This guy thinks serverless is impossible to leverage in an environment were uptime is paramount and scalability is important to control costs. He just isnt even on the same planet. He’s out the back working with the pizza boxes and changing hdd’s on ancient blades for his customers paying $100 a month management fee.
Have you ever heard of a load balancer?
Managing 60k active connections is nothing. I handle 25M+ a day. Granted theyre not as long lasting as a wow connection would be, but this isnt hard. Blizzard underestimated severely or decided this was ok from a business perspective, and both of those are unacceptable. This is a solved problem.
4 Likes
This goes to my point of the rollout feeling like a garage-shop operation, but the subscription still feeling premium.
Have you ever heard of IO? I’m certainly down to compare resumes if you wanna just whip them out.
The actual server side application isn’t the issue. It take a surprisingly small amount of resources to run. This isn’t a stateless environment like normal distributed micro-service architecture, where AWS and Azure type services are really meant for, this is a monolithic application that actively manages 2 way connections with (I assume) some proprietary socket-io or SignalR like technology. Connections are established and validated, events are raised on one client, sent to the server, validated against other events from other clients, then the winning event is distributed to all active connections.
The bottleneck isn’t the computation, the bottleneck is managing the active connections in a fully state-full environment. They are (rightly) limiting the number of active connections to keep the game playable.
1 Like
That’s what my guild did. 12 hours later - full.
Not sure how you think day 1 additions became full in less than 24.
We’ve moved 3 times. Done with it at this point since Blizz isn’t putting a cap on servers, only online caps.
Yes but honestly you make a decision on the longevity and cost to operate an antiquated architecture and application design - this technical debt ultimately has a negative impact on return on investment in such a low profit margin market. If they spent some time modernizing the application things would have been a little better for them 
Ive still yet to see a que. EU and OCE may be different but so far I’ve only swapped servers once. Left Mankrik and went to Old Blanchy. Since i moved, not a que in sight.
dont forget armyboxers that runs 30-60 accounts at the same time im sure theres more than one
That’s kind of what I do, professionally. I go to companies with 20 year old software and convince them that they need to modernize, then show them how to get it done without a complete rewrite.
Blizzard needs my services, it seems like. But given what they are working with, I don’t think it could have been any better. Do you know the tech behind the “megaserver” model?
Also, you said technical debt, so everyone take shot.
This thread is such an amazing example of people spouting engineering w.o context. (Comparing an ETL pipeline to a MMO is hilarious btw).
A MMO is not a social network, Ad tech, or IOT processing. There is world building, social, and fixed/“classic” product expectations that have to be considered. Its foolish to try to boil it down to “build it for scale”.
Could they find a way to fit 20k people in a single zone? Im sure they could even w.o instancing… would that be fun or the classic wow experience… oops no but you are an engineer and that is silly product work and you do not worry about such aspects.
There is also the issue of spreading to thin, could they spin up 100 servers? I am sure they could…oh but in 2 months when the hype dies most would be a ghost town which causes more people to leave? Uh oh… but engineers often think of such things.
White Knights…the biggest plague of modern society.
2 Likes
I bet they got them dual-core processors on fire over there LOL
No, I dont know specifically what ‘megaserver model’ refers to. I have limited experience working in the physical environment - briefly managed physical assets - have a few customers who are on very monolithic applications and helped them move into aws but my experience is that these are always very frought. The trade off between paying for money to modernize the application to leverage aws/azure completely to achieve cost savings from these environments is just too much. I am sure blizzard was in the same boat and wanted to do as little as possible to deliver the application into what is their environment today. that is something unforgivable for an asset that brings in as much money as the wow community for activision, but a travesty that they made such a poor decision when they would have unlimited support through aws because of who they are. AWS would have written white papers on this and presented it at REinvent because it would have been such a great marketing story.
I played flag football in high school, I’m on the same level as Tom Brady.
1 Like
With applications like this, hardware is really the limiting factor. Even if it was moved to the cloud (AKA Someone Else’s Computer) the IO is really the bottleneck. Virtualization can’t really fix that.
Story time. I worked with a company that had this massive in-house reporting application for comparing prices of retail items. Target, for example, is willing to pay very well to find out what Walmart is selling all their items for, and vice versa.
Anyway, they had a $90k physical server dedicated to the operations required to build these reports, and they brought me in to argue with their DBA. They already heavily leveraged Azure, and the guy had told them many times that they couldn’t move the database to Azure because the cloud couldn’t handle the IO requirement for some of these ETL processes.
I dig into it, and it turns out the guy was right. Every test I ran and all my research backed him up. They eventually let the guy go and moved the server anyway, and then failed back over and lost 48 hours of production. It just couldn’t handle it.
Megaservers use the idea of having several layers of machines spread out the IO and make certain decisions before escalating the next decision to the next server, in a sort of “chain”. So the outermost servers have 30K small connections to clients, and each one has a single, large dedicated pipe to an internal cluster for process escalation, and the software is picky about what needs to be escalated. It doesn’t scale infinitely, and the return per server added diminishes quite quickly, but for normal operation, it allows a single “realm” to house hundreds of thousands of objects instead of just tens of thousand.
But, like a guy up above said, then there’s the whole product owner thing. Being technically possible doesn’t make it the right decision, for the experience they are aiming for. I only brought it up to illustrate the lengths some companies have gone to looking for a solution.
2 Likes
I realize you clearly state this is an assumption, but why this number? Couldn’t it just as easily be 30,000 or 50,000?
Thanks, this was really insightful for me as someone who spends all day managing and delivering web applications on cloud computing assets - I feel like this isnt neccesarily the place to delve into the concepts deeper so I’ll go off and do my own supplementry reading xD.
In Blizzards’ case, its well known they deliver their services through AWS (Sydney Region is where the OCE realms are hosted for example) so you would think that they are just going with whatever they can get cost effective to deliver as cheaply as possible - I mean, there are high I/O optimized assets available in AWS for example but if your application cannot leverage autoscaling then you cant really take advantage.
So not “Literally every server” then?
Thanks for confirming that the person I responded to was full of it.
So I’m 100% ignorant of how servers actually work. Happy to admit it, it’s not my thing. Here’s my understanding as is, and I would love some feedback from the experts:
It seems that the population caps aren’t actually a technical limitation, or at least they shouldn’t be, assuming blizzard is competent, but are instead a design limitation. Theoretically, blizzard could probably put every single wow player into the game under the “Herod” server right? But the game was designed (resources, spawns, etc) around a specific max population number, and blizzards goal is to push the servers to be close to that number asap. Right now for launch they’re using the modern layering tech to get more people into the game, but they keep swearing up and down that that is temporary. If they just open the floodgates and put every single person into the game, how will they crush it down to vanilla-like populations later? They’ve said they will open new servers as the old ones fill, and they are. I agree that it seems like they seriously undershot the number to begin with, but it’s too late to change that now.
I guess my question is what should blizzard do now? Even if they opened up a dedicated server and called it the Herod Relief server, and it had no character creation option, it’s only purpose was for free immediate transfers off Herod to Herod Relief, I’m pretty sure a lot of people still wouldn’t leave.
1 Like