[Classic] The unacceptable state of Classic servers

This is going to another long post. Sorry.

I’m not taking any sides with what I am about to post (though it may seem like I am), but simply I am trying my best to take an educated guess as to what problems Blizzard is facing and deducting a (possibly wrong) conclusion from what Aggrend posted.

I’ll start by saying that I too am a Software Engineer with many years of experience working on complicated and diverse systems. I currently work on Self Driving/Autonomous Vehicle systems for one of the largest auto manufacturers in the world. To say that we need highly optimized code that can processes thousands, if not millions, of actions in a short window would be an understatement. Scalability is insanely important for us. If our systems fail, people could be seriously injured or worse.

Now you claim to also be a Software Engineer, but seem to have overlooked the constraints and limitations Aggrend mentioned in his post. I personally don’t feel our profession was disgraced, but instead some insight was given on the challenges Blizzard is facing. As an SE, I personally am intrigued by complicated problems. Isn’t that why we got into the industry? (jk it’s the money, obviously)

Given those constraints and limitations, it’s very believable that a viable solution does not exist (a solution beyond “just download more RAM 4Head” or a full on re-write). If those constraints and limitations were lifted? Then who knows. I am sure there is an ideal world somewhere in which the most perfect solution exists. A world with unlimited time and resources. However, as an SE you should understand that the ideal world with unlimited resources doesn’t really exist…

But maybe a solution does exist… We were also kindly informed that Classic is using the same infrastructure and technology as Retail, but Retail doesn’t see this problem too often. Trolls in General Discussion will say it’s because “everyone quit retail lol xD” but in reality it’s because the population is more spread out across hundreds of realms as opposed to like 4 and things like Cross-realm, sharding etc make it easier for the back-end processes of the game to handle millions of transactions. However, the Classic community has been very vocal about not wanting this technological solution. So then maybe given this constraint, the solution doesn’t exist? Can’t have your cake and eat it too.

You also mentioned the word Triple-A, implying that you are comparing to other games? WoW is very unique so it’s really hard to compare it to other games. I’m speaking from purely an engineering standpoint. Obviously every game is unique, but you would be surprised to learn that a lot of games use very similar technology when it comes to solving their problems. However, those solutions won’t necessarily work for WoW.

Let’s go over a scenario really quick based off what Aggrend described in regards to the persistent database that the entire game sits on, something any engineer could quickly imagine and then realize… oh yeah, this is complicated. (Oh btw, I might be wrong with all the small details but this is my guess on how things work in WoW).

You’re doing dailies and a new creature spawns in the world. Let’s say you’re playing a Mage and you cast a Frostbolt at this creature. A few things need to happen here. First of all, everyone in render distance of the creature needs to be sent a bunch of information regarding this creature (stuff like health and mana pools, geolocation information etc). Now when you cast your Frostbolt, information is sent from your client up to the server and then back to you. This information could contain things like how much damage your Frostbolt is going to do. But oh wait, we live in an open world where stuff like this needs to be broadcasted to hundreds(thousands?) of players at once. But oh wait, there’s hundreds of players at once doing the same thing to hundreds of different creatures or even other players across dozens of zones. Now all of a sudden you have thousands, if not millions, of transactions being written to a database somewhere concurrently. And given the nature of WoW, all of these need to be in near real time or the game play experience diminishes very rapidly.

Do you see how quickly the problem becomes exponential? Do you see why the database quickly reaches it’s limit when the concept of a mega-server exists? As Aggrend described, this is why queue times exist. There are software and hardware limitations to all of this. It’s like being stuck in traffic on your way home from work on a Friday evening. There are only so many lanes on the highway and there are only a finite amount of cars that can fill up those lanes before the average speed limit starts to drop, eventually leading to a full on traffic jam. Yeah the city/state can add more lanes, build more highways… but that takes time and resources. We all complain about not being able to merge onto the highway, so it’s natural to complain about queue times too. I’m not defending queue times. I hate them just as much as the next guy.

Now before people start arguing “but other games do it with no issues!!!” I will say that’s a fair statement, but that’s not always true either. Have we forgotten the FF14 Endwalker release that quickly? Anyone ever do the Eternal Battlegrounds in GW2 before? World PvP in ESO? Lots of lag and server stability issues, all the time. This problem is not unique to WoW. It’s the nature of MMO’s. Has anyone ever played an MMO that had 100% server up time with no lag, no stability/performance issues and a 100% perfect experience? Now I’m not saying we shouldn’t demand perfection. We absolutely should. We’re playing $15 a month to play the game… we should be able to play. We have every right to complain. But we should also realize that we don’t live in an ideal world and stuff breaks.

As far as non-MMO games are considered, it’s not even worth making the comparison because it’s comparing apples to oranges. Games like Overwatch, Call of Duty etc are session based games and should never be compared to an MMO. Yeah, Call of Duty doesn’t have queue times even though millions of players are logged in at once. Are they all interacting with each other at once? Nope. It’s sad that people have even tried to make this comparison.

One last point… Take something simple like changing your avatar or banner in Call of Duty, Overwatch, League of Legends etc… A lot of that is handled client side. If I go and change a spray or select a different skin, none of that information has to be sent back up to the server in real time (like in our Mage casting Frostbolt scenario) unless I am ready to join a multiplayer lobby. It’s like when you “Like” a post on Facebook. Sure, the front end does some magic and you see the thumbs up button light up and it shows you have “liked” that post, but because so much of the load is distributed to the client side, it doesn’t matter if it takes a few extra seconds to get to the back end.

What I’m saying is, in WoW we don’t have the luxury of allowing things to be delayed like that. When an action occurs anywhere in the world, the transaction of client → server → client(s) needs to be in real time. A situation very unique to WoW.

If WoW was written from scratch in 2022 would the engineers make different decisions? Yeah probably. But the fact is that it’s not written in 2022. The fact is none of us here have any idea what the code base looks like, what their tech debt is or what hurdles they have to overcome. To come out here and imply that they are giving us false information or lying about their situation is just… immature. And I don’t mean that solely directed at you. We as a community should be better at not devolving important conversations into trolling because we didn’t get an answer we like.

32 Likes

With all due respects,
I find your post rather condescending, saying this to me as if I didn’t know what client-server was or the basics of game development and insinuating that my post was devolving the conversation into “trolling”.

But I understand why people are skeptical, so let me lay down some technical stuff to hopefully put some minds at ease here so we can focus on the whole point of my rant rather than write novels on how difficult it is to be a game dev.

I’m going to correct you on your scenario.

1 - The client isn’t going to tell the server how much damage the frostbolt did, that’s just giving cheaters a free pass to send false information to the server, like telling the server that your frostbolt did 1 billion damage.
Might even break your server if some smarty pants decides to send a value larger than your value type and you don’t have the proper error handling for that. In multiplayer games, a lot of the work is done by the server and very little information is received from the player. (Emphasis on the bolded part here)
This in turn means that you also don’t have to send much more information to the player than the character position and the numbers for the client to use for the UI (Like how much health your target has, how much damage your frostbolt did so the combat text and log can show the right number, etc.)
Furthermore, when it comes to database, you could queue the data on the server as to not be spamming database requests. Good example of this would be metrics, you don’t just send metrics to the database whenever you have one, you queue them up and send them in bulk because they are far less important than gameplay data.

2 - You don’t broadcast to players in realtime, that’s just completely unrealistic. You broadcast on a tick of whatever timer you set. It’s why players aren’t synchronized with one another, what you see is an afterimage of other players.
You can see that in action when using Line of Sight in PvP. Try having someone cast a spell on you and hide before it ends. Ask them where you were standing when they finish casting the spell, it’s not gonna be where you see yourself on your own screen.
If there is one thing game developers are good at, it’s cheating, because you have to cheat to make something look good when in reality it’s just an illusion. Like how only the front of an asset get rendered while the back isn’t, or how older games uses doors and hallways as a way to load and unload rooms from memory.
This also ties in with #1, you can cheat by having both the client and server do the damage calculation so that the client can show you a number before the server even tell the client what really happened and if they don’t match, server’s value takes priority and the client adjust.
This can allow less messages sent from the server. Sending information in bulk is more efficient than sending it one at a time.
So no, it doesn’t have to be in “near realtime”.

See, here’s the thing… If we’re going the comparison route, yes, WoW has to deal with far more traffic than a game like LoL does, but on the flip side, it doesn’t have to be as precise as LoL needs to be. WoW can cheat in far more places than LoL can. You don’t get to just delay information in a game based solely around e-sport and milisecond gameplay like LoL.
Apples and oranges, yes, but WoW isn’t unique in how complex its challenges to overcome are.

Is it a simple fix? No, I never claimed it was, my claim was that it was doable, because it is. The question wasn’t whether it’s worth doing or not, because only the WoW devs themselves can answer that question.
If they find themselves stuck with too much constraints from the whole #nochanges, then perhaps they should communicate this to the players and tell us what the options are, not wait until the problem is causing major service degradation and preventing players from playing.

56 Likes

I never said the client is telling the server how much damage something did, I said somewhere along the route of client → server → client that calculation is done. You should re-read my post. We don’t work on WoW so all we can do is speculate. You’re just picking convenient talking points to pick apart that fit your narrative rather than actually trying to understand the large scale problem at hand.

Also it’s likely not true that very little information is received from the player. A lot of information HAS to be received from the player. Your movement in the world relative to other things, what spells or abilities you are casting/using, item usage etc. There is A LOT of information going from the client to the server, and vice versa.

I’m not even going to get into point #2. There is just too much to write about.

In any case, this is a pointless argument. I even as much said that a lot of what I said about the trolling etc was not directed solely at you, but the community as a whole. It was your choice to take it upon yourself and feel attacked by it. The part about you claiming we have been given false information or lies? Yeah. You did that.

It’s okay to just read something from a neutral perspective once in a while and not take things so personally. It’ll go a long way.

Anyways, have a good day. :slight_smile:

7 Likes

https://web.archive.org/web/20141019021233/http://us.battle.net/wow/en/forum/topic/13087818929?page=6#114

I’d say that’s pretty “near realtime.”

12 Likes

Thank you!

This was a great find and a great read. Guess I wasn’t too far off in my assessment then. 1ms seems pretty near realtime to me. :slight_smile:

2 Likes

Yes, but that is just one aspect of the game. Spell batching needed to be handled faster but it doesn’t mean everything is handled at that rate.

Again, I don’t know their code, I can only speak from an outside perspective here, but I doubt stuff like the chat system for example is working on the same loop as ability usage.
And he didn’t go into details on how the ability system work, nor should he, so who knows what kind of hack/cheat they do in that code to optimize it.

Take mounts inside building as an example, how you can ride them for a seemingly random amount of time inside a building. If you hit a movement key, you are more likely to get dismounted early, that right here is a proof that they have several layers of response. Some happen every few milliseconds, other happens every other second. Movement itself seems to be on a slower loop than abilities, going back to my LoS example from my post.

You’re not running the whole game in realtime and even abilities I doubt they are really processed in real time, just really really fast and decoupled from the rest of the game to prioritize them, might even be done on a case by case basis.

51 Likes

Time to reboot this thread now that WotLK is live and the queue problem has only gotten worse.
What’s the plan now Blizzard? :rage:

Queues are worst than in pre-patch and we still can’t get a sensible solution offered.

If you want people to transfer, you need to provide guarantees that we can come back to our server WITH our character names once the servers dies back down.
Because right now, most people would rather take the queue than deal with blindly using your free character transfer that has no guarantee that we won’t have to deal with the same issue that led us to transfer to megarealms in the first place.

Lock character creation entirely on megarealms (Even for people that already have characters on those realms) to avoid people taking character names that aren’t theirs and give a guarantee that once the servers dies back down that we can freely transfer back.
I will not pay to transfer my characters ever again, nor will anyone I know, I have friends that would rather quit than deal with any of this, guilds are being broken apart, and I will not blindly take your half arsed solution without a guarantee that I won’t get burned by it, nor will the majority of people I have spoken to.

Your company deserves absolutely 0 trust from us that you will do the right thing for the players, so the least you can do is provide some accountability for us to hold against you should you fail to provide what is even the bare minimum you should be doing for the players.

86 Likes

It’s been a week and we still have 0 word from you Blizzard.

Queues are still unacceptably high and you still won’t make any changes to your current failing plan to fix the issue. You won’t speak to the players about it either, too content to just ignore the issue and hope it goes away by itself and people forget about it in a month or so.

Raids are coming out soon, guilds won’t be able to do the raid because part of their guild can’t log on, most people on the mega realms couldn’t even level to 80 because of the service they pay for being unusable with no reasonable solution to deal with it.

This was unacceptable a month ago during pre-patch, it is beyond unacceptable now, I don’t even have the vocabulary knowledge necessary to describe just how horrifyingly bad of a company you are. We’re closing in on a month and a half of this non-sense and STILL you won’t offer the players anything reasonable, STILL you remain silent.

I mean, just look at Benediction queue since WotLK launch:

Absolutely ridiculous.

27 Likes

As a person with a regular job you don’t have a chance to play on some servers… While others play for hours or keep their characters ingame with hardware or other irregular stuff. To transfer on a server with low pop on classic is no help… you cant play anything x realm and auction house is also not x realm…

24 Likes

You would think they would refund gametime for the people who cant even play because they dont want to fix the problem. I think people just want some acknowledgement and promises because this is like the 3rd launch and the same problem. Why commit to keep playing when they cant even fix the problems that keep happening

21 Likes