Extended Maintenance talk - System Engineer PoV

So I’m willing to open this can of worms at this point.

I work as a System Engineer where we have regular maintenance periods, and anytime we are about to deploy any of our patches, hotfixes, programs, etc… we test them in our dev environment well before we put these updates into our live environment. (yes I understand emergency patches/hotfixes) Anytime something does happen that forces us to extend our maintenance window, there is some form of accountability. I’m not saying fire half the staff, but we as customers don’t get anything more than a “we are fixing this issue” but nothing of technical information/feedback regarding the “why” these extended/daily windows are happening. Today’s maintenance to fix scaling should’ve been found way earlier in the testing phase of TWW in my opinion, but it is what is is.

The daily maintenance, going into the extending of maintenance this often is not an acceptable thing this early after launch. I understand that this is a very very big game, and there are issues that need to be hotfixed, but the main question is: “Who is holding these SysAdmin’s/Dev’s accountable?” and “Why is there no feedback from the team/company explaining why we’re seeing this amount of downtime?”

You have a bunch of people that have taken time off work, or may not have as much time away from work as others and this morning period of time for maintenance really matters to them. (no I’m not a part of that group, I still WFH :D)

19 Likes

Respecting players time isn’t something Blizzard is known for anymore. I miss the days of Blizzards good communication.

Maybe one day they’ll have a change of heart and it won’t feel so much as a us vs them.

I understand down time but all I’m saying…it’s very interesting that this wasn’t happening during the EA or, heck, even done before the game went live.

13 Likes

I am in that group, but I also realize things happen and don’t go according to plan. I’m disappointed that two days in a row my window to play has been cut short. Tonight, it’s been deleted completely, but I can play tomorrow.

Some info for the masses would make it feel a bit better, even if it was total bull crap.

“DEAR GOD! YOU GUYS WON’T BELIEVE THIS! So, we were updating the game and Tony spilled coffee all over the servers… AGAIN! Tony now drinks from a toddler sippy cup at all times. Things will be in order shortly. We have backups. Thank you for your patience. Please post on the forums about Tony. He deserves it.”

Just a new humor bit every time they extend maintenance. I’d hang around for the next one if they were actually funny. It’d be like watching a text-only comedy show.

2 Likes

I’m cool with it as long as they credit our accounts for the extra downtime.

4 Likes

Exactly, Anytime we have issues we have to report what those problems were to at least the general management, who will then pass the feedback onto the customers as they see fit. 9/10 it’s really no big deal as long as there is transparency…

lololol blizzard has NEVER had good communication they have always been like this

4 Likes

Love the armchair quarterbacking.

2 Likes

I agree. Not one disruption that I recall even happened. I’m willing to bet money they didn’t want to risk falling into “legal territory” about the 3 days of early access they promised to people who chose to spend over $90 to get it.

3 Likes

Unless you’re a systems engineer for an triple A game dev company, it’s hard to believe you

5 Likes

It has always been this way. After BC or WOTLK originals came out, there were periods of all day maintenance. I’m not saying its right or wrong, but the way Blizz has been doing maintenance has existed for a couple decades at this point. I don’t think its going to change while this game exists.

2 Likes

No, they used to be all over the forums and even interact with players in the game.

1 Like

I don’t work for a Triple A game company. I do work in an environment that requires a production environment to have 24/7, 365 uptime (not possible but the general public should know what that means). I’ve had my issues at 1/2am that require me to be on site/troubleshooting on multiple occasions in order to have things up and accessible for the next working day.

3 Likes

The scale of the work you’re doing is like tic tac toe compared to this game.

5 Likes

Triple A game companies aren’t alone in handling scale in fact they aren’t even a blip on the radar of internet companies that deal with large scale. Educate yourself.

7 Likes

Taking time off for the beginning of a new expansion is a wildly optimistic move. But I applaud that misplaced optimism… the world (of warcraft) needs more of it.

99.99% SLAs are a contract, which are paid with that it mind.

Blizzard doesn’t have that with us, their SLA says they can take the service down for any length of time for any reason, with or without notice.

B. Alterations to the Platform and Availability. Blizzard may change, modify, suspend, or discontinue any aspect of the Platform, Games, or Accounts at any time, including removing items, or revising the effectiveness of items in an effort to balance a Game. Blizzard may also impose limits on certain features or restrict your access to parts or all of the Platform, Games, or Accounts without notice or liability. Except as otherwise set forth herein, Blizzard does not guarantee that any particular Platform, Game, or Account, or any particular features or components thereof, will be available at all times, at any given time, or in all countries and/or geographic locations, or that Blizzard will continue to offer the Platform, Game, or Account, or all features or components thereof, for any particular length of time. Availability is subject to change at any time, although we will endeavor to use reasonable commercial efforts to provide you prior notice, unless the discontinuance arises from a matter that is beyond Blizzard’s control or causes the provision of such advance notice not to be possible or feasible.
https://www.blizzard.com/en-us/legal/fba4d00f-c7e4-4883-b8b9-1b4500a402ea/blizzard-end-user-license-agreement

4 Likes

The fact that this is a triple A game dev company is exactly why this person’s comment matters. Blizzard should be head and shoulders above. But they are not. They are subpar at best.

6 Likes

Big companies just don’t really care as no one is going to rage quit and even if they did its a tiny amount of money they will lose from billions.

Think of it this way, you pay for electricity and internet , if the power grid goes down, does the power company write to you with exactly what the problem is?

For e.g. Dear valued customer, Johnny drilled to deep while putting in his new fence and broke a cable, we are now redirecting power from sub station 4, we have dispatched truck number 6 and its eta is 5 minutes.
No the power goes out its fixed when its fixed.
If the power comes back up a hour late are they going to fire there crew in truck 6? No

Companies don’t like to broadcast the details of their failures. That just compounds the failures and causes customers / stakeholders to ask follow-up questions that companies aren’t going to want to answer. This is pretty elementary.

Does your work environment require real-time interaction between 10s of thousands of online users, with minimal latency? I’ve seen what happens when a game company rushes out changes with little or no downtime. The results vary from unplayable lag to game-stopping bugs and crashes.

I’ve worked in places that have little or no tolerance for downtime - casinos are one example. Updates to customer accounts being continually sent from 1000s of machines, incoming feeds that update the odds of 1000s of games and racing events from around the world, online gambling interacting with mobile and online systems (which are being updated real time with odds, results, in game events), not to mention all the hotel reservation and hospitality data. All of that barely scratches the surface of the data the larger casinos in Vegas handle across their networks… and yet I would be surprised if it was a fraction of what a WoW server has to deal with. So while I also hate downtime, I appreciate that at the beginning of an expansion, it should be expected.

1 Like