“Sorry about the credit card numbers disappearing from my account so that you can’t bill me for any more time. I put a lot of research into the bug that caused them to vanish, but they cannot be restored at this time.”
23 Likes
I’m not sure that people understand how “backups” may not be the answer and may not even exist in a way that would resolve this problem.
However, it’s quite likely there is, or was, a path to recovery.
Generally, software services like this funnel fundamental actions, like object creation, modification, and deletion, through APIs (internal or external).
Also in general, these calls are logged with some detail. So whenever an object is created, something is written to a file somewhere describing that creation. Whenever an object is deleted, something is written describing the deletion.
Good design requires that all creation/modification/deletion proceeds through the API, and that there is no way to make these changes otherwise.
So if there is not some kind of “snapshot” (backup at a particular time) available, the next thing to do is “log diving.” You take what are probably some very large files and look through lines/entries that match the situation you are interested in. Say, “object deletion done as part of maintenance-process-x.”
Extracting and analyzing this information tells you what you need to know to confirm and perhaps reverse the undesirable change.
In my experience, they will have these logs, or they had them anyway, and if the logs are set up to automatically expire, the expiry is not going to be 24 hours, but more like 30 days or 1 year.
In any event, I believe Blizzard was well aware of the problem within 2-3 days of it starting, if not the same day.
Many services use what are called “canaries,” service objects that emulate customer resources, to alert them of changes that may be unexpected. So, hypothetically (and likely), Blizzard would have “canary users” with typical player resources attached. A wave of unexpected modifications ought to show up on a dashboard somewhere.
Well, anyway, this is how we ran things back where I used to work, and this is how everyone in the business does things.
Recovery process:
- If relevant snapshots don’t exist, go to the logs.
- Make a working copy of relevant logs, so that they don’t expire or go away for some other reason.
- Scan the logs for the events of interest.
- When the necessary information is gathered, use it as part of a procedure to restore the lost resources.
This is an approach that is commonly used.
If Blizzard let these logs expire (possible but unlikely as it is a terrible and obviously bad practice), or didn’t have them to begin with (I would find that impossible to believe), then there might be no path to recovery.
Bear in mind that there had to be logs to identify that the problem was occurring and to identify the nature of the problem. Those same logs would be the ones used for restoration.
I believe that this is occurring because management doesn’t want to do it, or fumbled the situation so badly initially that now they can’t.
The lack of appropriate messaging from Blizzard is IMO just as much a disaster as the issue itself.
17 Likes
At this point, everyone should know that nothing is safe in this game anymore. Last month it was guildbanks, tomorrow it could be your mount collection. You don’t own it - they owe you nothing. They will do nothing. They don’t care.
11 Likes
As a side-note, I notice the megathread in the Bugs forum got censored/closed about an hour ago - oh snap! 
I guess they really do want the victims to “just shut up and go away”
13 Likes
Technically speaking, as long as THIS thread exists, the bug reports thread is no longer needed, as the bug has been identified and there is a sticky thread here for it. So I wouldn’t worry too much about that at least. We’re rapidly approaching 50% of the post count of threads that have been pinned for a couple years.
Typically having multiple threads in multiple locations is counter-productive and it’s best to consolidate conversation about an issue in one place. Now if THIS thread gets closed, then we have a problem.
I would LOVE to be a fly on the wall in Blizzard HQ right now. It’s obvious that they are paying attention to these threads. Even if they refuse to give us any further responses, it’s pretty clear that people are paying attention and there are probably frantic conversations happening about what to do.
3 Likes
I pointed out earlier some simple math:
- Suppose there are 1 billion objects in WoW
- Suppose that it takes 1 kB to track the creation/modification/deletion of each object
- This = 1 PB (petabyte)
- 1 PB of “frequent access” S3 costs about $200,000/yr, but there will be a great discount to that (ballpark 70%), and the frequent access tier is probably not what’s needed either
- $200,000/7M = $0.03/customer per year
This gives you an idea of the order of magnitude of the expense of maintaining logs for this activity. It’s not expensive.
To add more context to that, WOW is an SQL database. Specifically, Oracle.
1 Like
Right. If for some reason object modification uses stored procedures, basic best practices will be to log those calls to a table then roll that table off into a text file.
Also: Oracle is brutally expensive and everyone despises dealing with them. A lot of organizations migrate off of it even when it takes years. AWS/Amazon is 100% Oracle-free now I believe and that was a truly epic effort.
2 Likes
Is this really that last response from a “Blue” on this incident?? After almost a thousand responses from unhappy (if not irate and/or outraged) customers?
Heh.
Earlier, I posted my own thread, wherein I listed the items that disappeared:
I also logged my own ticket:
https://us.battle.net/support/en/help/case/detail/100507149
Of the several THOUSAND items stolen (yes, STOLEN) from this specific Guild Bank by this glitch, I have received, to date, precisely NOTHING. No items restored, no other form of compensation. Just a pro forma apology issued from the executives on high, to be distributed by the GMs down to us unwashed masses of angry paying customers.
Congratulations, Blizzard, on becoming TRULY part of the Microsoft family of customer-indifferent corporations!
12 Likes
Ok so the EU forums work a little differently. They have a big thread in their CS forum which had been getting blue responses. And now like us, they also have a sticky in the general forum giving the same info as the one here.
CS thread
General thread
That way we can easily keep track of how our friends across the pond are handling the situation (since they’re having the same issue).
7 Likes
It is clear they don’t want to put in the man-hours necessary to make full restoration.
It doesn’t take many man-hours to provide some form of compensation. These items at the very least represent time spent, and we pay money to spend time in this game. Game time might be nice. I’m not a big money maker and I would be fine with a couple million in gold, but I know there are those who lost much more. Some people have suggested giving people special items they can’t get. Maybe give people one “wish” so to speak.
It doesn’t take many man-hours to properly apologize. A proper apology requires admitting specific fault. I personally see two main faults: Not properly testing feature that affect major infrastructure of the game. And esentually gaslighting players; making them think the items were safe and would be returned for over a month. “I’m sorry” is empty. “I’m sorry that I did (specific things) to you”, is more meaningful. Blizzard, you hurt people, and you continue to hurt people.
I think the apology should be as public as it can be. Instead of posting “what’s your favorite spec” on facebook, they should post their apology for all to see. Do that on all the social media.
12 Likes
The funny thing is, MySQL is capable of handling WOW’s back end just fine. And since they are both standard SQL, such a conversion would be possible.
1 Like
In order to support a dataset that large a lot of features beyond SQL have to be involved so it always turns out to be much more complex to migrate than expected, and usually it’s expected to be hard to begin with.
I personally am a PostgreSQL person ever since Oracle got involved in MySQL, but MariaDB is certainly an option for an alternative.
That is for damn sure.
2 Likes
Oh awesome, I’m going to replace MySQL on my Gentoo server with that, since it’s basically a drop-in replacement.
1 Like
There was a HUGE thread in the Bug category about the guild bank deleted items, that was still getting posts, until an hour ago when Blizzard locked and closed it…
This incident has had 3 major phases, and now they getting ready for the 4th phase:
Phase 1: Delete their items.
Phase 2: Do nothing and don’t acknowledge anything; wait nearly 2 months.
Phase 3: Post about “incomplete restorations” starting, but days later no one has gotten anything restored.
Phase 4: Bury this issue. Just make it go away.
15 Likes
As many others have stated, I lost EVERY item in SEVEN tabs in the guild bank. I received back, about 20 stacks (which seems to be better than most) of the HUNDREDS of stacks, unique items, mounts, pets, gear, etc.
What will be the compensation for this? Literally MILLIONS of gold worth of items… just gone? And you say “sorry”??
This isn’t good enough. You should be able to track the guilds that are affected by this and just drop some huge in their guild bank as compensation. Or a year of free play time. SOMETHING. Just a slap on the rear and “good game” isn’t going to cut it.
7 Likes
Actually, I would rate Microsoft CS as better than Blizzard CS, here’s why:
- usually a real-life human being (who may or may not have an Indian/foreign accent) answers the phone after a short wait with Microsoft support, compared to Blizzard ticket responses sometimes taking 1-2 weeks
- the real-life human being (despite the Indian accent) usually walks you thru your software issue/problem step-by-step and produces a fix/solution for you right there over the phone… while the first/initial ticket response from Blizzard is usually a “generic” automated message that doesn’t fix/do anything for your ticket issue(s)
- with Microsoft CS you get thru to a real-life human being “on the first try” (either thru phone or chat), but on the other hand with Blizzard CS tickets they usually need to be “escalated” or “appealed” a few times before a real-life human being actually looks at it
- typically with Microsoft CS you will get restored/made whole, but apparently Blizzard’s new policy is something along the lines of “we can’t do anything, tough luck” with players NOT made whole/properly compensated when a loss occurs
- etc
The last time I had a software issue (related to Office 2019) and needed help getting restored, the official Microsoft support line got me all hooked up/restored and the entire phone call took only like 18 minutes start-to-finish, was nice and quick… none of this “waiting weeks to hear something back” like we’ve witnessed with this guild banks issue
2 Likes
The bug report thread getting locked isn’t actually a problem, IF this thread continues to exist and not be locked. If this thread is locked, then your phase 4 will happen.
There really isn’t any need to have 2 threads in 2 separate locations about it. It is counterproductive and makes the issue more difficult to track. It’s far better to have all of the discussion about it consolidated into one post. I don’t fault them for this.
At this point, we’re just waiting. The backlash about this has reached social media and gaming sites, so they aren’t going to be able to ignore it. This is a huge blight on an otherwise great expansion. This puts into question Blizzard’s ability to continue to run the game at all.
7 Likes
Good points, I agree.
1 Like
I don’t know if you consider it a “major phase” but it all started with items that could not be moved in the guild bank. There were a few threads on that for a couple days before things disappeared.
1 Like