I don’t know how they run things on the backend, but I’m guessing some flavor of SQL. Unless they throw out every single practice known to companies with hundreds of millions of annual revenue (just WoW, not even full Activision Blizzard/MS), they would have some backups. Following best practices, they should have nightly backups and monthly backups, plus at least one prior to the release of the code that caused this issue.
Next, we know (because we can literally see it in game) that there is a transactional log of items put into and removed from the guild banks. There have been reports, and you can see references in here of people saying the same, that there may be some bugs with that, because people say they’ve noticed transactions not being captured (ie - it shows the 5 items being put in, but not when Bob pulled out 2).
Someone familiar with the tables that’s even somewhat decent at their job, using whatever their preferred ETL tool might be, could easily produce a list, with extremely high levels of accuracy, of every item that disappeared using a before and after snapshot with a transactional log of items manually added/removed by users between those two points in time. Personally, I’ve done something similar, using backups and audit logs to reconstruct point in time snapshots of what a table would have looked like during a month-end process, and it was accepted by auditors just as well as if it had been an actual database snapshot.
Given the apparent inability to restore items “due to how some of the data was lost”, leads to a combination of factors which must be true.
1 - The transactional log for guild banks is and has been broken for a while, and they don’t want to use it to restore the data. This, in my mind, is the most likely reason, because if this was working and trustworthy, then with backups, they could easily reconstruct the entire history and know what users might have themselves removed between snapshots A and B. Anything else not in B that was in A, and it wasn’t part of what users removed, is what was deleted. This transactional log doesn’t even require them to have data from the dawn of time. Transactional logs from the last 3 months is all we’d need, and people see stuff in those logs from before this time. So what happened to this data?
2 - They don’t have backups (or all backups got deleted/corrupted?) This seems a bit rough of an explanation, because even if you don’t have the snapshot from right before the release, that doesn’t matter if the transactional log is still good data. It just means more of the log has to be parsed in order to determine. Instead of the job running in 20 seconds, it takes 45 seconds (just random numbers, actual results depend on number or rows, format of data, etc.) The only way you cannot reconstruct the data is if you don’t have good before/after snapshots at all from any point in time (or, as mentioned above, problems with the transactional log.)
I personally wasn’t as hard hit as many. I lost a few dozen slots of items in my personal guild bank. I got back maybe 7 mails, including some large venom sacs and 2 34-slot bags. I got those mails in that “first wave” right after the announcement, and nothing since. Unlike others, nothing I lost was worth millions, or was irreplaceable. However, I also don’t fully know what I lost. I never took backups, screenshots or whatever of my bags. I know I lost things because one tab had been full, a dumping ground of old items to eventually sort through with no organization so just solid, and it became a checkerboard of items. I know I lost my stack of volatile fires, because there is no way I didn’t have any (and I still had a stack of all the others a couple of hundred of each). What else may have disappeared but I didn’t recognize because I just never expected to need to memorize or backup the data myself?
At this point, I don’t even care if I get the rest of my stuff back (although I do hope the rest of you get the important things you’ve lost.) I want to hear full explanations of what happened, how did you get to a point where you couldn’t reconstruct the data, and how do you plan to prevent similar scenarios from ever occurring again. I’ve been playing since Classic, occasionally with a second account. I’m reasonably certain my subscription on my main account has never even lapsed in all those years. I have the collector’s edition of every expansion except Classic. Giving all of that up would be hard, since this is where I also come to play with friends. But we need better from you, Blizzard.
You’re lucky, 21 guilds here, 5 of them missing 100% of items, and across all guilds I got 1 mail with 1 item in it. I can’t even bring myself to calculate how much I lost. Unlike Blizzard (which I still have to believe they do, and btw they use Oracle, it’s been all over their job postings for 20+ years), I took a backup of all my settings before the patch, and thus have a list of everything missing from my guild banks. It’s sad to think I do a better job of backups (I do nightly backups, as well as by hand backups before any update), but evidently I do.
They’ve never once said they don’t have backups. Heck, they’ve barely said anything at all concrete. All kinds of wishy washy “so sad, too bad” type of comments.
No you’re a Blizzard shill because you believe every lie they feed you and just go “ok, moving on.”
I know the amount of work it would take to (now) restore a backup, and try and reconcile the differences given the vast amount of data, and the amount of time it’s been (they should have shut guild banks off the moment this occurred to prevent this from getting worse like it did). And it’s really simple, the amount of work (man hours) is not worth to it Blizzard for the amount of people affected (complaining). It’s a simple business calculation, how much money will we possibly lose vs how much it will cost to fix.
And since it’s clear that Blizzard has kept this out of the main wow news sites (I’m talking about you WoWhead, Icy-Veins, MMO-Champion, etc.) that they are trying to minimize the amount of exposure.
But yeah as I said, keep towing the company line. I’m going to keep shouting from the rooftops until I get a reasonable explanation and restoration.
I’m known online as Kurn. I’ve been playing World of Warcraft, off and on (though mostly on), since 2005. I have served as a guild officer in multiple guilds, a healing lead, a raid leader and a guild leader. I have run a blog about WoW and my adventures within it since 2007. For a time, it was the most popular blog dealing with holy paladin stuff, through much of Wrath of the Lich King and Cataclysm. I had two podcasts to do with World of Warcraft as well. Finally, I have a small YouTube channel with almost 300 videos, most of which have to do with WoW or other Blizzard properties, which was started in 2006.
To be clear, I’m sharing this information so that you can understand that I am a long-time player who cares about WoW and I am someone who has given back to the community – and continues to do so.
I am writing this post today because of the guild bank issues that continue to plague hundreds, perhaps thousands, perhaps tens of thousands of guild banks. There are other issues that also need to be discussed, such as guild bank logs, warband bank issues, void bank issues and problems with reputation, not to mention weekly quests not resetting, though I have much less information on those. As such, I’ll keep my discussion to the guild bank issues, while bearing in mind that there are many systems that have been adversely affected since prior to the launch of The War Within. I did want to note that apparently the warband bank issues are related to the visibility of the items, but this does not seem to be the case with guild bank items.
TIMELINE OF EVENTS
On August 13/14 (depending on region), Patch 11.0.2 was released.
In the patch notes for this release, it noted that Cross-Realm Guild bank functionality was now available.
Shortly after this release, reports started to roll in that items were missing from guild banks.
US thread in the Bug Report forum:
EU thread in the Customer Support forum:
Of note, the US thread did not receive any responses from any Blizzard representatives, while the EU thread received several, mostly from Felranys, and many of these were apologetic in nature. These were mostly along the lines of “I’m sorry I don’t have any new news, details have not yet been shared with us”.
Due to how some of the data was lost, we’ve reached a point where the result will be an incomplete restoration for some guilds, and we do not have a way to restore the remaining missing items for them.
The important thing to note here is that this heavily implies that some guilds will have an incomplete restoration, yet the rest will have a complete one. However, based on the responses since then, it seems that the reverse is true. Some guilds had an incomplete restoration, while the rest have had nothing restored.
For myself, my personal guild bank is the one that’s affected. I received the following, after losing hundreds of patterns and entire tabs being wiped clean:
2x Pink Mageweave shirts
1x Wildvine Potion recipe
1x Wildthorn Mail recipe
1x Schematic: Dark Iron Rifle
Not even the items shown in the bank logs themselves as being deposited (and not withdrawn) were restored. I’ve read hundreds of comments and hundreds more forum posts from others who are worse off than me.
On September 23, the US thread in the Bug Report forum was closed without any response.
On October 2, the EU thread was closed with this response from Yryadorne, directing people to the general EU thread:
Almost two months after the patch, restorations still seem to be going out, though I still have yet to read about anyone being fully restored. (As a point of data, I have not received anything since that one mail in late September.)
RESTORATION OF ITEMS… AND TRUST
In my 19 years of playing World of Warcraft, I have never, ever, ever run into a situation with my character or a guildmate’s character where Blizzard did not do the right thing. As any of my former guildmates will attest to, I have opened a ridiculous number of tickets over the years to ask for Blizzard to fix something. A guildmate of mine had lost her Vashj vial for the Vials of Eternity quest (which led to the Hand of A’dal title). I have screenshots of me standing in Mount Hyjal, arguing passionately for my guildmate, and getting stomped by the incoming boss because I was talking to a GM about it. Ultimately, the GM discovered that my guildmate had accidentally dropped the quest – which had deleted her vial. Totally get it. It was my guildmate’s fault. The GM did the right thing in telling me that she had just picked up the quest a few days ago. The collaboration and communication from the GM in question was outstanding.
Years later, there was a bug in terms of normal/heroic switching in Blackrock Descent in Cataclysm, which meant that my raiding guild was stuck in two separate instances of BRD. Once again, I reached out to Blizzard for help and, once again, they did so. We went off to do Bastion of Twilight and the next raid night, we were properly synchronized to the right Blackrock Descent instance.
Never, in my long history in WoW, have I been let down by Blizzard the way I have been when it comes to the guild bank items disappearing.
(Yes, I know the GMs and community managers are different than the developers, and that the GMs are limited in what they can do. I’m using the GMs as proxies for my experiences with Blizzard as a whole. They take a lot of crap from users and they deserve better than they get.)
As a player, it is gut-wrenching to look at empty bank tabs where hundreds of items should be. And my items weren’t even all that personal. They were items I collected over more than a decade of play, storing away for a variety of reasons. Some are no longer available in the game. Some are available, but extremely limited in number (dragonscales, for example). And it just flat-out sucks that they’re gone. Part of me wants to trust Blizzard when they say, “we do not have a way to restore the remaining missing items”.
As someone who works in tech, though, I can’t trust what Blizzard is saying here. Blizzard has been running this game for more than 20 years (when you count alpha and beta and such). As such, they know how important data integrity is. They know that every byte of data matters. And they probably have PETABYTES of data. One gigabyte is 1024 megabytes. One terabyte is 1024 gigabytes. One petabyte is 1024 terabytes. Blizzard’s data probably is in the realm of petabytes. Worse, that data is always changing. Anytime you vendor a grey or disenchant an item or buy something or upgrade something or put something in the bank or loot something, the data changes. And that’s just inventory. What about levelling up? There’s experience to think about, reputation, how much mana or focus or rage or whatever you have, plus how much health you have. The data is always changing. It is almost never static.
Guild banks are included in that. Pull out 10k gold, deposit a BOE, withdraw a stack of Black Lotuses, drop in a stack of Flasks of Distilled Wisdom. It shifts, it changes. It’s not quite as dynamic as a character’s inventory, but in a large guild, yeah, stuff can be withdrawn or deposited, including repair costs and such.
What I’m trying to illustrate here is that, particularly for larger guilds, the state of the guild bank NOW is likely to be very different from the state of the guild bank when the bug occurred.
Let’s say that Blizzard has a snapshot of all the data from right before that maintenance period. The amount of work to reconcile all the changes between Day 0 (when the bug happened) and Day 50-something (which is the period of time we’re at right now) is likely to be A LOT.
So when a Blizzard representative tells me that they “do not have a way to restore the remaining missing items”, that’s where I’m stuck. That’s why I’m still out here making a big deal about this. You want me to believe that the devs who have made (and maintained) this game for 20 years did not make a backup of the game’s data before adding a patch to production servers?
I don’t buy that. Data integrity is vital. Anyone who has ever touched a live database knows that you back it up and, ideally, have another backup for it in another physical location, just in case.
You want me to believe that the devs who have made (and maintained) this game for 20 years did not keep backups of the game’s data for 30/60/90 days after the patch?
Super unlikely to me. Storage is cheap. For a game that just launched an expansion that still has subscription fees, storage is a pittance.
And yes, I know there have been layoffs. I know Blizzard has been affected. I’m saying that at this point, 20 years in, I’m pretty sure that any organization would have very specific processes for maintenance during the application of patches. What if the patch explodes and destroys things? Then they roll back to the backup and try to figure out what went wrong on their dev instances and then they try again. But they always should have the opportunity to roll back.
I’m not saying they should roll back NOW, mind you. I’m just saying that the data absolutely must exist somewhere. It doesn’t make any sense at all that they wouldn’t have this data from mid-August, prior to 11.0.2. That’s where I’m stuck.
Would it be an absolutely incredible amount of work to reconcile the state of guild banks between mid-August and now? Yes. My opinion is that if they have the data (and they should have the data), they can do it.
As such, I think Blizzard is saying one thing to make us think that they cannot restore things when, in my opinion, they are CHOOSING not to do so. And I think that’s a mistake. I’m not just saying that because I’m affected and I will benefit from them restoring the banks. I’m saying it because we’ve already lost trust in Blizzard by virtue of having lost the items in the first place, the possibility that they are able to and choosing not to hits even harder.
Everything in this game is a line in a database. If they won’t restore guild banks, who’s to say that this couldn’t happen to other things? And who’s to say that they won’t restore those other things? I don’t want to lose my Hand of A’dal title. What about my Rhok’delar? What about my Benediction? What about my reputation with, say, Timbermaw Hold? Or the Argent Dawn? What about my Feat of Strength, The Fifth Element? If guild bank items can get wiped like this, what’s to stop something from wiping out these achievements or items or titles? And what’s to stop Blizzard from NOT restoring them?
I get it, they’re not obliged to restore anything. I’ve read their end-user license agreement (EULA). I maintain, though, that there’s a difference between being obliged to do something and choosing to do the right thing. Further, they probably can’t be honest with us, due to the drama that any admission that they don’t want to waste the time on restoring the guild banks would cause.
So Blizzard is in a tough spot right now. The restoration of items may never occur. So will our trust in Blizzard ever be restored?
For me, it won’t be, not as it stands. I need a real explanation or a full restoration, ideally both, before I can put any faith back into Blizzard. Should I ultimately decide to step away, I know you won’t miss my single subscription. I would hope, though, that you have someone counting the number of people affected by this who decide to cancel their subscriptions. I have a feeling there will be more than few.
Well put. The problem isn’t that the items can’t be restored. The problem is their response (or lack thereof). Back in the day I would’ve expected a statement from Mike Morhaime about this.
They need to make some sort of effort to give some sort of compensation to affected guilds, even if the items can’t be restored.
for what benefit? what do they gain from making customers mad?
there’s another possibility. they have the data, but it’s not in a format they can use in the current system and they have no way to readily convert it.
They gain the time and effort (which translates directly to money) by not performing the restorations. Having manually reconciled databases myself (on an extremely small scale, especially compared to Blizzard), it’s a lot of work. Like, a LOT of work.
So I could easily see Blizzard saying “nah, don’t do it, don’t do too much to address it, it’s fine”. We don’t know how many people are affected, we have odd phrasing that implies one thing instead of the truth… it’s entirely possible this is the way they’re going if their major aim is to make and save money. Who’s going to miss my subscription? Yours? No one. It would have to be hundreds/thousands of people to make a real dent. And Blizz is (IMHO) counting on the new expansion to keep people distracted enough and engaged enough to overlook the guild bank issue. (Among other issues.)
It’s my two cents, but I think they’re really trying to just sweep things under the rug here.
An interesting idea, though if that’s the case, something changed between 11.0.1 and 11.0.2. I would think that it’s still a recent enough change that they could put data on a dev server, then send it through the conversion that went through on live (once they fix(?) what might have gone wrong) and then use that data. I think if that’s the issue, that could be easily automated. I really do think it’s the reconciliation between Then and Now that is causing issues.
Or the data is missing entirely. Which would be astounding.
Two things. Cost management and damage control. By not doing restorations they save alot of dev time which is better used to work on live game bugs and balances, especially delves which break in a new way every week. And by openly stating that restorations are coming, but may be “incomplete” they assure the playerbase as a whole that everything is fine, and Blizzard certainly didn’t lose millions of player items and tell them to pound sand.
Of course we don’t know if this is actually what they’re doing, because they’re not communicating AT ALL, which is usually a bad sign and why people are assuming the worst more and more now.
Ok. Do you have a list of items of what was lost? If so, how about compensating the gold value from the server’s AH? That’s the least Blizzard can do for not testing out the technical feature which caused the bug.
Many items required in-game time to gather, that equates to real world time. And real world time does have value. If an entire guild was harvesting ores/herbs/leather for the current expansion that can equate up to millions in gold. And seeing how a wow token is $20 usd to 200k gold. You can see where this is heading. Is Blizzard willing to at least compensate for the value lost?