Blizzard, Move to AWS (DDOS)

Sorani-barthilas · September 9, 2019, 5:29am

Calling someone in netops a sysadmin is the highest disrespect, I don’t enjoy having to clean up your mess.

We are currently expanding so quickly as people leave AWS due to privacy and data loss concerns that just this weekend we lit up 60Gbps of capacity in Singapore.

We have more than 80 crossconnects to AWS at this point because clients are refusing to leave sensitive data hosted there. You are a very small fry if you’re not aware of the privacy and dataloss concerns. Eventually once the management realizes the cost savings of having database heavy engines on hardware again as release cycles have died down, AWS gets shut down. This month alone was a 30-35Gbps shift away from AWS to our transit/colo.

Keep shilling I guess.

Clannad-darkspear · September 9, 2019, 5:31am

I’m confused as to why C2S is a thing if what you’re saying is true. LMAO

Also, the lack of proper design combined with confused management is always a good time. “iT r aWs fAuLt”

Valore-draka · September 9, 2019, 5:32am

I love watching geektalk. Seriously. I love it.

Sakuraba-stalagg · September 9, 2019, 5:35am

A shill would be better prepared

A shill would not be so ignorant as to call a datacenter a “server warehouse” lol

Cherrypie-skeram · September 9, 2019, 5:35am

Are you an AWS salesman? Seriously; I think the software engineers among us understand the appeal of leveraging the AWS cloud among others, but it takes a lot of hubris to believe that the software engineers at Activision Blizzard either didn’t consider deploying onto AWS.

We customers do have a right to demand that Activision Blizzard provide us with high availability, but we don’t have the mandate to demand where they deploy their infrastructure, nor do we have a full picture of the problem space to make such a recommendation.

I would leave it to them to figure it out, instead of designing software architecture by committee on the WoW public forums.

Clannad-darkspear · September 9, 2019, 5:36am

You already know it’s more than likely due to tech debt and already having massive server farms, with staff, that they didn’t feel like dealing with.

Racen-fairbanks · September 9, 2019, 9:27am

as someone who works in network administration, Sorani is the only person in this thread who knows what the hell they’re talking about.

Arkayenro-khazgoroth · September 9, 2019, 9:44am

oh well, if theres no need for anyone to tell you how wrong you are i guess we’ll just move on

Viscosity-stormrage · September 9, 2019, 11:36am

I also live on the moon full time

Wylkanth-frostwolf · September 9, 2019, 12:19pm

lol

Edit: Well I read most of this nonsense and TL:DR: OP is clueless. Move along.

Exziled-incendius · September 9, 2019, 12:43pm

AWS does offer some good DDoS protections with their WAF and some other mechanisms. However, DDoS protection is nearly impossible if the attacker(s) understand how this works.

The same DDoS guy on the first day also took down parts of twitch which is owned/hosted by Amazon.

Mantooth-bloodsail-buccaneers · September 9, 2019, 1:00pm

This is so true. So many IT managers want to put everything in the cloud now just to remove the responsibility of having to support anything.

Exziled-incendius · September 9, 2019, 2:13pm

Putting things on the cloud IS actually one of the best moves you can make. There’s no way you can build a fault-tolerant application on a “private cloud” because you simply will not have the resources at your disposal.

You should stick to your core competencies as a business. Blizz’s core competency is creating games… not worrying about this kind of crap. However, just because you’re in the cloud doesn’t mean all of your problems are instantly solved.

Bloodsong-deviate-delight · September 9, 2019, 2:16pm

AWS is not large enough for something on Blizzard’s scale. Besides, Blizzard is not self-hosting. They have servers all over the world.

Fabermor-frostwolf · September 9, 2019, 2:26pm

Clannad:

Fabermor:

Clannad:

Fabermor:

Clannad:

What gets hit? I don’t understand what you’re getting at in regards to saying something like AWS Shield Advanced can’t prevent this from happening for how long it happened. It WILL 100% prevent the majority of conventional DDOS attacks and a huge portion of larger companies leverage it. In the end let’s say I agree with you that a stateful MMO like WoW can’t leverage the same protection like web services. Shield Advance still provides 24/7 DRT (DDOS Response Team) services. Meaning if certain metrics point to a DDOS that you couldn’t 100% prevent Amazon themselves step in and handle it. It’s as simple as that. My post was saying they need to push off managing stuff like that on AWS and not trap themselves in their own internal mess. They aren’t a cloud company. Quit trying to be one.

Every enterprise data center service provider also has 24 hour response teams for networking… Blizzard’s Network is MANAGED, it is not something they them selves control aside from the LAN portion… Their partners do leg work, their only involvement is to approve access to workers to their cabinets if needed for serial connections or for log access

Yeah, but when you’re landlocked to your own internal data centers you aren’t able to provide anywhere near the services that a company like Amazon can throw your way. No way in hell. They could regurgitate their stack in a different region (different ISP as well) if they realized the DDOS was hitting a certain ISP. No way in hell can you do that if you are landlocked to your own data center. Blizzard isn’t rich enough for that no matter what you think. AWS by themselves make more money in a year than Activision-Blizzard is worth…

Except for the last time, in Blizzards case because they offer regionalized servers… cannot utilize MOST Of the AWS “DDOS” Shield serviceability with out horrendously screwing latency … So your West coast server, could only live within the west coast group so if all the pipes to that group got hit… You still have the exact same problem. When I worked at Microsoft we shared a data center with Blizzard… Who had more capacity than Microsoft did at the time (And Microsoft had 4 x 15GB pipes)

It’s cool you have a hard on for AWS, I’m a Certified Solutions Architect, my work uses AWS for our Applications… What your suggesting, AWS would build a custom solution for… which they’d do for about 5x the cost of what Blizzard has invested in each data center (and would pretty much be the exact same thing Blizzard has… They didn’t just hire some random basement nerds who don’t know how to build to scale)

And @Wildly, while data is data… The latency needed to access and relay that data matters… a lot, wow’s legacy client was a lot less sensitive and could deal with 100-300ms of latency and still be playable… Now days, thats not nearly the case

I don’t see how all of those pipes could be hit to the point it would cause the downtime we saw over the previous days. I think the difference in all of this stems from the post from earlier where Blizzard stated they have around 70 people managing these data centers. I’m willing to put money down that AWS has more than that sitting on their DDOS Response Team alone.

Also, I’m SA/Dev certified so nice man.

Than they really need to re-test you or revoke your certs, because you have no where near the grasp you seem to think you have on this topic

Exziled-incendius · September 9, 2019, 2:27pm

Are you serious? Do you even know anything about AWS?

Amazon and twitch probably get more traffic from all over the world in an hour than all of actiblizz gets in a day. On amazon’s “Prime Day” their NoSQL DynamoDB service handled > 7 trillion calls in 48 hours and peaked somewhere around 45 million requests per second. That’s just one small aspect of AWS, so can you imagine how many requests / sec they received through all of their infrastructure?

Blizz is a large company, but AWS could support 100% of blizzard in a heartbeat.

Clannad-darkspear · September 9, 2019, 2:30pm

lmao. AWS by itself makes more in a year than Acti-Blizzard is worth overall.

Zalanii-herod · September 9, 2019, 2:34pm

You can google ddos attacks for say last year and see these companies no one has ever heard of being ddos’ed like Ubisoft, Sony, Square Enix, Microsoft…

Clannad-darkspear · September 9, 2019, 2:51pm

Fabermor:

Clannad:

Fabermor:

Clannad:

Fabermor:

Clannad:

What gets hit? I don’t understand what you’re getting at in regards to saying something like AWS Shield Advanced can’t prevent this from happening for how long it happened. It WILL 100% prevent the majority of conventional DDOS attacks and a huge portion of larger companies leverage it. In the end let’s say I agree with you that a stateful MMO like WoW can’t leverage the same protection like web services. Shield Advance still provides 24/7 DRT (DDOS Response Team) services. Meaning if certain metrics point to a DDOS that you couldn’t 100% prevent Amazon themselves step in and handle it. It’s as simple as that. My post was saying they need to push off managing stuff like that on AWS and not trap themselves in their own internal mess. They aren’t a cloud company. Quit trying to be one.

Every enterprise data center service provider also has 24 hour response teams for networking… Blizzard’s Network is MANAGED, it is not something they them selves control aside from the LAN portion… Their partners do leg work, their only involvement is to approve access to workers to their cabinets if needed for serial connections or for log access

Yeah, but when you’re landlocked to your own internal data centers you aren’t able to provide anywhere near the services that a company like Amazon can throw your way. No way in hell. They could regurgitate their stack in a different region (different ISP as well) if they realized the DDOS was hitting a certain ISP. No way in hell can you do that if you are landlocked to your own data center. Blizzard isn’t rich enough for that no matter what you think. AWS by themselves make more money in a year than Activision-Blizzard is worth…

Except for the last time, in Blizzards case because they offer regionalized servers… cannot utilize MOST Of the AWS “DDOS” Shield serviceability with out horrendously screwing latency … So your West coast server, could only live within the west coast group so if all the pipes to that group got hit… You still have the exact same problem. When I worked at Microsoft we shared a data center with Blizzard… Who had more capacity than Microsoft did at the time (And Microsoft had 4 x 15GB pipes)

It’s cool you have a hard on for AWS, I’m a Certified Solutions Architect, my work uses AWS for our Applications… What your suggesting, AWS would build a custom solution for… which they’d do for about 5x the cost of what Blizzard has invested in each data center (and would pretty much be the exact same thing Blizzard has… They didn’t just hire some random basement nerds who don’t know how to build to scale)

And @Wildly, while data is data… The latency needed to access and relay that data matters… a lot, wow’s legacy client was a lot less sensitive and could deal with 100-300ms of latency and still be playable… Now days, thats not nearly the case

I don’t see how all of those pipes could be hit to the point it would cause the downtime we saw over the previous days. I think the difference in all of this stems from the post from earlier where Blizzard stated they have around 70 people managing these data centers. I’m willing to put money down that AWS has more than that sitting on their DDOS Response Team alone.

Also, I’m SA/Dev certified so nice man.

Than they really need to re-test you or revoke your certs, because you have no where near the grasp you seem to think you have on this topic

Aw, the personal attacks. Still haven’t seen any evidence brought forth showing their Advanced Shield, in tandem with their DRT, wouldn’t have stopped this in a fraction of the time it took Classic to get back on its feet. Keep trying.

Xanthak-durotan · September 9, 2019, 3:48pm

No. Its because the user base is so much more knowledgable than 3 years ago.