I’m not going to give away everything, but I can answer this question.
We get raw data on basically everything that’s happening in the game across all modes based on win/pick rate, as well as things like damage dealt/taken etc.
We are able to parse through this data to look at it through many lenses. This is where raw statistics become less useful and the skill and experience of a designer come into play more so that we can draw meaningful conclusions from the data, and decide what changes we want to make based on the results.
To get a full picture of what’s going on in the game, for example, I can and do look at things like win/loss data that’s filtered by different leagues and game modes. I can also see hero pick and ban popularity with this kind of granularity, which lets me see, for example, what the meta is looking like in the Bronze - Silver range as well as the Diamond - GM range.
While incredibly useful, I want to emphasize that it’s not just the raw data that’s important, but also having the game knowledge and experience to make correct conclusions based on the knowledge. This is something that’s a lot more esoteric, and there are often multiple right and wrong answers, which can and often does lead to a lot of debate within the design team and with the playerbase at large.
As an example, let’s say that you’re the designer and your job is to evaluate Samuro in the next balance patch with these facts (I made these up in my head right now because i’m at home, they are not what’s actually happening though there are truths in these statements):
He’s currently at a 55% win rate
He has a very low pick rate, let’s say he’s the 5th lowest pick hero in the game.
He’s considered a highly skilled but highly rewarding hero by the bulk of the community who argues that he should deserve a relatively high win rate as a reward for being hard to play. They argue that if you nerf or remove what’s unique about him, then why would anyone play a hard hero when they can pick someone like Raynor or Lili every game and just win more often?
Others feel he’s frustrating to play against and a different section of the community thinks he’s obviously way too strong and needs nerfs. They can and do show you videos of crazy things that he does that look and feel unfair.
The people that play him love the unique things he can do, like swapping to clones and tricking his opponents, which is what his entire design is based on. If you decide to change these things, the likelihood of having an extremely vocal and negative and reaction from his community is very high.
Do you make changes? If so, what do you do?
I can guarantee you not everyone will agree with your proposed changes, which is fine and normal. A large part of being a designer is having the courage to make these kinds of calls and dealing with the fallout, even if your ideas are not always perfect. Even so, there are many cases like this and there are many times where changes need to be made. That’s essentially what we spent the bulk of our time figuring out and doing.
Also as a side note, finding out community sentiment doesn’t come from data, and community sentiment is often a large part of the picture as well, so if you only rely on data to make changes, you’ll end up making a lot of weird changes that no one will agree with.
As an example, Probius has always had an extremely high win rate and statistically deserves heavy nerfs. I don’t think that’d go over too well with the community