Here you go, a perfect example:
He was whining that the sample was too large for a 5% variance, which is obviously wrong if N / 4 = 62,500 & N * 0.05 = 12,500. The variance Blizzard was using is close to 5x smaller than the variance he was calculating for. He was saying a 5% variance must have meant Blizzard was using a much smaller sample than a typical day’s worth of ladder games.
So he made a very basic error in calculating the expected variance of win-rates. This is just one of the easier to understand errors that he made. But I don’t expect you to understand any of this.
And this isn’t complicated either. The formula for calculating the variance is:
n * p * (1 - p)
The variance peaks at 0.5, and there are typically 250k ladder games/day:
n * 0.5 * (1 - 0.5) = 62,500
Blizzard was using +/-5% aka 250,000 * 0.05 = 12,500. So Blizzard’s acceptable variance is well below the maximum variance of a balanced game for that sample size. So he was wrong on some VERY basic math.
Furthermore he assumed that win-rates are a completely random process with p=0.5, which they are not. There are map pool and meta fluctuations. There are biases introduced by the ladder itself, which tries to optimize for a 50% win-rate. Etc, etc. So he was using a basic statistical test, saying Blizzard must have been using a smaller sample, despite the fact that they clearly said they were using an entire day’s worth of ladder games, while ignoring that the ladder behaves more like a Lotka–Volterra system than a truly random process.
When you consider this, it makes perfect sense that a 5% variation in the win-rates was acceptable. His test was basically showing a 5% win-rate over that sample would have an incredibly low P-value and therefore could not be an acceptable tolerance for balance, but that’s assuming it’s a completely random process which it is not.