We’ve all heard it a millions times at this point:
OMG OMG TRAP LOST IT MEANS PROTOSS ISN'T OP!
vs
OMG OMG HOW DID TRAP EVEN MAKE IT TO THE FINALS! PROTOSS IS OP!
With the dueling narratives of statistically inept dullards squawking across social media I thought I’d do a debunking of a common misconception: small samples of games are not statistically significant enough to support either conclusion.
We can calculate this quite simply. There are a lot of factors that can affect the outcome of a match ranging from skill to luck to how the players are feeling on that particular day. Any given match can have any possible combination of these factors, but we REALLY want to measure balance so how do we prove a match was caused by balance, and not these other factors? It’s very simple: for a sample of games, there must be a low probability that the outcome could occur if we assume the game is balanced. This is called the “statistical significance” and it is usually set to 1%. If there is a <=1% chance that this could occur given a balanced-game assumption, then that assumption is false.
To calculate this we use a binomial probability calculator:
https://stattrek.com/online-calculator/binomial.aspx
We punch in 0.5 for the expected result, meaning each player should win 50% of games (this is our balanced game assumption). We punch in 5 games since Dark vs Trap was 4-1 which is 5 games. We punch in 4 successes for the 4 games that Dark won. Then we click “calculate”:
https://i.imgur.com/Wv2PMz3.png
We now look at the “Cumulative probability: P(X >= x)” result, which tells us what the odds are that at least this severe of a result could occur. Those odds are: 18.75%. So this outcome could occur and is quite likely to occur under a balanced-game scenario, and is nowhere near our 1% statistical threshold, so we can definitively say the results are meaningless.
Just for argument’s sake, what would happen if we bumped it up to 100 games, but kept the win-rate the same? The odds go down to 0.0001%, which is well below our statistical threshold. Hopefully we can now appreciate the importance of sample size: to get a high statistical significance, you need a large sample.
Well I happen to have a large sample that uses far more sophisticated analysis techniques. The algorithm measures players’ performance in symmetrical matchups as a baseline for their skill, since symmetrical matchups are not affected by balance and correlate highly with the true skill level of a player. This can be used to measure trends for all players of a race. Cumulatively this represents all pro games recorded, aka tens of thousands of games, and the statistical significance is exceedingly high:
https://i.imgur.com/q5uzclg.png
The slopes of the lines are what matters. They measure how race does in a matchup relative to their symmetrical matchup performance:
TvP: 0.883
TvZ: 0.905
PvT: 0.907
PvZ: 0.902
ZvP: 0.887
ZvT: 0.888
As you can see, Protoss do amazing in both PvZ and PvT, netting 2.4% better performance than Terran in PvT, and 1.5% better than Zerg in PvZ. Interestingly, Terran does 1.7% better than Zerg in ZvT. It’s important to note that these are % of elo rankings, so it means the imbalance scales with skill. What is the total expected win-rate? Well we have to integrate the elo win-rate function for all skill levels:
https://i.imgur.com/rBIpKvq.png
The integral of this (assuming all skill levels play the same number of games) for PvZ is: 54.5%.
We can verify this over on Aligulac, which shows Protoss have a 51.69% win-rate (meaning the lower skill spectrum is playing more games):
http://aligulac.com/misc/balance/
Conclusion:
- A proper statistical analysis is complicated and difficult to do.
- Balance heavily favors Protoss in both matchups.
- The GSL Code S finals don’t matter for balance.
.