The effect of skill, quantified. DR278

Multiplayer Discussion
1

PART 1: ONE PARTICULAR MATCHUP

Let’s take one specific matchup: Mech Rogue vs Enrage Warrior. We’re not going to be looking at any other decks for the moment.

If we go to yesterday’s VS report, here are the raw matchup winrate numbers (in Mech Rogue’s favor):

  • Top 1000 Legend: 53.38% over 434 games
  • Legend: 54.41% over 944 games
  • D4 and up: 54.84% over 1310 games

Now first off, VS always fudges the last digit or two, I presume because they want to catch people stealing their numbers or something. 53.38% of 434 is 231.67, and assuming an integer number of games won we’d round to 232, which is 53.46%. So REALLY what we’re looking at is:

  • Top 1000 Legend: 232 of 434 games (53.46%)
  • Legend: 511 of 944 games (54.13%)
  • D4 and up: 718 of 1310 games (54.81%)

If we want D4-1 instead of D4 and up, we can now math that out. 718 minus 511 is 207 and 1310 minus 944 is 366.

  • D4-1: 207 of 366 games (56.56%)

So the difference in the matchup winrate for Mech Rogue vs Enrage Warrior, going from D4-1 to top Legend, is 3.1%. I contend that this particular winrate effect is 100% the consequence of the skill of the pilots of both decks increasing from average Diamond 4-1 skill to average top 1000 Legend skill. This belief is key to everything that follows. I think it’s an easy ask, but if you disagree well I guess you won’t find the rest very interesting.

PART 2: ESTIMATED WINRATE CALCULATION

So the core problem that we’re trying to solve is: when we’re going from a deck’s overall winrate at Diamond 4-1 to its overall winrate at top 1000 Legend, what percentage of that is skill?

The key to this is the estimated method for calculating winrate, as opposed to actual. To be clear here I didn’t choose these terms. Believe it or not, I haven’t invented every nerdy way of calculating Hearthstone statistics, and the estimated winrate calculation method was made by someone else and popularized by the Vicious Syndicate people. You might want to Google it (there’s a pretty solid Reddit post on it out there), but here’s a summary: you fill in the entire matchup winrate table for the meta, then for each deck you take its row on the table, multiply each value by that opponent’s deck popularity, then add all the results together to get overall winrate. In essence, matchup winrate times deck popularity equals overall winrate.

The way that we solve the problem posed above is by calculating the estimated winrate for a fictional meta. See, when I said we’re going from Diamond 4-1 to top 1000 Legend, that means going from D4-1 matchups × D4-1 popularity, to T1KL matchups × T1KL popularity. But let’s say we had a hypothetical meta with D4-1 matchups × T1KL popularity. Going from D4-1 to this hypothetical would isolate the effect of meta on winrate, because only popularity is changing. Going from the hypothetical to T1KL would isolate the effect of skill, because only matchup winrates are changing. With both the effects of meta upon winrate and skill upon winrate quantified, we can express each as a fraction of the total change.

PART 3: RESULTS

So I did the calculations described above. You can see my work here:
https://docs.google.com/spreadsheets/d/1HpNl7XpC8JvFwqQiwHE_rDNxKQJnf27GO9GZdGFtgkI/edit?usp=drivesdk

And here are the results. In the table below “skill diff” is how much more (or less) the deck would win due to skill differences IF the meta didn’t change with rank, and “% skill” is comparing the absolute value of that change to the sum of the absolute values of the skill change and the shifting meta change.

Class Deck skill diff % skill
Priest Control 2.34% 74%
Mage Rainbow 1.23% 46%
DH Relic 0.95% 59%
Warrior Enrage 0.77% 66%
Warrior Control 0.65% 36%
Rogue Secret 0.23% 64%
Rogue Miracle -0.16% 12%
Druid Ramp -0.24% 11%
Rogue Mech -0.34% 59%
Warlock Chad -0.38% 84%
DK Unholy -0.47% 57%
Hunter Hound -0.56% 94%
Warlock Imp -0.64% 67%
Shaman Totem -1.31% 64%
DK Blood -1.72% 89%
Druid Aggro -1.73% 94%
DK Plague -2.27% 90%
Druid Drum -2.28% 61%
Paladin Pure -2.55% 61%
Hunter Arcane -2.82% 93%
Warlock Curse -2.89% 90%
Priest Undead -2.97% 76%
Warlock Control -3.20% 80%
Paladin Earthen -3.64% 88%

For the vast majority of decks, skill effects winrate more than deck popularity. There are four exceptions currently:

  • Ramp Druid and Miracle Rogue are bad decks that top Legend players have an obsession with, and they try to make them work (by which I mean present opportunities for skillful play) when they simply won’t.
  • Rainbow Mage is almost a quarter of the meta at top Legend, so it makes sense that people would metagame against it extremely hard. However, it’s the second best deck in Standard at rewarding skill, and unlike Control Priest, which is a bad deck of you’re not a great pilot and becomes average if you are, Rainbow Mage is average to start out with. It’s a testament to the “skill factor” of the deck that it almost, but not quite, counteracts the attempts to bring it down, because without the hate it’d be over 52% winrate AND over 20% popularity.
  • Control Warrior is the one actual example of a deck benefitting more from the top Legend meta than from skill factor. Yes, the deck has a positive skill factor, but it gains almost twice as much from the top Legend meta just kinda being favorable to it. A genuine exception to the rule.

In any case, I hope you’ve enjoyed my TED Talk on how skill has a bigger effect on top Legend winrates than it being a pocket meta. Feel free to discuss.

8 Likes
2

Thank you for working it out. It is certainly congruent to my own experiences. Players at top 1k (where I am in for at least half a month every month) are definitely better at the game than sub 1k, and the difference starts to become more dramatic as you go towards the dumpster. People who are hard stuck in diamond are considerably worse than people who make legend regularly also.

1 Like
3

I’d specify “worse pilots.” One uncharitable reading of this has you calling them worse people, and I don’t think anyone is a worse person just because they have less Hearthstone piloting skill. It’s just a game at the end of the day.

I agree otherwise though.

4

Im trying to understand tired af writing this…

So hunter has the deck that is the most skill based ? Its 93%?

5

I believe it’s a value of what % of the change in win rate between lower and upper brackets is based on skill differences compared to the meta.

That can work for and against a deck. What hunter is showing is that it loses win rate as you climb, and that’s largely a result of people playing better against it, rather than it running into a new set of decks that happen to beat it.

1 Like
6

Small sample combined with small margins, at a glance it seems like there is a reasonable chance of false conclusions. Love your effort as always!

2 Likes
7

Or don’t want to win for its own sake so try and carve their own direction with decks, with all the pitfalls that entails. Smaller %, but important.

1 Like
8

Or do not want to spend as much time grinding.

1 Like
9

Can you rewrite your explanation in plain English?
I spent 15 minutes reading your post and couldn’t understand starting from Part 2.

I understand your Part 1 is basically saying:

Mech Rogue v Enrage Warrior Has Winrate (in favor of Rogue):
D4-D1: 56.56%
Legend Top1k: 53.46%
Your hypothesis is that the 3.1% difference comes from skill.

Your Part 2 proposed a testing method. However, it’s really hard to read.

Thanks

1 Like
10

Because it’s gibberish that he makes up. Don’t waste your time.

The whole post is ridiculous and should be taken ironically because it’s factually just a pile of mush… and I can’t even read it because I ignore this person.

6 Likes
11

And, approximately none of them made their own decks.

1 Like
12

? I’m exactly one of those. Got stuck for ages at diamond until my skill got to the point I overcame “flaws” in my deck. I don’t use deck builders, I make my own deck.

2 Likes
13
  1. Overall winrate = matchup winrate table × opponent deck popularity

Let’s imagine a simple 3 deck meta as an example. Say that Deck B has a 54% winrate against Deck A and a 44% winrate against Deck C. Obviously it has a 50% winrate in the mirror match, like all decks do. If Deck A is half of the meta, Deck B is a third of the meta, and Deck C is one sixth of the meta, then Deck B’s overall winrate is 51% — .54/2 + .5/3 + .44/6 = .51.

  1. Isolating the effect of skill

Overall winrate in Diamond = Diamond matchup winrate table × Diamond deck popularity
Overall winrate in Legend = Legend matchup winrate table × Legend deck popularity

So if we abbreviate matchup winrates as M and deck popularity as P, when we look at the difference between these overall winrates, we can visualize this as

M(D) × P(D) → M(L) × P(L)

Because both matchups AND deck popularity are changing at the same time, the effect of matchup winrates is not isolated. We can’t tell yet what part of that change is due to meta and what part due to skill.

So the way to fix this is to create a pit stop in a hypothetical meta in between.

. . .
M(D) × P(D) → M(L) × P(D)
↓ ↓
M(D) × P(L) → M(L) × P(L)

In the chart above any move to downward isolates the effect of deck popularity on winrate, because matchup winrates don’t change. Any move rightward isolates the effect of matchup winrates (skill) on overall winrate, because deck popularity doesn’t change.

Because matchup winrate data is more scarce at higher ranks, and deck popularity data is plentiful in both cases, I went down first and right second in my calculations. Thus the hypothetical meta in my calculations is Diamond matchup winrate table × Legend deck popularity.

No it isn’t and no there is not.

I’m not making any of this up, you simply refuse to accept that you’re wrong. (Edit: to clarify of course deck popularity has an effect on winrates, indeed that effect is isolated and measured using the method described above, but it’s simply not the dominant factor.)

1 Like
14

Tried to read it, to long, for those who managed to read it nice.

What i read makes no sense

4 Likes
15

“Aint nobody got time for that.”

2 Likes
16

Then just skip to part 3 for results I guess. But I do encourage you to find time.

2 Likes
17

Its an old meme I was referencing.

2 Likes
18

I kinda like to follow close what really determine a decks playrate so this is atleast interesting to read.

So if i understood this correctly you’re saying that rainbow mage has some equilibrium between being easy enough to be played by the average player and atleast a decent oportunity to find ways to outplay the opponent.

It’s actual playrate makes a ton of sense now.

2 Likes
19

Here’s the problem. You aren’t reaching the audience you’re likely trying to reach.

Why? Because the audience you want to reach are generally ones that aren’t very good with statistics and you use a lot of statistics style language.

Thus, you are getting a bunch of people responding not understanding what you’re saying.

Even in your “conclusion” area, you need to dumb it down. And by dumbing it down, I mean you need to come outright and say something like “X is the most skilled deck and Y is the most unskilled deck” or “if you are an average player, stop playing deck X if you expect to win”.

I’m not trying to say the forum is too dumb to understand, but instead your usage of language isn’t in layman’s terms even though you made an attempt to do so.

Or, to put it another way, you presented this like someone would present it to a class attending a college course (TED talk). You need to present it like you’re presenting it to 9th graders or a blue collared factory worker or something similar.

4 Likes
20

I actually took the time to unban this person for a second and read it.

The problem is exactly what I said it was: it is gibberish.

It does not work how they think it works.

They are not measuring skill, they’re just improperly using big words and calling themselves an expert.

I put them right back on ignore and now I’m 100% I don’t need to look again.

You can’t measure skill in this way. It’s not even a good proxy for skill because there’s too much shared variance because there is overlap in your samples.

If we start from the fact that we self-select into the pool for VS’s data set and then put forward that all of the games recorded for the brackets are the same small number of players, we figure out that we are looking at just a handful of people’s games instead of some objective skill rating.

If you want to just gloss over all the assumptions we violate when we manipulate data like the original post and just toss out garbage, fine, but that isn’t a statistical analysis it’s GIGO hiding behind big words.

4 Likes