Note: Sorry I can’t properly link or embed images. If there’s a work-around (?) I’m happy to edit them in. For now you can copy-paste this Imgur thread url and scroll to the appropriate “Fig” number reference in there: https://imgur.com/a/4DWoM2m
Introduction:
Last October, I posted to the OW University subreddit, lamenting that matches in silver, on my main account, were substantially harder to win than those in high gold, on my alt. (I suspect my skill ceiling is roughly low plat.) I lead the post with this S24 placements comparison graphic - 27% vs 60% win rate (main vs alt) averaged across all 3 roles (solo queue): https://i.imgur.com/imh2KGr.jpg
I speculated about the source of the apparent systematic bias perhaps being a disparity between comp SR and MMR. After discussions and further research, I decided that was most certainly nonsense, or at least the wrong framing - SR is MMR in comp. Particularly helpful was Kaawumba’s fantastic thread(s), referencing everything that’s officially known about the matchmaker. See the latest one for S28+.
In my mind, this left 2 plausible sources of systematic bias: player stats from within comp (which I can’t do much about), or QP stats/MMR, which I could potentially ‘adjust’. So I set out to test the latter, while seeing if I could also quantify the bias (or better still, just climb the cursed account!). Recording 30+ pieces of information per match, chipping away at it when my ME/CFS would allow.
Two sets of placements, on each account, clearly wasn’t enough matches for meaningful comparison with that, either. So I placed a few more times on my alt, as an additional control. Also to prove it wasn’t about to plummet to silver, too.
This Imgur post thread contains all the insightful graphs, plus relevant infographics/screenshots, with full explanation and some discussion appended to each image: https://imgur.com/a/4DWoM2m
⠀
Core findings (summary):
(1) Consistent evidence of matchmaker bias against tank role on my main. Initially stuck for 31 matches at 1900SR, 400 SR below my alt [Fig.1 https://imgur.com/w0qv5HU
], with 45% win rate [Fig.6 https://imgur.com/UJEHfuR
]. It stayed exactly level on SR due to the performance adjustments giving significantly more for wins vs losses [Fig.8 https://imgur.com/BcyX8Lz
]. Consistently strong performance was also seen with even higher average medal stats in losing matches [Fig.7 https://imgur.com/av6cI7B
] (i.e. my team mates were far more inconsistent). My stats did not fall away at all when tank SR did finally increase a little [Fig.18 https://imgur.com/bTwadki
]. I also estimated my tank QP MMR at 2500-2600 SR equivalent (see point 10).
(2) Matchmaking changed drastically for my tank role at the start of season 27, permitting me a totally unexpected and unprecedented 10 match win-streak! [Fig.2 https://imgur.com/eeG36L9
, Fig.1 https://imgur.com/w0qv5HU
] I gained 33 SR for the last 2 wins (28 for a couple before that) - a clear win-streak bonus. These matches felt like they’d been balanced for me, instead of counterbalancing my above average contributions (the rest of the time). This period ended, returning to a 45% win rate, stuck now at just over 2100 SR (200 SR below alt).
(3) Possibly circumstantial evidence: Throwing 10 QP support games (not advisable) appeared to unstick my competitive support matchmaking, allowing it to climb back into gold with ~59% win rate, pre season 27 [Fig.6 https://imgur.com/UJEHfuR
]. Tank remained stuck, actually losing more matches initially [Fig.1 https://imgur.com/w0qv5HU
]. Support match balance felt more favourable [Fig.17 https://imgur.com/cjsSxay
], then gradually slid down to feeling negatively biased, as it got stuck again at 2100SR, 300 SR below my alt.
(4) My alt held its rank over an additional 3 seasons of placements [Fig.1https://imgur.com/w0qv5HU
]. (101 comp matches now played on it, in total, across the 3 roles.) DPS climbed a little to borderline plat (despite being my weakest, least played role). Support fell a little (but I’d been playing lots of QP support on it with my masters skill friend). Tank feels under-ranked at 2300 SR and my attempts to carry also show up in average medal stats almost as high as my main [Fig.7 https://imgur.com/av6cI7B
].
(5) Incidence of leavers (3), throwers (11) and enemy ‘smurfs’ (19) was relatively low (out of 168 matches on my main). But about 50% of lost matches had one of these issues and a far lower percentage of losses on my alt were associated with these issues [Fig.12 https://imgur.com/nF31Ro6
].
(6) PotG was, predictably, awarded to a player on the winning team about 2/3rds of the time [Fig.14 https://imgur.com/w9k0xmy
]. But, beyond this, there was a noticeable bias towards enemy team PotGs, particularly in lost tank and DPS matches.
(7) On average, it appeared I was being pulled down into matches with slightly lower SR players [Fig.9 https://imgur.com/8qjpa3x
]. Ally and enemy teams ~10SR below. Which could help explain slightly lacklustre average SR gains, on my main (closer to 23 SR, than the expected average of 24). Like masters tanks get pulled up into GM/top 500 matches, one would expect the player distribution to pull the less played roles downwards the low end, too.
⠀
Incidental findings:
(8) More team mates in voice had only a small association with positive match outcome, despite comms being a personal strength [Fig.13 https://imgur.com/3jrU6wM
]. There were noticeably more team mates in voice in high gold vs low gold/silver.
(9) Playing after 11pm (GMT) was a bad idea, with a far more negative win-rate [Fig.15https://imgur.com/b9Ur9WZ
]. I’ll also be avoiding Sundays in future. (Note: I have a 26 hour rotating circadian rhythm, so those late night matches were still prime time for me.)
(10) Flex and ticket queues saw the same or marginally beneficial match balance/outcome [Fig.16 https://imgur.com/mlmLHjc
]. Which, I think (?) contradicts some popular sentiment. However, checking on this was an afterthought and there was too little data for a firm conclusion.
(11) Fairly consistent Quick Play MMR values can be estimated by averaging the SR, across many games, of the other players with visible profiles, on the role they played. At the end of the study, my tank was around 2500-2600 SR equivalent [Fig.20 https://imgur.com/GW5q336
]. DPS ~2300 SR (200 SR below the comp rank of my alt, after its 5 seasons of placements).
(12) My support QP MMR took a very long time to fully recover and stabilise, after hard-throwing 18 games [Fig.19 https://imgur.com/0lEOJXV
]. Please don’t try it yourselves. It seems that you only get to stomp low skilled enemy players for 1 game; I think the matchmaker immediately assumes there’s a different user on the account’s role. It gave me a lot of unwinnable and imbalanced/rubbish games for a while. Then, after about 24 try-hard games, a load of too easy games, where I did stomp, winning 17 out of 20 without MMR rising. Eventually my MMR plateaued again, after about 70 games, at 2300-2400 SR equivalent (in line with the comp support SR of my alt).
(13) Players in higher skilled QP games are more likely to have open profiles [Fig.20 https://imgur.com/GW5q336
]. The trend is based on extremely noisy data, but roughly speaking, for each 100 SR equivalent, that skill increase, there is one more profile visible.
⠀
General interest:
(14) Rein got the most PotGs on my main, by far! [Fig.3 https://imgur.com/py6hOP3
] Also on my alt, but more closely followed by Pharah [https://imgur.com/a/2TEoJOA
].
(15) Hanamura came up twice as often as most other maps (gods dammit!). [Fig.4 https://imgur.com/P0xRQGk
] Distribution looks random, of course, but are map probabilities weighted to have an even amount of each game mode, on average?
⠀
Discussion:
• Personal context: My top goal has been to climb my main, with all its unlocks and levels, to the modest SR of my alt. That’s close to where I think my absolute skill ceiling is, due to my slow (visual) information processing with ADHD-PI and dyslexia, plus ME/CFS getting in the way and slowing everything right down, a lot of the time.
I dearly love this game, but the competitive matchmaker has perplexed me for years. It seemed like the better my QP games were, with higher skilled friends (high gold, plat, diamond and masters), the less winnable my comp matches became, down in silver. I guessed there might reasonably be a causal link there (as I explained above).
⠀
⠀
• Matchmaker function: I think the most important point to highlight is the developer’s focus on making matches with 50/50 predicted outcomes.
They know players hate one-sided matches (and will stop playing if things feel too hopeless, too often). This is the main reason, that I’ve seen them offer, for not implementing an SR reset - months of horrendously random, messy matches (that wouldn’t achieve much, long term). I think this is totally fair.
But, I think it’s not possible to balance matches closely enough, in mid-ranks, while only considering the factors that they’ve made public. There’s hard stuck gold border accounts vs actual brand new players, smurfs, throwers, etc, all piled together in there. And they’ve admitted there are other (unspecified) data used in certain/niche circumstances, at least.
⠀
⠀
• Speculation: My guess was that performance stats from within competitive and/or from QP (or straight up QP MMR) are used to more finely balance teams in mid ranks.
Obviously, this would cause a kind of ‘stiction’, holding back climbing, because it would pit under-ranked players against equally under-ranked players (judging by their higher stats). Counter-productive to rank mobility. But it would still allow those who improved further to then climb.
Presumably there would be a cap on how much it will counter-balance a player. So, very under-ranked players (which I’m not, mostly) could always climb, too. Although, generally, smurf accounts, and high skilled new players, will climb up quickly, within a relatively small number of early matches on an account. While the account is still getting special treatment…
I think the devs have said that they keep new players away from most of the player base. I think they also make a point of balancing their early comp matches for them, too. To give them as much influence over the outcomes as possible, such that their rank moves in the right direction, initially. This would stem naturally from having fewer performance stats accumulated, to use to counter-balance them.
I think that’s a key question: who is the matchmaker balancing for, and who is being counter-balanced? Long time, regular players, it would take as part of the furniture. Except maybe under certain circumstances. Like, I feel that my high stats on tank might have passed a threshold, triggering the matchmaker to balance for me on that role (for a time), smoothing my climb.
My win streak started right at the start of a season, so I assumed it was a global change. But I’ve not heard of anyone else having had sudden success like that. So either it’s targeted, or randomly allotted…?
It definitely felt completely different. And I was expecting the usual 45% dross, so definitely not a case of expectation bias. I was actually disappointed it was winning while my support wasn’t (given that was dispelling my hopes of having figured out a QP link).
I previously thought the system might only counter-balance players it guesses are smurfing, deliberately avoiding ranking up, whatever. That it misidentified me and I was then seeing more enemy smurfs than other players, holding me back. Or, that it would set me, as an above average player for my rank, with a few other above average players, against super-smurfs. Which were more common than the devs expected, in my rank.
50% of losses on my main did involve a suspiciously high skilled enemy (or leaver/thrower). But I probably failed to identify more ‘smurfs’ on my own team, without the benefit of kill-cam, etc. And I saw relatively few obvious (DPS) smurfs, during this analysis, compared to previous experience. It seemed like mostly an ineffable issue, of overall balance of player skill, initiative, cooperation, etc.
⠀
⠀
• The analysis: When I started this little study, I promised myself that I’d publish it, regardless of outcome. I thought, most likely, it would be a “nothing to see here” debunking of my previous Reddit post.
But, while it’s very far from a proof (which would be impossible, looking at only one account), my results seem wholly compatible with QP stats influencing comp matchmaking. At least in my mid-low-ranks, where we have performance based adjustments (I doubt it’d be used at the top). Even the small drop in my alt’s support role could arguably have been expected, given that I was using my alt to play QP support with my much higher skilled friends.
However, after being 90% confident, at the time I took a break from playing in season 25, I then lost my confidence. The strength of the apparent effect waned, in season 26. It could be that the matchmaker looked only at (or weighted for) stats in recent games/matches. If QP stats are used, then that wouldn’t matter for players who only play comp.
I mostly play casually, given that my ME/CFS makes me too fatigued to play seriously most of the time. Hence this analysis taking 7 months. Hence why I play at least 4-6 warm up games, first, to be sure I’m up to speed. (Which might lower my QP performance, relatively.) And hence why I played a match of each role, together, each time, to control for any personal variation in ability, due to variable ill health.
I also do not recommend anyone else try throwing in QP. For a start, it’s an aweful thing to do to the other players and I felt bad doing it, despite approaching it to make it look as subtle as possible. It was a last ditch attempt to find something to alleviate years of frustration, and then report back for others. If you suspect this is an actual thing, a better plan is to only play comp on the role/account you are climbing (then use another socially).
Even if this study convinces no one of anything (let alone prompting Blizzard devs to tweak things), at least it still helped motivate me to just play more matches. It also took some of the dismay away from negative outcomes; at least I was getting useful data, every time. This has, at least, helped me climb my main roles 200 SR and DPS 400 SR (faster than expected, although it’s still 600 short of my alt). Also, I’ve learned the idiosyncracies of Google Docs spreadsheet (woot, lol!).
So, I do recommend match tracking, and analysing replays, to get a more objective perspective on what’s happening and how you are playing.
⠀
⠀
• Debunking: I would have liked to have been able to dismiss those conspiratorial sounding claims that the matchmaker ‘pushes’ people to the rank it thinks they should be; I think that idea came from a misconception about a win/loss adjustment factor that the devs had only in season 3, to correct a maths mistake with the SR distribution curve.
But, I admit, it does look kind of weirdly like my SRs have homed in on specific set-points, and stayed virtually glued there. My feeling is this is probably still just explainable with thresholds of when biasing effects fully kick in. Like glass ceilings.
⠀
• Finally: My Google Docs analysis spreadsheet is shared for potential scrutiny (and can be copied for personal reuse): https://docs.google.com/spreadsheets/d/1QOnzbXvYBWXqbj5X7pgNbp3ycNaUo3CYcFIJs0lRwQc/edit?usp=sharing
I’m happy to answer any quick (or long) questions based on people having read only this post, given that there’s a lot of text to wade through in the main Imgur thread-post: https://imgur.com/a/4DWoM2m