How Competitive Skill Rating Works (Season 14)

Competitive Discussion
19

In diamond yesterday (and I didn’t get the exact account), I started at 3005 won (got season high), then lost, and was at 3003 (I lost more than I had won in the previous game). Next game I won and got a new season high again (won more than I had lost in the previous game). If Diamond doesn’t have PBSR, then MMR would explain why I lost more and won more in a 3 game session. Thus, both wins and losses are affected by MMR.

I also agree, the less that is known about the player provides bigger swings. But big swings are relative to that perspective. In my case, the loss was bigger than the previous win. And the next win was bigger than both the loss and win for the amount of SR I gained.

20

That’s not PBSR. The game you lost, your team MMR was higher than the enemy team MMR, and the difference in that game was larger than the vs MMR difference in the first game that you won, thus you ended up at 3003 after going 1-1 for a net loss of 2 SR. And then on your very next game, your team MMR was lower than the enemy team’s MMR, and the MMR difference in this third game that you won was higher than the MMR difference in the previous game that you lost.

So we’re talking games where there’s a slight 51/49 or 48/52 chance because it’s actually not quite possible to make a perfect 50/50 match, even in populated ranks, if you don’t want to be stuck in queue for a long time. Summarizing this, you probably won a 50/50, lost a 51/49, won a 47/53 to get to your new high.

2 Likes
21

Got it, makes sense. Thanks!

1 Like
22

Yes.

It has happened in the past. http://overwatch.blizzplanet.com/blog/comments/blizzcon-2016-overwatch-qa-panel-transcript/3

I’m not convinced MMR significantly below SR happens anymore, but I expect the code for SR chasing MMR in a downward direction is still in the game.

There is the real question of exactly how really poor performance gets implemented. Does MMR move quickly and then SR chases it? Or does MMR and SR move at the same speed, and both get influenced by performance? Or does MMR generally move slowly, and SR will run ahead of it if performance is really bad?

At this point, I think the answer is MMR and SR move at the same speed (unless decayed or DCed) and both get influenced by performance, but I haven’t found a way to prove it.

23

I feel like that was something that went hand-in-hand with the Season 2 placements problem. Scott Mercer did mention “compressing” the ranks a little bit, but it seemed like he understated the “little bit” part because a lot of people who dropped their SR hard from the mid-ranks down to low-ranks (like, SR 50ish all the way down to the 20s) ended up placing pretty high in season 2, so they ended up dropping in SR again from Gold/high Silver to 3-digits in SR. So it played out like a half-assed hard MMR reset/targeted for accelerated MMR drop-or-increase system.

Unfortunately, this is kind of a sucky question to try to answer on an actual stage in front of people.

24

It’s definitely in the game. It happens if you go on loss streaks while performing like garbage. You can get to a point where you’re losing 80+ SR per game, but you have to be hardcore feeding and have lost like 10 games in a row.

1 Like
25

I have also noticed this once. In the competitive leaderboards (the “top 500 list”) you can check a button for it to only show players in your friends list (even if they’re not top 500). I once checked it, and it showed one of my friends at mid-masters rating (3500+) even though his profile showed 3000 because he had decayed. This should be easy enough to test, if it still happens.

Edit: I just went to check it again; it was season 12 and the leaderboard still shows his rating at 3725 even though his profile shows 3000 final SR (and 3749 for season high) for that season.

P.S. Kaawumba: thanks for this amazingly well made topic/analysis, btw! :slight_smile:

1 Like
26

What happens when you have a company that doesn’t reveal how their hidden systems work and you add a tonne of fans with a penchant for statistics and experimentation?

You get actual positive forum posts of people trying to work together and help each other solve a puzzle rather than… the alternative…

2 Likes
27

Did you happen to write down your SR each game while this was happening? Was it a new account? What season was it?

28

Ok, it looks like it is a real bug.

Assuming this bug doesn’t affect top 500 rewards (that is, you can’t get top 500 rewards if you have decayed out of it), it is a fairly minor bug.

29

I didn’t, and no it wasn’t a new account. I let a friend who had never played OW account play on one of my diamond accounts. He went on a massive losing streak (of course) and had normal losses until about 10 losses, at which point it went up until it hit about 90 SR lost one game. Then he won a game and gained the standard 25. I have no idea what the cap is on it.

And I can say with confidence that it does not work the other way; you can win 10 games in a row and still only gain 25.

30

Btw, before finding this thread, I always had the impression that the MMR might not be just a single number because it was supposedly “more complicated” than SR.

In ref. (58) Jeff says “MMR means matchmaking rating. It is a hidden number. It looks weird and mathematical”, so it seems he confirms that it is indeed a single number, but the follow up seems a little odd to me. How can a single number look “weird and mathematical”, at least compared to some other single number (the SR)? Is it just because it likely has a different normalization?

So I wondered if there were some other reasons or references according to which it is just a single number? (Which it most likely is, I’m just curious.)

31

Thank you for being willing to update your beliefs in light of new evidence. That skill is in short supply on the forums.

My interpretation is that Jeff is not a mathematician or a programmer, so floating point (C/FloatingPoint) looks weird and mathematical to him. To me floating point looks rather conventional. Yes, MMR presumably has a different mean, sigma, normalization, etc, from SR.

"Anyway, this leads me to matchmaking rating. This rating is the most important thing that we try to match on. … All the system does when it comes to matching on skill is attempt to match you with people of a similar number. " – Jeff Kaplan (Overwatch Forums)

These are the only two relevant quotes that I am familiar with. Note however, that pretty much all modern rating systems reduce skill to a single number.

P.S.

Some people say that MMR is actually two numbers: value +/- uncertainty. This is a semantic difference, but is not wrong in a literal sense. Other rating systems also have an uncertainty (K factor in Elo, sigma in TrueSkill, etc.).

“Play a lot of games, it (MMR) gets more certain. Don’t play Overwatch for a while, it gets less certain. The more certain the matchmaker is about your MMR, the less your MMR will change in either direction based on a win or loss” – Scott Mercer (Overwatch Forums)

It is certainly not some complicated object of many parts. Things like performance metrics influence how much MMR/SR move, but are not part of MMR/SR.

32

Yeah I was thinking something like that too. Although I’d like to think that the lead designer of a game this big is used to real numbers. :smiley: Maybe he was just trying to put it simply for the layman.

Anyways, thanks for the additional quotes!

1 Like
33

I’ve lost SR after a draw, is that normal?

I have a 56% win percentage and have dropped 100 SR since my placements.
I really wish the current ranking system was a lot clearer, and not so convoluted.

EDIT: When attempting to match two teams that are 50/50 does it attempt to match players with others with a lower skill to balance your team? Like for example (two - three experienced players and four - three lower skilled players?) vs (two skilled players, one moderate player and three lower skilled players?)

34

No. SR normally doesn’t change on a draw. There is an occasional bug, though, where people lose ~24 SR that they should not. See the bugs section in the original post.

Make sure to not include your placement games in your win percentage when looking at your SR change post placements. Also, if you include your SR change for each game I can say much more about whether of not something is broken with your account.

Experience, that is player level/time played, is not taken into account in competitive matchmaking.

Generally, in each match there will be players with highest/middle/lowest MMR in a given match. So technically, speaking, yes, “good” players are matched with “bad” players to balance the match. However, unless you are one of the best (GM+) or worst (low bronze) players in the game, you will be as likely to be the highest/best as the lowest/worst player in any given match.

Contrary to some conspiracy theories on the forums, there is no keeping gold skilled players in silver by consistently forcing them to carry bronze skilled players.

35

I can say I’ve taken a long break and was Mid plat to Low Diamond most of my ranked career. Since coming back I’ve been placed in Gold, and seem to lose an oddly large number of SR for my losses.

I do perform top of my class, and have often swayed games when assuming roles that are lacking.

I will take note of my SR changes as of today.

36

When you return from a long break, your MMR uncertainty goes way up for up to 25 games (including placements). During this time, your SR can move fairly violently, with early games mattering more than later games. After this period, your SR change should return to the usual +/- ~24 per game.

Thanks.

37

Anyone can confirm whether this is true or false?

After I climbed to diamond I feel like my SR is mostly affected by what my SR is compared to my team’s average. So, for example, if I’m at 3.1k and my team is 2.9k (same average SR for both teams) I will typically lose more and gain less than on a 3.2k game. I think I started noticing this last season while still in plat, where I would get 18 - 22 SR even if I hard carried a high gold / low plat game where both teams had virtually the same SR as well.

38

I’ve taken note of my SR changes as of last night.

Started at 1908
Win w/ Leavers on Enemy team +24
Second Win +23
Third win +27 (Won as 5v6)
Fourth Loss -22
Fifth Win +21
Sixth win +26 and into Gold / Season high

(Afterwards I stopped keeping track.)

One thing I’d like to mention is, these games were incredibly frustrating. There was no real composition to my team. No communication and players were wildly all over the map, not fun (Two of these games were won because the enemy team was C9, and I played from the CP finishing with G/ Objective Time)

Once I came over into Gold, I was presented with more well constructed teams but poor gamesense and performance. Games in this bracket are wildly unpredictable given people have some what of a fundamental understanding of the game, you’ll just have to hope they’re on your team otherwise you’re going to be punished.

While in silver, it’s possible to just brute force heals or go off in DPS and swing the game. But in the Gold bracket, teams are smarter and players have to perform at their critical roles. Suddenly climbing becomes more of a chore, and your team is either good or god awful.

Whilst in silver it just seemed like both teams were bad, and I could pretty much guarantee a win on my team. But in Gold it’s either your team is good and their team is bad, or vice versa. Rarer still is you finding your team being good, and their team being good (A fun/good game).

Swaying games in Gold would have to require Masters - Top 500 performance.

After last night I think MMR/SR is fundamentally sound, but I think needs to be improved on predicting a player’s experience. More importantly match making needs to be improved.