SR, MMR, and Role Queue

This is my take on a particular quote from Mercer in the Role Queue Update thread. It’s indicative of what I think about the current state of the matchmaking system.

This is a common misconception. My goal is primary to give information about how the system works. If I have no problem with criticism if it is criticizing the system as it is actually implemented. I criticize the system as well. I get frustrated when people hallucinate how the system works and criticize that hallucination.

MMR is not an integer, and will move a small amount (approximately 0.03 for established players) with each game.

Regarding streaks: Win probability changes slowly with rank because there are so many random factors in each individual match. Unfortunately, it follows from this that frequent and long streaks will occur, and a player’s rank will oscillate widely. Essentially, a player will tend to bounce between the range of where he is nearly guaranteed to win and where he is nearly guaranteed to lose. Historically, the range varied from player to player, but +/- 250 SR was common and +/- 500 was possible. This problem can be analyzed in depth, mathematically (Overwatch Forums). With role queue, and several months of data, streaks should be reduced in frequency and duration, and total SR swing should be less. How much less will have to be seen.

It behaves exactly like expected from TrueSkill: Why Match Quality is Frequently Poor.

SR is not used in matchmaking. Only MMR is. There is no “you’re doing well, time to carry a sack of potatoes” in Overwatch. I don’t comment on HoTS because I’ve barely played it.

MMR is measured in standard deviations, got it. Still not really useful information, because we don’t know if it’s standard deviations as a modifier of your SR or range, or standard deviations as a modifier to the average player overall.

Measuring it in regards to the average player makes sense in the context of fair matchmaking, by using it to push people toward their SR and setting the goalposts of 0 and 5000 at the highest attained values(or slightly above), you get a system that properly models the bell curve, which SR appears to do. You also get the effect we see where it becomes increasingly hard to gain SR approaching 5000, and to lose it approaching 0, because maintaining performance above or below 3 stdevs from the norm is extremely difficult.

However, in the context of the fixed system suggested, it still makes sense. If you have a measure of a player’s stdev from their rank, it’s easy to pair them with opposite players, match their performance with similar players on the other team, etc, to ensure a more fair and balanced game.

While the devs have said matchmaking is done based on MMR, not SR, that is a single line and not some code we’ve actually been able to examine. It would be perfectly reasonable for them to drop you into a pool(likely containing dozens or hundreds of queued players) based on the SR range the game will be, then match players out of the pool based on MMR. You would still be matching based on MMR, it’s just within the SR range.

‘Performance’ is not a metric easily measured in standard devations. A DPS playing with amazing tanks is going to be doing much more damage, getting many more eliminations, and suffering far fewer deaths than the same exact DPS playing with a potato roadhog and dps sigma. In this way, it’s quite possible that hitting a few one-sided wins would result in your stats appearing to have vastly improved compared to the average player, who is getting one-sided wins, one-sided losses, and a few fair games. Your stats in a one-sided win can easily be 5x or more your stats in a one-sided loss.

If MMR is easy enough to shift in that context, then it would make perfect sense that winning a few stomps results in you being paired with poor players. The resulting stomps will confuse your MMR, and perhaps you end up back in a winning streak.

Again, while I don’t buy into anecdotal evidence, there is a ton of it to be had. Smurfs largely agree that they will always end up paired with enemy smurfs if overperforming. Hundreds if not thousands of players feel they’re being forced into win and loss streaks.

Games in a coordinated stack feel utterly insane compared to normal games at the same SR. While stack matchmaking is weighted to be matched against other stacks, it’s clearly also got a skill factor. When you could still see stacks, my 4 stack would always be put against enemy 4-6 stacks. That’s fair, but they were always coordinated as well, resulting in long and fair games. I would expect that if there were no bias in the internal balancing, we would see the stacks that result from idiots using ‘stay as team’ as well, and it didn’t appear so.

The average ability of an opponent when queueing as a 2050~ sr 4 stack is equal or higher than the average ability of an opponent when queueing at 2700~ sr solo. This is something I do have enough games to be confident is past the observational bias mark. Whether it’s because stacks that don’t work well together break, or because there’s an internal bias, I can’t say.

In the context of what blizzard employees have posted, you can make a convincing case for either way. That said, my disclaimer on this topic still applies. While I personally am inclined to believe this system does exist, there is no proof one way or the other.

Further, if you’re using the potential existance of this system to claim you can’t climb, you are just plain wrong. If you are good enough for the matchmaker to try to balance downwards around you or put enemy smurfs against you, you are raking in performance based SR like crazy. If you can’t win half your games, or your performance based SR gains don’t outweigh your losses, any potential argument about being sandbagged falls apart because you’re obviously not doing great in the eyes of the system.

tldr; Maybe the games are handicapped, but if you can’t climb that certainly isn’t why.

1 Like

I don’t really know what you’re trying to say here, but MMR is not “a modifier of your SR range”, nor is it a “modifier to the average player”. It is a value that is used to place people in matches.

MMR is not used to push people anywhere.

If you can’t trust what the developers say about the system, there is not much to discuss. All you can say from looking at match data alone is “Hey, it sorta looks like Microsoft’s TrueSkill, which is well documented”.

Poor word choice, perhaps. But, if MMR is measured in standard deviations, they’re obviously using standard deviations in some way. Thus, MMR should either be ‘standard deviations from your current rank’ or ‘standard deviations from the average player’. It makes absolutely no sense to model a system in units of standard deviation if it is not actually a standard deviation of some sort.

This is objectively false. If your MMR is significantly below your SR, climbing will be very difficult until such time as you can shift it upwards. If your MMR is significantly above your SR, climbing will be very easy until it has time to drop. Your SR gains are clearly impacted by where the matchmaker thinks you should be, particularly at the start of a new season.

The developers intentionally keep everything to short and often cryptic notes. I have not called them a liar, simply said that their statement is up for interpretation.

If they limit the pool of potential players based on SR, then match based on MMR, they could say that they match based on MMR not SR without being in an outright lie.

1 Like

I have been pushing the line that SR and MMR are separate things in response to a number of threads, but I will say that the language Scott used in the the role queue update was confusing, specifically:

I am hoping that he really mean’t to say MMR there and a few other places in the update.


BTW Kaawumba, did you see this talk, its very enlightening:

Yeah, Scott pretending MMR doesn’t exist is very frustrating. I think he’s trying to keep it simple for the plebs, but it isn’t really working. At this point, I consider Seagull & Jeff Kaplan: NEW Hero Confirmed! - YouTube to be definitive that SR has no effect on matchmaking.

I have seen the talk. It is good. http://www.moserware.com/2010/03/computing-your-skill.html is also good.

SR might not have effect on matching (technically it shouldn’t), but it’s also indicative to a point.
A game with 5 people in gold, and 1 DPS in Silver. Can you guess who failed to deliver kills and refused to switch?
No one can convince me, that in the particular example, said DPS had similar MMR as that of the other participants.

IMO this could be the issue. That the system takes one number for performance for said rating and assumes you can fulfill that number under any circumstances.

As a healer, sometimes in a winning game you can pull off insane numbers. Other times you don’t have that opportunity (because the enemy team might tilt, start arguing, run in one by one and don’t do anything). You can’t heal, when your allies don’t lose any health.

Same thing with losses. Sometimes you can do insane numbers, top both healing and damage, despite losing. Other times, you get stomped so hard, you don’t even have the time to perform.

In all of these scenarios, the player is not doing anything differently than what he usually does, yet his performance will be vastly different in terms of numbers, that are devoid of context.

If your MMR is a standard diviation, with the standard itself being a flat number, the game can assume you overperformed or underperformed despite you playing the way you usually do, and as a result, you end up on streaks.

The best answer I can give is to watch the video - but its long (about 1 hour).

Therefore I will cut to the chase.
In your particular example the most likely scenario is that the DPS in question was grouped with a higher MMR player, hence the DPS’s SR and MMR were both lower than yours.

1 Like

It’s this one.

What I meant is that MMR is not used to rig matches. The MMR/SR differential is used to affect SR gains and losses, yes.

What is this, 2016? They got rid of that ages ago.

Certainly things could be better documented. But Blizzard’s statements are usually clear and fit with the matchmaking theory I’ve seen from independent (non-Blizzard) sources. Most of the “interpretations” I’ve seen are people trying to find some excuse for why they have poor teammates other than the obvious: that they themselves are a poor teammate, and they are matched fairly, on average.

1 Like

Everyone intelligent is aware you believe it to be that one. You obviously are one of the smartest posters here, can you objectively provide any sort of proof beyond saying it is so?

Seems a bit contradictory, but ok.

That is true of many posts, and it’s easy to see from the tone. However, their ulterior motive and general lack of awareness does not indicate that it is or is not the case. Suggesting so is a pretty severe fault in your logic. As I said in my first post, were this truly the case, anyone who belongs higher than they are would still be climbing due to PBSR. It still matters because it severely impacts match quality if that is how matchmaking is done.

1 Like

Here’s the actual quote: Seagull & Jeff Kaplan: NEW Hero Confirmed! - YouTube. You can interpret however you want. However, with sufficient math background, only one explanation is reasonable.

I was referring to your “particularly at the start of a new season”.

I have demonstrated that the problematic behavior can be understood by simulating a conventional, well-designed system. https://www.reddit.com/r/OverwatchUniversity/comments/aatezy/why_match_quality_is_frequently_poor/. By Occam’s Razor, I reject more obscure explanations.

1 Like

I’m not sure the ‘sufficient math background’ has any relevance here besides an attempt to invalidate my opinions via a self-proclaimed superiority. The question is not a mathematical one, but one of linguistics and trust.

Fair enough.

Occam’s Razor is a way of thinking, not empirical evidence. It might mean that it’s likely to be correct, but we both know it does nothing to actually back up that assertion.

My counterpoint: How do you measure standard deviations from the norm in the context of matches against varying opponents? We know MMR is measured in standard deviations. We know MMR is weighted based on wins, losses, and performance. By trying to measure these things against all players when different ranks will have drastically different performance in a non-linear fashion due to playstyles, you introduce mountains of factors that could destroy your data.

For example, more damage is almost always better. However, doing 2000 damage/min in a bush league game is much more likely than doing the same number in a grand master game where you attempt to limit ultimate feed and targets out of position are quickly killed. Likewise, achieving high healing numbers is much more difficult when your tanks aren’t constantly taking non-lethal chip damage and instead are being focus fired when they are killed. Given a player in gold, and a player in GM, many of their stats are liable to be comparable, but obviously the GM players’ mean more given they are against a GM opponent.

Wins, losses, and performance are all irrelevant until placed in the context of the opponents they were against. Measuring MMR as standard deviations within your range of skill(perhaps done by rank: bronze/silver/gold/etc) allows you to assess whatever variables you determine to be most relevant in the context of the level of gameplay you’re playing at. Thus, as a gamer and software developer myself(likely with less mathematical background than you), I would be inclined to measure in the context of a rank. This allows you to easily compare all performance between peers, whilst removing the bias due to the major game differences at varying levels of play.

If I were tasked with creating a system that achieved as close of games as possible, I would want to identify overperformers and underperformers and ensure that they were evenly distributed as much as possible within the provided constraints. Keep in mind that at it’s core, Blizzard is a company interested in making money, and similar tactics have been employed by many game companies in many ways.

All I’m getting from your recent posts is that you are unable to substantiate anything beyond that single line from Jeff. It’s a bit disappointing, since the response to all of these topics is ‘You’re crazy, go listen to Kaawumba!’.

So, allow me to ask you straight out. I acknowledge that any evidence in favor of weighted matchmaking is largely anecdotal and unsubstantiated. I acknowledge that Jeff has claimed they do not look at SR for matchmaking. Do you have anything else to offer me besides subjective viewpoints and ad-hominem? A shred of real evidence that the matchmaking is how you say it is? From what I can see, there is no objective proof that they DO or DO NOT ‘handicap’ matches, simply evidence to the possibility of either.

2 Likes

Why do I shoot up 500 SR after playing with the people I was supposed to be just as bad as? Why was I even in a match with them to begin with? Thoughts?

This is a broken procedure, and is not what they do. It would make ranking up harder than necessary, which would make match quality lower for everybody. What is done is described in How Competitive Skill Rating Works - Season 17 → Summary → Summarize matchmaking, rating, and progression for me. Note the references as well. That post is out of date because it doesn’t include role queue, but the basic ideas haven’t changed.

How Competitive Skill Rating Works - Season 17 has 65 references. Perhaps, though, you should start with http://www.moserware.com/2010/03/computing-your-skill.html. It’s about Microsoft’s TrueSkill rather than Blizzard’s matchmaker, but they aren’t really that different and that post does a good job of explaining the philosophy and math behind these kinds of systems.

You seem to believe that it is up to me to prove to you how things work, and unless I can do that, your beliefs are equally valid. That isn’t how knowledge works. You have to understand it for yourself. I can’t make you understand anything.

And we can’t both be right. Reality doesn’t care about validating your opinions.

You’re not really giving any context here, but the system is unstable because of the introduction of role queue. You need to play ~150 games for your new rating to believable. But if you maintain the higher rating after ~150 games then either you’ve gotten better as a player, or your skills are more suited to the new role-queue meta. There is nothing strange about either of these possibilities.

Its definetely not a single digit number, your MMR is not 1 for example, its more like 1.02357098000264 or something like that.

They use a number between -3 and 3 because of programmimg reasons, it is better to use small numbers with many digits after the decimal point (in programming this is known as a floating point) because it is just as precise as using really large numbers, while using much less memory space.

So your MMR would look more like 1.0000263588151425277, rather than just 1.

Probably they don’t have the same MMR, but remenber that you can’t see groups anymore so they probably were grouped with one of the gold players and that’s why they got into your game, not because the MMR system randomply put them there

The system in WoW worked damn fine for the most part at least during WotLK and Cata. At some point during one of those expansions, you were allowed to see your MMR. It didn’t ruin the game.

As to the case of Overwatch, the MMR can be translated. This number will correspond to approximate SR, that the system estimates you being capable of reaching. So, regardless of the actual number being 1.02357098000264 for programming reasons, they give you a visual representation of 4 digit number as to what SR the system believes you being capable of reaching at the time if you keep up what you’re doing. They could if they wanted to.

However, there are reasons they are not going to do that, and I believe it’s not, because “players would game the system”. Players are already gaming the system, wittingly or not.

The reasons they are not gonna show you are:

There will be those matches in which you’ve clearly outdone yourself, and yet your MMR will dip.
There will be those matches, which will clearly be what people refer to as “rigged” and it would be apparent, if you’re able to look at each of the 12 participants in a match.
Blizzard are not going to want you to see what happens to your MMR during streaks.

I never claimed to be right, and you keep jumping to personal insults and assuming I haven’t read your material. I asked you a simple question and you responded with more evasion. If you are to claim, as you have been, that it is a fair and unbiased matchmaking system… surely you would be able to support it with more than theories and insults.

The response to anyone suggesting an alternative system such as I’ve described has been to say they are crazy and to read your material. The truth is, your material does not adequately support or debunk anything with blizzard source material or hard evidence. It’s a very well written and thorough explanation, but much of it has been filled in with assumption. Your reddit post is a cute software exercise, but it is still just a demonstration of how you could get bad matches in a fair system. It is in no way proof of a fair system.

My issue is that while you believe the system to be fair, and you don’t seem to have encountered any educated opposition, you don’t have any actual evidence to support it besides one quote from Jeff. Your posts are long and intended to be intimidating, but when you sift through the distraction they lack actual evidence and instead fall back on trueskill’s behavior and a simulation that has some questionable mostly arbitrary parameters and a whole lot of assumption.

Sure, you can fairly call any idea to the contrary a conspiracy theory or unlikely. However, the attitude toward it has been far from that. People continually act like your summaries ‘prove’ something, when the truth is they’re just a whole lot of smoke and mirrors to conceal a lack of actual evidence.

2 Likes

You keep linking things, but have you actually read them?
Call me dumb if you like… maybe I am, or I am confused or something, but I already see an issue with this supposed explanation of how the system is supposed to play out when it comes to personal performance.

So let’s say I one-trick. Say I’m at 2000 current rating (which is SR, why current rating is called Skill Rating is beyond me).

Say the system already estimates me capable of reaching ~2250SR (that estimation is the MMR), because of how I performed in recent matches (and yes, this is possible, the way it was possible in WoW. I don’t care what actual numbers they use for programming purposes, they can correspond to a certain rating and can be translated and shown in a 4 digit number).

To reach 2250 SR, I need to keep winning. Win at least half my games if they take place on 2250MMR, win more than half my games if they take place lower. That’s all good and well.

But as to my performance, shouldn’t my performance be measured compared to people at my current rating (SR) and not my MMR? Because MMR is already an estimation of where I belong.
The way this is phrased, in order for me to get to where I’m supposed to be according to the system, I have to perform not better than where I am currently, but I have to perform even better than where I am estimated of capable of being.

If that’s the case, this would be truly ridiculous.