What Scott Mercer and Jeff Kaplan SAID about MMR

Doland-11979 · July 1, 2022, 5:29pm

The debate about algorithmic handicapping/balance in Overwatch has been seething on this and other Blizzard forums for over five years, with no direct answer from any Activision/Blizzard representative. What few replies we have from the Overwatch team on the topic were made in the decommissioned Battlenet forum, though never in direct reply to Cuthbert, the most vocal critic of the system:
Hey MICROSOFT, MMR/Forced 50% is Wrong for Online Games - Competitive Discussion - Overwatch Forums (blizzard.com)

And lately I notice that the conversation is being dominated by fact-free hot takes from players, who make no reference to primary source information. The overall lack of comment from or reference to verifiable information has made discussion unproductive and toxic. To ameliorate this problem, I have made the following collection of Overwatch developer posts, directly excerpted from “The Old Overwatch Forums Developer Post Archives by WyomingMyst.” I can’t link to the full archive, but anyone can find it with Google.

I welcome players from both sides of the debate to read the words that have been said on record about Match Making Rating, a mechanism of algorithmic handicapping/balancing, by Overwatch’s Principal Game Designer Scott Mercer and Game Director Jeff Kaplan. From the Battlenet forum compendium by WyomingMyst, in no particular order and with some repetition:

–

Skill Rating and Hidden MMR

Jeff Kaplan

Game Director

Jan 4, 2018

01/04/2018 02:31 PM - Posted by steelman

I’m a player who has only played less than 10 games in Diamond in all seasons. Whenever I play 4v4 Deathmatch, the enemy team always has Grand Masters. Is there any way to prevent this from happening?

MMR works very similarly to SR. There are some minor differences that make it feel worse though, when you just watch that number. For example, it’s possible to win a match and not gain any MMR. We make it so that if you win a match, you always gain SR – even if it’s just a little bit – to feel psychologically rewarding. But MMR’s entire goal is creating fair matches – which isn’t always fun to look at and certainly not “rewarding” for players looking for pats on the back or a sense of progression. So SR “chases” your MMR very closely, except in a rare case of severe SR decay at GM/Masters/Diamond level of play.

–

Jeff Kaplan

Game Director

Aug 23, 2017

Posted by Skrimiche

Wait… So if MMR is what decides who gets paired with who, then what is SR for? Just comp points and showing off?

That seems confusing…

SR closely chases your MMR up and down and is a more “digestible” number. With the exception of top players who have decayed, MMR and SR are closely linked…

–

Jeff Kaplan

Game Director

Aug 23, 2017

Posted by AnActualFrog

I’m guessing MMR is built off more than the player rank?

MMR works very similarly to SR. There are some minor differences that make it feel worse though, when you just watch that number. For example, it’s possible to win a match and not gain any MMR. We make it so that if you win a match, you always gain SR – even if it’s just a little bit – to feel psychologically rewarding. But MMR’s entire goal is creating fair matches – which isn’t always fun to look at and certainly not “rewarding” for players looking for pats on the back or a sense of progression. So SR “chases” your MMR very closely, except in a rare case of severe SR decay at GM/Masters/Diamond level of play.

–

Jeff Kaplan

Game Director

Aug 23, 2017

Skill Rating decays but your internal Matchmaking rating (the thing that determines who is matched against who – not SR) does not decay.

I’m not saying that you weren’t placed in an unfair match. I am just trying to clarify that if that did indeed happen, it wasn’t because of SR decay.

We have some changes to Season 6 that should help with some occurrences of mis-matches, especially in the top skill tiers. I can’t tell what tier you’re actually playing at because you posted on an alt account.

We posted a video yesterday that explains some changes that are coming that should help.

–

Jeff Kaplan

Game Director

Aug 23, 2017

Posted by Skrimiche

Wait… So if MMR is what decides who gets paired with who, then what is SR for? Just comp points and showing off?

That seems confusing…

SR closely chases your MMR up and down and is a more “digestible” number. With the exception of top players who have decayed, MMR and SR are closely linked…

Jeff Kaplan

Game Director

Aug 23, 2017

Posted by AnActualFrog

I’m guessing MMR is built off more than the player rank?

MMR works very similarly to SR. There are some minor differences that make it feel worse though, when you just watch that number. For example, it’s possible to win a match and not gain any MMR. We make it so that if you win a match, you always gain SR – even if it’s just a little bit – to feel psychologically rewarding. But MMR’s entire goal is creating fair matches – which isn’t always fun to look at and certainly not “rewarding” for players looking for pats on the back or a sense of progression. So SR “chases” your MMR very closely, except in a rare case of severe SR decay at GM/Masters/Diamond level of play.

–

Scott Mercer

Principal Designer

Mar 3, 2017

We’re pretty happy with how placements work right now, but we’re keeping an eye on them all the same. As we mentioned last season, we do lower your SR a bit initially but you also gain that SR back as you play. We’re trying to provide players with a positive initial experience of gaining SR at the beginning of a season without the matchmaking chaos of a more complete reset. It’s not perfect, but we like the results so far.

To clarify one thing, your initial MMR that we use to match you in placements isn’t simply based on the final result of the previous season. How you played in seasons before the prior one also factor into the calculations. Of course, how well you do in your placement matches also matters.

–

Scott Mercer

Principal Designer

Mar 3, 2017

Posted by Ungod

An example of what I was hoping for would maybe be, does the placement system put you against teams that you do have a significant chance of losing against, or even with, and then judge your performance?

We’re always trying to put you in fair matches that we think you have a 50% chance to win, and do it quickly so you’re playing Overwatch instead of staring at a queue timer. We never intentionally seek to put you in an unfair one.

I’ve seen comments like “I just won three games in a row, so the matchmaker put me in a bad stomp to get back to a 50% win rate”. It doesn’t do anything like that at all. It just keeps on trying to find fair matches. If you do win more than you lose, your SR will slowly go up. As that happens it’ll also put you up stronger opponents that match your new SR. That’s not so you lose, it’s to keep your matches fair.

There’s one other thing to remember, though. The results of a match that the matchmaker thought was perfectly fair don’t always result in a match that is hard fought every meter of the payload and you win by just holding them off from reaching the final checkpoint at 0:00 time remaining. Matches with teams of equal skill result in 3-0 stomps more than you think. Maybe an early fight snowballed out of control due to staggered spawns, maybe a player decided he was going to try to pickup a new hero that match, or perhaps your’s cat decided to play with your keyboard right before you used that Graviton Surge and it wildly missed its mark. We’re all human, and we don’t perform at the perfectly same level all the time. It’s one of the reasons competitive games are so much fun to play and watch.

–

Scott Mercer

Principal Designer

Mar 3, 2017

Posted by Verethragna

When exactly does the bonus SR end? It was at around 15 matches for me,(or 400 SR gain), so does it count matches, losses, or SR gain?

Thanks for the post btw.

It’s a bit fuzzy, but for most people it’s around 40-50 total matches played.

Scott Mercer

Principal Designer

Mar 3, 2017

Posted by CoffeeGirl

This is fantastic news, however the issue of trolls, leavers, and smurfs/derankers/throwers…has not been addressed. Would this be looked at in the future?

Oh, handling leaving is still high on our list of things to improve. It’s just a very, very difficult problem to solve in a way that the solutions don’t cause more problems than they solve, or that couldn’t be potentially exploited for SR gain. We have to be way more “mean” that we’d normally like to be to prevent exploits and discourage even worse behavior.

As Jeff mentioned back all the way back at Blizzcon, we’re still working to improve our player reporting to better identify and handle trolls and intentional throwers. Players intentionally deranking to misrepresent their skill (which does create unfair matches) is also definitely on our radar.

While I know a perfect Overwatch utopia of positive, communicative, teammates who’ll gladly play the tank or support when needed is a not completely realistic final goal, there’s still definitely room for a lot of improvement.

–

Scott Mercer

Principal Designer

Mar 3, 2017

Thanks for all the feedback about Season 4 of Competitive play and the changes we made. I wanted to first talk about how the distribution of skill rating looks across our competitive population. There’s always been a lot of speculation about this due to the lack of complete “official” data, and there are actually some very distinct differences between the perceived population based on third party sites and our complete data.

The data below is based on season 3 data (season 4 is too new), and the maximum (not current) skill rating of the player during the season. The values below are also rounded, so that’s why it doesn’t quite add up to 100%.

Bronze - 6%

Silver - 22%

Gold - 34%

Platinum - 23%

Diamond - 10%

Master - 3%

Grandmaster - <1%

Some interesting notes:

Median maximum skill rating is not 2500, it’s actually a little above 2300. So if you have a maximum skill rating of 2350, you have a higher SR than 50% of the population!
If you do a breakdown based upon the more volatile current skill rating, there’s even fewer players above 3000 SR than listed above. Only around 8% of the population was above 3000 SR for season 3 at any one time.

With this information in mind, let’s talk about skill decay. As mentioned in the competitive blog, only having to play 1 game per week didn’t make your current skill rating very accurate or meaningful. Current Skill Rating should reflect your skill rating as an active player. We also show your Season High Skill Rating in your profile, and that won’t change during a season due to inactivity decay. While you have to play 7 games in a week to avoid any decay, we wanted players to have flexibility within the week. Play in one long session on a weekend, or several smaller sessions during the week, as long as you play seven games it’s fine.

We felt like 3000 Skill Rating was a good breakpoint for the decay rules, as it was around 10% of the competitive population (see above). If you do decay, it only affects your current displayed skill rating. This decay does not affect the internal matchmaking rating we use, so we can still place you in fair matches. When you do come back and actively play matches, you’ll also typically gain more SR from a win until your displayed skill rating and internal matchmaking rating have again reached “equilibrium”.

If you’re curious about when you need to play a game to avoid decay, you can see that by looking on the on the right hand side of the information screen of the competitive play card.

Behind the scenes, we’re going to be making another change to our matchmaking that deserves its own discussion. The simple and primary goal of our matchmaker is creating fair matches. To do that, it evaluates potential matches by synthesizing an expected win %. The matchmaker is normally really good about creating matches with a win % that is close to 50%, but if the participants in the match are either at very high or very low SRs then it gets quite a bit more difficult. There simply aren’t many players at the extremes of SR to find. This is especially true in our lower population regions AND they’re playing at off peak times such as 04:00 in the morning AND the players queue in a large group. Normally in these situations, we wait to try and find a good match, but eventually compromises have to be made so the players in question don’t wait forever in queue.

Well, we’re not going to compromise as much anymore. The matchmaker will no longer create a match above a certain win percentage threshold, and we’re going to turn this new behavior on very soon. When we do, you could possibly wait a very long time or even not find a match at all in extreme cases. To find fair matches quickly, try to play during prime hours for your region. It also easier to match you if you’re solo or in a smaller group. In a future patch, when we think we can’t find a match based on the current matchmaking population then we’ll warn you that the wait might be a very long time. We didn’t want to overreact and create a strict rule such as “You can’t queue as a group above X Skill Rating.” There are some locations and times that can handle high skill groups and still find fair matches for them. We’ll start with this new threshold not being very aggressive, but we can adjust dynamically if needed. When we do add the UI, we’ll also make the check more aggressive.

Moving forward, we are currently looking at our win/loss streak bonuses and SR volatility, tuning and improving our expected win % calculations, and other elements of the matchmaking system. Competitive Play and Matchmaking are two systems that we’re constantly working to refine over time, and as always your feedback helps us tremendously. Keep it coming, and good luck in season 4!

–

Scott Mercer

Principal Designer

Mar 3, 2017

Thanks for all the feedback about Season 4 of Competitive play and the changes we made. I wanted to first talk about how the distribution of skill rating looks across our competitive population. There’s always been a lot of speculation about this due to the lack of complete “official” data, and there are actually some very distinct differences between the perceived population based on third party sites and our complete data.

The data below is based on season 3 data (season 4 is too new), and the maximum (not current) skill rating of the player during the season. The values below are also rounded, so that’s why it doesn’t quite add up to 100%.

Bronze - 6%

Silver - 22%

Gold - 34%

Platinum - 23%

Diamond - 10%

Master - 3%

Grandmaster - <1%

Some interesting notes:

Median maximum skill rating is not 2500, it’s actually a little above 2300. So if you have a maximum skill rating of 2350, you have a higher SR than 50% of the population!
If you do a breakdown based upon the more volatile current skill rating, there’s even fewer players above 3000 SR than listed above. Only around 8% of the population was above 3000 SR for season 3 at any one time.

With this information in mind, let’s talk about skill decay. As mentioned in the competitive blog, only having to play 1 game per week didn’t make your current skill rating very accurate or meaningful. Current Skill Rating should reflect your skill rating as an active player. We also show your Season High Skill Rating in your profile, and that won’t change during a season due to inactivity decay. While you have to play 7 games in a week to avoid any decay, we wanted players to have flexibility within the week. Play in one long session on a weekend, or several smaller sessions during the week, as long as you play seven games it’s fine.

We felt like 3000 Skill Rating was a good breakpoint for the decay rules, as it was around 10% of the competitive population (see above). If you do decay, it only affects your current displayed skill rating. This decay does not affect the internal matchmaking rating we use, so we can still place you in fair matches. When you do come back and actively play matches, you’ll also typically gain more SR from a win until your displayed skill rating and internal matchmaking rating have again reached “equilibrium”.

If you’re curious about when you need to play a game to avoid decay, you can see that by looking on the on the right hand side of the information screen of the competitive play card.

Behind the scenes, we’re going to be making another change to our matchmaking that deserves its own discussion. The simple and primary goal of our matchmaker is creating fair matches. To do that, it evaluates potential matches by synthesizing an expected win %. The matchmaker is normally really good about creating matches with a win % that is close to 50%, but if the participants in the match are either at very high or very low SRs then it gets quite a bit more difficult. There simply aren’t many players at the extremes of SR to find. This is especially true in our lower population regions AND they’re playing at off peak times such as 04:00 in the morning AND the players queue in a large group. Normally in these situations, we wait to try and find a good match, but eventually compromises have to be made so the players in question don’t wait forever in queue.

Well, we’re not going to compromise as much anymore. The matchmaker will no longer create a match above a certain win percentage threshold, and we’re going to turn this new behavior on very soon. When we do, you could possibly wait a very long time or even not find a match at all in extreme cases. To find fair matches quickly, try to play during prime hours for your region. It also easier to match you if you’re solo or in a smaller group. In a future patch, when we think we can’t find a match based on the current matchmaking population then we’ll warn you that the wait might be a very long time. We didn’t want to overreact and create a strict rule such as “You can’t queue as a group above X Skill Rating.” There are some locations and times that can handle high skill groups and still find fair matches for them. We’ll start with this new threshold not being very aggressive, but we can adjust dynamically if needed. When we do add the UI, we’ll also make the check more aggressive.

Moving forward, we are currently looking at our win/loss streak bonuses and SR volatility, tuning and improving our expected win % calculations, and other elements of the matchmaking system. Competitive Play and Matchmaking are two systems that we’re constantly working to refine over time, and as always your feedback helps us tremendously. Keep it coming, and good luck in season 4!

Ungod

Player

Mar 3, 2017

Great stuff and thank you very much for the post. Is there any chance that (next season) we’ll see some changes to the placement matches? A lot of people are very confused on how they work, and I know complete transparency isn’t possible, but some ideas would be nice. I know the idea is to place people lower than what they belong and have them climb up, but placing people drastically below where their last season left off just to face a frustrating climb is almost as bad as dropping from a high elo. Just some more info on the though process and how it works would be helpful.

–

Scott Mercer

Principal Designer

Mar 3, 2017

We’re pretty happy with how placements work right now, but we’re keeping an eye on them all the same. As we mentioned last season, we do lower your SR a bit initially but you also gain that SR back as you play. We’re trying to provide players with a positive initial experience of gaining SR at the beginning of a season without the matchmaking chaos of a more complete reset. It’s not perfect, but we like the results so far.

To clarify one thing, your initial MMR that we use to match you in placements isn’t simply based on the final result of the previous season. How you played in seasons before the prior one also factor into the calculations. Of course, how well you do in your placement matches also matters.

–

Scott Mercer

Principal Designer

Mar 3, 2017

Posted by Ungod

An example of what I was hoping for would maybe be, does the placement system put you against teams that you do have a significant chance of losing against, or even with, and then judge your performance?

We’re always trying to put you in fair matches that we think you have a 50% chance to win, and do it quickly so you’re playing Overwatch instead of staring at a queue timer. We never intentionally seek to put you in an unfair one.

I’ve seen comments like “I just won three games in a row, so the matchmaker put me in a bad stomp to get back to a 50% win rate”. It doesn’t do anything like that at all. It just keeps on trying to find fair matches. If you do win more than you lose, your SR will slowly go up. As that happens it’ll also put you up stronger opponents that match your new SR. That’s not so you lose, it’s to keep your matches fair.

There’s one other thing to remember, though. The results of a match that the matchmaker thought was perfectly fair don’t always result in a match that is hard fought every meter of the payload and you win by just holding them off from reaching the final checkpoint at 0:00 time remaining. Matches with teams of equal skill result in 3-0 stomps more than you think. Maybe an early fight snowballed out of control due to staggered spawns, maybe a player decided he was going to try to pickup a new hero that match, or perhaps your’s cat decided to play with your keyboard right before you used that Graviton Surge and it wildly missed its mark. We’re all human, and we don’t perform at the perfectly same level all the time. It’s one of the reasons competitive games are so much fun to play and watch.

–

Scott Mercer

Principal Designer

Mar 3, 2017

Posted by Verethragna

When exactly does the bonus SR end? It was at around 15 matches for me,(or 400 SR gain), so does it count matches, losses, or SR gain?

Thanks for the post btw.

It’s a bit fuzzy, but for most people it’s around 40-50 total matches played.

Scott Mercer

Principal Designer

Mar 3, 2017

Posted by CoffeeGirl

This is fantastic news, however the issue of trolls, leavers, and smurfs/derankers/throwers…has not been addressed. Would this be looked at in the future?

Oh, handling leaving is still high on our list of things to improve. It’s just a very, very difficult problem to solve in a way that the solutions don’t cause more problems than they solve, or that couldn’t be potentially exploited for SR gain. We have to be way more “mean” that we’d normally like to be to prevent exploits and discourage even worse behavior.

As Jeff mentioned back all the way back at Blizzcon, we’re still working to improve our player reporting to better identify and handle trolls and intentional throwers. Players intentionally deranking to misrepresent their skill (which does create unfair matches) is also definitely on our radar.

While I know a perfect Overwatch utopia of positive, communicative, teammates who’ll gladly play the tank or support when needed is a not completely realistic final goal, there’s still definitely room for a lot of improvement.

Thanks for the well written post on your matchmaking experiences, I’ll try to explain some of what’s going on.

The system does in fact try to place equal sized groups on opposite teams whenever possible. Your report for game 2 where both 3 player groups were on the same team definitely seems like something that shouldn’t happen based on the rules we’ve setup, so I’ll look into it further.

We do need to do a better job of not placing players into “unwinnable” matches. When the matchmaker creates a match, it determines the % chance for each team to win based on the match it made. The VAST majority of matches are usually near to 50% (especially if you’re a player closer to median skill rating and you’re not in a group), but I’ve definitely seen logs of matches where that’s really not the case and my eyebrows raise.

The unfortunate truth is that there is not always a “perfect” match for you, especially at very high (and very low!) skill ratings where there’s fewer players of similar skill. Then you throw in the desire to match groups vs. groups, with everybody having low latency, and doing ALL of this as fast as possible even though it’s the 3AM offpeak… it can get tough. We’ve tried different tunings with regards to wait times, and the improvements were unfortunately modest as we increased the time to wait. Still, this is an area we’re always looking to improve and tune better.

Fortunately, when we do put you in a match that we know isn’t a 50/50, we adjust your SR gain or loss based on your calculated change of winning. So if you did get placed into a match with only a 20% chance to win and then you lose, you shouldn’t lose much SR.

For matchmaking groups, there’s actually two separate issues that we try to solve. The first issue is “How do we handle groups formed of players with different MMRs?”. With season 2 we prevented players of REALLY disparate Skill Rating from grouping, but there’s still some variance we need to handle. Over time we’ve tested different models to try and see what’s best and are now using what tested most accurately. (Hint: it’s not simply averaging the MMRs)

The other issue is how do we model the synergistic effects of players being together in a group. As you noted, they have access to voice chat. Now here’s where things get interesting. This “massive” advantage actually differs based upon the skill rating of the group members. Based upon the data we’ve seen groups of low to mid SR players don’t see that much improvement to their win %. Higher SR players do see more notable improvements, but it’s not as huge as you might think. Still, we do take this into account when we predict the win% for each team. Regardless of how the data looks, we do know there’s a perception of a large advantage for groups. That’s one of the reasons why we explicitly try to match similar sized groups together.

So then why do points for losses and wins seem so random? Well, the amount of MMR (and SR) you go up or down isn’t simply a matter of whether you won or lost, and what was your predicted chance of winning. There’s a couple of other things at work. One is the matchmaker’s confidence in what your MMR should be. Play a lot of games, it gets more certain. Don’t play Overwatch for a while, it gets less certain. You go on a large win or loss streak, it gets less certain. The more certain the matchmaker is about your MMR, the less your MMR will change in either direction based on a win or loss.

As a minor factor, we also do evaluate how well you played the heroes you used in a match. The comparison is largely based on historical data of people playing a specific hero (not medals, not pure damage done), and we’ve done a lot of work to this system based on the community’s feedback. In fact, I’ve seen some people indicate that they don’t think we’re doing this anymore. We still are. While it’s a minor factor compared to wins/losses (The best way to increase your SR is still to play together and win as a team!), doing so does help us determine your skill more accurately and faster.

So take all that into account, the SR gain/loss after any single match can be a bit more “noisy” that it seems it should, but we’re asking it to look at a lot of different factors to do the best job it can creating fair matches for you.

As an aside to all of this…

“Fair” matches doesn’t always mean that every Ilios match goes 3-2 and 100-99 on the final point, or each team gets the payload to the end in overtime on Dorado, etc. Sometimes when two evenly matches teams play, the result can be one-sided. It just means that at that single moment in time the enemy team played better. It’s not always the matchmaker’s, your’s, or your team’s(!!!) fault that you got stomped.

–

Scott Mercer

Principal Designer

Oct 19, 2016

Posted by Light

Based on what you said, it sounds like the more you play the more certain the system is on where you should be. However, over time I hope to improve. Will the system thinking that I am where I should be have a negative result on me climbing in the future (via lower amount of SR gain from wins)?

Not in the long term. Something to remember is that when uncertainty decreases you also won’t lose as much SR when you do lose a match.

–

Scott Mercer

Principal Designer

Oct 20, 2016

Posted by Sciron

Posted by Nessuno

The effect of win/loss streaks seems to be the biggest culprit in the noise, and adds a lot of unnecessary randomness.

Whether or not your wins are strung together will have a huge impact on your “season high”, which determines both your end of season reward and your displayed tier level.

Posted by stephofni

the other two inputs into certainty make plenty of sense but it’s silly for the game to think because i’ve won 4 in a row, maybe i’m not the same player i’ve been for well over 200 games. or because i’ve lost 4, i might be a much worse player than i was yesterday on that +4 streak. someone’s skill changes, it’s going to reflect in their winrate. let it imo. trying to hurry that process along is just making a mess out of the median ranks.

These things. Seems streaks only serve to undo the system’s attempts to remain accurate.

We’ve actually done some tuning since launch so it takes several more wins or losses in a row to trigger full expansion of the uncertainty due to streaks. It’s been tricky to tune. We don’t want to have your SR seesawing wildly due to natural variance in wins and losses, but but we do want it to be meaningful enough so if someone’s MMR is incorrect (a new account, someone trying to smurf, etc.) we get them to their “proper” MMR quickly so they’re playing against the players who they should be playing.

–

Scott Mercer

Principal Designer

Oct 20, 2016

Posted by HimeSama

Here is my question: How can a mathematical equation be “uncertain”.

I know you can’t let us into the heads of your devs or reveal proprietary code, but I’m just curious. Not attacking or anything. Perhaps understanding how the system works will help people not get so frustrated when the MMR/SR system is taking a big one on their day.

The principle of “uncertainty” is in reference to measuring a player’s skill. Skill fluctuates, that’s just a given. You have good days and bad days. Your MMR is never supposed to cement at a fixed value. It’s supposed to ebb and flow. Uncertainty just measures how much it ebbs and flows. It’s not the calculation that’s uncertain, it’s the measurement of confidence in your past results shaping the system’s current impression of your skill.

Let’s say I play a match against you and we’re both brand new players. We both start at the seed value of 2500. You beat me. My rating drops to 2450 and yours increases to 2550. Is 2550 an accurate representation of your true potential? Probably not. It’s going to require more games – more data points – to determine whether that current rating is accurate. If you keep winning and winning and winning, the confidence in whether you are really 2550 is very low. If you keep losing and losing and losing, then you getting to 2550 in the first place may have been a fluke, so the confidence in whether you are really 2550 in that scenario is also very low. If you happen to go 50/50 by playing against other 2550 opponents, then confidence will be a bit higher that that’s roughly the range you should be at (depending on how consistent you are), so it makes sense to reduce the volatility a bit and reduce your rating change, since we have sufficient data to conclude that you actually belong there.

–

Scott Mercer

Principal Designer

Oct 20, 2016

Posted by D4G

As much as I love the post explaining match making something definitely needs improving.

matchmaking

This happened at 11:47am PST today and the 6 man said that they waited 6mins. When the average match takes 15 to 20 mins to complete 6 mins is way too short. I get it that it sucks for a 6 man to wait for a match but a match like this doesn’t help anyone and is a waste of both teams times.

Yeah, that’s a situation we’d definitely like to improve. The core problem is that there might not be a fair match for that 6 player group in your screenshot, not just after 6 minutes but after 15, 30, or 60. I know that people say that they’re willing to wait longer, but there are also practical limits to that patience. We’ve been erring more on the side of people at least playing the game instead of sitting in queue. Those 6 player stacks sometimes lose, too.

–

Jeff Kaplan

Game Director

Jul 6, 2016

07/06/2016 11:55 AM - Posted by Psiwolf

What was the reasoning behind punishing the winning team when someone leaves the game?

Skill Rating is very closely tied to your MMR. When we calculate MMR adjustment we look at things like was it a mis-match. The effect of someone leaving a match is different depending on when it happens. If it happens at the very end of a match it has very little relevance to MMR and thus Skill Rating. When it happens at the beginning of a match, it has a larger effect. Because no one sees MMR and MMR is not viewed as a “reward”, it’s fine that MMR works that way. MMR exists to create the best possible matches. But once we translated MMR into Skill Rating, it did not feel good. Also, in Quick Play, we backfill. So very few matches were completed with this sort of mis-match situation occuring.

So this was really a new issue caused by Skill Rating being derived from MMR (a non-player facing number) as well as no backfilling in competitive play. Hope that makes sense.

–

Jeff Kaplan

Game Director

Jun 30, 2016

Good feedback, WeaponsGrade. We’ll continue to make tweaks for next season. We do show you your career high and season high ranks, like you mention, but currently they are displayed in the rank up screen. Perhaps we can show that in more places.

Your first suggestion is a little tricky because Skill Rating is directly correlated to your matchmaking rating. To accomplish never going down in rank after ranking up, that would mean that Skill Rating would have to be separated from your MMR. This is possible but if we’re not careful it can be inflationary. Inflationary systems worked when we had 1 month seasons because we could reset before the negative side effects would set in. We’ll talk about it and see if there are some options out there for next season that might feel better.

–

Scott Mercer

Principal Designer

Mar 3, 2017

We’re pretty happy with how placements work right now, but we’re keeping an eye on them all the same. As we mentioned last season, we do lower your SR a bit initially but you also gain that SR back as you play. We’re trying to provide players with a positive initial experience of gaining SR at the beginning of a season without the matchmaking chaos of a more complete reset. It’s not perfect, but we like the results so far.

To clarify one thing, your initial MMR that we use to match you in placements isn’t simply based on the final result of the previous season. How you played in seasons before the prior one also factor into the calculations. Of course, how well you do in your placement matches also matters.

–

Scott Mercer

Principal Designer

Mar 3, 2017

Posted by Ungod

An example of what I was hoping for would maybe be, does the placement system put you against teams that you do have a significant chance of losing against, or even with, and then judge your performance?

We’re always trying to put you in fair matches that we think you have a 50% chance to win, and do it quickly so you’re playing Overwatch instead of staring at a queue timer. We never intentionally seek to put you in an unfair one.

I’ve seen comments like “I just won three games in a row, so the matchmaker put me in a bad stomp to get back to a 50% win rate”. It doesn’t do anything like that at all. It just keeps on trying to find fair matches. If you do win more than you lose, your SR will slowly go up. As that happens it’ll also put you up stronger opponents that match your new SR. That’s not so you lose, it’s to keep your matches fair.

There’s one other thing to remember, though. The results of a match that the matchmaker thought was perfectly fair don’t always result in a match that is hard fought every meter of the payload and you win by just holding them off from reaching the final checkpoint at 0:00 time remaining. Matches with teams of equal skill result in 3-0 stomps more than you think. Maybe an early fight snowballed out of control due to staggered spawns, maybe a player decided he was going to try to pickup a new hero that match, or perhaps your’s cat decided to play with your keyboard right before you used that Graviton Surge and it wildly missed its mark. We’re all human, and we don’t perform at the perfectly same level all the time. It’s one of the reasons competitive games are so much fun to play and watch.

–

Scott Mercer

Principal Designer

Mar 3, 2017

Posted by Verethragna

When exactly does the bonus SR end? It was at around 15 matches for me,(or 400 SR gain), so does it count matches, losses, or SR gain?

Thanks for the post btw.

It’s a bit fuzzy, but for most people it’s around 40-50 total matches played.

Scott Mercer

Principal Designer

Mar 3, 2017

Posted by CoffeeGirl

This is fantastic news, however the issue of trolls, leavers, and smurfs/derankers/throwers…has not been addressed. Would this be looked at in the future?

Oh, handling leaving is still high on our list of things to improve. It’s just a very, very difficult problem to solve in a way that the solutions don’t cause more problems than they solve, or that couldn’t be potentially exploited for SR gain. We have to be way more “mean” that we’d normally like to be to prevent exploits and discourage even worse behavior.

As Jeff mentioned back all the way back at Blizzcon, we’re still working to improve our player reporting to better identify and handle trolls and intentional throwers. Players intentionally deranking to misrepresent their skill (which does create unfair matches) is also definitely on our radar.

While I know a perfect Overwatch utopia of positive, communicative, teammates who’ll gladly play the tank or support when needed is a not completely realistic final goal, there’s still definitely room for a lot of improvement.

–

Scott Mercer

Principal Designer

Mar 3, 2017

Thanks for all the feedback about Season 4 of Competitive play and the changes we made. I wanted to first talk about how the distribution of skill rating looks across our competitive population. There’s always been a lot of speculation about this due to the lack of complete “official” data, and there are actually some very distinct differences between the perceived population based on third party sites and our complete data.

The data below is based on season 3 data (season 4 is too new), and the maximum (not current) skill rating of the player during the season. The values below are also rounded, so that’s why it doesn’t quite add up to 100%.

Bronze - 6%

Silver - 22%

Gold - 34%

Platinum - 23%

Diamond - 10%

Master - 3%

Grandmaster - <1%

Some interesting notes:

Median maximum skill rating is not 2500, it’s actually a little above 2300. So if you have a maximum skill rating of 2350, you have a higher SR than 50% of the population!
If you do a breakdown based upon the more volatile current skill rating, there’s even fewer players above 3000 SR than listed above. Only around 8% of the population was above 3000 SR for season 3 at any one time.

With this information in mind, let’s talk about skill decay. As mentioned in the competitive blog, only having to play 1 game per week didn’t make your current skill rating very accurate or meaningful. Current Skill Rating should reflect your skill rating as an active player. We also show your Season High Skill Rating in your profile, and that won’t change during a season due to inactivity decay. While you have to play 7 games in a week to avoid any decay, we wanted players to have flexibility within the week. Play in one long session on a weekend, or several smaller sessions during the week, as long as you play seven games it’s fine.

We felt like 3000 Skill Rating was a good breakpoint for the decay rules, as it was around 10% of the competitive population (see above). If you do decay, it only affects your current displayed skill rating. This decay does not affect the internal matchmaking rating we use, so we can still place you in fair matches. When you do come back and actively play matches, you’ll also typically gain more SR from a win until your displayed skill rating and internal matchmaking rating have again reached “equilibrium”.

If you’re curious about when you need to play a game to avoid decay, you can see that by looking on the on the right hand side of the information screen of the competitive play card.

Behind the scenes, we’re going to be making another change to our matchmaking that deserves its own discussion. The simple and primary goal of our matchmaker is creating fair matches. To do that, it evaluates potential matches by synthesizing an expected win %. The matchmaker is normally really good about creating matches with a win % that is close to 50%, but if the participants in the match are either at very high or very low SRs then it gets quite a bit more difficult. There simply aren’t many players at the extremes of SR to find. This is especially true in our lower population regions AND they’re playing at off peak times such as 04:00 in the morning AND the players queue in a large group. Normally in these situations, we wait to try and find a good match, but eventually compromises have to be made so the players in question don’t wait forever in queue.

Well, we’re not going to compromise as much anymore. The matchmaker will no longer create a match above a certain win percentage threshold, and we’re going to turn this new behavior on very soon. When we do, you could possibly wait a very long time or even not find a match at all in extreme cases. To find fair matches quickly, try to play during prime hours for your region. It also easier to match you if you’re solo or in a smaller group. In a future patch, when we think we can’t find a match based on the current matchmaking population then we’ll warn you that the wait might be a very long time. We didn’t want to overreact and create a strict rule such as “You can’t queue as a group above X Skill Rating.” There are some locations and times that can handle high skill groups and still find fair matches for them. We’ll start with this new threshold not being very aggressive, but we can adjust dynamically if needed. When we do add the UI, we’ll also make the check more aggressive.

Moving forward, we are currently looking at our win/loss streak bonuses and SR volatility, tuning and improving our expected win % calculations, and other elements of the matchmaking system. Competitive Play and Matchmaking are two systems that we’re constantly working to refine over time, and as always your feedback helps us tremendously. Keep it coming, and good luck in season 4!

–

Scott Mercer

Principal Designer

Oct 25, 2016

Right now on the PTR, we’re starting to test out changes for Season 3 of Competitive Play. While there’s one big change to Season 3, there are also some behind-the-scenes adjustments to how we’re calculating Skill Rating and Skill Tier placement that I wanted to discuss with everyone.

The largest visible change to Competitive Play will be to the off-season between Season 2 and Season 3. Based on community feedback, we’re going to shorten the off-season downtime to just one week, with Season 2 ending at 00:00 UTC on November 24, and Season 3 beginning on December 1. Additionally, during the off-season, we’ll display an in-game countdown to when the next season begins.

Before explaining the changes to Skill Rating and Skill Tiers, let me explain some of the goals for matchmaking in Competitive Play. First and foremost, we always want to provide the fairest matches that we can. Fair matches of skill between the teams provide the greatest chance for you to have fun in Overwatch. At the same time, we’d also like every new competitive season to feel like a fresh start. These two goals end up being somewhat contradictory. If we completely reset everyone’s Skill Rating (SR) at the start of a new season, then players of all skill levels would end up playing against each other and having poor quality matches until the system could reevaluate each player’s skill. Because of this, we don’t fully reset your SR when a new season begins, and instead use your SR from the previous season as a starting point.

Another area of Competitive Play we’re trying to improve for Season 2 is how we distribute everybody into their Skill Tiers (Bronze, Silver, Gold, Platinum, etc.) based on their SR. When Season 2 started, we had WAY more players in Gold and Platinum than we initially intended, and way fewer in Bronze and Silver. This was the result of how we calculated your initial SR for Season 2. We tried to partially reset player SR at the start of Season 2, but the results were not as we expected. Instead, below-average players started Season 2 at a higher SR than they should have been given their performance in Season 1. This meant that as they played in Season 2, their SR would often drop to a lower value, which didn’t feel great. It also meant that there was a much wider variation of skill in the Gold and Platinum tiers than we wanted. This is something we want to avoid in Season 3.

As a result, we’re testing a different way of determining your starting SR for Season 3 on the PTR. We’re leaning more towards trying to keep things fair rather than giving everyone a fresh start. We’re also going to initially tune your SR to be slightly lower to start. In turn, fewer players should start the season having their Skill Rating drastically drop despite having close to even wins and losses.

This change will mean that some players will not start in the same tier for Season 3 that they were placed in for Season 2, and that your SR gains from winning will be a little higher at the beginning of the season. After you play enough matches, however, your SR gains and losses will go back to normal.

We’ll be looking closely at the results of players placement on PTR and see if our goals are being met, and might make further adjustments during this testing cycle that might require a reset of placement matches. If that’s the case, we’re try to provide advanced notice whenever possible.

In the meantime, we really appreciate your feedback, and thanks for everyone who is helping us test on PTR!

–

Scott Mercer

Principal Designer

Oct 25, 2016

Posted by Taionyr

My biggest complaint is that the imbalanced S2 ranking will be the starting point for the changed S3. I don’t really see how that makes sense.

The matchmaking system is actually pretty resilient and it naturally improves match quality as more matches are played. Even now the current SR curve for the entire population is much healthier than the beginning of the season, and it will be even better by the time the season ends.