Why Handicapping (MMR) is Wrong for Competitive Play

Competitive Discussion
1283

This comment doesn’t make any sense with respect to how MM works.

So the genji in question played poorly and lost, what happens? Well, they drop in MMR (and SR) due to losing, and assuming they are not high ranked, drop a little bit extra due to performance adjustments. Then the Genji will be placed into games based on this new, lower MMR just like everyone else.

It almost sounds like you think the game “figures out” where the Genji belongs based on one game, but doesn’t move his SR appropriately. That isn’t accurate at all. The game doesn’t swing people’s MMR around wildly based on one game.

Maybe the genji just had one bad game, and normally plays better. In that case, they are at the MMR they belong. Everyone has bad games occasionally.

Maybe the player is trying to learn genji, and not doing well. They will continue to lose and fall in rank (both MMR and SR) until they reach where they belong based on their Genji play.

SR is irrelevant for match making, and generally just follows closely to MMR. So the “bad Genji” just has an MMR that moves up/down as they win/lose, just like anyone else.

It’s actually the opposite. Blizzard made the bold decision to implement a true competitive system. In beta they considered a hand-holdy, participation-trophy like system, but scrapped it under pressure from the community.

Not to be mean, but the people complaining about the system are casual players who are unfamiliar, or unfond, of competitive systems. Overwatch’s competitive ranking works the same essentially as every other competitive game, and in every game you get these same complaints.

These complaints are fundamentally anti-competitive in nature. No one is winning games and not climbing rank. The competitive players focus on improving, get good enough to win more games, and climb in rank. It’s the casual players who think they are good, who expect to be rewarded just for playing (not actively practicing) and get upset when they aren’t given achievements for being mediocre.

1 Like
1284

The example was a genji with a ton of games and a very poor WL ratio.

I also made no assumption as to how the MM works on this post. I don’t know how it works, I only know the devs said that there is a hidden MMR that after a bunch of games can very accurately place you in a given skill level.

All I’m saying is that, given the information we have, there may be some consequences to this MM system that are having negative side effects.

Jeff said they are dedicated to this and even have an engineer full time working on the matchmaking program’s algorythms. It’s fair then to assume that it’s not a simple thing and weird consequences may be arising from it.

1 Like
1285

And in the example, the genji had deranked from plat into gold. The player will continue to drop until they start to win as much as they lose.

Your question was “how does the MM handle this?” This is how: They are matched at their MMR, which falls if they lose more than they win, until they reach a 50% win-rate. Exactly what you want the system to do.

The Scott Mercer post Cuth has highlighted gives a lot of the details, and other posts expand more. The core principle isn’t that complicated.

You have an MMR that moves up when you win, and down when you lose. A few other factors effect how much it moves. You are matched on this MMR.

Of course it’s possible, and there have been bugs found and fixed in the past. However, there is overwhelming amounts of data that it works. There is endless videos of people playing competitive overwatch on youtube and twitch that show you there is no rigging or systematic unfair games. People move to the rank they belong. There is no video AT ALL showing a “good player” held down by constant bad teammates, because such a thing doesn’t happen.

In the old forums, I kept asking for video evidence. I got one person to admit they were wrong. After reviewing their own videos, they saw how bad they were playing, and realized it was their fault. Another person did post their videos, incredibly thinking they supported their point. What you say was a game where his teammates were consistently getting picks, and this player repeatedly suicided and threw. If you really want, I’ll even take the time to search through and drag up the old links.

So let’s not play this “maybe something weird is happening?” game. We know enough about how MM works. More importantly, we have essentially unlimited data from it. Cuthbert doesn’t claim that the problems with MM are rare bugs or issues, his point is that the core principles constantly create flawed matches.

If this was true, providing video proof would be effortless. The lack of evidence, given the ease of providing it if the theory was true, is all the proof a reasonable person needs.

1 Like
1286

You forgot that quote, so I’m just going to answer a quote from your post with a quote from my post.

1 Like
1287

I didn’t follow all the discussion but as a student in computer science that did a study on the ranking systems in video games:

Rating algorithms are incredibly complex for team based competitive games (sports or esports) as they are mostly inspired by the ELO system (which is was first developed for chess a 1v1 game).
The algorithms that have been developed since are the product of very long hours of works from very dedicated people and it’s not for naught that most of those are protected by copyrights or kept very secret.

Despite the fact that many great companies try to enhance the performance of their algorithms the more factuals, thus data, you want (and need) to take into consideration the more complex your algorithm is.

Not only they need to take into consideration the impact of one data (how much it will influence the final result) but also how it will interact with the others informations (ex: how should a loss but gold healing with Moira should hold compared to a win with bronze healing). And the more facts you want to consider, the more information you have, the more interactions you’ll need to check and control.

So, unless the mentioned post gives the detailed algorithm itself, it is not enough information and the core principle is that complicated.

(I’m also curious to know what you mean by core principle)

2 Likes
1288

The one sentence I didn’t bother to quote doesn’t change anything. The core principles are simple. The details behind a probabilistic (almost certainly bayesian) rank estimation can be complicated. It doesn’t matter though, because there is overwhelming evidence that the intended behavior arises as we’d expect from the basic principles.

Core principle: MMR goes up when you when, down when you lose, your matched on MMR. Secondary factors influence who much your MMR goes up/down. Basically, the same core principles of Elo.

Did you really study this topic? Because this statement is inaccurate. You can take a very simple algorithm in machine learning (like a one layer neural network, a decision tree, KNN , whatever) and apply as much data as you want to it. 10 examples or 10,000. The algorithm doesn’t change.

Now the more complex your algorithm is, generally the more data you need. I.e. with a larger neural network you need more training examples. Is that perhaps what you meant?

1 Like
1289

If you study CS and this particular aspect of rankings, what do you think may be some possible side effects of a matchmaking system trying to always create matches with a 50/50 chance, based on a hidden MMR, while balancing them to an SR.

Also, what are your thoughts on a system that starts everyone at a rank of 0 and just equates ranking movement based on win losses (and matches people only based of that simple rank number)?

1 Like
1290

you are missing the point completely. 12 people are pulled from que based on their SR, and are then grouped into teams based on their MMR to make a 50/50 match. the best players are going to be set up to carry the worst players.
this is the only way you can engineer a 50/50 outcome.
if you are performing on the high or low side of your SR, the game takes longer to find people to balance you out with.

you are arguing about how MMR works, he is arguing about how it is used.

4 Likes
1292

SR is not used in matchmaking. See

“If you do decay, it only affects your current displayed skill rating. This decay does not affect the internal matchmaking rating we use, so we can still place you in fair matches.” – Scott Mercer https://us.battle.net/forums/en/overwatch/topic/20753625906

“Skill Rating decays but your internal Matchmaking rating (the thing that determines who is matched against who – not SR) does not decay.” – Jeff Kaplan
https://us.battle.net/forums/en/overwatch/topic/20758686566#post-2

“We use MMR for matchmaking, not SR” – Overwatch official twitter
https://twitter.com/playoverwatch/status/850435344457543680

“SR isn’t used to make a match.” – William Warnecke
https://twitter.com/ww/status/867570441182826499

This assumes that their MMR is correct. If their MMR is wrong, the win probability will not be 50/50, and the result will be used to correct MMR.

This is a nonsensical statement, because SR is not used in matchmaking.

I am explaining and summarizing how MMR, SR, and matchmaking works, based on blue posts and data. He is hallucinating how MMR, SR, and matchmaking works, based on poor reading comprehension and the mistaken impression that there must be some explanation for his low rank other than his substandard play.

3 Likes
1293

Stop stop STOP. Freaking Cuthbert spreading these lies…

SR IS NOT USED IN MATCH MAKING AT ALL. PLAYERS ARE NOT PULLED FROM AN SR RANGE, AND THEN BALANCED ON MMR, OR ANY OTHER LOONEY COMBINATION OF SR AND MMR

I will provide you THREE explicit statements of this…

https://twitter.com/ww/status/867570441182826499
https://twitter.com/playoverwatch/status/849384478057299969?lang=en
https://twitter.com/playoverwatch/status/850435344457543680?lang=en

The matchmaker literally does not even know what your SR is. When a GM decays to 3000, they are placed right back into GM games. SR is purely a visual representation of your rank. It is no different than your tiers. Just like you aren’t “matched” based on your tier, i.e. you won’t only go against silver players if you are 1501 or 1599 SR, you aren’t matched based on your SR.

No, there are plenty of other ways. Like giving every player ONE number called MMR, making it move up/down based on win/loss, and then matching based on just that number. You know, like Overwatch does.

Because if you aren’t winning roughly 50% of your games, your MMR will either climb/fall, giving you harder/easier matches, until you get back to 50%.

2 Likes
1294

Man, you guys take the defense of the system seriously…

That baffles me even more than the complexity of a matchmaker.

I feel like I’m touching some taboo subject or something…

1 Like
1295

As we have said many times, we are fine with criticisms of the system. We have many of our own criticisms. But we want you to criticize the system as it actually is, not some hallucination of the system. Otherwise, you are just sucking oxygen out of the room that could otherwise be used to discuss and fix real problems.

4 Likes
1296

so you have 12 players, how do you sort them to make a 50/50 outcome other than sorting them on teams based on their mmr?

then how do most matches hit 50/50. the devs repeat this over and over that most teams have a statistical 50/50 shot based on match making.

look guys, im not saying anything crazy here. SR isnt used in matchmaking, i been wrong before, thats fine.

what im saying is they engineer the teams to be as balanced as they can be, based on mmr.

i look back at that post and see i really blended what i was trying to assert, and where i felt the person i tagged didnt understand about the OPs post.

1297

Step 1 is to ignore the OPs post, as it will only give you bad information in a very slow and confusing way.

There are many ways of accomplishing a 50% win percentage. Specifically, they do it as I describe in the post that you quoted (https://us.forums.blizzard.com/en/overwatch/t/why-handicapping-mmr-is-wrong-for-competitive-play/646/8?u=kaawumba-1133). The official (but somewhat less clear) version of this is at https://us.battle.net/forums/en/overwatch/topic/20753625906#post-13,

In short, raise MMR on a win, lower it on a loss, and match on MMR. People will rise or fall to their 50% level naturally. If they improve, compared to the community, they will be able to climb again.

1 Like
1298

ok ok i think we have our wires crossed. im talking about in one game. not over the series of multiple, or over all. how they balance for a 50/50 outcome in one game. i dont believe in forced wins or losses.
do you follow me here.

1300

It is already the case. How often you see close matches. That you would call gg even if you lose?

1301

They match on MMR. In an ideal world, in every match, every player would have identical MMR. In a slightly less ideal world, there is some spread in MMR, and the vast majority of players (everyone who is not at the bottom or the top of the scale) would be equally likely to be the bottom or the top or the middle player in the match.

Balancing in this way only predicts a fair (50/50) match. It does not guarantee it, because MMR may not be accurate.

1 Like
1302

This back and forth talk is all nonsense. They said explicitly that there is a complex system that tries to balance games.

It’s a nice idea for QP, but moving on…

This isn’t really doable in a game where you have tons of completely different heroes and different roles. Some guy may be good with a Tracer and suck with a Mercy.

You have people queuing who only know how to play some characters and then are forced into characters they don’t know how to play or are stubborn and end up with comps that won’t work. Then the balanced game is all of a sudden not balanced anymore.

Trying to balance games is pointless, unless you queue with a hero beforehand and then the MM can go and produce teams based off of that hero’s stats.

So it leads to frustration a lot of times. The solution may be some sort of hero or role queue or just plain win lose and random matchmaking. At least you know for certain you have to ride the variance.

1 Like
1303

I left this out, for simplicity. But since you brought it up: I agree that a predicted win percentage of 50% does not mean an actual win percentage of 50% even if all the MMRs are accurate. Boiling thousands of variables down to one MMR is horribly reductive. Just because teams are in theory equal does not mean it turns out that way in practice. Aside from the fairly obvious issue of wanting a balanced team, hero wise, and well suited to the map, sometimes teams just don’t work well together. You could imagine an example where a team has no leaders, or two leaders who refuse to work together.

It’s reasons like these that make a statement like predicted 50% win percentage for a single match not particularly valid. It is only really valid over very many games where the players are constantly being mixed (or you only match teams, not individuals). And by that time, people’s skills have changed, so it is always a moving target.

2 Likes
1304

So, there it is, I believe they should scrap the whole nonsense of saying they track your progress since beta and stating they have MMR and PBSR and just go ahead with a simple and transparent system of wins and losses, fixed points and decay.

1 Like