So it actually blew up on reddit: https://www.reddit.com/r/OverwatchUniversity/comments/aatezy/why_match_quality_is_frequently_poor/.
Nice! Great job man!
1 Like
This is the problem – a focus on math instead of user experience. You have to code a matchmaking system with logic and reason and rule sets based on experience, not based on some theoretical simulation. Blizzard has enough data to do this already. They choose not to.
I think the matchmaker they have right now is suitable for quickplay, not for competitive. In quickplay you want fair matches, teams that are manipulated so that one team does not have an advantage over the other.
In competitive you want a ladder that works itself out organically. The only way to truly evaluate players skills, is to allow them to play against each other in a scenario where everyone is assumed to have equal skill, not where the matchmaker arranges players so that each team has an equal chance of winning.
1 Like
Very well then. Create a matchmaking system that you think is better than Blizzard’s or Microsoft’s. Be detailed. Discuss edge cases where you system fails and how you will mitigate the failure. Be concerned not only for your experience, but for everyone’s experience.
Create a matchmaking system that you think is better than Blizzard’s or Microsoft’s
I’ve already done that. A number of professional poker systems in use in the real world are using my code for matchmaking.
I could fix Blizzard’s system for free if they were open to it. Since they’re not, the only reasonable conclusion is they like it the way it is. Broken.
Thank you for this… this really explains a lot of the inaccuracies in the matchmaker.
I don’t agree with your proposed improvements as they introduce things into the matchmaker simulations that would actually increase your “smurf” rating.
For example, by my experience, the 6-stack queue and solo q have wildly different skill levels, but use the same number and a player can go in and out of it at will… basically “resetting” their mu. A 2100 sr player in a 6-stack LFG environment, performs differently at 2100 solo q.
You can only insure that the numbers mean the same thing if they are playing the same game. This would require removing the barrier from solo q people from facing a 6-stack, even if another 6-stack is available for the 6stack to face.
As you showed, the more frequent there is an anomoly in the SR from their real rank of their opponants, it increases the amount of games it takes for the rating to stabilize.
You also bring up that players should use LFG to reduce the amount of inconsistances. However, LFG actually increases the range in which you a matched up with people. For example, the normal matchmaker generally around Gold keeps teammates and enemies within 15sr of your rank. However, in LFG you do not have that barrier… and the wide range of teammates you can have can open the ranges as far as 1000… same thing with any grouping mechanic (which doesn’t seem to be included in your model).
Adding guilds and clans also does not increase your model, unless the clan is restricted to the same 6 players in every game. This is way too unrealistic to expect from a playerbase that can’t seem to regroup very often outside of a playsession. If you assign 1 team SR to a group of players whose compesition can change… you’d have to pretty much multiply your inconsistancy modifier by 6 or more by the sheer number of different combinations of teammates AND their natural inconsistancies of their individual parts. You wouldn’t be able to have a general 100-300 inconsistancy… it would prob need to be closer to 1000-1500. Along with a internal smurf modification if someone with drastic skill differential is added or subtracted from the team from game to game. A clan system would be a mess and would have no way of determining an actual skill ranking this way.
The most important take away is that you will as an individual, have a better experience with the matchmaker if your play is consistant. Playing the same way, such as “always” in LFG… or always in solo q… or always with the same teammates. Modifying any of those factors can drastically increase the inaccuracy of rating system and thus putting you in an incorrect game because the game doesn’t rate you differently based on how you perform with friends, alone, in group queue, etc. It only has one rating for you. When you perform at drastically different skills in each scenerio you throw the matchmaker for a loop and have bad games.
the current system doesn’t have answers for its current fails. An example:
Let’s say I hop on comp and use my best hero’s only in order to make sure I keep my rank. then I am grouped with a team that has 3-4 DPS only players and we will definitely lose unless one of us does not play his best hero which would be a tank.
I decide to play tank (Reinhardt) in order to help the team win, he is not my best hero, but I can hold right click and shield my DPS.
I use Reinhardt and don’t do bad, but I don’t do well, however, my team is able to win. The next match I am faced with a similar scenario and because I just used Reinhardt I feel a bit more confident and use him again. I don’t do bad but I don’t do well, my team wins and they thank me for being rein when nobody else wanted to.
My metrics on Reinhardt were not very good so I am PUNISHED by the matchmaker algorithm believing that I belong at a lower rank so I am placed in order to lose. I did not throw, quite the opposite, I took a bullet by playing a position that I don’t even want to play in order for the good of the team, you might even say that I am the reason they won those games yet I am now marked for downrank.
Really think about how rotten that is combined with the fact most people don’t even realize that’s partially how the system works.
Share your code (or pseudocode that is close enough that a programmer could correctly recreate what you’ve done). Make modifications appropriate to Overwatch (as no variant of poker that I’m familiar with is 6v6)…
I agree with just about everything you wrote except this part. I don’t believe in the “forced” win loss thing. However, in plat and below, you will have lower SR gains because of pbsr.
If you win enough games “performing badly” according to stats because maybe filling actually earns you wins… or you win in ways that don’t accumulate stats… you will continue to get penalized SR until your SR catches up to your MMR.
You will still be matched up based on your MMR, not your SR so you can’t be put into an unwinable situation until your MMR gets raised too high because of flexing and rng that you can no longer win.
Agree with S23 fully. Been saying it for years, MMR is manipulation and nothing more. Simple is fair, take 12 people in an SR range and group them randomly or by giving the closest avg SR for the teams. They need to drop this magic secret MMR number which can only be based on hard stats and can never consider things like communication, flexibility, teamwork, commom sense, positive attitude…it just sees bang, bang stats.
2 Likes
I’m on board with that, just the idea that anyone is punished in any way shape or form by playing a hero they aren’t so good at but for the betterment of the team is a bad idea, especially when they don’t even know it’s happening.
1 Like
The reason that LFG should help is that it has role-queue, which should reduce inconsistency. In my experience with LFG, the groups are a little wider than solo queue, but not really bad (maybe +/- 150 SR). The LFG experience definitely varies though based on the rank/region/platform/time/day in which the player is playing.
Yes, in order for the team to have an SR, the team must be an identifiable entity. I’d propose a team of 9, (6 + 3 subs) with any changes to membership (except for adding players up to the cap) causing the team SR to be reset. Also, new teams can only form once per season, and an account can only be a member of one team.
While we are at it, because the team SR converges to the correct value so much faster, it would probably be appropriate to reset SR each season, and require 50 games in a season to get extra special team rewards.
Finally, keep in mind that my proposed improvements are the least thought out part of the post. They are more “things to think about” rather than “this is my position, and I’m sticking to it”.
Yes.
Unfortunately… there are too many loud voices that say, “i LoSt bEcAuSe oF mY DuMb tEaM, we should be judge based on our individual “skill””. So Bliz will never get rid of the pbsr system when they believe it will help calm down the people complaining about smurfs and the ones that want to be judged based on their individual number merits.
This is just false. The system does not rig games to push you to a specific rank.
Then please give some sort of explanation for why people repeatedly report going on loss streaks exactly at the threshold of a new tier. Did the matchmaker think they were getting boosted? Are there gateway matches? To say that “nothing different happens at tier thresholds” translates to “there is something going on that I am unaware of” because it does exist and is a source of frustration and confusion by who knows how many people playing this game.
That is displayed in this post.
Say you are a 2300 player… but you’ve gone on a pretty good run and are approaching 2500. Say you got good rng for those 200sr. But rng is a fickle thing… its random… and you are likely no longer going to any more of the string of 40% games you can’t lose. The other 60% of games come around… and the 20% you can control you lose also… because the enemies are better then you.
You will keep losing falling all the way back to 2300… or maybe even further if you get tilted at losing or the rng is not in your favor.
It is random. The odds are that it will equal itself out… but just like coin flips, it is possible to get 10heads in a row. Just like 10 tails in a row.
Just like the charts show… when there is inconsistancies in the rating, you get inconsistant results and poor matches. The more variables that threw in uncurtainty into the rankings for simulated games, the more likely there would be a game that didn’t turn out correctly.
You could have run into teammates having a bad game, or enemies having a really good one. Combine that with most players who DO experience loss streaks, they most likely are underperforming and just lose.
CSGO has this (in a weird way) has this via Trustfactor and their “Overwatch” system. What makes it ineffective is that the devs or Valve staff in general don’t double check it themselves, leaving it mostly to the community.
And speaking of hardware ID bans, doesn’t Blizzard already do this for hackers?
This doesn’t add up in my experience. In order for that to make sense I would have to believe that coincidentally the “other 60%” games happen to begin occurring right at the tier threshold, almost every time. I just don’t buy it.
What if it doesn’t though. What if the change in SR being 24 is actually 4sr x 6players. It is a built in thing we’ve already got. Since the matches are actually based on the team avg SR, not the individual player SR.
What I’m saying, is that even if you have one team SR, it would be no different than 6 people all playing the same time all the time and being diamond+. We must have already proven that new accounts that already play as a team and only in that way with the same people do NOT have the same SR or even the same change in SR.
Yes, you would pretty much have to have a SR reset every season, especially only allowing each player to have 1 team per account per season. The problem is, resetting SR will throw the matchmaker for a loop in those early games, just like a MMR reset would do in solo q. The problem being that unless the seasons are really short… like a month at most, the amount of games a full team will be able to put in will be very low before they break up, especially if they perform badly.
There will be VERY few teams that will even stay together long enough to get to their true rank. Thus making the whole system a waste of resources to even impliment.