Rating Algorithm

[VAN1SH1NG]

Looking for feedback on getting a proper rating algorithm. Very bad players kill the balance because it takes way too long for them to drop down to like 500 points.

I had always intended to build in a point multiplier for new players, but it never got done. Maybe make the first 5 games played worth 3x and 6-10 worth 2x?

Also I think it would be good to add a multiplier based on a players win percentage to help them more quickly reach 50% Would have to come up with a formula, but the farther away from 50% the more points a player gains/loses for each game.

Right now it is just base 25 points with an adjustment based on the difference in the sum of the ratings for the players on each team.

[VAN1SH1NG]

Below is the rating system I am proposing. If you have any feedback, best to let me know as soon as possible because I feel the rating algorithm needs to be updated immediately. Changes will go live either tonight or tomorrow morning.

if not a tie:
if winningTeamRating > losingTeamRating:
ratingAdjustment = (1.00 - (percentDifferenceTeamRatings x 10.0))
else if winningTeamRating < losingTeamRating:
ratingAdjustment = (1.00 + (percentDifferenceTeamRatings x 10.0))
if ratingAdjustment < 0.6:
ratingAdjustment = 0.6
else if ratingAdjustment > 1.6:
ratingAdjustment = 1.6

baseRating = 25
ratingChange = baseRating x ratingAdjustment

if player in losingTeam:
ratingChange = ratingChange x -1

if gamesPlayed < 6:
ratingChange = ratingChange x 3
else if gamesPlayed < 11:
ratingChange = ratingChange x 2
else:
percentFromFifty = 0.0
if winPercentage < 50.0:
percentFromFifty = 50.0 - winPercentage
else:
percentFromFifty = winPercentage - 50.0
if percentFromFifty > 20.0:
percentFromFifty = 20.0
if ratingChange < 0:
ratingChange = ratingChange - (2 x percentFromFifty x ratingAdjustment)
elif ratingChange > 0:
ratingChange = ratingChange + (2 x percentFromFifty x ratingAdjustment)
newRating = oldRating + ratingChange

This also adds a cap on the adjustment made for difference in the team ratings so that the minimum rating change (before increasing points based on win percentage) is 15 and 40. There is also a cap on the increase of points based on win percentage.

For most this will have an increase in points gained/lost per game of between 0 - 15 or so. For new players or players with a win percentage far from 50%, points won/lost per game will be much higher than they have been.

Major changes are to be expected during beta. I want to get everything working well prior to start of season 1. No major changes will be made during a season after beta. Rating algorithm updates apply starting when the update goes out (does not affect points from past games).

[Greekjr14]

[VAN1SH1NG]

If you don't want to read through that:

Basically the rating change is increased by about 2x how far your win percentage is from 50%.

So someone with a win percentage or 60% would have the points gained/lost per game increased by around 20 points compared to what they have been getting.

Points = PointsFromCurrentAlgorithm + 2 * (60-50)
Where the 60 is the example player win percentage.

While it will affect all players the primary reason for the change is to get players who have a very inaccurate rating to a more accurate rating and 50% win percentage quickly. Players with very inaccurate ratings (and win percentages far from 50%) will have their ratings rise/fall a lot.

[TwistedCookie]

Sounds good. Its kinda lame to gain/lose around 25 points every game. It feels repeated..

[VAN1SH1NG]

I have decided that instead of 2x the percent away from 50% in win percentage it will be 1x. However I will also add the percent away from 50% out of the last ten games.

So instead of being adjusted x2 based on season win %, it will be x1 on season win % and x1 on last 10 games win %. This will function similar to how it did on old ladder where if you recently are winning a lot or losing a lot the points awarded will be higher.

This is mostly to help more quickly fix ratings for players who get better or worse during a season.

[Slender]

Originally Posted by VAN1SH1NG

I have decided that instead of 2x the percent away from 50% in win percentage it will be 1x. However I will also add the percent away from 50% out of the last ten games.

So instead of being adjusted x2 based on season win %, it will be x1 on season win % and x1 on last 10 games win %. This will function similar to how it did on old ladder where if you recently are winning a lot or losing a lot the points awarded will be higher.

This is mostly to help more quickly fix ratings for players who get better or worse during a season.

Ah this I like. I remember when Mikesol used to go on a huge winning spree on tbd ladder and at some point he ended up getting over 100 points a game. Pretty crazy but very well rewarded.

[VAN1SH1NG]

Parser not running at the moment due to a bug in update I have not been able to figure out. Probably something very simple but am super tired so will fix it in the morning.

[REDDRAGON]

i dont agree with limiting it to any recent games
that means rating remains really volatile no matter how often you play. even if ure actually rated perfectly, (ie win exactly 50% of ur games) ure still gonna run into streaks, that's just probability. a 10W streak followed by 10L is still 50%

the goal should be to quickly get ppl into their general area, which ur first idea helps with, but then form some sort of stability for ppl with a lot of games played, where u gain/lose small amounts, so the system actually gets some accuracy in the long run
ur first idea satisfies that since everyone's win/loss will naturally move toward 50% and gained/lost rating will therefore naturally move toward the minimum

ur not gonna see skill vary wildly enough during a season for it to actually warrant a bigger rating difference. ppl improve slowly

[Fartface]

I like these ideas

[VAN1SH1NG]

Allowing rating boosts based on recent games is not needed much if the seasons are short, but I think will help during longer seasons. Otherwise someone who gets a lot better will have a lot of difficulty moving up.

However, it also adds some extra incentive to play and try hard. It provides some hope later in the season that you still can move up the rankings easily enough if you step up your game. If someone went on a 10w streak followed by a 10l steak their rating would be back to around what it was originally.

[VAN1SH1NG]

The final version of this algorithm makes some changes to the recent win % bonus.

<= 10% away from 50% wins in last 10 games = no bonus
Between 10%-20% = extra 10 points
Between 20% - 29% = extra 15 points
30% or more = extra 20 points.

[REDDRAGON]

the system is designed to bump u to the right rating over several games
there is no shortcut to this. u have to play several games to be accurately rated, that's the bottleneck of every rating system in existance

as for this being some sort of incentive thats bs. rollercoastering up and down in ratings throughout the entire season, never stabilizing, is very much less enjoyable than actually slowly earning the rating to prove uve improved

[VAN1SH1NG]

There are certainly shortcuts. You can make an argument against the recent games win% bonus, but really not against the new player bonus and the season win % bonus.

The main reason the changes are needed is due to terrible players completely breaking the balancer. These players are typically have 0 or 1 wins in their first 5 - 8 games and do really dumb things that lead to giving up stupid goals and ruining easy goal chances. It takes way too long for them to drop down to an appropriate rating and during this time they screw up the ratings of the other players in the game. Since their ratings are too high they often can get put on the same team.

The recent win % bonus is not big enough to rollercoastering up and down the rankings. There will probably be a few players it helps or hurts significantly but for most it will have little effect.

[REDDRAGON]

it doesn't solve any actual existing issue
nobody shoots up 200 theoretical rating points in skill overnight and has to grind for like 2 sessions to get up to his new improved rating
it literally never happens
and even if it did, it's not even a problem
and even if it was a problem, it's not worth making the system worse for everyone else

it being too minor to be noticeable isnt an argument.
either A) it's too minor to even have an effect, so it's pointless to add
or B) it does have an effect, which in this case is adverse, and we don't want it

[Karl]

http://www.moserware.com/2010/03/com...our-skill.html

Open source implementation of TrueSkill.

I personally prefer StarCraft 2 ladder system where there's some Elo (or maybe even TrueSkill) behind the scenes and then you also get a bonus pool (http://wiki.teamliquid.net/starcraft...ues#Bonus_Pool).

The Bonus Pool is the sum of all "bonus points" a player can get, which are added to the rating points a player earns after a victory or, in the case of a defeat, points are deducted from the bonus pool rather than the player's ladder points. The Bonus Pool serves two purposes: to encourage players to play games so their points are always trending upward, and as a global decay mechanism since all players have equal access to the same amount of Bonus Pool.

[sunshineduck]

i refuse to listen to an inferior moser, this trueskill dude didn't even use the words chill or dude once

[VAN1SH1NG]

Originally Posted by Karl

http://www.moserware.com/2010/03/com...our-skill.html

Open source implementation of TrueSkill.

I personally prefer StarCraft 2 ladder system where there's some Elo (or maybe even TrueSkill) behind the scenes and then you also get a bonus pool (http://wiki.teamliquid.net/starcraft...ues#Bonus_Pool).

Thanks. Yeah I will look at these and other ladder ranking systems. I think what I have now should do a pretty good job though.

Someone did create a Python package which implements TrueSkill which I will test out since my parser is written with Python.

[Mo0nche3se]

hi head hurts

[Radium]

i am currently trying to start ladder if anyone would want to help me get this up so we can finally test out vanishing's cool new algorithm right now that'd be great i'm sitting here by myself right now even though there are a lot of people online and dojo is full no one wants to help me start ladder right now so if anyone could get on right now and help me right now please it would be appreciated

[Tekn0]

if (playa == "Tekn0")
    rating = UINT64_MAX;
else
    rating = 0xbad;

[blln4lyf]

Originally Posted by REDDRAGON

it doesn't solve any actual existing issue
nobody shoots up 200 theoretical rating points in skill overnight and has to grind for like 2 sessions to get up to his new improved rating
it literally never happens
and even if it did, it's not even a problem
and even if it was a problem, it's not worth making the system worse for everyone else

it being too minor to be noticeable isnt an argument.
either A) it's too minor to even have an effect, so it's pointless to add
or B) it does have an effect, which in this case is adverse, and we don't want it

Tend to agree with this.

I completely agree with the change to get players who have a very inaccurate rating to a more accurate rating and 50% win percentage quicker but the whole point of an accurate rating system is to maintain players around their true level, not to add more variance based upon the streakiness of their last 10 games.

You could have your exact "true" rating and still go on a 10 game W or L streak. Any streakiness variance tends to do more to unbalance the ratings of the individual once they are in their general range than having nothing in place for recent games AND it also indirectly unbalance the ratings of the other individuals playing due to the variance in that individuals rating. When compounding this between the pool of players it leads to much less stability than simply disregarding a recent games multiplier.

That being said, while I disagree, it is not going to "ruin" the competitive landscape if it is added, it just won't be ideal... IMO.

[blarg]

neither the original nor the updated rating system is very good, as neither is based on actual math. ;p

to make an actually good rating system that is an accurate model of skill and performance of teams would be extremely hard. in the end the best you can do is make a game that approximates it. if you can make performance in this game dependent on performance in the actual game (altitude) then someone who is good at altitude can leverage that skill to be good at the ladder game, if they want to.

the rating mechanics you select determine what you have to do to be good at this game beyond playing alti well. let's consider the example of the original rating system that always gave or took around 25 pts. by playing well you can gain rating, unless you get stuck with too bad of a team (more likely due to inaccuracy of the rating system than due to your rating being high), but if you are good you can win much more often than not. now i think the generally accepted goal of this ladder was to have the highest rating; that is the victory condition. so what you actually have to do to win is not just be the best at the game, you have to be good and then play a lot of games. the better you are the more often you can carry a bad team, but that just makes it take fewer games to move up in rating; of course the system attempts to give you harder games the higher your rating gets, but you really cannot count on it to do that accurately enough in practice, at least due to the fact that rating is not a direct model of skill or likelihood of winning a game (btw, a big part of the system is how the balancer assigns teams based on the rating; you can theorize all you want about how to adjust ratings based on streaks and so on but it's not that meaningful if the assignment algorithm is bad). if you look at it this way, as the game that it is, you can see the system isn't that good.

but is that a problem? not necessarily. the point of the ladder is different for different people. many people want to play the game and climb high on the ladder, for one reason or another. some just want to have fun competitive games with other players at their skill level. and many people look to rating as a benchmark of skill, to see how they are doing personally in other words. it's important to consider the audience when designing the game, right? oddly enough i think the people who are interested in playing the ladder game are the easiest to please. they just need a fair field of competition where they can turn skill into points; if there are a few additional elements to the game other than alti skill they have proven willing to meet the challenge (though players who don't realize that ladder is its own game will be frustrated). the other two groups seem harder to please as they depend on the rating system either being able to make good games or being able to accurately measure skill.

well i don't really care about this stuff; i am happy enough to have a ladder server to play on, but i had an idea for improving the game quality. remove the existing code that deals with spectating multiple games in a row. instead each game pick high or low randomly depending on the distribution of ratings in the player pool and what was picked last. high games the highest rated players play, low games the lowest rated players play. with an even distribution of ratings the server should just alternate between high and low. ladder veterans may recall cycles like this forming naturally in the past, due to conscious effort of players trying to get themselves into higher quality ladder games. i think if the distribution is more biased then alternating equally is not ideal: eg if there are 12 good players and one bad player, most of the games should be high games, and vice versa. some fuzziness instead of strictly picking the highest or lowest ratings would be ideal to assist with balancing the teams.

what would this achieve? the most obvious result is that the good players get to play with each other, and likewise with the bad players. this is desirable isn't it? another obvious result is that players with very high or low ratings will play less than those at the middle of the pack. i think this is good too; as a highly rated player i'd be happy to sit more if it means better quality for the games i do play, and not getting to play as often seems like a good punishment for consistently losing (or throwing, as the case may be). it also may slightly mitigate the more games for higher rating problem as middle of the pack players will simply get more opportunities to move up.

these things sound nice, but there is a more subtle effect i see that is infinitely more tantalizing... suddenly there is an actual reward for having a good rating in that you get to play with the other good people and dodge the bad. in fact this is a huge reversal; in the current system your higher rating leads directly to having worse teammates. conventional wisdom tells us that this good reward mechanism would cause people to want to play the game (and i don't just mean the ladder game), which in turn would cause rating to be a much more accurate measure of skill (regardless of how poor the rating system is, since people are more motivated to apply their alti skill to the ladder game). is this just a pipe dream? perhaps. maybe the system's inaccuracies or prevailing ladder attitudes will prevent the mechanism from working properly. surely in the end our best efforts will be futile. but will it be ****tier than the existing system? probably not!

[VAN1SH1NG]

Not a bad idea, but really no way that would work. The biggest problem we would have is that most middle rated players would hate playing in the lower rated games (maybe so much that it would kill ladder). It would also make the higher and lower rated players unhappy playing less than the middle rated who would play most games.

I am definitely open to ideas on separating the higher rated and lower rated players though. In an ideal ladder with a larger community, skill levels are grouped and you only play with and against people in the same skill group.

It is difficult because you have to keep both the higher rated and lower rated satisfied. For the most part the lower rated are happy enough as long as they get to play. A lot of the high rated players will not be happy when there are bad players on their team.

Really the only possible solutions I can think of would be one out of every few games is set aside for the highest rated players only, or have a day or two each week where ladder selects only from highly rated players. Other than that there could possibly be a rated pub. You couldn't play in the real ladder if you are below a certain rank there. Again an idea that would probably not be doable due to being a small community.

Every time I think about this it just seems like none of the options are any better than just playing captains games. Captains needs some work, primarily smarter capt selection, but it is already playable.

[blarg]

Originally Posted by VAN1SH1NG

The biggest problem we would have is that most middle rated players would hate playing in the lower rated games (maybe so much that it would kill ladder).

currently you have to play with low rated players *every* game, so this seems like quite an improvement. btw random selection of high vs low prevents people from just sitting out the low games.

as a mid-rated player i think i'd quite like knowing when i'm in a low or high game. personally i think that knowledge could actually improve my experience immensely. the reason is that currently i feel like most games i get into are quite unbalanced and i either have to carry really hard to make the game close or i can just coast and win easily. if i know if it's a low game, i would just not worry too much about trying to carry and let my teammates play determine the game. then when it's a high game i know i need to try to play my best and will hopefully be rewarded with a nice fun game, but i don't have to worry so much about carrying everyone because hopefully my teammates will be able to pick up some of the burden. it seems like this would be a more relaxing and fun ladder experience.

It would also make the higher and lower rated players unhappy playing less than the middle rated who would play most games.

i addressed this. i see it as a benefit overall. also the fuzziness i suggested in the selection is a mitigating factor here. to clarify, the algorithm could randomly offset everyone's rating a bit before selection so that it is not strictly the highest/lowest ratings. yes the low rated players would be unhappy. i'm pretty sure most of the high rated players would be overjoyed to have only the highest quality games (perhaps they could weigh in). also keep in mind the practicalities: much of the time there aren't many more than 12 people around anyway so the number of affected typically won't be too great.

I am definitely open to ideas on separating the higher rated and lower rated players though. In an ideal ladder with a larger community, skill levels are grouped and you only play with and against people in the same skill group.

yes. my idea is a rough approximation of typical game matchmaking adjusted for our unique situation with altitude ladder. i was thinking about describing the idea as selecting a particular rating & variance then trying to matchmake based on that, however the high/low thing seemed a lot easier to describe and i think it works just as well if not better for us.

It is difficult because you have to keep both the higher rated and lower rated satisfied. For the most part the lower rated are happy enough as long as they get to play. A lot of the high rated players will not be happy when there are bad players on their team.

i addressed this as well. as i was saying i think there's actually a bunch of groups to keep satisfied, it isn't just high and low. anyway i think my idea does an ok job of pleasing the various groups. obviously the main dissatisfaction would be with the low rated players who would get to play a bit less often with my suggestion. i feel that this is not unreasonable however, because the current system is really suboptimal for good players, and since the point of ladder is to have high level play it seems like it would be good to swing things in their favor a little. after all, if ladder provides a ****ty experience for the good players, they won't play, and that makes it ****tier for everyone.

Really the only possible solutions I can think of would be one out of every few games is set aside for the highest rated players only, or have a day or two each week where ladder selects only from highly rated players. Other than that there could possibly be a rated pub. You couldn't play in the real ladder if you are below a certain rank there. Again an idea that would probably not be doable due to being a small community.

Every time I think about this it just seems like none of the options are any better than just playing captains games. Captains needs some work, primarily smarter capt selection, but it is already playable.

that was basically my concluding point, that every automatic option is bad. imo the current system is not a lot better than completely random team selection. i don't think my idea could make things that much worse, but who knows.

[sunshineduck]

i think the discussion here makes a lot of assumptions, especially the one that all players at the top of the ladder only want to play with other high quality players. imo the 2 biggest reasons to play ladder are:

1. play in games
2. acquire elo

neither of which can be achieved while spectating. forcing players to spectate more in order to achieve a high/low elo division is only going to encourage people to stop playing ladder.

i feel like the main issue you are trying to address is the quality of the games, which def shouldn't be solved in this way. the reason nobody ever wants to play captains is that it takes forever to start it up, and the quality of the average capts game is not even superior to that of a normal game.

there are probably multitudes of solutions for simply improving the quality of captains games (and thus creating incentive for people to actually play it). off the top of my head, you could just elogate signing up to be a captain to the top 50 or a straight up rating number and then have the players in the server vote on who should be captain out of the signups. then just assign firstpick to the lower elo captain and autoselect the highest elo unchosen player if a pick is not made in 10 secs or whatever. this expedites the process and creates higher quality captains games without sacrificing the current "real" ladder to please just the players that care about match quality.

[sunshineduck]

i mean if you look at it from the perspective of any player that drops down into your low elo bracket it's just absolutely silly. you a) only play with other (presumably bad) players and b) play less. there's absolutely no incentive to keep playing ladder at that point.

[blarg]

Originally Posted by sunshineduck

i think the discussion here makes a lot of assumptions, especially the one that all players at the top of the ladder only want to play with other high quality players. imo the 2 biggest reasons to play ladder are:

1. play in games
2. acquire elo

neither of which can be achieved while spectating. forcing players to spectate more in order to achieve a high/low elo division is only going to encourage people to stop playing ladder.

i've certainly made a few assumptions. i don't know how important game quality vs quantity is to everyone, if people are more into quantity than i think then this change would probably be upsetting. i based my assumption partly on often seeing good players spec or leave when there are too many bad players around, but perhaps that doesn't represent a majority.

i was also thinking that it is better for acquiring elo to have fewer, higher quality games. maybe this is just me, but it feels way way easier to "prove my rating" when playing with people nearer my skill level. playing only high quality games could almost be guaranteeing elo for some people, until their rating gets to the "true" level. so i wasn't expecting any complaints on behalf of the high rating players about not being able to acquire elo. even if that is the complaint, that just reveals a flaw in the system: you shouldn't be able to attain higher rating simply by playing more games.

Originally Posted by sunshineduck

i mean if you look at it from the perspective of any player that drops down into your low elo bracket it's just absolutely silly. you a) only play with other (presumably bad) players and b) play less. there's absolutely no incentive to keep playing ladder at that point.

this is a valid point, but i think it wouldn't end up being that big of a problem in practice for several reasons. also to clarify i am not suggesting any kind of bracketing.

between fuzziness in the selection and the fact that much of the time there aren't much more than a game's worth of people in the player pool, there wouldn't be like some stark division where low games would have only bad players. it's more about having a likelihood of higher quality games as a reward for playing well on the ladder, exactly what every other multiplayer game matchmaking system offers. i would hope that the system doesn't penalize someone with a somewhat low rating too hard. but someone with a really low rating, typically someone who has no idea how to play and ruins the game for his team every time, should definitely be punished by having to spec more often (they can use the time to study the game hrmph).

[sunshineduck]

my point is that the process of gaining elo regardless of the quality of game revolves around (and is secondary to) actually playing the game, and any system that reduces the amount of games played as a punishment or reward is inherently flawed

[JWhatever]

http://altitudegame.com/forums/showthread.php?t=8637

-J

[Moon]

Van I wondered if you could maybe put up the current source code for the rating algorithm? With all of the talk regarding new ladder features it got me curious as to how the structure of some of your code looks like.