The Algorithm: 2011

Now that my food coma is wearing off, I thought this would be a good time to go back and look at how the algorithm fared for the 2011 season. If we look at every bout from the end of Championships 2010 until the end of Championships 2011 (601 bouts in all), the algorithm correctly picked the winner in 491 of them (81.70%). This was slightly worse than our 85.1% from the 2010 season. However, given that we used the data from 2010 to train the algorithm, it would be unreasonable to expect the same result.

One of the effects that elicited several comments was the impact of new teams. Since we chose to assign a rating of 600 to any previously unranked team, some rather unfair alterations in the ratings of their first few opponents would result. And, while it's true that the proper ratings would eventually be restored after a few more bouts, it seems reasonable that those ripples reduced the overall effectiveness of the algorithm. Out of simple curiosity, I excluded any bout in which a completely new team participated. That reduced the total sample to 570 bouts, of which the algorithm correctly predicted 466 (81.75%). Clearly, the new bouts didn't really affect the overall accuracy.

In fact, our worst performance was from the final post-season tournaments. In those 80 bouts, the algorithm only correctly predicted 61 winners (76.25%) – special recognition has to go to London who had the largest number of post-season prediction upsets at 3. I'm not too worried about this, since it reflects the rather long layoff between regular season play and the post-season tournaments. After all, with more than a month in between, a lot can change in the skills and strategies of teams going to the post-season tournaments.

Of course, wins and losses are only the simplest way of judging the algorithm. What's actually more critical is how close the algorithm came in predicting the score – or more accurately the DoS – of the bout. In the following plot, I compare the actual result of each of the 601 bouts with the respective prediction from the algorithm.

The x-axis shows the difference between the actual result and the prediction, from the perspective of the home team. A positive value means that the home team performed better than predicted and a negative value means that the home team performed worse. Of course a zero means that the prediction was exactly right. The height of each bar shows the number of bouts that had that result. The fact that most of the bouts are between -0.2 and +0.2 shows that the algorithm was doing a pretty good job over the season. Of course, the algorithm is not expected to be a perfect predictor. Some statistical variation is expected. The dashed line actually shows the same distribution from all the previous bouts that were used to train the algorithm – that would be all bouts from 2005 through 2010 Championships. Only the height of the curve was scaled to match the number of bouts in 2011. It fits the data quite well – for those who geek out on this sort of thing, a chi-squared test gives a 75% probability that these two are describing the same distribution.

So where do we go from here? I think the overall performance of the algorithm has been very robust during it's first year, so I don't think there's any reason to change the fundamental structure. However, reducing the ripples caused by the introduction of new teams will likely improve the overall performance. A very good suggestion was made by bjmacke earlier this year that greatly reduces this effect by using a team's first game to set their initial rating (and not changing the rating of the opponent). It turns out that re-optimizing the algorithm with this simple change not only improves the overall accuracy, but also greatly reduces some of the non-intuitive biases that we had previously found, such as the home/away bias in post-season tournaments. I'm currently finalizing the optimizations and I expect that we'll roll out the upgrade somewhere around the beginning of the year. When that happens, you'll see the ratings shift around a little bit, so don't get freaked out if your team's rating drops by 50 points....everybody else will get shifted too.