A football rating system, discussions, ideas

[mr onemore]

[mr onemore]

Re: A football rating system, discussions, ideas

n=20	prediction	number	correct	%	target
H	> 0.544	8006	4403	55.0%	50
D	0.1050 - 0.4078	4003	1197	29.9%	30
A		4003	1609	40.2%	30

Hi Muppet, How do you feel about these? Worth carrying on, or need more info before deciding? As this is just an exercise - how about splitting your results into deciles - or even smaller intervals- and plotting the graphs. Would be very interesting to see them. There may well be something there that is worth pursuing. Of course if you have other ways of testing how good the model is - I'd like to hear them. Good work there Muppet - welcome to my world of time consuming ratings calculations :D

[mr onemore]

Re: A football rating system, discussions, ideas

Mr O or Muppet77, Could i possibly check if i would be right in saying that up until March 20th Arsenal have a rating of 11.33?

Hi Odds - sorry, need a bit more to go on to help you there... Dino - that's a sound suggestion, and I'm sure it does work very well. All I'm going to say here is that I feel it's a case of fine-tuning. And I'm afraid I'm against that approach on general principles. Not because I think you're wrong - indeed, I truly believe you are right. The fact remains that if I adopt your suggestion into the model, it's going to have a negligible effect on the prediction values that the model churns out. How can I be so sure? - well if you play around with the weightings calculator I attached to a previous post - you will find that the contribution of the original rating to a new rating reduces quite dramatically after a few new games have been rated. After 19 games, or half a season, I seem to remember it has about a 1/3 influence. After a whole season, it's just about gone. That means you could start a rating at 1000, instead of the normal 10 I use here, and after a season it would have still found it's proper value with regards the rest. You would of course have caused an unnecessary inflation in the ratings - best avoided. Conclusion - any rating near enough where it should be, will find it's own level very quickly. I leave predictions involving newly rated teams for half a season - by then they really should have found their proper level, without causing the ratings to inflate with any significance in the mean-time. Now, Dino, with leagues like the NFL, a very different type of league to the one we're following here - it probably is very necessary to adopt your approach - and I'd say you're absloutely right to explore this issue.:D I know that if I were looking at the NFL for instance, I'd use the pure matrix form of the LS method, as a lot of the assumptions we made about the approximated solutions just wouldn't sit right with the format of that competition. There's also not enough games played in a season to allow the rating to 'find it's level'. An iterative process there probably is necessary to speed things through. When this thread is finished !! I'd like to take a look at other sports - they've all got their own little intracacies...maybe will take a look at NFL - as there is absolutely loads of LS literature about that;)

[mr onemore]

Re: A football rating system, discussions, ideas Rounding off chapter 4 then - and the verdict is the model doesn't look quite good enough. There's still hope for predicting draws though - so let's carry it through and go against the bookie!! Intro - Ratings Lists, Metrics 1. Generating a rating list - Least Squares solution 2. Generating a rating list - A simplified LS solution + another solution that approximates a LS solution method. 3. Creating data for backtesting 4. The rating list as a prediction model - How good is the model? 5. The model versus the Bookie!! 6. Improving the Model - Fine tuning, combining different metrics, starting again with a new metric (that's what I normally do!!) Chapter 5 - The model versus the Bookie!! They laughed at Columbus. They laughed at Fulton. They laughed at the Wright brothers. But they also laughed at Bozo the Clown. - Carl Sagan Well pretty much the last legs for this metric - and the verdict of truth. Would it have made money against the bookies? And can we use it with confidence from now on to make money? As I said in the last section, even by just looking at the graph, this metric just does not show enough. However, in the 6th and 7th deciles where we are expecting draws, there is a glimmer of hope. With 27% of all games drawn, our model is predicting 31.2%. That's certainly worth further investigation... So how do we do this part? Simple really, if you have all the data required. First crank out the ratings for 2000 to 2004 and tidy up the list as before. Then we select our predicted draws from the previous table - that's ratings lying between 0.00 and 0.39. Then we use the real bookies odds for these games, place virtual money on them, and keep a count of the returns. Well, here's what we get. :(

Aug 2000 to Feb 2004
Rated Matches	1161
Draws Selected	253
Draws	73
Strike Rate	29%
Bets	253.00
Returns	235.47
Profit	- 17.53
Yield	- 6.9%

Yes, that's right I'm afraid. Those are minus signs, and that's a negative profit situation - known in the trade as a LOSS.:( But the good news is - that was play money :ok Should we be so critical? Well, there were only 26% draws in this period, and our model scored 29%. It's just that Ladbrokes, whose odds I used, were setting their odds as usual so that a 31% strike rate was needed to break even. At an exchange, like Betfair, there might have been some profit here - perhaps something for the die-hards of this metric to cling onto - but you would have needed average odds of 3.45 - quite rare to be honest. For the rest of us - it really is time to ditch this, and come up with a better metric - believe me, there are much better than this one:ok What about looking at homes, aways etc? Well, that can be done too, obviously. But in this particular case, the graph produced earlier was not too inspiring, and I certainly demand better when real money is involved. Hang on - can't we fine tune this - boost that strike rate, and turn it all around? Well possibly, and we'll take a quick look at these things next. It will be just a quick look though, as this metric is at death's door...We'll just mention a few things that can be used when things are going a bit better to start with. Such high hopes........:cry

[muppet77]

Re: A football rating system, discussions, ideas

decile	range	number	H	D	A
1st	> 1.165	2005	65%	21%	14%
2nd	0.897 - 1.165	2000	54%	27%	19%
3rd	0.714 - 0.896	2002	51%	28%	22%
4th	0.554 - 0.713	1998	50%	28%	22%
5th	0.408 - 0.553	2008	48%	28%	24%
6th	0.264 - 0.407	2001	43%	30%	26%
7th	0.105 - 0.263	1998	42%	30%	28%
8th	-0.086 - 0.104	1997	39%	29%	32%
9th	-0.359 - -0.085	1997	35%	27%	37%
10th	< -0.359	2002	28%	29%	43%
ALL		20016	45.7%	27.8%	26.6%

 

Here’s a breakdown of the results by each decile. Suppose the results are not encouraging looking atthe % homes in the first decile and % aways in the last. Don’t think prices will show a profitwith these results.

 

A lot of work there, but i think i agree, maybe a new metric is needed.

[mr onemore]

Re: A football rating system, discussions, ideas

Here’s a breakdown of the results by each decile. Suppose the results are not encouraging looking atthe % homes in the first decile and % aways in the last. Don’t think prices will show a profitwith these results.

I tell you what Muppet - that looks a very encouraging set of results. The graph - below - shows how well behaved it is, no anomolies to speak off. Actually, before ditching this just yet - you've enough data there to look at percentiles. That graph would certainly show better prices as the tails/extremes of the graph. Might be worth taking this ratings list on and going up against the bookie in some special bands - doubt it will be draws - but might be some value at the ends - strong homes, strong aways. How about posting the table, and I'll post up the graph. Interesting this - I still don't think this metric is good enough, but perhaps I didn't pursue this enough for all the divisions!! Good work Muppet:ok

[merlin]

Re: A football rating system, discussions, ideas Great graph there Mr.O and not so bad number crunching by Muppet.Looks to be good scope for the first decile on the home predictions, nearing 70%, and again with the last decile on the aways topping some 40% correct.Promising figures there and not all doom and gloom.Well done to you both!;)

[muppet77]

Re: A football rating system, discussions, ideas cheers boys. got one more idea that i'm gonna crunch now. may just improve the figures. will get back to you in a bit with another table of deciles. mmmmmmm deciles.

[muppet77]

Re: A football rating system, discussions, ideas no - same results with different starting rating for each division. gonna attempt to look at yield if i were to place these bets. may be a while!

[muppet77]

Re: A football rating system, discussions, ideas ok. results. unfortunately i only had access to odds from the 2000/01 season onwards up to this season, so that narrowed the test down to 9836 games examined. using the same deciles as above, the top decile were predicted "homes" and the bottom "aways". i used odds from one bookmaker only (ie not best odds). homes, 1018 bets, -5% yield aways, 1036 bets, -9% yield all, 2054 bets, -7% yield not the greatest, even though better odds can be found elsewhere, i'm not into looking at 14 companies' prices, just 2 or 3 internet accounts. looks like we need a new metric.

[mr onemore]

Re: A football rating system, discussions, ideas Hi Muppet, I'm afraid I have to agree - but hope you're inspired enough to have a look at some other way of measuring team performance. My conclusion is that the goal difference metric is quite good - but only as good as the Bookie. When they add in the overround, it just doesn't have enough to overcome this and make a consistent profit. However, as you have all the data set-up now, you might like to have a go at one of the 'fine-tuning' aspects I'm going to go through soon. Maintaining seperate home and away weightings can sometimes improve the predictions a little. At the end of the day though - let's look for something better;)

[muppet77]

Re: A football rating system, discussions, ideas totally agree mate. that 12% over-round is always gonna be the biggest hurdle. looking forward to the next chapter. ;)

[merlin]

Re: A football rating system, discussions, ideas Lots of hard work there fellas, I have a sweat on just reading about it.Looks like some disappointing results there in the final summation, but I'm sure that you will get there in the end.Remember that song about that little old ant trying to move a rubber tree plant - well he's got high hopes, an old Sinatra movie song. Maybe something may unwind as we continue with the Chapters eh!;)

[muppet77]

Re: A football rating system, discussions, ideas does anyone have any idea how you can use the normal distribution to find out the HDA probabilities, given the expected difference in strengths between teams, the mean and the standard deviation ??? i am aware of NORMDIST function on excel, but is this the way you do it?

[mr onemore]

Re: A football rating system, discussions, ideas I believe this kind of thing can be done - but there are one or two important points I think you have to consider. Are you sure your ratings expected differences between teams are normally distributed? You also need to be sure that the actual goal differences in a match are normally distributed. If so, you can certainly try to compare your ratings distribution with the actual distribution. But again you would have to be careful as your ratings are derived from the actual distribution anyway. I don't know the answers to these by the way.:( I'm not a statistics expert by any means - so anyone else who is, please feel free to add as I'd like to find out too.;)

[muppet77]

Re: A football rating system, discussions, ideas i think they approximate to the normal distribution, the curve looks bell shaped (if that's the test?) any ideas?

[dietcoke]

Re: A football rating system, discussions, ideas Muppet, are you doing these calculations on the Ratings?

[mr onemore]

Re: A football rating system, discussions, ideas

i think they approximate to the normal distribution, the curve looks bell shaped (if that's the test?) any ideas?

Well, you can standardise your ratings, and then check to see that approximately 68% of your data set lies within +- 1 standard deviation of the mean. That gives a rough idea to start with. To standardise your ratings, there is a function on excel called surprisingly STANDARDIZE which uses the mean and standard deviation. Just order your list and see if there are about the right number in the range. If the shape also looks 'right' as you say then the normal distribution curve is a fair enough assumption to model the ratings data I think.

[mr onemore]

Re: A football rating system, discussions, ideas I think there's going to be problem finding probabilties for HDA using this approach though. You might have to settle for H, or not H.

[muppet77]

Re: A football rating system, discussions, ideas dietcoke - i am using the diff in ratings. 66% lie within one standard dev either side of the mean, so looks good. Mr O, what would i use STANDARDIZE for? not with you, sorry. :unsure

[mr onemore]

Re: A football rating system, discussions, ideas Well if you standardise the data, and you find it fits a normal distribution, then you can start to use statistical methods to analyse the data, and the related equations for the standard normal curve. If you do the same with the actual goal differences that occurred, you can compare the data sets and perhaps come up with a theoretical estimate for the probability of a team winning. I think you should then be able to match up a rating boundary with an actual goal difference boundary. I have to admit that I haven't tried these things to any real extent - for practical purposes I like to think the samples I use are usually large enough not to have to model the distribution using a fit to a normal curve. Of course one of the advantages of fitting to a normal curve is you're going to be able to look at values at the 'tails' of the distribution where there is not normally much past data to go on - ie the extreme cases. I tend to look at the middles of distributions where there is a lot of data - usually looking for draws, just a personal call - but also I'm not convinced that all metrics used are normally distributed in any case.

[dietcoke]

Re: A football rating system, discussions, ideas

dietcoke - i am using the diff in ratings. 66% lie within one standard dev either side of the mean, so looks good. Mr O, what would i use STANDARDIZE for? not with you, sorry. :unsure

I found that this particular rating generated a slightly peaky distribution You can see just how far from perfect by calculating the SKEW and KURTOSIS in excel.

[dietcoke]

Re: A football rating system, discussions, ideas By peaky, I found that more matches than expected were rated around the -0.5 - +0.5 mark.

[dietcoke]

Re: A football rating system, discussions, ideas

I tend to look at the middles of distributions where there is a lot of data - usually looking for draws' date=' just a personal call - but also I'm not convinced that [i']all metrics used are normally distributed in any case.

Got to agree with you there Mr O. Draws are great. As you know I've been using RFO data to generate a rating which which maxes on the games most likely to draw. THe beauty of draws is that, overall the odds of a draw do not vary too much so you know that by teasing out just a 35% chance of a draw you can make money. I also agree that not all metrics are normal, although those based on goals scrored proably are close enough. Take something like distance travelled by the away team which many people believe has an effect. I suspect that this has little or no effect until a certain milage is reached.

[mr onemore]

Re: A football rating system, discussions, ideas Yep - in general most metrics will not be normally distributed - at least I tread very carefully before I assume that a particular metric can be modelled in this way. An interesting point in case here is applications of ratings to chessplayers. There is a very famous rating list here called the ELO ratings list. The developer - Arpad Elo !! - assumed that the performances of individual chess players are normally distributed around a particular base level or rating for each player. He then came up with the system - several versions in fact - and one was adopted. It was later found that performances for a chess player are far from normally distributed, but are in fact skewed a great deal. Young players tend to be continually increasing in their base level, with older players (oh dear...) going the other way. In fact there are very few players who maintain a constant level of performance. Now, as far as sports teams go, I would say the base level perhaps cycles somewhat - up then down then up then down. Teams don't really have a cycle from young to old like individuals, so perhaps there is a better argument for a normalised distribution in some cases - depending on the metric being used of course. However, I think you're on really dodgy ground if you pursue any form of ratings analysis based on normal distributions for participants in individual sports - tennis, boxing...I would suggest other ways have to be found I think the way I presented here has it's merits - dividing the distribution up into deciles, percentiles...depending on the size of the data set, but having a fair number of ratings in each band - then zoning in on particular areas where you expect certain things to be happening. It might not be the most sophisticated method, but it's simple, and does avoid making assumptions about a distribution that could be just wrong. As I say, it does mean that you probably have to leave the extreme ratings values alone though which is a major disadvantage I know, but that seems to be the price that's paid with this method. Any statisticians out there?? Please post your knowledge here, would love to know the real deal here... How do you deal with distributions that are not normal, and in fact you don't know what kind of distribution it is at all? And is it really that dangerous to assume a normal distribution because it looks about right, if in fact it isn't a normal distribution at all?? It can certainly be disguised as in the example of the chess world, where roughly half are always increasing, half decreasing, so the population peformance ratings look normally distributed, when in fact at an individual level the performances are anything but normally distributed.

[przeszczepan]

Re: A football rating system, discussions, ideas there's this test... don't remember the name and have little time so I'll just post it the way I remember it... test for a normal distribution of data: calculate skewness and kurtosis and finally the statistic: 1/6*(S^2+1/4*(K-3)^2) if its

[przeszczepan]

Re: A football rating system, discussions, ideas As for potential problems with not normal distributions... In fact I believe it shouldn't be such a problem. Well, at least not in the method you've presented. To give you a quick example, the LS is a basic method for estimation of parameters of an econometric model and you do not make any assumptions as for the distribution of variables involved there (and that's what we are looking at here I guess). The only thing some people consider is assuming a normal distribution of residuals but this again has no influence on goodness of the estimation procedure itself but is just necessary to run virtually any statistical test on the estimated model or make interval predictions. btw, it's enough to test for the distribution of ratings. if it's normal the difference will also be

[mr onemore]

Re: A football rating system, discussions, ideas

there's this test... don't remember the name and have little time so I'll just post it the way I remember it... test for a normal distribution of data: calculate skewness and kurtosis and finally the statistic: 1/6*(S^2+1/4*(K-3)^2) if its I haven't seen this before, przeszczepan, this is going to be very useful. If you remember the name of the test please post it up - I'd like to look at this. Nice input too dietcoke - I didn't know about the Kurtosis stat - I'm going to have to get some stats books out!! Well, we can certainly test for certain now whether the goal-dofference metric and the ratings it produces are normally distributed. That's important as I've tended to steer clear of assuming normal dist. for reasons above. In the more general case, with other metrics, these tests might give confidence to move away from the empirical approach I gave in the thread(deciles, pecentiles), and to study the ratings more analytically. Plenty for me to think about!! - thanks guys - keep it up:D

[muppet77]

Re: A football rating system, discussions, ideas for my data: mean 0.405 sd 0.420 kurtosis 4.975 skewness 0.319 przeszczepan's normtest 0.180 looks quite 'normal' by those standards.

[przeszczepan]

Re: A football rating system, discussions, ideas it's Jarque-Bera test- the major drawback is that it' power (don't know if it's the right expression in English) is low. another point is the difference in which Kurtosis is calculated- some substract 3 at the end and some do not. that's why there's K-3 in the equation. If your software returns the Kurtosis with the 3 already substracted, input just K into the equation. Hope that's clear enough:)