Testing Robustness of a System

[wettspou]

Hi guys, I maybe registered to this forum since 2005, but I have been inactive until 1-2 months ago and after reading, reading, reading, reading and my first attempts in setting up a system I am wondering: How many papertesting does a new system need, before I can be sure it is profitable? Lets say I am looking for a system to give me a ROI of 10%. What tests do I have to use? Can I be sure after only a few hundred bets that my system is indeed profitable? Are there any Tutorials on this because I couldn't find any. Further down on this page is this thread http://www.punterslounge.com/forum/f21/question-statistical-significance-98762/ but I don't really understand it hehe. So any help would be appreciated :) Edit: Hmpf, I just realized I posted in the wrong forum... Please feel free to move my thread over the the betting help :) Thanks alot!

[leskane]

Re: Testing Robustness of a System It doesn't take much of paper testing once you have the system and database to test on. From my experience, which pretty much included testing every soccer rating system out there, you will need historical data of at least 2 seasons from 4-5 different leagues in order to find out whether the system is indeed profitable. Less data might lead you into belief that you have found the Holy Grail, when the general rule of thumb says that 'there are no miracles in this world'. :-)

[knobbo]

Re: Testing Robustness of a System Is there a 'back testing for dummies' guide anywhere? Thanks

[leskane]

Re: Testing Robustness of a System I am not aware of such a guide, however it should be pretty straight forward... Copy paste the seasons you are interested to work with into an excel sheet (including relevant decimal odds) and place stakes assuming that at the time of wagering score is not available, then calculate profit/loss based on final score. Naturally excel sheet is just an example, you might prefer working with Matlab or any other better suited software.

[Grex]

Re: Testing Robustness of a System

......How many papertesting does a new system need' date=' before I can be sure it is profitable?.....[/quote'] The simple answer to this question is.... no matter how much paper-testing (back-testing) you do, you can NEVER be sure your system is profitable. I have developed systems using data from 10 previous seasons... and believed I had a sure-fire winner, only for it to go horrible wrong during "live" testing. I'm not a statistics wizard, so I can't advise you on complex issues such as statistical variance etc. But I would imagine that if you develop a system based on results for (say) seasons 2000 to 2004, and then paper-test it on seasons 2005 to 2009, and get similar performance to within +/- 5%, then you can have a high degree of confidence in your system. But then, I'm probably talking bollocks :loon

[leskane]

Re: Testing Robustness of a System

I have developed systems using data from 10 previous seasons... and believed I had a sure-fire winner' date=' only for it to go horrible wrong during "live" testing.[/quote'] I would suggest to concentrate on recent seasons only. Bookmakers base their odds on statistical models, which improve over the years. So indeed, whatever was profitable in 2005 will not necessarily be profitable in 2009. However the difference between 2008 and 2009 is known to be minor.

[wettspou]

Re: Testing Robustness of a System I am livetesting a System that gives me a ROI of 14% so far (46 picks, lol). So I guess the true profitability could be anywhere betweent -20 and +25% though because of my small sample... I just need a hint like "try the xxxxxxx-Test" or "just try this and that method" and I will and may also post my findings. But I don't know where to start on this, yet.

[relf]

Re: Testing Robustness of a System Hi wettspou, there are many types of tests available to determine different things. I could try to help but "46 picks and ROI 14%" aren't enough information for me to work on to determine the true ROI. Here's why: Scenario A: The average odds is 2.50. Out of 46, you won 21 and lost 25. ROI: approx 14% Scenario B: The average odds is 7.50. Out of 46, you won 7 and lose 39. ROI: approx 14% These 2 scenarios give the same ROI, but the variance (risk) for Scenario B is higher than Scenario A due to the difference in strike rates. This means, the range for Scenario B is much wider than Scenario A, i.e. the fluctuations of the ROI for Scenario B is bigger than Scenario A given the same sample size, which simply means you can win a lot, or lose a lot in Scenario B as compared to Scenario A. Hope this sounds clear.

[Rushian]

Re: Testing Robustness of a System This has been discussed in different threads in the past. People tend to use a "Chi Squared Test" which is designed to compare the number of winners you would expect to hit by sheer luck against the number of actual winners in your sample. I personally disagree with this as it ignores potential differences in the odds of your winning bets and the odds of your losing bets in the sample. In my opinion, a more appropriate test to use would be to compare the yield achieved against a potential negative yield. This is equivalent to calculating the probability of making losses in the long run. For more discussion on the subject and some guidelines on how to carry out the calculations have a look at this thread. Good luck!

[wettspou]

Re: Testing Robustness of a System @ relf: Yeah - that perfectly makes perfectly sense. My statistical education is some years back though, so I don't longer know which tests to run on what kind of problem. But I think I still understand the statistics ABC - so factoring in the variance is definitely the key part. If I don't succeed with setting this up myself, I might ask you again in a few weeks if I may :) @Rushian I will have a look into the Thread you've posted. But my feeling tells me, thats exactly what I am looking for. Thanks alot for the hint, I don't know why I didn't see this thread.

[ofcourseitwas]

Re: Testing Robustness of a System as for Rushian's advice, I think the relevance of the chi square test depends on whether you are basing your system on bookies odds (expected odds and value) or a selection criteria (expected result regardless of odds)... am i wrong? no idea, im trying to figure this out too...

[relf]

Re: Testing Robustness of a System

@ relf: Yeah - that perfectly makes perfectly sense. My statistical education is some years back though, so I don't longer know which tests to run on what kind of problem. But I think I still understand the statistics ABC - so factoring in the variance is definitely the key part. If I don't succeed with setting this up myself, I might ask you again in a few weeks if I may :)

No problem! Meanwhile, you can give the system some time and get more picks, since any small sample sizes wouldn't give you a good result in testing. (No disrespect here, but 46 is slightly too small of a sample size.) For example, if I say my system bets on 5 selections at even odds and they won 5 of them - it might be superb but people will still be skeptical that this system remains profitable after 50 or 100 or even more selections.

[protop]

Re: Testing Robustness of a System How would you analyse the following for robustness.

Amount Prediction Strike Rate 123 34% 32% 100 37% 35% 230 40% 41%

These are based on a probability model and are not generated from odds. How accurate is the model. They can be horses, dogs, it doesn’t matter.

Thanks

[relf]

Re: Testing Robustness of a System

How would you analyse the following for robustness.

Amount Prediction Strike Rate 123 34% 32% 100 37% 35% 230 40% 41%

These are based on a probability model and are not generated from odds. How accurate is the model. They can be horses, dogs, it doesn’t matter.

Thanks

May I ask what do the 2 percentages means? I assume that strike rate is the amount of success out of the number of attempts/selections, but what does the prediction percentage mean? (Sorry for poor understanding.)

[wettspou]

Re: Testing Robustness of a System

No problem! Meanwhile' date=' you can give the system some time and get more picks, since any small sample sizes wouldn't give you a good result in testing. (No disrespect here, but 46 is slightly too small of a sample size.) For example, if I say my system bets on 5 selections at even odds and they won 5 of them - it might be superb but people will still be skeptical that this system remains profitable after 50 or 100 or even more selections.[/quote'] You're absolutely right. 46 picks are a joke when speaking of samplesizes, i know. Though I'm glad I figured out how to test the chances of achieving a certain yield by luck or by skill. But now I am de-illusioned ... Even with 1000 picks, a yield of 15%, average odds of 2,55 and a StandardError of 0,04 there would be still an 11% chance that my true yield was less than 10%, but I would be pretty safely (99.33%) within a true yield of 5%. So theres a long way to go - hopefully my yield stays where it is now :loon

[relf]

Re: Testing Robustness of a System Just thought I should add that statisticians often believe something works as long as it's at least 95%. I have no idea why the magic number 95 (and not 94 or 96) is used but that's probably the risk they are willing to take. After all, there has to be a balance between risk and profit - believing something works at 50% chance would put one at a high risk, while a constraint of 99.99% would probably eliminate all the systems on this planet.

[protop]

Re: Testing Robustness of a System Hi Relf The first percentage is the calculated probability of the event happening and the second percentage is how often it actually happened. So 123 times i predicted that a horse had a 34% chance of winning, however these horses won 39 times out of 123 bets. Giving a strike rate of 32%. And so for all the percentages. Does this make it any clearer and have you any further thoughts. Thanks

[relf]

Re: Testing Robustness of a System

How would you analyse the following for robustness.

Amount Prediction Strike Rate 123 34% 32% 100 37% 35% 230 40% 41%

These are based on a probability model and are not generated from odds. How accurate is the model. They can be horses, dogs, it doesn’t matter.

Thanks

Hmm, there are a few things which you can do with the data, and you asked "how accurate the model is". On first glance, I would say the models are accurate. One way is to check the margin of error, which resembles what people term as confidence interval. But I find the 'margin of error' less complicated and easier to explain. It's almost as accurate as the confidence interval as well. Basically, the margin of error of a % would be 1/sqrt(n), n being the sample size. The bounds are then, the actual sample strike rate plus/minus the margin of error. (More information available: http://en.wikipedia.org/wiki/Margin_of_error) Case 1: The sample size is 123, so the margin of error would be 1/sqrt(123) = 9.0%. Thus, you can be approx. 95% confident that the true strike rate is between 23.0% and 41.0%. Since the model gives you 34%, it's accurate. Case 2: The sample size is 100, so the margin of error would be 1/sqrt(100) = 10.0%. Thus, you can be approx. 95% confident that the true strike rate is between 25.0% and 45.0%. Since the model gives you 37%, it's accurate. Case 3: The sample size is 230, so the margin of error would be 1/sqrt(230) = 6.6%. Thus, you can be approx. 95% confident that the true strike rate is between 34.4% and 47.6%. Since the model gives you 40%, it's accurate. The bounds are pretty wide largely due to the small sample size you have. Hope I did everything right.

[protop]

Re: Testing Robustness of a System Hi Relf Thanks for the reply. This certainly seems one way of measuring it, the only problem is as you say if the strike rate falls between the bands then it is presumed accurate. I take it though that the nearer the middle of the band the more accurate it is? Therefore I assuming that 34% is more accurate than say 27% although both figures fall within the bands. Need to have a read up on the link you sent, thanks for your thoughts.

[relf]

Re: Testing Robustness of a System

I take it though that the nearer the middle of the band the more accurate it is? Therefore I assuming that 34% is more accurate than say 27% although both figures fall within the bands.

Actually, you are not quite right about that. As long as the % falls in between the bands, there's nothing to prove whether 34% or 27% is more accurate. After all, the sample strike rate of 32% is not fixed and can fluctuate with more results, which means after a few weeks, the 32% might be closer to 27% than 34% now. Of course, one might argue that the model that resembles closely to the sample strike rate is the better one, but the main point is the sample strike rate is not fixed. Of course, with a large number of sample size, the bounds will shrink and we will almost surely know whether the model is good or bad.

[protop]

Re: Testing Robustness of a System Hi Relf just a thought is it better to combine all of tha data and test it as a whole, therefore having more data to play with or is it better to keep the probabilities separate to see where the deficiency in the model may lie. at the extremes for example. Thanks for your input

[relf]

Re: Testing Robustness of a System

Hi Relf just a thought is it better to combine all of tha data and test it as a whole, therefore having more data to play with or is it better to keep the probabilities separate to see where the deficiency in the model may lie. at the extremes for example. Thanks for your input

If the data are of the same situations, it would be okay to combine them. If they are not, then I would strongly advise against combining them. Not only it will lose accuracy, there might be unexpected outcomes like the Simpson's Paradox, which you might not want. (http://en.wikipedia.org/wiki/Simpson%27s_paradox)