Datapunter Posted June 11, 2009 Share Posted June 11, 2009 Here's a small Java program i made to filter out the Betfair Historic Data. If you've had a look at the original files you'll know most are well over 100Mb and this makes them very cumbersome to handle. This Java program can be used to read those original files and filter out only that data that you are specifically interested in. Resulting in a substantial reduction of file size making most data managable, although still big, into excel. Feel free to copy, modify and distribute. If you make a version that might be usefull for others please come back and add your version to the thread. Tomorrow i'll do a version with a MySQL database. Note: At the moment i do not have the time to explain how to program in Java nor how to use a MySQL database. This knowhow you'll have to get somewhere else. Don't let that stop you asking questions here. Plenty people on PL that master both and can provide answers to the less experienced members. Quote Link to comment Share on other sites More sharing options...
Datapunter Posted June 11, 2009 Author Share Posted June 11, 2009 Re: Betfair Historic Data, Java filter program(s) Version with simple file-to-file filter, suitable for any sport other than Horse Racing, contains a filter for Football, Correct Score markets.conv_959.zip Quote Link to comment Share on other sites More sharing options...
Datapunter Posted June 11, 2009 Author Share Posted June 11, 2009 Re: Betfair Historic Data, Java filter program(s) Same program, Horse Racing version.conv_960.zip Quote Link to comment Share on other sites More sharing options...
ScottyXS Posted June 11, 2009 Share Posted June 11, 2009 Re: Betfair Historic Data, Java filter program(s) Awesome stuff Eric. If you've already written one of these then sorry to repeat mate. -- Running Java Code Download Environment Before you can write or run Java programs (like the ones given by DP), you need to install the Java platform on your computer system. To just run Java code (like the .java code posted by DP), you need to download and install the JRE (Java Runtime Environment) software. Heres the latest version from Sun Microsystems. Once Java has been installed, you will be able to run Java program by running the java executable that was installed. You may find it convenient to add the location of this file to your operating system's path variable, otherwise you will need to explicitly refer to the absolute location of java. Editing System's Path Variable Windows 2000/XP users may set their path by right-clicking on 'My Computer' and selecting 'Properties'. Under the 'Advanced' tab, there is a button that allows you to set the 'Environment variables'. Click on this and alter the 'Path' variable so that it also contains the path to the Java executable. For example, if you have installed Java in c : \jdk and your path is currently set to C : \WINDOWS\SYSTEM32, then you would change your path to read C : \WINDOWS\SYSTEM32;c : \jdk\bin (without the spaces in : \, lol had to dodge the dreaded :\ smiley) When you open a new command prompt, it will reflect these changes and allow you to run java programs by typing "java". If you have installed the SDK, then you will also be able to run "javac" to compile stuff. Linux, UNIX, Solaris, FreeBSD users must set their PATH variable to point to where the java binaries have been installed. Please refer to your shell documentation if you have trouble doing this. For example, if you use bash as your shell, then you would add the following line to the end of your .bashrc: export PATH=/path/to/java: $PATH (but put no space between : $, lol had to dodge again the dreaded :$ smiley) Compiling Java Code Before the Java virtual machine (VM) can run a Java program, the program's Java source code must be compiled into byte-code using the javac compiler. To compile a Java source code file Foo.java, you would do the following: % javac -g Foo.java (This should be run in the command prompt (cmd.exe) or MSDOS) Javac is the compiler, -g just compiles it a with less hassle, and then the file name, please note that this should be run in the same folder the .java is held) If there are no errors in your source file, the Java compiler will produce one or more .class files (one .class file for each class defined in the Foo.java source file). For example, the results of a successful compile of Foo.java will produce a byte-code version of the class in a file named Foo.class. Running Java Code (Finally) Once you have successfully compiled your Java source code, you can invoke the Java VM to run your application's byte-code: i.e % java Foo (from Foo example) (again note that this must be run from where the class file is held) Quote Link to comment Share on other sites More sharing options...
beuher2000 Posted June 11, 2009 Share Posted June 11, 2009 Re: Betfair Historic Data, Java filter program(s) thanks a lot to both of you. trying to compile... Quote Link to comment Share on other sites More sharing options...
beuher2000 Posted June 11, 2009 Share Posted June 11, 2009 Re: Betfair Historic Data, Java filter program(s) Ok it works fine. Now i must find how to use the filtered file... :D maybe with the sql approach it would be easier to classify the data in an excel file like footballdata gives. great job anyway datapunter thanks again for help Quote Link to comment Share on other sites More sharing options...
beuher2000 Posted June 11, 2009 Share Posted June 11, 2009 Re: Betfair Historic Data, Java filter program(s) just a precision to compile java files you need to download the jdk here: http://java.sun.com/javase/downloads/index.jsp Quote Link to comment Share on other sites More sharing options...
KThom Posted June 11, 2009 Share Posted June 11, 2009 Re: Betfair Historic Data, Java filter program(s) Ok it works fine. Now i must find how to use the filtered file... :D maybe with the sql approach it would be easier to classify the data in an excel file like footballdata gives. great job anyway datapunter thanks again for help Hi Beuher, would you be able to zip and attache the filtered file? I maybe able to take it into work and have a play around with it. What format is it currently in? Quote Link to comment Share on other sites More sharing options...
ScottyXS Posted June 11, 2009 Share Posted June 11, 2009 Re: Betfair Historic Data, Java filter program(s) just a precision to compile java files you need to download the jdk here: http://java.sun.com/javase/downloads/index.jsp The JDK (Jave Development Kit) , the JDK is bundle of software that you can use to develop Java based software. The JRE (Jave Runtime Environment) the JRE is an implementation of the Java Virtual Machine which actually executes Java programs. All that is required to compile java code is JRE, JDK is for developing java software. Nonetheless JDK can be used to simply compile. Quote Link to comment Share on other sites More sharing options...
beuher2000 Posted June 11, 2009 Share Posted June 11, 2009 Re: Betfair Historic Data, Java filter program(s) The JDK (Jave Development Kit) , the JDK is bundle of software that you can use to develop Java based software. The JRE (Jave Runtime Environment) the JRE is an implementation of the Java Virtual Machine which actually executes Java programs. All that is required to compile java code is JRE, JDK is for developing java software. Nonetheless JDK can be used to simply compile. indeed i installed JRE but there wasnt the file javac so i installed jdk and it worked. no clue why though. Quote Link to comment Share on other sites More sharing options...
beuher2000 Posted June 11, 2009 Share Posted June 11, 2009 Re: Betfair Historic Data, Java filter program(s) Hi Beuher' date=' would you be able to zip and attache the filtered file? I maybe able to take it into work and have a play around with it. What format is it currently in?[/quote'] Problem is i have only one betfair data file that some friend gave to me. it s from may 2008. and i dunno if i can post this file on the forum... (cause of copyrights) Quote Link to comment Share on other sites More sharing options...
Datapunter Posted June 12, 2009 Author Share Posted June 12, 2009 Re: Betfair Historic Data, Java filter program(s) and i dunno if i can post this file on the forum... (cause of copyrights) Wouldn't post full files, copyright being one issue but disk-space on the PL server being the other. Don't think Betfair will mind however if some samples are posted in order to clarify the discussion in here. Quote Link to comment Share on other sites More sharing options...
Datapunter Posted June 12, 2009 Author Share Posted June 12, 2009 Re: Betfair Historic Data, Java filter program(s) Right here we go then, BetfairDataFilterToDB.java The .zip file contains the program as well as a MySQL dump file with the database structure. (also the JDBC connector i used to test, added for completeness ) The program will take an original Betfair data files and first extract only football data, then some more filters (up to you), then 3 markets being Match Odds, Over/Under 2.5 and Correct Score. The program fills 3 tables being: import_event: containing the match information and 3 ID's linking to the market data import_market: contains the market odds data import_selection: contains the names of the selections pretty simple so fairly self-explainatory. One tricky bit here is this: on a single line of data there is the name of the selection being a footballteam. You cannot determine however if this is the home or away team. For that you need to look at the full description where there is A v B. The problem is that Betfair in the full description has used abbreviations and not the full selection name. So in order to determine wether a selection is the home or away team we have to match the selection name to the abbreviation. And that is not always possible. Also if no bets where made on a selection, like with a tiny league, then there is no data, and without data you also cannot determine the home/away selection. About 90% is covered, depends on what you are after if you need to have a go at the other 10%.conv_962.zip Quote Link to comment Share on other sites More sharing options...
Datapunter Posted June 12, 2009 Author Share Posted June 12, 2009 Re: Betfair Historic Data, Java filter program(s) If you're going to use this to do some serious work i suggest you start a research thread of your own. Then we can talk and examine some situation more specific to what you are after. Might get confusing if we do that in this thread so let's keep this one more general. P.S. I'll do a horse racing version IF at least 3 people ask for it. Quote Link to comment Share on other sites More sharing options...
beuher2000 Posted June 12, 2009 Share Posted June 12, 2009 Re: Betfair Historic Data, Java filter program(s) Wouldn't post full files' date=' copyright being one issue but disk-space on the PL server being the other. Don't think Betfair will mind however if some samples are posted in order to clarify the discussion in here.[/quote'] ok then i uploaded the filtered file on web sharing site. If any problems with copyright or anything please delete the link. Anyway it s 'downloadable only 10 times. so here is the filtered file http://rapidshare.com/files/243719553/bfinf_other_080505to080511_080514085404FILTERED.csv.html as you can see it's betfair data for soccer from 5 may 2008 to 11 may 2008. not a big deal :( Quote Link to comment Share on other sites More sharing options...
beuher2000 Posted June 12, 2009 Share Posted June 12, 2009 Re: Betfair Historic Data, Java filter program(s) If you're going to use this to do some serious work i suggest you start a research thread of your own. Then we can talk and examine some situation more specific to what you are after. Might get confusing if we do that in this thread so let's keep this one more general. P.S. I'll do a horse racing version IF at least 3 people ask for it. Hi Erik, the thing is I find still really hard to use the filtered file in statistic purposes and for that maybe betfair data is not the best as you point out that some data is missing and the format is not friendly user (e.g. retrieving the home team...) Now i have to learn databases... :rollin Thanks again. Quote Link to comment Share on other sites More sharing options...
Datapunter Posted June 12, 2009 Author Share Posted June 12, 2009 Re: Betfair Historic Data, Java filter program(s) I had an additional program that read the data and then determined the price at which 50% of the total volume was matched. So caculate the total volume matched, run by all the odds and keep adding up the volume matched, until that total is more than 50% of the overall total, then i took that price a being the price i would most likely have been able to get had i wanted to bet on that match/selection. That's the kind of thing you need to think of. Plenty variations possible to that. - Some might need early odds so you'd need to add and look at the latest_matched and last_matched and compare that to the actual_off. - Some might be interested only in the price right before the off. - Some might want an in-running average. - Others might simply want a case by case archive. But it depends on the purpose of what you want the data for. Let's face it, what you are after is so specific you'll have to make an effort one way or the other. At least here you got a starting point. Quote Link to comment Share on other sites More sharing options...
muppet77 Posted June 14, 2009 Share Posted June 14, 2009 Re: Betfair Historic Data, Java filter program(s) wow this all looks very useful. good work datapunter. i now have less than the minimum points needed to access the historic BF data files. I am looking for BF odds for match corners: 12. any league, any season, but hopefully the EPL for last season as a starter. can anyone help me to extract this information. no problem if not, but its worth an ask. :ok Quote Link to comment Share on other sites More sharing options...
Datapunter Posted June 14, 2009 Author Share Posted June 14, 2009 Re: Betfair Historic Data, Java filter program(s) I'll see if i can get a sample of that data, then you can see if it's worth the effort. Quote Link to comment Share on other sites More sharing options...
muppet77 Posted June 14, 2009 Share Posted June 14, 2009 Re: Betfair Historic Data, Java filter program(s) cheers bud Quote Link to comment Share on other sites More sharing options...
nomadic Posted July 11, 2009 Share Posted July 11, 2009 Re: Betfair Historic Data, Java filter program(s) Tusen tack Erik. Applying these filters has been a huge time saver. ScottyXS - Thanks as well for the quick java tutorial. BTW, I also found I needed to download the JDK, as beuher2000 noted, in order to get the "javac" file. Quote Link to comment Share on other sites More sharing options...
nicc Posted January 9, 2012 Share Posted January 9, 2012 Re: Betfair Historic Data, Java filter program(s) I followed intructions but do not work!! In windows 7 changed environment variables patch in C:\WINDOWS\SYSTEM32;c:\jdk\bin and I downloaded all versions of java. How just to compile java code??? I wand to use file version BetfairDataFilterToDB Files saved on folder from desktop The file is not recognized in the command cmd!!!!! Any help please! Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.