Any Betfair data experts about ?

The problem is Betfair class keep bets as PE so they can be taken during the inplay period. I guess the best way is to determine the off time using the lowest IP time then use the last PE bet prior to that

noooooo..... not quite, getting the first IP time and then using the first PE prior to that, Occasionally returns a record a few hours prior to the off. much the same as using the DT_ACTUAL_OFF field.

using the first IP record itself produces under round markets.


bizarre why.... when the data files contain about 10mb ( yes thats 10,000,000    10......MIIIIIIILLLLLLLLION..... BYTES ) for each match,
They couldn't use 100 or so for such non entities at the SP.

Isn't it just a case of the smallest LAST_TAKEN column that is larger than the DT_ACTUAL_OFF column?

The dat by the way needs a mega amount of time spent on it for numerous reasons. It has to be meticulously cleansed as the markets are so tight if you leave it how it is then it can do more harm than good.

Looking at the data format some more, I think it was the introduction of 'keep bets' which caused the data to go a bit doo-lally.

If they changed from using the bet placed date to the bet matched date as the source of the date columns then it would all sort itself out again.

Doesn't look like it to me Feck,

because if that were the case then there would be zero 'PE' records with a LATEST_TAKEN after the kick off time.

i.e. the IN_PLAY field states that the bet was placed 'PE' pre-event, (prior to kick off), and the LATEST_TAKEN fields states the bet was matched 'IP' in-play.



So I'd speculate that the Bet-Placed-Date is used to derive the IN-PLAY field and the Bet-Matched-Date is used to derive the LATEST-TAKEN field.

I see what you mean now Escapee. All that's needed to resolve this is a list of eventid's and what time they were turned in play. Maybe not the sort of information they'd like to give out given their record.

I think the way the data's currently laid out means you'll never get an exact idea of the market at suspension just because a pre event bet at the correct starting price could also be last taken well into the game and wouldn't therefore show at the off time.

Looking at the data again the only way I could see of you getting some approximate odds would be to use the average FIRST_TAKEN IP odds at the DT ACTUAL_OFF time, that should hopefully be close to the actual start time odds as there would be keep bets in the market holding the prices and using the average should ensure both sides of the spread were included to eliminate the fact maybe just the lay side was taken first etc

just thought i'd ttt this to see if anyone else has any other methods

I've been experimenting with various criteria and approaches.
Using the PE record with the largest NUMBER_BETS seems to the best so far for creating a valid market.

I need to try it on more data files to confirm.

I don't see your problem. It is easy to work out the start time of the event. So, you then disregard all pre-event bets in the data that are matched after this time. Find the bet(s) that were matched closest to the off time. You could either use FIRST_TAKEN or LATEST_TAKEN to pick them, you'll still end up with something close to the right time.

This should get a 'good enough' measure of what was the SP at the time. To double-check, just ensure that the SP overround is close to 100%.

Aiming to be totally exact on the last traded odds seems a waste of effort. Just before the off, punters will have a choice of two odds to bet on for each runner/team (i.e. the back or the lay price) and in a stable, liquid market it is (mostly) random which one will have been traded last. So I wouldn't worry too much about exact precision here.

*Using that method I was getting markets with a range 97%-110%. with some as low as 89%

Mr Magoo
I can't see why it should have anything to do with keep-bets, since they are going to be matched against in-play bets and so it would be easy to mark them as such.


Going by the matches I looked at in detail, I reckon that records are marked as PE or IP according to the Bet-Placed-Date, not the Bet-Matched-Date.
I can't think of any other reason why there would be PE records with a latest taken timestamp dated after Kick off time.


I'd also hazard a guess that the whole data achiving process was designed/written before the 'Keep Bets' feature was introduced and therefore using either Bet-Placed or Bet-Matched date would have produced the same ( correct ) results and there wouldn't have been an issue.
But with the introduction of Keep Bets, it matters which date field is used to derive the IN-PLAY flag



Anyone know how to raise a bug/change request ticket with betfair ?

I guess you'd be better of collecting this data yourself for future events.

The Investor, if you know of an app/have an app which will collect data from markets simultaneously in one sport which you can set a market sample rate for that is stable and not affected in practice by the number of current markets being sampled/stored I would be interested in paying for it. Attempted to get this once but the developer Stefan let me down badly on here the sample rate was all over the place when the number of markets being monitored changed making it an expensive screensaver.

I reckon it'll cost you a fortune in data charges come Saturday afternoon, Investor.

It took us a while to figure out but here is the PHP code as is for getting the best possible overround. its not perfect, but from the tests we did it got us within +/- 5% from 100% overround with the most of them being +/- 1%. this isn't our import script - just a proof of concept... basically finding the smallest difference between LASTEST_TAKEN and KICKOFF timestamps.




$fp = fopen("bfinf_other_141027to141102_141105122445.csv", "rt" );

while ( $row = fgetcsv($fp, 2048) )
{
    if ( trim($row[3]) != "English Soccer/Barclays Premier League/Fixtures 01 November /Arsenal v Burnley" )
        continue;

    if ( trim($row[5] ) != "Match Odds" )
        continue;

    if ( trim($row[15]) == "IP" )
        continue;

    $kickoff_ts             = get_timestamp(rearrange_date(trim($row[4]).":00"));
    $bet_ts                 = get_timestamp(rearrange_date(trim($row[6].":00") ) );
    $latest_taken_ts        = get_timestamp(rearrange_date(trim($row[12].":00") ) );

    $difference = abs($latest_taken_ts - $kickoff_ts);
    $selection = trim($row[8]);

    if ( !$difference_selector[$selection]['difference'] )
    {
        $difference_selector[$selection]['difference'] = $difference;
        $difference_selector[$selection]['odds'] = trim($row[9]); 
    }
    else
    {
        if ( $difference < $difference_selector[$selection]['difference'] )
        {
            $difference_selector[$selection]['difference'] = $difference;
            $difference_selector[$selection]['odds'] = trim($row[9]);
        }
    }


}
print_r ( $difference_selector);



function rearrange_date ($date)
{
    $pieces = explode ( " ", $date );
    $date_pieces = explode ( "-", $pieces[0] );

    return $date_pieces[2]."-".$date_pieces[1]."-".$date_pieces[0] . " " . $pieces[1];


}



function get_timestamp($date)
{
    // If the date is a number, assume it's a timestamp
    if ($date == strval(intval($date))) {
        return $date;
    }

    $pi = explode(" ", $date);
    if (count($pi) == 2) {
        $date_p = explode("-", $pi[0]);
        $time_p = explode(":", $pi[1]);
        $timestamp = mktime(
            (int)$time_p[0],
            (int)$time_p[1],
            (int)$time_p[2],
            (int)$date_p[1],
            (int)$date_p[2],
            (int)$date_p[0]
        );
    } else {
        if (strlen($date) == 8) {
            $date_p = explode("/", $date);
            $date_p[2] = "20" . $date_p[2];
            $timestamp = gmmktime(0, 0, 0, (int)$date_p[1], (int)$date_p[0], (int)$date_p[2]);
        } else {
            $date_p = explode("-", $date);
            $timestamp = gmmktime(0, 0, 0, (int)$date_p[1], (int)$date_p[2], (int)$date_p[0]);
        }
    }

    return $timestamp;

}

Thanks for open-sourcing TJC.

If they changed from using the bet placed date to the bet matched date as the source of the date columns then it would all sort itself out again.

This is a pain all over (account statement, bet history etc.)

Who really gives a damn about placed date?
Matched date is so much more important.