Forums

Forum Chat

There is currently 1 person viewing this thread.
Last Saturday saw the worst outages of the Betfair exchange in years. We are sorry for the disruption to you, our customers, over the weekend and are determined not to let it happen again. Our teams have remedied the problems that led to the outages on Saturday. We've been making big changes to our platform in preparation for loads of new product features that we'll be offering this spring. We're all focused on delivering the service levels you've come to expect – and therefore we’d like to give you the chance to put any questions you have to some key members of the Betfair team: Phil Dixon, who runs our development and infrastructure organisations, and Lee Cowles, who is the commercial manager of our UK business.

We will be hosting a Q&A session in the Forum Chat section of the Betfair Community from 6pm until 7pm this Thursday (10th February).

Please send any questions you have to livechat@betfair.com, and we will endeavour to answer as many as possible. You can send any questions you have now, and responses will be posted on the night.

Unfortunately it is not possible for us to respond to each email individually but we will attempt to answer all questions raised via the live Q&A session.
Share |

Post your reply

Text Format: Table: Smilies:
Forum does not support HTML
Insert Photo
Cancel
sort by:
Show
per page
Replies: 31
By:
Betfair Community Manager 2
When: 10 Feb 11 17:57
Welcome to this evening’s forum Q&A, which we have organised to deal largely with the recent outage problems.

Thanks to all those who have already submitted questions in advance of this session - we will start to post answers to those now. As we have had many questions on a similar theme, you may find your exact question is not answered although we have tried to cover all topics.

Customers who wish to ask further questions during this session can email their questions to livechat@betfair.com.

Among the Betfair representatives answering questions this evening are Phil Dixon, who runs our development and infrastructure organisations, and Lee Cowles, who is the commercial manager of our UK business.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:03
With regards to Saturday’s fiasco, why are Betfair turning to more ‘Mickey Mouse’ type arcade games, at the detriment of the overall service?
Betfair offers a broad portfolio of products, including arcade, because we have seen demand for them and they are used by a significant number of our customers – whilst they are not everyone’s cup of tea, we have never compromised the delivery of our exchange service to accommodate them.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:03
When the outage occurred on Saturday, your splash screen said this was 'planned maintenance'. It obviously was not, so why put this screen up? Are you not at all prepared for 'unforeseen problems'? Then, following resumption, there was no message on your Service Problem regarding the outage for 3 hours. Do you not think that you could do a better job of keeping customers informed when there are problems with the site?
This is very fair feedback. Posting a 'planned maintenance' screen was obviously a mistake in our process. Clearly the outage was unexpected and this failure in our processes has been addressed internally. Generally speaking, we are determined to do a better job at keeping our customers informed in the future.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:05
Please could you explain the reasoning behind informing your customers of today’s planned interruption to the site in such a poor manner? I had no idea it was to take place, despite having been logged on for over three hours.  I don't regularly use the forum (although the messages posted on here were next to useless considering how late they were posted) and don't click on the home page. Surely the following should take place:

1) An acceptable period of warning. The maximum I have found on the forum is 9 minutes. Some of your warnings on there are less than 5 minutes.
2) Not relying on the forum for such notices
3) Emailing your customers about planned disruptions.
4) Ideally a message flash-up which requires confirmation it's been read (or at least the ability to have this option if desired).
Thanks so much for your question – the work we’ve carrying out to look into the recent outages has been going on since the weekend. The investigation has been evolving and led to a point today when the team decided to run some specific tests – it’s something that we decided to do only this morning so we communicated as quickly as possible in the circumstances. We appreciate that it didn’t give customers a lot of notice but we deliberately tried to find a window that would cause minimum disruption.  We take all your points on board about how we can communicate quickly and directly with customers – it’s great to hear your ideas and they are the sorts of things we’re looking into. Sorry again for the inconvenience to you today but thanks for sharing your thoughts with us.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:07
While I appreciate that it is not always possible to test for every eventuality, particularly simulating full production load, your performance following system upgrades over the years would seem to indicate you haven’t got this under control.  What steps are you taking to sort out your upgrade and testing approach?
We are running several threads to address our systems’ performance and stability. Last Saturday’s problems resulted from a major infrastructure upgrade, which we carried out ahead of the release of several upgrades and some great new product ideas. We are really sorry for the problems the recent instability has caused for some customers and we are working really hard to make sure that we can ensure all our customers can enjoy Betfair with confidence.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:09
Why is it that the longer you are logged in to Betfair the more memory gets sucked up by the iexplore process that it runs in ? By the end of the day I’m seeing more than 500Mb of memory being allocated to Betfair - the only solution is to logout then clear the cache and log back in again which is a royal pain.
Thanks for telling us about the problem. We were unaware of this issue and have passed your description to our team for immediate review.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:10
For all those watching this chat unfold, please feel free to open a thread in General Betting to discuss the answers etc. We'll monitor it.

We are looking at different and better ways to manage these sessions in future, by the way.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:11
In the event of a market crash and the inability to reopen the market prior to the off, it is my opinion that all bets should be voided. This is the only fair thing for all market players regardless of type of player they are. I do not believe Betfair wins any support by being seen to not accommodate the betting styles of all players - traders, gamblers etc.
Thanks for raising your concerns. However, we believe that any bets which have been matched prior to any site instability have been done so fairly and we do not believe it to be the case that customers (as a whole) would want all matched bets to be voided in such circumstances. As an example, consider a customer placing a matched bet in the morning, watching the relevant sporting event in the belief that the bet was a winning bet, yet later finding out that the bet was subsequently voided because of website issues, some hours after the bet was matched.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:12
You say the upgrade is about making way for "loads of new product features". Do you not realise that most of your regular customers (on whom your revenue and profits depend) do not want "loads of new product features" - they simply want a reliable, and well-performing site that they can trust. Bells and whistles don't butter parsnips.
You are absolutely right – the most important thing we can provide is a reliable and well-performing service and we are sorry that we failed to do so in the last couple of weeks. That said, whilst many of our customers love Betfair just the way it is, there are also large numbers of customers who find the existing site confusing and would like to see it more simply presented and easier to use. As Betfair has grown, we have put a huge amount of effort into stability of site performance. This has meant that we have not invested as much in the front end as we would have liked to. Much of our recent work has been to more cleanly separate the front end (presentation) from the back-end (processing bets and moving money). Now that is mostly completed, we can start to make changes as to how we present the site, without big risks to stability.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:14
In the event of a market crash and the inability to reopen the market prior to the off, it is my opinion that all bets should be voided. This is the only fair thing for all market players regardless of type of player they are. I do not believe Betfair wins any support by being seen to not accomodate the betting styles of all players - traders, gamblers etc.
If we have site issues, there is rarely a response that would be seen as fair by all customers – any bets which have been struck prior to any site instability have been done so fairly and voiding would be seen as unfair by customers with winning positions. The right answer is to make the site stable but there will always be risks. Over the years, we have had to respond to many unplanned situations and have pre-planned responses to many – we have never published clear guidelines on how we deal with outages and we will be looking at our procedures to see if we can make it clearer to interested customers how we will deal with outages. That said, we will never have rules that deal with every eventuality and will need to continue to try and make the best decisions for our customers depending on the specific circumstances at the time of any outage.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:16
My main concern over the recent site issues regard a possible contingency plan for dealing with matched bets. As a trader of the 'match-odds' markets on football I often have a position open with the intention of trading out prior to kick-off. As with a lot of Betfair customers I see myself as a trader and not a gambler so the thought of a match kicking off with such a position 'open' is a nightmare scenario.  Since Saturday’s problems I haven’t had the confidence to trade large sums for this reason. So, I would like to know:
a) is this something that I still need to be worried about? Could the problems happen again?
b) If it does happen is there anyway way to close / trade out of an open position? (even if this means taking a loss).
We are confident the problems that caused last Saturday's outages have been addressed. In the long-term the only way for us to minimise these types of issues is to continue to invest heavily in site stability and make sure that it doesn’t go down.  There are a number of specialist applications (see http://solutions.betfair.com) that can deal with some of the scenarios you mention but if the core of the site (bet matching) isn’t working then no-one, irrespective of channel or application, can trade.  We try and minimise risk but occasionally we need to make big changes to our systems in one go, as we did 2 weeks ago.  That is now complete and we are not planning changes on a similar scale for the foreseeable future.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:16
Is it not possible, when you place a bet , to specify that, in the event of an outage, you want that bet to stand or not? Either that or you need to cancel every bet placed on the events affected.
It’s theoretically possible but very complex and we believe our customers would benefit more from increased investment in stability.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:17
When are you going to change your rule in General 1.2 ‘All markets other than soccer markets and Australian markets - not suspending at the time of the 'off'’ which basically amounts to theft?   If a market is scheduled to be turned in-play but Betfair does not suspend the market at the time of the 'off' (so unmatched bets are not cancelled at that time), but the market is intentionally turned in-play at a later time during the event, all bets matched after the time of the 'off' will stand. Nobody should be penalised for having their pre match bets hoovered up if a match situation changes after the start of a contest.
The rule you refer to is one which has been drafted to cover the rare occurrences where the start of an event is missed. In such situations, a very high percentage of pre-off bets which are matched in-play, are matched at fair prices. If a material price-impacting event has occurred after the off in these circumstances, then our general policy would be not to turn the market in-play. Having had the rule in place for a number of years, feedback from customers indicates it’s striking the right balance. We would, however, be happy to review rules should that feedback change.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:18
Can you tell us more about the new product features that will be introduced in spring? If you can't go into specifics, can you tell us whether they will be related to the sports exchange (football in particular) or other areas of Betfair?
We can’t really go into specifics, but the new product features will cover a wide area, including new sports pages, in-play pages, mobile, and new games. There is a brief description of some of the in-play changes at https://promotions.betfair.com/in-play-uk.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:19
I've been dismayed to see more and more bloat added to the homepage of late. It's frustrating to wait 30 seconds plus for it to load, especially when a lot of that's down to plugs for the arcade, casino, poker etc, when all I'm interested in is the sports exchange. Will Betfair consider offering simplified access to sports exchange that bypasses this?
We acknowledge the page load performance of the sports home page and other parts of our website are unacceptable for our customers and have built a team with the sole objective of addressing these performance issues. You will first see these performance releases from around April and they will continue through the summer. The very first page we intend to relaunch is the sports home page. Further, for especially fast, direct access to the exchange, you may be interested in exploring the applications available in the Betfair app directory located at http://solutions.betfair.com
By:
Betfair Community Manager 2
When: 10 Feb 11 18:21
Betfair pages load much quicker on Chrome than Firefox, and I've seen Betfair suggest using it in the past. However, "what if?" figures on Chrome are completely spurious, which makes it unusable. Will Betfair consider addressing this problem, for those of us who would like to switch but can't till it's fixed?
Many thanks for making us aware of this. We will investigate this issue and schedule a fix as soon as possible.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:22
The Betfair lab rats created ‘watchlist’, and it is one of the best features to watch odds on a market live without refreshing or getting extra applications.  Sometimes it stops working after an upgrade and they have to do an emergency fix. Will you be supporting this feature through the coming upgrades?
Yes - however, as this is still a Betfair R&D product, support may have some lag. We'll make sure to notify the Labs team that you like their feature. Thanks for the feedback.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:22
I think it would be a good idea to be able to set a cable from my computer to my televison so that I can watch the live video stream that you provide, on sports that I bet on, whilst leaving the full screen of my computer available to access the Betfair interface or api bot that I use. This question is mainly pointed at your horse racing stream. What’s the chances of this?
Our contract with streaming providers restricts the size and usage of the pictures we stream, so we can only offer it in the existing format and up to a maximum bandwidth. It may be possible to view if you have an internet enabled TV or effectively set your TV up as a second monitor with the pop-out streaming viewer on the second screen but you wouldn’t be able to optimise for the full screen size.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:26
With your impeding launch of your new in-play campaign, do you plan to extend your horseracing coverage on Live Video to show the races that are covered on Racing UK?
Unfortunately, there are no immediate plans to reintroduce Racing UK streaming. However we are always reviewing our streaming content and hope to bring customers exciting new content where we can.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:27
As an in-running punter I find it annoying to see horses get matched at 1.01 and the other horse not to go below 1.5 in a very close finish when the only time the result is known is when the horse’s nose is on the line and Betfair have suspended the market late allowing on course punters an even bigger advantage. Has Betfair any plans to use technology on course to suspend the market? An example would be to have the suspend sign connected electronically to the starters button, and then to have a laser on the winning line like they use in athletics to suspend the market on the line. This would not allow the above example of people betting after the line.
Thanks for your suggestions. This is something we have discussed previously and will continue to investigate. Implementing such a solution however, is dependent upon a number of external factors outside of Betfair’s control. Timely management of markets and in this case, the suspension of horse races, is a key priority for our operational teams and we are currently using the best technology available in order to satisfy this.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:30
Can you confirm Saturday’s outage had nothing to do with an 'attack' by an outside agent?
Yes, our security teams have investigated thoroughly and we can definitely confirm that wasn't the case.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:32
Why on earth don't you have a back-up in place so that when the main server goes down you can switch to the back-up?
We have a number of back-up options in case of server issues but in the case of the outages on Saturday, we know they weren't related to server problems.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:36
Can you confirm if you know the specific reason why the site crashed multiple times last Saturday?
We have identified the issue and we've been working all week to make sure our solution is robust.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:36
Is the risk posed by site instability (and the effect instability has on your customers) due to upgrades always quantitatively justified by the benefits the upgrade brings?
We spend a lot of time planning upgrades and investment in the site and our products Our aim is to do this with the minimum disruption to, and maximum benefit for, our customers. Sometimes we don’t get the balance quite right, and for that we apologise.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:39
Will whatever you are intending to do in terms of new features shorten the delay between real live and what you call 'live', compared to what the delay is now?
We try and make our site as quick as possible, especially where prices and transaction processing are concerned. There will always be some latency or delay in the system and much of it is outside our control, for example the delay between what you call “real live” and TV pictures. We use the fastest sources we can find to manage our markets and can do nothing about delays in them. So whilst we will be working to speed up all of the pages on our site, it will have a limited impact on the delay you mention.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:42
The server is going down while you carry out "tests". This morning's lasted over 10 minutes (categorically NOT "1 or 2 minutes”). Why are you carrying out tests on the live system? Why don't you have a test system?
Sometimes you can only learn so much from a test environment.  Due to the nature of last Saturday's outages, we want to make sure that we have explored every explanation for the outage and that involved the test we ran today.  Sorry, the test did take longer than we anticipated but we learned a lot from it. We are planning some more tests tomorrow (the first is planned for 12.30 GMT). We have deliberately chosen times with the aim of causing minimal disruption to our customers whilst still allowing to us test under realistic conditions.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:46
There have been numerous occasions where I've rung Customer Service up to tell them Betfair has been down. They've had no idea.  So, they check it and say "erm ... yeah, you're right".  Isn't there an "alarm" that goes off when the site goes down?
We do have internal alerts but you’re right, over the years it’s often been a customer who tells customer services rather than the other way around. Over the years we’ve hugely improved our site monitoring but we still have a delay between our site support team seeing an issue and informing the Customer Service team. We’ll work to improve this but of course the main aim is to stabilise the site.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:50
"While I appreciate that it is not always possible to test for every eventuality, particularly simulating full production load, your performance following system upgrades over the years would seem to indicate you haven’t got this under control.  What steps are you taking to sort out your upgrade and testing approach?"

However, the answer doesn't address the question. I'm asking about upgrade and testing approach - that is the method used. Saying that you are running several threads to address systems performance and stability implies you are dealing with the issues that upgrades create, rather than preventing them in the first place.
We're upgrading the whole platform and have committed to invest heavily in it - we're part way through a project to create a site for the 21st century. Part of that investment is in our upgrade and testing approach – this is something that we take really seriously and have brought in a lot of experience and resource to help.
By:
Betfair Community Manager 2
When: 10 Feb 11 18:57
Why when the site is malfunctioning do you not restrict access to everyone? It seems grossly unfair that customer A can't login to cancel their bets while customer B, using a betting program via the API, is still allowed to use the site and take their bets.
As soon as we are aware of a significant issue, we do take down all parts of the site for exactly the reason you mention.
By:
Betfair Community Manager 2
When: 10 Feb 11 19:02
What was the cause of Saturday’s crash?
On 31 January we moved to a new datacentre and used that opportunity to significantly upgrade our infrastructure. Saturday's outage was the result of a misconfiguration that became apparent during a period of high activity. We fixed that issue and validated the fix during today's tests. During our test today, we spotted another issue that we have also addressed. We plan to double check this fix with additional tests tomorrow.
By:
Betfair Community Manager 2
When: 10 Feb 11 19:11
Okay, we're drawing the session to a close - thanks for taking part.

This sort of feedback is incredibly helpful to us as we are always working hard to improve things for our customers. We really appreciate you taking the time to join us this evening.
sort by:
Show
per page

Post your reply

Text Format: Table: Smilies:
Forum does not support HTML
Insert Photo
Cancel
‹ back to topics
www.betfair.com

New to Betfair?

You need to open an account before you can add content to the forum.

Opening an account only takes a few minutes.

register now