Forums

Forum Chat

There is currently 1 person viewing this thread.
Betfair Community Manager 2
14 Mar 11 13:09
Joined:
Date Joined: 22 Nov 10
| Topic/replies: 3,062 | Blogger: Betfair Community Manager 2's blog
Following on from Saturday’s site issues, Tony McAlister (Chief Technology Officer) and Niall Wass (Chief Marketing & Development Officer) will be hosting a live Q&A session from 6pm until 7pm GMT tonight in Forum Chat.

Please e-mail your questions to LiveChat@betfair.com.

You can send any questions you have now, and responses will be posted tonight. Unfortunately it is not possible for us to respond to each email individually, but we will attempt to answer all questions raised via the live Q&A session.
Share |

Post your reply

Text Format: Table: Smilies:
Forum does not support HTML
Insert Photo
Cancel
Page 1 of 2  •  Previous 1 | 2 | Next
sort by:
Show
per page
Replies: 43
By:
Betfair Community Manager 2
When: 14 Mar 11 18:47
Welcome to this evening’s forum Q&A, which we have organised to deal largely with the outage problems at the weekend. We will be starting at 6pm.

Thanks to all those who have already submitted questions in advance of this session - we will start to post answers to those now. As we have had many questions on a similar theme, you may find your exact question is not answered although we have tried to cover all topics.

Customers who wish to ask further questions during this session can email their questions to livechat@betfair.com.

Among the Betfair representatives answering questions this evening are Tony McAlister (Chief Technology Officer) and Niall Wass (Chief Marketing & Development Officer).
By:
Betfair Community Manager 2
When: 14 Mar 11 18:53
A thread has been opened in General Betting for discussing this Q&A session. It will be monitored by staff members.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:00
How does Betfair intend to Compensate Customers for the recent Unplanned Outages?
Betfair has listened to feedback from customers and we will be offering a 100% commission refund on a selection of this week's top markets across the Cheltenham Festival, the Champions League, the RBS Six Nations, the ICC Cricket World Cup and Barclays Premier League. The way this will work is that customers will place their bets as normal on any of the listed markets and then all commission paid on winning bets will be automatically credited back into their account within 24 hours of the market closing. You can read the full details at https://promotions.betfair.com/refund-offer
By:
Betfair Community Manager 2
When: 14 Mar 11 19:01
After all the instability and poor site performance - why should I continue to bet with you?
We are confident that we have taken every possible measure to fix the problems we had on Saturday and prevent them from happening again. We have also updated our operational processes and introduced a whole host of new monitoring to spot problematic issues on the Exchange. We have also imposed a change freeze for the next 7 days so we can maximise the stability of the site. We can assure you that it is in no one’s interests for there to be site outages and we will do everything within our power to avoid future incidents.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:03
Do you think, especially after the last Live Chat, you have satisfactorily addessed poor customer communications during operational failures?
We don’t think that Saturday was completely perfect by any stretch of the imagination. But during the last live chat, we did make a commitment to improving our customer communications and we have implemented a number or processes to support this. In general I think we improved in this area, however part of the problem for us is that at times (and Saturday is a good example) where we do not immediately know the root cause, we are restricted in what information we can pass to customers. We are looking at ways to address this issue. However, once we were sure what the issue was we communicated it in as honest and transparent a way possible.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:04
Why aren’t all channels taken down during an outage to prevent bets from being unfairly matched (i.e. API, Mobile, etc)?
As soon as we become aware of a significant issue, it is our policy to disable all betting channels to ensure a level playing field for customers.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:05
Why don’t you void bets so that everyone knows where they stand?
In circumstances where the website is unavailable due to an outage, there are some customers that feel that affected markets should be voided in their entirety. However, our rules do not allow for this and had we voided affected markets on Saturday we would have been deluged with complaints from those customers who had 'missed out' on winning bets. If we have site issues, there is rarely a response that would be seen as fair by all customers. However, any bets which have been struck prior to any site instability have been done so fairly and we do not believe that customers would want all matched bets to be voided in such circumstances.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:06
Why aren’t unmatched bets cancelled from the point the site goes down?
Our current procedure in any unplanned outage (regardless of length) is to suspend and cancel unmatched bets on all In-Play markets, all Financial Markets (Mon-Fri), and all Horse Racing and Greyhound Racing markets within 10 minutes of the 'off' time (this is an approximate timescale and is largely dependant on the opening show time).  We may also cancel bets on longer term markets if we feel they are materially affected e.g. A Tennis Tournament Winner market if a semi-final match was In-Play at the time of the outage. We always post a list of affected markets and any action taken as soon as possible, in the markets section of the forum following an unplanned outage.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:07
Do you think you have the right, mature IT management processes in place (ITIL for example), to ensure solid operations/service delivery?
We are confident that we have the right people, process and technology to ensure both reliable service delivery and change.  Our stability issues over the past weeks have shown that we have had our mix wrong in terms of the type and velocity of change we've put in front of our customers.  We are first freezing and then slowing our rate of change as we improve our quality and change processes to deliver both uptime and new product features for our customers.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:08
Do you think you have got the management balance right between implementing new functionality and "Keeping The Lights On", i.e growing the business by introducing new and enhanced products versus maintaining the existing products and protecting your current customer base?
From our customers' perspective, the balance clearly isn't quite right given recent stability concerns.  As such, we are first freezing and then slowing our new product release volume until we are confident that we have struck the right balance to enable us to both release new products and maintain the very high levels of service our customers rightly expect.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:11
A number of site problems seem to relate to ‘load’ issues. However you have done a number of upgrades in the past that were designed to address this problem. How come you are still encountering issues whenever there is excessive load on the site?
Our recent site problems have been related specifically to our web site, not our core betting exchange.  These problems were triggered by specific production load scenarios around new product features.  It turns out that we hadn't successfully replicated certain key production load scenarios before the products went live.  We've been continuously adjusting our quality process and we are confident that we will get this right.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:12
Can I ask how capacity was increased? To add three fold capacity in hours is quite an achievement.
Betfair has plenty of spare capacity in our production datacentres.  We simply allocated some of that spare capcity to our web server farm.  We still have plenty of remaining spare capacity.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:13
Please can you tell me how many test environments there are at Betfair and what methodology is used in terms of stress & contention testing to migrate system upgrades?
We have a huge range of test environments including development, performance/stress and staging. Even though our Quality Assurance team of over 100 people undertakes extensive testing, it is very difficult to replicate the actual live behaviour seen on the site.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:15
Could we not have a standalone site for horse racing? Surely you have to deliver consistency and continuity for racing punters?
All of our customers are of equal importance to us and we have a duty to deliver consistency and continuity to everybody, no matter what they are betting on. Our markets are separate and the interaction at a website level is also separate. Markets share physical infrastructure in order to scale more efficiently. All of that said, the issue on Saturday was not a problem with either the Exchange or the markets.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:16
Quite a lot of these problems occur after a release/upgrade. Why don’t you do more comprehensive testing to find any bugs before the site goes live?
The primary focus of our quality improvement is to get better at replicating true production scenarios in our testing. We invest significantly in quality but have realised that our testing scenarios and conditions aren't yet realistic enough. We are actively working on rapidly improving our ability to replicate production load scenarios in our testing environments.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:18
On Saturday, in response to your 6 Nations bet offer, I placed a £100 treble on the 6 Nations games. Italy duly obliged in the first game, placing me in a position where I could trade into a certain profit on the other games. I was unable to do this when I tried to access Betfair during the Wales-Ireland match. I would not have placed this bet had it not been for the Betfair offer. Will you void and therefore refund all losing bets for this offer?
We agree that this is a poor customer experience  and this weekend's performance has not been acceptable. We've received a lot of suggestions as to what to do. It is impossible to address everyone's concerns individually and even if we voided all bets then we let down customers who were in a winning position. On balance we decided we wanted to both apologise, give confidence of site stability and make a gesture to our customers to make up for experiences such as yours. So we have decided for the first time to offer a series of markets with 100% commission refund this week across racing, football and also on the sport you were betting on, rugby, next weekend. We hope this starts to restore your faith in us.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:19
Betfair started as a betting exchange, an alternative to traditional bookmakers. It has now become a website offering arcade, games, download & instant play casino's - all of which seem to take precedence over the one true unique selling point Betfair has. With so many weekly updates contributing to the downtime that could well have cost people thousands of pounds, would you concede that Betfair have gone too far in that direction?
We would agree that there has been too much downtime and we apologise for that. We are committed to radically improving the stability. We hope that we can still provide products that all customers want as many of these products you refer to are products that our customers told us they were playing elsewhere. The downtime was not linked to these products but if we can't improve and having too many products is the cause of this we'll be the first to look at it again.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:21
Was it caused by sabotage or a DDos attack?
No. We know that for certain.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:22
The last time the system crashed on a Saturday afternoon, you did some testing on the following Thursday. This involved stopping the live system, which caused API bots trading systems to crash. Will you be doing this again?
The last time we did this we were trying to check whether the fix we had in place would be completely effective under live conditions. As this particular problem is a completely different issue that we know we have fixed, we have no immediate plans to do this again.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:23
Are all the extras (arcade, poker, games etc) slowing the site down and increasing the chances of outages?
No. The performance of Betfair's Exchange and related core betting systems are not affected by any of our portfolio products.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:27
Prior to the last outage, were the latest fixes applied to a sandbox before the live system? If they were can you explain why the site issues continued?Saturday's outage was triggered by specific production load scenarios around new product features.  It turns out that we hadn't successfully replicated certain key production load scenarios before the products went live.  We've been continuously adjusting our quality process and we will get this right.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:30
Is it not the case that by recognising that you need to suspend all releases during the busiest week of the year in order to attempt to guarantee improved stability, you are making an implicit admission of the risk you run with your existing strategy?
To some extent you are right. Our freeze is about creating the greatest possible confidence for our customers in the stability of the site.  We intend to keep adding great new products, but right now our priority is reliability and customer confidence.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:31
Would you agree that the bug that caused Saturday's problems should never have reached the production system?  If so, were you aware that corners were being cut in Software Quality Assurance and what steps do you plan on undertaking to improve matters?
With the benefit of hindsight, of course we wish the bug hadn't made it through. Our focus now is to learn the lessons and ensure these scenarios don't happen again.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:32
Given the weekend's issues were all of your making - should you not waive this week's premium charge?
We all agree that the customer experience this week-end has not been acceptable. As above we've received a lot of suggestions as to what to do including specific refunds, voiding markets and so forth. On balance we decided we wanted to both say 'sorry', give the best chance of 100% site stability e.g. by stopping all releases this week, and then finally make a gesture to our customers. We know that might not fix everyone's issues from Saturday but we felt it was the right start.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:34
Why after restoring the system on Saturday night did it take so long to make an announcement as to what had happened?
We wanted to give you an honest appraisal of the situation and in order to do this we needed to make absolutely sure that we had our facts right.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:36
If a bet was placed before the breakdown with the intention of being traded in running will you refund them if they ended up being losing bets?
As explained above, if we have site issues, there is rarely a response that would be seen as fair by all customers.  Any bets which have been struck prior to any site instability have been done so fairly and voiding would be seen as unfair by customers with winning positions.  For example, consider a customer placing a matched bet in the morning, watching the relevant sporting event in the belief that the bet was a winning bet, yet later finding out that the bet was subsequently voided because of website issues, some hours after the bet was matched.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:40
Yesterday,you gave 1 minutes notice via your service page you were going to suspend the placing of bets for 5 minutes. I'm sure I speak for most people that we don't constantly refresh the service page and to give one minutes notice is totally unacceptable. Give me a believeable answer as to how you will make your communication much, much better.
You are absolutely right, we should have provided better warning. Following Saturday’s problems, we identified that the database had some threads that were frozen and we needed to free them up in order to guarantee performance for Sunday. In this instance we wanted to message people in the quickest way possible. In future, we want to use push notifications, like web messaging, that will give you a better warning more often.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:43
With regard to your refund offer how is that fair? Somebody who wasn't effected during the outage last week will benefit while I , who was effected , will not as I am not here for a large part of this week. You surely should only compensate those logged on during the outage.
Yes we appreciate some people will have that benefit. It is really hard to find a perfect solution as you will have seen from the other answers above such as voiding bets is unfair to winners. So it is not perfect but we thought it was better to include as many people as we could.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:44
Why do you continue to launch new products when you are not confident the site is stable – surely this only creates more instability?
What we have discovered is that we created a bit too much change in a short period.  We've learned our lessons and adjusted our technology and change processes.  Reliability and performance are our top priorities and we have adjusted our product delivery approach as a result.  The first example of this committment is a complete change freeze through 21st March.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:46
In your apology posting yesterday you mention the following “This has been an operational concern for several weeks as our traffic has reached record volumes week after week”. If you had concerns why didn’t you alert your customers?
We knew how to avoid the issue and believed it only to be caused by certain manual operations.  As a result, we believed we had mitigated the impact to our customers.  As it turns out, we missed an important automated job which resulted in Saturday's outage.  We have taken many steps to mitigate future occurences and are certain this issue won't happen again.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:48
Have Betfair considered charging high volume users (i.e. Bots)?
There are actually charges to deter extreme usage of the facilities provided by Betfair: (i) customers who place more than 1000 bets in any hour may be charged a transaction fee depending on the number of bets placed and the amount of commission they have already paid; and (ii) customers who make more than 20 data requests in any one second may also be charged a fee depending on the number and type of data requests made and whether they are also generating matched bets as opposed to just reading data.  Details of these fees are set out on the Betfair Charges page.  However, in this instance, our systems didn't have a fundamental problem with load.  As already explained, our recent product releases introduced a bug that  triggered performance and reliability problems for our web site.  We have found and fixed the core problem and don't expect a repeat.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:51
So anyone who got stuck in a large losing position on Saturday and has had to scale down or out as a result will get compensated less/not at all. Whereas anyone who got stuck in a large winning position will be able to bet more and thus benefit more. Who came up with a compensation idea that discriminates against those that lost the most? You should be refunding people commission from last week, not this week, and implementing a discount decay suspension.
We looked at a whole range of options but we felt the fairest option was to give a commission refund and announce it in advance so that everyone has the chance to benefit.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:52
In future, would it be possible to warn of a planned outage using the little speech bubble in the top right?
If you mean the inbox message upon login, then yes, we will be using this in future
By:
Betfair Community Manager 2
When: 14 Mar 11 19:56
Why are you adding whizzy add-ons to the site, when this merely slows response times from the site?
We are always trying to improve the customer experience and usability of the site and we are always focusing on performance as a part of these changes. However, the particular issue on Saturday was caused by a bug in one of the new product enhancements which we have identified and fixed.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:58
Is it not about time that you committed to some sort of Service Level Agreement with your customers? Ideally this SLA would result in you bearing a 'cost' of some sort when you are incapable of providing the service for which you are charging us?
From a financial point of view, it is in all of our interests to keep the products running. Site unavailability will obviously cost us in lost commission.  On this occasion we are also sharing the pain through the commission refund offer on a series of markets this week that will cost us a significant sum of money. So there is a very large cost to us for not meeting our customers service expectations but we think it is the right thing to do to begin to regain your confidence in us. Sorry once again for this. I think you make a valid suggestion though and it is one we will take on board and work out what we can do better.
By:
Betfair Community Manager 2
When: 14 Mar 11 19:59
After the recent instability problems - what have you done to guarantee the restored service will withstand the demands placed upon it by Cheltenham?
We have done everything in our power to maximise the stability of our site this week.  From the change freeze we've mentioned through to placing all teams on standby for rapid response to any unexpected problems.
By:
Betfair Community Manager 2
When: 14 Mar 11 20:01
As I can see from your explanation on the core threading issues you have encountered working in a similar environment (High Load server site) I want to congratulate you on the quick decision not to restart betting after the issues you encountered. I believe the decision actually saved your customers form further issues on Saturday.
One valid point would be to really look at would be to give users the option to access Betfair without all the “jazz and whistles” as just give users a slimmed down version to use especially during high load days. I have seen the issues you have encountered coincided with the  “jazz and whistles” upgrades so am not sure if this maybe related. Is this something that is under investigation?

That is a great idea and something we have already got our R&D team looking into.
By:
Betfair Community Manager 2
When: 14 Mar 11 20:02
Why give zero commission to all punters? Some individuals suffered directly as a result of the site failure and others "got lucky" - shouldn't you be looking to directly compensate those that lost out?
You’re right, everyone has a different position and felt lucky / unlucky or perhaps got exactly what they wanted to achieve from the markets on Saturday. As already explained, the issue we face is that we cannot void a market as this will obviously be unfair to customers with winning positions.  It is very difficult to find a perfect solution as you will have seen from the other answers above, but we thought it was better to include as many people as possible in the commission refund offer.
By:
Betfair Community Manager 2
When: 14 Mar 11 20:12
On Saturday there were three phases, system crashed, system volatile, system backup. Is there anyway of being informed that we are in the “system volatile” state?
We are looking into using the Web Messaging function for communicating with customers regarding the health of the site.  Sometimes, it is impossible to spot these things far enough in advance but when we can predict it, it is our intention to introduce a mechanism that will alert customers to this fact.
Page 1 of 2  •  Previous 1 | 2 | Next
sort by:
Show
per page

Post your reply

Text Format: Table: Smilies:
Forum does not support HTML
Insert Photo
Cancel
‹ back to topics
www.betfair.com

New to Betfair?

You need to open an account before you can add content to the forum.

Opening an account only takes a few minutes.

register now