nice to have a good explanation

I second that ^^

Tripling of webservers tells me they dont know much about capacity planning. Not sure why they would want to make this known to public

the fact that they're putting a freeze on their batsh1t crazy release schedule during one of the busiest weeks of the year implicitly confirms how risky it is.

Nice to get some sort of explanation and well done to betfair for that, but I cannot help thinking that had it not been for the massive outburst from customers then nothing would have been forthcoming.
Solutions they should consider;
1. Getting rid of the ridiculous casino/gaming add ons and separate them completely from the exchange site.
2. The new in-play pages are just awful, just drop them Betfair.

should be packaging up releases and doing them at set times after regression and performane testing has completed. sounds like they are using the live environment to do their testing!

I interviewed Tony McAlister about 12 months ago. Seemed a nice guy passionate about his job and the company but he has to count himself seriously lucky to be still in a job. If the site collapses during the festival then the whole management needs to walk. Cut the arcade junk and other rubbish clogging up the site and get back to being a sports betting exchange.

When I joined back in 2002 I tried to persuade gambling friends about the benefits of betfair. That was when the site was more stable and didn't try and ram the stupid games for degenerate gamblers down our throats. It's a shame the purple site can't match here for liquidity because people would be migrating there in their droves.

Stability is by far the most important issue

You can't go making releases that may affect this primary concern without doing thorough regression testing.

making any changes to the live environment without testing in the test environment shows the sort of people that you are dealing with

People that don't know what they are doing

Also this site was not up at 8pm, I know because I was trying to trade out of the Real Madrid game

Before you pat betfair on the back too much for providing an explanation, please realise on this occasion they had absolutely no choice.

Silverback you are dead right

This was a monumental balls up that stopped this site for 6 hours on a Saturday afternoon, early evening

"Tripling of webservers tells me they dont know much about capacity planning."
Yes but .. if I read their explanation right, the servers normally run multi threaded but were switching to single?

Now .. if they're running say quad core dual processors, was each server then running on say 1/8th capacity?!

I agree with the two main points made above: firstly that Betfair have given a decent explanation, for once, about what happened.  This took some courage as it exposes them to criticism about the way they handle their system changes and capacity management.  But, I for one feel more comfortable with the honesty, rather than being kept in the dark.  And secondly, anyone who has spent any time in IT operations/service delivery (and clearly there are some above), they will know that change is the enemy of solid uninterrupted operation.  I suspect even the novice would spot that the level of changes, and done on a continual basis (rather than batched up into releases)is a recipe for exactly what we have been experiencing.

But as I say - they have fessed up, and as long as they continue to do so, and explain the challenges they are facing, the more likely I am to be sympathetic.

Proof will be in the pudding. They are saying everything is fine now and no releases this week, So there should be no issues this week.

If thats so then its job well done finding the bottle neck. Their proceedures need more work!

"In tech speak, our servers weren't thread-safe on certain types of content changes.

This has been an operational concern for several weeks ...
We've fixed this problem now."

It was a concern for several weeks and 24 hours after the whole thing fell you've fixed it?

Oh ffs...nobody died...get a grip on reality.

saturdays outage cost me £380 and i go off lightly...

thousands and thousands were lost by traders because of yesterdays outage...

to think an explanation somewhat makes up for these losses is wide of the mark...

Jimmy, with Pricewise horses, a lot of people, with very big banks, lay the horse e.g. Alarazi in yesterdays Imperial cup at say 10.0. They do this for maybe £1000 or more i.e. liability £9k

Their objective is to trade out before the off to make a profit (but sometimes a loss) on the price movement

This provides the site with it's legendary liquidity for big punters

They also trade on football matches e.g. Man U vs Arsenal, sometimes for tens or even hundreds of thousands

If the site goes down they effectively have a bet of that size on the event

While you may say tough luck, if traders have no confidence in the stability of the site then they won't put money up and there will be no liquidity

No-one died, that is true. That however doesn't mean anything going wrong beneath that threshold doesn't matter

This situation was totally avoidable

Betfair acknowledged that they knew that there were problems at 2pm but did nothing to inform their customers

Depending on how much you lost yesterday, you are probably to that extent upset

The fact is not everyone is talking about missing a £5 bet, which I think you believe is the situation driving the anger

Any explanation as to why they put the Cricket back in play and tried to resume order with the Utd game when the site was clearly ****?

They didn't put the cricket back in play the second time, only the first time. (They did settle it instantly though)

Oh look I've just had the normal 15% weekly decay on my Betfair points. Whilst it hasn't affected me this time what if you hadn't made it to your normal rate because of the site unavailabilty? You actually pay them more commission if they provide a worse service. Surely there should be a built in compensation factor.

Don't hold you breath for any type of compensation.  Many times in the past BF have found it easy to write the words but then conveniently go quiet and hope nobody notices.

Just wanted to copy and paste the announcement here because knowing Betfair, it will dissappear within the next 24 - 48 hours.

  Betfair Customer Services Joined: 22 May 02
Replies: 1795 13 Mar 11 16:17   
On Saturday March 12, from 14:05 through 20:01, Betfair's website failed. We know how frustrating this is for our customers and offer our sincere apologies to all affected.

We are now looking at ways in which we can make up Saturday’s events to you.

We are working as hard as possible to ensure Betfair offers as reliable a site as possible. In a normal week we make at least 15 changes to the Betfair website but we have resolved not to release any new products or features for the next seven days. This should give maximum stability throughout a busy week that includes the Cheltenham Festival, cricket World Cup and Champions League football.

Below is an explanation of what went wrong and what we have done to fix the issue.

When the website failed on Saturday, our first step was to disable Betfair for all our customers on the web, API and mobile services.  Once we identified the actual problem, we determined that we needed our website "available" but with betting disallowed.  We recovered the site internally around 18:00 and re-enabled betting as of 20:00 once we were certain it was stable.

Here is what actually happened:

After performing certain types of website changes, an issue developed that caused our servers to temporarily slow down, processing just one thing at a time (single threading) instead of thousands of user requests in parallel.  This "single threading" behaviour was introduced some time ago to protect against occasional broken pages caused by serving content while it is changing.  In tech speak, our servers weren't thread-safe on certain types of content changes.

This has been an operational concern for several weeks as our traffic has reached record volumes week after week.  While we had several operational protections in place to limit these types of changes during peak load, we missed an important one.  Every 15 minutes, an automated process was publishing exactly the type of content that triggers the issue described above.  Yesterday we hit a tipping point as the web servers reached a point where it was taking longer than 15 minutes to complete their update - essentially rendering the servers unusable.

Then in an attempt to quickly shed load, we triggered a process to disable some of the computationally intensive features on the site.  Unfortunately, the way this was done triggered a complete recompile of every page on our site, for every user, in every locale.  Under our normal Saturday usage, recovery took several hours.

After spotting the pattern, we've recognised this has been going on with varying impact since February 8, 2011.  During periods of increased user traffic, our customers would experience this issue in the form of slow navigation or a "sticky" user experience.  Yesterday was simply a tipping point, made worse by our recovery attempt.

We've fixed this problem now.  We've disabled the original automated job and rebuilt it to update content safely.  We've tripled the capacity of our web server farm to spread our load even more thinly. We've fixed our process for disabling features so that we won't make things worse. We've updated our operational processes and introduced a whole new raft of monitoring to spot this type of issue.  We've also isolated the underlying web server issue so that we can change our content at will without triggering the switch to single-threading.

We believe these changes will bring the stability we all desire and thank you for your continued custom.

Yours faithfully,

Niall Wass – Chief Marketing & Development Officer
Tony McAlister – Chief Technology Officer

dont hold your breath on the compensation front.....they are the devil dress in sheeps clothing.....[smiley:crazy]

It will probably be something in the lines of not decaying the Betfair points for a week or so.
  It's almost impossible to do anything else that will benefit almost all customers and not cost Betfair to much.

How about a week-free of Premium Charge. Go on BF, you know you want to ...