Reading through the many threads following recent events, culminating in the catastrophic failure on Saturday, it’s clear that there are a lot of savvy and seasoned IT guys on the Forum. I thought it might be interesting to try and bring together in one thread the cumulative wisdom available on here, and offer Betfair a wealth of free and constructive advice.
Personally, I’m getting a bit old for the game, having being doing IT for over 30 years, and finding increasingly that I’m being over taken by bright young things and developments that my expanding senior moments are finding harder to grasp. However, even for an old buffer there seem to be a number of obvious things that could be addressed. I have no doubt those of more agile mind, and up-to-date with the latest thinking and developments, will have better informed views.
This may be wishful thinking, but I’m hoping people will confine their understandable anger to other threads, and concentrate here on dispassionate ideas and proposals.
Although it is clear that there is much to be done with amongst other things their ITIL processes (change, problem, RCA, capacity, demand management, OAT, etc.), and these certainly merit further discussion, I thought I would start with the organisation and management structure.
Looking at the Betfair Board and Executive team profiles, although there are a number of individuals who have done some leading edge technology developments and creations, there appears, on the face of it, little in the way of seasoned operational/service delivery experience.
Most organisations split their IT in some way between Development and Operations (new stuff – with say CTOs and Development Directors in charge, and “Keeping The Lights On” – with say Operations/Service Delivery Directors accountable). This is done for a reason. Within any organisation, and particularly so within companies like Betfair that seek rapid growth, there is a natural tension between innovation/developing new stuff to support the business plan to grow revenues with new and innovative products, whilst at the same time maintaining the current products in a way that protects the existing customer base. Getting this balance right is not always easy. Sorting out the responsibilities and accountabilities at the top of the shop is a good starting point.
The Operations/Service Delivery Director is measured and incentivised against “Keeping The Lights On”, whilst the Chief Technology Officer (CTO) is measured against the use of innovation/development to support the business plan. This structure generally brings with it the natural tension that the CTO has to satisfy the Ops Director that proposed changes (the enemy of rock solid operations), and the manner in which those changes are to be made, do not endanger the current operation. If, for whatever reason, the Ops Director is compelled to make changes against his/her better judgement, he/she will naturally ensure that all are fully aware and understand the risks and the potential consequences.
The fact that the “Apology and an explanation” was signed by the “Chief Marketing & Development Officer” and “Chief Technology Officer” begs the question in my mind: - where’s the operations guy? It suggests to me that Betfair would benefit from a good look at its senior organisation and a close review of the accountabilities and responsibilities with the intention of getting a better balance between new developments and retaining robust existing products.
Pay better wages, when your payroll consists of interns on peanuts, and experienced staff are offered starting salaries of circa £14k and expected to relocate abroad for this priviledge then you can expect problems.
Also sort out your so-called "talent aquistions managers" they are useless, all they do is spend their time spamming peeople on monster and linkedin. Another problem is that they are unable to read, what part of "not looking for work" are they unable to read ? Even if I was, do they seriously expect me to give up a 6 figure salary to come to betfair and start in an entry position on £14k ?
If you want proof of the "talent" that the TAM's are bringing to betfair, then that is easily measured by the roaring success of the "promotions manager" they recruited for malta.
Pay better wages, when your payroll consists of interns on peanuts, and experienced staff are offered starting salaries of circa £14k and expected to relocate abroad for this priviledge then you can expect problems.Also sort out your so-called "tale
1) It's always good to be honest with your customers about the reasons for outages, and to explain what you're doing to avoid future occurrances, so wd BF on that (at last)
2) 15 changes per week to the functionality (as opposed to just updating some text/images)of a 'live' database-driven website is 14 too many.
3) Fixes applied in the middle of a crash are unlikely to be a long-term solution.
4) Release schedules that are determined by Marketing Departments as opposed to Technical Departments almost always lead to the necessity for retrospective fixes, which lead to further retrospective fixes, which lead to ...
1) It's always good to be honest with your customers about the reasons for outages, and to explain what you're doing to avoid future occurrances, so wd BF on that (at last)2) 15 changes per week to the functionality (as opposed to just updating some
I agree with all of those points Gus. Picking up specifically on your number 3) - yep. There seemed to be a lot of wholesale changes taking place during recovery. I wonder how they were tested. That of course leads us to the whole area of the testing strategy and execution.
I agree with all of those points Gus. Picking up specifically on your number 3) - yep. There seemed to be a lot of wholesale changes taking place during recovery. I wonder how they were tested. That of course leads us to the whole area of the tes
i want to know if their developers, designers, analysts and architects are complaining about the "unnecessary" bureaucracy and "protracted" processes demanded by their Release Manager.
i want to know if the people requesting change are complaining about the "unnecessary" length of regression testing cycles before deployment into the live environment.
if they aren't, then stability isn't a priority.
i want to know if their developers, designers, analysts and architects are complaining about the "unnecessary" bureaucracy and "protracted" processes demanded by their Release Manager. i want to know if the people requesting change are complaining ab
Unless there's something we don't know, then that would seem the inevitable conclusion - which would tally with the Bs hiring Cs issue you raised earlier.
BrendanUnless there's something we don't know, then that would seem the inevitable conclusion - which would tally with the Bs hiring Cs issue you raised earlier.
iirc, it was all fine despite some seriously heavy activity.
also iirc, there was a moratorium on releasing anything new in the weeks before and during the competition.
iirc, it was all fine despite some seriously heavy activity. also iirc, there was a moratorium on releasing anything new in the weeks before and during the competition.
let's face it, if you're working in London and being paid £14,000 p.a. you'll find it difficult to concentrate on your job for worrying about where to find a cardboard box to live in.
let's face it, if you're working in London and being paid £14,000 p.a. you'll find it difficult to concentrate on your job for worrying about where to find a cardboard box to live in.
We're waiting for betting_quant, or someone else in the know, to provide some provenance for this figure. Personally, I find it difficult to believe.
GusWe're waiting for betting_quant, or someone else in the know, to provide some provenance for this figure. Personally, I find it difficult to believe.
well lets be honest, for a random call centre person in Malta who reads from a script, £14k is probably fine given the lower cost of living out there (and all the sun!).
well lets be honest, for a random call centre person in Malta who reads from a script, £14k is probably fine given the lower cost of living out there (and all the sun!).
Alex, I am, although saying that I'm very pi$$ed at anyone in the recruitment industry at the momement, they are absolute vermin, especially the ones in the betting industry.
The job was non IT, but I have enough experience of this industry to know that the pay and conditions in one department is very indicative of those in others.
yep wayne, job was in malta, I was told the cost of living was cheaper out there and thus the wages needed to reflect that.
Alex, I am, although saying that I'm very pi$$ed at anyone in the recruitment industry at the momement, they are absolute vermin, especially the ones in the betting industry.The job was non IT, but I have enough experience of this industry to know th
I'm not wanting an argument, but you posted something that seemed to imply it was IT, got everybody's attention, then you've just given a three sentence response where you could've settled it simply by saying what it was in a few words, but chose not to clarify it, which was interesting.
I'm guessing entry level BF trader in Malta might pay less than experienced IT bod in London, however.
I'm not wanting an argument, but you posted something that seemed to imply it was IT, got everybody's attention, then you've just given a three sentence response where you could've settled it simply by saying what it was in a few words, but chose not
ah, sorry JC, just realised my initial post did imply that.
I have no idea what god or bad IT salaries are, but my suspsion is that betfair are towards the bottom end of the pay scale.
ah, sorry JC, just realised my initial post did imply that.I have no idea what god or bad IT salaries are, but my suspsion is that betfair are towards the bottom end of the pay scale.
I've done a trawl to see if I can find some examples of Betfair's salaries. All of the job adverts I've seen say "salary negotiable". Looking at their jobs/careers stuff it looks at first sight to be pretty good. Training, competitive package, strong values etc. So unless someone has something more definite to go on, I for one am assuming that it's not salaries that are the issue.
Perhaps we could move on.
I've done a trawl to see if I can find some examples of Betfair's salaries. All of the job adverts I've seen say "salary negotiable". Looking at their jobs/careers stuff it looks at first sight to be pretty good. Training, competitive package, str
There is no evidence whatsoever that betfair pay below market rate. This is all started because betting_quant was upset with his job offer and now we understand that he is upset with all recruitment consultants.
It is tricky to know whether all recruitment consultants (who are guided by client constraints and economic realities) and betfair are out of line or whether betting_quant is.
There is no evidence whatsoever that betfair pay below market rate. This is all started because betting_quant was upset with his job offer and now we understand that he is upset with all recruitment consultants.It is tricky to know whether all recru
brendanuk1 The amount of technical roles currently empty would indicate that they are not paying top dollar.
Many organisations advertise jobs that don't exist because they know they have natural wastage and a list of potential future employees is good business sense. (I used to do it.)
subversion trouble is, their disastrous stability record makes speculation such as betting_quants very easy to believe
It is best to view such claims with an open mind. We don't have the facts let alone know what job he was applying for.
brendanuk1The amount of technical roles currently empty would indicate that they are not paying top dollar.Many organisations advertise jobs that don't exist because they know they have natural wastage and a list of potential future employees is good
Many organisations advertise jobs that don't exist because they know they have natural wastage and a list of potential future employees is good business sense. (I used to do it.)
thats just silly, do you work in IT?
Many organisations advertise jobs that don't exist because they know they have natural wastage and a list of potential future employees is good business sense. (I used to do it.)thats just silly, do you work in IT?
let's not lose sight of something in all this. the site was actually very stable for many months (including the world cup), so plainly they are technically capable of running a robust, trustable site.
the problems seem linked to changes implemented on a platform that was working fine, and which seem to be of questionable value judging by responses on here. so to me it seems a management/strategic failure more than proof that the IT people aren't up to the job.
let's not lose sight of something in all this. the site was actually very stable for many months (including the world cup), so plainly they are technically capable of running a robust, trustable site.the problems seem linked to changes implemented on
the site was actually very stable for many months (including the world cup),
I don't recall a week without a crash since about 2006, (I was whining about this before the recent bigger problems)
I do live on the site though tbf
the site was actually very stable for many months (including the world cup), I don't recall a week without a crash since about 2006, (I was whining about this before the recent bigger problems)I do live on the site though tbf
yeah, that's weird lori, because I'm on here loads too and honestly didn't notice anything for nearly a year.
whereas all this recent bullsh!t I've experienced virtually all of firsthand.
yeah, that's weird lori, because I'm on here loads too and honestly didn't notice anything for nearly a year.whereas all this recent bullsh!t I've experienced virtually all of firsthand.
brendanuk1 The amount of technical roles currently empty would indicate that they are not paying top dollar.
I can't imagine forcing your CTO to sign a public notice admitting a mammoth screw-up is going to do much for staff morale and future recruitment either!
All the on-going problems can be traced back to QA, either it's all been out-sourced, or management are in such a rush to force out changes that they're not waiting for tests to be completed.
brendanuk1The amount of technical roles currently empty would indicate that they are not paying top dollar.I can't imagine forcing your CTO to sign a public notice admitting a mammoth screw-up is going to do much for staff morale and future recruitme
i would like to see a service level agreement set out so we know what their targets are and what our risks maybe.
lets say 99.5% uptime and when a downtime occurs maximum outage of 1 hour (ideally less) to roll back changes / switch to contigency servers.
HOWEVER - EVEN WITH PROPER TESTING , BACKOUT PLANS ETC WE WILL NEVER GET 100% UPTIME WITH ANYONE. THEREFORE THERE WILL ALWAYS BE RISK FOR TRADERS BUT WE NEED TO BE ABLE TO QUANTIFY THE RISK .
i would like to see a service level agreement set out so we know what their targets are and what our risks maybe.lets say 99.5% uptime and when a downtime occurs maximum outage of 1 hour (ideally less) to roll back changes / switch to contigency serv
I'm not sure that I agree Viva with the site stability - more with Lori on that one. I've been around for about five years and seem to remember the site failing from time to time. Having said that, as they say in Operations: "The best you can hope for is a draw". We don't remember when it's going swimmingly, only when it fails.
I do agree with you though: it would be wrong to jump to the conclusion that the failures are because of an underpaid bunch of muppets. There are some things that wouldn't happen if they didn't have the requisite skills. Like you, I lean towards management failings, trying to push through too much change, not tensioning their people correctly, and enforcing tight process.
I'm not sure that I agree Viva with the site stability - more with Lori on that one. I've been around for about five years and seem to remember the site failing from time to time. Having said that, as they say in Operations: "The best you can hope
Very good point. Just saw the SLA idea on the forum Chat (I presume it was yours). Personally I'd think four 9s (99.99%) availability is very reasonably achievable. And yes Brendan some transparency in the measurement and reporting required.
Getinthere, BrendanVery good point. Just saw the SLA idea on the forum Chat (I presume it was yours). Personally I'd think four 9s (99.99%) availability is very reasonably achievable. And yes Brendan some transparency in the measurement and reporti
99.99 is 1min a week = no chance here, think they would struggle with 3 9s. And if downtime due to upgrades was included they wouldnt do 99%
http://en.wikipedia.org/wiki/High_availability
99.99 is 1min a week = no chance here, think they would struggle with 3 9s. And if downtime due to upgrades was included they wouldnt do 99%http://en.wikipedia.org/wiki/High_availability
I'm surprised we've had such different experiences. I remember a period of months when it was pretty bad about 18-15 months ago, then the most troublefree 12 months I'd experienced since day 1 (jun 06), followed by the worst couple of months.
I'm surprised we've had such different experiences. I remember a period of months when it was pretty bad about 18-15 months ago, then the most troublefree 12 months I'd experienced since day 1 (jun 06), followed by the worst couple of months.
I was pleasantly surprised it was ok during the world cup, as approaching 3pm on a saturday a few months before that, the site would predictably grind to a juddering halt.
I was pleasantly surprised it was ok during the world cup, as approaching 3pm on a saturday a few months before that, the site would predictably grind to a juddering halt.
You could well be right Viva. But people have certainly been reporting performance issues (myself included) since the last big crash. Thinking back, perhaps it had been relatively trouble free for the previous 12 months or so. I'll take a look at a couple of sites that are usually quick to report on outages, and see what the history says.
You could well be right Viva. But people have certainly been reporting performance issues (myself included) since the last big crash. Thinking back, perhaps it had been relatively trouble free for the previous 12 months or so. I'll take a look at
brendanuk1 14 Mar 11 18:50 Joined: 12 Jan 02 | Topic/replies: 14,296 | Blogger: brendanuk1's blog Many organisations advertise jobs that don't exist because they know they have natural wastage and a list of potential future employees is good business sense. (I used to do it.)
thats just silly, do you work in IT?
No. I ran and advised businesses with many employees though.
brendanuk114 Mar 11 18:50 Joined: 12 Jan 02 | Topic/replies: 14,296 | Blogger: brendanuk1's blogMany organisations advertise jobs that don't exist because they know they have natural wastage and a list of potential future employees is good business s
Yes, I'm probably setting the bar too high. Excluding scheduled downtime do we think 3 9s (99.9% - 10 mins a week) would: a) be realistically achievable, and b) satisfy us (I guess from a risk point of view mainly)?
BrendanYes, I'm probably setting the bar too high. Excluding scheduled downtime do we think 3 9s (99.9% - 10 mins a week) would: a) be realistically achievable, and b) satisfy us (I guess from a risk point of view mainly)?
do you have list of the tricks of the trade you could share with us?
artisan,
i would say 1 outage a month lasting no longer than 1 hour (ideally less). which i guess is around 99.85%
alex, do you have list of the tricks of the trade you could share with us? artisan,i would say 1 outage a month lasting no longer than 1 hour (ideally less). which i guess is around 99.85%
99.9 sounds ok they have a good a chance of failing as of succeeding imo.
Once real time results of third party monitoring is made public they could set up a market. Will Betfair reach its uptime SLA? Yes/No
99.9 sounds ok they have a good a chance of failing as of succeeding imo.Once real time results of third party monitoring is made public they could set up a market. Will Betfair reach its uptime SLA? Yes/No
I've taken a squint at the BetAngel site forum - they usually pick up on BF outages pretty quickly. I found the following bits and pieces:
"Mon Jan 31, 2011 11:37 am (Following scheduled maintenance) Still erratic at the moment, so please be careful before trying anything. We will be keeping an eye on things but it would be useful if anybody post issues to this thread.
It looks like the site is not fully up at the moment, the odds being returned by the API are all over the place.
Betfair Customer Services 03 Jul 10 13:42 Betfair would like to apologise for the difficulties some customers are experiencing on the website, particularly in respect of the ‘My Bets’ tab and markets prices not updating correctly.
We are investigating the cause of this problem as a matter of extreme urgency and will keep you updated.
Sun Jan 31, 2010 12:36 am Firstly some music to cheer you up while reading this post, I thought the band kind of suited:
At bang on midnight Betfair, yup you guessed it, died again! Not just the API but Betfair wide. I then spent 10 mins on hold being told all the customer services people were too busy to answer my call so eventually hung up!
Sat Jan 30, 2010 6:59 am Be careful this morning the API has been all over the shop overnight and is still playing up early this morning."
If that's all there was, then prior to the end of January this year, they've only had two major hits in the preceding 12 months, one in July (what dates were the World Cup?), and one in January last year. So on that basis, your recollection is more accurate than mine. Although I suspect all of the niggles (slow browsers, clearing cache, Aus wallets going etc.) won't be recorded, but certainly stick in people's minds.
VivaI've taken a squint at the BetAngel site forum - they usually pick up on BF outages pretty quickly. I found the following bits and pieces:"Mon Jan 31, 2011 11:37 am (Following scheduled maintenance)Still erratic at the moment, so please be care
In fact, thinking about it some more, your idea is definitely worth pursuing. BF should create a market on the chances of the site going down today. Site users, traders in particular, could hedge against being hit by such an eventuality (effectively take out insurance). The problem being that the premiums at the moment might be prohibitive.
In fact, thinking about it some more, your idea is definitely worth pursuing. BF should create a market on the chances of the site going down today. Site users, traders in particular, could hedge against being hit by such an eventuality (effectivel
you would of course have to keep adjusting your hedge to current exposure levels. after a month of not having an outage the price may be getting pretty short and expensive to increase your hedge :-)
if im on £14k in IT I could also fund my lifestyle by pulling the plug just after getting a wedge on.
you would of course have to keep adjusting your hedge to current exposure levels. after a month of not having an outage the price may be getting pretty short and expensive to increase your hedge :-)if im on £14k in IT I could also fund my lifestyle
It all seems very disjointed having data and people in Malta, Dublin, Australian, London etc. Did the major problems start when they started the major shift over to Dublin? Is there is a massive pre-occupation with making tax savings and this is forcing sub-optimal technical choices to made? It all seems such a waste of money moving everything just to deny the UK exchequer a few quid given how well Betfair has done over the last decade being based here.
It all seems very disjointed having data and people in Malta, Dublin, Australian, London etc. Did the major problems start when they started the major shift over to Dublin? Is there is a massive pre-occupation with making tax savings and this is forc
Seems there was a code freeze before going to Dublin, this caused a back log of changes and these were pushed through poorly tested and once problems started to happen they just ploughed on.
Seems there was a code freeze before going to Dublin, this caused a back log of changes and these were pushed through poorly tested and once problems started to happen they just ploughed on.
Getting back to the original topic of this thread:
"As an IT professional, what advice would you give Betfair?"
I'd add: There have , in the recent past (another poster mentioned the period before and during last year's World Cup, and i'd agree) , been long periods when the BF (Sports Exchange) site has been stable, reliable, and a joy to use.
What irritates me, and i'd guess quite a lot of other users is that, having achieved that astonishingly (given the demands) enviable level of stability / reliability, BF seem to have blown it.
Nothing that has been done during recent months has added to the BF (Sports Exchange) experience, whilst some things that have been done detract from it.
The most irritating thing, to this IT prfessional, is the perversity of sacrificing such genuinly admirable achievement for no apparent gain.
Just stop doing stuff!
Getting back to the original topic of this thread:"As an IT professional, what advice would you give Betfair?"I'd add: There have , in the recent past (another poster mentioned the period before and during last year's World Cup, and i'd agree) , been
Given the lack of relationship in function between say sports betting and the arcade stuff, if a lot of these new changes are for THAT perhaps they should be hosting that on completely separate servers, e.g. all such servers on arcade.betfair.com, perhaps even with a separate bank like the oz bank so it's all separate. Contain the disease so no upgrade for that has a chance of making the whole house of cards fall down.
Given the lack of relationship in function between say sports betting and the arcade stuff, if a lot of these new changes are for THAT perhaps they should be hosting that on completely separate servers, e.g. all such servers on arcade.betfair.com, pe
Problem with running markets on exchange failures is that some dodgt russian with a large botnet can probably lump on and then nuke the site. the CIA had terrorism prediction markets for a while but decided it was probably not a good idea for the same reasons.
Problem with running markets on exchange failures is that some dodgt russian with a large botnet can probably lump on and then nuke the site. the CIA had terrorism prediction markets for a while but decided it was probably not a good idea for the sam
Yep, some of the old adages spring to mind, like: "If it ain't broke don't fix it".
Just Checking
I agree that there is definitely something about segregation; getting the systems into manageable bite sized discrete chunks. We all know that large unwieldy systems are a nightmare to test, diagnose and fix when they snap. Mission critical stuff is deliberately broken down into as smaller units as possible for this very reason.
GusYep, some of the old adages spring to mind, like: "If it ain't broke don't fix it".Just CheckingI agree that there is definitely something about segregation; getting the systems into manageable bite sized discrete chunks. We all know that large u