General Betting
There is currently 1 person viewing this thread.
Reading through the many threads following recent events, culminating in the catastrophic failure on Saturday, it’s clear that there are a lot of savvy and seasoned IT guys on the Forum. I thought it might be interesting to try and bring together in one thread the cumulative wisdom available on here, and offer Betfair a wealth of free and constructive advice.
Personally, I’m getting a bit old for the game, having being doing IT for over 30 years, and finding increasingly that I’m being over taken by bright young things and developments that my expanding senior moments are finding harder to grasp. However, even for an old buffer there seem to be a number of obvious things that could be addressed. I have no doubt those of more agile mind, and up-to-date with the latest thinking and developments, will have better informed views.
This may be wishful thinking, but I’m hoping people will confine their understandable anger to other threads, and concentrate here on dispassionate ideas and proposals.
Although it is clear that there is much to be done with amongst other things their ITIL processes (change, problem, RCA, capacity, demand management, OAT, etc.), and these certainly merit further discussion, I thought I would start with the organisation and management structure.
Looking at the Betfair Board and Executive team profiles, although there are a number of individuals who have done some leading edge technology developments and creations, there appears, on the face of it, little in the way of seasoned operational/service delivery experience.
Most organisations split their IT in some way between Development and Operations (new stuff – with say CTOs and Development Directors in charge, and “Keeping The Lights On” – with say Operations/Service Delivery Directors accountable). This is done for a reason. Within any organisation, and particularly so within companies like Betfair that seek rapid growth, there is a natural tension between innovation/developing new stuff to support the business plan to grow revenues with new and innovative products, whilst at the same time maintaining the current products in a way that protects the existing customer base. Getting this balance right is not always easy. Sorting out the responsibilities and accountabilities at the top of the shop is a good starting point.
The Operations/Service Delivery Director is measured and incentivised against “Keeping The Lights On”, whilst the Chief Technology Officer (CTO) is measured against the use of innovation/development to support the business plan. This structure generally brings with it the natural tension that the CTO has to satisfy the Ops Director that proposed changes (the enemy of rock solid operations), and the manner in which those changes are to be made, do not endanger the current operation. If, for whatever reason, the Ops Director is compelled to make changes against his/her better judgement, he/she will naturally ensure that all are fully aware and understand the risks and the potential consequences.
The fact that the “Apology and an explanation” was signed by the “Chief Marketing & Development Officer” and “Chief Technology Officer” begs the question in my mind: - where’s the operations guy? It suggests to me that Betfair would benefit from a good look at its senior organisation and a close review of the accountabilities and responsibilities with the intention of getting a better balance between new developments and retaining robust existing products.
Personally, I’m getting a bit old for the game, having being doing IT for over 30 years, and finding increasingly that I’m being over taken by bright young things and developments that my expanding senior moments are finding harder to grasp. However, even for an old buffer there seem to be a number of obvious things that could be addressed. I have no doubt those of more agile mind, and up-to-date with the latest thinking and developments, will have better informed views.
This may be wishful thinking, but I’m hoping people will confine their understandable anger to other threads, and concentrate here on dispassionate ideas and proposals.
Although it is clear that there is much to be done with amongst other things their ITIL processes (change, problem, RCA, capacity, demand management, OAT, etc.), and these certainly merit further discussion, I thought I would start with the organisation and management structure.
Looking at the Betfair Board and Executive team profiles, although there are a number of individuals who have done some leading edge technology developments and creations, there appears, on the face of it, little in the way of seasoned operational/service delivery experience.
Most organisations split their IT in some way between Development and Operations (new stuff – with say CTOs and Development Directors in charge, and “Keeping The Lights On” – with say Operations/Service Delivery Directors accountable). This is done for a reason. Within any organisation, and particularly so within companies like Betfair that seek rapid growth, there is a natural tension between innovation/developing new stuff to support the business plan to grow revenues with new and innovative products, whilst at the same time maintaining the current products in a way that protects the existing customer base. Getting this balance right is not always easy. Sorting out the responsibilities and accountabilities at the top of the shop is a good starting point.
The Operations/Service Delivery Director is measured and incentivised against “Keeping The Lights On”, whilst the Chief Technology Officer (CTO) is measured against the use of innovation/development to support the business plan. This structure generally brings with it the natural tension that the CTO has to satisfy the Ops Director that proposed changes (the enemy of rock solid operations), and the manner in which those changes are to be made, do not endanger the current operation. If, for whatever reason, the Ops Director is compelled to make changes against his/her better judgement, he/she will naturally ensure that all are fully aware and understand the risks and the potential consequences.
The fact that the “Apology and an explanation” was signed by the “Chief Marketing & Development Officer” and “Chief Technology Officer” begs the question in my mind: - where’s the operations guy? It suggests to me that Betfair would benefit from a good look at its senior organisation and a close review of the accountabilities and responsibilities with the intention of getting a better balance between new developments and retaining robust existing products.
sort by:
Show
per page
Replies: 61
|
By:
Pay better wages, when your payroll consists of interns on peanuts, and experienced staff are offered starting salaries of circa £14k and expected to relocate abroad for this priviledge then you can expect problems.
Also sort out your so-called "talent aquistions managers" they are useless, all they do is spend their time spamming peeople on monster and linkedin. Another problem is that they are unable to read, what part of "not looking for work" are they unable to read ? Even if I was, do they seriously expect me to give up a 6 figure salary to come to betfair and start in an entry position on £14k ? If you want proof of the "talent" that the TAM's are bringing to betfair, then that is easily measured by the roaring success of the "promotions manager" they recruited for malta. |
|
By:
A's Hire A's and B's Hire C's
|
|
By:
don't go into nuclear power.
|
|
By:
betting_quant
I'm a bit stunned. Are they really trying to recruit experienced IT people on £14k? Surely not! To do what? Clean keyboards? |
|
By:
What job was offering £14k? Is that even a legal wage in the UK?
|
|
By:
1) It's always good to be honest with your customers about the reasons for outages, and to explain what you're doing to avoid future occurrances, so wd BF on that (at last)
2) 15 changes per week to the functionality (as opposed to just updating some text/images)of a 'live' database-driven website is 14 too many. 3) Fixes applied in the middle of a crash are unlikely to be a long-term solution. 4) Release schedules that are determined by Marketing Departments as opposed to Technical Departments almost always lead to the necessity for retrospective fixes, which lead to further retrospective fixes, which lead to ... |
|
By:
I agree with all of those points Gus. Picking up specifically on your number 3) - yep. There seemed to be a lot of wholesale changes taking place during recovery. I wonder how they were tested. That of course leads us to the whole area of the testing strategy and execution.
|
|
By:
i want to know if their developers, designers, analysts and architects are complaining about the "unnecessary" bureaucracy and "protracted" processes demanded by their Release Manager.
i want to know if the people requesting change are complaining about the "unnecessary" length of regression testing cycles before deployment into the live environment. if they aren't, then stability isn't a priority. |
|
By:
Dstyle
Well said, as with your questions on the other thread. |
|
By:
The fixes during an outage and short notice down times. tells me their route to getting things onto production is a piece of pish
|
|
By:
take a long look at the world cup last year, and try to figure out what went right.
|
|
By:
Brendan
Unless there's something we don't know, then that would seem the inevitable conclusion - which would tally with the Bs hiring Cs issue you raised earlier. |
|
By:
Viva
I don't do footy, and wasn't around at that time last year. What happened? Capacity issues? |
|
By:
iirc, it was all fine despite some seriously heavy activity.
also iirc, there was a moratorium on releasing anything new in the weeks before and during the competition. |
|
By:
I see that betting_quant is still miffed that betfair thought the fair market value for his expertise and service is £14,000 p.a.
|
|
By:
let's face it, if you're working in London and being paid £14,000 p.a. you'll find it difficult to concentrate on your job for worrying about where to find a cardboard box to live in.
|
|
By:
Gus
We're waiting for betting_quant, or someone else in the know, to provide some provenance for this figure. Personally, I find it difficult to believe. |
|
By:
I think the job was based in Malta.
|
|
By:
well lets be honest, for a random call centre person in Malta who reads from a script, £14k is probably fine given the lower cost of living out there (and all the sun!).
|
|
By:
Alex, I am, although saying that I'm very pi$$ed at anyone in the recruitment industry at the momement, they are absolute vermin, especially the ones in the betting industry.
The job was non IT, but I have enough experience of this industry to know that the pay and conditions in one department is very indicative of those in others. yep wayne, job was in malta, I was told the cost of living was cheaper out there and thus the wages needed to reflect that. |
|
By:
It was to be a betfair 'trader', you said so other day, not sure why you're not wanting to say?
|
|
By:
where did I say i didnt want to say ? I do apologise to all on betfair for not responding quick enough on the forum
|
|
By:
I'm not wanting an argument, but you posted something that seemed to imply it was IT, got everybody's attention, then you've just given a three sentence response where you could've settled it simply by saying what it was in a few words, but chose not to clarify it, which was interesting.
I'm guessing entry level BF trader in Malta might pay less than experienced IT bod in London, however. |
|
By:
ah, sorry JC, just realised my initial post did imply that.
I have no idea what god or bad IT salaries are, but my suspsion is that betfair are towards the bottom end of the pay scale. |
|
By:
I've done a trawl to see if I can find some examples of Betfair's salaries. All of the job adverts I've seen say "salary negotiable". Looking at their jobs/careers stuff it looks at first sight to be pretty good. Training, competitive package, strong values etc. So unless someone has something more definite to go on, I for one am assuming that it's not salaries that are the issue.
Perhaps we could move on. |
|
By:
There is no evidence whatsoever that betfair pay below market rate. This is all started because betting_quant was upset with his job offer and now we understand that he is upset with all recruitment consultants.
It is tricky to know whether all recruitment consultants (who are guided by client constraints and economic realities) and betfair are out of line or whether betting_quant is. |
|
By:
The amount of technical roles currently empty would indicate that they are not paying top dollar.
|
|
By:
trouble is, their disastrous stability record makes speculation such as betting_quants very easy to believe
|
|
By:
brendanuk1
The amount of technical roles currently empty would indicate that they are not paying top dollar. Many organisations advertise jobs that don't exist because they know they have natural wastage and a list of potential future employees is good business sense. (I used to do it.) subversion trouble is, their disastrous stability record makes speculation such as betting_quants very easy to believe It is best to view such claims with an open mind. We don't have the facts let alone know what job he was applying for. |
|
By:
Many organisations advertise jobs that don't exist because they know they have natural wastage and a list of potential future employees is good business sense. (I used to do it.)
thats just silly, do you work in IT? |
|
By:
let's not lose sight of something in all this. the site was actually very stable for many months (including the world cup), so plainly they are technically capable of running a robust, trustable site.
the problems seem linked to changes implemented on a platform that was working fine, and which seem to be of questionable value judging by responses on here. so to me it seems a management/strategic failure more than proof that the IT people aren't up to the job. |
|
By:
the site was actually very stable for many months (including the world cup),
I don't recall a week without a crash since about 2006, (I was whining about this before the recent bigger problems) I do live on the site though tbf |
|
By:
yeah, that's weird lori, because I'm on here loads too and honestly didn't notice anything for nearly a year.
whereas all this recent bullsh!t I've experienced virtually all of firsthand. |
|
By:
brendanuk1
The amount of technical roles currently empty would indicate that they are not paying top dollar. I can't imagine forcing your CTO to sign a public notice admitting a mammoth screw-up is going to do much for staff morale and future recruitment either! All the on-going problems can be traced back to QA, either it's all been out-sourced, or management are in such a rush to force out changes that they're not waiting for tests to be completed. |
|
By:
i would like to see a service level agreement set out so we know what their targets are and what our risks maybe.
lets say 99.5% uptime and when a downtime occurs maximum outage of 1 hour (ideally less) to roll back changes / switch to contigency servers. HOWEVER - EVEN WITH PROPER TESTING , BACKOUT PLANS ETC WE WILL NEVER GET 100% UPTIME WITH ANYONE. THEREFORE THERE WILL ALWAYS BE RISK FOR TRADERS BUT WE NEED TO BE ABLE TO QUANTIFY THE RISK . |
|
By:
They need third party monitoring of uptime and performance (response times) available to public
Bit of transparency |
|
By:
and for gawd's sake sort out the memory leak on this forum - BECAUSE DESPITE YOUR SERVICE POST - YOU STILL HAVEN'T FIXED IT!
|
|
By:
I'm not sure that I agree Viva with the site stability - more with Lori on that one. I've been around for about five years and seem to remember the site failing from time to time. Having said that, as they say in Operations: "The best you can hope for is a draw". We don't remember when it's going swimmingly, only when it fails.
I do agree with you though: it would be wrong to jump to the conclusion that the failures are because of an underpaid bunch of muppets. There are some things that wouldn't happen if they didn't have the requisite skills. Like you, I lean towards management failings, trying to push through too much change, not tensioning their people correctly, and enforcing tight process. |
|
By:
Getinthere, Brendan
Very good point. Just saw the SLA idea on the forum Chat (I presume it was yours). Personally I'd think four 9s (99.99%) availability is very reasonably achievable. And yes Brendan some transparency in the measurement and reporting required. |