Forums

General Betting

Welcome to Live View – Take the tour to learn more
Start Tour
There is currently 1 person viewing this thread.
Artisan
14 Mar 11 12:49
Joined:
Date Joined: 15 Jun 06
| Topic/replies: 208 | Blogger: Artisan's blog
Reading through the many threads following recent events, culminating in the catastrophic failure on Saturday, it’s clear that there are a lot of savvy and seasoned IT guys on the Forum.  I thought it might be interesting to try and bring together in one thread the cumulative wisdom available on here, and offer Betfair a wealth of free and constructive advice.

Personally, I’m getting a bit old for the game, having being doing IT for over 30 years, and finding increasingly that I’m being over taken by bright young things and developments that my expanding senior moments are finding harder to grasp.  However, even for an old buffer there seem to be a number of obvious things that could be addressed.  I have no doubt those of more agile mind, and up-to-date with the latest thinking and developments, will have better informed views.

This may be wishful thinking, but I’m hoping people will confine their understandable anger to other threads, and concentrate here on dispassionate ideas and proposals.

Although it is clear that there is much to be done with amongst other things their ITIL processes (change, problem, RCA, capacity, demand management, OAT, etc.), and these certainly merit further discussion, I thought I would start with the organisation and management structure.

Looking at the Betfair Board and Executive team profiles, although there are a number of individuals who have done some leading edge technology developments and creations, there appears, on the face of it, little in the way of seasoned operational/service delivery experience.

Most organisations split their IT in some way between Development and Operations (new stuff – with say CTOs and Development Directors in charge, and “Keeping The Lights On” – with say Operations/Service Delivery Directors accountable).  This is done for a reason.  Within any organisation, and particularly so within companies like Betfair that seek rapid growth, there is a natural tension between innovation/developing new stuff to support the business plan to grow revenues with new and innovative products, whilst at the same time maintaining the current products in a way that protects the existing customer base.  Getting this balance right is not always easy.  Sorting out the responsibilities and accountabilities at the top of the shop is a good starting point.

The Operations/Service Delivery Director is measured and incentivised against “Keeping The Lights On”, whilst the Chief Technology Officer (CTO) is measured against the use of innovation/development to support the business plan.  This structure generally brings with it the natural tension that the CTO has to satisfy the Ops Director that proposed changes (the enemy of rock solid operations), and the manner in which those changes are to be made, do not endanger the current operation.  If, for whatever reason, the Ops Director is compelled to make changes against his/her better judgement, he/she will naturally ensure that all are fully aware and understand the risks and the potential consequences.

The fact that the “Apology and an explanation” was signed by the “Chief Marketing & Development Officer” and “Chief Technology Officer” begs the question in my mind: - where’s the operations guy?  It suggests to me that Betfair would benefit from a good look at its senior organisation and a close review of the accountabilities and responsibilities with the intention of getting a better balance between new developments and retaining robust existing products.
Pause Switch to Standard View As an IT professional, what advice...
Show More
Loading...
Report betting_quant March 14, 2011 2:46 PM GMT
Pay better wages, when your payroll consists of interns on peanuts, and experienced staff are offered starting salaries of circa £14k and expected to relocate abroad for this priviledge then you can expect problems.

Also sort out your so-called "talent aquistions managers" they are useless, all they do is spend their time spamming peeople on monster and linkedin. Another problem is that they are unable to read, what part of "not looking for work" are they unable to read ? Even if I was, do they seriously expect me to give up a 6 figure salary to come to betfair and start in an entry position on £14k ?

If you want proof of the "talent" that the TAM's are bringing to betfair, then that is easily measured by the roaring success of the "promotions manager" they recruited for malta.
Report brendanuk1 March 14, 2011 3:02 PM GMT
A's Hire A's and B's Hire C's
Report viva el presidente! March 14, 2011 3:06 PM GMT
don't go into nuclear power.
Report Artisan March 14, 2011 3:08 PM GMT
betting_quant

I'm a bit stunned.  Are they really trying to recruit experienced IT people on £14k?  Surely not!  To do what?  Clean keyboards?
Report Chilly the Dog March 14, 2011 3:09 PM GMT
What job was offering £14k? Is that even a legal wage in the UK?
Report gus March 14, 2011 3:15 PM GMT
1) It's always good to be honest with your customers about the reasons for outages, and to explain what you're doing to avoid future occurrances, so wd BF on that (at last)

2) 15 changes per week to the functionality (as opposed to just updating some text/images)of a 'live' database-driven website  is 14 too many.

3) Fixes applied in the middle of a crash are unlikely to be a long-term solution.

4) Release schedules that are determined by Marketing Departments as opposed to Technical Departments almost always lead to the necessity for retrospective fixes, which lead to further retrospective fixes, which lead to ...
Report Artisan March 14, 2011 3:20 PM GMT
I agree with all of those points Gus.  Picking up specifically on your number 3) - yep.  There seemed to be a lot of wholesale changes taking place during recovery.  I wonder how they were tested.  That of course leads us to the whole area of the testing strategy and execution.
Report DStyle March 14, 2011 3:21 PM GMT
i want to know if their developers, designers, analysts and architects are complaining about the "unnecessary" bureaucracy and "protracted" processes demanded by their Release Manager.

i want to know if the people requesting change are complaining about the "unnecessary" length of regression testing cycles before deployment into the live environment.

if they aren't, then stability isn't a priority.
Report Artisan March 14, 2011 3:26 PM GMT
Dstyle

Well said, as with your questions on the other thread.
Report brendanuk1 March 14, 2011 3:34 PM GMT
The fixes during an outage and short notice down times. tells me their route to getting things onto production is a piece of pish
Report viva el presidente! March 14, 2011 3:37 PM GMT
take a long look at the world cup last year, and try to figure out what went right.
Report Artisan March 14, 2011 3:38 PM GMT
Brendan

Unless there's something we don't know, then that would seem the inevitable conclusion - which would tally with the Bs hiring Cs issue you raised earlier.
Report Artisan March 14, 2011 3:40 PM GMT
Viva

I don't do footy, and wasn't around at that time last year.  What happened?  Capacity issues?
Report viva el presidente! March 14, 2011 3:49 PM GMT
iirc, it was all fine despite some seriously heavy activity.

also iirc, there was a moratorium on releasing anything new in the weeks before and during the competition.
Report Alex the old wrinkled retainer March 14, 2011 4:22 PM GMT
I see that betting_quant is still miffed that betfair thought the fair market value for his expertise and service is £14,000 p.a.
Report gus March 14, 2011 4:46 PM GMT
let's face it, if you're working in London and being paid £14,000 p.a. you'll find it difficult to concentrate on your job for  worrying about where to find a cardboard box to live in.
Report Artisan March 14, 2011 4:49 PM GMT
Gus

We're waiting for betting_quant, or someone else in the know, to provide some provenance for this figure.  Personally, I find it difficult to believe.
Report waynebrayn March 14, 2011 4:52 PM GMT
I think the job was based in Malta.
Report Chilly the Dog March 14, 2011 5:01 PM GMT
well lets be honest, for a random call centre person in Malta who reads from a script, £14k is probably fine given the lower cost of living out there (and all the sun!).
Report betting_quant March 14, 2011 5:03 PM GMT
Alex, I am, although saying that I'm very pi$$ed at anyone in the recruitment industry at the momement, they are absolute vermin, especially the ones in the betting industry.

The job was non IT, but I have enough experience of this industry to know that the pay and conditions in one department is very indicative of those in others.

yep wayne, job was in malta, I was told the cost of living was cheaper out there and thus the wages needed to reflect that.
Report Just Checking March 14, 2011 5:20 PM GMT
It was to be a betfair 'trader', you said so other day, not sure why you're not wanting to say?
Report betting_quant March 14, 2011 5:24 PM GMT
where did I say i didnt want to say ? I do apologise to all on betfair for not responding quick enough on the forum
Report Just Checking March 14, 2011 5:30 PM GMT
I'm not wanting an argument, but you posted something that seemed to imply it was IT, got everybody's attention, then you've just given a three sentence response where you could've settled it simply by saying what it was in a few words, but chose not to clarify it, which was interesting.

I'm guessing entry level BF trader in Malta might pay less than experienced IT bod in London, however.
Report betting_quant March 14, 2011 5:43 PM GMT
ah, sorry JC, just realised my initial post did imply that.


I have no idea what god or bad IT salaries are, but my suspsion is that betfair are towards the bottom end of the pay scale.
Report Artisan March 14, 2011 5:46 PM GMT
I've done a trawl to see if I can find some examples of Betfair's salaries.  All of the job adverts I've seen say "salary negotiable".  Looking at their jobs/careers stuff it looks at first sight to be pretty good.  Training, competitive package, strong values etc.  So unless someone has something more definite to go on, I for one am assuming that it's not salaries that are the issue.

Perhaps we could move on.
Report Alex the old wrinkled retainer March 14, 2011 6:17 PM GMT
There is no evidence whatsoever that betfair pay below market rate.  This is all started because betting_quant was upset with his job offer and now we understand that he is upset with all recruitment consultants.

It is tricky to know whether all recruitment consultants (who are guided by client constraints and economic realities) and betfair are out of line or whether betting_quant is.
Report brendanuk1 March 14, 2011 6:28 PM GMT
The amount of technical roles currently empty would indicate that they are not paying top dollar.
Report subversion March 14, 2011 6:30 PM GMT
trouble is, their disastrous stability record makes speculation such as betting_quants very easy to believe
Report Alex the old wrinkled retainer March 14, 2011 6:39 PM GMT
brendanuk1
The amount of technical roles currently empty would indicate that they are not paying top dollar.

Many organisations advertise jobs that don't exist because they know they have natural wastage and a list of potential future employees is good business sense.  (I used to do it.)

subversion
trouble is, their disastrous stability record makes speculation such as betting_quants very easy to believe


It is best to view such claims with an open mind.  We don't have the facts let alone know what job he was applying for.
Report brendanuk1 March 14, 2011 6:50 PM GMT
Many organisations advertise jobs that don't exist because they know they have natural wastage and a list of potential future employees is good business sense.  (I used to do it.)

thats just silly, do you work in IT?
Report viva el presidente! March 14, 2011 6:51 PM GMT
let's not lose sight of something in all this. the site was actually very stable for many months (including the world cup), so plainly they are technically capable of running a robust, trustable site.

the problems seem linked to changes implemented on a platform that was working fine, and which seem to be of questionable value judging by responses on here. so to me it seems a management/strategic failure more than proof that the IT people aren't up to the job.
Report Lori March 14, 2011 6:53 PM GMT
the site was actually very stable for many months (including the world cup),

I don't recall a week without a crash since about 2006, (I was whining about this before the recent bigger problems)

I do live on the site though tbf
Report viva el presidente! March 14, 2011 6:58 PM GMT
yeah, that's weird lori, because I'm on here loads too and honestly didn't notice anything for nearly a year.

whereas all this recent bullsh!t I've experienced virtually all of firsthand.
Report yeahyeahwhatever March 14, 2011 7:02 PM GMT
brendanuk1
The amount of technical roles currently empty would indicate that they are not paying top dollar.


I can't imagine forcing your CTO to sign a public notice admitting a mammoth screw-up is going to do much for staff morale and future recruitment either!

All the on-going problems can be traced back to QA, either it's all been out-sourced, or management are in such a rush to force out changes that they're not waiting for tests to be completed.
Report getintheir March 14, 2011 7:03 PM GMT
i would like to see a service level agreement set out so we know what their targets are and what our risks maybe.

lets say 99.5% uptime and when a downtime occurs maximum outage of 1 hour (ideally less) to roll back changes / switch to contigency servers.

HOWEVER - EVEN WITH PROPER TESTING , BACKOUT PLANS ETC WE WILL NEVER GET 100% UPTIME WITH ANYONE.  THEREFORE THERE WILL ALWAYS BE RISK FOR TRADERS BUT WE NEED TO BE ABLE TO QUANTIFY THE RISK .
Report brendanuk1 March 14, 2011 7:03 PM GMT
They need third party monitoring of uptime and performance (response times) available to public

Bit of transparency
Report yeahyeahwhatever March 14, 2011 7:04 PM GMT
and for gawd's sake sort out the memory leak on this forum - BECAUSE DESPITE YOUR SERVICE POST - YOU STILL HAVEN'T FIXED IT!
Report Artisan March 14, 2011 7:05 PM GMT
I'm not sure that I agree Viva with the site stability - more with Lori on that one.  I've been around for about five years and seem to remember the site failing from time to time.  Having said that, as they say in Operations: "The best you can hope for is a draw".  We don't remember when it's going swimmingly, only when it fails.

I do agree with you though: it would be wrong to jump to the conclusion that the failures are because of an underpaid bunch of muppets.  There are some things that wouldn't happen if they didn't have the requisite skills.  Like you, I lean towards management failings, trying to push through too much change, not tensioning their people correctly, and enforcing tight process.
Report Artisan March 14, 2011 7:15 PM GMT
Getinthere, Brendan

Very good point.  Just saw the SLA idea on the forum Chat (I presume it was yours). Personally I'd think four 9s (99.99%) availability is very reasonably achievable.  And yes Brendan some transparency in the measurement and reporting required.
Report brendanuk1 March 14, 2011 7:20 PM GMT
99.99 is 1min a week = no chance here, think they would struggle with 3 9s. And if downtime due to upgrades was included they wouldnt do 99%

http://en.wikipedia.org/wiki/High_availability
Report viva el presidente! March 14, 2011 7:23 PM GMT
I'm surprised we've had such different experiences. I remember a period of months when it was pretty bad about 18-15 months ago, then the most troublefree 12 months I'd experienced since day 1 (jun 06), followed by the worst couple of months.
Report Just Checking March 14, 2011 7:27 PM GMT
I was pleasantly surprised it was ok during the world cup, as approaching 3pm on a saturday a few months before that, the site would predictably grind to a juddering halt.
Report Artisan March 14, 2011 7:29 PM GMT
You could well be right Viva.  But people have certainly been reporting performance issues (myself included) since the last big crash.  Thinking back, perhaps it had been relatively trouble free for the previous 12 months or so.  I'll take a look at a couple of sites that are usually quick to report on outages, and see what the history says.
Report Alex the old wrinkled retainer March 14, 2011 7:31 PM GMT
brendanuk1
14 Mar 11 18:50 Joined: 12 Jan 02 | Topic/replies: 14,296 | Blogger: brendanuk1's blog
Many organisations advertise jobs that don't exist because they know they have natural wastage and a list of potential future employees is good business sense.  (I used to do it.)

thats just silly, do you work in IT?



No.  I ran and advised businesses with many employees though.
Report Artisan March 14, 2011 7:36 PM GMT
Brendan

Yes, I'm probably setting the bar too high.  Excluding scheduled downtime do we think 3 9s (99.9% - 10 mins a week) would: a) be realistically achievable, and b) satisfy us (I guess from a risk point of view mainly)?
Report getintheir March 14, 2011 7:46 PM GMT
alex,

do you have list of the tricks of the trade you could share with us?   Mischief

artisan,

i would say 1 outage a month lasting no longer than 1 hour (ideally less). which i guess is around 99.85%
Report brendanuk1 March 14, 2011 7:59 PM GMT
99.9 sounds ok they have a good a chance of failing as of succeeding imo.

Once real time results of third party monitoring is made public they could set up a market. Will Betfair reach its uptime SLA? Yes/No
Report Artisan March 14, 2011 8:05 PM GMT
Viva

I've taken a squint at the BetAngel site forum - they usually pick up on BF outages pretty quickly.  I found the following bits and pieces:


"Mon Jan 31, 2011 11:37 am  (Following scheduled maintenance)
Still erratic at the moment, so please be careful before trying anything. We will be keeping an eye on things but it would be useful if anybody post issues to this thread.

It looks like the site is not fully up at the moment, the odds being returned by the API are all over the place.

Betfair Customer Services 03 Jul 10 13:42
Betfair would like to apologise for the difficulties some customers are experiencing on the website, particularly in respect of the ‘My Bets’ tab and markets prices not updating correctly.

We are investigating the cause of this problem as a matter of extreme urgency and will keep you updated.

Sun Jan 31, 2010 12:36 am
Firstly some music to cheer you up while reading this post, I thought the band kind of suited:

At bang on midnight Betfair, yup you guessed it, died again! Not just the API but Betfair wide. I then spent 10 mins on hold being told all the customer services people were too busy to answer my call so eventually hung up!

Sat Jan 30, 2010 6:59 am
Be careful this morning the API has been all over the shop overnight and is still playing up early this morning."

If that's all there was, then prior to the end of January this year, they've only had two major hits in the preceding 12 months, one in July (what dates were the World Cup?), and one in January last year.  So on that basis, your recollection is more accurate than mine.  Although I suspect all of the niggles (slow browsers, clearing cache, Aus wallets going etc.) won't be recorded, but certainly stick in people's minds.
Report Artisan March 14, 2011 8:09 PM GMT
Very good Brendan.  And given the general accuracy of the BF prices, people should be able to quantify their risk pretty accurately.
Report getintheir March 14, 2011 8:14 PM GMT
and hedge their risks accordingly. we'll all be jumping up and down with delight when there is an outage! Cool
Report Artisan March 14, 2011 8:16 PM GMT
In fact, thinking about it some more, your idea is definitely worth pursuing.  BF should create a market on the chances of the site going down today.  Site users, traders in particular, could hedge against being hit by such an eventuality (effectively take out insurance).  The problem being that the premiums at the moment might be prohibitive.
Report Artisan March 14, 2011 8:21 PM GMT
getinthere

I was posting at the same time as you - obviously we're all thinking along the same lines.
Report getintheir March 14, 2011 8:26 PM GMT
you would of course have to keep adjusting your hedge to current exposure levels. after a month of not having an outage the price may be getting pretty short and expensive to increase your hedge :-)

if im on £14k in IT I could also fund my lifestyle by pulling the plug just after getting a wedge on.
Report frog2 March 14, 2011 8:58 PM GMT
It all seems very disjointed having data and people in Malta, Dublin, Australian, London etc. Did the major problems start when they started the major shift over to Dublin? Is there is a massive pre-occupation with making tax savings and this is forcing sub-optimal technical choices to made? It all seems such a waste of money moving everything just to deny the UK exchequer a few quid given how well Betfair has done over the last decade being based here.
Report brendanuk1 March 14, 2011 9:11 PM GMT
Seems there was a code freeze before going to Dublin, this caused a back log of changes and these were pushed through poorly tested and once problems started to happen they just ploughed on.
Report gus March 14, 2011 9:51 PM GMT
Getting back to the original topic of this thread:

"As an IT professional, what advice would you give Betfair?"

I'd add: There have , in the recent past (another poster mentioned the period before and during last year's World Cup, and i'd agree) , been long periods when the BF (Sports Exchange) site has been stable, reliable, and a joy to use.

What irritates me, and i'd guess quite a lot of other users is that, having achieved that astonishingly (given the demands) enviable  level of stability / reliability, BF seem to have blown it.

Nothing that has been done during recent months has added to the BF (Sports Exchange) experience, whilst some things that have been done detract from it.

The most irritating thing, to this IT prfessional,  is the perversity of sacrificing such genuinly admirable achievement for no apparent gain.

Just stop doing stuff!
Report Just Checking March 14, 2011 10:02 PM GMT
Given the lack of relationship in function between say sports betting and the arcade stuff, if a lot of these new changes are for THAT perhaps they should be hosting that on completely separate servers, e.g. all such servers on arcade.betfair.com, perhaps even with a separate bank like the oz bank so it's all separate. Contain the disease so no upgrade for that has a chance of making the whole house of cards fall down.
Report Chilly the Dog March 15, 2011 11:12 AM GMT
Problem with running markets on exchange failures is that some dodgt russian with a large botnet can probably lump on and then nuke the site. the CIA had terrorism prediction markets for a while but decided it was probably not a good idea for the same reasons.
Report Artisan March 15, 2011 12:58 PM GMT
Gus

Yep, some of the old adages spring to mind, like: "If it ain't broke don't fix it".

Just Checking

I agree that there is definitely something about segregation; getting the systems into manageable bite sized discrete chunks.  We all know that large unwieldy systems are a nightmare to test, diagnose and fix when they snap.  Mission critical stuff is deliberately broken down into as smaller units as possible for this very reason.
Report Artisan March 15, 2011 1:03 PM GMT
Chilly

Good point, but, would the market not naturally price in that risk? Laugh
Report getintheir March 16, 2011 10:51 PM GMT
usually senior management chip in with helpful one liners like "just facking sort it" ..so that would be my advice.
Post Your Reply
<CTRL+Enter> to submit
Please login to post a reply.

Wonder

Instance ID: 13539
www.betfair.com