New Posts For Thread: Live ‘Site Performance’ forum chat – Thursday 10th February 6pm GMThttps://community.betfair.com/forum_chat/go/thread/view/94078/26924457/live-site-performance-forum-chat-thursday-10th-february-6pm-gmtLast Saturday saw the worst outages of the Betfair exchange in years. We are sorry for the disruption to you, our customers, over the weekend and are determined not to let it happen again. Our teams have remedied the problems that led to the outagesOkay, we're drawing the session to a close - thanks for taking part.This sort of feedback is incredibly helpful to us as we are always working hard to improve things for our customers. We really appreciate you taking the time to join us this evening.https://community.betfair.com/forum_chat/go/thread/view/94078/26924457/live-site-performance-forum-chat-thursday-10th-february-6pm-gmt?post_id=485447965#485447965Okay, we're drawing the session to a close - thanks for taking part.This sort of feedback is incredibly helpful to us as we are always working hard to improve things for our customers. We really appreciate you taking the time to join us this evening.Thu, 10 Feb 2011 13:11:17 -0600What was the cause of Saturday’s crash?On 31 January we moved to a new datacentre and used that opportunity to significantly upgrade our infrastructure. Saturday's outage was the result of a misconfiguration that became apparent during a period ofhttps://community.betfair.com/forum_chat/go/thread/view/94078/26924457/live-site-performance-forum-chat-thursday-10th-february-6pm-gmt?post_id=485446977#485446977What was the cause of Saturday’s crash?On 31 January we moved to a new datacentre and used that opportunity to significantly upgrade our infrastructure. Saturday's outage was the result of a misconfiguration that became apparent during a period ofThu, 10 Feb 2011 13:02:18 -0600Why when the site is malfunctioning do you not restrict access to everyone? It seems grossly unfair that customer A can't login to cancel their bets while customer B, using a betting program via the API, is still allowed to use the site and take theihttps://community.betfair.com/forum_chat/go/thread/view/94078/26924457/live-site-performance-forum-chat-thursday-10th-february-6pm-gmt?post_id=485446457#485446457Why when the site is malfunctioning do you not restrict access to everyone? It seems grossly unfair that customer A can't login to cancel their bets while customer B, using a betting program via the API, is still allowed to use the site and take theiThu, 10 Feb 2011 12:57:26 -0600"While I appreciate that it is not always possible to test for every eventuality, particularly simulating full production load, your performance following system upgrades over the years would seem to indicate you haven’t got this under control. Whhttps://community.betfair.com/forum_chat/go/thread/view/94078/26924457/live-site-performance-forum-chat-thursday-10th-february-6pm-gmt?post_id=485445701#485445701"While I appreciate that it is not always possible to test for every eventuality, particularly simulating full production load, your performance following system upgrades over the years would seem to indicate you haven’t got this under control. WhThu, 10 Feb 2011 12:50:21 -0600There have been numerous occasions where I've rung Customer Service up to tell them Betfair has been down. They've had no idea. So, they check it and say "erm ... yeah, you're right". Isn't there an "alarm" that goes off when the site goes down?Wehttps://community.betfair.com/forum_chat/go/thread/view/94078/26924457/live-site-performance-forum-chat-thursday-10th-february-6pm-gmt?post_id=485445193#485445193There have been numerous occasions where I've rung Customer Service up to tell them Betfair has been down. They've had no idea. So, they check it and say "erm ... yeah, you're right". Isn't there an "alarm" that goes off when the site goes down?WeThu, 10 Feb 2011 12:46:13 -0600The server is going down while you carry out "tests". This morning's lasted over 10 minutes (categorically NOT "1 or 2 minutes”). Why are you carrying out tests on the live system? Why don't you have a test system? Sometimes you can only learn so mhttps://community.betfair.com/forum_chat/go/thread/view/94078/26924457/live-site-performance-forum-chat-thursday-10th-february-6pm-gmt?post_id=485444845#485444845The server is going down while you carry out "tests". This morning's lasted over 10 minutes (categorically NOT "1 or 2 minutes”). Why are you carrying out tests on the live system? Why don't you have a test system? Sometimes you can only learn so mThu, 10 Feb 2011 12:42:57 -0600Will whatever you are intending to do in terms of new features shorten the delay between real live and what you call 'live', compared to what the delay is now?We try and make our site as quick as possible, especially where prices and transaction prochttps://community.betfair.com/forum_chat/go/thread/view/94078/26924457/live-site-performance-forum-chat-thursday-10th-february-6pm-gmt?post_id=485444521#485444521Will whatever you are intending to do in terms of new features shorten the delay between real live and what you call 'live', compared to what the delay is now?We try and make our site as quick as possible, especially where prices and transaction procThu, 10 Feb 2011 12:39:37 -0600Is the risk posed by site instability (and the effect instability has on your customers) due to upgrades always quantitatively justified by the benefits the upgrade brings?We spend a lot of time planning upgrades and investment in the site and our prhttps://community.betfair.com/forum_chat/go/thread/view/94078/26924457/live-site-performance-forum-chat-thursday-10th-february-6pm-gmt?post_id=485444229#485444229Is the risk posed by site instability (and the effect instability has on your customers) due to upgrades always quantitatively justified by the benefits the upgrade brings?We spend a lot of time planning upgrades and investment in the site and our prThu, 10 Feb 2011 12:36:37 -0600Can you confirm if you know the specific reason why the site crashed multiple times last Saturday?We have identified the issue and we've been working all week to make sure our solution is robust.https://community.betfair.com/forum_chat/go/thread/view/94078/26924457/live-site-performance-forum-chat-thursday-10th-february-6pm-gmt?post_id=485444177#485444177Can you confirm if you know the specific reason why the site crashed multiple times last Saturday?We have identified the issue and we've been working all week to make sure our solution is robust.Thu, 10 Feb 2011 12:36:12 -0600Why on earth don't you have a back-up in place so that when the main server goes down you can switch to the back-up?We have a number of back-up options in case of server issues but in the case of the outages on Saturday, we know they weren't relatedhttps://community.betfair.com/forum_chat/go/thread/view/94078/26924457/live-site-performance-forum-chat-thursday-10th-february-6pm-gmt?post_id=485443793#485443793Why on earth don't you have a back-up in place so that when the main server goes down you can switch to the back-up?We have a number of back-up options in case of server issues but in the case of the outages on Saturday, we know they weren't relatedThu, 10 Feb 2011 12:32:20 -0600