General Betting
Welcome to Live View – Take the tour to learn more
Start Tour
There is currently 1 person viewing this thread.
01 Apr 13
12:48
Joined:
Date Joined:
17 Sep 02
I have not heard of this term before. Is it something to do with using a bot?
"We would like to apologise for the Betfair exchange being unavailable on Saturday from 16:23 to 16:52 and later from 20:55 to 21:16. We disabled betting after web site speed was impacted by unusual and high-rate traffic from automated crawlers. Our decision to disable betting across all channels was in support of our Customer Commitment to maintain a level playing field for our customers.
In response we have implemented several changes to the web site since Saturday which will allow automated high loads to be managed more efficiently. We would advise any customers that are using automated crawlers on the web site to switch to using our API service as in future this type of activity may be blocked.
We apologise for the inconvenience caused and we do take these incidents very seriously. "
"We would like to apologise for the Betfair exchange being unavailable on Saturday from 16:23 to 16:52 and later from 20:55 to 21:16. We disabled betting after web site speed was impacted by unusual and high-rate traffic from automated crawlers. Our decision to disable betting across all channels was in support of our Customer Commitment to maintain a level playing field for our customers.
In response we have implemented several changes to the web site since Saturday which will allow automated high loads to be managed more efficiently. We would advise any customers that are using automated crawlers on the web site to switch to using our API service as in future this type of activity may be blocked.
We apologise for the inconvenience caused and we do take these incidents very seriously. "
Show More
surely when betfair take the website down of there own accord they could give some sort of warning(1 min maybe) allowing players to exit positions.
Web crawler
From Wikipedia, the free encyclopedia
For the search engine of the same name, see WebCrawler.
Not to be confused with offline reader.
A Web crawler is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing.
A Web crawler may also be called a Web spider,[1] an ant, an automatic indexer,[2] or (in the FOAF software context) a Web scutter.[3]
Web search engines and some other sites use Web crawling or spidering software to update their web content or indexes of others sites' web content. Web crawlers can copy all the pages they visit for later processing by a search engine that indexes the downloaded pages so that users can search them much more quickly.
Crawlers can validate hyperlinks and HTML code. They can also be used for web scraping (see also data-driven programming).
Contents [hide]
1 Web crawling
2 Selection policy
2.1 Focused crawling
2.1.1 Restricting followed links
2.1.2 URL normalization
2.1.3 Path-ascending crawling
2.1.4 Academic-focused crawler
2.2 Re-visit policy
2.3 Politeness policy
2.4 Parallelisation policy
3 Architectures
4 Crawler identification
5 Examples
5.1 Open-source crawlers
6 Crawling the deep web
6.1 Web crawler bias
7 See also
8 References
9 Further reading
[edit]Web crawling
A Web crawler starts with a list of URLs to visit, called the seeds. As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl frontier. URLs from the frontier are recursively visited according to a set of policies.
The large volume implies that the crawler can only download a limited number of the Web pages within a given time, so it needs to prioritize its downloads. The high rate of change implies that the pages might have already been updated or even deleted.
The number of possible crawlable URLs being generated by server-side software has also made it difficult for web crawlers to avoid retrieving duplicate content. Endless combinations of HTTP GET (URL-based) parameters exist, of which only a small selection will actually return unique content. For example, a simple online photo gallery may offer three options to users, as specified through HTTP GET parameters in the URL. If there exist four ways to sort images, three choices of thumbnail size, two file formats, and an option to disable user-provided content, then the same set of content can be accessed with 48 different URLs, all of which may be linked on the site. This mathematical combination creates a problem for crawlers, as they must sort through endless combinations of relatively minor scripted changes in order to retrieve unique content.
As Edwards et al. noted, "Given that the bandwidth for conducting crawls is neither infinite nor free, it is becoming essential to crawl the Web in not only a scalable, but efficient way, if some reasonable measure of quality or freshness is to be maintained."[4] A crawler must carefully choose at each step which pages to visit next.
The behavior of a Web crawler is the outcome of a combination of policies:[5]
a selection policy that states which pages to download,
a re-visit policy that states when to check for changes to the pages,
a politeness policy that states how to avoid overloading Web sites, and
a parallelization policy that states how to coordinate distributed web crawlers.
From Wikipedia, the free encyclopedia
For the search engine of the same name, see WebCrawler.
Not to be confused with offline reader.
A Web crawler is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing.
A Web crawler may also be called a Web spider,[1] an ant, an automatic indexer,[2] or (in the FOAF software context) a Web scutter.[3]
Web search engines and some other sites use Web crawling or spidering software to update their web content or indexes of others sites' web content. Web crawlers can copy all the pages they visit for later processing by a search engine that indexes the downloaded pages so that users can search them much more quickly.
Crawlers can validate hyperlinks and HTML code. They can also be used for web scraping (see also data-driven programming).
Contents [hide]
1 Web crawling
2 Selection policy
2.1 Focused crawling
2.1.1 Restricting followed links
2.1.2 URL normalization
2.1.3 Path-ascending crawling
2.1.4 Academic-focused crawler
2.2 Re-visit policy
2.3 Politeness policy
2.4 Parallelisation policy
3 Architectures
4 Crawler identification
5 Examples
5.1 Open-source crawlers
6 Crawling the deep web
6.1 Web crawler bias
7 See also
8 References
9 Further reading
[edit]Web crawling
A Web crawler starts with a list of URLs to visit, called the seeds. As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl frontier. URLs from the frontier are recursively visited according to a set of policies.
The large volume implies that the crawler can only download a limited number of the Web pages within a given time, so it needs to prioritize its downloads. The high rate of change implies that the pages might have already been updated or even deleted.
The number of possible crawlable URLs being generated by server-side software has also made it difficult for web crawlers to avoid retrieving duplicate content. Endless combinations of HTTP GET (URL-based) parameters exist, of which only a small selection will actually return unique content. For example, a simple online photo gallery may offer three options to users, as specified through HTTP GET parameters in the URL. If there exist four ways to sort images, three choices of thumbnail size, two file formats, and an option to disable user-provided content, then the same set of content can be accessed with 48 different URLs, all of which may be linked on the site. This mathematical combination creates a problem for crawlers, as they must sort through endless combinations of relatively minor scripted changes in order to retrieve unique content.
As Edwards et al. noted, "Given that the bandwidth for conducting crawls is neither infinite nor free, it is becoming essential to crawl the Web in not only a scalable, but efficient way, if some reasonable measure of quality or freshness is to be maintained."[4] A crawler must carefully choose at each step which pages to visit next.
The behavior of a Web crawler is the outcome of a combination of policies:[5]
a selection policy that states which pages to download,
a re-visit policy that states when to check for changes to the pages,
a politeness policy that states how to avoid overloading Web sites, and
a parallelization policy that states how to coordinate distributed web crawlers.
Thanks, I did look it up on Wikipedia but my brain clouded over after the first few sentences. Curious what it means in a Betfair sense.
Its a bit odd, I cant see why they dont just block them, and tell them to use the API.
They might have a problem identifying them, but I would have thought it strange that 1 "rogue" "web crawler" was able to bring down the site? If it is multiple "web crawlers" then that is almost a DDOS and would have thought Betfair would be totally within rights to ban/block them.
Doesnt really make much sense to me, sounds bollox. Just the term "web crawler" sounds like something from 2002
They might have a problem identifying them, but I would have thought it strange that 1 "rogue" "web crawler" was able to bring down the site? If it is multiple "web crawlers" then that is almost a DDOS and would have thought Betfair would be totally within rights to ban/block them.
Doesnt really make much sense to me, sounds bollox. Just the term "web crawler" sounds like something from 2002
Some conspiracy theorists believe there are intentional attempts to bring Betfair down by overloading the site. These people believe they target the weekend before large race meetings (e.g. Cheltenham, Aintree).
I have no idea whether they're tin foil hat people or on to something.
I have no idea whether they're tin foil hat people or on to something.
The site crashed because of Betfair's incompetence, they like to scapegoat 'bots' the same way the Government use immigrants.
there is specialist software that can prevent distributed denial of service DDOS attacks. I wouldnt think any large company has a problem with them now, they would be more of an inconvenience than anything.
.
http://www.telstra.com.au/business-enterprise/download/document/business-betfair-case-study.pdf
In response we have implemented several changes to the web site since Saturday
Just seems odd that they are vulnerable to "web crawlers" on friday and then after several unplanned changes over a bank holiday weekend everything is tickety boo today
.
http://www.telstra.com.au/business-enterprise/download/document/business-betfair-case-study.pdf
In response we have implemented several changes to the web site since Saturday
Just seems odd that they are vulnerable to "web crawlers" on friday and then after several unplanned changes over a bank holiday weekend everything is tickety boo today
The site crashed because of Betfair's incompetence, they like to scapegoat 'bots' the same way the Government use immigrants.
seems better explanation to me
seems better explanation to me
brendanuk1 01 Apr 13 14:12 Joined: 12 Jan 02 | Topic/replies: 21,726 | Blogger: brendanuk1's blog
Its a bit odd, I cant see why they dont just block them, and tell them to use the API.
They might have a problem identifying them, but I would have thought it strange that 1 "rogue" "web crawler" was able to bring down the site? If it is multiple "web crawlers" then that is almost a DDOS and would have thought Betfair would be totally within rights to ban/block them.
Doesnt really make much sense to me, sounds bollox. Just the term "web crawler" sounds like something from 2002
It's very difficult to block them. This is a decade old problem for Betfair:
http://www.guardian.co.uk/sport/2002/sep/26/horseracing.gregwood
Betfair over the years have taken action:
*Created an API
*Implemented data request and transaction charges (fighting the bots putting up tiny bets all over the place doing very little volume, or even doing nothing other than collecting data)
*Made the website price info slower when viewed when not logged in.
Betfair just need to be able to handle the load on weekends.
It will be a big embarrassment for Betfair if it goes down again during the Grand National.
Obviously very difficult for betfair, Investor. 10 years on and they're still searching for that missing link as to why the site grinds down/crashes around 3pm every Saturday 
I don't know what the odds of Betfair crashing due to "automated crawlers" are but I would take any 1.01 about Betfair crashing due to future "site enhancements" to prevent "automated crawlers"
Betfair just need to be able to handle the load on weekends.
-----------
this is the bit I'm not sure about. when it crashed a couple of saturdays ago there was nothing major on. the four mid-table 3pm EPL games probably accounted for less traffic than the manu-real game earlier in the week.
unless what it is is several people all using screen scrapers to try to archive pre-3pm price moves across all the leagues? but would that really generate enough load to crash the site?
-----------
this is the bit I'm not sure about. when it crashed a couple of saturdays ago there was nothing major on. the four mid-table 3pm EPL games probably accounted for less traffic than the manu-real game earlier in the week.
unless what it is is several people all using screen scrapers to try to archive pre-3pm price moves across all the leagues? but would that really generate enough load to crash the site?
Not just Betfair though is it. Bet365 has regularly crashed on Champions League nights right at 7.45pm.
there is specialist software that can prevent distributed denial of service DDOS attacks. I wouldnt think any large company has a problem with them now, they would be more of an inconvenience than anything.
mtgox, the main bitcoin exchange, is suffering for ddos attacks at the moment. They don't seem to be coping with them. The attackers crash the site, this causes panic selling and the bitcoin exchange rate drops. They then buy bitcoins and wait for a recovery.
mtgox, the main bitcoin exchange, is suffering for ddos attacks at the moment. They don't seem to be coping with them. The attackers crash the site, this causes panic selling and the bitcoin exchange rate drops. They then buy bitcoins and wait for a recovery.
Post Your Reply
Please login to post a reply.
Wonder
Instance ID: 13539