General Betting
There is currently 1 person viewing this thread.
I have had a few databases written to strip various bits from a couple of websites and fancy trying to write one of my own.
I wrote some code and managed to grab the bits of text I wanted with an incomplete program, where I was running to a break point.
When I tried to put the thing together, some of the getElementsByTagName bits that had previously grabbed bits of text ceased to work.
The plan was to grab the forecast prices to go with my ratings. So, starting from http://www.attheraces.com/tomorrowsracing/ I would first grab the racelinks e.g. http://www.attheraces.com/card.aspx?raceid=692426&meetingid=52194&date=2012-04-04&ref=fixtures&refsite=&nav=racecards and then grab the racetime, course and forecast SPs.
The grabbing of the links works fine. (NavigateAndWait is another bit of code that loads the webpage)
Set x = New HTMLDocument
x.body.innerHTML = NavigateAndWait("http://www.attheraces.com/tomorrowsracing/")
Set a = x.getElementsByTagName("span")
If a(i).className = "details" Then
s = "http://www.attheraces.com/" & Mid((a(i).parentNode), 8, 300)
(grabs http://www.attheraces.com/card.aspx?raceid=692426&meetingid=52194&date=2012-04-04&ref=fixtures&refsite=&nav=racecards )
and I managed to grab "14:00 Lingfield (A.W.), 4 Apr 2012" from that page. (I did have it working using the classname race_title, but that has given up the ghost now as well.)
set a = x.getElementsByTagName("h3")
s = a(i).innerText
but I can't seem to grab the forecast prices. The below worked in a previous bit of code, but not any more. Have tried many alternatives, without success.
Set a = x.getElementsByTagName("div")
If a(i).className = "forecast" Then
s = a(i).innerText
Can anyone offer some assistance?
I wrote some code and managed to grab the bits of text I wanted with an incomplete program, where I was running to a break point.
When I tried to put the thing together, some of the getElementsByTagName bits that had previously grabbed bits of text ceased to work.
The plan was to grab the forecast prices to go with my ratings. So, starting from http://www.attheraces.com/tomorrowsracing/ I would first grab the racelinks e.g. http://www.attheraces.com/card.aspx?raceid=692426&meetingid=52194&date=2012-04-04&ref=fixtures&refsite=&nav=racecards and then grab the racetime, course and forecast SPs.
The grabbing of the links works fine. (NavigateAndWait is another bit of code that loads the webpage)
Set x = New HTMLDocument
x.body.innerHTML = NavigateAndWait("http://www.attheraces.com/tomorrowsracing/")
Set a = x.getElementsByTagName("span")
If a(i).className = "details" Then
s = "http://www.attheraces.com/" & Mid((a(i).parentNode), 8, 300)
(grabs http://www.attheraces.com/card.aspx?raceid=692426&meetingid=52194&date=2012-04-04&ref=fixtures&refsite=&nav=racecards )
and I managed to grab "14:00 Lingfield (A.W.), 4 Apr 2012" from that page. (I did have it working using the classname race_title, but that has given up the ghost now as well.)
set a = x.getElementsByTagName("h3")
s = a(i).innerText
but I can't seem to grab the forecast prices. The below worked in a previous bit of code, but not any more. Have tried many alternatives, without success.
Set a = x.getElementsByTagName("div")
If a(i).className = "forecast" Then
s = a(i).innerText
Can anyone offer some assistance?
sort by:
Show
per page
Replies: 26
|
By:
I pull data off RP by parsing the DOM, never used ATR though so here goes:
Working off http://www.attheraces.com/card.aspx?raceid=692426&meetingid=52194&date=2012-04-04&ref=fixtures&refsite=&nav=racecards 2pm Lingfield This is how I understand things, you're using | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
[attempt #2 - didn't like all those tags first time around...]
I pull data off RP by parsing the DOM, never used ATR though so here goes: Working off http://www.attheraces.com/card.aspx?raceid=692426&meetingid=52194&date=2012-04-04&ref=fixtures&refsite=&nav=racecards 2pm Lingfield This is how I understand things, you're using h3 to grab race details. So before you pull the prices, are you also pulling runner ids (tr id="oddsrow_xxxxx")? I'm not sure if you're just after the first price, say span id="odd_xxxxxx_13" or if you want all prices for a particular horse? Won't you have an issue with not knowing which odds go with which horse though until you check every span id="odd_xxxxx_13" string? For what it's worth, I'd get a list of runnerIDs and then just search for all span id="odd_runnerID" strings using find/instr/mid whatever to get all odds from all bookmakers. A simpler approach may well be to use web queries and dump to Excel. Although it adds a few seconds onto each page's processing time, it's often much easier to find info on an Excel sheet. You can search and move around the sheet - it's certainly easier to find things that way, just locate the top horse and then use rngX.offset(X, X) to process all horses. You could also use rngX.hyperlinks(1) to pull the url of any links. I'll have a play though, see if I can get it all working with tags. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
ah... I didn't your post properly did I. damn. Are you using div to pull the forecast string, ie
Forecast: 6/4 Heartsong, 4/1 Subtle Knife, 9/2 Auntie Mabel, 5/1 Vexillum, Betty Brook I think you are... whereas I thought you wanted to pull all bookmakers odds from the table at the top of the screen. Looking at the following bit of your code: Set a = x.getElementsByTagName("div") If a(i).className = "forecast" Then s = a(i).innerText what happens when you run this - do you get errors? what do you get if you print out s? Where do you reset/increment i? thanks | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
Ah, could this be it, when you pull
x.body.innerHTML = NavigateAndWait("http://www.attheraces.com/tomorrowsracing/") and then, as you say, you use that to get an individual race-card... you are then pulling the innerHTML of that new page aren't you... because if you don't, you're never to find your forecast string - it's not there to be found. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
i dont have any experiance of data collection from websites using vba or any other programming languge, so if my question is naive sorry. i just checked the website mentioned in the discussion. do u look for a particular information using text analysis on the basic html code or does vba have a level of abstraction for the html code where u can stracture the html code(in a way similar to webbrowsers) and look for data methodically? or do the websites mentioned above have api
would be nice if someone can suggest me references on programming related to working with data from websites | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
You can get forecast data from Timeform using information from here http://forum.bdp.betfair.com/showthread.php?t=340
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
yeahyeahwhatever, It's as you described, The idea is to grab a an individual race link, use that web address to grab the date, race time, course and betting forecast (all as one text string) for all the races. I don't actually need to increment i, as each race has one forecast. (That just came over from the other program I started from.
The s = a(i).innerText shows nothing for s. (I'm using the Local window) and a break point after the line s = a(i).innerText I had it working once, but have tried many combinations of innertext/outertext/innerhtml/outerhtml/firstchild/siblings etc without being able to grad the forecast again. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
colonelll, it's not just vba, other computing languages (.NET, java, PHP) can also be used.)
I have had a few databases written. The 1st few used regular expressions. The web pages were downloaded, then the text was parsed with regular expressions. (which are complex) The method of navigating the DOM appears a lot more elegant and easier to understand. DOM introduction http://www.w3schools.com/dom/dom_intro.asp http://www.quirksmode.org/dom/intro.html How to Snatch HTML Using Visual Basic Code http://www.developer.com/services/article.php/3113371/How-to-Snatch-HTML-Using-Visual-Basic-Code.htm I'm tied to MS Access, as I have towards 50 or so databases. (horses, Greyhounds, Football, NFL) I am good once I have the data is databases, but am pretty poor at coding. One of the things I did design, was my own web based form system. All the horse pages link to the race pages. I have lots of stats & links, including links that play all the horse videos. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
rn-betfair, thanks for that, I'll take a look.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
ok Stow, nothing like fiddling around with code to stop it from working...
 You have NavigateAndWait("http://www.attheraces.com/tomorrowsracing/") which brings up that page, is that the only NavigateAndWait call you have? I would expect to see another one underneath the following bit of code: If a(i).className = "details" Then s = "http://www.attheraces.com/" & Mid((a(i).parentNode), 8, 300) (grabs http://www.attheraces.com/card.aspx?raceid=692426&meetingid=52194&date=2012-04-04&ref=fixtures&r... ) Something along the lines of x.body.innerHTML = NavigateAndWait(s) msgbox "value of i is " & i Set a = x.getElementsByTagName("div") If a(i).className = "forecast" Then s = a(i).innerText msgbox s We're just printing out i for reference - not important to the running of the code at all. You should now have the forecast string (stored in s) displayed in a pop-up messagebox. fingers crossed, that should do it. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
Thanks, I'll take a look this evening.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
colonell - sJudge has given you some good links there, but if you've never programmed before this is not a project for a beginner, not by any means. Although the DOM does indeed provide an elegant solution, it's not an easy thing to grasp for a novice.
You might want to look at using web queries to dump a web page onto an Excel sp and then searching for the info you want using VBA within Excel. Put a shout out if you want more help. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
thanks guys will have a llook on DOM as i have a few things in mind. yeahyeah had a bit of programming experiance but thanks for ur concern
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
strURL = "http://www.attheraces.com/card.aspx?raceid=692458&meetingid=52199&date=2012-04-05&ref=fixtures&refsite=&nav=racecards"
Set x = New HTMLDocument x.body.innerHTML = NavigateAndWait(strURL) Set a = x.getElementsByTagName("h3") s = a(i).innerText Set y = New HTMLDocument y.body.innerHTML = NavigateAndWait(strURL) MsgBox "value of s is " & s t = "test" MsgBox "value of t is " & t Set b = y.getElementsByTagName("div") If b(j).className = "forecast" Then t = b(j).innerText MsgBox "value of t is " & t End If ----------------------------------------------------------------- Break point set at End If Code runs S shown as "16:25 Clonmel (IRE), 5 Apr 2012" in the locals window Message box shows correct value of s ("16:25 Clonmel (IRE), 5 Apr 2012") I do not need a For next loop, though if I change s = a(i).innerText to s = a.innerText the code falls over and gives an error objecting to "s = a.innerText" (error 438 Oject doesn’t support this property or method) Do I need to reload the page as a new document? I think I tried with & without, without success in either case. I set t to equal “test” and it is show in the message box. The code runs to the break point (End If) and the line - MsgBox "value of t is " & t Does not produce a message box at all (???) and also the value of t is still “test” | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
A few things worth trying:
1) Has the page completely finished loading?... many webpages now use Ajax etc which dynamically load content. I would suggest manually checking the whole page html (from your prog) versus the "get source" from a real webbrowser. Check they are the same. If you're using a webbrowser the document_loaded type events are very cumbersome and don't work well for dynamic loading. You may end up with a loop that keeps checking the page has fully loaded before checking content ;) 2) your value for j is this correct? You should be looping through the objects of b maybe you are but not presented your code? | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
*You may end up having to code a loop that keeps checking the page has fully loaded before checking content ;)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
Getafix is correct in what he says about the page finishing loading, but as you have a NavigateandWait sub - which was clearly written by someone how knows what they're doing - and you're now "modifying"
 this code to try and get it to do something new.... I don't think you need worry about it. Please tell us you haven't changed anything in NavigateAndWait  let's try again, the variable s is not going to hold the forecast string because your revised code is not using the variable s in the way your previous example did! strURL = "http://www.attheraces.com/card.aspx?raceid=692458&meetingid=52199&date=2012-04-05&ref=fixtures&... Set x = New HTMLDocument x.body.innerHTML = NavigateAndWait(strURL) Set a = x.getElementsByTagName("h3") s = a(i).innerText Set y = New HTMLDocument y.body.innerHTML = NavigateAndWait(strURL) MsgBox "value of s is " & s t = "test" MsgBox "value of t is " & t Set b = y.getElementsByTagName("div") If b(j).className = "forecast" Then t = b(j).innerText MsgBox "value of t is " & t End If Your first NavigateAndWait statement is correct, but your second NavigateAndWait is pointless because it uses the same strUrl, and strUrl hasn't changed. The reason I told you that you needed to bring in another NavigateAndWait was because you needed one to bring up all of tomorrow's races, and then another one to bring up an individual race. Did you type in the code example I gave you - did that run? The problem(s) with the latest code you've posted is that you've introduced variable j - have you declared that or was it already in the program? What's the value of j? I think the reason you may not be seeing the final msgbox is that j is an invalid index and so that piece of code is just ignored. Also, do you know the data type of b? SJ, what you're trying to do is admirable, but I do think this may be a little bit too much for you. If you want to send me your code and I'll fix it and add comments so you can follow what's happening. I can't see us fixing it by posting on here because you're changing the code in each reply. My email is detectivefireplace@hotmail.co.uk | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
Thanks very much lads, I'll send you an e-mail yeahyeahwhatever.
I have not changed the NavigateAndWait statement. Your distinction between my coder and me was very accurate. One of us knows what we are doing wrt vb!  I thought the 2nd page load was probably pointless, but when something does not work, it's sometimes worth guessing a different method.I was not trying to get the whole thing working at this stage (that failed earlier!), just the grabbing of the date, racetime, course and the betting forecast from a single page. (2 fields) I previously managed to grab the racelinks ok separately. I did declare j as well as i. I don't believe that I need a For next loop, nor i or j, but as I said, if I turn s = a(i).innerText into s = a.innerText, I get the above mentioned error. I am just grabbing two fields off each race page. The 1st one works, but the grabbing the forecast prices does not. (It did once)  I think I tried setting i and j to zero and that was another bit of code guessing that didn't get me anywhere.  I'm not sure I know what you mean by the data type of b. I declared it in the same way from some previous code. I'll either have to spend more time reading my Access 2000 VBA handbook (Susann Novalis) or go on that course I was thinking of. http://www.microsofttraining.net/microsoft-access-training-london-2000-vba.php | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
notice the s in getElementsByTagName - this is plural. If you look at the docs you will see it returns an array/collection (you'll have to check as I can't rememver. I would assume there are many div tags on that webpage and that is why you have to cycle until you get to the one you want.
I.e., b = y.getElementsByTagName("div") for i = 0 to b.length -1 'or b.count - 1? If b(j).className = "forecast" Then t = b(j).innerText next i something along those lines. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
argh
b = y.getElementsByTagName("div") for j = 0 to b.length -1 'or b.count - 1? If b(j).className = "forecast" Then t = b(j).innerText next j | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
btw I don't program in vb but this should give you the jist!
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
Thanks v much. I did originally have the code as
For i = 0 To a.length - 1 but kept getting an error from this bit of code. It did not occur to me that this was cycling through the div nodes, I naively thought that the classname bit would just snap to the right bit of html like magic! | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
if it still errors what is the error message... Guesses: do vb indexes still start at 1? Or maybe one of the div's doesn't have a className and thus you have to check if that property exists or not because looking at it? Could stick a "try catch" around it if you want to be lazy.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
*Or maybe one of the div's doesn't have a className and thus you have to check if that property exists or not *before* looking at it?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
Much to my amazement and I'm sure some of yours! , I have got it sorted. (or very close to it)
I think I was mixing up some a and b's and x and y's in the code (from my copy & paste of repeating bits of code) and also the position of the loops were key. I don't understand all the code, particularly the actual adding of the data to the table just yet. I have it working, but don't understand it. If anyone could explain the code below to me, I'd be most grateful. I'll post a bit of the table in my next post for all the doubters.  I'm sure I'll have more questions to come! Set r = CurrentDb.OpenRecordset("Race") With r .AddNew strVerdict = a(i).innerText If Len(strVerdict) Then .fields("Verdict").Value = strVerdict End If .Update strVerdict = vbNullString | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
By:
|