Computer memory help

I hit a similar problem with excel 2003 as my files reached 150 mb size but in my case no message just the computer hanging up.

I tried expanding the memory from 2gb to 3gb but with the same negative result, I then found there was a limit to excel 2003 memory usage as its capped at 1gb so no matter how much memory you have available excel 2003 can only use 1gb.

Your problem concerns c++builder of which I am not familiar with but probably there is a application memory limitation
that caps the size of certain memory intensive operations, my advice would be to load the csv flies into another application and breakdown the file into more workable sizes both enabling the desired operation and speeding up the pc generally, after the resizing of the files you should be able to load with no further problems.

Thanks bff. Having just done a bit of googling I think you may be correct. Splitting the files would be more hassle than re-coding though.

It's more likely there's a limit to the amount of strings it can import/hold rather than the actual overall size of those strings.

I'm not clued up with C++ but you'll probably find they're are setting somewhere within the program where you can amend/increase the amounts it can handle I think it may be a file called LIMITS.H that'll show any limitations. Plus you may also be limited by the processor's own limits i.e. a 64 bit newer PC will have a lot higher limits than a 32Bit

What a wasted day. Upgrading the memory involved removing the keyboard (not easy) and it turned out my memory consisted of 1 x 2 GB and not the 2 x 1 GB I thought I had. This means I've also got to return one of the 2 x 2 GB memory modules.

http://www.festra.com/wwwboard/messages/12989.html

Thanks Joe. The above link would suggest otherwise though. I initially got the same "Out Of Memory" exception for the 144 MB file but was able to process that one after removing a few programmes from memory which would suggest it's not a TStringList problem.

I don't think there are any memory contraints with TStringList other than available memory Feck, it'll most likely be down to the number of strings that'll be limited either within some settings in the program or the processors limitations . Do you know how many records the 144mb and 155Mb held?

Interestingly I have tried importing the csv historic data files into excel and I get some of the file chopped off due to memory limitation, the files themselves are far to big for average application management and they do not cover all the markets for all sports but this falls inline with betfairs falling short policy for fully maintaining there services to customers as I assume that betfair fails to understand that most exchange users are more sophisticated than the average joe throwing his money into a bookmakers shop or website even though betfair now relies on the same inteligent professional punters that liquidate markets for there profits in the form of premium charging.

I don't know the number of records Joe. I could find out, but not without coding.

historic data we can use (still waiting) betfair starting price result for all sports (still waiting) more contact between the towers and the customers over betfairs direction and policy comitment(still waiting)
best betting exchange in the world, ( got that) oh well 1 out of 4 aint bad.

You think that a message issued by Microsoft saying "Out of Memory" means you are out of memory?

I like the open source approach to application development and upkeep and perhaps considering the immense work load betfair is under more members should help each other to implement matters concerning information management, afterall exchanges are the only betting sites that allow members to openly culture an edge.

Feck, couple of things you can try.

CTRL ALT DEL start windows task manager.
Right click your Process and select Priority realtime befaore executing the program.

Also your virtual memory could be changed to double your physical ram instead of windows managed size.
Both these fixes in conjunction should fix the problem unless it is an actual limitation of the program.

cheers.

Thanks BTO. Neither helped unfortunately.

I am pretty confident the answer to your particular problem will be to load historic data in csv format into some app that will then allow you to seperate the big file into smaller files for better manipulation, someone will have the answer soon

I could write code to do that bff. Wish I'd just done it instead of going through all this memory hassle.

I've used this csv splitter before feck, very simple and works fine. Might be worth using until you either find some fix or end up having to rewrite some of your code

http://www.fxfisherman.com/downloads/csv-splitter-1.1.zip

bff,

It is not memory causing the issue you are getting re the csv file being chopped off when loading into excel. It is a limitation of excel 2003 where a worksheet can only store 65,536 rows. Excel 2007 onwards can store 1 million rows. Try importing the csv files into an access/MySQL/SQL Server database and they load OK without truncation. One point to note is that if you are loading a lot of csv files into an access database there is an upper file size limit of 2gb for the access.mdb file.

might be worth a shot

http://www.delphibasics.co.uk/Article.asp?Name=Memory

// Free up the string list memory
   FreeAndNil(fileData);

when you dont need it anymore, incase some unused are hanging around

Thanks Joe, I'll take a look but I suspect I'd be better just writing a bit of code and getting it over and done with.

Brendan, thanks but it's not that. The file that's currently causing the problem is the first one to be assigned to the TStringList.

thanks dogform for the tip will look into sql/access as soon as I finish with my current vb/api project, just wish I had the free time I had when I didnt have a family to look after , not complaining though.

This is highlighting a fundamental flaw in your application's design. What if you wanted to decode and (presumably store in a DB) a 10gig file?

What you want is to do batching during the read. Read 10,000 lines at a time and process them. Since the file is a CSV and each line is independent you don't even care where you do the batching.

If you then convert the parsed CSV to a more efficient memory structure (from strings to ints and timestamps) you then save a huge amount of memory again.

Eg:

A CSV line consists of (this in no way reflects the data you're looking at but it serves the purpose):
2011-11-22T14:33:10.311+0010,2011-11-22T14:35:25.871+0010,5242341,1342524511,B,1.01,546.22,1234.12

This is
-two timestamps at millisecond accuracy (8 bytes as a 64 bit integer or 28 bytes as a string: 3.5x saving)
-two large numbers (8 bytes each as a 64 bit integer or 17 bytes as strings: 1.06x saving)
-one character B - 1 byte - no real saving to be had here.
-three decimal numbers  4 bytes each stored as a 32bt floating point number or 13 bytes as strings: 1.6x saving

Overall you can store it in 41 bytes as native data types , or 87 bytes for the strings, an overall saving of over 50%.

You also speed your program up because you dont need to faff about parsing strings all day, you parse each string once then access it natively in it's binary representation.

I agree it was lazy coding Chilly but I haven't found a cure yet for the laziness.

I'm not saying lazy, I'm just saying not designed with scalability in mind. As a professional coder I dream this stuff (whether I like it or not..) but I fully appreciate there are people (perhaps you are one) who dont have comp sci degrees and years of experience who just want to get stuff done.

I'm not saying lazy, I'm just saying not designed with scalability in mind. As a professional coder I dream this stuff (whether I like it or not..) but I fully appreciate there are people (perhaps you are one) who dont have comp sci degrees and years of experience who just want to get stuff done.

I'm sure I could've done it gooder Chilly but couldn't be @rsed. The thought of having to look up c++ file access functions was enough to put me off.

get couple of "interns" preferably with big tits and get them to type it in you know it makes sense

A fact of life regarding current pc capabilities is that once you start accumulating very large databases of info in what ever form it becomes impossible eventually to work on it with the data present in memory(ram) so the need to load and unload data becomes the mode of operation, this activity in itself then lends to using a programming language to circulate the databases as many database and spreadsheet programs don't cater for this ability.

This increase in I.T. knowledge needed when venturing into information studies is unavoidable unless limiting the
accuracy of data observation via limiting database sizes, it all boils down to accessing how much time and effort you have or can place into such research, as we all have come to learn in this sphere.

Are you specifying the 'BlockSize' in the TStringList Constructor ?

If not then there's a possibility that its starting with a small block size ( memory allocation ) and its getting bigger for each allocation ( it could be doubling or some other exponential formula ). Thus causing it to request
a excessively large memory allocation.


If its not got anything to do with initial BlockSize then I would look at 'Handles'. Its likely that the inerds of TStringList use handles to pass
arround pointers to the string value. And there is a limit to how many handles
a process can create (and indeed a system wide limit as well).

If you run out of available slots for Handles then you will get an out of memory message, even if its not out of memory. ( same is true for various resources like gdi, timers etc )

Its a long time since I looked at this issue, and it was on an old windows ME pc.
I think it is possible to increase the number of system wide handle slots available. using some utility in control panel ( can't remember how to do this off the top of my head )

Thanks Escapee but I can't find any BlockSize property associated with TStringList.

what version of Visual Studio are you using ? ( or is it Borland C ? )

I didn't find a lot of documentation on TStringList, most of it said "use the
documentation for TObjList"

to set the Block size try this:

int nBlockSize = 1024000;
TStringList myList = new TStringList( nBlockSize )

my documentation doesn't specify if Blocksize is actual memory allocation or number of list entries. So you might have to factor it up or down abit to get an optimal size for your app.

http://stackoverflow.com/questions/2680635/tstringlist-dynamic-array-or-linked-list-in-delphi

try the above link for your problem is almost identical and the solutions are there posted by programmers in the field.

It's embarcadero rad studio Escapee (used to be borland). I tried it but no matter how big I made the constructor parameter I still got the out of memory message at LoadFromFile.

stackoverflow is quite a good programers forum as they cover most programming langs/platforms and give free advive from begineers to pros.

excuse the typos

bff, thanks. I read that all the way through but I don't really see it helps.

I'd bite the bullet and do the parsing manually. Reading lines from a file in C++ is really easy: get onto google for fully working examples.

I tried the following

for (int i=1; ; i++) StringList->Add(i);

It went out of memory when StringList->Count was 38,873,827

for (int i=1; ; i++) StringList->Add(i*i);

went out of memory when StringList->Count was 31,099,062

which suggests it's not a max number of strings problem.

there was a reference to the limiatation of T-string memory sizes over 150mb plus and better aproachs for huge file sizes, just thought it might help.

I think you're right Chilly

In that sense it did bff. I was meaning from an easy answer point of view.

I'm getting round this by having the csv files read into a MySQL table using LOAD DATA INFILE. It gives me a fair amount of work to convert the code but it will probably end up faster and simpler to maintain. Thanks to all who took the trouble to offer a bit of help.

what would help you feck and everybody else in the future is if betfair created better and more manageable datafile
for post events so that those that need to can achieve there goals better after all we all know they don't want
thousands of users having to record live markets for them selves as its not good for there server bandwidth management so its no good betfair dishing out half hearted survives when they know it will effect them in the future.

Lets hope also they don't become user unfriendly towards data hungry customers in fear of spawning super users netting the top profits in the future as they now have better control of these profits via there premium charges so this helps to maintain the ethos they have bought from the 2 whizz kids that set betfair up.

There market lead in exchange and sports betting comes only from there distinct adoption of this model and to dilute it away towards general bookmaking would leave them with no MARKETABLE PRODUCT/SERVICE advantage.

Having the data available to all in order to save everyone collecting the same data would be a sensible thing bff. Betfair make the second by second data available through Fracsoft but charge so much for it that there's loads of bots collecting the same data. It's no wonder the site is forever crashing as they've even created a pirate market for the data.

If betfair did the maths and thought about things positively, if more powered punters were assisted in data sourcing
and application advantages then a new wave of big winners means more liquidity, more new markets, more commission charges, more premium charges, more big players playing against more big players at betfairs advantage, bigger turnover figures and profit reports means betfair gets more global foothold and respect and better share interest.

Wont stop them being able to have other new products and ideas similar to 1bet or sports books as long as they clarify the effect to consistent winners but I reckon most pros wont use sportsbooks when there is exchange prices
but sports books will help bring in money from average punters that want a fractional bet similar to wills and lads
without complication.

Betfair have to keep doing whats worked for them for so long and embracing the new vision for betting that they have pioneered solely, they are responding more to customers questions now and its all positive, they've even got a new boss coming who knows betting inside out which is very promising.

Betfairs biggest challenge in my view is to appeal to all types of bettors and cater for them all which is a hard task but one with a huge potential in the global betting playing field.


They also need staff that can deliver not just there time but empower betfair into more success and improvement.

"They also need staff that can deliver not just there time but empower betfair into more success and improvement."

Do you create generic corporate powerpoint presentations?

no, its just an opinion on a forum that caters for opinion, we are now six miles off the thread topic .

No problem with being off topic.
Virtually every other thread I've read involving Feck on this forum has been hijacked by him onto his obsession and I think it's fair to say actual hatred of "ferengi" - e.g. traders. Or to put it another way, people who try to make an honest quid on a betting exchange in a different way than Bishop Feck of the betting morality inquisition does, and therefore approves of.

It would be rude to the memory of all the other threads he's ruined to stay on topic on one of his .

ferengi's first baby steps are to download and import data for analysis

Don't let Cardinal Fang find out, he'll have Bishop Feck in a comfortable chair ASAP for such rumoured heresy.

lol

perhaps it should say on the betfair tin now, you can win bigger and easier on betfair with trading and better odds but please dont as your mis-using the site

If betfair did the maths and thought about things positively, if more powered punters were assisted in data sourcing and application advantages then a new wave of big winners means more liquidity, more new markets, more commission charges, more premium charges, more big players playing against more big players at betfairs advantage, bigger turnover figures and profit reports means betfair gets more global foothold and respect and better share interest.

Doubt it would mean more liquidity bff as it's mainly used by Ferengi w@nks like Just Bitching and copy cats. Far better for punters if none of it was available and everything bar the odds were hidden.

Feck,

Returning to your post about bulk loading the csv files into MySQL is there any chance you could share the code or procedure (PM/email if you wish)? My access database are creaking with the strain from the csv files and I keep meaning to switch to MySQL but for the pain of loading all the historic csv files individually.

"Doubt it would mean more liquidity bff as it's mainly used by Ferengi w@nks like Just Bitching and copy cats."

Just Bitching? That's quite rich. You're complaining about things and advocating the Feck Morally Approved (tm) way to use Betfair be forced upon everyone on every thread you touch.

feck sits on the moral high ground because he says so. Your all fookers bitches and ferengi if you stop him making money

I doubt any of you balloons cost me money. It's the imbecilic parasite friendly interface that costs me in the shape of reduced liquidity.

dogform, create the table with

CREATE TABLE  `bfData` (
  `SPORTS_ID` smallint(5) unsigned DEFAULT NULL,
  `EVENT_ID` int(10) unsigned DEFAULT NULL,
  `SETTLED_DATE` char(19) DEFAULT NULL,
  `COUNTRY` char(30) DEFAULT NULL,
  `FULL_DESCRIPTION` char(80) DEFAULT NULL,
  `COURSE` char(20) DEFAULT NULL,
  `SCHEDULED_OFF` char(19) DEFAULT NULL,
  `EVENT` char(50) DEFAULT NULL,
  `ACTUAL_OFF` char(19) DEFAULT NULL,
  `SELECTION_ID` int(10) unsigned DEFAULT NULL,
  `SELECTION` char(35) DEFAULT NULL,
  `ODDS` float DEFAULT NULL,
  `NUMBER_BETS` int(10) unsigned DEFAULT NULL,
  `VOLUME_MATCHED` float DEFAULT NULL,
  `LATEST_TAKEN` char(19) DEFAULT NULL,
  `FIRST_TAKEN` char(19) DEFAULT NULL,
  `WIN_FLAG` tinyint(3) unsigned DEFAULT NULL,
  `IN_PLAY` char(2) DEFAULT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

Import using

LOAD DATA INFILE 'FileName'
INTO TABLE bfData
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LNES TERMINATED BY '\n'
IGNORE 1 LINES;

Some of the files have to be doctored to accommodate the LOAD DATA criteria though.

nice and offensive Feck, still going to charm school