Computer memory help

I hit a similar problem with excel 2003 as my files reached 150 mb size but in my case no message just the computer hanging up.

I tried expanding the memory from 2gb to 3gb but with the same negative result, I then found there was a limit to excel 2003 memory usage as its capped at 1gb so no matter how much memory you have available excel 2003 can only use 1gb.

Your problem concerns c++builder of which I am not familiar with but probably there is a application memory limitation
that caps the size of certain memory intensive operations, my advice would be to load the csv flies into another application and breakdown the file into more workable sizes both enabling the desired operation and speeding up the pc generally, after the resizing of the files you should be able to load with no further problems.

It's more likely there's a limit to the amount of strings it can import/hold rather than the actual overall size of those strings.

I'm not clued up with C++ but you'll probably find they're are setting somewhere within the program where you can amend/increase the amounts it can handle I think it may be a file called LIMITS.H that'll show any limitations. Plus you may also be limited by the processor's own limits i.e. a 64 bit newer PC will have a lot higher limits than a 32Bit

http://www.festra.com/wwwboard/messages/12989.html

Thanks Joe. The above link would suggest otherwise though. I initially got the same "Out Of Memory" exception for the 144 MB file but was able to process that one after removing a few programmes from memory which would suggest it's not a TStringList problem.

Interestingly I have tried importing the csv historic data files into excel and I get some of the file chopped off due to memory limitation, the files themselves are far to big for average application management and they do not cover all the markets for all sports but this falls inline with betfairs falling short policy for fully maintaining there services to customers as I assume that betfair fails to understand that most exchange users are more sophisticated than the average joe throwing his money into a bookmakers shop or website even though betfair now relies on the same inteligent professional punters that liquidate markets for there profits in the form of premium charging.

historic data we can use (still waiting) betfair starting price result for all sports (still waiting) more contact between the towers and the customers over betfairs direction and policy comitment(still waiting)
best betting exchange in the world, ( got that) oh well 1 out of 4 aint bad.

I like the open source approach to application development and upkeep and perhaps considering the immense work load betfair is under more members should help each other to implement matters concerning information management, afterall exchanges are the only betting sites that allow members to openly culture an edge.

Thanks BTO. Neither helped unfortunately.

I could write code to do that bff. Wish I'd just done it instead of going through all this memory hassle.

bff,

It is not memory causing the issue you are getting re the csv file being chopped off when loading into excel. It is a limitation of excel 2003 where a worksheet can only store 65,536 rows. Excel 2007 onwards can store 1 million rows. Try importing the csv files into an access/MySQL/SQL Server database and they load OK without truncation. One point to note is that if you are loading a lot of csv files into an access database there is an upper file size limit of 2gb for the access.mdb file.

Thanks Joe, I'll take a look but I suspect I'd be better just writing a bit of code and getting it over and done with.

Brendan, thanks but it's not that. The file that's currently causing the problem is the first one to be assigned to the TStringList.

This is highlighting a fundamental flaw in your application's design. What if you wanted to decode and (presumably store in a DB) a 10gig file?

What you want is to do batching during the read. Read 10,000 lines at a time and process them. Since the file is a CSV and each line is independent you don't even care where you do the batching.

If you then convert the parsed CSV to a more efficient memory structure (from strings to ints and timestamps) you then save a huge amount of memory again.

Eg:

A CSV line consists of (this in no way reflects the data you're looking at but it serves the purpose):
2011-11-22T14:33:10.311+0010,2011-11-22T14:35:25.871+0010,5242341,1342524511,B,1.01,546.22,1234.12

This is
-two timestamps at millisecond accuracy (8 bytes as a 64 bit integer or 28 bytes as a string: 3.5x saving)
-two large numbers (8 bytes each as a 64 bit integer or 17 bytes as strings: 1.06x saving)
-one character B - 1 byte - no real saving to be had here.
-three decimal numbers  4 bytes each stored as a 32bt floating point number or 13 bytes as strings: 1.6x saving

Overall you can store it in 41 bytes as native data types , or 87 bytes for the strings, an overall saving of over 50%.

You also speed your program up because you dont need to faff about parsing strings all day, you parse each string once then access it natively in it's binary representation.

I'm not saying lazy, I'm just saying not designed with scalability in mind. As a professional coder I dream this stuff (whether I like it or not..) but I fully appreciate there are people (perhaps you are one) who dont have comp sci degrees and years of experience who just want to get stuff done.

I'm sure I could've done it gooder Chilly but couldn't be @rsed. The thought of having to look up c++ file access functions was enough to put me off.

A fact of life regarding current pc capabilities is that once you start accumulating very large databases of info in what ever form it becomes impossible eventually to work on it with the data present in memory(ram) so the need to load and unload data becomes the mode of operation, this activity in itself then lends to using a programming language to circulate the databases as many database and spreadsheet programs don't cater for this ability.

This increase in I.T. knowledge needed when venturing into information studies is unavoidable unless limiting the
accuracy of data observation via limiting database sizes, it all boils down to accessing how much time and effort you have or can place into such research, as we all have come to learn in this sphere.

Thanks Escapee but I can't find any BlockSize property associated with TStringList.

http://stackoverflow.com/questions/2680635/tstringlist-dynamic-array-or-linked-list-in-delphi

try the above link for your problem is almost identical and the solutions are there posted by programmers in the field.

stackoverflow is quite a good programers forum as they cover most programming langs/platforms and give free advive from begineers to pros.

bff, thanks. I read that all the way through but I don't really see it helps.

I tried the following

for (int i=1; ; i++) StringList->Add(i);

It went out of memory when StringList->Count was 38,873,827

for (int i=1; ; i++) StringList->Add(i*i);

went out of memory when StringList->Count was 31,099,062

which suggests it's not a max number of strings problem.

I think you're right Chilly