Reading large data

[onem]

Dear Tom,

I have very large datasets (about 3,000,000 KB each) in CSV format. This is an intraday US Treasury data.

RATS cannot read the data. I have the following message:

OPEN DATA C:\Users\Onem\Documents\RATSdata\trtwoyrtrans.csv
DATA(FORMAT=PRN,ORG=OBS)
## M4. A memory request for an additional -795082722 bytes cannot be satisfied

Is there a way to increase the memory of RATS?

Best regards,

Onem

[TomDoan]

The 64 bit Pro isn't constrained to 2Gb of memory---it can use virtual memory to several Tb. However, you might try putting in the specific number of observations on the DATA instruction.

[onem]

Hi Tom,

Thank you for your quick reply. As you mentioned, the problem is not the memory. I have the same message even with the first 500 observations.

OPEN DATA C:\Users\Onem\Documents\RATSdata\trtwoyrtrans.csv
DATA(FORMAT=PRN,ORG=OBS) 1 500
## M4. A memory request for an additional -795082722 bytes cannot be satisfied

Excel opens the data fine though. I am not sure what else can be the reason.

Regards,

Onem

[TomDoan]

Excel uses virtual memory (regardless of the operating system) to open particularly large files. It helps when the same company writes both the software and the operating system. If the files themselves have several Gb of text, then you would need to upgrade to WinRATS Pro and use the 64 bit executable. Are you using all that data in the same program?

[onem]

Tom,

I rearranged the data to make it smaller. It is now 460,000 KB, and still RATS give me the same error message.

OPEN DATA C:\Users\Onem\Documents\RATSdata\trtwoyrtrans.csv
DATA(FORMAT=PRN,ORG=OBS)
## M4. A memory request for an additional -793247792 bytes cannot be satisfied

Should I upload the data so you can see what the problem is?

Thank you,

Onem

[TomDoan]

Zip it and send it to support@estima.com. I don't think the forum software can take a file that large.

[onem]

Tom,

I sent the zipped file. Is it possible that one of the names of my variables is "Time"?

Thank you,

Onem

[TomDoan]

No. That's just a truly massive data set. It's not very big (in a relative sense) as a CSV file because there are so many 0 and 1 values. However, each of those expands to 8 bytes as a real, and when RATS is first trying to get it organized, each one of those values comes with about another 50-60 bytes for keeping track of everything. (Excel, BTW, does the same). All told, it creates a memory image of about 12 Gb. Once upon a time, RATS was written to better handle a simple data stream like what you have (though not of that magnitude), but we put in the process of building the memory image of the data set to take care of all the weird things that can happen in different data formats. For instance, the information in an Excel spreadsheet can be saved in almost any order, not just by rows. With some tweaks to the program (actually some major rewrites), we were able to process it to a RATS format file, which can be read without the overhead.