Cannot import large .dat files

Dear All,

I have some huge data files, containing around 15000000x2 matrices in seperate files. That is, one of my files contains a 15000000x2 matrix. When I import it into Matlab using importdata, only a fraction of it is imported. It seems there is a problem with the memory, but could anyone can help me to solve this?

I am using Dell latitude with 8GB memory, core i5.

Cheers

[per isakson]

    >> 15000000*2*8/1e6
    ans =
       240

That is 0.24GB, which shouldn't be a problem. At least not with a 64bit Matlab.

"only a fraction of it is imported" make me believe there is a problem with the file.

You must provide more detail. Error message. The ten first lines of the file.

[Batuhan]

Nope, there are no errors in the file. At least I can open it with vim.

Answer 1

[per isakson]

Comments:

• "[...]can open it with vim." That is not a sufficient requirement! There could be a special string indicating a missing numerical value in a line in the middle of the file. There could be a spurious non-printing character somewhere. Possibly, vim understands more text encodings than does importdata of Matlab.
• AFAIK: importdata analyzes (auto-magically) the beginning of the file to find out the format and delegates to an appropriate function. There might be a line which don't honor the format of the lines in the beginning.
• If you know the format it's better to use textscan.
• Wasn't there an error message?
• You must provide more detail. Error message. The ten first lines of the file.

[Batuhan]

I work on both my laptop and a remote computer for our calculations. I can import the whole data without any loss in the remote computer, but in my laptop the data is truncated. So I believe there is no problem with the data itself, because I also use it as an input for my perl scripts and no error is echoed.

In my laptop, no, I do not receive any error messages. Importing starts and the finalizes without any error, as if it is running OK.

Below are the first lines of the file

 0.02490588  -0.00045087
 -0.73235485   0.41171917
 -1.42063660   0.14335885
 -2.16042932  -1.16072912
 -1.90909512  -1.74951097
 -0.77953409   0.36309451
  0.21492125   1.87915631
  0.94789485   0.48371599
  0.81793610  -1.77439386
  0.52971379  -1.38098016
 -0.09243455  -0.10711347
 -0.53392565  -0.06699097
 -0.74784156  -0.84150355
 -1.06228134   0.83152723
 -0.27937340   1.52853082
  1.12518669   0.04544050
  1.15507537  -1.02233988
  1.67200132   1.44003002
  2.89863746   1.73281089
  2.91482059  -0.03389061
  1.41054370   0.82257573
  0.99218804   2.19622207
  0.68712796   0.79233637
  0.70598239   0.68184931
  0.30360987   1.86329137
  0.01083112   0.55908539
 -1.65630446   1.88928356
 -1.67732210   1.76437025
 -0.75000307  -0.21528305
 -0.45777996  -0.76524868

[per isakson]

Strange! I don't know. Guessing:

• Something happened when you copied the file to the laptop. Did you check the size of the file on the different computers?
• AFAIK: there is no default value or preference, which could cause this.
• With importdata one may set "Range:". Did you by mistake use an old value?

However, there are many Matlab function, which should read that file, e.g. textscan and load. (The flexibility of textscan is not needed.)

    >> fid = fopen( 'cssm.txt', 'r' ); % see doc for different encodings
    >> buf = textscan( fid, '%f%f', inf, 'CollectOutput', true );
    >> fclose(fid);

    >> M = load( 'cssm.txt', '-ascii' );

where cssm.txt contains the data from your comment (copy&paste).

You might test your file for spurious characters. Read as characters and search for characters other than, digit, dot, "white space", plus and minus. Empty cac indicates that none was found.

    >> str = fileread( 'cssm.txt' ); 
    >> cac = regexp( str, '[^\d\.\s\+\-]', 'match' ); 
    >> whos
      Name       Size             Bytes  Class     Attributes

      M         30x2                480  double              
      ans        1x1                  8  double              
      buf        1x1                592  cell                
      cac        0x0                  0  cell                
      fid        1x1                  8  double              
      str        1x807             1614  char

[Batuhan]

Dear Per,

Thank you for advice. Indeed 'load' function works much better and it loads the whole file without loss.

I am using Matlab on Linux Mint 13, maybe the problem is related with that.

The bottom line is; load function is better. Thank you for your help.

Batuhan