Fast Export Method

I have always used the export command to export a dataset to a comma delimited text file. When the files get a little larger (50-100MB) the export function seems to run very slow. Are there other functions that are much faster than the export dataset function?

My dataset is simple (just large). Col 1 is text 2:4 are numeric.

MyDS = dataset(MyData(:,1),MyData(:,2),MyData(:,3),MyData(:,4));
export(MyDS,'file','R:\Equity_Quant\BrianB\Factor Rotation\BulkData.txt','Delimiter',',');

Thanks much, Brian

[per isakson]

I cannot find the function, export. Writing to a text file is slow. Is binary an option to you?

[Walter Roberson]

export() is a method of the dataset class.

http://www.mathworks.com/help/toolbox/stats/dataset.export.html

[per isakson]

Walter, thanks!

Answer 1

[per isakson]

The functions save, load and whos is an alternative

    save( 'my_datasets.mat', 'MyDS' )
    save( 'my_datasets.mat', 'MyDS_2', '-append' )

or even faster

    save( 'my_datasets.mat', 'MyDS', '-v6' )

without guarantee. (Dataset does not overload save as far as I can see.)

--- Faster method to export to text file ---

Some data. I use three columns to avoid word wrap.

    N       = 1e1;
    MyData  = randn( N, 3 );

The variant with dataset

    tic
    ds  = dataset( MyData(:,1), MyData(:,2), MyData(:,3) );
    export( ds, 'file','c:\temp\test_ds.txt', 'Delimiter', ',' )    
    toc

produces this output

    Var1,Var2,Var3
    0.490621219889722,-0.64500446971637,1.6174852368957
    0.0660146644635964,-0.408559384693175,0.445251540387096
    ...

A faster variant

    tic
    ms  = permute( MyData, [ 2, 1 ] );
    fid = fopen( 'c:\temp\test_fp.txt', 'w' );
    fprintf( fid, '%s,%s,%s\n', 'Var1', 'Var2', 'Var3' );
    fprintf( fid, '%f,%f,%f\n', ms );
    fclose( fid );
    toc

produces this output

    Var1,Var2,Var3
    0.490621,-0.645004,1.617485
    0.066015,-0.408559,0.445252
    ...

I've run these two variants with different values for N. The faster variant is at least an order of magnitude faster.

With N=5e6 on my three years old vanilla desktop I get "Elapsed time is 16.478986 seconds." with the faster variant. That is 8 MB/s - something.

How many decimals do you need in the text file?

.

--- fprintf is hard to exceed ---

I added this test

    tic
    dlmwrite( 'c:\temp\test_dlm.txt', MyData )
    toc

With N=5e6 I got the following elapsed times

1. fprintf (A faster variant): Elapsed time is 16.507707 seconds.
2. dlmwrite: Elapsed time is 202.905913 seconds. (without header)
3. dateset.export: Elapsed time is 819.649789 seconds.

With plain Matlab I don't think there is a faster alternative. Maybe it is possible to do something faster with a MEX-function.

[Brian]

Those are saving .mat files through aren't they? That doesn't do me a lot of good if I am going to be reading them with a database engine to import the data. If I include the 'ascii' parameter I get the message that it can't write an unspported data tye to the ASCII file.

[per isakson]

That's correct. I didn't realize the purpose. Maybe it is an option to write directly to the database.

[Brian]

Since the database is on a server separate from my local machine, my job is running some calculations via matlab and exporting the large files to the database server so that I can do a bulk import via the SQL server. Hence the need for exporting the data. Would fprintf be my fastest option for writing to a text file?

Thanks a lot,

Brian