I have a 2 csv files one with 2.5gb and one with 450mb. I am using datastore to upload these function and need something like "outer join" to merge them. Any ideas?

I have a 2 csv files one with 2.5gb and one with 450mb. I am using datastore to upload these function and need something like "outer join" to merge them. Any ideas?

Answer 1

[Pratyush Swain]

Hi Rathies,

You can utilise the 'datastore' and 'outerjoin' functions to achieve this. Please refer to this example implementation:

% Create datastore objects for both CSV files
ds1 = datastore('file1.csv');
ds2 = datastore('file2.csv');

% Convert to tall tables
t1 = tall(ds1);
t2 = tall(ds2);

% Perform outer join
mergedTable = outerjoin(t1, t2, 'MergeKeys', true);

Datastores enable you to work with large data sets in small blocks that individually fit in memory, instead of loading the entire data set into memory at once. Tall arrays extend this capability to enable you to work with out-of-memory data using common functions.

For more information, please refer to following documentation links:

1- https://www.mathworks.com/help/matlab/import_export/tall-arrays.html

2- https://www.mathworks.com/help/matlab/ref/table.outerjoin.html