Processing Multiple CSV files one at a time and plotting them on the same graph/

I wrote a code which takes in two CSV files and synchronizes their data and plots it on a graph. With bigger files, it shows a memory issue and the program does not run. I have tried to split the big CSV file into multiple files and I want the program to read it one by one (not storing the data since doing this also lead to program failure with memory overload). I have attached my very first code (which worked fine for smaller CSV files) and the second code that I modified for larger memory files but I still cant seem to run it.

INITIAL CODE

%Initial Code

%reading and modifying the ring Data
sleeponData = readtable('x.csv');
sleeponTimestamp = sleeponData(:,"time");
sleeponTimestamparray = table2array(sleeponTimestamp);
sleeponSportRaw = table2array(sleeponData(:,"sportRaw"));
sleeponSportRaw = 5000*sleeponSportRaw; %use plotyy left for sensor y for ring

%converting epoch time to datetime

sleep_on_dt = datetime(sleeponData.time, 'ConvertFrom','epochtime','TicksPerSecond',1e3, 'Format', 'dd-MM-yyyy HH:mm:ss.SSSSSSSSS','TimeZone','+09:00');
sleepon_dt_shifted = sleep_on_dt + hours(9);
sleeponData.time = sleepon_dt_shifted;

%reading the pillow
data = importdata('sensordata.csv');
data(:,2) = [];
baseline = data(1,:);
baseline(1) = 0;

minus_bl = data - baseline;

minus_bl(end,:) = [];


tabledata = array2table(minus_bl);


%pillowTime = array2table(data(:,1));

sleeponTT = table2timetable(sleeponData);

pillow_time = datetime(tabledata.minus_bl1, 'ConvertFrom','posixtime','Format', 'dd-MM-yyyy HH:mm:ss.SSSSSSSSS','TimeZone','Asia/Tokyo');
tabledata.data1 = pillow_time;

pilowTT = table2timetable(tabledata);

%synchronizing the timestamp
%new = synchronize(sleeponData, data, 'secondly','linear');
new = synchronize(sleeponTT, pilowTT);

% plotyy(new.time, new.sportRaw, new.time, new{:,3:end});
figure
hold on
x = new.time;
y = new.sportRaw;
yyaxis right;
plot(x,y,'-x','Color', [.9 .4 1], 'markersize',15 );
ylabel('Movement');
title('Participant x');


z = new{:,3:end};
yyaxis left;
plot(x,z, '-');
ylabel('Sensor Readings');

MODIFIED CODE BELOW:

%MODIFIED CODE

%reading and plotting the Ring Data
sleeponData = readtable('sleepring.csv'); %small file
sleeponTimestamp = sleeponData(:,"time");
sleeponTimestamparray = table2array(sleeponTimestamp);
sleeponSportRaw = table2array(sleeponData(:,"sportRaw"));
sleeponSportRaw = 5000*sleeponSportRaw; %use plotyy left for sensor y for ring
%converting epoch time to datetime
sleep_on_dt = datetime(sleeponData.time, 'ConvertFrom','epochtime','TicksPerSecond',1e3, 'Format', 'dd-MM-yyyy HH:mm:ss.SSSSSSSSS','TimeZone','+09:00');
sleepon_dt_shifted = sleep_on_dt + hours(9);
sleeponData.time = sleepon_dt_shifted;
%reading the multiple sensor files
data = [];
filename = "sensordata_%i.csv";
figure
title('Sensor Data 1');
hold on
for i = 1:100;
    new_filename = sprintf(filename, i);
    if isfile(new_filename)
        newdata = importdata(new_filename);
        newdata(:,2) = [];
        data = [data; newdata];
        baseline = data(1,:);
        baseline(1) = 0;
        
        minus_bl = data - baseline;
        
        minus_bl(end,:) = [];
        
        tabledata = array2table(minus_bl);
        sleeponTT = table2timetable(sleeponData);
        pillow_time = datetime(tabledata.minus_bl1, 'ConvertFrom','posixtime','Format', 'dd-MM-yyyy HH:mm:ss.SSSSSSSSS','TimeZone','Asia/Tokyo');
        tabledata.data1 = pillow_time;
        
        pilowTT = table2timetable(tabledata);
        new = synchronize(sleeponTT, pilowTT);
        x = new.time;
        y = new.sportRaw;
        yyaxis right;
        plot(x,y,'-x','Color', [.9 .4 1], 'markersize',15 );
        ylabel('Movement');
        
        z = new{:,3:end};
        yyaxis left;
        plot(x,z, '-');
        ylabel('Sensor Readings');
    else
        break;
    end
end

[Eric Sofen]

I'm not sure what's wrong with your new code. It will be easier to diagnose with some sample data.

That said, this sounds like a situation wheretall arrays and tables might be useful. They use lazy evaluation to work with data that doesn't fit in memory.

[R]

Thank you so much for your suggestion! I have tried to use tall tables and arrays but I still get an error.

Error using matlab.graphics.chart.primitive.tall.Line/doPlot

Tall inputs must be real column vectors.

Error in matlab.graphics.chart.primitive.tall.Line

Error in tall/plot (line 77)

htemp = matlab.graphics.chart.primitive.tall.Line('XData', x, 'YData', y, ...

Error in MovementAnalyzer (line 63)

plot(x,z, '-');

Below is the sample data for 'examplesensordata.csv'. It has a total of 49 columns. The actual file consists of over 3 million rows.

NEW CODE WITH TALL ARRAYS AND TABLES

%reading ring data

sleeponData = readtable('sleepring.csv');

sleeponTimestamp = sleeponData(:,"time");

sleeponTimestamparray = table2array(sleeponTimestamp);

sleeponSportRaw = table2array(sleeponData(:,"sportRaw"));

sleeponSportRaw = 5000*sleeponSportRaw; %use plotyy left for sensor y for ring

%converting epoch time to datetime

sleep_on_dt = datetime(sleeponData.time, 'ConvertFrom','epochtime','TicksPerSecond',1e3, 'Format', 'dd-MM-yyyy HH:mm:ss.SSSSSSSSS','TimeZone','+09:00');

sleepon_dt_shifted = sleep_on_dt + hours(9);

sleeponData.time = sleepon_dt_shifted;

%reading sensor file as datastore

ds = datastore("examplesensordata.csv");

ds.VariableNames(1) = {'Time'};

%converting datastore to a tall table and removing the second column

tt_sleep = tall(ds);

T2 = removevars(tt_sleep,2);

T2_array = table2array(T2);

%reading baseline value

baseline = readtable("examplesensordata.csv");

baseline(:,2)= [];

baseline_value = baseline(1,:);

baseline_value_array = table2array(baseline_value);

baseline_value_array(1) = 0;

minus_bl = T2_array - baseline_value_array;

%removing the last row (cleaning data)

%minus_bl(end,:) = [];

%converting to tall

tabledata = array2table(minus_bl);

sleeponTT = table2timetable(sleeponData);

pillow_time = datetime(tabledata.minus_bl1, 'ConvertFrom','posixtime','Format', 'dd-MM-yyyy HH:mm:ss.SSSSSSSSS','TimeZone','Asia/Tokyo');

tabledata.data1 = pillow_time;

pilowTT = table2timetable(tabledata);

%synchronizing the timestamp

new = synchronize(sleeponTT, pilowTT);

new_clean = rmmissing(new);

figure

hold on

x = new_clean.time;

y = new_clean.sportRaw;

yyaxis right;

plot(x,y,'-x','Color', [.9 .4 1], 'markersize',15 );

ylabel('Movement Intensity by Ring');

title('Participantx');

z = new_clean{:,3:end};

yyaxis left;

plot(x,z, '-');

ylabel('Sensor Readings');