Ideal way to import csv data and create column vectors that will be variables i want to work with

Hello,

I have a .csv file that i wish to import and create column vecotrs (variables) that i can manipulate further with my code. Below is my code. I also wish to remove NaN values from all columns that have them. In additon i want to muliply all of the EMG data columns by 1000 which is why that number is there towards the end of my code. This code seems to work but i was told to not use eval in code writing and just wanted to see if someone can check it and offer any other suggestions to code it differently.

Thank you in advance!

filename='RESP_Trial_3.csv';
column_names={'RLT','RLTx','RLTy','RLTz','RUT','RUTx','RUTy','RUTz','RIC','RICx','RICy','RICz',...
    'RPEC','RPECx','RPECy','RPECz','RPS','RPSx','RPSy','RPSz','RD','RDx','RDy','RDz',...
    'RRA','RRAx','RRAy','RRAz','ROB','ROBx','ROBy','ROBz','Analog21','Analog22','Analog23','Sync'};
EMGVariables={'RLT','RUT','RIC','RPEC','RPS','RD','RRA','ROB','Sync'};
%read the imported matrix table and create data struct to store variables
%Create variables in the workspace with the same names as column names and
%assign the corresponding columns of the matrix as their values
matrix = readmatrix(filename,'HeaderLines', 7);
data = struct();
for i = 1:numel(column_names)
    variable_name = column_names{i};
    assignin('base',variable_name, matrix(:,i));
end
%Remove NaN
for i = 1:numel(EMGVariables)
    variable_name = EMGVariables{i};
    variable = 1000*eval(variable_name);
    variable(isnan(variable))=[];
    assignin('base', variable_name, variable);
end

[Stephen23]

"This code seems to work but i was told to not use eval in code writing..."

EVAL and ASSIGNIN and various others:

https://www.mathworks.com/matlabcentral/answers/304528-tutorial-why-variables-should-not-be-named-dynamically-eval

"...and just wanted to see if someone can check it and offer any other suggestions to code it differently"

Use a table.

Answer 1

[dpb]

filename='RESP_Trial_3.csv';
column_names={'RLT','RLTx','RLTy','RLTz','RUT','RUTx','RUTy','RUTz','RIC','RICx','RICy','RICz',...
              'RPEC','RPECx','RPECy','RPECz','RPS','RPSx','RPSy','RPSz','RD','RDx','RDy','RDz',...
              'RRA','RRAx','RRAy','RRAz','ROB','ROBx','ROBy','ROBz','Analog21','Analog22','Analog23','Sync'};
EMGVariables={'RLT','RUT','RIC','RPEC','RPS','RD','RRA','ROB','Sync'};
opt=detectImportOptions(filename);                  % get an import options object
opt.MissingRule='omitrow';                          % skip any incomplete rows
opt.ImportErrorRule='omitrow';                      % ditto any bad conversions...
tData=readtable(filename,opts,'HeaderLines', 7);    % and bring in the table
tData.Properties.VariableNames=column_names;        % and name variables as desired
tData{:,EMGVariables}=1000*tData{:,EMGVariables};   % and scale given subset by variable names

As the above illustrates, you can dynamically address any/all variables by name without there being a million different variables nor using very slow and difficult to debug eval expressions by the simple expedient of using a table.

Using the import object also lets you clean the table of any bad data on import with the important feature missing in the previous that the data as independent variables before was not necessarily consistent in length/location between the samples -- a missing value in one was not being compensated for in another column.

[dpb]

BTW, one might presume that the variable names are part of the header lines in the .csv file? If that's the case, then the code can be made fully generic with respect to the input file(s), using those data to set the variable names instead of having to hard/hand code them into the m-file itself. Always try to separate data from code if at all possible...

[Ines Shekhovtsov]

Thank you for your answer. The .csv is a bit of a complicated file and has two rows where variable names are listed. The EMG variable names are one row and gyroscope variable names are in another. I ran your code and was expecting to see the created variables in my workspace. I am a beginner coder in matlab and i think i tend to code very rudamentary by creating all variables in the beginning and then manipulating them throughout rest of code. I want to import the column data from that .csv file (data starts at line 7) and name all of the columns and create variables in my workspace. The array column_names is what i want my variables to be called. Thank you.

[dpb]

The above does what you want except the variables are named in the table -- a MUCH better way to handle multiple variables that go together. You can address them inside the table by their name just as were individual variables except you can now also do a lot more with the builtin features of MATLAB for tables.

Venture forth and learn the power of MATLAB; don't hamstring yourself by only walking when there's a powerful motorcycle right there for you to ride...

If you would attach a section of a sample file, it would then be possible to see how to fully automate the script.

If you would then outline the tasks to be done with the data, undoubtedly somebody would illustrate "the MATLAB way" to accomplish the job making use of the reasons for having MATLAB to begin with...

[dpb]

"...t wanted to see if someone can ... any other suggestions to code it differently"

What is the point of asking if aren't going to be willing to accept the advice offered?

[Ines Shekhovtsov]

EDIT: Seems i cannot attach the file as it is too large. Can you please let me know where i can drop the file in here? Thank you

Thank you, yes i would of course love to utilize the full potential of Matlab. I will upload the sample file here for you to see. I will outline what i am trying to do with the code below. I have written most of the code other than smoothing gyroscope data. It works yet i'm sure that i could have written it in a much more elegant and easier way.

First about the .csv file. Data starts on line 8. It contains EMG data, gyroscope data in x, y and z directions and a sync pulse column that helps us synchronize events with data. This data was collected using Trigno Discover software. EMG is sampled at 1925.9259Hz, Gyro data at 74.0741Hz, and analog sync pulse data at 2222.2222 Hz. (As a side note i am not sure why these sampling frequencies are not whole numbers). This sampling data can be found in header line 7. Emg variable names are on line 4, however they contain numbers after and i just want the variable name.

We plan to do much more with this data but here is a list of things my code currently does:

Remove NaN value from each data column since it is all sampled at different frequencies

bandpass filter all of the emg data (6th order butterworth filter 30-300Hz).

Using sync pulse i need to extract region of interest from it and align all of the EMG data and gyroscope data with it. So essentially, i would want to take a region of interest between two sync pulses and plot how the EMG looked as well as the correstponding gyroscope data all during the same time interval. This was a bit tricky to do because of the different sampling rates.

The plot should have EMG data on left and gyroscope data on right. The gyroscope data should be smoothed and plotted on same graph. Hope that makes sense.

So for example the first row on plot would have emg data plotted for bandpassed RUT variable on the left side and to the right side it will have RUTx, RUTy and RUTz variables (that have been smoothed) on one plot exactly to the right of the EMG plot.

Thank you for your time and willingness to help!

[Ines Shekhovtsov]

And as a reply to your other comment, i did not mean to imply that i was not trying to take any of your advice. As i explained i'm a very new beginner to coding in general and the code you wrote was still confusing to me since i didn't know how to call the variables from my script and start working with them.

[dpb]

See the doc @table and particularly look at the overview of the <Table top level page> and then most particularly <Access data in a table> link at the bottom of that page. If illustrates all the multiple ways in which one can address the data in a table -- by explicit name as you're used to coding with independent variables, but more efficiently from writing general code, by variables containing names or by indices.

That doesn't but just begin to touch the features available to make your life easier in peforming analyses on the data within the table...

[Ines Shekhovtsov]

Thank you, do you know how i can attach the .csv file on here. It is too large to just attach.

[dpb]

"...Seems i cannot attach the file as it is too large..."

Would only need first 100 lines or less to illustrate...open it in the editor and save just a piece of it to a new file and attach that. Or, as .csv it probably would zip down to small amount although somewhat less convenient.

"...Remove NaN value from each data column since it is all sampled at different frequencies..."

That may need some consideration of how you import the data, then. Probably the first thing to do after import would be to resample all to a common time base. But it 'splains what wasn't evident in the original post about NaN and independently removing NaN data. Would need to see the file to really tell the "ideal" way given the additional information; the above was based on erroneous (or at least incomplete) assumptions about the file structure.

[Ines Shekhovtsov]

Could i use websave to upload file to mathworks?