MATLAB Answers

0

Help definitely needed! How do I load multiple txt files and change the varible names of each to include the date from file

Asked by mashtine on 17 Feb 2014
Latest activity Commented on by mashtine on 24 Feb 2014
Hey everyone,
My coding skills have clearly dies on me with this one. I have a few txt files in directory that I am using textscan to load (massive files so I want to reduce memory). I would like a loop that goes through the list, loads in data into the workspace and gives each dataset a variable name that includes the date from the file name.
My file names are like this one midas_wind_198001-198012.txt from 1979-2013.
Here is the code I have thus far for loading but I am stuck on taking it further to renaming the variables. I do not want to use EVAL for efficiency reasons and I would like each dataset to be its own variable
fid = fopen('midas_wind_197701-197712.txt', 'r'); C = textscan(fid,'%s %s %f %f %s %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %s %f', 'Delimiter',','); fclose(fid);
Thanks a lot in advance!!

  0 Comments

Sign in to comment.

2 Answers

Answer by Mischa Kim
on 17 Feb 2014
Edited by Mischa Kim
on 17 Feb 2014
 Accepted Answer

Masao, assuming that the only the year changes in each filename, use
for ii = 1973:2013
f_name = strcat('midas_wind_',num2str(ii),'01-',num2str(ii),'12.txt');
fid = fopen(f_name, 'r');
scan_val = textscan(fid,'%s ... % !command cut off for readability!
v_name = genvarname(strcat('C',num2str(ii)));
eval([v_name ' = scan_val';]);
fclose(fid);
end

  7 Comments

Thanks again Mischa and Jos. Yes, thats the tricky thing with EVAL, it does allow me to get more freedom with variable naming etc but I am been told structs may be best. However, I am dealing with files that each have ~ 1.2 million rows and contain both string and numeric data. I assumed keeping each file separate would then be best as I have 1980-2013 but do you suggest using structs instead? Or would that complicate things further down the line when indexing and doing calculations with the data?
You guys have been very helpful so I look forward to your reply!
It all depends on what your doing. If you only need to process a single file at a time, why not store the values into a variable that always has the same name.
filename = 'xxx.txt'
A = load(filename) % keeping a fixed name ...
Result = Myfunction(A) % ... makes further coding easier
AnotherResult = MyOtherFunction(A)
disp(['A result of processing file: ' filename])
disp(Result)
I see no point in switching variable names here, as you have to edit the code every time. In humble my view, variables should be seen as coding elements that have a fixed name , but variable contents .
In addition, if you have two related variables, it is convenient to give them the same name but arrange them in an array:
Year(1) = 1990
Year(2) = 1996
Year(3) = 2001
Imagine using this design
Year1990 = 1990
Year1996 = 1996
Year2001 = 2001
Now see what happens if you need to change the third year
Year(3) = 1997 % easy to understand
Year2001 = 1997 % awkward!!

Sign in to comment.


Answer by Jos (10584)
on 17 Feb 2014

You can use cell or structs.
MyFiles = {'midas_wind_197701-197712.txt','whatever.txt','another name.txt'}
for k=1:numel(MyFiles) % loop over all the files
fid = fopen(MyFiles{k}, 'r');
tmp = textscan(fid,'%s %s %f %f %s %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %s %f', 'Delimiter',',');
% store
MyData{k} = tmp ; % option 1
MyDataToo(k).data = tmp ; % option 2
MyDataToo(k).filename = MyFiles{k} ; % for convenience you can store the
% filename in a separate field
fclose(fid);
end
Now the contents of MyData{x} or MyDataToo(k).data corresponds the file with the name MyFiles{x}

  1 Comment

Hey Jos,
I have used your code to create a cell array with the data and struct array. Unfortunately, both take up a large space when being saved (over 10 GB) whereas with Mischa method, my .mat file is only about 500 MB. Any reason why saving the variables separately causes this?

Sign in to comment.