append/save same variable with updated values into .mat file(row-wise)

What is the syntax for appending same variable with different values to an existing .mat file? When I use -append, I end up replacing the values!

Example:

   for col = 1:10

    out = zeros(1,1000000); %reset out to zero vector
    x=randn(1,100000);
    out=x.^2;

    if col == 1
    filename='z.mat';
    save(filename,'out','-v7.3'); % Write to MAT file
    else
    save(filename,'out','-v7.3','-append');
    end
 end

After running the above code, I have the replaced values instead of appending the updated/current values in out into the .mat file after every loop iteration.

Everytime, I iterate through the for loop, I want to save current values in the variable out(1x1000000) into the .mat file row-wise. At the end, I should have (10x1000000) matrix in z.mat and my variable out should hold only 1x1000000 vector during each iteration at a time.

How can I achieve this?

What am I missing?

Answer 1

[Guillaume]

The proper way to do what you want is to use matfile which lets you read and write to part of variables in a mat file.

filename='z.mat';
m = matfile(filename, 'Writable', true); %Note: writable is true by default IF the file does not exist
for row = 1:10 %why do you call it col when you're asking for rows?
  %out = zeros(1,1000000); %completely unnecessary
  x = randn(1,1000000);
  out = x.^2; %whatever you put into out is overwritten here anyway
  m.out(row, 1:1000000) = out;
end
clear m;

[Stephen23]

I accepted this answer as it is patently the best and correct answer.

Answer 2

[Thomas]

You are always overwriting the same variable 'out' in each of the iterations..

for col = 1:10

    out = zeros(1,1000000); %reset out to zero vector
    x=randn(1,1000000);
    out(col,:)=x.^2;

    if col == 1
    filename='z.mat';
    save(filename,'out','-v7.3'); % Write to MAT file
    else
    save(filename,'out','-v7.3','-append');
    end
 end

[zozo]

In your case, at the end I am holding 10x1000000 values in my variable 'out' and hence in my RAM (workspace).

But what I want is, at a time, I should hold only 1x1000000 in variable 'out' during each iteration. Only my z.mat file should hold all the 10x1000000 values in it at the end and 'out' should not exceed 1x1000000.

[Thomas]

True.. My bad..

http://www.mathworks.com/help/techdoc/matlab_prog/f10-61206.html#f10-61260

Note Saving with the -append switch does not append additional elements to an array that is already saved in a MAT-file

[zozo]

So, there is noway I can achieve this?

Answer 3

[Malcolm Lidierth]

See the Project Waterloo MAT-file Utilities, they will let you "grow" a version 6 MAT-file entry using AppendVector and AppendMatrix

http://www.mathworks.com/matlabcentral/fileexchange/12250

I assume you have a reason not to use rand(10,100000) ??

Answer 4

[Wilbert Sequeira Sandoval]

If what you want is to grow a matrix you have in the file then you need to first load the one in the file, say the matrix's variable name is A.

load(filename,'A');

Now if you want to append as rows, you do:

A=[A;B];

If you want to append as columns:

A =[A:B];

Where B is what you want to append(it must have the same number of columns as A if you're appending it as rows and the same number of rows if you're appending it as columns). It will be slow but it will work. Then at the end you use -append to save it to the file again(which will override the old A with the updated version).

[Stephen23]

" then you need to first load the one in the file"

This is not required at all. The function matfile makes accessing/extending data easy, and does not require loading the mat data into memory.

Answer 5

[Leigh Martin]

You can load all variables at once and create a concatenated array, but this doesn't work if the variables are too big to load into memory (which is presumably why you would want to do this in the first place). If you don't need the array saved as one big array, here's a useful work-around:

% this code saves variables named array_part1, array_part2... from your data
for i = 1:10
data = i*ones(1,5); % create some fake data
eval(sprintf('array_part%i = data', i))
save('file.mat', sprintf('array_part%i',i), '-append')
clear(sprintf('array_part%i',i))
end

you can then load whatever part you like using

    load('file.mat', sprintf('array_part%i',i))
eval(sprintf('array = array_part%i', i))