matlab disk space and memory issues

I am running a script that is formatting some data for use in a machine learning/classification system. However when running it MATLAB ends up using 100% of disk space and up to 80% of memory.

Even when I temporarily pause the script matlab is still using a good 50-60% of memory. This is even after waiting for several minutes after pausing the script.

Any way I can stop matlab needlessly using so much memory and hard disk space? Seems to me that it should not be doing this when an application is paused, though perhaps I am mistaken since my script initializes a 8192x140,000 matrix of zeroes and then replaces each column with new data.

Many thanks

Regards R

[Dennis]

I think that Matrix has a size of almost 9 GB, depending on the size of your RAM it might not fit in. I do not know why it should take alot of disk space though.

[cnfan_sd70]

Is it possibly the fact that I am looping round 140,000 times to iterate through some data, to then format the iterated section and then deposit it in the appropriate column of my matrix? I do know that loops are not great in matlab

[Steven Lord]

Is it possibly the fact that I am looping round 140,000 times to iterate through some data, to then format the iterated section and then deposit it in the appropriate column of my matrix?

It could be. Without seeing code it's very difficult to say. Assuming your matrix is a full real double matrix of size [8192 140000] allocating (potentially multiple copies of) a roughly eight and a half GB matrix is also a strong candidate for why your code uses so much memory.

I do know that loops are not great in matlab

That's not really true anymore. We've done a lot of work over the past 15-20 years to improve the performance of MATLAB, including improving the performance of loops. It's possible to write fast code that uses loops, slow code that uses loops, fast vectorized code, or slow vectorized code.

[cnfan_sd70]

Hi Steven, here I am posting a little bit of my code to show you what may be causing the memory problem (I cannot post all the code however)

         function  [dataVEC,labelsVEC] = NAME(DATA)
            dataVEC = zeros(8192,length(DATA));
            labelsVEC = zeros(length(DATA),1);
            for j = 1:length(DATA)
                a_1 = DATA{j,1}.a;%assign to required section of data vector
                a_1  = a_1 - mean(a_1);
                a_1 = a_1/norm(a_1);
                a_1 = [a_1(1,1:400) circshift(a_1(1,401:end),floor(randn*150))];
               data = feature_extract(a_1);
               dataVEC(:,j) = data;
               labelsVEC(j,1) = DATA{j,1}.label;
            end
         end

I have "hidden" some of code in a fake function called "feature_extract" so as to show you the basic creation of arrays etc in my code without giving too much away.

Some matlab functions I have used within my feature extraction are "isnan", "interp1", "findpeaks", "hilbert".

Hope this is enough info for you to take a guess as to whats going wrong! Cheers.

[cnfan_sd70]

Sorry I forgot to say - as you can see, before the for loop, the data matrix is initialised as an 8192x140,000 matrix of zeros. The labels matrix is initialised as an 140,000x1 vector of zeros. Each column of the data matrix is then assigned a piece of formatted data that is 8192x1 in size. Then it loops round and does the same basic operation again