Matlab doesn't seem to be using all the memory available to it and gives out of memory error.

I am running my code, and it seems to be needing more and more memory, I check the windows memory and it stays the same giving matlab 1 Gb instead of the 6 or 24 that is availible to it. I have been trying sparce I have "clearvars -global" and "clear all" at the beginning of my code I have reassignment to "=[]" and sparse, but Matlab doesn't use or direct me to where or why it can't use the the 32 it has detected of system memory but gives up when it knows it is using 2 Gb according to the memory command.

Error Message:

Out of memory. Type "help memory" for your options.
Error in dembedding_v8 (line 165)
fd_square_inv_alligned = s_param_inv_alligned .* y;
 
>> whos
  Name      Size                  Bytes  Class     Attributes

  x1        1x1                       8  double              
  x2        1x1                       8  double              
  x3        1x1                       8  double              
  x4        1x1                       8  double              
  x5        1x88                    176  char                
  x6        1x90                    180  char                
  x7        1x87                    174  char                
  x8        1x50000              400000  double              
  x9        1x1                       8  double              
  x10       1x1                       8  double              
  x11       1x1                       8  double              
  x12       1x1                       8  double              
  x13       1x1                       8  double              
  x14       100084x1             800672  double              
  x15       50000x1              800000  double    complex   
  x16       50000x1              800000  double    complex   
  x17       1x100000             800000  double              
  x18       1x1                       8  double              
  x19       1x1                       8  double              
  x20       1x100000             800000  double              
  y21       1x100000            1600000  double    complex   

note: summing the bytes in whos I get 6001290B or 6MB? which doesn't seem right

>> memory
Maximum possible array:       24202 MB (2.538e+10 bytes) *
Memory available for all arrays:       24202 MB (2.538e+10 bytes) *
Memory used by MATLAB:        2156 MB (2.261e+09 bytes)
Physical Memory (RAM):       32768 MB (3.436e+10 bytes)

*  Limited by System Memory (physical + swap file) available.

Edit: Thank you all for your help and insite with the .* failure(doing a matrix square instead of a 1 by x vector resutl I was getting a x by x size matrix) I am able to do 100,000,000 elements for accuracy and add a 1 million for loop all excecuting. Thank you all for your help and comments.

[dpb]

"note: summing the bytes in whos I get 6001290B or 6MB? which doesn't seem right"

And what doesn't seem right about it? You've got equivalent of two 100K double complex arrays which is same as four 100K doubles plus two more 100K doubles and a 50K.

(2*2 + 3)*100K + 50K --> 750K * 8 bytes/double --> 6000K --> 6MB

ignoring the odds and ends.

What do you think is wrong about that?

As far as MATLAB and using physical memory, that depends on the OS--what OS are your running? And, just for good measure, which ML release?

[Walter Roberson]

The "memory" function is only available for MS Windows.

[dpb]

Well, yes, but there are multiple Windwoes incarnations...but, granted, I should have asked for which Windows.

[Walter Roberson]

Out of memory. Type "help memory" for your options.

Error in dembedding_v8 (line 165)

fd_square_inv_alligned = s_param_inv_alligned .* y;

My suspicion is that you are using R2016b or later and that each of those two variables are vectors, but that they do not have the same orientation, so the code is equivalent to bsxfun(@times, s_param_inv_alligned, y) which can create a huge rectangular matrix, each element in s_param_inv_alligned multiplied by each element in y.

[Matt J]

note: summing the bytes in whos I get 6001290B or 6MB? which doesn't seem right

You cannot use whos to reliably determine the RAM consumed, because whos does not account for variables that share memory. For example, in the steps below, it appears to whos that A and B consume 2 GB, but really they consume only 1 GB, because A and B are really just copy-on-write pointers to the same data.

>> clear all; memory

Memory used by MATLAB:        2154 MB (2.258e+09 bytes)

>> A=rand(512,512,512); B=A;
>> Whos A B

  Name   Size            Kilobytes     Class    Attributes
                                                          
  A      512x512x512       1048576     double             
  B      512x512x512       1048576     double             
                                                          
>> memory

Memory used by MATLAB:        3178 MB (3.332e+09 bytes)

[Bruno Luong]

Out of memory. Type "help memory" for your options.
Error in dembedding_v8 (line 165)
fd_square_inv_alligned = s_param_inv_alligned .* y;

You should tell us the size of s_param_inv_alligned and y in your function dembedding_v8.

My guess is you multiply large column vector with large row vector.

[Jonathan Fernow]

Thank you all for your inputs.

@Walter Roberson @Bruno Luong: I am running Matlab 2018b. I have taken your input to avoid a square matrix and change to a parfor loop. I am able to run.

@Matt: to clarify whos is accurate if duplicated pointers

@Bruno I have been varying the length 1*10^6 is the typical length

[Matt J]

@Matt: to clarify whos is accurate if duplicated pointers

No, the opposite. whos does not offer a way to account for pointer duplication in the memory tally.

[Jonathan Fernow]

Thanks for the clarification Matt. So whos is safe to be a a worst case scenario where it will show potentiall more memory if totaled but not less.

[Bruno Luong]

I asked for size not for length. If you multiply a row vector with a column vector, MATLAB tries to allocate and return a big matrix.

It does not very matter to tell us the output of whos command performed on the base workspace. Please do it inside your function and post the result.

[Jonathan Fernow]

I ran the size command with my current run and it looks like it is a 1:1, it looks like the .* created a crazy square matrix when attempting that step.

%code executed
parfor i = 1:length(y1)
  z1(i) = x1(i) * y1(i);
end

 size(y1)
ans =         1   100000000
>> size(x1)
ans =        100000000           1
>> size(z1)
ans =        1   100000000
>> 

[Walter Roberson]

Your x1 is a column vector. Your y1 is row vector. Since R2016b your code has been the same as if you had written bsxfun(@times, x1, y1) which is like

repmat(x1, 1, length(y1)) .* repmat(y1, length(x1), 1)

producing a rectangular result.

You need x1.'.* y1

Answer 1

[Fikret Dogru]

Hello,

You can change your variables to single(variables) so it reduces the memory half and in the preferences increase java space to the maximum.

[Walter Roberson]

Increasing Java space is likely to make the problem worse. MATLAB reserves the Java space for Java use, and does not permit it to be used for other variables.

[Rik]

And although it good advice, the factor 2 reduction of memory requirements tends not to be the issue. Often people try to create arrays that are several orders of magnitude too large.

[Jonathan Fernow]

Rik was right as I had a larger issue, but thanks Fikret Dogru singles was a good suggestion, but some of my functions seem to be struggling with the singles implamentation. Singles ran into an issue when convrting the table to double was successful but the table to single seems fine.

Error using single
Conversion to single from string is not possible.

Error in myprogram (line 45)
column7  = single(mytable_element{:,7});

%had/have no problem
column7  = double(mytable_element{:,7});

[Walter Roberson]

This would probably be irrelevant if you were not using .* between a row vector and a column vector.