'Out of memory on device' error for GPU

When trying to replace a value in an existing large GPU array A, e.g. like this:

A(1,1)=rand(1,1,'single')

I get the error:

"Error using gpuArray/subsasgn

Out of memory on device. To view more detail about available memory on the GPU, use 'gpuDevice()'. If the problem persists, reset the GPU by calling 'gpuDevice(1)'."

I don't understand since I would not expect any additional GPU memory required for this operation. However, some closer investigation revealed that the amount of free GPU memory to enable even the simple operation above is roughly equal to the memory taken by A itself.

In cases like this, working with large variables, this means that at all times a huge chunk of GPU memory needs to remain available. Quite inefficient. Any explanation / solution for this issue would be very welcome, thanks!

I'm using Matlab 2016a

Here's the output from gpuDevice:

                      Name: 'Tesla K20c'
                     Index: 1
         ComputeCapability: '3.5'
            SupportsDouble: 1
             DriverVersion: 7.5000
            ToolkitVersion: 7.5000
        MaxThreadsPerBlock: 1024
          MaxShmemPerBlock: 49152
        MaxThreadBlockSize: [1024 1024 64]
               MaxGridSize: [2.1475e+09 65535 65535]
                 SIMDWidth: 32
               TotalMemory: 5.0330e+09
           AvailableMemory: 2.4498e+09
       MultiprocessorCount: 13
              ClockRateKHz: 705500
               ComputeMode: 'Default'
      GPUOverlapsTransfers: 1
    KernelExecutionTimeout: 0
          CanMapHostMemory: 1
           DeviceSupported: 1
            DeviceSelected: 1

Answer 1

[Edric Ellis]

This behaviour is related to what MATLAB calls "in-place optimization". There's some background in this entry in Loren's blog. Basically, MATLAB treats assignment of the form:

A(x) = B;

as equivalent to a function call a bit like this:

A = subsasgn(A, substruct('()', x), B);

To avoid duplicating A, the in-place optimization must take place. Unfortunately, this doesn't happen when executing the commands at the command-line, but it does happen inside functions. So, for example, on my GPU device which has 6GB of RAM, the following two commands error at the command-line, but succeed inside a function:

A = gpuArray.zeros(1, 2e9, 'uint16');
A(1) = 2;

[Peter]

thanks a lot Edric! I tried to test your suggestion to use an in-place operation to replace a part of a large GPU array A with B. Here's my testcode in which I just try to replace the first element of A:

keepfree=0.5;%percentage of GPU memory to keep free, call it a 'safety margin'
reservefactor=1.8;%when reservefactor<2, the not in-place assignment should fail because of memory shortage
bytessingle=4;%bytes for a single floating point number
reset(gpuDevice);
S=gpuDevice;
maxbytesA=S.AvailableMemory*(1-keepfree/100)/reservefactor;
maxnumelA=floor(maxbytesA/bytessingle);
A=rand(maxnumelA,1,'single');
whos A%to check the nr of bytes of A
A=gpuArray(A);
gpuDevice
A=inplacereplace(A);

According to Loren on in-place operations, the in-place operation should take place in a function that is in a function. So my function 'inplacereplace' which does the in-place replacement looks like this:

function x=inplacereplace(x)
x=in_inplacereplace(x);

function x=in_inplacereplace(x)
x(1,1)=1;

However, this approach apparently doesn't seem to be triggering an in-place operation since again the 'out of memory on device' error occurs. Am I doing it wrong?

[Joss Knight]

What Loren means is, you can't in-place replace something in the MATLAB workspace. The fundamental barrier is still in the way - if your function errors, MATLAB must be certain that it can return the MATLAB workspace - including your variable A - to the state it was in before you called your function. And it can't do that if it allows A to be operated on in-place. What Loren means is, put all that code into a function - doStuff - and call it from the command line. You can't return A to the workspace, so if you need to inspect it - plot it, look at some of the data in it etc - then you need to do that inside your function using plot, disp etc.

[Peter]

Joss, thanks a lot! I hope I get it: Loren indeed doesn't return x to the global workspace. When she calls inplaceTest(x), a local copy of x does have to be made in the workspace of inplaceTest, which explains the initial increase in memory usage in her Page File Usage History. Once inside she uses the in-place optimization by doing the operation from a function within a function.

So my approach is indeed not working since initially a copy of A is required for which there's not sufficient space in the GPU memory.

In my code, part of A is replaced with freshly incoming data in a loop. Every j'th loop iteration, some additional operations are done with A leading to some program output.

Do I get it right that the only way to do in-place operations is to put the entire program in a function which is again in a function?

[Joss Knight]

It's missing the point to trivially say just put a function inside a function. The fundamental discriminator is whether you are operating on data that is exposed in the MATLAB workspace. You could go twenty nested functions deep and you'd still never get in-place operations if the data being operated on exists in the MATLAB workspace. Can you see it in the Workspace browser window? If yes, operating on it will create a copy.