PCT GPU Computing: CUDA error was: setting the device when a process is active is not allowed

Hi, everyone!

It seems to me that there is a problem in Parallel Computing Toolbox (PCT) with CUDA operation. The problem is that the setting of CUDA device in PCT fails, if CUDA device has been set up earlier within the same process by someone else, therefore, I cannot use PCT functionality, working with CUDA in my application which is also used by CUDA.

I will give you a simple example. Here it is:

% Starting MATLAB Environment...

% 1. Load a DLL file used by CUDA (or MEX file, it does not matter, only for example)
loadlibrary('MyComputingLib.dll', 'MyComputingLib.h', 'alias', 'cuda_lib')

% 2. perform calculations...

% 3. try to use PCT CUDA...
gpuDevice()

% ... and the error message appears:

An unexpected error occurred during CUDA execution. The CUDA error was: setting the device when a process is active is not allowed.

If the aforementioned actions are performed in the reversed order, the error does not occur (My software works correctly with CUDA device setting). Why cannot MATLAB set up CUDA device, when, starting with CUDA 4.0 version, it is possible to set up CUDA device a multiple number of times within one process?

I think this limitation is the rudiment of what has been left from the old CUDA versions (old derivers), when it was impossible to set up a device a multiple number of times after the process start.

Thank you in advance for your answer.

Answer 1

[Joss Knight]

In MATLAB versions up to R2016b, MATLAB's built-in commands must be the first activity on each device within the MATLAB process after a device reset. If you are calling into code that makes a CUDA API call on any device before MATLAB does, MATLAB will error unless you call cudaDeviceReset() on that device before MATLAB tries to access it.

This happens because MATLAB sets device flags when switching to or resetting a device, and this is only allowed before any kernels have run after a reset.

As some have pointed out, if you have only one GPU you can work around this by making sure MATLAB (or any MATLAB workers in a pool) uses the GPU before any of your custom functionality. The more robust solution for your own multi-gpu custom code is to call cudaDeviceReset() on each device you have used before you return control to MATLAB.

This behaviour will no longer occur in the next version of MATLAB.

[Matt J]

As some have pointed out, if you have only one GPU you can work around this by making sure MATLAB (or any MATLAB workers in a pool) uses the GPU before any of your custom functionality.

Just a warning. I have observed that this workaround is reliable only when a single MATLAB session is using the GPU. If you start an additional MATLAB instance while a .dll in a pre-existing MATLAB session is using the GPU, then even if MATLAB commands are the first used in session #2, I encounter CUDA errors.

[Joss Knight]

That's a surprise. I wouldn't have expected that to happen. Is this the same "Cannot set device on active process" error?

[Matt J]

I did not keep track of the error messages, but I don't recall seeing that particular one.

I've been surmising that the GPU isn't distinguishing between MATLAB instances. If external CUDA code in MATLAB session #1 uses the GPU first, then subsequently MATLAB built-ins in session #2 use it, it is as if the same order of events took place in the same session.

Anyway, I would be grateful for some detailed documentation/guidelines on how external CUDA code (using loadlibrary etc...) can interact safely with the Optimization Toolbox and maybe also communicate gpuArray data to each other. Our lab has regular need of this.

Answer 2

[Matt J]

I seem to be experiencing the same thing. Hope TMW can comment...

However, if what I'm seeing is the same as you, a simple workaround is to do something like this in startup.m

 a=gpuArray(1); clear a

I find that as long as gpuArray is the first thing to use the GPU in the MATLAB session, things seem to work normally.

[David Pate]

I also had this problem in Matlab 2016a, and I'm using a solution similar to yours. In my startup file I have:

try
    gpu = gpuDevice;
    clear gpu;
catch
    disp('No GPU device found during startup');
end