GPU Coder used, but got error: Error generated while running CUDA-enabled program: [700,cudaE​rrorIllega​lAddress]

I'm using GPU coder to accelerate N-D interpolation function interpn().

Everything worked fine when my problem scale was small, i.e. interpolating in 1-D or 2-D dimensions. But when I throw a larger problem, of 5-D interpolation to it, MATLAB threw the following error message:

Error generated while running CUDA-enabled program: [700,cudaErrorIllegalAddress] an 
illegal memory access was encountered. To generate additional debugging information, 
enable the 'SafeBuild' GPU code generation option.

I followed the error message suggestion above and added the 'SafeBuild' GPU code generation option, and got the following error instead:

Error generated while running CUDA-enabled
program: [46,cudaErrorDevicesUnavailable] all
CUDA-capable devices are busy or unavailable

I know that when I wasn't using GPU Coder, the code still worked but kind of slow. But now when GPU coder is enabled, it is not working. I have NVIDIA RTX 2080ti with 11G memory. When I was using CPU to do the calculation, the problem took 1 hour and peak RAM use was around 40GB out of 64GB RAM. Could this error be because I ran out of GPU memory by establishing my problem too large? How can I tell if this is the case?

OK update: I rebooted my Windows PC, and ran the same program again, the following error showed instead:

Error generated while running CUDA-enabled
program: [2,cudaErrorMemoryAllocation] out of
memory

I think this confirms that indeed my problem is causing GPU to run out of memory just to start the problem. Now I know the problem, is there any suggestions how to split the problem for interpn? Thanks!

[Hariprasad Ravishankar]

Hi Tong,

Would it be possible for you to provide us with a reproduction example to investigate at our end?

Further, please let us know the MATLAB release you are currently working with.

Hari

[Tong Zhao]

Hi Hari,

Thank you for your attention. I got answer to this problem with some help. Indeed my problem was so big that it created arrays with size larger than 4GB (double arrays with size 2^32) to be interpolated. There's a limit to how large the indices of an array could be (int32), I think this might be why. I'm still curious though as to how one could expand the size limit to beyond (int32).

[First]

hi may i know the solution ？ i met same error of cuda out of memory but its quite weird. Same code can run cpu1+gtx980 but error above when i run on cpu2+gtx980 and cpu2 has more much more cores and memory than cpu1.

Answer 1

[Gautam]

Hi,

The error is most likely that a matrix is growing beyond it's intended size. Take a careful look at your code to analyze which matrix is growing beyond the intended value.

Growing size matrices are silently handled by MATLAB, but the coder generates CUDA C which does not handle growing sizes so it is mostly seg-faulting when you run the generated code.

Try running your code in MATLAB and check if every matrix at the end of its computation is having the intended size.

Also, when you run into a "cudaErrorIllegalAddress" you have to restart MATLAB and run your code. This is because, when the coder runs the generated code for the first time, it creates a CUDA context and it reuses the same context in the subsequent runs till the MATLAB application is closed. Now, when you encounter a seg-fault when running the generated code, the context gets corrupted, MATLAB doesn't know about it and so it tries to run the same context in the next run, which will fail. So the solution is to restart MATLAB.

Gautam