Debugging CUDA MEX in VS2017 (v2)

Hello,

I'm going to mirror a question asked two years because I've run into a similar issue :

https://ch.mathworks.com/matlabcentral/answers/450099-debugging-cuda-mex-in-vs2017-on-volta-gpu?s_tid=srchtitle

I've also asked this question at NVIDIA developper forum and will include the link for cross reference:

https://forums.developer.nvidia.com/t/debugging-cuda-mex-in-vs2017/172809

I've run into "CUDA_ERROR_ILLEGAL_ADDRESS", and using the host side debugger identified that it occurs when I try to fetch data back from the GPU while invoking "mxGPUCreateMxArrayOnCPU()". I would like to enter debugging mode inside the kernel to follow what is happening device side but I can't use the legacy debugger via VS2017 because it does not support my graphics card, and the next-gen debugger does not support remote debugging.

Does anyone know a workaround about this without having to resort to print statements?

Thanks in advance

[Joss Knight]

First, synchronize using cudaDeviceSynchronize(), then call cudaGetLastError() and make sure the error hasn't actually already occurred before you call mxGPUCreateMxArrayOnCPU. Hamza's suggestion of cuda-memcheck is a good idea. If everything is fine then you might want to post your kernel code here for us to debug by inspection.

Answer 1

[Hamza Butt]

According to NVIDIA's release notes for Nsight Compute, support for Pascal GPUs has been removed since 2020.1. Also, as you mentioned, remote debugging is not currently supported with Nsight Compute's next gen debugger. This is more suited for NVIDIA's forum, as the question is generic to debugging device-side CUDA C++, with or without MATLAB. However, I will try to help as much as I can, though this is a difficult question for someone outside NVIDIA. MATLAB does not include a debugger for GPU code, instead it relies on third party debugging tools for the CUDA infrastructure, and only NVIDIA can answer this question definitively.

Of the ideas that I can think of: Try using an older version of Nsight compute that does support your GPU, to see if it works. Maybe you can use cuda-gdb directly? If you think you are leaking memory, try compiling in debug mode and running it through cuda-memcheck. This should identify the location of code where it thinks you are leaking memory.

[Vyacheslav Samokhvalov]

Thank you for your answer!

For some reason somewhere between invoking mxGPUCopyFromMxArray, mxGPUGetData to work on the underlying data, and copying data back wtih mxGPUCreateMxArrayOnCPU there's unexpected behaviour. Because I cannot inspect what was happening on device code, to solve this issue I avoid using mxGPUArrays altogether, get pointers to data in Matlab arrays with mxGetSingles, do work with them in an outside CUDA library, and write data back to Matlab's mxArray.