Deep Learning - Data no longer exists on the device

I got the following error when trying to implement a deep learning NN in MATLAB.

Error using trainNetwork (line 154) The data no longer exists on the device.

Caused by:
    Error using nnet.internal.cnngpu.batchNormalizationForwardTrain
    The data no longer exists on the device.

Any suggestions?

Answer 1

[Joss Knight]

This would happen if you are resetting the device or switching device using reset or gpuDevice. Perhaps you have a custom layer which is innocuously calling gpuDevice(1); this is resetting the device and invalidating all GPU data.

[user06]

how to fix this???

[Joss Knight]

Reduce your MiniBatchSize in the trainingOptions.

[user06]

the same program is running fine on CPU but when I am running it on GPU it is giving error "The data no longer exists on the device"

[Joss Knight]

Usually it's because you're running out of GPU memory, so reduce the MiniBatchSize and see if the problem goes away. If not you're going to have to post your code so we can see what is going on.

[Zhipeng Fan]

trainning the segnet ,,

Error using trainNetwork (line 154) , cannot transfer gpuArray to single;

I don't know how to solve it,can you help me ?

[Walter Roberson]

Sounds to me as if you are out of GPU memory.

[Walter Roberson]

cui comments to Joss Knight:

thank you! my network is running out of GPU memory!

Answer 2

[Alvaro Lopez Anaya]

In my case, I had similar problems, despite of the fact that I have a gtx1080Ti.

As Joss said, reducing the MiniBatchSize solved my problem. It's all about the training options.