"Maximum variable size allowed on the device is exceeded." when using predict on GPU
61 vues (au cours des 30 derniers jours)
Afficher commentaires plus anciens
hanspeter
le 30 Oct 2024 à 9:58
Commenté : hanspeter
le 3 Nov 2024 à 13:17
Hello there,
I have a neural net and want to do a prediction using predict. When I do that I get the error
Error using dlnetwork/predict
Execution failed during layer(s) 'fc1'.
Caused by:
Error using parallel.internal.gpu.mtimesAdd
Maximum variable size allowed on the device is exceeded.
Afaik it's because on GPU Matlab only supports 2^32 elements in one array. Whats the best way to circumvent this?
I know I can disable the GPU completely by setting CUDA_VISIBLE_DEVICES to -1, but I still want to use it in other functions.
The data I pass into predict is a normal array and not a gpuarray.
Is there a way to force predict to use CPU?
Cheers
0 commentaires
Réponse acceptée
Joss Knight
le 1 Nov 2024 à 15:39
Modifié(e) : Joss Knight
le 1 Nov 2024 à 15:41
Is there a way to force predict to use CPU?
Yes. Pass data in as an in-memory array. If your data is on the device, you may need to call gather.
z = predict(net, gather(x));
If this does not work it will be because some of your network weights are on the GPU. So you may need to call gather on the network as well:
net = dlupdate(@gather, net);
net.State = dlupdate(@gather, net.State);
It's much more likely, however, that you're just passing too much data into your network. If you are trying to call predict on a large dataset, consider using minibatchpredict instead, to process the data in batches.
Plus de réponses (1)
Shivam Lahoti
le 30 Oct 2024 à 11:22
Hi hanspeter,
I understand you are encountering a “Maximum variable size allowed on the device is exceeded" error when using predict on a GPU.
As you mentioned, the current maximum number of elements allowed in a gpuArray is intmax('int32'), or 2^31. This is because CUBLAS passes array lengths as int (int32_T). This is a known limitation and is mentioned in the documentation link below:
Moreover, none of the following can exceed intmax('int32'):
- The number of elements of a dense array.
- The number of nonzero elements of a sparse array.
- The size in any given dimension. For example, zeros(0,3e9,'gpuArray') is not allowed.
I hope this helps you understand the issue.
Regards,
Shivam
0 commentaires
Voir également
Catégories
En savoir plus sur Image Data Workflows dans Help Center et File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!