"Maximum variable size allowed on the device is exceeded" error when using internal.s​tats.gpu.p​dist2?

Hello, I hope you are well. I'm encountering an issue with the pdist2 function on my GPUs. I have both an A6000 with 40 GB of memory and an A100 with 80 GB of memory,

load('Dataset_1.mat')
X=single(Dataset(:,[2 4]));

X_CPU = X(1:60000,:);
X_GPU = gpuArray(X_CPU); % Convert data to gpuArray 
 % Compute pairwise distances more efficiently
 D = pdist2(X_GPU, X_GPU, 'squaredeuclidean');

but I'm getting the same error on both devices. the error as shown below

Error using internal.stats.gpu.pdist2 Maximum variable size allowed on the device is exceeded.
Error in gpuArray/pdist2 (line 262) [doTranspose,D] = internal.stats.gpu.pdist2(X.',Y.',dist,additionalArg,smallestLargestFlag,d.AvailableMemory);

How can I solve this error?

Answer 1

[Meet]

Hi Med,

When calculating pairwise distances for your dataset "Dataset_1.mat," you encounter memory constraints due to the following reasons:

• X = 60000 x 2
• Pairwise distance matrix size= 60000 x 60000 = 36 x 10^8 elements

NVIDIA libraries that MATLAB uses for GPU computation store array lengths as 32-bit integers, limiting arrays to a maximum of intmax('int32') elements. This imposes a hard limit on array size.

As a workaround, you can try applying the pairwise distance calculation to smaller chunks of your dataset. This method involves processing smaller portions of the data and storing the results in a final cell array. It achieves the same outcome as applying the “pdist2” function to the entire dataset but handles it in more manageable parts, which are then aggregated.

load('Dataset_1.mat')

X = single(Dataset(:,[2 4]));
X_CPU = X(1:60000,:);
X_GPU = gpuArray(X_CPU); 

chunkSize = 10000; % Adjust chunkSize parameter based on available memory
numChunks = ceil(size(X_GPU, 1) / chunkSize);

% D_parts would store the final result of the pairwise distance of your dataset
D_parts = cell(numChunks, numChunks);

for i = 1:numChunks
    idx1 = (i-1)*chunkSize + 1 : min(i*chunkSize, size(X_GPU, 1));
    for j = 1:numChunks
        idx2 = (j-1)*chunkSize + 1 : min(j*chunkSize, size(X_GPU, 1));

        % Calculate distances for each chunk
        D_chunk = pdist2(X_GPU(idx1, :), X_GPU(idx2, :), 'squaredeuclidean');

        % Store the result in a cell array
        D_parts{i, j} = gather(D_chunk); % gather to CPU if GPU memory is a constraint
    end
end

You can refer to the resources below for more information:

https://www.mathworks.com/matlabcentral/answers/630363-error-using-gpuarray-maximum-variable-size-allowed-on-the-device-is-exceeded