Error with gpuArray. My code works on two computers but not a 3rd! "The data no longer exists" but it does!

I am doing some beamforming on GPU and it doesn't work on my newly installed computer. This code works on my workstation with a GTX 2080Ti, my laptop with an RTX A2000, but it wont run on my lab computer also with a GTX 2080Ti. This computer is a totally fresh install. The only thing that isn't new is the graphics card. I'm reusing the old one while a GTX4080 ships.

Here is the error from a smaller section of the command:

interp1(RcvData_gpu, delay_ind_gpu);

Error using gpuArray/interp1

The data no longer exists on the device.

But in the same moment i can test if those vars existOnGPU and I get true for both.

K>> existsOnGPU(RcvData_gpu)

existsOnGPU(delay_ind_gpu)

ans =

logical

1

ans =

logical

1

What gives? Any ideas?

Thanks in advance!

[Walter Roberson]

I wonder if somehow it is running out of memory?

[Daniel]

I dont think so... That generates a different error:

"Out of memory on device. To view more detail about available memory on the GPU, use 'gpuDevice()'. If the problem persists, reset the GPU by calling 'gpuDevice(1)'."

Which I am catching and handling by reducing the depth of the array.

It works just fine on identical hardware, or even lesser hardware. Its just this one new computer...

[Andrea Picciau]

Hey Daniel! Can you please post a bit more of what comes before that interp1 call?

[Daniel]

This are the 3 snippets of the code that generate the variables that are used in the interp1.

 %build return delay to each element from each point in beamformed
        %space. Same for each angle. In wavelengths
        return_delay_gpu = gpuArray(sqrt(reshape((x_gpu (:)' - Array_Positions_gpu )',length(za), length(xa),1,Nelements).^2 + z_gpu .^2));%factor in return lens

        %build receive aperture for each angle and beamformed pixel.
        %Generally a fixed F# versus depth. 
        RxAng_Correct_gpu  =gpuArray(abs(repmat(reshape((x_gpu (:)'-Array_Positions_gpu)',length(za), length(xa), 1, Nelements), 1, 1, na)...
            - repmat(reshape((z_gpu (:)'.*tand(IP.deg_rx))', length(za), length(xa),na), 1, 1, 1, Nelements))...
            <= reshape((z_gpu (:)'./cosd(IP.deg_rx)/2/AP.fnumber)', length(za), length(xa), na));

        %used later to minimize interp1 points
        RxAng_Correct_IDX_GPU = find(RxAng_Correct_gpu);

        % total tx/rx delay for each beamformed image pt converted to
        % samples
        total_delay_samples_gpu = ((TXDelay_gpu + return_delay_gpu - (2 * zacqstart)) * smps_wave);

        %ac is interpolation index offset to allow passing collapsed RF data (ie, 1 D array) into interp1.
        %for each beamformed point, the index represents where in the RF
        %data to grab a point. This is done for each array element RF line
        %and then will be summed.
        ac = reshape((ones(1,length(za)*length(xa)).*(0:na*Nelements-1)'*NptsRF)',length(za),length(xa),na,Nelements);
        %final index  in collapsed RF 1D format rawRF for interpolation
        %into beamformed space.
        delay_ind_gpu = gpuArray(ac+total_delay_samples_gpu + filt_shift);
%%
        t_gpu = gpuArray(((0:NptsRF-1)' + zacqstart)/IP.fs);
        %phase rotation line
        IQ_samp_gpu = gpuArray(complex(exp(-1i*wc*t_gpu)));
        %rotation for final image pts
        RotatePhase_gpu = gpuArray(complex(exp(1i*wc*delay_times_gpu)));

        %lump in admittance angle, window, RxAng and pahse correcton
        %AllCorrect_gpu = gpuArray(SenCutOff_gpu.*Rcv_window_gpu.*RxAng_Correct_gpu.*RotatePhase_gpu); % allocate GPU matrix

        AllCorrect_gpu = gpuArray(complex(zeros(size(RxAng_Correct_gpu)))); % allocate GPU matrix
        %just multiply non zero elements
        AllCorrect_gpu(RxAng_Correct_IDX_GPU) = RxAng_Correct_gpu(RxAng_Correct_IDX_GPU).*RotatePhase_gpu(RxAng_Correct_IDX_GPU); % allocate GPU matrix

%%

    RcvDataIn_gpu = gpuArray(double(RcvDataRawIn( :, transducer.ConnectorES, (j-1)*na+1: (j-1+bundle)*na))) ;
    RcvDataIn_gpu= filter(b_gpu,a_gpu,RcvDataIn_gpu);

    %put back in correct order
    %down mix data to Make it complex IQ

    %phase shift data and complex it
    RcvData_gpu = RcvDataIn_gpu.*IQ_samp_gpu;
    RcvData_gpu = filtfilt(bb_gpu,aa_gpu,RcvData_gpu)*2; % factor 2: to preserve the envelope amplitude
    %reorder indexes to prep for final interp1
    RcvData_gpu = reshape(RcvData_gpu, [NptsRF , Nelements, na, bundle]) ;
    RcvData_gpu = reshape(permute(RcvData_gpu,[1 3 2 4]), [Nsamples * Nelements, bundle]) ;
    % interpolate Rdata, sum up element contributions to a given pixel, and then compound image

    %interp1 data and then sum across elements for each beamformed pixel


    %%
    try
        BFOut_gpu = squeeze(sum(interp1(RcvData_gpu, delay_ind_gpu).*AllCorrect_gpu,4));
    catch gpuSqueezeEr
        disp(['Failed on BFOUT Squeeze:' newline 'Beamforming failed with the following message:' newline gpuSqueezeEr.message]);
        throw(gpuSqueezeEr);
    end

Failed on BFOUT Squeeze:

Beamforming failed with the following message:

The data no longer exists on the device.

[Oli Tissot]

Hi Daniel,

Thank you for reporting! I would like to reproduce the issue on my side to understand what is going on but some variables are not defined in your code snippet: x_gpu, Array_Positions_gpu, etc. What I need is some script that exhibit the issue and that I can reproduce on my side (everything must be defined there). Would it be it possible for you to get that and post it here for me? Then I could investigate the issue.

Cheers,

Oli

Answer 1

[Daniel]

I found the fix. But I don't get it. This code works in 2021b, but it does not work in 2022b. The one computer that it didnt work on had 2022b installed. I installed the older version and the code worked perfectly.

Does anyone from Mathworks know why interp1 fails like this in 2022b?