strange running time behavior when calculating on the gpu (CUDA)

Hey,

I'm trying to learn to use the gpu (CUDA). However, sometimes I get really bad running times. I tried to find an example as simple as possible to illustrate what I mean:

Therefore I multiply a matrix A with a vector v (once per iteration). In the next step I multiply the matrix again with the new vector (result vector of last step). On the CPU the running time increases with the iteration steps (as expected) while on the GPU at some point the running time seems to explode.

My code:

clear;

A = rand(2^14,2^14);
v=rand(2^14,1);
kmax = [250, 500, 1000, 2000, 4000];

for j=1:length(kmax)
   [tcpu(j), tgpu(j)] = cpuvsgpu(A, v, kmax(j));
end

As a result I get:

tcpu =

   25.3472   49.9852  100.8437  201.6964  407.3839

 tgpu =

    0.0969    0.0103   12.3973   37.7690   88.4201

My function:

function [tcpu, tgpu] = cpuvsgpu(A, v0, kmax)

% CPU
v = v0;

tic
for k=1:kmax
  v=A*v;
end
tcpu = toc;

% GPU
g_A = gpuArray(A);
g_v = gpuArray(v0);

tic
for k=1:kmax
  g_v=g_A*g_v;
end
tgpu = toc;

end

As you can see in the result the running time using the cpu doubles with the steps while the running time on the gpu behaves really weird for more than 500 steps. Am I using the tools to calculate on the GPU somehow totally wrong?

Thanks for any help

Answer 1

[Joss Knight]

The problem is asynchronous execution - you're missing a wait(gpuDevice). I got these timings:

tgpu =
    0.0204    0.0263   11.5714   35.2226   82.5351

Then I added a couple of lines to your code:

gpu = gpuDevice; % ADDED HERE
tic
for k=1:kmax
  g_v=g_A*g_v;
end
wait(gpu); % ADDED HERE
tgpu = toc;

And the result was these timings:

tgpu =
    5.9143   11.8319   23.6482   47.3094   94.6128

You see, all you were timing for the first 1000 or so iterations was how fast MATLAB could loop round launching more GPU kernels asynchronously.

You don't need to use wait to use the GPU - just to get sensible timings. Alternatively, use gputimeit. See Measure performance on the GPU.