GPU wrong results

Hello, I just started using GPU with matlab and I get some strange results. I run the following code

a = rand(2^10,2^10,'single');
G = gpuArray(a);
F = fft2(G);
err = isequal(gather(G), a)

Normally the value o 'err' should be 1, but in this case is not. If you call another function, e.g. instead of fft2 call atan, then the value of err is 1. Do you have any idea why this happens?

The GPU model is "GeForce GTX 580" and the matlab version is 2010b.

Thanks a lot.

[Jason Ross]

What is the value of err that you get?

[Georgios]

the value of err is 0 or 1 randomly

[Edric Ellis]

This is definitely unexpected, and you're right - there should be no difference between 'a' and 'gather(G)' for this case, and I don't see this problem on my C1060 card using R2011b. (As Jan points out, if you're doing computation on the GPU, then the values might diverge very slightly due to different algorithms being used).

Have you tried different NVIDIA drivers?

[Ben Tordoff]

Hi Georgios, I've been trying to reproduce this to no avail. Unfortunately I don't have access to a GTX 580, but I've tested running your code-snippet 1000 times in R2010b on each of: Quadro 600, GTX 460, GTX 480, Tesla C1060 and Tesla C2070. I can't get the test to fail even once on any of them.

What OS are you using? If you're not using the latest NVIDIA drivers, upgrading those is worth a try (as Edric suggests).

[Jan]

@Ben: I'm impressed by your GPU pool.

It seems to be obvious (we call such terms Wischi-Waschi in German, a kind of "limited scientfical power") that the GPU-processing is prone to drive problems and I assume is sensitve to overheating of the board also.

Is there a test-suite such that I can check my GPU before I trust the results e.g. for clinical decision making?

[Walter Roberson]

It appears that the German Wischi-Waschi came down to English as "wishy-washy". I had no idea of the origin of the English term before.

[Georgios]

Thank you all for your replies. The OS is ubuntu 10.4 and I think that we already use the latest driver (I will ask the administrator).

Answer 1

[Jan]

Do you expect absolutely equal results without rounding errors? This will happen as an exception only. Please check:

g = gather(G);
disp(max(abs(g(:) - a(:))));

[Georgios]

I checked

norm(g-a)

and sometimes is 0 and sometimes is of order 0.1 or 1. The result is random.

Also I found the 'isequal' comparison in matlab help under the GPU section. This means that they don't expect rounding errors!

[Jan]

Now I see: The F=fft2(G) is not involved in the comparison, it is just to stress the graphics board. This is strange.

[Jan]

A large FFT2 produces some heat on the GPU. Does this influence the local memory such that your data are simply corrupted?

[Georgios]

In my case, fft2 runs for less that a sec. I don't think this produces so much heat.

Answer 2

[Thomas]

Georgios, I ran it on our GPU pool

    gpuDevice
tic ()
a = rand(2^10,2^10,'single');
G = gpuArray(a);
F = fft2(G);
err = isequal(gather(G), a)
toc ();

The output was ans =

parallel.gpu.CUDADevice handle
Package: parallel.gpu

Properties:
                    Name: 'Tesla M2070-Q'
                   Index: 1
       ComputeCapability: '2.0'
          SupportsDouble: 1
           DriverVersion: 4.1000
      MaxThreadsPerBlock: 1024
        MaxShmemPerBlock: 49152
      MaxThreadBlockSize: [1024 1024 64]
             MaxGridSize: [65535 65535]
               SIMDWidth: 32
             TotalMemory: 5.6366e+09
              FreeMemory: 5.5281e+09
     MultiprocessorCount: 14
            ClockRateKHz: 1147000
             ComputeMode: 'Default'
    GPUOverlapsTransfers: 1
  KernelExecutionTimeout: 0
        CanMapHostMemory: 1
         DeviceSupported: 1
          DeviceSelected: 1

err =

     1

Elapsed time is 0.054194 seconds.

I am getting an err = 1 output..