CNN Performance: CPU Consistency vs. GPU Variance - Why?

Hello everyone,

I have executed CNN code multiple times using rng(0) with CPU and consistently obtained the same result. However, when I attempted to accelerate the training process using the GPU, the results differed. Has anyone else faced this issue?

Thank you in advance!

Answer 1

[Ruth]

Hi Hamza,

Even when using "gpurng" some small non-deterministic behavior is expected to happen in the GPU during training, particularly during the backward pass. This is out of our control.

However the behavior should be deterministic in the forward pass and subsequently at prediction time.

If one sets the learning rate to be almost zero (e.g. 1e-16, meaning nothing is updated in the backward pass), the output of training (using "rng" and "gpurng") should look deterministic.

Best wishes,

Ruth

Answer 2

[Edric Ellis]

I'm not certain if it will make everything consistent, but note that random state on the GPU is controlled by the gpurng function.

[Hamza]

@Edric Ellis thanks for your answer, I have tried that also and didnt work!