GPU computing: data no longer available on device??

Hi all,

the following code gives me an error sometimes:

if isa(rawData,'gpuArray')
    rawData = gather(rawData);
    gpuSwitcheroo = 1;
else
    gpuSwitcheroo = 0;
end

The error is: Error using gpuArray/gather The data no longer exists on the device.

I really cant make any sense of this, since the if statement should already check whether or not the variable is on the GPU, right?? How could it possibly not be on the GPU anymore if that statement just resulted in the variable being of type 'gpuArray'??

This only occurs on a system with very limited GPU memory, but I checked and during that specific operation the memory is not exceeded! It is however beforehand, and as a result of that, rawData is automatically gathered, hence the usage of that if-statement. But then how can the statement be true, if rawData has already been gathered??

I'm really confused, hopefully someone knows whats going on.

Thanks!

Answer 1

[Edric Ellis]

That error occurs when MATLAB resets the GPU "context". This generally happens when you either select a different GPU device, or reset the current device. In both cases, the class of your data doesn't change, but the ability to retrieve the contents is lost. You need to use the function existsOnGPU to tell whether or not the data is still accessible, like this:

g = gpuArray(1:10);
isa(g, 'gpuArray') && existsOnGPU(g) % returns true
reset(gpuDevice) % force invalidation of the GPU "context"
isa(g, 'gpuArray') % still true
existsOnGPU(g) % false

[Matthias]

Ok, that definitely is a clue as to what's going on, but I can't explain how it happens. In my code I try to execute functions on the GPU and if that fails I switch to the CPU as follows:

try
    % Operation on the GPU
    result = myFunction(data);
catch ME
    switch ME.identifier
        case 'parallel:gpu:array:OOM'
            rawData = gather(data);
            result = myFunction(data);
            reset(gpuDevice);
            gpuDevice(selectedGPU);
            data = gpuArray(data);
        otherwise
            rethrow(ME)
    end
end

This works as intented (I think, it doesnt give an error at least). The error occurs a few operations down the line, when I call another function, which takes the variable 'data' as input argument. Internal to the function the variable is called 'rawData' and the if-statement from my initial question is executed. So this all happens after the device is reset and newly selected and the data input to the function is transfered to the GPU again. The only thing I can think of is that I use the variable name rawData earlier in the level above that of the function where the error occurs, but I get rid of that variable with clearvars, so the GPU hopefully doesnt remember it anymore?? Actually, that really can't be it since the GPU is reset during that try-catch thing from above.

If you have any idea what might cause this unexpected behavior let me know, in the meantime I will probably plaster my code with existsOnGPU in order to try to debug this.

Thanks!