MATLAB Answers

reset(gpuDevice) does not work

58 views (last 30 days)
When I run the following code for values of n<5000 it runs just fine.
reset(gpuDevice);
n=5000;
a=gpuArray(rand(n));
b=gpuArray(rand(n));
tic
t=a'*a;
c=t\(a*b');
toc
But when I run it for n=5000 i get the error "Error using \ Call to Double LU on GPU failed with error status: unspecified launch failure."
If I try running the program again for any small value of n I get the error
"Error using parallel.gpu.CUDADevice/reset
An unexpected error pccured during CUDA execution. The CUDA error was " all CUDA -capable devices are busy or unavailable"
Also, if I execute the following command
g=gpuDevice;
disp(g.FreeMemory)
I get the answer to be NAN
I am unable to run the reset(gpuDevice) command. It gives the same error as above.
  2 Comments
arnold
arnold on 20 Aug 2016
Hi,
I was just now looking for this error, I have a similar problem on a machine at work. I tried using
class(a)
ans =
gpuArray
b = medfilt2(a,[9,9]);
Error using medfilt2gpumex
Failure in GPU implementation.
unspecified launch failure.
Error in gpuArray/medfilt2 (line 37)
b = medfilt2gpumex(varargin{:});
Filter sizes [7,7] and smaller work but 9 upwards gives this error. After that, the gpuDevice also shows
availableMemory: NaN
From this I can't use the GPU anymore without restarting Matlab. This is too bad since the GPU is 20 times faster doing this kind of calculations.
Setup:
  • Matlab 2016a
  • Windows 10 Pro 64 (all updates)
  • Intel 5960X
  • 64GB RAM
  • GTX1080 with 372.54 (newest driver).

Sign in to comment.

Accepted Answer

Alison Eele
Alison Eele on 25 Aug 2016
I think you are experiencing the symptoms of a kernel execution time-out. If the GPU is connected to a monitor (or in Windows the GPU is running in WDDM mode) then the operating system imposes a kill time out on any operation taking place on the GPU. The intention of this timeout is to allow screen display to continue. When this kill takes place on a MATLAB process using the GPU it disrupts our connection to the GPU and typically requires a restart of MATLAB to fix.
You can find out if a kernel time-out is in place on your GPU by executing the gpuDevice command in MATLAB. One of the properties listed will be:
KernelExecutionTimeout: 0
If this is 0 then there is no execution timeout being applied to that card. If it is 1 then the operating system is imposing a timeout (the exact timeout varies by operating system).
Ways to work around the issue:
  • If possible do computation in smaller pieces to avoid the timeout.
  • If there are multiple GPU cards in the computer and the computer is Windows then some NVIDIA cards can be switched from WDDM (display) to TCC (compute) mode using the nvidia-smi utility. TCC cards do not have an execution timeout. You cannot connect a display to a TCC mode card.
  • In Windows it is possible to lengthen the timeout using registry edits though as with all registry edits this should be done with care. https://msdn.microsoft.com/en-us/Library/Windows/Hardware/ff569918(v=vs.85).aspx
  5 Comments
arnold
arnold on 25 Sep 2016
Hi Alison,
I think most of my computations should be splittable into parallel tasks as I do a lot of element wise computations of image stacks. Can you hint me in the right direction as to how to split that up, maybe with blockfunctions?
regards Arnold

Sign in to comment.

More Answers (2)

Yahya Zakaria mohamed
Yahya Zakaria mohamed on 29 Jun 2017
Thank You. I faced the same problem, I disconnected the second monitor and no error appeared.

Ricardo de Azevedo
Ricardo de Azevedo on 19 Nov 2019
Edited: Ricardo de Azevedo on 21 Nov 2019
I am facing the same problem now training an RNN and have tried both the TdrDelay to longer and the TdrLevel to 0.
Error:
Error using gpuArray/gatherAn unexpected error occurred during CUDA execution. The CUDA error was:CUDA_ERROR_LAUNCH_FAILED
The weird thing is the network trains for a while and then crashes, I can't really tell what triggers it.
(Using Matlab 2019b and latest NVIDIA drivers 441.20 for GTX 1080 Ti)
  3 Comments
giorgio toscana
giorgio toscana on 7 Apr 2020
Hi Ricardo,
I will try them.
If the problem persists i'll contact the support with those info.
Thank you very much for your quick reply.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by