How can I fix the CUDNN errors when I'm running train with RTX 2080?

58 Ansichten (letzte 30 Tage)
Hello,
Recently, I bought 2 RTX 2080 gpu. We currently have 2 RTX 2080 and 1 TITAN xp gpu. I want these gpu to work in parallel. But, I am constantly getting the CUDNN_STATUS_EXECUTION_FAILED error. I've added the cuda cache to the system requirements in the environment variables as 512mb, but I still get the same error.
Training across multiple GPUs.
Initializing image normalization.
|=======================================================================================================================|
| Epoch | Iteration | Time Elapsed | Mini-batch | Validation | Mini-batch | Validation | Base Learning|
| | | (seconds) | Loss | Loss | RMSE | RMSE | Rate |
|=======================================================================================================================|
Error using trainNetwork (line 150)
Unexpected error calling cuDNN: *CUDNN_STATUS_EXECUTION_FAILED.*
Error in segnet_deadpixel_train (line 86)
[net info] = trainNetwork(pximds,lgraph,options);
Caused by:
Error using nnet.internal.cnn.ParallelTrainer/train (line 68)
Error detected on worker 1.
Error using nnet.internal.cnn.layer.util.Convolution2DGPUStrategy/forward (line 16)
Unexpected error calling cuDNN: CUDNN_STATUS_EXECUTION_FAILED._
Expect this error, when working with a single gpu with the TITAN xp is working well but the RTX 2080,it's working slowly and giving the following warning.
Warning: GPU is low on memory, which can slow performance due to additional data transfers with main memory. Try reducing
the 'MiniBatchSize' training option. This warning will not appear again unless you run the command :
warning('on','nnet_cnn:warning:GPULowMemory').
I've tried MATLAB 2018a and 2018b versions with windows 10 64bit. Which version of MATLAB should I use to resolve these issues? Which versions of CUDA and CUDNN support RTX 2080? How can i fix this errors ?
  1 Kommentar
Sumaiya Ahmad
Sumaiya Ahmad am 6 Feb. 2024
Bearbeitet: Sumaiya Ahmad am 6 Feb. 2024
I got this error:
"cuDNN failed to return a valid plan for cudnn Backend Execute for convolution."
I reduced my validation size and it worked! No more errors!

Melden Sie sich an, um zu kommentieren.

Akzeptierte Antwort

Joss Knight
Joss Knight am 5 Dez. 2018
Bearbeitet: Joss Knight am 5 Dez. 2018
This a known issue. Before you start anything else run
try
nnet.internal.cnngpu.reluForward(1);
catch ME
end
That should clear the issue.
  5 Kommentare
Joss Knight
Joss Knight am 5 Dez. 2018
When your device has to start paging, it goes A LOT slower. This is just a backup to help your training finish, but it will slow things down.
Secondly, if you're on Windows, multi-GPU training is slower.
Thirdly, if the MiniBatchSize is 1, multi-GPU training is pointless because there is no way to divide the mini-batch between workers. Set the miniBatchSize to 2. But the 2080 will still run out of memory.
Aydin Sümer
Aydin Sümer am 5 Dez. 2018
Bearbeitet: Aydin Sümer am 5 Dez. 2018
Thanks for your reply. I understood better now.

Melden Sie sich an, um zu kommentieren.

Weitere Antworten (2)

Joss Knight
Joss Knight am 5 Dez. 2018
Regarding issues with memory, the Titan XP has 12GB of memory while the RTX 2080 has only 8GB. You'll need to reduce your MiniBatchSize further to train SegNet on the 2080.
  4 Kommentare
Aydin Sümer
Aydin Sümer am 5 Dez. 2018
I'm using that configuration. I think, it's original version.
lgraph = segnetLayers(imageSize,numClasses,'vgg16');
That's my training options.
options = trainingOptions('sgdm', ...
'Momentum',0.9, ...
'InitialLearnRate',1e-3, ...
'L2Regularization',0.0005, ...
'MaxEpochs',100, ...
'MiniBatchSize',1, ...
'Shuffle','every-epoch', ...
'CheckpointPath', ConvnetFolder, ...
'VerboseFrequency',2,'ExecutionEnvironment','multi-gpu');
Joss Knight
Joss Knight am 5 Dez. 2018
I'm looking into this. It may be that the options to segnetLayers do not allow a small enough network for training on an 8GB GPU. You may have to edit the network manually to create a smaller one.

Melden Sie sich an, um zu kommentieren.


Julian Beckmann
Julian Beckmann am 7 Dez. 2021
I am getting the same error message (CUDNN_STATUS_EXECUTION_FAILED) trying to train a network on a local RTX 3070 gpu (using Matlab 2021a). The odd thing here is that I trained the same network structure with identical hyperparameters yesterday and it worked perfectly fine. However, starting this morning I receive the CUDNN_STATUS_EXECUTION_FAILED error message, whenever I try to start to train a network.
I literaly did not change anything regarding my hardware or software, the only thing I did was shutting my pc down yesterday in the evening and when I wanted to start training again this morning, I received the error message as described, which seems kind of odd to me.
I already tried Joss proposals to fix the issue, but it still persists.

Kategorien

Mehr zu GPU Computing finden Sie in Help Center und File Exchange

Tags

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by