A problem when using "multi-gpu" as "ExecutionEnvironment" for training a CNN
6 views (last 30 days)
I am experiencing weird problems when I use the “multi-gpu” as the “ExecutionEnvironment” in the training option for training a CNN.
I am using the simple CNN example shown in the MATLAB’s website: https://www.mathworks.com/help/deeplearning/ug/create-simple-deep-learning-network-for-classification.html
I only added the “multi-gpu” as the “ExecutionEnvironment” and changed the “MaxEpochs” to 50 in the training option.
I am using two of NVIDA RTX 2080 Ti 11GB from the same manufacturer (Galax). The two graphic cards are the same models from the same manufacturer.
In MATLAB R2020a, the CNN example works fine with faster training speeds when I use the “multi-gpu” as the “ExecutionEnvironment” in the training option. The training progress plot is attached below.
But in MATLAB R2022a, the training progress works really weird when I run the same code. The validation data curve goes high while the training data curve fluctuates at the middle and never goes up. The screenshot showing this is attached below.
And the validation accuracy in the command window after the training is also bad (63.92%) while the validation accuracy only in the training progress plot is good.
This issue also happens in other MATLAB versions later than R2020a. When I tested with a single GPU separately and individually, the training works fine.
Can anyone please help me with this issue? I need to use my two GPUs for the "multi-gpu" training in MATLAB R2022a for my work so I first need to fix this weird problem in the example.
I would appreciate any help. Thank you very much.