Warning: GPU is low on memory

29 views (last 30 days)
mohd akmal masud
mohd akmal masud on 21 Jun 2022
Commented: 은석 최 on 28 Jul 2022 at 8:13
Dear All,
I got this error when run data trained for Unet3Dlayer.
Warning: GPU is low on memory, which can slow performance due to additional data transfers with main memory. Try reducing the
'MiniBatchSize' training option. This warning will not appear again unless you run the command:
warning('on','nnet_cnn:warning:GPULowOnMemory').
Error using trainNetwork
GPU out of memory. Try reducing 'MiniBatchSize' using the trainingOptions function.
Caused by:
Error using gpuArray/cat
Out of memory on device. To view more detail about available memory on the GPU, use 'gpuDevice()'. If the problem persists,
reset the GPU by calling 'gpuDevice(1)'.
THIS IS MY CODE BELOW.
clc
clear all
close all
%testDataimages
DATASetDir = fullfile('C:\Users\USER\Downloads\NEW 3D U NET 128X128');
IMAGEDir = fullfile(DATASetDir,'ImagesTr');
volReader = @(x) matRead(x);
volds = imageDatastore(IMAGEDir, ...
'FileExtensions','.mat','ReadFcn',volReader);
% labelReader = @(x) matread(x);
matFileDir = fullfile('C:\Users\USER\Downloads\NEW 3D U NET 128X128\LabelsTr');
classNames = ["background", "tumor"];
pixelLabelID = [0 1];
% pxds = (LabelDirr,classNames,pixelLabelID, ...
% 'FileExtensions','.mat','ReadFcn',labelReader);
pxds = pixelLabelDatastore(matFileDir,classNames,pixelLabelID, ...
'FileExtensions','.mat','ReadFcn',@matRead);
volume = preview(volds);
label = preview(pxds);
up1 = uipanel;
h = labelvolshow(label, volume, 'Parent', up1);
h.CameraPosition = [4 2 -3.5];
h.LabelVisibility(1) = 0;
h.VolumeThreshold = 0.5;
volumeViewer(volume, label)
patchSize = [128 128 64];
patchPerImage = 16;
miniBatchSize = 8;
patchds = randomPatchExtractionDatastore(volds,pxds,patchSize, ...
'PatchesPerImage',patchPerImage);
patchds.MiniBatchSize = miniBatchSize;
dsTrain = transform(patchds,@augment3dPatch);
volLocVal = fullfile('C:\Users\USER\Downloads\NEW 3D U NET 128X128\imagesVal');
voldsVal = imageDatastore(volLocVal, ...
'FileExtensions','.mat','ReadFcn',volReader);
lblLocVal = fullfile('C:\Users\USER\Downloads\NEW 3D U NET 128X128\labelsVal');
pxdsVal = pixelLabelDatastore(lblLocVal,classNames,pixelLabelID, ...
'FileExtensions','.mat','ReadFcn',volReader);
dsVal = randomPatchExtractionDatastore(voldsVal,pxdsVal,patchSize, ...
'PatchesPerImage',patchPerImage);
dsVal.MiniBatchSize = miniBatchSize;
inputSize = [128 128 64];
numClasses = 2;
encoderDepth = 2;
lgraph = unet3dLayers(inputSize,numClasses,'EncoderDepth',encoderDepth,'NumFirstEncoderFilters',16)
figure,plot(lgraph);
%analyzeNetwork(lgraph1)
%analyzeNetwork(lgraph2)
maxEpochs = 100;
options = trainingOptions('adam', ...
'MaxEpochs',maxEpochs, ...
'InitialLearnRate',1e-3, ...
'LearnRateSchedule','piecewise', ...
'LearnRateDropPeriod',5, ...
'LearnRateDropFactor',0.97, ...
'ValidationData',dsVal, ...
'ValidationFrequency',200, ...
'Plots','training-progress', ...
'Verbose',false, ...
'MiniBatchSize',miniBatchSize);
doTraining = true;
if doTraining
modelDateTime = datestr(now,'dd-mmm-yyyy-HH-MM-SS');
[net,info] = trainNetwork(dsTrain,lgraph,options);
save(['trained3DUNet-' modelDateTime '-Epoch-' num2str(maxEpochs) '.mat'],'net');
else
load('trained3DVNet-07-Jun-2022-13-45-30-Epoch-250.mat');
end
ANYONE CAN HELP ME.
BECAUSE I THINK MY MEMORY GPU IS SUFFICIENT.
  3 Comments
Jan
Jan on 21 Jun 2022
@mohd akmal masud: Then the conclusion is, that the GPU on the laptop has more RAM than the machine cause the error. The solution is to use the laptop for this job. Or install a graphic card with more RAM.
Did you follow the advice to use the 'MiniBatchSize' option?
Sorry, this answer is trivial, but I do not see a chance to create a more useful answer.

Sign in to comment.

Accepted Answer

Joss Knight
Joss Knight on 22 Jun 2022
A 3-D U-net is a very large model. Try reducing patchSize, patchPerImage, miniBatchSize and inputSize.

More Answers (1)

Image Analyst
Image Analyst on 21 Jun 2022
Edited: Image Analyst on 21 Jun 2022
You can try reducing the minibatchsize to 4 or 2.
I had the same problem and a Mathworks deep learning expert engineer helped us figure it out. I had a GPU (two actually) in my computer and the "ExecutionEnvironment" was set up as auto, the default, so it tried to use the GPU. The problem was that my GPU was only 16 GB and we got the "Out of memory" error, even for mini-batches of size 2. So we changed the ExecutionEnvironment option to CPU and it worked. Even though I had only 32 GB of RAM on my computer, I actually had hundreds of GB of memory because if the CPU runs out of RAM it uses disk space as "virtual memory".
% Train network - 10000 epochs and 10 Augmentations per mask.
options = trainingOptions('adam', ...
'InitialLearnRate', 1e-3, ...
'MaxEpochs', 7000, ...
'VerboseFrequency', 10, ...
'MiniBatchSize', 16, ...
'Shuffle', 'every-epoch', ...
'Plots', 'training-progress', ...
'ExecutionEnvironment','cpu');
% 'CheckpointPath', checkpointPath );
  5 Comments
은석 최
은석 최 on 28 Jul 2022 at 8:13
Thank you for your advice! I set the mini-batch size to 2, so it's resolved.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by