GPU for ConvNN training out of memory

Hi.

I am working with applying one of the MATLAB neural network examples to a data set that I have. My operating system is MacOS Sierra 10.12.5. When I run the program on the CPU there are no errors. However, when I run it on the GPU I am getting an error:

"Out of memory on device. To view more detail about available memory on the GPU, use 'gpuDevice()'. If the problem persists, reset the GPU by calling 'gpuDevice(1)'."

CUDADevice with properties:

                      Name: 'GeForce GT 750M'
                     Index: 1
         ComputeCapability: '3.0'
            SupportsDouble: 1
             DriverVersion: 8
            ToolkitVersion: 8
        MaxThreadsPerBlock: 1024
          MaxShmemPerBlock: 49152
        MaxThreadBlockSize: [1024 1024 64]
               MaxGridSize: [2.1475e+09 65535 65535]
                 SIMDWidth: 32
               TotalMemory: 2.1470e+09
           AvailableMemory: 182575104
       MultiprocessorCount: 2
              ClockRateKHz: 925500
               ComputeMode: 'Default'
      GPUOverlapsTransfers: 1
    KernelExecutionTimeout: 1
          CanMapHostMemory: 1
           DeviceSupported: 1
            DeviceSelected: 1

The code is below. I was suggested to run smaller batch sizes, but I am already running 1 batch size and it does not work.

rootFolder = 'Desktop/Neural_Net/T_v1/';
categories = {'Good', 'Bad'};
Training_path = [rootFolder, 'Training/'];
rootFolder_Test = [rootFolder, 'Test/'];

TrainingSet = imageDatastore(fullfile(Training_path, 
categories), 'LabelSource', 'foldernames');

TestSet = imageDatastore(fullfile(rootFolder_Test, 
categories), 'LabelSource', 'foldernames');

tbl_train = countEachLabel(TrainingSet)
minSetCount = min(tbl_train{:,2}); % determine the smallest 
amount of images in a category
TrainingSet = splitEachLabel(TrainingSet, minSetCount, 
'randomize');
tbl_train_cut = countEachLabel(TrainingSet)

alex = alexnet;
layers = alex.Layers;
clear alex, 
layers(23) = fullyConnectedLayer(2);
layers(25) = classificationLayer; % rms different sgdm 
opts = trainingOptions('sgdm',  'InitialLearnRate', 0.001, 
'MaxEpochs', 1, 'MiniBatchSize', 1, 'ExecutionEnvironment', 
'gpu', 'OutputFcn', @plotTrainingAccuracy);
TrainingSet.ReadFcn = @readAndPreprocessImage;
myNet = trainNetwork(TrainingSet, layers, opts);
disp('Past myNet')
TestSet.ReadFcn = @readAndPreprocessImage;
predictedLabels = classify(myNet, TestSet); % PRobability 
normalized. Observer the softmax layer configure to output 
sofmax
accuracy = mean(predictedLabels == TestSet.Labels)
% High specificity 
function plotTrainingAccuracy(info)

persistent plotObj

if info.State == "start"
    plotObj = animatedline;
    xlabel("Iteration")
    ylabel("Training Accuracy")
elseif info.State == "iteration"
    addpoints(plotObj,info.Iteration,info.TrainingAccuracy)
    drawnow limitrate nocallbacks
end

end

function Iout = readAndPreprocessImage(filename)

          I = imread(filename);

          % create an RGB image. 
          if ismatrix(I)
              I = cat(3,I,I,I);
          end

          % Resize the image as required for the CNN. 
          Iout = imresize(I, [227 227]);           
      end

[Walter Roberson]

You might want to go down to single precision, especially on the GT 750M, as the double precision performance on it is only 1/24 of the single precision performance.

[Joss Knight]

single is the default, so that shouldn't be it. What is the result of gpuDevice when you first start up MATLAB? According to the above, you only have 182 MB of free memory, which is barely enough to hold a single copy of AlexNet.

[Ishtiaq Bercha]

Joss, I am using GTX 960m GPU and have a 3 layer CNN. I am getting the gpu out of memory error down to a mini batch size of 4. The GPU has a 2 GB RAM and 8 GB shared memory (although I am not sure how that benefits) on a 16GB ram alienware laptop. What are your recommendations. I am using medical images that are 512 x 512 x 3 in size. I am thinking about reducing the image size but are there any other ways I could get rid of the error? Thanks ---Ish Here is my code, btw.:

clc clear all % Train Network for Image Classification %Load the data as an ImageDatastore object.

        digitDatasetPath = fullfile('C:','Users','ihbercha', ...
            'Desktop','Machine Learning','Proj_F','Img_Data');
        imds = imageDatastore(digitDatasetPath, ...
            'IncludeSubfolders',true, ...
            'LabelSource','foldernames');

% The training set has 150 % images and the testing set has the remaining images from each %label % (260 x 2 total images) .

        numTrainingFiles =150;
        [imdsTrain,imdsTest] = splitEachLabel(imds,numTrainingFiles,'randomize');

% Define the convolutional neural network architecture.

        layers = [ ...
            imageInputLayer([512 512 3])

            convolution2dLayer(3,8,'Padding',1)
            reluLayer
            maxPooling2dLayer(3,'Stride',3)

           convolution2dLayer(3,16,'Padding',1)
            reluLayer
            maxPooling2dLayer(3,'Stride',3)

            convolution2dLayer(3,32,'Padding',1)
            reluLayer
            maxPooling2dLayer(3,'Stride',3)

            fullyConnectedLayer(2)
            softmaxLayer
            classificationLayer];

Set the options to the default settings for the stochastic gradient descent with momentum. Set the maximum number of epochs at 20, and start the training with an initial learning rate of 0.01.

        options = trainingOptions('sgdm', ...
            'MaxEpochs',12,...
            'InitialLearnRate',1e-2, ...
            'Verbose',0, ...
            'Plots','training-progress','MiniBatchSize',4, 'ExecutionEnvironment','gpu');
% Train the network. 

        net = trainNetwork(imdsTrain,layers,options);

% Run the trained network on the test set, which was not used to train the % network, and predict the image labels (digits).

        YPred = classify(net,imdsTest);
        YTest = imdsTest.Labels;
     cf  = confusionmat(YTest, YPred)
% Calculate the accuracy. Accuracy is the ratio of the number of true labels 
% in the test data matching the classifications from |classify|, to the number 
 % of images in the test data.

        accuracy = sum(YPred == YTest)/numel(YTest)

Answer 1

[Joss Knight]

You're going to struggle massively with a small mobile chip like the 750M with only 2GB of memory and a large network like AlexNet, especially when that memory is being shared by the graphics. MATLAB R2017a's deep learning support does memory paging back to CPU to improve the capacity so if you are on 16b you may find upgrading helps (but it might not). Also, minimize the amount of memory being used by other processes by closing everything else and switching off all hardware graphics acceleration. This includes MATLAB which you should start up in software opengl mode ( matlab -softwareopengl ).

[David Brenes]

I do have MATLAB R2017a, is there anything that must be specified in the code if I want to have memory paging back to CPU?

[Joss Knight]

No, it just happens.

We have MacBook Pros with the 750M in our test system and we just can't reliably train AlexNet on them because memory is so short. The system takes so much of it and then AlexNet needs a good GB to train. Probably if you want to play with Deep Learning on this machine you're going to have to reduce the size of the network - or use the CPU.

Answer 2

[ioannisbaptista]

I have the same problem.

The message "Out of memory on device. To view more detail about available memory on the GPU, use 'gpuDevice()'. If the problem persists, reset the GPU by calling 'gpuDevice(1)'." appears when I try to evaluate my trained CNN. But in my case, I'm using a GeForce 1060M GTX 6GB RAM.

Here's a piece of my code:

testData = load('testROI.mat');
[test_imds, test_pxds] = pixelLabelTrainingData(testData.gTruth);
testDataSet = pixelLabelImageDatastore(test_imds, test_pxds);
unetPxdsTruth = testDataSet.PixelLabelData;
unetpxdsResults = semanticseg(test_imds, unet); % error is caused by this line
unetMetrics = evaluateSemanticSegmentation(unetpxdsResults, unetPxdsTruth);

The command gpuDevice() shows the results below:

CUDADevice with properties:

                      Name: 'GeForce GTX 1060'
                     Index: 1
         ComputeCapability: '6.1'
            SupportsDouble: 1
             DriverVersion: 9.2000
            ToolkitVersion: 9.1000
        MaxThreadsPerBlock: 1024
          MaxShmemPerBlock: 49152
        MaxThreadBlockSize: [1024 1024 64]
               MaxGridSize: [2.1475e+09 65535 65535]
                 SIMDWidth: 32
               TotalMemory: 6.4425e+09
           AvailableMemory: 5.0524e+09
       MultiprocessorCount: 10
              ClockRateKHz: 1670500
               ComputeMode: 'Default'
      GPUOverlapsTransfers: 1
    KernelExecutionTimeout: 1
          CanMapHostMemory: 1
           DeviceSupported: 1
            DeviceSelected: 1

As you can see, there are more than 5GB of free memoy but, for some reason I don't understand, the out of memory problem happens. The curious thing is it doesn't happen with 500 images the training stage, but happens with 100 images in the test evaluating stage. It's important to emphasize that this evaluation atempt uses a pretrained CNN, so the training data is not in the GPU memory while doing this.

Does anyone please knows what might be going on?

[Andrea Picciau]

Hi ioannisbaptista, I gave you my answer here: https://www.mathworks.com/matlabcentral/answers/468181-out-of-memory-issue-on-evaluating-cnns