error while trying to train network using multiple cpus

Hi, I'm following the super resolution image neural network example from the mathwork website. Everything works fine except utilization of multiple cpus. My GPU is old. So, I've given up gpu option. But when I try to use multiple cpus by putting following options in the 'trainNetwork' I get errors.

,...
'ExecutionEnvironment', 'parallel');

Errors are as follows:

Starting parallel pool (parpool) using the 'local' profile ...
connected to 8 workers.
Error using trainNetwork (line 154)
MiniBatchable Datastore is not distributable.

Error in trainVDSR (line 75)
    net = trainNetwork(source,layers,options);

Caused by:
    Error using nnet.internal.cnn.DataDispatcherFactory.createDataDispatcher (line 100)
    MiniBatchable Datastore is not distributable.

Any idea why it's complaining?

Thank you, C

Answer 1

[Joss Knight]

The error means that the vdsrImagePatchDatastore does not support parallel training. This is a custom datastore created for this example. You could try adding the Distributable interface yourself to get it working.

I'll admit the error message is confusing and we should do something about it.

[Amal Abdulrahman]

Hi Joss, How can I add the Distributable interface ? I checked the link and it is really confusing. Can you give me clear steps to do so ? Thanks

[Joss Knight]

What is it that isn't clear? The doc link explains the steps - add the matlab.io.datastore.PartitionableByIndex mixin, add the partitionByIndex function, and it gives an example. Click on the link for partitionByIndex and it will describe the inputs and outputs of that function. If you can be specific about what you don't understand, I can elaborate.

[Amal Abdulrahman]

I am trying to run the example mentioned in the above. I did the modification suggested in the doc that you provided, but now I got another error message : Error using trainNetwork (line 154) Incorrectly defined MiniBatchable Datastore. Error in partitionByIndex method of F:\UPWORK\CNN\vdsrImagePatchDatastore.m at line 357: No appropriate method, property, or field 'InputImds' for class 'vdsrImagePatchDatastore'.

Error in test_image_enlargment (line 73) net = trainNetwork(source,layers,options);

Caused by: Error using nnet.internal.cnn.validateMiniBatchDatastore>checkPartitionByIndex (line 90) Incorrectly defined MiniBatchable Datastore. Error in partitionByIndex method of F:\UPWORK\CNN\vdsrImagePatchDatastore.m at line 357: No appropriate method, property, or field 'InputImds' for class 'vdsrImagePatchDatastore'.

What shall I do ? Thank you.

[Joss Knight]

You should debug your code using MATLAB's debugging tools. According to this error message, you are attempting to access a property in your partionByIndex function that doesn't exist.

[Matheus Ferreira]

I have the same problem, could you please be more specific on how to debug the code?

[Matheus Ferreira]

Error using trainNetwork (line 154) Incorrectly defined MiniBatchable Datastore. Error in partitionByIndex method of /media/matheus/dados/DeepLearning/TestSemanticSegmentationMatLab/PixelLabelImagePatchDatastore.m at line 223: No appropriate method, property, or field 'InputImds' for class 'PixelLabelImagePatchDatastore'.

Error in TestSemanticSegmentationMatLab (line 56) [net,info] = trainNetwork(ds,lgraph,options);

Caused by: Error using nnet.internal.cnn.validateMiniBatchDatastore>checkPartitionByIndex (line 90) Incorrectly defined MiniBatchable Datastore. Error in partitionByIndex method of /media/matheus/dados/DeepLearning/TestSemanticSegmentationMatLab/PixelLabelImagePatchDatastore.m at line 223: No appropriate method, property, or field 'InputImds' for class 'PixelLabelImagePatchDatastore'.

[Joss Knight]

The documentation instructs you to implement the function partitionByIndex, and it gives an example. You are not supposed to just copy the example. In this case, the custom datastore the example is using owns a property InputImds which holds the ImageDatastore object containing the input data. The vdsrImagePatchDatastore does not contain such a property, it holds onto the data in a different way. If you are going to customize it, you'll need to investigate how it works in order to write the method appropriately.

The premise of my answer is that if you wish for this example to support parallel training, you need sufficient MATLAB programming skills to be able to author a custom class. If not, your only hope (other than learning such skills) is to hope that another person frequenting this forum has already done this and can post their code.

[Matheus Ferreira]

Thank you for your prompt response. Much appreciated. I tried to implement the partitionByIndex function but I could not understand where should I insert the indices parameter in this function. I believe I should follow the implementation of shuffle, but partitionByIndex has another parameter called indices (partitionByIndex(ds, indices).

methods

function dsrand = shuffle(ds)

% To shuffle, shuffle underlying ImageDatastores
dsrand = copy(ds);
ord = randperm( numel(ds.InputImageDatastore.Files) );

dsrand.InputImageDatastore.Files = dsrand.InputImageDatastore.Files(ord);
dsrand.InputPixelLabelDatastore.shuffle(ord);
end  
end

Could you please post an example of Minibatch datastore that supports parallel computing? Thank you for your help!

[Joss Knight]

Sorry, this question is tantamount to asking me to write the code for you, and I unfortunately cannot oblige. Perhaps someone else is lurking here who can help?