Matlab Develop Custom Mini-Batch Datastore Example Bug?
1 Ansicht (letzte 30 Tage)
Ältere Kommentare anzeigen
Hello,
I was following the example of developing a custom mini-batch datastore through this link: https://www.mathworks.com/help/deeplearning/ug/develop-custom-mini-batch-datastore.html
In the function read of the example, after a file in the datastore is read in, the CurrentFileIndex is incremented by one. However, I was wondering if a client keeps reading the datastore that when it goes BEYOND the NumObservations, it will generate an error. To fix this problem, at the beginning of the read function, I would add the following:
if hasdata(ds)==false
reset(ds);
end
I'd appreciate very much if you could share your thoughts about this issue.
0 Kommentare
Antworten (1)
Dinesh
am 22 Dez. 2023
Hi Yong,
You're on the right path to ensure that your custom mini-batch datastore handles the end-of-data scenario gracefully. In MATLAB, when a datastore's read method is called and it has reached the end of the data (when "CurrentFileIndex" exceeds "NumObservations"), it is indeed good practice to check if there is more data to read using "hasdata". If there isn't, resetting the datastore with "reset(ds)" is a standard way to cycle back to the start of the data. This allows for continuous data reading, which is particularly useful during machine learning model training that requires multiple passes over the dataset (epochs).
Implementing this check and reset mechanism inside your read function makes your datastore robust and self-sufficient, as it can manage its state without relying on external control to avoid reading beyond the available data. Remember that if your datastore is going to be used in a parallel processing context, make sure that the reset operation is appropriately managed to prevent conflicts between different workers accessing the datastore. The documentation link that you mentioned will help you with everything.
0 Kommentare
Siehe auch
Kategorien
Mehr zu Preprocess Data for Deep Neural Networks finden Sie in Help Center und File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!