Effacer les filtres
Effacer les filtres

Matlab Develop Custom Mini-Batch Datastore Example Bug?

3 vues (au cours des 30 derniers jours)
Yong le 22 Mar 2022
Réponse apportée : Dinesh le 22 Déc 2023
I was following the example of developing a custom mini-batch datastore through this link: https://www.mathworks.com/help/deeplearning/ug/develop-custom-mini-batch-datastore.html
In the function read of the example, after a file in the datastore is read in, the CurrentFileIndex is incremented by one. However, I was wondering if a client keeps reading the datastore that when it goes BEYOND the NumObservations, it will generate an error. To fix this problem, at the beginning of the read function, I would add the following:
if hasdata(ds)==false
I'd appreciate very much if you could share your thoughts about this issue.

Réponses (1)

Dinesh le 22 Déc 2023
Hi Yong,
You're on the right path to ensure that your custom mini-batch datastore handles the end-of-data scenario gracefully. In MATLAB, when a datastore's read method is called and it has reached the end of the data (when "CurrentFileIndex" exceeds "NumObservations"), it is indeed good practice to check if there is more data to read using "hasdata". If there isn't, resetting the datastore with "reset(ds)" is a standard way to cycle back to the start of the data. This allows for continuous data reading, which is particularly useful during machine learning model training that requires multiple passes over the dataset (epochs).
Implementing this check and reset mechanism inside your read function makes your datastore robust and self-sufficient, as it can manage its state without relying on external control to avoid reading beyond the available data. Remember that if your datastore is going to be used in a parallel processing context, make sure that the reset operation is appropriately managed to prevent conflicts between different workers accessing the datastore. The documentation link that you mentioned will help you with everything.




Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by