Effacer les filtres
Effacer les filtres

How to organize a datastore with multiple devices for multiple days?

5 vues (au cours des 30 derniers jours)
So, I have a data from 75 smart meters for range of 17 days.
Each day all 75 smart meters did some measurements.
How it is better to organize the data? I really like the idea of datastore() in Matlab. But don't know how to do it automatically for so many variables(tables)
If I create a timetable for each day of each smart meter, it would require me to create 75*17 =1275 separate tables.
And also I would like to analyze each day (with all smart meter measurements combined) so it will make the total number 1275 +17 = 1292.
Can you suggest a good option of organizing such big amount of data?
I am analyzing it for load forecasting using Machine Learning.
Thank you very much in advance!
  1 commentaire
dpb
dpb le 22 Mai 2022
That's really not much data at all in terms of observations -- unless there are gigabytes of data which it wouldn't seem there could be but a few measurements from each device.
I'd just collect the whole dataset in memory in a timetable and use retime and similar functions to do operations over the data. memapfile is just going to get in the way here.

Connectez-vous pour commenter.

Réponse acceptée

Magsud Hasanov
Magsud Hasanov le 14 Juin 2022
I just used array with all paths to all files and made datastore(), then did readall()

Plus de réponses (1)

Seth Furman
Seth Furman le 30 Mai 2022
Provided your data are already stored in files on disk in a common format, you can create a tall timetable and perform operations on it as if it were a large timetable.
  1 commentaire
Magsud Hasanov
Magsud Hasanov le 13 Juin 2022
Hello, after making path into all files, I just used datastore() and then readall(). It created the big dataset. The amount of data was enough small to use readall(). Thank you very much for your time!

Connectez-vous pour commenter.

Catégories

En savoir plus sur Large Files and Big Data dans Help Center et File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by