Handling Big Data in MatLab
17 Ansichten (letzte 30 Tage)
Ältere Kommentare anzeigen
Buddhi Wimarshana
am 22 Jan. 2021
Kommentiert: Buddhi Wimarshana
am 24 Jan. 2021
Hi everyone,
I am working with large sized data and the followings are the steps I follow, but need your advice/help to correct the procedure? or make it better...
I receive the data as csv files (data is of mixed types like doubles, date-time ets.) and the whole data set spans across about 150 csv files each having more than 100MB of data. So, I combined them all into one matlab table so the table contains the whole data set. I can't (don't like to really) make matlab tables for each csv file, becuase of the continuity of the data across each csv file so I guess it would be harder to retrieve them later. A part of the table I created looks like below:
![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/496768/image.jpeg)
This Table is more than 2GB in size (26 columns and millions of raws). Then I saved this Table as a mat file so I can 'load' it back to workspace later and retrieve the necessary data segments as per requirements. For example, I may need to retrive and post-process data of 'CurrentA' and 'VoltageV' for Cycle_Index=1 & Step_Index=30.
My concerens are:
- this method is quite time taking, for example loading the mat file takes quite long, so wonder is there a better way? I saw using 'matfiles' we don't need to load the mat file to memory. But I am not sure this will allow me to retrieve data with much flexibility compared to original file loading to workspace method?
- not so sure about the total methodology I followed, is there a better way to handle these kind of large data in a better, efficient way ?
Many thanks in advance...
0 Kommentare
Akzeptierte Antwort
Steven Lord
am 22 Jan. 2021
Some of the tools and techniques described in the Large Files and Big Data category in the documentation may be of interest to you.
Weitere Antworten (0)
Siehe auch
Kategorien
Mehr zu Workspace Variables and MAT-Files finden Sie in Help Center und File Exchange
Produkte
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!