File version 7.3 and fast way to extract data from it.

Hi

I am creating data files using V7.3 to read back certain data and update the file continuously.

This is what I am using.

save("deviceid_ "+deviceid,'live_data_downloaded_data','ref_data_timestamp','ref_data_sensordata','-v7.3')

The data I am saving is from an HTTP API, with sensor data per minute. Therefore, it can get fairly large.

When I call the API again, I do not want to be downloading identical sample data. Therefore I would like to check the last time stamp from the data and adjust my call to the API.

When use the following statement.

loadingdata = matfile("deviceid_ "+deviceid, "Writable", true);

I would like to read the last time stamp from live_data_downloaded_data, but the matfile() function, takes a very long time to load due to the large dataset.

Is there a better way to check the last time stamp for each file I am saving?

Answer 1

[Walter Roberson]

v7.3 files can be somewhat slow for composite data types in which the parts potentially have different data types. The underlying file structure works better on arrays of consistent data types.

And that means that for storage space and time efficiency, instead of using a nonscalar struct array, it can be better to break the data up into separate variables each of which is uniform data type. This might not be as convenient but it works.

Also the underlying file structure does not work efficiently for nested data types. This can drive changing representations in inconvenient ways. For example instead of a cell array in which each entry is a double array of different sizes, merge all of the doubles into a single array, and have other arrays that give information about the sizes and relative offsets. (Doing this efficiently when you need to be able to dynamically update array sizes can be a challenge.)

v7 files were designed by Mathworks using native Mathworks data structures. They are not always as efficient in space as you might hope, but they work pretty well.

v7.3 file structures were designed by an international committee that deal more with blocks of uniform data. The file structure was flexible enough that Mathworks was able to find ways to store matlab variables... but the ways are sometimes kludges.

[Dharmesh Joshi]

My HTTP API output a JSON, with multiple variable types. Generally these are all Integers. Do you have any example of what you mean by breaking up the structure?

Would it be better to create CSV or excel file for my large data? What i need is to be able to see the timestamp in my list, then use that times stamp as my start time when downloading fresh data.