Large structure .mat files loading empty.

I have tried to load 2 different .mat files multiple times, and each time they load the fields contained in the structure, but each field just holds an empty array. These files are 1.67GB and 2GB. The PC I'm using has 16GB of RAM. MATLAB seems to get up to about 2.0GB of RAM usage as the file is being loaded, then once it's loaded it drops down to 600MB.

I'm fairly certain these files contain the appropriate data (or else they would not be the aforementioned sizes). They were also saved using the v-7.3 command, and via MATLAB 2018.

[Jan]

What is your question? All we know is that the contents of the loaded files differ from your expectations. Why are you convinced, that the files contain these data, although a load() seems to show, that this is not true? Can you provide the file? It does not work over this forum, I think, because the file size exceeds the allowed limits.

[Clifford Donovan]

Wow, after spending the past ~2 years learning MATLAB, a reply from Jan is like a celebrity sighting. I saved these files myself, is the first reason they should contain certain data. 2nd, even if that was completely false - a 2GB file just containing a few field names and empty arrays makes no sense. The file in windows explorer is around 2GB, but once the variable is loaded into MATLAB, the process is only taking ~600MB of RAM. Something doesn't add up.

I can provide you the file directly if you have a means for me to do so (e-mail for example and I can send a google drive link).

[Guillaume]

Yes, if you want us to look at a file, you'll have to provide a link to a (reputable) file sharing site such as google drive, dropbox, onedrive, etc.

Note that mat files don't have fields. They contain variables. Possibly some of these variables are structures which will have fields. Do you mean that all the variables in the mat files are empty (size 0x0)?

If you load a mat file into a variable (recommended) the variables are converted into fields. Perhaps that's what you mean.

Have you tried accessing the content with matfile?

I recently had a student come to see me with a bunch of mat files where one of the variable was corrupted. load did error but still loaded all the other variables into the workspace.

[Jan]

@Clifford Donovan: I'm happy if I can provide some help.

How did you save the files? Did you append variables to an existing MAT file by:

save(filename,variables,'-append')

? I'm not sure what happens, if you overwrite a variable by a smaller one. You can test this:

File = fullfile(tempdir, 'test.mat');
Q = rand(1, 100000);
P = rand(1, 1000);
save(File, 'Q', 'P', '-7.3')
dir(tempdir)
Q = rand(1, 10);
save(File, 'Q', '-append', '-7.3')
dir(tempdir)

Is the MAT file smaller after overwriting the formerly existing larger array? If so, this looks like a hint that the MAT files can be fragmented and contain zombie data. Then an almost empty file can require a lot of space.

Beside a copy of the MAT file, some code which reproduces the problem would be useful also. If the situation can be re-created by some dummy code as the above one, the problem might get clear. Maybe the files are simply corrupted?

[Clifford Donovan]

Both files should be able to be downloaded here:

https://1drv.ms/f/s!AtBiB1jt5a87iCoUVZBZraJM_CpH

[Guillaume]

Hum, yes as mat files, these files appear to be a huge waste of space, only containing a single structure of structures whose fields are all empty. The structure is only around 3000 bytes! However, there's a ton of metadata bloating the files.

So the question is: how did you create these files? Please show us the exact code. And what should actually be in the structure?

Your mat file are stored in the 7.3 mat-file format. It's a HDF5 format so you can see its content, including all the metadata with h5info. You'll see that the #refs# group is full of stuff that's not loaded by load. Unfortunately, I don't know enough about the mat file format to know if that's expected or not.

[Clifford Donovan]

Hi Guillaume,

It was saved quite simply via:

save ('ControlDataStruct.mat', 'ControlData', '-v7.3');

The fields within the structure should each contain multiple fields themselves. Some of those subfields contain another level of subfields, and some contain matrices of data. The files are the appropriate size for the amount of data that they contain - the structure variable took up a similar amount of RAM when originally in the workspace.

I see a similar result via h5info - the #refs# group indeed seems to contain the data at the endpoints of the fields of the structures - but without the appropriate field labels for organization.

[Guillaume]

I think you will have to contact Mathworks support for this. I don't think they've make the 7.3 format public so only they would be able to say what's going on here.

[Ramin Norouzi]

Did you find the reason for this problem? I have the same problem, and it is not just about large files. I have also small files (20 MB in size) which have this problem. I'm sure this problem did not exist before as I've opened and used my mat files many times before this problem, so the files where saved correctly and where accessible. But now they have only size on disk and when opened in MATLAB some variables (which are cell) are empty. I tried with different versions of MATLAB and on different PCs (Windows), but not fixed.

[Clifford Donovan]

I did not resolve the issue. I just used a previous backup of the data I had saved and had to reprocess it into MATLAB.

[Hussein Alaawieh]

One possible thing could be that you are saving instances of a user-defined class, so you need to run that class first (compile it) before loading the variable

[Elia Schiavon]

Are you writing and loading the files via the same MATLAB installation? I had a similar problem and in my case it was a toolbox dependencies issue, the MATLAB installation that generated the file had some toolboxes installed that were not on mine installation