MATLAB Answers

Speed up loading a single variable from a large mat file

49 views (last 30 days)
Wonsang You
Wonsang You on 18 Jul 2017
Commented: Walter Roberson on 25 Sep 2018
I was trying to load a single variable 'data' from a large .mat file (>2GB) using the following command in MATLAB R2016b on Mac OS.
data = load('db.mat','-mat','data');
However, it was too slow to load the single variable. Note that the variable 'data' is not a matrix but a structure. I already refer to the previous discussion but it didn't give solution. Could anyone please suggest an alternative way to speed up loading a single variable from a large mat file? Thank you for your help in advance.
  1 Comment
per isakson
per isakson on 18 Jul 2017
  • ">2GB" means version '-v7.3', I assume.
  • '-nocompression' improves speed, but not that much
  • AFAIK: '-v7.3' is slow. (I didn't test that since R2013a.)
  • Try HDF5 files with 'ChunkSize' Default: Not chunked

Sign in to comment.

Accepted Answer

Jan
Jan on 18 Jul 2017
Using a large MAT file and extracting a single variable takes time. There is no magic acceleration. The compression of the data of v7.3 files requires to decompress the complete or at least a large part of the data to extract you variable.
The clean solution would be not to use MAT files for this job.
  2 Comments
Walter Roberson
Walter Roberson on 25 Sep 2018
With -v7.3 mat files there is now a -nocompression option
This applies only for -v7.3 .
It is possible to have a -v7 (not -v7.3) .mat file that is over 2 gigabytes as long as any one variable is not over 2 gigabytes. However, compression would happen automatically for -v7 mat files as well.
Sometimes what helps is to use matfile with -v7.3 files . This can help in accessing members of a struct array. Reading between the lines, it might be able to load a single field of a struct array, but anything below that would be loaded as a whole.
For pure numeric arrays sometimes it makes sense to write them as pure binary files and memmapfile() . This can also be done for struct arrays, but in each case the data written per entry must be fixed size.

Sign in to comment.

More Answers (0)

Tags

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by