h5read being slow for files written by matlab
9 Ansichten (letzte 30 Tage)
Ältere Kommentare anzeigen
Thorbjørn Erik Køppen Christensen
am 30 Nov. 2018
Beantwortet: Preethi Ayyamperumal
am 5 Dez. 2018
I have some raw data in hdf5 file format (lz4 compressed) this is really quick to read, whcih is nice ;-).
I then do some stuff to the data, and save it in new hdf5 files (noncompressed) which is unreasonable slow to read in comparison:
>> tic;rawpix = h5read('raw_data.h5','/h5/data/path',[1 1 1], [1 1 1]);toc
Elapsed time is 0.154878 seconds.
>> tic;manpix = h5read('treated_data.h5','/h5/data/path', [1 1 1], [1 1 1]);toc
Elapsed time is 0.400602 seconds.
The raw data file is 3.8GiB (including 8 billion uint32's)
the treated one is 5GiB (including 0.7 billion doubles)
Is there a way to get theese speeds the same?
I think lz4 compressing as I'm writing the files would solve the problem, But I don't know how to make matlab use lz4 commpresion while writing, as it's not documented on The h5write documentation site, any ideas?
0 Kommentare
Antworten (1)
Preethi Ayyamperumal
am 5 Dez. 2018
Currently, MATLAB supports only gzip compression for creating HDF5 dataset. Th compression levels are between 0-9.
Refer to the following link for applying gzip compression:https://www.mathworks.com/help/matlab/ref/h5create.html#buvazss-2
Please try this compression method to see if it resolves the performance issue.
0 Kommentare
Siehe auch
Kategorien
Mehr zu HDF5 finden Sie in Help Center und File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!