How to save a huge matrix efficiently?

33 Ansichten (letzte 30 Tage)
DaWo
DaWo am 25 Nov. 2016
Kommentiert: DaWo am 28 Nov. 2016
Hello,
I want to save a huge matrix into a mat.file.
V = zeros(2000,2000,2000,'uint8');
loop filling the matrix;
save('test.mat','V','-v7.3');
clear all
This works very well! Then I want to get some parts of the variable V in the mat file.
data = matfile('test.mat');
V = data.V(:,:,1); % very slow ~20sec
V = data.V(:,1,:); % very slow ~20sec
V = data.V(1,:,:); % very slow ~20sec
V = data.V(1:500,1:500,1:500); % very fast although more bytes are extracted?! ~ 2sec
So now my question: Is there a possibility to save the matrix in an efficient way to get the slices faster? I already have some huge mat-files, where I can get slices of the volume very fast. This mat-files have the same properties like mine. They are a download, so I have no idea of the making. What I know is the structure of the variable and the type of saving (2182x2182x2242 uint8 and -v7.3). So I guess, it's possible to save the matrix elements in a more efficient way.
I hope you can help me. Best regards and thank you in anticipation!
PS: Why I need that? I'd seen, that normaly it is very fast to get a part of the matfile. So I can reduce working memory. Unfortunately I need this high resolution.
  1 Kommentar
Walter Roberson
Walter Roberson am 27 Nov. 2016
On my system I was able to create a 2000 by 2000 by 2000 uint8 matrix, barely. But saving it with save() was just toooo slow and I had to give it up.

Melden Sie sich an, um zu kommentieren.

Antworten (1)

Daniel kiracofe
Daniel kiracofe am 27 Nov. 2016
If you have enough memory and are using a 64 bit version of matlab, using load() to load the entire matrix into memory will be quicker. it will take some time to load the file, but after that each slice will be quicker.
If the matrices are sparse (i.e. they contain mostly zeros with a few non-zero values) then saving the file as a sparse matrix would be significantly quicker (https://www.mathworks.com/help/matlab/ref/sparse.html)
Beyond that, I don't really know any way to make matfile() run quicker. In order to get the slices, matfile() basically has to read the entire file. And that's just going to be slow because the file is big.
  1 Kommentar
DaWo
DaWo am 28 Nov. 2016
Thanks for the answer. The load function isn't my favorite solution, but for now it should work. ;)

Melden Sie sich an, um zu kommentieren.

Kategorien

Mehr zu Workspace Variables and MAT-Files finden Sie in Help Center und File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by