How do I write a 3D matrix to a file, and then read it back in?

I'm running 2D wavefield simulations over time and need to store the wavefield at each time-step. The whole wavefield matricies over time range up to 4200 x 2000 x 2000 in size, so quite large. Storing the whole wavefield over time in matlab is taking up all of its memory, so I'm resorting to writing them to files and then reading them back in when needed for calculations. How would I go about writing the 3D matrix to a file, then reading it back in when needed?

Any help is greatly appreciated, thank you!

[KSSV]

Read about saving data into .mat files. You can save 3D matrix one by one and read it one by one.

[Ralph Horsley]

Oh intersting, I'm not familiar with them so I'll do some reading, thank you!

[Ralph Horsley]

Wait, just to check, did you mean writing each 2D slice of the 3D matrix to a seperate .mat file? I can't find anything on writing 3D matrixicies to them.

[KSSV]

Not into multiple files.....into a single file.

[Ralph Horsley]

Oh that's brilliant, thank you! Does it store the values to their original precision?

[Ameer Hamza]

Ralph, .mat is native file format to save data in MATLAB. It store the values in exactly same precision, as stored in MATLAB workspace.

[Ralph Horsley]

You're right, they worked perfectly, thank you!

Answer 1

[Michael Croucher]

I haven't played with the matfile function in this way before so tried a little experiment that may also serve to help you use this functionality. No idea if this is the optimal way of using it though!

If your matrices are dense and double precision then they are indeed large! Each double is 8 bytes so the matrix is

4200*2000*2000*8/(1024*1024*1024) = 125.17 % Gigabytes

Let's do a demo on a smaller version of your problem. Say, 10 pages. Here's one way of creating some data one page at a time and then saving it to the matfile.

m = matfile('myFile3.mat');
x = 4200;y=2000; %size of each matrix
N=10;   % Number of pages
m.data = double.empty(x,y,0);  %Initialise an empty matrix in the matfile

for i=N:-1:1
   m.data(:,:,i) = rand(x,y);  %There's only ever one page of the 3D matrix in memory at any one time
end
    

You can subsequently get any submatrix of the data without loading it all by doing, for example,

m.data(1:5,1:5,3); % the first 5 x 5 sub matrix of page 3

You may notice that I am creating the matrix inside the .mat file in reverse order - starting from N=10 and going backwards. This is because I initially tried this

m = matfile('myFile3.mat');
x = 4200;y=2000; %size of each matrix
N=10;   % Number of pages
m.data = double.empty(x,y,0);

for i=1:N
   m.data(:,:,i) = rand(x,y);
end

and received the error message.

Error using mymatfile (line 7)
Variable 'data' has 2 dimensions in the file, this does not match the 3 dimensions in the
indexing subscripts. 

Which I wasn't expecting! It seems that setting m.data(:,:,1) makes m.data two dimensional. At this point I wonder if there is a better way to do this.

[Ralph Horsley]

Crikey that is an interesting method for it! I am using my universities bespoke computer lab for these simulations and the computers and servers are designed for handling and storing these large amounts of data so I may keep it simple with just storing them in mat files (though perhaps in single rather than double format), thank you for your effort though! One thing, I thought each double was 8 bits, not 8 bytes, I may be wrong though.

[Michael Croucher]

Definitely 8 bytes in a double. We can ask MATLAB:

>> a=1.0
a =
     1
>> whos
  Name      Size            Bytes  Class     Attributes

  a         1x1                 8  double

[Ameer Hamza]

Correct, double is 8 bytes. This describes how the arrangement of data in a double variable: https://en.wikipedia.org/wiki/Double-precision_floating-point_format