Remove rows from large MAT file

Hi, I have a large matrix stored in Mat FIle. I loaded the matfile:

db = matfile(myFile)

and got:

db = 

  matlab.io.MatFile

  Properties:
              Properties.Source: '.MyFile'
            Properties.Writable: true                                                                                                                                 
    Properties.ProtectedLoading: false                                                                                                                                                                                                                                              
                      hs: [216817664x532 uint32]      

I want to delete the lat row in hs field:

db.hs(216817664, :) = [];

But I got an error:

Requested 216817664x532 (429.7GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may
take a long time and cause MATLAB to become unresponsive. 

Is there any other way to remove rows from such large files?

[Image Analyst]

Perhaps memmapfile()?

[Zion Nahisi]

I don't think memmapfile() works on MAT file... this MAT file has another fields, so I can't use a dat file

[Ive J]

Have you tried tall arrays?

db = matfile(myFile)
t = tall(db.hs);
t(216817664, :) = []; % see write doc for saving this tall array 

[Zion Nahisi]

Same error, filed on

t = tall(db.hs);

since MATLAB cannot access the whole matrix for convert it to tall array.

Answer 1

[Vedant Shah]

Hi @Zion Nahisi,

MATLAB attempts to load the entire variable ‘hs’ into memory when a row deletion is requested but given the size of the matrix, this operation is not feasible.

A possible workaround is to create a new MAT-file that excludes the last row of the original matrix. To avoid memory overflow issues, the data should be copied in manageable chunks. The following code snippet demonstrates how this can be achieved:

src = matfile('MyFile.mat'); 
dst = matfile('NewFile.mat', 'Writable', true); 

chunkSize = 10000; 
numRows = size(src, 'hs', 1); 
numCols = size(src, 'hs', 2); 
for i = 1:chunkSize:(numRows - 1) 
    lastRow = min(i + chunkSize - 1, numRows - 1); 
    dst.hs(i:lastRow, :) = src.hs(i:lastRow, :); 
end 

This approach ensures that only a portion of the data is loaded into memory at any given time, making the process efficient and scalable for very large datasets.

For more information, please refer to the documentation using the following commands in the MATLAB command line: 

web(fullfile(docroot, " /matlab/ref/matlab.io.matfile.html"));