slicing a char array seems to internally allocate far more memory than expected

My laptop has 64G ram.

I am reading a large text file (> 8G) with s=fileread(filename).

The resulting char array is

K>> whos('s')

Name Size Bytes Class Attributes

s 1x8656516488 17313032976 char

Grabbing everything but the first, e.g. 200 characters results in this:

K>> v = s(201:end);

Requested 8656516288x1 (64.5GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time and cause MATLAB to become unresponsive.

See array size limit or preference panel for more information.

It seems that slicing a char array allocates 8x(number of chars). Is this expected?

[[Yes, I know of plenty of workarounds -- datastores, reading it pieces, fscanfs, etc., but this code is already a workaround to deal with the glacial speed of readtable(), so, I don't have to have it working, it's just somewhat surprising]]

Answer 1

[Walter Roberson]

That indexing 201:end is not internally optimized as passing subsref 'type', '()', 'subs', {{201, ':', 8656516488}}) and expecting the internal implementation of subsref to handle the range.

What is done instead is that the colon operator 201:8656516488 is executed, producing a double precision array of length 8656516288 to hold the indices, and that array is passed to subsref. But that array of indices is 64 gigabytes...

Copy the first 200 elements out of the array, and then

s(1:200) = [] ;

which only needs creating a vector of length 200 of double precision indices.

[Dmitry Kaplan]

Thank you, Walter. After (oh, god!) more than 30 years of Matlab, I still get hit with these weird subsref issues.

[My real-real issue, is that this code is in the middle of an ugly parfor with many large ranges -- all to try to "fix" the 300-second readtable() of a 8G file. So much for my 4xSSD Raid0 :-/

Dealing with dozens of these files as a datastore with custom ReadFcn helped a bit, but that's a whole different topic].

[Walter Roberson]

@Dmitry Kaplan

See https://www.mathworks.com/matlabcentral/answers/1645930-what-is-the-fastest-and-smartest-way-to-import-and-manage-plot-many-text-files-in-matlab#answer_891795 for potential speed improvements compared to readtable()

[Dmitry Kaplan]

Thank you. That's exactly the road I was following. Matlab is getting better at the "tens of gigabytes", but isn't quite there with the "hundreds of gigs or more" yet. Multithreaded C (or typescript) shreds through that much data, but yes, that's an unfair comparison. Thank you again for your help.

Answer 2

[Sourav Karmakar]

Hey Dmitri,

I've tried to create a char array of size 1x8656516488, but as it exceeds the maximum array size, it throws an error message. For example,

>> s = char(1:8656516488);
Requested 8656516488x1 (64.5GB) array exceeds maximum array size preference (48.0GB). This might cause
MATLAB to become unresponsive.

As you have 64G RAM , it should produce the same error. Try creating the 's' char array in your matlab workspace and check whether you are getting the same error message or not. Because, slicing only 200 chars from the array, does not change significantly in the memory( see difference between sizes of 's' and 'v' ).

You can refer to the following document for more reference:

Resolve "Out of Memory" Errors - MATLAB & Simulink (mathworks.com)

Hope this helps!

[Dmitry Kaplan]

Thank you. I belive that Walter's answer below is getting to the crux of the problem. The real issue is that I can easily allocate two 8G int8 arrays, I simply can't copy huge pieces of one into another without the internal allocation of huge double arrays to hold the indices.