Increase limit on Matrix size

Hi,

Presently matlab has put a limit of 10k on max array dimension. Matlab is not allowing me to use more than that. How can we increase it?

Thanks

[Walter Roberson]

Do you mean that you want ndims() of the array to be greater than 10000, or do you mean that you want one particular dimension of an array to be that large?

Did you accidentally use

A = zeros(1:10000)

instead of

A = zeros(1,10000)

[Walter Roberson]

I just created an array with over 100 million dimensions.

Answer 1

[Steven Lord]

There's a limit on the number of elements an array can have. With the 64-bit version of MATLAB that's a theoretical limit since you'd need a machine with hundreds of terabytes of memory to create one such array. But that limit is much, much more than 10000.

The old Student Edition of MATLAB that was distributed by Prentice-Hall (not the currently available Student Version from MathWorks) had a limitation of 128-by-128 on matrix size, but I believe that hasn't been for sale except maybe on eBay as a novelty for 15 years or so.

[I have a copy of the Student Edition version 5 for the Mac on my bookshelf, and to put its age into perspective the system requirements list System 7.1 or later (System 7.5 recommended) and the Notebook interface requires Microsoft Word 6.0. The manual inside is copyrighted 1997.]

You're going to need to clarify which version of MATLAB you're using and provide more details about what you're doing and what MATLAB is telling you before we can figure out where this limitation you've encountered is coming from.

[Rehan Rehan]

@Steven Lord

Its Matlab 2015 b , 64 bit, for windows 64 bit and license is academic.

[Walter Roberson]

Okay, and what was your command to create the array, and what was the error message?

How much RAM do you have installed?

What have you set your array size limit to in your preferences? http://www.mathworks.com/help/matlab/matlab_env/set-workspace-and-variable-preferences.html

[Rehan Rehan]

aaa=zeros(24000)

Error using zeros Requested 24000x24000 (4.3GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time and cause MATLAB to become unresponsive. See array size limit or preference panel for more information.

RAM is 4 GB

[Walter Roberson]

You can modify your preferences, according to the link I posted. That would allow you to use any hard drive swap space you have configured in your operating system to store part of the 4.3 gigabytes of array, as that array exceeds your RAM.

If you are working with arrays that large then you should be considering using sparse arrays.

You should also be asking whether you truly want a 24000x24000 square array, or if instead you intended to create a vector of length 24000, the syntax for which would be

aaa = zeros(240000, 1);

or

aaa = zeros(1, 24000);

[Ginaris Ajeng]

So, can I conclude that about this size of array it is dependent to the matlab version and size of our RAM?

[Walter Roberson]

The limits are:

• Prentice Hall version of MATLAB (book): 128 x 128
• 32 bit MATLAB: maximum array size is 2^31-1 bytes. This is an architectural limitation for compatibility with 32 bit architectures that reserve the top bit for privileged memory accesses (as SGI systems used to do.) The limit of 2^31-1 bytes is not strictly required for 32 bit MS Windows systems that have the appropriate XME hardware feature, so Mathworks could have pushed the limit up a bit
• 32 bit MATLAB: you run into difficulty creating arrays with more than about 2^27 dimensions. You might have been able to do 2^28-1 dimensions, perhaps; I did not check that. The header on the array has to store the size of each dimension, and eventually the amount of memory needed to do so reaches 2^31 bytes even though the amount of storage in the array might be small (think many many many 1's in the dimensions.)
• 64 bit MATLAB: no array can occupy more than 2^48-1 bytes, because all current versions of the x64 architecture use only 48 address pins.
• 64 bit MATLAB: the 2^48-1 byte limitation would apply to the dimension header as well, so you probably could not go beyond 2^45-1 dimensions.

Answer 2

[Nicole Rappaport]

my program needs to use a matrix of dimension 100,000 x 100,000 x 100,000. The responses I read above are too sophisticated for me. Could someone tell me a simple way to allow this? By the way, I am talking about MATLAB.

Thanks in advance,

Nicole

[Steven Lord]

An int8, uint8, or logical array that size requires a contiguous block of memory 1 petabyte in size (ask Google to compute "100000 times 100000 times 100000 bytes in petabytes") and a real double array that size will require 8 times as much memory. I feel safe in assuming that your computer doesn't have that much memory. You're going to need to rethink your approach.

[Walter Roberson]

It is not possible to do that unless the array is sparse.

Even if your array was uint8 or logical, it would require 10^15 bytes of memory, which exceeds 2^49 bytes, but the x64 architecture has a limit of 2^48 bytes on all known implementations.

The largest publicly known computer memory in the world is an HP server with 160 terabytes of memory https://www.forbes.com/sites/aarontilley/2017/05/16/hpe-160-terabytes-memory/ . Even with uint8 values, the array you want to create is more than 5 times larger than that. You would need 6 of those Largest Computer in the World to fit your array. And if your array was double precision, you would need 46 of them.

[Steven Lord]

To put it another way, Wikipedia states that 'as of March 2014 "the Library [U.S. Library of Congress] has collected about 525 terabytes of web archive data" and that it adds about 5 terabytes per month ("one terabyte = 1,024 gigabytes").' At that rate, the Library will have 1 petabyte (1000 terabytes) of web archive data:

>> startDate = datetime('March 1, 2014');
>> terabytesToAdd = 1000-525;
>> monthsToAdd = terabytesToAdd/5;
>> timeReaching1PB = startDate + calmonths(monthsToAdd)
timeReaching1PB = 
  datetime
   01-Feb-2022

a couple years from now.