Filter löschen
Filter löschen

preallocate array without initializing

23 Ansichten (letzte 30 Tage)
Ian am 30 Nov. 2018
Kommentiert: Walter Roberson am 2 Dez. 2018
Is there any way to preallocate a matrix without initializing it to either NaN's, zeros or ones?
I'm working with large image data, typically 1-4 billion pixels, and need to preallocate the array so I can read the data in in chunks. I don't need the array initialized because I will either fill the entire array if reading the data is successful, or throw an exception if it fails.
Preallocating the array with either zeros or NaN's takes matlab several seconds to initialize the array.
Allocating a large array like this in C++ returns immediately because it neither new(...) or malloc(...) bother to initialize the memory.
Is it possible to allocate the array either in a C++ mex file or using coder.ceval(...) to avoid the initialization time?

Akzeptierte Antwort

Steven Lord
Steven Lord am 30 Nov. 2018
If you want a matrix of class int16 filled with 0 elements, don't wrap a call to zeros in a call to int16. Instead use the typename input listed on the documentation page for the zeros function.
z = zeros(1e4, 1e5, 'int16');
Are you trying to read in a sequence of images that may take up more memory than you have avaialble (or a significant fraction of your available memory?) If so consider creating a tall array from an ImageDatastore and performing your analysis on the tall array. See this documentation page for more information on tall and datastore arrays.
  8 Kommentare
Ian am 2 Dez. 2018
Bearbeitet: Ian am 2 Dez. 2018
Which only shows how gray my beard is! Thanks for your comment. This was not a hardware feature of either the memory or the processors in the Z-80 and intel 8080 based systems I cut my teeth on.
For anyone like me both clueless about and interested in what "Demand-Zero-Paging" is and why it is important, here's a good brief description, in answer to a question about the concept:
If I understand it correctly, a request for a large block of zero'd memory can be memory-mapped to a single small segment of physical memory, and only redirected to a larger physical address space when actually written to.
That would explain why a call to a large quantity of zeros(...) is much faster than a call for nan(...) or ones(...).
Walter Roberson
Walter Roberson am 2 Dez. 2018
Some history . I have not found anything definitive as to when Demand Zero was introduced . Demand paging dates to 1961.
Demand Zero requires MMU which were not present in the intel 80* series until the 80286

Melden Sie sich an, um zu kommentieren.

Weitere Antworten (4)

Bruno Luong
Bruno Luong am 30 Nov. 2018
Bearbeitet: Bruno Luong am 30 Nov. 2018
MEX can do that, use for example this utility

per isakson
per isakson am 30 Nov. 2018
Bearbeitet: per isakson am 30 Nov. 2018
Try this
>> A(100,100) = 0;
>> whos A
Name Size Bytes Class Attributes
A 100x100 80000 double
I believe it is described by UndocumentedMatlab, but don't find it now. However, see the two first comments to the blog piece, which I linked to.
Maybe, more relevant
>> B(100,100)=uint8(0);
>> whos B
Name Size Bytes Class Attributes
B 100x100 10000 uint8

Ian am 30 Nov. 2018
A little experimenting has determined the following:
z = zeros(m,n);
is apparently quite fast, but only helpful if one needs a matrix of doubles.
clear z; tic(); z=zeros(10000,100000); toc()
Elapsed time is 0.000367 seconds.
>> whos z
Name Size Bytes Class Attributes
z 10000x100000 8000000000 double
>> clear z; tic(); z=int16(zeros(10000,100000)); toc()
Elapsed time is 5.164922 seconds.
>> whos z
Name Size Bytes Class Attributes
z 10000x100000 2000000000 int16
Unfortunately, converting the result to int16 is very slow and presumably uses about 5x final memory requirements in the process, which if the array is large, will cause problems.
PI's approach probably avoids creating the matrix first as doubles, but is also slow (but faster than the above):
>> clear z; tic(); z(10000,100000) = int16(0); toc()
Elapsed time is 1.023650 seconds.
>> clear z; tic(); z(10000,100000) = 0; toc()
Elapsed time is 5.692720 seconds.

Ian am 30 Nov. 2018
Bearbeitet: Ian am 30 Nov. 2018
BL is apparently correct -- it can be done in a MEX file -- and I have located a functional solution on Matlab File Eschange: 31362-uninit-create-an-uninitialized-variable-like-zeros-but-faster
It is a self-compiling MEX file which allows creation of matrices of any data type without initializing them.
clear z; tic(); tic(); z=uninit(23040,46080,'int16'); toc()
Elapsed time is 0.000231 seconds.
This solution is old (last updated 2011), but works in R2018a on MacOS and on Linux under R2017b.
It does leave the resulting matrix uninitialized.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by