Array too large for wdenoise?

Hello,

I am struggling with denoising a large dataset. I have a parquet file that has 8712 columns and each column has 800000 rows (conceptually it is signal data with each column representing a signal). Also, each column has a variable name, "0", "1", "2", etc. Since it is so large, I thought it would be best to use a datastore. Here it what I have so far:

ds=parquetDatastore("file.parquet"); %read data
isPartitionable(ds); %checking if I can split the data
ds.SelectedVariableNames=["0","1","2"]; %for practice/proof of concept I just chose the first three columns
ds2=read(ds); %get the first three columns and name ds2
%I want to denoise next, which requires an array and type double
ds2=table2array(ds2); %array
ds2T=transpose(ds2); %transpose
ds2Tdouble=double(ds2T); %double; note data was originally int8
ds2Tdouble_den=wdenoise(ds2Tdouble); %attempt to denoise
%when running the last line, I get the error:
%%%
Error using repmat
Requested 31x640000000000 (147819.5GB) array exceeds maximum array size preference (15.9GB). This might cause MATLAB to become
unresponsive.

Error in wavelet.internal.weightfromdata (line 163)
    deltaweight = repmat(temp_deltaweight,1,n);

Error in wavelet.internal.ebayesthresh (line 80)
weight = wavelet.internal.weightfromdata(x,30,transformtype);

Error in wavelet.internal.ebayesdenoise (line 32)
        xdecdenoised.cd{lev} = wavelet.internal.ebayesthresh(...

Error in wdenoise (line 273)
            wavelet.internal.ebayesdenoise(...

I am guessing this a problem with the size of my array. Any advice on how I can work around this?

Thank you

Answer 1

[arushi]

Hi Reid,

I understand that you are trying to denoise a signal and are getting an error because the size of the array that you are trying to use is exceeding your MATLAB’s memory preference.

Here are a few ways this issue can be resolved:

1. You can try to use tall arrays. These help to work with arrays that are too large to fit in the memory. You can find more information on them through the following MATLAB documentation: https://in.mathworks.com/help/releases/R2021b/matlab/import_export/tall-arrays.html
2. If the issue persists, you can untick the checkbox ‘Limit the maximum array size to a percentage of RAM’. To do this, you can go to: Preferences -> MATLAB -> Workspace. This allows you to use virtual memory in addition to the machine’s RAM. You can refer to the following documentation for more information about this: https://www.mathworks.com/help/releases/R2021b/matlab/matlab_env/set-workspace-and-variable-preferences.html
3. You can also refer to the following documentation for more information on resolving "Out of Memory" errors, if these occur at a later point: https://www.mathworks.com/help/releases/R2021b/matlab/matlab_prog/resolving-out-of-memory-errors.html

Hope this helps.