How to create datastore from cell array containing a lot of data?

I am trying to run an LSTM code with a frequency domain input with a cell array that is 184 x 1 called freqNorm. Each cell is a vector array of size 1 x 100001. When I try to run my code I get the following error:

Requested 400x127x100001 (18.9GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time and cause MATLAB to become unresponsive.

I read that I should create a datastore but I am not quite sure how to do this. The input data needs to still be in a 184 x 1 cell array. I tried using the following code, but i kept getting an error (as seen below the code).

save('freqNorm.mat','freqNorm');

freqNorm = datastore('C:\Users\Kim Cardillo\Desktop\Research\PRTools_Matlab\AEsignals_4to48kc_3.5-8mm\freqNorm.mat');

Error using datastore (line 125)

Cannot determine the datastore type for the specified location.

Specify the 'Type' name-value pair argument to indicate the type of datastore to create.

I am not quite sure what the 'Type' is nor do I know if what I am trying to do is correct. Can someone please help me.

Answer 1

[Devineni Aslesha]

Hi Kimberly,

Please use the below code to create datastore from cell array freqNorm.

tA = tall(freqNorm);
write('C:\Users\Kim Cardillo\Desktop\Research\PRTools_Matlab\AEsignals_4to48kc_3.5-8mm\freqNorm.mat', tA);
tds = datastore('C:\Users\Kim Cardillo\Desktop\Research\PRTools_Matlab\AEsignals_4to48kc_3.5-8mm\freqNorm.mat','Type','tall');

For more information, refer the following links,

https://www.mathworks.com/help/matlab/ref/tall.tall.html

https://www.mathworks.com/help/matlab/ref/tall.write.html

https://www.mathworks.com/help/matlab/ref/matlab.io.datastore.talldatastore.html

[Kimberly Cardillo]

Thank you. When I ran the code you posted, I got this error:

Invalid write location: C:\Users\Kim Cardillo\Desktop\Research\PRTools_Matlab\AEsignals_4to48kc_3.5-8mm\freqNorm.mat

Folder must either be empty or non-existing.

What does that mean?

[Devineni Aslesha]

Hi Kimberly,

In the write doc, you can see that the location should be

• Existing empty folder
• New folder that write creates

In your case, freqNorm.mat should be empty if it is already existing.