Deep Learning Memory issues for BiLstm mat files

Hi,

I have a huge dataset of over 20GB of numercial time-series data stored in files as shown below:

x1_1 = double(23*921600)

...

I have about 250 of these mat files. and the label or target of these are single row categorical array that shows 0 and 1.

y1_1= categorical(1*921600)

...

I loaded 5 of each file and was able to train and classiy them using a BiLSTM Netowrok of 3 hidden layers with over 300 neurorns. but now i want to run all of them. My example is very similar to this example, I also tried this which was no help. I know that i should use a datastore of some kind but i tried most[File,Tall] of them and couldn't solve the issue. Any ideas how to solve this issue? Thank you.

Answer 1

[Divya Gaddipati]

Hi,

You could use the Custom Mini-batch Datastore which uses the function sequenceDatastore that reads data from the specified folder and obtains labels from the subfolder names.

Refer to this link to learn more about how to train your networks.

[Kiyan Afsari]

Thank you for your answer. The issue that i am facing is that unlike the japanese vowel example, i do not have 1 label per sequence, i have a sequence label for each sequence for example:

x_train: 0 1 5 4 9 7 6 4 8 4 3 5 1 2

y_ label: 0 0 0 0 1 1 1 1 0 1 1 1 0

Something like that. I have sequence to sequence prediction rather than sequence to label.

  1×2 cell array
    {25×7680 double}    {1×7680 categorical}