Problem with processing large data set

I want to train a BiLSTM-CNN (Bidirectional Long Short-Term Memory Convolutional Neural Network) with 203457 timeseries. I am trying to store them in a table with one line and 203457 columns, each cell containing an array with 100,000 values, but my server freezes whenever I run my code. I don’t get any error messages so I am not sure what the problem is and how to fix it. Do you have any idea why this happens and how to solve this?

Thank you!

Answer 1

[Eric Sofen]

I'm not sure of the BiLSTM-CNN input requirements, but for starters you're probably better off storing those timeseries as rows in a table rather than variables (columns). You've got some 20 billion total values, so you may be hitting memory limits. You may want to look into using a tall variable for out-of-memory computation.

[Andrei-Ieronim Constantinescu]

Thank you very much for the response. Meanwhile, I found a forum suggesting that a fileDatastore is good for processing MAT files and big data due to the fact that it is a type of datastore. I will try both options and update here wich one is better.