Working on large .db file (database file) using MapReduce?

I have a very large signal file stored in a sqlite database file, it over 100 gb. I have the code section for extracting the data over to a matrix in matlab and storing it in a ,mat file, which has worked fine for small signals. However, I would like to do this to the large signal also but the signal is too large for memory.

I read that you can work on large files using MapReduce, however i only see how to use this with .csv files or other types of table data.

Is it possible to use MapReduce with this type of file? Or are there any other method to work on this signal without loading it into memory in its entirety?

Answer 1

[Samay Sagar]

Hi @Jørgen Fone Pedersen

You can use "mapreduce" while working with large SQLite Databases. You can use the "databaseDatastore" function to create a datastore for your database which can subsequently be used with the "mapreduce" function.

Refer the following documentations for more information:

• "databaseDatastore": https://www.mathworks.com/help/releases/R2024b/database/ug/matlab.io.datastore.databasedatastore.html
• "mapreduce": https://www.mathworks.com/help/releases/R2024b/matlab/ref/mapreduce.html
• Working with large data using MATLAB: https://www.mathworks.com/help/releases/R2024b/database/import-large-data-programmatically.html