More efficient jsonencode for large data?

So I am using matlabs jsonencode function to encode a structure array to a character array, and then write this to an output json text file. The structure array mat file equalls approximately 10GB. This takes both a long time, and alot of computer memory.

I then import the JSON text file into mongodb.

Is there a more efficient way to directly get the data into mongodb? Maybe the only option is the MatLab database toolbox...?

Thanks

Answer 1

[Seb]

I have found a temporary way to better do this.

I still use matlabs built in jsonencode function, and output a JSON file, but I do it in small stages/chunks. For example, I have a devision factor, which determines how many chunks I do. This then splits the data structure into row indices. I then encode the json within those row indices, and write that chunk of data to the file. I then go onto the next chunk and so forth until all data is written.

If someone is in need of the code then let me know and I can provide it. It does not take any less time than a single jsonencode call would be, however greatly reduces load and memory consumption on the computer, and can be made to be 32-bit compliant (each chunk cant be larger than ~1gb)

[Marcel]

Hi Seb, do you still have the code for this? I am interested in this. How do you puzzle together the parts in the json file? THANKS

Answer 2

[Carl]

You can try following the example in this File Exchange function, which uses the MongoDB Java driver to insert a document:

https://www.mathworks.com/matlabcentral/fileexchange/43171-use-of-mongodb-java-driver

The following Stackoverflow post also has some good suggestions:

https://stackoverflow.com/questions/3886461/connecting-to-mongodb-from-matlab?rq=1

With these approaches, you should be able to avoid writing text to disk, which may be more efficient. Note that this workflow has not been qualified, and it may or may not be more efficient than your current method.

Answer 3

[Marco Rossi]

Is matlab development team planning to remove this limitation? I also noticed that, when the 32 bit limit is overtaken, no warnings/errors are returned.

[Joris Brouwer]

I second that. Running into what seems to be exponentionally / halting jsonencode performance as well. Trying the chunked solution mentioned above.