Matlab is not utilizing full Memory capacity in Windows server

Hi,

I am trying to merge almost 510 .csv file (more than 40 GB of data) in a windows server. But it is utilizing only 7% of CPU and 17% of Memory. I have talked with the administrator, there is no user limitation in this case.

Server specification: Intel Xeon 2 GHz (16 Core) and 96 GB RAM capacity.

Any solution?

Thank you in advance.

/Kazi

[Walter Roberson]

How are you doing the merging?

Do the csv files each have a 1 line header that should be removed (except for at the very beginning) ?

[Kazi Main Uddin Ahmed]

None of my csv file has any header. My impression about this problem is the relationship of backend java program used by Matlab. But i am not sure.

[Walter Roberson]

How are you doing the merging? Are your csv pure numeric?

I cannot think of any backend java program that might be involved in merging csv files.

Perhaps you should just generate a command that would be like

system('copy F1.csv+F2.csv+F3.csv+F4.csv.....+R510.csv DestinationFile.csv')

[Mohammad Sami]

If you want to load the data into Matlab, try using fread instead of readtable. you can then just concatenate the char arrays. use textscan for conversion.

[Andrew Janke]

That sounds normal. Most Matlab stuff that doesn't call down into BLAS/LAPACK is single-threaded. And when you're reading CSV files, it'll do the reading incrementally. And as it reads the CSV data and converts the strings into numeric values, the in-memory representation of the data may be more compact than the size of those files on disk.

Is it not working? Are you trying to make it go faster?