Is it possible to use parfor to run a very large variable separately?

Is it possible to use parfor to run a very large variable separately? I've been finding a way to run a training data which is too big separately. Any suggestions?

Answer 1

[Shashank Prasanna]

Could you elaborate on "run a very large variable separately" There are things you can do parallely and there are few things you can't(if the output of one is needed as input for the next). The stuff you can do using Parallel Computing toolbox mostly fall under the embarrassingly parallel paradigm.

This might be helpful for when to use parfor: http://www.mathworks.com/help/distcomp/getting-started-with-parfor.html#brdqg4q-1

If you can provide further information on what you are doing then I will be able to give you a clearer picture.

[Lester Lim]

I am currently doing a project on machine learning and I have to put the trained data set into the classifier. The training data consists of 21 textures formed into a 21 columns. The 22nd column are labels. However, since the data is 170884x22, the data is too large to be put into the classifier. My school only has Matlab basic. Hence I am thinking of splitting the data and putting into the classifier. But im not sure how to do it due to my brief knowledge of matlab. Hence I came here to see if I could find any help since deadline is nearing...

[Shashank Prasanna]

What classifier is this? You can split the data, but you may have to train them one by one, and I am not sure you will be able to do this parallely. All classifiers have some weight tuning and I cant see how this can be done parallely. Cross validation however can be done parallely.

[Lester Lim]

I am using LDA currently but due to memory problems, im starting to look at naive bayes classifier. But let's just stick to LDA. Would it be possible in your opinion?

[Walter Roberson]

It is not going to be practical unless you are able to run on a machine with more memory than you have at the moment.

Each worker used through your Student Version is going to have the same 32 bit limit as your MATLAB Student Version, and so would be limited to on the order of 1 gigabyte of usable array memory.

You would get much further if the university allowed you to use one of its Academic licenses, in which case you could use a 64 bit version of MATLAB on a machine with much more memory.

[Lester Lim]

Good point, but if the computer had the same memory but the matlab was a 64bit, would it make much difference?

[Walter Roberson]

Possibly I am confusing you with someone else, but I thought you are executing on a machine with 3 gigabytes of memory? If so then... I don't think I recall seeing anyone give memory stats for that configuration.

More typical is for people to be using 4 Gb or more of memory on MATLAB 64 bit versions; with the 32 bit version of MATLAB they usually are limited to a little short of 800 Mb as the maximum array size, and with the 64 bit version the maximum array size goes up to at least 1500 Mb, and further as the memory increases past 4 Gb.

My use has been on Unix and OS-X (Mac) machines with the 64 bit MATLAB version; the memory stats are not available on those operating systems, so I cannot give comparison information. I think I had a MATLAB session up around 7 Gb on the main machine I use now; the one I used to use, I had MATLAB sessions up past 10 Gb.

[Lester Lim]

You're right the computer im using is a 3GB memory, which currently cant run the huge amounts of data im using. I think im gonna find another way to do the classifier probably using just simple statistics calculations. The school's computer has a lot of limitations and only basic matlab.

[Lester Lim]

You're right the computer im using is a 3GB memory, which currently cant run the huge amounts of data im using. I think im gonna find another way to do the classifier probably using just simple statistics calculations. The school's computer has a lot of limitations and only basic matlab.