Anonymous Functions and MapReduce

Hello there,

I have successfully created a home-made cluster with a couple of Macbooks. Everything seems alright, I've been doing a few tests and such. Now I want to run a MapReduce script which is fully working on my 'local' cluster profile and I have been using some anonymous functions for the Mapper and the Reducer because I need to handle several variables within the tasks.

I change the parallel pool from 'local' to 'homecluster', I've also set the Mapreducer environment but the following errors appear

Error using mapreduce (line 101)
An error occurred building a parallel.pool.Constant.

Error in SelectInitialCentroids (line 24)
outDataStore=mapreduce(DataStore,mapper,reducer,MapReduceEnv);

Error in main (line 35)
    centroids=SelectInitialCentroids(DS,K,MapReduceEnv);

Caused by:
    An error occurred interpreting function call.
    An error occurred interpreting function call.

Where 'mapper' and 'reducer' are my anonymous functions and 'MapReduceEnv' is the environment.

I think there might be a problem with my anonymous functions, although it works fine with the 'local' profile.

Any ideas? Thank you very much indeed.

PS: running Matlab R2015b on Mac OS X El Capitain

Update: the problem is indeed with the anonymous functions. I was able to run MapReduce tasks in my home cluster with Map and Reduce functions declared standard (i.e. not anonymous).

Answer 1

[AlessioX]

Allright. I guess I figured this out.

When opening the parallel pool, you must specify all the files with such functions: that file(s) must be available to all the workers/nodes in the pool:

myCluster=parpool('HomeCluster','AttachedFiles',{'file1.m'....'fileN.m'});

and then force the MapReduce Environment to that cluster (as in the original topic)

MapReduceEnv=mapreducer(myCluster);

Every time you call the MapReduce tasks, also, that environment must be specified (as in the original topic)

outDataStore=mapreduce(DataStore,mapper,reducer,MapReduceEnv);