Parfor detailed error message?

Hello,

I'm using parfor to run several simulations. After a unknown number of loops the simulation stops:

Error using simulation (line 227)
An UndefinedFunction error was thrown on the workers for 'fetch'.  This might
be because the file containing 'fetch' is not accessible on the workers.  Use
addAttachedFiles(pool, files) to specify the required files to be attached.
See the documentation for 'parallel.Pool/addAttachedFiles' for more details.

Line 227 is my parfor line, so there is no fetch.

I think I forgot somewhere to close the connection and running to the max. number. But I can't find it. How can I find the fetch function which generates this error? I have about 50 - 100 in my code and don't find the right one.

How can I get a more accurate error message?

Answer 1

[Rebecca Krosnick]

How do you create the database connection in your code? Note that when using the Database Toolbox with "parfor", you need to create the database connection on each worker, not just on the client. You should include the database connection statement inside of the "parfor" loop. Since a particular worker may execute multiple iterations of the "parfor", however, you should be sure to create the connection only once per worker. You can do this by creating a "persistent" variable. For example:

parpool
    parfor i=1:N
    output{i} = myFun(i)
end

where "myFun" is defined as:

function y = myFun(i)
    persistent conn
    if isempty(conn)
        conn = database(...)
    end
    curs = exec(conn, 'SELECT * FROM myTable'); % This on the workers
    curs = fetch(curs);
    y = curs.Data;
    ...
end

On each individual worker, the value of the persistent variable "conn" is retained in memory between calls to "myFun".

Also, are you using a JDBC or ODBC driver? When working with JDBC drivers, you need to make sure that the JDBC driver is actually loaded on all workers, so you may also need to use something like:

pctRunOnAll('javaaddpath(''yourdriver.jar'')')

after having opened the "parpool".

[Philipp Dreessen]

Thanks a lot, Rebecca! This saved me a lot of time. Especially the fact that I had to load the jdbc driver for each worker was something I didn't know. I had to make a little correction to your code, but below seems to be doing the job.

pctRunOnAll javaaddpath('yourdriver.jar')