Cannot use database cursor object inside a parfor

If I retrieve data from an sql query using the database interface of matlab database toolbox inside a parfor like this:

 parfor i = 1:n
   [...]
   cursorObj1 = fetch(exec(targetConn,sqlQueryString));
   queryData = cursorObj1.Data{1};
   [...]
 end

Matlab throws the error:

Cell contents reference from a non-cell array object.

Is there a way to accomplish the aforementioned operation inside a parfor? I know the basic rules of variables and objects use inside a parfor, but I can't find a solution for this specific case.

[TIANTIANBABA]

Did you find an solution yet?

Answer 1

[Edric Ellis]

You need to create the database connection targetConn directly on the workers. One way to achieve this is to use a parallel.pool.Constant to hold on to it. Something a bit like this:

connection = parallel.pool.Constant(@() database(...), @close);
parfor ...
  ...
  cursorObj1 = fetch(exec(connection.Value, ...));
  ...
end

Note that the second argument to the parallel.pool.Constant constructor is the "cleanup" function - this will close the underlying connection when connection goes out of scope.

[Tommaso Guseo]

Can I use parallel.pool.Constant also to send struct data to the worker to avoid initialization of a struct inside a parfor?

[AJ Geiger]

Not that I know :/ of if you want to debug a parfor loop I have found the best way to do it is to convert the parfor loop to just a for loop.

[TIANTIANBABA]

Hello. Even if I build the connection inside the parfor-loop, it seems that exec and fetch still can not work when inside the parfor-loop.

Here is my needs:

I have many sqlite files, and I need to export data from every sqlite file. I want to use parfor-loop to speed up, and inside the loop, I want each worker to connect to a different file, then do the data export work. But until now I don't succeed.