Potential bug with parfeval; cumulative slowing down after several hours of operation. Can even exceed 10x the initial compute time.

There seems to be a bug using parfeval in the parallel pressing tool box. After hours of running the computation time for the compute time of the parallel tasks start increasing.

I tried running in all serial mode and the compute time remains similar even after hours of operation.

I have monitored memory usage and doesn't increase with time.

I tried monitoring the parallel compute time and maintained a low and high compute time. Once the difference exceeds certain (20%) threshold, I manually performed the following;

delete(gcp('nocreate'));    

POOL=parpool('local', NO_PAR_POOLS);

The reset of the parallel pool seems to bring back the parallel compute time back to expected.

Here is the pseudo code:

%%%%%%%%%%%%%%%%%%%%%%%%

tic_sum1=0;

tic_sum1_high=0;

tic_sum1_low=0;

for mini_batch_no=1:NO_OF_MINI_BATCHES

tic;

% Launch (N-1) parallel asynchronous jobs

job{1} = parfeval(POOL, @read_dataset_from_hdd, 1, mini_batch_no, CONST_DATA, 1);

job{2} = parfeval(POOL, @compute_cpu_task, 1, BUFF_DATA_TRAIN.batch_file_read(:,:,:,:,set_cpu_num), CONST_DATA);

job{3} = parfeval(POOL, @compute_cpu_dwt_task, 1, BUFF_DATA_TRAIN.batch_file_process_1(:,:,:,:,set_cpu_dwt_num), CONST_DATA.IM_RESIZE, CONST_DATA);

% Perform the Nth parallel job on the host

compute_gpu_task({BUFF_DATA_TRAIN.batch_file_process_2(:,:,:,:,set_gpu_num), BUFF_DATA_TRAIN.batch_file_label_read(:,:,:,:,set_gpu_num)});

% Collect result from parallel jobs

result{1} = fetchOutputs(job{1});

result{2} = fetchOutputs(job{2});

result{3} = fetchOutputs(job{3});

tic_sum1=tic_sum1+toc;

%%%Perform reset of parallel pool if hi-low diff exceed threshold percentage

if (tic_sum1>=tic_sum1_high)

tic_sum1_high=tic_sum1;

end

if (tic_sum1<=tic_sum1_low)

tic_sum1_low=tic_sum1;

elseif (tic_sum1_low==0)

tic_sum1_low=tic_sum1;

end

tic_sum1=0;

if (tic_sum1_high~=0 && tic_sum1_low~=0)

if ((100*(tic_sum1_high-tic_sum1_low)/tic_sum1_low)>CPU_ALLOWABLE_COMPUTE_TIME_LOW_VS_HIGH_DIFF_PERCENTAGE)

delete(gcp('nocreate'));

POOL=parpool('local', NO_PAR_POOLS);

tic_sum1_high=0;

tic_sum1_low=0;

end

end

end

%%%%%%%%%%%%%%%%%%%%%%%%

The iterations go over 5-6 days and after the 1st day the total time of operation exceeds 10x the initial time.

[Pavel Sinha]

Is there a way around not doing the hard reset of the parallel pool?

[Edric Ellis]

Hm, what release of MATLAB/PCT are you using? What OS? Are you able to provide an MCVE?

[Pavel Sinha]

I am using Matlab R2017a on Windows10 system. HW: Dual CPU chips, each with 6 cores, 256GB system RAM with two GTX-1080-TI GPUs and two SSDs (one for OS and one workspace).

It's hard to provide an MCVE. I have to provide data which is big and the code base. Nothing of a small sandbox that exhibits the issue.

I also, checked; if I run without parfeval, that is simply do all the tasks sequentially, then the time taken to complete the inner loop (which is significant doing things sequentially) remains about the same even after hours of running and memory usage also doesn't grow with time.

[OCDER]

Perhaps a deadlock or Race condition?

https://www.mathworks.com/help/distcomp/further-notes-on-parallel-jobs.html

https://en.wikipedia.org/wiki/Race_condition

[Pavel Sinha]

Hi,

Thank you for your reply. Here are my thoughts:

1) It cannot be race condition because each of the async tasks are carried out on independent buffers. So, all read/write to/from happen within independent buffers during execution of the tasks. No communication or dependencies between the tasks, completely independent. All async tasks are synced, then data is collected before the next launch of the parallel threads happens, each time.

2) The above point also makes sure that there is no deadlock. It comes from the sync before launching the parallel jobs and no interaction between the async threads during execution.

[Pavel Sinha]

I found that by initializing Matlab with just 1 parallel worker from settings and then activate required number of parallel workers from code, fixes this issue to a great extent. As in, now it takes much longer time before the processing loops take up significantly longer time to compute than the initial once. But eventually once the processing loop exceeds 20% of the initial compute time, I still have to reset the parallel cores and re-assign the number of required parallel workers.