delete(filename) in parfor ?? does it execute right away? is it delayed ??

I am trying to use delete(filename) inside the iterations of the parfor, and it looks like the file is still there even when I see that the script has actually executed passed it by following the print outs. I do not see any no error or warning message about Transparence violation.

I am using some sizeable .bmp files as temprary store and have to delete them once they are consumed to avoid runing out of disk space, and would like the delete to execute as soon as possible.

code looks like this:

parfor idx=1:n
    imfilename=strcat('xxxx_',num2str(idx),'.bmp')
    imread(imfilename)
    .... some processing code
    delete(imfilename)
    ... more processing code
    fprintf('idx=%d executing here...\n',idx)
end

and I see the printouts "idx=?? executing here.." on the screen, but the file is still there in the folder and disk space is not freeing up??

-Thanks,

-Michael

Answer 1

[Walter Roberson]

My tests say they are deleted immediately, at least on MacOS.

td = tempdir();

parfor idx = 1 : 20
   t = getCurrentTask();
   if isempty(t)
     error('Bah! This is not a real parfor!');
   end
   tid = t.ID;
   outfile = fullfile(td, sprintf('junk_%d_%d.junk', tid, idx));
   [fid, msg] = fopen(outfile, 'w');
   if fid < 0
     error('failed to open file "%s" because "%s"', outfile, msg);
   end
   fwrite(fid, rand(1,1e6));
   fclose(fid);
   dinfo = dir( fullfile(td, '*.junk') );
   filenames = strjoin({dinfo.name}, ', ');
   fprintf('task %d iter %d before delete files are: %s\n', tid, idx, filenames);
   delete(outfile);
   dinfo = dir( fullfile(td, '*.junk') );
   filenames = strjoin({dinfo.name}, ', ');
   fprintf('task %d iter %d after delete files are: %s\n', tid, idx, filenames);
end

dinfo = dir( fullfile(td, '*.junk') );
filenames = strjoin({dinfo.name}, ', ');
if isempty(filenames)
   fprintf('after parfor, no remaining files\n');
else
   fprintf('after parfor files are: %s\n', filenames);
end

task 1 iter 1 before delete files are: junk_1_1.junk, junk_2_2.junk, junk_3_3.junk, junk_4_4.junk

task 3 iter 3 before delete files are: junk_1_1.junk, junk_2_2.junk, junk_3_3.junk, junk_4_4.junk

task 3 iter 3 after delete files are: junk_1_1.junk, junk_2_2.junk, junk_4_4.junk

task 1 iter 1 after delete files are: junk_2_2.junk, junk_4_4.junk

task 1 iter 6 before delete files are: junk_1_6.junk, junk_2_8.junk, junk_4_12.junk

task 1 iter 6 after delete files are: junk_2_8.junk, junk_3_9.junk, junk_4_12.junk

task 1 iter 5 before delete files are: junk_1_5.junk, junk_2_7.junk, junk_4_11.junk

task 1 iter 5 after delete files are: junk_2_7.junk, junk_4_11.junk

task 1 iter 14 before delete files are: junk_1_14.junk, junk_2_16.junk, junk_4_15.junk

task 1 iter 14 after delete files are: junk_4_15.junk

task 1 iter 18 before delete files are: junk_1_18.junk, junk_2_19.junk, junk_4_20.junk

task 1 iter 18 after delete files are: junk_2_19.junk, junk_4_20.junk

task 2 iter 2 before delete files are: junk_2_2.junk, junk_4_4.junk

task 2 iter 2 after delete files are:

task 2 iter 8 before delete files are: junk_1_5.junk, junk_2_8.junk, junk_3_9.junk, junk_4_12.junk

task 2 iter 8 after delete files are: junk_1_5.junk, junk_3_9.junk, junk_4_12.junk

task 2 iter 7 before delete files are: junk_2_7.junk, junk_4_11.junk

task 2 iter 7 after delete files are: junk_3_13.junk, junk_4_11.junk

task 2 iter 16 before delete files are: junk_1_14.junk, junk_2_16.junk, junk_4_15.junk

task 2 iter 16 after delete files are: junk_4_15.junk

task 2 iter 19 before delete files are: junk_1_18.junk, junk_2_19.junk, junk_4_20.junk

task 2 iter 19 after delete files are: junk_4_20.junk

task 3 iter 10 before delete files are: junk_1_6.junk, junk_2_8.junk, junk_3_10.junk, junk_4_12.junk

task 3 iter 10 after delete files are: junk_1_6.junk, junk_2_8.junk, junk_4_12.junk

task 3 iter 9 before delete files are: junk_1_5.junk, junk_2_7.junk, junk_3_9.junk, junk_4_11.junk

task 3 iter 9 after delete files are: junk_1_5.junk, junk_2_7.junk, junk_4_11.junk

task 3 iter 13 before delete files are: junk_1_14.junk, junk_2_16.junk, junk_3_13.junk, junk_4_15.junk

task 3 iter 13 after delete files are: junk_1_14.junk, junk_2_16.junk, junk_4_15.junk

task 3 iter 17 before delete files are: junk_1_18.junk, junk_2_19.junk, junk_3_17.junk

task 3 iter 17 after delete files are: junk_1_18.junk, junk_2_19.junk, junk_4_20.junk

task 4 iter 4 before delete files are: junk_2_2.junk, junk_4_4.junk

task 4 iter 4 after delete files are:

task 4 iter 12 before delete files are: junk_1_5.junk, junk_3_9.junk, junk_4_12.junk

task 4 iter 12 after delete files are: junk_1_5.junk, junk_3_9.junk

task 4 iter 11 before delete files are: junk_1_14.junk, junk_2_16.junk, junk_3_13.junk, junk_4_11.junk

task 4 iter 11 after delete files are: junk_1_14.junk, junk_2_16.junk, junk_3_13.junk

task 4 iter 15 before delete files are: junk_1_18.junk, junk_2_19.junk, junk_3_17.junk, junk_4_15.junk

task 4 iter 15 after delete files are: junk_1_18.junk, junk_2_19.junk, junk_3_17.junk

task 4 iter 20 before delete files are: junk_4_20.junk

task 4 iter 20 after delete files are:

after parfor, no remaining files

Notice that in each case, the "after delete" does not have any files junk_* starting with the number that is the same as the task number. For example,

task 3 iter 9 after delete files are: junk_1_5.junk, junk_2_7.junk, junk_4_11.junk

The junk_1 junk_2 junk_4 exist because those are being created simultaneously in other workers, but junk_3 has been deleted immediately in worker 3.

[Michael]

Thanks, Walter for your quick response, I just ran your code and it runs too. It's WIndows 10, must be something else in my code. WIll dig in and report later.

[Michael]

MY code had an error, file path was messed up. FIxed that, and it does work..