Retrieving data from createtask is very slow
4 Ansichten (letzte 30 Tage)
Ältere Kommentare anzeigen
I used createJob and createTask to execute a program on 16 notes, then retrived my data from each tasks. The result is very large, I can not use fetchOutputs or getAllOutputArgument.My network is a Gigabit LAN. However, I found that the speed of retrieving data is only 3M/s.It cost too much time is data transmission. This is my code:
Job=createJob(sched); set(Job,'PathDependencies',PathDependency},'FileDependencies','Reconstruction_MLOS_SMART_Parallel.m','IntegratingLOS.m','MARTUpdate.m','MARTInitialize.m'});
for ii=1:cpucorenum
curTask=newtask{ii,1};
PartStart=curTask(1);
PartEnd=curTask(end);
TaskObj(ii,1)=createTask(Job,@MARTUpdate,1,{VolumeSpace(:,:,PartStart:PartEnd), MFx, MFy, Xmesh, Ymesh, xmesh_jiu, ymesh_jiu, m, oriI, newI, PartStart, PartEnd, ParaSet});
end
submit(Job);
wait(Job,'finished');
for ii=1:cpucorenum
curTask=newtask{ii,1};
PartStart=curTask(1);
PartEnd=curTask(end);
results_t = get(TaskObj(ii,1), 'OutputArguments');
VolumeSpace(:,:,PartStart:PartEnd)=results_t{1,1};
destroy(TaskObj(ii,1));
clear results_t
end
%%clear the temporary value
clear results_t TaskObj
destroy(Job);
clear Job
The problem arised in ‘results_t = get(TaskObj(ii,1), 'OutputArguments')’. Too much time expended in this step.
I holp some one can help me. Thank you!
0 Kommentare
Antworten (1)
Edric Ellis
am 23 Jan. 2014
The time taken for fetching outputs is driven by the disk access speed, and the speed of MATLAB's own SAVE command. I ran the following test:
c = parcluster('local');
numMBytes = floor(logspace(1, 3, 10));
times = zeros(size(numMBytes));
for idx = 1:numel(numMBytes)
j = batch(c, @randi, 1, {[-10 10], numMBytes(idx), 1024^2, 'int8'});
wait(j);
tic
x = fetchOutputs(j);
times(idx) = toc;
delete(j);
end
plot(numMBytes, numMBytes ./ times);
xlabel 'Number of megabytes'
ylabel 'MB / s'
and saw on my system approximately 140 MB/s. This is R2013b on GLNXA64 using the local cluster which stores the outputs on a network disk.
Siehe auch
Kategorien
Mehr zu MATLAB Parallel Server finden Sie in Help Center und File Exchange
Produkte
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!