Why GPU computation does not scale linearly with the number of loops ?

Good evening everyone,

I'm currently trying to speed up my code by converting it to GPU computation. But I'm facing a problem I didn't have with CPU : The time doesn't scale linearly with the number of loops.

Here is the code :

Nloops = 500;

dt=1e-1; % time step
n=3e3; % number of vortices

z1=complex(gpuArray.randn(1,n),gpuArray.randn(1,n)); % vortices position
C1=repmat(1e-2*gpuArray.randn(n,1),1,n); % circulation

D=gpuArray.eye(n);
nD=~D;

tic

for ii=1:Nloops

      Z1 = repmat(z1,n,1);   
      z1 = z1 + (dt*0.5i/pi) * (sum((C1.*nD)./(Z1-Z1.'+D),1));

end

toc

The result is quite surprising

Nloops = 100 : Elapsed time is 0.014950 seconds.
Nloops = 500 : Elapsed time is 17.072178 seconds.

While on CPU it scales well ( 3 and 15 seconds respectively ), do you have an idea why it scales so badly on GPU ?

Answer 1

[Matt J]

tic and toc are not accurate measures of GPU execution time. Use gputimeit() instead.

[gdgc]

Thanks for the tip.

Unfortunately here it's not a problem of accuracy, as the algorithm is still a hundred to thousands times slower for 5 times more loops, whatever measurement I use.

[Matt J]

Not me. I ran your code as below and got 3.2969 sec for Nloop=500 versus 0.6609 sec for Nloop=100. In other words, it scaled very well. What graphics card are you using?

 function test

    dt=1e-1; % time step
    n=3e3; % number of vortices
    z1=complex(gpuArray.randn(1,n),gpuArray.randn(1,n)); % vortices position
    C1=repmat(1e-2*gpuArray.randn(n,1),1,n); % circulation
    D=gpuArray.eye(n);
    nD=~D;

    gputimeit(@()LoopPart)

    function LoopPart

        Nloops = 100;

        for ii=1:Nloops
              Z1 = repmat(z1,n,1);   
              z1 = z1 + (dt*0.5i/pi) * (sum((C1.*nD)./(Z1-Z1.'+D),1));
        end

    end

 end

[Walter Roberson]

For me the time scaled linearly the first time, but each time I ran after the first, the time increased. This could suggest memory not having been released.

(500 iterations, n=1e3 to fit in my memory)

>> testtime
ans =
               3.338060014
>> testtime
ans =
               4.613551083
>> testtime
ans =
               6.462555906
>> testtime
ans =
              12.783108898
>> clear all
>> testtime
ans =
          11.6822856971945

reset(gpuDevice(1)) seemed to help in my further tests.

[gdgc]

I ran your code, and effectively, written like this it scales well. But it's also much longer to execute, it took 3.4 sec and 17 sec.

While using my code the 100 loops case gives an instant answer, less than the blink of an eye.

I have a nvidia k620 at office, and gtx760m on my laptop, the results are similar. Obviously your card is much faster, do you notice the same problem using my code ?

[Matt J]

Walter, what MATLAB version and what graphics card are you using? My repeated runs are quite consistent

    >> test

    ans =

        3.3074

    >> test

    ans =

        3.3350

    >> test

    ans =

        3.3425

    >> test

    ans =

        3.3682

    >> test

    ans =

        3.3369

    >> clear all
    >> test

    ans =

        3.3089

[gdgc]

Ok, I understand why it was so long to plot the vortices after the time display. It was still working ... Thank you for your patience ! (crossing edits)

[Matt J]

Indeed. But I still don't understand why it's not taking just five blinks of an eye with 500 loops.

Sorry, forget what I said before. Your code isn't as fast as you think it is. The Nloop=500 case really does take 17 sec and the Nloop=100 case really does take 3.4 sec.

The reason Nloop=100 appears to be super-fast is because the GPU hasn't really finished working. It's just taking an early coffee break, which it sees that it can do because you haven't actually tried to use the result z1 for anything yet.

There are various asynchronous computing protocols that GPUs use. It's also the reason why tic/toc are unreliable and why you must use gputimeit().

[gdgc]

Thank you for your explanations. By the way, what graphic card do you have ? It might give me ideas of what I need for my next computer.

[Matt J]

I'm using the Titan X.

[Walter Roberson]

I am using a 650M card (2012 time frame) with R2017b.

I tried adding in a gather() to force the computation to finish, but it did not seem to make any difference. However, either way (with or without gather) if I run the tests close together then it takes additional time, but if I wait the timer is faster. This suggests the GPU might not have finished (even with the gather.)

Note: in the source below, I use 1e3 nodes not 3e3, so as to avoid filling my GPU memory.

 function testtime

    dt=1e-1; % time step
    n=1e3; % number of vortices
    z1 = complex(gpuArray.randn(1,n),gpuArray.randn(1,n)); % vortices position
    C1=repmat(1e-2*gpuArray.randn(n,1),1,n); % circulation
    D=gpuArray.eye(n);
    nD=~D;

    gputimeit(@()gather(LoopPart(z1)))

    function z1 = LoopPart(z1)

        Nloops = 500;

        for ii=1:Nloops
              Z1 = repmat(z1,n,1);   
              z1 = z1 + (dt*0.5i/pi) * (sum((C1.*nD)./(Z1-Z1.'+D),1));
        end

    end

 end

[Joss Knight]

Try removing the repmat from your loop, with automatic scalar expansion you don't need it: Z1-Z1.' == z1-z1.'