accelerate filter loop

Hi everybody, I need to filter large data sets of about a million lines and some 15 columns. The data are echosounder readings, one data line per second. The simple thing I have to do is to remove those lines that exceed 3 m depth from one line to the next as these jumps are not plausible (they might be caused by fishes or air bubbles under the ship). I reduced my code to only find out the 'good' line numbers for later processing (instead of filtering the whole data set). Although this is quite simple, it takes ages and the speed seems to decrease being quick for the first 200 k lines but hen significantly slowing down. Does anybody know a quicker method to do such filtering? This is the code I use:

 j=0;
    for i=2:length(depthAlpha)
            if abs(depthAlpha(i-1)-depthAlpha(i))<3
                j=j+1;
                goodlines(j,1)=i;
            end
    end

Answer 1

[Andrei Bobrov]

goodlines = find(abs(diff(depthAlpha(:)))<3)

Answer 2

[Jan]

It is getting slower for more than 200'000 lines? This sounds like a missed pre-allocation. Although I'd prefer Andrei's solution, here is the pre-allocation for educational reasons:

goodlines = zeros(1, length(depthAlpha));
j = 0;
for i = 2:length(depthAlpha)
  if abs(depthAlpha(i-1)-depthAlpha(i)) < 3
    j = j+1;
    goodlines(j) = i;
  end
end

You can use TIC/TOC to compare the speed.

Answer 3

[Christian]

Thank you Andrei and Jan! Andrei's solution significantly speeds up the process and, yes, lacking pre-allocation obviously was the problem for the deceleration at higher line numbers. Thanks again, Christian

[Jan]

Could you please post a speed comparison? I'm collecting arguments for my "For-loops versus vectorization" investigation.

[Andrei Bobrov]

Hi Jan! I look forward to publishing your investigation "For-loops versus vectorization"!