Deleting elements of an array with consecutively occurring NaN values

Hello, may inqure about something small. Am trying to filter a huge data I obatined from my measurements. Iwould like to delete array elements with NaN value and that occur consecutively in an array based on a thresh hold value. For example, i have matrix A, i can determine the position of the non-empty (not NaN) cells,as shown in the code below. I would then go ahead and find the diffenrece between the elemnts of NotNaN matrix,if the diffenerence is equal or greater than 3, then all the elements in matrix A, in between the positions defined by the NotNaN matrix (the adjacents elements whose diffenrence is greater or equal to 3) are deleted. So that the final output would be A= [34 45 10 22 36 33 28 21 NaN 20 98 NaN]

A = [43 NaN NaN NaN 45 NaN NaN 10 22 NaN NaN 36 33 28 21 NaN 20 98 NaN];

NotNaN=find(~isnan(A))

diff(NotNaN)

NotNaN =

1 5 8 9 12 13 14 15 17 18

ans =

4 3 1 3 1 1 1 2 1

Answer 1

[Dyuman Joshi]

Note that this removes any NaN groups occuring at the start or at the end of the data.

%Modified data
A = [43 NaN NaN NaN 45 NaN NaN 10 22 NaN NaN 36 33 28 21 NaN 20 98 NaN 17 NaN NaN NaN NaN NaN ...
    2 3 NaN 5 7 NaN 13 NaN NaN];
idx = find(isnan(A))

idx = 1×18

2 3 4 6 7 10 11 16 19 21 22 23 24 25 28 31 33 34

%Indices corresponding to single NaNs
out = setdiff(setdiff(idx,idx+1),idx-1)

out = 1×4

16 19 28 31

%Remove consecutive NaNs
A(setdiff(idx,out)) = []

A = 1×20

43 45 10 22 36 33 28 21 NaN 20 98 NaN 17 2 3 NaN 5 7 NaN 13

[okoth ochola]

Thanks Dyuman Joshi, how about if i wanted to exclude the NaN that occur three consecutive times and delete only the ones that occur four consecutive times and above. Any NaN that occur below four is left (the NaN occirring two and three consecutive times are not deleted), is there a way I can modify it to include this? Because I need to define the thresh hold fro deletion based on certain varriables.

[okoth ochola]

Thanks @Dyuman Joshi, how about if i wanted to exclude the NaN that occur three consecutive times and delete only the ones that occur four consecutive times and above. Any NaN that occur below four is left (the NaN occirring two and three consecutive times are not deleted), is there a way I can modify it to include this? Because I need to define the thresh hold fro deletion based on certain varriables.

[Dyuman Joshi]

I am unable to think of a vectorized solution to this approach yet, I will update if I find one. Meanwhile, here is a loop approach that achieves what you want to do, but it will be slow for large inputs

%Modified data
A = [NaN NaN NaN NaN 43 NaN NaN NaN 45 NaN NaN 10 22 NaN NaN 36 33 28 21 NaN 20 98 NaN 17 NaN NaN NaN NaN NaN ...
    2 3 NaN 5 7 NaN 13 NaN NaN NaN NaN];
%Adding a number to the end
A = [A 0];
disp(A)

  Columns 1 through 33

   NaN   NaN   NaN   NaN    43   NaN   NaN   NaN    45   NaN   NaN    10    22   NaN   NaN    36    33    28    21   NaN    20    98   NaN    17   NaN   NaN   NaN   NaN   NaN     2     3   NaN     5

  Columns 34 through 41

     7   NaN    13   NaN   NaN   NaN   NaN     0

%First NaN
idx = find(isnan(A),1);
%First number after NaN
k = find(~isnan(A(idx+1:end)),1)+idx;
thresh = 3;
while ~isempty(k)
    if k-idx>=thresh
        A(idx:k-1)=[];
        k=idx;
    end
    idx = find(isnan(A(k+1:end)),1)+k;
    k = find(~isnan(A(idx+1:end)),1)+idx;
end
%Remove the number
A = A(1:end-1);
disp(A)

    43    45   NaN   NaN    10    22   NaN   NaN    36    33    28    21   NaN    20    98   NaN    17     2     3   NaN     5     7   NaN    13

[okoth ochola]

thank you @Dyuman Joshi

Answer 2

[Stephen23]

A = [NaN,NaN,NaN,NaN,43,NaN,NaN,NaN,45,NaN,NaN,10,22,NaN,NaN,NaN,NaN,36,33,28,21,NaN,20,98,NaN,17,NaN,NaN,NaN,NaN,NaN,2,3,NaN,5,7,NaN,13,NaN,NaN,NaN,NaN]

A = 1×42

NaN NaN NaN NaN 43 NaN NaN NaN 45 NaN NaN 10 22 NaN NaN NaN NaN 36 33 28 21 NaN 20 98 NaN 17 NaN NaN NaN NaN

N = 3;
X = isnan(A(:));
D = cumsum([1;diff(X)~=0]);
F = @(v)sum(v)>N;
Y = grouptransform(ones(size(D)),D,F);
A(X&Y) = []

A = 1×25

43 NaN NaN NaN 45 NaN NaN 10 22 36 33 28 21 NaN 20 98 NaN 17 2 3 NaN 5 7 NaN 13

[okoth ochola]

Wow, thank you @Stephen23; However the my matlab is not recognizing the function grouptransform(), I guess it was added in more recent versions of matlab. Am using 2018a

[Dyuman Joshi]

Yes, grouptransform was added in just the next version of MATLAB - 2018b.