I tried Hampel filtering to discard the bad data, but it was not fruitful. How to avoid/filter

Dear all,

I have attached the data plot. I tried Hampel filtering to discard the bad data, but it was not fruitful. How to avoid/filter this bad data ?

[Rik]

What exactly defines bad data in your case?

[KSSV]

Read about isoutlier.

[Ismita]

@Rik if you see the plot, there are some peaks for corresponding value. How can I avoid those points (e.g., in this plot E_D is >12000 for one case, E_D>4000 for another case, etc......)? Thank you

@KSSV thank you

[Rik]

The function KSSV suggested should do the trick in this case. Feel free to post a follow up comment if you have questions.

[Ismita]

when I am running the following code in Matlab online it shows the plot, but it is not working while running in installed Matlab. Could you please suggest me the solution? Thanks again.

"Code:

x = 1:10;
A = [60 59 49 49 58 200 61 57 48 58];
[TF,L,U,C] = isoutlier(A);
plot(x,A,x(TF),A(TF),'x',x,L*ones(1,10),x,U*ones(1,10),x,C*ones(1,10))
legend('Original Data','Outlier','Lower Threshold','Upper Threshold','Center Value')

"

[Mathieu NOE]

hi

this code works on my R2020b

[Rik]

What release are you using?

[Mathieu NOE]

@Mst Ismita Tasnim

if you have an older release the second code example is for you

A = repmat([60 59 49 49 58 200 61 57 48 58],[1 10]);
nA = numel(A);
x = 1:nA;

% %% original code 
% [TF,L,U,C] = isoutlier(A);
% figure(1),plot(x,A,x(TF),A(TF),'dr',x,L*ones(1,nA),x,U*ones(1,nA),x,C*ones(1,nA))
% legend('Original Data','Outlier','Lower Threshold','Upper Threshold','Center Value')

%% alternative method
upper_threshold = 5; % above mean smoothed curve
lower_threshold = 25; % below mean smoothed curve

% apply smoothing (here twice)
N = 15;
B = myslidingavg(A, N); % or use smooth or smoothdata depending of what you have
B = myslidingavg(B, N); % or use smooth or smoothdata depending of what you have

TF = (A>B+upper_threshold) | (A<B-lower_threshold);
figure(2),plot(x,A,x,B,x,B-lower_threshold,x,B+upper_threshold,x(TF),A(TF),'dr')
legend('Original Data','Smoothed Data','Lower Threshold','Upper Threshold','Outlier')



function out = myslidingavg(in, N)
%   OUTPUT_ARRAY = MYSLIDINGAVG(INPUT_ARRAY, N)
%
%  The function 'slidingavg' implements a one-dimensional filtering, applying a sliding window to a sequence. Such filtering replaces the center value in
%  the window with the average value of all the points within the window. When the sliding window is exceeding the lower or upper boundaries of the input
%  vector INPUT_ARRAY, the average is computed among the available points. Indicating with nx the length of the the input sequence, we note that for values
%  of N larger or equal to 2*(nx - 1), each value of the output data array are identical and equal to mean(in).
%
%  *  The input argument INPUT_ARRAY is the numerical data array to be processed.
%  *  The input argument N  is the number of neighboring data points to average over for each point of IN.
%
%  *  The output argument OUTPUT_ARRAY is the output data array.

if (isempty(in)) | (N<=0)                                              % If the input array is empty or N is non-positive,
 disp(sprintf('SlidingAvg: (Error) empty input data or N null.'));     % an error is reported to the standard output and the
 return;                                                               % execution of the routine is stopped.
end % if

if (N==1)                                                              % If the number of neighbouring points over which the sliding 
 out = in;                                                             % average will be performed is '1', then no average actually occur and
 return;                                                               % OUTPUT_ARRAY will be the copy of INPUT_ARRAY and the execution of the routine
end % if                                                               % is stopped.

nx   = length(in);             % The length of the input data structure is acquired to later evaluate the 'mean' over the appropriate boundaries.

if (N>=(2*(nx-1)))                                                     % If the number of neighbouring points over which the sliding 
 out = mean(in)*ones(size(in));                                        % average will be performed is large enough, then the average actually covers all the points
 return;                                                               % of INPUT_ARRAY, for each index of OUTPUT_ARRAY and some CPU time can be gained by such an approach.
end % if                                                               % The execution of the routine is stopped.

out = zeros(size(in));         % In all the other situations, the initialization of the output data structure is performed.

if rem(N,2)~=1                 % When N is even, then we proceed in taking the half of it:
 m = N/2;                      % m = N     /  2.
else                           % Otherwise (N >= 3, N odd), N-1 is even ( N-1 >= 2) and we proceed taking the half of it:
 m = (N-1)/2;                  % m = (N-1) /  2.
end % if

for i=1:nx,                                                 % For each element (i-th) contained in the input numerical array, a check must be performed:
  dist2start = i-1;         %  index distance from current index to start index (1)
  dist2end = nx-i;          %  index distance from current index to end index (nx)
  if dist2start<m | dist2end<m        % if we are close to start / end of data, reduce the mean calculation on centered data vector reduced to available samples
      dd = min(dist2start,dist2end);    % min of the two distance (start or end)
  else
      dd = m;
  end % if
  out(i) = mean(in(i-dd:i+dd));     % mean of centered data , reduced to available samples at both ends of the data vector

end % for i

end

[Ismita]

Thank you so much

[Mathieu NOE]

My pleasure !