Eliminating the noise from the raw data using median filter

Hi everyone,

I have a raw csv file data and when I am to plot it, it seems pretty noisy. What I wanted to do ıs either the entire signal(data); or just along the decay curve to be smoothened; in other words; deleting the noise as much as possible. I'd be appreciated if anyone could help me out.

Here's the code that I am dealing with. Apart from reading the csv file, I also come across a similar code on the community so I was trying to modify it.

% Remove quiet parts of a signal.
clc; 
close all; 
clear;  
workspace; 
format long g;
format compact;
fontSize = 25;
data= readmatrix('T0027CH1.CSV');
x=data(:,1);
y=data(:,2);
y=(y+35.6);
x=(x+0.8*10^(-3));
N = 125013; % length(col1) if doing the whole signal
col1 = data(1:N, 1);
subplot(2, 2, 1);
plot(col1,y,'b-')
grid on
title('Original Signal', 'FontSize', fontSize);
subplot(2, 2, 2);
col1a = medfilt1(abs(col1),50000);
plot(col1a,y,'b-')
title('After median filter', 'FontSize', fontSize);
threshold = 50;
yline(threshold, 'Color', 'r', 'LineWidth', 2)
grid on;
badIndexes = col1a < threshold;
subplot(2, 2, 3);
plot(badIndexes,y, 'b-', 'LineWidth', 2);
title('Bad parts', 'FontSize', fontSize);
grid on;
% Remove bad indexes.
col1(badIndexes) = [];
subplot(2, 2, 4);
plot(col1,'b-')
grid on;
title('Bad parts removed', 'FontSize', fontSize);

[Mathieu NOE]

hi

to better help you we need the data file as well

[N/A]

Hi,

Sorry I thought I already attached my raw file. Please see enclosed the csv file.

Thank you.

Answer 1

[Mathieu NOE]

hello again

is it now smooth enough ?

% Remove quiet parts of a signal.
clc; 
close all; 
clear;  
workspace; 
format long g;
format compact;
fontSize = 25;
data= readmatrix('T0027CH1.CSV');
x=data(:,1);
y=data(:,2);
y=(y+35.6);
x=(x+0.8*10^(-3));

% removing x NaNs
idx = ~isnan(x);
x = x(idx);
y = y(idx);

ys = smoothdata(y,'gaussian',2500);

plot(x,y,'b',x,ys,'r')
legend('raw','smoothed');

[N/A]

Hi again thank you for getting back to me that early.

yes it is obvious that the decay curve behaves ın a more smooth way. However, may I ask is it a wrong approach to use a medfilt function to reduce the noise in the file? I mean it can be seen that there are lots of repetitive datas in the file so that is the reason why the plot is too rippled because of the noise.

Thank you again!

[Mathieu NOE]

hello again

I have nothing against medfilt

FYI, smoothdata will do the same job if you use the option 'movmedian'

here I tested different options , I let you choose which one is your prefered ;

the median filter cannot filter out the small quantization noise , in this aspect the gaussain or movmean is better; for the large spikes around t = 12 , movmedian is a better approach; you can also smooth the staircase aspect of the movmedian filter by using a second layer of filtering;

the possibilities are endless and smoothdata offers many options that are not available in other functions; there is still to explore if you want

% Remove quiet parts of a signal.
clc; 
close all; 
clear;  
workspace; 
format long g;
format compact;
fontSize = 25;
data= readmatrix('T0027CH1.CSV');
x=data(:,1);
y=data(:,2);
y=(y+35.6);
x=(x+0.8*10^(-3));

% removing x NaNs
idx = ~isnan(x);
x = x(idx);
y = y(idx);

ys1 = smoothdata(y,'gaussian',2500); 
ys2 = smoothdata(y,'movmedian',2500);
ys3 = smoothdata(ys2,'movmean',2500);

plot(x,y,'b',x,ys1,'r',x,ys2,'c',x,ys3,'m')
legend('raw','gaussian smoothed','movmedian smoothed','movmedian + movmean smoothed');

Zoom around t = 0

Zoom around t = 12 e -4 s

[Mathieu NOE]

hello

problem solved ?

[Mathieu NOE]

hello again

If my submission fullfills your request, do you mind accepting it ?

tx