Efficient Moving Quantile Function

37 Ansichten (letzte 30 Tage)
Matthew Casiano
Matthew Casiano am 11 Jan. 2020
Kommentiert: Benjamin Stucky am 6 Nov. 2023
Hi,
I am looking for thoughts on a recipe to perform a moving quantile in Matlab. Essentially I am trying to obtain the 98% value, for example, of a window that is instantaneously tracked through a large data set. I have tried looping in the quantile function, but this is extremely slow. I have had reasonable success using the sort function each time the window is moved, and then selecting the value based on the index that is 98% of the window size - see script below. But it is still several orders of magnitude slower than the built-in movmedian function. I don't see how a moving quantile should be much different then a moving median, just that the sorted value selected is not at 50%.
My thoughts are that after the initial window is sorted, an insertion sort algorithm is most appropriate since only a single value comes into the window each time the window moves 1 point. But really I am guessing at what is inside the movmedian function. I am not a programmer guy, so any advice here or examples on how to code an efficient moving quantile function is appreciated.
The example below is relatively quick with the given sampling rate, however in my example the sampling rate is 100,000 sps and takes much longer.
% Moving Quantile
fs=1000; % sampling rate (sps)
timeData=0:1/fs:40; % time (sec)
Data=rand(1,length(timeData)); % data (EU)
window=6300; % number of points used in 1 pt sliding window (pts)
dt_wnd=(window-1)/fs; % window time span (sec)
time_wnd=timeData(1)+dt_wnd/2:1/fs:timeData(end)-dt_wnd/2; % time vector, sliding window center points (sec)
Ind=floor(0.98*window); % index at 98% quantile
for j=1:length(time_wnd) % loop through all windows moving by 1 pt
SortedData=sort(Data(j:j+window-1)); % sorts data in the jth window
Val(j)=SortedData(Ind); % value at index
end
% Plotting
plot(timeData, Data)
axis([0 40 0.9 1.0])
hold on
plot(time_wnd,Val,'LineWidth',4)
hold off
title('Moving Quantile')
xlabel('time, s')
ylabel('Amplitude')
  1 Kommentar
Benjamin Stucky
Benjamin Stucky am 6 Nov. 2023
I am also interested in moving quantiles. The implementation of @Eike Petersen works great for smaller data sets. But I am dealing with very large time series. I found this C# reference implementation: https://aakinshin.net/posts/partitioning-heaps-quantile-estimator/ As I have no knowledge about C# and C++ wrappers, I am not able to create a mex file from this. If somebody could help to implement it, this would be nice.

Melden Sie sich an, um zu kommentieren.

Akzeptierte Antwort

Eike Petersen
Eike Petersen am 11 Dez. 2020
My guess is that your guess is pretty good. :-)
I just hand-implemented a version of this with a naive insertion sort algorithm, and it's roughly 50-80x slower than medfilt1. I'm not sure you can get much quicker without switching to C / a mex-file.
You can find it here; it works for N-D arrays, with NaNs, and with different padding strategies: https://de.mathworks.com/matlabcentral/fileexchange/84200-movquant

Weitere Antworten (0)

Produkte


Version

R2019b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by