Remove certain rows from cell array that deviates by more than 10 mean absolute deviations from a rolling centered median (excluding the observation under consideration) of 50 observations (25 observations before and 25 after).

I have a cellarray A which has 10 columns of which i need to apply this condition - "deviated by more than 10 mean absolute deviations from a rolling centered median (excluding the observation under consideration) of 50 observations (25 observations before and 25 after)" on column 7 of each cell and then remove those rows from cells. And i need to put another condition on column 6 of each cell that - if it is more than 50 times the median value, then i need to remove those rows too.

Suugest a faster way to put both these conditions on each cell and subsequently remove those rows from each cell.Using R2016a

[NS]

Kindly help?

Answer 1

[Image Analyst]

Try this. Replace my col7 with the data from your column 7. It extracts only the good data from col7. Bad data are identified by magenta asterisks. Requires the Signal Processing toolbox to get the moving median filter, medfilt1().

% Create sample data
col7 = randn(1, 1000) + 1;
% Make outliers
col7(50:100:end) = 30;
medianFilteredSignal = medfilt1(col7, 51);
x = 1 : length(col7);
plot(x, col7, 'b-');
grid on;
ylim([-1, 31]);
hold on;
plot(x, medianFilteredSignal, 'r-');
% Identify good signals where the difference between the
% mean signal and median filtered signal is less than 10 times the
% median filtered value.
goodIndexes = abs(col7 - medianFilteredSignal) < 10 * medianFilteredSignal;
badIndexes = ~goodIndexes;
% Plot a star over the bad points.
plot(x(badIndexes), col7(badIndexes), 'm*', 'MarkerSize', 15, 'LineWidth', 2);
% Enlarge figure to full screen.
set(gcf, 'Units', 'Normalized', 'OuterPosition', [0, 0.04, 1, 0.96]);
% Extract only the good points:
col7Good = col7(goodIndexes);

[NS]

Sir I need to perform this outlier condition on a cell array wherein the cells are split for each day. And I have Signal Analysis App , is it the one needed?

Is there a simpler way to put this condition using cellfun?

And i need to put another condition on column 6 of each cell that - if it is more than 50 times the median value, then i need to remove those rows too.

[NS]

Any solution with cellfun as these are outlier conditions to be applied on each day .

[Image Analyst]

Can you attach some data?

[NS]

Sir I have attached the sample data. I have to remove outliers using 2 conditions for each day. First split the data into days and then kindly help me execute these two conditions-

1. Delete entries for which the spread is more that 50 times the median spread on that day.
2. Delete entries for which the mid-quote deviated by more than 10 mean absolute deviations from a rolling centered median (excluding the observation under consideration) of 50 observations (25 observations before and 25 after).

In the data MIDQUOTE column is 9 and BASPREAD is 10.

The source of these two condition is http://public.econ.duke.edu/~get/browse/courses/201/spr09/DOWNLOADS/MicroStructureNoise/bhls_kernels_practice_08.pdf

[Image Analyst]

What is "the median spread"? How are you defining spread? Is it the standard deviation? Or +/- 2 or 3 standard deviations?

[NS]

Sir the spread is already calculated as OFR-BID and put in column 10 as BASPREAD. I need to delete all those rows in each daycell whose spread is more that 50 times the median spread on that day.

Median spread is the median of the BASPREAD in each cell of the cell array.

[NS]

Sir basically i want to replicate this R package code rmOutliers (Page 60) and rmLargeSpread(Page 59) in highfreuency package - https://cran.r-project.org/web/packages/highfrequency/highfrequency.pdf in MATLAB.

Kindly see these two codes for any other clarity.

[NS]

Dear Sir

Any help on this ? Its basically how you clean a quote data in finance using these two conditions on each day.