Script to Find Repeating Data within Tolerance

Hello,

I have a table that has over 6,000 rows of data with 8 columns, with each column representing a different data.

Many of the rows of data are similar but not exactly the same as other rows.

In other words, plotting column 1 vs column 2, and column 1 vs column 3, etc. gives a sinusoidal shape with a periodic behavior, but it is not exactly repeating, but slightly increasing.

With that in mind, I wrote a script such that it finds similar data within a tolerance, and removes that row of data if it is deemed as similar.

In short, the code is something like this:

for J=1:Length
    for I=1:Length
        if I~=J
            if abs(table.colA(J)-table.colA(I))<=tolerance && abs(table.colB(J)-table.colB(I))<=tolerance && % Repeat for other columns
                    table.colA(I)=-1;
                    table.colB(I)=-1;
                    % Repeat for other columns
            else
                
            end
        else
            
        end
    end
end

    % Then I keep only the rows whose values are greater than -1

What I have works, I got it to go from over 6,000 rows to about 150 rows

However it is very computationally expenisve, meaning it takes several hours to run

Does anyone have any advice to speed this up or make it more efficient?

Or even just a better way to do this task?

I know there is an easy function that removes duplicate rows in a table, but since none of the rows are exactly the same, this won't work.

Is it possible to do this to a tolerance, like two rows not exactly duplicates, but "close enough" to each other.

I was looking into it here and the in maual but couldn't find any easier way to do it than what I did above.

Any help or suggestions is appreciated, thanks.

Answer 1

[Jon]

I found the command uniquetol

I'm going to give that a shot but I believe it is what I'm looking for.

It seems I didn't look hard enough haha