How can I better classify my data

Hi everyone,

I am trying to separate data into two groups using clustering (k-means) as you can see in the picture attached. My algorithm work perfectly on some data (see pic1),the problem is that it does not work in some regions for other data (see pic2), do you have any suggestions on how I can better filter the upper points ? Do you think a regression on the lower points would solve the problem ? If yes how can I do it only for these lower points and not all the data ?

Thank you in advance :)

[the cyclist]

Can you upload the data, in a MAT file? The pictures are so small that it is difficult to discern what the red-circled region looks like.

In general, if the points are right on top of each other (not nicely separated as in Pic 1), no unsupervised clustering algorithm is going to be able to distinguish them.

[Adam Danz]

In pic #2, after running the k-means clustering fit a line to the blue data. Then compute the residual error of all points (blue and black) from that line. Set a very small threshold of error that can be used to categorize the two sets. This assumes the blue lines are nearlyl linear although the one on the left looks like it might be exponential in which case you'd want to fix an exp curve.

[Izem]

@the cyclist, thank you for your answer. I can't upload a mat file but I ll upload a more visible picture.

[Izem]

@Adam Danz, thank you for your answer. your solution seems good, do you know how can I skip the clustering part and fit directly the lower curve ? when calculating the sum of squarred errors how can chose the points from the line ? do you have an example ?

Thank you again.

[Adam Danz]

You wouldn't skip the cluster part. You would fit the line to the blue section of the cluster results. Then you would compute the residuals for all points, black and blue. The points along the curve that is fit to the blue line will have small residual error whereas the black points will have large error. As the cyclist aptly mentioned, the cluster of dots to the far right pose a problem but with this method, some should clearly be members of the blue line and others not.

To choose a residual error threshold, you could base it on the residuals from the blue line. For example, all dots that have a smaller residual error than the max residual error of the blue dots will be a member of the blue line. Or, maybe you want to use the mean residual error +/- 2 or 3 std.

[Izem]

Alright ! thank you sir.

[the cyclist]

I think there are a few approaches you could do here. Similar to what Adam suggested, you might also try doing kmeans with k=3, which I expect would separate the data into:

1. clearly in lower line
2. clearly in upper line
3. a bunch at the end

Then you could fit curves to clusters 1 and 2, and then assign the points of cluster 3 to the line it is closer to. Be aware that this is going to tend to underestimate the error in those points.

I think the best approach will depend on what you are doing next. It is conceivable to me that the best approach would be to ignore the cluster-3 data. Also, there are some clustering techniques that assign a probability that each point belongs to a particular cluster. That might be useful.

[Izem]

Thank you for your answer. I will try that since I can't ignore cluster-3 data.

[Izem]

@Adam Danz I clustered the data with k-means and fitted them using :

[pop,gof] = fit(X1,Y1,'poly2')

After that I only can plot the residuals using :

plot(pop,X1,Y1,'residuals')

and can't have access to the computed residuals of each point so that I can set a treshold and classify them, can you help please ?

[Adam Danz]

You can access the residuals using

 [pop,gof,output] = fit(X1,Y1,'poly2');
 resid = output.residuals;

however, that will only contain the residuals of the data used for fitting (blue dots). You need the residuals for all of the dots so you'll need to compute the residuals.

 [pop,gof,output] = fit(X1,Y1,'poly2'); 
 resids = pop(XALL) - YALL;

where XALL and YALL are the coordinates to all of the points. pop(XALL) produces the y-estimates. Subtract the actual y coordinates from the y-estimates to get the residuals.

For thresholding, use absolute values of the residuals.

[Izem]

Alright! thank you !

Answer 1

[Image Analyst]

I'd use fitPolynomialRANSAC(), if you have the Computer Vision Toolbox. Train it with values less than 1 and fit a second or third order polynomial. RANSAC had the advantage over "dumb" regressions in that it can fit a polynomial through extremely noisy neighborhoods. So the points in that shotgun blast cluster on the right won't be all in one class.

[Izem]

Thank you for the answer. I will download this toolbox.