Several ways, like smooth the boundary and find the residuals between the original coordinates and the smoothed coordinates. See attached demo for a start.
Or you could smooth the image with activecontour() or blurring and thresholding or using imclose(), then using xor() on the smoothed and original image and summing the white pixels in the result with bwarea() or sum().
Pick one and give it a try.
