Extracting moving foreground objects from an *almost* constant field of view video through stabilisation.

My video is of a coral reef environment. The camera was placed on a tripod on the seafloor. The coral is static but large fish are moving around them. My long term goal is to track the fish. I am exploring machine-learning options, but since the fish are in motion and the coral are static, I think I can produce a sequences of frame that are masks of just the fish to make training easier.

I am trying this by taking the difference in pixel values between frames as the regions of the frames where fish are will change but the regions where the coral are wil not:

absdiff_frame = imabsdiff(image_stack_bw{ii}, image_stack_bw{ii+window});

This works to some extent, but the masks also highlight the edges of the coral since the fixed camera is moving slightly with the current of the ocean.

To fix this, I am trying to stabilise the video. The best option so far seems to be point feature mapping. I know that my problem is more difficult than usual stabilisation as there are non static objects in the foreground. However, this video faces the same problem as I do and stabilises. I want to reproduce this code.

This is my current code:

vv = VideoWriter("stabilised_video.avi");

vv.FrameRate = 60;

open(vv)

v = VideoReader('video_to_stabilise.avi');

% Process all frames in the video

imgB = im2gray(im2single(read(v,(1))));

cumulativeTform = simtform2d;

i = 1;

while i < v.NumFrames

imgA = imgB;

imgB = im2gray(im2single(read(v,(i+1))));

tformAffine = cvexEstStabilizationTform(imgA,imgB);

sRtTform = cvexTformToSRT(tformAffine);

cumulativeTform = simtform2d(cumulativeTform.A * sRtTform.A);

imgBp = imwarp(imgB,cumulativeTform,OutputView=imref2d(size(imgB)));

writeVideo(vv,[imgB imgBp])

i = i+1;

end

close(vv)

And the functions:

function tform = cvexEstStabilizationTform(leftI,rightI,ptThresh)

if nargin < 3 || isempty(ptThresh)

ptThresh = 0.01;

end

pointsA = detectFASTFeatures(leftI,MinQuality= ptThresh);

pointsB = detectFASTFeatures(rightI,MinQuality= ptThresh);

[featuresA,pointsA] = extractFeatures(leftI,pointsA);

[featuresB,pointsB] = extractFeatures(rightI,pointsB);

indexPairs = matchFeatures(featuresA,featuresB);

pointsA = pointsA(indexPairs(:, 1),:);

pointsB = pointsB(indexPairs(:, 2),:);

tform = estgeotform2d(pointsB,pointsA,'affine');

function [tformOut,s,ang,t,R] = cvexTformToSRT(tformIn)

tformAffine1 = tformIn.A;

R = tformAffine1(1:2,1:2);

ang = mean([atan2(-R(3),R(1)) atan2(R(2),R(4))]);

s = mean(R([1 4])/cos(ang));

angd = rad2deg(ang);

t = tformAffine1(1:2,3);

tformOut = simtform2d(s,angd,t);

The stabilisation is bad. Few features are detected by FAST, which may be due to there being few distinct corners in my image. However the coral look very similar to the tree shown in video stabilisation using point feature mapping example. I have uploaded a segment of my video here.

Is there a way to improve this stabilisation? Or do you suggest any other approaches produce a video of masks of just the fish?

[Udit06]

Hi Ella,

You can consider using optical flow method to estimate the motion between frames. Optical flow can provide a dense motion field that might be more resilient to the lack of distinct features. You can refer to the following resource to understand more about optical flow.

https://web.stanford.edu/class/cs231m/lectures/lecture-7-optical-flow.pdf

You can use the MATLAB's opticalFlowLK function which is an optical flow implementation based on Lucas-Kanade method. Refer to the following documentation for more details:

https://www.mathworks.com/help/vision/ref/opticalflowlk.html

I hope this helps.