Calculate Optical Flow by Using Neighborhood Processing Subsystem Blocks

This example shows how to calculate optical flow in a video by using Neighborhood Processing Subsystem blocks. Optical flow is the distribution of the apparent velocities of objects in an image. Use optical flow to identify and track objects in a video.

Inspect Model

Open the model.

model = 'OpticalFlowNeighborhoodExample';
open_system(model);

The model references a video input, rhinos.avi, by using the From Multimedia File block from Computer Vision Toolbox. At each iteration, the model passes the current and previous frames of the video to the DUT subsystem, which performs the optical flow calculation. This calculation outputs an optical flow matrix which represents the apparent motion at each pixel of the video. The model overlays these calculated motion vectors over the input video.

Inspect DUT Subsystem

Open the DUT subsystem.

This example calculates optical flow by using the Lucas-Kanade method. The Lucas-Kanade method requires the values ${\mathit{I}}_{\mathit{x}}$, ${\mathit{I}}_{\mathit{y}}$, and ${\mathit{I}}_{\mathit{t}}$, which are the derivatives of pixel brightness along the horizontal direction, vertical direction, and time, respectively.

The Compute Ix Neighborhood Processing Subsystem block calculates ${\mathit{I}}_{\mathit{x}}$ by using a 1-by-5 neighborhood and these blocks. The Neighborhood control block NeighborhoodConfig specifies the neighborhood size using its Neighborhood size parameter.

The Compute Iy block calculates ${\mathit{I}}_{\mathit{y}}$ with the same blocks and a 5-by-1 neighborhood. The model calculates ${\mathit{I}}_{\mathit{t}}$ as the difference between the current video frame and previous video frame by using a Sum block.

The horizontal optical flow $\mathit{u}$ and vertical optical flow $\mathit{v}$ represent the solution to this equation.

$${\mathit{I}}_{\mathit{x}}\mathit{u}+{\mathit{I}}_{\mathit{y}}\mathit{v}+{\mathit{I}}_{\mathit{t}}=0$$

To solve this equation, the Lucas-Kanade method divides the input image into smaller sections and assumes a constant velocity in each section. Then it performs a weighted, least-square fit of the optical flow constraint equation to a constant model for ${\left[\mathit{u}\mathit{v}\right]}^{\mathit{T}}$ in each section $\Omega$. The method achieves this fit by minimizing this equation, where $\mathit{W}$ is a window function that emphasizes the constraints at the center of each section:

$$\sum _{\mathit{x}\in \Omega }{\mathit{W}}^2{\left[{\mathit{I}}_{\mathit{x}}\mathit{u}+{\mathit{I}}_{\mathit{y}}\mathit{v}+{\mathit{I}}_{\mathit{t}}\right]}^2$$

The solution to the minimization problem is:

$$\left[\begin{array}{cc}\sum _{}^{}{\mathit{W}}^2{\mathit{I}}_{\mathit{x}}^2& \sum _{}^{}{\mathit{W}}^2{\mathit{I}}_{\mathit{x}}{\mathit{I}}_{\mathit{y}}\\ \sum _{}^{}{\mathit{W}}^2{\mathit{I}}_{\mathit{x}}{\mathit{I}}_{\mathit{y}}& \sum _{}^{}{\mathit{W}}^2{\mathit{I}}_{\mathit{y}}^2\end{array}\right]\left[\begin{array}{c}\mathit{u}\\ \mathit{v}\end{array}\right]=-\left[\begin{array}{c}\sum _{}^{}{\mathit{W}}^2{\mathit{I}}_{\mathit{x}}{\mathit{I}}_{\mathit{y}}\\ \sum _{}^{}{\mathit{W}}^2{\mathit{I}}_{\mathit{y}}{\mathit{I}}_{\mathit{t}}\end{array}\right]$$

The example calculates ${\mathit{I}}_{\mathit{x}}^2$, ${\mathit{I}}_{\mathit{x}}{\mathit{I}}_{\mathit{y}}$, ${\mathit{I}}_{\mathit{y}}^2$, and ${\mathit{I}}_{\mathit{y}}{\mathit{I}}_{\mathit{t}}$ by using Product blocks. The example implements $\mathit{W}$by using Neighborhood Processing Subsystem blocks with 5-by-5 neighborhoods.

Inspect LK Method Subsystem

Open the LK Method subsystem.

The Calculate Eigenvalues Neighborhood Processing Subsystem calculates the eigenvalues of $\mathit{A}=\left[\begin{array}{cc}\mathit{a}& \mathit{b}\\ \mathit{c}& \mathit{d}\end{array}\right]=\left[\begin{array}{cc}\sum _{}^{}{\mathit{W}}^2{\mathit{I}}_{\mathit{x}}^2& \sum _{}^{}{\mathit{W}}^2{\mathit{I}}_{\mathit{x}}{\mathit{I}}_{\mathit{y}}\\ \sum _{}^{}{\mathit{W}}^2{\mathit{I}}_{\mathit{x}}{\mathit{I}}_{\mathit{y}}& \sum _{}^{}{\mathit{W}}^2{\mathit{I}}_{\mathit{y}}^2\end{array}\right]$ by solving the equation ${\lambda }_{\mathit{i}}=\frac{\mathit{a}+\mathit{c}}{2}\pm \frac{\sqrt{4{\mathit{b}}^2+{\left(\mathit{a}+\mathit{c}\right)}^2}}{2};\mathit{i}=1,2$.

The example implements the rest of the Lucas-Kanade method by using the Conditions subsystem and a MATLAB Function block. For information about these calculations, see opticalFlowLK (Computer Vision Toolbox).

Simulation and Results

Simulate the model. The model displays the input video with vectors overlaid, representing the optical flow.

Close the model.

close_system(model);