Create a 'fake' video from an image and previously calculated optical flow vectors

I calculated Horn & Schunk's Optical Flow on a video, thus obtaining a Vx and Vy matrix containing the velocity vectors of the pixels in the video.

Now I would like to do the reverse procedure, i.e., start from frame 1 of the video, apply the Optical Flow obtained frame by frame, and create a 'fake' video (which should look like the real video), in which the pixels move according to their value of Vx and Vy of each frame.

I tried to create a code and the pixels move but I always see frame1, i.e., not all the pixels move. Could someone help me with this? Thank you

here is the code:

video = VideoReader('autoincrocio.mp4'); %choose a video 
outputVideo_rec = VideoWriter('autoincrocio_INV.mp4', 'MPEG-4'); %name of the finale fake video

% Set the video properties, making sure they are within the allowed limits
outputVideo_rec.Quality = 100; % Imposta la qualità desiderata (può variare da 0 a 100)
outputVideo_rec.FrameRate = 10; %frame al secondo

open(outputVideo_rec);
%read frame 1, to get dimensions
frame1 = read(video,1);
[rows, cols, z] = size(frame1);

%loop for the n frames that I have
for p=1:30
    flow=flow_save{p}; %I saved the optical flow values for each frame. Flow_save is a 1x30 row vector where at each position is the optica flow of the 30 frames
    Vx = flow.Vx; 
    Vy = flow.Vy;
    
    %calculation displacements
    frameRate = videoReader.FrameRate;
    time_between_frames =  1/frameRate;
    displacementX = Vx *  time_between_frames;
    displacementY = Vy *  time_between_frames;
    
    % Amplifies displacements - this is because the values of displacementX and displacementY are too small, so I thought of amplifying them
    scale = 100; % You can adjust this value to get a more noticeable movement
    displacementX = displacementX * scale;
    displacementY = displacementY * scale;
    frame_sim=frame1;
    
    % Applies shifts and generates subsequent frames
    for i = 1:rows    
        for j = 1:cols
            new_x = round(i + displacementX(i, j));
            new_y = round(j + displacementY(i, j));
            
            % Checks that the new coordinates are within the image boundaries.
            if new_x >= 1 && new_x <= righe && new_y >= 1 && new_y <= colonne
                frame_sim(new_x, new_y,:)=frame1(i, j,:);   
            end
        end
    end
    writeVideo(outputVideo_rec, frame_sim);
end

% close video output
close(outputVideo_rec);
disp('Displacement video successfully generated.');

[Walter Roberson]

Are the Vx and Vy relative to the previous frame, or are they cumulative since the first frame?

[NICOLE LONGHI]

Vx and Vy are calculated for each frame, in fact the pixels move by that amount

Answer 1

[Walter Roberson]

    time_between_frames =  1/frameRate;
    displacementX = Vx *  time_between_frames;
    displacementY = Vy *  time_between_frames;

You are calculating one frame's-worth of displacement. That implies that you are moving relative to the previous frame.

    frame_sim=frame1;

But here you are resetting to the first frame. If you are doing relative motion to the previous frame, then frame_sim should be initialized to frame1 before the loop, but otherwise motion should be allowed to accumulate. If you are doing motion since the first frame then you need to account for the fact that you are now on frame #p not on frame #1, so you have had either (p)/frameRate motion or (p-1)/frameRate motion.

[NICOLE LONGHI]

Thank you very much! I modified the code like this:

...

%loop for the n frames that I have

for p=1:30

frame = read(video,p);

flow=flow_save{p}; %I saved the optical flow values for each frame. Flow_save is a 1x30 row vector where at each position is the optica flow of the 30 frames

Vx = flow.Vx;

Vy = flow.Vy;

%calculation displacements

time_between_frames = 1; the time between frames is 1 because now I calculate the time between one frame and the next frame so it is 1

displacementX = Vx * time_between_frames;

displacementY = Vy * time_between_frames;

frame_sim=frame;

...

In this way, at each iteration I read a new frame.

The resulting video is actually better with these changes, although I still see the equal background of frame 1, and above that the movement of later frames (basically I see a movement of an object above the still object image).