how do I play sound during drawing a live plot

I create a live plot in my code, which is a transformation of a sound file to a neuronal spike. I would like to play the sound during the plot drawing and save all of it to a video file.

here is the function that already creates the live plot and saves it to a avi file:

% plot_k_layer creates a very cool online plot of K layer spikes and saves

% it to GIF file.

% Inputs:

% k_spike_mat= matrix of K lines and t_vec length columns of the K layer spikes data

% stp = time step for decaying

% Outputs:

% matrix - the K spikes data

function matrix = plot_k_layer_video(k_spike_mat, stp, sampnum)

% create the needed data for plot.

tau = 0.25;

t_vec = 0:stp:5*tau;

filter = exp(-t_vec/tau);

matrix = conv2(k_spike_mat, filter');

matrix = matrix(1:size(k_spike_mat, 1), :);

p = sqrt(size(matrix, 2));

assert(p == round(p));

matrix = reshape(matrix, size(matrix, 1), p, p);

% create plot

fig1 = figure(1);

axis tight manual % this ensures that getframe() returns a consistent size

filename = ['sample' num2str(sampnum) '.gif'];

fps = 30;

for k = 1:round(1/(fps*stp)):size(matrix, 1)

figure(1) % fireworks

imagesc(reshape(matrix(k, :, :), size(matrix, 2), size(matrix, 3)));

caxis([0, max(matrix(:))]);

hold on;

p = size(matrix, 2);

% drow lines

for j = 1:(p - 1)

plot(j.*[1, 1] + 0.5, [0, p+1], 'w');

plot([0, p+1], j.*[1, 1] + 0.5', 'w');

end

xlim([0, p] + 0.5);

ylim([0, p] + 0.5);

title(['Sample number ' num2str(sampnum) ]);

subtitle(['time: ' num2str(k/10000) ' sec' ]);

hold off;

%axis equal;

drawnow;

pause(1/fps);

% Capture the plot as an image

frame = getframe(fig1);

im = frame2im(frame);

[imind,cm] = rgb2ind(im,256);

% Write to the GIF File

if k == 1

imwrite(imind,cm,filename,'gif', 'DelayTime', 2/fps, 'Loopcount',inf);

else

imwrite(imind,cm,filename,'gif', 'DelayTime', 2/fps,'WriteMode','append');

end

F(k) = getframe(gcf) ;

end

writerObj = VideoWriter('neuronfire.avi');

writerObj.FrameRate = 10;

% set the seconds per image

% open the video writer

open(writerObj);

% write the frames to the video

for i=1:round(1/(fps*stp)):size(matrix, 1)

% convert the image to a frame

frame = F(i) ;

writeVideo(writerObj, frame);

end

% close the writer object

close(writerObj);

which lines should I write to combine the sound file here?

this is a gif file of the video

thank you!

Answer 1

[Walter Roberson]

Computer Vision Toolbox

https://www.mathworks.com/help/vision/ref/vision.videofilewriter-system-object.html

There is also https://www.mathworks.com/help/dsp/ref/tomultimediafile.html from DSP toolbox, but that is for Simulink use only

[Mor Atzmon]

Thank you but I'm not sure how to use it. Any advice?

[Walter Roberson]

At the point you currently do

        writeVideo(writerObj, frame);

you would instead call

        writerObj(frame, sound_window)

where sound_window is the audio to be associated with the current frame.

Here writerObj is the result of calling vision.videoFileWriter() with a file name and with 'AudioInputPort', true

That is, you record a window of sound along with the image, as part of the multimedia file; then when you use an appropriate multimedia playback, the audio will be associated with the images.

[Mor Atzmon]

Is it only for 2021A? I work with 2020B (university licence) so I can't install it

[Walter Roberson]

It was introduced in R2012a. Not a typo --- 12 years ago.

[Mor Atzmon]

matlab couldn't find vision.videoFileWriter() function

[Walter Roberson]

vision.VideoFileWriter() with capital V on Video.

If that cannot be found, then it is possible that you do not have Computer Vision Toolbox installed.

Answer 2

[Sulaymon Eshkabilov]

You should include an audio file along with the videoFrame in your code, e.g.:

writerObj = videoFWriter('neuronfire.avi','my_music.wav');

See doc1 and doc2.

[Walter Roberson]

You cannot pass two file names to vision.videoFileWriter() or to audiowrite()