Error in "predict" in the "Gesture Recognition using Videos and Deep Learning" example

Hello

I am trying to run the example of "Gesture Recognition using Videos and Deep Learning" in Matlab.

However, when the code reaches to the line below

dlYPred = predict(slowFastClassifier,dlVideo);

Matlab returns the error below:

Error using predict
No valid system or dataset was specified.

Does anyone have any sugesttion how to resolve the issue?

Thanks

[Walter Roberson]

could you confirm that you already installed Computer Vision Toolbox™ Model for SlowFast Video Classification?

[SM]

Yes

I have installed Computer Vision Toolbox and Deep Learning Toolbox.

I have also installed Computer Vision Toolbox Model for SlowFast Video Classification.

[Walter Roberson]

The example worked for me on R2022a on MacOS Catalina. It took a bit of time to run.

[SM]

Thanks for your resposne.

Have you used ".mlx" file to run it or you created a ".m" file and reproduced the example?

Becauase, when I used the live script (.mlx) it runs with no problem. However, when I copy the codes into a ".m" file, and try to reprocude the same, results, I face such error. I have checket the syntax, and functions to be correct.

I cannot figure out what mistake I made?

Here is the code that I try to run:

I downloaded the files and put then in the path, and all functions are in the code directory.

clear
 clc

 %%% Expample Link: https://au.mathworks.com/help/vision/ug/gesture-recognition-using-slowfast-video-classification.html

 %{
For this code the slowFastVideoClassifier add-on must be installed
 %}

%%  
 downloadFolder='./Gesture'

%%                                      Getting the classes
 pretrainedDataFile =strcat(downloadFolder,'/slowFastPretrained_fourClasses.mat');
pretrained = load(pretrainedDataFile);
slowFastClassifier = pretrained.data.Params;

classes = slowFastClassifier.Classes


%%
 doTraining = false;

 groundTruthFolder = fullfile(downloadFolder,"groundTruthFolder");
if ~isfolder(groundTruthFolder)
    mkdir(groundTruthFolder);
end





groundTruthFolder = fullfile(downloadFolder,"groundTruthFolder");
trainingFolder = fullfile(downloadFolder,"videoScenes");

extractVideoScenes(groundTruthFolder,trainingFolder,classes);

%%

numFrames = 16;
frameSize = [112,112];
numChannels = 3;
isDataForTraining = true;
dsTrain = createFileDatastore(trainingFolder,numFrames,numChannels,classes,isDataForTraining);


baseNetwork = "resnet50-3d";
inputSize = [frameSize,numChannels,numFrames];
slowFast = slowFastVideoClassifier(baseNetwork,string(classes),InputSize=inputSize);
slowFast.ModelName = "Gesture Recognizer Using Deep Learning";


dsTrain = transform(dsTrain,@augmentVideo);


preprocessInfo.Statistics   = slowFast.InputNormalizationStatistics;
preprocessInfo.InputSize    = inputSize;
preprocessInfo.SizingOption = "resize";

dsTrain = transform(dsTrain,@(data)preprocessVideoClips(data,preprocessInfo));


params.Classes = classes;
params.MiniBatchSize = 5;
params.NumIterations = 600;
params.CosineNumIterations = [100 200 300];
params.SaveBestAfterIteration = 400;
params.MinLearningRate = 1e-4;
params.MaxLearningRate = 1e-3;
params.Momentum = 0.9;
params.Velocity = [];
params.L2Regularization = 0.0005;
params.ProgressPlot = false;
params.Verbose = true;
params.DispatchInBackground = true;
params.NumWorkers = 12;


%%                                              Train Video Classifieor

params.ModelFilename = "slowFastPretrained_fourClasses.mat";
if doTraining
    epoch = 1;
    bestLoss = realmax;
    accTrain = [];
    lossTrain = [];

    iteration = 1;
    start = tic;
    trainTime = start;
    shuffled = shuffleTrainDs(dsTrain);

    % Number of outputs is two: One for RGB frames, and one for ground truth labels.
    numOutputs = 2;
    mbq = createMiniBatchQueue(shuffled, numOutputs, params);
    
    % Use the initializeTrainingProgressPlot and initializeVerboseOutput
    % supporting functions, listed at the end of the example, to initialize
    % the training progress plot and verbose output to display the training
    % loss, training accuracy, and validation accuracy.
    plotters = initializeTrainingProgressPlot(params);
    initializeVerboseOutput(params);

    while iteration <= params.NumIterations

        % Iterate through the data set.
        [dlX1,dlY] = next(mbq);

        % Evaluate the model gradients and loss using dlfeval.
        [gradients,loss,acc,state] = ...
            dlfeval(@modelGradients,slowFast,dlX1,dlY);

        % Accumulate the loss and accuracies.
        lossTrain = [lossTrain, loss];
        accTrain = [accTrain, acc];

        % Update the network state.
        slowFast.State = state;

        % Update the gradients and parameters for the video classifier
        % using the SGDM optimizer.
        [slowFast,params.Velocity,learnRate] = ...
            updateLearnables(slowFast,gradients,params,params.Velocity,iteration);

        if ~hasdata(mbq) || iteration == params.NumIterations
            % Current epoch is complete. Do validation and update progress.
            trainTime = toc(trainTime);

            accTrain = mean(accTrain);
            lossTrain = mean(lossTrain);

            % Update the training progress.
            displayVerboseOutputEveryEpoch(params,start,learnRate,epoch,iteration,...
                accTrain,lossTrain,trainTime);
            updateProgressPlot(params,plotters,epoch,iteration,start,lossTrain,accTrain);

            % Save the trained video classifier and the parameters, that gave 
            % the best training loss so far. Use the saveData supporting function,
            % listed at the end of this example.
            bestLoss = saveData(slowFast,bestLoss,iteration,lossTrain,params);
        end

        if ~hasdata(mbq) && iteration < params.NumIterations
            % Current epoch is complete. Initialize the training loss, accuracy
            % values, and minibatchqueue for the next epoch.
            accTrain = [];
            lossTrain = [];
            
            epoch = epoch + 1;
            trainTime = tic;
            shuffled = shuffleTrainDs(dsTrain);
            mbq = createMiniBatchQueue(shuffled, numOutputs, params);            
        end

        iteration = iteration + 1;
    end

    % Display a message when training is complete.
    endVerboseOutput(params);

    disp("Model saved to: " + params.ModelFilename);
end

%%                                      Evaluate the trained data
isDataForTraining = false;
dsEval = createFileDatastore(trainingFolder,numFrames,numChannels,classes,isDataForTraining);
dsEval = transform(dsEval,@(data)preprocessVideoClips(data,preprocessInfo));

if doTraining
    transferLearned = load(params.ModelFilename);
    slowFastClassifier = transferLearned.data.slowFast;
end


numOutputs = 2;
mbq = createMiniBatchQueue(dsEval,numOutputs,params);

numClasses = numel(params.Classes);
cmat = sparse(numClasses,numClasses);

while hasdata(mbq)
    [dlVideo,dlY] = next(mbq);

    % Computer the predictions of the trained SlowFast 
    % video classifier.
    dlYPred = predict(slowFastClassifier,dlVideo);
    dlYPred = squeezeIfNeeded(dlYPred,dlY);

    % Aggregate the confusion matrix by using the maximum
    % values of the prediction scores and the ground truth labels.
    [~,YTest] = max(dlY,[],1);
    [~,YPred] = max(dlYPred,[],1);
    cmat = aggregateConfusionMetric(cmat,YTest,YPred);
end


evalClipAccuracy = sum(diag(cmat))./sum(cmat,"all")


figure
chart = confusionchart(cmat,classes);

Answer 1

[Walter Roberson]

Your code has

slowFastClassifier = pretrained.data.Params;

but the actual code (in both the posted example and the mlx file) is

slowFastClassifier = pretrained.data.slowFast;

Because of that, you were trying to run predict() on a struct(), which was getting you a predict() function defined by the System Identification Toolbox instead of the predict() method appropriate for the slowFastVideoClassifier object.

[SM]

Thank you so much, it fixed my issue.