Create Baseline Tests for MATLAB Code

Since R2024b

In some cases, qualifying the output of MATLAB^® code through automated testing is not straightforward. For example, a MATLAB function that performs complicated mathematical operations, such as solving a system of differential equations, might produce an output that requires manual inspection. With MATLAB Test™, you can create and run baseline tests to verify that such a function continues to produce the manually inspected "known good" output, referred to as baseline data. Baseline tests are useful for regression testing when you make changes to your code but require its output not to deviate from the baseline data.

This example shows how to create and run a baseline test for a simulator, which is implemented as a MATLAB function named pendulumSimulator. First, you manually verify that the function works as expected. Then, you write a baseline test to create baseline data and verify consistent function behavior. The pendulumSimulator function solves a system of differential equations to calculate the angular position and velocity of a pendulum at different points in time. To view the complete code for pendulumSimulator, see Summary of Simulator Function.

Inspect Simulation Results

In a file named pendulumSimulator.m in your current folder, create the pendulumSimulator function. To manually verify that the function works as expected, call the function with a set of simulation parameters, and then plot the returned angular position and velocity values against the simulated time.

simParams.length = 1.5;
simParams.stopTime = 10;
simParams.initialAngle = pi/6;
simParams.initialAngularVelocity = 0;

[t,y] = pendulumSimulator(simParams);

tiledlayout("horizontal")
nexttile
plot(t,y(:,1))
xlabel("Time (s)")
ylabel("Angle (rad)")
nexttile
plot(t,y(:,2))
xlabel("Time (s)")
ylabel("Angular Velocity (rad/s)")

Plots of the angle and angular velocity values returned by the pendulumSimulator function against time

The visualized data suggests that the simulated pendulum oscillates as expected. For example, the initial angular position and velocity are equal to the initial conditions specified in the simulation, and the oscillation period matches the theoretical value of $T = 2 π \sqrt{L / g}$ with L and g, respectively, representing the pendulum length and gravitational acceleration. For illustrative purposes, this level of inspection is sufficient in this example. You can consider the inspected values as the "known good" output of the function under test.

Write Baseline Test for Simulator Function

You can write baseline tests for your MATLAB code by creating a test class using baseline-specific parameterization. Specify parameterization properties in a properties block with the TestParameter, MethodSetupParameter, or ClassSetupParameter attribute. For more information about test parameterization, see Use Parameters in Class-Based Tests.

To write a baseline test for the pendulumSimulator function, in a file named SimulatorTest.m in your current folder, create the SimulatorTest test class by subclassing the matlab.unittest.TestCase class. Define the baseline data to use in the test by adding two parameterization properties that correspond to the two output arguments of the pendulumSimulator function. Then, use these properties to pass baseline data to a parameterized Test method, which implements the logic required for baseline testing. To view the complete code for SimulatorTest, see Test Class Definition.

Define Baseline Data

To define the baseline data to use in the test, in a properties block with the TestParameter attribute, add two properties named time and state that correspond to the first and second output arguments of the pendulumSimulator function. Set each property using the matlabtest.parameters.matfileBaseline function, which lets you define data in a MAT file as baseline data:

time property — Define the data in variable t of a MAT file named testdata.mat as the baseline data for the first output argument of pendulumSimulator.
state property — Define the data in variable y of a MAT file named testdata.mat as the baseline data for the second output argument of pendulumSimulator.

    properties (TestParameter)
        time = matlabtest.parameters.matfileBaseline( ...
            "testdata.mat",VariableName="t")
        state = matlabtest.parameters.matfileBaseline( ...
            "testdata.mat",VariableName="y")
    end

The MAT file does not need to exist when you create a test class. If you use baseline-specific qualification methods, such as verifyEqualsBaseline, or the matlabtest.constraints.EqualsBaseline constraint, the testing framework provides you with options to either create the MAT file or update it upon a qualification failure.

Implement Test Logic

To implement the logic required for baseline testing, in a methods block with the Test attribute, add a parameterized Test method named baselineTest that accepts the time and state properties as inputs. Then, implement the method by following these steps:

Return the function outputs to test by calling pendulumSimulator using the same simulation parameters as in the manual verification of the function.
Using a call to the verifyEqualsBaseline method, test the first function output actualTime against the baseline data represented by the time property.
Using another call to the verifyEqualsBaseline method, test the second function output actualState against the baseline data represented by the state property.

    methods (Test)
        function baselineTest(testCase,time,state)
            simParams.length = 1.5;
            simParams.stopTime = 10;
            simParams.initialAngle = pi/6;
            simParams.initialAngularVelocity = 0;

            [actualTime,actualState] = pendulumSimulator(simParams);

            testCase.verifyEqualsBaseline(actualTime,time)
            testCase.verifyEqualsBaseline(actualState,state)
        end
    end

Test Class Definition

This code provides the complete contents of the SimulatorTest class.

classdef SimulatorTest < matlab.unittest.TestCase
    properties (TestParameter)
        time = matlabtest.parameters.matfileBaseline( ...
            "testdata.mat",VariableName="t")
        state = matlabtest.parameters.matfileBaseline( ...
            "testdata.mat",VariableName="y")
    end

    methods (Test)
        function baselineTest(testCase,time,state)
            simParams.length = 1.5;
            simParams.stopTime = 10;
            simParams.initialAngle = pi/6;
            simParams.initialAngularVelocity = 0;

            [actualTime,actualState] = pendulumSimulator(simParams);

            testCase.verifyEqualsBaseline(actualTime,time)
            testCase.verifyEqualsBaseline(actualState,state)
        end
    end
end

Run Test to Create Baseline Data

Now that the SimulatorTest class definition is complete, you can run the baseline test to create baseline data in a MAT file and then verify consistent function behavior. In this example, you run the baseline test using the runtests function. You can also run the test interactively, for instance, using the MATLAB Test Manager app. For more information about ways to run tests, see Run MATLAB Tests.

Run the SimulatorTest test class. The baseline test fails because the testdata.mat file does not yet exist and there is no baseline data to compare the actual values against. However, the test diagnostics include a hyperlink for each qualification failure that enables you to create baseline data from the actual value that the framework records.

result = runtests("SimulatorTest");

Running SimulatorTest

================================================================================
Verification failed in SimulatorTest/baselineTest(time=testdata.mat(t),state=testdata.mat(y)).
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyEqualsBaseline failed.
    --> An error occurred when loading baseline data: 
    --> MATLAB:MatFile:NoFile
    --> Cannot access 't' because 'C:\work\testdata.mat' does not exist.
    --> Create baseline from recorded test data
    ------------------
    Stack Information:
    ------------------
    ...
    In C:\work\SimulatorTest.m (SimulatorTest.baselineTest) at 18
================================================================================

================================================================================
Verification failed in SimulatorTest/baselineTest(time=testdata.mat(t),state=testdata.mat(y)).
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyEqualsBaseline failed.
    --> An error occurred when loading baseline data: 
    --> MATLAB:MatFile:NoFile
    --> Cannot access 'y' because 'C:\work\testdata.mat' does not exist.
    --> Create baseline from recorded test data
    ------------------
    Stack Information:
    ------------------
    ...
    In C:\work\SimulatorTest.m (SimulatorTest.baselineTest) at 19
================================================================================
.
Done SimulatorTest
__________

Failure Summary:

     Name                                                                    Failed  Incomplete  Reason(s)
    =====================================================================================================================
     SimulatorTest/baselineTest(time=testdata.mat(t),state=testdata.mat(y))    X                 Failed by verification.

Because you already manually verified the actual values before creating the baseline test, approve them as baseline data by clicking both the Create baseline from recorded test data hyperlinks. Each time you click Create baseline from recorded test data, the testing framework saves the corresponding actual value to the MAT file and then displays a dialog box to confirm the successful operation. For example, when you click the first Create baseline from recorded test data hyperlink, the framework creates the testdata.mat file and saves the recorded values in actualTime as variable t to the file.

Now that baseline data exists for your test, you can verify that the pendulumSimulator function continues to produce the same outputs by running the baseline test. If a function output changes, for instance, due to code refactoring, the baseline test signals the change through a qualification failure. The test passes only if the function outputs remain the same as the baseline data in the MAT file. For instance, run the test again to test the function outputs against the baseline data in the t and y variables of testdata.mat. The test passes because pendulumSimulator has not changed since baseline data creation.

result = runtests("SimulatorTest");

Running SimulatorTest
.
Done SimulatorTest
__________

Run Test After Updating Function

Whenever you make changes to your MATLAB code, run the baseline tests to confirm that the code behavior aligns with the established baseline. If a baseline test fails, investigate the failure using the test diagnostics or debug it using the load method to identify the root cause. Baseline tests can be fragile due to their strict equality checks, meaning that a failure might result from the inherent fragility of the test itself. If you determine that the code behaves as expected, update the existing baseline data using the hyperlinks in the test diagnostics.

The pendulumSimulator function uses the ode45 solver to simulate the pendulum motion. For higher accuracy, update pendulumSimulator to use the ode78 solver instead.

function [t,y] = pendulumSimulator(params)
% Simulation parameters
l = params.length;
tspan = [0 params.stopTime];
y0 = [params.initialAngle params.initialAngularVelocity];

odefun = @(t,y) pendulum(t,y,l);
[t,y] = ode78(odefun,tspan,y0);  % Solve the pendulum equations
end

% System of differential equations to solve
function dydt = pendulum(~,y,l)
g = 9.8;   % Gravitational acceleration
dydt(1,1) = y(2);
dydt(2,1) = -(g/l) * sin(y(1));
end

Because you updated the pendulumSimulator function, run the baseline test in the SimulatorTest class to compare the new function outputs against the baseline data in testdata.mat. The test fails because the ode78 solver results in values that are not strictly equal to the existing baseline data. In particular, the test diagnostics indicate that the actual and baseline array sizes are not the same.

result = runtests("SimulatorTest");

Running SimulatorTest

================================================================================
Verification failed in SimulatorTest/baselineTest(time=testdata.mat(t),state=testdata.mat(y)).
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyEqualsBaseline failed.
    --> NumericComparator failed.
        --> Sizes do not match.
            
            Actual size:
               145     1
            Expected size:
               149     1
        
        Actual Value:
            145x1 double
        Expected Value:
            149x1 double
    --> Export test data to workspace | Update baseline from recorded test data
    ------------------
    Stack Information:
    ------------------
    ...
    In C:\work\SimulatorTest.m (SimulatorTest.baselineTest) at 18
================================================================================

================================================================================
Verification failed in SimulatorTest/baselineTest(time=testdata.mat(t),state=testdata.mat(y)).
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyEqualsBaseline failed.
    --> NumericComparator failed.
        --> Sizes do not match.
            
            Actual size:
               145     2
            Expected size:
               149     2
        
        Actual Value:
            145x2 double
        Expected Value:
            149x2 double
    --> Export test data to workspace | Update baseline from recorded test data
    ------------------
    Stack Information:
    ------------------
    ...
    In C:\work\SimulatorTest.m (SimulatorTest.baselineTest) at 19
================================================================================
.
Done SimulatorTest
__________

Failure Summary:

     Name                                                                    Failed  Incomplete  Reason(s)
    =====================================================================================================================
     SimulatorTest/baselineTest(time=testdata.mat(t),state=testdata.mat(y))    X                 Failed by verification.

To further investigate each qualification failure, export the actual value and the corresponding baseline data to the workspace by clicking Export test data to workspace in the test diagnostics. The framework exports the values as actual and baseline and displays a dialog box to confirm the successful operation.

For example, visualize the actual evaluation points (variable actualTime in the test file) and baseline evaluation points (variable t in the MAT file) using the values that are exported when you click the first Export test data to workspace hyperlink. Despite slight differences, both the actual and baseline vectors cover the entire simulation time.

plot(actual)
hold on
plot(baseline)
legend("Actual","Baseline",Location="northwest")

Plot including both the actual and baseline evaluation points. Both variables cover the entire simulation time.

After verifying the new function implementation using the exported test data, update the existing baseline data in the MAT file. To update the baseline data for a specific qualification, click the Update baseline from recorded test data hyperlink in the test diagnostics. The testing framework displays a dialog box, prompting you to confirm the update operation. Once you choose to proceed, the framework updates the MAT file using the new actual value.

For this example, update the baseline data in the t and y variables of testdata.mat by clicking both the Update baseline from recorded test data hyperlinks. Then, run the baseline test. The test passes because the updated baseline data in the MAT file matches the outputs that the new version of pendulumSimulator produces.

result = runtests("SimulatorTest");

Running SimulatorTest
.
Done SimulatorTest
__________

Summary of Simulator Function

This code provides the contents of the pendulumSimulator function, which uses the ode45 solver to simulate the motion of a pendulum, assuming no friction or air resistance.

function [t,y] = pendulumSimulator(params)
% Simulation parameters
l = params.length;
tspan = [0 params.stopTime];
y0 = [params.initialAngle params.initialAngularVelocity];

odefun = @(t,y) pendulum(t,y,l);
[t,y] = ode45(odefun,tspan,y0);  % Solve the pendulum equations
end

% System of differential equations to solve
function dydt = pendulum(~,y,l)
g = 9.8;   % Gravitational acceleration
dydt(1,1) = y(2);
dydt(2,1) = -(g/l) * sin(y(1));
end

The function accepts the simulation parameters, expressed in SI units, as a scalar structure with these fields:

length — Length of the pendulum
stopTime — Duration of the simulation
initialAngle — Initial angular position of the pendulum measured from the equilibrium axis
initialAngularVelocity — Initial angular velocity of the pendulum

The function returns the evaluation points and solutions as numeric arrays:

t — Evaluation points, returned as an N-by-1 column vector
y — Solutions, returned as an N-by-2 matrix in which the first and second columns, respectively, represent the angular position and velocity of the pendulum