testrunner

R2026b

Create test runner

Syntax

runner = testrunner

runner = testrunner("minimal")

runner = testrunner("textoutput")

runner = testrunner(___,Name=Value)

Description

runner = testrunner creates a default test runner, which is equivalent to the runner that the testing framework configures by default when you call the runtests function.

You can call the methods of runner to run and operate on your test suite and to customize running tests. For example, to run a suite of tests, use run(runner,suite).

runner = testrunner("minimal") creates a minimal runner with no plugins installed. The returned test runner is the simplest runner possible and produces no text output. Use this syntax when you want to have complete control over which plugins to add to the runner.

runner = testrunner("textoutput") creates a runner that is configured for text output. The output produced includes test progress as well as diagnostics in the event of test failures.

This syntax creates runners that tend to run tests more quickly, because the testing framework does not record diagnostics on test results produced by a nondefault runner. For more information, see Programmatically Access Test Diagnostics.

runner = testrunner(___,Name=Value) specifies options using one or more name-value arguments in addition to any of the input argument combinations in previous syntaxes. For example, runner = testrunner("textoutput",OutputDetail="Verbose") creates a test runner that displays test output detail at the maximum verbosity level. (since R2024b)

Examples

Create and Use Default Test Runner

Run a suite of tests with a default runner and access the results.

Create a function-based test sampleTest.m in your current folder. The file contains two tests that pass and one test that intentionally fails.

function tests = sampleTest
tests = functiontests(localfunctions);
end

function testA(testCase)      % Test passes
verifyEqual(testCase,2+3,5)
end

function testB(testCase)      % Test fails
verifyGreaterThan(testCase,13,42)
end

function testC(testCase)      % Test passes
verifySubstring(testCase,"Hello World!","llo")
end

Create a test suite from the tests in sampleTest.m. Then, create a default runner and run the tests.

suite = testsuite('SampleTest');
runner = testrunner;
results = run(runner,suite);

Running sampleTest
.
================================================================================
Verification failed in sampleTest/testB.
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyGreaterThan failed.
    --> The value must be greater than the minimum value.
    
    Actual Value:
        13
    Minimum Value (Exclusive):
        42
    ------------------
    Stack Information:
    ------------------
    In C:\work\sampleTest.m (testB) at 10
================================================================================
..
Done sampleTest
__________

Failure Summary:

     Name              Failed  Incomplete  Reason(s)
    ===============================================================
     sampleTest/testB    X                 Failed by verification.

Display the results from the second test.

results(2)

ans = 

  TestResult with properties:

          Name: 'sampleTest/testB'
        Passed: 0
        Failed: 1
    Incomplete: 0
      Duration: 0.3962
       Details: [1×1 struct]

Totals:
   0 Passed, 1 Failed, 0 Incomplete.
   0.39619 seconds testing time.

When you run tests with a default runner, the testing framework uses a DiagnosticsRecordingPlugin instance to record diagnostics on test results. Access the recorded diagnostics for the second test using the DiagnosticRecord field in the Details property on the TestResult object.

records = results(2).Details.DiagnosticRecord

records = 

  QualificationDiagnosticRecord with properties:

                          Event: 'VerificationFailed'
                     EventScope: TestMethod
                  EventLocation: 'sampleTest/testB'
          TestDiagnosticResults: [1×0 matlab.automation.diagnostics.DiagnosticResult]
     FrameworkDiagnosticResults: [1×1 matlab.automation.diagnostics.DiagnosticResult]
    AdditionalDiagnosticResults: [1×0 matlab.automation.diagnostics.DiagnosticResult]
                          Stack: [1×1 struct]
                         Report: 'Verification failed in sampleTest/testB.↵    ---------------------↵    Framework Diagnostic:↵    ---------------------↵    verifyGreaterThan failed.↵    --> The value must be greater than the minimum value.↵    ↵    Actual Value:↵        13↵    Minimum Value (Exclusive):↵        42↵    ------------------↵    Stack Information:↵    ------------------↵    In C:\work\sampleTest.m (testB) at 10'

Customize Runner to Generate Test Artifact

Generate JUnit-style test results by creating a minimal runner and then adding an XMLPlugin instance to the runner.

Create a function-based test sampleTest.m in your current folder. The file contains two tests that pass and one test that intentionally fails.

function tests = sampleTest
tests = functiontests(localfunctions);
end

function testA(testCase)      % Test passes
verifyEqual(testCase,2+3,5)
end

function testB(testCase)      % Test fails
verifyGreaterThan(testCase,13,42)
end

function testC(testCase)      % Test passes
verifySubstring(testCase,'hello, world','llo')
end

Create a test suite from the tests in sampleTest.m.

suite = testsuite('sampleTest');

Create a test runner with no plugins. This code creates a silent runner that produces no output. You can now install whatever plugins you like.

runner = testrunner('minimal');

Create an XMLPlugin instance that writes JUnit-style XML output to the file myTestResults.xml.

import matlab.unittest.plugins.XMLPlugin
xmlFile = 'myTestResults.xml';
p = XMLPlugin.producingJUnitFormat(xmlFile);

Add the plugin to the test runner and run the tests.

addPlugin(runner,p)
results = run(runner,suite);

Display the results from the second test.

results(2)

ans = 
  TestResult with properties:

          Name: 'sampleTest/testB'
        Passed: 0
        Failed: 1
    Incomplete: 0
      Duration: 0.0723
       Details: [1×1 struct]

Totals:
   0 Passed, 1 Failed (rerun), 0 Incomplete.
   0.0723 seconds testing time.

Check for diagnostics recorded on test results. If you were using a default runner, there would be a DiagnosticRecord field at this location. But because you are using a nondefault runner, the framework does not create such a field.

records = results(2).Details

records = struct with no fields.

Now, view the contents of the generated artifact.

disp(fileread(xmlFile))

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<testsuites>

  <testsuite errors="0" failures="1" name="sampleTest" skipped="0" tests="3" time="0.083731">
    <testcase classname="sampleTest" name="testA" time="0.0085045"/>
    <testcase classname="sampleTest" name="testB" time="0.0723">
      <failure type="VerificationFailure">Verification failed in sampleTest/testB.
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyGreaterThan failed.
    --&gt; The value must be greater than the minimum value.
    
    Actual Value:
        13
    Minimum Value (Exclusive):
        42
    ------------------
    Stack Information:
    ------------------
    In C:\TEMP\Examples\matlab-ex97531283\sampleTest.m (testB) at 10</failure>
    </testcase>
    <testcase classname="sampleTest" name="testC" time="0.0029273"/>
  </testsuite>

</testsuites>

Display Test Results in Command Window

Run a suite of tests with a runner that is configured for text output, and then access the results.

Create a function-based test sampleTest.m in your current folder. The file contains two tests that pass and one test that intentionally fails.

function tests = sampleTest
tests = functiontests(localfunctions);
end

function testA(testCase)      % Test passes
verifyEqual(testCase,2+3,5)
end

function testB(testCase)      % Test fails
verifyGreaterThan(testCase,13,42)
end

function testC(testCase)      % Test passes
verifySubstring(testCase,'hello, world','llo')
end

Create a test suite from the tests in sampleTest.m.

suite = testsuite('sampleTest');

Create a runner that produces text output and use it to run the tests.

runner = testrunner('textoutput');
results = run(runner,suite);

Running sampleTest
.
================================================================================
Verification failed in sampleTest/testB.
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyGreaterThan failed.
    --> The value must be greater than the minimum value.
    
    Actual Value:
        13
    Minimum Value (Exclusive):
        42
    ------------------
    Stack Information:
    ------------------
    In C:\TEMP\Examples\matlab-ex48684143\sampleTest.m (testB) at 10
================================================================================
..
Done sampleTest
__________

Failure Summary:

     Name              Failed  Incomplete  Reason(s)
    ===============================================================
     sampleTest/testB    X                 Failed by verification.

Display the results from the second test.

results(2)

ans = 
  TestResult with properties:

          Name: 'sampleTest/testB'
        Passed: 0
        Failed: 1
    Incomplete: 0
      Duration: 1.9894
       Details: [1×1 struct]

Totals:
   0 Passed, 1 Failed (rerun), 0 Incomplete.
   1.9894 seconds testing time.

Check for diagnostics recorded on test results. If you were using a default runner, there would be a DiagnosticRecord field at this location. But because you are using a nondefault runner, the framework does not create such a field.

records = results(2).Details

records = struct with no fields.

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: runner = testrunner("textoutput",OutputDetail="Verbose",LoggingLevel="Detailed")

OutputDetail — Display level of test output

0 | 1 | 2 | 3 | 4 | matlab.automation.Verbosity enumeration object | text representation of enumeration

Display level of test output, specified as an integer scalar from 0 through 4, a matlab.automation.Verbosity enumeration object, or a text representation of the enumeration.

Numeric Representation	Enumeration Member Name	Verbosity Description
0	None	No information
1	Terse	Minimal information
2	Concise	Moderate amount of information
3	Detailed	Some supplemental information
4	Verbose	Lots of supplemental information

By default, the test runner displays failing and logged events at the matlab.automation.Verbosity.Detailed level (level 3) and test run progress at the matlab.automation.Verbosity.Concise level (level 2).

Example: runner = testrunner("textoutput",OutputDetail="Verbose") creates a test runner that displays test output at the matlab.automation.Verbosity.Verbose level.

LoggingLevel — Verbosity level of logged diagnostics

1 (default) | 0 | 2 | 3 | 4 | matlab.automation.Verbosity enumeration object | text representation of enumeration

Verbosity level of logged diagnostics, specified as an integer scalar from 0 through 4, a matlab.automation.Verbosity enumeration object, or a text representation of the enumeration. The test runner includes diagnostics logged at the specified level and below.

Numeric Representation	Enumeration Member Name	Verbosity Description
0	None	No information
1	Terse	Minimal information
2	Concise	Moderate amount of information
3	Detailed	Some supplemental information
4	Verbose	Lots of supplemental information

By default, the test runner includes diagnostics logged at the matlab.automation.Verbosity.Terse level (level 1). To exclude logged diagnostics, specify LoggingLevel as matlab.automation.Verbosity.None (level 0).

Logged diagnostics are diagnostics that you supply to the unit testing framework with the log (TestCase) and log (Fixture) methods.

Example: runner = testrunner("textoutput",LoggingLevel="Detailed") creates a test runner that includes diagnostics logged at the matlab.automation.Verbosity.Detailed level and below.

Output Arguments

runner — Test runner

matlab.unittest.TestRunner object

Test runner, returned as a matlab.unittest.TestRunner object.

Tips

• You can use the testrunner function instead of the static methods of the matlab.unittest.TestRunner class. This table shows the testrunner function call equivalent to each static method of the matlab.unittest.TestRunner class.

  testrunner Function	matlab.unittest.TestRunner Class
  r = testrunner	r = matlab.unittest.TestRunner.withDefaultPlugins
  r = testrunner("minimal")	r = matlab.unittest.TestRunner.withNoPlugins
  r = testrunner("textoutput")	r = matlab.unittest.TestRunner.withTextOutput

Version History

Introduced in R2021a

R2026b: Rerunning failed tests uses a default test runner if the original runner no longer exists

When you run tests using a nondefault test runner created with the testrunner function and then rerun failed tests using the rerun link in the test summary, the testing framework reruns the failed tests using a default test runner if the original test runner no longer exists. In previous releases, the framework reruns failed tests using the original test runner.

R2024b: Control verbosity of test runner

To control the verbosity of the test runner, use the OutputDetail and LoggingLevel name-value arguments.