How to verify a function with an input and output throws a warning?

I am trying to verify the following function using unit testing:

function out = system_under_test(lookup)

% Set width for fit dependant on energy in measurement

switch lookup

case {45, 60}

out = 3;

case 154

out = 2;

case 227

out = 1;

otherwise

warning("OTHER:warning:id", "Invalid lookup value (%d) specified. Skipping.", lookup)

end

end

I am currently using the following code to test for warnings:

classdef MyTestCase < matlab.unittest.TestCase

properties (TestParameter)

lookup_ok = {45, 60, 154, 227};

lookup_expected = {3, 3, 2, 1};

lookup_warn = num2cell([1:44 46:59 61:153 155:226 228:230]);

end

methods (Test)

function ShouldWorkTest(me, lookup_ok, lookup_expected)

% Act

actual = system_under_test(lookup_ok);

% Assert

me.verifyEqual(actual, lookup_expected);

end

function ShouldWarnTest(me, lookup_warn)

% Assert

me.verifyWarning(system_under_test(lookup_warn), "OTHER:warning:id");

end

end

end

The ShouldWorkTest works as expected and properly satisfies the verifyEqual assertion.

The ShouldWarn test does not validate the warning is thrown but fails with the following exception although the warning is thrown as can be seen below:

Running MyTestCase

Warning: Invalid lookup value (1) specified. Skipping.

> In system_under_test (line 11)

In MyTestCase/ShouldWarnTest (line 27)

================================================================================

Error occurred in MyTestCase/ShouldWarnTest(lookup_warn=1) and it did not run to completion.

---------

Error ID:

---------

'MATLAB:unassignedOutputs'

--------------

Error Details:

--------------

Output argument "out" (and possibly others) not assigned a value in the execution with "system_under_test" function.

Error in MyTestCase/ShouldWarnTest (line 27)

me.verifyWarning(system_under_test(lookup_warn), "OTHER:warning:id");

================================================================================

I prefer not to return a value when an invalid lookup value is provided, but I do want to assert a warning is thrown. How can I do this?

[Jonathan]

I currently work around by catching and ignoring the 'MATLAB:unassignedOutputs' exception:

function ShouldWarnTest(me, lookup_warn)

% Assert

try

me.verifyWarning(system_under_test(lookup_warn), "OTHER:warning:id");

catch ME

switch ME.identifier

case 'MATLAB:unassignedOutputs'

% Do nothing

otherwise

rethrow(ME)

end

end

end

But I hope there is a more elegant way as this still not validate the warning is thrown, it just ignores the failure on the other exception.

[Stephen23]

"I prefer not to return a value when an invalid lookup value is provided.."

... and yet when you call it, you are demanding one output from the function. You can't have your cake and eat it too.

Why not just throw your own ERROR instead of a WARNING, rather than waiting until nothing gets returned and then throwing a default MATLAB error? All you are doing is relying on MATLAB to throw an error for you.

Or even simpler, just return a default invalid value, e.g. 0 or NaN. Simple to test for, easy to understand.

Answer 1

[Steven Lord]

The problem is indeed on this line and in your system_under_test function. Commenting it out so I can run other code later in this answer.

% me.verifyWarning(system_under_test(lookup_warn), "OTHER:warning:id");

This calls system_under_test with one input argument and one output argument, then passes the output argument from that call as the second input to the verifyWarning function. What does verifyWarning require its second input to be? From that documentation page: "verifyWarning(testCase,actual,identifier) verifies that actual is a function handle that issues the warning specified by identifier." [Emphasis added.]

So does your system_under_test function return a function handle as its output? It does not. It doesn't return any output in the case where the lookup value is not one of the ones in the first three cases, so MATLAB (correctly) throws an error. What you should do is pass a function handle to system_under_test into verifyWarning and have system_under_test return some value in the case where the lookup value was invalid.

me = matlab.unittest.TestCase.forInteractiveUse;
output = me.verifyWarning(@() system_under_test(-1), "OTHER:warning:id");

Verification passed.

Note the addition of the @() in the line above and the line assigning NaN to the output argument in system_under_test. This even allows you to check that it returns the "placeholder".

me.verifyEqual(output, NaN, "system_under_test should have returned NaN for invalid lookup value")

Verification passed.

function out = system_under_test(lookup)
    % Set width for fit dependant on energy in measurement
    switch lookup
        case {45, 60}
            out = 3;
        case 154
            out = 2;
        case 227
            out = 1;
        otherwise
            warning("OTHER:warning:id", "Invalid lookup value (%d) specified. Skipping.", lookup)
            out = NaN; % Need SOME value for the output argument, using NaN as placeholder
    end
end

Alternately you could have changed the warning in system_under_test into an error and used verifyError instead of verifyWarning (with the function handle but without an output argument; verifyError cannot return any output argument) as @Stephen23 suggested.

Answer 2

[Jonathan]

I currently found the following solution that works for me, but I don't like it's elegance:

function ShouldWarnTest(me, lookup_warn)

% Assert

try

_ = system_under_test(lookup_warn);

catch ME

switch ME.identifier

case 'MATLAB:unassignedOutputs'

w = warning('query','last');

me.verifyEqual(w.identifier, 'OTHER:warning:id')

otherwise

rethrow(ME);

end

end

end