Process Test Results with Custom Scripts

Testing your model often requires assessing conditions that ensure a test is valid, in addition to verifying model behavior. MATLAB^® Unit Test provides a framework for such assessments. In Simulink^® Test™, you can use the test case custom criteria to author specific assessments, and include MATLAB Unit Test qualifications in your script.

Custom criteria apply as post-simulation criteria to the simulation output. See Simulink.SimulationOutput. If you require run-time verifications, use a verify() statement in a Test Assessment or Test Sequence block. See Verify Model Simulation by Using When Decomposition.

MATLAB Testing Framework

A custom criteria script is a method of test, which is a matlab.unittest test case object. To enable the function, in the test case Custom Criteria section of the Test Manager, select function customCriteria(test). Inside the function, enter the custom criteria script in the embedded MATLAB editor.

The embedded MATLAB editor lists properties of test. Create test assessments using MATLAB Unit Test qualifications. Custom criteria supports verification and assertion type qualifications. See Table of Verifications, Assertions, and Other Qualifications. Verifications and assertions operate differently when custom criteria are evaluated:

• Verifications – Other assessments are evaluated when verifications fail. Diagnostics appear in the results. Use verifications for general assessments, such as checking simulation against expected outputs.

  Example: test.verifyEqual(lastValue,0)

• Assertions – The custom criteria script stops evaluating when an assertion fails. Diagnostics appear in the results. Use assertions for conditions that render the criteria invalid.

  Example: test.assertEqual(lastValue,0).

Define a Custom Criteria Script

This example shows how to create a custom criteria script for an autopilot test case.

1. Open the Test File

sltest.testmanager.load('AutopilotTestFile.mldatx');
sltest.testmanager.view

2. In the Test Browser, select AutopilotTestFile > Basic Design Test Cases > Requirement 1.3 Test.

3. In the test case, expand the Custom Criteria section.

4. Enable the custom criteria script by selecting function customCriteria(test).

5. In the embedded MATLAB editor, add the following script below the customCriteria.out = true line. This script gets and verifies the final values of the signals Phi and APEng. One of the test requirements is Signals_Req1_3 and sltest_simout is the simulation output.

% Get the last values

lastPhi = test.sltest_simout.get('Signals_Req1_3').get('Phi').Values.Data(end);

lastAPEng = test.sltest_simout.get('Signals_Req1_3').get('APEng').Values.Data(end);

% Verify the last values equal 0

test.verifyEqual(lastPhi,0,['Final Phi value: ',num2str(lastPhi),'.']);

test.verifyEqual(lastAPEng,false,['Final APEng value: ',num2str(lastAPEng),'.']);

6. Run the test case.

7. In the Results and Artifacts pane, expand the Custom Criteria Result. Both criteria pass.

8. Clear and close the Test Manager.

sltest.testmanager.clear
sltest.testmanager.clearResults
sltest.testmanager.close

Reuse Custom Criteria and Debug Using Breakpoints

This example shows how to author custom criteria in a standalone function and add a breakpoint to that function. The custom criteria function sltestCheckFinalRollRefValues.m is provided as the starting point for this example.

Using a standalone function lets you:

• Reuse the custom criteria in multiple test cases.

• Set breakpoints in the criteria script for debugging.

• Investigate the simulation output using the command line.

1. Open the function file in the MATLAB Live Editor, and open the test file in the Test Manager

open('sltestCheckFinalRollRefValues.m')

sltest.testmanager.load('AutopilotTestFile.mldatx');
sltest.testmanager.view

2. In the Test Browser pane of the Test Manager, select AutopilotTestFile > Basic Design Test Cases > Requirement 1.3 Test.

3. Expand the Custom Criteria section, and enable the custom criteria script by selecting function customCriteria(test).Then, in the embedded MATLAB editor, replace the content with the function call to the custom criteria:

sltestCheckFinalRollRefValues(test)

4. In sltestCheckFinalRollRefValues.m file in the MATLAB Live Editor, set a breakpoint by clicking on the 8 of line number 8.

5. In the Test Manager, run the test case. When the breakpoint is encountered, the test case pauses running, the command window displays the line where the breakpoint occurred, and the command line prompt changes to K>>, which indicates debugging mode.

6. Enter test at the command prompt to display the properties of the STMCustomCriteria object. The properties contain characteristics and simulation data output of the test case.

7. The property sltest_simout contains the simulation data. To view the data PhiRef, at the command line enter

test.sltest_simout.get('Signals_Req1_3').get('PhiRef')

8. In the sltestCheckFinalRollRefValues.m file in the MATLAB Live Editor, click Continue to continue running the custom criteria script.

9. In the Results and Artifacts pane, expand the Custom Criteria Result. Both criteria pass.

To reuse the script in another test case, call the same function from the Custom Criteria of that test case.

Custom Criteria Programmatic Interface Example

This example shows how to set and get custom criteria using the programmatic interface.

Before running this example, temporarily disable warnings that result from verification failures.

warning off Stateflow:Runtime:TestVerificationFailed;
warning off Stateflow:cdr:VerifyDangerousComparison;

Load a Test File and Get Test Case Object

tf = sltest.testmanager.load('AutopilotTestFile.mldatx');

ts = getTestSuiteByName(tf,'Basic Design Test Cases');

tc = getTestCaseByName(ts,'Requirement 1.3 Test');

Create the Custom Criteria Object and Set Criteria

Create the custom criteria object.

tcCriteria = getCustomCriteria(tc)

tcCriteria = 
  CustomCriteria with properties:

     Enabled: 0
    Callback: '% Return value: customCriteria...'

Create the custom criteria expression. This script gets the last value of the signal Phi and verifies that it equals 0.

criteria = ...
    sprintf(['lastPhi = test.sltest_simout.get(''Signals_Req1_3'')',...
	'.get(''Phi'').Values.Data(end);\n',...
	'test.verifyEqual(lastPhi,0,[''Final: '',num2str(lastPhi),''.'']);'])

criteria = 
    'lastPhi = test.sltest_simout.get('Signals_Req1_3').get('Phi').Values.Data(end);
     test.verifyEqual(lastPhi,0,['Final: ',num2str(lastPhi),'.']);'

Set and enable the criteria.

tcCriteria.Callback = criteria;
tcCriteria.Enabled = true;

Run the Test Case and Get the Results

Run the test case.

tcResultSet = run(tc);

Get the test case results.

tcResult = getTestCaseResults(tcResultSet);

Get the custom criteria result.

ccResult = getCustomCriteriaResult(tcResult)

ccResult = 
  CustomCriteriaResult with properties:

             Outcome: Failed
    DiagnosticRecord: [1x1 sltest.testmanager.DiagnosticRecord]

Restore warnings from verification failures.

warning on Stateflow:Runtime:TestVerificationFailed;
warning on Stateflow:cdr:VerifyDangerousComparison;

sltest.testmanager.clearResults
sltest.testmanager.clear
sltest.testmanager.close

Related Topics

• Test Models Using MATLAB Unit Test
• Create, Store, and Open MATLAB Figures