Main Content

Collecting Requirements-Based Testing Metrics using Continuous Integration

Overview

Requirements-based testing metrics allow you to assess the status and quality of your requirements-based testing (RBT) activities. You can visualize the results by using the Model Testing Dashboard. Metric collection can be incorporated into continuous integration (CI) workflows. Continuous collection of these metrics helps you to monitor the progression and quality of a project. This example uses GitLab to host the project source and Jenkins to build and test the project as well as archive the results.

Requirements

  • Use of MATLAB® projects

  • Use of the Simulink Test Manager test harness

Set Up the Project in Source Control

GitLab Setup

Create a GitLab project for source-controlling your project. For more information, see https://docs.gitlab.com/ee/README.html.

  1. Install the Git Client.

  2. Set up a branching workflow. Using GitLab, from the main branch, create a temporary branch for implementing changes to the model files. Integration engineers can use Jenkins test results to decide whether to merge a temporary branch into the main branch. For more information, see https://git-scm.com/book/en/Git-Branching-Branching-Workflows.

  3. Under Settings > Repository, protect the main branch by enforcing the use of merge requests when developers want to merge their changes into the main branch.

  4. Under Settings, on the Integrations page, add a webhook to the URL of your Jenkins project. This webhook triggers a build job on the Jenkins server.

Add the Project

This example uses the CruiseDashboardExample project. To create a working copy of this file, at the command line, type dashboardCCProjectStart.

dashboardCCProjectStart

Add all of the files in the project along with the files attached to this example using Git to the main branch. These scripts are used to run tests and collect the metrics.

Derived Artifact Filtering

Collecting metrics generates files that you typically do not want checked into source control. Git allows you to ignore files by adding filters to a text file named .gitignore located at the root directory. You can add the sample .gitignore file attached to this example which will filter the files generated by this example that do not need to be added to source control. For more information on .gitignore files, see https://git-scm.com/docs/gitignore.

Set Up the Project in the Continuous Integration Tool

The continuous integration tool automates the building and testing of the project. Many different tools can be used to automatically generate requirements-based testing results by following the same general steps. In this example, use Jenkins as the automation tool. To run the example, you must install the GitLab and MATLAB plugins for Jenkins.

Creating the Project

The CI tool will need integration to the source control repository of the project. This integration allows the CI tool to listen to changes and access the project to build. Jenkins provides a Freestyle project which serves as a generic template for projects which can work with any source control management (SCM). In the Freestyle project, add the source control information to enable the SCM to access the hosted project.

  1. Click New Item, fill in the name, and choose freestyle project

  2. In your Jenkins project, click Configure.

  3. On the General tab, specify a project name.

  4. On the Source Code Management tab, for the Repository URL field, specify the URL of your GitLab repository.

  5. On the Build Triggers tab, select Build when a change is pushed to GitLab.

  6. Configure the Build Environment to use the installed version of MATLAB needed by the project.

Building and Testing

The MATLAB plugin for Jenkins allows integration by specifying MATLAB commands as well as configuring testing, bypassing the need to use the command line. In this example, a single build step will be used to open the project, initialize the metrics infrastructure, run tests, and then collect results.

Click Add build step and choose Run MATLAB Command. Enter the command:

openProject(pwd);collectModelTestingResults();runTests();collectModelTestingResults();

Archiving and Consuming Metric Results

Metrics results can be archived during the build step and then reimported into MATLAB when you want to review them. In this example, the result collection script stores the metric data in the derived directory. Because some of the metrics rely on exported Simulink Test results, include the exported .mldatx files in the archive.

To archive results for later review, configure the CI system to export these files:

  • All files located in <project-root>/derived directory

  • All test results exported to <project-root>/testresults/.mldatx files.

For this example, use the Jenkins provided post-build action to archive artifacts produced during the build.

Click Add post-build action and choose Archive the artifacts. Enter the path:

derived/**,testresults/*.mldatx

to archive all files saved to that directory.

Finally you must click the Save button to save and close the configuration.

Running a Build Job in Jenkins

Jenkins is now configured to execute a new build job each time new changes to the project have been committed to the GitLab repository. You can also manually run a build by clicking on Build Now in the Jenkins project page.

Reviewing the Archived Results in MATLAB

Jenkins will store all the files generated and archived for each successful build and these can be viewed individually or downloaded together in a single zip file. To view the results in MATLAB:

  1. Get the version of the project that was used to generate the results from source control.

  2. Get the archived metric results from the archived location.

  3. Download and copy/extract the derived directory and all files into the root directory of the project.

  4. Download the archived, exported, Simulink Test results files and copy/extract these files as well.

  5. Open the project in MATLAB and open the Model Testing Dashboard. The dashboard displays the results generated from the CI build.

Alternative CI Integration using Command Line

If you use a different automation tool, you can alternatively use the command line for testing integration. Run the tests and collect metrics by running the appropriate commands through the command line interface along using the -batch flag.

For example, when you use this command, MATLAB opens the project, initializes the model testing results, runs all of the tests, collects the model metrics, and then shuts down.

matlab -c %LICENSE_PATH% -nosplash -logfile output.log -batch
"openProject(pwd);collectModelTestingResults();runTests();collectModelTestingResults(); exit;"