runperf

Run set of tests for performance measurement

Syntax

results = runperf

results = runperf(tests)

results = runperf(___,Name,Value)

Description

results = runperf runs all the tests in your current folder for performance measurements and returns an array of matlab.perftest.TimeResult objects. Each element in results corresponds to an element in the test suite.

The performance test framework runs the tests using a variable number of measurements to reach a sample mean with a 0.05 relative margin of error within a 0.95 confidence level. It runs the tests four times to warm up the code, and then between 4 and 256 times to collect measurements that meet the statistical objectives. If the sample mean does not meet the 0.05 relative margin of error within a 0.95 confidence level after 256 test runs, the performance test framework stops running the test and displays a warning. In this case, the matlab.perftest.TimeResult object contains information for the 4 warm-up runs and 256 measurement runs.

The runperf function provides a simple way to run a collection of tests as a performance experiment.

results = runperf(tests) runs a specified set of tests.

results = runperf(___,Name,Value) runs a set of tests with additional options specified by one or more name-value arguments.

Examples

Run Script as Performance Test

In your current working folder, create a script-based test, onesTest.m, that uses three different methods to initialize a 1000-by-1500 matrix of ones.

rows = 1000;
cols = 1500;

%% Ones Function
X = ones(rows,cols);

%% Loop Assignment Without Preallocation
for r = 1:rows
    for c = 1:cols
        X(r,c) = 1;
    end
end

%% Loop Assignment With Preallocation
X = zeros(rows,cols);
for r = 1:rows
    for c = 1:cols
        X(r,c) = 1;
    end
end

Run the script as a performance test. Your results might vary.

results = runperf('onesTest');

Running onesTest
.......... .......... .......... .......... ..........
.......... .......... .......... .......... ....
Done onesTest
__________

Display the results. The results variable is a 1-by-3 TimeResult array. Each element in the array corresponds to one of the tests defined in the code section in onesTest.m.

results

results = 

  1×3 TimeResult array with properties:

    Name
    Valid
    Samples
    TestActivity

Totals:
   3 Valid, 0 Invalid.
   4.4113 seconds testing time.

Display the measurement results for the second test, which loops the assignment without preallocation.

results(2)

ans = 

  TimeResult with properties:

            Name: 'onesTest/LoopAssignmentWithoutPreallocation'
           Valid: 1
         Samples: [4×7 table]
    TestActivity: [8×12 table]

Totals:
   1 Valid, 0 Invalid.
   3.1462 seconds testing time.

Display the complete table of test measurements.

results(2).TestActivity

results(2).TestActivity

ans =

  8×12 table

                       Name                        Passed    Failed    Incomplete    MeasuredTime    Objective         Timestamp             Host        Platform                     Version                                 TestResult                          RunIdentifier            
    ___________________________________________    ______    ______    __________    ____________    _________    ____________________    ___________    ________    __________________________________________    ________________________________    ____________________________________

    onesTest/LoopAssignmentWithoutPreallocation    true      false       false         0.39586        warmup      24-Jun-2019 16:50:25    MY-HOSTNAME     win64      9.7.0.1141441 (R2019b) Prerelease Update 2    [1×1 matlab.unittest.TestResult]    ab95d893-a080-4e98-9297-df697882b09d
    onesTest/LoopAssignmentWithoutPreallocation    true      false       false         0.38351        warmup      24-Jun-2019 16:50:25    MY-HOSTNAME     win64      9.7.0.1141441 (R2019b) Prerelease Update 2    [1×1 matlab.unittest.TestResult]    ab95d893-a080-4e98-9297-df697882b09d
    onesTest/LoopAssignmentWithoutPreallocation    true      false       false         0.37995        warmup      24-Jun-2019 16:50:25    MY-HOSTNAME     win64      9.7.0.1141441 (R2019b) Prerelease Update 2    [1×1 matlab.unittest.TestResult]    ab95d893-a080-4e98-9297-df697882b09d
    onesTest/LoopAssignmentWithoutPreallocation    true      false       false         0.38603        warmup      24-Jun-2019 16:50:26    MY-HOSTNAME     win64      9.7.0.1141441 (R2019b) Prerelease Update 2    [1×1 matlab.unittest.TestResult]    ab95d893-a080-4e98-9297-df697882b09d
    onesTest/LoopAssignmentWithoutPreallocation    true      false       false         0.38388        sample      24-Jun-2019 16:50:26    MY-HOSTNAME     win64      9.7.0.1141441 (R2019b) Prerelease Update 2    [1×1 matlab.unittest.TestResult]    ab95d893-a080-4e98-9297-df697882b09d
    onesTest/LoopAssignmentWithoutPreallocation    true      false       false         0.39803        sample      24-Jun-2019 16:50:27    MY-HOSTNAME     win64      9.7.0.1141441 (R2019b) Prerelease Update 2    [1×1 matlab.unittest.TestResult]    ab95d893-a080-4e98-9297-df697882b09d
    onesTest/LoopAssignmentWithoutPreallocation    true      false       false         0.39742        sample      24-Jun-2019 16:50:27    MY-HOSTNAME     win64      9.7.0.1141441 (R2019b) Prerelease Update 2    [1×1 matlab.unittest.TestResult]    ab95d893-a080-4e98-9297-df697882b09d
    onesTest/LoopAssignmentWithoutPreallocation    true      false       false         0.37702        sample      24-Jun-2019 16:50:28    MY-HOSTNAME     win64      9.7.0.1141441 (R2019b) Prerelease Update 2    [1×1 matlab.unittest.TestResult]    ab95d893-a080-4e98-9297-df697882b09d

The performance testing framework ran four warm-up runs, followed by four measurements runs (indicated as sample in the Objective column).

Display the mean measured time for the second test. To exclude data collected in the warm-up runs, use the values in the Samples field.

mean(results(2).Samples.MeasuredTime)

ans =

    0.3891

To compare the different initialization methods in the script, display the mean measured time for all the tests. Concatenate the values from the Samples field across the three elements in the results array. Then use varfun to group the table entries by name and compute the mean.

fullTable = vertcat(results.Samples);
varfun(@mean,fullTable,'InputVariables','MeasuredTime','GroupingVariables','Name')

ans =

  3×3 table

                       Name                        GroupCount    mean_MeasuredTime
    ___________________________________________    __________    _________________

    onesTest/OnesFunction                              65            0.0063079    
    onesTest/LoopAssignmentWithoutPreallocation         4              0.38909    
    onesTest/LoopAssignmentWithPreallocation           13             0.018792    

In the example output, the ones function was the fastest way to initialize the matrix to ones. The performance testing framework made 65 measurement runs for this test. Your results might vary.

Performance Tests from Select Unit Tests

In your current working folder, create a class-based test, preallocationTest.m, that compares different methods of preallocation.

classdef preallocationTest < matlab.perftest.TestCase
    methods(Test)
        function testOnes(testCase)
            x = ones(1,1e7);
        end
        
        function testIndexingWithVariable(testCase)
            id = 1:1e7;
            x(id) = 1;
        end
        
        function testIndexingOnLHS(testCase)
            x(1:1e7) = 1;
        end
        
        function testForLoop(testCase)
            for i=1:1e7
                x(i) = 1;
            end
        end
        
    end
end

The measurement boundary for the preallocationTest class is the test method. The time measurement for each test method includes all the code in method. For information on designating measurement boundaries, see the startMeasuring and stopMeasuring methods of matlab.perftest.TestCase.

Run performance tests for all the elements that contain 'Indexing' in the name. Your results might vary, and you might see a warning if runperf doesn't meet statistical objectives.

results = runperf('preallocationTest','Name','*Indexing*')

Running preallocationTest
.......... .......... .......
Done preallocationTest
__________


results = 

  1×2 TimeResult array with properties:

    Name
    Valid
    Samples
    TestActivity

Totals:
   2 Valid, 0 Invalid.
   2.4858 seconds testing time.

Display the mean measured time for each of the tests. Concatenate the values from the Samples field across the two elements in the results array. Then use varfun to group the table entries by name and compute the mean.

fullTable = vertcat(results.Samples);
varfun(@mean,fullTable,'InputVariables','MeasuredTime','GroupingVariables','Name')

ans =

  2×3 table

                       Name                       GroupCount    mean_MeasuredTime
    __________________________________________    __________    _________________

    preallocationTest/testIndexingWithVariable         6             0.16337     
    preallocationTest/testIndexingOnLHS               13            0.049936     

Input Arguments

tests — Suite of tests

character vector | cell array of character vectors

Suite of tests specified as a string array, character vector, or cell array of character vectors. Each character vector in the cell array can contain the name of a test file, a test class, a test suite element name, a package containing your test classes, a folder containing your test files, or a project folder containing test files.

Example: runperf('ATestFile.m')

Example: runperf('ATestFile/aTest')

Example: runperf('mypackage.MyTestClass')

Example: runperf(pwd)

Example: runperf({'mypackage.MyTestClass','ATestFile.m',pwd,'mypackage.subpackage'})

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Before R2021a, use commas to separate each name and value, and enclose Name in quotes.

Example: runperf(tests,'Name','productA_*') runs test elements with a name that starts with 'productA_'.

Test Identification

IncludeSubfolders — Indicator to run tests in subfolders

false (default) | true | 0 | 1

Indicator to run tests in subfolders, specified as false or true (0 or 1). By default the framework runs tests in the specified folders, but not in their subfolders.

IncludeSubpackages — Indicator to run tests in subpackages

false (default) | true | 0 | 1

Indicator to run tests in subpackages, specified as false or true (0 or 1). By default the framework runs tests in the specified packages, but not in their subpackages.

IncludeReferencedProjects — Indicator to include tests from referenced projects

false (default) | true | 0 | 1

Indicator to include tests from referenced projects, specified as false or true. For more information on referenced projects, see Componentize Large Projects.

InvalidFileFoundAction — Action to take against invalid test file

"warn" (default) | "error"

Action to take against an invalid test file in a folder or package that the function is processing, specified as "warn" or "error".

• "warn" — The function issues a warning for each invalid test file in a folder or package and runs the tests in the valid files.

• "error" — The function throws an error if it finds an invalid test file in a folder or package.

An invalid test file is a test file that the framework cannot run. Examples include a test file that contains syntax errors, a function-based test file that is missing local functions, and a file with a Test method that is passed an undefined parameterization property.

Test Filtering

BaseFolder — Name of base folder

string array | character vector | cell array of character vectors

Name of the base folder that contains the file defining the test class, function, or script, specified as a string array, character vector, or cell array of character vectors. This argument filters TestSuite array elements. For the testing framework to include a test in the suite, the Test element must be contained in one of the base folders specified by BaseFolder. If none of the Test elements matches a base folder, an empty test suite is returned. Use the wildcard character * to match any number of characters. Use the question mark character ? to match a single character. For test files defined in packages, the base folder is the parent of the top-level package folder.

Name — Name of test suite element

string array | character vector | cell array of character vectors

Name of the test suite element, specified as a string array, character vector, or cell array of character vectors. This argument filters TestSuite array elements. For the testing framework to include a test in the suite, the Name property of the Test element must match one of the names specified by Name. If none of the Test elements has a matching name, an empty test suite is returned. Use the wildcard character * to match any number of characters. Use the question mark character ? to match a single character.

For a given test file, the name of a test suite element uniquely identifies the smallest runnable portion of the test content. It includes the package name, file name (excluding the extension), procedure name, and information about parameterization.

ParameterProperty — Name of parameterization property

string array | character vector | cell array of character vectors

Name of a test class property that defines a parameter used by the test suite element, specified as a string array, character vector, or cell array of character vectors. This argument filters TestSuite array elements. For the testing framework to include a test in the suite, the Parameterization property of the Test element must contain at least one of the property names specified by ParameterProperty. If none of the Test elements has a matching property name, an empty test suite is returned. Use the wildcard character * to match any number of characters. Use the question mark character ? to match to a single character.

ParameterName — Name of parameter

string array | character vector | cell array of character vectors

Name of a parameter used by the test suite element, specified as a string array, character vector, or cell array of character vectors. MATLAB^® generates parameter names based on the test class property that defines the parameters:

• If the property value is a cell array, MATLAB generates parameter names from the elements of the cell array by taking into account their values, types, and dimensions.

• If the property value is a structure, MATLAB generates parameter names from the structure fields.

The ParameterName argument filters TestSuite array elements. For the testing framework to include a test in the suite, the Parameterization property of the Test element must contain at least one of the parameter names specified by ParameterName. If none of the Test elements has a matching parameter name, an empty test suite is returned. Use the wildcard character * to match any number of characters. Use the question mark character ? to match a single character.

ProcedureName — Name of test procedure

string array | character vector | cell array of character vectors

Name of the test procedure, specified as a string array, character vector, or cell array of character vectors. This argument filters TestSuite array elements. For the testing framework to include a test in the suite, the ProcedureName property of the Test element must match one of the procedure names specified by ProcedureName. If none of the Test elements has a matching procedure name, an empty test suite is returned. Use the wildcard character * to match any number of characters. Use the question mark character ? to match a single character.

In a class-based test, the name of a test procedure is the name of a Test method that contains the test. In a function-based test, it is the name of a local function that contains the test. In a script-based test, it is a name generated from the test section title. Unlike the name of a test suite element, the name of a test procedure does not include any package name, file name, or information about parameterization.

Superclass — Name of class that test class derives from

string array | character vector | cell array of character vectors

Name of the class that the test class derives from, specified as a string array, character vector, or cell array of character vectors. This argument filters TestSuite array elements. For the testing framework to include a test in the suite, the TestClass property of the Test element must point to a test class that derives from one of the classes specified by Superclass. If none of the Test elements matches a class, an empty test suite is returned.

Tag — Name of test element tag

string array | character vector | cell array of character vectors

Name of a test tag used by the test suite element, specified as a string array, character vector, or cell array of character vectors. This argument filters TestSuite array elements. For the testing framework to include a test in the suite, the Tags property of the Test element must contain at least one of the tag names specified by Tag. If none of the Test elements has a matching tag name, an empty test suite is returned. Use the wildcard character * to match any number of characters. Use the question mark character ? to match a single character.

Tips

• To customize the statistical objectives of the performance test, use the TimeExperiment class to construct and run the performance test.

• When you use shared test fixtures in your tests and specify the input to the runperf function as a string array or cell array of character vectors, the testing framework sorts the array to reduce shared fixture setup and teardown operations. As a result, the tests might run in an order that is different from the order of elements in the input array. For more information, see sortByFixtures.

• When you write class-based tests, you can run your tests as a standalone application (requires MATLAB Compiler™). Compiling performance tests is not currently supported. For more information, see Compile MATLAB Unit Tests.

Alternatives

To create a test suite explicitly, you can use the testsuite function or the matlab.unittest.TestSuite methods that create a suite. Then, you can run your performance test with the run method of your specified TimeExperiment.

Version History

Introduced in R2016a

R2022b: Specify action to take against invalid test files

To specify whether the framework issues a warning or throws an error when it encounters an invalid test file in a folder or package, use the InvalidFileFoundAction name-value argument.

R2022b: Parameter names generated from cell arrays are more descriptive

When you assign a nonempty cell array to a parameterization property, the testing framework generates parameter names from the elements of the cell array by taking into account their values, types, and dimensions. In previous releases, if the property value is a cell array of character vectors, the framework generates parameter names from the values in the cell array. Otherwise, the framework specifies parameter names as value1, value2, …, valueN.

If your code uses parameter names to create or filter test suites, replace the old parameter names with the descriptive parameter names. For example, update suite = testsuite(pwd,"ParameterName","value1") by replacing value1 with a descriptive parameter name.

R2022a: IncludeSubfolders treats folders and packages the same way

The IncludeSubfolders name-value argument treats folders and packages the same way. For example, runperf(pwd,IncludeSubfolders=true) runs all the tests in the current folder and any of its subfolders, including package folders. In previous releases, IncludeSubfolders ignores package folders.

R2021b: runperf ignores project files that do not define test procedures

The runperf function ignores any files in a MATLAB project that do not define test procedures. For example, if an abstract TestCase class definition file is labeled with the Test classification, the function ignores it. In previous releases, MATLAB produces an error if runperf is called on a project that uses the Test classification for any files other than concrete test files.

R2021b: Tests in projects cannot run without the Java Virtual Machine (JVM) software

If MATLAB runs without the Java^® Virtual Machine (JVM™) software, runperf cannot run the tests in a MATLAB project. The reason is that the project cannot be opened without the JVM software. In previous releases, when MATLAB runs without the JVM software, runperf creates a suite from the test files in the project and runs the suite.

R2019a: Run tests in a MATLAB project

When your current folder is a project root folder or when you pass the path to a project root folder to the runperf function, the function runs all test files contained in the specified project that are labeled with the Test classification.

R2019a: Run tests from referenced projects

To run the tests from referenced projects, use the IncludeReferencedProjects name-value argument.

R2019a: Performance test results are returned as TimeResult objects

The runperf function returns a matlab.perftest.TimeResult array containing the results of the specified performance tests. In previous releases, the function returns an array of matlab.unittest.measurement.MeasurementResult objects.

R2019a: Maximum number of sample measurements has increased

The default maximum number of sample measurements that runperf makes when running performance measurements has increased from 32 to 256.