runperf
Run set of tests for performance measurement
Description
results = runperf
runs all the tests in your current folder
for performance measurements and returns an array of
matlab.perftest.TimeResult
objects. Each element in
results
corresponds to an element in the test suite.
The performance test framework runs the tests using a variable number of
measurements to reach a sample mean with a 0.05 relative margin of error within a
0.95 confidence level. It runs the tests four times to warm up the code, and then
between 4 and 256 times to collect measurements that meet the statistical
objectives. If the sample mean does not meet the 0.05 relative margin of error
within a 0.95 confidence level after 256 test runs, the performance test framework
stops running the test and displays a warning. In this case, the
matlab.perftest.TimeResult
object contains information for the 4
warm-up runs and 256 measurement runs.
The runperf
function provides a simple way to run a
collection of tests as a performance experiment.
results = runperf(___,
runs a set of tests with additional options specified by one or more name-value
arguments.Name,Value
)
Examples
Run Script as Performance Test
In your current working folder, create a script-based test,
onesTest.m
, that uses three different methods to
initialize a 1000
-by-1500
matrix of
ones.
rows = 1000; cols = 1500; %% Ones Function X = ones(rows,cols); %% Loop Assignment Without Preallocation for r = 1:rows for c = 1:cols X(r,c) = 1; end end %% Loop Assignment With Preallocation X = zeros(rows,cols); for r = 1:rows for c = 1:cols X(r,c) = 1; end end
Run the script as a performance test. Your results might vary.
results = runperf('onesTest');
Running onesTest .......... .......... .......... .......... .......... .......... .......... .......... .......... .... Done onesTest __________
Display the results. The results
variable is a
1
-by-3
TimeResult
array. Each element in the array corresponds
to one of the tests defined in the code section in
onesTest.m
.
results
results = 1×3 TimeResult array with properties: Name Valid Samples TestActivity Totals: 3 Valid, 0 Invalid. 4.4113 seconds testing time.
Display the measurement results for the second test, which loops the assignment without preallocation.
results(2)
ans = TimeResult with properties: Name: 'onesTest/LoopAssignmentWithoutPreallocation' Valid: 1 Samples: [4×7 table] TestActivity: [8×12 table] Totals: 1 Valid, 0 Invalid. 3.1462 seconds testing time.
Display the complete table of test measurements.
results(2).TestActivity
results(2).TestActivity ans = 8×12 table Name Passed Failed Incomplete MeasuredTime Objective Timestamp Host Platform Version TestResult RunIdentifier ___________________________________________ ______ ______ __________ ____________ _________ ____________________ ___________ ________ __________________________________________ ________________________________ ____________________________________ onesTest/LoopAssignmentWithoutPreallocation true false false 0.39586 warmup 24-Jun-2019 16:50:25 MY-HOSTNAME win64 9.7.0.1141441 (R2019b) Prerelease Update 2 [1×1 matlab.unittest.TestResult] ab95d893-a080-4e98-9297-df697882b09d onesTest/LoopAssignmentWithoutPreallocation true false false 0.38351 warmup 24-Jun-2019 16:50:25 MY-HOSTNAME win64 9.7.0.1141441 (R2019b) Prerelease Update 2 [1×1 matlab.unittest.TestResult] ab95d893-a080-4e98-9297-df697882b09d onesTest/LoopAssignmentWithoutPreallocation true false false 0.37995 warmup 24-Jun-2019 16:50:25 MY-HOSTNAME win64 9.7.0.1141441 (R2019b) Prerelease Update 2 [1×1 matlab.unittest.TestResult] ab95d893-a080-4e98-9297-df697882b09d onesTest/LoopAssignmentWithoutPreallocation true false false 0.38603 warmup 24-Jun-2019 16:50:26 MY-HOSTNAME win64 9.7.0.1141441 (R2019b) Prerelease Update 2 [1×1 matlab.unittest.TestResult] ab95d893-a080-4e98-9297-df697882b09d onesTest/LoopAssignmentWithoutPreallocation true false false 0.38388 sample 24-Jun-2019 16:50:26 MY-HOSTNAME win64 9.7.0.1141441 (R2019b) Prerelease Update 2 [1×1 matlab.unittest.TestResult] ab95d893-a080-4e98-9297-df697882b09d onesTest/LoopAssignmentWithoutPreallocation true false false 0.39803 sample 24-Jun-2019 16:50:27 MY-HOSTNAME win64 9.7.0.1141441 (R2019b) Prerelease Update 2 [1×1 matlab.unittest.TestResult] ab95d893-a080-4e98-9297-df697882b09d onesTest/LoopAssignmentWithoutPreallocation true false false 0.39742 sample 24-Jun-2019 16:50:27 MY-HOSTNAME win64 9.7.0.1141441 (R2019b) Prerelease Update 2 [1×1 matlab.unittest.TestResult] ab95d893-a080-4e98-9297-df697882b09d onesTest/LoopAssignmentWithoutPreallocation true false false 0.37702 sample 24-Jun-2019 16:50:28 MY-HOSTNAME win64 9.7.0.1141441 (R2019b) Prerelease Update 2 [1×1 matlab.unittest.TestResult] ab95d893-a080-4e98-9297-df697882b09d
The performance testing framework ran four warm-up runs, followed by four
measurements runs (indicated as sample
in the
Objective
column).
Display the mean measured time for the second test. To exclude data
collected in the warm-up runs, use the values in the
Samples
field.
mean(results(2).Samples.MeasuredTime)
ans = 0.3891
To compare the different initialization methods in the script, display the
mean measured time for all the tests. Concatenate the values from the
Samples
field across the three elements in the
results
array. Then use varfun
to group the table entries by name and compute the mean.
fullTable = vertcat(results.Samples); varfun(@mean,fullTable,'InputVariables','MeasuredTime','GroupingVariables','Name')
ans = 3×3 table Name GroupCount mean_MeasuredTime ___________________________________________ __________ _________________ onesTest/OnesFunction 65 0.0063079 onesTest/LoopAssignmentWithoutPreallocation 4 0.38909 onesTest/LoopAssignmentWithPreallocation 13 0.018792
In the example output, the ones
function was the
fastest way to initialize the matrix to ones. The performance testing
framework made 65
measurement runs for this test. Your
results might vary.
Performance Tests from Select Unit Tests
In your current working folder, create a class-based test, preallocationTest.m
, that compares different methods of preallocation.
classdef preallocationTest < matlab.perftest.TestCase methods(Test) function testOnes(testCase) x = ones(1,1e7); end function testIndexingWithVariable(testCase) id = 1:1e7; x(id) = 1; end function testIndexingOnLHS(testCase) x(1:1e7) = 1; end function testForLoop(testCase) for i=1:1e7 x(i) = 1; end end end end
The measurement boundary for the preallocationTest
class is the test method. The time measurement for each test method includes all the code in method. For information on designating measurement boundaries, see the startMeasuring
and stopMeasuring
methods of matlab.perftest.TestCase
.
Run performance tests for all the elements that contain 'Indexing'
in the name. Your results might vary, and you might see a warning if runperf
doesn't meet statistical objectives.
results = runperf('preallocationTest','Name','*Indexing*')
Running preallocationTest .......... .......... ....... Done preallocationTest __________ results = 1×2 TimeResult array with properties: Name Valid Samples TestActivity Totals: 2 Valid, 0 Invalid. 2.4858 seconds testing time.
Display the mean measured time for each of the tests. Concatenate the values from the Samples
field across the two elements in the results
array. Then use varfun
to group the table entries by name and compute the mean.
fullTable = vertcat(results.Samples); varfun(@mean,fullTable,'InputVariables','MeasuredTime','GroupingVariables','Name')
ans = 2×3 table Name GroupCount mean_MeasuredTime __________________________________________ __________ _________________ preallocationTest/testIndexingWithVariable 6 0.16337 preallocationTest/testIndexingOnLHS 13 0.049936
Input Arguments
tests
— Suite of tests
character vector | cell array of character vectors
Suite of tests specified as a string array, character vector, or cell array of character vectors. Each character vector in the cell array can contain the name of a test file, a test class, a test suite element name, a package containing your test classes, a folder containing your test files, or a project folder containing test files.
Example: runperf('ATestFile.m')
Example: runperf('ATestFile/aTest')
Example: runperf('mypackage.MyTestClass')
Example: runperf(pwd)
Example: runperf({'mypackage.MyTestClass','ATestFile.m',pwd,'mypackage.subpackage'})
Name-Value Arguments
Specify optional pairs of arguments as
Name1=Value1,...,NameN=ValueN
, where Name
is
the argument name and Value
is the corresponding value.
Name-value arguments must appear after other arguments, but the order of the
pairs does not matter.
Before R2021a, use commas to separate each name and value, and enclose
Name
in quotes.
Example: runperf(tests,'Name','productA_*')
runs test elements
with a name that starts with 'productA_'
.
IncludeSubfolders
— Indicator to run tests in subfolders
false
(default) | true
| 0
| 1
Indicator to run tests in subfolders, specified as
false
or true
(0
or 1
). By default the
framework runs tests in the specified folders, but not in their
subfolders.
IncludeSubpackages
— Indicator to run tests in subpackages
false
(default) | true
| 0
| 1
Indicator to run tests in subpackages, specified as
false
or true
(0
or 1
). By default the
framework runs tests in the specified packages, but not in their
subpackages.
IncludeReferencedProjects
— Indicator to include tests from referenced projects
false
(default) | true
| 0
| 1
Indicator to include tests from referenced projects, specified as false
or true
. For more information on referenced projects, see Componentize Large Projects.
InvalidFileFoundAction
— Action to take against invalid test file
"warn"
(default) | "error"
Action to take against an invalid test file in a folder or package
that the function is processing, specified as "warn"
or "error"
.
"warn"
— The function issues a warning for each invalid test file in a folder or package and runs the tests in the valid files."error"
— The function throws an error if it finds an invalid test file in a folder or package.
An invalid test file is a test file that the framework cannot run.
Examples include a test file that contains syntax errors, a
function-based test file that is missing local functions, and a file
with a Test
method that is passed an undefined
parameterization property.
BaseFolder
— Name of base folder
string array | character vector | cell array of character vectors
Name of the base folder that contains the file defining the test class, function, or script,
specified as a string array, character vector, or cell array of character vectors. This
argument filters TestSuite
array elements. For the testing framework to
include a test in the suite, the Test
element must be contained in
one of the base folders specified by BaseFolder
. If none of the
Test
elements matches a base folder, an empty test suite is
returned. Use the wildcard character *
to match any number of
characters. Use the question mark character ?
to match a single
character. For test files defined in packages, the base folder is the parent of the
top-level package folder.
Name
— Name of test suite element
string array | character vector | cell array of character vectors
Name of the test suite element, specified as a string array, character vector, or cell
array of character vectors. This argument filters TestSuite
array
elements. For the testing framework to include a test in the suite, the
Name
property of the Test
element must match
one of the names specified by Name
. If none of the
Test
elements has a matching name, an empty test suite is
returned. Use the wildcard character *
to match any number of
characters. Use the question mark character ?
to match a single
character.
For a given test file, the name of a test suite element uniquely identifies the smallest runnable portion of the test content. It includes the package name, file name (excluding the extension), procedure name, and information about parameterization.
ParameterProperty
— Name of parameterization property
string array | character vector | cell array of character vectors
Name of a test class property that defines a parameter used by the test suite element, specified as a string array, character vector, or cell array of character vectors. This argument filters TestSuite
array elements. For the testing framework to include a test in the suite, the Parameterization
property of the Test
element must contain at least one of the property names specified by ParameterProperty
. If none of the Test
elements has a matching property name, an empty test suite is returned. Use the wildcard character *
to match any number of characters. Use the question mark character ?
to match to a single character.
ParameterName
— Name of parameter
string array | character vector | cell array of character vectors
Name of a parameter used by the test suite element, specified as a string array, character vector, or cell array of character vectors. MATLAB® generates parameter names based on the test class property that defines the parameters:
If the property value is a cell array, MATLAB generates parameter names from the elements of the cell array by taking into account their values, types, and dimensions.
If the property value is a structure, MATLAB generates parameter names from the structure fields.
The ParameterName
argument filters TestSuite
array elements. For the testing framework to include a test in the suite, the Parameterization
property of the Test
element must contain at least one of the parameter names specified by ParameterName
. If none of the Test
elements has a matching parameter name, an empty test suite is returned. Use the wildcard character *
to match any number of characters. Use the question mark character ?
to match a single character.
ProcedureName
— Name of test procedure
string array | character vector | cell array of character vectors
Name of the test procedure, specified as a string array, character vector, or cell
array of character vectors. This argument filters TestSuite
array
elements. For the testing framework to include a test in the suite, the
ProcedureName
property of the Test
element
must match one of the procedure names specified by ProcedureName
. If
none of the Test
elements has a matching procedure name, an empty
test suite is returned. Use the wildcard character *
to match any
number of characters. Use the question mark character ?
to match a
single character.
In a class-based test, the name of a test procedure is the name of a
Test
method that contains the test. In a function-based test, it
is the name of a local function that contains the test. In a script-based test, it is a
name generated from the test section title. Unlike the name of a test suite element, the
name of a test procedure does not include any package name, file name, or information
about parameterization.
Superclass
— Name of class that test class derives from
string array | character vector | cell array of character vectors
Name of the class that the test class derives from, specified as a string array,
character vector, or cell array of character vectors. This argument filters
TestSuite
array elements. For the testing framework to include a test
in the suite, the TestClass
property of the Test
element must point to a test class that derives from one of the classes specified by
Superclass
. If none of the Test
elements
matches a class, an empty test suite is returned.
Tag
— Name of test element tag
string array | character vector | cell array of character vectors
Name of a test tag used by the test suite element, specified as a string array, character
vector, or cell array of character vectors. This argument filters
TestSuite
array elements. For the testing framework to include a test
in the suite, the Tags
property of the Test
element must contain at least one of the tag names specified by Tag
.
If none of the Test
elements has a matching tag name, an empty test
suite is returned. Use the wildcard character *
to match any number
of characters. Use the question mark character ?
to match a single
character.
Tips
To customize the statistical objectives of the performance test, use the
TimeExperiment
class to construct and run the performance test.When you use shared test fixtures in your tests and specify the input to the
runperf
function as a string array or cell array of character vectors, the testing framework sorts the array to reduce shared fixture setup and teardown operations. As a result, the tests might run in an order that is different from the order of elements in the input array. For more information, seesortByFixtures
.When you write class-based tests, you can run your tests as a standalone application (requires MATLAB Compiler™). Compiling performance tests is not currently supported. For more information, see Compile MATLAB Unit Tests.
Alternatives
To create a test suite explicitly, you can use the testsuite
function or the matlab.unittest.TestSuite
methods that
create a suite. Then, you can run your performance test with the run
method of your specified TimeExperiment
.
Version History
Introduced in R2016aR2022b: Specify action to take against invalid test files
To specify whether the framework issues a warning or throws an error when it
encounters an invalid test file in a folder or package, use the
InvalidFileFoundAction
name-value argument.
R2022b: Parameter names generated from cell arrays are more descriptive
When you assign a nonempty cell array to a parameterization property, the testing framework generates parameter names from the elements of the cell array by taking into account their values, types, and dimensions. In previous releases, if the property value is a cell array of character vectors, the framework generates parameter names from the values in the cell array. Otherwise, the framework specifies parameter names as value1
, value2
, …, valueN
.
If your code uses parameter names to create or filter test suites, replace the old parameter
names with the descriptive parameter names. For example, update suite =
testsuite(pwd,"ParameterName","value1")
by replacing value1
with a descriptive parameter name.
R2022a: IncludeSubfolders
treats folders and packages the same way
The IncludeSubfolders
name-value argument treats folders and
packages the same way. For example,
runperf(pwd,IncludeSubfolders=true)
runs all the tests in the
current folder and any of its subfolders, including package folders. In previous
releases, IncludeSubfolders
ignores package folders.
R2021b: runperf
ignores project files that do not define test procedures
The runperf
function ignores any files in a MATLAB project that do not define test procedures. For example, if an
abstract TestCase
class definition file is labeled with the
Test
classification, the function ignores it. In previous
releases, MATLAB produces an error if runperf
is called on a
project that uses the Test
classification for any files other
than concrete test files.
R2021b: Tests in projects cannot run without the Java Virtual Machine (JVM) software
If MATLAB runs without the Java® Virtual Machine (JVM™) software, runperf
cannot run the tests in a
MATLAB project. The reason is that the project cannot be opened without the
JVM software. In previous releases, when MATLAB runs without the JVM software, runperf
creates a suite from the test
files in the project and runs the suite.
R2019a: Run tests in a MATLAB project
When your current folder is a project root folder or when you pass the path to a
project root folder to the runperf
function, the function runs
all test files contained in the specified project that are labeled with the
Test
classification.
R2019a: Run tests from referenced projects
To run the tests from referenced projects, use the
IncludeReferencedProjects
name-value argument.
R2019a: Performance test results are returned as TimeResult
objects
The runperf
function returns a matlab.perftest.TimeResult
array containing the results of the
specified performance tests. In previous releases, the function returns an array of
matlab.unittest.measurement.MeasurementResult
objects.
R2019a: Maximum number of sample measurements has increased
The default maximum number of sample measurements that
runperf
makes when running performance measurements has
increased from 32 to 256.
Ouvrir l'exemple
Vous possédez une version modifiée de cet exemple. Souhaitez-vous ouvrir cet exemple avec vos modifications ?
Commande MATLAB
Vous avez cliqué sur un lien qui correspond à cette commande MATLAB :
Pour exécuter la commande, saisissez-la dans la fenêtre de commande de MATLAB. Les navigateurs web ne supportent pas les commandes MATLAB.
Select a Web Site
Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select: .
You can also select a web site from the following list:
How to Get Best Site Performance
Select the China site (in Chinese or English) for best site performance. Other MathWorks country sites are not optimized for visits from your location.
Americas
- América Latina (Español)
- Canada (English)
- United States (English)
Europe
- Belgium (English)
- Denmark (English)
- Deutschland (Deutsch)
- España (Español)
- Finland (English)
- France (Français)
- Ireland (English)
- Italia (Italiano)
- Luxembourg (English)
- Netherlands (English)
- Norway (English)
- Österreich (Deutsch)
- Portugal (English)
- Sweden (English)
- Switzerland
- United Kingdom (English)