How is the execution order defined between multiple tests within matlab.uni​ttest.Test​Case.run?

Okay, I have a sequence of tests which need to be executed within a specified order. Is order guaranteed by position within the file? Or is there some other criteria for running test methods in sequence?

[Steven Lord]

Can you say a little more about what you're testing that requires your tests to be executed in a specific order? It's possible you have written an integration test in several pieces, when ideally you'd want to have those pieces each be independent unit tests and create a new larger test method that holds your entire integration test.

[William Hill]

This is mostly a hypothetical question. While I can certainly place the unit tests in sequence under a larger test method, it would require me to add more complexity to my application. I simply want to know how the sequence is determined, and whether such an addition is truly necessary.

Answer 1

[Andy Campbell]

Hi William,

The order is not guaranteed by design. This is so that the tests can follow the principle of Independent Test. By prinicple, each test should be independently runnable on its own, which allwos for the tests to be easily reproducible when failures are encountered, and any subsetof a test suite can be run. It also facilitates running them in parallel.

If there were a guaranteed order of tests, then tests would begin depending on one another which would result in an inability to run them each independently. Test B would not pass without first running Test A and so forth.

If you'd like a defined order it sounds more like an integration or system test, which should be written in a single test method.

Hope that helps! Andy

[Konstantin]

Hello Andy.

What is about using a test suite? I tried to separate the tests to different test classes, create a test suite and sort the order of the test in the suite. It seems to work. For example

 s = matlab.unittest.TestSuite.fromFolder(pwd)
 s.run
 s = s(end:-1:1)
 s.run

At the second call to the "run" method the tests are executed in reverse order

[Andy Campbell]

Hi Konstantin,

Yes this should work, also by design. In principle the tests should be able to run in any arbitrary order. So for example the following would also work:

 s = matlab.unittest.TestSuite.fromFolder(pwd)
 s.run
 s = s(randperm(numel(s)))
 s.run % run in random order

Note that doing this might be more inefficient because SharedTestFixtures and TestClassSetup/TestClassTeardown code may execute more times than necessary, but each test should still be valid.

I would discourage doing anything in the tests that would result in tests being dependent on the order in which they are run.

[Seungbum Koo]

How about the test setup? Will the class/method setup functions be called in the order they appear?

[Andy Campbell]

No, there is no guarantee of the order between different TestClassSetup/TestMethodSetup/etc methods defined in a class. Obviously you can be assured that all TestClassSetup methods will execute before any TestMethodSetup, and you also have assurance that setup methods defined in a base class execute before derived (and base class teardown methods execute after derived ones). However, at the same level of class hierarchy there is no guarantee of order, and they should all be independent.