TAP results do not show up during execution time, if a TestClassSetup is present
1 vue (au cours des 30 derniers jours)
Afficher commentaires plus anciens
Christian Heigele
le 7 Août 2018
Réponse apportée : Andy Campbell
le 9 Août 2018
Hi there
I have a problem. We use the Matlab testing framework to analyze our codebase. To track the results in our CI system TeamCity we use the TAP-format. Here we have the following problem:
If a test includes a TestClassSetup section, the TAP results show up only at the end, and not during the exection. This results in a few issues for us: - Timestamps might not be correct - If informative output is given per test-case, it is not shown together with the assertion statment.
We use the following (simplified) snippet to identify out TestSuite and execute it:
testSuite = matlab.unittest.TestSuite.fromFolder('.');
runner = matlab.unittest.TestRunner.withNoPlugins();
runner.addPlugin(matlab.unittest.plugins.TAPPlugin.producingOriginalFormat());
results = runner.run(testSuite);
With the following two classes the issue is reproducible (the content is of course made up & meaningless...):
classdef SomeTest < matlab.unittest.TestCase
properties (TestParameter)
param = {1, 2};
param2 = {1, 2};
end
methods (TestClassSetup)
function someSetup(testCase)
pause(0.1);
end
end
methods (Test)
function testMethod(self, param, param2)
fprintf('I''m here, with the params: %f/%f\n', param, param2);
pause(0.1);
self.assertGreaterThan(param, param2);
end
end
end
classdef SomeOtherTest < matlab.unittest.TestCase
properties (TestParameter)
param = {1, 2};
param2 = {1, 2};
end
methods (Test)
function testMethod(self, param, param2)
fprintf('I''m here, with the params: %f/%f\n', param, param2);
pause(0.1);
self.assertGreaterThan(param, param2);
end
end
end
If you copy all three files into one folder, and execute the runner, you'll see the output (assertions are simplified):
1..8
I'm here, with the params: 1.000000/1.000000
not ok 1 - SomeOtherTest/testMethod(param=1,param2=1)
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=1,param2=1) and it did not run to completion.
# ================================================================================
#
I'm here, with the params: 1.000000/2.000000
not ok 2 - SomeOtherTest/testMethod(param=1,param2=2)
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=1,param2=2) and it did not run to completion.
# ================================================================================
#
I'm here, with the params: 2.000000/1.000000
ok 3 - SomeOtherTest/testMethod(param=2,param2=1)
I'm here, with the params: 2.000000/2.000000
not ok 4 - SomeOtherTest/testMethod(param=2,param2=2)
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=2,param2=2) and it did not run to completion.
# ================================================================================
#
I'm here, with the params: 1.000000/1.000000
I'm here, with the params: 1.000000/2.000000
I'm here, with the params: 2.000000/1.000000
I'm here, with the params: 2.000000/2.000000
not ok 5 - SomeTest/testMethod(param=1,param2=1)
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=1,param2=1) and it did not run to completion.
# ================================================================================
#
not ok 6 - SomeTest/testMethod(param=1,param2=2)
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=1,param2=2) and it did not run to completion.
# ================================================================================
#
ok 7 - SomeTest/testMethod(param=2,param2=1)
not ok 8 - SomeTest/testMethod(param=2,param2=2)
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=2,param2=2) and it did not run to completion.
# ================================================================================
What I would expect is that also in the second case the Assertion statements (and the ok / not ok TAP flags) are aligned with the fprintf-statements.
Has anyone an idea?
0 commentaires
Réponse acceptée
Andy Campbell
le 9 Août 2018
This was also asked and answered on stack overflow:
https://stackoverflow.com/questions/51760637/tap-results-do-not-show-up-during-execution-time-if-a-testclasssetup-is-present/51775630#51775630
0 commentaires
Plus de réponses (0)
Voir également
Catégories
En savoir plus sur Write Unit Tests dans Help Center et File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!