Bug #9338: Import JUnit test report back into Harness - Testing - Golden Code Redmine

Bug #9338

Import JUnit test report back into Harness

Added by Tomasz Domin over 1 year ago. Updated about 1 year ago.

Status:

Closed

Priority:

Normal

Assignee:

Tomasz Domin

Target version:

Start date:

Due date:

% Done:

100%

billable:

vendor_id:

GCD

case_num:

version_reported:

version_resolved:

reviewer:

production:

env_name:

topics:

Related issues

History

#1 Updated by Tomasz Domin over 1 year ago

Related to Support #9270: design and implement an approach for executing multi-session 4GL testcases added

#2 Updated by Tomasz Domin over 1 year ago

Subject changed from Import JUnit report back into Harness to Import JUnit test report back into Harness

Harness has a new feature that allows running external processes that interact with Harness with remote API.
One example of such process would be starting external JUnit or ABLUnit testing tool. The tool produces a test report (in a JUnit Legacy format see: https://junit.org/junit5/docs/snapshot/user-guide/#junit-platform-reporting-legacy-xml) that needs to loaded back into Harness to allow unified test reporting.
The legacy JUnit format consists of elements of three levels: testsuites/testsuite/testcase.
Harness allows for four levels: TestPlan/TestSuite/Test/TestStep.
The difference is lack of TestStep level in Junit which should be a report of every assertion in specific testcase.

Due to that I will do following mapping:

   JUnit   ->    Harness
testsuites -> TestPlan
testsuite  -> TestSuite
testcase   -> Test which consists single step JUnitTestcase

Note that every Harness Test report would consist of a single step indicating it was executed as JUnit test - nothing fancy, just a placeholder for information provided by JUnit which Harness has no need to process.

In addition following needs to be possible:
- assign external test results to specific Harness TestPlan, so it will be possible to put all result into single TestPlan

There will be an additional Harness Testable created that reads JUnit report from file into Harness reporting structures.

#4 Updated by Vladimir Tsichevski over 1 year ago

I've implemented a tool, which is able to parse the JUnit5 legacy test output format and compare two reports. See #9134.

#5 Updated by Tomasz Domin over 1 year ago

Vladimir Tsichevski wrote:

I've implemented a tool, which is able to parse the JUnit5 legacy test output format and compare two reports. See #9134.

Thank you. Harness has its own XML file processing which I used to import JUnit5 legacy test files.
I guess the tool can be used later to compare baselines - once Harness allows for exporting JUnit5 legacy test files.

#6 Updated by Tomasz Domin over 1 year ago

I've implemented support for JUNIT-IMPORT tests.
It allows importing JUnit Legacy format test report from local filesystem. By default each testsuite is mapped into a new Harness TestSuite of a name derived from JUnit testsuite, but as an option it is possible to import all tests into a designated Harness TestSuite.
Imported report it merged into Harness TestPlan just before report generation - so it does not affect testing itself.

Example usage in Harness scenario - loads junit.xml file and puts all testcases as tests into start_tests_main TestSet:

    <junit-import filename="junit.xml" testset="start_tests_main"/>

#7 Updated by Tomasz Domin over 1 year ago

Assignee set to Tomasz Domin
% Done changed from 0 to 90

#8 Updated by Tomasz Domin over 1 year ago

In order to import test results from arbitrary place in a filesystem I've added support for filename variable substitutions
An example:

    <junit-import filename="%s/%s" testset="start_tests_main">
        <substitution variable="testDirectory" />
        <substitution variable="resultsXml" />
    </junit-import>

That will import junit result file from a paths specified by testDirectory and resultsXml variables.