Project

General

Profile

Bug #7608

Features supporting regression testing

Added by Tomasz Domin 10 months ago. Updated 7 months ago.

Status:
Feedback
Priority:
Normal
Assignee:
Tomasz Domin
Start date:
Due date:
% Done:

90%

billable:
No
vendor_id:
GCD
case_num:
version:

harness_20230721.diff Magnifier (63 KB) Tomasz Domin, 07/21/2023 06:24 AM

History

#1 Updated by Tomasz Domin 10 months ago

During development of regression testing I've made changes to harness to support process of debugging, tracing and stabilization.

The list of implemented features:
  • Restarting of Driver threads on failure - in case Driver thread fails completely it is restarted, so a total number of threads is kept as defined in master test plan
  • Terminate thread when in backout fails to prevent further failures - so failing backout threads do not accumulate, depends on Restarting of Driver threads on failure
  • Timeout multiplier - allows changing all timeouts in all tests in given tests scenario (-g 2.0)
  • Trace mode - increase logging verbosity to track connection and terminal issues (-d -d)
  • Tests white list - allows selecting tests to be executed out of entire tests scenario (-f test1,test2, in combination with already implemented black lists it can cover #1553
  • IN_PROGRESS test status - solely for reporting purposes
  • Reduce screen poll delay/increase screen poll frequency - the poll timeout is reduced from 500ms to 100ms, so harness do not miss any screen changes anymore
  • Limit maximum number of allowed parallel TestSets (-q 1) - allows executing TestSet in sequence despite master test scenario they are in parallel

Attached please find a patch harness_20230721.diff with all code changes.
Please review.

#2 Updated by Greg Shah 10 months ago

Code Review harness_20230721.diff

The changes are very good. I only have code standards issues to report:cat

1. isIgnored() needs javadoc (BarrierElement, ConcurrentElements

2. JobQueue.isEmpty() needs javadoc.

3. Please un-indent (de-dent? outdent?) the case "g": on line 597 of Harness.java so that it can be more easily read.

4. Driver, Harness, TestSet, ScreenPoller, LogHelper should only have one history number each (though the multiple entries can remain).

5. In TestPlanFactory, the outputDirectory.trim().length()>0 should be written outputDirectory.trim().length() > 0.

#3 Updated by Tomasz Domin 8 months ago

  • Subject changed from Features suppporting regression testing to Features supporting regression testing

Greg Shah wrote:

Code Review harness_20230721.diff

The changes are very good. I only have code standards issues to report:cat

1. isIgnored() needs javadoc (BarrierElement, ConcurrentElements

Done

2. JobQueue.isEmpty() needs javadoc.

Done

3. Please un-indent (de-dent? outdent?) the case "g": on line 597 of Harness.java so that it can be more easily read.

Done

4. Driver, Harness, TestSet, ScreenPoller, LogHelper should only have one history number each (though the multiple entries can remain).

Done

5. In TestPlanFactory, the outputDirectory.trim().length()>0 should be written outputDirectory.trim().length() > 0.

Done

Committed revision 30.

#4 Updated by Greg Shah 8 months ago

Code Review Harness Revision 30

The changes are good.

#5 Updated by Greg Shah 8 months ago

Please update the Documentation to match the changes.

#6 Updated by Greg Shah 8 months ago

I've updated the public release with rev 30.

Is there anything else for this task other than documentation?

#7 Updated by Tomasz Domin 8 months ago

Greg Shah wrote:

I've updated the public release with rev 30.

Is there anything else for this task other than documentation?

No, but the documentation is still pending.

#8 Updated by Roger Borrello 7 months ago

Tomasz... was there a change in revision 30 that would bring on this error:

2023-09-29 15:02:57 INFO    Harness  Enable debug mode 
2023-09-29 15:02:57 INFO    Harness  Start 
[Fatal Error] :6:38: Element type "exclude" must be followed by either attribute specifications, ">" or "/>".
Cannot load file (bad content).
org.xml.sax.SAXParseException: Element type "exclude" must be followed by either attribute specifications, ">" or "/>".
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:338)
        at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121)
        at com.goldencode.harness.XmlHelper.loadDocument(XmlHelper.java:79)
        at com.goldencode.harness.test.TestFactory.read(TestFactory.java:79)
        at com.goldencode.harness.TestPlanFactory.loadDuplicates(TestPlanFactory.java:1079)
        at com.goldencode.harness.TestPlanFactory.processTests(TestPlanFactory.java:840)
        at com.goldencode.harness.TestPlanFactory.processTestSets(TestPlanFactory.java:500)
        at com.goldencode.harness.TestPlanFactory.loadTestSets(TestPlanFactory.java:229)
        at com.goldencode.harness.TestPlanFactory.read(TestPlanFactory.java:78)
        at com.goldencode.harness.Harness.main(Harness.java:750)

For this code:
<?xml version="1.0"?>
<test name="sslsv9-1 " description="CRM activity report" >

    <!-- compare the reports -->
    <text-file-comparison baseline="sl/sslsv9-1" remote="/usr/report_testing/reports/sl/sslsv9-1">
       <exclude type="each-page" row="62" start="11" end="20" ignore-size="true" />
       <exclude type="each-page" row="62" start="24" end="29" ignore-size="true" />
       <exclude type="each-page" row="62" start="33" end="41" ignore-size="true" />
       <exclude type="each-page" row="" start="20" end="34" ignore-size="true" />
    </text-file-comparison>
</test>

#9 Updated by Tomasz Domin 7 months ago

Roger Borrello wrote:

Tomasz... was there a change in revision 30 that would bring on this error:
[...]
For this code:
[...]

There was no changes to script syntax.
There are several tests like:

   <text-file-comparison baseline="gso_67.out" remote="gso_67.out">
      <exclude type="each-page" row="0" start="113" end="132" />
      <exclude type="each-page" row="6" start="10" end="132" ignore-size="true" />
   </text-file-comparison>

EDIT: There are no tests having row=""
EDIT: Isnt remote mandatory ? - missed that

#10 Updated by Roger Borrello 7 months ago

Tomasz Domin wrote:

EDIT: There are no tests having row=""

Hmm... let me check this out. These tests were generated by the customer, and I am not sure if I ever ran these versions.

Also available in: Atom PDF