Project

General

Profile

Support #6862

build conversion regression testing tooling

Added by Greg Shah almost 4 years ago. Updated 8 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
Due date:
% Done:

0%

billable:
No
vendor_id:
GCD
case_num:
version_reported:
version_resolved:
reviewer:
production:
No
env_name:
topics:

History

#2 Updated by Roger Borrello about 3 years ago

Some apps build the archive.incremental target of a project. This could also be extended to have a archive.conv-regression target, which would collect all the things pertinent artifacts. The zip filename itself contains the branch/revision used to perform the conversion (or perhaps the p2j.jar used to convert could be stored in it, or version.properties if it is setup), so that you would know what version of FWD was used. FWD could contain the conv-regression script to compare:
  1. the artifacts in the project it is linked into with path to the baseline conversion zip file, or
  2. 2 zips containing the artifacts to compare

I'm not a fan of how we hook into zip files being unzipped prior to conversion. I would think that a branch should just have the revisions checked into it.

#3 Updated by Greg Shah over 1 year ago

  • Project changed from Conversion Tools to Testing

#4 Updated by Greg Shah over 1 year ago

Tomasz: Can you please summarize how the ChUI Regression testing handles this today? The older version of it had a script that had specific modes which could save off the generated code for a build and could compare the current build to a saved generation. We need a generic version of that for this task.

#5 Updated by Greg Shah 8 months ago

Greg Shah wrote:

Tomasz: Can you please summarize how the ChUI Regression testing handles this today? The older version of it had a script that had specific modes which could save off the generated code for a build and could compare the current build to a saved generation. We need a generic version of that for this task.

Tomasz?

#6 Updated by Greg Shah 8 months ago

The core idea here is to create tooling that implements a comparison of the converted output and a captured baseline. An old version of this for ChUI regression testing was just using diff and scripting to save off/compare against baselines.

Whatever we do must integrate with our CI/CD processing.

There are 2 expected scenarios:

  • The version of FWD being tested is expected to generate no differences in output. This one is easy, any differences are a failure.
  • The version of FWD being tested has some expected differences. This one is trickier, because we would want to define some rules for how to know if a diff is expected. The old ChUI regression testing had nothing automated for this. The user had to review the diffs manually. We don't want to do that. My initial thought is that in this case we should provide a list of regexes defining expected changes where at least one of these must be present in a diff for it to "pass". If all diffs are "expected" then the overall testing passes.

My instinct is that we should write some Java tooling to handle this process.

Documentation will be placed in Conversion Testing.

#7 Updated by Tomasz Domin 8 months ago

Greg Shah wrote:

Greg Shah wrote:

Tomasz: Can you please summarize how the ChUI Regression testing handles this today? The older version of it had a script that had specific modes which could save off the generated code for a build and could compare the current build to a saved generation. We need a generic version of that for this task.

Tomasz?

Well, actually I have a script that parses Harness HTML report into plain preformatted text. Tests are sorted and each test is reported separately so I can diff it with a baseline report.
I do not do any kind of archiving - just making sure container name thus report name included all relevant information about build or patch like:
  • java_8_pg_17_xfer_trunk_16060-20250723161816 - Java 8 with postgresql 17 trunk revision 16060 (there should be main mentioned as main regression tests)
  • java_8_xfer_ctrlc_8661b_15875-20250318112244 - Java 8 ctr-c test on branch 8661b revision 15875
  • java_8_xfer_9687c_16211_trunk_16184_all1-20250925212622 - Java 8 branch 9687c revision 15875 based on trunk 16184 scenario all1

#8 Updated by Greg Shah 8 months ago

I'm not talking about runtime testing. I'm talking about the way we used to use diff to compare the entire converted source code for the app to a captured baseline.

#9 Updated by Tomasz Domin 8 months ago

Greg Shah wrote:

I'm not talking about runtime testing. I'm talking about the way we used to use diff to compare the entire converted source code for the app to a captured baseline.

It was all manual.

#10 Updated by Greg Shah 8 months ago

It had been built into our scripts to capture the diffs, save off the newly built code and compare against a specific recent baseline. I guess we aren't doing any of that now. Is that right?

Also available in: Atom PDF