FWD » Testing

1. the artifacts in the project it is linked into with path to the baseline conversion zip file, or
2. 2 zips containing the artifacts to compare

I'm not a fan of how we hook into zip files being unzipped prior to conversion. I would think that a branch should just have the revisions checked into it.

Tomasz: Can you please summarize how the ChUI Regression testing handles this today? The older version of it had a script that had specific modes which could save off the generated code for a build and could compare the current build to a saved generation. We need a generic version of that for this task.

Greg Shah wrote:

Tomasz: Can you please summarize how the ChUI Regression testing handles this today? The older version of it had a script that had specific modes which could save off the generated code for a build and could compare the current build to a saved generation. We need a generic version of that for this task.

Tomasz?

The core idea here is to create tooling that implements a comparison of the converted output and a captured baseline. An old version of this for ChUI regression testing was just using diff and scripting to save off/compare against baselines.

Whatever we do must integrate with our CI/CD processing.

There are 2 expected scenarios:

• The version of FWD being tested is expected to generate no differences in output. This one is easy, any differences are a failure.
• The version of FWD being tested has some expected differences. This one is trickier, because we would want to define some rules for how to know if a diff is expected. The old ChUI regression testing had nothing automated for this. The user had to review the diffs manually. We don't want to do that. My initial thought is that in this case we should provide a list of regexes defining expected changes where at least one of these must be present in a diff for it to "pass". If all diffs are "expected" then the overall testing passes.

My instinct is that we should write some Java tooling to handle this process.

Documentation will be placed in Conversion Testing.

Greg Shah wrote:

Greg Shah wrote:

Tomasz: Can you please summarize how the ChUI Regression testing handles this today? The older version of it had a script that had specific modes which could save off the generated code for a build and could compare the current build to a saved generation. We need a generic version of that for this task.

Tomasz?

• java_8_pg_17_xfer_trunk_16060-20250723161816 - Java 8 with postgresql 17 trunk revision 16060 (there should be main mentioned as main regression tests)
• java_8_xfer_ctrlc_8661b_15875-20250318112244 - Java 8 ctr-c test on branch 8661b revision 15875
• java_8_xfer_9687c_16211_trunk_16184_all1-20250925212622 - Java 8 branch 9687c revision 15875 based on trunk 16184 scenario all1

I'm not talking about runtime testing. I'm talking about the way we used to use diff to compare the entire converted source code for the app to a captured baseline.

Greg Shah wrote:

I'm not talking about runtime testing. I'm talking about the way we used to use diff to compare the entire converted source code for the app to a captured baseline.

It was all manual.

It had been built into our scripts to capture the diffs, save off the newly built code and compare against a specific recent baseline. I guess we aren't doing any of that now. Is that right?

Greg Shah wrote:

It had been built into our scripts to capture the diffs, save off the newly built code and compare against a specific recent baseline. I guess we aren't doing any of that now. Is that right?

I will look into this with the expanded extent work.

I started to place the conversion testing scripts in the majic_baseline project, but I don't like that. The conversion testing should really be it's own project, because you should be able to run this on any conversion project. How is "conversion_testing" as a name?

I'm fine with conversion_testing.

Greg Shah wrote:

• The version of FWD being tested has some expected differences. This one is trickier, because we would want to define some rules for how to know if a diff is expected. The old ChUI regression testing had nothing automated for this. The user had to review the diffs manually. We don't want to do that. My initial thought is that in this case we should provide a list of regexes defining expected changes where at least one of these must be present in a diff for it to "pass". If all diffs are "expected" then the overall testing passes.

How feasible is this regex half? The idea sound good. But Claude tried writing such a set against a real comparison and the result wasn't spectacular.

The comparison was FWD 16766 against 16789, where the only substantive change was the BEGINS rework: 2543 changed source lines across 256 files. A six-pattern regex set written after reading the diff covered 2250 lines. But 293 lines escaped it. They were real misses the patterns did not anticipate:

<  new FindQuery(vendor, "upper(vendor.name) like ? and ...
>  new FindQuery(vendor, "upper(vendor.name) like upper(?) and ...

• a pattern set that covers a change well usually has to be written after seeing the diff, which weakens "declare intent up front"
• The skill required is high and the feedback is poor. You could cover up real problems or falsely show non-problems with poorly written regexes.

None of this says the approach is wrong. It does suggest the regex set is itself an artifact that needs review, and that generating a candidate set from an actual diff, then having a person tighten it, may be more realistic than expecting it to be hand-written cold. That is a place where AI assistance is plausibly the difference between the feature working and being quietly abandoned.

Greg Shah wrote:

I'm fine with conversion_testing.

That's OK, but I was also thinking to add --save-baseline as an argument to the FWD conversion, which would save off the artifacts. ArtifactManager has access to basepath and output-root by Configuration and maintains a registry of every artifact, with getArtifacts(), getArtifactsRelativePath() and getArtifactByFilter(). Each Artifact knows its relative path, absolute path and whether it exists. So why not do the work there? Scripts could drive the testing, but FWD can generate what is needed for the comparisons.

I think we agree that content has to be stored, not just fingerprints. Otherwise we would have enough information for the "alleged" regexes, or just diffs in general. A zip seems right, matching the suggestion in #6862-2, with a manifest inside it rather than beside it.

Metadata in three tiers: identity (FWD version including branch, timestamp, host); comparability (resolved basepath, output-root, pkgroot, the active profile set, fingerprints of the 4GL and schema actually read); and audit (the full resolved parameter set, artifact counts, whether javac succeeded).

The ArtifactManager is the right place for archiving.

We probably should archive the inputs (4GL sources) as well as our intermediate and output artifacts. The point here is that with real sources we can duplicate our results. With fingerprints only, we can only confirm if something was the same or not.