A range of things is needed for each test set:

• cfg
• database dumps
• data files
• application server cfg/setup
• database server cfg
• scripting for deployment/conversion/compilaton/build/execution

Each test set in the Testcases project needs to be rationalized for the above. This means:

1. Come up with a consistent design/approach- that meets the requirements of the full range of test sets.
2. Migrate each test set to the new approach.
3. Test and confirm it is all working.
4. Document it.

The configuration, setup and scripting standardization needs to be done for both OE and FWD. The idea is that each test set should follow the standard and we should find it easy to deploy/convert/compile/execute the tests on both OE and FWD.

I put some thought into this yesterday while working on the testcases project. I was wondering if we can consider "testcases" as a single big project and integrate in FWD all of the configuration changes per slices. In other words:

• for conversion, the settings are driven by profile configuration. What if we can define a default profile and separate profiles for certain testing slices. FWD will convert in a multi-profile fashion. This way, we shall be able to convert the whole testcases project with one shot, instead of engineering zset files for each slice.
• for build time, I am not sure if there is anything that shall change.
• for runtime:
  • I think most of the configuration can be driven by a client session configuration (session collation, session language, etc.). So, instead of running one single fat ./unittest.sh that will eventually scan all tests and run them, to have ./unittest.sh fire multiple clients with different configurations at client level.
  • For server-side configuration, I need to actively think of some examples. I can imagine unit tests exploiting a certain directoy.xml flag. So maybe we can engineer some directory.xml overlay at runtime over which we can write. A FWD extension like DIRECTORY:SESSION-CACHE-ENABLED = false can work (of course, the attributes shall rather match the actual path in directory.xml for ease). Runtime shall be slightly refactored to cache DirectoryBoolean instead of boolean in order to mutate at runtime. Tests would eventually change the setting in setup and adjust it back in teardown.
    • For example, the dynamic query parser that now uses the preprocessor works differently with/without the enabling it in directory.xml. So, unit tests pass only with that flag active (which is toggled false by default). However, I would imagine that unit tests shall rather be distinctive: pass without preprocessor if procepocessor is disabled or pass with preprocessor if preprocessor is enabled. I would prefer to toggle that option in directory.xml only for this test suite. Eventually, I would like to do some FWD specific unit tests that assert failure when flag is disabled (FWD exclusive unit tests).
  • I doubt there is a counterpart of directory in OE, so the FWD extension I propose is fine for wrapping in FWD exclusive code. Any configuration that shall occur for OE as well is most probably a client session configuration.
• database integration is a bit more tricky, but I am quite confident that testcases can simply manage a big fat set of databases with all kind of quirky stuff in it. Trivial testcases will use trivial tables, while complex testcases will use complex set of tables and triggers.

I would say that such approach would cover 99% of the configuration.

1. Come up with a consistent design/approach- that meets the requirements of the full range of test sets.

I think test-cases source code shall only embed FWD preprocessor or FWD extension directory session handle at most. Everything else can be driven by some multi-profile conversion. The setup of tests shall be able to set-up their proper configuration either with 4GL (SESSION) or FWD (DIRECTORY) + teardown.

Using profiles is definitely the expected path. We are planning to implement #7169 (the DependencyManager) to allow the order of profile conversion to be driven by FWD. If we find a feature missing from our profile support, we will consider adding it.

So maybe we can engineer some directory.xml overlay at runtime over which we can write.

Please see #1842 and #2316 which are both part of our plan.

Per-Test directory.xml Configuration¶

Tests need different server configurations — logging levels, pool sizes, ACLs, locking and tenancy settings. Keeping a full directory.xml per test does not scale: every schema change touches all of them, and they drift.

Approach¶

Keep one canonical directory.xml in the repo. Each test that needs something different ships a small overlay listing only what it changes. Before the server starts, a rewriter copies the canonical file, applies the overlay, and writes the result to the deploy location. The server reads it normally; nothing in the test knows.

The pieces¶

python3 support/update-directory.py \
    --base    cfg/directory.xml \
    --overlay tests/persist/locking/directory.json \
    --out     deploy/server/directory.xml

A suite with no overlay is deployed the canonical file unchanged, so adding the mechanism costs existing tests nothing.

The overlay¶

A flat map of location to value. Path segments are name attributes walked from the document root; \@attr targets an attribute on the node itself instead of its value child.

{
  "server/default/logging/loggers/com.goldencode.p2j.persist": "FINE",
  "server/standard/appservers/appsrv@maxConnectionsPerAgent": "1" 
}

The rewriter¶

• Always read --base, never the deployed file. Rewriting the deployed cous suite's settings in it, and tests silently inherit each other'sconfiguration.
• Anchor every path; never search. Node names repeat at different depths isive search for the first match retargets itself the day a node is addedabove the intended one.
• Exit non-zero on an unresolvable path. A silently ignored override means— without the configuration it claims to need. A path that is meant to create
  a missing node marks itself, so typos stay loud.
• Copy the generated file into the results. When a test fails, the first question is what it actually ran against.

Where it hooks¶

The rewriter is standalone, so it hangs wherever the runner already deploys wrapper around the server launch, or a per-suite setup step. It needs no knowledge of the tests and no changes to FWD.

Thanks for posting that good idea.

One reason I hesitate to use a python tool is that we intend to use this same approach for customers via the IDE support. That needs a Java solution rather than a one-off python tool.

Another thing to mention here is that we have this issue in production as well. And I have specified a solution in #1842. I think this is the proper answer because it will greatly help this exact case, but in a cleaner way. Please review #1842. It is something we can probably implement fairly quickly, I think.