FWD » Testing

A range of things is needed for each test set:

• cfg
• database dumps
• data files
• application server cfg/setup
• database server cfg
• scripting for deployment/conversion/compilaton/build/execution

Each test set in the Testcases project needs to be rationalized for the above. This means:

1. Come up with a consistent design/approach- that meets the requirements of the full range of test sets.
2. Migrate each test set to the new approach.
3. Test and confirm it is all working.
4. Document it.

The configuration, setup and scripting standardization needs to be done for both OE and FWD. The idea is that each test set should follow the standard and we should find it easy to deploy/convert/compile/execute the tests on both OE and FWD.

I put some thought into this yesterday while working on the testcases project. I was wondering if we can consider "testcases" as a single big project and integrate in FWD all of the configuration changes per slices. In other words:

• for conversion, the settings are driven by profile configuration. What if we can define a default profile and separate profiles for certain testing slices. FWD will convert in a multi-profile fashion. This way, we shall be able to convert the whole testcases project with one shot, instead of engineering zset files for each slice.
• for build time, I am not sure if there is anything that shall change.
• for runtime:
  • I think most of the configuration can be driven by a client session configuration (session collation, session language, etc.). So, instead of running one single fat ./unittest.sh that will eventually scan all tests and run them, to have ./unittest.sh fire multiple clients with different configurations at client level.
  • For server-side configuration, I need to actively think of some examples. I can imagine unit tests exploiting a certain directoy.xml flag. So maybe we can engineer some directory.xml overlay at runtime over which we can write. A FWD extension like DIRECTORY:SESSION-CACHE-ENABLED = false can work (of course, the attributes shall rather match the actual path in directory.xml for ease). Runtime shall be slightly refactored to cache DirectoryBoolean instead of boolean in order to mutate at runtime. Tests would eventually change the setting in setup and adjust it back in teardown.
    • For example, the dynamic query parser that now uses the preprocessor works differently with/without the enabling it in directory.xml. So, unit tests pass only with that flag active (which is toggled false by default). However, I would imagine that unit tests shall rather be distinctive: pass without preprocessor if procepocessor is disabled or pass with preprocessor if preprocessor is enabled. I would prefer to toggle that option in directory.xml only for this test suite. Eventually, I would like to do some FWD specific unit tests that assert failure when flag is disabled (FWD exclusive unit tests).
  • I doubt there is a counterpart of directory in OE, so the FWD extension I propose is fine for wrapping in FWD exclusive code. Any configuration that shall occur for OE as well is most probably a client session configuration.
• database integration is a bit more tricky, but I am quite confident that testcases can simply manage a big fat set of databases with all kind of quirky stuff in it. Trivial testcases will use trivial tables, while complex testcases will use complex set of tables and triggers.

I would say that such approach would cover 99% of the configuration.

1. Come up with a consistent design/approach- that meets the requirements of the full range of test sets.

I think test-cases source code shall only embed FWD preprocessor or FWD extension directory session handle at most. Everything else can be driven by some multi-profile conversion. The setup of tests shall be able to set-up their proper configuration either with 4GL (SESSION) or FWD (DIRECTORY) + teardown.

Using profiles is definitely the expected path. We are planning to implement #7169 (the DependencyManager) to allow the order of profile conversion to be driven by FWD. If we find a feature missing from our profile support, we will consider adding it.

So maybe we can engineer some directory.xml overlay at runtime over which we can write.

Please see #1842 and #2316 which are both part of our plan.