FWD » Core Development » Database

This task will be dedicated to the effort of writing multi session tests using harness. The results of running the tests with the latest trunk will be used as baseline. Any potential problems found in the trunk or general issues with harness will be discussed here.

Please commit to xfer/testcases.

Committed the tests to support/harness/multi-session-tests, testcases rev 1809.

Artur
Shouldnt be possible to run those tests in OE ?

Hi,
It is possible to run them in OE, just not in an automated mode. I ran the two procedures at the same time, pressed the keys and checked the output.

Oh, you meant running harness in OE... I did not try it. Is it a hard requirement?

Artur Școlnic wrote:

Oh, you meant running harness in OE... I did not try it. Is it a hard requirement?

I think so. I guess tests should be done as defined in #9270.

I would expect multi session tests to behave differently in OE compared to FWD since OE doesn't take transaction isolation seriously :)

Artur Școlnic wrote:

I would expect multi session tests to behave differently in OE compared to FWD since OE doesn't take transaction isolation seriously :)

I guess thats the point to discover such cases.

Is there a wiki or specific notes in #9270 on running harness in OE?

Artur Școlnic wrote:

Is there a wiki or specific notes in #9270 on running harness in OE?

Yep, there is Writing_Multi-User_Testcases, in #9270 I tried to make a template project to be filled with actual content.

The wiki is very new, let me know if I can help you if something is missing.

Tomasz Domin wrote:

Artur Școlnic wrote:

Oh, you meant running harness in OE... I did not try it. Is it a hard requirement?

I think so. I guess tests should be done as defined in #9270.

Correct. All of these tests should run using full automation on both OE and FWD.

Tomasz Domin wrote:

The wiki is very new, let me know if I can help you if something is missing.

Yes, please help me understand a few details.
1. What is PCT and how to configure and use it?
2. What is DLC?
3. Are there any windows scripts to run the tests?

Basically how to run harness on windows from scratch :)

Artur Școlnic wrote:

Tomasz Domin wrote:

The wiki is very new, let me know if I can help you if something is missing.

Yes, please help me understand a few details.
1. What is PCT and how to configure and use it?

Its support library for ant to allow using it with OE: https://github.com/Riverside-Software/pct

2. What is DLC?

Environental environment used by ant scripts point to directory OE is installed

3. Are there any windows scripts to run the tests?

There are none, as there are also no scripts for Linux.

See Writing_Multi-User_Testcases and Writing_Multi-User_Testcases

I made some edits to reflect this information (see Definitions) and some cleanup of the docs.

Should the tests be changed for OE?
In fwd the converted procedures are ran using client.sh, I am not sure how harness is supposed to start the procedures in OE.

Artur Școlnic wrote:

Should the tests be changed for OE?

Artur
The very first sentence from: Writing_Multi-User_Testcases
The goal of Multi-User Testcases project is allowing executing multi-user tests in both:FWD and OE runtime environments with no modifications to the sources of the tests.

In fwd the converted procedures are ran using client.sh, I am not sure how harness is supposed to start the procedures in OE.

In both cases procedures are started by ant. Its all in Writing_Multi-User_Testcases

So instructions like this one will be handled correctly?

-<send-text>

-<substitution name="value" spec="cd %s; ./client.sh client:cmd-line-option:startup-procedure=cross_session/full-transactions-find-first_control.p">

<parameter variable="runDir"/>

</substitution>

</send-text>

Artur Școlnic wrote:

So instructions like this one will be handled correctly?
[...]

Not in OE.

This is what I am talking about. There must be some differences in the tests when running harness in OE and FWD.

I think I understand now. The tests I added in support/harness/multi-session-tests were not meant to run on OE. Their purpose was to establish a baseline using trunk when it comes to transaction isolation behavior and then test tasks like #10583 to confirm that we don't introduce regressions.

Greg,
Should I convert/write a new test suite that is OE compatible?

You must change your approach to match the one documented in Writing Multi-User Testcases. In other words, do not put things into the test definition that are specific to either OE or to FWD. The tests must be written to run on both.

I think I understand now. The tests I added in support/harness/multi-session-tests were not meant to run on OE.

Please help me understand how the output of full_transaction-find-first_main4.txt is 1, 2, 2, 2. There is an update to 3 at some point, but it doesn't look like it is taking effect. I would expected to see 1, 2, 3, 3. Also, the control is 1, 1, 1, 2, which looks almost ok - why isn't the last one 3?

In note 4 I posted one example of seemingly incorrect behavior in trunk, but it's not the only one. full_transaction-find-first is one of them.

Artur Școlnic wrote:

In note 4 I posted one example of seemingly incorrect behavior in trunk, but it's not the only one. full_transaction-find-first is one of them.

In note 4 there is a test that targets FOR EACH. My comment in note 30 is about a FIND FIRST. The problem you are seeing is more spread than initially envisioned. Please take time to identify the issue and fix. multi-session-tests are hard to review considering they are not functionally stable.

• Please do a bit more testing on the locking mechanism. Even for #10583, it is of great concern whether we may leak locked (and stale) records from one session to another. For a bit of context, FWD is fully hydrating No-LOCK records, but only does projection queries on SHARE-LOCK or EXCLUSIVE-LOCK; mostly due to READ_COMMITTED transaction isolation level. Thus, we need to ensure that the READ_COMMITTED isolation is honored through the locking system and refetching system - which might conflict with #10583 implementation.
• Some caches may invalidate the intent in the tests. For instance, the Session.cache, reclaiming, FfCache, c3p0 etc. are all possible layers that hide design flaws. In other words, the tests should be executed in different configurations:
  • With no session.cache / minimal session cache; with/without ff cache; with/without reclaiming; with/without connection pooling; any combination of these.
• Different database engines should be tested: SQLServer, H2, MariaDB, PG (all that are supported in FWD).
• I am also interested in:
  • some tests that retrieve a record from the DB, updates it, releases it, at some point it is evicted from the session cache and then gathered back in the session cache (with CACHED state).
  • some tests with 2 users that work on the same record; one is committing the changes on it and the other should see it as STALE (related to READ_COMMITTED).
  • SCROLLING queries are different as they use a cursor underneath that does some caching on its own. In particular, BROWSED queries have a different behavior as OE doesn't instantly update browse views when a change occurs (Real-time). There is an algorithm to infer exactly how to assess modifications occurred on other sessions.
  • PreselectQuery has a SKIP-MISSING-RECORDS attribute that changes its behavior especially if a record is deleted while iterating (by this or another session)
  • rollback is a really nasty thing (in full or sub- transactions) and it should get better tested (#10583 may share records cross-session, but one record may be rollbacked and its state might get altered).
  • there are other kinds of queries that may trigger #10583: invalidated AdaptiveQuery, CompoundQuery that has an AQ that gets invalidated, CompoundQuery with a RAQ component, AdaptiveFind query, etc.
• I think we need better testing overall for dirty reads, non-repeatable reads and phantom reads + which are the OE DB defaults which should be honored. (PS: we had issues before when configuring SQLSever and MariaDB for customers to not properly set the transaction isolation and end up with non-functional run-time).

Eric/Greg/Constantin: these are some ideas especially in regard to #10583 Let me know if you have broader topics to assess with this.

My feeling is that we should take a step back and create a broader plan for this testing effort. I would suggest going back and forth through the Redmine tasks and identify common issues in synchronization at DB level. With that, we can design a more complex testsuite.

TL;DR I am worried to say the least that FWD's multi-user scenarios are tested with a couple of minutes/tests of stress. With more customers deploying FWD in tens/hundreds of users (our of which some are concurrent), I feel like this topic of multi-user testing is nowhere close to what we require.

Alexandru,
When it comes to issues found in trunk, should I open new tasks and postpone fixing them, or fix them first then continue with writing tests?

Yes, I agree that we have done a crappy job of building tests in this area. For years, I have been pleading with people to work on this very thing.

One thing that was holding us back was that we didn't have an approach for how to write such tests. That was solved in #9270.

Another thing I've been looking for: for Eric to gather up his flushing/validation and locking tests and check them into the testcases project with enough readme to explain what is there. He has a non-trivial set of tests that would be a good starting point if we can convince him to spend some minimal effort here so that we can move on.

I have previously requested that the tests be organized as follows:

• Flushing and Validation Tests
• Lock Tests
• Dirty Share (intra/cross)

Artur Școlnic wrote:

Alexandru,
When it comes to issues found in trunk, should I open new tasks and postpone fixing them, or fix them first then continue with writing tests?

Open tasks and postpone. We need the tests first.

Tomasz,
Is there a way to run the ABL Unit Tests separately (not in harness)?

Artur Școlnic wrote:

Tomasz,
Is there a way to run the ABL Unit Tests separately (not in harness)?

Sure. You can start them with ant the same way as harness starts them like in testcases project:
ant -Dtests.pattern=**/TestLockSession1.* -Dtests.dbdir=. -Dtests.db=testcases.db -Dtests.out=%s/build/cross_locking_session_001
-Dtests.dir=support/harness/test -Dtests.tempDir=build/cross_locking_session_001 %s

For FWD You can also you command line tools like ./run-tests.sh -p com.goldencode.testcases.tests.log_manager - see testcases/deploy/client.
For OE - please check OE documentation .

Sorry, I meant completely separate, as an ABL unit class. I see that a mutex is being used for synchronization between tests, it uses a socket tcp connection on a specific port, but there should be something to connect to first, I guess it is the harness server. Is there a way to start it just to listen for connections, so not for executing a test suite.
The reason I am asking is that I need to write the tests first in ABL unit, then integrate them in harness, for that I need abl harness api to work properly.

I have previously requested that the tests be organized as follows:

Looking at DatabasePersistence_Testcases, I could spot the mischievous CAN-FIND, triggers and multi-tenant subjects I completely missed from #10664-32. Oh well, I guess DatabasePersistence_Testcases does a better job than me trying to brainstorm edge cases in multi-session cases.

Open tasks and postpone. We need the tests first.

But we still need them to be fixed before moving on with #10583.

The reason I am asking is that I need to write the tests first in ABL unit, then integrate them in harness, for that I need abl harness api to work properly.

Just FYI, FWD output will be different from OE by design if opting for disabling cross-session dirty-share. But this is only about cross-session visibility of transient records before committing.

Just FYI, FWD output will be different from OE by design if opting for disabling cross-session dirty-share. But this is only about cross-session visibility of transient records before committing.

Understood. This is OK.

Artur Școlnic wrote:

Sorry, I meant completely separate, as an ABL unit class. I see that a mutex is being used for synchronization between tests, it uses a socket tcp connection on a specific port, but there should be something to connect to first, I guess it is the harness server. Is there a way to start it just to listen for connections, so not for executing a test suite.
The reason I am asking is that I need to write the tests first in ABL unit, then integrate them in harness, for that I need abl harness api to work properly.

Yep, you need harness server. Just start a simplified and almost empty scenario with all needed mutexes and event semaphores like that:

harness_server.xml

<?xml version="1.0"?>
<test-plan name="Harness server" description="Harness server" parallel = "2">
   <output-directory path="results" unique="true" />
   <target type="stream" shell="bash" />
   <resources>
      <create-event-sem name="setupCompleted" set="true"/>
      <create-event-sem name="testsCompleted" set="true"/>
      <create-mutex name="testsMutex"/>
      <create-mutex name="dsUnique"/>
      <create-mutex name="dsNonUnique"/>

      <create-event-sem name="runSession1" set="true"/>
      <create-event-sem name="runSession2" set="true"/>

   </resources>

   <test-set name="record_locking_test" description="Cross session record locking tests" threads="2" >
      <target type="remote" port="8881" threads="2" />
      <test filename="support/harness/test/wait_forever.xml" dependency="concurrent" group="xsession1" />
   </test-set>
</test-plan>

wait_forever.xml

<?xml version="1.0"?>
<test name="wait_forever" description="wait_forever" >
    <stream-wait-for value="this will never happen"  millis="100000000" newline="false"/>
</test>

Run it and you have harness server socket on port 8881:
java -jar ~/projects/harness/build/lib/harness.jar -d support/harness/test/harness_server.xml

This is exactly what I needed, thanks!

Going forward I will organize the tests like Greg mentioned in #10664-34.

*Flushing and Validation Tests
*Lock Tests
*Dirty Share (intra/cross)

Eric, are the flushing/validation and locking tests, Greg mentioned, the ones from testcases/tests/persistence/flushing_validation or are there other tests that are not yet committed to testcases?

Artur Școlnic wrote:

This is exactly what I needed, thanks!

I've added the example to the wiki.

Tomasz,
With the example from testcases I get this in build/*/results.xml

<failure message="Test engine was not initialized." type="java.lang.Exception"><![CDATA[java.lang.Exception: Test engine was not initialized.
    at com.goldencode.p2j.testengine.NoopDescriptor.execute(NoopDescriptor.java:211)
    at com.goldencode.p2j.testengine.NoopDescriptor.execute(NoopDescriptor.java:79)
    at com.goldencode.p2j.testengine.FWDTestEngine$FWDThrowableCollector.execute(FWDTestEngine.java:410)
    at com.goldencode.p2j.testengine.FWDTestEngine$FWDThrowableCollector.execute(FWDTestEngine.java:410)
]]></failure></testcase><system-err>Oct 13, 2025 2:19:21 PM com.goldencode.p2j.testengine.FWDTestEngine initialize
WARNING: Test engine cannot initialize and will be disabled: com.goldencode.p2j.cfg.ConfigurationException: Error loading bootstrap configuration file unit_test.xml
</system-err></testsuite>

Artur Școlnic wrote:

Tomasz,
With the example from testcases I get this in build/*/results.xml
[...]
The server is running and the build was successful. Could you please advise?

Artur
Which example do you refer to ? #10664-41 is about starting the server, not executing the tests.

Sorry, I meant the one from testcases testcases/support/harness/test.

Artur Școlnic wrote:

Sorry, I meant the one from testcases testcases/support/harness/test.

I need more information - did you convert the project testcases, did you start the server, how did you start the client etc.

I did exactly as the wiki states.
1. Change zfile_set.txt
2. and deploy.all for building testcases
3. start the server
4. run harness
Also tried to run one test separately with ant

ant -Dtests.pattern=**/TestLockSession1.* -Dtests.dbdir=. -Dtests.db=testcases.db -Dtests.out=/home/as/projects/testcases/build/cross_locking_session_001 -Dtests.dir=support/harness/test -Dtests.tempDir=build/cross_locking_session_001 junit-tests-pattern

For OE there are several issues.
1. The ablunit-tests-pattern is not located in build.xml so the correct file needs to be mentioned -f build_ablunit.xml ablunit-tests-pattern
2. Even with the correct command :

C:\projects\testcases>ant -Dtests.pattern=**/TestLockSession1.* -Dtests.dbdir=.
-Dtests.db=testcases.db -Dtests.out=C:\projects\testcases/build/cross_locking_se
ssion_001 -Dtests.dir=support/harness/test -Dtests.tempDir=build/cross_locking_s
ession_001 -f build_ablunit.xml ablunit-tests-pattern 

Buildfile: C:\projects\testcases\build_ablunit.xml
  [taskdef] Could not load definitions from resource PCT.properties. It could no
t be found.
  [typedef] Could not load definitions from resource types.properties. It could
not be found.

BUILD FAILED
C:\projects\testcases\build_ablunit.xml:61: Problem: failed to create task or type DBConnectionSet
Cause: The name is undefined.
Action: Check the spelling.
Action: Check that any custom tasks/types have been declared.
Action: Check that any <presetdef>/<macrodef> declarations have taken place.

Total time: 0 seconds

Artur Școlnic wrote:

I did exactly as the wiki states.
1. Change zfile_set.txt
2. and deploy.all for building testcases
3. start the server
4. run harness
Also tried to run one test separately with ant
[...]
same result.

I've updated to the lastest testcases and did the same and it works:

(note I am also working on log_manager at the moment)

~/projects/testcases$ more zfile_set.txt
#D ./support/ (*.[pPwW]|*.cls)
#X ./support/gen/*
#X ./harness.old/*
N ./tests/log_manager/ (*.[pPwW]|*.cls)
N ./tests/log_manager/log_entry_types/ (*.[pPwW]|*.cls)
N ./tests/log_manager/support/ (*.[pPwW]|*.cls)
#harness API
F support/harness/HarnessApi.cls
F support/harness/HarnessCommand.cls
F support/harness/HarnessConnection.cls
F support/harness/HarnessMessage.cls
F support/harness/readResponseHandler.p

#cross dirty share
F support/harness/test/TestNonUniqueDirtyShareSession1.cls
F support/harness/test/TestNonUniqueDirtyShareSession2.cls
F support/harness/test/TestUniqueDirtyShareSession1.cls
F support/harness/test/TestUniqueDirtyShareSession2.cls

#cross session locks
F support/harness/test/TestLockSession1.cls
F support/harness/test/TestLockSession2.cls

Harness server (I had to change port to 8882 in harness_server.xml):
~/projects/testcases$ java -jar ~/projects/harness/build/lib/harness.jar -d support/harness/test/harness_server.xml
session 1:

~/projects/testcases$ ant -Dtests.pattern=**/TestNonUniqueDirtyShareSession1.* -Dtests.dbdir=. -Dtests.db=testcases.db -Dtests.out=/home/tjd/projects/testcases/build/dirty_share_nonunique_session1 -Dtests.dir=support/harness/test -Dtests.tempDir=build/dirty_share_nonunique_session1 junit-tests-pattern
Buildfile: /home/tjd/projects/testcases/build.xml

download-ant-junitlauncher:

init-ant-junitlauncher:

junit-tests-pattern:
     [echo] Running JUnit tests for pattern '**/TestNonUniqueDirtyShareSession1.*'.
[junitlauncher2] Running com.goldencode.testcases.support.harness.test.TestNonUniqueDirtyShareSession1

BUILD SUCCESSFUL
Total time: 6 seconds

session 2:

~/projects/testcases$ ant -Dtests.pattern=**/TestNonUniqueDirtyShareSession2.* -Dtests.dbdir=. -Dtests.db=testcases.db -Dtests.out=/home/tjd/projects/testcases/build/dirty_share_nonunique_session1 -Dtests.dir=support/harness/test -Dtests.tempDir=build/dirty_share_nonunique_session1 junit-tests-pattern
Buildfile: /home/tjd/projects/testcases/build.xml

download-ant-junitlauncher:

init-ant-junitlauncher:

junit-tests-pattern:
     [echo] Running JUnit tests for pattern '**/TestNonUniqueDirtyShareSession2.*'.
[junitlauncher2] Running com.goldencode.testcases.support.harness.test.TestNonUniqueDirtyShareSession2

BUILD SUCCESSFUL
Total time: 3 seconds

Artur Școlnic wrote:

For OE there are several issues.
1. The ablunit-tests-pattern is not located in build.xml so the correct file needs to be mentioned -f build_ablunit.xml ablunit-tests-pattern

Nope, tjd/17.0.16:~/projects/testcases$ grep build_ablunit.xml *.xml
build.xml: <import if:set="pct-jar-exists" file="build_ablunit.xml" />

Please make sure you've set DLC environment variable correctly and there should be pct/PCT.jar file.

• Download PCT, build it, place the PCT.jar in %DLC%\pct\PTC.jar
• Create a tstcasesdb, start the server for it, import all the data from tstcasesdb.df

But even after that I get:

Buildfile: C:\projects\testcases\build_ablunit.xml

ablunit-tests-pattern:

clean.tests:

clean-e4gl:
     [exec] Execute failed: java.io.IOException: Cannot run program "sh" (in directory "C:\projects\testcases"): CreateProcess error=2, The system cannot find the file specified

clean.convert:

compile-tests-pattern:
     [echo] Building ABLUnit tests for pattern '**/TestLockSession1.*' in chui m
ode.
[PCTCompile] PCTCompile - Progress Code Compiler
[PCTCompile] Working with PCT source code, either because you asked for it or be
cause the compiled version is missing in JAR file
[PCTCompile] ** Incompatible data types in expression or assignment. (223)
[PCTCompile] ** C:\Users\ADMINI~1\AppData\Local\Temp\pct160610221\pct\compile.p
Could not understand line 234. (198)

BUILD FAILED
C:\projects\testcases\build_ablunit.xml:93: The following error occurred while executing this line:
C:\projects\testcases\build_ablunit.xml:480: Progress procedure failed - Error reading return value

Total time: 2 seconds

Wait, does the harness server also need to run alongside the FWD server?

Artur Școlnic wrote:

Wait, does the harness server also need to run alongside the FWD server?

In linux/FWD.

Artur Școlnic wrote:

Yes, there indeed are a few extra steps.

• Download PCT, build it, place the PCT.jar in %DLC%\pct\PCT.jar
• Create a tstcasesdb, start the server for it, import all the data from tstcasesdb.df

I think we need to mention these things in the wiki. Also, building PCT yourself is not a good idea, I had some issues because of it, downloading from https://repo1.maven.org/maven2/eu/rssw/pct/pct/226/pct-226.jar solved them.

Anyway...after solving these configuration issues, I am still not able to start the tests with ant on OE, I get OpenEdge Character client has stopped working, which is a pretty generic message, I have no clue as to what triggers it.
The tests run fine using Progress Developer Studio.

Artur Școlnic wrote:

Artur Școlnic wrote:

Yes, there indeed are a few extra steps.

• Download PCT, build it, place the PCT.jar in %DLC%\pct\PCT.jar
• Create a tstcasesdb, start the server for it, import all the data from tstcasesdb.df

I think we need to mention these things in the wiki. Also, building PCT yourself is not a good idea, I had some issues because of it, downloading from https://repo1.maven.org/maven2/eu/rssw/pct/pct/226/pct-226.jar solved them.

It was in the wiki already: Writing_Multi-User_Testcases
For the test I was using shared servers, they had already the setup needed.

Artur Școlnic wrote:

Anyway...after solving these configuration issues, I am still not able to start the tests with ant on OE, I get OpenEdge Character client has stopped working, which is a pretty generic message, I have no clue as to what triggers it.
The tests run fine using Progress Developer Studio.

Having single message wont help me either :)
Please include a big bigger log :)

Artur Școlnic wrote:

Wait, does the harness server also need to run alongside the FWD server?

Only when running tests from ant, in case harness supervises the tests the harness server is run automatically.

Tomasz Domin wrote:

It was in the wiki already: Writing_Multi-User_Testcases
For the test I was using shared servers, they had already the setup needed.

Yes, these things are mentioned in the wiki, but we need to have detailed instructions on how to set up from scratch the required dependencies for harness on OE, otherwise the next person that needs to use it, will encounter the same issues I am. If some of these things are already documented, the wiki needs to point to the documentation. I am happy to complete the wiki once I have a working set up.

Tomasz,

<stream-write value="ant -Dtests.pattern=**/TestUniqueDirtyShareSession1.* -Dtests.dbdir=. -Dtests.db=testcases.db -Dtests.out=%s/build/dirty_share_unique_session1 -Dtests.dir=support/harness/test -Dtests.tempDir=build/dirty_share_unique_session1 %s" newline="true">

Buildfile: C:\projects\testcases\build.xml

BUILD FAILED
Target "ablunit-tests-pattern" does not exist in the project "testcases".

Total time: 0 seconds

I am happy to complete the wiki once I have a working set up.

Please do make updates to the wiki page to add detail and clarify existing content.

I finally found the issue on OE.
HarnessConnection.cls does not compile (for me) due to:

define public property Connected as logical no-undo
        get:
            return valid-object(hSocket) and hSocket:connected().
        end get.

I also had to deviate a bit from the instructions in the wiki.
I will post here what I want to add and Tomasz can review it.

Prerequisites
There are following prerequisites for using the tool-set (ant/harness) to run tests:

• Use Progress Developer Studio's tool - Data Administration to create a new empty database. Make sure the physical name matches the one in build_ablunit.xml ant file, now it is hard-coded as tstcasesdb.
• The database files will be created in working dir of the current project, move them to testcases. Disconnect from the newly created db.
• Start the database server by executing this command in the proenv cmd:

proserve -S 7003 -n 10 -H localhost tstcasesdb.db

• Connect to the database and load the data definition from testcases/data/tstcasesdb.df

2. database must not be started, PCT will start tstcases.db database
I found that it has to be running before starting the tests.

• Download the PCT library from https://repo1.maven.org/maven2/eu/rssw/pct/pct/226/pct-226.jar
• Rename to PCT.jar and place it in a new dir pct in your progress installation dir (DLC).

There are two changes needed in the testcases project:
1. Solve the issue with valid-object in HarnessConnection.cls
2. Add -f build_ablunit.xml in the and command of the tests, otherwise the correct target is not identified.

Artur Școlnic wrote:

I finally found the issue on OE.
HarnessConnection.cls does not compile (for me) due to:
[...]
hSocket being a handle and not an object, I changed to valid-handle and everything runs smoothly on OE.

I git that code from Marian. I will retest it on shared servers.

Tomasz Domin wrote:

It was in the wiki already: Writing_Multi-User_Testcases
For the test I was using shared servers, they had already the setup needed.

Yes, these things are mentioned in the wiki, but we need to have detailed instructions on how to set up from scratch the required dependencies for harness on OE, otherwise the next person that needs to use it, will encounter the same issues I am. If some of these things are already documented, the wiki needs to point to the documentation. I am happy to complete the wiki once I have a working set up.

Artur
I simply cant because I dont have OE on my own. The shared servers had support for needed functions already and I'd have to break it in order to replay installation.
I guess I skipped OE database creation because its the same for entire testcases project, I need to link it to wiki: Writing_4GL_Testcases, there is procedure: support/db/reload_tstcasesdb.p

Artur Școlnic wrote:

Prerequisites
There are following prerequisites for using the tool-set (ant/harness) to run tests:

for OE

Which exactly OE version ? Maybe the differences are caused by OE runtime differences.

Artur Școlnic wrote:

There are two changes needed in the testcases project:
1. Solve the issue with valid-object in HarnessConnection.cls

I guess this may be a problem with OE 11.6 you are using.
It works fine with OE 11.7 where it was tested by me.
I've tested your change in OE 11.7 and it works as well, so we can commit your version.

2. Add -f build_ablunit.xml in the and command of the tests, otherwise the correct target is not identified.

No, its not neeeded.

Artur Școlnic wrote:

Prerequisites
There are following prerequisites for using the tool-set (ant/harness) to run tests:

for OE
1. database needs to be created in current directory (e.g. testcases).

I confirm that step was missing, its enough to call:

prowin32 -p support\db\create_tstcases_db.p except it creates database is db dir.

• Start the database server by executing this command in the proenv cmd:

proserve -S 7003 -n 10 -H localhost tstcasesdb.db

proserve testcases is enough

2. database must not be started, PCT will start tstcases.db database
I found that it has to be running before starting the tests.

Correct, when I was testing it was running all the time so I didnt have to start it.

3. DLC environmental variable needs to point OE installation directory (to access OE and PCT)

• Download the PCT library from https://repo1.maven.org/maven2/eu/rssw/pct/pct/226/pct-226.jar
• Rename to PCT.jar and place it in a new dir pct in your progress installation dir (DLC).

If it is missing, its present on shared server

There are two changes needed in the testcases project:
1. Solve the issue with valid-object in HarnessConnection.cls

I guess this may be a problem with OE 11.6 you are using.
It works fine with OE 11.7 where it was tested by me.
I've tested your change in OE 11.7 and it works as well, so we can commit your version.

2. Add -f build_ablunit.xml in the and command of the tests, otherwise the correct target is not identified.

No, its not needed. It should be automatically imported. There is a bug in latest build.xml - following line is missing at beginning of the file:

   <property environment="env" />

There were more changes to testcases build.xml which made it incompatible after I build the toolset - I will come back after I recap.

Tomasz Domin wrote:

prowin32 -p support\db\create_tstcases_db.p except it creates database is db dir.
proserve testcases is enough

I guess this may be a problem with OE 11.6 you are using.

Yes, with OE 11.6.3 which a lot of us are using, it is a problem.

You also mentioned a shared server you are using, I am not familiar with it, we are using a VM with windows server 2012 that has a basic OE development environment on it, not much else.

prowin32 -p support\db\create_tstcases_db.p except it creates database is db dir.
proserve testcases is enough

The database needs to be called tstcasesdb and it must be in the testcases root directory.

Our database should be in a subdirectory, so if that means we need to change our setup code or the commands to change the working dir before launching the OE database server, then please do that. We don't want our database files in the project root.

Artur Școlnic wrote:

Tomasz Domin wrote:

prowin32 -p support\db\create_tstcases_db.p except it creates database is db dir.
proserve testcases is enough

The database needs to be called tstcasesdb and it must be in the testcases root directory.

No, it does not need to stay in root, I've updated tests to reflect that.

I guess this may be a problem with OE 11.6 you are using.

Yes, with OE 11.6.3 which a lot of us are using, it is a problem.

You also mentioned a shared server you are using, I am not familiar with it, we are using a VM with windows server 2012 that has a basic OE development environment on it, not much else.

I dont have access and I didnt know about OE 11.6 requirement I guess that makes us even :), well I didn't even write that code.
The good thing is we have scripts tested in both OE 11.7 and 11.6 :)

• synchronize scripts with recent changes to testcases project
• database for OE is now stored in db directory
• updated readme.txt
• added standalone harness server
  Committed revision 1812.

It was tested under Linux and shared OE 11.7

Startup for windows now looks like (its in readme file):

For Windows (assuming having OE on PATH and DLC env variable set):
cd D:\tjd\testcases
prowin32 -p support\db\create_tstcases_db.p
proserve db\tstcasesdb
java -jar d:\tjd\harness\build\lib\harness.jar -d -shell cmd.exe -v testDirectory="D:\tjd\ablunit" -v antTarget=ablunit-tests-pattern support/harness/test/cross_session_test.xml
proshut db\tstcasesdb -by

Greg
Is there a global thread for testcases project except commit log ?

Is there a global thread for testcases project except commit log ?

There is no single Redmine task. We have many tasks related to this in Redmine.

Artur Școlnic wrote:

Eric, are the flushing/validation and locking tests, Greg mentioned, the ones from testcases/tests/persistence/flushing_validation or are there other tests that are not yet committed to testcases?

Those are not my test cases. I have neither used nor reviewed them, but they are likely fine.

I have a set of flush tests (~150) in the old testcases project, which you can use for inspiration, if you find them useful. None of these are multi-session, however.

Look in uast/flush/. There is not much in the way of instructions, but the starting point for batch mode is all-tests.p. This will log results to a file, but will require "clicking through" some error messages.

IIRC, this relies on the p2j_test schema of the old testcases project (at least the book table, possibly others), but it requires the installation of some trigger procedures (*-trig-proc.p) for logging.

Note that not all of the tests operated the same way as they did in OE at the time I was last using them. So, if you find differences, don't assume they are new regressions.

I'll continue looking for locking tests and any multi-session tests that look useful.

• Record creation
• Record creation index
• Record creation composite index
• Record creation mandatory field
• Record update
• Record update index
• Record update composite index
• Record update mandatory field

• Full transactions
• SubTransactions
• Rollback due to error
• Implicit transactions
• Implicit/Explicit validation
• Implicit/Explicit flushing

• Find first
• Find last
• Find where
• For each
• Dynamic query - for each

The final test suite is a result of combining the different variants of main and control sessions which results in 650 pairs of tests.

Alexandru and Eric
If something important is missing regarding just flushing and validation, please mention it.

Tomasz,
I would like to reuse some test classes in multiple test-sets, this means that the same socket will be used, currently I have an issue

IOException waiting for connection:Address already in use: bind

Artur Școlnic wrote:

Tomasz,
I would like to reuse some test classes in multiple test-sets, this means that the same socket will be used, currently I have an issue
[...]
Is there a way to use the same socket for all test sets?

I think so, but as TestSets are meant to be executed in parallel you need to make them run in sequence by using mutexes or -q 1 option or <test-plan parallel = "1">
Still - from what I see in a source code TestSet listening socket should be closed up to 100 ms after TestSet is completed, so you'd need to place a technical TestSet waiting for 1 second in between other TestSets - sorry but waiting for a server socket to close after TestSet is done is not implemented (I didn't think it through for all cases).

The connection is terminated after each test so I am not sure why the already is use error is thrown.

Do you mean client connection ? Server connection lasts for the lifetime of a TestSet, so new TestSet cannot reuse

The test sets are sequential, so it makes sense that a test set uses a socket, terminates the connection and it should be available for the next test set.

Harness TestSets are meant to be run in parallel by design, if you need to run them in sequence you need to make them run in sequence by using mutexes or -q 1 option or <test-plan parallel = "1">

ps. funny thing - I've just tried setting parallel = "1" and changed all the port in cross_session.xml and it all worked in sequence. So maybe the technical testset waiting for server socket to close is not needed actually.

I understand that individual tests are run in parallel, but the test sets are sequential, no?
I have a lot of scenarios like this:

   <test-set name="record_flushing_and_vaidation_test1" description="Cross session record validation/flushing tests." threads="2" >
      <target type="remote" port="8881" threads="2" />
      <test filename="support/harness/test/flushing_and_validation/TestFlushingValidationEquivalencyMain.xml" dependency="concurrent" group="xsession1" />
      <test filename="support/harness/test/flushing_and_validation/TestFlushingValidationEquivalencyControl.xml" dependency="concurrent" group="xsession1" />
   </test-set>

   <test-set name="record_flushing_and_vaidation_test2" description="Cross session record validation/flushing tests." threads="2" >
      <target type="remote" port="8881" threads="2" />
      <test filename="support/harness/test/flushing_and_validation/TestFlushingValidationEquivalencyMain.xml" dependency="concurrent" group="xsession1" />
      <test filename="support/harness/test/flushing_and_validation/TestFlushingValidationEquivalencyControlFindLast.xml" dependency="concurrent" group="xsession1" />
   </test-set>

Notice that the main session is the same class in both test sets and uses the same connection.

Basically I want to reuse test classes for more test sets.

Tomasz Domin wrote:

ps. funny thing - I've just tried setting parallel = "1" and changed all the port in cross_session.xml and it all worked in sequence. So maybe the technical testset waiting for server socket to close is not needed actually.

Oh...I see, test sets are also run in parallel. Indeed, after setting parallel = "1", the tests are executing correctly.

There is one issue though, the html output is quite weird...maybe because I use the same test for multiple different test sets.

Artur Școlnic wrote:

I understand that individual tests are run in parallel, but the test sets are sequential, no?

TestSets are meant to be executed in parallel, unless synchronized.
Test are meant to be run in sequence except they dependency is concurrent for the given group="xsession1"
So you can probably put all Tests into single TestSet as long as you define a different group for each pair of tests.

Notice that the main session is the same class in both test sets and uses the same connection.
Basically I want to reuse test classes for more test sets.

Its perfectly ok.

Tomasz,
We need to mention in the wiki that with FWD, unit_test.xml must be created from unit_test.xml.template and correct ports need to be assigned:

      <server insecure_port="3433"/>
      <server secure_port="3333"/>

I encountered a problem with the multi session test results with FWD.
I have a main session test that modifies a record and a control test that reads the record before the main's transaction is ended.
Basically each Control test is seeing the record that the previous test should have seen, in other words, it is one transaction behind.
I am not sure if this is an issue in trunk or harness.
The test case:
session1

define buffer itemBuf for item.

for each itemBuf:
    delete itemBuf.
end.

do transaction:
create itemBuf.
itemBuf.itemNum = 0.
itemBuf.itemName = "item1".
release itemBuf.
end.

do transaction:

    find itemBuf.
    itemBuf.itemName = "item2".
    display itemBuf.itemName.
    READKEY.

end.
READKEY.

session2:

define buffer itemBuf2 for item.

find first itemBuf2 no-lock.

display itemBuf2.itemName. // item1
READKEY.
release itemBuf2.

find first itemBuf2 no-lock.
display itemBuf2.itemName. // item2

Because of this problem, all the tests are failing in FWD. It seems to be a synchronization issue, I need to investigate it.

I solved all the issues with flushing_and_validation tests, there were some hard limits with abl unit and nested explicit transactions, so I had to remove tests that used them, though we still have nested implicit transactions.
The total number of tests as reported by harness is 1147, half making changes and the other half checking them in another session.

I will move on to locking tests.

I have to mention an issue I was having in case anybody else encounters it.
If an abl unit class has all capital letter words in the name, for example SomeTEST1.cls, it will be converted to camel case SomeTest1.java and the junit engine will fail to locate the class, which usually is easy to debug, but with harness, this issue is not reported anywhere, instead build successful is displayed, which is quite misleading.

there were some hard limits with abl unit and nested explicit transactions

Please provide more details. What are the limits?

Is this in OE or in FWD?

In a simple program, constructs like:

do transaction:
   do transaction:
      ...
   end.
end.

** WARNING--TRANSACTION keyword given within actual transaction level. (214)

Artur Școlnic wrote:

From my investigation, this is not a true warning and cannot be suppressed with SESSION:SUPPRESS-WARNINGS, it is rather a compiler generated message that includes the WARNING keyword. Anyway, it stops the tests from executing correctly in 4gl, same as in FWD. Maybe it can be solved in FWD, but since the behavior is similar to the 4gl one, I left it alone.

Possible solution is moving subtransaction into a sub-procedure. But that would make tests harder to read, especially those simple ones.

Artur Școlnic wrote:

In a simple program, constructs like:
[...]
are perfectly legal and cause no trouble.
The same construct in an ABL unit test raises a warning:
[...]
From my investigation, this is not a true warning and cannot be suppressed with SESSION:SUPPRESS-WARNINGS, it is rather a compiler generated message that includes the WARNING keyword. Anyway, it stops the tests from executing correctly in 4gl, same as in FWD. Maybe it can be solved in FWD, but since the behavior is similar to the 4gl one, I left it alone.

Isnt each ABLUnit test case started within an implicit transaction already ?

Tomasz Domin wrote:

Possible solution is moving subtransaction into a sub-procedure. But that would make tests harder to read, especially those simple ones.

Yes, but I settled for just having an implicit subtransaction triggered by a for each that does assigns.

Tomasz Domin wrote:

Isnt each ABLUnit test case started within an implicit transaction already ?

AFAIK yes, but I just wanted to go one level deeper and was surprised to see that nested do transaction blocks fail.

Artur Școlnic wrote:

Tomasz Domin wrote:

Isnt each ABLUnit test case started within an implicit transaction already ?

AFAIK yes, but I just wanted to go one level deeper and was surprised to see that nested do transaction blocks fail.

Isnt that interfering with transaction testing ? I mean - we can never start/control full transaction with test code.

Maybe I am not explaining myself clear, this is what fails:

    @Test.
    method public void Test1():
        do transaction:
            do transaction:
                ...
            end.    
        end.
    end method.

Artur Școlnic wrote:

Maybe I am not explaining myself clear, this is what fails:
[...]
This doesn't even make much sense, it just a corner case from my tests, if you replace the inner do transaction with a procedure or a for block that start an implicit transaction, the warning is not raised.

I meant something different - if starting a test case implicitly starts transaction we can only test subtransactions. Compiler complains at the second level of nesting as it is not aware that the test method would be run within a higher level transaction. Isnt that like this:

    @Test.
    method public void Test1(): -> transaction starts - level 0
        do transaction:         -> subtransaction - level 1
            do transaction:     -> subtransaction - level 2
                ...
            end.    
        end.
    end method.

Tomasz Domin wrote:

I meant something different - if starting a test case implicitly starts transaction we can only test subtransactions.

Not really, each test method that updates the database starts an implicit transaction, if we do not open another one implicit or explicit, then we are in a full transaction.
An example of what we can't test would be a for each that updates the database and we want to test it as a full transaction block, in abl unit it will always be at least level 1.

Artur Școlnic wrote:

Tomasz Domin wrote:

I meant something different - if starting a test case implicitly starts transaction we can only test subtransactions.

Not really, each test method that updates the database starts an implicit transaction, if we do not open another one implicit or explicit, then we are in a full transaction.
An example of what we can't test would be a for each that updates the database and we want to test it as a full transaction block, in abl unit it will always be at least level 1.

Dont we need tests running on level 0 ? Or test when we control when level 1 is entered ?
Honestly I dont know, just asking.

Tomasz Domin wrote:

Dont we need tests running on level 0 ? Or test when we control when level 1 is entered ?
Honestly I dont know, just asking.

Yes, we need to test full transactions as well as deeper level ones. But I don't see why we couldn't do it. Just don't start a transaction in the test method and you will have a level 0 transaction.
I am basing my conclusion on a the progress documentation about transactions, but it does not cover abl unit explicitly, I will try to understand from test if a method that updates the database implicitly starts a transaction before we panic further :)

Artur Școlnic wrote:

Tomasz Domin wrote:

Dont we need tests running on level 0 ? Or test when we control when level 1 is entered ?
Honestly I dont know, just asking.

Yes, we need to test full transactions as well as deeper level ones. But I don't see why we couldn't do it. Just don't start a transaction in the test method and you will have a level 0 transaction.
I am basing my conclusion on a the progress documentation about transactions, but it does not cover abl unit explicitly, I will try to understand from test if a method that updates the database implicitly starts a transaction before we panic further :)

In case transaction is started implicitly you wont be able to compare data before and after transaction commit/rollback, would you ?

    @Test.
    method public void Test1(): ??LVL0
        create itemBuf. // LVL0
        assign itemBuf.itemName = "item0" 
            itemBuf.itemNum = 1.              
    end.   

    @Test.
    method public void Test2(): // No transaction
        find first itemBuf.
    end.    

    @Test.
    method public void Test3(): // LVL0
        do transaction: // LVL0
             find first itemBuf.
        end.  
    end.

    @Test.
    method public void Test4(): //LVL0
        for each itemBuf: // LVL0
            itemBuf.itemName = "item1".
        end.  
    end.

    @Test.
    method public void Test5():// LVL0
        do transaction: //LVL0
            for each itemBuf: //LVL1
                itemBuf.itemName = "item2".
            end.  
        end.  
    end.

    @Test.
    method public void Test6(): //LVL0
        find first itemBuf. 
        itemBuf.price = 1.2. // LVL0
        do transaction: // LVL1 - 214 warning
            itemBuf.itemName = "item2".
        end.  
    end.

    @Test.
    method public void Test7(): //LVL0
        find first itemBuf.
        itemBuf.price = 1.2. //LVL0
        for each itemBuf: // LVL1 no warning
            delete itemBuf.
        end.
    end.
    @Test.
    method public void Test8(): //LVL0
        find first itemBuf.
        itemBuf.price = 1.2. //LVL0
        create mandBuf.
        for each itemBuf: //LVL1
             for each mandBuf exclusive-lock: //LVL1
                mandBuf.id = 1.
            end.
        end.
    end.

[25/10/31@00:56:53.516-1000] P-004500 T-004488 3 4GL 4GLTRANS       BEGIN TRANS 1115 [Test1 Transactions @ 29]
[25/10/31@00:56:53.516-1000] P-004500 T-004488 3 4GL 4GLTRANS       END TRANS 1115 [Test1 Transactions @ 33]
[25/10/31@00:56:53.516-1000] P-004500 T-004488 3 4GL 4GLTRANS       BEGIN TRANS 1178 [Test3 Transactions @ 43]
[25/10/31@00:56:53.516-1000] P-004500 T-004488 3 4GL 4GLTRANS       END TRANS 1178 [Test3 Transactions @ 45]
[25/10/31@00:56:53.516-1000] P-004500 T-004488 3 4GL 4GLTRANS       BEGIN TRANS 1210 [Test4 Transactions @ 50]
[25/10/31@00:56:53.516-1000] P-004500 T-004488 3 4GL 4GLTRANS       END TRANS 1210 [Test4 Transactions @ 52]
[25/10/31@00:56:53.517-1000] P-004500 T-004488 3 4GL 4GLTRANS       BEGIN TRANS 1242 [Test5 Transactions @ 57]
[25/10/31@00:56:53.517-1000] P-004500 T-004488 3 4GL 4GLTRANS       BEGIN SUB-TRANS 1243 [Test5 Transactions @ 58]
[25/10/31@00:56:53.517-1000] P-004500 T-004488 3 4GL 4GLTRANS       END SUB-TRANS 1243 [Test5 Transactions @ 60]
[25/10/31@00:56:53.517-1000] P-004500 T-004488 3 4GL 4GLTRANS       END TRANS 1242 [Test5 Transactions @ 61]
[25/10/31@00:56:53.517-1000] P-004500 T-004488 3 4GL 4GLTRANS       BEGIN TRANS 1274 [Test6 Transactions @ 65]
[25/10/31@00:56:53.517-1000] P-004500 T-004488 3 4GL 4GLTRANS       BEGIN SUB-TRANS 1275 [Test6 Transactions @ 68]
[25/10/31@00:56:53.517-1000] P-004500 T-004488 3 4GL 4GLTRANS       END SUB-TRANS 1275 [Test6 Transactions @ 70]
[25/10/31@00:56:53.517-1000] P-004500 T-004488 3 4GL 4GLTRANS       END TRANS 1274 [Test6 Transactions @ 71]
[25/10/31@00:56:53.517-1000] P-004500 T-004488 3 4GL 4GLTRANS       BEGIN TRANS 1306 [Test7 Transactions @ 74]
[25/10/31@00:56:53.517-1000] P-004500 T-004488 3 4GL 4GLTRANS       BEGIN SUB-TRANS 1307 [Test7 Transactions @ 77]
[25/10/31@00:56:53.517-1000] P-004500 T-004488 3 4GL 4GLTRANS       END SUB-TRANS 1307 [Test7 Transactions @ 79]
[25/10/31@00:56:53.517-1000] P-004500 T-004488 3 4GL 4GLTRANS       END TRANS 1306 [Test7 Transactions @ 80]
[25/10/31@01:10:09.234-1000] P-003816 T-004156 3 4GL 4GLTRANS       BEGIN TRANS 1077 [Test8 Transactions @ 75]
[25/10/31@01:10:09.234-1000] P-003816 T-004156 3 4GL 4GLTRANS       BEGIN SUB-TRANS 1079 [Test8 Transactions @ 80]
[25/10/31@01:10:09.234-1000] P-003816 T-004156 3 4GL 4GLTRANS       END SUB-TRANS 1079 [Test8 Transactions @ 82]
[25/10/31@01:10:09.234-1000] P-003816 T-004156 3 4GL 4GLTRANS       BEGIN SUB-TRANS 1080 [Test8 Transactions @ 80]
[25/10/31@01:10:09.234-1000] P-003816 T-004156 3 4GL 4GLTRANS       END SUB-TRANS 1080 [Test8 Transactions @ 82]
[25/10/31@01:10:09.234-1000] P-003816 T-004156 3 4GL 4GLTRANS       BEGIN SUB-TRANS 1081 [Test8 Transactions @ 80]
[25/10/31@01:10:09.234-1000] P-003816 T-004156 3 4GL 4GLTRANS       END SUB-TRANS 1081 [Test8 Transactions @ 82]
[25/10/31@01:10:09.234-1000] P-003816 T-004156 3 4GL 4GLTRANS       END TRANS 1077 [Test8 Transactions @ 84]

It seems the rules are:
1. If a test does database operations committed or not, a transaction is opened implicitly.
2. I another block starts a transaction, implicit or explicit, it does not start another transaction, as long as before it, there were no database operations (another transaction started)
3. You can have 'warning free' sub-transactions only by having implicit nested transactions.

Test6 raises the 214 warning because the itemBuf.price = 1.2. triggered a full transaction at the method scope, then I opened an explicit one with do transaction, the transactions are opened, but the warning is messing up the tests in FWD.
If we replace the do transaction with a for each, a sub-transaction is started without any warnings.
In Test8 I would expect a lvl2 transaction, but from the logs it seems that two lvl1 transactions are opened (the inner for each iterates twice).

Sorry, I still need to catch up here. Are you saying that in OE, the ABLUnit test code in #10664-86 fails to compile?

If so, is that a general failure (even outside of ABLUnit)? If so, then it is OK for us to change the 4GL code to be compilable in OE.

We must be able to fully test transaction processing. I would not expect ABLUnit to have any special transaction behavior inside OE itself, but it is possible that ABLUnit does implement some level of implicit transactions inside the 4GL code part of the implementation (which we replaced with a hand-coded Java version). We only want to add that behavior to the degree that it exists in OE.

In OE (and for FWD). external procedures and other top level blocks default to a sub-transaction. This gets combined with the behavior (in both OE and FWD) where database EDITS (but not temp-table edits) cause a full transaction to start. That means that you can get a full transaction at those levels at compile time depending on the code that is directly inside those blocks.

You can see this logic in rules/annotations/block_properties.rules where we set the level = prog.transaction. For example:

      <!-- database updates referencing a table node (or a buffer that is backed by a table) -->
      <!-- Special note for the KW_BUF_COPY check below: database_general.rules has rearranged
           the tree, such that both the source and destination buffer reside under the KW_TO
           node. This is done to make the method chaining and BufferCopy c'tor creation easier
           downstream. Hence, the "this.indexPos == 1" check to identify the destination buffer
           node. -->
      <rule>
         (type == prog.table or
          (type == prog.buffer and 
           getNoteLong("recordtype") == prog.table)) and
         (parent.type == prog.kw_assign                  or
          parent.type == prog.kw_create                  or
          parent.type == prog.kw_delete                  or
          parent.type == prog.kw_insert                  or
          parent.type == prog.kw_set                     or
          parent.type == prog.kw_update                  or
          (this.indexPos == 1        and 
           parent.type == prog.kw_to and
           parent.parent.type == prog.kw_buf_copy)       or
          (parent.type == prog.kw_raw_tran and
           leftSiblings == 1)                            or
          (parent.type == prog.kw_buffer          and
           parent.parent.type == prog.kw_raw_tran and
           parent.parent.getChildAt(1).id == parent.id))

         <action>reasons = execLib("reasons_set_table_update", reasons)</action>
         <action>level = prog.transaction</action>
      </rule>

The bottom line here: we must match the OE and ABLUnit behavior exactly. Please make sure to document any findings explicitly and in detail before we make any sweeping changes.

If our implementation of compatibility at the 4GL level leads to some limits in how we test transaction processing, then we should discuss how we get around those limits. It is possible that some test code might be needed outside of ABLUnit in that case.

Greg Shah wrote:

Sorry, I still need to catch up here. Are you saying that in OE, the ABLUnit test code in #10664-86 fails to compile?

No, it compiles, but raises ** WARNING--TRANSACTION keyword given within actual transaction level. (214). The tests are still executing, but if you have 100 tests raising this warning, you will have to click 100 dialog boxes with the warning in order to get rid of them (in gui).
The behavior in FWD is the same.
The problem is that this warning cannot be suppressed, so I just deleted all the tests with this construct and will no longer use it. There are other ways to trigger a sub-transaction.

There is no issue in FWD, just a 4gl inconvenience (which we replicate correctly).

Artur Școlnic wrote:

It seems the rules are:
1. If a test does database operations committed or not, a transaction is opened implicitly.
2. I another block starts a transaction, implicit or explicit, it does not start another transaction, as long as before it, there were no database operations (another transaction started)
3. You can have 'warning free' sub-transactions only by having implicit nested transactions.

These seem like normal rules for 4GL block where transactions are created implicitly if needed. So it seems ABLUnit does not add anything by itself.

• Lock type: no-lock, share-lock, exclusive-lock / no-wait
• Query type: find/first/last/where, for each/first/last/where, do preselect, dynamic.
• Release: explicit, on error, on transaction end, downgrade

AFAIK we should match the locking system to the OE one, so the results in OE will be considered the correct ones.
Test results
OE: 1232/1232, all pass.
FWD trunk: 1056/1232, 88 fail, 88 not run.

It seems like the failures are in a single test class that uses exclusive locks, the tests fail to find released records.

The ones that do not run are also in a single class, will investigate.

The tests are not checked in testcases, but I uploaded them to 10664a root directory for preliminary reviews. I figured it's better to commit them to testcases when all are done and approved.

Eric,
For review purposes, the actual test classes are in flushing_and_validation and locking directories in the root of 10664a.

The tests that are not running properly have this pattern:

find first itemBuf no-lock.
            itemBuf.itemName = itemBuf.itemName + "". //used just to raise the error
            CATCH e AS Progress.Lang.SysError:
               SignalControl().
               Assert:Equals(396, e:GetMessageNum(1)).
            END CATCH.

The test relies on the 396 error being caught to send a signal to the other session, this does not happen with FWD.
In 4gl, a no-lock record cannot be updated regardless of the value that is assigned.
In FWD, we are trying to get a lock on the record only if the assigned value is different than the one already in the buffer.

I will change the tests to assign a different value.

Greg, let me know if this issue deserves our further attention.

Artur Școlnic wrote:

I will change the tests to assign a different value.

After this and ignoring the error number (the wrong error was reported by FWD), all no-lock tests pass.

I computed some topics to look into based on recent work we've done for customers on High/Urgent priority. This list is not properly structured and can duplicate some topics you already considered. Please ingest this list and prioritize accordingly. I added Razvan and Andrei due to their work in this area. They have experience within customer application / clustering work and can confirm which areas are still fragile in FWD from the POV of multi-session scenarios.

Hints¶

• consider that __Lock is a meta table that can help identifyin what locks are set
• assert that __Lock metadata has different kind of locks set for different tests.

• 4GL is able to see record changes that were COMMITTED by other sessions like (just like PG)
• any test you find with changing/creating/deleting data for intra-session cases can be transformed so that the change/create/delete is done by another session withing a full transaction.
• while the logic for intra-session is clear as most of state resides in memory; the cross-session cases are a bit more complex (related to READ_COMMITTED isolation).
  • create per-se has a quirk that is disabled in FWD, so can be omitted from tests.

Topics¶

• some caches are cross-session (DynamicQueryHelper or FastFindCache)
• we need strong tests to understand whether they are properly synchronized and they do not leak.
• this can be stress tested.
• these caches shouldn't retain session specific data (i.e. FFC retaining memptr or other context local data).

• these are tests in /xfer/testcases/abl/tests/query/adaptive and explore query invalidation
• these can be suited for cross-session testing; running multiple such tests at once on persistent databases should not leak data.
• this can be stress tested.

• scrolling queries are usually used for browses to "cache" the records already iterated.
• committing on another transaction a change can change the order of a record, so that a single record can occur twice: once in the cursor (scrolling query cache) and once while still iterating.
• running off-end with such query will basically mark it as a full-set, so changes in other transactions won't be seen anymore

• preselect queries are only by id in 4GL. This means that data can change, but this query won't ever see new records.
• deleted records are skipped, so if another sessions commits the delete of a record, the preselect shouldn't see it anymore.
• check SKIP-DELETED-RECORD

• There were issues that involved killing a client (either through admin console or by raising a STOP)
• Sometimes, these can be caused by OOME.
• This made some locks to leak and records never accessbile.
• Arbitrary STOPs should be raised in tests just to check that locks do not leak

• This is something I did not explore in-depth, but I think there might be things that should be theoretically synchronized.
• The OO instances per-se can't be shared by multiple sessions, but maybe they happen to share some internals (like static members, pools, 3rd party resources, etc.)

• I am not sure if this is a proper topic, but browses in FWD can filter and sort through non-conventional 4GL flow
• I wonder if these "backdooors" into browses and data access is properly synchronized.

• These are usually tricky

• This shall be tested

• This shall be tested

• This shall be tested

Recent tasks¶

• Topic: Browse locks
• See #10000-22
• Premises: make a query with exclusive-lock and attach to browse.
• What to test: The exclusive locks shouldn't take action when displaying rows; browse uses no-lock by default.
• How to test: create a button that does GET CURRENT. This will force an exclusive-lock. Check __Lock to see that.

• Topic: Locking per component vs locking per GET
• See: #10000-22
• Premises: OPEN QUERY q FOR EACH tt EXCLUSIVE-LOCK, EACH tt2 NO-LOCK is a valid syntax, but at the same time GET NEXT q SHARE-LOCK is also correct.
• What to test: GET NEXT lock (I think) is more important than the query-level lock. Not specifying a lock will use the query-level lock per-component.
• How to test: check __Lock to see different locks with GET lock vs query level lock.

• Topic: Data display discrepany between CHUI CounterAct and GUI CounterAct
• See: #10038-74
• Premises: DmoVersioning is storing the shared version for each DMO that exists in-memory. If a DMO is no longer residing in memory, the version resets to 0.
• What to test: One session bring a record in memory and finishes a transaction and closes the session (the DMO is just loaded in buffer, not active). Another session get the record and updates it. The first session will continue execution, but the version will still be 0, so it will not refetch - it should.
• How to test: extend #10038-74 to explore DmoVersioning.
• Extra: We made a fix in #10038 that is not perfect from the performance POV, so we will need to wok on this more - so regression tests are required.

• Topic: the recid is meant to be unique across sessions; this is usually enforced by FWD sequences.
• See: #10094
• in 4GL, reclaiming is possible, but FWD doesn't allow it.
• I suggest making some tests to confirm the uniqueness of recid across sessions holds. This is especially important for multi-tenant environments.

Other tasks¶

• Something related with EDITING block and locks.

• There are plenty of tests descibed here, maybe some are worth to be investigated in multi-session environments

• This task is for a unit test that changed indexes for FIND. The intra-session topic was fixed, but committing the change from another session should also be considered.

+ Ovidiu

Razvan pointed out an issue with lock scopes in #10796, locking test suite already includes lock release/downgrade scope checks, I will also include lock acquiring scope checks.

Artur Școlnic wrote:

The tests that are not running properly have this pattern:
[...]

The test relies on the 396 error being caught to send a signal to the other session, this does not happen with FWD.
In 4gl, a no-lock record cannot be updated regardless of the value that is assigned.
In FWD, we are trying to get a lock on the record only if the assigned value is different than the one already in the buffer.

I will change the tests to assign a different value.

Greg, let me know if this issue deserves our further attention.

Yes, we should fix this.

Alex/Eric can determine if we do this now or if we do this in a separate task.

Alex/Eric can determine if we do this now or if we do this in a separate task.

IMHO, related tasks should be opened.

I agree, I will open related tasks for every issue found during testing.

Artur Școlnic wrote:

I finished writing the locking tests, the results are:
OE: 1610/1610
FWD: 411/1610
There are two main reasons why there are so few passing tests with FWD:

• With multi session tests, if there are failures in the control session, the communication between the sessions is corrupted and in many cases the results from the entire test class are skipped.
• Due to the conversational nature of the tests, if one of the test method fails in a test class, all the following test methods will fail, this is why there are so many failures in

I guess this can be solved by making tests more granular - like spliting each test into separate files (including setup/teardown) ? The problem is we'd end with 1610 source files which need to be supported.

We know there are failures, hence the skewed results, we simply need to work on the issues and bring the success rate to 100. If hundreds of tests are failing because of a single issues, it is not ideal from the point of view of knowing accurately where FWD is compared to OE, but it would be fairly easy to solve a lot of tests at once.
I don't like the idea of more granular tests, there are a lot of test files as it is, I tried to write them in such a way that they are reusable and no additional test files are needed for different tests (where it's possible).
For the future tests, I will try to come up with a way of not letting the failing assertions interfere with the session communication. Worst case, I will use only one assertion per test and send the signals before it.

Artur Școlnic wrote:

We know there are failures, hence the skewed results, we simply need to work on the issues and bring the success rate to 100. If hundreds of tests are failing because of a single issues, it is not ideal from the point of view of knowing accurately where FWD is compared to OE, but it would be fairly easy to solve a lot of tests at once.
I don't like the idea of more granular tests, there are a lot of test files as it is, I tried to write them in such a way that they are reusable and no additional test files are needed for different tests (where it's possible).
For the future tests, I will try to come up with a way of not letting the failing assertions interfere with the session communication. Worst case, I will use only one assertion per test and send the signals before it.

I fully agree, I was just wondering if I can help with tooling modification.

Tomasz Domin wrote:

I fully agree, I was just wondering if I can help with tooling modification.

Since you brought that up...

1. If a class test fails at some point and the communication is disrupted, the results for the entire class are skipped due to time out ('Build success' is not printed), is there a way to still report the passing tests?

2. There are a lot of situations where ant fails to find/compile/execute tests, but harness does not report the exact issue.

I can help, if you'd like, so it doesn't look like I am just demanding stuff :)

Artur Școlnic wrote:

Tomasz Domin wrote:

I fully agree, I was just wondering if I can help with tooling modification.

Since you brought that up...

1. If a class test fails at some point and the communication is disrupted, the results for the entire class are skipped due to time out ('Build success' is not printed), is there a way to still report the passing tests?

'Build success' is a string harness waits for to confirm ant finished tests execution. I guess we'd need to modify both build.xml scripts to print a customized message harness would wait for. Actually I was trying to find a way to report upstream ant finished its work like ant ; echo "ANT_DONE", but I could not find an unified way to do it in both windows CMD and Linux bash.

2. There are a lot of situations where ant fails to find/compile/execute tests, but harness does not report the exact issue.

The interface between ant and harness is very simple - its ascii stream based only. Maybe a dedicated harness task to interface ant would help - but that needs to be developed.