FWD » Testing

When we can write code in the form of a unit test, we want to do that. There are 3 unit test frameworks that I know of for the 4GL. OEUnit (used by a large customer), ABLUnit (used by most customers, including a customer with a large GUI application) and something called ProUnit (It is pretty old and we don't have any current customers that use it.)

We have implemented our own compatibility layer that provides OEUnit and ABLUnit support, running as a pluggable test engine in JUnit5. The core support is common code and you configure the conversion to tell it whether you are converting as OEUnit or ABLUnit. From there everything else is transparent. We use ABLUnit mode for all of our Testcases.

When OE executes 4GL code, it is always single-threaded (even when in PASOE). So OEUnit and ABLUnit have no concept of multi-threading or executing in multiple sessions. There is always just a single session and multiple tests get executed in sequence, never in parallel.

But we do have real scenarios in which there are multi-user/multi-session requirements:

• cross-session dirty share
• validation/flushing/trigger testcases
• record locking tests
• PASOE

In these cases, we will use the Harness to implement the multi-threaded testing. It provides robust support for implementing tests that can execute on multiple threads. We use it for one customer's REST application smoke testing, CTRL-C testing and ChUI Regression Testing today.

Both CTRL-C and ChUI regression testing are heavily multi-user and severely threaded. They work by building a queue of jobs, each job is a test. Each test can define dependencies on one another and can be written using synchronization primitives such as event or mutex semaphores. This makes it pretty easy to simulate multi-user workflows.

For the customer's smoke testing, we directly encode the REST calls into the Harness test and define a test as a sequence of multiple coordinated REST calls. Subsequent REST calls use data returned from prior calls as you would expect.

For the CTRL-C and ChUI regression testing, we are dealing more directly with a 4GL application. Nothing in the harness itself is specific to 4GL code or provides a direct mechanism to execute 4GL code. Instead, when needed, we start a FWD client session and then we manipulate it using TTY processing (send keys to the terminal and read/compare screens).

Although we could write interactive tests in ChUI for the above multi-session cases, I don't really want to do that. I think a better approach is to implement the individual tests as ABLUnit code and then provide some simple way to run an ABLUnit test and to read the result as XML or JSON output. I think there is even a standard for how JUnit results can be returned via XML. Our ABLUnit client is just a FWD client so we could run that the same way we run the FWD ChUI client. But if we add some light integration here, then we can pass off the results of the ABLUnit run and read it directly into the Harness, making the result much nicer (in my opinion).

I've long since asked Tomasz to implement some kind of prototype of this for cross-session dirty share testing. The idea was to use the same approach for the validation/flushing/triggers tests and I think we would do the same thing for the record locking and PASOE tests.

Making changes to the Harness to add such support would not be a big deal. Tomasz, Constantin and I have all done significant work on the Harness and would be involved with this discussion if not also with the implementation.

At first I was confused by the name, Bogdan who was looking into Harness before explained [[http://harness.io]] is not the one referred here :)

I guess we can use Harness to somehow orchestrate the execution of unit tests while other clients are holding records locks or making changes to the database, we will try to come up with a simple test and see how that goes.

The main assumption is to allow to run multi-session 4GL testcases in FWD as well as in OE environment.

The idea is to create a small Harness agent to allow tests orchestration and to expose it for ABLUnit testcases to use. It would be 4gl code for OE and Java code for FWD environments respectively.

• Operating system support - what operating system needs to be supported ?
• Are there examples of how to start and control a console process from harness - if its supported? I guess it was used during regression test baseline creation.

Marian - can you please provide source code for a sample test ? I guess you mentioned a locking test - one session locks a record and another checks if its locked. Both session are ABLUnit tests. We'd need to add code synchronizing with harness

I am investigating the way to collect and aggregate tests execution results.

Operating system support - what operating system needs to be supported ?

Both Windows and Linux. This is orthogonal with FWD/OE (it should run FWD and OE on both Windows and Linux).

Are there examples of how to start and control a console process from harness - if its supported? I guess it was used during regression test baseline creation.

Are you asking about how to run OE from the command line?

Greg Shah wrote:

Operating system support - what operating system needs to be supported ?

Both Windows and Linux. This is orthogonal with FWD/OE (it should run FWD and OE on both Windows and Linux).

I noticed #1547, so I guess it needs to be deeply tested under Windows.

Greg Shah wrote:

Are there examples of how to start and control a console process from harness - if its supported? I guess it was used during regression test baseline creation.

Are you asking about how to run OE from the command line?

No, I am asking if there are Harness examples of how to run OE from the command line. Otherwise I need to find it out and evaluate it myself - e.g. with System.exec, as until now I was only working with SSH transport.
And by support, I meant a dedicated support, System.exec is not a dedicated support for external tools integration :).

I think the OE launching needs to parallel how we launch FWD. To the degree that it is over ssh, this enables us to implement any of the ChUI directives (sending keys, reading screens).

To the same degree that we want OE launching to be local (for ABLUnit via ant), we will want the equivalent option for launching FWD locally (for ABLUnit via our client).

To say this a different way: we don't currently have any direct dependencies on FWD in the Harness. Whatever we do should be more generic and allow us to swap in OE or FWD as needed.

Greg Shah wrote:

I think the OE launching needs to parallel how we launch FWD. To the degree that it is over ssh, this enables us to implement any of the ChUI directives (sending keys, reading screens).

But what operating system limitations - like lack of SSH for Windows ?
Dont we need something like ShellTransport for Windows to be able to launch ABLUnit tests ?

I do think you can install OpenSSH in WSL but as I noted for ABLUnit, I think we should have a local launch option. There is no need for ChUI in that case.

A proposed way of using Harness test scenario to control ABLUnit test sessions.

• initiate ABLUnit test from Harness
• coordinate ABLUnit tests between themselves and Harness - my idea is to use Harness semaphores exported to ABLUnit tests
• process ABLUnit tests execution results

• setting up variables etc
• setting up semaphores to be used both by Harness tests and ABLUnit tests
• controlling order and parallel execution of single tests

• initiating single ABLUnit test execution by calling specified OS command (which would be configurable and system dependent) with ABLUnit test name as a parameter
• coordination between ABLUnit tests and between given ABLUnit test and Harness test using mutex semaphore and event semaphore elements
• checking for semaphore related timeouts
• importing tests results
• there would be one-to-one binding between Harness test element and ABLUnit test as we cant (probably) control order of ABLUnit test execution, unless order of test execution is not important (e.g. multiple locking tests on the same locked session)
• note there is an option to capture terminal output and process as well, but it would not be used for ABLUnit testing

1. ShellTransport that allows for running commands or scripts in local shell. I guess its needed for Windows which lacks remote shell over ssh.
2. API (socket ? webservice?) to allow ABLUnit tests to access Harness semaphores. Additional 4GL code needs to be created in order to let access Harness semaphores from 4GL code. API address needs to be passed to ABLUnit tests on startup as configuration variable
3. API (socket ? webservice?) to allow ABLUnit tests to report progress to Harness (plus 4GL code) - I guess its optional
4. import and integrate OE ABLUnit results.xml with Harness results
5. import and integrate FWD results.xml with Harness results
6. generate JUnit report out of Harness (#6265)

• remote terminal over SSH (Linux)
• local terminal (Linux/Windows)

Tomasz,

I might tend to over simplify things here but for concurent data access our needs are often quite simple, we need to lock/update/create records in one session and see what happens if we try to do the same on another session or see if the changes are visible or not (if yes, at which point exactly). For that we simply need to start a 4GL procedure that set the stage for the test and then execute the ABLUnit tests to assert the expectations.

If you guys really want to make this more chatty, have more test methods that run in a specific order and signal back Harness when they need to, maybe wait for answers from Harness we can certainly build a protocol - plain sockets will work both ways, soap/rest could be used from 4GL to ping back to Harness but there is no web-service in 4GL for Harness to call. If this is just to access semaphores in Harness then we already have a database available (both 4GL and FWD) so we can simply just use a table for 'semaphores'... it's true this will kinda break all together if persistence layer is completely broken, but then we will know :)

Using os command to start another process is something that we can do from the 4GL anyway, maybe easiest will be to just make a simple 4GL library that starts a 4GL client (with FWD specific code) and leave Harness for other use cases but this sounds like a missed opportunity to build something complicated that sounds like a lot of fun :)

Another easier option for us will be to simply use the 'application server', this is something that can be easily called asynchronous so we can use the server agent as another session - the drawback is we will rely on the FWD application server functionality to work but then we do have to use something even if that might broke at times.

Marian, those are my ideas based on conversation we had on Wednesday and here is my understanding of things:

Marian Edu wrote:

I might tend to over simplify things here but for concurent data access our needs are often quite simple, we need to lock/update/create records in one session and see what happens if we try to do the same on another session or see if the changes are visible or not (if yes, at which point exactly). For that we simply need to start a 4GL procedure that set the stage for the test and then execute the ABLUnit tests to assert the expectations.

You need to synchronize both/all (there can be more then two) sessions somehow and using 4GL mechanisms for that is not a way, as we want to test them in most cases :)

If you guys really want to make this more chatty, have more test methods that run in a specific order and signal back Harness when they need to, maybe wait for answers from Harness we can certainly build a protocol - plain sockets will work both ways, soap/rest could be used from 4GL to ping back to Harness but there is no web-service in 4GL for Harness to call. If this is just to access semaphores in Harness then we already have a database available (both 4GL and FWD) so we can simply just use a table for 'semaphores'... it's true this will kinda break all together if persistence layer is completely broken, but then we will know :)

Sockets is something that we trust that works, persistence is something we want to test :) and keep stable - so it cannot be used.

Another easier option for us will be to simply use the 'application server', this is something that can be easily called asynchronous so we can use the server agent as another session - the drawback is we will rely on the FWD application server functionality to work but then we do have to use something even if that might broke at times.

Application server is complex has its bugs. I guess idea of using Harness is to use a tool we have control of and we can trust (and fix it anytime if needed).

Tomasz Domin wrote:

Marian, those are my ideas based on conversation we had on Wednesday and here is my understanding of things:

I know, this was what was discussed... I didn't questioned your understanding :)

You need to synchronize both/all (there can be more then two) sessions somehow and using 4GL mechanisms for that is not a way, as we want to test them in most cases :)
Sockets is something that we trust that works, persistence is something we want to test :) and keep stable - so it cannot be used.

Socket it is then, this is going to be fun so let's build something ;)

Application server is complex has its bugs. I guess idea of using Harness is to use a tool we have control of and we can trust (and fix it anytime if needed).

Understood.

At this point though I don't know how to help, I was kinda expecting starting processes to be something like a 'task' in Harness but it doesn't seems to be anything like that and it looks to me it is designed to run on another machine (ssh) since there start-up options for host/port/user.

If there is an option to start an OS process in Harness please let me know how that is done for a simple scenario I've mentioned - just two processes to start with :)
Otherwise guess we can use 'run code' or something and craft something for starting a process in Java but this sounds like something to be included in Harness if we are to go that path anyway.

Thanks

API (socket ? webservice?) to allow ABLUnit tests to access Harness semaphores. Additional 4GL code needs to be created in order to let access Harness semaphores from 4GL code. API address needs to be passed to ABLUnit tests on startup as configuration variable

When the session is started, both OE and FWD provide a option that can be passed with custom content. In OE this is a command line parameter (-param), in FWD we call this a "startup parameter" and it can be provided via command line (client:cmd-line-option:parameter="some awesome content here", bootstrap cfg (category "client", group "cmd-line-option", key "parameter") or directory (in the runtime section under @parameter).

In the 4GL code, you would reference that string using SESSION:PARAMETER. Then you would parse out whatever content you pass, presumably an IP address and port number for the specific listener for that thread's semaphores.

API (socket ? webservice?) to allow ABLUnit tests to report progress to Harness (plus 4GL code) - I guess its optional

I don't think this is needed if we can report the results at the end.

I might tend to over simplify things here but for concurent data access our needs are often quite simple, we need to lock/update/create records in one session and see what happens if we try to do the same on another session or see if the changes are visible or not (if yes, at which point exactly). For that we simply need to start a 4GL procedure that set the stage for the test and then execute the ABLUnit tests to assert the expectations.

I'm confident this is not enough for many of the use cases we are trying to test. We've encoded these before in the Harness and having some semaphore mechanism was critical.

Even in your simple lock example, your approach (without some kind of signaling mechanism) would depend on something like a long PAUSE, which is a very fragile mechanism. On a heavily loaded system, you would have to make the pause long to make it reliable. It is not a nice way to write such tests. With mutex and event semaphores, we can make this coordination quite straightforward and we only will have timeouts if a test fails (e.g. a condition for which you await is never raised, so a timer pops and the test fails).

At this point though I don't know how to help, I was kinda expecting starting processes to be something like a 'task' in Harness but it doesn't seems to be anything like that and it looks to me it is designed to run on another machine (ssh) since there start-up options for host/port/user.

I agree that Tomasz has to do some development to enable the basic concept. But you'll be writing the 4GL sockets code, so you can design the protocol with Tomasz now (and probably even implement both sides of it, including in Java). The core concept is to deliver bidirectional event and mutex semaphore notifications such that the core semaphore processing is available in the 4GL code and integrates completely with the Harness semaphores.

Greg Shah wrote:

At this point though I don't know how to help, I was kinda expecting starting processes to be something like a 'task' in Harness but it doesn't seems to be anything like that and it looks to me it is designed to run on another machine (ssh) since there start-up options for host/port/user.

I agree that Tomasz has to do some development to enable the basic concept. But you'll be writing the 4GL sockets code, so you can design the protocol with Tomasz now (and probably even implement both sides of it, including in Java). The core concept is to deliver bidirectional event and mutex semaphore notifications such that the core semaphore processing is available in the 4GL code and integrates completely with the Harness semaphores.

• ABLUnit test would be client, Harness would be server
• Every command is a request (from ABLUnit test) with two possible responses OK or ERROR from Harness server
• semaphores are identified by name as defined in Harness Testcase
• executing command may block ABLUnit test execution e.g. resource is assigned
• disconnection from server should be treated as an unconditional error
• Following commands with one parameter which is a name of resource needs to be supported:
  • POSTES <name> - Post event semaphore
  • RESETES <name> - Reset Event Semaphore
  • WAITES <name> - Wait Event Semaphore - this may block
  • ACQUIREMTX <name> - Acquire mutex - this may block
  • RELEASEMTX <name> - Release mutex

Created task branch 9270a.

I'd need to add a command to read test report from file to Harness.
But lets start from ShellTransport and the server.

In FWD the usage of ABLUnit would normally "leverage" JUnit's reporting capabilities, depending on the launcher used. For shell execution, we use org.junit.platform.console.ConsoleLauncher.

Vladimir: If we want to output the results into a standardized XML or JSON format, do we need to do that ourselves or can we leverage JUnit to do it? I seem to recall that we might have to write another plug-in to do this.

Marian: Is there any equivalent to this in ABLUnit when run in OE? Some output artifact in XML or JSON that we can easily parse?

Greg Shah wrote:

Marian: Is there any equivalent to this in ABLUnit when run in OE? Some output artifact in XML or JSON that we can easily parse?

I've tested that yesterday and ABLUnit in OE produces results.xml which seems to be a normal JUnit XML file.

Perfect! If we can generate the same in FWD, then that will be better than having to build a custom back channel via socket.

Greg Shah wrote:

Perfect! If we can generate the same in FWD, then that will be better than having to build a custom back channel via socket.

In order to keep things simple I was rather considering loading from local filesystem (for ShellTransport) or scp plus loading from local filesystem for SSH.
There would be a dedicated Harness command to do that.

Tomasz Domin wrote:

Greg Shah wrote:

Perfect! If we can generate the same in FWD, then that will be better than having to build a custom back channel via socket.

In order to keep things simple I was rather considering loading from local filesystem (for ShellTransport) or scp plus loading from local filesystem for SSH.
There would be a dedicated Harness command to do that.

I don't think that would work. These "commands" can only be used from 4GL code since we are using 4GL sockets. The results can't be reported while the 4GL code is still running. Why not let ABLUnit complete and then when it returns we can read the results.xml?

Greg Shah wrote:

I don't think that would work. These "commands" can only be used from 4GL code since we are using 4GL sockets. The results can't be reported while the 4GL code is still running. Why not let ABLUnit complete and then when it returns we can read the results.xml?

That is exactly what I meant. Let ABLUnit test runner complete and read results.xml afterwards into Harness as a part of scenario - from filesystem or scp/filesystem.

Tomasz Domin wrote:

I think we should keep protocol as simple as possible - client-server architecture and readline based:

That could work but the problem is in 4GL we don't see the sockets like streams so readline isn't really a thing, we simply have to read bit by bit and check for 'new line'. I often prefer a message header - like type and body size if needed, so always read the header first (fix size) and then the body only if there is any.

• ABLUnit test would be client, Harness would be server

Agreed.

• Every command is a request (from ABLUnit test) with two possible responses OK or ERROR from Harness server
• semaphores are identified by name as defined in Harness Testcase
• executing command may block ABLUnit test execution e.g. resource is assigned
• disconnection from server should be treated as an unconditional error
• Following commands with one parameter which is a name of resource needs to be supported:
  • POSTES <name> - Post event semaphore
  • RESETES <name> - Reset Event Semaphore
  • WAITES <name> - Wait Event Semaphore - this may block
  • ACQUIREMTX <name> - Acquire mutex - this may block
  • RELEASEMTX <name> - Release mutex

Sounds good, let me know if we can go with the message header approach or you want readline one instead and I'll start crafting something on the 4GL side.

Greg Shah wrote:

Perfect! If we can generate the same in FWD, then that will be better than having to build a custom back channel via socket.

Greg, this is for sending back the content of 'result.xml' only right? Mind you this could be quite large depending on the number of tests executed.

I mean, otherwise the socket protocol to comunicate with Harness does stand right?

Marian Edu wrote:

Tomasz Domin wrote:

I think we should keep protocol as simple as possible - client-server architecture and readline based:

That could work but the problem is in 4GL we don't see the sockets like streams so readline isn't really a thing, we simply have to read bit by bit and check for 'new line'. I often prefer a message header - like type and body size if needed, so always read the header first (fix size) and then the body only if there is any.

Oh I see, so you'd like to have PASCAL strings in message.
We can Fix that by adding 6 digits at beginning of each message indicating a total message body size in bytes, no newlines in that case:like:
000002OK
000005ERROR
000021ERRORWith description
000011POSTEStest1
In case of desynchronization lets assume connection failure.

Just need to remember to flush at the at the end of each message.

Tomasz Domin wrote:

Oh I see, so you'd like to have PASCAL strings in message.
We can Fix that by adding 6 digits at beginning of each message indicating a total message body size in bytes, no newlines in that case:like:

Yep, that's about right only why 6 digits? I guess a 32-bit int will do just fine and I really like to have the message type in the header, most of the time we probably don't need an additional payload.

In case of desynchronization lets assume connection failure.

Totally.

Just need to remember to flush at the at the end of each message.

Yeah, sounds like a good idea only there isn't anything like that in 4GL... only thing we can do is to try to set the send/receive buffer size. In any case you know the payload size, so just make sure you block till you get the complete message. We can add an 'ACK' message if really needed.

Marian Edu wrote:

Tomasz Domin wrote:

Oh I see, so you'd like to have PASCAL strings in message.
We can Fix that by adding 6 digits at beginning of each message indicating a total message body size in bytes, no newlines in that case:like:

Yep, that's about right only why 6 digits? I guess a 32-bit int will do just fine and I really like to have the message type in the header, most of the time we probably don't need an additional payload.

I refrain the protocol from becoming binary (in 32-int case we'd need to process value according to host endian).
6 digits are for cases when test execution reports would need to be sent over socket (which I dont like).

Just need to remember to flush at the at the end of each message.

Yeah, sounds like a good idea only there isn't anything like that in 4GL... only thing we can do is to try to set the send/receive buffer size. In any case you know the payload size, so just make sure you block till you get the complete message. We can add an 'ACK' message if really needed.

I guess header would be needed we we had to extend protocol in future.
You always have block size - its either 6 or specified buffer size.

Marian Edu wrote:

Greg Shah wrote:

Perfect! If we can generate the same in FWD, then that will be better than having to build a custom back channel via socket.

Greg, this is for sending back the content of 'result.xml' only right? Mind you this could be quite large depending on the number of tests executed.

I'm hoping we can avoid sending the results via socket. If we just generate the results.xml the same way that OE ABLUnit does, then there is no need to send it over the socket. The Harness can just read the file directly.

I mean, otherwise the socket protocol to comunicate with Harness does stand right?

Yes

Tomasz Domin wrote:

But lets start from ShellTransport and the server.

I've created #9284 to discuss how it should be implemented as I have encountered issues related to lack of TTY capabilities of java executed subprocesses.

Marian Edu wrote:

Yeah, sounds like a good idea only there isn't anything like that in 4GL... only thing we can do is to try to set the send/receive buffer size. In any case you know the payload size, so just make sure you block till you get the complete message. We can add an 'ACK' message if really needed.

I've created a dedicate Harness feature that allows synchronizing external processes with Harness mutexes and sempahores.
I had to modify protocol a bit to allow commands parameters.
For more please check #9285

• there are two new targets - one is a stream for controlling local subprocesses which starts bash and the second one is a remote TCP server exposed on port 8889 with up to 4 remote sessions
• there are two mutexes - servermutex is used to control FWD server, xsessionmutex is test internal mutex to allow tests to synchronize
• there are three tests that run in parallel

<?xml version="1.0"?>
<test-plan name="Cross session test example" description="Cross session test example." >
   <output-directory path="results" unique="true" />
   <target type="stream" shell="bash" />
   <target type="remote" port="8889" threads="4" />
   <resources>
      <create-mutex name="xsessionmutex" />
      <create-mutex name="servermutex" />
   </resources>
   <test-set name="start_tests_main" description="First cross session tests" threads="3" >
      <test filename="cross_session_001.xml" dependency="concurrent" group="xsession1" />
      <test filename="cross_session_002.xml" dependency="concurrent" group="xsession1" />
      <test filename="cross_session_003.xml" dependency="concurrent" group="xsession1" />
   </test-set>

</test-plan>

Session one which starts and stops FWD server:

<?xml version="1.0"?>
<test name="cross_session_001" description="Start and stop the server" >
    <acquire-mutex name="xsessionmutex"/>
<!--
    Start the server after tests
-->
    <acquire-mutex name="servermutex"/>
    <stream-write value="cd /home/tjd/projects/testcases/deploy/server" newline="true"/>
    <stream-write value="./server.sh &amp;" newline="true"/>
    <stream-write value="./server.sh -w" newline="true"/>
    <stream-wait-for value="Result = STATUS_RUNNING" millis="60000"/>
    <release-mutex name="servermutex"/>
<!--
    Release application level mutex
-->
    <release-mutex name="xsessionmutex"/>
<!--
    Stop the server after tests
-->
        <acquire-mutex name="servermutex"/>
    <stream-write value="./server.sh -k" newline="true"/>
    <stream-wait-for value="Result = STATUS_STOPPED" millis="30000"/>
    <release-mutex name="servermutex"/>
    <pause millis="10000"/>
</test>

<?xml version="1.0"?>
<test name="cross_session_002" description="Start junit tests" >
    <pause millis="5000"/>
<!--
    Wait for server to start
-->
    <acquire-mutex name="servermutex"/>
<!--
    Start unit testing
-->
    <stream-write value="cd /home/tjd/projects/testcases" newline="true"/>
    <stream-write value="./unittest.sh  --select-package com.goldencode.testcases.tests.ui_tests.combo_box.screen_value.tests --reports-dir=/home/tjd/projects/testcases/deploy/client-chui" newline="true"/>
    <stream-wait-for value="Test run finished"  millis="30000"/>
    <release-mutex name="servermutex"/>
<!--
    Import JUnit report into harness
-->
    <junit-import filename="/home/tjd/projects/testcases/deploy/client-chui/TEST-fwd-junit.xml" testset="start_tests_main"/>
<!--
    Some synchronizatin with external application
-->
    <acquire-mutex name="xsessionmutex"/>
    <pause millis="10000"/>
    <release-mutex name="xsessionmutex"/>
    <pause millis="10000"/>
</test>

Session three is emulation of external application, it locks and releases xsessionmutex mutex with Harness TCP Socket API using netcat:

<?xml version="1.0"?>
<test name="cross_session_003" description="Remote semaphore example" >
    <stream-write value="cd /home/tjd/projects/testcases" newline="true"/>
    <stream-write value="(for a in '000021TEST|cross_session_03' '000024ACQUIREMTX|xsessionmutex' '000024RELEASEMTX|xsessionmutex' ; do echo -n $a; sleep 1; done;) | netcat -N localhost 8889" newline="true"/>
    <stream-wait-for value="OK"  millis="30000"/>
    <stream-wait-for value="OK"  millis="30000"/>
    <stream-wait-for value="OK"  millis="30000"/>
</test>

I have a question on initiating unit tests process for both - FWD and 4GL ABLUnit.
Its possible with command line (e.g. prowin32 -p ABLUnitCore.p -param "D:\tjd\testcases\tests\stop_handling\TestSuite.cls -outputLocation d:\tjd" or via ant.

Which should I focus on - note that Harness will just start a command line process and does not really care what process it will be as long as it produces JUnit5 legacy format report. But my work effort related with example implementation depends on this :)

Running with ant requires having ant toolset locally - I guess its present for 4GL installation anyway, isnt it ?
I can see that testcases project has a dedicated ant file for 4GL ABLUnit - not sure if it is complete, but at least it shows the way.

Tomasz Domin wrote:

I have a question on initiating unit tests process for both - FWD and 4GL ABLUnit.
Its possible with command line (e.g. prowin32 -p ABLUnitCore.p -param "D:\tjd\testcases\tests\stop_handling\TestSuite.cls -outputLocation d:\tjd" or via ant.

Which should I focus on - note that Harness will just start a command line process and does not really care what process it will be as long as it produces JUnit5 legacy format report. But my work effort related with example implementation depends on this :)

I would say ant is safest, you don't need to know what that ant target is doing as long as you can start it from harness :)

Running with ant requires having ant toolset locally - I guess its present for 4GL installation anyway, isnt it ?
I can see that testcases project has a dedicated ant file for 4GL ABLUnit - not sure if it is complete, but at least it shows the way.

Yes, to start ABL Unit we use PCT (https://github.com/Riverside-Software/pct) which is just a collection of ant tasks and is shipped with Progress installation. I would imagine for FWD implementation the engine can be started from an ant task as well.

I've modified session 2 to make it work under Windows - actually it does not need to wait for server or servers need to be started with ant as well - its to be decided later, now I was testing if new Harness functionality is working fine under Windows, and it seems it does:

Following worked well under windows and OE, and test report result was imported into Harness afterwards:

<?xml version="1.0"?>
<test name="cross_session_002" description="Start junit tests" >
    <acquire-mutex name="servermutex"/>
<!--
    Start unit testing
-->
    <stream-write value="cd D:\tjd\testcases\" newline="true"/>
    <stream-write value="ant -Dtests.module=stop_handling ablunit-test" newline="true"/>
    <stream-wait-for value="Total time:"  millis="90000"/>
    <release-mutex name="servermutex"/>
    <junit-import filename="D:\tjd\testcases\results.xml" testset="start_tests_main"/>
    <pause millis="10000"/>
</test>

I've invoked Harness in following way:
java -jar D:\tjd\harness\build\lib\harness.jar -d -d -shell cmd.exe cross_session_test.xml

There is a new Harness option -shell that overrides shell defined for stream sessions, default is sh, so in case sh is not installed on windows cmd.exe needs to be used.

Now I will try to make FWD unit tests to work with ant as well.

Marian Edu wrote:

Yes, to start ABL Unit we use PCT (https://github.com/Riverside-Software/pct) which is just a collection of ant tasks and is shipped with Progress installation. I would imagine for FWD implementation the engine can be started from an ant task as well.

Greg suggested to use plain commands instead ant. But it ended with constructs like (they are both equivalent - they start server, start ABLUnit tests, stop servers) for a bit modified scripts from #9270-35

For linux:

java -jar build/lib/harness.jar -d -d -v serverDirectory=/home/tjd/projects/testcases/deploy/server -v startServerCommand="./start_server.sh; ./server.sh -w" -v stopServerCommand="./stop_server.sh" -v testDirectory="/home/tjd/projects/testcases" -v startTestCommand="./unittest.sh  --select-package com.goldencode.testcases.tests.stop_handling --reports-dir=/home/tjd/projects/testcases" -v resultsXml=TEST-fwd-junit.xml cross_session_test.xml 

For Windows (it starts ant anyway):

java -jar build/lib/harness.jar -shell cmd.exe -d -d -v serverDirectory="D:\tjd\testcases\db" -v startServerCommand="proserve tstcasesdb.db & echo Result ^= STATUS_RUNNING" -v stopServerCommand="proshut -by tstcasesdb.db & echo Result ^= STATUS_STOPPED" -v testDirectory="D:\tjd\testcases" -v startTestCommand="ant -Dtests.module=stop_handling ablunit-test & echo Test ^run finished" -v resultsXml="results.xml" cross_session_test.xml 

I guess having ant targets would be easier to read.

Marian - what is a status of implementation of Harness API protocol in ABL ?

Tomasz Domin wrote:

Marian - what is a status of implementation of Harness API protocol in ABL ?

I'm working on it, just a few things that I would like to clarify:
- port number is hardcoded: 8889
- sounds like you expect the connection to be stateful - aka keep the connection alive when locking a semaphore, or it can be a stateless request/response (as in http 1.0)?
- if we want to lock a semaphore and that is already locked - we wait for something, this means that the connection will block and the server won't respond at all until it manages to lock the resource or it will respond with `ERROR` and we have to use some long pool mechanism to keep on trying till it's unlocked?

I'm thinking to add some kind of time-out so the test fail if the expected scenario doesn't happen so I would rather use the async read on the socket (wait-for) with a pause for time-out, is this something expected to run correctly in FWD socket implementation or I should rather just go for blocking reads?

Marian Edu wrote:

Tomasz Domin wrote:

Marian - what is a status of implementation of Harness API protocol in ABL ?

I'm working on it, just a few things that I would like to clarify:
- port number is hardcoded: 8889

No, its configurable per TestSet (group of test scenarios). That allows for parallel running of multilple TestSets.

- sounds like you expect the connection to be stateful - aka keep the connection alive when locking a semaphore, or it can be a stateless request/response (as in http 1.0)?

Correct, its should be stateful, or I'd say the connection should be persistent for duration of a test.

- if we want to lock a semaphore and that is already locked - we wait for something, this means that the connection will block and the server won't respond at all until it manages to lock the resource or it will respond with `ERROR` and we have to use some long pool mechanism to keep on trying till it's unlocked?

Correct, in case semaphore is locked connection will block. You will get ERROR in case there is non-testing error like given named semaphore does not exist.

I'm thinking to add some kind of time-out so the test fail if the expected scenario doesn't happen so I would rather use the async read on the socket (wait-for) with a pause for time-out, is this something expected to run correctly in FWD socket implementation or I should rather just go for blocking reads?

Timeouts are managed by Harness, in the worst case blocking would last until TestPlan/TestSet finishes and socket is closed.
Closing a socket without a reply is an error.

Tomasz Domin wrote:

No, its configurable per TestSet (group of test scenarios). That allows for parallel running of multilple TestSets.

Right, and what are we going to use to find out that port number then?

If you start the ABLUnit tests with ant - as in `ant -Dtests.module=stop_handling` - I don't see any parameters being sent, is there an environment variable that we are supposed to use?

Another example scenario that works now for both - ABLUnit and FWD Unit testing , requires harness/9270a/57

• there are two new targets - one is a stream for controlling local subprocesses which starts bash and the second one is a remote TCP server exposed on port 8889 with up to 4 remote sessions
• there are two mutexes - servermutex is used to control FWD server, xsessionmutex is test internal mutex to allow tests to synchronize
• defines variables that allow per deployment customization
  

  <?xml version="1.0"?>
  <test-plan name="Cross session test example" description="Cross session test example." >
     <output-directory path="results" unique="true" />
     <target type="stream" shell="bash" />
     <target type="remote" port="8889" threads="4" />
     <resources>
        <variable name="serverDirectory" type="String" initial="" prompt="Directory with server" />
        <variable name="startServerCommand" type="String" initial="" prompt="Command line to start server" />
        <variable name="stopServerCommand" type="String" initial="" prompt="Command line to stop server" />
        <variable name="testDirectory" type="String" initial="" prompt="Directory with tests" />
        <variable name="startTestCommand" type="String" initial="" prompt="Command line to start tests" />
        <variable name="resultsXml" type="String" initial="" prompt="results.xml native file name" />
  
        <create-mutex name="xsessionmutex" />
        <create-mutex name="servermutex" />
     </resources>
     <test-set name="start_tests_main" description="First cross session tests" threads="3" >
        <test filename="cross_session_001.xml" dependency="concurrent" group="xsession1" />
        <test filename="cross_session_002.xml" dependency="concurrent" group="xsession1" />
     </test-set>
  
  </test-plan>
  

Session one which starts and stops FWD server:

<?xml version="1.0"?>
<test name="cross_session_001" description="Start and stop the server" >
<!--
    Start the application server
-->
    <acquire-mutex name="servermutex"/>
    <stream-write value="cd %s" newline="true">
        <substitution variable="serverDirectory" />
    </stream-write>    
    <stream-write value="%s" newline="true">
        <substitution variable="startServerCommand" />
    </stream-write>    
    <stream-wait-for value="Result = STATUS_RUNNING" millis="60000"/>
    <release-mutex name="servermutex"/>
<!--
    Stop the application server after tests
-->
    <acquire-mutex name="servermutex"/>
    <stream-write value="%s" newline="true">
        <substitution variable="stopServerCommand" />
    </stream-write>    
    <stream-wait-for value="Result = STATUS_STOPPED" millis="30000"/>
    <release-mutex name="servermutex"/>
</test>

Session two - waits for server to be started, executes ABL unit tests with JUnit, imports results into Harness

<?xml version="1.0"?>
<test name="cross_session_002" description="Start junit tests" >
<!--
    Make sure this thread starts as second
-->
    <pause millis="1000"/>
<!--
    Wait for server to start
-->
    <acquire-mutex name="servermutex"/>
<!--
    Start unit testing
-->
    <stream-write value="cd %s" newline="true">
        <substitution variable="testDirectory" />
    </stream-write>    
    <stream-write value="%s" newline="true">
        <substitution variable="startTestCommand" />
    </stream-write>    
<!--
    Wait 120 seconds for tests to finish
-->
    <stream-wait-for value="Test run finished"  millis="120000"/>
    <release-mutex name="servermutex"/>
    <junit-import filename="%s/%s" testset="start_tests_main">
        <substitution variable="testDirectory" />
        <substitution variable="resultsXml" />
    </junit-import>
    <acquire-mutex name="xsessionmutex"/>
    <pause millis="10000"/>
    <release-mutex name="xsessionmutex"/>
    <pause millis="10000"/>
</test>

The problem I had was per platform differences requiring number of variables to be provided on a command line:
FWD unit testing under Linux:

java -jar build/lib/harness.jar -d -d -v serverDirectory=/home/tjd/projects/testcases/deploy/server -v startServerCommand="./start_server.sh; ./server.sh -w" -v stopServerCommand="./server.sh -k" -v testDirectory="/home/tjd/projects/testcases" -v startTestCommand="./unittest.sh  --select-package com.goldencode.testcases.tests.stop_handling --reports-dir=/home/tjd/projects/testcases" -v resultsXml=TEST-fwd-junit.xml cross_session_test.xml

ABLUnit testing under Windows (required some tricks for StreamWait):

java -jar build/lib/harness.jar -shell cmd.exe -d -d -v serverDirectory="D:\tjd\testcases\db" -v startServerCommand="proserve tstcasesdb.db & echo Result ^= STATUS_RUNNING" -v stopServerCommand="proshut -by tstcasesdb.db & echo Result ^= STATUS_STOPPED" -v testDirectory="D:\tjd\testcases" -v startTestCommand="ant -Dtests.module=stop_handling ablunit-test & echo Test ^run finished" -v resultsXml="results.xml" cross_session_test.xml

Marian Edu wrote:

Tomasz Domin wrote:

No, its configurable per TestSet (group of test scenarios). That allows for parallel running of multilple TestSets.

Right, and what are we going to use to find out that port number then?

If you start the ABLUnit tests with ant - as in `ant -Dtests.module=stop_handling` - I don't see any parameters being sent, is there an environment variable that we are supposed to use?

To simplify things lets start from ports being static - it can be any port but agreed between ABLUnit tests and Harness, as I cant find a proper way to parametrize ABLUnit tests.

Tomasz Domin wrote:

Marian Edu wrote:

Tomasz Domin wrote:

No, its configurable per TestSet (group of test scenarios). That allows for parallel running of multilple TestSets.

Right, and what are we going to use to find out that port number then?

If you start the ABLUnit tests with ant - as in `ant -Dtests.module=stop_handling` - I don't see any parameters being sent, is there an environment variable that we are supposed to use?

To simplify things lets start from ports being static - it can be any port but agreed between ABLUnit tests and Harness, as I cant find a proper way to parametrize ABLUnit tests.

Can't we pass this encoded in a string value of the -param OE command line parameter? In FWD this is the startup parameter that can be passed as client:cmd-line-option:parameter="value".

Greg Shah wrote:

Can't we pass this encoded in a string value of the -param OE command line parameter? In FWD this is the startup parameter that can be passed as client:cmd-line-option:parameter="value".

Invoking ABLUnit already is using -param for its own purposes: prowin -p ABLUnitCore.p -param testprocedure.p. or prowin -p ABLUnitCore.p -param testclass1.testclass,testclass2.testclass
From what I see it does not allow to pass any parameters on to test classes/procedures.

I've ported dirty-share cross session tests.
But I have a problem with unification of invocation of tests on different platform - basically to make using cross-session tests easier.
For single tests I've defined a number of parameters see: #9270-44.

• for Linux - bash script that invokes JUnit - input parameter is Java test class name
• for windows - ant script that invokes PCT or prowin.exe directly - input parameter is ABL test class name

The most limiting part is windows part, as its a third party tool we just can use.
I dont want to include a dedicated support in harness for any of these method - I guess harness should stay generic.

Greg, despite you have proposed to use plain commands I guess one way is to extend ant PCT scripts to support Linux/FWD and include it with harness. (not sure if possible, as I've also discovered ant+junit do not work with FWD ABLUnit correctly - will check it out).

PCT is an Apache licensed project. But to do this we would have to fork PCT since it is currently designed as a project for OpenEdge and I think it is even delivered with OpenEdge. So the people that write it may not have any interest in supporting FWD or even if they had interest, Progress would not allow it. So let's stay away from PCT.

Please find a way to provide an abstraction in the Harness which the tests can be written to connect/setup on both Windows and Linux and the right one is used when needed.

Greg Shah wrote:

PCT is an Apache licensed project. But to do this we would have to fork PCT since it is currently designed as a project for OpenEdge and I think it is even delivered with OpenEdge. So the people that write it may not have any interest in supporting FWD or even if they had interest, Progress would not allow it. So let's stay away from PCT.

We need only PCT when testing with OE. Ant scripts only refer to it via environment setting, as its delivered with OE and I guess it cant be used without OE. We don't need PCT for testing FWD.

What I've implemented is the way of invoking FWD ant test scripts with the same parameters as for PCT, the difference for Junit and for FWD is only a different ant target (we dont need PCT to execute tests for FWD). In such case instead of having 4-5 parameters for launching tests I can use only one - Testcase class name.

Please find a way to provide an abstraction in the Harness which the tests can be written to connect/setup on both Windows and Linux and the right one is used when needed.

Referring to what you wrote earlier:

Greg Shah wrote:

Operating system support - what operating system needs to be supported ?

Both Windows and Linux. This is orthogonal with FWD/OE (it should run FWD and OE on both Windows and Linux).

• OE would be handled with ant/PCT for both Windows/Linux (I presume its true for Linux, as I dont have a way to test it).
• FWD would be handled by ant/JUnit for both Windows/Linux.

OK

1. for OE
   1. database needs to be created in current directory (e.g. testcases).
   2. database must not be started, PCT will start tstcases.db database
   3. DLC variable needs to point OE installation directory (to access OE and PCT)
2. for FWD
   1. project needs to have test cases converted and compiled
   2. FWD server and database need to be started (scripts do not start database nor FWD)

┌──────────────────────────────────────────────────────────────────────────────────────────┐
│                                           Harness                                        │
│                                                                                          │
│┌────────────────────────────────────────────────────────────────────────────────────────┐│
││                                         Test Plan                                      ││
││                                                                                        ││
││┌──────────────────────────────────────────────────────────────────────────────────────┐││
│││                                        Test Set                                      │││
│││                          ┌───────────────┐     ┌──────────┐                          │││
│││                          │               │     │          │                          │││
│││                          │Event Semaphore│     │  Mutex   │                          │││
│││                          │               │     │          │                          │││
│││                          └───────────▲───┘     └─▲────────┘                          │││
│││                                      │           │                                   │││
│││┌─────────────────────────┐       ┌───┼───────────┼─┐      ┌─────────────────────────┐│││
││││     Stream Executor 1   │       │                 │      │     Stream Executor 2   ││││
││││ ┌─────────────────────┐ │       │    Harness API  │      │ ┌─────────────────────┐ ││││
││││ │  ant (Junit or PCT) │ │       │                 │      │ │  ant (Junit or PCT) │ ││││
││││ │ ┌─────────────────┐ │ │       │                 │      │ │ ┌─────────────────┐ │ ││││
││││ │ │  ABLUnit test   ┼─┼─┼───────►      Socket     ◄──────┼─┼─┼  ABLUnit test   │ │ ││││
││││ │ │    Session 1    │ │ │       └─────────────────┘      │ │ │    Session 2    │ │ ││││
││││ │ └┬────────────────┘ │ │       ┌─────────────────┐      │ │ └───────────────┬─┘ │ ││││
││││ │  │                  │ │       │                 │      │ │                 │   │ ││││
││││ │  │   JUnit report 1 ├─┼──────►│  JUnit import   │◄─────┼─┤ Junit report 2  │   │ ││││
││││ └──┼──────────────────┘ │       │                 │      │ └─────────────────┼───┘ ││││
│││└────┼────────────────────┘       └────────┬────────┘      └───────────────────┼─────┘│││
││└─────┼─────────────────────────────────┬───┼────┬──────────────────────────────┼──────┘││
│└──────┼───────────────────────────────┬─┼───┼────┼──┬───────────────────────────┼───────┘│
│       │                            ┌──▼─▼───▼────▼──▼─┐                         │        │
│       │                            │     Harness      │                         │        │
│       │                            │    Reporting     │                         │        │
│       │                            │                  │                         │        │
│       │                            └──────────────────┘                         │        │
└───────┼─────────────────────────────────────────────────────────────────────────┼────────┘
 ┌──────┼─────────────────────────────────────────────────────────────────────────┼───────┐ 
 │      │                              FWD or OE                                  │       │ 
 │  ┌───▼───────────────┐                                        ┌────────────────▼──┐    │ 
 │  │ Client session 1  │                                        │ Client session 2  │    │ 
 │  │                   │                                        │                   │    │ 
 │  └───────────────────┘                                        └───────────────────┘    │ 
 │                                                                                        │ 
 │                                                                                        │ 
 │                                                                                        │ 
 └────────────────────────────────────────────────────────────────────────────────────────┘  

Example test plan. Notice each TestSet opens its own Harness API socket, so they can be executed in parallel (assuming they do not share any FWD/OE resource).

<?xml version="1.0"?>
<test-plan name="Cross session tests" description="Cross session tests  Test Plan" parallel = "2">
   <output-directory path="results" unique="true" />
   <target type="stream" shell="bash" />
   <resources>
      <variable name="testDirectory" type="String" initial="" prompt="Directory with tests" />
      <variable name="antTarget" type="String" initial="" prompt="Ant target to run" />

      <create-event-sem name="setupCompleted" set="true"/>
      <create-event-sem name="testsCompleted" set="true"/>
      <create-mutex name="testsMutex"/>
      <create-mutex name="dsUnique"/>
      <create-mutex name="dsNonUnique"/>

      <create-event-sem name="runSession1" set="true"/>
      <create-event-sem name="runSession2" set="true"/>

   </resources>

   <test-set name="record_locking_test" description="Cross session record locking tests" threads="2" >
      <target type="remote" port="8881" threads="2" />
      <test filename="support/harness/test/cross_locking_session_1.xml" dependency="concurrent" group="xsession1" />
      <test filename="support/harness/test/cross_locking_session_2.xml" dependency="concurrent" group="xsession1" />

   </test-set>

   <test-set name="dirty_share_nonunique_cross_session_test" description="DirtyShare cross session tests" threads="2" >
      <target type="remote" port="8882" threads="2" />
      <test filename="support/harness/test/dirty_share_nonunique_session_1.xml" dependency="concurrent" group="xsession1" />
      <test filename="support/harness/test/dirty_share_nonunique_session_2.xml" dependency="concurrent" group="xsession1" />
   </test-set>

   <test-set name="dirty_share_unique_cross_session_test" description="DirtyShare cross session tests" threads="2" >
      <target type="remote" port="8883" threads="2" />
      <test filename="support/harness/test/dirty_share_unique_session_1.xml" dependency="concurrent" group="xsession2" />
      <test filename="support/harness/test/dirty_share_unique_session_2.xml" dependency="concurrent" group="xsession2" />
   </test-set>

</test-plan>

Example test session 1:

<?xml version="1.0"?>
<test name="dirty_share_nonunique_session1" description="Cross session dirty share tests with non-unique indexes session 1" >
    <stream-write value="cd %s" newline="true">
        <substitution variable="testDirectory" />
    </stream-write> 

    <stream-write value="ant -Dtests.pattern=**/TestNonUniqueDirtyShareSession1.* -Dtests.dbdir=. -Dtests.db=testcases.db -Dtests.out=%s/build/dirty_share_nonunique_session1 -Dtests.dir=support/harness/test -Dtests.tempDir=build/dirty_share_nonunique_session1 %s" newline="true">
        <substitution variable="testDirectory" />
        <substitution variable="antTarget" />
    </stream-write> 
<!--
    Wait up to 20 seconds for tests to finish
-->
    <stream-wait-for value="BUILD"  millis="10000" newline="false"/>
    <stream-wait-for value="Total time"  millis="10000" newline="false"/>

    <junit-import filename="%s/build/dirty_share_nonunique_session1/results.xml" testset="dirty_share_nonunique_cross_session_test">
        <substitution variable="testDirectory" />
    </junit-import>
</test>

Example test session 2:

<?xml version="1.0"?>
<test name="dirty_share_nonunique_session2" description="Cross session dirty share tests with non-unique indexes session 2" >
    <stream-write value="cd %s" newline="true">
        <substitution variable="testDirectory" />
    </stream-write> 

    <stream-write value="ant -Dtests.pattern=**/TestNonUniqueDirtyShareSession2.* -Dtests.dbdir=. -Dtests.db=testcases.db -Dtests.out=%s/build/dirty_share_nonunique_session2 -Dtests.dir=support/harness/test -Dtests.tempDir=build/dirty_share_nonunique_session2 %s" newline="true">
        <substitution variable="testDirectory" />
        <substitution variable="antTarget" />
    </stream-write> 
<!--
    Wait up to 20 seconds for tests to finish
-->
    <stream-wait-for value="BUILD"  millis="10000" newline="false"/>
    <stream-wait-for value="Total time"  millis="10000" newline="false"/>

    <junit-import filename="%s/build/dirty_share_nonunique_session2/results.xml" testset="dirty_share_nonunique_cross_session_test">
        <substitution variable="testDirectory" />
    </junit-import>
</test>

Tomasz Domin wrote:

If we keep ant as abstraction layer:

• OE would be handled with ant/PCT for both Windows/Linux (I presume its true for Linux, as I dont have a way to test it).
• FWD would be handled by ant/JUnit for both Windows/Linux.

Its implemented by implementing ant/JUnit target called junit-tests-pattern which accept the same parameters as ablunit-tests-pattern target.
So in the end invoking test subsession looks like following:

#Linux
java -jar harness.jar -d -v testDirectory="/home/tjd/projects/testcases" -v antTarget=junit-tests-pattern abl/test/cross_session_test.xml 

#Windows
java -jar harness.jar -d -shell cmd.exe -v testDirectory="D:\tjd\ablunit" -v antTarget=ablunit-tests-pattern abl/test/cross_session_test.xml

I've pushed testcases revision 1701 with updates to Harness API and including Dirty Share Cross session tests.
Test work correctly both under OE/Windows and FWD/Linux.
In order to run the tests harness version 9270a is required.

• validation/flushing/trigger testcases
• record locking tests
• PASOE

Tested with regression tests (4 runs) , results are the same as current harness trunk.

Tomasz Domin wrote:

Marian - are there already any tests to be worked on in testcases project in above-mentioned areas:

• validation/flushing/trigger testcases

For validation/triggers there are tests already, don't see any need to involve Harness in those cases though... that happens equally well in a single user session :)

For flushing we need indeed Harness (multi user) to see when changes are reflected to other users, nothing has been done on this subject.

• record locking tests

Same situation as for flushing, there is not much we can do about this in single user session... this is where I think we should start with Harness with a very simple tests before we can come up with some kind of 'methodology' on writing 'Harness driven' ABLUnit tests :(

• PASOE

There are tests for PASOE, not sure what could be tested with Harness in that area? We did proposed to use APPSRV instead of Harness with pure ABLUnit but then maybe Harness is considered more reliable than the APPSRV implementation atm so less noise introduced in those 'persistence' tests.

Marian Edu wrote:

Tomasz Domin wrote:

Marian - are there already any tests to be worked on in testcases project in above-mentioned areas:

• validation/flushing/trigger testcases

For validation/triggers there are tests already, don't see any need to involve Harness in those cases though... that happens equally well in a single user session :)

Arent there cross session triggers ? E.g. for database events ?

For flushing we need indeed Harness (multi user) to see when changes are reflected to other users, nothing has been done on this subject.

Do you have an example of such test ?

• record locking tests

Same situation as for flushing, there is not much we can do about this in single user session... this is where I think we should start with Harness with a very simple tests before we can come up with some kind of 'methodology' on writing 'Harness driven' ABLUnit tests :(

Do you have an example of such test ?

• PASOE

There are tests for PASOE, not sure what could be tested with Harness in that area? We did proposed to use APPSRV instead of Harness with pure ABLUnit but then maybe Harness is considered more reliable than the APPSRV implementation atm so less noise introduced in those 'persistence' tests.

Greg ?

Marian, I was rather asking if you maybe have some ABL test code I could use in examples to prove idea behind #9270

Tomasz Domin wrote:

Arent there cross session triggers ? E.g. for database events ?

No idea how is that implemented in FWD but ion 4GL there is no such a thing as 'cross session triggers', the triggers are plain 4GL procedures that will be execute inside the client session - the only thing the server knows about those is the name of the procedure that needs to be executed for each database event. If the client does not have those procedures available in PROPATH it will just throw the standard 'procedure not found' error :)

Marian, I was rather asking if you maybe have some ABL test code I could use in examples to prove idea behind #9270

No, we have no tests built for Harness as we do not know how are we to use them... it's probably like a chicken&egg situation, we couldn't design any tests based on a solution that was not there or am I missing something? :)

I see you have have some locking tests in support.harness.test, how do you run harness with those two test classes I have no idea though :)

We can maybe start by pointing us to the place where we can download Harness (https://proj.goldencode.com/projects/harness/wiki/Download), is there a prebuilt binary or we need to build from source code?

You did mentioned a revision number, no idea where is the Bazaar repo for Harness you're using - the one mentioned on Harness download page (bzr co https://proj.goldencode.com/downloads/harness/repo/ harness) looks like being on revision #34, last commit made by Constantin on 10.10.2024 so it's probably not the one you're referring to. We need to see how a Harness 'script' looks like, only interested on chaining ABLUnit tests right now, and how to launch it... before that is kinda hard to blindly write test cases.

Marian Edu wrote:

Tomasz Domin wrote:

Marian, I was rather asking if you maybe have some ABL test code I could use in examples to prove idea behind #9270

No, we have no tests built for Harness as we do not know how are we to use them... it's probably like a chicken&egg situation, we couldn't design any tests based on a solution that was not there or am I missing something? :)

I dont ask you for tests for Harness, as I know you don't have the new version capable of running parallel ABLUnit sessions yet.
I am asking if you maybe have any ABLUnit code for above cases which now have to be run manually in separate OE/FWD sessions.

Tomasz Domin wrote:

I dont ask you for tests for Harness, as I know you don't have the new version capable of running parallel ABLUnit sessions yet.
I am asking if you maybe have any ABLUnit code for above cases which now have to be run manually in separate OE/FWD sessions.

Ah, I see :)

No, we don't have any, thought there was supposed to be some 'old' persistence tests that are to be translated to ABLUnit but haven't seen any just yet :(

I don't think we have usable tests for multiple clients in PASOE. I'm implementing a new version of PASOE now in #8661 and when that is done, we can try to add tests against it. At first it could be something simple as running a PASOE server with one worker and testing if a second client fails the connect with the expected error. I'm not sure how tests are organized when running Harness, but the PASOE tests I have in mind will need different directory configs.

Marian - are there already any tests to be worked on in testcases project in above-mentioned areas:

• validation/flushing/trigger testcases

For validation/triggers there are tests already, don't see any need to involve Harness in those cases though... that happens equally well in a single user session :)

I disagree here. The proper point at which a record is actually flushed to the database is quite important and can only be seen in multi-user cases. This flushing is deeply bound to the validation and trigger processing in the 4GL.

This is why I put these items in the multi-user list.

For flushing we need indeed Harness (multi user) to see when changes are reflected to other users, nothing has been done on this subject.

• record locking tests

Same situation as for flushing, there is not much we can do about this in single user session... this is where I think we should start with Harness with a very simple tests before we can come up with some kind of 'methodology' on writing 'Harness driven' ABLUnit tests :(

• PASOE

There are tests for PASOE, not sure what could be tested with Harness in that area? We did proposed to use APPSRV instead of Harness with pure ABLUnit but then maybe Harness is considered more reliable than the APPSRV implementation atm so less noise introduced in those 'persistence' tests.

Code Review Task Branch 9270a Revisions 35 through 77

Overall, the changes are very good.

1. Shouldn't the abl/ directory be moved into the ~/secure/code/harness_repo/samples/ project (please create a better subdirectory name than abl)? I don't think they should be directly in the harness project.

2. In Harness.java, is this code finished?

                  else if (acfg instanceof StreamSessionSettings)
                  {

                  }

3. In Harness.java, there are some data members in the middle of the code. Please move them to the top of the class and write private static instead of static private:

   /**
    * Flag indicating if shutdown hook be installed
    */
   private boolean enableShutdownHook;

   /**
    * Global server port for all TestSets, overrides TestSets settings
    */
   static private int serverSocketPort = 0;

4. Harness and ScreenPoller need a history entry.

5. In multiple files, there is for( instead of for (.

Greg Shah wrote:

Code Review Task Branch 9270a Revisions 35 through 77

Overall, the changes are very good.

1. Shouldn't the abl/ directory be moved into the ~/secure/code/harness_repo/samples/ project (please create a better subdirectory name than abl)? I don't think they should be directly in the harness project.

As stated in #9270-66 I will remove abl directory from harness just before merging, as it is already part of testcases project (testcases revision 1701)

2. In Harness.java, is this code finished?

Yes, I wanted to make clear that I am handling all cases here so there is no doubt about it and I didnt forget about , I will put a comment into it.

3. In Harness.java, there are some data members in the middle of the code. Please move them to the top of the class and write private static instead of static private:
[...]

Moved to the right place.

4. Harness and ScreenPoller need a history entry.

Added history entries.

5. In multiple files, there is for( instead of for (.

Updated formatting of for loops.

Pushed up to revision 79.

Greg
I think second reviewer should be CA.

Code Review Task Branch 9270a Revisions 78 through 80

No objections. I did check in a minor reversion of whitespace removal as rev 81.

Constantin, please review.

• TestPlan - the sets are now a HashMap instead of List - should we use a LinkedHashMap, to preserve the test order?

Otherwise, I don't understand how ant is involved - at this time, to me it looks to be a 'hard dependency' to have a specific ant file with a specific target. If so, I suggest create a specific ant file for this, and have it in harness; and rely on that instead of i.e. having to ensure the testcases project (or any other project) has this specific ant file.

Constantin Asofiei wrote:

Tomasz, in the code the only issue I see is:

• TestPlan - the sets are now a HashMap instead of List - should we use a LinkedHashMap, to preserve the test order?

Correct, I've missed that. I've updated sets type to LinkedHashMap.
Pushed up to revision 82.

Otherwise, I don't understand how ant is involved - at this time, to me it looks to be a 'hard dependency' to have a specific ant file with a specific target. If so, I suggest create a specific ant file for this, and have it in harness; and rely on that instead of i.e. having to ensure the testcases project (or any other project) has this specific ant file.

There is no direct integration with ant, its mentioned here because the main goal of the task was to enable executing multi-session 4GL testcases in OE/Windows and FWD/Linux and for that purpose both changes to harness and ant were needed. Its more like an example that when using ant with #9270 its possible to deliver requested functionality.

I've executed regression testing of trunk/15830 with harness_9270/82. No problems found, tests completed the same way as for unmodified harness.
I guess its ready to merge to trunk - in meantime I will update documentation.

Tested for regressions twice, no issues found.
Pushed up to revision 36.

Working on an update of a documentation.

Tomasz: Please make it a priority to write the documentation in Writing Multi-User Testcases.

Greg Shah wrote:

Tomasz: Please make it a priority to write the documentation in Writing Multi-User Testcases.

I've committed the documentation.
Note that the example project testcases/harness/support with no issues (I had to update zfile_set.txt) - two tests are failing, which is expected.