FWD » Core Development » Base Language

• argument translation to remote definition type I think is done the same as the CALL statement does it...

What does that mean?

• versioninfo fields are read-only, but writing them is a runtime error and not compile error in FWD.

This one seems to have been changed.

Compiling session/local_version_info.p in testcases fails with:

    [javac] /home/gbb/Code/testcases-working/src/com/goldencode/testcases/session/LocalVersionInfo.java:97: error: method getOeclientType in class OeversionInfo cannot be applied to given types;
    [javac]             silent(() -> oeVersionInfo.ref().getOeclientType(new character("javaclient")));
    [javac]                                             ^
    [javac]   required: no arguments
    [javac]   found: character
    [javac]   reason: actual and formal argument lists differ in length
    [javac] /home/gbb/Code/testcases-working/src/com/goldencode/testcases/session/LocalVersionInfo.java:100: error: method getOemaintVersion in class OeversionInfo cannot be applied to given types;
    [javac]             silent(() -> oeVersionInfo.ref().getOemaintVersion(new character("test")));
    [javac]                                             ^
    [javac]   required: no arguments
    [javac]   found: character
    [javac]   reason: actual and formal argument lists differ in length
    [javac] /home/gbb/Code/testcases-working/src/com/goldencode/testcases/session/LocalVersionInfo.java:103: error: method getOemajorVersion in class OeversionInfo cannot be applied to given types;
    [javac]             silent(() -> oeVersionInfo.ref().getOemajorVersion(new character("test")));
    [javac]                                             ^
    [javac]   required: no arguments
    [javac]   found: character
    [javac]   reason: actual and formal argument lists differ in length
    [javac] /home/gbb/Code/testcases-working/src/com/goldencode/testcases/session/LocalVersionInfo.java:106: error: method getOeminorVersion in class OeversionInfo cannot be applied to given types;
    [javac]             silent(() -> oeVersionInfo.ref().getOeminorVersion(new character("test")));
    [javac]                                             ^
    [javac]   required: no arguments
    [javac]   found: character
    [javac]   reason: actual and formal argument lists differ in length
    [javac] 4 errors

For some reason the assignment of new values (oeVersionInfo:OEClientType = 'javaclient' no-error.) is converted into calling the getter methods, but the important part is that the compilation fails for the four Progress.Lang.OEVersionInfo attributes.

If this is how it should work, then the test needs to be changed to not expect a runtime error.

• VersionInfo:clientType supports only 4GLCLIENT and APPSERVER at this time

• SESSION:CURRENT-REQUEST-INFO (Progress.Lang.OERequestInfo) : VersionInfo property (Progress.Lang.OEVersionInfo) : OEClientType property
• SESSION:LOCAL-VERSION-INFO (Progreses.Lang.OEVersionInfo) : OEClientType property

To review Appserver_Support.

The client type can be accessed from both the session and the request.

Is it possible to associate the session in FWD with only one type of web requests / client? What about concurrent execution?

Where does RemoteWebRequest fit?

WebSpeed used to be OE app in the browser (in some script) running on the WebSpeed agent. So the client knew it is WebSpeed. In FWD instead of a live app we have dead endpoints. What is WebSpeed session then? I think this clientType is not applicable. Am I missing something?

• checkout xfer testcases project
• edit build.properties and set db.names=tstcasesdb
• copy file-cvt-list.txt.soap.rest.webhandler as file-cvt-list.txt
• run ant deploy.all
• for SOAP, make sure 'soap:address' is set to '/wsa/wsafwd' in fwd.wsdl. After that, run ant jar
• run ./install_spawner.sh
• in deploy/server, do cp directory.xml.template directory.xml
• edit directory.xml and:
  • replace "[os-user]" with your OS username
  • replace "[log-console]" with "true"
  • replace "[p2j-sport]" with "3333"
  • replace "[spawner-path]" with "/opt/spawner/spawn"
  • replace "[client-start-dir]" with your FWD client dir
  • fix "jdbc:h2:[client-start-dir]/deploy/db/tstcasesdb;DB_CLOSE_DELAY=-1;MV_STORE=FALSE;IFEXISTS=TRUE"
  • set REST/enabled to true
  • set SOAP/enabled to true
  • set webHandler/enabled to true
  • set scheduler/start_appserver/enabled to true
  • set processes/appserver_agent/enabled to true
  • set processes/appserver_process/enabled to true
  • change admin port from 7443 to 9443
• start the FWD server
  • check https://localhost:9443/rest/goldencodeService/pipe/char/mihai if it works
• install SoapUI
• the project files are:
  

  ./appsrv/soap/test/web/fwd-soapui-project.xml 
  ./appsrv/soap/test/web/pasoe-fwd-soapui-project.xml
  ./appsrv/rest/test/fwd-soapui-project.xml
  

Import these in SoapUI and see if these work. There is also a file-cvt-list.txt.rest_and_appserver, to run the plain appserver client tests, but I haven't run these yet. Maybe you can give it a go and post the details here?

• some errors throw by SERVER:CONNECT are not the same

Tests in testcases/appsrv/test/connect_errors/ are now successful with 6417a r14907 & r14908, except for one condition in session_model_mismatch.p. I don't know how to set return-value to the error message. The actual return value of ServerImpl.connect is a boolean.

Constantin Asofiei wrote:

Import these in SoapUI and see if these work. There is also a file-cvt-list.txt.rest_and_appserver, to run the plain appserver client tests, but I haven't run these yet. Maybe you can give it a go and post the details here?

I'll plan to run this batch of tests later on, because I guess it will present more issues for solving, instead of answers at this stage. It won't help with #6417-5 that is to make clear the requirement (do we want a runtime or compile error), or #6417-6 and #6417-7 because OEClientType tests are written in a way they can't fail (obviously not testing all types of clients). I will get back to those points when the rest is done.

Galya B wrote:

It won't help with #6417-5 that is to make clear the requirement (do we want a runtime or compile error),

We need to add the setter and make it just throw an ERROR condition - there is no error message displayed/logged. I can't explain why they would not leave this a hard compile error if the property is supposed to be read-only.

or #6417-6

There are tests IIRC for SESSION:CURRENT-REQUEST-INFO and SESSION:LOCAL-VERSION-INFO, for REST/SOAP/WEB.

Greg Shah wrote:

• a QUIT on the remote side quits the client

Solved in r14910. appsrv/test/test_raise_quit.p now is successful.

• versioninfo fields are read-only, but writing them is a runtime error and not compile error in FWD.

Solved in r14911. session/local_version_info.p now successful.

Constantin Asofiei wrote:

Import these in SoapUI and see if these work. There is also a file-cvt-list.txt.rest_and_appserver, to run the plain appserver client tests, but I haven't run these yet. Maybe you can give it a go and post the details here?

Actually, the set to be run is in appsrv/runall.p, because the lists for conversion contain procedures that can't be called independently, for example appsrv/api/dataset/get_current_request_info.p, which needs an argument. That's why it should be called from appsrv/test/dataset/test_get_current_request_info.p.

This is just a note.

• oerequestinfo: adapterType requires enum, which is not supported by FWD yet

I reworked com.goldencode.p2j.oo.applicationserver.AdapterTypes in r14912.

The only related tests I've found are appsrv/api/dataset/get_current_request_info.p and appsrv/api/dataset/get_current_response_info.p and they are not failing.

The skeleton might need to be changed on merge to include all the numeric associations.

oo4gl/Progress/ApplicationServer

enum Progress.ApplicationServer.AdapterTypes:
   define enum
   Unexpected = 0

   APSV

   SOAP

   REST

   WEB.

end class.

• agentId, sessionId, threadId: these are 'plugged in' but the Agent doesn't set the to anything (remain unknown).

I can find only one instance of Agent created, that is on appserver start. So the AgentPool is actually one agent per appserver. Am I correct? This means all tasks on the appserver are executed on the same agent, that's why the id is always 0.
Depends on directory configs.

Constantin Asofiei wrote:

• for SOAP, make sure 'soap:address' is set to '/wsa/wsafwd' in fwd.wsdl. After that, run ant jar

There are two fwd.wsdl files, where soap:address is full url. Where do I need to replace soap:address with the path?

appsrv/soap/proxygen/fwd.wsdl:

  <wsdl:service name="fwdService">
    <wsdl:port name="persistentObj" binding="tns:persistentObj">
<wsdl:documentation></wsdl:documentation>
      <soap:address location="http://localhost:9100/soap"/>
    </wsdl:port>
    <wsdl:port name="fwdObj" binding="tns:fwdObj">
<wsdl:documentation></wsdl:documentation>
      <soap:address location="http://localhost:9100/soap"/>
    </wsdl:port>
  </wsdl:service>

appsrv/config/fwd-pasoe/webapps/goldencode/WEB-INF/adapters/soap/goldencode/fwd.wsdl:

  <wsdl:service name="fwdService">
    <wsdl:port name="persistentObj" binding="tns:persistentObj">
<wsdl:documentation></wsdl:documentation>
      <soap:address location="http://DESKTOP-EIOD5PQ:8080/goldencode/soap"/>
    </wsdl:port>
    <wsdl:port name="fwdObj" binding="tns:fwdObj">
<wsdl:documentation></wsdl:documentation>
      <soap:address location="http://DESKTOP-EIOD5PQ:8080/goldencode/soap"/>
    </wsdl:port>
  </wsdl:service>

Galya B wrote:

Constantin Asofiei wrote:

• for SOAP, make sure 'soap:address' is set to '/wsa/wsafwd' in fwd.wsdl. After that, run ant jar

There are two fwd.wsdl files, where soap:address is full url. Where do I need to replace soap:address with the path?

You need to edit the fwd.wsdl file from the src/ folder, not abl/.

file-cvt-list.txt.soap.rest.webhandler (138 procedures) is a subset of file-cvt-list.txt.rest_and_appserver (988 procs). To run runall.p I need the latter and this is what I've used.

Several tests fail because appsrv/api/table/io_table_handle.p can't find table/help/get_session_dynamic_table.p although it's in the conversion list:

appsrv/test/table/test_input_output_table_handle.p| ABL TempTable type : - input/output-table-handle - start
appsrv/test/table/test_input_output_table_handle.p| UNEXPECTED_ERROR: calling procedure isValidTable shuld not throw error (** "table/help/get_session_dynamic_table.p" was not found. (293)) #22
appsrv/test/table/test_input_output_table_handle.p| UNEXPECTED_ERROR: on server after calling procedure remain a valid handle #27
appsrv/test/table/test_input_output_table_handle.p| UNEXPECTED_ERROR: calling procedure isValidTable shuld not throw error (** "table/help/get_session_dynamic_table.p" was not found. (293)) #34
appsrv/test/table/test_input_output_table_handle.p| UNEXPECTED_ERROR: calling procedure isValidTable shuld not throw error (** "table/help/get_session_dynamic_table.p" was not found. (293)) #46
appsrv/test/table/test_input_output_table_handle.p| UNEXPECTED_ERROR: on server after calling procedure remain a valid handle #51
appsrv/test/table/test_input_output_table_handle.p| UNEXPECTED_ERROR: calling procedure isValidTable shuld not throw error (** "table/help/get_session_dynamic_table.p" was not found. (293)) #58
appsrv/test/table/test_input_output_table_handle.p| ABL TempTable type : - input/output-table-handle - end
appsrv/test/table/test_input_table_handle.p| ABL TempTable type : - input-table-handle - start
appsrv/test/table/test_input_table_handle.p| UNEXPECTED_ERROR: calling procedure isValidTable shuld not throw error (** "table/help/get_session_dynamic_table.p" was not found. (293)) #21
appsrv/test/table/test_input_table_handle.p| UNEXPECTED_ERROR: on server after calling procedure remain a valid handle #26
appsrv/test/table/test_input_table_handle.p| UNEXPECTED_ERROR: calling procedure isValidTable shuld not throw error (** "table/help/get_session_dynamic_table.p" was not found. (293)) #33
appsrv/test/table/test_input_table_handle.p| UNEXPECTED_ERROR: calling procedure isValidTable shuld not throw error (** "table/help/get_session_dynamic_table.p" was not found. (293)) #45
appsrv/test/table/test_input_table_handle.p| UNEXPECTED_ERROR: on server after calling procedure remain a valid handle #50
appsrv/test/table/test_input_table_handle.p| UNEXPECTED_ERROR: calling procedure isValidTable shuld not throw error (** "table/help/get_session_dynamic_table.p" was not found. (293)) #57
appsrv/test/table/test_input_table_handle.p| ABL TempTable type : - input-table-handle - end
appsrv/test/table/test_output_table_handle.p| ABL TempTable type : - output-table-handle - start
appsrv/test/table/test_output_table_handle.p| UNEXPECTED_ERROR: calling procedure isValidTable shuld not throw error (** "table/help/get_session_dynamic_table.p" was not found. (293)) #18
appsrv/test/table/test_output_table_handle.p| UNEXPECTED_ERROR: calling procedure isValidTable shuld not throw error (** "table/help/get_session_dynamic_table.p" was not found. (293)) #30
appsrv/test/table/test_output_table_handle.p| UNEXPECTED_ERROR: on server after calling procedure should not remain a valid handle #35
appsrv/test/table/test_output_table_handle.p| UNEXPECTED_ERROR: calling procedure isValidTable shuld not throw error (** "table/help/get_session_dynamic_table.p" was not found. (293)) #42
appsrv/test/table/test_output_table_handle.p| UNEXPECTED_ERROR: calling procedure isValidTable shuld not throw error (** "table/help/get_session_dynamic_table.p" was not found. (293)) #54
appsrv/test/table/test_output_table_handle.p| UNEXPECTED_ERROR: on server after calling procedure should not remain a valid handle #59
appsrv/test/table/test_output_table_handle.p| UNEXPECTED_ERROR: calling procedure isValidTable shuld not throw error (** "table/help/get_session_dynamic_table.p" was not found. (293)) #66
appsrv/test/table/test_output_table_handle.p| ABL TempTable type : - output-table-handle - end

Greg Shah wrote:

• OO instance serialization and transmission to remote side (GES: are we already handling this in #4658?)

I guess it has to be handled in #4658.

• OO instance serialization and transmission to remote side (GES: are we already handling this in #4658?)

I guess it has to be handled in #4658.

Correct, unless something is absolutely needed for this task.

Marian, can you help with #6417-19. Something seems wrong with the paths to procedures for appsrv/runall.p.

agentId, sessionId, threadId: these are 'plugged in' but the Agent doesn't set the to anything (remain unknown).

Values added to sessionId and threadId in PASOE requests. r14913.

r14914 Fixes legacy appserver throw / return error.

Related tests appsrv/test/test_throw_error.p & appsrv/test/test_return_error.p.

appsrv/test/test_throw_error.p has an issue that needs to be fixed on merge: substitute({&msg_attr_test_value}, 'return-value', 'throwError') to be replaced by substitute({&msg_attr_test_value}, 'return-value', 'returnValue') as defined in appsrv/api/throw_error.p.

r14915 Handles legacy appserver stop error. Related test appsrv/test/test_raise_stop.p.

Constantin Asofiei wrote:

There is also a file-cvt-list.txt.rest_and_appserver, to run the plain appserver client tests, but I haven't run these yet. Maybe you can give it a go and post the details here?

Does noone run the tests regularly?

Until we are done with our work with #6853, we only run specific sets of tests manually to check functionality. Once #6853 is done, this will be done nightly at a minimum.

What is this port 8980? Something is off, but I can't tell what.

[24/01/11@16:48:55.347+0200] P-149379 T-000001 1 4GL -- (Procedure: '../../appsrv/soap/test/4glws/test_raise_stop.p') Error loading WSDL document  http://localhost:8980/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org :  WSDLException: faultCode=OTHER_ERROR: Unable to resolve imported document at 'http://localhost:8980/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org'.: java.net.ConnectException: Connection refused (Connection refused). (11748)

Otherwise https://localhost:9443/rest/goldencodeService/pipe/char/mihai works.

P.S. all configs are according to #6417-8.

Galya B wrote:

What is this port 8980? Something is off, but I can't tell what.

These files need to be changed manually to match the URL for the WSDL file (9443 instead of 8980):

./appsrv/soap/test/4glws/connect.i:9:hWebService:connect ('-WSDL http://localhost:8980/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org -Port {&portName}Obj').
./appsrv/soap/test/4glws/connect.i:12:hWebService:connect ('-WSDL http://localhost:8980/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org -Port fwdObj').

I missed this, is this when running something from SoapUI?

appsrv/soap/test/4glws.connect.i has the port 8980 hardcoded and soap tests ran from testcases are looking for the resources there.

Galya B wrote:

appsrv/soap/test/4glws.connect.i has the port 8980 hardcoded and soap tests ran from testcases are looking for the resources there.

Are you running the SOAP tests from FWD Client (and using the FWD web service component), via SopaUI, or both?

All tests fail on all clients with different errors.

The FWD client shows:

(Procedure: '../../appsrv/soap/test/4glws/test_return_error.p') Error loading WSDL document  http://localhost:8980/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org :  WSDLException: faultCode=OTHER_ERROR: Unable to resolve imported document at 'http://localhost:8980/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org'.: java.net.ConnectException: Connection refused (Connection refused). (11748)

I changed the port in appsrv/soap/test/4glws.connect.i to 9443 and now it fails with :

[24/01/12@14:41:56.648+0200] P-100032 T-000001 1 4GL -- (Procedure: '../../appsrv/soap/test/4glws/test_return_error.p') Error loading WSDL document  http://localhost:9443/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org :  WSDLException: faultCode=OTHER_ERROR: Unable to resolve imported document at 'http://localhost:9443/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org'.: java.net.SocketException: Unexpected end of file from server. (11748)

corresponding to server error:

24/01/12 14:45:11.632+0200 |  SEVERE | com.goldencode.p2j.util.ErrorManager | ThreadName:Conversation [00000041:bogus], Session:00000065, Thread:00000074, User:bogus | Invalid or inappropriate server handle specified for RUN fwdObj ... ON SERVER statement. (5453)

SoapUI shows NPE for many tests:

 -> [Character Match] XPathContains assertion failed for path [declare namespace ns="urn:tempuri-org:fwd" 
//ns:table_outResponse//ns:ttPrimitive//ns:ttPrimitiveRow//ns:dataChar] : NullPointerException:Cannot invoke "String.length()" because "s" is null
 -> [Date Match] XPathContains assertion failed for path [declare namespace ns="urn:tempuri-org:fwd" 
//ns:table_outResponse//ns:ttPrimitive//ns:ttPrimitiveRow//ns:dataDate] : NullPointerException:Cannot invoke "String.length()" because "s" is null

corresponding to server error:

24/01/12 14:38:06.841+0200 |  SEVERE | com.goldencode.p2j.rest.RestHandler | ThreadName:REST worker #0, Session:00000004, Thread:00000006, User:appserver_agent | Could not execute //dataset/get_current_response_info
java.lang.NullPointerException
    at com.goldencode.p2j.util.AppServerManager.invoke(AppServerManager.java:1292)
    at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.goldencode.p2j.net.LocalStaticRedirector.invokeCore(LocalStaticRedirector.java:167)
    at com.goldencode.p2j.net.InvocationStub.invoke(InvocationStub.java:144)
    at com.sun.proxy.$Proxy28.invoke(Unknown Source)
    at com.goldencode.p2j.util.AppServerHelper.invoke(AppServerHelper.java:2499)
    at com.goldencode.p2j.rest.ProgRestService.invoke(ProgRestService.java:196)
    at com.goldencode.p2j.rest.RestHandler.invoke(RestHandler.java:404)
    at com.goldencode.p2j.rest.RestHandler.lambda$handle$3(RestHandler.java:580)
    at com.goldencode.p2j.main.AppServerConnectionPool.lambda$dispatch$1(AppServerConnectionPool.java:248)
    at com.goldencode.p2j.main.LegacyServiceWorker.run(LegacyServiceWorker.java:305)
    at com.goldencode.p2j.main.AppServerConnectionPool.lambda$doInitialize$2(AppServerConnectionPool.java:479)
    at java.lang.Thread.run(Thread.java:750)
24/01/12 14:38:06.842+0200 | WARNING | org.eclipse.jetty.server.HttpChannel | ThreadName:qtp1394842015-53 | handleException /rest/goldencodeService//dataset/get_current_response_info java.io.IOException: java.lang.NullPointerException

Galya, http://localhost:9443/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org needs to use https and also the full URL I think needs to be this: https://localhost:9443/ws/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org (note the /ws/). This should work from the browser directly, too, to get the WSDL file.

1. focus on the 4GL tests which run SOAP, not SoapUI. Document which tests fail.
2. in SoapUI, there are failures, but not all needs to be addressed in this task; please focus on the Exceptions and Info test suites

Constantin Asofiei wrote:

Galya, http://localhost:9443/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org needs to use https and also the full URL I think needs to be this: https://localhost:9443/ws/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org (note the /ws/). This should work from the browser directly, too, to get the WSDL file.

Is this supposed to be changed in connect.i? It's hard-coded. Anyways, I'll try to figure out the magic of FWD test configurations and paths.

1. focus on the 4GL tests which run SOAP, not SoapUI. Document which tests fail.

I don't mix SOAP and SoapUI :) But I will probably need to run REST tests as well, because I have to figure out how to set clientType for the different supported types of requests (part of the requirements of the task).

Galya B wrote:

Constantin Asofiei wrote:

Galya, http://localhost:9443/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org needs to use https and also the full URL I think needs to be this: https://localhost:9443/ws/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org (note the /ws/). This should work from the browser directly, too, to get the WSDL file.

Is this supposed to be changed in connect.i? It's hard-coded. Anyways, I'll try to figure out the magic of FWD test configurations and paths.

Yes, in connect.i.

r14916 Adds conditions for unknown CURRENT-RESPONSE-INFO / CURRENT-REQUEST-INFO.

Constantin Asofiei wrote:

Galya B wrote:

Constantin Asofiei wrote:

Galya, http://localhost:9443/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org needs to use https and also the full URL I think needs to be this: https://localhost:9443/ws/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org (note the /ws/). This should work from the browser directly, too, to get the WSDL file.

Is this supposed to be changed in connect.i? It's hard-coded. Anyways, I'll try to figure out the magic of FWD test configurations and paths.

Yes, in connect.i.

Lovely, this is the 3rd type of exception, a new one:

[24/01/12@16:01:42.597+0200] P-120205 T-000001 1 4GL -- (Procedure: '../../appsrv/soap/test/4glws/test_raise_stop.p') Error loading WSDL document  https://localhost:9443/ws/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org :  WSDLException: faultCode=OTHER_ERROR: Unable to resolve imported document at 'https://localhost:9443/ws/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org'.: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target. (11748)

What certificate?

The batch client I run points to:

   <security>
      <truststore filename="../server/srv-certs.store" />
      <truststore alias="standard" />
   </security>

Isn't it enough?

No, this is about the HTTPS conection. I'm looking at it now.

Are you sure it should be https?

Galya B wrote:

Are you sure it should be https?

Yes.

Galya B wrote:

• argument translation to remote definition type I think is done the same as the CALL statement does it...

What does that mean?

Which types and what CALL?

While in deploy/client-chui, I've ran this and I don't get the certificate issue:

./client-swing.sh client:cmd-line-option:startup-procedure=appsrv/test/test_raise_stop.p

What do you use to run the tests?

appsrv/test/test_raise_stop.p is the legacy appserver test. Try:

appsrv/soap/test/4glws/test_return_error.p 
appsrv/soap/test/4glws/test_raise_stop.p
appsrv/soap/test/4glws/test_throw_error.p
appsrv/soap/test/4glws/test_return_value.p

1. https://localhost:9443/ws/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org does not work and https://localhost:9443/ws/wsa/wsafwd/wsdl?targetURI=urn:tempuri-org works - you can use the browser directly, this needs to retrieve the WSDL document. Note the missing / before the ? in the case where it works.
2. even with the WSDL URL 'fixed', then this does not discover any services.

This can be used to run it while in deploy/client-swing:

./client-swing.sh client:cmd-line-option:startup-procedure=appsrv/soap/test/4glws/test_raise_stop.p net:http_client:disable_ssl_certificate_validation=true ssl-socket:truststore:location=../server/srv-certs.store ssl-socket:truststore:password=[Ugpg637QuJAC\$\*lep95gqZ-Au\`k7nWSiW15j

Please look into both issues.

Galya B wrote:

Marian, can you help with #6417-19. Something seems wrong with the paths to procedures for appsrv/runall.p.

Gayla, I've somehow missed this... the file was added by Constantin, most probably to run all tests at once, not sure what is wrong with it, do you want me to look at it or problem was already solved by now?

Constantin Asofiei wrote:

Galya B wrote:

Constantin Asofiei wrote:

Galya, http://localhost:9443/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org needs to use https and also the full URL I think needs to be this: https://localhost:9443/ws/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org (note the /ws/). This should work from the browser directly, too, to get the WSDL file.

Is this supposed to be changed in connect.i? It's hard-coded. Anyways, I'll try to figure out the magic of FWD test configurations and paths.

Yes, in connect.i.

For the time being this is hardcoded in this include file, this is at least easier to change in only one place instead of all tests that are making an appsrv connection :)

We will see if/how we can parametrise the tests in order to override such hardcoded values - something that should be supported by both ABLUnit and the FWD implementation of it.

For SoapUI tests everything is based on an `endpoint` so the URL there is also hardcoded - we have tests for both classical and pasoe, as far as I remember the easiest way to run the tests was to update the `service endpoint` and `service path` custom properties on the project... sadly this was only done like that for the REST services while the SOAP projects seems to have the hardcoded URL's :(

Constantin Asofiei wrote:

Thanks. There are a couple of issues:

1. https://localhost:9443/ws/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org does not work and https://localhost:9443/ws/wsa/wsafwd/wsdl?targetURI=urn:tempuri-org works - you can use the browser directly, this needs to retrieve the WSDL document. Note the missing / before the ? in the case where it works.
2. even with the WSDL URL 'fixed', then this does not discover any services.

This can be used to run it while in deploy/client-swing:
[...]

Please look into both issues.

What do you mean by "not discover any services"? I'm still having a certificate issue:

Retrieving document at 'https://localhost:9443/ws/wsa/wsafwd/wsdl?targetURI=urn:tempuri-org'.
Error loading WSDL document  https://localhost:9443/ws/wsa/wsafwd/wsdl?targetURI=urn:tempuri-org :  WSDLException: faultCode=OTHER_ERROR: Unable to resolve imported document at 'https://localhost:9443/ws/wsa/wsafwd/wsdl?targetURI=urn:tempuri-org'.: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors. (11748)

The relevant section of the wiki is in TBD: Web Service SSL Connections.
Roger got the same issue in #5024-98, but I can't find an explanation how it got resolved.

The server .der certificate is imported in the JVM's cacert. I'm not sure about the appserver account certificates, if they need to be fixed somehow somewhere.

Otherwise the document opens fine in the browser.

Galya, the point here is to use ssl-socket:truststore to refer to srv-certs.store with the FWD server's certificate. This works with the bzr xfer testcases.

Galya B wrote:

Otherwise the document opens fine in the browser.

Does it open with both wsdl/?... and wsdl?... cases?

Constantin Asofiei wrote:

Galya B wrote:

Otherwise the document opens fine in the browser.

Does it open with both wsdl/?... and wsdl?... cases?

After the fix, it does, without the extra /.

Galya, the point here is to use ssl-socket:truststore to refer to srv-certs.store with the FWD server's certificate. This works with the bzr xfer testcases.

I do use it, that's configured in client.xml:

   <security>
      <truststore filename="../server/srv-certs.store" />
      <truststore alias="standard" />
   </security>
   <access>
      <password truststore="Ugpg637QuJAC$*lep95gqZ-Au`k7nWSiW15j" />
   </access>

Constantin, so you don't get the ssl issue in the output file or in the clientlog <cmd-line-option clientlog="file.log" />?

Galya B wrote:

Galya, the point here is to use ssl-socket:truststore to refer to srv-certs.store with the FWD server's certificate. This works with the bzr xfer testcases.

I do use it, that's configured in client.xml:
[...]

No, that's something else. Please test using the command I mentioned in #6417-45

Constantin Asofiei wrote:

Galya B wrote:

Galya, the point here is to use ssl-socket:truststore to refer to srv-certs.store with the FWD server's certificate. This works with the bzr xfer testcases.

I do use it, that's configured in client.xml:
[...]

No, that's something else. Please test using the command I mentioned in #6417-45

net:http_client:disable_ssl_certificate_validation=true hides the ssl problem indeed.

Then Port servicePort = getImplicitSoapPort(service.getPorts().values()); is null. The port called fwdObj, isn't it supposed to be defined in directory.xml?

I think this code should be executed, as the -Port from connect is set:

         if (options.getPortName() != null)
         {
            servicePort = service.getPort(options.getPortName());
         }

This is executed instead:

         Port servicePort = getImplicitSoapPort(service.getPorts().values());
         if (servicePort == null)
         {
            noServicesError();
            return ERR_NO_SERVICE_ERROR;
         }

P.S. The screenshot can be slightly confusing. Actually the highlighted breakpoint is not hit. Instead the one on line 431 is.

Please look into how getImplicitSoapPort calculates the implicit port and also compare fwd.wsdl from src/ with ./appsrv/soap/proxygen/fwd.wsdl, if there are any difference regarding fwdObj port.

The problem is that WebServiceImpl.connect expects explicit service name or binding name, while there is only a port in the connect options. In this case the code goes for the first service in the .wsdl and expects it to have only one port, otherwise an exception is thrown. In the wsdl there is one service with two ports, so it doesn't work.

There are a few ways to fix it:
1. in connect.i using a service name in the params;
2. in WebServiceImpl.connect read the port option, even when no service is explicitly requested, like so:

=== modified file 'src/com/goldencode/p2j/util/WebServiceImpl.java'
--- src/com/goldencode/p2j/util/WebServiceImpl.java    2023-05-19 19:46:27 +0000
+++ src/com/goldencode/p2j/util/WebServiceImpl.java    2024-01-15 14:53:02 +0000
@@ -427,7 +427,9 @@
          }

          service = (Service) services.values().iterator().next();
-         Port servicePort = getImplicitSoapPort(service.getPorts().values());
+
+         Port servicePort = options.getPortName() != null ? service.getPort(options.getPortName()) : 
+         getImplicitSoapPort(service.getPorts().values());
          if (servicePort == null)
          {
             noServicesError();

Marian Edu wrote:

Galya B wrote:

Marian, can you help with #6417-19. Something seems wrong with the paths to procedures for appsrv/runall.p.

Gayla, I've somehow missed this... the file was added by Constantin, most probably to run all tests at once, not sure what is wrong with it, do you want me to look at it or problem was already solved by now?

I'm trying to run as much as possible from file-cvt-list.txt.rest_and_appserver. The first issue with missing procedures in the list was described in #6417-19 and I fixed it by adding tests/table/help/get_session_dynamic_table.p.

Now there is another one: (Procedure: '../../appsrv/test/pipe/test_pipe_raw.p') ** "appsrv/test/pipe/test_pipe.i" was not found. (293). I tried to convert appsrv/test/pipe/test_pipe.i (although not sure if .i files are supposed to be converted with convert.list), but it doesn't compile afterwards. Do you know how to run the test appsrv/test/pipe/test_pipe_raw.p?

Something is still off.

ControlFlowOps is throwing errors:

         if (explicit && h.isUnknown())
         {
            String err = null;
            int num = -1;
            final String msg =
               "Could not evaluate procedure handle expression while trying to execute %s";
            err = String.format(msg, name);
            num = 2125;

            ErrorManager.recordOrThrowError(num, err, false);
            return false;
         }

Galya B wrote:

Now there is another one: (Procedure: '../../appsrv/test/pipe/test_pipe_raw.p') ** "appsrv/test/pipe/test_pipe.i" was not found. (293). I tried to convert appsrv/test/pipe/test_pipe.i (although not sure if .i files are supposed to be converted with convert.list), but it doesn't compile afterwards. Do you know how to run the test appsrv/test/pipe/test_pipe_raw.p?

I'm not sure what the issue could be here, the include file is there and the procedure compiles just fine in 4GL - only thing that is needed is to have the project's root in propath, you definitively don't need to add the include file to the convert list.

Marian Edu wrote:

Galya B wrote:

Now there is another one: (Procedure: '../../appsrv/test/pipe/test_pipe_raw.p') ** "appsrv/test/pipe/test_pipe.i" was not found. (293). I tried to convert appsrv/test/pipe/test_pipe.i (although not sure if .i files are supposed to be converted with convert.list), but it doesn't compile afterwards. Do you know how to run the test appsrv/test/pipe/test_pipe_raw.p?

I'm not sure what the issue could be here, the include file is there and the procedure compiles just fine in 4GL - only thing that is needed is to have the project's root in propath, you definitively don't need to add the include file to the convert list.

The procedure compiles fine, when .i is not in the convert list. But when it's executed, then I get the error (Procedure: '../../appsrv/test/pipe/test_pipe_raw.p') ** "appsrv/test/pipe/test_pipe.i" was not found.

It probably is the propath then. I have only:

      <node class="container" name="standard">
        <node class="container" name="appservers">
          <node class="appserver" name="app_server">
            <node-attribute name="propath" value=".:appsrv/api:"/>

appsrv/test is not in. Do you think I miss something else?

Galya B wrote:

It probably is the propath then. I have only:
[...]

appsrv/test is not in. Do you think I miss something else?

This is for the procedures that are to be ran from the application server - we deploy on the appsrv only the content of appsrv/api, the file you're referencing is only a test that we run from the client that will call out to the appsrv. When you convert the test procedures - under appsrv/test you only need the `testcases` root in `propath`. The path of the test procedure you're converting looks a bit odd (`../../appsrv/test/pipe/test_pipe_raw.p`)... this is relative to some path in deploy probably and propath only contains "." for the client so the current working folder is somehow important :(

Marian Edu wrote:

The path of the test procedure you're converting looks a bit odd (`../../appsrv/test/pipe/test_pipe_raw.p`)... this is relative to some path in deploy probably and propath only contains "." for the client so the current working folder is somehow important :(

The path in the test procedure itself ( appsrv/test/pipe/test_pipe_raw.p ) is not odd:

{appsrv/test/pipe/test_pipe.i 'pipe_raw.p' 'Mihai' 'character' true 'n/a'}

Galya B wrote:

I'm executing it from deploy/client-chui, where the client.xml config is: startup-procedure="../../appsrv/test/pipe/test_pipe_raw.p". Is this not supposed to work?

The path is: startup-procedure="appsrv/test/pipe/test_pipe_raw.p". Why the two up paths? This name doesn't reference something on disk, but something from name_map.xml.

Constantin Asofiei wrote:

Galya B wrote:

I'm executing it from deploy/client-chui, where the client.xml config is: startup-procedure="../../appsrv/test/pipe/test_pipe_raw.p". Is this not supposed to work?

The path is: startup-procedure="appsrv/test/pipe/test_pipe_raw.p". Why the two up paths? This name doesn't reference something on disk, but something from name_map.xml.

Nice, now I get:

(Procedure: 'appsrv/test/pipe/test_pipe_raw.p') ** "appsrv/test/pipe/test_pipe.i" was not found. (293)

#6417-68 I asked you about is not related. It happens with any 4glws test.

That's because of this statements:

copy-lob from file 'appsrv/test/pipe/test_pipe.i' to input_longchar.
copy-lob from file 'appsrv/test/pipe/test_pipe.i' to input_longchar.

This file needs to be made available to the FWD runtime.

Just create the deploy/client/appsrv/test/pipe/test_pipe.i file as a copy of appsrv/test/pipe/test_pipe.i.

Constantin Asofiei wrote:

That's because of this statements:
[...]

This file needs to be made available to the FWD runtime.

Just create the deploy/client/appsrv/test/pipe/test_pipe.i file as a copy of appsrv/test/pipe/test_pipe.i.

Is there no option to add the original location to a client propath?

Galya B wrote:

Is there no option to add the original location to a client propath?

You can add the full path for the testcases project to the propath, this will find it.

Constantin Asofiei wrote:

Galya B wrote:

Is there no option to add the original location to a client propath?

You can add the full path for the testcases project to the propath, this will find it.

Is this how it should look:

      <node class="container" name="standard">
        <node class="container" name="appservers">
          <node class="appserver" name="app_server">
            <node-attribute name="propath" value=".:appsrv/api:/home/gbb/code/testcases:"/>
          </node>
        </node>
        <node class="container" name="runtime">
          <node class="container" name="default">
            <node class="container" name="legacy-system">
              <node class="string" name="propath">
                <node-attribute name="value" value=".:appsrv/api:/home/gbb/code/testcases:"/>
              </node>
          </node>
        </node>

The only way to run appsrv/test/pipe/test_pipe_raw.p is as Marian suggested from the root of testcases. I would appreciate if someone confirms that the test can't be run successfully from the deploy dir.

Galya B wrote:

The only way to run appsrv/test/pipe/test_pipe_raw.p is as Marian suggested from the root of testcases. I would appreciate if someone confirms that the test can't be run successfully from the deploy dir.

I can confirm all paths used in tests and suport classes are relative to the project's root so if the PROPATH doesn't have the absolute path of this folder but a relative path that will be affected by the current working folder and the procedures/classes/includes (even other support files) referenced won't be resolved.

On our side since we're using PSDOE/Eclipse we've made simple wrapper classes to execute conversion, start server/client and we run those in run/debug configurations (we can override some default parameters if needed) - you can find those in testcases/src-dev. Point is we always run those from the project's root, even the FWD ABLUnit (with a custom tweak applied locally to be able to specify the location of unit_test.xml configuration file).

I continue to struggle with the SOAP tests.

Procedure to reproduce the issue (pretty much the connect.i):

define variable hWebService as handle    no-undo.
define variable hfwdObj     as handle    no-undo.

create server hWebService.
hWebService:connect('-WSDL https://localhost:9443/ws/wsa/wsafwd/wsdl?targetURI=urn:tempuri-org -Service fwdService -Port fwdObj').
run fwdObj set hfwdObj on hWebService no-error.

is converted to:

public class Soap
{
   private static final InvokeConfig RUN_CALL_SITE_1 = new InvokeConfig().setTarget("fwdObj");

   @LegacySignature(type = Type.VARIABLE, name = "hWebService")
   handle hWebService = TypeFactory.handle();

   @LegacySignature(type = Type.VARIABLE, name = "hfwdObj")
   handle hfwdObj = TypeFactory.handle();

   /**
    * External procedure (converted to Java from the 4GL source code
    * in galya/soap.p).
    */
   @LegacySignature(type = Type.MAIN, name = "galya/soap.p")
   public void execute()
   {
      externalProcedure(Soap.this, new Block((Body) () -> 
      {
         ServerFactory.createServer(hWebService);
         hWebService.unwrapConnectable().connect(new character("-WSDL https://localhost:9443/ws/wsa/wsafwd/wsdl?targetURI=urn:tempuri-org -Service fwdService -Port fwdObj"));
         RUN_CALL_SITE_1.clone().setProcedureHandle(hfwdObj).setServerHandle(hWebService).run();
      }));
   }
}

run fwdObj set hfwdObj on hWebService no-error. is converted to code with setProcedureHandle and setServerHandle. On execution in ControlFlowOps.invokeRemoteWithModeImpl this ends up in the second if, although the first if has the comment // this is a RUN ... SET ... ON SERVER webService:

      if (portHandle != null)
      {
         // this is a RUN ... SET ... ON SERVER webService
         invokePersistentImpl(asyncReq, remoteHandle, name, null, portHandle,
                              transactionDistinct, modes, args);
      }
      else
      {
         // this is a RUN ... ON SERVER appServer
         invokeImpl(asyncReq, remoteHandle, null, name,
                    false, false, false,
                    transactionDistinct, modes, args);
      }

Obviously the portHandle has not been set to the InvokeConfig. How is this supposed to work?

This always ends up hitting:

// the target portHandle must always be specified in case of 
// RUN port-type ON SERVER h-web-server statements.
// if is missing, the RUN statement is a no-op

I can't find a place in the code where InvokeConfig.portHandle is set.

So I just want to know if the code is broken and needs to be fixed, or the tests actually work for someone else (probably with a different connect.i).

Galya, this is a regression from back when I rewrote to emit InvokeConfig instead of ControlFlowOps. Please look into rules/convert/control_flow.rules, around this code on line 1234:

               <action>ref = ref.getImmediateChild(prog.kw_set, null)</action>
               <action>runRef = createJavaAst(java.method_call, "setProcedureHandle", runRef)</action>
               <action>ref.putAnnotation("peerid", runRef.id)</action>

If this has a KW_ON, then instead of setProcedureHandle I think is enough to emit a setPortHandle.

Finally to the next issue Host does not match certificate.:

SEVERE | com.goldencode.p2j.net.Dispatcher [Dispatcher.processInbound()] | PID:131472, ThreadName:main, Server Session:00000068, User:bogus | Unexpected throwable.
java.lang.SecurityException: Host does not match certificate.
    at com.goldencode.p2j.util.LowLevelSocketImpl.createSslSocket(LowLevelSocketImpl.java:400)

The service in wsdl has location https://10.39.2.86:9443/ws/wsa/wsafwd. I'm using VPN, so I guess this is the issue. I've added -nohostverify to the connect params, but it does nothing and the client still throws the exception. Is -nohostverify supposed to work?

Without VPN the same bummer, different IP... the local networking one. What do I do, what do I do... Fix the network interfaces or figure out how to use -nohostverify. None sounds good.

Doesn't localhost work instead of your IP?

I'm getting 302 with all SOAP endpoints, when executed from a FWD client (converted procedure). When the endpoints are hit in SoapUI they work as expected. I have no idea what configuration can lead to such outcome. I'm rebasing on latest trunk.

Constantin Asofiei wrote:

There is also a file-cvt-list.txt.rest_and_appserver, to run the plain appserver client tests, but I haven't run these yet. Maybe you can give it a go and post the details here?

I'm out of ideas why the appserver SOAP procedures end up with 302 SOAP fault, when started from the FWD client. I won't be able to run the SOAP tests in FWD client and present the results.

Also it's not clear which set of tests I need to run to complete the task. Only appsrv/runall.p is runnable. As I've mentioned before, the file-cvt-list.txt lists include many procedures that require arguments and are not directly runnable.

I still don't understand what this point involves:

argument translation to remote definition type I think is done the same as the CALL statement does it...

Also, I wanted a confirmation I'm understood, when I wrote in #6417-6 that SESSION:LOCAL-VERSION-INFO:OEClientType can't be set to anything but 4GLCLIENT and APPSERVER due to the fact that a FWD client is not limited to one type of connection to the appserver, in contrast with OE.

Galya, did you find a way to make -nohostverify work again?

Otherwise, AFAIK FWD server sockets bind to all hosts on the machine; I don't understand why localhost is not working for you in the WSDL URL like https://localhost:9443/ws/wsa/wsafwd/wsdl?targetURI=urn:tempuri-org. If this works in the browser, it must work from the FWD code, too.

Constantin Asofiei wrote:

Galya, did you find a way to make -nohostverify work again?

SOAP requests seem to be executed synchronously, so I've used the local context to set / get the value, so this is how it works on 6417a.

Otherwise, AFAIK FWD server sockets bind to all hosts on the machine; I don't understand why localhost is not working for you in the WSDL URL like https://localhost:9443/ws/wsa/wsafwd/wsdl?targetURI=urn:tempuri-org. If this works in the browser, it must work from the FWD code, too.

I get the wsdl fine even from the client, but check the wsdl doc for the url of the endpoints (the location of the service). It uses the eth0 IP.

Galya, I have an explanation for the 302 problem. The reason is the trailing '/' char in the request URL; it is not about localhost vs internal IP (at least here). If you use something like https://localhost:9443/ws/wsa/wsafwd/wsdl in SoapUI, you will see this in the raw log:

View log...View log...

Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "POST /ws/wsa/wsafwd HTTP/1.1[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "Accept-Encoding: gzip,deflate[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "Content-Type: text/xml;charset=UTF-8[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "SOAPAction: ""[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "Content-Length: 277[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "Host: localhost:9443[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "Connection: Keep-Alive[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "User-Agent: Apache-HttpClient/4.5.5 (Java/11.0.21)[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:urn="urn:tempuri-org:fwd">[\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "   <soapenv:Header/>[\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "   <soapenv:Body>[\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "      <urn:pipe_char>[\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "         <urn:inputValue>234</urn:inputValue>[\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "      </urn:pipe_char>[\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "   </soapenv:Body>[\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "</soapenv:Envelope>" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: << "HTTP/1.1 302 Found[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: << "Date: Tue, 23 Jan 2024 16:03:32 GMT[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: << "Location: https://localhost:9443/ws/wsa/wsafwd/[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: << "Content-Length: 0[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: << "Server: Jetty(9.4.22.v20191022)[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: << "[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "POST /ws/wsa/wsafwd/ HTTP/1.1[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "Accept-Encoding: gzip,deflate[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "Content-Type: text/xml;charset=UTF-8[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "SOAPAction: ""[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "Host: localhost[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "Content-Length: 277[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "Connection: Keep-Alive[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "User-Agent: Apache-HttpClient/4.5.5 (Java/11.0.21)[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:urn="urn:tempuri-org:fwd">[\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "   <soapenv:Header/>[\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "   <soapenv:Body>[\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "      <urn:pipe_char>[\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "         <urn:inputValue>234</urn:inputValue>[\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "      </urn:pipe_char>[\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "   </soapenv:Body>[\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: >> "</soapenv:Envelope>" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: << "HTTP/1.1 200 OK[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: << "Date: Tue, 23 Jan 2024 16:03:32 GMT[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: << "Content-Type: text/xml[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: << "Transfer-Encoding: chunked[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: << "Server: Jetty(9.4.22.v20191022)[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: << "[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: << "183[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: << "<?xml version='1.0' encoding='utf-8'?><SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><SOAP-ENV:Body><pipe_charResponse xmlns="urn:tempuri-org:fwd"><result xsi:nil="true"/><outputValue>234</outputValue></pipe_charResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: << "[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: << "0[\r][\n]" 
Tue Jan 23 18:03:32 EET 2024:DEBUG: << "[\r][\n]" 

I don't think we can make Axis2 to follow the 302 redirect. There is this code being executed when trying to follow redirects:

    /**
     * Entity enclosing requests cannot be redirected without user intervention
     * according to RFC 2616.
     *
     * @return false.
     *
     * @since 2.0
     */
    public boolean getFollowRedirects() {
        return false;
    }

PostMethod(EntityEnclosingMethod).getFollowRedirects() line: 213    
HttpMethodDirector.isRedirectNeeded(HttpMethod) line: 819    
HttpMethodDirector.executeMethod(HttpMethod) line: 178    
HttpClient.executeMethod(HostConfiguration, HttpMethod, HttpState) line: 397    
HTTPSender(AbstractHTTPSender).executeMethod(HttpClient, MessageContext, URL, HttpMethod) line: 621    
HTTPSender.sendViaPost(MessageContext, URL, String) line: 193    
HTTPSender.send(MessageContext, URL, String) line: 75    
CommonsHTTPTransportSender.writeMessageWithCommons(MessageContext, EndpointReference, OMOutputFormat) line: 404    
CommonsHTTPTransportSender.invoke(MessageContext) line: 231    
AxisEngine.send(MessageContext) line: 443    
OutInAxisOperationClient.send(MessageContext) line: 406    
OutInAxisOperationClient.executeImpl(boolean) line: 229    
OutInAxisOperationClient(OperationClient).execute(boolean) line: 165    
ServiceClient.sendReceive(QName, OMElement) line: 555    
WebServiceImpl.invoke(int, String, String, String) line: 820    
WebServiceMethodAccess.invoke(Object, int, Object...) line: not available    

The redirection is happening on the FWD server-side, somewhere in Jetty. Please debug and find if/how this can be configured, maybe we need to handle implicitly the target with a trailing '/'.

Otherwise, you can add the trailinig '/' in fwd.wsdl at <soap:address location="/wsa/wsafwd/" />, and the request gets posted, but there are still errors (something related to namespace, see SoapHandler.namespaceMatch). Also, I've used -WSDL https://localhost:9443/ws/wsa/wsafwd/wsdl?targetURI=urn:tempuri-org -Port fwdObj -nohostverify for the connection string and this patch:

=== modified file 'src/com/goldencode/p2j/soap/SoapHandler.java'
--- old/src/com/goldencode/p2j/soap/SoapHandler.java    2023-05-12 10:05:12 +0000
+++ new/src/com/goldencode/p2j/soap/SoapHandler.java    2024-01-23 16:14:19 +0000
@@ -702,7 +702,7 @@
          return;
       }

-      if (target.endsWith("/wsdl"))
+      if (target.endsWith("/wsdl") || target.endsWith("/wsdl/"))
       {
          String targetURI = request.getParameter("targetURI");
          Document doc = targetURI == null ? null : wsdlDocs.get(targetURI);

=== modified file 'src/com/goldencode/p2j/util/WebServiceImpl.java'
--- old/src/com/goldencode/p2j/util/WebServiceImpl.java 2023-05-19 19:46:27 +0000
+++ new/src/com/goldencode/p2j/util/WebServiceImpl.java 2024-01-23 16:05:43 +0000
@@ -82,6 +82,7 @@
 import org.apache.axis2.context.*;
 import org.apache.axis2.transport.http.*;
 import org.apache.axis2.wsdl.*;
+import org.apache.commons.httpclient.MultiThreadedHttpConnectionManager;
 import org.apache.ws.commons.schema.*;
 import javax.wsdl.*;
 import javax.wsdl.Binding;
@@ -427,7 +428,7 @@
          }

          service = (Service) services.values().iterator().next();
-         Port servicePort = getImplicitSoapPort(service.getPorts().values());
+         Port servicePort = options.getPortName() != null ? service.getPort(options.getPortName()) : getImplicitSoapPort(service.getPorts().values());
          if (servicePort == null)
          {
             noServicesError();
@@ -805,6 +806,13 @@
             sc.getOptions().setProperty(HTTPConstants.AUTHENTICATE, auth);
          }

+         MultiThreadedHttpConnectionManager httpmgr = new MultiThreadedHttpConnectionManager();
+         httpmgr.getParams().setBooleanParameter(LowLevelSocketImpl.P2J_NOHOSTVERIFY, 
+                                                 wsd.isNoHostVerify());
+         sc.getServiceContext()
+           .getConfigurationContext()
+           .setProperty(HTTPConstants.MULTITHREAD_HTTP_CONNECTION_MANAGER, httpmgr);
+
          sc.getOptions().setProperty(LowLevelSocketImpl.P2J_NOHOSTVERIFY, 
                                      wsd.isNoHostVerify());

I was thinking about redirect, but I was expecting 301 instead, so didn't dig deeper. In Jetty logs on the server nothing is found related to 301 / 302 / redirect.

Do we expect it to be some kind of misconfiguration in testcases, since prod apps are working fine?

Galya B wrote:

Do we expect it to be some kind of misconfiguration in testcases, since prod apps are working fine?

I don't think so. Please try and make both paths (with and without trailing '/' char) handled by SoapHandler - this should fix it.

<wsdl:definitions xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:S1="urn:soap-fault:details" xmlns:S2="urn:tempuri-org:fwd" xmlns:S3="urn:tempuri-org:persistent" xmlns:prodata="urn:schemas-progress-com:xml-prodata:0001" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:tns="urn:tempuri-org" xmlns:xsd="http://www.w3.org/2001/XMLSchema" name="fwd" targetNamespace="urn:tempuri-org">

<wsdl:operation name="pipe_char">
   <wsdl:input message="tns:fwd_pipe_char"/>
   <wsdl:output message="tns:fwd_pipe_charResponse"/>
   <wsdl:fault message="tns:FaultDetailMessage" name="fwdFault"/>
</wsdl:operation>

pipe_char (called by test_pipe_char.p) is one of the successful procedures, when executed from SoapUI, but not from the FWD client.
The operation uses the tns namespace, that is urn:tempuri-org, and it is the namespace of OMElement elRequest (where the variable ns comes from). Later we have a check that should be successful to return a result:

               if (namespaceMatch(ns, portOp.getInput()) &&
                   namespaceMatch(ns, portOp.getOutput()) &&
                   namespaceMatch(ns, portOp.getFault("FaultDetailMessage")))

In SoapHandler.namespaceMatch the following code msg.getPart(el instanceof Fault ? "FaultDetail" : "parameters").getElementName().getNamespaceURI() returns urn:tempuri-org:fwd, which doesn't match. That is because the actual input, output and fault messages of pipe_char are from other namespaces S1 (urn:soap-fault:details) and S2 (urn:tempuri-org:fwd):

<wsdl:message name="FaultDetailMessage">
   <wsdl:part element="S1:FaultDetail" name="FaultDetail"/>
</wsdl:message>
<wsdl:message name="fwd_pipe_char">
   <wsdl:part element="S2:pipe_char" name="parameters"/>
</wsdl:message>
<wsdl:message name="fwd_pipe_charResponse">
   <wsdl:part element="S2:pipe_charResponse" name="parameters"/>
</wsdl:message>

Constantin, I have no idea why the check and what can we do about it. Maybe everything used by testcases should be in one namespace?

The SOAP envelop is different.

SoapUI:

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:urn="urn:tempuri-org:fwd">
   <soapenv:Header/>
   <soapenv:Body>
      <urn:pipe_char>
         <urn:inputValue>Mihai</urn:inputValue>
      </urn:pipe_char>
   </soapenv:Body>
</soapenv:Envelope>

FWD client:

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Header />
   <soapenv:Body>
      <tns:pipe_char xmlns:tns="urn:tempuri-org">
         <tns:inputValue>Mihai</tns:inputValue>
      </tns:pipe_char>
   </soapenv:Body>
</soapenv:Envelope>

The targetNamespace should be used only for the wsdl elements declared without an explicit namespace, but in WebServiceImpl.invoke the client prepares the SOAP request by simply adding the targetNamespace to the request and then the server in SoapHandle.handle checks if that same target namespace corresponds to the namespaces of input, output of the operation, which they obviously don't according to the wsdl itself.

The root wsdl tag <wsdl:definitions> has one targetNamespace, while the <schema> for the operation has a different one in testcases. The FWD client was building the request body with the targetNamespace from <wsdl:definitions>, so I'm changing it to take the schema one. It seems to work that way.

Someone familiar with how FWD clients are used in prod for accessing soap services needs to validate the behavior in the testing phase.

Help!

Is this statement expected to throw an exception in the client?

private static final InvokeConfig RUN_CALL_SITE_15 = new InvokeConfig().setTarget("throw_error").setModes("IO");

[..]

silent(() -> RUN_CALL_SITE_15.clone().setInHandle(hfwdObj).run(new integer(9407), soapResult));

It's in a silent wrapper, but the external procedure execution still throws client-side. It is a soap request that fails.

Do we need to fix this behavior by informing the client somehow about no-error?

Galya, what test are you executing from the 4GL client via soap?

Constantin Asofiei wrote:

Galya, what test are you executing from the 4GL client via soap?

<cmd-line-option startup-procedure="appsrv/soap/test/4glws/test_throw_error.p" />

block-level on error undo, throw.
{common/log_hlp.i}

define variable outChar as character no-undo.

run log-it('ABL 4glws throw error procedure : - start').

{appsrv/soap/test/4glws/connect.i}

run throw_error in hfwdObj(input 9407, output soapResult) no-error.
assert-err-num(9407,
    substitute({&msg_call_error}, 'throw_error', ''), 
    {&line-number}).

run log-it('ABL 4glws throw error procedure : - end').

catch e as Progress.Lang.AppError :
    message e:ReturnValue.        
end catch.

finally:
{appsrv/soap/test/4glws/disconnect.i}    
end finally.

I think some legacy 4GL code is converted to SOAP endpoints and is executed from 4GL like WSA, ref in the table #6417-6. Please tell me if there is another one as well, I couldn't find any information in our docs / wikis.

Constantin, you won't be able to run appsrv/soap/test/4glws/test_throw_error.p from 4GL client without the fix for the namespaces in 6417a. I was asking about the expected behavior, because it seems very messy. I wonder if some customers call SOAP from 4GL clients in prod (I've seen some -WSDL connects in one of the customers, but I can't figure out if it's just a test or actually used in prod). I don't want to change core behavior if there are some workarounds working for customers.

I'm confused about this test. Galya, I assume you've added the input 9407 argument.

Marian: can you please advise with the above test? I assume error 9407 needs to be thrown by this call, and the catch block does nothing?

Galya B wrote:

I think some legacy 4GL code is converted to SOAP endpoints and is executed from 4GL like WSA, ref in the table #6417-6. Please tell me if there is another one as well, I couldn't find any information in our docs / wikis.

What do you mean here? The .wsm file configures which programs are exposed as SOAP.

Constantin Asofiei wrote:

I'm confused about this test. Galya, I assume you've added the input 9407 argument.

Yes, I've added 9407, because the external function throw_error.p requires an input param and throws error otherwise for invalid params. I'm also confused about the test.

Constantin Asofiei wrote:

Galya B wrote:

I think some legacy 4GL code is converted to SOAP endpoints and is executed from 4GL like WSA, ref in the table #6417-6. Please tell me if there is another one as well, I couldn't find any information in our docs / wikis.

What do you mean here? The .wsm file configures which programs are exposed as SOAP.

.wsm configure server-side the endpoints. The client is WSA (Web Services Adapter). Isn't WSA code converted and executed by FWD clients?

Galya B wrote:

.wsm configure server-side the endpoints. The client is WSA (Web Services Adapter). Isn't WSA code converted and executed by FWD clients?

I think I understand what you mean - when a SOAP request gets executed (regardless if is via SoapUI, another 4GL program, etc), the request is handled by a FWD client - and I think you can name these clients 'WSA'.

Constantin Asofiei wrote:

Galya B wrote:

.wsm configure server-side the endpoints. The client is WSA (Web Services Adapter). Isn't WSA code converted and executed by FWD clients?

I think I understand what you mean - when a SOAP request gets executed (regardless if is via SoapUI, another 4GL program, etc), the request is handled by a FWD client - and I think you can name these clients 'WSA'.

Do you mean the appserver client?

Constantin Asofiei wrote:

I'm confused about this test. Galya, I assume you've added the input 9407 argument.

Marian: can you please advise with the above test? I assume error 9407 needs to be thrown by this call, and the catch block does nothing?

Hmm, looks like the throw_error.p procedure was changed to take this additional parameter, it was most probably done for REST but then SOAP tests we had before were not updated. The thing I wanted to see there was the result when throwing an AppError with return-value vs. one with num-messages.

The 4GL test needs to be updated to send this new parameter as input - test_throw_error.p must send 9470 as input parameter when calling throw_error on the server, I will fix that on testcases.

Marian Edu wrote:

I will fix that on testcases.

And of course the proxy gen file must be updated and you might probably need the `wsdl` file as well.

Constantin, I think we need to fix the FWD client to never throw an exception from a faulty SOAP request but instead leave it to the server, so that no-error takes effect.

Galya B wrote:

Constantin, I think we need to fix the FWD client to never throw an exception from a faulty SOAP request but instead leave it to the server, so that no-error takes effect.

That is in WebServiceImpl.invoke:

ErrorManager.recordOrThrowError(11506,
               String.format(msg, operationName, faultMsg), false, true);

Marian, can you tell me how AIA and WSA are migrated to FWD? I made an attempt in #6417-6 to describe what I've found in the documentation, but it seems incomplete. These are adapters, AIA from what I read in 4GL doc is front-end lib that allows requests to the appserver. So my assumption is that in Progress there is an actual code written for the client using AIA and WSA.

Galya B wrote:

Marian, can you tell me how AIA and WSA are migrated to FWD? I made an attempt in #6417-6 to describe what I've found in the documentation, but it seems incomplete. These are adapters, AIA from what I read in 4GL doc is front-end lib that allows requests to the appserver. So my assumption is that in Progress there is an actual code written for the client using AIA and WSA.

Gayla, I think Constantin was working on FWD implementation - the 4GL client in this case is making a SOAP call and it just happens the SOAP endpoints are running on the AppSrv so it's the WSA involved somewhere in between but this is irelevant for the client, it can well be any SOAP endpoint. Normally the 4GL client will call in directly on an appsrv but those tests were just one way for Constantin to test the SOAP implementation in FWD.

REST and SOAP endpoints in FWD don't correspond directly to a certain adapter in Progress and these adapters are not converted to code in FWD, so AIA and WSA are not applicable values for clientType.

Galya B wrote:

REST and SOAP endpoints in FWD don't correspond directly to a certain adapter in Progress and these adapters are not converted to code in FWD, so AIA and WSA are not applicable values for clientType.

Exactly. WSA in OpenEdge is part of the middle layer deployed in tomcat - see https://docs.progress.com/bundle/openedge-application-and-integration-services-117/page/WSA-configuration-and-Web-service-deployment-tools.html, there is no 'client' equivalent in FWD terms for it.

For AIA, it may be for the 'legacy web', but I don't recall exactly. But, if lets say an appserver agent must identify itself as WSA/AIA for the duration of a request, depending on what kind of request is executes, then we need to implement this. Marian: does this ring a bell?

Marian Edu wrote:

The 4GL test needs to be updated to send this new parameter as input - test_throw_error.p must send 9470 as input parameter when calling throw_error on the server, I will fix that on testcases.

Please make the 4GL test which executes the SOAP request more 'explicit', to understand how the remote error propagates to the caller (if applicable).

Constantin Asofiei wrote:

For AIA, it may be for the 'legacy web', but I don't recall exactly. But, if lets say an appserver agent must identify itself as WSA/AIA for the duration of a request, depending on what kind of request is executes, then we need to implement this.

AIA is indeed 'legacy web', I've found a reference to LegacyWebServiceHandler somewhere in the documentation. And in Progress doc I've found AIA has a code (like a library) on the front-end. AIA is to be executed from the front-end.

The issue is that even the request can't be marked as AIA / WSA, because the context can come either from the endpoint conversion, or the client library / sdk / adapter. There is nothing in the endpoint identifying its relationship to one of the original types, and there is nothing in the client.

Galya B wrote:

The issue is that even the request can't be marked as AIA / WSA, because the context can come either from the endpoint conversion, or the client library / sdk / adapter. There is nothing in the endpoint identifying its relationship to one of the original types, and there is nothing in the client.

What I mean is that when the apsperver agent is executing a request from SoapHandler, that agent can identify 'client-type' as WSA (if this is how 4GL does it). Same for WebServiceHandler, if the request originates from this, then it can report AIA.

• LegacyWebServiceHandler is extended by:
  • RestHandler / rest/enabled
  • SoapHandler / soap/enabled)
• WebServiceHandler - implementation of OpenEdge.Web.WebHandler that handles OpenEdge.Web.WebRequest / webHandler/enabled.

In OE: https://community.progress.com/s/article/What-are-the-PASOE-transports

For Classic UBrokers, when the client needs to connect through the internet (HTTP/HTTPS), separate and distinct adapters AIA/WSA/REST/Mobile need to be installed and configured so that the client can connect via the AIA adapter to the Appserver for example.

PAS for OpenEdge is a Web server, which means that there is no need to install and configure adapters on a separate Web server. To put it simply: "transports replace adapters":

Transport	URL path
APSV	/apsv
REST	/rest
SOAP	/soap
WEB	/web

Basically, all our endpoints created by the three above-mentioned FWD classes create PASOE endpoints. So we can't mark the requests as coming from an adapter client to a legacy appserver.

AIA is configured in OE with jar in Tomcat and a servlet for the /aia path that calls com.progress.aia.Aia to handle the request: https://community.progress.com/s/article/P109721 .

https://docs.progress.com/bundle/openedge-classic-appserver-administration-117/page/AppServer-Internet-Adapter.html

The AppServer Internet Adapter (AIA) allows ABL clients and Open Clients (.NET and Java™) to access the AppServer or the OpenEdge Adapter for SonicMQ over the Internet.
The AIA is installed and runs as a Java servlet in most any Java container, which provides the Internet access to the AppServer.

So 4GL knows it's AIA when their jar is installed to handle the requests to /aia. The request itself to the aia endpoint is nothing unique.

The same with WSA. It's a custom handler mapped to a servlet path /wsa (https://docs.progress.com/bundle/openedge-webservices-development-117/page/Deploying-a-SOAP-Web-service.html):

The WSA is a Java servlet running on a Web server or stand-alone Java Servlet Engine (JSE). Situated between the client and AppServer, this Java servlet understands how to pass communications at run time between a Web service client and the AppServer.

WebSpeed is a web server with deployed WebSpeed Messenger (OE code) + logic in SpeedScript executed server-side (similar to .php):

WebSpeed Transaction Server becomes an application server for Web browser clients. The Transaction Server runs the SpeedScript, which generates HTML pages, and returns these Web pages to Web browser clients through the Messenger in response to Post and Get requests from the clients.
The Messenger runs on a Web server [..] and acts as the gateway and translator between Web requests and responses on the Web server side and the corresponding WebSpeed requests and responses on the Transaction Server side.

Again, we don't have that intermediate code that marks normal POST / GET requests as WebSpeed.

If any of those requests originate from OpenEdge.Web.WebHandler, it means the clientType is 4GLCLIENT.

We should have full support for WSA/PASOE SOAP and for REST Adapter/PASOE REST. I would expect that we can control any marking for these pretty easily.

We don't have proper webspeed support yet, at least not the "classic" version of webspeed that is implemented as CGI. We do support the webhandler, which is a kind of "do it yourself" web hook. In OE can handle some more complicated web requests which can't otherwise be handled. So I would say that our support for class webspeed and the PASOE WEB transport is not there, other than the webhandler portion (which only works in PASOE? I'm not sure about that). We do intend to add support for webspeed/PASOE WEB in #6506 (at least for the core CGI).

We don't support AIA, which I thought was only the gateway functionality in classic appserver where a web server is used to terminate the SSL connection and it just connects to appserver as a proxy. Right now we have no plans to support this. I thought that AIA doesn't have a PASOE equivalent for this, though perhaps I am mis-remembering.

In FWD the three types of legacy endpoints handlers are simply generated by openapi.openedge.export, and get converted each to a LegacyService with a type (REST, SOAP, WEBHANDLER). Again those types don't correspond to the above adapters or any specific client. In OE the clients of openapi are ABL / Java / .NET. It doesn't have to do with how the endpoints are generated.

My point is that if we need to know whether we are in classic REST, PASOE REST, WSA (classic SOAP), PASOE SOAP... we can determine this. These services are configured explicitly and we can mark them as classic or PASOE (if we don't already).

• 4GLCLIENT when an external procedure is running on legacy appserver from 4GL code
• JAVACLIENT when an external procedure is running on legacy appserver from Java code
• WSA when an external procedure is running on legacy appserver after -WSDL connect.
• APPSERVER when a startup procedure is running on legacy appserver
• MULTI-SESSION-AGENT when a startup procedure is running on pasoe appserver
• ? when an endpoint is executed without session (i.e. from SoapUI) (on PASOE only?)

clientType can have two values divided by comma, because only 4GL/Java/.NET are actual "clients", while AIA, WSA, WebSpeed are web adapters and both can co-exist.

AIA is neither REST, nor SOAP. It's one endpoint /aia with OE servlet handling custom HTTP requests generated by 4GL / OE Java / .NET clients. The so called Open Clients Java / .Net use OE classes - com.progress.open4gl.javaproxy.OpenAppObject / Progress.Open4GL.Proxy.OpenAppObject. 4GL clients probably have to use a custom implementation of OpenEdge.Web.WebHandler.

The annotation we have in our testcases openapi.openedge.export is probably used to open OpenEdge procedures to the servlet-adapter. In FWD we create separate endpoints for each procedure, while this is not the case in OE, where it's only one endpoint.

AIA is not FWD's WEBHANDLER. WebServiceHandler is simply implementation of OpenEdge.Web.WebHandler, that is (https://documentation.progress.com/output/oehttpclient/oe117/index.html):

Abstract class to use as a basis for building WebSpeed for PASOE request handlers

I don't understand how / if WEBHANDLER is used currently in FWD.

WEBSPEED is not REST or SOAP, it's a script server-side (like php) + custom GET/POST requests.

There is also the RA or REST Management Agent, but it's not in the clientType.

See, the paradox displayed by testcases:

appsrv/soap/test/4glws/test_throw_error.p connects to the SOAP agent with appsrv/soap/test/4glws/connect.i:

hWebService:connect ('-WSDL https://localhost:9443/ws/wsa/wsafwd/wsdl/?targetURI=urn:tempuri-org -Service fwdService -Port fwdObj').

Then appsrv/soap/test/4glws/test_throw_error.p runs throw_error, that is generated by appsrv/api/throw_error.p annotated as REST:

@openapi.openedge.export FILE(type="REST", executionMode="external", useReturnValue="false", writeDataSetBeforeImage="false").

This 'REST' endpoint ends up in the WSDL. So at this point we can't even determine the correct protocol by the openapi annotation.

The only way to mark clientType as WSA is to suppose all SOAP connections to legacy appserver are WSA.

I need to know where openapi.openedge.export is used as annotation in the original OE? I've found some references in their documentation, but it's not clear when it's used for what purpose and if it's manually or auto generated.

Galya B wrote:

I need to know where openapi.openedge.export is used as annotation in the original OE? I've found some references in their documentation, but it's not clear when it's used for what purpose and if it's manually or auto generated.

AFAIK 4GL annotations have no meaning at runtime, even the compiler treats them as comments. I don't recall if these are auto generated or hand-written (maybe they can be both?).

For note #6417-122 - you can not rely on the 4GL code being executed (a 4GL procedure can be annotated for both SOAP and REST, there is no restriction). What matters is who calls this code: modify the invoke code at the soap/rest/web handlers to pass a parameter which specifies a 'client-type override'; this will be passed to the agent and, when the override is set, it will report this client-type.

I've found an example of how the annotation openapi.openedge.export is created: https://community.progress.com/s/question/0D54Q00007qckNxSAI/creating-an-openedge-abl-service-using-openedge-1170. The example is for the /web transport (WebSpeed service) in PASOE servers. Business entity is created with resource url and crud operations and thus a .cls file is generated with annotations including openapi.openedge.export (REST). A Data Object service is created by importing the .cls and the annotation progress.service.resource defines the url path and name of the json object.

Another way to generate REST annotations for .p or .cls files in Project Explorer (legacy appserver): https://docs.progress.com/bundle/openedge-developer-studio-olh-117/page/Defining-REST-service-annotations.html. The OE tool 'REST Expose Editor' lists resources annotated with REST to associate an operation with a verb for a REST resource.

On the other hand SOAP doesn't seem to use the same annotations. https://docs.progress.com/bundle/openedge-webservices-development-117/page/Web-service-objects-and-the-WSDL-file.html :

During Web service deployment, the WSA generates a WSDL file that defines one or more Web service objects to describe the Web service. These Web service objects conform to the Open Client object model.

The legacy appserver WSA adapter is supposed to be used server-side, while clients are Java/.NET open clients. The server and client interfaces are generated with ProxyGen and rely on objects AppObject and SubAppObject.

For PASOE the soap services are deployed with a script deploySOAP.[bat|sh] using .wsm file generated by ProxyGen.

Question: AIA is simply allowing to connect to legacy appserver over http(s). This is mind-blowing. So the question: Is allowing the appserver to be accessed from another host via http enabled by default in FWD? Should server:connect("-URL http//IP/aia") work without any FWD configurations?

We don't support AIA and have no plans to do so.

Greg Shah wrote:

We don't support AIA and have no plans to do so.

AIA is enabling server:connect("-URL http//IP/aia") generally speaking and we already have it. We may not have the servlet adapter and Java / .NET clients with their interfaces, but the purpose of AIA is simply allowing internet connections to the legacy appserver, so we have that fulfilled. My question was if it's hidden behind configuration similar to OE or not and I think it's not.

https://docs.progress.com/bundle/openedge-classic-appserver-administration-117/page/AppServer-Internet-Adapter.html

The AppServer Internet Adapter (AIA) allows ABL clients and Open Clients (.NET and Java™) to access the AppServer or the OpenEdge Adapter for SonicMQ over the Internet.

This is my final conclusion after a lot of research and this is to be implemented:

I need some clarification on the resource/proxy/classloader thing: SOAPFaultImpl inherits HandleResource and the parent constructor calls ProxyFactory.isProxyClass, checking if ProxyFactory.loader (AsmClassLoader) is the same as SOAPFaultImpl.getClassLoader (AppClassLoader).

First, I don't understand what this is all about.
Then, ProxyFactory.loader is null, because AsmClassLoader hasn't been instantiated, so the client can't send SOAPFaultImpl.

Galya B wrote:

I need some clarification on the resource/proxy/classloader thing: SOAPFaultImpl inherits HandleResource and the parent constructor calls ProxyFactory.isProxyClass, checking if ProxyFactory.loader (AsmClassLoader) is the same as SOAPFaultImpl.getClassLoader (AppClassLoader).

You mean this code?

      // this will ensure the INSTANTIATING-PROCEDURE attribute is setup accordingly and also that
      // static resources will be registered with the correct top-level block
      if (processResource() && !ProxyFactory.isProxyClass(this.getClass()))
      {
         // for proxy cases, explicit registration is needed
         pm.processResource(this, dynamic);
      }

First, I don't understand what this is all about.

You have buffers which are proxies, and the processing can't be done when the buffer proxy is instantiated.

Then, ProxyFactory.loader is null, because AsmClassLoader hasn't been instantiated, so the client can't send SOAPFaultImpl.

Do you have -Djava.system.class.loader=com.goldencode.p2j.classloader.MultiClassLoader ?

Constantin Asofiei wrote:

Then, ProxyFactory.loader is null, because AsmClassLoader hasn't been instantiated, so the client can't send SOAPFaultImpl.

Do you have -Djava.system.class.loader=com.goldencode.p2j.classloader.MultiClassLoader ?

It fixes the NPE, but I wonder why is MultiClassLoader needed to return a SOAP fault? Is it something customers know about when using SOAP from ABL?

Galya B wrote:

... but I wonder why is MultiClassLoader needed to return a SOAP fault? Is it something customers know about when using SOAP from ABL?

I don't understand what you mean. MultiClassLoader is a requirement for the FWD server. How is this affecting the SOAP fault processing?

Constantin Asofiei wrote:

Galya B wrote:

... but I wonder why is MultiClassLoader needed to return a SOAP fault? Is it something customers know about when using SOAP from ABL?

I don't understand what you mean. MultiClassLoader is a requirement for the FWD server. How is this affecting the SOAP fault processing?

The issue I'm describing is in the FWD client, when a procedure is called via SOAP on the appserver. So the above-mentioned check is done client-side. I'm adding MultiClassLoader as class loader to the client run script.

Galya B wrote:

The issue I'm describing is in the FWD client, when a procedure is called via SOAP on the appserver. So the above-mentioned check is done client-side. I'm adding MultiClassLoader as class loader to the client run script.

OK, I understand now what you mean. SoapFaultImpl is instantiated on client-side, in WebServiceImpl.invoke. This isn't right. This code needs to be fixed so this will happen on server-side.

Constantin Asofiei wrote:

Galya B wrote:

The issue I'm describing is in the FWD client, when a procedure is called via SOAP on the appserver. So the above-mentioned check is done client-side. I'm adding MultiClassLoader as class loader to the client run script.

OK, I understand now what you mean. SoapFaultImpl is instantiated on client-side, in WebServiceImpl.invoke. This isn't right. This code needs to be fixed so this will happen on server-side.

Do we have customers calling soap endpoints from ABL? I wonder what I'm getting into making such core changes.

I need some clarification on the test in appsrv/soap/test/4glws/test_throw_error.p:

block-level on error undo, throw.
{common/log_hlp.i}

define variable outChar as character no-undo.

run log-it('ABL 4glws throw error procedure : - start').

{appsrv/soap/test/4glws/connect.i}

run throw_error in hfwdObj(input 9407, output soapResult) no-error.
assert-err-num(9407,
    substitute({&msg_call_error}, 'throw_error', ''), 
    {&line-number}).

run log-it('ABL 4glws throw error procedure : - end').

catch e as Progress.Lang.AppError :
    message e:ReturnValue.        
end catch.

finally:
{appsrv/soap/test/4glws/disconnect.i}    
end finally.

The error thrown by the external procedure should be silenced by no-error, then what AppError is caught after the end log?

Galya B wrote:

The error thrown by the external procedure should be silenced by no-error, then what AppError is caught after the end log?

Normally none, it's just that we used to have this `finally` block in all tests just to make sure we don't miss anything :)

You can safely ignore that in this case, as said most probably was only added as a default.

On QuitConditionException the appserver agent gets disconnected, but then AgentPool.idsToAppserver gets empty and consecutive requests from external clients fail due to NPE.

The OE doc says:

When QUIT is executed from within a procedure running on an AppServer, it terminates the ABL session running on the AppServer, causing the AppServer agent to shut down and returns to the ABL client session from which it was spawned.

So I guess this is the behavior disconnect in Agent is simulating. The issue is that Agent disconnects the AgentPool, but AppServerHelper for the LegacyServiceWorker remains connected. I think I've fixed it by disconnecting the helper in SoapHandler, but it may need to be fixed for other handlers too.

There are several common error messages for the SOAP requests, but it's difficult to compile requirements for when each one should be returned, because FWD has a very different execution environment / process. So I'll try to summarize here what shows up in the tests:

• An error was detected while executing the Web Service request. (10893)
  • classic test cases: dataset, exceptions, primitives, table, type
  • PASOE test cases: same
• An error was detected in the Web Service request. (10894)
  • classic test cases: primitives, table, exceptions
  • PASOE test cases: primitives, table, type
• An internal Web Service error occurred in the Web Services Adapter. (10902)
  • classic test cases: dataset, exceptions, type
  • PASOE test cases: not relevant
• Exception running ABL procedure:
  • we have it only in PASOE tests, but in webtests_from_logs dir there are also examples from classic SOAP requests
  • classic test logs: dataset
  • PASOE test cases: dataset, exceptions, primitives, type

There is already an attempt to match some errors to exceptions in the code, but it's far from sufficient. I may need to focus on everything else left in this task, because this issue is quite the puzzle.

I'm not sure if my reasoning is correct, but it looks like the same appserver agent can execute all types of appserver tasks (rest, soap, webhandler, classic, etc). The tasks get into a queue and are running in sequence. This means client type for the session (the connection between the agent and the server) should be set on task start and reset on task complete. Unfortunately the task itself is presented by Runnable and can't reveal where it comes from (the client type). And this makes simply setting an attr to the session not so simple.

• Agent.AgentCommand.doExecuteImpl
• LegacyServiceWorker.run

I'm not sure if my reasoning is correct, but it looks like the same appserver agent can execute all types of appserver tasks (rest, soap, webhandler, classic, etc). The tasks get into a queue and are running in sequence.

This is correct.

Unfortunately the task itself is presented by Runnable and can't reveal where it comes from (the client type). And this makes simply setting an attr to the session not so simple.

We could require the usage of a new interface that extends Runnable and has a method that returns an enum instance which describes the type.

Test in OE for client type:

Procedure running in batch client:

def var h as handle.

create server h.
h:connect("-DirectConnect -S 3111 -H localhost -sessionModel Session-Free", "", "").

message "session:local-version-info:OEClientType " session:local-version-info:OEClientType view-as alert-box error. 
message "session:current-request-info " session:current-request-info view-as alert-box error. 
message "session:current-response-info " session:current-response-info view-as alert-box error. 
message "h:request-info:OEClientType " h:request-info:VersionInfo:OEClientType view-as alert-box error. 
message "h:response-info:OEClientType " h:response-info:VersionInfo:OEClientType view-as alert-box error. 

run typeonapp.p on server h.

h:disconnect().
delete object h.

Results in clientlog:

4GLCLIENT
?
?
4GLCLIENT
JAVACLIENT

External procedure typeonapp.p running on appserver:

message "session:local-version-info:OEClientType " session:local-version-info:OEClientType view-as alert-box error. 
message "session:current-request-info " session:current-request-info view-as alert-box error. 
message "session:current-response-info " session:current-response-info view-as alert-box error. 

Results in appserver log:

APPSERVER
4GLCLIENT
APPSERVER

The same results with AIA (connecting to appserver via http url), when using localhost, i.e.:

h:connect("-URL http://localhost:8080/aia/Aia?AppService=gbb -sessionModel Session-Free").

As you can see the types in the request and response are different. Also JAVACLIENT came up from nowhere in the batch 4GL client. Also, AIA adapter is not indicating AIA client, at least testing with localhost.

Marian, do you have any idea why this test executed as batch in 4GL gives the following result:

def var h as handle.

create server h.
h:connect("-DirectConnect -S 3111 -H localhost -sessionModel Session-Free", "", "").

message h:response-info:VersionInfo:OEClientType.

h:disconnect().
delete object h.

Message displayed: JAVACLIENT.

I have no idea how it can be replicated in FWD if it's not explainable.

Galya B wrote:

Marian, do you have any idea why this test executed as batch in 4GL gives the following result:

[...]

Message displayed: JAVACLIENT.

I have no idea how it can be replicated in FWD if it's not explainable.

Gayla, we don't have the 'classic' application server available anymore so we can only test using the 'pacific' (pasoe) Tomcat based version... but anyway, that 'RESPONSE-INFO' is available even before connect - to set the 'CLIENT-PRINCIPAL' mainly. If you run that from 4GL client I don't see how it could ever be 'JAVACLIENT' unless it just somehow points to the last request served by the app server agent. In you test you didn't actually ran anything on the app server, try to do a call and see if that changes anything.

com.goldencode.p2j.util.decimal has:

   /** Maximum precision scale applied to any decimal instance */
   public static final int MAX_SCALE = 10;

But testcases test pipe_decimal_decimal2 in SOAP UI test suite appsrv/rest/test/fwd-soapui-project.xml expects the input number 2.123456789123456789 to be returned as output. The same test exists for PASOE.

Constantin, Marian, can you please elaborate on how these two can co-exist?

Greg, how is this going to work? I have more questions than the number of failing tests (quite a few and all over the place).

If it was only to fix the points in #6417-2 that's doable. Otherwise each one of the other failing tests is a task on its own. Like reworking decimal class.

Marian Edu wrote:

Galya B wrote:

Marian, do you have any idea why this test executed as batch in 4GL gives the following result:

[...]

Message displayed: JAVACLIENT.

I have no idea how it can be replicated in FWD if it's not explainable.

Gayla, we don't have the 'classic' application server available anymore so we can only test using the 'pacific' (pasoe) Tomcat based version... but anyway, that 'RESPONSE-INFO' is available even before connect - to set the 'CLIENT-PRINCIPAL' mainly. If you run that from 4GL client I don't see how it could ever be 'JAVACLIENT' unless it just somehow points to the last request served by the app server agent. In you test you didn't actually ran anything on the app server, try to do a call and see if that changes anything.

The following proc outputs JAVACLIENT and then APPSERVER on my appserver and APPSERVER twice on a different appserver:

def var h as handle.

create server h.
h:connect("-DirectConnect -S 3111 -H localhost -sessionModel Session-Free", "", "").

message h:response-info:VersionInfo:OEClientType.

run simple.p on server h.

message h:response-info:VersionInfo:OEClientType.

h:disconnect().
delete object h.

I remember that I've configured AIA (tomcat) a while ago, but it still doesn't make complete sense, because this proc is executed with a pure 4GL syntax. The connect should not go through Tomcat without -URL and the /aia context. I'll have to figure something out.

Galya B wrote:

com.goldencode.p2j.util.decimal has:

[...]

But testcases test pipe_decimal_decimal2 in SOAP UI test suite appsrv/rest/test/fwd-soapui-project.xml expects the input number 2.123456789123456789 to be returned as output. The same test exists for PASOE.

Constantin, Marian, can you please elaborate on how these two can co-exist?

Gayla, have to admit I'm as confused as you are... I'm looking at the log files and it does look like the assert is correct and somehow Progress managed to send back 15 decimals while it normally can't handle more than 10. Why and how it is doing that I do not know :(

Marian Edu wrote:

Gayla

It's Galya.

Galya B wrote:

Marian Edu wrote:

Gayla

It's Galya.

Galya :)

I'm looking back at the responses (in webtests_from_logs folder) and it does look like the assert it was indeed failing and Progress was only sending back the value truncated to 10 decimals - "Body match failed:<br>Path '$['response']['output_param']' expected value '2.123456789123456' and got value '2.1234567891'".

Marian Edu wrote:

I'm looking back at the responses (in webtests_from_logs folder) and it does look like the assert it was indeed failing and Progress was only sending back the value truncated to 10 decimals - "Body match failed:<br>Path '$['response']['output_param']' expected value '2.123456789123456' and got value '2.1234567891'".

Is the folder webtests_from_logs for logs from Progress runs? Then do we change the assert in the soapui tests?

May I get a confirmation that webtests_from_logs/results are results from Progress?

Galya B wrote:

May I get a confirmation that webtests_from_logs/results are results from Progress?

Yes, this is the output of a SoapUILog2Harness 'conversion program' which uses as input the SoapUI logs from running the tests in OpenEdge, to generate harness artifacts.

I find it very weird that so many tests were failing in OE, but not fixed, instead directly ported to testcases and still failing, but supposed to be used as a standard for the FWD code.

I find it very weird that so many tests were failing in OE, but not fixed,

Please be more specific about this. Are the tests broken in OE or are they designed to fail and FWD doesn't fail the same way?

Also: It can be that these tests are specific to a version of OE and you are testing in a different version. If that is the case, then we might want to make them work differently based on the version.

Greg Shah wrote:

I find it very weird that so many tests were failing in OE, but not fixed,

Please be more specific about this. Are the tests broken in OE or are they designed to fail and FWD doesn't fail the same way?

What is a test design designing failing tests called? I'm not familiar with this concept.

Also: It can be that these tests are specific to a version of OE and you are testing in a different version. If that is the case, then we might want to make them work differently based on the version.

I'm not testing OE. I'm looking at the tests results pushed to the testcases repository. Please ask the author for details. I'm taking the tests results as work that has already been assigned, completed and used as a reference for writing the testcases / SoapUI tests.

Galya B wrote:

I'm not testing OE.

Maybe we've got confused by your previous statement then... "I find it very weird that so many tests were failing in OE, but not fixed".

I'm looking at the tests results pushed to the testcases repository. Please ask the author for details. I'm taking the tests results as work that has already been assigned, completed and used as a reference for writing the testcases / SoapUI tests.

You really need to be more specific about this, there are different topics under the "Application Server" umbrela - direct remote-call from 4GL (CONNECT/RUN ON SERVER), SOAP and REST. For each of those, as far as I remember since it's been two years since last time we did anything in that area, we've tested parameter passing (data type conversion/validation), error handling and for 4GL RPC the OO serialisation.

I was looking at this and last changes there were made by Constantin probably when he did the (partial?) implementation. I'm not saying there isn't a chance we've broken some of existing tests - by changing the temp-table/dataset definition, updating records in the database or changing some support classes - but really just be more clear about what tests do you think are failing in OE, we will test those and if indeed broken will fix then as needed :)

For OE we were running those tests against the 'classical' application server and the newest 'pasoe', the 'classical' application server is not available anymore since OE12.x so we will probably be unable to test using that flavour once OE11.7 is retired (EOL is 2025).

The tests we made for 4GL RPC, and also consuming SOAP/web-services from 4GL client looks like were not 'migrated' to ABLUnit, @Greg please let me know if we should move those to ABLUnit as well. Not sure if FWD supports this (RPC, consume web-services from 4GL client) or just exposes SOAP/REST services - I think we were using those just to have some tests that can be ran from 4GL, otherwise SoapUI was used.

The SoapUI tests in testcases/appsrv/soap/test/web/fwd-soapui-project.xml are based on the OE results in testcases/webtests_from_logs/results/20211112_201954_soap_classic. There were (at the time someone ran the tests in OE) many tests failing in Progress, which can be seen in testcases/webtests_from_logs/results/20211112_201954_soap_classic/index.html.

Tests are supposed to pass. This is a base property of a software test. So it seems some tests have never passed successfully in Progress. Then these same tests are left in testcases/appsrv/soap/test/web/fwd-soapui-project.xml as a reference / standard for FWD behavior.

What do these SoapUI tests achieve? Are they to be used as they are as a standard for proper FWD behavior, or are they supposed to be fixed, or not used at all?

I find it very weird that so many tests were failing in OE, but not fixed,

Please be more specific about this. Are the tests broken in OE or are they designed to fail and FWD doesn't fail the same way?

What is a test design designing failing tests called? I'm not familiar with this concept.

In the development of FWD, we must get the error handling and boundary conditions to match OE. This means we have to write tests where the 4GL code is expected to fail and we are looking for a specific error or behavior. If seen, then the test passes. This might be done by looking at the ERROR-STATUS system handle or by storing local state in the test to detect the flow of control.

Greg please let me know if we should move those to ABLUnit as well.

Yes

Not sure if FWD supports this (RPC, consume web-services from 4GL client)

Yes, it is supported.

or just exposes SOAP/REST services - I think we were using those just to have some tests that can be ran from 4GL, otherwise SoapUI was used.

No need (or want) for SoapUI.

Galya B wrote:

The SoapUI tests in testcases/appsrv/soap/test/web/fwd-soapui-project.xml are based on the OE results in testcases/webtests_from_logs/results/20211112_201954_soap_classic. There were (at the time someone ran the tests in OE) many tests failing in Progress, which can be seen in testcases/webtests_from_logs/results/20211112_201954_soap_classic/index.html.

I don't know how those were generated, from Constantin's commit message (rev #1168) I would say it might be the logs from running the tests against FWD implementation using harness but he might be able to tell you more about it.

What do these SoapUI tests achieve? Are they to be used as they are as a standard for proper FWD behavior, or are they supposed to be fixed, or not used at all?

SoapUI projects were created for REST - appsrv/rest/test/fwd-soapui-project.xml and for SOAP - appsrv/soap/test/web/fwd-soapui-project.xml. For REST only logs when executed against Progress PASOE AppSrv exists since this is only available for PASOE, for SOAP there are log files for PASOE (pasoe-fwd-soapui-*.log) and for 'classical' (fwd-soapui-*.log).

Initially only the log files were supposed to be used for FWD implementation but then we've also added a number of 'assertions' in SoapUI project - on status code and response, still I think the log from SoapUI when ran in FWD should match the one we've got when running against 4GL (to some extent at least).

Greg Shah wrote:

Greg please let me know if we should move those to ABLUnit as well.

Yes

Not sure if FWD supports this (RPC, consume web-services from 4GL client)

Yes, it is supported.

OK then it make sense to have those moved to ABLUnit - I was simply overlook them when we've migrated the rest of testcases :(

Marian Edu wrote:

Galya B wrote:

The SoapUI tests in testcases/appsrv/soap/test/web/fwd-soapui-project.xml are based on the OE results in testcases/webtests_from_logs/results/20211112_201954_soap_classic. There were (at the time someone ran the tests in OE) many tests failing in Progress, which can be seen in testcases/webtests_from_logs/results/20211112_201954_soap_classic/index.html.

I don't know how those were generated, from Constantin's commit message (rev #1168) I would say it might be the logs from running the tests against FWD implementation using harness but he might be able to tell you more about it.

Constantin Asofiei wrote:

Galya B wrote:

May I get a confirmation that webtests_from_logs/results are results from Progress?

Yes, this is the output of a SoapUILog2Harness 'conversion program' which uses as input the SoapUI logs from running the tests in OpenEdge, to generate harness artifacts.

Let me get this coordinated. According to Constantin the dir webtests_from_logs presents the same results as the fwd-soapui-*.log files.

What do these SoapUI tests achieve? Are they to be used as they are as a standard for proper FWD behavior, or are they supposed to be fixed, or not used at all?

SoapUI projects were created for REST - appsrv/rest/test/fwd-soapui-project.xml and for SOAP - appsrv/soap/test/web/fwd-soapui-project.xml. For REST only logs when executed against Progress PASOE AppSrv exists since this is only available for PASOE, for SOAP there are log files for PASOE (pasoe-fwd-soapui-*.log) and for 'classical' (fwd-soapui-*.log).

Initially only the log files were supposed to be used for FWD implementation but then we've also added a number of 'assertions' in SoapUI project - on status code and response, still I think the log from SoapUI when ran in FWD should match the one we've got when running against 4GL (to some extent at least).

Well said. The asserts in the SoapUI tests indeed match those from the Progress runs, i.e. are failing.

In TDD failing tests means the implementation is flawed. In this case it means the tests are not fixed.

Does someone have anything to object my conclusion?

Galya B wrote:

Well said. The asserts in the SoapUI tests indeed match those from the Progress runs, i.e. are failing.

Again, I know nothing about anything that is found under the webtests_from_logs folder. I've already mentioned where the SoapUI projects and log files when ran against Progress are located, as far as I can see there isn't any assertion failed present in those log files so I imagine at that time the tests were passing.

In TDD failing tests means the implementation is flawed. In this case it means the tests are not fixed.

I've already said there might be changes in other code in testcases that might broke some of those tests but first step in identifying a test is failing is to actually ran the test isn't it? We're probably a bit late for TDD here, but anyway, failing tests in TDD is not that unusual... writing tests that fails is usually the first step ;)

Does someone have anything to object my conclusion?

Nothing at all, by that age I come to realise that resistance is futile :)

Marian, the work on the tests seems disjointed. I'm trying to help the team coordinate their versions to achieve a state where something can be completed. I'm glad you're able to contribute, but please make sure to comprehend what others say and collaborate, not simply repeat your single point of view.

Marian, let me help you understand what I'm working on here:

Greg Shah wrote:

These are items that can be found using the appserver test suite. The suite should run correctly/completely when this task is done.

Marian Edu wrote:

Galya B wrote:

In TDD failing tests means the implementation is flawed. In this case it means the tests are not fixed.

writing tests that fails is usually the first step ;)

Please read carefully what I've said. There are failing tests in TDD and that means the implementation is flawed. In our case it means the tests are flawed.

Marian, I believe acorn.ro is familiar not only with the *.log files, but also with the SoapUI tests:

gbb@gbb:~/Code/testcases/appsrv/soap/test/web$ bzr log fwd-soapui-project.xml
------------------------------------------------------------
revno: 1589
committer: Galya Bogdanova <gbb@goldencode.com>
branch nick: testcases
timestamp: Mon 2024-06-03 16:51:06 +0300
message:
  Fixes SOAP tests for classic appserver based on OE test results
------------------------------------------------------------
revno: 1273
committer: Constantin Asofiei <ca@goldencode.com>
branch nick: testcases
timestamp: Thu 2022-07-07 21:05:21 +0300
message:
  Reverted files from rev 1264 which should not have been committed.
------------------------------------------------------------
revno: 1264
committer: Constantin Asofiei <ca@goldencode.com>
branch nick: testcases
timestamp: Tue 2022-05-17 16:13:50 +0300
message:
  Added OO extent property examples for fuzzy matching at overload.
------------------------------------------------------------
revno: 1172
committer: Constantin Asofiei <ca@goldencode.com>
branch nick: testcases
timestamp: Wed 2021-11-24 17:53:34 +0200
message:
  Added misc focus cases.
------------------------------------------------------------
revno: 1164
committer: Mihai <mihai.tiganea@acorn.ro>
branch nick: testcases
timestamp: Wed 2021-10-27 10:16:27 +0300
message:
  soapUi test using pasoe server. load logs.
------------------------------------------------------------
revno: 1160
committer: Mihai <mihai.tiganea@acorn.ro>
branch nick: testcases
timestamp: Wed 2021-10-20 15:11:55 +0300
message:
  soap call - check tests and generate report for dataset usage
------------------------------------------------------------
revno: 1158
committer: Mihai <mihai.tiganea@acorn.ro>
branch nick: testcases
timestamp: Wed 2021-10-20 11:41:38 +0300
message:
  add missing tests in SoapUi for dataset 
------------------------------------------------------------
revno: 639
committer: Marian Edu <marian.edu@acorn.ro>
branch nick: testcases
timestamp: Thu 2020-06-11 18:39:15 +0300
message:
  update soapui project, http log files
------------------------------------------------------------
revno: 636
committer: Mihai <mihai.tiganea@acorn.ro>
branch nick: testcases
timestamp: Wed 2020-06-10 15:10:08 +0300
message:
  improve dataset tests for parsing arguments stage
------------------------------------------------------------
revno: 635
committer: Mihai <mihai.tiganea@acorn.ro>
branch nick: testcases
timestamp: Wed 2020-06-10 14:47:01 +0300
message:
  add soapUI tests for parsing arguments
------------------------------------------------------------
revno: 618
committer: Marian Edu <marian.edu@acorn.ro>
branch nick: testcases
timestamp: Tue 2020-05-26 12:41:15 +0300
message:
  soapui project
------------------------------------------------------------
revno: 617
committer: Marian Edu <marian.edu@acorn.ro>
branch nick: testcases
timestamp: Tue 2020-05-26 11:56:56 +0300
message:
  soapui project
------------------------------------------------------------
revno: 616
committer: Marian Edu <marian.edu@acorn.ro>
branch nick: testcases
timestamp: Tue 2020-05-26 11:50:26 +0300
message:
  more tests for extent
------------------------------------------------------------
revno: 614
committer: Marian Edu <marian.edu@acorn.ro>
branch nick: testcases
timestamp: Fri 2020-05-22 18:39:46 +0300
message:
  add more tests for exceptions
------------------------------------------------------------
revno: 613
committer: Marian Edu <marian.edu@acorn.ro>
branch nick: testcases
timestamp: Fri 2020-05-22 15:49:26 +0300
message:
  update all testsuites to use WsaURL in soapui project
------------------------------------------------------------
revno: 612
committer: Marian Edu <marian.edu@acorn.ro>
branch nick: testcases
timestamp: Fri 2020-05-22 15:19:00 +0300
message:
  update soapui project
------------------------------------------------------------
revno: 609
committer: Marian Edu <marian.edu@acorn.ro>
branch nick: testcases
timestamp: Fri 2020-05-22 14:36:17 +0300
message:
  appsrv soap tests
------------------------------------------------------------
revno: 379
author: Marian
committer: Marian Edu <marian.edu@acorn.ro>
branch nick: testcases
timestamp: Thu 2019-09-26 14:35:39 +0300
message:
  fix test cases for soap table parameters, add test for persistent procedure/function and log files for test suite run
------------------------------------------------------------
revno: 378
author: Marian
committer: Marian Edu <marian.edu@acorn.ro>
branch nick: testcases
timestamp: Thu 2019-09-26 11:21:31 +0300
message:
  add sub object in web service definition, test cases for this and log file of test suite run
------------------------------------------------------------
revno: 26
committer: Mihai <mihai.tiganea@acorn.ro>
branch nick: testcases
timestamp: Wed 2019-05-08 13:30:24 +0300
message:
  update tests - add rest psdoe settings
------------------------------------------------------------
revno: 12
committer: Marian Edu <marian.edu@acorn.ro>
branch nick: testcases
timestamp: Tue 2019-04-23 06:01:32 +0000
message:
  163cd8030b03eb428bd3b14bee983ca7f8f86918
------------------------------------------------------------
revno: 11
committer: Marian Edu <marian.edu@acorn.ro>
branch nick: testcases
timestamp: Thu 2019-04-18 17:32:58 +0000
message:
  6784cccc3cdbbe3b846a305bf5ada6d2757c3374

Galya: The failures you mention in #6417-162 are from the harness, probably something written by Constantin. These writing of these tests is just not complete yet. They are broken and will have to be fixed by someone.

Greg Shah wrote:

Galya: The failures you mention in #6417-162 are from the harness, probably something written by Constantin. These writing of these tests is just not complete yet. They are broken and will have to be fixed by someone.

Constantin said in #6417-157 webtests_from_logs are results from tests ran on Progress.

The tests including most of the asserts in webtests_from_logs match the tests and asserts in SoapUI testcases/appsrv/soap/test/web/fwd-soapui-project.xml.

You said I need to run the tests and make them successful. I found it weird, because initially I believed the tests are correct and present issues in FWD, but they are actually failing in Progress too. So I need to fix them.

So this was my attempt to answer your questions:

Greg Shah wrote:

I find it very weird that so many tests were failing in OE, but not fixed,

Please be more specific about this. Are the tests broken in OE or are they designed to fail and FWD doesn't fail the same way?

Also: It can be that these tests are specific to a version of OE and you are testing in a different version. If that is the case, then we might want to make them work differently based on the version.

Hope we're on the same page now and I continue working on fixing tests?

Galya: The failures you mention in #6417-162 are from the harness, probably something written by Constantin. These writing of these tests is just not complete yet. They are broken and will have to be fixed by someone.

Constantin said in #6417-157 webtests_from_logs are results from tests ran on Progress.

Yes

The tests including most of the asserts in webtests_from_logs match the tests and asserts in SoapUI testcases/appsrv/soap/test/web/fwd-soapui-project.xml.

At least the testcases/webtests_from_logs/results/20211112_201954_soap_classic/index.html is from the Harness not SoapUI. That doesn't mean the output is not from Progress. My point is simply that the tests have been written in multiple ways and just may not be complete yet.

You said I need to run the tests and make them successful. I found it weird, because initially I believed the tests are correct and present issues in FWD, but they are actually failing in Progress too. So I need to fix them.

It can be the case and is the case here.

Hope we're on the same page now and I continue working on fixing tests?

Yes

There are some 4GL tests which have not yet been migrated to ABLUnit. Marian's team will handle those:

The tests we made for 4GL RPC, and also consuming SOAP/web-services from 4GL client looks like were not 'migrated' to ABLUnit

Marian: Can you please list them here so that Galya knows not to touch those without discussing it with you?

Galya B wrote:

Marian, the work on the tests seems disjointed. I'm trying to help the team coordinate their versions to achieve a state where something can be completed. I'm glad you're able to contribute, but please make sure to comprehend what others say and collaborate, not simply repeat your single point of view.

Galya, sorry for stepping in, only tried to help so again sorry for the disruption.

I did mentioned the SoapUI projects were created by us ('blame' is not my favorite feature of a SCM). I won't repeat myself explaining how those and the log files were created. If those project files were updated by looking at some other log files used as reference then I don't have anything against it. I still don't understand how do you plan to fix those tests without testing against a real Progress application server but please there is no need to explain that to me :)

If you have any specific questions we will be glad to help (at least try to).

Greg Shah wrote:

Marian: Can you please list them here so that Galya knows not to touch those without discussing it with you?

Sure, the tests for RPC is in appsrv\test and for consuming WS from 4GL are in appsrv\soap\test\4glws folder.

Marian Edu wrote:

I still don't understand how do you plan to fix those tests without testing against a real Progress application server but please there is no need to explain that to me :)

I'm glad things you don't understand make you happy. I guess that's why you don't read other people's comments, to stay oblivious.

Don't read this to continue living in the bubble.Don't read this to continue living in the bubble.

Marian may just be referencing your comment from #6417-160 where you said "I'm not testing OE.".

I'm puzzled by webtests_from_logs/results/20211112_201954_soap_classic/test_set_2_exceptions/test_5_throw_error_1.html:

Request:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:urn="urn:tempuri-org:fwd">
   <soapenv:Header/>
   <soapenv:Body>
      <urn:throw_error>
         <urn:errorCode>1</urn:errorCode>
      </urn:throw_error>
   </soapenv:Body>
</soapenv:Envelope>

Response:

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <SOAP-ENV:Body>
    <SOAP-ENV:Fault>
      <faultcode>SOAP-ENV:Server</faultcode>
      <faultstring>An error was detected in the Web Service request. (10894)</faultstring>
      <detail>
        <ns1:FaultDetail xmlns:ns1="urn:soap-fault:details">
          <errorMessage>Error in SOAP parameter: errorCode</errorMessage>
          <requestID>f0e2aef24e958ce7:2a9a160a:2b0390a767fd:e00d#2708</requestID>
        </ns1:FaultDetail>
      </detail>
    </SOAP-ENV:Fault>
  </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

throw_error.p:

@openapi.openedge.export FILE(type="REST", executionMode="external", useReturnValue="false", writeDataSetBeforeImage="false").
    block-level on error undo, throw.

define input parameter errorCode as integer no-undo.

if errorCode > 0 then
    undo, throw new Progress.Lang.AppError("errorCode", errorCode).
else
    undo, throw new Progress.Lang.AppError("returnValue").

Sending 1 as errorCode input param results in Error in SOAP parameter. I'm not clear how passing in 1 as integer can lead to error in the input parameter. The same request with 0 passed successfully in test_4_throw_error.html.

Galya, the errorCode text from the Error in SOAP parameter: errorCode message is the first argument from new AppError("errorCode", errorCode). - it has no meaning in FWD runtime, is just the error text. If you change that, the error text will change, too.

For this test, FWD does not return (at least from the logs from that time) the proper error code and response:

Status code failed: expected 500 and received 200
Body match failed:
Path '/Envelope/Body/Fault/faultstring[0]' expected value 'An error was detected while executing the Web Service request. (10893)' and got value 'An error was detected in the Web Service request. (10894)'
Path '/Envelope/Body/Fault/detail/FaultDetail/errorMessage[0]' expected value 'An internal Web Service error occurred in the Web Services Adapter. (10902)' and got value 'Error in SOAP parameter: errorCode'

Constantin Asofiei wrote:

Galya, the errorCode text from the Error in SOAP parameter: errorCode message is the first argument from new AppError("errorCode", errorCode). - it has no meaning in FWD runtime, is just the error text. If you change that, the error text will change, too.

For this test, FWD does not return (at least from the logs from that time) the proper error code and response:
[...]

So webtests_from_logs/results/20211112_201954_soap_classic/test_set_2_exceptions/test_5_throw_error_1.html is a result from FWD?

Galya B wrote:

Constantin Asofiei wrote:

Galya, the errorCode text from the Error in SOAP parameter: errorCode message is the first argument from new AppError("errorCode", errorCode). - it has no meaning in FWD runtime, is just the error text. If you change that, the error text will change, too.

For this test, FWD does not return (at least from the logs from that time) the proper error code and response:
[...]

So webtests_from_logs/results/20211112_201954_soap_classic/test_set_2_exceptions/test_5_throw_error_1.html is a result from FWD?

Correct, these results were ran via harness on the FWD server. The SoapUI raw logs were considered as 'authoritative' to determine the request and expected response.

Constantin Asofiei wrote:

Galya B wrote:

So webtests_from_logs/results/20211112_201954_soap_classic/test_set_2_exceptions/test_5_throw_error_1.html is a result from FWD?

Correct, these results were ran via harness on the FWD server. The SoapUI raw logs were considered as 'authoritative' to determine the request and expected response.

So it seems I wasn't clear with my question when I got the answer in #6417-157.

We should separate our artifacts from OE and FWD. Either by more specific name or by separate directories.

I can't proceed with fixing all tests or related issues in FWD appservers without clear documentation what has been done as testing efforts and what needs to be done.

Can you start with this:

1. Setup equivalent testing environments in both OE and FWD.
2. Run the tests in both environments.
3. Assess and report the state.
   • What tests fail in OE and need to be modified/cleaned up?
   • What tests fail in FWD and need to be fixed in FWD?

Based on that we can make a plan. If some of the OE changes need to come from Marian's team, then we will get them to help.

Greg Shah wrote:

Setup equivalent testing environments in both OE and FWD.

Hasn't this already been done / documented with writing / running the tests previously? Should I start from zero?

Run the tests in both environments.

Which tests? I'm trying to understand which tests for many, many months. The SoapUI tests or the 4GL procs that don't have proper test wrappers? Where are the test suites to be executed?

Is this not what was documented in #6417-8 and following discussion?

Greg Shah wrote:

Is this not what was documented in #6417-8 and following discussion?

The "documentation" is explaining how to run testcases, which I already had for more than an year at that point.

The comment also says run all SoapUI tests. Am I supposed to set SOAP and REST api in Progress corresponding to what has been used when creating the tests and verify their validity?

As for the 4GL list that Constantin has never run, I explained in #6417-14 that it's not runnable. Some procedures don't have proper test wrappers. Basically this is not a test suite. Do I need to create one? If so, I'll just add the procs I've already tested in it.

The comment also says run all SoapUI tests. Am I supposed to set SOAP and REST api in Progress corresponding to what has been used when creating the tests and verify their validity?

Yes, I think that will be needed.

As for the 4GL list that Constantin has never run, I explained in #6417-14 that it's not runnable.

Aren't these the same tests I asked Marian's team to migrate to ABLUnit in #6417-174? If so, you don't have to run those now. After they are migrated we can test them, but for now you should focus on the tests written by Constantin.

Greg Shah wrote:

The comment also says run all SoapUI tests. Am I supposed to set SOAP and REST api in Progress corresponding to what has been used when creating the tests and verify their validity?

Yes, I think that will be needed.

So noone has any idea what has happened back in the days when the tests were created, so we should scratch those logged hours and start from scratch? This is fine, but it will obviously take the same time as the initial effort. Also it will be only for classic appserver, since at the moment there is no PASOE available to me.

As for the 4GL list that Constantin has never run, I explained in #6417-14 that it's not runnable.

Aren't these the same tests I asked Marian's team to migrate to ABLUnit in #6417-174?

I don't know. As I've said, I need documentation on testing and what and how needs to be run for this task. If it was a startup, I would set up their first tests, otherwise it's expected to receive some information on the current situation.

I need confirmation if I got it right: I need to figure out by myself how to setup Progress classic appserver with REST and SOAP apis that execute the testcases procedures in our SoapUI test suites. There is currently no progress files for endpoints configuration and no documentation I can use? And when the results are out from Progress and FWD and all issues fixed in FWD for classic appserver, the task can be closed?

Galya B wrote:

I need confirmation if I got it right: I need to figure out by myself how to setup Progress classic appserver with REST and SOAP apis that execute the testcases procedures in our SoapUI test suites. There is currently no progress files for endpoints configuration and no documentation I can use?

A few points on this one, if it doesn't help I apologise in advance :)
- there is no REST interface available in classical appserver
- SOAP interface for classical appserver is only available through WSA, documentation about how that can be deployed is on Progress web-site, maybe use this as a starting point - https://docs.progress.com/bundle/openedge-classic-appserver-administration-117/page/Configuring-a-Web-Services-Adapter-Installation.html
- the WSDL definition is created through 'ProxyGen' (also documented on Progress web-site), the ProxyGen definition for web-services can be found in appsrv/soap/proxygen/fwd.xpxg. Yes the paths in that file are hard-coded, nothing we can do about it, so to regenerate the WSDL one must replace C:\Projects\testcases_test in that file with the location of testcases project in own environment :(

Galya, I don't think you need to setup OpenEdge to run the REST/SOAP tests.

webtests_from_logs is meant to automate running the tests Marian's team wrote via harness, using as input the raw logs after running the tests via SoapUI in OpenEdge.

OTOH, I can't tell if what is currently in webtests_from_logs is in sync with the appsrv/ programs/tests and the raw SoapUI logs (appsrv/rest/test/fwd-soapui-project.xml appsrv/soap/test/web/pasoe-fwd-soapui-project.xml appsrv/soap/test/web/fwd-soapui-project.xml).

So maybe is worth it to re-generate the requests, responses and tests - see SoapUILog2Harness.java.

• branch needs rebase
• files with missing history entry:
  

  src/com/goldencode/p2j/main/AppServerConnectionPool.java
  src/com/goldencode/p2j/main/LegacyServiceHandler.java
  src/com/goldencode/p2j/main/RemoteAppServerConnectionPool.java
  src/com/goldencode/p2j/main/WebServiceHandler.java
  src/com/goldencode/p2j/main/StandardServer.java
  src/com/goldencode/p2j/oo/net/serverconnection/ClientSocket.java
  src/com/goldencode/p2j/rest/RestHandler.java
  src/com/goldencode/p2j/ui/LogicalTerminal.java
  src/com/goldencode/p2j/util/LegacyJavaAppserverClient.java
  src/com/goldencode/p2j/util/RemoteLegacyJavaAppserverClient.java
  

• src/com/goldencode/p2j/oo/lang/OeversionInfo.java
  • in the javadoc for oeclientType, please comment fully what is supposed to be from #6417-129
• src/com/goldencode/p2j/soap/SoapHandler.java
  • multi-exceptions in a catch block must each be on a separate line
• src/com/goldencode/p2j/ui/LogicalTerminal.java
  • getClientType was moved to SessionUtils. This breaks conversion of the SESSION:CLIENT-TYPE attribute - this needs to be fixed.
• src/com/goldencode/p2j/util/Agent.java
  • Agent's startup-procedure signature always defines a DEFINE INPUT PARAMETER startup-data AS CHARACTER NO-UNDO.
parameter - see https://docs.progress.com/bundle/openedge-classic-appserver-development-117/page/Startup-procedure.html. The changes to not pass startupParameter if is not set are wrong, this argument needs to be unknown.
• src/com/goldencode/p2j/util/RemoteErrorData.java
  • rename handleSoupFault to handleSoapFault. Check references in ErrorManager and WebServiceImpl, too.
• src/com/goldencode/p2j/util/AppServerManager.java
  • Throwable agentError = agentResult.getError(); - instead of doing instanceOf twice, first check against null and do this in a if/else/if pattern. This needs to have as few instructions as possible.
• src/com/goldencode/p2j/util/InvokeConfig.java
  • remove unneeded import
  • the CLIENT-TYPE management is done only if we are in a 'server' call. origClienTypes 'save and restore' must be included in this, too.
• src/com/goldencode/p2j/util/SessionUtils.java
  • for getClientType, the value of this attribute can be computed only once, and cached.
• src/com/goldencode/p2j/util/ErrorManager.java
  • remove unneeded import
  • private noRecordOrShowErrorWorker is never used?
  • getErrorHelper - please limit the line to 110 chars
  • this is an existing issue; in ErrorManager.writeLog(String), please change if (SessionManager.get().getSession() == null) to check also if SessionManager.get() is actually not null.
  • recordOrThrowError can be called from FWD client, too - why if (serverSide) { throw new ErrorConditionException } change?

The runtime needs to be regression tested with ChUI and other unit testing for customer apps.

Marian Edu wrote:

Galya B wrote:

I need confirmation if I got it right: I need to figure out by myself how to setup Progress classic appserver with REST and SOAP apis that execute the testcases procedures in our SoapUI test suites. There is currently no progress files for endpoints configuration and no documentation I can use?

A few points on this one, if it doesn't help I apologise in advance :)
- there is no REST interface available in classical appserver
- SOAP interface for classical appserver is only available through WSA, documentation about how that can be deployed is on Progress web-site, maybe use this as a starting point - https://docs.progress.com/bundle/openedge-classic-appserver-administration-117/page/Configuring-a-Web-Services-Adapter-Installation.html
- the WSDL definition is created through 'ProxyGen' (also documented on Progress web-site), the ProxyGen definition for web-services can be found in appsrv/soap/proxygen/fwd.xpxg. Yes the paths in that file are hard-coded, nothing we can do about it, so to regenerate the WSDL one must replace C:\Projects\testcases_test in that file with the location of testcases project in own environment :(

Thank you for the information.

Constantin Asofiei wrote:

Galya, I don't think you need to setup OpenEdge to run the REST/SOAP tests.

webtests_from_logs is meant to automate running the tests Marian's team wrote via harness, using as input the raw logs after running the tests via SoapUI in OpenEdge.

OTOH, I can't tell if what is currently in webtests_from_logs is in sync with the appsrv/ programs/tests and the raw SoapUI logs (appsrv/rest/test/fwd-soapui-project.xml appsrv/soap/test/web/pasoe-fwd-soapui-project.xml appsrv/soap/test/web/fwd-soapui-project.xml).

So maybe is worth it to re-generate the requests, responses and tests - see SoapUILog2Harness.java.

What is meant in SoapUILog2Harness by:

      generateRestTests(REST_APPSRV_LOG_FILES, "classic"); TODO: bad log

SoapUILog2Harness doesn't compile.

Comment this line, at the time this was written I think there was a problem with the log:

// generateRestTests(REST_APPSRV_LOG_FILES, "classic"); // TODO: bad log

javac is this:

javac -classpath $HOME/projects/p2j/build/lib/jackson-databind-2.16.1.jar:$HOME/projects/p2j/build/lib/axiom-api-1.4.0.jar:$HOME/projects/p2j/build/lib/jackson-core-2.17.0.jar SoapUILog2Harness.java

SoapUILog2Harness needs further adjustments, the logs exist as this now:

./appsrv/rest/test/fwd-soapui-appsrv.log.committed
./appsrv/rest/test/fwd-soapui-webhandler.log.committed
./appsrv/rest/test/fwd-soapui-pasoe.log.committed
./appsrv/soap/test/web/pasoe-fwd-soapui-primitives.log.committed
./appsrv/soap/test/web/pasoe-fwd-soapui-persistent.log.committed
./appsrv/soap/test/web/pasoe-fwd-soapui-info.log.committed
./appsrv/soap/test/web/pasoe-fwd-soapui-dataset.log.committed
./appsrv/soap/test/web/fwd-soapui-table.log.committed
./appsrv/soap/test/web/fwd-soapui-info.log.committed
./appsrv/soap/test/web/pasoe-fwd-soapui-parsing-arguments-dataset.log.committed
./appsrv/soap/test/web/pasoe-fwd-soapui-parsing-arguments-base-type.log.committed
./appsrv/soap/test/web/pasoe-fwd-soapui-wsdl.log.committed
./appsrv/soap/test/web/fwd-soapui-dataset.log.committed
./appsrv/soap/test/web/pasoe-fwd-soapui-table.log.committed
./appsrv/soap/test/web/fwd-soapui-parsing-arguments-base-type.log.committed
./appsrv/soap/test/web/fwd-soapui-parsing-arguments-dataset.log.committed
./appsrv/soap/test/web/fwd-soapui-exceptions.log.committed
./appsrv/soap/test/web/fwd-soapui-persistent.log.committed
./appsrv/soap/test/web/fwd-soapui-primitives.log.committed
./appsrv/soap/test/web/fwd-soapui-wsdl.log.committed
./appsrv/soap/test/web/pasoe-fwd-soapui-exceptions.log.committed

Constantin Asofiei wrote:

• src/com/goldencode/p2j/util/Agent.java
  • Agent's startup-procedure signature always defines a DEFINE INPUT PARAMETER startup-data AS CHARACTER NO-UNDO.
parameter - see https://docs.progress.com/bundle/openedge-classic-appserver-development-117/page/Startup-procedure.html. The changes to not pass startupParameter if is not set are wrong, this argument needs to be unknown.

This is an important fix in the execution of the logic that has to do with how it's implemented in FWD, not much related to OE interfaces. The moment there is mode "I", the param is to be validated and an empty param (when a startup is not defined in directory for the pool) there was an error. I haven't documented it back in the days, when I found it, but this was the only solution I've found, because the methods are reused by many different invocations.

I don't understand - startup procedure defines an input parameter, always. FWD must pass this argument, otherwise the call will fail.

Constantin Asofiei wrote:

I don't understand - startup procedure defines an input parameter, always. FWD must pass this argument, otherwise the call will fail.

The invoke methods are reused by normal SOAP calls that validate input param values. This you understand, right? The call failed with no startup procedure specified for the appserver pool.

I'm talking about this code in Agent.listen:

            SessionUtils.setClientType(AppServerManager.isPasoe() ?
                                          ClientType.MULTI_SESSION_AGENT :
                                          ClientType.APPSERVER);

            String startupParameter = pool.getStartupParameter();
            boolean isSuccess = StringHelper.hasContent(startupParameter) ?
               invoke(null, null, InvocationModes.IMMEDIATELY, startup, hStartup, "Startup",
                      "I", new character(startupParameter)) :
               invoke(null, null, InvocationModes.IMMEDIATELY, startup, hStartup, "Startup", null);

            if (!isSuccess)
            {
               LOG.warning("There were problems invoking the startup procedure, terminate the agent: " + pool.getName());
               // if there were problems invoking the startup procedure, terminate the agent
               return;
            }

This is executed when the agent is starting up - startup procedure is executed only once, at agent startup. This is not related to SOAP. This change needs to be reverted.

Constantin Asofiei wrote:

I'm talking about this code in Agent.listen:
[...]

This is executed when the agent is starting up - startup procedure is executed only once, at agent startup. This is not related to SOAP. This change needs to be reverted.

OK, let me try again....

• no startup config;
• startup config with value;
• startup config without value, i.e.:
  

   <node-attribute name="startup" value=""/>
  

The last example doesn't work. If you want me to revert the change, it will continue to fail agents startups.

Galya B wrote:

The last example doesn't work. If you want me to revert the change, it will continue to fail agents startups.

How does it fail? if (startup != null && startup.length() > 0) check will not move ahead with executing the Startup procedure if there is none configured (and I just tested this). Also, the changes are related to startup_parameter, and not startup node.

Constantin Asofiei wrote:

Galya B wrote:

The last example doesn't work. If you want me to revert the change, it will continue to fail agents startups.

How does it fail? if (startup != null && startup.length() > 0) check will not move ahead with executing the Startup procedure if there is none configured (and I just tested this). Also, the changes are related to startup_parameter, and not startup node.

Thanks for the clarification. I've completely forgotten what has been fixed with this fix months ago. You're right. It fixes a startup procedure being configured without startup params.

Galya B wrote:

It fixes a startup procedure being configured without startup params.

OK, just for clarification: if in OpenEdge you set as Startup procedure a program which doesn't have a single DEFINE INPUT PARAMETER startup-data AS CHARACTER NO-UNDO. parameter definition, the agent will not start. So even in if we end up in FWD with such a configuration, is not for FWD runtime to be fixed, but for the program specified as startup procedure.

Constantin Asofiei wrote:

• src/com/goldencode/p2j/util/SessionUtils.java
  • for getClientType, the value of this attribute can be computed only once, and cached.

This can be executed on a thread on the server that works for the appserver connection. Depending on the client, the value on the server gets changed. The client type should creep from the the client -> server connection to the server -> appserver connection. The second is one connection that serves many clients.

Constantin Asofiei wrote:

• recordOrThrowError can be called from FWD client, too - why if (serverSide) { throw new ErrorConditionException } change?

In silent the client should not throw execution logic related exceptions. Otherwise the client exits without the server knowing about it. The error should be sent to the server, that sends back the signal for session ended. I've fixed the issue I was able to reproduce with the server-side handleSoapFault. Other cases need to be fixed by calling a new server exported method, which I forgot and left in TODO.

Galya B wrote:

Other cases need to be fixed by calling a new server exported method, which I forgot and left in TODO.

Actually it's not that simple. For some reason it didn't work on the thread and I left the note that somehow it should be executed from the conversion thread.

New r15301 of 6417a based on trunk r15276 ready for review.

Constantin Asofiei wrote:

The runtime needs to be regression tested with ChUI and other unit testing for customer apps.

Why and what ChUI and what testing on customer apps? It's mostly SOAP related changes.

• src/com/goldencode/p2j/oo/lang/OeversionInfo.java
  • in the javadoc for oeclientType, please add in the comment what is supposed to be from #6417-129
• src/com/goldencode/p2j/util/InvokeConfig.java
  • I missed something in previous review - instead of doing serverHandle.unwrapConnectable() instanceof ServerImpl, we need serverHandle.getResource() instanceof ServerImpl. unwrapConnectable is meant to be followed always by an API call on that interface - otherwise, leaks will occur if the handle is not valid.
  • the location for this change can't be in InvokeConfig (again, I missed it in previous review); it needs to be in ControlFlowOps. Also, the server handle can also be assigned to SESSION - in this case, if (serverHandle == null) throw e for a QUIT no longer applies.
• src/com/goldencode/p2j/util/ErrorManager.java
  • do you have a test for the throwErrorConditionException being called from the FWD client side? Because the FWD client can throw ERROR conditions (although I don't have a test on hand) - and these will be raised back on the FWD server Conversation thread, via our remote protocol. Also by throw on the Conversion thread you meant throw on the Conversation thread?

Galya B wrote:

Constantin Asofiei wrote:

The runtime needs to be regression tested with ChUI and other unit testing for customer apps.

Why and what ChUI and what testing on customer apps? It's mostly SOAP related changes.

ErrorManager is refactored pretty extensively. I'll get back to you via email what I meant by ChUI and unit tests.

Constantin Asofiei wrote:

Review for 6417a/15301 built on trunk rev 15276:

• src/com/goldencode/p2j/oo/lang/OeversionInfo.java
  • in the javadoc for oeclientType, please add in the comment what is supposed to be from #6417-129

Client type is not done. It will have to be reviewed and tested further. It's way more complicated than what I thought in #6417-129, which is simply based on Progress documentation that is misleading.

• the location for this change can't be in InvokeConfig (again, I missed it in previous review); it needs to be in ControlFlowOps. Also, the server handle can also be assigned to SESSION - in this case, if (serverHandle == null) throw e for a QUIT no longer applies.

I don't understand. I can't simply move the logic of the RUN method further down in ControlFlowOps where the invoke methods are reused. The logic is strictly for a RUN call.

ErrorManager is refactored pretty extensively.

Not really. It's mostly a rename, because I couldn't track down what is on the server and what is on the client side (not for the first time), both being smushed together like always. Other than that there are a few new methods serving the fault handling. What is refactored?

I'll get back to you via email what I meant by ChUI and unit tests.

I'll be waiting.

Galya B wrote:

Client type is not done. It will have to be reviewed and tested further. It's way more complicated than what I thought in #6417-129, which is simply based on Progress documentation that is misleading.

Understood.

• the location for this change can't be in InvokeConfig (again, I missed it in previous review); it needs to be in ControlFlowOps. Also, the server handle can also be assigned to SESSION - in this case, if (serverHandle == null) throw e for a QUIT no longer applies.

I don't understand. I can't simply move the logic of the RUN method further down in ControlFlowOps where the invoke methods are reused. The logic is strictly for a RUN call.

See ControlFlowOps.invokeImpl line 7112 and ControlFlowOps.invokePersistentImpl line 7841 - these are the only places where a call over appserver is made. ControlFlowOps can be used on its own, InvokeConfig is just a wrapper/helper over ControlFlowOps APIs. And you can invoke functions over appserver, too, if you have the handle for a remote persistent procedure.

ErrorManager is refactored pretty extensively.

Not really. It's mostly a rename, because I couldn't track down what is on the server and what is on the client side (not for the first time), both being smushed together like always. Other than that there are a few new methods serving the fault handling. What is refactored?

throwErrorConditionException usage RemoteErrorData implementation, these are changes we need to validate they don't regress.

Constantin Asofiei wrote:

by throw on the Conversion thread you meant throw on the Conversation thread?

Yes.

the FWD client can throw ERROR conditions (although I don't have a test on hand) - and these will be raised back on the FWD server Conversation thread, via our remote protocol.

throwErrorConditionException usage RemoteErrorData implementation, these are changes we need to validate they don't regress.

The SOAP fault was thrown by the client and instead of being handled on the server, it crashed the client process, leaving the session open on the server. Now the SOAP fault is handled on its own, and I'm not aware of other similar types of errors I can raise on the client to test further if the 'fix' is fixing it. Hopefully the tests are going to help.

Constantin Asofiei wrote:

Also, the server handle can also be assigned to SESSION - in this case, if (serverHandle == null) throw e for a QUIT no longer applies.

What do you mean here? I'm trying to verify if the call is on the server or the client. Are you saying there is a scenario, where it's executed on the server, but the handle is null?

Galya B wrote:

Constantin Asofiei wrote:

Also, the server handle can also be assigned to SESSION - in this case, if (serverHandle == null) throw e for a QUIT no longer applies.

What do you mean here? I'm trying to verify if the call is on the server or the client. Are you saying there is a scenario, where it's executed on the server, but the handle is null?

I mean the serverHandle can reference the SESSION:HANDLE, and not a SERVER handle.

Can you check r15302.

Galya B wrote:

Can you check r15302.

The changes look good. I was trying to get a standalone test with FWD client throwing an ERROR condition, but no luck. Please proceed with testing.

Constantin Asofiei wrote:

Galya, I don't think you need to setup OpenEdge to run the REST/SOAP tests.

webtests_from_logs is meant to automate running the tests Marian's team wrote via harness, using as input the raw logs after running the tests via SoapUI in OpenEdge.

OTOH, I can't tell if what is currently in webtests_from_logs is in sync with the appsrv/ programs/tests and the raw SoapUI logs (appsrv/rest/test/fwd-soapui-project.xml appsrv/soap/test/web/pasoe-fwd-soapui-project.xml appsrv/soap/test/web/fwd-soapui-project.xml).

So maybe is worth it to re-generate the requests, responses and tests - see SoapUILog2Harness.java.

I've run SoapUILog2Harness.java. The output is the same.

Since the tests are up to date, how do I solve #6417-149?

Galya B wrote:

Since the tests are up to date, how do I solve #6417-149?

This is tricky; on a first glance, decimals incoming from outside OpenEdge (like SOAP or REST) are not limited to 10 decimal digits, and this limit is somehow bypassed. We need more experiments for this, not just 'pipe', i.e. what happens if you add 0.1 to the value, does it return more than 10 digits to the caller? Or if you assign the param to a value with 20 decimal digits, does this keep only 10 digits? I mean, there may be a place where this limit is reset back to default, and my thinking is it happens when some assignment is done on that value.

Constantin Asofiei wrote:

Galya B wrote:

Since the tests are up to date, how do I solve #6417-149?

This is tricky; on a first glance, decimals incoming from outside OpenEdge (like SOAP or REST) are not limited to 10 decimal digits, and this limit is somehow bypassed. We need more experiments for this, not just 'pipe', i.e. what happens if you add 0.1 to the value, does it return more than 10 digits to the caller? Or if you assign the param to a value with 20 decimal digits, does this keep only 10 digits? I mean, there may be a place where this limit is reset back to default, and my thinking is it happens when some assignment is done on that value.

Actually there is a similar issue we've encountered with integer startup parameters (ref #8206). There is a validation in OE that suggests the numbers are integers indeed (like The {name} parameter has too many digits and {name} must be set to a value between 500000 and 2147483647), but in some cases bigger numbers are allowed. Vladimir even thought it's the default behavior. Anyways, the point is that these two are probably related and I think it's an obvious bug in OE in how big numbers are handled. Reproducing it may not be a good idea, because it seems a bit sporadic from what I've seen with the startup params.

Maybe it's worth it to be tested further as you say.

There is something wrong with SOAP on trunk. Jetty returns 404, although the SoapHandler handles the requests successfully. I can't think of anything in testcases I might have changed to lead to such result. This is reproducible on 6417a, but also on trunk.

Constantin, can you please check if any SoapUI test for SOAP on classic is working with trunk in your setup? I've spent two hours on this and can't understand what's going on.

Galya, this must be from the Jetty upgrade - we need a baseRequest.setHandled(true); being called always in SoapHandler.handle. Now is being executed only in catch blocks - I think we need to just set the request as 'handled' like this:

      super.handle(target, baseRequest, request, response);

      if (baseRequest.isHandled())
      {
         return;
      }

      baseRequest.setHandled(true);

Radu: please check the same for the TenantHandler work.

Greg/Tomasz: is it worth it to integrate the #7994 work from 6667i into 6417a, so we can merge it to trunk?

Greg/Tomasz: is it worth it to integrate the #7994 work from 6667i into 6417a, so we can merge it to trunk?

Yes, anything safe that can be moved out of 6667i is a good thing.

Did 6417a pass its testing? If so, I'd like to get it merged.

Greg Shah wrote:

Did 6417a pass its testing? If so, I'd like to get it merged.

It passed, but my concern is that the testing for both the customer projects didn't even register the fail in all soap endpoints. I think we need to spend some time to write a Docker image with a few smoke tests to confirm the server, processes and clients are all running after each commit.

Constantin, do I need to merge 7994a into 6417a?

Galya, this must be from the Jetty upgrade - we need a baseRequest.setHandled(true); being called always in SoapHandler.handle.

I've noticed it too. I'll see if it works.

Galya B wrote:

Galya, this must be from the Jetty upgrade - we need a baseRequest.setHandled(true); being called always in SoapHandler.handle.

I've noticed it too. I'll see if it works.

It works. Fix in r15303.

Let me know about 7994a and merge.

Galya B wrote:

Let me know about 7994a and merge.

Lets get 6417a to trunk first and after that we'll check #7994 changes.

Constantin Asofiei wrote:

Galya B wrote:

Let me know about 7994a and merge.

Lets get 6417a to trunk first and after that we'll check #7994 changes.

Right. Actually #7994 is its own issue. Let Sergey merge it if it's needed on trunk.

6417a was merged to trunk as rev. 15287 and archived.

Created task branch 6417b from trunk rev 15298 to fix an existing bug exposed by the changes in 6417a.

The fix is in 6417b rev 15299.

6417b was merged to trunk rev 15299 and archived.

Just a quick note to let you know the tests for calling appsrv from 4GL client (RPC style) were converted to AblUnit and pushed into testcases project rev#1597 - folder `tests/appsrv/4gl_rpc`. Configuration of AppSrv connection string is done in 'appsrv.cfg' text file in project's root, currently is for a PASOE server running on localhost port 8810.

The tests for consuming web-services from 4GL client are being rewritten, will come back with a new message when available on testcases.

Constantin, give me more information on #7143-1012.

Also the revert in #6417-235 breaks a core functionality of SOAP, so we should address it somehow.

Created task branch 6417c from trunk rev 15303.

6417c rev 15304 backed out part of trunk rev 15287 in AppServerManager - the client needs to handle the response's return-value, error condition and stop; AppServerManager is executed on the remote side, not on the client-side.

Galya: we can use 6417d for the conversion fix of port-handle and fixing the 6417c part. For now, please review 6417c, if any other changes in 15287 would be dependent on what was reverted in AppServerManager.

Constantin Asofiei wrote:

6417c rev 15304 backed out part of trunk rev 15287 in AppServerManager - the client needs to handle the response's return-value, error condition and stop; AppServerManager is executed on the remote side, not on the client-side.

This is an intentional change related to:

• a QUIT on the remote side quits the client

It fixed appsrv/test/test_raise_quit.p.

Constantin Asofiei wrote:

6417c rev 15304 backed out part of trunk rev 15287 in AppServerManager - the client needs to handle the response's return-value, error condition and stop; AppServerManager is executed on the remote side, not on the client-side.

Actually related to:

• misc errors related to the ERROR, STOP, RETURN ERROR, raised by the remote side

Tested by appsrv/test/test_throw_error.p, appsrv/test/test_return_error.p, appsrv/test/test_raise_stop.p.

Can you explain why you back it out? What are the steps for reproducing an issue?

Galya B wrote:

Can you explain why you back it out? What are the steps for reproducing an issue?

The root cause is the change in AppServerManager - this is executed on the remote side, and not the caller side. These fixes need to be at the caller (i.e. ControlFlowOps).

The side effect was a null AsyncRequestImpl.result, for a RUN ... ASYNC remote call (which raised a QUIT condition), when the event procedure was executed.

We need to merge 6417c, if there are no concerns for other dependencies.

Constantin Asofiei wrote:

Galya B wrote:

Can you explain why you back it out? What are the steps for reproducing an issue?

The root cause is the change in AppServerManager - this is executed on the remote side, and not the caller side. These fixes need to be at the caller (i.e. ControlFlowOps).

I believe they need to be on the server-side, that's why they are there.

The side effect was a null AsyncRequestImpl.result, for a RUN ... ASYNC remote call (which raised a QUIT condition), when the event procedure was executed.

We need to merge 6417c, if there are no concerns for other dependencies.

Give me an example step by step how to reproduce the issue. If you don't, there is no way to fix the base issue described in this task.

Galya B wrote:

Constantin Asofiei wrote:

Galya B wrote:

Can you explain why you back it out? What are the steps for reproducing an issue?

The root cause is the change in AppServerManager - this is executed on the remote side, and not the caller side. These fixes need to be at the caller (i.e. ControlFlowOps).

I believe they need to be on the server-side, that's why they are there.

They can't be on AppServerManager. For example, ControlFlowOps.setReturnValue affects the current context - the call can be on a remote server (not in the same JVM), and this will not work.

Give me an example step by step how to reproduce the issue. If you don't, there is no way to fix the base issue described in this task.

I will later today, but at this time we need 6417c merged to trunk regardless if it affects the testcases you mentioned.

Constantin Asofiei wrote:

Galya B wrote:

Constantin Asofiei wrote:

Galya B wrote:

Can you explain why you back it out? What are the steps for reproducing an issue?

The root cause is the change in AppServerManager - this is executed on the remote side, and not the caller side. These fixes need to be at the caller (i.e. ControlFlowOps).

I believe they need to be on the server-side, that's why they are there.

They can't be on AppServerManager. For example, ControlFlowOps.setReturnValue affects the current context - the call can be on a remote server (not in the same JVM), and this will not work.

Give me an example step by step how to reproduce the issue. If you don't, there is no way to fix the base issue described in this task.

I will later today, but at this time we need 6417c merged to trunk regardless if it affects the testcases you mentioned.

I did a major rework to get to this solution. If you just revert it like that every time there is a bug not covered by tests, we won't move forward with FWD. But do what you know best.

Greg Shah wrote:

This regression is holding back a customer's testing. While it exists in trunk, they are blocked. It is urgent to get this fix into trunk.

Galya: Please split this discussion into 2 parts. The first, urgent part is to ensure that 6417c is OK to merge now. The second part, about the reasons that the original change was not OK, will happen after that.

I don't know if it's ok to revert it just like that, because it took me days to get to the solution and a lot of testing. Just revert it like that since the customer can't use an older version.

Also don't expect me to remember just like that what a change done months ago involves. That's what automation is about.

Greg Shah wrote:

Galya: Please split this discussion into 2 parts. The first, urgent part is to ensure that 6417c is OK to merge now. The second part, about the reasons that the original change was not OK, will happen after that.

'after that' I will need to figure out by myself what is the issue, instead of receiving reproducible steps immediately (basically the reason for the rollback).

I'm generally surprised by the urgency of an issue not on production. Even issues on production usually take a day or so for a hotfix.

Are you saying it is better to back out 6417a completely?

I mean when SOAP wasn't working for days after java 17 related changes merge it wasn't urgent.

Greg Shah wrote:

Are you saying it is better to back out 6417a completely?

I'm obviously saying that a hotfix takes some time and should be communicated between the person who is able to reproduce the bug and the dev who introduced it. And the hotfix rarely is a simple rollback.

If you want to rollback 6417a it's your idea.

Across all of our customers, SOAP is much less used than appserver. For this customer, it is close to a production issue since it is blocking a significant portion of their application and they have their own customer deadlines for deployment.

Greg Shah wrote:

Across all of our customers, SOAP is much less used than appserver. For this customer, it is close to a production issue since it is blocking a significant portion of their application and they have their own customer deadlines for deployment.

The importance not obvious from the tests available.

Are you saying it is better to back out 6417a completely?

I'm obviously saying that a hotfix takes some time and should be communicated between the person who is able to reproduce the bug and the dev who introduced it. And the hotfix rarely is a simple rollback.

If you want to rollback 6417a it's your idea.

This is the 3rd regression from 6417a and if the solution Constantin proposes is not obviously safe, then the logical conclusion (to me) is that all of 6417a should be reversed.

Greg Shah wrote:

This is the 3rd regression from 6417a and if the solution Constantin proposes is not obviously safe, then the logical conclusion (to me) is that all of 6417a should be reversed.

How 3rd?

I wondered how I get so many black points in the register. Now I see. All issues in FWD are on my account.

1. runtime issue in FileSystemOps.initFileInfo fixed in 15299
2. conversion issue in control_flow.rules fixed in 15300
3. this appserver issue

No one is saying everything is your fault. And who really cares about blame when we simply need to help a customer with an urgent issue?

Can we get over the arguing and find the right solution here?

Greg Shah wrote:

runtime issue in converison issue with FileSystemOps.initFileInfo fixed in 15299

This is not introduced by 6417. I'm not sure what you mean.

conversion issue in control_flow.rules fixed in 15300

The patch proposed by Constantin is indeed on this branch, but SOAP doesn't work without a change.

this appserver issue

This is an actual issue, unfortunately not even one test covers this critical functionality.

Greg Shah wrote:

No one is saying everything is your fault. And who really cares about blame when we simply need to help a customer with an urgent issue?

Can we get over the arguing and find the right solution here?

The right solution everywhere is on time code review and good automation. Otherwise even if you revert the whole branch, what do you expect me to change?

Constantin found the issue early today. If he spoke with me, we might have had already found the proper solution. If I got steps to reproduce and we communicated why this and not that has been implemented.

No one is saying everything is your fault. And who really cares about blame when we simply need to help a customer with an urgent issue?

Can we get over the arguing and find the right solution here?

The right solution everywhere is on time code review and good automation. Otherwise even if you revert the whole branch, what do you expect me to change?

I agree. That may mean that you have to write more tests and work through the painful process of making them succeed in both OE and FWD.

But for now, I need your assessment of 6417c. If you have no way to assess it, then we will remove 6417a completely and that will give us time to get it right.

Greg Shah wrote:

No one is saying everything is your fault. And who really cares about blame when we simply need to help a customer with an urgent issue?

Can we get over the arguing and find the right solution here?

The right solution everywhere is on time code review and good automation. Otherwise even if you revert the whole branch, what do you expect me to change?

I agree. That may mean that you have to write more tests and work through the painful process of making them succeed in both OE and FWD.

But for now, I need your assessment of 6417c. If you have no way to assess it, then we will remove 6417a completely and that will give us time to get it right.

'Time' doesn't close old tasks by itself. My assessment is that this functionality should not be reverted. Revert the whole branch, since you feel like it.

'Time' doesn't close old tasks by itself. My assessment is that this functionality should not be reverted.

That is not an option in the short term.

Revert the whole branch, since you feel like it.

Do you have an alternative?

Greg Shah wrote:

'Time' doesn't close old tasks by itself. My assessment is that this functionality should not be reverted.

That is not an option in the short term.

Revert the whole branch, since you feel like it.

Do you have an alternative?

Give me 24h and make Constantin call me.

• the test does not match the comment, I assume it was meant to be 'not equal':
  

        catch (QuitConditionException e)
        {
           if (serverHandle != null)
           {
              // don't propagate to the outer procedure QUIT executed on an appserver
              throw e;
           }
        }
  
  

test_return_error.p and test_throw_error.p still fail, I think are related to structured exceptions (as block-level on error undo, throw. is used).

There is also this fix for RUN ... SET portTypeHandle:

=== modified file 'rules/convert/control_flow.rules'
--- old/rules/convert/control_flow.rules        2024-06-19 13:17:03 +0000
+++ new/rules/convert/control_flow.rules        2024-06-20 12:39:54 +0000
@@ -1246,7 +1246,8 @@
                   downPath(this, prog.kw_sing_run, prog.kw_set) or 
                   downPath(this, prog.kw_singlton, prog.kw_set) or
                   downPath(this, prog.kw_set)
-                  
+               
+               <action>methodTxt = "setProcedureHandle"</action>
                <rule>downPath(this, prog.kw_persist, prog.kw_set)
                   <action>ref = copy.getImmediateChild(prog.kw_persist, null)</action>
                   <rule on="false">downPath(this, prog.kw_sing_run, prog.kw_set)
@@ -1254,12 +1255,15 @@

                      <rule on="false">downPath(this, prog.kw_singlton, prog.kw_set)
                         <action>ref = copy.getImmediateChild(prog.kw_singlton, null)</action>
+                        
+                        <action on="false">methodTxt = "setPortHandle"</action>
                         <action on="false">ref = copy</action>
                      </rule>
                   </rule>
                </rule>
+               
                <action>ref = ref.getImmediateChild(prog.kw_set, null)</action>
-               <action>runRef = createJavaAst(java.method_call, "setProcedureHandle", runRef)</action>
+               <action>runRef = createJavaAst(java.method_call, methodTxt, runRef)</action>
                <action>ref.putAnnotation("peerid", runRef.id)</action>
             </rule>

Greg: I think we can merge 6417c rev 15305. There is also an issue with (serviceName == null || data.services.contains(serviceName)) test not being used in the AppServerManager.lookupRemoteAppServer, but we can leave this for 6417d.

Go ahead and merge 6417c.

Constantin Asofiei wrote:

• the test does not match the comment, I assume it was meant to be 'not equal':
  [...]

This is exactly the type of 'smart' moves I wanted to prevent. It's not a mistake. GL.

There is also an issue with the socket connection and 'nohostverify' - I've found when trying to run some reports in a customer's app.

This is fixed in 6417c rev 15306 - the 'nohostverify' for SOAP requests (saved as a context-local flag) must be set/unset for that SOAP request, and also the context-local 'nohostverify' does not apply for normal SSL socket connections (the parameter must be used).

I'm merging 6417c now.

Branch 6417c was merged to trunk rev 15305 and archived.

• appsrv/test/test_raise_quit2.p
  

  block-level on error undo, throw.
  {common/log_hlp.i}
  
  run log-it('ABL raise quit : - start').
  
  {appsrv/test/connect.i}
  
  def var evtprc as log.
  
  procedure proc1.
     def input param p1 as char.
     evtprc = true.
     assert-null(p1, "parameter must be unknown", {&line-number}).
     assert-true(self:quit, "self:quit must be true", {&line-number}).
  end.
  
  def var p1 as char.
  evtprc = false.
  run raise_quit2.p on server appsrv asynchronous  event-procedure "proc1" (output p1).
  assert-not-err(
      substitute({&msg_call_no_error}, 'raise_quit', ''), 
      {&line-number}).
  pause 2.
  process events.
  assert-true(evtprc, "event-procedure must execute", {&line-number}).
  assert-true(not appsrv:connected(), "appsrv must be disconnected", {&line-number}).
  
  run log-it('ABL raise quit : - end').
  
  catch e as Progress.Lang.AppError :
      message e:ReturnValue.        
  end catch.
  
  finally:
  {appsrv/test/disconnect.i}    
  end finally.
  

• api/raise_quit2.p
  

  @openapi.openedge.export FILE(type="REST", executionMode="external", useReturnValue="false", writeDataSetBeforeImage="false").
  block-level on error undo, throw.
  
  def output param p1 as char.
  
  p1 = "here".
  
  quit.
  

There is another existing issue with marking the connection as disconnected for async cases - I have a fix for this.

Galya, please let me know when 6417d is available.

I'll get back to you with test_return_error.p and test_throw_error.p.

Constantin Asofiei wrote:

Galya, please let me know when 6417d is available.

6417d is created from trunk r15306 and the port handle patch is committed.

I'm looking into AppServerManager.java. I think the condition you added to lookupRemoteAppServer if (port != 0) should not be there. The first statement in the method handles the conditions for setting a default port. If both conditions are not met, then the following check on the port should fail.

Constantin Asofiei wrote:

There is also an issue with (serviceName == null || data.services.contains(serviceName)) test not being used in the AppServerManager.lookupRemoteAppServer

This has been reworked to server.aliases.contains(appService) to satisfy the test testcases/appsrv/test/connect_errors/service_name.p.

The fix for #7143-1017 is on 6417d.

Constantin: Please review.

Galya, please add back the if (port != 0) test - this is required for connection strings like -AppService app_server -S fwd1 -DirectConnect -sessionModel Session-free.

Constantin Asofiei wrote:

Galya, please add back the if (port != 0) test - this is required for connection strings like -AppService app_server -S fwd1 -DirectConnect -sessionModel Session-free.

Let me test it. I thought there should be either a default, or an explicit, no middle grounds.

Galya B wrote:

Constantin Asofiei wrote:

Galya, please add back the if (port != 0) test - this is required for connection strings like -AppService app_server -S fwd1 -DirectConnect -sessionModel Session-free.

Let me test it. I thought there should be either a default, or an explicit, no middle grounds.

https://docs.progress.com/bundle/abl-reference/page/CONNECT-method-AppServer.html

Connection parameter	Description
-S [ service-name / port-number ]	The UDP port number for a NameServer, or, if you connect directly, the TCP/IP port number for an AppServer connection. You can specify either an explicit port number or a service name. If you use a service name, the method uses the port number associated with that name in the TCP/IP services file. (Defaults to 5162)

Our -S is both serviceName and port. So the port can be declared 0 in the connect string, we don't know if this 0 is a missing port. So maybe the solution is to check if the -S is not a service name. What do you say about replacing if (port != 0) with if (serviceName == null)?

Note #6417-283 edited.

Galya B wrote:

Our -S is both serviceName and port. So the port can be declared 0 in the connect string, we don't know if this 0 is a missing port. So maybe the solution is to check if the -S is not a service name. What do you say about replacing if (port != 0) with if (serviceName == null)?

This will not work on how serviceName and port are computed:

      String serviceName = parms.get("-S");
      boolean directConnect = parms.containsKey("-DirectConnect");

      int port = 0;
      try
      {
         port = Integer.parseInt(serviceName);
      }
      catch (NumberFormatException e)
      {
         // ignore
      }

So you can have both port and serviceName. More, serviceName for URL connections is considered the path (I don't know what happens if you specify both -S and -URL).

At this time, please add back the if (port != 0) test. We will need to check what happens if -S 0 is specified.

Galya B wrote:

Constantin Asofiei wrote:

There is also an issue with (serviceName == null || data.services.contains(serviceName)) test not being used in the AppServerManager.lookupRemoteAppServer

This has been reworked to server.aliases.contains(appService) to satisfy the test testcases/appsrv/test/connect_errors/service_name.p.

That uses a -H localhost -S fwd1 -DirectConnect -sessionModel session-free and the test doesn't behave OK in FWD.

At this time, please do a filter on services, before the port check:

      if (serviceName != null)
      {
         foundServers = foundServers.stream()
                                    .filter(server -> server.services.contains(serviceName))
                                    .collect(Collectors.toList());
         if (foundServers.isEmpty())
         {
            ErrorManager.recordOrShowError(5484, false, "Unknown service " + appService + "  transport TCP.");
            ErrorManager.recordOrShowError(5468, false, "Application server connect failure.");
            return null;
         }
      }

Constantin Asofiei wrote:

Constantin Asofiei wrote:

There is also an issue with (serviceName == null || data.services.contains(serviceName)) test not being used in the AppServerManager.lookupRemoteAppServer

This has been reworked to server.aliases.contains(appService) to satisfy the test testcases/appsrv/test/connect_errors/service_name.p.

That uses a -H localhost -S fwd1 -DirectConnect -sessionModel session-free and the test doesn't behave OK in FWD.

appsrv/test/connect_errors/service_name.p is successful. Is it not?

Galya B wrote:

Constantin Asofiei wrote:

Constantin Asofiei wrote:

There is also an issue with (serviceName == null || data.services.contains(serviceName)) test not being used in the AppServerManager.lookupRemoteAppServer

This has been reworked to server.aliases.contains(appService) to satisfy the test testcases/appsrv/test/connect_errors/service_name.p.

That uses a -H localhost -S fwd1 -DirectConnect -sessionModel session-free and the test doesn't behave OK in FWD.

appsrv/test/connect_errors/service_name.p is successful. Is it not?

It's a 'false positive', this code is not executed in FWD:

if error-status:error then 
do:
// not executed    
    assert-true( error-status:get-number(1) = 5484, 
        'Conect method with wrong service_name should throw first error as 5484', 
        {&line-number}).

Constantin Asofiei wrote:

appsrv/test/connect_errors/service_name.p is successful. Is it not?

It's a 'false positive', this code is not executed in FWD:
[...]

Let's fix it. Just remove the if?

See r15311. I'm committing the test without the if, it works well now.

Constantin Asofiei wrote:

So you can have both port and serviceName. More, serviceName for URL connections is considered the path (I don't know what happens if you specify both -S and -URL).

Wait, what? It means if -S is a port, serviceName is the string representation of the port?

Galya B wrote:

Constantin Asofiei wrote:

So you can have both port and serviceName. More, serviceName for URL connections is considered the path (I don't know what happens if you specify both -S and -URL).

Wait, what? It means if -S is a port, serviceName is the string representation of the port?

Looks like so; but that's a question for 6417e.

I think we should revert the serviceName check. Less errors thrown, better. I think it's not correct now. What if it's a port and not a service name. Let's leave it for further investigation and make it proper from the parsing part.

Or at least I need to add one if (port == 0) before the filter.

No, no, I'm reverting, sorry, this is safer. Reverted in r15312.

Where to find the 'TCP/IP services file' in Progress, where the IP <-> service name is configured for appserver connect?

Galya B wrote:

Where to find the 'TCP/IP services file' in Progress, where the IP <-> service name is configured for appserver connect?

It's referring to C:\Windows\System32\drivers\etc\services or /etc/services. The file is used to resolve the port by the service name mapped in the OS file and specified by -S. And this is the serviceName we have in AppServerManager.lookupRemoteAppServer. We don't seem to have that lookup at the moment.

Constantin, I don't understand what is the function of the appserver config services in the context of the finding in note #6417-298:

        <node class="container" name="app_services">
          <node class="container" name="app_server">
            <node class="strings" name="services">
              <node-attribute name="values" value="fwd1"/>
              <node-attribute name="values" value="21200"/>
            </node>
          </node>
        </node>

Another question: How to configure a default appserver for Progress nameserver? There is no such config in the web manager, but there is a related error message, when the default is not found. I couldn't find info on the web.

Where to find the 'TCP/IP services file' in Progress, where the IP <-> service name is configured for appserver connect?

It's referring to C:\Windows\System32\drivers\etc\services or /etc/services. The file is used to resolve the port by the service name mapped in the OS file and specified by -S. And this is the serviceName we have in AppServerManager.lookupRemoteAppServer. We don't seem to have that lookup at the moment.

Correct, this is a system file. We never implemented OS level "services". Back when we first encountered this, we didn't have appserver support. It was only related to Progress database connections and since we weren't supporting Progress databases, we created a different concept. The CONNECT statement (which at that time was only supported in FWD for database connection) looked up the PostgreSQL connnection details in the directory instead of integrating with the OS services file.

Galya B wrote:

Constantin, I don't understand what is the function of the appserver config services in the context of the finding in note #6417-298:

[...]

When we added appserver, we just extended our usage of the directory to get the services info. This may still make sense, since putting the appservers into the services file is again a Progress-specific idea. In other words, there is very little value to having the OS manage this information. No general purpose usage of the services exist so why integrate to that level? Thus the directory seems fine.

Greg Shah wrote:

Galya B wrote:

Constantin, I don't understand what is the function of the appserver config services in the context of the finding in note #6417-298:

[...]

When we added appserver, we just extended our usage of the directory to get the services info. This may still make sense, since putting the appservers into the services file is again a Progress-specific idea. In other words, there is very little value to having the OS manage this information. No general purpose usage of the services exist so why integrate to that level? Thus the directory seems fine.

If the node services is supposed to be the mapping between the service name and port, then I'm puzzled by the directory.xml.template:

            <node class="integers" name="ports">
              <node-attribute name="values" value="21200"/>
              <node-attribute name="values" value="21300"/>
              <node-attribute name="values" value="21500"/>
            </node>
            <node class="strings" name="services">
              <node-attribute name="values" value="fwd1"/>
              <node-attribute name="values" value="21200"/>
              <node-attribute name="values" value="21300"/>
              <node-attribute name="values" value="21500"/>
            </node>

How does this mapping work (services being more than ports)? It's not consistent.

What's different in FWD is that appserver connect works without nameserver and without -DirectConnect. In Progress it never works that way. If there is no -DirectConnect, it falls back to nameserver and if the nameserver is not found, it fails. But in FWD -AppService still works. I think though that at least the port / service name and host name should not be taken into consideration in this case, because in Progress these are UDP port and host for the nameserver. So any validation in FWD of a nameserver host and port is meaningless when we're ok with accepting -AppService.

When I'm trying to say is: CONNECT with no -DirectConnect checks the -AppService (although not logical without NS) and uses the configs in directory, or fails without further validation of params. -DirectConnect on the other hand will have all params validated.

P.S. -AppService is used only with NS in OE, never taken into consideration with direct connect.

We have no need for the nameserver. We don't plan to implement a replacement for it. Any functionality choices should be assessed with those assumptions in mind.

Greg Shah wrote:

We have no need for the nameserver. We don't plan to implement a replacement for it. Any functionality choices should be assessed with those assumptions in mind.

Do we want Progress-like implementation or Progress-compatible one? Because currently it's Progress-compatible. I don't discuss the fact that there is no NS in FWD. I'm asking since there is no NS, why not support only DirectConnect and URL like what it would be if there was no NS in Progress (completely viable option).

Galya B wrote:

Greg Shah wrote:

We have no need for the nameserver. We don't plan to implement a replacement for it. Any functionality choices should be assessed with those assumptions in mind.

Do we want Progress-like implementation or Progress-compatible one? Because currently it's Progress-compatible. I don't discuss the fact that there is no NS in FWD. I'm asking since there is no NS, why not support only DirectConnect and URL like what it would be if there was no NS in Progress (completely viable option).

Just to clarify why I'm asking questions about CONNECT and our take on it. Part of the task is:

Greg Shah wrote:

some errors throw by SERVER:CONNECT are not the same

If the FWD implementation is not Progress-like, there is no way the errors to be the same.

Aren't we simulating the nameserver lookup with our services configuration in the directory?

Greg Shah wrote:

Aren't we simulating the nameserver lookup with our services configuration in the directory?

We can't simulate the host / udp port lookup and related error messages. Currently the port / host are always considered to be the appserver host/port instead of NS host/port, which is not correct when -AppService is used instead of -DirectConnect. As for the AppService name, I still don't understand how the mapping should work.
We're creating a strange non-Progress situation, where there is appservice name, but also host and port. So I'm proposing to skip the NS host / port validation as irrelevant, instead of erroneously apply the appserver host/port validation.

Constantin: Please provide some details about how this is supposed to work and why we have it.

Consider that applications can be passing this connect option from code that may be difficult to modify. If I understand correctly, we are just trying to bypass the nameserver nonsense and use the service name to find the right appserver in this case. We only need to have compatible error handling for errors we can reach.

Greg Shah wrote:

If I understand correctly, we are just trying to bypass the nameserver nonsense and use the service name to find the right appserver in this case.

This is the acceptable part. But the code currently doesn't completely reflect that by also trying to throw the additional params of host and port into the bucket (that are NS related). I want to clean it up by validating and applying host / port only when it's a direct connect (or url connect). When appservice is specified it will rely solely on the mapping in directory.

Galya B wrote:

Greg Shah wrote:

Aren't we simulating the nameserver lookup with our services configuration in the directory?

We can't simulate the host / udp port lookup and related error messages. Currently the port / host are always considered to be the appserver host/port instead of NS host/port, which is not correct when -AppService is used instead of -DirectConnect. As for the AppService name, I still don't understand how the mapping should work.

The documentation about AppSrv connect is here - https://documentation.progress.com/output/ua/OpenEdge_latest/pdsoe/PLUGINS_ROOT/com.openedge.pdt.langref.help/rfi1424920280586.html

Since FWD does not support NameServer (and that is hardly used in production in OE anyway) it looks to me 'DirectConnect' is considered as default in FWD while in OE defaults to false (NS). Using 'AppService@ with DirectConnect is not really an error, OE simply ignores that parameter for DirectConnect.

We're creating a strange non-Progress situation, where there is appservice name, but also host and port. So I'm proposing to skip the NS host / port validation as irrelevant, instead of erroneously apply the appserver host/port validation.

AppService name is not the same as 'service name' as in services that is merely a mapping between a port number and a service name - as in you can connect using `IP:22` or `HOST:ssh`.

Normally one can use either '-H' + '-S' (classical appsrv) or '-URL' (pasoe, or classical through AIA).

Marian, thanks for the references.

I've read the Progress documentation, so my question was if we do Progress-like or Progress-compatible implementation. In Progress as you've mentioned AppService is ignored with DirectConnect. In FWD it's not, so it's a mix and not match situation.

So my question in #6417-309 about the implementation in FWD still stands. I think I should implement it and wait for the review to discuss it further, as per usual.

I think I should implement it and wait for the review to discuss it further, as per usual.

Does this really have to be worked now or can we give Constantin a little time to respond to the above question?

Greg Shah wrote:

I think I should implement it and wait for the review to discuss it further, as per usual.

Does this really have to be worked now or can we give Constantin a little time to respond to the above question?

If time answers questions, then I guess we have it for branch 'e'.

Just a quick note to inform you the tests for consuming WS from 4GL (FWD) client were converted to ABLUnit - available in tests/appsrv/4gl_ws. Connection string used is setup in wssrv.cfg file in project's root.

The 4GL RPC tests in tests/appsrv/4gl_rpc can be configured by changing the connection string in appsrv.cfg file.

There were small (path) fixes in appsrv/api files - those are 'hosted' in the AppSrv, everything is committed in testcases rev#1601.

Old tests in appsrv/test (RPC) and appsrv/soap/test (WS) are to be removed so changing those is not considered for further conversion, if current tests needs to be changed/extended the changes should be done in the ABLUnit ones.

The documentation for the original implementation in FWD is at Appserver_Support and Chapter_15_Setup_and_Configuration_of_Legacy_Appservers

      if (!directConnect && appService == null)
      {
         appService = nameServerDefaults.get(host);
      }

1. the host is assumed to be the nameservice host
2. the default appService is retrieved for that nameserver

This is the only place where nameservers are used explicitly in FWD (and we need this added back).

Otherwise, as the FWD infrastructure for appservers is completely different than the one in OpenEdge (there are no nameservers, or actual URLs to validate), we may not be able to replicate any and all OpenEdge errors related to this.

But, we need to ensure that any legacy CONNECT string (which may be hard-coded in database, 4GL source code, configuration files, etc), can be mapped by FWD to an actual appserver, via the app_services configuration. More, in FWD the intent was to not create different appserver configurations for i.e. dev or prod, so for example aliases can have multiple values, for a i.e. -AppService dev and -AppService prod, which both semantically target the same appserver.

Is the confusion about the app_services/host and app_services/port that it can map to either -URL and -H/-S options? This was 'by intent' originally, and we can try to can separate the error messages by using the same rules as in OE, like:

-DirectConnect If specified, the -H and -S parameters are interpreted as the network address and TCP/IP port number of an AppServer connection. Otherwise, the -H and -S parameters are interpreted as the network address and UDP port number of a NameServer.

At this time, I think we need to build a list of various permutations of the CONNECT options, and how they fail in OpenEdge (the specific errors). We can use classic appserver for now to test in OpenEdge, I can help with PASOE.