FWD » Core Development » Base Language

Priority work: implement all usage of the server object handle in regards to connecting to a web service.

Most of the code is common to appserver or sockets. BUT there is this which does need to be added:

SET-CALLBACK-PROCEDURE()

1. REQUEST-INFO, RESPONSE-INFO attribute - the datatype is Progress.Lang.OERequestInfo class
2. RESET method, RESTART-ROWID attribute - following code shows that they do not apply to server object:
   

   def var h as handle.
   create server h.
   h:reset(). /* not queryable attribute for server widget */
   h:restart-rowid. /* not queryable attribute for server widget */
   

3. Note that the connect method uses different parameters to connect to a web services, but there is no difference conversion-wise.

Syntax for set-callback-procedure (used with a procedure handle):

set-callback-procedure(callback-name,internal-procedure [ , procedure-context ])

• APPLY-CALLBACK( ) method
• GET-CALLBACK-PROC-CONTEXT( ) method
• GET-CALLBACK-PROC-NAME( ) method

Browsing the OE Dev: Web Services book, I cant find any other methods/attributes related to web services, which we don't currently support.

Looks like I've missed this RUN statement syntax, related to web services:

RUN portTypeName [ SET hPortType ] ON [SERVER] hWebService [ NO-ERROR ].

1. fix the RUN SET ... ON SERVER statement
2. add SET-CALLBACK-PROCEDURE to PeristentProcedure.setCallbackProcedure

fix the RUN SET ... ON SERVER statement

Look like the ASYNC clause is supported (at least at compile-time) for this case too.
LE: plus the TRANSACTION DISTINCT clause, which is also allowed.

Another later edit: it is possible to use the SET hPortType specification even with the TRANSACTION DISTINCT and PERSISTENT SET clauses. I think the best solution is to add another handle parameter (right after the proc name) to all ControlFlowOps.invokeRemote APIs, to not duplicate them all... and if the standalone SET clause is not present, emit null for that parameter.

More info about RUN ... SET ... ON SERVER. Something like this works:

def var h1 as handle.
def var h2 as handle.

procedure proc0.
    message "bla".
end.

h2 = session.
run proc0 set h1 on server h2. /* runs proc0 */

message h1. /* this is unknown */

DEF VAR i AS INT.
DEF VAR j AS INT.
DEF VAR ch1 AS CHAR.
DEF VAR ch2 AS CHAR.

def var hs as handle.
def var hws as handle.
def var l as log.

create server hs.
l = hs:connect("-WSDL http://www.webservicex.net/convertMetricWeight.asmx?WSDL -Port MetricWeightUnitSoap").

if l = no 
then do:
  message "could not connect".
  return.
end.

run MetricWeightUnitSoap set hws on server hs.
if not valid-handle(hws)
then do:
  message "could not establish port".
  return.
end.

MESSAGE hws:TYPE hws:NAME. /* shows "PROCEDURE MetricWeightUnitSoap" */
i = 1000.
ch1 = "microgram".
ch2 = "milligram".

run ChangeMetricWeightUnit in hws (INPUT i, INPUT ch1, INPUT ch2, OUTPUT j).
message ch1 ch2 i j.

delete object hws.
hs:disconnect().
delete object hs.

In fixups/control_flow.rules refAsync is never assigned, but it is relied upon in the ASYNCH ON case and in the SET hPorttype case. The later case has no null protection, so it should abend. The former case is protected, but will just never trigger.

Everything else looks OK with the code.

It looks good. This will go into conversion testing next.

This is in testing now.

With this change, testing passed. Checked into bzr as 10193.

Comment from CA:

This and #1995 should be worked by the same person, as they are related (I think the web services need the SOAP features internally). And about the X-Document features: serialized XMLs (built using X-Document, then saved to a longchar var) can be passed as input to SOAP service invocation. We should be able to build the SOAP/Web Services without prior X-Document features, but we should test them with the XMLs too.

Following are findings about how 4GL implements the web service requests and what tools we need to address this. The most important tool in 4GL is the bprowsdldoc command which receives as argument the URI for a WSDL document and will generate reports on how the services exposed by the WSDL document can be used in 4GL.

About the WSDL document: 4GL will validate the connect options (service, binding and the port type) against the WSDL file. Basically, the WSDL document is an XML file (which we could parse and obtain a DOM Document instance), but this will force us to build tools for accessing the WSDL data in an easier way. The alternative is wsdl4j, which provides a WSDL reader and is realeased under Common Public License - v 1.0. It implements JSR 110 (Java APIs for WSDL) and is found here: http://wsdl4j.sourceforge.net/ ; Also, it is internally used by AXIS2.

Some details about connections: from what I can tell, establishing a web server connection just validates the passed parameters against the WSDL file (and possibly override some options, like -SOAPEndpoint). Once a web server "connection" was established, this means that a port type that can be executed with the RUN port-type SET h-port-type ON SERVER h-web-server statement. At this time, I think once the CONNECT statement establishes a connection, this only determines which port type will be exposed to the RUN statement (this is because a binding is associated with only one port type and, if the service was specified, the port must be specified (if the service has more than one port).

When the RUN statement is used to execute a port type, this only collects the information about the exposed operations (i.e. parameters and return values); the procedure attributes/methods (like get-signature, internal-entries, etc) are a no-op for the "port-type" procedure resource. More, I think no actual network connection to the remote web server is established.

Later on, when an operation is ran on the connected port type using a RUN operation IN h-port-type statement, the operation must be part of the loaded port type. Now, the interesting part. 4GL allows two modes for passing parameters: wrapped and unwrapped. In wrapped mode, the input will be the request message and the output will be the response message, from the SOAP Body node, in serialized XML form; these are passed as char or longchar variables (request as input and response as output). For the unwrapped mode, things start to get... complicated; 4GL will use parameter mode, type and order to map passed parameters to the SOAP response and request message elements; based on the order, it will try to convert from the 4GL data type to the WSDL message data type and viceversa; thus, you can use i.e. a 4GL char variable as output for a decimal value. More, you can use mixed mode: i.e. input parameters can be unwrapped, while the output be wrapped; not sure if mixed mode can be used inside the list of input or output parameters. A question here is how will 4GL handle a case when both the single input and single output are character: will see them as wrapped, unwrapped, combination?

Considering the mixed mode of passing arguments to the SOAP message (and back), I think we can use Apache AXIS2 as a tool to communicate with the web service, as it provides:
- connection pooling (for async requests, but not fully sure I will use this)
- both sync and async requests
- invoke the services java style or low-level, by passing the SOAP message directly
- the downside of AXIS is that it comes together with lots of dependencies (I will need to provide the minimal list of jars)

With AXIS2, at first I wanted to use this approach: provide at build time all the WSDL documents reached by the CONNECT statement and use AXIS to generate java code to access these services; using this generated code (and some other configurations), we could use java reflection to invoke the web service java-style, and not care about how the SOAP message is encoded. But, considering the mixed mode of sending and receiving a SOAP message, this will not save us the effort of checking each passed parameter against the message definition. Thus, as using java style calls does not worth the effort, I think the best solution is to send the SOAP message as wrapped (regardless of how the input parameters are specified) and use the wrapped received SOAP message to set the output variables accordingly.

Finally, web services allow temp-table data to be passed as parameters (both input and output); these are mapped to specially-configured complex data types in the WSDL document. We can add these later, but I will ask Guy for their WSDL documents, to make an idea of how complex are the web services they are using.

This is training server from progress:
http://wstraining.progress.com/index.html

I don't want to rely on progress or any other freely available services for the testing. Look at the WSDL structure, make a plan of cases and lets discuss it in #2208.

In regard to the CPL 1.0, it is acceptable to use code under that license.

License text:

http://opensource.org/licenses/CPL-1.0

Ultimately, we probably won't include any of these libraries in our "distributions" of P2J. Instead, we will provide a mechanism to easily pull down and patch the versions needed (perhaps through Maven). Customers can pull the specific versions manually too.

But even if we did ship the binaries under the P2J distribution, the CPL does not have the specialized linking behavior that is part of the GPL. This means that having Java code that depends upon the wsdl4j classes/jar does not merge P2J together with wsdl4j as a single work. In other words, P2J is not part of the "Program" nor is it a "Contribution". Note this text in section 1 of the license:

Contributions do not include additions to the Program which: (i) are separate modules of software distributed in conjunction with the Program under their own license agreement, and (ii) are not derivative works of the Program.

Thus, the restrictions you note are only in regard to the wsdl4j code itself, which is not a problem.

I have reviewed the above findings. The approach seems reasonable. I'll check the licenses for the AXIS2 dependencies when you post the list.

The wrapped mode is working nicely, async mode is almost done too (only remaining feature is canceling the requests), but unwrapped mode I think is going to be a PITA. We will need extensive testing to fully support it properly, as a message can have parts for which their type can be a complex type defined in some custom schema - and we will need to cover all possible schema type definition to be sure we get it right.

What we can do is this: from what I understand, the server project is using only wrapped mode, where each service operation call has at most one input and at most one output parameter, both of type char or longchar (I've double checked this in the 4GL source code and all service operation calls are seem to be invoked in wrapped mode). Thus, we can postpone the unwrapped mode to a later time.

Beside the unwrapped mode, what I need is to figure out how the soap faults are reported by axis2 and add support to create the ERROR-STATUS:ERROR-OBJECT-DETAIL. I will not go into building the SOAP-fault object, I just need to find out how the soap faults are reported by axis2 and invoke a SoapFactory.createSoapFault API which will create the object.

axiom-api-1.2.13.jar
axiom-impl-1.2.13.jar
axis2-adb-1.6.2.jar
axis2-kernel-1.6.2.jar
axis2-transport-http-1.6.2.jar
axis2-transport-local-1.6.2.jar
commons-httpclient-3.1.jar
commons-logging-1.1.1.jar
httpcore-4.0.jar
mail-1.4.jar
neethi-3.0.2.jar
wsdl4j-1.6.2.jar
XmlSchema-1.4.7.jar

• mail-1.4.jar is released under COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.0
• wsdl4j-1.6.2.jar is released under Common Public License - v 1.0
• all others are released under Apache License Version 2.0

1. Create new redmine tasks for the unwrapped mode and any error/fault processing that is not being implemented. Put enough detail in there for the future readers to understand exactly what needs to be done.

2. I believe that the CDDL 1.0 code can be used. Basically it is a weak copyleft license that does not have the linking requirements of the GPL. In other words, this code can be included on a source or binary basis in other projects (a "Larger Work") which has a different license (including a closed source license) and the source code of the closed license does not need to be released (under the CDDL or another license). Similar to the CPL, the CDDL requires providing the source code and an attribution to the project in our docs. Just like as mentioned above, we won't actually distribute this code with P2J and thus we will avoid even these other requirements.

http://opensource.org/licenses/CDDL-1.0

1.6. Larger Work means a work which combines Covered Software or portions thereof with code not governed by the terms of this License.

It looks to me like all of the additional jars should be OK.

Evgeny: please help me a little with something related to X-Noderef resources. Is there an existing way to serialize a XNoderef instance to a string? (i.e. return the entire XML sub-tree represented by a node in a string).

LE: Nevermind, there is the XmlHelper.write in P2J.

Constantin Asofiei wrote:

LE: Nevermind, there is the XmlHelper.write in P2J.

I've used for SAX writer:

   static String serializeNodeToString(Node node)
   throws TransformerFactoryConfigurationError,
          TransformerException
   {
      StreamResult xmlOutput = new StreamResult(new StringWriter());
      Transformer transformer = TransformerFactory.newInstance().newTransformer();
      transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
      transformer.transform(new DOMSource(node), xmlOutput);

Code Review 1224a

I like the improved abstraction of some parts of the runtime (e.g. ServerHelper). It also seems like you have cleaned up some latent bugs.

Unfortunately, there a huge amount code in WebServiceHelper that is specific to the 4GL SOAP implementation. As such, I can't really evaluate it effectively without going much deeper in that area of the 4GL, which I don't have time to do right now. On the surface it looks OK, but that is not a deep look.

Evgeny: please take a look at this code (realizing that it is an early/unfinished version). In particular, try to focus on the SOAP-specific parts and see if you can find anything of concern.

Other than that, my only question is in ControlFlowOps.initializeProxy(), is there any way the IllegalStateException can be thrown when processed from converted code?

Greg Shah wrote:

Other than that, my only question is in ControlFlowOps.initializeProxy(), is there any way the IllegalStateException can be thrown when processed from converted code?

Looking back at it, it is only called from invokeRemotePersistentSetAsyncWithMode which already is protected against calls using a web service; so, this exception can never be thrown.

Code Review 0109d

1. SOAPFactory.throwError() needs javadoc.

2. How safe is the use of IllegalArgumentException in SOAPHeaderImpl.getHeaderNode()/setHeaderNode()?

3. WebServiceHelper.msgParams needs javadoc.

4. Several of the files have explicit imports instead of using *.

5. WebServiceHelper.WebServiceConnectOptions has many methods without javadoc.

6. I see that you are using the Java 7 catch ( | ) syntax. That should be fine, but it is a more explicit dependency on Java 7 than we had before (I think we could still get away with compiling under Java 6 before). It isn't a problem, just a note.

7. I wonder if we have any potential character set conversion issues in our usage of the various InputStream/OutputStream (byte oriented streams versus character oriented reader/writer) constructs? Even if there are some issues there, I would say we create a separate task to consider/solve such issues. But I want us to know if there are any concerns there.

Otherwise, I don't see anything wrong. This version is much more complete, of course. But I still lack some domain understanding to provide a more complete evaluation.

Everything looks good. My only question is related to this (and the related commented-out code):

 * The {@link #CONNECT_OPTION_MAX_CONNECTIONS}, {@link #CONNECT_OPTION_CONNECTION_LIFETIME},
 * {@link #CONNECT_OPTION_NO_SESSION_REUSE} and {@link #CONNECT_OPTION_NO_HOST_VERIFY} options are
 * not implemented by {@link #connect}.

Does this mean these are not yet implemented or that don't need to be implemented?

Greg, all of these are not used by the server code. Also, the gain vs cost of implementing them I don't think is worth it (this will mean hacking AXIS2 to make connections persistent, hacking the AXIS2's SSL connection code to reuse the SSL sessions and a lot of other ugly changes).

I have no problem deferring this work. Please add comments to the javadoc to explain that the work is a TODO and the reason for the deferral.

In addition, add a task to Redmine to describe the work and make it related to this task.

OK, finish your own testing AND take into account any feedback from Evgeny from his code review of WebServiceHelper.invoke().

Then go ahead with regression testing.

Greg, I'm fine with how the runtime works with my testing. I'll put this through regression testing once I'm done with the appserver related testing, but I will not release it until I here from Evgeny.

main part of runtime testing has passed. need to double check the CTRL-C part.

Attached version is merged with rev 10438.

Code Review 0116a

I am fine with the changes. Go ahead and check it in.

Evgeny: I still want you to review this code and post your findings.

Constantin: If there are changes to make later based on the review, you can do so and check in without regression testing IF those changes are in code that cannot be reached by MAJIC.

Committed to bzr revision 10439.

Code Review 0116a:
1) com.goldencode.p2j.xml.XNodeRefImpl#getOwnerDocument()
Does RESTRICTED_DOCUMENT can affair to other methods in XNode ? if yes, we need merge this code with isInitialized() method.
2) AsyncRequestImpl
on line 613 is better to use notifyAll() instead of notify(), because it may lead to thread "starvation" problem.

Evgeny Kiselev wrote:

Code Review 0116a:
1) com.goldencode.p2j.xml.XNodeRefImpl#getOwnerDocument()
Does RESTRICTED_DOCUMENT can affair to other methods in XNode ? if yes, we need merge this code with isInitialized() method.

Access is restricted only to OWNER-DOCUMENT, for X-Nodes attached to a SOAP-Header or SOAP-Fault-Detail resource; the X-Nodes are valid, is just that in 4GL access to the SOAP Envelope via DOM is not allowed. But I think I've missed a case: when cloning the DOM Node in SOAPHeaderEntryImpl.setNode, I think RESTRICTED_DOCUMENT info should have been set at the clone's document. I will double check how it behaves and fix this.

2) AsyncRequestImpl
on line 613 is better to use notifyAll() instead of notify(), because it may lead to thread "starvation" problem.

This is safe, as there can be at most one thread waiting for this (when an explicit STOP condition is sent to terminate the request).