Project

General

Profile

Feature #9516

include pre-converted webspeed runtime code in FWD

Added by Greg Shah over 1 year ago. Updated 5 months ago.

Status:
Internal Test
Priority:
Normal
Assignee:
Constantin Asofiei
Target version:
-
Start date:
Due date:
% Done:

100%

billable:
No
vendor_id:
GCD
case_num:
version_reported:
version_resolved:
reviewer:
production:
No
env_name:
topics:
WEB/WebSpeed/WebHandler

ade.zip (17.7 MB) Roger Borrello, 01/08/2025 03:58 PM

abl_webspeed.zip (74.1 KB) Constantin Asofiei, 01/30/2025 01:17 PM

zfile_set.txt Magnifier (3.89 KB) Roger Borrello, 02/05/2025 06:47 PM

prepare_template.json (487 Bytes) Roger Borrello, 02/06/2025 09:55 AM

webspeed_sample.mkv (6.06 MB) Radu Apetrii, 06/17/2025 07:20 AM

Related issues

Related to Base Language - Feature #6506: implement CGI WebSpeed support and the standard web-disp.p dispatch functionality Test
Related to Base Language - Feature #6433: implement WEB-CONTEXT system handle WIP

History

#1 Updated by Greg Shah over 1 year ago

  • Related to Feature #6506: implement CGI WebSpeed support and the standard web-disp.p dispatch functionality added

#2 Updated by Greg Shah over 1 year ago

Starting with ADE tools 12.2, the code is released under the Apache 2 license. This means it is safe to include that code directly into FWD. Where we previously would have had customers include code from Possenet, we can now implement the versions from 12.8.4. We don't want every customer to have to find all the source in teh ADE code and include that in their conversion. Instead, I propose that we include all of the webspeed code (pre-converted) in FWD itself.

To be clear: I'm only talking about runtime code here. All the include files and any development time utilities (like the E4GL preprocessor) are not to be included here.

Scott/Marian: Can you please help us identify all of the WebSpeed runtime code in ADE 12.8.4?

#3 Updated by Greg Shah over 1 year ago

I think there are 3 parts to this work.

  1. Get the proper list of .p files (and any other procedures or code that must be there).
  2. Get all of the code in that list to properly convert and compile. This may include adding some conversion support for any missing 4GL features.
  3. Add runtime support for those missing 4GL features.

I don't actually know if there are actually any missing 4GL features, I just mention it so that we don't get surprised if there are some.

#4 Updated by Roger Borrello over 1 year ago

Is this the location of the code that will have all the parts we are concerned with? Or are those already stored somewhere?

https://community.progress.com/s/question/0D54Q00008AAMhESAX/ade-openedge-1221-development-tools-source-code?t=1601487321084

#5 Updated by Greg Shah over 1 year ago

No. As noted above, we MUST use 12.8.4 not 12.2.1.

Also: I've already downloaded and stored that code in ~/shared/projects/posse/ade_tools_apache_license_12_8_4.zip.

#6 Updated by Roger Borrello over 1 year ago

Greg Shah wrote:

No. As noted above, we MUST use 12.8.4 not 12.2.1.

Also: I've already downloaded and stored that code in ~/shared/projects/posse/ade_tools_apache_license_12_8_4.zip.

Should we have an "ade" project the same as we have "possenet" so that it can be checked out to the abl/ directory in a similar manner? It would simplify things, as we can use an option to indicate which is used in a project. The "ade" project can have a webspeed.txt file, which would specify the files to build for this task (in the -z syntax).

#7 Updated by Greg Shah over 1 year ago

Should we have an "ade" project the same as we have "possenet" so that it can be checked out to the abl/ directory in a similar manner?

Yes

It would simplify things, as we can use an option to indicate which is used in a project.

OK, but please note:

  • the new ADE 12.8.4 should be the default
  • all future customers should use ADE 12.8.4
  • the support for possenet will be removed as soon as our existing customer projects have shifted to to use ADE 12.8.4

The "ade" project can have a webspeed.txt file, which would specify the files to build for this task (in the -z syntax).

Hmmm. I agree this file should exist but I don't think it should be placed inside the ADE repo. We don't want to change that code at all, if possible.

#8 Updated by Constantin Asofiei over 1 year ago

Greg/Roger, please see #9488-11 - this includes the ADE source code and also is configured as a parse project for now.

#9 Updated by Greg Shah over 1 year ago

The webspeed.txt should be in that bzr repo.

#10 Updated by Roger Borrello over 1 year ago

The p2j_repo/ade project is a project to convert the OO parts of ADE 12.8.4 in the same manner this task is to convert the WebSpeed parts? I don't think naming the project "ade" is best, because we really should have the ADE 12.8.4 code in that project. Maybe it should have been "ade_oo" and this "ade_webspeed"?

#11 Updated by Constantin Asofiei over 1 year ago

Is it worth to use a single project for both webspeed and oo conversion, and use profiles to distinguish between them?

#12 Updated by Roger Borrello over 1 year ago

Constantin Asofiei wrote:

Is it worth to use a single project for both webspeed and oo conversion, and use profiles to distinguish between them?

I would think so. The project would have very little, and just checkout the "ade" project, with the top level directory as src (same as possenet) to the "abl/" directory. Then there could be 2 text files to be used as input files for -x in the conversion.

Are these supposed to generate jar files that would be included with a customer project?

#13 Updated by Greg Shah over 1 year ago

Is it worth to use a single project for both webspeed and oo conversion, and use profiles to distinguish between them?

Yes

Are these supposed to generate jar files that would be included with a customer project?

No, the results will be checked in to FWD itself.

#14 Updated by Roger Borrello over 1 year ago

Why wouldn't the project just create "fwd_ade" and be both the webspeed and OO components of ADE? Or am I missing something? Do we need to have it more granular? If so, how do we package FWD with WebSpeed, FWD with OO, etc?

#15 Updated by Greg Shah over 1 year ago

Why wouldn't the project just create "fwd_ade" and be both the webspeed and OO components of ADE?

It would.

Or am I missing something? Do we need to have it more granular? If so, how do we package FWD with WebSpeed, FWD with OO, etc?

The results of converting the webspeed AND the OO code will be to copy it into the FWD source tree and commit it there. Then it will always be in FWD from them on.

#16 Updated by Roger Borrello over 1 year ago

Do I have the rights to create a bzr repo on devsrv01? It looks like there was a little difference in possenet from most projects, as it has a repo/ directory rather than being called possenet_repo like the others.

Can you create /opt/secure/code/ade_repo with the contents of ade.zip attached as the initial check-in? It is zipped with src/ as the top-level directory, so ade would be the repo directory. That would match up well with possenet, except you wouldn't have to force the toplevel directory like we do with possenet.

#17 Updated by Greg Shah over 1 year ago

Yes, I think you have the rights to do that.

Nevertheless, I did it already.

cd /opt/secure/code
mkdir ade_tools_repo
cd ade_tools_repo
mkdir ade_tools_12_8_4
cd ade_tools_12_8_4
bzr init
unzip -q ~/ade.zip 
bzr add
bzr commit -m "Import of the unmodified files in the Apache 2.0 licensed ADE Tools 12.8.4 published by Progress.  The only change from the original zip is that the top level directory has been renamed from ADE-release-12.8.x/ to src/."

#18 Updated by Marian Edu over 1 year ago

Greg Shah wrote:

To be clear: I'm only talking about runtime code here. All the include files and any development time utilities (like the E4GL preprocessor) are not to be included here.

Scott/Marian: Can you please help us identify all of the WebSpeed runtime code in ADE 12.8.4?

I don't think you have to bother with those, especially if you already have the 'cgi-wrap' dispatcher... the 'classical' dispatcher is in web/objects/web-disp.p, the equivalent for PAS is web/objects/web-handler.p. I don't know if FWD will be able to use the wait-for web-notify on default-window but if you want to use the code as-is and converted this has to be implemented. The whole html-mapped mambo jambo is using actual UI components but either way the classical webspeed was not running in batch-mode just because of those reasons. If you do want to go that route then probably everything from web/objects and web/support is needed, also some files from webutil folder (webstart.p for sure, maybe reset.p, debug.p and ping.p although not really sure those are mandatory).

I've done this before for ganimede (pure 4GL), later migrated to node.js and became akera.io and never bothered to use any of Progress Webspeed runtime, as long as the wrap-cgi include files support all the APIs the implementation/runtime is not relevant for the client (imho).

#19 Updated by Greg Shah over 1 year ago

In a status meeting today, Scott said that they don't explicitly call any of the ADE webspeed procedures.

So: I think we won't convert anything that isn't needed to make web-disp.p work properly. I'm worried there may be implicit usage or nested usage of code in the ADE tools, for example if some code is run persistently to make a library of functions or internal procedures OR if super-procedures are in use there may be some implicit access to things that are outside of web-disp.p.

Scott: In PASOE, do you extend the web-handler.p?

#20 Updated by Constantin Asofiei over 1 year ago

Greg, we if we don't convert all dependencies, then we need to modify the ADE code; for example, if we omit a .p file which is ran just that it holds some helper procedures, then the entire web-disp.p will fail to run.

#21 Updated by Constantin Asofiei over 1 year ago

And here I mean programs ran from web-disp.p for 'normal usage', not ADE programs which can be ran by passing some query arguments at the call, or explicitly ran via the webspeed URL.

#22 Updated by Greg Shah over 1 year ago

Constantin Asofiei wrote:

And here I mean programs ran from web-disp.p for 'normal usage', not ADE programs which can be ran by passing some query arguments at the call, or explicitly ran via the webspeed URL.

That is exactly what I mean. We will convert any explicit or implicit usage of those files from web-disp.p but "we won't convert anything that isn't needed to make web-disp.p work properly". Sorry for the double negative but we are saying the same things, I think.

#23 Updated by Constantin Asofiei over 1 year ago

  • Related to Feature #6433: implement WEB-CONTEXT system handle added

#24 Updated by Constantin Asofiei over 1 year ago

I've ran conversion only for these files:

abl/src/web/objects/web-disp.p
abl/src/webutil/webstart.p
abl/src/web/objects/stateaware.p
abl/src/web/objects/web-util.p
abl/src/web/objects/session.p
abl/src/web/support/webinput.p
abl/src/web/support/webtog.p
abl/src/web/support/webradio.p
abl/src/web/support/weblist.p
abl/src/web/support/webedit.p
abl/src/web/support/tagrun.p
abl/src/web/support/webmsg.p

From these, FWD requires conversion and runtime stubs for:
  • web-context handle:
    • attributes:

config-name, current-environment, session-end, utc-offset

  • methods:

get-binary-data(), get-config-value(), get-cgi-list(), get-cgi-value(), get-cgi-long-value(), increment-exclusive-id(), urldecode() and url-decode()

  • compiler handle: error-row attribute. We need good default values for other 'unsupported' features like COMPILER:ERROR, to i.e. return false.

This include files look to be used:

./abl/src/web/method/admweb.i
./abl/src/web/method/cgiarray.i
./abl/src/web/method/cgidefs.i
./abl/src/web/method/cgiutils.i
./abl/src/web/method/cookies.i
./abl/src/web/method/message.i
./abl/src/web/method/proto.i
./abl/src/web/method/session.i
./abl/src/web/method/tagmap.i
./abl/src/web/method/webutils.i

I'll work on adding conversion and runtime stubs for these in #6433.

I'm not running reports and callgraph to see if there are other RUN statements or other features which are 'unsupported'.

#25 Updated by Constantin Asofiei over 1 year ago

I've found this statement usage in abl/src/web/objects/web-util.p.cache and abl/src/web/objects/stateaware.p.cache:

do on quit, leave:
message "quit".
end.

According to docs:

If you do not specify the UNDO option, then the current transaction is committed when the QUIT statement is executed.

My plan is to emit a Condition.QUIT_COMMIT, which will not rollback; I'll leave the runtime deferred for this.

#26 Updated by Constantin Asofiei over 1 year ago

The files currently can get converted and compiled with 6433a are attached.

Radu: for now, you can place them in your abl/ folder alongside the test program which would be ran for webspeed request. I expect 6433a to be in trunk in the next few days, so you will need to rebase your branch.

#27 Updated by Roger Borrello over 1 year ago

By using the attached prepare_template.json, I can:
  1. Checkout hotel_gui to adepre
  2. Place the prepare_template.json file in place
  3. Run ./json_template.sh
  4. Update bootstrap/set_project_properties.sh so needs_ade=true and needs_possenet=false
  5. Run ./bootstrap_conversion.sh -n -f ~/projects/fwd/6433a (I built the latest branch.)
  6. Place the attached zfile_set.txt file in place
  7. Run ant deployapp

And I have a build. I'm not sure if it helps to have a project, but I thought I'd document it.

#28 Updated by Roger Borrello over 1 year ago

  • File deleted (prepare_template.json)

#30 Updated by Roger Borrello over 1 year ago

  • Status changed from New to WIP
  • % Done changed from 0 to 50

When you follow the build steps, you get adepre.jar. Would this be how we would deliver this task? Like we do H2?

#31 Updated by Greg Shah over 1 year ago

I was planning for these converted classes to be directly checked in to the FWD project. The advantage of that approach is that it is less fragile. We won't be handling support issues where webspeed is missing because someone didn't deploy an additional jar.

The advantage of the separate jar approach is that it is easier to maintain on its own. But then again, it will need to be reconverted when FWD itself changes which is ALL THE TIME.

#32 Updated by Roger Borrello over 1 year ago

How best to merge, then? Should there be a separate build.xml in a directory from FWD and have all the source code of ADE checked into trunk?

#33 Updated by Greg Shah over 1 year ago

Roger Borrello wrote:

How best to merge, then? Should there be a separate build.xml in a directory from FWD and have all the source code of ADE checked into trunk?

I don't think we need a separate build.xml. If the code was checked into FWD trunk, then it will be included in p2j.jar.

#34 Updated by Roger Borrello over 1 year ago

Greg Shah wrote:

Roger Borrello wrote:

How best to merge, then? Should there be a separate build.xml in a directory from FWD and have all the source code of ADE checked into trunk?

I don't think we need a separate build.xml. If the code was checked into FWD trunk, then it will be included in p2j.jar.

Out of the build from my "adepre" project, is the below. What should the target directory structure look like, so that the classes can be found properly?

#35 Updated by Greg Shah over 1 year ago

Constantin will decide. We should isolate these files in their own package so they are easy to replace. You'll have to reconvert once you update the p2j.cfg.xml with the package paths.

#36 Updated by Roger Borrello over 1 year ago

Constantin, I have kind of project I haven't actually checked into bazaar, which is the below contents of zfile_set.txt, and the build.xml creates build/lib/adepre.jar.

F ./abl/ade/src/web/objects/web-disp.p
F ./abl/ade/src/webutil/webstart.p
F ./abl/ade/src/web/objects/stateaware.p
F ./abl/ade/src/web/objects/web-util.p
F ./abl/ade/src/web/objects/session.p
F ./abl/ade/src/web/support/webinput.p
F ./abl/ade/src/web/support/webtog.p
F ./abl/ade/src/web/support/webradio.p
F ./abl/ade/src/web/support/weblist.p
F ./abl/ade/src/web/support/webedit.p
F ./abl/ade/src/web/support/tagrun.p
F ./abl/ade/src/web/support/webmsg.p

The adepre.jar contains the classes under com/goldencode/adepre/, for example com/goldencode/adepre/ade/src/webspeed/.

What do you suggest for incorporating this into FWD project?

#37 Updated by Roger Borrello over 1 year ago

  • Status changed from WIP to Feedback

Just need a little direction into how best to merge into FWD.

#38 Updated by Greg Shah over 1 year ago

In #6506, we are putting the webspeed code into src/com/goldencode/p2j/webspeed/. So we would want a subdirectory under that to hold the preconverted ADE code.

The only question here is the pathing, since we must process these programs as executed from RUN statements and the like. This means that we should probably replicate the relative portion of the path that would be manually listed in the RUN statement. That way, we could have an implicit propath entry and name_map.xml content that will properly resolve. I don't want us to add abl/ into the path and probably not something named adepre which is pretty cryptic.

Roger: Make a proposal.

Constantin: Thoughts?

#39 Updated by Constantin Asofiei over 1 year ago

If we set com.goldencode.p2j.webspeed.<something?> as the root package, then the conversion will emit automatically the .java in this package. The abl/ folder is not emitted in name_map.xml.

As there are IN handle defs, we will need the name_map.xml in the package where the converted webspeed sources will be placed. And we can later on load this name_map.xml only if webspeed is enabled (SourceNameMapper already supports multiple name_map.xml files).

#40 Updated by Roger Borrello over 1 year ago

Isn't there kind of a chicken-egg situation here?
  1. which version of FWD to use for the conversion of webspeed so that we have a pre-converted webspeed to include in FWD?
  2. build FWD so that the pre-converted webspeed is part of p2j.jar

#41 Updated by Constantin Asofiei over 1 year ago

Roger Borrello wrote:

Isn't there kind of a chicken-egg situation here?
  1. which version of FWD to use for the conversion of webspeed so that we have a pre-converted webspeed to include in FWD?
  2. build FWD so that the pre-converted webspeed is part of p2j.jar
Well, this will be done in two phases:
  1. convert the webspeed programs and get the .java code and name_map.xml
  2. add them to FWD trunk via a branch

FWD trunk at this point should be able to convert the classes in #9516-24 and compile them.

#42 Updated by Roger Borrello over 1 year ago

I just connected trunk_15818 to my "adepre" project and ran ant clean convert.zset and created:

If I follow this, we need to place these somewhere in branch 6506a (without ./src/com/goldencode/adepre/ path) under src/com/goldencode/p2j/webspeed/.

Is that close? I'd need to understand what to do with the ui and dmo files, and how to merge name_map.xml

#43 Updated by Greg Shah about 1 year ago

For details on how we properly document the licensing of the ADE code, see Inclusion of Apache 2 Licensed ADE Code.

#44 Updated by Radu Apetrii about 1 year ago

While trying to set up the webspeed sample, I encountered something that I don't know how to fix. Basically, I updated zfile_set.txt to include the files mentioned in #9516-34, and I made sure that the propath includes both ./abl and ./abl/ade folders. However, when running ant deployapp (or ant clean convert.zset) I get the following error for web-util.p:

Can I get a bit of help for what I need to include and where? Thank you!

#45 Updated by Radu Apetrii about 1 year ago

One more thing, which might be needed in the future configurations: this is how web-disp.p is included in the 4GL environment:

srvrStartupParam=-p web\objects\web-disp.p -weblogerror

In my test project I had it directly in abl so it was easier to mention it in the directory, but things might need to be adjusted for the webspeed sample's directory.

#46 Updated by Constantin Asofiei about 1 year ago

You need to copy the skeleton project , and also add <parameter name="oo-skeleton-path" value="skeleton" /> in p2j.cfg.xml.

#47 Updated by Constantin Asofiei about 1 year ago

You will also need to add ${P2J_HOME}/abl/ade/src/ to the PROPATH in p2j.cfg.xml.

#48 Updated by Radu Apetrii about 1 year ago

Just a quick question. When running the web-disp.p program, at some point it calls the reset-tagmap-utilities from web-util.p. In there, there is this piece of code:

  /* Make sure the tagmap.dat file exists in the PROPATH */
  ASSIGN 
    tagmapfile = SEARCH("tagmap.dat":U)
    i-count    = 0.
  IF tagmapfile EQ ? THEN DO:
    DYNAMIC-FUNCTION ("logNote":U IN web-utilities-hdl, "Error":U,
                                 "The file '":U + "tagmap.dat":U + "' was not in your PROPATH":U) NO-ERROR.
    RETURN ERROR.
  END.

At the moment, this results in an error (tagmapfile is equal to ?) because tagmap.dat cannot be found. Where should I find this file, or how should I generate it?

#49 Updated by Radu Apetrii about 1 year ago

  • File tagmap.dat added

From what I'm seeing, this is the default tagmap.dat. Now, where should I place it? Or should something else happen?

#50 Updated by Constantin Asofiei about 1 year ago

  • File deleted (tagmap.dat)

#51 Updated by Radu Apetrii about 1 year ago

Mmm, I'm now facing another issue with a file I had to include. This is the piece of code:

FUNCTION getObjectType RETURNS CHARACTER
  ( ) :
/*------------------------------------------------------------------------------
  Purpose:  Returns the Type of the SmartObject, such as "SmartDataObject", etc.
            For an ADM Super procedure, this function returns "SUPER".
   Params:  <none>
------------------------------------------------------------------------------*/

  DEFINE VARIABLE cType AS CHARACTER NO-UNDO.

     NO-ERROR.

  RETURN cType.

END FUNCTION.

Now, FWD is uncertain what to do with the NO-ERROR, hence it throws a conversion error. Unfortunately, this comes from a deeply nested chain of include files so it's a bit difficult to determine the exact location. In the meantime, are there other suggestions other than just commenting out the NO-ERROR?

#52 Updated by Constantin Asofiei about 1 year ago

What file is this? Is this after parse (the .cache) or before it?

Try commenting the offending NO-ERROR. line.

#53 Updated by Radu Apetrii about 1 year ago

Constantin Asofiei wrote:

What file is this? Is this after parse (the .cache) or before it?

The file is a .cache file, and that code comes from some includes. The file is ade/src/webutil/debug.p.cache.

Try commenting the offending NO-ERROR. line.

That's what I'm searching right now, the root of the code.

#54 Updated by Constantin Asofiei about 1 year ago

Get the line from debug.p.cache and look into the debug.p.pphints for the include file matching that line.

#55 Updated by Radu Apetrii about 1 year ago

I've reached a point in which I think there's a conversion issue (maybe?). I'll post the code that's executing here, but I have a testcase created already if we need to make a new Redmine ticket.

In web-disp.p.cache, the program does this:

  /* Output any pending messages queued up by queue-message() */
  IF available-messages(?) THEN
    output-messages("all", ?, "Messages:").

The available-messages function is from web-util.p.cache:

FUNCTION available-messages RETURNS LOGICAL
  (INPUT p_grp AS CHARACTER) :

  DEFINE VARIABLE v-status AS LOGICAL NO-UNDO.

  FOR FIRST ttMessage WHERE
    (IF p_grp = ? THEN TRUE       /* match any group */
     ELSE ttMessage.grp = p_grp): /* else, match specified group */
    ASSIGN v-status = TRUE.
  END.

  RETURN v-status.
END FUNCTION. /* available-messages */

Notice how it takes an input parameter, and the call of the function passes ? as the argument. Now, the converted code looks like this:

            if ((FUNC_CALL_SITE_1.clone().execute(logical.class)).booleanValue())
            {
               FUNC_CALL_SITE_2.clone().execute("all", new character(), "Messages:");
            }

where FUNC_CALL_SITE_1 is available-messages. That ? was not included in the arguments, hence, when it checks the validity of arguments, the program throws an error saying that there's a mismatch in the number of parameters expected. I have a pretty small testcase with this issue that fails with both 6506a and trunk, so let me know if I should create another task.

#56 Updated by Radu Apetrii about 1 year ago

Another update. I modified for the moment the java file, just to pass that point (from FUNC_CALL_SITE_1.clone().execute(logical.class) to FUNC_CALL_SITE_1.clone().execute(logical.class, new unknown())), and I can say that now, when I do a webspeed request, I finally have some generated output printed back in the browser (or Postman):

I know there's still than Unable to run web object file message, but it's the first time some sort of output was printed in the browser from a webspeed request in the webspeed sample. I'm looking at the logs and I see all of this unimplemented feature things:

Should I implement these now since they're used in this example? Or just check what would really be necessary? I will also check that stream closing error.

#57 Updated by Radu Apetrii about 1 year ago

For the WEB-CONTEXT:GET-CONFIG-VALUE implementation, which seems to be quite important, 4GL reads the values from the ubroker.properties file. How does that translate into FWD? Will we have the same file in the project? Will we do our own version? Will we add nodes in the directory with these configurations? I would post a default ubroker.properties file here, but there might be some licensing issues.

#58 Updated by Constantin Asofiei about 1 year ago

Radu Apetrii wrote:

Will we add nodes in the directory with these configurations?

They should map to directory settings.

Now, for CURRENT-ENVIRONMENT, INCREMENT-EXCLUSIVE-ID, UTC-OFFSET, and other stubs in WebContextImpl - please implement them. Where you use UnimplementedFeature, make sure to also log the arguments, so we now without debugging what is missing for i.e. GET-CGI-VALUE or GET-CONFIG-VALUE.

#59 Updated by Constantin Asofiei about 1 year ago

Radu Apetrii wrote:

I have a pretty small testcase with this issue that fails with both 6506a and trunk, so let me know if I should create another task.

Please create a task for this and link it to this one.

You can also bypass by using an unknown var instead of the ? literal.

#60 Updated by Radu Apetrii about 1 year ago

Constantin Asofiei wrote:

Radu Apetrii wrote:

Will we add nodes in the directory with these configurations?

They should map to directory settings.

I assume I need to add those in the directory. Is there a suitable place (like for example under the WebspeedHandler node)?

#61 Updated by Constantin Asofiei about 1 year ago

Radu Apetrii wrote:

I assume I need to add those in the directory. Is there a suitable place (like for example under the WebspeedHandler node)?

Document them first. that's why the request to enhance the logging to include what arguments were used and 'unsupported.

#62 Updated by Radu Apetrii about 1 year ago

I've made some progress and I'm down to only these ones right now:

I've also included the argument values for get-config-value. And for get-config-name I'm a bit unsure how the implementation should be. The config name seems to be the broker name, but it's more of a guess. It's a bit confusing when looking at both the documentation and what I'm seeing in 4GL.

#63 Updated by Constantin Asofiei about 1 year ago

Radu:
  • WEB-CONTEXT:GET-CONFIG-VALUE(srvrAppMode) - this is the agent's appserver mode (like @Stateless)
  • WEB-CONTEXT:GET-CONFIG-VALUE(srvrDebug) - always 'no'
  • WEB-CONTEXT:GET-CONFIG-VALUE(applicationURL) - what does OE webspeed report for this? If needed, make it configurable at the webspeed configuration not the appserver
  • WEB-CONTEXT:GET-CONFIG-VALUE(defaultCookiePath) and defaultCookieDomain - this needs to be set at the webspeed configuration; use "/" and "localhost" as defaults
  • WEB-CONTEXT:GET-CONFIG-VALUE(defaultCookieDomain) - this method is not implemented. Argument value: defaultCookieDomain
  • WEB-CONTEXT:GET-CONFIG-VALUE(wsRoot) - don't we have the cgi.exe configured in directory already?
  • WEB-CONTEXT:GET-CONFIG-NAME always return the appserver name
  • MISSING: Open Web Stream: paged and page-size are not supported yet. - we need to add this, I assume there is some flushing done... but it shouldn't be mandatory to make the sample run.
  • MISSING: GET-CGI-LIST with FORM and QUERY - what does OE webspeed return? I assume QUERY is everything after the ? char ?

#64 Updated by Radu Apetrii about 1 year ago

Constantin, I've reached a point that needs some more implementations. Here's what the current server log shows:

25/06/06 10:31:10.840+0300 |  SEVERE | com.goldencode.p2j.util.UnimplementedFeature [UnimplementedFeature.UNSUPPORTED()] | ThreadName:Conversation [00000003:appserver_process], Session:00000003, ThreadId:00000005, User:appserver_process | UNSUPPORTED: COMPILER:ERROR attribute getter is not supported
25/06/06 10:31:26.213+0300 |  SEVERE | com.goldencode.p2j.util.UnimplementedFeature [UnimplementedFeature.UNSUPPORTED()] | ThreadName:Conversation [00000003:appserver_process], Session:00000003, ThreadId:00000005, User:appserver_process | UNSUPPORTED: COMPILER:ERROR attribute getter is not supported
25/06/06 10:31:47.319+0300 |  SEVERE | com.goldencode.p2j.util.UnimplementedFeature [UnimplementedFeature.UNSUPPORTED()] | ThreadName:Conversation [00000003:appserver_process], Session:00000003, ThreadId:00000005, User:appserver_process | UNSUPPORTED: COMPILER:FILENAME attribute getter is not supported
25/06/06 10:31:47.323+0300 |  SEVERE | com.goldencode.p2j.util.UnimplementedFeature [UnimplementedFeature.UNSUPPORTED()] | ThreadName:Conversation [00000003:appserver_process], Session:00000003, ThreadId:00000005, User:appserver_process | UNSUPPORTED: COMPILER:ERROR-ROW attribute getter is not supported

For example, the first one is called in web-util.p.cache at line:

  /* Did the code run okay?  Also trap for compiler error here, since some code 
     may run a program directly without running it through run-web-object */
  lRunOk = (NOT ERROR-STATUS:ERROR AND NOT COMPILER:ERROR).

How should I approach this?

#65 Updated by Constantin Asofiei about 1 year ago

Hmm... in FWD we emit true for CompilerOps.error(). Change it to FALSE for now.

#66 Updated by Radu Apetrii about 1 year ago

Constantin Asofiei wrote:

Hmm... in FWD we emit true for CompilerOps.error(). Change it to FALSE for now.

Thanks, it works!

Now I am able to run the example from webspeed sample without encountering errors. However, if I do multiple requests without closing the server in between, I sometimes get this error:

25/06/06 11:23:20.542+0300 |  SEVERE | com.goldencode.p2j.util.TransactionManager | ThreadName:Conversation [00000003:appserver_process], Session:00000003, ThreadId:00000004, User:appserver_process | Abnormal end; original error:
org.eclipse.jetty.http.BadMessageException: 500: No version
    at org.eclipse.jetty.http.HttpGenerator.generateResponse(HttpGenerator.java:393)
    at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:749)
    at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:248)
    at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:229)
    at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:555)
    at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:1009)
    at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:1086)
    at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:285)
    at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:269)
    at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:751)
    at com.goldencode.p2j.util.GenericOutputStream.closeOut(GenericOutputStream.java:356)
    at com.goldencode.p2j.util.GenericOutputStream.close(GenericOutputStream.java:371)
    at com.goldencode.p2j.util.StreamWrapper.assign(StreamWrapper.java:572)
    at com.apptechnologies.webspeedsample.ade.src.web.objects.WebDisp.lambda$null$5(WebDisp.java:474)
    ...

The instruction from GenericOutputStream.closeOut() that is being executed is stream.flush(), and from web-disp is webStreamStream.assign(StreamFactory.openWebStream(true)). This indicates that it tries to close the previous web stream first by doing a flush() and a close(), but it gets stuck at the first command. Any suggestions?

#67 Updated by Constantin Asofiei about 1 year ago

What's happening if flush is not executed? Maybe override this when you open the WEB stream?

#68 Updated by Radu Apetrii about 1 year ago

Constantin Asofiei wrote:

What's happening if flush is not executed? Maybe override this when you open the WEB stream?

I would like to answer this, but I wasn't able to replicate it after I posted the message. The behavior is very indeterminate. Sometimes it works fine, sometimes there's an EoFException when doing stream.flush() or stream.close(), sometimes I see ** Pipe to subprocess has been broken. (140) when sending the web-notify event. The thing is that the first call always works correctly, but from the second one forward things get a bit random. If you only take the last sentence into consideration, it looks like stuff isn't properly cleared after a request, I think.

#69 Updated by Constantin Asofiei about 1 year ago

Radu Apetrii wrote:

Constantin Asofiei wrote:

What's happening if flush is not executed? Maybe override this when you open the WEB stream?

I would like to answer this, but I wasn't able to replicate it after I posted the message. The behavior is very indeterminate. Sometimes it works fine, sometimes there's an EoFException when doing stream.flush() or stream.close(), sometimes I see ** Pipe to subprocess has been broken. (140) when sending the web-notify event. The thing is that the first call always works correctly, but from the second one forward things get a bit random. If you only take the last sentence into consideration, it looks like stuff isn't properly cleared after a request, I think.

Lets avoid the flush for now.

#70 Updated by Radu Apetrii about 1 year ago

Alright, so, the problem is that the program tries to write something to a closed stream. Here's how that happens. First, this is the 4GL instruction:

OUTPUT STREAM WebStream TO "WEB":U.

which converts to

webStreamStream.assign(StreamFactory.openWebStream(true));

and this is the assign method:

So, the key here is that stream and the s parameter are the exact same stream. Then, during the cleanup part, the stream is closed, and then it is assigned with s. But because s is the exact same, it was closed as well during the cleanup. So now we have assigned a closed stream to stream, which will then cause trouble later on.

#71 Updated by Constantin Asofiei about 1 year ago

What does StreamFactory.openWebStream(true) do with the GenericOutputStream.close() - it actually closes something? If yes, then that needs to be fixed.

#72 Updated by Radu Apetrii about 1 year ago

Constantin Asofiei wrote:

What does StreamFactory.openWebStream(true) do with the GenericOutputStream.close() - it actually closes something? If yes, then that needs to be fixed.

No, it just retrieves the stream. Here's the code:

  public static Stream openWebStream(boolean write, int pageSz, boolean paged)
   {
      UnimplementedFeature.missing("Open Web Stream: paged and page-size are not supported yet.");

      OutputStream os;
      try
      {
         os = WebHandlerHelper.getHttpResponse().getOutputStream();
         if (write)
         {
            return new GenericOutputStream(os);
         }
         else
         {
            UnimplementedFeature.missing("Open Web Stream: Input streams are not supported yet.");
         }
      }
      catch (IOException e)
      {
         LOG.log(Level.WARNING, "Could not retrieve output stream due to " + e.getMessage());
      }

      return null;
   }

The closing happens only inside assign.

#73 Updated by Radu Apetrii about 1 year ago

Constantin: is there a way to determine if two Stream s are the same? I'm thinking that inside assign, if the two streams represent the same one, a close() should not happen. If they are the same, we can skip over closing completely, or maybe maybe call flush() just to be sure that there's nothing left behind.

#74 Updated by Constantin Asofiei about 1 year ago

Lets make the WEB stream do nothing on flush or close - does this solve it?

#75 Updated by Radu Apetrii about 1 year ago

Constantin Asofiei wrote:

Lets make the WEB stream do nothing on flush or close - does this solve it?

Yes, in this case, I can run as many requests as I want and the output is fine every time.

#76 Updated by Constantin Asofiei about 1 year ago

Radu Apetrii wrote:

Constantin Asofiei wrote:

Lets make the WEB stream do nothing on flush or close - does this solve it?

Yes, in this case, I can run as many requests as I want and the output is fine every time.

Great!

#77 Updated by Radu Apetrii about 1 year ago

One more question. I'm looking at the two outputs:

  1. Progress output:
  1. FWD output:

The only difference between these two is that FWD has two extra lines in the beginning. However, at a closer inspection, shouldn't these be part of the response header? They are currently in the response body.

The 4GL code that is responsible for setting the first line is from web-util, function output-http-header:

  /* Output the header and associated value to the output stream */
  PUT STREAM WebStream CONTROL
        p_header
        p_value
        /* Newline must have both CR and LF even on UNIX.
           Bug: 97-03-04-008  Some web servers such as Netscape-Fasttrack 2.01
           don't like the CR character so allow the newline to be changed. */
        http-newline.

where p_header is Set-Cookie: and p_value is SERVER_CONNECTION_ID=local::app_server::0::e3b7c1d1-8078-4bc8-92a1-8b4598cd0534; path=/; domain=localhost. This converts into
         webStreamStream.putControl(new FieldEntry[]
         {
            new PutField(pHeader_1),
            new PutField(pValue_6),
            new PutField(httpNewline)
         });

which seems fine. Looking at these two pieces of code, I'm not sure how the line is supposed to end up in the response header. Does 4GL do some additional processing on the request and response?

#78 Updated by Constantin Asofiei about 1 year ago

Radu Apetrii wrote:

One more question. I'm looking at the two outputs:

What is the RAW response in Postman? Or curl? For both 4GL and FWD. Maybe FWD is missing some prolog like HTTP 1.1

#79 Updated by Radu Apetrii about 1 year ago

RAW response in 4GL:

RAW response in FWD:

Curl in 4GL:

Curl in FWD:

#80 Updated by Greg Shah about 1 year ago

Constantin Asofiei wrote:

Radu Apetrii wrote:

Constantin Asofiei wrote:

Lets make the WEB stream do nothing on flush or close - does this solve it?

Yes, in this case, I can run as many requests as I want and the output is fine every time.

Great!

The stream is provided by Jetty? Is that correct?

If so, I guess Jetty will close it and we don't have to. Is that the idea? I don't want us to leak resources, but if the resources get closed by jetty we should be fine.

#81 Updated by Constantin Asofiei about 1 year ago

Greg Shah wrote:

If so, I guess Jetty will close it and we don't have to. Is that the idea? I don't want us to leak resources, but if the resources get closed by jetty we should be fine.

Yes, we wrap the Jetty response 'output stream' inside a FWD stream, and write to it. We already not doing any flush/close for the other /rest or /soap legacy support, so is safe to leave the stream open and let Jetty handle it.

#82 Updated by Constantin Asofiei about 1 year ago

Greg, we still have to bring the ADE converted source code into FWD. So, I assume for webspeed we will have a com.goldencode.p2j.webspeed folder? If so, the problem is where to place the name_map.xml. I suggest to place it in com/goldencode/p2j/name_map.xml, and this will encode the webspeed stuff and any other programs (not webspeed) we need to include.

Radu: please send me the project you've been using.

#83 Updated by Greg Shah about 1 year ago

Greg, we still have to bring the ADE converted source code into FWD. So, I assume for webspeed we will have a com.goldencode.p2j.webspeed folder? If so, the problem is where to place the name_map.xml. I suggest to place it in com/goldencode/p2j/name_map.xml, and this will encode the webspeed stuff and any other programs (not webspeed) we need to include.

Agreed

#84 Updated by Radu Apetrii about 1 year ago

I will try to explain the issue I am seeing with the header content being shown in the body. Basically, in 4GL, one will open the stream, write the content, and then the browser will treat what's before \r\n\r\n as the header, and what's after as the body. In Jetty (i.e. in Java), when one calls getOutputStream() during the opening of the stream part, the program expects the headers to be already in place. So, in our scenario:
  • during opening the stream: it sees that no header was set, so it puts some default values (e.g. content-type: text/plain instead of text/html).
  • when writing to the stream: although the \r\n\r\n delimiter exists, the header was already set, meaning that what's written to the stream appears in the body. Not only that, but setting the header after getOutputStream() was called is forbidden. As far as I'm seeing, there are some exceptions, for example content-type which can be set after.

Am I allowed to change the converted Java code from web-util or web-disp, or should I treat that as a last resort? I will post shortly some solutions that I'm thinking about, although they are a bit ugly, I won't lie.

#85 Updated by Constantin Asofiei about 1 year ago

Radu Apetrii wrote:

Am I allowed to change the converted Java code from web-util or web-disp, or should I treat that as a last resort? I will post shortly some solutions that I'm thinking about, although they are a bit ugly, I won't lie.

The approach needs to be in FWD. Use something like this:
  • create a new WebOutputStream which inherits GenericOutputStream, and saves also the HTTP response instance
  • WebOutputStream uses as a cache a ByteArrayOutputStream (or StringBuilder?), until \r\n\r\n is encountered:
    • at that point, it parses the received text and sets the headers/cookies/etc
    • after this, it switches to 'output' mode and actually calls response.getOutputStream() and everything will be written to that

#86 Updated by Radu Apetrii about 1 year ago

Constantin Asofiei wrote:

Radu Apetrii wrote:

Am I allowed to change the converted Java code from web-util or web-disp, or should I treat that as a last resort? I will post shortly some solutions that I'm thinking about, although they are a bit ugly, I won't lie.

The approach needs to be in FWD. Use something like this:
  • create a new WebOutputStream which inherits GenericOutputStream, and saves also the HTTP response instance
  • WebOutputStream uses as a cache a ByteArrayOutputStream (or StringBuilder?), until \r\n\r\n is encountered:
    • at that point, it parses the received text and sets the headers/cookies/etc
    • after this, it switches to 'output' mode and actually calls response.getOutputStream() and everything will be written to that

I was literally writing the same thing, it's incredible. I might have a follow-up question, but let me double-check something.

#87 Updated by Radu Apetrii about 1 year ago

Done. Committed to 6506a rev. 15766. I added the implementation for WebOutputStream and replaced that occurrence of GenericOutputStream when opening the web stream. The FWD behavior now fully matches the 4GL one in the webspeed sample. I will send a short video so it's easier to see what I'm talking about.

#88 Updated by Radu Apetrii about 1 year ago

Here's the promised video.

#89 Updated by Constantin Asofiei about 1 year ago

Greg - even if we bring the webspeed files in FWD, the application converted .p's or .htm's can be dependent on include files like src/web/method/e4gl.i

For now, the sample requires (some of) these files:

ca@xever:~/projects/webspeed_sample$ ls -al abl/src/web/method/
total 172
drwxrwx--- 2 ca ca  4096 Jun 18 13:53 .
drwxrwx--- 3 ca ca  4096 Jun 18 13:53 ..
-rw-rw---- 1 ca ca  9972 May 15 15:28 admweb.i
-rw-rw---- 1 ca ca  3245 May 15 15:28 cgiarray.i
-rw-rw---- 1 ca ca  7468 May 15 15:28 cgidefs.i
-rw-rw---- 1 ca ca 52160 May 15 15:28 cgiutils.i
-rw-rw---- 1 ca ca 15562 May 15 15:28 cookies.i
-rw-rw---- 1 ca ca  3007 May 15 15:28 e4gl.i
-rw-rw---- 1 ca ca 21386 May 15 15:28 html-map.i
-rw-rw---- 1 ca ca  1295 May 15 15:28 htmoff.i
-rw-rw---- 1 ca ca 13578 May 15 15:28 message.i
-rw-rw---- 1 ca ca  6727 May 15 15:28 proto.i
-rw-rw---- 1 ca ca  1509 May 15 15:28 session.i
-rw-rw---- 1 ca ca  1040 May 15 15:28 tagmap.i
-rw-rw---- 1 ca ca  2747 May 15 15:28 webutils.i
-rw-rw---- 1 ca ca  1906 May 15 15:28 wrap-cgi.i

#90 Updated by Greg Shah about 1 year ago

Yes, that is understood. Includes from the ADE code can be used in customer 4GL code. For webspeed, it is required since some of those includes are added during e4gl preprocessing.

We should not use Possenet here. We should use 12.8.x ADE.

#91 Updated by Constantin Asofiei about 1 year ago

Greg Shah wrote:

Yes, that is understood. Includes from the ADE code can be used in customer 4GL code. For webspeed, it is required since some of those includes are added during e4gl preprocessing.

Yes, what I posted above is from ade (I just didn't use the abl/ade/ folder for the test).

We should not use Possenet here. We should use 12.8.x ADE.

Then this will be a configuration for the converted application.

6506a rev 15767 adds the ADE files related to webspeed (web-disp.p and minimum support programs) and FWD changes to support DMOs and programs (via name_map.xml). Greg, please review.

#92 Updated by Greg Shah about 1 year ago

  • Assignee changed from Roger Borrello to Constantin Asofiei
  • Status changed from Feedback to WIP

The changes look good to me. What is left to do?

#93 Updated by Constantin Asofiei about 1 year ago

Greg Shah wrote:

What is left to do?

Only if we get more dependencies after using a more real application/sample. Otherwise, it works.

#94 Updated by Greg Shah about 1 year ago

  • % Done changed from 50 to 100
  • Status changed from WIP to Internal Test

#95 Updated by Greg Shah 5 months ago

  • topics WEB/WebSpeed/WebHandler added

Also available in: Atom PDF