FWD » Core Development » Runtime Infrastructure

Batch processes can be made to work similar to PASOE agents, full server-side, assuming that there is no need of interactive terminal.

There are not many static fields in the p2j.ui package remaining. After we clean this up and make them context-local, what we need to do is run a FWD client only on server-side.

Initial proposal is to do this still via ClientDriver, which after authentication, if it sees server-side-resources=ALL, it switches automatically to this 'headless' mode (i.e. closes the socket, etc) and initializes ThinClient and other needed UI resources on server-side.

But, this does not get rid of the OS process launch, which can be problematic when many clients need to be launch at the same time.

Instead, the solution will be to provide FWD Java APIs to launch a batch process fully server-side, in 'headless' mode, directly from the applications code. This way, there will no longer be an OS process.

An additional complication is OS-GETENV. There are applications which pass state to the launched batch process via OS environment variables. This can't be made to work without adding a separate mechanism of passing these env vars to the FWD Java API launching the OS batch process, so that we still have context-local separation of OS-GETENV (otherwise, all clients would read FWD Server's env vars).

Constantin Asofiei wrote:

Instead, the solution will be to provide FWD Java APIs to launch a batch process fully server-side, in 'headless' mode, directly from the applications code. This way, there will no longer be an OS process.

Greg, an additional note: do we want to have REST APIs instead of Java APIs to launch the batch processes?

I would generally prefer that. But I want to make sure that we plan for the most likely usage scenario. How would it be used to replace the launching of _progres or pro sessions in a script?

Greg Shah wrote:

I would generally prefer that. But I want to make sure that we plan for the most likely usage scenario. How would it be used to replace the launching of _progres or pro sessions in a script?

• the REST API will authenticate with a FWD account which has the correct permissions to launch batch processes
• the payload will specify the FWD account under which to run the batch program, plus all the other configurations to initialize the FWD context (bootstrap config, OS env vars, workingDir, etc)

Also, this would make FWD easier to integrate with other apps. And also does not rely on a JVM process to connect to the FWD server and launch this.

OK, go ahead with the REST API.

The initial work on running client-side UI in server context was implemented in 9055a. It added very limited set of use cases, like the MESSAGE statement.

• mixed mode PUT and DISPLAY:
  • trunk server-side filesystem,streams: 11525ms/9869ms
  • trunk no-server-side: 11135ms/9936ms
  • 11371 server-side all: 2677ms/1823ms
• only PUT:
  • trunk server-side filesystem,streams: 121ms/4085ms
  • trunk no-server-side: 3919ms/3446ms
  • 11371 server-side all: 58ms/68ms
• only DISPLAY
  • 11371 server-side all: 2461ms/1717ms

I'll cleanup the changes, run automated review and start more extensive testing in real applications (I want to see if I can modify the batch launch code to actually be server-side=all for them). I also want to do some profiling to see where time is spent for the mixed PUT/DISPLAY case, the difference compared only with PUT is too high.

• the refactoring of the FileStream to keep an in-memory buffer was not yet made. I'll work on this in a separate branch (I've been thinking about this, but there are caveats to know when to flush to binary buffer, what happens when cursor is moved, etc).
• the configuration for server-side-resources remains in the directory - it can not be configured via bootstrap configurations. There are lots of 'chicken and the egg' issues and to add it complicates things a lot.
• REST APIs added:
  • Greg - do we want an ACL to allow a batch account to be used via REST?
  • GET /admin/launch - gets the list of all batch processes which can be launched; it skips only batch accounts configured as server or appserver.
    

    {
        "processes": {
            "serversidebatch": true,
            "appserver_agent": true
        }
    }
    

  • POST /admin/launch/<batchprocess> - launches a batch process, where the payload can specify custom OS properties and bootstrap configurations:
    

    {
        "properties" : 
        {
           "customproperty" : "blabla" 
        },
        "configs" : 
        {
           "client:cmd-line-option:startup-procedure" : "test1.p" 
        }
    }
    

• launching can also be done via ServerDriver -b - it will honor the server-side-resources=all from the directory

to allow a batch account to be used via REST

What do you mean by this? Used how or in what way?

Greg Shah wrote:

to allow a batch account to be used via REST

What do you mean by this? Used how or in what way?

I mean to be launched via REST. To allow explicitly this feauture for a process account.

Also I need to check what ACLs we have for the startup program - do we allow any program for REST launching?

I know the launching is secured via the token (so only an authenticated admin user can run this). So is more if we want another layer of checks.

About security: we already have EntryPointResource to secure the entry points. This should be enough.

Current changes are in 11371a rev 16687. I've also added server-side-resources=all support for launching classic appserver agents - in this mode, there will be no JVM for a FWD Client. This will help reducing the need for system memory usage if these clients are server-side, plus the FWD server startup time.

• there is an issue with WAIT-FOR (is not blocking). I'm looking into this.
• all STDOUT or other client-side output in this mode will go to the FWD Server's JVM STODUT or logging - this is the same as for MSA, I don't think we have another approach for this.
• currently filesystem and streams can work on server-side only and only if ALL are server-side. To keep this, we need back the #11271 support. But, considering the findings from this work (any terminal requirement like unnamed streams or UI statements must have the stream on the client), this is not something viable. I don't think we have any applications currently configured with server-side filesystem for interactive clients (not batch).
• more testing including MSA, classic agents and normal clients.

Greg - currently only REST allows specifying custom bootstrap config and OS environment overrides. I'm thinking to add this support also for ServerDriver -b, IMO it will complicate things if we want to launch these via shell scripts (i.e. maintaining the REST ADMIN user and password, ACLs, etc).

Constantin Asofiei wrote:

Greg - currently only REST allows specifying custom bootstrap config and OS environment overrides. I'm thinking to add this support also for ServerDriver -b, IMO it will complicate things if we want to launch these via shell scripts (i.e. maintaining the REST ADMIN user and password, ACLs, etc).

This support is in 11371a/16690; the syntax is:

ServerDriver -bcfg  client:cmd-line-option:startup-procedure=asyncreq4.p ... <list of config overrides> -benv somevar=foo ... <list of environment variables> -b <processname>

I'll look into changing scripts for existing apps to use ServerDriver -b to launch a batch process in server-side mode (the bootstrap config and environment needs to be the same as the batch process case with its own JVM).

For server-side=all, I need to change SESSION_FWD-LOGFILE_Attribute to return the server's log file, as in this case there is no FWD-specific client log file.

Constantin Asofiei wrote:

For server-side=all, I need to change SESSION_FWD-LOGFILE_Attribute to return the server's log file, as in this case there is no FWD-specific client log file.

Fixed in rev 16730.

Hynek - please take a look at the changes.

The last part I have on my list is to see how the logs which were on FWD client-side can still be on a separate file.

Constantin Asofiei wrote:

The last part I have on my list is to see how the logs which were on FWD client-side can still be on a separate file.

This part we can not do easily. Loggers are created via CentralLogger.get, for both FWD client and server code. More, there are classes which are used both on server and client (BDTs, UnimplementedFeature and more). So we can not separate even via lets say CentralLoggerClient.get, so we can control which kind of logging is used. Analyzing the stack is expensive (i.e. to determine if we are executing server or client code).

OTOH, all client code (which executes now server-side) will execute via local proxies (which are via LocalStaticRedirector and LocalRedirector handlers). So we can intercept this and switch the CentralLogger from server to client (or vice-versa) before the call, and restore it after the call. I'll look into how this can be done.

• LocalStaticRedirector and LocalRedirector do not apply the changes from the response - the invoke should have been like this:
  

     public Object invokeCore(Object proxy, Method method, Object[] args)
     throws Throwable
     {
        if (!methods.contains(method))
        {
           throw new SilentUnwindException();
        }
  
        if (state != null)
        {
           state.sync();
        }
  
        try
        {
           return method.invoke(instance, args);
        }
        finally
        {
           if (state != null)
           {
              state.reverseSync();
           }
        }
     }
  

  
  where reverseSync just gets the changes from the sink and applies them to the source. I recall checking this early on this work, I saw LT.getChanges() but I missed that LT.applyChanges is not being called after the response.
• for MSA agents (and classic agents), I'd like to use a single log file (rotated). Why: we can trim and start agents for 1000s of times per day maybe. We will just spam the OS filesystem with files (many of them I think just empty files).

for MSA agents (and classic agents), I'd like to use a single log file (rotated). Why: we can trim and start agents for 1000s of times per day maybe. We will just spam the OS filesystem with files (many of them I think just empty files)

Isn't this how OE does it? (that is what I recall) If so, then the customers will accept it.

I will note that I also recall hearing a customer say they liked the separate logs. Perhaps we make this configurable.

Hynek Cihlar wrote:

Code review 11371a. I'm assuming interactive client drivers are out of scope for this work.

1. ThinClient.getJavaCPInternal null guard is inverted

I'll fix this.

2. SET-WORKING-DIRECTORY performs a process-wide chdir when the filesystem is server-side

This is a FWD extension. I'll disallow this for server-side filesystem.

3. The Admin process launch applies no allowlist to caller-supplied bootstrap config

• do we allow just the entry point in the REST payload and everything else is from the directory?
• do we add ACLs to specify allowed configs sent via REST?

4. %pid no longer distinguishes legacy log files, and the collision is swallowed

I'll check this.

5. Should the interactive GUI drivers be explicitly checked and denied for all-server client?

I can add the check.

6. ServerStreamDaemon context-local has no cleanup(), open streams leak

I don't understand this one; streams are closed by the scopeable support - when that is reached, is assumed there are no streams active.

7. -bcfg/-benv argument accounting uses map size instead of tokens consumed

I'll fix this.

8. Dead osResources collection makes a config branch permanently false

Fixed.

9. Non-defensive cleanup in the headless launch finally

Yes, I'll protect each part in a try/catch block.

10. The following exception catches in ProcessRestHandler.java only log the message:

This is the 'standard' for all Admin handlers - there is no error stacktrace logged. If we fix it, we need to fix it in all.

Greg Shah wrote:

for MSA agents (and classic agents), I'd like to use a single log file (rotated). Why: we can trim and start agents for 1000s of times per day maybe. We will just spam the OS filesystem with files (many of them I think just empty files)

Isn't this how OE does it? (that is what I recall) If so, then the customers will accept it.

OE has a single log file (for the agent) with the OE-specific messages. I'll make the FWD log file (not the legacy file) configurable via directory.

I will note that I also recall hearing a customer say they liked the separate logs. Perhaps we make this configurable.

Most likely those were the legacy logs. I'll double-check these if they can be per-agent or per-appserver (including for MSA).

Constantin Asofiei wrote:

6. ServerStreamDaemon context-local has no cleanup(), open streams leak

I don't understand this one; streams are closed by the scopeable support - when that is reached, is assumed there are no streams active.

Unless I'm reading the code wrong, streams not closed in 4GL stay in StreamDaemon.WorkArea.streams and are dropped without closing on client/session termination. This is a pre-existing issue.

Hynek - please review the changes in 11371a rev 16731.

4. %pid no longer distinguishes legacy log files, and the collision is swallowed

LegacyLogOps.appserverNameLogConfigPairs holds a single configuration for each appserver - thus, the first 'pid', date or other non-static config is captured, it will be the log filename to be used. Not a current problem.

6. ServerStreamDaemon context-local has no cleanup(), open streams leak

ServerStreamDaemon - the streams are closed via their finalizable support (UnnamedStream and StreamWrapper) - there is nothing left behind.

Otherwise, 1, 2, 5, 7, 8, 9 are fixed.

• I've redirected to NullStream the output in case of server-side-resources=all, to not emit on server-side client code. But this is only for RedirectedTerminal (and frames). There are more cases of System.out usage, as I still see in ETF messages in stdout.log.
• fixed state sync for local redirectors (which opened other problems and were fixed as found).

I have not rebased yet. I've run ETF and is OK, ChUI is running. I'll do more tests 'as is' with other apps, and after that I'll rebase and re-test fully.

• the LOG field is a JVM-wide, static, final, instance, in each class where it is used. There are classes like Keyboard, BaseDataType and more which are used on both client and server. Thus, who wins? I don't see how we can switch between the client and server loggers as needed (maybe a proxy will help, but this will not solve the part bellow).
• the separation between client and server logs is done via static JVM-wide fiels in CentralLoggerClient and CentralLogger; logging can be done from multiple threads, thus ThreadLocal can't work - and ContextLocal can't be used either (as CentraLogger is used by ContextLocal!). We need another approach here. I don't know what that would be.
• so at this time I've paused this work.

11371a was rebased from trunk rev 16737 - new rev 16749.

Constantin, regarding the client/server split of the FWD logs when the client runs in the server JVM. I don't think we need to swap the loggers at all. The LOG fields can stay CentralLoggerServer instances, the decision where a record goes should be made at the single write point, CentralLoggerServer.doPublish. What is needed is a per-thread side marker (a plain ThreadLocal, no ContextLocal involved) set at the local proxy boundary. All client code in this mode is entered through LocalRedirector.invokeCore (and LocalStaticRedirector) via the ClientExports proxy, and all upcalls back to the server (e.g. ThinClient firing 4GL triggers via server.trigger() on the ServerExports proxy) go through the same redirectors. So invokeCore saves the current marker, sets CLIENT or SERVER based on the left flag RemoteObject.obtainLocalInstance already receives, and restores the saved value in finally. Because each invokeCore restores the previous value on exit, the marker is always correct for the code currently executing, no matter how deep the client/server nesting goes. A shared class like BaseDataType logs to the client file when called from client code and to the server file when called from a trigger body, which is exactly what happens today with two JVMs.

The rest already exists. describeContext puts the session id on every record, CentralLoggerServer already keeps per-client file handlers (setupClientHandler, publishClientLog, removeHandler) and the client logger levels come from the same getLoggerLevelMap(). So doPublish only checks the marker (plus a per-logger flag for the ui.client and ui.chui packages, which never execute as server code), looks up the handler by session id and falls back to the server log when anything is unknown, i.e. the current behavior. CentralLogger.initializeClient needs to handle the SERVER mode when isAllServerSide() and call setupClientHandler with a session based key, the launchHeadlessClient finally block removes the handler. Btw. the same marker gives us the single file variant Greg mentioned, just emit Side:client in the record context and skip the extra handlers.

Hynek, this can be done in a separate task - is not related just to server-side clients, but also to MSA. In the mean time, please review last commit.

I like the approach, let's move ahead with it. It is OK in a separate task.

11371a rev 16750 and 16751 have other fixes related to local redirector state sync and MSA session cleanup.

• the opened stream is a file resource
• both the FWD server and the FWD client's osUserName are identical
• both the FWD server and the FWD client are running on the same machine - this is the part I don't know how to check yet. The previous rules are sound IMO. I'm thinking of writing a UUID .pid file to the temp folder for the current context, and if both client and server can read the same file, we mark it as 'on the same machine'.

• FWD ChUI:
  • PUT - 4768ms
  • COPY-LOB - 112ms
• FWD batch client, server-side-resources=all
  • PUT - 124ms
  • COPY-LOB - 65ms
• FWD ChUI - COPY-LOB on server-side
  • PUT - 4602ms
  • COPY-LOB read on server - 75ms
  • COPY-LOB r/w on server - 39ms
• FWD batch client server-side-resources=all - COPY-LOB on server-side
  • PUT - 118ms
  • COPY-LOB read on server - 38ms
  • COPY-LOB r/w on server - 29ms

• COPY-LOB (TargetLobFile, SourceLobFile)
• WRITE/READ-JSON/XML (in p2j.persist, TargetData and SourceData)
• p2j.reporting package (includes Jasper and XPR support)
• reporting.poi - WorkbookWrapper class
• p2j.xml package - sax reader/writer and X-DOCUMENT support

I have not changed all these features to use server-side java.io streams - it was just a POC for COPY-LOB and see how extensive the changes must be.

What are the numbers? Time in ms? Something else?

I have not changed all these features to use server-side java.io streams

Actually, I'm surprised these things didn't already use server-side. That was the entire point of implementing it in the first place (i.e. all 4GL features using file system reads should be on the server when server-side file system support is enabled).

If some substantial portion of our usage is not redirected properly, we do need to fix that ASAP.

Greg Shah wrote:

What are the numbers? Time in ms? Something else?

They are milliseconds, I've edited the note.

Actually, I'm surprised these things didn't already use server-side. That was the entire point of implementing it in the first place (i.e. all 4GL features using file system reads should be on the server when server-side file system support is enabled).

Inherently the files need to live on the FWD client, and this is why FWD legacy streams were used to manage the files, even from FWD runtime. When I added the support for server-side filesystem/streams, this usage was not changed, as it allowed to access them via the FWD server (just that it still used the legacy streams).

If some substantial portion of our usage is not redirected properly, we do need to fix that ASAP.

I'm not sure what you mean here. The point of the note above is to allow the IO to be done from the FWD server via java.io streams, even for interactive clients, when the client and server run on the same machine and run with the same OS user. Any other case can't be supported - the file will need to be accessed via the FWD legacy stream code.

What I mean is that enabling server-side file system access should cause there to be no round trip to the client for any usage of file system writing. This should be the case for direct 4GL stream usage but also for any internal use of the client to read/write to the file system. These "internal" usages are exactly what you described above (COPY-LOB, XML, JSON...).

I was not making a distinction between using our legacy stream classes and direct Java stream i/o. My view was that there was a major savings in just eliminating the network round trips for each read/write. I was expecting that savings to already have been implemented in current trunk. If that is not the case, I'd like to understand it.

Greg Shah wrote:

What I mean is that enabling server-side file system access should cause there to be no round trip to the client for any usage of file system writing. This should be the case for direct 4GL stream usage but also for any internal use of the client to read/write to the file system. These "internal" usages are exactly what you described above (COPY-LOB, XML, JSON...).

Yes, with server-side file-system (what trunk has) and server-side-resources=all, the streams are accessed from the FWD Server JVM, there is no network round-trip. But there is still some overhead because of the legacy stream usage.

I was not making a distinction between using our legacy stream classes and direct Java stream i/o. My view was that there was a major savings in just eliminating the network round trips for each read/write. I was expecting that savings to already have been implemented in current trunk. If that is not the case, I'd like to understand it.

Current trunk has support for server-side-resources=filesystem,streams. But the runtime internal case still relies on legacy streams (which can be on server or client) - this is what I want to improve via the Java streams from FWD server, to have this implicitly for the runtime part.

But, another point to make: the explicit solution for server-side-resources=filesystem,streams is no longer supported in 11371a, you need to have server-side-resources=all, thus interactive clients can't use this. The benefit for these changes is only in specific cases (only PUT and named streams), as anything else has the terminal involved and the streams must live on the client. I can add back this solution to 11371a, but IMO the gain is minimal.

The benefit for these changes is only in specific cases (only PUT and named streams), as anything else has the terminal involved and the streams must live on the client. I can add back this solution to 11371a, but IMO the gain is minimal.

I'd still prefer to keep the support there. It gives us a tool for rewriting 4GL code using only PUT/named streams and getting a performance boost.

• Don't resolve from jar when the filename is blank.

Greg Shah wrote:

The benefit for these changes is only in specific cases (only PUT and named streams), as anything else has the terminal involved and the streams must live on the client. I can add back this solution to 11371a, but IMO the gain is minimal.

I'd still prefer to keep the support there. It gives us a tool for rewriting 4GL code using only PUT/named streams and getting a performance boost.

Understood; I'll add it back and check the other scenarios mentioned by Alex in #11779-50

The experiments with Java I/O were mostly to keep some gain for interactive clients, but I'll keep it, to remove the overhead from legacy streams.