FWD » Core Development » Runtime Infrastructure

Batch processes can be made to work similar to PASOE agents, full server-side, assuming that there is no need of interactive terminal.

There are not many static fields in the p2j.ui package remaining. After we clean this up and make them context-local, what we need to do is run a FWD client only on server-side.

Initial proposal is to do this still via ClientDriver, which after authentication, if it sees server-side-resources=ALL, it switches automatically to this 'headless' mode (i.e. closes the socket, etc) and initializes ThinClient and other needed UI resources on server-side.

But, this does not get rid of the OS process launch, which can be problematic when many clients need to be launch at the same time.

Instead, the solution will be to provide FWD Java APIs to launch a batch process fully server-side, in 'headless' mode, directly from the applications code. This way, there will no longer be an OS process.

An additional complication is OS-GETENV. There are applications which pass state to the launched batch process via OS environment variables. This can't be made to work without adding a separate mechanism of passing these env vars to the FWD Java API launching the OS batch process, so that we still have context-local separation of OS-GETENV (otherwise, all clients would read FWD Server's env vars).

Constantin Asofiei wrote:

Instead, the solution will be to provide FWD Java APIs to launch a batch process fully server-side, in 'headless' mode, directly from the applications code. This way, there will no longer be an OS process.

Greg, an additional note: do we want to have REST APIs instead of Java APIs to launch the batch processes?

I would generally prefer that. But I want to make sure that we plan for the most likely usage scenario. How would it be used to replace the launching of _progres or pro sessions in a script?

Greg Shah wrote:

I would generally prefer that. But I want to make sure that we plan for the most likely usage scenario. How would it be used to replace the launching of _progres or pro sessions in a script?

• the REST API will authenticate with a FWD account which has the correct permissions to launch batch processes
• the payload will specify the FWD account under which to run the batch program, plus all the other configurations to initialize the FWD context (bootstrap config, OS env vars, workingDir, etc)

Also, this would make FWD easier to integrate with other apps. And also does not rely on a JVM process to connect to the FWD server and launch this.

OK, go ahead with the REST API.

The initial work on running client-side UI in server context was implemented in 9055a. It added very limited set of use cases, like the MESSAGE statement.

• mixed mode PUT and DISPLAY:
  • trunk server-side filesystem,streams: 11525ms/9869ms
  • trunk no-server-side: 11135ms/9936ms
  • 11371 server-side all: 2677ms/1823ms
• only PUT:
  • trunk server-side filesystem,streams: 121ms/4085ms
  • trunk no-server-side: 3919ms/3446ms
  • 11371 server-side all: 58ms/68ms
• only DISPLAY
  • 11371 server-side all: 2461ms/1717ms

I'll cleanup the changes, run automated review and start more extensive testing in real applications (I want to see if I can modify the batch launch code to actually be server-side=all for them). I also want to do some profiling to see where time is spent for the mixed PUT/DISPLAY case, the difference compared only with PUT is too high.

• the refactoring of the FileStream to keep an in-memory buffer was not yet made. I'll work on this in a separate branch (I've been thinking about this, but there are caveats to know when to flush to binary buffer, what happens when cursor is moved, etc).
• the configuration for server-side-resources remains in the directory - it can not be configured via bootstrap configurations. There are lots of 'chicken and the egg' issues and to add it complicates things a lot.
• REST APIs added:
  • Greg - do we want an ACL to allow a batch account to be used via REST?
  • GET /admin/launch - gets the list of all batch processes which can be launched; it skips only batch accounts configured as server or appserver.
    

    {
        "processes": {
            "serversidebatch": true,
            "appserver_agent": true
        }
    }
    

  • POST /admin/launch/<batchprocess> - launches a batch process, where the payload can specify custom OS properties and bootstrap configurations:
    

    {
        "properties" : 
        {
           "customproperty" : "blabla" 
        },
        "configs" : 
        {
           "client:cmd-line-option:startup-procedure" : "test1.p" 
        }
    }
    

• launching can also be done via ServerDriver -b - it will honor the server-side-resources=all from the directory

to allow a batch account to be used via REST

What do you mean by this? Used how or in what way?

Greg Shah wrote:

to allow a batch account to be used via REST

What do you mean by this? Used how or in what way?

I mean to be launched via REST. To allow explicitly this feauture for a process account.

Also I need to check what ACLs we have for the startup program - do we allow any program for REST launching?

I know the launching is secured via the token (so only an authenticated admin user can run this). So is more if we want another layer of checks.

About security: we already have EntryPointResource to secure the entry points. This should be enough.

Current changes are in 11371a rev 16687. I've also added server-side-resources=all support for launching classic appserver agents - in this mode, there will be no JVM for a FWD Client. This will help reducing the need for system memory usage if these clients are server-side, plus the FWD server startup time.

• there is an issue with WAIT-FOR (is not blocking). I'm looking into this.
• all STDOUT or other client-side output in this mode will go to the FWD Server's JVM STODUT or logging - this is the same as for MSA, I don't think we have another approach for this.
• currently filesystem and streams can work on server-side only and only if ALL are server-side. To keep this, we need back the #11271 support. But, considering the findings from this work (any terminal requirement like unnamed streams or UI statements must have the stream on the client), this is not something viable. I don't think we have any applications currently configured with server-side filesystem for interactive clients (not batch).
• more testing including MSA, classic agents and normal clients.

Greg - currently only REST allows specifying custom bootstrap config and OS environment overrides. I'm thinking to add this support also for ServerDriver -b, IMO it will complicate things if we want to launch these via shell scripts (i.e. maintaining the REST ADMIN user and password, ACLs, etc).

Constantin Asofiei wrote:

Greg - currently only REST allows specifying custom bootstrap config and OS environment overrides. I'm thinking to add this support also for ServerDriver -b, IMO it will complicate things if we want to launch these via shell scripts (i.e. maintaining the REST ADMIN user and password, ACLs, etc).

This support is in 11371a/16690; the syntax is:

ServerDriver -bcfg  client:cmd-line-option:startup-procedure=asyncreq4.p ... <list of config overrides> -benv somevar=foo ... <list of environment variables> -b <processname>

I'll look into changing scripts for existing apps to use ServerDriver -b to launch a batch process in server-side mode (the bootstrap config and environment needs to be the same as the batch process case with its own JVM).

For server-side=all, I need to change SESSION_FWD-LOGFILE_Attribute to return the server's log file, as in this case there is no FWD-specific client log file.

Constantin Asofiei wrote:

For server-side=all, I need to change SESSION_FWD-LOGFILE_Attribute to return the server's log file, as in this case there is no FWD-specific client log file.

Fixed in rev 16730.

Hynek - please take a look at the changes.

The last part I have on my list is to see how the logs which were on FWD client-side can still be on a separate file.

Constantin Asofiei wrote:

The last part I have on my list is to see how the logs which were on FWD client-side can still be on a separate file.

This part we can not do easily. Loggers are created via CentralLogger.get, for both FWD client and server code. More, there are classes which are used both on server and client (BDTs, UnimplementedFeature and more). So we can not separate even via lets say CentralLoggerClient.get, so we can control which kind of logging is used. Analyzing the stack is expensive (i.e. to determine if we are executing server or client code).

OTOH, all client code (which executes now server-side) will execute via local proxies (which are via LocalStaticRedirector and LocalRedirector handlers). So we can intercept this and switch the CentralLogger from server to client (or vice-versa) before the call, and restore it after the call. I'll look into how this can be done.

• LocalStaticRedirector and LocalRedirector do not apply the changes from the response - the invoke should have been like this:
  

     public Object invokeCore(Object proxy, Method method, Object[] args)
     throws Throwable
     {
        if (!methods.contains(method))
        {
           throw new SilentUnwindException();
        }
  
        if (state != null)
        {
           state.sync();
        }
  
        try
        {
           return method.invoke(instance, args);
        }
        finally
        {
           if (state != null)
           {
              state.reverseSync();
           }
        }
     }
  

  
  where reverseSync just gets the changes from the sink and applies them to the source. I recall checking this early on this work, I saw LT.getChanges() but I missed that LT.applyChanges is not being called after the response.
• for MSA agents (and classic agents), I'd like to use a single log file (rotated). Why: we can trim and start agents for 1000s of times per day maybe. We will just spam the OS filesystem with files (many of them I think just empty files).

for MSA agents (and classic agents), I'd like to use a single log file (rotated). Why: we can trim and start agents for 1000s of times per day maybe. We will just spam the OS filesystem with files (many of them I think just empty files)

Isn't this how OE does it? (that is what I recall) If so, then the customers will accept it.

I will note that I also recall hearing a customer say they liked the separate logs. Perhaps we make this configurable.

Hynek Cihlar wrote:

Code review 11371a. I'm assuming interactive client drivers are out of scope for this work.

1. ThinClient.getJavaCPInternal null guard is inverted

I'll fix this.

2. SET-WORKING-DIRECTORY performs a process-wide chdir when the filesystem is server-side

This is a FWD extension. I'll disallow this for server-side filesystem.

3. The Admin process launch applies no allowlist to caller-supplied bootstrap config

• do we allow just the entry point in the REST payload and everything else is from the directory?
• do we add ACLs to specify allowed configs sent via REST?

4. %pid no longer distinguishes legacy log files, and the collision is swallowed

I'll check this.

5. Should the interactive GUI drivers be explicitly checked and denied for all-server client?

I can add the check.

6. ServerStreamDaemon context-local has no cleanup(), open streams leak

I don't understand this one; streams are closed by the scopeable support - when that is reached, is assumed there are no streams active.

7. -bcfg/-benv argument accounting uses map size instead of tokens consumed

I'll fix this.

8. Dead osResources collection makes a config branch permanently false

Fixed.

9. Non-defensive cleanup in the headless launch finally

Yes, I'll protect each part in a try/catch block.

10. The following exception catches in ProcessRestHandler.java only log the message:

This is the 'standard' for all Admin handlers - there is no error stacktrace logged. If we fix it, we need to fix it in all.

Greg Shah wrote:

for MSA agents (and classic agents), I'd like to use a single log file (rotated). Why: we can trim and start agents for 1000s of times per day maybe. We will just spam the OS filesystem with files (many of them I think just empty files)

Isn't this how OE does it? (that is what I recall) If so, then the customers will accept it.

OE has a single log file (for the agent) with the OE-specific messages. I'll make the FWD log file (not the legacy file) configurable via directory.

I will note that I also recall hearing a customer say they liked the separate logs. Perhaps we make this configurable.

Most likely those were the legacy logs. I'll double-check these if they can be per-agent or per-appserver (including for MSA).

Constantin Asofiei wrote:

6. ServerStreamDaemon context-local has no cleanup(), open streams leak

I don't understand this one; streams are closed by the scopeable support - when that is reached, is assumed there are no streams active.

Unless I'm reading the code wrong, streams not closed in 4GL stay in StreamDaemon.WorkArea.streams and are dropped without closing on client/session termination. This is a pre-existing issue.