FWD » Core Development » Runtime Infrastructure

Batch processes can be made to work similar to PASOE agents, full server-side, assuming that there is no need of interactive terminal.

There are not many static fields in the p2j.ui package remaining. After we clean this up and make them context-local, what we need to do is run a FWD client only on server-side.

Initial proposal is to do this still via ClientDriver, which after authentication, if it sees server-side-resources=ALL, it switches automatically to this 'headless' mode (i.e. closes the socket, etc) and initializes ThinClient and other needed UI resources on server-side.

But, this does not get rid of the OS process launch, which can be problematic when many clients need to be launch at the same time.

Instead, the solution will be to provide FWD Java APIs to launch a batch process fully server-side, in 'headless' mode, directly from the applications code. This way, there will no longer be an OS process.

An additional complication is OS-GETENV. There are applications which pass state to the launched batch process via OS environment variables. This can't be made to work without adding a separate mechanism of passing these env vars to the FWD Java API launching the OS batch process, so that we still have context-local separation of OS-GETENV (otherwise, all clients would read FWD Server's env vars).

Constantin Asofiei wrote:

Instead, the solution will be to provide FWD Java APIs to launch a batch process fully server-side, in 'headless' mode, directly from the applications code. This way, there will no longer be an OS process.

Greg, an additional note: do we want to have REST APIs instead of Java APIs to launch the batch processes?

I would generally prefer that. But I want to make sure that we plan for the most likely usage scenario. How would it be used to replace the launching of _progres or pro sessions in a script?

Greg Shah wrote:

I would generally prefer that. But I want to make sure that we plan for the most likely usage scenario. How would it be used to replace the launching of _progres or pro sessions in a script?

• the REST API will authenticate with a FWD account which has the correct permissions to launch batch processes
• the payload will specify the FWD account under which to run the batch program, plus all the other configurations to initialize the FWD context (bootstrap config, OS env vars, workingDir, etc)

Also, this would make FWD easier to integrate with other apps. And also does not rely on a JVM process to connect to the FWD server and launch this.

OK, go ahead with the REST API.

The initial work on running client-side UI in server context was implemented in 9055a. It added very limited set of use cases, like the MESSAGE statement.

• mixed mode PUT and DISPLAY:
  • trunk server-side filesystem,streams: 11525ms/9869ms
  • trunk no-server-side: 11135ms/9936ms
  • 11371 server-side all: 2677ms/1823ms
• only PUT:
  • trunk server-side filesystem,streams: 121ms/4085ms
  • trunk no-server-side: 3919ms/3446ms
  • 11371 server-side all: 58ms/68ms
• only DISPLAY
  • 11371 server-side all: 2461ms/1717ms

I'll cleanup the changes, run automated review and start more extensive testing in real applications (I want to see if I can modify the batch launch code to actually be server-side=all for them). I also want to do some profiling to see where time is spent for the mixed PUT/DISPLAY case, the difference compared only with PUT is too high.

• the refactoring of the FileStream to keep an in-memory buffer was not yet made. I'll work on this in a separate branch (I've been thinking about this, but there are caveats to know when to flush to binary buffer, what happens when cursor is moved, etc).
• the configuration for server-side-resources remains in the directory - it can not be configured via bootstrap configurations. There are lots of 'chicken and the egg' issues and to add it complicates things a lot.
• REST APIs added:
  • Greg - do we want an ACL to allow a batch account to be used via REST?
  • GET /admin/launch - gets the list of all batch processes which can be launched; it skips only batch accounts configured as server or appserver.
    

    {
        "processes": {
            "serversidebatch": true,
            "appserver_agent": true
        }
    }
    

  • POST /admin/launch/<batchprocess> - launches a batch process, where the payload can specify custom OS properties and bootstrap configurations:
    

    {
        "properties" : 
        {
           "customproperty" : "blabla" 
        },
        "configs" : 
        {
           "client:cmd-line-option:startup-procedure" : "test1.p" 
        }
    }
    

• launching can also be done via ServerDriver -b - it will honor the server-side-resources=all from the directory

to allow a batch account to be used via REST

What do you mean by this? Used how or in what way?

Greg Shah wrote:

to allow a batch account to be used via REST

What do you mean by this? Used how or in what way?

I mean to be launched via REST. To allow explicitly this feauture for a process account.

Also I need to check what ACLs we have for the startup program - do we allow any program for REST launching?

I know the launching is secured via the token (so only an authenticated admin user can run this). So is more if we want another layer of checks.

About security: we already have EntryPointResource to secure the entry points. This should be enough.

Current changes are in 11371a rev 16687. I've also added server-side-resources=all support for launching classic appserver agents - in this mode, there will be no JVM for a FWD Client. This will help reducing the need for system memory usage if these clients are server-side, plus the FWD server startup time.

• there is an issue with WAIT-FOR (is not blocking). I'm looking into this.
• all STDOUT or other client-side output in this mode will go to the FWD Server's JVM STODUT or logging - this is the same as for MSA, I don't think we have another approach for this.
• currently filesystem and streams can work on server-side only and only if ALL are server-side. To keep this, we need back the #11271 support. But, considering the findings from this work (any terminal requirement like unnamed streams or UI statements must have the stream on the client), this is not something viable. I don't think we have any applications currently configured with server-side filesystem for interactive clients (not batch).
• more testing including MSA, classic agents and normal clients.

Greg - currently only REST allows specifying custom bootstrap config and OS environment overrides. I'm thinking to add this support also for ServerDriver -b, IMO it will complicate things if we want to launch these via shell scripts (i.e. maintaining the REST ADMIN user and password, ACLs, etc).

Constantin Asofiei wrote:

Greg - currently only REST allows specifying custom bootstrap config and OS environment overrides. I'm thinking to add this support also for ServerDriver -b, IMO it will complicate things if we want to launch these via shell scripts (i.e. maintaining the REST ADMIN user and password, ACLs, etc).

This support is in 11371a/16690; the syntax is:

ServerDriver -bcfg  client:cmd-line-option:startup-procedure=asyncreq4.p ... <list of config overrides> -benv somevar=foo ... <list of environment variables> -b <processname>

I'll look into changing scripts for existing apps to use ServerDriver -b to launch a batch process in server-side mode (the bootstrap config and environment needs to be the same as the batch process case with its own JVM).