FWD » Core Development » Runtime Infrastructure

A customer that is heavily invested in classic appserver has to use the "trim" processing to stop appserver agents when they are consuming too many resources. They would prefer a "bounce" capability (stop and then restart) but OE doesn't provide it. This task is meant to expose some back capabilities in this regard. We must expose the rest of the appserver configuration as well as a view of the currently running agents.

See #6878 for an example of the kind of status reporting that is available in OE. We must provide this level or better.

I would prefer to implement this using a REST API. We will provide a CLI (command line interface) and would probably also implement a UI in the admin console. Having a REST API as the backing implementation gives customers the ultimate flexibility. They can create their own tooling if the built-in FWD tools don't meet their needs.

A first pass has been implemented for #7074 using the ServerDriver "status" approach. We will need to shift this to REST when possible.

A question: can there be several processes configured for one appserver? Is this a way to simply have different FWD configurations for the same definition?

Galya B wrote:

A question: can there be several processes configured for one appserver? Is this a way to simply have different FWD configurations for the same definition?

No, a single process account must be defined, this is enforced in AppServerLauncher.launch.

SecurityCache.getAccountsForAppserver returns an array, but I guess I should just ignore it and consider the first result.

Galya B wrote:

SecurityCache.getAccountsForAppserver returns an array, but I guess I should just ignore it and consider the first result.

You should handle both cases, when there are none setup and 2 or more, and show errors/warnings. I don't know how appserver configuration is enforced at the batch process config via AdminConsole.

For now I'm implementing only displaying the info read-only. If we add edit functionality later on, an error can be shown. When no process is defined, FWD process name will simply be empty. The trickier part is that the configs will reflect only the 'first' of the processes if multiple.

Galya B wrote:

For now I'm implementing only displaying the info read-only. If we add edit functionality later on, an error can be shown. When no process is defined, FWD process name will simply be empty. The trickier part is that the configs will reflect only the 'first' of the processes if multiple.

Understood, that's fine - just show a warning that there are multiple processes defined.

Constantin, I had to remove security:certificate:validate as you asked me in #7808-115, because it took me quite a while now to find out that the : symbols in the node id messes up directory reads. Is it that the prop is not valid anywhere in the code? All connections need to have a valid certificate?

Galya B wrote:

Constantin, I had to remove security:certificate:validate as you asked me in #7808-115, because it took me quite a while now to find out that the : symbols in the node id messes up directory reads.

OK.

Is it that the prop is not valid anywhere in the code? All connections need to have a valid certificate?

This prop is (was) used only when launching appserver agents or batch processes via the scheduler (i.e. they are 'spawned' and not launched explicitly). Until now, the default was 'false'; I still think we need to force it to 'true', but we need some tests of some apps to see if the appserver agents (or batch processes) still start, with that app configuration. Just make the change for now and lets not forget to do this test.

When the work is ready, Sergey and Constantin will review it.

Also, security:certificate:validate has been either removed or set to true in critical places and now I get an error, when testing with customer project:

24/10/10 11:58:43.513+0300 | WARNING | com.goldencode.p2j.net.RouterSessionManager | ThreadName:Incoming SSL Connector#15 | Incoming SSL Connector#15:Handshake timed out.
24/10/10 11:58:43.515+0300 | WARNING | com.goldencode.p2j.net.SSL | ThreadName:Incoming SSL Connector#15:nioSslTaskWorker#29 | InterruptedException
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1245)
    at java.util.concurrent.locks.ReentrantLock.tryLock(ReentrantLock.java:442)
    at com.goldencode.p2j.net.SSL.tryLock(SSL.java:393)
    at com.goldencode.p2j.net.SSL.wrap(SSL.java:537)
    at com.goldencode.p2j.net.SSL.handshake(SSL.java:503)
    at com.goldencode.p2j.net.SSL.run(SSL.java:316)
    at com.goldencode.p2j.net.SSL.notify(SSL.java:307)
    at com.goldencode.p2j.net.BlockingSSL.processInput(BlockingSSL.java:236)
    at com.goldencode.p2j.net.BlockingSSL.checkInput(BlockingSSL.java:115)
    at com.goldencode.p2j.net.NIOSslSocket.lambda$new$1(NIOSslSocket.java:175)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
24/10/10 11:58:43.515+0300 | WARNING | com.goldencode.p2j.net.NIOSslSocket | ThreadName:Incoming SSL Connector#15:nioSslTaskWorker#29 | Exception on SSL handshake 
java.lang.RuntimeException: Faild to acquire lock
    at com.goldencode.p2j.net.SSL.tryLock(SSL.java:402)
    at com.goldencode.p2j.net.SSL.wrap(SSL.java:537)
    at com.goldencode.p2j.net.SSL.handshake(SSL.java:503)
    at com.goldencode.p2j.net.SSL.run(SSL.java:316)
    at com.goldencode.p2j.net.SSL.notify(SSL.java:307)
    at com.goldencode.p2j.net.BlockingSSL.processInput(BlockingSSL.java:236)
    at com.goldencode.p2j.net.BlockingSSL.checkInput(BlockingSSL.java:115)
    at com.goldencode.p2j.net.NIOSslSocket.lambda$new$1(NIOSslSocket.java:175)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)

The above exception doesn't seem to prevent the scheduled processes from starting. I've encountered another error in web login - the customer login in the session shows an error with code 2512 not found, but I don't see an error in the logs, so maybe the login credentials have changed.

Is this task branch 8661a?

Is this task branch 8661a?

Yes

r15504 fixes persistent procedures not showing in the table. I broke it with the cleanup.

Here are the procs I'm testing the 'Session Procedure' column with:

appserver.p:

def var h as handle.

create server h.
h:connect("-S 21200 -H localhost -sessionModel Session-Free -DirectConnect").

run longrun.p on server h.

h:disconnect().
delete object h.

appserver-persistent.p:

def var h as handle.

create server h.
h:connect("-S 21200 -H localhost -sessionModel Session-Free -DirectConnect").

run longrun.p persistent on h asynchronous.

h:disconnect().
delete object h.

longrun.p:

message "started" view-as alert-box.
pause 30 no-message.
message "ended" view-as alert-box.

The second program appserver-persistent.p that runs longrun.p asynchronously has h:disconnect(). The 4GL docs said that pending asynchronous requests are cancelled and the corresponding event procedure is called for each request. Correct?

Sergey Ivanovskiy wrote:

The second program appserver-persistent.p that runs longrun.p asynchronously has h:disconnect(). The 4GL docs said that pending asynchronous requests are cancelled and the corresponding event procedure is called for each request. Correct?

Someone else can say how it's supposed to work. I'll tell you what it does for my test - it doesn't get cancelled and runs for a long time, so the proc doesn't seem to be executed async, probably because it's persistent (I may have created an incompatible mix).

Galya B wrote:

Constantin, I had to remove security:certificate:validate as you asked me in #7808-115, because it took me quite a while now to find out that the : symbols in the node id messes up directory reads. Is it that the prop is not valid anywhere in the code? All connections need to have a valid certificate?

Galya, this was not meant to remove the entire security:certificate:validate support from bootstrap config - just that directory-specific setting. Did you remove this security:certificate:validate=false completely, no longer supported via bootstrap option?

Constantin Asofiei wrote:

Galya B wrote:

Constantin, I had to remove security:certificate:validate as you asked me in #7808-115, because it took me quite a while now to find out that the : symbols in the node id messes up directory reads. Is it that the prop is not valid anywhere in the code? All connections need to have a valid certificate?

Galya, this was not meant to remove the entire security:certificate:validate support from bootstrap config - just that directory-specific setting. Did you remove this security:certificate:validate=false completely, no longer supported via bootstrap option?

Good time to know the answer. I haven't removed it from all places, where it's used, but let me revise. It should be deleted from the directory reads and process builder / spawn, and the ClientConfig wasn't used in another place, but I'll confirm if I've changed the bootstrap reads.

Galya B wrote:

Constantin Asofiei wrote:

Galya B wrote:

Constantin, I had to remove security:certificate:validate as you asked me in #7808-115, because it took me quite a while now to find out that the : symbols in the node id messes up directory reads. Is it that the prop is not valid anywhere in the code? All connections need to have a valid certificate?

Galya, this was not meant to remove the entire security:certificate:validate support from bootstrap config - just that directory-specific setting. Did you remove this security:certificate:validate=false completely, no longer supported via bootstrap option?

Good time to know the answer. I haven't removed it from all places, where it's used, but let me revise. It should be deleted from the directory reads and process builder / spawn, and the ClientConfig wasn't used in another place, but I'll confirm if I've changed the bootstrap reads.

Thanks, please double-check; what I'm interested in: a manually-started batch process should be able to specify security:certificate:validate=false at its bootstrap config, and work properly. For whatever is spawned via the FWD server, that security:certificate:validate=true should be enforced.

I've reverted the relevant changes in ClientCore and SecurityManager in r15506. Otherwise it's ok to remove the config, since it hasn't been used for the bootstrap value.

The changes rev 15506 (8661a) related to my view are good except a minor issue found in ClassicAppserverView, where the constants for columns widths are defined. Does it make sense to define these named constants for each column? There is column names array AGENT_COL_NAMES but their widths are defined in the code.

+      GridHelper.ColumnInfo<AppserverInfo.AgentInfo> col8 =
+         new GridHelper.ColumnInfo<>(AGENT_COL_NAMES[7],
+                                     agentInfo -> agentInfo.clientType == null ? "" : agentInfo.clientType);
+
+      col1.setWidthInPercents(10);
+      col2.setWidthInPercents(10);
+      col3.setWidthInPercents(16);
+      col4.setWidthInPercents(12);
+      col5.setWidthInPercents(12);
+      col6.setWidthInPercents(16);
+      col7.setWidthInPercents(12);
+      col8.setWidthInPercents(12);

Sergey Ivanovskiy wrote:

The changes rev 15506 (8661a) related to my view are good except a minor issue found in ClassicAppserverView, where the constants for columns widths are defined. Does it make sense to define these named constants for each column? There is column names array AGENT_COL_NAMES but their widths are defined in the code.
[...]

Up in r15507.

Constantin, how is the review going?

• please remove the src/com/goldencode/p2j/util/appserver/ empty package. Even if we want to move all appserver-related classes there, we can't move them all, as there are dependencies on customer code for these classes to be where they are now, and some of this code we can't refactor (don't have source access). For now, we need to leave p2j.util as is.
• ClientCore - no functional change, please revert the file.
• OERequestInfo
  • buildFromArray(Object[] arr) needs javadoc
  • using new OerequestInfo(); I don't think is OK. This will bypass reference counting. The OERequestInfo is a context-local singleton which is read via SessionUtils.currentRequestInfo(). Also, the single usage is from Agent$AgentCommand.oeRequestInfo, which is never read. So, can we remove this field and back out the buildFromArray changes?
• Agent.java
  • setExternalProcedure is being called with the target's name, which can be either an external procedure or an internal procedure/function. So, what do you want to track? If is the actual program being executed, then this needs to resolve the program name for the cases where it can invoke an internal procedure/function in a handle.
    • for the case of JavaInvokeConfig on line 1485, resolve the class name, too. cfg can never be null here.
    • on line 1405, resolve the resId external program (if specified)
    • on line 1215, for the InvokeConfig case, resolve the external program, if specified via InvokeConfig.inHandle, and not just the target. cfg can never be null here.
    • on line 936, resolve the class name from resId
• AgentPool.getAll() - we shouldn't expose the reference to the set, please return a copy.

Constantin Asofiei wrote:

• please remove the src/com/goldencode/p2j/util/appserver/ empty package. Even if we want to move all appserver-related classes there, we can't move them all, as there are dependencies on customer code for these classes to be where they are now, and some of this code we can't refactor (don't have source access). For now, we need to leave p2j.util as is.

I want to move the PASOE classes I'm adding in 8661b to the new package. Maybe for that purpose a separate merge is not needed, but maybe other classes like AppServerDefinition are also going there. I won't move other core appserver classes if this can break customer dependencies (hopefully soon we'll have proper java 9+ support to define the exposed interfaces).

• ClientCore - no functional change, please revert the file.

OK.

• OERequestInfo
  • buildFromArray(Object[] arr) needs javadoc

Added.

• using new OerequestInfo(); I don't think is OK. This will bypass reference counting. The OERequestInfo is a context-local singleton which is read via SessionUtils.currentRequestInfo(). Also, the single usage is from Agent$AgentCommand.oeRequestInfo, which is never read. So, can we remove this field and back out the buildFromArray changes?

Yes, I removed the code using the OerequestInfo object, because the array seemed to be empty in my tests. I wonder if this is always the case? I struggle to follow the lifecycle of OerequestInfo. My reasoning of leaving the transformation is that OerequestInfo at one point is supposed to be the central piece containing all the important data for the request (which it still is not) and it should be accessed from other parts of the code, not only the 4GL code. The reference you're referring to, is probably related to how it's used in 4GL. Am I correct? I actually didn't remove the original SessionUtils.currentRequestInfo().ref().fromArray(requestInfo); that should be tracking the reference (whatever that is), just added additional transformation to be able to use the data before the 4GL execution. I usually remove unused code, but this I got attached to, imagining I'll use it some day.

• Agent.java
  • setExternalProcedure is being called with the target's name, which can be either an external procedure or an internal procedure/function. So, what do you want to track? If is the actual program being executed, then this needs to resolve the program name for the cases where it can invoke an internal procedure/function in a handle.
    • for the case of JavaInvokeConfig on line 1485, resolve the class name, too. cfg can never be null here.
    • on line 1405, resolve the resId external program (if specified)
    • on line 1215, for the InvokeConfig case, resolve the external program, if specified via InvokeConfig.inHandle, and not just the target. cfg can never be null here.
    • on line 936, resolve the class name from resId

If by 'external procedure or an internal procedure/function' you mean nested calls, then it should be changed to the 'actual program'. All the invoke methods are super confusing, because I don't know how to test all the calls, but let me try and fix it.
For line 936 isn't character name The legacy 4GL name for the class method.? Please see what I've pushed, it doesn't look like the solution.

• AgentPool.getAll() - we shouldn't expose the reference to the set, please return a copy.

Roger that.

Changes up in r15509.

• AgentPool - Set.copyOf is not available in Java 8 - please replace this code.
• in Agent.java
  • invokeMethod around line 936 - use both the class name and the method name
  • invoke line 1411 - resId is null when invoking an external program, and not-null otherwise; in this case, when is not null, use both ext prog name and the target name.

Yes, I removed the code using the OerequestInfo object, because the array seemed to be empty in my tests. I wonder if this is always the case? I struggle to follow the lifecycle of OerequestInfo. My reasoning of leaving the transformation is that OerequestInfo at one point is supposed to be the central piece containing all the important data for the request (which it still is not) and it should be accessed from other parts of the code, not only the 4GL code.

The right way to access it is from SessionUtils.currentRequestInfo. Please remove AgentCommand.oeRequestInfo. If you mean you need OerequestInfo state read outside of the agent's context, lets discuss that, but current approach is not OK.

Constantin Asofiei wrote:

• invokeMethod around line 936 - use both the class name and the method name

How to get the class name from resId / handle (handle proc = handle.fromString(resId);) ?

• invoke line 1411 - resId is null when invoking an external program, and not-null otherwise; in this case, when is not null, use both ext prog name and the target name.

What target? Name?

   public AppServerInvocationResult invoke(final String    id,
                                           final int       requestId,
                                           final long      timeout,
                                           final character name,
                                           final String    resId,
                                           final boolean   function,
                                           final boolean   dynamicFunction,
                                           final boolean   superCall,
                                           final boolean   transactionDistinct,
                                           final String    modes,
                                           final Object... args)

Galya B wrote:

Constantin Asofiei wrote:

• invokeMethod around line 936 - use both the class name and the method name

How to get the class name from resId / handle (handle proc = handle.fromString(resId);) ?

Yes, extract the name from the handle - proc.get().getClass().getName().

• invoke line 1411 - resId is null when invoking an external program, and not-null otherwise; in this case, when is not null, use both ext prog name and the target name.

What target? Name?

Yes, name is the target - which can be external program when resId is null and internal procedure/function, otherwise.

Please check r15510.

Only one issue, please; in this code at line 1412:

         {
            String runningProgram = null;
            ExternalProgramWrapper proc = getProcedure(resId);
            if (proc != null)
            {
               runningProgram = proc.name().toStringMessage() + " # " + name.toStringMessage();
            }
            setRunningProgram(runningProgram);
         }

            String runningProgram = null;
            if (resId == null)
            {
               // external program call
               runningProgram = name.toStringMessage();
            }
            else
            {
...
}

Constantin Asofiei wrote:

Only one issue, please; in this code at line 1412:
[...]
should be like this:
[...]

In the else having runningProgram = proc.name().toStringMessage() + " # " + name.toStringMessage(); or only proc.name()?

Galya B wrote:

Constantin Asofiei wrote:

Only one issue, please; in this code at line 1412:
[...]
should be like this:
[...]

In the else having runningProgram = proc.name().toStringMessage() + " # " + name.toStringMessage(); ..

Both external program and target 'name'.

r15511.

The branch has been tested with a customer project and testcases.

How about I merge now?

After which branch?

Galya B wrote:

After which branch?

Sorry, I meant first, you can merge.

Galya B wrote:

8661a was merged to trunk as rev. 15504 and archived.

Is this needed in #7156?

Not at this time. We'll rebase that branch when all support is in trunk.

6377a r15513 fixes an issue with resolving the running procedure in Agent.invoke as originally reported by #7143-1268. Tested by Radu in #7143-1285. Waiting for merge approval.

File header in r15514.

Galya B wrote:

File header in r15514.

Thanks, you can merge to trunk now.

I think we need to reopen this task to get the REST API implemented. The admin console UI is useful but it will not be the way most customers need to use this feature.

Greg Shah wrote:

I think we need to reopen this task to get the REST API implemented. The admin console UI is useful but it will not be the way most customers need to use this feature.

Constantin had the actions exposed via java commands, so they are not available only via the console UI. I can't find the original task where he did it though.

#7074-6.