FWD » Core Development » User Interface

Create a special P2J client that is a dedicated client process launcher. This client must be able to run on any arbitrary system, not just on the server on which the P2J server executes. The idea is that this is a remote launcher that allows web clients, batch processes and appserver agent instances to be spawned on a system that is different from the P2J server's system.

This approach will trigger at least one known problem with cross-origin cookies used for authentication token processing. See #1811 note 599 for details.

To be clear, this special launcher client would be started on the remove system and it would be configured to connect to the P2J server and authenticate using certificates. Then it would wait for the P2J server to tell it to spawn a process. The same capabilities must be present as if the P2J server was spawning processing directly. This means there is some bidirectional data exchange that will have to occur as part of the spawning process.

The portions of the spawning process that use a login to the P2J server should translate to this approach naturally.

• the P2J Server can choose a P2J Client spawner randomly or using some load balancing policy (and this means all the machines are configured in the same way)
• each P2J Client spawner has a list of known OS usernames, and the P2J Server will choose the P2J client spawner to which the OS username belongs; problems may appear if the same OS username is registered with more than one machine.

Doesn't the 4GL appserver implementation have some features in this regard? We should at least document their basic approach here as part of this task.

Since we don't have any specific customer requirements right now, I'm inclined to start with a very simple load balancing approach (evenly distribute clients between remote launchers. We can possibly add some min/max range limits for each one. That is about it, unless the 4GL compatible approach is really simple to implement.

For remote client launching, first component we need is a broker; this will be a special P2J client which authenticates (via SSL) and registers with a remote server. The remote server will keep a registry of registered brokers and will use these brokers for remote launching P2J clients. Both web and appserver agent clients need to be covered by a broker. I think 4GL does some load balancing via the NameServer Load Balancer associated with an appserver - please check how this is done.

• When registering with the remote server, the P2J server will determine the broker's configuration (from the directory), which includes:
  - details about the appserver(s) which can be started on by this broker
  - details about the web clients (the P2J usernames?) which can be started by this broker
• when starting a remote client, if more than one broker is registered for that client (web or appserver agent), the P2J server should choose the broker depending on the load balancing policy in use:
  - for web clients, I think it should be OK to choose the broker with the lesser load
  - for appserver agents, we should use the 4GL's load balancing policy, if this can be easily identified (from the documentation or something else).
  This will mean the broker will need to expose some APIs related to how many agents were started by that broker (and are still running), to determine its load. Alternatively, I think this is easier to be managed on the P2J Server side, if the client (server-side part) knows which broker started it.
• the same broker client can be started multiple times; this will allow live attachement of brokers to a running P2J server.

• the broker clients will need special configuration in the directory, to identify themselves as brokers. The StandardServer.standardEntry will check if the client starts as a broker, similar to how the appserver agents are checked, via a if (BrokerManager.startBroker) { return false; }.
• when the P2J client acting as a broker is terminated, we need to choose if we want to either terminate all P2J clients started by this broker or let them run.
• the P2J broker client will need to export a special interface as a network server: this interface will be used by the P2J server to send a command to the remote P2J broker.

• create a new object-class in the dir_schema.xml file, named broker.
• the process object-class will need a new attribute, broker, with the name of the broker being started.
• I think we should keep the existing approach which allows startup of appserver agents/web clients on the same machine as the P2J server. If no brokers are configured for that web client/appserver agent, the existing infrastructure will be used. If brokers are configured, then the broker will be used.
• please check how the BrokerManager can be integrated with the existing ClientBuilder and ClientSpawner classes and document it here.
• ClientCore.start I think will need to be aware that this is a P2J broker client and bypass the normal startup (i.e. ThinClient, etc).

Please let me know if you have any comments.

For the beginning I have some comments and questions please.

1. P2J server and AppServer are distinct terms? If yes:
- StandardServer which implements MainEntry acts like an AppServer?
I saw that is used by the P2J clients to execute the converted applications because the clients are thin and acts like a presentation layer.
The business code is always executed on the server side (AppServer)
- ServerDriver used to start server acts like a P2J server?

2. The broker run on a remote machine A and the P2J server on another machine B.
The broker should connect and register as broker on a P2J server on a secured manner (SSL).
When a remote client is spawned by this broker on machine A the AppServer is running on machine A where the broker runs or on machine B where P2J runs?

3. The start of the remote broker is similar with the start of a P2J client like for example the Swing driver via a shell script?
What I have to ask here in fact is if the launching of brokers are not done by P2J server.

4. I didn't found a BrokerManager class inside P2J project.

Marius Gligor wrote:

For the beginning I have some comments and questions please.

1. P2J server and AppServer are distinct terms? If yes:

Yes, they are distinct. A P2J Server can have one or more appserver exposed via P2J processes.

- StandardServer which implements MainEntry acts like an AppServer?

StandardServer.standardEntry is the entry point which is reached by all started P2J Clients, when their server-side part is initiated. The ClientCore.start:264 line (running = main.standardEntry(params); will execute the server-side entry point of a P2J client. On a side note, I think ClientCore should be left alone. ClientDriver.start should choose between ClientCore and BrokerCore (a new class), depending on some bootstrap configuration.

I saw that is used by the P2J clients to execute the converted applications because the clients are thin and acts like a presentation layer.
The business code is always executed on the server side (AppServer)

Correct.

- ServerDriver used to start server acts like a P2J server?

Yes. ServerDriver relies on StandardServer.bootstrap to start a P2J Server.

2. The broker run on a remote machine A and the P2J server on another machine B.

Yes.

The broker should connect and register as broker on a P2J server on a secured manner (SSL).

Yes. It will be configured and authenticate as a P2J process account.

When a remote client is spawned by this broker on machine A the AppServer is running on machine A where the broker runs or on machine B where P2J runs?

Well, the appserver (i.e. the AppServerManager.startAppServer) will always execute on the P2J Server side (machine B). Machine A will hold only the P2J clients which act as the appserver agents. Currently, these agents are started via the ProcessClientSpawner class.

3. The start of the remote broker is similar with the start of a P2J client like for example the Swing driver via a shell script?

The P2J server has no knowledge about where the broker clients reside. These special P2J clients (brokers) need to be started manually or other way.

What I have to ask here in fact is if the lunching of brokers are not done by P2J server.

Correct, the P2J server is not involved in starting them.

4. I didn't found a BrokerManager class inside P2J project.

This is a new class which we need to add. It will provide broker management on P2J server side (i.e. which brokers are started, choosing a broker, etc). The launch command will be computed on P2J server side (what the ClientBuilder classes do), the broker will just delegate the command launching to its associated P2J client.

The remote launcher (broker) connect to the P2J server and authenticate using certificates.
Is any example in P2J on how to implements authentication based on certificates only?
So far all that I found require also an User ID and a Password.

Marius Gligor wrote:

The remote launcher (broker) connect to the P2J server and authenticate using certificates.
Is any example in P2J on how to implements authentication based on certificates only?
So far all that I found require also an User ID and a Password.

See the p2j.main.NativeSecureConnection class: this class is used by the spawner to authenticate with the specified P2J server and retrieve the command associated with the given UUID. By the way, I think the broker still needs to rely on the spawner to launch the P2J client; thus the P2J server will send the broker a UUID for the command + plus the P2J server details (in case when a process is launched) or the entire command, in case a web-client is launched.

I'm using your code from NativeSecureConnection class but we still need an account to connect to P2J server.
You are using the temporary account by getting the environment variables:

cfg.setConfigItem("access", "subject",  "id",   TemporaryAccount.fromHex(subject));
cfg.setConfigItem("access", "password", "user", TemporaryAccount.fromHex(password));

Marius Gligor wrote:

I'm using your code from NativeSecureConnection class but we still need an account to connect to P2J server.
You are using the temporary account by getting the environment variables:
[...]
If these credentials are missing from configuration the user will ask to enter a subject id and a password from stdin.
I tried both connectDirect and connectVirtual and both require subject id and password.

Yes, the broker needs a specific process account defined in the P2J directory. This process account will have a broker attribute, to identify itself as a broker.

• keystore, via security:keystore:filename, access:password:keystore, access:password:keyentry
• keystore alias, via security:keystore:processalias - this will identify the process cert alias (which needs to be the same as the P2J process Id).

Originally we thought that SSLCertGenUtil will not need to save the process private key in a separate keystore file, but it looks like this is no longer valid; I'll modify SSLCertGenUtil to allow saving of a process key store to a separate file.

Using SSLCertGenUtil I reconfigured the certificates/private keys in the server directory and I created a client.xml file for client strong authentication.
Now it's works like I expected and no user credentials are needed for authentication. The authentication is done based on certificates only.

I've implemented a P2J client (Broker) which connect to P2J server and authenticate using certificates.
Both the client and the sever exports some some services for register, unregister, etc.
It works fine when both server and client are started within Eclipse IDE.
When I tried to start the client and server from separate batch files the SSL authentication fails.
I exported the p2j-root alias as a PEM certificate and than I imported the certificate into the Java truststore.
Now when I tried to connect to the server I've got an "Authentication failed"
Do you have any idea how to fix this issue?

I've fixed the issue by rebuilding the P2J project. I deleted the p2j-root certificate from Java keystore.

Both web and appserver agent clients need to be covered by a broker.

Don't forget that we need to allow batch clients to be started too.

a list of user accounts that could be used to remote spawn clients on machine on which the broker runs

Where does the list come from?

I don't want the customer to have to maintain this on every system where there is a broker. It would be a source of lots of maintenance effort and support problems. Can we find an alternative approach that is dynamic? For example, can we ask a broker to try to login for a given user and fall back to a different broker if that did not work?

Greg, I was thinking for all the broker configuration to reside in the directory, similar to how the appserver is configured: the process has a broker attribute, and this attribute points to its configuration, in some <server-id>/brokers node. When the broker initializes on server-side, it reads its configuration from the directory, which includes info about which i.e. system usernames and/or P2J accounts and/or appserver names it can manage.

Yes I think that is better to keep broker configurations on server directory. In this case when a broker connect and register to the server it should send only the broker name.
Brokers having the same name acts like a load-balancing system since they have the same configuration. Is this true?

Marius Gligor wrote:

Yes I think that is better to keep broker configurations on server directory. In this case when a broker connect and register to the server it should send only the broker name.

You don't need to send the broker name, this can be determined from the associated P2J process account - the broker attribute.

Brokers having the same name acts like a load-balancing system since they have the same configuration. Is this true?

Yes, but not only this. The idea was that if a P2J account (be it appserver, web client or process) can be handled by multiple brokers, than a load-balancing policy applies.

Constantin Asofiei wrote:

Greg, I was thinking for all the broker configuration to reside in the directory, similar to how the appserver is configured: the process has a broker attribute, and this attribute points to its configuration, in some <server-id>/brokers node. When the broker initializes on server-side, it reads its configuration from the directory, which includes info about which i.e. system usernames and/or P2J accounts and/or appserver names it can manage.

That sounds reasonable, with one caveat: no configuration should be needed in the default setup. In such a case, if there is only 1 broker then that will be tried. And if there are more than one broker (but no user lists), then it will try them all in a round robin until one works.

Greg Shah wrote:

Constantin Asofiei wrote:

Greg, I was thinking for all the broker configuration to reside in the directory, similar to how the appserver is configured: the process has a broker attribute, and this attribute points to its configuration, in some <server-id>/brokers node. When the broker initializes on server-side, it reads its configuration from the directory, which includes info about which i.e. system usernames and/or P2J accounts and/or appserver names it can manage.

That sounds reasonable, with one caveat: no configuration should be needed in the default setup. In such a case, if there is only 1 broker then that will be tried. And if there are more than one broker (but no user lists), then it will try them all in a round robin until one works.

One more note: do you want to remove the current setup, which relies on the clientConfig nodes to launch a appserver/web client on the same machine as the P2J server? To me, this looks like the easiest way to setup a P2J server with a "built-in spawner".

To me, this looks like the easiest way to setup a P2J server with a "built-in spawner".

We should retain the ability to have a P2J server that has the built-in spawner.

Today I've continued to work on remote launcher client (broker).
I tried to spawn a web client locally via a remote launcher client running on the same machine where the P2J server runs.
I did the appropriate changes to prepare the spawn but I found an issue and I need some help to fix them. Let me to explain.
Both the server and the broker exports a lot of services.
When the broker connect and register he call a server exported method. Within this method I'm able to get the broker exported interface from server and do service calls on the interface.
Later when I tried to call broker services within another session I cannot do. The call remains blocked undefined.
It seems that the broker exported interface is available only within the broker session or perhaps special rights are necessary.
How could I do to access the broker exported services on server from other sessions?

I implemented a flaw on which the broker call a server exported method each second and check if a spawn task is scheduled for him, than he get the spawn parameters and do the spawn. This flaw woks fine but is not so elegant like calling broker interface directly.

I suspect this is an ACL problem. Each API that is exported must be configured with the appropriate ACLs to allow the actual access to occur at runtime.

I added the following node to server directory.xml but still not working.

      <node class="container" name="net">
        <node class="container" name="000100">
          <node class="strings" name="subjects">
            <node-attribute name="values" value="all_others"/>
          </node>
          <node class="netRights" name="rights">
            <node-attribute name="permissions" value="'00111111'B"/>
          </node>
          <node class="resource" name="resource-instance">
            <node-attribute name="reference" value="com.goldencode.p2j.main.BrokerClientServices"/>
            <node-attribute name="reftype" value="TRUE"/>
          </node>
        </node>
      </node>

I fixed the access to BrokerClientServices. The node that I inserted is OK. After a build all of P2J now it works.

Unfortunately I was wrong. I tested some old code on which the client connect to server and this works.
So I have to make more diggings.

I wasn't thinking correctly. On the client side there are no ACLs in play since the client's security manager is "empty". It approves all access decisions without any checking.

I don't know what is wrong in this case.

1. ClientDriver.start. This code is not reliable:
   

        if (BROKER_LOG.equals(logName))
            BrokerCore.start(config, args);
         else
            ClientCore.start(config, true, args);
   

   
   You should have a bootstrap config, something like "client:mode:broker=true": if this is set, start the broker, else the client.
   Also, in setClientLog you are suggesting that the broker's name can be set only via command line arguments. I don't like forcing the user to always use a command line argument, the broker's name should be a generic bootstrap config.
2. about the ACLs problem. Check the resource-plugins node in the directory, for the kind of registered resources. Depending on these resources, certain ACLs need to be added. Check #2124-38 for what was needed for MAJIC, when the web clients were added. Also, are you sure the thread from which you are accessing the network service has the proper context set?
   Also, you can debug into the Dispatcher.getRoutingKey and processInbound (for the NetResource usage) - and see where the access is denied.

Marius Gligor wrote:

Here is a simplified skeleton for remote brokers.

1. As I understood when a broker connect and register to P2J server no information is send to server. The broker configuration is keep in server directory.

- What kind of information should contains a directory entry for a broker?

If think is enough to keep a list of P2J account names, which are managed by this broker. And as Greg noted, "if there are more than one broker (but no user lists), then it will try them all in a round robin until one works."

- How broker configuration is used by clients and processes?

When the P2J server decides that a new remote client needs to be started, it will either choose a broker based on the account name (if brokers are configured), or it will fallback to the embedded spawner. Also, the P2J processes associated with the broker need to have a broker attribute set, with the broker name.

- How registered brokers are associated with the directory entries if more than one broker is registered?

Not sure what you mean here. If more than one broker (with the same name) is registered, then it will be treated as a distinct broker instance.

Could you please offer me a short description regarding how brokers works?

See note 7. Also, look at how the AppServerManager treats the Agents: each Agent thread has a listening loop, where it listens for commands. When a command is received, it executes it. You can use a similar approach:
- each broker has its own thread, with a loop listening for commands
- after the proper broker is picked, the BrokerManager sends a command to the Broker thread
- the Broker thread picks the command and sends it to its remote side (the P2J broker client).
If you think is best that each broker command is to be executed in a different thread, then be careful about the thread's context: if the broker's context is not set, then you will not be able to access the network servers.

Btw, what is your current approach? I wonder if it is not enough to use the broker from the thread where the authentication is performed... and do some context switch/restore to access the broker.

2. When a process is spawned remotely some launch parameters are specific to machines on which the brokers runs:

- classpath.
- native libraries path.

I think these can be determined from the runtime of the P2J client acting as a broker

- spawner location

I think a client config should be used for this - it will be a folder name where the spawner is installed, not the location of the spawn tool.

- the remote spawned client should use the IP address of P2J server instead localhost in order to make a connection with P2J server.

Agreed; but doesn't the P2J broker already know the IP address of the P2J server to which it is connected? Each broker instance will be dedicated to only one P2J Server, and all clients started by it will always connect to that P2J server.

Where should be kept all this information?

At this time, the only config which needs to be set at the client.xml for the broker client is the spawner folder; use a bootstrap config to determine it. Also, if it is missing, you can assume that there is a spawner folder in the same folder as where the P2J Client was started.

ca_upd20140707a.zip was committed to bzr rev 10567.

I did some debugs and I found:

1. When I call a client exported method from server, the request is arrive to client side.
In the Conversation.request line 222 the request message is stored and a waiting thread is notified.
Unfortunately nothing happen and the client goes to reader Protocol.Reader.run line 337 waiting to read new objects from network.
No reply message is send back to server and server remains in a waiting state forever Conversation.block line 298.

2. On the server side when a new broker is connected (register) I stored the session and the remote proxy objects in a per thread structure. More interesting using the broker stored session I'm able to obtain the remote network interface and do calls using proxy objects from other contexts.

params.remote = (BrokerClientServices) RemoteObject.obtainNetworkInstance(
                                       BrokerClientServices.class, 
                                       params.session);
params.remote.ping(params.uuid);

The code above woks when is executed at the broker registration time.
When I tried to execute the same code from another context it does not works.

My question is: Where is the problem on server side, client side or both?

1. if you are calling BrokerClientServices APIs via BrokerParameters.remote from a different thread, you need to switch contexts before doing so. The problem here is that you will need to save and expose the broker's context in i.e. BrokerParameters.context. If you choose this path, you need to secure the access to the broker's context very tight - but looking at RouterSessionManager.setContext, I don't think this is a good choice (security wise). Instead, see next point.
2. implement a server-side command listening loop, by calling a new API, i.e. BrokerServerServices.start, which will start the listening loop and wait for work. When work arrives, the work can be done in a separate thread, created via p2j.security.AssociatedThread - this new thread will inherit the security context from the creator.

I still have something that is not clear for me.

Let say we have 2 brokers in server directory broker1 and broker2. We have a process having broker attribute broker1.
At one moment we have 3 p2j clients acts as brokers which runs on 3 different machines connected to p2j server (registered).
We have no association between brokers name and the connected brokers.
Which broker is selected from registered brokers when we need broker1 or broker2?
Are all the 3 registered brokers equals from the selection point of view?

• we have broker names broker1 and broker2
• broker1 has 3 connected instances, broker2 has 5 connected instances. broker1 has authenticated using P2J process pbroker1 and broker2 has authenticated using P2J process pbroker2. How do you know which broker is associated with a certain context? Each P2J process will have an attribute with the broker's name, in its directory configuration:
  - see how the appserver attribute is set at the P2J process (in dir_schema.xml) and in the directory
  - see the SecurityManager.getAppserver, which determines the appserver name for the current context (from the ProcessAccount.appserver field).
  - see the SecurityManager.getAppserverForProcess, which determines the appserver name for the specified account (from the ProcessAccount.appserver field).
  Similar changes are needed for brokers - you need to mark a P2J process account as belonging to a certain broker, and you need to determine which broker is associated with the current context. Thus:
  - process pbroker1 will have a "broker=broker1" attribute
  - process pbroker2 will have a "broker=broker2" attribute
  Also, each broker name will have these nodes, under the security/server/{default|<server-name>}/ node:
  

  <node class="container" name="brokers">
     <node class="broker" name="broker1">
        <node-attribute name="account" value="p2j_account_1"/>
        <node-attribute name="account" value="p2j_account_2"/>
        ...
     </node>
  </node>
  

  
  You need to define a broker class in dir_schema.xml. When reading the broker configuration, use the p2j.util.Utils.findDirectoryNodePath API with the scope parameter set to false.
• lets say we want to launch a web client for P2J user "foo":
  - first step is to determine which broker(s) can handle this P2J user
  - second step is to apply a load-balancing policy over the broker(s) determined on previous step, and choose one, which will launch it.
  - a tricky problem is when the brokers have an empty user list, when load balancing will not be that easy to implement, as the broker's don't know in advance which users can manage.

To answer your questions:

We have no association between brokers name and the connected brokers.

See above how to make the associations. The broker name should not be at the client-side, it's something server specific; the client-side doesn't even care what name has the broker, all it cares is that the client is a broker. Thus, the remote:broker:name is not needed: to identify the client as a broker, you need to look into the BootstrapConfig instance and check for a client:mode:broker=true setting.

Which broker is selected from registered brokers when we need broker1 or broker2?

The broker names are not relevant in choosing a certain broker; use UUID's if you want to uniquely identify each connected instance. The broker instance knows its name (as it is something context-specific), which will be used to access the broker configuration from the directory. These configurations can be loaded once upon BrokerManager initialization, to not re-read it from the directory each time.

Are all the 3 registered brokers equals from the selection point of view?

Yes, only if a broker is configured to handle a specific P2J user, than all its instances can handle it. If the broker doesn't have a account list, than each broker instance needs to check if it can handle the specified account.

Marius Gligor wrote:

2. I did some directory configurations for brokers. Here I have a short question please:

The idea behind Utils.findDirectoryNodePath (and similar APIs) is to first look for a per-server configuration, and if it does not exist, use a global (default) configuration. Thus, i.e. if you have a server/standard/brokers node, that node will be used. If this node is missing, it will search for a server/default/brokers node (the global configuration).

About the update: do you think is still needed to associate the P2J process with a specific broker, in the directory? Your current approach is to configure the P2J client upfront as a broker.

The association between defined brokers inside directory and registered brokers (instances) is still no clear for me.
When the broker is connected to server he is authenticated using a P2J account in my case p2j_proc
On directory brokers configuration I have no informations regarding the account used by brokers to authenticate.
The only information we have is the account used by broker extracted from broker context when he ask server to register him as broker.
I think that the broker account should be the link between defined brokers and registered brokers.
What is in fact the rule?

Marius Gligor wrote:

The association between defined brokers inside directory and registered brokers (instances) is still no clear for me.
When the broker is connected to server he is authenticated using a P2J account in my case p2j_proc

p2j_proc is a process account, right?

On directory brokers configuration I have no informations regarding the account used by brokers to authenticate.

The brokers node does not need to know which P2J process accounts are configured to start a specific broker. To be able to retrieve the broker name associated with a P2J context, you need to add a new attribute at the P2J process account, named broker; this way you will be able to retrieve the broker's name, from the P2J context - see note 45. You will need to add some new SecurityManager APIs to retrieve the broker name, from the context.

The only information we have is the account used by broker extracted from broker context when he ask server to register him as broker.

Correct; this is why you need mark the P2J process account with the broker's name.

BTW, something I had wrong: at the moment the user wants to start a i.e. web process, the P2J process doesn't know which P2J account the user will connect as; all it knows is the OS username. Thus, the broker needs to know a list of OS users, not P2J account names. More, when BrokerManager.selectBroker is called in ClientBuilder.remoteStart, you need to pass the OS username as parameter, not the P2J broker account.

Some other notes: you need to add some checks so that a P2J client which is marked as a broker (via bootstrapconfig) will allow startup only if the P2J account is a process account which is marked as a broker.

OK tahnks. What I understood is that the brokers are always authenticated using a P2J process account not a user account.
The process account has an attribute "broker" which is used to link the broker instance to defined brokers.
The user accounts has also a "broker" attribute specifying which broker is used when a remote spawn is made for this account.
Web clients are indeed specials because they are use OS accounts instead P2J accounts.

Or more general. Both process and user account has the broker attribute.
On broker register the attribute is used to link the instance with defined brokers and on remote spawn to select the broker.

What I understood is that the brokers are always authenticated using a P2J process account not a user account.

Correct.

The process account has an attribute "broker" which is used to link the broker instance to defined brokers.

Correct.

The user accounts has also a "broker" attribute specifying which broker is used when a remote spawn is made for this account.

I don't understand this one. The idea is that the broker configuration (in the brokers node, the per-server/default configuration) knows a list of OS usernames. The P2J user accounts don't know which broker will be used to launch their P2J client, this is determined at runtime (depending on the load balancing policy, which brokers are registered, etc).

Marius Gligor wrote:

Or more general. Both process and user account has the broker attribute.

No, only the P2J process can have a broker attribute. We don't allow P2J users to authenticate as brokers (that's why I noted that we will allow startup only if the P2J account is a process account which is marked as a broker.).

Web clients are indeed specials because they are use OS accounts instead P2J accounts.

Batch processes need OS logins too.

I added the broker attribute to process account used by broker to authenticate:

<node class="container" name="processes">
  <node class="process" name="p2j_proc">
    <node-attribute name="enabled" value="TRUE"/>
    <node-attribute name="alias" value="p2j_proc_alias"/>
    <node-attribute name="description" value="a P2J batch process"/>
    <node-attribute name="server" value="FALSE"/>
    <node-attribute name="master" value="FALSE"/>
    <node-attribute name="broker" value="broker1"/>
  </node>
</node>

<node class="container" name="000500">
  <node class="strings" name="subjects">
    <node-attribute name="values" value="all_others"/>
  </node>
  <node class="systemRights" name="rights">
    <node-attribute name="check" value="true"/>
  </node>
  <node class="resource" name="resource-instance">
    <node-attribute name="reference" value="accounts"/>
    <node-attribute name="reftype" value="TRUE"/>
  </node>
</node>

1. In cases when the IF/FOR/ELSE block has a single statement, always enclose it in curly braces. i.e. Broker.java:82
   

               if (params != null && params.session != null)
                  params.session.terminate();
   

   
   should have been:
   

               if (params != null && params.session != null)
               {
                  params.session.terminate();
               }
   

   
   There are other files with the same problem (i.e BrokerCore.spawn, BrokerMAnager.initialize, BrokerParameters.equals/close, BrokerManager.findBroker, BrokerManager.getBrokers, etc).
2. Make sure the methods/fields are ordered properly, in the class (by their access modifiers).
3. Do not use System.err.println to log stuff. Instead, define a static variable like this:
   

      /** Logger. */
      private static final Logger LOG = LogHelper.getLogger(<class-name>.class.getName());
   

   
   and use that with a logging level.
4. the header in the new files, needs to be 98 lines long, like this:
   

   ** -#- -I- --Date-- ---------------------------------Description----------------------------------
   

5. Please rename Broker.haveAccounts and Broker.haveBrokers to hasAccounts and hasBrokers.
6. The while on BrokerCore.connect:216 needs to be on its own line. Same for the else if clause in BrokerParameters.equals:95.
7. BrokerParameters, BrokerSpawnParameters, BrokerSpawnResult - please make the fields private and access them via getters/setters.
8. BrokerParameters.commands - is this field used?
9. BrokerSpawnResult.exception - I think you should pass this to the server side, to log it in the server's log.
10. ClientDriver:115 - remove this empty line.
11. BrokerManager.spawn - the synchronization I think is too restrictive. You are allowing only one broker at a time to do its work:
    

          synchronized (activeBrokers)
          {
             // select a broker from list.
             BrokerParameters broker = schedule(brokers);                  
             // spawn P2J client 
             BrokerSpawnResult result = broker.remote.spawn(args);         
    
             // on error mark as available for schedule
             broker.scheduled = (result.exitCode == 0);
    
             return result;
          }
    

    
    I think it should be split this way:
    

          synchronized (activeBrokers)
          {
             // select a broker from list.
             BrokerParameters broker = schedule(brokers);                  
          }
    
          // spawn P2J client - the broker is already deactivated, so no one else can use it.
          BrokerSpawnResult result = broker.remote.spawn(args);         
    
          synchronized (activeBrokers)
          {
             // on error mark as available for schedule
             broker.scheduled = (result.exitCode == 0);
    
             return result;
          }
    

12. BrokerManager.schedule - if no broker is available, why are you resetting them all? Doesn't this break the current approach (which requires that a broker can handle only one request at a time)?
    

          // search for an available broker 
          for (BrokerParameters broker : brokers)
          {
             if (!broker.scheduled)
             {
                broker.scheduled = true;
                return broker;
             }
          }
    
          // all brokers was scheduled reset
          for (BrokerParameters broker : brokers)
             broker.scheduled = false;
    
          // pick the first broker from list mark as scheduled and return
          BrokerParameters broker = brokers.get(0);
          broker.scheduled = true;
          return broker;
    

    
    The key question here I think is: why can a broker handle only one request at a time? Is it possible for the broker client to execute the request in a separate thread, and start listening immediately after? Look into how async requests can be sent by the P2J net protocol - at least, the thread needs to be registered with the SessionManager as async.
13. I think you've missed something: if no broker is registered with a certain user, and there are running brokers, then P2J needs to try them all in a round robin until one works.

• concurrency issues: launch two or more remote clients (using the same broker) at the same time, to check for concurrency issues.
• edge cases: what happens when all brokers are busy, if the launch failed, if the broker gets terminated during a launch, etc.

Using our spawner we cannot manage the spawned P2J clients. When a P2J client is spawn the spawn returns but P2J client is running. We don't know how many P2J clients are still running at one moment.

I think the P2J server can track this, if the remote P2J client knows which broker started it (the broker's UUID). When the P2J client starts, it can send to the P2J server this broker UUID, and the BrokerManager can keep track of live P2J clients (via session listeners). But this will only limit us to the number of P2J clients still active, not the actual system load. And in a machine-load point of view, is best to know the actual machine load, not how many P2J clients are running. So I think we should go with the native approach, but this can be done after we make sure the remote launching works properly.

a). Web clients can be spawned local or remote using a broker. The cross domain issue cookie is true and occur even on the same machine.

Yes, the cross-domain cookie is a issue we need to solve, but can be deferred for another task (as is related to the web client protocol, not necessarily to remote spawners).

Batch clients are working when are local spawned as is already implemented but are not works with brokers.

How do you start them? Via ServerDriver's -b argument?

The client.xml file on Windows looks like:

Please post a minimal and maximal broker configuration for both Linux and Windows (if some nodes are only for windows and some only for linux).

I found why the batch clients does not work with brokers on my current implementation:
The batch client command is stored on the server side in ClientBuilder.pendingCmds having an UUID as a key.
Than a special client is spawned which create a JVM instance , connect to server, get the batch command by UUID and execute the command. All this happen inside the spawner in C code. This special client works local or remote when is spawned by broker. No problems here,

For remote execution I'm using placeholders for classpath, native library path and spawner file. The placeholders are replaced by broker when the client is spawn. The class path and native library are inherited from broker runtime context. The spawner file is get from config.xml file and all localhost strings are replaced with net:server:host parameter read from configuration file. see BrokerCore.spawn
This works fine for web clients but not for batch clients because web clients are always spawned by broker and placeholders are replaced before spawning.

When a batch client command is prepared for remote spawn the command arguments contains placeholders that should be replace on broker side.

Because the batch client is spawn within spawner code and not from broker palceholders remains unchanged and the Java command can not be executed due to errors in command line, arguments that are not understand by Java.

A possible solution is to replace holders on server side not on broker side.
In this case we have to get the values for placeholders from broker, or on broker registration to get this values and keep them on the broker structure.
A solution could be to implements this on spawner code by chancing the C code.
So far I'm thinking to the better solution.
What are your suggestions regarding this issue?

is best to know the actual machine load, not how many P2J clients are running. So I think we should go with the native approach

I think we can defer this part for now. Let's create a separate task for this. I don't think we need it for the current project.

a). Web clients can be spawned local or remote using a broker. The cross domain issue cookie is true and occur even on the same machine.

Yes, the cross-domain cookie is a issue we need to solve, but can be deferred for another task (as is related to the web client protocol, not necessarily to remote spawners).

Yes, this can be a separate task, but it is definitely needed for this project. Marius: work on this cross-domain task once this current task is complete.

I found a solution for batch clients which works. When a broker starts he export his JVM arguments and classpath into a file classpath.broker inside spawner directory.
When NativeSecureConnection.command is called within spawner in order to get the batch client command before returning the command arguments the placeholders for JVM
arguments and CLASSPATH are replaced from the exported file classpath.broker. All these happen on broker side.
The only problem could arise if broker has no enough rights to create a file inside the spawner folder. Usually the broker and spawner should reside on the same folder.
I'm thinking also to other kind of interprocess communications instead files suitable for our purposes.

A possible solution might be via an environment variable which is limited on size.

Marius Gligor wrote:

I found a solution for batch clients which works. When a broker starts he export his JVM arguments and classpath into a file classpath.broker inside spawner directory.

• when the broker invokes the spawn tool in server auth mode, it sends as arguments any placeholders which need to be replaced (i.e. classpath, libpath, etc)
• when the NativeSecureConnection class is invoked, you either pass these placeholders as parameters to the command method and prepare the command in the Java code, or the native code in launchP2JClient will prepare the retrieved command, to replace the placeholders. Also, do not allow launch if placeholdrs can't be resolved.

1. the header first line needs to have the Description text centered, as in note 56.
2. NativeSecureConnection:153 // add JVN arguments should be // add JVM arguments
3. why did you remove the ping service? This can be usefull, in i.e. the Administration Console, to test if a broker is alive/responsive, or for latency check. Please add it back, with a more appropriate javadoc.
4. BrokerManager:241 - the while needs to be on its own line
5. BrokerManager.spawn - the javadoc is formatted using a 80 char line instead of 98?
6. ClientBuilder.prepareToLaunch
   

            // localhost placeholder
            for (String arg : getSpawnArguments())
            {
               spawnCmd.add(arg.indexOf("localhost") > -1 ? 
                            arg.replace("localhost", "${remote.host}") : 
                            arg);
            }
   

   
   Why do you use "localhost" here? It seems too restrictive: what does this localhost mean? Is it the server's address? Also, is this part of a bootstrap config arg? If so, then search for the key, not for the value.
7. you have some referrences to application server, in Broker:26 and BrokerManager:125 and 128. Please change these to P2J server, to not confuse with the legacy appserver.
8. BrokerManager.spawn - before calling the remote proxy, please check if the broker is still alive. More, you should protect this for errors via a try/catch block - as the remote proxy call might fail. Same with BrokerManager.schedule - protect the remote proxy calls.

Beside the issues above, I don't see anything else wrong with the logic. So please make sure to test edge cases thoroughly, plus concurrency issues (launching two or more clients using the same broker, at the same time - you can place breakpoints into BrokerCore.spawn and BrokerManager.spawn and step through each call side-by-side).

When client is spawned locally I sow that localhost is used as host for server address but when the client is spawn remotely I think that real IP should be used.
For example in ProcessClientBuilder line 83 localhost is used and a remark:

// add host; this is hard-coded to localhost because is expected that the in-process JVM
// will connect to the same P2J server which initiated the request (as this is the P2J
// server knowing the command to be executed).

Marius Gligor wrote:

When client is spawned locally I sow that localhost is used as host for server address but when the client is spawn remotely I think that real IP should be used.
For example in ProcessClientBuilder line 83 localhost is used and a remark:
[...]
Is this true also for remote spawn?

• do we assume that both the server and the remote broker are on the same network?
• if they are not on the same network, then we need to determine an address in the same network as the remote broker.

• when local spawner is used, change it to localhost
• when remote spawner is used, change it to a server address in the same network as the broker client.

The broker an server could be on different networks for example if server is behind a network router.
I think that is better to let the broker to do placeholders replacements using the server address he know.
We could do changes to resolve localhost placeholders exactly where they are used. For example:

ProcessClientBuilder.getSpawnArguments(boolean remote)
{
...
   cmd.add(remote ? "${remote.host}" : "localhost");
}

Marius Gligor wrote:

The broker an server could be on different networks for example if server is behind a network router.
I think that is better to let the broker to do placeholders replacements using the server address he know.

Yes, you are correct, the broker already knows the P2J server address to which it has connected. Your approach is OK, but make sure to make the secure port configurable, too - when remote, let the broker replace it with the port from its bootstrap config.

Marius Gligor wrote:

1. I've fixed the issues reported on previous code review.

Please fix the header in the new files so that instead of this:

** -#- -I- --Date-- ----------------Description--------------------------------------------------

** -#- -I- --Date-- ---------------------------------Description----------------------------------

3. I did a lot tests and I found no problems so far. However in Linux p2j.jar and srv-certs.store must have root access when spawn batch clients.

The p2j.jar in the spawner folder (which must have 0550 permissions) is used only by NativeSecureConnection, when connecting to the remote P2J server, to retrieve the actual command. This P2J jar should not be used by the broker client.

Otherwise, the code looks OK, so good work on this one. Please run a single runtime regression and look for obvious abends during client startup/connection.

PS: please remove the author tag from the class javadoc - we don't use that.

Marius Gligor wrote:

I started the tests from the beginning with a Majic conversion.
Could be a conversion issue? Should we restart the MAIN part?

Yes, it might be. Please reconvert and run again.

OK. But first of all I have to fix a bug related to web client remote spawn.
The parameter client:web:host=<web server address> should be replaced with the IP of machine on which the web client is running not with the P2J server.
This is basically the same address on which broker runs.

Another issue that I found. The user working directory

<node class="string" name="workingDir">
  <node-attribute name="value" value="./"/>
</node>

Marius Gligor wrote:

Another issue that I found. The user working directory
[...]
depends on platform Windows or Linux.
When the server run on Windows and broker on Linux the working directory could be ".\" but this should be "./" on Linux

Hm... the working dir is specific to the P2J account being launched remotely, right? Although this looks like is valid only for remote processes, not web clients - see ProcessBuilderOptions.getNode, which reads the configs from the user first and server second. I think this should be made standard, ClientBuilderOptions.getNode APIs should search user first and server second.

1. make this a limitation, and allow a remote P2J client to be spawned only on windows or only on linux brokers.
2. allow two configurations, workingDir for windows and linux. And use the correct one, depending on which OS the broker is running.

When command is prepared on P2J server doesn't know which broker will be selected and on which platform is running.

I've fixed web server address for remote spawn. The solution is to not send this client:web:host=<web server address> parameter on remote spawn.
When client:web:host=<web server address> is missing the web server will choose the IP addresses and the browser will be redirected correct.
The browser and the web client must be on the same network.

make this a limitation, and allow a remote P2J client to be spawned only on windows or only on linux brokers.

For now, this will be a limitation. Make sure it is included in our documentation updates.

Marius, I think the only issue here is that a log is created for the error. I checked my logs and the message related to the "curFac has not yet been created" exists, but without a stacktrace.

OK, the stacktrace comes from a code in ControlFlowOps recently added... so you can stop your testing, is not your problem.

I'm good with the 0724a.zip changes. If your OK with the testing you've done, you can release them.

OK. I'm going to do some final tests on my network and than I will commit the changes and distribute to team members.

Before we close this task, it is important that the "P2J Runtime Installation, Configuration and Administration" book be updated with the full details of how to:

• install a remote launcher (e.g. what files go where)
• the OS dependencies that the remote launcher needs to have satisfied in order to properly run (e.g. security/permissions, environment...)
• the additional OS dependencies that the launched clients require in a remote launch scenario (only those dependencies, if any, that are different than we would already need for a standard client system)
• the P2J configuration of the remote launcher/broker
• the P2J configuration of the server (e.g. the directory changes) for remote launching
• limitations and issues

I'm not sure if I have forgotten something. The objective is to make the documentation complete in regards to the proper implementation of remote launching.

Marius: please make the changes to the book (it is all in bzr). Constantin will review it and will guide any questions.

To start, please note that when you edit an existing file, you should enclose your changes inside brackets like <mag>this is some added text</mag>. This lets the reviewer quickly find your changes and the reviewer can edit/cleanup the text and will remove the enclosing tags as part of the review. If you believe that some or all content is best put into a new chapter, that is fine too. Any new chapters must be added to the master document. Make sure you review the overall book first to know where your changes need to go.

OK. I did the checkout from bzr. I have a first question. If I want to put all in a new chapter this means to create a new odt file and add a link of the document to master.odm file?

If I want to put all in a new chapter this means to create a new odt file and add a link of the document to master.odm file?

Yes, exactly right. Make sure to use the books/templates/book_chapter.ott as the template for that chapter.