Project

General

Profile

Wiki » Documentation » Installation Configuration and Administration Guide »

Remote Launchers

Introduction

This chapter describes the installation and configuration of remote launching of FWD clients. A remote launcher client (also known as a “broker”) is a special FWD client designed to launch interactive web clients, batch processes and appserver clients (also known as “appserver agents”) on a remote host. Remote in this context means on a separate physical or virtual system from that on which the FWD server is running. Brokers connect to FWD servers using secured socket connections and use digital certificates for authentication.

IMPORTANT: If you are upgrading an existing broker deployment to trunk revision 16699 or later, read Migration: Upgrading to the Trunk Revision 16699 at the end of this chapter before you pick up that revision. The broker directory schema and the broker client bootstrap keys both changed incompatibly: an unmodified directory.xml prevents the FWD server from starting, and an unmodified broker client.xml fails silently.

System Requirements

Brokers are a special kind of FWD Client which connects to a running FWD Application Server and waits for a notification that a FWD Client process needs to be spawned. The system on which the Broker runs must meet all the requirements of a Client Installation.

Other requirements are related to the spawner tool, a native application which is used to launch a process in the context of a specific operating system (OS) user. For more details about installing and configuring the spawner tool see the Spawner Setup and Configuration chapter.

Installation and Configuration

The broker installation is similar to installation of a normal FWD client, configured to connect as a FWD process (non-interactively). On the remote machine create a folder which should contain a client.xml configuration file, the broker key store file, the broker trust store file and a batch file which is used to start the broker. The broker must have enough rights to access p2j.jar which contains the main class. The p2j.jar must be defined in the Java classpath. This client must also have access to the native library path. CLASSPATH and JVM arguments are inherited by spawned clients from the broker's context. As a consequence, the classpath and native library path must be accessible for both brokers and spawned clients. Basically the p2j.jar and dependencies should be installed somewhere within the machine's file system where the broker and spawned clients have access. For more details on how to install and start a FWD client see the Client_Installation chapter.

When a broker client starts, stderr will show console output. You can redirect it to a file to collect it.

A complete broker configuration file client.xml looks like:

<node type="client">
  <client>
    <mode broker="true" />
    <mode batch="true" />
    <driver background="true" />
    <driver type="gui_native" />
    <logging path="/home/fwd/projects/hotel_gui/deploy/logs/client_%uos_%pid_%uf_%g.log" rotationLimit="50000000" rotationCount="4" />
  </client>

  <net>
    <server host="192.168.1.10" />
    <server port="3433" />
    <server secure_port="3333" />
    <connection secure="true" />
  </net>

  <security>
    <certificate validate="true" />
    <truststore filename="../server/srv-certs.store" />
    <truststore alias="standard" />
    <keystore filename="../server/broker1-private-key.store" />
    <keystore processalias="broker1" />
    <authentication type="program"/>
  </security>

  <access>
    <password truststore="eS9-8b*7VeR6Z~@hfhAf9SuQkvq3wJtDg88J" />
    <password keystore="6MtJvx#38P$xy8Ymo?gh0Y7sE3ffKU)e5Vqw" />
    <password keyentry="gmxKx9(Bo12f8zvP0zlv&lt;6Rk+hc&gt;NBmkT63c" />
  </access>

  <remote>
    <retry count="10" />
    <retry seconds="10" />
    <spawner file="/opt/spawner/spawn" />
    <server host="192.168.1.11" />
    <agent user="fwd"/>
    <agent dedicatedMode="false"/>
    <java args="-Xmx512m -Djava.awt.headless=true -Djava.library.path=/home/fwd/projects/hotel_gui/p2j/build/lib/" />
    <java classpath="/home/fwd/projects/hotel_gui/p2j/build/lib/p2j.jar" />
  </remote>
</node>

The remote launch clients are using special configuration to indicate that they act as brokers. This parameter can be either preset into the client.xml file or can be sent as a command line parameter client:mode:broker=true which overrides the parameter from client.xml configuration file. As a general rule configuration file parameters can be overridden with command line parameters on broker start.

The net node in configuration file contains the FWD server address and the secure port used to connect to the server while security and access nodes contains SSL settings; is mandatory for the broker to connect in a secure mode and authenticate as a FWD process.

The security node defines the following parameters:

  1. security:truststore:filename define the location of trust store file containing server and root CA authority certificates identified by aliases. This file is the same as server trust store file.
  2. security:truststore:alias is the alias for the root CA authority certificate and must exists in the trust store file.
  3. security:keystore:filename parameter define the location of the key store file containing the user (process) public/private keys pair. A key store file could hold multiple public/private keys pairs each pair having is own alias (identifier).
  4. security:keystore:processalias parameter is the user (process) public/private keys pair identifier and must exist in the key store file. When keys are generated the alias attribute from process accounts is used as the key store identifier (alias) for a public/private key pair.
    <node class="process" name="p2j_proc">
  ...
  <node-attribute name="alias" value="p2j_proc_alias" />
</node>
  5. security:certificate:validate if true, then the server's certificate must be present in the trust store for authentication to succeed.
  6. security:authentication:type specifies the authentication type. Valid values: program

The access node holds the following parameters:

  1. access:password:truststore Specifies the password to be used to read/decrypt the trust store file.
  2. access:password:keystore Specifies the password to be used to read/decrypt the key store file.
  3. access:password:keyentry Specifies the password to be used to read/decrypt the in-memory key manager.

For more details see Bootstrap Configuration chapter.

Deprecated since trunk revision 16699.

The remote node contains broker specific parameters. When a broker is connected and registered to a FWD server it remains connected having an active session open. If for some reasons a broker is disconnected from server it will try to automatically reconnect to server, depending on the configured remote:retry:count and remote:retry:seconds parameters. Possible scenarios are server is shutdown or network errors. Parameter remote:retry:count defines the number of trials to reconnect having the default value set to 10. The value is decremented on each try and when it reached 0 the broker will exit. Parameter remote:retry:seconds defines the amount of time in seconds between each try having the default value set to 10. When a broker is terminated it is automatically unregistered from the server to which it was connected.

Brokers use the spawner tool in order to perform the remote launch. Parameter remote:spawner:file defines the location of the spawner tool's executable within machine's file system. The spawner tool is a native executable (binary program) that must be installed on the machine on which the broker runs and the broker must have sufficient OS permissions to access (and execute) it. For more details about installing and configuring the spawn tool see the Spawner Setup and Configuration chapter.

Since trunk revision 16699.

NOTE: The complete old-to-new key mapping, the failure modes of an unconverted file, and the rest of the upgrade procedure are in Broker Client Bootstrap Configuration.

<remote> category option was removed and <broker> and <client> category were used instead of <remote>.

<broker>
    <retry count="10" />
    <retry seconds="10" />
</broker>
<client>
    <spawner value="[location of the spawn binary file]" />
    <jvmArgs value="-Xmx1G -Djava.awt.headless=true -Djava.library.path=[path to p2j.jar] -Dfile.encoding=UTF-8 -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=[java debugger connection string]" />
    <classpath value="[classpath]" />
    <minAgentPort value="[min agent port]" />
    <maxAgentPort value="[max agent port]" />
    <web host="[the spawner ip address or host name]"/>
    <web dedicatedMode="[false|true]"/>
    <web defaultOsUser="[defaultOsUser]"/>
</client>

, where [java debugger connection string] can use this template dt_socket,address=[min agent port],server=y,suspend=n.
Two new parameters client:minAgentPort:value and client:maxAgentPort:value define java agents ports range were added to support remote debugging of spawned java clients. Each spawned client (by this broker configuration) got a unique debugger port from this range.

Server Configuration

On the server side the brokers are defined and configured inside the server directory. Each application server can have a list of brokers. Each broker on the list has a name and an optional list of account user names. The account attributes refer to user names for OS accounts which are used for remote spawning. These accounts must exist on the machine where the brokers (offering remote launch services for these users) reside. Brokers without a list of accounts (called generic brokers) are used when the OS user resolved for a remote spawn (i.e. specified by the user, for web clients, or configured at a FWD process) is not explicitly listed for any broker. In this case the generic brokers are used one by one until the remote spawn succeeds; if no generic broker can authenticate as the specified user, then the remote launch fails.

Brokers are defined inside the server directory by adding the following node to either /server/default or /server/<server-id> node. Default and explicit, per server, configurations are not merged.

<node class="container" name="brokers">
  <node class="broker" name="broker1">
    <node-attribute name="broker_account" value="broker1_process"/>
    <node-attribute name="host" value="192.168.100.28"/>
    <node-attribute name="account" value="os_account_1"/>
    <node-attribute name="account" value="os_account_2"/>
    <node class="container" name="portsRange">
      <node class="integer" name="from"><node-attribute name="value" value="7449"/></node>
      <node class="integer" name="to"><node-attribute name="value" value="7459"/></node>
    </node>
  </node>
  <node class="broker" name="broker2">
    <node-attribute name="broker_account" value="broker2_process"/>
    <node-attribute name="host" value="192.168.100.29"/>
  </node>
</node>

The attributes of a broker node (as defined in file dir_schema.xml) have the following usage:

node-attribute Type Multiple Mandatory Description
broker_account string no yes Name of the FWD process account this broker must authenticate as. Verified at registration.
host string no yes Host on which this broker runs. Must not resolve to the FWD server's loopback address - see Host Indexing and Ordering below.
account string yes no OS account user name used on remote spawn. A broker declared with no account attribute is a generic broker.

A broker node may additionally contain a portsRange container (from / to) giving the browser-facing embedded web server ports for clients spawned on that host. Reverse-proxy deployments require it on every broker; see Remote Web Client Spawning via Brokers.

Remote launchers are FWD clients using a server process account and digital certificates for authentication. Only process accounts can be used by brokers. The link between a broker and its process account is the broker_account attribute on the broker node, which names an account under /security/accounts/processes:

<node class="process" name="broker1_process">
  <node-attribute name="enabled" value="TRUE"/>
  <node-attribute name="server" value="FALSE"/>
  <node-attribute name="alias" value="broker1"/>
</node>

The alias attribute is the key store identifier of the public/private key pair this broker authenticates with; it must match security:keystore:processalias in the broker's client.xml.

IMPORTANT: broker_account and host are mandatory, and they are validated when the directory is loaded rather than when a broker registers. A directory carrying the earlier form — a broker attribute on a process account, or a broker node without these two attributes — fails to load, and the FWD server does not start. Existing directories must therefore be updated before upgrading; a stale broker definition is not ignored or skipped.

The two failures report as:

wrong attribute 'broker' in class 'process' node '<name>'
validation failed for class 'broker' node '<name>'

Process accounts used by brokers need to access server resources. For security reasons access to resources are protected and these broker accounts need special rights to have access to resources. This is done using ACL (Access Control List) settings inside the server directory. For more details see the Access Control Lists (ACLs) chapter.

When a broker is registered, the FWD user account is checked to be a process account, and to be the one named by the matching broker node's broker_account attribute. An ACL is needed for the broker to have access to the accounts resource. Something similar to the following node should be added to /security/acl/system node (at a minimum, the ACL number will probably need to be different in the actual entry):

 <node class="container" name="000500">
   <node class="strings" name="subjects">
     <node-attribute name="values" value="all_others"/>
   </node>
   <node class="systemRights" name="rights">
     <node-attribute name="check" value="true"/>
   </node>
   <node class="resource" name="resource-instance">
     <node-attribute name="reference" value="accounts"/>
     <node-attribute name="reftype" value="TRUE"/>
   </node>
 </node>

On the server side a broker services interface (API) is exported which allows communication between the server and broker. Brokers need to access exported services in order to register with the server and to allow the server to send commands down to the broker. In order to have access to the server's exported services, the following ACL node must be added to /security/acl/net node (at a minimum, the ACL number will probably need to be different in the actual entry):

<node class="container" name="002350">
  <node class="resource" name="resource-instance">
    <node-attribute name="reftype" value="TRUE"/>
    <node-attribute name="reference" value="com.goldencode.p2j.main.BrokerServerServices"/>
  </node>
  <node class="netRights" name="rights">
    <node-attribute name="permissions" value="'0101'B"/>
  </node>
  <node class="strings" name="subjects">
    <node-attribute name="values" value="all_others"/>
  </node>
</node>

NOTE: Some deployments carry an /security/acl/trustedspawner entry naming the broker. It is not enforced by the current engine and grants nothing; it may be removed. Authorization for broker spawning is the broker_account match described above, together with the spawn binary's own privileges for the target OS user.

Broker Startup and Registration

When the broker client process starts (BrokerCore.start(cfg)):

  1. initialize(cfg) reads the broker's client.xml (see Installation and Configuration above) and marks the process as a client (cfg.setServer(false)).
  2. SecurityManager.createInstance(cfg) loads the broker's key store and the server trust store.
  3. A retry loop (broker:retry:count × broker:retry:seconds) calls connect(sessMgr, cfg):
    1. sessMgr.connectDirect(...) opens a secure session to the server (TLS, on net:server:secure_port). The TLS handshake presents the broker's certificate; the server authenticates the session as the corresponding process account (program authentication).
    2. The broker obtains the server-exported BrokerServerServices proxy and calls registerBroker(...), passing dedicatedHost (client:web:host), associatedUser (client:web:defaultOsUser), dedicatedMode, the agent port range (client:minAgentPort / client:maxAgentPort), and debugger/JMX hints.
  4. On the server, BrokerManager.registerBroker(...):
    1. Requires that the authenticated subject is a process account — sm.isProcessDefined(sm.getUserId()). This is the check that consumes the accounts system ACL described under Server Configuration above; without that grant the lookup returns null and registration is rejected as if the account did not exist.
    2. Looks up the matching brokers/<name> entry by resolved host address (see Host Indexing and Ordering below) and verifies that the authenticated account equals broker_account. A mismatch throws and the registration is rejected.
    3. Builds a BrokerParameters record (random uuid, authenticated userId, agentAddress = host, associatedUser, dedicatedMode, a BrokerClientServices remote proxy back to the broker, and state = OPEN).
    4. Registers the host with the web clients manager, which pre-allocates that host's browser-facing client ports (and its agent ports) so later spawn requests can draw from them. See Remote Web Client Spawning via Brokers for how those resources are allocated and released.
  5. registerBroker returns the uuid; the broker keeps the session open and waits for spawn RPCs.

Multi-Broker Deployments

A single FWD server can drive several brokers — for horizontal scale-out (spawning web clients across many hosts) and/or to dedicate certain hosts to certain users.

Declaring More Than One Broker

Add additional <broker> nodes under server/default/brokers in directory.xml, each with its own broker_account, certificate/private-key, host, and portsRange. Each broker host runs its own broker-client process with its own broker*_client.xml (its own keystore / processalias). For example a second broker:

<node class="container" name="brokers">
  <node class="broker" name="broker1">
    <node-attribute name="broker_account" value="broker1_process"/>
    <node-attribute name="account" value="fwd"/>
    <node-attribute name="host" value="192.168.100.28"/>
    <node class="container" name="portsRange"> ... 7449..7459 ... </node>
  </node>
  <node class="broker" name="broker2">
    <node-attribute name="broker_account" value="broker2_process"/>
    <!-- no 'account' attribute => GENERIC broker (serves any user) -->
    <node-attribute name="host" value="192.168.100.29"/>
    <node class="container" name="portsRange"> ... </node>
  </node>
</node>

Each broker needs its matching process account, peers / private-keys certificate entries, and a brokers/<name> node whose broker_account names that process account — exactly as documented for broker1 under Reference configuration in Remote Web Client Spawning via Brokers. No trustedspawner grant is required; that ACL is not enforced by the current engine.

Host Indexing and Ordering

At registration each broker host is added to WebClientsManager with index broker.getIndex() + 1, where getIndex() is the broker's 1-based position in the brokers container (so localhost is index 1, the first declared broker is index 2, and so on). BrokerManager also guards against two brokers resolving to the same host. The web-client host index used by the browser side is therefore derived from this broker declaration order, not from any external host list.

IMPORTANT: A <broker> node must never be configured with host set to the FWD server's own loopback address (localhost or 127.0.0.1). Host index 1 is permanently reserved for the FWD server's loopback host. Were a broker declared on the loopback, it would be folded into that reserved index-1 slot at runtime while its clients were still assigned a mismatching declaration-order index in the reverse-proxy map — so the generated client-<index>-<port> proxy names would not match the runtime web-root, and that broker's clients could not be routed through the reverse proxy. Each broker must be declared on a distinct, non-loopback host.

NOTE: The engine enforces this constraint. At server startup, BrokerManager.initialize() rejects — with a SEVERE log — any broker whose host resolves to a loopback address, so such a broker is never registered and cannot spawn clients (a broker client that later connects claiming a loopback host is refused at registerBroker() with "No broker is defined for host …", since it was never added to the registry). The offline reverse-proxy map generator (ClientsToPortsGenerator.generateWithBrokerRanges()) skips the same broker while still consuming its declaration-order index, keeping the generated client-<index>-<port> map aligned with the runtime for the remaining brokers — see Remote Web Client Spawning via Brokers for the proxy map itself. Loopback detection is HostsManager.resolvesToLoopback(), which matches localhost, 127.0.0.0/8 and ::1. A violation therefore fails loudly at startup rather than as an unexplained reverse-proxy 404.

Host matching is performed on resolved addresses, not on the literal strings. When the server reads the brokers container at startup it resolves each broker's host attribute to its canonical address set (every address returned for the name), and at registerBroker it resolves the registering broker's reported dedicatedHost (client:web:host) the same way; a broker is matched when those address sets intersect (share at least one resolved address). Both resolutions are performed on the same FWD server host, so a given machine resolves to the same addresses on both sides regardless of how it is named — a literal host name, FQDN, short name, or IP all resolve consistently. The host attribute in directory.xml and the broker's own client:web:host therefore need not be written as the identical string; they only need to name the same machine as seen from the server. (This assumes the server's name resolution is stable between startup and broker registration, which holds because both run on that one server host.)

WARNING: The order of <broker> nodes in directory.xml is significant for host indexing. Keep it stable across deployments; reordering brokers shifts their indices.

Selecting a Broker for a Spawn Request

Selection is two-stage — first a candidate set by account, then load-based scheduling within it.

  1. Candidate set by account — BrokerManager.getBrokers(osUser) → getBrokersForUser(user):
    • A broker whose account list contains the user (broker.hasAccount(user)) is an account-specific broker for that user.
    • A broker with no account attribute (broker.noAccounts()) is a generic broker.
    • The server first tries account-specific brokers for the user; if none are registered/OPEN it falls back to generic brokers. (Account-specific spawns are attempted once; generic spawns may be retried across brokers.)
    • Only brokers currently in BrokerState.OPEN are considered.
  2. Load-based scheduling — BrokerManager.schedule(brokers, dedicatedUser, clientAddress):
    • Validates the requesting client's IP (InetAddress.getByName).
    • Computes the user's dedicated host set via getHostsForUser(dedicatedUser, ...). If the user is pinned to specific hosts, only brokers on those hosts are eligible; if the client's own address is among them it is preferred; otherwise any OPEN broker host is eligible (anyHost).
    • Among eligible brokers it picks the one reporting the lowest getRemote().getSystemLoading() (a live RPC to each broker). Throws if none are available.

Limitations

  1. Client working directory parameter read from server directory is platform specific. Processes are spawned on OS user's environments. By default the working directory will be set to user's home directory. Then, the spawner tool switches to the runtime resolved working directory for this client launch. As the working directory parameter is platform specific, all brokers with a certain name must run on the same OS. However using "." as the working directory is a valid directory name on both Windows and Linux. Using this approach, brokers are able to run on mixed platforms. Unfortunately the workingDirectory parameter is defined per server not per broker and the limitations are applied to all registered brokers regarding their names. If we have 3 brokers registered for broker1 running on 3 different machines and the working directory is set as absolute path like /opt/p2j/ for example all 3 brokers must run on the same OS family Linux, Unix. It is better to use a relative path for working directory like ./p2j/work_area because the client log files are created in the user working directory. With absolute paths the log files will be created in the same folder for all users. The same restriction is applied also in this case regarding OS family Linux, Unix. The working directory could be somethings like f:\p2j or .\p2j\work_area which are Windows OS specific. In this case the brokers must run on a Windows OS family and the working directory must exist on all machines where the brokers are running. The same rule is applied also to Linux/Unix; the working directory must exist on each machine where brokers are running. Using “.”, without a file separator, sets the working directory as the user's home directory since a the default current directory at login is the user's home directory. This is a cross platform folder name.
    When a process is spawned on an OS account, the working directory is first set to the user's home directory. This is done by spawner tool in C code. Then a change directory is made also by spawner in C code using the working directory parameter read from directory and sent as a command line argument.
    <node class="string" name="workingDir">
  <node-attribute name="value" value="." />
</node>

    For more details see Spawner Setup and Configuration chapter.
  2. For web clients the IP address of the embedded web server is the IP address of the machine where the web client has been spawned. When the web client is spawned remotely, the embedded web server selects reports its IP address and the browser client is redirected to this address. For this reason, the web browser must be able to reach this IP address (it must have a network route to this address or be on the same subnet as this address). The simplest way is to run on a machine from the same network/subnet.
  3. Batch clients are using an OS account to spawn batch processes. Directory settings are OS specific. On Windows we need systemUser and systemPassword while on Linux only systemUser is used. This OS account must exist on all machines on which we want to remote launch batch clients via brokers. OS account credentials are configured in node /server/runtime/<process-account>
    <node class="container" name="clientConfig">
  <node class="string" name="systemUser">
    <node-attribute name="value" value="username"/>
  </node>
  <node class="string" name="systemPassword">
    <node-attribute name="value" value="password"/>
  </node>
</node>

    For more details see Spawner Setup and Configuration chapter.

Missing in Implementation

System loading is used as a load-balancing parameter by the broker schedule algorithm. In the current implementation, load balancing only works on the Linux OS because the JVM provides enough information to properly detect the load. A pure Java solution which works on both Linux and Windows platforms is not yet available. A JNI solution will be designed and implemented in the future. A possible solution might be the use of SIGAR API by Hyperic.

Migration: Upgrading to the Trunk Revision 16699

Who Must Read This

This section describes every configuration change and every breaking change introduced by trunk revision 16699 (the merge of branch 9709e). Read it before you pick up that revision.

The release changes how brokers (remote launchers) and browser-facing web clients are configured. It does not change the 4GL-to-Java conversion pipeline, the generated Java, the runtime data types, the persistence layer or the database schema.

If your deployment... Impact Action
uses brokers / remote launchers (<node class="broker"> in directory.xml) Breaking. The FWD server will not start against an unmodified directory. Work through The Migration Checklist in full.
spawns web clients behind a reverse proxy (nginx, Apache, a load balancer) Breaking. Client route names changed; existing proxy rules stop matching. Sections Route Names Changed and Broker Port Ranges.
carries an acl/trustedspawner entry or a hosts.txt file Cleanup. Both are now dead configuration. Section Removed Configuration.
runs web clients only on the FWD server host, with no brokers and no proxy None. No action. Confirm you have no brokers container in directory.xml.
only converts 4GL to Java (no runtime deployment yet) None. No re-conversion is required. See What Is Not Affected.

IMPORTANT: The broker client and the FWD server must be upgraded together. The registration RPC (BrokerServerServices.registerBroker) gained parameters and the spawn-command placeholders were renamed in this release, so a pre-merge broker client cannot register with a post-merge server, and vice versa. Plan a coordinated restart of the server and every broker host.

Scope of the Release

The merge touches 19 files. They fall into four groups:

Group Files Customer-visible effect
Directory schema src/dir_schema.xml directory.xml must be updated before the server will start.
Broker / web-client runtime Broker, BrokerCore, BrokerDiscovery (new), BrokerManager, BrokerServerServices, ClientsToPortsGenerator, HostsManager, WebClientsManager Broker bootstrap keys, host indexing, port ranges, proxy route names.
Security ProcessAccount, SecurityCache, SecurityManager The broker attribute on a process account is gone; broker authorization moved to the broker node.
Configuration plumbing ConfigItem, BaseConfigItem (new) Broker-client bootstrap option names were re-categorised.
Container tooling tools/docker/*, tools/scripts/setup_ncurses6x.sh Optional; only if you build or run the FWD Docker images.

What Is Not Affected

Nothing in the conversion pipeline changed in this release. There are no changes to:

  • the TRPL rule sets under rules/ (convert/, annotations/, schema/, runtime/, fixups/);
  • the 4GL parser, preprocessor or schema processing;
  • p2j.cfg.xml, the conversion build.xml of an application project, or zfile_set.txt;
  • the generated Java, the runtime data types, BlockManager / TransactionManager, or the persistence layer;
  • the database schema, DMOs, or the DDL/UDF generation;
  • the CHUI, Swing or Web GUI rendering paths;
  • the Java source/target level (still 17).

You do not need to re-convert your application, re-generate DDL, or re-run the schema import for this release. A rebuild of the FWD jars and a redeploy of the runtime is sufficient.

Breaking Changes in Detail

1. Directory Schema: the broker and process Object Classes

This is the change that stops the server from starting, so handle it first.

The broker object class in dir_schema.xml changed from a leaf class with one optional attribute to a non-leaf class with two mandatory attributes:

Attribute Before After
account optional, multi-valued unchanged — optional, multi-valued
broker_account did not exist mandatory, single-valued
host did not exist mandatory, single-valued
(child nodes) not allowed (leaf="true") allowed (leaf="false") — this is what permits a portsRange container under a broker

At the same time, the broker attribute was removed from the process object class. Previously a broker was tied to its process account by putting broker="broker1" on the process account; that link now lives on the broker node as broker_account, pointing the other way.

IMPORTANT: These attributes are validated when the directory is loaded, not when a broker registers. A directory carrying the old form fails validation and the FWD server does not start. A stale broker definition is not ignored, skipped, or defaulted.

The failures look like this:

wrong attribute 'broker' in class 'process' node '<name>'
validation failed for class 'broker' node '<name>'

Before — the old form:

<node class="process" name="broker1_process">
  <node-attribute name="enabled"     value="TRUE"/>
  <node-attribute name="description" value="broker1_process"/>
  <node-attribute name="alias"       value="broker1"/>
  <node-attribute name="broker"      value="broker1"/>       <!-- REMOVE -->
</node>

<node class="container" name="brokers">
  <node class="broker" name="broker1">
    <node-attribute name="account" value="fwd"/>
  </node>
</node>

After — the new form:

<node class="process" name="broker1_process">
  <node-attribute name="enabled"     value="TRUE"/>
  <node-attribute name="description" value="broker1_process"/>
  <node-attribute name="alias"       value="broker1"/>
</node>

<node class="container" name="brokers">
  <node class="broker" name="broker1">
    <node-attribute name="broker_account" value="broker1_process"/>
    <node-attribute name="host"           value="192.168.100.28"/>
    <node-attribute name="account"        value="fwd"/>
    <node class="container" name="portsRange">
      <node class="integer" name="from"><node-attribute name="value" value="7449"/></node>
      <node class="integer" name="to"><node-attribute name="value" value="7459"/></node>
    </node>
  </node>
</node>

The brokers container lives under the server node, e.g. /server/default/brokers.

Authorization now runs off broker_account

At registration the server checks that the authenticated process account equals the matching broker node's broker_account. There is no longer a lookup from the process account to its broker. Two consequences:

  • A broker node with a broker_account that names no existing process account can never be registered.
  • Two brokers can no longer share one process account, because the account must match exactly one broker node's broker_account.
A broker may no longer be declared on the loopback host

Host index 1 is now permanently reserved for the FWD server's own loopback host. BrokerManager.initialize() rejects — with a SEVERE log — any broker whose host resolves to localhost, 127.0.0.0/8 or ::1, and the broker is never registered.

If your current directory declares a broker on localhost (a common single-host test setup), remove that broker node entirely and let the deployment run in local mode, or move the broker to a real, distinct, non-loopback address.

Two brokers may no longer resolve to the same host

Broker hosts are matched on resolved addresses, not on the literal string. If two <broker> nodes resolve to any common address, the second is rejected with a SEVERE log and ignored. A host name, an FQDN and an IP that all name the same machine are now the same host.

2. Broker Client Bootstrap Configuration: the remote Category Is Gone

The broker client's own bootstrap file (typically deploy/client/broker1_client.xml, or command-line overrides) used a remote category. That category was removed and its keys redistributed into the broker and client categories.

Old key New key Default
remote:retry:count broker:retry:count 10
remote:retry:seconds broker:retry:seconds 10
remote:spawner:file client:spawner:value ./spawn
remote:spawner:launchTimeout client:spawnerLaunchTimeout:value ClientSpawner.DEFAULT_TIMEOUT
remote:java:args client:jvmArgs:value the broker JVM's own input arguments
remote:java:classpath client:classpath:value the broker JVM's own classpath
remote:agent:host client:web:host the broker host's own name
remote:agent:user client:web:defaultOsUser ""
remote:agent:dedicatedMode client:web:dedicatedMode false
— (new in this release) client:minAgentPort:value 0
— (new in this release) client:maxAgentPort:value 0

WARNING: This change fails silently. client:mode:broker is unchanged, so a broker client with an old client.xml still starts and still enters broker mode — but every remote:* key is now an unrecognised category that is simply ignored, and the broker runs on defaults. In practice that means it looks for the spawner at ./spawn, inherits the broker JVM's own classpath and arguments, and reports its own local host name instead of the configured one. Registration then fails with "No broker is defined for host …", or spawns fail for reasons that do not point at the configuration. Do not rely on a startup error to catch this; convert the file.

Before:

<remote>
    <retry count="10" />
    <retry seconds="10" />
    <spawner file="/home/fwd/opt/hotel_gui/spawn" />
    <java classpath="/home/fwd/opt/hotel_gui/deploy/lib/p2j.jar" />
    <java args="-Xmx1G -Djava.awt.headless=true" />
    <agent host="192.168.100.28" />
    <agent user="fwd" />
    <agent dedicatedMode="false" />
</remote>

After:

<broker>
    <retry count="10" />
    <retry seconds="10" />
</broker>
<client>
    <spawner value="/home/fwd/opt/hotel_gui/spawn" />
    <classpath value="/home/fwd/opt/hotel_gui/deploy/lib/p2j.jar" />
    <jvmArgs value="-Xmx1G -Djava.awt.headless=true" />
    <minAgentPort value="9988" />
    <maxAgentPort value="9999" />
    <web host="192.168.100.28" />
    <web dedicatedMode="false" />
    <web defaultOsUser="fwd" />
</client>

Remember the three-level rule for these files: the first level is the category, the second is the group, and the group element's attributes are the keys. So client:spawner:value is written <client><spawner value="…"/></client>. Repeated group elements merge, which is why web appears three times above.

3. Reverse-Proxy Route Names Changed Format

The route name that identifies a spawned web client to an external proxy is generated by ClientsToPortsGenerator.getPortName(). The separator changed:

Before After
<namePrefix><hostIndex><offset> — e.g. client21 <namePrefix>-<hostIndex>-<offset> — e.g. client-2-1

The old un-hyphenated form was ambiguous (client211 — host 2 port 11, or host 21 port 1?). The new form is unambiguous, but every existing proxy rule stops matching, and the failure mode is a proxy 404 or 502 with no FWD-side error at all.

The nginx.conf example shipped in the FWD tree was written against the old form and must be regenerated too.

Both the running server and the offline generator derive the name the same way, so as long as you regenerate the map and rewrite the proxy rules from it, the two sides agree by construction.

4. Host Indexing Moved from hosts.txt to Directory Order

The hostIndex component of a route name used to come from a flat ./hosts.txt file, which HostsManager read at startup and appended to as brokers registered. That file is gone.

The host index is now derived entirely from directory.xml:

  • index 1 is permanently the FWD server's own loopback host;
  • every <broker> node consumes the next index in the order the nodes appear under the brokers container — the first broker is 2, the second 3, and so on;
  • a broker that declares no port range, or that is rejected as loopback, still consumes its slot, so the remaining indices stay aligned.

WARNING: The order of <broker> nodes in directory.xml is now significant and load-bearing. Adding, removing or reordering a broker renames every client route at and after that position. Fix the order when you migrate, and keep it stable thereafter.

Because the old indices came from an append-ordered file and the new ones come from declaration order, the indices your existing proxy map uses may not survive the upgrade even after you account for the hyphens. Regenerate the map rather than editing it by hand.

5. Each Broker Needs Its Own portsRange

Previously the global webClient/portsRange (from / to) supplied the browser-facing port range for every host, including broker hosts. Now each host carries its own range:

  • the FWD server's own loopback host (index 1) uses the global webClient/portsRange, as before;
  • each broker host uses its own brokers/<name>/portsRange, read relative to the broker node. The global range is not inherited.

Behind an external proxy this makes a per-broker portsRange mandatory. A broker without one is not "ports restricted", and the spawn fails in WebClientsManager.allocateClient() with:

Correct range: 'webClient/portsRange/from' < 'webClient/portsRange/to' must be provided

NOTE: That message names the global node even though the node actually missing is brokers/<name>/portsRange. Do not let it send you to the wrong place in the directory.

NOTE: A namePrefix node placed inside a broker's portsRange container is never read. namePrefix resolves only at webClient/portsRange/namePrefix (or clientConfig/portsRange/namePrefix) and is deployment-wide. This is harmless — uniqueness is carried by the host index — but do not expect a per-broker prefix to take effect.

6. ClientsToPortsGenerator Command Line Changed

Before After
Usage java …ClientsToPortsGenerator <directory.xml> <hosts.txt> java …ClientsToPortsGenerator <directory.xml> [host]
Second argument required — path to the hosts file optional — the host name/address that represents the local server in local mode; defaults to localhost, and is ignored in broker mode

The tool now picks its mode from the directory: if the brokers container holds any broker nodes it runs in broker mode (one entry per ports-restricted broker, host index from declaration order); otherwise it runs in local mode (a single host at index 1, using the global webClient/portsRange).

WARNING: An unchanged invocation is not rejected. ClientsToPortsGenerator directory.xml hosts.txt now parses hosts.txt as the host name, and in local mode it will happily emit backend lines reading hosts.txt:7449. Update every script and runbook that calls this tool.

7. Agent (Debug / JMX) Ports Are Now Per Broker

client:minAgentPort:value and client:maxAgentPort:value are read by the broker client and sent to the server at registration, so each broker host now has its own debug/JMX port pool. Each spawned client draws a distinct port from its host's range, and the literal address= in jvmArgs is rewritten per spawn.

Leave both at 0 to disable per-spawn agent ports. If jvmArgs carries -Xrunjdwp: or -Dcom.sun.management.jmxremote=true without a valid range, the single port parsed from jvmArgs is used as a fallback — which means only one client at a time can be debugged on that host.

A range whose upper bound exceeds 65535 is now rejected outright and logged, rather than attempting the allocation:

Agent ports range [<min>, <max>] for host '<host>' exceeds the maximum valid port 65535 and is ignored.

Removed Configuration — Safe to Delete

Item Status
./hosts.txt No longer read or written by any component. Delete it.
broker attribute on a <node class="process"> Removed from the schema — must be deleted or the directory fails to load.
<remote> category in the broker client bootstrap file Silently ignored — must be converted (see section 2).
acl/trustedspawner entries Not enforced by the current engine and grant nothing. Safe to remove. Broker spawn authorization is the broker_account match plus the spawn binary's own setuid privileges.
namePrefix under a brokers/<name>/portsRange container Never read. Harmless, but misleading — remove it.

ACLs That Are Still Required

The migration does not remove any ACL requirement. The broker's process account still needs:

  • read access to system/accounts (/security/acl/system) — registerBroker() confirms the registering session belongs to a process account, and without this grant the lookup returns nothing and registration is rejected with "The broker client must be authenticated using a process account." even though the account is valid;
  • access to the exported BrokerServerServices interface (/security/acl/net).

See Server Configuration earlier in this chapter for the exact nodes.

The Migration Checklist

Perform these steps with the FWD server and all brokers stopped. Steps 1–7 are configuration edits; nothing takes effect until the restart in step 10.

  1. Take a backup. Copy directory.xml, every broker client bootstrap file, your proxy configuration, and the current map.clients-to-backends. The directory edits below are not reversible by the engine.
  2. Build and stage the new engine. Build the merged trunk revision (./gradlew all) and stage the new jars where your deployment expects them. No re-conversion of the application is needed.
  3. Edit directory.xml — process accounts. Delete every broker attribute from every <node class="process"> node. Note which broker name each one referenced; you need it in the next step.
  4. Edit directory.xml — broker nodes. For each <node class="broker">: add broker_account naming its process account, add host naming the machine it runs on, and add a portsRange child container with from and to if this broker serves web clients behind a proxy. Remove any broker declared on localhost / 127.0.0.1; remove any duplicate that resolves to a host another broker already claims. Fix the order of the broker nodes now — it determines the host indices.
  5. Edit each broker client bootstrap file. Replace the <remote> block with the <broker> and <client> blocks shown in section 2. Check every value carries over: spawner path, classpath, JVM arguments, host, OS user, dedicated mode, retry count and interval.
  6. Delete the dead configuration. Remove hosts.txt and any acl/trustedspawner entries.
  7. Update your scripts. Any runbook, cron job or deployment script that invokes ClientsToPortsGenerator with a hosts.txt argument must drop it.
  8. Regenerate the proxy map. With the directory final, run:
    java com.goldencode.p2j.main.ClientsToPortsGenerator <directory.xml> [host]

9. Rewrite the proxy rules from the regenerated map. Every route name now carries hyphens and may carry a different host index. Build the rules from the generated map.clients-to-backends — do not patch the old ones. Keep the map file next to the proxy configuration it feeds.

10. Restart everything together. Start the FWD server first, then every broker host. A mixed old/new pair cannot register.

11. Verify. Work through Verification below.

Verification

  1. The server starts. A directory-validation failure at startup means step 3 or 4 is incomplete — look for wrong attribute 'broker' or validation failed for class 'broker' in the output.
  2. No SEVERE broker messages in the server log. Check for a rejected loopback broker, a broker with no host, or a host collision (see Error Messages).
  3. Every broker registers. Each broker client should log "Broker has been registered uid=…". A broker that keeps retrying has either not been converted (step 5) or does not match a broker node's host / broker_account.
  4. A web client spawns. Log in through the public proxy name. The browser must be redirected to https://<public-name>/<proxyPathSegment>/<namePrefix>-<hostIndex>-<offset>/ — never to an internal host:port. A redirect to an internal address means no forwarded host reached the server.
  5. The WebSocket upgrades on that same path. A 404 on a correct-looking path means the proxy map and the running server disagree — regenerate and reload.
  6. Check the generated map against the log. The route name the server builds and the name in map.clients-to-backends must be identical, character for character.

Error Messages and What They Mean

Message Cause Fix
wrong attribute 'broker' in class 'process' node '<name>' A process account still carries the removed broker attribute. Checklist step 3.
validation failed for class 'broker' node '<name>' A broker node is missing broker_account or host. Checklist step 4.
Broker '<name>' has no 'host' attribute and will be ignored. As above, caught at startup. Checklist step 4.
Broker '<name>' declares host '<host>' which resolves to the loopback address. A broker is declared on localhost / 127.0.0.1 / ::1. Remove the broker node, or move the broker to a distinct non-loopback host.
Brokers '<a>' and '<b>' both declare host '<host>'. Two broker nodes resolve to a common address. Give each broker a distinct host, or delete the duplicate node.
No brokers are defined for this P2J server. A broker client tried to register but the directory has no brokers container. Add the broker node, or stop running the broker client.
No broker is defined for host <host> The registering broker's client:web:host does not resolve to any declared broker's host — most often because the bootstrap file was not converted and the broker is reporting its own default host name. Checklist step 5; verify client:web:host.
The broker client must be authenticated using a process account. The session did not authenticate as a process account — usually the system/accounts ACL grant is missing. See ACLs That Are Still Required.
Account is not authorized to register this broker. The authenticated process account does not equal the broker node's broker_account. Correct broker_account, or the broker client's security:keystore:processalias.
Correct range: 'webClient/portsRange/from' < 'webClient/portsRange/to' must be provided A broker host has no portsRange. The message names the global node, but the missing one is brokers/<name>/portsRange. Checklist step 4.
Unregistered host: <host> / Add host: <host> A spawn was requested for a host that never registered. Confirm the broker registered and its host matches.
Host '<host>' is already registered at index <n>; ignoring re-registration with index <m>. Two registrations claim the same host with different indices — the deployment would be misaligned with the proxy map. Check for a duplicate broker node or a second broker client on the same host.
A proxy 404 or 502, with nothing logged on the FWD side The proxy map and the running server disagree — almost always a stale, un-regenerated map. Checklist steps 8 and 9.

Rollback

Roll back by restoring the backup taken in checklist step 1 and reverting the engine to the pre-merge revision. The two must move together: a restored old directory.xml will not load under the new engine, and a converted new directory.xml will not work under the old engine, because the old engine reads the broker attribute on the process account that the new form no longer has.

Keep the old and new directory.xml, broker bootstrap files, and proxy configuration side by side until the upgrade is verified.

Container Tooling (Optional)

These changes affect only deployments that build or run the FWD Docker images from tools/docker. They are additive; nothing existing is removed.

  • JDK 21 and JDK 25 are now supported alongside JDK 17. docker_build.sh determines the JDK from the distribution and rejects anything other than 17, 21 or 25.
  • Image names now carry a JDK suffix — …_jdk<version> — and PostgreSQL images additionally carry _pg<version>. Any script or compose file that pins an image name by its old form must be updated.
  • --pg= accepts a list. It now takes a single version, a comma-separated list (14,15,16,17), or all.
  • dbash.sh gained --jdk<21|25>, accepts comma-separated --manual_mount= and --manual_port= values, allows PGPORT to flow into the container, and no longer passes --rm when running detached.
  • New tool fbash.sh for handling FWD containers.
  • fwd_4.0_ubuntu_Dockerfile takes a DEPLOY_TYPE build argument, so the /opt/fwd symlink is no longer hard-wired to convert. This is what allows a runtime client image that carries no conversion jars.
  • setup_ncurses6x.sh resolves the target user's home directory from getent instead of assuming /home/<user>, and recognises the Linux Mint 20.x/21.x/22.x release names (ulyssa, wilma, zara, zena).

Reference

© 2004-2026 Golden Code Development Corporation. ALL RIGHTS RESERVED.

broker_registration.png (146 KB) Sergey Ivanovskiy, 08/07/2026 01:20 AM

broker_scheduling.png (121 KB) Sergey Ivanovskiy, 08/07/2026 01:20 AM