Stateless Server Clustering¶

• Stateless Server Clustering
  • Overview
    • Prerequisites for Customers
    • Solution for synchronization
  • Architecture Decisions
    • Topology: Peer-to-Peer
    • Region Configuration: REPLICATE + GLOBAL Scope
    • Delta Propagation
    • Network Partition Handling
  • Synchronized structures
    • Unique index tracking
    • Record locking
    • Metadata tables
    • Caches
    • Redis Directory
  • Directory configuration
  • Deployment/testing architecture
    • Meta Tables disabling
    • Starting Locator
    • Redis Directory
    • Docker infrastrucure
    • Graceful Shutdown

Overview¶

The goal of stateless FWD server clustering is to allow multiple FWD server instances to share the load for the same set of calling programs by synchronizing essential cross-session state between the JVMs. Each server operates independently for session-scoped data, but globally shared state (locks, unique index tracking, ...) is kept consistent across the cluster via Apache Geode.

Prerequisites for Customers¶

• For load balancing or high availability purposes, only application code that is truly stateless will be distributed across the cluster.
  • This means there can be no reliance on cross-session in-memory state between requests.
  • Stateful processing and long-lived sessions like interactive ChUI or GUI sessions are supported but their connections must be persistent. Those sessions stay with the specific FWD Application Server on which they were initially created.
• All FWD servers in the cluster must use the same converted application jar(s) and the same FWD version.
• By default, FWD runs in single-server mode with no extra configuration.

Solution for synchronization¶

Apache Geode was chosen as the distributed data grid technology. It is an open source solution which offers strong consistency.

Redis technology was used in order to implement centralized directory.

Architecture Decisions¶

Topology: Peer-to-Peer¶

Two Geode topologies were evaluated:

Aspect	Peer-to-Peer	Client/Server
Data storage	In FWD server JVM heap	Separate Geode server processes
Read performance	Excellent — data in local JVM	Network round-trip required
Additional infrastructure	Locator only (discovery)	Dedicated Geode servers + locator

Decision: Peer-to-peer topology was selected, primarily for superior read performance (data served from local JVM). With REPLICATE regions, data integrity is preserved even if an FWD server crashes. The implementation is designed to be topology-agnostic, allowing a switch to client/server mode if needed.

In order to connect to a geode cluster, a client/peer should connect ta a locator. Locator it a default node in geode that provide peer/client discovery.
Due to observed hardware performance variance across nodes, two locators are deployed to ensure high availability and eliminate a single point of failure. Default connection string to these locators is localhost[10334], localhost[10335]

Region Configuration: REPLICATE + GLOBAL Scope¶

• REPLICATE region type: Every member holds a full copy of all data. At least one server running guarantees no data loss.
• GLOBAL distribution scope: Distributed locks are acquired on region entries before propagating changes. Only one thread in the entire distributed system may modify a region entry at a time. This provides strong consistency without additional manual synchronization.

Some REPLICATE regions are used with scope DISTRIBUTED_ACK, (DistributedInMemoryLockManager because we implement our own distributed locking logic in place), other REPLICATE region use scope GLOBAL and we solely rely on Geode locking system.

Delta Propagation¶

For large objects with small changes (e.g., LockStatus), full object serialization on every update is wasteful. Geode's delta propagation was adopted:

• Classes implement Geode's Delta interface with toDelta()/fromDelta() methods.
• Only changed fields are serialized and sent to other members.
• Falls back to full object transfer if a member doesn't have the initial value.
• All operations still go through the put API, so consistency guarantees of GLOBAL scope are preserved.

Network Partition Handling¶

Definitions:

• Quorum: The minimum required "weight" or number of active members a distributed cluster must maintain to continue operating. It ensures that only one majority partition can make decisions during a cluster failure, avoiding conflicting updates.

• Split-Brain: A failure scenario during a network partition where the cluster splits into two or more independent, uncommunicative sub-clusters. Without safeguards, each side might continue to accept work, leading to divergent and silently conflicting data states. To implement this, each member in the cluster is assigned a weight. If a network partition is detected, the members evaluate whether they have sufficient quorum to continue operating by comparing the total weight of responsive members to the previous total. The side with insufficient quorum automatically shuts down, while only the partitioned side with a majority continues running. This strict behavior prevents a split-brain scenario.

Geode uses a weight-based quorum system:

• Each member is assigned a weight.
• On network partition detection, the side with insufficient quorum auto-shuts down.
• Only one side continues operating, preventing split-brain data corruption.

More details can be found here: https://geode.apache.org/docs/guide/114/managing/network_partitioning/how_network_partitioning_management_works.html

Synchronized structures¶

Unique index tracking¶

In order to track the unique indices in uncommitted transactions, the following items need to be synced:

• the maps from UniqueIndex(records, entries)
• the locks for protecting the critical section

In order to implement this, the exact instances for those were injected from the sub-classes(DistributedUniqueIndex and LocalUniqueIndex). In the local case, simple hash maps were injected, however in the distributed implementation Geode regions are being injected. As for the locks, for the local case, a simple ReentrantLock is used, however for the distributed case, an adapter over the Geode's DistributedLockService is used.

Record locking¶

Here mainly the distributed behaviour was achieved by extending the current InMemoryLockManager. In the sub-classes, the right implementation of maps were injected.
However, here, also some parts needed to be refactored in order to emulate the thread synchronization mechanisms:

• synchronized blocks replaced with explicit .lock()/.unlock() calls on distributed locks.
• wait()/notify() emulated using Geode CacheListener — the listener maintains a per-key monitor object; entries trigger notify() on the appropriate monitor.
• SessionToken now implements equals()/hashCode() — in a distributed environment, deserialized tokens are different object instances even for the same session. Identity comparison (==) was replaced with semantic equality.

Metadata tables¶

The synchronization for the metadata tables has not been implemented as a final decision on the approach has not been taken. There have been multiple discussions in #4369-55, #4369-66, #4369-68, #4369-72. An approach using Geode's structure is explained in #4369-100.

Caches¶

Even though the main caches don't need to be synced as their role is mainly performance, missing some invalidations can lead to functional problems. An example is FFCache, where invalidations are now propagated across the cluster. Here are some details:

• Each local FFCache invalidation triggers a Geode Function that executes the same invalidation type on all other members.
• A ThreadLocal boolean (isRemoteRequest) acts as a re-entrancy guard: set to true only in the Geode Function's execute method. The method that distributes invalidations checks this flag and only broadcasts if the invalidation originated locally — preventing an infinite broadcast loop. This works as Geode executes the functions on separate threads.

Redis Directory¶

RedisRemapper implements Remapper interface. RedisLockManager implements directory hierarchical locking mechanism. It is a centralized directory and all nodes should made reads and writes only from a single point of truth.

Data into redis is imported into redis from a input file in xml format (old directory.xml).

Changes to directory are persisted using versioning.
This implements the RedisSynchronizer, a thread that listen for redis updates using a RedisStream. RediStream ensure that the directory changes are always persisted, and reaches its listener. Once RedisSynchronizer has received a directory change it, saves the directory on the disk. This ensures that the directories does not get out of sync.

Directory configuration¶

For large production systems, Geode provides locators. Locators provide both discovery and load balancing services. You configure clients with a list of locator services and the locators maintain a dynamic list of member servers. A FWD server connectes to 2 locators (locator1 and locator2).

According to geode documentation and empiric observation, some nodes are more valuable to geode and their sudden shutdown is considered by geode as a split brain (which could lead to entire cluster shutdown).
For example in a multi nodes scenario, killing of a first created node (in geode terminology this node is called Coordinator) leads to cluster shutdown which is completely unacceptable. This is way, we had to ensure that 2 locators run at the same time.

Locators could be started from gfsh Geode utility: start locator --name=locator1 --port=10334, second locator is started with start locator --name=locator2 --port=10335 --locators=localhost[10334] in this way the second locator join the cluster.

Cluster mode is enabled via the FWD directory configuration:

Configuration	Location	Description
cluster-mode	server/default/cluster-mode	Enable/disable cluster mode (default: false)
geode/locatorHost1	server/default/geode/locatorHost1	Geode main locator hostname for peer discovery
geode/locatorPort1	server/default/geode/locatorPort1	Geode main locator port for peer discovery
geode/locatorHost2	server/default/geode/locatorHost2	Geode secondary locator hostname for peer discovery
geode/locatorPort2	server/default/geode/locatorPort2	Geode secondary locator port for peer discovery

Example config:

        <node class="boolean" name="cluster-mode">
          <node-attribute name="value" value="TRUE"/>
        </node>
        <node class="container" name="geode">
          <node class="string" name="locatorHost1">
            <node-attribute name="value" value="mainLocatorHost"/>
          </node>
          <node class="integer" name="locatorPort1">
            <node-attribute name="value" value="10334"/>
          </node>
          <node class="string" name="locatorHost2">
            <node-attribute name="value" value="secondaryLocatorHost"/>
          </node>
          <node class="integer" name="locatorPort2">
            <node-attribute name="value" value="10335"/>
          </node>
        </node>

Deployment/testing architecture¶

Meta Tables disabling¶

With the introduction of serverless clustering it was decided to not extent support of meta table to be distributed across cluster (at least for now). Since it is assumed that any business logic should not rely on any _meta tables content (at least for these that are changed at runtime) it introduces a few advantages and disadvantages.

Advantages:

- faster runtime, there is no need for metadata synchronization between threads/nodes.

Disadvantages:

- losses some 4gl metadata compatibility.

There is an option to disable the meta tables (by default it is enabled), in order to disable VSTs tables the following should be added under server/default:

<node class="boolean" name="lock-table">
  <node-attribute name="value" value="FALSE"/>
</node>
<node class="boolean" name="connection-table">
  <node-attribute name="value" value="FALSE"/>
</node>
<node class="boolean" name="transaction-table">
  <node-attribute name="value" value="FALSE"/>
</node>
<node class="boolean" name="user-table-stat">
  <node-attribute name="value" value="FALSE"/>
</node>
<node class="boolean" name="tenant-table">
  <node-attribute name="value" value="FALSE"/>
</node>

If the _meta tables are enabled they will work just per node (not cluster wise, for now).

Starting Locator¶

In order to start locator nodes, you should download Apache Geode (1.15.2) from https://geode.apache.org/releases/. Download the Binaries (TGZ).

Unzip the archive and go to apache-geode-1.15.2/bin. Run gfsh.

In order to start an Locator, you should run start locator --name=locator1 --port=10334. Please modify the port accordingly (keep in mind that the locators port should also be modified in directory.xml).

To start the second locator you need to run the following command start locator --name=locator2 --port=10335 --locators=localhost[10334]

Also, the following JVM arguments should be added to the server instance --add-exports=java.management/com.sun.jmx.remote.security=ALL-UNNAMED --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.time=ALL-UNNAMED --add-opens java.base/java.math=ALL-UNNAMED

Redis Directory¶

Install Redis sudo apt install redis -y.

If you run the server from docker container and redis on the host. You should edit /etc/redis/redis.conf, change the following line bind 127.0.0.1 -> bind 0.0.0.0

Node Bootstrap & Central Directory Configuration
This configuration initializes the primary node in the cluster, utilizing Redis as a centralized directory to coordinate all active nodes. The following configuration should be used to activate centralized directory.

<directory>
   <backend type="redis"/>
   <remap type="xml" />
   <xml filename="directory.xml"/>
   <redis host="redis_hostname" />
   <redis port="redis_port" />
   <redis forceLoad="true" />
</directory>

Operational Lifecycle

First Node Initialization (Seed): The initial cluster node uses the forceLoad="true" parameter. This instructs the node to read the local directory.xml file and seed the central Redis instance with the base directory data.

Subsequent Nodes: Follow-on nodes (does not need forceLoad="true" attribute) joining the cluster connect directly to the shared Redis directory and do not require the forceLoad parameter, preventing accidental overwrites of the live directory state.

Persistence & Versioning: To ensure durability, any runtime modifications made to the central directory are serialized back to the local disk in XML format (directory.xml). These updates are automatically tracked using a version number embedded within the XML tags.

Docker infrastrucure¶

If you want to start the application in docker environment, the patch from #11181-2 should be applied to server.sh script.

If you are running the server in docker, and the geode locators are running of local system then mainLocatorHost and secondaryLocatorHost must be the docker0 interface (172.17.0.1 in my case).

Graceful Shutdown¶

A critical requirement for clustering — when a container stops, the FWD server must disconnect from the Geode cluster cleanly (via JVM shutdown hooks). Without this, Geode detects a "network partition" and may shut down the entire cluster.

Forcefull kill of any node in the cluster in 2 locator setup will not kill the cluster entirely (Will continue to work). Forcefully killing all nodes will kill the cluster.

Currently, when the container is stopped the FWD server is not gracefully shut down. There are tasks #11157 and #11181 which should fix this.

---

© 2004-2026 Golden Code Development Corporation. ALL RIGHTS RESERVED.