Project

General

Profile

Branch 4369d: cluster lock manager — tests, changes and results

Summary

Branch 4369d replaces FWD's distributed lock manager and unique-key index in cluster mode with an owner-side design on Apache Geode 2.0.1: OwnerLockManager (record and table locks) and OwnerUniqueTracker (unique-key reservations). Every lock key has one owner, the primary of its Geode bucket, which decides alone; Geode's redundant copies provide failover.

Between 2026-10-22 and 2026-10-26 the branch was tested with converted 4GL code on real FWD servers, locally and on two hosts with up to 51 FWD servers, plus a bare-Geode harness and targeted correctness tests.

  • Correctness: about 145 million lock holds and 38 million transactions were checked on two hosts with no exclusivity violation and no duplicate unique key. All 33 functional scenarios pass in cluster mode.
  • Defects found and fixed: two in the branch's first version (a leaked table lock, unique keys released at the wrong block level) and six concurrency issues confirmed with deterministic tests (duplicate session ids across servers, a ghost-lock window, a stuck queue entry, a table-lock hole during a table EXCLUSIVE request, queue state shared between databases, a late lease revoke). The fixes were reviewed and approved after two further fix rounds.
  • Performance, leak and robustness fixes (2026-10-26, after the reference run): a lock manager leaked per stateless request (about one per transaction, never freed), now one per database; re-acquiring a held lock answered locally; cached liveness checks; the admin Record Locks view in one call per server; bounded, non-blocking lease give-backs; no lease revoke for a NO-WAIT table EXCLUSIVE request; CTRL-C effective at once while waiting on a remote owner; conserve-sockets=true by default; a concurrency-checks setting for the unique-key regions. Each was shown with a test before and after; the review asked for one fix round (three important findings), the re-review approved it with five minor findings, which are parked.
  • Reference run: the final build with all recommended settings, 25 + 25 servers (51 after a join), 6,000 users, kills, restarts, a join and three rebalances: all judged checks PASS, 10.5 M transactions, 39 M lock holds, no error outside a killed server.
  • Performance (reference run): a lock call to its owner takes 0.77 ms (p50) and 3.1 ms (p99) in steady load; 2,940 transactions/s at p50 38 ms, p99 94 ms. The lock logic itself is 2.5 % of a server's CPU.
  • Rebalance (reference run): after a restart or a join, one rebalance (bucket-count load probe, 8 threads, 30 s after the event) takes 2.9–3.5 s with 51 servers; lock-call p99 during it is 3.6–5.1 ms, throughput 98–100 %. Geode never rebalances by itself; a coordinator is recommended.
  • Failures: a killed server is detected in about 1.2–1.5 s with a 1000 ms member timeout; its locks are freed and its buckets fail over without loss.
  • All FWD changes since r16798 are uncommitted in the branch's working tree. The reference run used the build before the 2026-10-26 performance and leak fixes (md5 5b1bfd9c…); the current build (md5 c160cf2b…) has passed the local regression but not yet the two-host load test.

Test environment

Item Value
Local machine 3 FWD servers + 2 Geode locators, user-space PostgreSQL (port 5440)
Host cust004 192.168.1.83, 64 cores, 503 GB RAM, Ubuntu 24.04, PostgreSQL 16 (user-space instance, port 5440)
Host cust007 192.168.1.86, 128 cores, 503 GB RAM
Java OpenJDK 17, ZGC
Geode 2.0.1 (the branch's jars)
FWD server (load runs) 6 GB heap (12 GB in the first 4 + 4 run), 16 appserver sessions, 50 tenants (one database each) + landlord + default database
Geode ports locators 10334/10335, members 20000–20099 (direct) and 21000–21099 (membership), inside the firewall rules of cust004

Test tools

All test tools are in locktest/ (unversioned).

FWD 4GL scenario suite

A converted 4GL project (locktest/fwd/: schema lt with tables lrec, ukey, bulk and the multi-tenant mt; 4GL programs), a runner (locktest/bin/fwdtest.sh) and a Java driver (locktest/src/com/goldencode/lt/fwd/).

The runner starts FWD servers in four modes:
  • single — one server with FWD's local lock manager; the reference (oracle) for every scenario;
  • cluster — three servers with the branch's classes;
  • legacy — three servers with FWD's previous distributed manager; the "before" picture and the check whether a cluster-only failure comes from the branch;
  • negative — deliberately broken node-local classes, to prove the checks can fail.

The runner also kills, restarts, freezes (SIGSTOP/SIGCONT) and rebalances servers, and sets the member timeout, the lock regions' concurrency checks and the FastFindCache switch.

The 4GL actors are stateless: every call is a session-free appserver request that runs one transaction script (hold.p), with ready/await stages released through signal files, so a holder keeps its locks while other actors probe. A, B and C below are actors on server 1, 2 and 3.

Record locks

Id Steps and expected result
L1 A EXCLUSIVE r1 · B NO-WAIT → LOCKED · B waits · A commits → B still waits (A's lock downgraded to SHARE while its buffer is in scope) · A releases the buffer → B granted
L2 A and B SHARE on r2 → both OK, nobody blocked · B NO-WAIT EXCLUSIVE → LOCKED
L3 A and B SHARE on r3 · A upgrades to EXCLUSIVE → waits · B releases → A granted
L4 A EXCLUSIVE r4 and changes it · B NO-LOCK read → not blocked · A commits → B reads the new value
L5 A EXCLUSIVE r5, changes it, rolls back → B EXCLUSIVE granted, value unchanged
L6 A EXCLUSIVE r6 · B and C wait · A releases → exactly one is granted, the other still waits · its release → the other granted
L7 A EXCLUSIVE on 50 records spread over the id range (owners on all three servers) · B NO-WAIT on each → all LOCKED · A releases → all OK
L8 A EXCLUSIVE r8 · B waits with a short STOP-AFTER → TIMEOUT · B's session still answers · no table or record lock left behind for B
L9 A FOR EACH ... EXCLUSIVE-LOCK over 10 records · B NO-WAIT on each → LOCKED, on the next record → OK · A releases → all OK
L10 A OPEN QUERY + GET NEXT SHARE-LOCK · B SHARE → OK, NO-WAIT EXCLUSIVE → LOCKED · A closes the query → B EXCLUSIVE OK
L11 A NO-LOCK read · B changes the record · A FIND CURRENT ... EXCLUSIVE-LOCK → granted, CURRENT-CHANGED yes
L12 A DO FOR block: EXCLUSIVE, leave the block (transaction still open) · B NO-WAIT → LOCKED until A commits, then OK
L13 A EXCLUSIVE + RELEASE inside the transaction · B NO-WAIT → LOCKED until A commits, then OK

Unique keys

Id Steps and expected result
U1 A creates key K1 · B creates K1 → 132 · A rolls back → B creates K1 → OK
U2 A creates K2 in one committed transaction · B creates K2 → 132
U3 committed row OLD · A changes OLD→NEW · B creates NEW → 132, OLD → 132 · A commits → OLD OK, NEW 132
U4 committed row DEL · A deletes it · B creates DEL → 132 · A commits → OK
U5 two different keys of the same customer → both OK
U6 A creates SUB in an undone sub-transaction → B creates SUB → OK · kept sub-transaction → 132
U7 20 rounds: A and B create the same new key at the same moment → exactly one OK, one 132; one row per key
U8 two unique indexes: collisions on either index → 132; a failed create leaves nothing reserved
U9 A deletes 5 committed rows with FOR EACH ... DELETE (uncommitted) · B creates one key → 132 · A commits → OK
U10 change of a two-field key: old and new values → 132 until A commits, then the old one is free
U11 undone sub-transaction rekey and delete leave the original reservations correct

Table locks, _Lock, departures, rebalance

Id Steps and expected result
T1 B EXCLUSIVE on a bulk record · A deletes all bulk records in a FOR EACH ... DELETE loop → waits at that record · B releases → A completes, table empty
T2 A's delete loop in an open transaction · B NO-WAIT on a deleted record → LOCKED (deleted records keep their locks until commit) · A commits → NOTFOUND
S1 A EXCLUSIVE · _Lock shows one row, EXCLUSIVE, holder A · A releases → no row
D1 a client (separate JVM) holds lrec 500 EXCLUSIVE and creates a key in a request, and is killed with kill -9 · B's waiting request is granted no later than the killed request's time cap · the key is gone · the server serves the next request
D2 A EXCLUSIVE + key · B waits · kill -9 of A's server → B granted after the departure (waited until the kill) · A's key free · the server restarted, L7 again
D3 A EXCLUSIVE on L7's 50 records · B waits on one · kill -9 of an owner server → all other locks still held (backups took over) · A releases → B granted · all free
R1 A EXCLUSIVE on 50 records + 10 keys · B waits · server 3 restarted (it owns nothing) · rebalance moves lock primaries onto it (judged: at least one moved) → all locks still held, all keys still 132 · A releases → B granted
R2 A EXCLUSIVE · B waits · server frozen for half the member timeout → nothing lost, probes wait for the thaw · frozen for 4 × the timeout + 4 s → the server is removed, its locks are freed only after its removal, the survivors stay consistent
R3 6 traffic loops (2 per server) lock random records of a set of 200 for 20 ms each, 3 sessions wait on records held by a holder · server 3 restarted and a rebalance run under traffic → judged: no two holders at the same time, no errors, the waiters granted, traffic completed during the rebalance, lock primaries moved

Each cluster run has 1,089 checks: about 330 behaviour checks and about 760 clean-up checks (no lock or key left behind after each scenario), plus a run-level check that every server departure matches a kill, freeze or restart of the run. Hidden helper scenarios cover the export gate, leak detection, stale caches and a key-release regression test.

Two-host load test

The load test runs converted 4GL code on real FWD servers spread over two hosts, under a production-like multi-tenant load, while servers are killed, restarted, added and rebalanced. It answers two questions: do the locks and unique keys stay correct through all of that, and how long does a lock call take.

Tool: locktest/bin/fwdload.sh (preflight | deploy | db | start | calibrate | run | collect | report | stop | status), all settings in locktest/fwdload.conf (see How to run the tests).

Layout

Host What runs there
cust004 (64 cores) 25 FWD servers (node1–node25), Geode locator 1, the "rebalancer" (a small Geode member that holds no data and starts the rebalances), PostgreSQL 16 with all databases, the load driver for half of the users
cust007 (128 cores) 25 FWD servers (node26–node50), the server that joins during the run (node51), Geode locator 2, the load driver for the other half of the users

All servers form one Geode cluster across the two hosts. Every database has its two regions (locks, unique keys), 113 buckets each, one primary and one backup copy per bucket on two different servers. The primaries are spread over all servers of both hosts, so about half of all lock calls go to a server on the other host.

Application and data

  • The 4GL test project lt, converted with the branch's FWD. Databases: the default database lt, the landlord database and 50 tenant databases (one per tenant), 52 in total, all in the one PostgreSQL instance.
  • The table mt is multi-tenant: 100,000 records per tenant, two unique indexes. Its record locks and unique keys live in the tenant's own regions, so the load uses all 102 regions.
  • Each request runs loadhold.p through the server's appserver: it binds the session to the request's tenant, then runs hold.p, which executes a short script of steps (FIND with SHARE-LOCK or EXCLUSIVE-LOCK, pause, CREATE, DELETE, UNDO) as one 4GL transaction and returns the server's time stamps of every step.

Virtual users

  • 6,000 stateless users, 3,000 per host, 120 per server. A user thinks 2 s, sends one request (one transaction) to its server and waits for the answer; nothing is kept in the session between requests.
  • Each server has 16 appserver sessions. The requests of its users queue for them, so a slow server shows up as latency, measured from the moment the request was due, not from when a session picked it up.
  • Each request picks a random tenant, so every server works in all 50 tenants.

Transaction mix

Type Share (reference run) What it does
lock-excl 66 % 1–8 distinct mt records, locked in ascending id order (so no record-lock deadlock), each SHARE or EXCLUSIVE with 50 % chance, at least one EXCLUSIVE; hold 20 ms; commit
lock-share 10 % the same with all records SHARE
lock-upgrade 4 % the same, then the first record is upgraded from SHARE to EXCLUSIVE; two upgraders of the same record deadlock, which a 10 s STOP-AFTER breaks (counted apart, not as an error)
unique-create 9 % creates an mt record with new unique keys
unique-change 8 % deletes a record and creates it with another id and code (release old keys, reserve new ones)
unique-delete 1 % deletes a record created earlier
unique-hot 2 % create or delete on 20 hot keys per tenant that both hosts fight over; "already exists" (error 132) and "not found" are legal answers
  • A hot set of 1,000 of the 100,000 records per tenant gets 5 % of the record choices, so some records are contended between servers and hosts.
  • 5 % of all transactions end with UNDO (rollback), and another 5 % also run a sub-transaction that creates a key and is undone.
  • In the reference run this was about 2,940 transactions/s, 14,000 lock acquisitions/s and 11,500 releases/s.

Timeline of a main run

Phase Duration (reference run) What happens
Start and session warm-up 12 min all servers start; every server is warmed up in all 50 tenants on all its sessions before it gets users
Ramp 2.5 min users added gradually up to 6,000
Warm-up 5 min full load, not judged
Steady 5 min the baseline
Kill on cust007 1 s kill -9 of one server; its users pause, their in-flight requests fail (expected)
Restart, rebalance, warm-up 15 min the server is restarted at once; 30 s after it is ready one rebalance; the server warms up all tenants (about 12 min), then its users come back; 150 s to settle
Kill on cust004 15 min the same on the other host
Join 12 min a new server (node51) starts on cust007; 30 s after it is ready one rebalance; after its warm-up it gets users
Final steady 5 min the second baseline, with 51 servers
End checks and collection 10 min load stopped, region contents read, logs summarized on each host and collected
  • Calibration (optional, before the main run): the load is raised step by step (5 minutes per level) until the transaction p99 exceeds 1 s or an error occurs; the main run then uses the last level that held. The reference run used a fixed 6,000 users.

How correctness is judged

Check How Fails when
No exclusivity violation hold.p returns the server time when each lock was granted (end of the locking FIND) and released (end of the hold); the drivers store every interval; at the end all intervals of the same record from both hosts are compared, with the hosts' clocks aligned through the database (re-measured every 20 s) an EXCLUSIVE interval overlaps another interval of the same record by more than 3 ms
No duplicate or lost unique key the hot keys' final state in the database is compared with what the drivers committed; the servers' logs are searched for database unique-constraint errors any mismatch or any constraint error
No unexpected error every failed request is classified any error on a server that was not down at the time
No unexpected server removal the locators' membership views a server removed that the run did not kill
Recovery throughput in 10 s windows against the 120 s before the event throughput not back to 90 % within 60 s after the event
No leak after the load, the entries of all regions read through Geode's JMX manager any lock or unique-key entry left
Tenant regions at start regions of every server read right after start a tenant region missing on any server

What is measured

  • Transactions: throughput and latency (p50, p99, max) per window and per type.
  • Lock calls: every server writes latency histograms every 10 s (temporary instrumentation in the branch): the owner round trip of an acquire, by where the owner is (the same server, another server on the same host, the other host), the whole acquisition as the session sees it (waits included), releases and table-level calls. The report merges all servers per window (steady, each kill, each restart, each rebalance, the join, final steady).
  • Rebalances: duration, bucket and primary moves, primaries per server before and after.
  • Resources: CPU per host, GC per server and window, PostgreSQL connections, threads; Java Flight Recorder CPU profiles in separate runs.

Limits

  • The hot set is spread over 50 tenants, so contended hand-offs between hosts are relatively rare; a defect that breaks exclusivity only in a rare failover race could pass. There is no negative control (no run injects an overlap on purpose).
  • Only mt is used: table locks and leases, lrec and bulk operations are not driven by the load (they are covered by the scenario suite).
  • The leak check reads the regions only at the end.

Rebalance investigation

A bare-Geode harness (the branch's lock and unique classes without the FWD runtime), 20 nodes and 50 tenants (100 regions) on cust004, about 3,800 lock transactions per second with the unique-key workload and a violation checker. Each rebalance variant was run under this traffic. Findings: locktest/docs/2026-10-24-rebalance-spike-findings.md.

Correctness confirmation tests

locktest/test/correctness.sh and CorrectnessTest.java start Geode members in separate JVMs and drive the lock manager directly. Races are forced with test hooks in the FWD code that are off unless the system property fwd.4369.test.table is set. Each case prints VERDICT <case> PRESENT or ABSENT.

Performance and leak tests

locktest/test/perfleak.sh and PerfLeakTest.java work the same way (Geode members as separate JVMs on one host, verdict per case). The 14 default cases measure the performance and leak issues of 2026-10-26 before and after their fixes: tenant drop/re-init (leak), purge after the cache closed (purgeclose), liveness checks (livemembers), re-acquisitions (reacquire), the admin view (admin), a failover re-execution (failover), lease give-backs with an owner that does not answer (leaserelease, sweepburst, releasetimeout), NO-WAIT table EXCLUSIVE (nowaitrevoke, nowaitidle), CTRL-C during a remote wait (interrupt), an owner crash right after CTRL-C (ghostcrash) and a failed release (failedrelease). The tombstone counts are measured by the extra case tombstones (perfleak.sh tombstones, run once with each concurrency-checks setting).

Results

Scenario suite

Mode Result
Single PASS (1 known deviation)
Cluster PASS, 1,089 checks, 5 known deviations (all base FWD, see below)
Legacy (FWD's previous manager) 82 of 1,088 checks fail: undone unique keys stay reserved, a crashed or removed server's locks are never freed
Negative the broken classes are detected

Before the fixes described below, cluster mode failed one check (L8, a leaked table lock); the legacy comparison showed it came from the branch.

Two-host load test

What was tried

Each run changed one thing that the previous run had shown to be the main cost. All judged checks passed in every run.

Run Setup Transactions / lock holds checked What it showed What changed next
1 4 + 4 servers, 8,000 users, all events, 12 GB heap 9.06 M / 34.3 M 3,906 tx/s, p99 159 ms, but the live heap and the p99 grew over the run (p99 per 5 minutes from 128 to 914 ms): unique-key tombstones and session state 25 + 25 servers with 6 GB heap; the new defaults from the rebalance investigation (lazy buckets, 113 buckets, bucket-count probe, tenant regions at start)
Calibration 25 + 25 servers, FastFindCache on — 10,000 and 8,000 users failed (cust004, which also runs PostgreSQL, at 98.8 % CPU); 6,000 users held (p99 420 ms) 6,000 users for the event runs
2 25 + 25, 6,000 users, all events, FastFindCache on 10.31 M / 39.0 M lock call p99 14 ms in steady load; the CPU profile showed the FastFindCache broadcast as the main cost growing with the number of servers FastFindCache off
3 25 + 25, FastFindCache off, steady only 2.87 M / 11.4 M lock call p99 3.6 ms, CPU per transaction −33 %; still 4,243 threads per server (Geode's thread-owned connections) conserve-sockets=true
4 25 + 25, FastFindCache off + conserve-sockets=true, 6,000–10,000 users 5.53 M / 21.1 M lock call p99 3.1 ms, 1,019 threads per server; 8,000 users hold (p99 183 ms), at 10,000 cust004 saturates (p99 613 ms) the six concurrency fixes, final build
5 (reference) final build, all recommended settings, 25 + 25 (+1 joined), 6,000 users, all events 10.47 M / 39.0 M see below —

Steady phase, 50 servers, 6,000 users:

Measure FastFindCache on (run 2) Cache off (run 3) Cache off + conserve-sockets (run 4) Reference (run 5)
Transactions/s 2,889 2,936 2,936 2,938
Transaction p50 / p99 57 / 374 ms 40 / 102 ms 39 / 96 ms 39 / 94 ms
Lock call to the owner, p50 1.54 ms 0.77 ms 0.77 ms 0.77 ms
p90 5.1 ms 1.3 ms 1.3 ms 1.3 ms
p99 12.3 ms 3.6 ms 3.1 ms 3.1 ms
p99.9 41 ms 6.1 ms 5.1 ms 5.1 ms
CPU per transaction 37.9 core-ms 25.3 core-ms 23.7 core-ms —
cust004 CPU busy 92 % 63 % 58 % 56 %
Threads per FWD server 5,174 4,243 1,019 —
Largest GC pause 19 ms 4.6 ms 2.8 ms 2.6 ms

Reference run (run 5): final build, recommended settings

Build md5 5b1bfd9c66b02430c45e978f5488189d; 25 + 25 servers, a 51st joins during the run; 113 buckets, lazy bucket creation, bucket-count load probe, concurrency checks off on the lock regions, FastFindCache off, conserve-sockets=true, member timeout 1000 ms, rebalance with 8 threads 30 s after each restart or join, 50 tenants, 6 GB heap, 16 sessions per server; 6,000 users. Each setting was confirmed in the servers' logs and JVM properties. Results: locktest/results/fwdload-20261026_123644/report.txt.

Judged check Result
No exclusivity violation PASS: 0 in 39.0 M lock holds (19.5 M exclusive), largest overlap 0 ms
No duplicate or lost unique key PASS: 1,000 hot keys consistent with the database, 0 unique-constraint errors
No unexpected error PASS: 0; the only failed requests (74) went to a killed server while it was down
No unexpected server removal PASS: only the two kills, both seen as crashed members
Recovery after each kill, join and rebalance PASS (6 of 6): throughput 98–100 % during the event, 100–101 % within 60 s after it
No leak PASS: 0 lock or reservation entries left in the 102 regions after the load
Tenant regions at start PASS: 5,000 of 5,000 (5,100 of 5,100 after the join)

Latency per window (lock call = owner round trip of an acquire answered without waiting, over all owner locations; values are histogram bucket limits, the true value up to 20 % lower):

Window Lock call p50 p99 p99.9 Transaction p99 Throughput
Steady before the events (5 min) 0.77 ms 3.1 ms 5.1 ms 94 ms 2,938 tx/s
Kill of a server on cust007, first 30 s 0.90 ms 14 ms 1.6 s 1.8 s 95 %
Kill of a server on cust004, first 30 s 0.77 ms 12 ms 1.6 s 1.7 s 95 %
Kill, restart and warm-up, cust007 (12 min) 0.77 ms 3.6 ms 10 ms 120 ms 98 %
Kill, restart and warm-up, cust004 (12 min) 0.77 ms 3.1 ms 8.2 ms 112 ms 98 %
Rebalance 1 (after the cust007 restart) 0.77 ms 3.6 ms 6.1 ms 100 ms 99 %
Rebalance 2 (after the cust004 restart) 0.77 ms 4.1 ms 7.2 ms 104 ms 98 %
Rebalance 3 (after the join) 0.77 ms 5.1 ms 8.2 ms 107 ms 100 %
Join and warm-up of the new server (12 min) 0.77 ms 3.1 ms 6.1 ms 105 ms 100 %
Final steady (5 min) 0.77 ms 3.6 ms 7.2 ms 110 ms 2,938 tx/s
Whole run after the ramp (60 min) 0.77 ms 3.1 ms 7.2 ms 105 ms 2,912 tx/s
  • Owner location makes almost no difference: same server p50 0.51 / p99 2.0 ms (2 % of the calls), another server on the same host 0.77 / 3.1 ms (48 %), a server on the other host 0.77 / 3.1 ms (50 %). The spread between servers is small (per-server p99 2.6–3.6 ms).
  • Whole acquisition as the session sees it (table lock, waits and parks included): p50 0.90 ms, p99 4.1 ms; 0.08 % of the acquire attempts had to wait for another session's lock; 0.06 % of the acquisitions took 10 ms or more.
  • After a kill the calls whose bucket was owned by the killed server wait for the failover (member timeout 1000 ms): that is the 1.6 s p99.9 and the 1.7–1.8 s transaction p99 of the first 30 s; the other calls slow down only a little (p90 1.8–3.1 ms instead of 1.3 ms) and are back to normal over the whole restart window. Throughput over the first 30 s is 95 % of the steady level, the missing part being the killed server's own users.
  • Resources: cust004 (25 servers + PostgreSQL) 55 % CPU on average, cust007 (25–26 servers) 27 %; GC pauses at most 4.5 ms (0.002 % of the time); about 6,000 PostgreSQL connections (max 6,076).
  • A restarted or joined server is ready in 16–21 s but needs about 700 s of tenant warm-up before it gets its full share of users (see the base FWD findings).

For comparison, the bare-Geode harness measured an acquire p99 of 1.23 ms with 50 nodes.

Failure detection and its effect on lock acquisition

A server that dies plays two roles for the lock manager: it is the owner (bucket primary) of about 1/N of all lock keys, and its sessions are holders of locks owned by other servers. Until Geode's membership removes it, the cluster cannot tell a dead server from a slow one, so both roles wait for the failure detection; nothing else does.

Who What happens until the server is removed After removal
Lock calls for keys owned by the dead server (about 2 % of all calls with 50 servers) the call gets no answer and waits the backup copy becomes primary, the call is resent there and answered; no lock state is lost (the backup has it)
Sessions waiting for a lock that a session of the dead server holds they wait the branch frees the dead server's locks (0.04–0.13 s after the membership view) and the first waiter gets the lock
Lock calls for keys owned by live servers (about 98 %) answered normally —
The dead server's own users their requests fail (expected; not a lock effect) —
Example 1: the killed server holds a lock owned by another server. A session on server A holds an EXCLUSIVE lock on record 42; the lock entry lives on server B, the owner of record 42's bucket. A is killed (kill -9).
  1. From 0 to about 1.2 s: B still keeps A's lock on record 42; a session on server C that asks for record 42 waits. Until Geode removes A from the membership, B cannot tell a dead server from a slow one, and releasing early could give the lock away while A's session still uses it.
  2. At about 1.2–1.5 s (member timeout 1000 ms): Geode removes A. Every surviving server gets the "member crashed" event (LockMembershipHandler) and removes all locks and unique-key reservations of A from the entries it owns (OwnerLockManager.purgeDepartedMember, OwnerUniqueIndex.purgeDepartedMember), record 42 included; this takes 0.04–0.13 s.
  3. At about 1.25 s: the waiting session on C gets record 42 (measured: 1.25 s with a 1000 ms member timeout, 5.3 s with FWD's default 5000 ms).
Why this is safe and complete:
  • B did not fail, so the lock entry and its queue were never in danger; only A's own locks are removed.
  • A's transaction never committed: PostgreSQL rolls it back when A's database connections close, so record 42 is unchanged and can be given to the next session; A's uncommitted unique keys become free.
  • The purge runs a second time a few seconds later, for entries whose bucket moved to the server in the meantime. If a lock of A is still missed, the next request that would wait behind it removes it on the spot and decides again (logged as "lock … was held by a departed member; removed"), so a dead server's lock can never block anyone for good.
  • A lock request of A that arrives after A was removed is not granted, so it cannot leave a new lock behind.
  • Table leases held by A are dropped by their owners the same way.

Example 2: the killed server is the owner of a lock held by another server. A session on server B holds a lock whose entry lives on A, and A is killed. The lock is not lost: the backup copy of A's bucket on another server holds the same entry. When A is removed, that backup becomes primary with B's lock still in it; calls for that bucket wait the same 1.2–1.5 s and are then answered by the new owner.

All of this was tested with kill -9, where the operating system closes the dead process's connections at once. If a whole host is lost (power or network), the lock manager still frees the locks after the member timeout, but PostgreSQL notices the dead connections only through TCP keepalive, so its own row locks could stay longer (loss of a whole host is not covered by the tests).

So the failure detection time is the extra wait of the affected lock calls:

Member timeout Killed server detected after Waiting lock granted after Frozen server removed after
5000 ms (FWD default) 5.3–5.5 s 5.3 s 14.5 s
2000 ms 2.5 s 2.5 s 5.0 s
1000 ms 1.2–1.5 s 1.25 s 2.7 s

A frozen (hung, not dead) server takes about 2.5–3 times the member timeout to remove, because Geode first suspects it and then checks it again; lock calls for its keys wait that long.

Seen in the reference run (member timeout 1000 ms, 50 servers, 6,000 users), first 30 s after each kill:
  • lock-call p50 unchanged (0.77–0.90 ms): most calls are not affected;
  • 1.5–2.4 % of acquisitions took 10 ms or more (0.02 % in steady load): the share of calls owned by the killed server;
  • those calls waited about 1.6 s (lock-call p99.9 1.6 s), i.e. detection plus resend; transaction p99 1.7–1.8 s in the same 30 s;
  • no lock lost, no exclusivity violation, no failed request except those sent to the killed server;
  • throughput 95 % of the steady level (the killed server's users), 98 % over the whole restart window.

At 1000 ms there was no false removal, also with 51 members on two hosts under load (reference run: only the two kills were removed; the largest GC pause was 4.5 ms). A smaller member timeout shortens the wait but raises the risk of removing a healthy server during a long GC pause or a network hiccup; it should be checked on the production network.

CPU profile (50 servers, FastFindCache on)

Geode messaging was about 59 % of a server's Java CPU, the branch's lock code 2.5 %. The main per-transaction cost growing with the number of servers was FWD's FastFindCache: every record change sends a Java-serialized invalidation to all other servers. Turning it off produced the gains in the table above. Geode's thread-owned connections (conserve-sockets=false) created about 5,200 threads per server.

Rebalance

Background

The lock and unique-key state of every database lives in two Geode partitioned regions (<db>_owner_locks, <db>_unique_keys), each split into buckets with a primary and one backup copy. The primary's server answers all lock calls for that bucket. Geode restores lost backups by itself after a server leaves, but it never moves buckets or primaries to a new or restarted server: that needs a rebalance, which today is started only by an operator (gfsh rebalance) or a test.

Bucket creation (fill) modes

How the buckets are first created decides how much a later rebalance has to move.

Mode Behaviour Result
Eager (assignBuckets=true, the branch's original default) each server creates all buckets of a region as soon as it starts the first server up holds a copy of every bucket and all tombstones; in the investigation its heap filled (OutOfMemoryError with 1 GB, allocation stalls with 2 GB) and the cluster dropped to 400–1,400 transactions/s; rebalancing out of it under load ran at 6–13 moves/s (30–60 minutes)
Lazy (assignBuckets=false, new default) a bucket is created on its first key, on the least loaded server even when all servers are up before the load: lock primaries max/avg 1.08 → 1.02 after a 10 s rebalance
Late (harness only) buckets created once all members have joined perfectly balanced (max/avg 1.01)

With many tenants a second effect matters: FWD created a tenant database's regions on a server only at that tenant's first use there, so the first server that saw a tenant kept all its primaries. The branch now creates every tenant's regions on every server at start, from the landlord database (geode/tenantRegionsAtStart); in the 50-server run all 5,000 tenant regions were present on all servers right after start (5,100 on 51 servers after a join).

Methods tried

In the bare-Geode harness, all under lock traffic, 20 nodes, 100 regions, 113 buckets unless stated.

Method Result
gfsh rebalance (one region at a time) about 4.5 s per region; 200 regions took 15 minutes (harness, 50 nodes, 2026-10-22)
API rebalance, Geode's default size-based load probe chases the changing size of lock buckets: 23,737 moves planned for an already balanced cluster (balance got worse, 1.01 → 1.10), 24,430 after a restart (about 30 min)
API rebalance, bucket-count load probe balanced cluster: 23 moves in 2.2 s; after a restart: 1,100 moves + 392 primaries
8 resource-manager threads (gemfire.resource.manager.threads=8, API only) restart rebalance 65.9 s → 8.2 s (8 regions at a time); traffic 101 %, acquire p50 0.58 ms unchanged, p99 1.7 → 3.0 ms, no timeout or violation
Primaries only (restore redundancy with primary reassignment) 0.35–7.5 s, but a restarted server gets no primaries (it holds no copies yet)
1009 buckets (the original default) a restart needs 7,764 moves (estimated 1,100 s with 1 thread against 65.9 s with 113 buckets); the default probe's planning alone took over 541 s
Concurrency checks off on the lock regions (no tombstones) restart rebalance 4.5–5.2 s instead of 5.8–7.8 s; out of a skewed server 52 moves/s instead of 6–12; no ghost or lost lock in 4 provoked runs
Rebalance on every join (join ramp: 2 servers, 18 more join one by one under load) did not converge: about 13 moves/s whatever the number of coordinators (bound by the two source servers), Geode serializes concurrent rebalances per region
One rebalance after the last join (same ramp) worst: the first servers' heaps filled, cluster 800–1,400 transactions/s
One coordinator after a single restart or join into a balanced cluster works: 8.2 s, traffic unaffected

A rebalance does not interrupt lock acquisition when memory is not overloaded: in every such run throughput stayed at 99–101 % and acquire latency barely changed.

Recommended approach

From the methods tried, the combination that works:

Part Choice Why Status
Bucket creation lazy (ownerLocks/assignBuckets, ownerUnique/assignBuckets = false) no server holds a copy of every bucket; little to move later FWD default (changed in the branch)
Buckets per region 113 (ownerLocks/buckets, ownerUnique/buckets) with 1009 buckets a restart needs about 7 times as many moves (7,764 against 1,100) FWD default (changed in the branch)
Load probe bucket count (geode/loadProbe = bucketcount; FWD sets it before the Geode cache is created; an explicit -Dgemfire.ResourceManager.PR_LOAD_PROBE_CLASS wins; locators and other non-FWD members need the -D themselves) Geode's size probe chases the changing size of lock buckets and moves tens of thousands of buckets for nothing FWD default (changed in the branch)
Tenant regions created on every server at start, from the landlord database (geode/tenantRegionsAtStart) otherwise the first server that sees a tenant keeps all its primaries FWD default (changed in the branch)
Concurrency checks of the lock and unique-key regions on for now (geode/ownerLocks/concurrencyChecks, geode/ownerUnique/concurrencyChecks, both default true); off only after the targeted test below off removes the tombstones (faster moves, smaller heap, lock/release about 25–30 % faster locally) and was used for the lock regions in the reference run, but is safe only with one redundant copy and is unproven while Geode rebuilds a lost copy (see "Concurrency checks" below) settings; defaults unchanged
When to rebalance once, through the Geode API, about 30 s after the last restart or join has settled; not once per joining server concurrent rebalances are serialized per region and never converge during a join ramp test tool only (a small Geode member that holds no data); a coordinator in FWD is recommended
Rebalance threads 8 (gemfire.resource.manager.threads=8 on the member that starts the rebalance) 8 regions moved at a time: 65.9 s → 8.2 s in the harness test tool only

Results with the recommended approach (reference run)

Two-host load test, run 5: 25 + 25 servers under 6,000 users, 102 regions (51 databases × lock and unique-key region), 113 buckets each, one redundant copy. Each rebalance started 30 s after the restarted or joined server was ready.

Rebalance Trigger Duration Bucket moves Primaries moved Balance after (max/avg primaries per server) Lock call p50 / p99 during it Throughput
1 kill and restart of a server on cust007 3.5 s 410 174 1.07 0.77 / 3.6 ms 99 %
2 kill and restart of a server on cust004 3.4 s 408 176 1.08 0.77 / 4.1 ms 98 %
3 a new (51st) server joins on cust007 2.9 s 408 140 1.07 (unique-key regions) / 1.09 (lock regions) 0.77 / 5.1 ms 100 %
  • In steady load, without a rebalance, the lock call is 0.77 / 3.1 ms: a rebalance raises the p99 by 0.5–2 ms for about 3 s; p99.9 stays at 6–8 ms, and no request failed during a rebalance.
  • Before its rebalance a restarted or joined server holds no primaries (Geode does not move buckets to a new server by itself), so it takes no lock calls and nothing waits for it. After the rebalance it holds its share (the joined server: 106 and 112 primaries against an average of 113).
  • No region was ever without a host; after every rebalance all 51 lock regions had buckets on every server.
  • The new server has all tenant regions and is ready 16 s after its start; it then needs about 700 s of tenant warm-up before it gets its full share of users (base FWD, see below), independent of the rebalance.
Still to do:
  • a rebalance coordinator in FWD: one server (e.g. the oldest) starts an asynchronous rebalance through the API about 30 s after joins or restarts have settled, with 8 resource-manager threads and FWD's regions only;
  • until then, an operations procedure: start all servers before users connect, and run one rebalance after a restart or a join (as the test tool does);
  • before concurrency checks are turned off in production: a guard that refuses checks off with more than one redundant copy, and a test that releases locks while a new redundant copy is being built and then kills the primary (see "Concurrency checks" below).

Changes to FWD

All changes are in the branch's working tree on top of r16798, not committed. Current build build/lib/p2j.jar, md5 c160cf2bbf31419052f754bbbd7023d2.

Change Files Status
Leaked table SHARE lock after a STOP-AFTER-ended record-lock wait (L8): the table lock is restored on every exit that does not grant the record lock distributed/lock/OwnerLockManager.java reviewed, approved
Unique-key reservations registered at the wrong block level: UniqueTracker.Context.register ignored the depth it was given, so keys were not given back when their transaction ended (base FWD code, all trackers) persist/orm/UniqueTracker.java reviewed, approved
New defaults: lazy bucket creation, 113 buckets, bucket-count load probe OwnerLockManager, OwnerUniqueIndex, GeodeConfig, GeodeSettings reviewed, approved
Tenant regions at start new distributed/config/TenantRegions.java; TenantManager, Persistence, UniqueTracker, OwnerLockManager, OwnerUniqueIndex reviewed, approved
FastFindCache switch persistence/fast-find-cache (default on; off = no caching and no broadcast) persist/FastFindCache.java, distributed/cache/FastFindCacheInvalidationFunction.java tested
Six concurrency fixes (next section) OwnerLockFunction, OwnerLockManager, LockQueue, TableLeases, OwnerLockEntry, distributed/DistributedSequence.java reviewed, approved after two fix rounds
Performance, leak and robustness fixes of 2026-10-26 and the conserve-sockets default (section "Performance, leak and robustness fixes") OwnerLockManager, OwnerLockFunction, TableLeases, LockMembershipHandler, OwnerLockEntry, new distributed/lock/LiveMembers.java, OwnerUniqueIndex, OwnerUniqueFunction, GeodeConfig, GeodeSettings, persist/lock/LockManagerFactory.java reviewed; approved after one fix round, five minor findings parked
Lock-call latency histograms (geode/ownerLocks/histogramSec) and test hooks OwnerLockManager, OwnerLockFunction, TableLeases temporary, to be removed with the other [4369-locks] instrumentation

Concurrency issues: confirmation and fixes

Each issue was first shown with a deterministic test, then fixed, then re-tested.

# Issue Confirmation Fix After
1 Duplicate session ids across servers (the id sequence was atomic only within one JVM) 6,736 of 8,219 ids drawn more than once; two sessions with the same id and user both got EXCLUSIVE on one record the counter moved to a partitioned region with a cluster-wide atomic update 0 duplicates in 20,000
2 Ghost lock: a parked request granted to a session that had given up the session that gave up was recorded as the EXCLUSIVE holder every owner call carries an increasing number; a give-up is a fence, older requests of that session for that key are ignored absent
3 Queue entry left parked after an exception a request for the free key timed out after 3 s the owner unparks on any exception; the client sends CANCEL on any exception granted in 1 ms
4 Table locks during a table EXCLUSIVE request: another session could get table EXCLUSIVE while the requester held record locks, and a failed request lost the table SHARE both shown a new owner operation (CONVERT) turns the lease-held SHARE into the session's own SHARE on the owner; the EXCLUSIVE request is then an ordinary upgrade; the special lease drain path was removed both closed
5 Queue state shared between databases (same table and record id in two tenant databases) a free record waited 2 s behind another database's waiter; a cross-database wait cycle formed owner queues, parked registry and per-key locks keyed by region + key answered in 4 ms, no cycle
6 Late lease revoke a table SHARE waited 2.2 s without reason revokes carry the lease's identity; stale revokes ignored; draining cleared after a failed acquire 5 ms

Regression after the fixes: the 33-scenario cluster suite PASS (5 known), the key-release regression scenario PASS, smoke and cluster-start tests PASS, the bare harness's test list (465 s) ALL PASS, a contended load run with 0 violations.

Follow-up fixes after the reviews (each first reproduced with a new test, then fixed):

Review item Before Fix After
The session-id counter restarted at 0 when both copies of its bucket were lost ids 1 and 2 drawn twice while the counter's servers were killed counter back in the replicated region (survives while any server lives), each increment under a cluster-wide Geode lock 0 duplicates, also while the counter's servers are killed (about 2,700 ids/s)
A request parked on a former owner could be granted after the bucket's primary moved the bucket moved in 64 ms and the old primary granted the lock to a session that had given up every application runs inside Geode's primary lock of the bucket (computeWithPrimaryLocked); a server that is no longer primary drops its queue for the key and the client resends to the new primary absent
A CONVERT that failed after the owner applied it left the session's own table SHARE a late CONVERT left a SHARE that blocked another session's table EXCLUSIVE a fenced best-effort RELEASE on failure nothing left; the EXCLUSIVE is granted
Possible deadlock between the dead-server purge and a primary move (introduced by the previous fix) purge, request and move all stuck after 16 s one lock order on every path: bucket primary lock, then the key's stripe, then Geode's key lock; purge goes through the same path all finish in about 3 s
A bucket destroyed at the end of a move failed the call a RELEASE failed and the lock stayed held resent to the new primary when not yet applied the record is free
The primary check silently skipped if the Geode method were missing — the lock manager refuses to start in cluster mode (SEVERE) fail-safe

Final regression: all 16 confirmation cases absent (except the one that deliberately forces a duplicate id), the 33-scenario cluster suite PASS (5 known), the key-release regression scenario, smoke and cluster-start tests PASS, the bare harness's test list ALL PASS. Final build md5 5b1bfd9c66b02430c45e978f5488189d.

Known residuals (documented, low impact):
  • A request delayed more than 300 s (the lifetime of a give-up mark, ownerLocks/fenceKeep), or a bucket that moves away and back within about a second, can still produce a ghost lock.
  • A departed server's lock can stay in an entry when its bucket moved during both purge passes; the first request that would wait for it removes it (nobody is blocked, nothing granted twice), but it shows in the lock status and _Lock until then.
  • A lease's identity has millisecond resolution; a revoke arriving exactly while a lease is being acquired is dropped.
  • DistributedSecurityCacheGenerations has the same non-atomic counter update the session-id sequence had (outside this work).
  • The build of these fixes passed the one-hour two-host load test with kills, restarts, a join and rebalances (reference run, 0 violations in 39 M lock holds), but that load spreads its hot records over 50 tenants; a stress run with a concentrated hot set and many hand-offs during failovers was not done. The later performance and leak fixes have passed the local regression only.

Behaviour change to note: two sessions that both hold a table SHARE lock and both ask for table EXCLUSIVE now wait for each other until a timeout or STOP, as with FWD's single-server lock manager.

Performance, leak and robustness fixes (2026-10-26)

The tenant lock-manager leak was found in the reference run: node1's log showed 162,347 lock managers created in one hour, about one per transaction. With stateless requests a tenant's last session ends after nearly every request, FWD drops the tenant, and the next request initializes it again; every initialization created a new OwnerLockManager and added it to a static registry that was never pruned. The other items were open points of the design review. As for the concurrency issues, each was first shown with a test (perfleak.sh), then fixed and re-tested; the before figures come from the source before the fixes.

Item Before Fix After
Tenant lock-manager leak 20,000 tenant drop/re-init cycles: 20,000 managers registered, 20,000 start log lines, live heap +18.5 MB one manager per database, reused across re-initializations (OwnerLockManager.forDatabase, used by LockManagerFactory); the purge threads stop quietly when the cache is closing 1 manager, 1 log line, +2.3 MB; 0 warnings at shutdown (before: 12 in the test, about 67,000 at the end of the reference run)
Re-acquiring a lock of the type already held 2,000 owner calls for 1,000 SHARE + 1,000 EXCLUSIVE re-acquisitions, 386 / 160 µs each answered from the session's own record of its locks; upgrades, downgrades and releases still go to the owner; after a failed release the record is marked unverified and goes to the owner again 0 owner calls, 4 / 1 µs
Liveness checks built the set of all members per operation (O(N)); a request from a departed member was answered WAIT until its timeout, or forever 197 ns per check; the departed requester waited 3 s with no answer cached member view (new LiveMembers), a member missing from the cache rechecked against Geode's current view before acting; a departed requester gets an error 46–53 ns; error after 11 ms
Admin Record Locks view fetched every remote bucket on each refresh 3,000 record locks: 1,019 messages, 137 ms one function call per server and database (CollectFunction) 2 messages, 18 ms, same output
Tombstones no setting for the unique-key regions; tombstones not measured setting geode/ownerUnique/concurrencyChecks (default true); measured, see "Concurrency checks" below 20,000 releases leave 20,100 tombstones per copy with checks on, 0 with checks off
Lease give-backs could block a thread for about 90 s (one thread per member for all databases; give-backs also on session threads) owner not answering for 15 s: a session's release took 15,018 ms; an unrelated table EXCLUSIVE waited 14,708 ms give-backs on a fixed pool (ownerLocks/leaseReleaseThreads, 8) with one serial queue per owner server, failed ones resent as fences; the lease is blocked for local use until the give-back is settled (ownerLocks/leaseReleaseRetry, 30 s) 224 ms; 4 ms
A NO-WAIT table EXCLUSIVE request revoked every server's lease 1 revoke, 1 draining lease; another session's SHARE blocked 2 s a NO-WAIT request that cannot be granted at once sends only an "idle only" revoke, applied to leases nobody holds 0 draining leases; the SHARE took 29–95 ms
CTRL-C while waiting on a remotely owned key took effect only when the owner answered interrupt 100 ms into the first wait: 951–958 ms; during a later wait: 901–904 ms later waits on a helper thread (bounded, ownerLocks/callWaiters, 256), the session gives up at once and sends CANCEL; the first wait on a remote owner parks at most 100 ms (ownerLocks/firstParkRemote) 56–60 ms; 2 ms
conserve-sockets had to be set on the command line Geode's default false FWD sets it to true (geode/conserveSockets); an explicit -Dgemfire.conserve-sockets wins; locators do not need it true on every server of the cluster suite; -D…=false still wins

Review. An independent review asked for one fix round with three important findings, each reproduced with a new test before the fix:

Finding Before Fix After
After a CTRL-C the abandoned request could still be re-executed by Geode on a new owner when the old owner crashed, and grant a lock the session had given up (nothing would release it) ghostcrash: the new owner held the record for the session that gave up; another session timed out a parked ACQUIRE that Geode re-executes after its owner left is answered WAIT without being applied; a still-waiting session asks again entry empty, the other session granted
The give-back pool was unbounded: the idle sweep could start one thread per lease sweepburst (400 idle leases): 77 threads fixed pool fed by a serial queue per owner server 4 threads
After a failed RELEASE the local answer of a re-acquire could let two sessions hold the same record failedrelease: both sessions held it the record is unverified until the owner answers again the second session gets UNAVAILABLE

Seven minor findings were fixed in the same round, among them a table SHARE request now respects its lock timeout while a give-back is unsettled (9,533 ms → 999 ms with a 1 s timeout) and a NO-WAIT retry is no longer starved by idle leases (30 tries in 3 s → 2 tries in 115 ms). The re-review approved the round.

Regression after the round: the 14 performance and leak cases and the 16 concurrency cases as expected, the 33-scenario cluster suite PASS (5 known), the key-release regression scenario, smoke and cluster-start tests PASS, the bare harness's test list ALL PASS; every server writes its start line once and runs with conserve-sockets true.

Parked minor findings (open):
  • An owner that hangs without leaving the cluster turns its give-back queue into a chain of about 90 s per lease (callAttempts × callTimeout); fail fast suggested.
  • After a failed give-back the next lease request is not fenced, so a late RELEASE of the failed give-back can remove the new lease (existed before; a fenced lease request suggested).
  • The unverified mark is not applied to table locks held on the owner (no regression; table locks were answered locally before).
  • An idle-only revoke ignores leases the owner still lists after a failed give-back, so a NO-WAIT retry can still fail against them.
  • A table SHARE wait and the record wait can together take up to twice the lock timeout.
  • Not fixed by design: after a Geode auto-reconnect the lock manager keeps the old region and member id; its calls now fail with "left the cluster" instead of waiting. Recommended: disable-auto-reconnect=true with a supervisor that restarts the server.

Concurrency checks

Measured with two members: with checks on, 20,000 released locks leave 20,100 tombstones per copy and 20,000 released unique-key reservations 20,000 more (kept up to 8 minutes, until Geode's tombstone collection); with checks off, none, and the lock/release and reserve/release loops ran about 25 % and 30 % faster (one run each). A lock call applied on an owner that dies before answering and re-executed by Geode on the new owner is applied exactly once with checks on and off (failover), because the lock manager compares entry values and never relies on Geode's entry versions.

Verdict: checks off is safe with one redundant copy (the default) and not with two or more, where the surviving copies can differ after a failure and only the version checks reconcile them. It is unproven while Geode builds a new redundant copy (after a server loss or during a rebalance): a lock released during the copy could survive in the new copy, and after a failover to it come back as a lock nobody releases. The defaults stay on; before switching them off, a guard against more than one redundant copy and a test of that window are needed.

Findings outside the lock manager (base FWD)

  • FastFindCache in cluster mode broadcasts every record change to all other servers (see above); the largest performance cost at 50 servers.
  • Tenant warm-up: a restarted or new server needs about 700 s (14 s per tenant for 16 sessions) before it carries its full load; cause not yet isolated (connection pool of 2 per tenant database, tenant binding or a lock).
  • NullPointerException in RecordMeta.composeValidationSQLs:694 on the first concurrent unique-key validation after a server start.
  • Database connections grow as servers × tenant databases (6,076 PostgreSQL connections at 50 servers × 51 databases).
  • Threads: about 412 connection-pool helper threads per server (one pool per database).
  • Stale record caches across servers (known deviations F11, F21), _Lock shows only the local server's locks (F20), _Lock-Type is never written (F12), DELETE FROM deletes nothing (P3), servers ignore SIGTERM for 30 s.
  • FWD's previous unique index never frees the key of an undone create.
  • Tenant re-initialization per request: with stateless requests a tenant is dropped and initialized again after nearly every request (found through the lock-manager leak above). FWD's previous distributed lock manager (DistributedInMemoryLockManager) is also created at every initialization and adds a cache listener to the database's lock-table region that is never removed (from the code, to confirm at runtime); with many short sessions every lock-table update would run a growing number of listeners.

Recommendations

  1. Remove the temporary histograms, counters and test hooks, run the local regression and one two-host load run with the current build and the recommended configuration (it also confirms that the heap no longer grows), then commit the branch changes. The unique-key block-level fix and the tenant and FastFindCache changes touch base FWD code and should also be seen by a trunk reviewer and go through FWD's standard regression testing.
  2. Keep the new defaults: lazy buckets, 113 buckets, bucket-count load probe, tenant regions at start, one lock manager per database, conserve-sockets=true.
  3. Add a rebalance coordinator: one rebalance through the API, 8 resource-manager threads, about 30 s after a restart or join (see Rebalance); 2.9–3.5 s with 51 servers in the reference run.
  4. Fix the FastFindCache broadcast for cluster mode (batching, compact serialization or a per-table version), or run clustered servers with persistence/fast-find-cache=false (as in the reference run).
  5. Member timeout: start with 2000 ms in production, 1000 ms after checking it on the production network (no false removal at 1000 ms on the test hosts).
  6. Keep concurrency checks on (the default) until the guard and the copy-rebuild test of "Concurrency checks" are done.
  7. Run with Geode's disable-auto-reconnect=true and a supervisor that restarts a server that leaves the cluster; keep network partition detection on and two locators on different hosts.
  8. Decide on the parked minor findings; the unfenced lease request after a failed give-back and the unverified mark for table locks touch lock correctness and are small to fix.
  9. Start all servers before users connect; plan host capacity with headroom (the host also running PostgreSQL saturated first).
  10. Investigate the tenant warm-up time and the RecordMeta NullPointerException.

Not covered

  • Races that the fault-injection tests do not target, and a randomized stress test against a reference model.
  • A network split between hosts, loss of a whole host, redundancy 2, several permanent databases at once.
  • The 33 scenarios on the two hosts; the tenant scenarios (M1, M2) and a schema with many tables per tenant database.
  • Comparison with OpenEdge (lock grant order, message 121).
  • Long runs (the longest load run was about one hour).
  • Concurrency checks off while Geode builds a new redundant copy, followed by a failover (see "Concurrency checks").
  • The 2026-10-26 performance and leak fixes under the two-host load.

How to run the tests

All commands run on the development machine, from the branch directory (/home/es/gcd/branches/4369d). The README (locktest/README.md) has the same instructions.

Functional scenarios and concurrency tests (local, no hosts needed)

./gradlew jar                                       # FWD build: build/lib/p2j.jar
bash locktest/bin/fwdtest.sh build                  # the converted test project

# the 33 scenarios (also --mode=single, --mode=legacy); about 10 min in cluster mode
bash locktest/bin/fwdtest.sh start --mode=cluster
bash locktest/bin/fwdtest.sh run --mode=cluster
bash locktest/bin/fwdtest.sh stop

# the concurrency confirmation tests (each case prints VERDICT <case> PRESENT|ABSENT)
bash locktest/test/correctness.sh

# the performance and leak tests (14 cases; or name cases, e.g. perfleak.sh leak ghostcrash)
bash locktest/test/perfleak.sh

Results: locktest/results/fwd-<time>-<mode>/ (the RESULT: line of driver.log), locktest/results/correctness-<time>/ and locktest/results/perfleak-<time>/.

Two-host load test: prerequisites

1. Tunnels to both hosts (with your own login on the jump host, never -l mfg):

ssh www.goldencode.com -p 2222 -l es -NC -f -L2227:192.168.1.83:22   # cust004
ssh www.goldencode.com -p 2222 -l es -NC -f -L2228:192.168.1.86:22   # cust007

2. The ssh master connection to cust007. The scripts reach cust007 only through it; open it in your own terminal and type the password once:

ssh -o ControlMaster=yes -o ControlPath=/tmp/cm-mfg-cust007 -o ControlPersist=12h -fN -p 2228 mfg@localhost
ssh -o ControlPath=/tmp/cm-mfg-cust007 -O check -p 2228 mfg@localhost    # must say "Master running"

3. PostgreSQL 16 and the firewall rules on cust004 (admin, sudo). The firewall rules are not persistent and disappear on a reboot of cust004. Check:

ls /usr/lib/postgresql/16/bin/initdb
sudo iptables -S | grep 192.168.1.86

Expected: INPUT tcp 5440 from 192.168.1.86; INPUT and OUTPUT for 10334:10335, 20000:20099 and 21000:21099 (TCP, and UDP for 21000:21099) from and to 192.168.1.86; OUTPUT RELATED,ESTABLISHED. PostgreSQL 16 is installed from the Ubuntu packages without its system cluster (create_main_cluster = false in /etc/postgresql-common/createcluster.conf before apt-get install postgresql-16). A missing rule is added by the admin, e.g.:

sudo iptables -I INPUT -p tcp -s 192.168.1.86 --dport 5440 -j ACCEPT

4. The FWD build to test in build/lib/p2j.jar (./gradlew jar); the report records its md5.

Two-host load test: settings

All settings are in locktest/fwdload.conf; any of them can be overridden on the command line (e.g. SERVERS_H1=6 bash locktest/bin/fwdload.sh start). The recommended configuration:

Setting Value Meaning
SERVERS_H1, SERVERS_H2 25, 25 FWD servers on cust004 and cust007
BUCKETS 113 buckets of the lock and unique-key regions
LOAD_PROBE org.apache.geode.internal.cache.partitioned.BucketCountLoadProbe bucket-count load probe on every member
LOCK_CONCURRENCY_CHECKS false no tombstones in the lock regions
FFC false FastFindCache off
CONSERVE_SOCKETS true shared Geode connections
MEMBER_TIMEOUT_MS 1000 failure detection
REBALANCE_THREADS, REBALANCE_DELAY_SEC 8, 30 rebalance after a restart or join
TENANTS 50 tenant databases
SERVER_HEAP_MB, SESSIONS 6144, 16 per FWD server

The load mix (THINK_MS, HOLD_MS, MAX_RECORDS, SHARE_RATIO, UPGRADE_RATIO, HOT_RECORDS, HOT_RATIO, UNIQUE_RATIO, ROLLBACK_RATIO, SUBUNDO_RATIO), the timeline (RAMP_SEC, WARMUP_SEC, STEADY_SEC, RECOVER_SEC, FINAL_STEADY_SEC) and the calibration (CAL_USERS, CAL_DOWN, CAL_HOLD_SEC) are in the same file.

Two-host load test: run

cd /home/es/gcd/branches/4369d/locktest

bash bin/fwdload.sh preflight          # hosts, ssh master, Java, limits, free ports, PostgreSQL, firewall probes
bash bin/fwdload.sh deploy             # copies the build and the test project to ~/4369d/fwdload on both hosts
bash bin/fwdload.sh db init            # first time only, or to start fresh: lt + 50 tenant databases + landlord
bash bin/fwdload.sh start              # PostgreSQL, locators, all servers, rebalancer, configuration and tenant-region checks
bash bin/fwdload.sh calibrate          # optional (about 30 min): the user level that holds (p99 under 1 s, no errors)
bash bin/fwdload.sh run                # main run: load, kills, restarts, join, rebalances, judged checks
#    bash bin/fwdload.sh run --users=6000   (a fixed level instead of the calibrated one)
bash bin/fwdload.sh collect <RUNID>    # RUNID is printed by "run", e.g. fwdload-20261026_140000
bash bin/fwdload.sh report <RUNID>     # (re)analyses into results/<RUNID>/report.txt
bash bin/fwdload.sh stop               # stops everything on both hosts (recorded PIDs only)
bash bin/fwdload.sh status             # nothing should be alive afterwards

Exit codes: 0 OK / PASS, 2 a judged check failed, 3 needs the user (the cust007 master connection is gone or a host is unreachable), 1 another failure. Timing: start takes 5–15 minutes (tenant warm-up included), run about 85 minutes with 25 + 25 servers (the reference run: 12 minutes of session warm-up, 63 minutes of load, 10 minutes of checks and collection; a restarted or joined server warms up for about 12 minutes before it gets users).

Two-host load test: results

  • locktest/results/<RUNID>/report.txt: the judged checks (CHECK load/... PASS or FAIL), the numbers (INFO lines), the lock-call section per window and a final RESULT: line.
  • Raw summaries next to it: lockcalls-10s.tsv.gz, lockcalls-by-server.tsv.gz, gc-per-server.txt, rebalance-balance.txt, tenant-regions.tsv.
  • The full server logs stay on the hosts under ~/4369d/fwdload/fwd/; only a slim copy is collected.

Notes

  • On cust007 only ~/4369d is used; another application of the same user runs there and must not be touched.
  • After an interrupted run always run fwdload.sh stop and status.
  • To test another FWD build, rebuild and run deploy again.

References

  • locktest/docs/2026-10-23-fwd-4gl-scenarios-design.md — design of the scenario suite and the load test
  • locktest/docs/2026-10-23-fwd-spike-notes.md — FWD behaviour findings (P1–P9, F1–F25)
  • locktest/docs/2026-10-24-rebalance-spike-findings.md — rebalance investigation
  • locktest/.sdd/2026-10-23-fwd-4gl-scenarios-plan/fwdload-report.md — load test runs, lock-call tables, CPU analysis
  • locktest/.sdd/2026-10-23-fwd-4gl-scenarios-plan/correctness-report.md and correctness-review.md — concurrency issues
  • locktest/.sdd/2026-10-23-fwd-4gl-scenarios-plan/fwd-fixes-defaults-report.md — leaked table lock, key release, defaults
  • locktest/.sdd/2026-10-23-fwd-4gl-scenarios-plan/perf-leak-report.md and perf-leak-review.md — performance, leak and robustness fixes, their review and re-review
  • locktest/results/fwdload-20261026_123644/report.txt — the reference run (final build, recommended settings)
  • locktest/results/ — run results