Branch 4369d: two-host load test of the owner-side lock model (2026-10-22)¶

Test report and setup guide for the 50-node runs of branch 4369d (#4369) on two hosts, cust004 and cust007, done on 2026-10-22. Part 1 is for reviewers of the branch: what was tested, the numbers, the findings and what is still open. Part 2 is for whoever repeats the runs: hosts, ports, firewall, scripts, profiles, how to read the results and what went wrong on the way.

Companion documents in locktest/: README.md (how to use the harness), docs/2026-10-22-two-host-test-design.md (the approved design), docs/2026-10-22-two-host-test-plan.md (the implementation plan) and .sdd/2026-10-22-two-host-test-plan/progress.md (the ledger of every task, decision and run of the day).

Part 1: results and findings¶

1. Summary¶

The code under test is the branch tree at r16798 (no uncommitted source changes): OwnerLockManager, OwnerLockFunction, LockQueue, TableLeases and OwnerUniqueTracker / OwnerUniqueIndex / OwnerUniqueFunction on partitioned Geode 2.0.1 regions. On 2026-10-22 it was exercised by the locktest harness on a real Geode cluster of 50 member JVMs, 25 on each host, with two locators, no FWD server, no 4GL and no database. Every run passed every check:

profile	load	result	lock tx/s	unique tx/s	acquire p99	reserve p99	transaction p99
smoke2	1 + 1 nodes, 30 s	PASS	686	176	0.66 ms	0.98 ms	5.7 ms
prod50w	production-like, wide hot set, 180 s	PASS	4,922	n/a	17.7 ms	n/a	93 ms
prod50m	production-like, narrow hot set, 180 s	PASS	4,927	n/a	16.0 ms	n/a	89 ms
prod50w-u	prod50w + 20 % unique-key transactions, 180 s	PASS	3,942	981	22.3 ms	31.5 ms	91 ms
hot50-u	high contention + 30 % unique on 50 hot keys, 120 s	PASS	19,155	8,149	307 ms	211 ms	664 ms
prod50w-fast	prod50w with late buckets and ZGC, 120 s	PASS	4,933	n/a	1.17 ms	n/a	31 ms
prod50w-u-fast	prod50w-u with late buckets and ZGC, 120 s	PASS	3,951	985	1.23 ms	1.46 ms	31 ms

"Every check" means: 0 exclusivity violations, 0 errors, 0 stuck sessions, 0 lost or idle nodes, and for the unique tracker 0 duplicate keys, 0 self-TAKEN, 0 stale reservations and 0 reservations left at the end, in every run.

What the day showed:

• The lock protocol is cheap and never the bottleneck of the production-like profiles. An acquire or a release is one owner round trip of 0.8 ms at p50; of the 4.5 million calls of prod50w, 20 waited for a lock. The 10,000 sessions are think-time bound (about 99 tx/s per node), so the throughput is set by the profile, not by the code.
• The unique tracker holds under load. 981 unique-key transactions per second on top of the lock load, 176,591 in all, with the expected 43,759 error 132 on committed keys and no duplicate, lost or stale reservation; a reservation costs 0.92 ms at p50.
• Under high contention (hot50-u: 50 hot records and 50 hot unique keys shared by 10,000 sessions), 50 nodes sustained 19,155 lock and 8,149 unique transactions per second with 0 violations; the latency tail there is queueing on the hot keys, as intended.
• The latency tail of the SerialGC runs is the collector, not the code. The acquire/release p99 of 12-22 ms is young-generation pauses; with ZGC the p99 is 1.0-1.5 ms and 59 calls of 3 million exceed 5 ms (section 6.1).
• Buckets created before the cluster is complete make the start slow. The rebalance after a 50-node join took 183-669 s; creating the buckets once every node has joined (--assignBuckets=late) leaves 1-2 bucket moves in under a second and a more even primary spread (section 6.2). Both remedies are in the -fast profiles.

The same option sets were run on one host on 2026-10-14 (r16797 tree): the throughput is the same within 0.3 % and the p50 latencies within a few percent; section 5.5 has the comparison. Not done: any FWD-level test (ChUI, ETF, cluster application); see section 7.

2. What was tested, and what the tests actually do¶

Code. r16795 added OwnerLockManager (every lock change runs on the owner of the key), r16796 OwnerUniqueTracker (every unique key is reserved on its owner), r16797 table SHARE locks as member leases (TableLeases), owner call timeouts and fixes for ghost locks, failed releases on interrupt, lost unique reservations and a LockQueue deadlock; r16798 the review fixes. The harness compiles the lock package and the distributed/orm tracker classes from the branch source (locktest.sh build) and takes everything else from the branch's p2j.jar (./gradlew jar), so what runs is the reviewed code. The sections below say what the harness does with it; section 11 adds the regions and the mechanics a maintainer needs.

2.1 The cluster and a node¶

Every node is one JVM and one Geode member, as an FWD server is; the 50-node profiles run 25 on each host, with one locator per host. A node hosts every tenant (50): for each tenant it creates one OwnerLockManager over the partitioned region <tenant>_owner_locks (113 buckets, one redundant copy, as the FWD settings geode/ownerLocks/buckets and redundancy would) and, with the unique workload, one OwnerUniqueTracker over <tenant>_unique_keys. Geode spreads the buckets over the members, so every node owns about a fiftieth of every tenant's keys and serves the lock requests for them (the "owner calls" of the report), exactly as in a cluster of FWD servers. The lock functions and serializers are registered as FWD's GeodeConfig registers them.

Sessions are threads: --sessions per tenant per node, 4 × 50 = 200 per node, 10,000 in the cluster. Each session plays one FWD user on one tenant database and runs transactions back to back with a think time between them. What is emulated: the database (the committed unique keys and records live in a harness region), and there is no FWD session, RecordBuffer, directory or persistence. What is real: the lock managers, the lock queue, the table leases and their revocation, the membership handling and the whole owner-side unique reservation path, driven through the same entry points FWD uses (OwnerLockManager.lock/lockTable, UniqueTracker.Context).

2.2 A session's loop¶

Until the end of the run a session repeats: with probability --tableExclRate a table EXCLUSIVE (section 2.4); otherwise with probability --uniqueRatio a unique-key transaction (section 2.5); otherwise a lock transaction (section 2.3). Then it sleeps --thinkMs (2,000 ms in the production-like profiles, 300 ms in hot50-u). Nothing is counted before --warmupSec has passed. Any exception out of a transaction is counted as an error and logged with the session's name; the session goes on after 100 ms. Every session has its own random generator seeded from the run seed and its identity, so two runs of a profile draw the same records.

2.3 A lock transaction, step by step¶

1. Choose the records. 1 to --recordsPerTx of them (1-4 in the production-like profiles, 1-2 in hot50-u), each in a random table of 20 and, with probability --hotRatio, from the table's hot set (prod50w: 1,000 of 100,000 records at 5 %; prod50m: 200 at 20 %; hot50-u: 50 at 80 %), otherwise anywhere in the 100,000. The keys are sorted, so every session locks in the same order and record locks never deadlock each other.
2. Lock each record in that order, SHARE with probability --shareRatio (50 %; 30 % in hot50-u), otherwise EXCLUSIVE: OwnerLockManager.lock(type, record, true, 10 s). The call goes to the owner of the key, which grants it or parks the caller in its LockQueue until the holder releases (the "parked calls" of the report). The first record lock of a member on a table also takes the member's table lease, which from then on serves the implicit table SHARE lock of every record lock locally ("local, no call"). The time of the whole call is the acq_share / acq_excl latency. A 10 s timeout aborts the transaction (counted as aborted, every lock released).
3. Check it. After every acquire the session records its EXCLUSIVE lock in the checker region (putIfAbsent of the record key with its own name) and looks at what is there: another session recorded as EXCLUSIVE holder of the same record, or a session recorded as holding the table EXCLUSIVE, is a violation. A SHARE lock only looks. This is the exclusivity proof; it costs one or two extra Geode calls per lock and is on in every profile (--verify=true).
4. Maybe upgrade. A SHARE lock is upgraded to EXCLUSIVE with probability --upgradeRatio (5 %): lock(EXCLUSIVE, ...) on a record the session holds SHARE, the upgrade latency. Two sessions upgrading the same record deadlock by nature; the 10 s timeout breaks it, and the loser's transaction is aborted (the 13 timeouts of hot50-u).
5. Hold the locks for --holdMs (20 ms; 2 ms in hot50-u): the transaction's work.
6. Release in reverse order: remove the checker entry, then lock(NONE, record, true), one owner call each, the release latency.
7. The whole thing, from the first acquire to the last release, is the tx latency; the transaction counts as committed or aborted.

2.4 A table EXCLUSIVE¶

A bulk delete in FWD takes the table EXCLUSIVE; the harness does it with probability --tableExclRate per transaction (0.00001 in the production-like profiles: 3-7 per run; 0.0002 in hot50-u: 649 per run). lockTable(EXCLUSIVE, table, true) makes the owner revoke the table leases of every member holding one ("revokes sent / received" in the report) and wait for the record locks on the table to go; the call's time is the tbl_excl latency. The session then records the table in the checker region, holds the lock --tableExclHoldMs (50 ms) and restores the previous level. A record lock that another session manages to take on that table while the entry is there is a violation (step 3 above).

2.5 A unique-key transaction, step by step¶

The unique workload (--uniqueRatio of a session's transactions: 20 % in prod50w-u, 30 % in hot50-u) plays what FWD does for a record with unique indexes, in FWD's order (RecordBuffer, Validation), on a synthetic table with two unique indexes: code (one field) and (cust, seq) (two fields, seq sometimes unknown, in which case that key is not reserved). The DMO of that table is assembled at runtime with DmoMetadataManager.registerDmo under FWD's AsmClassLoader, as converted DMOs are.

1. Choose the operation: create, key change or delete with --uniqueOpMix (50/40/10). A change or a delete needs a committed record: the session draws up to 4 codes (from the hot set of 200 with probability 20 %, in hot50-u 50 at 80 %, otherwise from the tenant's 100,000) and looks each up in the emulated database; if none is committed the transaction becomes a create. New record ids encode node, session and sequence, so they are unique cluster-wide.
2. Lock the record EXCLUSIVE through the tenant's OwnerLockManager (10 s timeout), as FWD locks a record before changing it.
3. Build the change: a create gets a random code and (cust, seq); a change loads the committed values and changes either code or (cust, seq) (50/50); a delete loads them.
4. Open the transaction scope of the session's UniqueTracker.Context (beginTxScope). With probability --uniqueSubTxRatio (5 %) a second, nested scope is opened: the inner block of a 4GL transaction.
5. Reserve the keys on their owners: lockAndChange(tracker, record, table, true) for a create or a change, lockAndDelete(tracker, record) for a delete. This is the branch's OwnerUniqueTracker: one owner call per index whose key changed, each key reserved on the member that owns it; the u_reserve latency. A ValidationException is error 132: the key is reserved by another open transaction. The session reads the key's holder afterwards to classify it (held by an open transaction, holder gone, or the record's own id held by another member, which is the self-TAKEN failure), rolls the transaction back (rollback(true) + finished()) and counts it.
6. Record the open transaction in the _uopen checker region (so the stale scanner can tell an open transaction from a lost one).
7. Validate against the committed keys, as the database's unique index would at the INSERT or UPDATE: every new key is looked up in the emulated database; one committed to another record means rollbackChange(tracker, token) (the u_rollback latency), counted as error 132 (committed), and the transaction is rolled back. Otherwise unlock(tracker, token) ends the validation step (u_unlock); the reservations stay until the transaction ends.
8. Hold --holdMs, the transaction's work.
9. Undo the sub-transaction (the 5 % of step 4): the inner scope is ended with rollback(false) + finished(), its change is not written, and the outer transaction commits with nothing to write. Or roll the whole transaction back with probability --uniqueRollbackRatio (5 %): rollback(true) + finished(), nothing written, keys released.
10. Commit: first the emulated database (putIfAbsent of every new key with the record id: a key already committed to another live record is the duplicate failure; the old keys of a change and all keys of a delete are removed; the record is written), then commit(true) + finished() of the tracker scope, which gives the session's reservations back (the u_txend latency). The database before the tracker is FWD's order: a key is reserved from the reservation until after its commit, so no other transaction can commit it in between.
11. Clean up: remove the record from _uopen, release the record lock. The whole transaction is the u_tx latency.

Two faults can be injected to prove the checks (--injectFault, used only by test/fault-injection.sh): skipRelease makes 5 % of committed creates and changes skip finished(), so their keys stay reserved (stale and never-released must fire); bypassTracker makes 5 % of creates reserve nothing and commit after the database check alone, so two of them can commit the same key (duplicate must fire).

2.6 The checks running alongside¶

The exclusivity checker of steps 3 and 2.4 runs inside the sessions. Independently of them, every node scans every 5 s the reservations of the buckets it owns in every _unique_keys region: one older than --staleSec (30 s) whose transaction is not in _uopen is stale (a lost release). At the end of the run, after every session has stopped and every node has reached the end barrier, node 0 counts the reservations left in the _unique_keys regions: never released. Node 0 also counts the nodes that reported (lost nodes) and the nodes with sessions but no transaction (idle nodes). Section 4 lists all nine checks and how each was proven to fire.

2.7 What each profile stresses¶

profile	sessions	what it is	what it exercises
smoke2	8 (2 nodes × 2 tenants × 2)	30 s with the defaults: 1-2 records, 30 % SHARE, hold 2 ms, think 5 ms, 20 % unique	every code path across the two hosts, before an hour of 50-node runs
prod50w	10,000	production-like: 1-4 records, 50 % SHARE, 5 % upgrades, hold 20 ms, think 2 s, hot set 1,000 records at 5 %	the cost of the protocol with almost no contention: 25,000 owner calls/s, leases, the checker; 20 parked calls in 4.5 million
prod50m	10,000	the same with 200 hot records at 20 %	moderate contention on the hot records: 8 × the parked calls at the same p50
prod50w-u	10,000	prod50w with 20 % unique-key transactions, hot keys 200 at 20 %	the tracker under the same load: reservations, collisions on committed keys (25 % of unique transactions end in a 132), rollbacks, sub-transactions, deletes
hot50-u	10,000	50 hot records at 80 %, 1-2 records, 30 % SHARE, hold 2 ms, think 300 ms, 30 % unique on 50 hot keys at 80 %, 20 × the table EXCLUSIVE rate	saturation: the lock queue (154,424 parked calls, re-asks, owner busy), lease revocation (649 table EXCLUSIVE), upgrade deadlocks (13 timeouts), the tracker on 50 contended keys
*-fast	as the original	the same loads with buckets created after all nodes joined and ZGC	the same, with the startup and GC remedies of section 6

2.8 A run from start to end¶

The nodes of a host start 0.2 s apart. Each creates its cache, registers the functions, creates its regions, lock managers and trackers, publishes ready-N in the control region and waits until all N are there (--startTimeoutSec); with --assignBuckets=late it then creates the buckets of its share of the regions and waits until every region's buckets exist. Node 0 rebalances the lock and unique-key regions (--rebalance=true) and writes a start signal 3 s ahead, relative to its own clock, so the hosts' clock offset does not matter. All sessions start together, warm up for --warmupSec (not counted), then run the measured --durationSec. They stop; every node waits at an end barrier (a member leaving early would purge locks and move buckets, which would distort the others' numbers); node 0 counts the reservations left; every node publishes its counters and latency histograms to the control region; node 0 aggregates them into the report and the RESULT line; then all leave.

2.9 Not tested¶

Node, locator or host failures (members leaving under load); cluster sizes other than 50 (the 2-node smoke run aside); a single-host baseline on the r16798 tree; and anything above the lock manager API: ChUI/ETF/cluster-application tests of the branch are still owed (the 2026-10-21 wiki page Owner-Side Cluster Locking records the same status).

3. Test matrix¶

All 50-node profiles run with 1 GB node heaps and the option sets of cluster.conf (COMMON50: SerialGC and the same options as the earlier single-host runs, so the days stay comparable), plus --startTimeoutSec as a wait bound. Node 0 (cust004) rebalances once every node has joined (--rebalance=true) and writes the report.

profile	nodes	main options	warm-up / measured	run	result
smoke2	1 + 1	--tenants=2 --sessions=2 --uniqueRatio=0.2	5 / 30 s	cluster-20261022_154111-smoke2	PASS
prod50w	25 + 25	50 tenants, 4 sessions per tenant per node (10,000 sessions), 4 records per tx, 50 % SHARE, hold 20 ms, think 2,000 ms, hot set 1,000 records at 5 %, 113 buckets	180 / 180 s	cluster-20261022_132309-prod50w	PASS
prod50m	25 + 25	as prod50w, hot set 200 records at 20 %	180 / 180 s	cluster-20261022_133846-prod50m	PASS
prod50w-u	25 + 25	prod50w + --uniqueRatio=0.2 (create/change/delete 50/40/10, 5 % rollbacks, 5 % sub-transaction undo)	180 / 180 s	cluster-20261022_144731-prod50w-u	PASS
hot50-u	25 + 25	50 hot records at 80 %, 2 records per tx, hold 2 ms, think 300 ms, 4 sessions, + --uniqueRatio=0.3 --uniqueHotKeys=50 --uniqueHotRatio=0.8	10 / 120 s	cluster-20261022_151009-hot50-u	PASS

Diagnostic and experiment runs (60 / 120 s, prod50w options unless noted):

run	what differs	purpose
cluster-20261022_135855-prod50w	-Xlog:gc	GC diagnosis (SerialGC)
cluster-20261022_141101-prod50w	GC_OPTS=-XX:+UseZGC	GC confirmation
cluster-20261022_155429-prod50w	--assignBuckets=late (node 0 assigns)	rebalance experiment 1
cluster-20261022_160210-prod50w-fast	late + ZGC (node 0 assigns)	rebalance experiment 2
cluster-20261022_161118-prod50w-u-fast	prod50w-u, late (all nodes assign) + ZGC	rebalance experiment 3

The -fast profiles (prod50w-fast, prod50w-u-fast, hot50-u-fast) are the originals plus --assignBuckets=late and ZGC; hot50-u-fast has not been run yet.

4. Correctness checks¶

A run FAILs (exit code 2, last report line RESULT: FAIL name=count ..., up to 5 samples per failed check above it) when any of these is not zero:

check	what it catches	how
violations	two EXCLUSIVE holders of one record, or a record lock during a table EXCLUSIVE	the checker (--verify=true) records holders in a separate region and counts every overlap
errors	any exception in a session	counted per session
stuck sessions	a session that did not finish its transaction	end barrier
lost nodes	a member that left (lock purges, bucket moves)	node 0 counts the results published
idle nodes	a node whose sessions did nothing (found 2026-10-22: a clock-offset bug hid it)	per node transaction count
duplicate keys	a unique key committed to two live records	putIfAbsent on the emulated database at commit
self-TAKEN	a record's error 132 caused by its own reservation held by another member (a lost or late release)	holder read from <tenant>_unique_keys after a tracker 132
stale reservations	a reservation older than --staleSec (30 s) whose transaction is no longer open	every node scans the reservations it owns every 5 s
never released	reservations left once every session has stopped	node 0 counts the _unique_keys entries at the end

Reported, not judged: error 132 counts by cause (held by an open transaction / committed in the database / holder gone, which is a timing race between the 132 and the holder read).

The checks were proven before the runs, by breaking the workload on purpose on 2 local nodes (test/fault-injection.sh): --injectFault=skipRelease (5 % of committed creates and changes skip finished()) ends with RESULT: FAIL stale=8 never-released=8; --injectFault=bypassTracker (5 % of creates skip the tracker) ends with RESULT: FAIL duplicate=44; the clean run passes. The idle-node check has a fixture of the real bad run (test/idle-node.sh: node-1 did 0 transactions and the report said PASS; now RESULT: FAIL idle-nodes=1).

5. Numbers¶

All numbers come from the runs' report.txt (run ids in section 3 and the appendix); latencies are in ms, "p99" the 99th percentile.

5.1 Lock workload: prod50w, prod50m and the -fast variant¶

	prod50w	prod50m	prod50w-fast
collector, buckets	SerialGC, created at startup	SerialGC, created at startup	ZGC, created once all nodes joined
measured	180 s	180 s	120 s
transactions / s (per node)	4,922 (98-99)	4,927 (98-99)	4,933 (98-99)
acquire p50 / p99 / p99.9	0.82 / 17.7 / 137	0.82 / 16.0 / 137	0.81 / 1.17 / 1.71
release p50 / p99	0.71 / 13.2	0.71 / 12.2	0.70 / 1.02
upgrade p99	1.68	1.55	1.00
table EXCLUSIVE p99 (count)	61 (7)	43 (7)	99 (3)
transaction p50 / p99 / max	26.4 / 93 / 636	26.4 / 89 / 517	25.9 / 31 / 109
owner calls / s (per transaction)	24,912 (5.06)	24,934 (5.06)	24,972 (5.06)
owner fast call avg	1.23	1.20	0.75
parked calls (waited for a lock)	20	164	11
calls slower than 5 ms	71,403 (1.6 %)	68,985 (1.5 %)	59 (0.002 %)
table leases taken / revokes sent	50,550 / 11	50,550 / 11	50,200 / 4
CPU, all node JVMs	30.6 cores	30.8 cores	33.6 cores
threads per node at the end	545-799	552-792	523-790
heap after GC at the end (MB)	105-128	100-121	160-186
lock managers created, slowest node	137.8 s	124.0 s	1.4 s
rebalance after the last join	182.7 s	211.5 s	0.51 s (2 bucket moves)
primary buckets per node, min / max (max/avg)	79 / 140 (1.24)	82 / 135 (1.19)	104 / 121 (1.07)
checks	all 0	all 0	all 0

Each node runs 200 sessions that think 2 s between transactions, so about 99 tx/s per node is the ceiling of the profile and the throughput cannot tell the variants apart; the latencies can. prod50m (200 hot records at 20 %) parks eight times more calls than prod50w (164 against 20) at the same p50, which is the contention it is meant to produce. The p99 of the SerialGC runs is the collector (section 6.1); prod50w-fast shows the protocol's own tail: 1.17 ms at p99 and 1.71 ms at p99.9 for an acquire, 31 ms for a whole transaction that holds its locks for 20 ms.

5.2 Unique-key workload: prod50w-u and the -fast variant¶

	prod50w-u	prod50w-u-fast
collector, buckets	SerialGC, created at startup	ZGC, created once all nodes joined
measured	180 s	120 s
lock transactions / s	3,942	3,951
unique transactions / s (total; per node)	981 (176,591; 19-20)	985 (118,219; 19-20)
all transactions / s	4,923	4,936
unique operations created / changed / deleted	143,424 / 26,464 / 6,703	98,144 / 15,920 / 4,155
rollbacks: full / sub-transaction undone	50,256 / 6,739	28,460 / 4,807
error 132: open transaction / committed / holder gone	13 / 43,759 / 4	8 / 24,013 / 2
reserve (lockAndChange) p50 / p99 / max	0.92 / 31.5 / 1,356	0.94 / 1.46 / 6.3
transaction-end release p99	20.7	1.23
unique transaction p50 / p99 / p99.9	24.8 / 190 / 1,294	24.8 / 27.4 / 28.4
acquire p99 / release p99 (lock workload)	22.3 / 12.4	1.23 / 1.07
parked calls	73	23
calls slower than 5 ms	54,514 (1.4 %)	41
table leases taken	53,151	52,796
CPU, all node JVMs	30.7 cores	36.7 cores
rebalance after the last join (100 regions)	551 s	0.75 s (1 bucket move)
primary buckets per node, max/avg	1.19	1.07
duplicate / self-TAKEN / stale / never released	0 / 0 / 0 / 0	0 / 0 / 0 / 0

20 % of every session's transactions are unique-key transactions, so the lock rate is lower by construction and the sum is what compares with prod50w. The requested mix is 50/40/10, but a change or a delete needs an existing record of the session's own, so most transactions early in a run are creates. The 43,759 rollbacks on a committed key are the workload colliding on its hot keys on purpose (error 132 from the check against the emulated database), the 13 on an open transaction are collisions with a reservation another session still holds; both are the correct outcome. The reservation round trip is about 1 ms; the 31.5 ms p99 and the 1.3 s maxima of the SerialGC run are GC pauses (section 6.1).

5.3 High contention: hot50-u¶

50 hot records per table at 80 %, 50 hot unique keys at 80 %, 10,000 sessions: most transactions queue on the same few owners, which is the point of the profile.

	hot50-u (SerialGC, buckets at startup, 120 s measured)
lock transactions / s (per node)	19,155 (371-393)
unique transactions / s (total)	8,149 (977,874)
all transactions / s	27,304
aborted (lock timeout 10 s, breaks upgrade deadlocks)	13
table EXCLUSIVE	649
acquire p50 / p99 / max	1.87 / 307 / 10,494
release p99	49.7
transaction p50 / p99	12.2 / 664
table EXCLUSIVE p99	4,653
reserve p50 / p99	2.21 / 211
unique transaction p50 / p99	19.2 / 909
error 132: open transaction / committed / holder gone	29,751 / 587,026 / 27,222
owner calls / s	75,102
parked calls (total parked time)	154,424 (52,945 s)
re-asks / owner busy	49,587 / 5,502
owner fast call avg	7.2
leases taken / revokes sent	84,626 / 895
CPU, all node JVMs	116.7 of 192 cores (JIT 21.5 of them: 10 s warm-up is short)
rebalance after the last join	669 s
checks	all 0 (the 13 timeouts are aborted transactions, not errors)

The protocol itself is not the limit: p50 of an acquire is the owner round trip (1.9 ms), the p99 is queueing on the 50 hot keys, and the 154,424 parked calls with their re-asks are the lock queue doing its work. The 27,222 "holder gone" 132s are the race between a tracker 132 and the harness's holder read at 8,000 unique tx/s on 50 keys (the holder commits or rolls back in between); a real leak would show as stale or never-released, both 0.

5.4 Smoke run smoke2 (1 + 1 nodes)¶

686 lock tx/s and 176 unique tx/s (340-347 and 86-89 per node, i.e. both hosts work equally), acquire p99 0.66 ms, reserve p99 0.98 ms, 956 error 132 on committed keys, all checks 0. This is the run the acceptance script starts with: it proves the two hosts can talk before an hour of 50-node runs.

5.5 Comparison with the single-host runs of 2026-10-14¶

The same option sets were run on 2026-10-14 on one 64-core host (cust004, 50 nodes, r16797 tree, SerialGC); every run's compare.txt has the full side by side. The essentials, then / today:

	prod50w	prod50m	hot50 (one saturated host) / hot50-u
transactions / s	4,919 / 4,922	4,915 / 4,927	13,817 / 19,155 (+ 8,149 unique)
acquire p50	0.87 / 0.82	0.87 / 0.82	18.7 / 1.87
acquire p99	14.2 / 17.7	13.95 / 16.0	2,048 / 307
release p99	10.6 / 13.2	10.6 / 12.2	430 / 49.7
transaction p99	101 / 93	114 / 89	4,030 / 664
owner fast call avg	1.32 / 1.23	1.33 / 1.20	69 / 7.2
parked calls	25 / 20	183 / 164	283,137 / 154,424
calls slower than 5 ms	1.8 % / 1.6 %	1.9 % / 1.5 %	93 % / 15 %
CPU, node JVMs	28.8 / 30.6 cores	29.3 / 30.8 cores	63.7 of 64 / 116.7 of 192 cores

Reading: the throughput is identical (think-time bound), the p50s are 5 % lower, and the p99s of the production-like profiles are 15-25 % higher, which is that day's SerialGC pause distribution (the old runs had more calls over 5 ms but shorter ones), not the protocol (section 6.1). The differences mix the r16797 to r16798 code change with the cross-host network; no single-host run of r16798 was made. The hot50 line is indicative only: the old run saturated one host (63.7 of 64 cores) and had no unique workload.

6. Findings¶

6.1 The latency tail is SerialGC¶

Question: why is the acquire p99 17.7 ms when the p50 is 0.82 ms and only 20 calls of 4.5 million waited for a lock? Diagnosis on a short prod50w with -Xlog:gc on every node (SerialGC, 1 GB heap, -Xms64m):

• each node did about 105 young collections in the roughly 530 s of the run, one every 2-5 s, average pause 23 ms, maxima 148-326 ms (the longest ones are the harness's own System.gc() heap measurements outside the measured window); of 5,461 pauses, 380 were 5-10 ms, 2,687 10-20 ms, 2,142 20-50 ms, 244 over 50 ms;
• every node was paused 0.86-1.09 % of the last 150 s; a round trip touches two JVMs, so about 1.5 % of calls should be hit, and 1.47-1.63 % of the calls were slower than 5 ms, on both hosts alike;
• the heap stays at 60-230 MB because -Xms64m lets SerialGC keep a 20-50 MB young generation while the nodes allocate 17-22 MB/s.

Confirmation: the same run with -XX:+UseZGC and nothing else changed:

prod50w, 60 / 120 s	SerialGC	ZGC
transactions / s	4,926	4,948
acquire p50 / p99 / p99.9 (ms)	0.82 / 12.7 / 25.3	0.79 / 1.78 / 3.36
release p99 (ms)	11.9	1.36
transaction p99 (ms)	55.8	33.3
calls slower than 5 ms	46,171 (1.5 %)	349 (0.01 %)
parked calls	13	18
CPU, all node JVMs	31.2 cores	33.5 cores (+7 %)
heap after GC at the end (MB)	89-102	162-190

So the owner round trip is about 0.8 ms at p50 and under 2 ms at p99; everything above is the collector. The acceptance profiles keep SerialGC for comparability with 2026-10-14. For FWD this is a server JVM question: collector choice and -Xms/young-generation sizing of the servers, not a lock manager change.

6.2 Rebalance after a cluster start, and bucket placement¶

Every lock region (50) and unique-key region (50) has 113 buckets with one redundant copy. With buckets created at startup (geode/ownerLocks/assignBuckets=true, the FWD default) the first nodes to join get almost all of them, and node 0's rebalance after the last join moves them: 183-213 s for the 50 lock regions, 551 s for the 100 regions of prod50w-u, 669 s for hot50-u, and 903 s when the harness's own 100 regions were still included (the first prod50w-u attempt timed out on it). FWD's default of 1,009 buckets per region means roughly nine times as many buckets to move. After the rebalance the primaries were still uneven: 47 % on cust004, 53 % on cust007, busiest node 1.24 × the average.

The harness got --assignBuckets=late: nothing is created at startup; once every node has joined, every node creates the buckets of the tenants with t % nodes == idx (its lock region and its unique-key region), so Geode places them over the complete cluster.

prod50w, 60 / 120 s	buckets at startup (true)	late, node 0 creates all	late, every node its share
bucket creation	122-138 s after the cache	92-95 s (node 0, 50 regions, serial)	about 7 s per node (100 regions, in parallel)
rebalance	183-213 s, thousands of bucket transfers (27,989 for the 200 regions of the first prod50w-u attempt)	448 ms, 1 transfer	748 ms, 1 transfer (100 regions, prod50w-u-fast)
primaries per node, max / avg	1.21-1.24	1.08 (108-122)	1.07 (104-121)
primaries per host	47 % / 53 %	2,831 / 2,819	even (104-121 per node on both)
requests served, max / avg	1.21-1.24	1.08	1.06

Throughput and latency are unchanged by the mode (4,928-4,933 tx/s; acquire p99 14.2 ms with SerialGC, 1.17-1.23 ms with ZGC). Runs: cluster-20261022_155429-prod50w, cluster-20261022_160210-prod50w-fast, cluster-20261022_161118-prod50w-u-fast.

For a production FWD cluster the equivalent is: do not create the buckets before the cluster is complete (assignBuckets=false with all servers started before users log in, or a new "assign once N members joined" setting, which is what the late mode does), use fewer buckets than the default 1,009 per region (113 was enough for 50 nodes), and run gfsh rebalance at a quiet time after a server joins or leaves, since Geode never rebalances on its own.

6.3 Clock offset between the hosts (harness bug, fixed)¶

cust007's clock is about 466 s ahead of cust004's. The first cross-host smoke run passed with node-1 having done 0 transactions: the start time was node 0's absolute clock, so cust007's node had "started" 466 s ago and never ran. Fixed by scheduling with a relative delay, and the new idle-nodes check fails such a run. The branch itself compares no timestamps across members (OwnerLockEntry.since, UniqueReservation.since are informational and set by the owner); only the harness's stale check mixes clocks when a bucket moves between hosts. NTP on the hosts is still advisable.

6.4 Fixed ports and the ephemeral range (setup finding)¶

cust004's firewall drops everything but ssh, so Geode's ports were pinned to a fixed set for the firewall rules. The first set (40000-40099, 41000-41099) lay inside Linux's ephemeral range (32768-60999): four nodes failed with BindException: Address already in use because outbound connections had taken their port first. The set was moved below 32768 (section 8); preflight now refuses a fixed range inside the hosts' ephemeral range.

6.5 Smaller harness findings¶

• Geode's rebalance includes every partitioned region; the harness excludes its own _db/_uopen regions now (an FWD cluster has no such regions), which halved the prod50w-u rebalance.
• The stale scanner saw EntryDestroyedException for entries released between iteration and read (the errors samples of the prod50w-u and hot50-u reports, count 0); fixed to skip them. The samples in those two reports are this nuisance, not tracker errors.
• The 13 aborted transactions of hot50-u are the 10 s lock timeout breaking upgrade deadlocks (two sessions holding SHARE and both upgrading), which is the intended resolution.

7. Not covered and open items¶

• FWD-level test of the branch (ChUI/ETF/cluster application): not done; the harness drives the lock manager API, nothing above it.
• [4369-locks] TEMPORARY instrumentation in OwnerLockManager / OwnerLockFunction / TableLeases: to be removed before merge; the harness report's owner-call statistics depend on it, so the harness needs an update then.
• hot50-u-fast has not been run; the other two -fast profiles were run for 120 s, not 180 s.
• cust004 firewall rules are not persistent (iptables, added by hand on 2026-10-22; INPUT carries the TCP rule twice, harmless). They vanish on a reboot.
• Clocks 466 s apart; NTP.
• Harness minors deferred in the ledger: the idle check only flags a node with 0 transactions; start-nodes does not notice a dead slice shell before its 120 s wait; all.sh decides the selftest by the absence of "FAIL"; the 132 catch does not check the error number; deploy replaces the bundle even while a run is live.
• test/late-buckets.sh was written together with the late mode and never watched failing (the hosts' experiments read the same log lines it asserts).

Part 2: setup and how to repeat¶

8. Topology¶

        cust004 (192.168.1.83, 64 cores)               cust007 (192.168.1.86, 128 cores)
   +------------------------------------+        +------------------------------------+
   | locator-1  :10334  (tcp 20091)     |<------>| locator-2  :10335  (tcp 20092)     |
   | nodes 0..24   (node 0 reports)     |<------>| nodes 25..49                       |
   |   tcp-port 20000+N                 |  LAN   |   tcp-port 20000+N                 |
   |   membership 21000-21099 tcp+udp   |        |   membership 21000-21099 tcp+udp   |
   +------------------------------------+        +------------------------------------+
   every JVM: locators=192.168.1.83[10334],192.168.1.86[10335], bind-address = its host's LAN address
   development machine: ssh -p 2227 es@localhost -> cust004 (key), ssh -p 2228 mfg@localhost -> cust007 (password)

	cust004	cust007
reached as	ssh -p 2227 es@localhost (key; or a password from SSHPASS_cust004 with H1_AUTH=password)	ssh -p 2228 mfg@localhost (password, from SSHPASS_cust007 or SSHPASS)
LAN address	192.168.1.83	192.168.1.86
cores / RAM	64 / 503 GiB	128 / 503 GiB
Java	17.0.20	17.0.20
harness directory	~/4369d/locktest/	~/4369d/locktest/
firewall	iptables-nft, INPUT and OUTPUT policy DROP	none (policy ACCEPT)
other	older bundles ~/lt-20261014, ~/locktest (the 2026-10-14 runs)	only ~/4369d may be read or changed; Docker bridge 172.17.0.1

The ssh tunnels are created through the jump host www.goldencode.com (ssh port 2222): ssh www.goldencode.com -p 2222 -l es -NC -f -L2227:${cust004}:22 and ssh www.goldencode.com -p 2222 -l mfg -NC -f -L2228:${cust007}:22, with cust004 and cust007 set to the hosts' addresses (step 2 of section 10.1 has the full sequence). Both hosts reach each other directly on the LAN. Two locators, one per host, each started with both in its locators list so they share one membership view, as FWD's geode/locatorHost1/2 configuration does; locator-1 starts first, then locator-2, then the node slices. Locators are not in the fwd-servers group, so owner calls skip them.

Fixed Geode ports (GEODE_PORTS and LOC_TCP_PORT_BASE in cluster.conf), the only ones the firewall must open, both directions between the two LAN addresses:

ports	protocol	use
10334-10335	TCP	locators
20000-20099	TCP	direct channels (tcp-port): node N = 20000 + N, locator of host H = 20090 + H
21000-21099	TCP + UDP	membership (membership-port-range)

They must stay below the ephemeral range (/proc/sys/net/ipv4/ip_local_port_range, 32768-60999 on both hosts); preflight checks that, test/cluster-ports.sh checks the configuration, test/pinned-ports.sh that every JVM listens inside the set.

cust004 firewall rules. The five iptables rules that open this port set on cust004 (applied with sudo on 2026-10-22; not persistent, re-add after a reboot) are in step 3 of section 10.1, together with the check. A blocked port shows up within seconds: cluster.sh run probes each locator's port and direct-channel port from the other host right after starting it ("cust007 cannot reach 192.168.1.83:10334 (firewall?)").

Constraints on cust007. Only ~/4369d is read or written (plus the read-only host queries of preflight: Java version, nproc, ulimit -u, kernel threads-max / pid_max, systemd TasksMax, the ephemeral range). Every JVM keeps its files inside it: -XX:-UsePerfData, java.io.tmpdir under ~/4369d/locktest/tmp, the run's results directory as working directory. No keys are installed; the password is only ever in the SSHPASS_<host> or SSHPASS environment variable (sshpass -e), never in a file, a log or the shell history. Processes are stopped by their recorded pids only, and only if they still are load test processes (never pkill).

Thread budget. Every node keeps reader threads for every other member: about 190 + 80 × (nodes − 1) per node, so a 25-node slice of a 50-node cluster needs about 103,000 threads on its host. Measured at the end of the runs: 545-836 threads per node (up to 1,751 in hot50-u). preflight checks ulimit -u, TasksMax, threads-max and pid_max (both hosts: about 2 million or more).

9. Build, bundle, deploy¶

On the development machine, in the branch checkout:

./gradlew jar copyDependencies                 # the branch's p2j.jar and its jars in build/lib
bash locktest/bin/locktest.sh build            # compiles ../src/.../distributed/lock/*.java, the orm tracker
                                               #   classes, GeodeService, BlockSequence and everything under
                                               #   locktest/src into locktest/classes
bash locktest/bin/locktest.sh bundle           # locktest/locktest-bundle.tar.gz: classes, bin/, test/,
                                               #   cluster.conf and the 30 jars of jars.txt
bash locktest/bin/cluster.sh deploy            # scp + unpack into ~/4369d on both hosts (results/ kept)

jars.txt lists the jars the locator and the nodes load (Geode, FWD's p2j.jar, and for the unique workload commons-collections4, asm, reflections). A ClassNotFoundException in a node log on the hosts means a jar is missing from the list.

Harness files, for orientation:

file	role
bin/acceptance.sh	the whole acceptance in one command (sections 10.1 and 10.2)
bin/cluster.sh	two-host orchestration over ssh: preflight, deploy, run, status, collect, stop
cluster.conf	the hosts, the fixed ports, the profiles (PROFILE_*, HEAP_*, GC_*, BASELINE_*)
bin/locktest.sh	the harness itself: build, run (one host), locator, nodes, report, selftest, compare, bundle, and the detached steps cluster.sh uses (start-locator, start-nodes, state, stop, probe, preflight)
src/.../lock/LockLoadTest.java	the test program: locator mode, node mode (regions, sessions, barriers, checks, report)
src/.../lock/UniqueWorkload.java	one unique-key transaction, fault injection, 132 classification
src/.../lock/UniqueChecks.java	the stale scanner and the never-released count
src/.../lock/UniqueTable.java, UniqueRecords.java, LtTracker.java	the synthetic DMO, its records, the harness's OwnerUniqueTracker subclass
src/.../lock/SelfTest.java, Compare.java	the one-JVM tracker self-test; the comparison with a baseline
baseline/2026-10-14/{prod50w,prod50m,hot50}/	the 2026-10-14 reports and node results compared with (SOURCE names the run)
test/*.sh, test/all.sh	the harness's own tests (section 12)
results/	every run: run-* (one host), cluster-* (two hosts), acceptance-* (summaries)

10. Running¶

10.1 Step by step, from a fresh checkout to the results¶

Everything runs from the development machine, in the root of a checkout of branch 4369d (the paths below are relative to it); the hosts only need the bundle, which step 5 deploys. Times are those of 2026-10-22.

Step 1. What this machine needs. Java 17 (java -version), sshpass (command -v sshpass; install it with the package manager if missing), the branch checkout, and the two accesses: your ssh key accepted by es@cust004, and the password of mfg@cust007 (ask ES for it; it is never written down). Without a key on cust004 you need the password of es@cust004 as well: see "Passwords for both hosts" under step 5.

Step 2. Create the ssh tunnels and check where they land. Both go through the jump host www.goldencode.com (ssh port 2222): local port 2227 to cust004, logged in on the jump host as es, and local port 2228 to cust007, logged in as mfg. -N runs no command, -C compresses, -f puts the tunnel in the background, where it stays until killed:

cust004=192.168.1.83; cust007=192.168.1.86            # the hosts' addresses as seen from the jump host
ssh www.goldencode.com -p 2222 -l es  -NC -f -L2227:${cust004}:22
ssh www.goldencode.com -p 2222 -l mfg -NC -f -L2228:${cust007}:22
ssh -o BatchMode=yes -p 2227 es@localhost hostname    # must print cust004, without a prompt
ssh -p 2228 mfg@localhost hostname                    # asks the password, must print cust007

Another hostname means the tunnel forwards to another machine: fix the tunnel before going on (~/bin/gcd_remote.sh on ES's machine wraps the same two commands).

Step 3. Check the cust004 firewall rules, add them if missing. cust004's firewall drops everything but ssh (INPUT and OUTPUT policy DROP), so the fixed Geode ports of section 8 must be opened both ways between the two LAN addresses. The rules are not persistent: they are lost when cust004 reboots. cust007 has no firewall.

ssh -p 2227 es@localhost
sudo iptables -S INPUT | head -5      # the policy line, then the rules at the top of the chain

With the rules in place it looks like this: the policy, the two rules for 192.168.1.86 (tcp or udp first, depending on the order they were added), then the chain's own entries such as -A INPUT -i lo -j ACCEPT:

-P INPUT DROP
-A INPUT -s 192.168.1.86/32 -d 192.168.1.83/32 -i eth0 -p udp -m udp --dport 21000:21099 -j ACCEPT
-A INPUT -s 192.168.1.86/32 -d 192.168.1.83/32 -i eth0 -p tcp -m multiport --dports 10334:10335,20000:20099,21000:21099 -j ACCEPT
-A INPUT -i lo -j ACCEPT
...

Without them, no -A INPUT -s 192.168.1.86/32 ... line appears. In that case add the five rules (sudo on cust004), check again and log out:

sudo iptables -I INPUT  -s 192.168.1.86 -d 192.168.1.83 -i eth0 -p tcp -m multiport --dports 10334:10335,20000:20099,21000:21099 -j ACCEPT
sudo iptables -I INPUT  -s 192.168.1.86 -d 192.168.1.83 -i eth0 -p udp --dport 21000:21099 -j ACCEPT
sudo iptables -I OUTPUT -d 192.168.1.86 -o eth0 -p tcp -m multiport --dports 10334:10335,20000:20099,21000:21099 -j ACCEPT
sudo iptables -I OUTPUT -d 192.168.1.86 -o eth0 -p udp --dport 21000:21099 -j ACCEPT
sudo iptables -I OUTPUT -d 192.168.1.86 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
sudo iptables -S INPUT | head -5
sudo iptables -S OUTPUT | head -6
exit

The INPUT check then shows the lines above; the OUTPUT check shows the policy and the three rules for 192.168.1.86 (RELATED,ESTABLISHED, udp, tcp) at the top of its chain:

-P OUTPUT DROP
-A OUTPUT -d 192.168.1.86/32 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -d 192.168.1.86/32 -o eth0 -p udp -m udp --dport 21000:21099 -j ACCEPT
-A OUTPUT -d 192.168.1.86/32 -o eth0 -p tcp -m multiport --dports 10334:10335,20000:20099,21000:21099 -j ACCEPT
...

These are the rules applied on 2026-10-22 (INPUT then carried the TCP rule twice, which is harmless). A port that is still blocked shows up within seconds of a run: cluster.sh run probes each locator's ports from the other host and stops with "cust007 cannot reach 192.168.1.83:10334 (firewall?)".

Step 4. Build the branch jars (once, and again after every source change of the branch; a few minutes):

./gradlew jar copyDependencies

Step 5. Run the acceptance. One command does the rest: builds the harness, runs its own tests (about 15 min), bundles, deploys to both hosts, checks them and runs the five profiles (about 1 h):

bash locktest/bin/acceptance.sh

It asks once for the password of every host logged into with a password: mfg@cust007 and, with H1_AUTH=password, es@cust004 (typed, not echoed, not stored). Leave the terminal alone until it prints acceptance: done; summary: and the summary table. Expected durations: smoke2 about 3 min, prod50w and prod50m about 15 min each, prod50w-u about 22 min, hot50-u about 17 min. Variants:

SKIP_SUITE=1 bash locktest/bin/acceptance.sh                              # without the harness's own tests
bash locktest/bin/acceptance.sh smoke2                                    # a quick cross-host check only
bash locktest/bin/acceptance.sh prod50w-fast prod50w-u-fast hot50-u-fast  # the late-buckets + ZGC variants

Passwords for both hosts. cust004 is logged into with your ssh key (H1_AUTH=key in cluster.conf), cust007 with the password of mfg. Without a key on cust004, log into it with the password of es: set H1_AUTH=password in cluster.conf (once), and either let acceptance.sh ask for both passwords, or export them beforehand, one variable per host, in the shell that runs the commands:

read -rsp 'Password of es@cust004: '  SSHPASS_cust004; echo; export SSHPASS_cust004
read -rsp 'Password of mfg@cust007: ' SSHPASS_cust007; echo; export SSHPASS_cust007
bash locktest/bin/acceptance.sh                  # asks nothing more
bash locktest/bin/cluster.sh run smoke2          # every cluster.sh command works the same way

SSHPASS_<host> is the password of that host (the host name as in cluster.conf); SSHPASS alone serves every password host that has no variable of its own, which only helps when the accounts share a password. The read -rs form keeps the passwords out of the shell history; they are never written anywhere. A missing one stops every command at once with cluster: set SSHPASS_cust004 or SSHPASS (password of es@cust004).

Step 6. Read the result. The exit code says it: 0 = every profile passed, 2 = a check failed, 1 = a profile could not run. The summary is locktest/results/acceptance-<timestamp>/summary.txt, one line per profile, for example:

profile    result       tx/s    uniq/s   acq p99   rel p99    tx p99 u_res p99  vs 2026-10-14 (tx/s, acq p99)
prod50w    PASS         4922               17.66     13.18     93.18            +0.1%, +24.3%
prod50w-u  PASS         3942       981     22.27     12.42     91.14     31.49  -19.9%, +56.8%

A FAIL line is followed by failed checks: name=count .... details.txt next to it has the correctness section of every run; the full report.txt and compare.txt of each run are in locktest/results/cluster-<timestamp>-<profile>/ (section 10.4 explains how to read them; the RESULT line is the last line of report.txt, the samples of a failed check are just above it).

Step 7. If it stops early. A line acceptance: FAILED: ... names the step: no tunnel (step 2), no sshpass (step 1), the build, the harness's own tests (suite.log in the results directory), the deploy (deploy.log) or the preflight (preflight.log: a live pid of an earlier run, a locator port in use, thread limits, a fixed port inside the ephemeral range). A profile reported as NO RUN has its orchestrator log in run-<profile>.log; the usual causes and fixes are in the table of section 12. A normal run leaves nothing behind on the hosts; after an interrupted one, with the password exported:

read -rs SSHPASS_cust007 && export SSHPASS_cust007    # and SSHPASS_cust004 with H1_AUTH=password
bash locktest/bin/cluster.sh status      # what is still running on the hosts
bash locktest/bin/cluster.sh collect     # fetch a finished run the orchestrator missed
bash locktest/bin/cluster.sh stop        # stop a run's processes (by their pid files only)

Step 8. Single runs and experiments (once step 5 has deployed the current bundle):

read -rs SSHPASS_cust007 && export SSHPASS_cust007    # and SSHPASS_cust004 with H1_AUTH=password
bash locktest/bin/cluster.sh run prod50w                                    # one profile, about 15 min
bash locktest/bin/cluster.sh run prod50w --warmupSec=60 --durationSec=120   # a shorter one
GC_OPTS=-XX:+UseZGC bash locktest/bin/cluster.sh run prod50w                # another collector

Each run lands in locktest/results/cluster-<timestamp>-<profile>/ with its report.txt and compare.txt.

Only the harness's own tests, without the hosts (local JVMs, about 15 min): after step 4, bash locktest/bin/locktest.sh build and bash locktest/test/all.sh; the last line must be ALL PASS.

10.2 Everything in one command¶

bash locktest/bin/acceptance.sh                  # smoke2 prod50w prod50m prod50w-u hot50-u, about 1 h
bash locktest/bin/acceptance.sh smoke2 prod50w   # only these profiles
bash locktest/bin/acceptance.sh prod50w-fast     # a -fast profile (late buckets + ZGC)
SKIP_SUITE=1 bash locktest/bin/acceptance.sh     # without the harness's own tests (about 15 min less)
SKIP_BUILD=1 bash locktest/bin/acceptance.sh     # hosts already have the current bundle

Prerequisites: a checkout of the branch with ./gradlew jar copyDependencies done, Java 17, sshpass installed, the tunnels on 2227 and 2228 up, the firewall rules of section 8 in place. The script asks for the password of every password host (mfg@cust007; es@cust004 too with H1_AUTH=password; read -rs, not echoed, not stored) unless SSHPASS_<host> or SSHPASS is exported, then: checks the tunnels, builds, runs test/all.sh, bundles, deploys, runs preflight, runs the profiles one after another and writes results/acceptance-<timestamp>/summary.txt (one line per profile: result, tx/s, unique tx/s, p99s, change against 2026-10-14) and details.txt (the correctness section of every run). Exit code 0 = all passed, 2 = a check failed, 1 = a profile could not run. Verified end to end with smoke2 on 2026-10-22 (results/acceptance-20261022_153213; its summary.txt still shows the transaction count where later versions show the rate).

10.3 The pieces: cluster.sh commands¶

read -rs SSHPASS_cust007 && export SSHPASS_cust007   # password of mfg@cust007, typed (not in shell history)
read -rs SSHPASS_cust004 && export SSHPASS_cust004   # only with H1_AUTH=password (no key on cust004)
bash locktest/bin/cluster.sh preflight       # Java 17, thread limits, disk, free ports, live pids, clocks,
                                             #   ephemeral range on both hosts
bash locktest/bin/cluster.sh run smoke2      # 1 + 1 nodes, 1 min: proves the hosts can talk
bash locktest/bin/cluster.sh run prod50w     # a profile; extra --opt=value override it
bash locktest/bin/cluster.sh run prod50w --warmupSec=60 --durationSec=120
GC_OPTS=-XX:+UseZGC bash locktest/bin/cluster.sh run prod50w      # another collector
NODE_JVM_OPTS=-Xlog:gc:file=gc-%p.log bash locktest/bin/cluster.sh run prod50w   # extra JVM options
bash locktest/bin/cluster.sh status [RUN]    # state of a run on both hosts (running / done N / none)
bash locktest/bin/cluster.sh collect [RUN]   # copy a run's results here (also after an interrupted run)
bash locktest/bin/cluster.sh stop [RUN]      # stop a run's processes on both hosts, by their pid files

run does, in order: preflight; results/.last = the run name; start locator-1 on cust004 and probe its ports from cust007; start locator-2 and probe from cust004; start the node slices (cust004 nodes 0-24, cust007 nodes 25-49, each slice detached with setsid nohup, with HEAP, NODE_JVM_OPTS, GC_OPTS and the fixed ports); poll both hosts every 15 s (POLL_SEC) until both slices are done, a slice failed or vanished (then both hosts are stopped at once), or the deadline passed (startTimeoutSec + warmupSec + durationSec + 17 min); stop the locators; collect to locktest/results/<RUN>/{cust004,cust007}/; write report.txt (node 0's report) and compare.txt (against baseline/2026-10-14/<BASELINE_profile>). Exit 0 = PASS, 2 = FAIL, 1 = did not run. If the orchestrator dies, the run goes on: status, collect and stop take over (tested by killing it mid-run).

GC_<profile> in cluster.conf sets a profile's collector (GC_OPTS in the environment wins); the default is -XX:+UseSerialGC as on 2026-10-14. HEAP_<profile> is the node heap (1g for the 50-node profiles).

10.4 Reading a run¶

results/<RUN>/report.txt, top to bottom:

• options line: every option the nodes ran with (the single source of truth for a run's configuration);
• startup: nodes reported, cache creation, lock manager creation, bucket creation, heap, rebalance time;
• throughput: transactions and tx/s, per node min/avg/max (uneven = a slow host or an idle node);
• latency: acquire (SHARE / EXCLUSIVE), upgrade, release, table EXCLUSIVE and whole transaction, with count, mean, p50, p90, p99, p99.9 and max in ms;
• unique tracker (with --uniqueRatio > 0): unique tx/s, operations by kind, rollbacks, error 132 by cause, and the latency of reserve (u_reserve), unlock, rollbackChange, transaction end (u_txend) and the whole unique transaction (u_tx);
• owner calls: round trips by kind; fast calls avg is the cluster round trip, >5ms the count of slow ones (GC or network, see 6.1); parked calls waited for a lock; re-asks and owner busy are the queue protocol under contention; local, no call are the table SHARE locks served by the member lease;
• table leases: taken, revoked (by table EXCLUSIVE requests), given back;
• owner load balance: requests served and primary buckets per node (max/avg far above 1 = the first nodes own most keys), busiest owners;
• correctness: the checks of section 4, samples of failed ones, and RESULT: PASS or RESULT: FAIL ....

compare.txt puts the run's rows next to the 2026-10-14 baseline of the profile (tx/s, all tx/s, p50 and p99 of every operation, owner calls, leases, checks; unique rows n/a on the old side) with the caveats that apply. Per host, <host>/node-N.out has the node's log (progress, view size, bucket assignment, rebalance, any exception) and node-N.result its raw counters; locator-N.out and locator-N-geode.log the locators.

10.5 One host only¶

bash locktest/bin/locktest.sh run --nodes=10 --tenants=50 --durationSec=120 starts a locator and the nodes on this machine (results in results/run-<timestamp>/); locktest.sh help lists every option. This is what the harness's tests use, and what the 2026-10-14 runs on cust004 were.

11. How the harness works, as far as it matters for the results¶

• Tenants and regions. Tenant t is lt_tenantNNN; its lock region is lt_tenantNNN_owner_locks (113 buckets, 1 redundant copy in the profiles), its unique-key region lt_tenantNNN_unique_keys. Harness-owned regions, not under test: lt_tenantNNN_db (the emulated database: committed keys and records), lt_tenantNNN_uopen (record ids of open unique transactions, for the stale and 132 classification), lt_control (barriers, start signal, published results) and lt_verify (the exclusivity checker).
• Startup. Every node creates its cache, registers the lock and unique functions and their serializers as FWD's GeodeConfig does, creates its lock managers and trackers, publishes ready-N and waits for all N (--startTimeoutSec); with --assignBuckets=late it then creates the buckets of its share of the regions and waits until every region's buckets exist; node 0 rebalances (--rebalance=true, lock and unique regions only) and writes a relative start delay (3 s) into lt_control, so the hosts' clocks do not matter.
• Sessions. --sessions threads per hosted tenant per node (4 × 50 = 200 per node, 10,000 in all); what each does, transaction by transaction, is in sections 2.2 to 2.5. Warm-up is not measured.
• End. Sessions stop after --durationSec; nodes wait at an end barrier (a member leaving early would purge locks and move buckets), node 0 counts the reservations left, every node publishes its counters, node 0 aggregates them into the report, then all leave.
• Record ids encode node, session and sequence, so they are unique cluster-wide; the random seed is fixed (--seed=4369), so two runs of a profile draw the same records.

12. The harness's own tests and troubleshooting¶

bash locktest/test/all.sh runs, on this machine, the selftest and ten scripts (each takes an optional path to another locktest.sh, e.g. an unpacked bundle):

test	what it proves
locktest.sh selftest	the real tracker on the synthetic table in one JVM: basics, null seq, key change
unique-workload.sh	2 nodes with the unique workload on and off; operation mix line; AsmClassLoader only for unique runs
fault-injection.sh	skipRelease fails with stale + never-released, bypassTracker with duplicate, clean run passes
two-locators.sh	two locators listing each other share one view; state / stop behave
join-failure.sh	no locator, or not all members: exit 1 with the "only K of N members joined" message, no hang
pinned-ports.sh	every JVM listens inside the fixed port set (14 ports for 2 locators + 2 nodes)
compare.sh	the comparison with a baseline, n/a for missing metrics, the unique-share caveat
idle-node.sh	the fixture of the clock-offset run fails with idle-nodes=1
cluster-poll.sh	12 cases of the orchestrator's poll decision: wait, done, abort on a failed or vanished slice or the deadline
cluster-ports.sh	the fixed ranges are outside the ephemeral range; the old 40000/41000 ranges are reported
late-buckets.sh	2 nodes with --assignBuckets=late: the late assignment is logged after the barrier, the run passes

Status: ALL PASS (10 tests) in the acceptance run of 15:32; late-buckets.sh was added afterwards, and the complete suite of 11 was rerun on the final harness on 2026-10-22 in the evening: ALL PASS, exit 0.

What went wrong on 2026-10-22, and the fix (all in the ledger):

symptom	cause	fix
locator-2 never joined locator-1; TCP probes to the locator and membership ports blocked both ways	cust004 firewall (INPUT and OUTPUT policy DROP)	fixed port set + the rules of section 8
BindException: Address already in use on four nodes' tcp-port 400NN	fixed ports inside the ephemeral range	ranges moved to 20000/21000; preflight and cluster-ports.sh check
cross-host smoke2 PASS with node-1 at 0 tx	start time from node 0's clock, cust007 466 s ahead	relative start delay; idle-nodes check
prod50w-u: every node FATAL: no start signal after 900 s	node 0's rebalance of 200 regions took 903 s	harness regions excluded from the rebalance; -u profiles wait 1,800 s
back-to-back local runs: "port (10334) is not available"	the previous run's locator not yet gone	run waits for the locator to stop
stop counted 3 processes for 2 locators	Geode's own vf.gf.locator.pid	skipped
errors samples EntryDestroyedException under the unique workload	stale scanner reading a just-released entry	skipped
ClassNotFoundException in the bundle	jars missing from jars.txt	3 jars added
cluster.sh waiting up to 30 min for a host whose slice had died	no early abort	poll verdict: abort on a failed or vanished slice or the deadline
a refused run overwrote results/.last	.last written before preflight	written after preflight

Other things to know: a run leaves its node and locator processes on the hosts only until collect/stop; preflight refuses to start while pids of an earlier run are alive or a locator port is in use; under 50 node JVMs an interactive ssh session can time out, which is why every slice is detached.

13. Appendix: every two-host run of 2026-10-22¶

run	profile	nodes	outcome	purpose
cluster-20261022_111148-smoke2 (hosts only)	smoke2	1 + 1	locator-2 never joined	first cross-host attempt, firewall found
cluster-20261022_120943-smoke2	smoke2	n/a	refused in seconds: cust007 cannot reach 192.168.1.83:10334	after the port pinning, before the rules
cluster-20261022_123507-smoke2	smoke2	1 + 1	PASS, node-1 0 tx	rules in place; the clock-offset bug found
cluster-20261022_123938-smoke2	smoke2	1 + 1	PASS, node-1 0 tx	same bug, second run before the fix
cluster-20261022_124414-smoke2	smoke2	1 + 1	PASS, 347-349 tx/s per node	after the relative-start fix
cluster-20261022_125054-smoke2	smoke2	1 + 1	PASS (120 s)	.last fix: live run while a second one is refused
cluster-20261022_125924-smoke2	smoke2	1 + 1	PASS (120 s)	orchestrator killed mid-run; status / stop / collect resumed it
cluster-20261022_130459-prod50w	prod50w	25 + 25	did not start: BindException on 400NN	ports inside the ephemeral range
cluster-20261022_131947-smoke2	smoke2	1 + 1	PASS	the 20000/21000 ports and the swapped rules
cluster-20261022_132309-prod50w	prod50w	25 + 25	PASS, 4,922 tx/s	acceptance
cluster-20261022_133846-prod50m	prod50m	25 + 25	PASS, 4,927 tx/s	acceptance
cluster-20261022_135855-prod50w	prod50w, 60/120 s	25 + 25	PASS, 4,926 tx/s	GC diagnosis (-Xlog:gc)
cluster-20261022_141101-prod50w	prod50w, 60/120 s	25 + 25	PASS, 4,948 tx/s	ZGC confirmation
cluster-20261022_142220-prod50w-u	prod50w-u	25 + 25	did not start: no start signal in 900 s	rebalance of 200 regions took 903 s
cluster-20261022_144731-prod50w-u	prod50w-u	25 + 25	PASS, 3,942 + 981 tx/s	acceptance
cluster-20261022_151009-hot50-u	hot50-u	25 + 25	PASS, 19,155 + 8,149 tx/s	acceptance
cluster-20261022_154111-smoke2	smoke2	1 + 1	PASS, 686 + 176 tx/s	acceptance.sh end to end
cluster-20261022_155429-prod50w	prod50w, late, 60/120 s	25 + 25	PASS, rebalance 448 ms	rebalance experiment 1
cluster-20261022_160210-prod50w-fast	prod50w-fast, 60/120 s	25 + 25	PASS, acquire p99 1.17 ms	rebalance experiment 2
cluster-20261022_161118-prod50w-u-fast	prod50w-u-fast, 60/120 s	25 + 25	PASS, reserve p99 1.46 ms	rebalance experiment 3

Every run's report.txt, compare.txt and per-host logs are under locktest/results/ on the development machine; the hosts keep the same directories under ~/4369d/locktest/results/. The 2026-10-14 baselines are run-20261014_092943 (prod50w), run-20261014_103334 (prod50m) and run-20261014_083012 (hot50) from cust004:~/lt-20261014/locktest/results, copied to locktest/baseline/2026-10-14/.