Bug #11645
improve FWD Web Client ping-pong and timeouts
90%
History
#2 Updated by Constantin Asofiei 2 months ago
- Serban's findings at #11344-8 - the browser can throttle simple
setInterval, thus ping-pong may not be performed.
The old value of 30000ms was dangerous because of this: https://medium.com/@adithyaviswam/overcoming-browser-throttling-of-setinterval-executions-45387853a826
- we need to move the ping-pong to the FWD Java client side, so it does not rely on javascript to send it
- we need to move the other
setIntervalusage (like for reconnect) to Worker threads
#3 Updated by Teodor Gorghe about 2 months ago
- Assignee set to Teodor Gorghe
- Status changed from New to WIP
#4 Updated by Teodor Gorghe about 2 months ago
- Moved websocket ping-pong timer to server. Fixed additional bugs related with setInterval.
- As I understand about Workers setInterval from https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API, it just separates the work from main thread to a background thread. The main limitation is that it can't access DOM outside main javascript thread.
- The only scenario left which was feasable is the
pingPongWatcher, which has been removed on 11645a due to the logic move on the FWD client. - Other usages of
setIntervallikep2j.sso_reauth.js,p2j.perf_tests.js, etc., has been improved to reduce browser throttling (also some bug fixes onp2j.sso_reauth.js, for example, the case when a auth token expires).
#5 Updated by Teodor Gorghe about 2 months ago
Also, I have analyzed the case when these timers overlap and I don't see an issue when this may occur. The alternative was to use setTimeout in a loop to avoid callback overlapping, but it doesn't seem the case.
I have also analyzed the case when the connection gets torn down and as I can see in the PushMessagesWorker, if FWD client fails to send a message, puts it back in front to the message queue. Same instance of PushMessagesWorker is used for the restored websocket connection, so, it seems that nothing is lost.
On the other hand, if the browser client has something to sent, but in the meanwhile, the connection gets dropped, that message is lost.
#6 Updated by Teodor Gorghe about 2 months ago
- % Done changed from 0 to 100
#7 Updated by Sergey Ivanovskiy about 2 months ago
Have you tested the following scenario: the Java and JS web clients are running on different hosts, and the JS client loses connection due to local network issues (faults, interface changes, switching to Wi-Fi)? The JS client should reconnect to the Java client if the issue clears up within a known time interval.
#8 Updated by Teodor Gorghe about 2 months ago
On different hosts no, but I have tested the case when the browser tab enters in sleep and when the actual websocket connection gets torned down (chrome://discards freeze button).
I shall do the test using a remote host (for example, port forwarding the connection through devsrv01 local interface and loop it back using ssh port forwarding). This achieves ~500 ms ping.
#9 Updated by Sergey Ivanovskiy about 2 months ago
It makes sense to log in to the web client on the VM in order to simulate local network issues. The reconnect dialog should appear, and the connection should be restored once the simulated network issues are resolved.
#10 Updated by Teodor Gorghe about 1 month ago
Rebased 11645a to trunk rev 16695. Some work was done to integrate 11630a changes.
Also, I have committed the revision 16697:- Fixes after rebase. Replaced websocket protocol stream implementation with a chunked approach because we cannot interleave ping messages inside fragmented messages (RFC6455 5.4 page 35).
#11 Updated by Teodor Gorghe about 1 month ago
- Status changed from WIP to Review
- reviewer Sergey Ivanovskiy added
Sergey, can you review the changes?
I have done the testing with the ping (checked network interaction in the dev tools with some parameters which allows to monitor the behavior easily), including connection restoration.
#12 Updated by Teodor Gorghe about 1 month ago
Also, Constantin, I haven't documented here.
In RFC6455, there is already a ping-pong protocol specification, which reserves some special control frames for ping-pong implementation on protocol level.
- already implemented in Jetty.
- it is handled by the network stack. Doesn't require the javascript to be alive, so a browser tab can be in freeze.
- javascript side doesn't know when a ping message arrives, since this is handled exclusively by network stack. We need this ping-pong system to whatever check if a connection is alive, if not, we need to restore it.
- a proxy might keep the connection alive between server (FWD client) and proxy, but the other end might be closed.
This is the reason why I have kept the ping-pong own implementation, but I have switched the roles: FWD client now send the custom PING message and the browser (javascript application) responds with custom PONG.
#13 Updated by Sergey Ivanovskiy about 1 month ago
Teodor Gorghe wrote:
Sergey, can you review the changes?
I have done the testing with the ping (checked network interaction in the dev tools with some parameters which allows to monitor the behavior easily), including connection restoration.
Yes. I can review.
#14 Updated by Sergey Ivanovskiy about 1 month ago
Teodor, please run fwd-code-review for your changes. There are suspected issues:
MAJOR - functional WebClientProtocol.startKeepAlivePing: the echo comes from the browser's onmessage, so the deadline measures processing lag, not liveness, and outbound writes no longer buy tolerance (they used to reset Jetty's endpoint idle timeout via notIdle()). Deadline is min(3 × pingPongInterval, idleMs) — 90 s at defaults however large socketTimeout is. Reachable from 4GL today: HTML-BROWSER:PRINT() blocks the only thread that can answer, and past deadline + watchdog (~3.5 min) WatchdogTimer exits the client JVM, losing the session and its record locks. - performance same method: draining compares against a single previous sample, so a flat-or-growing queue is never draining, and the fall-through ping is appended to the tail of that same FIFO. On web CHUI, MSG_DRAW is one-way and this revision removed the client's outbound ping, so an idle-input report stream has zero inbound traffic → teardown on tick 6 (~180 s) of a perfectly-receiving peer. Sharper variant: one large in-flight message samples as queued == 0 → teardown at ~90 s. - functional same method: the deadline is hard-coded to pingPongInterval, silently redefining that key from "ping cadence" to "tolerated silence", with no separate knob and no way to disable. Confirmed against shipped configs — hotel_gui (600 s tolerance → 30 s deadline, while the browser still thinks it has ~5 min) and counteract (1 h → 90 s). - functional WebPageHandler.WebPageKeysProvider: pingPongInterval reaches the page unclamped while both Java readers now sanitise <= 0. With 0 (or -1, the convention of every neighbouring key) the page derives 100 ms and maxLostPings=2 → permanent reconnect loop ~300 ms after each connect. No schema validation anywhere. A regression: pre-change the same value was harmless. MINOR — grace path degrades liveness to "socket accepts bytes" and is self-suppressing (~240 s worst case vs the documented 90 s) _security_; MIN_PING_INTERVAL floor applied after the idle clamp, breaking the stated invariant; the keystroke-swallowing fix is incomplete and the disposal itself re-enables swallowing in the one case it changes; StatusCode.SHUTDOWN makes every reap log as "browser tab/window closed" at SEVERE, twice; pushWorker read off-lock without volatile; javadoc names the non-existent client/web/webSocketTimeout; MAX_IDLE_TIME javadoc overgeneralises (/api, network-test sockets still use it as a plain idle timeout); 2 style nits (missing blank line, 109-vs-110 separator). Two adjudications worth your attention: The headline finding was wrong. Two reviewers independently "verified against Jetty bytecode" that session.close(SHUTDOWN) half-closes to OSHUT and never delivers onClose, so no cleanup runs. I checked the bytecode myself: CloseStatus.isOrdinary covers only 1000, 1005 and ≥3000, so 1001 is abnormal — onOutgoingFrame sets CLOSED, returns true, and sendFrame → closeConnection → abort() → onClose. Rejected, along with the "locks linger 5 minutes / forever" claims built on it. Out of scope but material: while refuting the SSO-grace finding, a reviewer traced SsoTokenManager.forceLogout switching into the target session's context and then calling killSession on it — which trips the self-kill guard and throws into a silent catch. If that holds, the server-side SSO force-logout is dead code (and this revision's client-side teardown is genuinely load-bearing, vindicating its javadoc). It's in unchanged files, so I left it out of the report; it deserves its own issue and a second pass, as I haven't verified it independently.
I will review 11654a tomorrow.
#15 Updated by Sergey Ivanovskiy about 1 month ago
The first major is a real issue. I used again Claude Code:
REVIEW FINDING (abridged) "because the echo is emitted from the browser's onmessage handler, the deadline measures the client's inbound message-processing lag rather than its liveness - a page whose event loop is blocked [...] is declared 'gone or its page is no longer running' and closed. Outbound activity no longer buys any tolerance [...] Concrete blocker reachable from converted 4GL today: HTML-BROWSER:PRINT() [...] blocks the page task queue [...] a dialog left open past deadline + watchdog (~3.5 min at defaults) makes WatchdogTimer System.exit(-1) the client JVM and lose the FWD session and its record locks." FILES src/com/goldencode/p2j/ui/client/driver/web/WebClientProtocol.java src/com/goldencode/p2j/ui/client/driver/web/PushMessagesWorker.java ------------------------------------------------------------------------------------------------ 1. ROOT CAUSE ------------------------------------------------------------------------------------------------ Before r16675 a session survived on *either* direction of traffic. Jetty's EndPoint idle timeout is reset by every successful write (SocketChannelEndPoint.flush() -> notIdle()), so a client the server was still pushing output to was never reaped, no matter how long it had been silent. r16675 replaces that with a strictly inbound deadline: lastDataNanos is stamped only from onMessage (WebClientProtocol.java:463 and :478) and the tick closes the session when (now - lastDataNanos) > dataIdleMs. The only thing that can refresh it is the page's own echo of MSG_SERVER_PING, which is dispatched from the JS onmessage handler - i.e. from the page task queue. Any page whose task queue is blocked therefore looks dead, even though its host, socket and JVM are all healthy and the server's own writes are still completing normally. The queue-depth grace does not cover it: PushMessagesWorker.sendMessage() polls the message off the deque *before* writing (PushMessagesWorker.java:304), so getQueueSize() reads 0 while the write is in flight, stuckTicks never engages, and teardown happens at the bare deadline.
#16 Updated by Teodor Gorghe about 1 month ago
I will investigate!
#17 Updated by Teodor Gorghe about 1 month ago
Sergey, this is real and the part is that this is preexisting in trunk, because the browser ping-pong control timer was blocked by this browser print dialog.
I am finding a solution.
#18 Updated by Sergey Ivanovskiy about 1 month ago
- File proposal-issue-minor-functional-1.txt
added - File proposal-issue-minor-functional-2.txt
added - File proposal-issue-major-functional-1.txt
added - File proposal-issue-major-functional-2.txt
added - File proposal-issue-major-functional-3.txt
added - File proposal-issue-major-performance-1.txt
added - File proposal-issue-minor-functional-4.txt
added - File proposal-issue-minor-functional-5.txt
added - File proposal-issue-minor-functional-6.txt
added - File proposal-issue-minor-functional-3.txt
added
Teodor Gorghe wrote:
I will investigate!
There is another issue like ping pong cadence related to new hard coded constants and java web client logic. Please use Claude Code to assist in this work.
Fix proposals for the r16675 code review¶
Wrote 11 proposals in .tmp/, one per non-style finding of .tmp/code-review.textile. The two [MINOR] style findings - the GuiWebDriver.init blank line and the p2j.sso_reauth.js separator width - were skipped as requested, though the first is incidentally removed by proposal-issue-major-functional-3.
| File | Finding |
|---|---|
proposal-issue-major-functional-1.txt |
inbound-only deadline; outbound activity buys no tolerance |
proposal-issue-major-performance-1.txt |
backlog "draining" heuristic tears down healthy sessions |
proposal-issue-major-functional-2.txt |
deadline derived from pingPongInterval, ignoring configured idle tolerance |
proposal-issue-major-functional-3.txt |
PING_PONG_INTERVAL exported unclamped by WebPageKeysProvider |
proposal-issue-minor-security-1.txt |
backlog grace forges lastDataNanos with no peer evidence |
proposal-issue-minor-functional-1.txt |
MIN_PING_INTERVAL floor applied after the idle clamp |
proposal-issue-minor-functional-2.txt |
p2j.sso_reauth.js keystroke swallowing still occurs |
proposal-issue-minor-functional-3.txt |
StatusCode.SHUTDOWN reused for heartbeat expiry |
proposal-issue-minor-functional-4.txt |
pushWorker read off-lock, field not volatile |
proposal-issue-minor-functional-5.txt |
javadoc names non-existent client/web/webSocketTimeout |
proposal-issue-minor-functional-6.txt |
ConfigItem.MAX_IDLE_TIME javadoc overreaches |
Naming note: the review has no severe findings and one security finding, so that file uses -security- in the category slot rather than forcing it into functional|performance. Numbering restarts per severity+category pair, in review order.
Two things worth knowing before acting on these.
The five keep-alive findings are one rework, not five patches¶
Their fixes interact - one review bullet asks for outbound-write evidence to extend tolerance while another warns that outbound evidence must not substitute for peer liveness. That is resolved with two clocks: lastDataNanos stays inbound-only and sets the hard deadline; completed outbound writes (a new PushMessagesWorker.getTransmittedCount(), which unlike getQueueSize() sees the in-flight message) postpone it by a bounded, absolutely-capped grace. Each file owns a distinct mechanism, proposal-issue-major-performance-1 carries the composed tick body, and each states its dependencies. In particular, major-functional-1 must not land without minor-security-1 - the grace would be unbounded.
Verification is against the code, not the review text¶
Confirmed in the tree: the ConfigItem constructor contract behind the wrong key name; the three-reader split on pingPongInterval (WebPageHandler.java:251 unclamped vs. the two drivers); all three addWebSocketHandler registrations plus ReportWebServer.java:148 setting maxIdleTime to 36000000; that client/web/ slash notation exists only on the three lines this diff added; and that p2j.keyboard.js:319-336 already contains the exact listener-removal code needed for the SSO fix - inlined in init() with no way to call it, which is why extracting it is a refactor rather than new logic.
Nothing was compiled or run, so the code hunks are proposals rather than verified patches. Each file ends with concrete verification steps, and several note that the interval/deadline arithmetic has no test coverage at all today and would be worth extracting into a testable static helper.
The following changes
=== modified file 'src/com/goldencode/p2j/ui/client/driver/web/res/p2j.js'
--- old/src/com/goldencode/p2j/ui/client/driver/web/res/p2j.js 2026-05-26 07:34:53 +0000
+++ new/src/com/goldencode/p2j/ui/client/driver/web/res/p2j.js 2026-07-30 10:44:18 +0000
@@ -4,7 +4,7 @@
**
** Copyright (c) 2014-2026, Golden Code Development Corporation.
**
-** -#- -I- --Date-- ------------------------------Description----------------------------------
+** -#- -I- --Date-- ---------------------------------------Description----------------------------------------
** 001 MAG 20140110 First version.
** 002 MAG 20140204 Prevent back navigation. Disable browser context menu.
** 003 MAG 20140702 Add tty.clipboard module.
@@ -90,6 +90,7 @@
** changeDisplayStyle/changeVisibilityStyle. Removed unnecessary async from
** overrideDialogStyle/overrideTitleStyle. Added cachedTitle guard to setTitle.
** Added buttonsDiv null guard to setButtons.
+** 035 TG 20260728 Drove the loading progress bar easing from elapsed time instead of a tick count.
*/
/*
** This program is free software: you can redistribute it and/or modify
@@ -686,7 +687,9 @@
}
loadingProgressBar.set({"value" : value});
var v0 = loadingProgressBar.get("value");
- var time = 1;
+ // the easing below is driven by elapsed wall-clock time rather than by a tick count, so a
+ // throttled or delayed timer still yields the same curve instead of advancing more slowly
+ var started = Date.now();
if (progressTimer)
{
clearInterval(progressTimer);
@@ -722,8 +725,8 @@
progressTimer = null;
return;
}
+ var time = Math.max(1, (Date.now() - started) / 100);
var v = v0 + (v1 - v0) * ( 1 - 1 /(1 + Math.log2(time)));
- time++;
loadingProgressBar.set({"value" : v});
},
100);
are good!
#19 Updated by Sergey Ivanovskiy about 1 month ago
Updated the previous note as it was broken partly.
#20 Updated by Teodor Gorghe about 1 month ago
Sergey, this is my bad.
I have rebased the branch, but I have forgot to push it.
I have pushed it and please check 11645a/r16697.
#21 Updated by Sergey Ivanovskiy about 1 month ago
- File proposal-issue-major-functional-1.txt
added - File proposal-issue-minor-performance-1.txt
added - File proposal-issue-minor-functional-7.txt
added - File proposal-issue-minor-functional-6.txt
added - File proposal-issue-minor-functional-5.txt
added - File proposal-issue-minor-functional-4.txt
added - File proposal-issue-minor-functional-3.txt
added - File proposal-issue-minor-functional-2.txt
added - File proposal-issue-minor-functional-1.txt
added - File proposal-issue-critical-functional-1.txt
added
Code review and fix proposals - r16675 (server-driven websocket keep-alive)¶
Scope¶
Reviewed the branch diff against trunk: 15 files, ~830 added and ~300 removed lines. The revision does three things: it moves the websocket keep-alive heartbeat from the JS client to the server (new MSG_SERVER_PING, a server-side Timer in WebClientProtocol, dead-peer detection closing with private code 4001); it replaces websocket continuation frames with application-level MSG_CHUNK messages reassembled in the browser; and it hardens the SSO re-auth countdown (deadline-based ticks, a grace window, and a page reload on expiry in place of the deleted showSessionExpiredPage).
Modified areas: com.goldencode.p2j.ui.client.driver.web (5 java + 3 js), com.goldencode.p2j.ui.client.gui.driver.web (2 java + 1 js), com.goldencode.p2j.ui.client.chui.driver.web (1 java), com.goldencode.p2j.web (1), com.goldencode.p2j.util (1), com.goldencode.p2j.main (1).
Domains reviewed: fwd-web-gui-driver, fwd-chui-driver-streams, fwd-server-infrastructure, fwd-web-login-process.
Method¶
Seven parallel review passes (style, security, performance, and one functional pass per matched domain) produced 21 candidate findings. Each of the 16 non-style findings was then handed to an independent adversarial challenge pass instructed to identify the concrete trigger in the current code, or reject. Ten were confirmed, six rejected as theoretical. The challenge pass also downgraded three findings and rewrote one whose causal narrative was wrong.
Nothing was compiled or run. The proposals are code hunks with verification steps, not verified patches.
Confirmed findings¶
| Severity | Category | Location | Finding |
|---|---|---|---|
| CRITICAL | functional | WebClientProtocol.sendBinaryMessageStreamed |
A MSG_CHUNK transfer is not bound to a session; its tail is delivered to the next one |
| MAJOR | functional | p2j.socket.handleCloseEvent |
resetChunkTransfers() is unreachable on every client-initiated close |
| MINOR | functional | p2j.socket.collectChunk |
The MAX_CHUNK_BYTES abandon path poisons the transfer it abandons |
| MINOR | functional | p2j.sso_reauth.handleSsoReauth |
The expiry reload terminates the session it claims to re-attach to |
| MINOR | functional | p2j.sso_reauth.handleSsoReauth |
The reload raises the leave-confirmation; "Stay" freezes the overlay |
| MINOR | functional | p2j.sso_reauth.handleSsoReauth |
The expiry path leaves the IdP popup open as an orphan |
| MINOR | functional | PushMessagesWorker.getQueueSize |
Dead method whose javadoc documents a guard that was never built |
| MINOR | functional | WebClientProtocol.sendLock |
Field javadoc still documents the removed partial-frame contract |
| MINOR | functional | ConfigItem.MAX_IDLE_TIME |
Javadoc describes a page export that no shipped profile performs |
| MINOR | performance | WebClientProtocol.sendBinaryMessageStreamed |
The streamed payload accumulates in the push queue |
Five [MINOR] style findings (javadoc line length, two blank-line issues, header separator width in five files, one stray blank line) were reported without challenge, as style is judged against the rulebook rather than reachability.
The critical finding¶
queueChunk validates each piece against the current session and keeps no record of which session the transfer began on. Two independent routes carry a transfer's tail onto a later connection: the push queue survives the swap, because startPushWorker reuses the existing worker and onConnect never calls stopPushWorker (the sole caller is onClose's active-session branch, which the session != this.session guard skips for a replaced session); and the producer thread still inside the streaming loop simply observes the new session on its next call. This is the designed ordering on a half-open link - the page gives up after maxLostPings periods, roughly half the server idle timeout, so onConnect(new) precedes onClose(old) - and the endpoint really is the same object across a reconnect, since GenericWebSocketCreator returns the one GuiWebSocket instance for every upgrade.
On the page, collectChunk keys only on the transfer id with no notion of where a transfer starts, so an arriving tail begins a fresh reassembly and the piece carrying the last-flag re-enters messageHandler with whole[0] set to raw pixel or font data. 0x85 is MSG_QUIT, which calls doRedirectToLogoutPage() - a spurious logout roughly one time in 256.
Rejected findings¶
Recorded so they are not re-raised. Each was rejected after an attempt to construct a concrete trigger failed.
| Finding | Why rejected |
|---|---|
64 KiB CHUNK_PAYLOAD_SIZE caps WAN throughput |
Jetty 12 completes the send callback on flush into the kernel socket buffer, not on peer ACK, and queueChunk does not block per chunk - so the "128 serialized round trips" mechanism does not exist. Restoring 1 MiB would also mean per-piece humongous allocations |
Per-chunk synchronized (lock) contention |
The block holds only a null check, isOpen(), six byte stores and a non-blocking offerLast; the socket write happens on the worker thread under a different monitor |
| JS reassembly costs 2x memory | The previous continuation-frame path had the browser buffer and materialize the whole message internally - the same peak and the same copy, just below the JS boundary |
| JS per-message preamble runs 128x per image | Measured at 0.18 ms per image for the whole timer-op mix; feeding the silence watchdog per piece is the deliberate design |
ConfigItem.PING_PONG_INTERVAL javadoc omits clamping |
No ConfigItem javadoc in the file documents clamping, including two with the identical rule; the clamping is fully documented where it is implemented |
effectivePingPongInterval can return below MIN_PING_PONG_INTERVAL |
Requires webSocketTimeout under 3000 ms, which is below CLIENT_RESPONSE_TIMEOUT and self-defeating; the code warns for exactly this case, so it is signposted acceptance |
Fix proposals¶
One file per non-style finding, in .tmp/. Each states root cause with verified line references, concrete code hunks, dependencies, and verification steps.
| File | Severity | Finding |
|---|---|---|
proposal-issue-critical-functional-1.txt |
CRITICAL | MSG_CHUNK transfer not bound to a session |
proposal-issue-major-functional-1.txt |
MAJOR | resetChunkTransfers() unreachable on client-initiated closes |
proposal-issue-minor-functional-1.txt |
MINOR | MAX_CHUNK_BYTES abandon path poisons the transfer |
proposal-issue-minor-functional-2.txt |
MINOR | SSO expiry reload terminates the session |
proposal-issue-minor-functional-3.txt |
MINOR | Leave-confirmation on the expiry reload |
proposal-issue-minor-functional-4.txt |
MINOR | Orphaned IdP popup |
proposal-issue-minor-functional-5.txt |
MINOR | getQueueSize dead code |
proposal-issue-minor-functional-6.txt |
MINOR | sendLock javadoc |
proposal-issue-minor-functional-7.txt |
MINOR | ConfigItem.MAX_IDLE_TIME javadoc |
proposal-issue-minor-performance-1.txt |
MINOR | Streamed payload accumulates in the push queue |
The three chunk findings are one change¶
The critical finding, the major one, and minor-functional-1 all end at the same place: a piece that arrives with no matching reassembly in progress is dispatched with an arbitrary payload byte as its message type. The common remedy is a first-piece flag on the chunk header, so the peer refuses to begin a reassembly without it.
/** MSG_CHUNK flags bit 1: this piece opens the transfer. */ public static final byte CHUNK_FLAG_FIRST = 0x02;
if (!parts)
{
if (!first)
{
p2j.logger.error("Discarding orphaned message piece, transfer id " + id);
return null;
}
parts = chunkTransfers[id] = [];
}
The other two proposals state this as a hard dependency and are not safe to land alone: clearing state on close, or abandoning an oversized transfer, both leave a suffix that then starts a new reassembly - the same defect wearing a different hat. If only one change from this review is taken, take the first-piece flag.
The critical proposal additionally binds each transfer to a connection generation, which stops the producer feeding a dead transfer, and optionally purges queued output on session replacement.
The three SSO findings are one edit¶
minor-functional-2, -3 and -4 modify the same five lines of the countdown expiry branch and share one internalReload flag exported from p2j.socket. That flag makes onpagehide skip the quit-on-reload sessionStorage write and makes onbeforeunload skip the leave-confirmation, which together are what turn the reload into the re-attach the code comment already claims it performs. Apply them as a single change.
Risks worth reading before applying¶
Two proposals carry real risk and each offers a lower-risk variant alongside.
critical-functional-1section 2.3 - purging the push queue on session replacement drops draw output the peer has not yet received.onConnectresynchronizes throughinitRemoteClient, but this is the one part that changes behaviour for non-chunk traffic. Sections 2.1 and 2.2 are sufficient for correctness without it.minor-performance-1section 2.1 - the in-flight chunk semaphore introduces permit-lifecycle obligations on teardown and abandon paths, and a leak would hang the producer, which is worse than the footprint it fixes. Section 2.2 is an approximate fix at a fraction of the complexity, and is the recommended option unless a measurement shows the footprint matters.
minor-functional-2 section 2.1 also puts a deliberate hole in the long-standing quit-on-reload invariant. The hole is narrow - one caller, set immediately before an unload the page itself requested - but a future caller that sets the flag without unloading would leave a GUI page that no longer arms quit-on-reload.
Housekeeping¶
.tmp/ also contains proposal-issue-major-functional-2.txt, -major-functional-3.txt, -major-performance-1.txt and -minor-security-1.txt from an earlier review round. That round has already been applied - the code now has SESSION_HEARTBEAT_LOST = 4001 rather than a reused StatusCode.SHUTDOWN, a volatile pushWorker, client:web: key notation and clamping in WebPageHandler, which were exactly its subjects. Those four filenames imply membership in the current set and do not belong to it; delete or move them aside.
Fix proposals for the current r16675 code review¶
Wrote 10 proposals in .tmp/, one per non-style finding of .tmp/code-review.textile. The five [MINOR] style findings (javadoc line length, blank lines, header separator width) were skipped as requested.
| File | Severity | Finding |
|---|---|---|
proposal-issue-critical-functional-1.txt |
CRITICAL | MSG_CHUNK transfer not bound to a session; its tail is delivered to the next one |
proposal-issue-major-functional-1.txt |
MAJOR | resetChunkTransfers() unreachable on every client-initiated close |
proposal-issue-minor-functional-1.txt |
MINOR | MAX_CHUNK_BYTES abandon path poisons the transfer it abandons |
proposal-issue-minor-functional-2.txt |
MINOR | SSO expiry reload terminates the session it claims to re-attach to |
proposal-issue-minor-functional-3.txt |
MINOR | Expiry reload raises the leave-confirmation; "Stay" freezes the overlay |
proposal-issue-minor-functional-4.txt |
MINOR | Expiry path leaves the IdP popup open as an orphan |
proposal-issue-minor-functional-5.txt |
MINOR | PushMessagesWorker.getQueueSize dead, javadoc documents a guard never built |
proposal-issue-minor-functional-6.txt |
MINOR | WebClientProtocol.sendLock javadoc still documents partial frames |
proposal-issue-minor-functional-7.txt |
MINOR | ConfigItem.MAX_IDLE_TIME javadoc describes an export no profile performs |
proposal-issue-minor-performance-1.txt |
MINOR | Streamed payload accumulates in the push queue |
Numbering restarts per severity+category pair, in review order.
Stale files from the previous round¶
.tmp/ also contains proposal-issue-major-functional-2.txt, -major-functional-3.txt, -major-performance-1.txt and -minor-security-1.txt, which belong to an earlier review round (all written 2026-08-12 ~09:2x) and are not part of this set. That round appears to have been applied already - the code now has SESSION_HEARTBEAT_LOST = 4001 rather than a reused StatusCode.SHUTDOWN, a volatile pushWorker, client:web: key notation, and clamping in WebPageHandler, which were exactly its subjects. Delete them or move them aside before working from this directory; their filenames imply membership in the current set and they do not belong to it.
The chunk findings are one change, not four¶
critical-functional-1, major-functional-1 and minor-functional-1 all end at the same place: a piece that arrives with no matching reassembly in progress is dispatched with an arbitrary payload byte as its message type. The common remedy is the CHUNK_FLAG_FIRST bit specified in critical-functional-1 section 2.2 - the peer refuses to begin a reassembly without it. The other two proposals state that dependency and are not safe to land alone: clearing state on close, or abandoning an oversized transfer, both leave a suffix that then starts a new reassembly, which is the same defect wearing a different hat.
Likewise minor-functional-2, -3 and -4 edit the same five lines of the SSO expiry branch and share the internalReload flag; apply them as one edit.
Verification is against the code, not the review text¶
Confirmed in the tree while writing these: startPushWorker reuses the worker and onConnect never calls stopPushWorker, so the queue genuinely survives a session swap; closeWebSocketSafely nulls ws synchronously before the close event lands, so handleCloseEvent's this != ws early return really does skip the new cleanup; onpagehide writes the exitTheApplication sessionStorage object on a scripted reload and isRequiredToRedirect honours it for the whole watchdogTimeout window; getQueueSize has no caller anywhere in src/, test/ or tools/; and cleanupReauthUi is the only code path that closes reauthPopup.
Nothing was compiled or run, so the code hunks are proposals rather than verified patches. Each file ends with concrete verification steps. Two carry real risk and say so: critical-functional-1 section 2.3 (purging queued output on reconnect drops draw output not yet delivered) and minor-performance-1 section 2.1 (a permit leak would hang the producer, which is worse than the footprint it fixes) - both offer a lower-risk variant alongside.
#22 Updated by Sergey Ivanovskiy about 1 month ago
There is a hard-coded constant that should be fixed:
- put(ConfigItem.PING_PONG_INTERVAL, String.valueOf(20 * 60 * 1000)); + put(ConfigItem.PING_PONG_INTERVAL, String.valueOf(20 * 60 * 1000 / 3));
It looks like
WebConfigurationConstants.PING_PONG_INTERVAL can be used here instead.
Could you please explain the purpose of MSG_CHUNK sent from the Java web client to the browser web client? A similar message, MSG_PARTIAL, is sent in the opposite direction (from the browser web client to the Java web client). Why does sendPartial(ByteBuffer frame, boolean isLast) need to be changed?
#23 Updated by Sergey Ivanovskiy about 1 month ago
- Status changed from Review to WIP
- % Done changed from 100 to 90
I changed the task state to WIP - 90%.
#24 Updated by Teodor Gorghe about 1 month ago
I will take through the automated generated review finding.
Sergey, that section of sendPartial came from #11327, which splits that entire byte[] (like a big image), into chunks. At that time, I thought that using RFC6455 websocket partial frame specification (by Jetty's sendPartial method) is ideal, since there was just one main producer.
But now, when we have an async thread, the one which sends ping from the server side (from FWD client to browser), and we want to use a custom implemented PING-PONG to detect connection drops on browser-side, we can't have a send PING message and a big file transfer occurring in the same time because of websocket protocol limitation. The commit message also says it: "Fixes after rebase. Replaced websocket protocol stream implementation with a chunked approach because we cannot interleave ping messages inside fragmented messages (RFC6455 5.4 page 35)."
When using partial frames, the javascript side will notice these just when t*his ressembles, when the entire transmission finishes* (order of GB, or a very slow connection, < 2GB). In that time, the watchdog could trigger and kill the connection. By refactoring to use MSG_CHUNK, we make this being handled on application level, which allowed more control, especially a more accurate lastServerMessageAt measurement.
#25 Updated by Teodor Gorghe about 1 month ago
- % Done changed from 90 to 100
- Status changed from WIP to Review
Sergey Ivanovskiy wrote:
There is a hard-coded constant that should be fixed:
[...]
It looks likeWebConfigurationConstants.PING_PONG_INTERVALcan be used here instead.
Right.
Done in 11645a/r16698.
#26 Updated by Sergey Ivanovskiy about 1 month ago
Teodor Gorghe wrote:
Thanks for the detailed explanation, that clears it up. It makes sense to repeat here:I will take through the automated generated review finding.
Sergey, that section of
sendPartialcame from #11327, which splits that entirebyte[](like a big image), into chunks. At that time, I thought that using RFC6455 websocket partial frame specification (by Jetty'ssendPartialmethod) is ideal, since there was just one main producer.But now, when we have an async thread, the one which sends ping from the server side (from FWD client to browser), and we want to use a custom implemented PING-PONG to detect connection drops on browser-side, we can't have a send PING message and a big file transfer occurring in the same time because of websocket protocol limitation. The commit message also says it: "Fixes after rebase. Replaced websocket protocol stream implementation with a chunked approach because we cannot interleave ping messages inside fragmented messages (RFC6455 5.4 page 35)."
When using partial frames, the javascript side will notice these just when t*his ressembles, when the entire transmission finishes* (order of GB, or a very slow connection, < 2GB). In that time, the watchdog could trigger and kill the connection. By refactoring to use
MSG_CHUNK, we make this being handled on application level, which allowed more control, especially a more accuratelastServerMessageAtmeasurement.
- At the pure WebSocket Protocol Level (RFC 6455): Control frames (like PING / PONG) are explicitly allowed to be interleaved inside a fragmented message (FIN=0 sequence).
- At the Jetty / Application Level: While the RFC allows it, many higher-level frameworks or application-level protocol parsers cannot handle control messages interrupting an in-flight byte-stream without throwing an exception, dropping state, or corrupting client-side frame assembly.
Therefore the browser client parser couldn't cleanly handle a PING arriving in the middle of an RFC-level fragmented stream sendPartial, you chose to switch from protocol-level fragmentation to application-level chunking.
#27 Updated by Sergey Ivanovskiy about 1 month ago
Does it make sense to use a unified message type for chunks, replacing MSG_PARTIAL with MSG_CHUNK? The name MSG_PARTIAL was used because browsers lack APIs for protocol-level fragmentation (like Jetty's sendPartial).
#28 Updated by Teodor Gorghe about 1 month ago
Sergey Ivanovskiy wrote:
Thanks for the detailed explanation, that clears it up. It makes sense to repeat here:I know about PING / PONG control frames, but I can repeat the same note as #11645-12:
- At the pure WebSocket Protocol Level (RFC 6455): Control frames (like PING / PONG) are explicitly allowed to be interleaved inside a fragmented message (FIN=0 sequence).
- At the Jetty / Application Level: While the RFC allows it, many higher-level frameworks or application-level protocol parsers cannot handle control messages interrupting an in-flight byte-stream without throwing an exception, dropping state, or corrupting client-side frame assembly.
Therefore the browser client parser couldn't cleanly handle a
PINGarriving in the middle of an RFC-level fragmented streamsendPartial, you chose to switch from protocol-level fragmentation to application-level chunking.
- the main reason that we have PING-PONG is to detect if the connection has dropped.
- The protocol level PING-PONG makes the connection to not be dropped when in idle, but you can't see that from application.
#29 Updated by Teodor Gorghe about 1 month ago
Sergey Ivanovskiy wrote:
Does it make sense to use a unified message type for chunks, replacing MSG_PARTIAL with MSG_CHUNK? The name MSG_PARTIAL was used because browsers lack APIs for protocol-level fragmentation (like Jetty's sendPartial).
Didn't knew that there was an actual implementation, let me check.
#30 Updated by Teodor Gorghe about 1 month ago
Done in 11645a/r16699. MSG_PARTIAL implements the chunked transfer, but the direction was from browser to client. MSG_CHUNK was from client to browser, so it made sense to unify in a single name.
#31 Updated by Sergey Ivanovskiy about 1 month ago
Teodor Gorghe wrote:
Done in 11645a/r16699.
MSG_PARTIALimplements the chunked transfer, but the direction was from browser to client.MSG_CHUNKwas from client to browser, so it made sense to unify in a single name.
Agreed. Please check that you committed these revisions and applied the idea to use MS_CHUNK bidirectionally.
From my network view
Tree is up to date at revision 16697 of branch /home/sbi/secure/code/p2j_repo/p2j/active/11645a
#32 Updated by Teodor Gorghe about 1 month ago
I have already committed and the branch is at revision 16699:
------------------------------------------------------------ revno: 16699 committer: Teodor Gorghe <tg@goldencode.com> branch nick: 11645a timestamp: Fri 2026-08-14 06:05:20 +0000 message: Renamed MSG_CHUNK as MSG_PARTIAL. ------------------------------------------------------------ revno: 16698 committer: Teodor Gorghe <tg@goldencode.com> branch nick: 11645a timestamp: Thu 2026-08-13 10:37:58 +0000 message: Addressed code review #11645-21. ------------------------------------------------------------ revno: 16697 committer: Teodor Gorghe <tg@goldencode.com> branch nick: 11645a timestamp: Wed 2026-08-12 08:39:42 +0000 message: Fixes after rebase. Replaced websocket protocol stream implementation with a chunked approach because we cannot interleave ping messages inside fragmented messages (RFC6455 5.4 page 35).
Checkout (format: 2a)
Location:
checkout root: .
checkout of branch: bzr+ssh://localhost:2224/opt/secure/code/p2j_repo/p2j/active/11645a/
Related branches:
push branch: bzr+ssh://localhost:2224/opt/secure/code/p2j_repo/p2j/active/11645a/
#33 Updated by Sergey Ivanovskiy about 1 month ago
Yes, thanks, I got them. Please update to rev 16700, where I applied minor style fixes from the AI review.
#34 Updated by Sergey Ivanovskiy about 1 month ago
- File proposal-major-performance-1.txt
added - File proposal-major-functional-1.txt
added - File proposal-major-functional-2.txt
added - File proposal-major-security-1.txt
added - File code-review.textile
added
The committed rev 16699, 16700 looks good but the AI review found 4 major issues that seems worth to be taken into account.
Task #11645 - code review of r16696..r16699¶
Scope reviewed¶
Branch 11645a, diff r16695..r16699 - r16695 is the most recent trunk-nicked ancestor, so this is the full branch delta. 15 files, 2437 diff lines. The review was performed against r16699; the style fixes it produced are committed as r16700.
Subject of the change: the websocket keep-alive heartbeat moves from the JS client to the server (new MSG_SERVER_PING), large messages are reframed onto MSG_PARTIAL in both directions with first/last flags instead of native continuation frames, and the SSO re-auth expiry becomes an internal reload.
| Package | Files |
|---|---|
com.goldencode.p2j.ui.client.driver.web |
5 Java + 3 JS |
com.goldencode.p2j.ui.client.gui.driver.web |
2 Java + 1 JS |
com.goldencode.p2j.ui.client.chui.driver.web |
1 Java |
com.goldencode.p2j.main / com.goldencode.p2j.util / com.goldencode.p2j.web |
3 Java |
Method¶
Seven parallel review passes - style, security, performance, and one functional pass per matched domain (fwd-web-gui-driver, fwd-web-login-process, fwd-server-infrastructure, fwd-chui-driver-streams) - followed by 17 independent adversarial challenge passes, one per non-style finding. Each challenger was required to identify a concrete trigger in the current code or reject the finding.
Results at a glance¶
| Stage | Count |
|---|---|
| Raw findings from the review passes | 27 |
After dedup (5 reviewers converged on MessagesCollector.reset) |
27 to 23 non-style + 10 style |
| Non-style findings challenged | 17 |
| Confirmed | 13 |
| Rejected | 4 |
| Final: confirmed findings | 23 (5 MAJOR, 18 MINOR) |
| Category | MAJOR | MINOR | Total |
|---|---|---|---|
| functional | 2 | 5 | 7 |
| security | 0 | 2 | 2 |
| performance | 0 | 4 | 4 |
| style | 3 | 7 | 10 |
No MAJOR security or performance finding survived the challenge pass: the path-traversal finding was scoped down because its sink is pre-existing and untouched here, and the discardQueued accounting finding was downgraded because its impact is bounded.
Style fixes applied - committed as r16700¶
All 10 style findings are fixed. Verified: 0 violations remain among the lines this branch adds, both JS files pass node --check, and no double blank lines involving added lines remain.
WebClientProtocol.java:1860- javadoc line was 130 characters; rewrapped to four lines under 110.WebClientProtocol.java:1055- wrapped parameter continuation had 30 leading spaces where 32 are needed to align withint generation.WebClientProtocol.java:1968- two consecutive blank lines beforestopKeepAlivePing's javadoc.WebClientProtocol.java:312- added line was whitespace-only rather than empty (found by a second scan pass; the first missed whitespace-only lines).WebClientMessageTypes.java:501- two consecutive blank lines beforeMSG_SERVER_PING's javadoc.p2j.socket.js:644-648- the newMSG_PARTIALwire-format comment was orphaned above the unrelatedMSG_INVALIDATE_SELECTIONfield, with a whitespace-only line after it; moved ontoMSG_PARTIALitself, replacing the stale one-line comment there, and completed to document flags bit 1 (first piece) as well as bit 0 (last).p2j.socket.js:1897-for(was missing the required space after the keyword.p2j.socket.js:7032- whitespace-only line plus a double blank line afterbeginInternalReload.p2j.sso_reauth.js:435-452-cleanupReauthUi's javadoc, with its@param removeOverlay, had been separated from its subject by the newly insertedcloseReauthPopup; movedcloseReauthPopupabove the javadoc so each comment sits on its own subject.p2j.sso_reauth.js:16- history-entry line was 113 characters; rewrapped onto a continuation line.
History entries and copyright years were checked across all 15 files and are correct - every file has an entry, sequence IDs are successive, and where a file gained several entries only the first carries the ID.
MAJOR findings¶
Both are regressions introduced by this branch, and both have a written proposal.
- [MAJOR] functional
p2j.sso_reauthhandleSsoReauth- the expiry tick destroys the server-driven logout redirect.SsoTokenManager.handleInvalidTokenarmsforceLogoutatSsoTokenManager.java:518before callingsendSsoReauthat:539, while the module setsreauthDeadlineonly on receipt, so the server always kills the session first and the client tick always fires inside the shutdown. Nothing cancelscountdownIntervalon that path -cleanupReauthUiis reachable only from this module's own handlers, and there is no hook onMSG_SHUTTINGDOWN,MSG_QUITordoRedirectToLogoutPage- so the tick runsbeginInternalReload()+window.location.reload()mid-handshake. TheMSG_SHUTTINGDOWNack is lost soquit()burns the fullclientResponseTimeout, the pendingwindow.top.location.replace(logoutPage)is aborted, and the reloaded iframe points at an embedded servershutdownServer()has just torn down. The user lands on a browser network-error page instead of the login page. Before this branch the expiry path only replaced the document body, leaving the socket module intact to finish the redirect. Seeproposal-major-functional-1.txt. - [MAJOR] functional
WebClientProtocolonConnect- the newpushWorker.discardQueued()at:480-486throws away all pending server-to-browser output, justified by the comment "initRemoteClient below resyncs", butGuiWebDriver.initRemoteClient()is literally a no-op atGuiWebDriver.java:1033-1036; onlyChuiWebSimulatorresyncs. Trigger: a half-open socket where the browser's silence watchdog reconnects before Jetty deliversonClose, soonConnecttakes its session-replacement branch with a full backlog. The page was never reloaded, so the browser sendsMSG_PING_PONGnotMSG_PAGE_LOADEDand nothing requests a repaint - the canvas is left permanently stale. Note the discard has effect only on this path (elsewherepushWorkeris null), so it takes effect exactly where it is unsafe, and the cross-generation hazard it guards against is already covered twice byqueuePartial's generation check and the client'sresetChunkTransfers(). Seeproposal-major-functional-2.txt.
MINOR findings¶
Full text with verified triggers is in code-review.textile. Summarised here by area.
Reconnect and session-replacement races - one root cause, four faces¶
Five of the seven review passes converged independently on onConnect's session-replacement branch. It runs on the Jetty thread while the previous session's webWorker and the collector's asynchIOExecutor are still live, because startWebWorker no-ops when the worker exists and the replaced session's onClose is skipped by the session != this.session gate.
WebClientProtocol.MessagesCollector.reset- clears the plain unsynchronizedHashMap@s @partialMessagesandpayloadMessagesTypeswhile two other threads mutate them; a queuedAppendMessageTaskthen getsnullfrompartialMessages.get(msgId)and NPEs onchannel.write, and the catch handles onlyIOExceptionsosubmit()'sFutureTaskswallows it - a silently truncated upload. BecausenextMsgIdis page-scoped and restarts at 0, a stale queued task can also write into the new page's transfer id 0;PARTIAL_FLAG_FIRSTblocks that throughprocessPartialMesssagebut not through the executor queue.WebClientProtocol.onConnect- reads the non-volatile, unguardedcollectorfield from the Jetty thread while it is lazily assigned on thewebtaskworkerthread via the oneMSG_PARTIALswitch case that never takeslock.pushWorkerwas madevolatilein this same change;collectorwas not. A reconnect can seenull, skip the reset, and silently leak the open @FileChannel@s and temp files the reset exists to drop.PushMessagesWorker.discardQueued-queuedBytes.set(0L)races the worker's in-flight send, leaving the counter negative by one message size, andmessages.offerFirstruns aftermessages.clear()so a message from the superseded generation survives the discard. Downgraded from MAJOR: one worker thread means at most one message races per discard, every discard re-bases with an absoluteset, and pieces are capped at 64 KB - so the 32 MB guard is perturbed by tens of KB, not progressively raised. Seeproposal-major-performance-1.txt.WebClientProtocol.queuePartial- an abandoned streamed transfer is dropped with no notification to the waiter, soGuiWebSocket.createFontblocks forever:waitForResult(msgId)passes timeout0, which is implemented as "indefinitely", and nothing releases pending waiters on reconnect or close. Reachable throughFontManager's lazy cache-miss path for any font underdeploy/server/fonts. The same hang applies to non-streamedsendBinaryMessage+waitForResult(msgId)pairs -timeout 0waits are systemically unsafe across a reconnect.
Security¶
GuiWebSocket.processChannel- theMSG_FILE_UPLOADINGfilename reachesPaths.get()andFiles.newByteChannelwith nonormalize(), nogetFileName()and no containment check, andtmpDirends with a separator soconcatyields a traversable prefix. This is a real privilege-boundary crossing, not a user writing their own files: the spawned web client runs server-side, its websocket endpoint has no Origin check, and underOS_USER_OVERRIDEevery user's client runs as the sharedDEFAULT_OS_USER. FWD's ownUploadHandler:231-233already does the missing check. Scoped MINOR for this diff because the sink and its two twins are pre-existing and untouched - this branch only reworks the delivery route. Seeproposal-major-security-1.txt; raise as its own ticket.WebClientProtocol.MessagesCollector.processPartialMesssage- inbound reassembly has no aggregate size cap and no per-transfer expiry, the mirror image of theMAX_TRANSFER_BYTES/MAX_CHUNK_BYTEScaps this same branch added on the JS side. A page that setsPARTIAL_FLAG_FIRSTand never setsPARTIAL_FLAG_LASTgrows a temp file in the sharedjava.io.tmpdiruntil ENOSPC;onClosedoes not reset the collector, so while the socket is held open the window is attacker-controlled. Files are 0600 andDELETE_ON_CLOSE, so nothing survives process death. Related:processBinaryMessagegates onlength > PARTIAL_HEADER_SIZE, so a header-only 6-byte piece is dropped even carryingPARTIAL_FLAG_LAST- and the Java sender emits exactly such a terminator, so the guard should be>=.
Performance¶
WebClientProtocol.sendBinaryMessageStreamed-PARTIAL_PAYLOAD_SIZEis 64 KB where the previous path used 1 MB with a reused buffer, so a 3840x2160 image becomes 508 messages instead of 33 frames, each paying a lock acquisition on the same monitor asGuiWebEmulatedWindow.offer, two CAS ops, a deque node, a notify, and a latch plus park/unpark round trip. Nothing forces 64 KB - the server'ssetMaxBinaryMessageSizeis inbound-only and defaults to-1. Separately: Jetty 12.0.34 defaultsDEFAULT_MAX_FRAME_SIZEto 65536 with auto-fragment on, and a piece is 6 + 65536 = 65542 bytes, so every piece is split into a full frame plus a 6-byte continuation frame. The payload should be(64 * 1024) - PARTIAL_HEADER_SIZE.p2j.socket.jshandleMessageEvent- five timer operations per inbound message (new Date(),idleTimer.reset()'sclearInterval+setInterval, and asetTimeout/clearTimeoutpair) now run once per 64 KB piece instead of once per logical message, because the browser no longer reassembles natively. A 33 MB streamed image means ~2640 timer create/destroy operations and 528 extra macrotasks on the main thread. The code is unchanged; only the number of times it runs regressed.p2j.socket.jscollectChunk- holds every piece and then allocates the full-sizeUint8Arrayand copies into it, releasing pieces only after the copy loop, so peak transient main-thread memory is ~2x the message and the code's own limits permit ~128 MB. Alsototalis recomputed with a loop overpartsalthoughparts.bytesalready holds exactly that value. The original "extra copy the browser did not need" claim was dropped on verification - a browser reassembling fragments does the same accumulate-then-copy.p2j.socket.jssendPartialMessage- the payload is copied one byte at a time instead ofmsg.set(data.subarray(position, limit), PARTIAL_HEADER_SIZE); both operands are confirmed typed arrays so the replacement is exact. Measured on V8: 13.6 ms versus 1.2 ms per 10 MB. Low impact, but a correct one-line change on a line this branch already touches. The same loop exists inp2j.network_socket.js:294-298.
Documentation¶
ConfigItem.MAX_IDLE_TIME- the new javadoc says the setting applies "only whileclient:web:webSocketTimeoutis not positive", but no such key exists in any resolution path. The bootstrap option isclient:web:socketTimeoutand the directory node iswebClient/webSocketTimeout;BootstrapConfigresolves onlycategory():group():key()with no alias table. An administrator following the javadoc gets a silently ignored key while believing the timeout was raised. Especially misleading because neighbouring comments use the sameclient:web:form for keys that do exist.
Proposals¶
Four written proposals, in .tmp/rev16699/. Each covers problem, evidence and trigger, root cause, a recommended fix with code sketches, alternatives considered with reasons, risk, and a test plan.
| File | Finding | Status |
|---|---|---|
proposal-major-functional-1.txt |
SSO re-auth expiry tick destroys the logout redirect | MAJOR confirmed |
proposal-major-functional-2.txt |
onConnect discards the push backlog with no GUI resync |
MAJOR confirmed |
proposal-major-security-1.txt |
Unvalidated upload filename reaches a file-write sink | reported MAJOR, scoped MINOR here; pre-existing, own ticket |
proposal-major-performance-1.txt |
discardQueued breaks the queued-bytes accounting |
reported MAJOR, downgraded MINOR |
Because no MAJOR security or performance finding survived the challenge pass, the security and performance proposals cover the highest-ranked finding in each category - both of which were reported MAJOR before being adjusted.
Two proposals are coupled and should be settled together: functional-2 recommends removing the discardQueued() call entirely, which makes the performance-1 defect unreachable from onConnect but leaves the method broken. Fix or remove the method in the same change.
Rejected in the challenge pass¶
Recorded so they are not raised again. Full reasoning is at the end of code-review.textile.
WebClientProtocol.startKeepAlivePing- "the- tickMsslack narrows the configured tolerance to 60 s". The slack is load-bearing: becauseeffectivePingPongIntervalcaps the interval atidleMs/PING_INTERVAL_DIVISOR, the heartbeat deadline and Jetty's own idle deadline are the same instant, and the client echoes synchronously fromonmessageso the last inbound stamp sits one round trip after a tick. A strict>= dataIdleMscompare would miss that tick and fire a whole interval later - past the configured tolerance and past Jetty's idle close, losing the diagnostic 4001. The close actually lands at about 90 s with two unanswered pings; the proposed fix would be a regression. Residual below reporting threshold: the javadoc and WARNING describe the deadline as a single point, so the log can print "No websocket message received for 60000 ms (deadline=90000 ms)".p2j.socketmessageHandler- "the removed inboundMSG_PING_PONGcase leaves dead protocol traffic". The facts hold but the consequence does not:handleMessageEventunconditionally stampslastServerMessageAt, resetsidleTimer, zeroeslostPingsand callsreportPingRecovery()for any inbound frame, and the server stampslastDataNanosgenerically inonMessage- so the reply already delivers the side effect the handler existed for. A vestigial one-byte reply in unchanged code; hygiene, not a defect. Incidental: the comment atp2j.socket.js:5881is now stale.p2j.sso_reauthhandleSsoReauth- "the expiry path has no session-gone guard and no login-page fallback". Rejected as a duplicate of the MAJOR finding: same defect, same trigger, same one-line fix. Its supporting evidence was folded into that finding.PushMessagesWorker.awaitCapacity- "the per-limit wait is never notified betweenlowWaterandSTREAM_QUEUE_LIMIT, collapsing throughput to about 13 KB/s". The thresholds really are uncoupled, but the wait is timed at 5 s and on each wake the producer re-queues everything that drained, so the rate is socket-bound rather than notify-starved; holding the counter inside the un-notified band for a full 5 s needs a second unthrottled producer, meaning the socket is already saturated. Theweb-push-queue-low-byteskey also appears nowhere outside the Java source. Worth noting in passing: the javadoc invariant "a limit well below the high-water mark" is asserted in both files but unenforced, soSTREAM_QUEUE_LIMITwould be better derived fromhighWater()/lowWater()than hard-coded at 1 MB.
Artefacts¶
All under .tmp/rev16699/.
| File | Contents |
|---|---|
review-summary.textile |
this page |
proposal-major-functional-1.txt |
proposal - SSO re-auth expiry |
proposal-major-functional-2.txt |
proposal - push backlog discard |
proposal-major-security-1.txt |
proposal - upload filename traversal |
proposal-major-performance-1.txt |
proposal - queued-bytes accounting |
code-review.textile |
curated findings with full verified triggers |
code-review_uncurated.textile |
pre-challenge findings, for audit |
branch-r16695-r16700.diff |
the reviewed delta plus the style fixes, r16695..r16700 |
The reviewed diff was r16695..r16699; branch-r16695-r16700.diff additionally contains the r16700 style fixes. The two are identical apart from those fixes, and byte-identical to the reviewed diff apart from file mtimes in the +++ headers.
#35 Updated by Sergey Ivanovskiy about 1 month ago
- reviewer Hynek Cihlar added
- reviewer deleted (
Sergey Ivanovskiy)
Hynek, please review rev 16700. The changes looks good.
#37 Updated by Hynek Cihlar 20 days ago
- Status changed from Review to WIP
- % Done changed from 100 to 90
Code review 11645a revisions 16695..16700
I checked all the points below and they seem legit. I spent varying efforts on the points so please reasses.
- [MAJOR] functional
WebClientProtocol.onConnect: the newpushWorker.discardQueued()throws away the whole pending output queue on the justification "initRemoteClient below resyncs", butGuiWebDriver.initRemoteClient()is literally a no-op; onlyChuiWebSimulator.initRemoteClient()resyncs (clearCache/sendColorPalette/sendSwitchMode/triggerRepaint(true)). Trigger: the half-open reconnect — the page's silence watchdog runscloseWebSocketSafely+attemptToRestoreConnection, the server never saw the old CLOSE, so the new upgrade reaches the same endpoint object (GenericWebSocketCreatorreturns the singleton) andonConnectruns whilepushWorkeris still live. This is the only path where the discard has any effect, since the ordinary close path runsstopPushWorker(), which nullspushWorker. Two consequences: (1) on Web GUI nothing repaints —handleOpenEventtakes thepageLoadedWasSentbranch and sends onlyMSG_PING_PONG, which the server merely echoes — so every drawing message queued during the outage is lost and the canvas stays stale; (2) on both transports the discard also drops queued request/response traffic that no resync re-issues —StorageMessagingWebSocket.getStringValue/setStringValue/readClipboardValue/writeClipboardValueandshowChangePasswordDialogall block ingetResponse(msgId, 0)->waitForResult(msgId, 0), which arms no timer and loops onlock.wait(), so the calling thread parks permanently on an otherwise healthy session (reachable from 4GL via GET-KEY-VALUE / PUT-KEY-VALUE throughLocalStorageAccessor, and fromOsPassExpirationChecker). The pre-existingsessionInvalid/offerFirstrequeue machinery held exactly those messages for the replacement session. Recommend removing thediscardQueued()call, or gating it so only genuinely stale drawing output is dropped.
- [MAJOR] functional
PushMessagesWorker.discardQueued: it does not coordinate with the worker's failed-send requeue, so the discard is defeated. Interleaving:onConnect(Jetty thread, holdinglock) closes the old session and runsdiscardQueued()->messages.clear(); the worker, parked inlatch.await()insidesendMessage, sees that very failure, setssessionInvalid = trueand re-inserts the message withmessages.offerFirst(message)after the clear.onConnectthen doescollector.reset()andstartWebWorker()beforestartPushWorker(session), andrun()re-reads the session each iteration whilesendMessagehas no session or generation check — so the stale message is written to the brand-new socket ahead of the resync, defeating the generation guardqueuePartialenforces. The mirror interleaving is worse: ifsessionInvalid = truelands aftersetCurrentSession, nothing notifiessessionGuardagain and the worker parks on a healthy socket, stopping all client output including the keep-alive, until the browser's watchdog forces another reconnect. Fix by stamping the in-flight message with the connection generation and checking it insendMessage, mirroringqueuePartial.
- [MAJOR] functional
MessagesCollector.reset:partialMessagesandpayloadMessagesTypesare plainHashMap@s now mutated from three threads with no synchronization — @processPartialMesssageputs/gets on the webtaskworker thread (onMessage->webWorker.post->processBinaryMessage),AppendMessageTask.rungets on the collector's single-threadasynchIOExecutor, and the newreset()closes channels and clears both maps from the Jetty websocket thread viaonConnect.onConnect'ssynchronized (lock)does not help: neither other path takeslock, and the worker is not quiesced (the replaced-session path skipsonCloseentirely, andWebTaskWorker.kill()only interrupts plusjoin(1000)while the run loop finishes its current batch). Trigger: an inboundMSG_PARTIALtransfer in flight (drag-drop upload viauploadFileToJava->ReadFileChunkAndSend->SendMsg->sendPartialMessage, or anysend()payload aboveclient:web:maxBinaryMessage, e.g. a large paste) when a reconnect runsreset(): the executor thread then doespartialMessages.get(msgId)->null->channel.write(payloadData), an NPE not covered by the surroundingcatch (IOException)and buried in the discardedFuturefromsubmit, soAppendMessageTask.closenever runs and the fd and map entry leak. The load-bearing hazard is the unsynchronizedHashMap.clear()racing the worker'sput()— structural mutation of aHashMapfrom two threads can lose entries or corrupt the table. Routereset()throughwebWorker.post(or drain the executor first), and null-check the channel inAppendMessageTask.runregardless, sincereset()can always win the race against a queued task.
- [MAJOR] security
MessagesCollector.processPartialMesssage: the inboundMSG_PARTIALreassembly has no per-transfer cap, no aggregate cap and no expiry, whereas the same change added explicitMAX_TRANSFER_BYTES/MAX_CHUNK_BYTESplusdropTransfer/resetChunkTransferson the JS side. Trigger: the page sends pieces carryingPARTIAL_FLAG_FIRSTfor N distinct transfer ids and never setsPARTIAL_FLAG_LAST; each id opens a temp file plus aFileChannelunderjava.io.tmpdirand every further piece is appended without limit. The only releases arereset()on the nextonConnect(onClosedoes not reset) and process exit, both avoidable by holding the socket open. The per-process temp directory is mode 700, so there is no cross-user disclosure and the burned file descriptors only degrade the user's own spawned client; the cross-user leg is thatjava.io.tmpdiris the host's shared/tmp(no-Djava.io.tmpdiroverride on the spawn path, no quota), so one authenticated user can drive it to ENOSPC for every other user's client and for the FWD server. Two amplifiers: the new orphan-piece guard emits an unrate-limitedLOG.warningper frame onto the same filesystem, andasynchIOExecutoris a single-thread executor with an unbounded queue holding each received payload array, so a sender faster than the disk grows the client heap with no backpressure. Mirror the JS caps (per-transfer and aggregate bytes, a bound on concurrent transfer ids, an idle expiry) and reset the collector fromonCloseas well asonConnect.
- [MAJOR] functional
p2j.sso_reauth.js.handleSsoReauth: the countdown-expiry internal reload cannot "re-attach" a GUI session as its comment claims. On the reconnectWebClientProtocol.onConnectcalls exactly one driver hook,callbacks.initRemoteClient(), and the GUI implementation is a no-op. The reloaded page does sendMSG_PAGE_LOADED, butprocessBinaryMessageonly parks it inreceivedMessagesfor the one-shot startupwaitForResultinEmbeddedWebServerImpl.waitInitialization; noMSG_DESKTOP_RESIZEDis sent on load, anddesktopResizedonly redraws zooming or maximized windows.redrawZoomingWindows()is also the only caller ofclearPerWidgetCache/clearFullWindowCache, so the server still believes the browser holds hashes that died with the document. Result: the reloaded page attaches to a live GUI client and shows an empty virtual desktop — and is now also un-quittable, becauseinternalReloadsuppresses theonpagehideexitTheApplicationmarker thatisRequiredToRedirect()reads, so the old quit-on-reload path that used to release the session no longer runs. Concrete trigger with the session provably alive: the user completes sign-in,/reauthcb->processReauthCallbackclearsreauthInProgressand cancelsreauthTimeoutFuture(force-logout will now never fire), but the popup'spostMessageis dropped by theevent.origin !== serverOrigincheck (serverOriginfalls back towindow.location.originwhengetLogoutPage()does not parse, so a differing logout origin suffices); the user closes the popup,reauthInFlight()goes false, the grace is skipped and the tick reloads against a fully re-authenticated session. GiveGuiWebDriver.initRemoteClient()a real resync (clear both graphics caches and redraw all visible windows) before relying on a reload here, or keep the GUI expiry on a terminal path.
- [MINOR] functional
PushMessagesWorker.discardQueued:queuedBytes.set(0L)is not atomic with respect to the worker's dequeue-then-send, so the counter can go negative.onConnect(Jetty thread, holding onlylock) zeroes it while the worker holds onlysendLockand is parked inlatch.await()for a message it already removed withmessages.pollFirst(); on completionreleaseCapacity(msgSize)subtracts against the zeroed counter. While the value is negative, bothawaitCapacityoverloads return at their firstget(), raising the effective threshold of both the 32 MB high-water gate and the 1 MBSTREAM_QUEUE_LIMITby the bias. Bounded rather than unbounded: only one message is ever in flight, anddiscardQueuedre-floors to zero on every reconnect, so the bias cannot accumulate — but it should still be fixed by subtracting only what was actually removed, or by havingsendMessageskipreleaseCapacitywhen a discard intervened. Note the producer-side window is already closed: allpushMessagecall sites run insidesynchronized (lock).
- [MINOR] functional
WebClientProtocol.sendBinaryMessageStreamed: widening the handler tocatch (IOException | RuntimeException e)turns a real defect in the body stream into a silently malformed message. The trigger is the one the comment names and it is reachable end-to-end:GuiWebEmulatedWindow.encodeImagepasses the unclipped image dimensions todefineImageStreamedwhileVirtualScreenImpl.drawImageclips to the desktop size, soRawImageInputStreamindexes past its backing array and throwsArrayIndexOutOfBoundsExceptionfor any image larger than the desktop. The handler then queues a 6-bytelastpiece, socollectChunksees a complete transfer;p2j.screen.js'sdefineImagederives the body length from the header itself andsubarrayclamps silently, caching a short pixel array underkey. BecauseencodeImagealready registered the seal viaaddImageUsage/mapSealToUniqueIdbefore the send and returnsImageEncoding.HASHwith no pixels, this and every later draw of that image references pixels that never arrived, and the browser'sloadedImages.has(key)guard passes — a permanently garbled cached image, reported only at WARNING. Note this is genuinely new:RawImageInputStream.readdeclares noIOException, so the pre-change catch was inert on the image path and the AIOOBE used to propagate. Log at SEVERE and evict the image key (or register the seal only after the body is fully streamed).
- [MINOR] functional
p2j.socket.js.messageHandler: theMSG_SERVER_PINGecho goes throughsendNotification()->send(), which drops the message unlessisWebSocketConnected()holds — and that predicate includes the module-levelonlineflag, setfalseby the windowofflinelistener inme.initand only ever restored by the matchingonlinelistener; nothing re-validates it against the live socket. A browser whose OS reports no non-loopback interface therefore stops echoing while the websocket is perfectly usable (same-host deployment onhttps://localhost:7443/gui, including the virtual-desktop iframe, across an Ethernet/Wi-Fi drop or VPN flap).startKeepAlivePingarms its deadline on inbound silence alone (lastDataNanosis stamped only in the twoonMessageoverloads), so with the shipped defaults it closes a working session with 4001 after 60 s andonClosearms the process-killing watchdog. The two ends disagree silently:handleMessageEventresetsidleTimerand zeroeslostPingson every inbound frame, so the server's own pings keep the page's watchdog satisfied and the page shows nothing wrong until the close. Sinceonlineis still false on the new socket, the cycle repeats. Exempt the heartbeat echo from theonlinegate, or droponlinefromisWebSocketConnectedand rely onws.readyState, which is authoritative for this socket.
- [MINOR] functional
p2j.socket.js.beginInternalReload:internalReloadis set once and never cleared, suppressing both theonpagehidequit-on-reload arming and theonbeforeunloadconfirmation for the remaining life of the document.window.location.reload()only starts a navigation — the old document stays live and interactive until the new response commits — and this path exists precisely because the client may have just been killed, while the page is served by the client's own embedded Jetty whoseMAX_HTTP_IDLE_TIMEdefaults toINFINITE_TIMEOUT, so a stalled fetch can hold the flag true for an unbounded time on a page the user can still act on. Closing the tab in that window gives no leave confirmation and writes noexitTheApplicationmarker, so a same-tab return re-attaches silently instead of sendingMSG_QUIT. The change's own comment ("The interval is left running so a navigation the user declines is recoverable") contradicts itself: withinternalReloadtrue the beforeunload prompt is suppressed, so the user can never decline. Clear the flag on a bounded timer or onpageshow, or scope the suppression to the one unload that belongs to the reload.
PushMessagesWorker has two capacity gates over the same queuedBytes counter (PushMessagesWorker.java:247-302).
awaitCapacity() is the general gate: parks a producer at highWater() (32 MB default), releases at lowWater() (16 MB default). Two thresholds, 16 MB apart, so a producer that just drained does not immediately re-park.
awaitCapacity(long limit) is the new streamed-transfer gate. Parks at limit, releases at the same limit. No hysteresis. And it checks the same global queuedBytes, not the calling transfer's own bytes.
WebClientProtocol.queuePartial calls this gate with STREAM_QUEUE_LIMIT (1 MB, line 333), once per chunk of sendBinaryMessageStreamed (line 1058). That is the path GuiWebSocket uses to stream a screen image without buffering it whole (GuiWebSocket.java:1316).
Ordinary sends only go through the general gate, so unrelated output can fill the queue to 32 MB before anything throttles. If that backlog is already sitting there (slow browser, several sendPendingDrawingOps batches queued) and a new streamed transfer starts, its first queuePartial call sees queuedBytes already above 1 MB and parks before sending anything. With no hysteresis, it does not wake up again until the whole backlog, not just this transfer's share, drains below 1 MB. That is a deeper drain than the general gate ever requires (its own low-water release point is 16 MB).
- [MINOR] functional
WebClientProtocol.startKeepAlivePing: the javadoc states "the echo deadline isDATA_QUIET_MULTIPLIERintervals, or that idle timeout where it is longer, so a configured tolerance is never silently narrowed", but the deadline actually enforced isdataIdleMs - tickMs. The one-interval slack is intended and its rationale is in the inline comment — the defect is that the method javadoc asserts the opposite, and the logging carries the same inconsistency: with the shipped defaults (socketTimeout 90000, pingPongInterval 30000) the close is logged as "No websocket message received for 60003 ms (deadline=90000 ms)" and startup reports "heartbeat deadline=90000", both advertising a deadline the code does not apply — actively misleading when diagnosing an unexpected disconnect. Worth documenting in the same place: the slack costs a whole heartbeat and degenerates to a single heartbeat whenever thePING_INTERVAL_DIVISORcap binds, which is the case in both shipped profiles (default: ping at 30 s, close at 60 s; DEBUG_PROFILE: ping at 400 s, close at 800 s of a 1200 s tolerance), so one late or dropped echo tears the session down with no retry. A divisor of 4 would keep the slack and still leave a retry.
- [MINOR] style
WebConfigurationConstants.sanitizePingPongInterval: the new static methodssanitizePingPongIntervalandeffectivePingPongIntervalare inserted between interface constant fields (afterPING_INTERVAL_DIVISOR, beforeENABLE_DEBUG_LOGGING), violating the required member ordering — all data members must precede all methods. Move them after the last constant field.