FWD » Core Development » Runtime Infrastructure

• Serban's findings at #11344-8 - the browser can throttle simple setInterval, thus ping-pong may not be performed.

The old value of 30000ms was dangerous because of this: https://medium.com/@adithyaviswam/overcoming-browser-throttling-of-setinterval-executions-45387853a826

• we need to move the ping-pong to the FWD Java client side, so it does not rely on javascript to send it
• we need to move the other setInterval usage (like for reconnect) to Worker threads

• Moved websocket ping-pong timer to server. Fixed additional bugs related with setInterval.

• As I understand about Workers setInterval from https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API, it just separates the work from main thread to a background thread. The main limitation is that it can't access DOM outside main javascript thread.
• The only scenario left which was feasable is the pingPongWatcher, which has been removed on 11645a due to the logic move on the FWD client.
• Other usages of setInterval like p2j.sso_reauth.js, p2j.perf_tests.js, etc., has been improved to reduce browser throttling (also some bug fixes on p2j.sso_reauth.js, for example, the case when a auth token expires).

Also, I have analyzed the case when these timers overlap and I don't see an issue when this may occur. The alternative was to use setTimeout in a loop to avoid callback overlapping, but it doesn't seem the case.

I have also analyzed the case when the connection gets torn down and as I can see in the PushMessagesWorker, if FWD client fails to send a message, puts it back in front to the message queue. Same instance of PushMessagesWorker is used for the restored websocket connection, so, it seems that nothing is lost.
On the other hand, if the browser client has something to sent, but in the meanwhile, the connection gets dropped, that message is lost.

Have you tested the following scenario: the Java and JS web clients are running on different hosts, and the JS client loses connection due to local network issues (faults, interface changes, switching to Wi-Fi)? The JS client should reconnect to the Java client if the issue clears up within a known time interval.

On different hosts no, but I have tested the case when the browser tab enters in sleep and when the actual websocket connection gets torned down (chrome://discards freeze button).

I shall do the test using a remote host (for example, port forwarding the connection through devsrv01 local interface and loop it back using ssh port forwarding). This achieves ~500 ms ping.

It makes sense to log in to the web client on the VM in order to simulate local network issues. The reconnect dialog should appear, and the connection should be restored once the simulated network issues are resolved.

Rebased 11645a to trunk rev 16695. Some work was done to integrate 11630a changes.

• Fixes after rebase. Replaced websocket protocol stream implementation with a chunked approach because we cannot interleave ping messages inside fragmented messages (RFC6455 5.4 page 35).

Also, Constantin, I haven't documented here.
In RFC6455, there is already a ping-pong protocol specification, which reserves some special control frames for ping-pong implementation on protocol level.

• already implemented in Jetty.
• it is handled by the network stack. Doesn't require the javascript to be alive, so a browser tab can be in freeze.

• javascript side doesn't know when a ping message arrives, since this is handled exclusively by network stack. We need this ping-pong system to whatever check if a connection is alive, if not, we need to restore it.
• a proxy might keep the connection alive between server (FWD client) and proxy, but the other end might be closed.

This is the reason why I have kept the ping-pong own implementation, but I have switched the roles: FWD client now send the custom PING message and the browser (javascript application) responds with custom PONG.

Teodor Gorghe wrote:

Sergey, can you review the changes?

I have done the testing with the ping (checked network interaction in the dev tools with some parameters which allows to monitor the behavior easily), including connection restoration.

Yes. I can review.

Teodor, please run fwd-code-review for your changes. There are suspected issues:

  MAJOR
  - functional WebClientProtocol.startKeepAlivePing: the echo comes from the browser's onmessage, so the deadline measures
  processing lag, not liveness, and outbound writes no longer buy tolerance (they used to reset Jetty's endpoint idle timeout via
  notIdle()). Deadline is min(3 × pingPongInterval, idleMs) — 90 s at defaults however large socketTimeout is. Reachable from 4GL
  today: HTML-BROWSER:PRINT() blocks the only thread that can answer, and past deadline + watchdog (~3.5 min) WatchdogTimer exits
  the client JVM, losing the session and its record locks.
  - performance same method: draining compares against a single previous sample, so a flat-or-growing queue is never draining, and
  the fall-through ping is appended to the tail of that same FIFO. On web CHUI, MSG_DRAW is one-way and this revision removed the
  client's outbound ping, so an idle-input report stream has zero inbound traffic → teardown on tick 6 (~180 s) of a
  perfectly-receiving peer. Sharper variant: one large in-flight message samples as queued == 0 → teardown at ~90 s.
  - functional same method: the deadline is hard-coded to pingPongInterval, silently redefining that key from "ping cadence" to
  "tolerated silence", with no separate knob and no way to disable. Confirmed against shipped configs — hotel_gui (600 s tolerance →
  30 s deadline, while the browser still thinks it has ~5 min) and counteract (1 h → 90 s).
  - functional WebPageHandler.WebPageKeysProvider: pingPongInterval reaches the page unclamped while both Java readers now sanitise
  <= 0. With 0 (or -1, the convention of every neighbouring key) the page derives 100 ms and maxLostPings=2 → permanent reconnect
  loop ~300 ms after each connect. No schema validation anywhere. A regression: pre-change the same value was harmless.

  MINOR — grace path degrades liveness to "socket accepts bytes" and is self-suppressing (~240 s worst case vs the documented 90 s)
  _security_; MIN_PING_INTERVAL floor applied after the idle clamp, breaking the stated invariant; the keystroke-swallowing fix is
  incomplete and the disposal itself re-enables swallowing in the one case it changes; StatusCode.SHUTDOWN makes every reap log as
  "browser tab/window closed" at SEVERE, twice; pushWorker read off-lock without volatile; javadoc names the non-existent
  client/web/webSocketTimeout; MAX_IDLE_TIME javadoc overgeneralises (/api, network-test sockets still use it as a plain idle
  timeout); 2 style nits (missing blank line, 109-vs-110 separator).

  Two adjudications worth your attention:

  The headline finding was wrong. Two reviewers independently "verified against Jetty bytecode" that session.close(SHUTDOWN)
  half-closes to OSHUT and never delivers onClose, so no cleanup runs. I checked the bytecode myself: CloseStatus.isOrdinary covers
  only 1000, 1005 and ≥3000, so 1001 is abnormal — onOutgoingFrame sets CLOSED, returns true, and sendFrame → closeConnection →
  abort() → onClose. Rejected, along with the "locks linger 5 minutes / forever" claims built on it.

  Out of scope but material: while refuting the SSO-grace finding, a reviewer traced SsoTokenManager.forceLogout switching into the
  target session's context and then calling killSession on it — which trips the self-kill guard and throws into a silent catch. If
  that holds, the server-side SSO force-logout is dead code (and this revision's client-side teardown is genuinely load-bearing,
  vindicating its javadoc). It's in unchanged files, so I left it out of the report; it deserves its own issue and a second pass, as
  I haven't verified it independently.

The first major is a real issue. I used again Claude Code:

REVIEW FINDING (abridged)
   "because the echo is emitted from the browser's onmessage handler, the deadline measures the
   client's inbound message-processing lag rather than its liveness - a page whose event loop is
   blocked [...] is declared 'gone or its page is no longer running' and closed. Outbound activity
   no longer buys any tolerance [...] Concrete blocker reachable from converted 4GL today:
   HTML-BROWSER:PRINT() [...] blocks the page task queue [...] a dialog left open past deadline +
   watchdog (~3.5 min at defaults) makes WatchdogTimer System.exit(-1) the client JVM and lose the
   FWD session and its record locks." 

FILES
   src/com/goldencode/p2j/ui/client/driver/web/WebClientProtocol.java
   src/com/goldencode/p2j/ui/client/driver/web/PushMessagesWorker.java

------------------------------------------------------------------------------------------------
1. ROOT CAUSE
------------------------------------------------------------------------------------------------

Before r16675 a session survived on *either* direction of traffic. Jetty's EndPoint idle timeout is
reset by every successful write (SocketChannelEndPoint.flush() -> notIdle()), so a client the server
was still pushing output to was never reaped, no matter how long it had been silent.

r16675 replaces that with a strictly inbound deadline: lastDataNanos is stamped only from onMessage
(WebClientProtocol.java:463 and :478) and the tick closes the session when
(now - lastDataNanos) > dataIdleMs. The only thing that can refresh it is the page's own echo of
MSG_SERVER_PING, which is dispatched from the JS onmessage handler - i.e. from the page task queue.
Any page whose task queue is blocked therefore looks dead, even though its host, socket and JVM are
all healthy and the server's own writes are still completing normally.

The queue-depth grace does not cover it: PushMessagesWorker.sendMessage() polls the message off the
deque *before* writing (PushMessagesWorker.java:304), so getQueueSize() reads 0 while the write is
in flight, stuckTicks never engages, and teardown happens at the bare deadline.

I will investigate!

Sergey, this is real and the part is that this is preexisting in trunk, because the browser ping-pong control timer was blocked by this browser print dialog.

I am finding a solution.

Updated the previous note as it was broken partly.

Sergey, this is my bad.
I have rebased the branch, but I have forgot to push it.

I have pushed it and please check 11645a/r16697.