Bug #10485
App freeze due to lack of synchronization
100%
History
#1 Updated by Paula Păstrăguș 11 months ago
- Status changed from New to WIP
The application freeze occurs due to missing synchronization between PushMessagesWorker.sendMessage and the network test. Both threads (push worker and the download thread) invoke RemoteEndpoint.sendBytes() concurrently, even though it isn’t thread-safe. This race condition leads to IllegalStateException: Blocking message pending 10000 for BLOCKING, which prevents the latch from being released and causes the client to hang.
#2 Updated by Paula Păstrăguș 11 months ago
For additional details, please refer to entries #10428-24 through #10428-29
#4 Updated by Paula Păstrăguș 11 months ago
- Status changed from WIP to Review
- % Done changed from 0 to 100
- reviewer Hynek Cihlar, Sergey Ivanovskiy added
Created task branch 10485a.
Committed solution as rev 16135.
Hynek / Sergey please review.
#5 Updated by Hynek Cihlar 11 months ago
- % Done changed from 100 to 90
- Status changed from Review to WIP
Code review 10485a.
Please remove PushMessagesWorker.session and the related methods.
[ant:javac] /home/hc/gcd/p2j_repo/review/src/com/goldencode/p2j/ui/client/driver/web/NetworkTestHandler.java:219: error: unreported exception IOException; must be caught or declared to be thrown [ant:javac] webClientProtocol.sendBinaryMessageSync(message);
Sergey, please review.
#7 Updated by Paula Păstrăguș 11 months ago
Hynek Cihlar wrote:
Code review 10485a.
Please remove
PushMessagesWorker.session
If we remove the session, wouldn’t it also make sense to drop sessionInvalid and sessionGuard, since they only exist to manage the session state?
#8 Updated by Sergey Ivanovskiy 11 months ago
Hynek Cihlar wrote:
Code review 10485a.
Please remove
PushMessagesWorker.sessionand the related methods.[...]
Sergey, please review.
I will do this later but why not to change NetworkTestHandler? Is this issue is really related to #10428?
As this method is created for NetworkTestHandler usages and it was invoked via the different thread but all messages should be passed via PushMessagesWorker.
public void sendBinaryMessageSync(byte[] message)
{
synchronized (lock)
{
try
{
if (session.isOpen())
{
session.getRemote().sendBytes(ByteBuffer.wrap(message));
}
}
catch (IOException ex)
{
LOG.severe("", ex);
}
}
}
That was the current design violation.
#9 Updated by Hynek Cihlar 11 months ago
Paula Păstrăguș wrote:
Hynek Cihlar wrote:
Code review 10485a.
Please remove
PushMessagesWorker.sessionIf we remove the session, wouldn’t it also make sense to drop sessionInvalid and sessionGuard, since they only exist to manage the session state?
Hm, PushMessageWorker uses session for connection state control so it can't be removed. Ideally this should be moved to WebClientProtocol as well since the message sending is now there.
Please check whether the session state control doesn't pose another race condition issue.
#10 Updated by Sergey Ivanovskiy 11 months ago
public void sendBinaryMessageSync(byte[] message)
{
synchronized (lock)
{
try
{
if (session.isOpen())
{
session.getRemote().sendBytes(ByteBuffer.wrap(message));
}
}
catch (IOException ex)
{
LOG.severe("", ex);
}
}
}
this method of
WebClientProtocol is created for NetworkTestHandler usages and it was invoked via the different thread but it violates the design of PushMessagesWorker that was created in order that all messages should be passed via PushMessagesWorker. Is my understanding correct?
Greg, please review the code in the trunk, now Paula is trying to make this work via different threads.
#11 Updated by Paula Păstrăguș 11 months ago
Hynek / Sergey, I don’t think the solution I pushed is really viable. We’re synchronizing on the same lock that’s also used for reading keys, injecting keys, and other operations in WebClientProtocol. This can make the GUI feel sluggish, I noticed the slowdown when hovering over an image where a PS timer is defined to change it.
#12 Updated by Paula Păstrăguș 11 months ago
The race condition can only occur during a network test when a PS timer is active and its callback interacts with the WebSocket. I suggest to keep the retry mechanism for sending messages when an IllegalStateException is thrown.
#13 Updated by Sergey Ivanovskiy 11 months ago
Paula Păstrăguș wrote:
Hynek / Sergey, I don’t think the solution I pushed is really viable. We’re synchronizing on the same lock that’s also used for reading keys, injecting keys, and other operations in WebClientProtocol. This can make the GUI feel sluggish, I noticed the slowdown when hovering over an image where a PS timer is defined to change it.
I think we should support the current design that session.getRemote().sendBytes is accessed via the dedicated thread created by PushMessagesWorker,
otherwise we should invent new conception with multi-threaded access to session.getRemote().sendBytes .
#14 Updated by Paula Păstrăguș 11 months ago
Sergey Ivanovskiy wrote:
Is this issue is really related to #10428?
Not exactly, but since the main discussion took place in that task, I thought it would be useful to reference it.
#15 Updated by Paula Păstrăguș 11 months ago
- Status changed from WIP to Review
- File 10485a.patch
added - Assignee set to Paula Păstrăguș
I’ve attached a patch that switches the implementation to use a ReentrantLock. If this approach looks good, I’ll commit the changes and roll back the current solution.
Would this work for you? Hynek / Sergey, please take a look.
#16 Updated by Hynek Cihlar 11 months ago
Paula Păstrăguș wrote:
I’ve attached a patch that switches the implementation to use a
ReentrantLock. If this approach looks good, I’ll commit the changes and roll back the current solution.Would this work for you? Hynek / Sergey, please take a look.
I think it is OK to use a shared lock object. Just don't make it static. Yes, currently it doesn't make any functional difference to make it non-static, but let's make the intention clear. It belongs to the particual web socket object so it should be expressed such in the code. This may solve potential future issues.
Also please update the javadocs of the changed constructors.
Shouldn't sendString be protected, too?
#17 Updated by Paula Păstrăguș 11 months ago
Hynek Cihlar wrote:
Shouldn't
sendStringbe protected, too?
Indeed, it should have been protected too. I overlooked that detail.
#18 Updated by Sergey Ivanovskiy 11 months ago
Paula Păstrăguș wrote:
I’ve attached a patch that switches the implementation to use a
ReentrantLock. If this approach looks good, I’ll commit the changes and roll back the current solution.Would this work for you? Hynek / Sergey, please take a look.
Please make your changes in 10485a for review. Do you suppose to use PushMessageWorker.sendMessage from the different thread? It was supposed that PushMessageWorker.sendMessage was used from pushworker thread only.
#19 Updated by Paula Păstrăguș 11 months ago
Yes, sendMessage is only called from the push worker thread, so in practice it doesn’t really make sense to also guard sendString, since it’s used exclusively within that flow. My concern was more about future changes, if at some point another thread ends up calling sendString, having the lock in place would already prevent any issues.
#20 Updated by Greg Shah 11 months ago
Paula Păstrăguș wrote:
Yes, sendMessage is only called from the push worker thread, so in practice it doesn’t really make sense to also guard sendString, since it’s used exclusively within that flow. My concern was more about future changes, if at some point another thread ends up calling sendString, having the lock in place would already prevent any issues.
It is best to ensure that there is no way to use these methods which could break our synchronization.
#21 Updated by Paula Păstrăguș 11 months ago
- % Done changed from 90 to 100
Committed as rev 16136.
Please review.
#22 Updated by Hynek Cihlar 11 months ago
Paula Păstrăguș wrote:
Committed as rev 16136.
Please review.
The changes look good to me. Sergey, what do you think?
#23 Updated by Sergey Ivanovskiy 11 months ago
Let me explain my point of view. Please look at the usage of MSG_GET_WEB_BASIC_PALETTE of WebClientProtocol.GuiWebDriver.getBasicWebPalette sends a message via web socket and waits a result. In this situation we can said that the method synchronously returns the result. Compare this situation with WebClientProtocal.sendTextMessageSync(byte[] message) that is documented to send message synchronously but actually this message is just not queued via PushMessageWorker. Correct? This introduces confusions and makes PushMessageWorker useless because messages can be sent directly into the web socket. It seems for me that if NetworkTestHandler.startDownloadTest should measure the application download test, then it should use PushMessageWorker, but if this method should estimate indirectly the network speed with the server host, then it could use a new websocket channel.
The changes rev 16136 (10485a) should obviously fix the race condition. This methodpublic void sendTextMessageSync(String message) of WebClientProtocal has no usages and it seems that can be removed.
#24 Updated by Hynek Cihlar 11 months ago
Sergey raises a valid point. Adding synchronization logic to cover up a flawed design is not a good approach.
The original intent of the test logic was to bypass PushMessagesWorker entirely and measure raw network performance with as little interference from the framework as possible.
The best solution would be to open a completely new socket for the throughput measurement. This would also eliminate the need for additional synchronization.
Sergey, do you agree with this approach?
#25 Updated by Sergey Ivanovskiy 11 months ago
Yes, I agree.
#26 Updated by Paula Păstrăguș 10 months ago
- % Done changed from 100 to 20
- Status changed from Review to WIP
- I’ve created a new endpoint /network and registered it with the embedded server using:
server.addWebSocketHandler("/network", NetworkTestSocket.class);
- The core logic is not yet implemented. For now, I’ve only set up the skeleton to ensure the connection opens and messages can be exchanged. I’ll start working on the actual functionality once we settle a few important aspects:
- For the new JS WebSocket, should I create a dedicated file (e.g.,
p2j.network-socket.js)?
- I noticed that
p2j.socket.jsalready contains logic for WebSocket creation, retries, and ping-pong keepalives, but it seems tailored for a single socket rather than multiple ones. Should we extend that file or build a separate one for this new logic?
- When should the new JS WebSocket be opened? As soon as the client requests a network test, or right when the application loads its modules?
- Would there be any downside to opening the WebSocket immediately when the app starts? We could keep it alive with ping-pong, but if the client never initiates a network test, we’d be maintaining a connection that’s never used. What’s the best practice here?
- Should we include retry logic if the network WebSocket closes unexpectedly?
- These are the main points I’d like to clarify first. Once I'll have answers for these, I’ll be able to move forward, and of course, I’ll reach out if further questions come up.
#27 Updated by Greg Shah 10 months ago
The network test is only a diagnostic tool. Real users will almost never use it. The websocket should only ever be created when the test is used, it should not be there under normal conditions.
I don't think we need any special retry processing, since this is not a production connection. A best efforts check on network throughput is all we need to do.
#28 Updated by Paula Păstrăguș 10 months ago
- Status changed from WIP to Review
- % Done changed from 20 to 100
Committed as rev 16137: Added dedicated socket endpoint for network performance tests.
Hynek, Sergey please review.
#29 Updated by Paula Păstrăguș 10 months ago
- File netResults.png added
Here's an image with the results from my machine:

#30 Updated by Hynek Cihlar 10 months ago
- Status changed from Review to WIP
- % Done changed from 100 to 90
Code review 10485a.
Overall the changes look fine. Just a few points.
- The socket should be closed after the test is done.
mainSocketUrl.replace("ajax", "test");is fragile. If the host name will contain ajax in its name, for example, the resulting url will be incorrect. Use aURLobject and itspathnameproperty.- Do not import individual classes, but use asterisk. This complies with the code standards. For example
import org.eclipse.jetty.websocket.api.annotations.*.
#31 Updated by Paula Păstrăguș 10 months ago
- Status changed from WIP to Review
Addressed points 2 and 3 in rev 16138.
I double checked the first point, the socket is already closed at the end of the test (see p2j.perf_tests.js, line 1913), and it is also closed when the test is canceled. Or perhaps I missed something, could you clarify if that’s the case?
#32 Updated by Hynek Cihlar 10 months ago
Paula Păstrăguș wrote:
Addressed points 2 and 3 in rev 16138.
I double checked the first point, the socket is already closed at the end of the test (see p2j.perf_tests.js, line 1913), and it is also closed when the test is canceled. Or perhaps I missed something, could you clarify if that’s the case?
OK, if it is closed from JS it is fine.
Please test the case when the network connection is briefly interrupted. It is OK for the test to fail, but the user should be able to continue with his work, also the user should be able to run another test.
Please fix the type "Clossing network websocket...".
When the web socket is closed should also the associated handlers be removed before the reference is set to null in p2j.networkSocket.close()?
#33 Updated by Paula Păstrăguș 10 months ago
- File no-network.png added
Hynek Cihlar wrote:
Please test the case when the network connection is briefly interrupted. It is OK for the test to fail, but the user should be able to continue with his work, also the user should be able to run another test.
Hmm, if the network connection is interrupted, the main WebSocket will be closeed as well, which triggers this pop-up to appear:

When I reconnect to Wi-Fi and the main WebSocket connection is restored, the network test window doesn’t provide any feedback, no "Test abort" or "Test finished" message. It just stays frozen at the point where the connection was interrupted. However, the cancel button still works, and I can restart the tests manually. (the start button is disabled, like in the attached image)
Would it make sense to add something like context.log("Test finished.") to notify the user about the state of the test?
#34 Updated by Paula Păstrăguș 10 months ago
- File after.webm added
- File before.webm added
Hynek, I’ve attached two videos to illustrate the network test behavior when the internet connection is interrupted and then restored. The before.webm shows the current implementation, while the after.webm includes the patch.
I’d like your input: should we leave the behavior as it is, or also display the Test abort message when the network WebSocket is closed after an interruption?
#35 Updated by Hynek Cihlar 10 months ago
Paula Păstrăguș wrote:
I’d like your input: should we leave the behavior as it is, or also display the
Test abortmessage when the network WebSocket is closed after an interruption?
Paula, your implemented behavior is correct. Just please change the message so that it is clear the test failed due to connection error.
#36 Updated by Paula Păstrăguș 10 months ago
- % Done changed from 90 to 100
Committed as rev 16139.
Hynek, please review.
#37 Updated by Hynek Cihlar 10 months ago
Code review.
When logging the websocket close/error event, do not log just the reason property, but the whole object: console.log("WebSocket closed.", event);.
The connection error is output to the test results from the websocket's close handler. Should it be output from error handler instead so that only connection errors are output?
#38 Updated by Paula Păstrăguș 10 months ago
Hynek Cihlar wrote:
The connection error is output to the test results from the websocket's close handler. Should it be output from error handler instead so that only connection errors are output?
I believe we should always log whenever the WebSocket is closed, whether it’s intentional (by design) or due to a connection interruption / other causes.
#39 Updated by Paula Păstrăguș 10 months ago
Also, no error is raised when the internet connection is interrupted and then restored. I’ll double-check, but as I remember, the WebSocket just closes without throwing any error.. a bit strange, but that’s how it behaves.
#40 Updated by Hynek Cihlar 10 months ago
Paula Păstrăguș wrote:
Hynek Cihlar wrote:
The connection error is output to the test results from the websocket's close handler. Should it be output from error handler instead so that only connection errors are output?
I believe we should always log whenever the WebSocket is closed, whether it’s intentional (by design) or due to a connection interruption / other causes.
The thing is "Test failed: connection error" is output in all the cases (when close handler is invoked).
#41 Updated by Hynek Cihlar 10 months ago
Hynek Cihlar wrote:
Paula Păstrăguș wrote:
Hynek Cihlar wrote:
The connection error is output to the test results from the websocket's close handler. Should it be output from error handler instead so that only connection errors are output?
I believe we should always log whenever the WebSocket is closed, whether it’s intentional (by design) or due to a connection interruption / other causes.
The thing is "Test failed: connection error" is output in all the cases (when
closehandler is invoked).
You may also check wasClean of the close event.
#42 Updated by Paula Păstrăguș 10 months ago
I think Test failed on its own is enough, no need to mention connection error, since the exact reason isn’t that important here. Right?
#43 Updated by Paula Păstrăguș 10 months ago
I noticed that I was assigning null to the handlers too early. What happened is that the cleanup ran right after calling close(), before the socket had actually finished closing, which meant the onclose event couldn’t fire properly. I’ve moved the cleanup into the onclose handler so it runs at the right time.
#44 Updated by Paula Păstrăguș 10 months ago
- File net.patch
added
I’ve attached the patch. Let me know if there’s anything else I should adjust before I commit everything together. Thanks!
#45 Updated by Hynek Cihlar 10 months ago
Paula Păstrăguș wrote:
I think
Test failedon its own is enough, no need to mentionconnection error, since the exact reason isn’t that important here. Right?
If it failed, the user should know. The reason is useful I think.
#46 Updated by Paula Păstrăguș 10 months ago
Okay, so I can take the status code (per https://datatracker.ietf.org/doc/html/rfc6455#section-7.4.1) and map it to a user-friendly message?
#47 Updated by Hynek Cihlar 10 months ago
Paula Păstrăguș wrote:
Okay, so I can take the status code (per https://datatracker.ietf.org/doc/html/rfc6455#section-7.4.1) and map it to a user-friendly message?
I think a generic description of an unexpected network error plus the code in the message area is OK.
#48 Updated by Paula Păstrăguș 10 months ago
Addressed review as rev 16140.
#49 Updated by Hynek Cihlar 10 months ago
- Status changed from Review to Internal Test
Code review 10485a. The changes look good.
#50 Updated by Paula Păstrăguș 10 months ago
- Status changed from Internal Test to Review
The testing phase uncovered a small issue. When the connection was interrupted during the upload test, the behavior was incorrect because p2j.perf_tests.notifyClosing was calling context.stopped instead of stopNetworkTests. In the middle of an upload test, the abort handler needs to be executed.
Hynek, please review rev 16141: Fixed notifyClosing to call stopNetworkTests instead of context.stopped, ensuring the abort handler runs if the web socket closes during the upload test.
Aside from this, testing on a large GUI application passed successfully. I’ll continue next with the Hotel GUI.
#51 Updated by Paula Păstrăguș 10 months ago
Hotel GUI smoke testing also passed with the last rev of 10485a.
#52 Updated by Hynek Cihlar 10 months ago
- Status changed from Review to Internal Test
Code review 10485a. The changes look good.
Is there any more testing needed?
#53 Updated by Paula Păstrăguș 10 months ago
I believe Hotel Gui and the large GUI application provide sufficient coverage. If there's anything else you'd like me to test, feel free to mention it. Otherwise, I'm good to proceed with the merge.
#54 Updated by Hynek Cihlar 10 months ago
- Status changed from Internal Test to Merge Pending
Please merge 10485a to trunk now.
#55 Updated by Paula Păstrăguș 10 months ago
- Status changed from Merge Pending to Test
10485a was merged into trunk as rev 16188 and archived.