FWD » Core Development » Runtime Infrastructure

Tasks like #10941 and #10946 suggest that we need to improve our resiliency in our appserver (both classic and PASOE). Customer code should not be able to leave our appserver unresponsive or requiring a server restart to recover. This task should explore any changes we need to make in this area.

To the degree that OE is able to do better than FWD, any improvements would be considered a bug. Ultimately, FWD should handle such issues better than OE, so long as we don't compromise compatibility along the way. In short, we should do at least as well as OE and really, we should be able to do better.

With the recent fixes from #10941, the number of maximum concurrent requests FWD can handle is around 300 (with maxMsaSessionsPerAgent = 100), after that the server hangs and does not accept any more requests, nor does it produce any logs. The customers will most likely stumble upon this limitation soon, so it is a good place to start our resiliency work on.

As Constantin stated in #10941-35, MsaCancellableCallable.reserveSession gets stuck on completing the future:

Integer reserveSession(MultiSessionAppserver msaAppserver,
                             MultiSessionClientConnection invokeConnection)
      throws ExecutionException,
             InterruptedException
      {
         Future<Integer> reserveSessionFuture = msaAppserver.reserveSession(invokeConnection);
         setCurrentlyRunningFuture(reserveSessionFuture);
         int sessionId = reserveSessionFuture.get();
         setCurrentlyRunningFuture(null);
         return sessionId;
      }

From my investigations I concluded that sessions that should have been freed are not, I am not sure what is the reason yet.
The flow of the session allocation is quite simple, a session is requested, if not available, a listener is added to the list and when a session is freed, it gets it. Our problem is the last part, after a certain amount of waiting listeners, none of the sessions are being freed, indicating a deadlock of some kind. I will keep investigating.

• Requests a session for executing runPersistent, on pool initialization, an idle session is created, so one is available.
• The idle session is bound to the request/connection A and moved to the idleBoundSessions set.
• The MSA task is executed with the bound session.
• The session is then freed, but remains bound because later it will be used to execute other work on this connection. The session is unbound only on deleteProcedure (not sure on this one).

• Requests a session for executing runPersistent, the connection is not yet bound, so no bound session. There are no idle sessions left either.
• Creates a session listener Future that waits for a session to be freed, the call is blocking with get().
• Since we have only one thread, there is no other thread that could execute the onSessionFree callback so the thread is stuck on get().

More threads add complexity to the problem, but the issues is the same.
If there are a lot of incoming unbound connections and the existing ones are mostly bound, all the appserver threads will be busy waiting for a free connection sooner or later.

One solution would be to not count the bound session when deciding if a new one is to be created, be more lenient with the sessions per agent number.
I experimented a bit and was able to run 10000 concurrent requests with the limit of 100 sessions, the maximum number of active sessions was 178, so not so bad considering the alternatives.
Another solution would be a back pressure mechanism for free session requests, we can't have all the appserver threads trying to get a session, maybe put the requests in a queue and have the appserver threads take one at a time.

Artur Școlnic wrote:

One solution would be to not count the bound session when deciding if a new one is to be created, be more lenient with the sessions per agent number.
Another solution would be a back pressure mechanism for free session requests

By the 'appserver worker thread' I meant the LegacyServiceWorker.

Constantin, which solution do you think we should pursue?
In my opinion, we could be more lenient with the number of sessions as a workaround/immediate relief, while we work on improving the current architecture.

I've suggested a queue approach in #6673.

Is there a task or wiki on our PASOE implementation? I would like to know what our constraints and detailed specifications are.

We have some (limited) documentation in Appserver Support. The PASOE v2 implementation was done in #8661.

On another note, we are running the persistent procedure for each request, after the work with it is done, it is deleted, both actions need their own session. What if the persistent procedure is ran with the same parameters over and over, can't we cache the handle and use it on subsequent requests?
This way we no longer need to use two new sessions for each request and no longer run the procedure remotely which is quite costly.
I understand that there may be limitations, because the procs are tightly coupled with the invoke connection and the resource retention may be too large, but maybe it is worth looking into.
Constantin, Greg,
Does this make sense?

On another note, we are running the persistent procedure for each request, after the work with it is done, it is deleted, both actions need their own session.

Isn't this based on however the client code is written? I don't understand how we would merge the execution and deletion into one invocation if the client code has it separated.

Greg Shah wrote:

Isn't this based on however the client code is written? I don't understand how we would merge the execution and deletion into one invocation if the client code has it separated.

The idea is not to merge the invocations, but to not make them at all, if possible.
If a runPersistent is called for a certain procedure with certain arguments, I don't see why we can't store the handle and give it as a result for the next runPersistent with the same procedure and arguments.
The procedure cleanup can happen on cache expiry.

Artur Școlnic wrote:

Constantin, please review.

I'm good with the changes.

Artur Școlnic wrote:

I will test harness and concurrent requests in PASOE and classic.

This testing passed.

Artur, lets merge tomorrow after Alex unfreezes trunk.

I will test harness and concurrent requests in PASOE and classic.
Is there something else we could test?

See #10941-54.

• curl + visualVM:
  1000 concurrent requests on a an infinite loop for 30 minutes resulted in no errors on the server, but at the end the requests were handled much slower. Heap usage went from 1.5GB to 8GB, CPU usage from 30% to 70%.

• Jmeter:
  1000 concurrent requests on a 10 iteration loop resulted in no issues on the server, but the success rate was 50%, meaning that half of the requests did not reach the server, with curl such data is hard to gather.
  From the logged failures in jmeter, I think the issue is the jetty thread pool, it simply can't keep up with the amount of requests, I think the default max capacity is 200.

Artur, lets merge tomorrow after Alex unfreezes trunk.

The last pieces of puzzle are 10951a and 10591a, before I do a delivery of trunk.
For 10951a, #9687 is still pending. If Tomasz won't answer in the next hour, I will do the merge.

I think the issue is the jetty thread pool, it simply can't keep up with the amount of requests, I think the default max capacity is 200.

I would like us to calculate the right number automatically based on the number of possible concurrent sessions across all possible usage (interactive GUI/ChUI, REST, SOAP, appserver, webspeed/webhandler, non-legacy usage, admin console). It doesn't need to be one for one, but we must configure this to some reasonable default AND then also provide a way for the customer to override it explicitly in the directory.

I would like us to calculate the right number automatically based on the number of possible concurrent sessions across all possible usage (interactive GUI/ChUI, REST, SOAP, appserver, webspeed/webhandler, non-legacy usage, admin console). It doesn't need to be one for one, but we must configure this to some reasonable default AND then also provide a way for the customer to override it explicitly in the directory.

Is this a blocker for merging 10963a?

Alexandru Lungu wrote:

Is this a blocker for merging 10963a?

Yes, also some other testing we need to do.

The next steps for this task are:
1. Investigate the rapidly increasing memory usage noted in #10963-22. I uploaded the heap dump on devsrv:/tmp/heapdump-concurrent.hprof (13GB)
2. Run the customer jmeter test plan.
3. Make jetty thread pool configurable in directory (if not already) and find a suitable maximum number of connections.
4. I have some concerns about MSA tasks being executed with Future.get() without a timeout, I think if the task will hang, the legacy worker thread that executed the blocking call to the MSA thread will also hang and block indefinitely. This needs to be investigated.

Artur Școlnic wrote:

From the logged failures in jmeter, I think the issue is the jetty thread pool, it simply can't keep up with the amount of requests, I think the default max capacity is 200.

I tried to increase it, but the issue was not solved.
I checked the documentation for Jmeter and it seems like my test plan was not reasonable, I was starting 1000 threads in one second on the jmeter client which overloaded the server and the TLS handshake simply failed. Apache recommends a ramp up of 1 second per thread, which is quite slow for our use case, but a more realistic value would be nr_threads/10, so one thread will be spawned by jmeter each 0.1 seconds. With the new test plan, there are no more issues with unresolved requests.
200 threads in the jetty pool is quite reasonable considering that the requests are queued on reaching full capacity.
Another concern is that each thread carries a stack of 1Mb, so just adding thousand of threads will not increase throughput (likely), but will increase memory usage.

Do we still need to expose this via directory cfg so that we have some way to control it without code changes?

I can see where this may be something to reduce for smaller installations. What is this value set to for the web clients? We don't need this to be very high at all there.

A jetty thread pool is only used in GenericWebServer and is instantiated with a default constructor.
The default values in jetty 9.4:

Setting       Default Value
Min threads   8
Max threads   200
Idle timeout  60 000 ms (60 s)
Queue         Unbounded queue (e.g., BlockingArrayQueue)

Artur Școlnic wrote:

3. Make jetty thread pool configurable in directory (if not already) and find a suitable maximum number of connections.
4. I have some concerns about MSA tasks being executed with Future.get() without a timeout, I think if the task will hang, the legacy worker thread that executed the blocking call to the MSA thread will also hang and block indefinitely.

Constantin, I addressed these issues in 10963a/16313.
Please review when you get the chance.

I set the time out for the msa task a bit low, at 5 seconds, we will need to increase it for high workload scenarios and also make it configurable via directory.

4. I have some concerns about MSA tasks being executed with Future.get() without a timeout, I think if the task will hang, the legacy worker thread that executed the blocking call to the MSA thread will also hang and block indefinitely.

Do you have a recreate for this problem?

It's more of a potential issue, I don't have a recreate, but what will happen with trunk if a msa task executed a function in a persistent proc that does an infinite wait? I think it will hang the legacy worker and no more tasks will be executed on it.

I think the problem with the memory leak is with Map<String, MultiSessionClientConnection> connections = new ConcurrentHashMap<>(); in MultiSessionAppserverManager a new connection is created for each request and is not cleared. After 1k requests there are ~1k MultiSessionClientConnection objects in memory.

It is definitely an issue, but the memory retention is not significant, solving it did not solve the greater memory leak, there is something else.

Artur Școlnic wrote:

I think the problem with the memory leak is with Map<String, MultiSessionClientConnection> connections = new ConcurrentHashMap<>(); in MultiSessionAppserverManager a new connection is created for each request and is not cleared. After 1k requests there are ~1k MultiSessionClientConnection objects in memory.

As mentioned, this is not the main memory leak concern, but it needs to be solved, with enough requests the map can grow to hundreds of Mbs, we need to reuse the connections or remove them from the map after the session ends.

The leak is not present with classic appserver, after 4k requests in batches of 20, the used heap is < 1Gb.

Evidence points to SecurityContext tokenMap hogging the memory. The number of objects in the map is not changing, but the retained memory is growing. Individual nodes in the map went from consuming 5Mb after 1k requests to 30Mb after ~5k requests. It is hard to determine the point of entry for the leak because the map contains various types of objects, it seems like the memory is increasing across all types. Will investigate further.

I found another leak in TxWrapper$WorkArea, deregisterDynamicBuffer

// remove from global scope always
Map<RecordBuffer, Boolean> loaded = loadedBuffers.getDictionaryAtScope(-1, false);
if (loaded != null)
{
   loaded.remove(buffer);
}

Even with a fix for this, the leak is still present, albeit to a lesser degree.

I am beginning to think that we are not dealing with a single leak, but with multiple ones because the context is persisted between sessions.
I am not sure how the context is handled with classic appserver.
Constantin, I have a strong feeling that PASOE opened new opportunities for leaks because the context is being reused.

My original intention was that the context would be exited when a response was complete. In other words, we should be back to the "initial context" when we release the session after finishing a request. Are we leaving the context in place until the next time that session is used? That would be a problem.

Greg Shah wrote:

Are we leaving the context in place until the next time that session is used? That would be a problem.

Yes, we are. The session IDs are reused and the associated contexts also.
There is a resetContext method, but it is called only when a task fails with QuitConditionException, not sure why.
Hopefully, resetting the context at the end of session will solve our problems.

Artur, the context of a 4GL session can't be reset; see the docs at https://docs.progress.com/bundle/openedge-migrate-classic-appserver/page/Differences-between-classic-AppServer-WebSpeed-Transaction-Server-and-PAS-for-OpenEdge.html - Session-Managed is classic Stateless and Session-free is classic State-free. In OpenEdge, only State-reset resets the context.

On the heap you provided, my concern are RecordBuffer instances kept in BufferManager and TxWrapper - these seem to be leaking. Also the connections in the manager - these should either timeout or we need to clean them when the OpenClient calls disconnect.

1. in PASOE, each session that is created in FWD will have its own context.
2. the session's context will not be freed until it gets 'trimmed' - so any context-local caches will live until then.

Constantin Asofiei wrote:

On the heap you provided, my concern are RecordBuffer instances kept in BufferManager and TxWrapper - these seem to be leaking. Also the connections in the manager - these should either timeout or we need to clean them when the OpenClient calls disconnect.

I already noted these issues and solved them, this did not fix the leak completely.

But, please note that:

1. the session's context will not be freed until it gets 'trimmed' - so any context-local caches will live until then.

What is a 'trimmed' session exactly and when should it be trimmed?
Now it is being reused without any changes.
What happens:
1. Session 1 is created, it creates and associates context A.
2. Session 1 makes changes to the context A.
3. Session 1 finishes execution without clearing the context.
4. Session 1 is reused for another request, the same context A in the same state is used here.

Like I said, the leaks we see are due to the fact that until now the context was a short lived structure, now it accumulates data because of the latent leaks in FWD. At least this is how I understand it.

Constantin: Are you saying that in OE, a "stateless" session is actually stateful instead of starting from an initial context on each new request?

From the documentation:

On a bound session-managed connection, all remote procedures run in the session of a PAS for OpenEdge instance. The context at the initiation of a client request is exactly as it was when the previous request from the same client completed.

With bound sessions, that is understood. Binding makes a session stateful.

I'm asking about unbound stateless sessions.

• connect to server, call 1 runs a program and sets a global shared var in the session, disconnects and deletes the server
• connect to server, call 2 runs another program and reads the shared var in the session, disconnects and deletes the server - it sees the value set by the call 1

So both cases (managed and free) leave the context unaltered. In OE, only State-reset in classic actually destroys the context.

For unbound:

Because any ABL session in a PAS for OpenEdge session pool can handle any remote procedure request on any unbound connection, the context created by one remote procedure request is generally unavailable to the next request in the same connection sequence.

Artur Școlnic wrote:

For unbound:
[...]

They mean that if you do two requests in unbound mode, then the requests must not be dependent on each other, as there is no guarantee which context will end up executing it. In my test as there is only one context, then is 'guaranteed' that the same context will be used for both requests.

What a terrible design. I wonder how many customer applications actually rely upon that state (maybe without the developers realizing it).

We could setup a flag to force reset even for these "stateless" sessions. It might break some apps but it could be a huge benefit for those apps which don't need that state. Depending on periodic trimming makes no sense to me.

Greg Shah wrote:

What a terrible design. I wonder how many customer applications actually rely upon that state (maybe without the developers realizing it).

We could setup a flag to force reset even for these "stateless" sessions. It might break some apps but it could be a huge benefit for those apps which don't need that state. Depending on periodic trimming makes no sense to me.

FWIW - We do not rely on any residual context between sessions, and in fact have been tripped up by this on occasion in the past with shared variables, etc.

Another great source of leak is BufferManager.openBuffers.
Considering the following scenario:
A persistent procedure runs methods from another class, buffers are added to the method's scope and to the global scope:
bufferInitialized

if (global)
            {
               // the procedure is no longer on the stack, so we must explicitly add the buffer to the
               // PersistentProcScope.openBuffers collection for the PersistentProcScope which corresponds
               // with the procedure
               PersistentProcScope ppScope = persistProcScopes.computeIfAbsent(referent, k ->
               {
                  return new PersistentProcScope();
               });
               ppScope.openBuffers.add(buffer);
            }
         }
      }

      if (global)
      {
         openBuffers.add(true, buffer);

 if (referent == null || pm._isPersistentAndNotPendingDelete(referent)) //referent is the method
{
   // if no referent or procedure is still persistent, do not clean up yet
   return;
}

Not sure if the buffers should have not been added to the persistProcScopes in the first place...

The cause for openBuffers leaking is that inner persistent static procedures do not clean the openBuffers and other collections. Since it is static, deleting it does not make sense, but on scope ended we still need to clean it up.
I implemented a basic collection cleanup for the affected ones and the leak was solved.
Although this alleviated a lot of memory retention, there are other issues.
The next leak is in TemporaryBuffer$Context, the domains collection is retaining values.

      /** Map of master buffers to weak hash maps with slave buffer keys */
      private Map<TemporaryBuffer, Map<TemporaryBuffer, Object>> domains = new WeakHashMap<>();

The TemporaryBuffer keys from domains are strongly reachable from SecurityManager, so they are not deleted from the WeakHashMap, we need to refactor the code to use a HashMap and explicitly remove the statically defined buffers, I assume this should be done on scope close.

Artur, please provide some simple examples for #10963-56 and #10963-55 assumptions.

Constantin Asofiei wrote:

Artur, please provide some simple examples for #10963-56 and #10963-55 assumptions.

I am debugging a customer application, I can see clearly that the collections are growing from run to run.
It seems like the open client project i have does not work with persistent tables which is a hard requirement for replication.
As a general idea, the reproduce is not hard. You need to call a persistent procedure that calls a static method that defines a buffer for a persistent table.

I added the following fixes to the branch:
1. MSAManager connections not being clean up.
2. TxWrapper loadedBuffers was retaining buffers for global scope.
3. Buffer domains not being cleaned up on main buffer cleanup.
4. Static persistent procedures not cleaning up collections on scope deleted.

The leak is reduced by half with these changes, but it is still quite large.

I checked again and there is no leak with classic appserver, after thousands of requests the used memory increase is within margin of error.

I suspect the gap in retained memory is due to classic not holding on to the context, but this needs investigation.
We may need to change the approach when it comes to context usage in PASOE.

Just as an experiment, I tried cleaning the FWD context after a session is terminated and this solved the leak completely.
This may be a reasonable option, at least as a temporary workaround or backup solution.

See my thoughts from #10963-53. I think we should add this flag.

On the other hand, if we have real leaks here, then we do need to fix them.

Greg Shah wrote:

See my thoughts from #10963-53. I think we should add this flag.

I suppose the downside to resetting the context would be a loss in performance (maybe, not sure how many caches are context specific), but we may implement a common ground, maybe reuse the context for a certain amount of time or number of sessions.

On the other hand, if we have real leaks here, then we do need to fix them.

I agree, but this may take a while, considering the urgency of the issue, maybe we can omit the context reuse for the time being.

Please go ahead with the context reset and make it conditional based on a directory flag. We should probably default the flag to not reset (compatible mode).

We can measure the performance impact of the reset but I don't expect it to be too bad.

I agree, but this may take a while, considering the urgency of the issue, maybe we can omit the context reuse for the time being.

We can deliver in phases.

Greg Shah wrote:

Please go ahead with the context reset and make it conditional based on a directory flag. We should probably default the flag to not reset (compatible mode).

I added the code to the branch.

I think it would be best to keep only the PASOE stability fixes and the context reuse code in 10963a which we could deliver soon and make a b branch for the memory leak fixes, the review and testing for these can delay the branch quite a bit.

Constantin, if you agree, I will prepare the code and you can review the branches ASAP.

About the BufferManager changes - in TransactionManager.processScopeNotifications, there is this code:

         if (!start && blk.topLevel)
         {
            // procedure scopes need to be finished last
            wa.procScopeable.scopeFinished();
         }

This deletes all resources created in the static OO method.

I don't understand the original problem. Looking at how PersistentProcScope.openBuffers works in a simple test for a OO static method with an explicit DEFINE BUFFER, then BufferManager.openScope incorrectly marks an explicit buffer def as global because of the buffer.getPersistentProc() != null test. I think we need to distinguish between a buffer defined in the 'execute' method (for persistent programs) and one defined in some internal entry (internal procedure, etc).

The point of PersistentProcScope is to collect the buffers which are defined in the execute method (or in the static or instance reference for a OO class), as those must be available when some internal top-level block (method, procedure, function) gets executed.

So, are you talking about explicit buffers defined in some static OO method which get leaked?

Constantin Asofiei wrote:

So, are you talking about explicit buffers defined in some static OO method which get leaked?

Yes, the buffers are added to the current and global scopes, normally openBuffers is cleaned in scopeDeleted and for persistent procedures this happens because they are explicitly deleted, but the static methods used in the persistent proc are not cleared

Object referent = pm.getProcessedProcedure();
      if (referent == null || pm._isPersistentAndNotPendingDelete(referent))
      {
      // procedure clean up

Constantin Asofiei wrote:

This deletes all resources created in the static OO method.

It does not delete the openBuffers for the static method and global scopes, they just keep accumulating every time the buffers are initialized.

Where exactly are these buffers defined in 4gl?

start.p

run external.p persistent.

external.p

Class1:staticMethod().

Class1.cls

USING Progress.Lang.*.

BLOCK-LEVEL ON ERROR UNDO, THROW.

CLASS Class1: 

    method public static void staticMethod():
       def buffer buf1 for item.
       find first buf1 no-error.
       message "Static method ran persistent.".
    end method.

END CLASS.

buf1 is added to openBuffers of the staticMethod scope and global scope every time the method is called.

Artur Școlnic wrote:

buf1 is added to openBuffers of the staticMethod scope and global scope every time the method is called.

OK, this is similar to what I was testing. So, the assumption I think is: a DEF BUFFER inside an internal top-level block (i.e. not an external procedure) that must not register as 'global' or see itself as 'persistent'.

Ovidiu, please review the other persist changes (beside BufferManager)

I tested the performance of the customer application with the context delete and it was barely different, maybe a 10-20ms hit which is within the margin of error.

It seems like almost all the leaks discussed on the task are a consequence of a single issue. The procedure resources were not deleted because they were not being registered in the first place:
ProcedureManager

   public static void processResource(WorkArea wa, WrappedResource res, boolean dynamic)
   {
      // be a no-op in case of headless mode
      if (wa.tm == null || wa.tm.errHlp != null && wa.tm.errHlp.isHeadless())
      {
         return;
      }

Artur, that is a good find. I think the checks for isHeadless() in these ProcedureManager methods were added back then for the PL/Java compatibility - and PL/Java support was removed completely.

deregisterPendingResource(WindowWidget)
cleanupPending()
processResource(WorkArea, WrappedResource, boolean)

The other example of a 'headless' mode for a context I think is only the FWD Admin Console (and the REST APIs?). The point is: in headless mode, there is no real FWD context (or FWD client).

But, I wonder if we really need to set the MSA context as 'headless' in FWD. isHeadless() is used mainly from ErrorManager, for error processing.

For now, please try removing ErrorManager.getErrorHelper().setHeadless(true); from MultiSessionAppserverSecurityManager.setContext() and see what happens. If we have problems, then we need to check 'not PASOE' in ProcedureManager (and maybe more places), when 'headless' is true.

I cleaned up 10963a, it has the stability fix and the headless issue fix, I tested the pasoe project in pasoe and classic, no regressions were found.
I also stress tested the server with 10k concurrent requests ramped up over 100 seconds, there were no issues.
A large gui application that uses classic appserver was also tested, it was fine.
I think we can deliver 10963a.
In 10963b I will keep the other relevant code written for this task.

Did all of your testing have the "non-compatible context deletion" disabled?

Constantin: You control the merge.

Greg Shah wrote:

Did all of your testing have the "non-compatible context deletion" disabled?

There is no need to drop the context now, I kept the original code that reuses the context.

Artur, please do a test where in PASOE there is an appserver call which has:

def var h as handle.
h = h:next-sibling. // test 1
h = h:next-sibling no-error. // test 2

The idea is to not hit i.e. FWD client code via ErrorManager.

I've ran the test - the only issue is that TC.displayError ends up writing the error message to STDOUT for the FWD server:

      if ((client == null || !client.isRedirected()) && !isGuiBatchStdoutRedirected)
      {
         // if not redirected to a stream, write it to the console
         System.out.println(errmsg);
      }

We can fix this in another iteration.

• this for the appserver process:
  

              <node class="string" name="server-side-resources">
                <node-attribute name="value" value="all"/>
              </node>
  
  

• the resource-plugins list

After this, I could start in PASOE mode.

Some other notes why 'headless' mode was used initially: at that time, there was no server-side support for UI, including error display; and it was enabled in #9055, so 'headless' mode is obsolete.

In PASOE mode, shouldn't we be implementing server-side-resources automatically as all? It would not work any other way so I don't think we should have to explicitly add this to the directory.

Greg Shah wrote:

In PASOE mode, shouldn't we be implementing server-side-resources automatically as all? It would not work any other way so I don't think we should have to explicitly add this to the directory.

Yes, that should be done.

Artur Școlnic wrote:

Ovidiu, the leak fixes are in 10963b.

• BufferManager.java:
  • line 1394: I am not sure ObjectOps.isStaticReferent(referent) check is correct. The ObjectOps handles object-specific operations. The if block handles cleanup of other static resources (buffers and temp-tables);
  • lines 1409, 1417, 1425: I think the second parameter of getDictionaryAtScope() should be false? There is no point on creating a new (empty) dictionary for deleting from it (NOP).
• TxWrapper.java:
  • line 1473: it seems to me like loadedBuffers.getDictionaryAtScope(-1, false) != loadedBufs is always true. The left side of the operator specifies the global scope, while the loadedBufs a 'normal' one. The test would fail only when the user context is is destroyed, right?
• the branch needs a rebase since some classes were modified by other commits.
  LE:
• the copyright year should be updated to 2026
• LegacyJavaAppserverClient.java, line 42: a new ID and date are necessary for the new commit.

I don’t believe there are any remaining leaks aside from the one in connections; 10963b addresses that issue.

Constantin, my concern was about MSA tasks potentially hanging, which could prevent the supporting thread from accepting new sessions. As a possible solution, we could introduce a timeout for the futures executing these tasks. What are your thoughts on this approach?

The scenario I have in mind involves a long-running or stalled procedure, which can be easily reproduced using something like Thread.sleep. In such cases, we currently don’t seem to have any protection against a blocked thread, at least none that I’m aware of.

Constantin Asofiei wrote:

Greg Shah wrote:

In PASOE mode, shouldn't we be implementing server-side-resources automatically as all? It would not work any other way so I don't think we should have to explicitly add this to the directory.

Yes, that should be done.

I added this in 10963b.

I tested the PASOE application in both modes, the testing passed.

10963b was merged as trunk/16357 and ported to 10614_main/16227.

Artur/Constantin, should this go into test status, or back into WIP (it's marked 90%)? Is there something left to implement/fix? Does #10963-84 need to be addressed in another iteration, or did that make it into 10963b?

Eric Faulhaber wrote:

Does #10963-84 need to be addressed in another iteration, or did that make it into 10963b?

10963b does not have the fix for this.

• A jetty warning, I think in this instance the request was dropped due to high concurrency, the exact issue is not known at this moment
  

  26/01/29 02:57:00.797-0600 | WARNING | org.eclipse.jetty.io.AbstractConnection | ThreadName:qtp1011872961-449 | Failed callback
  java.nio.channels.WritePendingException
      at org.eclipse.jetty.server.internal.HttpConnection$SendCallback.reset(HttpConnection.java:751)
      at org.eclipse.jetty.server.internal.HttpConnection$HttpStreamOverHTTP1.send(HttpConnection.java:1447)
      at org.eclipse.jetty.server.HttpStream$Wrapper.send(HttpStream.java:185)
      at org.eclipse.jetty.server.HttpStream$Wrapper.send(HttpStream.java:185)
      at org.eclipse.jetty.session.AbstractSessionManager$SessionStreamWrapper.send(AbstractSessionManager.java:1491)
      at org.eclipse.jetty.server.internal.HttpChannelState$ChannelCallback.succeeded(HttpChannelState.java:1590)
      at org.eclipse.jetty.ee10.servlet.ServletChannel.onCompleted(ServletChannel.java:764)
      at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:430)
      at org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:470)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)
      at org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)
      at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1071)
      at org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)
      at org.eclipse.jetty.server.handler.EventsHandler.handle(EventsHandler.java:81)
      at org.eclipse.jetty.server.Server.handle(Server.java:182)
      at org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:678)
      at org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:416)
      at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
      at org.eclipse.jetty.io.ssl.SslConnection$SslEndPoint.onFillable(SslConnection.java:575)
      at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:390)
      at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:150)
      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
      at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
      at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)
      at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)
      at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
      at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)
      at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)
      at java.base/java.lang.Thread.run(Thread.java:840)
  
  

  
  I think this is a minor threat, or none at all, but needs to be investigated.

• The other issue looks bad:
  

  Caused by: java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
      at com.goldencode.p2j.util.TransactionManager.abnormalEnd(TransactionManager.java:6144)
  ...
  com.goldencode.p2j.persist.PersistenceException: Error scrolling
  Caused by: org.postgresql.util.PSQLException: Ran out of memory retrieving query results.
      at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2392)
  

  
  There are multiple severe errors and warnings indicating that pgsql and jvm ran out of memory.
  Of course, this is unrecoverable.
  Not sure if it is a real issue, config issue or just hardware limitations on my part.
  The heap size is 8GB for the server, it may be too small, still, it looks like we are trying to scroll through an immense result set and maybe we shouldn't.

I increased the heap to 16Gb and repeated the stress test 3 times, the OutOfMemoryError was solved, so it is something we should be aware of.

Regardin

WARNING | org.eclipse.jetty.io.AbstractConnection | ThreadName:qtp1011872961-449 | Failed callback
java.nio.channels.WritePendingException