FWD » Core Development » Database

FWD allows server A to connect to remote FWD server B, to access the database instance managed by server B. This requires directory configuration for the remote FWD server access, and also a direct JDBC connection for the database.

Server A AFAIK at this time uses only remote dirty share, persistence and primary key calculations. Other caches (like FFCache, ORM Session cache) for the database served by server B, I think live on server A, while they can never be invalidated by server B.

This task is meant to investigate and find solutions for these cases.

Another part on which the authoritative database and the remote database (and by these I mean the corresponding FWD servers) are not in sync is the DMOVersioning.

@Constantin Asofiei could you raise the prio to high as the workaround (in related issue) is on a per table basis on one entry now for the reportserver which is not a desired solution to the broader problem.

@Constantin Asofiei could you raise the prio to high as the workaround (in related issue) is on a per table basis on one entry now for the reportserver which is not a desired solution to the broader problem.

I reopen the OG task on VMA project and move it to High. This is in the public section like this one (Database) is not properly tracked for customer delivery.

• PSQL supports LISTEN/NOTIFY, which allows us to implement the Observer design pattern.
• What helps to us: we can have a deterministic approach about database cache invalidation, when multiple servers/sessions connect to the same database. We can have a database write trigger which notifies events to all listeners, to invalidate DMO cache.
• More about in: https://www.baeldung.com/java-postgresql-listen-notify-events

Things that distributed across the cluster:

- Record locks (LockManagerFactory) - regions <db>_lockTable, <db>_inPlay; delta-encoded LockStatus values

- Unique indexes (UniqueTracker) - DistributedUniqueIndex, 2 global regions

- FastFind cache invalidation (FastFindCache) - FastFindCacheInvalidationFunction broadcast

- Tenant registry (TenantManager) - generation region (ensure strong consistency) + TenantRegistryReloadFunction broadcast

- Web-service session tokens (WebTokenRegistryFactory) - DistributedWebTokenRegistry, region TOKENS_REGION_NAME

- Security cache generations (domains, domain types, users) (SecurityCacheGenerationsFactory) - DistributedSecurityCacheGenerations, region GENERATIONS_REGION_NAME

- DirectoryService (RedisRemapper), directory LockManager (right now Redisson locking backend, 4369c LUA script approach pending)

Things that still are not distributed : DmoVersioning, SecurityManager.cache

Eric, I think for UniqueTracker, DmoVersioning and FastFindCache we need to add remote support, for direct connections to a non-owner database.

It is a priority for #11730

Eric, I think for UniqueTracker, DmoVersioning and FastFindCache we need to add remote support, for direct connections to a non-owner database.

Constntin, do you consider this being a final fix for #11730 rather than the direct Java access full cache clear? I would personally say so, because the cache sweep looks more like an workaround of the capability of remote FWD server to share its DmoVersioning and FastFindCache. I really thought UniqueTracker is managed remotely, but you are right, it is not.

I have some changes for #11730 to improve the API, but I am not sure if I should hurry to deliver to the customer or take my time, build the native SQL wiki for modifying data properly, write tests and engineer that API completely. If so, please raise the priority of this task to match #11730 (High) and lets have it delivered by next Thursday.

PS: for dirty-share I reckon that, as long as the cross-session variant is disabled, there is no need to make it remote, though it has such implementation exactly for such cases. After #8388, remote dirty-share will be mostly obsolete.

Alexandru Lungu wrote:

Eric, I think for UniqueTracker, DmoVersioning and FastFindCache we need to add remote support, for direct connections to a non-owner database.

Constntin, do you consider this being a final fix for #11730 rather than the direct Java access full cache clear? I would personally say so, because the cache sweep looks more like an workaround of the capability of remote FWD server to share its DmoVersioning and FastFindCache. I really thought UniqueTracker is managed remotely, but you are right, it is not.

Yes, if we configure the two apps correctly, and the remote APIs are executed properly (the db owner notifies the client(s) properly), then there is no reason the use that workaround to clear the caches.

I have some changes for #11730 to improve the API, but I am not sure if I should hurry to deliver to the customer or take my time, build the native SQL wiki for modifying data properly, write tests and engineer that API completely. If so, please raise the priority of this task to match #11730 (High) and lets have it delivered by next Thursday.

Those apps already have some other code which uses hand-written SQL. I would rather deliver #11734 than move further into this workaround with manual cache invalidation. Once we have the APIs documented and working properly in #11730, we can work with the customer to improve/change the direct-SQL approach - but please make sure existing usage is not broken after your improvements, please.

Review 11734a:

• catch (RuntimeException exc) in publish is unclear. I guess it makes sense for deliver to fail and we should handle that, but unregisterring policy doesn't seem entirely right. I would expect to catch an actual reason. Maybe there is a timeout and we should retry. I am not entirelly sure, but conisder if we can improve this.
  • there are other similar patterns like catch (RuntimeException exc).
  • I would tend to say that a typed exception would be better to actually diagnose properly. There might be some kind of runtime exceptions that would be fatal and requires unwinding instead of recovering.
• please avoid superflous comments. AI is tending to generate 4+ lines commets for rather trivial or self-explainatory code. I mean, the more documentation, the better. But I would prefer to move that in the javadoc in order to be captured by javadoc generation instead of inlining. For instance

 +      // a meta database is embedded and private to this node, so it has nothing to exchange with another
+      // node; it must also be refused because it carries the *name* of its primary database, and both the
+      // broker registry and SERVICES are keyed by name - servicing it would create a second broker and
+      // silently displace the primary database's service

• wrap LOG.info("DIAG publish node=" + nodeID + " db=" + database.getName() + " " + idents); and other logs in isLoggable. This way we avoit the concatenation redundant overhead for cases where the logging level is not enough.

The changes are very extensive, so this would require lots of testing to be sure.

• if server A does changes, server B should have its caches invalidated
• if server B does changes, server A should have its cached invalidated

Also, I think the communication is async, right? In other words, an invalidation will reach the remove server slower (?) than the actual DB state. This is because there are two DmoVersioning instances - one that is live and the other that is updated asynchronously. With a RemoveDmoVersioning there should have been one single instance and synchronous source of truth. Though, I am not sure of the performance implications ... but as long as there is a read-only server in question, its version should not be consulted (unless it is exclusively locked, which means it is not read-only). The FFCache being remote might be indeed a performance draw-back.

I must admit that I am a bit unfamiliar with this architecture and maybe Constantin can provide more feedback on the changes. If having more instances of DmoVersioning and Ffcache that are async updated, then I am fine with the approach and I will review more in depth. But at this stage, I want to fully understand if this was the intended design.

Alexandru Lungu wrote:

Overall, I must admit that I was expecting to see a RemoteDmoVersioning instead of a RecordInvalidationService. Do I understand correctly that a server is able to broadcast invalidations and subscribers can pick up the invalidations?

Yes, that is right. But it is bidirectional. Every node, owner or not, publishes what it commits. The hook is in the common ORM code at SavepointManager, right after the transaction commit.

Also, I think the communication is async, right? In other words, an invalidation will reach the remove server slower (?) than the actual DB state. This is because there are two DmoVersioning instances - one that is live and the other that is updated asynchronously. With a RemoveDmoVersioning there should have been one single instance and synchronous source of truth. Though, I am not sure of the performance implications ... but as long as there is a read-only server in question, its version should not be consulted (unless it is exclusively locked, which means it is not read-only). The FFCache being remote might be indeed a performance draw-back.

A change on remote side is also async in this design. A change to an record should happen under an exclusive lock, locks the record with RemoteInMemoryLockManager, modifies it and broadcast the change. Then unlocks it using RemoteInMemoryLockManager. The broadcast will reach the server A before the lock.

Yes, performance was my main concern here. I lived under impression that DmoVersioning is consulted every cache hit and not if it is exclusively locked.

What is the next step here?

I must admit that I am a bit unfamiliar with this architecture and maybe Constantin can provide more feedback on the changes. If having more instances of DmoVersioning and Ffcache that are async updated, then I am fine with the approach and I will review more in depth. But at this stage, I want to fully understand if this was the intended design.

Constantin, can you provide a second pair of eyes. Please consider #11734-15 and Eduard's input on #11734-16.

• DB 'hotel' is owned by server A. Thus the 'cluster' has server A as owner and servers B and C as 'clients'.
• server B makes a change, thus in SavepointManager.publishInvalidations it uses RecordInvalidationMultiplexer (which is connected to server A) to notify about this record change
• server A receives the request on a Dispatcher thread, and in turn uses RecordInvalidationMultiplexerImpl.publish to publish only to itself and server C (as server B is the originating server for the change)
  • for server C, the request is done via Session.forwardNoReply - this is an async request and it does not wait for a response, thus it does not wait for the request to finish
  • the request target is the remote (on server C) RecordInvalidationListenerImpl.onInvalidations method
• server C in turn:
  • receives the request on a Dispatcher thread, and executes RecordInvalidationListenerImpl.onInvalidations
  • RecordInvalidationService.onPush then calls apply(event) for each RecordInvalidation received - this in turn takes care of DMO versioning, caches, and anything else which needs to be 'invalidated'.

• versioning.increment(dmoImplName); increments only once. Thus, are we in sync with the owner server, always? Or we are just 'eventually in sync'?
• why the need for !dmoInfo.isTempTable()? This must never be called for temp DMOs, right?
• all this code called in apply is thread-safe, correct?
• Alex - about the DMO versioning; considering that the APIS called by RecordInvalidationMultiplexerImpl.publish are not blocking, then we can't assume that the exclusive lock will still be held, untill after all nodes receive and consume the notifications. Thus, we can end up in a situation (Tx is the time axis):
  • T1: server B calls publishInvalidations after changing record R. This means server A executes the 'async request' to server C, and server B finishes the call.
  • T2: server B ends the commit and releases the lock for record R
  • T3: server C gets an exclusive-lock for record R, for which it uses its cache copy (at this point in time it no longer reflects the DB state, but is not marked as stale)
  • T4: server C only now processes the notification received from server A, about DMO versioning, caches, etc - record R is 'officially stale' with an exclusive-lock
  • I don't think we can allow this situation to happen. Making server A's publish notification blocking (thus waiting for a response from the remote server) may be a solution, but I don't know how to ensure no dead-locks can happen.
• otherwise, for NO-LOCK records, making it 'eventually stale' I don't think is an issue; just the EXCLUSIVE-LOCK records need always to be in sync with the main DB

I still need to finish looking into UniqueTracker changes.

• please fix copyright year and/or history entries for: RemotePersistence, DatabaseMultiplexer, SavepointManager, p2j.net.Session, p2j.net.BaseSession
• all new files: you need to have Copyright (c) 2026

• RemoteUniqueIndex - please add to an interface only the remote methods which are required; having these no-op (actually all the public methods are no-op) is really confusing, especially that they are also abstract in UniqueTracker$UniqueIndex?
• UniqueTrackerAuthority is executed on DB owner's side, by any remote clients
  • this is executed on a Dispatcher thread
  • it requires a context for that server-to-server session
  • it calls beginTxScope on that context via openRemoteScope
  • I do not see any call to a pseudo-commit/finished/etc to decrement these values
• ContextLock is used only by UniqueTracker$LocalUniqueIndex
  • this means that for a database running only in its own FWD server, we will track all the contexts keeping locks, etc. What's the performance impact for this?
  • if we don't have a remote connection to the database, we should still use a ReentrantLock ?
  • LocalUniqueIndex are global per database in the owner FWD server, right? So really I'm not sure why ContextLock is needed. You mean the lock can be obtained remotely by the non-owner FWD server, do some work, and after that release the lock? If this is the case, we need to have this operation entirely executed on the owner FWD server (lock, operation, unlock) and not split this in multiple calls.

Constantin Asofiei wrote:

Eduard, first some notes about my understanding of the implementation related to the record invalidation support:

• DB 'hotel' is owned by server A. Thus the 'cluster' has server A as owner and servers B and C as 'clients'.
• server B makes a change, thus in SavepointManager.publishInvalidations it uses RecordInvalidationMultiplexer (which is connected to server A) to notify about this record change
• server A receives the request on a Dispatcher thread, and in turn uses RecordInvalidationMultiplexerImpl.publish to publish only to itself and server C (as server B is the originating server for the change)
  • for server C, the request is done via Session.forwardNoReply - this is an async request and it does not wait for a response, thus it does not wait for the request to finish
  • the request target is the remote (on server C) RecordInvalidationListenerImpl.onInvalidations method
• server C in turn:
  • receives the request on a Dispatcher thread, and executes RecordInvalidationListenerImpl.onInvalidations
  • RecordInvalidationService.onPush then calls apply(event) for each RecordInvalidation received - this in turn takes care of DMO versioning, caches, and anything else which needs to be 'invalidated'.

Yes, that is correct.

Questions about RecordInvalidationService.apply(event):

• versioning.increment(dmoImplName); increments only once. Thus, are we in sync with the owner server, always? Or we are just 'eventually in sync'?

Lets take a example with 2 nodes (A - database owner, B - remote connection). DmoVersion counter on both nodes are 2 separate things and shouldn't be necessary equal.

Lets suppose that node A caches a record with id: 1, an entry in DmoVersioning is created with version = 0. Then 5 changes are performed of that record, and the version is incremented by 5.

Node B connects to the database, and caches the record and creates a entry in its own DmoVersioning with version = 0. A change on the record is made on node A, and event is created and broadcasted to node B, and then it will increment its entry.

The next time it will be get from cache, the versions will not correspond and the isStale check will not pass.

• why the need for !dmoInfo.isTempTable()? This must never be called for temp DMOs, right?

Actually we don't need this check at all, we will not get an invalidation service for _temp database at all (Persistence.startInvalidationService will reject temoprary databases).

• all this code called in apply is thread-safe, correct?

All it does it to change DmoVersioning.versions map which is a ConcurrentHashMap, and the code in FFCache.invalidate has a critical section which is synchronized.

Constantin Asofiei wrote:

About the UniqueTracker:

• I do not see any call to a pseudo-commit/finished/etc to decrement these values

The cleanup will be done in UniqueTracker.cleanup (normally they are called by TransactionManager but is not our case since is called in Dispatcher on server-to-server context, on remote server it does have its own Context) and ContextLock.cleanup (this will unlock the held locks, when the connection dies unexpectedly).

• ContextLock is used only by UniqueTracker$LocalUniqueIndex
  • LocalUniqueIndex are global per database in the owner FWD server, right? So really I'm not sure why ContextLock is needed. You mean the lock can be obtained remotely by the non-owner FWD server, do some work, and after that release the lock? If this is the case, we need to have this operation entirely executed on the owner FWD server (lock, operation, unlock) and not split this in multiple calls.

The owner's reserve returns with the index still locked, held across the caller's network hop and its database write, and released by a later commit or revert request that lands on a different dispatcher thread. That is the only reason ContextLock exists. This is a replicate o the UniqueTracker implementation. It locks the UniqueIndex in lockAndChange and lockAndDelete. And unlocks it in rollbackChange, or by unlock called by Validation.validateUniqueIndices

Eduard Soltan wrote:

Constantin Asofiei wrote:

About the UniqueTracker:

• I do not see any call to a pseudo-commit/finished/etc to decrement these values

The cleanup will be done in UniqueTracker.cleanup (normally they are called by TransactionManager but is not our case since is called in Dispatcher on server-to-server context, on remote server it does have its own Context) and ContextLock.cleanup (this will unlock the held locks, when the connection dies unexpectedly).

Ah, OK, the increment is done only once.

• ContextLock is used only by UniqueTracker$LocalUniqueIndex
  • LocalUniqueIndex are global per database in the owner FWD server, right? So really I'm not sure why ContextLock is needed. You mean the lock can be obtained remotely by the non-owner FWD server, do some work, and after that release the lock? If this is the case, we need to have this operation entirely executed on the owner FWD server (lock, operation, unlock) and not split this in multiple calls.

The owner's reserve returns with the index still locked, held across the caller's network hop and its database write, and released by a later commit or revert request that lands on a different dispatcher thread. That is the only reason ContextLock exists. This is a replicate o the UniqueTracker implementation. It locks the UniqueIndex in lockAndChange and lockAndDelete. And unlocks it in rollbackChange, or by unlock called by Validation.validateUniqueIndices

OK, what would be some stacktraces for the lock and unlock calls from the remote call?

Otherwise, the point for ContextLock where the owner's DB is not used remotely, still stands.

Constantin Asofiei wrote:

OK, what would be some stacktraces for the lock and unlock calls from the remote call?

Suppose we have server B runs the 4GL code and does not own hotel database ; server A owns the database.

1. Lock, taken on server B while serving reserve on server A, inside the flush of a record with a dirty unique index

B, conversation thread of the user context
   Queue.transact(Queue.java:751)                            <- blocks until A replies
   BaseSession.transact(BaseSession.java:273)
   HighLevelObject.transact(HighLevelObject.java:222)
   RemoteObject$RemoteAccess.invokeCore(RemoteObject.java:1468)
   InvocationStub.invoke(InvocationStub.java:144)
   $Proxy.reserve                                            <- UniqueTrackerMultiplexer proxy of this context
   RemoteUniqueTracker.lockAndChange(RemoteUniqueTracker.java:193)
   UniqueTracker$Context.lockAndChange(UniqueTracker.java:775)
   Validation.validateUniqueIndices(Validation.java:505)
   Validation.validateMaybeFlush(Validation.java:388)
   RecordBuffer.validateMaybeFlush(RecordBuffer.java:11845)
   ... RecordBuffer.validate()/flush()

On server A:

A, DISPACHER thread N, context = A's server-to-server session
   ContextLock.take(ContextLock.java:351)                    <- owner := Owner of B's session, holds = 1
   ContextLock.lockInterruptibly(ContextLock.java:201)
   ContextLock.lock(ContextLock.java:143)
   UniqueTracker$UniqueIndex.lock(UniqueTracker.java:1408)
   UniqueTracker.lockAndChange(UniqueTracker.java:392)       <- stock code, on the DMO materialized from the request
   UniqueTrackerAuthority.reserve(UniqueTrackerAuthority.java:175)
   UniqueTrackerMultiplexerImpl.reserve(UniqueTrackerMultiplexerImpl.java:136)

   Dispatcher.processInbound(Dispatcher.java:812)
   Dispatcher.dispatch(Dispatcher.java:942)
   Dispatcher$DispatcherStub.run(Dispatcher.java:1761)

2. Unlock, on Server A, when Server B flush succeeded:

SERVER B:

   $Proxy.commit -> InvocationStub.invoke -> ... -> Queue.transact   (same frames as above)
   RemoteUniqueTracker.unlock(RemoteUniqueTracker.java:278)
   UniqueTracker$Context.unlock(UniqueTracker.java:842)
   Validation.validateUniqueIndices(Validation.java:566)     <- finally
   Validation.validateMaybeFlush(Validation.java:388)
   RecordBuffer.validateMaybeFlush(RecordBuffer.java:11845)

Server A, DISPACHER thread, same context as above
   ContextLock.unlock(ContextLock.java:289)                  <- owner == Owner of B's session
   UniqueTracker$UniqueIndex.unlock(UniqueTracker.java:1416)
   UniqueTracker.unlock(UniqueTracker.java:524)
   UniqueTrackerAuthority.commit(UniqueTrackerAuthority.java:221)
   UniqueTrackerMultiplexerImpl.commit(UniqueTrackerMultiplexerImpl.java:154)
   java.lang.reflect.Method.invoke
   MethodInvoker.invoke(MethodInvoker.java:126)
   Dispatcher.processInbound(Dispatcher.java:812)
   Dispatcher.dispatch(Dispatcher.java:942)
   Dispatcher$DispatcherStub.run(Dispatcher.java:1761)

3. Unlock, on Server A, when Server B flush failed with a ValidationException* (the database rejected the row). From the catch on B:

Server B
   $Proxy.revert -> ... -> Queue.transact
   RemoteUniqueTracker.rollbackChange(RemoteUniqueTracker.java:303)   <- then token.consume()
   UniqueTracker$Context.rollbackChange(UniqueTracker.java:823)
   Validation.validateUniqueIndices(Validation.java:551)     <- catch (ValidationException)

SERVER A, dispatcher thread
   ContextLock.unlock(ContextLock.java:289)
   UniqueTracker$UniqueIndex.unlock(UniqueTracker.java:1416)
   UniqueTracker.rollbackChange(UniqueTracker.java:597)      <- private rollbackChange(Long, List): drop new key, restore old key, unlock
   UniqueTracker.rollbackChange(UniqueTracker.java:543)
   UniqueTrackerAuthority.revert(UniqueTrackerAuthority.java:259)
   UniqueTrackerMultiplexerImpl.revert(UniqueTrackerMultiplexerImpl.java:163)
   ... dispatcher frames as above

Otherwise, the point for ContextLock where the owner's DB is not used remotely, still stands.

The pick has to happen at creation, and creation knows nothing useful. The index is created by UniqueTracker.getInstance the first time any session touches the table, and the tracker is cached for the life of the server.

But I think the ContextLock can be optimized to take as parameter the Context at lock and unlock from the session's UniqueTracker.Context this way it can be taken in one dispatcher request and released in another.

For the stacktraces: I see it now, we have Validation which is still executed on server B (non-owner), and it requires to lock/unlock via the remote UniqueTracker.

The pick has to happen at creation, and creation knows nothing useful. The index is created by UniqueTracker.getInstance the first time any session touches the table, and the tracker is cached for the life of the server.

My point is more about when there is just server A using the database, and no other 'client' servers connect to it. In this case, what's the performance impact of ContextLock instead of the previous ReentrantLock?

But I think the ContextLock can be optimized to take as parameter the Context at lock and unlock from the session's UniqueTracker.Context this way it can be taken in one dispatcher request and released in another.

I don't think this is needed.

Are the other points I mentioned in WIP?

Constantin Asofiei wrote:

My point is more about when there is just server A using the database, and no other 'client' servers connect to it. In this case, what's the performance impact of ContextLock instead of the previous ReentrantLock?

This is some AI anaysis:

Are the other points I mentioned in WIP?

Committed on 11734a, rev. 16726. Extracted RemoteUniqueIndex methods into an interface and fixed history entries and java docs.

Constantin Asofiei wrote:

• I don't think we can allow this situation to happen. Making server A's publish notification blocking (thus waiting for a response from the remote server) may be a solution, but I don't know how to ensure no dead-locks can happen.

• otherwise, for NO-LOCK records, making it 'eventually stale' I don't think is an issue; just the EXCLUSIVE-LOCK records need always to be in sync with the main DB

Also fix this one in rev. 16727. I don't think a deadlock could happen: C's apply path touches DmoVersioning and the FastFindCache monitor only. Neither makes a network request or takes a lock a C thread could hold while waiting on A.

I have some changes that improve ContextLock performance.

Instead of a synchronized monitor plus a per-context set recording every acquire, it's now ReentrantLock's own synchronizer (AbstractQueuedSynchronizer) with the context written where the thread would go — one compare-and-set operation to take, one write to release, a FIFO queue for waiters.

1) Taking it — the ordinary case

UniqueTracker.lockAndChange walks uniqueIndexes[] and, for each index the record touches, calls ui.lock():

  ui.lock() → UniqueIndex.lock() → lock.lock() → ContextLock.lock()
     → await(0) → sync.tryAcquireNanos(1, 30s) → Sync.tryAcquire(1):
          OWNER.get()                  which context am I
          state == 0 → CAS 0 → 1       take it
          owner = me                   write the name
          return true

2) If another context has it

tryAcquire fails → AbstractQueuedSynchronizer puts a node for this thread on that lock's queue and parks it. It waits in 30-second slices (await, ContextLock.java): if a slice expires it logs the stall warning once and takes a fresh place in the queue. When the holder leaves, AbstractQueuedSynchronizer unparks exactly the first node — in arrival order, not a free-for-all.

3) Giving it back

 ContextLock.unlock() → sync.release(1) → Sync.tryRelease(1):
          owner != OWNER.get() → IllegalMonitorStateException
          state - 1; if 0 → owner = null
     → AbstractQueuedSynchronizer unparks the queue head if anyone is waiting

With this implementation the time take is comparable to the ReentrantLock taken time:

Committed on 11734a, rev. 16728.