Bug #11734
persistence caches for remote databases
100%
History
#1 Updated by Constantin Asofiei 21 days ago
FWD allows server A to connect to remote FWD server B, to access the database instance managed by server B. This requires directory configuration for the remote FWD server access, and also a direct JDBC connection for the database.
Server A AFAIK at this time uses only remote dirty share, persistence and primary key calculations. Other caches (like FFCache, ORM Session cache) for the database served by server B, I think live on server A, while they can never be invalidated by server B.
This task is meant to investigate and find solutions for these cases.
#3 Updated by Constantin Asofiei 16 days ago
Another part on which the authoritative database and the remote database (and by these I mean the corresponding FWD servers) are not in sync is the DMOVersioning.
#5 Updated by Alexandru Lungu 12 days ago
@Constantin Asofiei could you raise the prio to high as the workaround (in related issue) is on a per table basis on one entry now for the reportserver which is not a desired solution to the broader problem.
I reopen the OG task on VMA project and move it to High. This is in the public section like this one (Database) is not properly tracked for customer delivery.
#6 Updated by Teodor Gorghe 7 days ago
- PSQL supports
LISTEN/NOTIFY, which allows us to implement the Observer design pattern. - What helps to us: we can have a deterministic approach about database cache invalidation, when multiple servers/sessions connect to the same database. We can have a database write trigger which notifies events to all listeners, to invalidate DMO cache.
- More about in: https://www.baeldung.com/java-postgresql-listen-notify-events
#7 Updated by Eduard Soltan 7 days ago
Things that distributed across the cluster:
- Record locks (LockManagerFactory) - regions <db>_lockTable, <db>_inPlay; delta-encoded LockStatus values
- Unique indexes (UniqueTracker) - DistributedUniqueIndex, 2 global regions
- FastFind cache invalidation (FastFindCache) - FastFindCacheInvalidationFunction broadcast
- Tenant registry (TenantManager) - generation region (ensure strong consistency) + TenantRegistryReloadFunction broadcast
- Web-service session tokens (WebTokenRegistryFactory) - DistributedWebTokenRegistry, region TOKENS_REGION_NAME
- Security cache generations (domains, domain types, users) (SecurityCacheGenerationsFactory) - DistributedSecurityCacheGenerations, region GENERATIONS_REGION_NAME
- DirectoryService (RedisRemapper), directory LockManager (right now Redisson locking backend, 4369c LUA script approach pending)
Things that still are not distributed : DmoVersioning, SecurityManager.cache
#8 Updated by Constantin Asofiei 7 days ago
Eric, I think for UniqueTracker, DmoVersioning and FastFindCache we need to add remote support, for direct connections to a non-owner database.
It is a priority for #11730
#9 Updated by Eduard Soltan 6 days ago
- Assignee set to Eduard Soltan
#10 Updated by Alexandru Lungu 6 days ago
Eric, I think for UniqueTracker, DmoVersioning and FastFindCache we need to add remote support, for direct connections to a non-owner database.
Constntin, do you consider this being a final fix for #11730 rather than the direct Java access full cache clear? I would personally say so, because the cache sweep looks more like an workaround of the capability of remote FWD server to share its DmoVersioning and FastFindCache. I really thought UniqueTracker is managed remotely, but you are right, it is not.
I have some changes for #11730 to improve the API, but I am not sure if I should hurry to deliver to the customer or take my time, build the native SQL wiki for modifying data properly, write tests and engineer that API completely. If so, please raise the priority of this task to match #11730 (High) and lets have it delivered by next Thursday.
PS: for dirty-share I reckon that, as long as the cross-session variant is disabled, there is no need to make it remote, though it has such implementation exactly for such cases. After #8388, remote dirty-share will be mostly obsolete.
#11 Updated by Constantin Asofiei 6 days ago
Alexandru Lungu wrote:
Eric, I think for UniqueTracker, DmoVersioning and FastFindCache we need to add remote support, for direct connections to a non-owner database.
Constntin, do you consider this being a final fix for #11730 rather than the direct Java access full cache clear? I would personally say so, because the cache sweep looks more like an workaround of the capability of remote FWD server to share its
DmoVersioningandFastFindCache. I really thoughtUniqueTrackeris managed remotely, but you are right, it is not.
Yes, if we configure the two apps correctly, and the remote APIs are executed properly (the db owner notifies the client(s) properly), then there is no reason the use that workaround to clear the caches.
I have some changes for #11730 to improve the API, but I am not sure if I should hurry to deliver to the customer or take my time, build the native SQL wiki for modifying data properly, write tests and engineer that API completely. If so, please raise the priority of this task to match #11730 (High) and lets have it delivered by next Thursday.
Those apps already have some other code which uses hand-written SQL. I would rather deliver #11734 than move further into this workaround with manual cache invalidation. Once we have the APIs documented and working properly in #11730, we can work with the customer to improve/change the direct-SQL approach - but please make sure existing usage is not broken after your improvements, please.
#12 Updated by Eduard Soltan 6 days ago
- Priority changed from Normal to High
#13 Updated by Eduard Soltan 2 days ago
- % Done changed from 0 to 100
- Status changed from New to WIP
Remote DmoVersioning and FFCache¶
Hub topology. One RecordInvalidationBroker per database per node, but only the broker on the owning node is ever used. The owner's own service talks to it in process; non-owners reach it through RecordInvalidationMultiplexer, over the same virtual session already multiplexing record locks and identity allocation. Every node publishes what it commits; every other node is handed it.
Push only — no polling, no cursor. Subscribers register a sink at subscription time, and each event (cache invalidation event) is delivered to every sink but the publisher's, on the publisher's thread, at publication time. Nothing is retained after delivery.
Publish is driven from SavepointManager, immediately after the local versions are incremented on commit. Apply increments the affected records' DmoVersioning entries and invalidates the matching FastFindCache entries. Incrementing the canonical version is sufficient to make every cached copy on that node report itself stale through Session.getCached() — the record read path is unchanged.
Remote UniqueTracker¶
UniqueTracker.getInstance(DmoMeta, Database) now takes the Database instead of its name, and instantiates a RemoteUniqueTracker when !database.isLocal().
Caller side — RemoteUniqueTracker extends UniqueTracker. Overrides the tracker operations and forwards each as one request to a proxy for UniqueTrackerMultiplexer on the owning server. A multiplex ID identifies the target database, as with the lock and identity managers. The proxies are held per context (in a ContextLocal keyed by network session), not per tracker, because a tracker is a per-table singleton for the life of the server while the network session is per context.
Owner side — UniqueTrackerAuthority. One per owned database, reached via UniqueTrackerMultiplexerImpl extends DatabaseMultiplexer. It materialises a DMO of its own implementation class from the field values sent (DMOs aren't serializable, so records can't cross the wire) and then runs the stock lockAndChange / lockAndDelete / rollbackChange against the same trackers its own sessions use. That's why a violation needs no special handling — the legacy error is composed on the owner by existing code and propagates back as the exception of the call.
The split. The transaction, the JDBC connection, and the blocks that open and end scopes all belong to the caller. The owner is never told about scopes — it learns only that a record has reserved a key or no longer holds one. So the undo record stays on the caller, in its ordinary UniqueTracker.Context, and scope end drives release/restore in bulk, one request per index. Token is serializable and opaque to the caller.
Locking. reserve returns with the owner's indexes still held; commit or revert releases them, so a racing session waits for the caller's flush exactly as it would behind a local session. Since successive requests from one remote session arrive on different dispatcher-pool threads, a ReentrantLock can't be released by the second one — hence ContextLock, a mutex owned by the context rather than the thread, reentrant for that owner. LocalUniqueIndex uses it throughout; for a local session (one thread per context) the behaviour is identical to before.
Safety net. Keys reserved for a remote session are also tracked in that session's own Context on the owner, in a single scope no block ever ends, and untracked as the caller releases them. Its purpose is cleanup: when the session dies, ContextLocal cleanup releases whatever the caller never got to release, and ContextLock frees the locks it still held.
Committed on 11734a, rev 16724.
#14 Updated by Eduard Soltan 2 days ago
- Status changed from WIP to Review
- reviewer Alexandru Lungu, Constantin Asofiei added
Please review.