# BufferManager.java

    /*
    ** Module   : BufferManager.java
    ** Abstract : Coordinates life cycle of record buffers
    **
    ** Copyright (c) 2004-2025, Golden Code Development Corporation.
    **
    ** -#- -I- --Date-- --JPRM-- -----------------------------------Description-----------------------------------
    ** 001 ECF 20050926   @23246 Created initial version. Manages multiple
    **                           record buffers within each context. Maps
    **                           legacy transactions to database-level
    **                           transactions and registers buffers for commit
    **                           and rollback notifications.
    ** 002 ECF 20051108   @23376 Adjust to new architecture of one Persistence
    **                           instance per database. Most changes were in
    **                           the inner class DBTxWrapper.
    ** 003 ECF 20051205   @23670 Renamed method obtain() to get() for
    **                           consistency within package.
    ** 004 ECF 20060116   @23950 Fixed regression in initializeTransactions.
    **                           Was impossible to begin a new database
    **                           transaction.
    ** 005 ECF 20060120   @24037 Integration with P2J server. Replaced
    **                           ThreadLocal with ContextLocal.
    ** 006 ECF 20060130   @24285 Changed openScope signature. Now returns
    **                           current scope depth.
    ** 007 ECF 20060205   @24286 Implement BatchListener and register with the
    **                           TransactionManager for batchNotify events.
    **                           Triggers start/end of record buffer batch
    **                           assign mode.
    ** 008 ECF 20060428   @25869 Fixed dangling transaction defect. A database
    **                           level which is not explicitly committed or
    **                           rolled back by the TransactionManager is now
    **                           closed by auto-rollback processing.
    ** 009 GES 20060517   @26197 Match new Finalizable interface requirement.
    ** 010 ECF 20060610   @27090 Fixed retry defect. A new database level
    **                           transaction must be opened upon master
    **                           transaction retries. This prevents a later
    **                           commit or rollback from failing.
    ** 011 ECF 20060621   @27503 Added activeRecords method. Returns iterator
    **                           on the records currently stored in all record
    **                           buffers in the current client context for a
    **                           particular database.
    ** 012 ECF 20060627   @27587 Modified activeRecords method. Only include
    **                           non-transient records to avoid saving records
    **                           which were not meant to be saved.
    ** 013 ECF 20060628   @27668 Added getCurrentScope method. Reports current
    **                           buffer scope level.
    ** 014 ECF 20060719   @28110 Simplified activeRecords() implementation.
    ** 015 ECF 20060728   @28263 Moved DMO reference counting mechanism from
    **                           RecordBuffer. incrementDMOUseCount() and
    **                           decrementDMOUseCount() are now implemented
    **                           here.
    ** 016 GES 20060804   @28431 Some javadoc improvements.
    ** 017 ECF 20060916   @29664 Added DMO property-level undo support. A
    **                           scoped dictionary tracks all changes by DMO
    **                           type, necessary to undo changes to any block
    **                           scope. Also moved storage of record-level
    **                           reversible actions from RecordBuffer class.
    ** 018 ECF 20060919   @29695 Fixed DMO eviction. The DMO implementation
    **                           class must be passed to decrementDMOUseCount,
    **                           because there previously was not enough
    **                           information available to detect the proper
    **                           DMO type when evicting objects from the
    **                           persistence session.
    ** 019 ECF 20061010   @30307 Added support for auto-commit temp tables.
    **                           Buffers which auto-commit are not managed by
    **                           this class' transaction mechanism.
    ** 020 ECF 20061011   @30355 Rewrote logic to begin database transactions.
    **                           Eliminated initializeTransactions() method
    **                           and simplified scopeStart() and openScope()
    **                           logic. This change closes some loopholes
    **                           which would not have begun database level
    **                           transactions at the correct time.
    ** 021 ECF 20061018   @30486 Moved processing to clean up batch assign
    **                           mode from RecordBuffer. This was interfering
    **                           with end-of-scope processing in some cases.
    ** 022 ECF 20061114   @31159 Moved batch assign mode state management from
    **                           RecordBuffer to this class. Added package
    **                           private methods to allow RecordBuffer access
    **                           to batch mode state.
    ** 023 ECF 20061207   @31618 Fixed DMO validation problem. ChangeBroker
    **                           needs to be marked for a pending session
    **                           flush whenever a record buffer is added to
    **                           the set of dirty buffers in batch assign
    **                           mode.
    ** 024 ECF 20070130   @32037 Fixed handling of no-undo temp-tables. Change
    **                           undone with a database-level transaction
    **                           rollback is re-rolled forward and committed,
    **                           as if the rollback had not occurred.
    ** 025 ECF 20070412   @32976 Fixed end of transaction lock processing.
    **                           Record locks for a session are now managed by
    **                           RecordLockContext.
    ** 026 ECF 20070414   @32993 Fixed memory leak. DBTxWrapper was holding on
    **                           to record buffers after they had gone out of
    **                           scope. The new locking architecture has
    **                           eliminated the need for this object to hold
    **                           buffer references at all.
    ** 027 ECF 20070502   @33368 Added lookupBuffer(). Required changes to how
    **                           scoped dictionary for open buffers is managed
    **                           internally.
    ** 028 ECF 20070503   @33390 Added code to track all defined buffers in a
    **                           scoped dictionary. Added registerBuffer()
    **                           method to allow new buffers to register
    **                           themselves. Changed lookupBuffer() to use the
    **                           new dictionary of all buffers rather than
    **                           just open buffers.
    ** 029 ECF 20070515   @33668 Optimized startScope() slightly. Only process
    **                           DBTxWrapper if beginning a full transaction.
    ** 030 ECF 20070601   @33916 Fixed potential memory leak. Release all
    **                           reversible rollbacks at the end of a database
    **                           transaction, rather than only at rollback.
    ** 031 ECF 20070821   @34901 Added setSilentError() method. Enables error
    **                           suppression during query construction and/or
    **                           initialization to mimic Progress' lenient
    **                           error handling during its processing of a
    **                           where clause.
    ** 032 ECF 20070911   @35283 Replaced database name string with Database
    **                           information object. Required for non-local
    **                           database connections.
    ** 033 ECF 20071024   @35594 Added call to allow ConnectionManager to
    **                           clean up after a transaction ends. This was
    **                           necessary to avoid this cleanup too early in
    **                           the end-of-transaction logic flow. This
    **                           problem was possible with the previous
    **                           implementation, in which ConnectionManager
    **                           registered itself with the TM independently.
    ** 034 ECF 20071029   @35632 Changed in[de]crementDMOUseCount() signature.
    **                           Return use counts after change is made. Also
    **                           integrated some use of generics.
    ** 035 ECF 20080516   @38600 Added support for dirty index update sharing.
    **                           Integrated generics.
    ** 036 ECF 20080606   @38637 Fixed reversible rollback processing. Ensure
    **                           that it only takes place when rolling back
    **                           temp table transactions.
    ** 037 SVL 20080607   @38638 Fixed DBTxWrapper.iterate(). This method
    **                           needs to perform cleanup processing for the
    **                           master transaction by calling end() before
    **                           begin(). [ECF: same change made for retry()].
    ** 038 CA  20080808   @39357 Implemented no-undo temp-table rollback - on
    **                           db tx rollback, the re-roll actions are
    **                           processed for the no-undo buffers.
    ** 039 CA  20080826   @39568 Added rollbackPending which is called by TM
    **                           when rollback targets a parent block.
    ** 040 ECF 20080827   @39601 Fixed lookupBuffer(). Addition of
    **                           pendingBuffers logic regressed this method.
    ** 041 CA  20080820   @39451 Implemented DatabaseConnectionHandler - it
    **                           checks if an exception related to database
    **                           connectivity errors should be re-thrown or
    **                           not. Fixed leak in allBuffers scoping.
    ** 042 ECF 20080925   @39945 Modified for ReversibleChain API change. That
    **                           class' c'tor requires an additional argument.
    ** 043 ECF 20081020   @40207 Modified for Reversible API change. Method
    **                           Reversible.rollback() now requires arguments.
    ** 044 ECF 20081118   @40548 Fixed reversible rollback processing in
    **                           DBTxWrapper.rollback(). Needed a loop before
    **                           actual rollback, to call Reversibles'
    **                           prepareRollback() method.
    ** 045 ECF 20081127   @40711 Implement reclamation of temp table primary
    **                           keys. On full transaction commit/rollback,
    **                           keys which have been collected for recycling
    **                           are made available for re-use, or discarded,
    **                           respectively.
    ** 046 SVL 20081204   @40795 Added nestedOpen parameter to openScope()
    **                           method.
    ** 047 ECF 20090114   @41161 Minor change to DBTxWrapper. begin() is now
    **                           called from scopeStart() after all buffers
    **                           have been processed. It is called from
    **                           openScope() for a newly created DBTxWrapper.
    ** 048 ECF 20090121   @41193 Fixed defect in scopeStart()/scopeFinished().
    **                           Buffers were having their enterBlock() and
    **                           exitBlock() methods (respectively) invoked
    **                           too often in the case of nested, open buffer
    **                           scopes.
    ** 049 ECF 20090128   @41238 Changed activeRecords() to activeBuffers().
    **                           Now returns an iterator on all active record
    **                           buffers, rather than just the records they
    **                           contain. The caller required additional
    **                           information stored by the buffers themselves.
    ** 050 ECF 20090206   @41265 Added support for evicting unused DMOs in the
    **                           P2J session interceptor. Exposed a view of
    **                           DMOs currently in use so that the session
    **                           interceptor can evict unused DMOs after a
    **                           session flush. Also implemented a minor
    **                           performance improvement for batch mode. We
    **                           now pass the BufferManager instance to
    **                           RecordBuffer.cleanupBatchMode() method to
    **                           avoid an unnecessary context-local lookup.
    ** 051 ECF 20090218   @41330 Changed idiom for scoped maps of Reversibles.
    **                           The get*() methods which return these maps
    **                           will now return null if the map did not
    **                           previously exist, and the caller did not set
    **                           the create option to true. This avoids the
    **                           unnecessary creation of maps in the common
    **                           case of looking up an entry which will never
    **                           be found.
    ** 052 ECF 20090221   @41444 Added removeAllReversibles(). Also removed
    **                           unused parameter from decrementDMOUseCount().
    ** 053 ECF 20090315   @41608 Fixed NO-UNDO temp-table reversible rollback
    **                           processing. It is necessary to prepare the
    **                           reversibles for rollback before the old
    **                           session is rolled back. The reversals are
    **                           then executed in the new session. This allows
    **                           us to initialize lazy persistent collections
    **                           for extent fields before the old session is
    **                           gone. This is a corner case for reversible
    **                           deletes when the target DMO is also the
    **                           current record in the buffer at the time of
    **                           rollback.
    ** 054 ECF 20090610   @42632 Added evictDMOIfUnused(). Evicts DMO from
    **                           Hibernate session if no resource references it.
    ** 055 ECF 20090623   @42962 Simplified internals and reduced memory
    **                           footprint. Refactored four ScopedDictionary
    **                           instances for reversible/undoable processing into
    **                           one ScopedDictionary of UndoableData objects.
    **                           Simplified API to access reversible/undoable
    **                           action maps.
    ** 056 ECF 20090701   @43010 Fixed regressions introduced with #055 (@42962).
    **                           getReversibles() and getUndoables() were not
    **                           properly looking up or creating entries in the
    **                           backing ScopedDictionary.
    ** 057 ECF 20090702   @43027 Fixed additional regressions from #055 (@42962).
    **                           Reversible rollbacks were not being processed
    **                           correctly.
    ** 058 ECF 20090716   @43213 Enabled record buffers to be lazily initialized.
    **                           This required some refactoring with respect to
    **                           scope open/close logic. Also ensure change is
    **                           flushed to the database at the close of each
    **                           block.
    ** 059 ECF 20090724   @43415 Fixed reversible/undoable mapping. Use master
    **                           buffer instead of current buffer as map key. This
    **                           avoids losing undo information for shared temp
    **                           table buffers which go out of scope before their
    **                           master temp tables. Also changed session flush
    **                           strategy. We now try to flush before rollback at
    **                           any level, rather than on scope finish events.
    ** 060 ECF 20090729   @43447 Adapted Commitable API change. Added validate()
    **                           method to Commitable implementations.
    ** 061 CA  20090807   @43556 Reversibles must be kept by database and table
    **                           name, not by RecordBuffer instance - to ensure
    **                           proper reversible order when rolling back records
    **                           in a table with multiple buffers defined for it.
    **                           This fixes @43542.
    ** 062 ECF 20090810   @43582 Changed getBufferForEntity() to prevent NPE. This
    **                           is needed, as RecordBuffer.getDatabaseAndTable()
    **                           will now return null instead of throwing an
    **                           IllegalStateException if the buffer is not yet
    **                           initialized. Throw RuntimeException in various
    **                           places (rather than a ConditionException), if an
    **                           error is unrecoverable.
    ** 063 ECF 20090811   @43586 Changed decrementDMOUseCount() to prevent NPE.
    **                           In a case where the thread is interrupted due to
    **                           a session ending, it was possible for the
    **                           increment and decrement calls to become out of
    **                           balance.
    ** 064 ECF 20090811   @43598 Fixed regression in #061 (@43556). Reversibles
    **                           were being stored by database and table name, but
    **                           retrieved by RecordBuffer instance.
    ** 065 ECF 20090812   @43602 Changed Reversible key from String to new class
    **                           BufferType. Removed getBufferForEntity(), which
    **                           would fail to find a matching buffer in cases
    **                           where buffers had their scopes closed before a
    **                           transaction ended. A buffer lookup is no longer
    **                           needed, since BufferType contains all information
    **                           needed by former callers of that method.
    ** 066 ECF 20090816   @43665 Integrated with FieldAssigner. We get a context
    **                           local Scopeable from that class, which we notify
    **                           at each scope open and close event.
    ** 067 SVL 20091117   @44401 Added facility for releasing buffers into which a
    **                           new record was loaded (or the old record was
    **                           reloaded) at specific points of block processing.
    ** 068 CA  20091202   @44466 Reworked the rollback mechanism, to maintain
    **                           proper reversible order and thus to solve some
    **                           unique-constraint violations during rollback of
    **                           some bulk delete cases and mixed record operations.
    ** 069 SIY 20100429   @44838 Fixed double rollback in some conditions.
    ** 070 CS  20121031          Removed client specific references.
    ** 071 OM  20130902          Added getDMOUseCount() for detecting if a dmo is referred by more
    **                           than one record buffer.
    ** 072 HC  20130902          For #2164 added BufferManager initialization check.
    **                           BufferManager is initialized when its initialize()
    **                           method is called. If not initialized get() throws.
    ** 073 SVL 20130822          Added openFreeScope.
    ** 074 ECF 20131028          Made class and getCurrentScope() public and added import statement to
    **                           accommodate new lock sub-package.
    ** 075 SVL 20131028          Added openScopeAt and tracking functions for static temp tables.
    ** 076 CA  20131209          Added no-op deleted() and scopeDeleted() methods, requried by the
    **                           changes in Scopeable and Finalizable interfaces.
    ** 077 SVL 20140124          Added includeTransient parameter for activeBuffers() function.
    ** 078 SVL 20140221          [INPUT-]OUTPUT TABLE-HANDLE parameters are tracked.
    ** 079 CA  20140301          Allow buffers to survive the persistent procedure's external
    **                           procedure.
    ** 080 OM  20140417          Fixed typo in method name. Simplified string concatenation
    **                           operations.
    ** 081 CA  20140407          Fixed scopeDeleted(), when it is called on deletion of an appserver
    **                           persistent procedure.
    ** 082 CA  20140513          Added a weight for the context-local var, to ensure predetermined
    **                           order during context reset.
    ** 083 ECF 20140813          Manage batch assign brackets in a scoped manner, since now these
    **                           brackets can be nested (requirement of database triggers). Replaced
    **                           Apache commons logging with J2SE logging.
    ** 084 ECF 20150124          Lazily initialize ConnectionManager to avoid premature use of
    **                           persistence services during server startup.
    ** 085 ECF 20150220          Rewrote unique buffer name algorithm to ensure names are unique among
    **                           all buffers in the current context, but are consistently reused for a
    **                           given base, legacy name. This allows the dynamic query cache to
    **                           operate more efficiently.
    ** 086 CA  20150312          Fixed cleanup of tracked buffers; scope notifications are now
    **                           properly raised when the external procedure gets terminated.
    ** 087 CA  20150320          Backed out H086 - it caused a regression and the change is not part
    **                           of the root mem leak cause.
    ** 086 ECF 20150322          Partially rewrote unique buffer algorithm introduced with H086 to not
    **                           use String.format(), which is very inefficient. Added deregistration
    **                           of dynamic buffers to prevent leakage of tracked, dynamic buffers.
    **                           Made other performance improvements.
    ** 087 ECF 20150327          Enhanced logging to determine source of a conflicting transaction
    **                           when attempting to begin a new one.
    ** 088 ECF 20150906          Reimplemented implicit silent error mode for query substitution
    **                           parameter processing. Simplified and optimized in-use DMO tracking.
    ** 089 ECF 20150929          Optimized unused DMO eviction.
    ** 090 EVL 20160223          Javadoc fixes to make compatible with Oracle Java 8 for Solaris 10.
    ** 091 OM  20160212          Added cleanupPending() method. Finalizable API has changed. Upgrade
    **                           code to Java8.
    ** 092 ECF 20160321          Replaced ScopedDictionary with more performant ScopedList for
    **                           openBuffers data structure. Fixed memory leak in scopeDeleted. Fixed
    **                           removeAllReversibles to remove reversibles from parent scope as well
    **                           as current scope.
    ** 093 ECF 20160429          Optimized removeAllReversibles. Added toString to RefCount.
    ** 094 ECF 20160605          Replace TM.registerCommit with TM.registerCommitAt, to avoid extra
    **                           context-local dereferencing in TM.
    ** 095 ECF 20160823          Fixed primary key reclamation algorithm. Unfortunately, it was later
    **                           deemed that the reclamation concept itself seems inherently unsafe,
    **                           because of the potential for deadlocking due to out-of-order keys.
    **                           However, I want to keep this code around in case we figure out a way
    **                           to use this safely in the future.
    ** 096 ECF 20160829          Performance enhancement: track transaction status so other objects
    **                           in this context can use this local resource to check transaction
    **                           status instead of TransactionManager.
    ** 097 GES 20171207          Removed silent error mode processing.
    ** 098 CA  20180619          DBTxWrapper notifications are DB-specific, so
    **                           ConnectionManager.transactionEnd must be informed of the database
    **                           being processed.
    ** 099 CA  20181112          Some refactoring required to explicit begin or end a tx, usable by
    **                           the TRANSACTION:SET-COMMIT/ROLLBACK() method implementations.
    ** 100 OM  20181031          Added BufferManager-level unknown mode support.
    ** 101 OM  20190111          Avoided circular indirect recursivity at initialization when _Connect
    **                           meta-table is active.
    ** 102 HC  20190206          Fixed a potential NPE in buffer processing during an error
    **                           condition.
    **     OM  20190309          Mapped StaticTempTables by a key formed from the legacy name and the
    **                           This-Procedure to avoid collisions.
    **     ECF 20190313          Extracted RefCount inner class to a top level class.
    **     ECF 20190322          Refactored to make this class Commitable, to reduce the number of
    **                           individual registrations with the TransactionManager, and to
    **                           optimize notifications of transaction events to open buffers.
    **     CA  20190325          Implemented ProcedureHelper instead of direct usage of the static
    **                           API.  This allows the elimination of context local usage in
    **                           ProcedureManager.
    **     ECF 20190325          Replaced TransactionManager static calls with TransactionHelper
    **                           instance calls to reduce context-local lookups.
    ** 103 ECF 20190414          Track records currently being deleted by buffer type and primary key.
    ** 104 ECF 20190714          Added support for aggressive buffer flushing mode.
    **     ECF 20190725          Filter "fake" write trigger old buffers from those returned by
    **                           activeBuffers.
    **     ECF 20190805          Prevent an IllegalStateException when a master iterate call tries to
    **                           begin a transaction immediately after the DBTxWrapper c'tor just did
    **                           so, without an intervening commit or rollback. This can happen if the
    **                           object is constructed after commit processing, during iterate
    **                           processing, by schema trigger business logic.
    **     ECF 20190813          Change to RecordBuffer.release API.
    ** 105 CA  20190718          Lookup a table in the entire stacktrace, not just current program.
    **     CA  20190722          Allow lookup for handles registered as OUTPUT arguments.
    **     CA  20190724          As the legacy field names are emitted in the same case as they were
    **                           defined, the TempTableKey must be case-insensitive.
    **     ECF 20190725          Filter "fake" write trigger old buffers from those returned by
    **                           activeBuffers.
    **     OM  20190802          Added registry for looking up (static) buffers by their legacy name.
    **     CA  20190816          Static temp-tables are now created when their default buffer is
    **                           defined.
    ** 106 CA  20191013          Added where and sort clause translation in case of bound buffers.
    ** 107 AIL 20191119          Transaction rollback should be done only if a transaction is active.
    **     CA  20191119          Track the executing legacy class, to be able to register any buffer
    **                           to its source definition class.
    **     CA  20200110          Fixed issues with buffer registration for OE class hierarchy.
    ** 108 CA  20200323          A fix for binding a source buffer's DMO alias to the destination: this
    **                           must be permanent only in case of BIND arguments.  Otherwise, the
    **                           DMO alias binding is limited only for the duration of that top-level
    **                           block call.
    ** 109 ECF 20200419          New ORM implementation.
    **     CA  20200722          Performance improvements.
    **     CA  20200827          Changed pendingActiveDatabases to activeDatabases map to count the open buffers
    **                           for a certain database.
    ** 110 AIL 20200616          Added pm notification when a static record buffer is changed / reloaded.
    **     AIL 20200622          The pm should also be notified of temporary record buffer changes.
    **     ECF 20200909          Treat resetState as an undo operation and do not flush a buffer on release.
    **     CA  20200921          Avoid NPE in deregisterBindingBuffer if the buffer was already deregistered (as
    **                           it closed its scope).
    **     CA  20200927          Use IdentityHashMap instead of plain map when the key is a Class.
    **     CA  20201003          Replaced Guava identity HashSet with Collections.newSetFromMap(IdentityHashMap).
    **                           Allow dictionaries with an identity map.
    **     OM  20201231          Fixed buffer state in resetState().
    **     AIL 20210319          Commit instead of rollback no-undo tables.
    **     ECF 20210414          Changed scope transition behavior to only update scoped data structures when
    **                           entering or leaving block scopes which are either outside an application level
    **                           transaction or, if within a transaction, which have transaction block properties.
    **     OM  20210429          Release the current record at the end of the buffer scope and decrement the DMO
    **                           use count to allow the persistence to evict it if not in use.
    **     ECF 20210504          Fixed incorrect block depth used to register callbacks with the transaction
    **                           manager. Refactored begin/end transaction methods.
    **     AIL 20210507          Allow lookupByName search deeper for a buffer by its legacy name.
    **     ECF 20210519          Removed DMO use counting. Reference count is now maintained by BaseRecord.
    **     ECF 20210606          Removed short circuit check in scopeDeleted, which could potentially cause a
    **                           memory leak by bypassing resource cleanup.
    **     IAS 20210613          Get rid of the permanent connections notion.
    **     ECF 20210806          Fixed nested query error detection/reporting.
    **     ECF 20210924          Removed tracking of records currently being deleted. This is managed within
    **                           RecordBuffer and BaseRecord now.
    **     OM  20211006          Unbinding occurs on procedure return instead of TM's scope finished.
    **     AL2 20211015          Use real referents instead of class when registering buffers with byLegacyName.
    **     CA  20211025          pendingStaticTempTables must be emptied on cleanupPending().
    **     ECF 20211122          Added a global parameter to openScope, to allow registration of persistent
    **                           procedure buffers at the global scope of the openBuffers scoped list.
    **     ECF 20211124          Rolled back previous fix. Ensure the correct transition of a persistent
    **                           procedure buffer from uninitializedBuffers to openBuffers when that buffer is
    **                           initialized.
    **     OM  20211201          Improved transaction support for finally blocks.
    **     CA  20211222          Fixed memory leak - cleanup empty map values from byLegacyName map.
    **     CA  20220104          Fixed memory leak - when deleting a buffer, remove it from any 'persistent
    **                           procedure scope', too.
    **     CA  20220105          Fixed byLegacyName lookup - the buffers must not be moved to global scope, but
    **                           instead the lookup must use both the stack buffers (for i.e. internal procedures)
    **                           and the buffers defined at the internal procedure's this-procedure.
    **     CA  20220206          Each buffer knows the scope(s) at which it was registered with the allBuffers,
    **                           openBuffers, loadedBuffers and byLegacyName registries, so that the processing
    **                           during delete (at deregisterDynamicBuffer) will target explicit scopes, and will
    **                           avoid iterating all scopes.
    **                           Added 'dirtyBuffers' scoped dictionary, to track any buffer which had a Create,
    **                           Update or Delete operation in the current scope. The scope is cleared only when
    **                           the transaction block is processed, otherwise buffers are merged to previous
    **                           scope, but only if their record is actual 'dirty'.
    **     CA  20220210          Removed Commitable.rollbackPending and its notifications, as there is no actual
    **                           implementation of this method.
    **     CA  20220308          Fixed scopeFinished(), the condition to not merge dirty buffers to global block
    **                           was incorrect.
    **     TW  20220617          Use new TailMap for dirtyBuffers. This addresses an issue where new dirty buffers
    **                           were added to the collection during iterating it, resulting in a
    **                           ConcurrentModificationException (refs #6356).
    **     CA  20220707          Cache the converted Java names for the dynamic buffers.
    **     CA  20220901          Refactored scope notification support: ScopeableFactory was removed, and the
    **                           registration is now specific to each type of scopeable.  For each case, the block
    **                           will be registered for scope support (for that particular scopeable) only when
    **                           the scopeable is 'active' (i.e. unnamed streams or accumulators are used).  This
    **                           allows a lazy registration of scopeables, to avoid the unnecessary overhead of
    **                           processing all the scopeables for each and every block.
    **     CA  20220906          Removed state which is no longer used (PersistentProcScope.loadedBuffers and
    **                           undoData) and removed methods which are not being invoked.
    **                           Moved the parameter assigner scopeable support to AbstractParameter (as all
    **                           parameters require scope support, not just fields).
    **                           Extracted all BufferManager state which requires notification for
    **                           'isImportantBlock' to an external TxWrapper class (including the private class
    **                           here), to allow the scopeable BufferManager state related only to buffers (and
    **                           not transaction related) to be registered for scope notifications in a lazy
    **                           manner, when buffers are accessed/used/opened.
    **     CA  20221006          An improvement in deregisterBindingBuffer, do not use entrySet() iterator if both
    **                           key and value are not required.  Refs #6821
    **     CA  20221006          The 'nearestExternal' support needs to be in BufferManager, as it is used for
    **                           computing a scope in ChangeBroker - BufferManager is registered for external
    **                           block scope.
    **     CA  20221010          Performance improvements - avoid the ProcedureData lookup, by keeping a parallel
    **                           stack of this data for THIS-PROCEDURE.  Refs #6826
    **     CA  20220526          When a persistent program gets deleted, remove all dirty buffers associated with
    **                           it, from all scopes.
    **     CA  20220601          Nested batch assign mode is allowed in internal mode.
    **     CA  20220613          registerBindingBuffer must use as procedure the SOURCE-PROCEDURE in case of
    **                           OUTPUT and THIS-PROCEDURE in case of INPUT, for BIND parameters.
    **     CA  20220918          Do not create a procedure scope and cache it, if there is no buffer state to
    **                           cache.
    **     CA  20220919          PersistentProcScope.openBuffers can be an identity map, to allow better control
    **                           over the remove.
    **     CA  20221006          The 'nearestExternal' support needs to be in BufferManager, as it is used for
    **                           computing a scope in ChangeBroker - BufferManager is registered for external
    **                           block scope.
    **     CA  20221010          Performance improvements - avoid the ProcedureData lookup, by keeping a parallel
    **                           stack of this data for THIS-PROCEDURE.  Refs #6826
    **     CA  20230104          Keep a reverse mapping of bound buffers, per each external program, to avoid
    **                           iterating the entire set, when a scope is started.
    **     CA  20230116          Avoid using handle.unwrap, handle.getReference or other BDT usage from within FWD
    **                           runtime.
    **     HC  20230118          Eliminated some of the uses of String.toUpperCase and/or
    **                           String.toLowerCase for performance.
    ** 111 GBB 20230512          Logging methods replaced by CentralLogger/ConversionStatus.
    ** 112 AL2 20230613          Allow retrieving the ProcedureHelper with this manager to avoid context look-ups.
    ** 113 DDF 20230608          Made the size of the convertedNames LRUCache configurable.
    **     DDF 20230613          Removed the static block that reads the configuration size of the cache.
    **                           CacheManager will handle the finding of the configuration size and the
    **                           cache creation.
    **     DDF 20230627          Made the cache final.
    **     DDF 20230706          Replaced createLRUCache call with another one that will use a null
    **                           discriminator by default.
    ** 114 CA  20230724          Further reduce context-local usage.
    ** 115 CA  20231026          A persistent procedure with a pending delete is no longer persistent, so
    **                           scopeDeleted needs to be executed.
    **     CA  20231031          Added 'BlockDefinition' parameter to 'scopeStart'.
    ** 116 ASL 20231107          Implemented iterative search for ScopedDictionary key index in case of
    **                           postponed table handles.
    **     ASL 20231117          Modified externalScope stack to nextExtScope stack to easily access the nearest
    **                           external scope available. Changed the ScopedDictionary index search to a simple
    **                           expression.
    ** 117 CA  20231216          Avoid processing in maybeCloseSessions if activePersistenceContexts is empty.
    **     CA  20231221          Avoid lowercasing the name in TempTableKey and use case-insensitive hashcode.
    ** 118 DDF 20240130          Created isActiveBuffer method.
    **     DDF 20240130          Inlined condition in isActiveBuffer().
    ** 119 CA  20240206          If THIS-PROCEDURE is already on the stack, do not re-bind the bound buffers.
    ** 120 CA  20240302          A BIND at the param or argument has effect only if either the argument has
    **                           BY-REFERENCE or the target is REFERENCE-ONLY. Removed the 'bindBuffer' support
    **                           which sets and unsets the DMO alias as this was determined to be obsolete and/or
    **                           incorrect.
    ** 121 CA  20240404          Static temp-table or dataset resolution by name must be done using the currently
    **                           executing type and the static temp-table/dataset's defining type.
    ** 122 AL2 20240617          Made bufferInitialized return a boolean.
    ** 123 SP  20240801          Extracted byLegacy lambda to deregisterByLegacy() method.
    **     CA  20240809          Removed capturing lambda from 'setUnknownMode'.
    ** 124 CA  20240827          Added 'getTxWrapperHelper'.
    ** 125 OM  20240909          Improved Database API.
    ** 126 CA  20230924          Further reduce context-local lookup.
    ** 127 AS  20241016          Refactored dirtyBatchBuffers to a Map<RecordBuffer, Runnable>
    **                           containing the buffers and the code to restore their active state.
    ** 128 EAB 20241008          Added tableRecordBuffer to track and group buffers that reference the same DMO for
    **                           better organization and lookup.
    **     AL2 20241023          Created registerToBuffersByClass to de-duplicate code.
    **     AL2 20241023          Extended uses of buffersByDMOBufClass to work with persistent procs and
    **                           dynamic buffers.
    ** 129 CA  20241119          A DMO alias must not emit as a FQL reserved keyword.
    ** 130 OM  20241128          Multi-tenant runtime support: selected the proper persistence context, eventually
    **                           based on [sharedDb] parameter.
    ** 131 ES  20250130          Change generateUniqueBufferName method to generateBufferName, without adding the suffix
    **                           to the end of the javaname.
    **     ES  20250205          Added generateUniqueToken method and changed the allBuffers key from
    **                           getDMOAlias() to getDMOToken().
    ** 132 DDF 20250507          Improve performance by avoiding expensive lookups.
    */

    /*
    ** This program is free software: you can redistribute it and/or modify
    ** it under the terms of the GNU Affero General Public License as
    ** published by the Free Software Foundation, either version 3 of the
    ** License, or (at your option) any later version.
    **
    ** This program is distributed in the hope that it will be useful,
    ** but WITHOUT ANY WARRANTY; without even the implied warranty of
    ** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    ** GNU Affero General Public License for more details.
    **
    ** You may find a copy of the GNU Affero GPL version 3 at the following
    ** location: https://www.gnu.org/licenses/agpl-3.0.en.html
    **
    ** Additional terms under GNU Affero GPL version 3 section 7:
    **
    **   Under Section 7 of the GNU Affero GPL version 3, the following additional
    **   terms apply to the works covered under the License.  These additional terms
    **   are non-permissive additional terms allowed under Section 7 of the GNU
    **   Affero GPL version 3 and may not be removed by you.
    **
    **   0. Attribution Requirement.
    **
    **     You must preserve all legal notices or author attributions in the covered
    **     work or Appropriate Legal Notices displayed by works containing the covered
    **     work.  You may not remove from the covered work any author or developer
    **     credit already included within the covered work.
    **
    **   1. No License To Use Trademarks.
    **
    **     This license does not grant any license or rights to use the trademarks
    **     Golden Code, FWD, any Golden Code or FWD logo, or any other trademarks
    **     of Golden Code Development Corporation. You are not authorized to use the
    **     name Golden Code, FWD, or the names of any author or contributor, for
    **     publicity purposes without written authorization.
    **
    **   2. No Misrepresentation of Affiliation.
    **
    **     You may not represent yourself as Golden Code Development Corporation or FWD.
    **
    **     You may not represent yourself for publicity purposes as associated with
    **     Golden Code Development Corporation, FWD, or any author or contributor to
    **     the covered work, without written authorization.
    **
    **   3. No Misrepresentation of Source or Origin.
    **
    **     You may not represent the covered work as solely your work.  All modified
    **     versions of the covered work must be marked in a reasonable way to make it
    **     clear that the modified work is not originating from Golden Code Development
    **     Corporation or FWD.  All modified versions must contain the notices of
    **     attribution required in this license.
    */

    package com.goldencode.p2j.persist;

    import java.lang.reflect.*;
    import java.util.*;
    import java.util.concurrent.*;
    import java.util.function.*;
    import java.util.logging.*;
    import static java.util.AbstractMap.SimpleEntry;

    import com.goldencode.cache.*;
    import com.goldencode.p2j.convert.*;
    import com.goldencode.p2j.oo.lang.*;
    import com.goldencode.p2j.persist.dirty.*;
    import com.goldencode.p2j.persist.id.*;
    import com.goldencode.p2j.security.*;
    import com.goldencode.p2j.util.*;
    import com.goldencode.p2j.util.logging.CentralLogger;
    import com.goldencode.p2j.util.ErrorManager;
    import com.goldencode.util.*;
    import com.goldencode.util.Stack;

    /**
     * Manages all record buffers within the current user context.  This includes
     * transaction processing, buffer and block scope management.  A separate
     * instance of this class is created within each user context.  It acts as a
     * broker between the context local <code>TransactionManager</code>, the
     * persistence service layer which manages database resources, and multiple
     * record buffers opened by client code.
     * <p>
     * A record buffer registers itself with the buffer manager when client code
     * opens a new buffer scope.  If a transaction is active, that buffer is
     * immediately registered with the transaction manager for commit processing.
     * This allows the buffer to synchronize its state with the persistence
     * service when a transaction or subtransaction commit occurs.  If a
     * transaction is not active when a new buffer scope is opened, the buffer
     * manager keeps track of the open buffer, and the block scope at which it
     * was opened.  When a full transaction subsequently is opened, all buffers
     * with open scopes are batch registered for commit processing.
     * <p>
     * Also, when a full transaction begins, the buffer manager takes stock of
     * all databases referred to by open buffer scopes, and begins a new,
     * database-level transaction for each one.  The database transaction's life
     * cycle is managed by an inner class, {@link TxWrapper}.
     * This inner class registers itself for master commit and finish processing
     * with the transaction manager, such that it can reflect an application
     * level commit/rollback back to the underlying database transaction.
     * <p>
     * Updates made to existing records are tracked for undo purposes by this
     * class.  A scoped dictionary of reversible updates by DMO type is managed
     * by the context local instance of this class, for all open buffers in the
     * current client context.
     */
    public final class BufferManager
    implements Scopeable,
               BatchListener,
               StopConditionVetoHandler
    {
       /** Logger */
       private static final CentralLogger LOG = CentralLogger.get(BufferManager.class.getName());

       /** Stores context local instances of this class */
       private static final ContextLocal<BufferManager> context = new ContextLocal<BufferManager>()
       {
          @Override public WeightFactor getWeight() { return WeightFactor.LEVEL_2; }
          @Override protected BufferManager initialValue() { return (new BufferManager()); }
       };

       /** A cache of buffer-related fields defined in a specific legacy-converted class. */
       private static final Map<Class<?>, Set<Field>> classBufferFields = new ConcurrentHashMap<>();

       /** Map of buffer types to high-water block depths of any change to any record of that type */
       // intentionally package private for direct use from RecordBuffer
       final Map<BufferType, MutableInteger> changeScopes = new HashMap<>();

       /** Finalizable which clears batch mode at the appropriate times */
       private final Finalizable batchCleaner = new Finalizable()
       {
          @Override public void finished() { RecordBuffer.cleanupBatchMode(BufferManager.this); }
          @Override public void deleted()  { }
          @Override public void iterate()  { RecordBuffer.cleanupBatchMode(BufferManager.this); }
          @Override public void retry()    { RecordBuffer.cleanupBatchMode(BufferManager.this); }
       };

       /** The buffers which will be registered on the next scope entry. */
       private final Set<RecordBuffer> pendingBuffers = Collections.newSetFromMap(new IdentityHashMap<>());

       /** The defining class of a pending buffer, used when initializing an OE class and/or external program. */
       private final Map<RecordBuffer, Class<?>> pendingBufferClasses = new IdentityHashMap<>();

       /**
        * Scoped dictionary in which all defined record buffers are stored. This data structure uses
        * the DMO alias as keys.
        * TODO: this can't be by String, otherwise we can't separate the buffers to its defining class
        */
       private final ScopedDictionary<String, RecordBuffer> allBuffers = new ScopedDictionary<>();

       /**
        * Scoped dictionary with key as a DMO buffer interface and multiplex, and the value is a list of
        * RecordBuffers associated with them.
        */
       private final ScopedDictionary<SimpleEntry<String, Integer>, Set<RecordBuffer>> buffersByDMOBufClass =
                new ScopedDictionary<>();

       /**
        * Scoped dictionary in which all defined record buffers are stored. This data structure uses
        * the <strong>lowercased legacy name</strong> as keys.
        * <p>
        * Note: only the static buffers are kept in this structure. The dynamic resources cannot be
        *       stored here as their NAME attribute is mutable and will possible collide with other
        *       existing buffers.
        */
       private final ScopedDictionary<Object, Map<String, RecordBuffer>> byLegacyName = new ScopedDictionary<>();

       /** Buffers whose scopes have been opened, but are not yet initialized */
       private final Map<RecordBuffer, Boolean> uninitializedOpenBuffers = new WeakHashMap<>();

       /** Scoped dictionary in which open record buffers are stored */
       private final ScopedList<RecordBuffer> openBuffers = new ScopedList<>();

       /**
        * Scoped dictionary which contains currently active static temp tables (i.e. table scope was
        * opened and not yet closed). Keyed by a {@code TempTableKey} compound from the legacy table
        * name and the procedure where the temp table was defined.
        */
       private final ScopedDictionary<StaticDefinitionKey, StaticTempTable> openStaticTempTables =
          new ScopedDictionary<>();

       /** The registry with all static {@code Datasets} mapped by their name. */
       private ScopedDictionary<StaticDefinitionKey, DataSet> dataSetRegistry = new ScopedDictionary<>();

       /** A map of static temp-tables pending registration. */
       private final Map<StaticTempTable, Boolean> pendingStaticTempTables = new HashMap<>();

       /**
        * Scoped dictionary which contains the maps of output copiers for a procedure/function to
        * their 0-based indexes in the set of all TABLE-HANDLE parameters for the procedure/function
        * (excluding other types of parameters).
        */
       private final ScopedDictionary<OutputTableHandleCopier, Integer> thOutputParameters =
          new ScopedDictionary<>();

       /** Active persistence contexts (those which have an open database session) */
       private final Set<Persistence.Context> activePersistenceContexts = Collections.newSetFromMap(new IdentityHashMap<>());

       /** Stack of batch mode data */
       private final ArrayDeque<BatchModeData> batchModeStack = new ArrayDeque<>();

       /** Map with buffer-related data surviving a persistent proc's external procedure.*/
       private final Map<Object, PersistentProcScope> persistProcScopes = new IdentityHashMap<>();

       /** Primary keys pending reclamation */
       private final Map<Persistence, Set<Long>> pendingReclaimedKeys = new IdentityHashMap<>();

       /** Helper to use the ProcedureManager without any context local lookups. */
       private final ProcedureManager.ProcedureHelper pm = ProcedureManager.getProcedureHelper();

       /** Transaction helper */
       private final TransactionManager.TransactionHelper txHelper;

       /** Tx wrapper helper. */
       private TxWrapper.TxWrapperHelper txWrapperHelper;

       /** Change broker instance. */
       private final ChangeBroker changeBroker;

       /** Buffer names already converted. */
       private final LRUCache<String, String> convertedNames;

       /** Track the external scopes pushed on {@link BufferManager}. */
       private final Stack<Integer> nextExtScope = new Stack<>();

       /** Connection manager */
       private ConnectionManager connMgr = null;

       /** Number of nested queries being processed */
       private int queryDepth = 0;

       /** Unknown mode is {@code true} while evaluating query parameters */
       private boolean unknownMode = false;

       /** Does a nested query error condition currently exist? */
       private boolean nestedQueryError = false;

       /** The cached name converter. */
       private NameConverter nc = new NameConverter();

       /** Counter for ensuring token uniqueness and avoiding expensive lookups. */
       private long tokenCounter = 1;

       /**
        * Constructor; instances of this class are context-local and must be accessed using the static
        * {@link #get} method.
        */
       private BufferManager()
       {
          byLegacyName.setIdentityKeys(true);
          thOutputParameters.setIdentityKeys(true);

          txHelper = TransactionManager.getTransactionHelper();

          txHelper.registerBatchListener(this);
          txHelper.registerStopVetoHandler(this);

          changeBroker = ChangeBroker.get();

          convertedNames = CacheManager.createLRUCache(BufferManager.class, 10000);

          if (LOG.isLoggable(Level.FINE))
          {
             LOG.log(Level.FINE, "BufferManager created in context " + Utils.describeContext());
          }
       }

       /**
        * Retrieve the context-local instance of this class, instantiating it first if necessary.
        *
        * @return  Context-local instance of this class.
        */
       public static BufferManager get()
       {
          BufferManager bm = context.get();
          if (bm.connMgr == null)
          {
             ConnectionManager cm = ConnectionManager.get();
             if (bm.connMgr == null)
             {
                bm.connMgr = cm;
             }
             else if (bm.connMgr != cm)
             {
                // we have a problem: two ConnectionManager-s for same context
                throw new RuntimeException("Multiple ConnectionManager in same context.");
             }
          }

          if (bm.txWrapperHelper == null)
          {
             bm.txWrapperHelper = TxWrapper.getHelper(bm);
          }

          return bm;
       }

       /**
        * Register the {@link BufferManager} instance as for scope notifications at the current block.
        *
        * @param    bufferManager
        *           The {@link BufferManager} instance.
        */
       public static void registerScopeable(BufferManager bufferManager)
       {
          if (bufferManager == null)
          {
             bufferManager = get();
          }

          boolean isInitializing = bufferManager.pm.getInstantiatingExternalProgram() != null;
          if (isInitializing)
          {
             // there are cases like dataset builder where this is called even if we still are in the implicit
             // constructor for creating a new class/program.
             registerPendingScopeable(bufferManager);
             return;
          }

          // both ChangeBroker and BufferManager need to be added
          bufferManager.txHelper.registerBlockScopeable(bufferManager);
          bufferManager.txHelper.registerBlockScopeable(bufferManager.changeBroker);
       }

       /**
        * Register the {@link BufferManager} instance as pending scopeable, to be added to the next block.
        *
        * @param    bufferManager
        *           The {@link BufferManager} instance.
        */
       public static void registerPendingScopeable(BufferManager bufferManager)
       {
          if (bufferManager == null)
          {
             bufferManager = get();
          }

          // both ChangeBroker and BufferManager need to be added
          bufferManager.txHelper.registerPendingScopeable(bufferManager);
          bufferManager.txHelper.registerPendingScopeable(bufferManager.changeBroker);
       }

       /**
        * Register the {@link TxWrapper} scopeable as pending for the next opened block.
        *
        * @param    bufferManager
        *           The {@link BufferManager} instance.
        */
       public static void registerTxScopeable(BufferManager bufferManager)
       {
          if (bufferManager == null)
          {
             bufferManager = get();
          }

          bufferManager.txWrapperHelper.registerPendingScopeable();
       }

       /**
        * Cleanup of pending resources. This is called when the external procedure constructor failed
        * to build a new object because of an exception thrown in a member initialization (shared
        * frame/table).
        */
       public static void cleanupPending()
       {
          BufferManager bm = BufferManager.get();
          bm.pendingBuffers.clear();
          bm.pendingBufferClasses.clear();
          bm.pendingStaticTempTables.clear();
       }

       /**
        * Register this buffer as 'dirty' in the current scope.
        * <p>
        * This registration doesn't mean that at the time of {@link TxWrapper#commit}, {@link TxWrapper#rollback},
        * or {@link TxWrapper#validate} the buffer's record is still 'dirty' - this is used to limit the set of
        * buffers receiving such notifications, to avoid iterating over all {@link #openBuffers}.
        *
        * @param    recordBuffer
        *           The record buffer.
        */
       public void registerDirtyBuffer(RecordBuffer recordBuffer)
       {
          // although no state in BufferManager is touched when registering a dirty buffer, we need to register
          // it with the current block, as there is logic dependent on the 'depth' of the application stack which
          // uses buffers
          registerScopeable(this);

          txWrapperHelper.registerDirtyBuffer(recordBuffer);
       }

       /**
        * Register a pending buffer's defining class.
        *
        * @param    buffer
        *           The newly constructed buffer instance.
        * @param    cls
        *           The Java class associated with an external program or legacy OE class, where it
        *           was defined.
        */
       public void registerPendingBufferClass(RecordBuffer buffer, Class<?> cls)
       {
          pendingBufferClasses.put(buffer, cls);
       }

       /**
        * Get the defining class for a pending buffer.
        *
        * @param    buffer
        *           The newly constructed buffer instance.
        *
        * @return   The Java class associated with an external program or legacy OE class, where it
        *           was defined.
        */
       public Class<?> getPendingBufferClass(RecordBuffer buffer)
       {
          return pendingBufferClasses.get(buffer);
       }

       /**
        * Resolve the defining Java class for all Java fields which define a buffer.
        *
        * @param   referent
        *          The instance associated with an external program or legacy OE class.
        */
       public void resolvePendingBufferClasses(Object referent)
       {
          // TODO: this may be deprecated as we need this right after buffer instantiation, and not
          // after class initialization

          // process any buffer fields and save their classes
          Class<?> cls = referent.getClass();
          while (cls != BaseObject.class)
          {
             Set<Field> fields = classBufferFields.get(cls);
             if (fields == null)
             {
                fields = new HashSet<>();

                Field[] allFields = cls.getDeclaredFields();
                for (Field f : allFields)
                {
                   if (Modifier.isStatic(f.getModifiers()))
                   {
                      continue;
                   }

                   if (Buffer.class.isAssignableFrom(f.getType()))
                   {
                      fields.add(f);
                   }
                }

                classBufferFields.put(cls, fields);
             }

             for (Field f : fields)
             {
                try
                {
                   f.setAccessible(true);
                   BufferReference b = (BufferReference) f.get(referent);
                   registerPendingBufferClass(b.buffer(), cls);
                }
                catch (IllegalArgumentException | IllegalAccessException e)
                {
                   throw new RuntimeException(e);
                }
                finally
                {
                   f.setAccessible(false);
                }
             }

             cls = cls.getSuperclass();
          }
       }

       /**
        * Query the scope level which currently is open. This is the number of &quot;important&quot; scopes;
        * NO_TRANSACTION blocks inside an application transaction are ignored. So, this number may be lower
        * than the actual number of nested block scopes open, as tracked by the {@code TransactionManager}.
        *
        * @return  Number of &quot;important&quot; buffer scopes open, by the above definition.
        */
       public int getOpenBufferScopes()
       {
          return openBuffers.scopes();
       }

       /**
        * Find the nearest external block (from the bottom of the stack).
        *
        * @return   See above.
        */
       public int findNearestExternal()
       {
          return nextExtScope.empty() ? -1 : nextExtScope.peek();
       }

       /**
        * Check the specified exception and determine whether we want to allow it
        * to be honored by the {@link TransactionManager}, or whether we want to
        * veto the STOP at this scope.
        * <p>
        * This implementation is interested only in STOP conditions which have
        * been generated by database connection errors.  STOP processing at the
        * current scope is vetoed iff the exception is an instance of {@link
        * DatabaseConnectionException} AND either of the following criteria are
        * met:
        * <ul>
        *   <li>the failing database is one that should be permanently connected
        *       to this P2J server instance;  OR
        *   <li>there is any record buffer still in scope, which is associated
        *       with the failing database.
        * </ul>
        * <p>
        * Otherwise, normal STOP processing is allowed to continue.
        *
        * @param   exc
        *          The exception to be checked.
        *
        * @return  <code>true</code> if STOP should be vetoed;
        *          <code>false</code> if the STOP should be allowed.
        */
       public boolean vetoStop(StopConditionException exc)
       {
          // If the exception was not caused by a database connection problem, it
          // it is not in our domain to veto;  allow STOP processing to continue.
          if (!(exc instanceof DatabaseConnectionException))
          {
             return false;
          }

          DatabaseConnectionException dce = (DatabaseConnectionException) exc;
          Database database = dce.getDatabase();

          // on database connection errors, the exception should be re-thrown if this is a virtual database
          // and this block or a parent block uses a table from this database
          for (RecordBuffer buffer : allBuffers.values())
          {
             // a buffer is in use, the STOP must be vetoed
             if (buffer.isActive() && database.equals(buffer.getDatabase()))
             {
                return true;
             }
          }

          return false;
       }

       /**
        * Executes logic to explicitly begin a transaction.
        *
        * @param    inTx
        *           Flag indicating if we are in a transaction.
        * @param    fullTx
        *           Flag indicating if we are in a full transaction.
        * @param    blockDepth
        *           The current block depth (all blocks tracked by the {@code TransactionManager}).
        */
       public void beginTx(boolean inTx, boolean fullTx, int blockDepth)
       {
          txWrapperHelper.beginTx(inTx, fullTx, blockDepth);
       }

       /**
        * Executes logic to explicitly end a transaction.
        *
        * @param    inTx
        *           Flag indicating if we are in a transaction.
        * @param    fullTx
        *           Flag indicating if we are in a full transaction.
        */
       public void endTx(boolean inTx, boolean fullTx)
       {
          txWrapperHelper.endTx(inTx, fullTx);
       }

       /**
        * Executes logic required after a full transaction has ended.
        */
       public void endTxPost()
       {
          // if any DBs were used in this tx block and remained connected, disconnect now
          txWrapperHelper.endTxPost();
       }

       /**
        * Process a notification that a new block scope has been entered.  Notify
        * every buffer with an open scope that a new block scope has started.  If
        * the new block scope is within the context of a transaction, each buffer
        * with an open scope is registered for commit processing with the
        * transaction manager.  If the block scope represents the opening of a
        * new, full transaction, we also initialize database transactions for
        * each database referenced by a buffer with an open scope.
        *
        * @throws  ErrorConditionException
        *          if there is an error flushing pending changes to the database
        *          as we open a new database transaction.
        *
        * @param    block
        *           The explicit block definition which required this notification.
        */
       @Override
       public void scopeStart(BlockDefinition block)
       {
          if (txHelper.isExternalBlock())
          {
             nextExtScope.push(nextExtScope.size());
          }
          else
          {
             nextExtScope.push(nextExtScope.empty() ? -1 : nextExtScope.peek());
          }

          allBuffers.addScope(null);
          byLegacyName.addScope(null);
          buffersByDMOBufClass.addScope(null);
          openBuffers.pushScope();
          openStaticTempTables.addScope(null);
          dataSetRegistry.addScope(null);
          thOutputParameters.addScope(null);

          BatchModeData crtBatchScope = batchModeStack.peek();
          if (crtBatchScope != null && crtBatchScope.batchDepth == BatchModeData.INTERNAL)
          {
             // normally a new scope cannot be opened inside an INTERNAL batch mode. Except for the OO property
             // setters that can be invoked so that the output value to be set
             if (LOG.isLoggable(Level.FINE))
             {
                LOG.log(Level.INFO, block.type + " pushed on batch assign on top of an INTERNAL scope.");
             }
          }
          batchModeStack.push(new BatchModeData());

          if (!pendingStaticTempTables.isEmpty())
          {
             // always before pendingBuffers
             for (Map.Entry<StaticTempTable, Boolean> entry : pendingStaticTempTables.entrySet())
             {
                registerStaticTempTable(entry.getKey(), entry.getValue());
             }
             pendingStaticTempTables.clear();
          }

          if (!pendingBuffers.isEmpty())
          {
             for (RecordBuffer buffer : pendingBuffers)
             {
                Class<?> def = pendingBufferClasses.get(buffer);
                if (def == null)
                {
                   def = pm._thisProcedure().getClass();
                }

                registerBuffer(buffer);
             }
             pendingBuffers.clear();
             pendingBufferClasses.clear();
          }

          // Ensure that batch assign mode is turned off properly, regardless of
          // whether business logic invokes RecordBuffer.endBatch().  This
          // prevents a nesting error raised during RecordBuffer.startBatch(), if
          // a batch is interrupted and then a new one begins.
          txHelper.registerFinalizable(batchCleaner, false);

          int blockDepth = txHelper.getNestingLevel();

          if (LOG.isLoggable(Level.FINER))
          {
             LOG.log(Level.FINER,
                     Utils.describeContext() + ":  block scope " + blockDepth + " started");
          }
       }

       /**
        * Process a notification that a scope is about to be exited.  Notify each
        * record buffer with an open scope that the current block scope is ending.
        */
       @Override
       public void scopeFinished()
       {
          PersistentProcScope ppScope = null;
          Object referent = null;

          if (!txHelper.isGlobalBlock())
          {
             BlockType bt = txHelper.getBlockType();
             // if we have a persistent procedure, save the data
             if (bt == BlockType.EXTERNAL_PROC && pm.isThisProcedurePersistent())
             {
                referent = pm._thisProcedure();

                // in case of legacy OE classes, and when this is executed from a super-class 'execute'
                // surrogate, this will already exist
                ppScope = persistProcScopes.get(referent);
                if (ppScope == null)
                {
                   ppScope = new PersistentProcScope();
                   persistProcScopes.put(referent, ppScope);
                }
             }
          }

          if (ppScope != null)
          {
             // save the data for this external procedure, as these will need to be removed from the
             // global scope, when the procedure gets deleted.

             List<RecordBuffer> thisOpenBuffers = openBuffers.sublistAtScope(0);
             ppScope.openBuffers.addAll(thisOpenBuffers);

             Map<String, RecordBuffer> thisAllBuffers = allBuffers.getDictionaryAtScope(0, false);
             ppScope.allBuffers.putAll(thisAllBuffers);

             Map<SimpleEntry<String, Integer>, Set<RecordBuffer>> thisBuffersByDmoClass =
                      buffersByDMOBufClass.getDictionaryAtScope(0, false);
             ppScope.buffersByDMOBufClass.putAll(thisBuffersByDmoClass);

             checkLegacyNames(ppScope.byLegacyName, byLegacyName.getDictionaryAtScope(0, false));

             Map<StaticDefinitionKey, StaticTempTable> thisOpenStaticTempTables =
                openStaticTempTables.getDictionaryAtScope(0, false);
             ppScope.openStaticTempTables.putAll(thisOpenStaticTempTables);

             Map<StaticDefinitionKey, DataSet> thisDataSets = dataSetRegistry.getDictionaryAtScope(0, false);
             ppScope.datasets.putAll(thisDataSets);

             // add all these to the global scope, too

             if (!thisOpenBuffers.isEmpty())
             {
                openBuffers.addAll(true, thisOpenBuffers);
                for (RecordBuffer buffer : thisOpenBuffers)
                {
                   buffer.addOpenBuffersScope(1); // global scope
                }
             }

             if (!thisAllBuffers.isEmpty())
             {
                allBuffers.getDictionaryAtScope(allBuffers.size() - 1, true)
                          .putAll(thisAllBuffers);
             }

             if (!thisBuffersByDmoClass.isEmpty())
             {
                buffersByDMOBufClass.getDictionaryAtScope(buffersByDMOBufClass.size() - 1, true)
                                    .putAll(thisBuffersByDmoClass);
             }

             if (!thisOpenStaticTempTables.isEmpty())
             {
                openStaticTempTables.getDictionaryAtScope(openStaticTempTables.size() - 1, true)
                                    .putAll(thisOpenStaticTempTables);
             }

             // datasets are not pushed to global scope

             if (ppScope.allBuffers.isEmpty()           &&
                 ppScope.byLegacyName.isEmpty()         &&
                 ppScope.buffersByDMOBufClass.isEmpty() &&
                 ppScope.openBuffers.isEmpty()          &&
                 ppScope.openStaticTempTables.isEmpty())
             {
                // if there is no data required for this referent, do not save it.
                persistProcScopes.remove(referent);
             }
          }
          else
          {
             List<RecordBuffer> thisOpenBuffers = openBuffers.sublistAtScope(0);
             for (RecordBuffer buffer : thisOpenBuffers)
             {
                trackDatabase(buffer, false);
             }
          }

          nextExtScope.pop();
          openBuffers.popScope();
          allBuffers.deleteScope();
          byLegacyName.deleteScope();
          buffersByDMOBufClass.deleteScope();
          openStaticTempTables.deleteScope();
          dataSetRegistry.deleteScope();
          thOutputParameters.deleteScope();
          BatchModeData bmd = batchModeStack.pop();
          if (bmd.batchDepth == BatchModeData.INTERNAL)
          {
             throw new IllegalStateException("Cannot end a scope while an INTERNAL batch is active.");
          }

          if (LOG.isLoggable(Level.FINER))
          {
             LOG.log(Level.FINER,
                     Utils.describeContext() + ":  block scope " + txHelper.getNestingLevel() + " finished");
          }

          maybeCloseSessions();
       }

       /**
        * Provides notification that the external procedure scope has been deleted.
        * <p>
        * If we are deleting a persistent procedure, clean all its data from the global scope for the
        * following dictionaries: {@link #openBuffers}, {@link #allBuffers}, {@link #openStaticTempTables}.
        */
       @Override
       public void scopeDeleted()
       {
          Object referent = pm.getProcessedProcedure();
          if (referent == null || pm._isPersistentAndNotPendingDelete(referent))
          {
             // if no referent or procedure is still persistent, do not clean up yet
             return;
          }

          // TODO: is this in the right place, after the short-circuits above?
          maybeCloseSessions();

          PersistentProcScope scope = persistProcScopes.remove(referent);

          // this needs to be executed only when the procedure gets deleted.
          if (scope != null)
          {
             // remove all these from the global scope

             if (!scope.openBuffers.isEmpty())
             {
                openBuffers.removeAllFrom(true, new ArrayList<>(scope.openBuffers));

                for (RecordBuffer buffer : scope.openBuffers)
                {
                   trackDatabase(buffer, false);
                }
             }

             if (!scope.allBuffers.isEmpty())
             {
                txWrapperHelper.cleanDirtyBuffers(scope.allBuffers);

                allBuffers.getDictionaryAtScope(allBuffers.size() - 1, true)
                          .keySet()
                          .removeAll(scope.allBuffers.keySet());
             }

             if (!scope.buffersByDMOBufClass.isEmpty())
             {
                buffersByDMOBufClass.getDictionaryAtScope(buffersByDMOBufClass.size() - 1, true)
                                    .keySet()
                                    .removeAll(scope.buffersByDMOBufClass.keySet());
             }

             if (!scope.openStaticTempTables.isEmpty())
             {
                openStaticTempTables.getDictionaryAtScope(openStaticTempTables.size() - 1, true)
                                    .keySet()
                                    .removeAll(scope.openStaticTempTables.keySet());
             }

             // datasets are not pushed to global scope
          }
       }

       /**
        * Get the {@link ScopeId} for the instance.
        *
        * @return   {@link ScopeId#BUFFER_MANAGER}.
        */
       @Override
       public ScopeId getScopeId()
       {
          return ScopeId.BUFFER_MANAGER;
       }

       /**
        * Provides a notification that a batch is starting or ending.  Enters or
        * exits batch assignment mode for all record buffers in this context,
        * depending on the value of <code>start</code>.
        *
        * @param   start
        *          <code>true</code> if this is the start of a batch, otherwise
        *          <code>false</code>.
        *
        * @see     RecordBuffer#startBatch
        * @see     RecordBuffer#endBatch
        */
       public void batchNotify(boolean start)
       {
          if (start)
          {
             RecordBuffer.startBatch(this, true);
          }
          else
          {
             RecordBuffer.endBatch(this, false);
          }
       }

       /**
        * Marks the table for postponed delete, if it is possible. Postponed delete can take place if
        * the table is used as [INPUT-]OUTPUT TABLE-HANDLE parameter.
        *
        * @param  tempTable
        *         Temporary table to be marked for postponed delete.
        *
        * @return <code>true</code> if the table was postponed for deletion. <code>false</code> if
        *         this can't be done.
        */
       public boolean postponeTableHandleDelete(TempTable tempTable)
       {
          int nearestExt = findNearestExternal();
          if (nearestExt != -1)
          {
             int size = thOutputParameters.size();
             Map<OutputTableHandleCopier, Integer> map =
                   thOutputParameters.getDictionaryAtScope(size - nearestExt - 1, false);
             if (map != null)
             {
                Set<OutputTableHandleCopier> set = map.keySet();
                for (OutputTableHandleCopier copier : set)
                {
                   handle outputHandle = copier.getCalled();
                   if (outputHandle._isValid() && tempTable.equals(outputHandle.getResource()))
                   {
                      copier.setTableToDelete(tempTable);
                      return true;
                   }
                }

                // It's not a TABLE-HANDLE parameter, just a dynamic table. Clear all pending
                // TABLE-HANDLE parameter deletes which current handle value is not an unknown handle.
                for (OutputTableHandleCopier copier : set)
                {
                   if (!copier.getCalled()._isValid())
                   {
                      copier.setTableToDelete(null);
                   }
                }
             }
          }

          return false;
       }

       /**
        * Check whether the current block is within an application-level transaction.
        *
        * @return  {@code true} if we are in a transaction, else {@code false}.
        */
       public boolean isTransaction()
       {
          return txWrapperHelper.isTransaction();
       }

       /**
        * Get the number of block scope transitions made. This is meant to be a semaphore of sorts, to allow
        * a caller to determine whether this value has changed since some previous point when it was last
        * retrieved. Since it is only meant to detect that the current block has changed over time, we do not
        * protect against overflow.
        *
        * @return  The current unique block ID.
        */
       public long getScopeTransitions()
       {
          return txWrapperHelper.getScopeTransitions();
       }

       /**
        * Convenient method to retrieve the procedure helper. There are many cases where the buffer manager is
        * used and the procedure manager is required. Instead of using this as an intermediate and crowd it with
        * gateway methods towards procedure manager, let the user operate directly on the procedure helper.
        *
        * @return   The procedure helper used by this buffer manager.
        */
       public ProcedureManager.ProcedureHelper getProcedureHelper()
       {
          return pm;
       }

       /**
       * Search in all scopes the first record buffer that holds a specific DMO based on a provided PK.
       *
       * @param  entityName
       *         The name of the entity for which the record buffers are being searched.
       * @param  multiplex
        *        Optional multiplex ID (temp-tables only; otherwise {@code null}.
       * @param  pK
       *         The primary key used to identify a specific DMO.
       *
       * @return The first record buffer that matches the search criteria. If none is found,
       *         this returns {@code null}.
       */
       public RecordBuffer getRecordBuffersByDMOBufClass(String entityName, Integer multiplex, Long pK)
       {
          SimpleEntry<String, Integer> keyByDMOBuf = new SimpleEntry<>(entityName, multiplex);
          for (int i = 0; i < buffersByDMOBufClass.size(); i++)
          {
             Set<RecordBuffer> searchedRecordBuffers = buffersByDMOBufClass.getValueAtScope(keyByDMOBuf, i);
             if (searchedRecordBuffers == null)
             {
                continue;
             }

             RecordBuffer buffer = searchedRecordBuffers.stream()
                      .filter(r -> r.getCurrentRecord() != null && r.getCurrentRecord().primaryKey() == pK)
                      .findFirst()
                      .orElse(null);
             if (buffer != null)
             {
                return buffer;
             }
          }

          return null;
       }

       /**
        * Get the open buffers from the current scope.
        *
        * @return   See above.
        */
       Collection<RecordBuffer> getOpenBuffers()
       {
          return openBuffers.scopes() == 0 ? Collections.EMPTY_LIST : openBuffers.sublistAtScope(0);
       }

       /**
        * Get the current counter value of application-leve transactions which have been opened.
        *
        * @return  Transaction count.
        */
       long getTransactionCount()
       {
          return txWrapperHelper.getTransactionCount();
       }

       /**
        * Get the {@link TxWrapper transaction wrapper} used by this context.
        *
        * @return   See above.
        */
       TxWrapper.TxWrapperHelper getTxWrapperHelper()
       {
          return txWrapperHelper;
       }

       /**
        * Get the transaction helper object used by this context.
        *
        * @return  Transaction helper.
        */
       TransactionManager.TransactionHelper getTxHelper()
       {
          return txHelper;
       }

       /**
        * Get the {@link #connMgr} instance for this context.
        *
        * @return   See above.
        */
       public ConnectionManager getConnMgr()
       {
          return connMgr;
       }

       /**
        * Register an active persistence context, indicating a database session has been opened.
        *
        * @param   pc
        *          Persistence context.
        */
       void registerPersistenceContext(Persistence.Context pc)
       {
          activePersistenceContexts.add(pc);
       }

       /**
        * Register this static dataset definition.
        *
        *  @param    type
        *            The definition type.
        *  @param    name
        *            The legacy name.
        *  @param    ref
        *            The instance where it is defined.
        *  @param    dataset
        *            The dataset instance to register.
        */
       void registerDataSet(Class<?> type, String name, Object ref, DataSet dataset)
       {
          if (type == null)
          {
             LOG.log(Level.WARNING,
                     "No type specified when registering dataset " + name + " defined in ref " + ref.getClass());
             type = pm.getExecuting();
          }

          StaticDefinitionKey key = new StaticDefinitionKey(name, ref, type);

          dataSetRegistry.addEntry(false, key, dataset);
       }

       /**
        * Register a newly defined record buffer in the current scope.  Buffers are deregistered
        * implicitly when the scope in which they were registered ends.
        *
        * @param   buffer
        *          Buffer to be registered.
        */
       void registerBuffer(RecordBuffer buffer)
       {
          Object ref = pm._thisProcedure();

          registerToBuffersByClass(null, buffer);

          allBuffers.addEntry(false, buffer.getDMOToken(), buffer);
          buffer.addAllBufferScope(allBuffers.size());
          buffer.addAllBufferScope(byLegacyName.size());

          Map<Object, Map<String, RecordBuffer>> dict = byLegacyName.getDictionaryAtScope(0, true);
          Map<String, RecordBuffer> defBufs = dict.get(ref);
          if (defBufs == null)
          {
             dict.put(ref, defBufs = new CaseInsensitiveHashMap<>());
          }
          defBufs.put(buffer.getLegacyName(), buffer);
       }

       /**
        * Register as pending the newly defined record buffer. When the next scope
        * is entered, all the <code>pendingBuffers</code> will be registered with
        * that scope. Buffers are deregistered implicitly when the scope in which
        * they were registered ends.
        *
        * @param   def
        *          The Java class defining this buffer.
        * @param   buffer
        *          Buffer pending to be registered.
        */
       void registerPending(Class<?> def, RecordBuffer buffer)
       {
          registerPendingScopeable(this);

          pendingBuffers.add(buffer);

          registerPendingBufferClass(buffer, def);
       }

       /**
        * Deregister the given, dynamic record buffer by removing it from the buffer manager's
        * tracking.
        *
        * @param   buffer
        *          Dynamic record buffer to be removed.
        * @param   referent
        *          The persistent procedure where the buffer was defined; may be <code>null</code>.
        */
       void deregisterDynamicBuffer(RecordBuffer buffer, Object referent)
       {
          String key = buffer.getDMOToken();
          String lkey = buffer.getLegacyName();
          int openScopes = getOpenBufferScopes();

          // these scopes contain the buffer's known scopes where it was registered in the associated dictionaries
          // in some cases, this scope may no longer be on the stack or, as the blocks are exited and invoked
          // again, the scope on the stack may not be the same as the one on the stack at the time the buffer was
          // registered.  regardless, the goal of these scopes kept at the buffer is to avoid iterating the entire
          // associated dictionaries, and do a 'targeted approach': the buffer knows that it was registered with
          // that scope, so it may still be on that scope and thus it needs to be removed.

          // also, moving the buffer to the global scope is tracked at this time only for openBuffers, so in the
          // other cases it is removed from the global scope, too - as these are dictionaries with identity maps,
          // there is no sequential iteration as with openBuffers (which is a ScopedList).

          int[] allBufferScopes = buffer.getAllBuferScopes();
          int[] byLegacyNameScopes = buffer.getByLegacyNameScopes();
          int[] buffersByDmoClassScopes = buffer.getBuffersByDmoClassScopes();
          int[] openBuffersScopes = buffer.getOpenBuffersScope();

          int scopeSize = allBuffers.size();
          for (int scope : allBufferScopes)
          {
             int depth = scopeSize - scope;
             if (depth >= 0)
             {
                allBuffers.removeEntryAtScope(key, depth);
             }
          }
          // remove from global scope always
          allBuffers.removeEntryAtScope(key, -1);

          scopeSize = byLegacyName.size();
          for (int scope : byLegacyNameScopes)
          {
             int depth = scopeSize - scope;
             if (depth >= 0)
             {
                Map<Object, Map<String, RecordBuffer>> refs = byLegacyName.getDictionaryAtScope(depth, false);
                if (refs != null)
                {
                   deregisterByLegacy(lkey, refs);
                }
             }
          }

          // remove from global scope always
          Map<Object, Map<String, RecordBuffer>> refs = byLegacyName.getDictionaryAtScope(-1, false);
          if (refs != null)
          {
             deregisterByLegacy(lkey, refs);
          }

          // remove from buffers by DMO buf class
          SimpleEntry<String, Integer> entry =
                   new SimpleEntry<>(buffer.getDMOBufInterface().getName(), buffer.getMultiplexID());

          scopeSize = buffersByDMOBufClass.size();
          for (int scope : buffersByDmoClassScopes)
          {
             int depth = scopeSize - scope;
             if (depth >= 0)
             {
                Map<SimpleEntry<String, Integer>, Set<RecordBuffer>> refs1 =
                         buffersByDMOBufClass.getDictionaryAtScope(depth, false);
                if (refs1 != null)
                {
                   Set<RecordBuffer> buffers = refs1.get(entry);
                   if (buffers != null)
                   {
                      buffers.remove(buffer);
                      if (buffers.isEmpty())
                      {
                         refs1.remove(entry);
                      }
                   }
                }
             }
          }

          // deregister from global
          Map<SimpleEntry<String, Integer>, Set<RecordBuffer>> refs1 = buffersByDMOBufClass.getDictionaryAtScope(-1, false);
          if (refs1 != null)
          {
             Set<RecordBuffer> buffers = refs1.get(entry);
             if (buffers != null)
             {
                buffers.remove(buffer);
                if (buffers.isEmpty())
                {
                   refs1.remove(entry);
                }
             }
          }

          for (int scope : openBuffersScopes)
          {
             int depth = openScopes - scope;
             if (depth >= 0 && depth < openScopes)
             {
                openBuffers.removeFrom(depth, buffer);
             }
          }
          // remove from global scope always
          // openBuffers.removeFrom(openScopes - 1, buffer);

          txWrapperHelper.deregisterDynamicBuffer(buffer);

          if (referent != null)
          {
             PersistentProcScope procScope = persistProcScopes.get(referent);
             if (procScope != null)
             {
                procScope.allBuffers.remove(key);
                deregisterByLegacy(lkey, procScope.byLegacyName);
                procScope.buffersByDMOBufClass.remove(entry);
                procScope.openBuffers.remove(buffer);
             }
          }

          trackDatabase(buffer, false);
       }

       /**
        * Deregister the buffer from all the defined record buffers stored by the lowercased legacy name.
        *
        * @param   lkey
        *          Lowercased legacy name key
        * @param   refs
        *          Scoped dictionary that uses the lowercased legacy name as keys in which
        *          all defined record buffers are stored.
        */
       private void deregisterByLegacy(String lkey, Map<Object, Map<String, RecordBuffer>> refs)
       {
          Iterator<Map<String, RecordBuffer>> iter = refs.values().iterator();
          while (iter.hasNext())
          {
             Map<String, RecordBuffer> bufDefs = iter.next();
             if (bufDefs.remove(lkey) != null && bufDefs.isEmpty())
             {
                iter.remove();
             }
          }
       }

       /**
        * Begin batch assignment mode.  While in this mode, validation of record buffers is deferred. We remember
        * the scope at which this occurs in order to test for errors now and allow proper cleanup later.
        *
        * @param   internal
        *          Use {@code false} for a programmer required batch assign mode. These can be started at most one
        *          at each scope depth.<br>
        *          Use {@code true} for a FWD internal maintenance batch block when some data must be processed
        *          "in batch". In this case the depth, is not checked. Only one level of this can be added as the
        *          top-level. This mode must be be ended before starting any new batch.
        *
        * @see     RecordBuffer#startBatch
        * @see     RecordBuffer#endBatch
        * @see     RecordBuffer#cleanupBatchMode
        *
        * @throws  IllegalStateException
        *          if batch mode is already active in the current block or was started but never
        *          ended in a previous block.
        */
       void startBatchAssignMode(boolean internal)
       {
          registerScopeable(this);

          int currentScope = getOpenBufferScopes();
          BatchModeData bmd = batchModeStack.peek();

          if (!internal)
          {
             if (bmd == null)
             {
                throw new IllegalStateException("Internal error: no scopes found. Call scopeStart() first.");
             }

             if (bmd.batchDepth == currentScope)
             {
                throw new IllegalStateException("Cannot nest batch assign modes within the same scope");
             }

             if (bmd.batchDepth != BatchModeData.NOT_ACTIVE)
             {
                // this should only happen with hand-written or modified code; converted code generally
                // will have balanced pairs of start and end batch mode calls
                throw new IllegalStateException(
                   "Batch assign mode was opened but not closed in the same scope (" +
                   bmd.batchDepth + "; current scope is " + currentScope + ")");
             }

             bmd.activate(currentScope);
          }
          else
          {
             bmd = new BatchModeData();
             bmd.activate(BatchModeData.INTERNAL);
             batchModeStack.push(bmd);
          }
       }

       /**
        * End batch assignment mode if we are currently in batch assignment mode
        * and we are in the same scope at which batch assignment mode was
        * started.  Otherwise do nothing.
        *
        * @return  If the above conditions are met for batch mode to end, the
        *          map of record buffers which were modified during batch mode is
        *          returned, with the corresponding code to restore the active state.
        *          Otherwise, return <code>null</code>, indicating
        *          either we are not in batch mode, or we are not in the proper
        *          scope to end it, or the buffers already have been processed, or
        *          the program is ending and we are in cleanup mode.
        *
        * @see     RecordBuffer#startBatch
        * @see     RecordBuffer#endBatch
        * @see     RecordBuffer#cleanupBatchMode
        */
       Map<RecordBuffer, Runnable> endBatchAssignMode()
       {
          BatchModeData bmd = batchModeStack.peek();
          Map<RecordBuffer, Runnable> ret = null;

          if (bmd == null)
          {
             // exiting program
             return null;
          }

          if (bmd.batchDepth == BatchModeData.INTERNAL)
          {
             // the internal scope is removed completely once ended to allow the parent scope to be automatically
             // reactivated or to nest other new scopes to be pushed
             batchModeStack.pop();
             ret = bmd.dirtyBatchBuffers;
          }
          else
          {
             if (bmd.batchDepth == BatchModeData.NOT_ACTIVE || getOpenBufferScopes() != bmd.batchDepth)
             {
                return null;
             }

             ret = bmd.dirtyBatchBuffers;
             bmd.dirtyBatchBuffers = null; // free memory
             bmd.batchDepth = -1; // deactivate it
          }

          return ret;
       }

       /**
        * Indicate whether we currently are in batch assignment mode.
        *
        * @return  <code>true</code> if in batch assignment mode, else
        *          <code>false</code>.
        *
        * @see     #startBatchAssignMode
        * @see     #endBatchAssignMode
        */
       boolean isBatchAssignMode()
       {
          return batchModeStack.peek().batchDepth != BatchModeData.NOT_ACTIVE; // this includes INTERNAL mode, too
       }

       /**
        * If we currently are in batch assignment mode, add the given record
        * buffer to the set of buffers requiring validation at the end of the
        * batch. Otherwise, do nothing.
        *
        * @param   buffer
        *          Buffer to be added to the set of dirty batch mode buffers.
        *
        * @param   restoreActiveState
        *          The code that restores the active state of the buffer.
        *
        * @see     #startBatchAssignMode
        * @see     #endBatchAssignMode
        */
       void addDirtyBatchBuffer(RecordBuffer buffer, Runnable restoreActiveState)
       {
          BatchModeData bmd = batchModeStack.peek();
          if (bmd.batchDepth != BatchModeData.NOT_ACTIVE) // this includes INTERNAL mode, too
          {
             bmd.dirtyBatchBuffers.put(buffer, restoreActiveState);
          }
       }

       /**
        * Returns true if a buffer was inserted in the dirtyBatchBuffers map,
        * indicating that the setActiveBuffer was already called for it.
        *
        * @param buffer
        *        Buffer which should be checked against the dirtyBatchBuffers map.
        * @return true if the dirtyBatchBuffers map contains the buffer given as parameter.
        */
       boolean isBufferDirtyBatch(RecordBuffer buffer)
       {
           BatchModeData bmd = batchModeStack.peek();
           if (bmd.batchDepth != BatchModeData.NOT_ACTIVE)
           {
              return bmd.dirtyBatchBuffers.containsKey(buffer);
           }
           return false;
       }

       /**
        * Increment the query depth.
        * <p>
        * This method MUST be invoked in balanced pairs that involve a subsequent call to
        * {@link #decrementDepth}.  Calls may be nested, but they must be balanced, such that
        * at the end of all nested calls, <code>queryDepth</code> must be <code>0</code>.
        *
        * @return   <code>true</code> if the starting depth is 0.
        */
       boolean incrementDepth()
       {
          return (queryDepth++ == 0);
       }

       /**
        * Decrement the query depth.
        * <p>
        * This method MUST be invoked in balanced pairs that involve a subsequent call to
        * {@link #incrementDepth}.  Calls may be nested, but they must be balanced, such that
        * at the end of all nested calls, <code>queryDepth</code> must be <code>0</code>.
        *
        * @return   <code>true</code> if the resulting depth is 0.
        */
       boolean decrementDepth()
       {
          nestedQueryError = false;

          return (--queryDepth == 0);
       }

       /**
        * Check whether the current query depth is non-zero, indicating that we are processing a query within
        * the context of another query. If so, raise a 7254 error condition, unless the current query represents
        * a legacy FIND by RECID/ROWID, which is permitted to run within such context.
        *
        * @param   isFindByRowid
        *          A lambda expression which should return {@code true} if the current database operation
        *          represents a legacy FIND by RECID/ROWID, else {@code false}.
        */
       void checkNestedQuery(Supplier<Boolean> isFindByRowid)
       {
          if (queryDepth == 0 || isFindByRowid.get())
          {
             // normal state
             return;
          }

          Object referent = pm._thisProcedure();
          String iename = ProcedureManager.resolveClosestMethod(referent, true, false);
          String msg =
             "A FIND, FOR EACH or OPEN QUERY statement has been encountered in a user-defined " +
             "function, method or constructor '" +
             iename +
             "' which was called from a WHERE clause.  This statement cannot be executed in " +
             "this context";

          nestedQueryError = true;

          ErrorManager.recordOrThrowError(7254, msg, false);
       }

       /**
        * Indicate whether a nested query error has been detected and has not yet been cleared.
        *
        * @return  {@code true} if a nested query error has been detected, else {@code false}.
        */
       boolean isNestedQueryError()
       {
          return nestedQueryError;
       }

       /**
        * Changes the unknown mode.
        *
        * @param   unknownMode
        *          The new unknown mode.
        *
        * @return  A {@code Runnable} which should be called in a {@code finally} block, paired with the call to
        *          this method, which restores the unknown mode to its previous value.
        */
       boolean setUnknownMode(boolean unknownMode)
       {
          boolean previous = this.unknownMode;
          this.unknownMode = unknownMode;

          return previous;
       }

       /**
        * Checks whether the unknown mode is active.
        *
        * @return  the current state of unknown mode flag.
        */
       boolean isUnknownMode()
       {
          return unknownMode;
       }

       /**
        * Add potentially unused keys to be reclaimed after the end of the current transaction or
        * as buffer scopes are closed after a transaction. Some of these keys may not be collected,
        * if the locks on them are downgraded instead of released.
        *
        * @param   persistence
        *          Persistence helper object for a particular database.
        * @param   keys
        *          Keys to be reclaimed.
        */
       void addPendingReclaimedKeys(Persistence persistence, List<Long> keys)
       {
          if (keys.isEmpty())
          {
             return;
          }

          Set<Long> pending = pendingReclaimedKeys.get(persistence);
          if (pending == null)
          {
             pending = new HashSet<>();
             pendingReclaimedKeys.put(persistence, pending);
          }

          pending.addAll(keys);
       }

       /**
        * Reclaim unused keys collected during the transaction which just ended or from the buffer
        * scope which just closed.
        *
        * @param   persistence
        *          Persistence helper object for a particular database.
        * @param   released
        *          Keys actually released, due to the corresponding locks on them being released
        *          (as opposed to downgraded).
        */
       void reclaimPendingKeys(Persistence persistence, Set<Long> released)
       {
          if (released == null)
          {
             return;
          }

          Set<Long> pending = pendingReclaimedKeys.get(persistence);
          if (pending == null)
          {
             return;
          }

          IdentityManager identityManager = persistence.getIdentityManager(true);
          if (identityManager == null)
          {
             return;
          }

          // only reclaim keys actually released (not downgraded); downgraded keys are still
          // pinned by open buffers and cannot be reclaimed in their locked state
          released.retainAll(pending);

          int size = released.size();
          if (size == 0)
          {
             return;
          }

          try
          {
             identityManager.reclaimKeys(released.toArray(new Long[size]));
          }
          finally
          {
             if (pending.removeAll(released) && pending.isEmpty())
             {
                pendingReclaimedKeys.remove(persistence);
             }
          }
       }

       /**
        * Called by {@link RecordBuffer} when a new buffer scope is opened. It registers the open scope with the
        * buffer manager, so that the buffer can receive block scope start and stop notifications, and be
        * registered for commit/rollback/validate processing when a full transaction is begun.
        *
        * @param   buffer
        *          Buffer for which a new scope is being opened.
        */
       void openScope(RecordBuffer buffer)
       {
          boolean inTx = isTransaction();

          if (buffer.isActive())
          {
             openBuffers.add(false, buffer);
             buffer.addOpenBuffersScope(openBuffers.scopes());

             if (inTx)
             {
                maybeActivateTxWrapper(buffer.getDatabase());
             }

             trackDatabase(buffer, true);
          }
          else
          {
             // a dynamic buffer is added to the global scope; also provisionally mark a buffer associated with
             // a persistent procedure as needing to be added to the global scope; this may be overridden later
             // by bufferInitialized
             Boolean global = buffer.isDynamic() || buffer.getPersistentProc() != null
                              ? Boolean.TRUE
                              : Boolean.FALSE;
             uninitializedOpenBuffers.put(buffer, global);
          }
       }

       /**
        * Register static temp-table when its scope is opened.
        *
        * @param   table
        *          The static temp-table.
        * @param   global
        *          Indicates if it is a global temp-table (declared as SHARED GLOBAL).
        */
       void registerStaticTempTable(StaticTempTable table, boolean global)
       {
          Class<?> def = ((BufferImpl) table.defaultBufferHandleNative()).buffer().getDefiningType();
          StaticDefinitionKey key = new StaticDefinitionKey(table._name(), pm._thisProcedure(), def);
          openStaticTempTables.addEntry(global, key, table);
       }

       /**
        * Register static temp table, when the external program gets executed.
        *
        * @param   table
        *          The static temp-table.
        * @param   global
        *          Indicates if it is a global temp table (declared as SHARED GLOBAL).
        */
       void registerPendingStaticTempTable(StaticTempTable table, boolean global)
       {
          registerPendingScopeable(this);

          pendingStaticTempTables.put(table, global);
       }

       /**
        * Check if the specified handle instance is registered as an OUTPUT argument.
        *
        * @param    h
        *           The handle instance.
        *
        * @return   <code>true</code> if the handle originates from the caller as an OUTPUT argument.
        */
       public boolean isOutputTableHandle(handle h)
       {
          for (int i = 0; i < thOutputParameters.size(); i++)
          {
             Map<OutputTableHandleCopier, Integer> map = thOutputParameters.getDictionaryAtScope(i, true);
             for (OutputTableHandleCopier out : map.keySet())
             {
                if (out.getCalled() == h)
                {
                   return true;
                }
             }
          }
          return false;
       }

       /**
        * Register the output table copier.
        *
        * @param   copier
        *          Copier to register.
        */
       void registerOutputTableHandleCopier(OutputTableHandleCopier copier)
       {
          Map<OutputTableHandleCopier, Integer> map = thOutputParameters.getDictionaryAtScope(0, true);
          thOutputParameters.addEntry(false, copier, map.size());
          copier.setSiblingsMap(map);
       }

       /**
        * Unregister the output table copier <b>in the current scope</b>.
        *
        * @param   copier
        *          Copier to unregister.
        */
       void unregisterOutputTableHandleCopier(OutputTableHandleCopier copier)
       {
          thOutputParameters.removeEntryAtScope(copier, 0);
       }

       /**
        * Get definition for the active static temp table with the given name.
        *
        * @param   defType
        *          The type where this call is being made from.
        * @param   name
        *          Legacy name of the target static temp table.
        *
        * @return  Definition of the static temp table with the given name or <code>null</code> if
        *          there is no active table with the given name.
        */
       StaticTempTable getStaticTempTable(Class<?> defType, String name)
       {
          if (defType == null)
          {
             defType = pm.getExecuting();
          }

          if (defType == null)
          {
             throw new IllegalStateException("Null executing type when looking for " + name + " table");
          }

          if (!pendingStaticTempTables.isEmpty())
          {
             for (StaticTempTable table : pendingStaticTempTables.keySet())
             {
                Class<?> tdef = pendingBufferClasses.get(table.defaultBuffer());
                if (tdef == defType && table._name().equalsIgnoreCase(name))
                {
                   return table;
                }
             }
          }

          Class<?> type = defType;
          return (StaticTempTable) pm.searchInStack((ref) -> getStaticTempTable(type, name, ref));
       }

       /**
        * Get definition for the active static dataset with the given name.
        * <p>
        * The lookup is done first from super-classes and after that in {@link #dataSetRegistry}.
        *
        * @param   defType
        *          The type where this call is being made from.
        * @param   name
        *          Legacy name of the target static dataset.
        * @param   referent
        *          The reference used to lookup this dataset by name.
        *
        * @return  The found dataset with the same legacy name.
        */
       DataSet getStaticDataSet(Class<?> defType, String name, Object referent)
       {
          StaticDefinitionKey key = new StaticDefinitionKey(name, referent, defType);

          Object thisProc = pm._thisProcedure();
          PersistentProcScope ppScope = persistProcScopes.get(thisProc);
          DataSet ds = null;
          if (ppScope != null)
          {
             ds = ppScope.datasets.get(key);

             if (ds == null)
             {
                // check the defType hierarchy, in the same referent
                if (BaseObject.class.isAssignableFrom(defType))
                {
                   Class<?> type = defType.getSuperclass();
                   while (type != BaseObject.class)
                   {
                      StaticDefinitionKey key2 = new StaticDefinitionKey(name, referent, type);
                      ds = ppScope.datasets.get(key2);
                      if (ds != null)
                      {
                         break;
                      }

                      type = type.getSuperclass();
                   }
                }
             }
          }

          return ds != null ? ds : dataSetRegistry.lookup(key);
       }

       /**
        * Get definition for the active static temp table with the given name.
        *
        * @param   defType
        *          The type where this call is being made from.
        * @param   name
        *          Legacy name of the target static temp table.
        * @param   referent
        *          The external program where it was defined.
        *
        * @return definition of the static temp table with the given name or <code>null</code> if
        *         there is no active table with the given name.
        */
       StaticTempTable getStaticTempTable(Class<?> defType, String name, Object referent)
       {
          StaticDefinitionKey key = new StaticDefinitionKey(name, referent, defType);
          return openStaticTempTables.lookup(key);
       }

       /**
        * Get the depth of the scope at which the static temp table was opened.
        *
        * @param   name
        *          Legacy name of the target static temp table.
        *
        * @return zero-based index of the target scope, starting from the outermost scope. If there is
        *         no table with the given name, <code>-1</code> is returned.
        */
       int getStaticTempTableDepth(String name)
       {
          Integer val = (Integer) pm.searchInStack((ref) -> getStaticTempTableDepth(name, ref));

          return val == null ? -1 : val;
       }

       /**
        * Get the depth of the scope at which the static temp table was opened.
        *
        * @param   stt
        *          The static temp-table instance.
        *
        * @return zero-based index of the target scope, starting from the outermost scope. If there is
        *         no table with the given name, <code>-1</code> is returned.
        */
       int getStaticTempTableDepth(StaticTempTable stt)
       {
          Map<StaticDefinitionKey, StaticTempTable> scope = null;
          for (int i = 0; i < openStaticTempTables.size(); i++)
          {
             scope = openStaticTempTables.getDictionaryAtScope(i, false);
             if (scope.values().contains(stt))
             {
                return openStaticTempTables.size() - i - 1;
             }
          }
          return -1;
       }

       /**
        * Get the depth of the scope at which the static temp table was opened.
        *
        * @param   name
        *          Legacy name of the target static temp table.
        * @param   referent
        *          The external program where it was defined.
        *
        * @return zero-based index of the target scope, starting from the outermost scope. If there is
        *         no table with the given name, <code>-1</code> is returned.
        */
       int getStaticTempTableDepth(String name, Object referent)
       {
          Class<?> def = pm.getExecuting();
          StaticDefinitionKey key = new StaticDefinitionKey(name, referent, def);
          int topDepth = openStaticTempTables.locate(key);

          return topDepth == -1 ? -1 : openStaticTempTables.size() - topDepth - 1;
       }

       /**
        * Add the target buffer to the set of all buffers at the specified block depth.
        *
        * @param   blockDepth
        *          Zero-based depth of the block (starting from the outermost block) at which the
        *          buffer was opened.
        * @param   buffer
        *          Buffer to add.
        */
       void addToAllBuffersArray(int blockDepth, RecordBuffer buffer)
       {
          Object referent = pm._thisProcedure();

          registerToBuffersByClass(blockDepth, buffer);

          allBuffers.addEntryAt(blockDepth, buffer.getDMOToken(), buffer);
          buffer.addAllBufferScope(allBuffers.size() - blockDepth);
          buffer.addByLegacyNameScope(byLegacyName.size() - blockDepth);

          Map<Object, Map<String, RecordBuffer>> refs = byLegacyName.getDictionaryAtScope(blockDepth, true);

          Map<String, RecordBuffer> bufDefs = refs.get(referent);
          if (bufDefs == null)
          {
             refs.put(referent, bufDefs = new CaseInsensitiveHashMap<>());
          }
          bufDefs.put(buffer.getLegacyName(), buffer);
       }

       /**
        * Called by {@link TemporaryBuffer} when a new dynamic buffer scope is opened to register the
        * open scope with the buffer manager, so that the buffer can receive block scope start and
        * stop notifications, and be registered for commit processing when a full transaction is
        * begun.
        *
        * @param   openScopeDepth
        *          Zero-based depth of the open scopes (starting from the outermost scope) at which this
        *          buffer scope is opened.
        * @param   buffer
        *          Buffer for which a new scope is being opened.
        */
       void openScopeAt(int openScopeDepth, RecordBuffer buffer)
       {
          // TODO: this seems to be called only for 'openScopeDepth == 0'.

          if (buffer.isActive())
          {
             if (openScopeDepth == 0)
             {
                openBuffers.add(true, buffer);
                buffer.addOpenBuffersScope(1); // global scope
             }
             else
             {
                openBuffers.addAt(getOpenBufferScopes() - openScopeDepth, buffer);
                buffer.addOpenBuffersScope(openScopeDepth);
             }

             if (isTransaction())
             {
                maybeActivateTxWrapper(buffer.getDatabase());
             }

             trackDatabase(buffer, true);
          }
          else
          {
             uninitializedOpenBuffers.put(buffer, openScopeDepth == 0 ? Boolean.TRUE : Boolean.FALSE);
          }
       }

       /**
        * A buffer was initialized. If it was previously stored as an uninitialized, open buffer, move it into
        * the active, open buffers list. Perform special handling for persistent procedure buffers.
        *
        * @param   buffer
        *          Record buffer which was initialized.
        *
        * @return  {@code true} if this was properly registered as an open buffer. {@code false} if this couldn't
        *          be properly registered because the scope wasn't opened properly beforehand.
        */
       boolean bufferInitialized(RecordBuffer buffer)
       {
          Boolean global = uninitializedOpenBuffers.remove(buffer);

          if (global == null)
          {
             // buffer was not found among the pending, uninitialized buffers; nothing to do
             return buffer.getOpenScopeCount() > 0;
          }

          // dynamic buffers always are added to the global scope; persistent procedure buffers need further
          // inspection to determine whether they are globally scoped or not...
          if (global && !buffer.isDynamic())
          {
             Object referent = buffer.getPersistentProc();
             if (referent != null)
             {
                // a buffer associated with a persistent procedure will have been provisionally marked global,
                // but this is overridden if the persistent procedure is still on the stack (in which case it
                // is added to the scope corresponding with the buffer's activeScopeDepth)
                global = pm.isExecuted(referent);

                if (global)
                {
                   // the procedure is no longer on the stack, so we must explicitly add the buffer to the
                   // PersistentProcScope.openBuffers collection for the PersistentProcScope which corresponds
                   // with the procedure
                   PersistentProcScope ppScope = persistProcScopes.computeIfAbsent(referent, k ->
                   {
                      return new PersistentProcScope();
                   });
                   ppScope.openBuffers.add(buffer);
                }
             }
          }

          if (global)
          {
             openBuffers.add(true, buffer);
             buffer.addOpenBuffersScope(1); // global scope
          }
          else
          {
             int scopes = getOpenBufferScopes();
             if (buffer.activeScopeDepth <= scopes)
             {
                openBuffers.addAt(scopes - buffer.activeScopeDepth, buffer);
                buffer.addOpenBuffersScope(buffer.activeScopeDepth);
             }
          }

          trackDatabase(buffer, true);
          return true;
       }

       /**
        * Given a database which has one or more buffers with an open scope operating within an application
        * level transaction, possibly activate the transaction wrapper associated with that database.
        * Activation entails starting a database-level transaction and activating a savepoint manager
        * for that transaction.
        * <p>
        * If the transaction wrapper for the given database already is active, do nothing.
        *
        * @param   database
        *          Database with at least one buffer with an open scope.
        */
       void maybeActivateTxWrapper(Database database)
       {
          txWrapperHelper.maybeActivateTxWrapper(database);
       }

       /**
        * Find the record buffer associated with the given DMO alias.
        *
        * @param   alias
        *          Buffer DMO alias in business logic. (AKA Java name)
        *
        * @return  Associated record buffer or {@code null} if not found.
        */
       RecordBuffer lookupBuffer(String alias)
       {
          for (RecordBuffer buffer : pendingBuffers)
          {
             String token = buffer.getDMOToken();
             if (token != null && token.equals(alias))
             {
                return buffer;
             }
          }

          return allBuffers.lookup(alias);
       }

       /**
        * Find the record buffer associated with the given legacy name, which is the variable name
        * with which the buffer was defined. Because this method is legally called only after the
        * procedure block started, the lookup is not performed on a structure similar to
        * {@link #pendingBuffers}.
        *
        * @param   legacyName
        *          Buffer variable's legacy name in business logic.
        *
        * @return  Associated record buffer or {@code null} if not found.
        */
       RecordBuffer lookupByName(String legacyName)
       {
          // look in the current stack of executing Class

          Iterator<Object> it = pm.destinationIterator();
          while (it.hasNext())
          {
             Object ref = it.next();
             Map<String, RecordBuffer> bufDefs =
                      byLegacyName.lookup(ref, byLegacyName.size(), (Map<String, RecordBuffer> defs) ->
             {
                return defs != null && defs.containsKey(legacyName);
             });

             if (bufDefs != null && bufDefs.containsKey(legacyName))
             {
                return bufDefs.get(legacyName);
             }

             // lookup in the persistent procedure, too
             PersistentProcScope ppScope = persistProcScopes.get(ref);
             if (ppScope != null)
             {
                bufDefs = ppScope.byLegacyName.get(ref);
                if (bufDefs != null && bufDefs.containsKey(legacyName))
                {
                   return bufDefs.get(legacyName);
                }
             }
          }

          return null;
       }

       /**
        * Get an iterator on all open buffers for the given persistence object, within the current client context.
        *
        * @param   p
        *          A {@code Persistence} object which manages sessions. Only record buffers associated with this
        *          persistence object are returned.
        * @param   multiTenantOnly
        *          If {@code true}, only buffers of multi-tenant tables are filtered to be returned.
        *          Otherwise, all opened buffers are returned in the iterator.
        *
        * @return  Iterator on all (initialized) buffers.
        */
       Iterator<RecordBuffer> getBuffersOf(Persistence p, boolean multiTenantOnly)
       {
          Set<RecordBuffer> ret = new HashSet<>();
          for (RecordBuffer next : openBuffers)
          {
             if (next.isActive() && p.equals(next.getPersistence()))
             {
                if (!multiTenantOnly || next.getDmoInfo().multiTenant)
                {
                   ret.add(next);
                }
             }
          }

          return ret.iterator();
       }

       /**
        * Get an iterator on all open buffers which contain non-<code>null</code>
        * current records for the given persistence object, within the current
        * client context.  Auto-commit record buffers associated with a pending,
        * reversible rollback are not included in the iteration.  Records that
        * represent copies of DMOs from other sessions, obtained from a {@link
        * DirtyShareContext}, are likewise excluded.
        *
        * @param   persistence
        *          Persistence object which manages sessions.  Only record
        *          buffers associated with this persistence object are returned.
        *
        * @return  Iterator on all active buffers.
        */
       Iterator<RecordBuffer> activeBuffers(Persistence persistence)
       {
          return activeBuffers(persistence, false);
       }

       /**
        * Get an iterator on all open buffers which contain non-<code>null</code>
        * current records for the given persistence object, within the current
        * client context.  Auto-commit record buffers associated with a pending,
        * reversible rollback are not included in the iteration.  Records that
        * represent copies of DMOs from other sessions, obtained from a {@link
        * DirtyShareContext}, are likewise excluded.
        *
        * @param   persistence
        *          Persistence object which manages sessions.  Only record
        *          buffers associated with this persistence object are returned.
        * @param   includeTransient
        *          <code>true</code> to include transient records in the result set.
        *
        * @return  Iterator on all active buffers.
        */
       Iterator<RecordBuffer> activeBuffers(Persistence persistence, boolean includeTransient)
       {
          Set<RecordBuffer> set = new HashSet<>();
          for (RecordBuffer next : openBuffers)
          {
             if (isActiveBuffer(next, persistence, includeTransient))
             {
                set.add(next);
             }
          }

          return set.iterator();
       }

       /**
        * Check if the given buffer contains a non-<code>null</code> current record for
        * the given persistence object, within the current client context. Auto-commit
        * record buffers associated with a pending, reversible rollback are not included
        * in the iteration. Records that represent copies of DMOs from other sessions,
        * obtained from a {@link DirtyShareContext}, are likewise excluded.
        *
        * @param   buffer
        *          RecordBuffer that is checked.
        * @param   persistence
        *          Persistence object which manages sessions. Only record
        *          buffers associated with this persistence object are returned.
        * @param   includeTransient
        *          <code>true</code> to include transient records in the result set.
        *
        * @return  <code>true</code> if the buffer has a non-<code>null</code> current
        *          record, <code>false</code> otherwise.
        */
       static boolean isActiveBuffer(RecordBuffer buffer, Persistence persistence, boolean includeTransient)
       {
          if (buffer.isActive()                           &&
              (includeTransient || !buffer.isTransient()) &&
              !buffer.isDirtyCopy()                       &&
              !buffer.isFake()                            &&
              persistence.equals(buffer.getPersistence()) &&
              buffer.getCurrentRecord() != null)
          {
             return true;
          }

          return false;
       }

       /**
        * Evict the given DMO from the current ORM session if it is not referenced by any buffer or being
        * tracked for UNDO purposes. This method does not modify the DMO's in-use status.  It is intended to
        * quickly evict DMOs which have been loaded into the session by side effect.
        *
        * @param   persistence
        *          Persistence object which manages sessions.
        * @param   local
        *          Current user persistence context.
        * @param   dmo
        *          DMO instance whose reference count is to be decremented.
        *
        * @return  {@code true} if the DMO was evicted;  {@code false} if it was in use.
        *
        * @throws  PersistenceException
        *          if an error occurs detaching {@code dmo} from the persistence session.
        */
       boolean evictDMOIfUnused(Persistence persistence, Persistence.Context local, Record dmo)
       throws PersistenceException
       {
          if (!dmo.isInUse())
          {
             if (local == null)
             {
                local = persistence.getContext(!dmo._getRecordMeta().getDmoMeta().multiTenant);
             }
             persistence.evict(local, dmo);

             return true;
          }

          return false;
       }

       /**
        * Notify the buffer manager that a new record was loaded (or the old record was reloaded)
        * into the specified record buffer.
        *
        * @param   recordBuffer
        *          The record buffer in which a record was loaded (or the old record was reloaded).
        */
       void notifyRecordWasLoaded(RecordBuffer recordBuffer)
       {
          txWrapperHelper.notifyRecordWasLoaded(recordBuffer);
       }

       /**
        * Generate name (DMO alias) for a dynamic buffer. The name is
        * generated by using standard P2J buffer name conversion rules.
        *
        * @param   legacyBufferName
        *          Legacy name of the buffer.
        *
        * @return  name (alias) for a dynamic buffer.
        */
       String generateBufferName(String legacyBufferName)
       {
          String javaName = convertedNames.get(legacyBufferName);

          if (javaName == null)
          {
             javaName = nc.convert(legacyBufferName, MatchPhraseConstants.TYPE_DMO_ALIAS);

             convertedNames.put(legacyBufferName, javaName);
          }

          return javaName;
       }

       /**
        * Generate unique (across the session) name (DMO token) for a dynamic buffer. The name is
        * generated by using standard P2J buffer name conversion rules and if necessary, appending
        * <code>__{unique number}</code> postfix. This guarantees that the name will not contain
        * invalid characters and will be unique.
        *
        * @param   legacyBufferName
        *          Legacy name of the buffer.
        *
        * @return  unique name (alias) for a dynamic buffer.
        */
       String generateUniqueToken(String legacyBufferName)
       {
          String javaName = generateBufferName(legacyBufferName);

          StringBuilder buf = new StringBuilder();
          buf.append(javaName);
          buf.append("__");
          buf.append(tokenCounter++);
          return buf.toString();
       }

       /**
        * Iterate all active persistence contexts and close any open session, if there are no longer
        * any dependencies on it. This method deregisters an active persistence context by side
        * effect, if its open session was closed.
        */
       private void maybeCloseSessions()
       {
          if (activePersistenceContexts.isEmpty())
          {
             return;
          }

          Iterator<Persistence.Context> iter = activePersistenceContexts.iterator();
          while (iter.hasNext())
          {
             Persistence.Context pc = iter.next();

             if (pc.canCloseSession())
             {
                try
                {
                   if (pc.closeSession(false))
                   {
                      // remove the active persistence context if the close attempt succeeded
                      iter.remove();
                   }
                }
                catch (PersistenceException exc)
                {
                   if (LOG.isLoggable(Level.SEVERE))
                   {
                      String msg = "Error closing database session (" +
                                   pc.getPersistence().getDatabase(Persistence.PRIVATE_CTX) +
                                   ")";
                      LOG.log(Level.SEVERE, msg, exc);
                   }
                }
             }
          }
       }

       /**
        * Add the source buffers to the destination map.
        *
        * @param    dst
        *           The destination map.
        * @param    src
        *           The source map.
        */
       private void checkLegacyNames(Map<Object, Map<String, RecordBuffer>> dst,
                                     Map<Object, Map<String, RecordBuffer>> src)
       {
          for (Map.Entry<Object, Map<String, RecordBuffer>> entry : src.entrySet())
          {
             Object ref = entry.getKey();
             Map<String, RecordBuffer> srcBufs = entry.getValue();
             Map<String, RecordBuffer> dstBufs = dst.computeIfAbsent(ref, (k) -> new CaseInsensitiveHashMap<>());
             dstBufs.putAll(srcBufs);
          }
       }

       /**
        * Track the buffer's database.  This will increment the {@link TxWrapper.WorkArea#activeDatabases} counter when a
        * new buffer is activated, and decrement it when it exists its scope.
        *
        * @param    buffer
        *           The buffer to track.
        * @param    add
        *           Flag indicating if the buffer is opening or closing the scope.
        *
        * @throws   IllegalStateException
        *           If we are removing an active buffer and there is no entry for it in
        *           {@link TxWrapper.WorkArea#activeDatabases}.
        */
       private void trackDatabase(RecordBuffer buffer, boolean add)
       {
          txWrapperHelper.trackDatabase(buffer, add);
       }

       /**
        * Register the provided buffer to a provided block depth of {@link #buffersByDMOBufClass}.
        *
        * @param   blockDepth
        *          The block depth at which the buffer should be registered.
        * @param   buffer
        *          The buffer to be registered.
        */
       private void registerToBuffersByClass(Integer blockDepth, RecordBuffer buffer)
       {
          String dmoBufIface = buffer.getDMOBufInterface().getName();
          SimpleEntry<String, Integer> key = new SimpleEntry<>(dmoBufIface, buffer.getMultiplexID());
          Set<RecordBuffer> recordBuffersPerTable;
          if (blockDepth != null)
          {
             // add to specified scope
             recordBuffersPerTable = buffersByDMOBufClass.getValueAtScope(key, blockDepth);

             if (recordBuffersPerTable == null)
             {
                recordBuffersPerTable = new HashSet<>();
                buffersByDMOBufClass.addEntryAt(blockDepth, key, recordBuffersPerTable);
             }

             // this is required so that dynamic buffers have a change to do their own clear-up when deleted
             buffer.addBufferByClassScope(buffersByDMOBufClass.size() - blockDepth);
          }
          else
          {
             // add to last scope
             recordBuffersPerTable = buffersByDMOBufClass.getValueAtScope(key, 0);
             if (recordBuffersPerTable == null)
             {
                recordBuffersPerTable = new HashSet<>();
                buffersByDMOBufClass.addEntry(false, key, recordBuffersPerTable);

                // this is required so that dynamic buffers have a change to do their own clear-up when deleted
                buffer.addBufferByClassScope(buffersByDMOBufClass.size());
             }
          }

          recordBuffersPerTable.add(buffer);
       }

       /**
        * Indicate whether the current block is &quot;important&quot; from the standpoint of transitioning into
        * or out of its scope. A block is considered &quot;important&quot; in this sense if it is either
        * completely outside of an application transaction, or if it is within a transaction AND has transaction
        * block properties (i.e., is not a NO_TRANSACTION block).
        * <p>
        * This notion of importance is used to determine whether the various scoped data structures managed by
        * this class need to add or delete a scope when transitioning into or out of the block, respectively,
        * and whether we need to perform processing (e.g., copying/moving data between scopes) associated with
        * such transitions.
        *
        * @return  {@code true} if the block is considered &quot;important&quot; by the above criteria, else
        *          {@code false}.
        */
       public boolean isImportantBlockTransition()
       {
          return txWrapperHelper.isImportantBlockTransition();
       }

       /**
        * Container for batch assign mode information.
        */
       private static class BatchModeData
       {
          /** Marks a not active scope. */
          public static final int NOT_ACTIVE = -1;

          /**
           * Marks an internal scope. Internal scopes are scopes artificially introduced for a FWD specific
           * data management. This must always be the last scope and must be popped out before other normal (4GL-)
           * requested batch scopes are inserted.
           */
          public static final int INTERNAL = -2;

          /** Scope depth at which batch assign mode began; -1 if not in batch assign mode */
          private int batchDepth = NOT_ACTIVE;

          /** Map of dirty batch assign mode buffers with corresponding code that restores their active state.*/
          private Map<RecordBuffer, Runnable> dirtyBatchBuffers = null;

          /**
           * Default constructor.
           */
          BatchModeData()
          {
          }

          /**
           * Activate batch assign mode.
           *
           * @param   batchDepth
           *          Scope depth at which batch assign mode was activated.
           */
          void activate(int batchDepth)
          {
             this.batchDepth = batchDepth;
             dirtyBatchBuffers = new LinkedHashMap<>();
          }

          /**
           * Obtain a string encoding of the batch mode data.
           */
          @Override
          public String toString()
          {
             return "BMData{" +
                   (batchDepth == INTERNAL ? "INTERNAL" : batchDepth == NOT_ACTIVE ? "NOT-ACTIVE" : batchDepth) +
                   ", buffers: #" + (dirtyBatchBuffers == null ? 0 : dirtyBatchBuffers.size()) + '}';
          }

       }

       /**
        * Buffer-related data for a certain persistent procedure, surviving after its external
        * procedure is finished.
        */
       private class PersistentProcScope
       {
          /**
           * All defined record buffers for a persistent procedure. This data structure uses
           * the DMO alias as keys.
           * TODO: this can't be by String
           */
          private final Map<String, RecordBuffer> allBuffers = new HashMap<>();

          /**
           * Scoped dictionary in which all defined record buffers are stored. This data structure uses
           * the <strong>lowercased legacy name</strong> as keys.
           * <p>
           * Note: only the static buffers are kept in this structure. The dynamic resources cannot be
           *       stored here as their NAME attribute is mutable and will possible collide with other
           *       existing buffers.
           */
          private final Map<Object, Map<String, RecordBuffer>> byLegacyName = new IdentityHashMap<>();

          /**
           * Scoped dictionary with key as a DMO buffer interface and multiplex, and the value is a list of
           * RecordBuffers associated with them.
           */
          private final Map<SimpleEntry<String, Integer>, Set<RecordBuffer>> buffersByDMOBufClass =
                   new HashMap<>();

          /** Open record buffers for a persistent procedure. */
          private final Set<RecordBuffer> openBuffers = new LinkedHashSet<>();

          /**
           * Active static temp tables (i.e. table scope was opened and not yet closed) for a
           * persistent procedure.
           */
          private final Map<StaticDefinitionKey, StaticTempTable> openStaticTempTables = new HashMap<>();

          /** The static datasets defined in this instance. */
          private final Map<StaticDefinitionKey, DataSet> datasets = new HashMap<>();
       }

       /**
        * A temp-table key contains the lowercase legacy name of the dataset/table and the procedure where it
        * was created. This way we can be sure there are no collisions between datasets/temp tables having the
        * same name but created by different procedures.
        */
       private class StaticDefinitionKey
       {
          /** The lowercased temp-table legacy name. */
          private final String legacyName;

          /** The parent procedure where the temp-table was created. */
          private final Object parentProcedure;

          /** The Java class defining this table. */
          private final Class<?> def;

          /**
           * The hash value is computed when the key is created. These keys are immutable so the hash
           * is computed only once and cached.
           */
          private final int hash;

          /**
           * The contructors copies the parameters to internal data and computed the hash value.
           *
           * @param   legacyName
           *          The lowercased temp-table legacy name.
           * @param   parentProcedure
           *          The parent procedure where the temp-table was created.
           * @param   def
           *          The Java class defining this buffer.
           */
          public StaticDefinitionKey(String legacyName, Object parentProcedure, Class<?> def)
          {
             // do not lowercase, 'equals' uses 'equalsIgnoreCase' and the hash is case-insensitive
             this.legacyName = legacyName;
             this.parentProcedure = parentProcedure;
             this.def = def;
             int lhash = StringHelper.hashCodeCaseInsensitive(legacyName);
             hash = (lhash * 31 + parentProcedure.hashCode()) * 31 + def.hashCode();
          }

          /**
           * Checks whether the alien object is equals to this one.
           * @param   o
           *          The alien object.
           *
           * @return  {@code true} only if the two objects are the same (same temp-table created
           *          within the same procedure).
           */
          @Override
          public boolean equals(Object o)
          {
             if (this == o)
             {
                return true;
             }
             if (o == null || getClass() != o.getClass())
             {
                return false;
             }

             StaticDefinitionKey that = (StaticDefinitionKey) o;

             if (this.hash != that.hash)
             {
                return false;
             }

             if (!legacyName.equalsIgnoreCase(that.legacyName) || def != that.def)
             {
                return false;
             }
             return parentProcedure.equals(that.parentProcedure);
          }

          /**
           * Obtain the hash code as it was computed in the constructor.
           *
           * @return  the hash value for this key.
           */
          @Override
          public int hashCode()
          {
             return hash;
          }
       }
    }