PreselectQuery.java

/*
** Module   : PreselectQuery.java
** Abstract : A query which enables stateful, cursored navigation of a server side result set
**
** Copyright (c) 2004-2025, Golden Code Development Corporation.
**
** -#- -I- --Date-- --JPRM-- -----------------------------------Description-----------------------------------
** 001 ECF 20051115   @23396 Created initial version, based on prototype
**                           CursoredQuery. Backing support for queries
**                           which must support the Progress, row-level
**                           access semantic against a preselected result
**                           set.
** 002 ECF 20051129   @23575 Added lenient-off-end mode. When in this
**                           mode, QueryOffEndExceptions thrown by the
**                           underlying buffer during next and previous
**                           operations are suppressed.
** 003 ECF 20051216   @23750 Added scrolling support. Modified retrieval
**                           logic to allow for repositioning. Added many
**                           new methods.
** 004 ECF 20060105   @23831 Inserted abstraction layer between query and
**                           its scrollable results. Added the Results
**                           interface and its default implementation,
**                           SimpleResults. This allows subclasses to
**                           provide custom implementations by overriding
**                           the createResults method.
** 005 ECF 20060202   @24358 Added temp table support. Use buffer's
**                           multiplex ID as a query substitution
**                           parameter when executing HQL. Parameters are
**                           now managed separately, by component, to
**                           enable this processing.
** 006 ECF 20060208   @24721 Added support for client-side where clause
**                           processing. Results and SimpleResults are now
**                           top level classes. Modified all single-table
**                           constructor variants to accept an instance of
**                           WhereExpression. Modified Component to handle
**                           argument resolution more correctly. Results
**                           can now be provided by an external entity
**                           (CompoundQuery typically).
** 007 ECF 20060302   @25019 Add support for foreign key joins. Added new
**                           constructor and addComponent method variants
**                           which mirror existing versions, but accept an
**                           additional, inverse DMO parameter. Updated
**                           Component class, HQL generation, and query
**                           execution to integrate natural joins.
** 008 ECF 20060422   @25657 Added setIterationType method. Sets retrieval
**                           mode (first, last, next, etc.) for the most
**                           recently added query component.
** 009 ECF 20060423   @25661 Allow multiple client-side where clause
**                           expressions to be registered. This is needed
**                           because each table in a multi-table query can
**                           contain its own client-side where component.
** 010 ECF 20060424   @25670 Fix to several constructor variants. Swapped
**                           positions of sort and inverse parameters to
**                           match the natural order in which these
**                           parameters are emitted in converted code.
** 011 ECF 20060424   @25695 Added new constructor variants. Each existing
**                           variant now has a counterpart which does not
**                           expect a WhereExpression parameter.
** 012 ECF 20060424   @25699 Roll back previous change. Fixed ordering of
**                           some parameters in internal constructor
**                           calls.
** 013 ECF 20060425   @25718 Added setLenientOffEnd method. Required for
**                           conversion of certain OPEN QUERY constructs.
** 014 ECF 20060501   @25915 Added isPreselect method. Indicates query's
**                           preselection behavior.
** 015 ECF 20060531   @26850 Replaced EndConditionException with
**                           QueryOffEndException. The latter is a more
**                           specific version of the former, used to
**                           indicate a query has run off the end of its
**                           result set.
** 016 ECF 20060602   @26944 Optimization. HQL queries now will always
**                           attempt to load the full record, rather than
**                           just the primary key ID. This avoids the
**                           overhead of a separate load step.
** 017 ECF 20060616   @27412 Fix to regression caused by #016 (@26944).
**                           FilteredResults needs to store DMO instance
**                           within an array.
** 018 ECF 20060705   @27828 Fixed ClassCastException. Made fetch method
**                           tolerant of a primary key ID result (instead
**                           of DMO) in a result set.
** 019 ECF 20060713   @28051 Integrated HQLPreprocessor. An instance of
**                           this class is created for each component
**                           added to the query. This affects how the
**                           overall HQL query is composed.
** 020 ECF 20060719   @28116 Rolled back #016 (@26944). With the addition
**                           of the change broker architecture, it is
**                           actually possible to avoid loading the record
**                           at all in limited cases. In these cases, only
**                           a Hibernate proxy is created, which is far
**                           more efficient than retrieving the entire
**                           record up front. Additionally, #016 was
**                           problematic for locking, because the record
**                           could be changed by another user after it was
**                           fetched, but before the lock was acquired.
** 021 ECF 20060722   @28135 Changes to make inheritance of this class
**                           more flexible. In particular, this set of
**                           changes accommodates the AdaptiveQuery
**                           subclass.
** 022 ECF 20060725   @28188 Fixed regressions and implemented
**                           Finalizable. The latter is necessary to
**                           support retry, and to prevent a session from
**                           closing prematurely and discarding the
**                           query's results.
** 023 ECF 20060730   @28270 Client-side where clause processing changes.
**                           Moved FilteredResults into a top-level class.
** 024 ECF 20060801   @28352 Refactored execute() method and order by
**                           clause preparation code. Query substitution
**                           parameter preparation was moved into its own
**                           polymorphic method. Integrated SortCriterion
**                           class to replace code that was duplicative
**                           with that class.
** 025 ECF 20060803   @28396 Further refactoring. Added getRecordBuffers()
**                           method. Made results private again.
** 026 ECF 20060807   @28462 Fixed assembleOrderByClauseNoPreamble(). DMO
**                           alias needed to be trimmed of leading space.
** 027 ECF 20060829   @28954 Fixed join defect in constructor. We were
**                           creating a join and passing it to the wrong
**                           version of addComponent().
** 028 ECF 20060830   @28991 Additional join fix. The fix has been moved
**                           from a constructor to the most specific
**                           variant of addComponent().
** 029 ECF 20060831   @29087 Added support for sorting by an indexed
**                           property. Specifically, support sorting by
**                           a property of a DMO's associated list of
**                           composite properties.
** 030 ECF 20060901   @29139 Fixed NPE. Methods which accept a list of
**                           sort criteria must handle null list case.
** 031 ECF 20060911   @29461 Added debug code. Report DMOs found by the
**                           query if trace logging is enabled.
** 032 ECF 20060925   @29963 Check for a pending error before retrieving a
**                           record. Retrieval is not attempted if an
**                           error is pending in the ErrorManager.
** 033 ECF 20061003   @30154 Backed out #032 (@29963). This logic is only
**                           necessary for FindQuery and AdaptiveFind.
** 034 ECF 20061010   @30310 Minor change to initialization logic. Joins
**                           across different Persistence objects, rather
**                           than across databases, are now disallowed.
** 035 ECF 20061019   @30528 Fixed reposition support to better handle
**                           varying subclass behavior. This class now
**                           implements the SessionListener interface.
**                           Upon receiving a sessionClosing notification,
**                           a PreselectQuery will cache all of its
**                           results in a SimpleResults object and replace
**                           its existing results with that cache.
** 036 ECF 20061023   @30607 Moved errorIfNull logic to AbstractQuery base
**                           class. Method setErrorIfNull() is now
**                           required by the P2JQuery interface and is
**                           implemented in AbstractQuery instead of here.
** 037 ECF 20061027   @30752 Replaced Object with DataModelObject for DMO
**                           arguments to methods/constructors. Required
**                           for compile-time type safety.
** 038 ECF 20061102   @30897 Fixed issue querying row count. The cached
**                           row count must be reset to its default value
**                           when a new result set is retrieved. This
**                           forces the size() method to recalculate the
**                           proper row count.
** 039 ECF 20061102   @30959 Added support for reposition notification and
**                           auto-fetch features. Cleaned up reposition
**                           logic so that fetches after a reposition are
**                           (hopefully) correct now.
** 040 ECF 20061115   @31209 Changed call to set current buffer record.
**                           RecordBuffer API has changed to permit proper
**                           management of placeholder snapshot record.
**                           Added isResetSnapshot() method to allow
**                           subclasses to override this behavior.
** 041 ECF 20061120   @31326 Modified Component inner class. Expose
**                           restriction properties determined during HQL
**                           where clause preprocessing.
** 042 ECF 20061204   @31576 Integrated RepositionCache. This replaced a
**                           simple hash map approach which failed in
**                           cases where a lookup used an ID array which
**                           had fewer IDs than joined tables in a query.
**                           Also emulate Progress reposition quirk which
**                           repositions to last record in result set when
**                           no match is found for the target recid.
** 043 ECF 20061206   @31587 Integrated changes to AbstractQuery's
**                           afterReposition() signature. Failed attempts
**                           to reposition by ID cause afterReposition()
**                           to be invoked in notify-only mode.
** 044 ECF 20070225   @32227 Plug results resource leak. Clean up results
**                           resources when enclosing block ends. This was
**                           causing both prepared statements and result
**                           sets to remain unclosed in the backing JDBC
**                           connection.
** 045 ECF 20070225   @32232 Fixed regression caused by #044 (@32227).
**                           Finalizable which cleans up results resources
**                           must not be hard-coded to nearest enclosing
**                           scope where results are created. Subclasses
**                           may need to register this object to the
**                           global scope instead.
** 046 ECF 20070228   @32247 Optimized no-lock and temp table fetches. In
**                           these cases, it is not necessary to fetch the
**                           record in two steps. The no-lock case makes
**                           no guarantee that the record in memory has
**                           not been updated by other sessions. The temp
**                           table case guarantees that no other session
**                           can make an update to the record in memory,
**                           because a temp table record is inherently
**                           scoped to a session.
** 047 ECF 20070308   @32334 Fixed behavior for first/last. These should
**                           never raise an error condition, only a query
**                           off end condition.
** 048 ECF 20070328   @32648 Changed to support HQLPreprocessor caching.
**                           Use factory method to retrieve an instance of
**                           the preprocessor rather than invoking the
**                           constructor directly.
** 049 ECF 20070412   @32984 Accommodate method name change in
**                           RecordBuffer. setCurrentRecord --> setRecord.
** 050 ECF 20070415   @33002 Fixed locking problem. DMOs fetched "whole"
**                           with lock type NONE must be re-loaded with a
**                           lock if called for by the query component or
**                           lock type override in the retrieval method.
**                           In certain circumstances, these were being
**                           left without a lock.
** 051 ECF 20070416   @33026 Integrated user interrupt handling.
** 052 ECF 20070502   @33371 Changed to support HQLPreprocessor changes.
**                           The signature of that class' primary factory
**                           has changed.
** 053 ECF 20070504   @33409 Fixed load() implementation. We now retry
**                           reposition if it fails, assuming current
**                           result set did not contain target records.
** 054 ECF 20070508   @33459 Fixed sessionClosing() and session listener
**                           registration. Was not properly scoping
**                           registration for scrolling queries.
** 055 ECF 20070511   @33470 Fixed resetResults(). Additional state needed
**                           to be reset to default values.
** 056 ECF 20070517   @33670 Fixed abend in sessionClosing(). Check that
**                           all necessary buffers are still in scope
**                           before attempting to cache preselected
**                           results in a SimpleResults object. This
**                           prevents a fatal error when trying to
**                           retrieve results for a temp table which has
**                           been dropped by the time we get the session
**                           closing notification.
** 057 ECF 20070518   @33683 Changed access for getRecordBuffers(). Made
**                           public (was protected), since this is now a
**                           method of the Joinable interface.
** 058 ECF 20070706   @34402 Fixed selection of scope for SessionListener
**                           registration. Implemented new SessionListener
**                           method deregisteredSessionListener(). Also
**                           replaced StringBuffer with StringBuilder.
** 059 ECF 20070709   @34422 Fixed sessionClosing(). Was unsafe for
**                           certain non-scrolling results which do not
**                           support reset.
** 060 ECF 20070711   @34448 Extracted ScrollingResults inner class into a
**                           separate, top-level class.
** 061 ECF 20070716   @34510 Implement new cleanup() method as required by
**                           P2JQuery interface. Check whether query is
**                           standalone before registering a cleaner with
**                           the TM. This prevents us from cleaning up too
**                           aggressively in case we are wrapped by a
**                           QueryWrapper or contained in a CompoundQuery
**                           that is so wrapped.
** 062 ECF 20070720   @34611 Fixed fetch(). Always reload a DMO, even if
**                           it is in a temp table or has no lock. This is
**                           necessary to handle the case of fetching a
**                           cached DMO which may have been detached from
**                           its session previously. For a persistent DMO,
**                           the additional overhead should be small,
**                           since Hibernate only needs to go to the first
**                           level cache to find the object.
** 063 ECF 20070720   @34621 Implemented a better fix for fetch() than
**                           #062 (@34611). If DMO is a temp table, simply
**                           try to reassociate it with the Hibernate
**                           session, rather than going all the way back
**                           to the database (if DMO was detached). Can't
**                           do this for permanent tables, though.
** 064 ECF 20070821   @34907 Added support for temporary silent error
**                           mode. This allows query substitution parms to
**                           be processed safely during initialization of
**                           a query.
** 065 ECF 20070827   @34996 Optimized certain temp table queries. Only
**                           augment where clause with a temp buffer's
**                           multiplex ID when necessary. Use multiplex ID
**                           as a constant in where clause rather than as
**                           a substitution parameter.
** 066 ECF 20070927   @35276 Fixed reposition logic. Any reposition must
**                           first release all buffers. Changes to methods
**                           to get current row and report whether query
**                           is off-end.
** 067 ECF 20071030   @35634 Fixed fetch() to prevent loading stale temp
**                           table records. It is possible for a record to
**                           be evicted from the session cache while still
**                           held in a result set, such that the result
**                           set will return a stale record that is not a
**                           detached instance, even if the session is
**                           flushed before the fetch. We now force a
**                           reload of fetched temp table records to avoid
**                           this problem.
** 068 ECF 20071130   @36122 Removed isRefreshSnapshot() method. The
**                           RecordBuffer placeholder snapshot is no
**                           longer cleared by any query type when the
**                           query does not find a record to place in the
**                           buffer. Integrated generics.
** 069 ECF 20071206   @36267 Fixed off-end tracking. Replaced boolean flag
**                           with OffEnd enum to discern between off-end
**                           directions (i.e., off back vs off front of
**                           result set). Current off-end status and
**                           active sort index are reported to the record
**                           buffer when fetching a record.
** 070 ECF 20071218   @36463 Fixed order by clause generation. Was failing
**                           for sort criteria which included references
**                           to specific extents of indexed properties.
** 071 ECF 20080120   @37009 Removed restriction of dealing only with IDs.
**                           This query can now handle both Persistables
**                           and primary key IDs when retrieving data.
**                           This is necessary when the full record is
**                           needed immediately, not just an ID, such as
**                           in presort cases. Also added open() method to
**                           set proper semantics for a converted OPEN
**                           QUERY statement, and optionally to reposition
**                           to the first result.
** 072 ECF 20080222   @37144 Fixed NPE in sessionClosing().
** 073 ECF 20080310   @37481 Made changes necessary to support embedded,
**                           H2 database.
** 074 SVL 20080418   @38036 Added support for the inverseSorting flag.
** 075 SVL 20080421   @38080 Added support for legacy key join helpers.
** 076 SVL 20080425   @38119 lastRepo assignment has been fixed into
**                           reposition(int).
** 077 ECF 20080502   @23396 Backed out #076 (@38119). The reposition(int)
**                           implementation was generally correct and this
**                           change caused regressions. Fixed open() to
**                           execute the query immediately instead of
**                           doing this lazily. We now reposition on open
**                           only if the query is being browsed. Removed
**                           obsolete method isRepositionOnOpen().
** 078 SVL 20080508   @38238 Fixed incorrect record values after
**                           first-next, forward/backward(0) and
**                           last-prev execution sequences.
** 079 ECF 20080508   @38243 Fixed regression in reposition(int) which did
**                           not handle all non-positive rows correctly.
**                           Refactored open() method to use parent class.
**                           Override base lock type for temporary tables.
**                           This allows some downstream optimizations.
** 080 ECF 20080509   @38252 Fixed regression in open(). Removed check for
**                           standalone flag.
** 081 SVL 20080526   @38311 Preselect locking implemented.
** 082 SVL 20080604   @38534 Modified according to the new P2JQuery
**                           interface. previous() fixed.
** 083 ECF 20080604   @38608 Added placeholder for dirty database check.
** 084 SVL 20080611   @38685 Behavior of previous() fixed for the case 
**                           when the off-end is FRONT.
** 085 CA  20080618   @38869 When the JOINed foreign buffer does not refer
**                           any record, the query must return no results
**                           as it can not be executed.
** 086 CA  20080619   @38871 If executed within an OPEN QUERY statement, 
**                           the query must be removed from the session's
**                           listeners only when the enclosed external
**                           scope is exit. This fixes the case when the
**                           OPEN QUERY is executed inside a procedure.
** 087 ECF 20080709   @39086 Created cacheResults() method. Abstracted
**                           from sessionClosing() and made protected so
**                           subclasses could access this functionality.
** 088 ECF 20080710   @39090 Modified sessionClosing() and cacheResults().
**                           The latter no longer throws
**                           IllegalStateException.
** 089 ECF 20080714   @39127 Fixed NPE in debug logging.
** 090 ECF 20080723   @39173 Removed retry handling. This is now the
**                           responsibility of subclasses.
** 091 CA  20080729   @39218 Added back the retry handling. Now the retry
**                           will rely on the reposition mechanism to 
**                           retrieve the same record, but only when the
**                           query is associated with a FOR block.
** 092 SVL 20080730   @39237 Component.multiplexID field was removed.
**                           RecordBuffer's multiplex ID is used instead.
** 093 ECF 20080730   @39325 Added rowid support. Refactored reposition*()
**                           methods.
** 094 SVL 20080808   @39348 Fixed cacheResults(). Did not handle case
**                           when query has just executed but has not yet
**                           retrieved any results.
** 095 ECF 20080822   @39556 Fix for join validation. Abstracted check
**                           into separate method, verifyJoins(). Disable
**                           join buffer check for server-based joins.
**                           Other changes in support of global DMO change
**                           events.
** 096 CA  20080819   @39463 Support API change in ScrollingResults.
** 097 CA  20080828   @39610 Fix NPE caused by repositionByID(rowid, ...).
** 098 ECF 20081009   @40081 Added support for inlining query substitution
**                           parameters. Integrated HQLPreprocessor API
**                           changes and the ParameterIndices class.
** 099 ECF 20081030   @40308 Fixed regression introduced by #098 (@40081).
**                           Refactored Component inner class and code in
**                           top-level class which uses it.
** 100 ECF 20081105   @40311 Implemented registerRecordChangeListeners().
**                           Required by the Joinable interface.
** 101 ECF 20081106   @40324 Added _isOffEnd(). Required by P2JQuery
**                           interface change.
** 102 ECF 20081107   @40405 Moved multiplex ID position in order by
**                           clause. It used to be the first element, but
**                           that sometimes resulted in a less efficient
**                           index being selected for a query plan. Now it
**                           is either the last element, or second to last
**                           (if the primary key is part of the clause).
** 103 CA  20081210   @40839 Do not cleanup if the query was explicitly 
**                           closed.
** 104 CA  20081211   @40859 Fixed regression introduced by H103: on 
**                           sessionClosing event, if the query is closed,
**                           return true.
** 105 ECF 20081215   @40914 Removed flushBuffers() method. This work is
**                           now done when a component is added to the
**                           query. The previous invocation, during record
**                           retrieval, was too late. A validation error
**                           ignores silent error mode, since the flushing
**                           of the previous record in the buffer is not
**                           masked by silent error mode.
** 106 ECF 20090202   @41248 Rolled back #081 (@38311). Prelock mechanism
**                           has been disabled.
** 107 ECF 20090313   @41600 Changed createResults(). Added parameter to
**                           FilteredResults constructor.
** 108 SVL 20090316   @41610 assembleOrderByClauseNoPreamble() refactored,
**                           sort variable is protected instead of private 
**                           now.
** 109 GES 20090424   @41941 Import change.
** 110 ECF 20090512   @42153 Modified addComponent(). Now invokes flushAll()
**                           instead of flush() on backing buffer. This
**                           ensures all like buffers are allowed to flush
**                           their transient records properly.
** 111 ECF 20090603   @42599 Fixed fetch(). A dirty record in the backing
**                           buffer must not be replaced by the primary
**                           database's version of that record.
** 112 ECF 20090604   @42606 Added support for Progress isolation leak quirk.
**                           Modified execute() method to publish uncommitted
**                           changes with dirty share manager if necessary.
** 113 ECF 20090609   @42631 Fixed execute(). We now evict, if possible, those
**                           DMOs read into the Hibernate Session, whose
**                           uncommitted changes are leaked to other sessions
**                           early.
** 114 ECF 20090619   @42772 Added executeScroll(). Executes default variant
**                           of Persistence.scroll(String, Object[], Type[]).
**                           Children classes which need to invoke different
**                           variants must override this method.
** 115 ECF 20090630   @42985 Rewrote reposition logic. Moved from integer to
**                           recid type in reposition methods.
** 116 ECF 20090702   @43035 Fixed sessionClosing() and regressions in query
**                           (in the Progress OPEN QUERY sense) behavior.
**                           Added isNativeScrolling() to indicate that a base
**                           PreselectQuery instance used for a converted 4GL
**                           query behaves as if that query were defined as
**                           scrolling. Added error handling in caching
**                           results during a sessionClosing event.
** 117 ECF 20090703   @43051 Reimplemented load() method. This method now sets
**                           the backing buffer to unknown mode and throws
**                           MissingRecordException if a target record cannot
**                           be loaded. Also fixed cleanup(); the query is now
**                           flagged as closed once it runs.
** 118 ECF 20090707   @43101 Fixed defect related to processing reposition
**                           notifications. Make sure we don't release the
**                           buffers' current records and change other state
**                           when processing nested reposition calls during
**                           reposition notifications.
** 119 ECF 20090708   @43107 Partial rollback of #118 (@43101). Some state
**                           must be changed when processing nested reposition
**                           calls during reposition notifications.
** 120 ECF 20090711   @43192 Use new ScrollingResults c'tor.
** 121 ECF 20090716   @43220 Refactored for lazy record buffer initialization.
**                           Initialize backing buffers when their scopes have
**                           been opened. Otherwise allow initialization to be
**                           deferred.
** 122 SVL 20090723   @43344 Added setNonScrolling() function.
** 123 ECF 20090724   @43419 Made getPersistence() protected.
** 124 ECF 20090729   @43448 Fixed executeQuery(). Surrounded creation of
**                           ScrollingResults with push/pop calls for implicit
**                           transaction.
** 125 CA  20090731   @43468 Register the query for off-end listener 
**                           registration after block initialization (done
**                           only for block types FOR and FOR EACH). Added
**                           getOffEndListeners, which returns a list of all
**                           off-end listeners for all used buffers.
** 126 ECF 20090810   @43588 Changed to accommodate SessionListener API
**                           change. sessionClosing() method was replaced with
**                           sessionEvent().
** 127 ECF 20090811   @43600 Fixed resource cleanup. Refactored registration
**                           of a cleaner object with the TM, so that
**                           subclasses could use it. Ensured cleaner is only
**                           registered once.
** 128 SVL 20091030   @44296 Fixed forward(), backward(), currentRow() and next() functions.
** 129 OM  20130829          Added change detection to buffer on current() methods.
**                           Updated calls to match the new Persistence.load() signatures.
** 130 ECF 20131028          Import change.
** 131 CA  20131013          Added no-op deleted() method, required by the changes in Finalizable
**                           interface.
** 132 SVL 20140210          Use HQLExpression instead of HQL string.
** 133 SVL 20140604          Added support for INDEX-INFORMATION.
** 134 ECF 20140814          Replaced RecordBuffer.flushAll with flush in addComponent method.
** 135 SVL 20140909          Fixed null case for peekNext/peekPrevious. Changed some access
**                           modifiers. Added deleteResultListEntry.
** 136 SVL 20141209          Update row count after deleteResultListEntry.
** 137 SVL 20150212          Fixed last().
** 138 SVL 20150331          Added deleteResultListEntry(boolean).
** 139 OM  20150421          Changed the hqlPreprocessor of each components to expires on when
**                           resetArgs() is called based on nullArguments fingerprint.
**                           Replaced Apache logging with J2SE logging.
** 140 ECF 20150801          Implemented new Joinable methods necessary for compound query
**                           optimization. Refactored to remove inner class Component, which is
**                           now the top level class QueryComponent.
** 141 SVL 20150821          isScrolling was made public.
** 142 ECF 20150906          Reimplemented silent error mode processing of query substitution
**                           parameters. Fixed NPE related to outer joins.
** 143 ECF 20151214          Prevent releasing the current buffer record(s) under certain
**                           circumstances, when no record is found by the query.
** 144 EVL 20160217          Clean up comments from symbols invalid for Solaris to compile.
** 145 SVL 20160314          Added support for parameterFilter.
** 146 ECF 20160320          Force full records to be queried on the first pass.
** 147 ECF 20160512          Deregister as SessionListener during cleanup instead of letting TM
**                           call cleaner. Fixed coreFetch issue with null record.
** 148 ECF 20160610          Reverted #146.
** 149 SVL 20160608          The query can handle null IDs (for scrolling compound queries
**                           with OUTER join). Reposition validation moved away.
** 150 ECF 20160614          Added getEntities to collecte entity names associated with this
**                           query; modified executeScroll to pass these to Persistence.scroll.
** 151 SVL 20160721          Proper reset on query reopen.
** 152 OM  20160720          Added support for loading template records.
** 153 ECF 20160725          Fixed buffer lookup during HQL preprocessing.
** 154 ECF 20160823          Improved coreFetch to better detect missing records and avoid
**                           deadlock.
** 155 ECF 20160901          DirtyShareContext.isDirtyDelete signature change.
** 156 GES 20160804          Switched from WhereExpression to lambda usage.
**         20160919          All initialization is now done with a default constructor followed
**                           by a call to initialize(). This enables query instance members in
**                           converted code to be local variables that are effectively final.
**                           This is needed to support lambdas without moving queries to be
**                           instance members. Queries as instance members breaks recursion
**                           use cases.
** 157 SVL 20161007          Fix for the case when REPOSITION-TO-ROWID takes more ids than the
**                           actual query table count.
** 158 ECF 20161014          Fixed implementation of current() method in the event there is no
**                           current record in the buffer to reload, and this is not considered
**                           an error in the context of the query (e.g., query represents an
**                           outer join and record can be null).
** 159 OM  20170118          Fixed javadoc (Solaris compatibility).
**     OM  20170209          Added return value for repositioning methods.
** 160 GES 20171220          Changes to match new silent mode implementation.
** 161 OM  20171129          releaseBuffers() is implemented from P2JQuery.
**     ECF 20171202          Allow HQL to omit DMO alias qualifiers.
** 162 HC  20180130          Fixed an NPE condition in an outer join query where one of the query
**                           components yields no records.
** 163 OM  20180515          Added dynamic filters and dynamic sort. Fixed extra joins generation
**                           in assembleFromClause.
** 164 ECF 20180818          Temp-table fetch optimization.
**     OM  20180901          Improved dynamic filtering.
**     OM  20180924          Filtering on decimal fields uses STRING(f, format) function.
** 165 OM  20181023          Added external buffers when making the adaptive server join
**                           components. Avoided NPE in repositionByID().
** 166 ECF 20190106          Fixed size method to not report a size of 1 for an empty result set.
** 167 ECF 20190318          Refetch cached full records in coreFetch, most of which are likely
**                           detached from the Hibernate session.
** 168 SBI 20190401          Changed coreFetch in order to process the end of query for all query
**                           components.
**     ECF 20190415          Fixed quick-out test condition in cacheResults.
**     ECF 20190427          Fixed results caching regression.
** 169 ECF 20190813          Change to RecordBuffer.release and RecordBuffer.setRecord APIs.
** 170 OM  20190720          Added repositionByID() for internal usage.
**     CA  20190811          A dynamic query can be closed and then re-opened.
**     HC  20190823          Fixed an NPE condition.
** 171 SVL 20191002          Implemented createResultListEntry.
** 172 CA  20191009          Added where and sort clause translation in case of bound buffers.
** 173 SVL 20191218          Fixed caching logic driven by REPOSITION TO ROWID.
** 174 CA  20200110          Fixed finalizable leak when then query is used in a QueryWrapper.
** 175 ECF 20200419          Removed Hibernate dependencies.
** 176 AIL 20200908          Reset offSet flag when adding new row through createResultListEntry.
**     AIL 20200910          Restrict resetting offSet flag only for browsed queries.
**     ECF 20200919          Persistence.load API signature change.
**     OM  20201001          Dropped redundant ORDER BY elements in multi-table queries.
**     OM  20201007          Optimized SortCriterion by using DmoMeta data instead of map lookups.
**                           _multiplex sort component matches the index direction.
**     OM  20201120          Adjusted error messages.
**     CA  20210310          A dynamic predicate must set the default lock to NONE, instead of SHARE.
**     IAS 29210401          Update _UserTableStat "read" counter
**     IAS 20210419          UserTableStat counters' update
**     ECF 20210502          Adjusted initial components array list size.
**     CA  20210527          Removed isOffEnd implementation, as the AbstractQuery.isOffEnd is the correct
**                           one.
**     ECF 20210723          Minor cleanup.
**     OM  20220112          Added implementation of SKIP-DELETED-RECORD attribute.
**     CA  20220117          Fixed close for dynamic PreselectQuery's, the close(boolean) needs to be 
**                           implemented.
**     OM  20220118          The assembleOrderByClause() should not return null value.
**     SVL 20220127          Fixed the case when deleted records were fetched if the cursor contained full
**                           records.
**     CA  20220203          Fixed 'getRow' for SCROLLING mode, in this case check the cursor's row number.
**     OM  20220225          coreFetch() returns true if at least a record of the row was fetched.
**     OM  20220228          Reset and specially mark OUTER-JOIN buffers with no records.
**     SVL 20220412          Fixed a problem repositioning to the first result with next(LockType).
**     IAS 20220513          Fixed repositionByID logic for UNKNOWN value of the first argument.
**     OM  20220530          Fixed flush of nursery (only the affected index).
**     OM  20220701          Flatten calls to initialize().
**     CA  20221006          Added JMX instrumentation for query assemble. Refs #6814
**     CA  20221013          In 'repositionByID', if the row number is after the last row in the ResultSet,
**                           assume an error (query off-end) and do not try to fetch the record.
**     CA  20221109          Implemented runtime for EXCEPT/FIELDS options - only NO-LOCK records are loaded 
**                           as 'partial/incomplete'.   At this time, extent fields are always loaded.
**     CA  20220531          Ensure the sortTranslated flag is set when the sort is translated with the bound 
**                           buffers.
**     CA  20220919          Clean the current results before setting the new results.
**     CA  20221005          Fixed NPE in assembleFromClause.
**     OM  20221103          New class names for FQLPreprocessor, FQLExpression, FQLBundle, and FQLCache.
**     CA  20221130          Refactored the WHERE clause translation (when the bound and definition buffers
**                           are not the same), to be aware of the external buffers (i.e. added for CAN-FIND
**                           sub-select clauses).  The translate will be performed before the query is being
**                           executed.
**     SVL 20230108          Improved performance by replacing some "for-each" loops with indexed "for" loops.
**     SVL 20230109          Reflected PreselectQuery.components() now providing direct access to the
**                           components array in order to improve performance.
**     CA  20230110          Do not create 'whereExprList' until something is added to it.
**     SVL 20230113          Improved performance by replacing some "for-each" loops with indexed "for" loops.
**     CA  20221109          Implemented runtime for EXCEPT/FIELDS options - only NO-LOCK records are loaded 
**                           as 'partial/incomplete'.   At this time, extent fields are always loaded.
** 177 CA  20230227          Fixed where clause translation when the external buffer's binding and definition
**                           does not match.
** 178 CA  20230223          Fixed a bug for EXCEPT field processing in 'assembleSelectClause' - the prop.id
**                           was being used instead of prop.propId, which is not the same, in case of 
**                           denormalized extent. 
** 178 AL2 20230315          Avoid redundant fetch.
**         20230329          Avoid redundant fetch only when locking shouldn't be updated.
** 179 IAS 20230405          Fixed FILL support for recursive DATA-RELATION
**                           Added DATE-RELATION to the 'scroll' parameters
** 180 OM  20230404          The join navigation key may contain null/unknown values.
** 180 RAA 20230406          Added isOrderByRequired() which checks if the ORDER BY clause is really
**                           necessary (will the query retrieve a non-unique record?).
** 181 GBB 20230512          Logging methods replaced by CentralLogger/ConversionStatus.
** 182 SVL 20230516          Support for more granular configuration of dirty sharing functionality.
** 183 DDF 20230627          Replaced static variables of DirtyShareSupport with static method calls.
** 184 AB  20230803          Modified assembleWhereClause(),assembleFromClause() and prepareParameters() to 
**                           support OUTER-JOIN.
**     AB  20230803          Modified assembleWhereClause() and prepareParameters(), renamed assembleFromClause()
**                           to assembleFromAndWhereClaused, in order to support OUTER-JOIN.
**     AB  20230825          Modified coreFetch() to support OUTER-JOIN. Moved the responsibility of 
**                           throwing QueryOffEndException outside for loop.
**     AL2 20230825          Avoid calling errorNotOnFile if the fetched component was outer. 
**     AB  20230830          Modified assembleWhereClause() to assemble "on" or "where" clause, depending
**                           on the parameter "isOn".
** 185 AL2 20231027          Changes signature of preserveBuffersOnEmptyResults to expose the results.
** 186 OM  20231115          Fixed generation of nested multiplexed sub-queries.
** 187 RAA 20231006          Added P2JQueryExecutor layer to the query execution.
**     RAA 20231010          Removed DatabaseStatistics usage.         
** 188 AD  20231108          PreselectQuery now listens for deleted records, being sensitive
**                           to their changes.
**     AD  20231110          Added parameter forceOnlyId for ignoring already existing results,
**                           but use already existing primary keys to get accurate results from the database. 
**     AD  20231113          Deleted shouldRegisterRecordChangeListeners() calls in initialize methods and 
**                           refactored stateChanged() method.
**     AD  20231124          Reset forceOnlyId in cleanup and fix forceOnlyId assignment in stateChanged().
**     AL2 20231127          Rule out registering listeners according to shouldRegisterRecordChangeListeners.
** 189 AL2 20231219          Fixed NPE in PreselectQuery.close(). Dynamic query may not have yet any component.
** 190 ES  20231203          Changed first, last, next, previous, current, unique methods to return boolean,
**                           instead of void. These methods now returns true if operation was succesful, and false 
**                           not and lenientOffEnd is set on true, otherwise it throwd an QQE exception.   
** 191 RAA 20231220          Small performance improvements.
** 192 AB  20231002          Modified load() to handle correctly the OUTER-JOIN with preselect.
** 193 DDF 20240207          When caching results, the Result set of a non scrolling query may have not been
**                           accessed yet.
**     DDF 20240209          Do not set the results when the old and cached results are the same.
** 194 HC  20240222          Enabled JMX on FWD Client.
** 195 CA  20240331          Use ArrayList and Java for loops in query assembler.
** 196 ES  20240404          Persist the record in buffer in case of iteration in FOR EACH with break by statement.
** 197 CA  20240422          Allow the query to be reset to its initialization state, so the same instance can
**                           be used to execute child buffer FILL operations.
** 198 OM  20240430          Extents are handled like scalars. The FQL converter will filter them out if
**                           necessary, and SQLQuery will fetch them from secondary tables at hydration time.
** 199 CA  20240512          'prepareParameters' must resolve 'P2JQuery.ParamResolver' as these are deferred
**                           on each iteration of the outer component.
** 200 SB  20240620          Use the getter value of the DMO attribute to check for null argument when  
**                           composing WHERE clause for joins. Refs #8886.
** 201 SB  20240709          Fix a regression where composing WHERE clause for join with IS NULL would not
**                           remove an unknown/null parameter from the parameter list (see #8938). Refactor
**                           the WHERE clause assembly for joins to use getParameters.
** 202 CA  20240809          Avoid iterator usage, for performance improvement.
** 203 CA  20240809          Skip using JMX timers when JMX_DEBUG flag is not set.
** 204 CA  20230924          Further reduce context-local lookup.
** 205 TJD 20230516          Bringing back support for dirty records leaks to other sessions
**     TJD 20230724          Migrate earlyPublishEntities from List to Set to improve performance
**     TJD 20240912          DirtyShareSupport flags logic dependency update
** 206 ES  20240919          Create Results based on iterating flag in executeQuery.
** 207 AL2 20241002          Force the caching of results only by PK if there are too many to cache. 
** 208 AL2 20241017          Use the provided results implementation instead of the using the one in the
**                           query. This way, AdaptiveQuery.adapter will cache the low-level results instead
**                           of the high-level results. 
** 209 DDF 20230712          Added support for the the FORWARD-ONLY attribute.
**     DDF 20230713          Moved utility methods for the FORWARD-ONLY attribute to AbstractQuery.
**     DDF 20230717          Only allow scrolling when forwardOnly is false.
** 210 CA  20241030          Small improvement for 'getOffEndListeners' to avoid array resize.
** 211 SB  20241105          Release the last bound buffer of a query if REPOSITION-BY-ID has an unknown
**                           argument. Refs #9199.
** 212 LS  20241108          Created assembleComponent() that is used for every component when
**                           assembling the whole select clause.
** 213 OM  20241128          Multi-tenant runtime support: selected the proper persistence context, eventually
**                           based on [sharedDb] parameter.
**                           Implementation of [setMultiTenantAlternative] method.
** 214 AL2 20241209          For assembleSelectClause() do not force onlyPk, retrieve full records when possible.
** 215 ES  20250129          Avoid releasing the buffers in FIRST/LAST component when reaching the QueryOffEnd.
** 216 ICP 20250114          Adjusted assembleWhereClause() to detect if a server side join has a subselect.
** 217 ES  20250218          Make sure to release FIRST/LAST component in case is null and result set of a
**                           outer-join query is available.
** 218 AL2 20250226          Added shouldExecuteForward to encapsulate the decision on using ForwardResults.
** 219 AB2 20250419          Changed cachedResults to retrieve the array of primary keys of the object if 
**                           forceOnlyId is true. See #9722.
** 220 RNC 20250428          Always register the dynamic query as a global scope listener.
** 221 LS  20250416          Small refactoring in assembleComponent() to reuse the size of the array.
** 222 ES  20250508          Synchronize the query buffer if it is AUTO-SYNCHRONIZE attribute is set on true.  
*/

/*
** This program is free software: you can redistribute it and/or modify
** it under the terms of the GNU Affero General Public License as
** published by the Free Software Foundation, either version 3 of the
** License, or (at your option) any later version.
**
** This program is distributed in the hope that it will be useful,
** but WITHOUT ANY WARRANTY; without even the implied warranty of
** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
** GNU Affero General Public License for more details.
**
** You may find a copy of the GNU Affero GPL version 3 at the following
** location: https://www.gnu.org/licenses/agpl-3.0.en.html
** 
** Additional terms under GNU Affero GPL version 3 section 7:
** 
**   Under Section 7 of the GNU Affero GPL version 3, the following additional
**   terms apply to the works covered under the License.  These additional terms
**   are non-permissive additional terms allowed under Section 7 of the GNU
**   Affero GPL version 3 and may not be removed by you.
** 
**   0. Attribution Requirement.
** 
**     You must preserve all legal notices or author attributions in the covered
**     work or Appropriate Legal Notices displayed by works containing the covered
**     work.  You may not remove from the covered work any author or developer
**     credit already included within the covered work.
** 
**   1. No License To Use Trademarks.
** 
**     This license does not grant any license or rights to use the trademarks
**     Golden Code, FWD, any Golden Code or FWD logo, or any other trademarks
**     of Golden Code Development Corporation. You are not authorized to use the
**     name Golden Code, FWD, or the names of any author or contributor, for
**     publicity purposes without written authorization.
** 
**   2. No Misrepresentation of Affiliation.
** 
**     You may not represent yourself as Golden Code Development Corporation or FWD.
** 
**     You may not represent yourself for publicity purposes as associated with
**     Golden Code Development Corporation, FWD, or any author or contributor to
**     the covered work, without written authorization.
** 
**   3. No Misrepresentation of Source or Origin.
** 
**     You may not represent the covered work as solely your work.  All modified
**     versions of the covered work must be marked in a reasonable way to make it
**     clear that the modified work is not originating from Golden Code Development
**     Corporation or FWD.  All modified versions must contain the notices of
**     attribution required in this license.
*/

package com.goldencode.p2j.persist;

import java.io.*;
import java.sql.*;
import java.util.*;
import java.util.function.*;
import java.util.logging.*;
import com.goldencode.p2j.jmx.*;
import com.goldencode.p2j.persist.dirty.*;
import com.goldencode.p2j.persist.event.*;
import com.goldencode.p2j.persist.lock.*;
import com.goldencode.p2j.persist.orm.*;
import com.goldencode.p2j.util.*;
import com.goldencode.p2j.util.logging.CentralLogger;
import com.goldencode.p2j.util.ErrorManager;
import com.goldencode.util.*;

/**
 * Query which issues an FQL query to the database server and uses a
 * scrollable database cursor to navigate over the set of results.  This
 * query is used to back converted Progress queries which either explicitly
 * or implicitly cause a result list to be preselected.  Both single- and
 * multi-table queries are supported.  Because this class implements the
 * <code>Joinable</code> interface, instances may be added as components of
 * a {@link CompoundQuery}.
 * <p>
 * The query is lazily executed the first time a record retrieval/navigation
 * method is invoked.  Only the database identifiers (i.e., primary keys) of
 * the records are retrieved by the initial query.  On each subsequent fetch,
 * a row in this result list is retrieved, where each column contains the
 * primary key for the record associated with the corresponding query
 * component.  The record for each primary key is retrieved and stored in its
 * associated buffer.
 * <p>
 * At this time, outer joins are not supported.  Furthermore, components may
 * only specify NEXT as their iteration type at this time.  FIRST and LAST
 * currently are not supported.
 * <p>
 * When used to represent a Progress query (created with DEFINE or OPEN
 * QUERY), lenient-off-end must be enabled at construction.  This mode
 * suppresses end condition exceptions for <code>next</code> and
 * <code>previous</code> commands which run off the end of the query's
 * results.
 * <p>
 * This query may be integrated with a client-side where clause expression,
 * by passing a non-<code>null</code> <code>Supplier&lt;logical&gt;</code> argument
 * to one of the single-table constructor variants.  Note that additional
 * components may not be added to an instance so created, since this type of
 * query must act against no more than one record buffer.  If a multi-table
 * preselect query is called for, <code>CompoundQuery</code> must be used,
 * though its components may be instances of this class.
 */
public class PreselectQuery
extends AbstractQuery
implements Joinable,
           RecordChangeListener,
           QueryConstants,
           Finalizable,
           SessionListener
{
   /** 
    * A threshold for {@link #cacheResults(Results, boolean)} to decide when we should drop caching full DMO.
    * If this more records are cached, we will cache only the PK for the sake of the session cache performance.
    */
   protected static final int LENIENT_CACHING_THRESHOLD = 64;
   
   /** Empty array instance. */
   private static final ArrayList<SortCriterion> EMPTY_ARRAY = new ArrayList<>();
   
   /** Logger */
   private static final CentralLogger LOG = CentralLogger.get(PreselectQuery.class.getName());
   
   /** Instrumentation for {@link #assembleFQL()}. */
   private static final NanoTimer QUERY_ASSEMBLE = NanoTimer.getInstance(FwdServerJMX.TimeStat.OrmPreselectQueryAssemble);

   /** Has query been repositioned between result rows? */
   protected boolean betweenRows = true;
   
   /** Has the query run off either end of its result set? */
   protected OffEnd offEnd = OffEnd.NONE;
   
   /** FQL order by clause used across all query components */
   protected String sort;
   
   /** Components of the query;  at least 1 is required */
   protected ArrayList<QueryComponent> components;
   
   /**
    * Determines if the current row was deleted by DELETE-RESULT-LIST-ENTRY. Note that if the
    * query is off-end, it also can be "deleted".
    */
   protected boolean currentRowDeleted = false;
   
   /** Flag indicating if the {@link #sort} has been translated. */
   protected boolean sortTranslated = false;
   
   /** List of client-side where clause expressions */
   private List<Supplier<logical>> whereExprList = null;
   
   /** Persistence object which this query's components use */
   private Persistence persistence = null;
   
   /** Flag indicating results must be scrollable */
   private boolean scrolling = false;
   
   /** Cache of primary key lists to result set row numbers */
   private final RepositionCache repoCache = new RepositionCache();
   
   /** FQL query string to retrieve primary key ID arrays */
   private String fql = null;
   
   /** Result set for this query */
   private Results results = null;
   
   /** Does this query have any arguments which are runtime resolvable? */
   private boolean resolvableArgs = false;
   
   /** Number of total results returned */
   private int rowCount = -1;
   
   /** Is query registered for session event notification? */
   private boolean regWithSession = false;
   
   /** Has query been registered for resource cleanup? */
   private boolean regCleaner = false;
   
   /** Determines if this will be unregistered from ChangeBroker on cleanup. */
   protected boolean unregisterOnCleanup = false;
   
   /** Cleaner executed when this query's block is exit. */
   private final Finalizable cleaner = new Finalizable()
   {
      public void finished() { cleanup(); }
      public void iterate() {}
      public void retry() {}
      public void deleted() {}
   };
   
   /** Flag indicating that this instance was registered as a finalizable. */
   private boolean thisFinalizable = false;
   
   /** Force query to retrieve full records, rather than primary keys only */
   private boolean fullRecords = false;
   
   /** Force the query to retrieve only the ids */
   private boolean forceOnlyId = false;
   
   /** The list of sorting criteria added dynamically, at runtime. */
   private List<String> dynamicSorts = null;

   /** Flag indicated if the WHERE clause has been translated. */
   private boolean whereTranslated = false;
   
   /** Test whether the log messages with FINE priority are allowed. */
   private static final boolean LOG_FINE = LOG.isLoggable(Level.FINE);
   
   /**
    * Default constructor. Initialization is deferred until <code>initialize()</code> is
    * called.
    */
   public PreselectQuery()
   {
   }
   
   /**
    * Initializer logic which is typically used for a multi-table query. Details
    * specific to each table component of the query are added via one of the
    * <code>addComponent</code> method variants.
    * <p>
    * This code is intentionally separated from the constructor. The purpose of this separation
    * is to allow construction to occur in the prior scope while initialization can still occur
    * within a method or lambda that is inside the next block's scope.
    *
    * @param    sort
    *           FQL order by clause to be applied across all components of the query.
    *
    * @return   This query instance.
    */ 
   public PreselectQuery initialize(String sort)
   {
      // WARNING: any usage from FWD runtime of this API must give you a already translated sort
      // clause (in case of bound buffers).
      
      // when this is called from converted code, then the sort clause is not translated, and will
      // be translated on query execute!
      this.sort = sort;
      components = new ArrayList<>(3);
    
      TransactionManager.registerOffEndQuery(this);
    
      registerRecordChangeListeners(null);
      return this;
   }
   
   /**
    * Initializer logic which is typically used for a multi-table query. Details
    * specific to each table component of the query are added via one of the
    * <code>addComponent</code> method variants.
    * <p>
    * This variant is used by code which is converted from an OPEN QUERY
    * statement, which permits advancing off the end of a result set without
    * raising a <code>QueryOffEndException</code>.
    * <p>
    * This code is intentionally separated from the constructor. The purpose of this separation
    * is to allow construction to occur in the prior scope while initialization can still occur
    * within a method or lambda that is inside the next block's scope.
    *
    * @param    lenientOffEnd
    *           <code>true</code> if advancing past either end of the result
    *           set should NOT raise a <code>QueryOffEndException</code>;
    *           <code>false</code> if doing so should raise the exception.
    * @param    sort
    *           FQL order by clause to be applied across all components of the query.
    *
    * @return   This query instance.
    */
   public PreselectQuery initialize(boolean lenientOffEnd, String sort)
   {
      // WARNING: any usage from FWD runtime of this API must give you a already translated sort
      // clause (in case of bound buffers).
      
      // when this is called from converted code, then the sort clause is not translated, and will
      // be translated on query open!
      initialize(sort);
      this.lenientOffEnd = lenientOffEnd;
      
      return this;
   }
   
   /**
    * Initializer logic designed for a single-table preselect query.
    * The where clause must expect no substitution parameters.  A share lock
    * will be applied to retrieved records.
    * <p>
    * This code is intentionally separated from the constructor. The purpose of this separation
    * is to allow construction to occur in the prior scope while initialization can still occur
    * within a method or lambda that is inside the next block's scope.
    *
    * @param    dmo
    *           DMO proxy which defines buffer and record type.
    * @param    where
    *           FQL where clause.  May be <code>null</code>.
    * @param    whereExpr
    *           Client-side where clause expression.  May be <code>null</code>.
    * @param    sort
    *           FQL order by clause.
    *
    * @return   This query instance.
    */
   public PreselectQuery initialize(DataModelObject   dmo,
                                    String            where,
                                    Supplier<logical> whereExpr,
                                    String            sort)
   {
      return initialize(dmo, where, whereExpr, sort, null, null, null, null);
   }
   
   /**
    * Initializer logic designed for a single-table preselect query.
    * The where clause must expect no substitution parameters.  A share lock
    * will be applied to retrieved records.
    * <p>
    * This code is intentionally separated from the constructor. The purpose of this separation
    * is to allow construction to occur in the prior scope while initialization can still occur
    * within a method or lambda that is inside the next block's scope.
    *
    * @param    dmo
    *           DMO proxy which defines buffer and record type.
    * @param    where
    *           FQL where clause.  May be <code>null</code>.
    * @param    whereExpr
    *           Client-side where clause expression.  May be <code>null</code>.
    * @param    sort
    *           FQL order by clause.
    * @param    indexInfo
    *           Index information string as it is returned by INDEX-INFORMATION.
    *           May be <code>null</code> if it is not an OPEN QUERY case or if
    *           you don't need debug information about selected indexes.
    *
    * @return   This query instance.
    */
   public PreselectQuery initialize(DataModelObject   dmo,
                                    String            where,
                                    Supplier<logical> whereExpr,
                                    String            sort,
                                    String            indexInfo)
   {
      return initialize(dmo, where, whereExpr, sort, indexInfo, null, null, null);
   }
   
   /**
    * Initializer logic designed for a single-table preselect query.
    * The where clause must expect no substitution parameters.  A share lock
    * will be applied to retrieved records.
    * <p>
    * This code is intentionally separated from the constructor. The purpose of this separation
    * is to allow construction to occur in the prior scope while initialization can still occur
    * within a method or lambda that is inside the next block's scope.
    *
    * @param    dmo
    *           DMO proxy which defines buffer and record type.
    * @param    where
    *           FQL where clause.  May be <code>null</code>.
    * @param    whereExpr
    *           Client-side where clause expression.  May be <code>null</code>.
    * @param    sort
    *           FQL order by clause.
    * @param    inverse
    *           DMO to which this query should join via a foreign relation.
    *           May be <code>null</code>.
    *
    * @return   This query instance.
    */
   public PreselectQuery initialize(DataModelObject   dmo,
                                    String            where,
                                    Supplier<logical> whereExpr,
                                    String            sort,
                                    DataModelObject   inverse)
   {
      return initialize(dmo, where, whereExpr, sort, null, inverse, null, null);
   }
   
   /**
    * Initializer logic designed for a single-table preselect query.
    * The where clause must expect no substitution parameters.  A share lock
    * will be applied to retrieved records.
    * <p>
    * This code is intentionally separated from the constructor. The purpose of this separation
    * is to allow construction to occur in the prior scope while initialization can still occur
    * within a method or lambda that is inside the next block's scope.
    *
    * @param    dmo
    *           DMO proxy which defines buffer and record type.
    * @param    where
    *           FQL where clause.  May be <code>null</code>.
    * @param    whereExpr
    *           Client-side where clause expression.  May be <code>null</code>.
    * @param    sort
    *           FQL order by clause.
    * @param    indexInfo
    *           Index information string as it is returned by INDEX-INFORMATION.
    *           May be <code>null</code> if it is not an OPEN QUERY case or if
    *           you don't need debug information about selected indexes.
    * @param    inverse
    *           DMO to which this query should join via a foreign relation.
    *           May be <code>null</code>.
    *
    * @return   This query instance.
    */
   public PreselectQuery initialize(DataModelObject   dmo,
                                    String            where,
                                    Supplier<logical> whereExpr,
                                    String            sort,
                                    String            indexInfo,
                                    DataModelObject   inverse)
   {
      return initialize(dmo, where, whereExpr, sort, indexInfo, inverse, null, null);
   }
   
   /**
    * Initializer logic designed for a single-table preselect query.
    * A share lock will be applied to retrieved records.
    * <p>
    * This code is intentionally separated from the constructor. The purpose of this separation
    * is to allow construction to occur in the prior scope while initialization can still occur
    * within a method or lambda that is inside the next block's scope.
    *
    * @param    dmo
    *           DMO proxy which defines buffer and record type.
    * @param    where
    *           FQL where clause.  May be <code>null</code>.
    * @param    whereExpr
    *           Client-side where clause expression.  May be <code>null</code>.
    * @param    sort
    *           FQL order by clause.
    * @param    args
    *           Substitution parameters to be inserted into placeholders within
    *           the where clause.
    *
    * @return   This query instance.
    */
   public PreselectQuery initialize(DataModelObject   dmo,
                                    String            where,
                                    Supplier<logical> whereExpr,
                                    String            sort,
                                    Object[]          args)
   {
      return initialize(dmo, where, whereExpr, sort, null, null, args, null);
   }
   
   /**
    * Initializer logic designed for a single-table preselect query.
    * A share lock will be applied to retrieved records.
    * <p>
    * This code is intentionally separated from the constructor. The purpose of this separation
    * is to allow construction to occur in the prior scope while initialization can still occur
    * within a method or lambda that is inside the next block's scope.
    *
    * @param    dmo
    *           DMO proxy which defines buffer and record type.
    * @param    where
    *           FQL where clause.  May be <code>null</code>.
    * @param    whereExpr
    *           Client-side where clause expression.  May be <code>null</code>.
    * @param    sort
    *           FQL order by clause.
    * @param    indexInfo
    *           Index information string as it is returned by INDEX-INFORMATION.
    *           May be <code>null</code> if it is not an OPEN QUERY case or if
    *           you don't need debug information about selected indexes.
    * @param    args
    *           Substitution parameters to be inserted into placeholders within
    *           the where clause.
    *
    * @return   This query instance.
    */
   public PreselectQuery initialize(DataModelObject   dmo,
                                    String            where,
                                    Supplier<logical> whereExpr,
                                    String            sort,
                                    String            indexInfo,
                                    Object[]          args)
   {
      return initialize(dmo, where, whereExpr, sort, indexInfo, null, args, null);
   }
   
   /**
    * Initializer logic designed for a single-table preselect query.
    * A share lock will be applied to retrieved records.
    * <p>
    * This code is intentionally separated from the constructor. The purpose of this separation
    * is to allow construction to occur in the prior scope while initialization can still occur
    * within a method or lambda that is inside the next block's scope.
    *
    * @param    dmo
    *           DMO proxy which defines buffer and record type.
    * @param    where
    *           FQL where clause.  May be <code>null</code>.
    * @param    whereExpr
    *           Client-side where clause expression.  May be <code>null</code>.
    * @param    sort
    *           FQL order by clause.
    * @param    inverse
    *           DMO to which this query should join via a foreign relation.
    *           May be <code>null</code>.
    * @param    args
    *           Substitution parameters to be inserted into placeholders within
    *           the where clause.
    *
    * @return   This query instance.
    */
   public PreselectQuery initialize(DataModelObject   dmo,
                                    String            where,
                                    Supplier<logical> whereExpr,
                                    String            sort,
                                    DataModelObject   inverse,
                                    Object[]          args)
   {
      return initialize(dmo, where, whereExpr, sort, null, inverse, args, null);
   }
   
   /**
    * Initializer logic designed for a single-table preselect query.
    * A share lock will be applied to retrieved records.
    * <p>
    * This code is intentionally separated from the constructor. The purpose of this separation
    * is to allow construction to occur in the prior scope while initialization can still occur
    * within a method or lambda that is inside the next block's scope.
    *
    * @param    dmo
    *           DMO proxy which defines buffer and record type.
    * @param    where
    *           FQL where clause.  May be <code>null</code>.
    * @param    whereExpr
    *           Client-side where clause expression.  May be <code>null</code>.
    * @param    sort
    *           FQL order by clause.
    * @param    indexInfo
    *           Index information string as it is returned by INDEX-INFORMATION.
    *           May be <code>null</code> if it is not an OPEN QUERY case or if
    *           you don't need debug information about selected indexes.
    * @param    inverse
    *           DMO to which this query should join via a foreign relation.
    *           May be <code>null</code>.
    * @param    args
    *           Substitution parameters to be inserted into placeholders within
    *           the where clause.
    *
    * @return   This query instance.
    */
   public PreselectQuery initialize(DataModelObject   dmo,
                                    String            where,
                                    Supplier<logical> whereExpr,
                                    String            sort,
                                    String            indexInfo,
                                    DataModelObject   inverse,
                                    Object[]          args)
   {
      return initialize(dmo, where, whereExpr, sort, indexInfo, inverse, args, null);
   }
   
   /**
    * Initializer logic designed for a single-table preselect query.
    * The where clause must expect no substitution parameters.
    * <p>
    * This code is intentionally separated from the constructor. The purpose of this separation
    * is to allow construction to occur in the prior scope while initialization can still occur
    * within a method or lambda that is inside the next block's scope.
    *
    * @param    dmo
    *           DMO proxy which defines buffer and record type.
    * @param    where
    *           FQL where clause.  May be <code>null</code>.
    * @param    whereExpr
    *           Client-side where clause expression.  May be <code>null</code>.
    * @param    sort
    *           FQL order by clause.
    * @param    lockType
    *           Lock type to apply to records retrieved by this query.
    *
    * @return   This query instance.
    */
   public PreselectQuery initialize(DataModelObject   dmo,
                                    String            where,
                                    Supplier<logical> whereExpr,
                                    String            sort,
                                    LockType          lockType)
   {
      return initialize(dmo, where, whereExpr, sort, null, null, null, lockType);
   }
   
   /**
    * Initializer logic designed for a single-table preselect query.
    * The where clause must expect no substitution parameters.
    * <p>
    * This code is intentionally separated from the constructor. The purpose of this separation
    * is to allow construction to occur in the prior scope while initialization can still occur
    * within a method or lambda that is inside the next block's scope.
    *
    * @param    dmo
    *           DMO proxy which defines buffer and record type.
    * @param    where
    *           FQL where clause.  May be <code>null</code>.
    * @param    whereExpr
    *           Client-side where clause expression.  May be <code>null</code>.
    * @param    sort
    *           FQL order by clause.
    * @param    indexInfo
    *           Index information string as it is returned by INDEX-INFORMATION.
    *           May be <code>null</code> if it is not an OPEN QUERY case or if
    *           you don't need debug information about selected indexes.
    * @param    lockType
    *           Lock type to apply to records retrieved by this query.
    *
    * @return   This query instance.
    */
   public PreselectQuery initialize(DataModelObject   dmo,
                                    String            where,
                                    Supplier<logical> whereExpr,
                                    String            sort,
                                    String            indexInfo,
                                    LockType          lockType)
   {
      return initialize(dmo, where, whereExpr, sort, indexInfo, null, null, lockType);
   }
   
   /**
    * Initializer logic designed for a single-table preselect query.
    * <p>
    * This code is intentionally separated from the constructor. The purpose of this separation
    * is to allow construction to occur in the prior scope while initialization can still occur
    * within a method or lambda that is inside the next block's scope.
    *
    * @param    dmo
    *           DMO proxy which defines buffer and record type.
    * @param    where
    *           FQL where clause.  May be <code>null</code>.
    * @param    whereExpr
    *           Client-side where clause expression.  May be <code>null</code>.
    * @param    sort
    *           FQL order by clause.
    * @param    inverse
    *           DMO to which this query should join via a foreign relation.
    *           May be <code>null</code>.
    * @param    lockType
    *           Lock type to apply to records retrieved by this query.
    *
    * @return   This query instance.
    */
   public PreselectQuery initialize(DataModelObject   dmo,
                                    String            where,
                                    Supplier<logical> whereExpr,
                                    String            sort,
                                    DataModelObject   inverse,
                                    LockType          lockType)
   {
      return initialize(dmo, where, whereExpr, sort, null, inverse, null, lockType);
   }
   
   /**
    * Initializer logic designed for a single-table preselect query.
    * The where clause must expect no substitution parameters.
    * <p>
    * This code is intentionally separated from the constructor. The purpose of this separation
    * is to allow construction to occur in the prior scope while initialization can still occur
    * within a method or lambda that is inside the next block's scope.
    *
    * @param    dmo
    *           DMO proxy which defines buffer and record type.
    * @param    where
    *           FQL where clause.  May be <code>null</code>.
    * @param    whereExpr
    *           Client-side where clause expression.  May be <code>null</code>.
    * @param    sort
    *           FQL order by clause.
    * @param    indexInfo
    *           Index information string as it is returned by INDEX-INFORMATION.
    *           May be <code>null</code> if it is not an OPEN QUERY case or if
    *           you don't need debug information about selected indexes.
    * @param    inverse
    *           DMO to which this query should join via a foreign relation.
    *           May be <code>null</code>.
    * @param    lockType
    *           Lock type to apply to records retrieved by this query.
    *
    * @return   This query instance.
    */
   public PreselectQuery initialize(DataModelObject   dmo,
                                    String            where,
                                    Supplier<logical> whereExpr,
                                    String            sort,
                                    String            indexInfo,
                                    DataModelObject   inverse,
                                    LockType          lockType)
   {
      return initialize(dmo, where, whereExpr, sort, indexInfo, inverse, null, lockType);
   }
   
   /**
    * Initializer logic designed for a single-table preselect query.
    * <p>
    * This code is intentionally separated from the constructor. The purpose of this separation
    * is to allow construction to occur in the prior scope while initialization can still occur
    * within a method or lambda that is inside the next block's scope.
    *
    * @param    dmo
    *           DMO proxy which defines buffer and record type.
    * @param    where
    *           FQL where clause.  May be <code>null</code>.
    * @param    whereExpr
    *           Client-side where clause expression.  May be <code>null</code>.
    * @param    sort
    *           FQL order by clause.
    * @param    args
    *           Substitution parameters to be inserted into placeholders within
    *           the where clause.
    * @param    lockType
    *           Lock type to apply to records retrieved by this query.
    *
    * @return   This query instance.
    */
   public PreselectQuery initialize(DataModelObject   dmo,
                                    String            where,
                                    Supplier<logical> whereExpr,
                                    String            sort,
                                    Object[]          args,
                                    LockType          lockType)
   {
      return initialize(dmo, where, whereExpr, sort, null, null, args, lockType);
   }
   
   /**
    * Initializer logic designed for a single-table preselect query.
    * <p>
    * This code is intentionally separated from the constructor. The purpose of this separation
    * is to allow construction to occur in the prior scope while initialization can still occur
    * within a method or lambda that is inside the next block's scope.
    *
    * @param    dmo
    *           DMO proxy which defines buffer and record type.
    * @param    where
    *           FQL where clause.  May be <code>null</code>.
    * @param    whereExpr
    *           Client-side where clause expression.  May be <code>null</code>.
    * @param    sort
    *           FQL order by clause.
    * @param    indexInfo
    *           Index information string as it is returned by INDEX-INFORMATION.
    *           May be <code>null</code> if it is not an OPEN QUERY case or if
    *           you don't need debug information about selected indexes.
    * @param    args
    *           Substitution parameters to be inserted into placeholders within
    *           the where clause.
    * @param    lockType
    *           Lock type to apply to records retrieved by this query.
    *
    * @return   This query instance.
    */
   public PreselectQuery initialize(DataModelObject   dmo,
                                    String            where,
                                    Supplier<logical> whereExpr,
                                    String            sort,
                                    String            indexInfo,
                                    Object[]          args,
                                    LockType          lockType)
   {
      return initialize(dmo, where, whereExpr, sort, indexInfo, null, args, lockType);
   }
   
   /**
    * Initializer logic designed for a single-table preselect query.
    * <p>
    * This code is intentionally separated from the constructor. The purpose of this separation
    * is to allow construction to occur in the prior scope while initialization can still occur
    * within a method or lambda that is inside the next block's scope.
    *
    * @param    dmo
    *           DMO proxy which defines buffer and record type.
    * @param    where
    *           FQL where clause.  May be <code>null</code>.
    * @param    whereExpr
    *           Client-side where clause expression.  May be <code>null</code>.
    * @param    sort
    *           FQL order by clause.
    * @param    inverse
    *           DMO to which this query should join via a foreign relation.
    *           May be <code>null</code>.
    * @param    args
    *           Substitution parameters to be inserted into placeholders within
    *           the where clause.
    * @param    lockType
    *           Lock type to apply to records retrieved by this query.
    *
    * @return   This query instance.
    */
   public PreselectQuery initialize(DataModelObject   dmo,
                                    String            where,
                                    Supplier<logical> whereExpr,
                                    String            sort,
                                    DataModelObject   inverse,
                                    Object[]          args,
                                    LockType          lockType)
   {
      return initialize(dmo, where, whereExpr, sort, null, inverse, args, lockType);
   }
   
   /**
    * Initializer logic designed for a single-table preselect query.
    * <p>
    * This code is intentionally separated from the constructor. The purpose of this separation
    * is to allow construction to occur in the prior scope while initialization can still occur
    * within a method or lambda that is inside the next block's scope.
    *
    * @param    dmo
    *           DMO proxy which defines buffer and record type.
    * @param    where
    *           FQL where clause.  May be <code>null</code>.
    * @param    whereExpr
    *           Client-side where clause expression.  May be <code>null</code>.
    * @param    sort
    *           FQL order by clause.
    * @param    indexInfo
    *           Index information string as it is returned by INDEX-INFORMATION.
    *           May be <code>null</code> if it is not an OPEN QUERY case or if
    *           you don't need debug information about selected indexes.
    * @param    inverse
    *           DMO to which this query should join via a foreign relation.
    *           May be <code>null</code>.
    * @param    args
    *           Substitution parameters to be inserted into placeholders within
    *           the where clause.
    * @param    lockType
    *           Lock type to apply to records retrieved by this query. If {@code null}, the default lock level
    *           is assumed.
    *
    * @return   This query instance.
    */
   public PreselectQuery initialize(DataModelObject   dmo,
                                    String            where,
                                    Supplier<logical> whereExpr,
                                    String            sort,
                                    String            indexInfo,
                                    DataModelObject   inverse,
                                    Object[]          args,
                                    LockType          lockType)
   {
      if (lockType == null)
      {
         lockType = getDefaultLock();
      }
      
      components = new ArrayList<>(1);
      
      setErrorIfNull(false);
      addComponent(dmo, where, indexInfo, inverse, args, lockType, NEXT, false);
      
      BufferReference proxy = (BufferReference) dmo;
      RecordBuffer buffer = proxy.buffer();
      RecordBuffer defBuffer = proxy.definition();
      
      if (!buffersMatch(buffer, defBuffer))
      {
         sort = translateSort(buffer, defBuffer, sort);
         sortTranslated = true;
      }
      
      this.sort = sort;
      if (whereExpr != null)
      {
         addWhereExpression(whereExpr);
      }
      
      buffer.getTxHelper().registerOffEndQuery(this);
      registerRecordChangeListeners(null);
      return this;
   }
   
   /**
    * Translate the WHERE clause for all {@link #components}.
    */
   @Override
   public void translateWhere()
   {
      if (whereTranslated)
      {
         return;
      }
      
      whereTranslated = true;

      for (int i = 0; i < components.size(); i++)
      {
         QueryComponent qc = components.get(i);
         qc.translateWhere();
      }
   }
   
   /**
    * Set the behavior of this query as a record retrieval request moves off
    * the end of its results.  If set to lenient mode, invocations of
    * <code>next</code> and <code>previous</code> which do not find a record
    * do not raise a <code>QueryOffEndException</code>.  By default, this
    * would normally raise the exception.
    *
    * @param   lenientOffEnd
    *          <code>true</code> to suppress <code>QueryOffEndException</code>
    *          for <code>next</code> and <code>previous</code> requests;
    *          <code>false</code> to permit these exceptions.
    */
   public void setLenientOffEnd(boolean lenientOffEnd)
   {
      this.lenientOffEnd = lenientOffEnd;
   }
   
   /**
    * Add a component to the query.  The following defaults are used:
    * <ul>
    *   <li>no substitution parameters
    *   <li>share lock
    *   <li>inner join
    *   <li><code>NEXT</code> iteration type
    *   <li>no inverse join DMO
    * </ul>
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    */
   public void addComponent(DataModelObject dmo, String where)
   {
      addComponent(dmo, where, (String) null);
   }
   
   /**
    * Add a component to the query.  The following defaults are used:
    * <ul>
    *   <li>no substitution parameters
    *   <li>share lock
    *   <li>inner join
    *   <li><code>NEXT</code> iteration type
    *   <li>no inverse join DMO
    * </ul>
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    * @param   indexInfo
    *          Index information string as it is returned by INDEX-INFORMATION.
    *          May be <code>null</code> if it is not an OPEN QUERY case or if
    *          you don't need debug information about selected indexes.
    */
   public void addComponent(DataModelObject dmo, String where, String indexInfo)
   {
      addComponent(dmo, where, indexInfo, null, null, getDefaultLock(), NEXT, false);
   }
   
   /**
    * Add a component to the query.  The following defaults are used:
    * <ul>
    *   <li>no substitution parameters
    *   <li>share lock
    *   <li>inner join
    *   <li><code>NEXT</code> iteration type
    * </ul>
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    * @param   inverse
    *          DMO to which this query should join via a foreign relation.
    *          May be <code>null</code>.
    */
   public void addComponent(DataModelObject dmo, String where, DataModelObject inverse)
   {
      addComponent(dmo, where, null, inverse);
   }
   
   /**
    * Add a component to the query.  The following defaults are used:
    * <ul>
    *   <li>no substitution parameters
    *   <li>share lock
    *   <li>inner join
    *   <li><code>NEXT</code> iteration type
    * </ul>
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    * @param   indexInfo
    *          Index information string as it is returned by INDEX-INFORMATION.
    *          May be <code>null</code> if it is not an OPEN QUERY case or if
    *          you don't need debug information about selected indexes.
    * @param   inverse
    *          DMO to which this query should join via a foreign relation.
    *          May be <code>null</code>.
    */
   public void addComponent(DataModelObject dmo,
                            String where,
                            String indexInfo,
                            DataModelObject inverse)
   {
      addComponent(dmo, where, indexInfo, inverse, null, getDefaultLock(), NEXT, false);
   }
   
   /**
    * Add a component to the query.  The following defaults are used:
    * <ul>
    *   <li>share lock
    *   <li>inner join
    *   <li><code>NEXT</code> iteration type
    *   <li>no inverse join DMO
    * </ul>
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    * @param   args
    *          Substitution parameters to be inserted into placeholders within
    *          the where clause.
    */
   public void addComponent(DataModelObject dmo, String where, Object[] args)
   {
      addComponent(dmo, where, (String) null, args);
   }
   
   /**
    * Add a component to the query.  The following defaults are used:
    * <ul>
    *   <li>share lock
    *   <li>inner join
    *   <li><code>NEXT</code> iteration type
    *   <li>no inverse join DMO
    * </ul>
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    * @param   indexInfo
    *          Index information string as it is returned by INDEX-INFORMATION.
    *          May be <code>null</code> if it is not an OPEN QUERY case or if
    *          you don't need debug information about selected indexes.
    * @param   args
    *          Substitution parameters to be inserted into placeholders within
    *          the where clause.
    */
   public void addComponent(DataModelObject dmo, String where, String indexInfo, Object[] args)
   {
      addComponent(dmo, where, indexInfo, null, args, getDefaultLock(), NEXT, false);
   }
   
   /**
    * Add a component to the query.  The following defaults are used:
    * <ul>
    *   <li>share lock
    *   <li>inner join
    *   <li><code>NEXT</code> iteration type
    * </ul>
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    * @param   inverse
    *          DMO to which this query should join via a foreign relation.
    *          May be <code>null</code>.
    * @param   args
    *          Substitution parameters to be inserted into placeholders within
    *          the where clause.
    */
   public void addComponent(DataModelObject dmo,
                            String where,
                            DataModelObject inverse,
                            Object[] args)
   {
      addComponent(dmo, where, null, inverse, args);
   }
   
   /**
    * Add a component to the query.  The following defaults are used:
    * <ul>
    *   <li>share lock
    *   <li>inner join
    *   <li><code>NEXT</code> iteration type
    * </ul>
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    * @param   indexInfo
    *          Index information string as it is returned by INDEX-INFORMATION.
    *          May be <code>null</code> if it is not an OPEN QUERY case or if
    *          you don't need debug information about selected indexes.
    * @param   inverse
    *          DMO to which this query should join via a foreign relation.
    *          May be <code>null</code>.
    * @param   args
    *          Substitution parameters to be inserted into placeholders within
    *          the where clause.
    */
   public void addComponent(DataModelObject dmo,
                            String where,
                            String indexInfo,
                            DataModelObject inverse,
                            Object[] args)
   {
      addComponent(dmo, where, indexInfo, inverse, args, getDefaultLock(), NEXT, false);
   }
   
   /**
    * Add a component to the query.  The following defaults are used:
    * <ul>
    *   <li>no substitution parameters
    *   <li>inner join
    *   <li><code>NEXT</code> iteration type
    *   <li>no inverse join DMO
    * </ul>
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    * @param   lockType
    *          Lock type to apply to records retrieved by this component of
    *          the query.
    */
   public void addComponent(DataModelObject dmo, String where, LockType lockType)
   {
      addComponent(dmo, where, (String) null, lockType);
   }
   
   /**
    * Add a component to the query.  The following defaults are used:
    * <ul>
    *   <li>no substitution parameters
    *   <li>inner join
    *   <li><code>NEXT</code> iteration type
    *   <li>no inverse join DMO
    * </ul>
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    * @param   indexInfo
    *          Index information string as it is returned by INDEX-INFORMATION.
    *          May be <code>null</code> if it is not an OPEN QUERY case or if
    *          you don't need debug information about selected indexes.
    * @param   lockType
    *          Lock type to apply to records retrieved by this component of
    *          the query.
    */
   public void addComponent(DataModelObject dmo,
                            String where,
                            String indexInfo,
                            LockType lockType)
   {
      addComponent(dmo, where, indexInfo, null, null, lockType, NEXT, false);
   }
   
   /**
    * Add a component to the query.  The following defaults are used:
    * <ul>
    *   <li>no substitution parameters
    *   <li>inner join
    *   <li><code>NEXT</code> iteration type
    * </ul>
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    * @param   inverse
    *          DMO to which this query should join via a foreign relation.
    *          May be <code>null</code>.
    * @param   lockType
    *          Lock type to apply to records retrieved by this component of
    *          the query.
    */
   public void addComponent(DataModelObject dmo,
                            String where,
                            DataModelObject inverse,
                            LockType lockType)
   {
      addComponent(dmo, where, null, inverse, lockType);
   }
   
   /**
    * Add a component to the query.  The following defaults are used:
    * <ul>
    *   <li>no substitution parameters
    *   <li>inner join
    *   <li><code>NEXT</code> iteration type
    * </ul>
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    * @param   indexInfo
    *          Index information string as it is returned by INDEX-INFORMATION.
    *          May be <code>null</code> if it is not an OPEN QUERY case or if
    *          you don't need debug information about selected indexes.
    * @param   inverse
    *          DMO to which this query should join via a foreign relation.
    *          May be <code>null</code>.
    * @param   lockType
    *          Lock type to apply to records retrieved by this component of
    *          the query.
    */
   public void addComponent(DataModelObject dmo,
                            String where,
                            String indexInfo,
                            DataModelObject inverse,
                            LockType lockType)
   {
      addComponent(dmo, where, indexInfo, inverse, null, lockType, NEXT, false);
   }
   
   /**
    * Add a component to the query.  The following defaults are used:
    * <ul>
    *   <li>inner join
    *   <li><code>NEXT</code> iteration type
    *   <li>no inverse join DMO
    * </ul>
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    * @param   args
    *          Substitution parameters to be inserted into placeholders within
    *          the where clause.
    * @param   lockType
    *          Lock type to apply to records retrieved by this component of
    *          the query.
    */
   public void addComponent(DataModelObject dmo,
                            String where,
                            Object[] args,
                            LockType lockType)
   {
      addComponent(dmo, where, (String) null, args, lockType);
   }
   
   /**
    * Add a component to the query.  The following defaults are used:
    * <ul>
    *   <li>inner join
    *   <li><code>NEXT</code> iteration type
    *   <li>no inverse join DMO
    * </ul>
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    * @param   indexInfo
    *          Index information string as it is returned by INDEX-INFORMATION.
    *          May be <code>null</code> if it is not an OPEN QUERY case or if
    *          you don't need debug information about selected indexes.
    * @param   args
    *          Substitution parameters to be inserted into placeholders within
    *          the where clause.
    * @param   lockType
    *          Lock type to apply to records retrieved by this component of
    *          the query.
    */
   public void addComponent(DataModelObject dmo,
                            String where,
                            String indexInfo,
                            Object[] args,
                            LockType lockType)
   {
      addComponent(dmo, where, indexInfo, null, args, lockType, NEXT, false);
   }
   
   /**
    * Add a component to the query.  The following defaults are used:
    * <ul>
    *   <li>inner join
    *   <li><code>NEXT</code> iteration type
    * </ul>
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    * @param   inverse
    *          DMO to which this query should join via a foreign relation.
    *          May be <code>null</code>.
    * @param   args
    *          Substitution parameters to be inserted into placeholders within
    *          the where clause.
    * @param   lockType
    *          Lock type to apply to records retrieved by this component of
    *          the query.
    */
   public void addComponent(DataModelObject dmo,
                            String where,
                            DataModelObject inverse,
                            Object[] args,
                            LockType lockType)
   {
      addComponent(dmo, where, null, inverse, args, lockType);
   }
   
   /**
    * Add a component to the query.  The following defaults are used:
    * <ul>
    *   <li>inner join
    *   <li><code>NEXT</code> iteration type
    * </ul>
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    * @param   indexInfo
    *          Index information string as it is returned by INDEX-INFORMATION.
    *          May be <code>null</code> if it is not an OPEN QUERY case or if
    *          you don't need debug information about selected indexes.
    * @param   inverse
    *          DMO to which this query should join via a foreign relation.
    *          May be <code>null</code>.
    * @param   args
    *          Substitution parameters to be inserted into placeholders within
    *          the where clause.
    * @param   lockType
    *          Lock type to apply to records retrieved by this component of
    *          the query.
    */
   public void addComponent(DataModelObject dmo,
                            String where,
                            String indexInfo,
                            DataModelObject inverse,
                            Object[] args,
                            LockType lockType)
   {
      addComponent(dmo, where, indexInfo, inverse, args, lockType, NEXT, false);
   }
   
   /**
    * Add a component to the query.  The following defaults are used:
    * <ul>
    *   <li>no inverse join DMO
    * </ul>
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    * @param   args
    *          Substitution parameters to be inserted into placeholders within
    *          the where clause.
    * @param   lockType
    *          Lock type to apply to records retrieved by this component of
    *          the query.
    * @param   iteration
    *          Iteration/retrieval mode for this component of the query.  At
    *          this time, only <code>NEXT</code> is supported.  Future
    *          support for <code>FIRST</code>, <code>LAST</code>, and
    *          <code>UNIQUE</code> is expected.
    * @param   outer
    *          <code>true</code> if an outer join is required.  At this time,
    *          only inner joins are supported.  Future support for outer
    *          joins is expected.
    *
    * @throws  UnsupportedOperationException
    *          if <code>outer</code> is <code>true</code> (temporary);
    *          if <code>iteration</code> is not <code>NEXT</code> (temporary);
    *          if <code>dmo</code>'s associated database is different than
    *          that of a previously added component's database (cross-database
    *          preselect joins are not possible).
    */
   public void addComponent(DataModelObject dmo,
                            String where,
                            Object[] args,
                            LockType lockType,
                            int iteration,
                            boolean outer)
   {
      addComponent(dmo, where, (String) null, args, lockType, iteration, outer);
   }
   
   /**
    * Add a component to the query.  The following defaults are used:
    * <ul>
    *   <li>no inverse join DMO
    * </ul>
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    * @param   indexInfo
    *          Index information string as it is returned by INDEX-INFORMATION.
    *          May be <code>null</code> if it is not an OPEN QUERY case or if
    *          you don't need debug information about selected indexes.
    * @param   args
    *          Substitution parameters to be inserted into placeholders within
    *          the where clause.
    * @param   lockType
    *          Lock type to apply to records retrieved by this component of
    *          the query.
    * @param   iteration
    *          Iteration/retrieval mode for this component of the query.  At
    *          this time, only <code>NEXT</code> is supported.  Future
    *          support for <code>FIRST</code>, <code>LAST</code>, and
    *          <code>UNIQUE</code> is expected.
    * @param   outer
    *          <code>true</code> if an outer join is required.  At this time,
    *          only inner joins are supported.  Future support for outer
    *          joins is expected.
    *
    * @throws  UnsupportedOperationException
    *          if <code>outer</code> is <code>true</code> (temporary);
    *          if <code>iteration</code> is not <code>NEXT</code> (temporary);
    *          if <code>dmo</code>'s associated database is different than
    *          that of a previously added component's database (cross-database
    *          preselect joins are not possible).
    */
   public void addComponent(DataModelObject dmo,
                            String where,
                            String indexInfo,
                            Object[] args,
                            LockType lockType,
                            int iteration,
                            boolean outer)
   {
      BufferReference proxy = (BufferReference) dmo;
      RecordBuffer buffer = proxy.buffer();
      RecordBuffer defBuffer = proxy.definition();
      
      Supplier<String> translateWhere = null;
      if (where != null && (!buffersMatch(buffer, defBuffer) || anyBoundBufferInComponent()))
      {
         translateWhere = () ->
         {
            List<RecordBuffer> binding = new ArrayList<>();
            List<RecordBuffer> definition = new ArrayList<>();
            binding.add(buffer);
            definition.add(defBuffer);
            for (int i = 0; i < components.size(); i++)
            {
               QueryComponent qc = components.get(i);
               if (qc.getBuffer() != qc.getDefinitionBuffer())
               {
                  binding.add(qc.getBuffer());
                  definition.add(qc.getDefinitionBuffer());
               }
            }
   
            return translateWhere(binding, definition, where);
         };
      }

      QueryComponent comp = new QueryComponent(this,
                                               (BufferReference) dmo,
                                               null,
                                               where,
                                               translateWhere,
                                               indexInfo,
                                               lockType,
                                               args,
                                               iteration,
                                               outer);
      addComponent(comp);
   }
   
   /**
    * Add a component to the query.
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    * @param   inverse
    *          DMO to which this query should join via a foreign relation.
    *          May be <code>null</code>.
    * @param   args
    *          Substitution parameters to be inserted into placeholders within
    *          the where clause.
    * @param   lockType
    *          Lock type to apply to records retrieved by this component of
    *          the query.
    * @param   iteration
    *          Iteration/retrieval mode for this component of the query.  At
    *          this time, only <code>NEXT</code> is supported.  Future
    *          support for <code>FIRST</code>, <code>LAST</code>, and
    *          <code>UNIQUE</code> is expected.
    * @param   outer
    *          <code>true</code> if an outer join is required.  At this time,
    *          only inner joins are supported.  Future support for outer
    *          joins is expected.
    *
    * @throws  UnsupportedOperationException
    *          if <code>outer</code> is <code>true</code> (temporary);
    *          if <code>iteration</code> is not <code>NEXT</code> (temporary);
    *          if <code>dmo</code>'s associated database is different than
    *          that of a previously added component's database (cross-database
    *          preselect joins are not possible).
    */
   public void addComponent(DataModelObject dmo,
                            String where,
                            DataModelObject inverse,
                            Object[] args,
                            LockType lockType,
                            int iteration,
                            boolean outer)
   {
      addComponent(dmo, where, null, inverse, args, lockType, iteration, outer);
   }
   
   /**
    * Add a component to the query.
    *
    * @param   dmo
    *          DMO proxy which defines buffer and record type.
    * @param   where
    *          FQL where clause.  May be <code>null</code>.
    * @param   indexInfo
    *          Index information string as it is returned by INDEX-INFORMATION.
    *          May be <code>null</code> if it is not an OPEN QUERY case or if
    *          you don't need debug information about selected indexes.
    * @param   inverse
    *          DMO to which this query should join via a foreign relation.
    *          May be <code>null</code>.
    * @param   args
    *          Substitution parameters to be inserted into placeholders within
    *          the where clause.
    * @param   lockType
    *          Lock type to apply to records retrieved by this component of
    *          the query.
    * @param   iteration
    *          Iteration/retrieval mode for this component of the query.  At
    *          this time, only <code>NEXT</code> is supported.  Future
    *          support for <code>FIRST</code>, <code>LAST</code>, and
    *          <code>UNIQUE</code> is expected.
    * @param   outer
    *          <code>true</code> if an outer join is required.  At this time,
    *          only inner joins are supported.  Future support for outer
    *          joins is expected.
    *
    * @throws  UnsupportedOperationException
    *          if <code>outer</code> is <code>true</code> (temporary);
    *          if <code>iteration</code> is not <code>NEXT</code> (temporary);
    *          if <code>dmo</code>'s associated database is different than
    *          that of a previously added component's database (cross-database
    *          preselect joins are not possible).
    */
   public void addComponent(DataModelObject dmo,
                            String where,
                            String indexInfo,
                            DataModelObject inverse,
                            Object[] args,
                            LockType lockType,
                            int iteration,
                            boolean outer)
   {
      BufferReference proxy = (BufferReference) dmo;
      RecordBuffer buffer = proxy.buffer();
      RecordBuffer defBuffer = proxy.definition();
      
      Supplier<String> translateWhere = null;
      if (where != null && (!buffersMatch(buffer, defBuffer) || anyBoundBufferInComponent()))
      {
         translateWhere = () ->
         {
            List<RecordBuffer> binding = new ArrayList<>();
            List<RecordBuffer> definition = new ArrayList<>();
            binding.add(buffer);
            definition.add(defBuffer);
            for (int i = 0; i < components.size(); i++)
            {
               QueryComponent qc = components.get(i);
               if (qc.getBuffer() != qc.getDefinitionBuffer())
               {
                  binding.add(qc.getBuffer());
                  definition.add(qc.getDefinitionBuffer());
               }
            }
            return translateWhere(binding, definition, where);
         };
      }
      
      AbstractJoin join = processJoin(proxy, (BufferReference) inverse);
      
      QueryComponent comp = new QueryComponent(this,
                                               proxy,
                                               join,
                                               where,
                                               translateWhere,
                                               indexInfo,
                                               lockType,
                                               args,
                                               iteration,
                                               outer);
      addComponent(comp);
   }
   
   /**
    * Get the external buffers, if any, associated with this query.
    *
    * @return  Array of external buffers or {@code null}.
    */
   @Override
   public RecordBuffer[] getExternalBuffers()
   {
      return super.getExternalBuffers();
   }
   
   /**
    * Check if this type of query should register record change listeners. Use this 
    * to avoid computing any scope for registration and end up with a no registration
    * behavior. 
    * 
    * @return   {@code true} if we should invest time into computing the scope in order
    *           to call {@link #registerRecordChangeListeners} or {@code false} if we
    *           can skip doing any work to honor {@link #registerRecordChangeListeners}.
    */
   @Override
   public boolean shouldRegisterRecordChangeListeners()
   {
      return true;
   }
   
   /**
    * Register all <code>RecordChangeListener</code>s associated with this
    * query (if any), with the context-local <code>ChangeBroker</code> at the
    * indicated scope.
    * <p>
    * This implementation does nothing.
    * 
    * @param   scope
    *          Scope at which listeners should be registered with the change
    *          broker.
    */
   @Override
   public void registerRecordChangeListeners(Integer scope)
   {
      if (!shouldRegisterRecordChangeListeners())
      {
         return;
      }
      
      if (scope == null)
      {
         ChangeBroker.get().addListener(this);
         return;
      }
      
      ChangeBroker.get().addListener(this, scope);
      
      // if these were added as global listeners in ChangeBroker remember to remove them
      // when the query is destroyed. Otherwise, the ChangeBroker will drop them automatically
      // when the respective scope is finished.
      if (scope == 0)
      {
         unregisterOnCleanup = true;
      }
   }
   
   /**
    * Report the record buffers for whose changes this object is interested
    * in listening.  This listener will be registered with the {@link
    * ChangeBroker} to receive notifications of any change to DMOs whose
    * interface types match those returned by the {@link
    * RecordBuffer#getDMOInterface} methods of the returned buffers.
    *
    * @return  An iterator on the record buffers stored in each of this
    *          query's components.
    */
   @Override
   public Iterator<RecordBuffer> recordBuffers()
   {
      return Arrays.asList(getRecordBuffers()).iterator();
   }

   /**
    * Respond to a {@code RecordChangeEvent}.
    * 
    * @param   event
    *          Event which describes the DMO state change.
    */
   @Override
   public void stateChanged(RecordChangeEvent event) 
   throws PersistenceException
   {
      if (event.isDelete() && (event.getDMO() != null))
      {
         forceOnlyId = true;
      }
   }
   
   /**
    * Respond to a clean (i.e., transaction commit) session event by caching
    * all preselected results and using that cache to back the query's results
    * for all future fetches.  If all results already are cached, do nothing.
    * <p>
    * Respond to a session closing event by clearing all results, unless they
    * were just cached as described above.
    *
    * @param   event
    *          Session event type.
    *
    * @return  <code>true</code>, indicating this query should be
    *          deregistered as a session listener after this method returns;
    *          else <code>false</code>.
    * 
    * @throws  PersistenceException
    *          if there is an error caching results.
    */
   public boolean sessionEvent(SessionListener.Event event)
   throws PersistenceException
   {
      if (closed)
      {
         return true;
      }
      
      switch (event)
      {
         case TRANSACTION_COMMITTING:
            if (results != null && !results.isResultSetCached())
            {
               try
               {
                  Results cached = cacheResults(results, true);
                  if (results != cached)
                  {
                     results.sessionClosing();
                     setResults(cached);
                  }
               }
               catch (Exception exc)
               {
                  resetResults();
                  
                  throw new PersistenceException("Error caching results", exc);
               }
            }
            break;
            
         case SESSION_CLOSING_NORMALLY:
            if (results != null && !(results instanceof SimpleResults))
            {
               resetResults();
            }
            break;
            
         case SESSION_CLOSING_WITH_ERROR:
            resetResults();
            break;
            
         default:
            return false;
      }
      
      return true;
   }
   
   /**
    * Invoked when this session listener is removed from the list of
    * registered session listeners.
    */
   public void deregisteredSessionListener()
   {
      regWithSession = false;
   }
   
   /**
    * Provides a notification that the scope in which this query was created
    * is ending.  This is a no-op implementation required by the interface.
    */
   public void finished()
   {
   }
   
   /**
    * Provides a notification that the external program scope in which the object is registered is
    * is being deleted and the object's reference will be lost after this method is called.
    * <p>
    * This is a no-op implementation required by the interface.
    */
   public void deleted()
   {
   }
   
   /**
    * Provides a notification that the block whose scope in which the object
    * is registered is about to iterate and attempt another pass.  This is a
    * no-op implementation required by the interface.
    */
   public void iterate()
   {
   }
   
   /**
    * Provides a notification that the block whose scope in which the object
    * is registered is about to retry the same iteration (all loop control
    * data is unchanged).  This implementation sets the state of the query as
    * if a reposition to the current row has just occurred.  This will keep
    * the subsequent call to {@link #next()} at the top of the block body from
    * advancing to the next row.
    * <p>
    * This implementation retries the current record only if the query is 
    * associated with a FOR block (see {@link BlockType}).
    */
   public void retry()
   {
      BlockType type = TransactionManager.getBlockType();
      
      // only queries associated with a FOR block allows the retry of the 
      // current record
      switch (type)
      {
         case FOR_LOOP:
         case FOR_LOOP_WHILE:
         case FOR_LOOP_TO:
         case FOR_LOOP_TO_WHILE:
         case FOR_BLOCK:
         case FOR_BLOCK_WHILE:
         case FOR_BLOCK_TO:
         case FOR_BLOCK_TO_WHILE:
            betweenRows = true;
            break;
      }
      
      if (LOG_FINE)
      {
         LOG.log(Level.FINE, "Preparing query for retry [" + this + "]");
      }
   }
   
   /**
    * Create a string representation of this object's state, primarily for debug purposes.
    * 
    * @return  String representation of this object.
    */
   public String toString()
   {
      return "[" + getTableCount() + " table(s)] FQL: " + fql;
   }
   
   /**
    * Add a client-side where expression to this query.  Each initial query
    * result found by the server must be accepted by this expression before
    * becoming a part of the query's final result set.
    *
    * @param   whereExpr
    *          Client-side where clause expression.  May not be
    *          <code>null</code>.
    *
    * @throws  NullPointerException
    *          if <code>whereExpr</code> is <code>null</code>.
    */
   public void addWhereExpression(Supplier<logical> whereExpr)
   {
      if (whereExpr == null)
      {
         throw new NullPointerException("Client-side where expression may not be null");
      }
      
      if (whereExprList == null)
      {
         whereExprList = new ArrayList<>(1);
      }
      
      whereExprList.add(whereExpr);
   }
   
   /**
    * Set the iteration type for the most recently added query component.
    *
    * @param   iteration
    *          Retrieval mode from {@link QueryConstants}. Must not be FIRST or LAST (use
    *          {@link #setIterationType(int, String)} instead for these types.
    *
    * @throws  IllegalStateException
    *          If no component has yet been added.
    * @throws  IllegalArgumentException
    *          If <code>iteration</code> type is <code>FIRST</code> or <code>LAST</code>.
    */
   public void setIterationType(int iteration)
   {
      setIterationType(iteration, null);
   }
   
   /**
    * Set the iteration type for the most recently added query component.
    *
    * @param   iteration
    *          Retrieval mode from {@link QueryConstants}.
    * @param   sort
    *          Order by clause which should be associated with the most recently added query
    *          component. Must be non-<code>null</code> for iteration types <code>FIRST</code>
    *          and <code>LAST</code>.
    *
    * @throws  IllegalStateException
    *          If no component has yet been added.
    * @throws  IllegalArgumentException
    *          If <code>iteration</code> type is <code>FIRST</code> or <code>LAST</code>, but
    *          <code>sort</code> is <code>null</code>, or if <code>sort</code> represents an
    *          invalid FQL order by expression.
    */
   public void setIterationType(int iteration, String sort)
   {
      if (components.isEmpty())
      {
         throw new IllegalStateException(
            "Must add a query component before setting its iteration type");
      }
      
      QueryComponent comp = components.get(components.size() - 1);
      
      try
      {
         comp.setIteration(iteration, sort);
      }
      catch (PersistenceException exc)
      {
         // should only happen if we have a runtime programming error or conversion error
         // (or with bad, hand-written business logic)
         throw new IllegalArgumentException(exc);
      }
   }
   
   /**
    * Retrieve the first composite row of results for the query.  The
    * underlying buffer for each query component is updated with the
    * appropriate record.
    * <p>
    * If the query involves outer joins, some of the underlying buffers may
    * be empty when this method returns.
    *
    * @return  <code>true</code> if query could retrieve the first record sucefully;
    *          <code>false</code> if unsuccessful and <code>lenientOffEnd</code> is set on true.
    *
    * @throws  ErrorConditionException
    *          if the query or the fetch of any record fails, or if no records
    *          are available and the query is set to fail in this case.
    * @throws  QueryOffEndException
    *          if no records are available and the query is set not to fail
    *          in this case.
    * @throws  IllegalStateException
    *          if no query component has been added to this query.
    */
   public boolean first()
   {
      return first(null);
   }
   
   /**
    * Retrieve the first composite row of results for the query.  The
    * underlying buffer for each query component is updated with the
    * appropriate record.
    * <p>
    * If the query involves outer joins, some of the underlying buffers may
    * be empty when this method returns.
    *
    * @param   lockType
    *          Lock type to apply to records retrieved (overrides default
    *          lock type set for each query component).
    *          
    * @return  <code>true</code> if query could retrive the first record succefully;
    *          <code>false</code> if unsuccessful and <code>lenientOffEnd</code> is set on true.          
    *
    * @throws  ErrorConditionException
    *          if the query or the fetch of any record fails, or if no records
    *          are available and the query is set to fail in this case.
    * @throws  QueryOffEndException
    *          if no records are available and the query is set not to fail
    *          in this case.
    * @throws  IllegalStateException
    *          if no query component has been added to this query.
    */
   public boolean first(LockType lockType)
   {
      execute();
      
      boolean missing = false;
      boolean avail = false;
      do
      {
         try
         {
            avail = (missing ? results.next() : results.first());
            missing = false;
            
            offEnd = (avail ? OffEnd.NONE : OffEnd.BACK);
            betweenRows = false;
            fetch(avail, lockType, isErrorIfNull());
         }
         catch (MissingRecordException exc)
         {
            missing = true;
         }
      } while (missing);
      
      return avail;
   }
   
   /**
    * Retrieve the last composite row of results for the query.  The
    * underlying buffer for each query component is updated with the
    * appropriate record.
    * <p>
    * If the query involves outer joins, some of the underlying buffers may
    * be empty when this method returns.
    *
    *
    * @return  <code>true</code> if query could retrive the last record succefully;
    *          <code>false</code> if unsuccessful and <code>lenientOffEnd</code> is set on true. 
    *
    * @throws  ErrorConditionException
    *          if the query or the fetch of any record fails, or if no records
    *          are available and the query is set to fail in this case.
    * @throws  QueryOffEndException
    *          if no records are available and the query is set not to fail
    *          in this case.
    * @throws  IllegalStateException
    *          if no query component has been added to this query.
    */
   public boolean last()
   {
      return last(null);
   }
   
   /**
    * Retrieve the last composite row of results for the query.  The
    * underlying buffer for each query component is updated with the
    * appropriate record.
    * <p>
    * If the query involves outer joins, some of the underlying buffers may
    * be empty when this method returns.
    *
    * @param   lockType
    *          Lock type to apply to records retrieved (overrides default
    *          lock type set for each query component).
    *
    * @return  <code>true</code> if query could retrive the last record succefully;
    *          <code>false</code> if unsuccessful and <code>lenientOffEnd</code> is set on true. 
    *
    * @throws  ErrorConditionException
    *          if the query or the fetch of any record fails, or if no records
    *          are available and the query is set to fail in this case.
    * @throws  QueryOffEndException
    *          if no records are available and the query is set not to fail
    *          in this case.
    * @throws  IllegalStateException
    *          if no query component has been added to this query.
    */
   public boolean last(LockType lockType)
   {
      execute();
      
      boolean missing = false;
      boolean avail = false;
      do
      {
         try
         {
            avail = (missing ? results.previous() : results.last());
            missing = false;

            offEnd = (avail ? OffEnd.NONE : OffEnd.BACK);
            betweenRows = false;
            fetch(avail, lockType, isErrorIfNull());
         }
         catch (MissingRecordException exc)
         {
            missing = true;
         }
      } while (missing);
      
      return avail;
   }
   
   /**
    * Retrieve the next composite row of results for the query.  The
    * underlying buffer for each query component is updated with the
    * appropriate record.
    * <p>
    * If the query involves outer joins, some of the underlying buffers may
    * be empty when this method returns.
    *
    * @return  <code>true</code> if query could retrive the next record succefully;
    *          <code>false</code> if unsuccessful and <code>lenientOffEnd</code> is set on true. 
    *
    * @throws  ErrorConditionException
    *          if the query or the fetch of any record fails, or if no records
    *          are available and the query is set to fail in this case.
    * @throws  QueryOffEndException
    *          if no records are available and the query is set not to fail
    *          in this case.
    * @throws  IllegalStateException
    *          if no query component has been added to this query.
    */
   public boolean next()
   {
      return next(null);
   }
   
   /**
    * Retrieve the next composite row of results for the query.  The
    * underlying buffer for each query component is updated with the
    * appropriate record.
    * <p>
    * If the query involves outer joins, some of the underlying buffers may
    * be empty when this method returns.
    *
    * @param   lockType
    *          Lock type to apply to records retrieved (overrides default
    *          lock type set for each query component).
    *
    * @return  <code>true</code> if query could retrive the next record succefully;
    *          <code>false</code> if unsuccessful and <code>lenientOffEnd</code> is set on true. 
    *
    * @throws  ErrorConditionException
    *          if the query or the fetch of any record fails, or if no records
    *          are available and the query is set to fail in this case.
    * @throws  QueryOffEndException
    *          if no records are available and the query is set not to fail
    *          in this case.
    * @throws  IllegalStateException
    *          if no query component has been added to this query.
    */
   public boolean next(LockType lockType)
   {
      execute();
      
      boolean missing;
      boolean avail = false;
      do
      {
         missing = false;
         try
         {
            avail = false;
            if (offEnd == OffEnd.FRONT)
            {
               avail = results.first();
            }
            else
            {
               avail = (betweenRows && results.getRowNumber() != -1) || results.next();
            }
            
            offEnd = (avail ? OffEnd.NONE : OffEnd.BACK);
            fetch(avail, lockType, false);
         }
         catch (MissingRecordException exc)
         {
            missing = true;
         }
         finally
         {
            betweenRows = false;
         }
      } while (missing);
      
      return avail;
   }
   
   /**
    * Retrieve the previous composite row of results for the query.  The
    * underlying buffer for each query component is updated with the
    * appropriate record.
    * <p>
    * If the query involves outer joins, some of the underlying buffers may
    * be empty when this method returns.
    *
    * @return  <code>true</code> if query could retrive the previous record succefully;
    *          <code>false</code> if unsuccessful and <code>lenientOffEnd</code> is set on true. 
    *
    * @throws  ErrorConditionException
    *          if the query or the fetch of any record fails, or if no records
    *          are available and the query is set to fail in this case.
    * @throws  QueryOffEndException
    *          if no records are available and the query is set not to fail
    *          in this case.
    * @throws  IllegalStateException
    *          if no query component has been added to this query.
    */
   public boolean previous()
   {
      return previous(null);
   }
   
   /**
    * Retrieve the previous composite row of results for the query.  The
    * underlying buffer for each query component is updated with the
    * appropriate record.
    * <p>
    * If the query involves outer joins, some of the underlying buffers may
    * be empty when this method returns.
    *
    * @param   lockType
    *          Lock type to apply to records retrieved (overrides default
    *          lock type set for each query component).
    *
    * @return  <code>true</code> if query could retrive the previous record succefully;
    *          <code>false</code> if unsuccessful and <code>lenientOffEnd</code> is set on true.
    *
    * @throws  ErrorConditionException
    *          if the query or the fetch of any record fails, or if no records
    *          are available and the query is set to fail in this case.
    * @throws  QueryOffEndException
    *          if no records are available and the query is set not to fail
    *          in this case.
    * @throws  IllegalStateException
    *          if no query component has been added to this query.
    */
   public boolean previous(LockType lockType)
   {
      boolean setToLast = treatPreviousAsLast();
      execute();
      
      boolean missing;
      boolean avail = false;
      do
      {
         missing = false;
         try
         {
            avail = (setToLast ? results.last() : results.previous());
            if (avail)
            {
               offEnd = OffEnd.NONE;
            }
            else
            {
               offEnd = (setToLast ? OffEnd.BACK : OffEnd.FRONT);
            }
            fetch(avail, lockType, false);
         }
         catch (MissingRecordException exc)
         {
            missing = true;
         }
         finally
         {
            betweenRows = false;
         }
      } while (missing);
      
      return avail;
   }
   
   /**
    * Retrieve a unique record (only valid for a single-table query).
    *
    * @return  <code>true</code> if query could retrive the unique record succefully;
    *          <code>false</code> if unsuccessful and <code>lenientOffEnd</code> is set on true.
    *
    * @throws  ErrorConditionException
    *          if the query fails or if more than one result is found which
    *          matches the query criteria.
    * @throws  UnsupportedOperationException
    *          if this method is invoked on a multi-table query.
    * @throws  IllegalStateException
    *          if no query component has been added to this query.
    */
   public boolean unique()
   {
      return unique(null);
   }
   
   /**
    * Retrieve a unique record (only valid for a single-table query),
    * overriding the default lock type of the underlying query component.
    *
    * @param   lockType
    *          Lock type which should override that of the query component.
    *
    * @return  <code>true</code> if query could retrive the unique record succefully;
    *          <code>false</code> if unsuccessful and <code>lenientOffEnd</code> is set on true.
    *
    * @throws  ErrorConditionException
    *          if the query fails or if more than one result is found which
    *          matches the query criteria.
    * @throws  UnsupportedOperationException
    *          if this method is invoked on a multi-table query.
    * @throws  IllegalStateException
    *          if no query component has been added to this query.
    */
   public boolean unique(LockType lockType)
   {
      if (components.size() > 1)
      {
         throw new UnsupportedOperationException(
            "Unique retrieval only valid for a single-table preselect query");
      }
      
      boolean setToFirst = (results == null);
      execute();
      boolean avail = true;
      if (setToFirst)
      {
         avail = results.first();
      }
      
      if (!results.isFirst() || !results.isLast())
      {
         try
         {
            QueryComponent comp = components.get(0);
            RecordBuffer buffer = comp.getBuffer();
            getPersistence().uniqueResultViolation(buffer, null);
         }
         catch (PersistenceException exc)
         {
            ErrorManager.recordOrThrowError(exc);
            
            return false;
         }
      }
      
      offEnd = (avail ? OffEnd.NONE : OffEnd.BACK);
      
      try
      {
         fetch(avail, lockType, false);
      }
      catch (MissingRecordException exc)
      {
         ErrorManager.recordOrThrowError(exc);
      }
      
      return avail;
   }
   
   /**
    * Reload the current composite row of results for the query.
    * <p>
    * If the query involves outer joins, some of the underlying buffers may
    * be empty when this method returns.
    *
    * @return  <code>true</code> if query could retrive the current record succefully;
    *          <code>false</code> if unsuccessful and <code>lenientOffEnd</code> is set on true.
    *
    * @throws  ErrorConditionException
    *          if the reload triggers an error.
    * @throws  IllegalStateException
    *          if no query component has been added to this query.
    */
   public boolean current()
   {
      return current(null);
   }
   
   /**
    * Reload the current composite row of results for the query, overriding
    * the lock type to apply to each record.
    * <p>
    * If the query involves outer joins, some of the underlying buffers may
    * be empty when this method returns.
    *
    * @param   lockType
    *          Lock type which should override that of any query component.
    *
    * @return  <code>true</code> if query could retrive the current record succefully;
    *          <code>false</code> if unsuccessful and <code>lenientOffEnd</code> is set on true.
    *
    * @throws  ErrorConditionException
    *          if the reload triggers an error.
    * @throws  IllegalStateException
    *          if no query component has been added to this query.
    */
   public boolean current(LockType lockType)
   {
      if (_isOffEnd() || currentRowDeleted)
      {
         ErrorManager.displayError(4114, "No query record is available", false);
         return false;
      }
      
      execute();

      ArrayList<QueryComponent> components = components();
      for (int i = 0; i < components.size(); i++)
      {
         QueryComponent comp = components.get(i);
         LockType lt = (lockType == null ? comp.getLockType() : lockType);
         
         final RecordBuffer buffer = comp.getBuffer();
         buffer.armCurrentChanged();
         buffer.reload(lt, isErrorIfNull());
         buffer.updateCurrentChanged();
      }
      
      return true;
   }
   
   /**
    * Load one or more records into their associated buffers, given an array of primary key ID values, or the
    * actual DMOs themselves.  The IDs/DMOs are arranged to coincide with the tables underlying the query,
    * from left-most to right-most (in the sense of how they are joined by the query).
    * <p>
    * In the event an expected value cannot be loaded (e.g., the record has been deleted or is otherwise no
    * longer available), this method throws {@link MissingRecordException}, after setting the associated
    * buffer(s) into {@link RecordBuffer#setUnknownMode() unknown mode}. Note that a record being locked by
    * another session does not constitute a "missing" record. Note also that this exception is not thrown
    * until all the elements of {@code data} have been processed to the extent possible.
    * <p>
    * If the QueryComponent is not the first component in {@link #components()} 
    * and the query attached to it is an OUTER-JOIN, we set the buffer into 
    * {@link RecordBuffer#setUnknownMode() unknown mode} otherwise the buffer is released and the method
    * returns immediately.
    *
    * @param   data
    *          Array of primary key IDs or DMOs, one per table involved in the query.
    * @param   lockType
    *          Lock type which should by used.  Set to {@code null} to allow query to use its current lock
    *          type.
    * @param   silentIfNullId
    *          Do not raise {@link MissingRecordException} if some of the provided IDs are {@code null}.
    *          {@code null} IDs are valid for queries with OUTER join.
    *
    * @throws  MissingRecordException
    *          if any record cannot be loaded, because it is no longer available (e.g., deleted, etc.).
    * @throws  PersistenceException
    *          if there is an error loading data.
    */
   public void load(Object[] data, LockType lockType, boolean silentIfNullId)
   throws PersistenceException
   {
      int len = data.length;
      if (outer && len == 1 && data[0] == null)
      {
         // quick out: this is an OUTER-JOIN component positioned on a null record
         RecordBuffer buffer = components.get(0).getBuffer();
         buffer.release(true);
         // flag the buffer as 'OUTER' result of the query
         buffer.setUnknownMode();
         return;
      }
      
      boolean retry = (results != null);
      Long[] ids = new Long[len];
      for (int i = 0; i < len; i++)
      {
         Object datum = data[i];
         ids[i] = datum instanceof Record ? ((Record) datum).primaryKey() : (Long) datum;
      }
      
      try
      {
         boolean isNullId = false;
         for (int i = 0; i < len; i++)
         {
            if (ids[i] == null)
            {
               QueryComponent comp = components.get(i);
               
               if (comp.isOuter() && i > 0)
               {
                  RecordBuffer buffer = comp.getBuffer();
                  buffer.setUnknownMode();
               }
               else 
               {
                  isNullId = true;
                  break;
               }
            }
         }
         
         if (isNullId)
         {
            if (silentIfNullId)
            {
               setEmptyBuffersUnknown();
               return;
            }
            else
            {
               throw new MissingRecordException();
            }
         }
         
         if (!repositionByID(ids, true))
         {
            if (retry)
            {
               // Query did not have the target record(s) cached in its current
               // result set, so reset the results and try again.
               resetResults();
               if (!repositionByID(ids, true))
               {
                  throw new MissingRecordException();
               }
            }
            else
            {
               throw new MissingRecordException();
            }
         }
         
         fetch(offEnd == OffEnd.NONE, lockType, false);
      }
      catch (MissingRecordException exc)
      {
         setEmptyBuffersUnknown();
         
         throw exc;
      }
      finally
      {
         betweenRows = false;
      }
   }
   
   /**
    * Get all the off-end listeners associated with this query.
    * 
    * @return  the off-end listeners associated with this query.
    */
   public ArrayList<QueryOffEndListener> getOffEndListeners()
   {
      ArrayList<QueryOffEndListener> listeners = new ArrayList<>(components.size());

      for (int i = 0; i < components.size(); i++)
      {
         QueryComponent component = components.get(i);
         listeners.add(component.getBuffer().getQueryOffEndListener());
      }
      
      return listeners;
   }

   
   /**
    * Assemble an array of primary key IDs for the current record(s) in the
    * buffers underlying this query, from left-most to right-most (in the
    * sense of how the query joins the associated tables).  This method
    * essentially takes a lightweight snapshot of the currently loaded
    * record(s), so that this may be restored later.
    *
    * @return  Array of primary key IDs, one per table involved in the query.
    *
    * @see     #load(Object[], LockType, boolean)
    */
   public Object[] getRow()
   {
      return getRow(false);
   }
   
   /**
    * Assemble an array of primary key IDs for the current record(s) in the
    * buffers underlying this query, from left-most to right-most (in the
    * sense of how the query joins the associated tables).  This method
    * essentially takes a lightweight snapshot of the currently loaded
    * record(s), so that this may be restored later.
    * 
    * @param   forceIdOnly
    *          {@code true} only if the expected data are the PKs.
    *
    * @return  Array of primary key IDs, one per table involved in the query.
    *
    * @see     #load(Object[], LockType, boolean)
    */
   public Object[] getRow(boolean forceIdOnly)
   {
      if (results == null)
      {
         return null;
      }
      
      // test for empty result (may happen in case of outer joins)
      if (results.getRowNumber() == -1)
      {
         return null;
      }
      
      Object[] srcData = results.get(forceIdOnly);
      if (srcData == null)
      {
         return null;
      }
      
      int size = components.size();
      Object[] rowData = new Object[size];
      System.arraycopy(srcData, 0, rowData, 0, size);
      
      return rowData;
   }
   
   /**
    * Reposition the cursor such that a request to retrieve the next result
    * will retrieve the result which matches the specified primary key ID.  A
    * request to retrieve the previous result will retrieve the result
    * immediately previous to this, if any, in the results list.
    *
    * @param   id
    *          A primary key ID, representing the target record (in the case
    *          of a join, this ID is associated with the left-most record in
    *          the join).
    * 
    * @throws  ErrorConditionException
    *          if the specified ID represents the unknown value.
    */
   public void repositionByID(recid id)
   {
      if (!validateRepositionByID(id))
      {
         return;
      }
      
      Long[] serIDs = new Long[] { id.longValue() };
      
      if (!repositionByID(serIDs, false))
      {
         ErrorManager.recordOrThrowError(7331, "", "");
         // "Cannot reposition query <name> to recid/rowid(s) given".
      }
   }
   
   /**
    * Reposition the cursor such that a request to retrieve the next result
    * will retrieve the result which matches the specified array of primary
    * key IDs.  A request to retrieve the previous result will retrieve the
    * result immediately previous to this, if any, in the results list.
    *
    * @param   id1
    *          A primary key ID, representing the target record (in the case
    *          of a join, this ID is associated with the left-most record in
    *          the join).
    * @param   joinIDs
    *          All remaining primary key IDs, if any, arranged from left to
    *          right to coincide with the records being joined by the
    *          underlying query.
    * 
    * @throws  ErrorConditionException
    *          if any of the specified IDs represents the unknown value.
    */
   public void repositionByID(rowid id1, rowid...joinIDs)
   {
      if (id1 == null || id1.isUnknown())
      {
         QueryComponent qc = components.get(components.size() - 1);
         qc.getBuffer().release(true);
         ErrorManager.recordOrThrowError(7316, "Invalid rowid argument(s) given in query object method", false, false);
         return;
      }
      
      int len = joinIDs.length;
      Long[] serIDs = new Long[len + 1];
      serIDs[0] = id1.getValue();
      
      for (int i = 0; i < len; i++)
      {
         rowid next = joinIDs[i];
         
         serIDs[i + 1] = next.getValue();
      }
      
      if (!repositionByID(serIDs, false))
      {
         ErrorManager.recordOrThrowError(7331, "", "");
         // "Cannot reposition query <name> to recid/rowid(s) given".
      }
   }
   
   /**
    * Reposition the cursor such that a request to retrieve the next result will retrieve the
    * result which matches the specified array of primary key IDs. A request to retrieve the
    * previous result will retrieve the result immediately previous to this, if any, in the
    * results list.
    * <strong>Note:</strong> This is not public API.
    *
    * @param   joinIDs
    *          A set of primary key IDs, arranged from left to right to coincide with the records
    *          being joined by the underlying query.
    *
    * @throws  ErrorConditionException
    *          if any of the specified IDs represents the unknown value.
    *
    * @return  {@code true} on success.
    */
   @Override
   public boolean repositionByID(Long... joinIDs)
   {
      boolean ret = repositionByID(joinIDs, false);
      
      if (!ret)
      {
         ErrorManager.recordOrThrowError(7331, "", "");
         // Cannot reposition query <name> to recid/rowid(s) given.
      }
      
      return ret;
   }
   
   /**
    * Reposition the cursor to the specified row in the result set.  The row
    * indicates the position between two results, such that a call to
    * retrieve the next record will get the result at the specified row,
    * and a call to retrieve the previous result will get the result before
    * the new position.
    *
    * @param   row
    *          1-based index of the target position.
    *
    * @throws  ErrorConditionException
    *          if <code>row</code> represents unknown value.
    */
   public void reposition(NumberType row)
   {
      if (validateReposition(row))
      {
         reposition(row.intValue());
      }
   }
   
   /**
    * Reposition the query to the specified row in the result set.  The row
    * indicates the position between two results, such that a call to
    * retrieve the next record will get the record after the new position,
    * and a call to retrieve the previous record will get the record before
    * the new position.
    *
    * @param   row
    *          1-based index of the target position.
    *
    * @throws  ErrorConditionException
    *          if there is an error repositioning the cursor.
    */
   public void reposition(int row)
   {
      prepareReposition();
      
      if (results == null)
      {
         execute();
      }
      
      row--;   // adjust for 1-based row
      int currentRow = results.getRowNumber();
      if ((row >= 0 && currentRow == row) || results.setRowNumber(row))
      {
         offEnd = OffEnd.NONE;
         betweenRows = true;
      }
      else
      {
         offEnd = (row < 0 ? OffEnd.FRONT : OffEnd.BACK);
         betweenRows = false;
      }
      
      afterReposition(true);
   }
   
   /**
    * Advance the current cursor position forward by the specified number of rows.  This will
    * always leave the cursor between two results.  For example, given a request to move 1 row
    * forward:
    * <ul>
    *   <li>if the cursor currently is positioned <i>directly on a result</i>, this would move it
    *       ahead <i>one and a half positions</i>, to just beyond the following result;
    *   <li>if the cursor currently is positioned <i>between two results</i>, this would move it
    *       ahead <i>one position</i>, to just beyond the following result.
    * </ul>
    *
    * @param   rows
    *          Number of rows to scroll the cursor forward.
    *
    * @return  {@code true} if operation was successful and the cursor was moved the exact amount
    *          of rows as specified by the parameter. If {@code false} is returned then it is
    *          possible that cursor's position was moved but not with {@code rows} rows.
    *
    * @throws  ErrorConditionException
    *          if {@code rows} represents the unknown value; if there is an error repositioning
    *          the cursor.
    */
   public boolean forward(NumberType rows)
   {
      if (validateReposition(rows))
      {
         return forward(rows.intValue());
      }
      return false;
   }
   
   /**
    * Advance the current cursor position forward by the specified number of rows.  This will
    * always leave the cursor between two results.  For example, given a request to move 1 row
    * forward:
    * <ul>
    *   <li>if the cursor currently is positioned <i>directly on a result</i>, this would move it
    *       ahead <i>one and a half positions</i>, to just beyond the following result;
    *   <li>if the cursor currently is positioned <i>between two results</i>, this would move it
    *       ahead <i>one position</i>, to just beyond the following result.
    * </ul>
    *
    * @param   rows
    *          Number of rows to scroll the cursor forward.
    *
    * @return  {@code true} if operation was successful and the cursor was moved the exact amount
    *          of rows as specified by the parameter. If {@code false} is returned then it is
    *          possible that cursor's position was moved but not with {@code rows} rows.
    *
    * @throws  ErrorConditionException
    *          if there is an error repositioning the cursor.
    */
   public boolean forward(int rows)
   {
      if (rows == 0)
      {
         return true;
      }
      
      if (rows < 0)
      {
         return backward(-rows);
      }
      
      prepareReposition();
      
      if (results == null)
      {
         execute();
      }
      
      // Increment the requested number of rows by one if we are:
      // 1. on a record (i.e. not between rows) OR
      // 2. into front offend,
      // since the resulting position must be before the record *following*
      // the targeted jump.
      if (!betweenRows || (results.getRowNumber() == -1 && offEnd != OffEnd.BACK))
      {
         rows++;
      }
      
      boolean avail = results.scroll(rows);
      if (avail)
      {
         offEnd = OffEnd.NONE;
         betweenRows = true;
      }
      else
      {
         offEnd = OffEnd.BACK;
         betweenRows = false;
      }
      
      afterReposition(true);
      return avail;
   }
   
   /**
    * Move the current cursor position backward by the specified number of rows.  This will always
    * leave the cursor between two results.  For example, given a request to move 1 row backward:
    * <ul>
    *   <li>if the cursor currently is positioned <i>directly on a result</i>, this would move it
    *       back <i>one half position</i>, to just before the current result;
    *   <li>if the cursor currently is positioned <i>between two results</i>, this would move it
    *       back <i>one position</i>, to just before the current result.
    * </ul>
    *
    * @param   rows
    *          Number of rows to scroll the cursor backward.
    *
    * @return  {@code true} if operation was successful and the cursor was moved the exact amount
    *          of rows as specified by the parameter. If {@code false} is returned then it is
    *          possible that cursor's position was moved but not with {@code rows} rows.
    *
    * @throws  ErrorConditionException
    *          if {@code rows} represents the unknown value; if there is an error repositioning
    *          the cursor.
    */
   public boolean backward(NumberType rows)
   {
      if (validateReposition(rows))
      {
         return backward(rows.intValue());
      }
      return false;
   }
   
   /**
    * Move the current cursor position backward by the specified number of rows.  This will always
    * leave the cursor between two results.  For example, given a request to move 1 row backward:
    * <ul>
    *   <li>if the cursor currently is positioned <i>directly on a result</i>, this would move it
    *       back <i>one half position</i>, to just before the current result;
    *   <li>if the cursor currently is positioned <i>between two results</i>, this would move it
    *       back <i>one position</i>, to just before the current result.
    * </ul>
    *
    * @param   rows
    *          Number of rows to scroll the cursor backward.
    *
    * @return  {@code true} if operation was successful and the cursor was moved the exact amount
    *          of rows as specified by the parameter. If {@code false} is returned then it is
    *          possible that cursor's position was moved but not with {@code rows} rows.
    *
    * @throws  ErrorConditionException
    *          if there is an error repositioning the cursor.
    */
   public boolean backward(int rows)
   {
      if (rows == 0)
      {
         return true;
      }
      
      if (rows < 0)
      {
         return forward(-rows);
      }
      
      prepareReposition();
      
      if (results == null)
      {
         execute();
      }
      
      // Decrement the requested number of rows by one if we are:
      // 1. on a record (i.e. not between rows) AND
      // 2. NOT into back offend,
      // since the resulting position must be before the record *following*
      // the targeted jump.
      if (!betweenRows && offEnd != OffEnd.BACK)
      {
         rows--;
      }
      
      boolean avail = results.scroll(-rows);
      if (avail)
      {
         offEnd = OffEnd.NONE;
         betweenRows = true;
      }
      else
      {
         offEnd = OffEnd.FRONT;
         betweenRows = false;
      }
      
      afterReposition(true);
      return avail;
   }
   
   /**
    * Return the number of results found for this query.
    * <p>
    * Note that the first time this method is invoked, the database server may have to scroll to
    * the end of the result set, which may be a time-consuming process for a large result set.
    *
    * @return  Size of result set or value if the query is not open.
    */
   public integer size()
   {
      if (results == null)
      {
         return new integer();
      }
      
      if (rowCount < 0)
      {
         int row = results.getRowNumber();
         
         // if results.last() returns false, it means there are no results; otherwise add 1 to
         // the 0-based row number of the last result
         rowCount = results.last() ? results.getRowNumber() + 1 : 0;
         
         if (row == -1)
         {
            if (offEnd == OffEnd.BACK)
            {
               results.next();
            }
            else
            {
               results.reset();
            }
         }
         else
         {
            results.setRowNumber(row);
         }
      }
      
      return (new integer(rowCount));
   }
   
   /**
    * Progress-compatible function that returns the 1-based index of the
    * current row in the result set.
    *
    * @return  Unknown value if the query is not open or result set is empty.
    *          1 if the query is positioned before the first record.
    *          A value 1 greater than the number of rows in the query result
    *          set if the query is positioned beyond the last record.
    *          Otherwise returns the number of rows in the query result set.
    *
    */
   public integer currentRow()
   {
      if (results == null ||
          (offEnd == OffEnd.NONE && results.getRowNumber() == -1 && !isScrolling()))
      {
         return (new integer());
      }
      else
      {
         int size = size().intValue();
         int row = results.getRowNumber();
         
         if (size == 0)
         {
            return (new integer());
         }
         else if (offEnd == OffEnd.BACK)
         {
            return (new integer(size + 1));
         }
         else if (row == -1)
         {
            // We are at the front offend. Return unknown value if the query
            // is an adaptive one and we haven't retrieved any results using 
            // it.
            return !isNativelyPreselect() && (offEnd != OffEnd.FRONT)
                   ? new integer()
                   : new integer(1);
         }
         else
         {
            return (new integer(row + 1));
         }
      }
   }
   
   /**
    * Get the 1-based index of the current row in the result set.
    *
    * @return  Current row or the unknown value if the query has not yet
    *          executed.
    */
   public integer currentRowImpl()
   {
      if (results == null || offEnd != OffEnd.NONE)
      {
         return (new integer());
      }
      
      return (new integer(results.getRowNumber() + 1));
   }
   
   /**
    * Indicate whether the query has run off either end of its result set,
    * due to scrolling beyond the last record or before the first record.
    *
    * @return  <code>true</code> if no record can be retrieved at the
    *          cursor's current position;  otherwise <code>false</code>.
    */
   public boolean _isOffEnd()
   {
      boolean isOffEnd = (offEnd != OffEnd.NONE);
      if (!isOffEnd && results == null)
      {
         return false;
      }
      
      return isOffEnd;
   }
   
   /**
    * Indicate whether the query will retrieve full records, rather than primary keys only.
    * 
    * @return  <code>true</code> to return full records; <code>false</code> to return primary
    *          keys only.
    */
   public boolean isFullRecords()
   {
      return fullRecords;
   }
   
   /**
    * Force the query to retrieve full records, rather than primary keys only.
    */
   public void setFullRecords()
   {
      this.fullRecords = true;
   }
   
   /**
    * Reset the query by clearing its state to a known default.  This is
    * used to ensure that subsequent requests for FIRST and LAST record
    * retrievals operate on a known query/buffer state.
    * <p>
    * This implementation nulls out the current result set, so that the next
    * record retrieval request triggers a new server-side query.  Optionally,
    * it also resets resolvable substitution arguments and triggers where
    * expression arguments to be resolved, if a client-side where cause
    * expression is defined.
    *
    * @param   resolveArgs
    *          <code>true</code> if query substitution arguments should be
    *          resolved/reset as part of the reset;  <code>false</code> if
    *          existing argument values should be used for the next query
    *          execution.
    * @param   forceOrigArgs
    *          Flag indicating that the original arguments used at the construction of the query must be
    *          re-calculated.
    */
   @Override
   public void reset(boolean resolveArgs, boolean forceOrigArgs)
   {
      resetResults();
      
      if (resolveArgs && (resolvableArgs || forceOrigArgs))
      {
         for (int i = 0; i < components.size(); i++)
         {
            QueryComponent comp = components.get(i);
            comp.resetArgs(forceOrigArgs);
         }
      }
   }
   
   /**
    * Set each buffer backing a query component to unknown mode.
    *
    * @see     RecordBuffer#setUnknownMode
    */
   public void setUnknownRecord()
   {
      ArrayList<QueryComponent> components = components();
      for (int i = 0; i < components.size(); i++)
      {
         QueryComponent comp = components.get(i);
         comp.getBuffer().setUnknownMode();
      }
   }
   
   /**
    * Get the number of tables joined by this query.
    *
    * @return  Number of joined tables.
    */
   public int getTableCount()
   {
      return components == null ? 0 : components.size();
   }
   
   /**
    * Fetch the array of primary key IDs associated with the first result in
    * this query's result list.
    *
    * @return  Array of record IDs reflecting the first query result, or
    *          <code>null</code> if there is no such result.
    */
   public Object[] peekFirst()
   {
      execute();
      results.first();
      
      return getRow();
   }
   
   /**
    * Fetch the array of primary key IDs associated with the last result in
    * this query's result list.
    *
    * @return  Array of record IDs reflecting the last query result, or
    *          <code>null</code> if there is no such result.
    */
   public Object[] peekLast()
   {
      execute();
      results.last();
      
      return getRow();
   }
   
   /**
    * Fetch the array of primary key IDs associated with the next result in
    * this query's result list.
    *
    * @return  Array of record IDs reflecting the next query result, or
    *          <code>null</code> if there is no such result.
    */
   public Object[] peekNext()
   {
      execute();
      if (results.next())
      {
         return getRow();
      }
      else
      {
         return null;
      }
   }
   
   /**
    * Fetch the array of primary key IDs associated with the previous result
    * in this query's result list.
    *
    * @return  Array of record IDs reflecting the previous query result, or
    *          <code>null</code> if there is no such result.
    */
   public Object[] peekPrevious()
   {
      execute();
      if (results.previous())
      {
         return getRow();
      }
      else
      {
         return null;
      }
   }
   
   /**
    * Indicate whether this query preselects all of its results.
    *
    * @return  <code>true</code>.
    */
   public boolean isPreselect()
   {
      return true;
   }
   
   /**
    * Set the given results object into this query.
    *
    * @param   results
    *          New result set for this query.
    */
   public void setResults(Results results)
   {
      if (this.results != null)
      {
         // ensure the previous results are always cleaned up.
         this.results.cleanup();
      }
      this.results = results;
   }
   
   /**
    * Access the record buffers manipulated by this query, in order from
    * left-most (outermost) table of the query's join (if any) to right-most
    * (innermost) table.
    * 
    * @return  Array of one or more record buffers.
    */
   @Override
   public RecordBuffer[] getRecordBuffers()
   {
      RecordBuffer[] buffers = new RecordBuffer[getTableCount()];
      ArrayList<QueryComponent> components = components();
      if (components == null)
      {
         return buffers;
      }
      for (int i = 0; i < components.size(); i++)
      {
         QueryComponent comp = components.get(i);
         buffers[i] = comp.getBuffer();
      }
      
      return buffers;
   }
   
   /**
    * Get the query substitution arguments associated with this query, if any.
    * 
    * @return  Substitution arguments, in order, or <code>null</code> if there are none.
    */
   @Override
   public Object[] getArgs()
   {
      return components.get(0).getRawArgs();
   }
   
   /**
    * Get the original, FQL where clause associated with this query.
    * 
    * @return  Original where clause, or <code>null</code> if there is none.
    */
   @Override
   public String getOriginalWhere()
   {
      return components.get(0).getOriginalWhere();
   }
   
   /**
    * Get the legacy natural join, if any, associated with this query.
    * 
    * @return  Legacy join object, or <code>null</code> if none.
    */
   @Override
   public AbstractJoin getJoin()
   {
      return components.get(0).getJoin();
   }
   
   /**
    * Force this query to operate in dynamic retrieval mode, if it supports this mode.
    * 
    * @throws  UnsupportedOperationException
    *          always; this query type does not support dynamic retrieval.
    */
   @Override
   public void forceDynamicOperation()
   {
      throw new UnsupportedOperationException("PreselectQuery has no dynamic operation mode!");
   }
   
   /**
    * Get the sort phrase associated with this query.
    * 
    * @return  Sort phrase.
    */
   public String getSortPhrase()
   {
      return sort;
   }
   
   /**
    * Create an adaptive query component based on the information in this query (which is
    * presumed to be a single-table query), and given a list of adaptive query components which
    * represent outer, nested query loops, which will perform a server-side join to the
    * innermost, nested query component.
    * 
    * @param   joinList
    *          List of adaptive components representing nested query loops which will contain
    *          the adaptive query component returned by this method. The last component in the
    *          list represents the nested query loop immediately containing the returned query
    *          component, with which the join will be made.
    * @param   query
    *          Adaptive query which will manage the server-side join of <code>joinList</code> and
    *          the returned query component.
    * @param   iteration
    *          Iteration type: FIRST, LAST, or NEXT.
    * @param   outer
    *          <code>true</code> if the join should be a left outer join. This type of join is
    *          not supported at the time of this writing, so this parameter should always be
    *          <code>false</code>. It is here to support a planned, future enhancement.
    * @param   fallback
    *          A compound query component to use in the event the multi-table adaptive query
    *          switches from preselect to dynamic retrieval mode.
    * 
    * @return  An adaptive query component suitable for a server-side join, based on the
    *          information in this query.
    */
   @Override
   public AdaptiveComponent makeAdaptiveServerJoinComponent(List<AdaptiveComponent> joinList,
                                                            AdaptiveQuery query,
                                                            int iteration,
                                                            boolean outer,
                                                            CompoundComponent fallback)
   {
      QueryComponent original = components.get(0);
      original.processDynamicFilters();
      RecordBuffer buffer = original.getBuffer();
      AbstractJoin join = original.getJoin();
      Object[] args = original.getRawArgs();
      String where = original.getOriginalWhere();
      String indexInfo = original.getIndexInfo();
      LockType lockType = original.getLockType();
      
      QueryComponent.ServerJoinData sjd =
         QueryComponent.prepareServerJoinData(joinList, buffer, join, args);
      BufferReference inverse = (join != null ? join.getInverse() : null);
      AdaptiveComponent comp = new AdaptiveComponent(query,
                                                     buffer,
                                                     sjd.join,
                                                     where,
                                                     null,
                                                     sort,
                                                     indexInfo,
                                                     lockType,
                                                     sjd.args,
                                                     iteration,
                                                     outer,
                                                     inverse);
      comp.setReferenceSubs(sjd.referenceSubs);
      
      // gather all related buffers, including [relatedBuffers] from [sjd] and eventual external
      // buffers from original query component
      Set<RecordBuffer> relatedBuffers = new HashSet<>();
      if (sjd.relatedBuffers != null)
      {
         relatedBuffers.addAll(sjd.relatedBuffers);
      }
      RecordBuffer[] externalBuffers = fallback.getQuery().getExternalBuffers();
      if (externalBuffers != null && externalBuffers.length != 0)
      {
         relatedBuffers.addAll(Arrays.asList(externalBuffers));
      }
      if (!relatedBuffers.isEmpty())
      {
         comp.setRelatedBuffers(relatedBuffers);
      }
      
      comp.setFallback(fallback);
      
      if (iteration == FIRST || iteration == LAST)
      {
         try
         {
            comp.setIteration(iteration, sort);
         }
         catch (PersistenceException exc)
         {
            // should only happen if we have a runtime programming error or conversion error
            // (or with bad, hand-written business logic)
            throw new IllegalArgumentException(exc);
         }
      }
      
      return comp;
   }
   
   /**
    * Create a preselect query component based on the information in this query (which is
    * presumed to be a single-table query), and given a list of preselect query components which
    * represent outer, nested query loops, which will perform a server-side join to the
    * innermost, nested query component.
    * 
    * @param   joinList
    *          List of preselect components representing nested query loops which will contain
    *          the preselect query component returned by this method. The last component in the
    *          list represents the nested query loop immediately containing the returned query
    *          component, with which the join will be made.
    * @param   iteration
    *          Iteration type: FIRST, LAST, or NEXT.
    * @param   outer
    *          <code>true</code> if the join should be a left outer join. This type of join is
    *          not supported at the time of this writing, so this parameter should always be
    *          <code>false</code>. It is here to support a planned, future enhancement.
    * 
    * @return  A preselect query component suitable for a server-side join, based on the
    *          information in this query.
    */
   @Override
   public QueryComponent makePreselectServerJoinComponent(List<QueryComponent> joinList,
                                                          int iteration,
                                                          boolean outer)
   {
      QueryComponent original = components.get(0);
      original.processDynamicFilters();
      RecordBuffer buffer = original.getBuffer();
      AbstractJoin join = original.getJoin();
      Object[] args = original.getRawArgs();
      String where = original.getOriginalWhere();
      String indexInfo = original.getIndexInfo();
      LockType lockType = original.getLockType();
      
      QueryComponent.ServerJoinData sjd =
         QueryComponent.prepareServerJoinData(joinList, buffer, join, args);
      QueryComponent comp = new QueryComponent(this,
                                               buffer,
                                               sjd.join,
                                               where,
                                               null,
                                               indexInfo,
                                               lockType,
                                               sjd.args,
                                               iteration,
                                               outer);
      comp.setReferenceSubs(sjd.referenceSubs);
      comp.setRelatedBuffers(sjd.relatedBuffers);
      
      if (iteration == FIRST || iteration == LAST)
      {
         try
         {
            comp.setIteration(iteration, sort);
         }
         catch (PersistenceException exc)
         {
            // should only happen if we have a runtime programming error or conversion error
            // (or with bad, hand-written business logic)
            throw new IllegalArgumentException(exc);
         }
      }
      
      return comp;
   }
   
   /**
    * Indicate whether this query has a client-side where clause expression associated with it.
    * 
    * @return  <code>true</code> if there is a where expression; else <code>false</code>.
    */
   @Override
   public boolean hasWhereExpression()
   {
      return whereExprList != null && !whereExprList.isEmpty();
   }
   
   /**
    * Clean up query resources when this query's scope ends. This implementation cleans up any
    * open results and deregisters this session listener.
    * 
    * @see  P2JQuery#cleanup()
    */
   public void cleanup()
   {
      if (closed)
      {
         return;
      }
      
      forceOnlyId = false;
      closed = true;
      if (results != null)
      {
         results.cleanup();
         results = null;
      }
      
      if (regWithSession)
      {
         regWithSession = false;
         persistence.deregisterSessionListener(this);
      }
      
      if (unregisterOnCleanup)
      {
         ChangeBroker broker = ChangeBroker.get();
         broker.removeFromGlobal(this);
         
         unregisterOnCleanup = false;
      }
   }
   
   /**
    * Open the prepared query and reposition it to the first result if the query is being browsed.
    */
   public void open()
   {
      if (!closed)
      {
         close();
      }
      
      resetResults();
      
      setErrorIfNull(false);
      setLenientOffEnd(true);
      execute();
      
      if (shouldRegisterRecordChangeListeners())
      {
         int scope;
         if (isDynamicPredicate())
         {
            scope = 0;
         }
         else
         {
            scope = BufferManager.get().findNearestExternal();
            RecordBuffer[] buffers = getRecordBuffers();

            for (int i = 0; i < buffers.length; i++)
            {
               RecordBuffer buffer = buffers[i];
               scope = ((scope == -1)
                  ? buffer.getScopeOpenDepth()
                  : Math.min(scope, buffer.getScopeOpenDepth()));
            }
         }
         
         if (scope != -1)
         {
            registerRecordChangeListeners(Integer.valueOf(scope));
         }
      }
      
      super.open();
   }
   
   /**
    * Explicitly close the query.
    * <p>
    * This will remove any registered finalizable.
    */
   @Override
   public void close()
   {
      close(false);
   }
   
   /**
    * Explicitly close the prepared query.
    * 
    * @param    inResource
    *           Flag indicating this query is used in a QUERY legacy resource.
    */
   @Override
   public void close(boolean inResource)
   {
      super.close();
      
      if (thisFinalizable)
      {
         TransactionManager.removeFinalizable(this);
         thisFinalizable = false;
      }
      
      if (regCleaner)
      {
         TransactionManager.removeFinalizable(cleaner);
         regCleaner = false;
      }
      
      if (inResource && !unregisterOnCleanup)
      {
         // remove it explicitly
         ChangeBroker broker = ChangeBroker.get();
         broker.removeListener(this);
      }
   }
   
   /**
    * Override parent's no-op implementation to track whether client code has
    * set this as a scrollable query.  While this setting does not explicitly
    * affect this query's ability to respond to reposition requests (this
    * implementation is inherently scrollable), it does provide a hint to the
    * implementation regarding the use and scope of this query.  This method
    * must be invoked before any results are retrieved.
    */
   public void setScrolling()
   {
      scrolling = true;
   }
   
   /**
    * Disable results list scrolling/repositioning (if it was enabled by
    * default). This method must be invoked before any results are retrieved. 
    */
   public void setNonScrolling()
   {
      scrolling = false;
   }
   
   /**
    * Get the off-end status of this query, which indicates whether the query
    * has run off either end of its results.
    * 
    * @return  One of the {@link OffEnd} constants <code>NONE</code>,
    *          <code>FRONT</code>, or <code>BACK</code>.
    */
   public OffEnd getOffEnd()
   {
      return offEnd;
   }
   
   /**
    * Release all buffers. The position of the query is not affected.
    */
   public void releaseBuffers()
   {
      for (int i = 0; i < components.size(); i++)
      {
         QueryComponent c = components.get(i);
         c.getBuffer().release(false);
      }
   }
   
   /**
    * Adds a new constraint term to the where predicate of this query, effectively filtering out
    * any rows whose referenced field does not match the requested value. Semantically, the query
    * will select rows that match
    * <p>
    * {@code new-where := (fr = val) AND (old-query)}.
    * <p>
    * Note: The constraints are exclusive meaning that if constraints will be added for same
    * field of same buffer, the query will return an empty result because the field cannot be at
    * the same time equals to two different values.
    * <p>
    * Use {@link #clearDynamicFilters} to clean all dynamically added criteria and restore the
    * sorting order of the query to its original form.
    *
    * @param   fr
    *          A reference to field used as filtering criterion. Only the rows that matches the
    *          specified value for this field will be selected. Must not be {@code null}.
    * @param   val
    *          The filtering value for this constraint. Can be {@code null} or {@code unknown}, in
    *          which case the SQL {@code null} value will be assumed. In this case the predicate
    *          will look like:
    *           <p>
    *          {@code new-where := (fr IS  NULL) AND (old-query)}
    * @param   format
    *          The format to be used when comparing values. The reason for this parameter is that
    *          for some datatype (ex: {@code decimal}) the on-screen value can be different from
    *          the database (because of rounding).
    *
    * @return  {@code true} if the filtering constraint was successfully added. If the field
    *          reference is not related to the buffer(s) of this query, {@code false} is returned.
    */
   @Override
   public boolean addDynamicFilter(FieldReference fr, BaseDataType val, String format)
   {
      ArrayList<QueryComponent> components = components();
      for (int i = 0; i < components.size(); i++)
      {
         QueryComponent comp = components.get(i);
         if (fr.getParentBuffer() == comp.getBuffer())
         {
            comp.addDynamicFilter(fr, val, format);
            return true;
         }
      }
      
      return false;
   }
   
   /**
    * Clears any dynamically filters added at runtime. The query predicate will be restored to its
    * form from generated at conversion time. 
    */
   @Override
   public void clearDynamicFilters()
   {
      for (int i = 0; i < components.size(); i++)
      {
         QueryComponent comp = components.get(i);
         comp.clearDynamicFilters();
      }
   }
   
   /**
    * Adds a new sorting criterion as the strongest criterion for sorting the result of this
    * query. The other criteria remain queued, but with lower priority. 
    * <p>
    * Use {@link #clearDynamicSortCriteria} to clean all dynamically added criteria and restore the
    * sorting order of the query to its original form.
    *
    * @param   fr
    *          A reference to field used as new strongest criterion. The current criteria will
    *          be kept, but with lower priority.
    * @param   asc
    *          The direction of sorting for this criterion. Use {@code true} for ascending sort
    *          and {@code false} for descending sorting.
    */
   void addDynamicSortCriterion(FieldReference fr, boolean asc)
   {
      if (fr == null)
      {
         return; // no-op
      }
      ArrayList<QueryComponent> components = components();
      for (int i = 0; i < components.size(); i++)
      {
         QueryComponent comp = components.get(i);
         if (fr.getParentBuffer() == comp.getBuffer())
         {
            if (dynamicSorts == null)
            {
               dynamicSorts = new ArrayList<>();
            }
            dynamicSorts.add(0, fr + (asc ? " asc" : " desc"));
         }
      }
   }
   
   /**
    * Clears any dynamically sort criteria added at runtime. The query sort order will be restored
    * to its form from generated at conversion time. 
    */
   void clearDynamicSortCriteria()
   {
      dynamicSorts = null;
   }
   
   /**
    * Conversion of INDEX-INFORMATION attribute (KW_IDX_INFO).
    * <p>
    * Getter of INDEX-INFORMATION attribute
    *
    * @param    n
    *           An integer expression that evaluates to the level of join for which you want index
    *           information.
    *
    * @return   A character string consisting of a comma-separated list of the index or indexes
    *           the query uses at the level of join specified.
    */
   @LegacyAttribute(name = "INDEX-INFORMATION")
   public character indexInformation(NumberType n)
   {
      return new character(components.get(n.intValue() - 1).getIndexInfo());
   }
   
   /**
    * Indicate whether this query is a preselect query by its nature.
    *
    * @return  <code>true</code>.
    */
   public boolean isNativelyPreselect()
   {
      return true;
   }

   /**
    * Creates an entry in the result list for the current row. Implements the 4GL
    * CREATE-RESULT-LIST-ENTRY() method. Created entry contains id(s) of the record(s) currently
    * located in the backing buffer(s).
    *
    * @return <code>true</code> on success.
    */
   public logical createResultListEntry()
   {
      Results cached = cacheResults(results, false);
      if (results != cached)
      {
         setResults(cached);
      }
      
      Object[] row = new Object[getTableCount()];
      ArrayList<QueryComponent> components = components();
      for (int i = 0; i < components.size(); i++)
      {
         QueryComponent comp = components.get(i);
         RecordBuffer buffer = comp.getBuffer();
         Record dmo = buffer.getCurrentRecord();
         row[i] = dmo != null
               ? (isFullRecords() ? dmo : dmo.primaryKey()) 
               : null;
      }
      
      results.addRow(row, betweenRows);
      offEnd = isBrowsed() ? OffEnd.NONE : offEnd;
      betweenRows = false;
      rowCount++;
      return new logical(true);
   }
   
   /**
    * Deletes the current row of a query's result list. Implements the 4GL
    * DELETE-RESULT-LIST-ENTRY() method.
    *
    * @param  allowBetweenRows
    *         If <code>true</code> then the cursor can be positioned between rows (the next
    *         row will be deleted). <code>false</code> for conventional DELETE-RESULT-LIST-ENTRY()
    *         mode where the cursor should be positioned on a row (otherwise <code>false</code> is
    *         returned).
    *
    * @return <code>true</code> on success.
    */
   @Override
   public logical deleteResultListEntry(boolean allowBetweenRows)
   {
      // In allowBetweenRows mode we may want to delete multiple records without fetching them,
      // so currentRowDeleted state is ignored.
      if (currentRowDeleted && !allowBetweenRows)
      {
         return new logical(false);
      }
      
      Results cached = cacheResults(results, false);
      
      if (results != cached)
      {
         setResults(cached);
      }
      
      if (!_isOffEnd() &&
          cached.getNumberOfLoadedRows() > 0 &&
          (allowBetweenRows || !betweenRows))
      {
         int row = cached.getRowNumber();
         if (row >= 0)
         {
            int width = getTableCount();
            Long[] rsIDs = new Long[width];
            for (int i = 0; i < width; i++)
            {
               rsIDs[i] = cached.getID(i);
            }
            
            repoCache.remove(rsIDs);
         }
         currentRowDeleted = true;
         cached.deleteRow();
         betweenRows = true;
         rowCount--;
         return new logical(true);
      }
      
      return new logical(false);
   }
   
   /**
    * Indicate whether this query's <code>scrolling</code> flag has been set.
    * 
    * @return  <code>true</code> if this is a scrolling query;  else
    *          <code>false</code>.
    */
   public boolean isScrolling()
   {
      return (scrolling && !isForwardOnly()) || isNativelyScrolling();
   }
   
   /**
    * Indicate whether this query type is scrolling by its nature.
    * <p>
    * Subclasses which are not natively scrolling should override this method.
    * 
    * @return  <code>true</code>.
    */
   protected boolean isNativelyScrolling()
   {
      return true;
   }
   
   /**
    * Add one or more buffers to the list of those which this query references externally. These typically 
    * will be buffers referenced by converted CAN-FIND expressions embedded in the query's FQL expression, 
    * which are not the primary buffers associated with the query.
    * <p>
    * If the bound buffer differs from the definition buffer (see {@link #buffersMatch}) for any of the 
    * external buffers, then all {@link #components()} will have the {@link #translateWhere()} enabled 
    * (if not already enabled).
    * 
    * @param   dmos
    *          One or more external buffers.
    * 
    * @return  This query instance.
    */
   @Override
   public P2JQuery addExternalBuffers(DataModelObject... dmos)
   {
      super.addExternalBuffers(dmos);
      
      RecordBuffer[] buffers = getExternalBuffers();
      if (buffers == null)
      {
         return this;
      }
      
      RecordBuffer[] defs = getExternalBuffersDefs();
      boolean translate = false;
      for (int i = 0; i < buffers.length && !translate; i++)
      {
         translate = !buffersMatch(buffers[i], defs[i]);
      }
      
      if (translate)
      {
         Function<String, String> translateWhere = (where) ->
         {
            // just create two empty lists where the binding from the external buffers will be collected.
            List<RecordBuffer> binding = new ArrayList<>();
            List<RecordBuffer> definition = new ArrayList<>();
            return translateWhere(binding, definition, where);
         };
         
         ArrayList<QueryComponent> components = components();
         for (int i = 0; i < components.size(); i++)
         {
            QueryComponent qc = components.get(i); 
            qc.enableTranslateWhere(translateWhere);
         }
      }
      
      return this;
   }
   
   /**
    * Adds an alternative execution query, for the multi-tenant case in which the CAN-FIND nested buffer and
    * the main buffer of query component do not belong to same physical database.
    * <p>
    * This method will be generated only in case of multi-tenant projects, in the condition above-mentioned.
    *
    * @param   where
    *          FQL where clause.  May be {@code null}.
    * @param   whereExpr
    *          Client-side where clause expression. May be {@code null} but this is probably caused by an
    *          incorrect conversion.
    * @param   sort
    *          The sort order.
    *
    * @return  Self object so that the call to this method can be optionally chained with the constructor.
    */
   @Override
   public PreselectQuery setMultiTenantAlternative(String where, Supplier<logical> whereExpr, String sort)
   {
      return setMultiTenantAlternative(where, whereExpr, sort, null);
   }
   
   /**
    * Adds an alternative execution query, for the multi-tenant case in which the CAN-FIND nested buffer and
    * the main buffer of query component do not belong to same physical database.
    * <p>
    * This method will be generated only in case of multi-tenant projects, in the condition above-mentioned.
    *
    * @param   where
    *          FQL where clause.  May be {@code null}.
    * @param   whereExpr
    *          Client-side where clause expression. May be {@code null} but this is probably caused by an
    *          incorrect conversion.
    * @param   sort
    *          The sort order.
    * @param   args
    *          Substitution parameters for FQL queries. These will be used if not overridden by a record
    *          retrieval method.
    *
    * @return  Self object so that the call to this method can be optionally chained with the constructor.
    */
   @Override
   public PreselectQuery setMultiTenantAlternative(String where,
                                                   Supplier<logical> whereExpr,
                                                   String sort,
                                                   Object[] args)
   {
      RecordBuffer[] buffers = getExternalBuffers();
      if (buffers == null || buffers.length == 0)
      {
         return this; // no-op. This should not happen
      }
      
      RecordBuffer buffer = components.get(0).getBuffer();
      Persistence.Context privateContext = buffer.getPersistence().getContext(Persistence.PRIVATE_CTX);
      if (privateContext.getTenantId() == TenantManager.DEFAULT_TENANT_ID)
      {
         // running as default tenant, the query is intra-database (using the default database)
         return this;
      }
      
      boolean isMt = buffer.getDmoInfo().multiTenant;
      for (RecordBuffer eb : buffers)
      {
         if (eb.getDmoInfo().multiTenant != isMt)
         {
            // if the query is cross-physical database, re-initialize with the alternative execution path 
            return initialize(buffer, where, whereExpr, sort, null, null, args, null);
         }
      }
      
      // if we reach this point, all table are either public or all private, so the optimised query can be
      // safely executed
      return  this;
   }
   
   /**
    * Discard the current result set, clear the reposition cache, and reset
    * result-related state to default values.
    */
   protected void resetResults()
   {
      if (results != null)
      {
         results.cleanup();
      }
      setResults(null);
      offEnd = OffEnd.NONE;
      betweenRows = true;
      rowCount = -1;
      clearRepoCache();
      currentRowDeleted = false;
   }
   
   /**
    * Get the query components.
    *
    * @return  Query components.
    */
   protected ArrayList<QueryComponent> components()
   {
      return components;
   }
   
   /**
    * Indicate whether this query will throw a
    * <code>QueryOffEndException</code> when it runs off the end of its
    * results.
    *
    * @return  <code>true</code> if an exception will be thrown, else
    *          <code>false</code>.
    */
   protected boolean isLenientOffEnd()
   {
      return lenientOffEnd;
   }
   
   /**
    * Indicate whether a call to move to the previous record in the result
    * list should be treated as a call to move to the last record in the list.
    * This will be the case if either there are no results available, or if
    * the query is not off the front end of its results, but the row number
    * is -1 (indicating no results have yet been retrieved).
    * 
    * @return  <code>true</code> to treat a previous request as a request to
    *          move to the last record in the list;  else <code>false</code>.
    */
   protected boolean treatPreviousAsLast()
   {
      return (results == null || (offEnd != OffEnd.FRONT && results.getRowNumber() == -1));
   }
   
   /**
    * Walk through each component and set its backing buffer to unknown mode,
    * such that the invocation of any of its DMO-backed methods returns
    * unknown value.
    */
   protected void setEmptyBuffersUnknown()
   {
      ArrayList<QueryComponent> components = components();
      for (int i = 0; i < components.size(); i++)
      {
         QueryComponent comp = components.get(i);
         RecordBuffer buffer = comp.getBuffer();
         if (buffer.getCurrentRecord() == null)
         {
            buffer.setUnknownMode();
         }
      }
   }
   
   /**
    * Given a primary DMO buffer and an inverse DMO buffer which together
    * define a natural join, create the <code>AbstractJoin</code> helper
    * object which is used to implement the join.
    * <p>
    * It is intended that this method be called when adding a query component.
    * If no previous components have been added, a <code>DynamicJoin</code>
    * is created, and it is assumed that the inverse DMO is provided as a
    * query substitution parameter.  If a component previously has been added,
    * a <code>ServerJoin</code> is created, and it is assumed that the inverse
    * DMO will be resolved by a server-side join, rather than being provided
    * as a query substitution parameter.
    * <p>
    * If no inverse buffer is provided, no join is created.
    *
    * @param   primary
    *          Buffer reference to the primary DMO.
    * @param   inverse
    *          Buffer reference to the joined DMO, or <code>null</code> if no
    *          join is required.
    *
    * @return  A join helper object, or <code>null</code> if no natural join
    *          is required.
    */
   protected AbstractJoin processJoin(BufferReference primary, BufferReference inverse)
   {
      if (inverse == null)
      {
         return null;
      }
      
      AbstractJoin join = null;
      
      if (getTableCount() == 0)
      {
         // If we have an inverse proxy and this is the first component being
         // added, the inverse is being provided as a query substitution
         // parameter.
         join = Persistence.isForeignKeysEnabled()
                ? new DynamicJoin(primary.buffer(), inverse.buffer())
                : new DynamicLegacyKeyJoin(primary.buffer(), inverse.buffer());
      }
      else
      {
         // If we have an inverse proxy and this is not the first component
         // being added, the inverse is provided as a related DMO from a
         // related table.  In this case, the inverse was not provided as a
         // query substitution parameter.
         join = Persistence.isForeignKeysEnabled()
                ? new ServerJoin(primary.buffer(), inverse.buffer())
                : new ServerLegacyKeyJoin(primary.buffer(), inverse.buffer());
      }
      
      return join;
   }
   
   /**
    * Assemble the select clause for an FQL projection query, where the only
    * field retrieved for each buffer is the primary key. This uses the
    * Hibernate reserved keyword "id" to represent the primary key. Form of
    * the clause is:
    * <pre>
    *    select {buffer0.id} [, {buffer1.id} [...]]
    * </pre>
    *
    * @param   buf
    *          String buffer into which clause is assembled.
    */
   protected void assembleSelectClause(StringBuilder buf)
   {
      boolean needSelect = !fullRecords;

      ArrayList<QueryComponent> components = components();
      for (int i = 0; !needSelect && i < components.size(); i++)
      {
         QueryComponent comp = components.get(i);
         if (!isClientWhere() && !fullRecords && comp.isIdOnly())
         {
            needSelect = true;
         }
      }
      
      if (!needSelect)
      {
         return;
      }
      
      buf.append("select ");

      components = components();
      for (int i = 0; i < components.size(); i++)
      {
         QueryComponent comp = components.get(i);
         if (i > 0)
         {
            buf.append(", ");
         }
         assembleComponent(buf, comp, false);
      }
      
      buf.append(" ");
   }

   /**
    * Assemble a component that retrieves the full record or only primary key.
    * @param   buf
    *          String buffer into which clause is assembled.
    * @param   component
    *          The query component that needs to be assembled.
    * @param   forceOnlyPk
    *          Flag indicating if only the primary key should be fetched.
    */
   protected void assembleComponent(StringBuilder buf, QueryComponent component, boolean forceOnlyPk)
   {
      boolean qualified = !isOmitAlias();

      RecordBuffer buffer = component.getBuffer();

      if (qualified)
      {
         buf.append(buffer.getDMOAlias());
      }

      if (!isClientWhere() && (component.isIdOnly() && !fullRecords || forceOnlyPk))
      {
         if (qualified)
         {
            buf.append('.');
         }
         
         buf.append(DatabaseManager.PRIMARY_KEY);
      }
      else 
      {
         Property[] iprops = included == null ? null : included.get(buffer.getDMOProxy());
         Property[] xprops = excluded == null ? null : excluded.get(buffer.getDMOProxy());
         if (iprops != null || xprops != null)
         {
            DmoMeta meta = buffer.getDmoInfo();
            int size = meta.getFieldSize();
            Property[] props = new Property[size];
            
            // only permanent buffers can use include/exclude
            if (iprops == null)
            {
               // include all fields
               Iterator<Property> piter = meta.getFields(false);
               while (piter.hasNext())
               {
                  Property p = piter.next();
                  
                  if (p.propId > 0)
                  {
                     props[p.propId - 1] = p;
                  }
               }
            }
            else
            {
               System.arraycopy(iprops, 0, props, 0, size);
            }
            
            if (xprops != null)
            {
               // exclude properties
               for (int j = 0; j < size; j++)
               {
                  if (xprops[j] != null) props[j] = null;
               }
            }
            
            String alias = buffer.getDMOAlias();
            buf.append('.').append(DatabaseManager.PRIMARY_KEY);
            for (int j = 0; j < size; j++)
            {
               Property p = props[j];
               // extents are handled like scalars. The FQL converter will filter them out if necessary,
               // and the SQLQuery will fetch them from secondary tables at hydration time
               if (p == null)
               {
                  continue;
               }
               
               buf.append(", ").append(alias).append(".").append(p.name);
            }
         }
      }
   }
   
   /**
    * Assemble the from and the where clause for an FQL query. Now this method is capable of creating
    * the OUTER-JOIN clause as well. The class name of each DMO is decapitalized to produce
    * an alias for that object. Form of the clause is:
    * <pre>
    *    from {Buffer0} as {buffer0} 
    *    [[left outer] join {Buffer1} as {buffer1} on {JoinCondition1} [and {JoinCondition1_1}] [...]
    *    [, {Buffer2} as buffer2 [...]]
    *    [[left outer] join {Buffer3} as {buffer3} on {JoinCondition2} [and {JoinCondition2_1}] [...]]]
    *    [, {Buffer4} as buffer4 [...]]
    *    [where {WhereCondition1} [and {WhereCondition2}] [...]]]
    * </pre>
    * The FQL is creating by adding first the "from" KEYWORD, followed by:
    * 1) possible one or more different types of joins, each with one or more join conditions.
    * 2) not joined buffers.
    * At the end, the complete where clause is appended at the end of the FQL.
    * <p>
    *
    * @param   sortCriteria
    *          All known sort criteria. If a JOIN term is detected in the preprocessed query this list is
    *          simplified to allow a cleaner ORDER BY clause and let the query planner chose a better plan.
    *          
    * @return  FqlExpression
    *          The fql expression that contains both the from and where clause.
    * 
    * @throws  PersistenceException
    *          if there is an error creating the {@code FQLPreprocessor} from which the composite join data
    *          are gathered.
    */
   protected FQLExpression assembleFromAndWhereClauses(ArrayList<SortCriterion> sortCriteria)
   throws PersistenceException
   {
      FQLExpression buf = new FQLExpression();
      buf.append("from ");
      FQLExpression whereClause = new FQLExpression();
      ArrayList<QueryComponent> components = components();
      for (int i = 0; i < components.size(); i++)
      {

         QueryComponent comp = components.get(i);
         RecordBuffer buffer = comp.getBuffer();
         if (i > 0 && comp.isOuter())
         {
            buf.append(" left outer join ");
         }
         else if (i > 0)
         {
            buf.append(", ");
         }
         
         buf.append(buffer.getDMOImplementationName());
         
         if (!isOmitAlias())
         {
            buf.append(" as ");
            buf.append(buffer.getDMOAlias());
         }
         assembleWhereClause(sortCriteria, i, whereClause, buf, comp.isOuter() && i != 0);
        
      }
      if (!whereClause.isEmpty())
      {
         buf.append(whereClause);
      }

      buf.append(" ");
      
      // this set is used to avoid duplicating joins for composite tables if they occur as
      // normal joins required by the FQL preprocessor of each component and then needed by
      // joining the composite for soring purposes
      Set<String> joins = new HashSet<>(3);
      
      // add any ANSI joins required by the FQL preprocessor of each component
      components = components();
      for (int idx = 0; idx < components.size(); idx++)
      {
         QueryComponent comp = components.get(idx);
         Iterator<String> joinsIter = comp.compositeJoins();
         while (joinsIter.hasNext())
         {
            String aJoin = joinsIter.next();
            buf.append(aJoin);
            buf.append(" ");
            joins.add(aJoin);
         }
         
         // now simplify the [sortCriteria] is a server-side JOIN was detected while preprocessing the fql
         List<FQLPreprocessor.PropertyPair> joinPairs = comp.getFQLPreprocessor().getSubstPairs();
         if (joinPairs != null)
         {
            int delCount = 0;
            SortCriterion[] criteria = new SortCriterion[sortCriteria.size()]; 
            sortCriteria.toArray(criteria);
            
            for (int i = 0; i < criteria.length; i++)
            {
               if (criteria[i] == null)
               {
                  continue; // already dropped
               }
               
               for (FQLPreprocessor.PropertyPair pair : joinPairs)
               {
                  if (!pair.getOperator().equals("="))
                  {
                     continue; // ignore properties not joined with = operator 
                  }
                  
                  if (criteria[i].getOriginalName().equals(pair.getLeft()))
                  {
                     for (int j = i + 1; j < criteria.length; j++)
                     {
                        if (criteria[j] != null && criteria[j].getOriginalName().equals(pair.getRight()))
                        {
                           criteria[j] = null;
                           ++ delCount;
                           break; //j
                        }
                     }
                  }
                  else if (criteria[i].getOriginalName().equals(pair.getRight()))
                  {
                     for (int j = i + 1; j < criteria.length; j++)
                     {
                        if (criteria[j] != null && criteria[j].getOriginalName().equals(pair.getLeft()))
                        {
                           if (LOG_FINE)
                           {
                              LOG.log(Level.FINE, 
                                      "Dropped redundant sorting component " + criteria[j] + 
                                      ". The duplicate is " + criteria[i]);
                           }
                           
                           criteria[j] = null;
                           ++ delCount;
                           break; //j
                        }
                     }
                  }
               }
            }
            
            // check the remaining criteria
            if (delCount != 0)
            {
               for (int i = sortCriteria.size() - 1; i >= 0; i--)
               {
                  if (criteria[i] == null)
                  {
                     sortCriteria.remove(i);
                  }
               }
            }
         }
      }

      return buf;
   }
   
   /**
    * Assemble the restriction (i.e., where) clause for an FQL query. If more
    * than one component contains a where clause, they are combined with a
    * logical "and". Form of the clause is:
    * <pre>
    *    where ({component0_clause})
    *     [and ({component1_clause})
    *     [...]]
    * </pre>
    * <p>
    *
    * @param   sortCriteria
    *          List of <code>SortCriterion</code> objects which describe the
    *          sort behavior for this query.
    * @param   indices
    *          int which represents the position of the component we generate
    *          the where clause for.
    * @param   whereClause
    *          FQLExpression which represents the variable where the "where" clause is
    *          going to be saved.
    * @param   onClause
    *          FQLExpression which represents the variable where the "on" clause is
    *          going to be saved.
    * @param   isOn
    *          boolean that will help to distinguish between "where" and "and" and "on".
    *
    * @throws  PersistenceException
    *          if there is an error creating the <code>FQLPreprocessor</code>
    *          from which the preprocessed where clause is retrieved for each
    *          query component.
    */
   protected void assembleWhereClause(ArrayList<SortCriterion> sortCriteria, 
                                      int indices,
                                      FQLExpression whereClause, 
                                      FQLExpression onClause,
                                      boolean isOn) 
   throws PersistenceException
   {
      FQLExpression targetClause = isOn ? onClause : whereClause;
      ArrayList<QueryComponent> components = components();
      QueryComponent comp = components.get(indices);
      RecordBuffer rb = comp.getBuffer();
      boolean multiplex = rb.isMultiplexed();
      AbstractJoin join = comp.getJoin();
      FQLExpression subClause = comp.getWhere();
      if (subClause != null)
      {
         subClause.trim();
      }
      boolean hasJoin = (join != null);
      boolean hasWhere = subClause != null && !subClause.isEmpty();

      // if the component with a server join will specify the join in a subselect,
      // we don't want the join FQL to appear in the main where clause
      if (hasJoin && join.isServerJoin())
      {
         hasJoin = !comp.isJoinWithSubselect();
      }

      boolean hasContent = (hasJoin || multiplex || hasWhere);

      if (hasContent)
      {
         if (targetClause.isEmpty() && !isOn)
         {
            targetClause.append(" where (");
         }
         else if (!targetClause.isEmpty() && isOn)
         {
            targetClause.append(" on (");
         }
         else
         {
            targetClause.append(" and (");
         }
      }

      // Determine whether where clause should be augmented with an equality test for a
      // multiplex ID.  We need to do this when:
      //    a) we are querying a multiplexed temp table;  OR
      //    b) we are querying any temp table and have a non-empty where clause
      // but we skip it if that was already added in the subclause.
      if (multiplex && (subClause == null || !subClause.wasMultiplexed()))
      {
         if (hasJoin || hasWhere)
         {
            targetClause.append("(");
         }

         targetClause.append(rb.getDMOAlias());
         targetClause.append(".");
         targetClause.append(TemporaryBuffer.MULTIPLEX_FIELD_NAME);
         targetClause.append(" = ", true);

         if (hasJoin || hasWhere)
         {
            targetClause.append(") and ");
         }
      }
      
      // Add subclause for a join.
      if (hasJoin)
      {
         if (multiplex || hasWhere)
         {
            targetClause.append("(");
         }

         FQLExpression[][] fqls = join.getFQL();
         FQLExpression joinWhere = new FQLExpression();
         List<Serializable> params = join.getParameters();
         
         for (int j = 0; j < fqls.length; j++)
         {
            boolean nullArg = false;
            if (params != null)
            {
               Serializable val = params.get(j);
               if (val instanceof BaseDataType)
               {
                  BaseDataType param = (BaseDataType) val;
                  if (param.isUnknown())
                  {
                     nullArg = true;
                  }
               }
               else if (val == null)
               {
                  nullArg = true;
               }
            }
            
            joinWhere.append(
                     fqls[j][nullArg ? AbstractJoin.NULL_FIELD : AbstractJoin.NOT_NULL_FIELD]);

         }
         targetClause.append(joinWhere);

         if (multiplex || hasWhere)
         {
            targetClause.append(")");
         }

         if (hasWhere)
         {
            targetClause.append(" and ");
         }
      }
      
      if (hasWhere)
      {
         if (hasJoin || multiplex)
         {
            targetClause.append("(");
         }

         targetClause.append(subClause);

         if (hasJoin || multiplex)
         {
            targetClause.append(")");
         }
      }
      
      if (hasContent)
      {
         targetClause.append(") ");
      }
   }
   
   /**
    * Assemble the {@code order by} clause for an FQL query.
    * <p>
    * Any text-based sort criteria are augmented with the necessary upper() and rtrim() functions.
    *
    * @return  List of {@code SortCriterion} objects which describe the sort behavior for this query.
    *
    * @throws  PersistenceException
    *          if the raw sort clause provided at construction contains any unqualified property names;
    *          if the raw sort clause contains a reference to an unrecognized record buffer.
    */
   protected ArrayList<SortCriterion> assembleOrderByClause()
   throws PersistenceException
   {
      if (sort == null && dynamicSorts == null)
      {
         return EMPTY_ARRAY;
      }
      
      ArrayList<SortCriterion> ret = new ArrayList<>();
      
      if (dynamicSorts != null)
      {
         StringBuilder dynSort = new StringBuilder();
         for (Iterator<String> it = dynamicSorts.iterator(); it.hasNext(); )
         {
            dynSort.append(it.next());
            if (it.hasNext())
            {
               dynSort.append(", ");
            }
         }
         ret.addAll(assembleOrderByClause(dynSort.toString()));
         
         if (sort == null)
         {
            return ret;
         }
      }
      
      ret.addAll(assembleOrderByClause(sort));
      return ret;
   }
   
   /**
    * Generates the {@code order by} clause starting from the list of criteria, including the "order by"
    * keyword. Uses the form:
    * <pre>
    *   order by {{dmo.property} {asc/desc}} [, ...]
    * </pre>
    * <p>
    *
    * @param   sortCriteria
    *          The list of criteria to be added.
    * @param   buf
    *          The string builder to add the result.
    */
   private void generateOrderBy(ArrayList<SortCriterion> sortCriteria, StringBuilder buf)
   {
      if (sortCriteria == null)
      {
         return; // quick out
      }
      
      boolean orderBy = true;
      for (int i = 0; i < sortCriteria.size(); i++)
      {
         SortCriterion criterion = sortCriteria.get(i);
         if (orderBy)
         {
            buf.append(" order by ");
            orderBy = false;
         }
         else
         {
            buf.append(", ");
         }
         buf.append(criterion.toString());
      }
   }
   
   /**
    * Assemble the {@code order by} clause for an FQL query.
    * <p>
    * Any text-based sort criteria are augmented with the necessary upper() and rtrim() functions.
    *
    * @param   sort
    *          Raw sort clause to parse and augment if necessary.
    *
    * @return  List of {@code SortCriterion} objects which describe the sort behavior for this query.
    *
    * @throws  PersistenceException
    *          if {@code sort} contains any unqualified property names;
    *          if {@code sort} clause contains a reference to an unrecognized record buffer.
    */
   protected List<SortCriterion> assembleOrderByClause(String sort)
   throws PersistenceException
   {
      List<SortCriterion> sortCriteria = buildSortCriteria(sort);
      
      // assemble the order by clause, given the list of SortCriterion objects.
      return assembleOrderByClause(sortCriteria);
   }
   
   /**
    * Returns a list <code>SortCriterion</code> objects build from the given sort clause.
    *
    * @param   sort
    *          Raw sort clause to parse and augment if necessary.
    *
    * @return  List of <code>SortCriterion</code> objects which describe the
    *          sort behavior for this query.
    *
    * @throws  PersistenceException
    *          if <code>sort</code> contains any unqualified property names;
    *          if <code>sort</code> clause contains a reference to an
    *          unrecognized record buffer.
    */
   protected ArrayList<SortCriterion> buildSortCriteria(String sort)
   throws PersistenceException
   {
      // map record buffers used in the sort clause to their identifying aliases
      Map<String, RecordBuffer> aliases = new HashMap<>();
      ArrayList<QueryComponent> components = components();
      for (int i = 0; i < components.size(); i++)
      {
         QueryComponent comp = components.get(i);
         RecordBuffer buffer = comp.getBuffer();
         aliases.put(buffer.getDMOAlias(), buffer);
      }
      
      // Parse the sort clause, extracting each qualified DMO property
      // reference, matching it to its buffer (by alias), and creating a
      // SortCriterion object for it.  Also group lists of SortCriterion
      // objects by buffer, so that each query component can store its portion
      // of the sort clause.
      Map<RecordBuffer, List<SortCriterion>> subSortMap = new HashMap<>();
      ArrayList<SortCriterion> sortCriteria = new ArrayList<>();
      String[] parts = sort.split(",");
      for (int i = 0; i < parts.length; i++)
      {
         String part1 = parts[i];
         String part = part1.trim();
         int pos = part.indexOf(".");
         if (pos < 0)
         {
            throw new PersistenceException(
               "Order by clause contains unqualified property name:  " + part);
         }
         
         // Sanity.
         String alias = part.substring(0, pos);
         RecordBuffer buffer = aliases.get(alias);
         if (buffer == null)
         {
            throw new PersistenceException(
               "Order by clause references an unrecognized buffer:  " + alias + " (" + part + ")");
         }
         
         // Find or create the sub-list in which to store SortCriterion
         // objects for the current buffer.
         List<SortCriterion> subSortCrit = subSortMap.get(buffer);
         if (subSortCrit == null)
         {
            subSortCrit = new ArrayList<>();
            subSortMap.put(buffer, subSortCrit);
         }
         
         // create the SortCriterion instance and add it to the master list and to the appropriate sub-list
         SortCriterion crit = new SortCriterion(buffer, part);
         
         sortCriteria.add(crit);
         subSortCrit.add(crit);
      }
      
      // Match the sort criteria sub-list for each buffer, generate a normalized sort clause for
      // that sub-list, and store it in the appropriate query component.
      for (Map.Entry<RecordBuffer, List<SortCriterion>> next : subSortMap.entrySet())
      {
         RecordBuffer buffer = next.getKey();
         QueryComponent comp = findComponent(buffer);
         assert (comp != null);
         List<SortCriterion> subSortCrit = next.getValue();
         String subSort = SortCriterion.toRawSortPhrase(subSortCrit);
         SortIndex sortIndex = SortIndex.get(subSort, buffer);
         comp.setSortIndex(sortIndex);
      }
      
      return sortCriteria;
   }
   
   /**
    * Assemble the {@code order by} clause for an FQL query.
    * <p>
    * Any text-based sort criteria are augmented with the necessary upper() and rtrim() functions.
    *
    * @param   sortCriteria
    *          A list of {@code SortCriterion} objects which describe the sorting to be applied.
    *
    * @return  List of {@code SortCriterion} objects which describe the sort behavior for this query.
    */
   protected ArrayList<SortCriterion> assembleOrderByClause(List<SortCriterion> sortCriteria)
   {
      RecordBuffer buffer = getRecordBuffers()[0];
      boolean multiplex = (buffer.getMultiplexID() != null);
      ArrayList<SortCriterion> workList;
      
      if (sortCriteria == null)
      {
         if (multiplex)
         {
            workList = new ArrayList<>(1);
         }
         else
         {
            return null;
         }
      }
      else
      {
         workList = new ArrayList<>(sortCriteria.size() + 1);
      }
      
      if (multiplex)
      {
         // the _multiplex is injected so that the query to benefit from the matching index
         
         Boolean ascIndex = IndexHelper.get(buffer.getDatabase()).getIndexDirection(buffer, sort);
         StringBuilder sb = new StringBuilder();
         sb.append(buffer.getDMOAlias()).append(".").append(TemporaryBuffer.MULTIPLEX_FIELD_NAME);
         sb.append(ascIndex == null || ascIndex ? " asc" : " desc");
         
         try
         {
            workList.add(new SortCriterion(buffer, sb.toString()));
         }
         catch (PersistenceException exc)
         {
            LOG.log(Level.SEVERE, "Error adding multiplex ID to order by clause", exc);
         }
      }
      
      if (sortCriteria != null)
      {
         workList.addAll(sortCriteria);
      }
      
      return workList;
   }
   
   /**
    * Get this query's client-side where expressions.
    *
    * @return  Unmodifiable where expression list.
    */
   protected List<Supplier<logical>> getWhereExpressions()
   {
      return whereExprList == null ? Collections.emptyList() : Collections.unmodifiableList(whereExprList);
   }
   
   /**
    * Get an iterator on this query's client-side where expressions.
    *
    * @return  Where expression iterator.
    */
   protected Iterator<Supplier<logical>> whereExpressions()
   {
      return getWhereExpressions().iterator();
   }
   
   /**
    * Compose an FQL query statement to represent the combination of all query
    * components, execute it, and store the scrollable results.  If query has
    * been executed previously and we already have a result set, simply
    * return.
    *
    * @return  <code>true</code> if query was executed for the first time;
    *          <code>false</code> if query was executed previously and we
    *          already have a result set.
    *
    * @throws  ErrorConditionException
    *          if an error occurs executing the query.
    * @throws  IllegalStateException
    *          if no query component has been added to this query.
    */
   protected boolean execute()
   {
      // translate the sort criterion
      if (sort != null && !sortTranslated)
      {
         sortTranslated = true;
         
         ArrayList<QueryComponent> boundComponents = null;
         for (int i = 0; i < components.size(); i++)
         {
            QueryComponent component = components.get(i);
            RecordBuffer buffer = component.getBuffer();
            RecordBuffer definition = component.getDefinitionBuffer();
            if (!buffersMatch(buffer, definition))
            {
               if (boundComponents == null)
               {
                  boundComponents = new ArrayList<>();
               }
               
               boundComponents.add(component);
            }
         }
         
         if (boundComponents != null)
         { 
            // temporary switch to the definition buffer, get the sort criteria, then switch back
            int boundComponentsSize = boundComponents.size();
            RecordBuffer[] bound = new RecordBuffer[boundComponentsSize];
            RecordBuffer[] definition = new RecordBuffer[boundComponentsSize];
            int idx = 0;
            for (int i = 0; i < boundComponentsSize; i++)
            {
               QueryComponent component = boundComponents.get(i);
               RecordBuffer defBuffer = component.getDefinitionBuffer();
               bound[idx] = component.getBuffer();
               definition[idx] = defBuffer;
               idx = idx + 1;
               
               component.setBuffer(defBuffer);
            }
            
            List<SortCriterion> sorts;
            try
            {
               sorts = buildSortCriteria(sort);
            }
            catch (PersistenceException e)
            {
               throw new RuntimeException(e);
            }
            
            sort = translateSort(bound, definition, sorts);
            
            idx = 0;
            for (int i = 0; i < boundComponentsSize; i++)
            {
               boundComponents.get(i).setBuffer(bound[idx++]);
            }
         }
      }

      if (results != null)
      {
         return false;
      }
      
      if (components.isEmpty())
      {
         throw new IllegalStateException(
            "Preselect query must contain at least one query component");
      }
      
      if (!verifyJoins())
      {
         return true;
      }
      
      Persistence persistence = getPersistence();
      
      if (!regWithSession)
      {
         // Register to receive session closing notification.
         persistence.registerSessionListener(this, SessionListener.Scope.NONE, isSharedContext());
         regWithSession = true;
      }
      
      // if all buffers in the query are against temp-tables, fetch full records
      boolean allTemp = true;
      ArrayList<QueryComponent> components = components();
      for (int i = 0; allTemp && i < components.size(); i++)
      {
         allTemp = components.get(i).getBuffer().isTemporary();
      }
      
      if (allTemp)
      {
         setFullRecords();
      }
      
      Results r = null;
      try
      {
         // assemble FQL statement
         fql = assembleFQL();
         
         // Prepare substitution parameters.
         List<Object> argList = new ArrayList<>();
         prepareParameters(argList);
         Object[] args = argList.toArray();
         
         if (LOG_FINE)
         {
            LOG.log(Level.FINE, "FQL:  " + fql);
            StringBuilder buf = new StringBuilder("PARMS:  {");
            for (int i = 0; i < args.length; i++)
            {
               if (i > 0)
               {
                  buf.append(", ");
               }
               Object next = args[i];
               if (next instanceof BaseDataType)
               {
                  buf.append(((BaseDataType) next).toStringMessage());
               }
               else
               {
                  buf.append(next);
               }
            }
            buf.append("}");
            LOG.log(Level.FINE, buf.toString());
         }
         
         // possibly leak uncommitted updates to other sessions
         Iterator<QueryComponent> iter = components().iterator();
         while (iter.hasNext())
         {
            QueryComponent next = iter.next();
            FQLPreprocessor fqlPreproc = next.getFQLPreprocessor();
            Set<String> earlyPublish = fqlPreproc.getEarlyPublishEntities();
            if (!earlyPublish.isEmpty())
            {
               RecordBuffer buffer = next.getBuffer();
               DirtyShareContext dirtyContext = buffer.getDirtyContext();
               if (dirtyContext != null)
               {
                  List<Record> dmos = dirtyContext.updateSnapshots(earlyPublish, persistence);
                  
                  // Make sure DMOs do not bloat the ORM session.
                  for (Record dmo : dmos)
                  {
                     buffer.evictDMOIfUnused(dmo);
                  }
               }
            }
         }
         
         Long templateRowid = getTemplateQueryRowid(args);
         if (templateRowid != null)
         {
            r = new TemplateResults(templateRowid);
            setResults(r);
         }
         else
         {
            // execute the primary query
            r = P2JQueryExecutor.getInstance().execute(getRecordBuffers()[0].getDatabase(),
                                                       PreselectQuery::executeQuery, 
                                                       this,
                                                       persistence, 
                                                       fql,
                                                       args);
            r = createResults(r);
            setResults(r);
            
            registerCleaner();
         }
      }
      catch (PersistenceException exc)
      {
         ErrorManager.recordOrThrowError(exc);
      }
      finally
      {
         rowCount = -1;
      }
      
      return true;
   }
   
   /**
    * Detect whether the components of the query are shared or private from multi-tenant point of view.
    * In case of conflict, an error is logged.
    * 
    * @return  {@code true} if the buffers of teh query are (all) shared.
    */
   protected boolean isSharedContext()
   {
      int contextType = -1;
      for (QueryComponent comp : components())
      {
         boolean mt = comp.getBuffer().getDmoInfo().multiTenant;
         if (mt)
         {
            if (contextType == Persistence.SHARED_IDX)
            {
               contextType = -2;
               break;
            }
            contextType = Persistence.PRIVATE_IDX;
         }
         else
         {
            if (contextType == Persistence.PRIVATE_IDX)
            {
               contextType = -2;
               break;
            }
            contextType = Persistence.SHARED_IDX;
         }
      }
      
      if (contextType < 0)
      {
         LOG.log(Level.WARNING, "MTCtx: Failed TENANT context detection inPreselectQuery!");
      }
      
      return contextType == Persistence.SHARED_IDX;
   }
   
   /**
    * Indicate whether results resources should be cleaned up upon exiting
    * the global scope or upon exiting the nearest enclosing scope in which
    * the results were created.
    * <p>
    * This default implementation returns <code>false</code>, indicating
    * results should be cleaned up at the nearest enclosing scope.
    * Subclasses which require different behavior must override this method.
    *
    * @return  <code>false</code>.
    */
   protected boolean isTopLevelCleanup()
   {
      return false;
   }
   
   /**
    * Invoked just before query execution, this method checks whether all
    * components with a non-null JOIN have a valid inverse on which to perform
    * the join.  If not, set the query's results to an empty result set and
    * indicate failure.
    * 
    * @return  <code>true</code> if verification succeeded and query execution
    *          should continue;  else <code>false</code>
    */
   protected boolean verifyJoins()
   {
      ArrayList<QueryComponent> components = components();
      for (int i = 0; i < components.size(); i++)
      {
         QueryComponent comp = components.get(i);
         AbstractJoin join = comp.getJoin();
         
         if (join != null && !join.isServerJoin())
         {
            BufferReference ref = join.getInverse();
            Record record = ref.buffer().getCurrentRecord();
            
            if (record == null)
            {
               handleMissingJoinRecord();
               
               return false;
            }
         }
      }
      
      return true;
   }
   
   /**
    * Respond to the condition that a component with a non-null JOIN has a
    * missing inverse on which to perform the join.
    * <p>
    * This implementation creates an empty result set.
    */
   protected void handleMissingJoinRecord()
   {
      setResults(new SimpleResults(null));
   }
   
   /**
    * Attempt to cache all results from the specified target result set into a {@link Results}
    * instance. If this is a scrolling query, this will cache all results in the target result
    * set, including those already visited. Otherwise, only the current result and all remaining,
    * unvisited results are cached.
    * 
    * @param   target
    *          Result set which is to be cached, presumably because it will be invalid
    *          momentarily. If results previously were cached, {@code target} is simply returned.
    * @param   forceIdOnly
    *          {@code true} if the records should be cached only by their PK and not force full
    *          results iteration to hydrate the DMOs.
    * 
    * @return  Results from {@code target}, if any, cached in a {@code Results} object. If there
    *          were no results available to cache, an empty result set is returned.  If any of
    *          the record buffers backing this query have been closed, {@code null} is returned.
    */
   protected Results cacheResults(Results target, boolean forceIdOnly)
   {
      if (isResultSetCached())
      {
         // results are already cached
         return target;
      }
      
      // If any buffer associated with this query is closed, do not
      // attempt to cache results.  Instead, clean up resources;  this
      // query is done.
      ArrayList<QueryComponent> components = components();
      for (int i = 0; i < components.size(); i++)
      {
         QueryComponent comp = components.get(i);
         RecordBuffer buffer = comp.getBuffer();
         if (buffer.getOpenScopeCount() < 1)
         {
            resetResults();
            
            return null;
         }
      }
      
      // Remember current row.
      int currentRow = (isScrolling() ? target.getRowNumber() : 0);
      
      // Cache all preselected results in a SimpleResults object and set
      // that as the query's new backing results.
      ArrayList<Object[]> rows = new ArrayList<>(64);
      
      int numberOfRowsCached = 0;
      if (isScrolling())
      {
         target.reset();
         while (target.next())
         {
            boolean forceIdLenientCache = ++numberOfRowsCached > LENIENT_CACHING_THRESHOLD ? forceIdOnly : false;
            rows.add(target.get(forceIdLenientCache || this.forceOnlyId));
         }
      }
      else if (target.getRowNumber() == -1)
      {
         currentRow = -1;
         while (target.next())
         {
            rows.add(target.get(++numberOfRowsCached > LENIENT_CACHING_THRESHOLD ? forceIdOnly : false));
         }
      }
      else if (target.get() != null)
      {
         do
         {
            rows.add(target.get(++numberOfRowsCached > LENIENT_CACHING_THRESHOLD ? forceIdOnly : false));
         } while (target.next());
      }
      
      rows.trimToSize();
      
      // Override SimpleResults.cleanup() to do nothing, so we don't
      // immediately discard results when the current scope ends and our
      // Finalizable cleaner (installed in execute()) does its work.
      SimpleResults cached = new SimpleResults(rows)
      {
         public void cleanup() {}
      };
      
      // Reset to current row.
      cached.setRowNumber(currentRow);
      
      return cached;
   }
   
   /**
    * Prepare query substitution parameters for this query.  One argument must be added to
    * {@code args} list for each {@code ?} placeholder in the FQL query string for this query.
    *
    * @param   args
    *          Argument list to be populated.
    * @throws  PersistenceException
    *          if there is an error creating an {@code FQLPreprocessor} during the gathering of
    *          query substitution parameters.
    */
   protected void prepareParameters(List<Object> args)
   throws PersistenceException
   {
      List<Object> notOuterArgs = new ArrayList<>();
      for (int i = 0; i < components.size(); i++)
      {  
         QueryComponent next = components.get(i);
         RecordBuffer rb = next.getBuffer();
         List<Object> targetArgs = next.isOuter() && i > 0 ? args : notOuterArgs;
         if (rb.isMultiplexed())
         {        
            targetArgs.add(rb.getMultiplexID()); 
         }
         
         AbstractJoin join = next.getJoin();
         if (join != null)
         {
            List<Serializable> parameters = join.getParameters();
            if (parameters != null)
            {
               for (int j = 0; j < parameters.size(); j++)
               {
                  Serializable val = parameters.get(j); 
                  if (val instanceof BaseDataType)
                  {
                     BaseDataType param = (BaseDataType) val;
                     if (!param.isUnknown())
                     {
                        targetArgs.add(param);
                     }
                  }
                  else if (val != null)
                  {
                     targetArgs.add(val);
                  }
               }
            }
         }
         
         Object[] cArgs = next.getArgs();
         if (cArgs != null)
         {
            if (parameterFilter != null)
            {
               cArgs = parameterFilter.filterParameters(cArgs);
            }
            
            for (int j = 0; j < cArgs.length; j++)
            {
               if (cArgs[j] instanceof Resolvable)
               {   
                  targetArgs.add(((Resolvable) cArgs[j]).resolve());
               }
               else if (cArgs[j] instanceof P2JQuery.ParamResolver)
               {
                  targetArgs.add(((P2JQuery.ParamResolver) cArgs[j]).resolve());
               }
               else
               {
                  targetArgs.add(cArgs[j]);
               }
            }
         }
      }
      args.addAll(notOuterArgs);
   }
   
   /**
    * Collect the names of the entities (DMO types) associated with this query.
    * 
    * @return  Array of entity names.
    */
   protected Class<? extends Record>[] getEntities()
   {
      int len = components.size();
      Class<? extends Record>[] entities = new Class[len];
      for (int i = 0; i < len; i++)
      {
         entities[i] = components.get(i).getBuffer().getDMOImplementationClass();
      }
      
      return entities;
   }
   
   /**
    * Checks if this is a template query. A template query is a rowid query that looks for a
    * special template record that is not found in database. The template record is a record
    * having all fields set to default values. The id of such record is in P2J, by convention
    * negative.
    * In case this is a simple template query, this method returns the rowid of the template for
    * current buffer from the single.
    * <p>
    * <b>Note:</b><br>
    * The current implementation identifies only simple queries for template records:
    * <ol>
    *    <li>this {@code PreselectQuery} has a single component;</li>
    *    <li>the where predicate directly test whether the rowid/recid is equal to a value
    *          (this is done by the {@link FQLPreprocessor} of the component);</li>
    *    <li>the value (constant or passed in as query SUBST parameter) is negative (as noted
    *          above).</li>
    * </ol>
    * Although this kind of queries are sufficient for standard template record load according to
    * P4GL idiom, more complex queries can be issued, with predicates that basically match the
    * pattern but add supplementary AND/OR clauses. This kind of queries are not always of common
    * sense and the result in 4GL is not always as expected.<br>
    * For example:
    *       <p>{@code FOR EACH book WHERE RECID(book) = templateId OR book.isbn NE ?}</p>
    * will iterate only books that have the isbn not null, but will not contain the template
    * record in the result returned.
    * <p>
    * On the other hand, it does not make much sense to:
    * <ul>
    *    <li>iterate over the the template records of a table since there will always be at most
    *       one such record;</li>
    *    <li>add additional OR/AND clauses in WHERE predicate since, if the template record
    *       exists, it is well defined by it rowid, configured in database schema and
    *       immutable.</li>
    * </ul>
    * 
    * @param   args
    *          The actual parameters of the query.
    * 
    * @return  if this query looks for the template record of the component the rowid is returned
    *          and {@code null} otherwise. 
    * 
    */
   protected Long getTemplateQueryRowid(Object[] args)
   {
      if (components == null || components.size() != 1)
      {
         return null;
      }
      // TODO: need some attention:
      // - this query type is not only used for FOR EACH constructs. It is the base class for
      //       other query types as well. Need to check if template record changes applied to all
      //       these query types;
      
      FQLPreprocessor prep = null;
      try
      {
         prep = components.get(0).getFQLPreprocessor();
      }
      catch (PersistenceException e)
      {
         return null;
      }
      
      if (!prep.isFindByRowid())
      {
         return null;
      }
      
      long tId = prep.getFindByRowid(args);
      
      return tId < 0 ? tId : null;
   }
   
   /**
    * Execute the query to create a first level result set.
    * <p>
    * This default implementation returns an object which is a thin wrapper around
    * {@code ScrollableResults}.  Subclasses may override this method if they need a
    * {@code Results} implementation other than {@code ScrollingResults}.
    * <p>
    * In case when the {@code fql} represents a valid template record query (a test on recid /
    * rowid of the record against the unique {@code _file._template} for current record of the
    * single {@code QueryComponent}), a special {@link Results} object is returned with a single
    * element that is the template record for the buffer of the component.
    *
    * @param   persistence
    *          Object which is used to execute the query.
    * @param   fql
    *          FQL query string.
    * @param   args
    *          Substitution parameters to be inserted into placeholders within the FQL query string.
    *
    * @return  An instance of {@code ScrollingResults} which wrappers the 
    *          {@code ScrollableResults} object returned by the execution of the query.
    *
    * @throws  PersistenceException
    *          if there is an error executing the query.
    */
   protected Results executeQuery(Persistence persistence, String fql, Object[] args)
   throws PersistenceException
   {
      Long templateRowid = getTemplateQueryRowid(args);
      if (templateRowid != null)
      {
         return new TemplateResults(templateRowid);
      }
      
      return shouldExecuteForward() ?
             new ForwardResults(persistence, executeScroll(persistence, fql, args))  :              
             new ScrollingResults(persistence, executeScroll(persistence, fql, args));
   }
   
   /**
    * Execute the query to create a {@code ScrollableResults} result set.
    *
    * @param   persistence
    *          Object which is used to execute the query.
    * @param   fql
    *          FQL query string.
    * @param   args
    *          Substitution parameters to be inserted into placeholders within
    *          within the FQL query string.
    *
    * @return  The {@code ScrollableResults} object returned by the execution of the query.
    *
    * @throws  PersistenceException
    *          if there is an error executing the query.
    */
   protected <T> ScrollableResults<T> executeScroll(Persistence persistence, String fql, Object[] args)
   throws PersistenceException
   {
      int scrollMode = isScrolling()
                       ? ResultSet.TYPE_SCROLL_INSENSITIVE
                       : ResultSet.TYPE_FORWARD_ONLY;
      
      return persistence.scroll(getEntities(), fql, args, 0, 0, getDataRelation(), scrollMode);
   }
   
   /**
    * Optionally wrapper the result set created for the query with a filtering
    * <code>Results</code> implementation (for client-side where processing).
    * It is this object which actually will be used to retrieve the primary
    * key values of records to be loaded from the database.
    * <p>
    * If no filtering is necessary, simply return the results object which
    * was passed in.
    * <p>
    * Subclasses may override this method to insert their own layers into
    * result set processing.
    *
    * @param   results
    *          An object which manipulates the database result set returned
    *          from execution of the query.
    *
    * @return  <code>Results</code> object.
    *
    * @throws  PersistenceException
    *          if there is any error creating results.
    */
   protected Results createResults(Results results)
   throws PersistenceException
   {
      if (isClientWhere())
      {
         return (new FilteredResults(results,
                                     Arrays.asList(getRecordBuffers()),
                                     getWhereExpressions(),
                                     !fullRecords));
      }
      
      return results;
   }
   
   /**
    * Fetch all records for a given result row into their backing buffers.  Each result row
    * contains one column per query component.  Each column contains the unique ID of the record
    * to be fetched or the record itself. If the latter, the record is refetched in case it has
    * changed since the record was first retrieved.  If the scrollable cursor backing this query
    * currently is at a position where no retrieval is possible, the backing buffers will have
    * their current records released.
    * <p>
    * As particular case when the query looks for the template record of the buffer of the single
    * {@link QueryComponent}, the buffer is not populated with a record fetched from backing
    * database, instead the template record for the buffer is automatically created in-memory.
    * This {@code results} object has only one element to iterate on and attempting to iterate on
    * this object will throw {@link QueryOffEndException} and release the buffer. 
    *
    * @param   available
    *          {@code true} if the most recent scrollable cursor move left the cursor at an
    *          available result row;  {@code false} if the cursor cannot read a result at its
    *          current position.
    * @param   lockType
    *          Overriding lock type to apply to fetched records.  If {@code null}, default lock
    *          types are applied.
    * @param   errorIfNull
    *          {@code true} if failed retrievals should raise error.
    * 
    * @throws  MissingRecordException
    *          if the record was expected to be available, but wasn't.  This typically indicates
    *          the record was deleted between the time it initially was found by the query and the
    *          time it was to be fetched to store in the record buffer.
    * @throws  ErrorConditionException
    *          if the fetch of any record fails or if no records are available and the query is
    *          set to fail in this case.
    * @throws  QueryOffEndException
    *          if no records are available and the query is set not to fail in this case.
    */
   protected void fetch(boolean available, LockType lockType, boolean errorIfNull)
   throws MissingRecordException
   {
      currentRowDeleted = false;
      
      try
      {
         // under certain circumstances, we must bypass releasing the current record(s) (if any)
         // in the buffer(s), so we throw the off-end exception directly instead of setting the
         // current buffer record to null
         if (!available   &&
             rowCount < 0 &&
             !errorIfNull &&
             _isOffEnd()  &&
             ((isPresort() && isIterating()) ||
              preserveBuffersOnEmptyResults(results)))
         {
            if (!lenientOffEnd)
            {
               getRecordBuffers()[0].throwOffEnd();
            }
            
            return;
         }
         
         // check if a template search was issued. In this case we have only a single component.
         // We will load the template record in the respective buffer.
         if (results instanceof TemplateResults)
         {
            QueryComponent comp = components.get(0);
            RecordBuffer buffer = comp.getBuffer();
            if (available)
            {
               buffer.loadTemplateRecord(results.getID(0));
            }
            else
            {
               // will release the buffer and throw QueryOffEndException 
               buffer.setRecord(null, 
                                lockType,
                                false,
                                lenientOffEnd,
                                comp.getSortIndex(),
                                offEnd,
                                inverseSorting,
                                false,
                                false);
            }
         }
         else
         {
            // invalidated queries always fetch with LockType.NONE
            // SHARE or EXCLUSIVE locks needs updateLock in coreFetch
            boolean alreadyFetched = available && 
                                     results.isAutoFetch() &&
                                     !shouldUpdateAnyLock(lockType);
            if (!alreadyFetched)
            {
               Object[] data = available ? results.get(forceOnlyId) : null;
               coreFetch(data, lockType, available, errorIfNull, false);
            }
            
            if (available)
            {
               accumulate();
            }
         }
      }
      catch (QueryOffEndException exc)
      {
         if (!lenientOffEnd)
         {
            throw exc;
         }
      }
      catch (MissingRecordException exc)
      {
         throw exc;
      }
      catch (PersistenceException exc)
      {
         ErrorManager.recordOrThrowError(exc);
      }
   }
   
   /**
    * Fetch all records for a given result row into their backing buffers, given an array of data which
    * contains one primary key or DMO per query component.
    *
    * @param   data
    *          An array of primary key IDs or DMOs representing records to be fetched for the composite row.
    * @param   lockType
    *          Overriding lock type to apply to fetched records.  If {@code null}, default lock types are
    *          applied.
    * @param   errorIfNull
    *          {@code true} if failed retrievals should raise error.
    * @param   silentIfNullId
    *          If {@code true}, do not raise {@link QueryOffEndException} if some of the provided IDs are
    *          {@code null}.{@code null} IDs are valid for queries with OUTER join.
    *
    * @return  {@code true} if data was fetched and is available and {@code false} is the fetch failed and no
    *          error was raised (when {@code errorIfNull} is false).
    *
    * @throws  MissingRecordException
    *          if the record was expected to be available, but wasn't.  This typically indicates the record
    *          was deleted between the time it initially was found by the query and the time it was to be
    *          fetched to store in the record buffer.
    * @throws  PersistenceException
    *          if the fetch of any record fails.
    * @throws  ErrorConditionException
    *          if no records are available and the query is set to fail in this case.
    * @throws  QueryOffEndException
    *          if no records are available and the query is set not to fail in this case.
    */
   protected final boolean coreFetch(Object[] data,
                                     LockType lockType,
                                     boolean available,
                                     boolean errorIfNull,
                                     boolean silentIfNullId)
   throws PersistenceException
   {
      int recFetched = 0;
      QueryOffEndException throwOffEndException = null;
      ArrayList<QueryComponent> components = components();
      for (int i = 0; i < components.size(); i++)
      {
         QueryComponent comp = components.get(i);
         RecordBuffer buffer = comp.getBuffer();
         boolean dirtyCopy = false;
         Record dmo = null;
         LockType lt = lockType;
         if (data != null && data[i] != null && !currentRowDeleted)
         {
            LockType compLock = comp.getLockType();
            
            // Use default lock type if not overridden.
            if (lt == null)
            {
               lt = compLock;
            }
            
            // Individual result is either a DMO (i.e., Record) or a primary key ID.
            Object o = data[i];
            Long id = null;
            boolean updateLock;
            boolean fullRec = o instanceof Record;
            
            if (fullRec)
            {
               dmo = (Record) o;
               id = dmo.primaryKey();
               updateLock = (lt != LockType.NONE);
            }
            else
            {
               id = (Long) o;
               updateLock = true;
            }
            
            // if the DMO currently in the buffer has the same ID as the ID
            // we wish to load, but also represents a dirty copy (a copy of
            // uncommitted changes in another session), leave it in the buffer
            if (buffer.isDirtyCopy())
            {
               Record dirtyDMO = buffer.getCurrentRecord();
               if (dirtyDMO != null && id.equals(dirtyDMO.primaryKey()))
               {
                  dirtyCopy = true;
                  dmo = dirtyDMO;
               }
            }
            
            DirtyShareContext dirtyContext = DirtyShareSupport.isEnabledCrossSession() || DirtyShareSupport.isEnabledIntraSession() ?
                                                buffer.getDirtyContext() : null;
            String entity = buffer.getEntityName();
            
            // if we have an ID only, a cached DMO, or we are working with a temp-table, fetch
            // the current version of the record (dirty DMO copy excepted)
            boolean doFetch = (!fullRec || isResultSetCached() || !buffer.isTemporary()) && !dirtyCopy;
            
            if (doFetch)
            {
               // a record may have been retrieved from a cache of detached objects, OR its
               // state may have been changed (updated or deleted) since we first retrieved it,
               // so reload it by its ID
               Persistence p = getPersistence();
               if (updateLock)
               {
                  FibonacciCounter fib = new FibonacciCounter();
                  long timeout = fib.next();
                  
                  do
                  {
                     try
                     {
                        // force the load of the entire record
                        dmo = p.load(buffer.getDMOImplementationClass(), id, lt, timeout, updateLock, null);
                       
                        break;
                     }
                     catch (LockTimeoutException exc)
                     {
                        if (LOG_FINE)
                        {
                           String table = buffer.getTable();
                           RecordIdentifier<String> ident = new RecordIdentifier<>(table, id);
                           LOG.fine("[" +
                                    Utils.describeContext() +
                                    "] coreFetch lock timeout: " +
                                    timeout +
                                    " ms: " +
                                    ident);
                        }
                        
                        // check if record was deleted in uncommitted transaction in another session
                        if (dirtyContext != null && dirtyContext.isDirtyDelete(entity, id, true))
                        {
                           String table = buffer.getTable();
                           RecordIdentifier<String> ident = new RecordIdentifier<>(table, id);
                           
                           throw new MissingRecordException(ident);
                        }
                        
                        // steadily increase the timeout value
                        timeout = fib.next();
                        
                        // log deadlock warning if timeout is getting too long and wait
                        // indefinitely
                        if (timeout > 3600000L)
                        {
                           timeout = 0L;
                           if (LOG.isLoggable(Level.WARNING))
                           {
                              String table = buffer.getTable();
                              RecordIdentifier<String> ident = new RecordIdentifier<>(table, id);
                              LOG.log(Level.WARNING,
                                      "[" +
                                      Utils.describeContext() +
                                      "] Possible deadlocked record: " + ident,
                                      new Throwable());
                           }
                        }
                     }
                  } while (true);
               }
               else
               {
                  dmo = p.load(buffer.getDMOImplementationClass(), 
                               id, 
                               lt, 
                               0L, 
                               updateLock, 
                               // the entire record is loaded if the lock is not NONE
                               updateLock || dmo == null ? null : dmo._getReadFields());
               }
            }
            
            if (LOG_FINE)
            {
               StringBuilder buf = new StringBuilder("DMO:  ");
               if (id != null)
               {
                  buf.append(id);
               }
               else
               {
                  buf.append("<not found>");
               }
               
               LOG.log(Level.FINE, buf.toString());
               
               if (LOG.isLoggable(Level.FINEST))
               {
                  buf.setLength(0);
                  buf.append("DMO Detail:  ");
                  buf.append(dmo);
                  
                  LOG.log(Level.FINEST, buf.toString());
               }
            }
            
            // If we expected this record to be available, but it wasn't, it must have been deleted.
            // Likewise, if we know it was deleted in another context, this is an exceptional condition.
            // Throw an exception so the caller can figure out the appropriate recovery.
            if (dmo == null                                                          ||
                dirtyContext != null && dirtyContext.isDirtyDelete(entity, id, true) ||
                fullRec && dmo.checkState(DmoState.DELETED))
            {
               if (_isSkipDeletedRecord() || isFillingBrowseRows())
               {
                  String table = buffer.getTable();
                  RecordIdentifier<String> ident = new RecordIdentifier<>(table, id);
                  
                  throw new MissingRecordException(ident);
               }
               else
               {
                  // implementation of SKIP-DELETED-RECORD for -dynamic- queries. If positioned on a record
                  // which was deleted after being preselected, do not skip it, instead empty the buffer
                  // if the query attribute was set to false.
                  // NOTE: although the documentation expressly mentions that this attribute affects only
                  //       the dynamic queries, from my testcases, they seems to behave the same.
                  buffer.release(false);
                  continue; // with next buffer / component (main for)
               }
            }
         }
         
         // TODO: confirm or fix behavior:  first null record will raise error or end condition;
         //       remaining buffers will not have their current records released.
         SortIndex sortIndex = comp.getSortIndex();
         boolean outerJoin = comp.isOuter() && dmo == null  && i > 0;
         try
         {
            if (dmo != null ||
                (outerJoin && available) ||
                (comp.getIteration() != QueryConstants.LAST && 
                 comp.getIteration() != QueryConstants.FIRST))
            {               
               buffer.setRecord(dmo,
                        lt,
                        errorIfNull || silentIfNullId,
                        lenientOffEnd,
                        sortIndex,
                        offEnd,
                        inverseSorting,
                        dirtyCopy,     // TODO:  need better dirty check
                        false);
            }
            
            if (outerJoin)
            {
               buffer.setUnknownMode();
            }
            
            if (dmo != null)
            {               
               ++recFetched;
            }
            
            Buffer bufferImpl = (Buffer) buffer.getDMOProxy();
            if (bufferImpl.autoSynchronize().booleanValue())
            {
               bufferImpl.querySynchronize();
            }
         }
         catch (QueryOffEndException ex)
         {
            if (!this.lenientOffEnd && i == components.size() - 1)
            {
               throw ex;
            }
         }
         
         if (dmo == null && errorIfNull)
         {
            buffer.errorNotOnFile();
         }
      }
      
      return recFetched != 0;
   }
   
   /**
    * Indicate whether this query requires initial results to be filtered
    * through a client-side where clause expression.
    *
    * @return  <code>true</code> if client-side filter is required, else
    *          <code>false</code>.
    */
   protected final boolean isClientWhere()
   {
      return (!getWhereExpressions().isEmpty());
   }
   
   /**
    * Access the current <code>Results</code> object for this query.
    *
    * @return  <code>Results</code> object.
    */
   protected Results getResults()
   {
      execute();
      
      return results;
   }
   
   /**
    * Add a component to the query.
    *
    * @param   comp
    *          Query component to be added.
    *
    * @throws  UnsupportedOperationException
    *          if attempting an outer join (temporary);
    *          if iteration type is not <code>NEXT</code> (temporary);
    *          if the component's DMO's associated database is different than
    *          that of a previously added component's database (cross-database
    *          preselect joins are not possible).
    */
   protected void addComponent(QueryComponent comp)
   {
      if (isClientWhere())
      {
         throw new IllegalStateException(
            "Cannot have a multi-table query with a client-side where clause");
      }

      switch (comp.getIteration())
      {
         case NEXT:
         case FIRST:
         case LAST:
            break;
         default:
            if (!supportsArbitraryIteration())
            {
               throw new UnsupportedOperationException(
                  "Preselect query supports only NEXT/FIRST/LAST iteration");
            }
            break;
      }
      
      boolean isFirst = components.isEmpty();
      RecordBuffer buffer = comp.getBuffer();
      
      if (buffer.getOpenScopeCount() > 0)
      {
         buffer.initialize();
         
         // if there is a record waiting on the queried index in the record nursery, make it visible;
         // flush the record buffer, in case a transient, newly created record currently is in the buffer
         try
         {
            // TODO: find a more direct route to the numeric index ID that does not involve a lookup by name
            String idxName = buffer.getIndexHelper().getIndexForSort(buffer, sort);
            int idxId = buffer.getDmoInfo().getRecordMeta().getIndexIdByName(idxName);
            RecordNursery nursery = buffer.getNursery();
            nursery.makeVisible(buffer.getDmoInfo().getId(), buffer.getMultiplexID(), idxId);
            buffer.flush();
         }
         catch (ValidationException | PersistenceException exc)
         {
            // a validation error can only result from flushing an unrelated record currently in
            // the buffer in preparation for a query result; since this will not be within the
            // context of any silent error mode under which this query is running, we instruct
            // the error manager to not suppress the message
            ErrorManager.throwError(exc, false);
         }
         
         if (isFirst)
         {
            thisFinalizable = true;
            // Register for retry support.
            TransactionManager.registerFinalizable(this, false);
         }
         // Verify that no cross-connection join is being attempted.
         else if (!getPersistence().equals(buffer.getPersistence()))
         {
            throw new UnsupportedOperationException(
               "Preselect query cannot join across database connections");
         }
      }
      
      if (!isFirst)
      {
         comp.setNotTop();
      }
      
      // Add the component.
      components.add(comp);
      
      // Update cumulative substitution argument count.
      Object[] args = comp.getRawArgs();
      if (args != null)
      {
         int len = args.length;
         for (int i = 0; !resolvableArgs && i < len; i++)
         {
            if (args[i] instanceof Resolvable)
            {
               resolvableArgs = true;
            }
         }
      }
   }
   
   /**
    * Does this query support non-NEXT iteration components?
    * 
    * @return  <code>false</code>
    */
   protected boolean supportsArbitraryIteration()
   {
      return false;
   }
   
   /**
    * Indicate whether the record buffer(s) current record(s) should be preserved, in the event
    * the query does not find any results.
    * <p>
    * This implementation always returns false. Subclasses should override this behavior if
    * needed.
    * 
    * @param   results
    *          The results instance to decide if they are truly empty. The results are not visible
    *          in some sub-classes, so let them be passed by parameter.
    * 
    * @return  <code>false</code>.
    */
   protected boolean preserveBuffersOnEmptyResults(Results results)
   {
      return false;
   }
   
   /**
    * Clear the reposition cache of visited, virtual composite rows.
    */
   protected void clearRepoCache()
   {
      repoCache.clear();
   }
   
   /**
    * Get the FQL query which was executed to produce this query's results.
    *
    * @return  FQL query or <code>null</code> if the query has not yet been executed.
    */
   protected String getHQL()
   {
      return fql;
   }
   
   /**
    * Get the persistence service object used by this query.
    * 
    * @return  Persistence object.
    */
   protected Persistence getPersistence()
   {
      if (persistence == null)
      {
         // Since no cross-connection joins are permitted, it is safe to set
         // the query's Persistence instance using the first component only.
         RecordBuffer buffer = components.get(0).getBuffer();
         persistence = buffer.getPersistence();
      }
      
      return persistence;
   }
   
   /**
    * Register a <code>Finalizable</code> with the transaction manager for
    * resource cleanup when this query's scope ends.
    */
   protected void registerCleaner()
   {
      // If we are not a standalone query, we assume our container will
      // register for cleanup.
      if (regCleaner || !isStandalone())
      {
         return;
      }
      
      if (isTopLevelCleanup())
      {
         TransactionManager.registerTopLevelFinalizable(cleaner, true);
      }
      else
      {
         TransactionManager.registerFinalizable(cleaner, false);
      }
      
      regCleaner = true;
   }
   
   /**
    * Reposition the cursor such that a request to retrieve the next result
    * will retrieve the result which matches the specified array of primary
    * key IDs.  A request to retrieve the previous result will retrieve the
    * result immediately previous to this, if any, in the results list.
    *
    * @param   ids
    *          Array of primary key IDs, arranged from left to right to
    *          coincide with the records being joined by the underlying
    *          query.
    * @param   forceFetch
    *          <code>true</code> to override the current fetch-on-reposition
    *          setting to <code>true</code>, in case the underlying
    *          <code>Results</code> implementation attempts to fetch the
    *          record during repositioning;  <code>false</code> to honor the
    *          current setting.
    * 
    * @return  <code>true</code> if reposition was successful;
    *          <code>false</code> if unsuccessful.
    *
    * @throws  ErrorConditionException
    *          if there is an error repositioning the cursor.
    */
   protected boolean repositionByID(Long[] ids, boolean forceFetch)
   {
      integer oldCurrentRow = currentRow();

      prepareReposition();
      
      if (results == null)
      {
         execute();
      }
      
      ids = DBUtils.trimToTableCount(ids, this);
      int newRow = repoCache.getRow(ids);
      if (newRow >= 0)
      {
         int oldRow = results.getRowNumber();
         boolean avail;
         if (forceFetch)
         {
            boolean oldFetch = isFetchOnReposition();
            try
            {
               setFetchOnReposition(true);
               avail = results.setRowNumber(newRow);
            }
            finally
            {
               setFetchOnReposition(oldFetch);
            }
         }
         else
         {
            avail = results.setRowNumber(newRow);
         }
         if (avail)
         {
            offEnd = OffEnd.NONE;
            betweenRows = true;
         }
         else
         {
            int diff = oldRow - newRow;
            offEnd = (diff < 0 ? OffEnd.FRONT : OffEnd.BACK);
            betweenRows = false;
         }
         
         afterReposition(true);
         
         return true;
      }
      
      // Scan all results for a match.  Start from where cache last left off.
      boolean error = false;
      int size = repoCache.size();
      results.setRowNumber(size);
      error = results.getRowNumber() == -1;
      if (!error && results.get(0) == null)
      {
         // result set is fully cached (without a match) or it is empty
         error = size > 0 || !results.first();
      }
      
      int width = getTableCount();
      if (!error)
      {
         // For a partial match (number of provided IDs is less than the number of query tables)
         // 4GL *in most cases* returns the *last* row in the subset. E.g. "1 2 4" is returned for
         // the request "1" here:
         // 1 1 2
         // 1 2 3
         // 1 2 4
         // That may not be true for large data sets. 4GL documentation states that the order is
         // unpredictable.
         //
         // FWD caches in repoCache only a part of the full result set, up to the max row found
         // by IDs search because of performance considerations. In order to implement finding
         // of the last row, the following mechanics are implemented:
         // 1. FWD fetches at least one extra row to know where the target subset ends.
         // 2. FWD fetches more rows to get the full subset for the outer component into the cache
         // in order to to avoid putting the part of the following subsets into the cache. This is
         // necessary to avoid quick-out check of repoCache where non-last component is mistakenly
         // returned for the subset. E.g. in the following situation "2 1 2" is returned for the
         // request "2" while it is correct to return "2 1 3":
         // 1 1 2 <- in cache
         // 1 2 3 <- in cache
         // 1 2 4 <- in cache
         // 2 1 2 <- in cache
         // 2 1 3

         Integer partialMatchRow = null;
         Integer targetRow = null;
         Serializable outerId = null;
         do
         {
            boolean match = true;
            Long[] rsIDs = new Long[width];
            for (int i = 0; i < width; i++)
            {
               Long nextID = results.getID(i);
               rsIDs[i] = nextID;
               if (ids.length > i && ids[i] != null && !ids[i].equals(nextID))
               {
                  match = false;
               }
            }

            if (targetRow != null)
            {
               // we already found the target row, scroll to the end of outermost subset
               if (outerId.equals(rsIDs[0]))
               {
                  repoCache.add(rsIDs, results.getRowNumber());
                  continue;
               }
               else
               {
                  break;
               }
            }

            if (match)
            {
               repoCache.add(rsIDs, results.getRowNumber());
               outerId = rsIDs[0];
               if (rsIDs.length > ids.length)
               {
                  //partial match
                  partialMatchRow = results.getRowNumber();
               }
               else
               {
                  // complete match
                  targetRow = results.getRowNumber();
               }
            }
            else if (partialMatchRow != null)
            {
               // previous row was the last partial match in the subset, so it is our target
               if (outerId.equals(rsIDs[0]))
               {
                  repoCache.add(rsIDs, results.getRowNumber());
               }
               targetRow = partialMatchRow;
            }
            else
            {
               repoCache.add(rsIDs, results.getRowNumber());
            }
         }
         while (results.next());

         if (targetRow == null && partialMatchRow != null)
         {
            targetRow = partialMatchRow;
         }

         if (targetRow != null)
         {
            results.setRowNumber(targetRow);
            offEnd = OffEnd.NONE;
            betweenRows = true;
            afterReposition(true);
            return true;
         }
      }
      
      offEnd = OffEnd.BACK;
      betweenRows = false;
      afterReposition(false, oldCurrentRow);

      return false;
   }
   
   /**
    * Prepare to process a reposition request by releasing the current
    * records, if any, in each buffer managed by the query.  This occurs in
    * Progress unconditionally, even if the reposition does not actually move
    * the cursor from its current position.
    */
   protected void prepareReposition()
   {
      // Don't release buffer if we're processing a recursive reposition that
      // was triggered by a reposition notification.
      if (isRepositionNotificationActive())
      {
         return;
      }

      if (components != null)
      {
         for (int i = 0; i < components.size(); i++)
         {
            QueryComponent next = components.get(i);
            next.getBuffer().release(false);
         }
      }
   }
   
   /**
    * Check if any query component (added until the current call) has bound buffers.
    * 
    * @return   See above.
    */
   protected boolean anyBoundBufferInComponent()
   {
      for (int i = 0; i < components.size(); i++)
      {
         QueryComponent qc = components.get(i);
         RecordBuffer buf = qc.getBuffer();
         RecordBuffer defBuf = qc.getDefinitionBuffer();
         if (!buffersMatch(buf, defBuf))
         {
            return true;
         }
      }
      
      return false;
   }
   
   /**
    * Check if the lock should be updated for any component when fetching. This is used
    * to detect if we can bypass fetching to satisfy locks. If the provided lock type is
    * {@link LockType#NONE}, no lock update should happen. If the component specific 
    * lock mode is used, ensure that there is no particular (SHARE or EXCLUSIVE) lock 
    * request for any component, which may trigger fetching.
    * 
    * @param   lockType
    *          The lock mode for the records after query execution.
    *          
    * @return  {@code true} if one of the components has a record which should be fetched
    *          with another lock type.
    */
   protected boolean shouldUpdateAnyLock(LockType lockType)
   {
      if (lockType == LockType.NONE)
      {
         return false;
      }
      
      if (lockType != null)
      {
         return true;
      }
      
      for (int i = 0; i < components.size(); i++)
      {
         QueryComponent comp = components.get(i);
         LockType compLock = comp.getLockType();
         
         if (compLock != LockType.NONE)
         {
            return true;
         }
      }
      
      return false;
   }
   
   /**
    * Decide whether this query should have a {@link ForwardResults} back-end results provider
    * or it should eventually toggle a {@link ScrollingResults}
    * 
    * 
    * @return   {@code true} if this will generate {@link ForwardResults}
    */
   protected boolean shouldExecuteForward()
   {
      return !isScrolling() && isIterating();
   }
   
   /**
    * Find the query component associated with the given record buffer.
    *
    * @param   buffer
    *          Record buffer.
    *
    * @return  Associated query component, or <code>null</code> if not found.
    */
   private QueryComponent findComponent(RecordBuffer buffer)
   {
      ArrayList<QueryComponent> components = components();
      for (int i = 0; i < components.size(); i++)
      {
         QueryComponent comp = components.get(i);
         if (comp.getBuffer() == buffer)
         {
            return comp;
         }
      }
      
      return null;
   }
   
   /**
    * Assemble the FQL which is submitted to Hibernate in order to execute
    * the database query underlying this object. The FQL generated uses the
    * theta join style and includes a select clause, a from clause, an
    * optional where clause, and an order by clause (mandatory in all cases
    * except a unique find).
    *
    * @return  String
    *          A formatted FQL statement.
    *
    * @throws  PersistenceException
    *          if the sort criteria reference unqualified or unrecognized
    *          buffer names.
    */
   private String assembleFQL()
   throws PersistenceException
   {
      StringBuilder buf = new StringBuilder();

      translateWhere();
      
      if (FwdServerJMX.JMX_DEBUG)
      {
         QUERY_ASSEMBLE.timer(() -> assembleFQLImpl(buf));
      }
      else
      {
         assembleFQLImpl(buf);
      }
      
      return buf.toString().trim();
   }
   
   /**
    * Assemble the FQL which is submitted to Hibernate in order to execute
    * the database query underlying this object. The FQL generated uses the
    * theta join style and includes a select clause, a from clause, an
    * optional where clause, and an order by clause (mandatory in all cases
    * except a unique find).
    *
    * @param   buf
    *          The buffer where to assemble the FQL.
    *
    * @throws  PersistenceException
    *          if the sort criteria reference unqualified or unrecognized
    *          buffer names.
    */
   private void assembleFQLImpl(StringBuilder buf)
   throws PersistenceException
   {
      // need to put together order by clause first, as the sort criteria determined in this analysis may be
      // needed when composing the from-clause and where clause
      ArrayList<SortCriterion> sortCriteria = assembleOrderByClause();
      
      assembleSelectClause(buf);
      buf.append(assembleFromAndWhereClauses(sortCriteria).toFinalExpression());
      
      if (isOrderByRequired())
      {
         generateOrderBy(sortCriteria, buf);
      }
   }
   
   /**
    * Function for deciding whether the ORDER BY clause is required in the FQL.
    * For the moment, the ORDER BY clause can be skipped if this query will retrieve zero or one records.
    * In other words, if this query will retrieve a unique finding, no ORDER BY is necessary.
    * 
    * @return  {@code true} if the ORDER BY clause is necessary for the query,
    *          {@code false} otherwise.
    */
   private boolean isOrderByRequired()
   throws PersistenceException
   {
      ArrayList<QueryComponent> components = components();
      for (int i = 0; i < components.size(); i++)
      {
         if (!components.get(i).getFQLPreprocessor().isUniqueFind(true))
         {
            return true;
         }
      }
      
      return false;
   }
   
   /**
    * Validate a row identifier provided to a reposition request to ensure
    * that it does not represent the unknown value.
    *
    * @param   value
    *          ID to be validated.
    *
    * @return  <code>true</code> if the number is valid;  <code>false</code>
    *          if it is invalid, but we are in silent error mode.
    *
    * @throws  ErrorConditionException
    *          if the number is invalid and we are not in silent error mode.
    */
   private boolean validateRepositionByID(rowid value)
   {
      String msg = "Could not evaluate rowid during REPOSITION";
      
      return validateRepositionImpl(value, 7330, msg);
   }
   
   /**
    * Validate a record identifier provided to a reposition request to ensure
    * that it does not represent the unknown value.
    *
    * @param   value
    *          ID to be validated.
    *
    * @return  <code>true</code> if the number is valid;  <code>false</code>
    *          if it is invalid, but we are in silent error mode.
    *
    * @throws  ErrorConditionException
    *          if the number is invalid and we are not in silent error mode.
    */
   private boolean validateRepositionByID(recid value)
   {
      String msg = "Could not evaluate rowid during REPOSITION";
      
      return validateRepositionImpl(value, 7330, msg);
   }
   
   /**
    * Validate a numeric value provided to a reposition request to ensure
    * that it does not represent the unknown value.
    *
    * @param   value
    *          ID or row number/offset to be validated.
    *
    * @return  <code>true</code> if the number is valid;  <code>false</code>
    *          if it is invalid, but we are in silent error mode.
    *
    * @throws  ErrorConditionException
    *          if the number is invalid and we are not in silent error mode.
    */
   private boolean validateReposition(NumberType value)
   {
      String msg = "Could not evaluate reposition amount for query";
      
      return validateRepositionImpl(value, 3165, msg);
   }
   
   /**
    * Validate a numeric value provided to a reposition request to ensure
    * that it does not represent the unknown value.
    *
    * @param   value
    *          ID or row number/offset to be validated.
    * @param   errorNum
    *          Error number if validation fails.
    * @param   errorMsg
    *          Error message if validation fails.
    *
    * @return  <code>true</code> if the number is valid;  <code>false</code>
    *          if it is invalid.
    *
    * @throws  ErrorConditionException
    *          if the number is invalid and we are not in silent error mode.
    */
   private boolean validateRepositionImpl(BaseDataType value, int errorNum, String errorMsg)
   {
      if (value.isUnknown())
      {
         ErrorManager.displayError(errorNum, errorMsg, false);
         
         if (LOG.isLoggable(Level.SEVERE))
         {
            LOG.log(Level.SEVERE, errorMsg);
         }
         
         return false;
      }
      
      return true;
   }
   
   /**
    * Indicate whether the full result set for this query has been cached.
    * 
    * @return  {@code true} if there is a result set and it has been cached.
    */
   private boolean isResultSetCached()
   {
      return results != null && results.isResultSetCached();
   }
   
   /**
    * Special {@link Results} implementation for template records. Since a query on rowid/recid
    * will return only a single row, this class is built around its {@code Long} rowid
    * and provide fast access for a {@link PreselectQuery} aware of this query type to iterate
    * on this single-row result.
    */
   private static class TemplateResults
   implements Results
   {
      /** The id of the template record to be loaded. */
      private final Long templateRowid;
      
      /** 
       * The position of the cursor. This set of results always contain a single record, at
       * position 0. The natural starting position is -1, before the first row. The value 1 is
       * just after the last row. The class will maintain one of these three states.
       */
      private int cursor = -1;
      
      /**
       * The constructor. Builds a {@link Results} specific to a template rowid.
       * 
       * @param   templateRowid
       *          The rowid of the template record. Immutable.
       */
      public TemplateResults(Long templateRowid)
      {
         this.templateRowid = templateRowid;
      }
      
      /**
       * Move cursor to the first results row.
       *
       * @return  {@code true} if there are any results.
       */
      @Override
      public boolean first()
      {
         cursor = 0;
         return true; 
      }
      
      /**
       * Move cursor to the last results row.
       *
       * @return  {@code true} if there are any results.
       */
      @Override
      public boolean last()
      {
         cursor = 0;
         return true;
      }
      
      /**
       * Move cursor to the next results row.
       *
       * @return  {@code true} if there is a result under the cursor under the move.
       */
      @Override
      public boolean next()
      {
         ++ cursor;
         normalize();
         return isFirst();
      }
      
      /**
       * Move cursor to the previous results row.
       *
       * @return  {@code true} if there is a result under the cursor under the move.
       */
      @Override
      public boolean previous()
      {
         -- cursor;
         normalize();
         return isFirst();
      }
      
      /**
       * Is the cursor on the first row in the results set?
       *
       * @return  {@code true} if the cursor is on the first row.
       */
      @Override
      public boolean isFirst()
      {
         return cursor == 0;
      }
      
      /**
       * Is the cursor on the last row in the results set?
       *
       * @return  {@code true} if the cursor is on the first row.
       */
      @Override
      public boolean isLast()
      {
         return cursor == 0;
      }
      
      /**
       * Returns the row at the current position. It can be either only an array of PK or
       * an array of DMOs. 
       * 
       * @param   forceOnlyId
       *          Indicates whether it's only the primary keys that must be returned, 
       *          or original data found is returned (either PK or DMO).
       *
       * @return  Object array or {@code null}.
       */
      @Override
      public Object[] get(boolean forceOnlyId)
      {
         return (cursor == 0) ? new Object[] { templateRowid } : null; 
      }
      
      /**
       * Get the object at the current result row, at the specified column.
       *
       * @param   column
       *          Zero-based index column of the desired object.
       *
       * @return  Object at {@code column} or {@code null}.
       */
      @Override
      public Object get(int column)
      {
         return (cursor == 0 && column == 0) ? templateRowid : null;
      }
      
      /**
       * Get the primary key ID at the current result row, at the specified
       * column.
       *
       * @param   column
       *          Zero-based index column of the desired ID.
       *
       * @return  ID of the record or {@code null}.
       */
      @Override
      public Long getID(int column)
      {
         return (cursor == 0 && column == 0) ? templateRowid : null;
      }
      
      /**
       * Get the row number currently under the cursor.
       *
       * @return  Zero-based index of the current row, or {@code -1} if the cursor is not 
       *          currently on a result.
       */
      @Override
      public int getRowNumber() { return cursor == 0 ? 0 : -1; }
      
      /**
       * Set the row number currently under the cursor.
       *
       * @param   row
       *          Zero-based index of the row to be set as the current row.
       *
       * @return  {@code true} if there is a row at the specified row number; else {@code false}.
       */
      @Override
      public boolean setRowNumber(int row)
      {
         cursor = row;
         normalize();
         
         return isFirst();
      }
      
      /**
       * Scroll the cursor ahead by the specified number of rows.  If the number is negative, the
       * cursor is moved backward.  This may put the cursor off the end (either end) of the
       * query's results.
       *
       * @param   rows
       *          Number of rows to jump ahead or back.
       *
       * @return  {@code true} if there is a row at the new location; else {@code false}.
       */
      @Override
      public boolean scroll(int rows)
      {
         cursor += rows;
         normalize();
         
         return isFirst();
      }
      
      /**
       * Reset the cursor to its natural starting position, before the first result row.
       */
      @Override
      public void reset()
      {
         cursor = -1;
      }
      
      /**
       * Invoked when the current Hibernate session is about to close.  In this case this is a
       * no-op since the template record is not linked to Hibernate.
       */
      @Override
      public void sessionClosing()
      {
      }
      
      /**
       * Clean up and release any resources which this object is holding, such as open result
       * sets. In this case it does nothing as no resources are held.
       */
      @Override
      public void cleanup()
      {
      }
      
      /**
       * Internal worker that keeps the correctness of cursor position: always in one of the
       * positions: -1 = before the first record, 0 = the only record is selected, +1, after the
       * last record.
       */
      private void normalize()
      {
         if (cursor > 1)
         {
            cursor = 1;
         }
         else if (cursor < -1)
         {
            cursor = -1;
         }
         // else cursor is -1 (before), 0 (on single record), 1 (after)
      }
   }
}