Bug #9683
CompoundQuery is doing extra hydration attempts
80%
Related issues
History
#1 Updated by Alexandru Lungu over 1 year ago
- Subject changed from CompoundQuery is doing one extra hydration attempt to CompoundQuery is doing extra hydration attempts
Results.getRow:
- 21.3k come from
PreselectQuery.fetchand they are critical to hydrate the DMO and load the record inside the buffer. - 21.3k come from
CompoundQuery.processComponentafter running the underlying query. This is used to return the actual row retrieved by the query.- this is odd, because
peekmode is set to false. The javadoc states that:* @return If
peekistrue: an array of * the primary key IDs for the query component, if a result was * successfully retrieved;nullif no more results * were available.<p> * Ifpeekisfalse: always returns *null. - this is not true, due to the unconditioned by
peer:if (available) { return query.getRow(); }
- this is odd, because
- 21.3k come from
CompoundQuery.collectRowData. This is used when!offEndand!peerand it computes the return value ofretrieveImpl. This value is relevant only in the context of scrolling results as the ids are pushed onto thecursor. This is also indicated by the javadoc:* Assemble an array of primary key IDs or DMOs for the current records in * the buffers underlying this query, from left-most to right-most (in the * sense of how the query joins the associated tables). This method * essentially takes a lightweight snapshot of the currently loaded * records, so that this may be restored later. * <p> * This snapshot is only generated for scrolling queries.- Yet, this doesn't seem to hold as there is no conditional for this. Also, my profiled case uses the CQ in a FOR EACH, so it is not scrolling!
From my POV, 66% of the Results.getRow calls are redundant. The performance problem is that getRow attempt to hydrate the record! It will not actually use the rs because it can be found in the session cache, yet the Session.getCached call is time consuming (generates a RecordIdentifier instance with its hashCode and uses it inside a LRUCache).
- make
CompoundQuery.processComponentbehave properly according to the javadoc. Please identify why thereturn query.getRow();statement was introduced and if it can be removed entirely.- I am aware of the
if (subData == null)that comes afterprocessComponent, yet at that point we only want to know if we found something or not, not the actual data that should be gathered withquery.getRow(). - This topic requires extra attention and tests
- I am aware of the
- avoid calls to
@CompoundQuery.collectRowData@ if the query is non-scrolling. Find when this invariant was broken and why.- This topic requires extra attention and tests
#2 Updated by Alexandru Lungu over 1 year ago
- Assignee set to Artur Școlnic
- Priority changed from Normal to High
#3 Updated by Artur Școlnic over 1 year ago
- Status changed from New to WIP
#4 Updated by Artur Școlnic over 1 year ago
- Status changed from WIP to Review
- reviewer Alexandru Lungu added
Indeed there are callers of processComponent and collectRowData that do not use the row data, only the fact that it was found. In this cases, replacing getRow with a simple Object[1] to act as a flag seems to do the trick.
The code is in 9583a/15724.
#5 Updated by Alexandru Lungu over 1 year ago
- Status changed from Review to WIP
- % Done changed from 0 to 50
- for
collectRowData, if the query is non-scrolling and off-end, the code still returnsnew Object[1]indicating that something was actually retrieved. I agree that this is no a viable execution path because the caller ofcollectRowDatachecks that the query is notoffEnd, but for the sake of encapsulation, lets havecollectRowDatareturn null in case of off-end.
I must admit that I suggested (in a talk with Artur) the usage of new Object[1] as a viable marker for non-scrolling queries that something was retrieved, but realistically speaking, it was a very naive prototype to suggest that we don't need the getRow calls for non-scrolling queries. However, I think this task needs way more attention and testing overall because the changes affect the very core of CompoundQuery.
First and foremost, CompoundQuery can have multiple components so that the caller of retrieveImpl might expect to get an array of "N" elements instead of 1. I tend to agree that this execution path is, again, purely hypothetical, but in practice there are >10 FWD API points that end up in retrieveImpl (current, first, next, previous, last, first * their peek counter-parts + preselect case) and I can't tell if all such API are guaranteed to behave as we expect with the output of retrieveImpl.
Optional<Object[]> to properly mimic the viable outputs of retrieveImpl in a more type-safe manner. But, again, this was a naive prototype. Maybe the best way to go here is with a RetrieveResult object that will properly store the output of processComponent or/and retrieveImpl. I even see a better API like:
- hasRow() -> true/false whether the last retrieved row was
null. - getRow() -> that will actually encapsulate the
collectRowDatalogic
All consumers of processComponent and retrieveImpl will have to adapt to the new signature and handle RetrieveResult properly. In an ideal implementation, only scrolling queries will make use of getRow, while non-scrolling will use hasRow. The more exotic preselectResults can stay as it is (as a user of getRow) if we can't properly state that a non-scrolling query can't preselect.
CompoundQuery API in the context of RetrieveResult. I am not aware how to generate a CompoundQuery in preselect mode .. you need to spend some time to determine this
- I think you should generate a
CompoundQuerythat is browsed in an enhanced browse and you sort by a column in the UI.
#6 Updated by Artur Școlnic over 1 year ago
I committed an early implementation for the RetrieveResults, the code is in 9683a/15725. Code formatting, javadoc and proper encapsulation are not done yet.
#7 Updated by Alexandru Lungu over 1 year ago
9683a seems to be on the right track.
At this is in an early stage, I may say that it has quite a high cognitive complexity, mostly due to the high number of conditionals:
RetrieveResultshas 8 class members and 4 constructors. The constructors are setting only 0, 1 or 2 such members.- this represents a problem because, for example,
hasRowis set only in 1/4 constructors.
- this represents a problem because, for example,
getRowhas around 6 if conditionals and 3 switch conditionals. These generate many possible execution paths (in theory) - so we need to ensure all execution paths are tested completely and work as expected.- I don't see a place where
cachedRowis instantiated, but I see several places where it is used.
- I don't see a place where
RetrieveResults as an interface or abstract class and rather implement CollectingRowDataResults, PreselectingRetrieveResults and PeekResults to honor the right case?
CollectingRetrieveResultswill implementcollectRowDataand will be returned only in the place wherecollectRowDatawas actually used in trunk.PeekingRetrieveResults,PreselectingRetrieveResultswill implement their own switch-case I suppose. However, I think these can be merged into one singleCachedRetrieveResults
This is just an early design idea to kickstart a plan for segregating RetrieveResults's implementation in smaller parts, so please feel free to interpret it as you need and continue implementation.
#8 Updated by Artur Școlnic over 1 year ago
I committed rev 15726. It contains a new interface and a few implementing classes.
#10 Updated by Alexandru Lungu over 1 year ago
Review of 9683a:
There is still a place:-
- Object[] ids = retrieve(NEXT, lockType, iterating);
+
+ Object[] ids = retrieve(NEXT, lockType, iterating).getRow();
if (isScrolling() && !isPresorted())
{
cursor.addResultNext(ids);
}
-
+
return ids != null;
}
- There are some javadoc to change that still state
Array of record IDs reflecting the first query result - Small misalignment on
CompoundQuery::processComponentinvoking. Other places as well. - implements should be on a separate line
- I am not a personal fan of having overloaded constructors that are not related. For instance, calling
hasRowmakes sense only if using theRetrievePeekingResults(boolean hasRow)c'tor. Otherwise, it defaults to false ... is this right? The same goes forretrieveModeandquerythat are initialized with 0 and null ... which I think are not valid states always. Or if this is right, then setting them explicitly and making the props final is cleaner.- same applies to
RetrieveCachedResultsandRetrievePreselectResults. Please stick to one single c'tor that sets the final props accordingly. You can overload them as long as they are calling each other until a master c'tor and you pass default values on the way.
- same applies to
- there are some open braces on the same line as the statements
#11 Updated by Artur Școlnic over 1 year ago
- Status changed from WIP to Review
I addressed the review and fixed a few bugs detected during testing, the code is in 9683a/15727.
#12 Updated by Artur Școlnic about 1 year ago
Addressed a few formatting issues and rebased on trunk rev 15866.
#13 Updated by Artur Școlnic about 1 year ago
Rebased on trunk rev 15891. Alexandru, could you please look at the last commits, I would like to move on with testing 9683a, if the review is ok.
#14 Updated by Alexandru Lungu about 1 year ago
implementsshould be aligned topublic(at 3 spaces from the start of the line).- Typo in
RetrievePeekingResults.retrieveModejavadoc (mod instead of mode). - please make all fields of such
Retrievableimplementationsfinal. This will enforce setting them from the constructor and not miss the right initialization of any field.- please attempt to have only one constructor that sets fields. All of the other ctors should only call the "master" ctor with default values.
- this applies to all Retrievable implementations
- I want to avoid the high dependency between what API is called (hasRow, getRow) to the constructor overload (with hasRow or retrieveMode and query).
- New line is required after
RetrieveCachedResults.cachedRow - I think the combination of a ctor that provides a cachedRow and a setter for the cachedRow confusing. I might expect to either initialize the results with a cached row or set it along. This suggests the need to have the cachedRow field as final.
- Shouldn't the peeknext and peekprevious return
Retrievable? Add the row to the cursor only by callinggetRowinside of that. For non-scrolling queries, you won't need to call getRow.
hasRow and getRow in sync with the following examples:
return new RetrievePeekingResults(true);is returning an instance of peeking results that have a row. However, a call togetRowon this results will yield NPE (becausehasRowis true, butqueryis null).return new RetrievePeekingResults(navigation, query);is returning an instance of peeking results that can retrieve a row. However, a call tohasRowisfalse.retrieve(CURRENT, lockType, false) == nullwas changed to!retrieve(CURRENT, lockType, false).hasRow(). This is right, but according to current implementation,hasRowandgetRoware sometimes out of sync.RetrievePreselectResultscan return null withgetRowin case it is non-scrolling and off-end. SohasRowcan also be false in that case.
#15 Updated by Alexandru Lungu about 1 year ago
- Status changed from Review to WIP
- % Done changed from 50 to 80
#16 Updated by Artur Școlnic about 1 year ago
Rebased 9683a on trunk rev 16047. Addressed review points in rev 16053.
#17 Updated by Artur Școlnic about 1 year ago
- Status changed from WIP to Review
#18 Updated by Greg Shah about 1 year ago
- Related to Feature #6720: lazy hydration added
#19 Updated by Greg Shah about 1 year ago
We need to finish the work in #6720.
#20 Updated by Artur Școlnic about 1 year ago
#21 Updated by Greg Shah about 1 year ago
I assume we will finish this task first, but I am highlighting that we really need to finish the general solution to this problem, which is in task #6720.
#22 Updated by Andrei Plugaru 10 months ago
- Related to Feature #6695: Multi-table preselect query may underperform due to repetitive fetching added