FWD » Core Development » Database

Consider this test:

def var hb as handle.
def var hq as handle.

create buffer hb for table "book".
create query hq.
hq:add-buffer(hb).
hq:query-prepare("for each book fields (isbn) no-lock").
hq:query-open().
hq:get-first().
message hb::book-title. // error

For static queries, FIELDS and EXCEPT can appear only at DEFINE QUERY. In dynamic query case, FWD appends them to the OPEN QUERY statement, from where they get ignored.

More, FWD doesn't convert properly the DEFINE QUERY ... FIELDS case, either:

define query q for book fields (isbn).
open query q for each book.
get first q.
message book.book-title.

This was found in standalone tests.

Committed on 7999a, revision 14833.
- DynamicQueryHelper, added methods to traverse fields and except phrases of record phrase of the query string, and set include and exclude variables of AbstractQuery.
- in CompundQuery, added methods to set include and exclude of child components.

Ovidiu: Please review.

Review of 7999a, r14833.

Good job. The goal of the task was reached. But I have some things to note, mainly because of performance. They may not seem as much, but they add to global application speed:

• CompoundQuery.java:
  • history header: try keeping the information compact. The two entries are highly redundant;
  • there is a common typo in the new method names setIncludedCompoents() / setExcludedCompoents;
  • use an local variable to cache included.get(buffers[j].getDMOProxy()) value used in 2 places;
  • the code in these methods are pretty much identical. I think we can create a parametrised method to merge both of them;
• DynamicQueryHelper.java:
  • history header: see above. The closing tag */ is broken. Luckily it is closed by the copyright comment;
  • the import list contains new unneeded and unwanted entries;
  • line 650 and 663: the cast to AbstractQuery is not needed, the include() and exclude() methods are declared in P2JQuery interface;
  • the for loops at lines 644 and 657 can be replaces with a bit faster iterations over the Map's entry set. For example:
    

                Set<Map.Entry<Buffer, String[]>> kvPairs = include.entrySet();
                for (Map.Entry<Buffer, String[]> pair : kvPairs)
                {
                   String[] fields = pair.getValue();
    
                   if (fields != null)
                   {
                      BufferImpl buffer = (BufferImpl) pair.getKey();
                      query0.include(buffer.buffer().getDMOProxy(), fields);
                   }
                }
    

    The creation and iteration of KeySet and EntrySet are very similar but in the latter case the Map.get() call is replaced by the simple getValue() getter. Also, note the include.get(buffer) was invoked twice in initial code;
  • in methods collectFieldsList() and collectExcludeList():
    • as above they share pretty much of their code so they are candidate for merging into a single one, to avoid code duplication.
    • line 1452: I think the correct code is propList[j] = field.name; (the property name), not fieldAast.getText() (the legacy field name);
    • does collectFieldsList() throw any Exception?
    • the errMsg. Please be sure the message is the same as in OE, including inner spaces and final dot. Please split it on left aligned substrings which do not pass the standard line limit (110);
    • the int nrChildren = chAst.getNumImmediateChildren(); / for (int i = 1; i < nrChildren; i++) / Aast bufParamAast = chAst.getChildAt(i); is not recommended from PoV of performance. getNumImmediateChildren() iterate children to count them, the getChildAt(i) iterates them again. Use instead a iteration starting with getFirstChild() and advance with getNextSibling. This is probably the fastest way. Alternatively, there is getImmediateChild(int type, Aast start) method in AnnotatedAst but it's slower. The same stands for the inner loop;
    • AFAIK, there can be only one KW_FIELD / KW_EXCEPT sub-node of RECORD_PHRASE. In this case it's normal to leave the outer loop as soon as the (first) node was encountered and processed;
    • when computing the bufname. Instead of bufname.contains("."), please compute bufname.lastIndexOf(".") first (returns -1 if the needle is not present in the stack), cache it and reuse in substring.

Eduard, please address the review to get this done.

Committed on 7999a, revision 14837.

Addressed review in #7999-7. Also for second case in #7999-1, with DEFINE QUERY ... FIELDS, added support at conversion phase to add include/exclude method call for query variable.

Eduard, I rebased 7999a to latest trunk. It is now at rev. 14958. Please do the testing of it with POC and a large customer application regression tests.
I see changes in database_access.rules. Are these mandatory and require reconversion?

Ovidiu, please review latest 7999a.

Alexandru Lungu wrote:

I see changes in database_access.rules. Are these mandatory and require reconversion?

It is required for support of DEFINE QUERY ... FIELDS, because for now a fields/except clause inside a DEFINE QUERY is not taken into consideration at conversion phase. And it requires reconversion.
I tested conversion of Hotel_Gui with this change.

Review of 7999a/14954-14958.
The code looks promising, most of the issues below are low priority. but there are a couple of things in DynamicQueryHelper which seem unfinished. Has the code been incorrectly merged when rebased?

• please update the copyright year in file header to all affected files, regardless of the date in H section;
• database_access.rules:
  • line 420: routines exposed in CommonAstSupport are automatically available to TRPL code, no need to create a special worker. Actually, I have not seen it in use.
  • line 1739: variable name, typo bufnameJavanmae instead of bufnameJavaname ?
• CompoundQuery.java:
  • typo: setIncludedCompoents, setFieldsCompoents, nrCompoents
  • line 3487-3492: I guess a more expressive (and compact) initialization for fieldsCompMap would be:
    Map<DataModelObject, Property[]> fieldsCompMap = isInclude ? included : excluded;
  • setFieldsCompoents() can be private I think;
• DynamicQueryHelper.java:
  • line 92: invalid multi-line comment terminator. Does this compile?
  • lines 153, 520, 521, 1309, 1316: rogue empty lines
  • line 1324: @param for token parameter is missing from method javadoc
  • line 1351: we have getImmediateChild(tokenType) but your code is locally optimized to seek a specific token type and I agree with it;
  • line 1368: byLegacyName() requires that the argument to be normalized (lowercase). You may test with an exclude/field name with not matching casing;
  • line 1377: in case of an invalid field name does OE attempt to recover by looking for another list of fields? is it possible to have multiple fields/exclude options?
  • in parse() method, I think something is missing. The include and exclude maps are defined (line 347) and at lines 637/653 they are checked for null. Where is the private method exclude() called?
• PreselectQuery.java: missing H entry

Ovidiu Maxiniuc wrote:

• line 1368: byLegacyName() requires that the argument to be normalized (lowercase). You may test with an exclude/field name with not matching casing;

I tested with not matching cases, and it works fine with byLegacyName.

• line 1377: in case of an invalid field name does OE attempt to recover by looking for another list of fields? is it possible to have multiple fields/exclude options?

I checked in OE, it is not possible to have multiple fields/except options.

• in parse() method, I think something is missing. The include and exclude maps are defined (line 347) and at lines 637/653 they are checked for null. Where is the private method exclude() called?

It is missing, I do not why it was deleted at some point.

 include = collectFields(pAst, buffers, ProgressParserTokenTypes.KW_FIELD);
 exclude = collectFields(pAst, buffers, ProgressParserTokenTypes.KW_EXCEPT);

I tested this changed on POC, and it was causing some regressions:
1) in query 2 consecutive buffers, where referenced to the same persistence table. for each Customer1 fields (name), each Customer2 fields (name), like in this case Customer1 and Customer2 reference the same table Customer.

This issue is solved by changes in FqlToSqlConverter.

2) a issue related to #8259.

Commited on 7999a, revision 14959.

When the changes from this task are regression tested, please also include the following scenario mentioned in #10763:

def query q1 for a fields (a1).
open query q1 for each a no-lock.
get first q1 exclusive-lock.
message a.a1 a.a2.

We need to verify whether the get first q1 exclusive-lock operation reloads the full record.
This is important because an exclusive-lock on a query defined with a fields clause should trigger a full record fetch, even though the query was originally opened with NO-LOCK.

I have rebased the 7999a, to latest trunk. rev. 16321.

Added changes to handle the case when the query AST is retrieved from cache. Plus the #7999-14 is also handled.

I think the way by which we handle the LOCK type change and all fields retrieval is not right.

We retrieve all fields regardless of the FIELDS/EXCEPT clause, if the LOCK type set by the query is different then LockType.NONE. Which is not correct, because LockType.SHARE lock could also retrieve partial records. Only LockType.EXCLUSIVE lock retrieve the al the fields.

An update of lock does not necessary means that all fields should be retrieved.

NO-LOCK -> SHARE-LOCK - partial fields are retrieved.
NO-LOCK -> EXCLUSIVE-LOCK - all fields are retrieved.
SHARE-LOCK -> EXCLUSIVE-LOCK - all fields are retrieved.
SHARE-LOCK -> NO-LOCK -  partial fields are retrieved.
EXCLUSIVE-LOCK -> NO-LOCK - partial fields are retrieved.
EXCLUSIVE-LOCK -> SHARE-LOCK - partial fields are retrieved.

Things get very fuzzy when 2 buffers to the same table are used:

define buffer book1 for book.
define buffer book2 for book.

def query q1 for book1 fields (isbn), book2 fields (isbn).
open query q1 for each book1 NO-LOCK, each book2.
get first q1.
message book1.book-title book2.book-title.

NO-LOCK on the first buffer will cause retrieval of all fields in both buffers.

open query q1 for each book1, each book2 NO-LOCK.

However NO-LOCK on second buffer does not trigger the same behaviour.

Lorian/Artur: please advice on this topic.

+ added Artur and Lorian as watchers

I guess my question is why we check if the lock is different then LockType.NONE

=== modified file 'src/com/goldencode/p2j/persist/Persistence.java'
--- old/src/com/goldencode/p2j/persist/Persistence.java    2025-08-18 13:28:13 +0000
+++ new/src/com/goldencode/p2j/persist/Persistence.java    2025-11-06 15:39:50 +0000
@@ -685,6 +685,7 @@
 ** 218 LS  20250728          Modified 'tenantChanged' to skip buffer release if the tenant name has not changed.
 ** 219 OM  20250731          Dropped dead code related to old technology no longer in use.
 ** 220 LS  20250722          Changed 'load' to allow skipping full hydration.
+** 221 LS  20251106          Set 'partialFields' to null when the lock is not NO-LOCK.
 */

 /*
@@ -2638,7 +2639,11 @@
       RecordLockContext lockContext = local.getRecordLockContext();
       boolean resetLock = false;
       boolean vstTable = false;
-      
+
+      if (partialFields != null && lockType != LockType.NONE)
+      {
+         partialFields = null;
+      }
       if (temporary)
       {
          updateLock = false;

I think that can be changed to lockType.isExclusive()

Eduard Soltan wrote:

I have made a set of tests for the case when 2 buffers to the same table are used.

And actually the single case when the records are selected partially is this one:

[...]

In rest the records are always fetched fully.

I think this could be handled separately.

For now this branch is ready for review.

Could you review the branch?

Rebased 7999a to trunk rev. 16447.

This branch will work for the large majority of case.

There is only a small case that is not handled by this branch, having 2 buffers to the same table (BOTH have FIELDS or EXCEPT clause, first one and the second one have NO-LOCK). To solve this issue, it would require to add additional overhead of getting previous queries information.

I think this case should be handled in a separate branch, this branch could be reviewed.

While working on #11298, I have discovered that optimized CQ components does not have the included and excluded maps populated.
This is what I have changed to fix this (CQ.serverJoinPreselect and CQ.serverJoinAdaptive):

=== modified file 'src/com/goldencode/p2j/persist/CompoundQuery.java'
--- old/src/com/goldencode/p2j/persist/CompoundQuery.java    2026-04-06 12:34:41 +0000
+++ new/src/com/goldencode/p2j/persist/CompoundQuery.java    2026-04-06 13:49:32 +0000
@@ -5437,6 +5437,8 @@
          {
             return null;
          }
+         optimizedQuery.included = included;
+         optimizedQuery.excluded = excluded;

          return optimizedQuery;
       }
@@ -5494,6 +5496,8 @@
             optimizedQuery.addComponent(serverCopy);
             qcList.add(serverCopy);
          }
+         optimizedQuery.included = included;
+         optimizedQuery.excluded = excluded;

          return optimizedQuery;
       }

Teodor, keep in mind that excluded/included fields belong to a single component. So, if you join multiple components, how do you qualify the fields so you know which field belongs to which table?

Rebased 7999a to trunk rev. 16737.

- if (partialFields != null && lockType != LockType.NONE)
+ if (partialFields != null && lockType.isExclusive())

If the lock is exclusive, OE retrieves all data for a certain record. This change you propose makes SHARE locks behave different than exclusive. But what if the lock is being upgraded? Do we actually refetch the row on lock upgrade and ignore the FIELDS? My concern is not to reach a state where the record is exclusively locked, but only a part of fields are hydrated; I don't think this is a valid state in OE.

I see #7999-18, just make sure that plenty of unit tests cover this. We have faced a customer with an issue where FWD didn't fully hydrated a record on EXCLUSIVE-LOCK.

+ if (!isClientWhere() &&
+ (!fullRecords && component.isIdOnly() || forceOnlyPk) &&
+ included null &&
+ excluded null)

I don't know what to think about this change. These are cases of EXCLUSIVE-LOCK (and SHARE-LOCK) where we do projection queries. Any FIELDS selected here may become stale and incompatible with exclusive locking that ensures that we always lock the most up-to-date image of a record.

With this change, SHARE-LOCK and EXCLUSIVE-LOCK will retrieve FIELDS instead of projection. For SHARE-LOCK, I guess this is fine, but for EXCLUSIVE-LOCK we need to ensure included and excluded are always null. Also, the upgrading of lock should refetch the row with the latest data.

boolean doFetch = (!fullRec || updateLock || dmo.isStale()) && !dirtyCopy;

I think this change can impact the query execution a lot. TBH, this is the scariest change out of all.

• It is dropping the isResultsCached check. Is this correct? I much like that we can get results from cached results directly, but are they subject to STALE?
• Previously to your changes, temporary is dropped. I guess temporary records are never locked / stale or projected, so it is fine.

+ public boolean isStale()

I am skeptic about exposing isStale. This method synchronization. From the moment you check it to the moment you do an action a DMO can become stale. In your case doFetch may be false because the dmo is not stale, but by the time you reach if (doFetch), the DMO may become stale and you would have wanted doFetch to be true in fact.

What is the drive for doFetch change and the need for isStale?

Review of 7999a / r16750

This branch tricked me with a [merge] node at r16743 so the changes were incomplete at first, but things got clearer after identifying the correct branching revision.
I could not find severe issues, especially in the conversion (both static and dynamic). The following couple of issues is just nitpicking:

• database_access.rules:873: the definition of openQueryRef could be moved to 1923 to narrow its scope;
• PreselectQuery.java: 5935: && are not aligned to same screen column.

1. partialFields reset only for exclusive locks

What I measured:

FIELDS(customerName) NO-LOCK,        read an unlisted field  ->  error 8826
FIELDS(customerName) SHARE-LOCK,     read an unlisted field  ->  error 8826
FIELDS(customerName) EXCLUSIVE-LOCK, read an unlisted field  ->  value returned

So on this evidence OpenEdge does honour the field list under a share lock, and reads the whole record only under an exclusive one. If that holds generally, it suggests the current trunk behaviour is stricter than the 4GL for SHARE, and that the narrowing in this branch matches it.

The same pattern holds across 64 read-only combinations of per-buffer and GET lock, all matching OpenEdge.

2. Lock upgrade and refetch

This one was a genuine gap. RecordBuffer.upgradeLock() took the lock and returned without re-reading, so assigning a field under a share lock could leave the record exclusively locked and still incomplete. OpenEdge appears to re-read the whole record as part of that upgrade:

FIELDS(customerName) SHARE-LOCK, assign (book.f1 = 'something'), then read an unlisted field  ->  value returned
EXCEPT(customerEmail) SHARE-LOCK, assign (book.f1 = 'something'), then read the excluded one  ->  value returned

Addressed with a new RecordBuffer.upgradeLockForUpdate(), called from BaseRecord.lockForUpdate(): it takes the lock and then reloads the record if it is incomplete. Reading after the lock also gives the latest committed image, which is the property the review was asking for.

DELETE keeps the plain upgradeLock(). Nothing on the delete path needs a field other than the primary key, and OpenEdge does not appear to complete the record there either - a DELETE trigger sees exactly the projection, and FWD and OpenEdge produce identical output for that case.

3. Projection query under EXCLUSIVE-LOCK

On staleness: under an exclusive lock Persistence.load discards partialFields and re-reads the row after the lock has been taken, so the buffer ends up holding the latest committed image. The 16 exclusive cells of the matrix match OpenEdge. I could not find a case where a stale projection survives.

COmmitted on 7999a, rev. 16753.

• [MAJOR] functional DynamicQueryHelper.parse (lines 866-898): the new FIELDS/EXCEPT include/exclude application runs unconditionally, immediately before the pre-existing if (processor.delayedExecute() && query0 != null) guard on the next statement. On the QUERY-PREPARE path delayedExecute() unconditionally returns true, so at this point interpreter.interpret("execute") has not run and query0 is a constructed-but-uninitialized query (sort null, components null). AdaptiveQuery.include() merges the sort/index fields into the included set via collectIndexedFields(), which parses sort; with sort still null it contributes nothing, so a prepare string such as FOR EACH customer FIELDS(name) WHERE ... BY customer.city ends up including only name and never city — initialize() does not rebuild included, so the truncated set is what the query executes with, producing an incomplete DMO missing the ordering field. This is exactly what the AdaptiveQuery.include() override exists to prevent, and it hits every dynamic FIELDS/EXCEPT query with a BY clause on a permanent table. Apply include/exclude inside the onQueryOpen listener after interpret("execute") for the delayed case, guarding on query0 != null there.
• [MAJOR] functional CompoundQuery.setFieldsComponents: propagation assigns the raw map directly via comp.setInclude(newMap), bypassing AdaptiveQuery.include(), which merges the BY/sort-index fields into the include list via collectIndexedFields() ("necessary in order to add the indexed fields to the included fields in an incomplete DMO"). AbstractQuery.setInclude/setExclude are plain field assignments with no polymorphic hook, and neither CompoundQuery nor DynamicQuery overrides include(). Trigger: a converted multi-buffer OPEN QUERY q FOR EACH a, EACH b ... BY b.f where the FIELDS list for b omits f — the component loads an INCOMPLETE record lacking the sort column, and when the AdaptiveQuery switches from dynamic to preselect mode, prepareParameters() reads the sort property straight off the snapshot record, bypassing BaseRecord.checkIncomplete(), so breakValue is silently null/default and the re-issued query returns the wrong band of rows (skipped/duplicated records) with no error. Fix by applying the merge when propagating — e.g. override setInclude in AdaptiveQuery to run collectIndexedFields and union it into the incoming map, null-guarding the case where sort is not yet set.

These 2 seem less or more the same. There is indeed an important integration of fields with the fields used for indexing. An AQ that gets invalidated and reaches a RAQ will use the indexed fields to resolve records from the primary database against the record in the dirty or use the indexed fields to build up the active bundle.

For instance, an invalidated query that does next will use something like where ... and idxf1 > ? to gather the next record, using the last fetched record as a reference row. We decided at some point that in FWD the indexed fields can't be excluded by EXCLUDE or FIELDS. So, any query that uses FIELD or EXCLUDE will retrieve records with recid, the specified list of fields and always the indexed fields used by the order by. I am not sure if other use cases occured in the mean-time, but the invalidation one required collectIndexedFields. Otherwise, the idxf1 > ? will be resolved to a non-read field, yielding an error when doing NEXT (or PREV).

• [MAJOR] functional PreselectQuery.coreFetch (lines ~7620 and ~7676): both p.load(..., dmo == null ? null : dmo._getReadFields()) calls hand the row DMO's live readFields BitSet down the stack un-cloned. Persistence.load only nulls partialFields for lockType.isExclusive() and otherwise forwards the same object, and Session.getImpl mutates it in place (partialFields.andNot(dmo.readFields)) whenever the session-cached instance for that id is a different INCOMPLETE DMO than the one held in the current result row. Concrete, fully-reachable trigger: a FIELDS-restricted browse under a non-exclusive lock whose result rows hold full Record objects; the row's DMO is evicted from the session cache by the routine BufferManager.evictDMOIfUnused/session LRU path while still referenced by the result row; the same id is re-read with a different FIELDS subset, producing a new cached INCOMPLETE instance; the next coreFetch of the original row then has its own readFields silently cleared by the andNot, after which BaseRecord.checkIncomplete raises a spurious "missing from FIELDS phrase" runtime error (or, with reread-fields enabled, a hidden extra reload) for fields that were in fact read — aborting a running 4GL program. The codebase already guards against exactly this elsewhere (SQLQuery.hydrateRecordImpl clones the BitSet before mutating). Pass (BitSet) dmo._getReadFields().clone() at both call sites, or — preferably — have Session.getImpl copy partialFields before the andNot so every caller is fixed at once.

This is a very good point. The bitset is a mutable data structure. It think it would be safer to not allow _getReadFields to escape the instance and always clone the bit-set when getting the readFields. Check for other users that may maliciously use this getter.

• [MINOR] performance CompoundQuery.retrieveImpl: setFieldsComponents(true)/setFieldsComponents(false) are invoked from retrieveImpl, the per-row retrieval path used by both retrieve() and the preselectResults() loop, not the query-open path (propagation cannot simply move into open(), since that method runs before maybeOptimize() and must not call it). Whenever a FIELDS/EXCEPT clause is active, each row re-walks every component's getRecordBuffers() (which allocates a fresh array on every call) and reallocates a fresh IdentityHashMap per matching component, even though the maps are invariant after the first call — maybeOptimize() is idempotent and included/excluded are only ever set at query setup. The cost is real but modest (a handful of short-lived allocations per row, negligible next to the per-row DB fetch). The common non-FIELDS case is unaffected (both calls return immediately on a null map). Add a run-once guard inside retrieveImpl (cleared in open()) instead of re-propagating every row.
• [MINOR] performance DynamicQueryHelper.parse (lines 866-898): the include/exclude application loops test include.get(bufferAlias) != null / exclude.get(bufferAlias) != null, repeating a hash lookup for a value already held as pair.getValue(). This runs on the common path, including level-1 cache hits, whenever the prepare string carried a FIELDS/EXCEPT clause, and the guard is tautological — collectFields only ever stores a non-null array, so the branch can never be false. Test pair.getValue() != null instead (or drop the guard).

Reasonable points about performance. Whatever we can do only once at opening time should be done there.

• [MINOR] functional DynamicQueryHelper.prepare: in the dataset-sibling loop, bufferNames.put(b.buffer().getDMOAlias(), b.buffer()) reuses the outer loop variable b instead of the sibling buffer buf (dsBuffers.get(i).ref()) — a copy-paste defect (the line is byte-identical to the outer-loop registration two lines above). Dataset sibling buffers are consequently never registered in bufferNames; every dataset/FILL prepare over a valid dataset buffer hits this. It does not currently NPE the FIELDS/EXCEPT lookup — collectFields only ever keys on buffers already proven to be in the buffers list by validateBuffers — but the map is wrong by construction, and the subsequent substBuffers containsKey dedup misses those siblings (currently masked only because LinkedHashSet identity-dedups the same BufferImpl instances). Use buf.buffer().getDMOAlias() / buf.buffer().

Can't tell much about this finding per-se, but I am most certain that we have faces (many) issues in the past with the aliasing of buffers (sending buffers as parameters, using them in dataset, binding to other procedures, etc.). This whole DMOAlias saga should be carefully tested.

Conversion

• [MAJOR] functional database_access.rules.record_phrase FIELDS/EXCEPT emission: the buffer reference emitted for the new include/exclude call omits the static-buffer dereference (.get()) that every sibling rule in this file applies for a static buffer/temp-table. bufnameJavaname comes from execLib("get_javaname", fieldsClause.getPrevSibling()), and get_javaname has no static awareness — it only returns the localname annotation or the javaname of the refid target. Trigger: DEFINE STATIC TEMP-TABLE tt FIELD f AS CHAR. DEFINE QUERY q FOR tt FIELDS(f). OPEN QUERY q FOR EACH tt. — the pre-existing addBuffer rule resolves the same node through add_static_access and emits q.addBuffer(tt.get(), true), while the new block emits q.include(tt, "f") on the next line of generated code, which will not compile since a static buffer converts to a ContextLocal wrapper. Resolve the buffer reference through add_static_access (or the %s.get() suffix) the same way the pre-existing ascent rule does for the inline-FIELDS case.
• [MAJOR] functional database_access.rules.record_phrase FIELDS/EXCEPT emission: the new walk rule does not suppress the pre-existing ascent rule that also emits include/exclude for a FIELDS/EXCEPT clause carried directly on the OPEN QUERY's own record phrase, so both can fire for the same buffer. Trigger: DEFINE QUERY q FOR customer FIELDS(name). OPEN QUERY q FOR EACH customer EXCEPT(city). — legal 4GL, since the two clauses are matched by unrelated grammar productions (def_query_stmt and open_query_stmt). The new rule grafts q.include(customer, "name") while the pre-existing ascent rule grafts q.exclude(customer, "city"), leaving both included and excluded populated for the same DMO; because AbstractQuery.collectFields uses computeIfAbsent and only ever sets slots, a FIELDS clause on both DEFINE and OPEN yields the union of fields rather than the OPEN QUERY clause taking precedence, and when both included and excluded are non-null for one DMO, RandomAccessQuery.getPartialFields silently discards the include set and rebuilds from exclude alone — producing a wrong partial-field fetch set. The two paths must be made mutually exclusive: the new rule should skip a record phrase that already carries its own FIELDS/EXCEPT child, letting the OPEN QUERY clause win.
• [MINOR] functional database_access.rules.record_phrase FIELDS/EXCEPT emission, while loop: the loop that emits each field of the clause has no node-type or null guard on the fieldname annotation, unlike the sibling emission rule which guards with evalLib("fieldtype", ...). The post-parse fixup that strips non-field entries only applies when the clause's parent is a record_phrase, but a DEFINE QUERY's field list is parented by KW_FOR, so it is never sanitized. Trigger: the grammar's own documented "undocumented feature" where a bare record/table name is matched instead of an lvalue list (e.g. DEFINE QUERY q FOR customer FIELDS(customer).), or a field name shadowed by a variable/widget of the same name — both produce a clause child with no fieldname annotation. This does not NPE or corrupt data: createJavaAst(java.string, null, ref) emits the literal Java string "null", AbstractQuery.collectFields logs a warning and leaves that slot unresolved at runtime, but the buffer still gets a (wrong) entry in included, silently narrowing the projection to id/indexed fields only. Guard the loop body with the same evalLib("fieldtype", field.type) test used by the sibling rule.
• [MINOR] functional database_access.rules.is_define_query_buffer: the new fieldsClauses recording is nested inside the addBuffer rule and inherits that rule's unrelated guard excluding shared, non-NEW query declarations. Trigger: an importing procedure containing DEFINE SHARED QUERY q FOR customer FIELDS(name). OPEN QUERY q FOR EACH customer. — legal 4GL (DEFINE [[NEW] SHARED] QUERY ... FIELDS/EXCEPT) — where the guard evaluates false (KW_SHARED present, KW_NEW absent), so the recording rule body never runs; since the DEFINE QUERY's FIELDS node is unconditionally hidden elsewhere, the clause disappears with no diagnostic and the shared query opens with full records instead of the declared projection. The FIELDS/EXCEPT recording should be its own rule guarded only on the DEFINE-QUERY-buffer condition, not on the SHARED/NEW guard.

Not entirely sure about these, especially what "DEFINE STATIC TEMP-TABLE@ is :) Maybe you or Ovidiu can confirm or deny these bullets.

• [MINOR] performance PreselectQuery.coreFetch: passing dmo._getReadFields() unconditionally (including in the updateLock branch, which previously forced null) buys no reduction in the SQL Loader executes, since Loader.load() always runs the precomputed full-column loadSql regardless of partialFields — partialFields only decides which returned columns get copied into the DMO. On a session-cache miss, the row is now left INCOMPLETE, and the first later access to a field outside the projection forces a second, identical full-column SELECT for that row (or, via -rereadfields, another full load) — an extra round trip that did not happen before. This now also applies under a plain SHARE lock (paired with the Persistence lock-type change), whereas before the lock forced a full/null-partialFields load. Note this trades a genuine win on the cache-hit path (skips a previously-forced reload) for a loss on the cache-miss/widening path; consider reducing the actual SQL projection so a partial read costs less, rather than loading full columns into a record left incomplete.

• This is interesting; we emit full SQLs, but cherry-pick results based on the partialFields marker. I doubt this is a usual hit, because the session-cache will usually hit as the record was just hydrated from the result-set. I think this is indeed MINOR and can be deferred to another task. It is worth tho to put a debug breakpoint and check how often is this hit with a non-empty partialFields. It may be a separate optimization opportunity. It is important however to confirm it is not nullifying the changes in #7999 to actually parse FIELDS to end-up not using them in the SQL.

Style

• [MINOR] style DynamicQueryHelper: the import cleanup correctly drops the now-redundant explicit import java.util.List; / import java.util.Map; (covered by the existing import java.util.*;), but it also inserts a new blank line between import java.util.logging.*; and import antlr.*; that does not exist elsewhere in this file or in sibling files (AbstractQuery, CompoundQuery, PreselectQuery). Remove the extra blank line to match the established convention.

• [MINOR] style database_access.rules: the new local <variable> declarations (fieldsClause, bufnameAnnotation, bufnameJavaname, field, strField) are not column-aligned as required by this file's convention of padding name="..." so type= starts in the same column across a block of declarations.

• [MINOR] style database_access.rules: the new worker registration <worker class="com.goldencode.p2j.pattern.CommonAstSupport" namespace="cat" /> is never referenced anywhere in the file (no cat. usage) — CommonAstSupport is already registered under the common namespace elsewhere, making this a dead, unused declaration that should be removed.

• [MINOR] style PreselectQuery.assembleComponent: the new multi-line boolean condition does not fully align its && operators at the line ends:
  

  if (!isClientWhere()                                       &&
      (!fullRecords && component.isIdOnly() || forceOnlyPk) &&
      !hasFieldList)
  

  
  Line 1's && ends at column 67; line 2's && ends one column short at column 66. Add one space before the && on line 2 to align.

Alexandru Lungu wrote:

These 2 seem less or more the same. There is indeed an important integration of fields with the fields used for indexing. An AQ that gets invalidated and reaches a RAQ will use the indexed fields to resolve records from the primary database against the record in the dirty or use the indexed fields to build up the active bundle.

For instance, an invalidated query that does next will use something like where ... and idxf1 > ? to gather the next record, using the last fetched record as a reference row. We decided at some point that in FWD the indexed fields can't be excluded by EXCLUDE or FIELDS. So, any query that uses FIELD or EXCLUDE will retrieve records with recid, the specified list of fields and always the indexed fields used by the order by. I am not sure if other use cases occured in the mean-time, but the invalidation one required collectIndexedFields. Otherwise, the idxf1 > ? will be resolved to a non-read field, yielding an error when doing NEXT (or PREV).

The dynamic application moved into applyFieldsClauses(), called immediately for the FIND path (where execute has already been interpreted) and from inside the onQueryOpen listener right after interpret("execute") for the delayed one. Since the listener is only registered when query0 != null, the null case you flagged is covered by construction. I also added a null guard on the buffer lookup, which could have NPE'd on an unresolved alias.

setInclude and setExclude in AdaptiveQuery now route through a new applyIndexedFields() helper. It copies before writing, because setFieldsComponents shares the parent query's Property[] arrays and mutating them in place would corrupt the compound's own map, and it returns the original map untouched when there's nothing to add.

Two things I found while in there that weren't in the review. First, EXCEPT had the identical hole in reverse — nothing stopped EXCEPT idxf1 from dropping an index field, since exclude() was never overridden the way include() was, and PreselectQuery.assembleComponent nulls out any property present in xprops without checking. So exclude() is now overridden too, stripping index fields rather than adding them, and when every listed field turns out to be an index field the clause is skipped entirely rather than registered empty (an empty clause would suppress the id-only reduction and force a full-record fetch for no reason). Second, collectIndexedFields() had two latent NPEs: the catch around SortCriterion.parse logs and falls through, leaving sortCriteria null for the size() call on the next line, and an unresolvable field would NPE on p.name.

This is a very good point. The bitset is a mutable data structure. It think it would be safer to not allow _getReadFields to escape the instance and always clone the bit-set when getting the readFields. Check for other users that may maliciously use this getter.

BaseRecord._getReadFields() now returns (BitSet) readFields.clone() (null-safe) instead of the live field — the bit set can't escape the instance, which fixes both coreFetch call sites without touching them.
Session.getImpl now clones partialFields before partialFields.andNot(dmo.readFields), so it stops mutating a caller-owned object. Behaviour-neutral: no caller reads it back afterwards.

• [MINOR] performance CompoundQuery.retrieveImpl: setFieldsComponents(true)/setFieldsComponents(false) are invoked from retrieveImpl, the per-row retrieval path used by both retrieve() and the preselectResults() loop, not the query-open path (propagation cannot simply move into open(), since that method runs before maybeOptimize() and must not call it). Whenever a FIELDS/EXCEPT clause is active, each row re-walks every component's getRecordBuffers() (which allocates a fresh array on every call) and reallocates a fresh IdentityHashMap per matching component, even though the maps are invariant after the first call — maybeOptimize() is idempotent and included/excluded are only ever set at query setup. The cost is real but modest (a handful of short-lived allocations per row, negligible next to the per-row DB fetch). The common non-FIELDS case is unaffected (both calls return immediately on a null map). Add a run-once guard inside retrieveImpl (cleared in open()) instead of re-propagating every row.
• [MINOR] performance DynamicQueryHelper.parse (lines 866-898): the include/exclude application loops test include.get(bufferAlias) != null / exclude.get(bufferAlias) != null, repeating a hash lookup for a value already held as pair.getValue(). This runs on the common path, including level-1 cache hits, whenever the prepare string carried a FIELDS/EXCEPT clause, and the guard is tautological — collectFields only ever stores a non-null array, so the branch can never be false. Test pair.getValue() != null instead (or drop the guard).

Reasonable points about performance. Whatever we can do only once at opening time should be done there.

CompoundQuery — setFieldsComponents no longer runs per row. Guarded in retrieveImpl on component-list identity (fieldsPropagatedTo != comps), not a run-once flag cleared in open(), because components is nulled in three places (close(), addDynamicFilter(), clearDynamicFilters()) — a flag would have missed the latter two and left new joined components without the maps. Also nulled in close() next to components = null.

DynamicQueryHelper — the redundant include.get(bufferAlias) != null lookup is gone; the block became applyFieldsClauses() during the ordering fix and tests pair.getValue() != null, plus a rb != null guard that prevents an NPE on an unresolved alias.

• [MINOR] functional DynamicQueryHelper.prepare: in the dataset-sibling loop, bufferNames.put(b.buffer().getDMOAlias(), b.buffer()) reuses the outer loop variable b instead of the sibling buffer buf (dsBuffers.get(i).ref()) — a copy-paste defect (the line is byte-identical to the outer-loop registration two lines above). Dataset sibling buffers are consequently never registered in bufferNames; every dataset/FILL prepare over a valid dataset buffer hits this. It does not currently NPE the FIELDS/EXCEPT lookup — collectFields only ever keys on buffers already proven to be in the buffers list by validateBuffers — but the map is wrong by construction, and the subsequent substBuffers containsKey dedup misses those siblings (currently masked only because LinkedHashSet identity-dedups the same BufferImpl instances). Use buf.buffer().getDMOAlias() / buf.buffer().

Can't tell much about this finding per-se, but I am most certain that we have faces (many) issues in the past with the aliasing of buffers (sending buffers as parameters, using them in dataset, binding to other procedures, etc.). This whole DMOAlias saga should be carefully tested.

DynamicQueryHelper — the dataset-sibling loop now registers buf.buffer().getDMOAlias(), buf.buffer() instead of re-registering the outer b.

Style

• [MINOR] style DynamicQueryHelper: the import cleanup correctly drops the now-redundant explicit import java.util.List; / import java.util.Map; (covered by the existing import java.util.*;), but it also inserts a new blank line between import java.util.logging.*; and import antlr.*; that does not exist elsewhere in this file or in sibling files (AbstractQuery, CompoundQuery, PreselectQuery). Remove the extra blank line to match the established convention.

• [MINOR] style database_access.rules: the new local <variable> declarations (fieldsClause, bufnameAnnotation, bufnameJavaname, field, strField) are not column-aligned as required by this file's convention of padding name="..." so type= starts in the same column across a block of declarations.

• [MINOR] style database_access.rules: the new worker registration <worker class="com.goldencode.p2j.pattern.CommonAstSupport" namespace="cat" /> is never referenced anywhere in the file (no cat. usage) — CommonAstSupport is already registered under the common namespace elsewhere, making this a dead, unused declaration that should be removed.

• [MINOR] style PreselectQuery.assembleComponent: the new multi-line boolean condition does not fully align its && operators at the line ends:
  [...]
  Line 1's && ends at column 67; line 2's && ends one column short at column 66. Add one space before the && on line 2 to align.

Fixed all of that in 7999a rev. 16756.

[MAJOR] functional database_access.rules.record_phrase FIELDS/EXCEPT emission: the buffer reference emitted for the new include/exclude call omits the static-buffer dereference (.get()) that every sibling rule in this file applies for a static buffer/temp-table. bufnameJavaname comes from execLib("get_javaname", fieldsClause.getPrevSibling()), and get_javaname has no static awareness — it only returns the localname annotation or the javaname of the refid target. Trigger: DEFINE STATIC TEMP-TABLE tt FIELD f AS CHAR. DEFINE QUERY q FOR tt FIELDS. OPEN QUERY q FOR EACH tt. — the pre-existing addBuffer rule resolves the same node through add_static_access and emits q.addBuffer(tt.get(), true), while the new block emits q.include(tt, "f") on the next line of generated code, which will not compile since a static buffer converts to a ContextLocal wrapper. Resolve the buffer reference through add_static_access (or the %s.get() suffix) the same way the pre-existing ascent rule does for the inline-FIELDS case.

static buffer dereference. The buffer node is now resolved through its refid and passed to add_static_access, mirroring the addBuffer rule:

  <rule>bufferNode.isAnnotation("refid")
     <action>
        astid = execLib("add_static_access",
                        getAst(getReferenceNoteLong(bufferNode.id, "refid")),
                        ref.id)
     </action>
  </rule>

[MAJOR] functional database_access.rules.record_phrase FIELDS/EXCEPT emission: the new walk rule does not suppress the pre-existing ascent rule that also emits include/exclude for a FIELDS/EXCEPT clause carried directly on the OPEN QUERY's own record phrase, so both can fire for the same buffer. Trigger: DEFINE QUERY q FOR customer FIELDS. OPEN QUERY q FOR EACH customer EXCEPT. — legal 4GL, since the two clauses are matched by unrelated grammar productions (def_query_stmt and open_query_stmt). The new rule grafts q.include(customer, "name") while the pre-existing ascent rule grafts q.exclude(customer, "city"), leaving both included and excluded populated for the same DMO; because AbstractQuery.collectFields uses computeIfAbsent and only ever sets slots, a FIELDS clause on both DEFINE and OPEN yields the union of fields rather than the OPEN QUERY clause taking precedence, and when both included and excluded are non-null for one DMO, RandomAccessQuery.getPartialFields silently discards the include set and rebuilds from exclude alone — producing a wrong partial-field fetch set. The two paths must be made mutually exclusive: the new rule should skip a record phrase that already carries its own FIELDS/EXCEPT child, letting the OPEN QUERY clause win.

I think this is wrong.

I get the following error: FIELDS/EXCEPT belong on DEFINE QUERY, not on OPEN QUERY. (3638) when define FIELDS/EXCEPT phrase in the OPEN statement.

[MINOR] functional database_access.rules.record_phrase FIELDS/EXCEPT emission, while loop: the loop that emits each field of the clause has no node-type or null guard on the fieldname annotation, unlike the sibling emission rule which guards with evalLib("fieldtype", ...). The post-parse fixup that strips non-field entries only applies when the clause's parent is a record_phrase, but a DEFINE QUERY's field list is parented by KW_FOR, so it is never sanitized. Trigger: the grammar's own documented "undocumented feature" where a bare record/table name is matched instead of an lvalue list (e.g. DEFINE QUERY q FOR customer FIELDS.), or a field name shadowed by a variable/widget of the same name — both produce a clause child with no fieldname annotation. This does not NPE or corrupt data: createJavaAst(java.string, null, ref) emits the literal Java string "null", AbstractQuery.collectFields logs a warning and leaves that slot unresolved at runtime, but the buffer still gets a (wrong) entry in included, silently narrowing the projection to id/indexed fields only. Guard the loop body with the same evalLib("fieldtype", field.type) test used by the sibling rule.

Guarded the field-emission loop in database_access.rules so non-field nodes are skipped:

  <while>field != null
     <rule>evalLib("fieldtype", field.type)
        <action>strField = field.getAnnotation("fieldname")</action>
        <action>createJavaAst(java.string, strField, ref)</action>
     </rule>

     <action>field = field.getNextSibling()</action>
  </while>

I reviewed r16757.

• The most problematic issue I noticed being fixed was the b / buf swapped. 👍
• related to exclusion of indexed properties. As Alex said, we agreed that these properties cannot be excluded in a query. I did not comment because I thought we take care of this at a lower level. It seems that that was not the case. The code adds a bit of complexity but it is necessary. BTW, in CompoundQuery there is the fieldsPropagatedTo member field which is an ArrayList. It is assigned and compared to result of getComponents() (using = operator). I wonder whether the correct way to compare is to iterate it and compare each element since getComponents() is not stable (changes with dynamic filters) so the setFieldsComponents() might be called more than once.
• in RecordBuffer:9695, a typo: does acquires sound better instead of takes?

Ovidiu Maxiniuc wrote:

• related to exclusion of indexed properties. As Alex said, we agreed that these properties cannot be excluded in a query. I did not comment because I thought we take care of this at a lower level. It seems that that was not the case. The code adds a bit of complexity but it is necessary. BTW, in CompoundQuery there is the fieldsPropagatedTo member field which is an ArrayList. It is assigned and compared to result of getComponents() (using = operator). I wonder whether the correct way to compare is to iterate it and compare each element since getComponents() is not stable (changes with dynamic filters) so the setFieldsComponents() might be called more than once.

The check sits in retrieveImpl(). Identity is one reference compare per row; element-wise is O(components) per row. The redundant propagation it would avoid happens only when `components` is invalidated, (per dynamic-filter change or re-open), not per row. So we would be paying on the hot path to save work on a cold one.

Committed typo fix and fieldsPropagatedTo invalidation in CompoundQuery.initialize in rev. 16758.

Eduard Soltan wrote:

7999a was merged to trunk rev. 16764 and archived.

Eduard,

Unfortunately this commit introduces regression in big customer M application when trying to access PB:

...
26/09/18 23:26:55.433+0300 |  SEVERE | com.goldencode.p2j.util.ErrorManager | ThreadName:Conversation [00000075:bogus-gui_user], Session:00000117, ThreadId:00000107, User:bogus | Field IdPrj from mpl_opd record (recid 130430975) was missing from FIELDS phrase. (8826)
26/09/18 23:26:56.598+0300 |  SEVERE | com.goldencode.p2j.util.ErrorManager | ThreadName:Conversation [00000075:bogus-gui_user], Session:00000117, ThreadId:00000107, User:bogus | ** Unable to update mpl_opd Field. (142)
...

Can you please take a look. Because this is a kind of stopper for further application update.

What OpenEdge actually delivers¶

I think the rule is whatever OpenEdge has to ship to the client in order to make the client side join becomes readable; what it never ships stays missing. A sibling's predicate and an OF relation have to reach the client to parameterize the second query. A buffer's own where clause and a sort key are resolved in the database, so nothing is shipped and the field stays refused.

Committed on 7999a, rev. 16766.

Eduard, just one thing to check. Are the FIND triggers related to the partial FIELDS? If you do a FIND trigger, would the record reported by that trigger be partially hydrated ... I would think so. But make sure there is no quirk there like "fully hydrate if the session has find triggers".

Is TENANT-WHERE relevant? I would expect that it works like WHERE from the POV of field selection.

I don't think I have other objections. Please test these and lets proceed with review and testing.

Eduard Soltan wrote:

All the cases are handled in 7999a, rev. 16769. Please review.

Do you mean 7999b, not a, right? The a version was archived after merge in trunk. Or I missed for something?

Eugenie Lyzenko wrote:

Do you mean 7999b, not a, right? The a version was archived after merge in trunk. Or I missed for something?

Sorry, committed to 7999b.

Review of 7999b / r16769

• AbstractJoin.java:
  • line 5: (c) year not updated
• CompoundQuery.java:
  • line 345: missing ♯ for the new H entry;
  • lines 4602, 4608: this hybrid approach of chopping the line is not encouraged. More than that, all of them can stay on same line without overflowing the 110 chars limit;
  • line 4706: expression can be simplified. BufferReference extends DataModelObject so !(dmo instanceof DataModelObject) implies !(dmo instanceof BufferReference). The left operand can be safely dropped. This becomes more evident with the cast at line 4719;
• PreselectQuery.java:
  • line 8408: CAN-FIND can also be converted to select count(recid)... = 1 for unique cases. The current where.contains("exists") will cover only the non unique (first / last) cases.