Project

General

Profile

Bug #9365

The active buffer is no longer set for getters, so _getClob is not working

Added by Alexandru Lungu over 1 year ago. Updated over 1 year ago.

Status:
Test
Priority:
High
Target version:
-
Start date:
Due date:
% Done:

100%

billable:
No
vendor_id:
GCD
case_num:
version_reported:
version_resolved:
production:
No
env_name:
topics:

History

#1 Updated by Alexandru Lungu over 1 year ago

  • Assignee set to Artur Școlnic
  • Priority changed from Normal to High
  • reviewer Alexandru Lungu added

After the following change:

=== modified file 'src/com/goldencode/p2j/persist/RecordBuffer.java'
--- old/src/com/goldencode/p2j/persist/RecordBuffer.java    2024-11-20 08:24:11 +0000
+++ new/src/com/goldencode/p2j/persist/RecordBuffer.java    2024-11-22 08:46:09 +0000
@@ -1377,6 +1377,7 @@
 **                           nursery remove will do the job.
 **     AL2 20241112          Adapt to new nursery remove signature.
 ** 378 AP  20241009          Added new method to reset the sortIndex.
+** 379 AOG 20241121          Updated condition of currentRecord.setActiveBuffer only when invoking setter.
 */

 /*
@@ -12678,18 +12679,17 @@
                }
             }

-            // for any setter method and some getters (e.g., _getClob), the DMO needs access to this buffer
-            // the active buffer is set for each buffer is a single assign and just once for a buffer
-            // used in a batch assign.
-            boolean activeBufferSet = bufferManager.isBufferDirtyBatch(RecordBuffer.this);
-            if (!bufferManager.isBatchAssignMode() || !activeBufferSet)
-            {
-               restoreActiveState = currentRecord.setActiveBuffer(RecordBuffer.this);
-            }
-            
             boolean denormalized = false;
             if (isSetter)
             {
+               // for any setter method and some getters (e.g., _getClob), the DMO needs access to this buffer
+               // the active buffer is set for each buffer is a single assign and just once for a buffer
+               // used in a batch assign.
+               if (!bufferManager.isBatchAssignMode() || !bufferManager.isBufferDirtyBatch(RecordBuffer.this))
+               {
+                  restoreActiveState = currentRecord.setActiveBuffer(RecordBuffer.this);
+               }
+
                pm = dmoInfo.getPropsBySetterMap().get(method);
                if (pm == null)
                {

That reached trunk as rev. 15573. The activeBuffer is set only for setters. Thus _getClob won't work anymore. This should be fixed asap. Maybe we can work-around this specific case somehow - for instance, check if it is about _getClob in Handler.invoke. If so, so a temporary set of the active buffer and restore - just for the scope of the getter.

Also, can you test what happens in the following scenario:

function some-function-call returns char():
   tt2.f1 = 'a'. // getter
   return '';
end.

assign tt.f1 = some-function-call(). // setter

Will tt2 be marked as "dirty batch buffer" because it was used nested in a batch assign? If so, isn't this a problem as the restore won't properly kick in?

#2 Updated by Artur Școlnic over 1 year ago

  • Status changed from New to WIP

#3 Updated by Artur Școlnic over 1 year ago

  • Status changed from WIP to Review

isBatchAssignMode looks in the current scope, so in the case from note 1, tt2 won't be added to the dirty batch buffers. If in the some-function-call batch assign mode is used, restore will be performed by the logic of that batch. Overall I don't think this could be an issue.
I looked through the getters in Record and could not find another one (besides _GetCLOB) that needs the activeBuffer to be set, but we need to consider the possibility that such methods will be added or the existing one will be augmented.
I implemented setting the active buffer and restoring it's state for getters used as a standalone call or in a batch assign context.
Also I found some other issues regarding the active buffer and fixed them.
The code is in 9365a/15575.

#5 Updated by Alexandru Lungu over 1 year ago

  • % Done changed from 0 to 90

My understanding of the problem and proposed solution: The problem is that CLOB field getters don't have an active buffer. The implementation shows a new collection in BufferManager to keep the state on what buffers were set as active buffers to their records and use that to manage the restore timing.

Review of 9365a/15575

  • The collection of getterActiveBuffers leaks as there is no clear from this collection.
    • Also, I don't fully understand the motivation behind getterActiveBuffers. It seems to me that this stores the buffers that called getClob on assign (batch or not). The main goal of it seems to be the restore on batch end. However, I don't think delaying the restore till the batch end is actually needed. It is also tricky in the context of:
assign tt.f1 = buf1.clobField 
     tt.f2 = buf2.clobField
  • The implementation rules out setting the active buffer of the record when resolving buf2.clobField, because it is part of getterActiveBuffers already and its restore was delayed. However, if the buf2 and buf1 buffers imply different collations, this implementation won't work properly.
  • Artur, please attempt a less resource demanding solution (like managing getterActiveBuffers) with a more conditional imperative approach: if the getter is for a CLOB, simply set the activeBuffer and restore at the end in the finally block. Unless there is a strong lead on the need of restoring at the end of the batch for such case, then lets keep it simple.

#6 Updated by Artur Școlnic over 1 year ago

  • % Done changed from 90 to 0

getterActiveBuffers has a similar functionality to dirtyBathBuffers In a simple assign case, we can just check if a getter is getClob, and set the active buffer, the existing logic will restore it's state in the finally, but if is is used in a batch assign, then it's state can't be restored in the invoke of the getter, it has to be at the end of the batch, like the dirtyBathBuffers. I think tracking just the buffers used by getters that need an active buffer is more lightweight that tracking all the buffers that were set as active buffers.
When it comes to multiple buffers being used for the same table:

define temp-table tt no-undo field f1 as clob field f2 as clob.
define temp-table tt1 no-undo field f1 as clob field f2 as clob.
define buffer buf1 for tt1.
create tt1.
find buf1.

create tt.
assign tt.f1 = tt1.f1 tt.f2 = buf1.f2. // tt is in dirtyBatchBuffers, it's state will be restored at the end of the batch.
                                       // tt1 is in getterActiveBuffers, it's state will be restored at the end of the batch.
                                       // buf1 is in getterActiveBuffers, it's state will be restored at the end of the batch.

This case is pretty much similar to the ones where multiple buffers referencing the same table are being used in a batch assign.

#7 Updated by Artur Școlnic over 1 year ago

  • % Done changed from 0 to 90

#8 Updated by Alexandru Lungu over 1 year ago

but if is is used in a batch assign, then it's state can't be restored in the invoke of the getter, it has to be at the end of the batch, like the dirtyBathBuffers.

This is what I don't quite understand. Why can't it be restored in the invoke of the getter and why it has to be at the end of the batch? For setters, it should be because the validateMaybeFlush should acknowledge all changed fields in the batch. For getter, this doesn't hold.

#9 Updated by Artur Școlnic over 1 year ago

I wanted to be consistent with the way the restore is happening for all buffers, but I guess this is not imperative for getters, I will drop the getterActiveBuffers collection.

#10 Updated by Alexandru Lungu over 1 year ago

  • Status changed from Review to WIP

#11 Updated by Artur Școlnic over 1 year ago

  • Status changed from WIP to Review

I made the necessary changes and committed 15576, meanwhile I started a large app unit test module for testing.

#12 Updated by Alexandru Lungu over 1 year ago

Artur, please also design a small 4GL Unit Test suite with this edge case and add it to your existing suite. Also test that with 9365a. It is quite hard to find a CLOB and unit test it properly.

#13 Updated by Artur Școlnic over 1 year ago

The largest app unit test regression testing passed. I could think of a single scenario where the issue discussed in this ticked comes into play:

define temp-table tt no-undo field f1 as clob init ?.
define temp-table tt1 no-undo field f1 as int field f2 as clob COLUMN-CODEPAGE "UTF-16".

create tt.
tt.f1 = "CLOB".
create tt1.
assign tt1.f2 = tt.f1.
message tt1.f2 <> ?. // yes, both 4gl and FWD
message GET-CODEPAGE(tt1.f2). //4gl: UTF-16, FWD: ISO-8859-1

create tt1.
assign tt1.f1 = 1 tt1.f2 = tt.f1.
message tt1.f2 <> ?. // yes, both 4gl and FWD
message GET-CODEPAGE(tt1.f2). // 4gl: UTF-16, FWD: UTF-16

I will write these as 4gl abl unit tests and upload them to the testcases project.

#14 Updated by Alexandru Lungu over 1 year ago

Review of 9365a:
  • I am OK with the set/restore mechanism.
  • is BufferManager change still needed?
  • you also have a - iter.remove(); change. This looks out-of-context. Is this needed?
Artur, please also test:
  • tt1.f2 = tt.f1 without assign
  • buffer-copy construct

Please debug these test-cases and ensure the getter is working properly.

#15 Updated by Artur Școlnic over 1 year ago

Alexandru Lungu wrote:

Review of 9365a:
  • is BufferManager change still needed?

If you mean if (bmd.batchDepth != BatchModeData.NOT_ACTIVE && !bmd.dirtyBatchBuffers.containsKey(buffer)), then yes. If a restoring state runnable was assigned to a buffer, we don't want to replace it with null, which is possible sometimes.

  • you also have a - iter.remove(); change. This looks out-of-context. Is this needed?

Yes, this line removes the dirtyBatchBuffers before they reach cleanupBatchMode, clearing this collection is now done in cleanupBatchMode.

#16 Updated by Alexandru Lungu over 1 year ago

If you mean if (bmd.batchDepth != BatchModeData.NOT_ACTIVE && !bmd.dirtyBatchBuffers.containsKey(buffer)), then yes. If a restoring state runnable was assigned to a buffer, we don't want to replace it with null, which is possible sometimes.

Can you describe what are the possible cases in which this addition in required?
  • I want to understand if ruling out the restore registration is the right approach. Maybe you should call the existing restore and then override it.
  • Or maybe this is an invalid state that requires some error handling or at least some logging.
  • Can the null reach the registration on the first registration time so that the conditional you added will miss this case?

I am quite blind on what this problem solves and in what circumstances the null reaches the registration. Do you have a small test-case to share?

Yes, this line removes the dirtyBatchBuffers before they reach cleanupBatchMode, clearing this collection is now done in cleanupBatchMode.

I agree with the removal of iter.remove() because the clear is done anyway by the cleanupBatchMode.

#17 Updated by Artur Școlnic over 1 year ago

Alexandru Lungu wrote:

Can you describe what are the possible cases in which this addition in required?

This is a widespread issue really.

create tt1.
assign tt1.f1 = 1 tt1.f2 = tt.f2.

For the first call of the invoke in RecordBuffer for tt1.f1 = 1, tt1 will be set as the active buffer with the according runnable.
For the second call of the invoke in RecordBuffer for tt1.f2 = tt.f2, tt1 is already in dirtyBatchBuffers so restoreActiveState is null, but later it is added again to dirtyBatchBuffers.
if (batchAssign)
                  {
                     bufferManager.addDirtyBatchBuffer(RecordBuffer.this, restoreActiveState);
                  }

We don't have anything to prevent adding it again to the dirtyBatchBuffers with the null restoreActiveState.
Actually, we also could add a condition to the conditional above.

#18 Updated by Alexandru Lungu over 1 year ago

I think we need more consistency with the moment when the active buffer is set and activeBufferSet flag is used. So I would expect if (batchAssign && !activeBufferSet) to be a cleaner solution, avoiding passing down null to restoreActiveState.

#19 Updated by Artur Școlnic over 1 year ago

Made the discussed changes, committed rev 15577 and added TestCLOBGetter.cls to the testcases project.

#20 Updated by Alexandru Lungu over 1 year ago

  • Status changed from Review to Internal Test

Review of 15577:

  • I think there is a confusion. I was looking into a branch older than rev. 15573 in which the activeBuffer was set only for setters. Before that, there was an activeBufferSet flag which was a local variable:
    boolean activeBufferSet = bufferManager.isBufferDirtyBatch(RecordBuffer.this);
    • I was thinking of resurrecting it and use it in the if conditional. Having it as a class member causes harder management of the flag (e.g. bound buffers that should inherit that flag, nested logic, etc.)

#21 Updated by Artur Școlnic over 1 year ago

I committed rev 15578 and tested it, the testing went ok.

#22 Updated by Alexandru Lungu over 1 year ago

  • % Done changed from 90 to 100
  • Status changed from Internal Test to Merge Pending

I am OK with the changes in 9365a. Please merge to trunk now.

#23 Updated by Artur Școlnic over 1 year ago

  • % Done changed from 100 to 90

9365a is merged to trunk as rev 15579.

#24 Updated by Artur Școlnic over 1 year ago

  • Status changed from Merge Pending to Test
  • % Done changed from 90 to 100

Also available in: Atom PDF