Project

General

Profile

Bug #9701

RecordBuffer.updateCurrentChanged can be optimized to use a fast DMO equality check

Added by Alexandru Lungu over 1 year ago. Updated over 1 year ago.

Status:
Test
Priority:
Normal
Assignee:
Andrei Plugaru
Target version:
-
Start date:
Due date:
% Done:

100%

billable:
No
vendor_id:
GCD
case_num:
version_reported:
version_resolved:
production:
No
env_name:
topics:

History

#1 Updated by Alexandru Lungu over 1 year ago

  • Assignee set to Artur Școlnic

In a profiled scenario (#9692), there are 3.5k RecordBuffer.updateCurrentChanged calls. The DMOs used by this have 137 and 191 columns (mostly are expanded extents). Because of that, it generates >2M handler.getField calls, as this method is iterating all properties and calls handler.getField on the shadow copy record and on the current record. The worst part is that half of them resolve to Record._getDecimal calls, which instantiate a new uncachable decimal, just for temporary usage.

A possible attempt is to extend #9060 to cover decimals and record getters. But I think this path has a high chance to fail because decimal cache may have a high variance that won't solve #9692.

Thus, I suggest refining updateCurrentChanged to work with Java types instead of BDT. However, I can see some limitations on that on getters that are dependent upon context (i.e. _getCharacter which is dependent upon the case-sensitivity). However, in most cases, two equal Java types retrieved from the database should yield two equal BDT.

Mind that a really important invariant of updateCurrentChanged is that the shadowCopyRecord and currentRecord are on the same structure and DMO, so there is no way we can end up comparing decimals with different scale, characters with different case-sensitivity or others. However, extra caution should be taken for unknown cases.

#2 Updated by Andrei Plugaru over 1 year ago

  • Assignee changed from Artur Școlnic to Andrei Plugaru

#3 Updated by Andrei Plugaru over 1 year ago

  • Status changed from New to WIP

Created branch 9701a from rev. 15734.

#4 Updated by Andrei Plugaru over 1 year ago

At the moment, I am thinking of overriding the equals in BaseRecord and make it check the elements in data array. I think this is right based on the invariant posted by Alex.

#5 Updated by Andrei Plugaru over 1 year ago

I found this note in CompareOps.equals: // special case: an empty comhandle is considered be equal to the zero integer, so will take this into account.

LE: I actually don't think this is an issue, as we should have the same type on the same position in the data arrays, based on the invariant.

#6 Updated by Alexandru Lungu over 1 year ago

At the moment, I am thinking of overriding the equals in BaseRecord and make it check the elements in data array. I think this is right based on the invariant posted by Alex.

This is risky. I think there are place in FWD that use equals (from Object) for that matter and they rely on == comparison. Overriding equals may break such cases:

  • Please investigate if there are equals comparisons of BaseRecord across FWD.

#7 Updated by Andrei Plugaru over 1 year ago

Alexandru Lungu wrote:

  • Please investigate if there are equals comparisons of BaseRecord across FWD.

I actually found just one usage of Record.equals in LocalSyncher.preSynch: if (foreignDMO null || foreignDMO.equals(newForeignDMO)). We can easily replace this with == comparasion.

#8 Updated by Alexandru Lungu over 1 year ago

I actually found just one usage of Record.equals in LocalSyncher.preSynch: if (foreignDMO null || foreignDMO.equals(newForeignDMO)). We can easily replace this with == comparasion.

Got it, please go ahead.

Also, double-check this. It is crucial to ensure equals was not used somewhere else already for comparing BaseRecord.

#9 Updated by Andrei Plugaru over 1 year ago

equals is used explicitly only there, however I just realised it could be used as key in collections like Set or HashMaps, therefore equals will be called. And indeed, at a quick search, it is used in Set s multiple times.

So, I think I will abandon the idea to override equals, and, simply create another method which will do this checking.

#10 Updated by Andrei Plugaru over 1 year ago

  • Status changed from WIP to Review
  • reviewer Alexandru Lungu added

I committed 9701a/15735. As I said in the last note, I just created a new method in Record to check if the data arrays are equal. I have also ran the fwd tests of a large customer application, with an assert checking if my new method returns the same result as the old algorithm and it never failed.

Alex, please take a look when you are available.

#11 Updated by Alexandru Lungu over 1 year ago

  • reviewer Ovidiu Maxiniuc added

So, I think I will abandon the idea to override equals, and, simply create another method which will do this checking.

Good catch.

I have also ran the fwd tests of a large customer application, with an assert checking if my new method returns the same result as the old algorithm and it never failed.

Assert as in assert statement from Java. If so, did you run the Java process with -ea to activate it? Personally, I rather use debug mode and use a breakpoint for that matter.

Review of #9701:
  • I am OK with the changes.

In fact, I think this is functionally better, because String are compared case-sensitive now instead of using characters which can be case-insensitive. updateCurrentChanged should detect the case changes, but in trunk it doesn't ... with your changes it does.

I also want Ovidiu to have a second look to the changes. I also imagined something similar when posting #9701, so I may be biased in my review. Let Ovidiu provide a fresh view of this.

#12 Updated by Andrei Plugaru over 1 year ago

Alexandru Lungu wrote:

Assert as in assert statement from Java. If so, did you run the Java process with -ea to activate it?

Yes, I modified server.sh with that option. First, I just used an assert which always failed to be sure it was working as expected :)

Personally, I rather use debug mode and use a breakpoint for that matter.

Yeah, probably would have been easier.

#13 Updated by Greg Shah over 1 year ago

Assert as in assert statement from Java. If so, did you run the Java process with -ea to activate it? Personally, I rather use debug mode and use a breakpoint for that matter.

These must be removed. We never use Java assert. It has the potential to cause crashes in production depending on command line options which don't control. That is not OK.

#14 Updated by Andrei Plugaru over 1 year ago

I just used the assert to validate that I get the same results with the new implementation vs the old one while running some tests of a large customer application. I deleted the assert before committing.

#15 Updated by Greg Shah over 1 year ago

OK

#16 Updated by Andrei Plugaru over 1 year ago

Ovidiu, please take a look over the changes in 9701a when you are available.

#17 Updated by Alexandru Lungu over 1 year ago

Andrei, please do other remaining testing until Ovidiu provides a final Review (unit tests or/and ChUi regression tests). This should reach trunk in the next couple of hours.

#18 Updated by Andrei Plugaru over 1 year ago

I will be following this test plan:
  • unit tests for a large customer application [ai]
  • fwd tests of another large customer application [me]

#19 Updated by Andrei Plugaru over 1 year ago

The unit tests of a customer passed, however I have one more failing test in the fwd tests. So, I am running the fwd tests again.

#20 Updated by Andrei Plugaru over 1 year ago

I have ran the fwd tests and they show no regressions.

#21 Updated by Ovidiu Maxiniuc over 1 year ago

  • % Done changed from 0 to 100
  • Status changed from Review to Internal Test

I have reviewed the changes in 9701a/r15735.
I agree with the new code (with the reservation of the problem related to partial hydration, already documented in the javadoc).

#22 Updated by Andrei Plugaru over 1 year ago

Thanks, Ovidiu, for the review.

As we want to test it more aggressively, I propose the following tests, apart from the one already did for #9701-18:
  • unit tests of a large customer application [me]
  • harness tests of another client [ls]

Ovidiu/Alex, do you think anything else is necessary?

#23 Updated by Ovidiu Maxiniuc over 1 year ago

I think that should suffice. The changes are right at the heart of the persistence and surely the affected code will be executed with a lot of data.

#24 Updated by Eric Faulhaber over 1 year ago

Andrei Plugaru wrote:

Alexandru Lungu wrote:

  • Please investigate if there are equals comparisons of BaseRecord across FWD.

I actually found just one usage of Record.equals in LocalSyncher.preSynch: if (foreignDMO null || foreignDMO.equals(newForeignDMO)). We can easily replace this with == comparasion.

Please note that LocalSyncher is effectively dead code. We really should remove its use. It is based on the notion that we can have foreign keys between primary tables. We have long since determined that this is not feasible, since there is no referential integrity enforced by the Progress database, so much application code is written that would break referential integrity. This was causing database errors when applied to modern DBMS', so we stopped using it.

Persistence.isForeignKeysEnabled really should return false, or be removed entirely (along with code that depends on it returning true).

#25 Updated by Andrei Plugaru over 1 year ago

Eric Faulhaber wrote:

Please note that LocalSyncher is effectively dead code. We really should remove its use. It is based on the notion that we can have foreign keys between primary tables. We have long since determined that this is not feasible, since there is no referential integrity enforced by the Progress database, so much application code is written that would break referential integrity. This was causing database errors when applied to modern DBMS', so we stopped using it.

In the end, actually no changes to LocalSyncher were necessary for this task, but I see your point that we have to refactor to remove that parts of code.

If we are not in a rush for 9701a to be merged, we can do this here, if not, defer to another task.

#26 Updated by Andrei Plugaru over 1 year ago

As the proposed changes are not really related to this task, I created a new one - #9714, which will address the removal of LocalSyncher and the other necessary changes.

I will move on to the testing proposed in #9701-22.

#27 Updated by Andrei Plugaru over 1 year ago

All the testing showed no regressions. ✅

I think this can be queued in for merge.

#28 Updated by Alexandru Lungu over 1 year ago

  • Status changed from Internal Test to Merge Pending

Please merge 9701a to trunk now.

#29 Updated by Andrei Plugaru over 1 year ago

I have merged to trunk as rev 15750.

Danut, should I also port it to 7156c and 7156e?

#30 Updated by Dănuț Filimon over 1 year ago

Andrei Plugaru wrote:

I have merged to trunk as rev 15750.

Danut, should I also port it to 7156c and 7156e?

Port the change to both branches (7156c and 7156e). See #7156-457 for reference.

#31 Updated by Andrei Plugaru over 1 year ago

  • Status changed from Merge Pending to Test

I have now also ported the changes to 7156c/15546 and 7156e/15734.

Also available in: Atom PDF