Project

General

Profile

Bug #9669

Optimize RecordMeta.applyInitialValues

Added by Alexandru Lungu over 1 year ago. Updated over 1 year ago.

Status:
Test
Priority:
High
Target version:
-
Start date:
Due date:
% Done:

100%

billable:
No
vendor_id:
GCD
case_num:
version_reported:
version_resolved:
production:
No
env_name:
topics:

History

#1 Updated by Alexandru Lungu over 1 year ago

  • Assignee set to Artur Școlnic
  • Priority changed from Normal to High

In a profiled scenario (#9652), there are 3.6k calls to RecordMeta.applyInitialValues:

  • all are generated by RecordBuffer.create

The total time is 465ms and 63ms is own time. There are:

  • 104k calls to BitSet.set (401ms)

All times are affected by YK overhead.

Possible improvements:

  • Work similarly to RecordMeta.getValidatableProperties to leverage BitSet.or instead of setting the null properties one-by-one.
  • Check if we can set the initial values faster with a system array copy (at least when there are clearly no dynamicInitializers. In the profiled scenario there are none.

#3 Updated by Artur Școlnic over 1 year ago

  • Status changed from New to WIP

#4 Updated by Artur Școlnic over 1 year ago

  • Status changed from WIP to Review
  • reviewer Alexandru Lungu added

I committed the code to 9669a/15716, it is ready for review.

#5 Updated by Artur Școlnic over 1 year ago

  • Status changed from Review to WIP

I did a bit of testing and found some regressions, will fix them.

#6 Updated by Artur Școlnic over 1 year ago

  • Status changed from WIP to Review

I fixed the issue, 9669a/15717 is ready for review.

#7 Updated by Alexandru Lungu over 1 year ago

  • % Done changed from 0 to 70
  • Status changed from Review to WIP

Review of 9669a:

  • flip operation seems redundant. Instead of or-ing with initialDataBitSet and then flip, please keep the inverted state directly in initialDataBitSet to avoid flipping.
  • initialDataBitSet naming is not suggestive for the fact that the data is null or not. Please rename to initialDataNullProps.
  • if (data != null) is irrelevant because data can't be null at that point (you would have a NPE way before that at arraycopy or data[i] = value; from the DynamicInitializer.
  • DynamicInitializer.getProperties opening brace is not correctly formatted and the method doesn't have a javadoc. Also, it is placed in between class members.
  • // if the schema defines any dynamic initializers for this record, set them first is an invariant that is broken now. Before your changes, the static initial values have higher priority. After your changes, dynamic initializers have higher priority. Please make a test to confirm the order is not relevant or otherwise restore the original order of setters.

#8 Updated by Artur Școlnic over 1 year ago

Alexandru Lungu wrote:

  • flip operation seems redundant. Instead of or-ing with initialDataBitSet and then flip, please keep the inverted state directly in initialDataBitSet to avoid flipping.

My goal was to store the initial non null values, in case this bitset could be used in the future, but storing null properties bitset, for this issue specifically, works better indeed.

  • // if the schema defines any dynamic initializers for this record, set them first is an invariant that is broken now. Before your changes, the static initial values have higher priority. After your changes, dynamic initializers have higher priority. Please make a test to confirm the order is not relevant or otherwise restore the original order of setters.

When it comes to the dynamic initializers, before, they set the values first, then the static initializers ignored the set values, so the dynamic ones had the priority, once set, the dynamic values could not be unset.
Now the static initializers are setting all it's values, then the dynamic ones overwrite the already set (or not) properties, so the dynamic initializers also have the higher priority.

#9 Updated by Alexandru Lungu over 1 year ago

My goal was to store the initial non null values, in case this bitset could be used in the future, but storing null properties bitset, for this issue specifically, works better indeed.

Please apply the change to 9669a.

Now the static initializers are setting all it's values, then the dynamic ones overwrite the already set (or not) properties, so the dynamic initializers also have the higher priority.

Got it, it makes sense!

               nullProps.and(dynamicInitializers[i].getProperties());
               nullProps.andNot(dynamicInitializers[i].getProperties());

Doesn't this make nullProps full of 0? From my understanding:

nullProp dynamicInitializers[i].getProperties() ~dynamicInitializers[i].getProperties() (nullProp & dynamicInitializers[i].getProperties()) & (~dynamicInitializers[i].getProperties())
1 1 0 (1 & 1) & 0 = 0
1 0 1 (1 & 0) & 1 = 0
0 1 0 (0 & 1) & 0 = 0
0 0 1 (0 & 0) & 1 = 0

Please test dynamicInitializers case. I recommend doing different configurations and debug into understanding whether the final nullProp is set right.

#10 Updated by Artur Școlnic over 1 year ago

  • Status changed from WIP to Review

I addressed the review, the code is in rev 15718.

#11 Updated by Artur Școlnic over 1 year ago

Committed a small change to rev 15719.

#12 Updated by Artur Școlnic over 1 year ago

Considering the nature of the changes, I went ahead with the testing, FwdTests and 2 large customer application unit tests passed.

#13 Updated by Alexandru Lungu over 1 year ago

  • Status changed from Review to WIP

Considering the nature of the changes, I went ahead with the testing, FwdTests and 2 large customer application unit tests passed.

Please post the test-cases you built for testing 9669a.

#14 Updated by Alexandru Lungu over 1 year ago

Please post the test-cases you built for testing 9669a.

As part of 9669a review:

  • There is a misalignment of getProperties in regard to the javadoc.
  • nullProps = (BitSet) initialDataNullProps.clone(); this won't work. nullProps is sent as a parameter, so the assignment will simply change its reference and it won't propagate the constructed nullProps upstream.
    • Maybe you can build a blank nullProps, do the logic and return it. In the caller, simply assign it to nullProps
  • I suppose the or is redundant in the following code
                nullProps.or(dynamicInitializers[i].getProperties());
                nullProps.andNot(dynamicInitializers[i].getProperties());
    • Anyway, I think this approach is not completely right because you presume that the dynamicInitializers will always initialize with a non-unknown value. This is not true, the dynamic initializer can do the setting with unknown.

#15 Updated by Artur Școlnic over 1 year ago

Alexandru Lungu wrote:

  • Anyway, I think this approach is not completely right because you presume that the dynamicInitializers will always initialize with a non-unknown value. This is not true, the dynamic initializer can do the setting with unknown.

andNot Clears all of the bits in this BitSet (nullProps) whose corresponding bit is set in the specified BitSet (dyn props).
If the dynamic initializer sets a value, it will be unset in the nullProps, if it does not set it, the corresponding bit in nullProps will remain set.
The truth table for the operation we need is:

nullProps: 0 0 1 1
dynprops:  0 1 0 1
result:    0 0 1 0

andNot seems to be the exact operation we need.
The other issues were solved and the code committed to rev 15720.

#16 Updated by Artur Școlnic over 1 year ago

Alexandru Lungu wrote:

  • Anyway, I think this approach is not completely right because you presume that the dynamicInitializers will always initialize with a non-unknown value. This is not true, the dynamic initializer can do the setting with unknown.

Oh, I think I understood you wrong the first time, you mean that the properties is set even if the supplier of the dynamicInitializers provides a null value. Theoretically, this indeed is an issue with the new changes, though I am not sure if a dynamic initializer can supply a null value. The javadoc states that DynamicInitializer is A helper object which applies the result of an invocation of a dynamic initial value (e.g., "today" for a date, "now" for a datetime[tz]). In my understanding the values should always be non null;

#17 Updated by Artur Școlnic over 1 year ago

  • Status changed from WIP to Review

I committed the latest changes to rev 15721.

#18 Updated by Artur Școlnic over 1 year ago

Removed getProperties in rev 15722.

#19 Updated by Alexandru Lungu over 1 year ago

  • Status changed from Review to Internal Test
  • % Done changed from 70 to 100
I am OK with the changes:
  • please fix if (data[i] != null) { brace positioning.

This can go into internal testing.

#20 Updated by Artur Școlnic over 1 year ago

FwdTests passed.

#21 Updated by Alexandru Lungu over 1 year ago

  • Status changed from Internal Test to Merge Pending

Please merge 9669a to trunk after 9466b.

#22 Updated by Artur Școlnic over 1 year ago

Branch 9669a was merged to trunk as rev.15743 and archived.

#23 Updated by Alexandru Lungu over 1 year ago

  • Status changed from Merge Pending to Test

Also available in: Atom PDF