Bug #9669
Optimize RecordMeta.applyInitialValues
100%
History
#1 Updated by Alexandru Lungu over 1 year ago
- Assignee set to Artur Școlnic
- Priority changed from Normal to High
In a profiled scenario (#9652), there are 3.6k calls to RecordMeta.applyInitialValues:
- all are generated by
RecordBuffer.create
The total time is 465ms and 63ms is own time. There are:
- 104k calls to BitSet.set (401ms)
All times are affected by YK overhead.
Possible improvements:
- Work similarly to
RecordMeta.getValidatablePropertiesto leverageBitSet.orinstead of setting the null properties one-by-one. - Check if we can set the initial values faster with a system array copy (at least when there are clearly no
dynamicInitializers. In the profiled scenario there are none.
#3 Updated by Artur Școlnic over 1 year ago
- Status changed from New to WIP
#4 Updated by Artur Școlnic over 1 year ago
- Status changed from WIP to Review
- reviewer Alexandru Lungu added
I committed the code to 9669a/15716, it is ready for review.
#5 Updated by Artur Școlnic over 1 year ago
- Status changed from Review to WIP
I did a bit of testing and found some regressions, will fix them.
#6 Updated by Artur Școlnic over 1 year ago
- Status changed from WIP to Review
I fixed the issue, 9669a/15717 is ready for review.
#7 Updated by Alexandru Lungu over 1 year ago
- % Done changed from 0 to 70
- Status changed from Review to WIP
Review of 9669a:
flipoperation seems redundant. Instead ofor-ing withinitialDataBitSetand then flip, please keep the inverted state directly ininitialDataBitSetto avoid flipping.initialDataBitSetnaming is not suggestive for the fact that the data is null or not. Please rename toinitialDataNullProps.if (data != null)is irrelevant because data can't be null at that point (you would have a NPE way before that atarraycopyordata[i] = value;from theDynamicInitializer.DynamicInitializer.getPropertiesopening brace is not correctly formatted and the method doesn't have a javadoc. Also, it is placed in between class members.// if the schema defines any dynamic initializers for this record, set them firstis an invariant that is broken now. Before your changes, the static initial values have higher priority. After your changes, dynamic initializers have higher priority. Please make a test to confirm the order is not relevant or otherwise restore the original order of setters.
#8 Updated by Artur Școlnic over 1 year ago
Alexandru Lungu wrote:
flipoperation seems redundant. Instead ofor-ing withinitialDataBitSetand then flip, please keep the inverted state directly ininitialDataBitSetto avoid flipping.
My goal was to store the initial non null values, in case this bitset could be used in the future, but storing null properties bitset, for this issue specifically, works better indeed.
// if the schema defines any dynamic initializers for this record, set them firstis an invariant that is broken now. Before your changes, the static initial values have higher priority. After your changes, dynamic initializers have higher priority. Please make a test to confirm the order is not relevant or otherwise restore the original order of setters.
When it comes to the dynamic initializers, before, they set the values first, then the static initializers ignored the set values, so the dynamic ones had the priority, once set, the dynamic values could not be unset.
Now the static initializers are setting all it's values, then the dynamic ones overwrite the already set (or not) properties, so the dynamic initializers also have the higher priority.
#9 Updated by Alexandru Lungu over 1 year ago
My goal was to store the initial non null values, in case this bitset could be used in the future, but storing null properties bitset, for this issue specifically, works better indeed.
Please apply the change to 9669a.
Now the static initializers are setting all it's values, then the dynamic ones overwrite the already set (or not) properties, so the dynamic initializers also have the higher priority.
Got it, it makes sense!
nullProps.and(dynamicInitializers[i].getProperties());
nullProps.andNot(dynamicInitializers[i].getProperties());
Doesn't this make nullProps full of 0? From my understanding:
| nullProp | dynamicInitializers[i].getProperties() | ~dynamicInitializers[i].getProperties() | (nullProp & dynamicInitializers[i].getProperties()) & (~dynamicInitializers[i].getProperties()) |
|---|---|---|---|
| 1 | 1 | 0 | (1 & 1) & 0 = 0 |
| 1 | 0 | 1 | (1 & 0) & 1 = 0 |
| 0 | 1 | 0 | (0 & 1) & 0 = 0 |
| 0 | 0 | 1 | (0 & 0) & 1 = 0 |
Please test dynamicInitializers case. I recommend doing different configurations and debug into understanding whether the final nullProp is set right.
#10 Updated by Artur Școlnic over 1 year ago
- Status changed from WIP to Review
I addressed the review, the code is in rev 15718.
#11 Updated by Artur Școlnic over 1 year ago
Committed a small change to rev 15719.
#12 Updated by Artur Școlnic over 1 year ago
Considering the nature of the changes, I went ahead with the testing, FwdTests and 2 large customer application unit tests passed.
#13 Updated by Alexandru Lungu over 1 year ago
- Status changed from Review to WIP
Considering the nature of the changes, I went ahead with the testing, FwdTests and 2 large customer application unit tests passed.
Please post the test-cases you built for testing 9669a.
#14 Updated by Alexandru Lungu over 1 year ago
Please post the test-cases you built for testing 9669a.
As part of 9669a review:
- There is a misalignment of
getPropertiesin regard to the javadoc. nullProps = (BitSet) initialDataNullProps.clone();this won't work.nullPropsis sent as a parameter, so the assignment will simply change its reference and it won't propagate the constructednullPropsupstream.- Maybe you can build a blank
nullProps, do the logic and return it. In the caller, simply assign it tonullProps
- Maybe you can build a blank
- I suppose the
oris redundant in the following codenullProps.or(dynamicInitializers[i].getProperties()); nullProps.andNot(dynamicInitializers[i].getProperties());- Anyway, I think this approach is not completely right because you presume that the
dynamicInitializerswill always initialize with a non-unknown value. This is not true, the dynamic initializer can do the setting with unknown.
- Anyway, I think this approach is not completely right because you presume that the
#15 Updated by Artur Școlnic over 1 year ago
Alexandru Lungu wrote:
- Anyway, I think this approach is not completely right because you presume that the
dynamicInitializerswill always initialize with a non-unknown value. This is not true, the dynamic initializer can do the setting with unknown.
andNot Clears all of the bits in this BitSet (nullProps) whose corresponding bit is set in the specified BitSet (dyn props).
If the dynamic initializer sets a value, it will be unset in the nullProps, if it does not set it, the corresponding bit in nullProps will remain set.
The truth table for the operation we need is:
nullProps: 0 0 1 1 dynprops: 0 1 0 1 result: 0 0 1 0
andNot seems to be the exact operation we need.
The other issues were solved and the code committed to rev 15720.
#16 Updated by Artur Școlnic over 1 year ago
Alexandru Lungu wrote:
- Anyway, I think this approach is not completely right because you presume that the
dynamicInitializerswill always initialize with a non-unknown value. This is not true, the dynamic initializer can do the setting with unknown.
Oh, I think I understood you wrong the first time, you mean that the properties is set even if the supplier of the dynamicInitializers provides a null value. Theoretically, this indeed is an issue with the new changes, though I am not sure if a dynamic initializer can supply a null value. The javadoc states that DynamicInitializer is A helper object which applies the result of an invocation of a dynamic initial value (e.g., "today" for a date, "now" for a datetime[tz]). In my understanding the values should always be non null;
#17 Updated by Artur Școlnic over 1 year ago
- Status changed from WIP to Review
I committed the latest changes to rev 15721.
#18 Updated by Artur Școlnic over 1 year ago
Removed getProperties in rev 15722.
#19 Updated by Alexandru Lungu over 1 year ago
- Status changed from Review to Internal Test
- % Done changed from 70 to 100
- please fix
if (data[i] != null) {brace positioning.
This can go into internal testing.
#20 Updated by Artur Școlnic over 1 year ago
FwdTests passed.
#21 Updated by Alexandru Lungu over 1 year ago
- Status changed from Internal Test to Merge Pending
Please merge 9669a to trunk after 9466b.
#22 Updated by Artur Școlnic over 1 year ago
Branch 9669a was merged to trunk as rev.15743 and archived.
#23 Updated by Alexandru Lungu over 1 year ago
- Status changed from Merge Pending to Test