Project

General

Profile

Bug #9262

batch assign updates the dirty database on each assignment

Added by Constantin Asofiei almost 2 years ago. Updated almost 2 years ago.

Status:
Hold
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
Due date:
% Done:

0%

billable:
No
vendor_id:
GCD
case_num:
version_reported:
version_resolved:
reviewer:
production:
No
env_name:
topics:

History

#2 Updated by Constantin Asofiei almost 2 years ago

I found some cases where DirtyShareContext.insert is being called on each assignment in a batch assign, and not just at the end of the batch. This is accessing the dirty database via an insert/update, and it adds up.

An example which shows this is; in FWD, it takes ~1.5 seconds, while in OE ~350ms. Not all the time is spent in dirty database (I think ~200ms in FWD).

for each book transaction:
delete book.
end.

def var i as int.
def var m1 as int64.
def var m2 as int64.
def var ch as char.
def var d as date.
d = today.

m1 = mtime.
do i = 1 to 1000 transaction:
   ch = string(i).

   do transaction:
      create book.
      assign book.book-id = i
             book.isbn = ch
             book.book-title = ch
             book.author-id = i
             book.publisher = ch
             book.pub-date = d.
      release book.
   end.
end.
m2 = mtime.
message m2 - m1 "ms".

On each assignment in the batch-assign, the H2 dirty-database will be 'hit':

DefaultDirtyShareManager.trackChange(String, Record, List<String>, String[], boolean, IndexState) line: 1724    
DefaultDirtyShareManager.insert(String, Record, IndexState, boolean) line: 418    
DirtyShareContext.insert(RecordBuffer, Record, boolean) line: 353    
RecordBuffer$Handler.invoke(Object, Method, Object[]) line: 12544    
$__Proxy5.setBookId(NumberType) line: not available    
DirtyInsert.lambda$6() line: 75    
13892413.run() line: not available    
RecordBuffer.batch(Runnable) line: 3322    
DirtyInsert.lambda$5() line: 73    
1569400373.body() line: not available    
Block.body() line: 636    
BlockManager.processBody(BlockManager$WorkArea, Block, OnPhrase[], OnPhrase[]) line: 9291    
BlockManager.doBlockWorker(BlockManager$TransactionType, String[], String, OnPhrase[], Block) line: 10537    
BlockManager.doBlock(BlockManager$TransactionType, String, Block) line: 1535    
DirtyInsert.lambda$4() line: 70    
492258452.body() line: not available    
Block.body() line: 636    
BlockManager.processBody(BlockManager$WorkArea, Block, OnPhrase[], OnPhrase[]) line: 9291    
BlockManager.coreLoop(BlockType, boolean, BlockManager$TransactionType, String[], String, ToClause, LogicalOp, OnPhrase[], OnPhrase[], Block) line: 10999    
BlockManager.doLoopWorker(BlockManager$TransactionType, String[], String, ToClause, LogicalOp, OnPhrase[], Block) line: 10826    
BlockManager.doTo(BlockManager$TransactionType, String, ToClause, Block) line: 1929    
DirtyInsert.lambda$0() line: 66

In the #9214 scenario, DirtyShareContext.insert is executed ~0.5 million times, for each individual assignment in the batch assign.

I think we should delay this until batch ends, but a question here is: what happens if we are in the batch assign, but a function is called? I don't recall if we 'pause' this batch assign so the function is not executed in the 'batch assign' context.

#3 Updated by Alexandru Lungu almost 2 years ago

Can you do a similar test with 7156c? 7156c has a reword of RecordNursery and how the dirty changes are registered. More specific, in 7156c the setter is not updating the dirty database - it will only push the changes to the RecordNursery and cache them. The performance drawback you are seeing is due to an aggressive interaction with the dirty database that comes from 6667i in order to make the changes visible to cross-session ASAP. Disabling cross-session should avoid doing such registration.

I think I will need to fast-track the RecordNursery rework from 7156c (former 6667g changes) to trunk to avoid such issues.

#4 Updated by Constantin Asofiei almost 2 years ago

  • Status changed from New to Hold

Yes, with 7156c DirtyShareContext.insert is being called on batch end, so assuming the changes in 7156c will reach trunk, I'm placing this on hold. But, with trunk it is also being called on buffer.create() - this no longer happens:

DirtyShareContext.insert(RecordBuffer, Record, boolean) line: 327    
RecordNursery.indexUpdated(RecordBuffer, BitSet, boolean, BaseRecord) line: 158    
RecordBuffer.create() line: 10248    
$__Proxy5(BufferImpl).create() line: 1665

Is this intended?

#5 Updated by Alexandru Lungu almost 2 years ago

Yes, in 7156c the RecordNursery will hold onto all inserts/updates till the first query will occur. This is how trunk pre-6667i was working anyway (but only for CREATES as UPDATES were not supported by RecordNursery). Post-6667i, everything is sent to the dirty database super-eagerly.

Note however that 7156c was not tested against cross-session cases. It was designed to support intra-session as best as possible. Leveraging the cross-session with 7156c implementation is WIP. Most probably the directory.xml settings will decide if super-eager dirty database flushing should happen or not. In other words, if cross-session will be enabled, everything will be sent to the dirty database ASAP and RecordNursery will be a no-op. If cross-session will be disabled, RecordNursery will cache all operations as much as possible.

Your case in #9262-2 will still be slow if cross-session is enabled. But if it will be disabled and 7156c chanegs will reach trunk, the problem will be fixed.

Also available in: Atom PDF