FWD » Core Development » Database

We need a LRU cache for the FQLPreprocessor.translate result. The cache key will be similar to existing FQL keys, but without parameters and maybe others (keep it simple, just the bound/definition aliases, original where, etc).

Alexandru: can you add this to the existing sprint?

Alexandru: can you add this to the existing sprint?

Ok; do you have a small test-case in which that part of code is hit - to be used for testing?

The recreate is this:

def temp-table tt1 reference-only field f1 as int.

def temp-table tt2 field f2 as int.
   def var k as int.

procedure proc0.
   def input parameter table for tt1 bind.
   for each tt1 where tt1.f1 > 0:
      k = k + 1.
   end.
end.

create tt2.
tt2.f2 = 10.
release tt2.

def var i as int.
def var l1 as int64.
def var l2 as int64.

l1 = mtime.
do i = 1 to 100000:
   run proc0(input table tt2 bind) .
end.
l2 = mtime.

message k "in" (l2 - l1) "ms".

Try to also cache the translateSort result.

Main focus here is memory reduction.

Stefanel Pezamosca wrote:

First try: it seems like caching the translateSort output does not bring any improvements.

Please add an index to both temp-tables and test with that.

Stefanel Pezamosca wrote:

I committed to revision 15501: Added caching for translateSort(RecordBuffer, RecordBuffer, String) output.
Please review.

1. the synchronization. In AbstractQuery.translateSort() and also in FQLPreprocessor.translate(), when the sortCache / translateCache are accessed, the get and put use different synchronized blocks. The problem is that between them, other thread may access the cache object. I think a single lock on it should span on both accesses;
2. possible NPE in FQLPreprocessor$TranslateCacheKey. When the objects are compared in equals(), where.equals(that.where) is used. When the translate is called from AbstractQuery.translateWhere(), there is a guard for null where, but if the call comes from one of the static Buffer.delete() methods, the where parameter may be null.

Ovidiu Maxiniuc wrote:

1. the synchronization. In AbstractQuery.translateSort() and also in FQLPreprocessor.translate(), when the sortCache / translateCache are accessed, the get and put use different synchronized blocks. The problem is that between them, other thread may access the cache object. I think a single lock on it should span on both accesses;

I would rather avoid lock contention and let the threads do some additional work, but with the same results.

1. possible NPE in FQLPreprocessor$TranslateCacheKey. When the objects are compared in equals(), where.equals(that.where) is used. When the translate is called from AbstractQuery.translateWhere(), there is a guard for null where, but if the call comes from one of the static Buffer.delete() methods, the where parameter may be null.

Good catch. Stefanel: please add check at the top of FQLPreprocessor.translate, if the where clause is null, just exit and return null, as there is nothing to translate or cache.

What are the next steps with 9251b?

9251b should have been merged to trunk. It's just a null check.

Please add javadoc for the new params for the ctors in SortCacheKey and TranslateCacheKey. OTherwise 9251c/15631 looks ok. Please work with Eduard to confirm that the 7156d bug is solved with this.

Constantin Asofiei wrote:

Please add javadoc for the new params for the ctors in SortCacheKey and TranslateCacheKey. OTherwise 9251c/15631 looks ok. Please work with Eduard to confirm that the 7156d bug is solved with this.

javadoc fixed in revision 15632.

I tested the patch from #9251-20, and tested a few modules in the unit tests in a large customer application and did not encounter the regressions that I mentioned in #9494-1 and #9494-4.

Stefanel Pezamosca wrote:

Considering that 9251c is fixing the regression mentioned and I also retested #9251-3 with similar results, I guess that is ready to be merged?

What other testing did you do/is needed?

Please update the copyright year in the changed files.

Constantin Asofiei wrote:

Stefanel Pezamosca wrote:

Considering that 9251c is fixing the regression mentioned and I also retested #9251-3 with similar results, I guess that is ready to be merged?

What other testing did you do/is needed?

I ran the same tests that were done before for the 9251a and they are ok.

Please update the copyright year in the changed files.

Done.