FWD » Core Development » Database

08/18/2023

10:53 AM Bug #7454: Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

Dănuț Filimon wrote:
> upper(tt.f1) in ('A', 'B') and upper(tt.f1) is null look to be very specific.
Not really;... Alexandru Lungu

09:49 AM Bug #7454: Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

Alexandru Lungu wrote:
> Danut, please let me know if there is anything left to be worked on this task. If everythin... Dănuț Filimon

09:39 AM Bug #7454 (Review): Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

*Review of 7454a and 7454a_h2*
* I am OK with the changes.
* Regression tests passed.
* I am *no* longer hitting... Alexandru Lungu

04:15 AM Bug #7454: Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

I've looked into the problem and @upper@ methods are not reintroduced later in @FQLPreprocessor.mainWalk()@. It was a... Dănuț Filimon

07:22 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

Dănuț Filimon wrote:
> Can you please elaborate?
Create a dynamic buffer with @CREATE BUFFER buf@. Do similar wor... Alexandru Lungu

06:48 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

Alexandru Lungu wrote:
> What happens to the other buffers (on other tables / other multiplex)? What if we iterate w... Dănuț Filimon

04:24 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

*Review of 7535a*
* Doing a reclaim on @iterate@ looks right. However, I think @iterate@ is called only for the bu... Alexandru Lungu

02:28 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

Eric Faulhaber wrote:
> I wouldn't expect @BY RECID DESC@ to work. That doesn't look like valid 4GL code.
>
> Tha... Dănuț Filimon

08/17/2023

12:09 PM Bug #7535: Temporary tables require trailing recid component even for unique indexes

Dănuț Filimon wrote:
> When running the @test01.i with tt01 BY RECID DESC@ test, I end up with this select statement... Eric Faulhaber

07:37 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

Alexandru Lungu wrote:
> Danut, can you paste here the testcase which causes the issue in #7535-35? Thank you.
The ... Dănuț Filimon

07:07 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

Danut, can you paste here the testcase which causes the issue in #7535-35? Thank you. Alexandru Lungu

04:20 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

I also found @CompoundQuery.cleaner@ which is also a @Finalizable@ instance, it's a similar situation to @PreselectQu... Dănuț Filimon

03:58 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

*Committed 7535a/rev.14673*. This is a necessary change that I mentioned in #7535-29 about the @PrelectQuery.cleaner@... Dănuț Filimon

03:46 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

Alexandru Lungu wrote:
> Danut, regarding the recid mismatch, please refer to #7647-2. Use Ovidiu's patch just to cl... Dănuț Filimon

03:23 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

> Currently I can't test this assumption because of an ongoing issue with jboss repository resource which stops the p... Alexandru Lungu

03:20 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

Danut, regarding the recid mismatch, please refer to #7647-2. Use Ovidiu's patch just to clear up *some* mismatches. ... Alexandru Lungu

11:15 AM Bug #7488: Slow fast-copy with before tables in H2

*Review of 7404a*
This is a good start, but it is still far from ideal at this point. Radu, please consider doing ... Alexandru Lungu

06:26 AM Bug #7488: Slow fast-copy with before tables in H2

I've done some refactoring to @FastCopyHelper@:
* Added a cache that stores the instances of @FastCopyHelper@ based ... Radu Apetrii

09:42 AM Feature #7504: Export data tool from FWD into .d files

Another configuration option to consider is to allow a dump of only a subset of tables (perhaps an inclusive or exclu... Eric Faulhaber

08:21 AM Feature #7504: Export data tool from FWD into .d files

Greg Shah wrote:
> We might want to consider reading the configuration from the directory. In that case it will req... Alexandru Lungu

07:41 AM Feature #7504: Export data tool from FWD into .d files

Also I should point out that export will be used on production systems that won't have the standard application cfg p... Greg Shah

07:33 AM Feature #7504: Export data tool from FWD into .d files

We might want to consider reading the configuration from the directory. In that case it will require little or no pa... Greg Shah

07:19 AM Feature #7504: Export data tool from FWD into .d files

> Yes, any such utilities will often have some requirement for configuration. But we don't want them to be hard coded... Alexandru Lungu

07:11 AM Feature #7504: Export data tool from FWD into .d files

> Greg, by stored metadata dynamically you mean the beans we can find in the @dmo@ package (read annotations, hydrate... Greg Shah

06:49 AM Feature #7504: Export data tool from FWD into .d files

> I actually would like to move away from the current ant import.db and ImportWorker approach. These rely upon TRPL a... Alexandru Lungu

08:58 AM Bug #7454: Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

Alexandru Lungu wrote:
> I still have a concern with indexOf function. It takes two strings as parameters. It attemp... Dănuț Filimon

06:51 AM Bug #7454: Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

Regression tests passed.
Profiling tests shown -0.3% improvement (without temporary sequences). This is modest, but ... Alexandru Lungu

07:20 AM Bug #7496: finish support for query:forward-only attribute

*Committed 7496a/rev.14656*. I brought the changes made by Stefan in #7241 (7241b) to 7496a.
I've done tests with ... Dănuț Filimon

08/16/2023

06:52 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

When running the @test01.i with tt01 BY RECID DESC@ test, I end up with this select statement: ... Dănuț Filimon

03:44 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

I mentioned that keys are not reclaimed in simple @FOR EACH@ blocks and after making the necessary changes to reclaim... Dănuț Filimon

08/15/2023

03:45 AM Feature #7382: Check performance of delete from vs drop table in H2

Constantin Asofiei wrote:
> What I meant in FWD-H2 is to check if the number of selected records which are to be del... Dănuț Filimon

03:20 AM Bug #7641: Lower the number of Map.get() calls in the persistence layer

Made @fieldInfo@ a simple Java array, also added to @getMutableInfo@ a new parameter @fieldCount@, those meaning is n... Eduard Soltan

08/14/2023

02:46 AM Bug #7641: Lower the number of Map.get() calls in the persistence layer

Eduard, fieldInfo (and other structures related to fields) can be 'frozen' as a Java array once this table info was b... Constantin Asofiei

02:40 AM Feature #7382: Check performance of delete from vs drop table in H2

Alexandru, I didn't mean to change the DELETE SQL. What I meant in FWD-H2 is to check if the number of selected reco... Constantin Asofiei

08/11/2023

05:27 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

*Committed 7535a/rev.14672*. Applied the patch from #7535-26 after making another small change.
* Added reclaimKeys ... Dănuț Filimon

08/09/2023

04:56 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

Using the patch posted in #7535-26, I've done similar tests to #7535-15 on 7535a using FWD-22 and all the previous pa... Dănuț Filimon

08/08/2023

08:39 AM Feature #7504: Export data tool from FWD into .d files

From Stefan via email:
> I created an ExportWorker class for #7504 and now I am trying to understand @ant import.d... Greg Shah

07:03 AM Bug #7667: Support OUTER-JOIN with PRESELECT at conversion

Andrei, please address #7667-2. Also, please make some tests with @CAN-FIND@ for that matter. Alexandru Lungu

05:32 AM Bug #7667: Support OUTER-JOIN with PRESELECT at conversion

Please note that the conversion of the @CompoundQuery@ to the @PreselectQuery@ should not be applied if one of the co... Igor Skornyakov

04:15 AM Bug #7667 (WIP): Support OUTER-JOIN with PRESELECT at conversion

This is a continuation of the #6196 effort of optimizing @CompoundQuery@ to merge components that have OUTER-JOIN. In... Alexandru Lungu

04:06 AM Bug #7667 (Test): Support OUTER-JOIN with PRESELECT at conversion

Alexandru Lungu

04:40 AM Feature #7323: Implement soft unique index in FWD-H2

I retested with the latest changes in 7323a and got -1% improvement now. This starts looking good. At this point I am... Alexandru Lungu

03:53 AM Bug #7496: finish support for query:forward-only attribute

There are a few things wrong with the patch from #7496-22 which I just found out while testing.
# TemporaryBuffer do... Dănuț Filimon

03:27 AM Bug #7496: finish support for query:forward-only attribute

Danut, please find a concrete example where 7496a (together with 7241b) is failing before @SessionListener.Event.TRAN... Alexandru Lungu

08/07/2023

10:12 AM Bug #7641: Lower the number of Map.get() calls in the persistence layer

Committed on 7641a revision 14683.
@fieldInfo@ in @AbstractTempTable@ became an @ArrayList@, instead of @HashMap@.... Eduard Soltan

04:01 AM Bug #7641: Lower the number of Map.get() calls in the persistence layer

Created 7641a. Alexandru Lungu

10:01 AM Bug #7496: finish support for query:forward-only attribute

I previously tested a large customer application using the changes from 7241b/rev.14655, 7496a/rev.14655 and a few ch... Dănuț Filimon

05:23 AM Bug #7334: Reclaim used sessions to improve performance

Session reclaiming couldn't be disabled since most values either default to a set value or only disable the managing ... Dănuț Filimon

08/04/2023

10:08 AM Bug #7641: Lower the number of Map.get() calls in the persistence layer

In @TempTableMapper@ there is a @private final Map<TempTable, LegacyTableInfo> p2j = new HashMap<>();@ which should b... Eduard Soltan

06:51 AM Feature #7382: Check performance of delete from vs drop table in H2

From Constantin's solution, I think we can just generate @delete from tt1@ when we know this as the last multiplex is... Alexandru Lungu

08/03/2023

08:05 AM Feature #7323: Implement soft unique index in FWD-H2

I've committed to 7323a, rev. 14690 and 7323a_h2, rev. 23. I added support for the @DirectAccess@ idea mentioned in #... Radu Apetrii

05:05 AM Feature #7323: Implement soft unique index in FWD-H2

Alexandru Lungu wrote:
> *Radu, what is your input on this idea?*
I will investigate the aspects mentioned and I wi... Radu Apetrii

05:00 AM Feature #7323: Implement soft unique index in FWD-H2

I don't expect a miracle from @VALIDATE@, but 30k SELECT statements are in fact a lot (from my POV). I agree that the... Alexandru Lungu

04:28 AM Feature #7323: Implement soft unique index in FWD-H2

Alexandru Lungu wrote:
> Clearly we should see better improvement as now we have ~28k less SELECT statements, right?... Radu Apetrii

03:26 AM Bug #7488: Slow fast-copy with before tables in H2

Radu, there is some work in #7404 regarding the moving of most of the copy-temp-table operations to fast-copy using h... Alexandru Lungu

08/02/2023

04:19 AM Feature #7416: Reiterate DatabaseStatistics and leverage to other database profling tools

I have also created a script to analyze the following things:
* How many times each type of query was executed.
* W... Radu Apetrii

04:06 AM Feature #7416: Reiterate DatabaseStatistics and leverage to other database profling tools

I've added some changes to 7416b, rev. 14681. They include the following:
* Fixed the javadoc mentioned in #7416-23.... Radu Apetrii

08/01/2023

08:13 AM Feature #7365 (Closed): Avoid using reflection in generating BDT

Greg Shah

08:02 AM Feature #7365 (Test): Avoid using reflection in generating BDT

This was merged in trunk by 7026e and can be closed. Alexandru Lungu

08:02 AM Feature #7365 (Review): Avoid using reflection in generating BDT

Alexandru Lungu

04:55 AM Bug #7334 (WIP): Reclaim used sessions to improve performance

> Even though Session originally was intended to be used for a single connection check-out, should we be checking in ... Alexandru Lungu

07/31/2023

07:52 PM Feature #4722: data import should be able to run with only the converted application jar file (an...

After your requested update to my parameters, I have been able to make the connection and import... thank you for the... Roger Borrello

02:01 PM Bug #7647: Leverage internal H2 _ROWID_ with FWD recid

While working on first implementation of datasets I encountered an issue related to records from different temp-table... Ovidiu Maxiniuc

11:30 AM Bug #7647: Leverage internal H2 _ROWID_ with FWD recid

H2 is keeping an internal _rowid_ information for each row. For in-memory tables, this is basically the index at whic... Alexandru Lungu

11:15 AM Bug #7647 (New): Leverage internal H2 _ROWID_ with FWD recid

Alexandru Lungu

11:32 AM Bug #7334: Reclaim used sessions to improve performance

(In a different task) Alexandru Lungu wrote:
> > > Otherwise, there are 20 org.postgresql.jdbc.PgConnection, some ar... Eric Faulhaber

08:03 AM Bug #7641 (WIP): Lower the number of Map.get() calls in the persistence layer

Alexandru Lungu

07/29/2023

09:21 PM Bug #7645: Simplify _meta database initialization by removing <schema>.meta.xml resource

Currently, the population of schema related tables as part of the initialization of _meta database uses two sources:
... Ovidiu Maxiniuc

08:58 PM Bug #7645 (Test): Simplify _meta database initialization by removing <schema>.meta.xml resource

Ovidiu Maxiniuc

07/28/2023

03:22 PM Feature #7504: Export data tool from FWD into .d files

Greg Shah wrote:
> > The database can be spoiled with different kind of extra tables / fields we don't actually want... Ovidiu Maxiniuc

02:52 PM Feature #7504: Export data tool from FWD into .d files

I agree our ant build should have an option to use this, but I do want our utility code (Java) to be a complete stand... Greg Shah

07:51 AM Feature #7504: Export data tool from FWD into .d files

h3. Plan
We need an ant option for this. To be consistent, I suggest something like @ant export.db@ that relies on... Alexandru Lungu

11:14 AM Bug #7641: Lower the number of Map.get() calls in the persistence layer

From Constantin:
* *AbstractTempTable.getMutableInfo* - a plain array can be kept for cases when the @LegacyFieldI... Alexandru Lungu

11:12 AM Bug #7641 (Rejected): Lower the number of Map.get() calls in the persistence layer

Alexandru Lungu

10:23 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

Keys should be reclaimed at the end of each iteration and when the block finishes, they should not be reclaimed too e... Dănuț Filimon

09:22 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

Alexandru Lungu wrote:
> So what is the problem then? If @TemporaryBuffer.Context.commit@ is called:
> * is it call... Dănuț Filimon

08:37 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

So what is the problem then? If @TemporaryBuffer.Context.commit@ is called:
* is it called at bad times? too late or... Alexandru Lungu

08:14 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

Alexandru Lungu wrote:
> Note that @TemporaryBuffer.Context@ implements @Commitable@ and there is the whole reclaim ... Dănuț Filimon

07:57 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

> TemporaryBuffer.commit() (method was commented) and RecordBuffer.rollback() (method was mostly commented)
Note t... Alexandru Lungu

07:35 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

Alexandru Lungu wrote:
> *Please refactor the code in #7535-1 and identify where 2004 recid is reclaimed if there is... Dănuț Filimon

09:15 AM Bug #7567 (WIP): use Session.PK instead of DatabaseManager.PRIMARY_KEY wherever possible

Alexandru Lungu

07/27/2023

03:59 PM Feature #4722: data import should be able to run with only the converted application jar file (an...

Ovidiu, were you able to duplicate my issue at all? I still cannot get the script to work.
I have a very quick pro... Roger Borrello

09:16 AM Bug #7488: Slow fast-copy with before tables in H2

Do we really need the table names to be different at the database level? Why not reuse the same table and DMOs? It ... Greg Shah

09:03 AM Bug #7488: Slow fast-copy with before tables in H2

Greg Shah wrote:
> Perhaps they don't repeat because each time through a new dynamic tt is created (but it is really... Alexandru Lungu

08:21 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

Danut, I don't think the changes are quite right. @autoCommit@ means:
* "yes": there is no active transaction (@do t... Alexandru Lungu

07/26/2023

10:10 AM Feature #7382: Check performance of delete from vs drop table in H2

There are cases where the temp-table is not really multiplexed across the FWD session - only one 'instance' of this t... Constantin Asofiei

09:28 AM Bug #7454: Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

*Committed 7454a_h2/rev.30 and testcases/rev.2383*.
* Character literals were modified to be @ValueStringIgnoreCase... Dănuț Filimon

08:41 AM Bug #7454: Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

Alexandru Lungu wrote:
> My point was to do a bit of refactoring on Functions to allow both IGNORECASE and CASESENSI... Dănuț Filimon

04:02 AM Bug #7454: Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

My point was to do a bit of refactoring on Functions to allow both IGNORECASE and CASESENSITIVE as parameters for suc... Alexandru Lungu

03:44 AM Bug #7454: Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

Alexandru Lungu wrote:
> Danut, the following statements contradict:
> * @the UPPER function that returns Value.STR... Dănuț Filimon

03:19 AM Bug #7454: Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

Danut, the following statements contradict:
* @the UPPER function that returns Value.STRING_IGNORECASE;@: this theor... Alexandru Lungu

09:11 AM Bug #7488: Slow fast-copy with before tables in H2

I fixed the line longer than 110 characters and removed the invalidation block. I changed updatePeerRecords so now it... Ștefan Roman

08:10 AM Bug #7488: Slow fast-copy with before tables in H2

> the most consuming ones are between static tables and dynamic tables - and these don't quite repeat.
It is a com... Greg Shah

07:53 AM Bug #7488: Slow fast-copy with before tables in H2

Committed 7404a/rev. 14653. This resets the loose-copy flag if the loose property matching is the same as the same as... Alexandru Lungu

05:21 AM Bug #7488: Slow fast-copy with before tables in H2

Alexandru Lungu wrote:
> I will check next #7488-7: are there any copy operations that repeat (have the same source-... Alexandru Lungu

04:29 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

Alexandru Lungu wrote:
> Lets make baby steps here. We need the recid reclaim be sound first. At that point, we shal... Dănuț Filimon

04:12 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

Lets make baby steps here. We need the recid reclaim be sound first. At that point, we shall see how we can continue ... Alexandru Lungu

03:29 AM Bug #7421: Check only the indexes that were changed when using Validation.checkMaxIndexSize

I will redo the testing with *7421a/rev.14653* and @max-index-size@ set on 0. This will reduce the number of indexes ... Alexandru Lungu

07/25/2023

04:19 PM Feature #4722: data import should be able to run with only the converted application jar file (an...

The @registry.xml@ is not used with import. It may be opened as resource in common configuration, but its content is ... Ovidiu Maxiniuc

07:58 AM Feature #4722: data import should be able to run with only the converted application jar file (an...

Ovidiu Maxiniuc wrote:
> Roger, I used your jar and attempted to import a single table from your main database. It w... Roger Borrello

10:31 PM Feature #4722: data import should be able to run with only the converted application jar file (an...

Roger, I used your jar and attempted to import a single table from your main database. It works. The jar is good, pro... Ovidiu Maxiniuc

09:26 PM Feature #4722: data import should be able to run with only the converted application jar file (an...

Roger, could you put your @app.jar@ in a shared folder on devsrv01. I would like to attempt to debug against it, to u... Ovidiu Maxiniuc

09:33 AM Bug #7454: Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

Found an interesting example when experimenting with functions in H2.
Here is a simple table definition: @create t... Dănuț Filimon

04:37 AM Bug #7454: Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

Dănuț Filimon wrote:
> After testing a customer application, I noticed that there are instances of internal conversi... Alexandru Lungu

04:23 AM Bug #7454: Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

I looked into #7454-16, the change to the @Parser@ works properly since it has access to the database settings.
Af... Dănuț Filimon

06:43 AM Bug #7488: Slow fast-copy with before tables in H2

Last minute notice: @updatePeerRecords@ doesn't use prepared statements, but generates plain SQL statements. *Stefan*... Alexandru Lungu

06:37 AM Bug #7488: Slow fast-copy with before tables in H2

|_. Simple copy |_. Append |_. Loose-copy |_. Extents |_. Before |_. Time |_. Count |_. Avg |_. Master Copy |_. Exten... Alexandru Lungu

04:15 AM Bug #7488: Slow fast-copy with before tables in H2

*Review of 7404a, revision 14652.*
* The lines you changed now exceed 110 characters. Consider extracting the last... Alexandru Lungu

07/24/2023

10:00 AM Bug #7454: Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

Profiling shows -0.3% improvement. this is decent, but can be way better. I expect way more improvement after we get ... Alexandru Lungu

08:51 AM Bug #7454 (WIP): Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

Great!
I've added in rev. 29 the possibility to make sequences temporary. I just noticed that there are several at... Alexandru Lungu

07:28 AM Bug #7454: Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

Alexandru Lungu wrote:
> I found a regression while testing. I attempted a fix in 7454a_h2 / rev. 27. This has a wid... Dănuț Filimon

05:53 AM Bug #7454: Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

Also, Danut, please make the flag naming consistent with the option. Right now, the option is @DEFAULT_IGNORE_CASE@ a... Alexandru Lungu

05:39 AM Bug #7454: Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

I found a regression while testing. I attempted a fix in 7454a_h2 / rev. 27. This has a wide impact on direct-access,... Alexandru Lungu

09:21 AM Feature #7061 (Closed): Enable the use of lazy result sets in H2

Greg Shah

03:21 AM Feature #7061: Enable the use of lazy result sets in H2

This can be closed. Alexandru Lungu

09:06 AM Bug #7535: Temporary tables require trailing recid component even for unique indexes

After investigating how the recid is reclaimed for temporary tables, I made a few changes to allow reclaiming only wh... Dănuț Filimon

06:52 AM Feature #6813: analyze and improve performance of dynamic conversion of temp-tables and queries

Eric Faulhaber wrote:
> What is left to do on this task? Can we update the % done to a value that reflects the true ... Constantin Asofiei

04:55 AM Feature #7323: Implement soft unique index in FWD-H2

I think the numbers are quite relevant, so I will write them here:
|_. JMX |_. Before |_. After |
| Total | 29.91... Alexandru Lungu

03:24 AM Feature #7504 (WIP): Export data tool from FWD into .d files

Alexandru Lungu

07/22/2023

08:09 AM Feature #4722: data import should be able to run with only the converted application jar file (an...

Ovidiu Maxiniuc wrote:
> To remove this, please add @data/standard.dict@ to your application jar. It is needed by dy... Roger Borrello