FWD » Core Development » Database

01/19/2024

06:54 PM Feature #4722: data import should be able to run with only the converted application jar file (an...

The Docker implementation is the most stringent user of this data import feature, so while this is partially docker r... Roger Borrello

06:54 PM Feature #4722: data import should be able to run with only the converted application jar file (an...

Greg Shah wrote:
> Please test in a non-Docker environment.
It works in a normal environment. Roger Borrello

05:46 PM Feature #4722: data import should be able to run with only the converted application jar file (an...

Please test in a non-Docker environment. Greg Shah

05:36 PM Feature #4722: data import should be able to run with only the converted application jar file (an...

My testing is complete. There is now a multi-container configuration (server/client and postgres) for hotel_gui. Ther... Roger Borrello

04:52 PM Feature #4722: data import should be able to run with only the converted application jar file (an...

Greg Shah wrote:
> What testing is needed?
I've built some docker images using the branch, and I'm working on the... Roger Borrello

04:13 PM Feature #4722 (Internal Test): data import should be able to run with only the converted applicat...

Code Review Task Branch 4722c Revision 14935
The changes seem fine. What testing is needed? Greg Shah

03:50 PM Feature #4722 (Review): data import should be able to run with only the converted application jar...

Revision 14935 is ready for review. This holds up the ability to use a multi-container environment. Roger Borrello

03:41 PM Feature #4722 (WIP): data import should be able to run with only the converted application jar fi...

I created a new branch *4722c* due to an issue when attempting to run @import.sh@ in a container. The *createdb* is n... Roger Borrello

10:49 AM Support #4701 (Closed): try to improve H2 transaction commit performance

Eric Faulhaber

03:22 AM Support #4701: try to improve H2 transaction commit performance

This can be closed. Alexandru Lungu

01:23 AM Support #4701: try to improve H2 transaction commit performance

Is there anything left to do on this task, or can we consider the goals met? Eric Faulhaber

10:10 AM Feature #6628: port PostgreSQL native user defined functions to MariaDB

Thank you! Please rebase 6628a or port over the latest trunk, whichever is easier. We will need to regression test th... Eric Faulhaber

05:26 AM Feature #6628: port PostgreSQL native user defined functions to MariaDB

Added the changes from *6628a/rev.14614-14615* on top of *7156b/rev.14933*. I ran the regression tests and got betwee... Dănuț Filimon

02:07 AM Feature #6628: port PostgreSQL native user defined functions to MariaDB

Alexandru Lungu wrote:
> Danut already has the large application set-up with MariaDB lenient and can do the regressi... Dănuț Filimon

02:04 AM Feature #6628: port PostgreSQL native user defined functions to MariaDB

There are runtime changes and SQL UDF changes in this branch. Since the UDFs have changed, they need to be (re-)insta... Eric Faulhaber

02:00 AM Feature #6628: port PostgreSQL native user defined functions to MariaDB

Sure!
Danut already has the large application set-up with MariaDB lenient and can do the regression tests. Priorit... Alexandru Lungu

01:54 AM Feature #6628: port PostgreSQL native user defined functions to MariaDB

Alexandru, can you please assign this to someone on your team who has set up (or can) the large application environme... Eric Faulhaber

09:53 AM Feature #6496: add equivalent support for -trig command line option

I made a copy of 7156b and put my changes there, merged the changes to @SourceNameMapper@ that interfered with my cha... Alexandru Donica

02:11 AM Feature #6496 (Internal Test): add equivalent support for -trig command line option

Alexandru, please test the large application with this parameter you introduced:
* check there are no regressions wi... Alexandru Lungu

01:13 AM Feature #6496: add equivalent support for -trig command line option

Alexandru [either one ;)]: what is left to do to push this task over the finish line? Eric Faulhaber

09:02 AM Bug #7252: Use direct-access to support for-each queries without WHERE and BY clauses

I've created branches 7252b and 7252b_h2 for the experiment with the @SelectAll@ class. Radu Apetrii

07:58 AM Bug #6829: H2 forces re-parse of all prepared statements when metadata is changed

These are great news. Please do extensive testing: focus on smoke testing another customer application and regression... Alexandru Lungu

07:51 AM Bug #6829: H2 forces re-parse of all prepared statements when metadata is changed

Alexandru Lungu wrote:
> Danut, please analyze performance with NEVER_RECOMPILE. I don't recall if we have something... Dănuț Filimon

02:14 AM Bug #6829 (Internal Test): H2 forces re-parse of all prepared statements when metadata is changed

Recently, we done some changes that prevented us from using a "NEVER_RECOMPILE" kind of option. Back in #6829-1, it w... Alexandru Lungu

01:22 AM Bug #6829: H2 forces re-parse of all prepared statements when metadata is changed

Can this task be closed? Eric Faulhaber

05:15 AM Feature #6816 (Internal Test): improve PreselectQuery.assembleHQL

Alexandru Lungu

05:15 AM Feature #6816: improve PreselectQuery.assembleHQL

I am dumping some statistics with LTW right now. I will let you know if I continue seeing something obvious here. Alexandru Lungu

05:05 AM Feature #6816: improve PreselectQuery.assembleHQL

Give me a bit of time to check @PreselectQuery.assembleFQL@, maybe I can notice something else. I'm not sure if Alex ... Radu Apetrii

01:37 AM Feature #6816: improve PreselectQuery.assembleHQL

The changes implemented for this task have long since made their way into trunk. It is still marked 40% done. Is ther... Eric Faulhaber

03:21 AM Feature #6720: lazy hydration

On the POC, it regresses and crashes. I didn't attempt the tests on the new test suite. Alexandru Lungu

01:29 AM Feature #6720: lazy hydration

Is this far enough along that we can measure performance, or are the regressions blocking testing? We need to underst... Eric Faulhaber

02:43 AM Bug #6887: improve performance of dataset/table[-handle] parameters

Constantin, does it makes sense to do one more pass with JMX to see what the 3 cases which stood out earlier look lik... Eric Faulhaber

02:06 AM Feature #6414 (Closed): ERROR attribute for buffer, dataset and temp-table

Eric Faulhaber

01:43 AM Bug #7138 (Closed): Short circuit fake-update (update with a new value eqaul to the old value) in H2

I am closing this. Let me know if there is a reason it should remain open. We have done our internal testing and ther... Eric Faulhaber

01:26 AM Feature #6928 (Closed): H2 UPDATE performance improvement

I am closing this. I think 11 months of testing is probably sufficient :) Eric Faulhaber

01/18/2024

09:32 AM Feature #7382: Check performance of delete from vs drop table in H2

1. @8591 ms@ for @939_203@ deletes (classic execution, all deletes)
5. @7672 ms@ for @935_965@ deletes (optimized ex... Alexandru Donica

04:05 AM Feature #7382: Check performance of delete from vs drop table in H2

well 5 = 4 + 3. and 1 = 2 + 3.
(don't read this out of context) Alexandru Donica

03:40 AM Feature #7382: Check performance of delete from vs drop table in H2

Lets presume that A is the set of *all* deletes and C is the set of "candidate" deletes (the ones that *can* be optim... Alexandru Lungu

03:25 AM Feature #7382: Check performance of delete from vs drop table in H2

Alexandru Lungu wrote:
> * Total and Avg. time for classic deletes
> * Total and Avg. time for fast deletes
> * To... Alexandru Donica

01/17/2024

11:43 AM Feature #7382: Check performance of delete from vs drop table in H2

I really think this ratio will go up once we use expanded extents. There are many temp tables with extent fields. As ... Alexandru Lungu

09:59 AM Feature #7382: Check performance of delete from vs drop table in H2

I modified the code, now I only check for referential constraints, as they are the ones causing problems (Some tables... Alexandru Donica

09:34 AM Feature #7382: Check performance of delete from vs drop table in H2

As far as I know, the only cascading deletes that we would encounter would be for the non-expanded extents (which hav... Greg Shah

09:30 AM Feature #7382: Check performance of delete from vs drop table in H2

After some debugging, I found an error that I didn't know was thrown, which was happening because, when the classic d... Alexandru Donica

09:45 AM Bug #7496: finish support for query:forward-only attribute

I managed to solve all the regressions from the adaptive-scrolling test set and identify other problems in the POC wh... Dănuț Filimon

01/16/2024

10:13 AM Feature #7382: Check performance of delete from vs drop table in H2

This is quite strange - there are many "one time" temp-tables used in the large POC (or this is what I recall). Pleas... Alexandru Lungu

09:43 AM Feature #7382: Check performance of delete from vs drop table in H2

Scratch that, there was a little bug, but one that would show a weird error related to db because it was happening in... Alexandru Donica

08:26 AM Bug #7138 (Test): Short circuit fake-update (update with a new value eqaul to the old value) in H2

*Committed 7138a_h2 to FWD_H2 trunk as rev.39. Archived 7138a_h2.* Dănuț Filimon

07:54 AM Bug #8200: Explicit buffer used instead of the implicit one for a persistent table

Assuming we have a persistent table called @t1@ with only 1 field, @f1@, which is an integer.
For the following te... Andrei Bălteanu

07:53 AM Bug #8200 (Test): Explicit buffer used instead of the implicit one for a persistent table

Andrei Bălteanu

05:59 AM Bug #7448: Optimize FWD-H2 ValueTimestampTimeZone and maybe avoid caching

The improvement from removing the caching for this method is very small, 5 milliseconds faster, so a total of ~241 mi... Alexandru Donica

04:46 AM Bug #7252: Use direct-access to support for-each queries without WHERE and BY clauses

There are a lot of H2 queries that were replaced with direct-access. Is not that the "select all records" increased, ... Alexandru Lungu

01/15/2024

10:17 AM Feature #7382: Check performance of delete from vs drop table in H2

There is a bug/regression somewhere regarding the patch I posted above. When I try to smoke test on another large app... Alexandru Donica

09:28 AM Bug #8196: Reduce number of AbstractTempTable._hasRecords calls to avoid BufferManager.activeBuffers

I wasn't aware this could be called directly from the converted code. Alexandru Lungu

09:24 AM Bug #8196: Reduce number of AbstractTempTable._hasRecords calls to avoid BufferManager.activeBuffers

Alexandru Lungu wrote:
> This can be improved in two ways:
> * Check if we really need to check the transient recor... Dănuț Filimon

09:04 AM Bug #8196 (WIP): Reduce number of AbstractTempTable._hasRecords calls to avoid BufferManager.acti...

Dănuț Filimon

09:03 AM Bug #8196: Reduce number of AbstractTempTable._hasRecords calls to avoid BufferManager.activeBuffers

The POC application calls @AbstractTempTable.hasRecords@ and @clear@/@setCanUndo@ are not called at all. I noticed th... Dănuț Filimon

08:38 AM Bug #8196: Reduce number of AbstractTempTable._hasRecords calls to avoid BufferManager.activeBuffers

Danut, the code using @allBuffers@ is meant to check records that were not yet flushed to the database (transient) an... Alexandru Lungu

08:04 AM Bug #8196: Reduce number of AbstractTempTable._hasRecords calls to avoid BufferManager.activeBuffers

After a fix involving going over the buffers from @AbstractTempTable.allBuffers@ and execute the same logic from @act... Dănuț Filimon

04:25 AM Bug #8196: Reduce number of AbstractTempTable._hasRecords calls to avoid BufferManager.activeBuffers

While retesting different session reclaiming lifespans for #7167, it was noted that there is almost no difference bet... Dănuț Filimon

04:18 AM Bug #8196 (Closed): Reduce number of AbstractTempTable._hasRecords calls to avoid BufferManager.a...

Dănuț Filimon

09:27 AM Bug #7138: Short circuit fake-update (update with a new value eqaul to the old value) in H2

Danut, please go ahead and merge to FWD-H2 trunk.
Greg, this will reach FWD on the next H2 upgrade. Alexandru Lungu

08:54 AM Bug #7138: Short circuit fake-update (update with a new value eqaul to the old value) in H2

Alexandru: If you are OK with this, it can merge now. Greg Shah

08:48 AM Bug #7138: Short circuit fake-update (update with a new value eqaul to the old value) in H2

I retested POC on the original device and obtained a performance improvement for both the average of the last 20 runs... Dănuț Filimon

02:29 AM Bug #7138: Short circuit fake-update (update with a new value eqaul to the old value) in H2

Danut, this is a common pattern: in different days and on different platforms to see different results. The only way ... Alexandru Lungu

08:50 AM Bug #7448: Optimize FWD-H2 ValueTimestampTimeZone and maybe avoid caching

I was thinking of more than just avoiding the cache and maybe remote it entirely :) However, the timing is not that c... Alexandru Lungu

06:42 AM Bug #7448: Optimize FWD-H2 ValueTimestampTimeZone and maybe avoid caching

The average of 3 baseline runs for me was 14.665 seconds (so a total of 1_466_500 milliseconds for 100 iterations), a... Alexandru Donica

08:37 AM Bug #7252: Use direct-access to support for-each queries without WHERE and BY clauses

Alexandru Lungu wrote:
> First and foremost:
> See how many such queries are on POC. I just want to confirm my init... Radu Apetrii

03:56 AM Bug #7167: Associating records from opened buffers to new sessions is slow

I've got the results of how often is @activeBuffers@ hit from @AbstractTempTable._hasRecords@, and how often it is hi... Dănuț Filimon

01/12/2024

09:49 AM Bug #7138: Short circuit fake-update (update with a new value eqaul to the old value) in H2

I tested POC with 7156b/rev.14933 (which uses FWD-H2/rev.38) and 7138a_h2/rev.45. From 5 runs each (baseline and 7156... Dănuț Filimon

08:39 AM Bug #7448: Optimize FWD-H2 ValueTimestampTimeZone and maybe avoid caching

Alexandru, please keep the same profiling infrastructure you have and test with 7156b and the large POC. Please make ... Alexandru Lungu

08:13 AM Bug #7448: Optimize FWD-H2 ValueTimestampTimeZone and maybe avoid caching

I changed the @CurrentTimestamp.get@ method to keep track of the metrics you mentioned, and roamed around the menus o... Alexandru Donica

05:29 AM Feature #6815 (Review): configure all cache sizes in the directory, and create documentation for ...

*Committed 6815a/rev.14919*. Reduced the number of instrumented methods (based on results from a large application, t... Dănuț Filimon

04:04 AM Feature #6815: configure all cache sizes in the directory, and create documentation for them

Danut, please merge some of the statistics in one: get, getOrDefault can be merged. The same as put, putAll and putIf... Alexandru Lungu

04:01 AM Feature #6815: configure all cache sizes in the directory, and create documentation for them

Alexandru Lungu wrote:
> Why is it too big? Shouldn't it be similar to #6815-11, but with some extra columns?
> Any... Dănuț Filimon

03:50 AM Feature #6815: configure all cache sizes in the directory, and create documentation for them

Dănuț Filimon wrote:
> I added monitoring for the configured and the in use cache size (latest size of the cache), G... Alexandru Lungu

03:15 AM Feature #6815: configure all cache sizes in the directory, and create documentation for them

I added monitoring for the configured and the in use cache size (latest size of the cache), GET hit/miss and modified... Dănuț Filimon

01/11/2024

06:11 PM Bug #8183: debugging database module

Code review 8183a/14914:
The changes look useful, but there is a security concern, in that these public methods ca... Eric Faulhaber

05:51 AM Bug #7667: Support OUTER-JOIN with PRESELECT at conversion

Branch 7667b was merged to trunk revision 14916 and archived. Alexandru Lungu

05:48 AM Feature #6815 (WIP): configure all cache sizes in the directory, and create documentation for them

Alexandru Lungu wrote:
> What other statistics can be achieved? I suspect the ratio is the most important one, so le... Dănuț Filimon

05:39 AM Feature #6815: configure all cache sizes in the directory, and create documentation for them

What other statistics can be achieved? I suspect the ratio is the most important one, so lets get it done. Also, we n... Alexandru Lungu

05:28 AM Feature #6815: configure all cache sizes in the directory, and create documentation for them

*Committed 6815a/rev.14917*. Removed unnecessary methods and changes the container accessed for the @instrumentation@... Dănuț Filimon

03:52 AM Bug #7138: Short circuit fake-update (update with a new value eqaul to the old value) in H2

There are currently problems with the POC project and the performance tests will be put on *hold* until further notic... Dănuț Filimon

01/10/2024

07:12 AM Feature #6815 (Review): configure all cache sizes in the directory, and create documentation for ...

Created *6815a* and committed *rev.14916*. As suggested by Alexandru in #6815-9, I've created a way to instrument cac... Dănuț Filimon

01/09/2024

08:16 AM Bug #7454 (Closed): Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

Greg Shah

08:04 AM Bug #7454: Make ValueStringIgnoreCase the default generated value for setString in FWD-H2

This can be closed. Alexandru Lungu

08:16 AM Feature #7404 (Closed): Trasform replace-mode into append-mode when target table is empty

Greg Shah

08:04 AM Feature #7404: Trasform replace-mode into append-mode when target table is empty

This can be closed. Alexandru Lungu

08:16 AM Bug #7389 (Closed): Make fast-copy statements cachable

Greg Shah

08:03 AM Bug #7389: Make fast-copy statements cachable

This can be closed. Alexandru Lungu

08:03 AM Bug #7252: Use direct-access to support for-each queries without WHERE and BY clauses

Radu, lets see how we can "resurrect" this task. I think the initial statistic of *lots* of @SELECT@ statements that ... Alexandru Lungu

07:37 AM Bug #8183: debugging database module

This task should include an easy way to see a temp-table or permanent table's contents without having to do complex m... Greg Shah

05:06 AM Feature #7382: Check performance of delete from vs drop table in H2

Alexandru [ad], please refer to [[internal-documentation:H2_Database_Fork]] for further insights. Alexandru Lungu

04:38 AM Feature #7382: Check performance of delete from vs drop table in H2

> This is a patch of what I wrote so far. So from what I understand, instead of having a delete from tt where _multip... Alexandru Lungu

04:24 AM Feature #7382: Check performance of delete from vs drop table in H2

This is a patch of what I wrote so far. So from what I understand, instead of having a @delete from tt where _multipl... Alexandru Donica

04:20 AM Bug #7138: Short circuit fake-update (update with a new value eqaul to the old value) in H2

An update on the current test status:
* Rebased *7138a_h2* with *fwd-h2/rev.38*. *7138a_h2* is now at revision *45*;... Dănuț Filimon

01/08/2024

05:55 PM Bug #8183 (WIP): debugging database module

I created task branch 8183a and committed some of my code I have maintained for some time (r14914). It was designed n... Ovidiu Maxiniuc

04:57 PM Bug #8183: debugging database module

This task is the place for discussions on how to do database debugging and develop tools for doing this optimally.
Ovidiu Maxiniuc

04:53 PM Bug #8183 (WIP): debugging database module

Ovidiu Maxiniuc

09:13 AM Bug #7138: Short circuit fake-update (update with a new value eqaul to the old value) in H2

Alexandru Lungu wrote:
> Please let the application convert over night. After, we can go ahead with proper testing.
... Dănuț Filimon

09:10 AM Bug #7138: Short circuit fake-update (update with a new value eqaul to the old value) in H2

Please let the application convert over night. After, we can go ahead with proper testing. Alexandru Lungu

08:45 AM Bug #7138: Short circuit fake-update (update with a new value eqaul to the old value) in H2

I updated the sources for the customer applications but there are some regressions/files missing which I'll need to l... Dănuț Filimon

01/05/2024

09:14 AM Bug #7776 (Closed): Previous regression on FWD-H2 lazy mode

Greg Shah

04:07 AM Bug #7776 (Test): Previous regression on FWD-H2 lazy mode

This can be closed. Alexandru Lungu

09:14 AM Bug #8032 (Closed): Deleted records are returned by a preselect query

Greg Shah

04:06 AM Bug #8032: Deleted records are returned by a preselect query

5768a was merged to trunk. This task can be Closed. Alexandru Lungu

05:12 AM Feature #6720: lazy hydration

Eduard, please assist the regression hunting here. I am aware that you have a more suitable environment for regressio... Alexandru Lungu

05:08 AM Feature #7382: Check performance of delete from vs drop table in H2

Alexandru [ad], please provide a prototype for this task asap. Alexandru Lungu

04:04 AM Feature #7416 (Test): Reiterate DatabaseStatistics and leverage to other database profling tools

Alexandru Lungu

04:01 AM Bug #7138 (Internal Test): Short circuit fake-update (update with a new value eqaul to the old va...

Greg Shah wrote:
> What is the status of 7138a_h2? Does it need more review? More testing? It seems to provide a ... Alexandru Lungu

03:56 AM Bug #7167: Associating records from opened buffers to new sessions is slow

Danut, I don't think that the shift to 10s looks OK with these statistics. Mind that we "lock" a connection from the ... Alexandru Lungu

03:48 AM Feature #6825 (WIP): improve table meta runtime (TableMapper and other)

Alexandru [ad], please attempt to continue profiling and optimization of @TableMapper@. The recent changes shifted a ... Alexandru Lungu

12/25/2023

07:04 AM Bug #6767: Possible name clashes in DMO classes

The work redone in the branch 6767b, The branch is rebased to trunk rev. 14906 the branch revision is 14917.
Please,... Vladimir Tsichevski

12/22/2023

03:01 PM Bug #6767: Possible name clashes in DMO classes

The following methods should be also renamed in addition to the methods listed in #6767-9:... Vladimir Tsichevski

10:42 AM Bug #7138: Short circuit fake-update (update with a new value eqaul to the old value) in H2

What is the status of 7138a_h2? Does it need more review? More testing? It seems to provide a decent improvement o... Greg Shah

12:31 AM Bug #7496 (WIP): finish support for query:forward-only attribute

Since 7496b was merged, I'll put this issue into WIP. I've managed to find a few other issues with the implementation... Dănuț Filimon

12/21/2023

12:39 PM Feature #6442: missing encoding/collation features

The @DBCODEPAGE()@ and @DBCOLLATION()@ built-in function were implemented in #7326 and available in trunk starting wi... Ovidiu Maxiniuc

12:05 PM Feature #4722 (Test): data import should be able to run with only the converted application jar f...

Branch 4722b was merged into trunk revision 14895 and archived. Ovidiu Maxiniuc

11:24 AM Feature #4722 (Merge Pending): data import should be able to run with only the converted applicat...

Please merge 4722b after 8056b. Greg Shah

10:58 AM Feature #4722: data import should be able to run with only the converted application jar file (an...

I rebased the branch to trunk/14890 and fixed the (strange, albeit simple) conflicts.
Pushed up 4722b to revision 14... Ovidiu Maxiniuc

09:11 AM Feature #4722: data import should be able to run with only the converted application jar file (an...

Ovidiu : Please rebase and if the result is working with a smoke test, we will get it merged. Greg Shah

09:03 AM Feature #4722: data import should be able to run with only the converted application jar file (an...

Greg Shah wrote:
> Constantin: Please review the @memptr@ change in 4722b rev 14888. If this was a safe change, why... Constantin Asofiei

08:44 AM Feature #4722: data import should be able to run with only the converted application jar file (an...

Constantin: Please review the @memptr@ change in 4722b rev 14888. If this was a safe change, why didn't we make it b... Greg Shah

08:31 AM Feature #4722: data import should be able to run with only the converted application jar file (an...

Ovidiu, there are a couple of conflicts when attempting to rebase against trunk revision 14890. Could you take a look... Roger Borrello

08:28 AM Feature #4722: data import should be able to run with only the converted application jar file (an...

Eric Faulhaber wrote:
> I think the @import.sh@ changes are ok, but I did not confirm with a test. Has this been tes... Roger Borrello

11:23 AM Feature #8067: investigate adding automatically adding a FIELDS clause for simple record blocks

I fully agree with Eric and mark this as top focus in the implementation. However, I am not 100% sure this approach w... Alexandru Lungu

11:13 AM Feature #8067: investigate adding automatically adding a FIELDS clause for simple record blocks

Please note that an explicit FIELDS clause in 4GL code will raise an error if a field is accessed in downstream code ... Eric Faulhaber

10:50 AM Bug #7496: finish support for query:forward-only attribute

Branch 7496b was merged into trunk revision 14891 and archived.
This task is still open due to 7496a. Alexandru Lungu

10:07 AM Bug #7496: finish support for query:forward-only attribute

Doing it now - just 10 mins to check it out. Alexandru Lungu

07:52 AM Bug #7496: finish support for query:forward-only attribute

You can merge 7496b to trunk now. Greg Shah

04:57 AM Bug #7496: finish support for query:forward-only attribute

Greg, this seems like an easy performance gain. It was fully tested (except ETF). We can go ahead with the merge of 7... Alexandru Lungu

08:42 AM Bug #6767: Possible name clashes in DMO classes

Eric Faulhaber wrote:
> Vladimir, sorry for the long delay in reviewing this again. Please rebase this branch when y... Vladimir Tsichevski