FWD » Core Development » Database

We have entered a new phase of H2 performance improvement work, and there is already a set of changes we have been managing for H2. Since we are stuck for the time being on the deprecated PageStore engine (for performance reasons), and it is probable that many of our performance improvements will be implemented in that area of the code, it is unlikely many of our changes will be accepted back into the main project. It is time to get our fork of the H2 code base under internal version control.

Let's start with rev 6 as our initial trunk, put revs 7 and 8 in their own discrete branches, and plan to merge the rev 8 branch to trunk after some wider testing. We will need a wiki page with instructions specific to the build of our fork.

Should we use the same branch naming convention (task number + ascending letters) as we use in FWD? It probably makes sense to add something to the branch names, so that that there is a clear distinction between H2 branches and FWD branches. This will avoid confusion in discussions.

Should we use the same branch naming convention (task number + ascending letters) as we use in FWD? It probably makes sense to add something to the branch names, so that that there is a clear distinction between H2 branches and FWD branches. This will avoid confusion in discussions.

We definitely want to have the task-specific approach (task number + ascending letters). I agree, adding something to make the task branch name associated with h2 does make sense as well. Let's use <task_number><letter>_h2. This will be unambiguously H2 related while also linking back to te task in which the work is being done. We will assume a task branch without a project suffix is a FWD branch.

We'll need changes to create_task_branch.sh and archive_task_branch.sh on devsrv01 and the H2 repositories will need to be structured the same way that we use for FWD.

Greg Shah wrote:

Should we use the same branch naming convention (task number + ascending letters) as we use in FWD? It probably makes sense to add something to the branch names, so that that there is a clear distinction between H2 branches and FWD branches. This will avoid confusion in discussions.

We definitely want to have the task-specific approach (task number + ascending letters). I agree, adding something to make the task branch name associated with h2 does make sense as well. Let's use <task_number><letter>_h2. This will be unambiguously H2 related while also linking back to te task in which the work is being done. We will assume a task branch without a project suffix is a FWD branch.

We'll need changes to create_task_branch.sh and archive_task_branch.sh on devsrv01 and the H2 repositories will need to be structured the same way that we use for FWD.

I started to do the changes for the H2 project.

create_task_branch.sh and archive_task_branch.sh already have an argument named [project_path] which can be assigned to p2j_repo/fwd-h2 to make it work for our newly versioned H2. I find the branching of P2J the common case comparing to the H2 branching. Thus, we can explicitly state the project path when branching H2 and leave the default to P2J.

• create separate scripts for H2 (i.e. create_h2_task_branch.sh)
• add mandatory first parameter to the script (p2j or h2). However, this way we enforce the user to state p2j for each p2j branch.
• add an optional parameter to the script (p2j or h2). However, this would interfere with the [project_path] optional parameter. One is forced to state p2j of h2 even if the project path is custom.

I will not change the scripts for the moment and use the existing [project_path] to finish the version control of H2.

I'm glad the existing scripts worked.

Greg, I've done only one documentation change to Patching and Building Customized 3rd Party Libraries (H2 section), mentioning that we use a custom fork of H2 internally. Keeping patches for each "latest" trunk revision of H2 doesn't feel feasible. Therefore, the distribution of our FWD-H2 project sources should be considered and documented there.

I intend to do a documentation for the branching system of FWD-H2, but most of the source management reference is (impressively detailed) in Source Code Management. Should I refer to other wiki page or create a new one and refer to our existing "Source Code Management"?

See if you can just do some limited edits to Source Code Management to explain branching for FWD vs H2.

Greg Shah wrote:

See if you can just do some limited edits to Source Code Management to explain branching for FWD vs H2.

On it!

I am planning to merge 7046_h2 to the FWD-H2 trunk after handling #7085-3 (test suite segregation). There are still some issues with regression testing and PageStore, so I will keep you updated here on the progress.

Once we reach a consistent testing suite in FWD-H2 trunk, I will test and integrate in the FWD-H2 trunk: 6995a_h2 (H2 NO-UNDO changes), 6928a_h2 (UPDATE improvements) and 6829a_h2 (avoid re-parsing all prepared statements). 6995a_h2 and 6928a_h2 were tested with large customer applications and show a real performance improvement. I am going to test and profile 6829a_h2 as well before merging.

If we have all these in trunk, we can do a new FWD-H2 build for FWD. Further, 6995a_h2 is considered only if we patch FWD with #6995 (remove SQL Redo), so we will to integrate #6995 after this whole process.

My work on #7062 also has H2 changes and I will try to integrate them with this effort.

Alexandru Lungu wrote:

I am planning to merge 7046_h2 to the FWD-H2 trunk after handling #7085-3 (test suite segregation). There are still some issues with regression testing and PageStore, so I will keep you updated here on the progress.

• Integrated #7046 fix
• Integrated #7085 fix
• I've done only safe changes to the test-suite to allow the FWD-H2 trunk to pass them successfully (removed very specific edge cases and refactored some EXPLAIN tests to honor PageStore). Regression testing of H2 1.4.200 covered only a few features over PageStore. I added ./build.sh testMemory, ./build.sh testPersistent, ./build.sh testServer and ./build.sh testFWD to match our use-cases. These are documented in ./build.sh helper. It takes ~7min to run all testFWD.

UPDATE: regression testing passed; merged 6995a_h2 to FWD-H2 trunk including the addition of a NO-UNDO keyword being natively supported in H2.

UPDATE: regression testing passed; merged 6928a_h2 to FWD-H2 trunk including performance improvements for UPDATE statement, with in-memory and PageStore databases.

Will pre-built FWD-H2 revisions jars be uploaded into GCD repository ?

UPDATE: regression testing passed; merged 6829a_h2 to FWD-H2 trunk including performance improvements regarding less recompiling of prepared statements due to meta modifications.

Will pre-built FWD-H2 revisions jars be uploaded into GCD repository ?

Yes. Once we have the pending branches merged into the FWD-H2 trunk, we will have a new build of FWD-H2 available through the GCD repository. At least this is what I expect; Greg makes the call.

For H2, Eric will push this when it is ready.

Alexandru Lungu wrote:

Once we have the pending branches merged into the FWD-H2 trunk, we will have a new build of FWD-H2 available through the GCD repository.

Alexandru, if there is any change to the build process, beyond just checking out the trunk FWD-H2 code and building with the current instructions, please update the 3rd party library wiki accordingly. Thanks.

Eric Faulhaber wrote:

Alexandru, if there is any change to the build process, beyond just checking out the trunk FWD-H2 code and building with the current instructions, please update the 3rd party library wiki accordingly. Thanks.

There is no change in the build process. I changed 3rd party library wiki to exclusively refer to our FWD-H2 approach.

UPDATE: regression testing passed; merged 7062a_h2 to FWD-H2 trunk including support for direct access to internal H2 structures.

At this point, all the tested and profiled branches were merged into trunk. Eric, you can go ahead by building and uploading the new FWD-H2 version (from trunk/rev. 11).
7061a_h2 still needs intensive testing and optimization as it doesn't show much improvement in performance testing. Therefore, I don't suggest integrating it in the next FWD-H2 build.