FWD » Core Development » Database

03/12/2021

05:36 PM Feature #1587: implement full support for word indexes

BTW. I've just realized that it is possible to add a 'natural' implicit ordering to the queries using the @CONTAINS@ ... Igor Skornyakov

02:27 PM Feature #1587: implement full support for word indexes

I'm ok with using GUIDs. Eric Faulhaber

02:25 PM Feature #1587: implement full support for word indexes

Greg Shah wrote:
> Do dynamic temp-tables have names?
Yes, they are referenced in subsequent dynamic queries by n... Eric Faulhaber

02:20 PM Feature #1587: implement full support for word indexes

Greg Shah wrote:
> Do dynamic temp-tables have names?
If they are in the H2 database they should have at least so... Igor Skornyakov

02:16 PM Feature #1587: implement full support for word indexes

Do dynamic temp-tables have names? Greg Shah

02:10 PM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> Igor Skornyakov wrote:
> When you say "ensure that data objects created for word tables are... Igor Skornyakov

02:09 PM Feature #1587: implement full support for word indexes

Some google searches suggest '@@@' is a valid character in an SQL table name, but not in a 4GL table name. This will ... Eric Faulhaber

01:59 PM Feature #1587: implement full support for word indexes

Igor Skornyakov wrote:
> Another question. I need to ensure that data objects created for word tables are unique, ev... Eric Faulhaber

01:45 PM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> Yes, the dirty database should work properly with extent fields, if we go with the UDF appro... Igor Skornyakov

01:42 PM Feature #1587: implement full support for word indexes

Igor Skornyakov wrote:
> So I have to add a check for @isDirty@ and do not re-write the query for the 'dirty' data... Eric Faulhaber

01:32 PM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> Igor Skornyakov wrote:
> > What is 'dirty' database and should I consider such databases fo... Igor Skornyakov

01:22 PM Feature #1587: implement full support for word indexes

Igor Skornyakov wrote:
> What is 'dirty' database and should I consider such databases for the word tables support?
... Eric Faulhaber

05:28 AM Feature #1587: implement full support for word indexes

Another question. I need to ensure that data objects created for word tables are unique, even in the scope of a sessi... Igor Skornyakov

05:19 AM Feature #1587: implement full support for word indexes

Sorry for my ignorance. What is 'dirty' database and should I consider such databases for the word tables support?
T... Igor Skornyakov

04:17 PM Bug #5056 (Closed): Temp-tables and unique indexes

Eric Faulhaber

04:00 PM Bug #5056: Temp-tables and unique indexes

The validation works correctly for temp-tables now. It does not really make sense why OE shows error 132 multiple tim... Ovidiu Maxiniuc

11:39 AM Bug #5176: Buffer field attribute leak

Code review 3821c/12108:
Generally, the changes look good to me, though the change was so dense in @TableMapper@, ... Eric Faulhaber

03/11/2021

02:41 PM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> Just saw this in the H2 mailing list, FWIW. Noel is one of the main contributors to the proj... Igor Skornyakov

02:30 PM Feature #1587: implement full support for word indexes

Just saw this in the H2 mailing list, FWIW. Noel is one of the main contributors to the project. The OP was reporting... Eric Faulhaber

07:49 AM Bug #5121: Converted temp table should be using the legacy name instead of the converted name in ...

Eric Faulhaber wrote:
> Ovidiu, can you think of an efficient way to give access to @SharedVariableManager.lookupTem... Ovidiu Maxiniuc

09:22 PM Bug #5121: Converted temp table should be using the legacy name instead of the converted name in ...

Roger Borrello wrote:
> Greg Shah wrote:
> > I think the legacy name can be found in @expected@. It is the DMO we ... Eric Faulhaber

03/10/2021

04:27 PM Bug #5176 (WIP): Buffer field attribute leak

I committed the fix in 3821c as revision 12108. The update is a aggregate, it contains fixes for other smaller issues... Ovidiu Maxiniuc

04:08 PM Bug #5121: Converted temp table should be using the legacy name instead of the converted name in ...

Eric? Greg Shah

03:51 PM Bug #5121: Converted temp table should be using the legacy name instead of the converted name in ...

Greg Shah wrote:
> I think the legacy name can be found in @expected@. It is the DMO we are searching for, which in... Roger Borrello

01:17 PM Bug #5121: Converted temp table should be using the legacy name instead of the converted name in ...

I think the legacy name can be found in @expected@. It is the DMO we are searching for, which inherently maps 1-to-1... Greg Shah

12:02 PM Bug #5121: Converted temp table should be using the legacy name instead of the converted name in ...

At this point in processing I don't think we know the legacy name (since we haven't found it).
From @SharedVariabl... Roger Borrello

01:44 PM Bug #4173 (Closed): BUFFER-COPY/BUFFER-COMPARE compares table fields by name rather than data typ...

Greg Shah

01:29 PM Bug #4173 (WIP): BUFFER-COPY/BUFFER-COMPARE compares table fields by name rather than data type a...

This can be closed; the premises in #4173 are not accurate as there is a difference in implementation between @bufer-... Adrian Lungu

11:01 AM Bug #4173: BUFFER-COPY/BUFFER-COMPARE compares table fields by name rather than data type and order

Is this still an open issue? Greg Shah

01:42 PM Feature #3958 (Closed): BUFFER-COPY optimization

Greg Shah

01:22 PM Feature #3958: BUFFER-COPY optimization

This can be closed; the BUFFER-COPY implementation reached all the optimizations proposed here. Adrian Lungu

11:00 AM Feature #3958: BUFFER-COPY optimization

What is left to do on this task? Greg Shah

01:26 PM Feature #4011: database/persistence layer performance improvements

We have the idea in #4011-774 implemented as @DmoSignature@. Adrian Lungu

11:01 AM Feature #4011: database/persistence layer performance improvements

Is #4011-774 complete? Greg Shah

01:19 PM Feature #4055: optimize temp-table output parameter copying

#4055-63 keeps track of the things left to do. The opened ones are still some possible directions, which were explore... Adrian Lungu

10:59 AM Feature #4055: optimize temp-table output parameter copying

What is left to do on this task? Greg Shah

11:21 AM Feature #1587: implement full support for word indexes

The final logic for implicit sorting appeared to be more tricky than I've described initially. In particular, a situa... Igor Skornyakov

03/09/2021

05:04 PM Feature #1587: implement full support for word indexes

Ovidiu Maxiniuc wrote:
> These two classes are really specialised:
> * the @DDLGeneratorWorker@ is used by static c... Igor Skornyakov

05:02 PM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> Not as such. The documentation on FWD internals is the least developed of the various books.... Igor Skornyakov

04:25 PM Feature #1587: implement full support for word indexes

These two classes are really specialised:
* the @DDLGeneratorWorker@ is used by static conversion to get DDLs for pe... Ovidiu Maxiniuc

03:39 PM Feature #1587: implement full support for word indexes

Not as such. The documentation on FWD internals is the least developed of the various books.
The code which create... Eric Faulhaber

03:15 PM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> OK, please post any questions you may have about the temp-table creation, DDL, etc. Ovidiu o... Igor Skornyakov

03:07 PM Feature #1587: implement full support for word indexes

OK, please post any questions you may have about the temp-table creation, DDL, etc. Ovidiu or I will be able to get y... Eric Faulhaber

01:57 PM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> Igor Skornyakov wrote:
> Given that support for H2 already is implemented, what is the incr... Igor Skornyakov

01:02 PM Feature #1587: implement full support for word indexes

Igor Skornyakov wrote:
> According to #1587-327, you wanted also to implement word tables for @_temp@ db.
Given... Eric Faulhaber

11:58 AM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> Igor, other than finishing the sort implementation (and gauging its performance) and the 168... Igor Skornyakov

11:50 AM Feature #1587: implement full support for word indexes

Igor, other than finishing the sort implementation (and gauging its performance) and the 1688 conversion warning, wha... Eric Faulhaber

02:48 AM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> W.r.t. the command line options, we don't want to use system properties long term. For any o... Igor Skornyakov

02:44 PM Bug #5176: Buffer field attribute leak

I have really thought of this issue during last days.
The @DmoMeta@ carries immutable data, that is: information ... Ovidiu Maxiniuc

11:53 AM Bug #5176: Buffer field attribute leak

Ovidiu, if every buffer has its own writable version of a certain set of attributes, does it make sense for each buff... Eric Faulhaber

11:47 AM Support #5183: database collation

Some additional research to consider:
*1.* Locales
A "locale":https://en.wikipedia.org/wiki/Locale_(computer_so... Greg Shah

03/08/2021

05:54 PM Feature #1587: implement full support for word indexes

Code review 3821c, rev 12079, 12089:
The logic changes look good to me. I have not tested them.
W.r.t. the comm... Eric Faulhaber

03:35 AM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> Another question/concern: it seems like this approach requires the CTE. It follows that it w... Igor Skornyakov

03:31 AM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> In the absence of a BY clause, I don't know of any guaranteed ordering of the results of a q... Igor Skornyakov

11:03 PM Feature #1587: implement full support for word indexes

Another question/concern: it seems like this approach requires the CTE. It follows that it would disrupt your more pe... Eric Faulhaber

10:46 PM Feature #1587: implement full support for word indexes

This is a very interesting proposal, and I appreciate the creativity in coming up with it.
In the absence of a BY ... Eric Faulhaber

03/07/2021

10:57 AM Feature #1587: implement full support for word indexes

I doubt that it is possible to implement an implicit ordering imposed by @CONTAINS@ which is 100% compatible with 4GL... Igor Skornyakov

10:25 AM Feature #1587: implement full support for word indexes

I've created the following test to analyze the implicit sorting imposed by @CONTAINS@ (see @testcases/words/words-ord... Igor Skornyakov

03/06/2021

02:41 PM Feature #1587: implement full support for word indexes

The problem with ... Igor Skornyakov

08:21 AM Feature #1587: implement full support for word indexes

Added CNF ordering (see #1576-386).
Committed to 3821c/12089 Igor Skornyakov

05:10 AM Feature #1587: implement full support for word indexes

I understand that the order of terms in the @CONTAINS@@ expression is important for implicit sorting. So I'm going to... Igor Skornyakov

11:24 AM Support #5183: database collation

In the "Internationalize ABL Application" docs there are good details about codepages and all the character configura... Greg Shah

03/05/2021

05:17 PM Support #5183: database collation

The attached program (@cp_coll.p@) was written as a starting point to review the 4GL's collation, inspired by the exi... Eric Faulhaber

05:05 PM Support #5183: database collation

A review of how FWD collates its database data has raised some questions as to whether we are doing it correctly for ... Eric Faulhaber

05:00 PM Support #5183 (WIP): database collation

Eric Faulhaber

03/04/2021

08:02 AM Bug #5176: Buffer field attribute leak

My understanding is that in the 4GL every temp-table *instance* is a completely separate table. Multiple table insta... Greg Shah

08:10 PM Bug #5176: Buffer field attribute leak

I will try to summarize an ugly issue I am investigating lately.
I was working on a issue related to #3574-11 and ... Ovidiu Maxiniuc

07:07 PM Bug #5176 (Closed): Buffer field attribute leak

Ovidiu Maxiniuc

06:29 AM Feature #1587: implement full support for word indexes

I've noticed that queries with @CONTAINS@ run faster with CTE when the expression is 'simple' (its CNF contains only ... Igor Skornyakov

03/03/2021

03:46 PM Feature #1587: implement full support for word indexes

Added SQL re-writing using common table expressions (WITH clause).
Committed to 3821c/12077.
For PostgreSQL, it i... Igor Skornyakov

03/02/2021

02:41 AM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> Well, it is extra work to add it in @HQLPreprocessor@ and I imagine there probably is some w... Igor Skornyakov

03/01/2021

05:59 PM Feature #1587: implement full support for word indexes

Well, it is extra work to add it in @HQLPreprocessor@ and I imagine there probably is some work to ignore it when gen... Eric Faulhaber

05:19 PM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> The RTRIM is injected for text property references by the @HQLPreprocessor@. It is needed fo... Igor Skornyakov

05:09 PM Feature #1587: implement full support for word indexes

The RTRIM is injected for text property references by the @HQLPreprocessor@. It is needed for normal text property eq... Eric Faulhaber

04:44 PM Feature #1587: implement full support for word indexes

I've noticed some strange thing.
Consider the following 4GL code:... Igor Skornyakov

02/26/2021

05:42 PM Bug #4994 (WIP): NPE in AdaptiveQuery when loading null data for an outer join

In case of OUTER JOIN tables when there is no matching record, @data[i]@ is @null@. Rewriting the problematic line as... Ovidiu Maxiniuc

02:47 PM Support #4550: eliminate the need to place the p2jspi.jar in the JVM extension directory

It is good to know that the bug is just a configuration issue. At a minimum we need to improve our documentation. F... Greg Shah

02:28 PM Support #4550: eliminate the need to place the p2jspi.jar in the JVM extension directory

Greg Shah wrote:
> If I understand correctly, the @p2jspi.jar @ is hard coded to the equivalent of the 4GL 8859-1.
... Eric Faulhaber

07:53 AM Support #4550: eliminate the need to place the p2jspi.jar in the JVM extension directory

We also need to consider that the effective codepage for H2 temp-tables must be something that can be overridden. In... Greg Shah

07:49 AM Feature #4722: data import should be able to run with only the converted application jar file (an...

> Both of these dependencies could be specified as parameters to a new command line program
I don't understand why... Greg Shah

02:11 AM Bug #5056: Temp-tables and unique indexes

Can this issue be closed? Eric Faulhaber

02/25/2021

03:07 PM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> A separate thought: what if we remove the @distinct@ keyword? Unless I'm reading the plan wr... Igor Skornyakov

03:03 PM Feature #1587: implement full support for word indexes

I'm trying to figure out whether there is a way to express these search conditions without the hash joins (and someti... Eric Faulhaber

02:45 PM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> Thank you.
>
> Are these two statements functionally equivalent?
>
> 1)
>
> [...]
... Igor Skornyakov

02:37 PM Feature #1587: implement full support for word indexes

As I re-read that last post, I'm not sure my where clause makes sense in that second query...rethinking... Eric Faulhaber

02:35 PM Feature #1587: implement full support for word indexes

Thank you.
Are these two statements functionally equivalent?
1)... Eric Faulhaber

02:06 PM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> What is the SQL for the expression @"s0 | s1"@?
>
> For @"(s0 | s1) & s5*"@?
>
> (LE: ... Igor Skornyakov

01:59 PM Feature #1587: implement full support for word indexes

What is the SQL for the expression @"s0 | s1"@?
For @"(s0 | s1) & s5*"@?
(LE: fixed syntax error in original post) Eric Faulhaber

01:53 PM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> Igor, can't the first form of the SQL be simplified as follows?
> In other words, why is th... Igor Skornyakov

01:39 PM Feature #1587: implement full support for word indexes

Igor, can't the first form of the SQL be simplified as follows?
From:... Eric Faulhaber

11:02 AM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> Yes, I was looking at the sort part of the plan, but you are right that the parts below do l... Igor Skornyakov

11:01 AM Feature #1587: implement full support for word indexes

Yes, I was looking at the sort part of the plan, but you are right that the parts below do look somewhat faster. Plea... Eric Faulhaber

10:54 AM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> Igor Skornyakov wrote:
>
> > At this moment I'm looking how to change SQL re-writing usin... Igor Skornyakov

10:43 AM Feature #1587: implement full support for word indexes

Igor Skornyakov wrote:
> At this moment I'm looking how to change SQL re-writing using @WITH@ clause (see #1587-36... Eric Faulhaber

10:30 AM Feature #1587: implement full support for word indexes

I've added two things to the word tables support.
# The re-writing of the SQL queries with @CONTAINS@ re-writing can... Igor Skornyakov

04:27 AM Feature #1587: implement full support for word indexes

With a logically equivalent but different re-written SQL statement using @WITH@ clause:... Igor Skornyakov

03:46 AM Feature #1587: implement full support for word indexes

Adding index on the 'word' field in the word tables helps. The times are (see #1587-357)
|expr |records|e... Igor Skornyakov

03:09 AM Feature #1587: implement full support for word indexes

Word tables' tests to @sftp://xfer.goldencode.com/opt/testcases/@ revision 1008.
Tests (including Java project for g... Igor Skornyakov

02:07 AM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> Igor, is this test code (including the code which generates the test data) checked in?
No... Igor Skornyakov

02:04 AM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> Igor, while the improvement over the UDF approach is considerable in some cases, the new app... Igor Skornyakov

01:21 PM Bug #5056 (Test): Temp-tables and unique indexes

Patch committed as 3821c/12065. Eric Faulhaber

09:25 PM Bug #5056: Temp-tables and unique indexes

Eric Faulhaber wrote:
> Ovidiu, please test whether the attached patch resolves your original issue(s). It gets both... Ovidiu Maxiniuc

09:42 AM Feature #5165: disallow usage of an incompatible database instance

We can also validate the schema specified at the permament DMOs with the database tables/indexes. Constantin Asofiei

05:48 AM Feature #5165: disallow usage of an incompatible database instance

In OE, the database metaschema has CRC values stored for each table, index and schema trigger. When compiling 4GL to... Greg Shah

05:08 AM Feature #5165 (New): disallow usage of an incompatible database instance

Greg Shah

02/24/2021

06:35 PM Feature #1587: implement full support for word indexes

Igor, is this test code (including the code which generates the test data) checked in? Eric Faulhaber

05:22 PM Feature #1587: implement full support for word indexes

Igor, while the improvement over the UDF approach is considerable in some cases, the new approach is still not as per... Eric Faulhaber

02:33 PM Feature #1587: implement full support for word indexes

For the performance testing, I've used a modified approach described at #1587-340. Instead of generated "word" as all... Igor Skornyakov

02/23/2021

06:34 PM Bug #5056: Temp-tables and unique indexes

Similar to the patch proposed in #4884-15, I have fixed the cases identified above. The three main xfer testcases are... Ovidiu Maxiniuc

07:43 PM Bug #5056: Temp-tables and unique indexes

I found the cause for the 'regression' of your patch. It is probably not a single thing, but a couple of issues which... Ovidiu Maxiniuc

02/20/2021

11:26 AM Feature #1587: implement full support for word indexes

Finished with word tables support for denormalized extent fields. Tested with a large customer app. and with test app... Igor Skornyakov

07:46 PM Bug #5056: Temp-tables and unique indexes

I used your patch to run the xfer tests and things have changed. It is not perfect, but I cannot tell exactly why.
... Ovidiu Maxiniuc

02/19/2021

11:31 AM Feature #1587: implement full support for word indexes

Added word tables' support for custom extents (conversion and database import).
Committed to 3821c/12031 Igor Skornyakov

10:48 AM Feature #4397 (Closed): add database attrs, methods and options

Ovidiu has recently confirmed that testing passed for the features in this task. Greg Shah

02/18/2021

06:27 PM Bug #5056: Temp-tables and unique indexes

Eric,
I looked deeper into the server's log (there are a lot of @java.lang.IllegalStateException: Empty scopes in Un... Ovidiu Maxiniuc

06:19 PM Bug #5056: Temp-tables and unique indexes

Eric,
I am sorry to inform you that the patch is not making the testsets behave better. In fact, the test stops much... Ovidiu Maxiniuc

05:10 PM Bug #5056: Temp-tables and unique indexes

Ovidiu, please test whether the attached patch resolves your original issue(s). It gets both of the test cases you pr... Eric Faulhaber

05:00 PM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> Please don't. If we were starting from scratch, we could consider it, but the changes you sh... Igor Skornyakov

04:44 PM Feature #1587: implement full support for word indexes

Please don't. If we were starting from scratch, we could consider it, but the changes you should be making to DDL gen... Eric Faulhaber

04:15 PM Feature #1587: implement full support for word indexes

As I can see at this moment FWD has a dependency on Apache Velocity. Can I use it for DDL generation? This will make ... Igor Skornyakov

03:44 PM Feature #1587: implement full support for word indexes

I've resolved the issues described in #1587-349, implemented the generation of the word tables with indices/constrain... Igor Skornyakov

03:13 PM Feature #4022: upgrade to a newer PostgreSQL release

> * Did the package come out of the standard Ubuntu repos? Or did you have to add a PPA to the system?
Ubuntu.
... Stanislav Lomany

01:55 PM Feature #4022: upgrade to a newer PostgreSQL release

>> Were you able to leverage the pre-built PL/Java binaries?
>
> I installed PL/Java package using apt, along with ... Greg Shah

02/17/2021

04:02 PM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> Igor Skornyakov wrote:
> > I've noticed two things regarding custom extends.
> > # If the ... Igor Skornyakov

03:58 PM Feature #1587: implement full support for word indexes

Igor Skornyakov wrote:
> I've noticed two things regarding custom extends.
> # If the number of the @<extent-field@... Eric Faulhaber

02:57 PM Feature #1587: implement full support for word indexes

I've noticed two things regarding custom extends.
# If the number of the @<extent-field@ subnodes on the .hints file... Igor Skornyakov

02/16/2021

01:57 PM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> No problem. The annotation for the @wiext@ index looks correct to me as is.
Well, formally ... Igor Skornyakov

01:50 PM Feature #1587: implement full support for word indexes

Igor Skornyakov wrote:
> I'm talking about @name = "fextFirst", legacy = "f-ext"@. The word table name was also ba... Eric Faulhaber

01:21 PM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> I'm having trouble understanding the question. Please provide an example.
I mean the follow... Igor Skornyakov

01:10 PM Feature #1587: implement full support for word indexes

Igor Skornyakov wrote:
> I've noticed that for word index with customized extent fields the value of the @name@ attr... Eric Faulhaber

01:05 PM Feature #1587: implement full support for word indexes

I've noticed that for word index with customized extent fields the value of the @name@ attribute of the @IndexCompone... Igor Skornyakov

02/12/2021

01:11 PM Feature #1587: implement full support for word indexes

Eric Faulhaber wrote:
> There are two aspects of performance we need to understand and measure from this testing:
>... Igor Skornyakov

12:44 PM Feature #1587: implement full support for word indexes

There are two aspects of performance we need to understand and measure from this testing:
* read (upon which the c... Eric Faulhaber

11:01 AM Feature #1587: implement full support for word indexes

Here is the suggested structure of the test table for performance testing of the new @CONTAINS@ support:
The test ta... Igor Skornyakov