FWD » Core Development » Database

Branch 11903a was created from trunk r16769.

I used the codePage DMO annotation in processCodePages() instead of a second lookup. The information is already available, no need to search the whole schema for it.

Tested with both static and dynamic temp-tables.

Pushed up to revision 16774. Let me know if this should be merged into 11747a branch.

• Overview
• Reference 4GL behaviour
  • Code page expressions are evaluated when the table is defined
  • LIKE evaluates the source expression again
  • Dynamic tables inherit the resolved code page
  • The NEW SHARED definition is authoritative
• Changes
  • Conversion: expression code pages are no longer stored in the DMO
  • Conversion: code page registration, including LIKE fields and LIKE tables
  • Conversion: the NEW SHARED definition generates the shared DMO interface
  • Runtime: one code page store for static and dynamic temp-tables
  • Runtime: dynamic tables carry the LIKE source
  • Other changes
• Testcases

Overview¶

Branch 11903a (revisions 16800 to 16803, refs #11903 and #11747) fixes how FWD assigns the code page of temp-table CLOB fields declared with COLUMN-CODEPAGE, TTCODEPAGE or LIKE, for static and dynamic temp-tables and for NEW SHARED / SHARED definitions.

The changes are justified by the reference 4GL testcases attached to the task:

• ask-1.p defines NEW SHARED temp-tables tt01 to tt04, using literal and expression code pages, field-level and table-level LIKE, and dynamic tables built with ADD-LIKE-FIELD and CREATE-LIKE;
• ask-2.p defines the same tables as SHARED, with different COLUMN-CODEPAGE literals (these are ignored as authoritative is the new definitions from the ask-1.p procedure which invokes it);
• book.df is the schema of the database CLOB fields used as LIKE sources: clob-1250RO (1250) and clob-utf8basic (UTF-8);
• 11903_a.txt is the expected output, captured on OpenEdge, configured with -cpinternal ISO8859-1.

Each row printed by the testcases lists the GET-CODEPAGE value of every field of a table, in field order (? for non-LOB fields).

Reference 4GL behaviour¶

Code page expressions are evaluated when the table is defined¶

ask-1.p uses a counting function as code page expression. This proves the moment when the code-page expressions are evaluated (when the procedure is initialised, before any other code being executed).
Each call returns a different code page and displays a message:

FUNCTION func0 RETURNS CHAR ():
   c = c + 1.
   IF c EQ 1 THEN DO:
       MESSAGE "f0: 1250" c.
       RETURN      "1250".
   END. ELSE IF c EQ 2 THEN DO:
   ...
END.

MESSAGE "IF YES THEN RETURN ERROR.".

DEFINE NEW SHARED TEMP-TABLE tt01 NO-UNDO
   ...
   FIELD f01_6 AS   CLOB COLUMN-CODEPAGE func0().

The capture shows that all five calls are made before the first statement of the procedure body, one per definition and in definition order:

f0: 1250 1
f0: iso8859-15 2
f0: utf-16 3
f0: roman-8 4
f0: big-5 5
IF YES THEN RETURN ERROR.

The expression therefore cannot be stored as text in the DMO. It must be evaluated by the runtime when the procedure is instantiated, once for each definition that uses it.

LIKE evaluates the source expression again¶

A field defined LIKE a field with a code page expression does not reuse the value of the source. It evaluates the expression again, for its own definition. This holds for field-level LIKE, for LIKE chains, and for table-level LIKE:

DEFINE NEW SHARED TEMP-TABLE tt02 NO-UNDO
   ...
   FIELD f02_11 LIKE tt01.f01_6
.
DEFINE NEW SHARED TEMP-TABLE tt03 LIKE tt02.
DEFINE NEW SHARED TEMP-TABLE tt04 NO-UNDO
   ...
   FIELD f04_11 LIKE tt02.f02_11
   FIELD f04_12 LIKE tt01.f01_6
.

tt01: ? ISO8859-1 ISO8859-1 UTF-8 1250 UTF-8 1250
tt02: ? ISO8859-1 ISO8859-1 UTF-8 1250 UTF-8 ISO8859-1 ISO8859-1 UTF-8 1250 UTF-8 ISO8859-15
tt03: ? ISO8859-1 ISO8859-1 UTF-8 1250 UTF-8 ISO8859-1 ISO8859-1 UTF-8 1250 UTF-8 UTF-16
tt04: ? ISO8859-1 ISO8859-1 UTF-8 1250 UTF-8 ISO8859-1 ISO8859-1 UTF-8 1250 UTF-8 ROMAN-8 BIG-5

The last column of tt02 (ISO8859-15) and of tt03 (UTF-16), and the last two columns of tt04 (ROMAN-8 BIG-5), are the values of the 2nd to the 5th evaluation. The table-level LIKE of tt03 has no field definitions in the source code, but the expression of the inherited f02_11 is still evaluated for it.

The other columns show the static cases: a plain CLOB and a TTCODEPAGE CLOB get the session code page (ISO8859-1), a literal gets its value (UTF-8), and a LIKE of a database field gets its CLOB-CODEPAGE (1250, UTF-8).

Dynamic tables inherit the resolved code page¶

ADD-LIKE-FIELD and CREATE-LIKE do not evaluate the expression again. They copy the code page already resolved for the source field:

dtbh:ADD-LIKE-FIELD("f11", "tt02.f02_11")
...
dtbh:CREATE-LIKE("tt02").

dtt1: ISO8859-1 ISO8859-1 UTF-8 1250 UTF-8 ISO8859-1 ISO8859-1 UTF-8 1250 UTF-8 ISO8859-15 1250 UTF-8
dtt2: ? ISO8859-1 ISO8859-1 UTF-8 1250 UTF-8 ISO8859-1 ISO8859-1 UTF-8 1250 UTF-8 ISO8859-15

The same capture also shows the errors for invalid ADD-LIKE-FIELD sources (9055, 9055, 9054).

The NEW SHARED definition is authoritative¶

ask-2.p defines the shared tables with other literals:

DEFINE SHARED TEMP-TABLE tt01 NO-UNDO
   ...
   FIELD f01_3 AS   CLOB COLUMN-CODEPAGE "utf-32LE" 
   ...
   FIELD f01_6 AS   CLOB COLUMN-CODEPAGE "utf-32BE".

The second half of the capture, printed by ask-2.p, is identical to the first half. The SHARED procedure sees the code pages of the NEW SHARED definitions, and the expressions are not evaluated again:

tt01: ? ISO8859-1 ISO8859-1 UTF-8 1250 UTF-8 1250

Changes¶

Conversion: expression code pages are no longer stored in the DMO¶

rules/schema/fixups.xml used the text of the root of the COLUMN-CODEPAGE expression as the codePage DMO annotation. For COLUMN-CODEPAGE func0() this produced a meaningless code page:

@Property(id = 7, name = "f016", column = "f01_6", legacy = "f01_6", codePage = "func0", ...)

Now only a string literal is stored. An expression gets an empty annotation and is evaluated at runtime:

@Property(id = 7, name = "f016", column = "f01_6", legacy = "f01_6", codePage = "", ...)

Conversion: code page registration, including LIKE fields and LIKE tables¶

rules/annotations/codepage.rules:

• A literal COLUMN-CODEPAGE is removed from the AST. Its value is taken from the DMO annotation and no registerCodePage is emitted for it.
• A non-literal expression is moved to the procedure level and emitted as TemporaryBuffer.registerCodePage(buffer, property, () -> expression) in the Init block (rules/convert/buffer_definitions.rules).
• A field defined LIKE a temp-table field, with no COLUMN-CODEPAGE of its own, gets a copy of the expression of its source. The source is found by following the LIKE chain (resolve_col_cp), so tt04.f04_11 -> tt02.f02_11 -> tt01.f01_6 is resolved to func0().
• A table defined LIKE another temp-table has no field definitions in its AST: they are copied directly to the schema. The rule goes through the fields of the source table, and of its own source table recursively, and copies their expressions for the new table.

rules/fixups/post_parse_fixups.xml keeps the link from a DEFINE TEMP-TABLE ... LIKE to the definition of the source table as the like_tt_ref annotation. Before, it was used only for the NO-UNDO inheritance and then dropped.

For ask-1.p the Init block now registers every evaluation seen in the capture, including the table-level LIKE of tt03, which was missing before:

TemporaryBuffer.registerCodePage(tt01, "f016", () -> func0());
TemporaryBuffer.registerCodePage(tt04, "f0412", () -> func0());
TemporaryBuffer.registerCodePage(tt04, "f0411", () -> func0());
TemporaryBuffer.registerCodePage(tt03, "f0211", () -> func0());
TemporaryBuffer.registerCodePage(tt02, "f0211", () -> func0());

The field chains were resolved at runtime before (TemporaryBuffer.processCodePages looked up the LIKE source by name). This lookup is now done at conversion time.

Conversion: the NEW SHARED definition generates the shared DMO interface¶

Identical temp-table definitions share one DMO interface; the code page is not part of the comparison. rules/fixups/stable_naming.xml (Section B) assigned the generation of the interface to the last file in path order that defines the table. For ask-1.p / ask-2.p this was the SHARED definition in ask-2.p, so the shared interface carried the code pages of the view:

// Tt01_1.java, before
@Property(id = 4, name = "f013", column = "f01_3", legacy = "f01_3", codePage = "utf-32LE", ...)
@Property(id = 7, name = "f016", column = "f01_6", legacy = "f01_6", codePage = "utf-32BE", ...)

Now the first definition in path order that is not a view (NEW SHARED, or a local table) generates the interface. A view generates it only if no other definition exists:

// Tt01_1.java, after
@Property(id = 4, name = "f013", column = "f01_3", legacy = "f01_3", codePage = "utf-8", ...)
@Property(id = 7, name = "f016", column = "f01_6", legacy = "f01_6", codePage = "", ...)

Runtime: one code page store for static and dynamic temp-tables¶

• The code pages are kept by AbstractTempTable (setCodePage / getCodePage, keyed by property name) for both static and dynamic tables. They were in StaticTempTable before; TempTableBuilder threw IllegalStateException from setCodePage.
• The code page supplier API (setCodePageSupplier / getCodePageSupplier) is removed from TempTable, StaticTempTable and TempTableBuilder. The suppliers are only needed until they are evaluated.
• TemporaryBuffer.processCodePages is rewritten. It runs only for a master buffer (master == null); the other buffers of the table return immediately. For each CLOB property it uses the literal from the DMO annotation if there is one. Otherwise it evaluates the supplier registered with registerCodePage, once, and drops it.
• processCodePages is also called when a buffer is created for a dynamic temp-table, so a dynamic table gets the code pages from its DMO.

Runtime: dynamic tables carry the LIKE source¶

• P2JField has a new like attribute (table.field of the source field).
• TempTableBuilder.copyField sets it for ADD-LIKE-FIELD (by name and by field handle). addAllFields now copies each field with copyField, so CREATE-LIKE by name sets it too.
• DynamicTablesHelper.generateSchemaAst emits the like annotation. When the copied field has no code page of its own, it takes the code page of the source field: the literal from the source DMO, otherwise the value already resolved for the source table (getCodePage). This is how dtt1 and dtt2 get ISO8859-15 without evaluating func0() again.

Other changes¶

• TemporaryBuffer table proxy (invoke): all paths now go through invokeFunctionWithArguments, which unwraps a ConditionException from the reflection exception and rethrows it. Method.canAccess replaces the deprecated isAccessible. isDeleted / setDeleted are now private.
• I18nOps.getCodePage: the clob branch is removed, as clob is handled by the longchar branch.
• StaticTempTable and TempTableBuilder return the logical constants (logical.UNKNOWN, logical.FALSE, logical.of(...)) instead of new instances.
• orm.Query: UserTableStatUpdater.persist is called with db.toType(Database.Type.PRIMARY).
• SourceNameMapper: a missing map URL is skipped.

Testcases¶

ABLUnit testcases for the testcases project, built from the reference files, are in .docs/11903/testcases (see its README.md):

The database CLOB fields of book.df are added to the testcases database as a new BookCodepage table. The expected value of plain and TTCODEPAGE CLOBs is SESSION:CPINTERNAL, because the testcases server does not run with ISO8859-1.