The idea here is to add a facility to analytics which generates a complete gap analysis for an entire project as a single document.

Currently, I manually have to traverse reports (all of the ones that have support levels enabled) in the web UI to see each category that is marked with any support that is not CVT_LVL_FULL or RT_LVL_FULL. It is time consuming (and error prone) to type up each entry that is a gap.

Initially, if we just had a list of items with their non-full support levels, it would be enough to save a lot of time.

Each item might look something like this:

<keyword_or_category_descriptor> <4gl_feature_type> <cvt_support_level> <cvt_gap_details> <rt_support_level> <rt_gap_details>

It would also be useful to summarize the usage pattern from the associated details report. I can see 2 patterns here:

1. If there is a small number of usage locations (less than 11), then an explicit list of those locations can be provided.
2. If there are many usages (11 or more) then some summary like "X matches in Y files".

This would optimally be something generated by the ReportServer, driven by a request in the web UI. The output probably is best as formatted PDF. That can be converted in downstream usage to other forms (e.g. Textile).

The raw topics generated in #11415 can be simply parsed by a script and a list (CSV, textile, etc.) can be generated. Example generate now from the public kb is below.

FWD 4GL Feature Gap Analysis - Conversion / Runtime Support¶

Last updated: 2026-09-10 13:08 UTC
29 topics carrying a recorded support level, sorted alphabetically by keyword/descriptor.

Branch 11747a revision 16746 has the implementation of the gap analysis documentation generation tool.

Project Gap Analysis Document¶

• Project Gap Analysis Document
  • Introduction
  • Design
    • What is an item
    • The three groups
    • What makes a report a source of items
    • Presence is a gap
    • Duplicates
    • One keyword, several use cases
    • Unknown levels
  • Implementation
    • Report database
    • Carrying the group
    • Where the groups are declared
    • Built-in OO classes
    • The document model and renderer
    • Running inside a session
    • Statistics
  • Using it
    • Generating the document
      • System properties and classpath
      • Exit status and output
    • Prerequisites
    • What comes out
    • Reading an item
    • Adding marking for a new feature
    • When an item reports Unknown
  • Current limitations

Introduction¶

FWD Analytics already answers "what does this application use?" in detail: a hundred and more reports, each listing the categories of one 4GL feature with a support level attached. What it has not answered is the question a project actually starts with, which is "what in this application does FWD not yet support, and how much of a problem is that?"

This describes the facility which answers it. It reads the analytics report database and writes one document, organised the way a reader needs it rather than the way the reports are structured: every 4GL feature the project uses which FWD does not support completely, filed under the part of the language it belongs to, worst first, with what is missing and how much the project depends on it.

This was built on top of the gap detail storage of #11860. Nothing here changes how support levels are decided; it changes what is recorded alongside them and how the result is read.

Design¶

What is an item¶

An item is one 4GL feature, not one report row. That distinction is the whole of the design, because the reports overlap heavily: CAN-FIND is a category in Builtin Function Usage, again in Database Builtin Functions/Variables and again in Built-In Functions Called in WHERE Clauses. Three rows, one gap.

Every item carries a stable identity, built from three parts:

group / feature type / normalized name
base-language / function / can-find

The identity is stable across runs and across projects, so the same feature is the same item wherever it turns up. That is what makes it possible to merge the duplicates, and it is what a future annotation mechanism would key on.

The three groups¶

Every item is filed under exactly one of three top level groups, in this reading order:

1. Base Language — the language itself: statements, functions, expressions, error handling, sessions, security, codepages, and the built-in OO class library.
2. Database — buffers, queries, temp-tables, field handles, metaschema, embedded SQL.
3. User Interface — widgets, frames, windows, the clipboard, and the Windows-only facilities.

The important decision here is that a group describes the 4GL feature, not the report it appears in. A single report routinely spans all three: Builtin Function Usage holds ROWID (database), FRAME-COL (user interface) and SUBSTRING (base language). Filing by report would put all three in one place and be wrong for two of them.

So the group is recorded by the gap marking entry, which is where the feature is named in the first place. Of the twenty-four lookup maps in gap_analysis_marking.xml, nineteen hold entries of a single group and declare it once for the whole map. Five genuinely mix groups — attrmeth, stmts, funcs, globalVars and builtinClasses — and name one per entry.

What makes a report a source of items¶

A new per-report attribute, featureType, names the kind of 4GL thing a report's categories are: statement, function, attribute, data type, block option and so on. Its presence is what makes a report contribute to the document at all.

That is deliberate. Many reports list an application's own names — field names, variable names, class names, procedure names, RUN targets — and an application name is never a gap. The 408 field-name rows across three reports in one customer project are all blob and clob, which appear as exactly two rows in Database Field Usage (By Data Type). Those reports carry no featureType and contribute nothing.

Of the reports which carry support levels, sixty-four have a featureType and forty-one deliberately do not.

Presence is a gap¶

COM automation, ActiveX controls, window handles and .NET are not partially supported features. They reach outside the 4GL to a Windows facility a converted application does not have, so there is no support level to report and none of those reports carries one. Any use at all is the gap.

These reports are told apart by carrying a featureType but no support level expression, and they produce their own section of the document rather than support level tables.

Duplicates¶

Two reports seeing one feature are merged, in two passes.

The first pass merges by identity. Two candidates with the same group, feature type and name are the same feature by construction. A built-in class turns up in Class Name References wherever it is named and in OO References wherever it is used, and the first set is part of the second; reporting it twice with different counts would say nothing except that the two reports ask different questions. The candidate which saw the most occurrences is kept.

The second pass merges by occurrence set, which is what catches the same feature under different names. A name comparison misses KW_CAN_FIND against can-find() entirely; what is reliable is that both matched the same file, line and column every time. A narrower occurrence set is a different, narrower gap and stays its own item.

Comparing occurrence sets for every pair would mean reading every match in the project, so candidates are bucketed on a cheap fingerprint first and only collisions are confirmed exactly.

On a merge the more specific report wins the support level and the citation, but not necessarily the name: some reports name a feature by the parser token behind it, and a reader of the analysis knows the 4GL rather than FWD's token names. So can-find() wins over KW_CAN_FIND while the citation stays with the more specific report.

One keyword, several use cases¶

A marking entry is keyed by the 4GL keyword alone, with no receiver. attrmeth.put(prog.kw_clear, ...) is one entry covering every use of CLEAR, and CLEAR is four unrelated features: it empties a temp-table or a dataset, resets a CALL object, and clears what a frame or browse displays.

Splitting the marking is not the answer, and cannot be done reliably in any case: the receiver of an access through a plain HANDLE variable is only known at runtime. Instead the item keeps one group and the document prints a summary of the usages with examples underneath it — the kinds of thing the feature is used on, how many uses each accounts for, and a real line of source for each. The reviewer reads the split rather than inheriting a guess about it.

This is applied by rule, not by a list: the breakdown is computed for every attribute and method item, and printed whenever the item has more than one distinct receiver. Nothing has to maintain a register of which keywords are ambiguous, and a keyword which turns ambiguous in a later project is picked up on its own.

Unknown levels¶

An item whose support level is unknown is not a statement about the feature. It means the gap marking never reached it, which is a defect to fix rather than evidence of anything. Those items are always included and are called out as needing review, so that the marking gets corrected instead of quietly reporting a project as unsupported.

Implementation¶

Report database¶

Two columns were added.

Both are populated by the normal report run; no separate pass is involved.

Carrying the group¶

The group rides on the gap marking map value, the same way the gap details added by #11860 do. A map entry is either a plain support level or a GapMarking, and the marking rules read either form.

The three groups are named by the constants rw.group_base_lang, rw.group_database and rw.group_ui, which resolve through ReportWorker.resolveConstant().

At marking time the group is written as a note, support_level_gap_group, alongside the support level itself. The report walk reads the note and passes it to addMatchCategory(), which stores it in category.gap_group. That is the same path the conversion and runtime gap details already take.

Where the groups are declared¶

The nineteen unambiguous maps declare their group once, at the end of the init-rules in gap_analysis_marking.xml:

<rule>rw.setGapGroup(typeMatch     , rw.group_base_lang)</rule>
<rule>rw.setGapGroup(fieldRefs     , rw.group_database)</rule>
<rule>rw.setGapGroup(uiEvents      , rw.group_ui)</rule>

The five mixed maps name a group on each entry that needs one:

<rule>funcs.put(prog.kw_can_find, rw.gap(rw.group_database,
                                       rw.cvt_lvl_partial | rw.rt_lvl_full,
                                       "nested CAN-FIND inside a WHERE clause converts to a client-side expression",
                                       null))</rule>
<rule>attrs.put(prog.kw_search  , rw.gap(rw.group_ui, rw.cvt_lvl_full | rw.rt_lvl_full_restr))</rule>

Assignment is demand driven: an entry needs a group only once the feature actually surfaces as a gap. An entry without one defaults to base language, which is why the default is the group most features belong to.

A group written on an entry always beats the map's declaration, so a map can have a default and an exception at the same time. CONTAINS is the one such case today: it lives in typeMatch, whose declaration is base language, but it is the word index operator and therefore a database feature.

Built-in OO classes¶

builtinClasses is the one map which cannot hold a group next to a level, because it does not hold levels either. A built-in OO class is marked from the LegacyResourceSupport annotation on FWD's own implementation class, and the map is only a cache of what that lookup returned.

Those classes are grouped by package instead, in a twenty-fifth map, ooGroups, populated by addOoGroups() in gaps/expressions.rules. Everything defaults to base language, which is what the built-in class library mostly is — collections, strings, memptrs, JSON, reflection, HTTP, the web handlers. One package is not:

<rule>ooGroups.put("openedge.businesslogic.", rw.group_database)</rule>

rw.gapGroupFor() takes the longest matching prefix, so a class can still name its own group against the package it sits in.

The document model and renderer¶

Five classes in com.goldencode.p2j.report.server:

The model carries no presentation. Rendering to Textile is one renderer; anything else can be added without the analysis being redone.

Running inside a session¶

The build runs inside a report server session, which is what makes it honour the active file filter. A project's report database holds every file the conversion parsed, and the filter profile is what says which of them the project actually converts; reporting without it would report gaps in code the project does not convert.

The driver therefore opens the database the way the report web server does and establishes a session of its own, through two methods added to ReportApi: applyDefaultFileFilter(), which populates active_file for the session and names the profile applied, and releaseFileFilter(), which removes the session's rows afterwards. active_file is a global temporary table shared by every session and discriminated by a session id, so the driver uses a negative id, which the web server's own numbering can never produce.

Statistics¶

The statistics size the gaps rather than repeat them. A hundred features at partial support mean one thing in a twenty thousand line application and another in a two million line one.

Lines of code are taken from the existing analysis rather than recomputed. The source_line table holds every line of every file, blank ones and comments included, so counting its rows would not give lines of code; the report is produced by the walk itself, which knows which lines carry code and which arrived from an include file.

Two details of that report matter. Its last row is a totals row named "Totals (across n files)" rather than after a file, so summing a column without excluding it doubles every figure. And a file which contributes a schema as well as code has more than one row in the file table, so the rows must be matched to the filtered set with an EXISTS rather than a join, or such a file's lines are counted once per row.

Using it¶

Generating the document¶

The generator is a command line tool, com.goldencode.p2j.report.server.GapDocumentDriver. Run it from the project's root directory:

java -DP2J_HOME=. -cp "p2j/build/lib/*" \
     com.goldencode.p2j.report.server.GapDocumentDriver [project] [output file]

Both arguments are optional:

So all three of these are valid, and the last two are equivalent:

java -DP2J_HOME=. -cp "p2j/build/lib/*" com.goldencode.p2j.report.server.GapDocumentDriver
java -DP2J_HOME=. -cp "p2j/build/lib/*" com.goldencode.p2j.report.server.GapDocumentDriver MyApp
java -DP2J_HOME=. -cp "p2j/build/lib/*" com.goldencode.p2j.report.server.GapDocumentDriver MyApp myapp_gap_analysis.textile

System properties and classpath¶

P2J_HOME is the project root, not the FWD installation. It is what locates rptdb/, and it defaults to the current directory, so it can be omitted entirely when the tool is run from the project root. Set it when running from anywhere else:

java -DP2J_HOME=/path/to/project -cp "/path/to/project/p2j/build/lib/*" \
     com.goldencode.p2j.report.server.GapDocumentDriver MyApp

The classpath is FWD's own jars and their dependencies, which a converted project reaches through its p2j symlink at p2j/build/lib/. Nothing of the converted application is needed: the tool reads the report database, not the application.

No heap option is required for a project of ordinary size. A large project benefits from the same setting the report run uses, since the duplicate detection reads occurrence sets:

java -Xmx4G -DP2J_HOME=. -cp "p2j/build/lib/*" \
     com.goldencode.p2j.report.server.GapDocumentDriver MyApp

Exit status and output¶

The tool writes its progress to the console through the usual logging, ending with a line naming the file it wrote, how many items it found and how many need their marking reviewed:

INFO: Gap analysis: 96 candidate items
INFO: Gap analysis: 77 items after merging duplicates
INFO: Gap analysis written to /path/to/project/myapp_gap_analysis.textile: 77 items, 0 to review

It exits 0 on success and 1 if the report database could not be read or the document could not be written, with the cause logged.

Prerequisites¶

1. The analytics reports must already exist, which means a report run has been done for the project. If the gap marking rules have changed since that run, the conversion front end has to re-parse before the reports are regenerated: marking happens during the front end, which skips re-parsing when the 4GL sources have not changed, so a rules-only edit is otherwise ignored and the previous support levels are reported.
2. The report web server must not be running. H2 file-locks the report database, so the two cannot share it. This is not a limitation of the generator: the report run itself has the same constraint. Check with pgrep -f ReportWebServer and stop it first.

No server needs to be started. The driver opens the database itself.

What comes out¶

A Textile document with these sections:

• Summary — a count of items per group per support tier, which is the first thing to read.
• Base Language, Database, User Interface — the items, grouped by kind of feature, weakest support first, with conversion and runtime levels, use and file counts, and what is missing.
• Windows-only facilities — COM, ActiveX, window handles and .NET. The COM and ActiveX API surface is summarised by counts rather than listed, because a project which drives Excel reaches hundreds of members and no decision follows from reading the names one by one.
• Statistics.

From one test project:

Reading an item¶

|_. Feature|_. Conversion|_. Runtime|_. Uses|_. Files|_. What is missing|
|can-find()|Partial|Full|34|17|Conversion: nested CAN-FIND inside a WHERE clause
converts to a client-side expression. Also reported by: Built-In Functions Called in
WHERE Clauses and Builtin Function Usage|

The two support levels are independent and both are shown: full conversion support is worth nothing if the runtime is stubbed, which is why the summary tiers are decided by the weaker of the two. "Uses" and "Files" say how much the project depends on the feature. "What is missing" is the gap detail recorded by the marking, and "Also reported by" names the reports which were merged into this item.

Adding marking for a new feature¶

Nothing new is required. Add the entry to the appropriate map in rules/gaps/ as before, and:

• if the map declares a group, there is nothing else to do;
• if the map is one of the five which mix groups, write the group on the entry with rw.gap();
• if there is something worth saying about what is missing, say it in the same call, as the conversion or runtime detail. That text is what fills the "What is missing" column.

Then regenerate the reports, forcing the front end to re-parse, and run the tool again.

When an item reports Unknown¶

That is a marking defect, not a finding about the feature. The feature reached a report which carries support levels, but no marking rule assigned it one. Find the map the feature belongs in and add it.

Current limitations¶

• The "What is missing" column is empty for most items. The marking rules carry roughly 658 end-of-line comments which explain the gaps, and they have not yet been migrated into rw.gap() details. CAN-FIND shows what the column looks like once they are.
• Built-in OO classes and methods can never carry gap details, because their levels come from a Java annotation which has nowhere to record them.
• There is no web application integration. The document is generated from the command line only.
• Round-tripping a reviewer's annotations back into a regenerated document is out of scope.

Gap Analysis Document Generator: Performance and Presentation¶

• Gap Analysis Document Generator: Performance and Presentation
  • Introduction
  • The Problem
  • Analysis
    • Method
    • Where the time went
    • The dominant cost: one query shape
    • Why this did not show up earlier
    • The other costs
  • The Changes
    • Performance
    • Progress reporting
    • Statistics
    • Title cased headings
    • Colored support level cells
  • Verification
  • Results
  • Files Changed
  • Notes for the Reader

Introduction¶

The gap analysis document generator (com.goldencode.p2j.report.server.GapDocumentDriver) was first delivered in revision 16746. Its first run against a large customer project took twelve hours and produced almost no output while it ran, so there was no way to tell whether it was working or hung.

This change makes the same run take seventy six seconds on the same data, and makes it report what it is doing while it does it. It also carries a set of presentation changes to the document itself: the statistics section moved to the front and gained four new measures, every heading is title cased, and every support level cell is colored the way the reports color it.

Checked in to branch 11747a, revision 16768.

The Problem¶

Two problems, one of them hiding the other.

The run took twelve hours. That is long enough that nobody would run the generator twice in a working day, which in turn means the document cannot be regenerated casually after a marking change — and regenerating it after a marking change is the whole point of having a generator rather than a hand written document.

The run was also silent. Three log lines were written in twelve hours: the candidate count at the end of the first phase, the deduplicated count at the end of the second, and the final summary. Between 17:50 and 03:03 nothing was written at all. There was no way to distinguish progress from a hang, and no way to tell which phase was expensive — which is exactly the information needed to fix the first problem.

Analysis¶

Method¶

Each phase of GapDocumentBuilder was extracted into a throwaway harness and run separately against the real report database, so that each query could be timed on its own rather than inferred from the gaps between three log lines. The harness established a report server session and applied the default file filter exactly as the driver does, so the queries ran against the same active_file contents.

The reference project measures 47,109 rows in the file table, 29,881 distinct source files, 11,910,796 lines of code and a 133 GB report database. The measurements below were taken with -Xmx16g.

Where the time went¶

Three of the five phases were never a problem. Reading the gap categories aggregates every match of every gap category in a single query and does it in under a minute; the presence reports take three seconds. The cost was concentrated in two phases, and a third contributed a few minutes.

The dominant cost: one query shape¶

The usage breakdown reads the parser dump of every occurrence of every attribute and method item, so that an item used on more than one kind of receiver can be broken down by receiver. It ran this query once per item:

select m.text, m.fid, m.line
  from match m
  join active_file a on a.fid = m.fid and a.sid = ?
 where m.cid = ?
 order by m.fid, m.line

Measured on the largest such category — 45,117 occurrences — that query takes 442.6 seconds. There are 78 such categories on the reference project, which accounts for the nine hour block.

The reason is the join order H2 chooses. On the real table statistics it drives the query from active_file and reaches match through the foreign key index on match.fid, one file at a time, discarding everything whose cid is not the one asked for. The category's own occurrences are a rounding error next to the volume that plan reads: it walks essentially the whole match table for every category, and it does so in file order, which on a report database many times larger than memory is the worst possible page access pattern.

Rewriting the same question as an exists pins the driving table:

select substring(m.text, 1, 8192), m.fid, m.line
  from match m
 where m.cid = ?
   and exists (select 1 from active_file a where a.fid = m.fid and a.sid = ?)
 order by m.fid, m.line

The optimizer now enters through the index on match.cid, reads only that category's rows, and probes active_file once per row through the foreign key index it already has on fid. All 78 categories together: 2.6 seconds, 95,197 rows, 13.5 million characters.

Why this did not show up earlier¶

Two things conspired to hide it.

It does not reproduce at small scale. On a synthetic database with 200,000 match rows against 47,000 active files, H2 picks match as the driving table for the join form as well, and both forms are equally fast. The bad plan only appears once the match table is large enough relative to active_file for the cost model to prefer the other order. Any benchmark built on a scaled down database would have shown nothing.

An EXPLAIN on a synthetic database misleads unless the schema matches exactly. active_file carries a foreign key to file, and H2 creates an index on the referencing column to enforce it. A synthetic reproduction built without that constraint produces a completely different plan, and led initially to the wrong conclusion that active_file had no usable index at all. It has one, on fid; what it does not have is one on sid, which is not what these queries need.

The other costs¶

Duplicate detection re-read every occurrence. Two reports describing the same gap are recognized by their occurrence sets being identical, since the reports name the same feature differently. Candidates were bucketed on a cheap fingerprint — occurrence count, file count and the minimum and maximum file id and line — and every bucket holding more than one candidate then had all of its members' occurrences read back and compared exactly. On the reference project that meant reading occurrence sets for categories with hundreds of thousands of matches each, through the same cold random reads: two hours and forty five minutes to confirm what the fingerprint had already all but established.

The lines of code totals asked the filter question once per cell. The totals are summed from the custom "Lines of Code Analysis By File" report, whose rows have to be restricted to the files the filter profile selects. That restriction was written as a correlated exists on each numeric cell, so the file name lookup ran once for every cell of every numeric column rather than once per row: 145.9 seconds.

Report sizes counted rows the report run had already counted. reportSize() derived a report's occurrence count with count(m.id) over its matches. For the schema report "Fields by Table", that is 4,995,717 rows of cold reads — roughly six minutes — to arrive at a number the report run itself had already written into report_stats.

The Changes¶

Performance¶

1. The usage breakdown drives from the category. SQL_MATCH_TEXT replaces the join on active_file with an exists, as shown above. The occurrence's parser dump is also read through substring(m.text, 1, 8192) rather than in full: what the breakdown needs is the receiver, which is the dump's second line, plus the few lines a chained access walks through. The rest is the whole expression tree. On the reference project the longest of the 95,197 dumps read is 4,746 characters and the average is 142, so the cut never bites; an occurrence whose dump did not survive it would simply not be counted in the breakdown, which is already how the code treats a dump whose shape it does not recognize.
2. Duplicate detection no longer reads occurrences at all. SQL_ITEMS now computes two checksums over each category's occurrences in the pass it was already making: sum(mod(fid * 1000003 + line * 31 + col, 1000000007)) and sum(mod(line * 7919 + col * 131 + fid, 999999937)). Each term is reduced modulo a prime before it is summed, which keeps the total inside a bigint however many occurrences a category has. The two checksums join the counts and the bounds in the fingerprint, so candidates which share a fingerprint share an occurrence set and the exact comparison is no longer needed. The merge loop collapses accordingly: a bucket now yields exactly one survivor, the most specific report, with the others recorded against it.
3. The lines of code totals filter by row. SQL_LOC computes the set of surviving rowpos values once, in a subquery over the name column, and sums the numeric columns over that set with cell.rowpos in (...). The file name lookup stays an exists rather than a join, because a file which contributes a schema as well as code has more than one row in file and joining on the name would count its lines once per row.
4. Report sizes read the report's own statistics. SQL_REPORT_SIZE takes the occurrence count from report_stats.matches and counts only the categories, which is an indexed read. report_stats holds the unfiltered count, which is exactly what reportSize() has always returned, so this is a change of source and not of meaning.
5. The source reports are measured in one query. The include file report and the five user interface inventory reports are the largest reports the statistics touch. Asked one at a time they are six separate passes through a report database larger than memory; SQL_REPORT_USAGE asks for all of them at once, grouped by report title, and returns category, occurrence and file counts for each.

Progress reporting¶

Every phase now logs when it begins, and logs what it achieved and how long it took when it ends. Durations under a minute are written in seconds and longer ones as h:mm:ss. The usage breakdown, which is the only phase that reads a row per occurrence rather than a row per category, additionally reports every ten features. The statistics phase is broken into three reported sub-phases: the source and its lines, the includes and the user interface objects, and the schema. The driver logs what it is generating and where, names the filter profile it applied, and reports the total elapsed time alongside the item counts.

A representative run now looks like this:

INFO: Gap analysis of <project>, into /path/to/<project>_gap_analysis.textile
INFO: Gap analysis: file filter "Include All" applied
INFO: Gap analysis: reading the gap categories...
INFO: Gap analysis: 345 candidate items [36.8s]
INFO: Gap analysis: 272 items after merging duplicates [0.0s]
INFO: Gap analysis: breaking down how each attribute and method is used...
INFO: Gap analysis: usage breakdown, 70 of 78 features
INFO: Gap analysis: usage breakdown complete [0.7s]
INFO: Gap analysis: reading the features with no support level...
INFO: Gap analysis: features with no support level read [3.3s]
INFO: Gap analysis: measuring the project...
INFO: Gap analysis: counting the source and its lines...
INFO: Gap analysis: source counted [0.5s]
INFO: Gap analysis: counting the includes and the user interface objects...
INFO: Gap analysis: includes and user interface objects counted [29.8s]
INFO: Gap analysis: measuring the schema...
INFO: Gap analysis: schema measured [0.0s]
INFO: Gap analysis: project measured [30.3s]
INFO: Gap analysis written to ...: 272 items, 10 to review, in 76 seconds

Statistics¶

The section moved to the front of the document, immediately after the introductory note and before the summary. The figures are what the rest of the document is read against: a hundred partially supported features mean one thing in a twenty thousand line application and something else entirely in a twelve million line one, and a reader who meets the gaps first has nothing to weigh them against.

The temp-table schema no longer counts as a database. The schema reports carry it as a database of its own, which is what the reports of the web application want, since a temp-table is described by the same metadata as a permanent table and has to be filed somewhere. It is not a database the project connects to. SQL_DATABASES counts the categories of "Tables by Database" excluding the one named _temp.

Include file references say how far through the project they reach. The value is now written as 182,596 usages (in 24,645 files). How widely an include is used says more than how often: a hundred references from one program is a habit of that program, and the same hundred spread over a hundred programs is a dependency of the whole application.

Average program length is new: the total lines of code divided by the number of programs, where a program is one external procedure or one class, which is one source file either way. It is the crudest measure there is of how much work a feature woven through the application represents. Both figures it divides are rows immediately above it in the same table, so the reader can see what it is made of.

The user interface object inventory is new, drawn from the five reports added for it — Windows, Frames, Dialogs, Static Widget Usage and Dynamic Widget Usage — and presented in the same "usages in files" form, in the order a user meets them: the containers first, then what is placed inside them. A report database built before those reports exist simply has none of them, and the inventory is left out rather than reported as zero.

The section as it now renders:

Title cased headings¶

Every heading is title cased, as are the labels in the statistics table's measure column, so that the label column does not mix two conventions with the metric names in it.

The rule is conventional title case: every word is capitalized except the articles, the coordinating conjunctions and the short prepositions when they fall inside the title. Two details matter for this document in particular. A letter is only ever raised, never lowered, because the names here are 4GL and platform terms and COM/OCX, .NET, HWND and OO4GL are already written the way they are meant to be read. And only the first character of a word is considered, rather than the first letter found in it, so that a file extension stays .p rather than becoming .P.

Colored support level cells¶

Every support level cell in the item tables now carries the same background and foreground colors the reports use, so that a reader who knows the reports can see where a feature stands without reading the words. The colors are not duplicated: they are read from SupportLevelHelper.Descriptor, which is the single definition the reports and the wiki both describe, through the existing asRGBConstant() helper. A cell is written as Textile cell styling, for example:

|{background-color:#FF5000;color:#FFFFFF}. Stubs|

An unmarked feature keeps its own entry in that table, so an item whose level is LVL_UNKNOWN renders as a grey "Unknown" cell rather than as italic text.

Verification¶

The duplicate detection change was checked against the behaviour it replaces. Every fingerprint bucket holding more than one candidate had its members' exact occurrence sets read and compared, which is precisely what the old code did. 27 buckets, 27 confirmed identical, 0 false merges. The buckets are the expected ones: a statement seen by "Language Statement Usage" and by the area report that describes it better, a built-in class seen both where it is named and where it is used, and a keyword reported both by its token and by its 4GL spelling.

The dump truncation was checked against the data. Of the 95,197 parser dumps the usage breakdown reads, exactly one exceeds 4,096 characters, at 4,746. The limit was set to 8,192 accordingly.

The statistics were checked for equality with the figures the slower code produced. Every figure carried over unchanged, including the 4,995,717 field count now read from report_stats rather than counted, which confirms that the two sources agree.

The end to end run was repeated after each change, against the same report database, and the document compared. Item counts, group counts and the review count are unchanged throughout.

Results¶

Files Changed¶

No change was needed to DatabaseService, ReportApi or the report database schema. In particular, no index was added to active_file: the exists form rides the foreign key index the table already has on fid, and none of these queries filter on sid alone.

Notes for the Reader¶

The absolute timings in this note are specific to one report database on one machine, and the cold figures in particular are dominated by random reads against a store far larger than memory. What carries over to any project is the shape of the problem: the expensive queries were the ones asked once per category or once per cell, and each of the four changes replaces a repeated pass over the match table with a single pass or with a number already recorded.

The generator remains a command line tool. Nothing here changes how it is invoked.

Gap Analysis: Marking Corrections and Document Generation (r16781, r16782)¶

• Gap Analysis: Marking Corrections and Document Generation (r16781, r16782)
  • Introduction
  • Gap Marking Corrections
    • Levels which were wrong
    • Progress.Lang.Enum reported as entirely unsupported
    • The default which promoted four stubs
    • Metadata tables were filed under the base language
  • Reporting Corrections
    • ARGUMENTS is a preprocessor argument list, not a parameter list
    • The ActiveX controls were missing from the document
    • Telling an INI file environment from the Windows registry
  • Document Generation
    • The statistics became four tables
    • A row links to its own detail section
    • "Also reported by" is gone
    • Long tables collapse
    • Features With No Support Level became Platform Specific Features
    • CONNECT statement parameters
  • What This Changed in the dms Document
  • What Has Been Verified, and What Has Not
  • Open Questions
  • What Is Not Established
  • Sources

Introduction¶

Two revisions on branch 11747a, touching twelve files between them. They cover two strands of the same problem — what the gap marking says about a feature, and how the generated document says it — and they were committed together because several of the marking corrections were only visible once the document stopped obscuring them.

The reference project throughout is dms: 29,881 source files, 11.9 million lines, a 133 GB report database.

Gap Marking Corrections¶

Levels which were wrong¶

Progress.Lang.Enum reported as entirely unsupported¶

Every member of Progress.Lang.Enum read None / None, and the marking was not at fault in the way it looked.

The gap marking resolves a built-in OO class to the FWD Java class which implements it and reads the annotation there. Progress.Lang.Enum resolved to com.goldencode.p2j.oo.lang.Enum, which does not exist: the Java class is LegacyEnum, renamed because Enum collides with java.lang.Enum. Class.forName threw, and the marking recorded its class-not-found default of None / None. GetValue() had been marked full on LegacyEnum all along.

An entry in java_override_oo_names now maps it, in the same way Object, Class, Error and String were already mapped.

The default which promoted four stubs¶

A built-in OO member with no LegacyResourceSupport of its own now takes the level of the class which declares it. That correction is right for almost everything it touches, and wrong for four members.

Every LegacySignature member under com.goldencode.p2j.oo was read out of p2j.jar by reflection and every promoted member's Java body classified:

The 403 promoted to Full / Full are clean. The four are the indexed forms of Progress.Reflect.Variable, on a class marked CVT_LVL_FULL | RT_LVL_PARTIAL. They are now annotated as the stubs they are; re-audited, promoted members whose body is a stub went from 4 to 0.

dms does not use any of the four, so this never reached its document. It would have reached every other project's.

Metadata tables were filed under the base language¶

The document's Base Language group carried a Metadata Table section listing eighteen tables — _file, _field, _index, _sequence, _dbstatus and the rest — while the metadata field rows immediately below them were correctly under Database. One kind of gap was being read in two places.

The marking map had declared the right group since r16746; the group was being lost on the way to the report. Half of that was already fixed. What remained was the fallback: a table the map does not name is marked None / None, and a fallback carried no group, so it fell to the documented Base Language default. Nine of the eighteen came through that hole.

A metadata table or field reference is now recorded as a database feature whether or not the marking map names it.

Reporting Corrections¶

ARGUMENTS is a preprocessor argument list, not a parameter list¶

The Statement Option section held one row:

Read plainly that says a project passing arguments to 381 RUN statements has no support for doing so. It is not what the row means. A RUN statement's parameter list is LPARENS, which is fully supported; ARGUMENTS is the preprocessor argument list, which only a system that compiles at run time can act on.

The report database holds the 4GL source line behind every match. All 381 read like this:

run add_error_message "You have to enter a sublet type.".
run add_error_message("Receive ID is empty.")).
RUN add_error_message("One or more error(s) ocurred while processing stock number " + QU...
run vp_message = "Prospect " + STRING(vp_prspid) + " does not exist".

Every one is a RUN whose parameter list is malformed — a missing pair of parentheses, an extra closing one, or an assignment where a call was meant. The 4GL parser takes the trailing tokens as preprocessor arguments, which is legal, so the code compiles and the parameters are silently not passed.

The row is now named Preprocessor Arguments on a RUN Statement and carries a conversion gap detail:

If the OE project only runs in compiled form, then these are likely to be typos since they cannot work properly. In a development OE system or one that runs from source code, this could be valid code but cannot convert statically in FWD.

The level stays None / None, which is correct: FWD converts statically and cannot apply a preprocessor argument at run time.

The ActiveX controls were missing from the document¶

The section which holds everything whose presence alone is the gap said in its own introduction that it covered "COM automation, ActiveX controls and window handles" — and carried no list of ActiveX controls. The reports which find them had no featureType, which is the attribute that makes a report a source of gap analysis items.

OCX Control Loading by Descriptor now declares one, and the descriptor is reported without its 4GL string literal decoration — CtrlFrame rather than "CtrlFrame":U — the way the automation object report already reports a ProgID. On dms that is 622 controls across 575 files.

One caveat, and it matters. This is a list of control instances, not of control types. The descriptor is the name a control is loaded under within its container, not the ActiveX class behind it. The control type is not recoverable from the analytics run at all: the ProgID and CLSID live in the .wrx file, which is binary OLE compound storage and is never parsed. Where the actual controls in dms have been identified — the ProEssentials graphing controls, the TList grid, the RMPHTML editor — that was done by reading the .ocx binaries by hand, outside the analytics.

Telling an INI file environment from the Windows registry¶

The 4GL environment statements — LOAD, USE, GET-KEY-VALUE, PUT-KEY-VALUE, UNLOAD — read and write either an INI file or the Windows registry, and which one decides whether the use is portable at all. An INI file is read the same way anywhere; the registry exists only on Windows. describe_registry reported only the keyword, so the Windows Registry Usage section said KW_LOAD, KW_GET_K_V and so on, and carried none of that.

LOAD is the only one of the five which names the store, through its BASE-KEY clause — confirmed from the grammar, where base_key_clause is used by load_stmt and nothing else. A LOAD is now reported under the base key it names:

• LOAD BASE-KEY INI (INI file)
• LOAD BASE-KEY HKEY_LOCAL_MACHINE (Windows registry)
• LOAD BASE-KEY (set at run time)
• LOAD (no BASE-KEY, platform default)

USE, GET-KEY-VALUE, PUT-KEY-VALUE and UNLOAD act on whichever environment a LOAD put in scope and cannot be attributed to one or the other from the statement alone. The section says so.

From a source count, dms should split roughly 18 registry against 7 INI, plus one dynamic BASE-KEY p-baseKey.

Document Generation¶

The statistics became four tables¶

The document opened with a single thirty-row table carrying source counts, lines of code, include usage, the user interface inventory, the schema size and the gap count in one list — while the analysis below it is filed under three headings. A reader wanting to read the database gaps against the size of the schema had to find six schema rows among twenty-four others.

The statistics are now Code Summary, Database and User Interface, in the same order and under the same names as the groups the gaps are filed under, followed by Gaps — the former Summary table, which is a measure of the project like the others and is the one the rest of the document elaborates.

Possible Gaps and Features whose marking needs review were dropped from Code Summary, because the Gaps table carries both as its total and its Unknown column.

A row links to its own detail section¶

A feature with more than one kind of receiver gets a How It Is Used section further down. The table row now links to it, by a page-relative anchor so the link survives whatever the wiki page is eventually called:

|"@ERROR@":#usage-Base-Language-attribute-ERROR|...
h4(#usage-Base-Language-attribute-ERROR). ERROR — How It Is Used

The anchor is written explicitly rather than left to Redmine's heading-to-anchor rule, which the em dash in the heading would make unpredictable. It carries the group and the feature type as well as the name, because the same 4GL keyword can be an attribute of one thing and a method of another.

"Also reported by" is gone¶

A row with no gap details used to fill its "What is missing" column with the list of other reports the feature was seen in. That says nothing about the gap, and putting it where the gap details belong reads as though something had been recorded when nothing had.

Long tables collapse¶

A name table of more than twenty-five rows is wrapped in a {{collapse(Show Details)}} macro. On dms five sections collapse, the largest being the 622 OCX controls, which would otherwise turn one fact — how much of this is there — into twenty screens of scrolling. The rows are still there for a reader who wants them.

Features With No Support Level became Platform Specific Features¶

The old name stopped being true once the section took in features that convert and run perfectly well. Shared library calls and Windows registry usage are all marked Full / Full; what they cost is portability, not support. A native library has to exist on the new host, and a registry key does not exist off Windows.

Four sections were added, all read from reports which already existed:

These are read by report title, not by featureType. That matters: featureType marks a report whose categories name a 4GL feature, and a shell command category is a command line the application happens to contain. The route by title keeps that distinction and, as a side effect, works against an existing report database without a re-run.

CONNECT statement parameters¶

A CONNECT statement converts and runs, so it carries no gap — but what a reader of the database group needs to know is which parameters the project actually passes, because that is what decides whether the connections can be expressed against a FWD database at all. An inventory table at the end of the Database group now lists them. dms passes -db and no-error and nothing else.

What This Changed in the dms Document¶

The document regenerates in about fifteen seconds.

The base language fall is the WHERE clause functions leaving and the metadata tables moving to Database; the database rise is those eighteen tables arriving, less what left.

A diff of the report database's feature categories before and after accounts for every change: 626 added — 622 OCX controls, asc(), Hex Literal, the renamed ARGUMENTS row and one query option — 21 removed, and 157 changed, being 78 function rows, 51 metadata rows, 15 database triggers, 10 OO method and class rows and 3 others. Nothing changed that was not accounted for.

What Has Been Verified, and What Has Not¶

This is the part to read before relying on any level above.

The 2026-09-19 run was made against the r16781 content. So:

A conversion and report run is needed to confirm the second half. The dms/p2j symlink points at a stale copy of the branch and has to be repointed at ~/projects/11747a first, or the run will silently reproduce the old markings.

Short of a run, the following were verified directly:

• The Progress.Lang.Enum and Variable markings were read back out of the rebuilt jar by reflection.
• run_stmt_option_name was probed against a real dms AST and returns the new name on three ARGUMENTS nodes while leaving FILENAME, LPARENS and INT_PROC alone.
• The BASE-KEY navigation was probed against seven live nodes. The first version of it was wrong — the clause carries an EXPRESSION wrapper rather than the literal, so every LOAD would have been reported as "set at run time". That was caught by reading the AST rather than assuming its shape.
• ./gradlew jar and the testcases smoke test are clean on both revisions.

Open Questions¶

1. Do Possible Gaps and Features whose marking needs review belong in the Code Summary table? They measure the document rather than the project, and the Gaps table now carries both. They were removed; saying so here in case that is the wrong call.
2. Does dms run only from compiled code? If so, the 381 preprocessor argument uses are latent 4GL defects to be reported to the customer rather than a conversion gap.
3. Is an instance-level OCX list what is wanted, or should the work be done to name the actual controls by parsing the .wrx container or cross-referencing a hand-built map of the binaries?
4. Is Full / Stubs right for the four indexed Progress.Reflect.Variable forms, or is an indexed form better described as Full / None, being absent rather than incomplete?

What Is Not Established¶

• The r16782 markings have not been seen in a generated document, as set out above. The levels quoted for them are what the next run should produce.
• Nothing here was run against OpenEdge. Every statement about what FWD does is read from its source, its annotations or the report database.
• The 381 preprocessor argument uses were not exhaustively classified. Every one of the two dozen source lines read is a malformed call, but two dozen is not 381.
• The OCX list answers "how much ActiveX is embedded and where", not "which controls", and cannot answer the second without work outside the analytics.
• The audit behind the four Variable stubs covered com.goldencode.p2j.oo only, because that is where the changed default reaches. Whether any other package has a class marked better than its members deserve was not asked.

Sources¶

• Branch 11747a r16781 and r16782
• #11894 — GET-CLIENT returns the live client-principal rather than a copy
• #11888, #11892 — the marking evaluation and the correction queue these came out of
• Generated DMS Gap Analysis — the generated document the counts are read from

Every item is filed under exactly one of three top level groups, in this reading order:

It would be ideal to have a standard hierarchy for any artifact around FWD: knowledge-base, testcases structure, reports, etc.

In Hierarchy there is also persistence, ui and base_language defined. This was inferred from test-cases project. But, there is also "runtime infrastructure" as a different category. This includes:

AppServer connectivity (4GL-RPC, 4GL-WS, SOAP), the LOG-MANAGER and PROFILER session handles, the OE security model (AUDIT-CONTROL / AUDIT-POLICY / CLIENT-PRINCIPAL / SECURITY-POLICY / auth callbacks), and the ABLUnit framework itself.

Also, when working on that hierarchy, there was a decision to be taken for OO. In test-cases it is separate from these systems, but ultimately I decided to make it a "group" within the Base_Language system.