I had done this once many years ago and I forgot the process that I used to get it done. We'll have to figure that out again. It might be tricky because we use such an old version of ANTLR.

The intention is to update the syntax diagrams in the Conversion Reference as well as to provide popup support in our Eclipse (#3883) / IntelliJ (#6319) IDE plugins.

This will be easy to do once we move to ANTLR v4.

Analysis of Approach¶

Objective¶

Provide high-quality, readable "railroad" syntax diagrams for the full range of Progress 4GL syntax. These diagrams would be used in:

• Wiki Documentation (e.g. Conversion Reference)
• VS Code Hover Help

The idea has been to generate these diagrams using our ANTLRv4 grammar (see #1757 and #10804).

Because Progress 4GL is highly ambiguous (same text, different meaning), highly irregular (different text, same meaning) and context-dependent with thousands of keywords, the Progress grammar relies heavily on actions and semantic predicates. This complicates using the grammar as input to generate diagrams.

If the "production" grammar (the one we use for real parsing) is not well suited to user-level diagrams, then we should consider a "presentation" grammar that is never used for parsing but is just there to document the syntax in a user-friendly way. This grammar would ignore parser performance or ambiguity problems in favor of clean, human-readable rules. If we go with a presentation grammar, then we would need to consider how we create and/or maintain it.

Options¶

Use Production Grammar¶

The simplest option is to just generate diagrams from the production grammar. It is not clear if this will work well enough.

Worries:

• Production code makes for terrible documentation: Even if you use TRPL to strip out all the actions and semantic predicates, the structure of a production parser is inherently hostile to human comprehension. To make the parser fast and LL compliant, we use left-factoring, synthetic rules and deeply nested loops. A user doesn't want to see a railroad diagram of obscure sub-rules they want a simple, unified diagram of a statement.
• It is common for languages to have 2 grammars. One is the heavily optimized compiler grammar; the other is the formalized EBNF "specification" grammar used strictly for documentation and human readability.
• Seamless Text Integration: We need to integrate these diagrams with manually written text descriptions and Javadoc-style explanations. If we decouple the documentation grammar from the code, we can easily embed EBNF snippets or generated diagrams directly into any static documentation generation process without worrying about a parser optimization suddenly breaking the visual layout of your hover help.

Repeatable Automated Generation of Presentation Grammar¶

We could build an automated pipeline to transform the "production" grammar into a "presentation" grammar. We could modify the standard "Meta-Parser" for ANTLRv4 grammars to generate ASTs representing the production grammar. Then we could use TRPL implement transformations like:

• strip out actions and semantic predicates
• hide internal helper rules
• make some non-terminal rules and replace with the actual definitions to flatten the structure
• rename rules where they are cryptic or not user friendly
• replace rules where the original just can't be transformed reasonably

Then anti-parse the transformed ASTs into the presentation grammar.

Once the presentation grammar is ready, then we would use a tool like rrd-antlr4 to generate the railroad diagrams.

Meta-Parser DetailsMeta-Parser Details

Manually Maintained Presentation Grammar¶

Although our parser does change from time to time, the changes are generally minor. We do not expect major overhauls in the future. With this in mind and also the fact that we have a lot of other documentation to integrate with, it may not make sense to build a complicated autoamted transformation process to transform our production grammar into a presentation grammar.

As an alternative, we could simply build/maintain a presentation grammar from scratch. In other words, we would edit it independently of edits to the production grammar. Whenever the presentation grammar changes, we would re-generate the railroad diagram artifacts.

Limitations of Rendering Formats¶

• SVGs in Hovers: VS Code hover providers return Markdown. While Markdown supports images, VS Code's internal renderer strictly limits local SVG files due to cross-site scripting (XSS) security policies. SVGs must be embedded as Base64 Data URIs or converted to PNGs.
• Alternative Formats
  • Mermaid.js does not have a native "Railroad" or "Syntax Diagram" type. While you can technically force its flowchart or stateDiagram modules to connect nodes, it utilizes a force-directed layout. It cannot properly represent the standardized loops, optional bypasses, and rigid horizontal alignments expected in a true syntax diagram. It will look like a messy flowchart rather than a professional language specification.
  • Graphviz (DOT) can be manipulated into drawing railroad diagrams, but it is notoriously tedious. Because Graphviz is designed to route edges dynamically to minimize intersections, forcing it into the rigid, strictly ordered, horizontal "tracks" of a syntax diagram requires injecting invisible nodes, strict rank=same constraints, and highly specific port bindings. Nasty and a waste of time.
  • Also: the existing Java-based tools for railroad diagrams do not output DOT or Mermaid.

We will plan to generate .png artifacts for each railroad diagram.

Diagram Generator¶

We will utilize the existing Java ecosystem. Tools like rrd-antlr4 (by Bart Kiers) or its fork RRD-ANTLR4 (by flashpixx) will be used to ingest the grammar and generate standard PNGs (or SVGs that are then converted to PNGs).