Adding 4GL Syntax¶

Adding 4GL Syntax

Introduction¶

FWD is updated with additional 4GL language syntax:

To add 4GL syntax which exists in OpenEdge but is not yet supported in FWD.
To enhance/improve the 4GL with new syntax that does not exist in OpenEdge.

This document will outline the steps necessary to update FWD for either such enhancements.

This chapter is incomplete. It only documents how to add attributes, methods, and built-in functions (partial).

Process¶

The conversion process has 3 phases. The first phase is the "front end" which is about parsing the 4GL into Abstract Syntax Trees (ASTs). The subsequent phases ("middle" and "code back end") are responsible for the processing of those ASTs into the proper database artifacts (e.g. DDL, Java Data Model Objects) and the converted Java code (e.g. business logic, frame definitions, menu definitions).

Front End
- Add the token type(s) and keyword(s) to the parser.
- Add the new syntax grammar to the parser.
Code Back End
- Modify TRPL ruleset(s) to add support for converting the Progress AST (4GL syntax) into the Java AST (Java syntax).
- Add or modify Java runtime code to add "stub" methods that allow the new converted code to compile (using javac).

Parser¶

If an unexpected token: error occurs during conversion, you will need to look over the context surrounding the error, and determine if it's a new keyword, and if it's an attribute, method, or built-in function. As a keyword, you need to know if it's reserved or unreserved. As an attribute, you need to know if it's read-only or writable. Understanding these data items upfront will help with implementation.

Parser changes are most often made in src/com/goldencode/p2j/uast/progress.g, which is an ANTLR grammar for lexing and parsing 4GL syntax. When this file is built, ANTLR generates the Java code and output files:

ProgressParserTokenTypes.txt
ProgressParserTokenTypes.java
ProgressLexer.java
ProgressParser.java

These files are never to be edited on their own, since they get regenerated every time FWD is built.

Adding Tokens and Keywords¶

ASTs are tree-based data structures where each node has an integer type (called the token type) and some text (the token text). For example, when the 4GL code DEF VAR my-var AS INTEGER. is parsed, the first token has a symbolic token type of KW_DEFINE and text of "def". The entire 4GL source code is parsed into nodes of a tree using these token types, so adding any new syntax usually means that tokens must be added.

In progress.g, all tokens are explicitly defined symbols. ANTLR does have the ability to automatically define tokens but this facility MUST NOT be used. A token is a symbolic name for a specific integer type which is associated with the symbol when ANTLR generates the Java lexer and parser. All that needs to be done is to add the new symbol to the list of tokens (make sure to place it in the right location).

Tokens exist for every keyword and any other punctuation used in 4GL syntax. There are also tokens that are used as artificial nodes in the tree, that are created by the parser to properly structure or organize the tree but which don't correspond to any specific 4GL text.

If a token is associated with a keyword, in addition to adding the token, one must also add to the keywords list. The keywords list defines all valid keywords, their text, whether they are reserved or not, any abbreviation that is possible and the token they should generate when the keyword is matched.

Adding the Token¶

Follow these rules when adding tokens to progress.g:

Check to see if there is already a token (and keyword if needed) for the syntax you are adding. For example, many keywords are used for multiple purposes and may already exist in the parser. In this case, you can just add references to the keyword token instead of adding anything new. The best way to search for a keyword is to search for the quoted text of the unabbreviated keyword. For example, if you search progress.g for "define" (including the double quotes), you will find this line new Keyword("define" , 3, KW_DEFINE , true ), which is how a new keyword is created. Notice how there is a reference to the synbolic token KW_DEFINE which has been created elsewhere. Since both exist, any new 4GL grammar using KW_DEFINE could be implemented immediately.
The symbol name must be formed properly.
- All tokens have a defined prefix, then an underscore, then at most 8 characters of distinguishing text.
- Keyword prefix is KW_ (these are the most prevalent modifications).
- Tokens must be unique.
- Tokens must be all uppercase.
The symbol must be placed properly.
- All tokens must be alphabetized within the correct section.
- For keywords, determine if is reserved or not by reviewing Progress documentation or writing test 4GL code. They have a table ("Keyword Index") indicating if keywords are reserved or not.
  - Reserved keyword section is within BEGIN_RESERVED / END_RESERVED
  - Un-reserved keyword section is within BEGIN_UNRESERVED / END_UNRESERVED
  - Determine the minimum abbreviation (if any).
  - If the keyword is undocumented, write 4GL code to determine if it is reserved and what minimum abbreviation exists (if any).
For a token which has no OpenEdge counterpart, document it with a comment // FWD extension, not real 4GL! on the end of the line so that it is clear this is not a legacy 4GL feature.

Define your token, and add it to the correct location in the tokens { } section of progress.g.

Adding the Keyword¶

The progress.g module contains a keyword array containing:

The full keyword text for the keyword.
The minimum number of characters for matching
- > 0 indicates abbreviations are possible, and the value is the smallest number of characters that will be matched. For DEFINE, the abbreviation is 3.
- 0 indicates no abbreviations are possible for this keyword.
The integer token type to assign when a match occurs (e.g. KW-DEFINE for "define").
Whether or not it is a reserved keyword (true indicates reserved).

Make sure you make your token additions as mentioned above. You will need the token name in this area.

Add an entry to the Keyword[] array in the initializeKeywordDictionary() method of the ProgressLexer section of progress.g.

Add the following fields to a row using new Keyword(fullText, minChars, tokenType, reserved),.
- fullText (java.lang.String) : Full/unabbreviated literal text form of the keyword.
- minChars (int) : Minimum number of characters needed to match the keyword.
- tokenType (int) : The integer token type associated with this keyword in the Parser and Lexer.
- reserved (boolean) : The keyword is a reserved keyword in the source language.
Insert this in alphabetized order by the fullText field.
See initializeKeywordDictionary() Javadoc for details.
For a keyword which has no OpenEdge counterpart, document it with a comment // FWD extension, not real 4GL! on the end of the line so that it is clear this is not a legacy 4GL feature.

Parser Changes to Add a Handle-Based Attribute or Method¶

In progress.g there is a HashMap called attrsAndMethods to which you put a mapping between the new keyword token and the data type of the attribute or method. The syntax is:

attrsAndMethods.put( <keyword_token_type>, <attr_or_meth_data_type> );

The keyword_token_type is the symbolic token name for the keyword associated with the attribute or method.
The attr_or_meth_data_type is the data type of the attribute OR the data type of the return value for a method. The valid data types are tokens:
- for attributes the tokens are found within the BEGIN_ATTR / END_ATTR section
- for methods the tokens are found within the BEGIN_METH / END_METH section

For an attribute/method which has no OpenEdge counterpart, document it with a comment // FWD extension, not real 4GL! on the end of the line so that it is clear this is not a legacy 4GL feature.

Example BUFFER-NAME Attribute

attrsAndMethods.put( KW_BUF_NAME, ATTR_CHAR );

Example BUFFER-VALIDATE Method

attrsAndMethods.put( KW_BUF_VLID, METH_LOGICAL );

Parser Changes to Add a Built-in Function¶

In progress.g there is a method called initializeFunctionDictionary() which contains all the built-in function definitions. Each definition is added via the addBuiltinFunction() method of the SymbolResolver class. The syntax is:

sym.addBuiltinFunction( <name>, <function_return_data_type>, <returns_unknown> );

The parameters:

name (java.lang.String) : Function name as a quoted string literal for the unabbreviated keyword text.
function_return_data_type (int)
- Use the symbolic token name of the type of the function's return value.
- The function return types are tokens found within the BEGIN_FUNCTYPES / END_FUNCTYPES section.
returns_unknown (boolean) : Specifies if this built-in function can possibly return the unknown value.

For an function which has no OpenEdge counterpart, document it with a comment // FWD extension, not real 4GL! on the end of the line so that it is clear this is not a legacy 4GL feature.

Example CAN-FIND()

sym.addBuiltinFunction("can-find" , FUNC_LOGICAL , false);

TBD: special cases (functions that optionally can be missing parameter lists, functions that are really global vars)

Code Conversion¶

The conversion rules for most 4GL syntax can be added by making changes in TRPL rulesets.

TRPL Changes to Add a Handle-Based Attribute or Method¶

Planning¶

Before you start making changes, you must research or decide some things:

For attributes, determine if it is read-only or is read/write. Read-only attributes will have no setters, read/write attributes will have getters and setters.
Methods will just have the single Java method name.
Determine the list of method names and their signatures.
- Model the names after existing versions for related or similar attributes/methods.
  - Attributes should follow Java bean conventions (get____ or is_____ for the getters and set____ for setters).
  - Methods should just use a normal camel-cased method name that is a reasonable mapping for the original legacy name.
- Any parameter which has native Java literals that can optionally be passed instead of the BaseDataType wrapper, will require overloaded signatures.
  - character/longchar types will need to be overloaded with String
  - logical is overloaded with boolean
  - integer is overloaded with int
  - int64 is overloaded with long
  - decimal is overloaded with double
- If multiple different primative types can be used in the 4GL, define the wrapper type that can accept all of the values.
  - For parameters that can be either character or longchar the BaseDataType parent is Text.
  - For parameters that can be integer, int64 or decimal the BaseDataType parent is NumberType.
All attributes and methods must be defined in a Java interface that implements certain annotations. This is a core design element of how handle based attributes and methods are supported. It enables the type-less nature of handles. This means you must find an existing interface for your additions or you must add a new interface. Some guidelines:
- If only a single resource has that attribute to method, then the interface will be unique to that resource.
- If multiple resources all have the same attribute or method, then the interface used must be implemented by all of the Java implementation classes for those resources.
- It is OK to implement the interface at a level of the class hierarchy which is shared across all or some of the resources.
- Interfaces are named based on their purpose. Sometimes the purpose is specific to the resource type and sometimes the purpose is "cross-cutting" (not resource-specific) but instead has some functional purpose.
- Only implement attributes and methods in interfaces that cover the exact resources which support those attributes and methods.

Examples

read-only attribute in multiple resources - TBD
read-only attribute in single resource - TBD

read/write attribute in multiple resources

LABEL in many widgets, BUFFER and LAST-EVENT
Interface is CommonField, implemented in BufferFieldImpl, GenericWidget and other locations

3 signatures

   @LegacyAttribute(name = "LABEL")
   public character getLabel();

   @LegacyAttribute(name = "LABEL", setter = true)
   public void setLabel(String label);

   @LegacyAttribute(name = "LABEL", setter = true)
   public void setLabel(character label);

read/write attribute in single resource

ROW-RESIZABLE in BROWSE
Interface is BrowseInterface, implemented only in BrowseWidget

3 signatures

   @LegacyAttribute(name = "ROW-RESIZABLE", setter = true)
   public void setRowResizable(boolean flag);

   @LegacyAttribute(name = "ROW-RESIZABLE", setter = true)
   default public void setRowResizable(logical flag)

   @LegacyAttribute(name = "ROW-RESIZABLE")
   public logical isRowResizable();

method in multiple resources - TBD
method in single resource - TBD

Changes¶

Once the above research/details are available, the following changes can be made.

Add rules to rules/convert/methods_attributes.rules to convert the attribute or method.
- For attributes or methods associated with "instance" based handles (e.g. widgets which can have many different instances), add a new descriptor line to to the load_descriptors function in rules/convert/methods_attributes.rules.
  - The format of the descriptor is: list.put(keyword, wrapper_interface, getter_name, setter_name_or_null, instance).
  - Example: <rule>list.put(prog.kw_row_resz, execLib("cr_descr", "Browse" , "isRowResizable" , "setRowResizable" , true ))</rule>
- For attributes or methods associated with a system handle, look for  and add a section for the attribute/method there.
- Some attributes or methods may need changes in both locations (if they are used on both system handles and "instance" resources).
- Please add in the proper alphabetical order for the respective sections.
For any attribute which is read-only (cannot be assigned), the attribute must be registered with the read_only_attribute function in rules/include/common-progress.rules.
For an attribute/method which has no OpenEdge counterpart, document it with a comment FWD extension, not real 4GL! in rules/convert/methods_attributes.rules
Add rules to rules/gaps/expressions.rules in the addAttrsMeths function to add gap marking support.

TRPL Changes to Add a Built-in Function¶

TBD

Runtime Stubs¶

Without changes to the runtime, the conversion changes will generate Java code that will NOT compile (using javac). For this reason, it is common to "stub out" the runtime with empty methods that allow the converted code to compile. Of course, empty methods won't do anything at runtime but it will allow the application to function (so long as the feature is not essential).

Attributes/Methods¶

The following steps should be taken.

Add the method signatures for attribute/method to the Java interface identified above.
- Create the new interface if it does not exist, otherwise just edit the existing interface.
- Make sure add LegacyMethod or LegacyAttribute (getter and setter, if available) annotations. Examples can be seen in the section above. These annotations should only be in the interface, not in any implementation class.
Add the method signatures for attribute/method to the Java implementation class(es).
- Each Java method should have an empty method body (for void methods like setters) or a body that just returns unknown value (for getters or methods with a non-void return value).
- Do NOT include the LegacyMethod or LegacyAttribute annotations.
- If the interface was newly created, you must add it to the implements line for the implementation class.
If this is a method, add it to map_meth in SignatureHelper - specify the signature and return type.
If this is a new Java interface:
- Add "unwrap" APIs in the handle class for this interface.
- Add the interface must to the extends list in the HandleCommon interface.
If this is a system handle resource (e.g. SESSION), the methods should be static method calls, ensure that Java method(s) for this 4GL attribute/method exist in both the interface and as a static method in the implementation class (like SessionUtils).
Any new system handle must be added to HandleOps.isSystemHandle() and it must have an "asHandle" API in its implementation class, where a proxy will be created (see SessionUtils.asHandle()).

Built-in Functions¶

TBD

Project

General

Profile

FWD

Wiki