Provides specialized Progress 4GL intermediate form representation (abstract syntax tree or AST) to Java AST conversion services.

Author(s)
Greg Shah
Sergey Yevtushenko
Eric Faulhaber
Nick Saxon
Constantin Asofiei
Stanislav Lomany
Date
November 19, 2009
Access Control
CONFIDENTIAL

Contents

Pattern Engine Rule Sets
Data Type Mapping
Literals
Naming
Declaring Class Data Members and Local Variables
Operators and Expression Evaluation
Variable References
Assignments
Function/Procedure Definitions
Function Calls
Methods and Attributes
Control Flow
Common Language Statements
Transaction Processing and Block Properties
Accumulator behavior
Database Support
User Interface Support
Unreachable Code Processing
Unreferenced Tables/Fields Processing
Open Conversion Issues
Planned Optimizations, Refactoring and Future Development

Pattern Engine Rule Sets

A great deal of the conversion processing is handled from pattern engine rule sets rather than being encoded in Java classes.  This yields a simpler implementation whenever the conversion task is such that the pattern engine's tree walking can be used to mask the complexity.  There are cases where Java helpers are needed and in these cases, methods are exposed to the rule sets from conversion oriented pattern workers.

It is important to carefully review the standard conversion rule sets in the p2j/rules/convert directory of the project.

Data Type Mapping

Summary

The following is a summary of the approach to mapping Progress data types to Java data types.

Progress Type
Parser Token Type
Java Type(s)
Initial Value
Notes
character
VAR_CHAR
FIELD_CHAR
com.goldencode.p2j.util.character ""
Progress uses a single data type to handle both single character values and strings.

For situations such as event processing (keystrokes), the Java char may need to be used in preference.  In addition, the data storage itself could be handled by java.lang.String.

See wrapper considerations below.
integer
VAR_INT
FIELD_INT
com.goldencode.p2j.util.integer

(or int in cases where it is determined that unknown value can never be assigned or compared to this variable)
0
The Progress integer is a 32-bit signed integral value whose maximum value is 2147483647 and minimum value is -2147483648.  Overflows wrap around to the minimum value and underflows wrap around to the maximum value.  This is an exact match to the Java primitive int, except for the case where the unknown value is assigned or tested on this variable.

The logical operators have a predictable response to having the unknown value as one or both operands.  This processing is identical for all data types.  See the table below.

See wrapper considerations below.
decimal
VAR_DEC
FIELD_DEC
com.goldencode.p2j.util.decimal

(this class will be implemented using double and NaN to represent the unknown value; BigDecimal will be used as a backing data type in the case where it is required; a default number of significant digits right of the decimal point will be 10 but can optionally be set to a smaller value based on the "decimals" keyword override)
0
The Progress decimal is a signed floating point value whose maximum value is 50 significant digits in size (the mantissa or significand is a maximum of 50).  Up to 10 of those digits can be to the right of the decimal point (the exponent is a maximum of 10).  Thus the largest value is 99999999999999999999999999999999999999999999999999 and the minimum value is -99999999999999999999999999999999999999999999999999.  If all of the 10 possible digits of precision are used to the right of the decimal point, then only 40 digits can be used on the left. Overflows and underflows both cause the same runtime error "Decimal number is too large (536)".

The logical operators have a predictable response to having the unknown value as one or both operands.  This processing is identical for all data types.  See the table below.

For more details, please see Decimal Support.
logical
VAR_LOGICAL
FIELD_LOGICAL
com.goldencode.p2j.util.logical

(or boolean in cases where it is determined that unknown value can never be assigned or compared to this variable)
false
The Progress logical has multiple "modes" yes/no, true/false and a user defined value1/value2.  However, all of these are the exact equivalent of the Java boolean type which has only 2 states, true or false, except for the case where the unknown value is assigned or tested on this variable.

The logical operators respond differently to having the unknown value as an operand:

Operator
Left Operand Unknown
Right Operand Unknown Both Operands Unknown
EQ
false
false
true
NE
true
true
false
LT
?
?
false
LE
?
?
true
GT
?
?
false
GE
?
?
true
AND (logical type operands only)
false if right operand is false, ? if right operand is true
false if left operand is false, ? if left operand is true
?
OR (logical type operands only) true if right operand is true, ? if right operand is false
true if left operand is true, ? if left operand is false ?
NOT (logical type operands only) n/a
?
n/a

Note that any logical expression that evaluates to the unknown value is always considered false (e.g. for purposes of control flow in an IF statement).

See wrapper considerations below.
date
VAR_DATE
FIELD_DATE
com.goldencode.p2j.util.date
null
A custom wrapper class will be written to provide the proper semantics of date processing in Progress.  Where needed, an instance of java.util.Calendar (obtained through Calendar.getInstance() will be used for the calendar processing however all operators will operate on the day number directly for speed and simplicity.  The custom wrapper class will provide Progress semantics and will hide the complexities of dealing with  the J2SE Calendar interfaces.

The logical operators have a predictable response to having the unknown value as one or both operands.  This processing is identical for all data types.  See the table below.

Note that there are special considerations for handling dates within a database where clause.
rowid
VAR_ROWID
FIELD_ROWID
com.goldencode.p2j.util.integer n/a
The persistence runtime classes transparently map the ID field for the associated database row to an integer wrapper.
recid
VAR_RECID
FIELD_RECID
com.goldencode.p2j.util.integer n/a The persistence runtime classes transparently map the ID field for the associated database row to an integer wrapper.
raw
VAR_RAW
FIELD_RAW
com.goldencode.p2j.util.raw 0 size array
In the Java-based runtime, this is not backed by a pointer to memory.  Instead, it is backed by a byte array.
memptr
VAR_MEMPTR
com.goldencode.p2j.util.memptr uninitialized array
In the Java-based runtime, this is not backed by a pointer to memory.  Instead, it is backed by a byte array.
handle
VAR_HANDLE
FIELD_HANDLE
com.goldencode.p2j.util.handle no contained object
Note that the parser converts all widget handles into handle types since in Progress these are equivalent.

This class is really just a container that holds a backing object.  It provides unknown value processing, comparison/assignment operator support and the valid-handle() builtin function implementation.

The specific backing object will need to be determined on a case by case basis.  The handle gets dereferenced when used as a referent for methods and attributes.
com-handle n/a n/a n/a n/a

See also (these are rule sets containing the pattern engine logic that handles variable conversion):

rules/annotations/variable_definitions.rules (name conversion and class name determination)
rules/convert/variable_definitions.rules (variable definition and initialization)
rules/convert/variable_references.rules (variable references)
rules/convert/expressions.rules (unwrapping of variables)
rules/convert/literals.rules (initialization constants)

Decimal Support

The Progress decimal is a signed floating point value whose maximum value is 50 significant digits in size (the hh is a maximum of 50).  Up to 10 of those digits can be to the right of the decimal point (the exponent is a maximum of 10).  Thus the largest value is 99999999999999999999999999999999999999999999999999 and the minimum value is -99999999999999999999999999999999999999999999999999.  If all of the 10 possible digits of precision are used to the right of the decimal point, then only 40 digits can be used on the left. Overflows and underflows both cause the same runtime error "Decimal number is too large (536)".  

The Java primitive double can easily handle the smallest values that can be represented by a Progress decimal.  The Java double is based on a 64-bit IEEE 754 floating point value where the maximum value is 1.7976931348623157 * 10 308 and the minimum value is 4.9 * 10 -324 .  The biggest issue is in duplicating Progress behavior with small numbers near its limit of precision.  For example:

0.0000000001 / 2 * 2 = 0.0000000002

In Java this would be calculated as 0.0000000001.  The Progress rounding behavior for very small numbers is not limited to just the operators but rather it seems that they have an internal floating point representation that is precise to 11 digits.  Every time they convert the "external" representation into this form, the rounding occurs.

For example:

0.000000000149 + 0.000000000001 = .0000000001

0.00000000005 =  .0000000001

0.00000000015 =  .0000000002

In this case, the important rounding occurs before the operation rather than after.  The net of this analysis is that all Progress use of decimal variables must be reviewed to determine if there are any conditions which are sensitive to the boundary between the 10th and 11th digit of precision (to the right of the decimal point).  This would be rare EXCEPT in the condition where the result of a calculation is something like 2.999999999999996 which in Progress would be represented as 3.0 but which is directly represented in Java.

The following approach seems (based on some non-exhaustive testing) to exactly duplicate the default Progress results for any double d:

Math.round(d * 10000000000L) / 10000000000.D

For any decimal variable specified with the DECIMALS clause (some value x which is < 10), the proper result can be generated by converting the two constants (e.g. 10000000000L and 10000000000.D which are both 10 10 ) to a version based on 10 x .  This solution handles the rounding (leveraging the higher precision values just as the Progress "internal" floating point implementation does) before the truncation occurs.  This is quite important.

Interestingly, since the IEEE floating point format is really a data structure (e.g. the double has 3 fields for sign, exponent and significant)  rather than a primitive, it has forms reserved for Infinity and NaN (not a number).  The NaN in particular can be considered the analog to the Progress unknown value.  In fact, it is silently propagated through mathematic operators into the result just as Progress does.  So if one or both of the operands are NaN, the result is NaN.  With this idea, it may be feasible to duplicate the Progress behavior without a Double wrapper!  Some J2SE math functions (e.g. Math.round()) do generate results that are not NaN (an int or long set to 0 in the case of round()) so calls to such methods must be handled specially. The NaN comparison logic is different from the Progress unknown value comparison logic, so this would need to be handled.   See wrapper considerations below.

The logical operators respond differently to having the NaN as an operand:

Operator
Left Operand NaN
Right Operand NaN Both Operands NaN
==
false
false false
!=
true
true
true
<
false false false
<=
false false false
>
false false false
>=
false false false

As in all floating point implementations, the system's "epsilon" and rounding behavior must be matched up.  An epsilon is the smallest floating point value that can be added to another floating point value such that the result can be differentiated as not equivalent by the native environment's operators.  As with everything else about Progress decimal support, the Progress and Java differences are substantial.

In Java, the epsilon is defined (by the language specification) as being equal to 2 -52 or 2.22044604925 X 10 -16 .  This is an exceedingly small number, especially in comparison with the smallest possible Progress decimal number.  So the following application was written to identify the epsilon for Progress:

def var possibleEpsilon  as decimal init 1.0.
def var epsilon          as decimal init 0.0.
def var mantissaBits     as integer init 0.

repeat.
   if ((1.0 + possibleEpsilon) = 1.0) then leave.
   epsilon = possibleEpsilon.
   possibleEpsilon = possibleEpsilon / 2.0.
   mantissaBits = mantissaBits + 1.
   display possibleEpsilon format "99999999999.999999999999999" mantissaBits.
end.

message "Epsilon = " + string(epsilon).
message "Mantissa Bits = " + string(mantissaBits).

It turns out that this program will generate an infinite loop.  This is because once the possibleEpsilon is reduced down to 0.0000000001 (the smallest representable Progress decimal), the result of the possibleEpsilon / 2.0 sub-expression will always be 0.0000000001 instead of 0.00000000005.  Since a difference of 0.0000000001 can be detected as different from any other decimal value, another way of looking at this result is that there is no epsilon value that is "visible" to the programmer.  There is certainly an internal Progress epsilon value but due to the fact that Progress rounds all intermediate results (after each operator runs), the result is always something that cannot be smaller than 0.0000000001 unless it is 0.0!  So:

0.0000000001 / 2.0 results in 0.0000000001
0.0000000001 / 3.0 results in 0.0

The important result here is that epsilon testing (the "normalization" of floating point comparisons to detect if something is really different or not) is not an issue for Progress applications because of the rounding/truncation behavior.   Internally, the Progress system maintains a larger number of digits of precision (than 10) but whenever accessed from an application (via an operator, assignment...) it is always truncated to the lesser of 10 digits (or the value of the decimals option for that variable) and the truncation uses rounding to determine the least significant digit.  In order to ensure compatible floating point operations in Java, the following rounding algorithm will be required after every operator (assuming num is the double result to be rounded/truncated and the default of 10 digits is assumed):

Math.round(num * 10000000000L) / 10000000000.D

Given this rounding implementation on the Java side, the normal epsilon testing is not necessary AND it is reasonably easy to implement the Progress "decimals" option which allows the programmer to specify an arbitrary number of digits of precision that is less than the default (and maximum) 10 digits.  So as a more general algorithm (where x is the number of digits supported):

Math.round(num * 10x L) / 10x.D

The main idea here is that result is converted into the long primative type which is used to capture those digits that must be maintained and to drop the other digits, then it is converted back into a double .  The Math.round() handles the proper rounding behavior for the least significant digit.

The situation with large decimal numbers is also difficult.  The largest decimal values cannot be represented in a Java double without losing precision.  The Java double has a mantissa which is represented by 53 (52 explicit bits and 1 implicit bit in normalized forms) bits or ~16 decimal digits.  This means that any integer number with 17 or more digits can't be saved in double without precision loss (the least significant digits are lost).  Arithmetic and comparisons of numbers with > 16 digits will be incorrect when using a double.

To address this large number issue, decimal operations must be reviewed for any situation that could overflow this boundary.  In these cases, the BigDecimal class will be used.

The Java double overflows by ignoring any overflow and setting the value to the maximum value (positive or negative infinity).  The Java double underflows by "wrapping" to (positive or negative) 0.  Special "boundary" processing will need to be written for some generated code due to the fact that Progress has a significantly smaller precision, a significantly smaller maximum mantissa and different overflow/underflow behavior.

To Wrap or Not to Wrap?

Where possible, the resulting generated code will be significantly easier to write, read and maintain when using primatives instead of object wrappers (e.g. int instead of Integer).  There is also a performance advantage to using primatives since this avoids the overhead of temporary variables and method calls to handle expression processing. This is an issue for integer, decimal ( note that NaN may be a solution that still allows primative doubles to be used in this case ) and logical types since in Progress, such variables can be assigned and compared with the unknown value.  The unknown value is a similar thing as the Java concept of null .   The main difference is that Progress silently allows unknown value to "propagate" where Java tends to catastrophically fail at runtime (e.g. NullPointerException ).  In Progress an arithmetic sub-expression will return an unknown value if either or both operands are the unknown value.  Since primitive Java types cannot be assigned or compared with null , wrapper objects or some other solution must be used in that case.

In the following conditions an integer, decimal or logical variable must be represented by a wrapper class:
  1. All parameters defined via a "define parameter" language statement that have an input or input-output type are passed in and thus they may be passed as the unknown value.  An output type parameter doesn't require wrapper support in and of itself because its origin is known.
  2. If the variable is shared and is defined external to this procedure then a wrapper must be used because it must be assumed that it could be assigned the unknown value or compared against the unknown value either upstream.  Any new shared variables in this procedure do not need to be wrappered unless some other condition would cause this to occur.
  3. If the variable is explicitly initialized to the unknown value.
  4. If the unknown value is ever assigned to that variable through the assignment operator or the assign language statement.
  5. If the unknown value is ever used on the other side of any binary operator (all the arithmatic operators handle operands of unknown value by returning the unknown value) including multiply, divide, kw_mod, plus, minus, equals, not_eq, gt, lt, gte, lte, kw_and and kw_or.  The kw_begins, kw_matches and kw_contains binary operators do not operate on integer, decimal or logical variables so they are not at issue.
  6. If the variable is ever passed to an external procedure as an input-output or output parameter (in a run statement), a wrapper will be used to avoid a complex recursive/multi-file search for downstream comparisons/assignments against the unknown value.  An input type parameter doesn't require wrapper support in and of itself because its origin is known.  Input-output or output parameters of an external procedure (run statement) are potentially modified in the called procedure which means that this procedure could see an unknown value which is generated externally.
  7. If the value is ever assigned or compared to the return value of a function call which can return the unknown value.  A subset of the built-in functions can do this and some user-defined functions can return the unknown value.
  8. If the value is ever assigned or compared to the return value of a method or attribute.
  9. If the variable is defined as a parameter of a function (function declaration) that has an input or input-output type.  An output type parameter doesn't require wrapper support in and of itself because its origin is known.  If this function is only called from the current procedure, then the call sites can be examined to determine if the input can ever be unknown.  If the function is called externally (through the "in handle" form), then a multi-file review would have to occur which is beyond the scope of this pass of the conversion.
  10. If the variable is ever passed in a function call in a position that matches an input-output or output parameter.  Function call parms that match up with input-output and output types are obviously a problem in that hidden code may be invoked to assign the unknown value to that variable before returning.  It is possible that a local function could be analyzed but to limit the complexity of this processing, this will not be done.  Built-in functions and remote functions are not as easy to analyze.  In the case of built-in functions, each one would have to be reviewed manually as a "black box" since no source is available in these cases.
  11. If any variable or field in any expression can be the unknown value, then any other variable assigned from expression can be silently set to the unknown value as a result of the expression evaluation.  This last implication causes a requirement to iteratively process expressions expanding the list of possible nullable variables based on assignments from other variables/fields that are nullable, until no additional nullable variables are found.  This will take an undefined number of iterations to finish since each iteration can add to the list of nullable variables and then a subsequent scan will lead to new nullable variables that are only nullable because of the last scan's changes... and so on.
See also:

Wrapper Usage

Other Variable-Like Entities

Progress allows many user-defined resources that are not treated as variables.  A non-exhaustive list includes streams, frames, widgets, queries, buffers, and temp-tables.   These resources cannot be used in expressions nor can they be used as a storage type in the database, however in Java their representation will be as objects.  Each of these resources will be converted by resource-specific code and this section of the conversion does not address such resources.

Literals

The following is a summary of the approach to mapping Progress literals to Java literals.

Progress Literal
Parser Token Type
Java Literal/Object
Notes
true or yes
BOOL_TRUE
true
This is an exact match.
false or no
BOOL_FALSE
false
This is an exact match.
? (unknown value)
UNKNOWN_VAL
If used as the right side of an assignment, this is covnerted to a setUnknown() method call on the BaseDataType object that is the lvalue.

If used as 1 operand of a EQUALS, the other operand will be passed to CompareOps.isUnknown().

If used as 1 operand of a NOT_EQ, the other operand will be passed to CompareOps.notUnknown().

In Progress it isn't possible to compare two instances of the unknown value literal.

Other usage converts to an instance of com.goldencode.p2j.util.unknown.
At first glance, null is an exact match.  However, in order to eliminate the need to place null checks throughout code and to refactor much of the code, wrappers are being used which internalize the unknown value.
integer literal
NUM_LITERAL
Java integer literal
This is an exact match since Java integer literals default to int.
decimal literal
DEC_LITERAL
Java floating point literal As long as the value is within the mantissa limits listed above (16 decimal digits), this is an exact match since Java floating point literals default to doubles.  Otherwise, a BigDecimal must be used.
date literal
DATE_LITERAL
com.goldencode.p2j.util.date There is no date literal in Java.  An instance of the com.goldencode.p2j.util.date will be created instead.
string literal STRING
Java string literal
Any string options will be removed and any enclosing single or double quotes will be removed.  Escaped characters (using the tilde character) will be converted to native representations and any characters that need to be escaped will be so escaped:
  • '\b' - backspace
  • '\t' - horizontal tab
  • '\n' - linefeed
  • '\f' - formfeed
  • '\r' - carriage return
  • '\"' - double quote
  • '\'' - single quote
  • '\\' - backslash
  • '\ooo' where ooo is an octal character literal
  • '\uxxxx' where xxxx represents 4 hexidecimal digits - unicode character literal
The final result when output will be output inside enclosing double quotes but there is no need for those quotes inside the AST form.

See also:

LiteralConverter
ExpressionConverterWorker

Naming

Progress identifiers and Java identifiers are similar but there are some important incompatibilities.  Most importantly the following characters are valid in a Progress name but invalid in a Java name:

#
%
&
-  (this is the hyphen character, not the underscore)

While Java does allow Unicode letters to be included in identifier names, the plan is to limit valid letters to the 7-bit ASCII letters of a - z and A - Z.  Java identifiers will always start with one of these letters and in subsequent characters a letter, any digit (0 - 9), the underscore (_) and dollar sign ($) can all be used.  All of these Java identifier characters can also be present in a Progress identifier.

So there is an inherent problem caused by a reduced identifier namespace on the target platform.  The troublesome characters (all of which are used in real application code) must be converted to something that is unlikely to conflict with other visible (scopes must be taken into account) names.  Conflicts must be detected and resolved automatically, with overrides being available (via hints) for manual intervention.

As an example of the conflict, assume that hypens were converted directly to underscores.  The following 2 names "hello-world" and "hello_world" would both be converted to "hello_world".  Thus if both identifiers were in some accessible scope simultaneously, then the resulting code would not be correct.

One should note that in Progress, the convention is to use the hyphen as a word delimiter in an compound word identifier.  In Java, the convention is to use a "camelCase" identifier (Java identifiers are case-sensitive) for the same purpose.

For different types of names, different conversion strategies will be followed allowing the Java standards to be matched as closely as possible.  The following is a summary of the approachs to be used.

Progress Identifier
Java Identifier Strategy
Conflict Resolution
directory name
package name
Package naming will be somewhat direct (following the Java standard):

com.<customer>.<application>.<directory>
TBD
file name
class name
A set of hints will assist in class naming.  Where common sequences are encountered such characters might be expanded based on hints.  For example, so and po might be expanded to ServiceOrder and PurchaseOrder respectively.  A file ending in -r or -x might be have ReportBody or Export appended respectively.
TBD
procedure name
method name
TBD
TBD
function name
method name
TBD
TBD
variable name (normal variable that is assigned or passed as a parameter)
variable name
Hypens will be removed and the words will be camelcased.  This will result in lowercasing some letters (including the first char of the identifier) and uppercasing the first character of subsequent words.  Words will be assumed to be separated at the hyphens.
TBD
variable name (never assigned or passed as a parameter)
constant name
All letters will be uppercased and words will be separated by an underscore (_).
TBD
schema name
TBD
TBD TBD

See also:

NameConverter
NameConverterWorker

Declaring Class Data Members and Local Variables

Variables can be generated in a Progress program by any of the following language constructs:

Language Statement/Construct
Conditions
Options Are Controllable
define variable
always
yes
define parameter
when TABLE-HANDLE, AS or LIKE keywords are encountered
yes
format phrase
when following an unrecognized symbol and the AS or LIKE keywords are encountered
yes
function parameter
when TABLE-HANDLE or AS keywords are encountered no
message
when SET or UPDATE clause occurs with a child of SYMBOL and a child of the SYMBOL which is the AS keyword (as opposed to a child which is an lvalue that has an optional AS keyword which is bogus)
no

Reference IDs

In each of the locations above where options can be defined for a variable, those options are saved as annotations.  In addition any AST that represents a reference to that variable will have all the non-default options annotated.  To allow variable references to be cross-referenced back to the original AST node that defined the variable, a "temporary index" is maintained for each variable definition.  This index is not project unique (such as the AST node ID) but rather is only unique within a given file.  Each reference to a variable has the temporary index written to an annotation.  The reason the node IDs are not used for this purpose is because this cross-reference is best created at parse time and the node IDs don't get created until after parse time due to limitations of how ANTLR instantiates AST nodes.  Post parse fixups are used to convert these temporary ID annotations into an annotation containing the real node ID of the defining AST node.

Shared Variables

Support for shared variables is handled via a shared variable manager.  This runtime class is called SharedVariableManager .  It is based on the ScopedSymbolDictionary and is designed to exist in a per client/session basis (stored in the security context).  This allows all transactions to access these shared variables but for this access to be done maintaining context specific state.  This approach is a replacement for features that would normally be provided via thread local variables which can't be used in the transaction server.

Whenever a new external procedure scope is started, if new shared variable references are added, the corresponding Java source code will add a scope.  Likewise this same Java method will use a finally {} clause to delete the scope on exit.  Global variables are added to the global scope and regular shared variables are added to the current (top) scope.  Such additions only occur if the definition is defined with the NEW keyword.  This allows "downstream" methods to access the most current instance of a shared variable by executing a lookup using a given name and when the method that defines a new shared (non-global) variable ends, the scope must be deleted (deleting all references to variables defined in that scope).  Any process that accesses a shared variable gets the shared variable manager out of the current security context and then does a lookup of the variable name and assigns the reference to the resulting local instance.

Where possible, shared variable usage will be converted into object references passed as method parameters or contained in other classes passed as parameters.  In the first pass conversion, this is not planned due to the extreme use of shared variables (there can be tens or even hundreds of these defined in a procedure) and the arbitrary depth at which such references can be used makes it unwieldy to easily convert these to passed parameters.  For this reason, the original semantic must be maintained.  The shared variable manager provides this same semantic at runtime.

Note that this same infrastructure is used to provide support for shared streams .

See also:

rules/convert/variable_definitions.rules
rules/convert/input_output.rules
rules/convert/control_flow.rules
rules/annotations/shared_resources.rules

Extents

Progress arrays are very similar in nature to Java arrays.  A variable definition can be an array if it is explicitly specified or if it is not specified and the variable is defined like another variable or field (a LIKE clause) that is defined with extent > 0.  Either way, there will be a resulting KW_EXTENT with a numeric constant size and a "long" type annotation named "extent" on the variable definition root node.

The extent processing is mapped to array variable definitions and array variable initializers .  This is a fairly direct mapping.

The referencing of array elements is also very similar between Java and Progress (both use numeric expressions inside postfixed square brackets to index the element). The following differences exist:
  1. Progress indexes arrays using a 1-based index.  Java uses 0-based indices.
  2. There are certain language statements (e.g. DISPLAY) that support a range of indices using a KW_FOR NUM_LITERAL construct inside the square brackets (after the first numeric expression).  This form can't be used inside expressions but it will have to be handled.
  3. There are certain language statements (e.g. DISPLAY) that support referencing all elements when an array variable name is used with no subscript at all.
Requirement 1 is handled inline during conversion.  The Java subscript operator [] is directly mapped to the Progress extent operator [].  NumberType.subscript() is used to decrement the 1-based subscript and return a 0-based version.  For array variables, this method takes a reference to the BaseDataType[] being used.  This allows the subscript() method to handle bounds checking and generate the proper error.  Please note that using an array index that is out of bounds during a NO-ERROR (silent error mode) language statement/assignment will fail with an IndexOutOfBoundsException since we currently directly map array access into the standard Java subscript operator [].  While we do have a central method in NumberType.subscript() to handle the bounds checking and error generation, in silent error mode this will return an out of bounds value.

Note that any constant (NUM_LITERAL) array subscript in Progress is converted to the proper 0-based constant in Java (no use of the subscript() method is needed in that case).

For fields, since all access is via get/set methods, the [] operators are not used and the array index is passed as the last parameter of the method call (the only parm for a getter and the 2nd parm for a setter).  Bounds checking and error generation is handled inside the persistence classes in that case.

This same approach is also used in some of the frame field cases in order to properly dereference the widget in the frame (which is done via getter methods).

Requirements 2-3 are handled by expanding these array references into the proper list of references. Then all downstream processing is otherwise the same once the implicit reference is expanded into its explicit list.  This expansion works conditionally in case #2, only working where the start of the range is specified as a constant (NUM_LITERAL).  No support for array subscripts with ranges that use a non-constant expression as the start index as in myvar[ 2 + j FOR 3 ].  Such cases can not be used in an expression, instead they can only be used as implicit reference to a range of index positions.

Initial Values Logic

All variable definitions are explicitly initialized to some known value (see the above table ).  Either a null is assigned or the appropriate constructor is called with the literal initializer (default or as specified in the Progress source).  The initializers are all rooted at a node of type KW_INIT and these values can come from another variable or field (LIKE clause) or can be explicitly specified in the source code.  Either way, there will be a resulting list of initializers which are walked at conversion time to emit the appropriate constructors or null .

In the case of array processing (see Extents ) the number of explicit extents may exceed the number of initializers provided.  This is a different behavior than Java where one can specify one or the other but not both.  If there are no explicit initializers, then all elements of the array are initialized to the default value.  If any initializers are speciifed, these are applied to the matching elements in the array and then all remaining elements without an explicit initializer are initialized to the last initializer in the list.

Wrapper Usage

See To Wrap or Not To Wrap .

Based on the extreme complexity of this task, the effort to definitively determine whether an integer, decimal or logical variable requires a wrapper is going to be deferred at this time.
  As a first pass, all variables will always be wrapped with the ability to add an optional hint that allows an override to a primative type when it is known to be safe.

Note that moving to J2SE 5.0 does provide some small relief by providing a new language feature for auto-boxing and auto-unboxing.  This largely makes the use of wrappers transparent (it wraps or "boxes" a primitive into a wrapper when needed and unwraps or "unboxes" a wrapper into a primitive when needed).  This makes it possible to write arithmetic and logical expressions directly using wrappers.

However, the following limitations (which are also quite significant ) apply:
  1. null cannot be assigned to a primitive and any unboxing or usage of a null wrapper will result in a null pointer exception.  This means that null checks will have to be embedded throughout expressions which greatly reduces the value of auto-boxing/auto-unboxing.
  2. The == operator cannot be used because it does not give consistent results when comparing two operands where either one or the both of them are wrappers.  This is due to the fact that when this operator is dealing with objects (versus primatives), it is comparing object references and this will generally yield false-negatives in most situations.
  3. Overloaded method signatures can be tricky because method resolution does change with auto-boxing added.  These method resolution changes are supposed to be compatible but this remains to be seen.
  4. The combination of auto-boxing and generics means that the ternary operator will auto-cast the 2 expression results to the least common denominator result if their types are different.
At this time, we do not plan to implement J2SE 5.0 since the result is not much better.  We can do our null checks and then if everything is non-null, we can unwrap into temporary primitive variables and then wrap the result back up and assign them back.

The problem with this solution is that many expressions require significant refactoring to make this work.  For example, in the case where a null check fails, all possible assignments that could be forced to the unknown value must be detected.  In addition (possibly more important), any side effects of the expression must be triggered in a safe manner.  For example, if a function is called as part of an expression, that function may have side effects that the rest of the procedure relies upon.  The conclusion: not only is the refactoring of the expression highly non-trivial but there are very strong reasons to actually allow the expression to execute even in the presence of the unknown value (which is how Progress works today).  If null is used as a proxy for the unknown value, this aspect of the behavior cannot be duplicated since Java generates a runtime error for these cases.  Since the J2SE wrappers have no way of encoding the unknown value except setting the current reference to null , this eliminates the J2SE wrappers from being a useful/good solution to this problem.

The best solution is to implement custom classes (corresponding to each Progress data type) to handle all of the following:
  1. Stores the data in a form that provides an exact match with the Progress data.
  2. Stores the state of whether this instance is really equal to the unknown value.
  3. Implements all of the operators and replacements for the Progress built-in functions such that the 100% Progress compatible results are generated.  Note that for integer and logical, this is primarily a matter of handling the unknown value in a transparent manner.  For decimal, there is the added requirement for the rounding/truncation to be implemented after each operator and function. 
All of these custom classes will reside in the com.goldencode.p2j.util package and they will each have a class name the same as the lowercase data type name (e.g. class integer for the integer data type). This is unique (does not conflict with J2SE), is easy to read and looks more like a primative is being used.  The down side is that this is an exception to the normal Sun-recommended naming rules for classes.

In a perfect world, the exact look of (arithmetic and logical) expressions could be maintained if Java supported operator overloading.  Since that is not possible, methods (with intuitive names) will be written to substitute for all operators.  Unwrapping will only be needed in those places where the expression result requires a primative type.  So assignments and many method calls will use the object form returned as an expression result.  However, when expression results are used directly with a Java language construct (e.g. if/else, array subscripts) or in a J2SE-provided method that has primatives in the signature, the primatives must be used.  In these cases, the custom wrapper object will be unwrapped.

Note that because each of these classes will internally track whether the variable is set to the unknown value, all variables will have a valid (non-null) reference.  As long as the conversion code is written with this in mind, null pointer exceptions will be eliminated (from expressions).

Progress function/procedure parameters have input-output and output forms that cause the data in the calling procedure to be changed by the called function/procedure.  This is very similar to the C/C++ semantic of a parameter passed by pointer.  On return, the data pointed to by the calling procedure has been changed.  In Java this is easily handled when some kind of business/container object is passed as a parameter but it is not directly possible with the normal primatives or their standard J2SE wrappers (e.g. Integer) which are immutable.  While some instances of function/procedure calls may end up being generated in such a way that a more abstract object is passed in instead of the primatives, this will require significant refactoring (of procedural code into object oriented code) and may not always be possible.  For a general purpose solution to this problem, it seems better to ensure that all of the custom wrapper classes are designed as mutable, with an assignment-type interface and setters to allow modification of the wrapped data in the called method , where the variable being assigned was defined as an input-output or output parameter.

Detecting Constants

There is no concept of a constant variable in Progress but some variables may essentially be "read-only".  As an optimization (and a good practice for Java code) such variables should be converted as "final static".  To do this, it is required that all assignments, run statements and user functions... are examined to determine if any possible change can ever be made to each variable.  This presents similar issues as the To Wrap or Not To Wrap discussion, however it is likely that this is significantly more feasible to implement.

Format String Support

Variables can have explicit format strings or can implicitly have a format string based on a LIKE clause (where the database field or variable has an explicit format string).  In these cases, the any of the formatted output language statements will not use the default data type format string, but instead will honor the explicit or implicit format string that is specific to the variable.

These cases are detected and a class member will be used for each unique format string in a given class.  The member is created with the following approach:

public static final String FMT_STR_n = "...";

The number n will be a generated sequence number to make the string name unique.

At annotation time, a set (the format string is the key and the name of the class member is the value) of these are accumulated and are used to cross-reference all format string usage.  This allows a single definition to be reused in multiple places.  For this reason, if 2 variables are both explicitly defined with a format string of "x(40)", then the same FMT_STR_x constant will be used for both.

The proper hierarchy for determining the correct format string to use is:
  1. Use an explicit, in-place format string (part of a format phrase on any language statement that generates formatted output).
  2. If the variable is being used as a simple variable reference (rather than an expression or constant):
    1. Use the explicit format string for that variable, if one was specified.
    2. Use the implicit format string that was associated with the variable via a LIKE clause.
  3. Use the data type specific default format.
The formatted I/O processing (all putField variants) check on this (and honors it) for each field that is a simple variable reference.

Operators and Expression Evaluation

All operators are fully implemented except the highly specialized CONTAINS operator (see page 960 of the Progress 4GL Reference, in the section on the record phrase) which handles word index searches.

Each operator is converted to a static Java method call that is implemented in a data type wrapper class ( integer , decimal , date , logical , character ) or in a common operations class such as MathOps or CompareOps .  When an operator is encountered, the left operand (and sometimes the right operand in the case of date operations) is checked for its type.  To do this, the leftmost descendants are checked until a literal, variable, field or function is found.  All of these nodes have a type that determines the type of the resulting operation.  Depending on this type, the conversion strategy is picked.

The structure of the Java expression is an exact duplicate of the Progress source tree structure.  This is required to maintain logical correctness in the result.  The only deviations:
  1. The Progress logical NOT operator has a very low precedence and the Java NOT operator has a high precedence (the same as unary plus and unary minus).  If the real operator is used, the logical not would have to have its operand parenthesized to maintain the same result as in Progress.  If this was not handled, then the Java NOT would bind to the first emitted portion of the operand and this would be evaluated before any subsequent operator.  However, since logical.notOperator() is used instead, the expression subtree will automatically be paranthesized. This is the difference between:
  2. In Progress the equality and inequality operators have the same precedence level as the comparison operators (>, <, >=, <=) but in Java, these are split into 2 precedence levels.
The logic for the operator conversion is encoded in the OperatorConverter class.

Variable References

User Defined Variables

There are 2 forms of variable reference:
  1. A simple reference.
  2. An unwrapped reference.
A simple reference is used when passing a variable to a function or procedure call.  It is also used as the left operand in certain operators and as an lvalue for an assignment.  This is called a simple reference because it is emitted as the variable's object instance name (converted to the proper Java form).

An unwrapped reference is how the variable value is accessed in a Java language contruct.   This is usually a control flow construct like "if" or "switch" where the result of an expression must be unwrapped using booleanValue() or intValue() respectively.  For example, a Progress integer variable named "i" is represented by an integer variable named "i" in Java and when used in expressions requiring direct access to the primative value, it is dynamically unwrapped as "i.intValue()".  This works because Progress has no inline assignment forms inside expressions.  In Java there are many operators that modify and reassign their lvalue (e.g. ++ and -- increment/decrement their lvalue and reassign, the assignment operator itself can be used inside a Java expression and there are many other modify and reassign operators such as += or *=).

Without such assignment operators, expressions essentially access variables in a read-only manner with the exception of user-functions which can have input-output and output parameters.  Such user functions can modify the original variable's value and this is handled using the assign() method of the wrappers.  The method/attribute support has not been investigated fully at this time, but there may be implications for these as well.

Since null is not being used to track the unknown value, all wrappers are always expected to be valid for calling methods and operator replacements.  This eliminates the need to protect variable references using "bracketed" null checks.  This would have otherwise been necessary at the beginning of any expression that made a variable reference and possibly in the middle of expressions to check the return (or the input-output or output) from a function.

Each operator is implemented as methods that are "unknown value aware".   This way the operator will return unknown value or other correct results as needed.   For example, many Progress operators (e.g. + or -) silently return the unknown value if either or both operands are the unknown value.

Global Variables

Progress provides constructs that look like global variables (e.g. opsys, current-language, progress...).  These global variables are termed "built-in functions" by Progress but they act like read-only variables.  In particular, they take no parameters and even take no parenthesis.

A helper class com.goldencode.p2j.util.EnvironmentOps provides static method calls ("getters") to provide the values associated with the global variables.  Note that at this time, the return values are hard coded but eventually they will be backed by lookups from the P2J directory.  With this approach, the values to be passed at runtime can be controlled by the implementer and are not hard coded to a particular customer environment or application.

See also:

EnvironmentOps.java
convert/variable_references.rules

Assignments

All assignments (either "assignment" statements or the ASSIGN language statement) follow these conventions:
  1. Always have an lvalue as the 1st child.
  2. Never can be a declaration/definition of a new variable (this is not possible in Progress).
Since the wrapper approach eliminates the possibility of having a null pointer, all assignments can be made by calling the assign() method of the wrapper.  This is required because the standard Progress semantic is to set the value rather than the Java approach of setting the reference.  This difference can only really be seen in cases where a function or procedure with input parameters is called.  In these cases, the assigning of the value would "bleed through" into the calling procedure.

In a function or procedure where a variable is an input parameter, it can simply/safely be assigned the using the Java assignment operator.

The simple form of the ASSIGN statement is that which just is a list of lvalues, each followed by an assignment operator and an expression of an assignable type.  Such a form is converted by the same rule sets that convert normal ("standalone") assignments.  The more complex forms of the ASSIGN statement which handle the movement of data from input buffers to variables or fields, are not yet supported.

Function/Procedure Definitions

User-defined functions are differentiated from procedures in the following ways:
  1. Functions can (actually MUST) return a scalar instance of any of the basic data types, procedures can only set the return-value "global variable" which is a character value.
  2. Functions are limited in the resources they can define.
  3. Functions can't be referenced before they are defined, though a FORWARD declaration can be made which is just a signature definition.
Both of these constructs are emitted as Java instance methods (METHOD_DEF) for the class that maps to this file.  The function or procedure name is converted to a Java method name.  Any use of the KW_PRIVATE in the Progress source converts to a private access specifier, otherwise the default access is public.  Procedures always return void but functions always have a wrapper data type as a return value.

Both types of definition support input, input-output and output type parameters.  In Java, these parameters are all emitted as a simple variable definitions (of the form: classname reference ) that are comma-separated.  However, the assignment of these parameters is differentiated.  Input parameters have their reference assigned in order to ensure that calling code won't see a change in the parameter after the function/procedure returns.  Input-output and output parameters get their value assigned in order to ensure that the caller sees the change in contained data after return.  All assignments check the type of variable before emitting and use the "parmtype" annotation to make this determination.  The parser ensures that all parameters have this annotation, even if the KW_INPUT is omitted (it is optional in some circumstances).

The concept of an input-output or output function parameter is only implemented for user-defined functions and internal/external procedures.  No built-in Progress functions change the original referenced variable or field passed as a parameter. The only "returned" data from a built-in function appears to be the return value itself.  There may be environmental/database side effects of function calls to built-ins. However, no function parameters are ever modified based on calling a built-in function.

The case of input-output and output parameters is handled using the mutable wrappers where the called function/procedure will modify the contained data such that the caller will see a different value after the call using the same reference.  In this case, the normal assign() method of the wrapper will be called to set the value.  However it is important to note that in the case of an input parameter, the normal Java "=" assignment operator will be used to modify the reference rather than the value (referred to by the reference) since any changes should NOT be visible in the calling code.

Only scalar, table and buffer parameters are allowed.  No array parameters are possible in Progress.  At this time, there is no support for table or buffer parameters.

The user-defined function definition and procedure definition are converted directly to a Java method signature rooted at a METHOD_DEF node which is rooted as a peer with the external procedure's METHOD_DEF subtree.  The text of the METHOD_DEF is the function name (a valid Java function name) and the return type and access specifiers are defined in "rettype" and "access" annotations respectively.  All parameter definitions (if any exist) are rooted at the first child which is an LPARENS.  The next child is a BLOCK that contains the method body.

The creation of these nodes in the target tree is different since the structure of the source tree is different for the two cases.  In particular, the major difference is where the parameter definitions exist.  In the function definition, the parameters are child nodes (PARAMETER) of the FUNCTION and KW_FUNCT (the function definition itself).  In the procedure definition, the parameters are child nodes (DEFINE_PARAMETER) of the BLOCK that is a child to the PROCEDURE but is NOT a child of the KW_PROC which is the actual procedure definition.  Procedure definitions rely upon annotations to note how many parameters there are and then in the conversion step, an LPARENS node is either created (if "numparms" annotation in the PROCEDURE node > 0) or not.  If it is created, its ID is left behind as an annotation called "parmroot" in the PROCEDURE node.  All DEFINE_PARAMETER statements have a "procrefid" annotation that is the ID of the enclosing PROCEDURE node (or the AST root node which is BLOCK and defines the external procedure).  This allows the DEFINE_PARAMETER processing to occur much later than the PROCEDURE definition itself, yet it can reference its way back to the parent node at which the method signature must be rooted.  The order of the DEFINE_PARAMETER statements determines the order of the parameters in the signature (just as in Progress).  The function definition doesn't have this problem as it can emit in-line with the tree walk since the parameters are child nodes of the definition itself.

Input and input-output parameters never get initialized in a procedure or function since they always take the value from the caller.  Output parameters are always initialized.  DEFINE_PARAMETER allows explicit initialization using the KW_INIT, if this construct is found on an output parameter, an in-block (outside the method signature) assignment is made using the provided data.  If no explicit initializer is specified (this is always the case for function parameters since one cannot specifiy KW_INIT on a PARAMETER), then output parameters still get initialized in the block itself.  Procedure parameters get initialized at the same location as the DEFINE_PARAMETER appears in the block.  The default initializer is the same as for DEFINE_VARIABLE.  Function parameters are both more and less troublesome.  All function output parameters are always initialized to the unknown value, rather than following the rules of normal variable defaults.  Since one can never provide an explicit initializer to a function parameter, this means that there is only 1 case.  However, since Java cannot initialize parameters as part of the method signature itself, this initialization must occur in the block that follows the function definition. This is a problem because the PARAMETER nodes are children of the LPARENS and its parent KW_FUNCT but the BLOCK in which they must be attached is the right side sibling of the KW_FUNCT grandparent.  In other words, the BLOCK hasn't been created yet so it cannot be the target for the parameter creation.  To resolve this issue, a BOGUS node is created in place (as a child of the method signature) all parameters are created as children of this node.  They all get the proper IDs and parenting.  Then this node is detached from the tree and stored in a "depot" (see the TemplateWorker ) under the name "funcparms".  After creation of a matching BLOCK node, if the BLOCK is a child of FUNCTION (thus a sibling of KW_FUNCT), then the BOGUS node is retrieved from the depot and attached as the first child of the BLOCK.

See the convert/function_definitions.rules and convert/procedure_definitions.rules for more details.  Please note that the convert/variable_definitions.rules is also heavily involved with processing parameters.

Please also see the section on Control Flow for details on the RUN and RETURN language statements which are necessary to use function and procedures.

Function Calls

The following documents the mapping between built-in functions and their Java equivalents.  References to decimal, integer, logical, date, character, MathOps, and CompareOps refer to classes in package com.goldencode.p2j.util.

Progress Function
Java Equivalent
Category
Token Type
Nullable
Optional Parameters
DBCS Issues
Supported
Notes
_CBIT
character.testBitAt()
bit manipulation
FUNC_LOGICAL
yes
no
yes
yes
logical _cbit(string, integer)

The first parameter is a string with 0 or more characters. 

The second parameter is the bit position to test.  The input string is treated as a multibyte bitfield. The first character of the string corresponds to bit positions 0-7 (if it is non-DBCS and presumably in a DBCS character the positions would be 0-15).  In a non-DBCS charset, the second char would correspond to bits 8-15.

The return is always false if the empty string is input.  For any string of > 0 length, the return is true if the bit in the specified position is 1 and false if it is 0.

Sample Code:

def var i as int.
def var j as int.
def var b as char.
def var c as char.

do i = -1 to 257.
   b = "".
   c = chr(i).
   do j = 7 to 0 by -1.
      if _cbit(c, j)
         then b = b + "1".

         else b = b + "0".
   end.
   message i " = " b.
end.
ABSOLUTE
MathOps.abs(integer)
or
MathOps.abs(decimal)
math
FUNC_POLY yes
no
no
yes
returns INT or DEC depending on the type of the input
ACCUM
Accumulator (abstract base class);  concrete implementations:
  • AverageAccumulator
  • CountAccumulator
  • MaximumAccumulator
  • MinimumAccumulator
  • TotalAccumulator
database
FUNC_POLY no


yes
Resides in the com.goldencode.p2j.util package.

Each concrete subclass defines the following methods for results retrieval;  each class' implementations of these methods may return different data types, so these method signatures are named by convention only, and are not enforced by a common interface:
  • getResult() to retrieve the cumulative result
  • getResult(Resolvable breakGroupKey) to retrieve the current break group result
ALIAS
ConnectionManager.alias() database FUNC_CHAR
yes


yes

AMBIGUOUS RecordBuffer.wasAmbiguous() database
FUNC_LOGICAL no


yes

ASC
character.asc()
type conversion
FUNC_INT yes
yes
yes
yes
No source or target codepage support at this time.  Result in a DBCS environment may vary from the Progress implementation.
AVAILABLE
RecordBuffer.available()
database
FUNC_LOGICAL no


yes

CAN-DO
character.matchList()
security
FUNC_LOGICAL yes


yes

CAN-FIND RandomAccessQuery.hasFirst()
RandomAccessQuery.hasNext()
RandomAccessQuery.hasPrevious()
RandomAccessQuery.hasLast()
RandomAccessQuery.hasAny()
database
FUNC_LOGICAL no


yes

CAN-QUERY

UI
FUNC_LOGICAL yes


no

CAN-SET

UI
FUNC_LOGICAL yes


no

CAPS
character.toUpperCase()
string
FUNC_CHAR
yes
no
partial yes
It can contain DBCS but only the SBCS chars are uppercased.
CHR
character.chr()
type conversion
FUNC_CHAR
yes
yes
yes
yes
No source or target codepage support at this time.  Result in a DBCS environment may vary from the Progress implementation.
CODEPAGE-CONVERT

type conversion
FUNC_CHAR
yes
yes
yes
no

COMPARE
character.compare()
string
FUNC_LOGICAL
yes
yes
yes
yes
No collation table support at this time.  Result in a DBCS environment may vary from the Progress implementation.
CONNECTED ConnectionManager.connected() database FUNC_LOGICAL no no
yes
COUNT-OF
P2JQuery.size()
database
FUNC_INT
no
no

yes
Implemented by concrete query classes.
CURRENT-CHANGED

database
FUNC_LOGICAL
no
no

no

CURRENT-LANGUAGE
EnvironmentOps.getCurrentLanguage() I18N
VAR_CHAR
no
no

yes

CURRENT-RESULT-ROW
AbstractQuery.currentRow()
database
FUNC_INT
yes
no

yes
Implemented by AbstractQuery and overridden where necessary by concrete query implementations.
CURRENT-VALUE
database
FUNC_INT
yes
yes

no

DATASERVERS EnvironmentOps.getDataServerList() database
VAR_CHAR
no
no

early

DATE
see constructors for date class
date
FUNC_DATE
yes
no
no
yes

DAY date.day() which calls instance method date.getDayNum()
date
FUNC_INT
yes
no
no
yes

DBCODEPAGE
I18N
FUNC_CHAR
yes
yes

no

DBCOLLATION
I18N
FUNC_CHAR
yes
yes

no

DBNAME
EnvironmentOps.getCurrentDatabaseName() database
VAR_CHAR
yes
no

yes

DBPARAM
database
FUNC_CHAR yes
yes

no

DBRESTRICTIONS ConnectionManager.dbRestrictions() database
FUNC_CHAR yes
yes

yes

DBTASKID
database
FUNC_INT
yes
yes

no

DBTYPE ConnectionManager.dbType() database
FUNC_CHAR
yes
yes

yes

DBVERSION
database
FUNC_CHAR
yes
yes

no

DECIMAL
see constructors for decimal class
type conversion
FUNC_DEC
yes
no
no
yes

DYNAMIC-FUNCTION
function execution
FUNC_POLY yes
yes

no

ENCODE
SecurityOps.encode()
security
FUNC_CHAR
yes
no

early
The interface is completely supported and functional, however the calculated result is NOT compatible with the 4GL version.
ENTERED GenericWidget.isEntered()
UI
FUNC_LOGICAL
no
no

yes

ENTRY
character.entry()
string
FUNC_CHAR
yes
yes
yes
yes
Delimiter processing is *always* case-sensitive at this time.
ETIME date.elapsed()
time
FUNC_INT
no
yes

yes

EXP
MathOps.pow()
math
FUNC_DEC
yes
no

yes

EXTENT 0 if the variable or field is scalar
OR
array_var.length
arrays
FUNC_INT
no
no

yes
This is a direct conversion without any backing method or class.
FILL
character.fill() string
FUNC_CHAR
yes
no

yes

FIRST PresortQuery.isFirst() loops/transactions FUNC_LOGICAL
no
no

yes

FIRST-OF PresortQuery.isFirstOfGroup() loops/transactions FUNC_LOGICAL no
no

yes

FRAME-COL CommonFrame.frameCol()
UI
FUNC_DEC
no
yes

yes

FRAME-DB

UI
VAR_CHAR
no
no

no

FRAME-DOWN CommonFrame.frameDown() UI
FUNC_INT
no
yes

yes

FRAME-FIELD
LogicalTerminal.getFrameField() UI
VAR_CHAR no
no

yes

FRAME-FILE

UI
VAR_CHAR no
no

no

FRAME-INDEX LogicalTerminal.getFrameIndex() UI
VAR_INT
no
no

yes

FRAME-LINE CommonFrame.frameLine() UI
FUNC_INT no
yes

yes

FRAME-NAME

UI
VAR_CHAR
no
no



FRAME-ROW
CommonFrame.frameRow() UI
FUNC_DEC
no
yes

yes

FRAME-VALUE LogicalTerminal.getFrameValue()
LogicalTerminal.setFrameValue()
UI
VAR_CHAR no
no

yes

GATEWAYS
EnvironmentOps.getDataServerList()
database
FUNC_CHAR
no
no

early

GET-BITS

bit manipulation FUNC_INT
yes
no

no

GET-BYTE
BinaryData.getByte()
raw/memory access
FUNC_INT
yes
no

yes

GET-BYTE-ORDER

raw/memory access FUNC_INT
no
no

no

GET-BYTES

raw/memory access FUNC_POLY
yes
no

no
Returns RAW or MEMPTR.
GET-CODEPAGES

I18N
VAR_CHAR
no
no

no

GET-COLLATIONS

I18N
FUNC_CHAR
yes
no

no

GET-DOUBLE

raw/memory access FUNC_DEC
yes
no

no

GET-FLOAT

raw/memory access FUNC_DEC
yes
no

no

GET-LONG

raw/memory access FUNC_INT
yes
no

no

GET-POINTER-VALUE

raw/memory access FUNC_INT no
no

no

GET-SHORT

raw/memory access FUNC_INT
yes
no

no

GET-SIZE
memptr.length() raw/memory access FUNC_INT
no
no

yes

GET-STRING
BinaryData.getString() raw/memory access FUNC_CHAR
yes
yes

yes

GET-UNSIGNED-SHORT

raw/memory access FUNC_INT
yes
no

no

GO-PENDING LogicalTerminal.isGoPending() UI
VAR_LOGICAL
no
no

yes

IF THEN ELSE
condition ? expr1 : expr2
ternary
FUNC_POLY
yes
no

yes
Returns same type as the THEN and ELSE expressions evaluate to.
INDEX
character.indexOf()
string
FUNC_INT
yes
yes

yes

INPUT
Converts to a field-level getter in a custom frame interface OR to the CommonFrame.getScreenValue(widget) method if this is a character type override case (see notes).
UI
FUNC_POLY
no
yes

yes
The INPUT built-in function normally returns the same type as the single parameter (which must be an lvalue which is found in a frame that is in scope).  However, this function can be used in an expression that requires a character data type even in the case where the parameter is NOT of the character type!  This is like an override option and it must be detected from the SURROUNDING expression (above the FUNC_POLY node). For example,an direct assignment of the INPUT function's return type to a character lvalue is honored.  This case works differently than an assignment to a non-character type in the case that the value in the screen buffer is uninitialized (it will return the empty string like screen-value instead of returning the default value of the operand type).  This override support is implemented except for one case: the usage of INPUT as a parameter to another built-in function (or presumably a method) which expects a character parameter should also cause this override. At this time there is no support for this case. The resulting type (if not already character) will cause the wrong type to be emitted.  Note that this only is an issue for built-in functions (and methods).  If the parent is a user defined FUNC_* then the result is the same as the operand, no override occurs.  But a built-in function that takes a character type will get the result as a character instead of the operand type.

See the SCREEN-VALUE attribute which is a related issue.
INTEGER
see constructors for integer class
type conversion
FUNC_INT yes
no

yes

IS-ATTR-SPACE

UI
FUNC_LOGICAL yes
no

no

IS-LEAD-BYTE

I18N
FUNC_LOGICAL yes
no

no

KBLABEL
LogicalTerminal.kbLabel() UI
FUNC_CHAR
yes
no

yes

KEYCODE LogicalTerminal.keyCode() UI
FUNC_INT
yes
no

yes

KEYFUNCTION LogicalTerminal.keyFunction() UI
FUNC_CHAR
yes
no

yes

KEYLABEL LogicalTerminal.keyLabel()
UI
FUNC_CHAR
yes
no

yes

KEYWORD

Progress
FUNC_CHAR yes
no

no

KEYWORD-ALL
Progress FUNC_CHAR yes
no

no

LAST
PresortQuery.isLast() loops/transactions FUNC_LOGICAL no
no

yes

LAST-OF
PresortQuery.isLastOfGroup() loops/transactions FUNC_LOGICAL no
no

yes

LASTKEY
KeyReader.lastKey()
UI
VAR_INT
no
no

yes

LC
character.toLowerCase()
string
FUNC_CHAR
yes
no
partial
yes
It can contain DBCS but only the SBCS chars are lowercased.
LDBNAME
ConnectionManager.ldbName() database
FUNC_CHAR yes
yes

yes

LEFT-TRIM
character.leftTrim() string
FUNC_CHAR
yes
yes
yes
yes

LENGTH
character.length()
character.byteLength()
BinaryData.length()
raw/memory access
FUNC_INT
yes
yes

yes
No support for "column" based length at this time.
LIBRARY

R-code library FUNC_CHAR
yes
no

no

LINE-COUNTER
Stream.getNextLineNum()
I/O
FUNC_INT
no
yes

yes

LIST-EVENTS

UI
FUNC_CHAR yes
yes

no

LIST-QUERY-ATTRS

UI
FUNC_CHAR yes
no

no

LIST-SET-ATTRS

UI
FUNC_CHAR yes
no

no

LIST-WIDGETS

UI
FUNC_CHAR yes
yes

no

LOCKED
RecordBuffer.wasLocked() database
FUNC_LOGICAL
no
no

yes

LOG
MathOps.log()
math
FUNC_DEC
yes
yes

yes

LOOKUP character.lookup()
string
FUNC_INT
yes
yes
yes
yes
Delimiter processing is *always* case-sensitive at this time.
MAXIMUM
character.maximum()
date.maximum()
logical.maximum()
integer.maximum()
decimal.maximum()
math
FUNC_POLY
yes
yes

yes
Warning: variable args!  Numeric operands are widened (int to double) if there are a mixture.  All arguments (other than numerics) can only be compared against others of the same type.
MEMBER

R-code library
FUNC_CHAR
yes
no



MESSAGE-LINES
LogicalTerminal.getMessageLines() UI
VAR_INT
no
no

yes

MINIMUM
character.minimum()
date.minimum()
logical.minimum()
integer.minimum()
decimal.minimum()
math
FUNC_POLY yes
yes

yes
Warning: variable args!  Numeric operands are widened (int to double) if there are a mixture.  All arguments (other than numerics) can only be compared against others of the same type.
MONTH date.month() which calls instance method date.getMonthNum()
date
FUNC_INT
yes
no

yes

NEW
RecordBuffer.isNew() database
FUNC_LOGICAL
no
no

yes

NEXT-VALUE
database
FUNC_INT
yes
yes

no

NOT ENTERED
GenericWidget.isNotEntered()
UI
FUNC_LOGICAL no
yes

yes

NUM-ALIASES
ConnectionManager.numAliases()
database VAR_INT
no
no

yes

NUM-DBS
ConnectionManager.numDBs() database VAR_INT
no
no

yes

NUM-ENTRIES
character.numEntries()
character.numEntriesOf()
string
FUNC_INT
yes
yes
yes
yes
Delimiter processing is *always* case-sensitive at this time.
NUM-RESULTS P2JQuery.size()
database FUNC_INT
yes
no

yes
Implemented by concrete query classes.
OPSYS
EnvironmentOps.getOperatingSystem() OS environment VAR_CHAR
no
no

yes

OS-DRIVES
FileSystemOps.getRootList()
OS environment VAR_CHAR
no
no

yes

OS-ERROR
FileSystemOps.getLastError() OS environment VAR_INT
no
no

yes

OS-GETENV FileSystemOps.getProperty() OS environment
FUNC_CHAR
yes
no

yes

PAGE-NUMBER Stream.getPageNum()
I/O
FUNC_INT no
yes

yes

PAGE-SIZE Stream.getPageSize()
I/O
FUNC_INT
no
yes

yes

PDBNAME ConnectionManager.pdbName() database
FUNC_CHAR
yes
yes

yes

PROC-HANDLE

Progress environment VAR_INT
no
no

no
Is this actually polymorphic?
PROC-STATUS

Progress environment VAR_INT
no
no

no

PROGRAM-NAME EnvironmentOps.getSourceName() + a constant (public static final String progressSourceName) in each generated class.
Progress environment FUNC_CHAR
yes
no

yes
Warning: there are 2 flaws in this implementation.
  1. Progress returns the same filename as is used on the RUN statement (so the name can change from invocation to invocation).  Our version has a static string for the base name of the file (no pathing).
  2. It is possible that due to refactoring, code that calls program-name might have dependencies on a source file name that is not the source file name of the current class. Such cases might require a class and method specific mapping to a source file name (some kind of registry database).
PROGRESS
EnvironmentOps.getRuntimeType() Progress environment VAR_CHAR
no
no

yes

PROMSGS EnvironmentOps.getMessageSource() Progress environment VAR_CHAR no
no

yes

PROPATH EnvironmentOps.getSourcePath() Progress environment VAR_CHAR no
no

yes

PROVERSION EnvironmentOps.getVersion()
Progress environment VAR_CHAR no
no

yes

QUERY-OFF-END
P2JQuery.isOffEnd() database
FUNC_LOGICAL
yes
no

yes
Implemented by concrete query classes.
R-INDEX
character.lastIndexOf()
string
FUNC_INT
yes
yes

yes

RANDOM
MathOps.random()
math
FUNC_INT
yes
no

yes
Warning: it is possible to force Progress to always generate the same sequence of random numbers in every session! (This sounds like an awful idea but perhaps it is being used.)
RAW
database
FUNC_RAW
no
yes

no

RECID
RecordBuffer.recordID database FUNC_RECID
yes
no

yes
The type is mapped to integer.
RECORD-LENGTH

database
FUNC_INT
no
no

no

REPLACE
character.replaceAll()
string
FUNC_CHAR
yes
no

yes

RETRY
TransactionManager.isRetry()
loops/transactions
VAR_LOGICAL
no
no

yes

RETURN-VALUE
ControlFlowOps.getReturnValue()
procedure execution
VAR_CHAR
no
no

yes

RGB-VALUE

UI
FUNC_INT
yes
no

no

RIGHT-TRIM
character.rightTrim()
string
FUNC_CHAR
yes
yes

yes

ROUND
Math.round()
math
FUNC_DEC
yes
no

yes

ROWID
RecordBuffer.recordID database FUNC_ROWID
yes
no

yes
The type is mapped to integer.
SCREEN-LINES
LogicalTerminal.getScreenLines()
UI
VAR_INT
no
no

yes

SDBNAME
ConnectionManager.sdbName()
database FUNC_CHAR
yes
yes

yes

SEARCH
FileSystemOps.searchPath()
filesystem
FUNC_CHAR yes
no

yes

SEEK
Stream.getPosition()
filesystem
FUNC_INT
yes
yes

yes

SETUSERID

security
FUNC_LOGICAL
yes
yes

no

SQRT
MathOps.sqrt()
math
FUNC_DEC
yes
no

yes

STRING
character.valueOf() which uses a  wrapper-specific toString()
type conversion
FUNC_CHAR
yes
yes

yes

SUBSTITUTE
character.substitute()
string
FUNC_CHAR yes
yes
yes
yes
The variable length argument list is handled by an argument of type Object[].
SUBSTRING
character.substring()
string
FUNC_CHAR
yes
yes
yes
yes
No support for "raw", "fixed" or "column" based substrings at this time.
SUPER
function execution
FUNC_POLY
yes
yes

no

TERMINAL
LogicalTerminal.getTerminal()
UI
VAR_CHAR
no
no

yes

TIME
date.secondsSinceMidnight()
time
VAR_INT
no
no

yes

TODAY
new date()
date
VAR_DATE
no
no

yes

TO-ROWID

database FUNC_ROWID
yes
no

no

TRANSACTION
TransactionManager.isTransactionActive() loops/transactions
VAR_LOGICAL
no
no

yes

TRIM
character.trim()
string
FUNC_CHAR
yes
yes

yes

TRUNCATE
MathOps.truncate() math
FUNC_DEC
yes
no

yes

USERID
SecurityOps.getUserId() security
FUNC_CHAR
no
yes

yes

VALID-EVENT

UI
FUNC_LOGICAL
yes
yes

no

VALID-HANDLE
handle.isValid()
data types
FUNC_LOGICAL no
no

yes

WEEKDAY
date.weekday() which calls instance method date.getWeekDayNum()
date
FUNC_INT
yes
no

yes

WIDGET-HANDLE
handle.fromString()
UI
FUNC_HANDLE
yes
no

yes

YEAR date.year() which calls instance method date.getYearNum() date
FUNC_INT
yes
no

yes


See also:

rules/convert/builtin_functions.rules

Methods and Attributes

Progress 4GL provides an object-like construct with its HANDLE data type.  WIDGETs, QUERYs and other resources can be represented by a handle.  Using this handle one can access pre-defined attributes or call methods (invoke behavior).  These attributes and methods are specific to this resource type, they are built into the environment and cannot be extended by the user/programmer.

There are 3 possible forms of conversion:
  1. Special purpose accessors (e.g. SYSTEM HANDLES such as CURRENT-WINDOW) can be naturally converted to a static method call which accesses an object instance that is the current value.  The ATTRIBUTE or METHOD node itself can be directly translated to an instance method call made upon the returned object reference.
  2. Any singleton resource (e.g. SYSTEM HANDLES such as ERROR-STATUS) can be easily and naturally converted to a static method call.  The ATTRIBUTE or METHOD node itself can be directly translated to the static method call.  Note that for attributes, the getter methods will need to be backed by a context-local member.
  3. Instance resources (e.g. a WIDGET handle) act like instance members (accessed via a getter method call) or methods.  In this case the COLON patent node should be the peer node at which the method is converted.  Then the HANDLE will need to emit as an object REFERENCE of the proper class.
The following summarizes the method and attribute support that is implemented:

Progress Feature
Java Replacement
Type
Method or Attribute
Supported
Notes
DCOLOR
GenericWidget.getDcolor()
GenericWidget.setDcolor()
widget
ATTRIBUTE Yes

DESELECT-ROWS
GenericWidget.deselectRows()
browse
METHOD Yes

ENTRY
GenericWidget.entry() combo-box, selection list
METHOD
Yes

ERROR
ErrorManager.isError()
ERROR-STATUS system handle
ATTRIBUTE Yes

FETCH-SELECTED-ROW
GenericWidget.fetchSelectedRow() browse
METHOD Yes

FILE-NAME FileSystemOps.initFileInfo() to set
FileSystemOps.fileInfoGetName()  to get
FILE-INFO system handle ATTRIBUTE Yes
FILE-SIZE FileSystemOps.fileInfoGetSize()  to get FILE-INFO system handle ATTRIBUTE Yes
GET-NUMBER
Manager.getErrorNum() ERROR-STATUS system handle METHOD
Yes

GET-MESSAGE
ErrorManager.getErrorText() ERROR-STATUS system handle METHOD Yes

HANDLE
GenericWidget.getHandle()
widget
ATTRIBUTE Yes Should this really be converted into a widget object reference?
IS-SELECTED
GenericWidget.isSelected() combo-box, selection list METHOD Yes
LIST-ITEM-PAIRS
GenericWidget.getListItems()
GenericWidget.setListItems()
combo-box, selection list ATTRIBUTE Yes
LOOKUP
GenericWidget.lookup() combo-box, selection list METHOD Yes
NEXT-TAB-ITEM
GenericWidget.getNextTabItem()
GenericWidget.setNextTabItem()
widget
ATTRIBUTE Yes
NUM-MESSAGES
ErrorManager.numErrors() ERROR-STATUS system handle ATTRIBUTE
Yes

NUM-SELECTED-ROWS
GenericWidget.getNumSelectedRows()
GenericWidget.setNumSelectedRows()
browse ATTRIBUTE Yes
PARENT
GenericWidget.getParent()
GenericWidget.setParent()
frame
ATTRIBUTE Yes
PERSISTENT
false
THIS-PROCEDURE system handle
ATTRIBUTE
Yes
Temporary solution to satisfy the current application project for which we always know the result will be false.  A real implementation will eventually need to be made.
PFCOLOR
GenericWidget.getPfColor()
GenericWidget.setPfColor()
widget
ATTRIBUTE Yes
PRIVATE-DATA
GenericWidget.getPrivateData()
GenericWidget.setPrivateData()
frame
ATTRIBUTE Yes
READ-ONLY
GenericWidget.isReadOnly()
GenericWidget.setReadOnly()
widget
ATTRIBUTE
Yes

REFRESH
GenericWidget.refresh()
browse
METHOD Yes
SCROLLABLE
GenericWidget.isScrollable()
GenericWidget.setScrollable()
widget
ATTRIBUTE Yes
SCREEN-VALUE
CommonFrame.getScreenValue(widget) or widget setter method from the current frame interface.
widget
ATTRIBUTE
Yes
This duality is needed since the screen-value attribute will always return a character type (the frame interface getters have the same type as the data being represented/edited).  In the case where the "natural" data type is not character, this will behave differently when the screen buffer's version of this data is uninitialized.  In particular, the data will be returned as the empty string rather than as the default type for the natural data type.  So if widget i is an integer and it has never been initialized (copied into the screen buffer as a result of a display a screen-value setter call or via UI editing) then it will return "" instead of 0.

See the INPUT built-in function which has a similar issue.
SELECT-ALL
GenericWidget.selectAll()
browse
METHOD Yes
SELECT-FOCUSED-ROW
GenericWidget.selectFocusedRow()
browse
METHOD Yes
SELECTED
GenericWidget.isSelected()
GenericWidget.setSelected()
widget
ATTRIBUTE Yes
SENSITIVE
GenericWidget.isSensitive()
GenericWidget.setSensitive()
widget
ATTRIBUTE Yes
SET-REPOSITIONED-ROW
GenericWidget.setRepositionedRow()
browse
METHOD Yes
TITLE
GenericWidget.getTitle()
GenericWidget.setTitle()
frame
ATTRIBUTE Yes
VISIBLE
GenericWidget.isVisible()
GenericWidget.setVisible()
widget
ATTRIBUTE Yes

Instance oriented system handles:

Handle
Java Equivalent
CURRENT-WINDOW
LogicalTerminal.currentWindow()
ACTIVE-WINDOW
LogicalTerminal.activeWindow()
THIS-PROCEDURE
none at this time
FOCUS
LogicalTerminal.focus()
SESSION
none at this time
SELF LogicalTerminal.self()


See also:

rules/convert/methods_attributes.rules
rules/convert/assignments.rules (setters are implemented here)
rules/convert/variable_references.rules (for handle and system handle conversion)

Control Flow

Some implementation notes:
  1. IF/THEN/ELSE is completely supported as Java if/else.
  2. The IF function converts to the Java ternary operator.
  3. CASE statements with integral data types properly convert to the Java switch when using NUM_LITERALs for all WHEN clauses.
  4. CASE statements that have non-constant integrals (expressions rather than literals) and/or non-integer (character, date, decimal, logical) expressions properly convert to an if/else if/else if/else form.
  5. A LEAVE in a CASE statement breaks out of the closest enclosing non-DO block.  This is handled by using an existing label or manufacturing a new label (if one doesn't already exist) to identify the correct block.  Such changes are handled in the annotations processing, and the manufactured label and new label reference should be written into the source AST.  This means that an implicit behavior is made explicit and no changes to conversion code are needed.
  6. Simple DO blocks convert to the Java { }.
  7. Simple REPEAT blocks convert to the Java while (true) { }.
  8. DO/REPEAT/FOR with a WHILE expression converts to a Java while (expression) { }.
  9. DO/REPEAT/FOR with a var = expr1 TO expr2 converts to a Java for (var = expr1 ; var comparison expr2 ; var += increment) { }.
  10. Labels are supported (they are attached to the beginning of a block, exactly as in Java).
  11. LEAVE
  12. NEXT and NEXT label are translated directly to continue or continue label, except in the case where it is used in a non-iterating block in which case it is translated into a continue on the nearest enclosing iterating block.  If there is no such enclosing iterating block, it is treated as a LEAVE.
  13. STOP is implemented as a "throw new StopConditionException()".
  14. QUIT is implemented as "throw new QuitConditionException()".
  15. PAUSE is implemented as ControlFlow.pause(...).
  16. RETURN when used in a function, returns the result of the associated expression.
  17. RETURN when used in a procedure, if it has an expression, saves the evaluated result into a context-local runtime variable via the ControlFlowOps.setReturnValue() before emitting a Java return statement.  If there is no expression, then the setReturnValue() is assigned the empty string.  Whenever Progress code references RETURN-VALUE, this results in a call to ControlFlowOps.getReturnValue() which returns the current context-local value of this character variable.
  18. RETURN ERROR is implemented as a special ignore flag in the TransactionManager and "throw new ErrorConditionException()" where this exception is ignored until it reaches the caller of the top level block in the current method.
  19. RETURN NO-APPLY is implemented by calling LogicalTerminal.consumeEvent() before returning.
  20. RUN for internal procedures is implemented as a method call to the referenced instance member in the same class.  Any specified parameters are emitted as method arguments.
  21. RUN for external procedures is implemented as a method call to the execute() method of the Java class name generated from the file portion of the pathname text in the FILENAME child node.  First the class name and a unique Java variable name are generated.  Then an instance of the class is defined and initialized (using a default constructor).  Finally, the method call to the execute() method is emitted in such a manner as to ensure that any parameters are emitted as children of the method call node.
  22. RUN VALUE() is a variation on the procedure execution.   Both external and internal procedures can be called via this mechanism. The difference is that the procedure name is built dynamically (at runtime).
See also:

rules/annotations/annotations.xml
rules/annotations/block_properties.rules
rules/convert/control_flow.rules

Common Language Statements

Assignment Type Language Statements

The following language statements are handled with special processing:
  1. length (BinaryData.length)
  2. overlay (character.overlay)
  3. put-byte (BinaryData.setByte)
  4. put-string (BinaryData.setString)
  5. set-size (memptr.length)
  6. substring (character.replace)
All of these are "assignment type" language statements.  That is, language statements that are structured like an assignment of an expression result to a function call.  This required that the annotations processing rewrite the source tree to look more like a proper method call with the first parameter being the object reference.  For example, the assigned expression is inserted as the 2nd child of the KW_SUBSTR or KW_OVERLAY and the ASSIGN node is removed.

Once rewritten, the convert/language_statements.rules rule set handles a simple mapping of overlay to the proper instance method mapping.The result is designed as an instance method because the result must be assigned back to the value of the variable specified as the 1st child. This matches the method call semantic exactly.

Process Execution

External command execution is implemented in com.goldencode.p2j.util.ProcessOps.java.  The signature used is as follows:

void launch(String[] cmdlist, boolean silent, boolean wait)

This handles launching a child process in response to these language statements:
The COMMAND_TOKENS child is converted into an anonymous string array initializer and all children are converted into string literals or a string expression (the value() construct).  In this manner, the first parameter of the method call launch() simultaneously specifies the program to be executed and provides all command line arguments.

The 2nd and 3rd parameters are two flags which are generated based on optional (and mutually exclusive) keywords KW_SILENT and KW_NO_WAIT.  For this reason, in Progress there are 3 possible modes which can be encountered:

Options
silent flag
wait flag
none (the default)
false
true
KW_SILENT
true
true
KW_NO_WAIT
false
false

The "silent and no wait" mode doesn't exist in Progress.  All of these modes are properly supported using these flags.  In addition, the FileSystemOps.lastError (OS-ERROR) value is forced to 0 which is how Progress handles this value.

In Progress, one can launch an interactive child process with the NO-WAIT option.  However, this seems to invariably disable the terminal and neither the child process nor the Progress session can properly run.  For this reason, this mode is not supported nor is it expected to be in use in the field.

Command Text Parsing

The lexer and parser have been modified to change how the COMMAND_TOKENS children are created.  In particular, the lexer may deliver tokens broken into a form that does not match the whitespace delimited words that the author might have intended.  For example: the "word" (used in the shell to redirect STDERR to the same handle as STDOUT):

&2>1

would be lexed as 4 tokens:

UNKNOWN_TOKEN
NUM_LITERAL
GT
NUM_LITERAL

The lexer has been changed to return "hidden" tokens that represent whitespace.  During COMMAND_TOKENS processing, the parser checks these hidden tokens and marks the COMMAND_TOKENS children if they need to be merged back together or not.  Annotation processing then handles this merge as needed.  The result is the same set of commands as executed in Progress 4GL.

See Also

Please see the following rulesets for the implementation:

convert/process_launch.rules
convert/expressions.rules (this handles the VALUE() construct)
convert/input_output.rules

These classes back the runtime functions:

com.goldencode.p2j.util.Launcher.java (remote interface definition)
com.goldencode.p2j.util.ProcessOps.java (server side)
com.goldencode.p2j.util.ProcessDaemon.java (real process launcher)
com.goldencode.p2j.util.ProcessStream.java

The majority of the processing of INPUT_THRU, OUTPUT_THRU and INPUT_OUTPUT_THRU is implemented in Streams and I/O however, the process launching and stream connection is handled by the classes and rulesets noted above.

Limitations
  1. Remote I/O and Terminal Integration
  2. The option KW_NO_CONS (no-console) is not supported yet (it may be Windows-only).
  3. For some or all of these language statements, the process launched is really a shell which then executes the command line provided.  At this time, the shell is hard coded to "sh" but needs to be moved into the directory or provided as a conversion time hint that is passed to a modified version of launch.

OS-Independent Commands (File System)

The OS independent commands for manipulating the file system are implemented in com.goldencode.p2j.util.FileSystemOps.java.

The following language statements are supported:
Please see the following rulesets for the implementation:

convert/language_statements.rules
convert/literals.rules (handles FILENAME)
convert/expressions.rules (this handles the VALUE() construct)

Streams and I/O

The P2J project provides significant support for the Progress I/O processing and streams features. 

Summary

The following language statements which allow the creation, opening and closing of streams are supported:
The following language statements and functions which provide stream reading/writing (including formatted reads/writes) are supported:
Stream Types

Stream Type
Java Class
Input
Output
Notes
File or Device
com.goldencode.p2j.util.FileStream
Y
Y

Printer
com.goldencode.p2j.util.PrinterStream N
Y
On Unix, this will probably be implemented as a pipe to a child process of "lp".  As this type may only be needed on Windows, it isn't implemented at this time.
Directory
com.goldencode.p2j.util.DirectoryStream
(not yet implemented)
Y
N
Used for reading directory listings. Not created at this time since it is not used in the application.
Terminal
?
Y
Y
How this is to be implemented is TBD.
Clipboard
?
?
Y
Windows-only, not implemented at this time.  Note that via the CLIPBOARD system handle, reading the clipboard may be possible, though this has not been investigated.
Child Process
com.goldencode.p2j.util.PipeStream Y
Y
In Progress the same child process stream can be created as both input and output (simultaneously), however the Java class will only handle either input OR output.  Thus the converted code will have 2 streams created in this case and all operations will need to be properly destined for the correct stream.


Basic Design

A DEFINE STREAM corresponds to a local variable of type com.goldencode.p2j.util.Stream.  This class implements an API that provides the complete Progress semantics for all forms of supported I/O processing.  This is an abstract class and concrete subclasses actually provide the real implementation for each type of stream.  Note that the design is such that the vast majority of code is implemented in Stream and the subclasses only have to provide the minimum function to implement the specific resource being used.  This lets the same behavior be obtained from all types of streams and makes it easy to add more stream types with a small incremental effort.

Generally a stream is implemented as either an input OR an output but not as both.  The sole exception to this is the case of using a named stream in the INPUT-OUTPUT THROUGH, where the same stream name is used for both the input and output pipes (the complexity of this is managed inside com.goldencode.p2j.util.ProcessStream and in the rule-sets that convert the code..

There are 2 unnamed streams ("unnamed input" and "unnamed output") which are implicitly used when a named stream is not explicitly present in the source file.  These streams are stored as context local instances that are accessible throughout a user's session via the UnnamedStreams.input()  and UnnamedStreams.output() methods.  Language statements that operate differently based on whether the unnamed input or output is redirected use the UnnamedStreams class to detect these cases and access the stream(s).  See below .

For each named stream that is defined (and for the 2 unnamed streams), an instance of a RemoteStream is instantiated using a static method in StreamFactory .  This reference is named with a converted Java name or is accessed "anonymously" via UnnamedStreams.  Each RemoteStream will actually reference a remote concrete Stream sub-class such as FileStream or ProcessStream which operates on the client side of the system .

A concrete Stream subclass is instantiated and assigned to the named or unnamed stream instances whenever an INPUT FROM, OUTPUT TO, INPUT-THRU, OUTPUT-THRU or INPUT-OUTPUT THRU language statement is encountered.  The type of requested resource determines which class is instantiated.  This constructor opens the backing resource on the client.  At this point the stream (RemoteStream actually) reference is valid for use.

When a file is opened for output, any existing version of that file is removed and a 0 length file is created (unless APPEND is specified). That file is closed when the next stream is opened or by any explicit INPUT CLOSE, OUTPUT CLOSE or INPUT-OUTPUT CLOSE.

When a file is opened for input, the read pointer is located at the first byte of the file.

The STDIO pipes from a child process are fully supported.  The common process launching infrastructure is used to start the child process but a special ProcessStream class is used to implement the proper user-driven reading/writing support for these pipes.  All reads are done from a "combined" STDOUT and STDERR, where output will first be read from STDOUT and then from STDERR.  Due to the "combining" semantic of Progress, a polling method is used to perform I/O without blocking on one of the pipes (thus ignoring input from the other pipe).  In addition, an extra thread is used to wait for the termination of the child process.  This allows these polling loops to exit when the pipes are empty and the child process has exited, which eliminates infinite read blocking.

All of the formatted and unformatted reading and writing is provided by a common set of methods in Stream:
These handle the Progress semantics while actually reading/writing using a smaller set of very simple workers that are implemented in the subclasses.

The Stream class also implements the buffering of reads/writes, and all the line number, paging and column support.

Stream processing generates the EndConditionException on EOF (ENDKEY) and ErrorConditionException for ERROR condition generation.  This processing honors the NO-ERROR construct using the silent error mode in the ErrorManager class.

Implicit Close Behavior

Streams are automatically closed streams on exit from the scope in which they are defined, except for global streams which are closed on end of the session. The TransactionManager registerFinalizable() or registerTopLevelFinalizable() is used to obtain this support.  Global streams are registered with the global flag set true .  This registraton code is emitted into the client application, just after the reference is initialized.  Shared streams that are not "NEW" are just imported.  For this reason, they have already been registered for implcit close support in a previous scope.  This table summarizes the states in which registration occurs:

DEFINE STREAM Type
Registration
NEW SHARED
TransactionManager.registerTopLevelFinalizable(this, true)
NEW GLOBAL SHARED TransactionManager.registerFinalizable(this, true)
SHARED
n/a
(not shared at all)
TransactionManager.registerTopLevelFinalizable(this, true)

At each top level scope (trigger, external or internal procedure), the current unnamed input and output streams are "remembered" as the defaults for that scope.  Any close that occurs for the unnamed input or output streams which is not the result of assigning a new stream, causes the unnamed stream to be restored back to the default for that scope.  Since the Progress programmer can change the unnamed streams but not query them, this feature was needed otherwise an unnamed stream redirected in a calling scope could be modified downstream and upon return that stream might have been already closed.  This would lead to bizarre behavior.

Shared Streams

All NEW shared and NEW GLOBAL shared streams are supported using the SharedVariableManager .  This class has stream-specific methods to add and lookup streams.  Please see Shared Variables for more details.

References

rules/convert/base_structure.rules
rules/convert/input_output.rules
rules/convert/process_launch.rules

com.goldencode.p2j.util.Stream.java
com.goldencode.p2j.util.StreamWrapper.java
com.goldencode.p2j.util.FileStream.java
com.goldencode.p2j.util.ProcessStream.java
com.goldencode.p2j.util.ProcessOps.java

Limitations
  1. Although documented in Progress references, the default code-page conversion (stream to internal and vice versa) does not appear to actually be implemented OR perhaps this is locale specific.  No default codepage conversion is implemented in P2J.
  2. All of the explicit codepage conversion processing that is possible in Progress is missing.
  3. The effect of the BINARY option is only accounted for in READKEY processing and in this case it only modifies how newlines are returned.  If BINARY modifies other processing, it has not been identified and is thus unimplemented at this time.
  4. Progress references document that the null character terminates strings.  This is not implemented.
  5. DBCS support is not provided.
  6. UNBUFFERED mode is not supported (and is probably not needed).
  7. MAP/NO-MAP support does not exist.
Remote I/O and Terminal Integration

Introduction

The Progress 4GL is implemented as a client-centric environment.  Even the Progress database server is really just a (shared memory or network) connection to a process which takes a limited specification of index fields and the corresponding values or value ranges and returns records from that index.  All query where clause and sorting is handled on the client side!

In a P2J environment, the "client" is the "thin client".  This is the Java process written to connect to a P2J server and provide a remote (from the server) user interface.  It is run in the context of the user's operating system login or shell.  In other words, the user will be logged in to some operating system (e.g. Linux) and will obtain a shell or desktop.  If this is a terminal oriented system (CHUI) then that user will have logged in via a terminal session (via a serial line or a network protocol such as telnet or ssh).  After the login, the user will have some shell or desktop process running on that system.   From there the user either explicitly (with a command or icon) or implicitly (via a profile or startup script) will launch the P2J thin client Java process.  When this document references the "thin client" or "client", this is what is meant.

Stream and File System Remoting

When a Progress 4GL program reads/writes files, interacts with the file system or launches child processes, it is intrinsicly doing so on the client system.  This may not be evident in a system where the client and the P2J server are the same physical box.  But since all such activities in Progress 4GL are done in the context of a child process (e.g. on Unix, the executable "_progres") which is running with the login context of a specific user, the home directory and file system access rights of that specific user.  In a P2J server environment, the entire server is running in a process which is a daemon and while it does have an operating system security context, this is most certainly not the same context as a given user.  This means that every process launch *may* be dependent upon context that is only available on the client system.   The child process may be dependent upon access to the user's home directory or to data that is in a specific path that only exists on the client system.  Or perhaps the process itself only exists on the client system and/or has specific environment variables or other state/resources that are only available there.  For example, many Progress programs launch shell scripts or utilities that depend upon the client's context for current working directory and other environment variables (e.g. the Unix mail command).  For this reason, the end-points of all I/O operations (file, file system and process launching) must be executed on the client system.  In other words, they must be "remoted" to the client.  The business logic will still reside on the server.  This includes all logic that accesses and manipulates data, which defines the flow of control for the application.  However, when a read or write or file system search or process launch occurs, these operations must be remoted to the client but made to appear as if they were happening on the server.  This allows the logic to remain intact and the minimal amount of processing is pushed to the client.

The Stream class is abstract.  It defines the following abstract methods:
The backing file (FileStream) and process streams (ProcessStream) are Stream subclasses that implement concrete versions of these methods.  The RemoteStream class is used on the server and it redirects all calls to the abstract methods to instances of FileStream or ProcessStream.  This is handled using the LowLevelStream interface and a StreamDaemon class on the client side.  Each RemoteStream instance represents a single stream on the client and references that stream via an integer ID.  The StreamDaemon implements the LowLevelStream interface which is called by RemoteStream for service.  Each API has the innteger stream ID passed as its first parameter.  The StreamDaemon looks up the referenced stream and dispatches the method call to the real stream.

The StreamFactory class provides static factory methods (openFileStream() and openProcessStream()) to instantiate streams on teh client..  These methods call a client side export in the StreamDaemon to create the backing FileStream and ProcessStream resources on the client.  It obtains the integer stream ID in return and intitializes a RemoteStream instance using this ID.  The resulting RemoteStream object is returned to the caller (e.g. business logic) and all access to the RemoteStream is transparently redirected to the client.

INPUT THROUGH, OUTPUT THROUGH and INPUT-OUTPUT THROUGH all allow one to launch a child process and access the child's stdout/stderr for reading, or the child's stdin for writing or both.  This is handled by ProcessOps.  ProcessOps is the Progress compatible interface for process launching.  It accepts RemoteStream instances and uses a Launcher interface and the client-side ProcessDaemon instance on the client to actually process the process launch (using the J2SE Runtime.exec() method).  The resulting child process' standard I/O is connected to the client-side ProcessStream instances.  On the client side, when the child process is launched, its stdin and/or stdout/stderr are either connected to a pipe or to the parent process' terminal depending on the flow direction of the stream definition.  This is handled with the cooperation of ProcessDaemon.launch() and the CHARVA Toolkit.pseudoTerminalLaunch().

FileSystemOps provides a set of static methods to inspect and interact with the file system.  These are redirected to the client using the FileSystem interface which is serviced by the FileSystemDaemon class.

As a general rule, the maximum amount of processing that can be done on the server is done there and only a minimum remote interface is put in place to provide remote service.

For other process launching (UNIX, OS-COMMAND...), such items don't need to be read/written as streams on the server, but the launching must occur on the client anyway.  This means that the ProcessOps.launch(String[], boolean, boolean) method which is used in these cases, is serviced by ProcessDaemon as above, except the actual process launching mechanism is different (see below).

Non-Stream Process Launch Remoting

Normal process launching (UNIX, OS-COMMAND...) is done in 3 forms:
  1. Synchronous (lack of NO-WAIT option)
  2. Asynchronous (NO-WAIT option)
Some launched applications are synchronous and interactive (the user is allowed to interact with the child process' UI).  In such cases, the child process' standard I/O must be connected to the terminal (such as "vi").  Such programs cannot be run with the NO-WAIT option (it just doesn't work properly in Progress, your terminal gets very "confused").  The result is that the child process takes over the terminal and Progress is blocked (it is a synchronous call when NO-WAIT is not available).  This means that no output occurs to the screen except what is being done by the child process.

Java's Runtime.exec() process always creates a child process that whose STDIO is not connected to a terminal.  This means that interactive applications like "vi" will not properly work when launched by the standard J2SE process launching mechanism.  Since Progress allows child processes to be interactive in the current terminal, a different process launching mechanism is needed.

To duplicate this processing, the Charva implementation programs NCURSES to temporarily "shell out" or suspend current processing (see ThinClient.suspend()).  This means that NCURSES resets the terminal settings to those which were set at NCURSES init.  At that point, a child process is created (using fork).  After fork, the parent process will return and block (unless NO-WAIT was specified) until the child process exits.  If NO-WAIT is specified, the parent process resumes. The child process copies the command line parameters and then calls the Linux/UNIX execvp() system call.  When the child process is done, it exits.  The parent process will use ThinClient.resume() to re-enable the interactive terminal after it pauses (or just before it returns from ProcessDaemon.launch() if NO-WAIT was specified).  While the child process runs, all STDIO is directly connected to the parent process' terminal (the child process is directly connected to the user's current terminal). 

The process launching API exists in the Charva libTerminal.so to handle the fork() and execvp().

Either way, the worker for ProcessOps static methods is in the client's ProcessDaemon.  The ProcessDaemon uses features of the ThinClient to properly integrate with the terminal, to suspend/resume CHARVA and to handle the actual process launching.

Other applications are synchronous and non-interactive child processes.  Some of these may still write to STDOUT/STDERR.  Progress attaches the terminal I/O to the child process.  Progress itself can't know what processes are interactive and which are not.  This is true because the fact of whether a process is or is not interactive is something that cannot be inspected, it is implicit in the application (or applications if the child process launches other child proceses) being run.  In addition, there is no provision in the language for the programmer to provide a hint to Progress about this fact.  For this reason, we must conclude that Progress implements the launch of all synchronous processes in the same way for each process ("shelling out" the terminal, forking, execing to launch the process).

Finally, one can launch an asynchronous application (by adding the NO-WAIT option).  Such programs are always non-interactive applications.  However, if they do write to the STDOUT/STDERR, the result from an end-user perspective is that in a NO-WAIT condition, the child process' output is intermixed (in a non-deterministic manner) with output created by Progress.  For example (press the spacebar quickly when at the first "Press Space to Continue" prompt):

os-command no-wait "echo HELLO; sleep 3; echo 'HELLO, AGAIN!'".
def var c as char init "Hello World!".
display c.
enable.
os-command silent "echo GOODBYE".

The NO-WAIT launch in Progress don't seem to be really any different from the normal synchronous approach in terms of how the launch itself is done.  The only difference seems to be in whether or not Progress waits for the child to exit.  In a NO-WAIT case, the resume() is called immediately after launching, such that Charva can resume immediately.  In the synchronous case, resume() is not used until after the child process exits.  Either way, this simply means that the decision to wait or not to wait is always made just before the call the resume().

The remote worker methods handle NO-WAIT (making these methods synchronous - waiting for the child process to end - or asynchronous) as documented above.  No special proxies or other features are ever returned to the business logic for these ProcessOps.launch(String[], boolean, boolean) modes.  The server-side call invokes the remote worker which launches the child process on the client synchronously or asynchronously and then returns.  The server-side call then returns to the business logic.

Note that the ProcessOps class has some "cleaner" classes and extra cleanup threads to handle the proper close of the pipes and child process resources as needed.

The NCURSES-related changes are in Charva's libTerminal.so (toolkit.c) and the corresponding Toolkit.java.

NCURSES references:

http://www.faqs.org/docs/Linux-HOWTO/NCURSES-Programming-HOWTO.html
http://invisible-island.net/ncurses/ncurses.faq.html
http://www.tldp.org/HOWTO/Text-Terminal-HOWTO.html (especially the bit about termcap/terminfo and the part on psuedo-terminals)
http://www.cs.utk.edu/~shuford/terminal/termcap_news.txt   (search on "newterm(" for a bit of an example on multiple terminals)
http://dickey.his.com/ncurses/ncurses-intro.html

You will especially want to review:

initscr() - default init for single terminal applications, uses newterm() and other stuff under the covers
newterm() - creates a terminal, used instead of the initscr(), sets the current terminal to that newly created instance and returns a pointer to it
set_term() - switches the current terminal to the one passed, returns the previous terminal
delscreen() - deallocates memory for a terminal
SCREEN* - the data structure that holds the terminal "instance"
endwin() - used to reset the tty attributes/modes to the normal interactive shell (or whatever was set before ncurses started), is used at the end of all processing and also temporarily to "shell out" to another (non-ncurses) program
def_prog_mode() - temporarily saves off the current terminal settings, called prior to an endwin() such that the settings can be restored after "shelling out"

Stream Integration with the UI

In Progress 4GL there is also a concept of the "terminal".  The terminal is the user-interface (UI) with which an end-user will interact.  There are a wide range of language features provided to allow a UI to be created and maintained in Progress.  Progress does differentiate between "regular" streams (e.g. file system and child process I/O) and the terminal.  For example, there are certain operations that only are allowed on a non-terminal stream (e.g. PUT, EXPORT, IMPORT, SEEK...).  However, in other ways there is little differentiation between the terminal and streams.  In other words, the terminal can be redirected to a stream and many UI statements that normally only have interactive meaning become non-interactive and use their frame-oriented layout and formatting knowledge to read from the stream into a field list (INSERT/SET/UPDATE/PROMPT-FOR) or write from a field list to the stream (DISPLAY).  Some UI statements don't read or write to the stream, but do operate on the stream in other ways (DOWN, UP, HIDE, VIEW).

These reads/writes can be made on a replaced unnamed stream (e.g. INPUT FROM myfile.txt) or from a named stream (e.g. INPUT STREAM my-stream-name FROM myfile.txt).  In Progress, the former method operates on the "unnamed" stream which by default is the same as the terminal.  Thus, normally a UI language statement would operate upon the terminal by default, but some conditional processing (e.g. IF/THEN/ELSE) might redirect the unnamed input and/or output stream(s) such that a single UI statement would operate on either resource.  If that statement happens to be operating on the terminal, then the statement is interactive.  If it is stream based, the statement will have no interactive (end-user observable) side effects.  It is only at runtime that Progress can determine which "path" will be taken for these UI statements that can also be used on streams.  Since named streams can never reference the terminal, when a UI statement uses an explicit (named) stream, it is easy to identify that this is a stream statement and not really a UI statement.  However, when the unnamed stream has been redirected, these statements become "bimodal".  While there are some simple examples to the contrary, in many cases there is no way to know how that statement will be serviced (I/O versus UI) by analyzing the source code at conversion time.  Since such a case must inherently be determined at runtime, there must exist a common interface for these methods which then detects and uses the proper mode.  Consider this example:

def var redir as logical.
def var i     as integer.

/* some processing that calculates redir, possibly with UI or database reads */

if redir then
   input from file.txt.

set i.

message "i =" i.

Due to the runtime nature of this decision, backing methods for INSERT/SET/UPDATE/PROMPT-FOR/DISPLAY have been created in GenericFrame (actually, there is no support for INSERT support at this time).  All of the frame elements (in the proper order) being read are passed as a FrameElement array.  If an explicit stream is passed to SET/UPDATE/PROMPT-FOR then it is expected to be open for input (if not an error occurs).  In stream processing mode, a completely different operational mode is entered which uses an input reading method to feed the screen buffer from the input stream.  This bypasses the normal user interface processing for enable/disable, standard view/Z-order/refreshing support.  This mode is entered when a stream is explicitly passed to the API OR if no stream is explicitly passed but the unnamed input stream is redirected (has been explicitly opened as a stream).

There are 2 forms that can cause this to occur:
  1. The unnamed input stream is opened as a file or child process.  It defaults to the terminal, but once opened (and as long as this is not closed) it will reference the non-terminal stream.
  2. An explicit stream reference can be added to the INSERT/SET/UPDATE/PROMPT-FOR language statement.
Either way, this forces the input to be removed from the terminal/keyboard and obtained via the stream.  This also eliminates the ENABLE/DISABLE processing and the WAIT-FOR that is done inside the PROMPT-FOR portion of these statements.  Instead, the order of the fields in the SET/UPDATE/PROMPT-FOR is the order in which fields are read from the stream.  This makes these statements a kind of analog to the PUT language statement (this writes a list of fields to a stream).  It is important to note that the layout of the associated *frame* DOES NOT make any difference in how the data is read.  For example if the frame layout itself is ordered differently than the order of the fields in this language statement OR if the frame has other fields that are not listed in the language statement, these other fields and ordering are ignored.  It also means that the list of already enabled widgets makes no difference (only the fields listed explicitly on this statement will be read and they will be read in the exact order of this current statement).  One could say that this is really just a reading counterpart to PUT except for these two points:
  1. A single source code statement can both handle real UI and reading from the unnamed stream and this fact is detected and implemented at runtime only.
  2. These statements can also write data, either to the screen (if output is not redirected) or to a named or unnamed stream (if output is redirected).  This additional output only occurs if the stream in question does not have NO-ECHO specified (e.g. INPUT FROM myfile.txt NO-ECHO).
The Stream class supports setting the echo mode (it defaults to true ).  Please note that for streams opened via the INPUT-OUTPUT THROUGH language statement, this value must default to false , which is handled by emitting a setter into the business logic in these cases.  Interestingly, even if the input stream is marked as echo, if output is redirected and that output stream is marked as no-echo then the echo does not occur.  In other words, if both input and output are redirected, both streams must be marked as echo for the echo to occur.  If output is not redirected, then only the input stream's echo setting is taken into account since the screen is always considered to be in echo mode.

There is also an unusual behavior in this echo mode.  The SET/UPDATE/PROMPT-FOR language statements are "bimodal" (they handle both terminal and stream I/O).  In stream mode, if echo is off, no output occurs (to either the terminal or to a file if the unnamed out is redirected to file before the call occurs).  If echo is on, then output is generated AND it is based on the frame definition/screen-buffer that is in use.  However, the output is done in a "special" text only mode.  In other words, the output values are the exact text that has been parsed out of the input file AND it is NOT data type specific.  Any data in a fill-in field that has a data type that is non-character is treated as character.  In addition, it seems that any widget type other than fillin (e.g. combo-box or button but strangely enough also text!) generates no output at all.  In all cases (whether the widget does or doesn't emit), the column headings are done just as normal.  A *normally* right-aligned field like a decimal fill-in will have its column header right aligned BUT its data will be the unformatted, left-justified text just as it was read from the input stream.   For example:

┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│c[1]                 c[2]                 c[3]                 l    l2 dd                 i                 d│
│──────────────────── ──────────────────── ──────────────────── ──── ── ────────── ───────── ─────────────────│
│Double quoted text   'single              quoted               fals T  11/22/1999 -9,98787  +2,323243.32     │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│c[1]                 c[2]                 c[3]                 l    l2 dd                 i                 d│
│──────────────────── ──────────────────── ──────────────────── ──── ── ────────── ───────── ─────────────────│
Double quoted text    'single              quoted               Nope t  11/22/1999 -00998787 00002323243.32000│
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

The first output is the echo mode of SET.  The second output is using DISPLAY on the same data that was read.  The first 3 columns are character data and thus show no difference.  The l and l2 columns are logicals.  The logical data read from the file differs from the 2 outputs (they are both using format strings "Yep/Nope" and "t/f" respectively).  The dd field is a date and doesn't show any difference. The i (integer) and d (decimal) fields show the unformatted, left aligned text read from the file in the echo mode and the properly formatted, right aligned data in the DISPLAY.  The i format string is -99999999 and the d format string is 99999999999.9999 for both the SET and DISPLAY.

So both cases use the same frame layout and format strings, but the actual data is treated as unformatted text in the echo mode.  At this time, this special text mode for logicals, integers and decimals is not supported.

As an additional unusual behavior of this quirk, any errors resulting from truncating the input data cause this truncated data to be written back to the screen buffer before the error condition is raised (the SET/UPDATE form do not assign that data back but the data does exist in the screen buffer).  If echoing is active, the echo of that screen buffer will occur even though an error occurred!  At this time, this screen buffer copy is only supported for character data types, the other types do not get stored.

INSERT/SET/UPDATE/PROMPT-FOR are reading instructions when used with an input stream that is not the terminal.  Each has a list of lvalues (fields). Data is "read" from the stream on a whitespace delimited (and for character data, optionally double quote delimited) basis.  It is not possible to control the delimiter as can be done via IMPORT.  The parsed data is converted into each corresponding field's type and the data is then assigned into that field. Each field's format string (default, from dictionary, from var def, overridden in a format phrase) will be honored in that conversion.  The format string is obtained from the associated widget's configuration rather than being passed via the API itself.

All reading is done in the exact order of the widgets as listed in the frame element array passed to the SET/UPDATE/PROMPT-FOR APIs.  The order of these same fields in the frame itself is meaningless.  So:

form x y z with frame fr.
input from ./text.txt.
set z x y with frame fr.
input close.

The order of the fields in the form statement is the order of the data when displayed on the screen or when output is redirected to a stream.  BUT any stream input reading honors the order in the SET/UPDATE/PROMPT-FOR.  In the example above, this means that the first double quote or whitespace delimited entry on the current line in the input stream will be assigned to the "z" variable, the second to "x" and the third to "y".

It is important to note that the field-level data parsing is identical to that done for the IMPORT language statement.  So field skipping when reading an embedded hyphen, reading 1 line at a time (per SET/UPDATE/PROMPT-FOR), no assignment to the screen-buffer for fields read after the end of the line has been found, double quote processing as a single entry (allows embedded whitespace or hypens)... all these behaviors are the same. This is exposed by the Stream.readFieldWorker().  This worker is shared by the IMPORT implementation, but how the results are used (and the validation that is done) are quite different after the parsing is done.

In particular, this form of reading is uses the format string as a basis for the proper length of the data to read from the stream for the given field however the actual contents of the format string are not used to parse except for the logical type.  In that case, the values specified in the format (e.g. "Yep/Nope") will be honored.  For all other data types, the only affect of the format string is on the size.  An error condition is generated if data is read that is larger than can fit into the size specified by the format string.

The conversion from string data (as read from the file) to BaseDataType is handled by detecting the widget's data type and then instantiating the proper type using the constructor that using a string as input.

The way the logical formatting is processed is an exception.  This processing trucates the input text (if needed) to the maximum size of either of the 2 values in the format string (for "Yep/Nope" then maximum size would be 4).  Any truncation here is not an error (as it is with the other types), instead the data value (the string read from the stream) and the format string are passed to a special logical constructor for processing.  Please note that this means that (as in Progress), there is a quirk where a longer input (for example, "nopenotatall") could actually match the longer of the 2 format string values (for example, the false value of the "Yep/Nope" format).  This also applies to true/false and yes/no.  So a value of "yessiree" would be interpreted as true in any case where the longest format value is 3 characters.

For a more interesting example, please see testcases/uast/io_redirected_by_ui_stmts.p.

Consider these examples:

Example Program
Output
def var i as int.

input thru "echo 5".
output to ./test.txt.

prompt-for i.

output close.
message input i.
test.txt contains:

         i
----------
5

The message line on the terminal contains:

5
def var i as int.

input thru "echo 5".
output to ./test.txt.

prompt-for i.

message input i.
output close.
test.txt contains:

         i
----------
5
5

The message line on the terminal contains nothing.
def var i as int.

input thru "echo 5".

prompt-for i.

message input i.
The terminal displays:

┌──────────┐
│         i│
│──────────│
│5         │
└──────────┘

The message line on the terminal contains:

5

Please note that any statement that generates output (both PROMPT-FOR and MESSAGE in the examples above) will do so to the redirected stream.

A special case exists when input is NOT redirected BUT output IS redirected.  In this case PROMPT-FOR, SET and UPDATE all will actually allow interactive editing via the normal terminal (like a temporary switch to interactive mode) BUT then after the edits are done there is effectively an additional VIEW of the data to the output stream.

The most common redirected output statement is DISPLAY.  DISPLAY writes the given list of fields/expressions to the stream using the format and layout of the frame.  This is the most common way to generate a report in a file or for a printer.  This can be a redirection of the unnamed stream or the DISPLAY statement can also reference a named stream.

The format of the stream output will contain explicit (SKIP) and implicit (based on a list of fields that exceeds the display's width) newline characters exactly as the terminal would display.   The following example illustrates this:

Example Program
Output
def var txt1 as character.
def var txt2 as character.
def var i    as integer.

do i = 1 to 6:
   txt1 = txt1 + "Hello World! ".
end.

txt2 = txt1.

display txt1 format "x(78)" txt2 format "x(78)".

┌──────────────────────────────────────────────────────────────────────────────┐
│txt1                                                                          │
│txt2                                                                          │
│──────────────────────────────────────────────────────────────────────────────│
│Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! │
│Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! │
└──────────────────────────────────────────────────────────────────────────────┘

output to ./test.txt.

def var txt1 as character.

def var txt2 as character.
def var i    as integer.

do i = 1 to 6:
   txt1 = txt1 + "Hello World! ".
end.

txt2 = txt1.

display txt1 format "x(78)" txt2 format "x(78)".
test.txt contains:

txt1

txt2
------------------------------------------------------------------------------
Hello World! Hello World! Hello World! Hello World! Hello World! Hello World!
Hello World! Hello World! Hello World! Hello World! Hello World! Hello World!


It is also important to note that the box normally drawn around a frame is not output to the stream.  Instead there is an empty line where the top and bottom horizontal lines would be and leading spaces where the left vertical line would be.  No output is generated for the right vertical line.  If the frame is defined as a NO-BOX, then none of those spaces or extra lines will be output.

Column headings do get output using hyphen characters (and a space in between each column) as the separators.  NO-LABELS and SIDE-LABELS are honored just as they would be in the terminal.

In the case of the DISPLAY statement, it is important to note that it still copies data to the screen buffer and it is the resulting screen buffer that is written to the stream.

A close "cousin" of DISPLAY is the VIEW language statement.  This does not copy any new data to the screen buffer, however it does write the contents of the referenced screen buffer to the named or unnamed stream.  This is often used to output a static (unchanging) report header multiple times.  Output is done using "frames" (screen buffers).  If that frame (including the unnamed frame) is referenced for both file and terminal output, the data placed in that screenbuffer at the time of output will include all prior data.  For this reason Progress source code often uses a "fake" frame for file I/O so that that data doesn't appear on the terminal later.

The HIDE statement can also be used in these cases, but there appears to be no effect of such a statement (it is a NOP except in an obscure case -- see below).

The following examples show the affect of down frames and the DOWN and UP language statements.

Example Program
Output
output to ./test.txt.

def var i as integer.

do i = 1 to 6 with frame fr:
   display i with 12 down frame fr.
end.
test.txt contains:

         i
----------
         1
         2
         3
         4
         5
         6

output to ./test.txt.

def var i as integer.

do i = 1 to 6 with frame fr:
   display i with 12 down frame fr.
   up 1.
end.
test.txt contains:

         i
----------
         1
         2
         3
         4
         5
         6

output to ./test.txt.

def var i as integer.

do i = 1 to 6 with frame fr:
   display i with 12 down frame fr.
   down 1.
end.
test.txt contains:

         i
----------
         1

         2

         3

         4

         5

         6



def var i as integer.

do i = 1 to 6 with frame fr:
   display i with 12 down frame fr.
end.

The following is displayed on the terminal:
┌──────────┐

│         i│
│──────────│
│         1│
│         2│
│         3│
│         4│
│         5│
│         6│
│          │
│          │
│          │
│          │
│          │
│          │
└──────────┘

def var i as integer.

do i = 1 to 6 with frame fr:
   display i with 12 down frame fr.
   up 1.
end.
A series of screens is displayed on the terminal, with each screen alternating between a number (1-6) and a blank/empty output.  12 screens are displayed in all, each in turn.  The first looks like:

┌──────────┐
│         i│
│──────────│
│         1│
│          │
│          │
│          │
│          │
│          │
│          │
│          │
│          │
│          │
│          │
│          │
└──────────┘

def var i as int no-undo.          

do i = 1 to 10:
display "Count: " i with 10 down.
if i < 6
then down.
else up.
end.
The result on the terminal:

┌──────────────────┐
│                 i│
│        ──────────│
│Count:           1│
│Count:          10│
│Count:           9│
│Count:           8│
│Count:           7│
│Count:           6│
│                  │
│                  │
│                  │
│                  │
└──────────────────┘

def var i as int no-undo.          

output to tmpup1.out.

do i = 1 to 10:
display "Count: " i with 10 down.
if i < 6
then down.
else up.
end.
The result in the stream:


                 i
        ----------
Count:           1
Count:           2
Count:           3
Count:           4
Count:           5
Count:          10

def var i as integer.

do i = 1 to 6 with frame fr:
   display i with 12 down frame fr.
   down 1.
end.
The following is displayed on the terminal:
┌──────────┐
│         i│
│──────────│
│         1│
│          │
│         2│
│          │
│         3│
│          │
│         4│
│          │
│         5│
│          │
│         6│
│          │
└──────────┘


Following rules determine the Progress behavior of the DOWN/UP/SCROLL statements with a down frame, with or without the RETAIN/SCROLL n attributes:
  1. Up statements with a down frame (see testcases/uast/downframes/up_statements_with_a_down_frame.p):
  2. Down statements with a down frame (see testcases/uast/downframes/down_statements_with_a_down_frame.p):
  3. Down frame with stream output (see testcases/uast/downframes/down_frame_with_stream_output.p):
  4. Conditional up/down statements (see testcases/uast/downframes/up_down_conditional_with_a_down_frame.p):
  5. Up/Down statements with an uninitialized down frame (see testcases/uast/downframes/up_down_statements_with_a_down_frame_uninitialized.p):
  6. Retain n - down statements with a down frame (see testcases/uast/downframes/retain_n_down_statements_with_a_down_frame.p): For a down frame, when a down statement lands on a uninitialized row:
  7. Retain n - up statements with a down frame (see testcases/uast/downframes/retain_n_up_statements_with_a_down_frame.p): For a down frame, when a up statement goes past the first row:
  8. Scroll n - down statements with a down frame (see testcases/uast/downframes/scroll_n_down_statements_with_a_down_frame.p): For a down frame, when a down statement lands on a uninitialized row:
  9. Scroll n - up statements with a down frame (see testcases/uast/downframes/scroll_n_up_statements_with_a_down_frame.p): For a down frame, when a up statement goes past the first row:
  10. Scroll statement with a down frame (see testcases/uast/downframes/scroll_statement_with_a_down_frame.p): When from-current is not used:
  11. Scroll from current with a down frame (see testcases/uast/downframes/scroll_statement_from_current_with_a_down_frame.p): When from-current is used:
  12. SCROLL n and RETAIN n should never be used together (see langerf.pdf page 543).
Following rules determine the Progress behavior of the conditional DOWN statement with a streamed, down frame (with or without box) (NL = NewLine) (file testcases/uast/downframes/streamed_paged_frame1.p):
  1. If the current line is at the end of a page, then: a DOWN 1 statement will write a NL to the next page; a DOWN 2 statement will write a NL to the current page and a NL to the next page.
  2. If the current line is at position X having Y more rows in the current page, then a DOWN Y statement will write Y NL's to the current page and one more NL to the next page.
  3. If the current line is at position X having Y more rows in the current page, then a DOWN (Y+Z) statement will write Y+1 NL's to the current page and one more NL to the next page.
  4. The file testcases/uast/downframes/streamed_paged_frame2.p demonstrates how Progress computes the LINE-COUNTER and PAGE-NUMBER functions: after a down, if the page is ended, P2J erronetly will update the line counter to 1 and also increment the page number. Progress seems to do this when the subsequent statement (down/display/etc) which affects the stream buffer is executed. NOT IMPLEMENTED YET.
Some notes:
  1. DISPLAY and VIEW generate a fully formatted output (very close to the result when viewed on a terminal) where SET/PROMPT-FOR/UPDATE generate the field output in a "raw" form.   In the DISPLAY examples, the numbers are right aligned where in the PROMPT-FOR examples, the numbers are left aligned.
  2. When writing, the UP statement has differing behavior.  On the terminal, it seems to defeat the normal behavior of a DOWN frame.  On the stream, it usually does nothing, however in some cases it can cause a form of overwriting.
  3. When writing, DOWN seems to do a similar thing for both terminal and stream.  This behavior extends to the fact that some usage of DOWN will overwrite the same data rather than generating multiple lines.
  4. Although this is not shown above, when reading from a stream neither DOWN nor UP have any apparent effect.  Most important, these language statements do not change the current read position in the stream.
  5. DISPLAY and VIEW have a special behavior when output is redirected AND the output stream is paged AND the given frame is either PAGE-TOP or PAGE-BOTTOM.  In this case, the given frame is registered as a header (PAGE-TOP) or footer (PAGE-BOTTOM) and this will be displayed at the top and bottom of each page respectively.  The only difference is that DISPLAY will ALSO cause the frame to output immediately (as well as being "called back" for the header/footer) while VIEW only causes a registration but not an actual output.
  6. HIDE only has an effect in the case where output is redirected AND the output stream is paged AND the given frame is either PAGE-TOP or PAGE-BOTTOM.  In this case the frame is removed from the list of headers/footers.
  7. MESSAGE statements output to the stream but in the case where input is NOT redirected and output IS redirected, they have the curious behavior of displaying on both the terminal and the redirected output stream.  If input IS redirected (and output IS redirected), then there will be no interative terminal output, only output to the redirected output stream.
  8. MESSAGE SET/UPDATE using a redirected input stream operate just like a SET/UPDATE from a redirected input stream (it is a stream reading statement).
  9. ENABLE, PUT SCREEN, BELL, STATUS, PAUSE always operate on the interactive terminal and have no effect on any redirected streams.
The methods that are the equivalent to the DISPLAY, VIEW, HIDE, DOWN, UP statements are available in forms that take an explicit Stream reference as a parameter (see the CommonFrame interface and the GenericFrame implementation).  In the case where any of these are called with an explicit stream reference the UI runtime code detects that these are outputs to a stream, such that the terminal is temporarily redirected to the given stream.  Likewise, if an open has occurred on the unnamed output stream, this causes the terminal to be redirected to this stream (if it isn't already redirected).  The explicit stream reference form removes the redirection when the statement (e.g. DISPLAY) is complete.  The unnamed output stream terminal redirection remains in place until that unnamed output stream is closed.

The thin client maximizes the usage of the drawing and layout management of frames while replacing the back-end output technology.  This is done by implementing a pluggable replacement set of drawing primitives in a class named OutputPrimitives in CHARVA.  The CHARVA Toolkit implements a default instance of OutputPrimitives which maps all calls to the native Toolkit backing functions.  This means that by default, CHARVA uses NCURSES for all output.  The Toolkit has an interface to switch primitives and the thin client has a RedirectedTerminal class that implements this interface.  This class is used for each stream that is used for redirected output.  These drawing primitives duplicate the buffering, clipping, cursor movement, drawing and rendering (sync) functions using a stream as the output device.

In addition, the server's UI runtime code is aware of each session's unnamed streams.  This is used to enable use of unnamed streams from the UI language statements (which can be intermixed between "redirected" UI stmts and I/O stmts that can't be used on a terminal).  See UnnamedStreams .

PAGE-TOP (header) and PAGE-BOTTOM (footer) frames are honored and the DISPLAY, VIEW and HIDE language statements register and deregister these frames with the associated streams.  When the stream starts a new page it forces all headers in the active header list to render (in order of registration) and likewise, at page end all footers are rendered.  Any partial page at stream close will be extended to the page end and the footers will be rendered there.  PAGE-TOP and PAGE-BOTTOM frames that are not used on a paged output stream have no special effect (they are treated as normal frames).

In the Progress 4GL environment, certain frames can exhibit overwrite behavior.  This can occur because of a frame's implicit DOWN 1 behavior OR because of a programmer's explicit use of DOWN/UP (see examples above).  This appears to be a side-effect of the buffering strategy used by Progress.  Writing seems to occur to a 512 byte buffer (based on my testcases).  This buffer is flushed when Progress needs to write something that would exceed the 512 byte mark OR when the stream is closed.  This can be seen in the following testcase by changing the iterations from the default 8 to 7.  The pause then occurs before the output from cat.  Please note that this example does not exhibit the overwriting behavior (the cause of this is unknown at this time), but the testcase does show the buffering approach.
def var iterations as int init 8.

message "Redirect?" set redir as logical.

if redir then
output thru cat.

message "Iterations?" update iterations.

def var i as int.
def var c as char format "x(64)" init "1".

do i = 1 to 63:
c = c + "0".
end.

i = 0.
repeat while i < iterations:
i = i + 1.
display c with 1 down no-labels no-box.
end.

pause.
This overwrite behavior is not supported at this time.

Limitations:

Using Frames in Multistream Applications

Progress applications can use multiple streams simultaneously. An output stream is used typically with more than one frame and a single frame can be used with more than one output stream during the application run.  This raises questions about the frames and streams behavior in such a mixed environment.

There are three classes of behavior here:
The first class triggers some processing in the stream every time the stream gets prepared to get output from another frame. We will call it frame switching.

The second class triggers some processing in the frame every time the frame gets prepared to give output to another stream. We will call it stream switching.

The third class is just a boundary condition where no switching occurs.

For the purpose of this discussion, we won't distinguish the named streams from the redirected unnamed stream, which is the terminal, whenever appropriate, and use the term stream. The term terminal, conversly, will mean non-redirected unnamed stream.

The frame switching behavior is defined by the following rules:
The stream switching behavior is defined by the following rules:
The DOWN and UP statements work differently depending on the stream or terminal as the target. The following rules are applied, if the target device is a stream:
There are things specific to the terminal as the target device:

Transaction Processing and Block Properties

Summary

Progress 4GL has a transaction processing environment integrated into the base language.  While some explicit control is provided over how transactions occur, the vast majority of processing is implicit based on the operations, control flow structures, block properties and sequence of statements/blocks which is encoded by the programmer.

Block Types

In the 4GL, code can be structured into related sections called blocks.  In Progress v9, the following types of blocks are available:
A block groups code for several purposes:
Each block has a "header" and a "body".  All code that is executed in a block is contained in the block body.  The description of the block itself and any explicitly controlled options are defined in the block header.

The lines of code in a block's body are executed "top to bottom".  All code in the same block will share the same behavior in terms of transaction processing.  Absent any explicit changes to the flow of control (by language statements that cause a flow of control change) or the generation of a condition (such as an ERROR) which will cause the flow of control to change, all code in a given block is executed if the block body is entered or all the code is not executed if the block body is not entered.

Some blocks can be made to iteratively execute the code contained in the block body.  Such blocks are known as looping blocks.  Progress provides blocks that can explicitly loop based on programmer controlled expressions in the block header.

Blocks support transaction processing which is the ability to undo (reverse) edits to variables or the database that occurred within the scope of that block, when an abnormal condition is raised or when an explicit UNDO statement is encountered.  Some blocks can have their support for transactions explicitly coded via the TRANSACTION keyword in the block header.

All blocks have some default behavior when abnornal conditions occur.  Some blocks allow that behavior to be explicitly specified via "ON phrases" in the block header.

Some blocks provide retry support which is the re-execution of a block with the same (original) state as the optional action in response to an abnormal condition.

Some blocks can take parameters and be called explicitly (external procedures, internal procedures and user-defined functions) or are otherwise invoked as a callback (triggers).  Triggers are special in that they are top-level blocks but they can only be invoked indirectly in response to an event (usually associated with the user interface but that is not required) that has occurred.  In this document, such blocks are called "top-level blocks".

Other blocks (all forms of DO, REPEAT, FOR and EDITING) are not top-level blocks and as such can only be contained inside another block (top-level or otherwise).  In this document, such blocks are called "inner blocks".  Inner blocks are the only kind of block that can be labeled and that label can then be referenced in language statements that explicitly change the flow of control such as UNDO statement, LEAVE statement, NEXT statement and in ON phrases.

Some blocks can have their definitions made nested inside another block.  The top level blocks (except for triggers) cannot be nested. Inner blocks can be nested.  Do not confuse nesting with recursion.  The top level blocks can be called recursively (they can call themselves directly or indirectly causing multiple instantiations on the call stack).  But the top level blocks cannot have their definitions made on a nested basis.  For example, an external procedure cannot be defined inside another external procedure.

The following summarizes these behaviors:

Block Type Looping Default Transaction Level TRANSACTION Option ON Phrases Retry Support Parameters Callable (Via Name) Labeled Nestable
external procedure no sub-transaction no no yes yes yes (filename in a RUN statement) no no
internal procedure no sub-transaction no no yes yes yes (RUN statement) no no
function no sub-transaction no no yes yes yes (in an expression) no no
trigger no sub-transaction no no yes no no no yes
DO no no transaction yes yes yes (depending on options) no no yes yes
DO TO yes no transaction yes yes yes (depending on options) no no yes yes
DO WHILE yes no transaction yes yes yes (depending on options) no no yes yes
DO TO WHILE yes no transaction yes yes yes (depending on options) no no yes yes
REPEAT (all forms) yes sub-transaction yes yes yes no no yes yes
FOR FIRST no (unless there is an additional table defined with EACH table) sub-transaction yes yes yes no no yes yes
FOR LAST no (unless there is an additional table defined with EACH table) sub-transaction yes yes yes no no yes yes
FOR EACH yes sub-transaction yes yes yes no no yes yes
EDITING yes sub-transaction no no yes no no yes yes (editing blocks for other frames can be directly contained and any kind of editing block can be indirectly contained)

Conditions

A condition is an unusual event that may occur during the course of processing.  The Progress programmer must be able to specify how each procedure responds to the range of possible conditions.  Progress 4GL provides a mechanism to specify such behavior at the level of each block.

Progress defines blocks as having very specific "block properties" which define behavior associated with a block.  These properties can be explicitly defined or are implicit.  The beginning and end of each block corresponds to the beginning and end of a "scope".  These scopes can be nested and this structure allows a Progress programmer to control the granularity to which a block's properties are applied.

In Progress, all blocks have some form of implicit and/or explicit condition processing, properties and transaction support.

One of the most central behaviors Progress provides is UNDO.  This is similar to the concept of a rollback of a transaction and allows one to abort partial changes to variables (and other non-database resources) when a failure or problem is encountered.  This is provided by default to reduce the amount of manual work by the programmer.

The behavior of which scope gets undone in transaction processing rollbacks is defined by block properties.  In addition, these block properties also define how the block responds to each type of condition AFTER the UNDO occurs.  More specifically, the way that the application changes its flow of control in response to a condition is something that is defined by block properties (implicit or explicitly overridden where possible).

The following conditions are possible:

Condition
Default Action in Response
ERROR
If a transaction is active, UNDO the closest enclosing block that has the ERROR property (this will also be a transaction or sub-transaction).

If a transaction is not active, the UNDO will be a no operation.

Then RETRY the same block (or loop) that has the ERROR property.
ENDKEY
If a transaction is active, UNDO the closest enclosing block that has the ENDKEY property (this will also be a transaction or sub-transaction).

If a transaction is not active, the UNDO will be a no operation.

Only blocks with the ENDKEY property will have an interactions counter.  The interactions counter is a simple count of the number of input-blocking language statements have executed in the block. Any nested block that does not have an interactions counter of its own will have its interactions counted in the nearest enclosing block's interactions counter. This counter is used to determine the behavior of the END-ERROR event (see below).

Then LEAVE the same block (or loop) that has the ENDKEY property.
STOP
If a transaction is active, UNDO the block that defines the transaction (as opposed to a sub-transaction).

If a transaction is not active, the UNDO will be a no operation.

Then RETRY the "startup procedure".  This means that the *entire* top-level procedure is run again from the top.

Note that if this is generated based on a database disconnect, a special form of processing will occur where the default STOP behavior cannot be explicitly overridden until an exit to a scope in which no access is made to the database that was disconnected.  At this point the normal STOP processing is re-imposed (see Progress Programming Handbook page 5-24).
QUIT
If a transaction is active, COMMIT the block that defines the transaction (as opposed to a sub-transaction).

If a transaction is not active, the COMMIT will be a no operation.

Exit to the operating system.
END-ERROR
Technically, this really isn't a separate condition, but rather a key in the UI which generates 2 different conditions based on context.  An END condition is generated if the key press occurs during the first input-blocking language statement in the block.  An ERROR condition is generated if the key press occurs during a subsequent input-blocking language statement in the block.

The exception to this processing occurs for blocks that do not have the ENDKEY property (this can happen for some DO blocks and is always true for EDITING blocks). In such a case, the interactions counter is not maintained for the current block but rather is maintained at the level of the nearest enclosing block which has the ENDKEY property.

This means that the normal conversion of END-ERROR to ERROR is thus modified in such blocks. In the EDITING case especially, it is likely that an ERROR will be generated (because the EDITING block itself counts as an interaction and thus with a larger interactions counter it is more likely that an ERROR will be generated). What is especially different here is that the ERROR will be propagated to the enclosing block that has the ENDKEY property! So even if the DO or EDITING block has the ERROR property in this case, if they also do not have the ENDKEY property and the interactions counter (for the enclosing ENDKEY block) is < 2 then the ERROR will NOT be processed by the current block.

See above for ENDKEY and ERROR respectively.

Some language statements and all assignments that can generate the ERROR condition can be executed using the NO-ERROR keyword, which disables the generation of this condition.  When a failure occurs, instead of generating ERROR, the ERROR-STATUS system handle will have its state updated.  Please see Progress Programming Handbook page 5-6 and Progress Language Reference page 1337 for more details.

The possible actions in response to a condition:

Action
Description
UNDO
Rolls back all database, variable, temp-table and work-table modifications since the beginning of the target block.  This target block must have opened a transaction or it must be enclosed within a block that opened a transaction AND must itself be a sub-transaction.  In any other case, the UNDO is a no operation.

There is no way of disabling UNDO.  Any condition that is raised will always trigger an UNDO (with the exception of the implicit behavior for QUIT which is to COMMIT instead of UNDO).  In addition, one of the actions below will also be executed.  If not explicitly specified, the implicit action is always a RETRY.
LEAVE
Breaks (exits) out of the target block.  If the target block is a top-level block, the LEAVE will be converted to a RETURN.
RETRY
The target block (non-loop or loop) is executed from the top with the same state as when it last executed.

Since a retry of a block in which the state never changes from iteration to iteration would yield an infinite loop, Progress provides a feature to detect when an infinite loop could occur.  This is called Infinite Loop Protection (ILP).  RETRY will be converted to a LEAVE, NEXT or RETURN depending on the block type to which the RETRY is targetted.
NEXT
The next iteration of the loop is executed.  If the associated block is not a loop, then this is automatically treated as a LEAVE or a RETURN (see below).

Infinite Loop Protection (ILP) protects the NEXT action (when triggered from an ON phrase or from an UNDO statement).  In certain cases, a NEXT will be converted to a LEAVE or RETURN by the infinite loop protection logic depending on the block type to which the RETRY is targetted.
RETURN
The current procedure, function or trigger returns to the caller.  There are 5 forms:
  1. Raise an ERROR condition in the caller ("RETURN ERROR").  In a function, this can be specified but the error is ignored (not raised).
  2. Return from a procedure or trigger block and allow normal processing to continue ("RETURN").  If this executed from a procedure, then the RETURN-VALUE global character variable is set to the empty string "" which can be read in the caller.
  3. Return the value of one of the basic data types when returning from a function ("RETURN <expression>").
  4. Set the RETURN-VALUE global character variable which can be read in the caller when returning from a procedure ("RETURN <character_expression>").
  5. Return and disable normal event processing when returning from an event trigger block ("RETURN NO-APPLY").  This can also be specified in other block types, but the NO-APPLY will be ignored.

In all actions (except RETURN), the block which is the target of the action can be implicitly determined (if there is no label) OR it can be explicitly specified using a user-defined label for the block.

One cannot UNDO a specified label which is outside of the scope referenced in the associated action such as LEAVE or RETRY.  In other words, the target of the action will determine the flow of control for the program.  Any UNDO operation must be scoped inside (more deeply nested than) or equal to the block which is the target for the other action otherwise a compiler error will result.

Blocks have "properties" that match a condition name.  Thus there are ERROR, ENDKEY, STOP and QUIT properties.  The existence of a property (implicitly or explicitly) in a given block means that that block provides a predictable sequence of actions in response to the condition (matching that property) being raised.  To explicitly add a property to a block, one uses the ON ERROR, ON ENDKEY, ON STOP or ON QUIT clauses in the block header.  For each of these explicit overrides, one may also specify "UNDO, secondary_action".  UNDO is required and cannot be avoided.  The secondary action is one of those listed above.  If no secondary action is specified, then RETRY is the action executed by default.

The secondary action of an ON phrase can have an explicit label defining its target block.  If no label is specified BUT there is a label defined for the UNDO portion of the ON phrase, then that UNDO label will also define the target block for the secondary action.  If neither portion of the ON phrase defines a label, then the target block will be determined implicitly.  The logic for this is very complicated.  Please see below for details.

The implicit behavior of each block type is different in regards to condition handling.  Since all condition processing is handled based on whether the condition has a matching "property" for a given block, if a block doesn't have a property and that condition is raised, that block will be bypassed and the processing will occur in an enclosing block.  This means that if (for example) a block has the ERROR property, then in the case where an ERROR condition is raised, if that block is the innermost block which has the ERROR property, then that is the level at which implicit UNDO and following implicit action occurs.  The implicit action that occurs is defined differently for each condition, see above .  This default action ONLY OCCURS if the block implicitly has the matching property (e.g. the ENDKEY property when the ENDKEY condition is raised).

Please note that the UNDO action is special since it always occurs first (if it occurs at all), before the secondary action.  UNDO cannot be disabled however it only occurs when a transaction is active in the current or a containing scope.  Some blocks always have a given properties (implicit or explicit) and some blocks can only have explicit properties (i.e. DO).  If any property is not associated with a block, then no processing of that condition will occur in that scope.  This means that the stack unwinds past that block without stopping to do any processing.  Only those properties that are implicitly or explicitly associated with a scope will ever be processed in that scope.

Each property is independent.  For example, if a block has an ERROR property, that has no bearing on how it processes the ENDKEY.

The following are the implicit properties that are inherently provided with each block type:

Block Type
ERROR
ENDKEY
STOP
QUIT
FOR
Y
Y
N*
N*
REPEAT
Y
Y
N*
N*
DO
N* (except if the transaction keyword is present, then there is an implicit ERROR property)
N
N
N
procedure
Y
Y
Y* (only for the "startup procedure")
N*
trigger
Y
Y
N*
N*
editing
Y (for UPDATE except when the ERROR condition is generated by the END-ERROR key and the interactions counter is < 2) / N (for PROMPT-FOR and SET)
N
N
N
function
Y (always LEAVES the function with an unknown value return)
Y (always LEAVES the function with an unknown value return) N
N

(*) All of these entries have been found to be different than documented in the Progress Handbook (see the Block Properties summary table on page 3-3).

The DO block is the only block with no implicit behavior (except in the case where the TRANSACTION keyword or ON phrases are present).  All DO, REPEAT and FOR blocks can have the same set of explicit properties.  If explicitly set, this definition overrides the default action as documented above.

The DO block will have sub-transaction support if they contain ANY ON phrase.  The condition does not matter.  So an ON QUIT UNDO, RETRY is enough to force the DO block to have the sub-transaction property.

Please note that the function block is quite special.  In user defined functions, no user-input blocking statement can be executed.  This means that PROMPT-FOR/SET/UPDATE/INSERT/CHOOSE/WAIT-FOR/PAUSE and READKEY are all illegal, there are very limited choices of editing database fields within a function.  This combined with the fact that functions don't have the transaction property by default means that functions usually are sub-transactions  However, when a direct assignment (e.g. field = expression) occurs, this WILL cause the function block to obtain TRANSACTION level.  There is no way to make a function block a "NO" transaction (lacking the sub-transaction or transaction properties).  The ERROR and END conditions both default to LEAVE and the return value will be the unknown value in these cases.  This cannot be overridden.  Because of this behavior and the fact that one cannot specify an ON xxxx phrase with any of the 4 conditions, it is only possible to get a RETRY at the function block level by using the UNDO, RETRY. language statement when infinite loop protection is disabled (e.g. with the RETRY function). 

Infinite Loop Protection (ILP)

Infinite loop protection is a hidden feature of the block processing that is designed to modify the block behavior as specified by the 4GL programmer in order to avoid an infinite loop.  The idea is that an action that is specified by a 4GL programmer (RETRY or NEXT) which could perpetuate an infinite loop, will be silently converted (by the runtime) to a different action (LEAVE, NEXT or RETURN) in order to cause a change in the loop state that will avoid an infinite loop.  This silent conversion is done based on runtime state which is affected by the flow of control of the program.  So blocks and statements that are conditionally executed will change the state of the runtime and thus modify the conversions that occur.  For this reason, the behavior cannot be duplicated by static code modifications.

In particular, the following behavior is regulated:
This conversion will occur unless ILP has been disabled for the targetted block.  ILP becomes disabled based on runtime state which is maintained implicitly by particular language statements and by the block processing infrastructure of the runtime.

If ILP is NOT disabled, the following conversions occur:



Block Type
Top Level Block/Loop
Conversion
FOR EACH (all forms - even when there is a WHILE or TO/BY present)
no loop
RETRY converted to NEXT
NEXT no conversion, allowed to execute
DO TO/BY
no loop
RETRY converted to NEXT
NEXT no conversion, allowed to execute
REPEAT TO/BY
no loop
RETRY converted to NEXT
NEXT no conversion, allowed to execute
FOR FIRST/LAST (all forms - even when there is a WHILE or TO/BY present) no block
RETRY converted to LEAVE
NEXT converted to LEAVE
DO WHILE
no loop
RETRY converted to LEAVE
NEXT converted to LEAVE
REPEAT WHILE no loop
RETRY converted to LEAVE
NEXT converted to LEAVE
REPEAT
no loop
RETRY converted to LEAVE
NEXT converted to LEAVE
DO
no block
RETRY converted to LEAVE
NEXT converted to LEAVE
external procedure yes block
RETRY converted to RETURN
NEXT converted to RETURN
internal procedure
yes block
RETRY converted to RETURN
NEXT converted to RETURN
function yes block
RETRY converted to RETURN
NEXT converted to RETURN
trigger
yes block
RETRY converted to RETURN
NEXT converted to RETURN
editing
no loop
Infinite loop protection is always disabled in editing blocks (since they an only be called from UPDATE/SET/PROMPT-FOR as part of a user interaction), so no conversions ever occur in targetted editing blocks.

There are 2 cases where this ILP is disabled:
  1. A language statement which blocks for user input is executed before the point in the block where the condition is raised.  Please see Language Statements that Block for User Input for the list.  At the moment that such a statement executes, the infinite loop protection is disabled.  Since editing blocks are inherently processed as part of user input, infinite loop protection is meaningless for such loops.
  2. The RETRY built-in function is executed before the point in the block where the condition is raised.  At the moment that function executes, the infinite loop protection is disabled.
Until one of these 2 cases is executed, any CONDITION that generates a RETRY will result in a NEXT, LEAVE or RETURN. Only when the infinite loop protection is disabled will RETRY result in a RETRY!  Likewise NEXT may be allowed, but depending on the block type it may be converted to a LEAVE or RETURN.

All block types have retry support.  In other words, it is possible to retry any block including non-looping blocks and the top-level blocks (like a built-in function or internal procedure).  But whether the RETRY will be allowed or not is determined by the ILP processing at runtime.

It is important to note that the NEXT language statement is NOT affected by ILP.  ILP only affects ON phrases and the UNDO language statement.

A note in the UNDO language statement states that if "nothing changes during a RETRY of a block" it will convert the RETRY into a NEXT or LEAVE.  As far as can be determined using application code, the 4GL runtime does not actually monitor the state of  variables or other resources to detect if "nothing changes".  It looks like the disabling of ILP is taken as a proxy for whether things have changed in the current iteration.

The Progress READKEY documentation has a specific note regarding a special implementation of infinite loop protection.  It states that a count is used to convert UNDO, RETRY into UNDO, NEXT as well as the conversion of UNDO, NEXT into UNDO, LEAVE. While testing has not found any special counter behavior for READKEY, there is a "stutter quirk" (a counter based difference in ILP behavior) that has been found associated with ON ENDKEY UNDO, RETRY when it is used on certain block types.

The following code will cause a kind of "stuttering" where RETRY will sometimes be allowed to occur due to the state of a counter.  This stutter quirk only comes into play when an ENDKEY condition is raised and is handled by the "ON ENDKEY UNDO, RETRY" phrase.  Consider this code:

def var i as int no-undo.

do i = 1 to 5 on endkey undo, retry:
   message i.
   apply "endkey".
end.

The results:

1
2
3
3
4
4
5
5

The first 2 retries are converted to a NEXT but the 3rd RETRY is allowed to go forward as a RETRY (no conversion occurs).  Then the 4th RETRY is converted to a NEXT but the 5th RETRY is a RETRY.  This only occurs with the execution of the actions associated with an ON phrase and ONLY with ENDKEY.  If any other condition is targetted (e.g. ERROR), this will not occur.  Likewise, if any other action is specified (NEXT instead of RETRY), then this behavior does not occur.

Finally, this stutter mode is only seen in ON ENDKEY UNDO, RETRY phrases for the following block types:

Determining the Target and Meaning of UNDO, LEAVE, NEXT and RETRY

Each of these actions can explicitly reference a specific labeled block (e.g. LEAVE inner) or will implicitly reference a block.   In each of these cases, the action always operates in regards to a specific block. In these implicit cases (one in which no block label was specified in the 4GL source code), the 4GL compiler/runtime must determine the block which is referenced.  There are complex rules by which the UNDO, LEAVE, NEXT and RETRY actions are "bound" to a specific block as its target.  These actions do not always operate on the current block or even on an explicitly defined block target. This target block resolution is complicated by the block structure in which the on phrase or language statements are contained.  In particular, the target can reference enclosing blocks implicitly or explicitly.  The block properties of the enclosing blocks will also affect any implicit target calculation.

There are 3 ways that these actions can be invoked:
The UNDO language statement and ON phrases have the ability to specify up to 2 labels (one for the UNDO and one for the other action).

There are only 4 types of blocks in Progress which can be labeled: DO, REPEAT, FOR and EDITING.  Thus, only these block types can be explicitly targetted by the UNDO, LEAVE, NEXT or RETRY actions.  But when implicit block targets are calculated by Progress, it is possible for a non-labeled block to be the target of an action.  The non-labeled block types are special "top-level" blocks (EXTERNAL PROC, INTERNAL PROC, TRIGGER and USER DEFINED FUNCTION).  None of those non-labeled block types are iterating blocks.  This means that NEXT must be translated into something else (LEAVE in Progress terms).  In Progress, LEAVE on a top-level block is the same as a RETURN (with no modifiers or returned data).  So if LEAVE or NEXT do implicitly target a top-level block, then the behavior will logically be converted into a RETURN.  RETRY on a top-level block will be converted to a RETURN if infinite loop protection is NOT disabled.

All blocks in Progress implicitly have the ERROR property except for the DO and EDITING (PROMPT-FOR or SET forms) blocks.  The DO only will have the ERROR property when it has an ON ERROR phrase or if it has the TRANSACTION keyword.

Note that the ERROR property that exists for DO blocks and for the UPDATE form of EDITING blocks (and for the special "hidden" block in non-EDITING UPDATE blocks) is special in that it is not in effect in the following case:
  1. The block does not also have the ENDKEY property (EDITING blocks never have the ENDKEY property and DO blocks only have it if it is explicitly specified in an ON ENDKEY phrase); AND
  2. The ERROR condition was raised by the END-ERROR key (or an APPLY of that key); AND
  3. The current interactions counter is less than 2 (see END-ERROR for details).
This is runtime state that will modify how the ERROR property is processed when the ERROR condition is generated by the END-ERROR key. This means that an ERROR condition generated by END-ERROR key in an UPDATE block (EDITING or not) is always forwarded on to the containing block, even though all other ERROR conditions are handled as an UNDO, RETRY.

Most importantly, for the purposes of resolving unlabeled actions to blocks with the ERROR property, EDITING blocks (even UPDATE EDITING blocks) are treated as if they DO NOT have the ERROR property.

Considering this, the following are the possible implicit block target resolution algorithms:

Code Description
NEAR-ERR The nearest enclosing block with the ERROR property.

Example 1

outer:
repeat:
   inner:
   do on error undo, retry:
      <statement>
   end.
end.

In example 1, the nearest enclosing block (to the <statement>) with the ERROR properly is inner.

Example 2:

outer:
repeat:
   inner:
   do:
      <statement>
   end.
end.

In example 2, the nearest enclosing block (to the <statement>) with the ERROR property is outer.
CURRENT The "current" block.  This is the block which directly encloses the UNDO, NEXT or LEAVE language statement.  Or in the case of an ON phrase, it is the block upon which the ON phrase is specified.

outer:
repeat:
   inner:
   do on error undo, RETRY:
      <statement>
   end.
end.

In this example, for both the <statement> and the RETRY the current block is inner.
AS-UNDO The other action will be targetted at the same block as the associated UNDO is targetted.  If UNDO is targetted (implicitly or explicitly) to inner then the other action will be targetted to inner.

Assuming that the directly containing block is labeled "inner" and that block is contained inside a block labeled "outer", the following are possible formations of the UNDO language statement and ON phrases:

Feature UNDO Target Other Action Other Action Target Static Conversion Possible Compiler Error
UNDO. NEAR-ERR RETRY (implicit) AS-UNDO
n/a
UNDO inner. inner RETRY (implicit) AS-UNDO
If inner does not have the ERROR property.
UNDO outer. outer RETRY (implicit) AS-UNDO
If outer does not have the ERROR property.
UNDO, RETRY. NEAR-ERR RETRY AS-UNDO

UNDO, LEAVE. NEAR-ERR LEAVE AS-UNDO *
UNDO, NEXT. NEAR-ERR NEXT AS-UNDO *
UNDO, RETRY inner. NEAR-ERR RETRY inner
If inner does not have the ERROR property, then UNDO would choose a different block than inner.  Since RETRY is targetted at inner, this is a problem.  UNDO and RETRY must always be targetted at the same block.
UNDO, LEAVE inner. NEAR-ERR LEAVE inner
If inner does not have the ERROR property, then UNDO would choose a different block (a more enclosing one) than inner.  Since LEAVE must be targetted at a block that is the same as the UNDO target or which encloses the UNDO target, this is a problem.
UNDO, NEXT inner. NEAR-ERR NEXT inner
If inner does not have the ERROR property, then UNDO would choose a different block (a more enclosing one) than inner.  Since NEXT must be targetted at a block that is the same as the UNDO target or which encloses the UNDO target, this is a problem.  

In addition, if inner is not an iterating block the compiler fails.
UNDO, RETRY outer. NEAR-ERR RETRY outer
This is only possible if an inner block (one that more directly encloses the statement) does not have the ERROR property.  If inner does have such properties, then it would be implicitly chosen as the UNDO target and that would mismatch with the outer block as the action target, causing the compiler to error.
UNDO, LEAVE outer. NEAR-ERR LEAVE outer
The outer block or one of its nested blocks (which still encloses this statement) must have the ERROR property, otherwise the LEAVE will be targetted inside the UNDO target, which is not allowed.
UNDO, NEXT outer. NEAR-ERR NEXT outer
The outer block or one of its nested blocks (which still encloses this statement) must have the ERROR property, otherwise the NEXT will be targetted inside the UNDO target, which is not allowed.


In addition, if outer is not an iterating block the compiler fails.
UNDO inner, RETRY. inner RETRY AS-UNDO
inner must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO inner, LEAVE. inner LEAVE AS-UNDO * inner must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO inner, NEXT. inner NEXT AS-UNDO * inner must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO inner, RETRY inner. inner RETRY inner
inner must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO inner, LEAVE inner. inner LEAVE inner
inner must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO inner, NEXT inner. inner NEXT inner
inner must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO inner, RETRY outer. n/a n/a n/a
Invalid syntax, compiler will always error because RETRY and UNDO must target the same block.
UNDO inner, LEAVE outer. inner LEAVE outer
inner must have ANY property (not just ERROR) otherwise there is a compiler error.  (outer has no property dependencies)
UNDO inner, NEXT outer. inner NEXT outer
inner must have ANY property (not just ERROR) otherwise there is a compiler error.  (outer has no property dependencies)

In addition, if outer is not an iterating block the compiler fails.
UNDO outer, RETRY. outer RETRY AS-UNDO
outer must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO outer, LEAVE. outer LEAVE AS-UNDO * outer must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO outer, NEXT. outer NEXT AS-UNDO * outer must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO outer, RETRY inner. n/a n/a n/a
Invalid syntax, compiler will always error because RETRY and UNDO must target the same block.
UNDO outer, LEAVE inner. n/a n/a n/a
Invalid syntax, compiler will always error because LEAVE must target the same or a more enclosing block as UNDO.
UNDO outer, NEXT inner. n/a n/a n/a
Invalid syntax, compiler will always error because NEXT must target the same or a more enclosing block as UNDO.
UNDO outer, RETRY outer. outer RETRY outer
outer must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO outer, LEAVE outer. outer LEAVE outer
outer must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO outer, NEXT outer. outer NEXT outer
outer must have ANY property (not just ERROR) otherwise there is a compiler error.

In addition, if outer is not an iterating block the compiler fails.
ON <condition> UNDO. CURRENT RETRY (implicit) AS-UNDO
CURRENT must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO inner. inner RETRY (implicit) AS-UNDO
inner must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO outer. outer RETRY (implicit) AS-UNDO
outer must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO, RETRY. CURRENT RETRY AS-UNDO

ON <condition> UNDO, LEAVE. CURRENT LEAVE AS-UNDO *
ON <condition> UNDO, NEXT. CURRENT NEXT AS-UNDO * No error even if the implicit target block is not an iterating block.
ON <condition> UNDO, RETRY inner. CURRENT RETRY inner
CURRENT must be the same block as inner (RETRY and UNDO must always be at the targetted to the same block).
ON <condition> UNDO, LEAVE inner. CURRENT LEAVE inner
CURRENT must be the same block as inner or enclosed by inner.  The LEAVE target block needs no properties in particular.
ON <condition> UNDO, NEXT inner. CURRENT NEXT inner
CURRENT must be the same block as inner or enclosed by inner.  The NEXT target block needs no properties in particular.

However, if inner is not an iterating block the compiler fails.
ON <condition> UNDO, RETRY outer. n/a n/a n/a
Invalid syntax. The UNDO will target the current block which is more deeply nested than the RETRY target, which causes the compiler to fail.
ON <condition> UNDO, LEAVE outer. CURRENT LEAVE outer
The LEAVE target block needs no properties in particular.
ON <condition> UNDO, NEXT outer. CURRENT NEXT outer
The NEXT target block needs no properties in particular.

If outer is not an iterating block the compiler fails.
ON <condition> UNDO inner, RETRY. inner RETRY AS-UNDO
inner must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO inner, LEAVE. inner LEAVE AS-UNDO * inner must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO inner, NEXT. inner NEXT AS-UNDO * inner must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO inner, RETRY inner. inner RETRY inner
inner must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO inner, LEAVE inner. inner LEAVE inner
inner must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO inner, NEXT inner. inner NEXT inner
inner must have ANY property (not just ERROR) otherwise there is a compiler error.

If inner is not an iterating block the compiler fails.
ON <condition> UNDO inner, RETRY outer. n/a n/a n/a
Invalid syntax. The UNDO will target the inner block which is more deeply nested than the RETRY target, which causes the compiler to fail.
ON <condition> UNDO inner, LEAVE outer. inner LEAVE outer
inner must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO inner, NEXT outer. inner NEXT outer
inner must have ANY property (not just ERROR) otherwise there is a compiler error.

If outer is not an iterating block the compiler fails.
ON <condition> UNDO outer, RETRY. outer RETRY AS-UNDO
outer must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO outer, LEAVE. outer LEAVE AS-UNDO * outer must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO outer, NEXT. outer NEXT AS-UNDO * outer must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO outer, RETRY inner. n/a n/a n/a
Invalid syntax. The UNDO and RETRY must target the same block, otherwise the compiler fails.
ON <condition> UNDO outer, LEAVE inner. n/a n/a n/a
Invalid syntax. LEAVE must target the same or a more enclosing block as UNDO is targetting.
ON <condition> UNDO outer, NEXT inner. n/a n/a n/a
Invalid syntax. NEXT must target the same or a more enclosing block as UNDO is targetting.
ON <condition> UNDO outer, RETRY outer. outer RETRY outer
outer must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO outer, LEAVE outer. outer LEAVE outer
outer must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO outer, NEXT outer. outer NEXT outer
outer must have ANY property (not just ERROR) otherwise there is a compiler error.

If outer is not an iterating block the compiler fails.


In the above table <condition> can be ERROR, ENDKEY, STOP or QUIT.  The particular condition in question does not matter for the purposes of the behavior documented above.

The "Static Conversion Possible" column denotes the difference between runtime conversion of actions (infinite loop protection) versus static (compile time) conversion of actions.  A static conversion is one that will not change based on runtime state, but instead is based on data known at the time the source code is compiled.

At runtime, a RETRY action will be converted to a NEXT, LEAVE or RETURN (depending on target block type) if infinite loop protection is NOT disabled.

If the action is NEXT and was caused by an ON phrase or UNDO statement AND the target block is a non-iterating inner block (it is not a top-level block), then the NEXT will be converted to a LEAVE.

If the action is NEXT or LEAVE and the target block matched is a top level block (external procedure, internal procedure, function or trigger), then the action will be converted to a RETURN.

Rules for LEAVE statement processing:
  1. If there is an explicit label, that block is left no matter what properties it may have or not have.
  2. If there is no label, then the implicit target block must be found.  The nearest enclosing block type of the following will be the target:
  3. Please note that the simple DO block is the only inner block type that LEAVE will NEVER implicitly target.
  4. The properties of the blocks have nothing to do with this search process.
  5. If no inner block is found with the above process, then the LEAVE is considered to be targetting the nearest top-level block.  For this reason, the LEAVE statically converts to a RETURN.
Rules for NEXT statement processing:
  1. If there is an explicit label:
  2. If there is no label, then the implicit target block must be found.  This is a 2 phase process.
    1. The nearest enclosing looping block type will be the target:
    2. If there is no enclosing looping block between the NEXT statement and the nearest enclosing top-level block, then a search will be made to find the nearest enclosing FOR FIRST or FOR LAST block.  If one of these blocks is found, then the NEXTwill be statically converted to a LEAVE and targetted at that nearest enclosing FOR FIRST/LAST block.
  3. The 2nd phase of the implicit block search process is not attempted unless the first phase fails to find a match.  This means that a more deeply nested FOR FIRST/LAST block will NOT have preference to an outer looping block.  The looping blocks will always take precedence over the FOR FIRST/LAST.
  4. Please note that the simple DO block is the only inner block type that NEXT will NEVER implicitly target.
  5. The properties of the blocks have nothing to do with this search process.
  6. If no inner block is found with the above 2 phase process, then the NEXT is considered to be targetting the nearest top-level block.  For this reason, the NEXT converts to a RETURN.

Transactions

Progress 4GL transaction processing:
  1. Defines the start and end of transactions and sub-transactions.
  2. Implements the proper nesting of sub-transactions matching the block structuring of the program.
  3. Rolls back (UNDOes) database, variable, temp-table and work-table values to a known state upon certain "conditions" occurring.
  4. Commits all changes (permanently) to databases, variables, temp-tables and work-tables at the natural successful end of an active transaction. 
  5. Implements scoping facilities for reou
A transaction begins at the start of any block while the following are true:
A sub-transaction (a nested transaction) begins at the start of any block while the following are true:
Only 1 transaction can ever be active per user context.  Other nested blocks that have transaction properties, become nested sub-transactions.  For more details on which blocks can open a transaction, see page 12-17 of the Progress Programming Handbook.

Any block that has no direct access to the database cannot open a transaction (unless it is a block that has the TRANSACTION keyword).  Thus the same block type which would normally open a transaction when database updates or exclusive database reads are present, will not open a transaction (or in some cases a sub-transaction) when no database access is contained.

FOR block transaction anomaly: There is a case where reading a record in a FOR EACH with exclusive lock will not create a transaction.  In particular, if the buffer scope for the FOR EACH has been expanded beyond the FOR EACH block (there is a free reference outside that block) AND there is no data editing or other cause to create a transaction, then the FOR EACH will not create a transaction.  Also note that ALL buffers referenced (in the case where there are > 1 record phrases) must be scoped outside the FOR block in order for this anomaly to trigger.   If even 1 buffer (referenced in the FOR statement itself) is scoped to the FOR block itself, then the EXCLUSIVE-LOCK alone is enough to trigger this behavior.  When this case occurs, the FOR block is still treated as a sub-transaction.  See Progress Knowledge Base article KB13974.

Note that the rules defining a sub-transaction are a super-set of the rules defining a transaction.  Since a given block can be invoked with a call stack that is different depending on the path taken through the program, when a particular code block is invoked it is possible that sometimes a transaction will be active and other times not.  This means that the following cases are possible:
  1. The start of the block will start either a transaction or a sub-transaction, depending only on whether a transaction is already active.
  2. The start of the block will start a sub-transaction if a transaction is already active, but otherwise a transaction will not be started.
  3. The start of the block has no transaction or sub-transaction implications (notably any DO block without the ON ENDKEY, ON ERROR or TRANSACTION keywords).
To duplicate this behavior, the determination of whether a block opens a transaction or a sub-transaction must be handled at runtime.  Thus the same block will process differently depending on runtime conditions.

In this way, procedures are treated just like other block types.  Calls to procedures within an active transaction cause the procedure to open a sub-transaction.  If a procedure would not otherwise open a transaction and a transaction is not active, then a transaction is not started at the beginning of a procedure.  It is important to note that even though a procedure may not open a transaction, a nested block may still open a transaction.  For example, a nested FOR EACH that updates the database will have a transaction scoped to the FOR EACH block rather than to the containing procedure.

The end of any block that defines a transaction or sub-transaction defines the end of the corresponding transaction/subtransaction.  When a transaction scope ends successfully, all changes made during that scope are committed to the database.  When a sub-transaction scope ends successfully, all changes made during that scope are effectively added to the "transaction set".  This transaction set can be UNDOne as a set or committed as a set but once the end of a subtransaction occurs, one loses the ability to UNDO only that portion of the transaction.

In the event of any condition being raised, an UNDO (rollback) will occur (at either the transaction or sub-transaction level) and then a change of control flow to an enclosing scope will occur (to the same scope as the UNDO or to a scope that encloses the scope that was UNDOne.  The implicit or explicit definition of this behavior is encoded in the source application's block properties.  Each condition has a different set of default behavior in this regard.  For more details on determining the start and end of transactions, see page 12-8 of the Progress Programming Handbook

This ability to have nested sub-transactions allows blocks to be used to structure Progress 4GL programs such that the proper level of UNDO and other actions can be associated with the granularity or lack of granularity which the programmer desires.  This allows any active work (work that has not been added to the transaction set (defined above) to be rolled back partially (by UNDOing at the sub-transaction level) or for all work (the full transaction set) to be backed out (by UNDOing at the transaction level).

The sub-transaction commit point has the other behavior (in the case of a loop), that the "backup set" of all undoable resources (variables, database fields...) is updated at this same point.  This means that if a variable is defined external to a scope and the first iteration of a given sub-transaction loop updates its value and then commits, any this new value will be result of any UNDO that occurs (whether the variable was modified or not) until the next sub-transaction commit.  This can be described as updating the backup set at iteration.

Once a transaction is active, database fields, variables, temp-table and work-table values are saved off at the start of the block defining the transaction and again at the start of the block defining every sub-transaction.  One can UNDO all of these data structures to a known state by defining the block properties of each block such that the possible conditions implicitly UNDO the correct block OR the possible conditions are explicitly defined to UNDO a specific block.  In particular, labeled blocks can be specified as the target of an UNDO (or other action) by using the "ON <condition>" clause in a block.  If that named condition occurs, then the specified actions would be invoked on the specified blocks.

UNDO can ONLY EVER be done when a transaction is active.  The scope of the rollback is either explicitly defined by the programmer or is implicit, but in either case, each condition type matches up with a specific (and potentially different) rollback scope.  A rollback scope can be specified at any block that defines a transaction or sub-transaction.  When a transaction is not active, UNDO is a null operation.

There is no way to disable UNDO for a block, although individual local variables and temp-table/work-table fields can be specified as NO-UNDO, which will exclude them from the transaction rollback.  Database field changes are always rolled back.

Whether a block is a loop or not does not change how transaction processing works.  In fact, even a non-looping block can be retried.

Once the successful end of a transaction block is encountered, Progress commits all changes (permanently) to databases, variables, temp-tables and work-tables.  At the successful end of a sub-transaction, backup copies of the database fields (buffers), variables, temp-tables and work-tables are discarded.  A commit is a write of the modified value(s) back to the database or memory as necessary for the resource type being processed.  In case of the database, temp-tables and work-tables, this includes: When the natural flow of control of the procedure flows through an END statement for a block, this is considered the "successful end" of the block.  Please note that when LEAVE and RETURN are explicitly coded (and not triggered as a result of a CONDITION) then these are also considered the successful end of the containing block.  This means that all changes are committed in these cases.

Silent Error Mode (NO-ERROR keyword)

Progress provides the NO-ERROR keyword and a system handle called ERROR-STATUS to provide the ability to temporarily (for the duration of a single assignment or language statement) force a silent error mode.  In this mode, any generated errors will not raise a condition but will instead store the error number and error text in a list of errors.  Then the statement or assignment will silently return.

The ERROR-STATUS handle is used to access this stored data.

Language Statements Which Can Raise Conditions


Statement
Condition

Notes
ERROR
STOP
QUIT
ENDKEY
END-ERROR*
APPLY +
+
+
+
+
this statement can generate any event on a procedure including these special events that match "conditions" (it can also generate events for widgets but this is conceptually quite different from generating the conditions listed in this table)
ASSIGN +
-
-
-
-

BUFFER-COMPARE
+
-
-
-
-

BUFFER-COPY
+
-
-
-
-

CHOOSE +
+
-
+
+
interactive
COMPILE
+
-
-
-
-
provided here for completeness
CONNECT
+
-
-
-
-

COPY-LOB
+
-
-
-
-

CREATE
 ALIAS
 automation object
 CALL
 DATABASE
 SAX-READER
 SERVER-SOCKET
 SOCKET
 WIDGET-POOL
+
-
-
-
-

DDE
+
-
-
-
-
all kinds of DDE statement can raise ERROR
DELETE
 OBJECT
 PROCEDURE
 WIDGET-POOL
+
-
-
-
-

DICTIONARY
+
-
-
-
-
"The DICTIONARY statement is equivalent to RUN dict.p: it runs the Progress procedure called dict.p. Progress uses the regular search rules to find the dictionary procedure. The dictionary procedure is part of the Progress system software."
DISCONNECT
+
-
-
-
-

DISPLAY +
-
-
-
-

EDITING phrase +
+
+
+
+
The event processing inside the phrase must apply all events explicitly, either using APPLY statement or by using RETURN ERROR/STOP/QUIT statements. Also, this phrase is active only if input is from terminal. In general case it is necessary to check phrase body to determine which events can be generated.
EMPTY TEMP-TABLE +
-
-
-
-

ENTRY
+
-
-
-
-
function and statement
EXPORT
+
-
-
-
-

FIND +
-
-
+
-

IMPORT +
-
-
+
-

INPUT FROM +
-
-
+
+

INSERT +
+
-
+
+
interactive
LOAD
+
-
-
-
-

OUTPUT TO +
-
-
-
-

PROCESS EVENTS +
+
-
+
+
behaves like interactive
PROMPT-FOR +
+
-
+
+
interactive (equivalent to ENABLE, WAIT-FOR, DISABLE sequence of statements)
PUT-KEY-VALUE
+
-
-
-
-

QUIT
-
-
+
-
-
this statement seems the only way to generate this event (except APPLY)
RAW-TRANSFER
+
-
-
-
-

READKEY
+
-
-
+
-
Actually, the condition is not directly raised, instead the LASTKEY value is set to -1 (on ERROR) or -2 (on ENDKEY).  To raise the condition one then must "APPLY LASTKEY".
RELEASE
+
-
-
-
-
all kinds of RELEASE statement can raise ERROR
REPOSITION
+
-
-
-
-

RUN
+
+
-


all kinds of RUN statement can raise ERROR and STOP
SAVE CACHE +
-
-
-
-

SET +
+
-
+
+
interactive (equivalent to PROMPT-FOR and ASSIGN)
STOP
-
+
-
-
-

SUBSCRIBE
+
-
-
-
-

UNLOAD
+
-
-
-
-

UPDATE +
+
-
+
+
interactive
USE
+
-
-
-
-

VALIDATE
+
-
-
-
-

WAIT-FOR +
+
-
+
+
Waits for specified event to occur. Effectively this does allow other keyboard-driven events to occur as well.

Notes:
  1. END-ERROR is a key function that behaves as two separate events (see 5-19 in the Progress Programming Handbook): for the first input operation (any of the Language Statements that Block for User Input) of the current block's iteration (if it is a loop), raises the ENDKEY condition; for subsequent input operations, raise the ERROR condition.  Note that in a WAIT-FOR, the resulting condition is always ENDKEY.  In UPDATE/SET/PROMPT-FOR... (and for non-WAIT-FOR stmts), it differs based on whether this is the first or a subsequent interaction in this *iteration* of the block.  For example, in a REPEAT block that has 2 SET statements, on any given iteration of the block if you press the END-ERROR key while in the 1st SET statement the result is always ENDKEY condition and if you press the END-ERROR key in the 2nd SET statement, the result is the ERROR condition.
  2. Inner loops may have ON ERROR, ON ENDKEY, etc. phrases which refer to outer loop using appropriate labels.
  3. The default behavior if the ON ENDKEY phrase is omitted is UNDO, LEAVE. If ON ENDKEY phrase is partially specified then default is to LEAVE if other (NEXT, RETRY, or RETURN) action is not explicitly set.
  4. RETURN ERROR raises ERROR condition in calling procedure.
  5. The default behavior if the ON ERROR phase is omitted for REPEAT and FOR EACH blocks - UNDO, RETRY. If Progress detects possible infinite loop in FOR or iterating DO blocks then UNDO, NEXT is performed instead.
  6. It is unclear if UNSUBSCRIBE supports NO-ERROR option. Syntax diagram does not mention it but further description mentions it as a possible way to avoid raising ERROR.
  7. From the description it is not clear if the DICTIONARY is an equivalent to "RUN dict.p" or "RUN dict.p NO-ERROR".
  8. Quote from the reference: If you use READKEY, it intercepts any input from the user. Thus no widgets receive input. To pass the input to a widget, you must use the APPLY statement.  READKEY and widget usage are usually kept separate.  READKEY is normally used as part of the editing block functionality while widget events are part of the wait-for event driven functionality.  As noted above with the APPLY statement it is important to realize that widgets and events are a very different category of discussion than these block conditions.
  9. UNDO statement does not raise conditions by itself but may change execution flow if followed by LEAVE, NEXT, RETRY or RETURN. Also, in form UNDO, RETURN ERROR is may raise ERROR condition in the calling procedure (like RETURN ERROR).
  10. The default ENDKEY and ERROR processing is defined only for REPEAT and FOR loops, other loops must specify such processing explicitly using ON ERROR and ON ENDKEY phrases.

Language Statements that Block for User Input

The presence of language statements that block for user input modifies or controls the behavior of the END-ERROR key and of the infinite loop protection (ILP) for RETRY.

The following language statements do block for user input:
  1. CHOOSE
  2. ENABLE
  3. INSERT
  4. MESSAGE (only with VIEW-AS ALERT-BOX, UPDATE or SET)
  5. PROMPT-FOR
  6. READKEY on terminal (but not when reading from a file)
  7. SET
  8. UPDATE
  9. WAIT-FOR
  10. PAUSE (waits for "ANY-KEY") - Note that this statement is a special case in that it blocks for user input, but only sometimes does it  disable infinite loop protection.  For examples of this behavior, see testcases pause_infinite_loop_protection*.p.
There is a special case for the PAUSE statement.  Whether PAUSE does or does not disable infinite loop protection, is dependent upon the block type:
Perhaps surprising is that the following do not block for user-input:
  1. APPLY
  2. HIDE
  3. PROCESS-EVENTS
  4. VIEW

Language Statements that Update/Modify the Database

Absent the TRANSACTION keyword, only the presence of database updates or exclusive reads will cause a block to start a transaction.  The following language statements start a transaction, only if they reference a database table/field (after a record buffer has been created or filled).  If temp-table, work-table or normal variables are used, then a transaction is not started.  In addition, some of these statements can take a record as an option (as a shorthand way of referencing all fields in that record).  If this record is a TABLE, this will trigger a transaction.
  1. ASSIGN
  2. BUFFER-COPY
  3. CREATE
  4. DELETE
  5. INSERT
  6. RAW-TRANSFER
  7. SET
  8. UPDATE
  9. Explicit field-level use of the assignment operator on database records will start a transaction.
Note that the assignment operator can only be used without error on database records that have SHARE-LOCK or EXCLUSIVE-LOCK.  If the record is opened with NO-LOCK, then an ERROR will be generated.  The NO-ERROR keyword can be used on an assignment to suppress the error generation (the fact that it occurred can still be discovered via the system handle ERROR-STATUS) but this doesn't make the update successful.  In other words, one cannot successfully assign to a database record that is NO-LOCK but this still DOES mark the associated block as a transaction.

Usage of a BROWSE widget  (see DEFINE BROWSE) is a "black-box" language statement which reads from/writes to the database on behalf of the programmer.  Any BROWSE that uses the ENABLE keyword is not read-only (in Progress terms it is "updateable") and it will update/modify the database. All transaction support for updateable BROWSE is hidden inside the "black-box".  For this reason, the transaction property does not attach to the containing block.

It is not known at this time if statements such as SETUSERID implicitly access the database on the caller's behalf.  There may be other such language statements that read or update the database explicitly.

Language Statements that Read the Database with EXCLUSIVE LOCK

Absent the TRANSACTION keyword, only the presence of database updates or exclusive reads will cause a block to start a transaction.  The following language statements read from the database into a record buffer.  A lock can be exclusive explicitly with the EXCLUSIVE-LOCK keyword.
  1. FIND
  2. GET (with an explicit EXCLUSIVE-LOCK OR with no lock qualifiers when referencing a query with EXCLUSIVE-LOCK)
  3. FOR (all forms - EACH, FIRST, LAST and unqualified record phrase)
If temp-table or work-table records are read, then a transaction is not started.  Only exclusive database reads cause a transaction.

Usage of a BROWSE widget  (see DEFINE BROWSE) is a "black-box" language statement which reads from/writes to the database on behalf of the programmer.  BROWSE widgets are read-only when there is no use of the ENABLE keyword (in Progress terms it is not "updateable") and these can generate reads with EXCLUSIVE-LOCK (this can be set in the DEFINE BROWSE statement).  If the DEFINE BROWSE does not explicitly use the EXCLUSIVE-LOCK keyword, then even if the query was opened EXCLUSIVE-LOCK this is overridden to NO-LOCK (by default) or SHARE-LOCK (if specified).  So only a browse defined explicitly with EXCLUSIVE-LOCK will open the query as EXCLUSIVE-LOCK.  However, since such an approach is only allowed inside a block which already has transaction support (it must be placed inside a DO TRANSACTION or REPEAT TRANSACTION), the containing block will already have transaction support.  For this reason, browse usage can be ignored.

The REPOSITION statement fetches a record when the query is associated with a BROWSE widget.  Since the transaction processing associated with a browse in hidden inside the browse's "black-box", so also is any effect of the hidden/implicit record reading that occurs with this statement.  In other words, although this statement can have an implicit record read associated, there is no case where it causes the containing block to become a transaction.

It is not known at this time if statements such as SETUSERID implicitly access the database on the caller's behalf.  There may be other such language statements that read or update the database explicitly.   It is likely that such statements handle transactions internally (as a black-blox rather than associating a transaction with the calling block).

High Level Implementation Options

Many of the features that Progress provides relate to the proper flow of control in the resulting application.  While problems such as data processing fidelity were relatively easy to centralize or hide inside the wrapper classes , features related to flow of control are not easily centralized.   This is due to the fact that the all control flow constructs in Java cannot be split across two or more methods.  For example, one cannot centrally define catch () blocks that are then used in multiple places.  There must be a catch block in every method in which the matching logic must be triggered.  While one can take the user-defined logic and move that into methods that are called from many locations, all flow control constructs are built-in language features and they are essentially monolithic in the Java language.

The Progress feature set is a complex one with much implicit behavior that must be duplicated.  In many cases, this means that there will be more code in Java where no such code appears in the Progress source.  This has support, maintenance and conversion implications (the greater the difference in source and target tree structure, the greater the effort to convert).  For this reason, one must consider how to obtain complete fidelity while centralizing and hiding as much logic as possible in utility classes, leaving the generated source cleaner and more natural to maintain/support.

The following are the most feasible implementation options to provide complete fidelity with the Progress language/runtime feature set:

Option
Description
Pros
Cons
explicit hard coding in generated source
All control flow, logic and data processing is explicit in the generated source code.  Some features can be helped by hiding things like undo processing in the data type wrappers or utility classes like streams (with cleanup processing).  However, the core problem here is that the generated code still must hook into this processing.  This will take the form of try/catch/finally blocks, instrumentation calls to push/pop scopes, manufactured labels/bogus outer loops to handle things like retry and call-backs to utility classes to access or invoke that logic that can be centralized.
  1. Simple to understand and implement.
  2. Easiest to debug.
  3. Good performance (considering what has to be done).
  4. Most independent of external tools.
  1. Source code is most verbose of all options.
  2. Since most code is affected by the implicit features, this extra code will be in many/most locations.
  3. More effort later to convert any generated code to a more native Java approach.
indirect call mechanism using a utility class to implement the logic In this model, each block that has special properties would be implemented in 2 pieces:
  1. An "external" interface that exposes a well-known named method and interface.  This method would use reflection to obtain a Method object, it would package its parameters as an Object[] and invoke a special BlockProperties class that in turn would use the Method and Object[] to indirectly invoke the private worker.  If the return was not void, the Object would be cast to the proper type and then returned to the real caller.
  2. A private worker method that actually contains the converted logic.  This would be invoked via reflection from the external interface method.
The point here is that the block properties would not be emitted into the generated source, but would be centralized in the BlockProperties class.  Since the block properties class is just invoking the private worker as a method call, this method call can be inside a hand coded try/catch/finally with all special Progress-compatible logic completely hidden in this central location.
  1. Strong level of centralization.
  2. Source code is reasonably clean (but not completely clean) compared to the hard coded approach above.
  1. External interface methods have to be created for each public interface, for each internal procedure or function and for each block.  Thus the source code will still look different from the original and from how one would want to do it if manually coding.
  2. Harder to debug than the hard coded approach above.
  3. DO/REPEAT/FOR blocks would have to be refactored into methods which makes this extremely complicated (since all data must be either promoted to instance members, passed as parameters or encapsulated into another class and passed down.  This is made harder because anything nested inside eachblock has to have its data-access requirements "rolled up" into the containing block.
  4. Using reflection is a performance penalty, especially in the Mehtod object lookup (although the resulting Method could be cached in a static member).
bytecode instrumentation Generate the Java code just as one would normally write it by hand coding but after javac creates the resulting class files, rewrite the class file (statically as a 2nd compile step or dynamically at load time via a custom classloader) to instrument all the same logic and processing that would have been implemented explicitly in the source as in the hard coded option above.
  1. Cleanest source code.
  2. Easiest to convert away from Progress semantics and to a more natural Java approach.
  3. Performance is as good as the hard coded approach.
  1. Hardest to debug since there is truly hidden processing (only in the byte code and not in the source).
  2. Editing this code will likely require modifying some form of additional input (a second hints/annotations file or possibly some form of inline annotations) in order to specify the block properties etc... that must be applied (since it is not implicit in the bytecode) and the locations to which to apply this instrumentation. This makes maintenance harder than just editing a single source file.

At this time, the hard coded source approach is the simplest and easiest to achieve.  As a likely future optimization, the bytecode instrumentation approach will probably be provided in the future as an option.

Implementation Details

As mentioned above, Progress 4GL implements a great deal of its transaction and block behavior in an implicit manner.  One of the advantages Progress has in its interpreter is the embedding of knowledge about each block open and close and each loop iteration.

All Progress conditions will be mapped to custom Java exceptions.  This will allow the generation of such conditions to unwind the stack and to be caught/processed/rethrown where needed.  The base class will be com.goldencode.p2j.util.ConditionException which will allow for a common catch {} when this is useful.  Each of the 4 conditions will be mapped to a subclass:

Condition
Class
ERROR com.goldencode.p2j.util.ErrorConditionException
ENDKEY com.goldencode.p2j.util.EndConditionException
STOP
com.goldencode.p2j.util.StopConditionException
QUIT
com.goldencode.p2j.util.QuitConditionException

In the Java implementation, a TransactionManager class provides a central clearinghouse that tracks block open, close and loop iteration.  Transaction services will be provided by the TransactionManager based on method calls emitted at the start and end of each block, and when any loop iterates.  These hooks will provide the same level of processing that can be implemented internally within the Progress 4GL interpreter.

All blocks and loops which have properties will be emitted inside a try/finally pair.  The TransactionManager pushScope() will be called first.  Then the try {} will surround the loop or block itself.  The finally {} provides the ability to hook the TransactionManager popScope() method call.  This ensures that the scope will be properly managed no matter how we exit the block (via exception, natural block exit or return).

Inside the outer try block, there will be any loop control processing needed and the opening of the block itself.  This block will contain a special do { } while () loop to implement retry logic.  Inside this do while loop is another try {} block and a set of catch {} blocks to handle condition processing.  The actual block or loop body is emitted inside this try block.

Hooks and queries into the Transaction Manager allow it to handle much of the processing automatically, however all Java statements which affect flow control must be emitted into the source.  This is a requirement of the Java language as the flow control statements are converted into byte code (instructions) which are inherently local to the current method.  The Java instruction set (all possible byte codes) only provides the ability to handle local branching except for method calls, method returns and exception processing that can arbitrarily unwind the stack.

The following is a sample of the emitted code:

      TransactionManager.pushScope("label",
                                   TransactionManager.TRANSACTION,
                                   true,
                                   true,
                                   false,
                                   false);
     
      try
      {
         label:
         // if this is a loop, the loop control structure will emit here
         {
            TransactionManager.blockSetup();
            do
            {
               try
               {
                  // block body is emitted here
               }
              
               catch (ErrorConditionException err)
               {
                 
TransactionManager.honorError(err);
                  TransactionManager.rollback("label");
                  TransactionManager.triggerRetry(null);
               }
              
               catch (EndConditionException end)
               {
                  TransactionManager.rollback("label");
                  break label;
               }
              
               catch (RetryUnwindException ru)
               {
                  TransactionManager.honorRetry(ru);
               }
              
            }
            while (TransactionManager.needsRetry());
           
            if (TransactionManager.isBreakPending())
            {
               break label;
            }
           
            TransactionManager.iterate();
         }
      }
     
      finally
      {
         TransactionManager.popScope();
      }


The TransactionManager provides the following services via static methods that are context-local to the current user's context:
  1. Maintains the state of each block, tracks start and end and handles nesting of blocks.  TransactionManager.pushScope() and TransactionManager.popScope() are used to handle this maintenance.  Note that the popScope() is implemented inside a finally {} to ensure that it always gets called.  TransactionManager.iterate() is used to allow the Transaction Manager to implement state management for loops.
  2. UNDO support:
  3. RETRY state management/infinite loop protection:
  4. RETURN ERROR state/processing:
  5. Provides a hook for any object to get a callback on exit from the scope in which it is defined.  This provides an easy method for implementing cleanup processing.  All objects needing this support must be registered using TransactionManager.registerFinish() and must implement the Finalizable interface.
  6. Provides a hook for any object that requires notification of all block entrances and block exits.  This is provided by the Scopeable interface and such objects are registered via the TransactionManager.registerScopeable().
  7. Provides a session-level batch start and stop notification service.
  8. At every block start, stop and iteration the current thread's interrupted status is checked and if the thread has been previously interrupted, a StopConditionException will be thrown.  See TransactionManager.honorStopCondition().
All blocks will be labeled, with a manufactured label or one that comes from the original Progress source.  This facilitates condition processing, since processing that was previously implicit is now being made explicit.  This is also important since a simple block in Progress is implemented as a set of related blocks in Java, in order to always be sure that the flow control statement references the correct block, it is made explicit.

Silent error processing is provided using the ErrorManager class.  This class:
  1. Stores the state of whether errors should or should not be thrown.
  2. Tracks if an error has been thrown and the list of error text and numbers.
  3. Allows access to this data.
  4. Provides helpers to throw errors or record the error if silent mode is enabled.
Lower level interfaces are also provided to allow more control within a library, especially in the case where one must retain control over where to throw the error due to the inability to silently return from the location at which the error was encountered.  The ErrorUnwindException is provided to facilitate this requirement.

All data type wrappers, expression operator implementations and the runtime backing for built-in functions have been modified to be aware of the ErrorManager and silent error mode.  As a rule, where an RUNTIME errorn would normally be generated in Progress source, the P2J runtime will use the ErrorManager to honor the silent error mode.  COMPILE errors are not handled at all since no such errors should occur in the converted environment.  Note that there may be possibilities of runtime issues in Java that are compile-time issues in Progress.  At this time, such cases will throw Java exceptions such as IllegalArgumentException rather than attempt to duplicate Progress COMPILE time error strings.

Java has no undo or retry functionality.  This means that Progress and Java are mismatched in these areas.  For this reason, the vast majority of the processing in the Transaction Manager is related to one of these two areas.

See also:

TransactionManager
BlockDefinition
ErrorManager
ErrorUnwindException
Undoable
Commitable
Finalizable
Scopeable
ConditionException
RetryUnwindException
ContextLocal

rules/annotations/block_properties.rules
rules/annotations/leave_fixups.rules
rules/annotations/labels.rules
rules/convert/control_flow.rules
rules/convert/main_method.rules
rules/convert/java_template.tpl

Honoring Stop Conditions

Stop conditions can be generated by certain runtime errors such as a database disconnection or failures in stream processing.  Any runtime code running on a thread which is servicing converted Progress business logic will translate any InterruptedException into a StopConditionException.   Classes which provide such a service:
There are unusual cases in the runtime where processing is not occurring on a thread which is servicing converted Progress business logic.  For example, if a secondary thread is used to cleanup filesystem resource (e.g. close pipes or files) or if a secondary thread is used to copy data between two pipes.  Note that in these cases, there is no counterpart or proper mapping into a Progress semantic since such cases don't exist in the Progress runtime.  Generally, such cases silently exit or fail.  Classes which provide such behavior:
In addition, the STOP language statement in Progress allows an explicit stop condition to be raised.  In this case the business logic will explicitly throw a StopConditionException.

Stop condition processing is complicated by the Progress feature that the user may press the CTRL-C key combination at any time and this immediately interrupts the execution of the Progress program (which is single threaded).  In the P2J environment this is complicated by the split of UI processing into a separate process accessed via the network.  The UI client will monitor for the CTRL-C key combination and this will cause asynchronous processing of the interruption to occur.  In particular, if processing is currently occurring on the client side, the interruption will be converted to an interruption of that running thread which will be converted to a StopConditionException in the UI client.  This will then unwind the stack normally until the location in the business logic that handles (catches) the stop condition is reached.  If the current thread is processing on the server (such that the client cannot honor the UI since it is not actively running), then the server will be asynchronously notified and this will trigger a call to Thread.interrupt().  If the currently processing thread on the server (for that user's context) is blocked, an InterruptedException would be generated and this would be caught by the runtime as noted above and translated into a StopConditionException.  This is a key point: J2SE only generates an InterruptedException in specific cases where a thread blocks such as Object.wait() or Thread.sleep().  If the thread is not blocked, then J2SE will not generate an InterruptedException, instead it just sets the interrupt status of the thread.  The TransactionManager checks the interrupted status of the current thread at block start, stop and when a block iterates.  Any previous interruption will be honored by throwing a StopConditionException.  Please see above for more details on the TransactionManager.

Accumulator behavior

Contents

  1. ACCUMULATE statement
  2. DISPLAY with aggregate phrase
  3. ACCUM function
  4. Aggregator types
  5. Accumulator expressions
  6. Accumulator usage
  7. Initial (entry) value for accumulators
  8. Block-behavior for accumulators
  9. ACCUM scope computation
  10. Reset behavior for accumulators with simple blocks
  11. Reset behavior for accumulators with IF statement
  12. The "backup" mechanism
  13. ACCUM with nested RUN/TRIGGER statements

TBD

ACCUMULATE statement

The syntax of the ACCUMULATE statement is:

ACCUM {expression (aggregate-phrase)}

The following rules about ACCUM stmt must be taken into consideration:
- When converting this statement, it must be taken into consideration that it may contain more then one accumulation expression and also more then one aggregators for a certain expression. This knowledge is used by the annotations/accumulate.rules file to find the correct expression for a certain accumulator - in the post-rules section, it will find and set the scope-id for the expression to the computed id for the current accumulator.
- When aggregate-phrase contains more then one aggregator, this will trigger generation of an accumulator instance for each combination of (expression, aggregator).
- When the accumulation is performed while iterating a FOR EACH block with a BREAK BY option, the accumulation may be computed for each BREAK BY group, if the "BY break-group" is specified in the aggregate-phrase.

DISPLAY with aggregate phrase

The DISPLAY with aggregate phrase affects the accumulators in the following ways:
- on conversion, the accumulator must be scoped at least to the same block as the frame used in the display statement
- after the block is ended, the display statement will display a cumulative accumulation result
- if a sub-only with a BY group is used, then it will display the results for each BY value.

When an accumulator is used with a DISPLAY statement and with a ACCUM statement, then it must not emit an implicit c'tor; the rule is that, when the accumulator is used with an ACCUM statement, the ACCUM statement will be used to emit the c'tor - any DISPLAY with accum references will be ignored. The default c'tor (except COUNT and SUB-COUNT, which always use default c'tor) will be generated only when there are DISPLAY with accum references and there is no ACCUM statement references, for a certain accumulator. Also, the Accumulator.accumulate(BaseDataType) must not be restricted to a ExternalDataSource, to allow the ACCUM and DISPLAY statements to work together.

There are a few special cases, when an accumulator is used with a DISPLAY statement: During this document, there may be references to the DISPLAY accum statement or DISPLAY statement - this will always mean that the author is referring to the "DISPLAY with aggregator phrase" statement.

ACCUM function

The ACCUM function is used to retrieve the accumulation result at a certain time. This does not affect the accumulation result and will affect only the part where the scope-id is computed: the found block must enclose all accumulator references.

Aggregator types

The PROGRESS accumulator types are:

Aggregator long name Short name
AVERAGE AVG
COUNT -
TOTAL -
MINIMUM MIN
MAXIMUM MAX
SUB-AVERAGE -
SUB-COUNT -
SUB-TOTAL -
SUB-MINIMUM SUB-MIN
SUB-MAXIMUM SUB-MAX

For all accumulator types it can be specified a "BY break-group" option. When specifying such an option, the accumulation will be performed for each distinct break-group value; these values will be available for retrieving using the ACCUM function only in the current or an enclosed child block. After leaving the block which has the "BREAK BY group" option, the accumulation values for each break-group are no longer accessible.
The sub-only accumulators will behave the same as normal accumulators, if they have no break-by option attached.
If a "By group" is present and the accumulator is used with a DISPLAY stmt, the sub-only accumulators will display the accumulation result for each "BY" value; for normal accumulators, the DISPLAY stmt will display only a final result.

Accumulator expressions

Accumulator expressions can be cataloged as:
1. simple expressions: a variable/table field name
2. literals - a constant expressions
3. complex expressions
The accumulation performs the same for all expression types: the expression will be evaluated (if needed) and will be added to the primary result and sub-results, if needed.

Accumulator usage

In PROGRESS, the accumulator is identified by its expression, being aware of the operands' order. For example, "i + 0" and "0 + i" will represent different accumulators.
So, if the expression must be used to identify unique accumulators, the following rules are used at conversion time to compute the accumulator name:
- if a DISPLAY statement with a aggregator phrase is encountered, then the accumulation expression is checked with the global expr-to-name map (named "accumNames") which holds the computed java names (for each expression) extracted from the ACCUM statements and the global expr-to-name map (named "dispNames") which holds the computed java names for all accumulators used in DISPLAY with aggregator phrase stmts. If the expression is found in either the accumNames or dispNames maps, then the found name is returned. Else, a name like "accumDisp#"(where # holds for an unique index) is computed
- if an ACCUM statement is encountered, then the dispNames and accumNames maps will be checked if they contain the accumulator expression. If the expression is found, then the retrieved name is used; else, a name like 'accumType#' will be returned, where "type" represents the short-name for the accumulator (without the "sub-" prefix).
- if an ACCUM function is used, then the accumNames is first checked (to see if the accumulator was used with an ACCUM statement) and also it computes the default label and the type for this expression; after this, if the dispNames map contains the accum expression, then it will override the java name.

Initial (entry) value for accumulators

All accumulators start with its value set to PROGRESS unknown (?) value. This value will be maintained until a BLOCK is entered (the BLOCK header is executed), where the accumulation value will be set to:

Aggregator AVERAGE COUNT TOTAL MINIMUM MAXIMUM SUB-AVERAGE SUB-COUNT SUB-TOTAL SUB-MINIMUM SUB-MAXIMUM
Value 0 0 0 ? ? 0 0 0 ? ?

More about which blocks reset the accumulators and how this reset is done can be found in sections Reset behavior for accumulators with simple blocks and Reset behavior for accumulators with IF stmt.

Block-behavior for accumulators

The block behavior for accumulators affects how the accumulation is executed and also what value is retrieved when the ACCUM function is used. To implement this, the accumulators must accumulate their values into two distinct data bundles:
- the 'pending' accumulation data: when the accumulation is executed, the data must be accumulated to all parent blocks; so, the accumulation is done to all data found in the pending stack. This pending data is used when the accumulator exists within a certain block: the parent block will be updated with the pending accumulation value from the child block.
- the 'reported' accumulation stack (the bundleStack): for each block, there will be kept a set with all the eventually accumulated data for each break-group value and also the primary data bundle. When the accumulation is performed for this block, the primary data bundle and the break-group data - if needed - are updated
- the sub-only property: the accumulator will be marked as sub-only when it enters the block; this property is distinct for each block
- the accumulator must have knowledge that it enters a block which encloses a DISPLAY with aggregate phrase/ACCUM stmt for this accumulator, which may be executed; so, upon entry of a block which uses this accumulator, an Accumulator.reset() call is executed - this will set the "usage" flag to all parent blocks to true and also will reset the primary data bundle to its entry value.

Using the ACCUM function to retrieve the accumulated result presents a few usage scenarios:
1. Using the ACCUM function right after the ACCUM stmt has been executed:
repeat: 
...
accum i (count).
message accum count i.
...
end.
- the retrieved accumulation result will be the total accumulation result for this block (which is kept in the 'bundleStack').

2. Using the ACCUM function right after the a block which accumulated some data is executed:
B1:
repeat:
...
B2:
repeat:
...
accum i (count).
...
end.
message accum count i.
...
end.
- the retrieved accumulation result will be the accumulation result for block B2. If the body for block B2 did not execute, then the accumulation result will be the default "entry" value. The rule extracted from this example is that, when leaving a block which accumulated some data, it will set the 'reporting' data bundle (the primary data bundle from dataBundle stack) for the parent block (block B1) to be the total accumulated value (from the pendingStack) for block B2. In other words, block B1 will inherit and report the accumulation result from block B2, until a new ACCUM stmt is executed or another block is entered.

Other special cases about the ACCUM function will be presented in the following "reset" and "backup" sections.

The PROGRESS statements which present the above behavior are REPEAT, DO [TRANSACTION/ON END-KEY UNDO, LEAVE], FOR EACH/LAST/FIRST, OPEN QUERY ... DO WHILE AVAILABLE.
The editing blocks (PROMPT-FOR/UPDATE/SET statements) do not present special behavior when accumulators are used, so accumulator scope notifications will be ignored for these blocks. Also, DO statements (which have no transaction support) do not affect accumulators in any way.

ACCUM scope computation

A certain accumulator may be scoped to a certain block or may be promoted to an instance field, following those rules:
1. The computed block to which the accumulator will be scoped (and also the accumulator) presents these properties:
- it encloses all references for this accumulator
- the accumulator is not used with a PUT or FORM statement
- the block presents transaction managenment (transaction level annotation (trans_level) is not equal to prog.no_transaction)
- all frames used by a DISPLAY with aggregator phrase statement are enclosed within this block
- it is not the root block
2. In all other cases, it will be promoted to instance field

Also, if the same accumulation expression is used within a procedure, function or a trigger, after conversion will result in 3 different accumulators.
The computed accumulator scope is also used by the accumulator expression - when the accumulator expression is instantiated (field reference, literal, etc) it must be promoted or instantiated in the same block as the accumulator.

Reset behavior for accumulators with simple blocks

Accumulators present a special behavior when a block which encloses an ACCUM statement is not executed or the block is executed and the ACCUM statement is skipped. Before continuing, it must be noted that this behavior is presented with all aggregator types, with all REPEAT, DO [TRANSACTION/ON ...], FOR EACH/LAST/FIRST, OPEN QUERY ... DO WHILE ... statements and with both ACCUMULATE statement and DISPLAY with aggregate phrase statement. For this, the following testcases are used to determine the rules:
1. a block which does not use the ACCUM stmt is not executed
B1:
repeat k = 1 to 2:
accum i (total).
display accum total i with down column 0 frame f1.
B2:
repeat while false: /* no accumulator usage */
...
end.
display accum total i with down column 13 frame f2.
end.
Frames f1 and f2 will display the same value. This testcase shows that blocks which do not use the ACCUM statement for a certain accumulator will not affect it in any way. This behavior is also present if B2 has no ACCUM stmt and is executed. Also, any ACCUM function usage in block B2 will return the same value as if it was executed in block B1, as proved in the following testcase:
B1:
repeat k = 1 to 2:
accum i (total).
display accum total i with down column 0 frame f1.
B2:
repeat j = 1 to 1: /* no accum stmt usage */
display accum total i with down column 13 frame f2.
end.
display accum total i with down column 26 frame f2.
end.

This translates into: the accumulator must not set the "reported" bundle for block B1 to be the "pending" bundle computed in block B2, when B2 is leaved, because block B2 does not have any ACCUM stmts for this accumulator. Also, when executing a block which does not use the ACCUM statement, the accumulator instance must ignore any "scopeStart", "scopeFinished", "iterate", "retry" and "finished" calls.

2. a block which uses the ACCUM stmt is not executed
B1:
repeat k = 1 to 2:
accum i (total).
display accum total i with down column 0 frame f1.
B2:
repeat while false:
accum i (total).
end.
display accum total i with down column 13 frame f2.
end.
In this example, block B2 is not executed, but the block uses the ACCUM stmt; frames f1 and f2 will display different results:
- frame f1 will display the accumulated value for block B1
- frame f2 will display the "reset" value (0 in this case)
This is caused by the fact that, when executing the header for block B2, the accumulator is notified that it enters a block which may execute accumulation; so, the accumulator will be reset and also, when leaving block B2, it will set B1's "reported" bundle to be the "pending" (currently reset) bundle for block B2.

3. a block which uses the ACCUM stmt is executed, but the ACCUM stmt is skipped
B1:
repeat k = 1 to 2:
accum i (total).
display accum total i with down column 0 frame f1.
B2:
do transaction:
if false then
accum i (total).
end.
display accum total i with down column 13 frame f2.
end.
In this example, block B2 is executed, but the ACCUM stmt is not; frames f1 and f2 will also display different results:
- frame f1 will display the accumulated value for block B1
- frame f2 will display the "reset" value (0 in this case)
This is caused by the fact that, when executing the header for block B2, the accumulator is notified that it enters a block which may execute accumulation; so, the accumulator will be reset and also, when leaving block B2, it will set B1's "reported" bundle to be the "pending" (currently reset) bundle for block B2.

4. a block scoped to procedure block which does not use the ACCUM stmt is not executed, with accumulation performed in a previous block
B1:
repeat k = 1 to 2:
accum i (total).
end.
display accum total i with down column 0 frame f1.
B2:
repeat while false: /* no accumulator usage */
...
end.
display accum total i with down column 13 frame f2.
Frames f1 and f2 will display the same value. This testcase shows that blocks which do not use the ACCUM statement for a certain accumulator will not affect it in any way. Also, this proves that, regardless of scoping level, if a block doesn't use the ACCUM statement and is not executed, the accumulator is not reset. This behavior is also present if B2 has no ACCUM stmt and is executed.

5. a block scoped to procedure block which uses the ACCUM stmt is not executed, with accumulation performed in a previous block
B1:
repeat k = 1 to 2:
accum i (total).
end.
display accum total i with down column 0 frame f1.
B2:
repeat while false:
accum i (total).
end.
display accum total i with down column 13 frame f2.
In this example, block B2 is not executed, but the block uses the ACCUM stmt; frames f1 and f2 will also display different results:
- frame f1 will display the accumulated value for block B1
- frame f2 will display the "reset" value (0 in this case)
This is caused by the fact that, when executing the header for block B2, the accumulator is notified that it enters a block which may execute accumulation; so, the accumulator will be reset and also, when leaving block B2, it will set B1's "reported" bundle to be the "pending" (currently reset) bundle for block B2.
The testcase proves that the accumulator will be reset if a block which may use execute the ACCUM statement is entered, even if the block is scoped to procedure block.

6. a block scoped to procedure block which uses the ACCUM stmt is executed but the ACCUM stmt is skipped, with accumulation performed in a previous block
B1:
repeat k = 1 to 2:
accum i (total).
end.
display accum total i with down column 0 frame f1.
B2:
repeat:
if false then
accum i (total).
end.
display accum total i with down column 13 frame f2.
In this example, block B2 is executed, but the the ACCUM stmt is skipped; frames f1 and f2 will also display different results:
- frame f1 will display the accumulated value for block B1
- frame f2 will display the "reset" value (0 in this case)
This is caused by the fact that, when executing the header for block B2, the accumulator is notified that it enters a block which may execute accumulation; so, the accumulator will be reset and also, when leaving block B2, it will set B1's "reported" bundle to be the "pending" (currently reset) bundle for block B2.
The testcase proves that the accumulator will be reset if a block which may use execute the ACCUM statement is entered, even if the block is scoped to procedure block.

Reset behavior for accumulators with IF statement

The IF statement impact on the accumulator "reset" behavior is related to stopping block execution using an "IF condition", when the condition is false. This will mean that neither the block header or body are executed.
1. IF statement with the ACCUM statement
B1:
repeat k = 1 to 2:
accum i (total).
display accum total i with down column 0 frame f1.
if false then accum i (total).
display accum total i with down column 13 frame f2.
end.
Although the second accumulation statement does not execute at all, frame f1 and f2 will display the same results.
So, this proves that when an IF statement is executed and it encloses an ACCUM statement without an intermediate block, the accumulation value is not altered.

2. IF false statement with a block which does not execute
B1:
repeat k = 1 to 2:
accum i (total).
display accum total i with down column 0 frame f1.
if false then
B2:
repeat while false:
accum i (total).
end.
display accum total i with down column 13 frame f2.
end.
Although block B2 does not execute at all (header or body), frame f1 and f2 will display different results:
- frame f1 will display the current accumulation value for block B1
- frame f2 will always display the UNKNOWN (?) value
So, this may prove that, when an IF statement is executed and it encloses an ACCUM statement inside another block, the accumulation value is set to unknown; also, when executing the THEN/ELSE branch, the accumulation value will be set to the "reset" value only if the header for a block which encloses the ACCUM statement is executed; if no such blocks are encountered on the THEN/ELSE branch, the accumulation value will remain set to UNKNOWN. (this assumption is disproved on the following testcase)

3. IF true statement with a block which does not execute
B1:
repeat k = 1 to 2:
accum i (total).
display accum total i with down column 0 frame f1.
if true then do:
display accum total i with down column 13 frame f2.
B2:
repeat while false:
accum i (total).
end.
end.
display accum total i with down column 26 frame f3.
end.
Block B2 does not execute and frames f1, f2 and f3 will display different results:
- frame f1 will display the current accumulation value for block B1
- frame f2 will display the same data as frame f1
- frame f3 will display the "reset" value for the accumulator (in this case, "0"), because block B2 does not execute.
So, this proves that, when an IF statement is executed and it encloses an ACCUM statement inside another block, the accumulation value is NOT set to unknown - this is postponed until the THEN/ELSE branch has ended and only if no ACCUM statement have been executed on the THEN/ELSE branch or no blocks (body or header) which enclose the ACCUM statement are executed.

4. IF false statement scoped to procedure block with a block which does not execute
B1:
repeat k = 1 to 2:
accum i (total).
end.
display accum total i with down column 0 frame f1.
if false then
B2:
repeat while false:
accum i (total).
end.
display accum total i with down column 13 frame f2.
Block B2 does not execute at all (header or body) but frames f1 and f2 will display the same result.
So, this proves that, when the IF statement is scoped to the procedure block, it will not set to unknown the accumulator, even if it encloses a block which uses the accumulator.

Further more, scoping the IF to a DO statement:
B1:
repeat k = 1 to 2:
accum i (total).
end.
display accum total i with down column 0 frame f1.
do:
if false then
B2:
repeat while false:
accum i (total).
end.
end.
display accum total i with down column 13 frame f2.
will result in frames f1 and f2 displaying the same values. Experimenting with other statements which do not affect the accumulators (like SET/UPDATE/PROMPT-FOR - the editing blocks) showed the same behavior. The conclusion is that only IF statements which are enclosed within a block will set the accumulator to UNKNOWN.

The "backup" mechanism

The backup mechanism is related to cases when a block is entered and, before any other ACCUM/DISPLAY statement usage, the ACCUM function is used to retrieve the accumulated result. The following testcases describe this behavior:
1. display the accumulation result before any accumulation
B1:
repeat k = 1 to 1:
accum i (total).
display accum total i with down column 0 frame f1.
B2:
repeat j = 1 to 2:
display accum total i with down column 13 frame f2.
accum i (total).
display accum total i with down column 26 frame f3.
end.
display accum total i with down column 39 frame f4.
end.
In this case, frame f2 will always display the same result as frame f1 (on every B2 iteration) and frame f3 will display the current accumulation result for block B2; frame f4 will display the final accumulation result for block B2. This proves that PROGRESS presents some backup mechanism - used to report the "entry" accumulation value if no ACCUM stmts were executed in the current block before the ACCUM function call.

2. accumulation enclosed to the current block, but skipped
B1:
repeat k = 1 to 1:
accum i (total).
display accum total i with down column 0 frame f1.
B2:
repeat j = 1 to 2:
display accum total i with down column 13 frame f2.
if false then accum i (total). /*1*/
display accum total i with down column 26 frame f3.
accum i (total). /*2*/
display accum total i with down column 39 frame f4.
end.
display accum total i with down column 52 frame f5.
end.
In this case, frame f2 will not display the same result as frame f1:
- frame f1 will display the "reported" accumulation result for block B1
- frame f2 will display the same data as frame f3
- frame f3 will display:
    - when the first iteration is performed, the "reset" value (in this case, 0)
    - on subsequent iterations, the current accumulated result for block B2
- frame f4 will display the current accumulated result for block B2
- frame f5 will display the final accumulation result for block B2 (2 in this case)
So, after executing the "if false" statement, the accumulator will no longer report the "entry" value (the accumulation value for block B1).
This testcase also proves that, even if there are two ACCUM statements inside block B2, the second accumulation is executed. The difference between the two ACCUM statements is that the second one has as parent block B2 and the first one has as parent the IF statement.

Further more, investigating a testcase like:
B1:
repeat k = 1 to 1:
accum i (total).
display accum total i with down column 0 frame f1.
B2:
repeat j = 1 to 2:
display accum total i with down column 13 frame f2.
accum i (total). /*1*/
display accum total i with down column 26 frame f3.
if false then accum i (total). /*2*/
display accum total i with down column 39 frame f4.
end.
display accum total i with down column 52 frame f5.
end.
shows this:
- frame f1 will always display the current accumulation value for block B1
- frame f2 will display the "entry" accumulation value (the current accumulated value in block B1)
- frame f3 will always display the current accumulated value for block B2
- frame f4 will display the same values as frame f3
- frame f5 will display the total accumulated value in block B2
The conclusion from this testcase is that the "IF false" statement which uses the accumulator will reset it only if no other ACCUM statements were executed before it. So, the accumulator somehow knows to ignore the "if false then accum i (total)." statements after the accumulation has been performed within the current or a child block.

3. accumulation enclosed to the current block, but skipped - case 2
B1:
repeat k = 1 to 1:
accum i (total).
display accum total i with down column 0 frame f1.
B2:
repeat j = 1 to 2:
if false then
B3:
repeat while false:
accum i (total).
end.
display accum total i with down column 13 frame f2.
accum i (total).
display accum total i with down column 26 frame f3.
end.
display accum total i with down column 39 frame f4.
end.
In this case, frame f2 will not display the same result as frame f1:
- frame f1 will display the "reported" accumulation result for block B1
- frame f2 will always display UNKNOWN (?) value
- frame f3 will always display UNKNOWN (?) value
- frame f4 will display the accumulated result in block B2 (in this case, value 2)
It seems that, after executing the "IF false" statement and setting the accumulator to UNKNOWN, the reported value for block B2 will always be UNKNOWN, even if accumulation is performed for block B2.

4. accumulation enclosed to the current block, but skipped - case 3
B1:
repeat k = 1 to 1:
accum i (total).
display accum total i with down column 0 frame f1.
B2:
repeat j = 1 to 2:
B3:
repeat while false:
accum i (total).
end.
display accum total i with down column 13 frame f2.
accum i (total).
display accum total i with down column 26 frame f3.
end.
display accum total i with down column 39 frame f4.
end.
In this case, frame f2 will not display the same result as frame f1:
- frame f1 will display the "reported" accumulation result for block B1
- frame f2 will always display the reset value (0 in this case)
- frame f3 will always display the reset value (0 in this case)
- frame f4 will display the total accumulation result for block B2.
For this testcase, the "reported" result for block B2 will not be set to UNKNOWN after executing the header for block B3 - it will be reset. This "reset" value will always be reported by any ACCUM function call, regardless of any executed ACCUM statement, until another block with an enclosed ACCUM statement or an IF statement with an enclosed ACCUM statement is executed.

5. "pending" and "reported" values for a block
B1:
repeat k = 1 to 1:
if false then accum i (total). /*1*/

display accum total i with down column 0 frame f1.
B2:
repeat j = 1 to 3:
accum i (total). /*2*/
end.
display accum total i with down column 13 frame f2.

accum i (total). /*3*/
display accum total i with down column 26 frame f3.

B3:
repeat j = 1 to 4:
accum i (total). /*3*/
end.
display accum total i with down column 39 frame f4.
end.
In this example, frames f2 and f3 will always display the accumulated result for block B2 and frame f4 will always display the accumulated result for block B3:
- frame f1 will display 0 on first iteration and the current accumulated result for block B1, on subsequent iterations
- frame f2 will always display the accumulated result for block B2
- frame f3 will always display the same values as frame f2
- frame f4 will always display the accumulated result for block B3
So, when the third ACCUM statement is executed, the "reported" value for block B1 is not affected - after executing a block which uses the ACCUM statement, the "reported" value for the parent block (block B1 in this case) will always be the "pending" (computed) value for the executed block, until another block is executed.

More, this affects the reported value for the enclosed block:
B1:
repeat k = 1 to 1:
if false then accum i (total). /*1*/

display accum total i with down column 0 frame f1.
B2:
repeat j = 1 to 3:
message "B2:" + string(accum total i). /*1*/
accum i (total). /*2*/
end.
display accum total i with down column 13 frame f2.

accum i (total). /*3*/
display accum total i with down column 26 frame f3.

B3:
repeat j = 1 to 4:
message "B3:" + string(accum total i). /*2*/
accum i (total). /*4*/
end.
display accum total i with down column 39 frame f4.
end.
The ACCUM function used in the two message statements will behave differently:
- the ACCUM function in case #1 (block B2) will always display the accumulated (pending) value for block B1
- the ACCUM function in case #2 (block B3) will be affected by the fact that another block (block B2) has been executed - this will display 0 on the first iteration and the accumulated/pending result for block B3 on subsequent iterations.

ACCUM with nested RUN/TRIGGER statements

1. ACCUM with nested RUN statements (recursion)
def var i as int.
procedure proc1.
def input parameter p as int.
do transaction:
accum i (count).
if p <> 0
then run proc1(p - 1).
end.
put screen row p + 1 string(p) + " " + string(accum count i).
end.
run proc1(10).
For the testcases above, the ACCUM function will always return the same value. This is because accumulators - which are used inside a procedure/function - are alive and can be used only as along as the procedure is executed; this means that the accumulator is not scoped to the root external procedure, but is scoped to the internal procedure. In P2J, instead of scoping the accumulator to the internal procedure, we keep a stack with the "toplevel" status of each entered block. This way, when recursion occurs and the ACCUM statement is called, the accumulation will be performed only until a "toplevel" block is encountered - and so no other blocks from a previous recurssion will be affected.

2. ACCUM with nested TRIGGERS
def var j as int.
def var i as int.
def var k as int init 0.

form j with frame f0 column 39.
view frame f0.

on F7 of j in frame f0 do:
do transaction:
accum i (count).
end.
put screen row k + 1 column 1 string(k) + " " + string(accum count i).
if k < 10 then do:
k = k + 1.
apply "F7" to j in frame f0.
k = k - 1.
end.
put screen row k + 1 column 13 string(k) + " " + string(accum count i).
end.

apply "F7" to j in frame f0.
In this testcase, applying the same trigger as the one being executed will not be possible: in PROGRESS, the trigger will not go into recurssion and so the 'apply "F7" to j in frame f0.' statement inside the trigger definition will be ignored.

Database Support

Data Types

The following Progress data types are directly supported as persistable types using the P2J wrappers.  The mappings are made transparent to the application using custom, Hibernate user types.  See the com.goldencode.p2j.persist package documentation for additional details.

Progress Type
P2J Wrapper Type
character
character
date
date
decimal
decimal
integer
recid
integer
logical
logical
raw
raw

Relational Integrity and Natural Joins

An enhanced level of relational integrity is provided during conversion in that foreign key constaints are added to the database schema where the associated relations can be detected unequivocally at conversion.  Currently, this detection is limited to the analysis of instances of the <record> of <table> construct within a Progress record phrase.  Although this is a limited limited approach, since this construct is simply short-hand for a more verbose where clause construct, it is the only mechanism currently implemented, because it is reliable and easily automated (mostly).  This analysis and its effect on schema conversion and DMO creation is described in the com.goldencode.p2j.schema package documentation .

The introduction of foreign keys unfortunately increases the complexity of the runtime environment, because these resources now need to be used and managed transparently (since the pre-conversion application knows nothing of them).  This support is discussed in teh com.goldencode.p2j.persist package documentation .

Buffer Scoping

To understand the logic and rules of buffer scoping, please see Record Scoping .

The implementation of the scoping calculations is done completely during annotation processing, though it is so complicated that it takes 3 rule-sets to properly implement.  In addition, the complex arrangement of data structures required to calculate scopes means that several external classes and a custom worker were necessary to implement this processing in the most simple manner.

The result is a set of new BUFFER_SCOPE nodes which are each a child node of the BLOCK to which they are scoped.  Each one has a set of annotations necessary for downstream processing.  These annotations include the class name and Java name of the buffer, among other important information.

Each record and field reference has a " refid " annotation that is the AST ID of the BUFFER_SCOPE node to which the reference refers.  This allows the buffer's name to be dereferenced at conversion time.

Please see the following for more details:

rules/annotations/record_scoping_prep.rules
rules/annotations/record_scoping.rules
rules/annotations/record_scoping_post.rules

BufferScopeTracker
BufferList
BufferBlockState
BufferScopeWorker

Record Buffer Instantiation

The Java AST will get RecordBuffer instantiation or declaration nodes in the following cases:
  1. There is an explicit language statement (define buffer) which would cause a memory allocation to occur in Progress.
  2. There is an explicit language statement (define parameter or parameter) which would cause a buffer declaration to occur in a function or internal procedure.
  3. The first implicit reference to a buffer in the source file will cause a memory allocation of a buffer of the same name as the record.  This can be found by examining all BUFFER_SCOPE nodes and finding those references that have both the "implicit" and "first" annotations as true.
The location of the resulting nodes is determined by the following rules:
  1. A "define buffer" inside an internal procedure, trigger or function will emit a RecordBuffer.define() in place (in the main block of the internal procedure, trigger or function).  This allocates a new data model object.  Note that this works since the DEFINE_BUFFER nodes are moved/sorted to always be rooted as the first children of the BLOCK node in any internal procedure, trigger or function.
  2. An explicit DEFINE_PARAMETER inside an internal procedure is emitted to the method definition signature (it won't appear in a function or trigger).
  3. An explicit PARAMETER (that is a buffer definition) inside a function is emitted directly based on peerid (it won't appear in an internal procedure or trigger).
  4. All other buffers are instance members (this avoids the need for differentially constructing some in the external procedure and some as instance members via "promotion" due to usage across the entire class or due to being an explicit shared buffer definition).
Import statements for the generic com.goldencode.p2j.persist package are always added, as is an import for the custom package(s) that contain the application-specific data model objects.

The Java AST nodes that are created as a result of this logic are as follows:
  1. An imported shared buffer (define shared buffer) is emitted as a buffer declaration with an initialization based on calling SharedVariableManager.lookupBuffer().
  2. An exported shared buffer (define new shared buffer) is emitted as a standard buffer definition using RecordBuffer.define() with a second method call to SharedVariableManager.addBuffer() to export the buffer.  Note that in the case where the buffer is located as an instance member, the define and the addBuffer will be located in 2 places.  In particular, the addBuffer() will be done in line.
  3. Implicit buffers and non-shared explicit buffers (define buffer) are emitted as a RecordBuffer.define().
  4. Parameters (define parameter and parameter in a function definition) are emitted in the corresponding method signature as a REFERENCE_DEF (classname followed by the instance name).
All BUFFER_SCOPE nodes also generate a notification to RecordBuffer.openScope() to ensure that the database runtime code marks the end of each block for the associated record release and record lock release.

Please see:

Transaction Management in the persist package
rules/convert/buffer_definitions.rules

Buffer and Field References

Buffer/record references are emitted in 3 cases:
  1. The reference is a child of a KW_BUFFER node.  This conforms to the usage of a buffer reference as a RUN statement parameter or function call parameter.
  2. The reference's "bufreftype" annotation is not NO_REFERENCE and the 3rd level ancestor is not PARAMETER.
  3. The reference's "bufreftype" annotation is not NO_REFERENCE and the 3rd level ancestor is not DEFINE_PARAMETER.
The resulting node is a simple Java REFERENCE node with the buffer's Java name.

Field references are always emitted, but these translate into field-specific getter and setter calls on the data model object.  The getter and setter method names are determined at annotation time.

The assignments rule-set required changes to modify the normal ASSIGN node processing when the field reference is a setter.  This is required to ensure that the data model object (which is the first child) will emit as the referent of the setter method call.  It is also required such that the 2nd operand of the ASSIGN node would emit as the 2nd or 3rd child (which would be the 1st or 2nd parameter to the setter call).

The variable references rule-set was modified to change how subscripts work for fields.  Since Progress fields can be treated as array values, such parameters must be properly handled as the first integer index parameter of a getter or setter method call.  This is the reason that the 2nd operand of the ASSIGN node can be either the 1st or 2nd parameter to the setter method call.  The same logic as normal subscripting applies here too (all values are converted into 0-based indices from the Progress 1-based approach).

Please see:

rules/convert/assignments.rules
rules/convert/database_references.rules
rules/convert/variable_references.rules

WHERE Clause Processing

The following is the mapping of Progress WHERE clause constructs to HQL (Hibernate Query Language).

Progress WHERE
HQL
Notes
LPARENS
(

AND
and

OR
or

EQUALS
=

NOT_EQ
!=

GT
>

GTE
>=

LT
<

LTE
<=

BEGINS
like '<criteria>%'
The 2nd operand's text is converted into a Java string using ExpressionConversionWorker progressToJavaString() and then the '%' is appended.  Strings are enclosed in single quotes.
MATCHES
like '<criteria>'
The 2nd operand's text is converted into a Java string using ExpressionConversionWorker progressToJavaString() and then special matching characters are converted into the HQL equivalents using ExpressionConversionWorker convertToSQLLike().  Strings are enclosed in single quotes.
CONTAINS
n/a

NOT
not ( )
The child node of this operator is wrapped in () to ensure that the Progress precedence is maintained, since this is different compared to HQL NOT precedence.
BOOL_TRUE
true

BOOL_FALSE
false

NUM_LITERAL
text is emitted directly

DEC_LITERAL
text is emitted directly
STRING
'text'
The text is converted into a Java string using ExpressionConversionWorker progressToJavaString().  Strings are enclosed in single quotes.
DATE_LITERAL
'YYYY-MM-DD'
The text is turned into an instance of com.goldencode.p2j.util.date and the toStringSQL() method is used to generate the correct format.  The resulting string is enclosed in single quotes.

Note that there are special considerations for handling dates within a database where clause.
UNKNOWN_VAL
is null
is not null
null
If the parent note is EQUALS or NOT_EQ one of the two first forms are emitted.  Otherwise the 3rd form is emitted.
FUNC_RECID and oldtype == KW_RECID
<javaname_of_buffer>.id
This builtin function is directly converted to a reference to the special "id" field.
FUNC_ROWID and oldtype == KW_ROWID
<javaname_of_buffer>.id This builtin function is directly converted to a reference to the special "id" field.
FIELD_ in the current buffer
<buffer_qualifier>.<property_name>

LBRACKET
[]
Only FIELD_ extents are supported.

Subscripts are supported using [ ] but only if the index is a NUM_LITERAL.  Such an index is decremented by 1 and emitted as text inside the square brackets.

Anything not directly supported above is treated as a substitution parameter.  The HQL string will contain a '?' character at the point in the string in which the runtime substitution should occur.  The "root" node of such a sub-expression is emitted as a runtime expression in the generated business logic.  This result is passed as a query argument.  Hibernate handles the query substitutions using the result of the evaluated runtime expression.

Other notes:

Database Language Statements

Java classes referenced in the table below reside in the com.goldencode.p2j.persist package unless otherwise noted.
Statement
Java Equivalent
Notes
ACCUM
Accumulator (abstract base class);  concrete implementations:
  • AverageAccumulator
  • CountAccumulator
  • MaximumAccumulator
  • MinimumAccumulator
  • TotalAccumulator
  • reside in the com.goldencode.p2j.util package (following the Progress semantic, these classes are not strictly database related;  they can be used standalone with variables or added to a query)
  • same classes are used to handle both cumulative results and break group results
BUFFER COMPARE
RecordBuffer.compare

BUFFER COPY
RecordBuffer.copy

CLOSE QUERY
n/a
P2J queries are not explicitly closed;  this statement is dropped
CONNECT
ConnectionManager.connect

CREATE
RecordBuffer.create
No USING support at this time.

In Progress, CREATE does not immediately flush the new record to the database, since in most cases the default values assigned to a new record would violate uniqueness or nullability constraints.  Instead, the record is not written to the database until the first statement is executed, which would update an index for the record's table.

In our implementation, we defer writing the record to the database (or more accurately, persisting the record using Hibernate;  actual flushing to the database may be further deferred by Hibernate), until the earliest of the following occurs:
  1. all properties participating in ANY uniqueness constraint are updated (the last property updated in such a set triggers validation and flushing);
  2. a query which operates against the record's table is executed;
  3. the current record in the record buffer is replaced or released;
  4. a transaction or subtransaction commit occurs;
  5. the end of a converted ASSIGN statement is reached (RecordBuffer.endBatch() is invoked).

Note that our implementation departs slightly from Progress -- particularly with regard to #1 in the list above -- in that the setting of a property which does NOT participate in a unique constraint (but DOES participate in a non-unique index) WILL NOT trigger a buffer flush.  However, we believe the remaining conditions adequately make up for this departure.
CREATE ALIAS
ConnectionManager.createAlias

CREATE BUFFER

TBD;  dynamic buffer creation is not supported in the first release
CREATE DATABASE

TBD;  dynamic database creation is not supported in the first release
CREATE QUERY

TBD;  dynamic query creation is not supported in the first release
CREATE TEMP-TABLE

TBD;  dynamic temp-table creation is not supported in the first release
DEFINE BUFFER
RecordBuffer.define
see Record Buffer Instantiation
DEFINE QUERY
AbstractQueryWrapper (abstract base class);  concrete implementations:
  • CompoundQueryWrapper
  • PreselectQueryWrapper
  • RandomAccessQueryWrapper
These wrapper classes are used to support the DEFINE QUERY semantic of defining a query once and possibly opening it multiple times.  An instance is constructed once, as an instance variable, corresponding with the DEFINE QUERY statement.  Each corresponding OPEN QUERY is converted as an instance of a concrete subclass of AbstractQuery, which is set into the wrapper when opened.
DEFINE TEMP-TABLE
DEFINE WORK-TABLE
TemporaryBuffer.define no distinction is made between temp-tables and work-tables at conversion;  both are converted to DMO class definitions in the <...>.dmo._temp package (DMO interfaces) and the <...>.dmo._temp.impl package (DMO implementation classes) in the converted application hierarchy

see also Temporary and Work Table Support
DELETE
RecordBuffer.delete
no VALIDATE support at this time
DELETE ALIAS
ConnectionManager.deleteAlias

DISCONNECT
ConnectionManager.disconnect

FIND (standalone)
FindQuery
AdaptiveQuery

FIND (as preselect retrieval mechanism)
P2JQuery.first
P2JQuery.last
P2JQuery.next
P2JQuery.previous

FOR/DO/REPEAT
AbstractQuery (abstract base class);  concrete implementations:
  • CompoundQuery
  • PreselectQuery
    • AdaptiveQuery
    • PresortQuery
  • RandomAccessQuery
Which concrete implementation is chosen at conversion time depends upon the number of tables involved and upon the nature of the query.

For an individual table, PreselectQuery is chosen if it is defined explicitly in Progress using the PRESELECT keyword or if the Progress construct's explicit sort criteria do not match the index selected according to index selection rules.  PresortQuery is chosen if client-side sorting is required (i.e., the by clause contains an expression which cannot be converted to an HQL 'order by' clause, but must instead convert to a runtime expression).  Both of these types can be applied in a multi-table join situation.

AdaptiveQuery is chosen by default for a single table query, if other factors do not force a preselect variant.  This query operates in preselect mode until some change to a record forces it to operate in dynamic mode.  As soon as it can switch back to preselect mode, it will.

RandomAccessQuery is used by an AdaptiveQuery when switching to dynamic mode.  This is the most flexible query type (but also has the worst performance);  it is necessary to support the Progress "dynamic" record retrieval semantic, where updates to a record made during the loop can change the results found later in the loop (an effect [side effect?] of Progress' index implementation).

CompoundQuery is chosen for nested FOR loop conversions, where the component loops would convert to AdaptiveQuery instances or to a mixture of adaptive and preselect query variants.  CompoundQuery is a driver to which Joinable (implemented by RandomAccessQuery and PreselectQuery) query instances may be added as components.
GET
P2JQuery.first
P2JQuery.last
P2JQuery.next
P2JQuery.previous

RELEASE
RecordBuffer.release

REPOSITION
RandomAccessQuery.reposition

OPEN QUERY
AbstractQuery (abstract base class);  concrete implementations:
  • CompoundQuery
  • PreselectQuery
    • PresortQuery
  • RandomAccessQuery
    • FindQuery
in the event of an OPEN QUERY which references a query previously declared by a DEFINE QUERY, one of these concrete classes is set into the corresponding concrete wrapper implementation;  otherwise, an instance of one of these classes is used directly
VALIDATE
RecordBuffer.validate


Built-in Function Support

See all entries in this table with type "database
essimistic Record Locking Progress relies upon a pessimistic locking strategy consisting of three record locking modes:
References to these keywords have been converted to use constants defined as static variables of the LockType class.  These constants are used by the persist package internally.  They appear in converted application code as arguments to various query constructors and initialization methods, and in record retrieval methods (e.g., first , last , next , previous , current , and unique ), to correspond to explicitly specified lock requests in Progress record phrases.

For information regarding P2J's runtime pessimistic locking strategy, please refer to the persist package documentation .

Please see:

rules/convert/database_access.rules

Temporary and Work Table Conversion

Temp-tables and work-tables convert to the same temporary table implementation;  there is no distinction between the two in the P2J environment, so both are referred to generically as "temp tables" or "temporary tables" in this document.  Temp table structural information is extracted from DEFINE TEMP-TABLE and DEFINE WORK-TABLE statements in the Progress source code.  Each unique definition is converted into a specialized DMO interface and implementation class.  A Hibernate mapping document is created for each, and each receives an entry in the runtime dmo_index.xml document, which resides at the relative root of the dmo sub-package in the converted application's package hierarchy.

A temp table definition is considered unique if its structural organization and content (i.e., field and index elements) is unique across the application.  Currently, Hibernate contains no direct support for temp tables;  however, it does not preclude that Java objects be mapped to temporary database tables.  Furthermore, Hibernate requires that a Java object be mapped to only one backing table within a single SessionFactory context.  It also requires that these mappings be known at SessionFactory configuration, which for practical purposes (it is quite expensive) must occur infrequently and preferably during server initialization.  For these reasons, only the conversion of statically defined temp tables are supported at this time.

Please see the persist package documentation for more information about runtime temp table support.

Please see:

rules/convert/buffer_definitions.rules
rules/convert/database_access.rules

Client-Side Where Clause Processing

Certain where clauses in Progress cannot be converted to server-side queries expressed entirely in HQL.  Specifically, if a where clause contains a nested reference to a field in the record being retrieved, it must be tested against a client-side expression to determine whether it will be included in the query result.  By "nested", we mean embedded within a subexpression which cannot be expressed in HQL.  For instance, a field reference to the current buffer as a parameter to to a built-in or user function would qualify for client-side where clause processing:
def var snippet as character initial "whatever".
for each customer
 no-lock,
each invoice
where invoice.cust-num = customer.cust-num
and substring(invoice.category, 10, length(snippet)) = snippet
no-lock:
...
Although this is a trivial example, which could be expressed differently, it serves to illustrate the point:  the customer.name reference embedded within the built-in function substring causes this code to convert to Java code which executes a client-side where clause expression.

Progress where clauses which qualify for this type of handling convert, where possible, to a server-side component and a client-side component.  The server-side component is expressed in HQL and the client side component converts to an anonymous inner class which extends the abstract class WhereExpression .  At runtime, the query is executed (at a high level) in two phases:
  1. a server-side query which returns a preliminary, candidate result;
  2. a client-side expression which filters the candidate results.
The server-side query is necessary to restrict, to the extent possible, the amount of data pushed to the client for final evaluation, since this is likely to be the most expensive component of the process.  The greater the level of restriction performed at the server, the better.  That being said, it is not always possible to generate HQL which accomplishes this goal.  In the worst case scenario, no server-side restriction will be possible at all, and every record in the table will be retrieved and evaluated on the client.

Server-side restriction is possible only if the a top-level operation in the where clause expression is a logical AND, and at least some portion of one side of the expression can be evaluated at the database server.  In the above example, the invoice.cust-num = customer.cust-num component of the where clause can be expressed in HQL as invoice.cust-num = ? , where ? is substituted at runtime with each customer.cust-num value found in the outer loop.

The client-side portion of this expression is converted to an anonymous, inner subclass of WhereExpression , as the following code segment illustrates:
character snippet = new character("whatever");

WhereExpression whereExpr0 = new WhereExpression()
{
public Object[] getSubstitutions()
{
return new Object[]
{
new IntegerExpression()
{
public integer execute()
{
return snippet.length();
}
},
snippet
};
}

public logical evaluate(BaseDataType[] args)
{
return CompareOps.equals(character.substring(invoice.getCategory(), 10, (integer) args[0]), (character) args[1]);
}
};
During the client-side filtering phase of query execution, the evaluate() method defined above is invoked for each invoice record retrieved by the server-side phase. The WhereExpression class and the various query implementations ensure that the arguments/expressions returned by getSubstitutions() are resolved at the appropriate times during execution of the query loops (this is important, since it preserves the Progress semantic by ensuring that any side effects of invoking built-in or user functions -- such as state changes -- are maintained).  The resolved results of these arguments/expressions compose the args parameter passed to evaluate() .  If evaluate() returns true , the current invoice record is retained as a final query result;  if it returns false , the record is dropped from the query's final results.

NOTE:   the conditions which trigger client-side where clause processing as discussed above have been greatly reduced since this section was first written.  Since that time, we have introduced and implemented many server-side (i.e., database server, not P2J server) functions which minimize the circumstances which require client-side where clause processing.  Since this implementation, the only where clauses that require client-side where clause processing are those which integrate user-defined functions, because these may have side effects which cannot be predicted and must execute on the client.  All built-in Progress functions which are used in where clauses in the initial project to which P2J is targeted have now been implemented as server-side functions.  Please refer to the following package documentation for additional details on this implementation.  The information in the following package summaries take precedence over the information above:
Please also see:

rules/annotations/where_clause_pre_prep.rules
rules/annotations/where_clause_prep2.rules
rules/annotations/where_clause.rules
rules/annotations/where_clause_post.rules
rules/annotations/where_clause_post.rules
rules/convert/database_access.rules

Database Validation Expressions

Validation expressions and messages can be defined for database fields (and tables, though this is not supported at this time).  Field-level validation expressions are used by default in certain blocking, input, UI language statements.  Application code may provide an explicit override (using the VALIDATE keyword), in which case the default validation expression is ignored.

Any Progress code which is valid within the context of the business logic in which the corresponding field reference is made can be used within a validation expression.  Therefore, although this would be very bad form, it is legal for a validation expression to contain references to local variables and other constructs valid only within the enclosing scope of the field reference.  This is problematic for conversion, because it requires that validation expressions emulate this behavior in Java code.  To accomplish this, we duplicate the conversion expression as a subclass of Validatable , implemented as an inner class within each business logic class in which the field must be validated.

Although these expressions are defined at the database schema level, they are evaluated only by the UI.  Field-level validation expressions defined in the schema are not triggered by simple assignments .  The following UI language statements trigger field-level validation:
It is the first occurence of a reference to a validatable field, in association with one of these statements, which triggers the Validatable implementation to be emitted in the target business class.  It also triggers an instance of the Validatable implementation class to be created and registered with the UI widget associated with the validatable field.  In addition to the above listed statements, the FORM statement is given special consideration, because it may contain the first reference in a frame to a validatable field, even though it does not itself trigger validation.  Surprisingly, the SET and UPDATE options of the MESSAGE statement do not trigger field-level validation.

Please see:

rules/schema/p2o.xml
rules/fixups/schema-validations.rules
rules/annotations/validation-prep.rules
rules/annotations/validation.rules
rules/annotations/validation-post.rules
rules/convert/ui_statements.rules
rules/convert/expression.rules

Methods and Attribute Support

At this time, the following subset of database-related methods and attributes are supported:
Method/Attribute
Java Equivalent
Notes
CURRENT-RESULT-ROW
P2JQuery.currentRow()

GET-FIRST()
P2JQuery.first()

GET-LAST()
P2JQuery.last()

GET-NEXT()
P2JQuery.next()

GET-PREV()
P2JQuery.previous()

HANDLE
P2JQuery instance reference
There is no method equivalent in Java for this attribute;  the "handle" of a P2JQuery object is the object reference itself.  Therefore, the conversion of Progress code which dereferences the handle attribute from a query object is simply to emit the Java query object reference.
QUERY-OFF-END
P2JQuery.isOffEnd()

QUERY
N/A
Use of this attribute of browse is superfluous in Java, in the customer cases encountered thus far.  Therefore, it is simply removed from the tree in a customer-specific annotation ruleset.  If it is determined that this attribute is universally superfluous, this processing will be relocated to a common ruleset.

Known Issues

  1. When generating HQL expressions from a Progress where clause where a character comparison is necessary, the case sensitivity of a field or variable which is substituted at runtime for an HQL query substitution marker is only maintained if that field or variable is a direct operand of the comparison.  For instance, the Progress where clause customer.name = invoice.cust-name might convert to an HQL expression upper(?) = upper(invoice.cust-name), if both fields are case-insensitive.  However, the case sensitivity of customer.name may not be considered in the HQL generation if the field reference is more deeply nested.  For example, the Progress where clause substring(customer.name, 0, 5) = invoice.cust-name would convert to the same HQL phrase shown earlier if invoice.cust-name is case-insensitive, even if customer.name was case-sensitive.  This is considered to be a relatively obscure case;  it will be addressed as needed.
  2. For searches using HQL expressions which have an embedded, non-literal (i.e., dynamic), subscript reference (e.g., where myBuf.myField[?] = ...), P2J diverges from Progress in the type of error raised for a subscript out of bounds condition.  Because the query substitution is handled on the database server using this notation, Hibernate simply comes back with an empty result set in the case where the subscript is out of bounds, which will result in an error condition with text ** <XXXX> record not on file. (138) when a null DMO is set as the current record in a record buffer.  Progress raises a stop condition with text ** Array subscript <X> is out of range. (26).

User Interface Support

Basic Design

All static elements of the UI are removed from the business logic and used to create 2 UI specific classes for each unique frame:
  1. A custom screen buffer interface with getters, setters and widget accessors for the associated frame.
  2. A frame definition that can be used to instantiate and initialize a frame of the given layout/fields/configuration and its contained widgets.
These classes are used at runtime by the business logic.  In particular, the control flow is defined in the business logic.  This means that the controller of the UI is in the business logic, but the format, layout ... of the UI itself has been separated into UI specific classes.

The behavior of DISPLAY/UPDATE/SET/PROMPT-FOR is special because these statements depend upon a passed array of FrameElement instances which define the list of frame fields that are to be processed.  Each FrameElement is one of the following subclasses:
  1. Element - a displayable data value and the widget it is associated with
  2. WidgetElement - a widget with no data (e.g. a button)
  3. EmbeddedAssignment - a mechanism to delegate the execution of an assignment (and dynamic evaluation of the associated expression) during the assignment processing in these language statements.
For each of the statements the copy to/from the screen-buffer, the widget enable/disable, the view(), the wait-for() are all hidden in the runtime.  This processing is mapped into primitive operations but the logic is properly enforced in these higher level APIs such that the Progress semantic is maintained.  Please see GenericFrame for more details.

The rules/annotations/frame_scoping.rules is the core rule set that calculates the frame scoping, widget lists for each frame and many other critical annotations.  The majority of the UI related business logic annotations are done in rules/annotations/screen_buffer.rules.

The rules/convert/frame_generator.xml is a standalone rule set that is run between annotations and code conversion.  This rule set uses the previously stored annotations to generate the 2 classes above per frame.  The majority of the UI related business logic code conversion is done in rules/convert/ui_statements.rules.

UI Language Statements


Statement
Java Equivalent
Notes
APPLY
CommonFrame.apply()
LogicalTerminal.apply()

ASSIGN (input buffer form) Rewritten using the INPUT function such that this is converted as an explicit assignment statement that directly reads the field's getter method in the screen buffer.

BELL
LogicalTerminal.bell()

CHOOSE
CommonFrame.choose()
CLEAR
CommonFrame.clear()
CommonFrame.clearAll()

COLOR
LogicalTerminal.setColors()

CREATE WIDGET
not converted at this time
CREATE WIDGET-POOL
not converted at this time
DEFINE BROWSE
new BrowseWidget()
DEFINE BUTTON
new ButtonWidget()

DEFINE FRAME
new FrameWidget()
DEFINE RECTANGLE
not converted at this time
DELETE OBJECT
not converted at this time
DELETE WIDGET
not converted at this time

DISABLE
GenericWidget.disable()
CommonFrame.disable()
CommonFrame.disableExcept()

DISPLAY
CommonFrame.display()
DOWN
CommonFrame.down()
ENABLE
GenericWidget.enable()
CommonFrame.enable()
CommonFrame.enableExcept()

FORM
Converted into a frame definition class.

FRAME-VALUE
LogicalTerminal.setFrameValue()

HIDE
LogicalTerminal.hideMessage()
LogicalTerminal.hideAll()
CommonFrame.hide()

INPUT CLEAR
LogicalTerminal.inputClear()

MESSAGE
LogicalTerminal.message()
LogicalTerminal.messageBox()

NEXT-PROMPT
CommonFrame.nextPrompt()

ON
LogicalTerminal.registerRunnable()
LogicalTerminal.deregisterRunnable()
LogicalTerminal.remapKey()

PAUSE
LogicalTerminal.pauseBeforeHide()
LogicalTerminal.pause()

PROCESS-EVENTS
LogicalTerminal.processEvents()

PROMPT-FOR
CommonFrame.promptFor()

PUT-CURSOR LogicalTerminal.putCursor()

PUT-SCREEN LogicalTerminal.putScreen()
READKEY
KeyReader.readKey()

SCROLL
CommonFrame.scroll()

SET
CommonFrame.set()

STATUS
LogicalTerminal.statusDefault()
LogicalTerminal.statusInputOff()
LogicalTerminal.statusInput()

TERMINAL
LogicalTerminal.setTerminal()

UNDERLINE
LogicalTerminal.underline()
CommonFrame.underline()

UP
CommonFrame.up()

UPDATE
CommonFrame.update()

VIEW
CommonFrame.view()

WAIT-FOR
LogicalTerminal.waitFor()


See also:

rules/annotations/trigger_prep.rules
rules/annotations/set_update_embedded_assignments.rules
rules/annotations/frame_scoping.rules
rules/annotations/screen_buffers.rules
rules/annotations/validation_prep.rules
rules/annotations/validation.rules
rules/annotations/when_rewriting
rules/annotations/dynamic_ui_indexes
rules/convert/ui_statements.rules

Special UI Block Types/Callbacks

The Progress UI contains 2 special block types (trigger blocks and editing blocks) and 1 pure callback (validation expressions).  All 3 of these constructs are tightly integrated with language statements that interact with users.

Triggers are blocks that are executed when defined events occur in the user interface.  Instead of defining triggers like internal procedures, triggers are defined "in line" with the procedure or function being processed.  Though Progress treats a trigger as a distinct block type which has block properties ( see above ), it is not well seperated in terms of location in source code.   Triggers take no parameters and are not named.  Instead, they are registered as a set of events and a set of widgets which cause the trigger to be called.  When any of the listed events occurs for any of the listed widgets, the trigger will be executed.  Multiple triggers can be registered for the same event + widget combinations.  The last registration "wins" (is active).  The scoping for such registrations is at the procedure (internal/external), trigger and function level.  When such a new scope is opened, duplicate trigger registrations will hide registrations from previous scopes.  Likewise, the closing of a scope will remove all triggers registered in that scope, which will make previously hidden triggers "re-appear".  Within a single procedure, function or trigger (yes, triggers can be nested in other triggers), duplicate registrations hide previous registrations and this will not implicitly ever be undone (without the entire scope exiting and thus removing all triggers registered in that scope).  The explicit REVERT option can be used to deregister a specified trigger and then the trigger registered most recently (in this scope or a previous scope) will become the active trigger for that given event + widget combination.

One other interesting feature of trigger registration is that it is strictly based on the flow of control of the registering code.  If the flow of control never executes the branch of code in which a trigger is defined (e.g. an ELSE block that is never executed), then that trigger is not registered.  This means that the registration of triggers must be kept strictly in line with the matching control flow, even if the trigger block itself is refactored into another location.

Editing blocks are really special purpose key-press processing loops that are called in-line from the PROMPT-FOR processing (and from UPDATE/SET since they have an embedded PROMPT-FOR).  Such blocks are invoked before any other event processing associated with a key press.  They can override or modify all event processing associated with the key press (editing blocks take precedence over all other forms of event processing).   The editing block is peculiar in that it is completely integrated with the surrounding code (the block structuring of the surrounding code).  In particular, this means that one can execute low-level flow control statements such as NEXT or LEAVE and these can directly reference enclosing block names.  Although it may not be obvious, such blocks are in fact loops and one can use NEXT to force an iteration (restart from the top of the block).  Such blocks do have block properties ( see above ) but they are limited and cannot be customized (with the "ON clause").

Validation expressions are not a block type but instead are a direct evaluation of a logical expression to determine if the content of an associated field is or is not valid.  The expression is evaluated in the context of the defining program but it called when a field is changes in the UI and the user attempts to leave the field or screen.  If the validation expression returns FALSE, a second expression is evaluated to obtain message text and this message is displayed to the user.  The user is then placed back into that field (or just not allowed to leave) and is provided an opportunity to edit the field again.  If the validation expression evaluates TRUE, then the field is considered valid.  If no edits are made to a field but the screen is exited, all fields that had not yet been validated will be validated.  The first validation expression to fail will display the message and place the user back into the associated field to edit it.  Of particular interest is the fact that both the validation expression itself and the expression used to generate the message text are real expressions and can contain the full variety of function calls, operators, constants and resource references one would expect in a Progress expression.

In terms of scoping, all 3 constructs can access resources (vars, streams, buffers and frames) that are defined in the enclosing scope.  This complicates the generation of code immensely since the rules for scoping in Java are much more strict.  A great deal of processing was implemented in annotations to ensure that all such resources are "promoted" to instance members in the generated business logic classes.  This allows triggers and validation expressions to be emitted as inner classes while still accessing all the resources in the proper context.  In particular, streams, buffers and frames were all modified to ensure that *all* instances of such resources have unique names and thus all instances can always be made into instance members.  With variables, this was not desireable since the variables are so common and keeping a scoped approach yields a better result.  So variables are analyzed to determine which need promotion.  This promotion logic is in rules/annotations/scope_promotion.rules.

Triggers are implemented as a named inner class (each one gets a generated name) which extends Runnable.  The trigger block itself is emitted inside the run() method and it is treated as a top-level sub-transaction block in terms of the TransactionManager .  All local resources are promoted to instance members in the containing class so these resources are directly accessible.  A instance of the inner class is constructed (default constructor) and passed to the registration method, along with a definition of the event + widget combinations that are valid.  These event list + widget list combinations are defined as an instance of the WaitList class.

Editing blocks (loops actually) are implemented in-line with the original containing logic (at the location of the UPDATE/SET/PROMPT-FOR statement in the original control flow).  This is required to ensure that the integrated control flow processing (e.g. NEXT, LEAVE, RETURN...) occur in the proper context.  Java has no way to provide such direct control flow processing (such as break, continue or return) without the editing block being defined as a nested block in-line.  This requires "inverting" the normal UI processing.  Normally, the server's UI runtime drives the logical terminal based on well known rules AND most importantly without any return to the calling code.  In this case, the calling code really has control and must drive the UI processing.  This block is treated as a nested loop (e.g. like a REPEAT inner block) which is a sub-transaction in TransactionManager terms.   The emitted block handles ERROR and retry but END, STOP and QUIT are ignored (handled only by containing blocks). Special methods of the referenced frame are called at specific points to ensure that the UI is invoked at the proper times:
  1. CommonFrame.startEditingMode() - this code is executed once at entrance to the block.  The UI uses this to initialize and prepare for the "inverted" UI processing.
  2. CommonFrame.waitForNextKey() - this code is executed at the start of every iteration.  It returns when there is a key available for processing.
  3. CommonFrame.continueEditing() - this code is executed at the end of every iteration.  It allows any pending GO events to be processed and state to be maintained properly.  This is critical to the processing of NEXT and any retry (due to the ERROR condition being raised) as well, since in such cases any pending GO event must be cleared.
Validation expressions are emitted as inner classes that extend the Validator class.  This is a class that implements the Validatable interface.  Each validation expression (as defined in a format phrase for a variable/field override or as defined in the data dictionary for a database field) is registered with the associated widget and is thus called back as needed during the UI processing.  If a field has a default validation expression in the data dictionary AND there is no override in the business logic, then a factory method (RecordBuffer.getValidatable()) is used to obtain the Validatable instance to be registered.  Any override for a database field in the business logic will result in an inner class being generated and registered, just as for variables.

The validation expression itself almost always will have a reference to the current field or variable AND this reference must resolve to the latest or proposed version of the variable/field.  To handle this, the new value is passed as a parameter in the Validatable interface.  The emitted inner class casts this to a local variable of the proper type and name such that the expression will naturally access the local version in its expression evaluation.  This also handles the special case in Progress where an array element is referenced in a validation expression without the associated subscript.

See also:

CommonFrame.java
GenericFrame.java
LogicalTerminal.java
ThinClient.java
WaitList.java
Validatable.java interface
Validator.java

rules/annotations/trigger_prep.rules
rules/annotations/validation_prep.rules
rules/annotations/validation.rules
rules/convert/ui_statements.rules

Key and Condition Processing

The thin client has a separate thread that handles key reading from the terminal.  This reader thread provides typeahead functionality and polls the keyboard every 100ms.  It can be suspended and resumed, which is necessary while handling redirected input streams.  These are cases where any interactive input to the terminal must be routed to the stream in use (often a child process, for example a "vi" editor session) and thus the thin client's key reading cannot be done simultaneously.

The key reader thread fills a queue of input events.  This queue is read by the thin client during editing mode (anything that calls WAIT-FOR) and during an explicit READKEY.  When read during WAIT-FOR, the key events are processed in the widgets via APPLY.  READKEY bypasses all such APPLY processing and returns the latest key from the top of the queue.

The key reader uses features in the CHARVA Toolkit class (some of which are backed by native methods that use NCURSES) to read the keyboard in "raw" mode with the "keypad" features activated.  This means that all keystrokes such as CTRL-C will be read without signals being generated.  The "keypad" support means that NCURSES will convert special ("extended") keys like function keys or numpad keys into simple single keys rather than passing through the multiple scancodes that the keyboard actually generates. 

CTRL-C is handled specially.  It generates a STOP condition AND this works in both synchronous and asynchronous modes.  While processing is occuring on teh thin client, a STOP condition is raised directly on the client and this exception unwinds the stack back into the business logic on the server.  When processing is active on the server, the key reader thread will asynchronously interrupt the thread on the server (via a callback from the key reader thread to the server) which will raise the STOP condition there.  This happens using the current thread's "interrupted status" and the interrupt() method in Thread.   If server's thread is blocked in some form of wait, then an InterruptedException, IOInterruptedException or other related exception will be generated and this will be caught and converted into a StopConditionException.  All such locations have been instrumented with the proper catch blocks to ensure this happens.  If the server thread is not blocked in a wait, the interrupted status is set and at critical points (start scope, iterate and stop scope) for each block the TransactionManager checks this status and throws the StopConditionException is the thread was previously interrupted.

See:

KeyReader.java
ThinClient.java

Limitations

  1. ENABLE/DISABLE/SET/UPDATE/PROMPT-FOR/DISPLAY
  2. INSERT - no support exists.
  3. No "trigger phrase" support.  Regular triggers are fully supported but the "inline" triggers implemented in a variable or widget definition are not implemented at this time.
  4. Validation expressions for an array element that is referenced in the expression via subscript will fail.

Unreachable Code Processing

The purpose of the Unreachable Code Processing (UCP) is twofold:
  1. To eliminate code which during conversion may result in generation of Java code which will be considered unreachable by javac. Such code causes compilation errors and therefore must be avoided.
  2. To eliminate code which can never be accessed and thus is not actually part of the application.  This reduces the amount of conversion that must be done AND more importantly it yields a much better resulting application since it is smaller and easier to understand.
The reachability of statements depends on flow control statements such as IF, REPEAT, NEXT, LEAVE and so on. These statements change execution flow and under some circumstances may change reachability of code.

The approach used to determine reachability of code is simple: all nodes of the AST tree are visited and analyzed in order to determine if they can affect reachability of subsequent statements. The first node of the AST is always reachable.

Table below lists some examples of unreachable code:

Progress Code
Java Equivalent
Notes
repeat i = 1 to 10:
...
leave.
message "Hello!".
end.
for (i = 1; i <= 10;  i+= 1)
{
...
break;
}
The code after LEAVE is not reachable.
repeat i = 1 to 10:
...
if i > 7 then
leave.
else
next.
message "Hello!".
end.
for (i = 1; i <= 10;  i+= 1)
{
...
if (i > 7)
break;
else
continue;
}
The code after IF statement is unreachable.
label_xx:
repeat:
...
case i:
when 1 then
leave label_xx.
when 2 then
next label_xx.
otherwise
return.
end.
...
end.
labelXx:
while(true)
{
...
switch(i)
{
case 1:
break labelXx;
case 2:
continue labelXx;
default:
return;
}
...
}
The code after CASE statement is unreachable.

Statements

A significant number of Progress statements may affect reachability of the other statements. Some of the statements do that directly, because they change execution flow (e.g. IF, CASE or LEAVE). Other statements, such as the PROCEDURE, FUNCTION or ON statements do not change execution flow. Instead they result in marking code as always reachable because its reachability can't be checked or depends on external conditions.

The processing of individual statements is described below:
  1. NEXT, LEAVE, RETURN, QUIT, STOP
    These statements stop the execution flow and make remaining code in the AST subtree unreachable.   Note that a reachable LEAVE statement makes code after the loop which contains this LEAVE statement, reachable. If the LEAVE statement contains a label then code after the labeled loop will be marked as reachable.
  2. PROCEDURE, FUNCTION and ON statement
    The procedure file may be a mixture of the procedure itself and internal PROCEDURE and FUNCTION statements. The reachability of these pieces of code must be estimated independently. This is necessary because a particular PROCEDURE or FUNCTION can be reachable even if it is located in the middle of unreachable code. The same is true for the ON statement.
  3. IF
    The IF in general case can make other code unreachable only if  both THEN and ELSE clauses are present and both make remaining code unreachable (for example, end with LEAVE or RETURN statements). Other special case is the situation when IF statement condition can be easily calculated. In this case code in the THEN or ELSE clauses of the IF statement can be unreachable. Also, if code in the reachable part ends with the statements from listed in #1, then code after IF statement will be unreachable.
  4. REPEAT, DO, FOR loops
    These loops may have easily calculable or implicit (REPEAT loop) condition which may result to infinite loop or loop which does not allow execution of code after the loop (there are exceptions, see #7).
  5. CASE
    The CASE statement can make other code unreachable only if it contains THEN and OTHERWISE clauses and all clauses end with statements listed in #1.
  6. EDITING and TRIGGERS phrases
    These phrases may (and usually do) contain code which ends with statements listed in #1. Direct processing of such statements may result in incorrect marking of large portions of code as unreachable. In order to avoid this, after processing of these statements the reachability flag is restored to the state which it had before processing of these statements.
  7. STOP, ERROR, QUIT and ENDKEY conditions
    These conditions may change execution flow and therefore they must be taken into account during processing. In particular, some loops (FOR and REPEAT) have default processing for these conditions. Also, all loops may have explicitly specified condition handling specified with ON phrases. This handling may include LEAVE or LEAVE <label> statements and therefore change reachability of the code after loops.
  8. Statements which may raise STOP, ERROR, QUIT and ENDKEY conditions
    These statements do not need special handling except that presence of such statements must be tracked because they may affect processing described in #7.
The expression evaluation mentioned in the #3 and #4 may calculate simple expressions such as:
  1. TRUE
  2. FALSE
  3. variable = "value"
  4. "value" = variable
  5. variable <> "value"
  6. "value" <> variable
Please note that at this time, the expression evaluation of variables is limited to variables of type VAR_CHAR being compared (EQUALS or NOT_EQ) with a literal STRING.  Changes to UnreachableCodeWorker will be needed to provide a wider range of support in the future (variable type and value type information would have to be stored and used in the variable pool and evaluation respectively).

Values and variables are stored in a customer-specific rule set named customer_specific_variables.rules.  This allows customer specific character variables to be specified with a list of values that can possibly evaluate to true . This allows such simple comparisons to be evaluated statically, yielding a increase in known unreachable code.

See also:

UnreachableCodeWorker
rules/unreachable/unreachable.xml

Produced Output

The results of unreachable code processing are stored directly in the AST: each AST node receives annotation "reachable" which contains a boolean value true or false , representing the reachability of the code. Also, information about each unreachable node is printed to standard output. This output then can be stored in the log file and then used for analysis.

Known Limitations

  1. At present TRIGGER PROCEDURE and SUBSCRIBE statements are not supported.

Unreferenced Tables/Fields Processing

The main purpose of the Unreferenced Tables/Fields Processing (UTFP) is to minimize amount of data which need to be moved to converted system by eliminating unused tables and fields.

The general approach of UTFP is simple: scan ASTs and collect references to tables and fields. This approach is complicated with following issues which need to be handled:
  1. Fields referenced in the LIKE clause should not be taken into account.
  2. An application may create a temporary table as a structural copy of an existing table. This case should NOT be handled as if original table and its fields are referenced. The reason is that this temporary table definition will be statically defined in the target application based on the structure of the Progress table however that table does not need to exist in the target database in order for the temporary table to be supported.
  3. Some statements or variants of statements may reference entire table (i.e. all fields in table). In some cases such a statement may explicitly exclude some fields from the list of all fields.
  4. Tables and fields can be accessed indirectly via system tables such as _File.
List below contains all statements which may reference all fields in table:
  1. ASSIGN record [EXCEPT field]
  2. BUFFER-COMPARE [EXCEPT field] 1,2
  3. BUFFER-COPY    [EXCEPT field] 1,2
  4. DEFINE BROWSE ... DISPLAY ...
  5. DEFINE FRAME ... [EXCEPT field]
  6. DEFINE QUERY ... EXCEPT 1,3
  7. DISABLE ... ALL [EXCEPT field]
  8. DISPLAY ... [EXCEPT field]
  9. ENABLE ... ALL [EXCEPT field]
  10. EXPORT record [EXCEPT field]
  11. FORM record [EXCEPT field]
  12. IMPORT record [EXCEPT field]
  13. INSERT record [EXCEPT field]
  14. PROMPT-FOR record [EXCEPT field]
  15. SET record [EXCEPT field]
  16. UPDATE record [EXCEPT field]
  17. DO, FIND, FOR, OPEN QUERY, REPEAT, function CAN-FIND:
  18. record-phrase EXCEPT [field]
Notes:
  1. Note if USING phrase is present then statement does reference only specified fields.
  2. May contain NO-LOBS phrase which excludes LOBS from the list of referenced fields.
  3. Field list after EXCEPT can be empty, this means that all fields are referenced.

See also:

DatabaseReferenceWorker
rules/dbref/collect_refs.xml
rules/dbref/apply_refs.xml

Known Limitations

  1. At this time, no processing of runtime preprocessor arguments is handled.  Since tables and fields can be specified via this mechanism, based on runtime calculated expressions, the range of possible values must be provided via hints.
  2. No tracking is done of references on indexes. Since indexes are built using fields this may add references to the fields as well.
  3. Perhaps better handling of the system tables is possible.

Open Conversion Issues

  1. Database
  2. UI development and fixes
    1. Review all TODO markers in both ui and chui packages.  Prepare a report on any items that are left.  We will discuss the priorities.
    2. Make a list of all non-condition exceptions generated in the thin client.  Which of these should be translated into proper Progress ErrorConditionException (use ErrorManager.recordOrThrowError())?
    3. Using testcases/uast/on_entry_no_apply.p, the on-entry event is captured for field2 and returns no-apply. After leaving field1, in Progress field2 is skipped and field3 is focused. In P2J, field3 never gets focused, focus remains on field1.
    4. Using testcases/uast/trig-wait.p, type 3 in the first field (i) and then press enter.  This will cause a pause.  If you use CTRL-C there, it will abort the pause but it won't propagate the stop condition all the way out (exiting the app) like it does in Progress.
    5. at_base_field_max_quirk10.p right aligns a number when it should not.
    6. at_base_field_max_quirk5.p has a prompt-for that should truncate the data using format "x(2)" but does not.
    7. Number editing in P2J seems to disallow backspacing over the leftmost char but in Progress this is possible.
    8. Remove frame duplication (detect frames that are functionally identical and reuse the same definition in all business logic rather than creating a new one).
    9. Remove duplicate frame.openScope(), validation expressions (search on Validation2 and widgetDaysSupply) and frame elements from the business logic.
    10. Constant (literal) header expressions can be emitted in-line (via alternate constructors for HeaderElements) instead of as inner classes.
    11. Add ControlSetItem(String) and ControlSetItem(character) constructors.  They can simply use the same object as the value and label.  Investigate using Strings instead of the character for the label (this can't be done for the value, which must be a BaseDataType).
    12. Remove \n label packing code if it is unneeded.
    13. frame_generator.xml does not need the add_format function (it's results are not ever used).
    14. Remove duplicated code in frame definitions.  For example, there are often multiple outputs of the same setFormat(...) for the same widget are added to the frame def for @ base fields.
    15. Create and use (in frame definitions) common case widget constructors that take things like: label, format string, data type, legacy name... so that the majority of the setter code in the frame def can be eliminated (reduced to some constructor parameters).
    16. redirected_output_buffering.p output differs from 4GL (see SIY).
    17. CHOOSE issues:
    18. hidden attribute/visible issues
    19. handle data type cannot be displayed in P2J
    20. MESSAGE statement error handling deviations:
    21. MESSAGE ... VIEW-AS ALERT-BOX deviations:
    22. UI performance issues:
    23. FillIn UP/DOWN key movements shift the cursor to widgets above/below however there are cases in which the cursor is placed in the first valid cursor position instead of being placed in the exact column in which the cursor resided in the previous field.  If there is data in the position being targetted, then the cursor will move there.  Note that there is no issue with cases where the cursor is supposed to move to the beginning of the field (this works). UP key applied to a fillin in the top frame row moves the cursor to the first fillin in this row. Movement from fillin to a non-fillin widget and back with UP/DOWN keys should set the cursor position in fillin to the same as it was before we have left it.
    24. Hiding focused widget hides the widget itself, but the cursor remains on its position while it should move to the nearest widget.
    25. Cursor movements are constrained more than in Progress in number editing. See editing5.p.
    26. Converted help_precedence3.p testcase fails to compile.
    27. In the testcase testcases/uast/convert_hide.p HIDE <multiple widgets> statement is converted incorrectly so the converted testcase cannot be compiled.
    28. input_lastkey.p shows that certain terminal types have the initial lastkey value set to -1 (vt220, vt320) instead of 401 (xterm)
    29. Using SESSION:DATE-FORMAT to change the date component order dynamically will not be picked up by existing instances of date (on the client side) nor by existing instances of DateFormat (client side).  See date_component_order.p.
    30. FillIn bugs with NumberFormat (see testcases/uast/num_edit.p):
    31. A failure during auto-complete of a date field should display 00 in the day or month (depending on the field being completed) and the current year in the year portion.  In P2J we draw blanks instead.  Note that only uninitialized fields are affected by auto-complete.
    32. Eliminate cursor flashing during drawing.
    33. Name skip/space widgets skipX or spaceY instead of using exprZ.
    34. Expression widget numbering should use a per-frame counter instead of one that is per-file.  For example, a given frame should not have its first expression widget named expr17 because there happened to be other frames defined in the same file.
    35. Use a single alignment approach for widgets.  The format phrase usage of AT, TO and COLON should emit to the same thing as LEFT-ALIGN, RIGHT-ALIGN and COLON-ALIGN.  The widget interface for this should be setAlignment(int) where there are GenericWidget.LEFT_ALIGN, GenericWidget.RIGHT_ALIGN and GenericWidget.COLON_ALIGN constants passed as a parameter.  The setTo(), setAt() and setColon() would be dropped.
    36. If a combo-box is defined without a size phrase, its width is determined by the rendering size of the format string.  testcases/uast/combo_box/combo_box7.p shows this case.  Note that the BaseDataType.calcFormatLength() and BaseDataType.formatLength() can be used to calculate the rendered size of a format string.  This is the size of the variable portion of the combo-box.
    37. The "thumb" button on a scrollbar positions differently.  In particular, the algorithm that determines when it moves and where it is to be positioned is slightly different.  A user that is not comparing the two implementations side-by-side may not notice this, but in P2J the thumb button will move at different times and may be positioned differently at any given time.
    38. Determine the nature of some unexpected Progress behavior when displaying uninitialized field level widgets in the testcases/uast/uselect2.p testcase. When using a 2 DOWN frame, the selection  list is shown as uninitialized for the iteration 1 and 3, and as initialized for the iterations 2 and 4. This is expected. However, starting with the iteration 5, all subsequent iterations show the selection list as initialized. This does not happen when using 3 down frame, for instance.
    39. The double buffering output driver has solved many subtle issues in PUT SCREEN handling, but not all of them. There is a class of situations where it fails, backed by these testcases: put_screen_flush*.p. These are the observed facts and their interpretation:
    40. Browse issues. You can view reproductions using the browse_issues.p testcase. The reproductions theirselfes are placed in the code of this testcase and each item below references a reproduction from it.
  3. final runtime development:
    1. Client side support for:
    2. Directory API needs improvements.  For example, one would think that adding a single string node into the directory could be easily done with a single API call.  If you look in the API, you will find addNodeString().  This does not do what you might think (even after reading the javadoc).  So the docs are confusing and the simple API is missing.  Something similar to the MinStartupDirectory.addString() method is what would be useful.  All basic datatypes should have such simple helpers.
    3. Default P2J directory exports should not be directory driven otherwise simple/stupid admin mistakes can kill the server.  Shift the directory to a RemoteObject approach.
    4. The security manager must provide warnings before certificates actually expire so that production installations have adequate time to handle this issue.  This warning MUST NOT be only at startup as a server may run for months crossing the expiration boundry with no warning...  The lead time should be customizable via a directory entry.
    5. Review the state of output and input-output parameters for procedures/functions when the block aborts due to an error.  It is possible that in Progress the assignments never occur but in ours the assignments are done "as we go".  We may need to handle this on exit as a special kind of undo OR we can use temporaries and do a batch assign at the end (this may have negative consequences for database validation...).
    6. Constructs like opsys = "VMS" are not being recognized as dead code.
    7. Performance tune stream processing, especially redirected terminal mode (report generation can be slower in P2J by a noticeable amount of time).
    8. The build.xml is broken (it always compiles all frame classes, even when they have not changed).
    9. Logging improvements:
    10. Invent a strategy to obtain a certificate alias for a remote server using a reverse lookup of hostname/IP address and port/service name.  This is required if we want to support requester-side validation of a remote server's certificate in server-to-server connections (i.e., virtual sessions).  See [Router]SessionManager.connectVirtual(InetSocketAdress address, String certAlias, SessionListener notify) in the net package.  Currently, this method disables this check if a null certAlias is provided, which is the common case.  But there currently is no mechanism for the caller of this method to easily determine what to pass for a remote server's alias.  Currently, the directory contains a default/runtime/port_services path to enable lookups of port number by service name (since Java offers no standard way to do this).  We would probably want to merge this into a larger table that took hostnames/IP addresses and certificate aliases into account as well.
    11. Rename SecurityManager to SecurityMgr to avoid the name collisions with the java.lang version.  (NOTE [ECF]:  I disagree;  I don't like class names containing abbreviations -- also note it violates our coding standards.  The collision is easily resolved with an explicit import statement for the P2J SecurityManager.  You will rarely (if ever) need to access both in the same source file.  In fact, over 10+ years, I have never had to use java.lang.SecurityManager in normal Java development).
    12. Move the following into com/goldencode from the com/goldencode/p2j location:
    13. The SSL handshaking procedure of the protocol sends no client certificate to the server even when one is specified.
    14. Is any usage of a custom locale needed for purposes of non-database "character" type sorting and comparisons?
    15. RETURN ERROR should not work in a function (it should be like a RETURN ?).  Only procedures and triggers honor RETURN ERROR.
    16. unknown value in a CASE statement doesn't work as in 4GL
    17. each file has 2 entries in the registry.xml (with ./ prefix and without it), eliminate one (and fix whatever code is dependent upon it)
    18. p2j/testcases/uast/ieee_754_representation_issue.p shows a deviation where the inexact representation of Java's double causes two (different but close) double constants in the source program to be determined as the same number when they really aren't (this may only occur with numbers that are dependent upon the 11th digit to the right of the decimal point when there are 6 digits on the left --> thus causing the total number of representable significant digits to be too large for double to handle)
    19. Add BaseDataType constructors that allow a string name (which if non-null) would force a SharedVariableManager registration inside the constructor.
    20. frame_hiding02.p demonstrates TM behavior (probably in infinite loop protection) that is incorrect.  In particular, in Progress we see 0 1, 2, 3 on the first row of output but in P2J, there is a 0 1 and then the loop iterations end prematurely.
    21. helloworld.p has an NPE when run because of a conversion error in array var initialization.
    22. string_perf.p has an unreachable issue (failing code commented out).
    23. progress.g has a parsing issue when encountering a format string without quotes and a date format has more than 2 characters in the 1st or 2nd date component like 99-9999-99.
    24. scope promotion issue: update c1 c2 = c1 + "text". will create an EmbeddedAssignmentExpr which cannot resolve var c1
    25. Write directory merge tool.
    26. The XML schema for the directory is unnecessarily complicated.  While it makes sense to use child elements to multi-valued attributes, for the primitive node types, it is complete overkill.  A single string node can be specified as <node class="string" name="mynode" value="stuff" />. 
    27. Retry processing quirks:  when validation of a new temp-table record fails due to the buffer being flushed in preparation for a query (loosely meaning FIND, FOR, etc.), the 4GL displays more validation error messages than one would expect, if outside a transaction.  We do not emulate this behavior in P2J.  Note that the number of messages varies depending upon the existence of previous message and pause statements in the procedure.  See testcases uast/flush-by-query[1-4].p.  When inside a transaction, we behave the same as the 4GL (a single error message).  See testcases uast/flush-by-query[5-9].p.
  4. Build administrative tools.
    1. provide editing support for the server, security... configuration values in the directory that cannot be edited today
    2. finish the runtime/console mgmt features
    3. add a directory editor to the admin GUI

Planned Optimizations, Refactoring and Future Development

  1. Dead code removal:
  2. Expression rewriting:
  3. Remove unnecessary wrapping/unwrapping for expressions that where the primitive results could be directly used instead.
  4. Wrapping and unwrapping could be implemented in a more simple manner by always marking the literals for wrapping rather than marking the parent.  Perhaps other approaches might also simplify this.  Right now this mechanism is too hidden, obscure and complicated.
  5. Provide alternate signatures for LogicalTerminal.message() that allows BaseDataTypes to be passed instead of an AccessorWrapper.  Then change conversion for var refs to eliminate the "newAccessorWrapper(var)" and replace it with a simple "var".
  6. ControlFlowOps.setReturnValue("") can probably be done from inside the TransactionManager at every top-level scope open.  Then this code can be removed from the business logic.
  7. Progress 4GL if/else if/else if/else does not emit as a Java if/else if/else if/else but instead as an if/else { if/else { if/else {] }} which is sub-optimal from a readability perspective.
  8. Convert Progress 4GL if/else do: if/else end. into a Java if/else if/else construct which is easier to read.
  9. Functions should be analyzed to detect if they can ever trigger conditions to be raised such that they would need the TransactionManager infrastructure.  If not they can be emitted as much simpler methods.
  10. repeatEmit is probably no longer needed in literals.rules.
  11. The parser should implement format_string as KW_FORMAT^ STRING instead of KW_FORMAT^ expr.  This was done when processing initially because of hitting something that didn't parse right. At this point, the parsing of these obscure areas has been improved greatly such that this is no longer an issue.  The project must be reviewed to understand if this can be done.  Then all the rule sets would have to be modified to handle the missing EXPRESSION node.
  12. Format string constants that are not referenced in the business logic should not exist.
  13. The extra block if not an "integrated_simple_do" is not always needed in control_flow.rules.  This results in an unnecessary set of { } output in many blocks in the Java code.
  14. Eliminate undo properties from vars that don't need it.
  15. Hide undo registration (possibly inside the wrapper constructors).
  16. Emit an accessor call to the buffer (which returns a Resolveable or a  instead of generating new FieldReference(buffer, "field").
  17. Long lived user/process contexts will probably require us to deregister global resources (e.g. Streams) that are closed rather than leave the reference around in the finalizables list.  Or we can use a weak hash map.
  18. Reset the counter for "condition" and "editingLabel" in <init-rules> for annotations rule sets so that each file will have a 0 based set of indexes.
  19. Remove unused/unreferenced frames, streams, buffers, queries, variables...
  20. Replace all usage of @inheritdoc with explicit coding of the javadoc code, even though it duplicates text.
  21. Rename frame definition callback setup() instead of setUp().  Move the batch(false) call out of this setup method and back into the GenericFrame class which calls this method (there is no reason to emit this when it will always be needed anyway).
  22. The chui package's FillIn formatting classes (DateFormat...) should maximize the use of the BaseDataType formatting support instead of duplicating it.
  23. Constant variable detection and rewriting as final statics.
  24. AdaptiveQuery should be more restrictive in its decision to invalidated a preselected result set in the case of record deletes and insertions.  Specifically, these cases should only invalidate the result set if the deleted/inserted record would actually affect result set traversal.
  25. Query substitution parameter inlining:  currently, inlining of parameters is based upon database dialect.  We probably should base this on directory configuration, using reasonable defaults.
  26. Smarter implementation for record counting loops and record deleting loops.  Note:  need to consider pessimistic locking implications here.
  27. Avoid emitting unnecessary, top-level instance variables for function parameters, which are later hidden by block-level, local variables of the same name, and avoid initializing the top-level variables in the init() method of the inner subclass of Block which is emitted inside the method which represents the converted function, so long as those top-level instance variables are never needed.  There is some scenario where these are needed (not sure what that is at the time of writing), but in other cases, they are declared and initialized, but never subsequently read.  The testcase uast/query-test-for-scrolling.p, for instance, has a function updateRecords(input valOld as int, input valNew as int).  The valOld parameter generates both an instance variable in the top-level class QueryTestForScrolling, as well as an instance variable in the anonymous inner subclass of Block within the updateRecords() method.  In the init() method of that inner class, QueryTestForScrolling.this.valOld is initialized to the value of the anonymous inner class' valOld variable, but then the top-level valOld is never accessed.  Both the top-level class' valOld instance variable, and the initialization of it in the inner class' init() method (and in fact the entire init() method, since this is all that is done there) could be eliminated as a code cleanup optimization.
  28. Macro-pattern matching.
  29. "if ___ then x = y + z. else x = y + z + a." can be rewritten as "x = y + z.  if ___ then x += a."
  30. Emit primitives instead of wrappers based on hints or heuristics.
  31. How might we define beans for highly coupled/related shared variables such that all of these related members are stored and retrieved in the shared variable manager in one call?
  32. Improve refactoring by enhancing the call graph with the (currently lost) knowledge of loops/recursion and duplicate calls.  This knowledge may allow the aggregation of multiple procedures into a common class.
  33. Implement call graph mode for the pattern engine.  This will allow much more advanced analysis of resource usage (across scope boundaries), undoability, dead code analysis etc...
  34. Remove debug code in the Progress source.  (How do we identify such things?)
  35. Fix bug in date.neutralMillisToCalendar().  It does take the daylight savings time offset into account in that method.  The problem is that the DST offset is queried from a GregorianCalendar object that is set to the current (today's) date/time.  This means that if the current TZ is in DST right now, the DST offset will be positive for all dates!  That is why any date in the DST range is fine but those outside are off.  I need to think on a solution for this as it is not as easy as setting the target date/time into the calendar first, since anything very close to the DST transition point may still calculate incorrectly.  The following testcase shows the issue:
  36. Unimplemented Progress features:
  37. Deviations from Progress behavior:
  38. Complete support for database methods and attributes.  The first release supports only a handful of database-related methods and attributes.
  39. Support "preselect fetch" FIND statement where clauses.  These statements can have their own where clauses which further refine a find among the preselected results.  Currently, these are not handled.
  40. Support a multi-table AdaptiveQuery.  Currently, multi-table joins which cannot be converted to PreselectQuery convert instead to CompoundQuery.  The latter is inefficient because the join is performed at the database client and it therefore requires an order of magnitude more database-level queries for each join than would a multi-table AdaptiveQuery implementation.  A multi-table AdaptiveQuery would perform the join on the server when in preselect mode, but would fall back to using CompoundQuery when converting to dynamic mode.
  41. Future Development


Copyright (c) 2005-2010, Golden Code Development Corporation.
ALL RIGHTS RESERVED. Use is subject to license terms.