Skip navigation links

Package com.goldencode.p2j.convert

Provides specialized Progress 4GL intermediate form representation (abstract syntax tree or AST) to Java AST conversion services.

See: Description

Package com.goldencode.p2j.convert Description

Provides specialized Progress 4GL intermediate form representation (abstract syntax tree or AST) to Java AST conversion services.

Author(s)
 Greg Shah
Sergey Yevtushenko
Eric Faulhaber
Nick Saxon
Constantin Asofiei
Stanislav Lomany
Ovidiu Maxiniuc
Date
 June 21, 2013
Access Control
 CONFIDENTIAL

Contents

Pattern Engine Rule Sets
Data Type Mapping
Literals
Naming
Declaring Class Data Members and Local Variables
Operators and Expression Evaluation
Variable References
Assignments
Function/Procedure Definitions
Function Calls
Methods and Attributes
Control Flow
Common Language Statements
Transaction Processing and Block Properties
Accumulator behavior
Database Support
User Interface Support
Unreachable Code Processing
Unreferenced Tables/Fields Processing
Open Conversion Issues
Planned Optimizations, Refactoring and Future Development

Pattern Engine Rule Sets

A great deal of the conversion processing is handled from pattern engine rule sets rather than being encoded in Java classes.  This yields a simpler implementation whenever the conversion task is such that the pattern engine's tree walking can be used to mask the complexity.  There are cases where Java helpers are needed and in these cases, methods are exposed to the rule sets from conversion oriented pattern workers.

It is important to carefully review the standard conversion rule sets in the p2j/rules/convert directory of the project.

Data Type Mapping

Summary

The following is a summary of the approach to mapping Progress data types to Java data types.

Progress Type
 Parser Token Type
 Java Type(s)
 Initial Value
 Notes
character
 VAR_CHAR
FIELD_CHAR
 com.goldencode.p2j.util.character ""
 Progress uses a single data type to handle both single character values and strings.

For situations such as event processing (keystrokes), the Java char may need to be used in preference.  In addition, the data storage itself could be handled by java.lang.String.

See wrapper considerations below.
int64 VAR_INT64
FIELD_INT64		 com.goldencode.p2j.util.int64

(or long in cases where it is determined that unknown value can never be assigned or compared to this variable)		 0L Starting with version 10.2B, the Progress uses this type of integer for results and internal/intermediary computations.

The int64 is a 64-bit signed integral value whose maximum value is 9223372036854775807 and minimum value is -9223372036854775808.  Overflows wrap around to the minimum value and underflows wrap around to the maximum value. This is an exact match to the Java primitive long, except for the case where the unknown value is assigned or tested on this variable.

The logical operators have a predictable response to having the unknown value as one or both operands.  This processing is identical for all data types. See the table below.

See wrapper considerations below.
integer
 VAR_INT
FIELD_INT
 com.goldencode.p2j.util.integer

(or int in cases where it is determined that unknown value can never be assigned or compared to this variable)
 0
 The Progress integer is a 32-bit signed integral value whose maximum value is 2147483647 and minimum value is -2147483648. 32-but overflows will not occur as Progress works on 64-bit values even if both operands are 32 bits. This is an exact match to the Java primitive int, except for the cases where the unknown value is assigned or tested on this variable or when two operands are processed, they are automatically widened and the operation is performed on 64-bit.

The logical operators have a predictable response to having the unknown value as one or both operands.  This processing is identical for all data types.  See the table below.

See wrapper considerations below.
decimal
 VAR_DEC
FIELD_DEC
 com.goldencode.p2j.util.decimal

(this class will be implemented using double and NaN to represent the unknown value; BigDecimal will be used as a backing data type in the case where it is required; a default number of significant digits right of the decimal point will be 10 but can optionally be set to a smaller value based on the "decimals" keyword override)
 0
 The Progress decimal is a signed floating point value whose maximum value is 50 significant digits in size (the mantissa or significand is a maximum of 50).  Up to 10 of those digits can be to the right of the decimal point (the exponent is a maximum of 10).  Thus the largest value is 99999999999999999999999999999999999999999999999999 and the minimum value is -99999999999999999999999999999999999999999999999999.  If all of the 10 possible digits of precision are used to the right of the decimal point, then only 40 digits can be used on the left. Overflows and underflows both cause the same runtime error "Decimal number is too large (536)".

The logical operators have a predictable response to having the unknown value as one or both operands.  This processing is identical for all data types.  See the table below.

For more details, please see Decimal Support.
logical
 VAR_LOGICAL
FIELD_LOGICAL
 com.goldencode.p2j.util.logical

(or boolean in cases where it is determined that unknown value can never be assigned or compared to this variable)		 false
 The Progress logical has multiple "modes" yes/no, true/false and a user defined value1/value2.  However, all of these are the exact equivalent of the Java boolean type which has only 2 states, true or false, except for the case where the unknown value is assigned or tested on this variable.

The logical operators respond differently to having the unknown value as an operand:

Operator
 Left Operand Unknown
 Right Operand Unknown Both Operands Unknown
EQ
 false
 false
 true
NE
 true
 true
 false
LT
 ?
 ?
 false
LE
 ?
 ?
 true
GT
 ?
 ?
 false
GE
 ?
 ?
 true
AND (logical type operands only)
 false if right operand is false, ? if right operand is true
 false if left operand is false, ? if left operand is true
 ?
OR (logical type operands only) true if right operand is true, ? if right operand is false
 true if left operand is true, ? if left operand is false ?
NOT (logical type operands only) n/a
 ?
 n/a

Note that any logical expression that evaluates to the unknown value is always considered false (e.g. for purposes of control flow in an IF statement).

See wrapper considerations below.
date
 VAR_DATE
FIELD_DATE
 com.goldencode.p2j.util.date
 null
 A custom wrapper class will be written to provide the proper semantics of date processing in Progress.  Where needed, an instance of java.util.Calendar (obtained through Calendar.getInstance() will be used for the calendar processing however all operators will operate on the day number directly for speed and simplicity.  The custom wrapper class will provide Progress semantics and will hide the complexities of dealing with  the J2SE Calendar interfaces.

The logical operators have a predictable response to having the unknown value as one or both operands.  This processing is identical for all data types.  See the table below.

Note that there are special considerations for handling dates within a database where clause.
datetime VAR_DATETIME
FIELD_DATETIME		 com.goldencode.p2j.util.datetime null A custom wrapper class written to provides the proper semantics of datetime processing in Progress. Where needed, an instance of java.util.Calendar (obtained through Calendar.getInstance() will be used for the calendar processing however all operators will operate on the day number and time directly for speed and simplicity. The custom wrapper class provides Progress semantics and will hide the complexities of dealing with; the J2SE Calendar interfaces.

The logical operators have a predictable response to having the unknown value as one or both operands. This processing is identical for all data types. See the table below.

Note that there are special considerations for handling datetimes within a database where clause.
datetime-tz VAR_DATETIME_TZ
FIELD_DATETIME_TZ		 com.goldencode.p2j.util.datetimetz

Note that this is the only wrapper that does not have the same name with the datatype from Progress. This is because the hyphen found in the Progress name is not permitted in inside an identifier of the Java language. 		null A custom wrapper class written to provides the proper semantics of datetime-tz processing in Progress. Where needed, an instance of java.util.Calendar (obtained through Calendar.getInstance() will be used for the calendar processing however all operators will operate on the day number, time and timezone directly for speed and simplicity. The custom wrapper class provides Progress semantics and will hide the complexities of dealing with; the J2SE Calendar interfaces.

The logical operators have a predictable response to having the unknown value as one or both operands. This processing is identical for all data types. See the table below.

Note that there are special considerations for handling datetime-tz within a database where clause.
rowid
 VAR_ROWID
FIELD_ROWID
 com.goldencode.p2j.util.integer n/a
 The persistence runtime classes transparently map the ID field for the associated database row to an integer wrapper.
recid
 VAR_RECID
FIELD_RECID		 com.goldencode.p2j.util.integer n/a The persistence runtime classes transparently map the ID field for the associated database row to an integer wrapper.
raw
 VAR_RAW
FIELD_RAW
 com.goldencode.p2j.util.raw 0 size array
 In the Java-based runtime, this is not backed by a pointer to memory.  Instead, it is backed by a byte array.
memptr
 VAR_MEMPTR
 com.goldencode.p2j.util.memptr uninitialized array
 In the Java-based runtime, this is not backed by a pointer to memory.  Instead, it is backed by a byte array.
handle
 VAR_HANDLE
FIELD_HANDLE
 com.goldencode.p2j.util.handle no contained object
 Note that the parser converts all widget handles into handle types since in Progress these are equivalent.

This class is really just a container that holds a backing object.  It provides unknown value processing, comparison/assignment operator support and the valid-handle() builtin function implementation.

The specific backing object will need to be determined on a case by case basis.  The handle gets dereferenced when used as a referent for methods and attributes.
com-handle n/a n/a n/a n/a

See also (these are rule sets containing the pattern engine logic that handles variable conversion):

rules/annotations/variable_definitions.rules (name conversion and class name determination)
rules/convert/variable_definitions.rules (variable definition and initialization)
rules/convert/variable_references.rules (variable references)
rules/convert/expressions.rules (unwrapping of variables)
rules/convert/literals.rules (initialization constants)

Decimal Support

The Progress decimal is a signed floating point value whose maximum value is 50 significant digits in size (the hh is a maximum of 50).  Up to 10 of those digits can be to the right of the decimal point (the exponent is a maximum of 10).  Thus the largest value is 99999999999999999999999999999999999999999999999999 and the minimum value is -99999999999999999999999999999999999999999999999999.  If all of the 10 possible digits of precision are used to the right of the decimal point, then only 40 digits can be used on the left. Overflows and underflows both cause the same runtime error "Decimal number is too large (536)".  

The Java primitive double can easily handle the smallest values that can be represented by a Progress decimal.  The Java double is based on a 64-bit IEEE 754 floating point value where the maximum value is 1.7976931348623157 * 10 308 and the minimum value is 4.9 * 10 -324 .  The biggest issue is in duplicating Progress behavior with small numbers near its limit of precision.  For example:

0.0000000001 / 2 * 2 = 0.0000000002

In Java this would be calculated as 0.0000000001.  The Progress rounding behavior for very small numbers is not limited to just the operators but rather it seems that they have an internal floating point representation that is precise to 11 digits.  Every time they convert the "external" representation into this form, the rounding occurs.

For example:

0.000000000149 + 0.000000000001 = .0000000001

0.00000000005 =  .0000000001

0.00000000015 =  .0000000002

In this case, the important rounding occurs before the operation rather than after.  The net of this analysis is that all Progress use of decimal variables must be reviewed to determine if there are any conditions which are sensitive to the boundary between the 10th and 11th digit of precision (to the right of the decimal point).  This would be rare EXCEPT in the condition where the result of a calculation is something like 2.999999999999996 which in Progress would be represented as 3.0 but which is directly represented in Java.

The following approach seems (based on some non-exhaustive testing) to exactly duplicate the default Progress results for any double d:

Math.round(d * 10000000000L) / 10000000000.D

For any decimal variable specified with the DECIMALS clause (some value x which is < 10), the proper result can be generated by converting the two constants (e.g. 10000000000L and 10000000000.D which are both 10 10 ) to a version based on 10 x .  This solution handles the rounding (leveraging the higher precision values just as the Progress "internal" floating point implementation does) before the truncation occurs.  This is quite important.

Interestingly, since the IEEE floating point format is really a data structure (e.g. the double has 3 fields for sign, exponent and significant)  rather than a primitive, it has forms reserved for Infinity and NaN (not a number).  The NaN in particular can be considered the analog to the Progress unknown value.  In fact, it is silently propagated through mathematic operators into the result just as Progress does.  So if one or both of the operands are NaN, the result is NaN.  With this idea, it may be feasible to duplicate the Progress behavior without a Double wrapper!  Some J2SE math functions (e.g. Math.round()) do generate results that are not NaN (an int or long set to 0 in the case of round()) so calls to such methods must be handled specially. The NaN comparison logic is different from the Progress unknown value comparison logic, so this would need to be handled.   See wrapper considerations below.

The logical operators respond differently to having the NaN as an operand:

Operator
 Left Operand NaN
 Right Operand NaN Both Operands NaN
==
 false
 false false
!=
 true
 true
 true
<
 false false false
<=
 false false false
>
 false false false
>=
 false false false

As in all floating point implementations, the system's "epsilon" and rounding behavior must be matched up.  An epsilon is the smallest floating point value that can be added to another floating point value such that the result can be differentiated as not equivalent by the native environment's operators.  As with everything else about Progress decimal support, the Progress and Java differences are substantial.

In Java, the epsilon is defined (by the language specification) as being equal to 2 -52 or 2.22044604925 X 10 -16 .  This is an exceedingly small number, especially in comparison with the smallest possible Progress decimal number.  So the following application was written to identify the epsilon for Progress:

def var possibleEpsilon  as decimal init 1.0.
def var epsilon          as decimal init 0.0.
def var mantissaBits     as integer init 0.

repeat.
   if ((1.0 + possibleEpsilon) = 1.0) then leave.
   epsilon = possibleEpsilon.
   possibleEpsilon = possibleEpsilon / 2.0.
   mantissaBits = mantissaBits + 1.
   display possibleEpsilon format "99999999999.999999999999999" mantissaBits.
end.

message "Epsilon = " + string(epsilon).
message "Mantissa Bits = " + string(mantissaBits).

It turns out that this program will generate an infinite loop.  This is because once the possibleEpsilon is reduced down to 0.0000000001 (the smallest representable Progress decimal), the result of the possibleEpsilon / 2.0 sub-expression will always be 0.0000000001 instead of 0.00000000005.  Since a difference of 0.0000000001 can be detected as different from any other decimal value, another way of looking at this result is that there is no epsilon value that is "visible" to the programmer.  There is certainly an internal Progress epsilon value but due to the fact that Progress rounds all intermediate results (after each operator runs), the result is always something that cannot be smaller than 0.0000000001 unless it is 0.0!  So:

0.0000000001 / 2.0 results in 0.0000000001
0.0000000001 / 3.0 results in 0.0

The important result here is that epsilon testing (the "normalization" of floating point comparisons to detect if something is really different or not) is not an issue for Progress applications because of the rounding/truncation behavior.   Internally, the Progress system maintains a larger number of digits of precision (than 10) but whenever accessed from an application (via an operator, assignment...) it is always truncated to the lesser of 10 digits (or the value of the decimals option for that variable) and the truncation uses rounding to determine the least significant digit.  In order to ensure compatible floating point operations in Java, the following rounding algorithm will be required after every operator (assuming num is the double result to be rounded/truncated and the default of 10 digits is assumed):

Math.round(num * 10000000000L) / 10000000000.D

Given this rounding implementation on the Java side, the normal epsilon testing is not necessary AND it is reasonably easy to implement the Progress "decimals" option which allows the programmer to specify an arbitrary number of digits of precision that is less than the default (and maximum) 10 digits.  So as a more general algorithm (where x is the number of digits supported):

Math.round(num * 10x L) / 10x.D

The main idea here is that result is converted into the long primative type which is used to capture those digits that must be maintained and to drop the other digits, then it is converted back into a double .  The Math.round() handles the proper rounding behavior for the least significant digit.

The situation with large decimal numbers is also difficult.  The largest decimal values cannot be represented in a Java double without losing precision.  The Java double has a mantissa which is represented by 53 (52 explicit bits and 1 implicit bit in normalized forms) bits or ~16 decimal digits.  This means that any integer number with 17 or more digits can't be saved in double without precision loss (the least significant digits are lost).  Arithmetic and comparisons of numbers with > 16 digits will be incorrect when using a double.

To address this large number issue, decimal operations must be reviewed for any situation that could overflow this boundary.  In these cases, the BigDecimal class will be used.

The Java double overflows by ignoring any overflow and setting the value to the maximum value (positive or negative infinity).  The Java double underflows by "wrapping" to (positive or negative) 0.  Special "boundary" processing will need to be written for some generated code due to the fact that Progress has a significantly smaller precision, a significantly smaller maximum mantissa and different overflow/underflow behavior.

To Wrap or Not to Wrap?

Where possible, the resulting generated code will be significantly easier to write, read and maintain when using primatives instead of object wrappers (e.g. int instead of Integer).  There is also a performance advantage to using primatives since this avoids the overhead of temporary variables and method calls to handle expression processing. This is an issue for integer, decimal ( note that NaN may be a solution that still allows primative doubles to be used in this case ) and logical types since in Progress, such variables can be assigned and compared with the unknown value.  The unknown value is a similar thing as the Java concept of null .   The main difference is that Progress silently allows unknown value to "propagate" where Java tends to catastrophically fail at runtime (e.g. NullPointerException ).  In Progress an arithmetic sub-expression will return an unknown value if either or both operands are the unknown value.  Since primitive Java types cannot be assigned or compared with null , wrapper objects or some other solution must be used in that case.

In the following conditions an integer, decimal or logical variable must be represented by a wrapper class:
  1. All parameters defined via a "define parameter" language statement that have an input or input-output type are passed in and thus they may be passed as the unknown value.  An output type parameter doesn't require wrapper support in and of itself because its origin is known.
  2. If the variable is shared and is defined external to this procedure then a wrapper must be used because it must be assumed that it could be assigned the unknown value or compared against the unknown value either upstream.  Any new shared variables in this procedure do not need to be wrappered unless some other condition would cause this to occur.
  3. If the variable is explicitly initialized to the unknown value.
  4. If the unknown value is ever assigned to that variable through the assignment operator or the assign language statement.
  5. If the unknown value is ever used on the other side of any binary operator (all the arithmatic operators handle operands of unknown value by returning the unknown value) including multiply, divide, kw_mod, plus, minus, equals, not_eq, gt, lt, gte, lte, kw_and and kw_or.  The kw_begins, kw_matches and kw_contains binary operators do not operate on integer, decimal or logical variables so they are not at issue.
  6. If the variable is ever passed to an external procedure as an input-output or output parameter (in a run statement), a wrapper will be used to avoid a complex recursive/multi-file search for downstream comparisons/assignments against the unknown value.  An input type parameter doesn't require wrapper support in and of itself because its origin is known.  Input-output or output parameters of an external procedure (run statement) are potentially modified in the called procedure which means that this procedure could see an unknown value which is generated externally.
  7. If the value is ever assigned or compared to the return value of a function call which can return the unknown value.  A subset of the built-in functions can do this and some user-defined functions can return the unknown value.
  8. If the value is ever assigned or compared to the return value of a method or attribute.
  9. If the variable is defined as a parameter of a function (function declaration) that has an input or input-output type.  An output type parameter doesn't require wrapper support in and of itself because its origin is known.  If this function is only called from the current procedure, then the call sites can be examined to determine if the input can ever be unknown.  If the function is called externally (through the "in handle" form), then a multi-file review would have to occur which is beyond the scope of this pass of the conversion.
  10. If the variable is ever passed in a function call in a position that matches an input-output or output parameter.  Function call parms that match up with input-output and output types are obviously a problem in that hidden code may be invoked to assign the unknown value to that variable before returning.  It is possible that a local function could be analyzed but to limit the complexity of this processing, this will not be done.  Built-in functions and remote functions are not as easy to analyze.  In the case of built-in functions, each one would have to be reviewed manually as a "black box" since no source is available in these cases.
  11. If any variable or field in any expression can be the unknown value, then any other variable assigned from expression can be silently set to the unknown value as a result of the expression evaluation.  This last implication causes a requirement to iteratively process expressions expanding the list of possible nullable variables based on assignments from other variables/fields that are nullable, until no additional nullable variables are found.  This will take an undefined number of iterations to finish since each iteration can add to the list of nullable variables and then a subsequent scan will lead to new nullable variables that are only nullable because of the last scan's changes... and so on.
See also:

Wrapper Usage

Other Variable-Like Entities

Progress allows many user-defined resources that are not treated as variables.  A non-exhaustive list includes streams, frames, widgets, queries, buffers, and temp-tables.   These resources cannot be used in expressions nor can they be used as a storage type in the database, however in Java their representation will be as objects.  Each of these resources will be converted by resource-specific code and this section of the conversion does not address such resources.

Literals

The following is a summary of the approach to mapping Progress literals to Java literals.

Progress Literal
 Parser Token Type
 Java Literal/Object
 Notes
true or yes
 BOOL_TRUE
 true
 This is an exact match.
false or no
 BOOL_FALSE
 false
 This is an exact match.
? (unknown value)
 UNKNOWN_VAL
 If used as the right side of an assignment, this is covnerted to a setUnknown() method call on the BaseDataType object that is the lvalue.

If used as 1 operand of a EQUALS, the other operand will be passed to CompareOps.isUnknown().

If used as 1 operand of a NOT_EQ, the other operand will be passed to CompareOps.notUnknown().

In Progress it isn't possible to compare two instances of the unknown value literal.

Other usage converts to an instance of com.goldencode.p2j.util.unknown.
 At first glance, null is an exact match.  However, in order to eliminate the need to place null checks throughout code and to refactor much of the code, wrappers are being used which internalize the unknown value.
integer literal
 NUM_LITERAL
 Java integer literal
 This is an exact match since Java integer literals default to int.
int64 literal NUM_LITERAL
with "use64bit" annotation set to true.		 Java long literal This is an exact match.

The decision which type of numeric literal is taken dynamically at conversion time:
  1. If a literal is parsed and fits into 32-bit space, the int type is used;
  2. Else it it can be represented on 64 bits, the long type is used instead;
  3. Otherwise, it will be represented as a decimal/ double.
decimal literal
 DEC_LITERAL
 Java floating point literal As long as the value is within the mantissa limits listed above (16 decimal digits), this is an exact match since Java floating point literals default to doubles.  Otherwise, a BigDecimal must be used.
date literal DATE_LITERAL com.goldencode.p2j.util.date There is no date literal in Java. An instance of the com.goldencode.p2j.util.date will be created instead using the static method date.fromLiteral().
datetime literal DATETIME_LITERAL com.goldencode.p2j.util.datetime There is no datetime literal in Java. An instance of the com.goldencode.p2j.util.datetime will be created instead using the static method datetime.fromLiteral().
datetime-tz literal DATETIME_TZ_LITERAL com.goldencode.p2j.util.datetimetz There is no datetime-tz literal in Java. An instance of the com.goldencode.p2j.util.datetimetz will be created instead using the static method datetimetz.fromLiteral().
string literal STRING
 Java string literal
 Any string options will be removed and any enclosing single or double quotes will be removed.  Escaped characters (using the tilde character) will be converted to native representations and any characters that need to be escaped will be so escaped:
  • '\b' - backspace
  • '\t' - horizontal tab
  • '\n' - linefeed
  • '\f' - formfeed
  • '\r' - carriage return
  • '\"' - double quote
  • '\'' - single quote
  • '\\' - backslash
  • '\ooo' where ooo is an octal character literal
  • '\uxxxx' where xxxx represents 4 hexidecimal digits - unicode character literal
The final result when output will be output inside enclosing double quotes but there is no need for those quotes inside the AST form.

See also:

LiteralConverter
ExpressionConverterWorker

Naming

Progress identifiers and Java identifiers are similar but there are some important incompatibilities.  Most importantly the following characters are valid in a Progress name but invalid in a Java name:

#
%
&
-  (this is the hyphen character, not the underscore)

While Java does allow Unicode letters to be included in identifier names, the plan is to limit valid letters to the 7-bit ASCII letters of a - z and A - Z.  Java identifiers will always start with one of these letters and in subsequent characters a letter, any digit (0 - 9), the underscore (_) and dollar sign ($) can all be used.  All of these Java identifier characters can also be present in a Progress identifier.

So there is an inherent problem caused by a reduced identifier namespace on the target platform.  The troublesome characters (all of which are used in real application code) must be converted to something that is unlikely to conflict with other visible (scopes must be taken into account) names.  Conflicts must be detected and resolved automatically, with overrides being available (via hints) for manual intervention.

As an example of the conflict, assume that hypens were converted directly to underscores.  The following 2 names "hello-world" and "hello_world" would both be converted to "hello_world".  Thus if both identifiers were in some accessible scope simultaneously, then the resulting code would not be correct.

One should note that in Progress, the convention is to use the hyphen as a word delimiter in an compound word identifier.  In Java, the convention is to use a "camelCase" identifier (Java identifiers are case-sensitive) for the same purpose.

For different types of names, different conversion strategies will be followed allowing the Java standards to be matched as closely as possible.  The following is a summary of the approachs to be used.

Progress Identifier
 Java Identifier Strategy
 Conflict Resolution
directory name
 package name
 Package naming will be somewhat direct (following the Java standard):

com.<customer>.<application>.<directory>
 TBD
file name
 class name
 A set of hints will assist in class naming.  Where common sequences are encountered such characters might be expanded based on hints.  For example, so and po might be expanded to ServiceOrder and PurchaseOrder respectively.  A file ending in -r or -x might be have ReportBody or Export appended respectively.
 TBD
procedure name
 method name
 TBD
 TBD
function name
 method name
 TBD
 TBD
variable name (normal variable that is assigned or passed as a parameter)
 variable name
 Hypens will be removed and the words will be camelcased.  This will result in lowercasing some letters (including the first char of the identifier) and uppercasing the first character of subsequent words.  Words will be assumed to be separated at the hyphens.
 TBD
variable name (never assigned or passed as a parameter)
 constant name
 All letters will be uppercased and words will be separated by an underscore (_).
 TBD
schema name
 TBD
 TBD TBD

See also:

NameConverter
NameConverterWorker

Declaring Class Data Members and Local Variables

Variables can be generated in a Progress program by any of the following language constructs:

Language Statement/Construct
 Conditions
 Options Are Controllable
define variable
 always
 yes
define parameter
 when TABLE-HANDLE, AS or LIKE keywords are encountered
 yes
format phrase
 when following an unrecognized symbol and the AS or LIKE keywords are encountered
 yes
function parameter
 when TABLE-HANDLE or AS keywords are encountered no
message
 when SET or UPDATE clause occurs with a child of SYMBOL and a child of the SYMBOL which is the AS keyword (as opposed to a child which is an lvalue that has an optional AS keyword which is bogus)
 no

Reference IDs

In each of the locations above where options can be defined for a variable, those options are saved as annotations.  In addition any AST that represents a reference to that variable will have all the non-default options annotated.  To allow variable references to be cross-referenced back to the original AST node that defined the variable, a "temporary index" is maintained for each variable definition.  This index is not project unique (such as the AST node ID) but rather is only unique within a given file.  Each reference to a variable has the temporary index written to an annotation.  The reason the node IDs are not used for this purpose is because this cross-reference is best created at parse time and the node IDs don't get created until after parse time due to limitations of how ANTLR instantiates AST nodes.  Post parse fixups are used to convert these temporary ID annotations into an annotation containing the real node ID of the defining AST node.

Shared Variables

Support for shared variables is handled via a shared variable manager.  This runtime class is called SharedVariableManager .  It is based on the ScopedSymbolDictionary and is designed to exist in a per client/session basis (stored in the security context).  This allows all transactions to access these shared variables but for this access to be done maintaining context specific state.  This approach is a replacement for features that would normally be provided via thread local variables which can't be used in the transaction server.

Whenever a new external procedure scope is started, if new shared variable references are added, the corresponding Java source code will add a scope.  Likewise this same Java method will use a finally {} clause to delete the scope on exit.  Global variables are added to the global scope and regular shared variables are added to the current (top) scope.  Such additions only occur if the definition is defined with the NEW keyword.  This allows "downstream" methods to access the most current instance of a shared variable by executing a lookup using a given name and when the method that defines a new shared (non-global) variable ends, the scope must be deleted (deleting all references to variables defined in that scope).  Any process that accesses a shared variable gets the shared variable manager out of the current security context and then does a lookup of the variable name and assigns the reference to the resulting local instance.

Where possible, shared variable usage will be converted into object references passed as method parameters or contained in other classes passed as parameters.  In the first pass conversion, this is not planned due to the extreme use of shared variables (there can be tens or even hundreds of these defined in a procedure) and the arbitrary depth at which such references can be used makes it unwieldy to easily convert these to passed parameters.  For this reason, the original semantic must be maintained.  The shared variable manager provides this same semantic at runtime.

Note that this same infrastructure is used to provide support for shared streams .

See also:

rules/convert/variable_definitions.rules
rules/convert/input_output.rules
rules/convert/control_flow.rules
rules/annotations/shared_resources.rules

Extents

Progress arrays are very similar in nature to Java arrays.  A variable definition can be an array if it is explicitly specified or if it is not specified and the variable is defined like another variable or field (a LIKE clause) that is defined with extent > 0.  Either way, there will be a resulting KW_EXTENT with a numeric constant size and a "long" type annotation named "extent" on the variable definition root node.

The extent processing is mapped to array variable definitions and array variable initializers .  This is a fairly direct mapping.

The referencing of array elements is also very similar between Java and Progress (both use numeric expressions inside postfixed square brackets to index the element). The following differences exist:
  1. Progress indexes arrays using a 1-based index.  Java uses 0-based indices.
  2. There are certain language statements (e.g. DISPLAY) that support a range of indices using a KW_FOR NUM_LITERAL construct inside the square brackets (after the first numeric expression).  This form can't be used inside expressions but it will have to be handled.
  3. There are certain language statements (e.g. DISPLAY) that support referencing all elements when an array variable name is used with no subscript at all.
Requirement 1 is handled inline during conversion.  The Java subscript operator [] is directly mapped to the Progress extent operator [].  NumberType.subscript() is used to decrement the 1-based subscript and return a 0-based version.  For array variables, this method takes a reference to the BaseDataType[] being used.  This allows the subscript() method to handle bounds checking and generate the proper error.  Please note that using an array index that is out of bounds during a NO-ERROR (silent error mode) language statement/assignment will fail with an IndexOutOfBoundsException since we currently directly map array access into the standard Java subscript operator [].  While we do have a central method in NumberType.subscript() to handle the bounds checking and error generation, in silent error mode this will return an out of bounds value.

Note that any constant (NUM_LITERAL) array subscript in Progress is converted to the proper 0-based constant in Java (no use of the subscript() method is needed in that case).

For fields, since all access is via get/set methods, the [] operators are not used and the array index is passed as the last parameter of the method call (the only parm for a getter and the 2nd parm for a setter).  Bounds checking and error generation is handled inside the persistence classes in that case.

This same approach is also used in some of the frame field cases in order to properly dereference the widget in the frame (which is done via getter methods).

Requirements 2-3 are handled by expanding these array references into the proper list of references. Then all downstream processing is otherwise the same once the implicit reference is expanded into its explicit list.  This expansion works conditionally in case #2, only working where the start of the range is specified as a constant (NUM_LITERAL).  No support for array subscripts with ranges that use a non-constant expression as the start index as in myvar[ 2 + j FOR 3 ].  Such cases can not be used in an expression, instead they can only be used as implicit reference to a range of index positions.

Initial Values Logic

All variable definitions are explicitly initialized to some known value (see the above table ).  Either a null is assigned or the appropriate constructor is called with the literal initializer (default or as specified in the Progress source).  The initializers are all rooted at a node of type KW_INIT and these values can come from another variable or field (LIKE clause) or can be explicitly specified in the source code.  Either way, there will be a resulting list of initializers which are walked at conversion time to emit the appropriate constructors or null .

In the case of array processing (see Extents ) the number of explicit extents may exceed the number of initializers provided.  This is a different behavior than Java where one can specify one or the other but not both.  If there are no explicit initializers, then all elements of the array are initialized to the default value.  If any initializers are speciifed, these are applied to the matching elements in the array and then all remaining elements without an explicit initializer are initialized to the last initializer in the list.

Wrapper Usage

See To Wrap or Not To Wrap .

Based on the extreme complexity of this task, the effort to definitively determine whether an integer, decimal or logical variable requires a wrapper is going to be deferred at this time.   As a first pass, all variables will always be wrapped with the ability to add an optional hint that allows an override to a primative type when it is known to be safe.

Note that moving to J2SE 5.0 does provide some small relief by providing a new language feature for auto-boxing and auto-unboxing.  This largely makes the use of wrappers transparent (it wraps or "boxes" a primitive into a wrapper when needed and unwraps or "unboxes" a wrapper into a primitive when needed).  This makes it possible to write arithmetic and logical expressions directly using wrappers.

However, the following limitations (which are also quite significant ) apply:
  1. null cannot be assigned to a primitive and any unboxing or usage of a null wrapper will result in a null pointer exception.  This means that null checks will have to be embedded throughout expressions which greatly reduces the value of auto-boxing/auto-unboxing.
  2. The == operator cannot be used because it does not give consistent results when comparing two operands where either one or the both of them are wrappers.  This is due to the fact that when this operator is dealing with objects (versus primatives), it is comparing object references and this will generally yield false-negatives in most situations.
  3. Overloaded method signatures can be tricky because method resolution does change with auto-boxing added.  These method resolution changes are supposed to be compatible but this remains to be seen.
  4. The combination of auto-boxing and generics means that the ternary operator will auto-cast the 2 expression results to the least common denominator result if their types are different.
At this time, we do not plan to implement J2SE 5.0 since the result is not much better.  We can do our null checks and then if everything is non-null, we can unwrap into temporary primitive variables and then wrap the result back up and assign them back.

The problem with this solution is that many expressions require significant refactoring to make this work.  For example, in the case where a null check fails, all possible assignments that could be forced to the unknown value must be detected.  In addition (possibly more important), any side effects of the expression must be triggered in a safe manner.  For example, if a function is called as part of an expression, that function may have side effects that the rest of the procedure relies upon.  The conclusion: not only is the refactoring of the expression highly non-trivial but there are very strong reasons to actually allow the expression to execute even in the presence of the unknown value (which is how Progress works today).  If null is used as a proxy for the unknown value, this aspect of the behavior cannot be duplicated since Java generates a runtime error for these cases.  Since the J2SE wrappers have no way of encoding the unknown value except setting the current reference to null , this eliminates the J2SE wrappers from being a useful/good solution to this problem.

The best solution is to implement custom classes (corresponding to each Progress data type) to handle all of the following:
  1. Stores the data in a form that provides an exact match with the Progress data.
  2. Stores the state of whether this instance is really equal to the unknown value.
  3. Implements all of the operators and replacements for the Progress built-in functions such that the 100% Progress compatible results are generated.  Note that for integer and logical, this is primarily a matter of handling the unknown value in a transparent manner.  For decimal, there is the added requirement for the rounding/truncation to be implemented after each operator and function. 
All of these custom classes will reside in the com.goldencode.p2j.util package and they will each have a class name the same as the lowercase data type name (e.g. class integer for the integer data type). This is unique (does not conflict with J2SE), is easy to read and looks more like a primative is being used.  The down side is that this is an exception to the normal Sun-recommended naming rules for classes.

In a perfect world, the exact look of (arithmetic and logical) expressions could be maintained if Java supported operator overloading.  Since that is not possible, methods (with intuitive names) will be written to substitute for all operators.  Unwrapping will only be needed in those places where the expression result requires a primative type.  So assignments and many method calls will use the object form returned as an expression result.  However, when expression results are used directly with a Java language construct (e.g. if/else, array subscripts) or in a J2SE-provided method that has primatives in the signature, the primatives must be used.  In these cases, the custom wrapper object will be unwrapped.

Note that because each of these classes will internally track whether the variable is set to the unknown value, all variables will have a valid (non-null) reference.  As long as the conversion code is written with this in mind, null pointer exceptions will be eliminated (from expressions).

Progress function/procedure parameters have input-output and output forms that cause the data in the calling procedure to be changed by the called function/procedure.  This is very similar to the C/C++ semantic of a parameter passed by pointer.  On return, the data pointed to by the calling procedure has been changed.  In Java this is easily handled when some kind of business/container object is passed as a parameter but it is not directly possible with the normal primatives or their standard J2SE wrappers (e.g. Integer) which are immutable.  While some instances of function/procedure calls may end up being generated in such a way that a more abstract object is passed in instead of the primatives, this will require significant refactoring (of procedural code into object oriented code) and may not always be possible.  For a general purpose solution to this problem, it seems better to ensure that all of the custom wrapper classes are designed as mutable, with an assignment-type interface and setters to allow modification of the wrapped data in the called method , where the variable being assigned was defined as an input-output or output parameter.

Detecting Constants

There is no concept of a constant variable in Progress but some variables may essentially be "read-only".  As an optimization (and a good practice for Java code) such variables should be converted as "final static".  To do this, it is required that all assignments, run statements and user functions... are examined to determine if any possible change can ever be made to each variable.  This presents similar issues as the To Wrap or Not To Wrap discussion, however it is likely that this is significantly more feasible to implement.

Format String Support

Variables can have explicit format strings or can implicitly have a format string based on a LIKE clause (where the database field or variable has an explicit format string).  In these cases, the any of the formatted output language statements will not use the default data type format string, but instead will honor the explicit or implicit format string that is specific to the variable.

These cases are detected and a class member will be used for each unique format string in a given class.  The member is created with the following approach:

public static final String FMT_STR_n = "...";

The number n will be a generated sequence number to make the string name unique.

At annotation time, a set (the format string is the key and the name of the class member is the value) of these are accumulated and are used to cross-reference all format string usage.  This allows a single definition to be reused in multiple places.  For this reason, if 2 variables are both explicitly defined with a format string of "x(40)", then the same FMT_STR_x constant will be used for both.

The proper hierarchy for determining the correct format string to use is:
  1. Use an explicit, in-place format string (part of a format phrase on any language statement that generates formatted output).
  2. If the variable is being used as a simple variable reference (rather than an expression or constant):
    1. Use the explicit format string for that variable, if one was specified.
    2. Use the implicit format string that was associated with the variable via a LIKE clause.
  3. Use the data type specific default format.
The formatted I/O processing (all putField variants) check on this (and honors it) for each field that is a simple variable reference.

Operators and Expression Evaluation

All operators are fully implemented except the highly specialized CONTAINS operator (see page 960 of the Progress 4GL Reference, in the section on the record phrase) which handles word index searches.

Each operator is converted to a static Java method call that is implemented in a data type wrapper class ( integer , decimal , date , logical , character ) or in a common operations class such as MathOps or CompareOps .  When an operator is encountered, the left operand (and sometimes the right operand in the case of date operations) is checked for its type.  To do this, the leftmost descendants are checked until a literal, variable, field or function is found.  All of these nodes have a type that determines the type of the resulting operation.  Depending on this type, the conversion strategy is picked.

The structure of the Java expression is an exact duplicate of the Progress source tree structure.  This is required to maintain logical correctness in the result.  The only deviations:
  1. The Progress logical NOT operator has a very low precedence and the Java NOT operator has a high precedence (the same as unary plus and unary minus).  If the real operator is used, the logical not would have to have its operand parenthesized to maintain the same result as in Progress.  If this was not handled, then the Java NOT would bind to the first emitted portion of the operand and this would be evaluated before any subsequent operator.  However, since logical.notOperator() is used instead, the expression subtree will automatically be paranthesized. This is the difference between:
    • !i > 1
    • !(i > 1)
  2. In Progress the equality and inequality operators have the same precedence level as the comparison operators (>, <, >=, <=) but in Java, these are split into 2 precedence levels.
The logic for the operator conversion is encoded in the OperatorConverter class.

Variable References

User Defined Variables

There are 2 forms of variable reference:
  1. A simple reference.
  2. An unwrapped reference.
A simple reference is used when passing a variable to a function or procedure call.  It is also used as the left operand in certain operators and as an lvalue for an assignment.  This is called a simple reference because it is emitted as the variable's object instance name (converted to the proper Java form).

An unwrapped reference is how the variable value is accessed in a Java language contruct.   This is usually a control flow construct like "if" or "switch" where the result of an expression must be unwrapped using booleanValue() or intValue() respectively.  For example, a Progress integer variable named "i" is represented by an integer variable named "i" in Java and when used in expressions requiring direct access to the primative value, it is dynamically unwrapped as "i.intValue()".  This works because Progress has no inline assignment forms inside expressions.  In Java there are many operators that modify and reassign their lvalue (e.g. ++ and -- increment/decrement their lvalue and reassign, the assignment operator itself can be used inside a Java expression and there are many other modify and reassign operators such as += or *=).

Without such assignment operators, expressions essentially access variables in a read-only manner with the exception of user-functions which can have input-output and output parameters.  Such user functions can modify the original variable's value and this is handled using the assign() method of the wrappers.  The method/attribute support has not been investigated fully at this time, but there may be implications for these as well.

Since null is not being used to track the unknown value, all wrappers are always expected to be valid for calling methods and operator replacements.  This eliminates the need to protect variable references using "bracketed" null checks.  This would have otherwise been necessary at the beginning of any expression that made a variable reference and possibly in the middle of expressions to check the return (or the input-output or output) from a function.

Each operator is implemented as methods that are "unknown value aware".   This way the operator will return unknown value or other correct results as needed.   For example, many Progress operators (e.g. + or -) silently return the unknown value if either or both operands are the unknown value.

Global Variables

Progress provides constructs that look like global variables (e.g. opsys, current-language, progress...).  These global variables are termed "built-in functions" by Progress but they act like read-only variables.  In particular, they take no parameters and even take no parenthesis.

A helper class com.goldencode.p2j.util.EnvironmentOps provides static method calls ("getters") to provide the values associated with the global variables.  Note that at this time, the return values are hard coded but eventually they will be backed by lookups from the P2J directory.  With this approach, the values to be passed at runtime can be controlled by the implementer and are not hard coded to a particular customer environment or application.

See also:

EnvironmentOps.java
convert/variable_references.rules

Assignments

All assignments (either "assignment" statements or the ASSIGN language statement) follow these conventions:
  1. Always have an lvalue as the 1st child.
  2. Never can be a declaration/definition of a new variable (this is not possible in Progress).
Since the wrapper approach eliminates the possibility of having a null pointer, all assignments can be made by calling the assign() method of the wrapper.  This is required because the standard Progress semantic is to set the value rather than the Java approach of setting the reference.  This difference can only really be seen in cases where a function or procedure with input parameters is called.  In these cases, the assigning of the value would "bleed through" into the calling procedure.

In a function or procedure where a variable is an input parameter, it can simply/safely be assigned the using the Java assignment operator.

The simple form of the ASSIGN statement is that which just is a list of lvalues, each followed by an assignment operator and an expression of an assignable type.  Such a form is converted by the same rule sets that convert normal ("standalone") assignments.  The more complex forms of the ASSIGN statement which handle the movement of data from input buffers to variables or fields, are not yet supported.

Function/Procedure Definitions

User-defined functions are differentiated from procedures in the following ways:
  1. Functions can (actually MUST) return a scalar instance of any of the basic data types, procedures can only set the return-value "global variable" which is a character value.
  2. Functions are limited in the resources they can define.
  3. Functions can't be referenced before they are defined, though a FORWARD declaration can be made which is just a signature definition.
Both of these constructs are emitted as Java instance methods (METHOD_DEF) for the class that maps to this file.  The function or procedure name is converted to a Java method name.  Any use of the KW_PRIVATE in the Progress source converts to a private access specifier, otherwise the default access is public.  Procedures always return void but functions always have a wrapper data type as a return value.

Both types of definition support input, input-output and output type parameters.  In Java, these parameters are all emitted as a simple variable definitions (of the form: classname reference ) that are comma-separated.  However, the assignment of these parameters is differentiated.  Input parameters have their reference assigned in order to ensure that calling code won't see a change in the parameter after the function/procedure returns.  Input-output and output parameters get their value assigned in order to ensure that the caller sees the change in contained data after return.  All assignments check the type of variable before emitting and use the "parmtype" annotation to make this determination.  The parser ensures that all parameters have this annotation, even if the KW_INPUT is omitted (it is optional in some circumstances).

The concept of an input-output or output function parameter is only implemented for user-defined functions and internal/external procedures.  No built-in Progress functions change the original referenced variable or field passed as a parameter. The only "returned" data from a built-in function appears to be the return value itself.  There may be environmental/database side effects of function calls to built-ins. However, no function parameters are ever modified based on calling a built-in function.

The case of input-output and output parameters is handled using the mutable wrappers where the called function/procedure will modify the contained data such that the caller will see a different value after the call using the same reference.  In this case, the normal assign() method of the wrapper will be called to set the value.  However it is important to note that in the case of an input parameter, the normal Java "=" assignment operator will be used to modify the reference rather than the value (referred to by the reference) since any changes should NOT be visible in the calling code.

Only scalar, table and buffer parameters are allowed.  No array parameters are possible in Progress.  At this time, there is no support for table or buffer parameters.

The user-defined function definition and procedure definition are converted directly to a Java method signature rooted at a METHOD_DEF node which is rooted as a peer with the external procedure's METHOD_DEF subtree.  The text of the METHOD_DEF is the function name (a valid Java function name) and the return type and access specifiers are defined in "rettype" and "access" annotations respectively.  All parameter definitions (if any exist) are rooted at the first child which is an LPARENS.  The next child is a BLOCK that contains the method body.

The creation of these nodes in the target tree is different since the structure of the source tree is different for the two cases.  In particular, the major difference is where the parameter definitions exist.  In the function definition, the parameters are child nodes (PARAMETER) of the FUNCTION and KW_FUNCT (the function definition itself).  In the procedure definition, the parameters are child nodes (DEFINE_PARAMETER) of the BLOCK that is a child to the PROCEDURE but is NOT a child of the KW_PROC which is the actual procedure definition.  Procedure definitions rely upon annotations to note how many parameters there are and then in the conversion step, an LPARENS node is either created (if "numparms" annotation in the PROCEDURE node > 0) or not.  If it is created, its ID is left behind as an annotation called "parmroot" in the PROCEDURE node.  All DEFINE_PARAMETER statements have a "procrefid" annotation that is the ID of the enclosing PROCEDURE node (or the AST root node which is BLOCK and defines the external procedure).  This allows the DEFINE_PARAMETER processing to occur much later than the PROCEDURE definition itself, yet it can reference its way back to the parent node at which the method signature must be rooted.  The order of the DEFINE_PARAMETER statements determines the order of the parameters in the signature (just as in Progress).  The function definition doesn't have this problem as it can emit in-line with the tree walk since the parameters are child nodes of the definition itself.

Input and input-output parameters never get initialized in a procedure or function since they always take the value from the caller.  Output parameters are always initialized.  DEFINE_PARAMETER allows explicit initialization using the KW_INIT, if this construct is found on an output parameter, an in-block (outside the method signature) assignment is made using the provided data.  If no explicit initializer is specified (this is always the case for function parameters since one cannot specifiy KW_INIT on a PARAMETER), then output parameters still get initialized in the block itself.  Procedure parameters get initialized at the same location as the DEFINE_PARAMETER appears in the block.  The default initializer is the same as for DEFINE_VARIABLE.  Function parameters are both more and less troublesome.  All function output parameters are always initialized to the unknown value, rather than following the rules of normal variable defaults.  Since one can never provide an explicit initializer to a function parameter, this means that there is only 1 case.  However, since Java cannot initialize parameters as part of the method signature itself, this initialization must occur in the block that follows the function definition. This is a problem because the PARAMETER nodes are children of the LPARENS and its parent KW_FUNCT but the BLOCK in which they must be attached is the right side sibling of the KW_FUNCT grandparent.  In other words, the BLOCK hasn't been created yet so it cannot be the target for the parameter creation.  To resolve this issue, a BOGUS node is created in place (as a child of the method signature) all parameters are created as children of this node.  They all get the proper IDs and parenting.  Then this node is detached from the tree and stored in a "depot" (see the TemplateWorker ) under the name "funcparms".  After creation of a matching BLOCK node, if the BLOCK is a child of FUNCTION (thus a sibling of KW_FUNCT), then the BOGUS node is retrieved from the depot and attached as the first child of the BLOCK.

See the convert/function_definitions.rules and convert/procedure_definitions.rules for more details.  Please note that the convert/variable_definitions.rules is also heavily involved with processing parameters.

Please also see the section on Control Flow for details on the RUN and RETURN language statements which are necessary to use function and procedures.

Function Calls

The following documents the mapping between built-in functions and their Java equivalents.  References to decimal, integer, logical, date, character, MathOps, and CompareOps refer to classes in package com.goldencode.p2j.util.

Progress Function
 Java Equivalent
 Category
 Token Type
 Nullable
 Optional Parameters
 DBCS Issues
 Supported
 Notes
_CBIT
 character.testBitAt()
 bit manipulation
 FUNC_LOGICAL
 yes
 no
 yes
 yes
 logical _cbit(string, integer)

The first parameter is a string with 0 or more characters. 

The second parameter is the bit position to test.  The input string is treated as a multibyte bitfield. The first character of the string corresponds to bit positions 0-7 (if it is non-DBCS and presumably in a DBCS character the positions would be 0-15).  In a non-DBCS charset, the second char would correspond to bits 8-15.

The return is always false if the empty string is input.  For any string of > 0 length, the return is true if the bit in the specified position is 1 and false if it is 0.

Sample Code:

def var i as int.
def var j as int.
def var b as char.
def var c as char.

do i = -1 to 257.
   b = "".
   c = chr(i).
   do j = 7 to 0 by -1.
      if _cbit(c, j)
         then b = b + "1".
         else b = b + "0".
   end.
   message i " = " b.
end.
ABSOLUTE
 MathOps.abs(integer)
or
MathOps.abs(decimal)
 math
 FUNC_POLY yes
 no
 no
 yes
 returns INT or DEC depending on the type of the input
ACCUM
 Accumulator (abstract base class);  concrete implementations:
  • AverageAccumulator
  • CountAccumulator
  • MaximumAccumulator
  • MinimumAccumulator
  • TotalAccumulator
 database
 FUNC_POLY no

yes
 Resides in the com.goldencode.p2j.util package.

Each concrete subclass defines the following methods for results retrieval;  each class' implementations of these methods may return different data types, so these method signatures are named by convention only, and are not enforced by a common interface:
  • getResult() to retrieve the cumulative result
  • getResult(Resolvable breakGroupKey) to retrieve the current break group result
ADD-INTERVAL DateOps.addInterval() date/time FUNC_POLY yes no no yes
ALIAS
 ConnectionManager.alias() database FUNC_CHAR
 yes

yes
AMBIGUOUS RecordBuffer.wasAmbiguous() database
 FUNC_LOGICAL no

yes
ASC
 character.asc()
 type conversion
 FUNC_INT yes
 yes
 yes
 yes
 No source or target codepage support at this time.  Result in a DBCS environment may vary from the Progress implementation.
AVAILABLE
 RecordBuffer.available()
 database
 FUNC_LOGICAL no

yes
CAN-DO
 character.matchList()
 security
 FUNC_LOGICAL yes

yes
CAN-FIND RandomAccessQuery.hasFirst()
RandomAccessQuery.hasNext()
RandomAccessQuery.hasPrevious()
RandomAccessQuery.hasLast()
RandomAccessQuery.hasAny()
 database
 FUNC_LOGICAL no

yes
CAN-QUERY
UI
 FUNC_LOGICAL yes

no
CAN-SET
UI
 FUNC_LOGICAL yes

no
CAPS
 character.toUpperCase()
 string
 FUNC_CHAR
 yes
 no
 partial yes
 It can contain DBCS but only the SBCS chars are uppercased.
CHR
 character.chr()
 type conversion
 FUNC_CHAR
 yes
 yes
 yes
 yes
 No source or target codepage support at this time.  Result in a DBCS environment may vary from the Progress implementation.
CODEPAGE-CONVERT
type conversion
 FUNC_CHAR
 yes
 yes
 yes
 no
COMPARE
 character.compare()
 string
 FUNC_LOGICAL
 yes
 yes
 yes
 yes
 No collation table support at this time.  Result in a DBCS environment may vary from the Progress implementation.
CONNECTED ConnectionManager.connected() database FUNC_LOGICAL no no
yes
COUNT-OF
 P2JQuery.size()
 database
 FUNC_INT
 no
 no
yes
 Implemented by concrete query classes.
CURRENT-CHANGED
database
 FUNC_LOGICAL
 no
 no
no
CURRENT-LANGUAGE
 EnvironmentOps.getCurrentLanguage() I18N
 VAR_CHAR
 no
 no
yes
CURRENT-RESULT-ROW
 AbstractQuery.currentRow()
 database
 FUNC_INT
 yes
 no
yes
 Implemented by AbstractQuery and overridden where necessary by concrete query implementations.
CURRENT-VALUE
database
 FUNC_INT
 yes
 yes
no
DATASERVERS EnvironmentOps.getDataServerList() database
 VAR_CHAR
 no
 no
early
DATE
 see constructors for date class
 date
 FUNC_DATE
 yes
 no
 no
 yes
DATETIME see constructors for datetime class date/time FUNC_DATETIME yes no no yes
DATETIME-TZ see constructors for datetimetz class date/time FUNC_DATETIME_TZ yes no no yes
DAY date.day() which calls instance method date.getDayNum()
 date
 FUNC_INT
 yes
 no
 no
 yes
DBCODEPAGE
I18N
 FUNC_CHAR
 yes
 yes
no
DBCOLLATION
I18N
 FUNC_CHAR
 yes
 yes
no
DBNAME
 EnvironmentOps.getCurrentDatabaseName() database
 VAR_CHAR
 yes
 no
yes
DBPARAM
database
 FUNC_CHAR yes
 yes
no
DBRESTRICTIONS ConnectionManager.dbRestrictions() database
 FUNC_CHAR yes
 yes
yes
DBTASKID
database
 FUNC_INT
 yes
 yes
no
DBTYPE ConnectionManager.dbType() database
 FUNC_CHAR
 yes
 yes
yes
DBVERSION
database
 FUNC_CHAR
 yes
 yes
no
DECIMAL
 see constructors for decimal class
 type conversion
 FUNC_DEC
 yes
 no
 no
 yes
DYNAMIC-FUNCTION
function execution
 FUNC_POLY yes
 yes
no
ENCODE
 SecurityOps.encode()
 security
 FUNC_CHAR
 yes
 no
early
 The interface is completely supported and functional, however the calculated result is NOT compatible with the 4GL version.
ENTERED GenericWidget.isEntered()
 UI
 FUNC_LOGICAL
 no
 no
yes
ENTRY
 character.entry()
 string
 FUNC_CHAR
 yes
 yes
 yes
 yes
 Delimiter processing is *always* case-sensitive at this time.
ETIME date.elapsed()
 time
 FUNC_INT
 no
 yes
yes
EXP
 MathOps.pow()
 math
 FUNC_DEC
 yes
 no
yes
EXTENT 0 if the variable or field is scalar
OR
array_var.length
 arrays
 FUNC_INT
 no
 no
yes
 This is a direct conversion without any backing method or class.
FILL
 character.fill() string
 FUNC_CHAR
 yes
 no
yes
FIRST PresortQuery.isFirst() loops/transactions FUNC_LOGICAL
 no
 no
yes
FIRST-OF PresortQuery.isFirstOfGroup() loops/transactions FUNC_LOGICAL no
 no
yes
FRAME-COL CommonFrame.frameCol()
 UI
 FUNC_DEC
 no
 yes
yes
FRAME-DB
UI
 VAR_CHAR
 no
 no
no
FRAME-DOWN CommonFrame.frameDown() UI
 FUNC_INT
 no
 yes
yes
FRAME-FIELD
 LogicalTerminal.getFrameField() UI
 VAR_CHAR no
 no
yes
FRAME-FILE
UI
 VAR_CHAR no
 no
no
FRAME-INDEX LogicalTerminal.getFrameIndex() UI
 VAR_INT
 no
 no
yes
FRAME-LINE CommonFrame.frameLine() UI
 FUNC_INT no
 yes
yes
FRAME-NAME
UI
 VAR_CHAR
 no
 no


FRAME-ROW
 CommonFrame.frameRow() UI
 FUNC_DEC
 no
 yes
yes
FRAME-VALUE LogicalTerminal.getFrameValue()
LogicalTerminal.setFrameValue()
 UI
 VAR_CHAR no
 no
yes
GATEWAYS
 EnvironmentOps.getDataServerList()
 database
 FUNC_CHAR
 no
 no
early
GET-BITS
bit manipulation FUNC_INT
 yes
 no
no
GET-BYTE
 BinaryData.getByte()
 raw/memory access
 FUNC_INT
 yes
 no
yes
GET-BYTE-ORDER
raw/memory access FUNC_INT
 no
 no
no
GET-BYTES
raw/memory access FUNC_POLY
 yes
 no
no
 Returns RAW or MEMPTR.
GET-CODEPAGES
I18N
 VAR_CHAR
 no
 no
no
GET-COLLATIONS
I18N
 FUNC_CHAR
 yes
 no
no
GET-DOUBLE
raw/memory access FUNC_DEC
 yes
 no
no
GET-FLOAT
raw/memory access FUNC_DEC
 yes
 no
no
GET-LONG
raw/memory access FUNC_INT
 yes
 no
no
GET-POINTER-VALUE
raw/memory access FUNC_INT no
 no
no
GET-SHORT
raw/memory access FUNC_INT
 yes
 no
no
GET-SIZE
 memptr.length() raw/memory access FUNC_INT
 no
 no
yes
GET-STRING
 BinaryData.getString() raw/memory access FUNC_CHAR
 yes
 yes
yes
GET-UNSIGNED-SHORT
raw/memory access FUNC_INT
 yes
 no
no
GO-PENDING LogicalTerminal.isGoPending() UI
 VAR_LOGICAL
 no
 no
yes
IF THEN ELSE
 condition ? expr1 : expr2
 ternary
 FUNC_POLY
 yes
 no
yes
 Returns same type as the THEN and ELSE expressions evaluate to.
INDEX
 character.indexOf()
 string
 FUNC_INT
 yes
 yes
yes
INPUT
 Converts to a field-level getter in a custom frame interface OR to the CommonFrame.getScreenValue(widget) method if this is a character type override case (see notes).
 UI
 FUNC_POLY
 no
 yes
yes
 The INPUT built-in function normally returns the same type as the single parameter (which must be an lvalue which is found in a frame that is in scope).  However, this function can be used in an expression that requires a character data type even in the case where the parameter is NOT of the character type!  This is like an override option and it must be detected from the SURROUNDING expression (above the FUNC_POLY node). For example,an direct assignment of the INPUT function's return type to a character lvalue is honored.  This case works differently than an assignment to a non-character type in the case that the value in the screen buffer is uninitialized (it will return the empty string like screen-value instead of returning the default value of the operand type).  This override support is implemented except for one case: the usage of INPUT as a parameter to another built-in function (or presumably a method) which expects a character parameter should also cause this override. At this time there is no support for this case. The resulting type (if not already character) will cause the wrong type to be emitted.  Note that this only is an issue for built-in functions (and methods).  If the parent is a user defined FUNC_* then the result is the same as the operand, no override occurs.  But a built-in function that takes a character type will get the result as a character instead of the operand type.

See the SCREEN-VALUE attribute which is a related issue.
INTEGER
 see constructors for integer class
 type conversion
 FUNC_INT yes
 no
yes
INTERVAL DateOps.interval() date/time FUNC_INT64 yes no no yes
IS-ATTR-SPACE
UI
 FUNC_LOGICAL yes
 no
no
IS-LEAD-BYTE
I18N
 FUNC_LOGICAL yes
 no
no
KBLABEL
 LogicalTerminal.kbLabel() UI
 FUNC_CHAR
 yes
 no
yes
KEYCODE LogicalTerminal.keyCode() UI
 FUNC_INT
 yes
 no
yes
KEYFUNCTION LogicalTerminal.keyFunction() UI
 FUNC_CHAR
 yes
 no
yes
KEYLABEL LogicalTerminal.keyLabel()
 UI
 FUNC_CHAR
 yes
 no
yes
KEYWORD
Progress
 FUNC_CHAR yes
 no
no
KEYWORD-ALL
Progress FUNC_CHAR yes
 no
no
LAST
 PresortQuery.isLast() loops/transactions FUNC_LOGICAL no
 no
yes
LAST-OF
 PresortQuery.isLastOfGroup() loops/transactions FUNC_LOGICAL no
 no
yes
LASTKEY
 KeyReader.lastKey()
 UI
 VAR_INT
 no
 no
yes
LC
 character.toLowerCase()
 string
 FUNC_CHAR
 yes
 no
 partial
 yes
 It can contain DBCS but only the SBCS chars are lowercased.
LDBNAME
 ConnectionManager.ldbName() database
 FUNC_CHAR yes
 yes
yes
LEFT-TRIM
 character.leftTrim() string
 FUNC_CHAR
 yes
 yes
 yes
 yes
LENGTH
 character.length()
character.byteLength()
BinaryData.length()
 raw/memory access
 FUNC_INT
 yes
 yes
yes
 No support for "column" based length at this time.
LIBRARY
R-code library FUNC_CHAR
 yes
 no
no
LINE-COUNTER
 Stream.getNextLineNum()
 I/O
 FUNC_INT
 no
 yes
yes
LIST-EVENTS
UI
 FUNC_CHAR yes
 yes
no
LIST-QUERY-ATTRS
UI
 FUNC_CHAR yes
 no
no
LIST-SET-ATTRS
UI
 FUNC_CHAR yes
 no
no
LIST-WIDGETS
UI
 FUNC_CHAR yes
 yes
no
LOCKED
 RecordBuffer.wasLocked() database
 FUNC_LOGICAL
 no
 no
yes
LOG
 MathOps.log()
 math
 FUNC_DEC
 yes
 yes
yes
LOOKUP character.lookup()
 string
 FUNC_INT
 yes
 yes
 yes
 yes
 Delimiter processing is *always* case-sensitive at this time.
MAXIMUM
 character.maximum()
date.maximum()
logical.maximum()
integer.maximum()
decimal.maximum()		 math
 FUNC_POLY
 yes
 yes
yes
 Warning: variable args!  Numeric operands are widened (int to double) if there are a mixture.  All arguments (other than numerics) can only be compared against others of the same type.
MEMBER
R-code library
 FUNC_CHAR
 yes
 no


MESSAGE-LINES
 LogicalTerminal.getMessageLines() UI
 VAR_INT
 no
 no
yes
MINIMUM
 character.minimum()
date.minimum()
logical.minimum()
integer.minimum()
decimal.minimum()		 math
 FUNC_POLY yes
 yes
yes
 Warning: variable args!  Numeric operands are widened (int to double) if there are a mixture.  All arguments (other than numerics) can only be compared against others of the same type.
MONTH date.month() which calls instance method date.getMonthNum()
 date
 FUNC_INT
 yes
 no
yes
MTIME datetime.millisecondsSinceMidnight() date/time FUNC_INT yes yes yes
NEW
 RecordBuffer.isNew() database
 FUNC_LOGICAL
 no
 no
yes
NEXT-VALUE
database
 FUNC_INT
 yes
 yes
no
NOT ENTERED
 GenericWidget.isNotEntered()
 UI
 FUNC_LOGICAL no
 yes
yes
NOW datetime.now()
datetimetz.now()		 date/time FUNC_DATETIME
FUNC_DATETIME_TZ		 yes yes yes
NUM-ALIASES
 ConnectionManager.numAliases()
 database VAR_INT
 no
 no
yes
NUM-DBS
 ConnectionManager.numDBs() database VAR_INT
 no
 no
yes
NUM-ENTRIES
 character.numEntries()
character.numEntriesOf()
 string
 FUNC_INT
 yes
 yes
 yes
 yes
 Delimiter processing is *always* case-sensitive at this time.
NUM-RESULTS P2JQuery.size()
 database FUNC_INT
 yes
 no
yes
 Implemented by concrete query classes.
OPSYS
 EnvironmentOps.getOperatingSystem() OS environment VAR_CHAR
 no
 no
yes
OS-DRIVES
 FileSystemOps.getRootList()
 OS environment VAR_CHAR
 no
 no
yes
OS-ERROR
 FileSystemOps.getLastError() OS environment VAR_INT
 no
 no
yes
OS-GETENV FileSystemOps.getProperty() OS environment
 FUNC_CHAR
 yes
 no
yes
PAGE-NUMBER Stream.getPageNum()
 I/O
 FUNC_INT no
 yes
yes
PAGE-SIZE Stream.getPageSize()
 I/O
 FUNC_INT
 no
 yes
yes
PDBNAME ConnectionManager.pdbName() database
 FUNC_CHAR
 yes
 yes
yes
PROC-HANDLE
Progress environment VAR_INT
 no
 no
no
 Is this actually polymorphic?
PROC-STATUS
Progress environment VAR_INT
 no
 no
no
PROGRAM-NAME EnvironmentOps.getSourceName() + a constant (public static final String progressSourceName) in each generated class.
 Progress environment FUNC_CHAR
 yes
 no
yes
 Warning: there are 2 flaws in this implementation.
  1. Progress returns the same filename as is used on the RUN statement (so the name can change from invocation to invocation).  Our version has a static string for the base name of the file (no pathing).
  2. It is possible that due to refactoring, code that calls program-name might have dependencies on a source file name that is not the source file name of the current class. Such cases might require a class and method specific mapping to a source file name (some kind of registry database).
PROGRESS
 EnvironmentOps.getRuntimeType() Progress environment VAR_CHAR
 no
 no
yes
PROMSGS EnvironmentOps.getMessageSource() Progress environment VAR_CHAR no
 no
yes
PROPATH EnvironmentOps.getSourcePath() Progress environment VAR_CHAR no
 no
yes
PROVERSION EnvironmentOps.getVersion()
 Progress environment VAR_CHAR no
 no
yes
QUERY-OFF-END
 P2JQuery.isOffEnd() database
 FUNC_LOGICAL
 yes
 no
yes
 Implemented by concrete query classes.
R-INDEX
 character.lastIndexOf()
 string
 FUNC_INT
 yes
 yes
yes
RANDOM
 MathOps.random()
 math
 FUNC_INT
 yes
 no
yes
 Warning: it is possible to force Progress to always generate the same sequence of random numbers in every session! (This sounds like an awful idea but perhaps it is being used.)
RAW
database
 FUNC_RAW
 no
 yes
no
RECID
 RecordBuffer.recordID database FUNC_RECID
 yes
 no
yes
 The type is mapped to integer.
RECORD-LENGTH
database
 FUNC_INT
 no
 no
no
REPLACE
 character.replaceAll()
 string
 FUNC_CHAR
 yes
 no
yes
RETRY
 TransactionManager.isRetry()
 loops/transactions
 VAR_LOGICAL
 no
 no
yes
RETURN-VALUE
 ControlFlowOps.getReturnValue()
 procedure execution
 VAR_CHAR
 no
 no
yes
RGB-VALUE
UI
 FUNC_INT
 yes
 no
no
RIGHT-TRIM
 character.rightTrim()
 string
 FUNC_CHAR
 yes
 yes
yes
ROUND
 Math.round()
 math
 FUNC_DEC
 yes
 no
yes
ROWID
 RecordBuffer.recordID database FUNC_ROWID
 yes
 no
yes
 The type is mapped to integer.
SCREEN-LINES
 LogicalTerminal.getScreenLines()
 UI
 VAR_INT
 no
 no
yes
SDBNAME
 ConnectionManager.sdbName()
 database FUNC_CHAR
 yes
 yes
yes
SEARCH
 FileSystemOps.searchPath()
 filesystem
 FUNC_CHAR yes
 no
yes
SEEK
 Stream.getPosition()
 filesystem
 FUNC_INT
 yes
 yes
yes
SETUSERID
security
 FUNC_LOGICAL
 yes
 yes
no
SQRT
 MathOps.sqrt()
 math
 FUNC_DEC
 yes
 no
yes
STRING
 character.valueOf() which uses a  wrapper-specific toString()
 type conversion
 FUNC_CHAR
 yes
 yes
yes
SUBSTITUTE
 character.substitute()
 string
 FUNC_CHAR yes
 yes
 yes
 yes
 The variable length argument list is handled by an argument of type Object[].
SUBSTRING
 character.substring()
 string
 FUNC_CHAR
 yes
 yes
 yes
 yes
 No support for "raw", "fixed" or "column" based substrings at this time.
SUPER
function execution
 FUNC_POLY
 yes
 yes
no
TERMINAL
 LogicalTerminal.getTerminal()
 UI
 VAR_CHAR
 no
 no
yes
TIME
 date.secondsSinceMidnight()
 time
 VAR_INT
 no
 no
yes
TIMEZONE date.getDefaultTimeZoneOffset()
date.getOffsetForSpec()
datetimetz.getTimeZoneOffset()		 date/time FUNC_INT yes yes yes
TODAY
 date.today()
 date
 VAR_DATE
 no
 no
yes
TO-ROWID
database FUNC_ROWID
 yes
 no
no
TRANSACTION
 TransactionManager.isTransactionActive() loops/transactions
 VAR_LOGICAL
 no
 no
yes
TRIM
 character.trim()
 string
 FUNC_CHAR
 yes
 yes
yes
TRUNCATE
 MathOps.truncate() math
 FUNC_DEC
 yes
 no
yes
USERID
 SecurityOps.getUserId() security
 FUNC_CHAR
 no
 yes
yes
VALID-EVENT
UI
 FUNC_LOGICAL
 yes
 yes
no
VALID-HANDLE
 handle.isValid()
 data types
 FUNC_LOGICAL no
 no
yes
WEEKDAY
 date.weekday() which calls instance method date.getWeekDayNum()
 date
 FUNC_INT
 yes
 no
yes
WIDGET-HANDLE
 handle.fromString()
 UI
 FUNC_HANDLE
 yes
 no
yes
YEAR date.year() which calls instance method date.getYearNum() date
 FUNC_INT
 yes
 no
yes

See also:

rules/convert/builtin_functions.rules

Methods and Attributes

Progress 4GL provides an object-like construct with its HANDLE data type.  WIDGETs, QUERYs and other resources can be represented by a handle.  Using this handle one can access pre-defined attributes or call methods (invoke behavior).  These attributes and methods are specific to this resource type, they are built into the environment and cannot be extended by the user/programmer.

There are 3 possible forms of conversion:
  1. Special purpose accessors (e.g. SYSTEM HANDLES such as CURRENT-WINDOW) can be naturally converted to a static method call which accesses an object instance that is the current value.  The ATTRIBUTE or METHOD node itself can be directly translated to an instance method call made upon the returned object reference.
  2. Any singleton resource (e.g. SYSTEM HANDLES such as ERROR-STATUS) can be easily and naturally converted to a static method call.  The ATTRIBUTE or METHOD node itself can be directly translated to the static method call.  Note that for attributes, the getter methods will need to be backed by a context-local member.
  3. Instance resources (e.g. a WIDGET handle) act like instance members (accessed via a getter method call) or methods.  In this case the COLON patent node should be the peer node at which the method is converted.  Then the HANDLE will need to emit as an object REFERENCE of the proper class.
The following summarizes the method and attribute support that is implemented:

Progress Feature
 Java Replacement
 Type
 Method or Attribute
 Supported
 Notes
DCOLOR
 GenericWidget.getDcolor()
GenericWidget.setDcolor()
 widget
 ATTRIBUTE Yes
DESELECT-ROWS
 GenericWidget.deselectRows()
 browse
 METHOD Yes
ENTRY
 GenericWidget.entry() combo-box, selection list
 METHOD
 Yes
ERROR
 ErrorManager.isError()
 ERROR-STATUS system handle
 ATTRIBUTE Yes
FETCH-SELECTED-ROW
 GenericWidget.fetchSelectedRow() browse
 METHOD Yes
FILE-NAME FileSystemOps.initFileInfo() to set
FileSystemOps.fileInfoGetName()  to get		 FILE-INFO system handle ATTRIBUTE Yes
FILE-SIZE FileSystemOps.fileInfoGetSize()  to get FILE-INFO system handle ATTRIBUTE Yes
GET-NUMBER
 Manager.getErrorNum() ERROR-STATUS system handle METHOD
 Yes
GET-MESSAGE
 ErrorManager.getErrorText() ERROR-STATUS system handle METHOD Yes
HANDLE
 GenericWidget.getHandle()
 widget
 ATTRIBUTE Yes Should this really be converted into a widget object reference?
IS-SELECTED
 GenericWidget.isSelected() combo-box, selection list METHOD Yes
LIST-ITEM-PAIRS
 GenericWidget.getListItems()
GenericWidget.setListItems()		 combo-box, selection list ATTRIBUTE Yes
LOOKUP
 GenericWidget.lookup() combo-box, selection list METHOD Yes
NEXT-TAB-ITEM
 GenericWidget.getNextTabItem()
GenericWidget.setNextTabItem()		 widget
 ATTRIBUTE Yes
NUM-MESSAGES
 ErrorManager.numErrors() ERROR-STATUS system handle ATTRIBUTE
 Yes
NUM-SELECTED-ROWS
 GenericWidget.getNumSelectedRows()
GenericWidget.setNumSelectedRows()		 browse ATTRIBUTE Yes
PARENT
 GenericWidget.getParent()
GenericWidget.setParent()		 frame
 ATTRIBUTE Yes
PERSISTENT
 false
 THIS-PROCEDURE system handle
 ATTRIBUTE
 Yes
 Temporary solution to satisfy the current application project for which we always know the result will be false.  A real implementation will eventually need to be made.
PFCOLOR
 GenericWidget.getPfColor()
GenericWidget.setPfColor()		 widget
 ATTRIBUTE Yes
PRIVATE-DATA
 GenericWidget.getPrivateData()
GenericWidget.setPrivateData()		 frame
 ATTRIBUTE Yes
READ-ONLY
 GenericWidget.isReadOnly()
GenericWidget.setReadOnly()		 widget
 ATTRIBUTE
 Yes
REFRESH
 GenericWidget.refresh()
 browse
 METHOD Yes
SCROLLABLE
 GenericWidget.isScrollable()
GenericWidget.setScrollable()		 widget
 ATTRIBUTE Yes
SCREEN-VALUE
 CommonFrame.getScreenValue(widget) or widget setter method from the current frame interface.
 widget
 ATTRIBUTE
 Yes
 This duality is needed since the screen-value attribute will always return a character type (the frame interface getters have the same type as the data being represented/edited).  In the case where the "natural" data type is not character, this will behave differently when the screen buffer's version of this data is uninitialized.  In particular, the data will be returned as the empty string rather than as the default type for the natural data type.  So if widget i is an integer and it has never been initialized (copied into the screen buffer as a result of a display a screen-value setter call or via UI editing) then it will return "" instead of 0.

See the INPUT built-in function which has a similar issue.
SELECT-ALL
 GenericWidget.selectAll()
 browse
 METHOD Yes
SELECT-FOCUSED-ROW
 GenericWidget.selectFocusedRow()
 browse
 METHOD Yes
SELECTED
 GenericWidget.isSelected()
GenericWidget.setSelected()		 widget
 ATTRIBUTE Yes
SENSITIVE
 GenericWidget.isSensitive()
GenericWidget.setSensitive()		 widget
 ATTRIBUTE Yes
SET-REPOSITIONED-ROW
 GenericWidget.setRepositionedRow()
 browse
 METHOD Yes
TITLE
 GenericWidget.getTitle()
GenericWidget.setTitle()		 frame
 ATTRIBUTE Yes
VISIBLE
 GenericWidget.isVisible()
GenericWidget.setVisible()		 widget
 ATTRIBUTE Yes

Instance oriented system handles:

Handle
 Java Equivalent
CURRENT-WINDOW
 LogicalTerminal.currentWindow()
ACTIVE-WINDOW
 LogicalTerminal.activeWindow()
THIS-PROCEDURE
 none at this time
FOCUS
 LogicalTerminal.focus()
SESSION
 none at this time
SELF LogicalTerminal.self()


See also:

rules/convert/methods_attributes.rules
rules/convert/assignments.rules (setters are implemented here)
rules/convert/variable_references.rules (for handle and system handle conversion)

Control Flow

Some implementation notes:
  1. IF/THEN/ELSE is completely supported as Java if/else.
  2. The IF function converts to the Java ternary operator.
  3. CASE statements with integral data types properly convert to the Java switch when using NUM_LITERALs for all WHEN clauses.
  4. CASE statements that have non-constant integrals (expressions rather than literals) and/or non-integer (character, date, decimal, logical) expressions properly convert to an if/else if/else if/else form.
  5. A LEAVE in a CASE statement breaks out of the closest enclosing non-DO block.  This is handled by using an existing label or manufacturing a new label (if one doesn't already exist) to identify the correct block.  Such changes are handled in the annotations processing, and the manufactured label and new label reference should be written into the source AST.  This means that an implicit behavior is made explicit and no changes to conversion code are needed.
  6. Simple DO blocks convert to the Java { }.
  7. Simple REPEAT blocks convert to the Java while (true) { }.
  8. DO/REPEAT/FOR with a WHILE expression converts to a Java while (expression) { }.
  9. DO/REPEAT/FOR with a var = expr1 TO expr2 converts to a Java for (var = expr1 ; var comparison expr2 ; var += increment) { }.
  10. Labels are supported (they are attached to the beginning of a block, exactly as in Java).
  11. LEAVE
    • An unlabeled LEAVE outside of an iterating block acts as a RETURN statement.
    • Within an iterating block (even if inside a nested non-looping block), the LEAVE and LEAVE label are translated directly to break or break label respectively.
  12. NEXT and NEXT label are translated directly to continue or continue label, except in the case where it is used in a non-iterating block in which case it is translated into a continue on the nearest enclosing iterating block.  If there is no such enclosing iterating block, it is treated as a LEAVE.
  13. STOP is implemented as a "throw new StopConditionException()".
  14. QUIT is implemented as "throw new QuitConditionException()".
  15. PAUSE is implemented as ControlFlow.pause(...).
  16. RETURN when used in a function, returns the result of the associated expression.
  17. RETURN when used in a procedure, if it has an expression, saves the evaluated result into a context-local runtime variable via the ControlFlowOps.setReturnValue() before emitting a Java return statement.  If there is no expression, then the setReturnValue() is assigned the empty string.  Whenever Progress code references RETURN-VALUE, this results in a call to ControlFlowOps.getReturnValue() which returns the current context-local value of this character variable.
  18. RETURN ERROR is implemented as a special ignore flag in the TransactionManager and "throw new ErrorConditionException()" where this exception is ignored until it reaches the caller of the top level block in the current method.
  19. RETURN NO-APPLY is implemented by calling LogicalTerminal.consumeEvent() before returning.
  20. RUN for internal procedures is implemented as a method call to the referenced instance member in the same class.  Any specified parameters are emitted as method arguments.
  21. RUN for external procedures is implemented as a method call to the execute() method of the Java class name generated from the file portion of the pathname text in the FILENAME child node.  First the class name and a unique Java variable name are generated.  Then an instance of the class is defined and initialized (using a default constructor).  Finally, the method call to the execute() method is emitted in such a manner as to ensure that any parameters are emitted as children of the method call node.
  22. RUN VALUE() is a variation on the procedure execution.   Both external and internal procedures can be called via this mechanism. The difference is that the procedure name is built dynamically (at runtime).
    • The main problem with these statements is that resulting value of the variable passed to RUN value() statement contains the Progress-specific name of the internal or external procedure. In other words, runtime code must provide a way to translate Progress procedure names into Java class and methods names.
    • The following approach is used:
      • The value of the passed variable is translated into a Java class name using data in the directory which was generated at conversion time.
      • If such a Java class name is present then Java class is instantiated and execute() method is called.
      • If no such Java class name is present, then an internal procedure name is assumed and the Java method name is looked up using the directory information.
      • The instance method is called by ControlFlowOps.invoke() method via reflection on the instance associated with the caller as found from the current stack.
    • At this time, the implementation of the name mapping uses an XML data file which contains mapping between Progress file names/internal procedure names and Java class/method names. This information is collected during conversion and used by run-time code. The final implementation will use the directory.
See also:

rules/annotations/annotations.xml
rules/annotations/block_properties.rules
rules/convert/control_flow.rules

Common Language Statements

Assignment Type Language Statements

The following language statements are handled with special processing:
  1. length (BinaryData.length)
  2. overlay (character.overlay)
  3. put-byte (BinaryData.setByte)
  4. put-string (BinaryData.setString)
  5. set-size (memptr.length)
  6. substring (character.replace)
All of these are "assignment type" language statements.  That is, language statements that are structured like an assignment of an expression result to a function call.  This required that the annotations processing rewrite the source tree to look more like a proper method call with the first parameter being the object reference.  For example, the assigned expression is inserted as the 2nd child of the KW_SUBSTR or KW_OVERLAY and the ASSIGN node is removed.

Once rewritten, the convert/language_statements.rules rule set handles a simple mapping of overlay to the proper instance method mapping.The result is designed as an instance method because the result must be assigned back to the value of the variable specified as the 1st child. This matches the method call semantic exactly.

Process Execution

External command execution is implemented in com.goldencode.p2j.util.ProcessOps.java.  The signature used is as follows:

void launch(String[] cmdlist, boolean silent, boolean wait)

This handles launching a child process in response to these language statements:
The COMMAND_TOKENS child is converted into an anonymous string array initializer and all children are converted into string literals or a string expression (the value() construct).  In this manner, the first parameter of the method call launch() simultaneously specifies the program to be executed and provides all command line arguments.

The 2nd and 3rd parameters are two flags which are generated based on optional (and mutually exclusive) keywords KW_SILENT and KW_NO_WAIT.  For this reason, in Progress there are 3 possible modes which can be encountered:

Options
 silent flag
 wait flag
none (the default)
 false
 true
KW_SILENT
 true
 true
KW_NO_WAIT
 false
 false

The "silent and no wait" mode doesn't exist in Progress.  All of these modes are properly supported using these flags.  In addition, the FileSystemOps.lastError (OS-ERROR) value is forced to 0 which is how Progress handles this value.

In Progress, one can launch an interactive child process with the NO-WAIT option.  However, this seems to invariably disable the terminal and neither the child process nor the Progress session can properly run.  For this reason, this mode is not supported nor is it expected to be in use in the field.

Command Text Parsing

The lexer and parser have been modified to change how the COMMAND_TOKENS children are created.  In particular, the lexer may deliver tokens broken into a form that does not match the whitespace delimited words that the author might have intended.  For example: the "word" (used in the shell to redirect STDERR to the same handle as STDOUT):

&2>1

would be lexed as 4 tokens:

UNKNOWN_TOKEN
NUM_LITERAL
GT
NUM_LITERAL

The lexer has been changed to return "hidden" tokens that represent whitespace.  During COMMAND_TOKENS processing, the parser checks these hidden tokens and marks the COMMAND_TOKENS children if they need to be merged back together or not.  Annotation processing then handles this merge as needed.  The result is the same set of commands as executed in Progress 4GL.

See Also

Please see the following rulesets for the implementation:

convert/process_launch.rules
convert/expressions.rules (this handles the VALUE() construct)
convert/input_output.rules

These classes back the runtime functions:

com.goldencode.p2j.util.Launcher.java (remote interface definition)
com.goldencode.p2j.util.ProcessOps.java (server side)
com.goldencode.p2j.util.ProcessDaemon.java (real process launcher)
com.goldencode.p2j.util.ProcessStream.java

The majority of the processing of INPUT_THRU, OUTPUT_THRU and INPUT_OUTPUT_THRU is implemented in Streams and I/O however, the process launching and stream connection is handled by the classes and rulesets noted above.

Limitations
  1. Remote I/O and Terminal Integration
  2. The option KW_NO_CONS (no-console) is not supported yet (it may be Windows-only).
  3. For some or all of these language statements, the process launched is really a shell which then executes the command line provided.  At this time, the shell is hard coded to "sh" but needs to be moved into the directory or provided as a conversion time hint that is passed to a modified version of launch.

OS-Independent Commands (File System)

The OS independent commands for manipulating the file system are implemented in com.goldencode.p2j.util.FileSystemOps.java.

The following language statements are supported:
Please see the following rulesets for the implementation:

convert/language_statements.rules
convert/literals.rules (handles FILENAME)
convert/expressions.rules (this handles the VALUE() construct)

Streams and I/O

The P2J project provides significant support for the Progress I/O processing and streams features. 

Summary

The following language statements which allow the creation, opening and closing of streams are supported:
The following language statements and functions which provide stream reading/writing (including formatted reads/writes) are supported:
Stream Types

Stream Type
 Java Class
 Input
 Output
 Notes
File or Device
 com.goldencode.p2j.util.FileStream
 Y
 Y
Printer
 com.goldencode.p2j.util.ChUIPrinterStreamSupport N
 Y
 On Unix, this will probably be implemented as a pipe to a child process of "lp".  As this type may only be needed on Windows, it isn't implemented at this time.
Directory
 com.goldencode.p2j.util.DirectoryStream
(not yet implemented)
 Y
 N
 Used for reading directory listings. Not created at this time since it is not used in the application.
Terminal
 ?
 Y
 Y
 How this is to be implemented is TBD.
Clipboard
 ?
 ?
 Y
 Windows-only, not implemented at this time.  Note that via the CLIPBOARD system handle, reading the clipboard may be possible, though this has not been investigated.
Child Process
 com.goldencode.p2j.util.PipeStream Y
 Y
 In Progress the same child process stream can be created as both input and output (simultaneously), however the Java class will only handle either input OR output.  Thus the converted code will have 2 streams created in this case and all operations will need to be properly destined for the correct stream.


Basic Design

A DEFINE STREAM corresponds to a local variable of type com.goldencode.p2j.util.Stream.  This class implements an API that provides the complete Progress semantics for all forms of supported I/O processing.  This is an abstract class and concrete subclasses actually provide the real implementation for each type of stream.  Note that the design is such that the vast majority of code is implemented in Stream and the subclasses only have to provide the minimum function to implement the specific resource being used.  This lets the same behavior be obtained from all types of streams and makes it easy to add more stream types with a small incremental effort.

Generally a stream is implemented as either an input OR an output but not as both.  The sole exception to this is the case of using a named stream in the INPUT-OUTPUT THROUGH, where the same stream name is used for both the input and output pipes (the complexity of this is managed inside com.goldencode.p2j.util.ProcessStream and in the rule-sets that convert the code..

There are 2 unnamed streams ("unnamed input" and "unnamed output") which are implicitly used when a named stream is not explicitly present in the source file.  These streams are stored as context local instances that are accessible throughout a user's session via the UnnamedStreams.input()  and UnnamedStreams.output() methods.  Language statements that operate differently based on whether the unnamed input or output is redirected use the UnnamedStreams class to detect these cases and access the stream(s).  See below .

For each named stream that is defined (and for the 2 unnamed streams), an instance of a RemoteStream is instantiated using a static method in StreamFactory .  This reference is named with a converted Java name or is accessed "anonymously" via UnnamedStreams.  Each RemoteStream will actually reference a remote concrete Stream sub-class such as FileStream or ProcessStream which operates on the client side of the system .

A concrete Stream subclass is instantiated and assigned to the named or unnamed stream instances whenever an INPUT FROM, OUTPUT TO, INPUT-THRU, OUTPUT-THRU or INPUT-OUTPUT THRU language statement is encountered.  The type of requested resource determines which class is instantiated.  This constructor opens the backing resource on the client.  At this point the stream (RemoteStream actually) reference is valid for use.

When a file is opened for output, any existing version of that file is removed and a 0 length file is created (unless APPEND is specified). That file is closed when the next stream is opened or by any explicit INPUT CLOSE, OUTPUT CLOSE or INPUT-OUTPUT CLOSE.

When a file is opened for input, the read pointer is located at the first byte of the file.

The STDIO pipes from a child process are fully supported.  The common process launching infrastructure is used to start the child process but a special ProcessStream class is used to implement the proper user-driven reading/writing support for these pipes.  All reads are done from a "combined" STDOUT and STDERR, where output will first be read from STDOUT and then from STDERR.  Due to the "combining" semantic of Progress, a polling method is used to perform I/O without blocking on one of the pipes (thus ignoring input from the other pipe).  In addition, an extra thread is used to wait for the termination of the child process.  This allows these polling loops to exit when the pipes are empty and the child process has exited, which eliminates infinite read blocking.

All of the formatted and unformatted reading and writing is provided by a common set of methods in Stream:
These handle the Progress semantics while actually reading/writing using a smaller set of very simple workers that are implemented in the subclasses.

The Stream class also implements the buffering of reads/writes, and all the line number, paging and column support.

Stream processing generates the EndConditionException on EOF (ENDKEY) and ErrorConditionException for ERROR condition generation.  This processing honors the NO-ERROR construct using the silent error mode in the ErrorManager class.

Implicit Close Behavior

Streams are automatically closed streams on exit from the scope in which they are defined, except for global streams which are closed on end of the session. The TransactionManager registerFinalizable() or registerTopLevelFinalizable() is used to obtain this support.  Global streams are registered with the global flag set true .  This registraton code is emitted into the client application, just after the reference is initialized.  Shared streams that are not "NEW" are just imported.  For this reason, they have already been registered for implcit close support in a previous scope.  This table summarizes the states in which registration occurs:

DEFINE STREAM Type
 Registration
NEW SHARED
 TransactionManager.registerTopLevelFinalizable(this, true)
NEW GLOBAL SHARED TransactionManager.registerFinalizable(this, true)
SHARED
 n/a
(not shared at all)
 TransactionManager.registerTopLevelFinalizable(this, true)

At each top level scope (trigger, external or internal procedure), the current unnamed input and output streams are "remembered" as the defaults for that scope.  Any close that occurs for the unnamed input or output streams which is not the result of assigning a new stream, causes the unnamed stream to be restored back to the default for that scope.  Since the Progress programmer can change the unnamed streams but not query them, this feature was needed otherwise an unnamed stream redirected in a calling scope could be modified downstream and upon return that stream might have been already closed.  This would lead to bizarre behavior.

Shared Streams

All NEW shared and NEW GLOBAL shared streams are supported using the SharedVariableManager .  This class has stream-specific methods to add and lookup streams.  Please see Shared Variables for more details.

References
rules/convert/base_structure.rules
rules/convert/input_output.rules
rules/convert/process_launch.rules

com.goldencode.p2j.util.Stream.java
com.goldencode.p2j.util.StreamWrapper.java
com.goldencode.p2j.util.FileStream.java
com.goldencode.p2j.util.ProcessStream.java
com.goldencode.p2j.util.ProcessOps.java

Limitations
  1. Although documented in Progress references, the default code-page conversion (stream to internal and vice versa) does not appear to actually be implemented OR perhaps this is locale specific.  No default codepage conversion is implemented in P2J.
  2. All of the explicit codepage conversion processing that is possible in Progress is missing.
  3. The effect of the BINARY option is only accounted for in READKEY processing and in this case it only modifies how newlines are returned.  If BINARY modifies other processing, it has not been identified and is thus unimplemented at this time.
  4. Progress references document that the null character terminates strings.  This is not implemented.
  5. DBCS support is not provided.
  6. UNBUFFERED mode is not supported (and is probably not needed).
  7. MAP/NO-MAP support does not exist.
Remote I/O and Terminal Integration
Introduction

The Progress 4GL is implemented as a client-centric environment.  Even the Progress database server is really just a (shared memory or network) connection to a process which takes a limited specification of index fields and the corresponding values or value ranges and returns records from that index.  All query where clause and sorting is handled on the client side!

In a P2J environment, the "client" is the "thin client".  This is the Java process written to connect to a P2J server and provide a remote (from the server) user interface.  It is run in the context of the user's operating system login or shell.  In other words, the user will be logged in to some operating system (e.g. Linux) and will obtain a shell or desktop.  If this is a terminal oriented system (CHUI) then that user will have logged in via a terminal session (via a serial line or a network protocol such as telnet or ssh).  After the login, the user will have some shell or desktop process running on that system.   From there the user either explicitly (with a command or icon) or implicitly (via a profile or startup script) will launch the P2J thin client Java process.  When this document references the "thin client" or "client", this is what is meant.

Stream and File System Remoting

When a Progress 4GL program reads/writes files, interacts with the file system or launches child processes, it is intrinsicly doing so on the client system.  This may not be evident in a system where the client and the P2J server are the same physical box.  But since all such activities in Progress 4GL are done in the context of a child process (e.g. on Unix, the executable "_progres") which is running with the login context of a specific user, the home directory and file system access rights of that specific user.  In a P2J server environment, the entire server is running in a process which is a daemon and while it does have an operating system security context, this is most certainly not the same context as a given user.  This means that every process launch *may* be dependent upon context that is only available on the client system.   The child process may be dependent upon access to the user's home directory or to data that is in a specific path that only exists on the client system.  Or perhaps the process itself only exists on the client system and/or has specific environment variables or other state/resources that are only available there.  For example, many Progress programs launch shell scripts or utilities that depend upon the client's context for current working directory and other environment variables (e.g. the Unix mail command).  For this reason, the end-points of all I/O operations (file, file system and process launching) must be executed on the client system.  In other words, they must be "remoted" to the client.  The business logic will still reside on the server.  This includes all logic that accesses and manipulates data, which defines the flow of control for the application.  However, when a read or write or file system search or process launch occurs, these operations must be remoted to the client but made to appear as if they were happening on the server.  This allows the logic to remain intact and the minimal amount of processing is pushed to the client.

The Stream class is abstract.  It defines the following abstract methods:
The backing file (FileStream) and process streams (ProcessStream) are Stream subclasses that implement concrete versions of these methods.  The RemoteStream class is used on the server and it redirects all calls to the abstract methods to instances of FileStream or ProcessStream.  This is handled using the LowLevelStream interface and a StreamDaemon class on the client side.  Each RemoteStream instance represents a single stream on the client and references that stream via an integer ID.  The StreamDaemon implements the LowLevelStream interface which is called by RemoteStream for service.  Each API has the innteger stream ID passed as its first parameter.  The StreamDaemon looks up the referenced stream and dispatches the method call to the real stream.

The StreamFactory class provides static factory methods (openFileStream() and openProcessStream()) to instantiate streams on teh client..  These methods call a client side export in the StreamDaemon to create the backing FileStream and ProcessStream resources on the client.  It obtains the integer stream ID in return and intitializes a RemoteStream instance using this ID.  The resulting RemoteStream object is returned to the caller (e.g. business logic) and all access to the RemoteStream is transparently redirected to the client.

INPUT THROUGH, OUTPUT THROUGH and INPUT-OUTPUT THROUGH all allow one to launch a child process and access the child's stdout/stderr for reading, or the child's stdin for writing or both.  This is handled by ProcessOps.  ProcessOps is the Progress compatible interface for process launching.  It accepts RemoteStream instances and uses a Launcher interface and the client-side ProcessDaemon instance on the client to actually process the process launch (using the J2SE Runtime.exec() method).  The resulting child process' standard I/O is connected to the client-side ProcessStream instances.  On the client side, when the child process is launched, its stdin and/or stdout/stderr are either connected to a pipe or to the parent process' terminal depending on the flow direction of the stream definition.  This is handled with the cooperation of ProcessDaemon.launch() and the CHARVA Toolkit.pseudoTerminalLaunch().

FileSystemOps provides a set of static methods to inspect and interact with the file system.  These are redirected to the client using the FileSystem interface which is serviced by the FileSystemDaemon class.

As a general rule, the maximum amount of processing that can be done on the server is done there and only a minimum remote interface is put in place to provide remote service.

For other process launching (UNIX, OS-COMMAND...), such items don't need to be read/written as streams on the server, but the launching must occur on the client anyway.  This means that the ProcessOps.launch(String[], boolean, boolean) method which is used in these cases, is serviced by ProcessDaemon as above, except the actual process launching mechanism is different (see below).

Non-Stream Process Launch Remoting

Normal process launching (UNIX, OS-COMMAND...) is done in 3 forms:
  1. Synchronous (lack of NO-WAIT option)
    • interactive
    • non-interactive
  2. Asynchronous (NO-WAIT option)
Some launched applications are synchronous and interactive (the user is allowed to interact with the child process' UI).  In such cases, the child process' standard I/O must be connected to the terminal (such as "vi").  Such programs cannot be run with the NO-WAIT option (it just doesn't work properly in Progress, your terminal gets very "confused").  The result is that the child process takes over the terminal and Progress is blocked (it is a synchronous call when NO-WAIT is not available).  This means that no output occurs to the screen except what is being done by the child process.

Java's Runtime.exec() process always creates a child process that whose STDIO is not connected to a terminal.  This means that interactive applications like "vi" will not properly work when launched by the standard J2SE process launching mechanism.  Since Progress allows child processes to be interactive in the current terminal, a different process launching mechanism is needed.

To duplicate this processing, the Charva implementation programs NCURSES to temporarily "shell out" or suspend current processing (see ThinClient.suspend()).  This means that NCURSES resets the terminal settings to those which were set at NCURSES init.  At that point, a child process is created (using fork).  After fork, the parent process will return and block (unless NO-WAIT was specified) until the child process exits.  If NO-WAIT is specified, the parent process resumes. The child process copies the command line parameters and then calls the Linux/UNIX execvp() system call.  When the child process is done, it exits.  The parent process will use ThinClient.resume() to re-enable the interactive terminal after it pauses (or just before it returns from ProcessDaemon.launch() if NO-WAIT was specified).  While the child process runs, all STDIO is directly connected to the parent process' terminal (the child process is directly connected to the user's current terminal). 

The process launching API exists in the Charva libTerminal.so to handle the fork() and execvp().

Either way, the worker for ProcessOps static methods is in the client's ProcessDaemon.  The ProcessDaemon uses features of the ThinClient to properly integrate with the terminal, to suspend/resume CHARVA and to handle the actual process launching.

Other applications are synchronous and non-interactive child processes.  Some of these may still write to STDOUT/STDERR.  Progress attaches the terminal I/O to the child process.  Progress itself can't know what processes are interactive and which are not.  This is true because the fact of whether a process is or is not interactive is something that cannot be inspected, it is implicit in the application (or applications if the child process launches other child proceses) being run.  In addition, there is no provision in the language for the programmer to provide a hint to Progress about this fact.  For this reason, we must conclude that Progress implements the launch of all synchronous processes in the same way for each process ("shelling out" the terminal, forking, execing to launch the process).

Finally, one can launch an asynchronous application (by adding the NO-WAIT option).  Such programs are always non-interactive applications.  However, if they do write to the STDOUT/STDERR, the result from an end-user perspective is that in a NO-WAIT condition, the child process' output is intermixed (in a non-deterministic manner) with output created by Progress.  For example (press the spacebar quickly when at the first "Press Space to Continue" prompt):

os-command no-wait "echo HELLO; sleep 3; echo 'HELLO, AGAIN!'".
def var c as char init "Hello World!".
display c.
enable.
os-command silent "echo GOODBYE".

The NO-WAIT launch in Progress don't seem to be really any different from the normal synchronous approach in terms of how the launch itself is done.  The only difference seems to be in whether or not Progress waits for the child to exit.  In a NO-WAIT case, the resume() is called immediately after launching, such that Charva can resume immediately.  In the synchronous case, resume() is not used until after the child process exits.  Either way, this simply means that the decision to wait or not to wait is always made just before the call the resume().

The remote worker methods handle NO-WAIT (making these methods synchronous - waiting for the child process to end - or asynchronous) as documented above.  No special proxies or other features are ever returned to the business logic for these ProcessOps.launch(String[], boolean, boolean) modes.  The server-side call invokes the remote worker which launches the child process on the client synchronously or asynchronously and then returns.  The server-side call then returns to the business logic.

Note that the ProcessOps class has some "cleaner" classes and extra cleanup threads to handle the proper close of the pipes and child process resources as needed.

The NCURSES-related changes are in Charva's libTerminal.so (toolkit.c) and the corresponding Toolkit.java.

NCURSES references:

http://www.faqs.org/docs/Linux-HOWTO/NCURSES-Programming-HOWTO.html
http://invisible-island.net/ncurses/ncurses.faq.html
http://www.tldp.org/HOWTO/Text-Terminal-HOWTO.html (especially the bit about termcap/terminfo and the part on psuedo-terminals)
http://www.cs.utk.edu/~shuford/terminal/termcap_news.txt   (search on "newterm(" for a bit of an example on multiple terminals)
http://dickey.his.com/ncurses/ncurses-intro.html

You will especially want to review:

initscr() - default init for single terminal applications, uses newterm() and other stuff under the covers
newterm() - creates a terminal, used instead of the initscr(), sets the current terminal to that newly created instance and returns a pointer to it
set_term() - switches the current terminal to the one passed, returns the previous terminal
delscreen() - deallocates memory for a terminal
SCREEN* - the data structure that holds the terminal "instance"
endwin() - used to reset the tty attributes/modes to the normal interactive shell (or whatever was set before ncurses started), is used at the end of all processing and also temporarily to "shell out" to another (non-ncurses) program
def_prog_mode() - temporarily saves off the current terminal settings, called prior to an endwin() such that the settings can be restored after "shelling out"

Stream Integration with the UI

In Progress 4GL there is also a concept of the "terminal".  The terminal is the user-interface (UI) with which an end-user will interact.  There are a wide range of language features provided to allow a UI to be created and maintained in Progress.  Progress does differentiate between "regular" streams (e.g. file system and child process I/O) and the terminal.  For example, there are certain operations that only are allowed on a non-terminal stream (e.g. PUT, EXPORT, IMPORT, SEEK...).  However, in other ways there is little differentiation between the terminal and streams.  In other words, the terminal can be redirected to a stream and many UI statements that normally only have interactive meaning become non-interactive and use their frame-oriented layout and formatting knowledge to read from the stream into a field list (INSERT/SET/UPDATE/PROMPT-FOR) or write from a field list to the stream (DISPLAY).  Some UI statements don't read or write to the stream, but do operate on the stream in other ways (DOWN, UP, HIDE, VIEW).

These reads/writes can be made on a replaced unnamed stream (e.g. INPUT FROM myfile.txt) or from a named stream (e.g. INPUT STREAM my-stream-name FROM myfile.txt).  In Progress, the former method operates on the "unnamed" stream which by default is the same as the terminal.  Thus, normally a UI language statement would operate upon the terminal by default, but some conditional processing (e.g. IF/THEN/ELSE) might redirect the unnamed input and/or output stream(s) such that a single UI statement would operate on either resource.  If that statement happens to be operating on the terminal, then the statement is interactive.  If it is stream based, the statement will have no interactive (end-user observable) side effects.  It is only at runtime that Progress can determine which "path" will be taken for these UI statements that can also be used on streams.  Since named streams can never reference the terminal, when a UI statement uses an explicit (named) stream, it is easy to identify that this is a stream statement and not really a UI statement.  However, when the unnamed stream has been redirected, these statements become "bimodal".  While there are some simple examples to the contrary, in many cases there is no way to know how that statement will be serviced (I/O versus UI) by analyzing the source code at conversion time.  Since such a case must inherently be determined at runtime, there must exist a common interface for these methods which then detects and uses the proper mode.  Consider this example:

def var redir as logical.
def var i     as integer.

/* some processing that calculates redir, possibly with UI or database reads */

if redir then
   input from file.txt.

set i.

message "i =" i.

Due to the runtime nature of this decision, backing methods for INSERT/SET/UPDATE/PROMPT-FOR/DISPLAY have been created in GenericFrame (actually, there is no support for INSERT support at this time).  All of the frame elements (in the proper order) being read are passed as a FrameElement array.  If an explicit stream is passed to SET/UPDATE/PROMPT-FOR then it is expected to be open for input (if not an error occurs).  In stream processing mode, a completely different operational mode is entered which uses an input reading method to feed the screen buffer from the input stream.  This bypasses the normal user interface processing for enable/disable, standard view/Z-order/refreshing support.  This mode is entered when a stream is explicitly passed to the API OR if no stream is explicitly passed but the unnamed input stream is redirected (has been explicitly opened as a stream).

There are 2 forms that can cause this to occur:
  1. The unnamed input stream is opened as a file or child process.  It defaults to the terminal, but once opened (and as long as this is not closed) it will reference the non-terminal stream.
  2. An explicit stream reference can be added to the INSERT/SET/UPDATE/PROMPT-FOR language statement.
Either way, this forces the input to be removed from the terminal/keyboard and obtained via the stream.  This also eliminates the ENABLE/DISABLE processing and the WAIT-FOR that is done inside the PROMPT-FOR portion of these statements.  Instead, the order of the fields in the SET/UPDATE/PROMPT-FOR is the order in which fields are read from the stream.  This makes these statements a kind of analog to the PUT language statement (this writes a list of fields to a stream).  It is important to note that the layout of the associated *frame* DOES NOT make any difference in how the data is read.  For example if the frame layout itself is ordered differently than the order of the fields in this language statement OR if the frame has other fields that are not listed in the language statement, these other fields and ordering are ignored.  It also means that the list of already enabled widgets makes no difference (only the fields listed explicitly on this statement will be read and they will be read in the exact order of this current statement).  One could say that this is really just a reading counterpart to PUT except for these two points:
  1. A single source code statement can both handle real UI and reading from the unnamed stream and this fact is detected and implemented at runtime only.
  2. These statements can also write data, either to the screen (if output is not redirected) or to a named or unnamed stream (if output is redirected).  This additional output only occurs if the stream in question does not have NO-ECHO specified (e.g. INPUT FROM myfile.txt NO-ECHO).
The Stream class supports setting the echo mode (it defaults to true ).  Please note that for streams opened via the INPUT-OUTPUT THROUGH language statement, this value must default to false , which is handled by emitting a setter into the business logic in these cases.  Interestingly, even if the input stream is marked as echo, if output is redirected and that output stream is marked as no-echo then the echo does not occur.  In other words, if both input and output are redirected, both streams must be marked as echo for the echo to occur.  If output is not redirected, then only the input stream's echo setting is taken into account since the screen is always considered to be in echo mode.

There is also an unusual behavior in this echo mode.  The SET/UPDATE/PROMPT-FOR language statements are "bimodal" (they handle both terminal and stream I/O).  In stream mode, if echo is off, no output occurs (to either the terminal or to a file if the unnamed out is redirected to file before the call occurs).  If echo is on, then output is generated AND it is based on the frame definition/screen-buffer that is in use.  However, the output is done in a "special" text only mode.  In other words, the output values are the exact text that has been parsed out of the input file AND it is NOT data type specific.  Any data in a fill-in field that has a data type that is non-character is treated as character.  In addition, it seems that any widget type other than fillin (e.g. combo-box or button but strangely enough also text!) generates no output at all.  In all cases (whether the widget does or doesn't emit), the column headings are done just as normal.  A *normally* right-aligned field like a decimal fill-in will have its column header right aligned BUT its data will be the unformatted, left-justified text just as it was read from the input stream.   For example:

┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│c[1]                 c[2]                 c[3]                 l    l2 dd                 i                 d│
│──────────────────── ──────────────────── ──────────────────── ──── ── ────────── ───────── ─────────────────│
│Double quoted text   'single              quoted               fals T  11/22/1999 -9,98787  +2,323243.32     │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│c[1]                 c[2]                 c[3]                 l    l2 dd                 i                 d│
│──────────────────── ──────────────────── ──────────────────── ──── ── ────────── ───────── ─────────────────│
│ Double quoted text    'single              quoted               Nope t  11/22/1999 -00998787 00002323243.32000│
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
The first output is the echo mode of SET.  The second output is using DISPLAY on the same data that was read.  The first 3 columns are character data and thus show no difference.  The l and l2 columns are logicals.  The logical data read from the file differs from the 2 outputs (they are both using format strings "Yep/Nope" and "t/f" respectively).  The dd field is a date and doesn't show any difference. The i (integer) and d (decimal) fields show the unformatted, left aligned text read from the file in the echo mode and the properly formatted, right aligned data in the DISPLAY.  The i format string is -99999999 and the d format string is 99999999999.9999 for both the SET and DISPLAY.

So both cases use the same frame layout and format strings, but the actual data is treated as unformatted text in the echo mode.  At this time, this special text mode for logicals, integers and decimals is not supported.

As an additional unusual behavior of this quirk, any errors resulting from truncating the input data cause this truncated data to be written back to the screen buffer before the error condition is raised (the SET/UPDATE form do not assign that data back but the data does exist in the screen buffer).  If echoing is active, the echo of that screen buffer will occur even though an error occurred!  At this time, this screen buffer copy is only supported for character data types, the other types do not get stored.

INSERT/SET/UPDATE/PROMPT-FOR are reading instructions when used with an input stream that is not the terminal.  Each has a list of lvalues (fields). Data is "read" from the stream on a whitespace delimited (and for character data, optionally double quote delimited) basis.  It is not possible to control the delimiter as can be done via IMPORT.  The parsed data is converted into each corresponding field's type and the data is then assigned into that field. Each field's format string (default, from dictionary, from var def, overridden in a format phrase) will be honored in that conversion.  The format string is obtained from the associated widget's configuration rather than being passed via the API itself.

All reading is done in the exact order of the widgets as listed in the frame element array passed to the SET/UPDATE/PROMPT-FOR APIs.  The order of these same fields in the frame itself is meaningless.  So:

form x y z with frame fr.
input from ./text.txt.
set z x y with frame fr.
input close.

The order of the fields in the form statement is the order of the data when displayed on the screen or when output is redirected to a stream.  BUT any stream input reading honors the order in the SET/UPDATE/PROMPT-FOR.  In the example above, this means that the first double quote or whitespace delimited entry on the current line in the input stream will be assigned to the "z" variable, the second to "x" and the third to "y".

It is important to note that the field-level data parsing is identical to that done for the IMPORT language statement.  So field skipping when reading an embedded hyphen, reading 1 line at a time (per SET/UPDATE/PROMPT-FOR), no assignment to the screen-buffer for fields read after the end of the line has been found, double quote processing as a single entry (allows embedded whitespace or hypens)... all these behaviors are the same. This is exposed by the Stream.readFieldWorker().  This worker is shared by the IMPORT implementation, but how the results are used (and the validation that is done) are quite different after the parsing is done.

In particular, this form of reading is uses the format string as a basis for the proper length of the data to read from the stream for the given field however the actual contents of the format string are not used to parse except for the logical type.  In that case, the values specified in the format (e.g. "Yep/Nope") will be honored.  For all other data types, the only affect of the format string is on the size.  An error condition is generated if data is read that is larger than can fit into the size specified by the format string.

The conversion from string data (as read from the file) to BaseDataType is handled by detecting the widget's data type and then instantiating the proper type using the constructor that using a string as input.

The way the logical formatting is processed is an exception.  This processing trucates the input text (if needed) to the maximum size of either of the 2 values in the format string (for "Yep/Nope" then maximum size would be 4).  Any truncation here is not an error (as it is with the other types), instead the data value (the string read from the stream) and the format string are passed to a special logical constructor for processing.  Please note that this means that (as in Progress), there is a quirk where a longer input (for example, "nopenotatall") could actually match the longer of the 2 format string values (for example, the false value of the "Yep/Nope" format).  This also applies to true/false and yes/no.  So a value of "yessiree" would be interpreted as true in any case where the longest format value is 3 characters.

For a more interesting example, please see testcases/uast/io_redirected_by_ui_stmts.p.

Consider these examples:

Example Program
 Output
def var i as int.

input thru "echo 5".
output to ./test.txt.

prompt-for i.

output close.
message input i.
 test.txt contains:

         i
----------
5

The message line on the terminal contains:

5
def var i as int.

input thru "echo 5".
output to ./test.txt.

prompt-for i.

message input i.
output close.
 test.txt contains:

         i
----------
5
5

The message line on the terminal contains nothing.
def var i as int.

input thru "echo 5".

prompt-for i.

message input i.
 The terminal displays:

┌──────────┐
│         i│
│──────────│
│5         │
└──────────┘

The message line on the terminal contains:

5

Please note that any statement that generates output (both PROMPT-FOR and MESSAGE in the examples above) will do so to the redirected stream.

A special case exists when input is NOT redirected BUT output IS redirected.  In this case PROMPT-FOR, SET and UPDATE all will actually allow interactive editing via the normal terminal (like a temporary switch to interactive mode) BUT then after the edits are done there is effectively an additional VIEW of the data to the output stream.

The most common redirected output statement is DISPLAY.  DISPLAY writes the given list of fields/expressions to the stream using the format and layout of the frame.  This is the most common way to generate a report in a file or for a printer.  This can be a redirection of the unnamed stream or the DISPLAY statement can also reference a named stream.

The format of the stream output will contain explicit (SKIP) and implicit (based on a list of fields that exceeds the display's width) newline characters exactly as the terminal would display.   The following example illustrates this:

Example Program
 Output
def var txt1 as character.
def var txt2 as character.
def var i    as integer.

do i = 1 to 6:
   txt1 = txt1 + "Hello World! ".
end.

txt2 = txt1.

display txt1 format "x(78)" txt2 format "x(78)".

 ┌──────────────────────────────────────────────────────────────────────────────┐
│txt1                                                                          │
│txt2                                                                          │
│──────────────────────────────────────────────────────────────────────────────│
│Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! │
│Hello World! Hello World! Hello World! Hello World! Hello World! Hello World! │
└──────────────────────────────────────────────────────────────────────────────┘
output to ./test.txt.

def var txt1 as character.
def var txt2 as character.
def var i    as integer.

do i = 1 to 6:
   txt1 = txt1 + "Hello World! ".
end.

txt2 = txt1.

display txt1 format "x(78)" txt2 format "x(78)".
 test.txt contains:

txt1
txt2
------------------------------------------------------------------------------
Hello World! Hello World! Hello World! Hello World! Hello World! Hello World!
Hello World! Hello World! Hello World! Hello World! Hello World! Hello World!

It is also important to note that the box normally drawn around a frame is not output to the stream.  Instead there is an empty line where the top and bottom horizontal lines would be and leading spaces where the left vertical line would be.  No output is generated for the right vertical line.  If the frame is defined as a NO-BOX, then none of those spaces or extra lines will be output.

Column headings do get output using hyphen characters (and a space in between each column) as the separators.  NO-LABELS and SIDE-LABELS are honored just as they would be in the terminal.

In the case of the DISPLAY statement, it is important to note that it still copies data to the screen buffer and it is the resulting screen buffer that is written to the stream.

A close "cousin" of DISPLAY is the VIEW language statement.  This does not copy any new data to the screen buffer, however it does write the contents of the referenced screen buffer to the named or unnamed stream.  This is often used to output a static (unchanging) report header multiple times.  Output is done using "frames" (screen buffers).  If that frame (including the unnamed frame) is referenced for both file and terminal output, the data placed in that screenbuffer at the time of output will include all prior data.  For this reason Progress source code often uses a "fake" frame for file I/O so that that data doesn't appear on the terminal later.

The HIDE statement can also be used in these cases, but there appears to be no effect of such a statement (it is a NOP except in an obscure case -- see below).

The following examples show the affect of down frames and the DOWN and UP language statements.

Example Program
 Output
output to ./test.txt.

def var i as integer.

do i = 1 to 6 with frame fr:
   display i with 12 down frame fr.
end.
 test.txt contains:

         i
----------
         1
         2
         3
         4
         5
         6
output to ./test.txt.

def var i as integer.

do i = 1 to 6 with frame fr:
   display i with 12 down frame fr.
   up 1.
end.		 test.txt contains:

         i
----------
         1
         2
         3
         4
         5
         6
output to ./test.txt.

def var i as integer.

do i = 1 to 6 with frame fr:
   display i with 12 down frame fr.
   down 1.
end.		 test.txt contains:

         i
----------
         1

         2

         3

         4

         5

         6
def var i as integer.

do i = 1 to 6 with frame fr:
   display i with 12 down frame fr.
end.

 The following is displayed on the terminal:
┌──────────┐
│         i│
│──────────│
│         1│
│         2│
│         3│
│         4│
│         5│
│         6│
│          │
│          │
│          │
│          │
│          │
│          │
└──────────┘
def var i as integer.

do i = 1 to 6 with frame fr:
   display i with 12 down frame fr.
   up 1.
 end.		 A series of screens is displayed on the terminal, with each screen alternating between a number (1-6) and a blank/empty output.  12 screens are displayed in all, each in turn.  The first looks like:

┌──────────┐
│         i│
│──────────│
│         1│
│          │
│          │
│          │
│          │
│          │
│          │
│          │
│          │
│          │
│          │
│          │
└──────────┘

 
def var i as int no-undo.          
                                   
do i = 1 to 10:                    
  display "Count: " i with 10 down.
  if i < 6                         
    then down.                     
    else up.                       
end.
 The result on the terminal:

┌──────────────────┐
│                 i│
│        ──────────│
│Count:           1│
│Count:          10│
│Count:           9│
│Count:           8│
│Count:           7│
│Count:           6│
│                  │
│                  │
│                  │
│                  │
└──────────────────┘

 
def var i as int no-undo.          
                                   
output to tmpup1.out.   
                                   
do i = 1 to 10:                    
  display "Count: " i with 10 down.
  if i < 6                         
    then down.                     
    else up.                       
end.
 The result in the stream:


                 i
        ----------
Count:           1
Count:           2
Count:           3
Count:           4
Count:           5
Count:          10
def var i as integer.

do i = 1 to 6 with frame fr:
   display i with 12 down frame fr.
   down 1.
end.		 The following is displayed on the terminal:
┌──────────┐
│         i│
│──────────│
│         1│
│          │
│         2│
│          │
│         3│
│          │
│         4│
│          │
│         5│
│          │
│         6│
│          │
└──────────┘

Following rules determine the Progress behavior of the DOWN/UP/SCROLL statements with a down frame, with or without the RETAIN/SCROLL n attributes:
  1. Up statements with a down frame (see testcases/uast/downframes/up_statements_with_a_down_frame.p):
    • For a down frame, when an up statement lands on an uninitialized row,the screen buffer will be reseted.
    • Up statements which land on an initialized row will restore the screen buffer for that row.
    • If there is an attempt to go up past the first row, then FRAME-LINE will be set to FRAME-DOWN value and screen buffers will be reseted for alliterations.
  2. Down statements with a down frame (see testcases/uast/downframes/down_statements_with_a_down_frame.p):
    • For a down frame, when a down statement lands on a uninitialized row, the screen buffer will be reseted.
    • Down statements which land on a initialized row will restore the screen buffer for that row.
    • If there is an attempt to go down past the FRAME-DOWN row, then FRAME-LINE will be set to 1 and screen buffers will be reseted for all iterations.
  3. Down frame with stream output (see testcases/uast/downframes/down_frame_with_stream_output.p):
    • The up statement will have no effect on the screen buffer. The down statement will reset the screen buffer and advance to the next row.
    • If there is an attempt to go down past the FRAME-DOWN row, then a new row will be added to the frame.
    • FRAME-LINE for streamed frames will always be 0.
  4. Conditional up/down statements (see testcases/uast/downframes/up_down_conditional_with_a_down_frame.p):
    • if the up/down lands outside the frame, the frame gets cleared only when the conditional up/down is triggered by a down or display instruction.
    • after a conditional down, the "input frame f0 i" will return the value specified by the row at frame-line + 1.
    • after a conditional up, the "input frame f0 i" will return the value specified by the row at frame-line - 1
  5. Up/Down statements with an uninitialized down frame (see testcases/uast/downframes/up_down_statements_with_a_down_frame_uninitialized.p):
    • For uninitialized frames, performing up or down statements have the same behavior as if the frame would have been initialized.
    • Going up over the first row will reset all screen buffers and land on FRAME-DOWN row.
    • Going down over the FRAME-DOWN row will reset all screen buffers and will land on first row.
  6. Retain n - down statements with a down frame (see testcases/uast/downframes/retain_n_down_statements_with_a_down_frame.p): For a down frame, when a down statement lands on a uninitialized row:
    • if retain = 0, screen buffer will be reseted for all rows, FRAME-LINE is 1
    • if retain != 0, then the last RETAIN rows will be set to be the first RETAIN rows in the same order; FRAME-LINE is (RETAIN + 1). The screen buffer for the last (FRAME-DOWN - RETAIN) rows will be reset.
  7. Retain n - up statements with a down frame (see testcases/uast/downframes/retain_n_up_statements_with_a_down_frame.p): For a down frame, when a up statement goes past the first row:
    • if retain = 0, screen buffer will be reseted for all rows, FRAME-LINE is FRAME-DOWN
    • if retain != 0, then the first RETAIN rows will be set to be the last rows in the form, in the same order; FRAME-LINE is (FRAME-DOWN - RETAIN). The screen buffer for the first RETAIN rows will be reset.
  8. Scroll n - down statements with a down frame (see testcases/uast/downframes/scroll_n_down_statements_with_a_down_frame.p): For a down frame, when a down statement lands on a uninitialized row:
    • if scroll = 0, screen buffer will be reseted for all rows, FRAME-LINE is 1.
    • if scroll != 0, then the rows are pushed up SCROLL times, the last SCROLL lines being empty lines. FRAME-LINE will be set to FRAME-DOWN. The first original SCROLL rows will be lost and inaccessible.
  9. Scroll n - up statements with a down frame (see testcases/uast/downframes/scroll_n_up_statements_with_a_down_frame.p): For a down frame, when a up statement goes past the first row:
    • if scroll = 0, screen buffer will be reseted for all rows, FRAME-LINE is FRAME-DOWN
    • if scroll != 0, then the rows are pushed down SCROLL times, the first SCROLL lines being empty lines. FRAME-LINE will be set to FRAME-DOWN - SCROLL.The last original SCROLL rows will be lost and inaccessible.
  10. Scroll statement with a down frame (see testcases/uast/downframes/scroll_statement_with_a_down_frame.p): When from-current is not used:
    • FRAME-LINE remains unchanged
    • scroll down will act as if FRAME-LINE is 1 and from-current is used
    • scroll up will act as if FRAME-LINE is FRAME-DOWN and from-current is used
    • SCROLL and RETAIN values have no effect on the SCROLL statement
  11. Scroll from current with a down frame (see testcases/uast/downframes/scroll_statement_from_current_with_a_down_frame.p): When from-current is used:
    • FRAME-LINE remains unchanged
    • scroll from-current down will push-down the rows starting from FRAME-LINE. The FRAME-LINE row will be reset.
    • scroll from-current up will pull-up the rows starting from FRAME-LINE. The last row will be reset. The FRAME-LINE row will have the screen-buffer from FRAME-LINE+1. If FRAME-LINE is FRAME-DOWN then the last row will be reset.
    • SCROLL and RETAIN values have no effect on the SCROLL statement.
  12. SCROLL n and RETAIN n should never be used together (see langerf.pdf page 543).
Following rules determine the Progress behavior of the conditional DOWN statement with a streamed, down frame (with or without box) (NL = NewLine) (file testcases/uast/downframes/streamed_paged_frame1.p):
  1. If the current line is at the end of a page, then: a DOWN 1 statement will write a NL to the next page; a DOWN 2 statement will write a NL to the current page and a NL to the next page.
  2. If the current line is at position X having Y more rows in the current page, then a DOWN Y statement will write Y NL's to the current page and one more NL to the next page.
  3. If the current line is at position X having Y more rows in the current page, then a DOWN (Y+Z) statement will write Y+1 NL's to the current page and one more NL to the next page.
  4. The file testcases/uast/downframes/streamed_paged_frame2.p demonstrates how Progress computes the LINE-COUNTER and PAGE-NUMBER functions: after a down, if the page is ended, P2J erronetly will update the line counter to 1 and also increment the page number. Progress seems to do this when the subsequent statement (down/display/etc) which affects the stream buffer is executed. NOT IMPLEMENTED YET.
Some notes:
  1. DISPLAY and VIEW generate a fully formatted output (very close to the result when viewed on a terminal) where SET/PROMPT-FOR/UPDATE generate the field output in a "raw" form.   In the DISPLAY examples, the numbers are right aligned where in the PROMPT-FOR examples, the numbers are left aligned.
  2. When writing, the UP statement has differing behavior.  On the terminal, it seems to defeat the normal behavior of a DOWN frame.  On the stream, it usually does nothing, however in some cases it can cause a form of overwriting.
  3. When writing, DOWN seems to do a similar thing for both terminal and stream.  This behavior extends to the fact that some usage of DOWN will overwrite the same data rather than generating multiple lines.
  4. Although this is not shown above, when reading from a stream neither DOWN nor UP have any apparent effect.  Most important, these language statements do not change the current read position in the stream.
  5. DISPLAY and VIEW have a special behavior when output is redirected AND the output stream is paged AND the given frame is either PAGE-TOP or PAGE-BOTTOM.  In this case, the given frame is registered as a header (PAGE-TOP) or footer (PAGE-BOTTOM) and this will be displayed at the top and bottom of each page respectively.  The only difference is that DISPLAY will ALSO cause the frame to output immediately (as well as being "called back" for the header/footer) while VIEW only causes a registration but not an actual output.
  6. HIDE only has an effect in the case where output is redirected AND the output stream is paged AND the given frame is either PAGE-TOP or PAGE-BOTTOM.  In this case the frame is removed from the list of headers/footers.
  7. MESSAGE statements output to the stream but in the case where input is NOT redirected and output IS redirected, they have the curious behavior of displaying on both the terminal and the redirected output stream.  If input IS redirected (and output IS redirected), then there will be no interative terminal output, only output to the redirected output stream.
  8. MESSAGE SET/UPDATE using a redirected input stream operate just like a SET/UPDATE from a redirected input stream (it is a stream reading statement).
  9. ENABLE, PUT SCREEN, BELL, STATUS, PAUSE always operate on the interactive terminal and have no effect on any redirected streams.
The methods that are the equivalent to the DISPLAY, VIEW, HIDE, DOWN, UP statements are available in forms that take an explicit Stream reference as a parameter (see the CommonFrame interface and the GenericFrame implementation).  In the case where any of these are called with an explicit stream reference the UI runtime code detects that these are outputs to a stream, such that the terminal is temporarily redirected to the given stream.  Likewise, if an open has occurred on the unnamed output stream, this causes the terminal to be redirected to this stream (if it isn't already redirected).  The explicit stream reference form removes the redirection when the statement (e.g. DISPLAY) is complete.  The unnamed output stream terminal redirection remains in place until that unnamed output stream is closed.

The thin client maximizes the usage of the drawing and layout management of frames while replacing the back-end output technology.  This is done by implementing a pluggable replacement set of drawing primitives in a class named OutputPrimitives in CHARVA.  The CHARVA Toolkit implements a default instance of OutputPrimitives which maps all calls to the native Toolkit backing functions.  This means that by default, CHARVA uses NCURSES for all output.  The Toolkit has an interface to switch primitives and the thin client has a RedirectedTerminal class that implements this interface.  This class is used for each stream that is used for redirected output.  These drawing primitives duplicate the buffering, clipping, cursor movement, drawing and rendering (sync) functions using a stream as the output device.

In addition, the server's UI runtime code is aware of each session's unnamed streams.  This is used to enable use of unnamed streams from the UI language statements (which can be intermixed between "redirected" UI stmts and I/O stmts that can't be used on a terminal).  See UnnamedStreams .

PAGE-TOP (header) and PAGE-BOTTOM (footer) frames are honored and the DISPLAY, VIEW and HIDE language statements register and deregister these frames with the associated streams.  When the stream starts a new page it forces all headers in the active header list to render (in order of registration) and likewise, at page end all footers are rendered.  Any partial page at stream close will be extended to the page end and the footers will be rendered there.  PAGE-TOP and PAGE-BOTTOM frames that are not used on a paged output stream have no special effect (they are treated as normal frames).

In the Progress 4GL environment, certain frames can exhibit overwrite behavior.  This can occur because of a frame's implicit DOWN 1 behavior OR because of a programmer's explicit use of DOWN/UP (see examples above).  This appears to be a side-effect of the buffering strategy used by Progress.  Writing seems to occur to a 512 byte buffer (based on my testcases).  This buffer is flushed when Progress needs to write something that would exceed the 512 byte mark OR when the stream is closed.  This can be seen in the following testcase by changing the iterations from the default 8 to 7.  The pause then occurs before the output from cat.  Please note that this example does not exhibit the overwriting behavior (the cause of this is unknown at this time), but the testcase does show the buffering approach. 
def var iterations as int init 8.

message "Redirect?" set redir as logical.

if redir then
   output thru cat.

message "Iterations?" update iterations.

def var i as int.
def var c as char format "x(64)" init "1".

do i = 1 to 63:
   c = c + "0".
end.

i = 0.
repeat while i < iterations:
   i = i + 1.
   display c with 1 down no-labels no-box.
end.

pause.
This overwrite behavior is not supported at this time.

Limitations:

Using Frames in Multistream Applications
Progress applications can use multiple streams simultaneously. An output stream is used typically with more than one frame and a single frame can be used with more than one output stream during the application run.  This raises questions about the frames and streams behavior in such a mixed environment.

There are three classes of behavior here:
  • a single stream receives output from the multiple frames
  • a single frame gives output to the multiple streams
  • multiple streams receive output from multiple frames, one frame per stream and their relations don't change
The first class triggers some processing in the stream every time the stream gets prepared to get output from another frame. We will call it frame switching.

The second class triggers some processing in the frame every time the frame gets prepared to give output to another stream. We will call it stream switching.

The third class is just a boundary condition where no switching occurs.

For the purpose of this discussion, we won't distinguish the named streams from the redirected unnamed stream, which is the terminal, whenever appropriate, and use the term stream. The term terminal, conversly, will mean non-redirected unnamed stream.

The frame switching behavior is defined by the following rules:
  • every stream keeps track of the frame it was last used with
  • whenever the frame is about to produce some new output, the last used frame of the stream is compared with the target frame for the current operation and when these are different, the frame switching takes place
  • PAGE-TOP and PAGE-BOTTOM frames do not participate in the frame switching when they are used as page elements
  • the switching itself causes flushing of the buffers of the remote terminal associated with the stream and then the streams buffers so that no partial output remains buffered anywhere
  • flushing the terminal is a no-operation.
The stream switching behavior is defined by the following rules:
  • every frame keeps track of the stream it was last used with
  • VIEW, DISPLAY and DOWN statements check the last used stream of the frame with the target stream for the operation and when these are different, the stream switching takes place
  • HIDE statement is known not to check for the stream switch
  • the switching itself causes flushing of the buffers of the remote terminal associated with the last used stream and then the streams buffers so that no partial output remains buffered anywhere
The DOWN and UP statements work differently depending on the stream or terminal as the target. The following rules are applied, if the target device is a stream:
  • UP statement is a no-operation for streams
  • for this very reason, no up/down pending counters are maintained for streams and those are reset  to 0 with every DOWN
  • conditional DOWN still sets the down pending flag, which is checked with VIEW, DISPLAY or unconditional DOWN
  • the effect of the conditional DOWN, when it's applied, differs from a DOWN 1:
    • conditional DOWN flushes the current remote terminal's buffers to the stream and is a no-operation when the buffers are empty
    • unconditional DOWN N flushes the current remote terminal's buffers to the stream and puts a newline on the stream when the buffers are empty, then adds N - 1 newlines
  • conditional DOWN is OK for 1 down frames
  • both forms of DOWN clear the current iteration of the frame and reset pending flag and counters
  • implicit conditional DOWN is no different
  • frames that were used with a stream at least once, are permanently flagged
  • the above mentioned flag is checked when the frame gets cleared with DOWN processor for the terminal as target
There are things specific to the terminal as the target device:
  • the stream association flag mentioned above triggers a different frame iteration clearing procedure during DOWN processing:
    • the down frame iteration cursor remains unchanged;
    • clearing is initiated
    • clearing the current iteration causes an implicit PAUSE on the terminal
  • the visibility attribute of the frame is meaningful only for the terminal use of the frame and is preserved across the stream-oriented operations.

Transaction Processing and Block Properties

Summary

Progress 4GL has a transaction processing environment integrated into the base language.  While some explicit control is provided over how transactions occur, the vast majority of processing is implicit based on the operations, control flow structures, block properties and sequence of statements/blocks which is encoded by the programmer.

Block Types

In the 4GL, code can be structured into related sections called blocks.  In Progress v9, the following types of blocks are available:
A block groups code for several purposes:
Each block has a "header" and a "body".  All code that is executed in a block is contained in the block body.  The description of the block itself and any explicitly controlled options are defined in the block header.

The lines of code in a block's body are executed "top to bottom".  All code in the same block will share the same behavior in terms of transaction processing.  Absent any explicit changes to the flow of control (by language statements that cause a flow of control change) or the generation of a condition (such as an ERROR) which will cause the flow of control to change, all code in a given block is executed if the block body is entered or all the code is not executed if the block body is not entered.

Some blocks can be made to iteratively execute the code contained in the block body.  Such blocks are known as looping blocks.  Progress provides blocks that can explicitly loop based on programmer controlled expressions in the block header.

Blocks support transaction processing which is the ability to undo (reverse) edits to variables or the database that occurred within the scope of that block, when an abnormal condition is raised or when an explicit UNDO statement is encountered.  Some blocks can have their support for transactions explicitly coded via the TRANSACTION keyword in the block header.

All blocks have some default behavior when abnornal conditions occur.  Some blocks allow that behavior to be explicitly specified via "ON phrases" in the block header.

Some blocks provide retry support which is the re-execution of a block with the same (original) state as the optional action in response to an abnormal condition.

Some blocks can take parameters and be called explicitly (external procedures, internal procedures and user-defined functions) or are otherwise invoked as a callback (triggers).  Triggers are special in that they are top-level blocks but they can only be invoked indirectly in response to an event (usually associated with the user interface but that is not required) that has occurred.  In this document, such blocks are called "top-level blocks".

Other blocks (all forms of DO, REPEAT, FOR and EDITING) are not top-level blocks and as such can only be contained inside another block (top-level or otherwise).  In this document, such blocks are called "inner blocks".  Inner blocks are the only kind of block that can be labeled and that label can then be referenced in language statements that explicitly change the flow of control such as UNDO statement, LEAVE statement, NEXT statement and in ON phrases.

Some blocks can have their definitions made nested inside another block.  The top level blocks (except for triggers) cannot be nested. Inner blocks can be nested.  Do not confuse nesting with recursion.  The top level blocks can be called recursively (they can call themselves directly or indirectly causing multiple instantiations on the call stack).  But the top level blocks cannot have their definitions made on a nested basis.  For example, an external procedure cannot be defined inside another external procedure.

The following summarizes these behaviors:

Block Type Looping Default Transaction Level TRANSACTION Option ON Phrases Retry Support Parameters Callable (Via Name) Labeled Nestable
external procedure no sub-transaction no no yes yes yes (filename in a RUN statement) no no
internal procedure no sub-transaction no no yes yes yes (RUN statement) no no
function no sub-transaction no no yes yes yes (in an expression) no no
trigger no sub-transaction no no yes no no no yes
DO no no transaction yes yes yes (depending on options) no no yes yes
DO TO yes no transaction yes yes yes (depending on options) no no yes yes
DO WHILE yes no transaction yes yes yes (depending on options) no no yes yes
DO TO WHILE yes no transaction yes yes yes (depending on options) no no yes yes
REPEAT (all forms) yes sub-transaction yes yes yes no no yes yes
FOR FIRST no (unless there is an additional table defined with EACH table) sub-transaction yes yes yes no no yes yes
FOR LAST no (unless there is an additional table defined with EACH table) sub-transaction yes yes yes no no yes yes
FOR EACH yes sub-transaction yes yes yes no no yes yes
EDITING yes sub-transaction no no yes no no yes yes (editing blocks for other frames can be directly contained and any kind of editing block can be indirectly contained)

Conditions

A condition is an unusual event that may occur during the course of processing.  The Progress programmer must be able to specify how each procedure responds to the range of possible conditions.  Progress 4GL provides a mechanism to specify such behavior at the level of each block.

Progress defines blocks as having very specific "block properties" which define behavior associated with a block.  These properties can be explicitly defined or are implicit.  The beginning and end of each block corresponds to the beginning and end of a "scope".  These scopes can be nested and this structure allows a Progress programmer to control the granularity to which a block's properties are applied.

In Progress, all blocks have some form of implicit and/or explicit condition processing, properties and transaction support.

One of the most central behaviors Progress provides is UNDO.  This is similar to the concept of a rollback of a transaction and allows one to abort partial changes to variables (and other non-database resources) when a failure or problem is encountered.  This is provided by default to reduce the amount of manual work by the programmer.

The behavior of which scope gets undone in transaction processing rollbacks is defined by block properties.  In addition, these block properties also define how the block responds to each type of condition AFTER the UNDO occurs.  More specifically, the way that the application changes its flow of control in response to a condition is something that is defined by block properties (implicit or explicitly overridden where possible).

The following conditions are possible:

Condition
 Default Action in Response
ERROR
 If a transaction is active, UNDO the closest enclosing block that has the ERROR property (this will also be a transaction or sub-transaction).

If a transaction is not active, the UNDO will be a no operation.

Then RETRY the same block (or loop) that has the ERROR property.
ENDKEY
 If a transaction is active, UNDO the closest enclosing block that has the ENDKEY property (this will also be a transaction or sub-transaction).

If a transaction is not active, the UNDO will be a no operation.

Only blocks with the ENDKEY property will have an interactions counter.  The interactions counter is a simple count of the number of input-blocking language statements have executed in the block. Any nested block that does not have an interactions counter of its own will have its interactions counted in the nearest enclosing block's interactions counter. This counter is used to determine the behavior of the END-ERROR event (see below).

Then LEAVE the same block (or loop) that has the ENDKEY property.
STOP
 If a transaction is active, UNDO the block that defines the transaction (as opposed to a sub-transaction).

If a transaction is not active, the UNDO will be a no operation.

Then RETRY the "startup procedure".  This means that the *entire* top-level procedure is run again from the top.

Note that if this is generated based on a database disconnect, a special form of processing will occur where the default STOP behavior cannot be explicitly overridden until an exit to a scope in which no access is made to the database that was disconnected.  At this point the normal STOP processing is re-imposed (see Progress Programming Handbook page 5-24).
QUIT
 If a transaction is active, COMMIT the block that defines the transaction (as opposed to a sub-transaction).

If a transaction is not active, the COMMIT will be a no operation.

Exit to the operating system.
END-ERROR
 Technically, this really isn't a separate condition, but rather a key in the UI which generates 2 different conditions based on context.  An END condition is generated if the key press occurs during the first input-blocking language statement in the block.  An ERROR condition is generated if the key press occurs during a subsequent input-blocking language statement in the block.

The exception to this processing occurs for blocks that do not have the ENDKEY property (this can happen for some DO blocks and is always true for EDITING blocks). In such a case, the interactions counter is not maintained for the current block but rather is maintained at the level of the nearest enclosing block which has the ENDKEY property.

This means that the normal conversion of END-ERROR to ERROR is thus modified in such blocks. In the EDITING case especially, it is likely that an ERROR will be generated (because the EDITING block itself counts as an interaction and thus with a larger interactions counter it is more likely that an ERROR will be generated). What is especially different here is that the ERROR will be propagated to the enclosing block that has the ENDKEY property! So even if the DO or EDITING block has the ERROR property in this case, if they also do not have the ENDKEY property and the interactions counter (for the enclosing ENDKEY block) is < 2 then the ERROR will NOT be processed by the current block.

See above for ENDKEY and ERROR respectively.

Some language statements and all assignments that can generate the ERROR condition can be executed using the NO-ERROR keyword, which disables the generation of this condition.  When a failure occurs, instead of generating ERROR, the ERROR-STATUS system handle will have its state updated.  Please see Progress Programming Handbook page 5-6 and Progress Language Reference page 1337 for more details.

The possible actions in response to a condition:

Action
 Description
UNDO
 Rolls back all database, variable, temp-table and work-table modifications since the beginning of the target block.  This target block must have opened a transaction or it must be enclosed within a block that opened a transaction AND must itself be a sub-transaction.  In any other case, the UNDO is a no operation.

There is no way of disabling UNDO.  Any condition that is raised will always trigger an UNDO (with the exception of the implicit behavior for QUIT which is to COMMIT instead of UNDO).  In addition, one of the actions below will also be executed.  If not explicitly specified, the implicit action is always a RETRY.
LEAVE
 Breaks (exits) out of the target block.  If the target block is a top-level block, the LEAVE will be converted to a RETURN.
RETRY
 The target block (non-loop or loop) is executed from the top with the same state as when it last executed.

Since a retry of a block in which the state never changes from iteration to iteration would yield an infinite loop, Progress provides a feature to detect when an infinite loop could occur.  This is called Infinite Loop Protection (ILP).  RETRY will be converted to a LEAVE, NEXT or RETURN depending on the block type to which the RETRY is targetted.
NEXT
 The next iteration of the loop is executed.  If the associated block is not a loop, then this is automatically treated as a LEAVE or a RETURN (see below).

Infinite Loop Protection (ILP) protects the NEXT action (when triggered from an ON phrase or from an UNDO statement).  In certain cases, a NEXT will be converted to a LEAVE or RETURN by the infinite loop protection logic depending on the block type to which the RETRY is targetted.
RETURN
 The current procedure, function or trigger returns to the caller.  There are 5 forms:
  1. Raise an ERROR condition in the caller ("RETURN ERROR").  In a function, this can be specified but the error is ignored (not raised).
  2. Return from a procedure or trigger block and allow normal processing to continue ("RETURN").  If this executed from a procedure, then the RETURN-VALUE global character variable is set to the empty string "" which can be read in the caller.
  3. Return the value of one of the basic data types when returning from a function ("RETURN <expression>").
  4. Set the RETURN-VALUE global character variable which can be read in the caller when returning from a procedure ("RETURN <character_expression>").
  5. Return and disable normal event processing when returning from an event trigger block ("RETURN NO-APPLY").  This can also be specified in other block types, but the NO-APPLY will be ignored.

In all actions (except RETURN), the block which is the target of the action can be implicitly determined (if there is no label) OR it can be explicitly specified using a user-defined label for the block.

One cannot UNDO a specified label which is outside of the scope referenced in the associated action such as LEAVE or RETRY.  In other words, the target of the action will determine the flow of control for the program.  Any UNDO operation must be scoped inside (more deeply nested than) or equal to the block which is the target for the other action otherwise a compiler error will result.

Blocks have "properties" that match a condition name.  Thus there are ERROR, ENDKEY, STOP and QUIT properties.  The existence of a property (implicitly or explicitly) in a given block means that that block provides a predictable sequence of actions in response to the condition (matching that property) being raised.  To explicitly add a property to a block, one uses the ON ERROR, ON ENDKEY, ON STOP or ON QUIT clauses in the block header.  For each of these explicit overrides, one may also specify "UNDO, secondary_action".  UNDO is required and cannot be avoided.  The secondary action is one of those listed above.  If no secondary action is specified, then RETRY is the action executed by default.

The secondary action of an ON phrase can have an explicit label defining its target block.  If no label is specified BUT there is a label defined for the UNDO portion of the ON phrase, then that UNDO label will also define the target block for the secondary action.  If neither portion of the ON phrase defines a label, then the target block will be determined implicitly.  The logic for this is very complicated.  Please see below for details.

The implicit behavior of each block type is different in regards to condition handling.  Since all condition processing is handled based on whether the condition has a matching "property" for a given block, if a block doesn't have a property and that condition is raised, that block will be bypassed and the processing will occur in an enclosing block.  This means that if (for example) a block has the ERROR property, then in the case where an ERROR condition is raised, if that block is the innermost block which has the ERROR property, then that is the level at which implicit UNDO and following implicit action occurs.  The implicit action that occurs is defined differently for each condition, see above .  This default action ONLY OCCURS if the block implicitly has the matching property (e.g. the ENDKEY property when the ENDKEY condition is raised).

Please note that the UNDO action is special since it always occurs first (if it occurs at all), before the secondary action.  UNDO cannot be disabled however it only occurs when a transaction is active in the current or a containing scope.  Some blocks always have a given properties (implicit or explicit) and some blocks can only have explicit properties (i.e. DO).  If any property is not associated with a block, then no processing of that condition will occur in that scope.  This means that the stack unwinds past that block without stopping to do any processing.  Only those properties that are implicitly or explicitly associated with a scope will ever be processed in that scope.

Each property is independent.  For example, if a block has an ERROR property, that has no bearing on how it processes the ENDKEY.

The following are the implicit properties that are inherently provided with each block type:

Block Type
 ERROR
 ENDKEY
 STOP
 QUIT
FOR
 Y
 Y
 N*
 N*
REPEAT
 Y
 Y
 N*
 N*
DO
 N* (except if the transaction keyword is present, then there is an implicit ERROR property)
 N
 N
 N
procedure
 Y
 Y
 Y* (only for the "startup procedure")
 N*
trigger
 Y
 Y
 N*
 N*
editing
 Y (for UPDATE except when the ERROR condition is generated by the END-ERROR key and the interactions counter is < 2) / N (for PROMPT-FOR and SET)
 N
 N
 N
function
 Y (always LEAVES the function with an unknown value return)
 Y (always LEAVES the function with an unknown value return) N
 N

(*) All of these entries have been found to be different than documented in the Progress Handbook (see the Block Properties summary table on page 3-3).

The DO block is the only block with no implicit behavior (except in the case where the TRANSACTION keyword or ON phrases are present).  All DO, REPEAT and FOR blocks can have the same set of explicit properties.  If explicitly set, this definition overrides the default action as documented above.

The DO block will have sub-transaction support if they contain ANY ON phrase.  The condition does not matter.  So an ON QUIT UNDO, RETRY is enough to force the DO block to have the sub-transaction property.

Please note that the function block is quite special.  In user defined functions, no user-input blocking statement can be executed.  This means that PROMPT-FOR/SET/UPDATE/INSERT/CHOOSE/WAIT-FOR/PAUSE and READKEY are all illegal, there are very limited choices of editing database fields within a function.  This combined with the fact that functions don't have the transaction property by default means that functions usually are sub-transactions  However, when a direct assignment (e.g. field = expression) occurs, this WILL cause the function block to obtain TRANSACTION level.  There is no way to make a function block a "NO" transaction (lacking the sub-transaction or transaction properties).  The ERROR and END conditions both default to LEAVE and the return value will be the unknown value in these cases.  This cannot be overridden.  Because of this behavior and the fact that one cannot specify an ON xxxx phrase with any of the 4 conditions, it is only possible to get a RETRY at the function block level by using the UNDO, RETRY. language statement when infinite loop protection is disabled (e.g. with the RETRY function). 

Infinite Loop Protection (ILP)

Infinite loop protection is a hidden feature of the block processing that is designed to modify the block behavior as specified by the 4GL programmer in order to avoid an infinite loop.  The idea is that an action that is specified by a 4GL programmer (RETRY or NEXT) which could perpetuate an infinite loop, will be silently converted (by the runtime) to a different action (LEAVE, NEXT or RETURN) in order to cause a change in the loop state that will avoid an infinite loop.  This silent conversion is done based on runtime state which is affected by the flow of control of the program.  So blocks and statements that are conditionally executed will change the state of the runtime and thus modify the conversions that occur.  For this reason, the behavior cannot be duplicated by static code modifications.

In particular, the following behavior is regulated:
This conversion will occur unless ILP has been disabled for the targetted block.  ILP becomes disabled based on runtime state which is maintained implicitly by particular language statements and by the block processing infrastructure of the runtime.

If ILP is NOT disabled, the following conversions occur:



Block Type
 Top Level Block/Loop
 Conversion
FOR EACH (all forms - even when there is a WHILE or TO/BY present)
 no loop
RETRY converted to NEXT
NEXT no conversion, allowed to execute
DO TO/BY
 no loop
RETRY converted to NEXT
NEXT no conversion, allowed to execute
REPEAT TO/BY
 no loop
RETRY converted to NEXT
NEXT no conversion, allowed to execute
FOR FIRST/LAST (all forms - even when there is a WHILE or TO/BY present) no block
RETRY converted to LEAVE
NEXT converted to LEAVE
DO WHILE
 no loop
RETRY converted to LEAVE
NEXT converted to LEAVE
REPEAT WHILE no loop
RETRY converted to LEAVE
NEXT converted to LEAVE
REPEAT
 no loop
RETRY converted to LEAVE
NEXT converted to LEAVE
DO
 no block
RETRY converted to LEAVE
NEXT converted to LEAVE
external procedure yes block
RETRY converted to RETURN
NEXT converted to RETURN
internal procedure
 yes block
RETRY converted to RETURN
NEXT converted to RETURN
function yes block
RETRY converted to RETURN
NEXT converted to RETURN
trigger
 yes block
RETRY converted to RETURN
NEXT converted to RETURN
editing
 no loop
 Infinite loop protection is always disabled in editing blocks (since they an only be called from UPDATE/SET/PROMPT-FOR as part of a user interaction), so no conversions ever occur in targetted editing blocks.

There are 2 cases where this ILP is disabled:
  1. A language statement which blocks for user input is executed before the point in the block where the condition is raised.  Please see Language Statements that Block for User Input for the list.  At the moment that such a statement executes, the infinite loop protection is disabled.  Since editing blocks are inherently processed as part of user input, infinite loop protection is meaningless for such loops.
  2. The RETRY built-in function is executed before the point in the block where the condition is raised.  At the moment that function executes, the infinite loop protection is disabled.
Until one of these 2 cases is executed, any CONDITION that generates a RETRY will result in a NEXT, LEAVE or RETURN. Only when the infinite loop protection is disabled will RETRY result in a RETRY!  Likewise NEXT may be allowed, but depending on the block type it may be converted to a LEAVE or RETURN.

All block types have retry support.  In other words, it is possible to retry any block including non-looping blocks and the top-level blocks (like a built-in function or internal procedure).  But whether the RETRY will be allowed or not is determined by the ILP processing at runtime.

It is important to note that the NEXT language statement is NOT affected by ILP.  ILP only affects ON phrases and the UNDO language statement.

A note in the UNDO language statement states that if "nothing changes during a RETRY of a block" it will convert the RETRY into a NEXT or LEAVE.  As far as can be determined using application code, the 4GL runtime does not actually monitor the state of  variables or other resources to detect if "nothing changes".  It looks like the disabling of ILP is taken as a proxy for whether things have changed in the current iteration.

The Progress READKEY documentation has a specific note regarding a special implementation of infinite loop protection.  It states that a count is used to convert UNDO, RETRY into UNDO, NEXT as well as the conversion of UNDO, NEXT into UNDO, LEAVE. While testing has not found any special counter behavior for READKEY, there is a "stutter quirk" (a counter based difference in ILP behavior) that has been found associated with ON ENDKEY UNDO, RETRY when it is used on certain block types.

The following code will cause a kind of "stuttering" where RETRY will sometimes be allowed to occur due to the state of a counter.  This stutter quirk only comes into play when an ENDKEY condition is raised and is handled by the "ON ENDKEY UNDO, RETRY" phrase.  Consider this code:

def var i as int no-undo.

do i = 1 to 5 on endkey undo, retry:
   message i.
   apply "endkey".
end.

The results:

1
2
3
3
4
4
5
5

The first 2 retries are converted to a NEXT but the 3rd RETRY is allowed to go forward as a RETRY (no conversion occurs).  Then the 4th RETRY is converted to a NEXT but the 5th RETRY is a RETRY.  This only occurs with the execution of the actions associated with an ON phrase and ONLY with ENDKEY.  If any other condition is targetted (e.g. ERROR), this will not occur.  Likewise, if any other action is specified (NEXT instead of RETRY), then this behavior does not occur.

Finally, this stutter mode is only seen in ON ENDKEY UNDO, RETRY phrases for the following block types:

Determining the Target and Meaning of UNDO, LEAVE, NEXT and RETRY

Each of these actions can explicitly reference a specific labeled block (e.g. LEAVE inner) or will implicitly reference a block.   In each of these cases, the action always operates in regards to a specific block. In these implicit cases (one in which no block label was specified in the 4GL source code), the 4GL compiler/runtime must determine the block which is referenced.  There are complex rules by which the UNDO, LEAVE, NEXT and RETRY actions are "bound" to a specific block as its target.  These actions do not always operate on the current block or even on an explicitly defined block target. This target block resolution is complicated by the block structure in which the on phrase or language statements are contained.  In particular, the target can reference enclosing blocks implicitly or explicitly.  The block properties of the enclosing blocks will also affect any implicit target calculation.

There are 3 ways that these actions can be invoked:
The UNDO language statement and ON phrases have the ability to specify up to 2 labels (one for the UNDO and one for the other action).

There are only 4 types of blocks in Progress which can be labeled: DO, REPEAT, FOR and EDITING.  Thus, only these block types can be explicitly targetted by the UNDO, LEAVE, NEXT or RETRY actions.  But when implicit block targets are calculated by Progress, it is possible for a non-labeled block to be the target of an action.  The non-labeled block types are special "top-level" blocks (EXTERNAL PROC, INTERNAL PROC, TRIGGER and USER DEFINED FUNCTION).  None of those non-labeled block types are iterating blocks.  This means that NEXT must be translated into something else (LEAVE in Progress terms).  In Progress, LEAVE on a top-level block is the same as a RETURN (with no modifiers or returned data).  So if LEAVE or NEXT do implicitly target a top-level block, then the behavior will logically be converted into a RETURN.  RETRY on a top-level block will be converted to a RETURN if infinite loop protection is NOT disabled.

All blocks in Progress implicitly have the ERROR property except for the DO and EDITING (PROMPT-FOR or SET forms) blocks.  The DO only will have the ERROR property when it has an ON ERROR phrase or if it has the TRANSACTION keyword.

Note that the ERROR property that exists for DO blocks and for the UPDATE form of EDITING blocks (and for the special "hidden" block in non-EDITING UPDATE blocks) is special in that it is not in effect in the following case:
  1. The block does not also have the ENDKEY property (EDITING blocks never have the ENDKEY property and DO blocks only have it if it is explicitly specified in an ON ENDKEY phrase); AND
  2. The ERROR condition was raised by the END-ERROR key (or an APPLY of that key); AND
  3. The current interactions counter is less than 2 (see END-ERROR for details).
This is runtime state that will modify how the ERROR property is processed when the ERROR condition is generated by the END-ERROR key. This means that an ERROR condition generated by END-ERROR key in an UPDATE block (EDITING or not) is always forwarded on to the containing block, even though all other ERROR conditions are handled as an UNDO, RETRY.

Most importantly, for the purposes of resolving unlabeled actions to blocks with the ERROR property, EDITING blocks (even UPDATE EDITING blocks) are treated as if they DO NOT have the ERROR property.

Considering this, the following are the possible implicit block target resolution algorithms:

Code Description
NEAR-ERR The nearest enclosing block with the ERROR property.

Example 1

outer:
repeat:
   inner:
   do on error undo, retry:
      <statement>
   end.
end.

In example 1, the nearest enclosing block (to the <statement>) with the ERROR properly is inner.

Example 2:

outer:
repeat:
   inner:
   do:
      <statement>
   end.
end.

In example 2, the nearest enclosing block (to the <statement>) with the ERROR property is outer.
CURRENT The "current" block.  This is the block which directly encloses the UNDO, NEXT or LEAVE language statement.  Or in the case of an ON phrase, it is the block upon which the ON phrase is specified.

outer:
repeat:
   inner:
   do on error undo, RETRY:
      <statement>
   end.
end.

In this example, for both the <statement> and the RETRY the current block is inner.
AS-UNDO The other action will be targetted at the same block as the associated UNDO is targetted.  If UNDO is targetted (implicitly or explicitly) to inner then the other action will be targetted to inner.

Assuming that the directly containing block is labeled "inner" and that block is contained inside a block labeled "outer", the following are possible formations of the UNDO language statement and ON phrases:

Feature UNDO Target Other Action Other Action Target Static Conversion Possible Compiler Error
UNDO. NEAR-ERR RETRY (implicit) AS-UNDO
n/a
UNDO inner. inner RETRY (implicit) AS-UNDO
If inner does not have the ERROR property.
UNDO outer. outer RETRY (implicit) AS-UNDO
If outer does not have the ERROR property.
UNDO, RETRY. NEAR-ERR RETRY AS-UNDO

UNDO, LEAVE. NEAR-ERR LEAVE AS-UNDO *
UNDO, NEXT. NEAR-ERR NEXT AS-UNDO *
UNDO, RETRY inner. NEAR-ERR RETRY inner
If inner does not have the ERROR property, then UNDO would choose a different block than inner.  Since RETRY is targetted at inner, this is a problem.  UNDO and RETRY must always be targetted at the same block.
UNDO, LEAVE inner. NEAR-ERR LEAVE inner
If inner does not have the ERROR property, then UNDO would choose a different block (a more enclosing one) than inner.  Since LEAVE must be targetted at a block that is the same as the UNDO target or which encloses the UNDO target, this is a problem.
UNDO, NEXT inner. NEAR-ERR NEXT inner
If inner does not have the ERROR property, then UNDO would choose a different block (a more enclosing one) than inner.  Since NEXT must be targetted at a block that is the same as the UNDO target or which encloses the UNDO target, this is a problem.  

In addition, if inner is not an iterating block the compiler fails.
UNDO, RETRY outer. NEAR-ERR RETRY outer
This is only possible if an inner block (one that more directly encloses the statement) does not have the ERROR property.  If inner does have such properties, then it would be implicitly chosen as the UNDO target and that would mismatch with the outer block as the action target, causing the compiler to error.
UNDO, LEAVE outer. NEAR-ERR LEAVE outer
The outer block or one of its nested blocks (which still encloses this statement) must have the ERROR property, otherwise the LEAVE will be targetted inside the UNDO target, which is not allowed.
UNDO, NEXT outer. NEAR-ERR NEXT outer
The outer block or one of its nested blocks (which still encloses this statement) must have the ERROR property, otherwise the NEXT will be targetted inside the UNDO target, which is not allowed.


In addition, if outer is not an iterating block the compiler fails.
UNDO inner, RETRY. inner RETRY AS-UNDO
inner must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO inner, LEAVE. inner LEAVE AS-UNDO * inner must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO inner, NEXT. inner NEXT AS-UNDO * inner must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO inner, RETRY inner. inner RETRY inner
inner must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO inner, LEAVE inner. inner LEAVE inner
inner must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO inner, NEXT inner. inner NEXT inner
inner must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO inner, RETRY outer. n/a n/a n/a
Invalid syntax, compiler will always error because RETRY and UNDO must target the same block.
UNDO inner, LEAVE outer. inner LEAVE outer
inner must have ANY property (not just ERROR) otherwise there is a compiler error.  (outer has no property dependencies)
UNDO inner, NEXT outer. inner NEXT outer
inner must have ANY property (not just ERROR) otherwise there is a compiler error.  (outer has no property dependencies)

In addition, if outer is not an iterating block the compiler fails.
UNDO outer, RETRY. outer RETRY AS-UNDO
outer must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO outer, LEAVE. outer LEAVE AS-UNDO * outer must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO outer, NEXT. outer NEXT AS-UNDO * outer must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO outer, RETRY inner. n/a n/a n/a
Invalid syntax, compiler will always error because RETRY and UNDO must target the same block.
UNDO outer, LEAVE inner. n/a n/a n/a
Invalid syntax, compiler will always error because LEAVE must target the same or a more enclosing block as UNDO.
UNDO outer, NEXT inner. n/a n/a n/a
Invalid syntax, compiler will always error because NEXT must target the same or a more enclosing block as UNDO.
UNDO outer, RETRY outer. outer RETRY outer
outer must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO outer, LEAVE outer. outer LEAVE outer
outer must have ANY property (not just ERROR) otherwise there is a compiler error.
UNDO outer, NEXT outer. outer NEXT outer
outer must have ANY property (not just ERROR) otherwise there is a compiler error.

In addition, if outer is not an iterating block the compiler fails.
ON <condition> UNDO. CURRENT RETRY (implicit) AS-UNDO
CURRENT must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO inner. inner RETRY (implicit) AS-UNDO
inner must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO outer. outer RETRY (implicit) AS-UNDO
outer must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO, RETRY. CURRENT RETRY AS-UNDO

ON <condition> UNDO, LEAVE. CURRENT LEAVE AS-UNDO *
ON <condition> UNDO, NEXT. CURRENT NEXT AS-UNDO * No error even if the implicit target block is not an iterating block.
ON <condition> UNDO, RETRY inner. CURRENT RETRY inner
CURRENT must be the same block as inner (RETRY and UNDO must always be at the targetted to the same block).
ON <condition> UNDO, LEAVE inner. CURRENT LEAVE inner
CURRENT must be the same block as inner or enclosed by inner.  The LEAVE target block needs no properties in particular.
ON <condition> UNDO, NEXT inner. CURRENT NEXT inner
CURRENT must be the same block as inner or enclosed by inner.  The NEXT target block needs no properties in particular.

However, if inner is not an iterating block the compiler fails.
ON <condition> UNDO, RETRY outer. n/a n/a n/a
Invalid syntax. The UNDO will target the current block which is more deeply nested than the RETRY target, which causes the compiler to fail.
ON <condition> UNDO, LEAVE outer. CURRENT LEAVE outer
The LEAVE target block needs no properties in particular.
ON <condition> UNDO, NEXT outer. CURRENT NEXT outer
The NEXT target block needs no properties in particular.

If outer is not an iterating block the compiler fails.
ON <condition> UNDO inner, RETRY. inner RETRY AS-UNDO
inner must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO inner, LEAVE. inner LEAVE AS-UNDO * inner must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO inner, NEXT. inner NEXT AS-UNDO * inner must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO inner, RETRY inner. inner RETRY inner
inner must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO inner, LEAVE inner. inner LEAVE inner
inner must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO inner, NEXT inner. inner NEXT inner
inner must have ANY property (not just ERROR) otherwise there is a compiler error.

If inner is not an iterating block the compiler fails.
ON <condition> UNDO inner, RETRY outer. n/a n/a n/a
Invalid syntax. The UNDO will target the inner block which is more deeply nested than the RETRY target, which causes the compiler to fail.
ON <condition> UNDO inner, LEAVE outer. inner LEAVE outer
inner must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO inner, NEXT outer. inner NEXT outer
inner must have ANY property (not just ERROR) otherwise there is a compiler error.

If outer is not an iterating block the compiler fails.
ON <condition> UNDO outer, RETRY. outer RETRY AS-UNDO
outer must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO outer, LEAVE. outer LEAVE AS-UNDO * outer must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO outer, NEXT. outer NEXT AS-UNDO * outer must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO outer, RETRY inner. n/a n/a n/a
Invalid syntax. The UNDO and RETRY must target the same block, otherwise the compiler fails.
ON <condition> UNDO outer, LEAVE inner. n/a n/a n/a
Invalid syntax. LEAVE must target the same or a more enclosing block as UNDO is targetting.
ON <condition> UNDO outer, NEXT inner. n/a n/a n/a
Invalid syntax. NEXT must target the same or a more enclosing block as UNDO is targetting.
ON <condition> UNDO outer, RETRY outer. outer RETRY outer
outer must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO outer, LEAVE outer. outer LEAVE outer
outer must have ANY property (not just ERROR) otherwise there is a compiler error.
ON <condition> UNDO outer, NEXT outer. outer NEXT outer
outer must have ANY property (not just ERROR) otherwise there is a compiler error.

If outer is not an iterating block the compiler fails.


In the above table <condition> can be ERROR, ENDKEY, STOP or QUIT.  The particular condition in question does not matter for the purposes of the behavior documented above.

The "Static Conversion Possible" column denotes the difference between runtime conversion of actions (infinite loop protection) versus static (compile time) conversion of actions.  A static conversion is one that will not change based on runtime state, but instead is based on data known at the time the source code is compiled.

At runtime, a RETRY action will be converted to a NEXT, LEAVE or RETURN (depending on target block type) if infinite loop protection is NOT disabled.

If the action is NEXT and was caused by an ON phrase or UNDO statement AND the target block is a non-iterating inner block (it is not a top-level block), then the NEXT will be converted to a LEAVE.

If the action is NEXT or LEAVE and the target block matched is a top level block (external procedure, internal procedure, function or trigger), then the action will be converted to a RETURN.

Rules for LEAVE statement processing:
  1. If there is an explicit label, that block is left no matter what properties it may have or not have.
  2. If there is no label, then the implicit target block must be found.  The nearest enclosing block type of the following will be the target:
    • DO WHILE
    • DO TO
    • DO TO/WHILE
    • REPEAT (all forms)
    • FOR EACH (all forms)
    • FOR FIRST (all forms)
    • FOR LAST (all forms)
    • EDITING
  3. Please note that the simple DO block is the only inner block type that LEAVE will NEVER implicitly target.
  4. The properties of the blocks have nothing to do with this search process.
  5. If no inner block is found with the above process, then the LEAVE is considered to be targetting the nearest top-level block.  For this reason, the LEAVE statically converts to a RETURN.
Rules for NEXT statement processing:
  1. If there is an explicit label:
    • If the block is an iterating block, then the next iteration of the block is invoked.
    • If the block is non-iterating:
      • If the block is a FOR FIRST/FOR LAST, then the NEXT is statically converted to a LEAVE.
      • If the block is a DO, the compiler errors.
  2. If there is no label, then the implicit target block must be found.  This is a 2 phase process.
    1. The nearest enclosing looping block type will be the target:
      • DO WHILE
      • DO TO
      • DO TO/WHILE
      • REPEAT (all forms)
      • FOR EACH (all forms)
      • EDITING
      • If one of these blocks is found, then the NEXT iteration of that block will be invoked.
    2. If there is no enclosing looping block between the NEXT statement and the nearest enclosing top-level block, then a search will be made to find the nearest enclosing FOR FIRST or FOR LAST block.  If one of these blocks is found, then the NEXTwill be statically converted to a LEAVE and targetted at that nearest enclosing FOR FIRST/LAST block.
  3. The 2nd phase of the implicit block search process is not attempted unless the first phase fails to find a match.  This means that a more deeply nested FOR FIRST/LAST block will NOT have preference to an outer looping block.  The looping blocks will always take precedence over the FOR FIRST/LAST.
  4. Please note that the simple DO block is the only inner block type that NEXT will NEVER implicitly target.
  5. The properties of the blocks have nothing to do with this search process.
  6. If no inner block is found with the above 2 phase process, then the NEXT is considered to be targetting the nearest top-level block.  For this reason, the NEXT converts to a RETURN.

Transactions

Progress 4GL transaction processing:
  1. Defines the start and end of transactions and sub-transactions.
  2. Implements the proper nesting of sub-transactions matching the block structuring of the program.
  3. Rolls back (UNDOes) database, variable, temp-table and work-table values to a known state upon certain "conditions" occurring.
  4. Commits all changes (permanently) to databases, variables, temp-tables and work-tables at the natural successful end of an active transaction. 
  5. Implements scoping facilities for reou
A transaction begins at the start of any block while the following are true:
A sub-transaction (a nested transaction) begins at the start of any block while the following are true:
Only 1 transaction can ever be active per user context.  Other nested blocks that have transaction properties, become nested sub-transactions.  For more details on which blocks can open a transaction, see page 12-17 of the Progress Programming Handbook.

Any block that has no direct access to the database cannot open a transaction (unless it is a block that has the TRANSACTION keyword).  Thus the same block type which would normally open a transaction when database updates or exclusive database reads are present, will not open a transaction (or in some cases a sub-transaction) when no database access is contained.

FOR block transaction anomaly: There is a case where reading a record in a FOR EACH with exclusive lock will not create a transaction.  In particular, if the buffer scope for the FOR EACH has been expanded beyond the FOR EACH block (there is a free reference outside that block) AND there is no data editing or other cause to create a transaction, then the FOR EACH will not create a transaction.  Also note that ALL buffers referenced (in the case where there are > 1 record phrases) must be scoped outside the FOR block in order for this anomaly to trigger.   If even 1 buffer (referenced in the FOR statement itself) is scoped to the FOR block itself, then the EXCLUSIVE-LOCK alone is enough to trigger this behavior.  When this case occurs, the FOR block is still treated as a sub-transaction.  See Progress Knowledge Base article KB13974.

Note that the rules defining a sub-transaction are a super-set of the rules defining a transaction.  Since a given block can be invoked with a call stack that is different depending on the path taken through the program, when a particular code block is invoked it is possible that sometimes a transaction will be active and other times not.  This means that the following cases are possible:
  1. The start of the block will start either a transaction or a sub-transaction, depending only on whether a transaction is already active.
  2. The start of the block will start a sub-transaction if a transaction is already active, but otherwise a transaction will not be started.
  3. The start of the block has no transaction or sub-transaction implications (notably any DO block without the ON ENDKEY, ON ERROR or TRANSACTION keywords).
To duplicate this behavior, the determination of whether a block opens a transaction or a sub-transaction must be handled at runtime.  Thus the same block will process differently depending on runtime conditions.

In this way, procedures are treated just like other block types.  Calls to procedures within an active transaction cause the procedure to open a sub-transaction.  If a procedure would not otherwise open a transaction and a transaction is not active, then a transaction is not started at the beginning of a procedure.  It is important to note that even though a procedure may not open a transaction, a nested block may still open a transaction.  For example, a nested FOR EACH that updates the database will have a transaction scoped to the FOR EACH block rather than to the containing procedure.

The end of any block that defines a transaction or sub-transaction defines the end of the corresponding transaction/subtransaction.  When a transaction scope ends successfully, all changes made during that scope are committed to the database.  When a sub-transaction scope ends successfully, all changes made during that scope are effectively added to the "transaction set".  This transaction set can be UNDOne as a set or committed as a set but once the end of a subtransaction occurs, one loses the ability to UNDO only that portion of the transaction.

In the event of any condition being raised, an UNDO (rollback) will occur (at either the transaction or sub-transaction level) and then a change of control flow to an enclosing scope will occur (to the same scope as the UNDO or to a scope that encloses the scope that was UNDOne.  The implicit or explicit definition of this behavior is encoded in the source application's block properties.  Each condition has a different set of default behavior in this regard.  For more details on determining the start and end of transactions, see page 12-8 of the Progress Programming Handbook

This ability to have nested sub-transactions allows blocks to be used to structure Progress 4GL programs such that the proper level of UNDO and other actions can be associated with the granularity or lack of granularity which the programmer desires.  This allows any active work (work that has not been added to the transaction set (defined above) to be rolled back partially (by UNDOing at the sub-transaction level) or for all work (the full transaction set) to be backed out (by UNDOing at the transaction level).

The sub-transaction commit point has the other behavior (in the case of a loop), that the "backup set" of all undoable resources (variables, database fields...) is updated at this same point.  This means that if a variable is defined external to a scope and the first iteration of a given sub-transaction loop updates its value and then commits, any this new value will be result of any UNDO that occurs (whether the variable was modified or not) until the next sub-transaction commit.  This can be described as updating the backup set at iteration.

Once a transaction is active, database fields, variables, temp-table and work-table values are saved off at the start of the block defining the transaction and again at the start of the block defining every sub-transaction.  One can UNDO all of these data structures to a known state by defining the block properties of each block such that the possible conditions implicitly UNDO the correct block OR the possible conditions are explicitly defined to UNDO a specific block.  In particular, labeled blocks can be specified as the target of an UNDO (or other action) by using the "ON <condition>" clause in a block.  If that named condition occurs, then the specified actions would be invoked on the specified blocks.

UNDO can ONLY EVER be done when a transaction is active.  The scope of the rollback is either explicitly defined by the programmer or is implicit, but in either case, each condition type matches up with a specific (and potentially different) rollback scope.  A rollback scope can be specified at any block that defines a transaction or sub-transaction.  When a transaction is not active, UNDO is a null operation.

There is no way to disable UNDO for a block, although individual local variables and temp-table/work-table fields can be specified as NO-UNDO, which will exclude them from the transaction rollback.  Database field changes are always rolled back.

Whether a block is a loop or not does not change how transaction processing works.  In fact, even a non-looping block can be retried.

Once the successful end of a transaction block is encountered, Progress commits all changes (permanently) to databases, variables, temp-tables and work-tables.  At the successful end of a sub-transaction, backup copies of the database fields (buffers), variables, temp-tables and work-tables are discarded.  A commit is a write of the modified value(s) back to the database or memory as necessary for the resource type being processed.  In case of the database, temp-tables and work-tables, this includes: When the natural flow of control of the procedure flows through an END statement for a block, this is considered the "successful end" of the block.  Please note that when LEAVE and RETURN are explicitly coded (and not triggered as a result of a CONDITION) then these are also considered the successful end of the containing block.  This means that all changes are committed in these cases.

Silent Error Mode (NO-ERROR keyword)

Progress provides the NO-ERROR keyword and a system handle called ERROR-STATUS to provide the ability to temporarily (for the duration of a single assignment or language statement) force a silent error mode.  In this mode, any generated errors will not raise a condition but will instead store the error number and error text in a list of errors.  Then the statement or assignment will silently return.

The ERROR-STATUS handle is used to access this stored data.

Language Statements Which Can Raise Conditions


Statement
 Condition
Notes
ERROR
 STOP
 QUIT
 ENDKEY
 END-ERROR*
APPLY +
 +
 +
 +
 +
 this statement can generate any event on a procedure including these special events that match "conditions" (it can also generate events for widgets but this is conceptually quite different from generating the conditions listed in this table)
ASSIGN +
 -
 -
 -
 -
BUFFER-COMPARE
 +
 -
 -
 -
 -
BUFFER-COPY
 +
 -
 -
 -
 -
CHOOSE +
 +
 -
 +
 +
 interactive
COMPILE
 +
 -
 -
 -
 -
 provided here for completeness
CONNECT
 +
 -
 -
 -
 -
COPY-LOB
 +
 -
 -
 -
 -
CREATE
 ALIAS
 automation object
 CALL
 DATABASE
 SAX-READER
 SERVER-SOCKET
 SOCKET
 WIDGET-POOL
 +
 -
 -
 -
 -
DDE
 +
 -
 -
 -
 -
 all kinds of DDE statement can raise ERROR
DELETE
 OBJECT
 PROCEDURE
 WIDGET-POOL
 +
 -
 -
 -
 -
DICTIONARY
 +
 -
 -
 -
 -
 "The DICTIONARY statement is equivalent to RUN dict.p: it runs the Progress procedure called dict.p. Progress uses the regular search rules to find the dictionary procedure. The dictionary procedure is part of the Progress system software."
DISCONNECT
 +
 -
 -
 -
 -
DISPLAY +
 -
 -
 -
 -
EDITING phrase +
 +
 +
 +
 +
 The event processing inside the phrase must apply all events explicitly, either using APPLY statement or by using RETURN ERROR/STOP/QUIT statements. Also, this phrase is active only if input is from terminal. In general case it is necessary to check phrase body to determine which events can be generated.
EMPTY TEMP-TABLE +
 -
 -
 -
 -
ENTRY
 +
 -
 -
 -
 -
 function and statement
EXPORT
 +
 -
 -
 -
 -
FIND +
 -
 -
 +
 -
IMPORT +
 -
 -
 +
 -
INPUT FROM +
 -
 -
 +
 +
INSERT +
 +
 -
 +
 +
 interactive
LOAD
 +
 -
 -
 -
 -
OUTPUT TO +
 -
 -
 -
 -
PROCESS EVENTS +
 +
 -
 +
 +
 behaves like interactive
PROMPT-FOR +
 +
 -
 +
 +
 interactive (equivalent to ENABLE, WAIT-FOR, DISABLE sequence of statements)
PUT-KEY-VALUE
 +
 -
 -
 -
 -
QUIT
 -
 -
 +
 -
 -
 this statement seems the only way to generate this event (except APPLY)
RAW-TRANSFER
 +
 -
 -
 -
 -
READKEY
 +
 -
 -
 +
 -
 Actually, the condition is not directly raised, instead the LASTKEY value is set to -1 (on ERROR) or -2 (on ENDKEY).  To raise the condition one then must "APPLY LASTKEY".
RELEASE
 +
 -
 -
 -
 -
 all kinds of RELEASE statement can raise ERROR
REPOSITION
 +
 -
 -
 -
 -
RUN
 +
 +
 -

all kinds of RUN statement can raise ERROR and STOP
SAVE CACHE +
 -
 -
 -
 -
SET +
 +
 -
 +
 +
 interactive (equivalent to PROMPT-FOR and ASSIGN)
STOP
 -
 +
 -
 -
 -
SUBSCRIBE
 +
 -
 -
 -
 -
UNLOAD
 +
 -
 -
 -
 -
UPDATE +
 +
 -
 +
 +
 interactive
USE
 +
 -
 -
 -
 -
VALIDATE
 +
 -
 -
 -
 -
WAIT-FOR +
 +
 -
 +
 +
 Waits for specified event to occur. Effectively this does allow other keyboard-driven events to occur as well.

Notes:
  1. END-ERROR is a key function that behaves as two separate events (see 5-19 in the Progress Programming Handbook): for the first input operation (any of the Language Statements that Block for User Input) of the current block's iteration (if it is a loop), raises the ENDKEY condition; for subsequent input operations, raise the ERROR condition.  Note that in a WAIT-FOR, the resulting condition is always ENDKEY.  In UPDATE/SET/PROMPT-FOR... (and for non-WAIT-FOR stmts), it differs based on whether this is the first or a subsequent interaction in this *iteration* of the block.  For example, in a REPEAT block that has 2 SET statements, on any given iteration of the block if you press the END-ERROR key while in the 1st SET statement the result is always ENDKEY condition and if you press the END-ERROR key in the 2nd SET statement, the result is the ERROR condition.
  2. Inner loops may have ON ERROR, ON ENDKEY, etc. phrases which refer to outer loop using appropriate labels.
  3. The default behavior if the ON ENDKEY phrase is omitted is UNDO, LEAVE. If ON ENDKEY phrase is partially specified then default is to LEAVE if other (NEXT, RETRY, or RETURN) action is not explicitly set.
  4. RETURN ERROR raises ERROR condition in calling procedure.
  5. The default behavior if the ON ERROR phase is omitted for REPEAT and FOR EACH blocks - UNDO, RETRY. If Progress detects possible infinite loop in FOR or iterating DO blocks then UNDO, NEXT is performed instead.
  6. It is unclear if UNSUBSCRIBE supports NO-ERROR option. Syntax diagram does not mention it but further description mentions it as a possible way to avoid raising ERROR.
  7. From the description it is not clear if the DICTIONARY is an equivalent to "RUN dict.p" or "RUN dict.p NO-ERROR".
  8. Quote from the reference: If you use READKEY, it intercepts any input from the user. Thus no widgets receive input. To pass the input to a widget, you must use the APPLY statement.  READKEY and widget usage are usually kept separate.  READKEY is normally used as part of the editing block functionality while widget events are part of the wait-for event driven functionality.  As noted above with the APPLY statement it is important to realize that widgets and events are a very different category of discussion than these block conditions.
  9. UNDO statement does not raise conditions by itself but may change execution flow if followed by LEAVE, NEXT, RETRY or RETURN. Also, in form UNDO, RETURN ERROR is may raise ERROR condition in the calling procedure (like RETURN ERROR).
  10. The default ENDKEY and ERROR processing is defined only for REPEAT and FOR loops, other loops must specify such processing explicitly using ON ERROR and ON ENDKEY phrases.

Language Statements that Block for User Input

The presence of language statements that block for user input modifies or controls the behavior of the END-ERROR key and of the infinite loop protection (ILP) for RETRY.

The following language statements do block for user input:
  1. CHOOSE
  2. ENABLE
  3. INSERT
  4. MESSAGE (only with VIEW-AS ALERT-BOX, UPDATE or SET)
  5. PROMPT-FOR
  6. READKEY on terminal (but not when reading from a file)
  7. SET
  8. UPDATE
  9. WAIT-FOR
  10. PAUSE (waits for "ANY-KEY") - Note that this statement is a special case in that it blocks for user input, but only sometimes does it  disable infinite loop protection.  For examples of this behavior, see testcases pause_infinite_loop_protection*.p.
There is a special case for the PAUSE statement.  Whether PAUSE does or does not disable infinite loop protection, is dependent upon the block type:
Perhaps surprising is that the following do not block for user-input:
  1. APPLY
  2. HIDE
  3. PROCESS-EVENTS
  4. VIEW

Language Statements that Update/Modify the Database

Absent the TRANSACTION keyword, only the presence of database updates or exclusive reads will cause a block to start a transaction.  The following language statements start a transaction, only if they reference a database table/field (after a record buffer has been created or filled).  If temp-table, work-table or normal variables are used, then a transaction is not started.  In addition, some of these statements can take a record as an option (as a shorthand way of referencing all fields in that record).  If this record is a TABLE, this will trigger a transaction.
  1. ASSIGN
  2. BUFFER-COPY
  3. CREATE
  4. DELETE
  5. INSERT
  6. RAW-TRANSFER
  7. SET
  8. UPDATE
  9. Explicit field-level use of the assignment operator on database records will start a transaction.
Note that the assignment operator can only be used without error on database records that have SHARE-LOCK or EXCLUSIVE-LOCK.  If the record is opened with NO-LOCK, then an ERROR will be generated.  The NO-ERROR keyword can be used on an assignment to suppress the error generation (the fact that it occurred can still be discovered via the system handle ERROR-STATUS) but this doesn't make the update successful.  In other words, one cannot successfully assign to a database record that is NO-LOCK but this still DOES mark the associated block as a transaction.

Usage of a BROWSE widget  (see DEFINE BROWSE) is a "black-box" language statement which reads from/writes to the database on behalf of the programmer.  Any BROWSE that uses the ENABLE keyword is not read-only (in Progress terms it is "updateable") and it will update/modify the database. All transaction support for updateable BROWSE is hidden inside the "black-box".  For this reason, the transaction property does not attach to the containing block.
It is not known at this time if statements such as SETUSERID implicitly access the database on the caller's behalf.  There may be other such language statements that read or update the database explicitly.

Language Statements that Read the Database with EXCLUSIVE LOCK

Absent the TRANSACTION keyword, only the presence of database updates or exclusive reads will cause a block to start a transaction.  The following language statements read from the database into a record buffer.  A lock can be exclusive explicitly with the EXCLUSIVE-LOCK keyword.
  1. FIND
  2. GET (with an explicit EXCLUSIVE-LOCK OR with no lock qualifiers when referencing a query with EXCLUSIVE-LOCK)
  3. FOR (all forms - EACH, FIRST, LAST and unqualified record phrase)
If temp-table or work-table records are read, then a transaction is not started.  Only exclusive database reads cause a transaction.

Usage of a BROWSE widget  (see DEFINE BROWSE) is a "black-box" language statement which reads from/writes to the database on behalf of the programmer.  BROWSE widgets are read-only when there is no use of the ENABLE keyword (in Progress terms it is not "updateable") and these can generate reads with EXCLUSIVE-LOCK (this can be set in the DEFINE BROWSE statement).  If the DEFINE BROWSE does not explicitly use the EXCLUSIVE-LOCK keyword, then even if the query was opened EXCLUSIVE-LOCK this is overridden to NO-LOCK (by default) or SHARE-LOCK (if specified).  So only a browse defined explicitly with EXCLUSIVE-LOCK will open the query as EXCLUSIVE-LOCK.  However, since such an approach is only allowed inside a block which already has transaction support (it must be placed inside a DO TRANSACTION or REPEAT TRANSACTION), the containing block will already have transaction support.  For this reason, browse usage can be ignored.

The REPOSITION statement fetches a record when the query is associated with a BROWSE widget.  Since the transaction processing associated with a browse in hidden inside the browse's "black-box", so also is any effect of the hidden/implicit record reading that occurs with this statement.  In other words, although this statement can have an implicit record read associated, there is no case where it causes the containing block to become a transaction.

It is not known at this time if statements such as SETUSERID implicitly access the database on the caller's behalf.  There may be other such language statements that read or update the database explicitly.   It is likely that such statements handle transactions internally (as a black-blox rather than associating a transaction with the calling block).

High Level Implementation Options

Many of the features that Progress provides relate to the proper flow of control in the resulting application.  While problems such as data processing fidelity were relatively easy to centralize or hide inside the wrapper classes , features related to flow of control are not easily centralized.   This is due to the fact that the all control flow constructs in Java cannot be split across two or more methods.  For example, one cannot centrally define catch () blocks that are then used in multiple places.  There must be a catch block in every method in which the matching logic must be triggered.  While one can take the user-defined logic and move that into methods that are called from many locations, all flow control constructs are built-in language features and they are essentially monolithic in the Java language.

The Progress feature set is a complex one with much implicit behavior that must be duplicated.  In many cases, this means that there will be more code in Java where no such code appears in the Progress source.  This has support, maintenance and conversion implications (the greater the difference in source and target tree structure, the greater the effort to convert).  For this reason, one must consider how to obtain complete fidelity while centralizing and hiding as much logic as possible in utility classes, leaving the generated source cleaner and more natural to maintain/support.

The following are the most feasible implementation options to provide complete fidelity with the Progress language/runtime feature set:

Option
 Description
 Pros
 Cons
explicit hard coding in generated source
 All control flow, logic and data processing is explicit in the generated source code.  Some features can be helped by hiding things like undo processing in the data type wrappers or utility classes like streams (with cleanup processing).  However, the core problem here is that the generated code still must hook into this processing.  This will take the form of try/catch/finally blocks, instrumentation calls to push/pop scopes, manufactured labels/bogus outer loops to handle things like retry and call-backs to utility classes to access or invoke that logic that can be centralized.
  1. Simple to understand and implement.
  2. Easiest to debug.
  3. Good performance (considering what has to be done).
  4. Most independent of external tools.
  1. Source code is most verbose of all options.
  2. Since most code is affected by the implicit features, this extra code will be in many/most locations.
  3. More effort later to convert any generated code to a more native Java approach.
indirect call mechanism using a utility class to implement the logic In this model, each block that has special properties would be implemented in 2 pieces:
  1. An "external" interface that exposes a well-known named method and interface.  This method would use reflection to obtain a Method object, it would package its parameters as an Object[] and invoke a special BlockProperties class that in turn would use the Method and Object[] to indirectly invoke the private worker.  If the return was not void, the Object would be cast to the proper type and then returned to the real caller.
  2. A private worker method that actually contains the converted logic.  This would be invoked via reflection from the external interface method.
The point here is that the block properties would not be emitted into the generated source, but would be centralized in the BlockProperties class.  Since the block properties class is just invoking the private worker as a method call, this method call can be inside a hand coded try/catch/finally with all special Progress-compatible logic completely hidden in this central location.
  1. Strong level of centralization.
  2. Source code is reasonably clean (but not completely clean) compared to the hard coded approach above.
  1. External interface methods have to be created for each public interface, for each internal procedure or function and for each block.  Thus the source code will still look different from the original and from how one would want to do it if manually coding.
  2. Harder to debug than the hard coded approach above.
  3. DO/REPEAT/FOR blocks would have to be refactored into methods which makes this extremely complicated (since all data must be either promoted to instance members, passed as parameters or encapsulated into another class and passed down.  This is made harder because anything nested inside eachblock has to have its data-access requirements "rolled up" into the containing block.
  4. Using reflection is a performance penalty, especially in the Mehtod object lookup (although the resulting Method could be cached in a static member).
bytecode instrumentation Generate the Java code just as one would normally write it by hand coding but after javac creates the resulting class files, rewrite the class file (statically as a 2nd compile step or dynamically at load time via a custom classloader) to instrument all the same logic and processing that would have been implemented explicitly in the source as in the hard coded option above.
  1. Cleanest source code.
  2. Easiest to convert away from Progress semantics and to a more natural Java approach.
  3. Performance is as good as the hard coded approach.
  1. Hardest to debug since there is truly hidden processing (only in the byte code and not in the source).
  2. Editing this code will likely require modifying some form of additional input (a second hints/annotations file or possibly some form of inline annotations) in order to specify the block properties etc... that must be applied (since it is not implicit in the bytecode) and the locations to which to apply this instrumentation. This makes maintenance harder than just editing a single source file.

At this time, the hard coded source approach is the simplest and easiest to achieve.  As a likely future optimization, the bytecode instrumentation approach will probably be provided in the future as an option.

Implementation Details

As mentioned above, Progress 4GL implements a great deal of its transaction and block behavior in an implicit manner.  One of the advantages Progress has in its interpreter is the embedding of knowledge about each block open and close and each loop iteration.

All Progress conditions will be mapped to custom Java exceptions.  This will allow the generation of such conditions to unwind the stack and to be caught/processed/rethrown where needed.  The base class will be com.goldencode.p2j.util.ConditionException which will allow for a common catch {} when this is useful.  Each of the 4 conditions will be mapped to a subclass:

Condition
 Class
ERROR com.goldencode.p2j.util.ErrorConditionException
ENDKEY com.goldencode.p2j.util.EndConditionException
STOP
 com.goldencode.p2j.util.StopConditionException
QUIT
 com.goldencode.p2j.util.QuitConditionException

In the Java implementation, a TransactionManager class provides a central clearinghouse that tracks block open, close and loop iteration.  Transaction services will be provided by the TransactionManager based on method calls emitted at the start and end of each block, and when any loop iterates.  These hooks will provide the same level of processing that can be implemented internally within the Progress 4GL interpreter.

All blocks and loops which have properties will be emitted inside a try/finally pair.  The TransactionManager pushScope() will be called first.  Then the try {} will surround the loop or block itself.  The finally {} provides the ability to hook the TransactionManager popScope() method call.  This ensures that the scope will be properly managed no matter how we exit the block (via exception, natural block exit or return).

Inside the outer try block, there will be any loop control processing needed and the opening of the block itself.  This block will contain a special do { } while () loop to implement retry logic.  Inside this do while loop is another try {} block and a set of catch {} blocks to handle condition processing.  The actual block or loop body is emitted inside this try block.

Hooks and queries into the Transaction Manager allow it to handle much of the processing automatically, however all Java statements which affect flow control must be emitted into the source.  This is a requirement of the Java language as the flow control statements are converted into byte code (instructions) which are inherently local to the current method.  The Java instruction set (all possible byte codes) only provides the ability to handle local branching except for method calls, method returns and exception processing that can arbitrarily unwind the stack.

The following is a sample of the emitted code:

      TransactionManager.pushScope("label",
                                   TransactionManager.TRANSACTION,
                                   true,
                                   true,
                                    false,
                                   false);
     
      try
      {
         label:
         // if this is a loop, the loop control structure will emit here
         {
            TransactionManager.blockSetup();
             do
            {
               try
               {
                  // block body is emitted here
               }
              
               catch (ErrorConditionException err)
               {
                  TransactionManager.honorError(err);
                  TransactionManager.rollback("label");
                  TransactionManager.triggerRetry(null);
               }
              
               catch (EndConditionException end)
               {
                  TransactionManager.rollback("label");
                  break label;
               }
              
               catch (RetryUnwindException ru)
               {
                  TransactionManager.honorRetry(ru);
               }
              
            }
            while (TransactionManager.needsRetry());
           
            if (TransactionManager.isBreakPending())
            {
               break label;
            }
           
            TransactionManager.iterate();
         }
      }
     
      finally
      {
         TransactionManager.popScope();
      }

The TransactionManager provides the following services via static methods that are context-local to the current user's context:
  1. Maintains the state of each block, tracks start and end and handles nesting of blocks.  TransactionManager.pushScope() and TransactionManager.popScope() are used to handle this maintenance.  Note that the popScope() is implemented inside a finally {} to ensure that it always gets called.  TransactionManager.iterate() is used to allow the Transaction Manager to implement state management for loops.
  2. UNDO support:
    • When variables and record buffers (database, temp-tables and work-tables) are defined, they register with the TransactionManager as undoable unless the Progress source definition was NO-UNDO.  All such Java objects must implement the Undoable Interface.  This interface provides the ability to clone (backup) and assign back (undo) the state of an Undoable object.
    • To obtain rollback services, an object must be added to the undoable list via TransactionManager.register().
    • At every transaction and sub-transaction start, any Java objects registered in the undo list will be cloned and stored in a map that is unique to the block being opened.  The key for each copy will be the reference to the object being copied.  A static method (TransactionManager.makeBackup())  is called just inside each block, after any loop control variables are incremented or initialized.  This is required because of how Progress implements its backup set, which occurs after loop control variable changes are made.  Otherwise, this processing would have been integrated into TransactionManager.pushScope() and TransactionManager.iterate().
    • Rollback support for variables and record buffers via TransactionManager.triggerRollback() which can be called by user code (a conversion of the UNDO language statement) or in response to the condition processing (a catch block).
    • At the successful completion of every transaction and sub-transaction, all registered undoable objects have their changes committed.  In terms of the database, this means that the changes are added to the Hibernate session.  Due to nesting, it is possible to partially commit changes but then to backout all changes at the transaction level.  This works for all types of undoable objects, although the database changes are maintained differently because of Hibernate.  At this time only the database fields implement the special Commitable interface and get a call to commit(boolean transaction) upon each transaction or sub-transaction.  Variables don't require such support since they naturally commit at the end of each block and instead they only get rolled back on a failure, otherwise the backup sets are just garbage collected when they go out of scope.  This processing can be called from either TransactionManager.iterate() or TransactionManager.popScope() depending on the method of successful exit from the block.
    • To obtain commit services, an object must be added to the commitable list via TransactionManager.register() or TransactionManager.registerCommit().
  3. RETRY state management/infinite loop protection:
    • An extra do {} while () loop encloses the block body in the emitted code.
    • The transaction manager associates a BlockDefinition object with each block and maintains the proper nesting.  Each such definition has retry related state variables.
    • The first time through any loop or block, the state is initialized to disable retry and enable infinite loop protection.
    • During any condition processing (a catch block), if retry was implicitly or explicitly defined in the original Progress source, then retry would be enabled for that associated block.  The TransactionManager.triggerRetry() is called to set the state (and possibly throw an exception).
    • If a retry is requested for a block that is not the current block, the RetryUnwindException is thrown to unwind the stack to the proper block.  Each block catches this exception and tests if this is the target for the retry.  If not, the exception is rethrown.  When the proper block is reached, the retry flag is set and normal processing of the do {} while () loop continues.  The TransactionManager.honorRetry() is called in each catch block to implement this logic.
    • Every iteration of a loop reset the retry state such that it is as if we are starting fresh.
    • The do {} while () loop calls TransactionManager.needsRetry() to determine whether or not it needs to loop.
    • needsRetry() implements infinite loop protection (in the case where a retry is requested, it is sometimes converted into another operation).
    • Input blocking language statements notify the TransactionManager such that the current scope's infinite loop protection can be disabled.  Similarly, the replacement for the RETRY built-in function is backed by a TransactionManager method isRetry().  Before this method returns its value, the current scope's infinite loop protection is disabled (see TransactionManager.disableLoopProtection()).
  4. RETURN ERROR state/processing:
    • Normal ERROR conditions are thrown "inline", wherever the problem occurs.
    • This language statement must bypass normal ERROR condition processing and instead should be processed in the caller of the current method, no matter how deeply nested the block is in which this is thrown.
    • To throw this, one calls TransactionManager.triggerErrorInCaller().
    • A special ignore flag is kept to track if this special mode is being used.  It is checked using To throw this, one calls TransactionManager.honorError() in catch blocks which process the ErrorConditionException and if this returns silently, then normal error processing is honored.  Otherwise it will rethrow the exception, propagating it upwards.  This results in unwinding the stack to the top-level block of the method.  At this point, the exception is thrown once more but first the ignore flag is cleared.  The caller will then catch and process this error.
  5. Provides a hook for any object to get a callback on exit from the scope in which it is defined.  This provides an easy method for implementing cleanup processing.  All objects needing this support must be registered using TransactionManager.registerFinish() and must implement the Finalizable interface.
  6. Provides a hook for any object that requires notification of all block entrances and block exits.  This is provided by the Scopeable interface and such objects are registered via the TransactionManager.registerScopeable().
  7. Provides a session-level batch start and stop notification service.
  8. At every block start, stop and iteration the current thread's interrupted status is checked and if the thread has been previously interrupted, a StopConditionException will be thrown.  See TransactionManager.honorStopCondition().
All blocks will be labeled, with a manufactured label or one that comes from the original Progress source.  This facilitates condition processing, since processing that was previously implicit is now being made explicit.  This is also important since a simple block in Progress is implemented as a set of related blocks in Java, in order to always be sure that the flow control statement references the correct block, it is made explicit.

Silent error processing is provided using the ErrorManager class.  This class:
  1. Stores the state of whether errors should or should not be thrown.
  2. Tracks if an error has been thrown and the list of error text and numbers.
  3. Allows access to this data.
  4. Provides helpers to throw errors or record the error if silent mode is enabled.
Lower level interfaces are also provided to allow more control within a library, especially in the case where one must retain control over where to throw the error due to the inability to silently return from the location at which the error was encountered.  The ErrorUnwindException is provided to facilitate this requirement.

All data type wrappers, expression operator implementations and the runtime backing for built-in functions have been modified to be aware of the ErrorManager and silent error mode.  As a rule, where an RUNTIME errorn would normally be generated in Progress source, the P2J runtime will use the ErrorManager to honor the silent error mode.  COMPILE errors are not handled at all since no such errors should occur in the converted environment.  Note that there may be possibilities of runtime issues in Java that are compile-time issues in Progress.  At this time, such cases will throw Java exceptions such as IllegalArgumentException rather than attempt to duplicate Progress COMPILE time error strings.

Java has no undo or retry functionality.  This means that Progress and Java are mismatched in these areas.  For this reason, the vast majority of the processing in the Transaction Manager is related to one of these two areas.

See also:

TransactionManager
BlockDefinition
ErrorManager
ErrorUnwindException
Undoable
Commitable
Finalizable
Scopeable
ConditionException
RetryUnwindException
ContextLocal

rules/annotations/block_properties.rules
rules/annotations/leave_fixups.rules
rules/annotations/labels.rules
rules/convert/control_flow.rules
rules/convert/main_method.rules
rules/convert/java_template.tpl

Honoring Stop Conditions

Stop conditions can be generated by certain runtime errors such as a database disconnection or failures in stream processing.  Any runtime code running on a thread which is servicing converted Progress business logic will translate any InterruptedException into a StopConditionException.   Classes which provide such a service:
There are unusual cases in the runtime where processing is not occurring on a thread which is servicing converted Progress business logic.  For example, if a secondary thread is used to cleanup filesystem resource (e.g. close pipes or files) or if a secondary thread is used to copy data between two pipes.  Note that in these cases, there is no counterpart or proper mapping into a Progress semantic since such cases don't exist in the Progress runtime.  Generally, such cases silently exit or fail.  Classes which provide such behavior:
In addition, the STOP language statement in Progress allows an explicit stop condition to be raised.  In this case the business logic will explicitly throw a StopConditionException.

Stop condition processing is complicated by the Progress feature that the user may press the CTRL-C key combination at any time and this immediately interrupts the execution of the Progress program (which is single threaded).  In the P2J environment this is complicated by the split of UI processing into a separate process accessed via the network.  The UI client will monitor for the CTRL-C key combination and this will cause asynchronous processing of the interruption to occur.  In particular, if processing is currently occurring on the client side, the interruption will be converted to an interruption of that running thread which will be converted to a StopConditionException in the UI client.  This will then unwind the stack normally until the location in the business logic that handles (catches) the stop condition is reached.  If the current thread is processing on the server (such that the client cannot honor the UI since it is not actively running), then the server will be asynchronously notified and this will trigger a call to Thread.interrupt().  If the currently processing thread on the server (for that user's context) is blocked, an InterruptedException would be generated and this would be caught by the runtime as noted above and translated into a StopConditionException.  This is a key point: J2SE only generates an InterruptedException in specific cases where a thread blocks such as Object.wait() or Thread.sleep().  If the thread is not blocked, then J2SE will not generate an InterruptedException, instead it just sets the interrupt status of the thread.  The TransactionManager checks the interrupted status of the current thread at block start, stop and when a block iterates.  Any previous interruption will be honored by throwing a StopConditionException.  Please see above for more details on the TransactionManager.

Accumulator behavior

Contents

  1. ACCUMULATE statement
  2. DISPLAY with aggregate phrase
  3. ACCUM function
  4. Aggregator types
  5. Accumulator expressions
  6. Accumulator usage
  7. Initial (entry) value for accumulators
  8. Block-behavior for accumulators
  9. ACCUM scope computation
  10. Reset behavior for accumulators with simple blocks
  11. Reset behavior for accumulators with IF statement
  12. The "backup" mechanism
  13. ACCUM with nested RUN/TRIGGER statements

TBD

ACCUMULATE statement

The syntax of the ACCUMULATE statement is:

ACCUM {expression (aggregate-phrase)}

The following rules about ACCUM stmt must be taken into consideration:
- When converting this statement, it must be taken into consideration that it may contain more then one accumulation expression and also more then one aggregators for a certain expression. This knowledge is used by the annotations/accumulate.rules file to find the correct expression for a certain accumulator - in the post-rules section, it will find and set the scope-id for the expression to the computed id for the current accumulator.
- When aggregate-phrase contains more then one aggregator, this will trigger generation of an accumulator instance for each combination of (expression, aggregator).
- When the accumulation is performed while iterating a FOR EACH block with a BREAK BY option, the accumulation may be computed for each BREAK BY group, if the "BY break-group" is specified in the aggregate-phrase.

DISPLAY with aggregate phrase

The DISPLAY with aggregate phrase affects the accumulators in the following ways:
- on conversion, the accumulator must be scoped at least to the same block as the frame used in the display statement
- after the block is ended, the display statement will display a cumulative accumulation result
- if a sub-only with a BY group is used, then it will display the results for each BY value.

When an accumulator is used with a DISPLAY statement and with a ACCUM statement, then it must not emit an implicit c'tor; the rule is that, when the accumulator is used with an ACCUM statement, the ACCUM statement will be used to emit the c'tor - any DISPLAY with accum references will be ignored. The default c'tor (except COUNT and SUB-COUNT, which always use default c'tor) will be generated only when there are DISPLAY with accum references and there is no ACCUM statement references, for a certain accumulator. Also, the Accumulator.accumulate(BaseDataType) must not be restricted to a ExternalDataSource, to allow the ACCUM and DISPLAY statements to work together.

There are a few special cases, when an accumulator is used with a DISPLAY statement: During this document, there may be references to the DISPLAY accum statement or DISPLAY statement - this will always mean that the author is referring to the "DISPLAY with aggregator phrase" statement.

ACCUM function

The ACCUM function is used to retrieve the accumulation result at a certain time. This does not affect the accumulation result and will affect only the part where the scope-id is computed: the found block must enclose all accumulator references.

Aggregator types

The PROGRESS accumulator types are:

Aggregator long name Short name
AVERAGE AVG
COUNT -
TOTAL -
MINIMUM MIN
MAXIMUM MAX
SUB-AVERAGE -
SUB-COUNT -
SUB-TOTAL -
SUB-MINIMUM SUB-MIN
SUB-MAXIMUM SUB-MAX

For all accumulator types it can be specified a "BY break-group" option. When specifying such an option, the accumulation will be performed for each distinct break-group value; these values will be available for retrieving using the ACCUM function only in the current or an enclosed child block. After leaving the block which has the "BREAK BY group" option, the accumulation values for each break-group are no longer accessible.
The sub-only accumulators will behave the same as normal accumulators, if they have no break-by option attached.
If a "By group" is present and the accumulator is used with a DISPLAY stmt, the sub-only accumulators will display the accumulation result for each "BY" value; for normal accumulators, the DISPLAY stmt will display only a final result.

Accumulator expressions

Accumulator expressions can be cataloged as:
1. simple expressions: a variable/table field name
2. literals - a constant expressions
3. complex expressions
The accumulation performs the same for all expression types: the expression will be evaluated (if needed) and will be added to the primary result and sub-results, if needed.

Accumulator usage

In PROGRESS, the accumulator is identified by its expression, being aware of the operands' order. For example, "i + 0" and "0 + i" will represent different accumulators.
So, if the expression must be used to identify unique accumulators, the following rules are used at conversion time to compute the accumulator name:
- if a DISPLAY statement with a aggregator phrase is encountered, then the accumulation expression is checked with the global expr-to-name map (named "accumNames") which holds the computed java names (for each expression) extracted from the ACCUM statements and the global expr-to-name map (named "dispNames") which holds the computed java names for all accumulators used in DISPLAY with aggregator phrase stmts. If the expression is found in either the accumNames or dispNames maps, then the found name is returned. Else, a name like "accumDisp#"(where # holds for an unique index) is computed
- if an ACCUM statement is encountered, then the dispNames and accumNames maps will be checked if they contain the accumulator expression. If the expression is found, then the retrieved name is used; else, a name like 'accumType#' will be returned, where "type" represents the short-name for the accumulator (without the "sub-" prefix).
- if an ACCUM function is used, then the accumNames is first checked (to see if the accumulator was used with an ACCUM statement) and also it computes the default label and the type for this expression; after this, if the dispNames map contains the accum expression, then it will override the java name.

Initial (entry) value for accumulators

All accumulators start with its value set to PROGRESS unknown (?) value. This value will be maintained until a BLOCK is entered (the BLOCK header is executed), where the accumulation value will be set to:

Aggregator AVERAGE COUNT TOTAL MINIMUM MAXIMUM SUB-AVERAGE SUB-COUNT SUB-TOTAL SUB-MINIMUM SUB-MAXIMUM
Value 0 0 0 ? ? 0 0 0 ? ?

More about which blocks reset the accumulators and how this reset is done can be found in sections Reset behavior for accumulators with simple blocks and Reset behavior for accumulators with IF stmt.

Block-behavior for accumulators

The block behavior for accumulators affects how the accumulation is executed and also what value is retrieved when the ACCUM function is used. To implement this, the accumulators must accumulate their values into two distinct data bundles:
- the 'pending' accumulation data: when the accumulation is executed, the data must be accumulated to all parent blocks; so, the accumulation is done to all data found in the pending stack. This pending data is used when the accumulator exists within a certain block: the parent block will be updated with the pending accumulation value from the child block.
- the 'reported' accumulation stack (the bundleStack): for each block, there will be kept a set with all the eventually accumulated data for each break-group value and also the primary data bundle. When the accumulation is performed for this block, the primary data bundle and the break-group data - if needed - are updated
- the sub-only property: the accumulator will be marked as sub-only when it enters the block; this property is distinct for each block
- the accumulator must have knowledge that it enters a block which encloses a DISPLAY with aggregate phrase/ACCUM stmt for this accumulator, which may be executed; so, upon entry of a block which uses this accumulator, an Accumulator.reset() call is executed - this will set the "usage" flag to all parent blocks to true and also will reset the primary data bundle to its entry value.

Using the ACCUM function to retrieve the accumulated result presents a few usage scenarios:
1. Using the ACCUM function right after the ACCUM stmt has been executed: 
repeat: 
   ...
   accum i (count).
   message accum count i.
   ...
end.
- the retrieved accumulation result will be the total accumulation result for this block (which is kept in the 'bundleStack').

2. Using the ACCUM function right after the a block which accumulated some data is executed: 
B1:
repeat: 
   ...
   B2:
   repeat:
      ...
      accum i (count).
      ...
   end.
   message accum count i.
   ...
end.
- the retrieved accumulation result will be the accumulation result for block B2. If the body for block B2 did not execute, then the accumulation result will be the default "entry" value. The rule extracted from this example is that, when leaving a block which accumulated some data, it will set the 'reporting' data bundle (the primary data bundle from dataBundle stack) for the parent block (block B1) to be the total accumulated value (from the pendingStack) for block B2. In other words, block B1 will inherit and report the accumulation result from block B2, until a new ACCUM stmt is executed or another block is entered.

Other special cases about the ACCUM function will be presented in the following "reset" and "backup" sections.

The PROGRESS statements which present the above behavior are REPEAT, DO [TRANSACTION/ON END-KEY UNDO, LEAVE], FOR EACH/LAST/FIRST, OPEN QUERY ... DO WHILE AVAILABLE.
The editing blocks (PROMPT-FOR/UPDATE/SET statements) do not present special behavior when accumulators are used, so accumulator scope notifications will be ignored for these blocks. Also, DO statements (which have no transaction support) do not affect accumulators in any way.

ACCUM scope computation

A certain accumulator may be scoped to a certain block or may be promoted to an instance field, following those rules:
1. The computed block to which the accumulator will be scoped (and also the accumulator) presents these properties:
- it encloses all references for this accumulator
- the accumulator is not used with a PUT or FORM statement
- the block presents transaction managenment (transaction level annotation (trans_level) is not equal to prog.no_transaction)
- all frames used by a DISPLAY with aggregator phrase statement are enclosed within this block
- it is not the root block
2. In all other cases, it will be promoted to instance field

Also, if the same accumulation expression is used within a procedure, function or a trigger, after conversion will result in 3 different accumulators.
The computed accumulator scope is also used by the accumulator expression - when the accumulator expression is instantiated (field reference, literal, etc) it must be promoted or instantiated in the same block as the accumulator.

Reset behavior for accumulators with simple blocks

Accumulators present a special behavior when a block which encloses an ACCUM statement is not executed or the block is executed and the ACCUM statement is skipped. Before continuing, it must be noted that this behavior is presented with all aggregator types, with all REPEAT, DO [TRANSACTION/ON ...], FOR EACH/LAST/FIRST, OPEN QUERY ... DO WHILE ... statements and with both ACCUMULATE statement and DISPLAY with aggregate phrase statement. For this, the following testcases are used to determine the rules:
1. a block which does not use the ACCUM stmt is not executed 
B1:
repeat k = 1 to 2:
   accum i (total).
   display accum total i with down column 0 frame f1.
   B2:
   repeat while false: /* no accumulator usage */
      ...
   end.
   display accum total i with down column 13 frame f2.
end.
Frames f1 and f2 will display the same value. This testcase shows that blocks which do not use the ACCUM statement for a certain accumulator will not affect it in any way. This behavior is also present if B2 has no ACCUM stmt and is executed. Also, any ACCUM function usage in block B2 will return the same value as if it was executed in block B1, as proved in the following testcase: 
B1:
repeat k = 1 to 2:
   accum i (total).
   display accum total i with down column 0 frame f1.
   B2:
   repeat j = 1 to 1: /* no accum stmt usage */
      display accum total i with down column 13 frame f2.
   end.
   display accum total i with down column 26 frame f2.
end.

This translates into: the accumulator must not set the "reported" bundle for block B1 to be the "pending" bundle computed in block B2, when B2 is leaved, because block B2 does not have any ACCUM stmts for this accumulator. Also, when executing a block which does not use the ACCUM statement, the accumulator instance must ignore any "scopeStart", "scopeFinished", "iterate", "retry" and "finished" calls.

2. a block which uses the ACCUM stmt is not executed 
B1:
repeat k = 1 to 2:
   accum i (total).
   display accum total i with down column 0 frame f1.
   B2:
   repeat while false:
      accum i (total).
   end.
   display accum total i with down column 13 frame f2.
end.
In this example, block B2 is not executed, but the block uses the ACCUM stmt; frames f1 and f2 will display different results:
- frame f1 will display the accumulated value for block B1
- frame f2 will display the "reset" value (0 in this case)
This is caused by the fact that, when executing the header for block B2, the accumulator is notified that it enters a block which may execute accumulation; so, the accumulator will be reset and also, when leaving block B2, it will set B1's "reported" bundle to be the "pending" (currently reset) bundle for block B2.

3. a block which uses the ACCUM stmt is executed, but the ACCUM stmt is skipped 
B1:
repeat k = 1 to 2:
   accum i (total).
   display accum total i with down column 0 frame f1.
   B2:
   do transaction:
      if false then
         accum i (total).
   end.
   display accum total i with down column 13 frame f2.
end.
In this example, block B2 is executed, but the ACCUM stmt is not; frames f1 and f2 will also display different results:
- frame f1 will display the accumulated value for block B1
- frame f2 will display the "reset" value (0 in this case)
This is caused by the fact that, when executing the header for block B2, the accumulator is notified that it enters a block which may execute accumulation; so, the accumulator will be reset and also, when leaving block B2, it will set B1's "reported" bundle to be the "pending" (currently reset) bundle for block B2.

4. a block scoped to procedure block which does not use the ACCUM stmt is not executed, with accumulation performed in a previous block 
B1:
repeat k = 1 to 2:
   accum i (total).
end.
display accum total i with down column 0 frame f1.
B2:
repeat while false: /* no accumulator usage */
   ...
end.
display accum total i with down column 13 frame f2.
Frames f1 and f2 will display the same value. This testcase shows that blocks which do not use the ACCUM statement for a certain accumulator will not affect it in any way. Also, this proves that, regardless of scoping level, if a block doesn't use the ACCUM statement and is not executed, the accumulator is not reset. This behavior is also present if B2 has no ACCUM stmt and is executed.

5. a block scoped to procedure block which uses the ACCUM stmt is not executed, with accumulation performed in a previous block 
B1:
repeat k = 1 to 2:
   accum i (total).
end.
display accum total i with down column 0 frame f1.
B2:
repeat while false:
   accum i (total).
end.
display accum total i with down column 13 frame f2.
In this example, block B2 is not executed, but the block uses the ACCUM stmt; frames f1 and f2 will also display different results:
- frame f1 will display the accumulated value for block B1
- frame f2 will display the "reset" value (0 in this case)
This is caused by the fact that, when executing the header for block B2, the accumulator is notified that it enters a block which may execute accumulation; so, the accumulator will be reset and also, when leaving block B2, it will set B1's "reported" bundle to be the "pending" (currently reset) bundle for block B2.
The testcase proves that the accumulator will be reset if a block which may use execute the ACCUM statement is entered, even if the block is scoped to procedure block.

6. a block scoped to procedure block which uses the ACCUM stmt is executed but the ACCUM stmt is skipped, with accumulation performed in a previous block 
B1:
repeat k = 1 to 2:
   accum i (total).
end.
display accum total i with down column 0 frame f1.
B2:
repeat:
   if false then
      accum i (total).  
end.
display accum total i with down column 13 frame f2.
In this example, block B2 is executed, but the the ACCUM stmt is skipped; frames f1 and f2 will also display different results:
- frame f1 will display the accumulated value for block B1
- frame f2 will display the "reset" value (0 in this case)
This is caused by the fact that, when executing the header for block B2, the accumulator is notified that it enters a block which may execute accumulation; so, the accumulator will be reset and also, when leaving block B2, it will set B1's "reported" bundle to be the "pending" (currently reset) bundle for block B2.
The testcase proves that the accumulator will be reset if a block which may use execute the ACCUM statement is entered, even if the block is scoped to procedure block.

Reset behavior for accumulators with IF statement

The IF statement impact on the accumulator "reset" behavior is related to stopping block execution using an "IF condition", when the condition is false. This will mean that neither the block header or body are executed.
1. IF statement with the ACCUM statement 
B1:
repeat k = 1 to 2:
   accum i (total).
   display accum total i with down column 0 frame f1.
   if false then accum i (total).
   display accum total i with down column 13 frame f2.
end.
Although the second accumulation statement does not execute at all, frame f1 and f2 will display the same results.
So, this proves that when an IF statement is executed and it encloses an ACCUM statement without an intermediate block, the accumulation value is not altered.

2. IF false statement with a block which does not execute 
B1:
repeat k = 1 to 2:
   accum i (total).
   display accum total i with down column 0 frame f1.
   if false then
      B2:
      repeat while false:
         accum i (total).
      end.
   display accum total i with down column 13 frame f2.
end.
Although block B2 does not execute at all (header or body), frame f1 and f2 will display different results:
- frame f1 will display the current accumulation value for block B1
- frame f2 will always display the UNKNOWN (?) value
So, this may prove that, when an IF statement is executed and it encloses an ACCUM statement inside another block, the accumulation value is set to unknown; also, when executing the THEN/ELSE branch, the accumulation value will be set to the "reset" value only if the header for a block which encloses the ACCUM statement is executed; if no such blocks are encountered on the THEN/ELSE branch, the accumulation value will remain set to UNKNOWN. (this assumption is disproved on the following testcase)

3. IF true statement with a block which does not execute 
B1:
repeat k = 1 to 2:
   accum i (total).
   display accum total i with down column 0 frame f1.
   if true then do:
      display accum total i with down column 13 frame f2.      
      B2:
      repeat while false:
         accum i (total).
      end.
   end.
   display accum total i with down column 26 frame f3.
end.
Block B2 does not execute and frames f1, f2 and f3 will display different results:
- frame f1 will display the current accumulation value for block B1
- frame f2 will display the same data as frame f1
- frame f3 will display the "reset" value for the accumulator (in this case, "0"), because block B2 does not execute.
So, this proves that, when an IF statement is executed and it encloses an ACCUM statement inside another block, the accumulation value is NOT set to unknown - this is postponed until the THEN/ELSE branch has ended and only if no ACCUM statement have been executed on the THEN/ELSE branch or no blocks (body or header) which enclose the ACCUM statement are executed.

4. IF false statement scoped to procedure block with a block which does not execute 
B1:
repeat k = 1 to 2:
   accum i (total).
end.
display accum total i with down column 0 frame f1.
if false then 
   B2:
   repeat while false:
      accum i (total).
   end.
display accum total i with down column 13 frame f2.
Block B2 does not execute at all (header or body) but frames f1 and f2 will display the same result.
So, this proves that, when the IF statement is scoped to the procedure block, it will not set to unknown the accumulator, even if it encloses a block which uses the accumulator.

Further more, scoping the IF to a DO statement: 
B1:
repeat k = 1 to 2:
   accum i (total).
end.
display accum total i with down column 0 frame f1.
do:
   if false then 
      B2:
      repeat while false:
         accum i (total).
      end.
end.
display accum total i with down column 13 frame f2.
will result in frames f1 and f2 displaying the same values. Experimenting with other statements which do not affect the accumulators (like SET/UPDATE/PROMPT-FOR - the editing blocks) showed the same behavior. The conclusion is that only IF statements which are enclosed within a block will set the accumulator to UNKNOWN.

The "backup" mechanism

The backup mechanism is related to cases when a block is entered and, before any other ACCUM/DISPLAY statement usage, the ACCUM function is used to retrieve the accumulated result. The following testcases describe this behavior:
1. display the accumulation result before any accumulation 
B1:
repeat k = 1 to 1:
   accum i (total).
   display accum total i with down column 0 frame f1.
   B2:
   repeat j = 1 to 2:
      display accum total i with down column 13 frame f2.
      accum i (total).
      display accum total i with down column 26 frame f3.
   end.
   display accum total i with down column 39 frame f4.
end.
In this case, frame f2 will always display the same result as frame f1 (on every B2 iteration) and frame f3 will display the current accumulation result for block B2; frame f4 will display the final accumulation result for block B2. This proves that PROGRESS presents some backup mechanism - used to report the "entry" accumulation value if no ACCUM stmts were executed in the current block before the ACCUM function call.

2. accumulation enclosed to the current block, but skipped 
B1:
repeat k = 1 to 1:
   accum i (total).
   display accum total i with down column 0 frame f1.
   B2:
   repeat j = 1 to 2:
      display accum total i with down column 13 frame f2.
      if false then accum i (total). /*1*/
      display accum total i with down column 26 frame f3.
      accum i (total). /*2*/
      display accum total i with down column 39 frame f4.
   end.
   display accum total i with down column 52 frame f5.
end.
In this case, frame f2 will not display the same result as frame f1:
- frame f1 will display the "reported" accumulation result for block B1
- frame f2 will display the same data as frame f3
- frame f3 will display:
    - when the first iteration is performed, the "reset" value (in this case, 0)
    - on subsequent iterations, the current accumulated result for block B2
- frame f4 will display the current accumulated result for block B2
- frame f5 will display the final accumulation result for block B2 (2 in this case)
So, after executing the "if false" statement, the accumulator will no longer report the "entry" value (the accumulation value for block B1).
This testcase also proves that, even if there are two ACCUM statements inside block B2, the second accumulation is executed. The difference between the two ACCUM statements is that the second one has as parent block B2 and the first one has as parent the IF statement.

Further more, investigating a testcase like: 
B1:
repeat k = 1 to 1:
   accum i (total).
   display accum total i with down column 0 frame f1.
   B2:
   repeat j = 1 to 2:
      display accum total i with down column 13 frame f2.
      accum i (total). /*1*/
      display accum total i with down column 26 frame f3.
      if false then accum i (total). /*2*/
      display accum total i with down column 39 frame f4.
   end.
   display accum total i with down column 52 frame f5.
end.
shows this:
- frame f1 will always display the current accumulation value for block B1
- frame f2 will display the "entry" accumulation value (the current accumulated value in block B1)
- frame f3 will always display the current accumulated value for block B2
- frame f4 will display the same values as frame f3
- frame f5 will display the total accumulated value in block B2
The conclusion from this testcase is that the "IF false" statement which uses the accumulator will reset it only if no other ACCUM statements were executed before it. So, the accumulator somehow knows to ignore the "if false then accum i (total)." statements after the accumulation has been performed within the current or a child block.

3. accumulation enclosed to the current block, but skipped - case 2 
B1:
repeat k = 1 to 1:
   accum i (total).
   display accum total i with down column 0 frame f1.
   B2:
   repeat j = 1 to 2:
      if false then
         B3:
         repeat while false:
            accum i (total).
         end.
      display accum total i with down column 13 frame f2.
      accum i (total).
      display accum total i with down column 26 frame f3.
   end.
   display accum total i with down column 39 frame f4.
end.
In this case, frame f2 will not display the same result as frame f1:
- frame f1 will display the "reported" accumulation result for block B1
- frame f2 will always display UNKNOWN (?) value
- frame f3 will always display UNKNOWN (?) value
- frame f4 will display the accumulated result in block B2 (in this case, value 2)
It seems that, after executing the "IF false" statement and setting the accumulator to UNKNOWN, the reported value for block B2 will always be UNKNOWN, even if accumulation is performed for block B2.

4. accumulation enclosed to the current block, but skipped - case 3 
B1:
repeat k = 1 to 1:
   accum i (total).
   display accum total i with down column 0 frame f1.
   B2:
   repeat j = 1 to 2:
      B3:
      repeat while false:
         accum i (total).
      end.
      display accum total i with down column 13 frame f2.
      accum i (total).
      display accum total i with down column 26 frame f3.
   end.
   display accum total i with down column 39 frame f4.
end.
In this case, frame f2 will not display the same result as frame f1:
- frame f1 will display the "reported" accumulation result for block B1
- frame f2 will always display the reset value (0 in this case)
- frame f3 will always display the reset value (0 in this case)
- frame f4 will display the total accumulation result for block B2.
For this testcase, the "reported" result for block B2 will not be set to UNKNOWN after executing the header for block B3 - it will be reset. This "reset" value will always be reported by any ACCUM function call, regardless of any executed ACCUM statement, until another block with an enclosed ACCUM statement or an IF statement with an enclosed ACCUM statement is executed.

5. "pending" and "reported" values for a block 
B1:
repeat k = 1 to 1:
   if false then accum i (total). /*1*/
   
   display accum total i with down column 0 frame f1.
   B2:
   repeat j = 1 to 3:
      accum i (total). /*2*/
   end.
   display accum total i with down column 13 frame f2.
   
   accum i (total). /*3*/
   display accum total i with down column 26 frame f3.
   
   B3:
   repeat j = 1 to 4:
      accum i (total). /*3*/
   end.
   display accum total i with down column 39 frame f4.
end.
In this example, frames f2 and f3 will always display the accumulated result for block B2 and frame f4 will always display the accumulated result for block B3:
- frame f1 will display 0 on first iteration and the current accumulated result for block B1, on subsequent iterations
- frame f2 will always display the accumulated result for block B2
- frame f3 will always display the same values as frame f2
- frame f4 will always display the accumulated result for block B3
So, when the third ACCUM statement is executed, the "reported" value for block B1 is not affected - after executing a block which uses the ACCUM statement, the "reported" value for the parent block (block B1 in this case) will always be the "pending" (computed) value for the executed block, until another block is executed.

More, this affects the reported value for the enclosed block: 
B1:
repeat k = 1 to 1:
   if false then accum i (total). /*1*/
   
   display accum total i with down column 0 frame f1.
   B2:
   repeat j = 1 to 3:
      message "B2:" + string(accum total i). /*1*/
      accum i (total). /*2*/
   end.
   display accum total i with down column 13 frame f2.
   
   accum i (total). /*3*/
   display accum total i with down column 26 frame f3.
   
   B3:
   repeat j = 1 to 4:
      message "B3:" + string(accum total i). /*2*/
      accum i (total). /*4*/
   end.
   display accum total i with down column 39 frame f4.
end.
The ACCUM function used in the two message statements will behave differently:
- the ACCUM function in case #1 (block B2) will always display the accumulated (pending) value for block B1
- the ACCUM function in case #2 (block B3) will be affected by the fact that another block (block B2) has been executed - this will display 0 on the first iteration and the accumulated/pending result for block B3 on subsequent iterations.

ACCUM with nested RUN/TRIGGER statements

1. ACCUM with nested RUN statements (recursion) 
def var i as int.
procedure proc1.
def input parameter p as int.
  do transaction:
     accum i (count).
     if p <> 0
       then run proc1(p - 1).
  end.
  put screen row p + 1 string(p) + " " + string(accum count i).
end.
run proc1(10).
For the testcases above, the ACCUM function will always return the same value. This is because accumulators - which are used inside a procedure/function - are alive and can be used only as along as the procedure is executed; this means that the accumulator is not scoped to the root external procedure, but is scoped to the internal procedure. In P2J, instead of scoping the accumulator to the internal procedure, we keep a stack with the "toplevel" status of each entered block. This way, when recursion occurs and the ACCUM statement is called, the accumulation will be performed only until a "toplevel" block is encountered - and so no other blocks from a previous recurssion will be affected.

2. ACCUM with nested TRIGGERS 
def var j as int.
def var i as int.
def var k as int init  0.

form j with frame f0 column 39.
view frame f0.

on F7 of j in frame f0 do:
  do transaction:
     accum i (count).
  end.
   put screen row k + 1 column 1 string(k) + " " + string(accum count i).
  if k < 10 then do:
      k = k + 1.
      apply "F7" to j in frame f0.
      k = k - 1.
  end.
  put screen row k + 1 column 13 string(k) + " " + string(accum count i).
end.

apply "F7" to j in frame f0.
In this testcase, applying the same trigger as the one being executed will not be possible: in PROGRESS, the trigger will not go into recurssion and so the 'apply "F7" to j in frame f0.' statement inside the trigger definition will be ignored.

Database Support

Data Types

The following Progress data types are directly supported as persistable types using the P2J wrappers.  The mappings are made transparent to the application using custom, Hibernate user types.  See the com.goldencode.p2j.persist package documentation for additional details.

Progress Type
 P2J Wrapper Type
character
 character
date
 date
datetime datetime
datetime-tz datetimetz
(At the moment of writting this there are some issues persisting multicolumn values with Hibernate.)
decimal
 decimal
integer
recid
 integer
int64 int64
logical
 logical
raw
 raw

Relational Integrity and Natural Joins

An enhanced level of relational integrity is provided during conversion in that foreign key constaints are added to the database schema where the associated relations can be detected unequivocally at conversion.  Currently, this detection is limited to the analysis of instances of the <record> of <table> construct within a Progress record phrase.  Although this is a limited limited approach, since this construct is simply short-hand for a more verbose where clause construct, it is the only mechanism currently implemented, because it is reliable and easily automated (mostly).  This analysis and its effect on schema conversion and DMO creation is described in the com.goldencode.p2j.schema package documentation .

The introduction of foreign keys unfortunately increases the complexity of the runtime environment, because these resources now need to be used and managed transparently (since the pre-conversion application knows nothing of them).  This support is discussed in teh com.goldencode.p2j.persist package documentation .

Buffer Scoping

To understand the logic and rules of buffer scoping, please see Record Scoping .

The implementation of the scoping calculations is done completely during annotation processing, though it is so complicated that it takes 3 rule-sets to properly implement.  In addition, the complex arrangement of data structures required to calculate scopes means that several external classes and a custom worker were necessary to implement this processing in the most simple manner.

The result is a set of new BUFFER_SCOPE nodes which are each a child node of the BLOCK to which they are scoped.  Each one has a set of annotations necessary for downstream processing.  These annotations include the class name and Java name of the buffer, among other important information.

Each record and field reference has a " refid " annotation that is the AST ID of the BUFFER_SCOPE node to which the reference refers.  This allows the buffer's name to be dereferenced at conversion time.

Please see the following for more details:

rules/annotations/record_scoping_prep.rules
rules/annotations/record_scoping.rules
rules/annotations/record_scoping_post.rules

BufferScopeTracker
BufferList
BufferBlockState
BufferScopeWorker

Record Buffer Instantiation

The Java AST will get RecordBuffer instantiation or declaration nodes in the following cases:
  1. There is an explicit language statement (define buffer) which would cause a memory allocation to occur in Progress.
  2. There is an explicit language statement (define parameter or parameter) which would cause a buffer declaration to occur in a function or internal procedure.
  3. The first implicit reference to a buffer in the source file will cause a memory allocation of a buffer of the same name as the record.  This can be found by examining all BUFFER_SCOPE nodes and finding those references that have both the "implicit" and "first" annotations as true.
The location of the resulting nodes is determined by the following rules:
  1. A "define buffer" inside an internal procedure, trigger or function will emit a RecordBuffer.define() in place (in the main block of the internal procedure, trigger or function).  This allocates a new data model object.  Note that this works since the DEFINE_BUFFER nodes are moved/sorted to always be rooted as the first children of the BLOCK node in any internal procedure, trigger or function.
  2. An explicit DEFINE_PARAMETER inside an internal procedure is emitted to the method definition signature (it won't appear in a function or trigger).
  3. An explicit PARAMETER (that is a buffer definition) inside a function is emitted directly based on peerid (it won't appear in an internal procedure or trigger).
  4. All other buffers are instance members (this avoids the need for differentially constructing some in the external procedure and some as instance members via "promotion" due to usage across the entire class or due to being an explicit shared buffer definition).
Import statements for the generic com.goldencode.p2j.persist package are always added, as is an import for the custom package(s) that contain the application-specific data model objects.

The Java AST nodes that are created as a result of this logic are as follows:
  1. An imported shared buffer (define shared buffer) is emitted as a buffer declaration with an initialization based on calling SharedVariableManager.lookupBuffer().
  2. An exported shared buffer (define new shared buffer) is emitted as a standard buffer definition using RecordBuffer.define() with a second method call to SharedVariableManager.addBuffer() to export the buffer.  Note that in the case where the buffer is located as an instance member, the define and the addBuffer will be located in 2 places.  In particular, the addBuffer() will be done in line.
  3. Implicit buffers and non-shared explicit buffers (define buffer) are emitted as a RecordBuffer.define().
  4. Parameters (define parameter and parameter in a function definition) are emitted in the corresponding method signature as a REFERENCE_DEF (classname followed by the instance name).
All BUFFER_SCOPE nodes also generate a notification to RecordBuffer.openScope() to ensure that the database runtime code marks the end of each block for the associated record release and record lock release.

Please see:

Transaction Management in the persist package
rules/convert/buffer_definitions.rules

Buffer and Field References

Buffer/record references are emitted in 3 cases:
  1. The reference is a child of a KW_BUFFER node.  This conforms to the usage of a buffer reference as a RUN statement parameter or function call parameter.
  2. The reference's "bufreftype" annotation is not NO_REFERENCE and the 3rd level ancestor is not PARAMETER.
  3. The reference's "bufreftype" annotation is not NO_REFERENCE and the 3rd level ancestor is not DEFINE_PARAMETER.
The resulting node is a simple Java REFERENCE node with the buffer's Java name.

Field references are always emitted, but these translate into field-specific getter and setter calls on the data model object.  The getter and setter method names are determined at annotation time.

The assignments rule-set required changes to modify the normal ASSIGN node processing when the field reference is a setter.  This is required to ensure that the data model object (which is the first child) will emit as the referent of the setter method call.  It is also required such that the 2nd operand of the ASSIGN node would emit as the 2nd or 3rd child (which would be the 1st or 2nd parameter to the setter call).

The variable references rule-set was modified to change how subscripts work for fields.  Since Progress fields can be treated as array values, such parameters must be properly handled as the first integer index parameter of a getter or setter method call.  This is the reason that the 2nd operand of the ASSIGN node can be either the 1st or 2nd parameter to the setter method call.  The same logic as normal subscripting applies here too (all values are converted into 0-based indices from the Progress 1-based approach).

Please see:

rules/convert/assignments.rules
rules/convert/database_references.rules
rules/convert/variable_references.rules

WHERE Clause Processing

The following is the mapping of Progress WHERE clause constructs to HQL (Hibernate Query Language).

Progress WHERE
 HQL
 Notes
LPARENS
 (
AND
 and
OR
 or
EQUALS
 =
NOT_EQ
 !=
GT
 >
GTE
 >=
LT
 <
LTE
 <=
BEGINS
 like '<criteria>%'
 The 2nd operand's text is converted into a Java string using ExpressionConversionWorker progressToJavaString() and then the '%' is appended.  Strings are enclosed in single quotes.
MATCHES
 like '<criteria>'
 The 2nd operand's text is converted into a Java string using ExpressionConversionWorker progressToJavaString() and then special matching characters are converted into the HQL equivalents using ExpressionConversionWorker convertToSQLLike().  Strings are enclosed in single quotes.
CONTAINS
 n/a
NOT
 not ( )
 The child node of this operator is wrapped in () to ensure that the Progress precedence is maintained, since this is different compared to HQL NOT precedence.
BOOL_TRUE
 true
BOOL_FALSE
 false
NUM_LITERAL
 text is emitted directly
DEC_LITERAL
 text is emitted directly
STRING
 'text'
 The text is converted into a Java string using ExpressionConversionWorker progressToJavaString().  Strings are enclosed in single quotes.
DATE_LITERAL
 'YYYY-MM-DD'
 The text is turned into an instance of com.goldencode.p2j.util.date and the toStringSQL() method is used to generate the correct format.  The resulting string is enclosed in single quotes.

Note that there are special considerations for handling dates within a database where clause.
UNKNOWN_VAL
 is null
is not null
null
 If the parent note is EQUALS or NOT_EQ one of the two first forms are emitted.  Otherwise the 3rd form is emitted.
FUNC_RECID and oldtype == KW_RECID
 <javaname_of_buffer>.id
 This builtin function is directly converted to a reference to the special "id" field.
FUNC_ROWID and oldtype == KW_ROWID
 <javaname_of_buffer>.id This builtin function is directly converted to a reference to the special "id" field.
FIELD_ in the current buffer
 <buffer_qualifier>.<property_name>
LBRACKET
 []
 Only FIELD_ extents are supported.

Subscripts are supported using [ ] but only if the index is a NUM_LITERAL.  Such an index is decremented by 1 and emitted as text inside the square brackets.

Anything not directly supported above is treated as a substitution parameter.  The HQL string will contain a '?' character at the point in the string in which the runtime substitution should occur.  The "root" node of such a sub-expression is emitted as a runtime expression in the generated business logic.  This result is passed as a query argument.  Hibernate handles the query substitutions using the result of the evaluated runtime expression.

Other notes:

Database Language Statements

Java classes referenced in the table below reside in the com.goldencode.p2j.persist package unless otherwise noted.
Statement
 Java Equivalent
 Notes
ACCUM
 Accumulator (abstract base class);  concrete implementations:
  • AverageAccumulator
  • CountAccumulator
  • MaximumAccumulator
  • MinimumAccumulator
  • TotalAccumulator
  • reside in the com.goldencode.p2j.util package (following the Progress semantic, these classes are not strictly database related;  they can be used standalone with variables or added to a query)
  • same classes are used to handle both cumulative results and break group results
BUFFER COMPARE
 RecordBuffer.compare
BUFFER COPY
 RecordBuffer.copy
CLOSE QUERY
 n/a
 P2J queries are not explicitly closed;  this statement is dropped
CONNECT
 ConnectionManager.connect
CREATE
 RecordBuffer.create
 No USING support at this time.

In Progress, CREATE does not immediately flush the new record to the database, since in most cases the default values assigned to a new record would violate uniqueness or nullability constraints.  Instead, the record is not written to the database until the first statement is executed, which would update an index for the record's table.

In our implementation, we defer writing the record to the database (or more accurately, persisting the record using Hibernate;  actual flushing to the database may be further deferred by Hibernate), until the earliest of the following occurs:
  1. all properties participating in ANY uniqueness constraint are updated (the last property updated in such a set triggers validation and flushing);
  2. a query which operates against the record's table is executed;
  3. the current record in the record buffer is replaced or released;
  4. a transaction or subtransaction commit occurs;
  5. the end of a converted ASSIGN statement is reached (RecordBuffer.endBatch() is invoked).

Note that our implementation departs slightly from Progress -- particularly with regard to #1 in the list above -- in that the setting of a property which does NOT participate in a unique constraint (but DOES participate in a non-unique index) WILL NOT trigger a buffer flush.  However, we believe the remaining conditions adequately make up for this departure.
CREATE ALIAS
 ConnectionManager.createAlias
CREATE BUFFER
TBD;  dynamic buffer creation is not supported in the first release
CREATE DATABASE
TBD;  dynamic database creation is not supported in the first release
CREATE QUERY
TBD;  dynamic query creation is not supported in the first release
CREATE TEMP-TABLE
TBD;  dynamic temp-table creation is not supported in the first release
DEFINE BUFFER
 RecordBuffer.define
 see Record Buffer Instantiation
DEFINE QUERY
 AbstractQueryWrapper (abstract base class);  concrete implementations:
  • CompoundQueryWrapper
  • PreselectQueryWrapper
  • RandomAccessQueryWrapper
 These wrapper classes are used to support the DEFINE QUERY semantic of defining a query once and possibly opening it multiple times.  An instance is constructed once, as an instance variable, corresponding with the DEFINE QUERY statement.  Each corresponding OPEN QUERY is converted as an instance of a concrete subclass of AbstractQuery, which is set into the wrapper when opened.
DEFINE TEMP-TABLE
DEFINE WORK-TABLE
 TemporaryBuffer.define no distinction is made between temp-tables and work-tables at conversion;  both are converted to DMO class definitions in the <...>.dmo._temp package (DMO interfaces) and the <...>.dmo._temp.impl package (DMO implementation classes) in the converted application hierarchy

see also Temporary and Work Table Support
DELETE
 RecordBuffer.delete
 no VALIDATE support at this time
DELETE ALIAS
 ConnectionManager.deleteAlias
DISCONNECT
 ConnectionManager.disconnect
FIND (standalone)
 FindQuery
AdaptiveQuery
FIND (as preselect retrieval mechanism)
 P2JQuery.first
P2JQuery.last
P2JQuery.next
P2JQuery.previous
FOR/DO/REPEAT
 AbstractQuery (abstract base class);  concrete implementations:
  • CompoundQuery
  • PreselectQuery
    • AdaptiveQuery
    • PresortQuery
  • RandomAccessQuery
 Which concrete implementation is chosen at conversion time depends upon the number of tables involved and upon the nature of the query.

For an individual table, PreselectQuery is chosen if it is defined explicitly in Progress using the PRESELECT keyword or if the Progress construct's explicit sort criteria do not match the index selected according to index selection rules.  PresortQuery is chosen if client-side sorting is required (i.e., the by clause contains an expression which cannot be converted to an HQL 'order by' clause, but must instead convert to a runtime expression).  Both of these types can be applied in a multi-table join situation.

AdaptiveQuery is chosen by default for a single table query, if other factors do not force a preselect variant.  This query operates in preselect mode until some change to a record forces it to operate in dynamic mode.  As soon as it can switch back to preselect mode, it will.

RandomAccessQuery is used by an AdaptiveQuery when switching to dynamic mode.  This is the most flexible query type (but also has the worst performance);  it is necessary to support the Progress "dynamic" record retrieval semantic, where updates to a record made during the loop can change the results found later in the loop (an effect [side effect?] of Progress' index implementation).

CompoundQuery is chosen for nested FOR loop conversions, where the component loops would convert to AdaptiveQuery instances or to a mixture of adaptive and preselect query variants.  CompoundQuery is a driver to which Joinable (implemented by RandomAccessQuery and PreselectQuery) query instances may be added as components.
GET
 P2JQuery.first
P2JQuery.last
P2JQuery.next
P2JQuery.previous
RELEASE
 RecordBuffer.release
REPOSITION
 RandomAccessQuery.reposition
OPEN QUERY
 AbstractQuery (abstract base class);  concrete implementations:
  • CompoundQuery
  • PreselectQuery
    • PresortQuery
  • RandomAccessQuery
    • FindQuery
 in the event of an OPEN QUERY which references a query previously declared by a DEFINE QUERY, one of these concrete classes is set into the corresponding concrete wrapper implementation;  otherwise, an instance of one of these classes is used directly
VALIDATE
 RecordBuffer.validate

Built-in Function Support

See all entries in this table with type "database
essimistic Record Locking Progress relies upon a pessimistic locking strategy consisting of three record locking modes:
References to these keywords have been converted to use constants defined as static variables of the LockType class.  These constants are used by the persist package internally.  They appear in converted application code as arguments to various query constructors and initialization methods, and in record retrieval methods (e.g., first , last , next , previous , current , and unique ), to correspond to explicitly specified lock requests in Progress record phrases.

For information regarding P2J's runtime pessimistic locking strategy, please refer to the persist package documentation .

Please see:

rules/convert/database_access.rules

Temporary and Work Table Conversion

Temp-tables and work-tables convert to the same temporary table implementation;  there is no distinction between the two in the P2J environment, so both are referred to generically as "temp tables" or "temporary tables" in this document.  Temp table structural information is extracted from DEFINE TEMP-TABLE and DEFINE WORK-TABLE statements in the Progress source code.  Each unique definition is converted into a specialized DMO interface and implementation class.  A Hibernate mapping document is created for each, and each receives an entry in the runtime dmo_index.xml document, which resides at the relative root of the dmo sub-package in the converted application's package hierarchy.

A temp table definition is considered unique if its structural organization and content (i.e., field and index elements) is unique across the application.  Currently, Hibernate contains no direct support for temp tables;  however, it does not preclude that Java objects be mapped to temporary database tables.  Furthermore, Hibernate requires that a Java object be mapped to only one backing table within a single SessionFactory context.  It also requires that these mappings be known at SessionFactory configuration, which for practical purposes (it is quite expensive) must occur infrequently and preferably during server initialization.  For these reasons, only the conversion of statically defined temp tables are supported at this time.

Please see the persist package documentation for more information about runtime temp table support.

Please see:

rules/convert/buffer_definitions.rules
rules/convert/database_access.rules

Client-Side Where Clause Processing

Certain where clauses in Progress cannot be converted to server-side queries expressed entirely in HQL.  Specifically, if a where clause contains a nested reference to a field in the record being retrieved, it must be tested against a client-side expression to determine whether it will be included in the query result.  By "nested", we mean embedded within a subexpression which cannot be expressed in HQL.  For instance, a field reference to the current buffer as a parameter to to a built-in or user function would qualify for client-side where clause processing: 
def var snippet as character initial "whatever".
for each customer
         no-lock,
    each invoice
         where invoice.cust-num = customer.cust-num
         and substring(invoice.category, 10, length(snippet)) = snippet
         no-lock:
   ...
Although this is a trivial example, which could be expressed differently, it serves to illustrate the point:  the customer.name reference embedded within the built-in function substring causes this code to convert to Java code which executes a client-side where clause expression.

Progress where clauses which qualify for this type of handling convert, where possible, to a server-side component and a client-side component.  The server-side component is expressed in HQL and the client side component converts to an anonymous inner class which extends the abstract class WhereExpression .  At runtime, the query is executed (at a high level) in two phases:
  1. a server-side query which returns a preliminary, candidate result;
  2. a client-side expression which filters the candidate results.
The server-side query is necessary to restrict, to the extent possible, the amount of data pushed to the client for final evaluation, since this is likely to be the most expensive component of the process.  The greater the level of restriction performed at the server, the better.  That being said, it is not always possible to generate HQL which accomplishes this goal.  In the worst case scenario, no server-side restriction will be possible at all, and every record in the table will be retrieved and evaluated on the client.

Server-side restriction is possible only if the a top-level operation in the where clause expression is a logical AND, and at least some portion of one side of the expression can be evaluated at the database server.  In the above example, the invoice.cust-num = customer.cust-num component of the where clause can be expressed in HQL as invoice.cust-num = ? , where ? is substituted at runtime with each customer.cust-num value found in the outer loop.

The client-side portion of this expression is converted to an anonymous, inner subclass of WhereExpression , as the following code segment illustrates: 
character snippet = new character("whatever");

WhereExpression whereExpr0 = new WhereExpression()
{
   public Object[] getSubstitutions()
   {
      return new Object[]
      {
         new IntegerExpression()
         {
            public integer execute()
            {
               return snippet.length();
            }
         },
         snippet
      };
   }

   public logical evaluate(BaseDataType[] args)
   {
      return CompareOps.equals(character.substring(invoice.getCategory(), 10, (integer) args[0]), (character) args[1]);
   }
};
During the client-side filtering phase of query execution, the evaluate() method defined above is invoked for each invoice record retrieved by the server-side phase. The WhereExpression class and the various query implementations ensure that the arguments/expressions returned by getSubstitutions() are resolved at the appropriate times during execution of the query loops (this is important, since it preserves the Progress semantic by ensuring that any side effects of invoking built-in or user functions -- such as state changes -- are maintained).  The resolved results of these arguments/expressions compose the args parameter passed to evaluate() .  If evaluate() returns true , the current invoice record is retained as a final query result;  if it returns false , the record is dropped from the query's final results.

NOTE:   the conditions which trigger client-side where clause processing as discussed above have been greatly reduced since this section was first written.  Since that time, we have introduced and implemented many server-side (i.e., database server, not P2J server) functions which minimize the circumstances which require client-side where clause processing.  Since this implementation, the only where clauses that require client-side where clause processing are those which integrate user-defined functions, because these may have side effects which cannot be predicted and must execute on the client.  All built-in Progress functions which are used in where clauses in the initial project to which P2J is targeted have now been implemented as server-side functions.  Please refer to the following package documentation for additional details on this implementation.  The information in the following package summaries take precedence over the information above:
Please also see:

rules/annotations/where_clause_pre_prep.rules
rules/annotations/where_clause_prep2.rules
rules/annotations/where_clause.rules
rules/annotations/where_clause_post.rules
rules/annotations/where_clause_post.rules
rules/convert/database_access.rules

Database Validation Expressions

Validation expressions and messages can be defined for database fields (and tables, though this is not supported at this time).  Field-level validation expressions are used by default in certain blocking, input, UI language statements.  Application code may provide an explicit override (using the VALIDATE keyword), in which case the default validation expression is ignored.

Any Progress code which is valid within the context of the business logic in which the corresponding field reference is made can be used within a validation expression.  Therefore, although this would be very bad form, it is legal for a validation expression to contain references to local variables and other constructs valid only within the enclosing scope of the field reference.  This is problematic for conversion, because it requires that validation expressions emulate this behavior in Java code.  To accomplish this, we duplicate the conversion expression as a subclass of Validatable , implemented as an inner class within each business logic class in which the field must be validated.

Although these expressions are defined at the database schema level, they are evaluated only by the UI.  Field-level validation expressions defined in the schema are not triggered by simple assignments .  The following UI language statements trigger field-level validation:
It is the first occurence of a reference to a validatable field, in association with one of these statements, which triggers the Validatable implementation to be emitted in the target business class.  It also triggers an instance of the Validatable implementation class to be created and registered with the UI widget associated with the validatable field.  In addition to the above listed statements, the FORM statement is given special consideration, because it may contain the first reference in a frame to a validatable field, even though it does not itself trigger validation.  Surprisingly, the SET and UPDATE options of the MESSAGE statement do not trigger field-level validation.

Please see:

rules/schema/p2o.xml
rules/fixups/schema-validations.rules
rules/annotations/validation-prep.rules
rules/annotations/validation.rules
rules/annotations/validation-post.rules
rules/convert/ui_statements.rules
rules/convert/expression.rules

Methods and Attribute Support

At this time, the following subset of database-related methods and attributes are supported:
Method/Attribute
 Java Equivalent
 Notes
CURRENT-RESULT-ROW
 P2JQuery.currentRow()
GET-FIRST()
 P2JQuery.first()
GET-LAST()
 P2JQuery.last()
GET-NEXT()
 P2JQuery.next()
GET-PREV()
 P2JQuery.previous()
HANDLE
 P2JQuery instance reference
 There is no method equivalent in Java for this attribute;  the "handle" of a P2JQuery object is the object reference itself.  Therefore, the conversion of Progress code which dereferences the handle attribute from a query object is simply to emit the Java query object reference.
QUERY-OFF-END
 P2JQuery.isOffEnd()
QUERY
 N/A
 Use of this attribute of browse is superfluous in Java, in the customer cases encountered thus far.  Therefore, it is simply removed from the tree in a customer-specific annotation ruleset.  If it is determined that this attribute is universally superfluous, this processing will be relocated to a common ruleset.

Known Issues

  1. When generating HQL expressions from a Progress where clause where a character comparison is necessary, the case sensitivity of a field or variable which is substituted at runtime for an HQL query substitution marker is only maintained if that field or variable is a direct operand of the comparison.  For instance, the Progress where clause customer.name = invoice.cust-name might convert to an HQL expression upper(?) = upper(invoice.cust-name), if both fields are case-insensitive.  However, the case sensitivity of customer.name may not be considered in the HQL generation if the field reference is more deeply nested.  For example, the Progress where clause substring(customer.name, 0, 5) = invoice.cust-name would convert to the same HQL phrase shown earlier if invoice.cust-name is case-insensitive, even if customer.name was case-sensitive.  This is considered to be a relatively obscure case;  it will be addressed as needed.
  2. For searches using HQL expressions which have an embedded, non-literal (i.e., dynamic), subscript reference (e.g., where myBuf.myField[?] = ...), P2J diverges from Progress in the type of error raised for a subscript out of bounds condition.  Because the query substitution is handled on the database server using this notation, Hibernate simply comes back with an empty result set in the case where the subscript is out of bounds, which will result in an error condition with text ** <XXXX> record not on file. (138) when a null DMO is set as the current record in a record buffer.  Progress raises a stop condition with text ** Array subscript <X> is out of range. (26).

User Interface Support

Basic Design

All static elements of the UI are removed from the business logic and used to create 2 UI specific classes for each unique frame:
  1. A custom screen buffer interface with getters, setters and widget accessors for the associated frame.
  2. A frame definition that can be used to instantiate and initialize a frame of the given layout/fields/configuration and its contained widgets.
These classes are used at runtime by the business logic.  In particular, the control flow is defined in the business logic.  This means that the controller of the UI is in the business logic, but the format, layout ... of the UI itself has been separated into UI specific classes.

The behavior of DISPLAY/UPDATE/SET/PROMPT-FOR is special because these statements depend upon a passed array of FrameElement instances which define the list of frame fields that are to be processed.  Each FrameElement is one of the following subclasses:
  1. Element - a displayable data value and the widget it is associated with
  2. WidgetElement - a widget with no data (e.g. a button)
  3. EmbeddedAssignment - a mechanism to delegate the execution of an assignment (and dynamic evaluation of the associated expression) during the assignment processing in these language statements.
For each of the statements the copy to/from the screen-buffer, the widget enable/disable, the view(), the wait-for() are all hidden in the runtime.  This processing is mapped into primitive operations but the logic is properly enforced in these higher level APIs such that the Progress semantic is maintained.  Please see GenericFrame for more details.

The rules/annotations/frame_scoping.rules is the core rule set that calculates the frame scoping, widget lists for each frame and many other critical annotations.  The majority of the UI related business logic annotations are done in rules/annotations/screen_buffer.rules.

The rules/convert/frame_generator.xml is a standalone rule set that is run between annotations and code conversion.  This rule set uses the previously stored annotations to generate the 2 classes above per frame.  The majority of the UI related business logic code conversion is done in rules/convert/ui_statements.rules.

UI Language Statements


Statement
 Java Equivalent
 Notes
APPLY
 CommonFrame.apply()
LogicalTerminal.apply()
ASSIGN (input buffer form) Rewritten using the INPUT function such that this is converted as an explicit assignment statement that directly reads the field's getter method in the screen buffer.
BELL
 LogicalTerminal.bell()
CHOOSE
 CommonFrame.choose()
CLEAR
 CommonFrame.clear()
CommonFrame.clearAll()
COLOR
 LogicalTerminal.setColors()
CREATE WIDGET
 not converted at this time
CREATE WIDGET-POOL
 not converted at this time
DEFINE BROWSE
 new BrowseWidget()
DEFINE BUTTON
 new ButtonWidget()
DEFINE FRAME
 new FrameWidget()
DEFINE RECTANGLE
 not converted at this time
DELETE OBJECT
 not converted at this time
DELETE WIDGET
 not converted at this time
DISABLE
 GenericWidget.disable()
CommonFrame.disable()
CommonFrame.disableExcept()
DISPLAY
 CommonFrame.display()
DOWN
 CommonFrame.down()
ENABLE
 GenericWidget.enable()
CommonFrame.enable()
CommonFrame.enableExcept()
FORM
 Converted into a frame definition class.
FRAME-VALUE
 LogicalTerminal.setFrameValue()
HIDE
 LogicalTerminal.hideMessage()
LogicalTerminal.hideAll()
CommonFrame.hide()
INPUT CLEAR
 LogicalTerminal.inputClear()
MESSAGE
 LogicalTerminal.message()
LogicalTerminal.messageBox()
NEXT-PROMPT
 CommonFrame.nextPrompt()
ON
 LogicalTerminal.registerRunnable()
LogicalTerminal.deregisterRunnable()
LogicalTerminal.remapKey()
PAUSE
 LogicalTerminal.pauseBeforeHide()
LogicalTerminal.pause()
PROCESS-EVENTS
 LogicalTerminal.processEvents()
PROMPT-FOR
 CommonFrame.promptFor()
PUT-CURSOR LogicalTerminal.putCursor()
PUT-SCREEN LogicalTerminal.putScreen()
READKEY
 KeyReader.readKey()
SCROLL
 CommonFrame.scroll()
SET
 CommonFrame.set()
STATUS
 LogicalTerminal.statusDefault()
LogicalTerminal.statusInputOff()
LogicalTerminal.statusInput()
TERMINAL
 LogicalTerminal.setTerminal()
UNDERLINE
 LogicalTerminal.underline()
CommonFrame.underline()
UP
 CommonFrame.up()
UPDATE
 CommonFrame.update()
VIEW
 CommonFrame.view()
WAIT-FOR
 LogicalTerminal.waitFor()

See also:

rules/annotations/trigger_prep.rules
rules/annotations/set_update_embedded_assignments.rules
rules/annotations/frame_scoping.rules
rules/annotations/screen_buffers.rules
rules/annotations/validation_prep.rules
rules/annotations/validation.rules
rules/annotations/when_rewriting
rules/annotations/dynamic_ui_indexes
rules/convert/ui_statements.rules

Special UI Block Types/Callbacks

The Progress UI contains 2 special block types (trigger blocks and editing blocks) and 1 pure callback (validation expressions).  All 3 of these constructs are tightly integrated with language statements that interact with users.

Triggers are blocks that are executed when defined events occur in the user interface.  Instead of defining triggers like internal procedures, triggers are defined "in line" with the procedure or function being processed.  Though Progress treats a trigger as a distinct block type which has block properties ( see above ), it is not well seperated in terms of location in source code.   Triggers take no parameters and are not named.  Instead, they are registered as a set of events and a set of widgets which cause the trigger to be called.  When any of the listed events occurs for any of the listed widgets, the trigger will be executed.  Multiple triggers can be registered for the same event + widget combinations.  The last registration "wins" (is active).  The scoping for such registrations is at the procedure (internal/external), trigger and function level.  When such a new scope is opened, duplicate trigger registrations will hide registrations from previous scopes.  Likewise, the closing of a scope will remove all triggers registered in that scope, which will make previously hidden triggers "re-appear".  Within a single procedure, function or trigger (yes, triggers can be nested in other triggers), duplicate registrations hide previous registrations and this will not implicitly ever be undone (without the entire scope exiting and thus removing all triggers registered in that scope).  The explicit REVERT option can be used to deregister a specified trigger and then the trigger registered most recently (in this scope or a previous scope) will become the active trigger for that given event + widget combination.

One other interesting feature of trigger registration is that it is strictly based on the flow of control of the registering code.  If the flow of control never executes the branch of code in which a trigger is defined (e.g. an ELSE block that is never executed), then that trigger is not registered.  This means that the registration of triggers must be kept strictly in line with the matching control flow, even if the trigger block itself is refactored into another location.

Editing blocks are really special purpose key-press processing loops that are called in-line from the PROMPT-FOR processing (and from UPDATE/SET since they have an embedded PROMPT-FOR).  Such blocks are invoked before any other event processing associated with a key press.  They can override or modify all event processing associated with the key press (editing blocks take precedence over all other forms of event processing).   The editing block is peculiar in that it is completely integrated with the surrounding code (the block structuring of the surrounding code).  In particular, this means that one can execute low-level flow control statements such as NEXT or LEAVE and these can directly reference enclosing block names.  Although it may not be obvious, such blocks are in fact loops and one can use NEXT to force an iteration (restart from the top of the block).  Such blocks do have block properties ( see above ) but they are limited and cannot be customized (with the "ON clause").

Validation expressions are not a block type but instead are a direct evaluation of a logical expression to determine if the content of an associated field is or is not valid.  The expression is evaluated in the context of the defining program but it called when a field is changes in the UI and the user attempts to leave the field or screen.  If the validation expression returns FALSE, a second expression is evaluated to obtain message text and this message is displayed to the user.  The user is then placed back into that field (or just not allowed to leave) and is provided an opportunity to edit the field again.  If the validation expression evaluates TRUE, then the field is considered valid.  If no edits are made to a field but the screen is exited, all fields that had not yet been validated will be validated.  The first validation expression to fail will display the message and place the user back into the associated field to edit it.  Of particular interest is the fact that both the validation expression itself and the expression used to generate the message text are real expressions and can contain the full variety of function calls, operators, constants and resource references one would expect in a Progress expression.

In terms of scoping, all 3 constructs can access resources (vars, streams, buffers and frames) that are defined in the enclosing scope.  This complicates the generation of code immensely since the rules for scoping in Java are much more strict.  A great deal of processing was implemented in annotations to ensure that all such resources are "promoted" to instance members in the generated business logic classes.  This allows triggers and validation expressions to be emitted as inner classes while still accessing all the resources in the proper context.  In particular, streams, buffers and frames were all modified to ensure that *all* instances of such resources have unique names and thus all instances can always be made into instance members.  With variables, this was not desireable since the variables are so common and keeping a scoped approach yields a better result.  So variables are analyzed to determine which need promotion.  This promotion logic is in rules/annotations/scope_promotion.rules.

Triggers are implemented as a named inner class (each one gets a generated name) which extends Runnable.  The trigger block itself is emitted inside the run() method and it is treated as a top-level sub-transaction block in terms of the TransactionManager .  All local resources are promoted to instance members in the containing class so these resources are directly accessible.  A instance of the inner class is constructed (default constructor) and passed to the registration method, along with a definition of the event + widget combinations that are valid.  These event list + widget list combinations are defined as an instance of the WaitList class.

Editing blocks (loops actually) are implemented in-line with the original containing logic (at the location of the UPDATE/SET/PROMPT-FOR statement in the original control flow).  This is required to ensure that the integrated control flow processing (e.g. NEXT, LEAVE, RETURN...) occur in the proper context.  Java has no way to provide such direct control flow processing (such as break, continue or return) without the editing block being defined as a nested block in-line.  This requires "inverting" the normal UI processing.  Normally, the server's UI runtime drives the logical terminal based on well known rules AND most importantly without any return to the calling code.  In this case, the calling code really has control and must drive the UI processing.  This block is treated as a nested loop (e.g. like a REPEAT inner block) which is a sub-transaction in TransactionManager terms.   The emitted block handles ERROR and retry but END, STOP and QUIT are ignored (handled only by containing blocks). Special methods of the referenced frame are called at specific points to ensure that the UI is invoked at the proper times:
  1. CommonFrame.startEditingMode() - this code is executed once at entrance to the block.  The UI uses this to initialize and prepare for the "inverted" UI processing.
  2. CommonFrame.waitForNextKey() - this code is executed at the start of every iteration.  It returns when there is a key available for processing.
  3. CommonFrame.continueEditing() - this code is executed at the end of every iteration.  It allows any pending GO events to be processed and state to be maintained properly.  This is critical to the processing of NEXT and any retry (due to the ERROR condition being raised) as well, since in such cases any pending GO event must be cleared.
Validation expressions are emitted as inner classes that extend the Validator class.  This is a class that implements the Validatable interface.  Each validation expression (as defined in a format phrase for a variable/field override or as defined in the data dictionary for a database field) is registered with the associated widget and is thus called back as needed during the UI processing.  If a field has a default validation expression in the data dictionary AND there is no override in the business logic, then a factory method (RecordBuffer.getValidatable()) is used to obtain the Validatable instance to be registered.  Any override for a database field in the business logic will result in an inner class being generated and registered, just as for variables.

The validation expression itself almost always will have a reference to the current field or variable AND this reference must resolve to the latest or proposed version of the variable/field.  To handle this, the new value is passed as a parameter in the Validatable interface.  The emitted inner class casts this to a local variable of the proper type and name such that the expression will naturally access the local version in its expression evaluation.  This also handles the special case in Progress where an array element is referenced in a validation expression without the associated subscript.

See also:

CommonFrame.java
GenericFrame.java
LogicalTerminal.java
ThinClient.java
WaitList.java
Validatable.java interface
Validator.java

rules/annotations/trigger_prep.rules
rules/annotations/validation_prep.rules
rules/annotations/validation.rules
rules/convert/ui_statements.rules

Key and Condition Processing

The thin client has a separate thread that handles key reading from the terminal.  This reader thread provides typeahead functionality and polls the keyboard every 100ms.  It can be suspended and resumed, which is necessary while handling redirected input streams.  These are cases where any interactive input to the terminal must be routed to the stream in use (often a child process, for example a "vi" editor session) and thus the thin client's key reading cannot be done simultaneously.

The key reader thread fills a queue of input events.  This queue is read by the thin client during editing mode (anything that calls WAIT-FOR) and during an explicit READKEY.  When read during WAIT-FOR, the key events are processed in the widgets via APPLY.  READKEY bypasses all such APPLY processing and returns the latest key from the top of the queue.

The key reader uses features in the CHARVA Toolkit class (some of which are backed by native methods that use NCURSES) to read the keyboard in "raw" mode with the "keypad" features activated.  This means that all keystrokes such as CTRL-C will be read without signals being generated.  The "keypad" support means that NCURSES will convert special ("extended") keys like function keys or numpad keys into simple single keys rather than passing through the multiple scancodes that the keyboard actually generates. 

CTRL-C is handled specially.  It generates a STOP condition AND this works in both synchronous and asynchronous modes.  While processing is occuring on teh thin client, a STOP condition is raised directly on the client and this exception unwinds the stack back into the business logic on the server.  When processing is active on the server, the key reader thread will asynchronously interrupt the thread on the server (via a callback from the key reader thread to the server) which will raise the STOP condition there.  This happens using the current thread's "interrupted status" and the interrupt() method in Thread.   If server's thread is blocked in some form of wait, then an InterruptedException, IOInterruptedException or other related exception will be generated and this will be caught and converted into a StopConditionException.  All such locations have been instrumented with the proper catch blocks to ensure this happens.  If the server thread is not blocked in a wait, the interrupted status is set and at critical points (start scope, iterate and stop scope) for each block the TransactionManager checks this status and throws the StopConditionException is the thread was previously interrupted.

See:

KeyReader.java
ThinClient.java

Limitations

  1. ENABLE/DISABLE/SET/UPDATE/PROMPT-FOR/DISPLAY
    • The KW_UNL_HID (unless-hidden keyword) is not supported.
    • Certain runtime configuration in the FRAME_PHRASE and FORMAT_PHRASE is not supported: AT COLUMN-OF/ROW-OF/X-OF/Y-OF.
  2. INSERT - no support exists.
  3. No "trigger phrase" support.  Regular triggers are fully supported but the "inline" triggers implemented in a variable or widget definition are not implemented at this time.
  4. Validation expressions for an array element that is referenced in the expression via subscript will fail.

Unreachable Code Processing

The purpose of the Unreachable Code Processing (UCP) is twofold:
  1. To eliminate code which during conversion may result in generation of Java code which will be considered unreachable by javac. Such code causes compilation errors and therefore must be avoided.
  2. To eliminate code which can never be accessed and thus is not actually part of the application.  This reduces the amount of conversion that must be done AND more importantly it yields a much better resulting application since it is smaller and easier to understand.
The reachability of statements depends on flow control statements such as IF, REPEAT, NEXT, LEAVE and so on. These statements change execution flow and under some circumstances may change reachability of code.

The approach used to determine reachability of code is simple: all nodes of the AST tree are visited and analyzed in order to determine if they can affect reachability of subsequent statements. The first node of the AST is always reachable.

Table below lists some examples of unreachable code:

Progress Code
 Java Equivalent
 Notes
 
repeat i = 1 to 10:
   ...
   leave.
   message "Hello!".
end.
 
for (i = 1; i <= 10;  i+= 1)
{
   ...
   break;
}
 The code after LEAVE is not reachable. 
repeat i = 1 to 10:
   ...
   if i > 7 then
      leave.
   else
      next.
   message "Hello!".
end.
 
for (i = 1; i <= 10;  i+= 1)
{
   ...
   if (i > 7)
      break;
   else
      continue;
}
 The code after IF statement is unreachable.
 
label_xx:
repeat:
   ...
   case i:
      when 1 then
         leave label_xx.
      when 2 then
         next label_xx.
      otherwise
         return.
   end.
   ...
end.
 
labelXx:
while(true)
{
   ...
   switch(i)
   {
      case 1:
         break labelXx;
      case 2:
         continue labelXx;
      default:
         return;
   }
   ...
}
 The code after CASE statement is unreachable.

Statements

A significant number of Progress statements may affect reachability of the other statements. Some of the statements do that directly, because they change execution flow (e.g. IF, CASE or LEAVE). Other statements, such as the PROCEDURE, FUNCTION or ON statements do not change execution flow. Instead they result in marking code as always reachable because its reachability can't be checked or depends on external conditions.

The processing of individual statements is described below:
  1. NEXT, LEAVE, RETURN, QUIT, STOP
    These statements stop the execution flow and make remaining code in the AST subtree unreachable.   Note that a reachable LEAVE statement makes code after the loop which contains this LEAVE statement, reachable. If the LEAVE statement contains a label then code after the labeled loop will be marked as reachable.
  2. PROCEDURE, FUNCTION and ON statement
    The procedure file may be a mixture of the procedure itself and internal PROCEDURE and FUNCTION statements. The reachability of these pieces of code must be estimated independently. This is necessary because a particular PROCEDURE or FUNCTION can be reachable even if it is located in the middle of unreachable code. The same is true for the ON statement.
  3. IF
    The IF in general case can make other code unreachable only if  both THEN and ELSE clauses are present and both make remaining code unreachable (for example, end with LEAVE or RETURN statements). Other special case is the situation when IF statement condition can be easily calculated. In this case code in the THEN or ELSE clauses of the IF statement can be unreachable. Also, if code in the reachable part ends with the statements from listed in #1, then code after IF statement will be unreachable.
  4. REPEAT, DO, FOR loops
    These loops may have easily calculable or implicit (REPEAT loop) condition which may result to infinite loop or loop which does not allow execution of code after the loop (there are exceptions, see #7).
  5. CASE
    The CASE statement can make other code unreachable only if it contains THEN and OTHERWISE clauses and all clauses end with statements listed in #1.
  6. EDITING and TRIGGERS phrases
    These phrases may (and usually do) contain code which ends with statements listed in #1. Direct processing of such statements may result in incorrect marking of large portions of code as unreachable. In order to avoid this, after processing of these statements the reachability flag is restored to the state which it had before processing of these statements.
  7. STOP, ERROR, QUIT and ENDKEY conditions
    These conditions may change execution flow and therefore they must be taken into account during processing. In particular, some loops (FOR and REPEAT) have default processing for these conditions. Also, all loops may have explicitly specified condition handling specified with ON phrases. This handling may include LEAVE or LEAVE <label> statements and therefore change reachability of the code after loops.
  8. Statements which may raise STOP, ERROR, QUIT and ENDKEY conditions
    These statements do not need special handling except that presence of such statements must be tracked because they may affect processing described in #7.
The expression evaluation mentioned in the #3 and #4 may calculate simple expressions such as:
  1. TRUE
  2. FALSE
  3. variable = "value"
  4. "value" = variable
  5. variable <> "value"
  6. "value" <> variable
Please note that at this time, the expression evaluation of variables is limited to variables of type VAR_CHAR being compared (EQUALS or NOT_EQ) with a literal STRING.  Changes to UnreachableCodeWorker will be needed to provide a wider range of support in the future (variable type and value type information would have to be stored and used in the variable pool and evaluation respectively).

Values and variables are stored in a customer-specific rule set named customer_specific_variables.rules.  This allows customer specific character variables to be specified with a list of values that can possibly evaluate to true . This allows such simple comparisons to be evaluated statically, yielding a increase in known unreachable code.

See also:

UnreachableCodeWorker
rules/unreachable/unreachable.xml

Produced Output

The results of unreachable code processing are stored directly in the AST: each AST node receives annotation "reachable" which contains a boolean value true or false , representing the reachability of the code. Also, information about each unreachable node is printed to standard output. This output then can be stored in the log file and then used for analysis.

Known Limitations

  1. At present TRIGGER PROCEDURE and SUBSCRIBE statements are not supported.

Unreferenced Tables/Fields Processing

The main purpose of the Unreferenced Tables/Fields Processing (UTFP) is to minimize amount of data which need to be moved to converted system by eliminating unused tables and fields.

The general approach of UTFP is simple: scan ASTs and collect references to tables and fields. This approach is complicated with following issues which need to be handled:
  1. Fields referenced in the LIKE clause should not be taken into account.
  2. An application may create a temporary table as a structural copy of an existing table. This case should NOT be handled as if original table and its fields are referenced. The reason is that this temporary table definition will be statically defined in the target application based on the structure of the Progress table however that table does not need to exist in the target database in order for the temporary table to be supported.
  3. Some statements or variants of statements may reference entire table (i.e. all fields in table). In some cases such a statement may explicitly exclude some fields from the list of all fields.
  4. Tables and fields can be accessed indirectly via system tables such as _File.
List below contains all statements which may reference all fields in table:
  1. ASSIGN record [EXCEPT field]
  2. BUFFER-COMPARE [EXCEPT field] 1,2
  3. BUFFER-COPY    [EXCEPT field] 1,2
  4. DEFINE BROWSE ... DISPLAY ...
  5. DEFINE FRAME ... [EXCEPT field]
  6. DEFINE QUERY ... EXCEPT 1,3
  7. DISABLE ... ALL [EXCEPT field]
  8. DISPLAY ... [EXCEPT field]
  9. ENABLE ... ALL [EXCEPT field]
  10. EXPORT record [EXCEPT field]
  11. FORM record [EXCEPT field]
  12. IMPORT record [EXCEPT field]
  13. INSERT record [EXCEPT field]
  14. PROMPT-FOR record [EXCEPT field]
  15. SET record [EXCEPT field]
  16. UPDATE record [EXCEPT field]
  17. DO, FIND, FOR, OPEN QUERY, REPEAT, function CAN-FIND:
  18. record-phrase EXCEPT [field]
Notes:
  1. Note if USING phrase is present then statement does reference only specified fields.
  2. May contain NO-LOBS phrase which excludes LOBS from the list of referenced fields.
  3. Field list after EXCEPT can be empty, this means that all fields are referenced.

See also:

DatabaseReferenceWorker
rules/dbref/collect_refs.xml
rules/dbref/apply_refs.xml

Known Limitations

  1. At this time, no processing of runtime preprocessor arguments is handled.  Since tables and fields can be specified via this mechanism, based on runtime calculated expressions, the range of possible values must be provided via hints.
  2. No tracking is done of references on indexes. Since indexes are built using fields this may add references to the fields as well.
  3. Perhaps better handling of the system tables is possible.

Open Conversion Issues

  1. Database
    • Unqualified field references (e.g. "display id" instead of "display customer.id") will reference an already created named buffer (define buffer ccc for customer) for the same table if there is no default buffer already created. (ECF: please doc and remove)
    • Raise the STOP condition for out of bounds array subscripts inside of where clause.
    • Implement runtime support for multi-table, mixed retrieval-type queries (e.g., REPEAT PRESELECT EACH a, FIRST b, LAST c) for:
      • PresortQuery
      • PreselectQuery (currently, we have an implementation which uses CompoundQuery in preselect mode to meet this need, but a more efficient implementation may be possible using subselects)
    • Temp-tables don't always get buffer scopes at the external proc as is done today, they can be scoped to specific nested blocks in Progress.  They probably can participate normally in scoping (today record_scoping.rules has some code at the top that hard codes temp-table cases to the external proc) which is an easy fix but needs testing.
    • Multiple buffer scopes are opened in the same block for the same buffer.  This doesn't make any sense.  See codes/crewcode.p.
    • recid(currentBuffer) in client side where clauses is incorrect?
    • Add an extent size query method for extent fields and change the current implementation in builtin_functions.rules which hard codes the extent value as a NUM_LITERAL.
    • The STOP condition generated by a DB disconnect is a special form that cannot be caught until the first block that doesn't access the database in question.
    • P2JIndexComponent needs type information, or at least we need to know when a component is a text component (JPRM watch point @32647).  Currently, this class' getNameCaseAware() method does not have enough information to wrap a text component in the rtrim() SQL function, unless that component also happens to be case-insensitive.  This hurts us with temp table indexes, because this method is used to create indexes on temp table at runtime, and these indexes will not be usable by the database, because they will not match the SQL we submit, which embeds the rtrim() function to match Progress' behavior.
    • Uncommitted transaction updates visibility.  This is implemented partially in that converted code can perform dirty reads when validating DMO changes.  However, a loophole still exists for new code which does not go through the legacy layer of the persistence framework.  This loophole must be closed.  We can probably extend our session interceptor implementation to handle this.
    • Implement secure login to DB server from Hibernate;  currently, userid and password are stored in clear text in the directory
    • look into validation of extent fields;  confirm correct behavior
    • There are cases where the 4GL will report an ambiguous name, but P2J does not consider it ambiguous. This probably means that we have been too aggressive about "promoting" namespaces in the SchemaDictionary.  In addition, there may be suppressed AmbiguousNameException instances in progress.g and SymbolResolver which further complicates this problem.
    • Our PL/Java-based, server-side function implementation currently does not take into account the following:
      • precision of decimal arguments and return values (test that PL/Java does deliver parms with the correct scale when set in DDL, check that our methods are returning BigInteger values with the right scale based on what is passed in)
      • case-sensitivity of character arguments and return values
      • timezone of date arguments and return values
    • upper() wrapping for case-insensitive fields (and literals and query subsitution parms)
      • is done too aggressively in where clauses/query subst (it should only be done for the immediate children of a comparison operator), which can cause things like this where clause example: "... substringOf(upper(myfield), 1, 5) = 'SOMETHING' ...", which works for this substring() replacement but for other built-ins it may not
      • is ignored for string concat operator and character functions (we rely upon the fact that all fields, literals and query subst data are already uppercased)
      • can be moved into the P2J runtime where it belongs, leaving the generated source code significantly better
    • rtrim() wrapping for character fields is done too aggressively in where clauses/query subst (it should only be done for the immediate children of a comparison operator), which can cause things like this where clause example: "... lengthOf(upper(rtrim(myfield))) > num ...", which would be incorrect
    • Date out-of-bounds issues.  Need to support database-specific date range checking for dates which are hard-coded into where clauses.  Currently, this check is only done for query substitution parameters.  A very rough out-of-bounds check is done at conversion time in order to log a warning message, but this is insufficient in the long term.
    • IS NULL performance may have issues
    • shared temp-table may leak memory in the UNDO implementation
    • Runtime error in server logs: JDBC error dropping a temp-table during session exit. This is more of an annoyance than a real problem as there is no persistent data to corrupt in a TT.
    • Runtime error in server logs: "Can't operate on a closed statement".  This is a warning that Hibernate logs sometimes when closing the delegate ScrollingResults object used by a ProgressiveResults object (converted FOR EACH loops).  Previous fixes may have made this better, but problems remain. One of the issues was in ScrollingResults.cleanup. What might cause this would be a race condition, where two threads call ScrollingResults.sessionClosing  at the same time - but I'm not sure this can happen; I mention this because, once the ScrollingResults.closed flag is set to TRUE, it no longer be set back to FALSE; only explication would be a race condition, so synchronizing the ScrollingResults.cleanup method on its object instance may be a solution.
    • Progress-compatibility of DB error messages should be fully provided.  This will require mapping info for the legacy DB names (in the DMO index?) and some changes to error processing in the runtime.
    • v10+ schema .df parsing will require updates as there are new options and syntax to support.
    • Rollback processing for foreign key association auto-synchronization.
    • More robust locking strategy for records resolved by the ForeignResolver during foreign key association auto-synchronization.  Currently, these records are locked exclusively, but only until their association object references are updated, then the locks are released.  This could cause deadlock problems during subtransaction rollback, if a lock previously held cannot be reacquired during rollback.
    • Disallow cross-database joins (PreselectQuery) during conversion.  Currently, PreselectQuery.addComponent() disallows adding cross-database query components at runtime as a last-ditch safety check, but by then this is too late.  This does not appear to be an issu, but should be addressed for the long term.
    • PostgreSQL Dependencies
      • PL/Java is used to implement server-side built-in functions.  We would need to create similar functionality for other databases.  Some vendors support Java natively for custom function implementations, but the level of support and the difficulty of the implementation is still unknown.  Other vendors (e.g., SQL Server) do not support the definition of custom functions using Java.  For these, we would have to create a PL/Java-like layer, probably using C/JNI.  For databases which do not support custom functions at all, we would have to convert differently (i.e., using client-side where clause processing instead of embedding server-side functions in HQL).  This is the least desirable option, both because the performance would be unacceptable and because the converted output would differ from the standard.  These database implementations are effectively removed as candidates for a scalable, converted system.
      • During schema conversion, indexes are used to implement non-nullable constraints.  Must research whether this is the case with all databases, or whether a different approach would be necessary.
      • During data import, we issue an "analyze" command after each table is loaded to generate statistics, allowing subsequent queries against that table to be faster.  This is specific to PostgreSQL.
      • Not necessarily specific to PostgreSQL, but some databases do not support this needed feature:  PostgreSQL allows the use of expressions when defining table indexes.  We leverage this feature by wrapping text columns in the rtrim() SQL function to match Progress' behavior, and for case-insensitive text components, we further wrap that expression inside the upper() SQL function.
    • PostgreSQL Limitations
      • Collation strategy cannot be made dynamic.  Collation behavior for a database cluster is determined at database installation, by using a custom locale during initdb step.  It may not be changed thereafter.
      • No support for mixed ASC and DESC indexes.  Progress allows an index to contain multiple fields which sort in different directions.  PostgreSQL (and possibly other databases) do not by default.  This will result in performance penalties (possibly severe, for large tables) in converted applications which rely on such mixed direction sorting.  Currently, we have no workaround for this issue for PostgreSQL, which has no syntax for specifying sort direction for index columns.  There may be a way to leverage custom operators to assist with this, but further research is required to determine if this in fact has potential for the development of a solution to the problem.  Note:  PostgreSQL v8.3 is supposed to add support for mixed direction indices, but we will have to change our data import processing to handle this capability.
      • PostgreSQL query planner doesn't take NOT NULL constraint into account for field F when optimizing queries such as select ... from ... where F is null...  Instead of returning an empty result set immediately (since it is impossible for F to ever be null based upon the NOT NULL constraint), it does a sequential table scan (which can be very expensive for a large table, since every row must be visited to determine there is no such record).  There may be a way to tune the query planner to be smarter about this.  Another alternative is to create a limited range index (i.e., create index i on table t where id is null).  This will result in a very fast response to this type of query, but it has drawbacks:  it adds another index to maintain and is not portable, since not all databases will support a where clause in a create index statement.
    • Progress index bracketing bug we do not replicate:  during where clause processing, expressions (or portions thereof) which test whether a field (of any type) is less than, or is less than or equal to, unknown value (as a constant or as the result of a sub-expression) will trigger an index bracketing bug.  The result of this defect is that all records which would be matched had that subexpression evaluated to true, are included in that index bracket.  We have determined that this must be an index bracketing defect rather than an expression processing defect, because the same sub-expression on a stand-alone basis (i.e., outside of a where clause) behaves normally.  That is, it matches the normal expression processing rules for comparison with unknown value.  An illustration of this defect is provided with P2J uast testcase where-clause-index-unknown-value-bug.p.  This defect was found using Progress v9.1C.  We do not know if it is present in other versions.  At the time of this writing, there are no plans to mimic this obviously incorrect behavior, since it seems unlikely that there are legitimate use cases in production code which would rely upon it.  If any cases are found which inadvertently rely upon this behavior, it is more likely that the Progress source code should be fixed.
    • Unknown value embedded within a where clause:  currently, we "inline" the most common unknown value references within a where clause, both static references (i.e., a direct comparison between a database field and the ? symbol), and variable references (i.e., a direct comparison between a database field and a variable set to unknown value at runtime).  "Inlined" here means these comparisons are replaced with the appropriate {database_field} is [not] null clause in the HQL which is submitted to Hibernate.  In the static case, the replacement is done at conversion time;  in the variable case, at runtime (by the HQLPreprocessor).  However, several corner cases remain unaddressed:
      • The unknown value reference (static or variable) is embedded in a non-logical subexpression of a direct field comparison, such that the evaluation of the subexpression would result in a comparison of the database field and unknown value (rather than the more correct {database_field} is [not] null idiom).  These cases can be addressed by expanding our replacement/inlining coverage in both the conversion and in the persistence runtime.  Examples:
        • where ... some-database-field >= ? is handled
        • where ... some-database-field >= some-var (where some-var is set to unknown value at runtime) is handled
        • where ... some-database-field >= (? + 5) (however unlikely) is not handled
        • where ... some-database-field >= (some-var + 5) (where some-var is set to unknown value at runtime) is not handled
      • Unknown value (as NULL) is passed from the database into a server-side (e.g., PL/Java) function.  It is unclear whether this is truly a problem, or whether the server-side expression processing within a database already correctly handles this situation.  If truly an issue, this problem potentially could be addressed by wrapping every such instance with case...when... handling logic, though the impact on performance would have to be considered carefully.  I suspect this would add noticeable overhead (and could provide for some very tortuous HQL), but since embedding the database field within the PL function presumably already precludes any index-based optimizations, the incremental hit may be acceptable.  Note that this is only an issue for nullable database columns (non-mandatory fields in Progress-speak), so we may elect to at least flag potential problem areas during conversion by recognizing the condition during where clause conversion and issuing a warning to the conversion log.  Example:
        • where ... field-1 = dec(field-2) (where field-2 is not mandatory and potentially represents unknown value) is not handled;  although the implementation of the dec() function correctly returns unknown value, this leaves us with an equality comparison between a database column and NULL inside the SQL database server, when it should really be processing field1 is null.  TODO:  test whether such processing in an RDBMS really differs from Progress.
    • TODO: currently, when there are multiple buffers defined for the same table, to fix the rollback processing, the BufferManager keeps together all reversibles related to the same table belonging to the same database. This ensures proper reversible processing on rollback. But, the rollback related part in Commitable can be reworked to allow multiple buffers for a certain table in a database. This will eliminate the small overhead of rolling back a buffer which has nothing to do (rollback was performed by one of its siblings).
      Note: a case which needs to be taken into consideration and checked is when the foreign-keys are enabled. In this case, if the child is rolled back first in a Child.Parent relation, there might be some problems.
  2. UI development and fixes
    1. Review all TODO markers in both ui and chui packages.  Prepare a report on any items that are left.  We will discuss the priorities.
    2. Make a list of all non-condition exceptions generated in the thin client.  Which of these should be translated into proper Progress ErrorConditionException (use ErrorManager.recordOrThrowError())?
    3. Using testcases/uast/on_entry_no_apply.p, the on-entry event is captured for field2 and returns no-apply. After leaving field1, in Progress field2 is skipped and field3 is focused. In P2J, field3 never gets focused, focus remains on field1.
    4. Using testcases/uast/trig-wait.p, type 3 in the first field (i) and then press enter.  This will cause a pause.  If you use CTRL-C there, it will abort the pause but it won't propagate the stop condition all the way out (exiting the app) like it does in Progress.
    5. at_base_field_max_quirk10.p right aligns a number when it should not.
    6. at_base_field_max_quirk5.p has a prompt-for that should truncate the data using format "x(2)" but does not.
    7. Number editing in P2J seems to disallow backspacing over the leftmost char but in Progress this is possible.
    8. Remove frame duplication (detect frames that are functionally identical and reuse the same definition in all business logic rather than creating a new one).
    9. Remove duplicate frame.openScope(), validation expressions (search on Validation2 and widgetDaysSupply) and frame elements from the business logic.
    10. Constant (literal) header expressions can be emitted in-line (via alternate constructors for HeaderElements) instead of as inner classes.
    11. Add ControlSetItem(String) and ControlSetItem(character) constructors.  They can simply use the same object as the value and label.  Investigate using Strings instead of the character for the label (this can't be done for the value, which must be a BaseDataType).
    12. Remove \n label packing code if it is unneeded.
    13. frame_generator.xml does not need the add_format function (it's results are not ever used).
    14. Remove duplicated code in frame definitions.  For example, there are often multiple outputs of the same setFormat(...) for the same widget are added to the frame def for @ base fields.
    15. Create and use (in frame definitions) common case widget constructors that take things like: label, format string, data type, legacy name... so that the majority of the setter code in the frame def can be eliminated (reduced to some constructor parameters).
    16. redirected_output_buffering.p output differs from 4GL (see SIY).
    17. CHOOSE issues:
      • field mode cursor movement is sometimes broken
      • row mode group navigation is sometimes broken
      • bell is not always occurring when it should
      • cursor drawing is sometimes wrong
    18. hidden attribute/visible issues
    19. handle data type cannot be displayed in P2J
    20. MESSAGE statement error handling deviations:
      • apply error in a called user-defined function just displays "?" instead of propagating the error further (is this a more general issue?)
      • page-number(s) on a closed stream displays an error and then a 2nd line with "?"
      • fields need ErrorManager warning mode
    21. MESSAGE ... VIEW-AS ALERT-BOX deviations:
      • conversion issues related to TITLE in a ALERT-BOX (when the TITLE is a character expression, not constant). testcase: alert_box_width2b.p
      • an edge case when the max length of the message lines is 20 and a SKIP is used. testcase: alert_box_width2.p
    22. UI performance issues:
      • optimize multiple screen definition updates (which have no intervening visible change to the user)
        • pushScreenDefinition() can be deferred until at least GenericFrame.openScope()
        • by using the state sync approach, these can even be queued up/batched which is probably the better approach
      • alternatively, consider pullScreenDefinition() call made from the client when a new frame is about to be instantiated
        • it acts as an ultimately deferred pushScreenDefinition
        • screen definition won't be transferred if not required by this specific run of the application
        • repeatable calls will be eliminated since the client will maintain a registry
      • optimize single value (non-batch) screen definition updates
        • dynamic setting of screen definition values can be "broken out" as separate calls to the client to handle application of specific values only, instead of using the pushScreenDefinition() to apply the change
        • change batching can be redesigned to batch individual calls as "orders"
        • opportunities:
          • the COLOR statement (setColors)
          • runtime frame options (dynamic expression use in frame phrases):
            • setDynamicTitle
            • setColumn
            • setRow
            • setDown
            • setDcolor
      • Drawing should never be done during pushScreenDefinition!  Even if something has changed, it should never be visible to the user until the next view()... so this is wasted effort and potentially incorrect visually.  Currently the COLOR stmt relies upon this behavior, but if split off as a separate client export, this would be avoided.
      • eliminate extra drawing and screen buffer processing in ThinClient.promptFor()
      • check on whether the following is still true:
        • Toolkit.blankBox is called by containers to clear all contents every time draw is called, even when this is completely unnecessary.
        • Toolkit.blankBox is called by sub-containers even when an enclosing container has already called this.  For example, Window calls this, then Container calls this and then Scrollpane calls this...
        • Toolkit.blankBox is called 3-9 times for every end-user keystroke in LoginClient.
        • Toolkit.blankBox is called hundreds of times after the last keystroke in LoginClient.
      • creation of the frame proxy (Proxy.newProxyInstance()) in GenericFrame.createFrame() is expensive because of the large size of the CommonFrame interface
        • cache/pool instances and reuse them over time?
        • use something like CGLIB to do the instrumentation?
    23. FillIn UP/DOWN key movements shift the cursor to widgets above/below however there are cases in which the cursor is placed in the first valid cursor position instead of being placed in the exact column in which the cursor resided in the previous field.  If there is data in the position being targetted, then the cursor will move there.  Note that there is no issue with cases where the cursor is supposed to move to the beginning of the field (this works). UP key applied to a fillin in the top frame row moves the cursor to the first fillin in this row. Movement from fillin to a non-fillin widget and back with UP/DOWN keys should set the cursor position in fillin to the same as it was before we have left it.
    24. Hiding focused widget hides the widget itself, but the cursor remains on its position while it should move to the nearest widget.
    25. Cursor movements are constrained more than in Progress in number editing. See editing5.p.
    26. Converted help_precedence3.p testcase fails to compile.
    27. In the testcase testcases/uast/convert_hide.p HIDE <multiple widgets> statement is converted incorrectly so the converted testcase cannot be compiled.
    28. input_lastkey.p shows that certain terminal types have the initial lastkey value set to -1 (vt220, vt320) instead of 401 (xterm)
    29. Using SESSION:DATE-FORMAT to change the date component order dynamically will not be picked up by existing instances of date (on the client side) nor by existing instances of DateFormat (client side).  See date_component_order.p.
    30. FillIn bugs with NumberFormat (see testcases/uast/num_edit.p):
      • Deleting or back-spacing over last char in P2J moves cursor out of the field but in the 4GL it does not.
      • Overwriting the last char properly moves the cursor out of the field BUT subsequent keystrokes should BELL.  Instead P2J insert characters and shift everything left.
      • If you are in overwrite mode in one field and then use CURSOR-RIGHT to move the cursor into the next field, the 4GL DOES NOT auto-zap but P2J does.  So in "99,999", the first character typed (e.g. '4') after cursoring in will yield a "40,000" in the 4GL and a "00,004" in P2J.
      • Extra space added to the right of the "left user text" in an uninitialized field with formats "+>>>>>>>", "$(>>>>>)", "$>>>>>" (and others).
      • Pressing '0' in an uninitialized "(>>>)" field causes the cursor to be re-drawn outside and far to the right of the field BUT subsequent key presses enter data properly into the field and reset the cursor to the right position.  Another example of this same problem: in a ">>,>>>" field, enter '0', '1', '9', '1'.  This will result in 191 in the field BUT there will be a temporary misplacement of the cursor between the '0' and '1'.
      • "-999" allows interactive editing of the '-' sign character when not-negative.
      • A field with a 0 value and a format with an optional sign character (e.g. "-999") will display with '-' if the user explicitly presses '-' to make the value negative.  In other words, a negative 0 can be displayed as "-000".
      • Fields with formats that have a sign character (e.g. "-999") don't allow using arrow (e.g. CURSOR-LEFT) keys to move the cursor into the sign character but can in the 4GL.
      • Can't enter data on left of decimal point if not in insert mode for decimal formats: "zzz.99", "-zzz.", "+zzzz.999", "(zzz.9)", "zz,zzz.999".
      • Initial cursor placement in format "zzzzz+" should be on the right (the + char) instead of on the left.
      • Initial cursor placement and subsequent cursor draws for "$(zzzzz)" format all draw 1 char too far right.
      • Cursor placement is too far right in format "-zzz,zzz,zzz" after activation and insertion of a single "0" character (in the rightmost position).
      • integer fill-in with a value of "1017" and format of "zzzzzzz", delete on the first digit results in "17" in P2J but "017" in 4GL
      • integer fill-in with a value of "1000" and format of "zzzzzzz", insert mode off, disengage auto-zap (cursor left/cursor right, for example), move cursor to digit "1" and type "0"; P2J compresses all 0's into a single "0" while 4GL just replaces "1" with "0" for a result of "0000" (similar root cause as above with 1017)
      • integer fill-in with a value of "0" and format of "zzzzzzz", delete on "0" results in "0" in P2J but clears the widget completely  in 4GL
      • integer fill-in with a value of "1" and format of "zzzzzzz", insert mode off, attempts to enter mode digits must fail while P2J behaves as if insert mode is active
      • Decimal cursor placement is incorrect when all digits to the left of the decimal point are zero suppressed.
      • The sign and/or left user text appears 1 char too far left with "+>>>>>>>", "$>>>>>", "$(>>>>>)".  The cursor position is also wrong in this case.
    31. A failure during auto-complete of a date field should display 00 in the day or month (depending on the field being completed) and the current year in the year portion.  In P2J we draw blanks instead.  Note that only uninitialized fields are affected by auto-complete.
    32. Eliminate cursor flashing during drawing.
    33. Name skip/space widgets skipX or spaceY instead of using exprZ.
    34. Expression widget numbering should use a per-frame counter instead of one that is per-file.  For example, a given frame should not have its first expression widget named expr17 because there happened to be other frames defined in the same file.
    35. Use a single alignment approach for widgets.  The format phrase usage of AT, TO and COLON should emit to the same thing as LEFT-ALIGN, RIGHT-ALIGN and COLON-ALIGN.  The widget interface for this should be setAlignment(int) where there are GenericWidget.LEFT_ALIGN, GenericWidget.RIGHT_ALIGN and GenericWidget.COLON_ALIGN constants passed as a parameter.  The setTo(), setAt() and setColon() would be dropped.
    36. If a combo-box is defined without a size phrase, its width is determined by the rendering size of the format string.  testcases/uast/combo_box/combo_box7.p shows this case.  Note that the BaseDataType.calcFormatLength() and BaseDataType.formatLength() can be used to calculate the rendered size of a format string.  This is the size of the variable portion of the combo-box.
    37. The "thumb" button on a scrollbar positions differently.  In particular, the algorithm that determines when it moves and where it is to be positioned is slightly different.  A user that is not comparing the two implementations side-by-side may not notice this, but in P2J the thumb button will move at different times and may be positioned differently at any given time.
    38. Determine the nature of some unexpected Progress behavior when displaying uninitialized field level widgets in the testcases/uast/uselect2.p testcase. When using a 2 DOWN frame, the selection  list is shown as uninitialized for the iteration 1 and 3, and as initialized for the iterations 2 and 4. This is expected. However, starting with the iteration 5, all subsequent iterations show the selection list as initialized. This does not happen when using 3 down frame, for instance.
    39. The double buffering output driver has solved many subtle issues in PUT SCREEN handling, but not all of them. There is a class of situations where it fails, backed by these testcases: put_screen_flush*.p. These are the observed facts and their interpretation:
      • in Progress, there is a delay between some frame output and copying that output to the master buffer, which is used to optimize the subsequent output;
      • this delay affects the way the PUT SCREEN output gets cleared naturally with the frame output;
      • this delay also affects the way the final frame picture looks like with the mixture of PUT SCREENs;
      • if no PUT SCREENs are involved, this delay does not cause any difference in the output besides the degree of the output optimization, which can be measured as the number of characters in the produced diff;
      • the reason for the delay is that Progress saves the frame output to the master buffer only when a frame switch occurs: any operation that performs output to a different frame B, comparing with the previous such operation with a frame A, saves the output of the frame A at this moment;
      • if the frame B from the case above is an overlay frame being displayed on top of the frame A, then the existance of the delay directly affects the calculated diff for the frame B, since this calculation could have been done either with the previous contents of the master buffer or the most current ones;
      • the testcases clearly demonstrate that the diff calculation happens before the update to the master buffer; that's why the frame B looks differently when displayed for the first time after displaying A and a PUT SCREEN, and the second time, when the master buffer has been updated yet;
      • the right way to fix this issue in P2J is to separate the regular syncing of the output in sync() method of the DoubleBufferedTerminal.java from copying that output to the master buffer, which happens simultaneously in today's implementation. sync() should produce the diff based on the existing contents of the master buffer and leave it unmodified. A new method, say, flush(), should be added, which updates the master buffer. This method should be called from the view() method whenever the latter detects the frame switch;
      • a fix as described has been attempted but had to be backed out due to massive regressions in PUT SCREEN output and probably beyond. Separating sync() from the master buffer update is a very sensitive operation with the current implementation of the frame output. Implementor of the fix should be ready to face it.
    40. Browse issues. You can view reproductions using the browse_issues.p testcase. The reproductions theirselfes are placed in the code of this testcase and each item below references a reproduction from it.
      • In 4GL you cannot change the READ-ONLY attribute while the browse has focus (message "You cannot change the READ-ONLY attribute for BROWSE {browse_name} while that browser has focus. (4517)" will be displayed). In P2J, if you will change read-only mode while the browse isn't focused, you will get full-screen graphical artifacts. (see reproductions c9, c10 from browse_issues.p)
      • In 4GL, if you will close the query while the editable record has invalid value (does not conform browse VALIDATE statement), then "No query record is available. (4114)" and "Warning: Progress cannot get the row you've just begun editing with a Share-Lock. (4316)" messages will be displayed, the active record will remain on the clean browse area. Further actions in this state may lead to a non-intuitive behavior. (c11)
      • In 4GL, if you will close the query while the editable record has invalid value (does not conform index), then the empty browse area will be displayed. Further actions in this state will lead to the normal browse "no query opened" state. (c12)
      • In, 4GL BROWSE:REFRESH() function, 4GL will roll back the old value of the edited field, but you will leave the field, the new value will be displayed again. Depending on the validness of this value, it will be committed or not (i.e. you may have invalid values in a browse, but on the next refresh, they will go away). (c13)
      • Validation option into DEF BROWSE .. ENABLE .. VALIDATE doesn't work in P2J. (c14)
      • Added records shouldn't be displayed into browse after BROWSE:REFRESH(). (c15)
      • Browse shouldn't appear on repositions or on BROWSE:REFRESH() if it is currently hidden. (c16)
      • After the backing query of an editable browse has been reopened, you cannot edit browse cells anymore. (c17)
      • Browse receives focus too early. (b1)
  3. final runtime development:
    1. Client side support for:
      • DirectoryService API (note NumberFormat needs to use the proper getGroupSeparator() instead of using direct access to NumberType.GROUP_SEP...)
      • Logging (in ThinClient, ErrorManager...).  Provide a custom remote log class with simple methods to handle the remote entries on the server.
    2. Directory API needs improvements.  For example, one would think that adding a single string node into the directory could be easily done with a single API call.  If you look in the API, you will find addNodeString().  This does not do what you might think (even after reading the javadoc).  So the docs are confusing and the simple API is missing.  Something similar to the MinStartupDirectory.addString() method is what would be useful.  All basic datatypes should have such simple helpers.
    3. Default P2J directory exports should not be directory driven otherwise simple/stupid admin mistakes can kill the server.  Shift the directory to a RemoteObject approach.
    4. The security manager must provide warnings before certificates actually expire so that production installations have adequate time to handle this issue.  This warning MUST NOT be only at startup as a server may run for months crossing the expiration boundry with no warning...  The lead time should be customizable via a directory entry.
    5. Review the state of output and input-output parameters for procedures/functions when the block aborts due to an error.  It is possible that in Progress the assignments never occur but in ours the assignments are done "as we go".  We may need to handle this on exit as a special kind of undo OR we can use temporaries and do a batch assign at the end (this may have negative consequences for database validation...).
    6. Constructs like opsys = "VMS" are not being recognized as dead code.
    7. Performance tune stream processing, especially redirected terminal mode (report generation can be slower in P2J by a noticeable amount of time).
    8. The build.xml is broken (it always compiles all frame classes, even when they have not changed).
    9. Logging improvements:
      • Review all project files for:
        • Open logging todos (e.g. // TODO: log this) and implement the proper logging.
        • In any location which already logs, make sure that any exceptions are not just concatinated via toString() or "+" operators but instead are passed directly to the Logger.log() method call.
        • For any logging calls that are expensive (that do string concatination, data lookups...) should be wrappered in an if (logger.isLoggable(LEVEL)) to decrease the overhead when logging is disabled.
      • Add trace level logging to the TransactionManager in key locations.
      • Shift ECF log usage to the standardized approach.
      • Add socket endpoint info (IP, port) into LogHelper.describeContext().
    10. Invent a strategy to obtain a certificate alias for a remote server using a reverse lookup of hostname/IP address and port/service name.  This is required if we want to support requester-side validation of a remote server's certificate in server-to-server connections (i.e., virtual sessions).  See [Router]SessionManager.connectVirtual(InetSocketAdress address, String certAlias, SessionListener notify) in the net package.  Currently, this method disables this check if a null certAlias is provided, which is the common case.  But there currently is no mechanism for the caller of this method to easily determine what to pass for a remote server's alias.  Currently, the directory contains a default/runtime/port_services path to enable lookups of port number by service name (since Java offers no standard way to do this).  We would probably want to merge this into a larger table that took hostnames/IP addresses and certificate aliases into account as well.
    11. Rename SecurityManager to SecurityMgr to avoid the name collisions with the java.lang version.  (NOTE [ECF]:  I disagree;  I don't like class names containing abbreviations -- also note it violates our coding standards.  The collision is easily resolved with an explicit import statement for the P2J SecurityManager.  You will rarely (if ever) need to access both in the same source file.  In fact, over 10+ years, I have never had to use java.lang.SecurityManager in normal Java development).
    12. Move the following into com/goldencode from the com/goldencode/p2j location:
      • cfg (at least the bootstrap config stuff)
      • the generic (non-Progress) util/ stuff
      • directory/
      • net/
      • security/
      • the non-Progress stuff in main/
      • As part of this work:
        • make sure that any dependencies on Progress-specific stuff are removed if otherwise a class would be independent
        • remove dependencies between packages where possible (especially dependencies on the UI package like ThinClient and LogicalTerminal (example: the use of LogicalTerminal.message() and ThinClient.message() from the util/ErrorManager)
    13. The SSL handshaking procedure of the protocol sends no client certificate to the server even when one is specified.
    14. Is any usage of a custom locale needed for purposes of non-database "character" type sorting and comparisons?
    15. RETURN ERROR should not work in a function (it should be like a RETURN ?).  Only procedures and triggers honor RETURN ERROR.
    16. unknown value in a CASE statement doesn't work as in 4GL
    17. each file has 2 entries in the registry.xml (with ./ prefix and without it), eliminate one (and fix whatever code is dependent upon it)
    18. p2j/testcases/uast/ieee_754_representation_issue.p shows a deviation where the inexact representation of Java's double causes two (different but close) double constants in the source program to be determined as the same number when they really aren't (this may only occur with numbers that are dependent upon the 11th digit to the right of the decimal point when there are 6 digits on the left --> thus causing the total number of representable significant digits to be too large for double to handle)
    19. Add BaseDataType constructors that allow a string name (which if non-null) would force a SharedVariableManager registration inside the constructor.
    20. frame_hiding02.p demonstrates TM behavior (probably in infinite loop protection) that is incorrect.  In particular, in Progress we see 0 1, 2, 3 on the first row of output but in P2J, there is a 0 1 and then the loop iterations end prematurely.
    21. helloworld.p has an NPE when run because of a conversion error in array var initialization.
    22. string_perf.p has an unreachable issue (failing code commented out).
    23. progress.g has a parsing issue when encountering a format string without quotes and a date format has more than 2 characters in the 1st or 2nd date component like 99-9999-99.
    24. scope promotion issue: update c1 c2 = c1 + "text". will create an EmbeddedAssignmentExpr which cannot resolve var c1
    25. Write directory merge tool.
      • See name_map.xml created by rules/annotations/name_map_helpers.rules and rules/annotations/collect_names.rules.  This is a file with a merge tag that stores the attachment path in the main directory.  One good result of this approach is that the directory is always maintained as a proper directory and any other merge files can be integrated one by one or in a batch.
      • An alternate approach advocated by NVS was to create a master "template" directory that defines includes.  This is more of a pull concept.  It would require that all of the dependencies are resolvable at merge time, otherwise the master file cannot be turned into a proper directory.
    26. The XML schema for the directory is unnecessarily complicated.  While it makes sense to use child elements to multi-valued attributes, for the primitive node types, it is complete overkill.  A single string node can be specified as <node class="string" name="mynode" value="stuff" />. 
    27. Retry processing quirks:  when validation of a new temp-table record fails due to the buffer being flushed in preparation for a query (loosely meaning FIND, FOR, etc.), the 4GL displays more validation error messages than one would expect, if outside a transaction.  We do not emulate this behavior in P2J.  Note that the number of messages varies depending upon the existence of previous message and pause statements in the procedure.  See testcases uast/flush-by-query[1-4].p.  When inside a transaction, we behave the same as the 4GL (a single error message).  See testcases uast/flush-by-query[5-9].p.
  4. Build administrative tools.
    1. provide editing support for the server, security... configuration values in the directory that cannot be edited today
    2. finish the runtime/console mgmt features
    3. add a directory editor to the admin GUI

Planned Optimizations, Refactoring and Future Development

  1. Dead code removal:
    • Unreferenced variables removal is simplistic at this time and there are cases which can be removed but are not:
      • new shared vars that are never used downstream (requires call-graph based processing)
      • we simply check for any node that has a refid to the var def, but there are cases that are not real source code references so the vars stay around (but are really dead anyway)
    • Unreferenced streams, frames, buffers... all can be removed too.
    • conditional expressions that always evaluate true or false
    • remove concatenation of empty strings
    • use of GUI-specific stuff in a CHUI app
    • use of persistence in an app that never uses run persistent
    • Convert "for each table: delete table. end." into a simpler form (a single method call to something like RecordBuffer.deleteAll(table).  Some other forms would be possible too, taking a where clause... to allow deletion of all records matching is given criteria.
    • Convert "for each table: i = i + 1. end." to something like RecordBuffer.count(table).    Some other forms would be possible too, taking a where clause... to allow a count of all records matching is given criteria.
    • IF/ELSE unreachable processing needs to promote the THEN/ELSE child nodes into the enclosing block.  This can be single statements (e.g. even "blocks" like REPEAT) or complex groups (any case of a simple DO block).  In the complex group case, all children of the simple DO block are reparented into the enclosing block at the index position of the statement node (STATEMENT/KW_IF...).  Today such things are put in an unnecessary DO block.
    • Deeper analysis of IF/CASE and other condition expressions. For example, a logical var that is initialized or assigned false and then never otherwise changed (or passed as an output or input-output parm) is effectively a constant.  This can be used to detect a larger amount of dead code.
    • An example of expressions that can be cleaned up:
      • this code:
        • for each itemloc where itemloc.item = cur-item and itemloc.site = (if "" <> "" then "" else itemloc.site)
      • can be converted into this:
        • for each itemloc where itemloc.item = cur-item
      • these cleanups are not limited to where clauses, but one should note that any expression rewriting must ensure that the index selection that would normally be driven by a more complex where clause is not broken (see ECF or GES on this)
    • unreachable processing should be rewritten to run in a loop (and/or using ascent rules) to detect and "rollup" dead code.  The issue is that if all contents of a particular IF block turn out to be unreachable BUT the IF *is* reachable, then the IF can be removed.  This is handled by a fixed number of multiple sequential unreachable cleanup runs in annotations today but would be better handled by a loop until no more removals can be found in a given pass.
    • variable definitions made in unreachable code actually still exist and can be used in Progress code (note that this is tougher in cases where there is a message statement or format_phrase var definition because there are other side effects)
    • dead code can have an effect on frame formatting (e.g. format string processing and width calcs for @ base fields) because the removed code, though dead in Java, is still processed by the Progress compiler --> the order of processing can change formatting (see testcases/uast/deadcode_vs_frame1.p)
    • frame which is completely scoped to the dead code, can be safely removed along with the dead code (requires taking into account frame scoping before dead code removal)
    • LEAVE/NEXT rewriting in annotations should be moved into fixups.  This will allow simplification of the code handling this as well as improvements in unreachable processing (which happens after fixups and before annotations).  For example, there is a case where an unlabeled next that isn't in a loop acts as a return, but today unreachable code processing doesn't detect this (see testcases/uast/next_as_return.p).
    • Detect cases where a handle is not shared AND is not passed as a parameter AND is only assigned a single type of object.  In this case, replace the usage with the object directly and remove the handle var.  The only other problem is to ensure that the valid-handle() method is not being used on that handle.
  2. Expression rewriting:
    • call i.increment() and i.decrement() instance members for expressions like i = i + 1
    • convert x <> ? to !x.isUnknown() and convert x = ? to x.isUnknown()
    • constant expressions that can be evaluated at conversion time
      • constant array indices can be decremented by 1 instead of written as an expression
      • 3 + 1 * 2 would be converted as the literal 8
      • "Hello " + "World" would be converted as "Hello World"
    • Convert > 2 chained string concatenations from the character.concatinate() "operator" to chained StringBuffer.append().
    • if the same string literal is used in multiple places (in the same scope), implement a String variable and then reference
  3. Remove unnecessary wrapping/unwrapping for expressions that where the primitive results could be directly used instead.
    • When a CompareOps method is used directly in a control flow statement such as IF, rework the method name into one which returns a boolean to avoid the need to wrap in CompareOps and then unwrap in calling code.
    • Allow fields and wrappers (vars) to be compared directly from primative Java types (e.g. int).  This can easily be done with character and String but there is probably value in overloading the equals() and compareTo() methods... CompareOps and other changes may be needed too.
    • Allow fields and wrappers (vars) to be assigned directly from primative Java types (e.g. int).
    • These improvements may all the BaseDataType class to have the knowledge of the logical class removed if the unknown value testing does not need to be there anymore.
  4. Wrapping and unwrapping could be implemented in a more simple manner by always marking the literals for wrapping rather than marking the parent.  Perhaps other approaches might also simplify this.  Right now this mechanism is too hidden, obscure and complicated.
  5. Provide alternate signatures for LogicalTerminal.message() that allows BaseDataTypes to be passed instead of an AccessorWrapper.  Then change conversion for var refs to eliminate the "newAccessorWrapper(var)" and replace it with a simple "var".
  6. ControlFlowOps.setReturnValue("") can probably be done from inside the TransactionManager at every top-level scope open.  Then this code can be removed from the business logic.
  7. Progress 4GL if/else if/else if/else does not emit as a Java if/else if/else if/else but instead as an if/else { if/else { if/else {] }} which is sub-optimal from a readability perspective.
  8. Convert Progress 4GL if/else do: if/else end. into a Java if/else if/else construct which is easier to read.
  9. Functions should be analyzed to detect if they can ever trigger conditions to be raised such that they would need the TransactionManager infrastructure.  If not they can be emitted as much simpler methods.
  10. repeatEmit is probably no longer needed in literals.rules.
  11. The parser should implement format_string as KW_FORMAT^ STRING instead of KW_FORMAT^ expr.  This was done when processing initially because of hitting something that didn't parse right. At this point, the parsing of these obscure areas has been improved greatly such that this is no longer an issue.  The project must be reviewed to understand if this can be done.  Then all the rule sets would have to be modified to handle the missing EXPRESSION node.
  12. Format string constants that are not referenced in the business logic should not exist.
  13. The extra block if not an "integrated_simple_do" is not always needed in control_flow.rules.  This results in an unnecessary set of { } output in many blocks in the Java code.
  14. Eliminate undo properties from vars that don't need it.
  15. Hide undo registration (possibly inside the wrapper constructors).
  16. Emit an accessor call to the buffer (which returns a Resolveable or a  instead of generating new FieldReference(buffer, "field").
  17. Long lived user/process contexts will probably require us to deregister global resources (e.g. Streams) that are closed rather than leave the reference around in the finalizables list.  Or we can use a weak hash map.
  18. Reset the counter for "condition" and "editingLabel" in <init-rules> for annotations rule sets so that each file will have a 0 based set of indexes.
  19. Remove unused/unreferenced frames, streams, buffers, queries, variables...
  20. Replace all usage of @inheritdoc with explicit coding of the javadoc code, even though it duplicates text.
  21. Rename frame definition callback setup() instead of setUp().  Move the batch(false) call out of this setup method and back into the GenericFrame class which calls this method (there is no reason to emit this when it will always be needed anyway).
  22. The chui package's FillIn formatting classes (DateFormat...) should maximize the use of the BaseDataType formatting support instead of duplicating it.
  23. Constant variable detection and rewriting as final statics.
  24. AdaptiveQuery should be more restrictive in its decision to invalidated a preselected result set in the case of record deletes and insertions.  Specifically, these cases should only invalidate the result set if the deleted/inserted record would actually affect result set traversal.
  25. Query substitution parameter inlining:  currently, inlining of parameters is based upon database dialect.  We probably should base this on directory configuration, using reasonable defaults.
  26. Smarter implementation for record counting loops and record deleting loops.  Note:  need to consider pessimistic locking implications here.
  27. Avoid emitting unnecessary, top-level instance variables for function parameters, which are later hidden by block-level, local variables of the same name, and avoid initializing the top-level variables in the init() method of the inner subclass of Block which is emitted inside the method which represents the converted function, so long as those top-level instance variables are never needed.  There is some scenario where these are needed (not sure what that is at the time of writing), but in other cases, they are declared and initialized, but never subsequently read.  The testcase uast/query-test-for-scrolling.p, for instance, has a function updateRecords(input valOld as int, input valNew as int).  The valOld parameter generates both an instance variable in the top-level class QueryTestForScrolling, as well as an instance variable in the anonymous inner subclass of Block within the updateRecords() method.  In the init() method of that inner class, QueryTestForScrolling.this.valOld is initialized to the value of the anonymous inner class' valOld variable, but then the top-level valOld is never accessed.  Both the top-level class' valOld instance variable, and the initialization of it in the inner class' init() method (and in fact the entire init() method, since this is all that is done there) could be eliminated as a code cleanup optimization.
  28. Macro-pattern matching.
    • large amounts of code have been "cut-n-pasted" all over the place
      • e.g. select output destination in report generation is ~200 lines of this
      • some of these pieces of cut-n-paste code may have been slightly modified (sometimes in incompatible ways and sometimes in a manner that may be handled by method parameters in the target system)
    • other places in the code may be highly similar or even the same, simply by the fact that the same logic is written separately in multiple places
    • the optimal result is to create one or more methods than can be reused from many locations
    • with the current expression engine/pattern engine support, one can easily write a small ruleset that matches on very complex patterns BUT to do this one must know what the pattern looks like first
    • how might we search throughout the project for such candidates without knowing the pattern in advance?
      • make the full list of live ASTs
      • start at the root of the first AST and generate a ruleset that matches this entire tree
      • search all ASTs for a match, save results, reduce the pattern by walking down the tree and iteratively generating a ruleset that matches this entire sub-tree and searching all ASTs...
      • issues to solve:
        • how to build the expressions and how to submit them to the pattern engine?
        • do we make a first pass to build all possible expressions (a potentially huge list of expressions!) which then are tested in a second pass? OR do we have some more automated driver that iteratively processes things (this would require a pattern engine that can be run against different profiles in the same JVM)
        • do we take some kind of "template" approach where we pass an AST (rooting the subtree to match) into the pattern engine instead of a ruleset
        • real matches are likely even though an exact match with the tree is not there, to handle this some form of fuzzy matching (setting a heuristic that a 90% match is still valid) OR allowing the pruning of likely variable stuff out of the tree and still allowing a match in the target when additional nodes exist that don't exist in the original
        • how do we rank matches?
          • probably based on the % of nodes that match
          • some measure of the deviation caused by nodes that exist in the target or source but not both
          • the number of matches across the list of ASTs
        • how do we know when to stop creating a match pattern for a given input
          • matching a single node is likely to be a waste of time
          • even matching small subtrees of 2-3 nodes is likely to be a waste of time
          • perhaps a more general rule can be made that patterns only found WITHIN language statements are too small to be matched upon (e.g. matching on all DEFINE VARIABLE subtrees is useless as we already know that these subtrees appear everywhere and by themselves they don't constitute a useful match)
      • SO: we are really trying to match common patterns made by groups of:
        • top-level parser/Progress language constructs like language statements, assignments, loops...
        • expressions that are of high complexity (how to determine this?) might also be good candidates for replacement as a method
  29. "if ___ then x = y + z. else x = y + z + a." can be rewritten as "x = y + z.  if ___ then x += a."
  30. Emit primitives instead of wrappers based on hints or heuristics.
    • Heuristic based optimizations will be written to unwrap variables once into temporary primitives and reuse these until reassignment or modification of the original wrapper instance.  For example, if there is > 1 read of the same variable without that variable being modified, the unwrapping would occur once and the same temporary primitive would be used multiple times.  This makes the code more readable and improves performance.  Issues to resolve include:
      • how to name the temp variables?
      • how to detect when it is safe to reuse an already unwrapped variable and when something could be modified?
  31. How might we define beans for highly coupled/related shared variables such that all of these related members are stored and retrieved in the shared variable manager in one call?
  32. Improve refactoring by enhancing the call graph with the (currently lost) knowledge of loops/recursion and duplicate calls.  This knowledge may allow the aggregation of multiple procedures into a common class.
  33. Implement call graph mode for the pattern engine.  This will allow much more advanced analysis of resource usage (across scope boundaries), undoability, dead code analysis etc...
  34. Remove debug code in the Progress source.  (How do we identify such things?)
  35. Fix bug in date.neutralMillisToCalendar().  It does take the daylight savings time offset into account in that method.  The problem is that the DST offset is queried from a GregorianCalendar object that is set to the current (today's) date/time.  This means that if the current TZ is in DST right now, the DST offset will be positive for all dates!  That is why any date in the DST range is fine but those outside are off.  I need to think on a solution for this as it is not as easy as setting the target date/time into the calendar first, since anything very close to the DST transition point may still calculate incorrectly.  The following testcase shows the issue:
    •          date firstP2JDate = new date(2,1,2006);    //EST
    •          date secondP2JDate = new date(7,4,2006);    //EDT
    •          Date firstJavaDate = firstP2JDate.dateValue(TimeZone.getDefault());
    •          Date secondJavaDate = secondP2JDate.dateValue(TimeZone.getDefault());
    •          DateFormat format = new SimpleDateFormat("MM/dd/yyyy 'at' HH:mm:ss z");
    •          System.out.println("P2J First: " + firstP2JDate.toString());
    •          System.out.println("Java First: " + format.format(firstJavaDate));
    •          System.out.println("P2J Second: " + secondP2JDate.toString());
    •          System.out.println("Java Second: " + format.format(secondJavaDate));
  36. Unimplemented Progress features:
    • database
      • dynamic database support
        • CREATE BUFFER
        • CREATE DATABASE
        • CREATE QUERY
        • CREATE TEMP-TABLE
        • methods/attributes for using query handles and other dynamic database resources
        • All the backing support (compatibility logic) for these features is implemented, but some of the conversion processing needed to convert queries needs to be moved into the runtime.
      • database triggers
        • DISABLE TRIGGERS
        • TRIGGER PROCEDURE
      • sequences
        • CURRENT-VALUE()
        • CURRENT-VALUE statement
        • NEXT-VALUE()
      • temp-table handle as parameters
      • CONTAINS operator
      • misc built-in functions
        • CURRENT-CHANGED
        • DBPARAM
        • DBTASKID
        • DBVERSION
        • RECORD-LENGTH
        • TO-ROWID
      • internationalization is partial
        • DBCODEPAGE()
        • DBCOLLATION()
      • stored procedures
        • RUN STORED-PROCEDURE
        • CLOSE STORED-PROCEDURE
      • two-phase commit
      • schema meta-data access (e.g. _file)
    • base language
      • Windows-specific features
        • registry and environment support
          • GET-KEY-VALUE
          • LOAD
          • PUT-KEY-VALUE
          • UNLOAD
          • USE
        • system dialogs
          • SYSTEM-DIALOG COLOR
          • SYSTEM-DIALOG FONT
          • SYSTEM-DIALOG GET-FILE
          • SYSTEM-DIALOG PRINTER-SETUP
          • SYSTEM-DIALOG HELP
        • "well known" named stream I/O destinations
          • CLIPBOARD
          • PRINTER
        • Active-X / COM support
          • COM-HANDLE data type
          • CREATE <com_object>
          • RELEASE OBJECT
        • DDE support
          • DDE ADVISE
          • DDE EXECUTE
          • DDE GET
          • DDE INITIATE
          • DDE REQUEST
          • DDE SEND
          • DDE TERMINATE
      • Finish BigDecimal support for decimal literals and for all decimal math functions.
      • user-defined functions with IN <procedure_handle> or IN SUPER.
      • shared library support
        • CALL
        • RELEASE EXTERNAL
      • appserver
        • CREATE SERVER
        • RUN ON SERVER (invoking remote procedures on an appserver)
        • TRANSACTION-MODE AUTOMATIC
      • persistent procedures, super procedures
        • DELETE PROCEDURE
        • RUN PERSISTENT (no persistent procedure support is provided)
        • RUN SUPER
      • named events support is handled in the conversion and is stubbed out in the runtime, but the runtime needs to be finished (see NamedEventManager.java)
        • PUBLISH
        • SUBSCRIBE
        • UNSUBSCRIBE
      • internationalization and sorting
        • CURRENT-LANGUAGE
        • CODEPAGE-CONVERT()
        • IS-LEAD-BYTE()
        • GET-CODEPAGES()
        • GET-COLLATIONS()
      • 22% of built-in functions:
        • DYNAMIC-FUNCTION()
        • SETUSERID()
        • SUPER()
        • PROC-STATUS()
        • PROC-HANDLE()
      • persistent triggers
        • ON PERSISTENT (all other forms of triggers are already supported)
      • misc
        • PROMSGS
        • PROPATH assignment is supported in the runtime but not in conversion
        • SETUSERID
      • raw/memptr support
        • GET-BITS()
        • GET-BYTE-ORDER()
        • GET-BYTES()
        • GET-DOUBLE()
        • GET-FLOAT()
        • GET-LONG()
        • GET-POINTER-VALUE()
        • GET-SHORT()
        • GET-UNSIGNED-SHORT()
        • PUT-BUTS
        • PUT-BYTES
        • PUT-DOUBLE
        • PUT-FLOAT
        • PUT-LONG
        • PUT-SHORT
        • PUT-UNSIGNED-SHORT
        • RAW()
        • RAW
        • SET-BYTE-ORDER
        • SET-POINTER-VALUE
      • XML
        • CREATE X-DOCUMENT
        • CREATE X-NODEREF
      • sockets
        • CREATE SERVER-SOCKET
        • CREATE SOCKET
      • VIEW with a list of widgets, what seems to be an undocumented language feature
    • user interface
      • finish editor widget support, phase 3: all methods/attributes, all options, all word wrapping and editing features
      • frame-index is only implemented with CHOOSE, but in reality it should be maintained by any data input statement
      • remaining CHUI support
        • menus/sub-menus
      • remaining GUI support
        • base frame implementation
        • drawing/layout for GUI widgets
        • new widgets
          • image
          • rectangles
      • dynamic widget/frame support
        • CREATE WIDGET/OBJECT
        • CREATE BROWSE
        • CREATE WIDGET-POOL
        • DELETE WIDGET/OBJECT
        • DELETE WIDGET-POOL
      • remaining methods/attributes (75%)
      • miscellaneous built-in functions
        • CAN-QUERY
        • CAN-SET
        • FRAME-DB
        • FRAME-FILE
        • FRAME-NAME
        • IS-ATTR-SPACE
        • LIST-EVENTS
        • LIST-QUERY-ATTRS
        • LIST-SET-ATTRS
        • LIST-WIDGETS
        • LOAD-PICTURE
        • RGB-VALUE
        • VALID-EVENT
      • language statements
        • DEFINE IMAGE
        • DEFINE MENU
        • DEFINE RECTANGLE
        • DEFINE SUB-MENU
        • INSERT
      • Buttons don't honor the default attribute in CHUI. A number of experiments show that this attribute is ignored in the Progress character mode. Documentation is not clear regard to this attribute, but it mentions that presence of default button in the frame disables handling of the RETURN key by fill-in widgets and this does not happen in character mode.  In GUI mode this will need to be added.
      • accumulation "variables" (like a "naked" count variable) can be referenced directly in Progress without the "accum 1 count" construct.  This is a parse-time problem but it also has consequences that are harder to fix since such vars can be assigned!
      • validation expressions, screen-value and the INPUT builtin function all allow variables/fields to be treated bimodally (as their normal type and as a character type, depending on context), better detection logic is needed during conversion, including having a database of signatures for builtin functions, methods...
      • aggregation phrases in a display statement are supported, but the implementation is limited to service the simpler use cases we have encountered to date;  specifically, the following cases will break the current implementation:
        • currently, there is a simplifying assumption that all active aggregate phrases for a frame are grouped together in the same display statement;  thus, the "same" aggregate phrase appearing in multiple display statements will not be processed correctly, whether:
          • in the same loop, or
          • across different loops
        • the "when" option will disrupt accumulation by producing a null frame element (Progress accumulates the row data, regardless of whether "when" masks display of the data);  because we drive accumulation from within the frame element, this implementation breaks down in this situation
          • a related problem is that the "when" option is not honored currently, if it appears after an aggregate phrase in a display statement
    • internationalization
      • code page conversions
      • collation
      • "translation manager" features
      • string options processing
      • usage of Java resource bundles where possible
  37. Deviations from Progress behavior:
    • parser
      • An empty frame phrase is legal as in "repeat with j = 1 to 5:".  This is the equivalent to "repeat j = 1 to 5:".  The problem is that the down_clause rule isn't conditional, it matches as soon as there is an expression in LA1.
      • The file testaces/uast/two_frames_same_name.p defines two frames: one in the procedure file and one in a local procedure. Although both frames have the same name, P2J creates 2 fields with the same name in the generated class. Also, there is only one frame definition for both generated fields.
    • user interface
      • Applying ENTRY event to other widget in trigger attached to EDIT widget may not have visual feedback in 4GL - cursor remains inside EDIT widget (see event_behavior_edit.p testcase) while in P2J cursor is moved to activated widget. This is definitely a bug in 4GL while P2J handles this case correctly and this results to visible difference in appearance. Note that input focus is correctly moved so next key typed by the user is forwarded to correct widget.
      • PUT SCREEN can "bleed through" a frame in Progress because certain fields (e.g. fill-ins) only seem to output to some character positions and blank positions don't get output as spaces.  This means that a prior put screen can bleed through in Progress.  In the P2J implementation, this does not occur.
      • In Progress DOWN frames behave differently when CHOOSE is invoked for a variable that is in a DOWN frame when CHOOSE is in ROW mode. In this case no intermediate pauses are performed and all values are scrolled up so only the last value is visible in the frame (it occupies the top data row in the frame).
      • FRAME-INDEX may return incorrect values (in comparison with Progress) if CHOOSE is invoked for regular (extent = 1) variables or for a combination of array and regular variables.   This happens because information about array indexes associated with a particular variable instance is lost during conversion.  Since this function is intended for use with arrays, real applications should not have any problems with that.
      • FillIn widget
        • when editing a decimal number, after entering the last (rightmost) digit to the right of the decimal point) using the backspace key (or left arrow) will jump 2 characters to the left (in Progress) instead of 1 (in P2J and as most users would expect)
      • An explicit PAUSE statement will ignore the HELP key (by default F2 or any key that has been remapped to the HELP function).  It does not trigger help nor does it clear the pause.  Some tests show that implicit pauses (like those caused by a block exiting/iterating with a frame scoped in that block) may not provide this same "feature".
      • A string override on a date base field will display with separators in Progress but not in P2J.  "hello" @ mydate will display as "he/lo/  ".  This doesn't seem very useful so it has not been duplicated.
      • A string override in a logical base field will change the value of the field in a way that can be edited.  "ye" @ mybool format "yes/no" will display AND edit as "yep".  This is unusual because normally edited data is not changed when an override occurs which has mismatched types.
      • LEAVE and ENTRY events show inconsistent behavior if there are triggers on either of them that return NO-APPLY. The reaction may even depend on the source of the event. Cursor movement keys, tabs and back tabs and APPLY ENTRY all may produce varying results. The most often, the cursor left and right keys behave in the most logical way: the focused widget remains in focus and does not lose its highlight. If this is the case, repeatable application of those keys produces predictable and repeatable results. The other times, the focused widget loses its highlight *and* the state of the widgets changes invisibly, so that the repeatable application of the keys produces ever changing results! In P2J this behavior is not reproduced and the focused widget always retains focus.
      • ON ... REVERT is implemented with deviations. In Progress,
        • triggers having widget list (even though they may have ANYWHERE option as well) can't be reverted entirely down to the no trigger at all; the first defined trigger is "sticky" and REVERT operation has no effect on it. See the testcases/ui/revert1.p testcase.
        • global ANYWHERE triggers seem to have a single item representing them instead of a normal stacking. No matter how many ON registrations were made, a single ON ... REVERT deregisters the trigger entirely.See the testcases/ui/revert2.p testcase.
        • the findings above have to be further tested to determine the applicable scoping (within the current block or globally).
      • READKEY in editing block may be responsible for firing triggers on the fly for some high level events. CHOOSE event for buttons is known to be such an event. The list of events in this category and details of how it works need to be investigated. See the testcases/uast/readkey_trigger*.p testcases.
    • Some error messages are slightly different:
      • Database lock conflicts don't show the TTY of the owning user.
      • Database field and table names in error messages are the new ones.
      • Stream access failures don't show the source code name of the stream.
      • Some runtime generated stream exceptions (e.g. "Attempt to read from closed stream ____. (1386)" may appear as unspecified IOExceptions today where this translates into error text that may say something like "IOException during read. (-1)".
    • Condition processing:
      • CTRL-C while the server is in a tight DO loop doesn't honor the stop until after the loop exits.  If the loop is very long (or infinite), the STOP will never be honored.
    • The program-name function may show different output than in Progress due to different decompositions of function in both the generated code and differing code paths in the runtime.
    • regular stream I/O
      • Although documented in Progress references, the default code-page conversion (stream to internal and vice versa) does not appear to actually be implemented OR perhaps this is locale specific.  No default codepage conversion is implemented in P2J.
      • All of the explicit codepage conversion processing that is possible in Progress is missing.
      • The effect of the BINARY option is only accounted for in READKEY processing and in this case it only modifies how newlines are returned.  If BINARY modifies other processing, it has not been identified and is thus unimplemented at this time.
      • Progress references document that the null character terminates strings.  This is not implemented.
      • DBCS support is not provided.
      • UNBUFFERED mode is not supported (and is probably not needed).
      • MAP/NO-MAP support does not exist.
    • redirected I/O
      • The special text mode echo quirk for logicals, integers and decimals is not supported.  Likewise the screen buffer copy on error is only supported for character types, not all types.
      • Mixed usage of the PUT language statement and UI statement output redirection support (e.g. DISPLAY) may not operate properly.
      • The overwrite behavior of certain frames (based on implicit DOWN or explicit DOWN/UP) is not supported as there does not seem to be a useful reason to actually code an application to do this.
      • Stream width is hard coded to a maximum of 512 columns and 128 lines.  A more dynamic/flexible approach should be implemented based on the (column and line) values in the current frame however this has not been handled at this time.
    • Using an array index that is out of bounds during a NO-ERROR (silent error mode) language statement/assignment will fail with an IndexOutOfBoundsException since we currently directly map array access into the standard Java subscript operator [].  While we do have a central method in NumberType.subscript() to handle the bounds checking and error generation, in silent error mode this will return an out of bounds value.
    • IF/THEN/ELSE that are detected as having an unreachable branch (the THEN or ELSE) will have that branch removed.  This is logically correct, however in the case where the IF expression has side effects (e.g. a user defined function that changes some program state), then this code would not be executed in the rewritten generated code.
    • Any NO-ERROR expression that causes an error to be raised, does not immediately abort in P2J as it would in Progress.  Assignment processing is protected such that no assignment is made in this case, BUT any side effects of the expression that might be sequenced after the error will occur in P2J when they would not in Progress.  For example, use of user defined functions can have side-effects.  Since in P2J, a user defined function in an expression would still execute after the error occurred, an unexpected behavior could be caused.
    • NUM_LITERAL of 999999999999 (12 digits) can be encoded in a progress source file but an error does not occur until runtime. To avoid a javac problem (compile time), we detect this case during conversion and throw an exception.
    • DEC_LITERAL mantissas (the significand or the digits to the left of the decimal point) can be up to 50 digits in Progress, the current implementation of P2J only supports up to 16.  Detect and use a BigInteger implemention instead.  This may have to be done as a configuration value for the entire server/client (use use of the decimal class).  It also has implications for code emit since the Java double can't be used to represent the DEC_LITERAL.
    • Handle data type's default format string is larger because the 32-bit integer hashcode that is used to back the handle's representation is too large to fit in the normal Progress default.
    • No support for array subscripts with ranges that use a non-constant expression as the start index as in myvar[ 2 + j FOR 3 ].  Such cases can not be used in an expression, instead they can only be used as implicit reference to a range of index positions.
    • No conversion of Progress strings to Java strings is done at runtime.  All strings passed are expected to be Java strings.  This behavior may need changing in the case where a Progress-compatible string (one that has Progress-specific escape sequences) is built at runtime but needs to be used as a Java string.
    • Based on testing a wide range of Progress features (escape sequences in string literals, reading from files/processes, built-in functions...), embedded null characters in a character var can only be generated using the get-string() builtin function on either a raw or memptr var.  Even in this case, the var must contain null bytes AND the get-string() call must specify an explicit length that causes the copied data to include a range that contains the null byte.  This processing is fully implemented in P2J.  Once you have a character var with a null character embedded, the support for that data varies depending upon the part of Progress that is in use.  All character related operators (especially the comparison ones), built-in functions and language statements need to be reviewed for their compatibility with such embedded null byte cases.  The following are already handled: + (concatination operator), asc(), chr(), string(), substring() and = (assignment operator).  Note that chr() cannot return a null byte, and both substring() and the assignment operator are completely insensitive to null bytes.
    • frame-index, lastkey and other variable-like builtin functions can silently ignore assignment usage in business logic (the errors occur and no change occurs to the value) --> this is a form of dead code
    • The following features aren't useful in converted Java code (and are not currently converted at all):
      • DICTIONARY
      • COMPILE
      • KEYWORD
      • KEYWORD-ALL
      • LIBRARY
      • MEMBER
      • PROMSGS (function and statement)
      • SAVE-CACHE
      • SHOW-STATS
  38. Complete support for database methods and attributes.  The first release supports only a handful of database-related methods and attributes.
  39. Support "preselect fetch" FIND statement where clauses.  These statements can have their own where clauses which further refine a find among the preselected results.  Currently, these are not handled.
  40. Support a multi-table AdaptiveQuery.  Currently, multi-table joins which cannot be converted to PreselectQuery convert instead to CompoundQuery.  The latter is inefficient because the join is performed at the database client and it therefore requires an order of magnitude more database-level queries for each join than would a multi-table AdaptiveQuery implementation.  A multi-table AdaptiveQuery would perform the join on the server when in preselect mode, but would fall back to using CompoundQuery when converting to dynamic mode.
  41. Future Development
    • AST ID stability.
    • Better regression testing harness.
    • Naming
      • Add hints support for naming overrides.
      • Cross-namespace (vars, streams....) conflicts.
      • Minimize the use of disambiguating text in the names (like adding Frame to every frame name).
    • Remove fileMatch in all annotations rules (we always persist now so this is just gorp).
    • Add an option to allow get/setProperty() in FileSystemOps to be backed by the directory instead of by Java system properties.
    • Update the lexer and parser for v10.1C changes (it is up to date with 10.1B).

Copyright (c) 2005-2010, Golden Code Development Corporation.
ALL RIGHTS RESERVED. Use is subject to license terms.


Skip navigation links
Copyright (c) 2004-2017, Golden Code Development Corporation.
ALL RIGHTS RESERVED. Use is subject to license terms.