FWD » Core Development » Base Language

types: DATETIME, LONGCHAR, INT64 (needed for DB support of _lock-recid and for sequences); DATETIME, NOW, ADD-INTERVAL, MTIME

Some other thoughts on LONGCHAR:

0. Look through the com/goldencode/p2j/util/character class carefully. Also look at BaseDataType.
1. Search through the Progress 4GL Language Reference to find all the unique ways that LONGCHAR is used. Look carefully at those features.
2. Write testcases to explore these features.
3. Expand the testcases to explore how LONGCHAR is interoperable with CHARACTER. In what ways can they be used interchangeably? For example, most code (like built-in functions) that can take an INTEGER can also take a DECIMAL because there are implicit conversions between the two types. Likewise, arithmetic and comparison operators work on both numeric types. This kind of "hidden" behavior is important to consider in our design.
4. Are there any ways to directly access specific characters by index position that are NOT supported for CHARACTER?

Document your findings here.

• CHARACTER data that is not limited to 32K in size. LONGCHAR variables can be any size (limited by system resources).
• metadata:
  - code page information (defaults to -cpinternal)
  - other information ( the codepage is fixed or not)

• FIX-CODEPAGE statement
• CODEPAGE–CONVERT function
• COPY-LOB statement

• the LONGCHAR is converted to -cpinternal
• most functions which alter the string only apply to LONGCHAR values that are in -cpinternal
• all functions which deal with LONGCHAR values and have offset and length input parameters use character offsets and
  lengths, not byte values.

• default initial value "" (like character)
• the documentation states that: you cannot use the INITIAL option to specify an initial value for this data type as part of the definition of a variable, procedure parameter, or class-based property. But in fact specifying initial value will assign a value to the LONGCHAR variable.
• defining a LONGCHAR variable supports the same options as a CHARACTER variable, except for the FORMAT option and all VIEW-AS options except VIEW-AS EDITOR LARGE.

• Export statement: for LONGCHAR only if it's only field in the list. This will save the data in (UTF8)
• "Remote functions". If you invoke a function on an AppServer, the function cannot return a value as a LONGCHAR.
• The expression COLUMN is not valid for a LONGCHAR variable.(to be tested)

My only question so far is related to the codepage info. Is there a difference between how character and longchar handle codepages? The points about codepages don't make this clear. For example, as far as I understand, the character data is stored and processed in the 4GL using cpinternal. Only input and output of character values is done in a different codepage. Of course, my understanding may be defective. :)

• CODEPAGE–CONVERT works the same way for CHAR and LONGCHAR.
• FIX-CODEPAGE and COPY-LOB cannot take CHAR parameters.

• for a CHAR parameter there are CHARACTER, FIXED, COLUMN, and RAW as length/position types
• for LONGCHAR only CHARACTER is a valid length/position type.

Operators:

• LT,GT,LE,GE,EQ,NE between two LONGCHARS or a LONGCHAR and a CHARACTER -> values are converted to -cpinternal
• MATCHES: only the first parameter could be LONGCHAR. The LONGCHAR variable value converted to -cpinternal prior to
  comparison.
• BEGINS: only the first parameter could be LONGCHAR. The LONGCHAR variable value converted to -cpinternal prior to
  comparison. BEGINS always uses the -cpcoll collation.

• if a or b is LONGCHAR the result is LONGCHAR.
• the result has the codpage of the expression on the left

• there are default assignment character conversions (CHAR <-> LONGCHAR <-> CLOB ...)
• expr is converted to -cpinternal or fixed code page(if longchar_var code page is fixed)
• char_var = longchar_var . Works if LONGCHAR could be fit to a CHAR (32k?). For a LONGCHAR with a larger size there will be a runtime error.
• a LONGCHAR cannot be used in a place where a CHAR is required. There will be a compile time error. Also for functions taking only LONGCHAR parameters passing a CHAR will result in a compile time error.

Note:
The undocumented function _cbit() works only for CHAR data type and not for LONGCHARS.

From Adrian:

I believe that this longchar type could be implemented on the existing character class. I didn't find any case(a function, operator or just a phrase in the documentation) that will suggest that longchar is a different type from char apart from the size and some emphasis on codepage(protecting this with fix-codepage).

We need to consider this carefully. Your descriptions of the features do note that in the 4GL, there are cases where there are runtime limits (e.g. errors that can occur) and runtime behaviors that are specific to LONGCHAR. We may need to duplicate these features exactly. For example, the SUBSTRING function has very specific behavior with LONGCHAR and different behavior with CHARACTER.

It also seems like there is very specific codepage behavior when using operators on LONGCHAR. I don't think CHARACTER has any such behavior like that.

Some of the other limits could possibly be implemented at conversion time, but I think we might want to implement with a real LONGCHAR class to get it exactly right.

I reviewed the 4gl operator functions and procedures implemented by the character class and also the ones taking LONCHARs as parameters or having a LONGCHAR result type.

Functions that will take only CHARACTER parameters. This will remain implemented only in character class.

_cbit
asc

Functions/operators having a main parameter of type LONGCHAR or CHARACTER and the result of the function determined by the type of this parameter(c/lc means CHARACTER or LONGCHAR):

c = c + c                lc = lc + c    lc = c + lc    lc = lc + lc 
c = substitute(c,c/lc)   lc = substitute(lc,c/lc)
c = substring(c,..)      lc = substring(lc,..)
c = caps(c)              lc = caps(lc)              
c = replace(c,c/lc,c/lc) lc = replace(lc,c/lc,c/lc)
c = right-trim(c)        lc = right-trim(lc)
c = left-trim(c)         lc = left-trim(lc)
c = trim(c)              lc = trim(lc)
c = entry(,c)            lc = entry(,lc)

Functions that could take a combination of LONGCHAR and CHARACTER variables:

compare(c/lc,c/lc)
index(c/lc,c/lc)
r-index(c/lc,c/lc)
overlay(c/lc, , ) = c/lc
num-entries(c/lc)
lookup(c,c/lc)

Functions with parameters of the same type( no mixing):

minimum(c,c) minimum(lc,lc)
maximum(c,c) maximum(lc,lc)

Restrictions on the type specification:

length(c,["CHARACTER" | "RAW" | "COLUMN"])
length(lc,"CHARACTER")

substring(c,pos,len,["CHARACTER" | "FIXED" | "RAW" | "COLUMN"])
substring(lc,pos,len,"CHARACTER")

Functions related to LONGCHAR:

FIX-CODEPAGE(lc) = codepage
IS-CODEPAGE-FIXED(lc)
GET-CODEPAGE(lc)
CODEPAGE-CONVERT(c/lc,target_codepage,source_codepage) - also for CHARACTER type but not implemented
BASE64-DECODE(lc)
lc = BASE64-ENCODE()
COPY-LOB    
SHA1-DIGEST(c/lc,c/lc)
ENCRYPT(c/lc,,,)
MD5-DIGEST(c/lc,c/lc)
GENERATE-PBE-KEY(c/lc)
NORMALIZE(c/lc)

A lot of the functions/procedures listed above could end up being replicated in both character and longchar classes(probably with all parameter type combinations).

In order to go further with implementing the LONGCHAR data type I created the longchar class as a copy of the character class and (successfully)tested the base64-encode and base64-decode functions.

• codepage
• fixedcodepage

• CODEPAGE-CONVERT
• FIX-CODEPAGE
• GET-CODEPAGE

For example, as far as I understand, the character data is stored and processed in the 4GL using cpinternal. Only input and output of character values is done in a different codepage. Of course, my understanding may be defective. :)

• cpinternal (PUT-STRING)
• utf-8 (EXPORT)
• cpstream (COPY-LOB TO FILE)

Whatever the 4GL internal representation (and it is probably cpinternal ...) is for a LONGCHAR, the value will always be converted to one codepage or another. The same will be in P2J, just care should be taken to make the right conversion and produce the same output. The same for assignment of values to a LONGCHAR.

Please advise if I should go on with this class and implement all the methods needed.

1. We don't want to duplicate code. BUT
2. We need to limit the operators, built-in functions and so forth as needed to exactly match longchar.

We definitely don't want to substantially duplicate the character class in longchar. Before going further with the current approach, let's consider/discuss the following. Can we meet all objectives using one of these options:

A. Create longchar as a subclass of character (and overriding methods where needed). It seems we might encounter issues with implementing proper runtime limits on usage of longchar in this case. But it would be easy to maximize the common code.

B. Create a common base class for both character and longchar. Move common code there and then specialize as needed.

In both cases, we must consider that most operators and built-in functions are implemented using static methods. Both options above may encounter some difficulties in that regard.

Please consider these and offer your thoughts (and other options too should you have any).

• rules needed for adding a new type
• rules needed to set the result type to CHAR/LONGCHAR on functions that will take both type of parameters and the result is based on the type of the parameter

• make the change ( like adding a new function)
• fix the errors ( check what rules need to be changed / added)
• verify if all functionality is in place ( and take into consideration if there will be undesired functionality)

I considered that making longchar class a subclass of character or extracting common functionality in a base class could be done as a refactoring at the end (wishful thinking) or, more probably, on the first big issue encountered that will suggest(or impose) a solution or another. If I'll start with a subclass of the character class the same reasoning could be applied.

Apart from a subclass or a common base class as final implementation solutions I do not see what else could be done.

Go forward with this approach:

1. Make longchar a subclass of character. Override only as minimally necessary. If is OK to make the value, unknown and/or caseSens members of character protected if longchar needs direct access.

2. To the extent that the static methods in character need to be aware of both classes, then it is best to move those methods to a new class TextOps. I don't want to make the methods in character have references to the child class longchar. That is poor design. Make a list of the stuff that MUST move out. We may want to just move all function and operator support out to be consistent.

Is there anything about longchar that cannot be cleanly supported using the above approach?

• numEntriesOf (NUM-ENTRIES 4GL function)
• toUpperCase (CAPS 4GL function)
  along with utility methods fixupRegexChar and safeDelimiter

The problem:
Progress built-in function num-entries(c/lc,c). The first parameter could be CHARACTER or LONGCHAR but the second only CHARACTER.

The current corresponding method signature is:
public static integer numEntries(character list, character delimit)

This will allow using both character and longchar as parameters in P2J but in 4GL there will be an error on using a longchar for the second parameter.
It is like the hierarchy is upside down. If the character class will be derived from longchar then:
public static integer numEntries(longchar list, character delimit)
will limit the second parameter to the character class.

The rule in 4GL seems to be that you can use a CHARACTER as a function parameter defined to take a LONGCHAR.

a LONGCHAR cannot be used in a place where a CHAR is required. There will be a compile time error. Also for functions taking only LONGCHAR parameters passing a CHARACTER will result in a compile time error.

An update of this statement:

• _CBIT
• Asc
• MINIMUM and MAXIMUM functions where both parameters should be of the same type
• User defined function parameters( and statements ).

• FIX-CODEPAGE
• IS-CODEPAGE-FIXED
• GET-CODEPAGE
• INPUT-OUTPUT parameters (for INPUT parameters works)
  The first three built-in functions are LONGCHAR specifics.

I'm thinking of inversing the hierarchy and making longchar the parent class.

This is a list of static fields/methods that should be moved from the character class to the new TextOps class.

Static fields to be moved to TextOps (all this are used by the compare method and this was moved to TextOps)
CASE_INSENSITIVE
CASE_SENSITIVE
CHARACTER
COLUMN
EQ
FIXED
GE
GT
LE
LT
NE
RAW

Static methods to be moved to TextOps:
_begins
begins
_matches
matches
byteLength
byteLengthOf
compare
entry
fill
indexOf
lastIndexOf
trim
leftTrim
rightTrim
length
lengthOf
lookup
matches
matchesList
numEntries
replaceAll
substitute
substring
toLowerCase
toUpperCase

fixupRegexChar
genNegativeIndexError
genOutsideRangeError
safeDelimiter

Static methods NOT to be moved to TextOps:
testBitAt
asc
chr
maximum
minimum
concat

convertToRegEx
convertToSQLLike
convertToSQLLike
convertToSQLBegins
convertToSQLBegins
fromExportString

progressToJavaString
progressSpacifyNull
javaSpacifyNull
javaTruncateNull
preprocessFormatString
postprocessStringLiteral
readchar

valueOf

This refactoring will cause changes in:
com.goldencode.p2j.convert.ExpressionConversionWorker ( toUpperCase )
com.goldencode.p2j.persist.DMOValidator ( toUpperCase )
com.goldencode.p2j.persist.FieldReference ( toUpperCase )
com.goldencode.p2j.persist.Persistence ( rightTrim )
com.goldencode.p2j.persist.pl.Functions ( begins, entry, indexOf,... )
com.goldencode.p2j.preproc.Preprocessor ( lengthOf )
com.goldencode.p2j.uast.ExpressionEvaluator ( begins, matches)
com.goldencode.p2j.ui.client.FillIn ( lengthOf )

Note:
value, unknown and caseSens fields remained private as I modified direct field access with getValue(), isUnknown(), isCaseSensitive()

After looking over Adrian's findings, to me it looks like the best solution is to have two classes - character and longchar - both inheriting the same base class. This way, we can enforce (at compile time) the parameter types for all functions which expect a parameter to be only char or longchar (like minimum/maximum or num-entries case).

def var ch as char.
def var l as longchar.
do i = 1 to 17000:
   ch = ch + "1".
end.
l = "".

l = ch + ch. /* case 1 */
l = (l + ch) + ch. /* case 2 */
l = l + (ch + ch). /* case 3 */

1. as the resulted length is greater than the maximum length for a character value (31984), it will result in a runtime error, as when concatening ch with itself, the result is still a character variable, which can not hold the combined string (of length 34000).
2. the paranthesis are to emphasize that when concatenating a longchar with a char, it results a longchar. Here, the concatenation finishes, with l holding a 34000-char string.
3. produces the same runtime error as case 1 - the concatenation in the paranthesis is done first, which produces a character which can not hold the resulted string of length 34000.

Finally, please confirm that all the longchar functions mentioned until now will need to be supported.

to me it looks like the best solution is to have two classes - character and longchar - both inheriting the same base class

Yes, I completely agree.

 please confirm that all the longchar functions mentioned until now will need to be supported.

Yes, but only for conversion purposes. You can and should defer runtime implementation where possible.

Normally, we always define parameters as the exact match to the type originally specified. I prefer to stay with that approach as it is more clear to the reader of the code, what the intent was.

I think using a base class for the hierarchy makes sense. This parameter case is messy, but it also should be rare, since it has little possible useful purpose. So if there is a bit of extra messiness in this case, I can accept that.

How about using a technique similar to the decimal.PrecisionResetter inner class? This is a case where in downstream procedure, the precision of a shared decimal variable can be temporarily forced to a new value and then reset upon leaving the nested scope. In this longchar/char param case, it seems like for OUTPUT/INPUT-OUTPUT parms we could save a reference to the original var in an inner class instance and then reset the original parm values as needed based on the scope notifications. A special wrapping constructor or static factory method would be used to prepare the parm before the call. It can setup the new instance with the proper type and the inner class instance prepped. The registration would have to be done inside the function, although we can make a facility similar to TransactionManager.registerNextExternal() to hide this away.

OK, I think I understand your approach. As we will use a base class (maybe named "text"?), all the built-in functions which can accept both character and longchar for a parameter will have that parameter set to the base class; the character/longchar types will be set only when it is sure the parameter can be only of that type.

Actually, I was still wanting the parameter set to the specific type. The reason is that this parameter mismatch case should be rare. So instead of optimizing for that case, I want to just put in special handling only when needed.

Yes, I like the idea of the Text base class. But I don't want to force all parameters to that type.

I refer to the built-in functions which can accept both character and longchar for a parameter, like replace. As it was moved to TextOps, we can set its parameter to Text instead of character. If we don't do this, we will have to create a TextOps.replaceAll API for each element of the (String, character, longchar)x(String, character, longchar)x(String, character, longchar) cartesian product (because replace accepts 3 parameters and each can be set to either string constant, character or longchar).
If you worry about their return type, this will still need to be Text, because as far as I can tell, the type of the returned value is dependent on the type of some parameter. This applies to the concatenation operator too: the returned value can be either character or longchar.

Sorry, I missed the "builtin-functions" part. Yes, I agree that the built-ins should take the base class.

In regards to the return type, I think using generics <T extends Text> is a good solution, especially for the cases where the return type matches one of the inputs.

Feedback:

1. The Progress documentation states that the default initializer for longchar is unknown value. But notes above state otherwise. If the Progress documentation is right, then longchar needs to be added to function is_unknown_init_var_def in common-progress.rules and the default initializer in variable_definitions.rules would not be "". Please confirm this and if the Progress documentation is wrong, put comments in both of those rulesets to note that fact.

2. Copyright date updates are needed in expressions.rules.

3. Should byteLength() remain in character since longchar can't be used on a byte basis?

Otherwise, I think this is ready for testing.

The code changes look good. My only concern is that the issues 1 (default init) and 3 (byte length) should have the javadoc changed to explain that the Progress documentation is incorrect. Can you do that before you leave?

Eugenie is making a change on his update too, so we have a bit of time...

For byteLength, OK, I've changed the javadoc. But I don't understand what javadoc I should change related to default init - in converted code, a longchar variable is emitted and initialized as:

longchar l = new longchar("");

Just put it in the following:

1. In the TRPL rules where it emits "" as the default.
2. In the class level Javadoc for longchar.

That should be sufficient.

I have applied it. There are some compile warnings:

[javac] /mnt/san/sata/gc/20121029/p2j/src/com/goldencode/p2j/util/TextOps.java:3361: warning: unmappable character for encoding UTF8
    [javac]     * <li> NFD � Canonical Decomposition
    [javac]                ^
    [javac] /mnt/san/sata/gc/20121029/p2j/src/com/goldencode/p2j/util/TextOps.java:3362: warning: unmappable character for encoding UTF8
    [javac]     * <li> NFC � Canonical Decomposition, followed by Canonical Composition
    [javac]                ^
    [javac] /mnt/san/sata/gc/20121029/p2j/src/com/goldencode/p2j/util/TextOps.java:3363: warning: unmappable character for encoding UTF8
    [javac]     * <li> NFKD � Compatibility Decomposition
    [javac]                 ^
    [javac] /mnt/san/sata/gc/20121029/p2j/src/com/goldencode/p2j/util/TextOps.java:3364: warning: unmappable character for encoding UTF8
    [javac]     * <li> NFKC � Compatibility Decomposition, followed by Canonical Composition
    [javac]                 ^
    [javac] /mnt/san/sata/gc/20121029/p2j/src/com/goldencode/p2j/util/TextOps.java:3365: warning: unmappable character for encoding UTF8
    [javac]     * <li> NONE � Returns the source string unchanged

This is just in the javadoc, so I think it can be fixed before check-in but without further testing.

1. a bug in operator.rules which leaked the chp_wrapper to other nodes on line 452 (needed to add protection for prog.plus)
2. TextOps.concat can be set as parameter for lots of builtin functions/methods, so for now I think is best to let it return character, not Text. When we will add full longchar support, we will either have to find each and every case of builtin function/method which has a character parameter and change it to accept Text parameter instead (if it accepts longchar values too) or we will have to wrap each expression which returns Text (passed to a character parameter) with a new character c'tor.
3. fields defined as character can be set to expressions which are evaluated using Text APIs (which return Text) or longchar variables, i.e.:
   

   def var ch as char.
   def temp-table tt1 field f1 as char field f2 as int.
   def var lc as longchar.
   
   overlay(tt1.f1, 1) = ch.
   tt1.f1 = lc.
   

   
   Thus, we will either have to wrap the parameter depending on the fields type, using new longchar or new character c'tor, or (the easiest way) would be to add setters which accept a Text value, for each character field.

 I think is best to let it return character

That is fine for milestone 4.

After that:

we will either have to find each and every case of builtin function/method which has a character parameter and change it to accept Text parameter instead (if it accepts longchar values too)

Yes, this is the way.

And for longchar passed to user defined functions using character (and vice versa), we will just wrap in the proper type.

or (the easiest way) would be to add setters which accept a Text value, for each character field.

Yes and no. We would change the setter to take Text (just like the setter for both integer and decimal fields is always NumberType). We don't want to add new setters, the Text setter should be sufficient.

I've spoken with Eric on this and he agrees. If you need to make the change now, do it. Otherwise, we will make the related changes after milestone 4.

Passed regression testing, committed to bzr revision 10165.

Passed conversion regression testing, committed to bzr revision 10215.

I agree. The code is fine. Check it in and distribute it.

Committed to bzr revision 10259.

1. character/longchar interchangeability. Note that this is related to integer/int64 and date/datetime/datetime-tz, when OUTPUT parameters are used (all cases should use some common approach). See notes 23 and 25.
2. implement the character limitation related to length.
3. the hard part will be to determine how the codepage and -cpinternal can be implemented (I suspect we could use "file.encoding" property to some extent).
4. implement the codepage-related statements/functions:
   

   FIX-CODEPAGE(lc) = codepage
   IS-CODEPAGE-FIXED(lc)
   GET-CODEPAGE(lc)
   CODEPAGE-CONVERT(c/lc,target_codepage,source_codepage) - also for CHARACTER type but not implemented
   NORMALIZE(c/lc)
   

• COPY-LOB is a challenge on its own
  For the above, an initial estimate is 60 hours.

Should we make a separate task for the SecurityOps stuff? I refer to these statements/functions (estimate is 8 hours for them):

ENCRYPT
GENERATE-PBE-KEY
GENERATE-PBE-SALT
SHA1-DIGEST
MDG-DIGEST

We will deal with COPY-LOB in #2135, which is not needed for milestone 7.

We will handle the hashing and encryption functions in #2136, which is not needed for milestone 7.

Is this work complete? If not, what exactly is left?

As far as I observed, 54.1 should already be at the same level as int/in64 and date/datetime.
54.3 must be investigated against 4GL/OE.
Quick looking over the code, I see that code-page support is mainly missing (including conversion of some parts).

I will start working on this issue tomorrow.

Please just add a new task for the remaining work. Do not start work on it. We will schedule it for later.