FWD » Core Development » Base Language

As noted during #4384 work, Marian's team found that CHR and ASC were not double byte enabled, they use the wrong default codepage and validation is incomplete. This task is for tracking the cleanup work on those built-ins.

Please see #4761 for more details.

• if a character code is not defined/supported for 8859-x or 1252 a non-empty character is still returned. In the case of UTF-8, an empty string will be returned instead;
• the functions are multi-byte, meaning that they will successfully encode/decode characters like € in UTF-8 which occupies 3 bytes (14844588 / 0x00E282AC);
• a funny thing: running
  message asc("www", "UTF-8", "UTF-8").
  will print 7829367. In hexadecimal this number is 0x00777777. Now, looking at character map we can see that asc("w") = 119 = 0x77. Running
  message chr(7829367, "UTF-8", "UTF-8")
  will print back www! Of course, this is valid not only for w character, but seems limited to multi-byte codepages.
• chr(-1, "1252", "1252") will actually return a value but it is meaningless (ÿÿÿÿÿÿÿÿ1). Also it seems to alter over time;
• executing chr(188, "ISO8859-15", "ISO8859-15") with the default CPINTERNAL of "ISO8859-1" will print ¼. However, ¼ is not part of the target CP "ISO8859-15", Œ should have been printed instead. But this character is not part of the CPINTERNAL, so the character with same code is printed.
• "1252" defines at code 140 the character Œ (\u0152). The same character can be found in 8859-15 at position 188 (as noted in previous item). Theoretically the character should have been printed by
  message chr(140, "ISO8859-15", "1252").
  But it is not, instead errors 6063 and 1586 are issued.

chr(-1, "1252", "1252") will actually return a value but it is meaningless (ÿÿÿÿÿÿÿÿ1). Also it seems to alter over time;

Yikes! I wonder if this is memory overflow/underflow problem. If the 4GL directly adds the value (e.g. -1) to a C/C++ pointer (memory address) then a negative value might be looking outside of the conversion tables.

Unless this proves to be stable in some way that an application can use, we will probably consider this an unimplemented "quirk".

This is probably still a work in progress but just mentioned this here since we've found some of our tests in OO implementation were failing and that turned out to be because of some CHR related changes. Previously the CHR returned empty string, as it should or at least this is what 4GL does, now it returns space (%20). There was previously a condition in I18nOps that returned null for codes less or equal to zero, that was removed - not sure about the high watermark test (65535) though.

Marian, please provide the (isolated) testcases you refer to in note #4766-5. Please specify the active CP you are working with.

Marian Edu wrote:

[...] There was previously a condition in I18nOps that returned null for codes less or equal to zero, that was removed

It was not, see I18nOps.java:570

   private static String get4glCharacter(int ascCode, Charset charset)
   {
      if (ascCode < 0)
      {
         return null;
      }

not sure about the high watermark test (65535) though.

it returns the empty string. I do not think the new implementation will return a space except when the parameter is 0x20 / 32dec.

Ovidiu Maxiniuc wrote:

Marian Edu wrote:

[...] There was previously a condition in I18nOps that returned null for codes less or equal to zero, that was removed

It was not, see I18nOps.java:570

The test there is for less than zero, I was (trying) to refer to the case when the code number is actually zero, sorry for not being clear in my message :(

CHR gives " " (space) now while in 4GL is "" (empty).

Nice catch. The problem is not actually in I18nOps (which converts chr(0) to "\0") but in character constructor. Before assigning the value to its internal value, it is a bit processed by Text.javaSpacifyNull(). Apparently the instances of character data type are spacified in 4GL. Since there is a single \0 character, it will be converted to space. This is new for me. I will address this issue in the next commit.

The fix for chr(0) was committed in revision 12898/3821c.

Code Review Task Branch 3821c Revision 12898

The changes look good.

What is left to do in this task? Please note that in #6428 Joe is fixing an issue related to lead-byte processing in CHR. I don't know what else is needed but would like to list the items here.

I've tried some examples in 4GL and I discovered that for "€" symbol chr and asc behavior is weird.

display chr(14844588, "UTF-8", "UTF-8"). // display €
display chr(49827, "UTF-8", "UTF-8"). //display £

display asc("€", "UTF-8", "UTF-8"). // display 49792
display asc("£", "UTF-8", "UTF-8"). // display 49827

I can see what is displayed only if I set cpinternal to UTF-8. I haven't found other symbol with the same behavior.

Also, what is left to do on this task?

Delia, look at #4766-3, there are certainly several quirks of 4GL that may not translate "naturally" in the Java world (or maybe any other world). The goal of this task is to stress ASC and CHR as much as possible to check where 4GL breaks the normality. I expect that FWD will work in 90% of the cases with expected input, providing a predictable output. But what happens to the remaining 10%? Some customers may consciously (or not) rely on that 10% and FWD will fail in that case.

Please invest some time to stress these construct in different environments (with different codepages, characters, numbers, etc.). Ultimately, we need to spot some rules (i.e. message asc("www", "UTF-8", "UTF-8"). is working due to some predicable rules) and have them in FWD. For some cases (i.e. chr(-1, "1252", "1252")), the output is unpredictable as so we can't implement a proper Java replacement. In that case, we can consider it implemented.

Overall, an example like #4766-17 should work in FWD. If it doesn't we need to understand why asc("€", "UTF-8", "UTF-8") is 49792 and not 14844588. In this place, some investigation work is deemed: try different characters, collations, etc. and find out why the answer is like that (e.g. is always the most significant byte? is always the least significant byte? does this hold for any collation? etc.).

cpinternal and cpstream are vital for having this working. But mind that these are run-time settings. The encoding of the source-file is also relevant. If you hard-code characters in the source file, they will be encoded in the source file's cp. If they are read from a file, then the characters will use the cp of that file. My recommendation is to provide some structure to your investigation and make clear statements on your findings (i.e. 4GL yield X when the cpstream and cpinternal are set to Y and the source file encoding/input file encoding is Z). Your investigation shall be in this 3D space, highly dependent on the variants. When a clear rule is identified, it is safe to assume it in FWD.

I've tried to run a small program with cpinternal and cpstream set to CP949 and the result was a window with the following message: Unable to open word-break table file 248. (2736) The word-rule file specified by the -ttwdrul parameter is invalid. (9258). I receive this result for all other Japanese, Chinese and Korean encodings.
It could also be possible that this error might be coming from the OE or VM. But I have no way of proving that.

Is is necessary to work with multi-bytes encodings like those? And how should I proceed?

I don't know the answer, but by default most operating systems do not install support for DBCS codepages unless you explicitly request it. In other words, you may have to install that support at the OS level first.

Do those codepages show up in the list of codepages supported by OE? If not, then I wouldn't expect it to work.

Greg Shah wrote:

I don't know the answer, but by default most operating systems do not install support for DBCS codepages unless you explicitly request it. In other words, you may have to install that support at the OS level first.

Do those codepages show up in the list of codepages supported by OE? If not, then I wouldn't expect it to work.

I've added these languages on OS, but it still shows me the same message.

Multi-bytes codepages supported by OE:
https://docs.progress.com/bundle/openedge-abl-internationalize-applications/page/OpenEdge-support-for-multi-byte-code-pages.html#OpenEdge-support-for-multi-byte-code-pages

It can be supported in OE but that doesn't mean that the support is always installed. You can see the list of active codepages using the get-codepages() 4GL built-in function.

Delia Mitric wrote:

I've tried to run a small program with cpinternal and cpstream set to CP949 and the result was a window with the following message: Unable to open word-break table file 248. (2736) The word-rule file specified by the -ttwdrul parameter is invalid. (9258).

Are you specifying the '-ttwdrul' startup parameter, what is the OE version are you using? There are a number of cases logged on a similar issue, for instance if you are on a 11.6+ OE version this might help: [[https://community.progress.com/s/article/Unable-to-start-client-application-after-upgrading-from-11-4-to-11-6-when-using-code-page-other-than-the-installed-default-code-page-Error-message-2736-and-9258]]

Marian Edu wrote:

Are you specifying the '-ttwdrul' startup parameter, what is the OE version are you using? There are a number of cases logged on a similar issue, for instance if you are on a 11.6+ OE version this might help: [[https://community.progress.com/s/article/Unable-to-start-client-application-after-upgrading-from-11-4-to-11-6-when-using-code-page-other-than-the-installed-default-code-page-Error-message-2736-and-9258]]

I use 11.6+ version and adding -ttwdrul parameter solves the message written before, but when I try to run a program with one of DBCS codepages it displays two errors: Code page conversion table for ISO8859-1 to CP936 was not found in convmap.cp (6063) Could not find conversion table for r-code. (4706).
Are these errors normal even if I've found CP936 codepage with collation BASIC in the list of active codepages displayed by get-codepages() function?
I receive approximately the same errors related to convmap.cp using other codepages (for cpinternal and cpstream) found in the list of active codepages (with weird variations like different messages).

• cpinternal = ISO8859-1 / UTF-8 / 1252 chr(164, "ISO8859-15", "ISO8859-1") results in ¤. This character doesn't exists in 8859-15. It should display €. It seems like it looks just on source-codepage when it makes the conversion because on 164 position UTF-8 has nothing, but:
• cpinternal = ISO8859-15 chr(164, "ISO8859-15", "ISO8859-1") results in € character, which means that it doesn't base just on source-codepage because chr(164) in ISO8859-15 is €, but chr(164) in ISO8859-1 is ¤. Souldn't the result to be ? ?
• cpinternal = ROMAN-8 / IBM850 etc... chr(164, "ISO8859-15", "ISO8859-1") results in the character found on 164 position in cpinternal codepage
• cpinternal = UTF-8 chr(164, "ISO8859-15", "ISO8859-15") results in ¤. As i said before, chr(164) in UTF-8 results in nothing and chr(164) in 8859-15 is €.
• like Ovidiu says: executing chr(188, "ISO8859-15", "ISO8859-15") with the default CPINTERNAL of "ISO8859-1" will print ¼. However, ¼ is not part of the targetCP "ISO8859-15", Œ should have been printed instead. But this character is not part of the CPINTERNAL, so the character with same code is printed.

These are just a few weird examples that go to a chaotic "rule" based on multiple if statements in FWD if we want to implement them.
Probably, these weird results come from some mapping files used by 4GL.
Should I make chr(expression, target-codepage, source-codepage) to work as expected (related to cpinternal, source-codepage and target-codepage all combined) or take every single weird example and add an if to "translate" it in FWD?

Delia Mitric wrote:

I use 11.6+ version and adding -ttwdrul parameter solves the message written before, but when I try to run a program with one of DBCS codepages it displays two errors: Code page conversion table for ISO8859-1 to CP936 was not found in convmap.cp (6063) Could not find conversion table for r-code. (4706).
Are these errors normal even if I've found CP936 codepage with collation BASIC in the list of active codepages displayed by get-codepages() function?

There are many encoding settings used by the Progress session - internal, term, stream, log, print, r-code (in/out) - check out 'session system handle' in Progress help. Problem is you must use the same, or compatible, encodings for all those subsystems else it will greet you with that error about missing conversion mapping :(

In your particular case it looks like you're running with r-code and the sources were compiled with ISO8859-1 encoding, you might try to just remove the r-code files and let the compiler work with source code or re-compile using the encoding you plan to use as cpinternal (CP936).

• If char exists in cpi, get the code on which it exists in cpi: code_cpi. If char isn't in cpi, asc returns 63 (ASCII code for "?").
• If code_cpi is a valid code in source, get the characater on the code_cpi from source codepage: char_source. If code_cpi represents an undefined in source codepage, asc returns -1.
• Last step is to verify if char_source is in target codepage. If it is, asc returns the code on which char-source is in target: code_target. If char_source doesn't exists in target, asc returns -1.

• how errors occur when converting between different codepages relative to convmap.cp
• UTF-16(BE/LE) and UTF-32(BE/LE) codepages
• ASC(s, target, source), where s represents a string with a length > 1

I have some problems. Until now I've tested without changing source file encoding. I noticed that the rules doesn't match for all cases if source file encoding is different.
I get errors and I tried to solve them but the problem persists. If file source encoding is UTF-8 and cpinternal is 1252/CP949/CP936 or any code page that is not compatible with ISO8859-15, it shows me: Code page conversion table for ISO8859-15 to 1252 was not found in convmap.cp. (6063) Could not find conversion table for r-code. (4706). I've tried to remove .r files, but the errors persist.
This happens also for other source file encodings.

The source file encoding will affect the content of the string literals and I think it will also affect the processing of special characters and escape sequences by the preprocessor (e.g. #8027).

Marian: What other things are affected by the source encoding?

At a minimum, if you have comparison text in string literals in your tests, then the source encoding will change the results. It is important that our approach takes this into account. In FWD, we must define the source encoding at conversion time to properly read the 4GL source code (see Input File Encoding).

If in OE a file has the encoding set to a specific charset, it must be set also added in a .hints file in order to help conversion to understand the characters like in OE. For example, this is the content of a <filename>.p.hints file, assuming that the filename.p is encoded in ISO8859-1:

 <?xml version="1.0" encoding="UTF-8" standalone="no"?>
<hints>
    <preprocessor>
        <source-charset  value="ISO8859-1"/>
    </preprocessor>
</hints>

4GL's ASC function is affected by the relation between source file encoding and cpinternal code page. For example, if source file encoding is 1252 and cpinternal is UTF-8, display asc("€"). will display -1. This probably happens because "€" in 1252 encoding is represented by a single byte: 0x80, while in UTF-8 this byte has an undefined character on it.

After conversion .java files are encoded by default in UTF-8. How can I identify source file encoding from I18nOps class in order to use it in implementation of ASC functionality? Should I take this value from .hints file and assume this file is created?

4GL's ASC function is affected by the relation between source file encoding and cpinternal code page. For example, if source file encoding is 1252 and cpinternal is UTF-8, display asc("€"). will display -1. This probably happens because "€" in 1252 encoding is represented by a single byte: 0x80, while in UTF-8 this byte has an undefined character on it.

What is the converted Java for this specific example? I would expect that we read the 1252 character in properly and converted it to UTF-8. Is the problem that our conversion processing leaves the string as a valid UTF-8 Euro character instead of undefined?

After conversion .java files are encoded by default in UTF-8. How can I identify source file encoding from I18nOps class in order to use it in implementation of ASC functionality? Should I take this value from .hints file and assume this file is created?

The hints are not there at runtime. But we can change the converted code in some way to ensure that we match the OE behavior. Let me see the current Java code first.

Greg Shah wrote:

4GL's ASC function is affected by the relation between source file encoding and cpinternal code page. For example, if source file encoding is 1252 and cpinternal is UTF-8, display asc("€"). will display -1. This probably happens because "€" in 1252 encoding is represented by a single byte: 0x80, while in UTF-8 this byte has an undefined character on it.

What is the converted Java for this specific example? I would expect that we read the 1252 character in properly and converted it to UTF-8. Is the problem that our conversion processing leaves the string as a valid UTF-8 Euro character instead of undefined?

After conversion .java files are encoded by default in UTF-8. How can I identify source file encoding from I18nOps class in order to use it in implementation of ASC functionality? Should I take this value from .hints file and assume this file is created?

The hints are not there at runtime. But we can change the converted code in some way to ensure that we match the OE behavior. Let me see the current Java code first.

The Java code for display("€"). with source file encoding 1252 and the .hints file like this:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<hints>
    <preprocessor>
        <source-charset  value="windows-1252"/>
    </preprocessor>
</hints>

is:

FrameElement[] elementList0 = new FrameElement[]
         {
            new Element(asc("€"), frame0.widgetExpr1())
         };

If I don't add any .hints file, the Java code is:

 FrameElement[] elementList0 = new FrameElement[]
         {
            new Element(asc("�"), frame0.widgetExpr1())
         };

If we use .hints file, the resulted Java code has correct characters, but with UTF-8 byte representation. This means that some character that is represented on a single byte in 1252 (for example €) will be represented on 3 bytes in UTF-8 in the Java code.
My need is to have access to the source file encoding because ASC function depends on it. I need to know what is the source encoding of the character to see its initial byte representation and to make the verification relative to sourceCP and targetCP parameters of ASC function.

Greg Shah wrote:

After conversion .java files are encoded by default in UTF-8. How can I identify source file encoding from I18nOps class in order to use it in implementation of ASC functionality?

The hints are not there at runtime. But we can change the converted code in some way to ensure that we match the OE behavior. Let me see the current Java code first.

Progress is using the 'stream' encoding (cpstream) when reading any files, including source code - this affects the compiler when the source code gets compiled. There are two parameters there cprcodein and cprcodeout that I don't think it makes any sense in FWD - aka it will always have to be UTF8 since this is the encoding used by Java for 'text segments' (if such a thing even exists in Java). Of course when running on source code (compile on the fly) the same process happens at runtime.

Since FWD has an extra step added in between - the conversion - I think there are two options here, either save converted code in UTF-8 (do the codepage conversion during code conversion based on source code encoding specified, this is I think the current situation) or choose to keep the same encoding as the 4GL source code and specify that using the encoding javac parameter. Since the converted code is probably seen as derived resource and not meant to be edited manually I wouldn't even bother to keep the original source code encoding - again, this is something that one has to be aware but is not saved anywhere in the 4GL source code.

Yes, the .class files have a pool of String constants. AFAIK, they are always in UTF-8, regardless of the input sources. It's javac's job to do the unification and do the CP conversions on the fly, during the compile process.

Basically, FWD will do the same (generate the sources in UTF8, regardless of input files CP), provided that it knows the encoding of the 4GL original sources. In this case, if no CP was specified in the hints. It simply copied the bytes from input to output.

Note:
In the case of import, we have a global CP from cfg/p2j.xml, if that is specified using cpstream parameter. If no CP is encountered in the footer of the .d file, that will be used. I Do not remember having a similar setting for conversion. Instead, for parsing 4GL sources with different encoding, we add the -encoding parameter to the java conversion process.

Moving to UTF-8 as the standard for source code is an "upgrade" that is generally quite safe to do. This case of ASC() will require something different to get compatible results. That is OK.

We can output the source encoding into the converted code in some way. The nicest way would probably be to put it in class-level annotations for the containing business logic class. This would be nice because it could be in one place and is already associated with file-level state. I suspect there will be other cases where we may need this knowledge (in addition to ASC()) so having it in one place is cleaner.

The downside is that the ASC() function conversion would have still have to pass that information in some way or a link to the class from which it can be retrieved. Passing a class instance along is not nice. As an alternative, we could optionally create a static class member (e.g. a String like "1252") that stores this knowledge and then pass it along on any ASC() function (or other 4GL usage that needs it). That isn't as nice as the annotation but it is pretty clear, can easily be referenced without passing this.class around and would only be needed in those classes that depend on it. Let's go ahead with this idea.

I Do not remember having a similar setting for conversion. Instead, for parsing 4GL sources with different encoding, we add the -encoding parameter to the java conversion process.

No, we must not do this. Instead we set the value as noted in #4766-30.

Greg, ProcedureManager.thisProcedure() gives you the external program on top of stack, and the ProcedureManager$ProcedureData can store this annotation, which in turn can be resolved via THIS-PROCEDURE.

So, if we need to store the source encoding as an annotation at the Java class, that will work; I assume we will use this only if the source encoding is not UTF-8.

Greg Shah wrote:

We can output the source encoding into the converted code in some way.

Greg, just to be clear here...in 4GL it is NOT possible to have different encoding used across code base, it always must be the same (the stream codepage, which defaults to the internal one but that is another story). Even in Java the compiler will treat all source code equally, using the same encoding - either specified or the system default. Eclipse allows setting different encodings for each resource (file/folder) and are able to handle that in the builder but imho that is never a good idea to have different encodings used within the same code base.

Is it the CPSTREAM at COMPILE time? If so, then one set of .r files can be combined from the .r output of multiple different COMPILE statements and thus different encodings.

In regard to ProcedureManager.thisProcedure(), I was hoping to avoid the performance hit and avoid adding a deep dependency on the class structure to something that is supposed to be pretty independent (a "simple" ASC() function).

In regard to ProcedureManager.thisProcedure(), I was hoping to avoid the performance hit and avoid adding a deep dependency on the class structure to something that is supposed to be pretty independent (a "simple" ASC() function).

But with that said, Constantin's idea from #4766-39 is probably the best idea. We would just add a class-level annotation and then look it up in the runtime.

Greg Shah wrote:

Is it the CPSTREAM at COMPILE time?

Yes, unless running on source code so there is no r-code so the compile will be done before execution.

If so, then one set of .r files can be combined from the .r output of multiple different COMPILE statements and thus different encodings.

It is of course possible to compile different sets using different encodings but this is just if one wants to shoot himself in the foot imho as it only complicates the build process and add run-time codepage conversion overhead when this isn't really necessary for r-code that isn't meant to be human-readable in the first place.

Nevertheless, there is the rcode-info handle that can be used to read the encoding of an r-code file (codepage attribute). So if multiple encodings are used across compiled r-code the compiler will know how to handle the 'text segments' in those. The point is all those are strictly related to the COMPILER which I do not think it has an equivalent in FWD, you can say r-code in FWD is always UTF-8, it really doesn't matter. Source code on the other hand is human readable so supporting different encodings for the source code is a requirement but as I see it 'source code' here really means 4GL code that is going to be converter not the generated Java code, going even further to support different encodings for 'r-code' is not helping at all imho.

I agree that our equivalent of r-code (the converted Java classes) will always be encoded with UTF-8. Previously, the actual 4GL source encoding never mattered. Now these testcases show that the ASC() built-in depends upon the 4GL source encoding AND our implicit conversion to UTF-8 for the "r-code" causes a deviation in behavior for ASC().

Thus to fix it, we must know the original 4GL source encoding at runtime so that we can change ASC() to return a compatible result.

About #4766-35, I've tried some examples and I've changed the cpstream parameter to be different from the file encoding, and it doesn't seem to alter the result.
Take for example:
Encoded with 1252
cpstream = UTF-8

display asc("€", "UTF-8", "UTF-8"). // -1
display asc("€", "UTF-8"). // -1
display asc("€"). // -1

If cpstream is responsible for reading source code files, shouldn't these results be different from -1? More precisely, 0xE282AC = 14844588.

I'm sorry if I wasn't very clear earlier, but what I was trying to say is that if we have the converted java classes with .hints files, we only need the name of the source codepage and not the .java class encoded with a different codepage than UTF-8.
To be clear:
file.p - encoded in 1252:

display asc("€").
display asc("¥").

Byte representation (1252):
€ - 0x80
¥ - 0xA5

Byte representation (UTF-8):
€ - 0xE282AC
¥ - 0xC2A5

Well, with .hints file, the .java converted file will contain the wanted characters, but in UTF-8 encoding (different byte representation, but no different characters). This is not a problem at all as long as we have the correct character and the name of the source encoding in order to use them (in ASC) to find the source codepage byte representation for the character, whatever the encoding of java converted classes is.

As I understand from the discussion above, this can be done by adding an annotation as Constantin said #4766-39 and use it an runtime?

As I understand from the discussion above, this can be done by adding an annotation as Constantin said #4766-39 and use it an runtime?

Yes

Given the following case:

If we have a help.i file with source encoding = UTF-8 :

function get_euro returns integer():
    display asc("€").
    display asc("ô").
    display asc("ϰ").
    return 1.
end function.

This function is used in file.p, encoded with ISO8859-1:

{help.i}
get_euro(). 

For cpinternal = UTF-8 the results are : 14844588 and 50100 and 53168 (UTF-8 codes)
For cpinternal = ISO8859-15 the results are: 164 and 244 and 63 (ISO8859-15 codes)
This means that .i file is treated in its encoding independently from source encoding of file.p .

After conversion, .i file is copied in the .java file that uses it and if the source encodings are different it will not work like in OE because text literals from .java file are treated based on the source charset of the main .java file (set in the class annotation).

Should I consider this case?
If yes, should I use a similar approach like the one described in the above discussion?

Yuck! Well done in finding this weird behavior.

Should I consider this case?

Good question. I think this is a similar case as Marian's comment:

It is of course possible to compile different sets using different encodings but this is just if one wants to shoot himself in the foot imho as it only complicates the build process and add run-time codepage conversion overhead when this isn't really necessary for r-code that isn't meant to be human-readable in the first place.

This kind of thing seems like a bad idea. But then again, it is possible. If we exclude it, then we are leaving some work for the future. All too often we find that real customers do things like this, even when it is a bad idea.

If yes, should I use a similar approach like the one described in the above discussion?

The annotation approach won't work for this case since the section of included code will be arbitrary. There is no easy way to add annotations only to the code that came from the include. This is because the included code can be anything. It doesn't have to have any "nice boundaries" like a class, internal procedure or function. You can easily emit code that is a small part of a single line, is an entire line or crosses multiple lines. The code can be part of a larger construct (e.g. an internal proc) or can even be part of the start or end of a larger construct without containing the entire construct. There is no consistent way to annotate such code.

This means that our solution will have to change the converted code for ASC() itself. I think we should do something like this:

1. For each ASC() location, lookup the encoding for the file in which that function was defined.
2. Compare it to UTF-8.
   • If that input file (which could be a procedure, class or include) is already encoded in UTF-8, then nothing else needs to be done.
   • If it is a different encoding, then annotate the function call to store the encoding that needs to be emitted downstream.
3. Somewhere in the core conversion rules, we should create constant data members (e.g. final private static String ENCODING_ISO8859_15 = "ISO8859-15";) that define each of the encodings that can be referenced.
4. In convert/builtin_functions.rules, emit an extra parameter with the associated constant data member.
5. Modify the Java runtime for ASC() to:
   • Add variants that accept the extra parameter.
   • Properly implement the behavior needed.

Can we have the actual string in an include file, like asc({somechar.i})?, where somechar.i has "a"? Because if we are talking about encodings, it may be that what matters is where the string is defined, and not the include file of the ASC.

Also, can we have something like asc({somechar1.i} + {somechar2.i})? i.e. the text from multiple include files.

In these cases the BOM should be used when each file is opened (be it an .p, .w or .i). Unfortunately, it is often absent for utf-8 (most editors assume this encoding by default nowadays) and other CPs do not have such 'luxury'.

IMHO, we should handle all source files as written with same encoding. The 4GL developer should be consistent.

I am also thinking of another scenario: if the 4GL sources are compiled using a specific CP, so the .r files contain the text with this CP. What happens when they are executed using a different internal CP parameter?

Constantin Asofiei wrote:

Can we have the actual string in an include file, like asc({somechar.i})?, where somechar.i has "a"? Because if we are talking about encodings, it may be that what matters is where the string is defined, and not the include file of the ASC.

Also, can we have something like asc({somechar1.i} + {somechar2.i})? i.e. the text from multiple include files.

Yes, I tested and we can have cases like those.
In these cases, should I put the annotation on the text, right? Even it is represented by a literal or a char, longchar..

asc({somechar1.i} + {somechar2.i})

Can we go the extra mile and have asc("{somechar1.i}x{somechar2.i}")? Does the preprocessor work inside string literals? Or maybe asc({somechar1.i}x{somechar2.i}), but have the somechar1 as "a and somechar2 as b".

Alexandru Lungu wrote:

asc({somechar1.i} + {somechar2.i})

Can we go the extra mile and have asc("{somechar1.i}x{somechar2.i}")? Does the preprocessor work inside string literals? Or maybe asc({somechar1.i}x{somechar2.i}), but have the somechar1 as "a and somechar2 as b".

Yes, both cases are possible in OE; first if somechar1.i contains a and somechar2.i contains b, second if somechar1.i contains "a and somechar2.i contains b"

I've noticed multiple times that OE has an "nondeterministic" behavior for the same examples with the same combination of parameters/characters.
For example, when I'm trying to identify if the content of an included file is converted to the encoding of the file where it is copied. Sometimes the result is expected (the content is converted to the encoding of the "target" file) sometimes the result is unexpected (like I described above when the included file is treated is its encoding).
If I restart the VM or OE and recompile the files with same parameters, sometimes the behavior changes.

Example 1:
utf8.i file encoded with UTF-8:

function get_euro returns integer():
    display asc("€").
    display asc("ô").

    display "€".
    display "ô".

    return 1.
end function.

is used in 1252.p file encoded with 1252:

{utf8.i}
get_euro().

cpinternal = UTF-8

Result 1: -1 -1 â│¬ ô - this is the expected behavior if we think the .i file is copied in the target file
Result 2: 14844588 50100 € ô - the unexpected behavior (.i file is treated in its encoding)

Example 2:
utf.p encoded with UTF-8:

display asc("€").

cpinternal = UTF-8

Result 1: 14844588 - expected
Result 2: -1 - unexpected

What is the 4GL code for the COMPILE of these examples?

Greg Shah wrote:

What is the 4GL code for the COMPILE of these examples?

Sorry, what do you mean when you say the 4GL code for COMPILE?

You must run a the 4GL COMPILE language statement with very specific encoding options to get the compiled result to be consistent. How you are compiling?

Greg Shah wrote:

You must run a the 4GL COMPILE language statement with very specific encoding options to get the compiled result to be consistent. How you are compiling?

Isn't a pro -p .. -cpinternal compiling 'on the fly' with the specified encodings?

Constantin Asofiei wrote:

Greg Shah wrote:

You must run a the 4GL COMPILE language statement with very specific encoding options to get the compiled result to be consistent. How you are compiling?

Isn't a pro -p .. -cpinternal compiling 'on the fly' with the specified encodings?

I don't know. The fact that there were inconsistent results suggests:

1. OE has some compile or other factor that we aren't controlling explicitly which is the cause. OR
2. OE is truly indeterminate with this behavior.

I hope it is not cause 2. Having an explicit COMPILE should reduce the variables.

Just to add to the confusion: RCODE-INFO:CODEPAGE states this is the encoding of all texts in the text segment. But, what if you have a program with two or more includes, each include file with its own encoding?

I was not aware of this attribute. Its mere existence means each compiled piece of code my have independent CPs decided at compile time, which the runtime can read and use for correct string conversion operations. If/when that would be implemented in FWD, it would always return UNICODE, the de-facto CP of Java.

Therefore, writing 4GL code which correctly handles CP conversion, requires checking of this attribute. If the 4GL programmer use CHR/ASC without checking this attribute means s/he assumes a specific CP and that might not be the case if the procedure/class was compiled into a .r binary using a different CP.

Just to clarify a few things here, the whole idea of include files encoded in different code page than the source code where those are used is something that is not supported in 4GL - at least not as you seems to think, there is no per-file encoding setting so the AVM can magically figure out each file encoding of all pieces and manage to put everything together like pieces from different puzzles :)

• regardless of the OE IDE (Eclipse) that might support different encoding per file the AVM will really treat all source code as any other file on the OS file system - CPSTREAM is the session setting that specify that encoding.
• since the COMPILE session can take a specific RCODEOUT encoding the RCODE has that piece of information saved in the binary file because of the text segments (aka the string values in the source code - yes, the source itself and all include files used). When compiling the CPSTREAM and CPRCODEOUT encodings MUST be compatible, for unicode code pages there seems to be errors even if the CPINTERNAL is not unicode.
• in theory one can run RCODE compiled with different encodings, the AVM will handle conversion between those code pages and the internal codepage as needed. Of course those encodings MUST be compatible, otherwise the famous 'code page conversion table for X to Y was not found' error (6063) will occur at runtime.
• there is also the RCODEIN startup parameter that can be used to override the RCODE-INFO:ENCODING saved in the compiled rcode, this could help go around the codepage conversion error but the results might be wrong as the AVM will interpret the text segments in the rcode as having the specified encoding even if the one used at compile time is different.
• the compiler will handle code page conversion between the CPSTREAM and the CPRCODEOUT, the initial encoding in source code isn't saved anywhere.
• using 'extended' characters in source code is not a good practice (to say at least), hard-coded values in general but the code base is full of error/log messages that might include those. However the source code base encoding should be known and enforced really I've never (ever) saw and situation where different parts of the code base is compiled with different code pages.

• one can specify both the source and target codepage when using it
• the default for target is CPINTERNAL
• the default for source is CPINTERNAL if variables are used or the text segment encoding for string values
• the text segment is the one saved in RCODE-INFO for rcode or the CPSTREAM when running on source code - compile on the fly.
• the 4GL developer doesn't need to care about RCODE encoding to be able to use ASC, really it doesn't - I personally don't see any use case for the whole RCODE-INFO system handle :)
• specifying the source code page for string values (with extended character) WILL be affected by the CPSTREAM when running on source code or the setting used when the file is compiled.

Unless the COMPILE statement is actually supported in FWD I really don't see any need to implement that RCODE-INFO:ENCODING, the way I see it the CPSTREAM used when compiling in 4GL is the equivalent of the source code encoding set at conversion. From that point on you can say the text segments in 'RCODE' is really UTF-8 so that system handle property will always just return UTF-8 :)

You can go the extra mile and allow them to specify the conversion RCODEOUT code page and use that encoding in generated Java classes, then you must specify the source code encoding when compiling the Java classes and it will eventually end-up in a binary class file (strings there are UTF-8 as far as I know, to the extent one can speak about text in a binary file of course). Saving that information in every source code using annotations is of course possible but what value it will bring, it's going to be exactly the same in all code base unless you provide option to convert parts of the code base using different conversion encoding :)

Greg Shah wrote:

You must run a the 4GL COMPILE language statement with very specific encoding options to get the compiled result to be consistent. How you are compiling?

I'm not compiling in a specific way. I'm not using COMPILE statement or compile from the command line.
I modify startup.pf file (cpinternal / cpstream parameters) and run the program from OE after saving the files.

In some way, the results I get after opening VM are consistent. After more examples, the results are chaotic and make no sense.

1. Don't use the procedure editor or any IDE support to implicitly compile.
2. Don't run from source code (which is a kind of implicit compile).
3. Create a 4GL program that explicitly uses the COMPILE statement with the exact features needed. Run that program from the command line and explicitly pass any cp* values as command line options.

This will give you complete control over the results and should be stable.

Is there something different implemented in this convmap than is implemented using the character sets in Java? In other words, would our dynamic approach of using Java charsets also detect that these characters (like the Euro symbol) cannot be converted?

If the convmap behavior is just the standard way that the charsets would work and is compatible with how Java would do it, then we can detect this issue at runtime and map it into the same result as in the 4GL.

We need to know if the behavior differs (between OE and Java) or not.

Greg Shah wrote:

Is there something different implemented in this convmap than is implemented using the character sets in Java? In other words, would our dynamic approach of using Java charsets also detect that these characters (like the Euro symbol) cannot be converted?

If the convmap behavior is just the standard way that the charsets would work and is compatible with how Java would do it, then we can detect this issue at runtime and map it into the same result as in the 4GL.

We need to know if the behavior differs (between OE and Java) or not.

• 1252 maps € on 128 code
• IBM858 maps € on 213 code
• IBM850 doesn't map €

BUT, when we try to make asc("€", "IBM850", "1252") we receive 213 (the code from IBM858 - this is how is mapped).

What do you suggest?

Delia Mitric wrote:

BUT, when we try to make asc("€", "IBM850", "1252") we receive 213 (the code from IBM858 - this is how is mapped).

Delia, this really depends on what encoding is used for the source code - what you see is probably just an oddity of the 'procedure editor' (any 4GL editor widget as a matter of fact). You are probably running with default encoding (ISO8859-1) and there is no euro sign there still the editor will accept that if you paste it and it will display just fine. However, just trying to get the screen-value of that editor widget will show you that 'euro sign' is not valid - just try to type '€ a' and it will show like '? a'. Just to be sure try to save that code you run in a file on disk and open it with another editor where you can change the encoding or a hex editor - you will see the 'euro sign' is saved as 0x80 (alt 128) which is the windows shortcut for international characters so it's more like a 'mirage', there is no euro sign in ISO8859-1.

Delia Mitric wrote:

My question is how to introduce this convmap tables that specify very clearly (and have no general rule) how a character from an encoding is mapped into another encoding?

Progress does provide some default i18n support but I think from the very beginning they did provided the option to roll your own. Updating conversion tables, collations, word breaking rules and all that is possible in Progress... how many are actually doing that in the real world I really have no idea, I suspect very few if any. It might just be something they thought to make it extensible/customisable when implemented but no one felt incline to do so :)

https://docs.progress.com/bundle/openedge-abl-internationalize-applications/page/Modify-compile-and-provide-access-to-the-CONVMAP-file.html

I am also thinking of another scenario: if the 4GL sources are compiled using a specific CP, so the .r files contain the text with this CP. What happens when they are executed using a different internal CP parameter?

This for Ovidiu's comment. In #8027, Octavian did some work into processing ~u0041 kind of constructs into a single character. In the .r files, the strings are already processed, so the compilation takes care of ~u0041 and converts to the right character. Now, if you use more specific ~u1234 (to yield a character from outer space), maybe we can learn more on how the COMPILE works. As previously mentioned, CLI should be used and the .r file introspected. From my POV, the Java strings should be the same as the .r strings. Any visual differences in the terminal or GUI may be due to a different CP at run-time! In one experiment, the character from the .r code (generated by ~uXXXX) was different from the one on screen.

Delia Mitric wrote:

I think these tables are really relevant for functions like ASC/CHR, mostly for characters that have different codes from an encoding to another.

I've never said isn't relevant, it's how Progress works and in this specific case it's even 'documented' as not really the 'standard'. To be perfectly compatible with Progress those convmap files must be used and don't just use the codepage conversion from Java. But the 4GL has a lot of history behind and many of those encodings are not even used and therefore there is really no need to bother with conversion support between them but this is just my PoV, to implement that 100% as in Progress the conversion map files must be used - as I've said there are other things like collations and word breaking rules that also come into play.

We actually have the same issue when writing tests, chr returns valid 'symbol' for code points that should not be mapped for a specific code page and that is also because of those 'deviations' implemented through the convmap files :(

OK, so the default OE convmap files do not implement a standard charset conversion. Let's move forward in this way:

1. At this time, FWD does not support customized convmap (nor other tables like collation or word break). We will not implement this at this time. In the future, if a customer requires it, we can implement these features. So far, no customer actually has had any such customization.
2. We must implement the same behavior as OE for the default convmap on those character sets which we support. This is currently 8859-1, 8859-15, 1252. In #6431 we have to also add both UTF-8 and CP936.
3. I'm open to supporting all charsets that OE supports. We probably should just do that now otherwise it will be extra work to debug and extend until the end of time.

The bottom line: make FWD operate like the default OE approach even if it means we have to do some custom nonsense in addition to the standard Java charset processing.

Thank you for your answers!🙂

We must implement the same behavior as OE for the default convmap on those character sets which we support. This is currently 8859-1, 8859-15, 1252. In #6431 we have to also add both UTF-8 and CP936.

It is ok to implement this tables using hash maps as constants in the class I'm working or to save them into separate files accessed at runtime?

I'm open to supporting all charsets that OE supports. We probably should just do that now otherwise it will be extra work to debug and extend until the end of time.

So, I will focus to introduce the convmaps just for the encodings that are already supported by FWD for the start?

We must implement the same behavior as OE for the default convmap on those character sets which we support. This is currently 8859-1, 8859-15, 1252. In #6431 we have to also add both UTF-8 and CP936.

It is ok to implement this tables using hash maps as constants in the class I'm working or to save them into separate files accessed at runtime?

Yes, use in-memory maps. Later on, if we add some ability to customize, we will provide directory configuration. We would not use separate files in the file system.

I'm open to supporting all charsets that OE supports. We probably should just do that now otherwise it will be extra work to debug and extend until the end of time.

So, I will focus to introduce the convmaps just for the encodings that are already supported by FWD for the start?

No, I think it is better to support all encodings now unless you assess it will take a crazy amount of time. My thinking is that doing it later (a little at a time) will mean more work in the long term than just doing it once. Each time someone does this, they will have to get "up to speed" and learn what you already know. Let's get it done now.

I am a bit pessimistic about this.

The difference between FWD and OE is that the OE actually stores the strings using the selected CP in memory, while in FWD we rely on String for that. OE will actually do the conversion by rewriting the bytes in memory or reinterpreting them (that's the role of the two CP parameters in ASC / CHR functions). OTOH we always let Java handle the character data using internal encoding, only using the CP when the character data is 'exported' in some way.

Having a byte in a character buffer in memory does not lead to a meaning of it, without knowing the CP used when it was stored. Of course, a global CP encoding exists in OE, but sometimes in multi-language application, that is not enough. This was the reason for the 3rd parameter of ASC / CHR functions. So ASC(ch, CP1, CP2) should be interpreted as: "what is the code of the first character stored in ch buffer in the CP1 encoding, assuming the string data ch was not using the CPinternal encoding, but CP2?". And the answer requires to analyse that first byte of ch using CP2 (source CP) and interpret it as a specific character 'alpha'. Then look up it in the CP1 (target CP) table. If found, return the code; otherwise fail.

Historically speaking, the multiple single-byte CPs and multi-bytes support in C/C++ and other languages was a shabby solution to an actual problem: allow usage of a larger character-set, but keeping backward compatibility with existing binaries. The sole idea of having multiple single-byte CPs means there will NOT be a full conversion map between them. Each CP stores different characters in order to cover necessities of a specific languages. Having a full mapping between two CPs means they are just permutations of each other and that is not useful.

Java was born with that lesson learned. The String will store the UNICODE. We work directly, with the 'characters', not one of its image through an CP encoding.

• use byte array and actual encoding of the characters in required CPs in a wrapper class as replacement of String;
• have a hard hit on performance when we do the extra conversions from Java's native ('alpha') to CPinternal then to CP specified in the ASC parameter.

Java was born with that lesson learned. The String will store the UNICODE. We work directly, with the 'characters', not one of its image through an CP encoding.

I fully understand. It is one of many, many, many times we have had to "dumb down" our implementation from a good (a.k.a. more modern) starting point to provide compatibility with an environment designed in the 1980s.

The customers already have these legacy applications. We must provide that compatibility to make it feasible to move to the modern Java-based stack.

have a hard hit on performance when we do the extra conversions from Java's native ('alpha') to CPinternal then to CP specified in the ASC parameter

This one.

I'm working on adding the convmaps from OE and for now I added just the convmaps for combinations of codepages cp1-cp2, where at least one of these codepages is a 1-byte codepage (conversion tables have 256 elements).
I implemented these tables using lists of 256 elements that are initialized when a call of ASC is made with a specific combination of codepages: ASC(ch, targetCP, sourceCP) -> will "invoke" a method that initializes the "convmap" if it isn't initialized yet.

OE contains other convmaps between multi-byte codepages (like CP949 and UTF-8). These convmaps are much bigger (aprox. 13000 elements). As I've noticed, the rule I found for ASC works for these combinations of codepages (I didn't test every character because there are thousands).
Do you think it is necessary to add also these huge convmaps in FWD?

Delia Mitric wrote:

I'm working on adding the convmaps from OE and for now I added just the convmaps for combinations of codepages cp1-cp2, where at least one of these codepages is a 1-byte codepage (conversion tables have 256 elements).
I implemented these tables using lists of 256 elements that are initialized when a call of ASC is made with a specific combination of codepages: ASC(ch, targetCP, sourceCP) -> will "invoke" a method that initializes the "convmap" if it isn't initialized yet.

Delia, we've also added more tests for chr/asc and how text segments in source code gets interpreted when compiling (or run on source code, compile on the fly). The tests for rcode-info are using `compile` statement so do not think those really applies for FWD but we've added just to try to explain how OE works when compiling source code - aka the stream and rcode-out encoding used for the session play a role in this case. Not sure if rcode-in session parameter needs to be tested, this is not recommended to be used anyway as it simply overrides the encoding saved in the r-code pretending to be be something else, this might cause issues but guess the use case was to avoid unnecessary code page conversions on large r-code base compiled with a different rcode-out encoding, hopping there are no non-ascii characters in the code base :)

All the tests for i18n stuff is in tests/i18n folder of testcases project.

Do you think it is necessary to add also these huge convmaps in FWD?

Unfortunately, yes.

We can provide these conversion tables are resources inside the jar, we only need to write a class that initializes from them when needed (maybe with a factory method so that the instances of conversions can be cached). This way, we can add supplementary conversion tables (as text files), as needed, without additional coding.

Ovidiu Maxiniuc wrote:

We can provide these conversion tables are resources inside the jar, we only need to write a class that initializes from them when needed (maybe with a factory method so that the instances of conversions can be cached). This way, we can add supplementary conversion tables (as text files), as needed, without additional coding.

I was thinking at something like that too, but in #4766-76 was said:

Yes, use in-memory maps. Later on, if we add some ability to customize, we will provide directory configuration. We would not use separate files in the file system.

One thing to note is that UTF-16 / -16LE / -16BE and UTF-32 / -32LE / -32BE present weird behavior.
As I've noticed is not the single case when these codepages conduct to strange results. In #9445-8 are mentioned problems with these encodings.
In my cases, I've tried to use them in combinations of codepages as parameters for ASC. I can't identify a general behavior.

Another thing to note:
I've run more examples and I found that the conversion tables are used in some cases, not always (when they're defined).
I've noticed that they are used when a character c from cp1 codepage doesn't exists in the target codepage cp2 and conversion table for cp1-cp2 combo exists, example #4766-72.

Ovidiu Maxiniuc wrote:

We can provide these conversion tables are resources inside the jar, we only need to write a class that initializes from them when needed (maybe with a factory method so that the instances of conversions can be cached). This way, we can add supplementary conversion tables (as text files), as needed, without additional coding.

Just a small thing here, OE uses a 'compiled' version of the conversion map (binary) - $DLC/convmap.cp, this basically have the built-in conversion maps from $DLC/prolang/convmap. If you need 100% compatibility with OE I would focus on supporting the 'convert syntax' in the .dat files and compile that to any binary format you see fit. In case someone is using a custom conversion map(s) those conversion map definition files will need to be 'compiled', just like in OE.

Delia Mitric wrote:

One thing to note is that UTF-16 / -16LE / -16BE and UTF-32 / -32LE / -32BE present weird behavior.
As I've noticed is not the single case when these codepages conduct to strange results. In #9445-8 are mentioned problems with these encodings.
In my cases, I've tried to use them in combinations of codepages as parameters for ASC. I can't identify a general behavior.

It worth nothing to say UTF-8 is the only unicode encoding supported as internal code page. From our tests using those encodings for stream will have strange effects, the code sometimes shows compile errors (if ran on source code) or the execution will be simply wrong, I would think the compiler isn't really able to 'digest' code written using those encodings and are only supported for 'data exchange'.

We can provide these conversion tables are resources inside the jar, we only need to write a class that initializes from them when needed (maybe with a factory method so that the instances of conversions can be cached). This way, we can add supplementary conversion tables (as text files), as needed, without additional coding.

I was thinking at something like that too, but in #4766-76 was said:

Yes, use in-memory maps. Later on, if we add some ability to customize, we will provide directory configuration. We would not use separate files in the file system.

It is perfectly fine to load these resources from the jar at runtime in order to create the in-memory maps. Lazy loading is good because in most cases we only will have to load a small number of the maps.

We definitely want the resources to be in the jar file. Let's discuss the format to use.

We can provide these conversion tables are resources inside the jar, we only need to write a class that initializes from them when needed (maybe with a factory method so that the instances of conversions can be cached). This way, we can add supplementary conversion tables (as text files), as needed, without additional coding.

Just a small thing here, OE uses a 'compiled' version of the conversion map (binary) - $DLC/convmap.cp, this basically have the built-in conversion maps from $DLC/prolang/convmap. If you need 100% compatibility with OE I would focus on supporting the 'convert syntax' in the .dat files and compile that to any binary format you see fit. In case someone is using a custom conversion map(s) those conversion map definition files will need to be 'compiled', just like in OE.

For now I want to avoid this. When we need to add this feature (custom conversion map support), we can add this parser/transform into our tooling.

But for the built-in/standard conversions, I don't want to read the convmap files directly so that there cannot be any complaint about copyright issues. Instead, we can write 4GL code to convert each char in a given source charset to the result in the target charset. This can be written directly into the format we will load from our jars. In other words, you cannot copyright the knowledge of what the conversion does, only the expression of that conversion as a file (convmap). It is stupid but we must be careful.

Greg Shah wrote:

But for the built-in/standard conversions, I don't want to read the convmap files directly so that there cannot be any complaint about copyright issues. Instead, we can write 4GL code to convert each char in a given source charset to the result in the target charset.

As part of the changes done on the asc/chr tests I've already used this kind of 'conversion maps' but only using UTF-8 as 'common language', normally the routines are there just needs to be called with proper source/target code pages.

This can be written directly into the format we will load from our jars.

For tests we wanted to keep that as simple as possible so we just dump code points mapping from source/target, each mapping pair on a new line - codes are in decimal format not the hex format used in convmap files (well they do mix hex/decimal, hex is only used for extended characters and 8-bit is using decimal).

In other words, you cannot copyright the knowledge of what the conversion does, only the expression of that conversion as a file (convmap). It is stupid but we must be careful.

I'm not suggesting to use the conversion files from OE, the documentation for the syntax of those files is not extensively documented but one can make sense of it to some extent and I don't think having an antlr parser for that is a copyright violation :)

I'm not suggesting to use the conversion files from OE, the documentation for the syntax of those files is not extensively documented but one can make sense of it to some extent and I don't think having an antlr parser for that is a copyright violation :)

It certainly isn't but if it is there someone might think we used it on the copyrighted stuff. If we write 4GL code to generate this, there is no concern and no need to write the parser (until a customer requires it for their own maps).

Delia Mitric wrote:

One thing to note is that UTF-16 / -16LE / -16BE and UTF-32 / -32LE / -32BE present weird behavior.
As I've noticed is not the single case when these codepages conduct to strange results. In #9445-8 are mentioned problems with these encodings.
In my cases, I've tried to use them in combinations of codepages as parameters for ASC. I can't identify a general behavior.

This is strange for me. That's because these are, in fact, the no-op encoding, keeping a bijection with UNICODEs (only for first 64Ki characters in case of UTF-16 variants).

Another thing to note:
I've run more examples and I found that the conversion tables are used in some cases, not always (when they're defined). I've noticed that they are used when a character c from cp1 codepage doesn't exists in the target codepage cp2 and conversion table for cp1-cp2 combo exists, example #4766-72.

That's the reason for having multiple CP, to cover multiple sets of characters. If they were define not to overlap at all (I do not think such examples exists, the basic punctuation and numbers are probably mandatory) then the CP conversion would be useless (that is, noting to translate).

For the resource files:
I do not think there are many multibyte CPs. And at 16 bit, the table would become pretty difficult to create / manage manually. So they should be generated. Either using 4GL code, or in Java. I am thinking whether we can generate the conversion tables on-the-fly.
Here is the pseudo-code for conversion for two 8 bit CPs: cp1 to cp2:

private byte[] conv = new byte[256];

for (int i = 0; i < 256; i++) {
   String chStr = new String({(byte) i}, cp1); // chStr.charAt(0) is the character

   try {
       byte[] cp2Code = chStr.getBytes(cp2);   // cp2Code.length should be 1
       conv[i] = cp2Code[0];
   } catch (UnsupportedEncodingException uee) {
       // skip it since this char is not supported/present in cp2
   }
}

I am curious how the results of this algorithm compare on 4GL and Java.

Ovidiu Maxiniuc wrote:

For the resource files:
I do not think there are many multibyte CPs. And at 16 bit, the table would become pretty difficult to create / manage manually. So they should be generated. Either using 4GL code, or in Java. I am thinking whether we can generate the conversion tables on-the-fly.
Here is the pseudo-code for conversion for two 8 bit CPs: cp1 to cp2:
[...]Of course, this would work only if cp1 to cp2 are two installed (provided by Java) StringEncoder s.

I am curious how the results of this algorithm compare on 4GL and Java.

The algorithm doesn't generate the same default convmaps provided by OE .
The main difference is that if a character c from cp1 doesn't exists in cp2, in OE it is mapped to another character (from cp2). See #4766-72 table 128 mapped to 213 case (euro character).
The algorithm (in Java ) gives 63 for not found characters in cp2, which is the code of ? (unknown).

Delia Mitric wrote:

The main difference is that if a character c from cp1 doesn't exists in cp2, in OE it is mapped to another character (from cp2). See #4766-72 table 128 mapped to 213 case (euro character).

And these are the cases when my rule need to look at the convmaps (when c is not found in target_codepage), in rest it doesn't need them.

Don't look at the convmap. Just write the generator code in 4GL. We don't care what is in the convmap because we are just trying to duplicate how the 4GL behaves. In the end, I presume they will be the same.

My point: you can't get the proper result by generating the maps in Java.

Delia Mitric wrote:

Delia Mitric wrote:

The main difference is that if a character c from cp1 doesn't exists in cp2, in OE it is mapped to another character (from cp2). See #4766-72 table 128 mapped to 213 case (euro character).

And these are the cases when my rule need to look at the convmaps (when c is not found in target_codepage), in rest it doesn't need them.

To be clear here:

1. For copyright reasons we must not look at the convmap files shipped with OE. Instead we can write 4GL code to test all inputs for each combination and thus we can generate the same output as we presumeably would find in the convmap files themselves. Doing it this was is completely safe from an intellectual property perspective.
2. We cannot use Java to generate these maps since the OE mappings have differences. So the code must be written in the 4GL.
3. We can analyze the generated maps and only store the differences between Java and OE. In other words, we can probably reduce these maps down to the very small sets of OE-specific mappings. If a conversion is not in our loaded maps, then we can default to whatever the Java conversion would emit.

I'll be interested to know how many map entries are really needed (for the diffs). If the number is small enough, we can encode these as static data in our I18N classes somewhere and just load the map as a static resource at class init. If the diff mappings is a larger data set, then we will stick with the "load the resource lazily from the jar" approach.

Another thing: there are some "empty" tables (especially for combinations with cp1 - UTF16/LE/BE). That means that cp1 and UTF-16/Le/BE are compatible; the tables are empty, but the conversion works (in some way).
Do you think is necessary to generate files for these cases? (Consider that I still can't identify a clear rule for how ASC/CHR works for these UNICODE codepages UTF-16, UTF-32...)

It will depend on how we encode this data. If the mappings are resources in the jar then an empty file (maybe with a comment inside that says: there are no differences) might work.

Or we can encode the knowledge into our I18N classes in some way. All else equal, you can pick the approach that is easiest to maintain and understand.

The problem is Java Charset doesn't support all 4GL supported codepages (for example 708, 709, IBM851) and this "filtering" cannot be done for all files.
If this codepages aren't supported by Java, I don't know how we should obtain the same results as in 4GL.

Please check on each of these cases. Java may actually have a charset for it, but just by a different name. For example, CP708 is supported in Java as ASMO-708 or as ISO 8859-6.

Greg Shah wrote:

I'll be interested to know how many map entries are really needed (for the diffs). If the number is small enough, we can encode these as static data in our I18N classes somewhere and just load the map as a static resource at class init. If the diff mappings is a larger data set, then we will stick with the "load the resource lazily from the jar" approach.

There are files with thousands of different entries. For example, a file can contain 7689 or 16960 entries (conversion between multi-byte charsets), but there are also files with 0 entries so I think we will use the second variant.

Please check on each of these cases. Java may actually have a charset for it, but just by a different name. For example, CP708 is supported in Java as ASMO-708 or as ISO 8859-6.

There are still some encodings that cannot be used by Java (MAZOVIA, ROMAN-8, 709, 710...) https://docs.oracle.com/javase/8/docs/technotes/guides/intl/encoding.doc.html.

One thing to note: in 4GL, these conversion tables works in both ways.
For example, the table:
source = 1252
target = ibm850
it is used for 1252 - > ibm850 conversion, but also for ibm850 -> 1252 conversion if no noinverse keyword is specified.

I covered this by "inversing" the contents of generated files and put them in new files to express the conversion from target to source. This is important because the content of diffs files are different if we change the way.

Greg Shah wrote:

Don't look at the convmap. Just write the generator code in 4GL. We don't care what is in the convmap because we are just trying to duplicate how the 4GL behaves. In the end, I presume they will be the same.

Something to note here: the generated convmaps aren't the same like the default convmaps. I used ASC to generate them and this shows that the default convmaps aren't used every time.

Since we have .txt files for the diffs, where (and how) should I put these files in order to load them at runtime?

They must be loaded from our jar file. Since they are static, we will check them in to the FWD project itself and change the build to include them in the jar.

How about com.goldencode.p2j.util.i18n.resource.*? It should be isolated for these special "resources" and it should be specific to I18N.

The files themselves should probably be named something like <source_charset>_to_<target_charset>_mapping_diffs.dat. Such names would then be pretty clear in their purpose.

It is necessary to add files for these cases?

Does OE have a default conversion that they implement in these cases? If it doesn't have a default conversion, then is there some error?

Either way, we should duplicate the OE behavior.

Greg Shah wrote:

It is necessary to add files for these cases?

Does OE have a default conversion that they implement in these cases? If it doesn't have a default conversion, then is there some error?

Either way, we should duplicate the OE behavior.

There is no error. Probably there is a logic implemented in 4gl relative to Unicode based codepages. I've noticed that every codepage "compatible" with UTF-8 is also compatible with UTF-16/ UTF-32 and their variants (LE/BE) even though no specific convmap is found.

• chr(14844588) in UTF-8 is EURO
• 14844588 is hex is E282AC

• ASC(chr(14844588), "UTF-16", "UTF-16"). run with cpinternal = UTF-8 results in: 11305698 in hex = AC82E2
• ASC(chr(14844588), "UTF-16LE", "UTF-16LE"). run with cpinternal = UTF-8 results in: 11305698 in hex = AC82E2
• ASC(chr(14844588), "UTF-16BE", "UTF-16BE"). run with cpinternal = UTF-8 results in: 11305698 in hex = AC82E2

All of UTF-16/UTF-16LE/UTF-16BE give the same results and I think it is not correct because they're refer to different byte representations. I think at least UTF-16BE must give this result: 14844588 which means E282AC in hex. Same for UTF-32 ant its representations.
It is possible to be a bug in the Progress version I use? (11.6)

Delia Mitric wrote:

• ASC(chr(14844588), "UTF-16", "UTF-16"). run with cpinternal = UTF-8 results in: 11305698 in hex = AC82E2
• ASC(chr(14844588), "UTF-16LE", "UTF-16LE"). run with cpinternal = UTF-8 results in: 11305698 in hex = AC82E2
• ASC(chr(14844588), "UTF-16BE", "UTF-16BE"). run with cpinternal = UTF-8 results in: 11305698 in hex = AC82E2

The character returned by CHR is using the internal codepage (UTF-8 if this is what you're using) but then you say the source codepage is something else when calling ASC.

ASC(CHR(14844588, 'UTF-8', 'UTF-8'), "UTF-16LE", "UTF-8") => 8364 (x20AC)
ASC(CHR(14844588, 'UTF-8', 'UTF-8'), "UTF-16BE", "UTF-8") => 44064 (xAC20)

It is possible to be a bug in the Progress version I use? (11.6)

Not really, 12.8 gives the same results but isn't really a bug.

ASC(CHR(14844588, 'UTF-8', 'UTF-8'), "UTF-16LE", "UTF-16LE") => 11305698
ASC(CHR(14844588, 'UTF-8', 'UTF-8'), "UTF-16BE", "UTF-16BE") => 11305698

Marian Edu wrote:

Delia Mitric wrote:

• ASC(chr(14844588), "UTF-16", "UTF-16"). run with cpinternal = UTF-8 results in: 11305698 in hex = AC82E2
• ASC(chr(14844588), "UTF-16LE", "UTF-16LE"). run with cpinternal = UTF-8 results in: 11305698 in hex = AC82E2
• ASC(chr(14844588), "UTF-16BE", "UTF-16BE"). run with cpinternal = UTF-8 results in: 11305698 in hex = AC82E2

The character returned by CHR is using the internal codepage (UTF-8 if this is what you're using) but then you say the source codepage is something else when calling ASC.

Yes, because I'm trying to see how "conversion" from cpinternal to source codepage is made. I think cases like this one are unwanted but they're still work in 4GL :(.

Delia Mitric wrote:

Yes, because I'm trying to see how "conversion" from cpinternal to source codepage is made. I think cases like this one are unwanted but they're still work in 4GL :(.

But there is no conversion there, for CHR you don't specify the source not target so it's cpinternal and for ASC you specify the same values for source/target - it's just that the character doesn't have the encoding you specify as source. There seems to be some kind of 'conversion' given it does returns something but the behaviour isn't documented and I have no idea what is doing if one 'purposely' misleading it like this :)

Delia Mitric wrote:

It is possible to be a bug in the Progress version I use? (11.6)

• it is the 8364th character in the Unicode set therefore we say its decimal code point is 8364;
• in hexadecimal this value is written as 20AC so, in Java we encode the character as \u20AC. This is also the encoding for UTF-16, where all the characters use exactly 16 bits (2 bytes): #20 #AC. This is the BE encoding, where the most significant byte first. In LE the order of the bytes is reversed.
• in UTF-8, the number of bytes is variable, but the idea is to allow most used characters to use shorter sequences. Some kind of Huffman code, like Morse. This is an prefix-free encoding, therefore some byte values are reserved as prefixes. Because of this, the information for less used characters (like €) will require more bytes than the 16-bit representation. The actual payload is actually split between all 3 bytes. See the Description in wikipedia. In our case:

20AC(16) = 0010 0000 1010 1100 (2) but we group the bytes as 0010 000010 101100 (2)

If you link these bytes you'll get #E282AC (14,844,588) in BE or #AC82E2 (11,305,698) in LE encoding.

This was encoding in I18nOps.get4glCharacter() .

Ovidiu Maxiniuc wrote:

Let's use € (EURO SIGN) as reference.

• it is the 8364th character in the Unicode set therefore we say it's decimal code point is 8364;
• in hexadecimal this value is written as 20AC so, in Java we encode the character as \u20AC. This is also the encoding for UTF-16, where all the characters use exactly 16 bits (2 bytes): #20 #AC. This is the BE encoding, where the most significant byte first. In LE the order of the bytes is reversed.
• in UTF-8, the number of bytes is variable, but the idea is to allow most used characters to use shorter sequences. Some kind of Huffman code, like Morse. This is an prefix-free encoding, therefore some byte values are reserved as prefixes. Because of this, the information for less used characters (like €) will require more bytes than the 16-bit representation. The actual payload is actually split between all 3 bytes. See the Description in wikipedia. In our case:

20AC(16) = 0010 0000 1010 1100 (2) but we group the bytes as 0010 000010 101100 (2)

First code point	Last code point	Byte 1	Byte 2	Byte 3	Byte 4
U+0800	U+FFFF	1110 wwww	10 xxxxyy	10 yyzzzz	-
U+20AC		1110 0010	10 000010	10 101100	-

If you link these bytes you'll get #E282AC (14,844,588) in BE or #AC82E2 (11,305,698) in LE encoding.

This was encoding in I18nOps.get4glCharacter() .

Thank you for your explanation! :)

Marian Edu wrote:

There seems to be some kind of 'conversion' given it does returns something but the behaviour isn't documented and I have no idea what is doing if one 'purposely' misleading it like this :)

What I don't understand is why BE result is the same as the LE result and I don't know how should I treat these cases.

Delia Mitric wrote:

What I don't understand is why BE result is the same as the LE result and I don't know how should I treat these cases.

You mean for:

ASC(chr(14844588), "UTF-16LE", "UTF-16LE"). run with cpinternal = UTF-8 results in: 11305698 in hex = AC82E2
ASC(chr(14844588), "UTF-16BE", "UTF-16BE"). run with cpinternal = UTF-8 results in: 11305698 in hex = AC82E2

I am sure there is some kind of an explanation.

• in the first case, external ASC will take that buffer and will assume it contains UTF-16LE - encoded text data (the source, second parameter). That would be two 16-bits characters: \u82E2 (苢)and \u00ac (¬). It should report back the code of first character using UTF-16LE (the destination, first parameter), 82E2. I cannot explain the #(00)AC prefix;
• in second case, the rationale is similar, just the order of the bytes reversed: \uE282 () and \uAC00 (가). Again, I would expect #82E2 to be printed, but not #(00)AC prefix.

I created the following piece of code to see the bytes of character encoding (requires -cpinternal UTF-8):

define variable euro as character.
euro = chr(14844588).

function toHex returns character (INPUT vInt AS int):
  RETURN upper(System.Convert:ToString(vInt, 16)).
END.

define variable ptr as raw.

put-string(ptr, 1) = euro.                                    // ---> E2 82 AC 0
// put-string(ptr, 1) = CHR(164, "UTF-16LE", "ISO8859-15").   // ---> AC 20 0 0
// put-string(ptr, 1) = CHR(164, "UTF-16BE", "ISO8859-15").   // ---> 20 AC 0 0
// put-string(ptr, 1) = chr(14844588, "ISO8859-15", "UTF-8"). // ---> A4 0 0 0

message toHex(get-byte(ptr, 1)) toHex(get-byte(ptr, 2)) toHex(get-byte(ptr, 3)) toHex(get-byte(ptr, 4)).

An explanation for the result you see may be an optimisation. Since the source and target parameters of the ASC are the same, the function returns as a no-op. To simulate this, instead of using the put-string, we can use put-long:

put-long(ptr, 1) = asc(euro).

Thank you for the explanation!

I've investigated myself to see if the problems comes from how bytes are stored in memory. ASC and CHR don't have anything to do with that.

put-long(ptr, 1) = 14844588. 
put-string(ptr2, 1) = "€". 
message toHex(get-byte(ptr, 1)) toHex(get-byte(ptr, 2)) toHex(get-byte(ptr, 3)) toHex(get-byte(ptr, 4)). // -> AC 82 E2 
message toHex(get-byte(ptr2, 1)) toHex(get-byte(ptr2, 2)) toHex(get-byte(ptr2, 3)) toHex(get-byte(ptr2, 4)). // -> E2 82 AC

Is it possible that these results might come from the fact that the Windows distribution is Little Endian?

I've committed partial modifications for ASC and CHR functionality to 4766a rev. 16068.
Please ignore the things related to annotations (we do not need them).

The summary is we need to find a way to automate the testing for chr and asc - we can go through single-byte chars (for now) and see how this works on the combinations for: source and target codepage, plus cpinternal (3 cases each, so 24 cases).

cpinternal looks like it affects assignment: ch = chr(.., target-cp, source-cp) will convert to 'cpinternal' on assignment to ch. This still needs to be discussed and see how can be solved, as 'chr' can appear as argument to i.e. RUN or a builtin function.

For now, lets see how we can leverage memptr to check the result of chr (Paul/Ovidiu - see if SET-BYTE-ORDER can alleviate the big/little endian problem).

For OUTPUT TO ... CONVERT - we can create 255 files with a single char, and after that use combinations of cpstream, cpinternal and target/source codepage. The assumption is we will use a EXPORT (maybe with memptr?). After that, we can compare the 4GL and FWD output.

There's something else which I can't find if it was mentioned in this task: in FWD (and Java) all Strings (literals or 'instances') are UTF. And for conversion, we (unless I remember wrong) always emit the .java with UTF encoding.

So, considering that all runtime is UTF, custom codepages (and cpinternal) comes into play only when we convert from one codepage to another or when we use numeric values. Because, for example, when you do an ASC(...), that resulting number should be in the cpinternal or the target codepage's context, and not UTF. The same for CHR, the number it receives is in the context of cpinternal/source codepage, but the result must be in the target codepage context.

To be clear about what was said here about include files and .p files with different encodings:
After more investigation, found that ASC and CHR aren't affected directly by this. I think this should be treated like a separate problem: how files are converted if they have different encodings in order to obtain the correct characters given as parameter for ASC .

Also, we do not need to know anymore the encoding of source file using annotations. If we have the first parameter of ASC (which can be a character or a string!)
converted correctly, it is ok. The further checks is made based on cpinternal, (maybe cpstream?), targetCodepage and sourceCodepage parameters.

A weird case I've found is this:

cpinternal = UTF-8, cpstream = UTF-8

MESSAGE "1. " CHR (ASC ("efc","UTF-8", "UTF-8"),"UTF-8", "UTF-8"). // will display nothing

DEF VAR res AS INTEGER NO-UNDO.
res = ASC ("efc", "UTF-8", "UTF-8").
MESSAGE "2. " CHR (res, "UTF-8", "UTF-8"). // will display "cfe" 

If the result of ASC is kept in a variable, CHR works, otherwise not. This happens for a parameter with length > 1 and < 4.

Constantin Asofiei wrote:

For now, lets see how we can leverage memptr to check the result of chr (Paul/Ovidiu - see if SET-BYTE-ORDER can alleviate the big/little endian problem).

I tried to convert the function that Ovidiu suggested in #4766-109 but I don't think that we have support for the System references because the conversion would fail. I thought about using this function:

FUNCTION toHex RETURNS CHARACTER (INPUT vInt AS INTEGER):
   IF vInt = 0 THEN RETURN "0".
   IF vInt = ? THEN RETURN "?".

   DEFINE VARIABLE hexDigits AS CHARACTER INITIAL "0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F".
   DEFINE VARIABLE hexResult AS CHARACTER.
   DEFINE VARIABLE r AS INTEGER NO-UNDO.
   DO WHILE vInt > 0:
      r = vInt MODULO 16.
      hexResult = ENTRY(r + 1, hexDigits) + hexResult.
      vInt = TRUNCATE(vInt / 16, 0).
   END.

   RETURN hexResult.
END.

It is a standalone function that replaces the dependency on System.Convert and no matter the system architecture it'll always produce the big-endian representation.

I've ran some minimal tests with Delia to try and get a clear direction regarding what was found until now.

We've ran this program with the code written in notepad++ with UTF-8 as the encoding:

         DEFINE VARIABLE data1 AS RAW NO-UNDO.
         DEFINE VARIABLE data2 AS RAW NO-UNDO.

         PUT-STRING(data1, 1) = "€".
         PUT-STRING(data2, 1) = CHR(14844588).

         MESSAGE "Decimal: " GET-BYTE(data1, 1) "|" GET-BYTE(data1, 2) "|" GET-BYTE(data1, 3) "|" GET-BYTE(data1, 4) "|" GET-BYTE(data1, 5) "|" GET-BYTE(data1, 6) "|" GET-BYTE(data1, 7) "|" GET-BYTE(data1, 8). 
         MESSAGE "Hex: " toHex(GET-BYTE(data1, 1)) "|" toHex(GET-BYTE(data1, 2)) "|" toHex(GET-BYTE(data1, 3)) "|" toHex(GET-BYTE(data1, 4)) "|" toHex(GET-BYTE(data1, 5)) "|" toHex(GET-BYTE(data1, 6)) "|" toHex(GET-BYTE(data1, 7)) "|" toHex(GET-BYTE(data1, 8)).
         MESSAGE "------------------------------------------------------------".
         MESSAGE "Decimal: " GET-BYTE(data2, 1) "|" GET-BYTE(data2, 2) "|" GET-BYTE(data2, 3) "|" GET-BYTE(data2, 4) "|" GET-BYTE(data2, 5) "|" GET-BYTE(data2, 6) "|" GET-BYTE(data2, 7) "|" GET-BYTE(data2, 8). 
         MESSAGE "Hex: " toHex(GET-BYTE(data2, 1)) "|" toHex(GET-BYTE(data2, 2)) "|" toHex(GET-BYTE(data2, 3)) "|" toHex(GET-BYTE(data2, 4)) "|" toHex(GET-BYTE(data2, 5)) "|" toHex(GET-BYTE(data2, 6)) "|" toHex(GET-BYTE(data2, 7)) "|" toHex(GET-BYTE(data2, 8)).

Decimal: 195 | 162 | 194 | 130 | 194 | 172 | 0 | 0
Hex:     C3  | A2  | C2  | 82  | C2  | AC  | 0 | 0
------------------------------------------------------------
Decimal: 226 | 130 | 172 |  0  |  ?  |  ?  | ? | ?
Hex:     E2  | 82  | AC  |  0  |  0  |  0  | 0 | 0

Then we also set -cpstream to UTF-8 which returned the following result:

Decimal: 226 | 130 | 172 |  0  |  ?  |  ?  | ? | ?
Hex:     E2  | 82  | AC  |  0  |  0  |  0  | 0 | 0
------------------------------------------------------------
Decimal: 226 | 130 | 172 |  0  |  ?  |  ?  | ? | ?
Hex:     E2  | 82  | AC  |  0  |  0  |  0  | 0 | 0

Then we set the -cpinternal and -cpstream to ISO8859-1:

Decimal: 226 | 130 | 172 |  0  |  ?  |  ?  |  ?  |  ?  
Hex:     E2  | 82  | AC  |  0  |  0  |  0  |  0  |  0
------------------------------------------------------------
Decimal:  0  |  ?  |  ?  |  ?  |  ?  |  ?  |  ?  |  ?
Hex:      0  |  0  |  0  |  0  |  0  |  0  |  0  |  0

Like Delia found out, this tests show:
1. That the CHR function may not be dependable on cpstream
2. That the encoding of the file matters but ASC and CHR aren't affected directly by this while literals are
3. Literals are likely transformed (or interpreted) in -cpstream encoding which affects the internal representation because...
4. The type of the encoding (single or multiple byte) also matters.

3 and 4 are the most probable causes that Delia found for which those weird text are printed on the screen.

This is besides the purposes of this task but every client should be made aware that, if their code is dependent upon such a case where they had a programmer that was using a certain encoding (different than UTF-8) while writing the code, with a different -cpstream and -cpinternal we will not be able to provide 100% the same functionality. For conversion, we are obligated to assume that the encoding of the file and -cpstream are the same.

Please compile the file to a .r and run that via "pro -p", not from the procedure editor. It may matter what cpinternal/cpstream you have in the procedure editor, when the compile statement is ran.

Constantin Asofiei wrote:

Please compile the file to a .r and run that via "pro -p", not from the procedure editor. It may matter what cpinternal/cpstream you have in the procedure editor, when the compile statement is ran.

I ran the procedure via the commandline: prowin -p play.p -cpinternal CP1 -cpstream CP2

Delia, please work on automating using an approach like this:

message session:cpinternal.

def var m as memptr.
def var i as int.
def var lc as longchar.

output to b.txt.
do i = 0 to 255:
  set-size(m) = 2.
  put-string(m, 1) = chr(i, "IBM850", "iso8859-1") no-error.
  message i error-status:error error-status:get-message(1).

  message string(base64-encode(m)).
end.
output close.

• script to change the cpinternal to the 3 ones we are interested in (UTF-8, ISO8859-1, IBM850) - windows and linux, as we need to run it in both
• for asc/chr the source/target codepage - combinations of the 3
• a way to save the output from 4GL and compare it automatically in FWD

Instead of base64-encode, you can use the 'to-hex' approach.

During making tests for CHR, I've found another weird case.
The tests compare the byte representation of the result given by FWD CHR function with the one given by 4GL CHR function.

For cpinternal = cpstream = UTF-8 and

def var m as memptr no-undo.
set-size(m) = 4.
put-string(m,1) = chr(255, "ibm850", "ibm850").
message chr(255, "ibm850", "ibm850").
message length(chr(255, "ibm850", "ibm850")).
message toHex(get-byte(m, 1)) toHex(get-byte(m, 2)) toHex(get-byte(m, 3)) toHex(get-byte(m, 4)).

The results is:
Progress // or a weird string
1
FF 0 0 0

• If I try to save the result in a file, the resulted character is ÿ - character placed on FF code in ISO8859-1.
• For some decimal code points like 128, 129, 130 ...., no character is displayed, the length is 0, but the byte representation is correct, fact that sustains that there are some default values (probably from ISO8859-1) for 128-255 codes "verified" in UTF-8

A problem I've noticed while CHR testing based on byte representation:

For put-string(m,1) = chr(255, "ibm850", "ibm850"). cpinternal = cpstream = UTF-8, toHex(get-byte(m, 1)) toHex(get-byte(m, 2)) should be FF 0 - ÿ character, but in FWD all characters/strings are stored as UTF-8 and the byte representation in FWD for this case is C3 BF which will not match the 4GL result.

In 4GL, if cpinternal = cpstream = "ibm850", what do you get? I assume still FF 0.

You are correct that FWD stores the char as UTF-8, and maybe the problem is at put-string, to convert the bytes to cpinternal and store it as that in the memptr.

Constantin Asofiei wrote:

In 4GL, if cpinternal = cpstream = "ibm850", what do you get? I assume still FF 0.

Yes, same result.

Greg, please see the test in #4766-123 - this contradicts what I suggested to use cpinternal for PUT/GET-STRING. The bytes generated by put-string really need to be in the codepage of what chr used.

I think we need to have a codepage at the character instance. If null, then that is cpinternal; otherwise, we know that the UTF-8 character.value String needs to be interpreted in a different codepage.

Now, when we convert i.e. ibm850 to UTF-8, not all chars may have equivalents (or correct equivalents), if I understand correctly. This is another problem.

Greg, please see the test in #4766-123 - this contradicts what I suggested to use cpinternal for PUT/GET-STRING.

The thing I don't understand about #4766-123 is this output:

Progress // or a weird string

Which I guess is from message chr(255, "ibm850", "ibm850"). but it should not result in the text "Progress".

The bytes generated by put-string really need to be in the codepage of what chr used.

OK, I can accept that.

I think we need to have a codepage at the character instance. If null, then that is cpinternal; otherwise, we know that the UTF-8 character.value String needs to be interpreted in a different codepage.

OK, this makes sense. It will potentially impact every location where we read the value and assume it is cpinternal. This means that we really need to put this logic into common code and then call it from any code that honors codepage conversion.

Now, when we convert i.e. ibm850 to UTF-8, not all chars may have equivalents (or correct equivalents), if I understand correctly. This is another problem.

UTF-8 should have all possible characters from ibm850. Isn't the issue the other way around (UTF-8 to ibm850)? And in that case don't we just need to raise the same errors as OE would do?

I'm probably not following your point.

After I ran some examples in ChUI (and GUI) mode, these are the conclusions:

• CHR isn't affected directly by cpinternal, but some statements which can be applied on CHR's result can be affected by cpinternal
• if we store CHR's result in variable ch1, ch1 stores a character encoded in target codepage parameter; so cpinternal doesn't affect the assignment
• PUT-STRING (m, 1) = CHR (...) is not affected by cpinternal, but m "stores" the character relative to target codepage CHR's parameter
• OUTPUT TO .. [CONVERT ...] is not affected by cpinternal or cpstream because it seems to simply write the bytes resulted from CHR. The encoding of "output file" is important.
• DISPLAY and MESSAGE are affected by cpinternal just in GUI mode

I've found a weird example (ChUI mode):
cpinternal = cpstream = UTF-8

DEFINE VAR m AS RAW NO-UNDO.

PUT-STRING(m, 1) = CHR(6645347, "UTF-8", "UTF-8").
MESSAGE CHR(6645347, "UTF-8", "UTF-8").
MESSAGE toHex(GET-BYTE(m,1)) toHex(GET-BYTE(m,2)) toHex(GET-BYTE(m,3)) toHex(GET-BYTE(m,4)).

Result:
cfe
63 ? ? ?

6645347 is not corresponding for a lead-byte value in UTF-8.

I don't know how to interpret this.
I expect to receive these bytes: 63 66 65 ? (which represents cfe).

In Progress documentation for CHR I've found:
If the value of expression is in the range of 1 to 255, CHR returns a single character. This character might not be printable or might not display on certain terminals. For a value greater than 255, the CHR function checks whether the value represents a valid character within the source code page. For a double-byte or Unicode source code page, it checks for a corresponding lead-byte value. If the integer value corresponds to a valid lead-byte and the value represents a valid character in the source code page, CHR returns the character. Otherwise, the character returns a null string.

Probably I shouldn't treat this case even if the result from MESSAGE statement seems to be (somehow) correct, right?

Another weird case:
cpinternal = cpstream = IBM850

PUT-STRING(m, 1) = CHR(159, "ISO8859-1", "IBM850").
MESSAGE toHex(GET-BYTE(m,1)) toHex(GET-BYTE(m,2)) toHex(GET-BYTE(m,3)) toHex(GET-BYTE(m,4)).

but ISO8859-1 encodes "" (a non-printable char) on 9F byte.

In FWD we need to know the character (and the encoding of it) returned by CHR to find the right byte representation. Because it is not sure what character is returned by CHR (159, "ISO8859-1", "IBM850") in 4GL (this is dependent of terminal encoding and so on), I implemented these cases using a "fallback" : if we reach this point, return the character encoded in source codepage on given code.

We could return the ISO8859-1 result: "", but this will lead to another byte representation: 3F which means ? character (unknown) and this is not the expected result.

Delia Mitric wrote:

I've found a weird example (ChUI mode):
cpinternal = cpstream = UTF-8
[...]

Result:
cfe
63 ? ? ?

6645347 is not corresponding for a lead-byte value in UTF-8.

I don't know how to interpret this.

I do not think it's strange. The 6645347 integer is in internal hex representation, as you noted 0x63 0x66 0x65 0x00. These do not form a single UTF encoding. Instead there are 3 separate characters, those3 you already mentioned (and a string terminator): cfe\0. I think it's logical that chr() will process only the first full character it sees in the buffer, ignoring the rest.

Ovidiu Maxiniuc wrote:

I do not think it's strange. The 6645347 integer is in internal hex representation, as you noted 0x63 0x66 0x65 0x00. These do not form a single UTF encoding. Instead there are 3 separate characters, those3 you already mentioned (and a string terminator): cfe\0. I think it's logical that chr() will process only the first full character it sees in the buffer, ignoring the rest.

Ok, but MESSAGE CHR(6645347, "UTF-8", "UTF-8"). is cfe. Why CHR does not process just the first character here ?

message is behaving strangely sometime. But, for a greater confusion try this:

define variable v as character.
v = CHR(6645347, "UTF-8", "UTF-8").
message 1 v.
message 2 CHR(6645347).
message 3 CHR(6645347, "UTF-8", "UTF-8").

I think we need to support the normal/valid values. Passing an edge-case parameter to CHR should be handled after that. This may be non-deterministic, or garbage, as you observed in #4766-131.

Something similar to what is presented in #4766-131, but I don't know how should I handle this:

cpinternal = cpstream = ISO8859-1

PUT-STRING(m,1) = chr(128, "iso8859-1", "iso8859-1").
MESSAGE LENGTH(chr(128, "iso8859-1", "iso8859-1")).
MESSAGE toHex(GET-BYTE(m,1)) toHex(GET-BYTE(m,2)) toHex(GET-BYTE(m,3)) toHex(GET-BYTE(m,4)).

result:

1
80 0 ? ?

Ovidiu Maxiniuc wrote:

message is behaving strangely sometime. But, for a greater confusion try this:
[...]
If 6645347 is replaced by 14844588 (utf-8 encoding of euro sign) the result is expected, though.

I think we need to support the normal/valid values. Passing an edge-case parameter to CHR should be handled after that. This may be non-deterministic, or garbage, as you observed in #4766-131.

I've tested the same example but for OUTPUT TO ... EXPORT:

PUT-STRING(m, 1) =  CHR(6645347, "UTF-8", "UTF-8").
EXPORT toHex(get-byte(m, 1)) toHex(get-byte(m, 2)) toHex(get-byte(m, 3)).

EXPORT  CHR(6645347, "UTF-8", "UTF-8").

and in the file are written these bytes: 0x63 0x66 0x65 0x00 but the bytes taken from memptr are: 0x63 0x00 0x00 0x00.
So, is this a non-deterministic behavior of OUTPUT TO... and MESSAGE or CHR really returns a string, but PUT-STRING/GET-BYTE treats its result different?

I remember we have discussed this in a previous meet. The main problem is that, in 4GL, the CHR() function will perform its computation and store the result in the destination buffer (variable, buffer-field or intermediary value) using the encoding it was instructed (the optional target-cp parameter). This will override the internal CP setting. At the moment the value is read, that encoding is no longer available. Accessing the value will use the cpinternal to extract data as a character. However, if the application logic is aware of that encoding, it can use CODEPAGE-CONVERT to make additional changes, practically returning the buffer content using the right encoding.

In FWD we use the native String instead of C/C++ style memory allocation. the result of chr() function is no longer storing the result into the destination buffer, but return a String, always in Java's internal encoding. To be more confusing, before JDK9 it used an array of char (UTF16 - making it faster), the newer JDKs MAY use byte array using UTF-8 (more compact for common ASCII / Latin1 characters). This will cause us to see different results between Java8 and Java17. However, this is not observed in normal usage, with a fixed CP.

An idea I had is to enhance the character data with additional information about where the content came from. An hybrid approach. This would allow FWD to access information like the source of its value, more precisely, if it is the result of character.chr() and TextOps.codePageConvert() function. Note that the latter already stores the target CP if the processed/returned value is a longchar. In that case, an additional byte buffer may be added to store the same value as the 4GL would do. Of course, this need more refinement about how this new internal data will be further used.

An idea I had is to enhance the character data with additional information about where the content came from.

I think this is the same idea that Constantin was suggesting in #4766-127. Great minds think alike!

We should move ahead with this idea.

CHR isn't affected directly by cpinternal,

Delia, in #4766-129 you posted this. I think it is generally correct, except for the cases where CHR() does not explicitly define the source or target codepages. In other words, the 1 parameter version chr(<number>) will automatically use cpinternal as its source and target codepages. The 2 parameter version chr(<number>, <target_cp>) means that the source cp is cpinternal.

Greg Shah wrote:

An idea I had is to enhance the character data with additional information about where the content came from.

I think this is the same idea that Constantin was suggesting in #4766-127. Great minds think alike!

We should move ahead with this idea.

In some cases, to know just where the content comes from isn't enough. One of these cases is #4766-135 when CHR will return "" but in 4GL the buffer contains: 0x80 0x00 0x00 0x00. This affects some results if we base just on the character's value and the codepage where the value comes from.
I also think we should store an additional byte buffer to cover these cases.

CHR isn't affected directly by cpinternal,

Delia, in #4766-129 you posted this. I think it is generally correct, except for the cases where CHR() does not explicitly define the source or target codepages. In other words, the 1 parameter version chr(<number>) will automatically use cpinternal as its source and target codepages. The 2 parameter version chr(<number>, <target_cp>) means that the source cp is cpinternal.

Yes, that's right. When source or target codepages aren't specified they're replaced by cpinternal value.

Delia Mitric wrote:

In some cases, to know just where the content comes from isn't enough. One of these cases is #4766-135 when CHR will return "" (a non-printable character) but in 4GL the buffer contains: 0x80 0x00 0x00 0x00. This affects some results if we base just on the character's value and the codepage where the value comes from.
I also think we should store an additional byte buffer to cover these cases.

Another way to treat it is to return String.valueOf((char) 128) (from CHR). This will put the UNICODE value in the character's value which is a non-printable character and we can have access to the byte from this string to use it further in PUT-STRING, for example.

Delia Mitric wrote:

Another way to treat it is to return String.valueOf((char) 128) (from CHR). This will put the UNICODE value in the character's value which is a non-printable character and we can have access to the byte from this string to use it further in PUT-STRING, for example.

We analyzed a bit what would take to make this possible and there are some other things that need to be taken care of which I think can increase the complexity of the code.

One such thing is the fact that in java, bytes are represented as a signed integer with the range of [-128, 127] (inclusive) while Progress encodes bytes in unsigned integers with the range of [0, 255] (inclusive). For encoding and decoding, we would need to have an offset of 128 to make sure the representation is accurate.

I understand your concern but I do not think the sign is relevant here. Think of the hex (or binary) representation: -128 is 0x80 and 127 if 0xFF (or binary). There are the same 8 bits and the character encoders in Java are well aware of the signed data type.

Returning to chr(128), assuming UTF8 internal CP (aka: chr(128, 'UTF-8', 'UTF-8')): I do not think it need to go deeper with it. CHR expects the first parameter to be a (valid) encoding of a character and 128 is NOT. (In UTF8, bytes starting with 10 are trailing bytes after a leading byte, they cannot exist by themselves). If internal CP is 8859-1, there is no character defined with this encoding.

String.valueOf((char) 128) is something else: the parameter of valueOf() is the Unicode order (code point), which is identical with UTF-16 encoding (at least for the first 60K chars). When executed from Java17, we have Character.getName(128) = "PADDING CHARACTER", but its encoding in UTF-8 is C280 as returned by String.encodeUTF8(LATIN1, new byte[]{(byte)128}, true).

In conclusion, using 0x80 as 4GL does, will break the meaning of the character, turning it from a non-standardised value to a well-know (read: documented) value is not desired. If the application gives a meaning to chr(128) in 8895-1 then it's its own responsibility to interpret it when it is read (from memory or database). But in the context of using UTF-8 the read value in FWD is not the same it was written.

I've added the codePage information in Test class and initialized it when a character instance is made based on the result of CHR function. (+ modified the assign method for character )
Also, I've modified the BinaryData.setString methods (PUT-STRING) in order to manage the data in the specified codePage if the data is character and its codePage is not NULL.

Now, the tests for CHR for 1-255 decimal code points and the three code pages : UTF-8, ISO8859-1, IBM850 pass.

About OUTPUT TO ... statement in combination with one of the MESSAGE/EXPORT or DISPLAY statements applied on CHR's results (or a character variable), these are the conclusions:
(Consider cpinternal as cpi, cpstream as cps, character (or the CHR's result) as c, c character's encoding/codePage as enc and the hexadecimal representation of c in enc as hex)

• MESSAGE, EXPORT and DISPLAY leads to the same results
• it doesn't matter if cpi != enc or cps != enc
• like I've said above, cpi doesn't affect CHR's results, but it affects MESSAGE, EXPORT and DISPLAY statements, that's why the result from output file can differ if cpi != cps; hex is "stored" relative to cpi and it can be a valid character in cpi or not.
• if cpi = cps, then the written internal representation in the output files are exactly the one that represents c in enc
• if cpi != cps, then the hex relative to cpi (a valid character c2 or not) is written in output file relative to cps. For example:

OUTPUT TO results.txt.
DISPLAY CHR(164, "ISO8859-1", "ISO8859-1").
OUTPUT CLOSE.

c = ¤
cpi = IBM850
cps = UTF-8
hex = 0xA4
0xA4 in cpi is ñ

The output file contains:

C3 B1 - the representation of ñ in cps which is UTF-8

The defined streams / OUTPUT STREAM TO ... seem to work the same like I've described in #4766-144.

I'm trying to make some (automatized) tests which compares the FWD results (from OUTPUT [STREAM] TO ...) with the ones given by 4GL.

I started to do these tests for OUTPUT TO ... MESSAGE CHR(...), but I've reached a problem:

When trying to "generate" and store the 4GL's results, I've noticed that there are some combinations that lead to the crash of the program, for example:

OUTPUT TO file.txt.
MESSAGE CHR(164, "ISO8859-1", "ISO8859-1"). with cpinternal = UTF-8 and cpstream = ISO8859-1.

When I run the procedure which contains this example from command line using pro, it crashes. When I run it using prowin it don't crash, but in the output file are written strange things (15245 characters; probably there is a nondeterministic behavior).

To be mentioned this happens just when "messaging" to a file because here is implied cpstream.

I'm trying to make some (automatized) tests which compares the FWD results (from OUTPUT [STREAM] TO ...) with the ones given by 4GL.

Just related to this; you can try to make tests either using unit testing framework or harness. If you need command line specific options (like it seems), then I would incline to use harness as testing framework: harness
If Greg and agrees, then Florin [fer] can assist with setting it up.

In OE, one must provide the proper command line arguments to set the CP* values. Since we use the PCT ("Progress Compile Tools") ant scripting to execute ABLUnit tests in OE, that set of scripts would have to be written to handle this need. That will likely be the case whether we use the harness or not, since I recall that we use these ant scripts even from the harness.

In FWD, we have a similar capability but this is done via the bootstrap configuration. We would have to have customized bootstrap cfg files for specific test sets. This would have to be done in a manner that our ABLUnit client can handle. See Running the ABLUnit Client for details.

Let's start without the harness in this case. Try to solve the problem with proper configuration.

• Windows: 0D 0A
• Linux: 0A.

An idea is to split the binary content based on the binary representation of the endline. To achieve this, I need to know how is represented endline character in this current file (which can come from Windows/Linux/something else). I was thinking of writing the endline character at the start of the file followed by an easy to read delimiter/character.

About the modifications made on Text class (for character to store the information about the codePage in which it is encoded):
cpinternal = cpstream = ISO8859-1

The encoding of the file is ISO8859-1 - this is important for the literal

DEFINE VARIABLE ch1 AS CHARACTER NO-UNDO.
DEFINE VARIABLE m   AS MEMPTR    NO-UNDO.
SET-SIZE(m) = 8.

ch1 = "¤" + CHR(164, "UTF-8", "ISO8859-1").
PUT-STRING(m, 1) = ch1.

MESSAGE toHex(GET-BYTE(m, 1)) toHex(GET-BYTE(m, 2)) toHex(GET-BYTE(m, 3)) toHex(GET-BYTE(m, 4)) toHex(GET-BYTE(m, 5)) toHex(GET-BYTE(m, 6)).

Results in:
A4 C2 A4 00 00 00

A4 - from the literal - encoded in cpinternal relative to file's encoding
C2A4 - from CHR result encoded in target-codePage

In this case, when we build a character variable through concatenation, which will be (in FWD) the codePage's value ?
If we set the attribute codePage to "UTF-8", PUT-STRING will "write" these bytes: C2 A4 C2 A4 which is incorrect.

(I found this case when generated some results files for OUTPUT TO ...)

Found a problem in FWD during generating results for OUTPUT STREAM TO ... .
This is the testcase:

DEFINE STREAM sout.
OUTPUT STREAM sout TO test.txt.

DEFINE VARIABLE i AS INTEGER NO-UNDO.
DEFINE VARIABLE line AS CHARACTER NO-UNDO.
DO i = 1 TO 10:
    line = STRING(i).
    DISPLAY STREAM  sout line WITH FRAME f NO-BOX NO-LABELS STREAM-IO.
    DOWN WITH FRAME f.
END.

OUTPUT STREAM sout CLOSE.

• in Progress test.txt file contains: ShowHide

  1
  2
  3
  4
  5
  6
  7
  8
  9
  10

• In FWD test.txt file contains: ShowHide

  10

Should I open a new task for it?

Delia Mitric wrote:

Found a problem in FWD during generating results for OUTPUT STREAM TO ... .
This is the testcase:
[...]

• in Progress test.txt file contains: {{collapse
  1
  2
  3
  4
  5
  6
  7
  8
  9
  10
  }}

• In FWD test.txt file contains: {{collapse
  10

}}

Should I open a new task for it?

I think, this was also found here: #10131-22.

Delia Mitric wrote:

Found a problem in FWD during generating results for OUTPUT STREAM TO ... .
This is the testcase:
[...]

Should I open a new task for it?

Created a new task for this in Base Language: #10527

This is an example that highlights that PUT and DISPLAY lead to different results (for some cases) when writing in a file using OUTPUT TO:

cpinternal = UTF-8
cpstream = ISO8859-1

OUTPUT TO TEST.TXT.

DISPLAY CHR(194, "ISO8859-1", "ISO8859-1")  with FRAME f NO-LABELS NO-BOX STREAM-IO.
PUT UNFORMATTED SKIP.

PUT UNFORMATTED CHR(194, "ISO8859-1", "ISO8859-1").

OUTPUT CLOSE.

The bytes from the output file:
80 0d 0a

This behavior occurs for 128-255 decimal code points.
Display writes bytes for CHR (194, "ISO8859-1", "ISO8859-1") and for CHR (195, "ISO8859-1", "ISO8859-1"). In these cases, PUT writes a null string (0 bytes, not even 3F which means unknown).
For example, for CHR (199, "ISO8859-1", "ISO8859-1") DISPLAY writes 3F (which can be consider equivalent to a null string/ unknown).

No idea where the valid bytes written by DISPLAY come from.

For item 2 in #4766-16, in the Progress column of your table, when it says "~u2591", what does that mean? Are you saying that a quoted string of 8 characters is written the stream? What is the hex encoded output in the file?

Greg Shah wrote:

For item 2 in #4766-16, in the Progress column of your table, when it says "~u2591", what does that mean? Are you saying that a quoted string of 8 characters is written the stream? What is the hex encoded output in the file?

Sorry, I wrote the string value, not the hex output for this item. The hex encoded output is 7e 75 32 35 39 31 .

Hmmm. That may explain things.

I wonder if this really has nothing to do with the PUT or DISPLAY. The problem here is that the 4GL preprocessor normally converts escape sequences like "~u2591" into the associated character in the character set used for encoding the source code. If that encoding is 8859-1 and there is no equivalent for that character, the preprocessor may just emit the 8859-1 text for ~, u, 2, 5, 9, 1. In other words, if it doesn't know what to do with a character it may just NOT CONVERT AT ALL.

def var txt as char.

txt = "~u2591".

if substring(txt, 1, 1, "character") eq "~~" AND
   substring(txt, 2, 1, "character") eq "u"  AND
   substring(txt, 3, 1, "character") eq "2"  AND
   substring(txt, 4, 1, "character") eq "5"  AND
   substring(txt, 5, 1, "character") eq "9"  AND
   substring(txt, 6, 1, "character") eq "1" then

   do:
      message "The preprocessor is confused and bailed.".
   end.
else
   do:
      message "The escape sequence was processed as character '" + txt + "'".
   end.

Please run this with different encodings (on FWD and OE) and report the results.

Greg Shah wrote:

Please run this with different encodings (on FWD and OE) and report the results.

I've modified the code in order to obtain the hex representation of the character.
These are the results:

To be mentioned that 3F represents the ? character. So FWD seems to "know" that the sequence can't be interpreted relative to cpinternal.

OK, so this behavior has nothing to do with the stream output. It is a quirk of the preprocessor. Please create a new task in the Conversion Tools project for this bug.

Greg Shah wrote:

OK, so this behavior has nothing to do with the stream output. It is a quirk of the preprocessor. Please create a new task in the Conversion Tools project for this bug.

Created a new task in Conversion Tools project: #10678.

Before adding the the logic for "conversion" from cpinternal to cpstream for writing in files, there are some things to be fixed:

• in Progress we can build a character variable (let's name it str1) by concatenating a literal with a result of CHR and so on. The hex representation of str1 is the hex representation of each concatenated "part": hex representation of the literal (relative to cpinternal) + hex representation of the CHR result (relative to the targetCP) and so on.
  How should we store this information in FWD? We discussed about the codepage information for a character type variable, but just in case it comes from CHR function. What if str1 is built from many CHR results with different targetCodepages? Which will be the "general" codepage?

• when "converting" characters from cpinternal to cpstream in order to write the correct bytes in the output file, we may have a string/many lines to write. Each character from the string should be converted and this can be pretty expensive. How should we treat this?
  A good thing to know: I've noticed that the conversion from cpinternal to cpstream is quite similar to how CHR works: it uses convmaps if the character from sourceCP is not encoded in targetCP.

What if str1 is built from many CHR results with different targetCodepages?

I see your point that a single flag might not work. Please show a 4GL example here, especially how the resulting character value could contain content encoded in more than two codepages.

when "converting" characters from cpinternal to cpstream in order to write the correct bytes in the output file, we may have a string/many lines to write. Each character from the string should be converted and this can be pretty expensive. How should we treat this?

Don't we already handle this conversion in the Stream and subclasses?

Greg Shah wrote:

What if str1 is built from many CHR results with different targetCodepages?

I see your point that a single flag might not work. Please show a 4GL example here, especially how the resulting character value could contain content encoded in more than two codepages.

Sure, here is an example:

DEF  VAR str1 AS CHARACTER NO-UNDO.
DEF VAR m AS RAW.
str1 = CHR(164, "UTF-8", "ISO8859-1") + CHR(176, "IBM850", "ISO8859-1") + CHR(144, "ISO8859-1", "IBM850").

PUT-STRING(m, 1) = str1.

MESSAGE toHex(GET-BYTE(m, 1)) toHex(GET-BYTE(m, 2)) toHex(GET-BYTE(m, 3)) toHex(GET-BYTE(m, 4)) toHex(GET-BYTE(m, 5)).

The Progress result: C2 A4 F8 C9 00

• 164 is A4 in hex
• 176 is B0 in hex
• 144 is 90 in hex

• On A4 in ISO8859-1 is encoded ¤ character
• ¤ in UTF-8 is encoded on C2 A4

• On B0 in ISO8859-1 is encoded ° character
• ° character in IBM850 is encoded on F8

• On 90 in IBM850 is encoded É character
• É character in ISO8859-1 is encoded on C9

Greg Shah wrote:

Don't we already handle this conversion in the Stream and subclasses?

As I've noticed, there isn't a real "conversion" between cpinternal and cpstream, but in most cases it works, I mean the cases when a char from cpinternal is encoded in cpstream because here the character is not "transformed" in another char. In this cases, things work normally. The problem arises when a character encoded in cpinternal is not encoded in cpstream codepage, so we need convmaps.

Also, I'm not sure that the conversion using OUTPUT TO .... CONVERT SOURCE ... TARGET ... works as expected, but that's another story.

Delia Mitric wrote:

Sure, here is an example:
[...]

This is very interesting. It shows how OE will use the memory storage of str1 as a destination buffer for the result of intermediary evaluations without any postprocessing (code page conversions). As we see in the binary representation, the result is a set of characters using mixed CPs. Unless the application has additional information on how to decode it (which CP to use for individual bytes), the result is practically unreadable.

The only solution to this problem is that we do the very same thing as OE, meaning to drop Java native support for String and implement a byte-array replacement. But I do not think we should go on this slope: these are more like very particular cases, examples of chaotic programming which break the common sense and basic paradigm learned in computer science. It looks like an obfuscation solution with non benign intentions.

That is a fair point, Ovidiu. I've been thinking about a similar solution. As you note, we would not want to change our entire character processing to use byte[] just so that we can handle this CHR edge case.

We could implement a character subclass (EncodedString?) which is only created when the edge case(s) are hit. So, normally the entire system continues working with character, including the value returned from CHR. But when the CHR edge case is hit, we could return EncodedString instead. Then we would need to implement changes in other parts of the code (e.g. String concatenation, operators, certain assign() calls) to handle things differently. It would give us the "memory" of the problem and the tools to implement behavior since the problem can be seen and managed.

Before we do this, we need to understand the value of it. The key question: would real application code ever depend on this behavior?

Here is the current status of this task:¶

• Delia's work provided a clear picture in terms of what we need to make the CHR function work.
• When convmaps are needed, we choose to load them dynamically from the jar (and maybe cache it) to make the process more optimized
• There were still some edge-cases needed to be addressed especially with the UTF-8 and ISO8859-1 encoding
• In this task it was also attempted to solve the different behavior of the MESSAGE, DISPLAY, PUT and EXPORT statements but there is a lot more work to do on that

Here are the main concerns we come across:¶

• Every character in the system needs to be represented through a byte buffer
  • Solution: Every character can be converted from UTF-16 to the target encoding if the situation requires it, provided we have the correct cpinternal and cpstream values

• Buffer holding bytes from multiple encodings (str = CHR(159, "ISO8859-1", "IBM850") + CHR(159, "IBM850", "UTF-8") + CHR(159, "IBM850", "ISO8859-15))
  • Solution: We'll defer the implementation for this problem as it heavily complicates things but the solution would be to keep the current implementation and convert to a new class (e.g. EncodedString) when such an edge-case is found

• Use of different cpstream and cpinternal values
  • This don't directly affect the CHR and ASC functions (except source and target codepages are omitted) but have an impact on the output statements.

• Compiling source code into multiple encodings (i.e different cpstream values for different source code files)
  • This is possible to accomplish with Progress but highly unlikely that someone will ever do it. The solution would be to place an annotation at the class level or in the pphints files corresponding to the 4gl program. Will also be deferred for the purposes of this task.

• Wrong cpstream value
  • If there will ever be a case where a certain Progress program was written in a IDE with an encoding different to the cpstream value there would be impossible to consistently return the correct results (without guessing the actual encoding)

• Size of the convmaps
  • Solution: Store only the differences and load them as a resource from the jar at runtime.

Here is what I was able to do until now:¶

• Comprehensive Analysis of the Progress OpenEdge Conversion Map
• ISO 8859-1 Explained

I have to point out that these documents were generated using the "smartest" LLM from Google (Gemini Pro) with the Deep Research option. I've analyzed every information, taken out all the hallucinations and verified all information that I was able to but do take them with a grain of salt.

Those documents provided a good starting point. I've also added all other relevant information I could find.

I've found that there is a library called International Components for Unicode (ICU), that is embedded in Progress, used to perform operations with all the supported Unicode encodings. In Progress, those are: UTF-8, UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32BE, UTF-32LE, UCS-2 and maybe also GB18030.

I thought that, because of the high complexity and performance issues related to using convmaps for multi-byte encodings (such as UTF-8), the operation must be handed to this ICU library to handle them as fast as possible while still return the correct results. This is partially correct.

I've concluded that this library is used only when looking up attributes (e.g. is alphanumeric, lead-byte, upper-case, lower-case, etc.) of the respective characters and not for conversion.

This brings us back to the convmaps. While Delia's rule covers maybe up to 90% of the cases, I think that the issue lies on how we interpret the convmaps themselves. The most complete documentation I was able to find is here: Progress Internationalization Guide
In this document, the main concepts involved in this task are listed. I even found the documentation for the syntax for the convmap tables which Marian brought to the discussion a while back. This will be useful if, for some reason ever, we decide to implement support for custom conversion maps.

Now, what it is never mentioned in the documentation is what actually are the freaking numbers in that table representing. We must not rule out the possibility that the meaning could be different across multiple code pages (single-byte or multi-byte).

There is one table in particular that has gotten my attention and is the conversion table from ISO8859-1 to UTF-8. This table defines a one to one match between the two for the first 255 characters. The problem arises when trying to convert characters above or equal to 128 which in UTF-8 is a invalid position. In this case, progress knows to return the correct bytes of the PAD control character in UTF-8 which are C280 even if the table states the 128 value for the character on the position 128.

The only way that this would make sense is if those values are actually the Unicode code points that point to the correct character. This would be consistent with the other tables for UTF-8. For example the conversion between 1252 and UTF-8, the table states the value 0x20AC on position 128 which in the 1252 code page represents the euro sign. In this case, Progress also returns the correct bytes in UTF-8. The Unicode code point for the euro sign is U+20AC.

So this points to the fact that at least for the Unicode code pages Progress is using the universal code point behind the scenes to do the proper conversion.

Very nice analysis!

• Buffer holding bytes from multiple encodings (str = CHR(159, "ISO8859-1", "IBM850") + CHR(159, "IBM850", "UTF-8") + CHR(159, "IBM850", "ISO8859-15))
  • Solution: We'll defer the implementation for this problem as it heavily complicates things but the solution would be to keep the current implementation and convert to a new class (e.g. EncodedString) when such an edge-case is found

Please create a task for this deferred work and link it here as related.

• Compiling source code into multiple encodings (i.e different cpstream values for different source code files)
  • This is possible to accomplish with Progress but highly unlikely that someone will ever do it. The solution would be to place an annotation at the class level or in the pphints files corresponding to the 4gl program. Will also be deferred for the purposes of this task.

Please create a task for this deferred work and link it here as related.

Created branch 4766b from trunk rev. 16366.

Starting fresh as we found new cases which need to be covered and previous changes not guaranteed to be fully correct. I'll keep Delia's work intact and use it as a reference for what needs to be done.

Rebased branch 4766b with trunk rev. 16407 and committed rev. 16408 with the initial implementation for the CHR function.

This commit also includes changes from #11149.

I found a differences in FWD related to byte representation of character values. The exclusion of byte 0 will affect the CHR testcases since we're working with bytes.

Review of 4766b / r16409.

This is really deeper set of changes that I expected. I see the codePage was pushed up from longchar to parent class Text (as protected), so it is now available to character as well. However, this is not accessible from 4GL code. Therefore, the attribute is explicit in longchar and implicit in character. The way I understand, the codePage is set when a chr creates a character right value and it should be propagated as long a possible, so that the information is accessible when the binary representation (of the respective CP) is needed. There are some edge cases (line when two character values with different codePage are concatenated), but generally, I think the idea should work. We will handle the edge-case in subsequent iterations.

• I18nOps.java
  • lines 181-191: missing javadocs (I understand, this is WiP so some of their semantic might change)
  • lines 704-713: the combination containsKey() / get is a double lookup in a map. A more efficient (but also compact) form with identical result is:

    validCodePageConv.computeIfAbsent(fromCodePage, s -> new HashSet<>()).add(toCodePage);

  • lines 718-727: identical pattern as above
  • lines 733/735: double map lookup again. To avoid it, extract validCodePageConv.get(fromCodePage) into a local variable. If not null, check if it contains toCodePage;
  • lines 739-744: again, 2 double lookups. But also I think there is a logic flaw in the conditions which may lead to NPE. If !irreversibleConvs.containsKey(toCodePage), validCodePageConv.containsKey(toCodePage) is not checked, so validCodePageConv.get(toCodePage) may return null causing a NPE on line 743;
  • lines 750/752: same pattern;
  • lines 697, 702, 729, 748, 1016, 1087: new methods lack javadocs;
  • line 874: method's return type is changed to wrapper. This might cause conversion issues;
  • line 876: the default value adjustments for CPs were dropped. The CPs are the (optional) parameters of 4GL crt function. The defaulting should happen;
  • lines 897-898: First: the comment was converted to an unused String variable. ErrorManager will automatically compose the message based on error code and parameters, allowing for future internationalisation. The comment facilitate the understanding from reader's PoV. Second: throw was converted to show. Please confirm that this is the case;
  • line 911: (and several occurrences downward) for unassignable returned values, please avoid non-trivial creation of BDT objects by using:
    • character.UNKNOWN for unknown instances (i.e. instead of new character()). There is also EMPTY_STRING for "".
    • character.of() for fast access to a cached constant;
  • line 976: please use CharacterCodingException instead of much wider Exception, even if unexpected blocks. NPEs or other runtime exception could be unintentionally caught;
  • line 1056/1057: NumberFormatException is not caught;
  • line 1873: this class should be implemented as an immutable map key, with precomputed hash value for fast map access.
• character.java
  • lines 493, 499: missing javadocs
  • lines 737, 771, ...: please use character.UNKNOWN constant for returned value (see above);
  • line 996: can be dropped, the local value not used further on;
  • lines 1064/1065: there is a constant CP_UNDEFINED defined in I18nOps.

Here's a problem. Say we have the following procedure that is encoded in Windows 1252:

DEFINE VARIABLE intData AS INTEGER NO-UNDO.
intData = ASC("€").
MESSAGE intData.

I confirmed that the byte that represents the euro sign is 80, compiled this procedure into a .r file with different cpstream and cpinternal values then I made another procedure to check the RCODE-INFO:CODEPAGE value and to run the compiled procedure. I also used different values for running this "checking" procedure.

• RCODE-INFO:CODEPAGE always shows the cpinternal value used to run the procedure that compiles is into the .r file (the cpinternal value at compilation)
• For almost every single combination of the codepages used, the number returned by the simple procedure above is different

This goes to show that there are many other variables involved when deciding the output for ASC.

I could not find a simple, logic rule to be able to accurately and consistently predict the output in those complicated testcases. There is clearly some conversion involved. The code works but it might not be worth going as far as handling these cases where one would compile one procedure with some values for cpinternal and cpstream and then running them with completely different values and as far as I know, we don't offer support for this.

I'll continue working on finding a general solution for the normal testcases to at least have a baseline for the asc function that is able to work as expected and later we can add support for these cases.

The thing is that in FWD we don't have access to the bytes of the source file. We always need to assume that the encoding of the source file is the one set by the cpstream parameter (which defaults to UTF-8 or otherwise needs to be specified in the .p.hints file or in p2j.cfg.xml).

• please add history entries, update copyright year and javadocs as needed
• build.xml - we usually copy the src/ resources to build/classes folder (and build/classes.aop/ automatically) and the jar target is just updated to include those file extensions
• BinaryData - please optimize setString
• I18NOps - static fields like cachedConvmaps need to be either concurrent maps or synchronized
• are you sure character.chr can't receive a long value in 4GL? The parameter type was changed from long to int. This is a matter of compilable code, not that a 64-bit value exceeding integer.max is not allowed at runtime (raises exception).
• util/convmaps/*.dat files are not added to the branch

Otherwise, please commit to old testcases project all tools or additional 4GL code which were used for testing. Post here how to use them, also.

Thanks for the review, I'll make the changes before the final commit. I'll also include the convmaps in the next commit. For now, I committed the procedures used to generate the differences in the convmaps on the testcases project from xfer, in the char_conversions_util folder.

I've gone more in-depth on what ASC does in some cases. Take the following (simple) case:

File encoding: 1252 <- € is represented on byte 80

DEFINE VARIABLE intData AS INTEGER NO-UNDO.
intData = ASC("€").
MESSAGE intData.
MESSAGE HexUtils:intToHex(intData).

Consider cpinternal as ISO8859-1 for convenience. It plays a role which I'll describe but can be taken out for simplicity's sake as it defines a one-to-one conversion mapping to codepages ISO8859-15 and 1252.

For cpstream set as ISO8859-1, ISO8859-15, 1252 the result is the byte 80 which is expected.

For cpstream set as IBM850, the result is the byte C7. To achieve this result, we look at the byte present in the file which is 80. Take that byte and represent it using cpstream. We check the ISO8859-1_to_IBM850 convmap which is inversible (means we can also use it to convert from target to source) and see that the byte 80 in IBM is mapped to 199 in ISO8859-1. 199 is C7 in hex which is the result returned.

This shows why we need to make sure we have the correct cpstream value to be able to return the correct result at byte-level. I will make this assumption when implementing the ASC function.

Another thing that might be worth mentioning is that, if we change the example above to intData = ASC("€", "ISO8859-1"). and run it with cpstream IBM850, it results in -1 being returned and the Code page conversion table for ISO8859-1 to ISO8895-1 was not found in convmap.cp. (6063) and 1587 exceptions being thrown which is funny 😂.

This is another example of how using the wrong cpstream value can influence the outcome and why we need to avoid it.

EDIT: Typo in the second parameter of ASC lead to the behavior described above marked with the line-strike and should not be considered.

The CPSTREAM only comes into play in the MESSAGE statement, right? My understanding is that ASC does not consider CPSTREAM at all.

Greg Shah wrote:

The CPSTREAM only comes into play in the MESSAGE statement, right?

No, that's not the case. When I use the MESSAGE statement, I only print letters and numbers which are available in any encoding and don't print special characters at all. I calculate the byte representation with the following method:

    METHOD PUBLIC STATIC CHARACTER intToHex(INPUT vInt AS INTEGER):
        IF vInt = 0 THEN RETURN "0".
        IF vInt = ? THEN RETURN "?".

        DEFINE VARIABLE hexDigits AS CHARACTER INITIAL "0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F" NO-UNDO.
        DEFINE VARIABLE hexResult AS CHARACTER NO-UNDO.
        DEFINE VARIABLE r AS INTEGER NO-UNDO.
        DO WHILE vInt > 0:
            r = vInt MODULO 16.
            hexResult = ENTRY(r + 1, hexDigits) + hexResult.
            vInt = TRUNCATE(vInt / 16, 0).
        END.

        RETURN hexResult.
    END METHOD.

This is independent of the cpstream and cpinternal values.

My understanding is that ASC does not consider CPSTREAM at all.

Actually, the example I've posted above shows that it does for literals.

This won't be the case if instead of the character literal we place the result of a CHR call. Delia proved that the result of a CHR call is not affected by cpstream/cpinternal values.

You're saying that intData = ASC("€"). will change the result based on the CPSTREAM? That is very unexpected. I expect it to vary with CPINTERNAL but not with CPSTREAM.

Greg Shah wrote:

You're saying that intData = ASC("€"). will change the result based on the CPSTREAM? That is very unexpected. I expect it to vary with CPINTERNAL but not with CPSTREAM.

I know. This happens because the encoding of the file is different than cpstream.

What I think progress does for literals is, it takes the individual bytes from the source file, no matter the encoding, and thinks of them through the cpstream code page then converts them to the cpinternal as it is instructed. This guarantees that every character literal in the program will be represented in cpinternal.

I think Marian also pointed this before but we were not able to see it.

That would be the CPSTREAM at the time of COMPILE. Our general approach to string literals is currently to yield UTF-8 Java source code, which after javac will be in the constant pool of the class file in a form that is mostly UTF-8 (I think it handles the null character differently).

In memory, these characters will be in the Unicode-16 format of Java strings. So if we needed to implement this behavior of the 4GL, we would have to record more information in the converted code. As Marian noted, it isn't necessarily a good idea to do this in the 4GL so we might not want to support that behavior. BUT before we decide how to implement, we must document the behavior that exists.

So how should I move forward with this?

To implement the ASC function I'll need to look into the .hints file to see if there's a source-charset defined and if not then to assume the cpstream for string literals and perform the necessary conversions.

We can write code when we have:

1. Fully detailed rules that describe the behavior of ASC and CHR. The details that we have are linked in a Google doc in #4766-172. I have done a quick read through, I have some questions.
   • The document reads as if it was mostly written by AI. I apologize if you wrote it by hand, but there are many places in the document where it uses terminology that sounds like it was written by Progress Software Corp or some 4GL developer that is a fan of OpenEdge.
     • "modern OpenEdge applications" (we rarely if ever would refer to any 4GL application as modern)
     • "Progress OpenEdge Relational Database Management System (RDBMS)" (while by some strict criteria, the OE database could loosely be called relational, it lacks important features that we consider essential for a true RDBMS, we would never call it that)
     • Lots of references to ABL and even a mention of Advanced Business Language. (there is nothing advanced about the language and we generally try to use 4GL which is more accurate and avoids the "ABL" marketing term)
     • "it follows a rigorous decision tree" (weird insertion of "rigorous", are we really sure that the ASC implementation is rigorous?)
   • It is OK to use AI like this, but we must recognize the limitations of that approach.
   • The biggest problem with relying upon AI for this is that it can only go by what is written. This is primarily the OE documentation and then some amount of forum posts. We have found that these sources often leave much unstated and sometimes they are simply wrong.
   • Another problem with AI is that it can hallucinate. It is possible that some of this is just made up.
   • I don't see enough detail in the ASC and CHR sections to tell us how to write a fully compatible replacement.
     • It provides no details on the specific error processing logic, including error codes and message text for each error.
     • It leaves key details undocumented, for example "ASC typically evaluates the first character or returns -1 depending on strictness settings".
   • My sense here is that we need to verify and extend this documentation using testcases to prove the behavior and to explore areas that could not be explored by AI.
2. A comprehensive set of testcases for both ASC and CHR, for a subset of source/target/CPINTERNAL/CPSTREAM values. We can make this "supported configuration" list to know what testcases to write. This must be comprehensive enough to prove:
   • That the logic/rules of ASC/CHR as documented is correct.
   • That our implementation is compatible.
   • In the future, that we don't have regressions when we make changes.

Paul: Please discuss with Constantin and report back with a proposed plan. Apologies again, if I have treated your personal work like AI output. Correct me if that is the case.

Also, the Google doc should probably be a .pdf attached here and then the specific rules/logic ofr ASC and CHR should be a wikie page in our docs or posted in notes here.

No need to apologize. Yes, the document was generated with AI and yes, there were some hallucinations but I've reviewed every information and left only what I could verify to be true, I mentioned this here :)

I have to point out that these documents were generated using the "smartest" LLM from Google (Gemini Pro) with the Deep Research option. I've analyzed every information, taken out all the hallucinations and verified all information that I was able to but do take them with a grain of salt.

I did this to summarize everything that is available online. The rule that I've come up with for CHR is based only on what Delia found before and the many experiments I ran. The code is available on the 4766b branch.

I haven't wrote tests yet because I thought it would be too much time consuming. I know that Marian wrote some tests for this. See #4766-80

The AI output is useful as is our own analysis of documentation and other web sources. But ultimately, the tests are what matter. Make them comprehensive including covering the full range of error processing, boundary conditions and other weirdness. Testcases form a spec that is precise and proven to be correct by the runtime implementation of OE itself. If we miss something, we add more tests until it is complete.

The problem with the documentation is that it is incomplete and sometimes incorrect. Even with unlimited time, you cannot create a correct spec because a large amount of the actual behavior is just hidden inside the runtime but is not documented anywhere. If we could write our implementation using only the documentation, we would have completed it long ago. The actual runtime behavior is always much more complex than the documentation and often has some really weird undocumented behavior.