FWD » Core Development » Base Language

The conversion is fixed in 6129a. Please make the changes in 6129a.

Marian: Please review #6389-3 and post any thoughts.

Greg Shah wrote:

Marian: Please review #6389-3 and post any thoughts.

Greg, I've never used COLUMN type on LENGTH function before and doing a couple of quick tests I have to agree with Ovidiu and just make that return the same result as when CHARACTER type is used for time being and then worry more about it when we have a use-case that fails... I've couldn't thought of anything new, trying different codepages (internal/stream) for client startup, chui/gui doesn't really make any difference so no idea when the results might be different nor why this was added in the first place :(

I think this is related to chinese characters:

def var ch as char.
ch =  CHR(14990001, "cp936","UTF-8").
message length(ch, "column").

Some details here: https://knowledgebase.progress.com/articles/Article/P108864 and here: https://stackoverflow.com/questions/3634627/how-to-know-the-preferred-display-width-in-columns-of-unicode-characters

Thank you. My search were in vain. So it is the 3rd bullet (CJK glyphs).

I tried to launch the pro/prowin with the CP parameters from KB but I am getting the following errors:

DO NOT CONTINUE. Character set cp936 requires DBE PROGRESS.  You may corrupt files or databases. (3624)
Unable to open word-break table file 247. (2736)
The word-rule file specified by the -ttwrdrul parameter is invalid. (9258)

I added runtime support for LENGTH(..., 'COLUMN'). For the moment it delegates to LENGTH(..., 'CHARACTER') routine. A message will be logged in this event. Committed to 6129a as r13875.

Some notes. I did further investigations based on Constantin example in note 7.
Using the ChUI client, and altering the code to look like:

message ch "character"           length(ch, "character").
message ch "char"                length(ch, "char").
message ch "raw"                 length(ch, "raw").
message ch "column"              length(ch, "column").

üï character 2
üï char 2
üï raw 2
üï column 2

üï character 0
üï char 0
üï raw 2
üï column 2

The funny thing is, FWD will (correctly ???) decode it as 亱.

A second issue here is related to other differences of the result when the same code is executed on ChUI compared to GUI. Consider the code:

message ch 'trim("+++") + "+++"' length(ch, trim("+++") + "+++").

üï trim("+++") + "+++" 2

** The data type argument value must be "raw", "character" or "column". (1186)
üï trim("+++") + "+++"

Because the implementation of the LENGTH function reside in com.goldencode.p2j.util package, the attempt to use LogicalTerminal.isChui() will create unwanted dependency to com.goldencode.p2j.ui.*;. So I commented out the code, but kept it.

Maybe the sample attached to this KB entry could help.

[[https://community.progress.com/s/article/P193813]]

I wrote an implementation for the columnLength method, but it relies on the jdk.internal.org.jline.utils.WCWidth class which is in the internal packages of java, not meant to be used, the problem with these packages is that java does not offer support for them and can even exclude them in any update, however the code we need is opensource and distributable under the BSD license. My question is, can we use the code from WCWidth?

Artur, you mean this project? https://github.com/jline/jline3

Greg: please comment about BSD.

Constantin Asofiei wrote:

Artur, you mean this project? https://github.com/jline/jline3

No, it it a class in the internal java package jdk.internal.org.jline.utils.

The same class exists in FWD as p2j/build/lib/jline-2.14.6.jar:jline.console.WCWidth. I'm confused why you need the one from JDK.

I was not aware we already have it :), thank you.

DEFINE VARIABLE c1 AS CHARACTER FORMAT "x(35)" NO-UNDO.

c1 = CHR(14990001, "cp936","UTF-8") + // 2
     CHR(14990725, "cp936","UTF-8") + // 2
     "a" + // 1
     "b".  // 1

message length(c1,  "raw"). // 6
message length(c1,  "character"). // 6
message length(c1,  "column"). // 6

I have a fix for the COLUMN parameter, but it looks like the same method could be used for all the parameters.

Should I commit the fix for the "Column" parameter, or fix the other ones too?

Did you test in both ChUI, GUI and batch process?

I am using tests like the one in #6389-21.

Ovidiu and Eugenie: Please review.

The approach used in 15563 should work for 16-bit Java characters. If 4GL LENGTH function always return number of bytes for string provided. According to Progress doc(yes we can not trust it with 100% degree) this function returns number of chars(or symbols another word).

But I'm not sure it will work for complex symbols exceeding 16-bit, having low surrogate and high surrogate pair for single symbol. The regular Java DK has API to work for such cases, no need to use extra libraries. The concept is 'code point'. Every symbol can be represented with single 32-bit code point. The Character.charCount() can be used to detect how much chars need to represent given code point.

However the main question here I think is to detect what is the output of the LENGTH in 4GL. Bytes or symbols? What if to use different code pages for CPINTERNAL/CPSTREAM. Honestly I doubt if return value is bytes, because for UI layout this is useless info and can produce bad layout in 4GL.

Review of 6389a/15563

• your sample in note #6389-23 seems a bit wrong. I was able to run it using utf-8 cpinternal and

      CHR(14990001, "UTF-8") + // 2
      CHR(14990725, "UTF-8") + // 2

  with the following result:

  亱佅ab raw: 8
  亱佅ab character:4
  亱佅ab column:6

• the implementation of the jline.console.WCWidth.wcwidth(). It seems not to return the exact values we need. For some quite usual values, it returns 0 or -1 (see javadoc):

  Non-spacing and enclosing combining characters (general category code Mn or Me in the Unicode database) have a column width of 0.
  Other C0/C1 control characters and DEL will lead to a return value of -1.

  I think 4GL returns 1 in these cases.
• the jline library itself. I cannot find a jdk.internal.org.jline.utils in JDK1.8. More than that, your code says: import jline.console.WCWidth;. Indeed, it appears in the Software Dependencies page, but I did not see any other usage of it. If there had existed, they were removed. We usually prefer to have as less dependencies as possible. I think we wanna drop it.
• I am not sure 55357 is a valid character. For utf-16, you can use redmine to see how they look: & = &, � = �, é = é, 亱 = 亱 (chr(14990001)), 佅 = 佅 (chr(14990725)). I was not able to get a character out of this number neither on utf-8.

My conclusion: I think we should implement a fast method which scans the Unicode planes for CJK characters and 2 based on whether a specific character is part of the charts specified in this wikipedia article.

Eugenie Lyzenko wrote:

However the main question here I think is to detect what is the output of the LENGTH in 4GL. Bytes or symbols? What if to use different code pages for CPINTERNAL/CPSTREAM. Honestly I doubt if return value is bytes, because for UI layout this is useless info and can produce bad layout in 4GL.

Eugenie, please see the article quoted by Marian in note #6389-11:

Characters in the CJK (Chinese, Japanese and Korean) languages are historically double-width characters: they take up two print columns to draw the glyph. The ABL FORMAT defines the number of print columns, not the number of characters. The number of bytes is irrelevant; it just happens that the CJK characters are encoded using three bytes in UTF-8. [...] The CJK characters are sorted into specific ranges (“blocks”) within Unicode. Characters within these blocks are all double-width. They require three bytes to encode them in UTF-8.

I was operating under the assumption that chr(, "utf-8") encodes a character using utf-8 and 4gl will calculate the length accordingly, apparently it is not enough, I redid all the testing using utf-8 cpinternal.
Ovidiu, won't manually scanning the Unicode planes for any type of characters limit the functionality of the length function to these specific characters? CJK is just a subset of multi byte chars. Also, from what I have found, there are characters in the 2 and 3 byte range that occupy a single column, a lot of thai characters for example.

It looks like the number of columns means the number of columns that a character would occupy if it had been displayed correctly (???).
There are multi byte chars that I am not able to display correclty, for these, the length in columns is 1, because the output is a substitute char that occupies a single column.
There are instances when I am able to display a character that is clearly wide, but 4gl returns the length of 0. I wonder if the displayability (and correct column length) of a character depends on the codepages in convmap.cp.

DEFINE VARIABLE c1 AS CHARACTER.

c1 = CHR(0xE0AF8C). // ௌ
display c1. // displays correctly
message "bytes: " + string(length(c1,  "raw")). // 3
message "characters: " + string(length(c1,  "character")). // 1
message "columns: " +  string(length(c1,  "column")). // 0

c1 = CHR(0xE0AFB5). // ௵
display c1. // displays correctly
message "bytes: " + string(length(c1,  "raw")). // 3
message "characters: " + string(length(c1,  "character")). // 1
message "columns: " +  string(length(c1,  "column")). // 1, but looks like it should be at least 2

c1 = CHR(0xD494). // Ԫ
display c1. // unable to display, displays '?'
message "bytes: " + string(length(c1,  "raw")). // 2
message "characters: " + string(length(c1,  "character")). // 1 
message "columns: " +  string(length(c1,  "column")). // 1, the length of ?, not Ԫ 

c1 = CHR(0xF097BB90). // 𗻐
display c1. // unable to display
message "bytes: " + string(length(c1,  "raw")). // 4
message "characters: " + string(length(c1,  "character")). // 1 
message "columns: " +  string(length(c1,  "column")). // 1

c1 = CHR(0xE5BFA7). // 忧
display c1. // displays correctly
message "bytes: " + string(length(c1,  "raw")). // 3
message "characters: " + string(length(c1,  "character")). // 1 
message "columns: " +  string(length(c1,  "column")). // 2

c1 = CHR(0xf09f9881). //GRINNING FACE WITH SMILING EYES emoji
display c1. // displays correctly
message "bytes: " + string(length(c1,  "raw")). // 4
message "characters: " + string(length(c1,  "character")). // 1
message "columns: " +  string(length(c1,  "column")). // 1

Are these results from OE?

Yes, the results are from OE. In FWD the length function does not work with any of the parameters if multi byte chars are used, it returns 0 for the test cases above, but that is due to the fact that the chars are not recognized at all, the value for them is ""(empty).

Artur Școlnic wrote:

Ovidiu, won't manually scanning the Unicode planes for any type of characters limit the functionality of the length function to these specific characters? CJK is just a subset of multi byte chars. Also, from what I have found, there are characters in the 2 and 3 byte range that occupy a single column, a lot of thai characters for example.
It looks like the number of columns means the number of columns that a character would occupy if it had been displayed correctly (???).

Yes, the number of bytes in UTF-8 encoding has nothing to do with the number of columns. See the bold text in my message for Eugenie.

There are multi byte chars that I am not able to display correclty, for these, the length in columns is 1, because the output is a substitute char that occupies a single column.
There are instances when I am able to display a character that is clearly wide, but 4gl returns the length of 0. I wonder if the displayability (and correct column length) of a character depends on the codepages in convmap.cp.

I do not think 4GL does all these computations internally. I think it defer it to a OS low-level routine.

I assume the FWD implementation you mention is using jline library?

One more thing: Ԫ (Ԫ) (0xD494, which seems to be correctly displayed by Thunderbird but not by Firefox) is reported by UTF-8 tool as having 0xD4AA encoding (see also here).

Artur Școlnic wrote:

Yes, the results are from OE. In FWD the length function does not work with any of the parameters if multi byte chars are used, it returns 0 for the test cases above, but that is due to the fact that the chars are not recognized at all, the value for them is ""(empty).

I am guessing you mix the UTF-8 encoding used by CHR function and the Unicode offset. They are different, although, only for ASCII range, they overlap. The jline library checks the latter.

To extract the column lengths for all 64Ki Unicode characters we can use the following procedure (do not forget to start OE with a codepage which covers the full Unicode set, for example utf-8):

output to "/tmp/length.log".

define variable ch as integer.
do ch = 0 to 0xFFFF:
    message length(chr(ch, 'UTF-8', 'UTF-16'), 'column').
end.

Correct me if I am wrong, but the main problem with an approach like this is the fact that we can't rely on intervals, in the same interval there are various column lengths, this is an example:

char  :length in columns
60880 :1
60881 :1
60882 :0
60883 :2
60884 :2
60885 :1
60886 :0
60887 :3
60888 :3
60889 :1

We can keep this data in a file (or an array) directly, no need to complicate things with intervals.

But: are we sure characters in a certain string are not 'merged' into i.e. 3 columns even if both are 2 columns?

I extended the extractor to:

output to "/tmp/length.log".

define variable ch as integer.
do ch = 0 to 0xFFFF:
    message length(chr(ch, 'UTF-8', 'UTF-16'), 'column').
end.

output to "/tmp/length-intervals.log".

define variable intervalStart as integer.
define variable cols as integer.
define variable lastCols as integer.

intervalStart = -1.
lastCols = -1.
do ch = 0 to 0x10000:
    cols = length(chr(ch, 'UTF-16'), 'column').
    if cols ne lastCols then do:
        if intervalStart ge 0 then
            message lastCols ":" intervalStart "-" (ch - 1).
        intervalStart = ch.
        lastCols = cols.
    end.
end.

Constantin Asofiei wrote:

But: are we sure characters in a certain string are not 'merged' into i.e. 3 columns even if both are 2 columns?

From the testing I have done, I saw that the column length is a 'per character' attribute, I have not seen any merging being done. Maybe someone else can confirm this.

I am sorry, I forgot to add the 'destination' parameter for chr function. I updated the procedures above. It turn out there are only 230 intervals, which is more manageable.

Here:Here:

output to "/tmp/length.log".

define variable ch as integer.
do ch = 0 to 0xFFFF:
    message length(chr(ch, 'UTF-8', 'UTF-16'), 'column').
end.

output to "/tmp/length-intervals.log".

function toHex returns character (input i as integer):
    define variable s as char.
    define variable d as int.
    s = ''.
    do while i > 0:
        d = i mod 16.
        i = (i - d) / 16.

        if d < 10 then s = string(d) + s.
                  else s = chr(asc('A') + d - 10) + s.
    end.
    s = '0000' + s.

    return substring(s, length(s) - 3, 4).
end.

define variable intervalStart as integer.
define variable cols as integer.
define variable lastCols as integer.

intervalStart = -1.
lastCols = -1.
do ch = 0 to 0x10000:
    cols = length(chr(ch, 'UTF-8', 'UTF-16'), 'column').
    if cols ne lastCols then do:
        if intervalStart ge 0 then
            message lastCols ":" toHex(intervalStart) "-" toHex(ch - 1).
        intervalStart = ch.
        lastCols = cols.
    end.
end.

Maybe there is one more thing we can try, I was thinking that the column length is determined by the width of the character in pixels divided by the width of one column, so when a char width exceeds the column width in pixels, it is displayed on one more and so on. When it comes to text size and font, the width of a column should also be scaled accordingly. Maybe there is someone that worked with UI that could provide some feedback on this?

My understanding from that time is the 'column' length is to align the text for i.e. printing. So this can not depend on any pixels-per-column setting.