FWD » Core Development » Conversion Tools

Take a look at the following 4GL example.

start.w:

def var a as char initial "".
a = 'text1'~'text2~'text3~';'.
message a.

The second line, when executing the COMPILE PREPROCESS statement in the 4GL environment, is:

a = 'text1'~''text2~'text3~';'.

The same line in .w.cache with trunk is:

a = 'text1'~'text2~'text3~''.

The same line in .w.cache with 6859b is:

a = 'text1'~'text2~'text3~'`.

The thing that is causing issues in the environment I am testing is the missing semicolon (with both trunk and 6859b), but as it can be seen before text2, there is also a double quote/apostrophe in 4GL.

a = 'text1'~'text2~'text3~'`.

I think this is a regression from #6859 :/ The trailing back tick shouldn't be there.

PS: #9903 may be the culprit

Alex/Danut: if you don't plan on assigning this issue to somebody from your team, can you please suggest a place where I should start investigating? I'm interested in solving the semicolon thing: a semicolon disappears when going from .w to .w.cache (note #10569-1). Thank you!

Radu Apetrii wrote:

Alex/Danut: if you don't plan on assigning this issue to somebody from your team, can you please suggest a place where I should start investigating? I'm interested in solving the semicolon thing: a semicolon disappears when going from .w to .w.cache (note #10569-1). Thank you!

I think you can start from src/com/goldencode/p2j/preproc/text.g.

I don't think this change is OK. I deliberately excluded those chars from that rule (i.e. // it is OK to match " or ' as the 2nd or later char).

It seems to me that the 4GL is not implementing the alternative coding in this case, but we are implementing it. It is not clear why this is the case. I also don't understand why the 4GL generates an extra ' before text2. That doesn't make sense. We need to understand why the 4GL does these things.

To be a bit more clear: we have applications that use a 4GL quirk where ' and " characters can be embedded inside symbol names/arbitrary text. In those cases, these quotes are not part of string literals. This is why the characters are matched at that location in text.g. See the malformed_symbol rule in progress.g.

I think the work should be done in 6859b, is that right?

Yes, that would probably be for the best.

The changes fix the testcase mentioned in #10569-1.

Radu, can you check if that's happening for your specific file as well?

Octavian Adrian Gavril wrote:

Radu, can you check if that's happening for your specific file as well?

Yes, I'll write a note here when I have the results.

Please make sure that these cases have been added to the Preprocessor Testcases (unit tests).

All of the unit tests should now pass?

Radu Apetrii wrote:

Octavian Adrian Gavril wrote:

Radu, can you check if that's happening for your specific file as well?

Yes, I'll write a note here when I have the results.

Indeed, the problem is solved. ✅

The cause of failing is this scenario:

"analyze-suspend"~;

            else if ((LA(1)=='"') && (LA(2)=='~')) {
                match("\"~\"");
            }

I did some experiments that add an extra ' as OE does, but this changed how the lexer process next input. For example, an extra ' after 'test1'~' would close the ASTRING and text2~'text3~';'. would be CODE.

Since the preprocessor tests are stable, is the current fix sufficient for this task?

What extra testing is needed for this issue? Should we run conversion of some apps?

We can take this approach:

1. Run the front end on an application. Confirm there are no errors.
2. Compare the .cache files with the trunk version. Confirm there are no unexpected differences.
3. If there are no cache file differences at all, then we don't need to test that application further. If there are any differences (even expected ones), then we should test full conversion and compare the source output to that generated by trunk. If there are no unexpected differences, the application is OK.

We should test with all major applications.

Greg Shah wrote:

We can take this approach:

1. Run the front end on an application. Confirm there are no errors.
2. Compare the .cache files with the trunk version. Confirm there are no unexpected differences.
3. If there are no cache file differences at all, then we don't need to test that application further. If there are any differences (even expected ones), then we should test full conversion and compare the source output to that generated by trunk. If there are no unexpected differences, the application is OK.

We should test with all major applications.

Sure. I'm on it.

This approach should test only the changes related to this task or it should include all the changes from 6859b? If we test 6859b/rev16145, then I expect to see differences caused by other changes as well.

I described the testing for all of 6859b. You can document that in #6859 itself. I don't see a reason to do separate testing by each subtask. The overall testing is enough (when combined with passing the unit tests).

I noticed some suspect differences for ChUI regression testing app:

   <customer_code_redacted>

But this syntax is not compiling in OE:

def var a as char.
a = "a".
a = "|"+a+"|" + "~n".

It's weird that with/without my changes, that file is converted successfully. But the previous testcase is not:

main.p:4:3: unexpected char: 0xFFFF
     [java]     at com.goldencode.p2j.preproc.TextLexer.nextToken(TextLexer.java:265)
     [java]     at antlr.TokenBuffer.fill(TokenBuffer.java:69)

• +"| is CODE
• " + " is a QSTRING. This way, the new line will be translated and the next code will be broken because of my last changes that set the 'in string' flag to false when we have "~n.

The expected result should be:

              +"|" + "~n".
             /  |   \  \
            /   |    |  \ 
           /    |    |   \
          V     V    V    V
       CODE QSTRING CODE QSTRING

Greg, do you recall why is it OK to match " or ' as the 2nd or later char in CODE? I used these changes and there are no differences for both my testcase and syman/dc/so-labr1.w, comparing the trunk with 6859b.

=== modified file 'src/com/goldencode/p2j/preproc/text.g'
--- old/src/com/goldencode/p2j/preproc/text.g    2025-12-05 12:41:56 +0000
+++ new/src/com/goldencode/p2j/preproc/text.g    2025-12-11 12:39:16 +0000
@@ -1797,6 +1797,8 @@
                 // it is OK to match " or ' as the 2nd or later char
                 ~(
                      ' '
                    | '\t'
                    | '\n'
                    | '&'
                    | '/' 
+                   | '\"'
+                   | '\''
                    | '*'
                    | '(' 
                    | ')' 

Could this be a reason why using ' or " as CODE is allowed on trunk?

Here is another testcase:

def var txt as char.

txt = "txt"~; ".
txt = "txt"~;".

txt = "a" +"b ~; ".

Preprocessor output in OE:

def var txt as char.

txt = "txt"~;; ".         <--- last quote open a string literal
txt = "txt"~;".           <--- ; is not duplicated so we are in a string

txt = "a" +"b ~;; ".      <--- ; is duplicated because we are not in a string due to missing white space before quote

It looks like we can open a string only if the quote/apostrophe is preceded by white space. So, I really think that it is OK to consume " or ' as the second or later char from CODE. Therefore, the changes mentioned in #10569-39 are incorrect. We should find another way to fix the issue described in #10569-36.

Greg, do you recall why is it OK to match " or ' as the 2nd or later char in CODE?

I don't recall exactly but I do think the ChUI regression app issue would be one of the cases. I do remember having to write that code explicitly while working on some other applications, so it will probably be found elsewhere too.

Please make sure to add all of these testcases into our ABLUnit preproc test set.

It looks like we can open a string only if the quote/apostrophe is preceded by white space. So, I really think that it is OK to consume " or ' as the second or later char from CODE.

Good find.

Also: My understanding is that the highlighting in the Progress IDE may be written as a one-off IDE-specific parser and is not the same code that implements the COMPILE in OE. So: the highlighting might be different than the compiler would accept.

Look at this testcase:

def var txt as char.

txt = "txt"~; ".
txt = "txt"~; "."    // not duplicated

txt = "txt"~; ".
txt = "txt"a~; "."   // not duplicated

txt = "txt"~; ".
txt = "txt" 
~; "."  // duplicated because is preceded by new line

txt = "txt"~; ".
txt = "txt" ~; "."   // duplicated

txt = "txt"~; ".
txt = "txt"|~; "."   // duplicated

Preprocessor output in OE:

def var txt as char.

txt = "txt"~;; ".
txt = "txt"~; "."    // not duplicated

txt = "txt"~;; ".
txt = "txt"a~; "."   // not duplicated

txt = "txt"~;; ".
txt = "txt" 
~;; "."  // duplicated because is preceded by new line

txt = "txt"~;; ".
txt = "txt" ~;; "."   // duplicated

txt = "txt"~;; ".
txt = "txt"|~;; "."   // duplicated

While checking the testcases mentioned in #10569,in #10569-40 and #10569-42, I noticed that the ; is duplicated only if tilde is the first character of our CODE rule. Basically, this happens only when it's preceded by non-consumed characters from the CODE rule:

          ~(
                 ' '
               | '\t'
               | '\n'
               | '&'
               | '/'
               | '(' 
               | ')'
               | '['
               | ']'
               | ',' 
               | '=' 
               | '<' 
               | '>'
               | '{'
               | '}'
               | '|'
           )

So, I think the right way to handle this in FWD is to allow the doubling only in the CODE rule, and only if it starts with a tilde ~. This avoids inside strings duplicates of ; (QSTRING and ASTRING) and matches how the OE preprocessor works.

This seems reasonable.

Code Review Task Branch 6859b Revision 16327

1. The changes to CODE seem like they might fix the +"a" case but would fail in the + "a" case (with intervening whitespace between the + and the string). Have you tested that?

2. The // it is OK to match " or ' as the 2nd or later char comment in that CODE rule is no longer accurate.

3. Shouldn't the changes in ClearStream.read() be limited only to the ~; case (which is the specific case where ; gets duplicated)?

Greg Shah wrote:

Code Review Task Branch 6859b Revision 16327

1. The changes to CODE seem like they might fix the +"a" case but would fail in the + "a" case (with intervening whitespace between the + and the string). Have you tested that?

In that case, CODE rule will be stopped by the first whitespace and the quote will open the STRING rule. So, I think it's safe. This testcase is converted successfully:

def var a as char.
a = "|" +      "A" + "|" + "~n".
MESSAGE a.

2. The // it is OK to match " or ' as the 2nd or later char comment in that CODE rule is no longer accurate.

You're right.

3. Shouldn't the changes in ClearStream.read() be limited only to the ~; case (which is the specific case where ; gets duplicated)?

They are limited only to the ~; case because env.tildeMode is 2 when the previous character was ~. So, the character x in the sequence ~x is duplicated only if x is not a semicolon ;.

1. The changes to CODE seem like they might fix the +"a" case but would fail in the + "a" case (with intervening whitespace between the + and the string). Have you tested that?

In that case, CODE rule will be stopped by the first whitespace and the quote will open the STRING rule. So, I think it's safe. This testcase is converted successfully:
[...]

Good. Please add this testcase and also variants that use ' and which mix " and '. Also, work with Radu to include his testcase from #6506-156 as well as variants that use ' and which mix " and '.

3. Shouldn't the changes in ClearStream.read() be limited only to the ~; case (which is the specific case where ; gets duplicated)?

They are limited only to the ~; case because env.tildeMode is 2 when the previous character was ~. So, the character x in the sequence ~x is duplicated only if x is not a semicolon ;.

OK, please enhance the comment there to make this obvious.

Greg Shah wrote:

Good. Please add this testcase and also variants that use ' and which mix " and '. Also, work with Radu to include his testcase from #6506-156 as well as variants that use ' and which mix " and '.

Sure!

OK, please enhance the comment there to make this obvious.

I fixed code comments in 6859b/16328.