FWD » Core Development » Conversion Tools

The function issue is actually a 4GL preprocessor quirk. I've created a testcase and checked it into Old UAST Testcases (see testcases/preproc/post_string_alternative_coding_quirk.p). The fix will need to be in ClearStream (search on the usage of getAlternativeCoding()). The only tricky part is that we must know when we are adjacent to and following a string literal. We do get a notification of being inside a string, but we need to check that the notification occurs at exactly the right moment AND we would have to set an extra flag followsString and clear that flag on the next character. Since this was a typo in the customer code where it was found, work on the "bug" was deferred.

The testcase:

// outside of comments and strings there is a quirk of the "alternative codings" 
// - the ;) alternative coding construct will be converted by the preproc to the } character
// - this occurs in comments and in regular code
// - no conversion occurs inside strings (either '' or "" style)
// - the quirk is that any ;) in code that immediately follows a string literal (either '' or "" style) will NOT convert but will emit as a single ) char
// - no whitespace or other chars can be between the string literal and the ;), the quirk only happens if it is directly following
// - this quirk does not manifest in comments
// - since alternative codings don't convert in strings, this quirk also doesn't appear in strings

def var txt1 as char.
def var txt2 as char.
def var txt3 as char.
def var i as int extent 2.

function func0 returns int(input i as char).
end.

// in each of the following comments, the ;) will be converted into the } character

/* "";) */
// "";)
/* ";) */
// ";)
/* '';) */
// '';)
/* ';) */
// ';)

// preprocesses to:
// func0("").
func0("";).

// preprocesses to:
// func0('').
func0('';).

function func-garbage returns int (input t1 as char, input-output t2 as char, input t3 as char):
   return 14.
end.

// preprocesses to:
// i[1] = 3.
i;<1;> = 3.

// preprocesses to:
// func-garbage("whatever", txt2, "whatever").
func-garbage("whatever", txt2, "whatever";).

// preprocesses to:
// func-garbage("whatever", txt2, 'whatever').
func-garbage("whatever", txt2, 'whatever';).

txt3 = ";)".
txt3 = ';)'.

// preprocesses to:
// txt1 = (txt2 + "stuff").
txt1 = (txt2 + "stuff";).

// preprocesses to:
// txt1 = (txt2 + 'stuff').
txt1 = (txt2 + 'stuff';).

// OO support would also be affected if the ;) occurs after a string, but it is not specific
// to function parms or OO parms
// def var basic as oo.basic.BasicMethods.
// basic = new oo.basic.BasicMethods().
// basic.something(9, txt2, "";).

Code Review Task Branch 6308a Revision 15851

My only concern is that this quirk will occur for any alternative coding. My original testcase only tests ;). Please check if this occurs for the other alternative codings. If so, then your change will work.

If only ;) has this quirk, then we need to limit the change accordingly.

Greg Shah wrote:

Code Review Task Branch 6308a Revision 15851

My only concern is that this quirk will occur for any alternative coding. My original testcase only tests ;). Please check if this occurs for the other alternative codings. If so, then your change will work.

If only ;) has this quirk, then we need to limit the change accordingly.

• ;>
• ;<
• ;*
• ;%
• ;)
• ;(

Have this behaviour.

• ;&
• ;'
• ;?

I updated this, new rev is 15852.

while working on this I saw that the inString variable is not set correctly when reading the stream:
We have for example the following "text";&
When the read() function from ClearStream reaches ; the inString boolean is still set on True.
As a workaround i put this in text.g:

         // most (but not all) alternative codings that follow (without whitespace) a string literal
         // have a 4GL preproc quirk where the ; is eaten and the following char is left behind
         if (LA(1) == 0x3B                                    &&   // 0x3B is ; 
             ClearStream.isAlternativeCoding(LA(1), LA(2)))
         {
            if (LA(2) != 0x26                                 &&   // 0x26 is &
                LA(2) != 0x27                                 &&   // 0x27 is '
                LA(2) != 0x3F)                                     // 0x3F is ?
            {
                consume();
                text.setLength(text.length() - 1);
            }
            else
            {
                consume();
                consume();
                text.setLength(text.length() - 2);
                if (LA(1) == 0x20)
                {
                    consume();
                    text.setLength(text.length() - 1);
                }
            }
         }

Post this as a diff, please.

I reverted this change since it wasn't producing what I expected, but in return I added:

                 'a'..'z'
               | '-'
            )+
-           
+         | ' '  
          | // nothing, this is really "code" 
       )
       // this will be overridden by the literals testing when necessary
       { 
+         if (text.toString().equals("& "))
+         {
+            text.setLength(text.length() - 2);
+         }

As a summary, in text.g I added this to now match & and consume it since that's the behavior in 4GL.

Please show me some 4GL examples of this behavior.

When preprocessing the following

& GLOBAL-DEFINE arg 1
&GLOBAL-DEFINE arg 1
txt = "txt";& .
txt = 'txt';& .
txt = 'txt';& GLOBAL-DEFINE arg 1 .
txt = 'txt';&GLOBAL-DEFINE arg 1 .

GLOBAL-DEFINE arg 1

txt = "txt".
txt = 'txt'.
txt = 'txt'GLOBAL-DEFINE arg 1 .
txt = 'txt'

Code Review Task Branch 6308a Revision 15903

I'm a little nervous about this change.

Please add preprocessor tests for these cases. This should include the test code in #6308-16, but it also needs to be expanded to check the full range of ASTMT matches. This means adding tests for:

&scoped-define
&message
&undefine
&if
&then
&elseif
&else
&endif
&analyze-suspend
&analyze-resume

These tests should be written not just as copies of the &global-define but in the proper structure for the given ASTMT.

Also, for things like the &global-define, don't repeat the definition name arg unless you are indeed testing whether the name is overridden by subsequent definitions. Instead, the test code should use different names and check which ones actually get defined.

• &scoped-define
• &if
• &then
• &elseif
• &else
• &endif

Another bug I've found:
While preprocessing this:

/*if and elseif*/
&IF DEFINED(arg01) = 0 &THEN   <- 3 spaces
    "undefined" 
"elseif arg01 global";&ELSEIF DEFINED(arg01) = 1 &THEN
    "global" 
"elseif arg01 scoped";&ELSEIF DEFINED(arg01) = 3 &THEN
    "scoped" 
&ELSE
    "fileref" 
&ENDIF

/*if and elseif*/
   <-- 3 spaces
    "undefined" 
"elseif arg01 global" 

/*if and elseif*/
    "undefined" 
"elseif arg01 global" 

Same behavior described in #6859-19 for &ENDIF

Another bug:

"undefine arg01";&UNDEFINE arg01/*end undefine*/ <- space
/*after undefine*/

"undefine arg01";&UNDEFINE arg01/*end undefine*/
/*after undefine*/

"undefine arg01";&UNDEFINE arg01 /*end undefine*/
/*after undefine*/

"undefine arg01";&UNDEFINE arg01 /*end undefine*/ <- space
/*after undefine*/

OE results:

"undefine arg01"/*end undefine*/
/*after undefine*/

"undefine arg01"/*end undefine*//*after undefine*/

"undefine arg01"/*end undefine*/
/*after undefine*/

"undefine arg01"/*end undefine*/ 
/*after undefine*/

What's happening is after &UNDEFINE a single char is consumed. (maybe is a token since WS and NL are consumed)
I fixed this by adding in the parser at the ppstatement:

-         | undefine
+         | undefine { consume(); }

Stefan Vieru wrote:

[...]
My interpretation is that OE just "eats" what is after &THEN.
How should I approach this?
Note: I tested with &IF and "if arg01";&IF, with one space or 3 spaces, the same results in FWD and OE like above.

I have tested with a comment after &THEN and a string, the conclusion that I've reached is that OE consumes any WS after &THEN until reaching anything else than a WS.

&IF DEFINED(arg06) = 0 &THEN /* after then*/ 

• STAR_COMMENT STAR_COMMENT
• STAR_COMMENT STRING
• STRING STAR_COMMENT

Only 1 and 2 will print, the difference being that in the first case we have 2 STAR_COMMENT in a row, they will both be printed.
In the second case we print the STRING if the condition is true.
Third case is printed fully only when the condition is true.

As a bottom line, after a ATHEN if there are multiple STAR_COMMENT right after it, they will be printed regardless of the condition. This is the case only for STAR_COMMENT, SLASH_SLASH doesn't have the same behavior.

I had to add this in condtext in the Parser:

      tmp0:ATHEN
+     (
+        ctmp0:STAR_COMMENT { env.print(ctmp0.getText()); }
+     )*

      tmp4:ATHEN
+     (
+        ctmp4:STAR_COMMENT { env.print(ctmp4.getText()); }
+     )*

A present bug:
Preprocessing this in OE allows the file to be included:

"txt";(tests/preproc/support/check_defined.i arg01}

"txt"(tests/preproc/support/check_defined.i arg01}

                 | '\\'                    // allow \ by itself
                 |   {
                        env.setInString(false);
                     }
                     '\"'
                     (
                        '\"' {env.setInString(true);}
                        |
                     )

The changes in the current version include the following, after ASTMT

       {
          if (text.toString().equals("&THEN") || text.toString().equals("&ENDIF"))
          {
             env.setInStatement(false);
             while (LA(1) == ' ' || LA(1) == '\t')
             {
                consume();

&IF DEFINED(arg02) = 0 &THEN
&IF {&arg01} = YES &THEN
    &SCOPED-DEFINE arg02 YES
  /* arg02 YES */
&ELSE
    &SCOPED-DEFINE arg02 NO
  /* arg02 NO */
&ENDIF
&ENDIF

&THEN // message

The cause is that after the &THEN of the first &IF, we set InStatement on false.
Removing this fixed the issue, but when we have:

That code was deliberately put there, so removing it is probably going to regress something.

This is a very tricky area because we had to add this state management in a way we would not have otherwise chosen. See this comment for ASTMT:

      // we used to set this after we matched the ampersand, but it turns out that there can be
      // constructs like &{&METADEFINE}-define where {&METADEFINE} expands to global or scoped
      // for which the 4GL happily treats the cumulative result as &global-define or
      // &scoped-define, this was broken when in-stmt was set too late because the next
      // characters read were marker chars not the expanded text; by setting this earlier
      // we eliminate the marker chars and everything works as in the 4GL
      env.setInStatement(true);

This inStat flag changes how we handle markers, how we handle deferred processing and how comments are matched. The regression is likely to be in one of those places.

Greg Shah wrote:

The cause is that after the &THEN of the first &IF, we set InStatement on false.
Removing this fixed the issue, but when we have:

That code was deliberately put there, so removing it is probably going to regress something.

This is a very tricky area because we had to add this state management in a way we would not have otherwise chosen. See this comment for ASTMT:

[...]

This inStat flag changes how we handle markers, how we handle deferred processing and how comments are matched. The regression is likely to be in one of those places.

I added previously that setter, because it helped process the // comments that are right after THEN and ENDIF. It wasn't previously put there and I removed it, I just reverted my change that caused issues.

OK, good.

Decided on the following change:

-         if (text.toString().equals("&THEN") || text.toString().equals("&ENDIF"))
-         {
-            while (LA(1) == ' ' || LA(1) == '\t')
-            {
-               consume();
-               text.setLength(text.length() - 1);
-            }
-            if (LA(1) == '/' && LA(2) == '/')
-            {
-               env.setInStatement(false);
-            }
-         }
-         else if (text.toString().equals("& ") && env.getTildeOffset() == 0)
+         if (text.toString().equals("& ") && env.getTildeOffset() == 0)
          {
             text.setLength(text.length() - 2);
          }
@@ -1624,6 +1622,21 @@
             }
          }
       }
+      (
+         {$getText.equals("&THEN") || $getText.equals("&ENDIF")}?
+            (
+                  {!(LA(2) >= 'a' && LA(2) <= 'z') &&
+                   !(LA(2) >= 'A' && LA(2) <= 'Z') &&
+                   !(LA(2) >= '0' && LA(2) <= '9')}? ' '!
+               |  // nothing
+            )
+            {
+               if (LA(1) == '/' && LA(2) == '/')
+               {
+                  env.setNeedsNewLine(true);
+                  env.setInStatement(false);
+               }
+            }
+         |  // nothing
+      )

The change I'm proposing now is basically consuming all the WS without appending them to the buffer, if @LA is not an alphabetical character or number. This stopping when we hit a alphabetical character or number.
I'll add some comments.

All changes have been moved to 6859b.

• alternative_coding_quirk_01
• alternative_coding_quirk_01_analyze
• alternative_coding_quirk_01_combined_directives_global
• alternative_coding_quirk_01_combined_directives_scoped
• alternative_coding_quirk_01_combined_directives_undefined
• alternative_coding_quirk_01_else
• alternative_coding_quirk_01_else_if
• alternative_coding_quirk_01_endif
• alternative_coding_quirk_01_if
• alternative_coding_quirk_01_message
• alternative_coding_quirk_01_post_string
• alternative_coding_quirk_01_then
• alternative_coding_quirk_01_tilde_strings
• alternative_coding_quirk_01_undefine
• alternative_coding_quirk_02
• alternative_coding_quirk_02_left_bracket
• alternative_coding_quirk_02_right_bracket
• alternative_coding_quirk_03
• alternative_coding_quirk_04
• alternative_coding_quirk_05
• alternative_coding_quirk_05_left_curly
• alternative_coding_quirk_05_post_curly
• alternative_coding_quirk_05_right_curly
• alternative_coding_quirk_06
• alternative_coding_quirk_07
• alternative_coding_quirk_08_global
• alternative_coding_quirk_08_scoped

Alexandru Lungu wrote:

Octavian, can you check if there is something left to be fixed here? Just check if the tests here are passing with trunk, tests were committed and if the fixed reached trunk.

Sure. I'm looking into it.

Stefan Vieru wrote:

Tests that cover this:

• alternative_coding_quirk_01
• alternative_coding_quirk_01_analyze
• alternative_coding_quirk_01_combined_directives_global
• alternative_coding_quirk_01_combined_directives_scoped
• alternative_coding_quirk_01_combined_directives_undefined
• alternative_coding_quirk_01_else
• alternative_coding_quirk_01_else_if
• alternative_coding_quirk_01_endif
• alternative_coding_quirk_01_if
• alternative_coding_quirk_01_message
• alternative_coding_quirk_01_post_string
• alternative_coding_quirk_01_then
• alternative_coding_quirk_01_tilde_strings
• alternative_coding_quirk_01_undefine
• alternative_coding_quirk_02
• alternative_coding_quirk_02_left_bracket
• alternative_coding_quirk_02_right_bracket
• alternative_coding_quirk_03
• alternative_coding_quirk_04
• alternative_coding_quirk_05
• alternative_coding_quirk_05_left_curly
• alternative_coding_quirk_05_post_curly
• alternative_coding_quirk_05_right_curly
• alternative_coding_quirk_06
• alternative_coding_quirk_07
• alternative_coding_quirk_08_global
• alternative_coding_quirk_08_scoped

alternative_coding_quirk_06 is the only test that is currently failing.

This is the testcase that fails:

;?u0042
;?011
;?~
;?\

This is the output in FWD:

~uu0042
~011    
~~~

\

This is the output in OE:

~u0042B
~011    
~~~
~\\

I used this patch to solve the last part of differences:

=== modified file 'src/com/goldencode/p2j/preproc/ClearStream.java'
--- old/src/com/goldencode/p2j/preproc/ClearStream.java    2026-03-09 11:33:44 +0000
+++ new/src/com/goldencode/p2j/preproc/ClearStream.java    2026-04-28 13:46:57 +0000
@@ -802,7 +802,18 @@
          int i;

          // step 1. read all leaders and the byte that follows them
-         while (nextChar == '~' || (nextChar == '\\' && unixEscapes))
+
+         char leader;
+         if (nextChar == '\\' && unixEscapes)
+         {
+            leader = '\\';
+         }
+         else
+         {
+            leader = '~';
+         }
+
+         while (nextChar == leader)
          {
             leaders.push(Character.valueOf((char)nextChar));
             n++;
@@ -872,7 +883,7 @@
          boolean passThru = false;

          // from now on, there is one leader on the stack
-         char leader = ((Character)leaders.pop()).charValue();
+         leader = ((Character)leaders.pop()).charValue();
          int nextCharTemp = nextChar;

          if (notInStrings && notInComments)

~uu0042
~011    
~~~
~\

1. We need to handle Unicode translation right at the preprocessing stage (related to #10028 and #3971);
2. The actual duplication should happen after the character is processed but before it is returned (like ~u0028 becoming B) and then duplicate the resulting character. Now, it looks like we treat u as the first character after tilde and we duplicate it.

I suggest we create a separate task to track the remaining 8 failing tests (documented in Preprocessor_Testcases). These are specific edge cases, and moving them to a dedicated tasks will allow us to close the initial testing tasks which are otherwise complete. These should be addressed following the completion of #10028.