FWD » Testing

We will implement a driver that directly executes the preprocessor and compares the resulting output against baselines. The harness would work really well for this since we have good file comparison tooling there.

We can use the same harness approach for testing expected preproc failures with those tests having to be coded to the error output rather than the source code output.

Documentation is in Preprocessor Testcases.

Created the following methods, adapted to the new testcases project:

    @Test.
    METHOD PUBLIC VOID TestPositionalArgs(  ):  
        def var x as char.
        x = {tests/support/show_preproc_args.i "1stpos" &name1="1" /* "tuff" */ &name2="2"}
        Assert:Equals(' ~{name1}  ~{name2} 1stpos ~{1} &name1=1 ~{2} /* ~{3} tuff ~{4} */ ~{5} &name2=2 ~{6}  ~{7} 1stpos &name1=1 /* tuff */ &name2=2 ~{*} "" ~{* (named)}', x).
    END METHOD.

    @Test.
    METHOD PUBLIC VOID TestNamedArgs(  ):  
        def var x as char.
        x = {tests/support/show_preproc_args.i &name1="1" /* "tuff" */ &name2="2"}
        Assert:Equals('1 ~{name1} 2 ~{name2} 1 ~{1} 2 ~{2}  ~{3}  ~{4}  ~{5}  ~{6}  ~{7} 1 2 ~{*} "&name1="1" &name2="2"" ~{* (named)}', x).
    END METHOD.

Nice. I think this approach can work well for some tests, maybe even all tests that are expected to "pass" in OE. For tests where the preprocessor should fail, we will need a different approach (for some ideas, see #6860).

There will be some tests which are hard to check in an ABLUnit test method. I still think we might need some level of text comparison to check complex cases.

There will be some tests which are hard to check in an ABLUnit test method. I still think we might need some level of text comparison to check complex cases.

What about Java unit tests exactly for the Java classes involved in the preprocessing? A framework can be built around it to standardize the test writing, but the overall attempt can revolve around an automatic testing engine like JUnit that can clearly state the mismatches in preprocessing.

I also agree with text comparison for that matter: baseline vs .cache file - which will stay in-memory for the sake of unit testing.

How should I test like in #6860?
I created the tests in 4GL but when converting with FWD, FWD doesn't have COMPILER handle support (CompilerOps is empty).

I have this testcase:

     @Test.
     METHOD PUBLIC VOID TestMissingIncludeFile(  ):
         DEF VAR errNum AS INTEGER  EXTENT 2.
          ASSIGN
              errNum[1] = 293
              errNum[2] = 193.

          COMPILE "tests/preproc/support/missing_include_file.p" NO-ERROR. // this file contains just {missing.i}
          AssertExt:Errors(errNum).                                   
          AssertExt:Error(293, '** "missing.i" was not found. (293)').
     END METHOD.

• braces.test
• clear.test
• file
• file2
• toplevel.test

What should I do about these?

Will look into harness for those tests that are using compile statement.

Alexandru Lungu wrote:

There will be some tests which are hard to check in an ABLUnit test method. I still think we might need some level of text comparison to check complex cases.

What about Java unit tests exactly for the Java classes involved in the preprocessing? A framework can be built around it to standardize the test writing, but the overall attempt can revolve around an automatic testing engine like JUnit that can clearly state the mismatches in preprocessing.

I also agree with text comparison for that matter: baseline vs .cache file - which will stay in-memory for the sake of unit testing.

I'm open to JUnit but the Harness does have good file comparison tools built in, it can report results in JUnit format (for nice integration with CI/CD dashboards) and it even has techniques to running different tools/commands in OE vs FWD.

How should I test like in #6860?
I created the tests in 4GL but when converting with FWD, FWD doesn't have COMPILER handle support (CompilerOps is empty).

The idea in #6860-8 is that we may not be able to use ABLUnit for our approach. I've made the same point here in #6859-8. Instead of ABLUnit, we encode the tests as 4GL source code and then we:

1. The "core" of each test is one or more 4GL source files (.p, .i, .cls...). This is not ABLUnit code.
2. Implement a driver for each environment:
   • In OE, the driver would be 4GL code that runs a COMPILE to get the listing (this is the equivalent of the .cache file.
   • In FWD, the driver would be a Java program that runs the Preprocessor directly (which will generate the .cache file).
3. For each 4GL testcase, we write a harness test to:
   • call the correct driver depending on whether we are in OE or FWD
   • compare the listing/cache file output to previously captured baselines
4. Create a test plan that includes all the tests and which generates the JUnit XML output that reports the results.

As noted in #6859-8, we can't support all use cases via ABLUnit. For this reason, I don't want to implement some of our tests using ABLUnit and other tests using JUnit or the Harness. Let's keep it consistent and have a single approach.

I guess the invalid input cases would have to be handled directly by the driver code, but I'm open to alternatives.

I have this testcase:
[...]
When running in 4GL the tests passes, but in FWD when compiling the file mentioned it doesn't throw and error, resulting in the test failing in FWD.

This is an example of why we can't use ABLUnit for some cases.

Tomasz can help with understanding the Harness approach including how to setup tests that can run under both OE and FWD but with different commands.

There is also the Harness Documentation.

Greg Shah wrote:

Tomasz can help with understanding the Harness approach including how to setup tests that can run under both OE and FWD but with different commands.

There is also the Harness Documentation.

Just a remark. Exporting JUnit reports from Harness is waiting to be reviewed, merged (and documented) as part of #9270.

While working with harness I developed a test that uses text-file-comparison
But when comparing the .cache and .4gl_preproc_output files I have some differences:
test file:

if {&bogus} eq 3 then
   message "Existing scoped definition WAS NOT redefined using global-define.".

{tests/preproc/support/redefine_argument_nested.i &argname="14"}
/*newline*/

if {&argname} ne 14 then message "ERROR: missing named argument 'argname'!".

&GLOBAL-DEFINE argname 9

if {&argname} eq 14 then
   message "Named argument WAS NOT redefined using global-define."./*nospace*/

if 3 eq 3 then
   message "Existing scoped definition WAS NOT redefined using global-define.".

if 14 ne 14 then message "ERROR: missing named argument 'argname'!".

if 14 eq 14 then
   message "Named argument WAS NOT redefined using global-define."./*nospace*/ <-- there is a space here
/*newline*/

.4gl_preproc_output file has the content:

if 3 eq 3 then
   message "Existing scoped definition WAS NOT redefined using global-define.".

if 14 ne 14 then message "ERROR: missing named argument 'argname'!".

if 14 eq 14 then
   message "Named argument WAS NOT redefined using global-define."./*nospace*/
 <-- there is a space here
/*newline*/

That seems like a bug to fix. Let's fix it (in a newly created task) once the test suite is ready.

Greg Shah wrote:

1. The "core" of each test is one or more 4GL source files (.p, .i, .cls...). This is not ABLUnit code.
2. Implement a driver for each environment:
   • In OE, the driver would be 4GL code that runs a COMPILE to get the listing (this is the equivalent of the .cache file.
   • In FWD, the driver would be a Java program that runs the Preprocessor directly (which will generate the .cache file).
3. For each 4GL testcase, we write a harness test to:
   • call the correct driver depending on whether we are in OE or FWD
   • compare the listing/cache file output to previously captured baselines
4. Create a test plan that includes all the tests and which generates the JUnit XML output that reports the results.

• for OE, I will implement a 4GL code that uses COMPILE on a set of files.
• for FWD, after discussing with Alex we've found com.goldencode.p2j.preproc.Preprocessor that is kind of but not really a driver, in the sense that it can be run from CLI but has no help or something like that. I can look into this class and make some improvements so I can use it in Harness testing

Now for the Harness part if someone has some documentation that would be helpful. For now I have a few tests that just compare the baseline with the .cache file. (text-file-comparison)
But the cache file has been produced after running front part of conversion. From what i understood is that Harness tests should be able to run this Preprocessor driver and compare the .cache file with the baseline.
In OE I see that there is a HarnessApi and maybe I should use that? I don't really know, but the test files in OE should run the procedure I will implement (that compiles the files) and compares the output with the already existing baseline (expected to be 100% passed here).
Let me know if my line of thinking is correct.

Stefan Vieru wrote:

Greg Shah wrote:

1. The "core" of each test is one or more 4GL source files (.p, .i, .cls...). This is not ABLUnit code.
2. Implement a driver for each environment:
   • In OE, the driver would be 4GL code that runs a COMPILE to get the listing (this is the equivalent of the .cache file.
   • In FWD, the driver would be a Java program that runs the Preprocessor directly (which will generate the .cache file).
3. For each 4GL testcase, we write a harness test to:
   • call the correct driver depending on whether we are in OE or FWD
   • compare the listing/cache file output to previously captured baselines
4. Create a test plan that includes all the tests and which generates the JUnit XML output that reports the results.

Regarding this:

• for OE, I will implement a 4GL code that uses COMPILE on a set of files.
• for FWD, after discussing with Alex we've found com.goldencode.p2j.preproc.Preprocessor that is kind of but not really a driver, in the sense that it can be run from CLI but has no help or something like that. I can look into this class and make some improvements so I can use it in Harness testing

Sure. One of the things that should be done is to move the core processing out of the constructor. That would make it easier to understand and use. Other changes that make sense are welcome.

Now for the Harness part if someone has some documentation that would be helpful. For now I have a few tests that just compare the baseline with the .cache file. (text-file-comparison)

You've looked through Documentation?

Also, there are major updates to the harness in #9270. There are multi-session tests in Testcases which can be run using that version of the harness.

Tomasz: We need documentation updates for these new features and for understanding the techniques you are introducing like running both OE and FWD code from the same test plans.

But the cache file has been produced after running front part of conversion. From what i understood is that Harness tests should be able to run this Preprocessor driver and compare the .cache file with the baseline.

Yes. The cache file shouldn't be any different but there is no reason to run the entire front end just to test the preprocessor.

In OE I see that there is a HarnessApi and maybe I should use that?

I have no idea what that is.

I don't really know, but the test files in OE should run the procedure I will implement (that compiles the files) and compares the output with the already existing baseline (expected to be 100% passed here).

Exactly. The one issue here is the tilde bug in OE.

The general idea is good. I would probably just encode the parameters are extent variables and call GeneratePreprocOutput4GL.p in a loop.

Please do plan for the scenario that we need to be testing with inputs that have different source code character encodings AND different CPINTERNAL values in the session running the COMPILE. There is good discussion in both #4766 and #8027.

Greg Shah wrote:

The general idea is good. I would probably just encode the parameters are extent variables and call GeneratePreprocOutput4GL.p in a loop.

Please do plan for the scenario that we need to be testing with inputs that have different source code character encodings AND different CPINTERNAL values in the session running the COMPILE. There is good discussion in both #4766 and #8027.

A question about this, how would I call this Driver from harness .xml test?
With this abordation (inpout parameters), how would I pass those parameters to the procedure through cli?
I have found -param but this has to be accessed inside procedure with SESSION:PARAM, it's not passing input parameters.
Reference: https://community.progress.com/s/article/26806

A question about this, how would I call this Driver from harness .xml test?

It can be called with direct Java code (if there is Java logic that is best encoded in the test itself.

Or it can be run as a shell/command line program.

With this abordation (inpout parameters), how would I pass those parameters to the procedure through cli?
I have found -param but this has to be accessed inside procedure with SESSION:PARAM, it's not passing input parameters.
Reference: https://community.progress.com/s/article/26806

I'm not sure what you mean by abordation but parameters cannot be passed to the 4GL startup procedure (the procedure executed directly from the command line). As you note, you can pack some data into the -param and then access it via SESSION:PARAM.

There is no such thing as a parameter (input or otherwise) to a 4GL startup procedure.

Help me understand what you need to use input parameters to achieve.

In #9854-28 is the cli command for progress that i think i should use. It works for procedures without input parameters, but since you said there's no such thing for startup procedures, then my Driver should be updated, no?

It works for procedures without input parameters, but since you said there's no such thing for startup procedures, then my Driver should be updated, no?

Correct. You can have a startup program that reads SESSION:PARAM and translates that into parameters for GeneratePreprocOutput4GL.p which would be a "worker" program.

Or you can change GeneratePreprocOutput4GL.p to remove the parameter usage and have it read SESSION:PARAM to get its input.

Also, since the worker is not a class, let's use the lowercase naming approach that is more standard (tests/preproc/generate_preproc_output.p).

I've created the following ant task in build.xml:

     <target name="preproc-tests" depends="init">
       <java classname="com.goldencode.p2j.preproc.Preprocessor" 
             fork="true" 
             failonerror="true" 
             dir="${basedir}">
          <arg value="-out" />
          <arg value="${cvt.home}/${preproc_output}/${preproc_input}.${preproc_postfix}" />
          <arg value="${preproc_input}"/>
          <classpath refid="compile.classpath"/>
       </java>
     </target>

As an update:
Tested this on Linux and it works:
project-index.xml

<?xml version="1.0"?>
<project-index>
   <defaults>
      <attribute name="millis" value="10000" />
   </defaults>
   <paths>
      <test path="." />
      <test path="tests" />
      <test path="tests/preproc" />
      <test path="tests/preproc/tests" />
      <baseline path="tests/preproc/support/baseline" />
   </paths>
</project-index>

test-plan.xml

<?xml version="1.0"?>
<test-plan name="preproc_testing_plan" description="Automated preprocessor testing." >
   <project-index filename="tests/preproc/project-index.xml" />
   <output-directory path="results" unique="true" />
   <target type="stream" shell="bash" />
   <resources>
      <variable name="antTarget"       type="String" initial="preproc-tests" />
      <variable name="preprocInput"    type="String" initial="tests/preproc/support" />
      <variable name="preprocOutput"   type="String" initial="." />
      <variable name="preprocPostfix"  type="String" initial="preproc" />
   </resources>

   <test-set name="preproc-baseline-comparison-tests">
      <test filename="missing_equals_in_named_args.xml"/>
   </test-set>
</test-plan>

missing_equals_in_named_args.xml

<?xml version="1.0"?>
<test name="missing_equals_in_named_args" description="" >
   <stream-write value="ant %s -Dpreproc_input=%s/missing_equals_in_named_args.p -Dpreproc_output=%s -Dpreproc_postfix=%s" newline="true">
      <substitution variable="antTarget" />
      <substitution variable="preprocInput" />
      <substitution variable="preprocOutput" />
      <substitution variable="preprocPostfix" />
   </stream-write>
   <stream-wait-for value="BUILD SUCCESSFUL" newline="false" />
   <text-file-comparison baseline="missing_equals_in_named_args.p.preproc" actual="cvt/tests/preproc/support/missing_equals_in_named_args.p.preproc" />
</test>

Windows update:
I created a different testplan for windows test_plan_OE

<?xml version="1.0"?>
<test-plan name="preproc_test_plan" description="Atomated preprocessor testing." >
   <project-index filename="tests/preproc/project-index.xml" />
   <output-directory path="results" unique="true" />
   <target type="stream" shell="bash" />

   <test-set name="generate_cache_files">
      <test filename="missing_equals_in_named_args_OE.xml"/>
   </test-set>
</test-plan>

And the test:

<?xml version="1.0"?>
<test name="missing_equals_in_named_args" description="" >
   <stream-write value='pro -p tests/preproc/generate_preproc_output_worker.p -b -param "tests/preproc/support/missing_equals_in_named_args.p,cvt/tests/preproc/support,oe_preproc"' newline="true" />
   <stream-wait-for value="SUCCESS" newline="false" />
   <text-file-comparison baseline="missing_equals_in_named_args.p.preproc" actual="cvt/tests/preproc/support/missing_equals_in_named_args.p.oe_preproc" />
</test>

While debugging the FWD Preprocessor I have found that:
When preprocessing

   
&SCOPED-DEFINE nothing better than nothing
&GLOBAL-DEFINE something something else

&if false &then
   /* this can never be included since the expression is false */
&elseif defined(something) &then
   /* welcome to a preprocessed program */
&else
   /* this can never be included since the elseif will always be true */
&endif

         String[] list = env.getOpt().getProPath();

         for (int j = 0; j < list.length; j++)
         {

   /* welcome to a preprocessed program */

My question would be: should i include somewhere in Preprocessor.java the logic that would created the path to the preproc_output?

Yes, the idea seems reasonable though I wonder if it will change with #7180.

Right now as it stands, this ant command generates the files in @cvt/

@ but if the dir doesn't exist then the output is not directed to the a file.

I assume that we don't have this hard coded to @cvt/@.

Overall, this is all moving along nicely.

In regard to the different test plans/tests for OE vs FWD, please look into ways to unify these into a single version. Tomasz can help with that. If it is possible (I think it is) and the result is reasonable, then I'd prefer a single set of definitions to maintain instead of having the duplication.

Also, please don't use uppercase in any of the names except for class/interface filenames (e.g. test_plan_oe.xml instead of test_plan_OE.xml).

In OE the preprocessor IF would go on the 2nd statement, while in FWD it will go in the first. The reason being here:

com/goldencode/p2j/preproc/Preprocessor.java:850

This is a configuration issue (the preproc needs to be told about the propath). But I agree that we should provide a default of . if we haven't otherwise been configured with one. Please make the change.

Yes, it doesn't have to be hardcoded, it can be taken from p2j.cfg i think.
Also another issues I've found during preprocessing is that:
when a // comment is found, progress adds a NL. In FWD that doesn't happen.
When we have ;) progress translates that into just ), but in FWD it stays the same.
I guess i have to look into text.g, could you please confirm that?

When we have ;) progress translates that into just ), but in FWD it stays the same.
I guess i have to look into text.g, could you please confirm that?

See ClearStream and speak with Octavian.

Also another issues I've found during preprocessing is that: when a // comment is found, progress adds a NL. In FWD that doesn't happen.

Anything that is a bug should have a new task opened (Conversion Tools project).

Greg Shah wrote:

When we have ;) progress translates that into just ), but in FWD it stays the same.
I guess i have to look into text.g, could you please confirm that?

See ClearStream and speak with Octavian.

I created #9871. We had a look through the source code. Let me know if I should take this issue or what to do with it.

Let me know if I should take this issue or what to do with it.

Not yet. Let's get the test suite in place first. Then we can fix this and any other issues found from running the test suite.

I added all the tests I've done and also the OE driver on testcases rev 1735

To run the tests with FWD: (assuming we are at testcases root)
java -jar ../harness_9270a/build/lib/harness.jar tests/preproc/test_plan.xml
harness_9270a is the branch from #9270

To run the tests with OE:
java -jar ../harness_9270a/build/lib/harness.jar -shell cmd.exe tests/preproc/test_plan.xml

I created branch 6859a with revision 15847.

Have you migrated or created all tests noted in #6859-5?

All testcases from old_tescases are migrated with the harness counterpart on testcases/1735. Isn't that all?

Greg Shah wrote:

• Once those old testcases are migrated and working, we would evaluate the following to identify other tests that we need to add:
  • The feature list in Preprocessor
  • The preprocessor code itself, both in Preprocessor as well as in the two grammars (text.g and braces.g)
  • Preprocessor bug issues in Redmine

This still needs attention. The idea is to improve our coverage over the cases that were written in the old testcases project.

Okay, got it.

I have a question about how to handle the testing of ;'.
In the wiki mentioned above, it is stated that ;' converts to ', when in progress and in progress wiki it's stated to convert to `.
Which one is correct? Since the Preprocessor also converts to '.

Stefan Vieru wrote:

I have a question about how to handle the testing of ;'.
In the wiki mentioned above, it is stated that ;' converts to ', when in progress and in progress wiki it's stated to convert to `.
Which one is correct? Since the Preprocessor also converts to '.

Check it with test code on OE.

I created the following test:

&SCOPED-DEFINE text1 "test1" 
&GLOBAL-DEFINE text2 "test2" 

// these should convert
MESSAGE ;(&text1 ;).
MESSAGE ;(&text1;).

// theses shouldn't convert
MESSAGE ;( &text1 ;).
MESSAGE ;( &text1;).

// these should convert
MESSAGE ;(&text2 ;).
MESSAGE ;(&text2;).

// theses shouldn't convert
MESSAGE ;( &text2 ;).
MESSAGE ;( &text2;).

// existing import
// when trying with missing import it 
;(tests/preproc/support/keep_tildes.p;)

In OE it outputs to:

// these should convert

MESSAGE "test1".
MESSAGE "test1".

// theses shouldn't convert

MESSAGE .
MESSAGE .

// these should convert

MESSAGE "test2".
MESSAGE "test2".

// theses shouldn't convert

MESSAGE .
MESSAGE .

// existing import

// when trying with missing import it 

def var txt as char.
txt = "hel~nlo".
message txt.

Braces parsing failed!
java.lang.NullPointerException: Cannot read the array length because "rawResults" is null
// these should convert
MESSAGE "test1".
MESSAGE "test1".

// theses shouldn't convert
/
Warning [./tests/preproc/support/alternative_coding_quirk_07.p line 9, col 19]: Include file "&text1" not found.
/
Exception: antlr.TokenStreamIOException: Braces parsing failed!
            ClearStream state:  clearCount = 0; inComment = no; inString = no; keepTildes = no
Scanning:  
Scanning: LA(1)='.' LA(2)='
'
 scope 1 --- #0:0/{}{}
    fwd-version=(global,"FWD v4.0.0_undefined_undefined_15811")
    window-system=(builtin,"null")
    file-name=(builtin,"null")
    opsys=(builtin,"null")
    batch-mode=(builtin,"null")
    text2=(global,""test2"")
    line-number=(builtin,"null")
    process-architecture=(builtin,"null")
    sequence=(builtin,"null")
 scope 2 --- #0:0/{}{}
---  scope 3 --- #1:56./tests/preproc/support/alternative_coding_quirk_07.p{}{}
---     text1=(scoped,""test1"")
=================================
Preprocessor done.

Greg Shah wrote:

Stefan Vieru wrote:

I have a question about how to handle the testing of ;'.
In the wiki mentioned above, it is stated that ;' converts to ', when in progress and in progress wiki it's stated to convert to `.
Which one is correct? Since the Preprocessor also converts to '.

Check it with test code on OE.

Yes, i have a testcase for that and it converts to `

Open bugs as needed.

I created 6859b/15856
It was needed due to an issue when running Preprocessor standalone, it wouldn't be able to create necessary dirs and subdirs.

While developing some more tests i have found that:

txt1 = 'txt1';*.
txt2 = "txt2";*.
txt3 = 'txt1'~;*.
txt4 = "txt2"~;*.

In OE we get:

txt1 = 'txt1'*.
txt2 = "txt2"*.
txt3 = 'txt1'~;;*.
txt4 = "txt2"~;;*.

But in FWD:

txt1 = 'txt1'*.
txt2 = "txt2"*.
txt3 = 'txt1'~;*.
txt4 = "txt2"~;*.

• ;?n converts to ~nn (;? = ~)

• ~;* stays ~;*
• ~;& stays ~;&
• ~;< stays ~;<
• ~;> stays ~;>
• ~;% stays ~;%
• ~;?n stays ~;?n

It seems this is another special case of #6308. Does this only happen after a string literal?

Greg Shah wrote:

It seems this is another special case of #6308. Does this only happen after a string literal?

No, it happens regardless is after a string literal or not.

OK, then open a new task for this additional "quirk". Make sure we have tests that explore these cases. You will have to expand the tests past those in #6859-66, since those are all about following a string literal.

I created the following small sample:

// the alternative coding construct should not convert in string
txt = ";&".
txt = ';&'.

/*the ; from the alternative coding that is placed after a string
should be consumed */
txt = 'txt';&GLOBAL-DEFINE arg01 1.
{tests/preproc/support/check_global_defined.i arg01}
txt = "txt";&GLOBAL-DEFINE arg02 1.
{tests/preproc/support/check_global_defined.i arg02}

txt = 'txt';&SCOPED-DEFINE arg03 1.
{tests/preproc/support/check_scope_defined.i arg03}
txt = "txt";&SCOPED-DEFINE arg04 1.
{tests/preproc/support/check_scope_defined.i arg04}

/*tilde should duplicate ; and stop the alternative coding from being processed*/
txt = 'txt'~;&GLOBAL-DEFINE arg05 1.
{tests/preproc/support/check_undefined.i arg05}
txt = "txt"~;&GLOBAL-DEFINE arg06 1.
{tests/preproc/support/check_undefined.i arg06}

txt = 'txt'~;&SCOPED-DEFINE arg07 1.
{tests/preproc/support/check_undefined.i arg07}
txt = "txt"~;&SCOPED-DEFINE arg08 1.

txt = ";&".
txt = ';&'.

/*the ; from the alternative coding that is placed after a string
should be consumed */
txt = 'txt'
   "arg01 defined globally".

txt = "txt" 
   "arg02 defined globally".

txt = 'txt'
   "arg03 defined in scope".

txt = "txt" 
   "arg04 defined in scope".

/*tilde should duplicate ; and stop the alternative coding from being processed*/
txt = 'txt'~;;&GLOBAL-DEFINE arg05 1.

   "arg05 is not defined".

txt = "txt"~;;&GLOBAL-DEFINE arg06 1.

   "arg06 is not defined".

txt = 'txt'~;;&SCOPED-DEFINE arg07 1.

   "arg07 is not defined".

txt = "txt"~;;&SCOPED-DEFINE arg08 1.

   "arg08 is not defined".

txt = ~;&SCOPED-DEFINE arg09 1"txt"~;&SCOPED-DEFINE arg10 1.

Strange behavior is found when preprocessing this:

~;&GLOBAL-DEFINE abb1 1
{tests/preproc/support/check_defined.i abb1}
~;&GLOBAL-DEFINE abb2 1
{tests/preproc/support/check_defined.i abb2}
~;&GLOBAL-DEFINE abb3 1
{tests/preproc/support/check_defined.i abb3}
~;&GLOBAL-DEFINE abb4 1
{tests/preproc/support/check_defined.i abb4}
~;&GLOBAL-DEFINE abb5 1
{tests/preproc/support/check_defined.i abb5}
~;&GLOBAL-DEFINE abb6 1
{tests/preproc/support/check_defined.i abb6}
~;&GLOBAL-DEFINE abb7 1
{tests/preproc/support/check_defined.i abb7}
~;&GLOBAL-DEFINE abb8 1
{tests/preproc/support/check_defined.i abb8}
~;&GLOBAL-DEFINE abb9 1
{tests/preproc/support/check_defined.i abb9}
~;&GLOBAL-DEFINE abb10 1
{tests/preproc/support/check_defined.i abb10}
~;&GLOBAL-DEFINE abb11 1
{tests/preproc/support/check_defined.i abb11}
~;&GLOBAL-DEFINE abb12 1
{tests/preproc/support/check_defined.i abb12}
~;&GLOBAL-DEFINE abb13 1
{tests/preproc/support/check_defined.i abb13}
~;&GLOBAL-DEFINE abb14 1
{tests/preproc/support/check_defined.i abb14}
~;&GLOBAL-DEFINE abb15 1
{tests/preproc/support/check_defined.i abb15}
~;&GLOBAL-DEFINE abb16 1
{tests/preproc/support/check_defined.i abb16}
~;&GLOBAL-DEFINE abb17 1
{tests/preproc/support/check_defined.i abb17}
~;&GLOBAL-DEFINE abb18 1
{tests/preproc/support/check_defined.i abb18}

Are you talking about OE or FWD?

Greg Shah wrote:

Are you talking about OE or FWD?

Stefan is no longer at the office, but I can answer this. It is in OE. Stefan is designing unit tests in OE, but after using COMPILE with the code in #6859-71, the preprocessed code was simply halted after the 10th define. Maybe he can provide more details.

Sounds like an OE bug which we won't reproduce. We should keep some record of this, but the OE tests will have to be split up to avoid the bug.

• tilde + alternative coding converts in // comment
• ";( doesn't process into "{, same for ' (documented a bit in #6308-25)

I have updated 6859b with r15982
The change contains the new flags for each of the comment and string type.
Greg, could you please take a look?

1. 6859b - updated
2. 9886a - needs a rebase, everything else is fine
3. 9905a - needs a rebase, everything else is fine
4. 6308a - updated
5. 9918a

I am now working towards incrementally updating the branches where there are no dependencies, so I started off with 6859b.

I updated testcases with the latest tests, also moved them to tests/conversion/preprocessor.
Will have to adapt FWD driver with the possibility of executing it on a directory, right now it only works with a filename as a parameter. This is proven a must for correct jacoco coverage.

• added possibility of running the FWD driver with a directory as parameter
• added a regex to differentiate between files and directories
• move the preprocessing logic in preprocess method
• update the help

While analyzing conversion logs with this changes I cam across an issue and created the following testcase:

testcasetestcase

DEFINE VARIABLE i               AS INTEGER    NO-UNDO.
DEFINE VARIABLE list            AS CHARACTER  NO-UNDO.

list = "3456789".

&SCOPED-DEFINE nr 5

&SCOPED-DEFINE embed &IF {&nr} < ~{&num~} &THEN~
 WHEN ~{&num~} THEN message "{&nr} < ~{&num}".~
 &ENDIF

DO i = 1 TO NUM-ENTRIES(list):
   CASE i:
      &SCOPED-DEFINE num 1
      {&embed} 
      &SCOPED-DEFINE num 2
      {&embed} 
      &SCOPED-DEFINE num 3
      {&embed} 
      &SCOPED-DEFINE num 4
      {&embed} 
      &SCOPED-DEFINE num 5
      {&embed} 
      &SCOPED-DEFINE num 6
      {&embed} 
      &SCOPED-DEFINE num 7
      {&embed} 
      &SCOPED-DEFINE num 8
      {&embed} 
   END CASE.
END.

With my changes I did get an error, the &THEN and WHEN from the SCOPED-DEFINE would just become &THENWHEN after consuming all the WS after &THEN. I reverted this change and kept all WS after &THEN.
Now after reverting this I have the same error as on trunk during the execution of FWD driver:

errorerror

DEFINE VARIABLE iSDOLoop            AS INTEGER    NO-UNDO.
DEFINE VARIABLE cSDOList            AS CHARACTER  NO-UNDO.
DEFINE VARIABLE hSDOHandle          AS HANDLE     NO-UNDO.
DEFINE VARIABLE lStatic             AS LOGICAL    NO-UNDO.

cSDOList = "7896".
lStatic = true.

DO iSDOLoop = 1 TO NUM-ENTRIES(cSDOList):
  hSDOHandle = WIDGET-HANDLE(ENTRY(iSDOLoop,cSDOList)).
  IF lStatic THEN
  DO:
   CASE iSDOLoop:

   END CASE.
  END.
END.
/
Warning [./abl/test.p line 19, col 29]: Preprocessor evaluation of expression '2 < ~{&num~}' failed, returning false.
Warning [./abl/test.p line 21, col 29]: Preprocessor evaluation of expression '2 < ~{&num~}' failed, returning false.
Warning [./abl/test.p line 23, col 29]: Preprocessor evaluation of expression '2 < ~{&num~}' failed, returning false.
Warning [./abl/test.p line 25, col 29]: Preprocessor evaluation of expression '2 < ~{&num~}' failed, returning false.
Warning [./abl/test.p line 27, col 29]: Preprocessor evaluation of expression '2 < ~{&num~}' failed, returning false.
Warning [./abl/test.p line 29, col 29]: Preprocessor evaluation of expression '2 < ~{&num~}' failed, returning false.
Warning [./abl/test.p line 31, col 29]: Preprocessor evaluation of expression '2 < ~{&num~}' failed, returning false.
Warning [./abl/test.p line 33, col 29]: Preprocessor evaluation of expression '2 < ~{&num~}' failed, returning false.
Preprocessor done.

Now, if we run convert.front on this, we get the following cache filecache file

DEFINE VARIABLE iSDOLoop            AS INTEGER    NO-UNDO.
DEFINE VARIABLE cSDOList            AS CHARACTER  NO-UNDO.
DEFINE VARIABLE hSDOHandle          AS HANDLE     NO-UNDO.
DEFINE VARIABLE lStatic             AS LOGICAL    NO-UNDO.

cSDOList = "7896".
lStatic = true.

DO iSDOLoop = 1 TO NUM-ENTRIES(cSDOList):
  hSDOHandle = WIDGET-HANDLE(ENTRY(iSDOLoop,cSDOList)).
  IF lStatic THEN
  DO:
   CASE iSDOLoop:

             WHEN 3 THEN message 2 3.  
             WHEN 4 THEN message 2 4.  
             WHEN 5 THEN message 2 5.  
             WHEN 6 THEN message 2 6.  
             WHEN 7 THEN message 2 7.  
             WHEN 8 THEN message 2 8.  
   END CASE.
  END.
END.

Greg, is this worth investigating further?

I've rebased 6859b and created r16036.
I committed all the changes from preprocessor bug-fix branches (except #10099) and now I am testing a customer application.

Code Review Task Branch 6859b Revisions 16029 through 16036

These changes will need to be tested with the full range of customer applications.

1. In ClearStream, ;' now returns ` (backtick) instead of ' (single quote). That is different than the 4GL docs state. Are you sure this is correct?

2. In ClearStream.read() you added this:

                     /*
                     here we have an alternative comment which is ~, and such we have to process it
                     having ~n or any char
                     */

What is an "alternative comment"? Also, this should be using // not /* */.

3. The alternative coding changes can output the same alternative coding multiples times when the character is ~. That code needs comments to explain why that is correct. Also, it is not clear if that is needed for keep-tildes mode or not. If it is specific to that behavior, then we need to limit it to when that flag is on.

4. I'm worried that the translation of \r to \n is not correct. I understand that we would have to make changes to the Progress lexer but I think we should allow \r to flow through. We saw a bug recently where the 4GL code expected that character and we silently "ate it".

5. The header is mis-formatted in Environment:

** 020 CA  20201015          Replaced java.util.Stack with ArrayDeque (as synchronization is not required).
*/
/*
** 021 SV  20250410          Add the needsNewLine flag. Used in case of double slash comments. ref 9886
**     SV  20250508          Add isAfterTilde flag.
**     SV  20250520          Remove the flag and add tildeOffset, used for counting the remaining chars 
**                           after a tilde that are to be returned without processing.
**     SV  20250616          Move towards using the different flags for each type of comment and string
*/

This is also a problem in Preprocessor:

** 042 FER 20250408          Added an if to check the printToStdout flag in Preprocessor() before logging errors.
*/
/*
** 042 SV  20250414          Added the possibility of creating the dirs necessary to output file. (ref #6859)
**     SV  20250620          Created the possibility of running the driver with a directory as a parameter
*/

And in text.g:

** 034 CA  20230727          ScopedSymbolDictionary now uses a CaseInsensitiveHashMap for case-insensitive
**                           dictionaries, so 'processKey' is no longer needed.
*/
/*
** 035 SV  20250411          Set a new flag that means a NL must be appended after a slash_slash token.
**     SV  20250414          Added a consume in the case we have a alternative coding construct
**                           right after a string.
**     SV  20250508          Process ASTMT based on isAfterTilde value.
**     SV  20250512          Updated this with a condition to consume '& '.
**     SV  20250520          Add tildeOffset check for ASTMT.
**     SV  20250528          Move the flag at the beggining of the SLASH_SLASH
**     SV  20250603          Separated the comment and string flags into separate ones depending on the
**                           type of comment and string respectively.
**     SV  20250616          Process STAR_COMMENTs that are placed after preprocessor directives
**                           Consume the first WS/NL after an UNDEFINE statement
**     SV  20250703          Fix an issue when having the flag set and reaching EOF.
*/

6. In Preprocessor, please use a static member and pre-compile the String regex = ".*\\.(p|P|w|W|cls)$";.

7. Line 1633 in text.g has $getText.equals("&THEN") || $getText.equals("&ENDIF" which is not a good way to match things. For example, the text doesn't have to match case. We've already matched the token, so use th etoken type. In the previous action, we have newType which can be hoised in scope and compared with ATHEN or AENDIF token types.

1. OE preprocessing does this, in documentation yes, it is said to be ;' -> ', but in OE ;' -> `, unless I'm doing something wrong.
2. meant alternative coding, I'll fix this
5. I usually update the history entries at the end in the case I have to rebase (read recently through some wiki about this approach).
7. I checked now, will remove the text part and check for the newType, I just moved this here but previously it was before newType was assigned.

Updated points 7 and 2 in r16037.

I've updated the Preprocessor class with the compiled regex.
r16038

Greg Shah wrote:

Code Review Task Branch 6859b Revisions 16029 through 16036

3. The alternative coding changes can output the same alternative coding multiples times when the character is ~. That code needs comments to explain why that is correct. Also, it is not clear if that is needed for keep-tildes mode or not. If it is specific to that behavior, then we need to limit it to when that flag is on.

Could you please elaborate this point? I think I might be missing something.

Greg Shah wrote:

Code Review Task Branch 6859b Revisions 16029 through 16036

4. I'm worried that the translation of \r to \n is not correct. I understand that we would have to make changes to the Progress lexer but I think we should allow \r to flow through. We saw a bug recently where the 4GL code expected that character and we silently "ate it".

I'll remove this part then.

Stefan Vieru wrote:

Greg Shah wrote:

Code Review Task Branch 6859b Revisions 16029 through 16036

3. The alternative coding changes can output the same alternative coding multiples times when the character is ~. That code needs comments to explain why that is correct. Also, it is not clear if that is needed for keep-tildes mode or not. If it is specific to that behavior, then we need to limit it to when that flag is on.

Could you please elaborate this point? I think I might be missing something.

If you're referring to the case where:
in OE

;?n
;?.

Preprocesses to

~nn
~..

This should only be in the case of keepTildes? If so, without tildes how would it look like? Would the files with keepTildes false will ultimately be the .cache files?

Stefan Vieru wrote:

Greg Shah wrote:

Code Review Task Branch 6859b Revisions 16029 through 16036

3. The alternative coding changes can output the same alternative coding multiples times when the character is ~. That code needs comments to explain why that is correct. Also, it is not clear if that is needed for keep-tildes mode or not. If it is specific to that behavior, then we need to limit it to when that flag is on.

Could you please elaborate this point? I think I might be missing something.

I'm talking about this code:

         if (nextChar == ';')
         {
            if (notInStrings)
            {
               int pastNextChar = mread();
               if (isAlternativeCoding(nextChar, pastNextChar))
               {
                  int alternativeCoding = getAlternativeCoding(pastNextChar);

                  if (alternativeCoding == '~' && !isInComment())
                  {
                     /*
                     here we have an alternative comment which is ~, and such we have to process it
                     having ~n or any char
                     */
                     pastNextChar = mread();
                     if (pastNextChar == '~')
                     {
                        super.unread(alternativeCoding);
                        clearCount++;
                     }
                     else
                     {
                        super.unread(pastNextChar);
                     }
                     super.unread(alternativeCoding);
                     continue;
                  }

                  super.unread(alternativeCoding);
                  if (!(alternativeCoding == '{'))
                  {
                     // the alternative codings ;), ;( that convert into { and } are needed to be interpreted
                     // the rest should just be printed
                     clearCount++;
                  }
                  // reread the modified stream
                  continue;
               }
               else
               {
                  super.unread(pastNextChar);
                  return nextChar;
               }
            }
            else
               return nextChar;
         }

Notice how if alternativeCoding == '~' we can unread alternativeCoding multiple times.

Stefan Vieru wrote:

Greg Shah wrote:

Code Review Task Branch 6859b Revisions 16029 through 16036

4. I'm worried that the translation of \r to \n is not correct. I understand that we would have to make changes to the Progress lexer but I think we should allow \r to flow through. We saw a bug recently where the 4GL code expected that character and we silently "ate it".

I'll remove this part then.

We probably should not, at this time.

Doing so will require more development and we probably should defer that work to another task. Please speak with Octavian and Ovidiu. I think we hit a problem recently related to \r but I don't recall the task number.

We should add a task to implement support for \r to the same level as is available in the 4GL. Then link it here and to the other task that I can't recall.

Finished the code review changes.
Tested two customer application with minimal conversion output changes.
Will work towards documenting the changes I've done and create a commit.

Code Review Task Branch 6859b Revision 16040:

I've analyzed each individual task. Everything looks good, and I have no objections.

During compiling the following appears:

    [exec] > Task :ant-chk_antlr_text
     [exec] [ant:antlr] ANTLR Parser Generator   Version 2.7.7 (20060906)   1989-2005
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:925: warning:nondeterminism upon
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:925:     k==1:WS
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:925:     k==2:EOF,TAB,WS,COMMENT,COMM_OPEN,CODE,COMM_CLOSE,LBRACE,RBRACE,APOST,QUOTE,XAPOST,XQUOTE,AGLOBAL,ASCOPED,AMESSAGE,AUNDEFINE,AIF,ATHEN,AELSEIF,AELSE,AENDIF,ASUSPEND,ARESUME,ASTMT,NL,STAR,AMPER,PPNAME,ALPHA,SPECIAL,DIGIT,EQUALS,ASTRING,QSTRING,STRING,STAR_COMMENT,SLASH_SLASH,BREAK_CODE_CHUNKS
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:925:     between alt 1 and exit branch of block
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:926: warning:nondeterminism upon
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:926:     k==1:STAR_COMMENT
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:926:     k==2:EOF,TAB,WS,COMMENT,COMM_OPEN,CODE,COMM_CLOSE,LBRACE,RBRACE,APOST,QUOTE,XAPOST,XQUOTE,AGLOBAL,ASCOPED,AMESSAGE,AUNDEFINE,AIF,ATHEN,AELSEIF,AELSE,AENDIF,ASUSPEND,ARESUME,ASTMT,NL,STAR,AMPER,PPNAME,ALPHA,SPECIAL,DIGIT,EQUALS,ASTRING,QSTRING,STRING,STAR_COMMENT,SLASH_SLASH,BREAK_CODE_CHUNKS
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:926:     between alt 1 and exit branch of block
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:945: warning:nondeterminism upon
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:945:     k==1:WS
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:945:     k==2:TAB,WS,COMMENT,COMM_OPEN,CODE,COMM_CLOSE,LBRACE,RBRACE,APOST,QUOTE,XAPOST,XQUOTE,AGLOBAL,ASCOPED,AMESSAGE,AUNDEFINE,AIF,AELSEIF,AELSE,AENDIF,ASUSPEND,ARESUME,ASTMT,NL,STAR,AMPER,PPNAME,ALPHA,SPECIAL,DIGIT,EQUALS,ASTRING,QSTRING,STRING,STAR_COMMENT,SLASH_SLASH,BREAK_CODE_CHUNKS
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:945:     between alt 1 and exit branch of block
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:946: warning:nondeterminism upon
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:946:     k==1:STAR_COMMENT
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:946:     k==2:TAB,WS,COMMENT,COMM_OPEN,CODE,COMM_CLOSE,LBRACE,RBRACE,APOST,QUOTE,XAPOST,XQUOTE,AGLOBAL,ASCOPED,AMESSAGE,AUNDEFINE,AIF,AELSEIF,AELSE,AENDIF,ASUSPEND,ARESUME,ASTMT,NL,STAR,AMPER,PPNAME,ALPHA,SPECIAL,DIGIT,EQUALS,ASTRING,QSTRING,STRING,STAR_COMMENT,SLASH_SLASH,BREAK_CODE_CHUNKS
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:946:     between alt 1 and exit branch of block
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1000: warning:nondeterminism upon
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1000:     k==1:WS
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1000:     k==2:EOF,TAB,WS,COMMENT,COMM_OPEN,CODE,COMM_CLOSE,LBRACE,RBRACE,APOST,QUOTE,XAPOST,XQUOTE,AGLOBAL,ASCOPED,AMESSAGE,AUNDEFINE,AIF,ATHEN,AELSEIF,AELSE,AENDIF,ASUSPEND,ARESUME,ASTMT,NL,STAR,AMPER,PPNAME,ALPHA,SPECIAL,DIGIT,EQUALS,ASTRING,QSTRING,STRING,STAR_COMMENT,SLASH_SLASH,BREAK_CODE_CHUNKS
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1000:     between alt 1 and exit branch of block
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1001: warning:nondeterminism upon
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1001:     k==1:STAR_COMMENT
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1001:     k==2:EOF,TAB,WS,COMMENT,COMM_OPEN,CODE,COMM_CLOSE,LBRACE,RBRACE,APOST,QUOTE,XAPOST,XQUOTE,AGLOBAL,ASCOPED,AMESSAGE,AUNDEFINE,AIF,ATHEN,AELSEIF,AELSE,AENDIF,ASUSPEND,ARESUME,ASTMT,NL,STAR,AMPER,PPNAME,ALPHA,SPECIAL,DIGIT,EQUALS,ASTRING,QSTRING,STRING,STAR_COMMENT,SLASH_SLASH,BREAK_CODE_CHUNKS
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1001:     between alt 1 and exit branch of block
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1033: warning:nondeterminism upon
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1033:     k==1:WS
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1033:     k==2:EOF,TAB,WS,COMMENT,COMM_OPEN,CODE,COMM_CLOSE,LBRACE,RBRACE,APOST,QUOTE,XAPOST,XQUOTE,AGLOBAL,ASCOPED,AMESSAGE,AUNDEFINE,AIF,ATHEN,AELSEIF,AELSE,AENDIF,ASUSPEND,ARESUME,ASTMT,NL,STAR,AMPER,PPNAME,ALPHA,SPECIAL,DIGIT,EQUALS,ASTRING,QSTRING,STRING,STAR_COMMENT,SLASH_SLASH,BREAK_CODE_CHUNKS
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1033:     between alt 1 and exit branch of block
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1034: warning:nondeterminism upon
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1034:     k==1:STAR_COMMENT
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1034:     k==2:EOF,TAB,WS,COMMENT,COMM_OPEN,CODE,COMM_CLOSE,LBRACE,RBRACE,APOST,QUOTE,XAPOST,XQUOTE,AGLOBAL,ASCOPED,AMESSAGE,AUNDEFINE,AIF,ATHEN,AELSEIF,AELSE,AENDIF,ASUSPEND,ARESUME,ASTMT,NL,STAR,AMPER,PPNAME,ALPHA,SPECIAL,DIGIT,EQUALS,ASTRING,QSTRING,STRING,STAR_COMMENT,SLASH_SLASH,BREAK_CODE_CHUNKS
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1034:     between alt 1 and exit branch of block
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1055: warning:nondeterminism upon
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1055:     k==1:WS
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1055:     k==2:EOF,TAB,WS,COMMENT,COMM_OPEN,CODE,COMM_CLOSE,LBRACE,RBRACE,APOST,QUOTE,XAPOST,XQUOTE,AGLOBAL,ASCOPED,AMESSAGE,AUNDEFINE,AIF,ATHEN,AELSEIF,AELSE,AENDIF,ASUSPEND,ARESUME,ASTMT,NL,STAR,AMPER,PPNAME,ALPHA,SPECIAL,DIGIT,EQUALS,ASTRING,QSTRING,STRING,STAR_COMMENT,SLASH_SLASH,BREAK_CODE_CHUNKS
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1055:     between alt 1 and exit branch of block
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1056: warning:nondeterminism upon
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1056:     k==1:STAR_COMMENT
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1056:     k==2:EOF,TAB,WS,COMMENT,COMM_OPEN,CODE,COMM_CLOSE,LBRACE,RBRACE,APOST,QUOTE,XAPOST,XQUOTE,AGLOBAL,ASCOPED,AMESSAGE,AUNDEFINE,AIF,ATHEN,AELSEIF,AELSE,AENDIF,ASUSPEND,ARESUME,ASTMT,NL,STAR,AMPER,PPNAME,ALPHA,SPECIAL,DIGIT,EQUALS,ASTRING,QSTRING,STRING,STAR_COMMENT,SLASH_SLASH,BREAK_CODE_CHUNKS
     [exec] [ant:antlr] /home/sv/gcd/branches/6859b/src/com/goldencode/p2j/preproc/text.g:1056:     between alt 1 and exit branch of block

In each of these cases, you'll have to change this:

      (WS)*
      (
         ctmp0:STAR_COMMENT { env.print(ctmp0.getText()); }
      )*
<pre>

Into this:

</pre>
      (
         options { generateAmbigWarnings=false; }
         :
         WS
      )*
      (
         options { generateAmbigWarnings=false; }
         :
         ctmp0:STAR_COMMENT { env.print(ctmp0.getText()); }
      )*

It should be safe to do that for these cases.

I've updated this bit in r16042.

We need to write documentation for this test suite in Preprocessor Testcases.

There's testcases revisions 1735 and 1780 which I will have to look into and document. Alexandru also mentioned that I should revert the changes related to #9903 from 6859b to allow #10109 to continue being worked on.

Dănuț Filimon wrote:

Alexandru also mentioned that I should revert the changes related to #9903 from 6859b to allow #10109 to continue being worked on.

I can't find what revision was committed in 6859b for #9903, there are no references task numbers and no notes mentioning the specific revision(s) for this issue. If I were to take it by the changes, I think it is revision 16141 and 16142.

Preprocessor tests status:

ddf@ddfmachine:~/gcd/testcases$ java -cp ../harness/build/lib/harness.jar com.goldencode.harness.Harness tests/conversion/preprocessor/test_plan.xml 
2025-09-16 14:00:06 INFO    Harness  Start 
2025-09-16 14:00:06 INFO    Harness  Running test plan...  
2025-09-16 14:00:06 INFO    TestPlan STARTSET|preproc_test_plan/PRE_CONDITION 
2025-09-16 14:00:06 INFO    TestPlan ENDSET|preproc_test_plan/PRE_CONDITION 
2025-09-16 14:00:06 INFO    TestPlan STARTSET|preproc_test_plan/NONE 
2025-09-16 14:02:16 INFO    TestPlan ENDSET|preproc_test_plan/NONE 
2025-09-16 14:02:16 INFO    TestPlan STARTSET|preproc_test_plan/POST_CONDITION 
2025-09-16 14:02:16 INFO    TestPlan ENDSET|preproc_test_plan/POST_CONDITION 
2025-09-16 14:02:16 INFO    Harness  Completed in 2 minutes and 10.222 seconds. Status = FAILED.
Generating HTML reports...  
2025-09-16 14:02:16 INFO    Harness  Reports generation Completed in 0.086 seconds. 
2025-09-16 14:02:16 INFO    Harness  End 

The failures:

1. 1. alternative_coding_test_set: alternative_coding_quirk_06: failure in step 4: 'Line size mismatch (line # = 42, base = 18, actual = 25).'
2. preproc_test_set: 1. missing_include_file: failure in step 4: 'Unexpected EOF in baseline at line # 0.'
2. preprocessor_name_references_01: failure in step 4: 'Mismatched data at absolute row 3, relative row -1, page # -1, page size -1, last page false, column index 12. Expected 'r' (0x72) and found 'u' (0x75).'
3. shell: failure in step 4: 'Line size mismatch (line # = 0, base = 26, actual = 31).'
4. tildes_in_preproc_defines: failure in step 4: 'Line size mismatch (line # = 18, base = 36, actual = 38).'
3. unicode_test_set: control_characters: failure in step 4: 'Line size mismatch (line # = 5, base = 12, actual = 20).'alternative_coding_test_set: alternative_coding_quirk_06: failure in step 4: 'Line size mismatch (line # = 42, base = 18, actual = 25).'
2. preproc_test_set: 1. missing_include_file: failure in step 4: 'Unexpected EOF in baseline at line # 0.'
2. preprocessor_name_references_01: failure in step 4: 'Mismatched data at absolute row 3, relative row -1, page # -1, page size -1, last page false, column index 12. Expected 'r' (0x72) and found 'u' (0x75).'
3. shell: failure in step 4: 'Line size mismatch (line # = 0, base = 26, actual = 31).'
4. tildes_in_preproc_defines: failure in step 4: 'Line size mismatch (line # = 18, base = 36, actual = 38).'
3. unicode_test_set: control_characters: failure in step 4: 'Line size mismatch (line # = 5, base = 12, actual = 20).'

I found that 6859b/16136 & 16137 are the ones that fix the #9903 problem, but also fix a number of other tests.

Because the number of tests failing skyrocket when removing the revisions for #9903, I will leave 6859b intact. I will take the branch and run a new conversion with a large customer application and wrap up the regression test plan.

Dănuț Filimon wrote:

Because the number of tests failing skyrocket when removing the revisions for #9903, I will leave 6859b intact. I will take the branch and run a new conversion with a large customer application and wrap up the regression test plan.

The customer conversion is done, I did not expect that kind of change from 6859b. I've posted more in #10262-31.

I updated https://proj.goldencode.com/projects/p2j/wiki/Preprocessor_Testcases#Current-Status and https://proj.goldencode.com/projects/p2j/wiki/Preprocessor_Testcases#Running-in-FWD

I am investigating #10262-31 and I am also looking into expanding the test cases where possible.

Dănuț Filimon wrote:

I am investigating #10262-31 and I am also looking into expanding the test cases where possible.

Yes, please ensure that both the #10262-31 and the #10109 case are represented by preproc testcases. Optimally, these testcases would not require E4GL (in the case of #10109).

Greg Shah wrote:

Dănuț Filimon wrote:

I am investigating #10262-31 and I am also looking into expanding the test cases where possible.

Yes, please ensure that both the #10262-31 and the #10109 case are represented by preproc testcases. Optimally, these testcases would not require E4GL (in the case of #10109).

I am not sure how to write a preprocessor test for #10262-31 since the conversion output must be compared (java files).

Dănuț Filimon wrote:

Greg Shah wrote:

Dănuț Filimon wrote:

I am investigating #10262-31 and I am also looking into expanding the test cases where possible.

Yes, please ensure that both the #10262-31 and the #10109 case are represented by preproc testcases. Optimally, these testcases would not require E4GL (in the case of #10109).

I am not sure how to write a preprocessor test for #10262-31 since the conversion output must be compared (java files).

You're right, #10262-31 is not a preprocessor issue.

I wrote the test for #10109 + semicolon tests, can I commit them to testcases? As for the AdaptiveFind scenario, I could not reproduce it.

Yes, you can add them.

Committed testcases/1811. Added semicolon preprocessor tests.

I need clarification. There are two separate targets responsible with preprocessing, one is with keeptildes option and one without it. Each generates .keeptildes.preproc/.preproc file for a certain .p test file. But when I debug the preproc-tests-unix-with-tildes target for example, the debugger catch a breakpoint and I see that .preproc file is empty, not the .keeptildes.preproc file. It seems like each target generates the wrong output file. I assumed that preproc-tests-unix-with-tildes is responsible generating .keeptildes.preproc file, isn't that right?

From the build.xml:

    <target name="preproc-tests-unix-with-tildes" if="isUnix" depends="init" >
      <echo message="Running preproc-tests-unix" />
      <java classname="com.goldencode.p2j.preproc.Preprocessor" 
            fork="true" 
            failonerror="false" 
            dir="${basedir}">
         <arg value="-unixescapes" />
         <arg value="-keeptildes" />
         <arg value="-out" />
         <arg value="${cvt.home}/${preproc_input}.${preproc_postfix}" />
         <arg value="${preproc_input}"/>
         <classpath refid="compile.classpath"/>
      </java>
    </target>

    <target name="preproc-tests-unix-no-tildes" if="isUnix" depends="init" >
      <echo message="Running preproc-tests-unix" />
      <java classname="com.goldencode.p2j.preproc.Preprocessor" 
            fork="true" 
            failonerror="false" 
            dir="${basedir}">
         <arg value="-unixescapes" />
         <arg value="-out" />
         <arg value="${cvt.home}/${preproc_input}.keeptildes.${preproc_postfix}" />
         <arg value="${preproc_input}"/>
         <classpath refid="compile.classpath"/>
      </java>
    </target>

    <target name="preproc-tests-unix" if="isUnix" depends="preproc-tests-unix-no-tildes, preproc-tests-unix-with-tildes" />

preproc-tests-unix-with-tildes does indeed include the -keeptildes command line option and preproc-tests-unix-no-tildes does not have the -keeptildes.

However, you have indeed found a bug in the build script. The output filename that includes .keeptildes. should be preproc-tests-unix-with-tildes but it is actually used for preproc-tests-unix-no-tildes. Please just reverse them.

And we should compare the OE output with the .keeptildes. file, right? In the current targets, we compare with the .preproc file. That's why the current state is working. Because the file names are switched.

Octavian Adrian Gavril wrote:

And we should compare the OE output with the .keeptildes. file, right?

Yes, that is correct. The OE output can only match with the keeptildes output because of a bug in the 4GL preproc listing.

I've committed the changes into testcases/1818. Reversed output filenames across preprocessor targets.

I tested the latest changes of 6859b with ChUI regression testing application. I run the frontend and there were some cache file differences so I continued with full conversion. There are some source differences due to other conversion changes that were added in trunk.

Therefore, I'm planning a rebase for 6859b in order to get clearer results.

Rebased branch 6859b with trunk/rev16310. The new revision is 16326.

I've committed new semicolon preprocessor testcases into testcases/1820. These are related to #10569-45.

I updated the current status of the preprocessor test in order to include the new test suite. Preprocessor_Testcases

What is the status of retesting 6859b?

Greg Shah wrote:

What is the status of retesting 6859b?

If I recall correctly, #10569 needs to be reviewed first. The changes are in 6859b/16327.

Octavian, please check #6506-159 for another test that can be added to the test suite (assuming it doesn't just fall into the same category as the other one presented earlier).

Radu Apetrii wrote:

Octavian, please check #6506-159 for another test that can be added to the test suite (assuming it doesn't just fall into the same category as the other one presented earlier).

Thank you, Radu! I will include #6506-159 in the test suite as well.

Rebased branch 6859b with trunk/16435. The new revision is 16453.

• an OO application;
• two large GUI applications;
• a multi-tenant applications.

I've committed string concatenation tests in testcases/1828. These are in preproc_test_set. They are related to #10569, #6506-156 and #6506-159 + other variants that use mixed " and '. The only difference is that tilde is duplicating the next char:

--- /testcases/tests/conversion/preprocessor/support/baseline/string_concatenation.p.preproc    2026-02-23 14:30:46.424783852 +0200
+++ /testcases/cvt/tests/conversion/preprocessor/support/string_concatenation.p.keeptildes.preproc    2026-02-23 15:23:08.333224506 +0200
@@ -25,7 +25,7 @@
 MESSAGE "5. Mixed/Varied-Space:" a VIEW-AS ALERT-BOX.

 /* 6. Extreme Mixed / No Spacing (Lexer Stress) */
-a = '|'+"A"+"|"+'~nn'.
+a = '|'+"A"+"|"+'~n'.
 MESSAGE "6. Mixed/No-Space:" a VIEW-AS ALERT-BOX.

 /* 7. The Alternator (Switching delimiters with wide gaps) */
@@ -50,7 +50,7 @@
 MESSAGE "11. Format Mixed/Mixed-Space:" a VIEW-AS ALERT-BOX.

 /* 12. Format Tight Mixed (Lexer Stress) */
-a=string(9,">>>>9")+'|'+b+'~nn'.
+a=string(9,">>>>9")+'|'+b+'~n'.
 MESSAGE "12. Format Tight Mixed:" a VIEW-AS ALERT-BOX.

@@ -59,23 +59,23 @@
 /* 13. Variant: Original with Tilde and Space */
 display
    "A:" when l
-   "B"~   at 36
+   "B"~  at 36
    with frame f1 title "Test 13: Original Display".

 /* 14. Variant: Single Quotes and No-Space between string and attribute */
 display
    'A:'when l
-   'B'~  at 36
+   'B'~ at 36
    with frame f2 title "Test 14: Sngl/No-Space".

 /* 15. Variant: Wide Whitespace between String and Keyword */
 display
    "A:"       when l
-   'B'~        at 36
+   'B'~       at 36
    with frame f3 title "Test 15: Wide-Space Display".

 /* 16. Variant: The Chaos Display (Tilde touching the attribute) */
 display
    "A:"  when l
-   'B'~aat 36
+   'B'~at 36
    with frame f4 title "Test 16: Chaos Spacing".

This issue is not a regression of the latest changes. So, I think it's a scenario that's not handled by 6859b.

Please create a new task for these deviations. We will defer work on those.

Greg Shah wrote:

Please create a new task for these deviations. We will defer work on those.

Done! The issue is described in #11250.

The OO application converted successfully. The only differences are about the order of some comments.

I've restarted conversion testing as there are new changes in 6859b/16454.
Conversion for ChUI regression testing application completed successfully and there are no differences. Continuing with the other apps.

I've retested in advance conversion for an OO application as well. The output is the same as before latest changes: different order of comments.

Conversion of a multi-tenant application completed successfully. I need to get a baseline to check if there are any differences.

1. The comment ordering issue persists, similar to what we saw during the OO app conversion. Should we address this now, or fix it in a separate task?
2. There differences related to how & is processed as an include parameter. Here is the testcase:

start.pstart.p

def var a as char initial "test".
{custom.i &name   = "a" 
           &silent = "silent" 
           &p1     = "test" 
           &p2     = "&"} 

     if opsys = "unix"  then unix {&silent}
     value("a/b" + {&name} + ".sh")
{&p1} {&p2} {&p3} {&p4} {&p5} {&p6} {&p7} {&p8} {&p9} {&p10} {&p11} {&p12}.

This is the output of 6859b preprocessor:

def var a as char initial "test".
     if opsys = "unix"  then unix silent
     value("a/b" + a + ".sh")
test          .

This is the output of trunk preprocessor:

def var a as char initial "test".
     if opsys = "unix"  then unix silent
     value("a/b" + a + ".sh")
test &          .  

And this is the output of OE preprocessor:

def var a as char initial "test".
     if opsys = "unix"  then unix silent
     value("a/b" + a + ".sh")
test          .

So, it seems that trunk keeps & while 6859b doesn't. But the good thing is that OE preproccesor doesn't keep &, as well.
That being said, if we ignore comments order, we can consider that there are no regressions for multi-tenant app conversion and we can continue testing conversion with other apps.

About the & behavior, I'm a bit surprised with the result but I'm glad we match OE so that is OK. Please add this case to our preproc testcases.

The comment changes are most likely related to the way that 6859b changes how whitespace is output. Comment conversion is very sensitive to that, especially newline processing. I don't consider that a regression, we have some known issues to resolve in the preproc and they also show some difference in whitespace. For now, let's accept this result as passing.

Great, I'm adding that to the preproc testcases now and moving forward with the conversion tests.

I've added the testcase mentioned in #6859-150 into testcases/1830.

Conversion of a large GUI application completed successfully. The only differences are comments order and corresponding line-number metadata (this is expected due to comment placement). Now, I'm waiting for the results of another large GUI application.

Radu confirms that conversion for another large GUI application completed successfully and there are no differences aside the comments order.

What additional testing do we need for this?

Rebased branch 6859b with trunk 16450. New revision is 16470.