FWD » Core Development » Conversion Tools

A thought about the case like define variable ch as character initial "~000doesntmatter":U no-undo..

In FWD, the character(String) constructor does javaSpacifyNull. I don't think we should do this at parse time with the literal - we should just escape the null char and let it emit as it was in the 4GL code.

This also helps the anti-parser to emit the similar 4GL code as the source.

Agreed.

I finished preprocessor testcases for printable ASCII characters (octal representation). I continue with code page-dependent characters and representative Unicode cases.

I’ve created branch 10028a to continue working on this. From what I recall, there are two sets of tests that I created for the testcases project: control_characters.xml and printableASCIICharacters.xml. I will start by getting these fixed first. Currently, both test sets are failing.

Some issues with the control_characters.p test set involved two aspects regarding how the null character (~000) is handled.

First, the current implementation calculates octal sequences and returns the results directly. For the NULL character, this resulted in returning a 0 character, which breaks the string. To match the OE behavior, we now return a new line when the octal result is zero.

The second issue affecting NULL character handling was the progressSpacifyNull method. This method is triggered by the STRING rule in text.g. While the logic was correct, it correctly transformed the input "This has a ~000 (null) char." into "This has a ", the entire process is occurring at runtime in OE. The actual OE preprocessor output is "This has a ~000 (null) char.". Therefore, I moved the progressSpacifyNull method call from the STRING rule to the progressToJavaString method, ensuring it executes before the string input is processed.

Another issue causing the control_characters.p test set to fail is how the \t character is handled. The current implementation replaces the tab character with spaces, and I am not entirely sure why this is necessary. The following method is responsible:

   /**
     * Expands tabs with spaces right in the ANTLRStringBuilder text.
     * This version is used here because according to experiments,
     * the tabs expansion for the preprocessor variables is immediate.
     */
   public void tab()
   {
      int c  = getColumn();
      int nc = (((c-1)/tabsize) + 1) * tabsize + 1;  // calculate tab stop
      setColumn(nc);

      text.setLength(text.length() - 1);    // replace '\t' in the buffer
      for (; c < nc; c ++)                  // add more spaces
         text.append(' ');
   }

While I understand the first part, calculating the new column number based on the tab size, it is confusing why we are replacing the tab with literal space characters.
If I remove that replacement logic from the tab method and apply the changes mentioned above for null characters, the control_characters.p tests pass.

• removing the tab replacement with white spaces.
• moving the progressSpacifyNull method call from STRING to progressToJavaString.
• printing translated octal sequences if needed.
• handling the octal sequences inside the CODE rule:
  • if we are at the beginning, we consume the octal sequence only:
    

         message ~042 test! ". // message " test! ".
         // visible preprocessor output should be: message ~042" test! ".
         // where ~042 is CODE and " is starting a STRING.      
    

  • if we are already inside a CODE rule processing, we consume the octal sequence and continue the CODE rule without the escaped character:
    

        def frame a~042. // def frame a".
        // visible preprocessor output should be: def frame a~042.
        // where a~042. is CODE.
    

The changes are committed in 10028a/16654.

I added printableASCIICharacters.p to the test plan, fixed the include file path from tests and adjusted the baseline output for the tests. All of these are committed in testcases/1856.

I added unicode_escape_sequences.p to cover several cases involving unicode sequences in different contexts. Based on these and the cases related to octal sequences, I'm preparing a solution that uses the source-charset parameter to handle escape sequences when needed. The latest tests and changes have been committed to testcases/1857.