FWD » Core Development » Conversion Tools

A thought about the case like define variable ch as character initial "~000doesntmatter":U no-undo..

In FWD, the character(String) constructor does javaSpacifyNull. I don't think we should do this at parse time with the literal - we should just escape the null char and let it emit as it was in the 4GL code.

This also helps the anti-parser to emit the similar 4GL code as the source.

Agreed.

I finished preprocessor testcases for printable ASCII characters (octal representation). I continue with code page-dependent characters and representative Unicode cases.

I’ve created branch 10028a to continue working on this. From what I recall, there are two sets of tests that I created for the testcases project: control_characters.xml and printableASCIICharacters.xml. I will start by getting these fixed first. Currently, both test sets are failing.

Some issues with the control_characters.p test set involved two aspects regarding how the null character (~000) is handled.

First, the current implementation calculates octal sequences and returns the results directly. For the NULL character, this resulted in returning a 0 character, which breaks the string. To match the OE behavior, we now return a new line when the octal result is zero.

The second issue affecting NULL character handling was the progressSpacifyNull method. This method is triggered by the STRING rule in text.g. While the logic was correct, it correctly transformed the input "This has a ~000 (null) char." into "This has a ", the entire process is occurring at runtime in OE. The actual OE preprocessor output is "This has a ~000 (null) char.". Therefore, I moved the progressSpacifyNull method call from the STRING rule to the progressToJavaString method, ensuring it executes before the string input is processed.

Another issue causing the control_characters.p test set to fail is how the \t character is handled. The current implementation replaces the tab character with spaces, and I am not entirely sure why this is necessary. The following method is responsible:

   /**
     * Expands tabs with spaces right in the ANTLRStringBuilder text.
     * This version is used here because according to experiments,
     * the tabs expansion for the preprocessor variables is immediate.
     */
   public void tab()
   {
      int c  = getColumn();
      int nc = (((c-1)/tabsize) + 1) * tabsize + 1;  // calculate tab stop
      setColumn(nc);

      text.setLength(text.length() - 1);    // replace '\t' in the buffer
      for (; c < nc; c ++)                  // add more spaces
         text.append(' ');
   }

While I understand the first part, calculating the new column number based on the tab size, it is confusing why we are replacing the tab with literal space characters.
If I remove that replacement logic from the tab method and apply the changes mentioned above for null characters, the control_characters.p tests pass.