FWD » Core Development » Conversion Tools

• in OE with /* */ comments we don't have two /r/n, but with // we do. The following processes as it is.
  

  /* outside of comments and strings there is a quirk of the "alternative codings" */
  message "test".
  /* - the ;) alternative coding construct will be converted by the preproc to the } character */
  

• if we have // instead of /* */, the output is:
  

  // outside of comments and strings there is a quirk of the "alternative codings" 
  
  message "test".
  // - the ;) alternative coding construct will be converted by the preproc to the } character
  
  

• in OE we have \r\n when we have a NL; Meaning when we have a // comment, we have two /r/n after it.
• in FWD whether we use Preprocessor on windows or unix we have ONLY \n, due to the following I think
  

           // normalize new lines
           if (nextChar == '\r')
           {
              int pastNextChar = mread();
              if (pastNextChar != '\n')
              {
                 super.unread(pastNextChar);
              }
              nextChar = '\n';
           }
           if (nextChar == '\n')
           {
              return nextChar;
           }
  

The same normalization also destroys a CR inside a value, and this issue cannot be tested by the current harness.

Two updates.

First, a second symptom of the same blanket normalization: a CR that is data rather than a line terminator does not survive a define value. Filed separately as #11836, because the fix differs in kind, but they share a root cause and should be fixed together.

Second, and more important for planning: the existing suite cannot provide a testcase for this issue at all. text-file-comparison compares with BufferedReader.readLine(), which treats LF, a bare CR and CRLF all as line terminators, so a line-ending difference is invisible to it by design. #9894 can never go red under the current harness. Covering it needs a byte-level comparison step.

text-file-comparison normalizes line terminators via readLine(), so the exact difference that issue is about cannot fail a test. The wiki already recommends cmp as a manual cross-check for precisely this reason, and warns that because the comparison is tolerant, "baseline corruption can sit in the repository indefinitely without any test failing."

Proposal: add a byte-exact comparison step to the harness and use it for the small number of tests where the bytes matter — #9894's line terminators, and any test whose baseline contains a bare CR. This makes the existing manual advice enforceable and closes the corruption hole at the same time.

The alternative — check byte-exactness only at baseline capture/refresh time — catches damage introduced in transit but not a regression in FWD's own output, so it is the weaker option.