Project

General

Profile

Bug #10099

Tilde consumes the octals/follow-up characters, when in OE this doesn't happen

Added by Stefan Vieru over 1 year ago. Updated 16 days ago.

Status:
Internal Test
Priority:
Normal
Assignee:
Stefan Vieru
Target version:
-
Start date:
Due date:
% Done:

100%

billable:
No
vendor_id:
GCD
case_num:
version_reported:
version_resolved:
reviewer:
Greg Shah
production:
No
env_name:
topics:
4GL Preprocessor

Related issues

Related to Testing - Support #6859: preprocessor tests Test
Related to Conversion Tools - Bug #11835: -keeptildes keeps escape text for macro-expanded content, where OE emits only the byte Review
Related to Conversion Tools - Bug #11837: preprocessor expands a TAB byte to spaces where OE emits the byte Review

History

#1 Updated by Stefan Vieru over 1 year ago

  • Subject changed from Tilde consumes the octals, when in OE this doesn't happen to Tilde consumes the octals/follow-up characters, when in OE this doesn't happen

When preprocessing the following test: (same behavior is present with ~ directly)

;?" 
;?'
;?011
;?~
;?\
;?t
;?r
;?n
;?E
;?b
;?f

The output of OE:
~"" 
~''
~011    
~~~
~\\
~t    
~r

~n

~E
~b
~f

VS the output of FWD:

~" 
~'
~       

~;?
~  
~
~

~
~
~

#2 Updated by Stefan Vieru over 1 year ago

  • Status changed from New to WIP
  • Assignee set to Stefan Vieru

#3 Updated by Stefan Vieru over 1 year ago

I've created r15995.
The change covers all the cases from above except:
  • \
  • ~

These characters are trickier to print without more serious changes to ClearStream.
The changes now allow the appending of the resulting character after ~ + character/octal.

#4 Updated by Stefan Vieru over 1 year ago

  • Status changed from WIP to Review
  • % Done changed from 0 to 100

Greg, could you please take a look at my changes? Let me know if I should do anything else here.

#5 Updated by Stefan Vieru over 1 year ago

#6 Updated by Stefan Vieru about 1 year ago

  • Status changed from Review to WIP
  • % Done changed from 100 to 80

#7 Updated by Stefan Vieru about 1 year ago

  • reviewer Greg Shah added

#8 Updated by Stefan Vieru about 1 year ago

  • Status changed from WIP to Review
  • % Done changed from 80 to 100

Fixed in 6859b #6859.
To have \ escaped, the Preprocessor should be run with -windowsescapes.

#9 Updated by Greg Shah about 1 year ago

What revision of 6859b has the fix?

#10 Updated by Stefan Vieru about 1 year ago

r16040

@@ -814,10 +837,10 @@

          // from now on, there is one leader on the stack
          char leader = ((Character)leaders.pop()).charValue();
+         int nextCharTemp = nextChar;

          if (notInStrings && notInComments)
          {
-            translated = true;
             switch (nextChar)
             {
                case '"':
@@ -830,6 +853,7 @@
                   {
                      passThru = true;
                   }
+                  translated = true;
                   break;
                case '{':
                case ';':
@@ -837,24 +861,37 @@
                   break;
                case 't':
                   nextChar = 0x09;
+                  translated = true;
                   break;
                case 'r':
                   nextChar = 0x0D;
+                  translated = true;
                   break;
                case 'n':
                   nextChar = 0x0A;
+                  translated = true;
                   break;
                case 'E':
                   nextChar = 0x1B;
+                  translated = true;
                   break;
                case 'b':
                   nextChar = 0x08;
+                  translated = true;
                   break;
                case 'f':
                   nextChar = 0x0C;
+                  translated = true;
                   break;
                case 'u':
                   passThru = true;
+                  translated = true;
+                  break;
+               case '\\':
+                  if (!unixEscapes)
+                  {
+                     translated = true;
+                  }
                   break;
                default:
                   translated = false;
@@ -866,12 +903,25 @@
                if (nextChar >= '0' && nextChar <= '9')
                {
                   // the Progress PP doesn't check bytes 2 and 3
+                  int byte1 = nextChar;
                   int byte2 = mread();
                   int byte3 = mread();
-                  nextChar = 64 * (nextChar - '0') + 8 * (byte2 - '0')
+                  nextChar = 64 * (byte1 - '0') + 8 * (byte2 - '0')
                              + (byte3 - '0');
                   if (nextChar == marker)
                      throw new IOException("octal escape matches marker");
+                  super.unread(nextChar);
+                  if (keepTildes)
+                  {
+                     // here we would have to match the OE output by keeping the escape character and the
+                     // escaped character, followed by the resulted translation
+                     super.unread(byte3);
+                     super.unread(byte2);
+                     super.unread(byte1);
+                     super.unread(leader);
+                     clearCount += 4;
+                  }
+                  continue;
                }
             }
          } // not in strings

#11 Updated by Stefan Vieru about 1 year ago

The above is only a bit, there's more at the end where we use nextCharTemp.

#12 Updated by Stefan Vieru about 1 year ago

Testcases that cover this:
  • alternative_coding_quirk_06

#13 Updated by Greg Shah about 1 year ago

  • Status changed from Review to Internal Test

I've reviewed this change and all the related change in 6859b. It is difficult to assess by code review alone, if these changes are correct. At this point, we have to carefully test the full range of customer applications to confirm if this is safe.

#14 Updated by Greg Shah 7 months ago

  • topics 4GL Preprocessor added

#15 Updated by Greg Shah 16 days ago

  • Related to Bug #11835: -keeptildes keeps escape text for macro-expanded content, where OE emits only the byte added

#16 Updated by Greg Shah 16 days ago

  • Related to Bug #11837: preprocessor expands a TAB byte to spaces where OE emits the byte added

#17 Updated by Greg Shah 16 days ago

The residual failure in alternative_coding_quirk_06 is NOT this defect.

alternative_coding_quirk_06 is still failing and a reader is likely to conclude the fix regressed. It did not — the escape-text behaviour this issue fixed works. The residual difference is tab expansion (#11837):

OE : txt = "text"~~~t<TAB>.
FWD: txt = "text"~~~t␣␣␣␣␣␣␣␣.

Also worth recording, because it is an easy misreading: nine baselines contain a bare CR and every one belongs to a failing test. That correlation is coincidental. This test emits its bare CR correctly, so it is not evidence of a CR problem.

Also available in: Atom PDF