FWD » Core Development » Database

def var h as handle.
def temp-table tt1 field f1 as int
                   field f2 as char.

def var fooClass as oo.FooTest.

create tt1.
tt1.f1 = 4.
release tt1.

def query hquery for tt1.

do transaction on error undo, throw:
   def var ch1 as char.
   ch1 = "FOR EACH tt1 WHERE tt1.f1 < 9 and tt1.f2 = 'bc".

   message length(ch1).
   ch1 = ch1 + chr(13).
   ch1 = ch1 + "'".

   query hQuery:QUERY-PREPARE(ch1).

   message ch1.
end.   

The following code will execute without any exception in 4gl, however in FWD it will raised the following error ** Unmatched quote found in procedure. (133).

Alternatively this test case also reproduce the issues:

def var h as handle.
def temp-table tt1 field f1 as int
                   field f2 as char.

def var fooClass as oo.FooTest.

create tt1.
tt1.f1 = 4.
release tt1.

def query hquery for tt1.

do transaction on error undo, throw:
   def var ch1 as char.
   ch1 = "FOR EACH tt1 WHERE tt1.f1 < 9 and tt1.f2 = 'bc\r'".

   message length(ch1).

   query hQuery:QUERY-PREPARE(ch1).

   message ch1.
end.   

Our preprocessor/lexer strips '\r' characters in a naive way that needs to be refined.

Eduard, is 'bc\r' right? Shouldn't it be 'bc~r' to actually generate the CR?

Please also see #9967, which is somehow related to the escape char used under linux.

Alexandru Lungu wrote:

Eduard, is 'bc\r' right? Shouldn't it be 'bc~r' to actually generate the CR?

It is either of them. In both cases in fwd a exception is thrown. The original case in customer application was with ch = ch + char(13).

The escape sequences should include an additional '\' prefix. The current revision handles this only for '\u' within comments. The '\r' character is processed only in specific contexts. I'm currently investigating this.

Branch 9969a was rebased with trunk. The new revision is 15901.

I made a few changes that fix the way Java escaping is handled. First, in ClearStream.java sequences (with '\') are translated exactly when unixEscapes is true. Then, in character.progressToJavaString everything escaped with "/" is double escaped to avoid honoring Java escaping. These changes are made in 9969a/15902. With these changes, #9967 is also fixed.

Review of 9969a/r15902.

• UnicodeTranslationException: missing javadocs for both the class and the constructor (I am aware this was committed in 15900, for #8027). Please take this opportunity and add them now;
• progress.g: My understanding is that the inlines for non ' and non " were replaced with the new rules (NON_QUOTE_CHAR, and NON_DOUBLE_QUOTE_CHAR), so I expect the result will not change. Will these fix the issue in note #9969-11?
• ClearStream and character: I think that is exactly what the H entries describe.

Ovidiu Maxiniuc wrote:

Review of 9969a/r15902.

• progress.g: My understanding is that the inlines for non ' and non " were replaced with the new rules (NON_QUOTE_CHAR, and NON_DOUBLE_QUOTE_CHAR), so I expect the result will not change. Will these fix the issue in note #9969-11?

Yes because without changes, ANTLR generates a token set to match non quote characters. That token set doesn't contain '\r':

            else if ((_tokenSet_7.member(LA(1)))) {
                matchNot('\'');
            }

Separating that case will not exclude '\r':

    public final void mNON_DOUBLE_QUOTE_CHAR(boolean _createToken) throws RecognitionException, CharStreamException, TokenStreamException {
        int _ttype; Token _token=null; int _begin=text.length();
        _ttype = NON_DOUBLE_QUOTE_CHAR;
        int _saveIndex;

        matchNot('\"');
        if ( _createToken && _token==null && _ttype!=Token.SKIP ) {
            _token = makeToken(_ttype);
            _token.setText(new String(text.getBuffer(), _begin, text.length()-_begin));
        }
        _returnToken = _token;
    }

Ovidiu Maxiniuc wrote:

Review of 9969a/r15902.

• UnicodeTranslationException: missing javadocs for both the class and the constructor (I am aware this was committed in 15900, for #8027). Please take this opportunity and add them now;

Fixed in 9969a/15903.

Is conversion regression tesing necessary or run-time testing is enough?

This affects all string processing in the lexer in a profound way. Full conversion testing is needed.

The conversion of a large GUI client application has been completed successfully. The conversion log is clean and there are no unexpected changes. All unit tests passed. ✅

I will get the results for another GUI application and an application server application soon. In the meantime, I am converting the ChUI regression testing application.

The conversion of the ChUI regression test application completed successfully, but the source compilation failed. The cause is the changes in the secondary part of the problem, those in character.java. The problematic scenario is ch = "\\". The code converted with 15903 looks like this: ch.assign("\\\"). I need to get rid of all backslashes if windows is enabled.

The conversion of another large GUI customer application was successfully completed. No source code differences were detected, the runtime behaves as expected, and the logs show no anomalies. ✅

Eduard mentioned in #4602-50 that exception is not thrown anymore.

I used this testcase to see if the behavior is as expected:
[...]

Converted code with unixEscapes on true:
[...]

Converted code with unixEscapes on false:
[...]

What is the output for OE?

Greg Shah wrote:

I used this testcase to see if the behavior is as expected:
[...]

Converted code with unixEscapes on true:
[...]

Converted code with unixEscapes on false:
[...]

What is the output for OE?

This one:

---------------------------
Message (Press HELP to view stack trace)
---------------------------
• \225 € \U0020AC • \u2022 ~u202 ~u2222 
 \r      \t \\ \
---------------------------
OK   Help   
---------------------------

Shouldn't I wait for the appserver application conversion test results?

Conversion of appserver application completed successfully. The runtime is also working fine.

Octavian, I think there may be a problem. In a project there is a var defined like:

define variable ch as character initial "~000doesntmatterelse":U no-undo.

Note the null character first (escaped) this is emitted 'as is' (unescaped) in the Java code, making it 1. uneditable using gedit or something like this, and 2. in eclipse, the line with the var will 'drop' everything after the null char.

We need to escape the null char.

Constantin Asofiei wrote:

Octavian, I think there may be a problem. In a project there is a var defined like:
[...]

Note the null character first (escaped) this is emitted 'as is' (unescaped) in the Java code, making it 1. uneditable using gedit or something like this, and 2. in eclipse, the line with the var will 'drop' everything after the null char.

We need to escape the null char.

I think this is an exception that could be handled in the same way as control characters:

               else if (result == 126)
               {
                  super.unread('~');
                  clearCount++;
               }
               else if (!notInStrings)
               {
                  switch(result)
                  {
                     case '\b':
                        super.unread('b');
                        clearCount++;
                        return '~';
                     case '\t':
                        super.unread('t');
                        clearCount++;
                        return '~';
                     case '\n':
                        super.unread('n');
                        clearCount++;
                        return '~';
                     case '\f':
                        super.unread('f');
                        clearCount++;
                        return '~';
                     case '\r':
                        super.unread('r');
                        clearCount++;
                        return '~';
                     case '\\':
                        super.unread('\\');
                        clearCount++;
                        return '~';
                  }

This is more related to #8027 than this task.

I think we need to handle and escape all control characters - see https://www.ascii-code.com/characters/control-characters

I don't know from which task the problem arises in the customer project (this was the last commit I saw somehow related) - please work on #8027 if is from there. We need this fixed, as it is now the code doesn't convert properly. I haven't checked the .class bytecode if is right or not.

Constantin Asofiei wrote:

I think we need to handle and escape all control characters - see https://www.ascii-code.com/characters/control-characters

I don't know from which task the problem arises in the customer project (this was the last commit I saw somehow related) - please work on #8027 if is from there. We need this fixed, as it is now the code doesn't convert properly. I haven't checked the .class bytecode if is right or not.

Sure! I'll take care of it.

After upgrading to trunk 15913 (including 15910), the antlr build target results in a lot (30+) of nondeterminism errors, like this:

   progress.g: warning:lexical nondeterminism between rules WS and NON_QUOTE_CHAR upon
   progress.g:     k==1:'\u0000'..' ','\u007f'
   progress.g:     k==2:<end-of-token>
   progress.g:     k==3:<end-of-token>
   progress.g:     k==4:<end-of-token>

Ovidiu Maxiniuc wrote:

After upgrading to trunk 15913 (including 15910), the antlr build target results in a lot (30+) of nondeterminism errors, like this:
[...]I know the full parser will be rewritten in a future but having all these, even benignant, may hide (make difficult to spot) eventual errors in the future.

I tested and this is fixed if protected access modifier is added to NON_QUOTE_CHAR and NON_DOUBLE_QUOTE_CHAR rules. I will include this changes in the fix for #9969-36.