FWD » Core Development » Conversion Tools

In conversion and analytics, where we report line/column from cache file we also should report the original unpreprocessed file (proc/class/include) and line/column.

I added some logic to implement the idea above, so the symbols are described with more details in pphints file, for example

&SCOPED-DEFINE num1 message 10000000
&SCOPED-DEFINE num1 10000000
&SCOPED-DEFINE num2 1000000000
&SCOPED-DEFINE num3 100000000000

generates:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!--Preprocessor hints--><hints>
  <preprocessor-output>
    <symbol column="1" end-column="29" end-line="1" line="1" name="num1" type="scoped" value="message 10000000"/>
    <symbol column="1" end-column="29" end-line="1" line="1" name="num1" type="scoped" value="10000000"/>
    <symbol column="1" end-column="31" end-line="1" line="1" name="num2" type="scoped" value="1000000000"/>
    <symbol column="1" end-column="33" end-line="1" line="1" name="num3" type="scoped" value="100000000000"/>
  </preprocessor-output>
</hints>

As you can see, all of the symbols have the same starting line, this is because this is with respect to the .cache, so the line is an imaginary one. It is still needed to caclulate line shifts.

The next steps should be I think storing the multi-line scoped and global defines, as well as &if... directives and &message& in a similar fashion, because they also increase the line counter.

Florin Eugen Rotaru wrote:

The next steps should be I think storing the multi-line scoped and global defines,

I worked on this, the pphints now correctly store the starting and ending line for the cases where the definitions is on multiple lines:

&SCOPED-DEFINE num0 message 10000000.~
 message 1.~
 message 1.~
 message 1.~
 message 1.~
 message 2.

&SCOPED-DEFINE num1 message "hello".

&SCOPED-DEFINE num2 message "hello world".

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!--Preprocessor hints--><hints>
  <preprocessor-output>
    <symbol column="1" end-column="1" end-line="6" line="1" name="num0" type="scoped" value="message 10000000. message 1. message 1. message 1. message 1. message 2."/>
    <symbol column="1" end-column="1" end-line="3" line="2" name="num1" type="scoped" value="message &quot;hello&quot;."/>
    <symbol column="1" end-column="1" end-line="5" line="3" name="num2" type="scoped" value="message &quot;hello world&quot;."/>
  </preprocessor-output>
</hints>

The end-column I think should always be 1, since all of the defines need an empty line as terminators.

I also noticed the possibility to have includes like

{%arg

}

Because this also increases the line discrepancy between the original files and the cache files, I think the pphints should also store this information (same for the {include.i} statements (i.e. the start/end line/col for references, not only the lines where they are written)).

I started working on that and made a first committ to 3881a.

I continued extending the pphints format by adding a child tag for includes and references. Basically, the parent's properties are for the .cache code and the child for the original .p code.

For example:

.p:

&SCOPED-DEFINE num0 message 10000000.~~n~
 message 1.~~n~
 message 1.~~n~
 message 1.~~n~
 message 1.~~n~
 message 2.

{&num0

   }

.cache:

message 10000000.
 message 1.
 message 1.
 message 1.
 message 1.
 message 2.

.pphints:

...
     <symbol column="1" end-column="1" end-line="7" line="1" name="num0" type="scoped" value="message 10000000.~n message 1.~n message 1.~n message 1.~n message 1.~n message 2."/>
    <reference column="1" end-column="12" end-line="7" line="2" name="num0" type="scoped" value="message 10000000.~n message 1.~n message 1.~n message 1.~n message 1.~n message 2.">
      <reference-call end-column="5" line-span="4"/>  // this is the child describes the {&...} call
    </reference>
...

The logic is identical for file includes.

This way, we can map the cache line/col to the original ones in two phases:

1) Expand the preprocessor definitions

2) Shrink the code which was expanded during preprocessor.

Florin Eugen Rotaru wrote:

I have implemented a first working method which computes the original line/column (currently taking into consideration only includes)

{{collapse(Method)
[...]
}}

I will have to test additionally to make sure it works for all cases (e.g. with newlines at the end of the included code, etc). Then I will add similar logic for other possible preprocessor elements.

Last branch revision: 3881a/16024.

I'm continuing to work on the getOriginalLocation() method, I think we can make the code that treats the includes and references generic because they have the same principle: replace the {...} with actual code in the caches.

For this I've created an abstract class called AbstractInjectableHint and made IncludeHint and ReferenceHint extend it.

public abstract class AbstractInjectableHint
{
   private int callStartLine = 0;
   private int callLineSpan = 0;
   private int callEndColumn = 0;

   public int getCallStartLine() 
   {
      return callStartLine;
   }

   public void setCallStartLine(int callStartLine) 
   {
      this.callStartLine = callStartLine;
   }

   public int getCallLineSpan()
   {
      return callLineSpan;
   }

   public void setCallLineSpan(int callLineSpan)
   {
      this.callLineSpan = callLineSpan;
   }

   public int getCallEndColumn()
  {
      return callEndColumn;
   }

   public void setCallEndColumn(int callEndColumn)
   {
      this.callEndColumn = callEndColumn;
   }
}

The call is the actual {...} statement.

With the changes from branch 3881a, one of the next steps is addressing this bug:

for the 4GL code

&SCOPED-DEFINE num0 message 10000000.

{&num0} {&num0} {&num0}

the newly added properties of <reference> and the new tag <reference-call are not very exact if the {&...} are on the same line:

    <reference column="1" end-column="18" end-line="2" line="2" name="num0" type="scoped" value="message 10000000.">
      <reference-call end-column="8" line="3" line-span="1"/>
    </reference>
    <reference column="19" end-column="18" end-line="2" line="2" name="num0" type="scoped" value="message 10000000.">
      <reference-call end-column="31" line="3" line-span="1"/>
    </reference>
    <reference column="37" end-column="18" end-line="2" line="2" name="num0" type="scoped" value="message 10000000.">
      <reference-call end-column="54" line="3" line-span="1"/>
    </reference>

The end-columns are incorrect.

I am also noticing that the tag <symbol name="fwd-version" type="global" value="FWD v4.0.0_undefined_undefined_16003"/> went missing, so this should also be checked at some point.

What I would do next here is:

1. Make sure the bug describe above is fixed (i.e. the pphints metadata correctly stores information about the lines, columns, spans of preprocessor structures)

2. Extend it by adding metadata for &IF, &THEN, &ELSEIF, &ELSE, and &ENDIF preprocessor directives, because these will not be present in the preprocessed files, so we have to know how many lines and columns to retrieve/collapse in order to convert the line/col location.

3. &UNDEFINE preprocessor directive

4. Once those are finished, the getOriginalLocation() describe in #3881-8 method has to consider all these metadata. My intention was to process the @{file.i} and {&reference} calls similarly, in a generic way.