FWD

2. I have seen the following error many times while using convert.front to parse my application code:
...
Analysis of the code shows that the error is being thrown when the built-in custom class Progress.Lang.AppError is referenced, for instance my line of code might be:
UNDO, THROW NEW Progress.Lang.AppError ("some error occurred").

I've created a test program with only this one line of code. When I parse it using java -classpath p2j/build/lib/p2j.jar com.goldencode.p2j.convert.ConversionDriver f2 undo_throw.p, it parses successfully and displays this console output:

undo_throw.p
   Recursive level 1 parse of class: ./skeleton/oo4gl/Progress/Lang/AppError.cls
   Recursive level 2 parse of class: ./skeleton/oo4gl/Progress/Lang/ProError.cls
   Recursive level 3 parse of class: ./skeleton/oo4gl/Progress/Lang/Error.cls
   Recursive level 3 parse of class: ./skeleton/oo4gl/Progress/Lang/Object.cls
   Recursive level 4 parse of class: ./skeleton/oo4gl/Progress/Lang/Class.cls
   Recursive level 5 parse of class: ./skeleton/oo4gl/Progress/Lang/ParameterList.cls
   Done 1st level parsing for ./skeleton/oo4gl/Progress/Lang/AppError.cls

If you do not see this, then I suspect you need to lay down the "skeleton classes". You can read about these in:

Skeleton Classes

And download the ones we have created here:

Skeleton Classes Download

3. I am also seeing this error a lot:
...
[java] Caused by: java.lang.RuntimeException: Cannot find class/interface ReportEmailSender in PROPATH.

This is most likely a missing ./cfg/class_map.xml configuration file or a missing class in that file. For details, see:

Class Map

4. Which ADM1 and ADM2 code should I copy from my DLC\src and DLC\gui directories?

As a start, you should probably copy all the include files and any procedures that you know you call. When you put them in place, make sure that the PROPATH configuration in ./cfg/p2j.cfg.xml is setup to find things based in the directory structure you've added. After everything parses, you will be able to tell which includes are actually referenced and which are not, which will allow you to prune the directory down to the minimum.

To figure out the exact ADM1/ADM2 procedures can be done with the Call Graph Analyzer, but using that is beyond the scope of this post. :) For now, add those procedures you know you use. You can always add more later and re-run the parsing and report generation. Because it is an automated process, you can incrementally modify the project until you have everything figured out exactly.

Thanks for the prompt response Greg - I found the documentation covering class_map.xml and skeleton classes not long after making this post, and I'm working through getting it all set up.

My pleasure.

By the way, I did upload a newer version of the skeleton classes just before I posted. If you downloaded the 20160210 version, please download the 20170714 version that is now available.

That's the version I got. I tried running convert.front again with no success (but a few different errors than last time), I assume I've not quite got everything right yet.

1. Do I need to add:
<parameter name="oo-skeleton-path" value="./p2j/skeleton/" />

to my ./cfg/p2j.cfg.xml file, or will it automatically pick that up? The documentation mentions that ./p2j/skeleton/ is the default.

2. I am still not getting success with our own classes, it looks like the parser assumes that the classes have already been processed because it's missing a ".lexer" file, e.g.

[java] .\abl\Batch\Daily\check-newprice.p
     [java]    Recursive level 1 parse of class: .\Source\ReportEmailSender.cls
     [java]    Done 1st level parsing for .\Source\ReportEmailSender.cls
     [java] Failure in file '.\abl\Batch\Daily\check-newprice.p':
     [java] com.goldencode.ast.AstException: Error processing .\abl\Batch\Daily\check-newprice.p
     [java]     at com.goldencode.p2j.uast.AstGenerator.processFile(AstGenerator.java:962)
     [java]     at com.goldencode.p2j.uast.ScanDriver.lambda$scan$0(ScanDriver.java:375)
     [java]     at com.goldencode.p2j.uast.ScanDriver.scan(ScanDriver.java:410)
     [java]     at com.goldencode.p2j.uast.ScanDriver.scan(ScanDriver.java:248)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.runScanDriver(ConversionDriver.java:496)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.front(ConversionDriver.java:377)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.main(ConversionDriver.java:2005)
     [java] Caused by: java.lang.RuntimeException: Failed level 1 parse of class ReportEmailSender
     [java]     at com.goldencode.p2j.uast.SymbolResolver.loadClass(SymbolResolver.java:2389)
     [java]     at com.goldencode.p2j.uast.ProgressParser.user_defined_type_name(ProgressParser.java:7304)
     [java]     at com.goldencode.p2j.uast.ProgressParser.var_type(ProgressParser.java:46125)
     [java]     at com.goldencode.p2j.uast.ProgressParser.as_clause(ProgressParser.java:20380)
     [java]     at com.goldencode.p2j.uast.ProgressParser.def_var_stmt(ProgressParser.java:10937)
     [java]     at com.goldencode.p2j.uast.ProgressParser.define_stmt(ProgressParser.java:9454)
     [java]     at com.goldencode.p2j.uast.ProgressParser.stmt_list(ProgressParser.java:23372)
     [java]     at com.goldencode.p2j.uast.ProgressParser.statement(ProgressParser.java:6087)
     [java]     at com.goldencode.p2j.uast.ProgressParser.single_block(ProgressParser.java:4958)
     [java]     at com.goldencode.p2j.uast.ProgressParser.block(ProgressParser.java:4678)
     [java]     at com.goldencode.p2j.uast.ProgressParser.external_proc(ProgressParser.java:4605)
     [java]     at com.goldencode.p2j.uast.AstGenerator.parse(AstGenerator.java:1487)
     [java]     at com.goldencode.p2j.uast.AstGenerator.processFile(AstGenerator.java:957)
     [java]     ... 6 more
     [java] Caused by: com.goldencode.ast.AstException: Error processing .\Source\ReportEmailSender.cls
     [java]     at com.goldencode.p2j.uast.AstGenerator.preScanClass(AstGenerator.java:1444)
     [java]     at com.goldencode.p2j.uast.SymbolResolver.loadClass(SymbolResolver.java:2374)
     [java]     ... 18 more
     [java] Caused by: java.io.FileNotFoundException: .\Source\ReportEmailSender.cls.lexer (The system cannot find the path specified)
     [java]     at java.io.FileOutputStream.open0(Native Method)
     [java]     at java.io.FileOutputStream.open(FileOutputStream.java:270)
     [java]     at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
     [java]     at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
     [java]     at java.io.FileWriter.<init>(FileWriter.java:90)
     [java]     at com.goldencode.p2j.uast.LexerDumpFilter.openOutput(LexerDumpFilter.java:318)
     [java]     at com.goldencode.p2j.uast.LexerDumpFilter.<init>(LexerDumpFilter.java:184)
     [java]     at com.goldencode.p2j.uast.AstGenerator.prepareLexer(AstGenerator.java:1569)
     [java]     at com.goldencode.p2j.uast.AstGenerator.preScanClass(AstGenerator.java:1433)
     [java]     ... 19 more

1. Do I need to add:
<parameter name="oo-skeleton-path" value="./p2j/skeleton/" />

to my ./cfg/p2j.cfg.xml file, or will it automatically pick that up? The documentation mentions that ./p2j/skeleton/ is the default.

You should not need to specify it. It should default to ./p2j/skeleton/. If you unzipped directly in p2j/, it will have created the correct directory structure.

We will respond separately about item 2.

java.io.FileNotFoundException: .\Source\ReportEmailSender.cls.lexer (The system cannot find the path specified)

The path on this looks wrong. Some questions:

• Can you please post the entry for this file from class_map.xml?
• The actual class file exists as .\abl\Source\ReportEmailSender.cls?

I see where my class_map.xml was wrong, I was missing .\abl\ from the front of all the value attributes.

So now I have entries like this:

<entry key="ReportEmailSender" value=".\abl\Source\ReportEmailSender.cls"/>

But still get an error like this:

[java] .\abl\Source\ReportEmailSender.cls
     [java] Failure in file '.\abl\Source\ReportEmailSender.cls':
     [java] com.goldencode.ast.AstException: Error processing .\abl\Source\ReportEmailSender.cls
     [java]     at com.goldencode.p2j.uast.AstGenerator.processFile(AstGenerator.java:962)
     [java]     at com.goldencode.p2j.uast.ScanDriver.lambda$scan$0(ScanDriver.java:375)
     [java]     at com.goldencode.p2j.uast.ScanDriver.scan(ScanDriver.java:410)
     [java]     at com.goldencode.p2j.uast.ScanDriver.scan(ScanDriver.java:248)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.runScanDriver(ConversionDriver.java:496)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.front(ConversionDriver.java:377)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.main(ConversionDriver.java:2005)
     [java] Caused by: java.lang.IllegalArgumentException: character to be escaped is missing
     [java]     at java.util.regex.Matcher.appendReplacement(Matcher.java:809)
     [java]     at java.util.regex.Matcher.replaceAll(Matcher.java:955)
     [java]     at java.lang.String.replaceAll(String.java:2223)
     [java]     at com.goldencode.p2j.uast.SymbolResolver.convertSeparator(SymbolResolver.java:5754)
     [java]     at com.goldencode.p2j.uast.SymbolResolver.pkgExists(SymbolResolver.java:5861)
     [java]     at com.goldencode.p2j.uast.SymbolResolver.addPackage(SymbolResolver.java:2669)
     [java]     at com.goldencode.p2j.uast.ProgressParser.using_stmt(ProgressParser.java:7582)
     [java]     at com.goldencode.p2j.uast.ProgressParser.stmt_list(ProgressParser.java:23739)
     [java]     at com.goldencode.p2j.uast.ProgressParser.statement(ProgressParser.java:6087)
     [java]     at com.goldencode.p2j.uast.ProgressParser.single_block(ProgressParser.java:4958)
     [java]     at com.goldencode.p2j.uast.ProgressParser.block(ProgressParser.java:4678)
     [java]     at com.goldencode.p2j.uast.ProgressParser.external_proc(ProgressParser.java:4605)
     [java]     at com.goldencode.p2j.uast.AstGenerator.parse(AstGenerator.java:1487)
     [java]     at com.goldencode.p2j.uast.AstGenerator.processFile(AstGenerator.java:957)
     [java]     ... 6 more

The first executable line of code is:

USING Progress.Lang.*.

and I can see that the parser fails at the USING statement.

I've verified that I've unpacked the 20170714 skeleton files in to %FWD%\p2j\skeleton, and everything looks okay in there.

Neil Treeby wrote:

But still get an error like this:

[java] .\abl\Source\ReportEmailSender.cls
[java] Failure in file '.\abl\Source\ReportEmailSender.cls':
[java] com.goldencode.ast.AstException: Error processing .\abl\Source\ReportEmailSender.cls
[java] at com.goldencode.p2j.uast.AstGenerator.processFile(AstGenerator.java:962)
[java] at com.goldencode.p2j.uast.ScanDriver.lambda$scan$0(ScanDriver.java:375)
[java] at com.goldencode.p2j.uast.ScanDriver.scan(ScanDriver.java:410)
[java] at com.goldencode.p2j.uast.ScanDriver.scan(ScanDriver.java:248)
[java] at com.goldencode.p2j.convert.ConversionDriver.runScanDriver(ConversionDriver.java:496)
[java] at com.goldencode.p2j.convert.ConversionDriver.front(ConversionDriver.java:377)
[java] at com.goldencode.p2j.convert.ConversionDriver.main(ConversionDriver.java:2005)
[java] Caused by: java.lang.IllegalArgumentException: character to be escaped is missing
[java] at java.util.regex.Matcher.appendReplacement(Matcher.java:809)
[java] at java.util.regex.Matcher.replaceAll(Matcher.java:955)
[java] at java.lang.String.replaceAll(String.java:2223)
[java] at com.goldencode.p2j.uast.SymbolResolver.convertSeparator(SymbolResolver.java:5754)
[java] at com.goldencode.p2j.uast.SymbolResolver.pkgExists(SymbolResolver.java:5861)
[java] at com.goldencode.p2j.uast.SymbolResolver.addPackage(SymbolResolver.java:2669)
[java] at com.goldencode.p2j.uast.ProgressParser.using_stmt(ProgressParser.java:7582)
[java] at com.goldencode.p2j.uast.ProgressParser.stmt_list(ProgressParser.java:23739)
[java] at com.goldencode.p2j.uast.ProgressParser.statement(ProgressParser.java:6087)
[java] at com.goldencode.p2j.uast.ProgressParser.single_block(ProgressParser.java:4958)
[java] at com.goldencode.p2j.uast.ProgressParser.block(ProgressParser.java:4678)
[java] at com.goldencode.p2j.uast.ProgressParser.external_proc(ProgressParser.java:4605)
[java] at com.goldencode.p2j.uast.AstGenerator.parse(AstGenerator.java:1487)
[java] at com.goldencode.p2j.uast.AstGenerator.processFile(AstGenerator.java:957)
[java] ... 6 more

The problem I think is specific to Windows OS (specifically, the name separator char...); this code in SymbolResolver:

   private String convertSeparator(String name)
   {
      return name.replaceAll("\\.", File.separator); 
   }

   private String convertSeparator(String name)
   {
      return name.replace('.', File.separatorChar);
   }

Edit the p2j/src/com/goldencode/p2j/uast/SymbolResolver.java and apply the change suggested by Constantin, then rebuild FWD using gradlew.bat all. Then try parsing again.

That appears to have worked, at least on a small subset of source files. Thank you for all your help today.

I'm going to kick off a full analysis of the entire application now, but since I'm off until Tuesday I won't get back to it until then if there are more questions.

I've spotted a few possible problems with the parser throwing errors for "good" ABL code.

1. The following code compiles and executes in 10.2B08:

CREATE BROWSE vhBrowse IN WIDGET-POOL "ERRORS-POOL" NO-ERROR
        ASSIGN
            FRAME = FRAME frCheckErrors:HANDLE
            QUERY = QUERY qrQuery:HANDLE
            TITLE = ?
            COL   = 2
            ROW   = 2
            WIDTH = 75
            DOWN  = 11
            VISIBLE = TRUE
            SENSITIVE = TRUE
            COLUMN-SCROLLING = FALSE
            SEPARATORS = TRUE.

During parsing, this code throws the following errors:

[java] Warning: lookup of query '=' failed at line 2274 column 27
     [java] line 2271:65: unexpected token: NO-ERROR
     [java]     at com.goldencode.p2j.uast.ProgressParser.create_browse_stmt(ProgressParser.java:25269)
     [java]     at com.goldencode.p2j.uast.ProgressParser.stmt_list(ProgressParser.java:23788)
     [java]     at com.goldencode.p2j.uast.ProgressParser.statement(ProgressParser.java:6087)

To correct the code for valid parsing, the following changes are required:

DEFINE VARIABLE qh AS HANDLE NO-UNDO.

ASSIGN qh = QUERY qrQuery:HANDLE.

CREATE BROWSE vhBrowse IN WIDGET-POOL "ERRORS-POOL" /*NO-ERROR*/
        ASSIGN
            FRAME = FRAME frCheckErrors:HANDLE
    /*        QUERY = QUERY qrQuery:HANDLE*/
            QUERY = qh
            TITLE = ?
            COL   = 2
            ROW   = 2
            WIDTH = 75
            DOWN  = 11
            VISIBLE = TRUE
            SENSITIVE = TRUE
            COLUMN-SCROLLING = FALSE
            SEPARATORS = TRUE.

2. ABL allows temp-table definitions to have non-reserved keywords as field names. This causes parsing errors. For example:

DEFINE TEMP-TABLE ttOut NO-UNDO
       FIELD Branch AS INTEGER
       FIELD Dept AS CHARACTER
       FIELD InvoiceYear AS INTEGER
       FIELD InvoiceMonth AS INTEGER
       FIELD Shipments AS DECIMAL
       FIELD RETURNS AS DECIMAL
       INDEX pu-idx IS PRIMARY UNIQUE Branch Dept InvoiceYear InvoiceMonth.

The above is a valid temp-table definition that compiles and executes; RETURNS is a non-reserved keyword. The FWD parser throws an error when it encounters the definition of the "RETURNS" field:

[java] com.goldencode.ast.AstException: Error processing .\abl\Source\responseShipmentsAndReturns.p
     [java]     at com.goldencode.p2j.uast.AstGenerator.processFile(AstGenerator.java:962)
     [java]     at com.goldencode.p2j.uast.ScanDriver.lambda$scan$0(ScanDriver.java:375)
     [java]     at com.goldencode.p2j.uast.ScanDriver.scan(ScanDriver.java:410)
     [java]     at com.goldencode.p2j.uast.ScanDriver.scan(ScanDriver.java:248)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.runScanDriver(ConversionDriver.java:496)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.front(ConversionDriver.java:377)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.main(ConversionDriver.java:2005)
     [java] Caused by: java.lang.RuntimeException: Cannot find class/interface DECIMAL in PROPATH.
     [java]     at com.goldencode.p2j.uast.SymbolResolver.loadClass(SymbolResolver.java:2337)
     [java]     at com.goldencode.p2j.uast.ProgressParser.user_defined_type_name(ProgressParser.java:7304)
     [java]     at com.goldencode.p2j.uast.ProgressParser.as_field_clause(ProgressParser.java:50085)
     [java]     at com.goldencode.p2j.uast.ProgressParser.add_temp_table_field(ProgressParser.java:49582)
     [java]     at com.goldencode.p2j.uast.ProgressParser.def_temp_table_stmt(ProgressParser.java:10821)
     [java]     at com.goldencode.p2j.uast.ProgressParser.define_stmt(ProgressParser.java:9436)
     [java]     at com.goldencode.p2j.uast.ProgressParser.stmt_list(ProgressParser.java:23372)
     [java]     at com.goldencode.p2j.uast.ProgressParser.statement(ProgressParser.java:6087)
     [java]     at com.goldencode.p2j.uast.ProgressParser.single_block(ProgressParser.java:4958)
     [java]     at com.goldencode.p2j.uast.ProgressParser.block(ProgressParser.java:4678)
     [java]     at com.goldencode.p2j.uast.ProgressParser.external_proc(ProgressParser.java:4605)
     [java]     at com.goldencode.p2j.uast.AstGenerator.parse(AstGenerator.java:1487)
     [java]     at com.goldencode.p2j.uast.AstGenerator.processFile(AstGenerator.java:957)
     [java]     ... 6 more

3. ABL allows the use of literal Hex values and the compiler correctly interprets them as integer literals. For example:

IF GET-BYTE(mXMLData, 1) <> 0x3C THEN

is okay, but the FWD parser will throw an error:

[java] line 190:34: expecting KW_THEN, found 'x3C'
     [java]     at antlr.Parser.match(Parser.java:211)
     [java]     at com.goldencode.p2j.uast.ProgressParser.then_clause(ProgressParser.java:34804)
     [java]     at com.goldencode.p2j.uast.ProgressParser.if_stmt(ProgressParser.java:29004)
     [java]     at com.goldencode.p2j.uast.ProgressParser.stmt_list(ProgressParser.java:23453)
     [java]     at com.goldencode.p2j.uast.ProgressParser.statement(ProgressParser.java:6087)
     [java]     at com.goldencode.p2j.uast.ProgressParser.single_block(ProgressParser.java:4958)
     [java]     at com.goldencode.p2j.uast.ProgressParser.block(ProgressParser.java:4678)
     [java]     at com.goldencode.p2j.uast.ProgressParser.inner_block(ProgressParser.java:5895)
     [java]     at com.goldencode.p2j.uast.ProgressParser.single_block(ProgressParser.java:4950)
     [java]     at com.goldencode.p2j.uast.ProgressParser.block(ProgressParser.java:4678)
     [java]     at com.goldencode.p2j.uast.ProgressParser.external_proc(ProgressParser.java:4605)
     [java]     at com.goldencode.p2j.uast.AstGenerator.parse(AstGenerator.java:1487)
     [java]     at com.goldencode.p2j.uast.AstGenerator.processFile(AstGenerator.java:957)
     [java]     at com.goldencode.p2j.uast.ScanDriver.lambda$scan$0(ScanDriver.java:375)
     [java]     at com.goldencode.p2j.uast.ScanDriver.scan(ScanDriver.java:410)
     [java]     at com.goldencode.p2j.uast.ScanDriver.scan(ScanDriver.java:248)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.runScanDriver(ConversionDriver.java:496)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.front(ConversionDriver.java:377)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.main(ConversionDriver.java:2005)
     [java] line 193:49: unexpected token: x3C
     [java]     at com.goldencode.p2j.uast.ProgressParser.lvalue(ProgressParser.java:13643)
     [java]     at com.goldencode.p2j.uast.ProgressParser.primary_expr(ProgressParser.java:53111)
     [java]     at com.goldencode.p2j.uast.ProgressParser.chained_object_members(ProgressParser.java:19400)
     [java]     at com.goldencode.p2j.uast.ProgressParser.assign(ProgressParser.java:34739)

To prevent the parser error, change the Hex literal to an integer literal, e.g.

IF GET-BYTE(mXMLData, 1) <> 60 THEN

4. Walking the AST throws an error when it encounters a user-defined function that returns a CLASS:

FUNCTION newAppError RETURNS /*CLASS*/ Progress.Lang.AppError PRIVATE
        (INPUT ipcMessage AS CHARACTER) FORWARD.

results in the following errors thrown during the AST walk, which results in termination of the analysis at that point:

[java] EXPRESSION EXECUTION ERROR:
     [java] ---------------------------
     [java] throwException(errmsg)
     [java] ^  { Unrecognized return type FUNC_CLASS [KW_FUNCT id <1451698947492> 356:1] }
     [java] ---------------------------
     [java] EXPRESSION EXECUTION ERROR:
     [java] ---------------------------
     [java] cls = execLib("get_function_return_class", ftype)
     [java]       ^  { Expression execution error @1:1 }
     [java] ---------------------------
     [java] ERROR:
     [java] com.goldencode.p2j.pattern.TreeWalkException: ERROR!  Active Rule:
     [java] -----------------------
     [java]       RULE REPORT      
     [java] -----------------------
     [java] Rule Type :   WALK
     [java] Source AST:  [ FUNCTION ] BLOCK/FUNCTION/KW_FUNCT/ @356:1 {1451698947492}
     [java] Copy AST  :  [ FUNCTION ] BLOCK/FUNCTION/KW_FUNCT/ @356:1 {1451698947492}
     [java] Condition :  throwException(errmsg)
     [java] Loop      :  false
     [java] --- END RULE REPORT ---
     [java] 
     [java] 
     [java] 
     [java]     at com.goldencode.p2j.pattern.PatternEngine.run(PatternEngine.java:1068)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.processTrees(ConversionDriver.java:1128)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.front(ConversionDriver.java:382)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.main(ConversionDriver.java:2005)
     [java] Caused by: com.goldencode.expr.ExpressionException: Expression execution error @1:7 [KW_FUNCT id=1451698947492]
     [java]     at com.goldencode.p2j.pattern.AstWalker.walk(AstWalker.java:274)
     [java]     at com.goldencode.p2j.pattern.AstWalker.walk(AstWalker.java:209)
     [java]     at com.goldencode.p2j.pattern.PatternEngine.apply(PatternEngine.java:1631)
     [java]     at com.goldencode.p2j.pattern.PatternEngine.processAst(PatternEngine.java:1529)
     [java]     at com.goldencode.p2j.pattern.PatternEngine.processAst(PatternEngine.java:1477)
     [java]     at com.goldencode.p2j.pattern.PatternEngine.run(PatternEngine.java:1032)
     [java]     ... 3 more
     [java] Caused by: com.goldencode.expr.ExpressionException: Expression execution error @1:7
     [java]     at com.goldencode.expr.Expression.execute(Expression.java:484)
     [java]     at com.goldencode.p2j.pattern.Rule.apply(Rule.java:491)
     [java]     at com.goldencode.p2j.pattern.Rule.executeActions(Rule.java:738)
     [java]     at com.goldencode.p2j.pattern.Rule.coreProcessing(Rule.java:707)
     [java]     at com.goldencode.p2j.pattern.Rule.apply(Rule.java:530)
     [java]     at com.goldencode.p2j.pattern.RuleContainer.apply(RuleContainer.java:583)
     [java]     at com.goldencode.p2j.pattern.RuleSet.apply(RuleSet.java:98)
     [java]     at com.goldencode.p2j.pattern.AstWalker.walk(AstWalker.java:261)
     [java]     ... 8 more
     [java] Caused by: com.goldencode.expr.ExpressionException: Expression execution error @1:1
     [java]     at com.goldencode.expr.Expression.execute(Expression.java:484)
     [java]     at com.goldencode.p2j.pattern.Rule.apply(Rule.java:491)
     [java]     at com.goldencode.p2j.pattern.Rule.executeActions(Rule.java:738)
     [java]     at com.goldencode.p2j.pattern.Rule.coreProcessing(Rule.java:707)
     [java]     at com.goldencode.p2j.pattern.Rule.apply(Rule.java:530)
     [java]     at com.goldencode.p2j.pattern.NamedFunction.execute(NamedFunction.java:449)
     [java]     at com.goldencode.p2j.pattern.AstSymbolResolver.execute(AstSymbolResolver.java:709)
     [java]     at com.goldencode.p2j.pattern.CommonAstSupport$Library.execLib(CommonAstSupport.java:1262)
     [java]     at com.goldencode.expr.CE107.execute(Unknown Source)
     [java]     at com.goldencode.expr.Expression.execute(Expression.java:391)
     [java]     ... 15 more
     [java] Caused by: com.goldencode.p2j.pattern.CommonAstSupport$UserGeneratedException: Unrecognized return type FUNC_CLASS [KW_FUNCT id <1451698947492> 356:1]
     [java]     at com.goldencode.p2j.pattern.CommonAstSupport$Library.throwException(CommonAstSupport.java:2528)
     [java]     at com.goldencode.p2j.pattern.CommonAstSupport$Library.throwException(CommonAstSupport.java:2513)
     [java]     at com.goldencode.expr.CE126.execute(Unknown Source)
     [java]     at com.goldencode.expr.Expression.execute(Expression.java:391)
     [java]     ... 24 more

Thank you for finding and documenting these issues. I've moved them into official bug reports:

#3351 query qualifier parsing
#3352 non-reserved keywords as tt field name
#3353 hexidecimal literals
#3354 func_class return value

Your account is authorized to directly create new bug reports. I've made you a "watcher" on each of these so that you can see the headway we make or participate in the discussion.

After I got a clean parse completed with a full ant convert.front, I proceeded with a full ant rpt-no-front which ran for approximately 90 minutes, and completed successfully.

I started the report server with ./deploy/server/report.cmd, registered the certificate, and was able to connect to https://localhost:8443, log in, and change the default admin password.

However all the reports except the Lines of Code Analysis by File and Lines of Code Analysis by Directory are not returning results, and the Call Graph tabs don't work.

For the reports, nothing shows up in the log file.

For the Call Graph tabs, I get the following in the log file:

java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at com.goldencode.p2j.report.server.ReportProtocol.processIncoming(ReportProtocol.java:386)
    at com.goldencode.p2j.report.server.ReportProtocol.onMessage(ReportProtocol.java:239)
    at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.eclipse.jetty.websocket.common.events.annotated.CallableMethod.call(CallableMethod.java:71)
    at org.eclipse.jetty.websocket.common.events.annotated.OptionalSessionCallableMethod.call(OptionalSessionCallableMethod.java:72)
    at org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onTextMessage(JettyAnnotatedEventDriver.java:234)
    at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
    at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:66)
    at org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onTextFrame(JettyAnnotatedEventDriver.java:226)
    at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:162)
    at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:367)
    at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:176)
    at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:105)
    at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.forwardIncoming(CompressExtension.java:142)
    at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:85)
    at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:220)
    at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
    at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:256)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:675)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:505)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
    at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
    at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
    at java.lang.Thread.run(Unknown Source)
    Caused by: java.lang.IllegalAccessError: tried to access method 
    com.google.common.collect.Iterators.emptyIterator()Lcom/google/common/collect/UnmodifiableIterator; from class org.janusgraph.graphdb.query.QueryProcessor
    at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:66)
    at com.google.common.collect.Iterables$5.iterator(Iterables.java:614)
    at org.janusgraph.graphdb.tinkerpop.optimize.JanusGraphStep.lambda$new$64(JanusGraphStep.java:69)
    at org.apache.tinkerpop.gremlin.process.traversal.step.map.GraphStep.processNextStart(GraphStep.java:136)
    at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
    at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:179)
    at com.goldencode.p2j.report.server.CallGraphApi.getMissingExternalTargets(CallGraphApi.java:472)
    ... 37 more
    java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at com.goldencode.p2j.report.server.ReportProtocol.processIncoming(ReportProtocol.java:382)
    at com.goldencode.p2j.report.server.ReportProtocol.onMessage(ReportProtocol.java:239)
    at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.eclipse.jetty.websocket.common.events.annotated.CallableMethod.call(CallableMethod.java:71)
    at org.eclipse.jetty.websocket.common.events.annotated.OptionalSessionCallableMethod.call(OptionalSessionCallableMethod.java:72)
    at org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onTextMessage(JettyAnnotatedEventDriver.java:234)
    at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
    at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:66)
    at org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onTextFrame(JettyAnnotatedEventDriver.java:226)
    at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:162)
    at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:367)
    at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:176)
    at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:105)
    at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.forwardIncoming(CompressExtension.java:142)
    at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:85)
    at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:220)
    at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
    at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:256)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:675)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:505)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
    at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
    at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
    at java.lang.Thread.run(Unknown Source)
    Caused by: java.lang.NullPointerException
    at java.util.LinkedHashSet.<init>(Unknown Source)
    at com.goldencode.p2j.report.server.CallGraphApi.readGraph(CallGraphApi.java:240)
    ... 37 more

The missing details reports is a bug that only occurs on Windows. My colleague Eric has coincidentally been working that bug today for another thread in this same forum. Later tonight, he will post here with details on the fix.

The call graph tabs are only functional if the call graph analysis processing was run. It will run if you use ant callgraph or ant report_server (this is a combination of ant rpt and ant callgraph). Our documentation on the call graph tabs is not written yet. However, most of the Call Graph in FWD v3.0 is still accurate. At least the parts about how to configure the call graph and handle the hints. But the reports and the UI for visualizing the call graph are all new and not yet documented. The call graph processing is very powerful. It will take some time to get the hints right because of the pervasiveness of dynamic invocations (e.g. RUN VALUE(some_runtime_expression)). Take a look at that chapter and I can answer questions in the short term.

Neil Treeby wrote:

... all the reports except the Lines of Code Analysis by File and Lines of Code Analysis by Directory are not returning results

The problem is specific to Windows, and I've committed a fix to our version control system. I don't know when this will be released publicly, so please apply the patch for this, which I've posted at #3356-1, then rebuild FWD.

In testing this fix, I noticed another Windows-specific problem with the File List Filters view, for which I've opened #3357. This issue is not yet fixed, but it will not block you from using the other main features of the analytics tool.

The patch worked, thanks.

Now I'm working on getting a callgraph generated, I assume this will take some time to build since I am going to have to do the whole tree for now; it would take me all day to figure out possible entry points for the rootlist.

One thing I found interesting in the 3.0 documentation for generating the Call Graph - do we really need to specify programs that are executed using RUN program.p ON /*SERVER*/ some-server-hdl.? To my mind such a call is an extension of the call graph of the calling procedure, just running on a different server to the client.

Neil Treeby wrote:

One thing I found interesting in the 3.0 documentation for generating the Call Graph - do we really need to specify programs that are executed using RUN program.p ON /*SERVER*/ some-server-hdl.? To my mind such a call is an extension of the call graph of the calling procedure, just running on a different server to the client.

1. if RUN VALUE .. ON SERVER is used, then this is a ambiguous call-site and will need to be resolved via hints
2. if an explicit program name is set, i.e. RUN program.p ON SERVER, then:
   • if program.p exists in the current code set, then it will be resolved
   • otherwise, program.p will be marked as a Missing Program

So, you need to disambiguate via hints only RUN VALUE cases. There is a report in the callgraph section of the Analytics web app, called Ambiguous Call Sites - this will show you all call-sites which need to be disambiguated.

The documentation for the latest Call Graph support is now complete. Please see these chapters:

Call Graph Analyzer
Call Graph Visualization
Call Graph Reports

Greg Shah wrote:

Thank you for finding and documenting these issues. I've moved them into official bug reports:

#3351 query qualifier parsing
#3352 non-reserved keywords as tt field name
#3353 hexidecimal literals
#3354 func_class return value

Your account is authorized to directly create new bug reports. I've made you a "watcher" on each of these so that you can see the headway we make or participate in the discussion.

All of these issues are resolved in FWD v3.2.0 which is the latest stable public release.