FWD » Core Development » Runtime Infrastructure

Dependency de.intarsys.opensource:jPod is used only in ReportServlet.java and can be replaced by org.apache.pdfbox:pdfbox.

What is the user / pass for the reporting login in testcases? I tried admin/admin, but I get socket closed (1011) - Cannot call method public void com.goldencode.p2j.report.server.ReportProtocol#onConnect(org.eclipse.jetty.websocket.api.Se in the front-end and no logs in the back-end and the loader doesn't disappear.

I assume you've ran the report tasks? The default user is admin and the default password is the app's name as configured in build.properties.

But ReportServlet is from FWD Admin console - for that try admin/test123.

ant report_server gives a lot of errors and eventually fails. Ends up with:

     [java] com.goldencode.ast.AstException: Error processing ./words/words.p
     [java]     at com.goldencode.p2j.uast.AstGenerator.processFile(AstGenerator.java:1034)
     [java]     at com.goldencode.p2j.uast.ScanDriver.lambda$scan$0(ScanDriver.java:406)
     [java]     at com.goldencode.p2j.uast.ScanDriver.scan(ScanDriver.java:442)
     [java]     at com.goldencode.p2j.uast.ScanDriver.scan(ScanDriver.java:271)
     [java]     at com.goldencode.p2j.convert.TransformDriver.runScanDriver(TransformDriver.java:393)
     [java]     at com.goldencode.p2j.convert.TransformDriver.front(TransformDriver.java:258)
     [java]     at com.goldencode.p2j.convert.TransformDriver.executeJob(TransformDriver.java:989)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.main(ConversionDriver.java:1272)
     [java] Caused by: java.lang.NullPointerException
     [java]     at com.goldencode.p2j.uast.ProgressParser.primary_expr(ProgressParser.java:59255)
     [java]     at com.goldencode.p2j.uast.ProgressParser.chained_object_members(ProgressParser.java:23324)
     [java]     at com.goldencode.p2j.uast.ProgressParser.un_type(ProgressParser.java:58952)
     [java]     at com.goldencode.p2j.uast.ProgressParser.prod_expr(ProgressParser.java:58819)
     [java]     at com.goldencode.p2j.uast.ProgressParser.sum_expr(ProgressParser.java:42577)
     [java]     at com.goldencode.p2j.uast.ProgressParser.compare_expr(ProgressParser.java:58405)
     [java]     at com.goldencode.p2j.uast.ProgressParser.log_not_expr(ProgressParser.java:58257)
     [java]     at com.goldencode.p2j.uast.ProgressParser.bitwise_xor_expr(ProgressParser.java:58188)
     [java]     at com.goldencode.p2j.uast.ProgressParser.log_and_expr(ProgressParser.java:58127)
     [java]     at com.goldencode.p2j.uast.ProgressParser.expr(ProgressParser.java:13341)
     [java]     at com.goldencode.p2j.uast.ProgressParser.primary_expr(ProgressParser.java:59304)
     [java]     at com.goldencode.p2j.uast.ProgressParser.chained_object_members(ProgressParser.java:23324)
     [java]     at com.goldencode.p2j.uast.ProgressParser.un_type(ProgressParser.java:58952)
     [java]     at com.goldencode.p2j.uast.ProgressParser.prod_expr(ProgressParser.java:58819)
     [java]     at com.goldencode.p2j.uast.ProgressParser.sum_expr(ProgressParser.java:42577)
     [java]     at com.goldencode.p2j.uast.ProgressParser.compare_expr(ProgressParser.java:58405)
     [java]     at com.goldencode.p2j.uast.ProgressParser.log_not_expr(ProgressParser.java:58257)
     [java]     at com.goldencode.p2j.uast.ProgressParser.bitwise_xor_expr(ProgressParser.java:58188)
     [java]     at com.goldencode.p2j.uast.ProgressParser.log_and_expr(ProgressParser.java:58127)
     [java]     at com.goldencode.p2j.uast.ProgressParser.expr(ProgressParser.java:13341)
     [java]     at com.goldencode.p2j.uast.ProgressParser.where_clause(ProgressParser.java:41703)
     [java]     at com.goldencode.p2j.uast.ProgressParser.record_phrase(ProgressParser.java:40009)
     [java]     at com.goldencode.p2j.uast.ProgressParser.each_first_last_spec(ProgressParser.java:38942)
     [java]     at com.goldencode.p2j.uast.ProgressParser.for_stmt(ProgressParser.java:27301)
     [java]     at com.goldencode.p2j.uast.ProgressParser.block_list(ProgressParser.java:27134)
     [java]     at com.goldencode.p2j.uast.ProgressParser.inner_block(ProgressParser.java:9456)
     [java]     at com.goldencode.p2j.uast.ProgressParser.single_block(ProgressParser.java:8395)
     [java]     at com.goldencode.p2j.uast.ProgressParser.block(ProgressParser.java:7520)
     [java]     at com.goldencode.p2j.uast.ProgressParser.external_proc(ProgressParser.java:7447)
     [java]     at com.goldencode.p2j.uast.AstGenerator.parse(AstGenerator.java:1599)
     [java]     at com.goldencode.p2j.uast.AstGenerator.processFile(AstGenerator.java:1022)
     [java]     ... 7 more
     [java] 
     [java] ./words/words.p
     [java] Failure in file './words/words.p':
     [java] ERROR:
     [java] java.lang.NullPointerException
     [java]     at com.goldencode.p2j.uast.MemberData.lambda$parseFinished$0(MemberData.java:307)
     [java]     at com.goldencode.p2j.uast.MemberData.parseFinished(MemberData.java:328)
     [java]     at com.goldencode.p2j.uast.ClassDefinition.lambda$null$15(ClassDefinition.java:1561)
     [java]     at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
     [java]     at com.goldencode.p2j.uast.ClassDefinition.lambda$parseFinished$16(ClassDefinition.java:1561)
     [java]     at java.util.LinkedHashMap$LinkedKeySet.forEach(LinkedHashMap.java:559)
     [java]     at com.goldencode.p2j.uast.ClassDefinition.parseFinished(ClassDefinition.java:1561)
     [java]     at com.goldencode.p2j.uast.SymbolResolver.lambda$parseFinished$1(SymbolResolver.java:1286)
     [java]     at java.lang.Iterable.forEach(Iterable.java:75)
     [java]     at com.goldencode.p2j.uast.SymbolResolver.parseFinished(SymbolResolver.java:1286)
     [java]     at com.goldencode.p2j.uast.ScanDriver.scan(ScanDriver.java:500)
     [java]     at com.goldencode.p2j.uast.ScanDriver.scan(ScanDriver.java:271)
     [java]     at com.goldencode.p2j.convert.TransformDriver.runScanDriver(TransformDriver.java:393)
     [java]     at com.goldencode.p2j.convert.TransformDriver.front(TransformDriver.java:258)
     [java]     at com.goldencode.p2j.convert.TransformDriver.executeJob(TransformDriver.java:989)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.main(ConversionDriver.java:1272)

But has many more issues. The log is 34MB, can't upload it here.

The ant report_server is related to FWD Analytics and is not related to ReportServlet from our admin console. The issue you've found is something unrelated to this task.

A side observation is that the server struggles with such big data when generating the report (ReportsManager.buildReport) performed before the pdf is sent to the browser. I couldn't find the criteria of splitting the documents, but for so much data it only created two splits. I guess it's not relevant to most customers. Just a note for a future performance testing.

Galya B wrote:

A side observation is that the server struggles with such big data when generating the report (ReportsManager.buildReport) performed before the pdf is sent to the browser. I couldn't find the criteria of splitting the documents, but for so much data it only created two splits. I guess it's not relevant to most customers. Just a note for a future performance testing.

There is the maximal report pages number parameter given in request parameters for a target report. This value splits large documents. I guess that de.intarsys.opensource:jPod was used intentionally because the apache pdf api had memory issue and does not work with the customer reports. I do not know if that memory issue was gone out or fixed with new versions of apache pdf api.

Sergey Ivanovskiy wrote:

Galya B wrote:

A side observation is that the server struggles with such big data when generating the report (ReportsManager.buildReport) performed before the pdf is sent to the browser. I couldn't find the criteria of splitting the documents, but for so much data it only created two splits. I guess it's not relevant to most customers. Just a note for a future performance testing.

There is the maximal report pages number parameter given in request parameters for a target report. This value splits large documents. I guess that de.intarsys.opensource:jPod was used intentionally because the apache pdf api had memory issue and does not work with the customer reports. I do not know if that memory issue was gone out or fixed with new versions of apache pdf api.

There was an issue at one point indeed, but then I added MemoryUsageSetting to pdfMerger.mergeDocuments to define the max size of a partition of data to be kept in memory by the merger and things worked out. By default it uses the whole available memory with MemoryUsageSetting.setupMainMemoryOnly() and this is an issue.

Sergey/Hynek: Please review.

Code review 8656a revisions 15156..15159.

The code changes remove the zipping part for the split pdfs. Was this part of the requirements for this issue?

Otherwise the changes look good.

Hynek Cihlar wrote:

The code changes remove the zipping part for the split pdfs. Was this part of the requirements for this issue?

I've mentioned in #8656-8 that

Method writeSplittedDocument wasn't used since the beginning of the class existence in 2017, so I removed it.

There is no better reasoning behind it.

I did a comparison of the original behavior and output and the new one and shared the results in #8656-8. I can do more load testing, but this is likely to lead to revealing other weak points and non-related changes. I think I've tested the behavior directly related to the change well.

You can merge to trunk now.