FWD » Core Development » Conversion Tools

I have done some profiling and experiments for hotel_gui conversion, and I have a summary about what is the time spent on conversion.

Let's suppose that the conversion baseline time is 3:30 minutes
I have optimized item 4, 7, 8, 11, and the conversion time has been reduced to 2:47 minutes.
Serializing/Deserializing AST into a binary form instead of XML provided additional improvement just to 2:07 minutes.

I'll put the changes here as it is right now here, and I will do refine them until we reach to a good and safe state.

Another conversion improvement opportunity: implement AST hand-off through conversion stages, which might optimize 5 and 6.

Conversion measurements were done with trunk rev 16665.

• Initial binary AST work.

Teodor, for AST serialization, we still plain .ast files for debugging work. Please default to binary and via a p2j.cfg.xml flag, it can be selected either file or binary. Thanks.

I think we will need a tool to convert the binary form into the text/XML form.

I have done making a tool which translates binary ast to an xml form.

I started doing conversion testing on two projects. On the first project, the conversion time has been reduced from 3 hours to 1.58 hours. For the second project, time spent on conversion has been reduced from 10 hours to 4.19 hours.

I have diffed the generated source code and there is no difference.

Teodor, 11650a has more changes than the binary AST feature. Also, please do not use System.getProperty - the flag needs to be configured via p2j.cfg.xml.

I know, I have already done this, but these are uncommitted yet. I need to test the tool for converting binary AST to XML AST.

Currently, the binary form is used only for .ast and not for .schema, .jast files. I don't think it will provide substantial performance improvement because they don't get serialized to XML as much as .ast files, but should I keep as it is or I should serialize entirely in binary form?

• Made CFG_BINARY_AST configurable through p2j.cfg.xml. Added BinaryAstToXml.java.
• BinaryAstToXml.java is a tool which converts binary AST to a human readable AST XML form.

Teodor, please put the binary AST changes in a separate branch.

.dict and .p2o are also used at runtime, and I'm not sure if is best to leave a binary version or the XML version behind. .jast may also help, they are read in the brew phase.

Otherwise, what extension does the binary AST form have?

Constantin Asofiei wrote:

Otherwise, what extension does the binary AST form have?

They have the same file extension, .ast, for simplicity, and both forms (XML and binary) has some magic bytes, to check which form it is.

It is checked first for binary form (always starting with FAST). If the magic bytes doesn't match, it tries to parse as XML file.

I'm also curious, how did the disk space required for the full conversion change?

Required disk space has been decreased significantly, about 75% for a full conversion on a big project.

Moved micro-optimizations like these for evalLib, dir listing, to 11650b.

11650a now contains only the AST tree serialization in binary form.

• the new files use a 80-char line length limit; the standard length is 110. Please adjust comments/etc.
• XmlFilePlugin should be created with a flag telling it to use 'binary serialization' - do not interrogate Configuration in this class
• EXT_AST should not be needed; in binary-ast mode, any AST file (.schema, .dict, etc) should be managed in binary mode (anything managed by XmlFilePlugin is an AST).
• the binary form uses the same extension as the XML text form. So there is no need to preserve info about the file's extension. And the name of the AST implementation class is already written in the header.
• please enhance the tool to receive a source folder and a destination folder, where:
  • it reads all binary ASTs from source
  • it writes the XML form in the destination folder, using the same structure (and extension, etc)
  • be careful not to overwrite the binary form
• the point is - lets get all AST forms used by conversion to binary format. There may be some changes for runtime loading of the .dict/.p2o.

• previously it took 9h21m and ~60GB of disk space
• now it took 5h24m and ~35GB of disk space

Teodor, actually, can you sub-class XmlFilePlugin for the binary-ast mode? Is more clear this way.

With some uncommitted changes (with jast/dict/schema in binary form), hotel_gui converts in 2:05 minutes, which doesn't include 11650b micro-optimizations.

Teodor, the changes look good. I've tested also Hotel GUI reports and it works.

I ran the conversion for an app, which it failed. Currently investigating the issue.

Fixed the issue on 11650a/r16673, the issue was related to string null coercion, wasn't right for jast (java null parameter was converted to nothing instead of null).

Conversion + import + smoke testing for a large GUI app has passed.
I am investigating the issue reported by Artur and continue testing ETF + another project.

Runtime + conversion testing for ChUI regression tests passed.

Thanks Razvan!
Two more projects conversion + import + regression testing has passed.

• Conversion on the first one completed in 4h24m, compared to the original which is 6h41m, which means an improvement of 34.1%.
  • Memory-wise, the project now takes up 15.7GB instead of 35.6GB.
• Conversion on the second project completed in 6h48m, compared to the original which is 10h6m, which means an improvement of 32.7%.
  • Memory-wise, the project now takes up 24.4GB instead of 56,7GB.

Take the memory comparisons with a grain of salt, as I believe I haven't done them on the exact same revisions. Anyway, I hope I'm not tripping, but overall this looks extremely good.

From Dănuț:

Since I am unable to log into redmine and post on the task, the performance results were:

Teodor Gorghe wrote:

From Dănuț:

Since I am unable to log into redmine and post on the task, the performance results were:

	PG	MariaDB
trunk	22562.71	23476.96
11650a	21357.93	23142.14

It is not clear the units of measure or what is being tested here.

Greg Shah wrote:

It is not clear the units of measure or what is being tested here.

Is this a database import? It isn't clear what test is being executed. I guess it isn't conversion since it should not differ much between databases.

Dănuț also reported the conversion time through email, which has decreased from 18h26m to 8h31m.

Greg Shah wrote:

Is this a database import? It isn't clear what test is being executed. I guess it isn't conversion since it should not differ much between databases.

It is the runtime performance testing for 100 runs. Conversion took 8h31m as mentioned above. deploy + jar + prepare took 531 minutes.

In summary, conversion and runtime testing for all projects has passed.
Constantin, I think this is ready for merge after the trunk release.

Also, 11650b changes are still, which are still relevant, providing additional 20% improvement. I think it can deferred after #3211 to avoid conflicts.

Should we consider a p2j.cfg.xml flag to force all ASTs into XML mode? Using the utility to convert one at a time is not convenient when we are looking at a lot of ASTs.

We would certainly default everything to binary mode.

Teodor Gorghe wrote:

Committed revision 16670 on task branch 11650a:

• Made CFG_BINARY_AST configurable through p2j.cfg.xml. Added BinaryAstToXml.java.

I have to say that being able to read the .ast/.jast is a lot more convenient, this convenience helps us to debug and find the problems faster. Even if we have a way to keep the old readable ast/jast files with the p2j.cfg.xml flag, I feel that we are missing an opportunity to have a faster conversion for testing purposes only because we can't read those files.

Maybe a separate tool can be used to translate the binary ast to the old plain file, it might receive a folder/file that will have to go to this process without being required to run a separate conversion with this feature disabled.

Danut, Teodor already added a tool to decode a single file or a full folder of AST files. So if you convert 'in binary' you can decode the entire cvt/ folder if you want.

This is how to use it:

java -cp deploy/lib/p2j.jar com.goldencode.p2j.util.BinaryAstToXml
Usage: BinaryAstToXml <input.ast> [output.xml]
       BinaryAstToXml <source-dir> <destination-dir>

Example:

java -cp deploy/lib/p2j.jar com.goldencode.p2j.util.BinaryAstToXml cvt cvt_xml
...
Converted 310 binary AST file(s), skipped 122 non-binary file(s), refused 0 destination binary AST(s), 0 failure(s).

Dănuț also reported the conversion time through email, which has decreased from 18h26m to 8h31m.

Danut, please make sure you did not convert the POC with Octavian -numThreads multi-threaded conversion. I say this because on the machine where the POC is installed I delivered tests for Octavian's branch; hopefully you did not include that in the conversion testing. Just make sure it is not a false positive here.

Alexandru Lungu wrote:

Dănuț also reported the conversion time through email, which has decreased from 18h26m to 8h31m.

Danut, please make sure you did not convert the POC with Octavian -numThreads multi-threaded conversion. I say this because on the machine where the POC is installed I delivered tests for Octavian's branch; hopefully you did not include that in the conversion testing. Just make sure it is not a false positive here.

This option was not used.

I related this to #1755, because in that task there is an idea to store the AST directly in a DB to leverage query planner and executor. This idea will conflict with #1755 as both suggest changing the representation of the AST. Of course, we can keep both on-disk and in-DB, but it may be redundant (?).

I don't want to mitigate the work being done already in #11650, but I wonder if we can have the best of both words here: representation of AST as fast as this task shows and the TRPL resolution as fast as suggested in #1755.

If in #1755 we choose to use an H2 database, then the serialization/deserialization will be done by H2, which I doubt is using a verbose XML anyway. As for debugging purposes, maybe using the H2 console or some H2 queries would be enough anyway.

11650a is just a small change, which is centralized to XmlFilePlugin, which implements AST file save as binary AST, instead of in a XML format.

The save-load cycles for each file, for each conversion stage, still takes a huge amount of time, but at least it was reduced to a smaller fraction.

#1755 is the way to speedup even further, as proved in #11650, but the work in there might eliminate the need to serialize ASTs in the filesystem. Also for DB representation, a relational database might be slow because it needs a predefined structure and the insertion/projection speed for a tree-structure data is slower than compared to a non-relational database such as neo4j.

My plan with #1755 was that there would be no filesystem storage.

Teodor, only part which may be problematic after 1752a multi-threaded conversion may be RuleContainer.resolvedFunctions - access to this is not thread-safe.

We'll wait for 1752a to be in trunk, and after that we'll retest this. Also, remind me please, what's the expected gain on a large app conversion?

Please rebase 11650b - after that I'll do the review.

I have rebased 11650b (based of trunk 16746), latest revision is 16748.

Compared the performance of converting hotel_gui.
I see a performance increase of ~19%, trunk finishes conversion in 75 seconds, while 11650b, in 61 seconds.

Currently, I am profiling it once again, and if there is something easy to improve, I will include in this branch. Most of the time is spent on parsing stage, but we will improve that after migration to ANTLR 4.

• fixed the single threaded conversion overhead caused by multi-threaded conversion changes. Was related with AnnotatedAst identity map, and the AST copy was slow, which involved in manually recreating the idMap and lots of hashmap put calls.

The conversion speed can be further improved, but the problem comes to the fact that lots of threads are waiting on the Compiler.getCompiledInstance lock because other threads are also compiling.

To reduce that time spent there, one idea which I have started to prototype is to precompile these expressions during built-time. Right now, there are around 42k compiled expressions which are bundled in jar (currently in p2j.jar, Constantin suggested to bring them in separate jar).
I have a prototype of working pre-compilation, but right now, there is not so much performance difference because I need to cache some reflection objects, like Class.forName, Constructor instances, etc.

I will document here where is the real improvement.

As we discussed at noon, I agree that there is room for improvement here. My point of interest is the primarily the dynamic conversion. The already converted code will surely boost the conversion of the temp-tables and dynamic queries, at least when they are running the first time, in cold mode. Several seconds is a usual time today for waiting the necessary TRPL code to be assembled. After the system gets warmer, the specific caches come into play, but these are out of scope of current discussion.

• each profile (.xml) is converted to a Java class. They will have to be preprocessed so that the .rules are included in the proper order;
• rules-sets gets converted into methods;
• the TRPL functions will map to Java functions;
• we can even keep the original /*comments*/;
• the .tpl will be classes and each declaration a static builder method returning new instances of the templates;
• <rule> tags become if statements, <while> will map to.. Java while statements, etc
• the converter will be probably fast enough and the compiler will give early warnings and errors;
• if needed, in a distant future we could drop the TRPL and write conversion directly in Java if that proves beneficial. The full language is available, instead of constraints of TRPL language. The debugging would be much easier with Java source code available;
• instead of tens of thousands of nameless assembled expressions, the code will be composed of several well structured Java classes.

I have fixed the issues related with #11650-52, but currently, I am a bit disappointed.

The initial measurement showed that there were 51 seconds spent on Expression.getCompiledInstance. After precompiling changes, the time has decreased to just 6 seconds (there are still expressions that are being compiled during runtime, for example, related with callgraph and DDL).

But that difference can't be seen much on multi-threaded conversion (decreasing the total conversion time just to 52 seconds) and also when converting large project, because that gets cached, multi-threading reduces the time spent on compilation, etc.

For now, I will attach the changes here (or I should put in a different branch, eg. 11650c), or maybe put in 11650b.
I think the TRPL compilation during build time into bytecode, or even porting the rules into Scala is a more solid idea (which is way more expressive than Java), because the number is small compared with loading 29k classes (which is required by the conversion) during runtime using Java Reflection API (spending about 5 seconds).

Teodor, lets keep the TRPL expression improvements in a separate branch (or even task?).

Please try a larger project; I suspect the contention problem may be just for the small Hotel GUI project?

As noted in #1747, we won't move to Java source as a replacement to TRPL because TRPL has many features that don't translate directly into Java syntax. Without those features, the Java code becomes much more verbose and fragile.

The point of TRPL is to let the engine and the language take away much of the boilerplate tree processing that would otherwise be needed in normal Java code. That is a core design requirement.

With that said, we have considered moving to a JVM-hosted language AND also the idea of making TRPL a first class language (still JVM hosted) in its own right (e.g. IDE support for editing, source level debugging etc...).

In 2009, I did a prototype of a multithreaded TRPL engine using a more pure-Java approach. It still remains in the source tree but was never turned into something real. See the code in com.goldencode.trpl.*.

Another optimisation idea: it is probably not much speedup, but will also make the generated code cleaner.

So: instead of emitting the wrappers like new integer(), new logical(), etc, we should be using the lookup forms: integer.of(), logical.of(). Or, even better, the constants: logical.TRUE, integer.UNKNOWN, etc and even extend the set with integer.ZERO (or integer._0). The constants will have a good impact on runtime since no new objects are constructed when not needed and also they are not map looked up like the static .of() variant do.

As noted above, the expected impact is low since there are not so much instances of constants used in 4GL code to be processed.