FWD » Core Development » Conversion Tools

Customers have reported that we always convert the schema in incremental mode (see #10175-26). For large projects, this can add a significant amount of time. It doesn't make any sense if the .df files are unchanged. Even if they are changed, then we should force a full conversion instead of incremental.

Can we just disable M0 in incremental mode?

Greg Shah wrote:

Can we just disable M0 in incremental mode?

The problem with is that for #9457 app, front-end schema parsing takes a lot (the problem is from AnnotatedAst.nearest IIRC).

Disabling M0 will need to be done only for the main db, and not the temp schemas. Otherwise, I would suggest disabling schema parsing on front-end for incremental (or maybe just change build.xml to use F0 instead of F2).

Disabling M0 will need to be done only for the main db, and not the temp schemas.

Before #6083, we could implement something to bypass the permanent schemata in M0 during incremental mode. After #6083, there is no need to bypass because we would only convert those tables and nothing more.

Otherwise, I would suggest disabling schema parsing on front-end for incremental (or maybe just change build.xml to use F0 instead of F2).

This seems like a quick win.

As far as I can tell, there doesn't seem to be any information (e.g. hash) for the schema file in the cvtdb database. There's only the cvt/data folder which stores the information about the schema, it will be helpful if we create a table in cvtdb h2 database that can be used for the incremental conversion (just like the FILE_SIGNATURE table).

I don't object, but can we just add the schemata to the FILE_SIGNATURE table?

Greg Shah wrote:

I don't object, but can we just add the schemata to the FILE_SIGNATURE table?

Yes, I will do this instead.

1. SchemaLoader.importSchema() should call ConversionData.saveFileHash(srcfile); to save the df (If the df is not changes, this means that the dict, p2o will be the same).
2. Add SchemaLoader.hasSchemaChanges() to go over the current metadata import file and other import files.
3. Add an additional ConversionData.mustConvertFile that avoids checking for .ast (I am not sure if there are any hint files for schema?)
4. During the incremental conversion check of the conversion files, set mode.schema to false if SchameLoader.hasSchemaChanges() returns false.

The approach seems correct to me.

I am not sure if there are any hint files for schema?

Yes, there are schema hints files. See Conversion Hints.

Constantin: Do we already detect that UAST .hints file changes are a cause to reconvert the associated program?

Greg Shah wrote:

Constantin: Do we already detect that UAST .hints file changes are a cause to reconvert the associated program?

Not at this time. There is also #4867 for .ext-hints

• SchemaLoader - already skipped by the changes
• Schema Fixups (data dictionary)
• P2O Generation (database schema files)
• P2O Post-processing (database schema files)
• Generate Plain Old Java Objects (database DMOs)

Maybe not all of them, but only the conversion steps that process the schemas and create their respective p2o/dict/schema files. Currently investigating...

I created 10208a from trunk/16005 and committed an initial version in 10208a/16006. The changes are for #10208-9.

#10208-13 can be achieved by correctly setting CvtRunMode.skipDBSchema, I made it protected and did the following for a test:

            if (mode.schema)
            {
               SchemaLoader loader = new SchemaLoader();
               mode.schema = loader.hasSchemaChanges();
               ((ConversionDriver.CvtRunMode) mode).skipDBSchema = !mode.schema;
            }

Greg, the skipDBSchema is quite old. I suggest removing it and using the schema flag to do the job.

Dănuț Filimon wrote:

Greg, the skipDBSchema is quite old. I suggest removing it and using the schema flag to do the job.

Doing that would be consistent in M0 and M1 modes but inconsistent in F2 and F3 modes. Please check what the impact is in those cases.

In our weekly meeting, I mentioned that I would not modify how the M0, M1, F2 and F3 modes are used. I decided to take a similar approach and create a similar mode to CvtRunMode called IncrementalCvtRunMode. The only difference is that the IncrementalCvtRunMode will only be merged with the JobDefinition.mode during the incremental conversion.

I committed 10208a/16007. Added IncrementalCvtRunMode and used to skip most of the phases that involve the database schema.

Rebased 10208a to latest trunk/16074, the branch is now at revision 16076.

Code Review Tk Branch 10208a Revision 16078

Overall, I agree with the changes for when we are running incremental conversion. Are the changes to SchemaLoader.importAll() safe in a full conversion mode?

OK, then add a comment to explain that in full conversion mode, all schemata will be detected as "changed" and will be converted.

Committed 10208a/16079, added a detailed javadoc to schemasChanged set class members in TransformDriver and SchemaLoader.

Rebased 10208a to latest trunk/16110, the branch is now at revision 16116.

Found an issue with ETF conversion, I will be investigating.

• schemasChanged = null means all schemas should be converted as we are not running an incremental conversion
• calling getSchemaChanges() will always initialize schemasChanged to a HashSet

I tested incremental conversion for two customers and even if the schema was not changed, it was still running the SchemaLoader and other steps. The commands I used were:

 
   <target name="convert.zset" 
           depends="init, check-zfile-set" 
           description="Convert the 4GL source code to Java source code, using the z-file set file as input." 
           if="zfile-set">
      <record name="cvt_${LOG_STAMP}.log" action="start"/>
      <java classname="com.goldencode.p2j.convert.ConversionDriver" 
            fork="true" 
            failonerror="true" 
            dir="${basedir}">
         <jvmarg value="-server"/>
         <jvmarg value="-Xmx${conversionHeap}"/>
         <jvmarg value="-XX:-OmitStackTraceInFastThrow"/>
         <jvmarg value="-DP2J_HOME=${p2j.home}"/>
         <arg value="-ZId2"/>
         <arg value="f2+m0+cb"/>
         <arg value="${zfile}"/>
         <classpath refid="compile.classpath"/>
      </java>
      <record name="cvt_${LOG_STAMP}.log" action="stop"/>
   </target>

and

   <target name="convert.zset" 
           depends="init, check-zfile-set" 
           description="Convert the 4GL source code to Java source code, using the z-file set file as input." 
           if="zfile-set">
      <record name="cvt_${LOG_STAMP}.log" action="start"/>
      <java classname="com.goldencode.p2j.convert.ConversionDriver" 
            fork="true" 
            failonerror="true" 
            dir="${basedir}">
         <jvmarg value="-server"/>
         <jvmarg value="-Xmx${conversionHeap}"/>
         <jvmarg value="-XX:-OmitStackTraceInFastThrow"/>
         <jvmarg value="-DP2J_HOME=${p2j.home}"/>
         <arg value="-ZId2"/>
         <arg value="f2+m0+cb"/>
         <arg value="${zfile}"/>
         <classpath refid="compile.classpath"/> 
      </java>
      <record name="cvt_${LOG_STAMP}.log" action="stop"/>
   </target>

     [java] ------------------------------------------------------------------------------
     [java] Detecting Frame Interfaces
     [java] ------------------------------------------------------------------------------
     [java]
     [java] Elapsed job time:  00:00:00.072
     [java] EXPRESSION EXECUTION ERROR:
     [java] ---------------------------
     [java] frameInterfaces    = createMapFrameAstKeyToString("frameInterfaces")
     [java]                      ^  { org.h2.jdbc.JdbcSQLDataException: Deserialization failed, cause: "java.lang.NullPointerException: Cannot invoke ""com.goldencode.ast.Aast.duplica>
     [java] ---------------------------
     [java] ERROR:
     [java] java.lang.RuntimeException: ERROR!  Active Rule:
     [java] -----------------------
     [java]       RULE REPORT
     [java] -----------------------
     [java] Rule Type :   INIT
     [java] Source AST:  null
     [java] Copy AST  :  null
     [java] Condition :  frameInterfaces    = createMapFrameAstKeyToString("frameInterfaces")
     [java] Loop      :  false
     [java] --- END RULE REPORT ---
     [java]
     [java]
     [java]
     [java]     at com.goldencode.p2j.pattern.PatternEngine.run(PatternEngine.java:1111)
     [java]     at com.goldencode.p2j.convert.TransformDriver.processTrees(TransformDriver.java:591)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.generateFrames(ConversionDriver.java:845)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.back(ConversionDriver.java:581)
     [java]     at com.goldencode.p2j.convert.TransformDriver.executeJob(TransformDriver.java:1010)
     [java]     at com.goldencode.p2j.convert.ConversionDriver.main(ConversionDriver.java:1315)
     [java] Caused by: com.goldencode.expr.ExpressionException: Expression execution error @1:22
     [java]     at com.goldencode.expr.Expression.execute(Expression.java:495)
     [java]     at com.goldencode.p2j.pattern.Rule.apply(Rule.java:500)
     [java]     at com.goldencode.p2j.pattern.PatternEngine.applyGlobal(PatternEngine.java:1754)
     [java]     at com.goldencode.p2j.pattern.PatternEngine.run(PatternEngine.java:1041)
     [java]     ... 5 more
     [java] Caused by: java.lang.RuntimeException: org.h2.jdbc.JdbcSQLDataException: Deserialization failed, cause: "java.lang.NullPointerException: Cannot invoke ""com.goldencode.ast>
     [java]     at com.goldencode.p2j.convert.db.H2Map.lambda$new$0(H2Map.java:236)
     [java]     at com.goldencode.p2j.convert.db.DBHelper.executeQuery(DBHelper.java:452)
     [java]     at com.goldencode.p2j.convert.db.H2Map.<init>(H2Map.java:239)
     [java]     at com.goldencode.p2j.convert.db.ConversionData.createMap(ConversionData.java:833)
     [java]     at com.goldencode.p2j.pattern.CommonAstSupport$Library.createMapFrameAstKeyToString(CommonAstSupport.java:6309)
     [java]     at com.goldencode.expr.CE14478.execute(Unknown Source)
     [java]     at com.goldencode.expr.Expression.execute(Expression.java:398)
     [java]     ... 8 more
     [java] Caused by: org.h2.jdbc.JdbcSQLDataException: Deserialization failed, cause: "java.lang.NullPointerException: Cannot invoke ""com.goldencode.ast.Aast.duplicate()"" because >
     [java]     at org.h2.message.DbException.getJdbcSQLException(DbException.java:590)
     [java]     at org.h2.message.DbException.getJdbcSQLException(DbException.java:429)
     [java]     at org.h2.message.DbException.get(DbException.java:194)
     [java]     at org.h2.util.JdbcUtils.deserialize(JdbcUtils.java:435)
     [java]     at org.h2.jdbc.JdbcConnection.convertToDefaultObject(JdbcConnection.java:2058)
     [java]     at org.h2.jdbc.JdbcResultSet.getObject(JdbcResultSet.java:525)
     [java]     at com.goldencode.p2j.convert.db.H2Map.lambda$new$0(H2Map.java:211)
     [java]     ... 14 more
     [java] Caused by: java.lang.NullPointerException: Cannot invoke "com.goldencode.ast.Aast.duplicate()" because "ast" is null
     [java]     at com.goldencode.p2j.uast.FrameAstKey.readExternal(FrameAstKey.java:265)
     [java]     at java.base/java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:2306)
     [java]     at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2255)
     [java]     at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1733)
     [java]     at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:509)
     [java]     at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:467)
     [java]     at org.h2.util.JdbcUtils.deserialize(JdbcUtils.java:433)
     [java]     ... 17 more
     [java]

Committed 10208a/16118. SchemaLoader should have schemasChanged passed in runSchemaLoader().

Mutable databases will always cause the schema to be converted, the problem was that I was having the schemasChanged set with the mutable database, but I was not using the collection in runSchemaLoader(). I retested incremental conversion with the fix on the same application and it was ok.

I am not sure about the frames problem since it seems to get stuck there, so I am running another full conversion.

Testing went well, Radu also got the error from #10208-37 and it was related to trunk/16111 (since he converted with trunk/16110 + 10208a, he found the same issue).

Conclusion, testing passed and 10208a can be merged.

I'm OK with a merge but please coordinate with Constantin to ensure that we don't negatively affect customer deliveries for today.