FWD » Core Development » Conversion Tools

Some schema changes may not affect all programs in an application. Optimally, this would be limited to those programs that reference a given table (including any field change/add/delete or index change/add/delete or any table-level option change/add/delete) or sequence which is changed. The idea is that most schema changes only affect a small amount of the application yet we force a full reconversion. Let's not do that.

Note: Everything I mention will consider the implementation from 10208a to be part of this task.

1. Mode the schema check before depFiles is populated so that schemasChanged will contain the schemas that were modified.
2. In each converted <file>.ast we have access to db_references annotation and BUFFER_SCOPE ast nodes that contain the name of the database and tables, maybe we can use this information to find if the changed table from the schema is among them. But there should be cases where incremental conversion should not reconvert files even if the changed table is among them:
• Adding/Deleting a field/index/sequence/trigger that is not used

1. How to determine what table/sequence was changed? I suggest using ConversionData to store information about the table, we can compute a hash for each table depending on the fields and options, indexes, and triggers. The same thing should be done to sequences.

1. When we have changed schemas, I iterate over the ones that are modified and call loader.getModifiedSchemaTables(changed) to get the set of tables and sequences that were modified. The set can be empty, meaning that no table or sequence was modified and in this case we can remove the schema from schemasChanged collection. We store the non-empty sets in a map Map<String, Set<String>> schemaTablesChanged = null;.
2. In ConversionData, I created a new table called table_signature with two fields table_sig and hash (I will need to store the type - table/sequence and create a constraint next). Added saveSchemaTableHash() method to insert into the table, isSchemaModified() method to check if there are matching records in the table and containsModifiedSchemaTable() method that will check if a file uses a schema and contains any of the modified tables/sequences. To check if a file needs to be reconverted, db_references annotation is checked to confirm the schema is used and then all BUFFER_SCOPE nodes at the first level are searched in the set of modified tables.

• I am calculating the table hash and overriding before I can compare it
• No difference between sequences/tables stored.

• Even if a sequence is changed, there's no need to reconvert a file for this.

Dănuț Filimon wrote:

Issues:

• I am calculating the table hash and overriding before I can compare it

I have the dict files available and I can load them, I was using importSchema() which saved new hash values. This fixes the issue.

Rebased 6083a to latest trunk/16124.

We could calculate the checksum for the original input text for that table. That would be more consistent with our file-based approach for source code (and presumeably with the file-based approach we take to the .df checking).

Greg Shah wrote:

We could calculate the checksum for the original input text for that table. That would be more consistent with our file-based approach for source code (and presumeably with the file-based approach we take to the .df checking).

This should work.

• same table name, but different databases. This will not work out with the current implementation when inserting into table_signature. The solution will be to store the database and table name and create a primary key for those two fields.

• The original text of the table, e.g.: guest
• The db_signature of the table (annotation of the TABLE node)
• A way to determine is the table/field/index properties were changed (AREA, DESCRIPTION, DUMP-NAME, FORMAT, INITIAL, LABEL, POSITION, MAX-WIDTH, COLUMN-LABEL, ORDER, UNIQUE, PRIMARY, INDEX-FIELD, ASCENDING, DESCENDING) - this is the problem

Dănuț Filimon wrote:

To calculate the checksum for a table, we need:

• The original text of the table, e.g.: guest
• The db_signature of the table (annotation of the TABLE node)
• A way to determine is the table/field/index properties were changed (AREA, DESCRIPTION, DUMP-NAME, FORMAT, INITIAL, LABEL, POSITION, MAX-WIDTH, COLUMN-LABEL, ORDER, UNIQUE, PRIMARY, INDEX-FIELD, ASCENDING, DESCENDING) - this is the problem

I've come up with this method and placed it in AstManager

   public int computeAstHash(Aast ast)
   {
      int result = ast.getType();
      if (ast.getText() != null && !ast.getText().isEmpty())
      {
         result = 31 * result + ast.getText().hashCode();
      }

      for (int i = 0 ; i < ast.getNumImmediateChildren(); i++)
      {
         result = 31 * result + computeAstHash(ast.getChildAt(i));
      }

      return result;
   }

The hash is correct in the end, but the problem I noticed it that even if I change a table it will give the old hash (because I did not load the new schema yet).

• The old hash will be used because I did not load the new schema yet.
• I can't run the SchemaLoader because I will override the old hashes.

If we are operating on the original text from the .df then we would NOT be calculating any changes in the AST.

Greg Shah wrote:

If we are operating on the original text from the .df then we would NOT be calculating any changes in the AST.

I am working with the dict files and not the df file. I already know that the schema file was modified (10208a changes), so I am trying to determine what table was modified after the modified schema is processed into a dict file during incremental conversion.

The original detection of a change to the .df might just be from whitespace changes or ordering differences (content is the same but in different order). Processing based on the ASTs will not necessarily match the same results, when a user might expect it to do so.

We could extend our checksum processing to calculate checksums of sections (e.g. a table and all its fields/indexes) of the .df instead of calculating the checksum for the entire file.

So one section will be like this:

ADD TABLE "guest" 
  AREA "Schema Area" 
  DESCRIPTION "Guest staying in a room" 
  DUMP-NAME "guest" 

ADD FIELD "stay-id" OF "guest" AS integer 
  FORMAT "->,>>>,>>9" 
  INITIAL "0" 
  LABEL "Stay ID" 
  POSITION 2
  MAX-WIDTH 4
  COLUMN-LABEL "Stay ID" 
  ORDER 10

... 

ADD INDEX "stay-order" ON "guest" 
  AREA "Schema Area" 
  UNIQUE
  PRIMARY
  INDEX-FIELD "stay-id" ASCENDING 
  INDEX-FIELD "order" ASCENDING

Yes, and it just needs to be checked by tablename and schema (the table is uniquely named per schema).

The only issue that I can see right now, is that the last table parsed will contain the df footer:

That must be fixed otherwise the last table may always be detected as changed. The . can be used to detect the footer and avoid futher checksumming.

Greg Shah wrote:

The only issue that I can see right now, is that the last table parsed will contain the df footer:

That must be fixed otherwise the last table may always be detected as changed. The . can be used to detect the footer and avoid futher checksumming.

Good point, I committed 6083a/16127.

Rebased 6083a to latest trunk/16147, the branch is now at revision 16154.

Danut, can you document which changes are mandatory for customer projects once 6083a gets into trunk?

Alexandru Lungu wrote:

Danut, can you document which changes are mandatory for customer projects once 6083a gets into trunk?

There's #10536-1 and the required changes for some projects in #10536-5. All projects will need to have their configuration updated, because name_map.xml will be defaulted to the cvt folder.

• name_map.xml must be included in the <project>/build/classes/ and be part of the jar, it will no longer be placed in both the project root and src/<pkgroot>
• name_map.xml no longer needs to be deleted since deleting the cvt folder will also delete the file
• ensure the cvt folder is created by the prepare target (not all projects require this)

EDIT: additional change

I have reconsidered the name_map.xml changes. It was my mistake to force that into this task. Please separate that change into another task (and branch). This will make 6083a a non-breaking change which can be tested and merged safely and quickly.

Greg Shah wrote:

I have reconsidered the name_map.xml changes. It was my mistake to force that into this task. Please separate that change into another task (and branch). This will make 6083a a non-breaking change which can be tested and merged safely and quickly.

I'll make the necessary changes. The changes are not that complex, so I don't think there will be a need for regression testing.

Dănuț Filimon wrote:

Greg Shah wrote:

I have reconsidered the name_map.xml changes. It was my mistake to force that into this task. Please separate that change into another task (and branch). This will make 6083a a non-breaking change which can be tested and merged safely and quickly.

I'll make the necessary changes. The changes are not that complex, so I don't think there will be a need for regression testing.

Committed 6083a/16223. I also had to reintroduce some changes to copy the name_map.xml to the right project location when deleting the mappings.

Code Review Task Branch 6083a Revision 1622

I'm fine with the changes.

Please do enough testing to confirm it is safe.

Greg Shah wrote:

Code REview Task Branch 6083a Revision 1622

I'm fine with the changes.

Please do enough testing to confirm it is safe.

Only incremental conversion needs to be tested, to ensure mappings from name_map.xml are removed properly. I'll go with ETF, ChUI and another small customer application.

Committed 6083a/16224. Fixed file copy and file separator.

Code Review Task Branch 6083a Revision 16224

No objection.