Project

General

Profile

Bug #7645

Simplify _meta database initialization by removing <schema>.meta.xml resource

Added by Ovidiu Maxiniuc 11 months ago. Updated 4 months ago.

Status:
Test
Priority:
High
Target version:
-
Start date:
Due date:
% Done:

100%

billable:
No
vendor_id:
GCD
case_num:
version_reported:
version_resolved:

History

#1 Updated by Ovidiu Maxiniuc 11 months ago

Currently, the population of schema related tables as part of the initialization of _meta database uses two sources:
  • the <schema>.meta.xml xml file generated during conversion and usually stored with the respective DMOs
  • the scan of the dmo/ folder of each schema for interfaces carrying @Table annotation.
We should simplify the process and drop the dependency of the XML resource because of some reasons:
  • only three records are created after processing several MiB of XML. The information is mainly duplicated between the two sources, but this file requires parsing and the 99% of the information is not used. The entries from MetadataManager.SystemTable marked with IGNORE_META_XML are ignored;
  • the creation of the XML itself at conversion time requires additional CPU;
  • the metadata obtained by the mandatory registration of the DMOs with the DmoMetadataManager is richer and freely accessible.
After removing the XML resource:
  • possible ambiguities are eliminated (where is the data source? in the event of a bug, if the sources contain different data which is to be trusted?)
  • the only records which now require the XML are MetaDb, MetaTrans and MetaStartup need new data source (if any);
  • IGNORE_META_XML can/should be dropped;
  • we get small performance boosts at database initialization and schema generation. However, these might not be directly noticeable due the moments when they occur.

The metaschema.xml TRPL code (which generates the XML resource) should be reviewed carefully, to make sure we have every aspect of the metadata which is addressed there covered elsewhere.

#3 Updated by Eric Faulhaber 5 months ago

  • Priority changed from Normal to High
  • case_num set to PI-4893

#6 Updated by Eric Faulhaber 5 months ago

  • Assignee set to Eduard Soltan
  • case_num deleted (PI-4893)

#7 Updated by Eduard Soltan 4 months ago

  • Status changed from New to WIP

#8 Updated by Eduard Soltan 4 months ago

Committed on 7645a. revision 15017.

I moved the logic from metaschema.xml into MetadataManger class, and created a new method for creation, population and saving of new record of each type: MetaDb, MetaFile, MetaIndex and MetaindexComponent.

Also introduced logic for creation of the aux meta record(Actindex, Actlock, etc).

#9 Updated by Eric Faulhaber 4 months ago

Eduard, is this ready for review? Is there anything left to do on this, other than regression testing?

#10 Updated by Eduard Soltan 4 months ago

Eric Faulhaber wrote:

Eduard, is this ready for review? Is there anything left to do on this, other than regression testing?

Yes, it can be reviewed.

#11 Updated by Eduard Soltan 4 months ago

Eduard Soltan wrote:

Eric Faulhaber wrote:

Eduard, is this ready for review? Is there anything left to do on this, other than regression testing?

Yes, it can be reviewed. I tested it on customer tests, and it seems good.

#12 Updated by Alexandru Lungu 4 months ago

  • Status changed from WIP to Review
  • % Done changed from 0 to 100

#13 Updated by Ovidiu Maxiniuc 4 months ago

  • Status changed from Review to WIP
  • % Done changed from 100 to 90

Review of 7645a/15017.

So far, so good. I am OK with the revision, I like the splitting of the code in very specific methods.

Some things I notices in this revision:
  • there are some unused imports (java.net.*, com.goldencode.util.*, org.w3c.dom.*);
  • method saveMetaSetup() can be private (and moved down in the appropriate section). Unless there is there a reason to keep it public? Why rename it to saveMetaRecord, maybe?
  • method populateDatabase():
    • please update the javadoc. It still say "information read from XML files stored in the application jar". This is not the case, any more;
    • it uses { at the end of try / for / while (Java standard). Please move them to next line (GCD code style). Same for method savePropertyMetaRecord();
    • the lists is too vague as an identifier. It might have been suggested by the IDE, but the human reader will understand better the code if a more suggestive name is used (for example: dmoInterfacesList);
    • typo in indecies identifier;
  • methods saveMetaDbRecord(), saveFileMetaRecord(), savePropertyMetaRecord(), saveIndexMetaRecord(), saveIndexComponentMetaRecord(), and createMetaAux():
    • lines dmo.primaryKey(Long.parseLong(String.valueOf(recId))); look strange. Why not using a cast to long? Maybe use long as the type of recId in populateDatabase(), the only place where these methods are being called;
    • maybe change the names of the methods. they do not save anything, just create and initialize a record to be saved later by saveMetaSetup();
    • since the methods also initialize the record to be returned from the passed arguments (like Property, P2JIndex), this fact can be specified in the javadoc, too;
    • I think it would be more readable if the declaration and initialization of dmo is done by a single statement, on a single line;
  • method saveAuxMetaRecords():
    • The multiple lookups in tableSet set can be replaced by the faster and more compact way line this:
            for (String tableName : tableSet)
            {
               switch (tableName)
               {
                  case "_actindex": createMetaAux(p, ++recId, "Actindex"); break;
                  case "_actlock":  createMetaAux(p, ++recId, "Actlock");  break;
                  [...]
               }
            }
      
    • I guess each of these tables will have to be populated with specific values in the future, as the customers will need access to these tables, so this method will be updated at respective moments;
    • method prepareRecord(), fieldClassMap, PK_TYPE fields are not needed any more so they can be dropped;
  • other things I identified while doing the review, not introduced by this update:
    • the SystemTable.forIface() has a String parameter. I wonder whether we can boost the performance by comparing the interfaces using = instead of String.equalsIgnoreCase(). In both cases where this method is called, the interface Class is available.
    • there is a typo with isManadatory() method. Please correct it.
  • question: when the SystemTable._File is created, it uses MetadataManager::populateFileTable() as initializer. This will take each table from the database (method populateFileTable()) and use FileTableRecordBuilder.build() to build a record which will save (setMetaTableRecord()) similar records in _File, _Field and the other meta tables. It looks to me that these pieces of information are at lease overwritten if not duplicated.

There is only one affected file MetadataManager.java. We also need changes in conversion, so that the <schema>.meta.xml resource is not generated any more. Since the meta information is picked from the annotation, this resource is no longer necessary. Therefore I expect there will be a new revision which removes rules/metaschema.xml from FWD project and the code from ConversionDriver.middle() which generates metaschema XML. Please feel free to update the status of the task when these are ready.

#14 Updated by Eric Faulhaber 4 months ago

  • Assignee changed from Eduard Soltan to Ovidiu Maxiniuc

#15 Updated by Eduard Soltan 4 months ago

Committed on 7645a, revision 15018.

In this commit I addressed most of the review issues, however I was investigated the Ovidiu's question. To see if existing code might be used to import metarecords.

#16 Updated by Ovidiu Maxiniuc 4 months ago

  • % Done changed from 90 to 100
  • Status changed from WIP to Review

The fixes are done. I also did a bit of cleanup and code maintenance.
Here is a bit of code which dumps the metadata content:

output to /tmp/7645.out.

for each _Db:
   message "D: " recid(_Db) _Db-name _Db-Type _Db-Comm _Db-Slave.
   for each _File of _Db:
      message "~tT: " recid(_File) _Db-recid _File-Name _File-Number _Template _Owner _User-Misc _File-Label _Tbl-Type _numfld _Prime-Index _Valmsg _Valexp _Dump-name.

      for each _Field of _File:
         message "~t~tF: " recid(_Field) _File-recid _Field-Name _Data-Type _Mandatory _Fld-case _Label _Col-label _Format _Order _Field-Physpos _Width.
      end.

      for each _Index of _File:
         message "~t~tI: " recid(_Index) _File-recid _Index-Name _Unique.

         for each _Index-Field of _Index:
            message "~t~t~tC: " recid(_Index-Field) _Index-recid  _Field-recid  _Index-Seq.
         end.
      end.

   end.
end.

It is focused on the items which are now processed from annotations instead of XML. I compared the output with and without the 7645a and they are identical up to the RECIDs of the records (note that they are used in PARENT-ID relations).

Committed revision 15054 (after rebase).

Although I am pretty confident of the code, it should be reviewed. Thank You!

#17 Updated by Ovidiu Maxiniuc 4 months ago

  • Status changed from Review to Test

Branch 7645a was merged into trunk as rev. 15070 and archived.

Reconversion is not mandatory. But if done the <db-name>.meta.xml files will not be created any more.
You need to update the build script to skip packaging the deprecated artefact (old locations: <pkgroot>/dmo/<db-name>/<db-name>.meta.xml). Otherwise the build script may end with file not found error.

#18 Updated by Eugenie Lyzenko 4 months ago

Ovidiu Maxiniuc wrote:

Branch 7645a was merged into trunk as rev. 15070 and archived.

Reconversion is not mandatory. But if done the <db-name>.meta.xml files will not be created any more.
You need to update the build script to skip packaging the deprecated artefact (old locations: <pkgroot>/dmo/<db-name>/<db-name>.meta.xml). Otherwise the build script may end with file not found error.

Can you please specify what scripts do you mean? How can I find it in specific application project?

#19 Updated by Eric Faulhaber 4 months ago

Eugenie Lyzenko wrote:

Ovidiu Maxiniuc wrote:

Branch 7645a was merged into trunk as rev. 15070 and archived.

Reconversion is not mandatory. But if done the <db-name>.meta.xml files will not be created any more.
You need to update the build script to skip packaging the deprecated artefact (old locations: <pkgroot>/dmo/<db-name>/<db-name>.meta.xml). Otherwise the build script may end with file not found error.

Can you please specify what scripts do you mean? How can I find it in specific application project?

Ovidiu, do we actually copy these specific files around in build.xml (or build_db.xml), or are they just caught up in the more general file spec of a broader copy?

If the former, please provide an example of the necessary changes in the hotel[_gui] project.

If the latter, it seems that the problem will naturally solve itself for any new conversion. Before re-converting an existing project with this new trunk revision for the first time, however, a manual delete of these files from the old location you specified will be necessary.

#20 Updated by Ovidiu Maxiniuc 4 months ago

The hotel project built without any issues, without any additional changes. The .meta.xml files had been generated directly into the <pkgroot>/dmo/<schema>. They were matching the following rule:

      <!-- copy all XML and WSDL documents -->
      <copy todir="${build.home}/classes">
         <fileset dir="${src.home}" includes="**/*.xml **/*.wsdl" />
      </copy>

so they were copied to expected locations. The same for <jar rule, so they got packages as well.

In case of a full/clean conversion like ant deploy.all for hotel projects, the artefacts are deletes with the directories cleaned (generated src, classes, classes.aop, and finally, lib). In case an incremental built is used, it is possible that .meta.xml will reach the final location, but it won't be used since there is no code to load it any more.

Indeed, deleting them is probably a good thing. Having these around (with possible deprecated content, as the source .df might evolve) may be confusing for customers. So deleting them manually is recommended (although not mandatory) if the project is not clean built.

Also available in: Atom PDF