# DataModelWorker.java

    /*
    ** Module   : DataModelWorker.java
    ** Abstract : Pattern worker with specific knowledge of the data model
    **
    ** Copyright (c) 2005-2023, Golden Code Development Corporation.
    **
    ** -#- -I- --Date-- --JPRM-- -----------------------------------Description-----------------------------------
    ** 001 ECF 20050426   @21002 Created initial version. Modeled after
    **                           uast.JavaPatternWorker.
    ** 002 ECF 20050526   @21303 Simplified worker library methods. The new
    **                           expression engine implementation enables us
    **                           to reference objects directly in expressions,
    **                           so much of the worker library implementation
    **                           which jumped through hoops to use AST ids in
    **                           lieu of object references was eliminated.
    ** 003 ECF 20050614   @21536 Added capability to sort class properties.
    **                           The order associated with the peer field's
    **                           "order" value is used to sort into ascending
    **                           order. Also removed visitAst to prevent root
    **                           node creation when visiting a previously
    **                           generated AST. This is now done from rulesets
    **                           using the createRootAst user function.
    ** 004 ECF 20050714   @21703 Added capability to sort class ASTs within a
    **                           data model root AST. Classes are sorted by
    **                           dependency, with dependent classes ordered
    **                           after those classes upon which they are
    **                           dependent. Also removed unused methods.
    ** 005 ECF 20050720   @21776 Modified sortProperties method. Remove
    **                           temporary "order" annotation from properties
    **                           after the sort completes.
    ** 006 ECF 20050805   @22012 Rewrote algorithm to sort class ASTs by
    **                           dependency. The old approach did not always
    **                           properly resolve dependencies.
    ** 007 ECF 20051005   @22961 Change to P2O filename generation. Now ".p2o"
    **                           replaces the ".schema" extension, instead of
    **                           being appended to it.
    ** 008 ECF 20051027   @23215 Added user functions. normalizeUniqueIndexes
    **                           calculates minimal set of unique constraints
    **                           for a list of indexes. nonRedundantIndexes
    **                           calculates the non-redundant subset of a list
    **                           of indexes.
    ** 009 ECF 20060504   @25998 Fixed property sorting. Collections.sort()
    **                           was not working properly in the temp table
    **                           case. Changed collection type from TreeSet to
    **                           ArrayList.
    ** 010 GES 20070629   @34339 Removed terse flag from AST persistence so
    **                           that we retain line/column info in our ASTs.
    ** 011 ECF 20080730   @39270 Reduced memory footprint. Implemented finish
    **                           method to clean up resources. Removed current
    **                           instance variable, which held an unused, hard
    **                           reference to a potentially very large AST.
    ** 012 GES 20090515   @42219 Moved to AstManager from AstRegistry/AstPersister.
    ** 013 GES 20090518   @42395 Import change.
    ** 014 ECF 20121220          Fixed sortClasses to accommodate sequence ASTs,
    **                           which are siblings of class ASTs. This method
    **                           assumed class ASTs were the only children of the
    **                           data model root AST.
    ** 015 SVL 20130624          Added support for virtual files.
    ** 016 ECF 20131001          Added virtualFile parameter to additional methods; added
    **                           loadRelatedTree user function.
    ** 017 ECF 20131031          Fixed regression for runtime conversion.
    ** 018 OM  20140117          Added fieldId annotation for runtime index access.
    ** 019 VMN 20140202          Added shift of order attribute for denormalized fields.
    ** 020 VMN 20140502          Added helper methods for processing backticks.
    ** 021 ECF 20150328          Added support for cross-schema natural joins.
    ** 022 ECF 20150715          Replace StringBuffer with StringBuilder.
    ** 023 ECF 20160215          Fixed cross-referencing of denormalized extent fields to original
    **                           field ASTs in .schema file. If a peerid already exists, don't
    **                           overwrite it. This effectively leaves field AST linked to first,
    **                           exploded, property AST.
    ** 024 OM  20160905          Small optimization.
    ** 025 CA  20200412          Added incremental conversion support.
    ** 026 CA  20200416          Reduce the memory footprint for incremental conversion.
    ** 027 CA  20200714          Reserved properties have their 'fieldid' already set by SchemaDictionary.
    ** 028 OM  20200924          P2JIndexComponent carries multiple information to avoid map lookups for them.
    ** 029 CA  20211214          The AST manager plugin must be context-local, for runtime conversion, as the
    **                           InMemoryRegistryPlugin is not thread-safe.
    **                           All worker's state must be context-local, for runtime conversion to work in
    **                           multi-context mode, as the pattern workers are singletons.
    **     OM  20220713          The fieldId is kept for all denormalized properties of same legacy field.
    **     OM  20220721          Added [propertyId] annotation for tables with denormalized fields.
    **                           Fixed [PropertyComparator] implementation.
    ** 030 GBB 20230512          Logging methods replaced by CentralLogger/ConversionStatus.
    */

    /*
    ** This program is free software: you can redistribute it and/or modify
    ** it under the terms of the GNU Affero General Public License as
    ** published by the Free Software Foundation, either version 3 of the
    ** License, or (at your option) any later version.
    **
    ** This program is distributed in the hope that it will be useful,
    ** but WITHOUT ANY WARRANTY; without even the implied warranty of
    ** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    ** GNU Affero General Public License for more details.
    **
    ** You may find a copy of the GNU Affero GPL version 3 at the following
    ** location: https://www.gnu.org/licenses/agpl-3.0.en.html
    **
    ** Additional terms under GNU Affero GPL version 3 section 7:
    **
    **   Under Section 7 of the GNU Affero GPL version 3, the following additional
    **   terms apply to the works covered under the License.  These additional terms
    **   are non-permissive additional terms allowed under Section 7 of the GNU
    **   Affero GPL version 3 and may not be removed by you.
    **
    **   0. Attribution Requirement.
    **
    **     You must preserve all legal notices or author attributions in the covered
    **     work or Appropriate Legal Notices displayed by works containing the covered
    **     work.  You may not remove from the covered work any author or developer
    **     credit already included within the covered work.
    **
    **   1. No License To Use Trademarks.
    **
    **     This license does not grant any license or rights to use the trademarks
    **     Golden Code, FWD, any Golden Code or FWD logo, or any other trademarks
    **     of Golden Code Development Corporation. You are not authorized to use the
    **     name Golden Code, FWD, or the names of any author or contributor, for
    **     publicity purposes without written authorization.
    **
    **   2. No Misrepresentation of Affiliation.
    **
    **     You may not represent yourself as Golden Code Development Corporation or FWD.
    **
    **     You may not represent yourself for publicity purposes as associated with
    **     Golden Code Development Corporation, FWD, or any author or contributor to
    **     the covered work, without written authorization.
    **
    **   3. No Misrepresentation of Source or Origin.
    **
    **     You may not represent the covered work as solely your work.  All modified
    **     versions of the covered work must be marked in a reasonable way to make it
    **     clear that the modified work is not originating from Golden Code Development
    **     Corporation or FWD.  All modified versions must contain the notices of
    **     attribution required in this license.
    */
    package com.goldencode.p2j.schema;

    import java.io.*;
    import java.util.*;
    import java.util.function.*;
    import java.util.logging.*;

    import com.goldencode.ast.*;
    import com.goldencode.p2j.cfg.*;
    import com.goldencode.p2j.convert.*;
    import com.goldencode.p2j.convert.db.*;
    import com.goldencode.p2j.pattern.*;
    import com.goldencode.p2j.persist.*;
    import com.goldencode.p2j.security.*;
    import com.goldencode.p2j.xml.*;

    /**
     * A pattern worker that provides services to support the conversion of Progress schema data into
     * a relational data model AST.
     */
    public final class DataModelWorker
    extends AbstractConversionWorker
    implements DataModelTokenTypes
    {
       /** Logger. */
       private static final ConversionStatus LOG = ConversionStatus.get(DataModelWorker.class);

       /** File extension appended to schema filenames */
       private static final String P2O_POSTFIX = ".p2o";

       /** Context local work area. */
       private static final ContextLocal<WorkArea> context = new ContextLocal<WorkArea>()
       {
          protected WorkArea initialValue()
          {
             return new WorkArea();
          };
       };

       /**
        * Worker routine to centralize creation of new data model ASTs.  All ID
        * setup and parent linkages are established if the parent AST is passed
        * as a parameter, otherwise only an unidentified, standalone AST is
        * created. If a parent AST is passed, a resolver instance should also
        * be provided.
        *
        * @param    type
        *           The token type for the data model AST.
        * @param    text
        *           The text for the AST.
        * @param    parent
        *           The parent AST node or <code>null</code>.
        * @param    resolver
        *           The current instance of the {@link AstSymbolResolver} in use
        *           or <code>null</code>.
        *
        * @return   The AST that was created.
        */
       public static DataModelAst createAst(int type,
                                            String text,
                                            Aast parent,
                                            AstSymbolResolver resolver)
       {
          DataModelAst ast = new DataModelAst();
          initializeAst(ast, type, text, parent, resolver);

          return ast;
       }

       /**
        * Worker routine for peer AST creation.  All ID setup and parent linkages
        * are established and bidirectional peer ID annotations are written
        * in source and target nodes.
        *
        * @param    type
        *           The token type for the AST.
        * @param    text
        *           The text for the AST.
        * @param    parent
        *           The parent AST node.
        * @param    resolver
        *           The current instance of the {@link AstSymbolResolver} in use.
        *
        * @return   The AST that was created.
        */
       public static DataModelAst createPeerAst(Aast source,
                                                int type,
                                                String text,
                                                Aast parent,
                                                AstSymbolResolver resolver)
       {
          DataModelAst target = createAst(type, text, parent, resolver);

          // save our cross-references; we do not use base class' crossReferencePeerAsts method,
          // because we do not want to overwrite an existing peer id, if it exists
          if (!source.isAnnotation(PEER_ID))
          {
             source.putAnnotation(PEER_ID, target.getId());
          }
          if (!target.isAnnotation(PEER_ID))
          {
             target.putAnnotation(PEER_ID, source.getId());
          }

          return target;
       }

       /**
        * Default constructor which initializes libraries.
        */
       public DataModelWorker()
       {
          super();

          setLibrary(new Library());
       }

       /**
        * Worker routine to centralize creation of a new data model AST that is
        * a root node.  A new file ID is registered in the {@link AstManager}
        * and set into the AST.
        *
        * @param   peerRoot
        *          Root node of Progress schema AST currently being processed.
        * @param   virtualFile
        *          If <code>true</code> the file is not supposed to be persisted into the file system,
        *          the file name is used to map some ID to the AST
        *
        * @return  The root AST that was created.
        *
        * @throws  ConfigurationException
        *          if P2J environment is not properly initialized.
        */
       public Aast createRootAst(Aast peerRoot, boolean virtualFile)
       throws ConfigurationException
       {
          String filename = (virtualFile ? peerRoot.getText() : generateFilename(peerRoot));

          return createRootAst(peerRoot, filename, virtualFile);
       }

       /**
        * Worker routine to centralize creation of a new data model AST that is
        * a root node.  A new file ID is registered in the {@link AstManager}
        * and set into the AST.
        *
        * @param   peerRoot
        *          Root node of Progress schema AST currently being processed.
        * @param   filename
        *          The file in which to persist the new AST.
        * @param   virtualFile
        *          If <code>true</code> the file is not supposed to be persisted into the file system,
        *          the file name is used to map some ID to the AST
        *
        * @return  The root AST that was created.
        *
        * @throws  ConfigurationException
        *          if P2J environment is not properly initialized.
        */
       public Aast createRootAst(Aast peerRoot, String filename, boolean virtualFile)
       throws ConfigurationException
       {
          WorkArea wa = context.get();

          // reset the current/root AST to null
          wa.root = null;
          getResolver().setTargetRootAst(null);

          // create a new target tree
          wa.root = createAst(DATA_MODEL, "data model", null, null);

          if (virtualFile)
          {
             initializeRootAst(wa.root, filename, getResolver());
          }
          else
          {
             File file = Configuration.toFile(filename);
             initializeRootAst(wa.root, file, getResolver());
          }

          getResolver().setTargetRootAst(wa.root);

          // cross-reference peerid annotations
          peerRoot.putAnnotation("peerid", wa.root.getId());
          wa.root.putAnnotation("peerid", peerRoot.getId());

          return wa.root;
       }

       /**
        * Resolve <code>constant</code> to a literal which can be compiled into
        * expressions.
        *
        * @param   constant
        *          Constant indicating a data model token name.
        *
        * @return  Token type associated with the token name.
        */
       public Object resolveConstant(String constant)
       {
          return DataModelAst.resolveConstant(constant);
       }

       /**
        * Hook to provide termination processing for a pattern worker.
        */
       public void finish()
       {
          context.get().root = null;

          super.finish();
       }

       /**
        * Generate a unique data model AST persistence filename based on the
        * current source AST filename. The source file name's extension is
        * removed and replaced with ".p2o".
        *
        * @param   ast
        *          Current source AST.
        *
        * @return   A unique filename based on the source file root name with
        *          ".p2o" appended.
        */
       private String generateFilename(Aast ast)
       {
          String filename = ast.getFilename();
          filename = filename.substring(0, filename.lastIndexOf("."));
          StringBuilder buf = new StringBuilder(filename);
          buf.append(P2O_POSTFIX);

          return buf.toString();
       }

       /**
        * Tests one class AST for dependence upon another. The dependency test
        * includes the tested class <code>x</code>, as well as the entire tree
        * of its dependencies. In other words, test whether <code>x</code> <i>or
        * any class upon which it is dependent</i> is dependent upon
        * <code>y</code>.
        *
        * @param   x
        *          Class AST node whose dependencies are being checked.
        * @param   y
        *          Class AST node against which to test <code>x</code> for
        *          dependence.
        *
        * @return  <code>true</code> if <code>x</code> (or any of its dependent
        *          classes) is dependent upon <code>y</code>.
        */
       private boolean isXDependentUponY(Aast x, Aast y)
       {
          Long testId = y.getId();
          AstSymbolResolver resolver = AstSymbolResolver.getResolver();
          int len = x.annotationSize("dependency");
          for (int i = 0; i < len; i++)
          {
             Long depId = (Long) x.getAnnotation("dependency", i);
             Aast depAst = resolver.getAst(depId);

             if (depAst == null)
             {
                // dependency references a different schema; we don't care about these for sorting
                // purposes
                return false;
             }

             if (depId.equals(testId) || isXDependentUponY(depAst, y))
             {
                return true;
             }
          }

          return false;
       }

       /**
        * Helper to create, edit, move and delete data model ASTs.
        */
       public class Library
       {
          /** The XML <code>schema</code> element for the temporary schema. */
          private XmlAst schemaElem = null;

          /** The list of <code>class</code> elements in dmo_index.xml. */
          private List<DMOElementInfo> tempSchema = new LinkedList<>();

          /**
           * Create a new data model AST that is the root for the data model
           * hierarchy. This can later be retrieved with the {@link #getAstRoot}
           * user function.
           *
           * @param   peerRoot
           *          Root node of Progress schema AST currently being processed.
           * @param   virtualFile
           *          If <code>true</code> the file is not supposed to be persisted into the file
           *          system, the file name is used to map some ID to the AST
           *
           * @return  The root AST that was created.
           *
           * @throws  ConfigurationException
           *          if P2J environment is not properly initialized.
           */
          public Aast createRootAst(Aast peerRoot, boolean virtualFile)
          throws ConfigurationException
          {
             return DataModelWorker.this.createRootAst(peerRoot, virtualFile);
          }

          /**
           * Create a new data model AST that is the root for the data model
           * hierarchy. This can later be retrieved with the {@link #getAstRoot}
           * user function.
           *
           * @param   peerRoot
           *          Root node of Progress schema AST currently being processed.
           * @param   filename
           *          The file in which to persist the new AST.
           * @param   virtualFile
           *          If <code>true</code> the file is not supposed to be persisted into the file
           *          system, the file name is used to map some ID to the AST
           *
           * @return  The root AST that was created.
           *
           * @throws  ConfigurationException
           *          if P2J environment is not properly initialized.
           */
          public Aast createRootAst(Aast peerRoot, String filename, boolean virtualFile)
          throws ConfigurationException
          {
             return DataModelWorker.this.createRootAst(peerRoot, filename, virtualFile);
          }

          /**
           * General purpose data model AST creation function which can create a
           * <code>DataModelAst</code> node using the given token type and text.
           * The created node will then be attached to the specified parent. This
           * is used in cases where the conversion is simple enough to drive
           * it completely from the rule set rather than from a custom worker.
           *
           * @param    type
           *           The token type for the data model AST.
           * @param    text
           *           The text for the AST.
           * @param    parent
           *           The ID of the parent AST node.
           *
           * @return   The unique ID if the AST was created successfully or
           *           <code>null</code> if the parent ID is invalid, if the type
           *           is not valid or on any unexpected error during AST
           *           creation.
           */
          public Aast createAst(int type, String text, Aast parent)
          {
             return DataModelWorker.createAst(type, text, parent, getResolver());
          }

          /**
           * General purpose data model AST creation function which can create a
           * <code>DataModelAst</code> node using the given token type and text.
           * The created node will then be attached to the specified parent. Most
           * importantly, on success the ID of the generated AST node will be
           * stored in the current source AST node's "peerid" annotation. This
           * provides a standard mechanism to duplicate the structure of a source
           * AST in the target data model AST.  The source AST's ID will also
           * be stored in the target AST's "peerid" annotation which allows
           * subsequent bidirectional referencing between these peers. This
           * is used in cases where the conversion is simple enough to drive
           * it completely from the rule set rather than from a custom worker.
           *
           * @param    type
           *           The token type for the data model AST.
           * @param    text
           *           The text for the AST.
           * @param    parent
           *           The ID of the parent AST node.
           *
           * @return   The unique ID if the AST was created successfully or
           *           <code>null</code> if the parent ID is invalid, if the type
           *           is not valid or on any unexpected error during AST
           *           creation.
           */
          public Aast createPeerAst(int type, String text, Aast parent)
          {
             return DataModelWorker.createPeerAst(getCopy(),
                                                  type,
                                                  text,
                                                  parent,
                                                  getResolver());
          }

          /**
           * Return the AST node that is the root of the data model AST hierarchy.
           *
           * @return  Data model AST root node.
           */
          public Aast getAstRoot()
          {
             return context.get().root;
          }

          /**
           * Set the AST node that is the root of the data model AST hierarchy and make it the target
           * root AST of the resolver.
           */
          public void setAstRoot(Aast root)
          {
             context.get().root = (DataModelAst) root;
             getResolver().setTargetRootAst(root);
          }

          /**
           * Common case for data model AST creation which creates a
           * <code>DataModelAst</code> node using the passed token type and text
           * and then attaches that node to the parent specified by the "peerid"
           * annotation of the source AST's parent node. This greatly simplifies
           * creation of entire hierarchies of structure that match between the
           * source and target ASTs.  This uses the services of {@link
           * #createPeerAst(int,String,Aast)}.
           *
           * @param    type
           *           The token type for the data model AST.
           * @param    text
           *           The text for the AST.
           *
           * @return   The unique ID if the AST was created successfully or
           *           <code>null</code> if the parent ID is invalid, if the type
           *           is not valid or on any unexpected error during AST
           *           creation.
           */
          public Aast createPeerAst(int type, String text)
          {
             Long id = (Long) getCopy().getParent().getAnnotation(PEER_ID);
             if (id == null)
             {
                return null;
             }
             Aast parentPeer = getResolver().getAst(id);

             return createPeerAst(type, text, parentPeer);
          }

          /**
           * Returns a formatted string using the specified format string and parameter optionally
           * containing backticks.
           *
           * @param    format
           *           Format string.
           * @param    parameter
           *           String parameter optionally containing backticks..
           *
           * @return   Returns a formatted string using the specified format string and parameter
           *           optionally containing backticks.
           */
          public String formatWithEscapedParameter(String format, String parameter)
          {
             return DBUtils.formatWithEscapedParameter(format,  parameter);
          }

          /**
           * Returns a formatted string using the specified format string and arguments optionally
           * containing backticks.
           *
           * @param    format
           *           Format string.
           * @param    parameter
           *           String parameter optionally containing backticks..
           * @param    index
           *           Nullable value.
           *
           * @return   Returns a formatted string using the specified format string and arguments
           *           optionally containing backticks.
           */
          public String formatWithEscapedParameter(String format, String parameter, Long index)
          {
             return DBUtils.formatWithEscapedParameter(format,  parameter, index);
          }

          /**
           * Returns a formatted string using the specified format string and arguments optionally
           * containing backticks.
           *
           * @param    format
           *           Format string.
           * @param    table
           *           String parameter optionally containing backticks.
           * @param    param
           *           String parameter optionally containing backticks.
           *
           * @return   Returns a formatted string using the specified format string and arguments
           *           optionally containing backticks.
           */
          public String formatWithEscapedParameters(String format, String table, String param)
          {
             return DBUtils.formatWithEscapedParameters(format,  table, param);
          }

          /**
           * Shifts order attribute for denormalized fields and sorts <code>PROPERTY</code> children
           * ASTs of <code>parent</code> in place into ascending order, according to the numeric
           * value specified by the "order" annotation. Sibling ASTs of other types are left in
           * place.
           *
           * @param   parent
           *          Parent AST of type <code>CLASS</code>.
           */
          public void sortProperties(Aast parent)
          {
             List<Aast> list = new ArrayList<>();
             Aast child = parent.getImmediateChild(PROPERTY, null);
             while (child != null)
             {
                list.add(child);
                child = parent.getImmediateChild(PROPERTY, child);
             }

             list.sort(new PropertyComparator());

             // Shift order if repeatable orders are found (used if some properties are denormalized)
             Long lastOrder = Long.MIN_VALUE;
             Long currentOrder;
             long shift = 0;
             Iterator<Aast> iter = list.iterator();
             while (iter.hasNext())
             {
                child = iter.next();
                currentOrder = (Long) child.getAnnotation("order");
                if (currentOrder != null)
                {
                   if (currentOrder.equals(lastOrder))
                   {
                      shift++;
                   }
                   if (shift > 0)
                   {
                      child.putAnnotation("order", shift + currentOrder);
                   }
                   lastOrder = currentOrder;
                }
             }

             // Replace temporary "order" annotation with "fieldId" annotation
             long fieldId = 0;
             long propertyId = 0;
             iter = list.iterator();
             String lastLegacyName = null;
             while (iter.hasNext())
             {
                child = iter.next();

                if (child.isAnnotation("fieldId"))
                {
                   continue;
                }

                // Commented because this order is used for metadata generation
                //    child.removeAnnotation("order");
                String legacyName = (String) child.getAnnotation("historical");
                if (!legacyName.equalsIgnoreCase(lastLegacyName))
                {
                   lastLegacyName = legacyName;
                   ++ fieldId;
                }
                child.putAnnotation("fieldId", fieldId);

                // [fieldId] increments only for each filed but [propertyId] for each property
                ++ propertyId;
                child.putAnnotation("propertyId", propertyId);
             }

             // if denormalized fields were encountered, the number of properties is larger than those of fields
             parent.putAnnotation("hasDenormalizedFields", fieldId != propertyId);

             // If list contains 1 or fewer entries, no need to sort.
             if (list.size() <= 1)
             {
                return;
             }

             Aast preProp = null;
             Aast postProp = null;
             child = (Aast) parent.getFirstChild();

             // Detect first non-PROPERTY child before PROPERTY ASTs.
             while (child != null)
             {
                if (child.getType() == PROPERTY)
                {
                   break;
                }
                preProp = child;
                child = (Aast) child.getNextSibling();
             }

             // Detect first non-PROPERTY child after PROPERTY ASTs.
             while (child != null)
             {
                if (child.getType() != PROPERTY)
                {
                   postProp = child;
                   break;
                }
                child = (Aast) child.getNextSibling();
             }

             // Determine first and last PROPERTY ASTs, chaining together
             // siblings in between.
             Aast first = null;
             Aast last = null;
             iter = list.iterator();
             for (int i = 0; iter.hasNext(); i++)
             {
                child = iter.next();
                if (i == 0)
                {
                   first = child;
                }

                if (last != null)
                {
                   last.setNextSibling(child);
                }

                last = child;
             }

             // Link first PROPERTY AST to head of list or to parent.
             if (preProp == null)
             {
                parent.setFirstChild(first);
             }
             else
             {
                preProp.setNextSibling(first);
             }

             // Link last PROPERTY AST to tail of list (may be null).
             last.setNextSibling(postProp);
          }

          /**
           * Sorts <code>CLASS</code> children ASTs of <code>parent</code>
           * in place into ascending order, according to the association
           * dependencies of each class. Classes are sorted after classes upon
           * which they have dependencies. Within dependency levels, they are
           * sorted alphabetically by class name.
           *
           * @param   parent
           *          Parent AST of type <code>DATA_MODEL</code>.
           */
          public void sortClasses(Aast parent)
          {
             SortedSet<Aast> set = new TreeSet<Aast>(new ClassComparator());
             Aast child = parent.getImmediateChild(CLASS, null);
             while (child != null)
             {
                set.add(child);
                child = parent.getImmediateChild(CLASS, child);
             }

             // If set contains 1 or fewer entries, no need to sort.
             if (set.size() <= 1)
             {
                return;
             }

             // Remove all CLASS nodes from their original locations.
             Iterator<Aast> iter = set.iterator();
             while (iter.hasNext())
             {
                Aast cls = iter.next();
                cls.remove();
             }

             // Classes are now sorted into 2 broad categories:  independent and
             // dependent. They are also sorted alphabetically within those
             // categories. We must now sort out the dependent classes to account
             // for the actual dependencies.
             List<Aast> list = new LinkedList<Aast>();
             iter = set.iterator();
             while (iter.hasNext())
             {
                Aast cls = iter.next();
                if (cls.annotationSize("dependency") == 0)
                {
                   // Because of the first level sort, all independent classes
                   // will always be added before any dependent classes.
                   list.add(cls);
                   continue;
                }

                // Walk backwards through the list until we find a class upon
                // which the current class is dependent. Add the current class
                // just after it. Not the most efficient algorithm, but it will
                // do...
                int pos = list.size();
                ListIterator<Aast> subIter = list.listIterator(pos);
                while (subIter.hasPrevious())
                {
                   Aast next = subIter.previous();
                   if (isXDependentUponY(cls, next))
                   {
                      break;
                   }
                   pos--;
                }

                list.add(pos, cls);
             }

             /*
             // Debug output.
             AstSymbolResolver resolver = AstSymbolResolver.getResolver();
             iter = list.iterator();
             while (iter.hasNext())
             {
                Aast next = iter.next();
                System.out.println(next.getText());
                int size = next.annotationSize("dependency");
                for (int i = 0; i < size; i++)
                {
                   Long depId = (Long) next.getAnnotation("dependency", i);
                   System.out.println("   " + resolver.getAst(depId).getText());
                }
             }
             */

             // Re-attach to tree in sorted order. Remove temporary list of
             // dependencies from dependent class AST nodes.
             iter = list.iterator();
             while (iter.hasNext())
             {
                child = iter.next();
                child.removeAnnotation("dependency");
                parent.addChild(child);
             }
          }

          /**
           * Load an AST from a file with the same name as the currently loaded file, but with the
           * given extension instead.
           *
           * @param   extension
           *          Filename extension of related AST file.
           */
          public void loadRelatedTree(String extension)
          {
             try
             {
                String file = getSource().getFilename();
                int pos = file.lastIndexOf('.');
                Aast root = AstManager.get().loadTree(file.substring(0, pos) + "." + extension);
                registerTree(root);
             }
             catch (Exception exc)
             {
                LOG.log(Level.SEVERE, "", exc);
             }
          }

          /**
           * Save the current target data model AST to file if the tree is not
           * empty (the root has at least 1 child).  If this AST is empty, it
           * will be removed from the {@link AstManager} which will essentially
           * disable further use of this AST.
           *
           * @return   <code>true</code> if the tree is persisted,
           *           <code>false</code> if the tree is empty or on any failure
           *           during persistence.
           */
          public boolean persistTarget()
          {
             WorkArea wa = context.get();

             String filename = generateFilename(getSource());

             // bypass persistence if there is an empty tree
             if (wa.root.getNumImmediateChildren() == 0)
             {
                try
                {
                   String normal = Configuration.normalizeFilename(filename);

                   // no need to maintain this value in the registry
                   AstManager.get().removeTree(normal);
                }
                catch (Exception exc)
                {
                   LOG.log(Level.SEVERE, "", exc);
                }

                return false;
             }

             try
             {
                filename = Configuration.normalizeFilename(filename);

                // read the existing target tree
                AstManager.get().saveTree(wa.root, filename, false);

                // ensure the AST node values remain in the map as their ID, and not the full AST.
                ConversionData.compact("tmpTabNodes", filename);
             }
             catch (Exception exc)
             {
                // some failure, we can't persist!
                LOG.log(Level.SEVERE, "", exc);

                return false;
             }

             return true;
          }

          /**
           * Convenience method to expose to rulesets the {@link P2JIndex} static method of the same name.
           *
           * @param   indexes
           *          The list of indexes.
           *
           * @see  P2JIndex#normalizeUniqueIndexes
           */
          public List<P2JIndex> normalizeUniqueIndexes(List<P2JIndex> indexes)
          {
             return P2JIndex.normalizeUniqueIndexes(indexes, P2JIndexComponent.PROPERTY);
          }

          /**
           * Convenience method to expose to rulesets the {@link P2JIndex} static method of the same name.
           *
           * @param   indexes
           *          The list of indexes.
           *
           * @see  P2JIndex#nonRedundantIndexes
           */
          public List<P2JIndex> nonRedundantIndexes(List<P2JIndex> indexes)
          {
             return P2JIndex.nonRedundantIndexes(indexes);
          }

          /**
           * Give a XML structure, attach the {@link #schemaElem} element and populate it with all
           * the DMO information stored in {@link #tempSchema}.
           *
           * @param    indexRoot
           *           The root element for the dmo_index.xml file.
           */
          public void storeTempSchema(XmlAst indexRoot)
          {
             if (schemaElem == null)
             {
                return;
             }

             XmlAst indexElem = (XmlAst) indexRoot.getImmediateChild(XmlTokenTypes.ELEMENT_NODE, null);
             schemaElem.remove();
             indexElem.addChild(schemaElem);

             BiConsumer<XmlAst, DMOElementInfo> storeAttributes = (element, info) ->
             {
                XmlAst elAttrSet = (XmlAst) XmlPatternWorker.createAst(XmlTokenTypes.ATTR_SET,
                                                                       "",
                                                                       element);
                for (String key : info.attributes.keySet())
                {
                   String value = info.attributes.get(key);

                   XmlAst elAttr = (XmlAst) XmlPatternWorker.createAst(XmlTokenTypes.ATTRIBUTE_NODE,
                                                                       key,
                                                                       elAttrSet);
                   XmlPatternWorker.createAst(XmlTokenTypes.CONTENT, value, elAttr);
                }
             };

             BiConsumer<XmlAst, DMOElementInfo>[] storeChildren = new BiConsumer[1];
             storeChildren[0] = (parent, info) ->
             {
                if (info.children == null)
                {
                   return;
                }

                for (DMOElementInfo childInfo : info.children)
                {
                   XmlAst dmoElem = (XmlAst) XmlPatternWorker.createAst(XmlTokenTypes.ELEMENT_NODE,
                                                                        childInfo.tag,
                                                                        parent);
                   storeAttributes.accept(dmoElem, childInfo);
                   storeChildren[0].accept(dmoElem, childInfo);
                }
             };

             Set<String> stored = new HashSet<>();
             Iterator<DMOElementInfo> iter = tempSchema.iterator();
             while (iter.hasNext())
             {
                DMOElementInfo dmoInfo = iter.next();
                String iface = dmoInfo.attributes.get("interface");
                if (stored.contains(iface))
                {
                   continue;
                }
                stored.add(iface);
                XmlAst dmoElem = (XmlAst) XmlPatternWorker.createAst(XmlTokenTypes.ELEMENT_NODE,
                                                                     dmoInfo.tag,
                                                                     schemaElem);

                storeAttributes.accept(dmoElem, dmoInfo);
                storeChildren[0].accept(dmoElem, dmoInfo);
             }
          }

          /**
           * Load the specified _temp schema from the XML structure into the {@link #tempSchema}.
           *
           * @param    indexElem
           *           The XML element for the <code>dmo-index</code> node.
           * @param    schemaElem
           *           The schema element to load.  May be <code>null</code>, if this does not exist
           *           already.
           * @param    schema
           *           The schema name.
           *
           * @return   The newly created XML element for the schema, or the previously sent
           *           schemaElem instance.
           */
          public XmlAst restoreTempSchema(XmlAst indexElem, XmlAst schemaElem, String schema)
          {
             if (this.schemaElem != null)
             {
                // do only once
                return this.schemaElem;
             }

             if (schemaElem == null)
             {
                schemaElem = (XmlAst) XmlPatternWorker.createAst(XmlTokenTypes.ELEMENT_NODE,
                                                                 "schema",
                                                                 indexElem);

                XmlAst elAttrSet = (XmlAst) XmlPatternWorker.createAst(XmlTokenTypes.ATTR_SET,
                                                                       "",
                                                                       schemaElem);
                XmlAst elAttr = (XmlAst) XmlPatternWorker.createAst(XmlTokenTypes.ATTRIBUTE_NODE,
                                                                   "name",
                                                                   elAttrSet);
                XmlPatternWorker.createAst(XmlTokenTypes.CONTENT, schema, elAttr);

                elAttr = (XmlAst) XmlPatternWorker.createAst(XmlTokenTypes.ATTRIBUTE_NODE,
                                                             "impl",
                                                             elAttrSet);
                XmlPatternWorker.createAst(XmlTokenTypes.CONTENT, schema + ".impl", elAttr);
             }

             this.schemaElem = schemaElem;

             BiConsumer<XmlAst, DMOElementInfo> restoreAttributes = (parent, info) ->
             {
                XmlAst attrs = (XmlAst) parent.getImmediateChild(XmlTokenTypes.ATTR_SET, null);

                XmlAst attr = (XmlAst) attrs.getImmediateChild(XmlTokenTypes.ATTRIBUTE_NODE, null);
                while (attr != null)
                {
                   String key = attr.getText();
                   String value = XmlPatternWorker.getAttributeValue(attrs, key);

                   info.putAttribute(key, value);

                   attr = (XmlAst) attrs.getImmediateChild(XmlTokenTypes.ATTRIBUTE_NODE, attr);
                }
             };

             BiConsumer<XmlAst, DMOElementInfo>[] restoreChildren = new BiConsumer[1];
             restoreChildren[0] =
             (parent, info) ->
             {
                XmlAst child = (XmlAst) parent.getImmediateChild(XmlTokenTypes.ELEMENT_NODE, null);

                while (child != null)
                {
                   String tag = child.getText();

                   DMOElementInfo childInfo = createDMOChildInfo(info, tag);
                   restoreAttributes.accept(child, childInfo);
                   restoreChildren[0].accept(child, childInfo);

                   child = (XmlAst) parent.getImmediateChild(XmlTokenTypes.ELEMENT_NODE, child);
                }
             };

             XmlAst dmoClass = (XmlAst) schemaElem.getImmediateChild(XmlTokenTypes.ELEMENT_NODE, null);
             while (dmoClass != null)
             {
                if ("class".equals(dmoClass.getText()))
                {
                   DMOElementInfo dmoInfo = createDMOClassInfo();

                   restoreAttributes.accept(dmoClass, dmoInfo);
                   restoreChildren[0].accept(dmoClass, dmoInfo);
                }

                dmoClass.remove();

                dmoClass = (XmlAst) schemaElem.getImmediateChild(XmlTokenTypes.ELEMENT_NODE, null);
             }

             return schemaElem;
          }

          /**
           * Create a structure to hold all the information for a <code>class</code> element in
           * the dmo_index.xml file.
           *
           * @return   See above.
           */
          public DMOElementInfo createDMOClassInfo()
          {
             DMOElementInfo classInfo = new DMOElementInfo("class");

             this.tempSchema.add(classInfo);

             return classInfo;
          }

          /**
           * Create a structure to hold additional information, and attach it to the given parent.
           *
           * @param    parent
           *           The parent structure, from dmo_index.xml.
           * @param    tag
           *           The that for this structure.
           *
           * @return   See above.
           */
          public DMOElementInfo createDMOChildInfo(DMOElementInfo parent, String tag)
          {
             if (parent.children == null)
             {
                parent.children = new LinkedList<>();
             }

             DMOElementInfo child = new DMOElementInfo(tag);

             parent.children.add(child);

             return child;
          }
       }

       /**
        * A simple helper class to load the content of a dmo_index.xml file, without having to know
        * all the semantics of the nodes.
        */
       public static class DMOElementInfo
       {
          /** The XML element tag. */
          private final String tag;

          /** The XML element attributes. */
          private Map<String, String> attributes = new LinkedHashMap<>();

          /** The children for this node. */
          private List<DMOElementInfo> children = null;

          /**
           * Create a new instance with the given tag.
           *
           * @param    tag
           *           The XML element tag.
           */
          public DMOElementInfo(String tag)
          {
             this.tag = tag;
          }

          /**
           * Set an attribute for this XML element, in the {@link #attributes} map.
           *
           * @param    name
           *           The attribute's name.
           * @param    val
           *           The attribute's value.
           */
          public void putAttribute(String name, String val)
          {
             attributes.put(name, val);
          }
       }

       /**
        * Comparator implementation which sorts ASTs which have an "order"
        * annotation into ascending order, according to the numeric value of
        * that attribute.
        */
       private static class PropertyComparator
       implements Comparator
       {
          /**
           * Compare two ASTs and return a value indicating their relative sort
           * order. It is assumed that each object parameter represents an
           * annotated AST, and that each such AST has an "order" annotation.
           *
           * @param   o1
           *          First AST.
           * @param   o2
           *          Second AST.
           *
           * @return  A negative number if <code>o1</code> &lt; <code>o2</code>,
           *          a negative number if <code>o1</code> &gt; <code>o2</code>,
           *          else <code>0</code>.
           */
          public int compare(Object o1, Object o2)
          {
             if (o1 == o2)
             {
                return 0; // evidently
             }

             // the objects represent our data model ASTs.
             Aast a1 = (Aast) o1;
             Aast a2 = (Aast) o2;

             Long order1 = (Long) a1.getAnnotation("order");
             Long order2 = (Long) a2.getAnnotation("order");

             if (order1 != null && order2 != null)
             {
                return (int) (order1 - order2);
             }

             Long index1 = (Long) a1.getAnnotation("customextent");
             Long index2 = (Long) a2.getAnnotation("customextent");

             if (index1 != null && index2 != null)
             {
                return (int) (index1 - index2);
             }

             System.out.println("Unexpected equal properties: " + a1.getDescriptiveTokenText() +
                                " and " + a2.getDescriptiveTokenText());
             return 0;
          }
       }

       /**
        * Comparator implementation which sorts ASTs which represent data model
        * object (DMO) classes. These classes are object representations of
        * relational tables. Foreign key relations create dependencies between
        * these tables. This comparator sorts classes such that independent
        * classes appear first in the sequence, followed by classes which are
        * dependent upon them.
        * <p>
        * This sort only separates completely independent classes from those
        * which have dependencies on other classes, but it <i>does not</i> sort
        * out the dependencies themselves. A second pass sort is used for that.
        */
       private static class ClassComparator
       implements Comparator
       {
          /**
           * Compare two ASTs and return a value indicating their relative sort
           * order. It is assumed that each object parameter represents an
           * annotated AST, and that each such AST represents a data model object
           * class.
           *
           * @param   o1
           *          First AST.
           * @param   o2
           *          Second AST.
           *
           * @return  A negative number if <code>o1</code> &lt; <code>o2</code>,
           *          a negative number if <code>o1</code> &gt; <code>o2</code>,
           *          else <code>0</code>.
           */
          public int compare(Object o1, Object o2)
          {
             // Objects represent our data model ASTs.
             Aast a1 = (Aast) o1;
             Aast a2 = (Aast) o2;

             // Independent classes sort earlier than dependent ones.
             int len1 = a1.annotationSize("dependency");
             int len2 = a2.annotationSize("dependency");
             if (len1 == 0 && len2 > 0)
             {
                return -1;
             }
             if (len2 == 0 && len1 > 0)
             {
                return 1;
             }

             // No sorting could be determined by dependency, so just sort
             // in ascending alpha order by name.
             return (a1.getText().compareTo(a2.getText()));
          }
       }

       /**
        * Context local work area.
        */
       private static class WorkArea
       {
          /** Root of the target tree currently being processed (created). */
          private DataModelAst root = null;
       }
    }