# SchemaWorker.java

    /*
    ** Module   : SchemaWorker.java
    ** Abstract : Exposes schema analysis capability to pattern engine
    **
    ** Copyright (c) 2005-2025, Golden Code Development Corporation.
    **
    ** -#- -I- --Date-- --JPRM-- -----------------------------------Description-----------------------------------
    ** 001 ECF 20050321   @20999 Created initial version. Detects and records
    **                           natural table relation data among Progress
    **                           schema tables.
    ** 002 ECF 20050526   @21290 Changes to support new expression engine
    **                           implementation. Changed library registration
    **                           mechanism based on superclass' modifications
    **                           to support single library per pattern worker
    **                           limit.
    ** 003 ECF 20050613   @21477 Reworked to handle adding inverse end of each
    **                           detected join. Also added method to enable
    **                           dumping all detected joins at the end of a
    **                           source scan.
    ** 004 GES 20050706   @21669 Minor signature change for a utility method.
    ** 005 ECF 20051005   @22983 Update to store data to .schema file instead
    **                           of .dict file. Necessitated by change to
    **                           schema fixups ruleset, which preserves
    **                           original AST as .dict and saves fixed up file
    **                           with a .schema extension.
    ** 006 ECF 20060302   @24859 Fixed regression related to #005 (@22983).
    **                           Natural join information was not properly
    **                           persisting in the .schema file, due to AST ID
    **                           mismatches between join data (based on .dict
    **                           ASTs) and the newer, target, .schema AST.
    ** 007 ECF 20060325   @25241 Removed code which created duplicate schema
    **                           file with the '.original' extension. This
    **                           duplication is no longer necessary and is the
    **                           source of constant registry problems.
    ** 008 ECF 20070302   @32265 Ignore self referencing join. That is, a join
    **                           between a table and itself. This could happen
    **                           if two buffers on the same backing table were
    **                           joined using the OF keyword.
    ** 009 GES 20070629   @34341 Removed terse flag from AST persistence so
    **                           that we retail line/column info in our ASTs.
    ** 010 GES 20070911   @35143 Minor change to how the schema dictionary is
    **                           obtained.
    ** 011 ECF 20080730   @39272 Reduced memory footprint. Implemented finish
    **                           method. Made creation of RelationAnalyzer
    **                           (which pins SchemaDictionary) lazy. Added a
    **                           user function to release SchemaDictionary.
    ** 012 GES 20081010   @40337 Fix to allow UAST hints to load properly.
    ** 013 ECF 20081216   @40938 Fixed attachRelationalData(). Allow foreign
    **                           relation information to be stored for temp
    **                           tables. The only restriction now is that both
    **                           tables in the join need to reside in the same
    **                           database.
    ** 014 GES 20090515   @42223 Moved to AstManager from AstRegistry/AstPersister.
    ** 015 ECF 20130124          Added worker function to get schema config.
    ** 016 OM  20130808          Added getForeignKey() to obtain the list of the fields used to join
    **                           two Progress tables.
    **                           joins Set morphed into a Hashtable for getter fast lookup.
    **                           Generified private/local variables except for deprecated methods.
    ** 017 CA  20140114          Added getLegacyTableName API to resolve the legacy table name
    **                           (case-sensitive) from a table name.
    ** 018 ECF 20131129          Added support for metaschema joins.
    ** 019 ECF 20150112          Added support for metadata tables implemented in a primary database.
    ** 020 ECF 20150326          Added nearly complete support for cross-schema natural joins. The
    **                           feature still missing is joins from permanent tables to temp-tables.
    **                           The other direction is supported.
    ** 021 ECF 20150715          Replace StringBuffer with StringBuilder.
    ** 022 ECF 20151210          Default _User dump file name to "_User.d" for Unix systems, else
    **                           "_user.d".
    ** 023 OM  20160905          Small optimizations.
    ** 024 OM  20180118          Dropped "primary" notion for databases in p2j.cfg.xml. Maintenance.
    ** 025 CA  20180507          Added loadNonDefaults, to load non-default schemas.
    **     CA  20190513          Fixed buffer usage from super-classes, in OO.
    ** 026 GES 20190620          Added multiple inheritance support (for OO interface defs).
    ** 027 CA  20200412          Added incremental conversion support.
    **     OM  20210909          Used SchemaDictionary.META_DB instead of hardcoded constant.
    **     OM  20211122          Added hints-based support for 'unloading' schema.
    **     OM  20220330          Moved database conversion artifacts to ${cvtpath} folder.
    **                           Made SCHEMA_POSTFIX static field public.
    **     TJD 20220504          Synced deprecation in javadoc and annotation
    **     CA  20230110          Let 'getImmediateChild' work with a set, if an array is used for the types.
    ** 028 GBB 20230512          Logging methods replaced by CentralLogger/ConversionStatus.
    ** 029 OM  20230630          Added support for the newly observed PARENT_ID_RELATION between joined tables.
    ** 030 CA  20230801          The key in the 'joins' map is formed from table names: in case of temp-tables,
    **                           this does not guarantee that the temp-table definition match.  For these cases,
    **                           when the computed join doesn't match a previously computed one, do not emit a
    **                           relation for these tables.
    ** 031 AL2 20240530          Use AstManager to initialize the directory instead of File.
    ** 032 CA  20240601          Schema dictionary files are loaded from the outermost grand-parent to the direct
    **                           parent, as i.e. any temp-table definition in current class will hide definitions
    **                           with the same name, from parents.
    ** 033 OM  20241030          Added _sec-authentication-system and _sec-authentication-domain tables to
    **                           metaTablesInPersistentDB so that they get moved from standard to primary dbs.
    ** 034 CA  20250319          This worker can be used before the 'qualified_oo_name' is set in annotation
    **                           (like in schema conversion phase), so ensure we can load the ClassDefinition.
    */

    /*
    ** This program is free software: you can redistribute it and/or modify
    ** it under the terms of the GNU Affero General Public License as
    ** published by the Free Software Foundation, either version 3 of the
    ** License, or (at your option) any later version.
    **
    ** This program is distributed in the hope that it will be useful,
    ** but WITHOUT ANY WARRANTY; without even the implied warranty of
    ** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    ** GNU Affero General Public License for more details.
    **
    ** You may find a copy of the GNU Affero GPL version 3 at the following
    ** location: https://www.gnu.org/licenses/agpl-3.0.en.html
    **
    ** Additional terms under GNU Affero GPL version 3 section 7:
    **
    **   Under Section 7 of the GNU Affero GPL version 3, the following additional
    **   terms apply to the works covered under the License.  These additional terms
    **   are non-permissive additional terms allowed under Section 7 of the GNU
    **   Affero GPL version 3 and may not be removed by you.
    **
    **   0. Attribution Requirement.
    **
    **     You must preserve all legal notices or author attributions in the covered
    **     work or Appropriate Legal Notices displayed by works containing the covered
    **     work.  You may not remove from the covered work any author or developer
    **     credit already included within the covered work.
    **
    **   1. No License To Use Trademarks.
    **
    **     This license does not grant any license or rights to use the trademarks
    **     Golden Code, FWD, any Golden Code or FWD logo, or any other trademarks
    **     of Golden Code Development Corporation. You are not authorized to use the
    **     name Golden Code, FWD, or the names of any author or contributor, for
    **     publicity purposes without written authorization.
    **
    **   2. No Misrepresentation of Affiliation.
    **
    **     You may not represent yourself as Golden Code Development Corporation or FWD.
    **
    **     You may not represent yourself for publicity purposes as associated with
    **     Golden Code Development Corporation, FWD, or any author or contributor to
    **     the covered work, without written authorization.
    **
    **   3. No Misrepresentation of Source or Origin.
    **
    **     You may not represent the covered work as solely your work.  All modified
    **     versions of the covered work must be marked in a reasonable way to make it
    **     clear that the modified work is not originating from Golden Code Development
    **     Corporation or FWD.  All modified versions must contain the notices of
    **     attribution required in this license.
    */

    package com.goldencode.p2j.schema;

    import java.io.*;
    import java.util.*;
    import java.util.function.*;
    import java.util.logging.*;

    import com.goldencode.ast.*;
    import com.goldencode.p2j.cfg.*;
    import com.goldencode.p2j.convert.*;
    import com.goldencode.p2j.pattern.*;
    import com.goldencode.p2j.uast.*;

    /**
     * A pattern worker which exposes schema dictionary and table relation
     * analysis functionality to the pattern engine.
     */
    public class SchemaWorker
    extends AbstractConversionWorker
    implements SchemaParserTokenTypes
    {
       /** Fixed-up schema AST filename extension. */
       public static final String SCHEMA_POSTFIX = ".schema";

       /** List of all Progress field token types */
       private static final Set<Integer> ALL_FIELD_TYPES = new HashSet<>();

       /** Map of metadata table names (lowercase) to respective dump file names, if any */
       private static final Map<String, String> metaTableDumpFiles = new HashMap<>();

       /** Names of metadata tables (lowercase) implemented in a primary database */
       private static final Set<String> metaTablesInPersistentDB;

       /** Logger */
       private static final ConversionStatus LOG = ConversionStatus.get(SchemaWorker.class);

       static
       {
          // initialize list of field token types
          for (int i = BEGIN_FIELDTYPES + 1; i < END_FIELDTYPES; i++)
          {
             ALL_FIELD_TYPES.add(i);
          }

          // initialize list of field token types used only by metadata
          for (int i = BEGIN_METATYPES + 1; i < END_METATYPES; i++)
          {
             ALL_FIELD_TYPES.add(i);
          }

          // initialize map of metadata tables to their respective dump file names, if any;
          // on Unix systems, default dump name is "_User.d", on Windows, "_user.d"
          // TODO: check other platform defaults and implement as necessary
          boolean unix = "UNIX".equals(Configuration.getParameter("opsys"));
          metaTableDumpFiles.put("_user", unix ? "_User.d" : "_user.d");

          // initialize set of metadata tables to be implemented in each primary database
          Set<String> set = new LinkedHashSet<>();
          //set.add("_connect");
          set.add("_user");
          set.add("_sec-authentication-system");
          set.add("_sec-authentication-domain");
          metaTablesInPersistentDB = Collections.unmodifiableSet(set);
       }

       /** Schema loader, used to load schema ASTs */
       private SchemaLoader loader = new SchemaLoader();

       /** Provides schema information lookup; loads default database(s) */
       private SchemaDictionary dictionary = new SchemaDictionary((Set<String>) null);

       /** Names of databases which contain tables involved in relations */
       private Set<String> databases = new HashSet<>();

       /**
        * Set of all table relations detected in a scanning session. They are hashed using a
        * combination of the table names that assures that there are no access collisions.
        */
       private Map<String, TableRelation> joins = new Hashtable<>();

       /** The set of "related tables" where their relation definition does not match. */
       private Set<String> joinCollisions = new HashSet<>();

       /** Set of IDs of index ASTs to be marked as referenced by "use-index" */
       private Set<String> usedIndexes = new HashSet<>();

       /** The class definition where this schema was loaded. */
       private ClassDefinition clsDef = null;

       /**
        * Default constructor which defines the symbol libraries to be registered.
        *
        * @throws  SchemaException
        *          if there is any error initializing the schema loader or
        *          schema dictionary.
        */
       public SchemaWorker()
       throws SchemaException
       {
          super();

          setLibrary(new SchemaSupport());
       }

       /**
        * Hook to provide termination processing for a pattern worker.
        */
       public void finish()
       {
          loader = null;
          dictionary = null;
          databases = null;
          joins = null;
          joinCollisions = null;
          usedIndexes = null;
          clsDef = null;

          super.finish();
       }

       /**
        * Helper method to append an extension to a filename if it is not
        * already there.
        *
        * @param   filename
        *          Filename which may or may not already end with the extension.
        * @param   extension
        *          Extension to be set at the end of <code>filename</code>.
        *
        * @return  <code>filename</code> with the required extension.
        */
       public static String adjustFilename(String filename, String extension)
       {
          if (filename.endsWith(extension))
          {
             return filename;
          }

          return filename + extension;
       }

       /**
        * Load a persisted schema AST from the specified file. This is intended primarily to load
        * temp- and work-tables into an instance of the dictionary which already has a primary schema loaded.
        *
        * @param   dictionary
        *          The schema dictionary instance.
        * @param   file
        *          The name of a persisted schema file which is associated with a Progress source code file.
        *
        * @return  an instance of {@code SchemaException} if any error occurs reading the schema data from
        *          persistence and {@code null} on success.
        */
       public static SchemaException loadFromPersistence(SchemaDictionary dictionary, String file)
       {
          String fname = adjustFilename(file, AstGenerator.DICT_POSTFIX);
          File   fil   = new File(fname);
          if (fil.exists())
          {
             try
             {
                dictionary.loadFromPersistence(fil.getAbsolutePath());
             }
             catch (SchemaException e)
             {
                return e;
             }
          }

          return null;
       }

       /**
        * This method is called each time a new AST is visited, just before the
        * pattern engine processes that AST.  We exploit this hook by loading the
        * current temp/work-table schema (P2O) AST into the <code>P2OLookup</code>
        * class' state.  This enables us to resolve references to temp and work
        * tables, which are local to the currently loaded source file AST.
        *
        * @param   ast
        *          The root node of the source AST about to be processed by the
        *          pattern engine.
        */
       @Override
       public void visitAst(Aast ast)
       {
          clsDef = SymbolResolver.resolveClassDefinition(ast);
       }

       /**
        * Composes a normalized index name, which is the fully qualified table
        * name in all lowercase, in the form:
        * <code>&lt;database_name&gt;&lt;table_name&gt;&lt;index_name&gt;</code>.
        *
        * @param   index
        *          AST which defines index.
        *
        * @return  Normalized index name as described above.
        */
       private static String normalizedIndexName(Aast index)
       {
          String buf = TableRelation.normalizedTableName(index.getParent()) + '.' + index.getText();

          return buf.toLowerCase();
       }

       /**
        * Figures out which schema file to load as the basis for updates. If
        * an original schema file (i.e., one that ends with ".original") exists,
        * it is returned. Otherwise, the file represented by <code>filename</code>
        * is returned.
        *
        * @param   filename
        *          Root schema AST filename.
        *
        * @return  Original filename or base filename if an original has not
        *          been created yet.
        *
        * @throws  ConfigurationException
        *          if P2J_HOME has not been defined.
        */
       private File getSchemaLoadFile(String filename)
       throws ConfigurationException
       {
          return Configuration.toFile(filename);
       }

       /**
        * Attaches relation data ASTs to any table which has been found in a natural join.
        * <p>
        * The <code>schemaRoot</code> AST will have been loaded from the <code>.schema</code> file;
        * however, all join information is based upon ASTs loaded from the primordial
        * <code>.dict</code> file (needed by the <code>SchemaDictionary</code>). Thus, table IDs
        * within <code>TableRelation</code> objects will not match those in <code>schemaRoot</code>
        * tables.  We must therefore use the normalized table name to associated the table relations
        * with the table ASTs found in the schema AST.  The same technique applies to indexes.
        *
        * @param   schemaRoot
        *          AST which represents a permanent or transient database schema.
        * @param   file
        *          File to which schema AST is saved after updates have been made.
        * @param   tables
        *          Map of normalized table names to table ASTs.
        * @param   indexes
        *          Map of normalized index names to index ASTs.
        */
       private void persistSchema(Aast schemaRoot,
                                  File file,
                                  Map<String, Aast> tables,
                                  Map<String, Aast> indexes)
       throws SchemaException
       {
          Map<String, TableRelation> joins = new Hashtable<>(this.joins);
          joins.keySet().removeAll(joinCollisions);

          // attach relational data to tables for which such data has been collected
          for (TableRelation join : joins.values())
          {
             attachRelationData(join, tables, schemaRoot);
          }

          // annotate indexes which are referenced by "use-index"
          for (String usedIndex : usedIndexes)
          {
             Aast index = indexes.get(usedIndex);
             if (index != null)
             {
                index.putAnnotation("use-index", Boolean.TRUE);
             }
          }

          try
          {
             // Persist modified AST.
             String filename = file.getAbsolutePath();
             AstManager.get().saveTree(schemaRoot, filename, false);
          }
          catch (Exception exc)
          {
             throw new SchemaException("Error overwriting schema file", exc);
          }
       }

       /**
        * Walk the given schema AST and collect every table and index node by their normalized names
        * in the given maps.
        *
        * @param   schemaRoot
        *          Schema AST.
        * @param   tables
        *          Map in which to store table nodes.
        * @param   indexes
        *          Map in which to store index nodes.
        */
       private void gatherTableAndIndexAsts(Aast schemaRoot,
                                            Map<String, Aast> tables,
                                            Map<String, Aast> indexes)
       {
          Iterator<Aast> iter = schemaRoot.iterator(2, null);
          while (iter.hasNext())
          {
             Aast next = iter.next();
             switch (next.getType())
             {
                case TABLE:
                case TEMP_TABLE:
                case WORK_TABLE:
                   tables.put(TableRelation.normalizedTableName(next), next);
                   break;

                case INDEX:
                   if (indexes != null)
                   {
                      indexes.put(normalizedIndexName(next), next);
                   }
                   break;
             }
          }
       }

       /**
        * Given a database and record AST, scan the collected join information
        * to see if this record requires table relation information ASTs to be
        * attached. For each join which was recorded in which this table
        * participates on the foreign key end, add an AST describing the join.
        * <p>
        * The ASTs which are attached are of token type ONE_TO_ONE or MANY_TO_ONE,
        * depending upon the circumstances of the join. Each has one child AST
        * of type KEY_FIELD for each field which participates in the join. In
        * addition, each has an annotation to record the ID of the table on the
        * other end of the join. The ASTs are attached as direct children of the
        * table to which they refer.
        *
        * @param   join
        *          Object which describes a natural relation between tables.
        * @param   tables
        *          Map of record-type AST nodes, indexed by ID. This map may or
        *          may not contain one or both tables described by
        *          <code>join</code>.
        *
        * @throws  SchemaException
        *          if any error occurs attaching a join information AST to
        *          the table AST <code>ast</code>.
        */
       private void attachRelationData(TableRelation join, Map<String, Aast> tables, Aast schemaRoot)
       throws SchemaException
       {
          String key;

          // Get the node within the AST to be updated, which represents the
          // left side of this join. If this join has a primary end, it will be
          // the left table.
          key = TableRelation.normalizedTableName(join, join.getLeft());
          Aast left = tables.get(key);

          // Get the node within the AST to be updated, which represents the
          // right side of this join. If this join has a foreign end, it will be
          // the right table.
          key = TableRelation.normalizedTableName(join, join.getRight());
          Aast right = tables.get(key);

          // graft join data onto each table iff we have the ASTs from both sides of the join loaded
          if (left != null && right != null)
          {
             String database = schemaRoot.getText();
             Set<String> keyFields = join.getKey();

             if (database.equalsIgnoreCase(left.getParent().getText()))
             {
                graftJoinNode(join.getInverseType(), keyFields, left, right);
             }

             if (database.equalsIgnoreCase(right.getParent().getText()))
             {
                graftJoinNode(join.getType(), keyFields, right, left);
             }
          }
       }

       /**
        * Graft an AST node which describes a natural join onto a target table
        * AST. The node's token type describes the nature of the join. It has
        * an annotation with the ID of the table on the inverse end of the
        * relation. The node has one child per field involved in the key which
        * binds the tables.
        *
        * @param   type
        *          Token type of the join node.
        * @param   key
        *          Set of names of fields which form the key between <code>target</code> and
        *          <code>inverse</code>. In the case of a metaschema join, this will contain only one
        *          field name, and that field will only be present in the <code>target</code> table.
        * @param   target
        *          Table AST to which join node is added.
        * @param   inverse
        *          AST which represents the table on the inverse end of the relation.
        *
        * @throws  SchemaException
        *          if an error occurs adding the join AST node.
        */
       private void graftJoinNode(int type, Set<String> key, Aast target, Aast inverse)
       throws SchemaException
       {
          String table = TableRelation.normalizedTableName(inverse);
          String database = TableRelation.normalizedDatabaseName(inverse);
          AstSymbolResolver resolver = AstSymbolResolver.getResolver();

          // create relation AST child to graft to the target table node
          Aast relAst = new ProgressAst();
          initializeAst(relAst, type, table, target, resolver);

          // add an annotation indicating the table on the other end of the relation
          relAst.putAnnotation("references", inverse.getId());
          relAst.putAnnotation("database", database);

          // add each field as a child component of the relation node
          for (String fieldName : key)
          {
             Aast compAst = new ProgressAst();
             initializeAst(compAst, KEY_FIELD, fieldName, relAst, resolver);

             // Annotate with a reference to actual, local field.
             Aast field = target.getImmediateChild(ALL_FIELD_TYPES, null);
             while (field != null)
             {
                if (fieldName.equals(field.getText()))
                {
                   compAst.putAnnotation("refid", field.getId());
                   break;
                }
                field = target.getImmediateChild(ALL_FIELD_TYPES, field);
             }

             // annotate with a cross-reference to same field in inverse table
             field = inverse.getImmediateChild(ALL_FIELD_TYPES, null);
             while (field != null)
             {
                if (fieldName.equals(field.getText()))
                {
                   compAst.putAnnotation("crossrefid", field.getId());
                   break;
                }
                field = inverse.getImmediateChild(ALL_FIELD_TYPES, field);
             }
          }
       }

       /**
        * Provides schema-related services to pattern engine users. An object of
        * this class is registered with the symbol resolver to provide user
        * function support for user expressions.
        */
       public class SchemaSupport
       {
          /** List of strings comprising the redundant index report */
          private List<String> redundantIndexReport = new ArrayList<>();

          /** List of strings comprising the index component report */
          private List<String> indexComponentReport = new ArrayList<>();

          /** Unqualified, lower case names of all metadata tables for this project */
          private Set<String> metaNames = Configuration.getSchemaConfig().getMetadata().getTables();

          /**
           * Return instance of schema dictionary.
           *
           * @return  Current instance of schema dictionary.
           */
          public SchemaDictionary getDictionary()
          {
             return dictionary;
          }

          /**
           * Release the in-use dictionary.
           */
          public void releaseDictionary()
          {
             dictionary = null;
          }

          /**
           * Resolve the legacy table name, given the (possible) qualified table name.
           *
           * @param    name
           *           The (possibly) qualified table name.
           *
           * @return   See above.
           */
          public String getLegacyTableName(String name)
          {
             try
             {
                Aast table = dictionary.getTable(name);

                String legacyName = (String) table.getAnnotation("legacy_name");
                if (legacyName == null)
                {
                   throw new SchemaException("Legacy name not found for table:" + name);
                }

                return legacyName;
             }
             catch (SchemaException e)
             {
                throw new RuntimeException("Could not resolve legacy name for table: " + name);
             }
          }

          /**
           * Load all non-default schemas into the dictionary.
           *
           * @throws  SchemaException
           *          if any error occurs loading source file-specific schema information.
           */
          public void loadNonDefaults()
          throws SchemaException
          {
             loader.loadNonDefaults(dictionary);
          }

          /**
           * Initialize the schema dictionary for use with the current source
           * file AST. Resets the state of the dictionary, but keeps default
           * databases loaded. Also checks for hints to see if any additional
           * databases need to be loaded and/or aliases assigned.
           *
           * @param   filename
           *          Name of the source file represented by the AST currently
           *          being processed in the pattern engine. Any temp- and work-
           *          tables defined in this source file are loaded into the
           *          dictionary.
           *
           * @return  <code>true</code>.
           *
           * @throws  SchemaException
           *          if any error occurs loading source file-specific schema
           *          information.
           */
          public boolean initializeDictionary(String filename)
          throws SchemaException
          {
             String originalFileName = filename;

             // Get a clean instance of the dictionary.
             dictionary = new SchemaDictionary((Set<String>) null);

             // Load hints for this source file, if any.
             UastHints hints = new UastHints(filename);

             // load extra databases into dictionary, if any
             Set<String> dbList = hints.getDatabases();
             if (dbList != null)
             {
                for (String db : dbList)
                {
                   dictionary.loadSchema(db);
                }
             }

             // Create database aliases, if any.
             List<String[]> aliases = hints.getAliases();
             if (aliases != null)
             {
                Iterator<String[]> iter = aliases.iterator();
                while (iter.hasNext())
                {
                   String[] pair = iter.next();

                   if (!dictionary.createAlias(pair[0], pair[1]))
                   {
                      LOG.log(Level.WARNING, "Dictionary could not create alias " + pair[0] + " for database " + pair[1]);
                   }
                }
             }

             // first load the parent hierarchy
             if (clsDef != null)
             {
                SchemaException[] exc = new SchemaException[1];

                Consumer<ClassDefinition> consumer = (ClassDefinition cls) ->
                {
                   if (exc[0] == null)
                   {
                      exc[0] = loadFromPersistence(dictionary, cls.getFilename());
                   }
                };

                clsDef.processParentGraph(consumer, false);

                if (exc[0] != null)
                {
                   throw exc[0];
                }
             }

             // Load AST which defines temp- and work-tables for the current source file.
             filename = Configuration.normalizeFilename(originalFileName + AstGenerator.DICT_POSTFIX);
             if (AstManager.get().isExistingAst(filename))
             {
                Aast root = AstManager.get().loadTree(filename);
                dictionary.loadFromAst(root, false);
             }

             return true;
          }

          /**
           * Get the {@link SchemaConfig} that was read from the project
           * configuration.
           *
           * @return  Schema configuration.
           */
          public SchemaConfig getSchemaConfig()
          {
             return Configuration.getSchemaConfig();
          }

          /**
           * Return a string which describes a join between two tables. This is
           * useful for debug and ad-hoc reporting purposes.
           *
           * @param   left
           *          Left-hand table of the join (fully qualified).
           * @param   right
           *          Right-hand table of the join (fully qualified).
           *
           * @return  A string describing the relation between these tables,
           *          if any.  <code>null</code> if there was an error.
           */
          public String describeJoin(String left, String right)
          {
             // Ignore a table which is joined to itself.
             if (left.equals(right))
             {
                return null;
             }

             String description = null;

             try
             {
                RelationAnalyzer analyzer = new RelationAnalyzer(dictionary);
                TableRelation join = analyzer.createJoin(left, right);
                description = join.toString();
             }
             catch (SchemaException exc)
             {
                description = exc.getMessage();
             }
             catch (Exception exc)
             {
                LOG.log(Level.WARNING,"", exc);
             }

             return description;
          }

          /**
           * Create a {@link TableRelation} object based the natural join of
           * the specified tables, and add it to the set of joins being collected
           * by this worker. Error messages are printed to <code>stderr</code>.
           *
           * @param   left
           *          Left-hand table of the join (fully qualified).
           * @param   right
           *          Right-hand table of the join (fully qualified).
           *
           * @return  The instance of a <code>TableRelation</code>, if the join
           *          was successfully recorded;  <code>null</code> if there was
           *          an error.
           */
          public TableRelation recordJoin(String left, String right)
          {
             // Ignore a table which is joined to itself.
             if (left.equals(right))
             {
                return null;
             }

             /*
              * OM: The TableRelation is reflexive from Progress point of view. This is different from
              * normal SQL notion of table join by foreign key.
              * The Progress imposes that:
              * - the name of the fields that compose the index to have the same name in both tables,
              *       if the relation is specified all fields participate in the join implicitly.
              * - such combination to be unique.
              * The analyzer will automatically detect the unique indexed table and choose it as the
              * sql left side of the join.
              * Because of that, createJoin(left, right) and createJoin(right, left) will generate
              * "equal" objects, so we can freely reorder the tables.  If we index the "joins"
              * map using the alphabetic order of the table names we can quickly lookup the relation
              * between the two tables.
              *
              * ECF: Exception to this rule is metaschema tables, which use "real" foreign key joins.
              * In this case, when creating the cache key, we set the table with the foreign key
              * reference as the left table and the table with the primary key referent as the right
              * table, regardless of their names' alphabetical order.
              */

             TableRelation join = null;
             String joinLookup = null;
             boolean isMeta = areMetaTables(left, right);

             if (isMeta)
             {
                join = lookupMetaJoin(left, right);
             }
             else
             {
                joinLookup = left.compareTo(right) > 0 ? right + " " + left : left + " " + right;
                if (joinCollisions.contains(joinLookup))
                {
                   return null;
                }
                join = joins.get(joinLookup);
             }

             try
             {
                TableRelation found = join;

                RelationAnalyzer analyzer = new RelationAnalyzer(dictionary);
                join = analyzer.createJoin(left, right);

                /*
                 * Save the relation for later lookups. It will:
                 * - cache the value for future calls for same tables (which can be swapped)
                 * - allow extracting the fields of key for nested can-find cases where the relation
                 *       between the two tables has to be expanded explicitly in hql.
                 */
                if (isMeta)
                {
                   joinLookup = TableRelation.normalizedTableName(join, join.getRight())  + " " +
                                TableRelation.normalizedTableName(join, join.getLeft());
                }

                // check the string representation which uses the entire relation - if these do not match, then we
                // have a collision
                if (found != null)
                {
                   if (found.toString().equals(join.toString()))
                   {
                      return found;
                   }
                   else
                   {
                      joinCollisions.add(joinLookup);
                      return null;
                   }
                }

                joins.put(joinLookup, join);
                databases.addAll(join.getDatabases());
             }
             catch (Exception exc)
             {
                LOG.log(Level.SEVERE, "", exc);
             }

             return join;
          }

          /**
           * Return a string which contains the fields (space separated) that are part of the foreign
           * key that joins two tables. The relation should already have been added with
           * {@link #recordJoin(String, String)}.
           *
           * @param   left
           *          Left-hand table of the join (fully qualified).
           * @param   right
           *          Right-hand table of the join (fully qualified).
           *
           * @return  A string describing the foreign key of these tables, if any, and
           *          <code>null</code> if tables are unrelated or there was an error.
           */
          public String getForeignKey(String left, String right)
          {
             // Eliminate common cases
             if (left == null || right == null || left.equals(right))
             {
                return null;
             }

             TableRelation join = null;
             if (areMetaTables(left, right))
             {
                join = lookupMetaJoin(left, right);
             }
             else
             {
                String lookup = left.compareTo(right) > 0 ? right + " " + left : left + " " + right;
                if (joinCollisions.contains(lookup))
                {
                   return null;
                }

                join = joins.get(lookup);
             }

             if (join == null)
             {
                return null;
             }

             if (join.getType() == PARENT_ID_RELATION)
             {
                return null; // ""; ?
             }

             StringBuilder sb = new StringBuilder();
             for (String k : join.getKey())
             {
                if (sb.length() > 0)
                {
                   sb.append(' ');
                }
                sb.append(k);
             }

             return sb.toString();
          }

          /**
           * Add index AST to the set of indexes referenced by the Progress
           * "use-index" directive. Update the set of databases to be persisted
           * as a result of this discovery. The actual annotation occurs later,
           * when the corresponding database AST is persisted.
           *
           * @param   index
           *          AST of the index to be marked.
           */
          public void markIndexUsed(Aast index)
          {
             usedIndexes.add(normalizedIndexName(index));
             databases.add(index.getAncestor(-1).getText());
          }

          /**
           * Graft all discovered natural join information into the appropriate schema ASTs and
           * persist those ASTs for both permanent and temp-tables.
           * <p>
           * TODO: determine whether natural joins are legal from permanent tables to temp-tables and
           * if so, add support for this idiom. This will require a more precise way to identify the
           * temp-table end of the join than the database-qualified table name method used currently.
           *
           * @param   sourceFiles
           *          Names of source files in which temp-table natural joins were detected.
           *
           * @throws  ConfigurationException
           *          if P2J_HOME environment variable is not set.
           * @throws  SchemaException
           *          if there were errors updating or saving the affected schema ASTs.
           */
          public void persistSchemas(Set<String> sourceFiles)
          throws ConfigurationException,
                 SchemaException
          {
             SchemaConfig schemaConfig = Configuration.getSchemaConfig();
             Map<File, Aast> schemaMap = new LinkedHashMap<>();

             // first pass collects permanent tables and indexes from databases participating in
             // natural joins
             Map<String, Aast> tables = new HashMap<>();
             Map<String, Aast> indexes = new HashMap<>();
             for (String next : databases)
             {
                if (next.startsWith("@"))
                {
                   continue;
                }

                if (SchemaDictionary.META_DB.equals(next))
                {
                   next = schemaConfig.getMetadata().getName();
                }

                String filename = schemaConfig.getSchemaFileName(next, SCHEMA_POSTFIX);
                File saveFile = new File(filename);
                File loadFile = getSchemaLoadFile(filename);
                if (loadFile.exists())
                {
                   Aast schemaRoot = loader.loadAst(loadFile);
                   schemaMap.put(saveFile, schemaRoot);
                   gatherTableAndIndexAsts(schemaRoot, tables, indexes);
                }
             }

             // second pass attaches join data and persists the schema ASTs
             for (Map.Entry<File, Aast> entry : schemaMap.entrySet())
             {
                File saveFile = entry.getKey();
                Aast schemaRoot = entry.getValue();
                persistSchema(schemaRoot, saveFile, tables, indexes);
             }

             // process temp-table schemas
             for (String filename : sourceFiles)
             {
                filename = adjustFilename(filename, SCHEMA_POSTFIX);
                File loadFile = getSchemaLoadFile(filename);
                if (loadFile.exists())
                {
                   Map<String, Aast> tempTables = new HashMap<>();
                   Map<String, Aast> tempIndexes = new HashMap<>();
                   Aast schemaRoot = loader.loadAst(loadFile);
                   gatherTableAndIndexAsts(schemaRoot, tempTables, tempIndexes);
                   tables.putAll(tempTables);
                   try
                   {
                      persistSchema(schemaRoot, loadFile, tables, tempIndexes);
                   }
                   finally
                   {
                      tables.keySet().removeAll(tempTables.keySet());
                   }
                }
             }
          }

          /**
           * Dump to <code>stdout</code> the set of natural joins detected by
           * all invocations of {@link #recordJoin} during a scan of some set
           * of Progress source code.
           */
          public void dumpNaturalJoins()
          {
             System.out.println("ALL NATURAL JOINS: ");
             System.out.println("------------------");
             for (Map.Entry<String, TableRelation> pair : joins.entrySet())
             {
                TableRelation join = pair.getValue();
                String key = pair.getKey();
                if (joinCollisions.contains(key))
                {
                   System.out.println("COLLISION - natural join ignored on tables: " + key);
                }
                else
                {
                   System.out.println(join.toString());
                }
             }
          }

          /**
           * @deprecated
           */
          @Deprecated
          public void analyzeIndexComponents(String table, Map indices)
          {
             List<String> report = new ArrayList<>();
             Set<String> done = new HashSet<>();

             Iterator<String> iter = indices.keySet().iterator();
             while (iter.hasNext())
             {
                // Get current index name.
                String name = iter.next();
                done.add(name);

                // Get list of current index component fields.
                List fields = (List) indices.get(name);

                Iterator iter2 = indices.keySet().iterator();
                while (iter2.hasNext())
                {
                   String key = (String) iter2.next();

                   // Skip indices already reviewed.
                   if (done.contains(key))
                   {
                      continue;
                   }

                   List fields2 = (List) indices.get(key);

                   // Iterate through "outer" index' fields.
                   boolean problem = false;
                   Iterator subIter = fields.iterator();
                   for (int i = 0; subIter.hasNext(); i++)
                   {
                      String next = (String) subIter.next();
                      int index = fields2.indexOf(next);
                      if (index >= 0 && index != i)
                      {
                         if (!problem)
                         {
                            problem = true;
                            report.add("   " + name + ":  " + fields);
                            report.add("   " + key + ":  " + fields2);
                         }
                         report.add("      " + next + ":  " + name + "[" + i +
                                    "] ... " + key + "[" + index + "]");
                      }
                   }
                }
             }

             if (report.size() > 0)
             {
                indexComponentReport.add(table + ":");
                iter = report.iterator();
                while (iter.hasNext())
                {
                   indexComponentReport.add(iter.next());
                }
             }
          }

          /**
           * @deprecated
           */
          @Deprecated
          public void analyzeIndexRedundancy(String table, Map indices)
          {
             List<String> report = new ArrayList<>();
             Set<String> done = new HashSet<>();

             Iterator<String> iter = indices.keySet().iterator();
             while (iter.hasNext())
             {
                // Get current index name.
                String name = iter.next();
                done.add(name);

                // Get list of current index component fields.
                List fields = (List) indices.get(name);
                int size = fields.size();

                Iterator iter2 = indices.keySet().iterator();
                while (iter2.hasNext())
                {
                   String key = (String) iter2.next();

                   // Skip indices already reviewed.
                   if (done.contains(key))
                   {
                      continue;
                   }

                   // Get list of compare target's component fields.
                   List tgtFields = (List) indices.get(key);
                   int tgtSize = tgtFields.size();

                   String nameA;
                   String nameB;
                   List listA;
                   List listB;

                   if (size < tgtSize)
                   {
                      nameA = name;
                      nameB = key;
                      listA = fields;
                      listB = tgtFields;
                   }
                   else
                   {
                      nameA = key;
                      nameB = name;
                      listA = tgtFields;
                      listB = fields;
                   }

                   // Compare first N components of larger list with all
                   // components of smaller list.
                   if (listA.equals(listB.subList(0, listA.size())))
                   {
                      report.add("   " + nameA + " is redundant with " + nameB + ":");
                      report.add("      " + listA);
                      report.add("      " + listB);
                   }
                }
             }

             if (report.size() > 0)
             {
                redundantIndexReport.add(table + ":");
                iter = report.iterator();
                while (iter.hasNext())
                {
                   redundantIndexReport.add(iter.next());
                }
             }
          }

          /**
           * @deprecated
           */
          @Deprecated
          public void generateIndexReports(String filename)
          throws IOException
          {
             try (BufferedWriter bw = new BufferedWriter(new FileWriter(filename)))
             {
                bw.write("REDUNDANT INDICES:");
                bw.newLine();
                bw.write("------------------");
                bw.newLine();

                if (redundantIndexReport.size() == 0)
                {
                   bw.write("none");
                   bw.newLine();
                   bw.newLine();
                }
                else
                {
                   Iterator iter = redundantIndexReport.iterator();
                   while (iter.hasNext())
                   {
                      bw.write((String) iter.next());
                      bw.newLine();
                   }
                   bw.newLine();
                }

                bw.write("PROBLEM INDEX COMPONENTS:");
                bw.newLine();
                bw.write("-------------------------");
                bw.newLine();

                if (indexComponentReport.size() == 0)
                {
                   bw.write("none");
                   bw.newLine();
                   bw.newLine();
                }
                else
                {
                   Iterator iter = indexComponentReport.iterator();
                   while (iter.hasNext())
                   {
                      bw.write((String) iter.next());
                      bw.newLine();
                   }
                   bw.newLine();
                }
             }
          }

          /**
           * Determine whether the given tables are both metaschema tables.
           *
           * @param   left
           *          Left-hand table of a potential join (fully qualified).
           * @param   right
           *          Right-hand table of a potential join (fully qualified).
           *
           * @return  <code>true</code> if <em>both</em> tables are metaschema tables, else
           *          <code>false</code>.
           */
          public boolean areMetaTables(String left, String right)
          {
             String unqualLeft = left.substring(left.lastIndexOf('.') + 1).toLowerCase();
             String unqualRight = right.substring(right.lastIndexOf('.') + 1).toLowerCase();

             return (metaNames.contains(unqualLeft) && metaNames.contains(unqualRight));
          }

          /**
           * Return the set of (lowercased) metadata table names to be implemented in each permanent
           * database, instead of in an in-memory database.
           * <p>
           * A metadata table name's inclusion in this set will change how a table's schema is
           * converted, how its data (if any) is imported, and may change how queries involving the
           * table are converted.
           *
           * @return  Set of (lowercased) metadata table names to be implemented in each permanent
           *          database.
           */
          public Set<String> getMetaTablesInPersistentDB()
          {
             return metaTablesInPersistentDB;
          }

          /**
           * Return the unqualified data dump file name associated with the given metadata table.
           * These aren't necessarily included in the schema export for the metadata schema, as they
           * are for normal schemas.
           *
           * @param   metaTable
           *          Name of the metadata table.
           *
           * @return  Dump file name or <code>null</code> if this table isn't meant to be imported.
           */
          public String getMetaTableDumpFile(String metaTable)
          {
             return metaTableDumpFiles.get(metaTable.toLowerCase());
          }

          /**
           * Look up the cached table relation information for the given tables, if any, as if both
           * tables represent metaschema tables. If there is no cached, natural join between them,
           * return <code>null</code>.
           *
           * @param   left
           *          Left-hand table of a potential join (fully qualified).
           * @param   right
           *          Right-hand table of a potential join (fully qualified).
           *
           * @return  Table relation information, if any is cached, else <code>null</code>.
           */
          private TableRelation lookupMetaJoin(String left, String right)
          {
             // try lookup both ways, because we don't know which end is the foreign end, but
             // only one will work, since we figure this out when adding new entries
             String lookup = left + " " + right;
             if (joinCollisions.contains(lookup))
             {
                return null;
             }
             TableRelation join = joins.get(lookup);
             if (join == null)
             {
                lookup = right + " " + left;
                if (joinCollisions.contains(lookup))
                {
                   return null;
                }
                join = joins.get(lookup);
             }

             return join;
          }
       }
    }