SchemaWorker.java
/*
** Module : SchemaWorker.java
** Abstract : Exposes schema analysis capability to pattern engine
**
** Copyright (c) 2005-2025, Golden Code Development Corporation.
**
** -#- -I- --Date-- --JPRM-- -----------------------------------Description-----------------------------------
** 001 ECF 20050321 @20999 Created initial version. Detects and records
** natural table relation data among Progress
** schema tables.
** 002 ECF 20050526 @21290 Changes to support new expression engine
** implementation. Changed library registration
** mechanism based on superclass' modifications
** to support single library per pattern worker
** limit.
** 003 ECF 20050613 @21477 Reworked to handle adding inverse end of each
** detected join. Also added method to enable
** dumping all detected joins at the end of a
** source scan.
** 004 GES 20050706 @21669 Minor signature change for a utility method.
** 005 ECF 20051005 @22983 Update to store data to .schema file instead
** of .dict file. Necessitated by change to
** schema fixups ruleset, which preserves
** original AST as .dict and saves fixed up file
** with a .schema extension.
** 006 ECF 20060302 @24859 Fixed regression related to #005 (@22983).
** Natural join information was not properly
** persisting in the .schema file, due to AST ID
** mismatches between join data (based on .dict
** ASTs) and the newer, target, .schema AST.
** 007 ECF 20060325 @25241 Removed code which created duplicate schema
** file with the '.original' extension. This
** duplication is no longer necessary and is the
** source of constant registry problems.
** 008 ECF 20070302 @32265 Ignore self referencing join. That is, a join
** between a table and itself. This could happen
** if two buffers on the same backing table were
** joined using the OF keyword.
** 009 GES 20070629 @34341 Removed terse flag from AST persistence so
** that we retail line/column info in our ASTs.
** 010 GES 20070911 @35143 Minor change to how the schema dictionary is
** obtained.
** 011 ECF 20080730 @39272 Reduced memory footprint. Implemented finish
** method. Made creation of RelationAnalyzer
** (which pins SchemaDictionary) lazy. Added a
** user function to release SchemaDictionary.
** 012 GES 20081010 @40337 Fix to allow UAST hints to load properly.
** 013 ECF 20081216 @40938 Fixed attachRelationalData(). Allow foreign
** relation information to be stored for temp
** tables. The only restriction now is that both
** tables in the join need to reside in the same
** database.
** 014 GES 20090515 @42223 Moved to AstManager from AstRegistry/AstPersister.
** 015 ECF 20130124 Added worker function to get schema config.
** 016 OM 20130808 Added getForeignKey() to obtain the list of the fields used to join
** two Progress tables.
** joins Set morphed into a Hashtable for getter fast lookup.
** Generified private/local variables except for deprecated methods.
** 017 CA 20140114 Added getLegacyTableName API to resolve the legacy table name
** (case-sensitive) from a table name.
** 018 ECF 20131129 Added support for metaschema joins.
** 019 ECF 20150112 Added support for metadata tables implemented in a primary database.
** 020 ECF 20150326 Added nearly complete support for cross-schema natural joins. The
** feature still missing is joins from permanent tables to temp-tables.
** The other direction is supported.
** 021 ECF 20150715 Replace StringBuffer with StringBuilder.
** 022 ECF 20151210 Default _User dump file name to "_User.d" for Unix systems, else
** "_user.d".
** 023 OM 20160905 Small optimizations.
** 024 OM 20180118 Dropped "primary" notion for databases in p2j.cfg.xml. Maintenance.
** 025 CA 20180507 Added loadNonDefaults, to load non-default schemas.
** CA 20190513 Fixed buffer usage from super-classes, in OO.
** 026 GES 20190620 Added multiple inheritance support (for OO interface defs).
** 027 CA 20200412 Added incremental conversion support.
** OM 20210909 Used SchemaDictionary.META_DB instead of hardcoded constant.
** OM 20211122 Added hints-based support for 'unloading' schema.
** OM 20220330 Moved database conversion artifacts to ${cvtpath} folder.
** Made SCHEMA_POSTFIX static field public.
** TJD 20220504 Synced deprecation in javadoc and annotation
** CA 20230110 Let 'getImmediateChild' work with a set, if an array is used for the types.
** 028 GBB 20230512 Logging methods replaced by CentralLogger/ConversionStatus.
** 029 OM 20230630 Added support for the newly observed PARENT_ID_RELATION between joined tables.
** 030 CA 20230801 The key in the 'joins' map is formed from table names: in case of temp-tables,
** this does not guarantee that the temp-table definition match. For these cases,
** when the computed join doesn't match a previously computed one, do not emit a
** relation for these tables.
** 031 AL2 20240530 Use AstManager to initialize the directory instead of File.
** 032 CA 20240601 Schema dictionary files are loaded from the outermost grand-parent to the direct
** parent, as i.e. any temp-table definition in current class will hide definitions
** with the same name, from parents.
** 033 OM 20241030 Added _sec-authentication-system and _sec-authentication-domain tables to
** metaTablesInPersistentDB so that they get moved from standard to primary dbs.
** 034 CA 20250319 This worker can be used before the 'qualified_oo_name' is set in annotation
** (like in schema conversion phase), so ensure we can load the ClassDefinition.
*/
/*
** This program is free software: you can redistribute it and/or modify
** it under the terms of the GNU Affero General Public License as
** published by the Free Software Foundation, either version 3 of the
** License, or (at your option) any later version.
**
** This program is distributed in the hope that it will be useful,
** but WITHOUT ANY WARRANTY; without even the implied warranty of
** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
** GNU Affero General Public License for more details.
**
** You may find a copy of the GNU Affero GPL version 3 at the following
** location: https://www.gnu.org/licenses/agpl-3.0.en.html
**
** Additional terms under GNU Affero GPL version 3 section 7:
**
** Under Section 7 of the GNU Affero GPL version 3, the following additional
** terms apply to the works covered under the License. These additional terms
** are non-permissive additional terms allowed under Section 7 of the GNU
** Affero GPL version 3 and may not be removed by you.
**
** 0. Attribution Requirement.
**
** You must preserve all legal notices or author attributions in the covered
** work or Appropriate Legal Notices displayed by works containing the covered
** work. You may not remove from the covered work any author or developer
** credit already included within the covered work.
**
** 1. No License To Use Trademarks.
**
** This license does not grant any license or rights to use the trademarks
** Golden Code, FWD, any Golden Code or FWD logo, or any other trademarks
** of Golden Code Development Corporation. You are not authorized to use the
** name Golden Code, FWD, or the names of any author or contributor, for
** publicity purposes without written authorization.
**
** 2. No Misrepresentation of Affiliation.
**
** You may not represent yourself as Golden Code Development Corporation or FWD.
**
** You may not represent yourself for publicity purposes as associated with
** Golden Code Development Corporation, FWD, or any author or contributor to
** the covered work, without written authorization.
**
** 3. No Misrepresentation of Source or Origin.
**
** You may not represent the covered work as solely your work. All modified
** versions of the covered work must be marked in a reasonable way to make it
** clear that the modified work is not originating from Golden Code Development
** Corporation or FWD. All modified versions must contain the notices of
** attribution required in this license.
*/
package com.goldencode.p2j.schema;
import java.io.*;
import java.util.*;
import java.util.function.*;
import java.util.logging.*;
import com.goldencode.ast.*;
import com.goldencode.p2j.cfg.*;
import com.goldencode.p2j.convert.*;
import com.goldencode.p2j.pattern.*;
import com.goldencode.p2j.uast.*;
/**
* A pattern worker which exposes schema dictionary and table relation
* analysis functionality to the pattern engine.
*/
public class SchemaWorker
extends AbstractConversionWorker
implements SchemaParserTokenTypes
{
/** Fixed-up schema AST filename extension. */
public static final String SCHEMA_POSTFIX = ".schema";
/** List of all Progress field token types */
private static final Set<Integer> ALL_FIELD_TYPES = new HashSet<>();
/** Map of metadata table names (lowercase) to respective dump file names, if any */
private static final Map<String, String> metaTableDumpFiles = new HashMap<>();
/** Names of metadata tables (lowercase) implemented in a primary database */
private static final Set<String> metaTablesInPersistentDB;
/** Logger */
private static final ConversionStatus LOG = ConversionStatus.get(SchemaWorker.class);
static
{
// initialize list of field token types
for (int i = BEGIN_FIELDTYPES + 1; i < END_FIELDTYPES; i++)
{
ALL_FIELD_TYPES.add(i);
}
// initialize list of field token types used only by metadata
for (int i = BEGIN_METATYPES + 1; i < END_METATYPES; i++)
{
ALL_FIELD_TYPES.add(i);
}
// initialize map of metadata tables to their respective dump file names, if any;
// on Unix systems, default dump name is "_User.d", on Windows, "_user.d"
// TODO: check other platform defaults and implement as necessary
boolean unix = "UNIX".equals(Configuration.getParameter("opsys"));
metaTableDumpFiles.put("_user", unix ? "_User.d" : "_user.d");
// initialize set of metadata tables to be implemented in each primary database
Set<String> set = new LinkedHashSet<>();
//set.add("_connect");
set.add("_user");
set.add("_sec-authentication-system");
set.add("_sec-authentication-domain");
metaTablesInPersistentDB = Collections.unmodifiableSet(set);
}
/** Schema loader, used to load schema ASTs */
private SchemaLoader loader = new SchemaLoader();
/** Provides schema information lookup; loads default database(s) */
private SchemaDictionary dictionary = new SchemaDictionary((Set<String>) null);
/** Names of databases which contain tables involved in relations */
private Set<String> databases = new HashSet<>();
/**
* Set of all table relations detected in a scanning session. They are hashed using a
* combination of the table names that assures that there are no access collisions.
*/
private Map<String, TableRelation> joins = new Hashtable<>();
/** The set of "related tables" where their relation definition does not match. */
private Set<String> joinCollisions = new HashSet<>();
/** Set of IDs of index ASTs to be marked as referenced by "use-index" */
private Set<String> usedIndexes = new HashSet<>();
/** The class definition where this schema was loaded. */
private ClassDefinition clsDef = null;
/**
* Default constructor which defines the symbol libraries to be registered.
*
* @throws SchemaException
* if there is any error initializing the schema loader or
* schema dictionary.
*/
public SchemaWorker()
throws SchemaException
{
super();
setLibrary(new SchemaSupport());
}
/**
* Hook to provide termination processing for a pattern worker.
*/
public void finish()
{
loader = null;
dictionary = null;
databases = null;
joins = null;
joinCollisions = null;
usedIndexes = null;
clsDef = null;
super.finish();
}
/**
* Helper method to append an extension to a filename if it is not
* already there.
*
* @param filename
* Filename which may or may not already end with the extension.
* @param extension
* Extension to be set at the end of <code>filename</code>.
*
* @return <code>filename</code> with the required extension.
*/
public static String adjustFilename(String filename, String extension)
{
if (filename.endsWith(extension))
{
return filename;
}
return filename + extension;
}
/**
* Load a persisted schema AST from the specified file. This is intended primarily to load
* temp- and work-tables into an instance of the dictionary which already has a primary schema loaded.
*
* @param dictionary
* The schema dictionary instance.
* @param file
* The name of a persisted schema file which is associated with a Progress source code file.
*
* @return an instance of {@code SchemaException} if any error occurs reading the schema data from
* persistence and {@code null} on success.
*/
public static SchemaException loadFromPersistence(SchemaDictionary dictionary, String file)
{
String fname = adjustFilename(file, AstGenerator.DICT_POSTFIX);
File fil = new File(fname);
if (fil.exists())
{
try
{
dictionary.loadFromPersistence(fil.getAbsolutePath());
}
catch (SchemaException e)
{
return e;
}
}
return null;
}
/**
* This method is called each time a new AST is visited, just before the
* pattern engine processes that AST. We exploit this hook by loading the
* current temp/work-table schema (P2O) AST into the <code>P2OLookup</code>
* class' state. This enables us to resolve references to temp and work
* tables, which are local to the currently loaded source file AST.
*
* @param ast
* The root node of the source AST about to be processed by the
* pattern engine.
*/
@Override
public void visitAst(Aast ast)
{
clsDef = SymbolResolver.resolveClassDefinition(ast);
}
/**
* Composes a normalized index name, which is the fully qualified table
* name in all lowercase, in the form:
* <code><database_name><table_name><index_name></code>.
*
* @param index
* AST which defines index.
*
* @return Normalized index name as described above.
*/
private static String normalizedIndexName(Aast index)
{
String buf = TableRelation.normalizedTableName(index.getParent()) + '.' + index.getText();
return buf.toLowerCase();
}
/**
* Figures out which schema file to load as the basis for updates. If
* an original schema file (i.e., one that ends with ".original") exists,
* it is returned. Otherwise, the file represented by <code>filename</code>
* is returned.
*
* @param filename
* Root schema AST filename.
*
* @return Original filename or base filename if an original has not
* been created yet.
*
* @throws ConfigurationException
* if P2J_HOME has not been defined.
*/
private File getSchemaLoadFile(String filename)
throws ConfigurationException
{
return Configuration.toFile(filename);
}
/**
* Attaches relation data ASTs to any table which has been found in a natural join.
* <p>
* The <code>schemaRoot</code> AST will have been loaded from the <code>.schema</code> file;
* however, all join information is based upon ASTs loaded from the primordial
* <code>.dict</code> file (needed by the <code>SchemaDictionary</code>). Thus, table IDs
* within <code>TableRelation</code> objects will not match those in <code>schemaRoot</code>
* tables. We must therefore use the normalized table name to associated the table relations
* with the table ASTs found in the schema AST. The same technique applies to indexes.
*
* @param schemaRoot
* AST which represents a permanent or transient database schema.
* @param file
* File to which schema AST is saved after updates have been made.
* @param tables
* Map of normalized table names to table ASTs.
* @param indexes
* Map of normalized index names to index ASTs.
*/
private void persistSchema(Aast schemaRoot,
File file,
Map<String, Aast> tables,
Map<String, Aast> indexes)
throws SchemaException
{
Map<String, TableRelation> joins = new Hashtable<>(this.joins);
joins.keySet().removeAll(joinCollisions);
// attach relational data to tables for which such data has been collected
for (TableRelation join : joins.values())
{
attachRelationData(join, tables, schemaRoot);
}
// annotate indexes which are referenced by "use-index"
for (String usedIndex : usedIndexes)
{
Aast index = indexes.get(usedIndex);
if (index != null)
{
index.putAnnotation("use-index", Boolean.TRUE);
}
}
try
{
// Persist modified AST.
String filename = file.getAbsolutePath();
AstManager.get().saveTree(schemaRoot, filename, false);
}
catch (Exception exc)
{
throw new SchemaException("Error overwriting schema file", exc);
}
}
/**
* Walk the given schema AST and collect every table and index node by their normalized names
* in the given maps.
*
* @param schemaRoot
* Schema AST.
* @param tables
* Map in which to store table nodes.
* @param indexes
* Map in which to store index nodes.
*/
private void gatherTableAndIndexAsts(Aast schemaRoot,
Map<String, Aast> tables,
Map<String, Aast> indexes)
{
Iterator<Aast> iter = schemaRoot.iterator(2, null);
while (iter.hasNext())
{
Aast next = iter.next();
switch (next.getType())
{
case TABLE:
case TEMP_TABLE:
case WORK_TABLE:
tables.put(TableRelation.normalizedTableName(next), next);
break;
case INDEX:
if (indexes != null)
{
indexes.put(normalizedIndexName(next), next);
}
break;
}
}
}
/**
* Given a database and record AST, scan the collected join information
* to see if this record requires table relation information ASTs to be
* attached. For each join which was recorded in which this table
* participates on the foreign key end, add an AST describing the join.
* <p>
* The ASTs which are attached are of token type ONE_TO_ONE or MANY_TO_ONE,
* depending upon the circumstances of the join. Each has one child AST
* of type KEY_FIELD for each field which participates in the join. In
* addition, each has an annotation to record the ID of the table on the
* other end of the join. The ASTs are attached as direct children of the
* table to which they refer.
*
* @param join
* Object which describes a natural relation between tables.
* @param tables
* Map of record-type AST nodes, indexed by ID. This map may or
* may not contain one or both tables described by
* <code>join</code>.
*
* @throws SchemaException
* if any error occurs attaching a join information AST to
* the table AST <code>ast</code>.
*/
private void attachRelationData(TableRelation join, Map<String, Aast> tables, Aast schemaRoot)
throws SchemaException
{
String key;
// Get the node within the AST to be updated, which represents the
// left side of this join. If this join has a primary end, it will be
// the left table.
key = TableRelation.normalizedTableName(join, join.getLeft());
Aast left = tables.get(key);
// Get the node within the AST to be updated, which represents the
// right side of this join. If this join has a foreign end, it will be
// the right table.
key = TableRelation.normalizedTableName(join, join.getRight());
Aast right = tables.get(key);
// graft join data onto each table iff we have the ASTs from both sides of the join loaded
if (left != null && right != null)
{
String database = schemaRoot.getText();
Set<String> keyFields = join.getKey();
if (database.equalsIgnoreCase(left.getParent().getText()))
{
graftJoinNode(join.getInverseType(), keyFields, left, right);
}
if (database.equalsIgnoreCase(right.getParent().getText()))
{
graftJoinNode(join.getType(), keyFields, right, left);
}
}
}
/**
* Graft an AST node which describes a natural join onto a target table
* AST. The node's token type describes the nature of the join. It has
* an annotation with the ID of the table on the inverse end of the
* relation. The node has one child per field involved in the key which
* binds the tables.
*
* @param type
* Token type of the join node.
* @param key
* Set of names of fields which form the key between <code>target</code> and
* <code>inverse</code>. In the case of a metaschema join, this will contain only one
* field name, and that field will only be present in the <code>target</code> table.
* @param target
* Table AST to which join node is added.
* @param inverse
* AST which represents the table on the inverse end of the relation.
*
* @throws SchemaException
* if an error occurs adding the join AST node.
*/
private void graftJoinNode(int type, Set<String> key, Aast target, Aast inverse)
throws SchemaException
{
String table = TableRelation.normalizedTableName(inverse);
String database = TableRelation.normalizedDatabaseName(inverse);
AstSymbolResolver resolver = AstSymbolResolver.getResolver();
// create relation AST child to graft to the target table node
Aast relAst = new ProgressAst();
initializeAst(relAst, type, table, target, resolver);
// add an annotation indicating the table on the other end of the relation
relAst.putAnnotation("references", inverse.getId());
relAst.putAnnotation("database", database);
// add each field as a child component of the relation node
for (String fieldName : key)
{
Aast compAst = new ProgressAst();
initializeAst(compAst, KEY_FIELD, fieldName, relAst, resolver);
// Annotate with a reference to actual, local field.
Aast field = target.getImmediateChild(ALL_FIELD_TYPES, null);
while (field != null)
{
if (fieldName.equals(field.getText()))
{
compAst.putAnnotation("refid", field.getId());
break;
}
field = target.getImmediateChild(ALL_FIELD_TYPES, field);
}
// annotate with a cross-reference to same field in inverse table
field = inverse.getImmediateChild(ALL_FIELD_TYPES, null);
while (field != null)
{
if (fieldName.equals(field.getText()))
{
compAst.putAnnotation("crossrefid", field.getId());
break;
}
field = inverse.getImmediateChild(ALL_FIELD_TYPES, field);
}
}
}
/**
* Provides schema-related services to pattern engine users. An object of
* this class is registered with the symbol resolver to provide user
* function support for user expressions.
*/
public class SchemaSupport
{
/** List of strings comprising the redundant index report */
private List<String> redundantIndexReport = new ArrayList<>();
/** List of strings comprising the index component report */
private List<String> indexComponentReport = new ArrayList<>();
/** Unqualified, lower case names of all metadata tables for this project */
private Set<String> metaNames = Configuration.getSchemaConfig().getMetadata().getTables();
/**
* Return instance of schema dictionary.
*
* @return Current instance of schema dictionary.
*/
public SchemaDictionary getDictionary()
{
return dictionary;
}
/**
* Release the in-use dictionary.
*/
public void releaseDictionary()
{
dictionary = null;
}
/**
* Resolve the legacy table name, given the (possible) qualified table name.
*
* @param name
* The (possibly) qualified table name.
*
* @return See above.
*/
public String getLegacyTableName(String name)
{
try
{
Aast table = dictionary.getTable(name);
String legacyName = (String) table.getAnnotation("legacy_name");
if (legacyName == null)
{
throw new SchemaException("Legacy name not found for table:" + name);
}
return legacyName;
}
catch (SchemaException e)
{
throw new RuntimeException("Could not resolve legacy name for table: " + name);
}
}
/**
* Load all non-default schemas into the dictionary.
*
* @throws SchemaException
* if any error occurs loading source file-specific schema information.
*/
public void loadNonDefaults()
throws SchemaException
{
loader.loadNonDefaults(dictionary);
}
/**
* Initialize the schema dictionary for use with the current source
* file AST. Resets the state of the dictionary, but keeps default
* databases loaded. Also checks for hints to see if any additional
* databases need to be loaded and/or aliases assigned.
*
* @param filename
* Name of the source file represented by the AST currently
* being processed in the pattern engine. Any temp- and work-
* tables defined in this source file are loaded into the
* dictionary.
*
* @return <code>true</code>.
*
* @throws SchemaException
* if any error occurs loading source file-specific schema
* information.
*/
public boolean initializeDictionary(String filename)
throws SchemaException
{
String originalFileName = filename;
// Get a clean instance of the dictionary.
dictionary = new SchemaDictionary((Set<String>) null);
// Load hints for this source file, if any.
UastHints hints = new UastHints(filename);
// load extra databases into dictionary, if any
Set<String> dbList = hints.getDatabases();
if (dbList != null)
{
for (String db : dbList)
{
dictionary.loadSchema(db);
}
}
// Create database aliases, if any.
List<String[]> aliases = hints.getAliases();
if (aliases != null)
{
Iterator<String[]> iter = aliases.iterator();
while (iter.hasNext())
{
String[] pair = iter.next();
if (!dictionary.createAlias(pair[0], pair[1]))
{
LOG.log(Level.WARNING, "Dictionary could not create alias " + pair[0] + " for database " + pair[1]);
}
}
}
// first load the parent hierarchy
if (clsDef != null)
{
SchemaException[] exc = new SchemaException[1];
Consumer<ClassDefinition> consumer = (ClassDefinition cls) ->
{
if (exc[0] == null)
{
exc[0] = loadFromPersistence(dictionary, cls.getFilename());
}
};
clsDef.processParentGraph(consumer, false);
if (exc[0] != null)
{
throw exc[0];
}
}
// Load AST which defines temp- and work-tables for the current source file.
filename = Configuration.normalizeFilename(originalFileName + AstGenerator.DICT_POSTFIX);
if (AstManager.get().isExistingAst(filename))
{
Aast root = AstManager.get().loadTree(filename);
dictionary.loadFromAst(root, false);
}
return true;
}
/**
* Get the {@link SchemaConfig} that was read from the project
* configuration.
*
* @return Schema configuration.
*/
public SchemaConfig getSchemaConfig()
{
return Configuration.getSchemaConfig();
}
/**
* Return a string which describes a join between two tables. This is
* useful for debug and ad-hoc reporting purposes.
*
* @param left
* Left-hand table of the join (fully qualified).
* @param right
* Right-hand table of the join (fully qualified).
*
* @return A string describing the relation between these tables,
* if any. <code>null</code> if there was an error.
*/
public String describeJoin(String left, String right)
{
// Ignore a table which is joined to itself.
if (left.equals(right))
{
return null;
}
String description = null;
try
{
RelationAnalyzer analyzer = new RelationAnalyzer(dictionary);
TableRelation join = analyzer.createJoin(left, right);
description = join.toString();
}
catch (SchemaException exc)
{
description = exc.getMessage();
}
catch (Exception exc)
{
LOG.log(Level.WARNING,"", exc);
}
return description;
}
/**
* Create a {@link TableRelation} object based the natural join of
* the specified tables, and add it to the set of joins being collected
* by this worker. Error messages are printed to <code>stderr</code>.
*
* @param left
* Left-hand table of the join (fully qualified).
* @param right
* Right-hand table of the join (fully qualified).
*
* @return The instance of a <code>TableRelation</code>, if the join
* was successfully recorded; <code>null</code> if there was
* an error.
*/
public TableRelation recordJoin(String left, String right)
{
// Ignore a table which is joined to itself.
if (left.equals(right))
{
return null;
}
/*
* OM: The TableRelation is reflexive from Progress point of view. This is different from
* normal SQL notion of table join by foreign key.
* The Progress imposes that:
* - the name of the fields that compose the index to have the same name in both tables,
* if the relation is specified all fields participate in the join implicitly.
* - such combination to be unique.
* The analyzer will automatically detect the unique indexed table and choose it as the
* sql left side of the join.
* Because of that, createJoin(left, right) and createJoin(right, left) will generate
* "equal" objects, so we can freely reorder the tables. If we index the "joins"
* map using the alphabetic order of the table names we can quickly lookup the relation
* between the two tables.
*
* ECF: Exception to this rule is metaschema tables, which use "real" foreign key joins.
* In this case, when creating the cache key, we set the table with the foreign key
* reference as the left table and the table with the primary key referent as the right
* table, regardless of their names' alphabetical order.
*/
TableRelation join = null;
String joinLookup = null;
boolean isMeta = areMetaTables(left, right);
if (isMeta)
{
join = lookupMetaJoin(left, right);
}
else
{
joinLookup = left.compareTo(right) > 0 ? right + " " + left : left + " " + right;
if (joinCollisions.contains(joinLookup))
{
return null;
}
join = joins.get(joinLookup);
}
try
{
TableRelation found = join;
RelationAnalyzer analyzer = new RelationAnalyzer(dictionary);
join = analyzer.createJoin(left, right);
/*
* Save the relation for later lookups. It will:
* - cache the value for future calls for same tables (which can be swapped)
* - allow extracting the fields of key for nested can-find cases where the relation
* between the two tables has to be expanded explicitly in hql.
*/
if (isMeta)
{
joinLookup = TableRelation.normalizedTableName(join, join.getRight()) + " " +
TableRelation.normalizedTableName(join, join.getLeft());
}
// check the string representation which uses the entire relation - if these do not match, then we
// have a collision
if (found != null)
{
if (found.toString().equals(join.toString()))
{
return found;
}
else
{
joinCollisions.add(joinLookup);
return null;
}
}
joins.put(joinLookup, join);
databases.addAll(join.getDatabases());
}
catch (Exception exc)
{
LOG.log(Level.SEVERE, "", exc);
}
return join;
}
/**
* Return a string which contains the fields (space separated) that are part of the foreign
* key that joins two tables. The relation should already have been added with
* {@link #recordJoin(String, String)}.
*
* @param left
* Left-hand table of the join (fully qualified).
* @param right
* Right-hand table of the join (fully qualified).
*
* @return A string describing the foreign key of these tables, if any, and
* <code>null</code> if tables are unrelated or there was an error.
*/
public String getForeignKey(String left, String right)
{
// Eliminate common cases
if (left == null || right == null || left.equals(right))
{
return null;
}
TableRelation join = null;
if (areMetaTables(left, right))
{
join = lookupMetaJoin(left, right);
}
else
{
String lookup = left.compareTo(right) > 0 ? right + " " + left : left + " " + right;
if (joinCollisions.contains(lookup))
{
return null;
}
join = joins.get(lookup);
}
if (join == null)
{
return null;
}
if (join.getType() == PARENT_ID_RELATION)
{
return null; // ""; ?
}
StringBuilder sb = new StringBuilder();
for (String k : join.getKey())
{
if (sb.length() > 0)
{
sb.append(' ');
}
sb.append(k);
}
return sb.toString();
}
/**
* Add index AST to the set of indexes referenced by the Progress
* "use-index" directive. Update the set of databases to be persisted
* as a result of this discovery. The actual annotation occurs later,
* when the corresponding database AST is persisted.
*
* @param index
* AST of the index to be marked.
*/
public void markIndexUsed(Aast index)
{
usedIndexes.add(normalizedIndexName(index));
databases.add(index.getAncestor(-1).getText());
}
/**
* Graft all discovered natural join information into the appropriate schema ASTs and
* persist those ASTs for both permanent and temp-tables.
* <p>
* TODO: determine whether natural joins are legal from permanent tables to temp-tables and
* if so, add support for this idiom. This will require a more precise way to identify the
* temp-table end of the join than the database-qualified table name method used currently.
*
* @param sourceFiles
* Names of source files in which temp-table natural joins were detected.
*
* @throws ConfigurationException
* if P2J_HOME environment variable is not set.
* @throws SchemaException
* if there were errors updating or saving the affected schema ASTs.
*/
public void persistSchemas(Set<String> sourceFiles)
throws ConfigurationException,
SchemaException
{
SchemaConfig schemaConfig = Configuration.getSchemaConfig();
Map<File, Aast> schemaMap = new LinkedHashMap<>();
// first pass collects permanent tables and indexes from databases participating in
// natural joins
Map<String, Aast> tables = new HashMap<>();
Map<String, Aast> indexes = new HashMap<>();
for (String next : databases)
{
if (next.startsWith("@"))
{
continue;
}
if (SchemaDictionary.META_DB.equals(next))
{
next = schemaConfig.getMetadata().getName();
}
String filename = schemaConfig.getSchemaFileName(next, SCHEMA_POSTFIX);
File saveFile = new File(filename);
File loadFile = getSchemaLoadFile(filename);
if (loadFile.exists())
{
Aast schemaRoot = loader.loadAst(loadFile);
schemaMap.put(saveFile, schemaRoot);
gatherTableAndIndexAsts(schemaRoot, tables, indexes);
}
}
// second pass attaches join data and persists the schema ASTs
for (Map.Entry<File, Aast> entry : schemaMap.entrySet())
{
File saveFile = entry.getKey();
Aast schemaRoot = entry.getValue();
persistSchema(schemaRoot, saveFile, tables, indexes);
}
// process temp-table schemas
for (String filename : sourceFiles)
{
filename = adjustFilename(filename, SCHEMA_POSTFIX);
File loadFile = getSchemaLoadFile(filename);
if (loadFile.exists())
{
Map<String, Aast> tempTables = new HashMap<>();
Map<String, Aast> tempIndexes = new HashMap<>();
Aast schemaRoot = loader.loadAst(loadFile);
gatherTableAndIndexAsts(schemaRoot, tempTables, tempIndexes);
tables.putAll(tempTables);
try
{
persistSchema(schemaRoot, loadFile, tables, tempIndexes);
}
finally
{
tables.keySet().removeAll(tempTables.keySet());
}
}
}
}
/**
* Dump to <code>stdout</code> the set of natural joins detected by
* all invocations of {@link #recordJoin} during a scan of some set
* of Progress source code.
*/
public void dumpNaturalJoins()
{
System.out.println("ALL NATURAL JOINS: ");
System.out.println("------------------");
for (Map.Entry<String, TableRelation> pair : joins.entrySet())
{
TableRelation join = pair.getValue();
String key = pair.getKey();
if (joinCollisions.contains(key))
{
System.out.println("COLLISION - natural join ignored on tables: " + key);
}
else
{
System.out.println(join.toString());
}
}
}
/**
* @deprecated
*/
@Deprecated
public void analyzeIndexComponents(String table, Map indices)
{
List<String> report = new ArrayList<>();
Set<String> done = new HashSet<>();
Iterator<String> iter = indices.keySet().iterator();
while (iter.hasNext())
{
// Get current index name.
String name = iter.next();
done.add(name);
// Get list of current index component fields.
List fields = (List) indices.get(name);
Iterator iter2 = indices.keySet().iterator();
while (iter2.hasNext())
{
String key = (String) iter2.next();
// Skip indices already reviewed.
if (done.contains(key))
{
continue;
}
List fields2 = (List) indices.get(key);
// Iterate through "outer" index' fields.
boolean problem = false;
Iterator subIter = fields.iterator();
for (int i = 0; subIter.hasNext(); i++)
{
String next = (String) subIter.next();
int index = fields2.indexOf(next);
if (index >= 0 && index != i)
{
if (!problem)
{
problem = true;
report.add(" " + name + ": " + fields);
report.add(" " + key + ": " + fields2);
}
report.add(" " + next + ": " + name + "[" + i +
"] ... " + key + "[" + index + "]");
}
}
}
}
if (report.size() > 0)
{
indexComponentReport.add(table + ":");
iter = report.iterator();
while (iter.hasNext())
{
indexComponentReport.add(iter.next());
}
}
}
/**
* @deprecated
*/
@Deprecated
public void analyzeIndexRedundancy(String table, Map indices)
{
List<String> report = new ArrayList<>();
Set<String> done = new HashSet<>();
Iterator<String> iter = indices.keySet().iterator();
while (iter.hasNext())
{
// Get current index name.
String name = iter.next();
done.add(name);
// Get list of current index component fields.
List fields = (List) indices.get(name);
int size = fields.size();
Iterator iter2 = indices.keySet().iterator();
while (iter2.hasNext())
{
String key = (String) iter2.next();
// Skip indices already reviewed.
if (done.contains(key))
{
continue;
}
// Get list of compare target's component fields.
List tgtFields = (List) indices.get(key);
int tgtSize = tgtFields.size();
String nameA;
String nameB;
List listA;
List listB;
if (size < tgtSize)
{
nameA = name;
nameB = key;
listA = fields;
listB = tgtFields;
}
else
{
nameA = key;
nameB = name;
listA = tgtFields;
listB = fields;
}
// Compare first N components of larger list with all
// components of smaller list.
if (listA.equals(listB.subList(0, listA.size())))
{
report.add(" " + nameA + " is redundant with " + nameB + ":");
report.add(" " + listA);
report.add(" " + listB);
}
}
}
if (report.size() > 0)
{
redundantIndexReport.add(table + ":");
iter = report.iterator();
while (iter.hasNext())
{
redundantIndexReport.add(iter.next());
}
}
}
/**
* @deprecated
*/
@Deprecated
public void generateIndexReports(String filename)
throws IOException
{
try (BufferedWriter bw = new BufferedWriter(new FileWriter(filename)))
{
bw.write("REDUNDANT INDICES:");
bw.newLine();
bw.write("------------------");
bw.newLine();
if (redundantIndexReport.size() == 0)
{
bw.write("none");
bw.newLine();
bw.newLine();
}
else
{
Iterator iter = redundantIndexReport.iterator();
while (iter.hasNext())
{
bw.write((String) iter.next());
bw.newLine();
}
bw.newLine();
}
bw.write("PROBLEM INDEX COMPONENTS:");
bw.newLine();
bw.write("-------------------------");
bw.newLine();
if (indexComponentReport.size() == 0)
{
bw.write("none");
bw.newLine();
bw.newLine();
}
else
{
Iterator iter = indexComponentReport.iterator();
while (iter.hasNext())
{
bw.write((String) iter.next());
bw.newLine();
}
bw.newLine();
}
}
}
/**
* Determine whether the given tables are both metaschema tables.
*
* @param left
* Left-hand table of a potential join (fully qualified).
* @param right
* Right-hand table of a potential join (fully qualified).
*
* @return <code>true</code> if <em>both</em> tables are metaschema tables, else
* <code>false</code>.
*/
public boolean areMetaTables(String left, String right)
{
String unqualLeft = left.substring(left.lastIndexOf('.') + 1).toLowerCase();
String unqualRight = right.substring(right.lastIndexOf('.') + 1).toLowerCase();
return (metaNames.contains(unqualLeft) && metaNames.contains(unqualRight));
}
/**
* Return the set of (lowercased) metadata table names to be implemented in each permanent
* database, instead of in an in-memory database.
* <p>
* A metadata table name's inclusion in this set will change how a table's schema is
* converted, how its data (if any) is imported, and may change how queries involving the
* table are converted.
*
* @return Set of (lowercased) metadata table names to be implemented in each permanent
* database.
*/
public Set<String> getMetaTablesInPersistentDB()
{
return metaTablesInPersistentDB;
}
/**
* Return the unqualified data dump file name associated with the given metadata table.
* These aren't necessarily included in the schema export for the metadata schema, as they
* are for normal schemas.
*
* @param metaTable
* Name of the metadata table.
*
* @return Dump file name or <code>null</code> if this table isn't meant to be imported.
*/
public String getMetaTableDumpFile(String metaTable)
{
return metaTableDumpFiles.get(metaTable.toLowerCase());
}
/**
* Look up the cached table relation information for the given tables, if any, as if both
* tables represent metaschema tables. If there is no cached, natural join between them,
* return <code>null</code>.
*
* @param left
* Left-hand table of a potential join (fully qualified).
* @param right
* Right-hand table of a potential join (fully qualified).
*
* @return Table relation information, if any is cached, else <code>null</code>.
*/
private TableRelation lookupMetaJoin(String left, String right)
{
// try lookup both ways, because we don't know which end is the foreign end, but
// only one will work, since we figure this out when adding new entries
String lookup = left + " " + right;
if (joinCollisions.contains(lookup))
{
return null;
}
TableRelation join = joins.get(lookup);
if (join == null)
{
lookup = right + " " + left;
if (joinCollisions.contains(lookup))
{
return null;
}
join = joins.get(lookup);
}
return join;
}
}
}