ConversionDriver.java
/*
** Module : ConversionDriver.java
** Abstract : runs the entire conversion process for a file or set of files
**
** Copyright (c) 2006-2024, Golden Code Development Corporation.
**
** -#- -I- --Date-- --JPRM-- -----------------------------------Description-----------------------------------
** 001 GES 20060131 @24171 First version. Supports:
** - schema loader/schema fixups
** - scan driver (preprocess/lex/parse/persist)
** - post parse fixups
** - call graph generation
** - unreachable code processing
** - annotations
** - frame generator
** - base structure
** - core conversion
** - brew
** - the above steps are separated into a
** front end and back end that are separately
** or jointly runnable
** - scanning results are fully or partially
** cleared when running the front end
** - quick mode - reset with cached front end
** results before running back end
** 002 GES 20060131 @24172 Added middle processing.
** 003 ECF 20060203 @24343 Enabled temp table P2O processing. Had been
** commented out pending testing.
** 004 GES 20060217 @24612 Ensure that hints files don't get deleted.
** 005 ECF 20060210 @24504 Make temp table processing conditional. Only
** attempted if source-level temp table schema
** files exist.
** 006 ECF 20060220 @24618 Fixed metadata schema file handling. Needed
** additional logic to correctly omit this file
** from the list of fixed-up schema files.
** 007 GES 20060308 @24932 Only cleanup generated schema files when
** schema processing is being run (in front
** end).
** 008 GES 20060311 @24973 Force different match lists into the name
** converter for different types of pattern
** engine runs.
** 009 ECF 20060405 @25372 Made archival of key files (with .pristine
** extension) optional. This option is enabled
** with the '-A' command line option.
** 010 GES 20071015 @35527 Filenames including regex characters like
** square brackets, need to be escaped in
** certain cases.
** 011 GES 20090105 @41010 Added file-based filename list as an option.
** The filespec option is VERY slow when there
** are large numbers of files. This allows a
** very large file list to be passed and/or
** a custom list to be passed which cannot
** easily be duplicated by a single filespec.
** Eliminated use of the FileSpec class during
** cleanup of prior scan processing results.
** This is significantly faster.
** 012 GES 20090424 @41928 Import change.
** 013 GES 20090429 @42046 Match package and class name changes.
** 014 GES 20090515 @42204 Removed quick mode and archive support. These
** did not work properly and used unnecessary
** features in the backing infrastructure.
** 015 GES 20091117 @44417 Modified the name converter loading to be aware
** of database defaults when necessary.
** 016 ECF 20101006 Allow conversion to proceed if no database metadata
** file has been configured.
** 017 CA 20101102 Call the generate_ddl.xml as part of M0, for each
** schema and registered target DB.
** 018 LMR 20110106 Fixes to reflect renaming of load() method in
** Configuration class to getSchemaConfig() and
** removal of its 'key' parameter.
** 019 GES 20110824 cleanScanResults() was updated to clear from some
** "fakeout" directories. These directories hold
** class/interface definitions of the Progress
** built-ins and .NET classes/interfaces that are in
** use. In particular, the ASTs cannot be reused if
** there has been parser/lexer changes.
** 020 OM 20121121 Add support for writing DDL schema for sequences.
** 021 ECF 20130125 Added database metadata support.
** 022 ECF 20130624 Additional metadata support to generate DDL and export data.
** 023 CA 20140313 Changed callgraph generation to use a callgraph DB.
** 024 HC 20140613 Logging added.
** 025 ECF 20141218 Made metadata-related steps conditional; some projects do not use
** metadata.
** 026 VIG 20141121 Added menus generation.
** 027 ECF 20150327 Refactored middle conversion to support cross-schema natural joins.
** 028 ECF 20150804 Exit with System.exit(1) on fatal conversion error.
** 029 OM 20150824 Added pre-processing for P2O Generation for temp-table schema files.
** 030 GES 20160224 Cleanup file list inputs (trim whitespace and ignore blank lines).
** 031 IAS 20160331 Add input streams' close.
** 032 ECF 20160302 Added "abort-on-error" option for the scan portion of the front end
** and implemented multithreaded front end scan.
** 033 GES 20160311 Added gap analysis marking to the front end processing.
** 034 IAS 20160509 Fixed DDL dialects processing in case of more than one
** default namespace.
** 035 OM 20161111 Implemented SHARED FRAMEs hierarchy.
** 036 ECF 20170213 Fixed typo in syntax output.
** 037 CA 20170825 Added a mechanism to inject new ASTs into the source list, generated
** by the TRPL rules.
** Added code to collect and generate code for ADM embedded windows.
** 038 CA 20170827 Added MX mode: in this mode, the database schema(s) are no longer
** processed - it is assumed already generated by a previous M0 run;
** this allows running the conversion without having to re-generate DMO
** and DDL for the physical database. Consider MX mode experimental,
** if in doubt, run with M0 mode.
** 039 GES 20180521 Added "Early Annotations" as a new step in the front end. This is
** intended to implement annotations processing that is safe for usage
** in reporting. No changes should be made to the trees which would not
** normally be put into post parse fixups.
** 040 GES 20181022 Massive refactoring to move most code into a common abstract base
** class. This allows a much simpler implementation of alternate
** drivers.
** 041 GES 20181209 Added annotations_prep phase.
** 042 ECF 20190218 Added blacklist (filespec + ignore list) mode.
** 043 CA 20200412 Added incremental conversion support.
** 044 OM 20200618 Conversion runs in headless mode (there is no client side to print the errors).
** 045 CA 20201008 Added generation for legacy open client proxy Java programs.
** 046 GES 20210707 Removed some dead code and moved to use of some common logic in the FileList
** class.
** OM 20210923 Added -P<name> command line option for specifying the configuration profile.
** CA 20211012 Fixed generated WSDL and reworked SOAP support to rely on namespace, when
** resolving an operation.
** OM 20220325 Added warning message when working in m1 profile.
** OM 20220330 Moved conversion artifacts to ${cvtpath} folder.
** CA 20220412 Added file-set support (either -Z command-line option or p2j.cfg.xml
** configuration).
** OM 20220713 Added support for generating POJOs.
** CA 20220501 Each profile has the same structure as the main 'global' config. Allow multiple
** profiles to be ran at once, with the conversion switching the state between
** profiles, when a resource (like a file or namespace) is being processed. Only
** front phase is supported at this time.
** CA 20220428 Legacy OpenClient proxy conversion must be done for each (Sub)AppObject, as a
** program may appear in multiple sections.
** CA 20220513 Optimized proxy generation, all programs belonging to multiple (Sub)AppObjects
** are processed multiple times, in batches.
** CA 20220516 The (Sub)AppObject JASTs must be tracked by Java package, too, as the same name
** may appear in multiple (Sub)AppObject configs, but for different packages.
** Allow standalone generation of the Java code for the OpenClient proxy programs.
** CA 20220517 Generate proxy OpenClient programs only for programs part of the initial
** conversion list.
** CA 20220526 Allow the proxy classes and methods to use the original legacy names, via the
** 'proxy-legacy-names' config parameter.
** CA 20220603 Allow the generation of proxy Java programs in the folder specified by
** 'proxy-output-root'.
** 047 OM 20240311 Do not generate the <schema>.meta.xml any more. The information is already
** present in DMO interface annotations.
** 048 GES 20241218 Added syntax help and javadoc for the -Z file set support.
*/
/*
** This program is free software: you can redistribute it and/or modify
** it under the terms of the GNU Affero General Public License as
** published by the Free Software Foundation, either version 3 of the
** License, or (at your option) any later version.
**
** This program is distributed in the hope that it will be useful,
** but WITHOUT ANY WARRANTY; without even the implied warranty of
** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
** GNU Affero General Public License for more details.
**
** You may find a copy of the GNU Affero GPL version 3 at the following
** location: https://www.gnu.org/licenses/agpl-3.0.en.html
**
** Additional terms under GNU Affero GPL version 3 section 7:
**
** Under Section 7 of the GNU Affero GPL version 3, the following additional
** terms apply to the works covered under the License. These additional terms
** are non-permissive additional terms allowed under Section 7 of the GNU
** Affero GPL version 3 and may not be removed by you.
**
** 0. Attribution Requirement.
**
** You must preserve all legal notices or author attributions in the covered
** work or Appropriate Legal Notices displayed by works containing the covered
** work. You may not remove from the covered work any author or developer
** credit already included within the covered work.
**
** 1. No License To Use Trademarks.
**
** This license does not grant any license or rights to use the trademarks
** Golden Code, FWD, any Golden Code or FWD logo, or any other trademarks
** of Golden Code Development Corporation. You are not authorized to use the
** name Golden Code, FWD, or the names of any author or contributor, for
** publicity purposes without written authorization.
**
** 2. No Misrepresentation of Affiliation.
**
** You may not represent yourself as Golden Code Development Corporation or FWD.
**
** You may not represent yourself for publicity purposes as associated with
** Golden Code Development Corporation, FWD, or any author or contributor to
** the covered work, without written authorization.
**
** 3. No Misrepresentation of Source or Origin.
**
** You may not represent the covered work as solely your work. All modified
** versions of the covered work must be marked in a reasonable way to make it
** clear that the modified work is not originating from Golden Code Development
** Corporation or FWD. All modified versions must contain the notices of
** attribution required in this license.
*/
/*
TODOS:
- back end for the data side (create database(s), run import)
- separate modes for various cleanup (deletion of different sets of output)
- more granular debug options (step specific instead of global)
- call graph mode for back-end processing
*/
package com.goldencode.p2j.convert;
import com.goldencode.ast.*;
import com.goldencode.p2j.cfg.*;
import com.goldencode.p2j.convert.db.*;
import com.goldencode.p2j.pattern.*;
import com.goldencode.p2j.schema.*;
import com.goldencode.p2j.uast.*;
import com.goldencode.p2j.util.*;
import java.io.*;
import java.util.*;
/**
* Provides a harness to drive the end-to-end conversion process for a
* user-defined list of files.
* <p>
* This program provides a command line interface. See {@link #main} for the
* syntax.
*/
public class ConversionDriver
extends TransformDriver
{
/** Match list for source code name conversion. */
private static final String codenames = Configuration.getParameter("codenames");
/** Associates mode names with detailed mode definitions. */
private static final Map<String, RunMode> modes = new HashMap<>();
/** Help text, which needs to be implemented in each subclass. */
private static final String[] helpText = new String[]
{
"---------------------------- P2J Conversion Driver -----------------------------",
"",
"Syntax:",
" java -DP2J_HOME=<home> ConversionDriver [-options]* <mode> <file-specs>",
"",
"Where",
" Options (F, S, X and Z are mutually exclusive. If none is specified <file-specs>",
" is a list of files added to command line):",
" Dn = set DEBUG level 'n'. Must be a numeric digit between " + MSG_NONE + " (none)",
" and " + MSG_TRACE + " (trace) inclusive",
" F = <file-specs> is a <filename> containing a custom FILE-LIST (deprecated)",
" S = <file-specs> is composed of a root <directory> and \"<file-filter>\"",
" to select SPECIFIC files",
" X = <file-specs> is similar to S but has an additional custom EXCLUDE",
" list file \"<filename>\" to be ignored (deprecated)",
" N = NO recursion in <file-filter> (S options)",
" Z = <file-set> is a <filename> containing a list of directives which define a file",
" set to process; these directives provide a superset of the capabilities of F, S",
" and X options",
" A = ABORT scan part of front end on an error",
" Tn = specify number of THREADS 'n' to use for front end file scan ('n'",
" must be >= 0; defaults to 1; set to 0 to use 1 thread per",
" available CPU)",
" WARNING: this mode is not compatible if there are legacy .cls files",
" L = LIST files to be processed only; do not perform any further task",
" I = INCREMENTAL conversion; only changed/new programs will be parsed",
" and converted",
" Pname = the <name> of the configuration PROFILE follows the option. Can appear multiple times,",
" to specify which profiles are included in this run.",
"",
" Mode (one or more of the following values, use '+' to delimit if there are two",
" or more values; don't insert spaces!)",
" -------------------------------- Front End --------------------------------",
" F0 = preprocessor (in honor cache mode), lexer, parser, AST persistence and",
" post-parse fixups",
" F1 = F0 + forced preprocessor execution",
" F2 = F1 + schema loader/fixups",
" F3 = F2 + call graph generation",
"",
" ---------------------------------- Middle ---------------------------------",
" M0 = schema annotations, P2O generation, docs, DMO generation, brew for DMO",
" classes",
" M1 = M1 + length statistics",
"",
/*
" ----------------------Data Back End------------------------",
" DB = generate DDL, create the database(s) and import data",
"",
*/
" ------------------------------ Code Back End ------------------------------",
" CB = unreachable code, annotations, frame generator, base structure, core",
" conversion and brew",
"",
" filelist = arbitrary list of absolute and/or relative file names to scan",
" (default approach)",
" directory = a relative or absolute path (only used when using <file-filter>,",
" S/X options)",
" file-filter= file filter specification of the filenames in the given directory",
" to scan, should be enclosed in double quotes if any of the",
" wildcard characters '*' or '?' are used (only used with S/X",
" modes)",
" filename = when used with the -F option, this is a single absolute or",
" relative name of a text file that contains a custom list of",
" absolute and/or relative file names of files to scan (i.e. the",
" file list will be read from a file instead of being hard coded",
" on the command line); there must be one file name per line in",
" the file;",
" when used with the -X option, this is a single absolute or",
" relative name of a text file that contains a custom list of",
" specifications of file names to ignore; all other files",
" specified by the primary <file-filter> are scanned.",
" when used with the -Z option, this is a single absolute or",
" relative name of a text file that contains directives that",
" define a file set (i.e. the list of files to be processed);",
" These directives are available, one directive per line:",
" D <recursive_top_level_dir> <file_spec> adds a top level",
" directory and a file spec which will be recursively",
" processed to add 0 or more files into the list\n",
" N <non_recursive_top_level_dir> <file_spec> adds a top level",
" directory and file spec which will be non-recursively",
" processed to add 0 or more files into the list",
" F <absolute_or_relative_file_name> explicitly adds a specific",
" file into the list",
" X <exclusion_filter_expression_including_wildcards> filter to",
" remove 0 or more files from the list",
" I <inclusion_filter_expression_including_wildcards> filter to",
" add something removed back into the list",
};
/** Job title, which needs to be implemented in each subclass. */
private static final String TITLE = "P2J Conversion Driver";
static
{
modes.put("F0", new CvtRunMode(true , false, false, false, false, false, false, false, false));
modes.put("F1", new CvtRunMode(true , true , false, false, false, false, false, false, false));
modes.put("F2", new CvtRunMode(true , true , true , false, false, false, false, false, false));
modes.put("F3", new CvtRunMode(true , true , true , true , false, false, false, false, false));
modes.put("M0", new CvtRunMode(false, false, false, false, true , false, false, false, false));
modes.put("M1", new CvtRunMode(false, false, false, false, true , true , false, false, false));
modes.put("MX", new CvtRunMode(false, false, false, false, true , false, true, false, false));
modes.put("MY", new CvtRunMode(false, false, false, false, true , false, true, false, true));
modes.put("CB", new CvtRunMode(false, false, false, false, false, false, false, true, false));
}
/** The list of Java AST (abstract syntax tree) files. */
private FileList jasts = null;
/** Java ASTs (abstract syntax trees) created during frame generation. */
private String[] frames = null;
/** Java ASTs (abstract syntax trees) created during menus generation. */
private String[] menus = null;
/** The list of database schema files. */
private FileList dschemas = null;
/** The list of database P2O files. */
private FileList dp2os = null;
/** The list of Progress source file specific P2O files. */
private FileList sp2os = null;
/**
* Creates an instance with a specific configuration.
*
* @param job
* The details of this driver run.
*/
ConversionDriver(JobDefinition job)
{
super(job);
}
/**
* Drive the middle of the conversion process using the schema dictionary
* lists generated from the output of the <code>SchemaLoader</code> and
* the file list of Progress AST files that is already defined in this
* instance.
* <p>
* This runs the following:
* <ul>
* <li> schema annotations (analyze Progress source code)
* <li> P2O generation for each database schema and the schema files
* associated with all temp-tables (stored in Progress source
* file specific filenames)
* <li> length statistics (optional)
* <li> generate Hibernate mapping documents
* <li> DMO generation for each database schema and the schema files
* associated with all temp-tables (stored in Progress source
* file specific filenames)
* <li> brew (converts from JAST to <code>.java</code>)
* </ul>
*
* @throws ConfigurationException If there is problem reading/parsing the configuration.
* @throws AstException If there is an problem processing ASTs.
* @throws SchemaException If there is a problem with the schema.
* @throws IOException If there is a problem with I/O.
*/
protected void middle()
throws ConfigurationException,
AstException,
SchemaException,
IOException
{
CvtRunMode cmode = (CvtRunMode) mode;
// ensure that downstream processing has the proper list of files
convertSourceNamesToAstNames();
processTrees("Schema Annotations (scan Progress source code)",
"schema/annotations",
datanames,
true,
asts,
debug);
boolean omitMeta = omitSchemaMetadata();
if (!cmode.skipDBSchema)
{
// build database schema list
dschemas = createDatabaseList(".schema", omitMeta);
}
// build Progress source schema list
FileList sschemas = convertSourceNamesToDictNames(".schema");
if (!cmode.skipDBSchema)
{
processTrees("P2O Generation (database schema files)",
"schema/p2o",
datanames,
true,
dschemas,
debug);
}
// convert source file schema files to P2O (if any exist for these specific source files)
if (sschemas.size() > 0)
{
processTrees("Pre-processing P2O Generation (temp-table schema files)",
"schema/p2o_pre",
datanames,
true,
sschemas,
debug);
processTrees("P2O Generation (temp-table schema files)",
"schema/p2o",
datanames,
true,
sschemas,
debug);
}
if (!cmode.skipDBSchema)
{
processTrees("P2O Post-processing (database schema files)",
"schema/p2o_post",
datanames,
true,
dschemas,
debug);
}
// convert source file schema files to P2O (if any exist for these specific source files)
if (sschemas.size() > 0)
{
processTrees("P2O Post-processing (temp-table schema files)",
"schema/p2o_post",
datanames,
true,
sschemas,
debug);
}
if (!cmode.skipDBSchema)
{
// build database p2o list
dp2os = createDatabaseList(".p2o", omitMeta);
if (cmode.length)
{
System.out.println(
"WARNING: The data analyser is not currently in use and it is not being maintained.\n" +
"\tThe attempt to enable it can cause the conversion to halt especially if the project\n" +
"\tsettings are different than those the application expected at the time it was developed.");
processTrees("Database Field Length Statistics",
"schema/data_analyzer",
datanames,
true,
dp2os,
debug);
}
}
// build Progress source P2O list
sp2os = convertSourceNamesToDictNames(".p2o");
if (!cmode.omitSchemaBrew)
{
FileList dmos = generateDMOs(cmode.skipDBSchema);
brew(dmos, "Data Model Objects (DMOs)");
FileList pojos = generatePOJOs(cmode.skipDBSchema);
brew(pojos, "Plain 0ld Java 0bjects (P0J0s)");
}
if (!cmode.skipDBSchema && !cmode.omitSchemaBrew)
{
// the ddl table and indexes are created at once with dmo interfaces
System.out.println("TODO: merge hand-written DMOs into generated DDL output " +
"([merge-dmo-root] entry in [p2j.cfg.xml])");
}
}
/**
* Drive the back end of the conversion process using the file list
* of Progress AST files that is already defined in this instance.
* <p>
* This runs the following:
* <ul>
* <li> unreachable code analysis
* <li> code conversion annotations
* <li> frame generator
* <li> business logic base structure
* <li> core conversion
* <li> brew (converts from JAST to <code>.java</code>)
* </ul>
*
* @throws ConfigurationException If there is problem reading/parsing the configuration.
* @throws AstException If there is an problem processing ASTs.
* @throws SchemaException If there is a problem with the schema.
* @throws IOException If there is a problem with I/O.
*/
protected void back()
throws ConfigurationException,
AstException,
SchemaException,
IOException
{
// ensure that downstream processing has the proper list of files
convertSourceNamesToAstNames();
if (Configuration.getParameter("add-embedded-windows", false))
{
processTrees("Generate embedded code for ADM windows",
"adm/adm_windows",
null,
false,
asts,
debug);
}
processTrees("Unreachable Code Analysis",
"unreachable/unreachable",
null,
false,
asts,
debug);
processTrees("Code Conversion Annotations Prep",
"annotations/annotations_prep",
codenames,
false,
asts,
debug);
processTrees("Code Conversion Annotations",
"annotations/annotations",
codenames,
false,
asts,
debug);
generateWsdls(debug);
generateFrames();
generateMenus();
processTrees("Business Logic Base Structure",
"convert/base_structure",
codenames,
false,
asts,
debug);
generateProxyPrograms();
processTrees("Core Code Conversion",
"convert/core_conversion",
codenames,
false,
asts,
debug);
// ensure that downstream processing has the proper list of files
convertSourceNamesToJavaAstNames();
brew(jasts, "Business Logic and Frames Classes");
}
/**
* Gather all SOAP operations from all files, regardless if they are included in the incremental conversion
* or not, and generate the WSDLs.
*
* @param debug
* The debug level.
*
* @throws ConfigurationException If there is problem reading/parsing the configuration.
* @throws AstException If there is an problem processing ASTs.
*/
static void generateWsdls(int debug)
throws AstException,
ConfigurationException
{
List<String> files = ConversionData.listWithServicesFiles();
String[] astFiles = new String[files.size()];
for (int i = 0; i < files.size(); i++)
{
astFiles[i] = files.get(i) + ".ast";
}
FileList asts = new ExplicitFileList(astFiles);
processTrees("Generate WSDL files",
"convert/soap_services",
codenames,
false,
asts,
debug);
}
/**
* Generate the Java code for legacy programs which are exposed via the OpenClient replacement.
* <p>
* A 4GL legacy program may be configured with one or more (Sub)AppObject. For this reason, the processing
* is done in 'batches', where a program will appear in a single (Sub)AppObject configuration, and this
* will either create or update the JAST for that (Sub)AppObject.
*
* @throws ConfigurationException If there is problem reading/parsing the configuration.
* @throws AstException If there is an problem processing ASTs.
*/
protected void generateProxyPrograms()
throws AstException,
ConfigurationException
{
List<LegacyProxyConfig> proxyConfigs = ServiceSupport.getProxyConfigs();
if (proxyConfigs == null)
{
return;
}
NameConverter nc = new NameConverter();
boolean legacyNames = Configuration.getParameter("proxy-legacy-names", false);
java.util.function.Function<LegacyProxyConfig, String> getAppObjectKey = (cfg) ->
{
String javaName = legacyNames ? cfg.getAppObject()
: nc.convert(cfg.getAppObject(), MatchPhraseConstants.TYPE_CLASS);
return cfg.getPkg() + "." + javaName;
};
Set<String> allCvtFiles = new HashSet<>();
for (String file : asts.listFilenames())
{
file = FileOperationsWorker.findFile(file, null);
allCvtFiles.add(file);
}
allCvtFiles.addAll(Arrays.asList(asts.listFilenames()));
// get the proxy config for each file
Map<String, Integer> programCounter = new HashMap<>();
Map<String, List<LegacyProxyConfig>> proxyFileSets = new HashMap<>();
for (LegacyProxyConfig cfg : proxyConfigs)
{
Map<String, LegacyProxyClientConfig> proxyFiles = cfg.getProxyPrograms();
for (Map.Entry<String, LegacyProxyClientConfig> entry : proxyFiles.entrySet())
{
String file = entry.getKey();
LegacyProxyClientConfig lpcc = entry.getValue();
String filename = lpcc.getFullName() + "." + lpcc.getProgExt();
String cvtFile = FileOperationsWorker.findFile(filename, null) + ".ast";
if (!allCvtFiles.contains(cvtFile))
{
continue;
}
List<LegacyProxyConfig> proxyCfgs = proxyFileSets.computeIfAbsent(file, (k) -> new ArrayList<>());
proxyCfgs.add(cfg);
}
programCounter.compute(getAppObjectKey.apply(cfg),
(appObj, counter) -> (counter == null
? proxyFiles.size()
: counter + proxyFiles.size()));
}
// when there are multiple configurations for a file, create a LegacyProxyConfig with just that file
// and add it explicitly to its list, while removing the file from the 'main' configuration
for (Map.Entry<String, List<LegacyProxyConfig>> entry : proxyFileSets.entrySet())
{
String file = entry.getKey();
List<LegacyProxyConfig> proxyCfgs = entry.getValue();
if (proxyCfgs.size() <= 1)
{
continue;
}
List<LegacyProxyConfig> fileCfgs = new ArrayList<>();
entry.setValue(fileCfgs);
for (LegacyProxyConfig cfg : proxyCfgs)
{
LegacyProxyClientConfig progCfg = cfg.removeProxyProgram(file);
LegacyProxyConfig dup = new LegacyProxyConfig(cfg);
dup.addProxyProgram(progCfg);
fileCfgs.add(dup);
}
}
// create the lists of LegacyProxyConfig instances, so that a file is not processed twice in a list
List<Set<LegacyProxyConfig>> proxyUniqueConfigs = new ArrayList<>();
while (!proxyFileSets.isEmpty())
{
Set<LegacyProxyConfig> uniqueConfigs = Collections.newSetFromMap(new IdentityHashMap<>());
proxyUniqueConfigs.add(uniqueConfigs);
Iterator<String> iter = proxyFileSets.keySet().iterator();
while (iter.hasNext())
{
String file = iter.next();
List<LegacyProxyConfig> fileCfgs = proxyFileSets.get(file);
LegacyProxyConfig cfg = fileCfgs.remove(0);
uniqueConfigs.add(cfg);
if (fileCfgs.isEmpty())
{
iter.remove();
}
}
}
// process all configurations in batches
int pass = 0;
int size = proxyUniqueConfigs.size();
for (Set<LegacyProxyConfig> uniqueConfigs : proxyUniqueConfigs)
{
pass = pass + 1;
ServiceSupport.setCurrentProxyConfigs(new ArrayList<>(uniqueConfigs));
List<String> proxyFiles = new ArrayList<>();
Set<String> doneAppObjects = new HashSet<>();
for (LegacyProxyConfig cfg : uniqueConfigs)
{
List<String> files = cfg.getProxyProgramFiles();
proxyFiles.addAll(files);
String appObjectKey = getAppObjectKey.apply(cfg);
int remainder = programCounter.compute(appObjectKey,
(appObj, counter) -> counter - files.size());
if (remainder == 0)
{
doneAppObjects.add(appObjectKey);
}
else if (remainder < 0)
{
throw new RuntimeException("Unbalanced " + appObjectKey);
}
}
if (!proxyFiles.isEmpty())
{
ExplicitFileList proxyAsts = new ExplicitFileList(Configuration.isCaseSensitive());
for (String proxyFile : proxyFiles)
{
String astFile = FileOperationsWorker.findFile(proxyFile, null);
if (astFile == null)
{
System.out.println("WARNING: Could not resolve program for proxy file: " + proxyFile);
continue;
}
proxyAsts.add(astFile + ".ast");
}
processTrees("Annotate Proxy Client Programs - pass #" + pass + " of " + size,
"annotations/proxy_programs",
codenames,
false,
proxyAsts,
debug);
processTrees("Generate Proxy Client Programs - pass #" + pass + " of " + size,
"convert/proxy_programs",
codenames,
false,
proxyAsts,
debug);
FileList files = ServiceSupport.getProxyClientPrograms(doneAppObjects);
if (files.size() > 0)
{
ServiceSupport.setBrewProxy(true);
try
{
brew(files, "for Proxy Client Programs - pass #" + pass + " of " + size);
}
finally
{
ServiceSupport.setBrewProxy(false);
}
}
}
}
}
/**
* Drives the execution of the frame generator rule-set using the Progress
* ASTs as input and capturing the list of Java AST files generated. These
* generated files are all added to the list of Java AST files to be
* processed downstream.
*
* @throws ConfigurationException If there is problem reading/parsing the configuration.
* @throws AstException If there is an problem processing ASTs.
*/
private void generateFrames()
throws AstException,
ConfigurationException
{
// TODO: change to a more adequate naming
processTrees("Detecting Frame Interfaces",
"annotations/frame_generator_pre",
null,
false,
asts,
debug);
// enable new file tracking mode
JavaPatternWorker.enableNewFileTracking();
processTrees("Frame Generator",
"convert/frame_generator",
codenames,
false,
asts,
debug);
// add generated JASTs to the overall list (used with brew later)
JavaPatternWorker.disableNewFileTracking();
frames = JavaPatternWorker.getPersistedFiles();
}
/**
* For each .p2o file associated with a schema and for each specified dialect, generate the
* specified type of DDL.
* <p>
* This is done by calling the ruleset specified and setting the <code>dbName</code> and
* <code>targetDb</code> parameters accordingly.
*
* @param ddlType
* Type of DDL being generated (used only for console message).
* @param schemas
* List of names of schemas to process.
* @param dialectList
* List of database dialects for which to generate DDL or <code>null</code> if this
* information is to be read from the configuration.
* @param ruleset
* Name of ruleset to execute (without file extension suffix).
*
* @throws ConfigurationException
* if the schema configuration can't be loaded.
*/
private void generateDDL(String ddlType,
List<String> schemas,
String dialectList,
String ruleset)
throws ConfigurationException
{
SchemaConfig config = Configuration.getSchemaConfig();
// for each .p2o file and for each read dialect, generate the DDL
Map<String, String> vars = new HashMap<>();
for (String schema: schemas)
{
String dialects_ = dialectList;
String p2oName = config.getSchemaResource(schema, ".p2o");
ExplicitFileList files = new ExplicitFileList(new String[] { p2oName });
vars.put("dbName", "\"" + schema + "\"") ;
// for each dialect
if (dialects_ == null)
{
dialects_ = config.getNamespaceParameter(schema, "ddl-dialects");
}
if (dialects_ == null)
{
System.out.println("WARNING: no DDL dialect specified for schema " + schema);
continue;
}
String[] dialects = dialects_.split(",");
for (String dialect : dialects)
{
dialect = dialect.trim().toLowerCase();
vars.put("targetDb", "\"" + dialect + "\"");
// generate the DDL
processTrees("Generate " + ddlType +
" DDL scripts for schema '" + schema +
"' and target DB '" + dialect + "'",
ruleset,
datanames,
true,
files,
debug,
vars);
}
}
}
/**
* Drives the execution of the DMO creation rule-set using the P2O lists
* as input and capturing the list of Java AST files generated.
*
* @param skipDBSchema
* <code>true</code> if the physical DB schemas are not processed.
*
* @return The list of JAST files to be "brewed".
*
* @throws ConfigurationException If there is problem reading/parsing the configuration.
* @throws AstException If there is an problem processing ASTs.
*/
private FileList generateDMOs(boolean skipDBSchema)
throws AstException,
ConfigurationException
{
// enable new file tracking mode
JavaPatternWorker.enableNewFileTracking();
if (!skipDBSchema)
{
processTrees("Generate Data Model Objects (database DMOs)",
"schema/java_dmo",
datanames,
true,
dp2os,
debug);
}
if (sp2os.size() > 0)
{
processTrees("Generate Data Model Objects (temp-table DMOs)",
"schema/java_dmo",
datanames,
true,
sp2os,
debug);
}
// add generated JASTs to the overall list (used with brew later)
JavaPatternWorker.disableNewFileTracking();
String[] dmoList = JavaPatternWorker.getPersistedFiles();
return new ExplicitFileList(dmoList);
}
/**
* Drives the execution of the POJO creation rule-set using the P2O lists as input and capturing the list
* of Java AST files generated.
*
* @param skipDBSchema
* {@code true} if the physical DB schemas are not processed.
*
* @return The list of JAST files to be "brewed" to POJO classes.
*
* @throws ConfigurationException
* If there is problem reading/parsing the configuration.
* @throws AstException
* If there is an problem processing ASTs.
*/
private FileList generatePOJOs(boolean skipDBSchema)
throws AstException,
ConfigurationException
{
JavaPatternWorker.enableNewFileTracking(); // enable new file tracking mode
if (!skipDBSchema)
{
processTrees("Generate Plain Old Java Objects (database DMOs)",
"schema/java_pojo",
datanames,
true,
dp2os,
debug);
}
if (sp2os.size() > 0)
{
processTrees("Generate Plain Old Java Objects (temp-table POJOs)",
"schema/java_pojo",
datanames,
true,
sp2os,
debug);
}
// add generated JASTs to the overall list (used with brew later)
JavaPatternWorker.disableNewFileTracking();
return new ExplicitFileList(JavaPatternWorker.getPersistedFiles());
}
/**
* Drives the execution of the frame generator rule-set using the Progress
* ASTs as input and capturing the list of Java AST files generated. These
* generated files are all added to the list of Java AST files to be
* processed downstream.
*
* @throws ConfigurationException
* If there is problem reading/parsing the configuration.
* @throws AstException
* If there is an problem processing ASTs.
*/
private void generateMenus()
throws AstException,
ConfigurationException
{
// enable new file tracking mode
JavaPatternWorker.enableNewFileTracking();
processTrees("Menu Generator",
"convert/menu_generator",
codenames,
false,
asts,
debug);
// add generated JASTs to the overall list (used with brew later)
JavaPatternWorker.disableNewFileTracking();
menus = JavaPatternWorker.getPersistedFiles();
}
/**
* Convert Java AST (JAST) files into Java source code (<code>.java</code>
* files). The output is written into the configured project's output
* path.
*
* @param files
* The list of JAST files to process.
* @param extra
* Text to append to the displayed header.
*
* @throws ConfigurationException If there is problem reading/parsing the configuration.
* @throws AstException If there is an problem processing ASTs.
*/
private void brew(FileList files, String extra)
throws ConfigurationException,
AstException
{
processTrees("Generate Java Source " + extra,
"convert/brew",
null,
false,
files,
debug);
}
/**
* Using the source name list as input, this method creates duplicate
* lists with the proper <code>.ast</code> or <code>.jast</code> file
* suffixes on each entry. Appended to this list is the list of Java ASTs
* created during frame generation. The results are saved into the
* Java AST name list.
*/
private void convertSourceNamesToJavaAstNames()
{
List<String> oldjasts = new ArrayList<>();
// merge the lists
oldjasts.addAll(Arrays.asList(source.withSuffix(".jast")));
oldjasts.addAll(Arrays.asList(frames));
oldjasts.addAll(Arrays.asList(menus));
// save the results
jasts = new ExplicitFileList(oldjasts.toArray(new String[0]));
}
/**
* Container class that aggregates a set of options to control a run of the conversion
* driver.
*/
static class CvtRunMode
extends RunMode
{
/** Run the length statistics during the middle. */
private final boolean length;
/** When <code>true</code>, the physical DB schemas are not processed.*/
private final boolean skipDBSchema;
/** Flag indicating if the brew of schema DMOs will be omitted. */
private final boolean omitSchemaBrew;
/**
* Builds an instance and initializes all data.
*
* @param front
* If the front phase should be executed.
* @param preproc
* Force the preprocessor to run.
* @param fschema
* Force the schema loading/fixups to run.
* @param callgraph
* Force the callgraph generation to run.
* @param middle
* If the middle phase should be executed.
* @param length
* If the schema length statistics should be calculated.
* @param skipDBSchema
* <code>true</code> if the physical DB schemas are not processed.
* @param back
* If the code back end phase should be executed.
* @param omitSchemaBrew
* Do not brew any Java ASTs for the schema part.
*/
CvtRunMode(boolean front,
boolean preproc,
boolean fschema,
boolean callgraph,
boolean middle,
boolean length,
boolean skipDBSchema,
boolean back,
boolean omitSchemaBrew)
{
super(front, preproc, fschema, callgraph, middle, back);
this.length = length;
this.skipDBSchema = skipDBSchema;
this.omitSchemaBrew = omitSchemaBrew;
}
/**
* Logically OR the given mode flags with the current mode flags and
* return an instance that has all flags active that are active in
* either of the 2 original input modes. No changes are made to the
* original instances.
*
* @param run
* The mode to logically OR with this instance's data.
*
* @return The logically OR'd result.
*/
public RunMode merge(RunMode run)
{
RunMode rm = super.merge(run);
if (!(run instanceof CvtRunMode))
{
return run;
}
CvtRunMode cvt = (CvtRunMode) run;
return new CvtRunMode(rm.front,
rm.preproc,
rm.schema,
rm.callgraph,
rm.middle,
(this.length || cvt.length),
(this.skipDBSchema || cvt.skipDBSchema),
rm.back,
this.omitSchemaBrew || cvt.omitSchemaBrew);
}
}
/**
* Provides a command line interface for an end user to drive the scanning harness.
* <p>
* The program will return a non-zero code if there is a fatal error running the conversion.
* <p>
* Syntax:
* <pre>{@code java -DP2J_HOME=<home> ConversionDriver [-options]* <mode> <file-specs>}</pre>
* Where:
* <ul>
* <li>Options (F, S, X and Z are mutually exclusive. If none is specified <file-specs> is a list of files
* added to command line):
* <ul>
* <li>Dn = set debug level 'n' (must be a numeric digit between {@code MSG_NONE} (0) and
* {@code MSG_TRACE} (3) inclusive
* <li>I = incremental conversion; only changed/new programs will be parsed and converted.
* <li>F = {@code <file-specs>} is a {@code <filename>} containing a custom FILE-LIST, instead of
* using the explicit command line file list (deprecated)
* <li>S = {@code <file-specs>} is composed of a root {@code <directory>} and a {@code
* "<file-filter>"}, to select SPECIFIC files instead of an explicit command line file list
* <li>X = {@code <file-specs>} is similar to S but has an additional custom EXCLUDE list file
* {@code "<filename>"} to be ignored (deprecated)
* <li>N = NO recursion in {@code <file-filter>} (S options)",
* <li>Z = {@code <file-set>} is a {@code <filename>} containing a list of directives which define a file
* set to process; these directives provide a superset of the capabilities of F, S and X options
* <li>A = ABORT scan part of front end on an error
* <li>Tn = specify number of THREADS 'n' to use for front end file scan ('n' must be {@code >=}
* 0; defaults to 1; set to 0 to use 1 thread per available CPU)
* WARNING: this mode is not compatible if there are legacy .cls files
* <li>L = LIST files to be processed only; do not perform any further task
* <li>Pname = the {@code <name>} of the configuration PROFILE follows the option
* </ul>
* <li>{@code mode} (one or more of the following values, use '+' to delimit if there are two or more
* values; don't insert spaces!):
* <ul>
* <li>Front End
* <ul>
* <li>F0 = preprocessor (in honor cache mode), lexer, parser, AST persistence and
* post-parse fixups
* <li>F1 = F0 + forced preprocessor execution
* <li>F2 = F1 + schema loader/fixups
* <li>F3 = F2 + call graph generation
* </ul>
* <li>Middle
* <ul>
* <li>M0 = schema annotations, P2O generation, DMO generation, brew of DMO classes
* <li>M1 = M1 + length statistics
* </ul>
* <li>Code Back End
* <ul>
* <li>CB = unreachable code, annotations, frame generator, base structure, core conversion
* and brew
* </ul>
* </ul>
* <li>explicit command line list (PLAIN, default approach if none of S, P, or F are present)
* <ul>
* <li>{@code <filelist>} is arbitrary list of absolute and/or relative file names to scan, this
* list is hard coded in the command line itself
* <li>any number of files may be listed on the command line, subject to command shell limits
* </ul>
* <li>custom file list (FILE approach)
* <ul>
* <li>{@code <filename>} is a single absolute or relative filename of a text file that contains a
* custom list of absolute and/or relative file names of files to scan
* <li>in other words, the file list will be read from a file instead of being hard coded on the
* command line
* <li>there must be one filename per line in the file
* <li>there is no limit of the number of files in the list
* </ul>
* <li>file-spec mode (expressly SPECIFIED files approach)
* <ul>
* <li>{@code <directory>} is the root directory to search for files
* <li>{@code "filespec"} is the file specification that will be used to create a list of files to
* process, should be enclosed in double quotes if any of the wildcard characters '*' or '?'
* are used
* </ul>
* <li>custom ignore list (file-spec mode with additional EXCLUDE list)
* <ul>
* <li>{@code <directory>} is the root directory to search for files
* <li>{@code "filespec"}" is the file specification that will be used to create a list of files
* to process, should be enclosed in double quotes if any of the wildcard characters '*' or
* '?' are used
* <li>{@code <filename>} is a single absolute or relative filename of a text file that contains a
* custom list of specifications of file names to ignore
* <li>asterisk (*) and question mark (?) wildcards are permitted to represent zero or more and
* exactly one wildcard characters, respectively
* <li>files specified by the primary {@code filespec} which are not matched by the ignore list
* are scanned as they would be in {@code filespec} mode.
* </ul>
* <li>file set (list of directives in a text file)
* <ul>
* <li>{@code <filename>} is a single absolute or relative filename of a text file that contains a
* list of directives that define the file set (the list of files to be processed)
* <li>These directives are available, one directive per line:
* <ul>
* <li>{@code D <recursive_top_level_dir> <file_spec>} adds a top level directory and a file
* spec which will be recursively processed to add 0 or more files into the list
* <li>{@code N <non_recursive_top_level_dir> <file_spec>} adds a top level directory and file
* spec which will be non-recursively processed to add 0 or more files into the list
* <li>{@code F <absolute_or_relative_file_name>} explicitly adds a specific file into the list
* <li>{@code X <exclusion_filter_expression_including_wildcards>} filter to remove 0 or more files from the list
* <li>{@code I <inclusion_filter_expression_including_wildcards>} filter to add something removed back into the list
* </ul>
* </ul>
* </ul>
*
* @param args
* List of command line arguments.
*/
public static void main(String[] args)
{
JobDefinition job = processCommandLine(args, modes, helpText);
ErrorManager.setHeadless();
if (job != null)
{
ConversionDriver driver = new ConversionDriver(job);
driver.executeJob(TITLE);
}
}
}