AstGenerator.java
/*
** Module : AstGenerator.java
** Abstract : Creates an AST from a single source file using Progress preprocessor and parser tools
**
** Copyright (c) 2005-2025, Golden Code Development Corporation.
**
** -#- -I- --Date-- --JPRM-- -----------------------------------Description-----------------------------------
** 001 ECF 20050121 @19369 Created initial version.
** 002 GES 20050126 @19446 Added support for caching preprocessor
** output. In input, will bypass preprocessing
** (even if requested) if there is a
** pre-existing cache file that has a later
** timestamp than the input file.
** 003 GES 20050204 @19569 Added support for hints processing to load
** the list of non-default databases.
** 004 GES 20050207 @19625 Added preprocessor runtime arguments support
** with input from the hints file.
** 005 GES 20050210 @19747 Added variable, widget, frame and alias
** hints support.
** 006 GES 20050218 @19861 Modified creation of the FileScope object
** to match new features that properly
** implement runtime arguments. This new
** approach also simplifies this code greatly.
** Also added code to check if the parser
** reached EOF or exited prematurely. In the
** case where an early exit is detected, an
** diagnostic error message is output.
** 007 ECF 20050118 @19890 Changed type of AST this class generates. We
** now create instances of AnnotatedAst instead
** of UnifiedAST. The latter has been obsoleted
** and removed from the project.
** 008 GES 20050122 @19904 Added fixup processing for the AST, after it
** is built. This allows parents and IDs to be
** assigned to the tree before it is ever used.
** Also added basic lexer dumping and greater
** external control over state variables.
** 009 GES 20050124 @19948 Major rework to rationalize the constructors,
** option processing and most importantly to
** add persistence support. ASTs can be
** persistently cached and loaded transparently
** using this class.
** 010 ECF 20050225 @19959 Made AST_POSTFIX constant public. This is
** needed to provide user functions the ability
** to persist ASTs using a consistent filename
** extension.
** 011 ECF 20050225 @20060 Moved a modified version of findFile method
** to the common, Utils class. We now call that
** method (findFileInPath) from here.
** 012 GES 20050303 @20165 Minor change to match the changed DumpTree
** interface.
** 013 GES 20050307 @20198 Made all POSTFIX constants public so that
** external users don't have to hard-code the
** same strings.
** 014 GES 20050323 @20479 Converted to use ProgressAst instead of
** AnnotatedAst which is now abstract.
** 015 GES 20050328 @20508 Moved AST class processing into parser to
** centralize this function and make clients
** simpler and more independent.
** 016 ECF 20050331 @20614 If in AST persistence mode, also persist a
** *.p.schema file for each source file. This
** contains temp-table schema information.
** 017 GES 20050411 @20665 New SymbolResolver constructor signature.
** 018 ECF 20050411 @20699 Modified to conform with new SymbolResolver
** and Preprocessor changes. Removed redundant
** argument from call to symbol resolver's
** persistSchemaData method. Catch new exception
** thrown by Preprocessor constructor.
** 019 ECF 20050419 @20853 Modified to minimize having to delete the AST
** registry. Previously, re-generating an AST
** which was registered meant deleting the
** registry and re-scanning completely. Now, we
** reset the next node ID in the registry for
** each AST just before applying parent and ID
** fixups. This allows AST file IDs to be
** reused without making the entire registry
** stale. Note: any existing references to the
** old IDs WILL be stale, however.
** 020 NVS 20050516 @21179 New Preprocessor option is enabled - produce
** the hints file. The file is generated with
** the base file name and ".pphints" postfix.
** 021 ECF 20060202 @24255 Store temp/work-table schema information
** using ".dict" extension. Previous ".schema"
** extension now is used later, when schema
** fixups are applied.
** 022 GES 20070313 @32396 Support global dynamic propaths and file
** specific propath hints.
** 023 GES 20070403 @32714 Support hint control over escape sequence
** processing in the preprocessor.
** 024 GES 20070423 @33209 Added preprocessor marker hint support.
** 025 GES 20070629 @34344 Minor interface change.
** 026 GES 20070816 @34889 Added preprocessor hint support for built-in
** variables BATCH-MODE, OPSYS and
** WINDOW-SYSTEM.
** 027 GES 20070830 @34948 Added WebSpeed/Blue Diamond preprocessing
** support. This takes any input file that is
** HTML, and converts it into 4GL source code.
** The resulting source code is then run through
** the preprocessor, lexer and parser just as
** if it had been static 4GL source.
** 028 GES 20070913 @35130 Made this class safe to use reentrantly.
** Also added file-level locking to ensure
** process-wide exclusive access.
** 029 GES 20070923 @35183 Added keyword ignore support to the lexer
** init which is equivalent to the Progress -k
** startup parameter (honors a keyword "forget"
** file).
** 030 GES 20070926 @35241 Pass the escape processing mode to the lexer
** to allow it to honor or ignore backslash
** escapes.
** 031 GES 20080318 @37661 Changed approach to read characters (the
** entire unicode charset) instead of dealing
** with bytes. This makes it safe to process
** files with I18N requirements.
** 032 GES 20090427 @42023 Import change.
** 033 GES 20090429 @42056 Match package and class name changes.
** 034 GES 20090515 @42226 Moved to AstManager from AstRegistry/AstPersister.
** 035 GES 20090518 @42404 Fixup for usage of brainwash().
** 036 GES 20100922 Normalize the filename at the top of processFile()
** so that lockTree() and brainwash() both use the
** same input string. This eliminates duplicate
** registry entries for the same file. Added
** parameter to differentiate the behavior of
** brainwash() where 4GL code trees now will save off
** comments and other hidden tokens as shadow nodes.
** 037 ECF 20101001 Added logic to query case-sensitivity parameter
** from Configuration and disable it in preprocessor
** options if set to 'false'. Use file and path
** separators appropriate to original platform of the
** project.
** 038 GES 20110712 Now the code passes the case-sensitivity flag to
** the symbol resolver (to enable case-insensitive
** file-system searches).
** 039 GES 20110822 Added a pre-scanning pass for parsing class files
** which is needed to allow class resources to be
** accessed out-of-order (earlier than their defs
** appear in the file).
** 040 GES 20111203 Fixed regression in an printf error message spec.
** Reworked propath processing code to be static and
** to be exposed publicly so it can be reused.
** 041 OM 20130607 Fixed imports that conflicted with Hibernate namespace.
** 042 GES 20140313 Safety code so that input file failures get handled gracefully.
** 043 ECF 20150715 Replace StringBuffer with StringBuilder.
** 044 GES 20160225 Honor OO skeleton path override.
** 045 IAS 20160331 Add input streams' close.
** 046 ECF 20160302 Made processFile and removeDuplicate threadsafe.
** 047 GES 20160225 Add OO class resource cleanup.
** 048 HC 20160802 Fixed handling of file separators when including files during
** preprocessing.
** 049 OM 20170123 Paths returned by Configuration already use the current OS
** separators.
** 050 GES 20170303 Added support for configuring the preprocessor debug flag.
** 051 GES 20170708 Added a silent flag.
** HC 20170708 Added source-charset uast preprocessor hint.
** Fixed loading of preprocessed file cache (*.cache) when the file
** included non-ascii (multibyte) characters.
** CA 20170711 1st level OO class scanning is decoupled from full parse: first, all
** references from members or inheritance are resolved, for the entire
** graph, and only after that the reached classes are parsed.
** HC 20170726 Made AstGenerator to report failure when ProgressParser encounters
** errors.
** 052 GES 20180226 Improve error handling by ensuring that the parser knows the
** filename being processed.
** CA 20180321 Added support for PROCESS-ARCHITECTURE preprocessor var.
** 053 CA 20181121 preScanClass must inherit the PROPATH from the caller.
** 054 HC 20190227 Fixed text encoding for the preprocessed cache file.
** 055 CA 20190513 Fixed buffer usage from super-classes, in OO.
** 056 ECF 20190619 Added support for include file reference prefix remapping.
** 057 GES 20190625 Removed previous .NET dependency tracking approach.
** RFB 20190911 Schema processing may occur which references include files and
** preprocessor directives. However, the hints file was not being
** honored, since it was not accessible from progress.g's parser.
** Added a getHints and storage method for this.
** 058 CA 20200412 Added incremental conversion support.
** 059 CA 20200428 Process enums for incremental conversion support.
** GES 20200522 Fix a dependency problem with enum parsing by pre-loading the most important
** built-in 4GL OO classes. This only happens just before the first time a class
** is parsed.
** OM 20210913 Added flags capability to AST registry.
** OM 20211122 Added hints-based support for 'unloading' schema.
** OM 20220405 XmlFilePlugin can load resources from application's jar.
** GES 20220517 Added getAstName().
** CA 20220727 Improved memory management for parsing; cleanup is done in two phases:
** 1. after each legacy class file has finished parsing, the SchemaDictionary will
** keep only protected temp-tables (all private tables are removed, as these
** can't be reached from sub-classes; all permanent tables are removed, as each
** class has its own reference to the permanent tables). Also, the class def
** instance will reduce its own used memory, which is not required when parsing
** sub-classes.
** 2. after parsing of the entire file set is finished, any SchemaDictionary or
** other ASTs referenced by the ClassDefinition are released.
** 060 GBB 20230512 Logging methods replaced by CentralLogger/ConversionStatus.
** 061 OM 20230115 Replaced absolutePath(), relativePath(), upPath() and downPath() with faster
** versions, based on node types.
** 062 GES 20241112 Rewrite the tree locking data structures for HTML filenames.
** 063 CA 20241026 Fixed an issue with circular references when parsing classes: if a parent is
** already 'pre-scanned', avoid parsing the class and let it parse when it's reached
** again.
** CA 20241104 Pre-scan must use the banned schemas (from the hints).
** CA 20241118 Mark already parsed .cls file's ASTs as 'original' in the registry.
** 064 CA 20250422 If there are duplicated PROPATH entries, leave only the first occurrence, as this
** will result two matches found when automatically adding .cls depdendencies at
** conversion.
*/
/*
** This program is free software: you can redistribute it and/or modify
** it under the terms of the GNU Affero General Public License as
** published by the Free Software Foundation, either version 3 of the
** License, or (at your option) any later version.
**
** This program is distributed in the hope that it will be useful,
** but WITHOUT ANY WARRANTY; without even the implied warranty of
** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
** GNU Affero General Public License for more details.
**
** You may find a copy of the GNU Affero GPL version 3 at the following
** location: https://www.gnu.org/licenses/agpl-3.0.en.html
**
** Additional terms under GNU Affero GPL version 3 section 7:
**
** Under Section 7 of the GNU Affero GPL version 3, the following additional
** terms apply to the works covered under the License. These additional terms
** are non-permissive additional terms allowed under Section 7 of the GNU
** Affero GPL version 3 and may not be removed by you.
**
** 0. Attribution Requirement.
**
** You must preserve all legal notices or author attributions in the covered
** work or Appropriate Legal Notices displayed by works containing the covered
** work. You may not remove from the covered work any author or developer
** credit already included within the covered work.
**
** 1. No License To Use Trademarks.
**
** This license does not grant any license or rights to use the trademarks
** Golden Code, FWD, any Golden Code or FWD logo, or any other trademarks
** of Golden Code Development Corporation. You are not authorized to use the
** name Golden Code, FWD, or the names of any author or contributor, for
** publicity purposes without written authorization.
**
** 2. No Misrepresentation of Affiliation.
**
** You may not represent yourself as Golden Code Development Corporation or FWD.
**
** You may not represent yourself for publicity purposes as associated with
** Golden Code Development Corporation, FWD, or any author or contributor to
** the covered work, without written authorization.
**
** 3. No Misrepresentation of Source or Origin.
**
** You may not represent the covered work as solely your work. All modified
** versions of the covered work must be marked in a reasonable way to make it
** clear that the modified work is not originating from Golden Code Development
** Corporation or FWD. All modified versions must contain the notices of
** attribution required in this license.
*/
package com.goldencode.p2j.uast;
import java.io.*;
import java.nio.charset.*;
import java.util.*;
import java.util.logging.*;
import antlr.*;
import com.goldencode.ast.*;
import com.goldencode.p2j.cfg.*;
import com.goldencode.p2j.convert.*;
import com.goldencode.p2j.convert.db.*;
import com.goldencode.p2j.util.Utils;
import com.goldencode.p2j.e4gl.*;
import com.goldencode.p2j.preproc.*;
import com.goldencode.p2j.preproc.Options;
/**
* Returns an AST for a specified Progress source file by creating this AST
* (using the preprocessor and parser) or by loading an already existing
* persisted AST from a file.
* <p>
* Optionally preprocesses, then parses a single Progress source file into an
* abstract syntax tree (AST). Optionally accepts a <code>PROPATH</code>
* setting which is used to allow the preprocessor to locate relative source
* file references (includes). If this is not provided, the current directory
* is assumed to be the top level directory for all filename resolution.
* <p>
* Caching of the preprocessor results can be optionally enabled. In this
* case the output will be stored in a file of the same base name as the
* original with an appended '.cache'. For example, the preprocessor output
* for utilities/top-menu.p would be stored in utilities/top-menu.p.cache.
* <p>
* If the given input file is HTML, it is converted into 4GL source code by
* the embedded 4GL (E4GL) preprocessor. The resulting source code is then
* run through the preprocessor, lexer and parser just as if it had been
* static 4GL source. Note that the same flags that control the running and
* caching of the 4GL preprocessor, control the running and caching of the
* embedded 4GL (E4GL) preprocessor. Note that the only difference is that
* the output of the E4GL preprocessor is always written to file. See
* {@link #processFile} for more details on this support.
* <p>
* For debugging and regression testing purposes, it is possible to save
* the output of the lexer and/or the parser in a more human readable format.
* The {@link #setDumpLexer} and {@link #setDumpParser} methods provide
* this capability. Note that the resulting files will be generated with the
* original source filename and an appended '.lexer' and '.parser'
* respectively. The existence of these files does not alter any subsequent
* processing and subsequent runs will replace these files with no warning.
* <p>
* An additional option is provided to enable AST persistence. If this is
* enabled, it changes the processing in 2 ways. First, if a valid persisted
* AST already exists, the AST returned by this class will be loaded from
* that file AND the preprocessor/parser will not be invoked. Second, in the
* case that no persisted AST exists yet OR the persisted AST is not later in
* timestamp than the source file, the output from the parser will be
* persisted into a file with the original source filename and an appended
* '.ast'.
* <p>
* By default, preprocessing, preprocessor output caching and AST persistance
* are all enabled BUT lexer and parser output is disabled.
*/
public class AstGenerator
implements HintsConstants,
E4GLConstants
{
/**
* Text to append to a filename to generate a unique name for persisting
* an AST.
*/
public static final String AST_POSTFIX = ".ast";
/** Text to append to a filename to generate a unique name for caching. */
public static final String CACHE_POSTFIX = ".cache";
/** Text to append to a filename to generate a unique name for hints. */
public static final String HINTS_POSTFIX = ".pphints";
/**
* Text to append to a filename to generate a unique name for saving
* human-readable lexer output.
*/
public static final String LEXER_POSTFIX = ".lexer";
/**
* Text to append to a filename to generate a unique name for saving
* human-readable parser output.
*/
public static final String PARSER_POSTFIX = ".parser";
/**
* Text to append to a filename to generate a unique name for persisting
* schema dictionary information.
*/
public static final String DICT_POSTFIX = ".dict";
/** The flag for marking a DICT schema tree in AST registry. */
public static final int FLAG_DICT = AstManager.FLAG_USER;
/** A filename extention to recognize as an HTML file. */
public static final String HTML1_POSTFIX = ".html";
/** A filename extention to recognize as an HTML file. */
public static final String HTML2_POSTFIX = ".htm";
/** Key for the file separator configuration parameter. */
private static final String FILESEP_KEY = "file-separator";
/** Key for the path separator configuration parameter. */
private static final String PATHSEP_KEY = "path-separator";
/** Key for the global propath configuration parameter. */
private static final String PROPATH_KEY = "propath";
/** Key to enable global directory prepending to the propath. */
private static final String DYN_PROPATH_PREPEND_KEY = "dynamic-prepend";
/** Key to enable global directory appending to the propath. */
private static final String DYN_PROPATH_APPEND_KEY = "dynamic-append";
/** Key for the keyword ignore configuration parameter. */
private static final String IGNORE_KEY = "keyword-ignore";
/** Key for the global, file system case sensitivity key. */
private static final String CASE_SENS_KEY = "case-sensitive";
/** Key for the error logfile. */
private static final String ERR_DEST_KEY = "error-dest";
/** Key for the warning logfile. */
private static final String WARN_DEST_KEY = "warn-dest";
/** Logger */
private static final ConversionStatus LOG = ConversionStatus.get(AstGenerator.class);
/** Global manager for file locking and file ID services. */
private static AstManager mgr = null;
/** Error destination. */
private static Writer err = new PrintWriter(System.err);
/** Warning destination. */
private static Writer warn = err;
/** Flag to track whether early builtin class initialization is complete. */
private static boolean earlyClassInit = false;
/** Flag indicating whether silent mode is required. */
private boolean silent = false;
/** Flag indicating whether inline preprocessing is needed. */
private boolean preprocess = true;
/** Indicates whether caching of preprocessor results should be done. */
private boolean cache = true;
/** Flag indicating whether dumping of lexer output is needed. */
private boolean dumpLexer = false;
/** Flag indicating whether dumping of parser output is needed. */
private boolean dumpParser = false;
/** Flag indicating whether storing of a persistent AST is needed. */
private boolean astPersist = true;
/** Array of propath path elements. */
private String[] paths = null;
/**
* Represents any non-default, file specific behavior that must be
* handled for the preprocessor or parser.
*/
private UastHints hints = null;
static
{
try
{
String name = Configuration.getParameter("registry");
mgr = AstManager.initialize(new XmlFilePlugin(name, Configuration.isImportMode()));
// honor the skeleton path override when the SR is being used for normal conversion
SymbolResolver.initSkeletonPath();
String errDest = Configuration.getParameter(ERR_DEST_KEY);
String warnDest = Configuration.getParameter(WARN_DEST_KEY);
try
{
if (errDest != null)
{
err = new BufferedWriter(new FileWriter(errDest));
warn = err;
}
if (warnDest != null)
{
warn = new BufferedWriter(new FileWriter(warnDest));
}
}
catch (Exception e)
{
// use the defaults
}
}
catch (AstException ae)
{
// TODO: log this? get ready for null pointer exceptions later
}
}
/**
* Default constructor which initializes the propath setting based on
* the project configuration.
*/
public AstGenerator()
{
}
/**
* Sort the given list to ensure that any HTML files are listed first.
* <p>
* Make sure all HTML files appear first since they get converted into 4GL
* source and these generated source file names may be separately
* duplicated in this list. By ensuring that the HTML files are listed
* first, the processing of the HTML files can also remove any duplicate
* 4GL source file names from the list, ensuring that no unnecessary
* processing is done. See {@link #processFile}.
*
* @param list
* The list to sort.
*
* @return The sorted list.
*/
public static File[] prioritize(File[] list)
{
ArrayList<File> html = new ArrayList<>();
ArrayList<File> src = new ArrayList<>();
ArrayList<File> full = new ArrayList<>();
// separate HTML and 4GL source files, leave the order otherwise
// the same
for (int i = 0; i < list.length; i++)
{
if (isHTML(list[i].getName()))
{
html.add(list[i]);
}
else
{
src.add(list[i]);
}
}
if (!html.isEmpty())
{
full.addAll(html);
full.addAll(src);
full.toArray(list);
}
return list;
}
/**
* Uses the filename extension to determine if this is an HTML file or
* not. See {@link #HTML1_POSTFIX} and {@link #HTML2_POSTFIX}.
*
* @param filename
* The filename to test.
*
* @return <code>true</code> if the filename ends with one of the
* known HTML extensions.
*/
public static boolean isHTML(String filename)
{
String lower = filename.toLowerCase();
return lower.endsWith(HTML1_POSTFIX) || lower.endsWith(HTML2_POSTFIX);
}
/**
* Common setup for preprocessor options which uses hints and global
* configuration as available.
*
* @param paths
* The propath;
* @param file
* The specific source file being processed. May be
* <code>null</code> if this preprocessor is not associated with
* a specific 4GL source file.
* @param hints
* Any 4GL source file hints that may exist. May be
* <code>null</code> if no hints exists or if there is no
* 4GL source file associated.
*
* @return The new options instance, suitably configured.
*/
public static Options buildOptions(String[] paths, File file, UastHints hints)
{
Options options = new Options();
options.setProPath(paths);
if (file != null)
{
File hintsFile = createUniqueFile(file.getAbsolutePath(), HINTS_POSTFIX);
options.setHintsFile(hintsFile.getAbsolutePath());
}
char marker = '\000';
boolean unixEscapes = true;
boolean batchMode = false;
boolean debug = false;
String opsys = null;
String winsys = null;
int arch = 32;
if (hints != null)
{
// honor file or directory level overrides
marker = hints.getMarker();
unixEscapes = hints.isUnixEscapes();
batchMode = hints.isBatchMode();
opsys = hints.getOpsys();
winsys = hints.getWindowSystem();
debug = hints.isPreprocDebug();
arch = hints.getProcessArchitecture();
}
else
{
// use global values
marker = UastHints.defaultMarker();
unixEscapes = UastHints.defaultUnixEscapes();
batchMode = UastHints.defaultBatch();
opsys = UastHints.defaultOpsys();
winsys = UastHints.defaultWinsys();
debug = UastHints.defaultPreprocDebug();
arch = UastHints.defaultArch();
}
options.setMarker(marker);
options.setUnixEscapes(unixEscapes);
options.setBatchMode(batchMode);
options.setDebug(debug);
options.setProcessArchitecture(arch);
if (opsys != null)
{
options.setOpsys(opsys);
}
if (winsys != null)
{
options.setWindowSystem(winsys);
}
String caseSensStr = Configuration.getParameter(CASE_SENS_KEY);
if ("false".equals(caseSensStr))
{
options.setCaseSensitive(false);
}
String fileSep = Configuration.getParameter(FILESEP_KEY, File.separator);
options.setFileSeparator(fileSep);
String pathSep = Configuration.getParameter(PATHSEP_KEY, File.pathSeparator);
options.setPathSeparator(pathSep);
options.setBasepath(Configuration.getParameter(ELEM_BASEPATH, "."));
Map<String, String> includeFilePrefixMap = Configuration.getIncludeFilePrefixMap();
options.setIncludeFilePrefixMap(includeFilePrefixMap);
return options;
}
/**
* Gets the PROPATH based on the global configuration and any file-specific configuration
* provided. Source-platform path separators will be honored during parsing of the paths.
*
* @param paths
* A single string with one or more paths to be searched for Progress files, with
* each path separated by the current platform's file separator character.
* May be {@code null}.
* @param hints
* File-specific hints or {@code null} if they don't exist or if there is no specific
* source file being processed.
*/
public static String[] getPropath(String paths, UastHints hints)
{
StringBuilder sb = new StringBuilder();
// honor the passed propath if given
if (paths != null)
{
sb.append(paths);
}
String replace = "";
String sourceDir = "";
int type = UastHints.PATH_MOD_NONE;
// lookup file-specific overrides
if (hints != null)
{
type = hints.getPathModType();
replace = hints.getPathModTxt();
sourceDir = hints.getSourceDir();
if (type == UastHints.PATH_MOD_REPLACE)
{
// override with a file specific propath
sb.setLength(0);
sb.append(replace);
}
}
// was there either a passed-in or file-specific propath?
if (sb.length() == 0)
{
// [propath] returned by Configuration already uses current platform file separator
String global = Configuration.getParameter(PROPATH_KEY);
if (global != null && global.length() > 0)
{
// use the global propath
sb.append(global);
}
else
{
// fallback plan (manufacture a default propath)
sb.append('.').append(File.pathSeparator);
}
}
// process file-specific prepend/append cases (the dynamic cases will already have replace
// set to be the same as sourceDir by the hints object)
switch (type)
{
case UastHints.PATH_MOD_PREPEND:
case UastHints.PATH_MOD_DYN_PREPEND:
prependPath(sb, replace);
break;
case UastHints.PATH_MOD_APPEND:
case UastHints.PATH_MOD_DYN_APPEND:
appendPath(sb, replace);
break;
}
// process global dynamic prepend/append cases
if (Configuration.getParameter(DYN_PROPATH_PREPEND_KEY) != null)
{
prependPath(sb, sourceDir);
}
else if (Configuration.getParameter(DYN_PROPATH_APPEND_KEY) != null)
{
appendPath(sb, sourceDir);
}
// no need to convert file separator chars to the current platform, [propath] returned by
// Configuration already uses the correct file separator
String[] res = sb.toString().split(File.pathSeparator);
// remove duplicated entries
LinkedHashSet<String> l = new LinkedHashSet<>(Arrays.asList(res));
res = l.toArray(new String[0]);
return res;
}
/**
* Gets the state of the silent mode flag indicating if the <code>System.out</code> output
* should be suppressed.
*
* @return <code>true</code> if silent mode is enabled, otherwise <code>false</code>.
*/
public synchronized boolean getSilent()
{
return silent;
}
/**
* Sets the state of the silent mode flag indicating if the <code>System.out</code> output
* should be suppressed.
*
* @param silent
* <code>true</code> if silent mode is enabled, otherwise <code>false</code>.
*/
public synchronized void setSilent(boolean silent)
{
this.silent = silent;
}
/**
* Sets the state of the flag indicating if the preprocessor should be
* run.
*
* @param preprocess
* <code>true</code> if the preprocessor is to be run, otherwise
* <code>false</code>.
*/
public synchronized void setPreprocess(boolean preprocess)
{
this.preprocess = preprocess;
}
/**
* Gets the state of the flag indicating if the preprocessor should be
* run.
*
* @return <code>true</code> if the preprocessor is to be run, otherwise
* <code>false</code>.
*/
public synchronized boolean getPreprocess()
{
return preprocess;
}
/**
* Sets the state of the flag indicating if the preprocessor output should
* be cached. This value is dependent upon whether or not the
* preprocessor is run, if the preprocessor is not run, then this value
* is meaningless.
*
* @param cache
* <code>true</code> if the preprocessor output is to be cached,
* otherwise <code>false</code>.
*/
public synchronized void setCache(boolean cache)
{
this.cache = cache;
}
/**
* Gets the state of the flag indicating if the preprocessor output should
* be cached. This value is dependent upon whether or not the
* preprocessor is run, if the preprocessor is not run, then this value
* is meaningless.
*
* @return <code>true</code> if the preprocessor output is to be cached,
* otherwise <code>false</code>.
*/
public synchronized boolean getCache()
{
return cache;
}
/**
* Sets the state of the flag indicating if lexer output should be
* dumped.
*
* @param dumpLexer
* <code>true</code> if lexer output is to be dumped, otherwise
* <code>false</code>.
*/
public synchronized void setDumpLexer(boolean dumpLexer)
{
this.dumpLexer = dumpLexer;
}
/**
* Accesses the state of the flag indicating if lexer output should be
* dumped.
*
* @return <code>true</code> if lexer output is to be dumped, otherwise
* <code>false</code>.
*/
public synchronized boolean getDumpLexer()
{
return dumpLexer;
}
/**
* Sets the state of the flag indicating if parser output should be
* dumped.
*
* @param dumpParser
* <code>true</code> if parser output is to be dumped, otherwise
* <code>false</code>.
*/
public synchronized void setDumpParser(boolean dumpParser)
{
this.dumpParser = dumpParser;
}
/**
* Accesses the state of the flag indicating if parser output should be
* dumped.
*
* @return <code>true</code> if parser output is to be dumped,
* otherwise <code>false</code>.
*/
public synchronized boolean getDumpParser()
{
return dumpParser;
}
/**
* Sets the state of the flag indicating if a newly parsed AST should be
* persisted.
*
* @param astPersist
* <code>true</code> to persist a new AST, otherwise
* <code>false</code>.
*/
public synchronized void setAstPersist(boolean astPersist)
{
this.astPersist = astPersist;
}
/**
* Accesses the state of the flag indicating if new ASTs should be
* persisted.
*
* @return <code>true</code> to persist a new AST, otherwise
* <code>false</code>.
*/
public synchronized boolean getAstPersist()
{
return astPersist;
}
/**
* Sets the PROPATH which defines how Progress source files are found.
*
* @param paths
* A single string with one or more paths to be searched for Progress files, with
* each path separated by the current platform's file separator character.
*/
public synchronized void setPaths(String paths)
{
this.paths = getPropath(paths, hints);
}
/**
* Accesses the PROPATH as defined by the project configuration or by
* the user of this class.
*
* @return Array of paths to be searched for Progress files.
*/
public synchronized String[] getPaths()
{
return paths;
}
/**
* Getter for our hints RFB20190911
*
* @return UastHints if any.
*/
public synchronized UastHints getHints()
{
return hints;
}
/**
* Mark the filename's AST as 'original'. This is used by already parsed .cls files, from pre-scan.
*
* @param filename
* Name of the source file to mark.
*/
public void markOriginal(String filename)
{
try
{
mgr.lockTree(filename, true);
long id = mgr.getTreeId(filename);
mgr.setFlag(id, AstManager.FLAG_ORIGINAL);
}
catch (Exception exc)
{
throw new AstException("Error marking as original " + filename, exc);
}
finally
{
mgr.unlockTree(filename);
}
}
/**
* Create or load an AST given a specific source filename. If AST
* persistence is activated, a valid persisted AST will be loaded. If no
* valid persisted AST exists or if AST persistence is disabled, then the
* AST will be created. If creation is used (as opposed to loading a
* persisted AST), this method will optionally preprocess the Progress
* source file and will parse the loaded file (preprocessed or not) into
* an abstract syntax tree.
* <p>
* The caller may provide a file list such that duplicate source names
* can be removed. This is not a generic service, but rather one that is
* specific to HTML files. In the case of an HTML file that has embedded
* 4GL (E4GL) code, the E4GL preprocessor will be invoked. This HTML file
* will then be converted into a normal 4GL source file. Since that source
* file is generated by the preprocessor, it may overwrite a file that is
* already there from a previous run (if caching is off or if the cached
* file is older than the timestamp on the HTML file). Since a valid 4GL
* source file may already exist in the file system, it is possible that
* that source file name (with a .p, .w or .i instead of the .html) may
* also appear in the file list that is used by the caller. Since the
* caller doesn't know the generated file name in the E4GL case, the
* pruning of that "duplicate" name is delegated to this method. The file
* list will be edited to remove any output file name that the E4GL
* preprocessor generates, if it is separately listed. This avoids
* duplicate processing of that file, since it was already processed
* fully to an AST when it was listed as an HTML file, it doesn't need to
* be processed again later. This processing is dependent upon all the
* HTML files being processed before any 4GL source files. The
* {@link #prioritize} method can be used to sort the list properly. Note
* that any removal is done by setting the duplicate entry to
* <code>null</code>. The calling code must be prepared to ignore any
* such <code>null</code> entries.
* <p>
* If this file represents a <code>.cls</code> file (a class or interface)
* it is assumed that the file is not a built-in 4GL class, nor is it a
* .NET class.
*
* @param filename
* Name of the source file to process; if this is not an absolute
* path, it should be relative to the propath.
* @param list
* The list of files upon which the generator is iterating in the
* calling method. The list may be edited as noted above. This
* may be <code>null</code> if no such editing is needed.
* @param idx
* Index position of the element being processed in the given
* list. Ignored if the list is <code>null</code>.
*
* @return The root AST of the parsed output.
*
* @throws AstException
* if the file cannot be found, or if there is some other error
* during processing.
*/
public Aast processFile(String filename, List<File> list, int idx)
throws AstException
{
filename = Configuration.normalizeFilename(filename);
// remember the original file name
String original = filename;
try
{
// exclude all other access to this file
// TODO: doesn't this locking need to be on filename.ast instead of just filename?
mgr.lockTree(filename, list != null);
// should we honor a previously persisted AST?
if (astPersist && !ConversionData.mustConvertFile(filename))
{
// check if the AST exists and has a later timestamp than the
// source file
if (checkForValidAst(filename))
{
// load the AST and return it
return loadPersistedAst(filename);
}
}
// does the filename reference an html file
if (isHTML(filename))
{
// we may have hints to honor for e4gl preprocessing
String hintsFile = findHintsFile(filename);
if (hintsFile != null)
{
hints = new UastHints(hintsFile);
}
E4GLPreprocessor e4gl = createE4GLPreprocessor(filename);
// use calc'd output name for downstream usage
filename = calcOutputFilename(e4gl);
// drive the web preprocessor
preprocessE4GL(e4gl, original, filename);
// rewrite the tree locking filename
mgr.rewriteTreeName(original, filename);
// make sure that caller's file list is updated to avoid
// containing both the html and the generated .p/.w/.i
if (list != null)
{
removeDuplicate(list, filename);
// set the current list element to be the output filename
// so that any downstream processing can find the AST
list.set(idx, new File(filename));
}
}
// load hints if they are available (must be done before propath
// processing)
if (hints == null)
{
hints = new UastHints(filename);
}
// setup propath from configuration
setPaths(null);
Aast root = parse(filename, hints.getBannedDatabases());
if (astPersist)
{
// this is not pre-scan, save the file hash
ConversionData.saveFileHash(filename);
}
return root;
}
catch (Exception exc)
{
throw new AstException("Error processing " + filename, exc);
}
finally
{
// clear our file-specific state (this releases these objects from
// the heap but more importantly it creates a clean slate for the
// next use of this method)
paths = null;
hints = null;
// clear our lock
mgr.unlockTree(filename);
}
}
/**
* Given a 4GL source code filename return the name of the AST file.
*
* @param filename
* The original source filename.
*
* @return The target AST filename.
*/
public static String getAstName(String filename)
{
return filename + AST_POSTFIX;
}
/**
* Builds a unique filename based on an original filename and a passed
* postfix. This method takes the original filename and appends a fixed
* value to create a new, unique cache filename.
*
* @param filename
* The original source filename.
* @param postfix
* Text to append to the original filename.
*
* @return The target cache filename.
*/
private static File createUniqueFile(String filename, String postfix)
{
return new File(filename + postfix);
}
/**
* Prepare the data stream which will be used by the parser. The file is
* first located in the file system. If this object is set to preprocess
* inline, this is done first, then an input stream is opened on the
* resulting data buffer. If inline preprocessing is turned off, the input
* stream is opened directly on the specified file.
*
* @param filename
* Name of the source file to process; if this is not an absolute
* path, it should be relative to the propath.
*
* @return Parser-ready input stream.
*
* @throws FileNotFoundException
* if <code>filename</code> does not exist.
* @throws IOException
* If a problem occurs during preprocessing or caching.
*/
private Reader prepareDataStream(String filename)
throws FileNotFoundException,
IOException
{
File file = Utils.findFileInPath(filename, paths);
Reader in = null;
if (preprocess)
{
in = preprocess(file);
}
else
{
InputStream is = new FileInputStream(file);
Charset charset = hints.getSourceCharset();
charset = charset == null ? Charset.defaultCharset() : charset;
in = new BufferedReader(new InputStreamReader(is, charset));
}
return in;
}
/**
* Find the UAST hints file associated with the given embedded 4GL (E4GL)
* program (which is an HTML file). This is done by removing the file
* extension to get a base name and then searching for a single file with
* the pattern <code>basename.*.</code> concatinated with the extension
* defined by {@link UastHints#HINTS_SUFFIX}. If more than 1 file is
* found, then none will be used since this code has no way of determining
* what the precedence order would be. Note that there is no known
* functional reason why more than one hints file would match this
* specification.
*
* @param filename
* The filename on which to base the hints filename.
*
* @return The hints filename (if one and only one exists) or
* <code>null</code> if no hints file can be found.
*/
private String findHintsFile(String filename)
{
File html = new File(filename);
String base = html.getName();
String lower = base.toLowerCase();
int len = base.length();
// calculate the end of the portion of the base name without the
// extension
if (lower.endsWith(HTML1_POSTFIX))
{
len -= HTML1_POSTFIX.length();
}
else if (lower.endsWith(HTML2_POSTFIX))
{
len -= HTML2_POSTFIX.length();
}
else
{
// not an html file
return null;
}
// build our file spec (as a regular expression)
String sb = base.substring(0, len) + "\\..*\\." + UastHints.HINTS_SUFFIX;
// determine the containing directory
String dpath = html.getParent();
if (dpath == null)
{
// fallback to the current directory
dpath = ".";
}
File dir = new File(dpath);
// get a list of files/sub-directories
String[] rawResults = dir.list();
String spec = sb;
String hintFile = null;
for (int i = 0; i < rawResults.length; i++)
{
if (rawResults[i].matches(spec))
{
if (hintFile == null)
{
hintFile = String.format("%s%s%s",
dpath,
File.separator,
rawResults[i]);
}
else
{
// more than one result, this is invalid so "bail out"
return null;
}
}
}
return hintFile;
}
/**
* Manage the process of initializing the E4GL preprocessor including
* the honoring of hints for <code>wsoptions</code> and the compatibility
* <code>mode</code>.
*
* @param filename
* The HTML file to preprocess.
*
* @return The instance that will be used to preprocess the specified
* HTML file.
*/
private E4GLPreprocessor createE4GLPreprocessor(String filename)
{
String modename = null;
String wsoptions = null;
if (hints != null)
{
wsoptions = hints.getWSOptions();
modename = hints.getE4GLMode();
}
else
{
// even if there is no hints file that is specific to the given
// source code, we may still need to honor the global configuration
wsoptions = Configuration.getParameter(ELEM_WSOPTIONS);
modename = Configuration.getParameter(ELEM_MODE);
}
int mode = DEFAULT_MODE;
if (modename != null)
{
mode = E4GLPreprocessor.decodeMode(modename);
if (mode == UNKNOWN_MODE)
mode = DEFAULT_MODE;
}
return new E4GLPreprocessor(filename, mode, wsoptions);
}
/**
* Determine the output file name to be used for the current HTML file
* being processed. If a hint file exists and has an explicit override
* specified, that will be used. Otherwise, the filename will be
* calculated based on any encoded options and the original HTML source
* file name.
*
* @param e4gl
* The preprocessor associated with the current file.
*
* @return The calculated name.
*/
private String calcOutputFilename(E4GLPreprocessor e4gl)
{
String output = null;
// force any encoded options to be picked up
e4gl.scanForWSOptions();
if (hints != null)
{
// honor any specified override (even if there are hints, the
// output file name may still be null)
output = hints.getE4GLOutputFile();
}
// calculate output name if needed
return output != null ? output : e4gl.calculateOutputDest();
}
/**
* Perform inline preprocessing of the specified source file.
*
* @param file
* File to be preprocessed.
*
* @return Input source connected to the beginning of the data buffer
* created by the {@link com.goldencode.p2j.preproc.Preprocessor}.
*
* @throws FileNotFoundException
* if <code>filename</code> does not exist.
* @throws IOException
* during a write or close failure.
*/
private Reader preprocess(File file)
throws FileNotFoundException,
IOException
{
boolean alreadyCached = false;
File cacheFile = null;
// Target output stream.
CharArrayWriter os = new CharArrayWriter();
if (cache)
{
cacheFile = createUniqueFile(file.getAbsolutePath(),
CACHE_POSTFIX);
if (cacheFile.exists() &&
cacheFile.lastModified() > file.lastModified())
{
alreadyCached = true;
}
}
if (!alreadyCached)
{
// If caching is NOT in effect or if the cache file is out of date,
// activate the preprocessor.
// Set up options.
Options options = buildOptions(paths, file, hints);
// Implement preprocessor runtime arguments, if they exist.
// It is OK for this to be null in the FileScope constructor.
Map args = hints.getArguments();
FileScope fs = new FileScope(file.getAbsolutePath(), args, options, false);
fs.setCharset(hints.getSourceCharset());
// Run preprocessor.
try
{
new Preprocessor(fs, os, err, warn, options);
}
catch (PreprocessorException exc)
{
LOG.log(Level.SEVERE, "", exc);
}
// Implement caching if enabled - this works because the Preprocessor
// constructor doesn't return until all preprocessing is complete
// AND because we can get a copy of the output more than once so
// this code doesn't conflict with other downstream code that uses
// the preprocessor output (we can be inserted transparently).
if (cache)
{
// If this filename exists as a directory, we can't cache.
if (!cacheFile.isDirectory())
{
// Remove the target file if it exists.
if (cacheFile.exists())
{
cacheFile.delete();
}
OutputStream fos = new FileOutputStream(cacheFile);
Writer out = new BufferedWriter(new OutputStreamWriter(fos, hints.getSourceCharset()));
// Cache the output and close the cache file.
out.write(os.toCharArray());
out.close();
}
}
}
else
{
// Take the cached version as input.
Reader in = new BufferedReader(new InputStreamReader(new FileInputStream(cacheFile),
hints.getSourceCharset()));
try
{
final int BUFF_SIZE = 8192;
char[] chrs = new char[BUFF_SIZE];
int read = 0;
while (read >= 0)
{
read = in.read(chrs, 0, BUFF_SIZE);
if (read > 0)
{
os.write(chrs, 0, read);
}
}
}
finally
{
in.close();
}
}
// Return reader with the preprocessed results.
return new CharArrayReader(os.toCharArray());
}
/**
* Perform E4GL preprocessing of the specified source HTML file into the
* given 4GL output file.
*
* @param e4gl
* The file-specific preprocessor instance.
* @param source
* HTML source file.
* @param target
* 4GL output file.
*/
private void preprocessE4GL(E4GLPreprocessor e4gl,
String source,
String target)
{
boolean alreadyCached = false;
if (cache)
{
File sourceFile = new File(source);
File cacheFile = new File(target);
if (cacheFile.exists() &&
cacheFile.lastModified() > sourceFile.lastModified())
{
alreadyCached = true;
}
}
if (!alreadyCached)
{
// if caching is NOT in effect or if the cache file is out of date,
// activate the preprocessor, otherwise let the downstream code use
// the file that already exists
// run the E4GL preprocessor
try
{
// the result will be cached automatically
e4gl.preprocess(false, null);
}
catch (Exception exc)
{
LOG.log(Level.SEVERE, "", exc);
}
}
}
/**
* Search the list to find a duplicate to the given name and remove it
* if found. Once the first matching entry is found, it will be set to
* <code>null</code> and the method will return.
*
* @param list
* Array to edit.
* @param duplicate
* Name of the file to remove. This may be <code>null</code>
* but then why would you call this method?
*/
private void removeDuplicate(List<File> list, String duplicate)
{
int size = list.size();
for (int i = 0; i < size; i++)
{
File file = list.get(i);
if (file != null && file.toString().equals(duplicate))
{
list.set(i, null);
return;
}
}
}
/**
* Performs first-level scanning of OO classes.
*
* @param filename
* Name of the source file to process; if this is not an absolute
* path, it should be relative to the propath.
* @param builtin
* <code>true</code> if this class file represents a built-in class or interface.
* @param dotnet
* <code>true</code> if this class file represents a .NET class or interface.
* @param propath
* The current <code>PROPATH</code> to honor. May not be <code>null</code>.
*
* @throws AstException
* if the file cannot be found, or if there is some other error during processing.
*/
void preScanClass(String filename, boolean builtin, boolean dotnet, String[] propath)
throws AstException,
ANTLRException,
IOException
{
filename = Configuration.normalizeFilename(filename);
boolean caseSens = Configuration.getParameter(CASE_SENS_KEY, true);
UastHints saveHints = hints;
String[] savePaths = paths;
try
{
// exclude all other access to this file
mgr.lockTree(filename, false);
// TODO: hints will not work, as this is called recursive... need a stack approach
hints = new UastHints(filename);
this.paths = propath;
SymbolResolver sym = new SymbolResolver(true,
caseSens,
propath,
silent,
true,
hints.getBannedDatabases());
processParserHints(sym);
// class definitions require a 2-pass run, where the 1st pass preloads
// resources that can be accessed before their definitions occur on the
// 2nd pass (e.g. an instance method can be called in an expression
// earlier in the file than where that method's definition appears)
ProgressLexer lexer = prepareLexer(filename, sym);
ProgressParser parser = null;
parser = new ProgressParser(lexer, sym);
parser.setFilename(filename);
parser.setBuiltInCls(builtin);
parser.setDotNetCls(dotnet);
parser.pre_scan_class();
}
catch (ParentUnavailableException e)
{
// let the exception propagate, everything will be re-parsed when this same class is re-parsed.
throw e;
}
catch (Exception exc)
{
throw new AstException("Error processing " + filename, exc);
}
finally
{
// clear our file-specific state (this releases these objects from
// the heap but more importantly it creates a clean slate for the
// next use of this method)
paths = savePaths;
hints = saveHints;
// clear our lock
mgr.unlockTree(filename);
}
// if everything was completed OK and this is a built-in class, do a full parse to save
// the AST
if (builtin)
{
boolean oldAstPersist = this.astPersist;
this.astPersist = true;
try
{
processFile(filename, null, -1);
}
finally
{
this.astPersist = oldAstPersist;
}
}
}
/**
* Parse the source code read from the specified filename into an abstract
* syntax tree. Uses the Progress parser and its helpers to do this work.
*
* @param filename
* The name of the file being processed.
*
* @return Root of the AST created by the parser.
*
* @throws ANTLRException
* if any error occurs while lexing or parsing the source code.
*/
private Aast parse(String filename, Set<String> bannedSchemas)
throws AstException,
ANTLRException,
IOException
{
boolean caseSens = Configuration.getParameter(CASE_SENS_KEY, true);
SymbolResolver sym = new SymbolResolver(true, caseSens, paths, silent, true, bannedSchemas);
processParserHints(sym);
// the very first time we try to parse a class, we must ensure that the critical built-in classes
// are already loaded; without this some classes (enums especially) can cause a parsing failure
// during pre-scanning which is avoided when these same classes are already loaded
if (filename.endsWith(SymbolResolver.CLASS_EXT))
{
synchronized (AstGenerator.class)
{
if (!earlyClassInit)
{
earlyClassInit = true;
sym.loadClass(SymbolResolver.ROOT_OBJ_NAME);
}
}
}
ProgressLexer lexer = prepareLexer(filename, sym);
ProgressParser parser = null;
parser = new ProgressParser(lexer, sym);
parser.setFilename(filename);
// Store off our AstGenerator instance RFB20190911
parser.setGenerator(this);
boolean complete = parser.external_proc();
if (!complete)
{
Token next = lexer.nextToken();
LOG.log(Level.WARNING,
String.format("Parser did NOT process to EOF " +
"(no match with token immediately BEFORE %s:%s " +
"in file '%s' with token text '%s').\n",
next.getLine(),
next.getColumn(),
filename,
next.getText()));
}
// get the resulting AST
AnnotatedAst result = (AnnotatedAst) parser.getAST();
// makes sure that parents and IDs are properly assigned (this is a
// form of post-processing)
result.brainwash(filename, true);
if (result.downPath(ProgressParser.CLASS_DEF) ||
result.downPath(ProgressParser.INTERFACE_DEF) ||
result.downPath(ProgressParser.ENUM_DEF))
{
// this is a class - save the AST IDs where is required
boolean isClass = result.downPath(ProgressParser.CLASS_DEF);
boolean isEnum = result.downPath(ProgressParser.ENUM_DEF);
boolean isIface = result.downPath(ProgressParser.INTERFACE_DEF);
int tok = isClass ? ProgressParser.CLASS_DEF
: isIface ? ProgressParser.INTERFACE_DEF
: ProgressParser.ENUM_DEF;
Aast ref = result.getImmediateChild(tok, null);
tok = isClass ? ProgressParser.KW_CLASS
: isIface ? ProgressParser.KW_INTERFAC
: ProgressParser.KW_ENUM;
ref = ref.getImmediateChild(tok, null);
ref = ref.getImmediateChild(ProgressParser.SYMBOL, null);
String qname = ref.getText();
ClassDefinition cdef = SymbolResolver.getClassDefinition(qname);
cdef.parseFinished(false);
}
// save off output if requested, THIS MUST BE DONE AFTER FIXUPS!
if (dumpParser)
generateParseReport(filename, result);
if (astPersist)
{
persistAst(filename, result);
// persist schema dictionary state
File schemaFile = createUniqueFile(filename, DICT_POSTFIX);
try
{
sym.persistSchemaData(filename, schemaFile.getAbsolutePath());
}
catch (Exception exc)
{
throw new AstException(
"Error persisting schema dictionary state to " + schemaFile,
exc);
}
}
int errCount = parser.getErrorCount();
if (errCount > 0)
{
throw new RuntimeException("Parser encountered " + errCount + " errors");
}
return result;
}
/**
* Prepare a lexer instance for the given file using the supporting
* resolver instance as needed.
*
* @param filename
* The file being lexed.
* @param sym
* The symbol resolver instance being used for this file.
*
* @return A token stream source for the given file.
*/
private ProgressLexer prepareLexer(String filename, SymbolResolver sym)
throws IOException
{
// preprocess file or load source from cache
Reader in = prepareDataStream(filename);
ProgressLexer lexer = null;
if (dumpLexer)
{
File lexerFile = createUniqueFile(filename, LEXER_POSTFIX);
if (lexerFile.exists() && lexerFile.isFile())
{
lexerFile.delete();
}
lexer = new LexerDumpFilter(in, sym, lexerFile);
}
else
{
lexer = new ProgressLexer(in, sym);
}
// keyword ignore support must be implemented after the lexer is
// instantiated since the keyword processing is not finished
// initializing until that point
String ignore = Configuration.getParameter(IGNORE_KEY);
if (ignore != null)
{
String[] list = readIntoArray(ignore);
if (list != null)
{
lexer.ignoreKeywords(list);
}
}
// force the escape mode
lexer.setUnixEscapes(hints.isUnixEscapes());
return lexer;
}
/**
* Drive the process by which the hints file is queried for parser
* related hints, any hints found are implemented.
*
* @param sym
* Provides access to the symbol namespaces such that hints can
* be loaded.
*/
private void processParserHints(SymbolResolver sym)
{
// if databases need to be connected before parsing, we load them here
sym.loadSchemaDatabases(hints.getDatabases());
// if aliases need to be added before parsing, we load them here; NOTE
// that this MUST be done AFTER database loads
List<String[]> aliases = hints.getAliases();
if (aliases != null)
{
sym.loadAliases(aliases);
}
// if variables/widgets need to be added to the dictionary, do this now
Variable[] varlist = hints.getVariables();
if (varlist != null)
{
sym.loadVariableSymbols(varlist);
}
}
/**
* Read a simple text file into an array where each line in the input file
* is a separate element in the output array.
*
* @param filename
* Name of the text file to read.
*
* @return Array of text that was read or <code>null</code> if there
* was any error.
*/
private String[] readIntoArray(String filename)
{
String[] out = new String[0];
try
{
ArrayList<String> list = new ArrayList<>();
try (BufferedReader in = new BufferedReader(new FileReader(filename)))
{
String next = in.readLine();
while (next != null)
{
list.add(next);
next = in.readLine();
}
out = list.toArray(out);
}
}
catch (Exception exc)
{
LOG.log(Level.WARNING,"", exc);
}
return out;
}
/**
* Calculates a unique stream name for storing the output, based on the
* input file name and a postfix. Takes the input filename (including the
* absolute path) and appends the a user-defined postfix to it. As long
* as the resulting name is not a directory, it is deleted if it exists
* and the new stream is then returned.
*
* @param original
* Input file name.
*
* @return An open stream on which to write the parser output.
*/
private PrintStream createUniqueStream(String original, String postfix)
{
PrintStream out = null;
if (original != null)
{
try
{
// build a unique filename based on the original
File parse = createUniqueFile(original, postfix);
// if this filename exists as a directory, we can't save results
if (!parse.isDirectory())
{
// remove the target file if it exists.
if (parse.exists())
{
parse.delete();
}
out = new PrintStream( new FileOutputStream(parse) );
}
}
catch (Exception excpt)
{
LOG.log(Level.WARNING,"", excpt);
}
}
return out;
}
/**
* Dump an AST into a file with a human readable report format. A filename
* is generated off the original source file's name, with a standard
* extension of <code>PARSER_POSTFIX</code>.
*
* @param filename
* Original source file being processed.
* @param ast
* The AST to dump.
*/
private void generateParseReport(String filename, Aast ast)
{
PrintStream out = createUniqueStream(filename, PARSER_POSTFIX);
DumpTree visitor = new DumpTree(out);
visitor.visit(ast);
out.close();
}
/**
* Persist an AST into an XML file which can be turned into a valid AST
* using <code>AstPersister</code>. A filename is generated off the
* original source file's name, with a standard extension of
* <code>AST_POSTFIX</code>.
*
* @param filename
* Original source file being processed.
* @param ast
* The AST to dump.
*/
private void persistAst(String filename, Aast ast)
throws AstException
{
File target = createUniqueFile(filename, AST_POSTFIX);
String fname = null;
try
{
fname = target.getCanonicalPath();
}
catch (IOException ioe)
{
throw new AstException("Invalid filename " + filename, ioe);
}
AstManager.get().saveTree(ast, fname, false);
}
/**
* Checks whether or not the source filename has a valid persisted AST.
* Validity is determined based on the fact that the AST exists AND that
* it's timestamp is later than that of the source file.
*
* @param filename
* The source filename to check for a matching persisted AST.
*
* @return <code>true</code> if a persisted AST exists AND that AST
* has a timestamp later than that of the source file,
* <code>false</code> otherwise.
*/
private boolean checkForValidAst(String filename)
{
File src = new File(filename);
File ast = createUniqueFile(filename, AST_POSTFIX);
return ast.exists() && ( ast.lastModified() > src.lastModified() );
}
/**
* Loads a previously persisted AST for a specified source file.
*
* @param filename
* The source filename for which a matching persisted AST will
* be loaded.
*
* @return The AST associated with the specified source file.
*
* @throws AstException
* if there are any problems with loading the AST.
*/
private Aast loadPersistedAst(String filename)
throws AstException
{
File source = createUniqueFile(filename, AST_POSTFIX);
String fname = null;
try
{
fname = source.getCanonicalPath();
}
catch (IOException ioe)
{
throw new AstException("Invalid filename " + filename, ioe);
}
return AstManager.get().loadTree(fname);
}
/**
* Prepend the given text to the path list in the buffer, optionally
* separating the path elements with the file system path separator if
* a path separator is not already present.
*
* @param sb
* Buffer containing current path list.
* @param txt
* Path element to prepend (insert at position 0).
*/
private static void prependPath(StringBuilder sb, String txt)
{
if (sb.charAt(0) != File.pathSeparatorChar)
{
sb.insert(0, File.pathSeparatorChar);
}
sb.insert(0, txt);
}
/**
* Append the given text to the path list in the buffer, optionally
* separating the path elements with the file system path separator if
* a path separator is not already present.
*
* @param sb
* Buffer containing current path list.
* @param txt
* Path element to append.
*/
private static void appendPath(StringBuilder sb, String txt)
{
int i = sb.length() - 1;
if (sb.charAt(i) != File.pathSeparatorChar)
{
sb.append(File.pathSeparatorChar);
}
sb.append(txt);
}
}