# AstGenerator.java

    /*
    ** Module   : AstGenerator.java
    ** Abstract : Creates an AST from a single source file using Progress preprocessor and parser tools
    **
    ** Copyright (c) 2005-2025, Golden Code Development Corporation.
    **
    ** -#- -I- --Date-- --JPRM-- -----------------------------------Description-----------------------------------
    ** 001 ECF 20050121   @19369 Created initial version.
    ** 002 GES 20050126   @19446 Added support for caching preprocessor
    **                           output.  In input, will bypass preprocessing
    **                           (even if requested) if there is a
    **                           pre-existing cache file that has a later
    **                           timestamp than the input file.
    ** 003 GES 20050204   @19569 Added support for hints processing to load
    **                           the list of non-default databases.
    ** 004 GES 20050207   @19625 Added preprocessor runtime arguments support
    **                           with input from the hints file.
    ** 005 GES 20050210   @19747 Added variable, widget, frame and alias
    **                           hints support.
    ** 006 GES 20050218   @19861 Modified creation of the FileScope object
    **                           to match new features that properly
    **                           implement runtime arguments. This new
    **                           approach also simplifies this code greatly.
    **                           Also added code to check if the parser
    **                           reached EOF or exited prematurely. In the
    **                           case where an early exit is detected, an
    **                           diagnostic error message is output.
    ** 007 ECF 20050118   @19890 Changed type of AST this class generates. We
    **                           now create instances of AnnotatedAst instead
    **                           of UnifiedAST. The latter has been obsoleted
    **                           and removed from the project.
    ** 008 GES 20050122   @19904 Added fixup processing for the AST, after it
    **                           is built. This allows parents and IDs to be
    **                           assigned to the tree before it is ever used.
    **                           Also added basic lexer dumping and greater
    **                           external control over state variables.
    ** 009 GES 20050124   @19948 Major rework to rationalize the constructors,
    **                           option processing and most importantly to
    **                           add persistence support. ASTs can be
    **                           persistently cached and loaded transparently
    **                           using this class.
    ** 010 ECF 20050225   @19959 Made AST_POSTFIX constant public. This is
    **                           needed to provide user functions the ability
    **                           to persist ASTs using a consistent filename
    **                           extension.
    ** 011 ECF 20050225   @20060 Moved a modified version of findFile method
    **                           to the common, Utils class. We now call that
    **                           method (findFileInPath) from here.
    ** 012 GES 20050303   @20165 Minor change to match the changed DumpTree
    **                           interface.
    ** 013 GES 20050307   @20198 Made all POSTFIX constants public so that
    **                           external users don't have to hard-code the
    **                           same strings.
    ** 014 GES 20050323   @20479 Converted to use ProgressAst instead of
    **                           AnnotatedAst which is now abstract.
    ** 015 GES 20050328   @20508 Moved AST class processing into parser to
    **                           centralize this function and make clients
    **                           simpler and more independent.
    ** 016 ECF 20050331   @20614 If in AST persistence mode, also persist a
    **                           *.p.schema file for each source file. This
    **                           contains temp-table schema information.
    ** 017 GES 20050411   @20665 New SymbolResolver constructor signature.
    ** 018 ECF 20050411   @20699 Modified to conform with new SymbolResolver
    **                           and Preprocessor changes. Removed redundant
    **                           argument from call to symbol resolver's
    **                           persistSchemaData method. Catch new exception
    **                           thrown by Preprocessor constructor.
    ** 019 ECF 20050419   @20853 Modified to minimize having to delete the AST
    **                           registry. Previously, re-generating an AST
    **                           which was registered meant deleting the
    **                           registry and re-scanning completely. Now, we
    **                           reset the next node ID in the registry for
    **                           each AST just before applying parent and ID
    **                           fixups. This allows AST file IDs to be
    **                           reused without making the entire registry
    **                           stale. Note: any existing references to the
    **                           old IDs WILL be stale, however.
    ** 020 NVS 20050516   @21179 New Preprocessor option is enabled - produce
    **                           the hints file. The file is generated with
    **                           the base file name and ".pphints" postfix.
    ** 021 ECF 20060202   @24255 Store temp/work-table schema information
    **                           using ".dict" extension. Previous ".schema"
    **                           extension now is used later, when schema
    **                           fixups are applied.
    ** 022 GES 20070313   @32396 Support global dynamic propaths and file
    **                           specific propath hints.
    ** 023 GES 20070403   @32714 Support hint control over escape sequence
    **                           processing in the preprocessor.
    ** 024 GES 20070423   @33209 Added preprocessor marker hint support.
    ** 025 GES 20070629   @34344 Minor interface change.
    ** 026 GES 20070816   @34889 Added preprocessor hint support for built-in
    **                           variables BATCH-MODE, OPSYS and
    **                           WINDOW-SYSTEM.
    ** 027 GES 20070830   @34948 Added WebSpeed/Blue Diamond preprocessing
    **                           support. This takes any input file that is
    **                           HTML, and converts it into 4GL source code.
    **                           The resulting source code is then run through
    **                           the preprocessor, lexer and parser just as
    **                           if it had been static 4GL source.
    ** 028 GES 20070913   @35130 Made this class safe to use reentrantly.
    **                           Also added file-level locking to ensure
    **                           process-wide exclusive access.
    ** 029 GES 20070923   @35183 Added keyword ignore support to the lexer
    **                           init which is equivalent to the Progress -k
    **                           startup parameter (honors a keyword "forget"
    **                           file).
    ** 030 GES 20070926   @35241 Pass the escape processing mode to the lexer
    **                           to allow it to honor or ignore backslash
    **                           escapes.
    ** 031 GES 20080318   @37661 Changed approach to read characters (the
    **                           entire unicode charset) instead of dealing
    **                           with bytes. This makes it safe to process
    **                           files with I18N requirements.
    ** 032 GES 20090427   @42023 Import change.
    ** 033 GES 20090429   @42056 Match package and class name changes.
    ** 034 GES 20090515   @42226 Moved to AstManager from AstRegistry/AstPersister.
    ** 035 GES 20090518   @42404 Fixup for usage of brainwash().
    ** 036 GES 20100922          Normalize the filename at the top of processFile()
    **                           so that lockTree() and brainwash() both use the
    **                           same input string. This eliminates duplicate
    **                           registry entries for the same file. Added
    **                           parameter to differentiate the behavior of
    **                           brainwash() where 4GL code trees now will save off
    **                           comments and other hidden tokens as shadow nodes.
    ** 037 ECF 20101001          Added logic to query case-sensitivity parameter
    **                           from Configuration and disable it in preprocessor
    **                           options if set to 'false'. Use file and path
    **                           separators appropriate to original platform of the
    **                           project.
    ** 038 GES 20110712          Now the code passes the case-sensitivity flag to
    **                           the symbol resolver (to enable case-insensitive
    **                           file-system searches).
    ** 039 GES 20110822          Added a pre-scanning pass for parsing class files
    **                           which is needed to allow class resources to be
    **                           accessed out-of-order (earlier than their defs
    **                           appear in the file).
    ** 040 GES 20111203          Fixed regression in an printf error message spec.
    **                           Reworked propath processing code to be static and
    **                           to be exposed publicly so it can be reused.
    ** 041 OM  20130607          Fixed imports that conflicted with Hibernate namespace.
    ** 042 GES 20140313          Safety code so that input file failures get handled gracefully.
    ** 043 ECF 20150715          Replace StringBuffer with StringBuilder.
    ** 044 GES 20160225          Honor OO skeleton path override.
    ** 045 IAS 20160331          Add input streams' close.
    ** 046 ECF 20160302          Made processFile and removeDuplicate threadsafe.
    ** 047 GES 20160225          Add OO class resource cleanup.
    ** 048 HC  20160802          Fixed handling of file separators when including files during
    **                           preprocessing.
    ** 049 OM  20170123          Paths returned by Configuration already use the current OS
    **                           separators.
    ** 050 GES 20170303          Added support for configuring the preprocessor debug flag.
    ** 051 GES 20170708          Added a silent flag.
    **     HC  20170708          Added source-charset uast preprocessor hint.
    **                           Fixed loading of preprocessed file cache (*.cache) when the file
    **                           included non-ascii (multibyte) characters.
    **     CA  20170711          1st level OO class scanning is decoupled from full parse: first, all
    **                           references from members or inheritance are resolved, for the entire
    **                           graph, and only after that the reached classes are parsed.
    **     HC  20170726          Made AstGenerator to report failure when ProgressParser encounters
    **                           errors.
    ** 052 GES 20180226          Improve error handling by ensuring that the parser knows the
    **                           filename being processed.
    **     CA  20180321          Added support for PROCESS-ARCHITECTURE preprocessor var.
    ** 053 CA  20181121          preScanClass must inherit the PROPATH from the caller.
    ** 054 HC  20190227          Fixed text encoding for the preprocessed cache file.
    ** 055 CA  20190513          Fixed buffer usage from super-classes, in OO.
    ** 056 ECF 20190619          Added support for include file reference prefix remapping.
    ** 057 GES 20190625          Removed previous .NET dependency tracking approach.
    **     RFB 20190911          Schema processing may occur which references include files and
    **                           preprocessor directives. However, the hints file was not being
    **                           honored, since it was not accessible from progress.g's parser.
    **                           Added a getHints and storage method for this.
    ** 058 CA  20200412          Added incremental conversion support.
    ** 059 CA  20200428          Process enums for incremental conversion support.
    **     GES 20200522          Fix a dependency problem with enum parsing by pre-loading the most important
    **                           built-in 4GL OO classes. This only happens just before the first time a class
    **                           is parsed.
    **     OM  20210913          Added flags capability to AST registry.
    **     OM  20211122          Added hints-based support for 'unloading' schema.
    **     OM  20220405          XmlFilePlugin can load resources from application's jar.
    **     GES 20220517          Added getAstName().
    **     CA  20220727          Improved memory management for parsing; cleanup is done in two phases:
    **                           1. after each legacy class file has finished parsing, the SchemaDictionary will
    **                              keep only protected temp-tables (all private tables are removed, as these
    **                              can't be reached from sub-classes; all permanent tables are removed, as each
    **                              class has its own reference to the permanent tables).  Also, the class def
    **                              instance will reduce its own used memory, which is not required when parsing
    **                              sub-classes.
    **                           2. after parsing of the entire file set is finished, any SchemaDictionary or
    **                              other ASTs referenced by the ClassDefinition are released.
    ** 060 GBB 20230512          Logging methods replaced by CentralLogger/ConversionStatus.
    ** 061 OM  20230115          Replaced absolutePath(), relativePath(), upPath() and downPath() with faster
    **                           versions, based on node types.
    ** 062 GES 20241112          Rewrite the tree locking data structures for HTML filenames.
    ** 063 CA  20241026          Fixed an issue with circular references when parsing classes: if a parent is
    **                           already 'pre-scanned', avoid parsing the class and let it parse when it's reached
    **                           again.
    **     CA  20241104          Pre-scan must use the banned schemas (from the hints).
    **     CA  20241118          Mark already parsed .cls file's ASTs as 'original' in the registry.
    ** 064 CA  20250422          If there are duplicated PROPATH entries, leave only the first occurrence, as this
    **                           will result two matches found when automatically adding .cls depdendencies at
    **                           conversion.
    */

    /*
    ** This program is free software: you can redistribute it and/or modify
    ** it under the terms of the GNU Affero General Public License as
    ** published by the Free Software Foundation, either version 3 of the
    ** License, or (at your option) any later version.
    **
    ** This program is distributed in the hope that it will be useful,
    ** but WITHOUT ANY WARRANTY; without even the implied warranty of
    ** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    ** GNU Affero General Public License for more details.
    **
    ** You may find a copy of the GNU Affero GPL version 3 at the following
    ** location: https://www.gnu.org/licenses/agpl-3.0.en.html
    **
    ** Additional terms under GNU Affero GPL version 3 section 7:
    **
    **   Under Section 7 of the GNU Affero GPL version 3, the following additional
    **   terms apply to the works covered under the License.  These additional terms
    **   are non-permissive additional terms allowed under Section 7 of the GNU
    **   Affero GPL version 3 and may not be removed by you.
    **
    **   0. Attribution Requirement.
    **
    **     You must preserve all legal notices or author attributions in the covered
    **     work or Appropriate Legal Notices displayed by works containing the covered
    **     work.  You may not remove from the covered work any author or developer
    **     credit already included within the covered work.
    **
    **   1. No License To Use Trademarks.
    **
    **     This license does not grant any license or rights to use the trademarks
    **     Golden Code, FWD, any Golden Code or FWD logo, or any other trademarks
    **     of Golden Code Development Corporation. You are not authorized to use the
    **     name Golden Code, FWD, or the names of any author or contributor, for
    **     publicity purposes without written authorization.
    **
    **   2. No Misrepresentation of Affiliation.
    **
    **     You may not represent yourself as Golden Code Development Corporation or FWD.
    **
    **     You may not represent yourself for publicity purposes as associated with
    **     Golden Code Development Corporation, FWD, or any author or contributor to
    **     the covered work, without written authorization.
    **
    **   3. No Misrepresentation of Source or Origin.
    **
    **     You may not represent the covered work as solely your work.  All modified
    **     versions of the covered work must be marked in a reasonable way to make it
    **     clear that the modified work is not originating from Golden Code Development
    **     Corporation or FWD.  All modified versions must contain the notices of
    **     attribution required in this license.
    */

    package com.goldencode.p2j.uast;

    import java.io.*;
    import java.nio.charset.*;
    import java.util.*;
    import java.util.logging.*;

    import antlr.*;
    import com.goldencode.ast.*;
    import com.goldencode.p2j.cfg.*;
    import com.goldencode.p2j.convert.*;
    import com.goldencode.p2j.convert.db.*;
    import com.goldencode.p2j.util.Utils;
    import com.goldencode.p2j.e4gl.*;
    import com.goldencode.p2j.preproc.*;
    import com.goldencode.p2j.preproc.Options;

    /**
     * Returns an AST for a specified Progress source file by creating this AST
     * (using the preprocessor and parser) or by loading an already existing
     * persisted AST from a file.
     * <p>
     * Optionally preprocesses, then parses a single Progress source file into an
     * abstract syntax tree (AST). Optionally accepts a <code>PROPATH</code>
     * setting which is used to allow the preprocessor to locate relative source
     * file references (includes). If this is not provided, the current directory
     * is assumed to be the top level directory for all filename resolution.
     * <p>
     * Caching of the preprocessor results can be optionally enabled.  In this
     * case the output will be stored in a file of the same base name as the
     * original with an appended '.cache'.  For example, the preprocessor output
     * for utilities/top-menu.p would be stored in utilities/top-menu.p.cache.
     * <p>
     * If the given input file is HTML, it is converted into 4GL source code by
     * the embedded 4GL (E4GL) preprocessor. The resulting source code is then
     * run through the preprocessor, lexer and parser just as if it had been
     * static 4GL source. Note that the same flags that control the running and
     * caching of the 4GL preprocessor, control the running and caching of the
     * embedded 4GL (E4GL) preprocessor. Note that the only difference is that
     * the output of the E4GL preprocessor is always written to file. See
     * {@link #processFile} for more details on this support.
     * <p>
     * For debugging and regression testing purposes, it is possible to save
     * the output of the lexer and/or the parser in a more human readable format.
     * The {@link #setDumpLexer} and {@link #setDumpParser} methods provide
     * this capability.  Note that the resulting files will be generated with the
     * original source filename and an appended '.lexer' and '.parser'
     * respectively.  The existence of these files does not alter any subsequent
     * processing and subsequent runs will replace these files with no warning.
     * <p>
     * An additional option is provided to enable AST persistence.  If this is
     * enabled, it changes the processing in 2 ways.  First, if a valid persisted
     * AST already exists, the AST returned by this class will be loaded from
     * that file AND the preprocessor/parser will not be invoked.  Second, in the
     * case that no persisted AST exists yet OR the persisted AST is not later in
     * timestamp than the source file, the output from the parser will be
     * persisted into a file with the original source filename and an appended
     * '.ast'.
     * <p>
     * By default, preprocessing, preprocessor output caching and AST persistance
     * are all enabled BUT lexer and parser output is disabled.
     */
    public class AstGenerator
    implements HintsConstants,
               E4GLConstants
    {
       /**
        * Text to append to a filename to generate a unique name for persisting
        * an AST.
        */
       public static final String AST_POSTFIX = ".ast";

       /** Text to append to a filename to generate a unique name for caching. */
       public static final String CACHE_POSTFIX = ".cache";

       /** Text to append to a filename to generate a unique name for hints. */
       public static final String HINTS_POSTFIX = ".pphints";

       /**
        * Text to append to a filename to generate a unique name for saving
        * human-readable lexer output.
        */
       public static final String LEXER_POSTFIX = ".lexer";

       /**
        * Text to append to a filename to generate a unique name for saving
        * human-readable parser output.
        */
       public static final String PARSER_POSTFIX = ".parser";

       /**
        * Text to append to a filename to generate a unique name for persisting
        * schema dictionary information.
        */
       public static final String DICT_POSTFIX = ".dict";

       /** The flag for marking a DICT schema tree in AST registry. */
       public static final int FLAG_DICT = AstManager.FLAG_USER;

       /** A filename extention to recognize as an HTML file. */
       public static final String HTML1_POSTFIX = ".html";

       /** A filename extention to recognize as an HTML file. */
       public static final String HTML2_POSTFIX = ".htm";

       /** Key for the file separator configuration parameter. */
       private static final String FILESEP_KEY = "file-separator";

       /** Key for the path separator configuration parameter. */
       private static final String PATHSEP_KEY = "path-separator";

       /** Key for the global propath configuration parameter. */
       private static final String PROPATH_KEY = "propath";

       /** Key to enable global directory prepending to the propath. */
       private static final String DYN_PROPATH_PREPEND_KEY = "dynamic-prepend";

       /** Key to enable global directory appending to the propath. */
       private static final String DYN_PROPATH_APPEND_KEY = "dynamic-append";

       /** Key for the keyword ignore configuration parameter. */
       private static final String IGNORE_KEY = "keyword-ignore";

       /** Key for the global, file system case sensitivity key. */
       private static final String CASE_SENS_KEY = "case-sensitive";

       /** Key for the error logfile. */
       private static final String ERR_DEST_KEY = "error-dest";

       /** Key for the warning logfile. */
       private static final String WARN_DEST_KEY = "warn-dest";

       /** Logger */
       private static final ConversionStatus LOG = ConversionStatus.get(AstGenerator.class);

       /** Global manager for file locking and file ID services. */
       private static AstManager mgr = null;

       /** Error destination. */
       private static Writer err = new PrintWriter(System.err);

       /** Warning destination. */
       private static Writer warn = err;

       /** Flag to track whether early builtin class initialization is complete. */
       private static boolean earlyClassInit = false;

       /** Flag indicating whether silent mode is required. */
       private boolean silent = false;

       /** Flag indicating whether inline preprocessing is needed. */
       private boolean preprocess = true;

       /** Indicates whether caching of preprocessor results should be done. */
       private boolean cache = true;

       /** Flag indicating whether dumping of lexer output is needed. */
       private boolean dumpLexer = false;

       /** Flag indicating whether dumping of parser output is needed. */
       private boolean dumpParser = false;

       /** Flag indicating whether storing of a persistent AST is needed. */
       private boolean astPersist = true;

       /** Array of propath path elements. */
       private String[] paths = null;

       /**
        * Represents any non-default, file specific behavior that must be
        * handled for the preprocessor or parser.
        */
       private UastHints hints = null;

       static
       {
          try
          {
             String name = Configuration.getParameter("registry");
             mgr = AstManager.initialize(new XmlFilePlugin(name, Configuration.isImportMode()));

             // honor the skeleton path override when the SR is being used for normal conversion
             SymbolResolver.initSkeletonPath();

             String errDest  = Configuration.getParameter(ERR_DEST_KEY);
             String warnDest = Configuration.getParameter(WARN_DEST_KEY);

             try
             {
                if (errDest != null)
                {
                   err  = new BufferedWriter(new FileWriter(errDest));
                   warn = err;
                }
                if (warnDest != null)
                {
                   warn = new BufferedWriter(new FileWriter(warnDest));
                }
             }

             catch (Exception e)
             {
                // use the defaults
             }
          }

          catch (AstException ae)
          {
             // TODO: log this? get ready for null pointer exceptions later
          }
       }

       /**
        * Default constructor which initializes the propath setting based on
        * the project configuration.
        */
       public AstGenerator()
       {
       }

       /**
        * Sort the given list to ensure that any HTML files are listed first.
        * <p>
        * Make sure all HTML files appear first since they get converted into 4GL
        * source and these generated source file names may be separately
        * duplicated in this list. By ensuring that the HTML files are listed
        * first, the processing of the HTML files can also remove any duplicate
        * 4GL source file names from the list, ensuring that no unnecessary
        * processing is done.  See {@link #processFile}.
        *
        * @param    list
        *           The list to sort.
        *
        * @return   The sorted list.
        */
       public static File[] prioritize(File[] list)
       {
          ArrayList<File> html = new ArrayList<>();
          ArrayList<File> src  = new ArrayList<>();
          ArrayList<File> full = new ArrayList<>();

          // separate HTML and 4GL source files, leave the order otherwise
          // the same
          for (int i = 0; i < list.length; i++)
          {
             if (isHTML(list[i].getName()))
             {
                html.add(list[i]);
             }
             else
             {
                src.add(list[i]);
             }
          }

          if (!html.isEmpty())
          {
             full.addAll(html);
             full.addAll(src);
             full.toArray(list);
          }

          return list;
       }

       /**
        * Uses the filename extension to determine if this is an HTML file or
        * not.  See {@link #HTML1_POSTFIX} and {@link #HTML2_POSTFIX}.
        *
        * @param    filename
        *           The filename to test.
        *
        * @return   <code>true</code> if the filename ends with one of the
        *           known HTML extensions.
        */
       public static boolean isHTML(String filename)
       {
          String lower = filename.toLowerCase();
          return lower.endsWith(HTML1_POSTFIX) || lower.endsWith(HTML2_POSTFIX);
       }

       /**
        * Common setup for preprocessor options which uses hints and global
        * configuration as available.
        *
        * @param    paths
        *           The propath;
        * @param    file
        *           The specific source file being processed. May be
        *           <code>null</code> if this preprocessor is not associated with
        *           a specific 4GL source file.
        * @param    hints
        *           Any 4GL source file hints that may exist.  May be
        *           <code>null</code> if no hints exists or if there is no
        *           4GL source file associated.
        *
        * @return   The new options instance, suitably configured.
        */
       public static Options buildOptions(String[] paths, File file, UastHints hints)
       {
          Options options = new Options();

          options.setProPath(paths);

          if (file != null)
          {
             File hintsFile = createUniqueFile(file.getAbsolutePath(), HINTS_POSTFIX);
             options.setHintsFile(hintsFile.getAbsolutePath());
          }

          char    marker      = '\000';
          boolean unixEscapes = true;
          boolean batchMode   = false;
          boolean debug       = false;
          String  opsys       = null;
          String  winsys      = null;
          int     arch        = 32;

          if (hints != null)
          {
             // honor file or directory level overrides
             marker      = hints.getMarker();
             unixEscapes = hints.isUnixEscapes();
             batchMode   = hints.isBatchMode();
             opsys       = hints.getOpsys();
             winsys      = hints.getWindowSystem();
             debug       = hints.isPreprocDebug();
             arch        = hints.getProcessArchitecture();
          }
          else
          {
             // use global values
             marker      = UastHints.defaultMarker();
             unixEscapes = UastHints.defaultUnixEscapes();
             batchMode   = UastHints.defaultBatch();
             opsys       = UastHints.defaultOpsys();
             winsys      = UastHints.defaultWinsys();
             debug       = UastHints.defaultPreprocDebug();
             arch        = UastHints.defaultArch();
          }

          options.setMarker(marker);
          options.setUnixEscapes(unixEscapes);
          options.setBatchMode(batchMode);
          options.setDebug(debug);
          options.setProcessArchitecture(arch);

          if (opsys != null)
          {
             options.setOpsys(opsys);
          }

          if (winsys != null)
          {
             options.setWindowSystem(winsys);
          }

          String caseSensStr = Configuration.getParameter(CASE_SENS_KEY);

          if ("false".equals(caseSensStr))
          {
             options.setCaseSensitive(false);
          }

          String fileSep = Configuration.getParameter(FILESEP_KEY, File.separator);
          options.setFileSeparator(fileSep);

          String pathSep = Configuration.getParameter(PATHSEP_KEY,  File.pathSeparator);
          options.setPathSeparator(pathSep);

          options.setBasepath(Configuration.getParameter(ELEM_BASEPATH, "."));

          Map<String, String> includeFilePrefixMap = Configuration.getIncludeFilePrefixMap();
          options.setIncludeFilePrefixMap(includeFilePrefixMap);

          return options;
       }

       /**
        * Gets the PROPATH based on the global configuration and any file-specific configuration
        * provided.  Source-platform path separators will be honored during parsing of the paths.
        *
        * @param    paths
        *           A single string with one or more paths to be searched for Progress files, with
        *           each path separated by the current platform's file separator character.
        *           May be {@code null}.
        * @param    hints
        *           File-specific hints or {@code null} if they don't exist or if there is no specific
        *           source file being processed.
        */
       public static String[] getPropath(String paths, UastHints hints)
       {
          StringBuilder sb = new StringBuilder();

          // honor the passed propath if given
          if (paths != null)
          {
             sb.append(paths);
          }

          String replace    = "";
          String sourceDir  = "";

          int type = UastHints.PATH_MOD_NONE;

          // lookup file-specific overrides
          if (hints != null)
          {
             type      = hints.getPathModType();
             replace   = hints.getPathModTxt();
             sourceDir = hints.getSourceDir();

             if (type == UastHints.PATH_MOD_REPLACE)
             {
                // override with a file specific propath
                sb.setLength(0);
                sb.append(replace);
             }
          }

          // was there either a passed-in or file-specific propath?
          if (sb.length() == 0)
          {
             // [propath] returned by Configuration already uses current platform file separator
             String global = Configuration.getParameter(PROPATH_KEY);

             if (global != null && global.length() > 0)
             {
                // use the global propath
                sb.append(global);
             }
             else
             {
                // fallback plan (manufacture a default propath)
                sb.append('.').append(File.pathSeparator);
             }
          }

          // process file-specific prepend/append cases (the dynamic cases will already have replace
          // set to be the same as sourceDir by the hints object)
          switch (type)
          {
             case UastHints.PATH_MOD_PREPEND:
             case UastHints.PATH_MOD_DYN_PREPEND:
                prependPath(sb, replace);
                break;
             case UastHints.PATH_MOD_APPEND:
             case UastHints.PATH_MOD_DYN_APPEND:
                appendPath(sb, replace);
                break;
          }

          // process global dynamic prepend/append cases
          if (Configuration.getParameter(DYN_PROPATH_PREPEND_KEY) != null)
          {
             prependPath(sb, sourceDir);
          }
          else if (Configuration.getParameter(DYN_PROPATH_APPEND_KEY) != null)
          {
             appendPath(sb, sourceDir);
          }

          // no need to convert file separator chars to the current platform, [propath] returned by
          // Configuration already uses the correct file separator
          String[] res = sb.toString().split(File.pathSeparator);
          // remove duplicated entries
          LinkedHashSet<String> l = new LinkedHashSet<>(Arrays.asList(res));
          res = l.toArray(new String[0]);
          return res;
       }

       /**
        * Gets the state of the silent mode flag indicating if the <code>System.out</code> output
        * should be suppressed.
        *
        * @return   <code>true</code> if silent mode is enabled, otherwise <code>false</code>.
        */
       public synchronized boolean getSilent()
       {
          return silent;
       }

       /**
        * Sets the state of the silent mode flag indicating if the <code>System.out</code> output
        * should be suppressed.
        *
        * @param    silent
        *           <code>true</code> if silent mode is enabled, otherwise <code>false</code>.
        */
       public synchronized void setSilent(boolean silent)
       {
          this.silent = silent;
       }

       /**
        * Sets the state of the flag indicating if the preprocessor should be
        * run.
        *
        * @param    preprocess
        *           <code>true</code> if the preprocessor is to be run, otherwise
        *           <code>false</code>.
        */
       public synchronized void setPreprocess(boolean preprocess)
       {
          this.preprocess = preprocess;
       }

       /**
        * Gets the state of the flag indicating if the preprocessor should be
        * run.
        *
        * @return   <code>true</code> if the preprocessor is to be run, otherwise
        *           <code>false</code>.
        */
       public synchronized boolean getPreprocess()
       {
          return preprocess;
       }

       /**
        * Sets the state of the flag indicating if the preprocessor output should
        * be cached.  This value is dependent upon whether or not the
        * preprocessor is run, if the preprocessor is not run, then this value
        * is meaningless.
        *
        * @param    cache
        *           <code>true</code> if the preprocessor output is to be cached,
        *           otherwise <code>false</code>.
        */
       public synchronized void setCache(boolean cache)
       {
          this.cache = cache;
       }

       /**
        * Gets the state of the flag indicating if the preprocessor output should
        * be cached.  This value is dependent upon whether or not the
        * preprocessor is run, if the preprocessor is not run, then this value
        * is meaningless.
        *
        * @return   <code>true</code> if the preprocessor output is to be cached,
        *           otherwise <code>false</code>.
        */
       public synchronized boolean getCache()
       {
          return cache;
       }

       /**
        * Sets the state of the flag indicating if lexer output should be
        * dumped.
        *
        * @param    dumpLexer
        *           <code>true</code> if lexer output is to be dumped, otherwise
        *           <code>false</code>.
        */
       public synchronized void setDumpLexer(boolean dumpLexer)
       {
          this.dumpLexer = dumpLexer;
       }

       /**
        * Accesses the state of the flag indicating if lexer output should be
        * dumped.
        *
        * @return    <code>true</code> if lexer output is to be dumped, otherwise
        *            <code>false</code>.
        */
       public synchronized boolean getDumpLexer()
       {
          return dumpLexer;
       }

       /**
        * Sets the state of the flag indicating if parser output should be
        * dumped.
        *
        * @param    dumpParser
        *           <code>true</code> if parser output is to be dumped, otherwise
        *           <code>false</code>.
        */
       public synchronized void setDumpParser(boolean dumpParser)
       {
          this.dumpParser = dumpParser;
       }

       /**
        * Accesses the state of the flag indicating if parser output should be
        * dumped.
        *
        * @return    <code>true</code> if parser output is to be dumped,
        *            otherwise <code>false</code>.
        */
       public synchronized boolean getDumpParser()
       {
          return dumpParser;
       }

       /**
        * Sets the state of the flag indicating if a newly parsed AST should be
        * persisted.
        *
        * @param    astPersist
        *           <code>true</code> to persist a new AST, otherwise
        *           <code>false</code>.
        */
       public synchronized void setAstPersist(boolean astPersist)
       {
          this.astPersist = astPersist;
       }

       /**
        * Accesses the state of the flag indicating if new ASTs should be
        * persisted.
        *
        * @return   <code>true</code> to persist a new AST, otherwise
        *           <code>false</code>.
        */
       public synchronized boolean getAstPersist()
       {
          return astPersist;
       }

       /**
        * Sets the PROPATH which defines how Progress source files are found.
        *
        * @param    paths
        *           A single string with one or more paths to be searched for Progress files, with
        *           each path separated by the current platform's file separator character.
        */
       public synchronized void setPaths(String paths)
       {
          this.paths = getPropath(paths, hints);
       }

       /**
        * Accesses the PROPATH as defined by the project configuration or by
        * the user of this class.
        *
        * @return   Array of paths to be searched for Progress files.
        */
       public synchronized String[] getPaths()
       {
          return paths;
       }

       /**
        * Getter for our hints RFB20190911
        *
        * @return   UastHints if any.
        */
       public synchronized UastHints getHints()
       {
          return hints;
       }

       /**
        * Mark the filename's AST as 'original'.  This is used by already parsed .cls files, from pre-scan.
        *
        * @param   filename
        *          Name of the source file to mark.
        */
       public void markOriginal(String filename)
       {
          try
          {
             mgr.lockTree(filename, true);
             long id = mgr.getTreeId(filename);
             mgr.setFlag(id, AstManager.FLAG_ORIGINAL);
          }
          catch (Exception exc)
          {
             throw new AstException("Error marking as original " + filename, exc);
          }
          finally
          {
             mgr.unlockTree(filename);
          }
       }

       /**
        * Create or load an AST given a specific source filename.  If AST
        * persistence is activated, a valid persisted AST will be loaded. If no
        * valid persisted AST exists or if AST persistence is disabled, then the
        * AST will be created.  If creation is used (as opposed to loading a
        * persisted AST), this method will optionally preprocess the Progress
        * source file and will parse the loaded file (preprocessed or not) into
        * an abstract syntax tree.
        * <p>
        * The caller may provide a file list such that duplicate source names
        * can be removed. This is not a generic service, but rather one that is
        * specific to HTML files. In the case of an HTML file that has embedded
        * 4GL (E4GL) code, the E4GL preprocessor will be invoked. This HTML file
        * will then be converted into a normal 4GL source file. Since that source
        * file is generated by the preprocessor, it may overwrite a file that is
        * already there from a previous run (if caching is off or if the cached
        * file is older than the timestamp on the HTML file). Since a valid 4GL
        * source file may already exist in the file system, it is possible that
        * that source file name (with a .p, .w or .i instead of the .html) may
        * also appear in the file list that is used by the caller. Since the
        * caller doesn't know the generated file name in the E4GL case, the
        * pruning of that "duplicate" name is delegated to this method. The file
        * list will be edited to remove any output file name that the E4GL
        * preprocessor generates, if it is separately listed. This avoids
        * duplicate processing of that file, since it was already processed
        * fully to an AST when it was listed as an HTML file, it doesn't need to
        * be processed again later. This processing is dependent upon all the
        * HTML files being processed before any 4GL source files. The
        * {@link #prioritize} method can be used to sort the list properly. Note
        * that any removal is done by setting the duplicate entry to
        * <code>null</code>. The calling code must be prepared to ignore any
        * such <code>null</code> entries.
        * <p>
        * If this file represents a <code>.cls</code> file (a class or interface)
        * it is assumed that the file is not a built-in 4GL class, nor is it a
        * .NET class.
        *
        * @param   filename
        *          Name of the source file to process; if this is not an absolute
        *          path, it should be relative to the propath.
        * @param   list
        *          The list of files upon which the generator is iterating in the
        *          calling method. The list may be edited as noted above. This
        *          may be <code>null</code> if no such editing is needed.
        * @param   idx
        *          Index position of the element being processed in the given
        *          list. Ignored if the list is <code>null</code>.
        *
        * @return  The root AST of the parsed output.
        *
        * @throws  AstException
        *          if the file cannot be found, or if there is some other error
        *          during processing.
        */
       public Aast processFile(String filename, List<File> list, int idx)
       throws AstException
       {
          filename = Configuration.normalizeFilename(filename);

          // remember the original file name
          String original = filename;

          try
          {
             // exclude all other access to this file
             // TODO: doesn't this locking need to be on filename.ast instead of just filename?
             mgr.lockTree(filename, list != null);

             // should we honor a previously persisted AST?
             if (astPersist && !ConversionData.mustConvertFile(filename))
             {
                // check if the AST exists and has a later timestamp than the
                // source file
                if (checkForValidAst(filename))
                {
                   // load the AST and return it
                   return loadPersistedAst(filename);
                }
             }

             // does the filename reference an html file
             if (isHTML(filename))
             {
                // we may have hints to honor for e4gl preprocessing
                String hintsFile = findHintsFile(filename);

                if (hintsFile != null)
                {
                   hints = new UastHints(hintsFile);
                }

                E4GLPreprocessor e4gl = createE4GLPreprocessor(filename);

                // use calc'd output name for downstream usage
                filename = calcOutputFilename(e4gl);

                // drive the web preprocessor
                preprocessE4GL(e4gl, original, filename);

                // rewrite the tree locking filename
                mgr.rewriteTreeName(original, filename);

                // make sure that caller's file list is updated to avoid
                // containing both the html and the generated .p/.w/.i
                if (list != null)
                {
                   removeDuplicate(list, filename);

                   // set the current list element to be the output filename
                   // so that any downstream processing can find the AST
                   list.set(idx, new File(filename));
                }
             }

             // load hints if they are available (must be done before propath
             // processing)
             if (hints == null)
             {
                hints = new UastHints(filename);
             }

             // setup propath from configuration
             setPaths(null);

             Aast root = parse(filename, hints.getBannedDatabases());
             if (astPersist)
             {
                // this is not pre-scan, save the file hash
                ConversionData.saveFileHash(filename);
             }

             return root;
          }

          catch (Exception exc)
          {
             throw new AstException("Error processing " + filename, exc);
          }

          finally
          {
             // clear our file-specific state (this releases these objects from
             // the heap but more importantly it creates a clean slate for the
             // next use of this method)
             paths = null;
             hints = null;

             // clear our lock
             mgr.unlockTree(filename);
          }
       }

       /**
        * Given a 4GL source code filename return the name of the AST file.
        *
        * @param    filename
        *           The original source filename.
        *
        * @return   The target AST filename.
        */
       public static String getAstName(String filename)
       {
          return filename + AST_POSTFIX;
       }

       /**
        * Builds a unique filename based on an original filename and a passed
        * postfix.  This method takes the original filename and appends a fixed
        * value to create a new, unique cache filename.
        *
        * @param    filename
        *           The original source filename.
        * @param    postfix
        *           Text to append to the original filename.
        *
        * @return   The target cache filename.
        */
       private static File createUniqueFile(String filename, String postfix)
       {
          return new File(filename + postfix);
       }

       /**
        * Prepare the data stream which will be used by the parser. The file is
        * first located in the file system. If this object is set to preprocess
        * inline, this is done first, then an input stream is opened on the
        * resulting data buffer. If inline preprocessing is turned off, the input
        * stream is opened directly on the specified file.
        *
        * @param   filename
        *          Name of the source file to process; if this is not an absolute
        *          path, it should be relative to the propath.
        *
        * @return  Parser-ready input stream.
        *
        * @throws  FileNotFoundException
        *          if <code>filename</code> does not exist.
        * @throws  IOException
        *          If a problem occurs during preprocessing or caching.
        */
       private Reader prepareDataStream(String filename)
       throws FileNotFoundException,
              IOException
       {
          File file = Utils.findFileInPath(filename, paths);
          Reader in = null;

          if (preprocess)
          {
             in = preprocess(file);
          }
          else
          {
             InputStream is = new FileInputStream(file);
             Charset charset = hints.getSourceCharset();
             charset = charset == null ? Charset.defaultCharset() : charset;
             in = new BufferedReader(new InputStreamReader(is, charset));
          }

          return in;
       }

       /**
        * Find the UAST hints file associated with the given embedded 4GL (E4GL)
        * program (which is an HTML file). This is done by removing the file
        * extension to get a base name and then searching for a single file with
        * the pattern <code>basename.*.</code> concatinated with the extension
        * defined by {@link UastHints#HINTS_SUFFIX}. If more than 1 file is
        * found, then none will be used since this code has no way of determining
        * what the precedence order would be. Note that there is no known
        * functional reason why more than one hints file would match this
        * specification.
        *
        * @param    filename
        *           The filename on which to base the hints filename.
        *
        * @return   The hints filename (if one and only one exists) or
        *           <code>null</code> if no hints file can be found.
        */
       private String findHintsFile(String filename)
       {
          File   html  = new File(filename);
          String base  = html.getName();
          String lower = base.toLowerCase();

          int len = base.length();

          // calculate the end of the portion of the base name without the
          // extension
          if (lower.endsWith(HTML1_POSTFIX))
          {
             len -= HTML1_POSTFIX.length();
          }
          else if (lower.endsWith(HTML2_POSTFIX))
          {
             len -= HTML2_POSTFIX.length();
          }
          else
          {
             // not an html file
             return null;
          }

          // build our file spec (as a regular expression)
          String sb = base.substring(0, len) + "\\..*\\." + UastHints.HINTS_SUFFIX;

          // determine the containing directory
          String dpath = html.getParent();

          if (dpath == null)
          {
             // fallback to the current directory
             dpath = ".";
          }

          File dir = new File(dpath);

          // get a list of files/sub-directories
          String[] rawResults = dir.list();
          String   spec       = sb;
          String   hintFile   = null;

          for (int i = 0; i < rawResults.length; i++)
          {
             if (rawResults[i].matches(spec))
             {
                if (hintFile == null)
                {
                   hintFile = String.format("%s%s%s",
                                            dpath,
                                            File.separator,
                                            rawResults[i]);
                }
                else
                {
                   // more than one result, this is invalid so "bail out"
                   return null;
                }
             }
          }

          return hintFile;
       }

       /**
        * Manage the process of initializing the E4GL preprocessor including
        * the honoring of hints for <code>wsoptions</code> and the compatibility
        * <code>mode</code>.
        *
        * @param    filename
        *           The HTML file to preprocess.
        *
        * @return   The instance that will be used to preprocess the specified
        *           HTML file.
        */
       private E4GLPreprocessor createE4GLPreprocessor(String filename)
       {
          String modename  = null;
          String wsoptions = null;

          if (hints != null)
          {
             wsoptions = hints.getWSOptions();
             modename  = hints.getE4GLMode();
          }
          else
          {
             // even if there is no hints file that is specific to the given
             // source code, we may still need to honor the global configuration
             wsoptions = Configuration.getParameter(ELEM_WSOPTIONS);
             modename = Configuration.getParameter(ELEM_MODE);
          }

          int mode = DEFAULT_MODE;

          if (modename != null)
          {
             mode = E4GLPreprocessor.decodeMode(modename);

             if (mode == UNKNOWN_MODE)
                mode = DEFAULT_MODE;
          }

          return new E4GLPreprocessor(filename, mode, wsoptions);
       }

       /**
        * Determine the output file name to be used for the current HTML file
        * being processed. If a hint file exists and has an explicit override
        * specified, that will be used. Otherwise, the filename will be
        * calculated based on any encoded options and the original HTML source
        * file name.
        *
        * @param    e4gl
        *           The preprocessor associated with the current file.
        *
        * @return   The calculated name.
        */
       private String calcOutputFilename(E4GLPreprocessor e4gl)
       {
          String output = null;

          // force any encoded options to be picked up
          e4gl.scanForWSOptions();

          if (hints != null)
          {
             // honor any specified override (even if there are hints, the
             // output file name may still be null)
             output = hints.getE4GLOutputFile();
          }

          // calculate output name if needed
          return output != null ? output : e4gl.calculateOutputDest();
       }

       /**
        * Perform inline preprocessing of the specified source file.
        *
        * @param   file
        *          File to be preprocessed.
        *
        * @return  Input source connected to the beginning of the data buffer
        *          created by the {@link com.goldencode.p2j.preproc.Preprocessor}.
        *
        * @throws  FileNotFoundException
        *          if <code>filename</code> does not exist.
        * @throws  IOException
        *          during a write or close failure.
        */
       private Reader preprocess(File file)
       throws  FileNotFoundException,
               IOException
       {
          boolean alreadyCached = false;
          File    cacheFile     = null;

          // Target output stream.
          CharArrayWriter os = new CharArrayWriter();

          if (cache)
          {
             cacheFile = createUniqueFile(file.getAbsolutePath(),
                                          CACHE_POSTFIX);

             if (cacheFile.exists() &&
                 cacheFile.lastModified() > file.lastModified())
             {
                alreadyCached = true;
             }
          }

          if (!alreadyCached)
          {
             // If caching is NOT in effect or if the cache file is out of date,
             // activate the preprocessor.

             // Set up options.
             Options options = buildOptions(paths, file, hints);

             // Implement preprocessor runtime arguments, if they exist.
             // It is OK for this to be null in the FileScope constructor.
             Map       args = hints.getArguments();
             FileScope fs   = new FileScope(file.getAbsolutePath(), args, options, false);
             fs.setCharset(hints.getSourceCharset());

             // Run preprocessor.
             try
             {
                new Preprocessor(fs, os, err, warn, options);
             }
             catch (PreprocessorException exc)
             {
                LOG.log(Level.SEVERE, "", exc);
             }

             // Implement caching if enabled - this works because the Preprocessor
             // constructor doesn't return until all preprocessing is complete
             // AND because we can get a copy of the output more than once so
             // this code doesn't conflict with other downstream code that uses
             // the preprocessor output (we can be inserted transparently).
             if (cache)
             {
                // If this filename exists as a directory, we can't cache.
                if (!cacheFile.isDirectory())
                {
                   // Remove the target file if it exists.
                   if (cacheFile.exists())
                   {
                      cacheFile.delete();
                   }

                   OutputStream fos = new FileOutputStream(cacheFile);
                   Writer out = new BufferedWriter(new OutputStreamWriter(fos, hints.getSourceCharset()));
                   // Cache the output and close the cache file.
                   out.write(os.toCharArray());
                   out.close();
                }
             }
          }
          else
          {
             // Take the cached version as input.
             Reader in   = new BufferedReader(new InputStreamReader(new FileInputStream(cacheFile),
                                                                    hints.getSourceCharset()));

             try
             {
                final int BUFF_SIZE = 8192;
                char[] chrs = new char[BUFF_SIZE];
                int read  = 0;

                while (read >= 0)
                {
                   read = in.read(chrs, 0, BUFF_SIZE);
                   if (read > 0)
                   {
                      os.write(chrs, 0, read);
                   }
                }
             }
             finally
             {
                in.close();
             }
          }

          // Return reader with the preprocessed results.
          return new CharArrayReader(os.toCharArray());
       }

       /**
        * Perform E4GL preprocessing of the specified source HTML file into the
        * given 4GL output file.
        *
        * @param    e4gl
        *           The file-specific preprocessor instance.
        * @param    source
        *           HTML source file.
        * @param    target
        *           4GL output file.
        */
       private void preprocessE4GL(E4GLPreprocessor e4gl,
                                   String           source,
                                   String           target)
       {
          boolean alreadyCached = false;

          if (cache)
          {
             File sourceFile = new File(source);
             File cacheFile  = new File(target);

             if (cacheFile.exists() &&
                 cacheFile.lastModified() > sourceFile.lastModified())
             {
                alreadyCached = true;
             }
          }

          if (!alreadyCached)
          {
             // if caching is NOT in effect or if the cache file is out of date,
             // activate the preprocessor, otherwise let the downstream code use
             // the file that already exists

             // run the E4GL preprocessor
             try
             {
                // the result will be cached automatically
                e4gl.preprocess(false, null);
             }
             catch (Exception exc)
             {
                LOG.log(Level.SEVERE, "", exc);
             }
          }
       }

       /**
        * Search the list to find a duplicate to the given name and remove it
        * if found. Once the first matching entry is found, it will be set to
        * <code>null</code> and the method will return.
        *
        * @param    list
        *           Array to edit.
        * @param    duplicate
        *           Name of the file to remove. This may be <code>null</code>
        *           but then why would you call this method?
        */
       private void removeDuplicate(List<File> list, String duplicate)
       {
          int size = list.size();
          for (int i = 0; i < size; i++)
          {
             File file = list.get(i);
             if (file != null && file.toString().equals(duplicate))
             {
                list.set(i, null);
                return;
             }
          }
       }

       /**
        * Performs first-level scanning of OO classes.
        *
        * @param   filename
        *          Name of the source file to process; if this is not an absolute
        *          path, it should be relative to the propath.
        * @param   builtin
        *          <code>true</code> if this class file represents a built-in class or interface.
        * @param   dotnet
        *          <code>true</code> if this class file represents a .NET class or interface.
        * @param   propath
        *          The current <code>PROPATH</code> to honor.  May not be <code>null</code>.
        *
        * @throws  AstException
        *          if the file cannot be found, or if there is some other error during processing.
        */
       void preScanClass(String filename, boolean builtin, boolean dotnet, String[] propath)
       throws AstException,
              ANTLRException,
              IOException
       {
          filename = Configuration.normalizeFilename(filename);

          boolean caseSens = Configuration.getParameter(CASE_SENS_KEY, true);

          UastHints saveHints = hints;
          String[] savePaths = paths;

          try
          {
             // exclude all other access to this file
             mgr.lockTree(filename, false);

             // TODO: hints will not work, as this is called recursive... need a stack approach
             hints = new UastHints(filename);

             this.paths = propath;

             SymbolResolver sym = new SymbolResolver(true,
                                                     caseSens,
                                                     propath,
                                                     silent,
                                                     true,
                                                     hints.getBannedDatabases());
             processParserHints(sym);

             // class definitions require a 2-pass run, where the 1st pass preloads
             // resources that can be accessed before their definitions occur on the
             // 2nd pass (e.g. an instance method can be called in an expression
             // earlier in the file than where that method's definition appears)

             ProgressLexer  lexer  = prepareLexer(filename, sym);
             ProgressParser parser = null;

             parser = new ProgressParser(lexer, sym);
             parser.setFilename(filename);
             parser.setBuiltInCls(builtin);
             parser.setDotNetCls(dotnet);
             parser.pre_scan_class();
          }

          catch (ParentUnavailableException e)
          {
             // let the exception propagate, everything will be re-parsed when this same class is re-parsed.
             throw e;
          }

          catch (Exception exc)
          {
             throw new AstException("Error processing " + filename, exc);
          }

          finally
          {
             // clear our file-specific state (this releases these objects from
             // the heap but more importantly it creates a clean slate for the
             // next use of this method)
             paths = savePaths;
             hints = saveHints;

             // clear our lock
             mgr.unlockTree(filename);
          }

          // if everything was completed OK and this is a built-in class, do a full parse to save
          // the AST
          if (builtin)
          {
             boolean oldAstPersist = this.astPersist;
             this.astPersist = true;
             try
             {
                processFile(filename, null, -1);
             }
             finally
             {
                this.astPersist = oldAstPersist;
             }
          }
       }

       /**
        * Parse the source code read from the specified filename into an abstract
        * syntax tree. Uses the Progress parser and its helpers to do this work.
        *
        * @param   filename
        *          The name of the file being processed.
        *
        * @return  Root of the AST created by the parser.
        *
        * @throws  ANTLRException
        *          if any error occurs while lexing or parsing the source code.
        */
       private Aast parse(String filename, Set<String> bannedSchemas)
       throws AstException,
              ANTLRException,
              IOException
       {
          boolean caseSens = Configuration.getParameter(CASE_SENS_KEY, true);

          SymbolResolver sym = new SymbolResolver(true, caseSens, paths, silent, true, bannedSchemas);

          processParserHints(sym);

          // the very first time we try to parse a class, we must ensure that the critical built-in classes
          // are already loaded; without this some classes (enums especially) can cause a parsing failure
          // during pre-scanning which is avoided when these same classes are already loaded
          if (filename.endsWith(SymbolResolver.CLASS_EXT))
          {
             synchronized (AstGenerator.class)
             {
                if (!earlyClassInit)
                {
                   earlyClassInit = true;
                   sym.loadClass(SymbolResolver.ROOT_OBJ_NAME);
                }
             }
          }

          ProgressLexer  lexer  = prepareLexer(filename, sym);
          ProgressParser parser = null;

          parser = new ProgressParser(lexer, sym);
          parser.setFilename(filename);

          // Store off our AstGenerator instance RFB20190911
          parser.setGenerator(this);

          boolean complete = parser.external_proc();

          if (!complete)
          {
             Token  next  = lexer.nextToken();
             LOG.log(Level.WARNING,
                     String.format("Parser did NOT process to EOF "        +
                                      "(no match with token immediately BEFORE %s:%s " +
                                      "in file '%s' with token text '%s').\n",
                                   next.getLine(),
                                   next.getColumn(),
                                   filename,
                                   next.getText()));
          }

          // get the resulting AST
          AnnotatedAst result = (AnnotatedAst) parser.getAST();

          // makes sure that parents and IDs are properly assigned (this is a
          // form of post-processing)
          result.brainwash(filename, true);

          if (result.downPath(ProgressParser.CLASS_DEF)     ||
              result.downPath(ProgressParser.INTERFACE_DEF) ||
              result.downPath(ProgressParser.ENUM_DEF))
          {
             // this is a class - save the AST IDs where is required
             boolean isClass = result.downPath(ProgressParser.CLASS_DEF);
             boolean isEnum  = result.downPath(ProgressParser.ENUM_DEF);
             boolean isIface = result.downPath(ProgressParser.INTERFACE_DEF);
             int tok = isClass ? ProgressParser.CLASS_DEF
                               : isIface ? ProgressParser.INTERFACE_DEF
                                         : ProgressParser.ENUM_DEF;
             Aast ref = result.getImmediateChild(tok, null);
             tok = isClass ? ProgressParser.KW_CLASS
                           : isIface ? ProgressParser.KW_INTERFAC
                                     : ProgressParser.KW_ENUM;
             ref = ref.getImmediateChild(tok, null);
             ref = ref.getImmediateChild(ProgressParser.SYMBOL, null);
             String qname = ref.getText();
             ClassDefinition cdef = SymbolResolver.getClassDefinition(qname);
             cdef.parseFinished(false);
          }
          // save off output if requested, THIS MUST BE DONE AFTER FIXUPS!
          if (dumpParser)
             generateParseReport(filename, result);

          if (astPersist)
          {
             persistAst(filename, result);

             // persist schema dictionary state
             File schemaFile = createUniqueFile(filename, DICT_POSTFIX);
             try
             {
                sym.persistSchemaData(filename, schemaFile.getAbsolutePath());
             }
             catch (Exception exc)
             {
                throw new AstException(
                   "Error persisting schema dictionary state to " + schemaFile,
                   exc);
             }
          }

          int errCount = parser.getErrorCount();
          if (errCount > 0)
          {
             throw new RuntimeException("Parser encountered " + errCount + " errors");
          }

          return result;
       }

       /**
        * Prepare a lexer instance for the given file using the supporting
        * resolver instance as needed.
        *
        * @param    filename
        *           The file being lexed.
        * @param    sym
        *           The symbol resolver instance being used for this file.
        *
        * @return   A token stream source for the given file.
        */
       private ProgressLexer prepareLexer(String filename, SymbolResolver sym)
       throws IOException
       {
          // preprocess file or load source from cache
          Reader in = prepareDataStream(filename);

          ProgressLexer lexer = null;
          if (dumpLexer)
          {
             File lexerFile = createUniqueFile(filename, LEXER_POSTFIX);

             if (lexerFile.exists() && lexerFile.isFile())
             {
                lexerFile.delete();
             }

             lexer = new LexerDumpFilter(in, sym, lexerFile);
          }
          else
          {
             lexer = new ProgressLexer(in, sym);
          }

          // keyword ignore support must be implemented after the lexer is
          // instantiated since the keyword processing is not finished
          // initializing until that point
          String ignore = Configuration.getParameter(IGNORE_KEY);

          if (ignore != null)
          {
             String[] list = readIntoArray(ignore);

             if (list != null)
             {
                lexer.ignoreKeywords(list);
             }
          }

          // force the escape mode
          lexer.setUnixEscapes(hints.isUnixEscapes());

          return lexer;
       }

       /**
        * Drive the process by which the hints file is queried for parser
        * related hints, any hints found are implemented.
        *
        * @param    sym
        *           Provides access to the symbol namespaces such that hints can
        *           be loaded.
        */
       private void processParserHints(SymbolResolver sym)
       {
          // if databases need to be connected before parsing, we load them here
          sym.loadSchemaDatabases(hints.getDatabases());

          // if aliases need to be added before parsing, we load them here; NOTE
          // that this MUST be done AFTER database loads
          List<String[]> aliases = hints.getAliases();
          if (aliases != null)
          {
             sym.loadAliases(aliases);
          }

          // if variables/widgets need to be added to the dictionary, do this now
          Variable[] varlist = hints.getVariables();
          if (varlist != null)
          {
             sym.loadVariableSymbols(varlist);
          }
       }

       /**
        * Read a simple text file into an array where each line in the input file
        * is a separate element in the output array.
        *
        * @param    filename
        *           Name of the text file to read.
        *
        * @return   Array of text that was read or <code>null</code> if there
        *           was any error.
        */
       private String[] readIntoArray(String filename)
       {
          String[] out = new String[0];

          try
          {
             ArrayList<String> list = new ArrayList<>();

             try (BufferedReader in = new BufferedReader(new FileReader(filename)))
             {
                String next = in.readLine();

                while (next != null)
                {
                   list.add(next);
                   next = in.readLine();
                }

                out = list.toArray(out);
             }
          }

          catch (Exception exc)
          {
             LOG.log(Level.WARNING,"", exc);
          }

          return out;
       }

       /**
        * Calculates a unique stream name for storing the output, based on the
        * input file name and a postfix.  Takes the input filename (including the
        * absolute path) and appends the a user-defined postfix to it.  As long
        * as the resulting name is not a directory, it is deleted if it exists
        * and the new stream is then returned.
        *
        * @param    original
        *           Input file name.
        *
        * @return   An open stream on which to write the parser output.
        */
       private PrintStream createUniqueStream(String original, String postfix)
       {
          PrintStream out = null;

          if (original != null)
          {
             try
             {
                // build a unique filename based on the original
                File parse = createUniqueFile(original, postfix);

                // if this filename exists as a directory, we can't save results
                if (!parse.isDirectory())
                {
                   // remove the target file if it exists.
                   if (parse.exists())
                   {
                      parse.delete();
                   }

                   out = new PrintStream( new FileOutputStream(parse) );
                }
             }
             catch (Exception excpt)
             {
                LOG.log(Level.WARNING,"", excpt);
             }
          }

          return out;
       }

       /**
        * Dump an AST into a file with a human readable report format. A filename
        * is generated off the original source file's name, with a standard
        * extension of <code>PARSER_POSTFIX</code>.
        *
        * @param    filename
        *           Original source file being processed.
        * @param    ast
        *           The AST to dump.
        */
       private void generateParseReport(String filename, Aast ast)
       {
          PrintStream out     = createUniqueStream(filename, PARSER_POSTFIX);
          DumpTree    visitor = new DumpTree(out);

          visitor.visit(ast);
          out.close();
       }

       /**
        * Persist an AST into an XML file which can be turned into a valid AST
        * using <code>AstPersister</code>. A filename is generated off the
        * original source file's name, with a standard extension of
        * <code>AST_POSTFIX</code>.
        *
        * @param    filename
        *           Original source file being processed.
        * @param    ast
        *           The AST to dump.
        */
       private void persistAst(String filename, Aast ast)
       throws AstException
       {
          File   target = createUniqueFile(filename, AST_POSTFIX);
          String fname  = null;

          try
          {
             fname = target.getCanonicalPath();
          }

          catch (IOException ioe)
          {
             throw new AstException("Invalid filename " + filename, ioe);
          }

          AstManager.get().saveTree(ast, fname, false);
       }

       /**
        * Checks whether or not the source filename has a valid persisted AST.
        * Validity is determined based on the fact that the AST exists AND that
        * it's timestamp is later than that of the source file.
        *
        * @param    filename
        *           The source filename to check for a matching persisted AST.
        *
        * @return   <code>true</code> if a persisted AST exists AND that AST
        *           has a timestamp later than that of the source file,
        *           <code>false</code> otherwise.
        */
       private boolean checkForValidAst(String filename)
       {
          File src = new File(filename);
          File ast = createUniqueFile(filename, AST_POSTFIX);

          return ast.exists() && ( ast.lastModified() > src.lastModified() );
       }

       /**
        * Loads a previously persisted AST for a specified source file.
        *
        * @param    filename
        *           The source filename for which a matching persisted AST will
        *           be loaded.
        *
        * @return   The AST associated with the specified source file.
        *
        * @throws   AstException
        *           if there are any problems with loading the AST.
        */
       private Aast loadPersistedAst(String filename)
       throws AstException
       {
          File source = createUniqueFile(filename, AST_POSTFIX);

          String fname  = null;

          try
          {
             fname = source.getCanonicalPath();
          }

          catch (IOException ioe)
          {
             throw new AstException("Invalid filename " + filename, ioe);
          }

          return AstManager.get().loadTree(fname);
       }

       /**
        * Prepend the given text to the path list in the buffer, optionally
        * separating the path elements with the file system path separator if
        * a path separator is not already present.
        *
        * @param    sb
        *           Buffer containing current path list.
        * @param    txt
        *           Path element to prepend (insert at position 0).
        */
       private static void prependPath(StringBuilder sb, String txt)
       {
          if (sb.charAt(0) != File.pathSeparatorChar)
          {
             sb.insert(0, File.pathSeparatorChar);
          }

          sb.insert(0, txt);
       }

       /**
        * Append the given text to the path list in the buffer, optionally
        * separating the path elements with the file system path separator if
        * a path separator is not already present.
        *
        * @param    sb
        *           Buffer containing current path list.
        * @param    txt
        *           Path element to append.
        */
       private static void appendPath(StringBuilder sb, String txt)
       {
          int i = sb.length() - 1;

          if (sb.charAt(i) != File.pathSeparatorChar)
          {
             sb.append(File.pathSeparatorChar);
          }

          sb.append(txt);
       }
    }