# PatternEngine.java

    /*
    ** Module   : PatternEngine.java
    ** Abstract : Main driver for the pattern rules framework
    **
    ** Copyright (c) 2005-2023, Golden Code Development Corporation.
    **
    ** -#- -I- --Date-- --JPRM-- -----------------------------------Description-----------------------------------
    ** 001 ECF 20050301   @20131 Created initial version. Provides a command
    **                           line interface and API to apply a pipeline of
    **                           rulesets against ASTs.
    ** 002 ECF 20050302   @20150 Fixed defect in finish method. Made sure
    **                           resolver has no ASTs in scope when global,
    **                           post-rules are applied.
    ** 003 ECF 20050310   @20255 Added -Q command line option and quiet mode
    **                           support. When not in quiet mode, engine now
    **                           reports to stdout the name of each source
    **                           file as it is processed.
    ** 004 GES 20050310   @20261 Added new default workers for Dictionary
    **                           and FileOperations.
    ** 005 ECF 20050310   @20264 Added support for global includes of named
    **                           expression libraries. Previously these were
    **                           only possible at the ruleset level.
    ** 006 ECF 20050310   @20275 Added support for literal resolution.
    **                           AstSymbolResolver uses this support to
    **                           resolve literals for the expression parser.
    ** 007 GES 20050311   @20294 Added a more generic run() method that
    **                           takes both a profile and a caller created
    **                           iterator. The original run method is left
    **                           alone because it MUST implement the
    **                           initialize() call because this is how the
    **                           astSpecs get initialized. A better approach
    **                           would be to create a more generic, common
    **                           run method but this would require more
    **                           significant changes to the configuration
    **                           loading.  Instead we maintain the old
    **                           approach (unchanged) while opening up a new
    **                           option for external callers of the pattern
    **                           engine.  The new option allows the iterator
    **                           to be created and maintained elsewhere which
    **                           allows for dynamic modifications of that
    **                           iterator while inside the pattern engine's
    **                           core run() loop that calls processAst().
    ** 008 ECF 20050314   @20329 Added support for user variables. Variables
    **                           have one of two scopes: global (for variables
    **                           defined in the main, pipeline profile) and
    **                           local (for variables defined within an
    **                           individual ruleset). Default values may be
    **                           overridden at the command line.
    ** 009 ECF 20050315   @20340 Added astSpecs method. Provides an iterator
    **                           over the FileList objects used to find ASTs.
    ** 010 ECF 20050315   @20344 Added callback to notify pattern worker just
    **                           before a new AST is processed. Calls visitAst
    **                           method for each pattern worker.
    ** 011 ECF 20050315   @20408 Call graphs (really trees) are constructed
    **                           from persistence, based on either the root
    **                           node list or AST specs from the profile or
    **                           command line. Each call tree is walked and
    **                           two new lists of rules (ascent-rules and
    **                           descent-rules) can be defined, both at the
    **                           global and ruleset levels, to trigger actions
    **                           based upon changes in AST scope during the
    **                           walk.
    ** 012 GES 20050323   @20474 Converted use of AnnotatedAst copy
    **                           constructor to the use of the Aast duplicate
    **                           method, which does the same job but allows
    **                           for better abstraction.
    ** 013 ECF 20050328   @20518 Changed references to "target" AST to "copy"
    **                           AST, to reflect similar changes in the symbol
    **                           resolver implementation. Added a call to
    **                           resolver's cleanUp method after a ruleset has
    **                           processed an AST hierarchy, to enable
    **                           resolver to clean up internal resources.
    ** 014 ECF 20050415   @20741 Removed use of AstGenerator. Use AstPersister
    **                           instead to allow ASTs of any kind to be
    **                           processed. AST specs refer to persisted AST
    **                           files directly, rather than indirectly via
    **                           source code files.
    ** 015 ECF 20050422   @20852 Introduced pattern worker namespaces. Changed
    **                           resolveToLiteral method to accept and use an
    **                           optional namespace parameter. Changed pattern
    **                           worker registration methods to also use an
    **                           optional namespace parameter. Namespaces
    **                           enable us to prevent collisions between
    **                           pattern workers when resolving constants, if
    **                           multiple workers can resolve a constant to
    **                           potentially different literals.
    ** 016 ECF 20050423   @20858 Disallow registration of multiple instances
    **                           of the same pattern worker. If an instance of
    **                           a particular pattern worker class is already
    **                           registered, subsequent attempts to register
    **                           that worker type are silently ignored. This
    **                           allows individual rulesets to load pattern
    **                           workers as necessary, without overlap.
    ** 017 GES 20050428   @20919 Added global next-child rules when walking
    **                           call graphs. These rules will be triggered
    **                           with the parent node in scope when moving
    **                           between children (but not on the move to
    **                           the first child or from the last child).
    ** 018 ECF 20050430   @20966 Changes related to namespace support for user
    **                           functions and variables. Moved all pattern
    **                           worker management to AstSymbolResolver.
    **                           Constant resolution is now also handled by
    **                           the resolver. Configuration of the resolver
    **                           is now handled at the superclass level.
    ** 019 ECF 20050526   @21300 Changes to support new expression engine
    **                           implementation. Reimplemented variable
    **                           registration to account for new variable
    **                           initializers feature. Modified command line
    **                           arguments to allow for a user-defined debug
    **                           reporting level. Ensure default pattern
    **                           workers are registered under valid names.
    ** 020 SIY 20050531   @21336 Exclude non-existent files from the
    **                           processing. Some other minor cleanups.
    ** 021 ECF 20050601   @21344 Don't add AST spec if fileSpec is an empty
    **                           string. This triggered default behavior in
    **                           FileList which searches for everything under
    **                           the specified directory.
    ** 022 GES 20050602   @21380 Moved some processing (keeping a list of
    **                           RuleSets) into RuleContainer to provide for
    **                           arbitrary nesting of rule sets.
    ** 023 ECF 20050616   @21503 Refactored initialize, run, and finish
    **                           methods. The methods initialize and finish
    **                           are now public and no longer apply global
    **                           init and post rules. The version of the run
    **                           method which accepted a file iterator is now
    **                           renamed processFiles. This method applies
    **                           global init and post rules directly (as does
    **                           the remaining run method). The processFiles
    **                           method is designed to be safely run multiple
    **                           times between bracketed (external) calls to
    **                           initialize and finish. The run method remains
    **                           as a convenience to bundle initialization,
    **                           AST processing and termination processing for
    **                           a single set of ASTs.
    ** 024 ECF 20050716   @21733 Added support for ruleset maximum walk depth.
    **                           This is an optimization which avoids visiting
    **                           unnecessary levels of an AST when applying a
    **                           specific ruleset.
    ** 025 GES 20050728   @21900 Added support for the global setting of
    **                           honoring the hidden flag of an AST during
    **                           the walk.  Defaults to false, but when
    **                           enabled, non-root AST nodes that are accessed
    **                           via a non-view iterator will drop from the
    **                           walk if their hidden flag is set.
    ** 026 ECF 20050811   @22086 Added support for read-only mode. When in
    **                           read-only mode, the AST walker is handed the
    **                           source AST as both its source and copy AST
    **                           (i.e., there is no copy). Also, the copy ID
    **                           map is empty. This mode gives us a moderate
    **                           performance enhancement when running rulesets
    **                           which do not modify and persist the copy AST
    **                           as part of their work (e.g., reporting).
    ** 027 ECF 20050819   @22173 Prevent duplicate AST processing and permit
    **                           premature termination of a walk. A set of
    **                           processed AST IDs is maintained in call graph
    **                           mode to prevent the same AST branch from
    **                           being processed twice. Also, a new flag in
    **                           the symbol resolver indicates whether a walk
    **                           should be ended prematurely. This is set by
    **                           CommonAstSupport and checked and reset during
    **                           processAst in the pattern engine.
    ** 028 GES 20051005   @22976 Added an explicit file list mode to the
    **                           command line interface.
    ** 029 ECF 20051201   @23760 Modified read only mode. AstRegistry is only
    **                           saved when not in read only mode.
    ** 030 GES 20060129   @24133 FileList related changes.
    ** 031 GES 20060129   @24139 Simple change that allows multiple different
    **                           pattern engines to sequentially run.
    ** 032 ECF 20060421   @25644 New debug feature. Any uncaught error thrown
    **                           within the run() method now triggers a report
    **                           about the most recently active rule.
    ** 033 GES 20070511   @33477 Exposed variable initialization as a public
    **                           method.
    ** 034 ECF 20080730   @39269 Reduced memory footprint. Ensure rule list
    **                           elements are cleaned up after a run. Ensure
    **                           non-global ruleset variables are cleared
    **                           after the ruleset has been applied. Release
    **                           resources to allow GC after a run.
    ** 035 GES 20090421   @41827 Matched utility class package change.
    ** 036 GES 20090429   @42048 Match package and class name changes.
    ** 037 GES 20090515   @42213 Moved to AstManager from AstRegistry/AstPersister.
    ** 038 GES 20130203          Moved to using an Initializer class to allow a
    **                           simple instance to be assigned to a new variable
    **                           in addition to the previous expression approach.
    ** 039 SVL 20130624          Allow pattern engine to manipulate ASTs in memory rather than
    **                           read/write them using file system.
    ** 040 CA  20131022          Added conditional rule-set processing.
    ** 041 OM  20140122          Moved finish() into a finally to be sure the taken resources are
    **                           freed when leaving the run(String profile).
    ** 042 CA  20140313          Changed callgraph generation to use a callgraph DB.
    ** 043 SVL 20140320          Minor improvement to error messages.
    ** 044 CA  20140408          Fixed filename matching for source-code originating from a different
    **                           OS than the one where conversion is ran.
    ** 045 ECF 20140403          Added special mode and c'tor for runtime conversion use; implemented
    **                           additional generics.
    ** 046 ECF 20150331          Added variant of storeObject to allow passing of ConversionPool task
    **                           results to the next pattern engine in a pipeline.
    ** 047 ECF 20150413          Very minor format cleanup.
    ** 048 ECF 20150715          Replace StringBuffer with StringBuilder.
    ** 049 EVL 20160225          Javadoc fixes to make compatible with Oracle Java 8 for Solaris 10.
    ** 050 GES 20170509          Changes for call graph v3.
    **     ECF 20170607          Use TreeWalkException in place of RuntimeException.
    ** 051 CA  20170825          Added a mechanism to inject new ASTs into the source list, generated
    **                           by the TRPL rules.
    ** 052 ECF 20171228          Improve performance by reducing context-local lookups.
    ** 053 ECF 20190620          EmptyIterator API change.
    ** 054 OM  20210923          Added -profile command line option for specifying the configuration profile.
    **     CA  20211214          Fixed cleanup in case the conversion abends.
    **     OM  20211215          Print the descriptive text of the root node if file name is not available.
    **     OM  20220405          XmlFilePlugin can load resources from application's jar.
    **     CA  20221010          Avoid using iterators on ArrayList - instead, use a 'for' loop and get the
    **                           element by index, as iterators are more expensive in terms of memory and
    **                           performance.  Refs #6813
    **     TJD 20220504          Upgrade do Java 11 minor changes
    **     CA  20220501          Each profile has the same structure as the main 'global' config.  Allow multiple
    **                           profiles to be ran at once, with the conversion switching the state between
    **                           profiles, when a resource (like a file or namespace) is being processed.  Only
    **                           front phase is supported at this time.
    **     ECF 20220515          Added PatternWorker.leaveAst hook to allow for post-AST walk processing.
    **     CA  20221010          Avoid using iterators on ArrayList - instead, use a 'for' loop and get the
    **                           element by index, as iterators are more expensive in terms of memory and
    **                           performance.  Refs #6813
    **     SVL 20230108          Improved performance by replacing some "for-each" loops with indexed "for" loops.
    ** 055 GBB 20230512          Logging methods replaced by CentralLogger/ConversionStatus.
    ** 056 CA  20231129          Small changes to avoid 'invokeinterface' in the compiled bytecode.
    */

    /*
    ** This program is free software: you can redistribute it and/or modify
    ** it under the terms of the GNU Affero General Public License as
    ** published by the Free Software Foundation, either version 3 of the
    ** License, or (at your option) any later version.
    **
    ** This program is distributed in the hope that it will be useful,
    ** but WITHOUT ANY WARRANTY; without even the implied warranty of
    ** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    ** GNU Affero General Public License for more details.
    **
    ** You may find a copy of the GNU Affero GPL version 3 at the following
    ** location: https://www.gnu.org/licenses/agpl-3.0.en.html
    **
    ** Additional terms under GNU Affero GPL version 3 section 7:
    **
    **   Under Section 7 of the GNU Affero GPL version 3, the following additional
    **   terms apply to the works covered under the License.  These additional terms
    **   are non-permissive additional terms allowed under Section 7 of the GNU
    **   Affero GPL version 3 and may not be removed by you.
    **
    **   0. Attribution Requirement.
    **
    **     You must preserve all legal notices or author attributions in the covered
    **     work or Appropriate Legal Notices displayed by works containing the covered
    **     work.  You may not remove from the covered work any author or developer
    **     credit already included within the covered work.
    **
    **   1. No License To Use Trademarks.
    **
    **     This license does not grant any license or rights to use the trademarks
    **     Golden Code, FWD, any Golden Code or FWD logo, or any other trademarks
    **     of Golden Code Development Corporation. You are not authorized to use the
    **     name Golden Code, FWD, or the names of any author or contributor, for
    **     publicity purposes without written authorization.
    **
    **   2. No Misrepresentation of Affiliation.
    **
    **     You may not represent yourself as Golden Code Development Corporation or FWD.
    **
    **     You may not represent yourself for publicity purposes as associated with
    **     Golden Code Development Corporation, FWD, or any author or contributor to
    **     the covered work, without written authorization.
    **
    **   3. No Misrepresentation of Source or Origin.
    **
    **     You may not represent the covered work as solely your work.  All modified
    **     versions of the covered work must be marked in a reasonable way to make it
    **     clear that the modified work is not originating from Golden Code Development
    **     Corporation or FWD.  All modified versions must contain the notices of
    **     attribution required in this license.
    */

    package com.goldencode.p2j.pattern;

    import java.io.*;
    import java.text.*;
    import java.util.*;
    import java.util.logging.*;

    import com.goldencode.ast.*;
    import com.goldencode.p2j.cfg.*;
    import com.goldencode.p2j.convert.*;
    import com.goldencode.util.*;
    import com.goldencode.p2j.uast.*;
    import com.goldencode.p2j.util.*;
    import org.apache.tinkerpop.gremlin.structure.*;
    import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.*;

    /**
     * An engine which applies rules to the nodes of an abstract syntax tree
     * (AST), in order to analyze or transform it, based upon patterns detected
     * in the AST data. Rules are organized into rulesets, and rulesets are
     * organized into a pipeline. Pipelines are applied against one or more
     * AST hierarchies, which are ultimately created by either:
     * <ul>
     * <li>deriving them from source code files written using the well-defined
     *     syntax of a programming language; or
     * <li>building them up dynamically in memory, based upon some well-defined
     *     structure.
     * </ul>
     * A pattern engine profile is used to organize the information managed by
     * the pattern engine for a particular application. A profile defines:
     * <ul>
     * <li>the worker objects used to support the desired application;
     * <li>the ASTs to be processed and the order in which they are to be
     *     processed;
     * <li>the pipeline of rulesets to be applied against the ASTs, and the order
     *     in which they are to be applied;
     * <li>the rules contained in each ruleset, and the timing and order in which
     *     these rules are to be applied against the ASTs;
     * <li>the conditions defined for each rule;
     * <li>the list of actions to be executed conditionally for each rule.
     * </ul>
     * A {@link ConfigLoader} object is used to load a profile into the engine,
     * or a profile may be configured programatically using the public methods
     * of this class.
     * <p>
     * A profile can be defined to perform a variety of analysis or transformation
     * applications. This is done via the {@link #run} method. The logical program
     * flow (assuming a pre-defined profile is being used) is as follows:
     * <ul>
     *  <li>begin
     *  <li>perform global initialization processing:
     *  <ul>
     *   <li>create a new symbol resolver
     *   <li>initialize and register default worker objects
     *   <li>load the pattern configuration profile
     *   <li>set a default specification to find ASTs if not provided by profile
     *   <li>apply global, user-defined init-rules
     *  </ul>
     *  <li>for each AST to be processed:
     *  <ul>
     *   <li>for each ruleset defined in the pipeline:
     *   <ul>
     *    <li>make a deep copy of the source AST hierarchy; this copy becomes
     *        the editable source copy for the current ruleset pass and becomes
     *        the source AST for the next ruleset in the pipeline
     *    <li>perform ruleset-specific initialization processing:
     *    <ul>
     *     <li>configure the symbol resolver for the current ruleset
     *     <li>set the source and copy root ASTs as the current ASTs in the
     *         symbol resolver
     *     <li>apply user-defined init-rules for the ruleset
     *    </ul>
     *    <li>walk the AST (or the filtered view of results of previous ruleset),
     *        applying all walk-rules of the ruleset to the current AST
     *    <li>perform ruleset-specific termination processing:
     *    <ul>
     *     <li>set the source and copy root ASTs as the current ASTs in the
     *         symbol resolver
     *     <li>apply user-defined post-rules for the ruleset
     *    </ul>
     *    <li>use the copy AST from the last ruleset as the source AST for the
     *        next ruleset
     *   </ul>
     *  </ul>
     *  <li>perform global termination processing:
     *  <ul>
     *   <li>apply global, user-defined post-rules
     *   <li>call each registered worker object's termination processing hook
     *   <li>null out the symbol resolver
     *  </ul>
     *  <li>end
     * </ul>
     * <p>
     * In cases where multiple file lists must be processed using a single
     * profile, the caller may choose to use the following idiom instead:
     * <ol>
     * <li>call {@link #initialize} <b>once only</b>.
     * <li>call {@link #processFiles} one or more times.
     * <li>call {@link #finish} <b>once only</b>.
     * </ol>
     * This results in a slightly different logic flow:
     * <ul>
     *  <li>begin
     *  <li>perform global initialization processing (single, external call):
     *  <ul>
     *   <li>create a new symbol resolver
     *   <li>initialize and register default worker objects
     *   <li>load the pattern configuration profile
     *   <li>set a default specification to find ASTs if not provided by profile
     *  </ul>
     *  <li>for each file list to be iterated and processed (possibly multiple,
     *      external calls):
     *  <ul>
     *   <li>apply global, user-defined init-rules
     *   <li>for each ruleset defined in the pipeline:
     *   <ul>
     *    <li>make a deep copy of the source AST hierarchy; this copy becomes
     *        the editable source copy for the current ruleset pass and becomes
     *        the source AST for the next ruleset in the pipeline
     *    <li>perform ruleset-specific initialization processing:
     *    <ul>
     *     <li>configure the symbol resolver for the current ruleset
     *     <li>set the source and copy root ASTs as the current ASTs in the
     *         symbol resolver
     *     <li>apply user-defined init-rules for the ruleset
     *    </ul>
     *    <li>walk the AST (or the filtered view of results of previous ruleset),
     *        applying all walk-rules of the ruleset to the current AST
     *    <li>perform ruleset-specific termination processing:
     *    <ul>
     *     <li>set the source and copy root ASTs as the current ASTs in the
     *         symbol resolver
     *     <li>apply user-defined post-rules for the ruleset
     *    </ul>
     *    <li>use the copy AST from the last ruleset as the source AST for the
     *        next ruleset
     *   </ul>
     *   <li>apply global, user-defined post-rules
     *  </ul>
     *  <li>perform global termination processing (single, external call):
     *  <ul>
     *   <li>call each registered worker object's termination processing hook
     *   <li>null out the symbol resolver
     *  </ul>
     *  <li>end
     * </ul>
     * <p>
     * The pattern engine can walk either an arbitrary list of ASTs (default
     * mode) or can traverse a call graph (a tree, really). In call graph mode,
     * the engine uses one or more file specifications to load persisted call
     * graph profiles and then walks the call trees defined by these profiles.
     * If no call graph profiles are specified, it will try to load the project-
     * specific root node list, which is a list of source file paths which represent
     * all known program entry points for the target application.
     * <p>
     * <strong>Note:</strong> at this time, the pattern engine is <b>not</b>
     * designed to process multiple profiles in parallel.  However, it IS possible
     * to run multiple pattern engines (with different profiles if desired)
     * SEQUENTIALLY as long as the <code>initialize</code> method is called at
     * the beginning and the <code>finish</code> method is called at the
     * end of each run AND in each case a different instance of the pattern
     * engine is used.  Note that usage of the {@link #run} method naturally
     * handles the proper bracketing of <code>initialize</code> and
     * <code>finish</code>.  Multiple instances of this class should NOT be
     * created at the same time.  The singleton pattern is not explicitly
     * enforced at this time, as this limitation may be lifted with further
     * development.
     *
     * @see  com.goldencode.p2j.uast.RootNodeList
     */
    public final class PatternEngine
    extends RuleContainer
    {
       /** List of AST filespecs used to locate persisted AST files. */
       private final ArrayList<FileList> astSpecs = new ArrayList<>();

       /** List of newly created AST files by the rules processed by this engine. */
       private final ArrayList<String> newProgramFiles = new ArrayList<>();

       /** List of ASTs to be processed. */
       private final ArrayList<Aast> targetAsts = new ArrayList<>();

       /** List of explicit target AST names. */
       private final ArrayList<String> targetNames = new ArrayList<>();

       /** Logger. */
       private static final ConversionStatus LOG = ConversionStatus.get(PatternEngine.class);

       /** Default root directory used to locate persisted AST files. */
       private String directory = null;

       /** Filespec used to locate persisted AST files relative to a directory. */
       private String fileSpec = null;

       /** Object used by rules to resolve expression variables and functions. */
       private AstSymbolResolver resolver = null;

       /** Current AST results view from last ruleset processing in pipeline. */
       private Iterator<Aast> view = EmptyIterator.get();

       /** Set of visited AST IDs for use with call graph mode to prevent dups */
       private HashSet<Long> processedAsts = new HashSet<>();

       /** Graph DB instance. */
       private Graph graph = null;

       /** Honor hidden processing mode flag. */
       private boolean honorHidden = false;

       /** Operate in read-only mode; do not copy source AST for each ruleset */
       private boolean readOnly = false;

       /** Mapping of user variable names to initializer expression strings or values. */
       private Map<String, Object> varInitializers = null;

       /** Ordered set of target paths of files to be processed. */
       private final LinkedHashSet<String> targetPaths = new LinkedHashSet<>();

       /** Rule currently being (or most recently) processed */
       private Rule activeRule = null;

       /** System time at beginning of run. */
       private long jobStartTime = 0L;

       /** Map to store resulting objects like ASTs or streams during rules pipeline processing. */
       private final HashMap<StorageKey, Object> storedObjects = new HashMap<>();

       /** Flag indicating that conditional rule-set processing is activated or not. */
       private boolean conditionalRuleSets = false;

       /** The flags which must be set at the rule-set for the rule-set to be processed. */
       private final HashSet<String> activatedFlags = new HashSet<>();

       /**
        * In callgraph walking mode, this will be used to filter the nodes which will be processed.
        */
       private GraphNodeFilter nodeFilter = null;

       static
       {
          try
          {
             String name = Configuration.getParameter("registry");
             AstManager.initialize(() -> new XmlFilePlugin(name, Configuration.isImportMode()));
          }
          catch (AstException ae)
          {
             // TODO: log this? get ready for null pointer exceptions later
          }
       }

       /**
        * Default constructor.
        */
       public PatternEngine()
       {
          this(null, null);
       }

       /**
        * Constructor which accepts a directory and filespec to use instead of
        * those (if any) contained in a pattern profile.
        *
        * @param   directory
        *          Directory used to find ASTs to load; will override any
        *          provided in the pattern profile if not <code>null</code>.
        * @param   fileSpec
        *          File specification used to find ASTs to load; will override
        *          any provided in the pattern profile if not <code>null</code>.
        */
       public PatternEngine(String directory, String fileSpec)
       {
          super(null);

          this.directory = directory;
          this.fileSpec = fileSpec;
       }

       /**
        * A constructor for runtime conversion use to support a limited use mode, which does not
        * support some options, such as callgraph mode.
        *
        * @param   profile
        *          Name of the profile to be loaded. It is up to the {@link ConfigLoader}
        *          implementation to understand what to do with this name.
        * @param   conditional
        *          If <code>true</code>, enable conditional rule-set loading for runtime query mode.
        *
        * @throws  IllegalStateException
        *          if invoked when not in runtime use mode.
        * @throws  ConfigurationException
        *          if there is an error loading the profile.
        */
       public PatternEngine(String profile, boolean conditional)
       throws ConfigurationException
       {
          this(null, null);

          if (!Configuration.isRuntimeConfig())
          {
             throw new IllegalStateException("This pattern engine mode is only for use at runtime");
          }

          if (conditional)
          {
             setConditionalRuleSets(true);
             addConditionalFlag(ConversionFlag.RUNTIME_QUERY.toString());
          }

          // load and initialize the profile
          initialize(profile);

          // temporarily null out the context-local resolver; this will be set at each invocation of
          // runInMemory
          AstSymbolResolver.reset();
       }

       /**
        * Adds a flag to the {@link #activatedFlags} set, which will be checked only when
        * {@link #conditionalRuleSets} is activated.
        * <p>
        * Use {@link #setConditionalRuleSets(boolean)} to activate these flags before starting
        * conversion.
        *
        * @param    flag
        *           the conditional flag to add.
        */
       public void addConditionalFlag(String flag)
       {
          activatedFlags.add(prepareFlag(flag));
       }

       /**
        * Removes a flag from the {@link #activatedFlags} set.
        * <p>
        * Use {@link #setConditionalRuleSets(boolean)} to deactivate these flags after conversion is
        * finished.
        *
        * @param    flag
        *           the conditional flag to add.
        */
       public void removeConditionalFlag(String flag)
       {
          activatedFlags.remove(prepareFlag(flag));
       }

       /**
        * Enable or disable conditional rule-sets mode.
        * <p>
        * When activated, only rule-sets marked with flags part of the {@link #activatedFlags}
        * set will be included.
        *
        * @param    state
        *           The new state of the conditional rule-sets mode.
        */
       public void setConditionalRuleSets(boolean state)
       {
          conditionalRuleSets = state;
       }

       /**
        * Check if conditional rule-sets is activated.
        *
        * @return    The state of the {@link #conditionalRuleSets} flag.
        */
       public boolean isConditionalRuleSets()
       {
          return conditionalRuleSets;
       }

       /**
        * Check if the given flag is part of the {@link #activatedFlags} set.
        *
        * @param    flag
        *           The flag to check.
        *
        * @return   <code>true</code> if {@link #conditionalRuleSets conditional rule-set} processing
        *           is not activated or the flag is part of the {@link #activatedFlags} set.
        *           <code>false</code> if the flag is not part of the  {@link #activatedFlags} set.
        */
       public boolean isFlagActivated(String flag)
       {
          return !conditionalRuleSets || activatedFlags.contains(prepareFlag(flag));
       }

       /**
        * Check if the given flags are part of the flags registered in the {@link #activatedFlags} set.
        * <p>
        * The flags will be checked only when {@link #conditionalRuleSets} is activated.
        *
        * @param    flags
        *           The flags to check.
        *
        * @return   <code>true</code> if at least one flag is part of the
        *           {@link #activatedFlags} set or if {@link #conditionalRuleSets} is not
        *           activated.
        *           <code>false</code> if no flag from the given set is part of the
        *           {@link #activatedFlags} set.
        */
       public boolean activeFlags(Set<String> flags)
       {
          if (!conditionalRuleSets)
          {
             return true;
          }

          for (String flag : flags)
          {
             if (isFlagActivated(flag))
             {
                return true;
             }
          }

          return false;
       }

       /**
        * Set the Graph DB instance used by this pattern engine.  A non-null {@link #graph} instance
        * will enable callgraph processing mode.
        *
        * @param    graph
        *           The graph DB instance.
        */
       public void setGraph(Graph graph)
       {
          this.graph = graph;
       }

       /**
        * Gets the flag that determines if the hidden status of an AST node
        * should be honored.
        *
        * @return   <code>true</code> if a node's hidden flag should be honored.
        */
       public boolean isHonorHidden()
       {
          return honorHidden;
       }

       /**
        * Sets the flag that determines if the hidden status of an AST node
        * should be honored.
        *
        * @param    hide
        *           <code>true</code> if a node's hidden flag should be honored.
        */
       public void setHonorHidden(boolean hide)
       {
          honorHidden = hide;
       }

       /**
        * Set state of read-only mode. When in read-only mode, a copy of the
        * source AST is <i>not</i> made before each AST walk. By default,
        * read-only mode is turned off and a copy is made. Read-only mode is
        * available as an optimization to avoid the overhead of this potentially
        * expensive operation if it is not needed.
        * <p>
        * In read-only mode, the source AST is used as both the source and copy
        * AST for each rule-set processed. Rule-sets should never modify the AST
        * being walked when read-only mode is enabled. Note that currently there
        * is no enforcement of this contract. It is on the "honor system" only.
        *
        * @param   readOnly
        *          <code>true</code> to enable read-only mode; <code>false</code>
        *          to disable it.
        */
       public void setReadOnly(boolean readOnly)
       {
          this.readOnly = readOnly;
       }

       /**
        * Store a mapping of user variable names to initializer expression
        * overrides (in infix notation), as extracted from the command line
        * arguments passed to this process.
        *
        * @param   initializers
        *          Key/value pairs of variable names to initializer expression
        *          strings. The strings must represent valid expressions in infix
        *          notation.
        */
       public void setVariableInitializers(Map<String, Object> initializers)
       {
          varInitializers = initializers;
       }

       /**
        * Add a directory and file specification pair which will allow the engine
        * to locate a set of persisted AST files to be processed into ASTs, which
        * will in turn be applied against a pipeline of rulesets. Any number of
        * such pairs may be added; however, if this object was constructed with
        * an overriding pair, calls to this method will be ignored, and the
        * overriding pair will be used instead.
        *
        * @param   directory
        *          Relative or absolute directory in which to begin a recursive
        *          search for AST files matching the file specification defined
        *          by <code>fileSpec</code>. If relative, the path will be
        *          prepended with the <code>P2J_HOME</code> path.
        * @param   fileSpec
        *          A regular expression indicating a filename mask to match when
        *          searching for persisted AST files to process for the current
        *          pattern matching session.
        * @param   force
        *          <code>true</code> to force the spec to be added.
        *
        * @throws  ConfigurationException
        *          if <code>P2J_HOME</code> has not been defined for the current
        *          process.
        */
       public void addAstSpec(String directory, String fileSpec, boolean force)
       throws ConfigurationException
       {
          // An empty string for fileSpec is meaningless.
          if (fileSpec != null && fileSpec.trim().length() == 0)
          {
             return;
          }

          // Ignore parameters if the user has overridden from the constructor.
          if (this.directory != null && !force)
          {
             return;
          }

          File dir = Configuration.toFile(directory);
          astSpecs.add(new FileSpecList(dir, fileSpec, true));
       }

       /**
        * Add a list of persisted AST files to be processed into ASTs, which
        * will in turn be applied against a pipeline of rulesets. Any number of
        * such lists may be added.
        *
        * @param   list
        *          An existing, non-null list of files.
        */
       public void addAstSpec(FileList list)
       {
          // a null list is meaningless
          if (list == null)
          {
             return;
          }

          astSpecs.add(list);
       }

       /**
        * Add the specified AST name as a target.
        *
        * @param    name
        *           An AST name.
        */
       public void addTarget(String name)
       {
          targetNames.add(name);
       }

       /**
        * Add AST to be processed against a pipeline of rule sets.
        *
        * @param ast
        *        AST to be processed.
        */
       public void addTargetAst(Aast ast)
       {
          targetAsts.add(ast);
       }

       /**
        * Get an iterator over the ast specifications ({@link
        * com.goldencode.p2j.util.FileList} objects). Items are iterated in the
        * order in which they were added.
        *
        * @return  Iterator over AST spec <code>FileList</code> objects used to
        *          find ASTs to process.
        */
       public Iterator<FileList> astSpecs()
       {
          return astSpecs.iterator();
       }

       /**
        * Override the superclass' implementation of this method to allow for
        * replacement of variables' initializer expressions with values provided
        * by the user. Delegates eventual registration of the variable to the
        * superclass' implementation.
        * <p>
        * Register a new user variable with this container. If the variable's
        * name is already in use within this rule container's scope, it cannot
        * be re-used and this method will fail. If both <code>type</code> and
        * <code>expression</code> are provided, <code>type</code> must match the
        * return type of <code>expression</code>.
        *
        * @param   name
        *          Name by which this variable will be referenced in expressions.
        * @param   type
        *          Data type of the variable. May be <code>null</code> if and
        *          only if <code>expression</code> is not <code>null</code>
        *          <b>and</b> <code>expression</code>'s return type is not
        *          <code>null</code>.
        * @param   expression
        *          Expression which will be used to initialize the variable each
        *          time it is {@link com.goldencode.expr.Variable#reset reset}
        *          by the pattern engine. May be <code>null</code> if and only
        *          if <code>type</code> is not <code>null</code>.
        *
        * @throws  com.goldencode.expr.SymbolException
        *          if the variable's name is already in use in this container's
        *          scope.
        * @throws  com.goldencode.expr.ExpressionException
        *          if <code>type</code> is a class which is not assignable from
        *          <code>expression</code>'s return type.
        * @throws  IllegalArgumentException
        *          if <code>type</code> is not provided and it cannot be
        *          determined from <code>expression</code>.
        *
        * @see     com.goldencode.expr.SymbolResolver#resetVariables
        */
       @Override
       public void registerVariable(String name, Class<?> type, String expression)
       {
          if (varInitializers != null)
          {
             Object value = varInitializers.get(name);

             if (value != null)
             {
                if (value instanceof String)
                {
                   // assume this is an override expression
                   expression = (String) value;
                }
                else
                {
                   // this is the value itself, not an expression
                   super.registerVariableValue(name, type, value);
                   return;
                }
             }
          }

          super.registerVariable(name, type, expression);
       }

       /**
        * Perform a discrete pipeline pattern run against a set of target ASTs. This mode runs init
        * rules, processes the list of ASTs, then runs post rules. It is intended for runtime
        * conversion use in a long-lived pattern engine.
        */
       public void run()
       {
          AstSymbolResolver.setLocal(resolver);

          try
          {
             // run global init rules.
             applyGlobal(RULE_INIT);

             // Process each in-memory AST
             for (int i = 0; i < targetAsts.size(); i++)
             {
                Aast targetAst = targetAsts.get(i);
                processAst(targetAst);
             }

             // run global post rules.
             applyGlobal(RULE_POST);
          }
          catch (Throwable thr)
          {
             StringBuilder buf = new StringBuilder();
             String sep = System.getProperty("line.separator");

             if (activeRule != null)
             {
                buf.append("ERROR!  Active Rule:");
                buf.append(sep);
                buf.append(activeRule.createReport());
                buf.append(sep);
             }
             else
             {
                buf.append("ERROR!  Active rule could not be determined: ");
                buf.append(thr.getMessage());
             }

             throw new RuntimeException(buf.toString(), thr);
          }
          finally
          {
             AstSymbolResolver.reset();
             targetAsts.clear();
          }
       }

       /**
        * Run the pattern engine once using the specified configuration profile.
        * This will load the profile and initialize the pattern engine and its
        * supporting workers. It will then apply the pipeline of rulesets defined
        * in the profile to every AST it finds based upon the AST specifications
        * found in the profile (or provided as overrides from the command line).
        * After all ASTs have been processed, each registered pattern worker is
        * called to perform its post-processing, if any, and the pattern engine
        * itself performs some clean-up processing. Finally, the registry of ASTs
        * is saved, in case any changes have occurred.
        *
        * @param   profile
        *          Name of the profile to be loaded. It is up to the {@link
        *          ConfigLoader} implementation to understand what to do with
        *          this name.
        *
        * @throws  ConfigurationException
        *          if any error occurs loading the specified configuration
        *          profile.
        * @throws  AstException
        *          if any error occurs loading a persisted AST.
        */
       public void run(String profile)
       throws ConfigurationException,
              AstException
       {
          if (graph != null)
          {
             String nodeFilterClass = Configuration.getParameter("graph-node-filter", null);
             if (nodeFilterClass == null)
             {
                throw new ConfigurationException(
                   "The graph-node-filter configuration parameter is not set!");
             }

             try
             {
                Class<?> nodeFilterClz = Class.forName(nodeFilterClass);
                if (!GraphNodeFilter.class.isAssignableFrom(nodeFilterClz))
                {
                   throw new ConfigurationException(
                      nodeFilterClz + " does not implement the GraphNodeFilter interface!");
                }

                this.nodeFilter = (GraphNodeFilter) nodeFilterClz.getDeclaredConstructor().newInstance();
             }
             catch (ReflectiveOperationException e)
             {
                throw new ConfigurationException(
                   "Could not instantiate the node-filter from " + nodeFilterClass, e);
             }
          }

          try
          {
             // Load and initialize the profile.
             initialize(profile);

             // Run global init rules.
             applyGlobal(RULE_INIT);

             Configuration config = Configuration.getInstance();

             // Process each stored AST or walk call graph, depending on the mode of operation.
             Iterator<String> iter = targetPaths.iterator();
             while (iter.hasNext())
             {
                String path = iter.next();

                config.withFileProfile(path);

                if (graph != null)
                {
                   Aast ast = AstManager.get().loadTree(path);
                   long id = ast.getId();
                   Vertex node = nodeFilter.acceptNode(id);
                   if (node != null)
                   {
                      walkGraph(node);
                   }
                }
                else
                {
                   processAst(path);
                }
             }

             // Process each in-memory AST
             for (int i = 0; i < targetAsts.size(); i++)
             {
                Aast targetAst = targetAsts.get(i);
                processAst(targetAst);
             }

             // Run global post rules.
             resolver.setAsts(null, null);
             applyGlobal(RULE_POST);
          }
          catch (Throwable thr)
          {
             StringBuilder buf = new StringBuilder();
             String sep = System.getProperty("line.separator");

             if (activeRule != null)
             {
                buf.append("ERROR!  Active Rule:");
                buf.append(sep);
                buf.append(activeRule.createReport());
                buf.append(sep);
             }
             else
             {
                buf.append("ERROR!  Active rule could not be determined: ");
                buf.append(thr.getMessage());
             }

             String msg = buf.toString();
             Aast srcAst = resolver.getSourceAst();
             if (srcAst != null)
             {
                TreeWalkException exc = new TreeWalkException(msg, thr);
                exc.setSourceAstId(srcAst.getId());
                exc.setCopyAstId(resolver.getCopyAst().getId());
                exc.setFilename(srcAst.getFilename());

                throw exc;
             }
             else
             {
                throw new RuntimeException(msg, thr);
             }
          }
          finally
          {
             // Perform post-processing and cleanup.
             finish();
          }
       }

       /**
        * This will apply the pipeline of rulesets defined in the currently
        * loaded profile to every AST it finds based upon the given iterator
        * that lists the <code>File</code> objects that must be processed (and
        * the inherent order in which these files must be processed).
        * <p>
        * It is assumed that the caller is responsible for loading a profile
        * before this method is invoked and for conducting termination processing
        * afterwards.
        *
        * @param   iter
        *          An iterator of <code>File</code> objects to process.
        *
        * @throws  AstException
        *          if any error occurs loading a persisted AST.
        */
       public void processFiles(Iterator<File> iter)
       throws AstException
       {
          try
          {
             // Run global init rules.
             resolver.setAsts(null, null);
             applyGlobal(RULE_INIT);

             // For each file in the list, process the associated AST.
             while (iter.hasNext())
             {
                File next = iter.next();

                // Do not try to process non-existent files.
                if (next.exists())
                {
                   processAst(next.getAbsolutePath());
                }
             }

             // Run global post rules and clean up resolver.
             resolver.setAsts(null, null);
             applyGlobal(RULE_POST);
          }
          finally
          {
             resolver.cleanUp();
          }
       }

       /**
        * Initialize the pattern engine by creating a new symbol resolver
        * instance, registering default pattern workers, loading the specified
        * profile, ensuring an AST spec is available, and applying all global,
        * user-defined initialization rules.
        *
        * @param   profile
        *          Name of the profile to be loaded. It is up to the {@link
        *          ConfigLoader} implementation to understand what to do with
        *          this name.
        *
        * @throws  ConfigurationException
        *          if any error occurs loading the specified configuration
        *          profile.
        */
       public void initialize(String profile)
       throws ConfigurationException
       {
          jobStartTime = System.currentTimeMillis();

          // Clear set of processed ASTs.
          if (graph != null)
          {
             processedAsts.clear();
          }

          // Create a new AST symbol resolver.
          resolver = AstSymbolResolver.create(this);

          // Register commonly used pattern workers.
          registerDefaultWorkers();

          // Load the profile.
          ConfigLoader loader = new ConfigLoader(this);
          loader.load(profile);

          // Add an AST spec using overrides, if no other has been loaded.
          if (astSpecs.isEmpty() && directory != null)
          {
             addAstSpec(directory, fileSpec, true);
          }

          // Gather set of paths we need to process.
          gatherTargetPaths();
       }

       /**
        * Perform cleanup/termination post-processing, including applying all
        * global, user-defined post-rules, calling each pattern worker's {@link
        * PatternWorker#finish} method, and nulling out the symbol resolver.
        */
       public void finish()
       throws AstException
       {
          // Call each worker to hook post-processing.
          Iterator<PatternWorker> iter = resolver.workers();
          while (iter.hasNext())
          {
             PatternWorker worker = iter.next();
             worker.finish();
          }

          // Clean up all contained rules and rulesets.
          cleanup();

          activeRule = null;
          processedAsts = new HashSet<>();
          varInitializers = null;

          // Null out the resolver and clear some static state to allow future pattern engine runs to succeed.
          resolver = null;
          AstSymbolResolver.reset();

          // If not in quiet mode, report elapsed job time.
          if (getDebugLevel() >= MSG_STATUS)
          {
             long elapsed = System.currentTimeMillis() - jobStartTime;
             Date date = new Date(elapsed - TimeZone.getDefault().getRawOffset());
             SimpleDateFormat format = new SimpleDateFormat("HH:mm:ss.SSS");

             System.out.println("Elapsed job time:  " + format.format(date));
          }

          if (!readOnly)
          {
             AstManager.get().save();
          }
       }

       /**
        * Store the given object under the given id.
        *
        * @param id
        *        Arbitrary object id.
        * @param object
        *        Object to store.
        */
       public void storeObject(String id, Object object)
       {
          storedObjects.put(new StorageKey(id), object);
       }

       /**
        * Store the given object under the given category and id.
        *
        * @param category
        *        Arbitrary object category.
        * @param id
        *        Arbitrary object id.
        * @param object
        *        Object to store.
        */
       public void storeObject(String category, String id, Object object)
       {
          storedObjects.put(new StorageKey(category, id), object);
       }

       /**
        * Get the object stored under the given id.
        *
        * @param  id
        *         Object id.
        *
        * @return stored object.
        */
       public Object getStoredObject(String id)
       {
          return storedObjects.get(new StorageKey(id));
       }

       /**
        * Store the given object under the given key.
        *
        * @param key
        *        Storage key under which to store the object.
        * @param object
        *        Object to store.
        */
       public void storeObject(StorageKey key, Object object)
       {
          storedObjects.put(key, object);
       }

       /**
        * Get the object stored under the given category and id.
        *
        * @param  category
        *         Object category.
        * @param  id
        *         Object id.
        *
        * @return stored object.
        */
       public Object getStoredObject(String category, String id)
       {
          return storedObjects.get(new StorageKey(category, id));
       }

       /**
        * Get an unmodifiable map of all objects currently stored with this pattern engine.
        *
        * @return  Map of stored objects.
        */
       public Map<StorageKey, Object> getAllStoredObjects()
       {
          return Collections.unmodifiableMap(storedObjects);
       }

       /**
        * Clear the map of objects currently stored with this pattern engine.
        */
       public void clearStoredObjects()
       {
          storedObjects.clear();
       }

       /**
        * Set the rule which is currently active.  This is used for debug
        * reporting.
        *
        * @param   activeRule
        *          Currently active rule.
        */
       void setActiveRule(Rule activeRule)
       {
          this.activeRule = activeRule;
       }

       /**
        * Get the list of newly added program file names.
        *
        * @return   The {@link #newProgramFiles}.
        */
       public List<String> getNewProgramFiles()
       {
          return newProgramFiles;
       }

       /**
        * Clear the list of newly, generated, {@link #newProgramFiles program file names}.
        */
       public void clearNewProgramFiles()
       {
          newProgramFiles.clear();
       }

       /**
        * Add the given filename as being newly generated, so it will be registered and processed
        * by other rule-sets, downstream.
        *
        * @param    file
        *           The new program file name.
        */
       public void addNewProgramFile(String file)
       {
          this.newProgramFiles.add(file);
       }

       /**
        * Prepare the given flag by trimming any whitespace and converting it to uppercase.
        *
        * @param    flag
        *           The flag to be prepared.
        *
        * @return   The prepared flag.
        */
       private String prepareFlag(String flag)
       {
          return flag.trim().toUpperCase();
       }

       /**
        * Walk the call tree recursively, depth-first, from the specified node.
        * At each descent down to a new nesting level, apply the global descent
        * rules, if any. At each ascent back up to a previous nesting level,
        * apply the global ascent rules, if any. Process the AST represented by
        * <code>node</code>, then iterate through <code>node</code>'s children,
        * invoking this method recursively for each child found.  Apply the
        * global next-child rule (with the parent node in scope) when moving
        * between any child (but NOT when moving down to the first child which is
        * handled by the descend rule OR when moving up from the last child which
        * is handled by the ascend rule).  In other words, the next-child rule
        * will be triggered between the first and second child, the second and
        * third child... and when moving between the second-to-last and the last
        * child.
        *
        * @param   node
        *          Call graph node at which to begin the walk. Contains the
        *          project-relative path of the AST it represents.
        *
        * @throws  AstException
        *          if any error occurs generating the copy AST.
        */
       private void walkGraph(Vertex node)
       throws AstException
       {
          // Apply global AST descent rules.
          apply(resolver, RULE_DESCENT);

          // Process a Vertex once and only once.
          Long id = (Long) node.id();
          if (!processedAsts.contains(id) && nodeFilter.acceptNode(node))
          {
             String filename = node.<String>property("os-filename").value();
             Aast ast = AstManager.get().loadTree(filename + ".ast");
             processAst(ast);
             processedAsts.add(id);
          }
          else
          {
             return;
          }

          boolean firstChild = true;
          GraphTraversalSource g = node.graph().traversal();

          // GES_TODO: Does it make sense to map the descent, ascent and next-child events to a call
          // graph walk?  It seems like it is not the best fit, especially since there is no inherent
          // ordering of edges.  The order is unpredictable and thus the results could be different
          // from run to run.

          // the idea here is that we only process based on a specific edge label (e.g. "calls" for an
          // invocation), otherwise will process "contains", "includes" and all other types of
          // relationships and will arrive at entry-point vertices from non-call-sites
          Iterator<Vertex> iter = g.V(node).out("calls");

          // Process all outbound vertices recursively.
          while (iter.hasNext())
          {
             Vertex vertex = iter.next();

             if (!nodeFilter.acceptNode(node))
             {
                continue;
             }

             // Determine if this is the first time through the loop.
             if (firstChild)
             {
                // Update our state, subsequent iterations mean we are visiting
                // siblings.
                firstChild = false;
             }
             else
             {
                // Apply global next-child AST rules (only on the movement
                // between children --> not on entering the first or leaving the
                // last child since descend and ascend respectively can be used
                // for those purposes).
                apply(resolver, RULE_NEXT_CHILD);
             }

             walkGraph(vertex);
          }

          // Apply global AST ascent rules.
          apply(resolver, RULE_ASCENT);
       }

       /**
        * Apply the pipeline of rulesets currently loaded against a single AST.
        * Each ruleset is applied in turn, in the order in which they were loaded.
        * The source AST is provided by the {@link AstManager#loadTree} method. It
        * is copied for the first ruleset applied, and any edits are made to the
        * copy rather than to the original. Once that ruleset has been fully
        * processed, the copy becomes the source AST to the next ruleset in the
        * pipeline. In such a way, an AST may be incrementally refined by each
        * ruleset and passed on to the next in the pipeline for further
        * processing.
        *
        * @param   astName
        *          Name of the AST persisted file to be processed.
        *
        * @return  Processed copy of AST.
        *
        * @throws  AstException
        *          if any error occurs generating the copy AST.
        *
        * @see     #apply
        */
       private Aast processAst(String astName)
       throws AstException
       {
          // Get the original source AST from persistence.
          Aast ast = AstManager.get().loadTree(astName);
          return processAst(ast);
       }

       /**
        * Apply the pipeline of rulesets currently loaded against a single AST.
        * Each ruleset is applied in turn, in the order in which they were loaded.
        * The source AST is provided by the {@link AstManager#loadTree} method. It
        * is copied for the first ruleset applied, and any edits are made to the
        * copy rather than to the original. Once that ruleset has been fully
        * processed, the copy becomes the source AST to the next ruleset in the
        * pipeline. In such a way, an AST may be incrementally refined by each
        * ruleset and passed on to the next in the pipeline for further
        * processing.
        *
        * @param   ast
        *          AST to be processed.
        *
        * @return  Processed copy of AST.
        *
        * @throws  AstException
        *          if any error occurs generating the copy AST.
        *
        * @see     #apply
        */
       private Aast processAst(Aast ast)
       throws AstException
       {
          // notify all pattern workers that we are about to process the given AST; this is the *source* AST
          // instance
          Iterator<PatternWorker> wrkIter = resolver.workers();
          while (wrkIter.hasNext())
          {
             PatternWorker worker = wrkIter.next();
             worker.visitAst(ast);
          }

          // report filename being processed if quiet mode is off
          if (getDebugLevel() >= MSG_STATUS)
          {
             String filename = ast.getFilename();
             if (filename == null)
             {
                // print the descriptive text of the root node if file name is not available
                System.out.printf("%s (%#016x): Unknown file name. %n",
                                  ast.getDescriptiveTokenText(),
                                  ast.getId());
             }
             else
             {
                System.out.println(filename);
             }
          }

          // Apply each ruleset. Unless in read-only mode, a copy is made each
          // iteration through the loop, which may be modified by rules. The
          // copy becomes the original for the next pass.
          ArrayList<RuleListElement> rules = ruleList();
          if (rules != null)
          {
             int size = rules.size();
             RuleListElement next;
             for (int i = 0; i < size; i++)
             {
                next = rules.get(i);
                if (next.isContainer())
                {
                   RuleSet ruleSet = (RuleSet) next;
                   ast = apply(ruleSet, ast);
                }
             }
          }

          // notify all pattern workers that we have finished processing the given AST; at this point, the ast
          // variable refers to the *copy* AST instance
          wrkIter = resolver.workers();
          while (wrkIter.hasNext())
          {
             PatternWorker worker = wrkIter.next();
             worker.leaveAst(ast);
          }

          // return the final AST copy, possibly transformed
          return ast;
       }

       /**
        * Apply the given ruleset against the specified source AST, using the
        * rules and configuration information stored within the ruleset. This
        * method first makes a deep copy of the source AST hierarchy, of which
        * <code>source</code> is the root node. Each AST node in the copy tree
        * is linked by ID to its original counterpart in the source tree, but
        * there are no shared object references between the trees, so the copy
        * nodes may be edited, moved, or otherwise modified by rules within the
        * ruleset without impacting the original, source AST.
        * <p>
        * Next, the engine's symbol resolver is configured to use the correct
        * expression library (that of the current ruleset merged with the global
        * expression library). The root source and copy ASTs are set into the
        * resolver as the current source and copy AST nodes, respectively. The
        * ruleset's user-defined init-rules, if any, are then applied.
        * <p>
        * Next, the ruleset's walk-rules are applied. Depending upon the {@link
        * RuleSet#setInputMode input mode} of the ruleset, this means either the
        * entire source tree is walked (<code>RuleSet.INPUT_TREE</code>), or only
        * those AST nodes added to the filtered set of results defined by the
        * last ruleset processed in this pipeline (if any) are visited, regardless
        * of the tree structure (<code>RuleSet.INPUT_VIEW</code>). In either case,
        * at each node visited in the source tree, the symbol resolver is updated
        * with the current source node and the corresponding copy node. By
        * convention, the source node is considered "read-only"; rules must not
        * modify it. Rules may make edits to the copy node, including annotating
        * it, moving it, or removing it from the copy tree entirely. A reference
        * to the copy node may also be added to a collection of results
        * maintained by the symbol resolver. An iterator on this collection, is
        * stored as an optional input to the next ruleset in the pipeline.
        * <p>
        * Next, the root source and copy ASTs are again set into the resolver
        * as the current source and copy AST nodes, respectively, and the
        * ruleset's user-defined post-rules, if any, are then applied.
        * <p>
        * Finally, the (possibly edited) copy tree is returned, so that it can
        * become an input to the next ruleset in the pipeline.
        *
        * @param   ruleSet
        *          Ruleset to be applied against an AST.
        * @param   source
        *          Source AST to be processed by the ruleset.
        *
        * @return  The copy AST root node, after the ruleset has been applied.
        */
       private Aast apply(RuleSet ruleSet, Aast source)
       {
          try
          {
             // Make a deep copy of the source AST which will become the copy
             // AST of this pass, unless in read-only mode.
             HashMap<Long, Aast> idMap = new HashMap<>();
             Aast copy = source;
             if (!readOnly)
             {
                copy = source.duplicate(idMap);
             }

             // Configure the resolver for this ruleset pass.
             resolver.reset(ruleSet);
             resolver.setAsts(source, copy);

             // Reset all user variables local to this ruleset to their default
             // values.
             resolver.resetVariables(ruleSet, false);

             // Implement the global honorHidden override (overrides whatever is
             // set in the rule-set attribute).  Note: unless honorHidden is true,
             // whatever is set in the ruleSet's attribute will take precedence.
             if (honorHidden)
             {
                ruleSet.setHonorHidden(true);
             }

             // Run ruleset's init rules.
             ruleSet.apply(resolver, RULE_INIT);

             // Initialize the ruleset. If the ruleset indicates a tree walk is
             // necessary, and walk has not been ended prematurely, walk the AST,
             // applying this ruleset to each node.
             if (!resolver.isEndWalk() &&
                 (ruleSet.hasType(RULE_WALK)    ||
                  ruleSet.hasType(RULE_DESCENT) ||
                  ruleSet.hasType(RULE_ASCENT)  ||
                  ruleSet.hasType(RULE_NEXT_CHILD)))
             {
                AstWalker walker = new AstWalker(ruleSet, idMap);

                // Walk the tree or the view created by the last ruleset, depending
                // upon the input mode.
                if (ruleSet.getInputMode() == RuleSet.INPUT_VIEW)
                {
                   view = walker.walk(resolver, view);
                }
                else
                {
                   view = walker.walk(resolver, source, ruleSet.getDepth());
                }
             }

             if (resolver.isEndWalk())
             {
                // Reset end walk flag in resolver.
                resolver.setEndWalk(false);

                // Warn user if message level permits.
                if (getDebugLevel() >= MSG_STATUS)
                {
                   System.out.println("AST walk ended prematurely");
                }
             }
             else
             {
                // Perform any post-processing defined by this ruleset.
                resolver.setAsts(source, copy);
                ruleSet.apply(resolver, RULE_POST);
             }

             return copy;
          }
          finally
          {
             // Clear all user variables local to this ruleset.
             resolver.resetVariables(ruleSet, true);

             // Allow resolver to clean up its resources.
             resolver.cleanUp();
          }
       }

       /**
        * Runs the global rules defined for a pipeline and DOES NOT recursively
        * call any rules in any contained ruleset.
        *
        * @param   type
        *          Either <code>RULE_INIT</code> or <code>RULE_POST</code>.
        */
       private void applyGlobal(int type)
       {
          ArrayList<RuleListElement> rules = ruleList();
          if (rules == null)
          {
             return;
          }

          int size = rules.size();
          RuleListElement next;
          for (int i = 0; i < size; i++)
          {
             next = rules.get(i);

             if (!next.isContainer())
             {
                Rule rule = (Rule) next;
                rule.apply(resolver, type);
             }
          }
       }

       /**
        * Assemble the master set of filenames to be processed. This is a set
        * of source filenames if in default mode;  it is a set of persisted
        * call graph filenames if in call graph mode.
        *
        * @throws  ConfigurationException
        *          if any error with the configuration is encountered.
        */
       private void gatherTargetPaths()
       throws ConfigurationException
       {
          // consolidate all files derived from AST specs into master set.
          Iterator<FileList> iter = astSpecs.iterator();
          if (Configuration.isImportMode())
          {
             // assume the path is correct (import only one database at a time) inside jar and just add them
             // to [targetPaths]
             while (iter.hasNext())
             {
                FileList next = iter.next();
                if (next instanceof FileSpecList)
                {
                   FileSpecList fileList = (FileSpecList) next;
                   targetPaths.add(fileList.getStartDir() + "/" + fileList.getFileSpec());
                }
                else
                {
                   File[] files = next.list();
                   for (File file : files)
                   {
                      targetPaths.add(file.getName());
                   }
                }
             }
          }
          else
          {
             while (iter.hasNext())
             {
                FileList fileList = iter.next();
                File[] files = fileList.list();
                int len = files.length;
                for (int i = 0; i < len; i++)
                {
                   targetPaths.add(toNormalizedPath(files[i].getAbsolutePath()));
                }
             }

             for (int i = 0; i < targetNames.size(); i++)
             {
                String name = targetNames.get(i);
                targetPaths.add(toNormalizedPath(new File(name).getAbsolutePath()));
             }
          }

          // Warn if nothing to process and not in quiet mode.
          if (targetPaths.isEmpty() && targetAsts.isEmpty() && getDebugLevel() >= MSG_STATUS)
          {
             StringBuilder buf = new StringBuilder();
             StringHelper.appendOnNewLine(buf, "*********");
             StringHelper.appendOnNewLine(buf, "Warning:  no files matching the given criteria were found");
             if (!astSpecs.isEmpty())
             {
                StringHelper.appendOnNewLine(buf, "Specification(s) (relative to project root):");
                iter = astSpecs.iterator();
                while (iter.hasNext())
                {
                   FileList fileList = iter.next();

                   if (fileList instanceof FileSpecList)
                   {
                      FileSpecList fsl = (FileSpecList) fileList;
                      String dir = fsl.getStartDir().getAbsolutePath();

                      buf.append("   Starting Directory:  ")
                         .append(Configuration.normalizeFilename(dir))
                         .append(";  RegEx File Specification:  ")
                         .append(fsl.getFileSpecAsRegEx());
                   }
                   else
                   {
                      String[] list = fileList.listFilenames();

                      StringHelper.appendOnNewLine(buf, "   List:");

                      for (int j = 0; j < list.length; j++)
                      {
                         buf.append("      ")
                            .append(Configuration.normalizeFilename(list[j]));
                      }
                   }
                   StringHelper.appendOnNewLine(buf, "");
                }
                StringHelper.appendOnNewLine(buf, "*********");
                LOG.log(Level.WARNING, buf.toString());
             }
          }
       }

       /**
        * Normalize the specified path if possible, to eliminate unnecessary,
        * embedded, relative references and to prepend the project's home
        * directory, if <code>path</code> is not absolute. If <code>path</code>
        * is already absolute, it is not converted.
        *
        * @param   path
        *          An absolute path in the project's home directory, or
        *          <code>path</code> itself if it is already an absolute path.
        *
        * @throws  ConfigurationException
        *          if the project home directory is not configured.
        */
       private String toNormalizedPath(String path)
       throws ConfigurationException
       {
          path = Configuration.normalizeFilename(path);
          File file = Configuration.toFile(path);

          return file.getAbsolutePath();
       }

       /**
        * Registers all default pattern workers. Default pattern workers are
        * those that supply basic services which are likely to be used across a
        * wide range of applications. Rather than requiring that they be defined
        * in each pattern profile, they are registered once here.
        * <p>
        * Currently, the default workers are instances of:
        * <ul>
        * <li>{@link CommonAstSupport}
        * <li>{@link PropsWorker}
        * <li>{@link DictionaryWorker}
        * <li>{@link FileOperationsWorker}
        * </ul>
        *
        * @throws  ConfigurationException
        *          if any exception is thrown by the default constructor of a
        *          default worker. The exception is caught and chained as the
        *          root cause of the <code>ConfigurationException</code>.
        */
       private void registerDefaultWorkers()
       throws ConfigurationException
       {
          try
          {
             registerWorker("common", new CommonAstSupport(this));
             registerWorker("prop", new PropsWorker());
             registerWorker("dict", new DictionaryWorker());
             registerWorker("io", new FileOperationsWorker());
          }
          catch (Exception exc)
          {
             throw new ConfigurationException("Error initializing pattern engine", exc);
          }
       }

       /**
        * Emit a syntax statement and optional error message to {@code stderr}, then exit the process with the
        * specified return code. This method is called from {@link #main} when the user passes command line
        * parameters which are not viable.
        *
        * @param   msg
        *          Optional error message; may be {@code null}.
        * @param   rc
        *          Process return code used with {@code System.exit()}.
        */
       private static void syntax(String msg, int rc)
       {
          if (msg != null)
          {
             LOG.log(Level.SEVERE, msg);
          }

          String[] staticText =
          {
             "Syntax:",
             "   java PatternEngine [options]",
             "                      [\"<variable>=<expression>\"...]",
             "                      <profile>",
             "                      [<directory> \"<filespec>\" | <filelist>]",
             "",
             "where",
             "   options:",
             "      -d <debuglevel> Message output mode; debuglevel values:",
             "                         0 = no message output",
             "                         1 = status messages only",
             "                         2 = status + debug messages",
             "                         3 = verbose trace output",
             "      -c              Call graph walking mode",
             "      -f              Explicit file list mode",
             "      -h              Honor hidden mode",
             "      -r              Read-only mode",
             "      -profile <name> Sets the configuration profile",
             "",
             "   variable   = variable name",
             "   expression = infix expression used to initialize variable",
             "   profile    = rules pipeline configuration filename",
             "   directory  = directory in which to search recursively for persisted ASTs",
             "   filespec   = file filter specification to use in persisted AST search",
             "   filelist   = arbitrary list of absolute and/or relative file names of " +
                             "persisted AST files to process (-f mode)"
          };

          LOG.log(Level.INFO, StringHelper.arrayToString(staticText));

          System.exit(rc);
       }

       /**
        * Command line driver to launch the pattern engine, using a pre-defined pattern profile.
        *
        * @param   args
        *          Expected arguments (in order) are as follows:
        *          <ul>
        *             <li>[-options] any combination of the following options (case-insensitive):
        *               <ul>
        *                  <li>-d debuglevel = level of error/status reporting desired.
        *                  <li>-c = call graph processing mode; use the {@code directory} and {@code filespec}
        *                      parameters (see below) to specify persisted ASTs which should be processed. If
        *                      those parameters are not provided, process the ASTs referenced in the project's
        *                      root node list.
        *                  <li>-f = explicit file list; use the {@code filelist} parameters to specify persisted
        *                      AST files which should be processed.
        *                  <li>-h = turn on honor hidden mode which drops AST nodes from the walk in the case
        *                      where their hidden flag is {@code true} (this honor hidden mode is off by default)
        *                  <li>-profile <name> = sets the configuration profile to {@code name}
        *               </ul>
        *             <li>{@code variable=expression} (optional): a key/value pair which overrides the initializer
        *                 expression (if any) of a user variable defined within the configuration profile.
        *                 {@code variable} is the variable name and <code>expression</code> is the initializer
        *                 expression override (which must resolve to a value of the variable's data type).
        *             <li>{@code profile} (required): name of the pattern configuration profile to be loaded for
        *                 this run.
        *             <li>{@code directory} (optional): starting directory used for a recursive search for
        *                 persisted AST files to process. Overrides any such data loaded from the profile. If in
        *                 call graph mode, represents the directory
        *             <li>{@code filespec} (required if and only if {@code directory} was provided): regular
        *                 expression (in double quotes) used as a filename mask in combination with
        *                 {@code directory} to find persisted AST files to process. Overrides any such data loaded
        *                 from the profile.
        *             <li>{@code filelist} (required if and only if explicit file list mode is enabled): the
        *                 remaining arguments specify an arbitrary list of persisted AST files to process.
        *                 Overrides any such data loaded from the profile.
        *          </ul>
        */
       public static void main(String[] args)
       {
          int debugLevel = MSG_STATUS;

          int len = args.length;
          if (len < 1)
          {
             syntax(null, -1);
          }

          int count = 0;

          boolean metd = false;
          boolean metc = false;
          boolean metf = false;
          boolean meth = false;
          boolean metr = false;
          boolean callGraphMode = false;
          boolean explicit = false;
          boolean honorHidden = false;
          boolean readOnly = false;
          ArrayList<String> cfgProfile = new ArrayList<>();

          // Process options first.
          for (int i = 0; i < len && args[i].startsWith("-"); i++, count++)
          {
             // Debug level argument.
             if (args[i].equalsIgnoreCase("-d"))
             {
                if (metd)
                {
                   syntax("Only one -d option is allowed", -2);
                }

                if (args.length <= i + 1)
                {
                   syntax("-d should be followed by debuglevel argument", -3);
                }

                boolean err = false;
                try
                {
                   debugLevel = Integer.parseInt(args[i + 1]);
                }
                catch (NumberFormatException exc)
                {
                   err = true;
                }

                if (err || debugLevel < MSG_NONE || debugLevel > MSG_TRACE)
                {
                   syntax("Invalid debuglevel value:  " + args[i + 1], -4);
                }

                metd = true;
                i++;
                count++;

                continue;
             }

             // Call graph mode argument.
             if (args[i].equalsIgnoreCase("-c"))
             {
                if (metc)
                {
                   syntax("Only one -c option is allowed", -5);
                }
                callGraphMode = true;
                metc = true;

                continue;
             }

             // Explicit file list mode argument.
             if (args[i].equalsIgnoreCase("-f"))
             {
                if (metf)
                {
                   syntax("Only one -f option is allowed", -6);
                }
                explicit = true;
                metf = true;

                continue;
             }

             // Honor hidden mode argument.
             if (args[i].equalsIgnoreCase("-h"))
             {
                if (meth)
                {
                   syntax("Only one -h option is allowed", -7);
                }
                honorHidden = true;
                meth = true;

                continue;
             }

             // Read-only mode.
             if (args[i].equalsIgnoreCase("-r"))
             {
                if (metr)
                {
                   syntax("Only one -r option is allowed", -8);
                }
                readOnly = true;
                metr = true;

                continue;
             }

             // configuration profile
             if (args[i].equalsIgnoreCase("-profile"))
             {
                if (cfgProfile != null)
                {
                   syntax("Only one -profile option is allowed", -12);
                }

                i++;
                count++;
                if (i >= args.length)
                {
                   syntax("The -profile option requires a profile <name> argument", -13);
                }

                cfgProfile.add(args[i]);

                continue;
             }

             syntax("Invalid option:  " + args[i], -9);
          }

          // Process variable initializer overrides.
          HashMap<String, Object> initializers = null;
          while (count < len)
          {
             String arg = args[count];
             int pos = arg.indexOf("=");
             if (pos < 0)
             {
                break;
             }

             String key = arg.substring(0, pos);
             String value = arg.substring(pos + 1);

             if (initializers == null)
             {
                initializers = new HashMap<>();
             }

             initializers.put(key, value);

             count++;
          }

          if (len <= count)
          {
             syntax(null, -10);
          }

          String profile = args[count++];

          // Process AST filespec arguments.
          String directory = null;
          String fileSpec = null;
          String[] filelist = null;

          if (explicit)
          {
             filelist = new String[len - count];

             for (int j = 0; count < len; j++, count++)
             {
                filelist[j] = args[count];
             }
          }
          else
          {
             if (len == count + 2)
             {
                directory = args[count++];
                fileSpec = args[count++];
             }

             if (len > count)
             {
                syntax(null, -11);
             }
          }

          // set the configuration profile
          if (cfgProfile != null)
          {
             Configuration.setDefaultProfiles(cfgProfile);
          }

          try
          {
             // Instantiate pattern engine and run...
             PatternEngine engine = new PatternEngine(directory, fileSpec);
             setDebugLevel(debugLevel);

             if (explicit)
             {
                engine.addAstSpec(new ExplicitFileList(filelist));
             }

             if (callGraphMode)
             {
                // TODO: should this be switched to CallGraphHelper.getGraph() which doesn't create indexes?
                engine.setGraph(CallGraphWorker.getGraph());
             }
             engine.setHonorHidden(honorHidden);
             engine.setReadOnly(readOnly);
             engine.setVariableInitializers(initializers);
             engine.run(profile);
          }
          catch (Exception exc)
          {
             if (debugLevel >= MSG_DEBUG)
             {
                // prints the exception with causes
                LOG.log(Level.SEVERE, "ERROR:", exc);
             }
             else
             {
                // for MSG_STATUS prints only the direct stacktrace
                StringBuilder sb = new StringBuilder("ERROR:");
                sb.append(System.lineSeparator());
                dumpThrowableNoCause(sb, exc);
                LOG.log(Level.SEVERE, sb.toString());
             }
          }
          finally
          {
             CallGraphHelper.shutdownGraph();
          }
       }

       /**
        * Dump the details of the throwable into the given string buffer without including the cause.
        *
        * @param    sb
        *           The string builder the throwable to be appended to.
        * @param    t
        *           The throwable to dump.
        */
       private static void dumpThrowableNoCause(StringBuilder sb, Throwable t)
       {
          sb.append(t.getClass().getName());
          String msg = t.getMessage();
          if (msg != null && msg.trim().length() > 0)
          {
             sb.append(": ");
             sb.append(msg);
          }
          sb.append(System.lineSeparator());

          StackTraceElement[] stackTraceElements = t.getStackTrace();
          for (int i = 0; i < stackTraceElements.length; i++)
          {
             StackTraceElement stackTraceElement = stackTraceElements[i];
             sb.append("        at ");
             sb.append(stackTraceElement);
             if (i < stackTraceElements.length - 1)
             {
                sb.append(System.lineSeparator());
             }
          }
       }

       /**
        * Container for an arbitrary category and id that is supposed to be used as a key for the
        * {@link PatternEngine#storedObjects} map.
        */
       public static class StorageKey
       {
          /** Arbitrary category. <code>null</code> if no category is specified. */
          private final String category;

          /** Arbitrary id. */
          private final String id;

          /**
           * Creates a key with no category specified.
           *
           * @param id
           *        Arbitrary id.
           */
          public StorageKey(String id)
          {
             this(null, id);
          }

          /**
           * Convenience constructor.
           *
           * @param category
           *        Arbitrary category.
           * @param id
           *        Arbitrary id.
           */
          public StorageKey(String category, String id)
          {
             this.category = category;
             this.id = id;
          }

          /**
           * Calculate a hash code for this object.
           *
           * @return  hash code.
           */
          public int hashCode()
          {
             int result = 17;

             result = 37 * result + (category == null ? 0 : category.hashCode());
             result = 37 * result + (id == null ? 0 : id.hashCode());

             return result;
          }

          /**
           * Test this object for equality (equivalence) with the given object.
           *
           * @param   o
           *          Object to test.
           *
           * @return  <code>true</code> if this object is the same instance as the
           *          given object, or it is considered equivalent to the given
           *          object.  <code>StorageKey</code> instances are equivalent if
           *          they contain the same category and id.
           */
          public boolean equals(Object o)
          {
             if (!(o instanceof StorageKey))
             {
                return false;
             }

             if (o == this)
             {
                return true;
             }

             StorageKey other = (StorageKey) o;
             return (category == null ? other.category == null :
                                        category.equals(other.category)) &&
                    (id == null ? other.id == null :
                                   id.equals(other.id));
          }

          /**
           * Produce a string representation of the internal state of this object.
           *
           * @return  String representation of this object.
           */
          public String toString()
          {
             return "[" + category + ", " + id + "]";
          }
       }
    }