PatternEngine.java
/*
** Module : PatternEngine.java
** Abstract : Main driver for the pattern rules framework
**
** Copyright (c) 2005-2023, Golden Code Development Corporation.
**
** -#- -I- --Date-- --JPRM-- -----------------------------------Description-----------------------------------
** 001 ECF 20050301 @20131 Created initial version. Provides a command
** line interface and API to apply a pipeline of
** rulesets against ASTs.
** 002 ECF 20050302 @20150 Fixed defect in finish method. Made sure
** resolver has no ASTs in scope when global,
** post-rules are applied.
** 003 ECF 20050310 @20255 Added -Q command line option and quiet mode
** support. When not in quiet mode, engine now
** reports to stdout the name of each source
** file as it is processed.
** 004 GES 20050310 @20261 Added new default workers for Dictionary
** and FileOperations.
** 005 ECF 20050310 @20264 Added support for global includes of named
** expression libraries. Previously these were
** only possible at the ruleset level.
** 006 ECF 20050310 @20275 Added support for literal resolution.
** AstSymbolResolver uses this support to
** resolve literals for the expression parser.
** 007 GES 20050311 @20294 Added a more generic run() method that
** takes both a profile and a caller created
** iterator. The original run method is left
** alone because it MUST implement the
** initialize() call because this is how the
** astSpecs get initialized. A better approach
** would be to create a more generic, common
** run method but this would require more
** significant changes to the configuration
** loading. Instead we maintain the old
** approach (unchanged) while opening up a new
** option for external callers of the pattern
** engine. The new option allows the iterator
** to be created and maintained elsewhere which
** allows for dynamic modifications of that
** iterator while inside the pattern engine's
** core run() loop that calls processAst().
** 008 ECF 20050314 @20329 Added support for user variables. Variables
** have one of two scopes: global (for variables
** defined in the main, pipeline profile) and
** local (for variables defined within an
** individual ruleset). Default values may be
** overridden at the command line.
** 009 ECF 20050315 @20340 Added astSpecs method. Provides an iterator
** over the FileList objects used to find ASTs.
** 010 ECF 20050315 @20344 Added callback to notify pattern worker just
** before a new AST is processed. Calls visitAst
** method for each pattern worker.
** 011 ECF 20050315 @20408 Call graphs (really trees) are constructed
** from persistence, based on either the root
** node list or AST specs from the profile or
** command line. Each call tree is walked and
** two new lists of rules (ascent-rules and
** descent-rules) can be defined, both at the
** global and ruleset levels, to trigger actions
** based upon changes in AST scope during the
** walk.
** 012 GES 20050323 @20474 Converted use of AnnotatedAst copy
** constructor to the use of the Aast duplicate
** method, which does the same job but allows
** for better abstraction.
** 013 ECF 20050328 @20518 Changed references to "target" AST to "copy"
** AST, to reflect similar changes in the symbol
** resolver implementation. Added a call to
** resolver's cleanUp method after a ruleset has
** processed an AST hierarchy, to enable
** resolver to clean up internal resources.
** 014 ECF 20050415 @20741 Removed use of AstGenerator. Use AstPersister
** instead to allow ASTs of any kind to be
** processed. AST specs refer to persisted AST
** files directly, rather than indirectly via
** source code files.
** 015 ECF 20050422 @20852 Introduced pattern worker namespaces. Changed
** resolveToLiteral method to accept and use an
** optional namespace parameter. Changed pattern
** worker registration methods to also use an
** optional namespace parameter. Namespaces
** enable us to prevent collisions between
** pattern workers when resolving constants, if
** multiple workers can resolve a constant to
** potentially different literals.
** 016 ECF 20050423 @20858 Disallow registration of multiple instances
** of the same pattern worker. If an instance of
** a particular pattern worker class is already
** registered, subsequent attempts to register
** that worker type are silently ignored. This
** allows individual rulesets to load pattern
** workers as necessary, without overlap.
** 017 GES 20050428 @20919 Added global next-child rules when walking
** call graphs. These rules will be triggered
** with the parent node in scope when moving
** between children (but not on the move to
** the first child or from the last child).
** 018 ECF 20050430 @20966 Changes related to namespace support for user
** functions and variables. Moved all pattern
** worker management to AstSymbolResolver.
** Constant resolution is now also handled by
** the resolver. Configuration of the resolver
** is now handled at the superclass level.
** 019 ECF 20050526 @21300 Changes to support new expression engine
** implementation. Reimplemented variable
** registration to account for new variable
** initializers feature. Modified command line
** arguments to allow for a user-defined debug
** reporting level. Ensure default pattern
** workers are registered under valid names.
** 020 SIY 20050531 @21336 Exclude non-existent files from the
** processing. Some other minor cleanups.
** 021 ECF 20050601 @21344 Don't add AST spec if fileSpec is an empty
** string. This triggered default behavior in
** FileList which searches for everything under
** the specified directory.
** 022 GES 20050602 @21380 Moved some processing (keeping a list of
** RuleSets) into RuleContainer to provide for
** arbitrary nesting of rule sets.
** 023 ECF 20050616 @21503 Refactored initialize, run, and finish
** methods. The methods initialize and finish
** are now public and no longer apply global
** init and post rules. The version of the run
** method which accepted a file iterator is now
** renamed processFiles. This method applies
** global init and post rules directly (as does
** the remaining run method). The processFiles
** method is designed to be safely run multiple
** times between bracketed (external) calls to
** initialize and finish. The run method remains
** as a convenience to bundle initialization,
** AST processing and termination processing for
** a single set of ASTs.
** 024 ECF 20050716 @21733 Added support for ruleset maximum walk depth.
** This is an optimization which avoids visiting
** unnecessary levels of an AST when applying a
** specific ruleset.
** 025 GES 20050728 @21900 Added support for the global setting of
** honoring the hidden flag of an AST during
** the walk. Defaults to false, but when
** enabled, non-root AST nodes that are accessed
** via a non-view iterator will drop from the
** walk if their hidden flag is set.
** 026 ECF 20050811 @22086 Added support for read-only mode. When in
** read-only mode, the AST walker is handed the
** source AST as both its source and copy AST
** (i.e., there is no copy). Also, the copy ID
** map is empty. This mode gives us a moderate
** performance enhancement when running rulesets
** which do not modify and persist the copy AST
** as part of their work (e.g., reporting).
** 027 ECF 20050819 @22173 Prevent duplicate AST processing and permit
** premature termination of a walk. A set of
** processed AST IDs is maintained in call graph
** mode to prevent the same AST branch from
** being processed twice. Also, a new flag in
** the symbol resolver indicates whether a walk
** should be ended prematurely. This is set by
** CommonAstSupport and checked and reset during
** processAst in the pattern engine.
** 028 GES 20051005 @22976 Added an explicit file list mode to the
** command line interface.
** 029 ECF 20051201 @23760 Modified read only mode. AstRegistry is only
** saved when not in read only mode.
** 030 GES 20060129 @24133 FileList related changes.
** 031 GES 20060129 @24139 Simple change that allows multiple different
** pattern engines to sequentially run.
** 032 ECF 20060421 @25644 New debug feature. Any uncaught error thrown
** within the run() method now triggers a report
** about the most recently active rule.
** 033 GES 20070511 @33477 Exposed variable initialization as a public
** method.
** 034 ECF 20080730 @39269 Reduced memory footprint. Ensure rule list
** elements are cleaned up after a run. Ensure
** non-global ruleset variables are cleared
** after the ruleset has been applied. Release
** resources to allow GC after a run.
** 035 GES 20090421 @41827 Matched utility class package change.
** 036 GES 20090429 @42048 Match package and class name changes.
** 037 GES 20090515 @42213 Moved to AstManager from AstRegistry/AstPersister.
** 038 GES 20130203 Moved to using an Initializer class to allow a
** simple instance to be assigned to a new variable
** in addition to the previous expression approach.
** 039 SVL 20130624 Allow pattern engine to manipulate ASTs in memory rather than
** read/write them using file system.
** 040 CA 20131022 Added conditional rule-set processing.
** 041 OM 20140122 Moved finish() into a finally to be sure the taken resources are
** freed when leaving the run(String profile).
** 042 CA 20140313 Changed callgraph generation to use a callgraph DB.
** 043 SVL 20140320 Minor improvement to error messages.
** 044 CA 20140408 Fixed filename matching for source-code originating from a different
** OS than the one where conversion is ran.
** 045 ECF 20140403 Added special mode and c'tor for runtime conversion use; implemented
** additional generics.
** 046 ECF 20150331 Added variant of storeObject to allow passing of ConversionPool task
** results to the next pattern engine in a pipeline.
** 047 ECF 20150413 Very minor format cleanup.
** 048 ECF 20150715 Replace StringBuffer with StringBuilder.
** 049 EVL 20160225 Javadoc fixes to make compatible with Oracle Java 8 for Solaris 10.
** 050 GES 20170509 Changes for call graph v3.
** ECF 20170607 Use TreeWalkException in place of RuntimeException.
** 051 CA 20170825 Added a mechanism to inject new ASTs into the source list, generated
** by the TRPL rules.
** 052 ECF 20171228 Improve performance by reducing context-local lookups.
** 053 ECF 20190620 EmptyIterator API change.
** 054 OM 20210923 Added -profile command line option for specifying the configuration profile.
** CA 20211214 Fixed cleanup in case the conversion abends.
** OM 20211215 Print the descriptive text of the root node if file name is not available.
** OM 20220405 XmlFilePlugin can load resources from application's jar.
** CA 20221010 Avoid using iterators on ArrayList - instead, use a 'for' loop and get the
** element by index, as iterators are more expensive in terms of memory and
** performance. Refs #6813
** TJD 20220504 Upgrade do Java 11 minor changes
** CA 20220501 Each profile has the same structure as the main 'global' config. Allow multiple
** profiles to be ran at once, with the conversion switching the state between
** profiles, when a resource (like a file or namespace) is being processed. Only
** front phase is supported at this time.
** ECF 20220515 Added PatternWorker.leaveAst hook to allow for post-AST walk processing.
** CA 20221010 Avoid using iterators on ArrayList - instead, use a 'for' loop and get the
** element by index, as iterators are more expensive in terms of memory and
** performance. Refs #6813
** SVL 20230108 Improved performance by replacing some "for-each" loops with indexed "for" loops.
** 055 GBB 20230512 Logging methods replaced by CentralLogger/ConversionStatus.
** 056 CA 20231129 Small changes to avoid 'invokeinterface' in the compiled bytecode.
*/
/*
** This program is free software: you can redistribute it and/or modify
** it under the terms of the GNU Affero General Public License as
** published by the Free Software Foundation, either version 3 of the
** License, or (at your option) any later version.
**
** This program is distributed in the hope that it will be useful,
** but WITHOUT ANY WARRANTY; without even the implied warranty of
** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
** GNU Affero General Public License for more details.
**
** You may find a copy of the GNU Affero GPL version 3 at the following
** location: https://www.gnu.org/licenses/agpl-3.0.en.html
**
** Additional terms under GNU Affero GPL version 3 section 7:
**
** Under Section 7 of the GNU Affero GPL version 3, the following additional
** terms apply to the works covered under the License. These additional terms
** are non-permissive additional terms allowed under Section 7 of the GNU
** Affero GPL version 3 and may not be removed by you.
**
** 0. Attribution Requirement.
**
** You must preserve all legal notices or author attributions in the covered
** work or Appropriate Legal Notices displayed by works containing the covered
** work. You may not remove from the covered work any author or developer
** credit already included within the covered work.
**
** 1. No License To Use Trademarks.
**
** This license does not grant any license or rights to use the trademarks
** Golden Code, FWD, any Golden Code or FWD logo, or any other trademarks
** of Golden Code Development Corporation. You are not authorized to use the
** name Golden Code, FWD, or the names of any author or contributor, for
** publicity purposes without written authorization.
**
** 2. No Misrepresentation of Affiliation.
**
** You may not represent yourself as Golden Code Development Corporation or FWD.
**
** You may not represent yourself for publicity purposes as associated with
** Golden Code Development Corporation, FWD, or any author or contributor to
** the covered work, without written authorization.
**
** 3. No Misrepresentation of Source or Origin.
**
** You may not represent the covered work as solely your work. All modified
** versions of the covered work must be marked in a reasonable way to make it
** clear that the modified work is not originating from Golden Code Development
** Corporation or FWD. All modified versions must contain the notices of
** attribution required in this license.
*/
package com.goldencode.p2j.pattern;
import java.io.*;
import java.text.*;
import java.util.*;
import java.util.logging.*;
import com.goldencode.ast.*;
import com.goldencode.p2j.cfg.*;
import com.goldencode.p2j.convert.*;
import com.goldencode.util.*;
import com.goldencode.p2j.uast.*;
import com.goldencode.p2j.util.*;
import org.apache.tinkerpop.gremlin.structure.*;
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.*;
/**
* An engine which applies rules to the nodes of an abstract syntax tree
* (AST), in order to analyze or transform it, based upon patterns detected
* in the AST data. Rules are organized into rulesets, and rulesets are
* organized into a pipeline. Pipelines are applied against one or more
* AST hierarchies, which are ultimately created by either:
* <ul>
* <li>deriving them from source code files written using the well-defined
* syntax of a programming language; or
* <li>building them up dynamically in memory, based upon some well-defined
* structure.
* </ul>
* A pattern engine profile is used to organize the information managed by
* the pattern engine for a particular application. A profile defines:
* <ul>
* <li>the worker objects used to support the desired application;
* <li>the ASTs to be processed and the order in which they are to be
* processed;
* <li>the pipeline of rulesets to be applied against the ASTs, and the order
* in which they are to be applied;
* <li>the rules contained in each ruleset, and the timing and order in which
* these rules are to be applied against the ASTs;
* <li>the conditions defined for each rule;
* <li>the list of actions to be executed conditionally for each rule.
* </ul>
* A {@link ConfigLoader} object is used to load a profile into the engine,
* or a profile may be configured programatically using the public methods
* of this class.
* <p>
* A profile can be defined to perform a variety of analysis or transformation
* applications. This is done via the {@link #run} method. The logical program
* flow (assuming a pre-defined profile is being used) is as follows:
* <ul>
* <li>begin
* <li>perform global initialization processing:
* <ul>
* <li>create a new symbol resolver
* <li>initialize and register default worker objects
* <li>load the pattern configuration profile
* <li>set a default specification to find ASTs if not provided by profile
* <li>apply global, user-defined init-rules
* </ul>
* <li>for each AST to be processed:
* <ul>
* <li>for each ruleset defined in the pipeline:
* <ul>
* <li>make a deep copy of the source AST hierarchy; this copy becomes
* the editable source copy for the current ruleset pass and becomes
* the source AST for the next ruleset in the pipeline
* <li>perform ruleset-specific initialization processing:
* <ul>
* <li>configure the symbol resolver for the current ruleset
* <li>set the source and copy root ASTs as the current ASTs in the
* symbol resolver
* <li>apply user-defined init-rules for the ruleset
* </ul>
* <li>walk the AST (or the filtered view of results of previous ruleset),
* applying all walk-rules of the ruleset to the current AST
* <li>perform ruleset-specific termination processing:
* <ul>
* <li>set the source and copy root ASTs as the current ASTs in the
* symbol resolver
* <li>apply user-defined post-rules for the ruleset
* </ul>
* <li>use the copy AST from the last ruleset as the source AST for the
* next ruleset
* </ul>
* </ul>
* <li>perform global termination processing:
* <ul>
* <li>apply global, user-defined post-rules
* <li>call each registered worker object's termination processing hook
* <li>null out the symbol resolver
* </ul>
* <li>end
* </ul>
* <p>
* In cases where multiple file lists must be processed using a single
* profile, the caller may choose to use the following idiom instead:
* <ol>
* <li>call {@link #initialize} <b>once only</b>.
* <li>call {@link #processFiles} one or more times.
* <li>call {@link #finish} <b>once only</b>.
* </ol>
* This results in a slightly different logic flow:
* <ul>
* <li>begin
* <li>perform global initialization processing (single, external call):
* <ul>
* <li>create a new symbol resolver
* <li>initialize and register default worker objects
* <li>load the pattern configuration profile
* <li>set a default specification to find ASTs if not provided by profile
* </ul>
* <li>for each file list to be iterated and processed (possibly multiple,
* external calls):
* <ul>
* <li>apply global, user-defined init-rules
* <li>for each ruleset defined in the pipeline:
* <ul>
* <li>make a deep copy of the source AST hierarchy; this copy becomes
* the editable source copy for the current ruleset pass and becomes
* the source AST for the next ruleset in the pipeline
* <li>perform ruleset-specific initialization processing:
* <ul>
* <li>configure the symbol resolver for the current ruleset
* <li>set the source and copy root ASTs as the current ASTs in the
* symbol resolver
* <li>apply user-defined init-rules for the ruleset
* </ul>
* <li>walk the AST (or the filtered view of results of previous ruleset),
* applying all walk-rules of the ruleset to the current AST
* <li>perform ruleset-specific termination processing:
* <ul>
* <li>set the source and copy root ASTs as the current ASTs in the
* symbol resolver
* <li>apply user-defined post-rules for the ruleset
* </ul>
* <li>use the copy AST from the last ruleset as the source AST for the
* next ruleset
* </ul>
* <li>apply global, user-defined post-rules
* </ul>
* <li>perform global termination processing (single, external call):
* <ul>
* <li>call each registered worker object's termination processing hook
* <li>null out the symbol resolver
* </ul>
* <li>end
* </ul>
* <p>
* The pattern engine can walk either an arbitrary list of ASTs (default
* mode) or can traverse a call graph (a tree, really). In call graph mode,
* the engine uses one or more file specifications to load persisted call
* graph profiles and then walks the call trees defined by these profiles.
* If no call graph profiles are specified, it will try to load the project-
* specific root node list, which is a list of source file paths which represent
* all known program entry points for the target application.
* <p>
* <strong>Note:</strong> at this time, the pattern engine is <b>not</b>
* designed to process multiple profiles in parallel. However, it IS possible
* to run multiple pattern engines (with different profiles if desired)
* SEQUENTIALLY as long as the <code>initialize</code> method is called at
* the beginning and the <code>finish</code> method is called at the
* end of each run AND in each case a different instance of the pattern
* engine is used. Note that usage of the {@link #run} method naturally
* handles the proper bracketing of <code>initialize</code> and
* <code>finish</code>. Multiple instances of this class should NOT be
* created at the same time. The singleton pattern is not explicitly
* enforced at this time, as this limitation may be lifted with further
* development.
*
* @see com.goldencode.p2j.uast.RootNodeList
*/
public final class PatternEngine
extends RuleContainer
{
/** List of AST filespecs used to locate persisted AST files. */
private final ArrayList<FileList> astSpecs = new ArrayList<>();
/** List of newly created AST files by the rules processed by this engine. */
private final ArrayList<String> newProgramFiles = new ArrayList<>();
/** List of ASTs to be processed. */
private final ArrayList<Aast> targetAsts = new ArrayList<>();
/** List of explicit target AST names. */
private final ArrayList<String> targetNames = new ArrayList<>();
/** Logger. */
private static final ConversionStatus LOG = ConversionStatus.get(PatternEngine.class);
/** Default root directory used to locate persisted AST files. */
private String directory = null;
/** Filespec used to locate persisted AST files relative to a directory. */
private String fileSpec = null;
/** Object used by rules to resolve expression variables and functions. */
private AstSymbolResolver resolver = null;
/** Current AST results view from last ruleset processing in pipeline. */
private Iterator<Aast> view = EmptyIterator.get();
/** Set of visited AST IDs for use with call graph mode to prevent dups */
private HashSet<Long> processedAsts = new HashSet<>();
/** Graph DB instance. */
private Graph graph = null;
/** Honor hidden processing mode flag. */
private boolean honorHidden = false;
/** Operate in read-only mode; do not copy source AST for each ruleset */
private boolean readOnly = false;
/** Mapping of user variable names to initializer expression strings or values. */
private Map<String, Object> varInitializers = null;
/** Ordered set of target paths of files to be processed. */
private final LinkedHashSet<String> targetPaths = new LinkedHashSet<>();
/** Rule currently being (or most recently) processed */
private Rule activeRule = null;
/** System time at beginning of run. */
private long jobStartTime = 0L;
/** Map to store resulting objects like ASTs or streams during rules pipeline processing. */
private final HashMap<StorageKey, Object> storedObjects = new HashMap<>();
/** Flag indicating that conditional rule-set processing is activated or not. */
private boolean conditionalRuleSets = false;
/** The flags which must be set at the rule-set for the rule-set to be processed. */
private final HashSet<String> activatedFlags = new HashSet<>();
/**
* In callgraph walking mode, this will be used to filter the nodes which will be processed.
*/
private GraphNodeFilter nodeFilter = null;
static
{
try
{
String name = Configuration.getParameter("registry");
AstManager.initialize(() -> new XmlFilePlugin(name, Configuration.isImportMode()));
}
catch (AstException ae)
{
// TODO: log this? get ready for null pointer exceptions later
}
}
/**
* Default constructor.
*/
public PatternEngine()
{
this(null, null);
}
/**
* Constructor which accepts a directory and filespec to use instead of
* those (if any) contained in a pattern profile.
*
* @param directory
* Directory used to find ASTs to load; will override any
* provided in the pattern profile if not <code>null</code>.
* @param fileSpec
* File specification used to find ASTs to load; will override
* any provided in the pattern profile if not <code>null</code>.
*/
public PatternEngine(String directory, String fileSpec)
{
super(null);
this.directory = directory;
this.fileSpec = fileSpec;
}
/**
* A constructor for runtime conversion use to support a limited use mode, which does not
* support some options, such as callgraph mode.
*
* @param profile
* Name of the profile to be loaded. It is up to the {@link ConfigLoader}
* implementation to understand what to do with this name.
* @param conditional
* If <code>true</code>, enable conditional rule-set loading for runtime query mode.
*
* @throws IllegalStateException
* if invoked when not in runtime use mode.
* @throws ConfigurationException
* if there is an error loading the profile.
*/
public PatternEngine(String profile, boolean conditional)
throws ConfigurationException
{
this(null, null);
if (!Configuration.isRuntimeConfig())
{
throw new IllegalStateException("This pattern engine mode is only for use at runtime");
}
if (conditional)
{
setConditionalRuleSets(true);
addConditionalFlag(ConversionFlag.RUNTIME_QUERY.toString());
}
// load and initialize the profile
initialize(profile);
// temporarily null out the context-local resolver; this will be set at each invocation of
// runInMemory
AstSymbolResolver.reset();
}
/**
* Adds a flag to the {@link #activatedFlags} set, which will be checked only when
* {@link #conditionalRuleSets} is activated.
* <p>
* Use {@link #setConditionalRuleSets(boolean)} to activate these flags before starting
* conversion.
*
* @param flag
* the conditional flag to add.
*/
public void addConditionalFlag(String flag)
{
activatedFlags.add(prepareFlag(flag));
}
/**
* Removes a flag from the {@link #activatedFlags} set.
* <p>
* Use {@link #setConditionalRuleSets(boolean)} to deactivate these flags after conversion is
* finished.
*
* @param flag
* the conditional flag to add.
*/
public void removeConditionalFlag(String flag)
{
activatedFlags.remove(prepareFlag(flag));
}
/**
* Enable or disable conditional rule-sets mode.
* <p>
* When activated, only rule-sets marked with flags part of the {@link #activatedFlags}
* set will be included.
*
* @param state
* The new state of the conditional rule-sets mode.
*/
public void setConditionalRuleSets(boolean state)
{
conditionalRuleSets = state;
}
/**
* Check if conditional rule-sets is activated.
*
* @return The state of the {@link #conditionalRuleSets} flag.
*/
public boolean isConditionalRuleSets()
{
return conditionalRuleSets;
}
/**
* Check if the given flag is part of the {@link #activatedFlags} set.
*
* @param flag
* The flag to check.
*
* @return <code>true</code> if {@link #conditionalRuleSets conditional rule-set} processing
* is not activated or the flag is part of the {@link #activatedFlags} set.
* <code>false</code> if the flag is not part of the {@link #activatedFlags} set.
*/
public boolean isFlagActivated(String flag)
{
return !conditionalRuleSets || activatedFlags.contains(prepareFlag(flag));
}
/**
* Check if the given flags are part of the flags registered in the {@link #activatedFlags} set.
* <p>
* The flags will be checked only when {@link #conditionalRuleSets} is activated.
*
* @param flags
* The flags to check.
*
* @return <code>true</code> if at least one flag is part of the
* {@link #activatedFlags} set or if {@link #conditionalRuleSets} is not
* activated.
* <code>false</code> if no flag from the given set is part of the
* {@link #activatedFlags} set.
*/
public boolean activeFlags(Set<String> flags)
{
if (!conditionalRuleSets)
{
return true;
}
for (String flag : flags)
{
if (isFlagActivated(flag))
{
return true;
}
}
return false;
}
/**
* Set the Graph DB instance used by this pattern engine. A non-null {@link #graph} instance
* will enable callgraph processing mode.
*
* @param graph
* The graph DB instance.
*/
public void setGraph(Graph graph)
{
this.graph = graph;
}
/**
* Gets the flag that determines if the hidden status of an AST node
* should be honored.
*
* @return <code>true</code> if a node's hidden flag should be honored.
*/
public boolean isHonorHidden()
{
return honorHidden;
}
/**
* Sets the flag that determines if the hidden status of an AST node
* should be honored.
*
* @param hide
* <code>true</code> if a node's hidden flag should be honored.
*/
public void setHonorHidden(boolean hide)
{
honorHidden = hide;
}
/**
* Set state of read-only mode. When in read-only mode, a copy of the
* source AST is <i>not</i> made before each AST walk. By default,
* read-only mode is turned off and a copy is made. Read-only mode is
* available as an optimization to avoid the overhead of this potentially
* expensive operation if it is not needed.
* <p>
* In read-only mode, the source AST is used as both the source and copy
* AST for each rule-set processed. Rule-sets should never modify the AST
* being walked when read-only mode is enabled. Note that currently there
* is no enforcement of this contract. It is on the "honor system" only.
*
* @param readOnly
* <code>true</code> to enable read-only mode; <code>false</code>
* to disable it.
*/
public void setReadOnly(boolean readOnly)
{
this.readOnly = readOnly;
}
/**
* Store a mapping of user variable names to initializer expression
* overrides (in infix notation), as extracted from the command line
* arguments passed to this process.
*
* @param initializers
* Key/value pairs of variable names to initializer expression
* strings. The strings must represent valid expressions in infix
* notation.
*/
public void setVariableInitializers(Map<String, Object> initializers)
{
varInitializers = initializers;
}
/**
* Add a directory and file specification pair which will allow the engine
* to locate a set of persisted AST files to be processed into ASTs, which
* will in turn be applied against a pipeline of rulesets. Any number of
* such pairs may be added; however, if this object was constructed with
* an overriding pair, calls to this method will be ignored, and the
* overriding pair will be used instead.
*
* @param directory
* Relative or absolute directory in which to begin a recursive
* search for AST files matching the file specification defined
* by <code>fileSpec</code>. If relative, the path will be
* prepended with the <code>P2J_HOME</code> path.
* @param fileSpec
* A regular expression indicating a filename mask to match when
* searching for persisted AST files to process for the current
* pattern matching session.
* @param force
* <code>true</code> to force the spec to be added.
*
* @throws ConfigurationException
* if <code>P2J_HOME</code> has not been defined for the current
* process.
*/
public void addAstSpec(String directory, String fileSpec, boolean force)
throws ConfigurationException
{
// An empty string for fileSpec is meaningless.
if (fileSpec != null && fileSpec.trim().length() == 0)
{
return;
}
// Ignore parameters if the user has overridden from the constructor.
if (this.directory != null && !force)
{
return;
}
File dir = Configuration.toFile(directory);
astSpecs.add(new FileSpecList(dir, fileSpec, true));
}
/**
* Add a list of persisted AST files to be processed into ASTs, which
* will in turn be applied against a pipeline of rulesets. Any number of
* such lists may be added.
*
* @param list
* An existing, non-null list of files.
*/
public void addAstSpec(FileList list)
{
// a null list is meaningless
if (list == null)
{
return;
}
astSpecs.add(list);
}
/**
* Add the specified AST name as a target.
*
* @param name
* An AST name.
*/
public void addTarget(String name)
{
targetNames.add(name);
}
/**
* Add AST to be processed against a pipeline of rule sets.
*
* @param ast
* AST to be processed.
*/
public void addTargetAst(Aast ast)
{
targetAsts.add(ast);
}
/**
* Get an iterator over the ast specifications ({@link
* com.goldencode.p2j.util.FileList} objects). Items are iterated in the
* order in which they were added.
*
* @return Iterator over AST spec <code>FileList</code> objects used to
* find ASTs to process.
*/
public Iterator<FileList> astSpecs()
{
return astSpecs.iterator();
}
/**
* Override the superclass' implementation of this method to allow for
* replacement of variables' initializer expressions with values provided
* by the user. Delegates eventual registration of the variable to the
* superclass' implementation.
* <p>
* Register a new user variable with this container. If the variable's
* name is already in use within this rule container's scope, it cannot
* be re-used and this method will fail. If both <code>type</code> and
* <code>expression</code> are provided, <code>type</code> must match the
* return type of <code>expression</code>.
*
* @param name
* Name by which this variable will be referenced in expressions.
* @param type
* Data type of the variable. May be <code>null</code> if and
* only if <code>expression</code> is not <code>null</code>
* <b>and</b> <code>expression</code>'s return type is not
* <code>null</code>.
* @param expression
* Expression which will be used to initialize the variable each
* time it is {@link com.goldencode.expr.Variable#reset reset}
* by the pattern engine. May be <code>null</code> if and only
* if <code>type</code> is not <code>null</code>.
*
* @throws com.goldencode.expr.SymbolException
* if the variable's name is already in use in this container's
* scope.
* @throws com.goldencode.expr.ExpressionException
* if <code>type</code> is a class which is not assignable from
* <code>expression</code>'s return type.
* @throws IllegalArgumentException
* if <code>type</code> is not provided and it cannot be
* determined from <code>expression</code>.
*
* @see com.goldencode.expr.SymbolResolver#resetVariables
*/
@Override
public void registerVariable(String name, Class<?> type, String expression)
{
if (varInitializers != null)
{
Object value = varInitializers.get(name);
if (value != null)
{
if (value instanceof String)
{
// assume this is an override expression
expression = (String) value;
}
else
{
// this is the value itself, not an expression
super.registerVariableValue(name, type, value);
return;
}
}
}
super.registerVariable(name, type, expression);
}
/**
* Perform a discrete pipeline pattern run against a set of target ASTs. This mode runs init
* rules, processes the list of ASTs, then runs post rules. It is intended for runtime
* conversion use in a long-lived pattern engine.
*/
public void run()
{
AstSymbolResolver.setLocal(resolver);
try
{
// run global init rules.
applyGlobal(RULE_INIT);
// Process each in-memory AST
for (int i = 0; i < targetAsts.size(); i++)
{
Aast targetAst = targetAsts.get(i);
processAst(targetAst);
}
// run global post rules.
applyGlobal(RULE_POST);
}
catch (Throwable thr)
{
StringBuilder buf = new StringBuilder();
String sep = System.getProperty("line.separator");
if (activeRule != null)
{
buf.append("ERROR! Active Rule:");
buf.append(sep);
buf.append(activeRule.createReport());
buf.append(sep);
}
else
{
buf.append("ERROR! Active rule could not be determined: ");
buf.append(thr.getMessage());
}
throw new RuntimeException(buf.toString(), thr);
}
finally
{
AstSymbolResolver.reset();
targetAsts.clear();
}
}
/**
* Run the pattern engine once using the specified configuration profile.
* This will load the profile and initialize the pattern engine and its
* supporting workers. It will then apply the pipeline of rulesets defined
* in the profile to every AST it finds based upon the AST specifications
* found in the profile (or provided as overrides from the command line).
* After all ASTs have been processed, each registered pattern worker is
* called to perform its post-processing, if any, and the pattern engine
* itself performs some clean-up processing. Finally, the registry of ASTs
* is saved, in case any changes have occurred.
*
* @param profile
* Name of the profile to be loaded. It is up to the {@link
* ConfigLoader} implementation to understand what to do with
* this name.
*
* @throws ConfigurationException
* if any error occurs loading the specified configuration
* profile.
* @throws AstException
* if any error occurs loading a persisted AST.
*/
public void run(String profile)
throws ConfigurationException,
AstException
{
if (graph != null)
{
String nodeFilterClass = Configuration.getParameter("graph-node-filter", null);
if (nodeFilterClass == null)
{
throw new ConfigurationException(
"The graph-node-filter configuration parameter is not set!");
}
try
{
Class<?> nodeFilterClz = Class.forName(nodeFilterClass);
if (!GraphNodeFilter.class.isAssignableFrom(nodeFilterClz))
{
throw new ConfigurationException(
nodeFilterClz + " does not implement the GraphNodeFilter interface!");
}
this.nodeFilter = (GraphNodeFilter) nodeFilterClz.getDeclaredConstructor().newInstance();
}
catch (ReflectiveOperationException e)
{
throw new ConfigurationException(
"Could not instantiate the node-filter from " + nodeFilterClass, e);
}
}
try
{
// Load and initialize the profile.
initialize(profile);
// Run global init rules.
applyGlobal(RULE_INIT);
Configuration config = Configuration.getInstance();
// Process each stored AST or walk call graph, depending on the mode of operation.
Iterator<String> iter = targetPaths.iterator();
while (iter.hasNext())
{
String path = iter.next();
config.withFileProfile(path);
if (graph != null)
{
Aast ast = AstManager.get().loadTree(path);
long id = ast.getId();
Vertex node = nodeFilter.acceptNode(id);
if (node != null)
{
walkGraph(node);
}
}
else
{
processAst(path);
}
}
// Process each in-memory AST
for (int i = 0; i < targetAsts.size(); i++)
{
Aast targetAst = targetAsts.get(i);
processAst(targetAst);
}
// Run global post rules.
resolver.setAsts(null, null);
applyGlobal(RULE_POST);
}
catch (Throwable thr)
{
StringBuilder buf = new StringBuilder();
String sep = System.getProperty("line.separator");
if (activeRule != null)
{
buf.append("ERROR! Active Rule:");
buf.append(sep);
buf.append(activeRule.createReport());
buf.append(sep);
}
else
{
buf.append("ERROR! Active rule could not be determined: ");
buf.append(thr.getMessage());
}
String msg = buf.toString();
Aast srcAst = resolver.getSourceAst();
if (srcAst != null)
{
TreeWalkException exc = new TreeWalkException(msg, thr);
exc.setSourceAstId(srcAst.getId());
exc.setCopyAstId(resolver.getCopyAst().getId());
exc.setFilename(srcAst.getFilename());
throw exc;
}
else
{
throw new RuntimeException(msg, thr);
}
}
finally
{
// Perform post-processing and cleanup.
finish();
}
}
/**
* This will apply the pipeline of rulesets defined in the currently
* loaded profile to every AST it finds based upon the given iterator
* that lists the <code>File</code> objects that must be processed (and
* the inherent order in which these files must be processed).
* <p>
* It is assumed that the caller is responsible for loading a profile
* before this method is invoked and for conducting termination processing
* afterwards.
*
* @param iter
* An iterator of <code>File</code> objects to process.
*
* @throws AstException
* if any error occurs loading a persisted AST.
*/
public void processFiles(Iterator<File> iter)
throws AstException
{
try
{
// Run global init rules.
resolver.setAsts(null, null);
applyGlobal(RULE_INIT);
// For each file in the list, process the associated AST.
while (iter.hasNext())
{
File next = iter.next();
// Do not try to process non-existent files.
if (next.exists())
{
processAst(next.getAbsolutePath());
}
}
// Run global post rules and clean up resolver.
resolver.setAsts(null, null);
applyGlobal(RULE_POST);
}
finally
{
resolver.cleanUp();
}
}
/**
* Initialize the pattern engine by creating a new symbol resolver
* instance, registering default pattern workers, loading the specified
* profile, ensuring an AST spec is available, and applying all global,
* user-defined initialization rules.
*
* @param profile
* Name of the profile to be loaded. It is up to the {@link
* ConfigLoader} implementation to understand what to do with
* this name.
*
* @throws ConfigurationException
* if any error occurs loading the specified configuration
* profile.
*/
public void initialize(String profile)
throws ConfigurationException
{
jobStartTime = System.currentTimeMillis();
// Clear set of processed ASTs.
if (graph != null)
{
processedAsts.clear();
}
// Create a new AST symbol resolver.
resolver = AstSymbolResolver.create(this);
// Register commonly used pattern workers.
registerDefaultWorkers();
// Load the profile.
ConfigLoader loader = new ConfigLoader(this);
loader.load(profile);
// Add an AST spec using overrides, if no other has been loaded.
if (astSpecs.isEmpty() && directory != null)
{
addAstSpec(directory, fileSpec, true);
}
// Gather set of paths we need to process.
gatherTargetPaths();
}
/**
* Perform cleanup/termination post-processing, including applying all
* global, user-defined post-rules, calling each pattern worker's {@link
* PatternWorker#finish} method, and nulling out the symbol resolver.
*/
public void finish()
throws AstException
{
// Call each worker to hook post-processing.
Iterator<PatternWorker> iter = resolver.workers();
while (iter.hasNext())
{
PatternWorker worker = iter.next();
worker.finish();
}
// Clean up all contained rules and rulesets.
cleanup();
activeRule = null;
processedAsts = new HashSet<>();
varInitializers = null;
// Null out the resolver and clear some static state to allow future pattern engine runs to succeed.
resolver = null;
AstSymbolResolver.reset();
// If not in quiet mode, report elapsed job time.
if (getDebugLevel() >= MSG_STATUS)
{
long elapsed = System.currentTimeMillis() - jobStartTime;
Date date = new Date(elapsed - TimeZone.getDefault().getRawOffset());
SimpleDateFormat format = new SimpleDateFormat("HH:mm:ss.SSS");
System.out.println("Elapsed job time: " + format.format(date));
}
if (!readOnly)
{
AstManager.get().save();
}
}
/**
* Store the given object under the given id.
*
* @param id
* Arbitrary object id.
* @param object
* Object to store.
*/
public void storeObject(String id, Object object)
{
storedObjects.put(new StorageKey(id), object);
}
/**
* Store the given object under the given category and id.
*
* @param category
* Arbitrary object category.
* @param id
* Arbitrary object id.
* @param object
* Object to store.
*/
public void storeObject(String category, String id, Object object)
{
storedObjects.put(new StorageKey(category, id), object);
}
/**
* Get the object stored under the given id.
*
* @param id
* Object id.
*
* @return stored object.
*/
public Object getStoredObject(String id)
{
return storedObjects.get(new StorageKey(id));
}
/**
* Store the given object under the given key.
*
* @param key
* Storage key under which to store the object.
* @param object
* Object to store.
*/
public void storeObject(StorageKey key, Object object)
{
storedObjects.put(key, object);
}
/**
* Get the object stored under the given category and id.
*
* @param category
* Object category.
* @param id
* Object id.
*
* @return stored object.
*/
public Object getStoredObject(String category, String id)
{
return storedObjects.get(new StorageKey(category, id));
}
/**
* Get an unmodifiable map of all objects currently stored with this pattern engine.
*
* @return Map of stored objects.
*/
public Map<StorageKey, Object> getAllStoredObjects()
{
return Collections.unmodifiableMap(storedObjects);
}
/**
* Clear the map of objects currently stored with this pattern engine.
*/
public void clearStoredObjects()
{
storedObjects.clear();
}
/**
* Set the rule which is currently active. This is used for debug
* reporting.
*
* @param activeRule
* Currently active rule.
*/
void setActiveRule(Rule activeRule)
{
this.activeRule = activeRule;
}
/**
* Get the list of newly added program file names.
*
* @return The {@link #newProgramFiles}.
*/
public List<String> getNewProgramFiles()
{
return newProgramFiles;
}
/**
* Clear the list of newly, generated, {@link #newProgramFiles program file names}.
*/
public void clearNewProgramFiles()
{
newProgramFiles.clear();
}
/**
* Add the given filename as being newly generated, so it will be registered and processed
* by other rule-sets, downstream.
*
* @param file
* The new program file name.
*/
public void addNewProgramFile(String file)
{
this.newProgramFiles.add(file);
}
/**
* Prepare the given flag by trimming any whitespace and converting it to uppercase.
*
* @param flag
* The flag to be prepared.
*
* @return The prepared flag.
*/
private String prepareFlag(String flag)
{
return flag.trim().toUpperCase();
}
/**
* Walk the call tree recursively, depth-first, from the specified node.
* At each descent down to a new nesting level, apply the global descent
* rules, if any. At each ascent back up to a previous nesting level,
* apply the global ascent rules, if any. Process the AST represented by
* <code>node</code>, then iterate through <code>node</code>'s children,
* invoking this method recursively for each child found. Apply the
* global next-child rule (with the parent node in scope) when moving
* between any child (but NOT when moving down to the first child which is
* handled by the descend rule OR when moving up from the last child which
* is handled by the ascend rule). In other words, the next-child rule
* will be triggered between the first and second child, the second and
* third child... and when moving between the second-to-last and the last
* child.
*
* @param node
* Call graph node at which to begin the walk. Contains the
* project-relative path of the AST it represents.
*
* @throws AstException
* if any error occurs generating the copy AST.
*/
private void walkGraph(Vertex node)
throws AstException
{
// Apply global AST descent rules.
apply(resolver, RULE_DESCENT);
// Process a Vertex once and only once.
Long id = (Long) node.id();
if (!processedAsts.contains(id) && nodeFilter.acceptNode(node))
{
String filename = node.<String>property("os-filename").value();
Aast ast = AstManager.get().loadTree(filename + ".ast");
processAst(ast);
processedAsts.add(id);
}
else
{
return;
}
boolean firstChild = true;
GraphTraversalSource g = node.graph().traversal();
// GES_TODO: Does it make sense to map the descent, ascent and next-child events to a call
// graph walk? It seems like it is not the best fit, especially since there is no inherent
// ordering of edges. The order is unpredictable and thus the results could be different
// from run to run.
// the idea here is that we only process based on a specific edge label (e.g. "calls" for an
// invocation), otherwise will process "contains", "includes" and all other types of
// relationships and will arrive at entry-point vertices from non-call-sites
Iterator<Vertex> iter = g.V(node).out("calls");
// Process all outbound vertices recursively.
while (iter.hasNext())
{
Vertex vertex = iter.next();
if (!nodeFilter.acceptNode(node))
{
continue;
}
// Determine if this is the first time through the loop.
if (firstChild)
{
// Update our state, subsequent iterations mean we are visiting
// siblings.
firstChild = false;
}
else
{
// Apply global next-child AST rules (only on the movement
// between children --> not on entering the first or leaving the
// last child since descend and ascend respectively can be used
// for those purposes).
apply(resolver, RULE_NEXT_CHILD);
}
walkGraph(vertex);
}
// Apply global AST ascent rules.
apply(resolver, RULE_ASCENT);
}
/**
* Apply the pipeline of rulesets currently loaded against a single AST.
* Each ruleset is applied in turn, in the order in which they were loaded.
* The source AST is provided by the {@link AstManager#loadTree} method. It
* is copied for the first ruleset applied, and any edits are made to the
* copy rather than to the original. Once that ruleset has been fully
* processed, the copy becomes the source AST to the next ruleset in the
* pipeline. In such a way, an AST may be incrementally refined by each
* ruleset and passed on to the next in the pipeline for further
* processing.
*
* @param astName
* Name of the AST persisted file to be processed.
*
* @return Processed copy of AST.
*
* @throws AstException
* if any error occurs generating the copy AST.
*
* @see #apply
*/
private Aast processAst(String astName)
throws AstException
{
// Get the original source AST from persistence.
Aast ast = AstManager.get().loadTree(astName);
return processAst(ast);
}
/**
* Apply the pipeline of rulesets currently loaded against a single AST.
* Each ruleset is applied in turn, in the order in which they were loaded.
* The source AST is provided by the {@link AstManager#loadTree} method. It
* is copied for the first ruleset applied, and any edits are made to the
* copy rather than to the original. Once that ruleset has been fully
* processed, the copy becomes the source AST to the next ruleset in the
* pipeline. In such a way, an AST may be incrementally refined by each
* ruleset and passed on to the next in the pipeline for further
* processing.
*
* @param ast
* AST to be processed.
*
* @return Processed copy of AST.
*
* @throws AstException
* if any error occurs generating the copy AST.
*
* @see #apply
*/
private Aast processAst(Aast ast)
throws AstException
{
// notify all pattern workers that we are about to process the given AST; this is the *source* AST
// instance
Iterator<PatternWorker> wrkIter = resolver.workers();
while (wrkIter.hasNext())
{
PatternWorker worker = wrkIter.next();
worker.visitAst(ast);
}
// report filename being processed if quiet mode is off
if (getDebugLevel() >= MSG_STATUS)
{
String filename = ast.getFilename();
if (filename == null)
{
// print the descriptive text of the root node if file name is not available
System.out.printf("%s (%#016x): Unknown file name. %n",
ast.getDescriptiveTokenText(),
ast.getId());
}
else
{
System.out.println(filename);
}
}
// Apply each ruleset. Unless in read-only mode, a copy is made each
// iteration through the loop, which may be modified by rules. The
// copy becomes the original for the next pass.
ArrayList<RuleListElement> rules = ruleList();
if (rules != null)
{
int size = rules.size();
RuleListElement next;
for (int i = 0; i < size; i++)
{
next = rules.get(i);
if (next.isContainer())
{
RuleSet ruleSet = (RuleSet) next;
ast = apply(ruleSet, ast);
}
}
}
// notify all pattern workers that we have finished processing the given AST; at this point, the ast
// variable refers to the *copy* AST instance
wrkIter = resolver.workers();
while (wrkIter.hasNext())
{
PatternWorker worker = wrkIter.next();
worker.leaveAst(ast);
}
// return the final AST copy, possibly transformed
return ast;
}
/**
* Apply the given ruleset against the specified source AST, using the
* rules and configuration information stored within the ruleset. This
* method first makes a deep copy of the source AST hierarchy, of which
* <code>source</code> is the root node. Each AST node in the copy tree
* is linked by ID to its original counterpart in the source tree, but
* there are no shared object references between the trees, so the copy
* nodes may be edited, moved, or otherwise modified by rules within the
* ruleset without impacting the original, source AST.
* <p>
* Next, the engine's symbol resolver is configured to use the correct
* expression library (that of the current ruleset merged with the global
* expression library). The root source and copy ASTs are set into the
* resolver as the current source and copy AST nodes, respectively. The
* ruleset's user-defined init-rules, if any, are then applied.
* <p>
* Next, the ruleset's walk-rules are applied. Depending upon the {@link
* RuleSet#setInputMode input mode} of the ruleset, this means either the
* entire source tree is walked (<code>RuleSet.INPUT_TREE</code>), or only
* those AST nodes added to the filtered set of results defined by the
* last ruleset processed in this pipeline (if any) are visited, regardless
* of the tree structure (<code>RuleSet.INPUT_VIEW</code>). In either case,
* at each node visited in the source tree, the symbol resolver is updated
* with the current source node and the corresponding copy node. By
* convention, the source node is considered "read-only"; rules must not
* modify it. Rules may make edits to the copy node, including annotating
* it, moving it, or removing it from the copy tree entirely. A reference
* to the copy node may also be added to a collection of results
* maintained by the symbol resolver. An iterator on this collection, is
* stored as an optional input to the next ruleset in the pipeline.
* <p>
* Next, the root source and copy ASTs are again set into the resolver
* as the current source and copy AST nodes, respectively, and the
* ruleset's user-defined post-rules, if any, are then applied.
* <p>
* Finally, the (possibly edited) copy tree is returned, so that it can
* become an input to the next ruleset in the pipeline.
*
* @param ruleSet
* Ruleset to be applied against an AST.
* @param source
* Source AST to be processed by the ruleset.
*
* @return The copy AST root node, after the ruleset has been applied.
*/
private Aast apply(RuleSet ruleSet, Aast source)
{
try
{
// Make a deep copy of the source AST which will become the copy
// AST of this pass, unless in read-only mode.
HashMap<Long, Aast> idMap = new HashMap<>();
Aast copy = source;
if (!readOnly)
{
copy = source.duplicate(idMap);
}
// Configure the resolver for this ruleset pass.
resolver.reset(ruleSet);
resolver.setAsts(source, copy);
// Reset all user variables local to this ruleset to their default
// values.
resolver.resetVariables(ruleSet, false);
// Implement the global honorHidden override (overrides whatever is
// set in the rule-set attribute). Note: unless honorHidden is true,
// whatever is set in the ruleSet's attribute will take precedence.
if (honorHidden)
{
ruleSet.setHonorHidden(true);
}
// Run ruleset's init rules.
ruleSet.apply(resolver, RULE_INIT);
// Initialize the ruleset. If the ruleset indicates a tree walk is
// necessary, and walk has not been ended prematurely, walk the AST,
// applying this ruleset to each node.
if (!resolver.isEndWalk() &&
(ruleSet.hasType(RULE_WALK) ||
ruleSet.hasType(RULE_DESCENT) ||
ruleSet.hasType(RULE_ASCENT) ||
ruleSet.hasType(RULE_NEXT_CHILD)))
{
AstWalker walker = new AstWalker(ruleSet, idMap);
// Walk the tree or the view created by the last ruleset, depending
// upon the input mode.
if (ruleSet.getInputMode() == RuleSet.INPUT_VIEW)
{
view = walker.walk(resolver, view);
}
else
{
view = walker.walk(resolver, source, ruleSet.getDepth());
}
}
if (resolver.isEndWalk())
{
// Reset end walk flag in resolver.
resolver.setEndWalk(false);
// Warn user if message level permits.
if (getDebugLevel() >= MSG_STATUS)
{
System.out.println("AST walk ended prematurely");
}
}
else
{
// Perform any post-processing defined by this ruleset.
resolver.setAsts(source, copy);
ruleSet.apply(resolver, RULE_POST);
}
return copy;
}
finally
{
// Clear all user variables local to this ruleset.
resolver.resetVariables(ruleSet, true);
// Allow resolver to clean up its resources.
resolver.cleanUp();
}
}
/**
* Runs the global rules defined for a pipeline and DOES NOT recursively
* call any rules in any contained ruleset.
*
* @param type
* Either <code>RULE_INIT</code> or <code>RULE_POST</code>.
*/
private void applyGlobal(int type)
{
ArrayList<RuleListElement> rules = ruleList();
if (rules == null)
{
return;
}
int size = rules.size();
RuleListElement next;
for (int i = 0; i < size; i++)
{
next = rules.get(i);
if (!next.isContainer())
{
Rule rule = (Rule) next;
rule.apply(resolver, type);
}
}
}
/**
* Assemble the master set of filenames to be processed. This is a set
* of source filenames if in default mode; it is a set of persisted
* call graph filenames if in call graph mode.
*
* @throws ConfigurationException
* if any error with the configuration is encountered.
*/
private void gatherTargetPaths()
throws ConfigurationException
{
// consolidate all files derived from AST specs into master set.
Iterator<FileList> iter = astSpecs.iterator();
if (Configuration.isImportMode())
{
// assume the path is correct (import only one database at a time) inside jar and just add them
// to [targetPaths]
while (iter.hasNext())
{
FileList next = iter.next();
if (next instanceof FileSpecList)
{
FileSpecList fileList = (FileSpecList) next;
targetPaths.add(fileList.getStartDir() + "/" + fileList.getFileSpec());
}
else
{
File[] files = next.list();
for (File file : files)
{
targetPaths.add(file.getName());
}
}
}
}
else
{
while (iter.hasNext())
{
FileList fileList = iter.next();
File[] files = fileList.list();
int len = files.length;
for (int i = 0; i < len; i++)
{
targetPaths.add(toNormalizedPath(files[i].getAbsolutePath()));
}
}
for (int i = 0; i < targetNames.size(); i++)
{
String name = targetNames.get(i);
targetPaths.add(toNormalizedPath(new File(name).getAbsolutePath()));
}
}
// Warn if nothing to process and not in quiet mode.
if (targetPaths.isEmpty() && targetAsts.isEmpty() && getDebugLevel() >= MSG_STATUS)
{
StringBuilder buf = new StringBuilder();
StringHelper.appendOnNewLine(buf, "*********");
StringHelper.appendOnNewLine(buf, "Warning: no files matching the given criteria were found");
if (!astSpecs.isEmpty())
{
StringHelper.appendOnNewLine(buf, "Specification(s) (relative to project root):");
iter = astSpecs.iterator();
while (iter.hasNext())
{
FileList fileList = iter.next();
if (fileList instanceof FileSpecList)
{
FileSpecList fsl = (FileSpecList) fileList;
String dir = fsl.getStartDir().getAbsolutePath();
buf.append(" Starting Directory: ")
.append(Configuration.normalizeFilename(dir))
.append("; RegEx File Specification: ")
.append(fsl.getFileSpecAsRegEx());
}
else
{
String[] list = fileList.listFilenames();
StringHelper.appendOnNewLine(buf, " List:");
for (int j = 0; j < list.length; j++)
{
buf.append(" ")
.append(Configuration.normalizeFilename(list[j]));
}
}
StringHelper.appendOnNewLine(buf, "");
}
StringHelper.appendOnNewLine(buf, "*********");
LOG.log(Level.WARNING, buf.toString());
}
}
}
/**
* Normalize the specified path if possible, to eliminate unnecessary,
* embedded, relative references and to prepend the project's home
* directory, if <code>path</code> is not absolute. If <code>path</code>
* is already absolute, it is not converted.
*
* @param path
* An absolute path in the project's home directory, or
* <code>path</code> itself if it is already an absolute path.
*
* @throws ConfigurationException
* if the project home directory is not configured.
*/
private String toNormalizedPath(String path)
throws ConfigurationException
{
path = Configuration.normalizeFilename(path);
File file = Configuration.toFile(path);
return file.getAbsolutePath();
}
/**
* Registers all default pattern workers. Default pattern workers are
* those that supply basic services which are likely to be used across a
* wide range of applications. Rather than requiring that they be defined
* in each pattern profile, they are registered once here.
* <p>
* Currently, the default workers are instances of:
* <ul>
* <li>{@link CommonAstSupport}
* <li>{@link PropsWorker}
* <li>{@link DictionaryWorker}
* <li>{@link FileOperationsWorker}
* </ul>
*
* @throws ConfigurationException
* if any exception is thrown by the default constructor of a
* default worker. The exception is caught and chained as the
* root cause of the <code>ConfigurationException</code>.
*/
private void registerDefaultWorkers()
throws ConfigurationException
{
try
{
registerWorker("common", new CommonAstSupport(this));
registerWorker("prop", new PropsWorker());
registerWorker("dict", new DictionaryWorker());
registerWorker("io", new FileOperationsWorker());
}
catch (Exception exc)
{
throw new ConfigurationException("Error initializing pattern engine", exc);
}
}
/**
* Emit a syntax statement and optional error message to {@code stderr}, then exit the process with the
* specified return code. This method is called from {@link #main} when the user passes command line
* parameters which are not viable.
*
* @param msg
* Optional error message; may be {@code null}.
* @param rc
* Process return code used with {@code System.exit()}.
*/
private static void syntax(String msg, int rc)
{
if (msg != null)
{
LOG.log(Level.SEVERE, msg);
}
String[] staticText =
{
"Syntax:",
" java PatternEngine [options]",
" [\"<variable>=<expression>\"...]",
" <profile>",
" [<directory> \"<filespec>\" | <filelist>]",
"",
"where",
" options:",
" -d <debuglevel> Message output mode; debuglevel values:",
" 0 = no message output",
" 1 = status messages only",
" 2 = status + debug messages",
" 3 = verbose trace output",
" -c Call graph walking mode",
" -f Explicit file list mode",
" -h Honor hidden mode",
" -r Read-only mode",
" -profile <name> Sets the configuration profile",
"",
" variable = variable name",
" expression = infix expression used to initialize variable",
" profile = rules pipeline configuration filename",
" directory = directory in which to search recursively for persisted ASTs",
" filespec = file filter specification to use in persisted AST search",
" filelist = arbitrary list of absolute and/or relative file names of " +
"persisted AST files to process (-f mode)"
};
LOG.log(Level.INFO, StringHelper.arrayToString(staticText));
System.exit(rc);
}
/**
* Command line driver to launch the pattern engine, using a pre-defined pattern profile.
*
* @param args
* Expected arguments (in order) are as follows:
* <ul>
* <li>[-options] any combination of the following options (case-insensitive):
* <ul>
* <li>-d debuglevel = level of error/status reporting desired.
* <li>-c = call graph processing mode; use the {@code directory} and {@code filespec}
* parameters (see below) to specify persisted ASTs which should be processed. If
* those parameters are not provided, process the ASTs referenced in the project's
* root node list.
* <li>-f = explicit file list; use the {@code filelist} parameters to specify persisted
* AST files which should be processed.
* <li>-h = turn on honor hidden mode which drops AST nodes from the walk in the case
* where their hidden flag is {@code true} (this honor hidden mode is off by default)
* <li>-profile <name> = sets the configuration profile to {@code name}
* </ul>
* <li>{@code variable=expression} (optional): a key/value pair which overrides the initializer
* expression (if any) of a user variable defined within the configuration profile.
* {@code variable} is the variable name and <code>expression</code> is the initializer
* expression override (which must resolve to a value of the variable's data type).
* <li>{@code profile} (required): name of the pattern configuration profile to be loaded for
* this run.
* <li>{@code directory} (optional): starting directory used for a recursive search for
* persisted AST files to process. Overrides any such data loaded from the profile. If in
* call graph mode, represents the directory
* <li>{@code filespec} (required if and only if {@code directory} was provided): regular
* expression (in double quotes) used as a filename mask in combination with
* {@code directory} to find persisted AST files to process. Overrides any such data loaded
* from the profile.
* <li>{@code filelist} (required if and only if explicit file list mode is enabled): the
* remaining arguments specify an arbitrary list of persisted AST files to process.
* Overrides any such data loaded from the profile.
* </ul>
*/
public static void main(String[] args)
{
int debugLevel = MSG_STATUS;
int len = args.length;
if (len < 1)
{
syntax(null, -1);
}
int count = 0;
boolean metd = false;
boolean metc = false;
boolean metf = false;
boolean meth = false;
boolean metr = false;
boolean callGraphMode = false;
boolean explicit = false;
boolean honorHidden = false;
boolean readOnly = false;
ArrayList<String> cfgProfile = new ArrayList<>();
// Process options first.
for (int i = 0; i < len && args[i].startsWith("-"); i++, count++)
{
// Debug level argument.
if (args[i].equalsIgnoreCase("-d"))
{
if (metd)
{
syntax("Only one -d option is allowed", -2);
}
if (args.length <= i + 1)
{
syntax("-d should be followed by debuglevel argument", -3);
}
boolean err = false;
try
{
debugLevel = Integer.parseInt(args[i + 1]);
}
catch (NumberFormatException exc)
{
err = true;
}
if (err || debugLevel < MSG_NONE || debugLevel > MSG_TRACE)
{
syntax("Invalid debuglevel value: " + args[i + 1], -4);
}
metd = true;
i++;
count++;
continue;
}
// Call graph mode argument.
if (args[i].equalsIgnoreCase("-c"))
{
if (metc)
{
syntax("Only one -c option is allowed", -5);
}
callGraphMode = true;
metc = true;
continue;
}
// Explicit file list mode argument.
if (args[i].equalsIgnoreCase("-f"))
{
if (metf)
{
syntax("Only one -f option is allowed", -6);
}
explicit = true;
metf = true;
continue;
}
// Honor hidden mode argument.
if (args[i].equalsIgnoreCase("-h"))
{
if (meth)
{
syntax("Only one -h option is allowed", -7);
}
honorHidden = true;
meth = true;
continue;
}
// Read-only mode.
if (args[i].equalsIgnoreCase("-r"))
{
if (metr)
{
syntax("Only one -r option is allowed", -8);
}
readOnly = true;
metr = true;
continue;
}
// configuration profile
if (args[i].equalsIgnoreCase("-profile"))
{
if (cfgProfile != null)
{
syntax("Only one -profile option is allowed", -12);
}
i++;
count++;
if (i >= args.length)
{
syntax("The -profile option requires a profile <name> argument", -13);
}
cfgProfile.add(args[i]);
continue;
}
syntax("Invalid option: " + args[i], -9);
}
// Process variable initializer overrides.
HashMap<String, Object> initializers = null;
while (count < len)
{
String arg = args[count];
int pos = arg.indexOf("=");
if (pos < 0)
{
break;
}
String key = arg.substring(0, pos);
String value = arg.substring(pos + 1);
if (initializers == null)
{
initializers = new HashMap<>();
}
initializers.put(key, value);
count++;
}
if (len <= count)
{
syntax(null, -10);
}
String profile = args[count++];
// Process AST filespec arguments.
String directory = null;
String fileSpec = null;
String[] filelist = null;
if (explicit)
{
filelist = new String[len - count];
for (int j = 0; count < len; j++, count++)
{
filelist[j] = args[count];
}
}
else
{
if (len == count + 2)
{
directory = args[count++];
fileSpec = args[count++];
}
if (len > count)
{
syntax(null, -11);
}
}
// set the configuration profile
if (cfgProfile != null)
{
Configuration.setDefaultProfiles(cfgProfile);
}
try
{
// Instantiate pattern engine and run...
PatternEngine engine = new PatternEngine(directory, fileSpec);
setDebugLevel(debugLevel);
if (explicit)
{
engine.addAstSpec(new ExplicitFileList(filelist));
}
if (callGraphMode)
{
// TODO: should this be switched to CallGraphHelper.getGraph() which doesn't create indexes?
engine.setGraph(CallGraphWorker.getGraph());
}
engine.setHonorHidden(honorHidden);
engine.setReadOnly(readOnly);
engine.setVariableInitializers(initializers);
engine.run(profile);
}
catch (Exception exc)
{
if (debugLevel >= MSG_DEBUG)
{
// prints the exception with causes
LOG.log(Level.SEVERE, "ERROR:", exc);
}
else
{
// for MSG_STATUS prints only the direct stacktrace
StringBuilder sb = new StringBuilder("ERROR:");
sb.append(System.lineSeparator());
dumpThrowableNoCause(sb, exc);
LOG.log(Level.SEVERE, sb.toString());
}
}
finally
{
CallGraphHelper.shutdownGraph();
}
}
/**
* Dump the details of the throwable into the given string buffer without including the cause.
*
* @param sb
* The string builder the throwable to be appended to.
* @param t
* The throwable to dump.
*/
private static void dumpThrowableNoCause(StringBuilder sb, Throwable t)
{
sb.append(t.getClass().getName());
String msg = t.getMessage();
if (msg != null && msg.trim().length() > 0)
{
sb.append(": ");
sb.append(msg);
}
sb.append(System.lineSeparator());
StackTraceElement[] stackTraceElements = t.getStackTrace();
for (int i = 0; i < stackTraceElements.length; i++)
{
StackTraceElement stackTraceElement = stackTraceElements[i];
sb.append(" at ");
sb.append(stackTraceElement);
if (i < stackTraceElements.length - 1)
{
sb.append(System.lineSeparator());
}
}
}
/**
* Container for an arbitrary category and id that is supposed to be used as a key for the
* {@link PatternEngine#storedObjects} map.
*/
public static class StorageKey
{
/** Arbitrary category. <code>null</code> if no category is specified. */
private final String category;
/** Arbitrary id. */
private final String id;
/**
* Creates a key with no category specified.
*
* @param id
* Arbitrary id.
*/
public StorageKey(String id)
{
this(null, id);
}
/**
* Convenience constructor.
*
* @param category
* Arbitrary category.
* @param id
* Arbitrary id.
*/
public StorageKey(String category, String id)
{
this.category = category;
this.id = id;
}
/**
* Calculate a hash code for this object.
*
* @return hash code.
*/
public int hashCode()
{
int result = 17;
result = 37 * result + (category == null ? 0 : category.hashCode());
result = 37 * result + (id == null ? 0 : id.hashCode());
return result;
}
/**
* Test this object for equality (equivalence) with the given object.
*
* @param o
* Object to test.
*
* @return <code>true</code> if this object is the same instance as the
* given object, or it is considered equivalent to the given
* object. <code>StorageKey</code> instances are equivalent if
* they contain the same category and id.
*/
public boolean equals(Object o)
{
if (!(o instanceof StorageKey))
{
return false;
}
if (o == this)
{
return true;
}
StorageKey other = (StorageKey) o;
return (category == null ? other.category == null :
category.equals(other.category)) &&
(id == null ? other.id == null :
id.equals(other.id));
}
/**
* Produce a string representation of the internal state of this object.
*
* @return String representation of this object.
*/
public String toString()
{
return "[" + category + ", " + id + "]";
}
}
}