public class AstGenerator extends java.lang.Object implements HintsConstants, E4GLConstants
Optionally preprocesses, then parses a single Progress source file into an
abstract syntax tree (AST). Optionally accepts a PROPATH
setting which is used to allow the preprocessor to locate relative source
file references (includes). If this is not provided, the current directory
is assumed to be the top level directory for all filename resolution.
Caching of the preprocessor results can be optionally enabled. In this case the output will be stored in a file of the same base name as the original with an appended '.cache'. For example, the preprocessor output for utilities/top-menu.p would be stored in utilities/top-menu.p.cache.
If the given input file is HTML, it is converted into 4GL source code by
the embedded 4GL (E4GL) preprocessor. The resulting source code is then
run through the preprocessor, lexer and parser just as if it had been
static 4GL source. Note that the same flags that control the running and
caching of the 4GL preprocessor, control the running and caching of the
embedded 4GL (E4GL) preprocessor. Note that the only difference is that
the output of the E4GL preprocessor is always written to file. See
processFile(java.lang.String, java.util.List<java.io.File>, int)
for more details on this support.
For debugging and regression testing purposes, it is possible to save
the output of the lexer and/or the parser in a more human readable format.
The setDumpLexer(boolean)
and setDumpParser(boolean)
methods provide
this capability. Note that the resulting files will be generated with the
original source filename and an appended '.lexer' and '.parser'
respectively. The existence of these files does not alter any subsequent
processing and subsequent runs will replace these files with no warning.
An additional option is provided to enable AST persistence. If this is enabled, it changes the processing in 2 ways. First, if a valid persisted AST already exists, the AST returned by this class will be loaded from that file AND the preprocessor/parser will not be invoked. Second, in the case that no persisted AST exists yet OR the persisted AST is not later in timestamp than the source file, the output from the parser will be persisted into a file with the original source filename and an appended '.ast'.
By default, preprocessing, preprocessor output caching and AST persistance are all enabled BUT lexer and parser output is disabled.
Modifier and Type | Field and Description |
---|---|
static java.lang.String |
AST_POSTFIX
Text to append to a filename to generate a unique name for persisting
an AST.
|
private boolean |
astPersist
Flag indicating whether storing of a persistent AST is needed.
|
private boolean |
cache
Indicates whether caching of preprocessor results should be done.
|
static java.lang.String |
CACHE_POSTFIX
Text to append to a filename to generate a unique name for caching.
|
private static java.lang.String |
CASE_SENS_KEY
Key for the global, file system case sensitivity key.
|
static java.lang.String |
DICT_POSTFIX
Text to append to a filename to generate a unique name for persisting
schema dictionary information.
|
private boolean |
dumpLexer
Flag indicating whether dumping of lexer output is needed.
|
private boolean |
dumpParser
Flag indicating whether dumping of parser output is needed.
|
private static java.lang.String |
DYN_PROPATH_APPEND_KEY
Key to enable global directory appending to the propath.
|
private static java.lang.String |
DYN_PROPATH_PREPEND_KEY
Key to enable global directory prepending to the propath.
|
private static java.io.Writer |
err
Error destination.
|
private static java.lang.String |
ERR_DEST_KEY
Key for the error logfile.
|
private static java.lang.String |
FILESEP_KEY
Key for the file separator configuration parameter.
|
private UastHints |
hints
Represents any non-default, file specific behavior that must be
handled for the preprocessor or parser.
|
static java.lang.String |
HINTS_POSTFIX
Text to append to a filename to generate a unique name for hints.
|
static java.lang.String |
HTML1_POSTFIX
A filename extention to recognize as an HTML file.
|
static java.lang.String |
HTML2_POSTFIX
A filename extention to recognize as an HTML file.
|
private static java.lang.String |
IGNORE_KEY
Key for the keyword ignore configuration parameter.
|
static java.lang.String |
LEXER_POSTFIX
Text to append to a filename to generate a unique name for saving
human-readable lexer output.
|
private static AstManager |
mgr
Global manager for file locking and file ID services.
|
static java.lang.String |
PARSER_POSTFIX
Text to append to a filename to generate a unique name for saving
human-readable parser output.
|
private java.lang.String[] |
paths
Array of propath path elements.
|
private static java.lang.String |
PATHSEP_KEY
Key for the path separator configuration parameter.
|
private boolean |
preprocess
Flag indicating whether inline preprocessing is needed.
|
private static java.lang.String |
PROPATH_KEY
Key for the global propath configuration parameter.
|
private boolean |
silent
Flag indicating whether silent mode is required.
|
private static java.io.Writer |
warn
Warning destination.
|
private static java.lang.String |
WARN_DEST_KEY
Key for the warning logfile.
|
ATTR_CLASSNAME, ATTR_DB, ATTR_DTYPE, ATTR_ESCAPE, ATTR_NAME, ATTR_POS, ATTR_TYPE, ATTR_VAL, ELEM_ALIAS, ELEM_ARCH, ELEM_ARG, ELEM_ARRAY, ELEM_BASEPATH, ELEM_BATCH, ELEM_CALLGRAPH, ELEM_DATABASE, ELEM_DEBUG, ELEM_E4GL, ELEM_ESC, ELEM_INCLUDE_SPEC, ELEM_MARKER, ELEM_MODE, ELEM_OPSYS, ELEM_OUTPUT, ELEM_PREPROC, ELEM_PROPATH, ELEM_ROOTLIST, ELEM_SCHEMA, ELEM_TABLE, ELEM_UAST, ELEM_VAR, ELEM_WINSYS, ELEM_WSOPTIONS, SOURCE_CHARSET
BLUE_DIAMOND, DEFAULT_MODE, MODE_NAMES, OPT_FOUND, OPT_INCLUDE, OPT_KEEP, OPT_WEB_OBJ, UNKNOWN_MODE, WEBSPEED, WEBSPEED_STRICT
Constructor and Description |
---|
AstGenerator()
Default constructor which initializes the propath setting based on
the project configuration.
|
Modifier and Type | Method and Description |
---|---|
private static void |
appendPath(java.lang.StringBuilder sb,
java.lang.String txt)
Append the given text to the path list in the buffer, optionally
separating the path elements with the file system path separator if
a path separator is not already present.
|
static Options |
buildOptions(java.lang.String[] paths,
java.io.File file,
UastHints hints)
Common setup for preprocessor options which uses hints and global
configuration as available.
|
private java.lang.String |
calcOutputFilename(E4GLPreprocessor e4gl)
Determine the output file name to be used for the current HTML file
being processed.
|
private boolean |
checkForValidAst(java.lang.String filename)
Checks whether or not the source filename has a valid persisted AST.
|
private E4GLPreprocessor |
createE4GLPreprocessor(java.lang.String filename)
Manage the process of initializing the E4GL preprocessor including
the honoring of hints for
wsoptions and the compatibility
mode . |
private static java.io.File |
createUniqueFile(java.lang.String filename,
java.lang.String postfix)
Builds a unique filename based on an original filename and a passed
postfix.
|
private java.io.PrintStream |
createUniqueStream(java.lang.String original,
java.lang.String postfix)
Calculates a unique stream name for storing the output, based on the
input file name and a postfix.
|
private java.lang.String |
findHintsFile(java.lang.String filename)
Find the UAST hints file associated with the given embedded 4GL (E4GL)
program (which is an HTML file).
|
private void |
generateParseReport(java.lang.String filename,
Aast ast)
Dump an AST into a file with a human readable report format.
|
boolean |
getAstPersist()
Accesses the state of the flag indicating if new ASTs should be
persisted.
|
boolean |
getCache()
Gets the state of the flag indicating if the preprocessor output should
be cached.
|
boolean |
getDumpLexer()
Accesses the state of the flag indicating if lexer output should be
dumped.
|
boolean |
getDumpParser()
Accesses the state of the flag indicating if parser output should be
dumped.
|
java.lang.String[] |
getPaths()
Accesses the PROPATH as defined by the project configuration or by
the user of this class.
|
boolean |
getPreprocess()
Gets the state of the flag indicating if the preprocessor should be
run.
|
static java.lang.String[] |
getPropath(java.lang.String paths,
UastHints hints)
Gets the PROPATH based on the global configuration and any file-specific configuration
provided.
|
boolean |
getSilent()
Gets the state of the silent mode flag indicating if the
System.out output
should be suppressed. |
static boolean |
isHTML(java.lang.String filename)
Uses the filename extension to determine if this is an HTML file or
not.
|
private Aast |
loadPersistedAst(java.lang.String filename)
Loads a previously persisted AST for a specified source file.
|
private Aast |
parse(java.lang.String filename)
Parse the source code read from the specified filename into an abstract
syntax tree.
|
private void |
persistAst(java.lang.String filename,
Aast ast)
Persist an AST into an XML file which can be turned into a valid AST
using
AstPersister . |
private java.io.Reader |
prepareDataStream(java.lang.String filename)
Prepare the data stream which will be used by the parser.
|
private ProgressLexer |
prepareLexer(java.lang.String filename,
SymbolResolver sym)
Prepare a lexer instance for the given file using the supporting
resolver instance as needed.
|
private static void |
prependPath(java.lang.StringBuilder sb,
java.lang.String txt)
Prepend the given text to the path list in the buffer, optionally
separating the path elements with the file system path separator if
a path separator is not already present.
|
private java.io.Reader |
preprocess(java.io.File file)
Perform inline preprocessing of the specified source file.
|
private void |
preprocessE4GL(E4GLPreprocessor e4gl,
java.lang.String source,
java.lang.String target)
Perform E4GL preprocessing of the specified source HTML file into the
given 4GL output file.
|
(package private) void |
preScanClass(java.lang.String filename,
boolean builtin,
boolean dotnet)
Performs first-level scanning of OO classes.
|
static java.io.File[] |
prioritize(java.io.File[] list)
Sort the given list to ensure that any HTML files are listed first.
|
Aast |
processFile(java.lang.String filename,
java.util.List<java.io.File> list,
int idx)
Create or load an AST given a specific source filename.
|
private void |
processParserHints(SymbolResolver sym)
Drive the process by which the hints file is queried for parser
related hints, any hints found are implemented.
|
private java.lang.String[] |
readIntoArray(java.lang.String filename)
Read a simple text file into an array where each line in the input file
is a separate element in the output array.
|
private void |
removeDuplicate(java.util.List<java.io.File> list,
java.lang.String duplicate)
Search the list to find a duplicate to the given name and remove it
if found.
|
void |
setAstPersist(boolean astPersist)
Sets the state of the flag indicating if a newly parsed AST should be
persisted.
|
void |
setCache(boolean cache)
Sets the state of the flag indicating if the preprocessor output should
be cached.
|
void |
setDumpLexer(boolean dumpLexer)
Sets the state of the flag indicating if lexer output should be
dumped.
|
void |
setDumpParser(boolean dumpParser)
Sets the state of the flag indicating if parser output should be
dumped.
|
void |
setPaths(java.lang.String paths)
Sets the PROPATH which defines how Progress source files are found.
|
void |
setPreprocess(boolean preprocess)
Sets the state of the flag indicating if the preprocessor should be
run.
|
void |
setSilent(boolean silent)
Sets the state of the silent mode flag indicating if the
System.out output
should be suppressed. |
public static final java.lang.String AST_POSTFIX
public static final java.lang.String CACHE_POSTFIX
public static final java.lang.String HINTS_POSTFIX
public static final java.lang.String LEXER_POSTFIX
public static final java.lang.String PARSER_POSTFIX
public static final java.lang.String DICT_POSTFIX
public static final java.lang.String HTML1_POSTFIX
public static final java.lang.String HTML2_POSTFIX
private static final java.lang.String FILESEP_KEY
private static final java.lang.String PATHSEP_KEY
private static final java.lang.String PROPATH_KEY
private static final java.lang.String DYN_PROPATH_PREPEND_KEY
private static final java.lang.String DYN_PROPATH_APPEND_KEY
private static final java.lang.String IGNORE_KEY
private static final java.lang.String CASE_SENS_KEY
private static final java.lang.String ERR_DEST_KEY
private static final java.lang.String WARN_DEST_KEY
private static AstManager mgr
private static java.io.Writer err
private static java.io.Writer warn
private boolean silent
private boolean preprocess
private boolean cache
private boolean dumpLexer
private boolean dumpParser
private boolean astPersist
private java.lang.String[] paths
private UastHints hints
public AstGenerator()
public static java.io.File[] prioritize(java.io.File[] list)
Make sure all HTML files appear first since they get converted into 4GL
source and these generated source file names may be separately
duplicated in this list. By ensuring that the HTML files are listed
first, the processing of the HTML files can also remove any duplicate
4GL source file names from the list, ensuring that no unnecessary
processing is done. See processFile(java.lang.String, java.util.List<java.io.File>, int)
.
list
- The list to sort.public static boolean isHTML(java.lang.String filename)
HTML1_POSTFIX
and HTML2_POSTFIX
.filename
- The filename to test.true
if the filename ends with one of the
known HTML extensions.public static Options buildOptions(java.lang.String[] paths, java.io.File file, UastHints hints)
paths
- The propath;file
- The specific source file being processed. May be
null
if this preprocessor is not associated with
a specific 4GL source file.hints
- Any 4GL source file hints that may exist. May be
null
if no hints exists or if there is no
4GL source file associated.public static java.lang.String[] getPropath(java.lang.String paths, UastHints hints)
paths
- A single string with one or more paths to be searched for Progress files, with
each path separated by the current platform's file separator character.
May be null
.hints
- File-specific hints or null
if they don't exist or if there is no specific
source file being processed.public boolean getSilent()
System.out
output
should be suppressed.true
if silent mode is enabled, otherwise false
.public void setSilent(boolean silent)
System.out
output
should be suppressed.silent
- true
if silent mode is enabled, otherwise false
.public void setPreprocess(boolean preprocess)
preprocess
- true
if the preprocessor is to be run, otherwise
false
.public boolean getPreprocess()
true
if the preprocessor is to be run, otherwise
false
.public void setCache(boolean cache)
cache
- true
if the preprocessor output is to be cached,
otherwise false
.public boolean getCache()
true
if the preprocessor output is to be cached,
otherwise false
.public void setDumpLexer(boolean dumpLexer)
dumpLexer
- true
if lexer output is to be dumped, otherwise
false
.public boolean getDumpLexer()
true
if lexer output is to be dumped, otherwise
false
.public void setDumpParser(boolean dumpParser)
dumpParser
- true
if parser output is to be dumped, otherwise
false
.public boolean getDumpParser()
true
if parser output is to be dumped,
otherwise false
.public void setAstPersist(boolean astPersist)
astPersist
- true
to persist a new AST, otherwise
false
.public boolean getAstPersist()
true
to persist a new AST, otherwise
false
.public void setPaths(java.lang.String paths)
paths
- A single string with one or more paths to be searched for Progress files, with
each path separated by the current platform's file separator character.public java.lang.String[] getPaths()
public Aast processFile(java.lang.String filename, java.util.List<java.io.File> list, int idx) throws AstException
The caller may provide a file list such that duplicate source names
can be removed. This is not a generic service, but rather one that is
specific to HTML files. In the case of an HTML file that has embedded
4GL (E4GL) code, the E4GL preprocessor will be invoked. This HTML file
will then be converted into a normal 4GL source file. Since that source
file is generated by the preprocessor, it may overwrite a file that is
already there from a previous run (if caching is off or if the cached
file is older than the timestamp on the HTML file). Since a valid 4GL
source file may already exist in the file system, it is possible that
that source file name (with a .p, .w or .i instead of the .html) may
also appear in the file list that is used by the caller. Since the
caller doesn't know the generated file name in the E4GL case, the
pruning of that "duplicate" name is delegated to this method. The file
list will be edited to remove any output file name that the E4GL
preprocessor generates, if it is separately listed. This avoids
duplicate processing of that file, since it was already processed
fully to an AST when it was listed as an HTML file, it doesn't need to
be processed again later. This processing is dependent upon all the
HTML files being processed before any 4GL source files. The
prioritize(java.io.File[])
method can be used to sort the list properly. Note
that any removal is done by setting the duplicate entry to
null
. The calling code must be prepared to ignore any
such null
entries.
If this file represents a .cls
file (a class or interface)
it is assumed that the file is not a built-in 4GL class, nor is it a
.NET class.
filename
- Name of the source file to process; if this is not an absolute
path, it should be relative to the propath.list
- The list of files upon which the generator is iterating in the
calling method. The list may be edited as noted above. This
may be null
if no such editing is needed.idx
- Index position of the element being processed in the given
list. Ignored if the list is null
.AstException
- if the file cannot be found, or if there is some other error
during processing.private static java.io.File createUniqueFile(java.lang.String filename, java.lang.String postfix)
filename
- The original source filename.postfix
- Text to append to the original filename.private java.io.Reader prepareDataStream(java.lang.String filename) throws java.io.FileNotFoundException, java.io.IOException
filename
- Name of the source file to process; if this is not an absolute
path, it should be relative to the propath.java.io.FileNotFoundException
- if filename
does not exist.java.io.IOException
- If a problem occurs during preprocessing or caching.private java.lang.String findHintsFile(java.lang.String filename)
basename.*.
concatinated with the extension
defined by UastHints.HINTS_SUFFIX
. If more than 1 file is
found, then none will be used since this code has no way of determining
what the precedence order would be. Note that there is no known
functional reason why more than one hints file would match this
specification.filename
- The filename on which to base the hints filename.null
if no hints file can be found.private E4GLPreprocessor createE4GLPreprocessor(java.lang.String filename)
wsoptions
and the compatibility
mode
.filename
- The HTML file to preprocess.private java.lang.String calcOutputFilename(E4GLPreprocessor e4gl)
e4gl
- The preprocessor associated with the current file.private java.io.Reader preprocess(java.io.File file) throws java.io.FileNotFoundException, java.io.IOException
file
- File to be preprocessed.Preprocessor
.java.io.FileNotFoundException
- if filename
does not exist.java.io.IOException
- during a write or close failure.private void preprocessE4GL(E4GLPreprocessor e4gl, java.lang.String source, java.lang.String target)
e4gl
- The file-specific preprocessor instance.source
- HTML source file.target
- 4GL output file.private void removeDuplicate(java.util.List<java.io.File> list, java.lang.String duplicate)
null
and the method will return.list
- Array to edit.duplicate
- Name of the file to remove. This may be null
but then why would you call this method?void preScanClass(java.lang.String filename, boolean builtin, boolean dotnet) throws AstException, antlr.ANTLRException, java.io.IOException
filename
- Name of the source file to process; if this is not an absolute
path, it should be relative to the propath.builtin
- true
if this class file represents a built-in
class or interface.dotnet
- true
if this class file represents a .NET
class or interface.AstException
- if the file cannot be found, or if there is some other error during processing.antlr.ANTLRException
java.io.IOException
private Aast parse(java.lang.String filename) throws AstException, antlr.ANTLRException, java.io.IOException
filename
- The name of the file being processed.antlr.ANTLRException
- if any error occurs while lexing or parsing the source code.AstException
java.io.IOException
private ProgressLexer prepareLexer(java.lang.String filename, SymbolResolver sym) throws java.io.IOException
filename
- The file being lexed.sym
- The symbol resolver instance being used for this file.java.io.IOException
private void processParserHints(SymbolResolver sym)
sym
- Provides access to the symbol namespaces such that hints can
be loaded.private java.lang.String[] readIntoArray(java.lang.String filename)
filename
- Name of the text file to read.null
if there
was any error.private java.io.PrintStream createUniqueStream(java.lang.String original, java.lang.String postfix)
original
- Input file name.private void generateParseReport(java.lang.String filename, Aast ast)
PARSER_POSTFIX
.filename
- Original source file being processed.ast
- The AST to dump.private void persistAst(java.lang.String filename, Aast ast) throws AstException
AstPersister
. A filename is generated off the
original source file's name, with a standard extension of
AST_POSTFIX
.filename
- Original source file being processed.ast
- The AST to dump.AstException
private boolean checkForValidAst(java.lang.String filename)
filename
- The source filename to check for a matching persisted AST.true
if a persisted AST exists AND that AST
has a timestamp later than that of the source file,
false
otherwise.private Aast loadPersistedAst(java.lang.String filename) throws AstException
filename
- The source filename for which a matching persisted AST will
be loaded.AstException
- if there are any problems with loading the AST.private static void prependPath(java.lang.StringBuilder sb, java.lang.String txt)
sb
- Buffer containing current path list.txt
- Path element to prepend (insert at position 0).private static void appendPath(java.lang.StringBuilder sb, java.lang.String txt)
sb
- Buffer containing current path list.txt
- Path element to append.