public class ExpressionLexer extends antlr.CharScanner implements ExpressionParserTokenTypes, antlr.TokenStream
ExpressionParser
.
Each token has an integer token type by which the parser references and matches tokens. This is done to make a high performance parser. To make this work, the lexer must create a token object when it matches a top level rule, this token object includes the token type and the actual text found in the original character stream. The lexer's main job is to lookahead into the character stream, switch into the top level rule that matches some set of characters and when this is complete, to create a valid token object and return this to the parser. The lexer thus implements a token stream interface from the parser's perspective.
Please note that this is a generated file using ANTLR 2.7.4 and a
grammar specified in expression.g
.
Symbol resolution is the process by which the proper token type is assigned to a token created from a match to the generic symbol rule. In the lexer, the only symbol resolution that can be done is to match with reserved keywords. Reserved keywords are given preference over other symbol types in any case where there is a conflict.
Other key design points:
nextToken
. This
method has a for-loop that uses up to 3 characters of lookahead
to identify which of the top level token rules to call. To do
this, each token rule's left-most match characters are "rolled up"
and tested in this top level rule.
All token types are defined as integer constants in
which is an interface that
the parser, lexer and other related classes all implement. This allows
all of these classes to directly refer to the token types and share this
common set of definitions. All top-level lexer rules generate a token
of the same name in the ExpressionParserTokenTypes
ExpressionParserTokenTypes
interface.
There is a tokens { } section in the parser where artificial tokens are
defined (tokens that are not backed by a lexer rule). These tokens are
also added to the ExpressionParserTokenTypes
interface and
can thus be referenced directly by the lexer and parser.
Modifier and Type | Field and Description |
---|---|
static antlr.collections.impl.BitSet |
_tokenSet_0 |
static antlr.collections.impl.BitSet |
_tokenSet_1 |
static antlr.collections.impl.BitSet |
_tokenSet_2 |
private static java.util.Map |
keywords
Map of keywords to symbol tokens
|
private boolean |
noAssign
If
true , convert assignment to equality operator |
private int |
posParmCount
Positional parameter counter
|
_returnToken, caseSensitive, caseSensitiveLiterals, commitToPath, EOF_CHAR, hashString, inputState, literals, saveConsumedInput, tabsize, text, tokenObjectClass, traceDepth
AND, ASSIGN, BIT_AND, BIT_COMP, BIT_OR, BIT_XOR, BOOL_FALSE, BOOL_TRUE, CAST, COMMA, DEC_LITERAL, DIGIT, DIVIDE, DOT, DSTRING, EOF, EQUALS, GOTO, GT, GTE, HASH, HEX_LITERAL, HEXDIGIT, IDENTITY, IS_NULL, LETTER, LPARENS, LSHIFT, LT, LTE, METH_BOOLEAN, METH_DOUBLE, METH_INT, METH_LONG, METH_OBJ, MINUS, MODULO, MULTIPLY, NOT, NOT_EQ, NOT_NULL, NULL, NULL_TREE_LOOKAHEAD, NUM_LITERAL, OR, PLUS, POSPARM, RPARENS, RSHIFT, SSTRING, STRING, SYM_CHAR, SYMBOL, UN_MINUS, VALID_SYM_CHAR, WS, ZRSHIFT
Constructor and Description |
---|
ExpressionLexer(antlr.InputBuffer ib) |
ExpressionLexer(java.io.InputStream in) |
ExpressionLexer(antlr.LexerSharedInputState state) |
ExpressionLexer(java.io.Reader in) |
Modifier and Type | Method and Description |
---|---|
static void |
main(java.lang.String[] args)
Provides a command line interface for an end user to drive and/or test
the ExpressionLexer class.
|
void |
mAND(boolean _createToken)
Matches the "&&" string.
|
void |
mASSIGN(boolean _createToken)
Matches the "=" character.
|
void |
mBIT_AND(boolean _createToken)
Matches the '&' character.
|
void |
mBIT_COMP(boolean _createToken)
Matches the '~' character.
|
void |
mBIT_OR(boolean _createToken)
Matches the '|' character.
|
void |
mBIT_XOR(boolean _createToken)
Matches the '^' character.
|
void |
mCOMMA(boolean _createToken)
Matches the ',' character.
|
protected void |
mDIGIT(boolean _createToken)
Matches any numeric digit:
0 - 9 |
void |
mDIVIDE(boolean _createToken)
Matches the '/' character.
|
protected void |
mDSTRING(boolean _createToken)
Matches an opening double quote, arbitrary contents and an ending double
quote.
|
void |
mEQUALS(boolean _createToken)
Matches the "==" string.
|
void |
mGT(boolean _createToken)
Matches the '>' character.
|
void |
mGTE(boolean _createToken)
Matches the '>=' character sequence.
|
void |
mHASH(boolean _createToken)
Matches the '#' character.
|
protected void |
mHEXDIGIT(boolean _createToken)
Matches any valid letter that can be used in a hexidecimal number.
|
private static long[] |
mk_tokenSet_0() |
private static long[] |
mk_tokenSet_1() |
private static long[] |
mk_tokenSet_2() |
protected void |
mLETTER(boolean _createToken)
Matches any alphabetic character:
a - z |
void |
mLPARENS(boolean _createToken)
Matches the '(' character.
|
void |
mLSHIFT(boolean _createToken)
Matches the "<<" string.
|
void |
mLT(boolean _createToken)
Matches the '<' character.
|
void |
mLTE(boolean _createToken)
Matches the '<=' character sequence.
|
void |
mMINUS(boolean _createToken)
Matches the '-' character.
|
void |
mMODULO(boolean _createToken)
Matches the '%' character.
|
void |
mMULTIPLY(boolean _createToken)
Matches the '*' character.
|
void |
mNOT_EQ(boolean _createToken)
Matches the inequality string.
|
void |
mNOT(boolean _createToken)
Matches the logical negation character.
|
void |
mNUM_LITERAL(boolean _createToken)
Matches all forms of valid integer literals and decimal literals.
|
void |
mOR(boolean _createToken)
Matches the "||" string.
|
void |
mPLUS(boolean _createToken)
Matches the '+' character.
|
protected void |
mPOSPARM(boolean _createToken)
Matches a positional parameter placeholder.
|
void |
mRPARENS(boolean _createToken)
Matches the ')' character.
|
void |
mRSHIFT(boolean _createToken)
Matches the ">>" string.
|
protected void |
mSSTRING(boolean _createToken)
Matches an opening single quote, arbitrary contents and an ending single
quote.
|
void |
mSTRING(boolean _createToken)
Matches any string literal which is enclosed in a pair of single quotes
or a pair of double quotes.
|
protected void |
mSYM_CHAR(boolean _createToken)
Matches all characters that can be made part of a valid user-defined
symbol (except alphabetic and numeric characters).
|
void |
mSYMBOL(boolean _createToken)
Match a valid symbol name.
|
protected void |
mVALID_SYM_CHAR(boolean _createToken)
Matches any single character that can appear in the 2nd or later position
in a symbol (matches any
LETTER, DIGIT or SYM_CHAR ). |
void |
mWS(boolean _createToken)
Matches any amount of whitespace in a program and sets the token type to
"skip".
|
void |
mZRSHIFT(boolean _createToken)
Matches the ">>>" string.
|
antlr.Token |
nextToken() |
void |
setNoAssignment() |
append, append, commit, consume, consumeUntil, consumeUntil, getCaseSensitive, getCaseSensitiveLiterals, getColumn, getCommitToPath, getFilename, getInputBuffer, getInputState, getLine, getTabSize, getText, getTokenObject, LA, makeToken, mark, match, match, match, matchNot, matchRange, newline, panic, panic, reportError, reportError, reportWarning, resetText, rewind, setCaseSensitive, setColumn, setCommitToPath, setFilename, setInputState, setLine, setTabSize, setText, setTokenObjectClass, tab, testLiteralsTable, testLiteralsTable, toLower, traceIn, traceIndent, traceOut, uponEOF
private static final java.util.Map keywords
private int posParmCount
private boolean noAssign
true
, convert assignment to equality operatorpublic static final antlr.collections.impl.BitSet _tokenSet_0
public static final antlr.collections.impl.BitSet _tokenSet_1
public static final antlr.collections.impl.BitSet _tokenSet_2
public ExpressionLexer(java.io.InputStream in)
public ExpressionLexer(java.io.Reader in)
public ExpressionLexer(antlr.InputBuffer ib)
public ExpressionLexer(antlr.LexerSharedInputState state)
public void setNoAssignment()
public static void main(java.lang.String[] args)
Syntax:
java ExpressionLexer <expression>
args
- List of command line arguments.public antlr.Token nextToken() throws antlr.TokenStreamException
nextToken
in interface antlr.TokenStream
antlr.TokenStreamException
public final void mWS(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
Spaces, tabs and carriage returns and line feeds (newlines) are all matched.
This is a top level lexer rule which means that there is an associated
WS
token.
antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mSYMBOL(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
_ # $ : .
).
Symbols are matched case-insensitively.
Once a symbol has been found, a keyword lookup occurs. If matched, the
the token's type is overridden from the default (SYMBOL
)
to the artificial token type associated with the keyword.
This is a top level lexer rule which means that there is an associated
SYMBOL
token.
antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
protected final void mPOSPARM(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
protected final void mLETTER(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
a - z
antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
protected final void mVALID_SYM_CHAR(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
LETTER, DIGIT or SYM_CHAR
). This is
simply a helper rule to make references easier.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mASSIGN(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
ASSIGN
token type, unless setNoAssignment()
has been
invoked, in which case an EQUALS
token type is created.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mCOMMA(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
COMMA
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mEQUALS(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
EQUALS
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mOR(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
OR
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mAND(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
AND
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mNOT(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
NOT
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mRSHIFT(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
RSHIFT
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mLSHIFT(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
LSHIFT
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mZRSHIFT(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
ZRSHIFT
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mBIT_AND(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
BIT_AND
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mBIT_OR(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
BIT_OR
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mBIT_XOR(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
BIT_XOR
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mBIT_COMP(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
BIT_COMP
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mNOT_EQ(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
NOT_EQ
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mGT(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
GT
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mLT(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
LT
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mGTE(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
GTE
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mLTE(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
LTE
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mLPARENS(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
LPARENS
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mRPARENS(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
RPARENS
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mPLUS(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
PLUS
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mMINUS(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
MINUS
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mMULTIPLY(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
MULTIPLY
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mDIVIDE(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
DIVIDE
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mMODULO(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
MODULO
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mHASH(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
HASH
token type.antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mNUM_LITERAL(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
NUM_LITERAL
token type.
HEX_LITERAL
token type.
DEC_LITERAL
token type.
DEC_LITERAL
token type.
DEC_LITERAL
token type.
This is a top level rule. The token type defaults to
NUM_LITERAL
and is overridden by specific actions depending
on the scenario matched.
antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
protected final void mDIGIT(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
0 - 9
antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
protected final void mHEXDIGIT(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
public final void mSTRING(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
protected final void mSSTRING(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
Any newlines inside the string are identified and the lexer's internal newline counter is properly maintained. The newlines are not maintained in the output string.
The greedy option does not need to be disabled here as the closure rule termination is built into the subrule itself: it accepts anything that isn't an unescaped single quote character (see above).
Tabs and spaces are maintained inside strings. Opening and closing single quote characters are dropped.
antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
protected final void mDSTRING(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
Any newlines inside the string are identified and the lexer's internal newline counter is properly maintained. The newlines are not maintained in the output string.
The greedy option does not need to be disabled here as the closure rule termination is built into the subrule itself: it accepts anything that isn't an unescaped double quote character (see above).
Tabs and spaces are maintained inside strings. Opening and closing double quote characters are dropped.
antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
protected final void mSYM_CHAR(boolean _createToken) throws antlr.RecognitionException, antlr.CharStreamException, antlr.TokenStreamException
$ _
Note that this intentionally matches the normal Java symbol rules.
antlr.RecognitionException
antlr.CharStreamException
antlr.TokenStreamException
private static final long[] mk_tokenSet_0()
private static final long[] mk_tokenSet_1()
private static final long[] mk_tokenSet_2()