public class Token extends Object implements Position
Tokenizer
.
A token consists of a position, a type and up to three string values. The first is the trigger. Along with the type this uniquely identifies what kind of token we're looking at. For example the input "Hello", the id Hello and a special id like #Hello all have the same content (Hello). However, they will have different types (STRING, ID, SPECIAL_ID). However, the two special ids $Hello and #Hello will both have the same token type and content. Those will differ in their trigger which will be # and $ respectively.
As already shown, the content contains the effective content of the token which should be used for further processing. Finally the source contains the complete text which was consumed while reading the token. If we look at a string constant "Hello" the content will be Hello and the source will be "Hello".
The basic token types supported are:
Modifier and Type | Class and Description |
---|---|
static class |
Token.TokenType
Contains the different token types supported by this class.
|
Modifier and Type | Field and Description |
---|---|
protected int |
pos |
Modifier and Type | Method and Description |
---|---|
Token |
addToContent(char ch)
Adds the given character to the content (and the source) but not to the trigger
|
Token |
addToContent(Char ch)
Adds the given Char to the content (and the source) but not to the trigger
|
Token |
addToSource(Char ch)
Adds the given Char to the source of this token, but neither to the trigger nor to the content.
|
Token |
addToTrigger(Char ch)
Adds the given Char to the trigger (and the source) but not to the content
|
static Token |
create(Token.TokenType type,
Position pos)
Creates a new token with the given type, using the given position as location info.
|
static Token |
createAndFill(Token.TokenType type,
Char ch)
Creates a new token with the given type, using the Char a initial trigger and content.
|
String |
getContents()
Returns the effective content of this token
|
int |
getLine()
Returns the line number of this position.
|
int |
getPos()
Returns the character position within the line of this position
|
String |
getSource()
Returns the complete source string consumed while parsing this token
|
String |
getTrigger()
Returns the string or character which further specifies this token.
|
Token.TokenType |
getType()
Returns the basic classification of this token
|
boolean |
hasContent(String content)
Determines if the given content matches the content of this token.
|
boolean |
is(Token.TokenType type)
Determines if the token has the given type
|
boolean |
isDecimal()
Determines if this token is a decimal number.
|
boolean |
isEnd()
Determines if this is an end of input token
|
boolean |
isIdentifier(String... values)
Determines if this token is an identifier.
|
boolean |
isInteger()
Determines if this token is an integer number.
|
boolean |
isKeyword(String... keywords)
Determines if this token is a keyword.
|
boolean |
isNotEnd()
Opposite of
isEnd() . |
boolean |
isNumber()
Determines if this token is an integer or decimal number.
|
boolean |
isScientificDecimal()
Determines if this token is a scientific decimal number (e.g.
|
boolean |
isSpecialIdentifier(String... triggers)
Determines if this token is a special identifier.
|
boolean |
isSpecialIdentifierWithContent(String trigger,
String... contents)
Determines if this token is a special identifier with the given trigger.
|
boolean |
isString()
Determines if this token is a string constant
|
boolean |
isSymbol(String... symbols)
Determines if this token is a symbol.
|
boolean |
matches(Token.TokenType type,
String trigger)
Determines if this token has the given type and trigger.
|
void |
setContent(String content)
Externally sets the content used for this token.
|
void |
setSource(String source)
Externally sets the source used for this token.
|
void |
setTrigger(String trigger)
Externally sets the trigger used for this token.
|
Token |
silentAddToContent(char ch)
Adds a character to the content without adding it to the source.
|
String |
toString() |
boolean |
wasTriggeredBy(String... triggers)
Determines if this token was triggered by one of the given triggers.
|
public static Token create(Token.TokenType type, Position pos)
type
- the type if this token. Can be further specified by supplying a trigger.pos
- the location of this tokenpublic static Token createAndFill(Token.TokenType type, Char ch)
type
- the type if this token. The supplied Char will be used as initial part of the trigger to further
specify the tokench
- first character of the content and trigger of this token. Also specifies the position of the token.public Token addToTrigger(Char ch)
ch
- the character to add to the trigger and sourcepublic Token addToSource(Char ch)
ch
- the character to add to the sourcepublic Token addToContent(Char ch)
ch
- the character to add to the content and sourcepublic Token addToContent(char ch)
ch
- the character to add to the content and sourcepublic Token silentAddToContent(char ch)
ch
- the character to add to the contentpublic String getTrigger()
public Token.TokenType getType()
public String getContents()
public String getSource()
public int getLine()
Position
public int getPos()
Position
public void setTrigger(String trigger)
This will neither change the content nor the source of this token.
trigger
- the new trigger of this tokenpublic void setContent(String content)
This will neither change the trigger nor the source of this token.
content
- the new content of this tokenpublic void setSource(String source)
This will neither change the trigger nor the content of this token.
source
- the new source of this tokenpublic boolean isEnd()
public boolean isNotEnd()
isEnd()
.public boolean matches(Token.TokenType type, String trigger)
type
- the expected typetrigger
- the expected triggerpublic boolean wasTriggeredBy(String... triggers)
triggers
- a list of possible triggers to compare topublic boolean hasContent(String content)
content
- the content to check forpublic boolean is(Token.TokenType type)
type
- the expected typepublic boolean isSymbol(String... symbols)
If a list of symbols is given, this method checks that the trigger matches one of them.
symbols
- the symbols to check for. If the list es empty, only the token type is checked.public boolean isKeyword(String... keywords)
If a list of symbols is given, this method checks that the trigger matches one of them.
keywords
- the keywords to check for. If the list es empty, only the token type is checked.public boolean isIdentifier(String... values)
If a list of values is given, this method checks that the content matches one of them.
values
- the values to check for. If the list es empty, only the token type is checked.public boolean isSpecialIdentifier(String... triggers)
If a list of triggers is given, this method checks that the trigger matches one of them.
triggers
- the triggers to check for. If the list es empty, only the token type is checked.public boolean isSpecialIdentifierWithContent(String trigger, String... contents)
If a list of contents is given, this method checks that the content matches one of them.
trigger
- the trigger of the special idcontents
- the content to check for. If the list es empty, only the token type and the trigger is checked.public boolean isInteger()
public boolean isDecimal()
public boolean isScientificDecimal()
public boolean isNumber()
public boolean isString()
Copyright © 2020. All rights reserved.