Scanner (Eclipse ECJ 4.3.1 API)

java.lang.Object
- org.eclipse.jdt.internal.compiler.parser.Scanner

All Implemented Interfaces:

TerminalTokens

Direct Known Subclasses:

RecoveryScanner
```
public class Scanner
extends Object
implements TerminalTokens
```
IMPORTANT NOTE: Internal Scanner implementation. It is mirrored in org.eclipse.jdt.core.compiler public package where it is API. The mirror implementation is using the backward compatible ITerminalSymbols constant definitions (stable with 2.0), whereas the internal implementation uses TerminalTokens which constant values reflect the latest parser generation state.

Field Summary

Fields
Modifier and Type	Field and Description
`static String`	`BINARY_LITERAL_NOT_BELOW_17`
`static int`	`BracketKinds`
`char[][][][]`	`charArray_length`
`boolean`	`checkNonExternalizedStringLiterals`
`static int`	`COMMENT_ARRAYS_SIZE`
`int`	`commentPtr`
`int[]`	`commentStarts`
`int[]`	`commentStops`
`int[]`	`commentTagStarts`
`long`	`complianceLevel`
`boolean`	`containsAssertKeyword`
`static int`	`CurlyBracket`
`char`	`currentCharacter`
`int`	`currentPosition`
`boolean`	`diet`
`static String`	`END_OF_SOURCE`
`int`	`eofPosition`
`int`	`foundTaskCount`
`char[][]`	`foundTaskMessages`
`int[][]`	`foundTaskPositions`
`char[][]`	`foundTaskPriorities`
`char[][]`	`foundTaskTags`
`static int`	`HIGH_SURROGATE_MAX_VALUE`
`static int`	`HIGH_SURROGATE_MIN_VALUE`
`static String`	`ILLEGAL_HEXA_LITERAL`
`int`	`initialPosition`
`boolean`	`insideRecovery`
`static String`	`INVALID_BINARY`
`static String`	`INVALID_CHAR_IN_STRING`
`static String`	`INVALID_CHARACTER_CONSTANT`
`static String`	`INVALID_DIGIT`
`static String`	`INVALID_ESCAPE`
`static String`	`INVALID_FLOAT`
`static String`	`INVALID_HEXA`
`static String`	`INVALID_HIGH_SURROGATE`
`static String`	`INVALID_INPUT`
`static String`	`INVALID_LOW_SURROGATE`
`static String`	`INVALID_OCTAL`
`static String`	`INVALID_UNDERSCORE`
`static String`	`INVALID_UNICODE_ESCAPE`
`boolean`	`isTaskCaseSensitive`
`protected int`	`lastCommentLinePosition`
`protected int`	`lastPosition`
`int[]`	`lineEnds`
`int`	`linePtr`
`static int`	`LOW_SURROGATE_MAX_VALUE`
`static int`	`LOW_SURROGATE_MIN_VALUE`
`protected int`	`nlsTagsPtr`
`static String`	`NULL_SOURCE_STRING`
`static int`	`OptimizedLength`
`boolean`	`recordLineSeparator`
`boolean`	`returnOnlyGreater`
`static int`	`RoundBracket`
`boolean`	`scanningFloatLiteral`
`boolean`	`skipComments`
`char[]`	`source`
`long`	`sourceLevel`
`static int`	`SquareBracket`
`int`	`startPosition`
`static char`	`TAG_POSTFIX`
`static int`	`TAG_POSTFIX_LENGTH`
`static char[]`	`TAG_PREFIX`
`static int`	`TAG_PREFIX_LENGTH`
`char[][]`	`taskPriorities`
`char[][]`	`taskTags`
`boolean`	`tokenizeComments`
`boolean`	`tokenizeWhiteSpace`
`static String`	`UNDERSCORES_IN_LITERALS_NOT_BELOW_17`
`boolean`	`unicodeAsBackSlash`
`static String`	`UNTERMINATED_COMMENT`
`static String`	`UNTERMINATED_STRING`
`boolean`	`useAssertAsAnIndentifier`
`boolean`	`useEnumAsAnIndentifier`
`boolean`	`wasAcr`
`char[]`	`withoutUnicodeBuffer`
`int`	`withoutUnicodePtr`

Fields inherited from interface org.eclipse.jdt.internal.compiler.parser.TerminalTokens
TokenNameabstract, TokenNameAND, TokenNameAND_AND, TokenNameAND_EQUAL, TokenNameassert, TokenNameAT, TokenNameboolean, TokenNamebreak, TokenNamebyte, TokenNamecase, TokenNamecatch, TokenNamechar, TokenNameCharacterLiteral, TokenNameclass, TokenNameCOLON, TokenNameCOMMA, TokenNameCOMMENT_BLOCK, TokenNameCOMMENT_JAVADOC, TokenNameCOMMENT_LINE, TokenNameconst, TokenNamecontinue, TokenNamedefault, TokenNameDIVIDE, TokenNameDIVIDE_EQUAL, TokenNamedo, TokenNameDOT, TokenNamedouble, TokenNameDoubleLiteral, TokenNameELLIPSIS, TokenNameelse, TokenNameenum, TokenNameEOF, TokenNameEQUAL, TokenNameEQUAL_EQUAL, TokenNameERROR, TokenNameextends, TokenNamefalse, TokenNamefinal, TokenNamefinally, TokenNamefloat, TokenNameFloatingPointLiteral, TokenNamefor, TokenNamegoto, TokenNameGREATER, TokenNameGREATER_EQUAL, TokenNameIdentifier, TokenNameif, TokenNameimplements, TokenNameimport, TokenNameinstanceof, TokenNameint, TokenNameIntegerLiteral, TokenNameinterface, TokenNameLBRACE, TokenNameLBRACKET, TokenNameLEFT_SHIFT, TokenNameLEFT_SHIFT_EQUAL, TokenNameLESS, TokenNameLESS_EQUAL, TokenNamelong, TokenNameLongLiteral, TokenNameLPAREN, TokenNameMINUS, TokenNameMINUS_EQUAL, TokenNameMINUS_MINUS, TokenNameMULTIPLY, TokenNameMULTIPLY_EQUAL, TokenNamenative, TokenNamenew, TokenNameNOT, TokenNameNOT_EQUAL, TokenNamenull, TokenNameOR, TokenNameOR_EQUAL, TokenNameOR_OR, TokenNamepackage, TokenNamePLUS, TokenNamePLUS_EQUAL, TokenNamePLUS_PLUS, TokenNameprivate, TokenNameprotected, TokenNamepublic, TokenNameQUESTION, TokenNameRBRACE, TokenNameRBRACKET, TokenNameREMAINDER, TokenNameREMAINDER_EQUAL, TokenNamereturn, TokenNameRIGHT_SHIFT, TokenNameRIGHT_SHIFT_EQUAL, TokenNameRPAREN, TokenNameSEMICOLON, TokenNameshort, TokenNamestatic, TokenNamestrictfp, TokenNameStringLiteral, TokenNamesuper, TokenNameswitch, TokenNamesynchronized, TokenNamethis, TokenNamethrow, TokenNamethrows, TokenNametransient, TokenNametrue, TokenNametry, TokenNameTWIDDLE, TokenNameUNSIGNED_RIGHT_SHIFT, TokenNameUNSIGNED_RIGHT_SHIFT_EQUAL, TokenNamevoid, TokenNamevolatile, TokenNamewhile, TokenNameWHITESPACE, TokenNameXOR, TokenNameXOR_EQUAL

Constructor Summary

Constructors
Constructor and Description
`Scanner()`
`Scanner(boolean tokenizeComments, boolean tokenizeWhiteSpace, boolean checkNonExternalizedStringLiterals, long sourceLevel, char[][] taskTags, char[][] taskPriorities, boolean isTaskCaseSensitive)`
`Scanner(boolean tokenizeComments, boolean tokenizeWhiteSpace, boolean checkNonExternalizedStringLiterals, long sourceLevel, long complianceLevel, char[][] taskTags, char[][] taskPriorities, boolean isTaskCaseSensitive)`

Method Summary

Methods
Modifier and Type	Method and Description
`boolean`	`atEnd()`
`void`	`checkTaskTag(int commentStart, int commentEnd)`
`char[]`	`getCurrentIdentifierSource()`
`String`	`getCurrentStringLiteral()`
`int`	`getCurrentTokenEndPosition()`
`char[]`	`getCurrentTokenSource()`
`char[]`	`getCurrentTokenSourceString()`
`int`	`getCurrentTokenStartPosition()`
`String`	`getCurrentTokenString()`
`int`	`getLineEnd(int lineNumber)`
`int[]`	`getLineEnds()`
`int`	`getLineNumber(int position)` Search the line number corresponding to a specific position
`int`	`getLineStart(int lineNumber)` Search the source position corresponding to the beginning of a given line number Line numbers are 1-based, and relative to the scanner initialPosition.
`int`	`getNextChar()`
`boolean`	`getNextChar(char testedChar)`
`int`	`getNextChar(char testedChar1, char testedChar2)`
`boolean`	`getNextCharAsDigit()`
`boolean`	`getNextCharAsDigit(int radix)`
`boolean`	`getNextCharAsJavaIdentifierPart()`
`boolean`	`getNextCharAsJavaIdentifierPartWithBoundCheck()`
`int`	`getNextCharWithBoundChecks()`
`int`	`getNextToken()`
`void`	`getNextUnicodeChar()`
`NLSTag[]`	`getNLSTags()`
`char[]`	`getRawTokenSource()`
`char[]`	`getRawTokenSourceEnd()`
`char[]`	`getSource()`
`protected boolean`	`isFirstTag()`
`static boolean`	`isIdentifier(int token)`
`static boolean`	`isKeyword(int token)`
`static boolean`	`isLiteral(int token)`
`void`	`jumpOverMethodBody()`
`boolean`	`jumpOverUnicodeWhiteSpace()`
`void`	`pushLineSeparator()`
`void`	`pushUnicodeLineSeparator()`
`void`	`recordComment(int token)`
`void`	`resetTo(int begin, int end)` Reposition the scanner on some portion of the original source.
`protected void`	`scanEscapeCharacter()`
`int`	`scanIdentifier()`
`int`	`scanIdentifierOrKeyword()`
`int`	`scanIdentifierOrKeywordWithBoundCheck()`
`int`	`scanNumber(boolean dotPrefix)`
`void`	`setSource(char[] sourceString)`
`void`	`setSource(char[] contents, CompilationResult compilationResult)`
`void`	`setSource(CompilationResult compilationResult)`
`String`	`toString()`
`String`	`toStringAction(int act)`
`void`	`unicodeInitializeBuffer(int length)`
`void`	`unicodeStore()`
`void`	`unicodeStore(char character)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

sourceLevel
```
public long sourceLevel
```

complianceLevel
```
public long complianceLevel
```

useAssertAsAnIndentifier

public boolean useAssertAsAnIndentifier

containsAssertKeyword
```
public boolean containsAssertKeyword
```

useEnumAsAnIndentifier
```
public boolean useEnumAsAnIndentifier
```

recordLineSeparator
```
public boolean recordLineSeparator
```

currentCharacter
```
public char currentCharacter
```

startPosition
```
public int startPosition
```

currentPosition
```
public int currentPosition
```

initialPosition
```
public int initialPosition
```

eofPosition
```
public int eofPosition
```

skipComments
```
public boolean skipComments
```

tokenizeComments
```
public boolean tokenizeComments
```

tokenizeWhiteSpace
```
public boolean tokenizeWhiteSpace
```

source
```
public char[] source
```

withoutUnicodeBuffer
```
public char[] withoutUnicodeBuffer
```

withoutUnicodePtr
```
public int withoutUnicodePtr
```

unicodeAsBackSlash
```
public boolean unicodeAsBackSlash
```

scanningFloatLiteral
```
public boolean scanningFloatLiteral
```

COMMENT_ARRAYS_SIZE
```
public static final int COMMENT_ARRAYS_SIZE
```
See Also:
Constant Field Values

commentStops
```
public int[] commentStops
```

commentStarts
```
public int[] commentStarts
```

commentTagStarts
```
public int[] commentTagStarts
```

commentPtr
```
public int commentPtr
```

lastCommentLinePosition
```
protected int lastCommentLinePosition
```

foundTaskTags
```
public char[][] foundTaskTags
```

foundTaskMessages
```
public char[][] foundTaskMessages
```

foundTaskPriorities
```
public char[][] foundTaskPriorities
```

foundTaskPositions
```
public int[][] foundTaskPositions
```

foundTaskCount
```
public int foundTaskCount
```

taskTags
```
public char[][] taskTags
```

taskPriorities
```
public char[][] taskPriorities
```

isTaskCaseSensitive
```
public boolean isTaskCaseSensitive
```

diet
```
public boolean diet
```

lineEnds
```
public int[] lineEnds
```

linePtr
```
public int linePtr
```

wasAcr
```
public boolean wasAcr
```

END_OF_SOURCE

public static final String END_OF_SOURCE

See Also:: Constant Field Values

INVALID_HEXA

public static final String INVALID_HEXA

See Also:: Constant Field Values

INVALID_OCTAL

public static final String INVALID_OCTAL

See Also:: Constant Field Values

INVALID_CHARACTER_CONSTANT

public static final String INVALID_CHARACTER_CONSTANT

See Also:: Constant Field Values

INVALID_ESCAPE

public static final String INVALID_ESCAPE

See Also:: Constant Field Values

INVALID_INPUT

public static final String INVALID_INPUT

See Also:: Constant Field Values

INVALID_UNICODE_ESCAPE

public static final String INVALID_UNICODE_ESCAPE

See Also:: Constant Field Values

INVALID_FLOAT

public static final String INVALID_FLOAT

See Also:: Constant Field Values

INVALID_LOW_SURROGATE

public static final String INVALID_LOW_SURROGATE

See Also:: Constant Field Values

INVALID_HIGH_SURROGATE

public static final String INVALID_HIGH_SURROGATE

See Also:: Constant Field Values

NULL_SOURCE_STRING

public static final String NULL_SOURCE_STRING

See Also:: Constant Field Values

UNTERMINATED_STRING

public static final String UNTERMINATED_STRING

See Also:: Constant Field Values

UNTERMINATED_COMMENT

public static final String UNTERMINATED_COMMENT

See Also:: Constant Field Values

INVALID_CHAR_IN_STRING

public static final String INVALID_CHAR_IN_STRING

See Also:: Constant Field Values

INVALID_DIGIT

public static final String INVALID_DIGIT

See Also:: Constant Field Values

INVALID_BINARY

public static final String INVALID_BINARY

See Also:: Constant Field Values

BINARY_LITERAL_NOT_BELOW_17

public static final String BINARY_LITERAL_NOT_BELOW_17

See Also:: Constant Field Values

ILLEGAL_HEXA_LITERAL

public static final String ILLEGAL_HEXA_LITERAL

See Also:: Constant Field Values

INVALID_UNDERSCORE

public static final String INVALID_UNDERSCORE

See Also:: Constant Field Values

UNDERSCORES_IN_LITERALS_NOT_BELOW_17

public static final String UNDERSCORES_IN_LITERALS_NOT_BELOW_17

See Also:: Constant Field Values

OptimizedLength
```
public static final int OptimizedLength
```
See Also:
Constant Field Values

charArray_length

public final char[][][][] charArray_length

TAG_PREFIX
```
public static final char[] TAG_PREFIX
```

TAG_PREFIX_LENGTH

public static final int TAG_PREFIX_LENGTH

TAG_POSTFIX
```
public static final char TAG_POSTFIX
```
See Also:
Constant Field Values

TAG_POSTFIX_LENGTH
```
public static final int TAG_POSTFIX_LENGTH
```
See Also:
Constant Field Values

nlsTagsPtr
```
protected int nlsTagsPtr
```

checkNonExternalizedStringLiterals

public boolean checkNonExternalizedStringLiterals

lastPosition
```
protected int lastPosition
```

returnOnlyGreater
```
public boolean returnOnlyGreater
```

insideRecovery
```
public boolean insideRecovery
```

RoundBracket
```
public static final int RoundBracket
```
See Also:
Constant Field Values

SquareBracket
```
public static final int SquareBracket
```
See Also:
Constant Field Values

CurlyBracket
```
public static final int CurlyBracket
```
See Also:
Constant Field Values

BracketKinds
```
public static final int BracketKinds
```
See Also:
Constant Field Values

LOW_SURROGATE_MIN_VALUE
```
public static final int LOW_SURROGATE_MIN_VALUE
```
See Also:
Constant Field Values

HIGH_SURROGATE_MIN_VALUE
```
public static final int HIGH_SURROGATE_MIN_VALUE
```
See Also:
Constant Field Values

HIGH_SURROGATE_MAX_VALUE
```
public static final int HIGH_SURROGATE_MAX_VALUE
```
See Also:
Constant Field Values

LOW_SURROGATE_MAX_VALUE
```
public static final int LOW_SURROGATE_MAX_VALUE
```
See Also:
Constant Field Values

Constructor Detail

Scanner
```
public Scanner()
```

Scanner

public Scanner(boolean tokenizeComments,
       boolean tokenizeWhiteSpace,
       boolean checkNonExternalizedStringLiterals,
       long sourceLevel,
       long complianceLevel,
       char[][] taskTags,
       char[][] taskPriorities,
       boolean isTaskCaseSensitive)

Scanner

public Scanner(boolean tokenizeComments,
       boolean tokenizeWhiteSpace,
       boolean checkNonExternalizedStringLiterals,
       long sourceLevel,
       char[][] taskTags,
       char[][] taskPriorities,
       boolean isTaskCaseSensitive)

Method Detail

atEnd
```
public final boolean atEnd()
```

checkTaskTag

public void checkTaskTag(int commentStart,
                int commentEnd)
                  throws InvalidInputException

Throws:: InvalidInputException

getCurrentIdentifierSource

public char[] getCurrentIdentifierSource()

getCurrentTokenEndPosition

public int getCurrentTokenEndPosition()

getCurrentTokenSource
```
public char[] getCurrentTokenSource()
```

getCurrentTokenString

public final String getCurrentTokenString()

getCurrentTokenSourceString

public char[] getCurrentTokenSourceString()

getCurrentStringLiteral

public final String getCurrentStringLiteral()

getRawTokenSource

public final char[] getRawTokenSource()

getRawTokenSourceEnd

public final char[] getRawTokenSourceEnd()

getCurrentTokenStartPosition

public int getCurrentTokenStartPosition()

getLineEnd

public final int getLineEnd(int lineNumber)

getLineEnds
```
public final int[] getLineEnds()
```

getLineStart
```
public final int getLineStart(int lineNumber)
```
Search the source position corresponding to the beginning of a given line number Line numbers are 1-based, and relative to the scanner initialPosition. Character positions are 0-based. e.g. getLineStart(1) --> 0 indicates that the first line starts at character 0. In case the given line number is inconsistent, answers -1.

Parameters:
lineNumber - int

Returns:
int

getNextChar
```
public final int getNextChar()
```

getNextCharWithBoundChecks

public final int getNextCharWithBoundChecks()

getNextChar

public final boolean getNextChar(char testedChar)

getNextChar

public final int getNextChar(char testedChar1,
              char testedChar2)

getNextCharAsDigit

public final boolean getNextCharAsDigit()
                                 throws InvalidInputException

Throws:: InvalidInputException

getNextCharAsDigit

public final boolean getNextCharAsDigit(int radix)

getNextCharAsJavaIdentifierPartWithBoundCheck

public boolean getNextCharAsJavaIdentifierPartWithBoundCheck()

getNextCharAsJavaIdentifierPart

public boolean getNextCharAsJavaIdentifierPart()

scanIdentifier

public int scanIdentifier()
                   throws InvalidInputException

Throws:: InvalidInputException

getNextToken

public int getNextToken()
                 throws InvalidInputException

Throws:: InvalidInputException

getNextUnicodeChar

public void getNextUnicodeChar()
                        throws InvalidInputException

Throws:: InvalidInputException

getNLSTags
```
public NLSTag[] getNLSTags()
```

getSource
```
public char[] getSource()
```

isFirstTag
```
protected boolean isFirstTag()
```

jumpOverMethodBody

public final void jumpOverMethodBody()

jumpOverUnicodeWhiteSpace

public final boolean jumpOverUnicodeWhiteSpace()
                                        throws InvalidInputException

Throws:: InvalidInputException

pushLineSeparator
```
public final void pushLineSeparator()
```

pushUnicodeLineSeparator

public final void pushUnicodeLineSeparator()

recordComment
```
public void recordComment(int token)
```

resetTo
```
public void resetTo(int begin,
           int end)
```
Reposition the scanner on some portion of the original source. The given endPosition is the last valid position. Beyond this position, the scanner will answer EOF tokens (ITerminalSymbols.TokenNameEOF).

Parameters:
begin - the given start position
end - the given end position

scanEscapeCharacter

protected final void scanEscapeCharacter()
                                  throws InvalidInputException

Throws:: InvalidInputException

scanIdentifierOrKeywordWithBoundCheck

public int scanIdentifierOrKeywordWithBoundCheck()

scanIdentifierOrKeyword
```
public int scanIdentifierOrKeyword()
```

scanNumber

public int scanNumber(boolean dotPrefix)
               throws InvalidInputException

Throws:: InvalidInputException

getLineNumber
```
public final int getLineNumber(int position)
```
Search the line number corresponding to a specific position

Parameters:
position - int

Returns:
int

setSource

public final void setSource(char[] sourceString)

setSource

public final void setSource(char[] contents,
             CompilationResult compilationResult)

setSource

public final void setSource(CompilationResult compilationResult)

toString
```
public String toString()
```
Overrides:

toString in class Object

toStringAction

public String toStringAction(int act)

unicodeInitializeBuffer

public void unicodeInitializeBuffer(int length)

unicodeStore
```
public void unicodeStore()
```

unicodeStore

public void unicodeStore(char character)

isIdentifier

public static boolean isIdentifier(int token)

isLiteral

public static boolean isLiteral(int token)

isKeyword

public static boolean isKeyword(int token)

Class Scanner

Field Summary

Fields inherited from interface org.eclipse.jdt.internal.compiler.parser.TerminalTokens

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

sourceLevel

complianceLevel

useAssertAsAnIndentifier

containsAssertKeyword

useEnumAsAnIndentifier

recordLineSeparator

currentCharacter

startPosition

currentPosition

initialPosition

eofPosition

skipComments

tokenizeComments

tokenizeWhiteSpace

source

withoutUnicodeBuffer

withoutUnicodePtr

unicodeAsBackSlash

scanningFloatLiteral

COMMENT_ARRAYS_SIZE

commentStops

commentStarts

commentTagStarts

commentPtr

lastCommentLinePosition

foundTaskTags

foundTaskMessages

foundTaskPriorities

foundTaskPositions

foundTaskCount

taskTags

taskPriorities

isTaskCaseSensitive

diet

lineEnds

linePtr

wasAcr

END_OF_SOURCE

INVALID_HEXA

INVALID_OCTAL

INVALID_CHARACTER_CONSTANT

INVALID_ESCAPE

INVALID_INPUT

INVALID_UNICODE_ESCAPE

INVALID_FLOAT

INVALID_LOW_SURROGATE

INVALID_HIGH_SURROGATE

NULL_SOURCE_STRING

UNTERMINATED_STRING

UNTERMINATED_COMMENT

INVALID_CHAR_IN_STRING

INVALID_DIGIT

INVALID_BINARY

BINARY_LITERAL_NOT_BELOW_17

ILLEGAL_HEXA_LITERAL

INVALID_UNDERSCORE

UNDERSCORES_IN_LITERALS_NOT_BELOW_17

OptimizedLength

charArray_length

TAG_PREFIX

TAG_PREFIX_LENGTH

TAG_POSTFIX

TAG_POSTFIX_LENGTH

nlsTagsPtr

checkNonExternalizedStringLiterals

lastPosition

returnOnlyGreater

insideRecovery

RoundBracket

SquareBracket

CurlyBracket

BracketKinds

LOW_SURROGATE_MIN_VALUE