com.oracle.js.parser

Class Lexer

java.lang.Object

com.oracle.js.parser.Scanner

com.oracle.js.parser.Lexer

All Implemented Interfaces:

StringPool

public class Lexer
extends Scanner
implements StringPool

Responsible for converting source content into a stream of tokens.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Lexer.LexerToken Helper class for Lexer tokens, e.g XML or RegExp tokens.
protected static interface	Lexer.LineInfoReceiver interface to receive line information for multi-line literals.
static class	Lexer.RegexToken Temporary container for regular expressions.
static class	Lexer.XMLToken Temporary container for XML expression.

Field Summary

Fields inherited from class com.oracle.js.parser.Scanner

ch0, ch1, ch2, ch3, content, limit, line, position

Constructor Summary

Constructors

Constructor and Description
Lexer(Source source, int start, int len, TokenStream stream, boolean scripting, int ecmaScriptVersion, boolean shebang, boolean isModule, boolean pauseOnFunctionBody, boolean allowBigInt) Constructor
Lexer(Source source, TokenStream stream, boolean scripting, int ecmaScriptVersion, boolean shebang, boolean isModule, boolean allowBigInt) Constructor

Method Summary

Modifier and Type	Method and Description
protected void	add(TokenType type, int start) Add a new token to the stream.
protected void	add(TokenType type, int start, int end) Add a new token to the stream.
boolean	canStartLiteral(TokenType token) Return true if the given token can be the beginning of a literal.
boolean	checkIdentForKeyword(long token, String keyword)
protected static int	convertDigit(char ch, int base) Convert a digit to a integer.
protected void	error(String message, TokenType type, int start, int length) Generate a runtime exception
protected boolean	isEOL(char ch) Test whether a char is valid JavaScript end of line
protected boolean	isEscapeCharacter(char ch) Is the given character a valid escape char after "\" ?
static boolean	isJSEOL(char ch) Test whether a char is valid JavaScript end of line
static boolean	isJSWhitespace(char ch) Test whether a char is valid JavaScript whitespace
protected boolean	isStringDelimiter(char ch) Test if char is a string delimiter, e.g.
static boolean	isStringLineTerminator(char ch) Test whether a char is valid JavaScript end of string.
protected boolean	isWhitespace(char ch) Test whether a char is valid JavaScript whitespace
void	lexify() Breaks source content down into lex units, adding tokens to the token stream.
protected static String	message(String msgId, String... args) Get the correctly localized error message for a given message id format arguments
protected boolean	scanLiteral(long token, TokenType startTokenType, Lexer.LineInfoReceiver lir) Check whether the given token represents the beginning of a literal.
protected void	scanNumber() Scan a number.
protected void	scanString(boolean add) Scan over a string literal.
protected void	scanTemplateSpan() Continue scanning a template literal after an expression.
com.oracle.truffle.api.strings.TruffleString	stringIntern(String candidate)
com.oracle.truffle.api.strings.TruffleString	stringIntern(com.oracle.truffle.api.strings.TruffleString candidate)
Lexer.RegexToken	valueOfPattern(int start, int length) Convert a regex token to a token object.
com.oracle.truffle.api.strings.TruffleString	valueOfRawString(long token) Get the raw string value of a template literal string part.

Methods inherited from class com.oracle.js.parser.Scanner

atEOF, charAt, reset, skip

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Lexer

public Lexer(Source source,
             TokenStream stream,
             boolean scripting,
             int ecmaScriptVersion,
             boolean shebang,
             boolean isModule,
             boolean allowBigInt)

Constructor

Parameters:

source - the source

stream - the token stream to lex

scripting - are we in scripting mode

ecmaScriptVersion - ECMAScript language version

shebang - do we support shebang

isModule - are we in module

Lexer

public Lexer(Source source,
             int start,
             int len,
             TokenStream stream,
             boolean scripting,
             int ecmaScriptVersion,
             boolean shebang,
             boolean isModule,
             boolean pauseOnFunctionBody,
             boolean allowBigInt)

Constructor

Parameters:

source - the source

start - start position in source from which to start lexing

len - length of source segment to lex

stream - token stream to lex

scripting - are we in scripting mode

ecmaScriptVersion - ECMAScript language version

shebang - do we support shebang

isModule - are we in module

pauseOnFunctionBody - if true, lexer will return from Lexer.lexify() when it encounters a function body. This is used with the feature where the parser is skipping nested function bodies to avoid reading ahead unnecessarily when we skip the function bodies.

Method Detail

add

protected void add(TokenType type,
                   int start,
                   int end)

Add a new token to the stream.

Parameters:

type - Token type.

start - Start position.

end - End position.

add

protected void add(TokenType type,
                   int start)

Add a new token to the stream.

Parameters:

type - Token type.

start - Start position.

isJSWhitespace

public static boolean isJSWhitespace(char ch)

Test whether a char is valid JavaScript whitespace

Parameters:

ch - a char

Returns:

true if valid JavaScript whitespace

isJSEOL

public static boolean isJSEOL(char ch)

Test whether a char is valid JavaScript end of line

Parameters:

ch - a char

Returns:

true if valid JavaScript end of line

isStringLineTerminator

public static boolean isStringLineTerminator(char ch)

Test whether a char is valid JavaScript end of string. Line separators and paragraph separators can appear in JavaScript string literals.

Parameters:

ch - a char

Returns:

true if valid JavaScript end of string

isStringDelimiter

protected boolean isStringDelimiter(char ch)

Test if char is a string delimiter, e.g. '\' or '"'.

Parameters:

ch - a char

Returns:

true if string delimiter

isWhitespace

protected boolean isWhitespace(char ch)

Test whether a char is valid JavaScript whitespace

Parameters:

ch - a char

Returns:

true if valid JavaScript whitespace

isEOL

protected boolean isEOL(char ch)

Test whether a char is valid JavaScript end of line

Parameters:

ch - a char

Returns:

true if valid JavaScript end of line

valueOfPattern

public Lexer.RegexToken valueOfPattern(int start,
                                       int length)

Convert a regex token to a token object.

Parameters:

start - Position in source content.

length - Length of regex token.

Returns:

Regex token object.

canStartLiteral

public boolean canStartLiteral(TokenType token)

Return true if the given token can be the beginning of a literal.

Parameters:

token - a token

Returns:

true if token can start a literal.

scanLiteral

protected boolean scanLiteral(long token,
                              TokenType startTokenType,
                              Lexer.LineInfoReceiver lir)

Check whether the given token represents the beginning of a literal. If so scan the literal and return true, otherwise return false.

Parameters:

token - the token.

startTokenType - the token type.

lir - LineInfoReceiver that receives line info for multi-line string literals.

Returns:

True if a literal beginning with startToken was found and scanned.

convertDigit

protected static int convertDigit(char ch,
                                  int base)

Convert a digit to a integer. Can't use Character.digit since we are restricted to ASCII by the spec.

Parameters:

ch - Character to convert.

base - Numeric base.

Returns:

The converted digit or -1 if invalid.

checkIdentForKeyword

public boolean checkIdentForKeyword(long token,
                                    String keyword)

scanString

protected void scanString(boolean add)

Scan over a string literal.

Parameters:

add - true if we are not just scanning but should actually modify the token stream

scanTemplateSpan

protected final void scanTemplateSpan()

Continue scanning a template literal after an expression.

isEscapeCharacter

protected boolean isEscapeCharacter(char ch)

Is the given character a valid escape char after "\" ?

Parameters:

ch - character to be checked

Returns:

if the given character is valid after "\"

scanNumber

protected void scanNumber()

Scan a number.

lexify

public void lexify()

Breaks source content down into lex units, adding tokens to the token stream. The routine scans until the stream buffer is full. Can be called repeatedly until EOF is detected.

valueOfRawString

public com.oracle.truffle.api.strings.TruffleString valueOfRawString(long token)

Get the raw string value of a template literal string part.

Parameters:

token - template string token

Returns:

raw string

stringIntern

public com.oracle.truffle.api.strings.TruffleString stringIntern(com.oracle.truffle.api.strings.TruffleString candidate)

Specified by:

stringIntern in interface StringPool

stringIntern

public com.oracle.truffle.api.strings.TruffleString stringIntern(String candidate)

message

protected static String message(String msgId,
                                String... args)

Get the correctly localized error message for a given message id format arguments

Parameters:

msgId - message id

args - format arguments

Returns:

message

error

protected void error(String message,
                     TokenType type,
                     int start,
                     int length)
              throws ParserException

Generate a runtime exception

Parameters:

message - error message

type - token type

start - start position of lexed error

length - length of lexed error

Throws:

ParserException - unconditionally