BufferedTokenStream Class

Namespace: Stimulsoft.Data.Expressions.Antlr.Runtime

Buffer all input tokens but do on-demand fetching of new tokens from lexer. Useful when the parser or lexer has to set context/mode info before proper lexing of future tokens. The ST template parser needs this, for example, because it has to constantly flip back and forth between inside/output templates. E.g., <names:{hi, <it>}> has to parse names as part of an expression but “hi, <it>” as a nested template. You can’t use this stream if you pass whitespace or other off-channel tokens to the parser. The stream can’t ignore off-channel tokens. (UnbufferedTokenStream is the same way.) This is not a subclass of UnbufferedTokenStream because I don’t want to confuse small moving window of tokens it uses for the full buffer.

Inheritance

Implements: ITokenStream, ITokenStreamInformation

Constructors

constructor(tokenSource: ITokenSource)

Parameters

• tokenSource (ITokenSource)

Properties

Methods

Method Details

consume

consume(): void

Move the input pointer to the next incoming token. The stream must become active with LT(1) available. consume() simply moves the input pointer so that LT(1) points at the next input symbol. Consume at least one token. Walk past any token not on the channel the parser is listening to.

fetch

fetch(n: number): void

add n elements to buffer

Parameters

• n (number)

fill

fill(): void

getTokens

getTokens(start: number, stop: number, types: BitSet): IToken[]

Given a start and stop index, return a List of all tokens in the token type BitSet. Return null if no tokens were found. This method looks at both on and off channel tokens.

Parameters

• start (number)
• stop (number)
• types (BitSet)

Returns IToken[]

implements

implements(): any[]

Returns any[]

la

la(i: number): number

Parameters

• i (number)

Returns number

lb

lb(k: number): IToken

Parameters

• k (number)

Returns IToken

lt

lt(k: number): IToken

Parameters

• k (number)

Returns IToken

mark

mark(): number

Returns number

release

release(marker: number): void

Parameters

• marker (number)

reset

reset(): void

rewind

rewind(marker: number): void

Parameters

• marker (number)

seek

seek(index: number): void

Parameters

• index (number)

setup

setup(): void

sync

sync(i: number): void

Make sure index i in tokens has a token.

Parameters

• i (number)

toString

toString(): string

Returns string

toString2

toString2(start: number, stop: number): string

Parameters

• start (number)
• stop (number)

Returns string

Fields