| ANTLRStringStream |
A pretty quick CharStream that pulls all data from an array directly. Every method call counts in the lexer. Java’s strings aren’t very good so I’m avoiding. |
| BaseRecognizer |
A generic recognizer that can handle recognizers generated from lexer, parser, and tree grammars. This is all the parsing support code essentially; most of it is error recovery stuff and backtracking. |
| BitSet |
A stripped-down version of org.antlr.misc.BitSet that is just good enough to handle runtime requirements such as FOLLOW sets for automatic error recovery. |
| BufferedTokenStream |
Buffer all input tokens but do on-demand fetching of new tokens from lexer. Useful when the parser or lexer has to set context/mode info before proper lexing of future tokens. The ST template parser needs this, for example, because it has to constantly flip back and forth between inside/output templates. E.g., <names:{hi, <it>}> has to parse names as part of an expression but “hi, <it>” as a nested template. You can’t use this stream if you pass whitespace or other off-channel tokens to the parser. The stream can’t ignore off-channel tokens. (UnbufferedTokenStream is the same way.) This is not a subclass of UnbufferedTokenStream because I don’t want to confuse small moving window of tokens it uses for the full buffer. |
| CharStreamConstants |
|
| CharStreamState |
When walking ahead with cyclic DFA or for syntactic predicates, we need to record the state of the input stream (char index, line, etc…) so that we can rewind the state after scanning ahead. This is the complete state of a stream. |
| ClassicToken |
A Token object like we’d use in ANTLR 2.x; has an actual string created and associated with this object. These objects are needed for imaginary tree nodes that have payload objects. We need to create a Token object that has a string; the tree node will point at this token. CommonToken has indexes into a char stream and hence cannot be used to introduce new strings. |
| CommonToken |
|
| DFA |
A DFA implemented as a set of transition tables. Any state that has a semantic predicate edge is special; those states are generated with if-then-else structures in a specialStateTransition() which is generated by cyclicDFA template. There are at most 32767 states (16-bit signed short). Could get away with byte sometimes but would have to generate different types and the simulation code too. For a point of reference, the Java lexer’s Tokens rule DFA has 326 states roughly. |
| LegacyCommonTokenStream |
The most common stream of this.tokens is one where every token is buffered up and this.tokens are prefiltered for a certain channel (the parser will only see these this.tokens and cannot change the filter channel number during the parse). TODO: how to access the full token stream? How to track all this.tokens matched per rule? |
| ParserRuleReturnScope |
Rules that return more than a single value must return an object containing all the values. Besides the properties defined in RuleLabelScope.predefinedRulePropertiesScope there may be user-defined return values. This class simply defines the minimum properties that are always defined and methods to access the others that might be available depending on output option such as template and tree. Note text is not an actual property of the return value, it is computed from start and stop using the input stream’s toString() method. I could add a ctor to this so that we can pass in and store the input stream, but I’m not sure we want to do that. It would seem to be undefined to get the .text property anyway if the rule matches tokens from multiple input streams. I do not use getters for fields of objects that are used simply to group values such as this aggregate. The getters/setters are there to satisfy the superclass interface. |
| RecognizerSharedState |
The set of fields needed by an abstract recognizer to recognize input and recover from errors etc… As a separate state object, it can be shared among multiple grammars; e.g., when one grammar imports another. These fields are publically visible but the actual state pointer per parser is protected. |
| RewriteOperation |
|
| TokenChannels |
|
| TokenTypes |
|
| Tokens |
|
