ANTLR4: How to enter tokens

Question

I am trying to implement a preprocessor for DSL modeled after the CPP example in code / add-ons. However, I do not use the factory token. Is it required? The emit call (token) does not inject tokens into the token stream as expected.

Here's the lexer:

// string-delimited path  
SPATH     :  '"' (~[\n\r])*? '"'
                {
                 emit();  // inject the current token
                 // launch another lexer on the include file, get tokens,
                 // emit them all at once here
                 List<CommonToken> tokens = Preprocessor.include(getText());
                 if (null != tokens) {
                   for (CommonToken tok : tokens) {
                     emit(tok);
                   }
                 }
               }
      ;

Here's the include method:

@SuppressWarnings("unchecked")
public static List<CommonToken> include(String filename) {
    List<CommonToken> tokens = null;
    try (FileReader fr = openFile(filename.substring(1, filename.length() - 1));
            BufferedReader br = new BufferedReader(fr)) {
        ANTLRInputStream input = new ANTLRInputStream(br);
        PreprocessorLexer lexer = new PreprocessorLexer(input);

        tokens = (List<CommonToken>) lexer.getAllTokens();

    } catch (IOException ioe) {
        log.error("Can't load ~{}~", ioe.getLocalizedMessage());
    }
    return tokens;
}

Answer 1

To provide this function, you need to override Lexer.nextToken. In your lexer, keep the Deque<Token>entered tokens that have not yet been returned nextToken. When the queue is empty, your implementation nextTokenshould return the next token in accordance with the implementation of the superclass.

Here is a sample code. I did not try to compile or run it so that it was not perfect.

private final Deque<Token> pendingTokens = new ArrayDeque<>();

@Override
public Token nextToken() {
    Token pending = pendingTokens.pollFirst();
    if (pending != null) {
        return pending;
    }

    Token next = super.nextToken();
    pending = pendingTokens.pollFirst();
    if (pending != null) {
        pendingTokens.addLast(next);
        return pending;
    }

    return next;
}