Access channels in ANTLR 4 and analyze them separately

Question

I included my comments in a separate channel in ANTLR 4. In my case, this is channel 2.

This is my lexical grammar.

COMMENT: '/*' .*? '*/' -> channel(2) ; 

I want to access this channel 2 and parse this channel to accumulate comments. So I included this as in parsing the grammar below

 comment :COMMENT ; 

In a programme

  string s = " paring string" AntlrInputStream input = new AntlrInputStream(s); CSSLexer lexer = new CSSLexer(input); CommonTokenStream tokens = new CommonTokenStream(lexer,2); 

Then I want to parse on tokens

 var xr = parser.comment().GetRuleContexts<CommentContext>(); 

because I want to get information from CommentContext object like Start.Column etc.

EDIT:

This is an improved question.

To be more specific, I want to get all the tokens in channel 2 and analyze them using the comment grammar to get all the comments on the list ( IReadOnly<CommentContext> ) so that I can iterate through each of them and access information such as start line, start column, end column of the end line, and token text.

 CommonTokenStream tokens = new CommonTokenStream(lexer,2); 

This does not give me tokens in channel 2. And one more thing I discovered until these tokens are passed as arguments to the XParser parser = new XParser(tokens);

Then only I can access the tokens by calling GetTokens() . In the volumes, I see that there are comments identified as tokens, and is in channel 2. Although CommentTokenStrem displays the channel number as described above. It contains all the tokens.

• What is the reason for the inability to access tokens until the parser object is created using tokens?

• I want to get CommentTokenStrem in channel 2 and pass it to the creation of an XParser object to parse these tokens using my comment grammar. What is the best way to do this in the ANTLR 4 API?

Answer 1

CommonTokenStream internally tracks all tokens from any channel. The only thing you will not see when you call getTokens() is the lexer rules in which the -> skip action was performed (the marker was not even created for these rules).

You can watch the markers on channel 2 using TokenStream.LT and IntStream.consume .

Java example

 CommonTokenStream cts = new CommonTokenStream(tokenSource, 2); List<Token> tokens = new ArrayList<Token>(); while (cts.LA(1) != EOF) { tokens.add(cts.LT(1)); cts.consume(); } 

C # example:

 CommonTokenStream cts = new CommonTokenStream(tokenSource, 2); IList<IToken> tokens = new List<IToken>(); while (cts.La(1) != Eof) { tokens.Add(cts.Lt(1)); cts.Consume(); } 

Answer 2

How about this:

  var allowedChannels = new[] { 2 }; // add more if you need to var tokensImInterestedIn = tokens.GetTokens().Where(token => allowedChannels.Contains(token.Channel) && token.Type != CSSLexer.Eof).ToArray(); // if you're just interested in one particular channel var tokensImInterestedIn = tokens.GetTokens().Where(token => token.Channel == 2) && token.Type != CSSLexer.Eof).ToArray(); 

Answer 3

Alternatively, you can put all other tokens in another channel and use the default channel for your parser.

Of course, this will not work if you have two parsers that expect tokens in separate channels.

Answer 4

ANTLR 4 C #:

  using Antlr4.Runtime; ... MyLexer lexer = new MyLexer (inputStream); var tokenstream = new CommonTokenStream(lexer, TokenConstants.HiddenChannel); IList<IToken> tokens = new List<IToken>(); while (tokenstream.La(1) != TokenConstants.Eof) { tokens.Add(tokenstream.Lt(1)); tokenstream.Consume(); } foreach (IToken iToken in tokens) { Console.WriteLine(" Line : {0} Text : {1} ", iToken.Line, iToken.Text ); }