How to ignore the delimiter when it comments or quotes

Question

I am writing a parser to find string concatenation expressions. I have a number of lines that are enclosed in parentheses, arising mainly from a function call.

For example, ("one"+"two"+"three") -> ("one"|"two"|"three") is a simple case, and I can handle it.

More complex is (null, "one"+"two"+"three", null) -> (null, "one"|"two"|"three", null) , but I can parse it with boost::tokenizer .

(null, "one"+"two"+"three,four", 1 /* third parameter can be: 1, 2, 3 */) , in such a complex example, I suggest parsing with boost::spirit , but I need help in writing some rules for him.

Further:

It seems that escaped_list_separator from boost::tokenizer is what I need. But I have one problem:

  using namespace std; using namespace boost; string s = "Field 1,\"putting quotes around fields, allows commas\",Field 3"; tokenizer<escaped_list_separator<char> > tok(s,escaped_list_separator<char>("", ",", "\"")); for(tokenizer<escaped_list_separator<char> >::iterator beg=tok.begin(); beg!=tok.end();++beg){ cout <<"~~~"<< *beg << "\n"; } 

removes " for me. You can save quotes in the output like this

 Field 1 "putting quotes around fields, allows commas" Field 3 

Answer 1

Basically, you can use operator- with matching characters:

  rule = '"' >> (char_ - '"') >> '"'; 

Also see operator ~ to invert the encoding.

If you are interested in speeding up quotes inside quotes, as well as possibly in style comments at the same time, I recommend looking at my answer here:

• How to make my split work only on one real line and be able to skip quoted parts of the line?

Display (partially) quoted cells in CSV files, including escaped quotation marks inside strings.

Other items of interest:

• Cross-platform way to get the line number of the INI file where this parameter is specified

  Ignoring comments and escaping comment delimiters

• Parsing strings with boost :: spirit

  Displays various styles for quoted string parsing, including built-in escaped quotation marks.