Writing a compiler for DSL in python

Question

I am writing a python game and decided to create a DSL for map data files. I know that I can write my own parser with a regular expression, but I am wondering if there are existing python tools that can do this more easily, for example re2c, which is used in the PHP engine.

Additional Information:

• Yes, I need DSL, and even if I didn’t, I still need experience building and using one in a project.

• DSL contains only data (declarative?), It does not get "done". Most lines look like this:

  SOMETHING: !abc @123 #xyz/123

  I just need to read the data tree.

Answer 1

Pyparsing has always fascinated me . The author, Paul McGuire, is active on the python / comp.lang.python list and has always been very helpful on any request.

Answer 2

There are many Python analysis tools: http://nedbatchelder.com/text/python-parsers.html

Answer 3

Here's an approach that works very well.

 abc= ONETHING( ... ) xyz= ANOTHERTHING( ... ) pqr= SOMETHING( this=abc, that=123, more=(xyz,123) ) 

declarative. Easy to parse.

BUT...

This is actually Python. Several class declarations and the work is done. DSLs are actually class declarations.

The important thing is that DSL just creates objects. When you define a DSL, you must first start with the object model. Later you put some syntax around this object model. You do not start with the syntax; you start with the model.

Answer 4

Yes, there are many - too many - parsing tools, but not in the standard library.

From what I saw, PLY and SPARK are popular. PLY is like yacc, but you do everything in Python because you write the grammar in the dock.

Personally, I like the concept of parser combinators (taken from functional programming), and I really like pyparsing : you write your grammar and actions directly in python, and it's easy to get started. As a result, I created my own types of node tree with actions, instead of using their default type ParserElement .

Otherwise, you can also use an existing declarative language, such as YAML .

Answer 5

I wrote something similar in work to read SNMP notification definitions and automatically generate Java classes and SNMP MIB files. Using this small DSL, I could write 20 lines of my specification, and it would generate about 80 lines of Java code and a 100-line MIB file.

To implement this, I simply used direct Python string processing (split (), slicing, etc.) to parse the file. I believe that the Pythons string parameters are sufficient for most of my (simple) parsing.

Besides the libraries mentioned by others, if I were to write something more complex and necessary for proper analysis, I would probably use ANTLR , which supports Python (and other languages).

Answer 6

Peter

DSL is good, so you don’t need to defend yourself :-) However, did you consider the internal DSL? They have so many pros compared to external (analyzed) DSLs that they, at least, deserve attention. Mixing DSL with native language permissions really solves a lot of problems for you, and Python is actually not so bad for internal DSLs with the with statement.

Answer 7

For "small languages" like the one you describe, I use simple split, shlex (the mind that # defines the comment) or regular expressions.

 >>> line = 'SOMETHING: !abc @123 #xyz/123' >>> line.split() ['SOMETHING:', '!abc', '@123', '#xyz/123'] >>> import shlex >>> list(shlex.shlex(line)) ['SOMETHING', ':', '!', 'abc', '@', '123'] 

Below is an example, since I don’t know exactly what you are looking for.

 >>> import re >>> result = re.match(r'([AZ]*): !([az]*) @([0-9]*) #([a-z0-9/]*)', line) >>> result.groups() ('SOMETHING', 'abc', '123', 'xyz/123') 

Answer 8

In declarative python lines, I wrote a helper module called "bpyml" that allows you to declare data in python in a more structured XML way without detailed tags, it can also be converted to / from XML, but python is valid.

https://svn.blender.org/svnroot/bf-blender/trunk/blender/release/scripts/modules/bpyml.py

Example usage http://wiki.blender.org/index.php/User:Ideasman42#Declarative_UI_In_Blender