Get trace information, including SyntaxError from compilation ()

Question

Main problem

It appears that the SyntaxError (and TypeError s) raised by the compile() function are not included in the stack trace returned by sys.exc_info() , but are printed as part of the formatted output using traceback.print_exc .

Example

For example, given the following code (where filename is the name of the file containing the Python code with the line $ flagrant syntax error ):

 import sys from traceback import extract_tb try: with open(filename) as f: code = compile(f.read(), filename, "exec") except: print "using sys.exc_info:" tb_list = extract_tb(sys.exc_info()[2]) for f in tb_list: print f print "using traceback.print_exc:" from traceback import print_exc print_exc() 

I get the following output (where <scriptname> is the name of the script containing the code above):

 using sys.exc_info: ('<scriptname>', 6, 'exec_file', 'code = compile(f.read(), filename, "exec")') using traceback.print_exc: Traceback (most recent call last): File "<scriptname>", line 6, in exec_file code = compile(f.read(), filename, "exec") File "<filename>", line 3 $ flagrant syntax error ^ SyntaxError: invalid syntax 

My questions

I have three questions:

• Why does the trace with sys.exc_info() not include the frame in which SyntaxError was created?
• How to traceback.print_exc get information about the missing frame?
• What is the best way to keep the "extracted" trace information, including SyntaxError , in a list? Should the last element of the list (i.e., information from the stack frame representing where SyntaxError occurred) be created using filename and the SyntaxError exception object SyntaxError ?

---

Case Example

In context, here is my use case for trying to get a full stack trace extraction.

I have a program that essentially implements DSL using exec with some files containing user-written Python code. (Regardless of whether this is a good DSL implementation strategy, I am more or less stuck with it.) If I encounter an error in the user code, I would (in some cases) as an interpreter to save the error for later, and Do not vomit traces of the stack and death. Therefore, I have a ScriptExcInfo class specifically designed to store this information. Here (a slightly edited version) is this __init__ class method, complete with a rather ugly workaround for the problem described above:

 def __init__(self, scriptpath, add_frame): self.exc, tb = sys.exc_info()[1:] self.tb_list = traceback.extract_tb(tb) if add_frame: # Note: I'm pretty sure that the names of the linenumber and # message attributes are undocumented, and I don't think there's # actually a good way to access them. if isinstance(exc, TypeError): lineno = -1 text = exc.message else: lineno = exc.lineno text = exc.text # '?' is used as the function name since there no function context # in which the SyntaxError or TypeError can occur. self.tb_list.append((scriptpath, lineno, '?', text)) else: # Pop off the frames related to this `exec` infrastructure. # Note that there no straightforward way to remove the unwanted # frames for the above case where the desired frame was explicitly # constructed and appended to tb_list, and in fact the resulting # list is out of order (!!!!). while scriptpath != self.tb_list[-1][0]: self.tb_list.pop() 

Note that “pretty ugly” I mean, this workaround turns what should be a one-parameter, 4-line __init__ function to require two arguments and take 13 lines.

Answer 1

The only difference between your two approaches is that print_exc() prints a formatted exception. For SyntaxError , which includes formatting the information in this exception, which includes the actual string that caused the problem.

For the trace itself, print_exc() uses sys.exc_info()[2] , the same information that you already use to create the trace. In other words, it does not receive more information than you already do, but you ignore the exception information itself:

 >>> import traceback >>> try: ... compile('Hmmm, nope!', '<stdin>', 'exec') ... except SyntaxError as e: ... print ''.join(traceback.format_exception_only(type(e), e)) ... File "<stdin>", line 1 Hmmm, nope! ^ SyntaxError: invalid syntax 

Here traceback.format_exception_only() is the undocumented function used by traceback.print_exc() to format the exception value. All information is available to you for the exception:

 >>> dir(e) ['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__getitem__', '__getslice__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__unicode__', 'args', 'filename', 'lineno', 'message', 'msg', 'offset', 'print_file_and_line', 'text'] >>> e.args ('invalid syntax', ('<stdin>', 1, 11, 'Hmmm, nope!\n')) >>> e.filename, e.lineno, e.offset, e.text ('<stdin>', 1, 11, 'Hmmm, nope!\n') 

Also see the documentation for traceback.print_exception() :

(3) if the type is SyntaxError , and the value is in the appropriate format, it prints the line where the syntax error occurred, using a caret indicating the approximate position of the error.

and SyntaxError documentation :

Instances of this class have filename , lineno , offset and text attributes for easier access to details. str() exception instance returns only the message.

The fact that the syntax error line is not included in the trace is logical; code with a syntax error could not be executed, so no execution block was created for it. And the exception is thrown by the compile() function, the bottommost frame of execution.

As such, you are stuck with your ugly approach; this is the right approach for handling SyntaxError exceptions. However attributes documented.

Note that exception.message usually set to exception.args[0] , and str(exception) usually gives you the same message (if args longer you get str(exception.args) ), although some types of exceptions provide a custom __str__ which often just gives you exception.args[0] ).