PyTuple_SetItem Limitations

Question

I have a Python extension module that creates a tuple as an attribute of another object and sets elements in a tuple. Whenever I execute this module in Python, I keep getting a SystemError: bad argument to internal function error SystemError: bad argument to internal function

After reading the documents for PyTuple and debugging my program for several hours, I still could not figure out what the hell was going on. Running my program through the debugger showed that the problem arose inside the library call inside the Python interpreter. So finally, I looked at the Python source code and finally understood the problem. The PyTuple_SetItem function has an interesting limitation that I did not know about, and I cannot find explicitly documented.

Here is an important function in the Python source (edited for clarity):

 int PyTuple_SetItem(register PyObject *op, register Py_ssize_t i, PyObject *newitem) { ..... if (!PyTuple_Check(op) || op->ob_refcnt != 1) { Py_XDECREF(newitem); PyErr_BadInternalCall(); return -1; } ..... } 

An important line is the condition op-> ob_refcnt! = 1 . So here is the problem: you cannot even call PyTuple_SetItem if Tuple does not have a ref-count of 1. It seems like the idea is that you should never use PyTuple_SetItem , except immediately after creating the tuple using PyTuple_New() . I assume this makes sense, since Tuples should be immutable, after all, so this limitation helps keep your C code in line with abstractions of a system like Python.

However, I cannot find this restriction documented. Relevant documents look here and here , none of which indicate this restriction. The docs basically say that when PyTuple_New(X) is called, all elements in the tuple are initialized to NULL . Since NULL not a valid Python value, it is important for the extension programmer to make sure that all slots in the Tuple are filled with the correct Python values ​​before returning the Tuple to the interpreter. But he does not say anywhere that this should be done, while the Tuple object has a value of ref 1.

So the problem is that I basically encoded myself in the corner, because I did not know about this (undocumented?) Restriction on PyTuple_SetItem . My code is structured in such a way that it is very difficult to insert elements into a tuple until the tuple itself becomes an attribute of another object. So, when it comes time to populate the elements in the tuple, the tuple already has a higher ref value.

I will probably have to rebuild my code, but I was seriously thinking about temporarily setting the ref value on Tuple to 1, insert the elements, and then restore the original link count. Of course, this is a terrible hack, I know, and not some permanent solution. Regardless, I would like to know if the requirement for the number of links to Tuple is documented anywhere. Is this just a detail of the implementation of CPython, or is it that API users can rely on expected behavior?

Answer 1

I'm sure you can get around the limitations by using PyTuple_SET_ITEM instead of PyTuple_SetItem . PyTuple_SET_ITEM is a macro defined in tupleobject.h as follows:

 #define PyTuple_SET_ITEM(op, i, v) (((PyTupleObject*)(op))->ob_item[i] = v 

So, if you are absolutely, definitely and completely sure that:

• op is a tuple object
• you have not initialized slot i in the tuple so far
• you have a link to v and you want the tuple to steal it and
• there is no other Python object using a tuple for anything before you call PyTuple_SET_ITEM

then I think you can use PyTuple_SET_ITEM .

Answer 2

The Python C API is very underestimated, and I won’t be surprised if this restriction was not mentioned anywhere.

Of course, you should never change tuples as soon as something holds them back; either go to the elements you want to put in the tuple, or use a list.