Warn every (nested) function with free variables (recursively)

Question

I would like to do the following:

for every nested function f anywhere in this_py_file: if has_free_variables(f): print warning 

Why? First of all, as insurance against late binding closure, as described elsewhere . Namely:

 >>> def outer(): ... rr = [] ... for i in range(3): ... def inner(): ... print i ... rr.append(inner) ... return rr ... >>> for f in outer(): f() ... 2 2 2 >>> 

And whenever I get a warning about a free variable, I would either add an explicit exception (in the rare case when I would like this behavior), or fix it like this:

  ... def inner ( i = i ):

Then the behavior is more like nested classes in Java (where any variable that should be used in the inner class must be final ).

(As far as I know, in addition to solving the late binding problem, this will also contribute to better memory utilization, because if the function “closes” some variables in the outer region, then the outer region cannot be garbage collected until the function is around. Correct ?)

I cannot find a way to get functions nested in other functions. Currently, the best way I can come up with is an analyzer tool that seems to work a lot.

Answer 1

Consider the following function:

 def outer_func(): outer_var = 1 def inner_func(): inner_var = outer_var return inner_var outer_var += 1 return inner_func 

The __code__ object can be used to restore the code object of an internal function:

 outer_code = outer_func.__code__ inner_code = outer_code.co_consts[2] 

Free variables can be restored from this code object:

 inner_code.co_freevars # ('outer_var',) 

You can check whether to check the code object with:

 hasattr(inner_code, 'co_freevars') # True 

After you get all the functions from your file, it might look something like this:

 for func in function_list: for code in outer_func.__code__.co_consts[1:-1]: if hasattr(code, 'co_freevars'): assert len(code.co_freevars) == 0 

Someone who knows more about internal work is likely to give a better explanation or a more concise solution.

Answer 2

To “hold” your nested functions (even if you override them), you will need to use eval to make the definition variable names for each declaration.

 def outer(): rr = [] for i in range(3): eval("def inner"+str(i)+"""(): print """+str(i)) rr.append(eval("inner"+str(i))) return rr for f in outer(): f() 

prints

 1 2 3 

Answer 3

I also wanted to do this in Jython. But the method shown in the accepted answer does not work there, because co_consts not available for the code object. (Also, there seems to be no other way to request a code object to get the code objects of nested functions.)

But, of course, there are code objects somewhere, we have the initial and full access, so it’s just a matter of finding a simple path within a reasonable amount of time. So, here is one way that works. (Hold on to your seats.)

Suppose the mod module has the following code:

 def outer(): def inner(): print "Inner" 

First, get the external function code object directly:

 code = mod.outer.__code__ 

In Jython, this is an instance of PyTableCode , and reading the source code, we find that the actual functions are implemented in a Java class made from your given script, which is referenced by object funcs code. (All of these scripted classes are subclasses of PyFunctionTable , which is why the declared type is funcs .) This is not visible from inside Jython, as a result of the magic mechanism, which is a design way to say that you are "addressing these things at your own risk.

So, we need to dive a little in Java. A class like this does the trick:

 import java.lang.reflect.Field; public class Getter { public static Object getFuncs(Object o) throws NoSuchFieldException, IllegalAccessException { Field f = o.getClass().getDeclaredField("funcs"); f.setAccessible(true); return f.get(o); } } 

Back to Jython:

 >>> import Getter >>> funcs = Getter.getFuncs(mod.outer.__code__) >>> funcs mod$py@1bfa3a2 

Now this funcs object has all the functions declared anywhere in the Jython script (those that are nested arbitrarily, inside classes, etc.). In addition, it has fields containing the corresponding code objects.

 >>> fields = funcs.class.getDeclaredFields() 

In my case, the code object corresponding to the nested function is the last one:

 >>> flast = fields[-1] >>> flast static final org.python.core.PyCode mod$py.inner$24 

To get the code object of interest:

 >>> flast.setAccessible(True) >>> inner_code = flast.get(None) #"None" because it a static field. >>> dir(inner_code) co_argcount co_filename co_flags co_freevars co_name co_nlocals co_varnames co_cellvars co_firstlineno 

And everything else matches the accepted answer, i.e. check out co_freevars , (which is in Jython, unlike co_consts ).

It’s good that this approach is that you list exactly all the code objects that are declared anywhere in the source code file: functions, methods, generators, regardless of whether they are installed or not or something they are under something or under each other. They have nowhere else to hide.

Answer 4

you need import copy and use rr.append(copy.copy(inner))

https://pymotw.com/2/copy/