Generate a function with arguments filled when creating it?

Question

My goal is to dynamically generate functions and then save them in a file. For example, in my current attempt, When create_file called

 import io def create_file(a_value): a_func = make_concrete_func(a_value) write_to_file([a_func], '/tmp/code.py') def make_concrete_func(a_value): def concrete_func(b, k): return b + k + a_value return concrete_func def write_to_file(code_list, path): import inspect code_str_list = [inspect.getsource(c) for c in code_list] with open(path, 'w') as ofh: for c in code_str_list: fh = io.StringIO(c) ofh.writelines(fh.readlines()) ofh.write('\n') create_file('my_value') 

The output I want is (file /tmp/code.py ):

 def concrete_func(b, k): return b + k + 'my_value' 

The output I get (file '/tmp/code.py' ):

 def concrete_func(b, k): return b + k + a_value 

UPDATE: my solution uses inspect.getsource , which returns a string. I wonder if I have limited your options since most of the solutions below offer a string replacement. There is no need to use inspect.getsource for the solution. You could write it anyway to get the desired result.

UPDATE 2: The reason I am doing this is because I want to create a file for Amazon Lambda. Amazon Lambda takes the python file and its virtual environment and runs it for you (freeing you from worries about scalability and resiliency). You must tell Lambda which file and which function to call, and Lambda will execute it for you.

Answer 1

Use getsource to convert the function to a string and replace the variable names with simple string manipulations.

 from inspect import getsource def write_func(fn, path, **kwargs): fn_as_string = getsource(fn) for var in kwargs: fn_as_string = fn_as_string.replace(var, kwargs[var]) with open(path, 'a') as fp: # append to file fp.write('\n' + fn_as_string) def base_func(b, k): return b + k + VALUE # add quotes to string literals write_func(base_func, '/tmp/code.py', VALUE="'my value'") # you should replace the function name if you write multiple functions to the file write_func(base_func, '/tmp/code.py', base_func='another_func', VALUE='5') 

Output is expected in /tmp/code.py:

 def base_func(b, k): return b + k + 'my value' def another_func(b, k): return b + k + 5 

Answer 2

A function definition does not search for its free variables (variables that are not defined in the function itself) during the definition. That is concrete_func here:

 def make_concrete_func(a_value): def concrete_func(b, k): return b + k + a_value return concrete_func 

does not look up a_value when it is defined, instead it will contain code to load a_value from its closure (simplified closing function) at runtime.

This can be seen by parsing the returned function:

 f = make_concrete_func(42) import dis print dis.dis(f) 3 0 LOAD_FAST 0 (b) 3 LOAD_FAST 1 (k) 6 BINARY_ADD 7 LOAD_DEREF 0 (a_value) 10 BINARY_ADD 11 RETURN_VALUE None 

Perhaps you can do what you want by editing the byte code. This has been done before ( http://bytecodehacks.sourceforge.net/bch-docs/bch/module-bytecodehacks.macro.html ..shudder).

Answer 3

Try it. Note that I added another parameter to write_to_file

 def write_to_file(code_list, path,a_value): print "lc",code_list code_str_list = [inspect.getsource(c) for c in code_list] with open(path, 'w') as ofh: for c in code_str_list: c= c.replace('a_value','\''+a_value+'\'') fh = io.StringIO(c) ofh.writelines(fh.readlines()) ofh.write('\n') 

Answer 4

If the file should not be human readable and you believe that attackers will not manipulate them, combining functools.partial and pickle may be the most pythonic approach. However, it has drawbacks that I do not quite understand: on the one hand, it does not work with locally defined functions (or, possibly, locally defined variables in general?).

I can just ask my own question.

 import functools import pickle def write_file_not_working(): def concrete_func_not_working(b, k, a_value): return b + k + a_value with open('data.pickle', 'wb') as f: data = functools.partial(concrete_func_not_working, a_value='my_value') pickle.dump(data, f, pickle.HIGHEST_PROTOCOL) def use_file_not_working(): with open('data.pickle', 'rb') as f: resurrected_data = pickle.load(f) print(resurrected_data('hi', 'there')) def top_level_concrete_func(b, k, a_value): return a_value + b + k def write_file_working(): with open('working.pickle', 'wb') as f: data = functools.partial(top_level_concrete_func, a_value='my_working_value') pickle.dump(data, f, pickle.HIGHEST_PROTOCOL) def use_file_working(): with open('working.pickle', 'rb') as f: resurrected_data = pickle.load(f) print(resurrected_data('hi', 'there')) if __name__ == "__main__": write_file_working() use_file_working() write_file_not_working() use_file_not_working() 

Answer 5

@Ben made me realize that I didn't need to use a line-based approach to generate code, and that I could use serialization. Instead of a limited pickle library, I used a dill that overcomes the constraint mentioned by Ben

So, I finally did something like.

 import dill def create_file(a_value, path): a_func = make_concrete_func(a_value) dill.dump(a_func, open(path, "wb")) return path def make_concrete_func(a_value): def concrete_func(b, k): return b + k + a_value return concrete_func if __name__ == '__main__': path = '/tmp/code.dill' create_file('Ben', path) a_func = dill.load(open(path, "rb")) print(a_func('Thank ', 'You ')) 

Answer 6

if the function you want to create has a specific template, I would create a template for it and use it to create functions in bulk

 >>> def test(*values): template=""" def {name}(b,k): return b + k + {value} """ for i,v in enumerate(values): print( template.format(name="func{}".format(i),value=repr(v)) ) >>> test("my_value",42,[1]) def func0(b,k): return b + k + 'my_value' def func1(b,k): return b + k + 42 def func2(b,k): return b + k + [1] >>>