Reassigning a function attribute makes it "inaccessible"

Question

I have a simple little decorator that caches the results of function calls in a dict as a function attribute.

 from decorator import decorator def _dynamic_programming(f, *args, **kwargs): try: f.cache[args] except KeyError: f.cache[args] = f(*args, **kwargs) return f.cache[args] def dynamic_programming(f): f.cache = {} return decorator(_dynamic_programming, f) 

Now I want to add the ability to clear the cache. Therefore, I change the dynamic_programming() function as follows:

 def dynamic_programming(f): f.cache = {} def clear(): f.cache = {} f.clear = clear return decorator(_dynamic_programming, f) 

Now suppose I use this little thing to implement the Fibonacci number function:

 @dynamic_programming def fib(n): if n <= 1: return 1 else: return fib(n-1) + fib(n-2) >>> fib(4) 5 >>> fib.cache {(0,): 1, (1,): 1, (2,): 2, (3,): 3, (4,): 5} 

But now, when I clear the cache, something strange is happening:

 >>> fib.clear() >>> fib.cache {(0,): 1, (1,): 1, (2,): 2, (3,): 3, (4,): 5} 

Or (with the Python kernel running) do the opposite:

 >>> fib.clear() >>> fib(4) 5 >>> fib.cache {} 

Why is the cache somehow inaccessible after the first access to it, that is, does not change when clear() called after a call, or after clear() called?

(By the way, I know a solution for clearing the cache correctly: calling f.cache.clear() instead of assigning {} works for it as expected. I'm just interested in the reason for the failure of the assignment.)

Answer 1

The problem is the decorator module. If you added some print statements to your decorator:

 from decorator import decorator def _dynamic_programming(f, *args, **kwargs): print "Inside decorator", id(f.cache) try: f.cache[args] except KeyError: f.cache[args] = f(*args, **kwargs) return f.cache[args] def dynamic_programming(f): f.cache = {} print "Original cache", id(f.cache) def clear(): f.cache = {} print "New cache", id(f.cache) f.clear = clear return decorator(_dynamic_programming, f) @dynamic_programming def fib(n): if n <= 1: return 1 else: return fib(n-1) + fib(n-2) print fib(4) print id(fib.cache) fib.clear() print id(fib.cache) print fib(10) print id(fib.cache) 

It outputs (skipped duplicate lines):

 Original cache 139877501744024 Inside decorator 139877501744024 5 139877501744024 New cache 139877501802208 139877501744024 Inside decorator 139877501802208 89 139877501744024 

As you can see, the cache inside the decorator changes according to the clear function. However, cache access to which from __main__ does not change. Printing cache outside and inside the decorator gives a sharper image (again, duplicates are missing):

 Inside decorator {} Inside decorator {(1,): 1} Inside decorator {(2,): 2, (0,): 1, (1,): 1} Inside decorator {(2,): 2, (0,): 1, (3,): 3, (1,): 1} 5 Outside {(2,): 2, (0,): 1, (3,): 3, (1,): 1, (4,): 5} Inside decorator {} Inside decorator {(1,): 1} Inside decorator {(2,): 2, (0,): 1, (1,): 1} Inside decorator {(2,): 2, (0,): 1, (3,): 3, (1,): 1} Inside decorator {(2,): 2, (0,): 1, (3,): 3, (1,): 1, (4,): 5} Inside decorator {(0,): 1, (1,): 1, (2,): 2, (3,): 3, (4,): 5, (5,): 8} Inside decorator {(0,): 1, (1,): 1, (2,): 2, (3,): 3, (4,): 5, (5,): 8, (6,): 13} Inside decorator {(0,): 1, (1,): 1, (2,): 2, (3,): 3, (4,): 5, (5,): 8, (6,): 13, (7,): 21} Inside decorator {(0,): 1, (1,): 1, (2,): 2, (8,): 34, (3,): 3, (4,): 5, (5,): 8, (6,): 13, (7,): 21} Inside decorator {(0,): 1, (1,): 1, (2,): 2, (8,): 34, (3,): 3, (9,): 55, (4,): 5, (5,): 8, (6,): 13, (7,): 21} 89 Outside {(2,): 2, (0,): 1, (3,): 3, (1,): 1, (4,): 5} 

As you can see, internal changes are not reflected externally. The problem is that there is a line inside the decorator module (inside the class that it used to create the decorator):

 self.dict = func.__dict__.copy() 

And then later :

 func.__dict__ = getattr(self, 'dict', {}) 

Basically, __dict__ outside is different from __dict__ inside. It means that:

• __dict__ copied (not specified) by the decorator
• When the cache changes, it changes the internal __dict__ , not the external __dict__
• Therefore, the cache used by _dynamic_programming is cleared, but you cannot see it from outside, since the __dict__ decorator still points to the old cache (as you can see above, inside the cache updated, but the external cache remains the same)

So, to summarize, this is a problem with the decorator module.

Answer 2

so @ matsjoyce's answer is very interesting and in-depth, and I know you already have a solution, but I always find it clearer to write my own decorators:

 def dynamic_programming(f): def wrapper(*args, **kwargs): try: return wrapper.cache[args] except KeyError: res = wrapper.cache[args] = f(*args, **kwargs) return res wrapper.cache = {} wrapper.clear = wrapper.cache.clear return wrapper