Why does Python @staticmethods interact so badly with decorated classes?

Question

Recently, the StackOverflow community has helped me develop a rather concise @memoize decorator that can decorate not only functions, but also methods and classes in a general way, i.e. without any foresight of decorating.

One of the problems I ran into is that if you decorated a class with @memoize and then tried to decorate one of its methods with @staticmethod , it would not work as expected, i.e. you would not be able to call ClassName.thestaticmethod() at all. The original solution that I came up with looked like this:

 def memoize(obj): """General-purpose cache for classes, methods, and functions.""" cache = obj.cache = {} def memoizer(*args, **kwargs): """Do cache lookups and populate the cache in the case of misses.""" key = args[0] if len(args) is 1 else args if key not in cache: cache[key] = obj(*args, **kwargs) return cache[key] # Make the memoizer func masquerade as the object we are memoizing. # This makes class attributes and static methods behave as expected. for k, v in obj.__dict__.items(): memoizer.__dict__[k] = v.__func__ if type(v) is staticmethod else v return memoizer 

But then I found out about functools.wraps , which is designed to make the decorator function masquerade as a decorated function in a much cleaner and more complete way, and I really accepted it like this:

 def memoize(obj): """General-purpose cache for class instantiations, methods, and functions.""" cache = obj.cache = {} @functools.wraps(obj) def memoizer(*args, **kwargs): """Do cache lookups and populate the cache in the case of misses.""" key = args[0] if len(args) is 1 else args if key not in cache: cache[key] = obj(*args, **kwargs) return cache[key] return memoizer 

Although this looks very good, functools.wraps provides absolutely no support for either staticmethod or classmethod s. For example, if you tried something like this:

 @memoize class Flub: def __init__(self, foo): """It is an error to have more than one instance per foo.""" self.foo = foo @staticmethod def do_for_all(): """Have some effect on all instances of Flub.""" for flub in Flub.cache.values(): print flub.foo Flub('alpha') is Flub('alpha') #=> True Flub('beta') is Flub('beta') #=> True Flub.do_for_all() #=> 'alpha' # 'beta' 

This will work with the first implementation of @memoize , which I have listed, but will raise a TypeError: 'staticmethod' object is not callable with the second.

I really wanted to solve this simply by using functools.wraps without returning this __dict__ ugliness, so I actually redefined my own staticmethod in pure Python, which looked like this:

 class staticmethod(object): """Make @staticmethods play nice with @memoize.""" def __init__(self, func): self.func = func def __call__(self, *args, **kwargs): """Provide the expected behavior inside memoized classes.""" return self.func(*args, **kwargs) def __get__(self, obj, objtype=None): """Re-implement the standard behavior for non-memoized classes.""" return self.func 

And this, as far as I can tell, works great next to the second @memoize implementation, which I list above.

So my question is: why does the standard built-in staticmethod behave correctly as it sees fit and / or why doesn't functools.wraps expect this situation and solve it for me?

Is this a bug in Python? Or in functools.wraps ?

What are the warnings about overriding the built-in staticmethod ? As I said, now it works fine, but I'm afraid there might be some hidden incompatibility between my implementation and the embedded implementation, which may explode later.

Thanks.

Edit to clarify: in my application, I have a function that performs an expensive search and is often called, so I remembered it. It is pretty simple. In addition to this, I have several classes that represent files, and having multiple instances representing the same file in the file system usually results in an inconsistent state, so it is important to force only one instance for the file name. In essence, it’s trivial to adapt the @memoize decorator for this purpose and preserve functionality as a traditional memoizer.

Real world examples in three different @memoize applications @memoize given here:

• In function
• By method
• In a class containing staticmethods

Answer 1

A few considerations for you:

• The operation of the static method is completely orthogonal to the class decorator operator. Creating a function in staticmethod only affects what happens when searching for attributes. A class decorator is compile-time conversion in a class.

• There is no "error" in functools.wraps. All he does is copy function attributes from one function to another.

• As currently written, your memoize tool does not account for different call signatures for class methods and staticmethods. This weakness in memoize is not in the class tools themselves.

I think you introduced tools like class decorators, static methods, class methods, and functools to have some kind of interdependent intelligence. Instead, all of these tools are very simple and require that the programmer consciously develop their interactions.

ISTM consists in the fact that the stated goal is somewhat unproven: "a decorator who is able to decorate not only functions, but also methods and classes in a general way, that is, without any foreseeing of being a decoration "

It is not clear what semantics memoize will be in each scenario. And there is no way for simple Python components to automatically compose themselves in such a way that you can guess what you really wanted to do.

My recommendation is that you start with a list of developed memoize examples using various objects. Then start building your current solution to get them working one at a time. At each step you will find out where your specification does not match what meoize actually does.

Another thought is that functools.wraps and class decorators are not absolutely necessary for this problem. Both can be implemented manually. Start by wiring your tool to do what you want. After it works, look at replacing steps with wrappers and decorators. It hits, trying to force the tools to your will in situations where they may not be suitable.

Hope this helps.

Answer 2

A class decoration is used to potentially alter the construction of a class. It is kind of convenient, but not quite the same as __new__ .

 # Make the memoizer func masquerade as the object we are memoizing. # This makes class attributes and static methods behave as expected. for k, v in obj.__dict__.items(): memoizer.__dict__[k] = v.__func__ if type(v) is staticmethod else v return memoizer 

The above code forces your shell to use the methods inside your instance.

 class Flub: @memoize @staticmethod def do_things(): print 'Do some things.' Flub.do_things() 

I believe that this should be the code that you should use - remember that if you do not get any arguments, then args [0] will go to IndexError

Answer 3

The problem is that your decorator takes a class (i.e. an instance of type ) and returns a function. This is (to a large extent) the software equivalent of error; classes may look like functions in that they can be called (like a constructor), but that does not mean that the function that returns the instance is equivalent to a class of type of that instance. For example, there is no way for instanceof give the correct result, and in addition, your decorated class can no longer be a subclass (because it is no longer a class!)

What you need to do is adapt your decorator to determine when it is called in the class, and in this case build a wrapper class (using the class syntax or the type 3 argument constructor) that has the required behavior. Either this, or memoize __new__ (although keep in mind that __init__ will be called in the return value of __new__ , if it is of the corresponding type, even if it is an existing instance).