What is the design reason for if __new__ does not return an instance of cls, python does not call __init__?

Question

There are many questions in SO why python does not always call __init__ after creating an object. The answer, of course, is contained in this excerpt from the documentation:

If __new__() returns an instance of cls , then the new instance of __init__() will be called as __init__(self[, ...]) , where self is the new instance and the rest of the arguments are the same as those passed to __new__() .

If __new__() does not return an instance of cls , then the new __init__() method will not be called.

What is the design reason for this?

Answer 1

__init__ acts as a constructor. He needs an instance to do his job, for example, set attributes, etc. If __new__ does not return an explicitly returned instance, then None returned by default.

Imagine what happens when __init__ gets None as input and tries to set attributes? This will throw an exception called "AttributeError: 'NoneType' object has no attribute xxxxx".

Therefore, I find it natural that you do not call __init__ when __new__ returns None.

Answer 2

In Python 2, you cannot call a regular method with the first argument being something other than an instance of the class (or subclass):

 class Foo(object): def __init__(self): pass Foo.__init__() # TypeError: unbound method __init__() must be called with Foo instance as first argument (got nothing instead) Foo.__init__(3) # TypeError: unbound method __init__() must be called with Foo instance as first argument (got int instance instead) 

So __init__ not called because it cannot do anything but throw an exception right away. Do not try to call it more useful (although I don't think I have ever seen code use this).

Python 3 has a slightly simpler implementation of the method, and this restriction no longer applies, but the semantics of __new__ same. In any case, it makes no sense to try to run the class initializer on a foreign object.

---

For a more constructive answer, rather than “because it is,” answer:

Overriding __new__ is already a strange thing. By default, it returns an uninitialized object, which is a concept that Python is trying to hide. If you override it, you are likely to do something like this:

 class Foo(object): def __new__(cls, some_arg): if some_arg == 15: # 15 is a magic number for some reason! return Bar() else: return super(Foo, cls).__new__(cls, some_arg) 

Imagine a Python variant that is unconditionally called __init__ by return value. I immediately see a number of problems.

When you return Bar() , should Python call Bar.__init__ (which was already called in this case) or Foo.__init__ (which for a completely different type will break any Bar guarantees)?

The answer should probably be called Bar.__init__ . Does this mean that you need to return an uninitialized instance of Bar , using the return Bar.__new__(Bar) instead? Python very rarely requires you to call dunder methods outside of using super , so that would be very unusual.

Where would Bar.__init__ arguments Bar.__init__ ? Both Foo.__new__ and Foo.__init__ are passed with the same arguments - those that are passed to type.__call__ , which Foo(...) processes. But if you explicitly call Bar.__new__ , you should not remember the arguments that you wanted to pass to Bar.__init__ . You cannot store them on the new Bar object, because that is what Bar.__init__ should do! And if you just said that it receives the same arguments that were passed to Foo , you strictly limit what types can be returned from __new__ .

Or, what if you want to return an object that already exists? Python has no way to indicate that the object is already “initialized” - since uninitialized objects are a temporary and mostly internal thing of interest to __new__ , so you would have no way not to call __init__ again.

The current approach is a bit awkward, but I don't think there is a better alternative. __new__ should create space to store a new object, and returning another type of object is just a very strange thing; this is the least surprising and most useful way that Python can handle this.

If this restriction is bothering you, remember that the whole dance of __new__ and __init__ is just the behavior of type.__call__ . You can freely define your __call__ behavior in the metaclass or just change the class for the factory function.

Answer 3

This is probably not the reason for the design, but a great result if this design decision is that the class should not return an instance of itself:

 class FactoryClass(): def __new__(klass, *args, **kwargs): if args or kwargs: return OtherClass(*args, **kwargs) return super().__new__(klass) 

It can be useful sometimes. Obviously, if you are returning some other class object, you do not want the init method of the factory class to be called.

The real reason, which, I hope, illustrates the above example, and, apparently, any other design decision would be pointless. New is a constructor for Python class objects, not init. Init is no different from any other class method (except that it is automatically called by magic after the object has been created); regardless of whether they are called from them, depends on what happens in the constructor. This is just a natural way to do something. Any other way will be broken.

Answer 4

If __new__ does not return an instance of cls , then passing the return value to cls.__init__ can lead to very bad things.

In particular, Python does not require you to return an instance of the same type of class.

Take this contrived example:

 In [1]: class SillyInt(int): ...: def __new__(cls): ...: return "hello" ...: In [2]: si = SillyInt() In [3]: si Out[3]: 'hello' 

So, we created a subclass of int , but our __new__ method returned a str object.

It would not make much sense to pass this line to our inherited int.__init__ for instantiation. Even if it “works” in the particular case (in the sense that it does not throw any errors), we could create an object in an inconsistent state.

---

Well, what about a more specific (albeit still far-fetched) example?

Suppose we create two classes that set the same attribute as part of their initialization (in the __init__ method):

 In [1]: class Tree(): ...: def __init__(self): ...: self.desc = "Tree" ...: In [2]: class Branch(): ...: def __init__(self): ...: self.desc = "Branch" ...: 

If we now create a subclass that in the user method __new__ wants to return a different type of object (which can sometimes make sense in practice, even if it is not here!):

 In [3]: class SillyTree(Tree): ...: def __new__(cls): ...: return Branch() ...: 

When creating an instance of this subclass, we see that we have an object of the expected type and that the Branch initializer was the one that was called:

 In [4]: sillyTree = SillyTree() In [5]: sillyTree.desc Out[5]: 'Branch' 

What would happen if Python unconditionally named a inherited initializer?

Well, we can verify this by calling the initializer directly.

As a result, we get a Branch instance that was initialized as a Tree , leaving the object in a very unexpected state:

 In [6]: SillyTree.__init__(sillyTree) In [7]: sillyTree.desc Out[7]: 'Tree' In [8]: isinstance(sillyTree, Branch) Out[8]: True In [9]: isinstance(sillyTree, Tree) Out[9]: False