Python __reduce __ / copy_reg semantic and statefull unickler

Question

I want to implement etching support for objects that belong to my extension library. There is a global instance of the Service class initialized at startup. All these objects are created as a result of some calls to the service method and, in essence, belong to it. The service knows how to serialize them in binary buffers and how to deserialize buffers back to objects.

It seems that Pythons __ reduce__ should serve my purpose - to implement etching support. I started to implement one and realized that there was a problem with unpickler (the first element is an od tuple, which is expected to be returned __ reduce__). This unickick function needs an instance of the service in order to be able to convert the input buffer to an object. Here are some pseudo-code to illustrate the problem:

class Service(object): ... def pickleObject(self,obj): # do serialization here and return buffer ... def unpickleObject(self,buffer): # do deserialization here and return new Object ... class Object(object): ... def __reduce__(self): return self.service().unpickleObject, (self.service().pickleObject(self),) 

Pay attention to the first element in the tuple. He doesn’t like the Python pickler: he says that this is an example of a method and cannot be pickled. Obviously, pickler is trying to save the procedure in the output file and wants the service instance along with the function name, but this does not mean that I want to happen. I do not want (and really can not: Service is not selected) to store the service together with all objects. I want the service instance to be created before calling pickle.load and somehow this instance is used during the spill.

Here, where I came with the copy_reg module. Again it turned out that he should solve my problems. This module allows you to register pickler and unpickler manipulators for each type dynamically, and they should be used later for objects of this type. Therefore, I added this registration to the Service design:

 class Service(object): ... def __init__(self): ... import copy_reg copy_reg( mymodule.Object, self.pickleObject, self.unpickleObject ) 

self.unpickleObject is now a related method that takes a service as the first parameter and a buffer as the second. self.pickleObject also binds the method, taking care of the service and the object for the brine. copy_reg requires the pickleObject procedure to follow the semantics of the reducer and return the same tuple as before. And here the problem arose again: what should I return as the first element of the tuple?

 class Service(object): ... def pickleObject(self,obj): ... return self.unpickleObject, (self.serialize(obj),) 

In this form, the pickle again complains that it cannot pickle the instance method. I tried None - he doesn't like it either. I put some kind of dummy function there. This works - it means the serialization stage went fine, but during the scattering it calls this dummy function, not the unpickler, which I registered for the mymodule.Object type in the Service constructor.

So now I am at a loss. Sorry for the long explanation: I did not know how to ask this question in several lines. I can summarize my questions as follows:

• Why does copy_reg semantics require me to return a sloppy procedure from pickleObject if I have to register it myself?
• Is there any reason to prefer the copy_reg.constructor interface for registering a sloppy program?
• How to make a pickle to use the registered multiplier that I registered, instead of one inside the stream?
• What should I return as the first element in a tuple as the value of the pickleObject result? Is there a “right” meaning?
• Am I right about all this? Is there any other / simpler solution?

Thank you for your time.

Gennady

Answer 1

First of all, the copy_reg module copy_reg unlikely to help you here: this is, first of all, a way to add __reduce__ similar functions to classes that do not have this method and do not offer any special abilities (for example, if you want to sort objects from any library which does not support her).

The called, returned __reduce__ , must be localizable in an environment where the object should be unpainted, so the instance method is not suitable. As mentioned in the Pickle Documentation :

In an environment with an incomplete shell, this object must be either a class or a callable registered as a "safe constructor" (see below), or it must have the __safe_for_unpickling__ attribute with a true value.

So, if you defined a function (not a method) as follows:

 def _unpickle_service_object(buffer): # Grab the global service object, however that is accomplished service = get_global_service_object() return service.unpickleObject(buffer) _unpickle_service_object.__safe_for_unpickling__ = True 

Now you can use this _unpickle_service_object function in the return value of your __reduce__ methods __reduce__ that your objects are bound to the new environment by the global Service object when being scattered.