Gracefully degrading etching in Python

Question

(You can read this question for some background)

I would like to have a graceful and humiliating way to sort objects in Python.

When etching an object, let it be called its main object; sometimes Pickler raises an exception because it cannot determine any subobject of the main object. For example, the error that I get a lot is “incapable brine module objects”. This is because I am referring to a module from the main object.

I know that I can write something to replace this module with a facade that will contain the attributes of the module, but it will have its own problems (1).

So, what I would like is an etching function that automatically replaces modules (and any other hard-to-reach objects) with facades that contain their attributes. This may not provide perfect etching, but in many cases this would be sufficient.

Is there anything similar? Does anyone have an idea how to approach this?

---

(1) One problem is that a module can reference other modules from the inside.

Answer 1

You can decide and implement how any previously unsealed type is pickled and not printed: see the standard library module copy_reg (renamed copyreg in Python 3. *).

Essentially, you need to provide a function that, given an instance of the type, reduces it to a tuple - with the same protocol as it is reduced (except that the special reduction method does not accept any arguments, provided that it is called directly on object, and the function you provide will take the object as the only argument).

As a rule, the returned tuple has 2 elements: the called tuple and the argument tuple for passing it. The caller must be registered as a "safe constructor" or equivalent to the __safe_for_unpickling__ attribute with a true value. These elements will be pickled, and at different times the called one will be called with the given arguments and should return an unpublished object.

For example, suppose you just want to sort the modules by name, so that multiplying them simply means re-importing them (i.e. for simplicity you don't need dynamically modified modules, nested packages, etc., just top-level modules). Then:

 >>> import sys, pickle, copy_reg >>> def savemodule(module): ... return __import__, (module.__name__,) ... >>> copy_reg.pickle(type(sys), savemodule) >>> s = pickle.dumps(sys) >>> s "c__builtin__\n__import__\np0\n(S'sys'\np1\ntp2\nRp3\n." >>> z = pickle.loads(s) >>> z <module 'sys' (built-in)> 

I use the old-fashioned ASCII brine form, so s , the string containing the brine, is easy to examine: it instructs to scatter a call to the built-in import function with the string sys as its only argument. And z shows that it really returns us the sys built-in module as a result of scattering at will.

Now you need to make things a little more complicated than just __import__ (you have to deal with saving and restoring dynamic changes, moving the nested namespace, etc.), and therefore you will also have to call copy_reg.constructor (passing as an argument your own function that does this job) before the copy_reg module save function, which returns your other function (and, if in a separate run, also before you decompose these pickles that you made using the specified function). But I hope that these simple cases help show that in fact there is nothing special, that "purely" is "difficult"!)

Answer 2

How about the next one, which is a shell that you can use to transfer some modules (possibly any module) into something that is pickled. You can then subclass the Pickler object to check if the target object is a module, and if so, wrap it. Does this do what you desire?

 class PickleableModuleWrapper(object): def __init__(self, module): # make a copy of the module namespace in this instance self.__dict__ = dict(module.__dict__) # remove anything that going to give us trouble during pickling self.remove_unpickleable_attributes() def remove_unpickleable_attributes(self): for name, value in self.__dict__.items(): try: pickle.dumps(value) except Exception: del self.__dict__[name] import pickle p = pickle.dumps(PickleableModuleWrapper(pickle)) wrapped_mod = pickle.loads(p) 

Answer 3

Hmmm, something like this?

 import sys attribList = dir(someobject) for attrib in attribList: if(type(attrib) == type(sys)): #is a module #put in a facade, either recursively list the module and do the same thing, or just put in something like str('modulename_module') else: #proceed with normal pickle 

Obviously, this would lead to an extension of the pickle class with an overridden dump method ...