How to convert a generator or iterator to a list recursively

Question

I want to convert a generator or iterator to a list recursively.
I wrote the code below, but it looks naive and ugly and can be removed in doctest.

Q1. Help me in a good version.
Q2. How to indicate an object is immutable or not?

import itertools def isiterable(datum): return hasattr(datum, '__iter__') def issubscriptable(datum): return hasattr(datum, "__getitem__") def eagerlize(obj): """ Convert generator or iterator to list recursively. return a eagalized object of given obj. This works but, whether it return a new object, break given one. test 1.0 iterator >>> q = itertools.permutations('AB', 2) >>> eagerlize(q) [('A', 'B'), ('B', 'A')] >>> test 2.0 generator in list >>> q = [(2**x for x in range(3))] >>> eagerlize(q) [[1, 2, 4]] >>> test 2.1 generator in tuple >>> q = ((2**x for x in range(3)),) >>> eagerlize(q) ([1, 2, 4],) >>> test 2.2 generator in tuple in generator >>> q = (((x, (y for y in range(x, x+1))) for x in range(3)),) >>> eagerlize(q) ([(0, [0]), (1, [1]), (2, [2])],) >>> test 3.0 complex test >>> def test(r): ... for x in range(3): ... r.update({'k%s'%x:x}) ... yield (n for n in range(1)) >>> >>> def creator(): ... r = {} ... t = test(r) ... return r, t >>> >>> a, b = creator() >>> q = {'b' : a, 'a' : b} >>> eagerlize(q) {'a': [[0], [0], [0]], 'b': {'k2': 2, 'k1': 1, 'k0': 0}} >>> test 3.1 complex test (other dict order) >>> a, b = creator() >>> q = {'b' : b, 'a' : a} >>> eagerlize(q) {'a': {'k2': 2, 'k1': 1, 'k0': 0}, 'b': [[0], [0], [0]]} >>> test 4.0 complex test with tuple >>> a, b = creator() >>> q = {'b' : (b, 10), 'a' : (a, 10)} >>> eagerlize(q) {'a': ({'k2': 2, 'k1': 1, 'k0': 0}, 10), 'b': ([[0], [0], [0]], 10)} >>> test 4.1 complex test with tuple (other dict order) >>> a, b = creator() >>> q = {'b' : (b, 10), 'a' : (a, 10)} >>> eagerlize(q) {'a': ({'k2': 2, 'k1': 1, 'k0': 0}, 10), 'b': ([[0], [0], [0]], 10)} >>> """ def loop(obj): if isiterable(obj): for k, v in obj.iteritems() if isinstance(obj, dict) \ else enumerate(obj): if isinstance(v, tuple): # immutable and iterable object must be recreate, # but realy only tuple? obj[k] = tuple(eagerlize(list(obj[k]))) elif issubscriptable(v): loop(v) elif isiterable(v): obj[k] = list(v) loop(obj[k]) b = [obj] loop(b) return b[0] def _test(): import doctest doctest.testmod() if __name__=="__main__": _test() 

Answer 1

To avoid a bad effect on the original object, you basically need the copy.deepcopy option ... subtly modified because you need to include generators and iterators in lists (in any case, a deep copy will not generate deep copy generators). Please note that some influence on the original object, unfortunately, is inevitable, because generators and iterators are "exhausted" as a side effect of iterations all the way (whether turning them into lists or for any other purposes), therefore it is simply impossible, so that you both leave the original object alone and that this generator or other iterator turns into a list as a result of the "deep-copy" option.

Unfortunately, the copy module cannot be customized, so the alternative ares, either copy-paste-edit, or a thin (sigh) monkey patch, depending on the (double-sigh) private variable of the _deepcopy_dispatch module (which means that your the corrected version may not be preserved when updating the version in Python, say, from 2.6 to 2.7, hypothetically). In addition, the monkey patch must be removed after each use of your eagerize (so as not to affect other deepcopy applications). So, suppose that instead we choose the path for copy-paste.

Let's say we start with the latest version, the one online here . Of course you need to rename the module; rename the externally visible deepcopy function to eagerize on line 145; a significant change occurs on lines 161-165, which are annotated in the specified version:

 161 : copier = _deepcopy_dispatch.get(cls) 162 : if copier: 163 : y = copier(x, memo) 164 : else: 165 : tim_one 18729 try: 

We need to insert between line 163 and 164 the logic “otherwise, if it is iterable, it will expand it to a list (that is, it uses the _deepcopy_list function as a copier”. Thus, these lines become:

 161 : copier = _deepcopy_dispatch.get(cls) 162 : if copier: 163 : y = copier(x, memo) elif hasattr(cls, '__iter__'): y = _deepcopy_list(x, memo) 164 : else: 165 : tim_one 18729 try: 

That's all: only two lines added. Please note that I left only the original line numbers in order to clearly understand where exactly these two lines need to be inserted, and not to number two new lines. You also need to rename other instances of the deepcopy identifier (indirect recursive calls) to eagerize .

You should also delete lines 66-144 (small copy functionality that you do not need) and configure lines 1-65 correctly (docstrings, import, __all__ , etc.).

Of course, you want to work out a copy of the text version of copy.py , here , and not the annotated version that I have (I used the annotated version to clarify exactly where the changes were needed!).