What is the best way to share the value of all the generators created by the function?

Question

Here I asked a question about izip_longest from itertools .

His code is:

 def izip_longest_from_docs(*args, **kwds): # izip_longest('ABCD', 'xy', fillvalue='-') --> Ax By C- D- fillvalue = kwds.get('fillvalue') def sentinel(counter = ([fillvalue]*(len(args)-1)).pop): yield counter() # yields the fillvalue, or raises IndexError fillers = repeat(fillvalue) iters = [chain(it, sentinel(), fillers) for it in args] try: for tup in izip(*iters): yield tup except IndexError: pass 

An error has appeared in the documentation in the pure Python equivalent of this function. The error was that the real function was IndexError , and the aforementioned equivalent did not IndexError exceptions that were raised inside the generators sent as function parameters.

@agf solved the problem and gave a patched version of the pure Python equivalent.

But at the same time, when he wrote his decision, I made my own. And at the same time, I ran into one problem, which, I hope, will be dissolved by asking this question.

The code I came up with is this:

 def izip_longest_modified_my(*args, **kwds): # izip_longest('ABCD', 'xy', fillvalue='-') --> Ax By C- D- fillvalue = kwds.get('fillvalue') class LongestExhausted(Exception): pass def sentinel(fillvalue = fillvalue, counter = [0]): def ret(): counter[0] += 1 if counter[0] == len(args): raise LongestExhausted yield fillvalue return ret() fillers = repeat(fillvalue) iters = [chain(it, sentinel(), fillers) for it in args] try: for tup in izip(*iters): yield tup except LongestExhausted: pass 

There is a generator in the sentinel source code that performs a lazy evaluation. So counter() returned only when the iterator created using the chain function really needs it.

In my code, I added counter , which contains a list of a single value [0] . The reason for this was to place the mutable object in a place where all returned ret() iterators can be accessed and modified. The only place I found suitable was in function_defaults of sentinel .

If I put it inside the sentinel function, then counter will be assigned [0] each time sentinel called, and these will be different lists for all ret() s:

 def sentinel(fillvalue = fillvalue): counter = [0] def ret(): counter[0] += 1 if counter[0] == len(args): raise LongestExhausted yield fillvalue return ret() 

I tried setting it outside the sentinel function:

 counter = 0 def sentinel(fillvalue = fillvalue): def ret(): counter += 1 if counter == len(args): raise LongestExhausted yield fillvalue return ret() 

But the exception has grown: UnboundLocalError: local variable 'counter' referenced before assignment .

I added the global , but this did not help (I think because counter really not in the global scope):

 counter = 0 def sentinel(fillvalue = fillvalue): global counter def ret(): counter += 1 if counter == len(args): raise LongestExhausted yield fillvalue return ret() 

So my question is :

Is the approach I used (to put mutable list counter = [0] in function_defaults ) the best in this case, or is there a better way to solve this problem?

Answer 1

This has been asked many times in many forms. Read any number of other questions about mutable default arguments and the new Python 3 nonlocal . On Python 2, you can use the attribute:

 def sentinel(fillvalue = fillvalue): def ret(): sentinel.counter += 1 if sentinel.counter == len(args): raise LongestExhausted yield fillvalue return ret() sentinel.counter = 0 

or use global inside ret and inside izip_longest , so you always refer to a global variable:

 global counter counter = 0 def sentinel(fillvalue = fillvalue): def ret(): global counter counter += 1 if counter == len(args): raise LongestExhausted yield fillvalue return ret() 

However, using global limits you to only one izip_longest at a time - see comments for another answer.

You also define a new ret each time sentinel is called (once per iterator) - instead, you can do something like

 global counter counter = 0 arglen = len(args) def ret(): global counter counter += 1 if counter == arglen: raise LongestExhausted return fillvalue def sentinel(): yield ret() 

Sample code for sentinel outside izip_longest in your question from comments:

 def sentinel(counter, arglen, fillvalue): def ret(): counter[0] += 1 if counter[0] == arglen: raise LongestExhausted yield fillvalue return ret() def izip_longest_modified_my(*args, **kwds): # izip_longest('ABCD', 'xy', fillvalue='-') --> Ax By C- D- fillvalue = kwds.get('fillvalue') class LongestExhausted(Exception): pass fillers = repeat(fillvalue) counter = [0] arglen = len(args) iters = [chain(it, sentinel(counter, arglen, fillvalue), fillers) for it in args] try: for tup in izip(*iters): yield tup except LongestExhausted: pass 

Here, you again use the list as a container to get around issues related to external areas in Python 2.

Answer 2

Using global is a bad idea, IMHO. You need to make sure that the reset counter is correct between calls. But more seriously, it is a generator; you don’t even need streaming to have several calls to the generator in flight at the same time, which will destroy chaos with any attempt to intelligently use the global mode to track the state.

You can simply explicitly pass the reference to the mutable object to the sentinel and then to ret. It looks like your code controls all the calls. Function parameters are an original and boring way to transfer links between areas!