Switching context with "exit"

Question

I read the gevent tutorial and saw this interesting snippet:

import gevent def foo(): print('Running in foo') gevent.sleep(0) print('Explicit context switch to foo again') def bar(): print('Explicit context to bar') gevent.sleep(0) print('Implicit context switch back to bar') gevent.joinall([ gevent.spawn(foo), gevent.spawn(bar), ]) 

In which the flow of execution looks like this foo → bar → foo → bar. Impossible to do the same without the gevent module, but with yield operations? I tried to do this with yield, but for some reason I can't get it to work ... :(

Answer 1

The generators used for this purpose are often called tasks (among many other terms), and I will use this term for clarity. Yes it is possible. There are actually several approaches that work and make sense in some contexts. However, none (that I know) work without an equivalent for at least one of gevent.spawn and gevent.joinall . For more powerful and well-designed, an equivalent is required for both.

The main problem is this: generators can be paused (when they press yield ), but what is it. To turn them off again, you need another code that calls next() on them. In fact, you even need to call next() newly created generator so that it starts to start somewhere. Similarly, the generator itself is not the best place to decide what to do next. So, you need a cycle that initiates each time slice of tasks (starts them until the next yield ) and switches between them an unlimited time. This is commonly called a scheduler. They tend to get really hairy very quickly, so I will not try to write a complete planner in one answer. However, there are some basic concepts that I can try to explain:

• Usually, yield used as control back to the sheduler (actually similar to gevent.sleep(0) in your code). This means that the generator does everything that it wants to do, and when it is in the place where the context switch is convenient and possibly useful, it yield s.
• In Python 3.3+, yield from is a very useful tool for delegating to another generator. If you cannot use it, you must force the scheduler to emulate the call stack and return the route values ​​to the right place and do something like result = yield subtasks() in your tasks. This is slower, harder to implement, and is unlikely to provide useful stack traces ( yield from does this for free). But until recently, it was the best we had.
• Depending on your use case, you may need a wide range of task management tools. In ordinary examples, new tasks arise, waiting for the completion of the task, waiting for the completion of any of several tasks, failure detection (uncaught exception) of other tasks, etc. Usually they are handled by the scheduler, and tasks are provided with an API to communicate with the scheduler. The optimal way (but not always ideal) for this message is yield special values.
• One rather important difference between generator tasks and gevent (and similar libraries) is that context switches in the latter are implicit, and tasks are trivial for defining context switches: only those things that yield [from] can run the scheduler code, For example, you can make sure that the piece of code is atomic (wrt other tasks, if you add streams to the mix, you should worry about them independently) by simply looking at the code without checking what it calls.

Finally, you might be interested in the Greg Ewing tutorial on creating such a scheduler. (This triggered python-ideas while brainstorming about what is now PEP 3156. These mail streams may also be of interest to you, although the web archive is not suitable for reading hundreds of letters in dozens of threads written half a year ago. )

Answer 2

The key is to realize that you will need to provide your own driving cycle - I have provided a simple demo below. I was lazy and used the Queue object to provide FIFO, I did not use python for a significant project for a while.

 #!/usr/bin/python import Queue def foo(): print('Constructing foo') yield print('Running in foo') yield print('Explicit context switch to foo again') def bar(): print('Constructing bar') yield print('Explicit context to bar') yield print('Implicit context switch back to bar') def trampoline(taskq): while not taskq.empty(): task = taskq.get() try: task.next() taskq.put(task) except StopIteration: pass tasks = Queue.Queue() tasks.put(foo()) tasks.put(bar()) trampoline(tasks) print('Finished') 

And at startup:

 $ ./coroutines.py Constructing foo Constructing bar Running in foo Explicit context to bar Explicit context switch to foo again Implicit context switch back to bar Finished