Replacing C ++ STL output iterator with Python generator

Question

Python has no built-in equivalent to OutputIterator; in particular, embedded or standard library containers do not support any common interface that allows client code to send data to them without knowing the specific type of container.

According to @Steven Rumbalski comment and @Glenn Maynard answer , this is usually not a problem, because a function that would accept the OutputIterator argument in C ++ would simply be written as a generator in python.

Usually I have no problem using generators and I never felt like I needed an OutputIterator in Python. However, in this case I am stuck.

I am reprogramming in Python some of the algorithms from the Boost Graph library. A typical graph traversal algorithm, such as depth_first_search , takes a visitor object as a parameter. A visitor is, in fact, a bunch of callback functions that the traversal algorithm calls when it encounters various events during its execution (for example, it detects a new vertex, examines an edge, etc.). In C ++, I have one or more of these callback functions that send data to the OutputIterator objects that the visitor object received when it was initialized from the client code. (For example, how exactly topological_sort is implemented: it receives an OutputIterator, passes it to the dfs_visitor object, the visitor then “tracks” the finished_vertex event and sends the vertices received to it to the specified OutputIterator. Of course, more complex cases require several OutputIterator objects and several callback functions.)

How to achieve the same result with Python generators?

I need to somehow send the data in the "style" of the generator, from depth_first_search to several designated data consumers. I just can't figure out how to do this. (I am using Python 3.3.)

Answer 1

Can you just pass the callback functions?

 def depth_first_search(some_args, on_edge=lambda e:None, on_vertex=lambda v:None): ... on_edge(some_edge) on_vertex(some_vertex) def edge_handler(e): print "E", e def vertex_handler(v): print "V", v depth_first_search(..., on_edge=edge_handler, on_vertex=vertex_handler) 

Or enter your destination:

 def depth_first_search(some_args, on_edge=lambda e:None, on_vertex=lambda v:None): ... yield "edge", some_edge yield "vertex", some_vertex for t, value in depth_first_search(...): if t == 'edge': # ... elif t == 'vertex': # ... 

Answer 2

I think the generator.send method does what you want here:

 def depth_first_search(some_args, edge_consumer, vertex_consumer): # start the generators next(edge_consumer) next(vertex_consumer) ... edge_consumer.send(some_edge) vertex_consumer.send(some_vertex) ... # this raises GeneratorExit at the 'yield' in the generator edge_consumer.close() vertex_consumer.close() def edge_handler(): while True: e = yield print "E", e def vertex_handler(): while True: v = yield print "V", v depth_first_search(..., edge_handler(), vertex_handler()) 

Answer 3

I don’t know if this will work, but what about using a library like multimethods to implement the multiple send function to do what you need? My Internet has problems with DNS, so I can’t find the syntax, so you will get pseudocode instead of real Python, but here is a general idea of ​​what I am saying:

 def send_to(data, consumer): workfunc = dispatch(type(consumer)) workfunc(data, consumer) def send_to_list(data, consumer): consumer.append(data) def send_to_set(data, consumer): consumer.add(data) def send_to_file_obj(data, consumer): consumer.write(data) 

Some plumbing is needed to, of course, connect the work functions to the send function, and this is something that I really can’t find right now, since my DNS is not working. (StackOverflow, fortunately, is still in my browser cache). Therefore, I am afraid that this answer is long in general terms and short in specifics, but I hope it will at least point you in a useful direction.