How to handle multiple results from a coroutine function?

Question

I have some generators that do some searching things, and I use another generator to complete them:

def searching_stuff_1(): # searching yield 1 # and searching yield 2 yield 3 def searching_stuff_2(): yield 4 yield 5 def gen(): yield from searching_stuff_1() yield from searching_stuff_2() for result in gen(): print(result) 

So now I am wondering how I can rewrite it into an asynchronous version that can give multiple values ​​in search_stuff_1 and search_stuff_2.

I've tried:

 import asyncio async def searching_stuff_1(): f = asyncio.Future() result = [] await asyncio.sleep(1) #searching result.append(1) #searching result.append(2) result.append(3) f.set_result(result) return f async def searching_stuff_2(): f = asyncio.Future() result = [] await asyncio.sleep(1) result.append(4) result.append(5) f.set_result(result) return f async def producer(): coros = [searching_stuff_1(), searching_stuff_2()] for future in asyncio.as_completed(coros): yield await future async def consumer(xs): async for future in xs: for r in future.result(): print(r) loop = asyncio.get_event_loop() loop.run_until_complete(consumer(producer())) loop.close() 

However, in this version, I have to add all the results to the list and transfer it to the Future instance. I am wondering if there is a better way to handle multiple results from the coroutine function. Is it possible that I can still get these numbers?

Answer 1

Yes, you can still get these numbers in the asynchronous version, these are Asynchronous generators , you can use async for (PEP492) and asynchronous messages (PEP530) , like this, rewrite from your first example. although this requires a python version higher or equal to 3.6

 import asyncio async def searching_stuff_1(): # searching yield 1 # and searching yield 2 yield 3 async def searching_stuff_2(): yield 4 yield 5 async def gen(): async for i in searching_stuff_1(): yield i async for i in searching_stuff_2(): yield i async def gen_all(): return [i async for i in gen()] if __name__ == "__main__": result = asyncio.get_event_loop().run_until_complete(gen_all()) print(result) 

wanting to run two async async generators, you can use asyncio.gather .
But since asyncio.gather collects the result of coroutines in asynchronous mode, you need to combine each result from the asynchronous generator separately with async def gen2 before calling asyncio.gather.

 async def gen2(coro): return [i async for i in coro()] # combine two async async def gen_all2(): return list(chain.from_iterable(await gather(gen2(searching_stuff_1), gen2(searching_stuff_2)))) 

To prove our point, we can change searching_stuff to:

 async def searching_stuff_1(): print("searching_stuff_1 begin") # searching yield 1 await asyncio.sleep(1) # and searching yield 2 yield 3 print("searching_stuff_1 end") async def searching_stuff_2(): print("searching_stuff_2 begin") yield 4 await asyncio.sleep(1) yield 5 print("searching_stuff_2 end") 

and then make a move:

 result = asyncio.get_event_loop().run_until_complete(gen_all()) print(result) > searching_stuff_2 begin > searching_stuff_1 begin > searching_stuff_2 end > searching_stuff_1 end > [1, 2, 3, 4, 5]