Asyncio Sets HTTP Requests Slower?

Question

I use Asyncio and Requests to test a series of HTTP requests.

For some reason, it uses Asyncio a little slower than just direct requests. Any idea why? Am I using Asyncio incorrectly?

import asyncio import functools import requests import time ts = time.time() for i in range(10): @asyncio.coroutine def do_checks(): loop = asyncio.get_event_loop() req = loop.run_in_executor(None, functools.partial(requests.get, "http://google.com", timeout=3)) resp = yield from req print(resp.status_code) loop = asyncio.get_event_loop() loop.run_until_complete(do_checks()) te = time.time() print("Version A: " + str(te - ts)) ts = time.time() for i in range(10): r = requests.get("http://google.com", timeout=3) print(r.status_code) te = time.time() print("Version B: " + str(te - ts)) 

Conclusion:

Version A = Asyncio; Version B = Requests

 200 200 200 200 200 200 200 200 200 200 Version A: 5.7215821743011475 200 200 200 200 200 200 200 200 200 200 Version B: 5.320340156555176 

Answer 1

You wait for each request to complete before starting the next one. This way you have the overhead of an event loop without any benefits.

Try the following:

 import asyncio import functools import requests import time ts = time.time() loop = asyncio.get_event_loop() @asyncio.coroutine def do_checks(): futures = [] for i in range(10): futures.append(loop.run_in_executor(None, functools.partial(requests.get, "http://google.com", timeout=3))) for req in asyncio.as_completed(futures): resp = yield from req print(resp.status_code) loop.run_until_complete(do_checks()) te = time.time() print("Version A: " + str(te - ts)) ts = time.time() for i in range(10): r = requests.get("http://google.com", timeout=3) print(r.status_code) te = time.time() print("Version B: " + str(te - ts)) 

This is what I get when I run it:

 $ python test.py 200 ... Version A: 0.43438172340393066 200 ... Version B: 1.6541109085083008 

Much faster, but in reality these are just spawning streams and waiting for the http library to finish working, you do not need asyncio for this.

You might want to check out aiohttp as it was created for use with asyncio . requests is a fabulous library, but it is not made for asyncio .

Answer 2

Just for completeness, here is a really quick implementation of asyncio

 import aiohttp import asyncio import time async def main(n): ts = time.time() session = aiohttp.ClientSession() fs = [session.get('http://google.com') for _ in range(n)] for f in asyncio.as_completed(fs): resp = await f print(resp.status) await resp.release() session.close() te = time.time() print("Aiohttp version: " + str(te - ts)) loop = asyncio.get_event_loop() loop.run_until_complete(main(10)) loop.close() 

The code is python 3.5 and higher.

 ~> python asyncioreq.py 200 ... Aiohttp version: 0.15974688529968262 

Hope someone can use it;)