Receive and store images in python asynchronously

Question

The following code is a sample of non-asynchronous code, is there a way to get images asynchronously?

import urllib for x in range(0,10): urllib.urlretrieve("http://test.com/file %s.png" % (x), "temp/file %s.png" % (x)) 

I also saw the Grequests library, but I could not understand much if this is possible or how to do it from the documentation.

Answer 1

You do not need a third-party library. Just create a thread for each request, start the threads, and then wait until they are all completed in the background, or continue the application while loading images.

 import threading results = [] def getter(url, dest): results.append(urllib.urlretreave(url, dest)) threads = [] for x in range(0,10): t = threading.Thread(target=getter, args=('http://test.com/file %s.png' % x, 'temp/file %s.png' % x)) t.start() threads.append(t) # wait for all threads to finish # You can continue doing whatever you want and # join the threads when you finally need the results. # They will fatch your urls in the background without # blocking your main application. map(lambda t: t.join(), threads) 

Optionally, you can create a thread pool that will receive urls and dests from the queue.

If you are using Python 3, it is already implemented for you in the futures module.

Answer 2

Something like this should help you

 import grequests urls = ['url1', 'url2', ....] # this should be the list of urls requests = (grequests.get(u) for u in urls) responses = grequests.map(requests) for response in responses: if 199 < response.status_code < 400: name = generate_file_name() # generate some name for your image file with extension like example.jpg with open(name, 'wb') as f: # or save to S3 or something like that f.write(response.content) 

Here, only loading the images will be parallel, but writing each content of the image to a file will be sequential so that you can create a stream or do something else to make it parallel or asynchronous