Geek Answers Handbook

Urlopen / requests.get does not work on threads created in imported modules

I have a problem with urlopen (and request.get)

In my program, if I run it inside a thread (I tested using it multiprocessingtoo) [update: thread created by the imported module], it will not work until the program finishes.

By default, it doesn’t start, I mean it doesn’t even start: the timeout (here 3 seconds) will never work, and no will be created on the website .

Here is my simplified code:

import threading,urllib2,time

def dlfile(url):
  print 'Before request'
  r = urllib2.urlopen(url, timeout=3)
  print 'After request'
  return r

def dlfiles(*urls):
  threads = [threading.Thread(None, dlfile, None, (url,), {}) for url in urls]
  map(lambda t:t.start(), threads)

def main():
    dlfiles('http://google.com')

main()
time.sleep(10)
print 'End of program'

My conclusion:

Before request
End of program
After request

Unfortunately, the code that I write in SO works as expected (ie, "Before requesting / after requesting / completing the program"), and I still cannot reproduce the problem with simplified code.

, , - - . , , .

, , , interwebs

UPDATE

threadtest.py

import threading,urllib2,time
def log(a):print(a)
def dlfile(url):
  log('Before request')
  r = urllib2.urlopen(url, timeout=3)
  log('After request')
  return r

def dlfiles(*urls):
  threads = [threading.Thread(None, dlfile, None, (url,), {}) for url in urls]
  map(lambda t:t.start(), threads)

def main():
    dlfiles('http://google.com')

main()
for i in range(5):
    time.sleep(1)
    log('Sleep')
log('End of program')

threadtest-import.py

import threadtest

:

$ python threadtest.py
Before request
After request
Sleep
Sleep
Sleep
Sleep
Sleep
End of program

$ python threadtest-import.py
Before request
Sleep
Sleep
Sleep
Sleep
Sleep
End of program
After request

, , : ? ?

? , urlopen, .

+4
2

, @user3351750 .

. threadtest-import.py threadtest , , - * ( ) . IIRC re urllib. , .

, . , .

.. :

import threadtest #do nothing except declarations
threadtest.run() #do the work

:

import threadtest #declarations + work


main()
for i in range(5):
    time.sleep(1)
    log('Sleep')
log('End of program')

run:

def run():
    main()
    for i in range(5):
        time.sleep(1)
        log('Sleep')
    log('End of program')

, * , , .

0

. .

def main():
    dlfiles('http://google.fr')

url.

threads = [threading.Thread(None, dlfile, None, (url,), {}) for url in urls]

, urls .

:

def main():
    dlfiles('http://google.fr', 'http://google.com', 'http://google.gg')
+1

More articles:


All Articles