Python urllib2 Main problem with Auth

Question

Update: Based on Lee's comment, I decided to condensate my code into a really simple script and run it from the command line:

import urllib2 import sys username = sys.argv[1] password = sys.argv[2] url = sys.argv[3] print("calling %s with %s:%s\n" % (url, username, password)) passman = urllib2.HTTPPasswordMgrWithDefaultRealm() passman.add_password(None, url, username, password) urllib2.install_opener(urllib2.build_opener(urllib2.HTTPBasicAuthHandler(passman))) req = urllib2.Request(url) f = urllib2.urlopen(req) data = f.read() print(data) 

Unfortunately, it still will not generate the Authorization (per Wireshark) header: (

I have a problem sending basic AUTH to urllib2. I looked through this article and followed suit. My code is:

 passman = urllib2.HTTPPasswordMgrWithDefaultRealm() passman.add_password(None, "api.foursquare.com", username, password) urllib2.install_opener(urllib2.build_opener(urllib2.HTTPBasicAuthHandler(passman))) req = urllib2.Request("http://api.foursquare.com/v1/user") f = urllib2.urlopen(req) data = f.read() 

I see the following on Wire through wirehark:

 GET /v1/user HTTP/1.1 Host: api.foursquare.com Connection: close Accept-Encoding: gzip User-Agent: Python-urllib/2.5 

You can see that authorization is not sent, and when I send a request through curl: curl -u user:password http://api.foursquare.com/v1/user

 GET /v1/user HTTP/1.1 Authorization: Basic =SNIP= User-Agent: curl/7.19.4 (universal-apple-darwin10.0) libcurl/7.19.4 OpenSSL/0.9.8k zlib/1.2.3 Host: api.foursquare.com Accept: */* 

For some reason, my code doesn't seem to send authentication - does anyone see what I am missing?

thank

-simon

Answer 1

The problem may be that the Python libraries, in accordance with the HTTP standard, first send an unauthenticated request, and then only if it answered with a repeat of 401, they are the correct credentials. If Foursquare servers do not perform “fully standard authentication,” then the libraries will not work.

Try using headers for authentication:

 import urllib2, base64 request = urllib2.Request("http://api.foursquare.com/v1/user") base64string = base64.b64encode('%s:%s' % (username, password)) request.add_header("Authorization", "Basic %s" % base64string) result = urllib2.urlopen(request) 

Had the same problem as you, and found a solution from this thread: http://forums.shopify.com/categories/9/posts/27662

Answer 2

(copy-paste / adapted from https://stackoverflow.com/a/125478/ ).

First you can subclass urllib2.BaseHandler or urllib2.HTTPBasicAuthHandler and implement http_request so that each request has a corresponding Authorization header.

 import urllib2 import base64 class PreemptiveBasicAuthHandler(urllib2.HTTPBasicAuthHandler): '''Preemptive basic auth. Instead of waiting for a 403 to then retry with the credentials, send the credentials if the url is handled by the password manager. Note: please use realm=None when calling add_password.''' def http_request(self, req): url = req.get_full_url() realm = None # this is very similar to the code from retry_http_basic_auth() # but returns a request object. user, pw = self.passwd.find_user_password(realm, url) if pw: raw = "%s:%s" % (user, pw) auth = 'Basic %s' % base64.b64encode(raw).strip() req.add_unredirected_header(self.auth_header, auth) return req https_request = http_request 

Then, if you are lazy like me, set the handler globally

 api_url = "http://api.foursquare.com/" api_username = "johndoe" api_password = "some-cryptic-value" auth_handler = PreemptiveBasicAuthHandler() auth_handler.add_password( realm=None, # default realm. uri=api_url, user=api_username, passwd=api_password) opener = urllib2.build_opener(auth_handler) urllib2.install_opener(opener) 

Answer 3

The second parameter should be a URI, not a domain name. i.e.

 passman = urllib2.HTTPPasswordMgrWithDefaultRealm() passman.add_password(None, "http://api.foursquare.com/", username, password) 

Answer 4

Here is what I use to solve a similar problem that I encountered while trying to access the MailChimp API. It does the same, just formatted better.

 import urllib2 import base64 chimpConfig = { "headers" : { "Content-Type": "application/json", "Authorization": "Basic " + base64.encodestring("hayden:MYSECRETAPIKEY").replace('\n', '') }, "url": 'https://us12.api.mailchimp.com/3.0/'} #perform authentication datas = None request = urllib2.Request(chimpConfig["url"], datas, chimpConfig["headers"]) result = urllib2.urlopen(request) 

Answer 5

I would suggest that the current solution is to use my urllib2_prior_auth package, which solves this pretty nicely (I'm working on including it in the standard library.