Python urllib2, basic HTTP authentication and tr.im

Question

I play trying to write code to use the tr.im API to shorten the url.

After reading http://docs.python.org/library/urllib2.html, I tried:

TRIM_API_URL = 'http://api.tr.im/api' auth_handler = urllib2.HTTPBasicAuthHandler() auth_handler.add_password(realm='tr.im', uri=TRIM_API_URL, user=USERNAME, passwd=PASSWORD) opener = urllib2.build_opener(auth_handler) urllib2.install_opener(opener) response = urllib2.urlopen('%s/trim_simple?url=%s' % (TRIM_API_URL, url_to_trim)) url = response.read().strip() 

response.code is 200 (I think it should be 202). The url is valid, but standard HTTP authentication did not seem to work, since the Abbreviated URL is not listed in my list of URLs ( http://tr.im/?page=1 ).

After reading http://www.voidspace.org.uk/python/articles/authentication.shtml#doing-it-properly I also tried:

  TRIM_API_URL = 'api.tr.im/api' password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm() password_mgr.add_password(None, TRIM_API_URL, USERNAME, PASSWORD) auth_handler = urllib2.HTTPBasicAuthHandler(password_mgr) opener = urllib2.build_opener(auth_handler) urllib2.install_opener(opener) response = urllib2.urlopen('http://%s/trim_simple?url=%s' % (TRIM_API_URL, url_to_trim)) url = response.read().strip() 

But I get the same results. (response.code is 200, and the url is valid, but not recorded in my account at http://tr.im/ .)

If I use the query string parameters instead of basic HTTP authentication, for example:

  TRIM_API_URL = 'http://api.tr.im/api' response = urllib2.urlopen('%s/trim_simple?url=%s&username=%s&password=%s' % (TRIM_API_URL, url_to_trim, USERNAME, PASSWORD)) url = response.read().strip() 

... then not only the URL is valid, but it will also be written to my tr.im account. (Although response.code is still 200.)

There should be something wrong in my code (not the tr.im API), because

 $ curl -u yacitus:xxxx http://api.tr.im/api/trim_url.json?url=http://www.google.co.uk 

... returns:

 {"trimpath":"hfhb","reference":"nH45bftZDWOX0QpVojeDbOvPDnaRaJ","trimmed":"11\/03\/2009","destination":"http:\/\/www.google.co.uk\/","trim_path":"hfhb","domain":"google.co.uk","url":"http:\/\/tr.im\/hfhb","visits":0,"status":{"result":"OK","code":"200","message":"tr.im URL Added."},"date_time":"2009-03-11T10:15:35-04:00"} 

... and the URL appears in my list of URLs at http://tr.im/?page=1 .

And if I run:

 $ curl -u yacitus:xxxx http://api.tr.im/api/trim_url.json?url=http://www.google.co.uk 

... again, I get:

 {"trimpath":"hfhb","reference":"nH45bftZDWOX0QpVojeDbOvPDnaRaJ","trimmed":"11\/03\/2009","destination":"http:\/\/www.google.co.uk\/","trim_path":"hfhb","domain":"google.co.uk","url":"http:\/\/tr.im\/hfhb","visits":0,"status":{"result":"OK","code":"201","message":"tr.im URL Already Created [yacitus]."},"date_time":"2009-03-11T10:15:35-04:00"} 

The note code is 201 and the message is "The URL tr.im has already been created [yacitus]."

I do not have to perform basic HTTP authentication correctly (in any attempt). Can you identify my problem? Perhaps I should look and see what is sent by wire? I have never done this before. Are there any Python APIs that I can use (possibly in pdb)? Or is there another tool (preferably for Mac OS X) that I can use?

Answer 1

This seems to work very well (taken from another thread)

 import urllib2, base64 request = urllib2.Request("http://api.foursquare.com/v1/user") base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '') request.add_header("Authorization", "Basic %s" % base64string) result = urllib2.urlopen(request) 

Answer 2

Really cheap solution:

 urllib.urlopen('http://user:[e-mail]/api') 

(which you can decide is not suitable for a number of reasons, such as URL security)

Github API Example :

 >>> import urllib, json >>> result = urllib.urlopen('https://personal-access-token:[e-mail]/repos/:owner/:repo') >>> r = json.load(result.fp) >>> result.close() 

Answer 3

Check out this SO post answer , and also see the basic tutorial on authentication from urllib2 missing a guide .

In order for basic urllib2 authentication to work, the HTTP response must contain the 401 Unauthorized HTTP code and the "WWW-Authenticate" key with the value "Basic" otherwise Python will not send your login information and you will need to either use Requests or urllib.urlopen(url) with your username in the URL or add a header, as in @Flowpoke answer .

You can view your error by putting urlopen in a try block:

 try: urllib2.urlopen(urllib2.Request(url)) except urllib2.HTTPError, e: print e.headers print e.headers.has_key('WWW-Authenticate') 

Answer 4

The recommended way is to use the requests module :

 #!/usr/bin/env python import requests # $ python -m pip install requests ####from pip._vendor import requests # bundled with python url = 'https://httpbin.org/hidden-basic-auth/user/passwd' user, password = 'user', 'passwd' r = requests.get(url, auth=(user, password)) # send auth unconditionally r.raise_for_status() # raise an exception if the authentication fails 

This uses a single source Python 2/3 compatible version of urllib2 :

 #!/usr/bin/env python import base64 try: from urllib.request import Request, urlopen except ImportError: # Python 2 from urllib2 import Request, urlopen credentials = '{user}:{password}'.format(**vars()).encode() urlopen(Request(url, headers={'Authorization': # send auth unconditionally b'Basic ' + base64.b64encode(credentials)})).close() 

Python 3.5+ introduces HTTPPasswordMgrWithPriorAuth() , which allows:

.. eliminate the unnecessary processing of the 401 response or unconditionally send the credentials on the first request to communicate with servers that return 404 instead of 401 if the authorization header is not sent.

 #!/usr/bin/env python3 import urllib.request as urllib2 password_manager = urllib2.HTTPPasswordMgrWithPriorAuth() password_manager.add_password(None, url, user, password, is_authenticated=True) # to handle 404 variant auth_manager = urllib2.HTTPBasicAuthHandler(password_manager) opener = urllib2.build_opener(auth_manager) opener.open(url).close() 

It is easy to replace HTTPBasicAuthHandler() with ProxyBasicAuthHandler() if necessary in this case.

Answer 5

The same solutions are used as Python urllib2 Basic authentication problem .

see https://stackoverflow.com/a/412960/ you can subclass urllib2.HTTPBasicAuthHandler add an Authorization header for each request that matches a known URL.

 class PreemptiveBasicAuthHandler(urllib2.HTTPBasicAuthHandler): '''Preemptive basic auth. Instead of waiting for a 403 to then retry with the credentials, send the credentials if the url is handled by the password manager. Note: please use realm=None when calling add_password.''' def http_request(self, req): url = req.get_full_url() realm = None # this is very similar to the code from retry_http_basic_auth() # but returns a request object. user, pw = self.passwd.find_user_password(realm, url) if pw: raw = "%s:%s" % (user, pw) auth = 'Basic %s' % base64.b64encode(raw).strip() req.add_unredirected_header(self.auth_header, auth) return req https_request = http_request 

Answer 6

I would suggest that the current solution is to use my urllib2_prior_auth package, which solves this pretty nicely (I'm working on including it in the standard library.

Answer 7

Try python-request or python-grab