In Python 3.2, I can open and read the HTTPS web page with http.client, but urllib.request will not be able to open the same page

Question

I want to open and read https://yande.re/ with urllib.request , but I get an SSL error. I can easily open and read the page using http.client with this code:

 import http.client conn = http.client.HTTPSConnection('www.yande.re') conn.request('GET', 'https://yande.re/') resp = conn.getresponse() data = resp.read() 

However, the following code using urllib.request does not work:

 import urllib.request opener = urllib.request.build_opener() resp = opener.open('https://yande.re/') data = resp.read() 

This results in an error: ssl.SSLError: [Errno 1] _ssl.c:392: error:1411809D:SSL routines:SSL_CHECK_SERVERHELLO_TLSEXT:tls invalid ecpointformat list . Why can I open a page using HTTPSConnection but not openener.open?

Edit: Here is my version of OpenSSL and the trace from trying to open https://yande.re/

 >>> import ssl; ssl.OPENSSL_VERSION 'OpenSSL 1.0.0a 1 Jun 2010' >>> import urllib.request >>> urllib.request.urlopen('https://yande.re/') Traceback (most recent call last): File "<pyshell#3>", line 1, in <module> urllib.request.urlopen('https://yande.re/') File "C:\Python32\lib\urllib\request.py", line 138, in urlopen return opener.open(url, data, timeout) File "C:\Python32\lib\urllib\request.py", line 369, in open response = self._open(req, data) File "C:\Python32\lib\urllib\request.py", line 387, in _open '_open', req) File "C:\Python32\lib\urllib\request.py", line 347, in _call_chain result = func(*args) File "C:\Python32\lib\urllib\request.py", line 1171, in https_open context=self._context, check_hostname=self._check_hostname) File "C:\Python32\lib\urllib\request.py", line 1138, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [Errno 1] _ssl.c:392: error:1411809D:SSL routines:SSL_CHECK_SERVERHELLO_TLSEXT:tls invalid ecpointformat list> >>> 

Answer 1

What a coincidence! I have the same problem as you, with an additional complication: I am for the proxy. I found this regarding https-not-working-with-urllib. Fortunately, they posted a workaround.

 import urllib.request import ssl ##uncomment this code if you're behind a proxy ##https port is 443 but it doesn't work for me, used port 80 instead ##proxy_auth = '{0}://{1}:{2}@{3}'.format('https', 'username', 'password', ## 'proxy:80') ##proxies = { 'https' : proxy_auth } ##proxy = urllib.request.ProxyHandler(proxies) ##proxy_auth_handler = urllib.request.HTTPBasicAuthHandler() ##opener = urllib.request.build_opener(proxy, proxy_auth_handler, ## https_sslv3_handler) https_sslv3_handler = urllib.request.HTTPSHandler(context=ssl.SSLContext(ssl.PROTOCOL_SSLv3)) opener = urllib.request.build_opener(https_sslv3_handler) urllib.request.install_opener(opener) resp = opener.open('https://yande.re/') data = resp.read().decode('utf-8') print(data) 

Btw, thanks for showing how to use http.client . I did not know that there is another library that can be used to connect to the Internet .;)

Answer 2

This is due to a bug in the early version 1.x of OpenSSL for cryptography of an elliptic curve . Take a closer look at the relevant part of the exception:

 _ssl.c:392: error:1411809D:SSL routines:SSL_CHECK_SERVERHELLO_TLSEXT:tls invalid ecpointformat list 

This is an error from the OpenSSL library base code, which is the result of incorrect extension of the TLS extension in EC format. One workaround is to use SSLv3 instead of SSLv23, another workaround is to use a cipher suite specification that disables all ECC encryption lists (I had good results with ALL:-ECDH , use openssl ciphers for testing). The fix is ​​updating OpenSSL.

Answer 3

The problem is with the host names you give in two examples:

 import http.client conn = http.client.HTTPSConnection('www.yande.re') conn.request('GET', 'https://yande.re/') 

and...

 import urllib.request urllib.request.urlopen('https://yande.re/') 

Please note that in the first example, you ask the client to make a connection to the host: www.yande.re, and in the second example urllib will first analyze the URL https://yande.re 'and then try the request on the yande.re host

Although www.yande.re and yande.re can resolve to the same IP address, from the point of view of the web server they are different virtual hosts. I assume that you had an SNI configuration problem on your web server. Seeing that the original question was published on May 21, and the current certificate on yande.re starts on May 28, I think that you have already fixed this problem?

Answer 4

Try the following:

 import connection #imports connection import url url = 'http://www.google.com/' webpage = url.open(url) try: connection.receive(webpage) except: webpage = url.text('This webpage is not available!') connection.receive(webpage)