Geek Answers Handbook

Urllib can't read https

(Python 3.4.2) Can anyone help me get https pages from urllib? I spent hours trying to figure it out.

Here is what I'm trying to do (pretty simple):

import urllib.request
url = "".join((baseurl, other_string, midurl, query))
response = urllib.request.urlopen(url)
html = response.read()

Here my error is displayed at startup:

File "./script.py", line 124, in <module>
    response = urllib.request.urlopen(url)
  File "/usr/lib/python3.4/urllib/request.py", line 153, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 455, in open
    response = self._open(req, data)
  File "/usr/lib/python3.4/urllib/request.py", line 478, in _open
    'unknown_open', req)
  File "/usr/lib/python3.4/urllib/request.py", line 433, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 1244, in unknown_open
    raise URLError('unknown url type: %s' % type)
urllib.error.URLError: <urlopen error unknown url type: 'https>

I also tried using data = None to no help:

response = urllib.request.urlopen(url, data=None)

I also tried this:

import urllib.request, ssl
https_sslv3_handler = urllib.request.HTTPSHandler(context=ssl.SSLContext(ssl.PROTOCOL_SSLv3))
opener = urllib.request.build_opener(https_sslv3_handler)
urllib.request.install_opener(opener)
resp = opener.open(url)
html = resp.read().decode('utf-8')
print(html)

A similar error occurs with this ^ script, where the error is found on the line "resp = ..." and complains that "https" is an unknown type of URL.

Python was compiled with SSL support on my computer (Arch Linux). I tried reinstalling python3 and openssl several times, but this does not help. I am not trying to completely remove python and then reinstall, because I will also need to remove many other programs on my computer.

Does anyone know what is going on?

----- EDIT -----

, . url ":" , , urllib . "% 3A", . !!!

+6
4

, , - .

:

from urllib.request import urlopen
resp = urlopen('https://github.com')
print(resp.read())
+3
urllib.error.URLError: <urlopen error unknown url type: 'https>

'https, https , http://, 'https://, , , . , URL.

+1

SSL

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

url = input('Enter - ')
html = urllib.request.urlopen(url, context=ctx).read()
+1

, URL- https, http.

>>> from urllib.request import urlopen
>>> urlopen('http://google.com')
<http.client.HTTPResponse object at 0xb770252c>
>>> urlopen('https://google.com')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/local/lib/python3.7/urllib/request.py", line 548, in _open
    'unknown_open', req)
  File "/usr/local/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.7/urllib/request.py", line 1387, in unknown_open
    raise URLError('unknown url type: %s' % type)
urllib.error.URLError: <urlopen error unknown url type: https>

Ubuntu 16.04 Python 3.7. Ubuntu Python 3.5 /usr/bin, 3.7 /usr/local/bin. , 3.5, /usr/bin/openssl 3.7, :

>>> import ssl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/ssl.py", line 98, in <module>
    import _ssl             # if we can't import it, let the error propagate
ModuleNotFoundError: No module named '_ssl'

Having examined this link , I changed SSL = / usr / local / ssl to SSL = / usr in 3.7 source dir Modules / Setup.dist, and also copied it to Setup and then rebuilt Python 3.7.

$ ./configure
$ make
$ make install

This is now fixed:

>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.0.2g  1 Mar 2016'
>>> urlopen('https://www.google.com') 
<http.client.HTTPResponse object at 0xb74c4ecc>
>>> urlopen('https://www.google.com').read()
b'<!doctype html>...

and 3.7 OpenSSL support was successfully implemented. Note that the Ubuntu "openssl version" command is not complete until you download it in Python.

0
source

More articles:


All Articles