Install package dependencies for Scrapy

Question

So, among the many packages that users should install for Scrapy, I think I'm having problems with pyOpenSSL.

When I try to create a Scrapy training project, I get the following output:

Traceback (most recent call last): File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "C:\Python27\lib\runpy.py", line 72, in _run_code exec code in run_globals File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 168, in <module> execute() File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 122, in execute cmds = _get_commands_dict(settings, inproject) File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 46, in _get_comma nds_dict cmds = _get_commands_from_module('scrapy.commands', inproject) File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 29, in _get_comma nds_from_module for cmd in _iter_command_classes(module): File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 20, in _iter_comm and_classes for module in walk_modules(module_name): File "C:\Python27\lib\site-packages\scrapy\utils\misc.py", line 68, in walk_mo dules submod = import_module(fullpath) File "C:\Python27\lib\importlib\__init__.py", line 37, in import_module __import__(name) File "C:\Python27\lib\site-packages\scrapy\commands\bench.py", line 3, in <mod ule> from scrapy.tests.mockserver import MockServer File "C:\Python27\lib\site-packages\scrapy\tests\mockserver.py", line 6, in <m odule> from twisted.internet import reactor, defer, ssl File "C:\Python27\lib\site-packages\twisted\internet\ssl.py", line 59, in <mod ule> from OpenSSL import SSL File "build\bdist.win32\egg\OpenSSL\__init__.py", line 8, in <module> File "build\bdist.win32\egg\OpenSSL\rand.py", line 11, in <module> File "build\bdist.win32\egg\OpenSSL\_util.py", line 3, in <module> ImportError: No module named cryptography.hazmat.bindings.openssl.binding 

And when I looked for this last error (without a module named cryptography.hazmat), I see a couple of mentions of pyOpenSSL. So I went ahead and tried to run easy_install pyOpenSSL==0.14 to make sure this is the latest version, but when I do this, I get this output:

 c:\python27\include\pymath.h(22) : warning C4273: 'round' : inconsistent dll lin kage C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\INCLUDE\math.h(51 6) : see previous definition of 'round' c:\users\bk\appdata\local\temp\easy_install-tztawu\cryptography-0.4\temp\easy_in stall-svxsjy\cffi-0.8.2\c\misc_win32.h(225) : error C2632: 'char' followed by 'b ool' is illegal c:\users\bk\appdata\local\temp\easy_install-tztawu\cryptography-0.4\temp\easy_in stall-svxsjy\cffi-0.8.2\c\misc_win32.h(225) : warning C4091: 'typedef ' : ignore d on left of 'unsigned char' when no variable is declared c/_cffi_backend.c(5295) : warning C4146: unary minus operator applied to unsigne d type, result still unsigned c/_cffi_backend.c(5296) : warning C4146: unary minus operator applied to unsigne d type, result still unsigned c/_cffi_backend.c(5297) : warning C4146: unary minus operator applied to unsigne d type, result still unsigned c/_cffi_backend.c(5298) : warning C4146: unary minus operator applied to unsigne d type, result still unsigned error: Setup script exited with error: command '"C:\Program Files (x86)\Microsof t Visual Studio 12.0\VC\BIN\cl.exe"' failed with exit status 2 

So, I lost a little what I need to do to start Scrapy correctly

Answer 1

I got the same error on Mac OS.

I decided to use it with openssl 0.13 instead of the latest version.

 easy_install pyOpenSSL==0.13 

or

 pip install pyOpenSSL==0.13 

Answer 2

I highly recommend using conda instead of pip , especially when using Windows. Among many other things, it will capture the appropriate binaries for your system. It makes setting up the python scientific environment (think Scipy, Numpy, Pandas ...) a breeze.

So read Anaconda , install Anaconda , then do:

 conda create -n scrapyenv python=2 # creates a new py2 environment activate scrapyenv # switch to the new environment conda install scrapy # install scrapy 

Steps 1 and 2 are only necessary if you want it to be encapsulated in a separate environment. Btw, the whole battery of useful packages will be installed if you do conda install anaconda .

Additionally, in case conda does not include pyOpenSSL or you DO NOT want to install anaconda , please take a look at point 9 of the How to install Scrapy on 64-bit Windows 7 tutorial.

Answer 3

I had the same problems and tried to solve it with the first answer, but it does not work. In the end, I remove pyOpenSSL and load pyopenssl, setup. And the problem is solved. Piopensle URL: https://launchpad.net/pyopenssl

Answer 4

You must update pip before trying to install Scrapy :

 pip install --upgrade pip pip install Scrapy