Twisted critical raw error in a workbook

Question

I am new to programming and I am trying to learn scrapy using the tutorial: http://doc.scrapy.org/en/latest/intro/tutorial.html

So, I ran the scraw crawl dmoz command and got this error:

2015-07-14 16:11:02 [scrapy] INFO: Scrapy 1.0.1 started (bot: tutorial) 2015-07-14 16:11:02 [scrapy] INFO: Optional features available: ssl, http11 2015-07-14 16:11:02 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'tu torial.spiders', 'SPIDER_MODULES': ['tutorial.spiders'], 'BOT_NAME': 'tutorial'} 2015-07-14 16:11:05 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsol e, LogStats, CoreStats, SpiderState Unhandled error in Deferred: 2015-07-14 16:11:06 [twisted] CRITICAL: Unhandled error in Deferred: 2015-07-14 16:11:07 [twisted] CRITICAL: 

I am using windows 7 and python 2.7. Does anyone know what the problem is? How can i fix this?

EDIT: My spider file code:

 # This package will contain the spiders of your Scrapy project # # Please refer to the documentation for information on how to create and manage # your spiders. import scrapy class DmozSpider(scrapy.Spider): name = "dmoz" allowed_domains = ["dmoz.org"] start_urls = [ "http://www.dmoz.org/computers/programming/languages/python/books/", "http://www.dmoz.org/computer/programming/languages/python/resources/" ] def parse(self, response): filename = response.url.split("/")[-2] + '.html' with open(filename,'wb') as f: f.write(response.body) 

items.py code:

 import scrapy class DmozItem(scrapy.Item): title = scrapy.Field() link = scrapy.Field() desc = scrapy.Field() 

picket list:

• bootstrap-admin (0.3.3)
• cffi (1.1.2)
• characteristic (14.3.0)
• cryptography (0.9.3)
• cssselect (0.9.1)
• Django (1.7.7)
• django-auth-ldap (1.2.4)
• django-debug-toolbar (1.3.0)
• django-mssql (1.6.2)
• django-pyodbc (0.2.6)
• django-pyodbc-azure (1.2.2)
• django-redator (0.2.3)
• django-reversion (1.8.5)
• django-summernote (0.6.0)
• django-windows-tools (0.1.1)
• django-wysiwyg-redactor (0.4.3.2)
• enum34 (1.0.4)
• ez-setup (0.9)
• flup (1.0.2)
• idna (2.0)
• ipaddress (1.0.13)
• iso8601 (0.1.4)
• logging (0.4.9.6)
• lxml (3.4.4)
• mechanize (0.2.5)
• MySQL-python (1.2.4)
• pbr (0.10.8)
• Pillow (2.7.0)
• pip (7.1.0)
• pyasn1 (0.1.8)
• pyasn1-modules (0.0.6)
• pycparser (2.14)
• pymongo (2.6)
• pyodbc (3.0.7)
• pyOpenSSL (0.15.1)
• pypm (1.4.3)
• python-ldap (2.4.18)
• pythonselect (1.3)
• pywin32 (218.3)
• queuelib (1.2.2)
• Scrapy (1.0.1)
• selenium (2.44.0)
• service identification (14.0.0)
• setuptools (18.0.1)
• six (1.9.0)
• sqlparse (0.1.15)
• stevedore (1.3.0)
• Twisted (15.2.1)
• virtualenv (1.11.6)
• virtualenv-clone (0.2.5)
• virtualenvwrapper (4.3.2)
• virtualenvwrapper-powershell (12.7.8)
• w3lib (1.11.0)
• xlrd (0.9.2)
• zope.interface (4.1.2)

thanks for attention and sry for my poor english, not my native language.

Answer 1

I also begin to study radiation therapy and come across the same question with yours. After struggling with it in the afternoon, finally, I found it due to the pywin32 module loading without installation. You can try entering the command below in cmd to complete the installation of the pywin32 module and retry the scan:

python python27 \ scripts \ pywin32_postinstall.py -install

Hope this helps!

Answer 2

The short answer is you are missing pywin32!

Other answers are mostly correct, but not 100% correct. pywin32 is not a pip installation! You should download the installer package from here:

http://sourceforge.net/projects/pywin32/files/pywin32/

Make sure you get the correct bit: 32 or 64. In my case, I did not understand that I had a 32-bit version of Python installed on my 64-bit computer, and the installer did not work with "Cannot find Python 2.7 installation in the registry, "I had to install the 32-bit version of pywin32. As soon as I did this, the scanning site worked.

Answer 3

I do not see what you are doing with the elements as an entry in the file. But there may be imports. Try, if this does not work, try installing pywin -update and installing pip Twisted -update, which should reinstall the damaged files. Plus, I don’t know if there was a problem with Stack, but you had some inappropriate identification. from scrapy.spiders import Spider

 from {Projectname}.items import {Itemclass} import scrapy class DmozSpider(scrapy.Spider): name = "dmoz" allowed_domains = ["dmoz.org"] start_urls = [ "http://www.dmoz.org/computers/programming/languages/python/books/", "http://www.dmoz.org/computer/programming/languages/python/resources/"] def parse(self, response): filename = response.url.split("/")[-2] + '.html' with open(filename,'wb') as f: f.write(response.body) 

Answer 4

Scrapy crashes with: ImportError: not a single module named win32api

You need to install pywin32 due to this Twisted error.