Documente Academic
Documente Profesional
Documente Cultură
Release 0.23.0
Scrapy developers
Contents
1 2
Getting help First steps 2.1 Scrapy at a glance 2.2 Installation guide . 2.3 Scrapy Tutorial . . 2.4 Examples . . . . . Basic concepts 3.1 Command line tool 3.2 Items . . . . . . . 3.3 Spiders . . . . . . 3.4 Selectors . . . . . 3.5 Item Loaders . . . 3.6 Scrapy shell . . . 3.7 Item Pipeline . . . 3.8 Feed exports . . . 3.9 Link Extractors . . Built-in services 4.1 Logging . . . . . 4.2 Stats Collection . 4.3 Sending e-mail . 4.4 Telnet Console . 4.5 Web Service . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
Solving specic problems 5.1 Frequently Asked Questions 5.2 Debugging Spiders . . . . . 5.3 Spiders Contracts . . . . . . 5.4 Common Practices . . . . . 5.5 Broad Crawls . . . . . . . . 5.6 Using Firefox for scraping . 5.7 Using Firebug for scraping . 5.8 Debugging memory leaks . 5.9 Downloading Item Images . 5.10 Ubuntu packages . . . . . . 5.11 Scrapyd . . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
109 111 112 113 117 117 119 129 133 138 143 143 150 162 164 166 173 173 191 193 194 195
Extending Scrapy 6.1 Architecture overview . 6.2 Downloader Middleware 6.3 Spider Middleware . . . 6.4 Extensions . . . . . . . 6.5 Core API . . . . . . . . Reference 7.1 Requests and Responses 7.2 Settings . . . . . . . . . 7.3 Signals . . . . . . . . . 7.4 Exceptions . . . . . . . 7.5 Item Exporters . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
All the rest 8.1 Release notes . . . . . . . . 8.2 Contributing to Scrapy . . . 8.3 Versioning and API Stability 8.4 Experimental features . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
ii
Contents
Contents
CHAPTER 1
Getting help
Having trouble? Wed like to help! Try the FAQ its got answers to some common questions. Looking for specic information? Try the genindex or modindex. Search for information in the archives of the scrapy-users mailing list, or post a question. Ask a question in the #scrapy IRC channel. Report bugs with Scrapy in our issue tracker.
CHAPTER 2
First steps
<div id="description"> Short documentary made for Plymouth City Museum and Art Gallery regarding the setup of an exhibit abo ...
Finally, the le size is contained in the second <p> tag inside the <div> tag with id=specifications:
<div id="specifications"> <p> <strong>Category:</strong> <a href="/cat/4">Movies</a> > <a href="/sub/35">Documentary</a> </p> <p> <strong>Total size:</strong> 150.62 megabyte</p>
For more information about XPath see the XPath reference. Finally, heres the spider code:
from scrapy.contrib.spiders import CrawlSpider, Rule from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor from scrapy.selector import Selector
class MininovaSpider(CrawlSpider): name = mininova allowed_domains = [mininova.org] start_urls = [http://www.mininova.org/today] rules = [Rule(SgmlLinkExtractor(allow=[/tor/\d+]), parse_torrent)] def parse_torrent(self, response): sel = Selector(response) torrent = TorrentItem() torrent[url] = response.url torrent[name] = sel.xpath("//h1/text()").extract() torrent[description] = sel.xpath("//div[@id=description]").extract() torrent[size] = sel.xpath("//div[@id=info-left]/p[2]/text()[2]").extract() return torrent
This uses feed exports to generate the JSON le. You can easily change the export format (XML or CSV, for example) or the storage backend (FTP or Amazon S3, for example). You can also write an item pipeline to store the items in a database very easily.
Youll notice that all eld values (except for the url which was assigned directly) are actually lists. This is because the selectors return lists. You may want to store single values, or perform some additional parsing/cleansing to the values. Thats what Item Loaders are for.
Support for extending Scrapy by plugging your own functionality using signals and a well-dened API (middlewares, extensions, and pipelines). Wide range of built-in middlewares and extensions for: cookies and session handling HTTP compression HTTP authentication HTTP cache user-agent spoong robots.txt crawl depth restriction and more Robust encoding support and auto-detection, for dealing with foreign, non-standard and broken encoding declarations. Support for creating spiders based on pre-dened templates, to speed up spider creation and make their code more consistent on large projects. See genspider command for more details. Extensible stats collection for multiple spider metrics, useful for monitoring the performance of your spiders and detecting when they get broken An Interactive shell console for trying XPaths, very useful for writing and debugging your spiders A System service designed to ease the deployment and run of your spiders in production. A built-in Web service for monitoring and controlling your bot A Telnet console for hooking into a Python console running inside your Scrapy process, to introspect and debug your crawler Logging facility that you can hook on to for catching errors during the scraping process. Support for crawling based on URLs discovered through Sitemaps A caching DNS resolver
OpenSSL. This comes preinstalled in all operating systems except Windows (see Platform specic installation notes) pip or easy_install Python package managers
Dont use the python-scrapy package provided by Ubuntu, they are typically too old and slow to catch up with latest Scrapy. Instead, use the ofcial Ubuntu Packages, which already solve all dependencies for you and are continuously updated with the latest bug xes.
These are basically: scrapy.cfg: the project conguration le tutorial/: the projects python module, youll later import your code from here. tutorial/items.py: the projects items le. tutorial/pipelines.py: the projects pipelines le.
10
tutorial/settings.py: the projects settings le. tutorial/spiders/: a directory where youll later put your spiders.
This may seem complicated at rst, but dening the item allows you to use other handy components of Scrapy that need to know how your item looks like.
11
Crawling To put our spider to work, go to the projects top level directory and run:
scrapy crawl dmoz
The crawl dmoz command runs the spider for the dmoz.org domain. You will get an output similar to this:
2014-01-23 2014-01-23 2014-01-23 2014-01-23 2014-01-23 2014-01-23 2014-01-23 2014-01-23 2014-01-23 2014-01-23 2014-01-23 18:13:07-0400 18:13:07-0400 18:13:07-0400 18:13:07-0400 18:13:07-0400 18:13:07-0400 18:13:07-0400 18:13:07-0400 18:13:08-0400 18:13:09-0400 18:13:09-0400
[scrapy] INFO: Scrapy started (bot: tutorial) [scrapy] INFO: Optional features available: ... [scrapy] INFO: Overridden settings: {} [scrapy] INFO: Enabled extensions: ... [scrapy] INFO: Enabled downloader middlewares: ... [scrapy] INFO: Enabled spider middlewares: ... [scrapy] INFO: Enabled item pipelines: ... [dmoz] INFO: Spider opened [dmoz] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/L [dmoz] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/L [dmoz] INFO: Closing spider (finished)
Pay attention to the lines containing [dmoz], which corresponds to our spider. You can see a log line for each URL dened in start_urls. Because these URLs are the starting ones, they have no referrers, which is shown at the end of the log line, where it says (referer: <None>). But more interesting, as our parse method instructs, two les have been created: Books and Resources, with the content of both URLs.
What just happened under the hood?
Scrapy creates scrapy.http.Request objects for each URL in the start_urls attribute of the Spider, and assigns them the parse method of the spider as their callback function. These Requests are scheduled, then executed, and scrapy.http.Response objects are returned and then fed back to the spider, through the parse() method. Extracting Items
Introduction to Selectors
There are several ways to extract data from web pages. Scrapy uses a mechanism based on XPath or CSS expressions called Scrapy Selectors. For more information about selectors and other extraction mechanisms see the Selectors documentation. Here are some examples of XPath expressions and their meanings: /html/head/title: selects the <title> element, inside the <head> element of a HTML document /html/head/title/text(): selects the text inside the aforementioned <title> element. 12 Chapter 2. First steps
//td: selects all the <td> elements //div[@class="mine"]: selects all div elements which contain an attribute class="mine" These are just a couple of simple examples of what you can do with XPath, but XPath expressions are indeed much more powerful. To learn more about XPath we recommend this XPath tutorial. For working with XPaths, Scrapy provides a Selector class, it is instantiated with a HtmlResponse or XmlResponse object as rst argument. You can see selectors as objects that represent nodes in the document structure. So, the rst instantiated selectors are associated to the root node, or the entire document. Selectors have four basic methods (click on the method to see the complete API documentation). xpath(): returns a list of selectors, each of them representing the nodes selected by the xpath expression given as argument. css(): returns a list of selectors, each of them representing the nodes selected by the CSS expression given as argument. extract(): returns a unicode string with the selected data. re(): returns a list of unicode strings extracted by applying the regular expression given as argument.
Trying Selectors in the Shell
To illustrate the use of Selectors were going to use the built-in Scrapy shell, which also requires IPython (an extended Python console) installed on your system. To start a shell, you must go to the projects top level directory and run:
scrapy shell "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/"
Note: Remember to always enclose urls with quotes when running Scrapy shell from command-line, otherwise urls containing arguments (ie. & character) will not work. This is what the shell looks like:
[ ... Scrapy log here ... ]
2014-01-23 17:11:42-0400 [default] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programmin [s] Available Scrapy objects: [s] crawler <scrapy.crawler.Crawler object at 0x3636b50> [s] item {} [s] request <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> [s] response <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> [s] sel <Selector xpath=None data=u<html>\r\n<head>\r\n<meta http-equiv="Conten> [s] settings <CrawlerSettings module=None> [s] spider <Spider default at 0x3cebf50> [s] Useful shortcuts: [s] shelp() Shell help (print this help) [s] fetch(req_or_url) Fetch request (or URL) and update local objects [s] view(response) View response in a browser In [1]:
After the shell loads, you will have the response fetched in a local response variable, so if you type response.body you will see the body of the response, or you can type response.headers to see its headers.
13
The shell also pre-instantiate a selector for this response in variable sel, the selector automatically chooses the best parsing rules (XML vs HTML) based on responses type. So lets try it:
In [1]: sel.xpath(//title) Out[1]: [<Selector xpath=//title data=u<title>Open Directory - Computers: Progr>] In [2]: sel.xpath(//title).extract() Out[2]: [u<title>Open Directory - Computers: Programming: Languages: Python: Books</title>] In [3]: sel.xpath(//title/text()) Out[3]: [<Selector xpath=//title/text() data=uOpen Directory - Computers: Programming:>] In [4]: sel.xpath(//title/text()).extract() Out[4]: [uOpen Directory - Computers: Programming: Languages: Python: Books] In [5]: sel.xpath(//title/text()).re((\w+):) Out[5]: [uComputers, uProgramming, uLanguages, uPython]
Now, lets try to extract some real information from those pages. You could type response.body in the console, and inspect the source code to gure out the XPaths you need to use. However, inspecting the raw HTML code there could become a very tedious task. To make this an easier task, you can use some Firefox extensions like Firebug. For more information see Using Firebug for scraping and Using Firefox for scraping. After inspecting the page source, youll nd that the web sites information is inside a <ul> element, in fact the second <ul> element. So we can select each <li> element belonging to the sites list with this code:
sel.xpath(//ul/li)
As we said before, each .xpath() call returns a list of selectors, so we can concatenate further .xpath() calls to dig deeper into a node. We are going to use that property here, so:
sites = sel.xpath(//ul/li) for site in sites: title = site.xpath(a/text()).extract() link = site.xpath(a/@href).extract() desc = site.xpath(text()).extract() print title, link, desc
Note:
For a more detailed description of using nested selectors, see Nesting selectors and Working with relative
14
XPaths in the Selectors documentation Lets add this code to our spider:
from scrapy.spider import Spider from scrapy.selector import Selector class DmozSpider(Spider): name = "dmoz" allowed_domains = ["dmoz.org"] start_urls = [ "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/", "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/" ] def parse(self, response): sel = Selector(response) sites = sel.xpath(//ul/li) for site in sites: title = site.xpath(a/text()).extract() link = site.xpath(a/@href).extract() desc = site.xpath(text()).extract() print title, link, desc
Notice we import our Selector class from scrapy.selector and instantiate a new Selector object. We can now specify our XPaths just as we did in the shell. Now try crawling the dmoz.org domain again and youll see sites being printed in your output, run:
scrapy crawl dmoz
Using our item Item objects are custom python dicts; you can access the values of their elds (attributes of the class we dened earlier) using the standard dict syntax like:
>>> item = DmozItem() >>> item[title] = Example title >>> item[title] Example title
Spiders are expected to return their scraped data inside Item objects. So, in order to return the data weve scraped so far, the nal code for our Spider would be like this:
from scrapy.spider import Spider from scrapy.selector import Selector from tutorial.items import DmozItem class DmozSpider(Spider): name = "dmoz" allowed_domains = ["dmoz.org"] start_urls = [ "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/", "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/" ] def parse(self, response):
15
sel = Selector(response) sites = sel.xpath(//ul/li) items = [] for site in sites: item = DmozItem() item[title] = site.xpath(a/text()).extract() item[link] = site.xpath(a/@href).extract() item[desc] = site.xpath(text()).extract() items.append(item) return items
Note: You can nd a fully-functional variant of this spider in the dirbot project available at https://github.com/scrapy/dirbot Now doing a crawl on the dmoz.org domain yields DmozItems:
[dmoz] DEBUG: Scraped from <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> {desc: [u - By David Mertz; Addison Wesley. Book in progress, full text, ASCII format. Asks f link: [uhttp://gnosis.cx/TPiP/], title: [uText Processing in Python]} [dmoz] DEBUG: Scraped from <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> {desc: [u - By Sean McGrath; Prentice Hall PTR, 2000, ISBN 0130211192, has CD-ROM. Methods to link: [uhttp://www.informit.com/store/product.aspx?isbn=0130211192], title: [uXML Processing with Python]}
That will generate a items.json le containing all scraped items, serialized in JSON. In small projects (like the one in this tutorial), that should be enough. However, if you want to perform more complex things with the scraped items, you can write an Item Pipeline. As with Items, a placeholder le for Item Pipelines has been set up for you when the project is created, in tutorial/pipelines.py. Though you dont need to implement any item pipeline if you just want to store the scraped items.
2.4 Examples
The best way to learn is with examples, and Scrapy is no exception. For this reason, there is an example Scrapy project named dirbot, that you can use to play and learn more about Scrapy. It contains the dmoz spider described in the tutorial. This dirbot project is available at: https://github.com/scrapy/dirbot
16
It contains a README le with a detailed description of the project contents. If youre familiar with git, you can checkout the code. Otherwise you can download a tarball or zip le of the project by clicking on Downloads. The scrapy tag on Snipplr is used for sharing code snippets such as spiders, middlewares, extensions, or scripts. Feel free (and encouraged!) to share any code there. Scrapy at a glance Understand what Scrapy is and how it can help you. Installation guide Get Scrapy installed on your computer. Scrapy Tutorial Write your rst Scrapy project. Examples Learn more by playing with a pre-made Scrapy project.
2.4. Examples
17
18
CHAPTER 3
Basic concepts
The directory where the scrapy.cfg le resides is known as the project root directory. That le contains the name of the python module that denes the project settings. Here is an example:
[settings] default = myproject.settings
19
Scrapy X.Y - no active project Usage: scrapy <command> [options] [args] Available commands: crawl Run a spider fetch Fetch a URL using the Scrapy downloader [...]
The rst line will print the currently active project, if youre inside a Scrapy project. In this, it was run from outside a project. If run from inside a project it would have printed something like this:
Scrapy X.Y - project: myproject Usage: scrapy <command> [options] [args] [...]
Creating projects The rst thing you typically do with the scrapy tool is create your Scrapy project:
scrapy startproject myproject
That will create a Scrapy project under the myproject directory. Next, you go inside the new project directory:
cd myproject
And youre ready to use the scrapy command to manage and control your project from there. Controlling projects You use the scrapy tool from inside your projects to control and manage them. For example, to create a new spider:
scrapy genspider mydomain mydomain.com
Some Scrapy commands (like crawl) must be run from inside a Scrapy project. See the commands reference below for more information on which commands must be run from inside projects, and which not. Also keep in mind that some commands may have slightly different behaviours when running them from inside projects. For example, the fetch command will use spider-overridden behaviours (such as the user_agent attribute to override the user-agent) if the url being fetched is associated with some specic spider. This is intentional, as the fetch command is meant to be used to check how spiders are downloading pages.
20
scrapy <command> -h
There are two kinds of commands, those that only work from inside a Scrapy project (Project-specic commands) and those that also work without an active Scrapy project (Global commands), though they may behave slightly different when running from inside a project (as they would use the project overridden settings). Global commands: startproject settings runspider shell fetch view version Project-only commands: crawl check list edit parse genspider deploy bench startproject Syntax: scrapy startproject <project_name> Requires project: no Creates a new Scrapy project named project_name, under the project_name directory. Usage example:
$ scrapy startproject myproject
genspider Syntax: scrapy genspider [-t template] <name> <domain> Requires project: yes
21
Create a new spider in the current project. This is just a convenient shortcut command for creating spiders based on pre-dened templates, but certainly not the only way to create spiders. You can just create the spider source code les yourself, instead of using this command. Usage example:
$ scrapy genspider -l Available templates: basic crawl csvfeed xmlfeed $ scrapy genspider -d basic from scrapy.spider import Spider class $classname(Spider): name = "$name" allowed_domains = ["$domain"] start_urls = ( http://www.$domain/, ) def parse(self, response): pass $ scrapy genspider -t basic example example.com Created spider example using template basic in module: mybot.spiders.example
crawl Syntax: scrapy crawl <spider> Requires project: yes Start crawling a spider. Usage examples:
$ scrapy crawl myspider [ ... myspider starts crawling ... ]
check Syntax: scrapy check [-l] <spider> Requires project: yes Run contract checks. Usage examples:
$ scrapy check -l first_spider * parse * parse_item second_spider
22
* parse * parse_item $ scrapy check [FAILED] first_spider:parse_item >>> RetailPricex field is missing [FAILED] first_spider:parse >>> Returned 92 requests, expected 0..4
list Syntax: scrapy list Requires project: yes List all available spiders in the current project. The output is one spider per line. Usage example:
$ scrapy list spider1 spider2
edit Syntax: scrapy edit <spider> Requires project: yes Edit the given spider using the editor dened in the EDITOR setting. This command is provided only as a convenient shortcut for the most common case, the developer is of course free to choose any tool or IDE to write and debug his spiders. Usage example:
$ scrapy edit spider1
fetch Syntax: scrapy fetch <url> Requires project: no Downloads the given URL using the Scrapy downloader and writes the contents to standard output. The interesting thing about this command is that it fetches the page how the the spider would download it. For example, if the spider has an USER_AGENT attribute which overrides the User Agent, it will use that one. So this command can be used to see how your spider would fetch certain page. If used outside a project, no particular per-spider behaviour would be applied and it will just use the default Scrapy downloder settings. Usage examples:
23
$ scrapy fetch --nolog http://www.example.com/some/page.html [ ... html content here ... ] $ scrapy fetch --nolog --headers http://www.example.com/ {Accept-Ranges: [bytes], Age: [1263 ], Connection: [close ], Content-Length: [596], Content-Type: [text/html; charset=UTF-8], Date: [Wed, 18 Aug 2010 23:59:46 GMT], Etag: ["573c1-254-48c9c87349680"], Last-Modified: [Fri, 30 Jul 2010 15:30:18 GMT], Server: [Apache/2.2.3 (CentOS)]}
view Syntax: scrapy view <url> Requires project: no Opens the given URL in a browser, as your Scrapy spider would see it. Sometimes spiders see pages differently from regular users, so this can be used to check what the spider sees and conrm its what you expect. Usage example:
$ scrapy view http://www.example.com/some/page.html [ ... browser starts ... ]
shell Syntax: scrapy shell [url] Requires project: no Starts the Scrapy shell for the given URL (if given) or empty if not URL is given. See Scrapy shell for more info. Usage example:
$ scrapy shell http://www.example.com/some/page.html [ ... scrapy shell starts ... ]
parse Syntax: scrapy parse <url> [options] Requires project: yes Fetches the given URL and parses with the spider that handles it, using the method passed with the --callback option, or parse if not given. Supported options: --callback or -c: spider method to use as callback for parsing the response --rules or -r: use CrawlSpider rules to discover the callback (ie. spider method) to use for parsing the response --noitems: dont show scraped items
24
--nolinks: dont show extracted links --depth or -d: depth level for which the requests should be followed recursively (default: 1) --verbose or -v: display information for each depth level Usage example:
$ scrapy parse http://www.example.com/ -c parse_item [ ... scrapy log lines crawling example.com spider ... ] >>> STATUS DEPTH LEVEL 1 <<< # Scraped Items -----------------------------------------------------------[{name: uExample item, category: uFurniture, length: u12 cm}] # Requests [] -----------------------------------------------------------------
settings Syntax: scrapy settings [options] Requires project: no Get the value of a Scrapy setting. If used inside a project itll show the project setting value, otherwise itll show the default Scrapy value for that setting. Example usage:
$ scrapy settings --get BOT_NAME scrapybot $ scrapy settings --get DOWNLOAD_DELAY 0
runspider Syntax: scrapy runspider <spider_file.py> Requires project: no Run a spider self-contained in a Python le, without having to create a project. Example usage:
$ scrapy runspider myspider.py [ ... spider starts crawling ... ]
version Syntax: scrapy version [-v] Requires project: no Prints the Scrapy version. If used with -v it also prints Python, Twisted and Platform info, which is useful for bug reports.
25
deploy New in version 0.11. Syntax: scrapy deploy [ <target:project> | -l <target> | -L ] Requires project: yes Deploy the project into a Scrapyd server. See Deploying your project. bench New in version 0.17. Syntax: scrapy bench Requires project: no Run quick benchmark test. Benchmarking.
3.2 Items
The main goal in scraping is to extract structured data from unstructured sources, typically, web pages. Scrapy provides the Item class for this purpose. Item objects are simple containers used to collect the scraped data. They provide a dictionary-like API with a convenient syntax for declaring their available elds.
26
Note: Those familiar with Django will notice that Scrapy Items are declared similar to Django Models, except that Scrapy Items are much simpler as there is no concept of different eld types.
You can specify any kind of metadata for each eld. There is no restriction on the values accepted by Field objects. For this same reason, there isnt a reference list of all available metadata keys. Each key dened in Field objects could be used by a different components, and only those components know about it. You can also dene and use any other Field key in your project too, for your own needs. The main goal of Field objects is to provide a way to dene all eld metadata in one place. Typically, those components whose behaviour depends on each eld use certain eld keys to congure that behaviour. You must refer to their documentation to see which metadata keys are used by each component. Its important to note that the Field objects used to declare the item do not stay assigned as class attributes. Instead, they can be accessed through the Item.fields attribute. And thats all you need to know about declaring items.
3.2. Items
27
... KeyError: lala >>> product.get(lala, unknown field) unknown field >>> name in product True # is name field populated?
# is last_updated populated?
Accessing all populated values To access all populated values, just use the typical dict API:
>>> product.keys() [price, name] >>> product.items() [(price, 1000), (name, Desktop PC)]
28
You can also extend eld metadata by using the previous eld metadata and appending more values, or changing existing values, like this:
class SpecificProduct(Product): name = Field(Product.fields[name], serializer=my_serializer)
That adds (or replaces) the serializer metadata key for the name eld, keeping all the previously existing metadata values.
3.3 Spiders
Spiders are classes which dene how a certain site (or group of sites) will be scraped, including how to perform the crawl (ie. follow links) and how to extract structured data from their pages (ie. scraping items). In other words, Spiders 3.3. Spiders 29
are the place where you dene the custom behaviour for crawling and parsing pages for a particular site (or, in some cases, group of sites). For spiders, the scraping cycle goes through something like this: 1. You start by generating the initial Requests to crawl the rst URLs, and specify a callback function to be called with the response downloaded from those requests. The rst requests to perform are obtained by calling the start_requests() method which (by default) generates Request for the URLs specied in the start_urls and the parse method as callback function for the Requests. 2. In the callback function, you parse the response (web page) and return either Item objects, Request objects, or an iterable of both. Those Requests will also contain a callback (maybe the same) and will then be downloaded by Scrapy and then their response handled by the specied callback. 3. In callback functions, you parse the page contents, typically using Selectors (but you can also use BeautifulSoup, lxml or whatever mechanism you prefer) and generate items with the parsed data. 4. Finally, the items returned from the spider will be typically persisted to a database (in some Item Pipeline) or written to a le using Feed exports. Even though this cycle applies (more or less) to any kind of spider, there are different kinds of default spiders bundled into Scrapy for different purposes. We will talk about those types here.
Spider arguments can also be passed through the Scrapyd schedule.json API. See Scrapyd documentation.
30
Spider class scrapy.spider.Spider This is the simplest spider, and the one from which every other spider must inherit from (either the ones that come bundled with Scrapy, or the ones that you write yourself). It doesnt provide any special functionality. It just requests the given start_urls/start_requests, and calls the spiders method parse for each of the resulting responses. name A string which denes the name for this spider. The spider name is how the spider is located (and instantiated) by Scrapy, so it must be unique. However, nothing prevents you from instantiating more than one instance of the same spider. This is the most important spider attribute and its required. If the spider scrapes a single domain, a common practice is to name the spider after the domain, or without the TLD. So, for example, a spider that crawls mywebsite.com would often be called mywebsite. allowed_domains An optional list of strings containing domains that this spider is allowed to crawl. Requests for URLs not belonging to the domain names specied in this list wont be followed if OffsiteMiddleware is enabled. start_urls A list of URLs where the spider will begin to crawl from, when no particular URLs are specied. So, the rst pages downloaded will be those listed here. The subsequent URLs will be generated successively from data contained in the start URLs. start_requests() This method must return an iterable with the rst Requests to crawl for this spider. This is the method called by Scrapy when the spider is opened for scraping when no particular URLs are specied. If particular URLs are specied, the make_requests_from_url() is used instead to create the Requests. This method is also called only once from Scrapy, so its safe to implement it as a generator. The default implementation uses make_requests_from_url() to generate Requests for each url in start_urls. If you want to change the Requests used to start scraping a domain, this is the method to override. For example, if you need to start by logging in using a POST request, you could do:
def start_requests(self): return [FormRequest("http://www.example.com/login", formdata={user: john, pass: secret}, callback=self.logged_in)] def logged_in(self, response): # here you would extract links to follow and return Requests for # each of them, with another callback pass
make_requests_from_url(url) A method that receives a URL and returns a Request object (or a list of Request objects) to scrape. This method is used to construct the initial requests in the start_requests() method, and is typically used to convert urls to requests.
3.3. Spiders
31
Unless overridden, this method returns Requests with the parse() method as their callback function, and with dont_lter parameter enabled (see Request class for more info). parse(response) This is the default callback used by Scrapy to process downloaded responses, when their requests dont specify a callback. The parse method is in charge of processing the response and returning scraped data and/or more URLs to follow. Other Requests callbacks have the same requirements as the Spider class. This method, as well as any other Request callback, must return an iterable of Request and/or Item objects. Parameters response (:class:~scrapy.http.Response) the response to parse log(message[, level, component ]) Log a message using the scrapy.log.msg() function, automatically populating the spider argument with the name of this spider. For more information see Logging.
Spider example
Another example returning multiples Requests and Items from a single callback:
from from from from scrapy.selector import Selector scrapy.spider import Spider scrapy.http import Request myproject.items import MyItem
class MySpider(Spider): name = example.com allowed_domains = [example.com] start_urls = [ http://www.example.com/1.html, http://www.example.com/2.html, http://www.example.com/3.html, ] def parse(self, response): sel = Selector(response) for h3 in sel.xpath(//h3).extract(): yield MyItem(title=h3)
32
CrawlSpider class scrapy.contrib.spiders.CrawlSpider This is the most commonly used spider for crawling regular websites, as it provides a convenient mechanism for following links by dening a set of rules. It may not be the best suited for your particular web sites or project, but its generic enough for several cases, so you can start from it and override it as needed for more custom functionality, or just implement your own spider. Apart from the attributes inherited from Spider (that you must specify), this class supports a new attribute: rules Which is a list of one (or more) Rule objects. Each Rule denes a certain behaviour for crawling the site. Rules objects are described below. If multiple rules match the same link, the rst one will be used, according to the order theyre dened in this attribute. This spider also exposes an overrideable method: parse_start_url(response) This method is called for the start_urls responses. It allows to parse the initial responses and must return either a Item object, a Request object, or an iterable containing any of them.
Crawling rules
class scrapy.contrib.spiders.Rule(link_extractor, callback=None, cb_kwargs=None, follow=None, process_links=None, process_request=None) link_extractor is a Link Extractor object which denes how links will be extracted from each crawled page. callback is a callable or a string (in which case a method from the spider object with that name will be used) to be called for each link extracted with the specied link_extractor. This callback receives a response as its rst argument and must return a list containing Item and/or Request objects (or any subclass of them). Warning: When writing crawl spider rules, avoid using parse as callback, since the CrawlSpider uses the parse method itself to implement its logic. So if you override the parse method, the crawl spider will no longer work. cb_kwargs is a dict containing the keyword arguments to be passed to the callback function follow is a boolean which species if links should be followed from each response extracted with this rule. If callback is None follow defaults to True, otherwise it default to False. process_links is a callable, or a string (in which case a method from the spider object with that name will be used) which will be called for each list of links extracted from each response using the specied link_extractor. This is mainly used for ltering purposes. process_request is a callable, or a string (in which case a method from the spider object with that name will be used) which will be called with every request extracted by this rule, and must return a request or None (to lter out the request).
CrawlSpider example
Lets now take a look at an example CrawlSpider with rules: 3.3. Spiders 33
scrapy.contrib.spiders import CrawlSpider, Rule scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor scrapy.selector import Selector scrapy.item import Item
class MySpider(CrawlSpider): name = example.com allowed_domains = [example.com] start_urls = [http://www.example.com] rules = ( # Extract links matching category.php (but not matching subsection.php) # and follow links from them (since no callback means follow=True by default). Rule(SgmlLinkExtractor(allow=(category\.php, ), deny=(subsection\.php, ))), # Extract links matching item.php and parse them with the spiders method parse_item Rule(SgmlLinkExtractor(allow=(item\.php, )), callback=parse_item), ) def parse_item(self, response): self.log(Hi, this is an item page! %s % response.url) sel = Selector(response) item = Item() item[id] = sel.xpath(//td[@id="item_id"]/text()).re(rID: (\d+)) item[name] = sel.xpath(//td[@id="item_name"]/text()).extract() item[description] = sel.xpath(//td[@id="item_description"]/text()).extract() return item
This spider would start crawling example.coms home page, collecting category links, and item links, parsing the latter with the parse_item method. For each item response, some data will be extracted from the HTML using XPath, and a Item will be lled with it. XMLFeedSpider class scrapy.contrib.spiders.XMLFeedSpider XMLFeedSpider is designed for parsing XML feeds by iterating through them by a certain node name. The iterator can be chosen from: iternodes, xml, and html. Its recommended to use the iternodes iterator for performance reasons, since the xml and html iterators generate the whole DOM at once in order to parse it. However, using html as the iterator may be useful when parsing XML with bad markup. To set the iterator and the tag name, you must dene the following class attributes: iterator A string which denes the iterator to use. It can be either: iternodes - a fast iterator based on regular expressions html - an iterator which uses Selector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem for big feeds xml - an iterator which uses Selector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem for big feeds It defaults to: iternodes. itertag A string with the name of the node (or element) to iterate in. Example:
34
itertag = product
namespaces A list of (prefix, uri) tuples which dene the namespaces available in that document that will be processed with this spider. The prefix and uri will be used to automatically register namespaces using the register_namespace() method. You can then specify nodes with namespaces in the itertag attribute. Example:
class YourSpider(XMLFeedSpider): namespaces = [(n, http://www.sitemaps.org/schemas/sitemap/0.9)] itertag = n:url # ...
Apart from these new attributes, this spider has the following overrideable methods too: adapt_response(response) A method that receives the response as soon as it arrives from the spider middleware, before the spider starts parsing it. It can be used to modify the response body before parsing it. This method receives a response and also returns a response (it could be the same or another one). parse_node(response, selector) This method is called for the nodes matching the provided tag name (itertag). Receives the response and an Selector for each node. Overriding this method is mandatory. Otherwise, you spider wont work. This method must return either a Item object, a Request object, or an iterable containing any of them. process_results(response, results) This method is called for each result (item or request) returned by the spider, and its intended to perform any last time processing required before returning the results to the framework core, for example setting the item IDs. It receives a list of results and the response which originated those results. It must return a list of results (Items or Requests).
XMLFeedSpider example
These spiders are pretty easy to use, lets have a look at one example:
from scrapy import log from scrapy.contrib.spiders import XMLFeedSpider from myproject.items import TestItem class MySpider(XMLFeedSpider): name = example.com allowed_domains = [example.com] start_urls = [http://www.example.com/feed.xml] iterator = iternodes # This is actually unnecessary, since its the default value itertag = item def parse_node(self, response, node): log.msg(Hi, this is a <%s> node!: %s % (self.itertag, .join(node.extract()))) item = Item() item[id] = node.xpath(@id).extract() item[name] = node.xpath(name).extract()
3.3. Spiders
35
Basically what we did up there was to create a spider that downloads a feed from the given start_urls, and then iterates through each of its item tags, prints them out, and stores some random data in an Item. CSVFeedSpider class scrapy.contrib.spiders.CSVFeedSpider This spider is very similar to the XMLFeedSpider, except that it iterates over rows, instead of nodes. The method that gets called in each iteration is parse_row(). delimiter A string with the separator character for each eld in the CSV le Defaults to , (comma). headers A list of the rows contained in the le CSV feed which will be used to extract elds from it. parse_row(response, row) Receives a response and a dict (representing each row) with a key for each provided (or detected) header of the CSV le. This spider also gives the opportunity to override adapt_response and process_results methods for pre- and post-processing purposes.
CSVFeedSpider example
Lets see an example similar to the previous one, but using a CSVFeedSpider:
from scrapy import log from scrapy.contrib.spiders import CSVFeedSpider from myproject.items import TestItem class MySpider(CSVFeedSpider): name = example.com allowed_domains = [example.com] start_urls = [http://www.example.com/feed.csv] delimiter = ; headers = [id, name, description] def parse_row(self, response, row): log.msg(Hi, this is a row!: %r % row) item = TestItem() item[id] = row[id] item[name] = row[name] item[description] = row[description] return item
SitemapSpider class scrapy.contrib.spiders.SitemapSpider SitemapSpider allows you to crawl a site by discovering the URLs using Sitemaps. It supports nested sitemaps and discovering sitemap urls from robots.txt. sitemap_urls A list of urls pointing to the sitemaps whose urls you want to crawl. 36 Chapter 3. Basic concepts
You can also point to a robots.txt and it will be parsed to extract sitemap urls from it. sitemap_rules A list of tuples (regex, callback) where: regex is a regular expression to match urls extracted from sitemaps. regex can be either a str or a compiled regex object. callback is the callback to use for processing the urls that match the regular expression. callback can be a string (indicating the name of a spider method) or a callable. For example:
sitemap_rules = [(/product/, parse_product)]
Rules are applied in order, and only the rst one that matches will be used. If you omit this attribute, all urls found in sitemaps will be processed with the parse callback. sitemap_follow A list of regexes of sitemap that should be followed. This is is only for sites that use Sitemap index les that point to other sitemap les. By default, all sitemaps are followed. sitemap_alternate_links Species if alternate links for one url should be followed. These are links for the same website in another language passed within the same url block. For example:
<url> <loc>http://example.com/</loc> <xhtml:link rel="alternate" hreflang="de" href="http://example.com/de"/> </url>
With sitemap_alternate_links set, this would retrieve both URLs. sitemap_alternate_links disabled, only http://example.com/ would be retrieved. Default is sitemap_alternate_links disabled.
SitemapSpider examples
With
Simplest example: process all urls discovered through sitemaps using the parse callback:
from scrapy.contrib.spiders import SitemapSpider class MySpider(SitemapSpider): sitemap_urls = [http://www.example.com/sitemap.xml] def parse(self, response): pass # ... scrape item here ...
Process some urls with certain callback and other urls with a different callback:
from scrapy.contrib.spiders import SitemapSpider class MySpider(SitemapSpider): sitemap_urls = [http://www.example.com/sitemap.xml] sitemap_rules = [ (/product/, parse_product),
3.3. Spiders
37
(/category/, parse_category), ] def parse_product(self, response): pass # ... scrape product ... def parse_category(self, response): pass # ... scrape category ...
Follow sitemaps dened in the robots.txt le and only follow sitemaps whose url contains /sitemap_shop:
from scrapy.contrib.spiders import SitemapSpider class MySpider(SitemapSpider): sitemap_urls = [http://www.example.com/robots.txt] sitemap_rules = [ (/shop/, parse_shop), ] sitemap_follow = [/sitemap_shops] def parse_shop(self, response): pass # ... scrape shop here ...
3.4 Selectors
When youre scraping web pages, the most common task you need to perform is to extract data from the HTML source. There are several libraries available to achieve this: BeautifulSoup is a very popular screen scraping library among Python programmers which constructs a Python object based on the structure of the HTML code and also deals with bad markup reasonably well, but it has one drawback: its slow. lxml is a XML parsing library (which also parses HTML) with a pythonic API based on ElementTree (which is not part of the Python standard library). 38 Chapter 3. Basic concepts
Scrapy comes with its own mechanism for extracting data. Theyre called selectors because they select certain parts of the HTML document specied either by XPath or CSS expressions. XPath is a language for selecting nodes in XML documents, which can also be used with HTML. CSS is a language for applying styles to HTML documents. It denes selectors to associate those styles with specic HTML elements. Scrapy selectors are built over the lxml library, which means theyre very similar in speed and parsing accuracy. This page explains how selectors work and describes their API which is very small and simple, unlike the lxml API which is much bigger because the lxml library can be used for many other tasks, besides selecting markup documents. For a complete reference of the selectors API see Selector reference
Using selectors To explain how to use the selectors well use the Scrapy shell (which provides interactive testing) and an example page located in the Scrapy documentation server: http://doc.scrapy.org/en/latest/_static/selectors-sample1.html Heres its HTML code:
<html> <head> <base href=http://example.com/ /> <title>Example website</title> </head> <body> <div id=images> <a href=image1.html>Name: My image <a href=image2.html>Name: My image <a href=image3.html>Name: My image <a href=image4.html>Name: My image <a href=image5.html>Name: My image </div> </body> </html>
1 2 3 4 5
3.4. Selectors
39
Then, after the shell loads, youll have a selector already instantiated and ready to use in sel shell variable. Since were dealing with HTML, the selector will automatically use an HTML parser. So, by looking at the HTML code of that page, lets construct an XPath (using an HTML selector) for selecting the text inside the title tag:
>>> sel.xpath(//title/text()) [<Selector (text) xpath=//title/text()>]
As you can see, the .xpath() method returns an SelectorList instance, which is a list of new selectors. This API can be used quickly for extracting nested data. To actually extract the textual data, you must call the selector .extract() method, as follows:
>>> sel.xpath(//title/text()).extract() [uExample website]
Notice that CSS selectors can select text or attribute nodes using CSS3 pseudo-elements:
>>> sel.css(title::text).extract() [uExample website]
Now were going to get the base URL and some image links:
>>> sel.xpath(//base/@href).extract() [uhttp://example.com/] >>> sel.css(base::attr(href)).extract() [uhttp://example.com/] >>> sel.xpath(//a[contains(@href, "image")]/@href).extract() [uimage1.html, uimage2.html, uimage3.html, uimage4.html, uimage5.html] >>> sel.css(a[href*=image]::attr(href)).extract() [uimage1.html, uimage2.html, uimage3.html, uimage4.html, uimage5.html] >>> sel.xpath(//a[contains(@href, "image")]/img/@src).extract() [uimage1_thumb.jpg, uimage2_thumb.jpg, uimage3_thumb.jpg, uimage4_thumb.jpg, uimage5_thumb.jpg] >>> sel.css(a[href*=image] img::attr(src)).extract() [uimage1_thumb.jpg, uimage2_thumb.jpg, uimage3_thumb.jpg,
40
uimage4_thumb.jpg, uimage5_thumb.jpg]
Nesting selectors The selection methods (.xpath() or .css()) returns a list of selectors of the same type, so you can call the selection methods for those selectors too. Heres an example:
>>> links = sel.xpath(//a[contains(@href, "image")]) >>> links.extract() [u<a href="image1.html">Name: My image 1 <br><img src="image1_thumb.jpg"></a>, u<a href="image2.html">Name: My image 2 <br><img src="image2_thumb.jpg"></a>, u<a href="image3.html">Name: My image 3 <br><img src="image3_thumb.jpg"></a>, u<a href="image4.html">Name: My image 4 <br><img src="image4_thumb.jpg"></a>, u<a href="image5.html">Name: My image 5 <br><img src="image5_thumb.jpg"></a>] >>> for index, link in enumerate(links): args = (index, link.xpath(@href).extract(), link.xpath(img/@src).extract()) print Link number %d points to url %s and image %s % args Link Link Link Link Link number number number number number 0 1 2 3 4 points points points points points to to to to to url url url url url [uimage1.html] [uimage2.html] [uimage3.html] [uimage4.html] [uimage5.html] and and and and and image image image image image [uimage1_thumb.jpg] [uimage2_thumb.jpg] [uimage3_thumb.jpg] [uimage4_thumb.jpg] [uimage5_thumb.jpg]
Using selectors with regular expressions Selector also have a .re() method for extracting data using regular expressions. However, unlike using .xpath() or .css() methods, .re() method returns a list of unicode strings. So you cant construct nested .re() calls. Heres an example used to extract images names from the HTML code above:
>>> sel.xpath(//a[contains(@href, "image")]/text()).re(rName:\s*(.*)) [uMy image 1, uMy image 2, uMy image 3, uMy image 4, uMy image 5]
Working with relative XPaths Keep in mind that if you are nesting selectors and use an XPath that starts with /, that XPath will be absolute to the document and not relative to the Selector youre calling it from. For example, suppose you want to extract all <p> elements inside <div> elements. First, you would get all <div> elements:
>>> divs = sel.xpath(//div)
At rst, you may be tempted to use the following approach, which is wrong, as it actually extracts all <p> elements from the document, not only those inside <div> elements:
3.4. Selectors
41
This is the proper way to do it (note the dot prexing the .//p XPath):
>>> for p in divs.xpath(.//p) >>> print p.extract() # extracts all <p> inside
For more details about relative XPaths see the Location Paths section in the XPath specication. Using EXSLT extensions Being built atop lxml, Scrapy selectors also support some EXSLT extensions and come with these pre-registered namespaces to use in XPath expressions: prex re set namespace http://exslt.org/regular-expressions http://exslt.org/sets usage regular expressions set manipulation
Regular expressions
The test() function for example can prove quite useful when XPaths starts-with() or contains() are not sufcient. Example selecting links in list item with a class attribute ending with a digit:
>>> doc = """ ... <div> ... <ul> ... <li class="item-0"><a href="link1.html">first item</a></li> ... <li class="item-1"><a href="link2.html">second item</a></li> ... <li class="item-inactive"><a href="link3.html">third item</a></li> ... <li class="item-1"><a href="link4.html">fourth item</a></li> ... <li class="item-0"><a href="link5.html">fifth item</a></li> ... </ul> ... </div> ... """ >>> sel = Selector(text=doc, type="html") >>> sel.xpath(//li//@href).extract() [ulink1.html, ulink2.html, ulink3.html, ulink4.html, ulink5.html] >>> sel.xpath(//li[re:test(@class, "item-\d$")]//@href).extract() [ulink1.html, ulink2.html, ulink4.html, ulink5.html] >>>
Warning: C library libxslt doesnt natively support EXSLT regular expressions so lxmls implementation uses hooks to Pythons re module. Thus, using regexp functions in your XPath expressions may add a small performance penalty.
42
Set operations
These can be handy for excluding parts of a document tree before extracting text elements for example. Example extracting microdata (sample content taken from http://schema.org/Product) with groups of itemscopes and corresponding itemprops:
>>> ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... >>> >>> doc = """ <div itemscope itemtype="http://schema.org/Product"> <span itemprop="name">Kenmore White 17" Microwave</span> <img src="kenmore-microwave-17in.jpg" alt=Kenmore 17" Microwave /> <div itemprop="aggregateRating" itemscope itemtype="http://schema.org/AggregateRating"> Rated <span itemprop="ratingValue">3.5</span>/5 based on <span itemprop="reviewCount">11</span> customer reviews </div> <div itemprop="offers" itemscope itemtype="http://schema.org/Offer"> <span itemprop="price">$55.00</span> <link itemprop="availability" href="http://schema.org/InStock" />In stock </div> Product description: <span itemprop="description">0.7 cubic feet countertop microwave. Has six preset cooking categories and convenience features like Add-A-Minute and Child Lock.</span> Customer reviews: <div itemprop="review" itemscope itemtype="http://schema.org/Review"> <span itemprop="name">Not a happy camper</span> by <span itemprop="author">Ellie</span>, <meta itemprop="datePublished" content="2011-04-01">April 1, 2011 <div itemprop="reviewRating" itemscope itemtype="http://schema.org/Rating"> <meta itemprop="worstRating" content = "1"> <span itemprop="ratingValue">1</span>/ <span itemprop="bestRating">5</span>stars </div> <span itemprop="description">The lamp burned out and now I have to replace it. </span> </div> <div itemprop="review" itemscope itemtype="http://schema.org/Review"> <span itemprop="name">Value purchase</span> by <span itemprop="author">Lucas</span>, <meta itemprop="datePublished" content="2011-03-25">March 25, 2011 <div itemprop="reviewRating" itemscope itemtype="http://schema.org/Rating"> <meta itemprop="worstRating" content = "1"/> <span itemprop="ratingValue">4</span>/ <span itemprop="bestRating">5</span>stars </div> <span itemprop="description">Great microwave for the price. It is small and fits in my apartment.</span> </div> ... </div> """ for scope in sel.xpath(//div[@itemscope]):
3.4. Selectors
43
... print "current scope:", scope.xpath(@itemtype).extract() ... props = scope.xpath( ... set:difference(./descendant::*/@itemprop, ... .//*[@itemscope]/*/@itemprop)) ... print " properties:", props.extract() ... print ... current scope: [uhttp://schema.org/Product] properties: [uname, uaggregateRating, uoffers, udescription, ureview, ureview] current scope: [uhttp://schema.org/AggregateRating] properties: [uratingValue, ureviewCount] current scope: [uhttp://schema.org/Offer] properties: [uprice, uavailability] current scope: [uhttp://schema.org/Review] properties: [uname, uauthor, udatePublished, ureviewRating, udescription] current scope: [uhttp://schema.org/Rating] properties: [uworstRating, uratingValue, ubestRating] current scope: [uhttp://schema.org/Review] properties: [uname, uauthor, udatePublished, ureviewRating, udescription] current scope: [uhttp://schema.org/Rating] properties: [uworstRating, uratingValue, ubestRating] >>>
Here we rst iterate over itemscope elements, and for each one, we look for all itemprops elements and exclude those that are themselves inside another itemscope.
44
xpath(query) Find nodes matching the xpath query and return the result as a SelectorList instance with all elements attened. List elements implement Selector interface too. query is a string containing the XPATH query to apply. css(query) Apply the given CSS selector and return a SelectorList instance. query is a string containing the CSS selector to apply. In the background, CSS queries are translated into XPath queries using cssselect library and run .xpath() method. extract() Serialize and return the matched nodes as a list of unicode strings. Percent encoded content is unquoted. re(regex) Apply the given regex and return a list of unicode strings with the matches. regex can be either a compiled regular expression or a string which will be compiled to a regular expression using re.compile(regex) register_namespace(prex, uri) Register the given namespace to be used in this Selector. Without registering namespaces you cant select or extract data from non-standard namespaces. See examples below. remove_namespaces() Remove all namespaces, allowing to traverse the document using namespace-less xpaths. See example below. __nonzero__() Returns True if there is any real content selected or False otherwise. In other words, the boolean value of a Selector is given by the contents it selects. SelectorList objects class scrapy.selector.SelectorList The SelectorList class is subclass of the builtin list class, which provides a few additional methods. xpath(query) Call the .xpath() method for each element in this list and return their results attened as another SelectorList. query is the same argument as the one in Selector.xpath() css(query) Call the .css() method for each element in this list and return their results attened as another SelectorList. query is the same argument as the one in Selector.css() extract() Call the .extract() method for each element is this list and return their results attened, as a list of unicode strings. re() Call the .re() method for each element is this list and return their results attened, as a list of unicode strings. __nonzero__() returns True if the list is not empty, False otherwise. 3.4. Selectors 45
Heres a couple of Selector examples to illustrate several concepts. In all cases, we assume there is already an Selector instantiated with a HtmlResponse object like this:
sel = Selector(html_response)
1. Select all <h1> elements from a HTML response body, returning a list of Selector objects (ie. SelectorList object):
sel.xpath("//h1")
2. Extract the text of all <h1> elements from a HTML response body, returning a list of unicode strings:
sel.xpath("//h1").extract() sel.xpath("//h1/text()").extract() # this includes the h1 tag # this excludes the h1 tag
3. Iterate over all <p> tags and print their class attribute:
for node in sel.xpath("//p"): ... print node.xpath("@class").extract()
Heres a couple of examples to illustrate several concepts. In both cases we assume there is already an Selector instantiated with a XmlResponse object like this:
sel = Selector(xml_response)
1. Select all <product> elements from a XML response body, returning a list of Selector objects (ie. a SelectorList object):
sel.xpath("//product")
2. Extract all prices from a Google Base XML feed which requires registering a namespace:
sel.register_namespace("g", "http://base.google.com/ns/1.0") sel.xpath("//g:price").extract()
Removing namespaces
When dealing with scraping projects, it is often quite convenient to get rid of namespaces altogether and just work with element names, to write more simple/convenient XPaths. You can use the Selector.remove_namespaces() method for that. Lets show an example that illustrates this with Github blog atom feed. First, we open the shell with the url we want to scrape:
$ scrapy shell https://github.com/blog.atom
Once in the shell we can try selecting all <link> objects and see that it doesnt work (because the Atom XML namespace is obfuscating those nodes):
>>> sel.xpath("//link") []
46
But once we call the Selector.remove_namespaces() method, all nodes can be accessed directly by their names:
>>> sel.remove_namespaces() >>> sel.xpath("//link") [<Selector xpath=//link data=u<link xmlns="http://www.w3.org/2005/Atom>, <Selector xpath=//link data=u<link xmlns="http://www.w3.org/2005/Atom>, ...
If you wonder why the namespace removal procedure is not always called, instead of having to call it manually. This is because of two reasons which, in order of relevance, are: 1. Removing namespaces requires to iterate and modify all nodes in the document, which is a reasonably expensive operation to performs for all documents crawled by Scrapy 2. There could be some cases where using namespaces is actually required, in case some element names clash between namespaces. These cases are very rare though.
By quickly looking at that code, we can see the name eld is being extracted from two different XPath locations in the page: 1. //div[@class="product_name"]
47
2. //div[@class="product_title"] In other words, data is being collected by extracting it from two XPath locations, using the add_xpath() method. This is the data that will be assigned to the name eld later. Afterwords, similar calls are used for price and stock elds (the later using a CSS selector with the add_css() method), and nally the last_update eld is populated directly with a literal value (today) using a different method: add_value(). Finally, when all data is collected, the ItemLoader.load_item() method is called which actually populates and returns the item populated with the data previously extracted and collected with the add_xpath(), add_css(), and add_value() calls.
So what happens is: 1. Data from xpath1 is extracted, and passed through the input processor of the name eld. The result of the input processor is collected and kept in the Item Loader (but not yet assigned to the item). 2. Data from xpath2 is extracted, and passed through the same input processor used in (1). The result of the input processor is appended to the data collected in (1) (if any). 3. This case is similar to the previous ones, except that the data is extracted from the css CSS selector, and passed through the same input processor used in (1) and (2). The result of the input processor is appended to the data collected in (1) and (2) (if any). 4. This case is also similar to the previous ones, except that the value to be collected is assigned directly, instead of being extracted from a XPath expression or a CSS selector. However, the value is still passed through the input processors. In this case, since the value is not iterable it is converted to an iterable of a single element before passing it to the input processor, because input processor always receive iterables. 5. The data collected in steps (1), (2), (3) and (4) is passed through the output processor of the name eld. The result of the output processor is the value assigned to the name eld in the item. Its worth noticing that processors are just callable objects, which are called with the data to be parsed, and return a parsed value. So you can use any function as input or output processor. The only requirement is that they must accept one (and only one) positional argument, which will be an iterator. Note: Both input and output processors must receive an iterator as their rst argument. The output of those functions can be anything. The result of input processors will be appended to an internal list (in the Loader) containing the collected values (for that eld). The result of the output processors is the value that will be nally assigned to the item.
48
The other thing you need to keep in mind is that the values returned by input processors are collected internally (in lists) and then passed to output processors to populate the elds. Last, but not least, Scrapy comes with some commonly used processors built-in for convenience.
As you can see, input processors are declared using the _in sufx while output processors are declared using the _out sufx. And you can also declare a default input/output processors using the ItemLoader.default_input_processor and ItemLoader.default_output_processor attributes.
The precedence order, for both input and output processors, is as follows: 1. Item Loader eld-specic attributes: field_in and field_out (most precedence) 2. Field metadata (input_processor and output_processor key) 3.5. Item Loaders 49
3. Item Loader defaults: ItemLoader.default_input_processor() ItemLoader.default_output_processor() (least precedence) See also: Reusing and extending Item Loaders.
and
By accepting a loader_context argument the function is explicitly telling the Item Loader that is able to receive an Item Loader context, so the Item Loader passes the currently active context when calling it, and the processor function (parse_length in this case) can thus use them. There are several ways to modify Item Loader context values: 1. By modifying the currently active Item Loader context (context attribute):
loader = ItemLoader(product) loader.context[unit] = cm
2. On Item Loader instantiation (the keyword arguments of Item Loader constructor are stored in the Item Loader context):
loader = ItemLoader(product, unit=cm)
3. On Item Loader declaration, for those input/output processors that support instatiating them with a Item Loader context. MapCompose is one of them:
class ProductLoader(ItemLoader): length_out = MapCompose(parse_length, unit=cm)
50
response (Response object) The response used to construct the selector using the default_selector_class, unless the selector argument is given, in which case this argument is ignored. The item, selector, response and the remaining keyword arguments are assigned to the Loader context (accessible through the context attribute). ItemLoader instances have the following methods: get_value(value, *processors, **kwargs) Process the given value by the given processors and keyword arguments. Available keyword arguments: Parameters re (str or compiled regex) a regular expression to use for extracting data from the given value using extract_regex() method, applied before processors Examples:
>>> from scrapy.contrib.loader.processor import TakeFirst >>> loader.get_value(uname: foo, TakeFirst(), unicode.upper, re=name: (.+)) FOO
add_value(eld_name, value, *processors, **kwargs) Process and then add the given value for the given eld. The value is rst passed through get_value() by giving the processors and kwargs, and then passed through the eld input processor and its result appended to the data collected for that eld. If the eld already contains collected data, the new data is added. The given field_name can be None, in which case values for multiple elds may be added. And the processed value should be a dict with eld_name mapped to values. Examples:
loader.add_value(name, uColor TV) loader.add_value(colours, [uwhite, ublue]) loader.add_value(length, u100) loader.add_value(name, uname: foo, TakeFirst(), re=name: (.+)) loader.add_value(None, {name: ufoo, sex: umale})
replace_value(eld_name, value) Similar to add_value() but replaces the collected data with the new value instead of adding it. get_xpath(xpath, *processors, **kwargs) Similar to ItemLoader.get_value() but receives an XPath instead of a value, which is used to extract a list of unicode strings from the selector associated with this ItemLoader. Parameters xpath (str) the XPath to extract data from re (str or compiled regex) a regular expression to use for extracting data from the selected XPath region Examples:
# HTML snippet: <p class="product-name">Color TV</p> loader.get_xpath(//p[@class="product-name"]) # HTML snippet: <p id="price">the price is $1200</p> loader.get_xpath(//p[@id="price"], TakeFirst(), re=the price is (.*))
51
add_xpath(eld_name, xpath, *processors, **kwargs) Similar to ItemLoader.add_value() but receives an XPath instead of a value, which is used to extract a list of unicode strings from the selector associated with this ItemLoader. See get_xpath() for kwargs. Parameters xpath (str) the XPath to extract data from Examples:
# HTML snippet: <p class="product-name">Color TV</p> loader.add_xpath(name, //p[@class="product-name"]) # HTML snippet: <p id="price">the price is $1200</p> loader.add_xpath(price, //p[@id="price"], re=the price is (.*))
replace_xpath(eld_name, xpath, *processors, **kwargs) Similar to add_xpath() but replaces collected data instead of adding it. get_css(css, *processors, **kwargs) Similar to ItemLoader.get_value() but receives a CSS selector instead of a value, which is used to extract a list of unicode strings from the selector associated with this ItemLoader. Parameters css (str) the CSS selector to extract data from re (str or compiled regex) a regular expression to use for extracting data from the selected CSS region Examples:
# HTML snippet: <p class="product-name">Color TV</p> loader.get_css(p.product-name) # HTML snippet: <p id="price">the price is $1200</p> loader.get_css(p#price, TakeFirst(), re=the price is (.*))
add_css(eld_name, css, *processors, **kwargs) Similar to ItemLoader.add_value() but receives a CSS selector instead of a value, which is used to extract a list of unicode strings from the selector associated with this ItemLoader. See get_css() for kwargs. Parameters css (str) the CSS selector to extract data from Examples:
# HTML snippet: <p class="product-name">Color TV</p> loader.add_css(name, p.product-name) # HTML snippet: <p id="price">the price is $1200</p> loader.add_css(price, p#price, re=the price is (.*))
replace_css(eld_name, css, *processors, **kwargs) Similar to add_css() but replaces collected data instead of adding it. load_item() Populate the item with the data collected so far, and return it. The data collected is rst passed through the output processors to get the nal value to assign to each item eld. get_collected_values(eld_name) Return the collected values for the given eld. get_output_value(eld_name) Return the collected values parsed using the output processor, for the given eld. This method doesnt populate or modify the item at all. 52 Chapter 3. Basic concepts
get_input_processor(eld_name) Return the input processor for the given eld. get_output_processor(eld_name) Return the output processor for the given eld. ItemLoader instances have the following attributes: item The Item object being parsed by this Item Loader. context The currently active Context of this Item Loader. default_item_class An Item class (or factory), used to instantiate items when not given in the constructor. default_input_processor The default input processor to use for those elds which dont specify one. default_output_processor The default output processor to use for those elds which dont specify one. default_selector_class The class used to construct the selector of this ItemLoader, if only a response is given in the constructor. If a selector is given in the constructor this attribute is ignored. This attribute is sometimes overridden in subclasses. selector The Selector object to extract data from. Its either the selector given in the constructor or one created from the response given in the constructor using the default_selector_class. This attribute is meant to be read-only.
53
Another case where extending Item Loaders can be very helpful is when you have multiple source formats, for example XML and HTML. In the XML version you may want to remove CDATA occurrences. Heres an example of how to do it:
from scrapy.contrib.loader.processor import MapCompose from myproject.ItemLoaders import ProductLoader from myproject.utils.xml import remove_cdata class XmlProductLoader(ProductLoader): name_in = MapCompose(remove_cdata, ProductLoader.name_in)
And thats how you typically extend input processors. As for output processors, it is more common to declare them in the eld metadata, as they usually depend only on the eld and not on each specic site parsing rule (as input processors do). See also: Declaring Input and Output Processors. There are many other possible ways to extend, inherit and override your Item Loaders, and different Item Loaders hierarchies may t better for different projects. Scrapy only provides the mechanism; it doesnt impose any specic organization of your Loaders collection - thats up to you and your projects needs.
class scrapy.contrib.loader.processor.TakeFirst Return the rst non-null/non-empty value from the values received, so its typically used as an output processor to single-valued elds. It doesnt receive any constructor arguments, nor accept Loader contexts. Example:
>>> from scrapy.contrib.loader.processor import TakeFirst >>> proc = TakeFirst() >>> proc([, one, two, three]) one
class scrapy.contrib.loader.processor.Join(separator=u ) Returns the values joined with the separator given in the constructor, which defaults to u . It doesnt accept Loader contexts. When using the default separator, this processor is equivalent to the function: u .join Examples:
54
>>> from scrapy.contrib.loader.processor import Join >>> proc = Join() >>> proc([one, two, three]) uone two three >>> proc = Join(<br>) >>> proc([one, two, three]) uone<br>two<br>three
class scrapy.contrib.loader.processor.Compose(*functions, **default_loader_context) A processor which is constructed from the composition of the given functions. This means that each input value of this processor is passed to the rst function, and the result of that function is passed to the second function, and so on, until the last function returns the output value of this processor. By default, stop process on None value. This behaviour can be changed by passing keyword argument stop_on_none=False. Example:
>>> from scrapy.contrib.loader.processor import Compose >>> proc = Compose(lambda v: v[0], str.upper) >>> proc([hello, world]) HELLO
Each function can optionally receive a loader_context parameter. For those which do, this processor will pass the currently active Loader context through that parameter. The keyword arguments passed in the constructor are used as the default Loader context values passed to each function call. However, the nal Loader context values passed to functions are overridden with the currently active Loader context accessible through the ItemLoader.context() attribute. class scrapy.contrib.loader.processor.MapCompose(*functions, **default_loader_context) A processor which is constructed from the composition of the given functions, similar to the Compose processor. The difference with this processor is the way internal results are passed among functions, which is as follows: The input value of this processor is iterated and each element is passed to the rst function, and the result of that function (for each element) is concatenated to construct a new iterable, which is then passed to the second function, and so on, until the last function is applied for each value of the list of values collected so far. The output values of the last function are concatenated together to produce the output of this processor. Each particular function can return a value or a list of values, which is attened with the list of values returned by the same function applied to the other input values. The functions can also return None in which case the output of that function is ignored for further processing over the chain. This processor provides a convenient way to compose functions that only work with single values (instead of iterables). For this reason the MapCompose processor is typically used as input processor, since data is often extracted using the extract() method of selectors, which returns a list of unicode strings. The example below should clarify how it works:
>>> def filter_world(x): ... return None if x == world else x ... >>> from scrapy.contrib.loader.processor import MapCompose >>> proc = MapCompose(filter_world, unicode.upper) >>> proc([uhello, uworld, uthis, uis, uscrapy]) [uHELLO, uTHIS, uIS, uSCRAPY]
As with the Compose processor, functions can receive Loader contexts, and constructor keyword arguments are used as default context values. See Compose processor for more info.
55
56
request - a Request object of the last fetched page. You can modify this request using replace() or fetch a new request (without leaving the shell) using the fetch shortcut. response - a Response object containing the last fetched page sel - a Selector object constructed with the last response fetched settings - the current Scrapy settings
Then, the shell fetches the URL (using the Scrapy downloader) and prints the list of available objects and useful shortcuts (youll notice that these lines all start with the [s] prex):
[s] Available Scrapy objects: [s] crawler <scrapy.crawler.Crawler object at 0x1e16b50> [s] item {} [s] request <GET http://scrapy.org> [s] response <200 http://scrapy.org> [s] sel <Selector xpath=None data=u<html>\n <head>\n <meta charset="utf-8> [s] settings <CrawlerSettings module=None> [s] spider <Spider default at 0x20c6f50> [s] Useful shortcuts: [s] shelp() Shell help (print this help) [s] fetch(req_or_url) Fetch request (or URL) and update local objects [s] view(response) View response in a browser >>>
57
>>> sel.xpath(//title/text()).extract() [uSlashdot: News for nerds, stuff that matters] >>> request = request.replace(method="POST") >>> fetch(request) [s] Available Scrapy objects: [s] crawler <scrapy.crawler.Crawler object at 0x1e16b50> ... >>>
class MySpider(Spider): name = "myspider" start_urls = [ "http://example.com", "http://example.org", "http://example.net", ] def parse(self, response): # We want to inspect one specific response. if ".org" in response.url: from scrapy.shell import inspect_response inspect_response(response) # Rest of parsing code.
When you run the spider, you will get something similar to this:
2014-01-23 17:48:31-0400 [myspider] DEBUG: Crawled (200) <GET http://example.com> (referer: None) 2014-01-23 17:48:31-0400 [myspider] DEBUG: Crawled (200) <GET http://example.org> (referer: None) [s] Available Scrapy objects: [s] crawler <scrapy.crawler.Crawler object at 0x1e16b50> ... >>> response.url http://example.org
Nope, it doesnt. So you can open the response in your web browser and see if its the response you were expecting:
58
Finally you hit Ctrl-D (or Ctrl-Z in Windows) to exit the shell and resume the crawling:
>>> ^D 2014-01-23 17:50:03-0400 [myspider] DEBUG: Crawled (200) <GET http://example.net> (referer: None) ...
Note that you cant use the fetch shortcut here since the Scrapy engine is blocked by the shell. However, after you leave the shell, the spider will continue crawling where it stopped, as shown above.
59
Write items to a JSON le The following pipeline stores all scraped items (from all spiders) into a a single items.jl le, containing one item per line serialized in JSON format:
import json class JsonWriterPipeline(object): def __init__(self): self.file = open(items.jl, wb) def process_item(self, item, spider): line = json.dumps(dict(item)) + "\n" self.file.write(line) return item
Note: The purpose of JsonWriterPipeline is just to introduce how to write item pipelines. If you really want to store all scraped items into a JSON le you should use the Feed exports.
Duplicates lter A lter that looks for duplicate items, and drops those items that were already processed. Let say that our items have an unique id, but our spider returns multiples items with the same id:
from scrapy.exceptions import DropItem class DuplicatesPipeline(object): def __init__(self): self.ids_seen = set() def process_item(self, item, spider): if item[id] in self.ids_seen:
60
raise DropItem("Duplicate item found: %s" % item) else: self.ids_seen.add(item[id]) return item
The integer values you assign to classes in this setting determine the order they run in- items go through pipelines from order number low to high. Its customary to dene these numbers in the 0-1000 range.
61
JSON lines FEED_FORMAT: jsonlines Exporter used: JsonLinesItemExporter CSV FEED_FORMAT: csv Exporter used: CsvItemExporter XML FEED_FORMAT: xml Exporter used: XmlItemExporter Pickle FEED_FORMAT: pickle Exporter used: PickleItemExporter Marshal FEED_FORMAT: marshal Exporter used: MarshalItemExporter
3.8.2 Storages
When using the feed exports you dene where to store the feed using a URI (through the FEED_URI setting). The feed exports supports multiple storage backend types which are dened by the URI scheme. The storages backends supported out of the box are: Local lesystem FTP S3 (requires boto) Standard output Some storage backends may be unavailable if the required external libraries are not available. For example, the S3 backend is only available if the boto library is installed.
62
Any other named parameter gets replaced by the spider attribute of the same name. For example, %(site_id)s would get replaced by the spider.site_id attribute the moment the feed is being created. Here are some examples to illustrate: Store in FTP using one directory per spider: ftp://user:password@ftp.example.com/scraping/feeds/%(name)s/%(time)s.json Store in S3 using one directory per spider: s3://mybucket/scraping/feeds/%(name)s/%(time)s.json
63
Standard output The feeds are written to the standard output of the Scrapy process. URI scheme: stdout Example URI: stdout: Required external libraries: none
3.8.5 Settings
These are the settings used for conguring the feed exports: FEED_URI (mandatory) FEED_FORMAT FEED_STORAGES FEED_EXPORTERS FEED_STORE_EMPTY FEED_URI Default: None The URI of the export feed. See Storage backends for supported URI schemes. This setting is required for enabling the feed exports. FEED_FORMAT The serialization format to be used for the feed. See Serialization formats for possible values. FEED_STORE_EMPTY Default: False Whether to export empty feeds (ie. feeds with no items). FEED_STORAGES Default:: {} A dict containing additional feed storage backends supported by your project. The keys are URI schemes and the values are paths to storage classes. FEED_STORAGES_BASE Default:
64
A dict containing the built-in feed storage backends supported by Scrapy. FEED_EXPORTERS Default:: {} A dict containing additional exporters supported by your project. The keys are URI schemes and the values are paths to Item exporter classes. FEED_EXPORTERS_BASE Default:
FEED_EXPORTERS_BASE = { json: scrapy.contrib.exporter.JsonItemExporter, jsonlines: scrapy.contrib.exporter.JsonLinesItemExporter, csv: scrapy.contrib.exporter.CsvItemExporter, xml: scrapy.contrib.exporter.XmlItemExporter, marshal: scrapy.contrib.exporter.MarshalItemExporter, }
65
SgmlLinkExtractor class scrapy.contrib.linkextractors.sgml.SgmlLinkExtractor(allow=(), deny=(), allow_domains=(), deny_domains=(), deny_extensions=None, restrict_xpaths=(), tags=(a, area), attrs=(href), canonicalize=True, unique=True, process_value=None) The SgmlLinkExtractor extends the base BaseSgmlLinkExtractor by providing additional lters that you can specify to extract links, including regular expressions patterns that the links must match to be extracted. All those lters are congured through these constructor parameters: Parameters allow (a regular expression (or list of)) a single regular expression (or list of regular expressions) that the (absolute) urls must match in order to be extracted. If not given (or empty), it will match all links. deny (a regular expression (or list of)) a single regular expression (or list of regular expressions) that the (absolute) urls must match in order to be excluded (ie. not extracted). It has precedence over the allow parameter. If not given (or empty) it wont exclude any links. allow_domains (str or list) a single value or a list of string containing domains which will be considered for extracting the links deny_domains (str or list) a single value or a list of strings containing domains which wont be considered for extracting the links deny_extensions (list) a list of extensions that should be ignored when extracting links. If not given, it will default to the IGNORED_EXTENSIONS list dened in the scrapy.linkextractor module. restrict_xpaths (str or list) is a XPath (or list of XPaths) which denes regions inside the response where links should be extracted from. If given, only the text selected by those XPath will be scanned for links. See examples below. tags (str or list) a tag or a list of tags to consider when extracting links. Defaults to (a, area). attrs (list) list of attributes which should be considered when looking for links to extract (only for those tags specied in the tags parameter). Defaults to (href,) canonicalize (boolean) canonicalize each scrapy.utils.url.canonicalize_url). Defaults to True. extracted url (using
unique (boolean) whether duplicate ltering should be applied to extracted links. process_value (callable) see process_value BaseSgmlLinkExtractor class constructor argument of
66
BaseSgmlLinkExtractor class scrapy.contrib.linkextractors.sgml.BaseSgmlLinkExtractor(tag=a, attr=href, unique=False, process_value=None) The purpose of this Link Extractor is only to serve as a base class for the SgmlLinkExtractor. You should use that one instead. The constructor arguments are: Parameters tag (str or callable) either a string (with the name of a tag) or a function that receives a tag name and returns True if links should be extracted from that tag, or False if they shouldnt. Defaults to a. request (once its downloaded) as its rst parameter. For more information, see Passing additional data to callback functions. attr (str or callable) either string (with the name of a tag attribute), or a function that receives an attribute name and returns True if links should be extracted from it, or False if they shouldnt. Defaults to href. unique (boolean) is a boolean that species if a duplicate ltering should be applied to links extracted. process_value (callable) a function which receives each value extracted from the tag and attributes scanned and can modify the value and return a new one, or return None to ignore the link altogether. If not given, process_value defaults to lambda x: x. For example, to extract links from this code:
<a href="javascript:goToPage(../other/page.html); return false">Link text</a>
Command line tool Learn about the command-line tool used to manage your Scrapy project. Items Dene the data you want to scrape. Spiders Write the rules to crawl your websites. Selectors Extract the data from web pages using XPath. Scrapy shell Test your extraction code in an interactive environment. Item Loaders Populate your items with the extracted data. Item Pipeline Post-process and store your scraped data. Feed exports Output your scraped data using different formats and storages. Link Extractors Convenient classes to extract links to follow from pages.
67
68
CHAPTER 4
Built-in services
4.1 Logging
Scrapy provides a logging facility which can be used through the scrapy.log module. The current underlying implementation uses Twisted logging but this may change in the future. The logging service must be explicitly started through the scrapy.log.start() function.
69
70
71
Or you can instantiate it passing a Scrapy settings object, which will respect the settings:
mailer = MailSender.from_settings(settings)
class scrapy.mail.MailSender(smtphost=None, mailfrom=None, smtpuser=None, smtppass=None, smtpport=None) Parameters smtphost (str) the SMTP host to use for sending the emails. If omitted, the MAIL_HOST setting will be used. mailfrom (str) the address used to send emails (in the From: header). If omitted, the MAIL_FROM setting will be used. smtpuser the SMTP user. If omitted, the MAIL_USER setting will be used. If not given, no SMTP authentication will be performed. smtppass (str) the SMTP pass for authentication. smtpport (boolean) the SMTP port to connect to smtptls enforce using SMTP STARTTLS smtpssl enforce using a secure SSL connection classmethod from_settings(settings) Instantiate using a Scrapy settings object, which will respect these Scrapy settings. Parameters settings (scrapy.settings.Settings object) the e-mail recipients send(to, subject, body, cc=None, attachs=()) Send email to the given recipients. Parameters to (list) the e-mail recipients subject (str) the subject of the e-mail cc (list) the e-mails to CC body (str) the e-mail body attachs (iterable) an iterable of tuples (attach_name, mimetype, file_object) where attach_name is a string with the name that will appear on the e-mails attachment, mimetype is the mimetype of the attachment and file_object is a readable le object with the contents of the attachment
73
MAIL_PORT Default: 25 SMTP port to use for sending emails. MAIL_USER Default: None User to use for SMTP authentication. If disabled no SMTP authentication will be performed. MAIL_PASS Default: None Password to use for SMTP authentication, along with MAIL_USER. MAIL_TLS Default: False Enforce using STARTTLS. STARTTLS is a way to take an existing insecure connection, and upgrade it to a secure connection using SSL/TLS. MAIL_SSL Default: False Enforce connecting using an SSL encrypted connection
You need the telnet program which comes installed by default in Windows, and most Linux distros.
74
: : : : : : : : : :
75
To resume:
telnet localhost 6023 >>> engine.unpause() >>>
To stop:
telnet localhost 6023 >>> engine.stop() Connection closed by foreign host.
class scrapy.contrib.webservice.crawler.CrawlerResource Provides access to the main Crawler object that controls the Scrapy process. Available by default at: http://localhost:6080/crawler
Stats Collector JSON-RPC resource
class scrapy.contrib.webservice.stats.StatsResource Provides access to the Stats Collector used by the crawler. Available by default at: http://localhost:6080/stats
Spider Manager JSON-RPC resource
You can access the spider manager JSON-RPC resource through the Crawler JSON-RPC resource at: http://localhost:6080/crawler/spiders
Extension Manager JSON-RPC resource
You can access the extension manager JSON-RPC resource through the Crawler JSON-RPC resource at: http://localhost:6080/crawler/spiders Available JSON resources These are the JSON resources available by default:
Engine status JSON resource
class scrapy.contrib.webservice.enginestatus.EngineStatusResource Provides access to engine status metrics. Available by default at: http://localhost:6080/enginestatus
77
The list of web service resources available by default in Scrapy. You shouldnt change this setting in your project, change WEBSERVICE_RESOURCES instead. If you want to disable some resource set its value to None in WEBSERVICE_RESOURCES.
78
79
if self._spider_name is None: return status for sp, st in status[spiders].items(): if sp.name == self._spider_name: return st def getChild(self, name, txrequest): return EngineStatusResource(name, self.crawler)
80
print(x) def cmd_list_available(args, opts): """list-available - list name of available spiders""" for x in jsonrpc_call(opts, crawler/spiders, list): print(x) def cmd_list_resources(args, opts): """list-resources - list available web service resources""" for x in json_get(opts, )[resources]: print(x) def cmd_get_spider_stats(args, opts): """get-spider-stats <spider> - get stats of a running spider""" stats = jsonrpc_call(opts, stats, get_stats, args[0]) for name, value in stats.items(): print("%-40s %s" % (name, value)) def cmd_get_global_stats(args, opts): """get-global-stats - get global stats""" stats = jsonrpc_call(opts, stats, get_stats) for name, value in stats.items(): print("%-40s %s" % (name, value)) def get_wsurl(opts, path): return urljoin("http://%s:%s/"% (opts.host, opts.port), path) def jsonrpc_call(opts, path, method, *args, **kwargs): url = get_wsurl(opts, path) return jsonrpc_client_call(url, method, *args, **kwargs) def json_get(opts, path): url = get_wsurl(opts, path) return json.loads(urllib.urlopen(url).read()) def parse_opts(): usage = "%prog [options] <command> [arg] ..." description = "Scrapy web service control script. Use %prog help " \ "to see the list of available commands." op = optparse.OptionParser(usage=usage, description=description) op.add_option("-H", dest="host", default="localhost", \ help="Scrapy host to connect to") op.add_option("-P", dest="port", type="int", default=6080, \ help="Scrapy port to connect to") opts, args = op.parse_args() if not args: op.print_help() sys.exit(2) cmdname, cmdargs, opts = args[0], args[1:], opts commands = get_commands() if cmdname not in commands: sys.stderr.write("Unknown command: %s\n\n" % cmdname) cmd_help(None, None) sys.exit(1) return commands[cmdname], cmdargs, opts def main(): cmd, args, opts = parse_opts()
81
try: cmd(args, opts) except IndexError: print(cmd.__doc__) except JsonRpcError as e: print(str(e)) if e.data: print("Server Traceback below:") print(e.data)
Logging Understand the simple logging facility provided by Scrapy. Stats Collection Collect statistics about your scraping crawler. Sending e-mail Send email notications when certain events occur. Telnet Console Inspect a running crawler using a built-in Python console. Web Service Monitor and control a crawler using a web service.
82
CHAPTER 5
83
5.1.13 Why does Scrapy download pages in English instead of my native language?
Try changing the default Accept-Language request header by overriding the DEFAULT_REQUEST_HEADERS setting.
84
Or by setting a global download delay in your project with the DOWNLOAD_DELAY setting.
85
5.1.22 Simplest way to dump all my scraped items into a JSON/CSV/XML le?
To dump into a JSON le:
scrapy crawl myspider -o items.json -t json
5.1.23 Whats this huge cryptic __VIEWSTATE parameter used in some forms?
The __VIEWSTATE parameter is used in sites built with ASP.NET/VB.NET. For more info on how it works see this page. Also, heres an example spider which scrapes one of these sites.
5.1.24 Whats the best way to parse big XML/CSV data feeds?
Parsing big feeds with XPath selectors can be problematic since they need to build the DOM of the entire feed in memory, and this can be quite slow and consume a lot of memory. In order to avoid parsing all the entire feed at once in memory, you can use the functions xmliter and csviter from scrapy.utils.iterators module. In fact, this is what the feed spiders (see Spiders) use under the cover.
5.1.26 How can I see the cookies being sent and received from Scrapy?
Enable the COOKIES_DEBUG setting.
86
5.1.30 Im scraping a XML document and my XPath selector doesnt return any items
You may need to remove namespaces. See Removing namespaces.
This way to access the crawler object is deprecated, the code should be ported to use from_crawler class method, for example:
class SomeExtension(object): @classmethod def from_crawler(cls, crawler): o = cls() o.crawler = crawler return o
Scrapy command line tool has some backwards compatibility in place to support the old import mechanism (with a deprecation warning), but this mechanism may not work if you use Scrapy differently (for example, as a library).
87
class MySpider(Spider): name = myspider start_urls = ( http://example.com/page1, http://example.com/page2, ) def parse(self, response): # collect item_urls for item_url in item_urls: yield Request(url=item_url, callback=self.parse_item) def parse_item(self, response): item = MyItem() # populate item fields yield Request(url=item_details_url, meta={item: item}, callback=self.parse_details) def parse_details(self, response): item = response.meta[item] # populate more item fields return item
Basically this is a simple spider which parses two pages of items (the start_urls). Items also have a details page with additional information, so we use the meta functionality of Request to pass a partially populated item.
Using the --verbose or -v option we can see the status at each depth level:
$ scrapy parse --spider=myspider -c parse_item -d 2 -v <item_url> [ ... scrapy log lines crawling example.com spider ... ] >>> DEPTH LEVEL: 1 <<< # Scraped Items -----------------------------------------------------------[] # Requests ----------------------------------------------------------------[<GET item_details_url>]
88
Checking items scraped from a single start_url, can also be easily achieved using:
$ scrapy parse --spider=myspider -d 3 http://example.com/page1
open_in_browser will open a browser with the response received by Scrapy at that point, adjusting the base tag so that images and styles are displayed properly.
5.2.4 Logging
Logging is another useful option for getting information about your spider run. Although not as convenient, it comes with the advantage that the logs will be available in all future runs should they be necessary again:
from scrapy import log def parse_details(self, response): item = response.meta.get(item, None) if item: # populate more item fields
89
This callback is tested using three built-in contracts: class scrapy.contracts.default.UrlContract This contract (@url) sets the sample url used when checking other contract conditions for this spider. This contract is mandatory. All callbacks lacking this contract are ignored when running the checks:
@url url
class scrapy.contracts.default.ReturnsContract This contract (@returns) sets lower and upper bounds for the items and requests returned by the spider. The upper bound is optional:
@returns item(s)|request(s) [min [max]]
class scrapy.contracts.default.ScrapesContract This contract (@scrapes) checks that all the items returned by the callback have the specied elds:
@scrapes field_1 field_2 ...
Each contract must inherit from scrapy.contracts.Contract and can override three methods: class scrapy.contracts.Contract(method, *args) Parameters method (function) callback function to which the contract is associated args (list) list of arguments passed into the docstring (whitespace separated) adjust_request_args(args) This receives a dict as an argument containing default arguments for Request object. Must return the same or a modied version of it. pre_process(response) This allows hooking in various checks on the response received from the sample request, before its being passed to the callback. post_process(output) This allows processing the output of the callback. Iterators are converted listied before being passed to this hook. Here is a demo contract which checks the presence of a custom header in the response received. scrapy.exceptions.ContractFail in order to get the failures pretty printed:
from scrapy.contracts import Contract from scrapy.exceptions import ContractFail class HasHeaderContract(Contract): """ Demo contract which checks the presence of a custom header @has_header X-CustomHeader """ name = has_header def pre_process(self, response): for header in self.args: if header not in response.headers: raise ContractFail(X-CustomHeader not present)
Raise
91
Note that you will also have to shutdown the Twisted reactor yourself after the spider is nished. This can be achieved by connecting a handler to the signals.spider_closed signal. What follows is a working example of how to do that, using the testspiders project as example.
from from from from from twisted.internet import reactor scrapy.crawler import Crawler scrapy import log, signals testspiders.spiders.followall import FollowAllSpider scrapy.utils.project import get_project_settings
spider = FollowAllSpider(domain=scrapinghub.com) settings = get_project_settings() crawler = Crawler(settings) crawler.signals.connect(reactor.stop, signal=signals.spider_closed) crawler.configure() crawler.crawl(spider) crawler.start() log.start() reactor.run() # the script will block here until the spider_closed signal was sent
def setup_crawler(domain): spider = FollowAllSpider(domain=domain) settings = get_project_settings() crawler = Crawler(settings) crawler.configure() crawler.crawl(spider) crawler.start() for domain in [scrapinghub.com, insophia.com]: setup_crawler(domain) log.start() reactor.run()
92
Then you re a spider run on 3 different Scrapyd servers. The spider would receive a (spider) argument part with the number of the partition to crawl:
curl http://scrapy1.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=1 curl http://scrapy2.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=2 curl http://scrapy3.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=3
93
94
When doing broad crawls its common to crawl a lot of index web pages; AjaxCrawlMiddleware helps to crawl them correctly. It is turned OFF by default because it has some performance overhead, and enabling it for focused crawls doesnt make much sense. 5.5. Broad Crawls 95
96
5.7.1 Introduction
This document explains how to use Firebug (a Firefox add-on) to make the scraping process easier and more fun. For other useful Firefox add-ons see Useful Firefox add-ons for scraping. There are some caveats with using Firefox add-ons to inspect pages, see Caveats with inspecting the live browser DOM . In this example, well show how to use Firebug to scrape data from the Google Directory, which contains the same data as the Open Directory Project used in the tutorial but with a different face. Firebug comes with a very useful feature called Inspect Element which allows you to inspect the HTML code of the different page elements just by hovering your mouse over them. Otherwise you would have to search for the tags manually through the HTML body which can be a very tedious task. In the following screenshot you can see the Inspect Element tool in action.
At rst sight, we can see that the directory is divided in categories, which are also divided in subcategories. However, it seems that there are more subcategories than the ones being shown in this page, so well keep looking:
97
As expected, the subcategories contain links to other subcategories, and also links to actual websites, which is the purpose of the directory.
So, based on that regular expression we can create the rst crawling rule:
Rule(SgmlLinkExtractor(allow=directory.google.com/[A-Z][a-zA-Z_/]+$, ), parse_category, follow=True, ),
The Rule object instructs CrawlSpider based spiders how to follow the category links. parse_category will be a method of the spider which will process and extract data from those pages. This is how the spider would look so far:
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor from scrapy.contrib.spiders import CrawlSpider, Rule
98
class GoogleDirectorySpider(CrawlSpider): name = directory.google.com allowed_domains = [directory.google.com] start_urls = [http://directory.google.com/] rules = ( Rule(SgmlLinkExtractor(allow=directory\.google\.com/[A-Z][a-zA-Z_/]+$), parse_category, follow=True, ), ) def parse_category(self, response): # write the category page data extraction code here pass
99
As you can see, the page markup is not very descriptive: the elements dont contain id, class or any attribute that clearly identies them, so well use the ranking bars as a reference point to select the data to extract when we construct our XPaths. After using FireBug, we can see that each link is inside a td tag, which is itself inside a tr tag that also contains the links ranking bar (in another td). So we can select the ranking bar, then nd its parent (the tr), and then nally, the links td (which contains the data we want to scrape). This results in the following XPath:
//td[descendant::a[contains(@href, "#pagerank")]]/following-sibling::td//a
Its important to use the Scrapy shell to test these complex XPath expressions and make sure they work as expected. Basically, that expression will look for the ranking bars td element, and then select any td element who has a descendant a element whose href attribute contains the string #pagerank Of course, this is not the only XPath, and maybe not the simpler one to select that data. Another approach could be, for example, to nd any font tags that have that grey colour of the links, Finally, we can write our parse_category() method:
def parse_category(self, response): sel = Selector(response)
# The path to website links in directory page links = sel.xpath(//td[descendant::a[contains(@href, "#pagerank")]]/following-sibling::td/font) for link in links: item = DirectoryItem() item[name] = link.xpath(a/text()).extract() item[url] = link.xpath(a/@href).extract() item[description] = link.xpath(font[2]/text()).extract() yield item
Be aware that you may nd some elements which appear in Firebug but not in the original HTML, such as the typical case of <tbody> elements. or tags which Therefer in page HTML sources may on Firebug inspects the live DOM
100
As you can see, that report also shows the age of the oldest object in each class. If you do have leaks, chances are you can gure out which spider is leaking by looking at the oldest request or response. You can get the oldest object of each class using the get_oldest() function like this (from the telnet console). Which objects are tracked? The objects tracked by trackrefs are all from these classes (and all its subclasses): scrapy.http.Request scrapy.http.Response scrapy.item.Item scrapy.selector.Selector scrapy.spider.Spider
101
A real example Lets see a concrete example of an hypothetical case of memory leaks. Suppose we have some spider with a line similar to this one:
return Request("http://www.somenastyspider.com/product.php?pid=%d" % product_id, callback=self.parse, meta={referer: response}")
That line is passing a response reference inside a request which effectively ties the response lifetime to the requests one, and that would denitely cause memory leaks. Lets see how we can discover which one is the nasty spider (without knowing it a-priori, of course) by using the trackref tool. After the crawler is running for a few minutes and we notice its memory usage has grown a lot, we can enter its telnet console and check the live references:
>>> prefs() Live References SomenastySpider HtmlResponse Selector Request 1 3890 2 3878 oldest: oldest: oldest: oldest: 15s ago 265s ago 0s ago 250s ago
The fact that there are so many live responses (and that theyre so old) is denitely suspicious, as responses should have a relatively short lifetime compared to Requests. So lets check the oldest response:
>>> from scrapy.utils.trackref import get_oldest >>> r = get_oldest(HtmlResponse) >>> r.url http://www.somenastyspider.com/product.php?pid=123
There it is. By looking at the URL of the oldest response we can see it belongs to the somenastyspider.com spider. We can now go and check the code of that spider to discover the nasty line that is generating the leaks (passing response references inside requests). If you want to iterate over all objects, instead of getting the oldest one, you can use the iter_all() function:
>>> from scrapy.utils.trackref import iter_all >>> [r.url for r in iter_all(HtmlResponse)] [http://www.somenastyspider.com/product.php?pid=123, http://www.somenastyspider.com/product.php?pid=584, ...
Too many spiders? If your project has too many spiders, the output of prefs() can be difcult to read. For this reason, that function has a ignore argument which can be used to ignore a particular class (and all its subclases). For example, using:
>>> from scrapy.spider import Spider >>> prefs(ignore=Spider)
Wont show any live references to spiders. scrapy.utils.trackref module Here are the functions available in the trackref module. 102 Chapter 5. Solving specic problems
class scrapy.utils.trackref.object_ref Inherit from this class (instead of object) if you want to track live instances with the trackref module. scrapy.utils.trackref.print_live_refs(class_name, ignore=NoneType) Print a report of live references, grouped by class name. Parameters ignore (class or classes tuple) if given, all objects from the specied class (or tuple of classes) will be ignored. scrapy.utils.trackref.get_oldest(class_name) Return the oldest object alive with the given class name, or None if none is found. Use print_live_refs() rst to get a list of all tracked live objects per class name. scrapy.utils.trackref.iter_all(class_name) Return an iterator over all objects alive with the given class name, or None if none is found. print_live_refs() rst to get a list of all tracked live objects per class name. Use
The telnet console also comes with a built-in shortcut (hpy) for accessing Guppy heap objects. Heres an example to view all Python objects available in the heap using Guppy:
>>> x = hpy.heap() >>> x.bytype Partition of a set of 297033 objects. Total size = 52587824 bytes. Index Count % Size % Cumulative % Type 0 22307 8 16423880 31 16423880 31 dict 1 122285 41 12441544 24 28865424 55 str 2 68346 23 5966696 11 34832120 66 tuple 3 227 0 5836528 11 40668648 77 unicode 4 2461 1 2222272 4 42890920 82 type 5 16870 6 2024400 4 44915320 85 function 6 13949 5 1673880 3 46589200 89 types.CodeType 7 13422 5 1653104 3 48242304 92 list 8 3735 1 1173680 2 49415984 94 _sre.SRE_Pattern 9 1209 0 456936 1 49872920 95 scrapy.http.headers.Headers <1676 more rows. Type e.g. _.more to view.>
You can see that most space is used by dicts. Then, if you want to see from which attribute those dicts are referenced, you could do:
>>> x.bytype[0].byvia Partition of a set of 22307 objects. Total size = 16423880 bytes. Index Count % Size % Cumulative % Referred Via: 0 10982 49 9416336 57 9416336 57 .__dict__ 1 1820 8 2681504 16 12097840 74 .__dict__, .func_globals 2 3097 14 1122904 7 13220744 80 3 990 4 277200 2 13497944 82 "[cookies]" 4 987 4 276360 2 13774304 84 "[cache]" 5 985 4 275800 2 14050104 86 "[meta]"
103
4 251160 2 14301264 87 0 196888 1 14498152 88 3 188160 1 14686312 89 0 155016 1 14841328 90 Type e.g. _.more to view.>
As you can see, the Guppy module is very powerful but also requires some deep knowledge about Python internals. For more info about Guppy, refer to the Guppy documentation.
104
If you need something more complex and want to override the custom images pipeline behaviour, see Implementing your custom Images Pipeline.
And set the IMAGES_STORE setting to a valid directory that will be used for storing the downloaded images. Otherwise the pipeline will remain disabled, even if you include it in the ITEM_PIPELINES setting. For example:
IMAGES_STORE = /path/to/valid/dir
105
File system storage The images are stored in les (one per image), using a SHA1 hash of their URLs for the le names. For example, the following image URL:
http://www.example.com/image.jpg
Where: <IMAGES_STORE> is the directory dened in IMAGES_STORE setting full is a sub-directory to separate full images from thumbnails (if used). For more info see Thumbnail generation.
Thumbnail generation The Images Pipeline can automatically create thumbnails of the downloaded images. In order use this feature, you must set IMAGES_THUMBS to a dictionary where the keys are the thumbnail names and the values are their dimensions. For example:
IMAGES_THUMBS = { small: (50, 50), big: (270, 270), }
When you use this feature, the Images Pipeline will create thumbnails of the each specied size with this format:
<IMAGES_STORE>/thumbs/<size_name>/<image_id>.jpg
Where: <size_name> is the one specied in the IMAGES_THUMBS dictionary keys (small, big, etc) <image_id> is the SHA1 hash of the image url Example of image les stored using small and big thumbnail names:
106
The rst one is the full image, as downloaded from the site. Filtering out small images You can drop images which are too small, by specifying the minimum allowed size in the IMAGES_MIN_HEIGHT and IMAGES_MIN_WIDTH settings. For example:
IMAGES_MIN_HEIGHT = 110 IMAGES_MIN_WIDTH = 110
Note: these size constraints dont affect thumbnail generation at all. By default, there are no size constraints, so all images are processed.
Those requests will be processed by the pipeline and, when they have nished downloading, the results will be sent to the item_completed() method, as a list of 2-element tuples. Each tuple will contain (success, image_info_or_failure) where: success is a boolean which is True if the image was downloaded successfully or False if it failed for some reason image_info_or_error is a dict containing the following keys (if success is True) or a Twisted Failure if there was a problem. url - the url where the image was downloaded from. This is the url of the request returned from the get_media_requests() method. path - the path (relative to IMAGES_STORE) where the image was stored checksum - a MD5 hash of the image contents The list of tuples received by item_completed() is guaranteed to retain the same order of the requests returned from the get_media_requests() method. Heres a typical value of the results argument:
107
[(True, {checksum: 2b00042f7481c7b056c4b410d28f33cf, path: full/7d97e98f8af710c7e7fe703abc8f639e0ee507c4.jpg, url: http://www.example.com/images/product1.jpg}), (True, {checksum: b9628c4ab9b595f72f280b90c4fd093d, path: full/1ca5879492b8fd606df1964ea3c1e2f4520f076f.jpg, url: http://www.example.com/images/product2.jpg}), (False, Failure(...))]
By default the get_media_requests() method returns None which means there are no images to download for the item. item_completed(results, items, info) The ImagesPipeline.item_completed() method called when all image requests for a single item have completed (either nished downloading, or failed for some reason). The item_completed() method must return the output that will be sent to subsequent item pipeline stages, so you must return (or drop) the item, as you would in any pipeline. Here is an example of the item_completed() method where we store the downloaded image paths (passed in results) in the image_paths item eld, and we drop the item if it doesnt contain any images:
from scrapy.exceptions import DropItem def item_completed(self, results, item, info): image_paths = [x[path] for ok, x in results if ok] if not image_paths: raise DropItem("Item contains no images") item[image_paths] = image_paths return item
108
3. Update package lists and install scrapy-VERSION, replace VERSION by a known Scrapy version (i.e.: scrapy0.22:
sudo apt-get update && sudo apt-get install scrapy-VERSION
Note: Repeat step 3 if you are trying to upgrade Scrapy. Warning: python-scrapy is a different package provided by ofcial debian repositories, its very outdated and it isnt supported by Scrapy team.
5.11 Scrapyd
Scrapyd has been moved into a separate project. Its documentation is now hosted at: http://scrapyd.readthedocs.org/
109
5.12.4 Settings
The settings used to control the AutoThrottle extension are: AUTOTHROTTLE_ENABLED AUTOTHROTTLE_START_DELAY AUTOTHROTTLE_MAX_DELAY AUTOTHROTTLE_DEBUG CONCURRENT_REQUESTS_PER_DOMAIN CONCURRENT_REQUESTS_PER_IP DOWNLOAD_DELAY For more information see Throttling algorithm. AUTOTHROTTLE_ENABLED Default: False Enables the AutoThrottle extension. AUTOTHROTTLE_START_DELAY Default: 5.0 The initial download delay (in seconds).
110
AUTOTHROTTLE_MAX_DELAY Default: 60.0 The maximum download delay (in seconds) to be set in case of high latencies. AUTOTHROTTLE_DEBUG Default: False Enable AutoThrottle debug mode which will display stats on every response received, so you can see how the throttling parameters are being adjusted in real time.
5.13 Benchmarking
New in version 0.17. Scrapy comes with a simple benchmarking suite that spawns a local HTTP server and crawls it at the maximum possible speed. The goal of this benchmarking is to get an idea of how Scrapy performs in your hardware, in order to have a common baseline for comparisons. It uses a simple spider that does nothing and just follows links. To run it use:
scrapy bench
2013-05-16 13:08:46-0300 [scrapy] INFO: Scrapy 0.17.0 started (bot: scrapybot) 2013-05-16 13:08:47-0300 [follow] INFO: Spider opened 2013-05-16 13:08:47-0300 [follow] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items 2013-05-16 13:08:48-0300 [follow] INFO: Crawled 74 pages (at 4440 pages/min), scraped 0 items (at 0 i 2013-05-16 13:08:49-0300 [follow] INFO: Crawled 143 pages (at 4140 pages/min), scraped 0 items (at 0 2013-05-16 13:08:50-0300 [follow] INFO: Crawled 210 pages (at 4020 pages/min), scraped 0 items (at 0 2013-05-16 13:08:51-0300 [follow] INFO: Crawled 274 pages (at 3840 pages/min), scraped 0 items (at 0 2013-05-16 13:08:52-0300 [follow] INFO: Crawled 343 pages (at 4140 pages/min), scraped 0 items (at 0 2013-05-16 13:08:53-0300 [follow] INFO: Crawled 410 pages (at 4020 pages/min), scraped 0 items (at 0 2013-05-16 13:08:54-0300 [follow] INFO: Crawled 474 pages (at 3840 pages/min), scraped 0 items (at 0 2013-05-16 13:08:55-0300 [follow] INFO: Crawled 538 pages (at 3840 pages/min), scraped 0 items (at 0 2013-05-16 13:08:56-0300 [follow] INFO: Crawled 602 pages (at 3840 pages/min), scraped 0 items (at 0 2013-05-16 13:08:57-0300 [follow] INFO: Closing spider (closespider_timeout) 2013-05-16 13:08:57-0300 [follow] INFO: Crawled 666 pages (at 3840 pages/min), scraped 0 items (at 0 2013-05-16 13:08:57-0300 [follow] INFO: Dumping Scrapy stats: {downloader/request_bytes: 231508, downloader/request_count: 682, downloader/request_method_count/GET: 682, downloader/response_bytes: 1172802, downloader/response_count: 682, downloader/response_status_count/200: 682, finish_reason: closespider_timeout, finish_time: datetime.datetime(2013, 5, 16, 16, 8, 57, 985539), log_count/INFO: 14, request_depth_max: 34, response_received_count: 682, scheduler/dequeued: 682, scheduler/dequeued/memory: 682, scheduler/enqueued: 12767, scheduler/enqueued/memory: 12767,
5.13. Benchmarking
111
start_time: datetime.datetime(2013, 5, 16, 16, 8, 47, 676539)} 2013-05-16 13:08:57-0300 [follow] INFO: Spider closed (closespider_timeout)
That tells you that Scrapy is able to crawl about 3900 pages per minute in the hardware where you run it. Note that this is a very simple spider intended to follow links, any custom spider you write will probably do more stuff which results in slower crawl rates. How slower depends on how much your spider does and how well its written. In the future, more cases will be added to the benchmarking suite to cover other common scenarios.
Then, you can stop the spider safely at any time (by pressing Ctrl-C or sending a signal), and resume it later by issuing the same command:
scrapy crawl somespider -s JOBDIR=crawls/somespider-1
112
5.15 DjangoItem
DjangoItem is a class of item that gets its elds denition from a Django model, you simply create a DjangoItem and specify what Django model it relates to. Besides of getting the model elds dened on your item, DjangoItem provides a method to create and populate a Django model instance with the item data.
5.15. DjangoItem
113
from django.db import models class Person(models.Model): name = models.CharField(max_length=255) age = models.IntegerField()
To obtain the Django model from the item, we call the extra method save() of the DjangoItem:
>>> person = p.save() >>> person.name John >>> person.age 22 >>> person.id 1
The model is already saved when we call save(), we can prevent this by calling it with commit=False. We can use commit=False in save() method to obtain an unsaved model:
>>> person = p.save(commit=False) >>> person.name John >>> person.age 22 >>> person.id None
Note: elds added to the item wont be taken into account when doing a save() And we can override the elds of the model with your own:
class PersonItem(DjangoItem): django_model = Person name = Field(default=No Name)
This is useful to provide properties to the eld, like a default or any other property that your project uses. 114 Chapter 5. Solving specic problems
Then you need to add /home/projects/mysite to the PYTHONPATH environment variable and set up the environment variable DJANGO_SETTINGS_MODULE to mysite.settings. That can be done in your Scrapys settings le by adding the lines below:
import sys sys.path.append(/home/projects/mysite) import os os.environ[DJANGO_SETTINGS_MODULE] = mysite.settings
Notice that we modify the sys.path variable instead the PYTHONPATH environment variable as we are already within the python runtime. If everything is right, you should be able to start the scrapy shell command and import the model Person (i.e. from myapp.models import Person). Frequently Asked Questions Get answers to most frequently asked questions. Debugging Spiders Learn how to debug common problems of your scrapy spider. Spiders Contracts Learn how to use contracts for testing your spiders. Common Practices Get familiar with some Scrapy common practices. Broad Crawls Tune Scrapy for crawling a lot domains in parallel. Using Firefox for scraping Learn how to scrape with Firefox and some useful add-ons. Using Firebug for scraping Learn how to scrape efciently using Firebug.
5.15. DjangoItem
115
Debugging memory leaks Learn how to nd and get rid of memory leaks in your crawler. Downloading Item Images Download static images associated with your scraped items. Ubuntu packages Install latest Scrapy packages easily on Ubuntu Scrapyd Deploying your Scrapy project in production. AutoThrottle extension Adjust crawl rate dynamically based on load. Benchmarking Check how Scrapy performs on your hardware. Jobs: pausing and resuming crawls Learn how to pause and resume crawls for large spiders. DjangoItem Write scraped items using Django models.
116
CHAPTER 6
Extending Scrapy
6.1.1 Overview
The following diagram shows an overview of the Scrapy architecture with its components and an outline of the data ow that takes place inside the system (shown by the green arrows). A brief description of the components is included below with links for more detailed information about them. The data ow is also described below.
117
6.1.2 Components
Scrapy Engine The engine is responsible for controlling the data ow between all components of the system, and triggering events when certain actions occur. See the Data Flow section below for more details. Scheduler The Scheduler receives requests from the engine and enqueues them for feeding them later (also to the engine) when the engine requests them. Downloader The Downloader is responsible for fetching web pages and feeding them to the engine which, in turn, feeds them to the spiders. Spiders Spiders are custom classes written by Scrapy users to parse responses and extract items (aka scraped items) from them or additional URLs (requests) to follow. Each spider is able to handle a specic domain (or group of domains). For more information see Spiders. Item Pipeline The Item Pipeline is responsible for processing the items once they have been extracted (or scraped) by the spiders. Typical tasks include cleansing, validation and persistence (like storing the item in a database). For more information see Item Pipeline. Downloader middlewares Downloader middlewares are specic hooks that sit between the Engine and the Downloader and process requests when they pass from the Engine to the Downloader, and responses that pass from Downloader to the Engine. They provide a convenient mechanism for extending Scrapy functionality by plugging custom code. For more information see Downloader Middleware. Spider middlewares Spider middlewares are specic hooks that sit between the Engine and the Spiders and are able to process spider input (responses) and output (items and requests). They provide a convenient mechanism for extending Scrapy functionality by plugging custom code. For more information see Spider Middleware.
6.1.3 Data ow
The data ow in Scrapy is controlled by the execution engine, and goes like this: 1. The Engine opens a domain, locates the Spider that handles that domain, and asks the spider for the rst URLs to crawl. 2. The Engine gets the rst URLs to crawl from the Spider and schedules them in the Scheduler, as Requests. 118 Chapter 6. Extending Scrapy
3. The Engine asks the Scheduler for the next URLs to crawl. 4. The Scheduler returns the next URLs to crawl to the Engine and the Engine sends them to the Downloader, passing through the Downloader Middleware (request direction). 5. Once the page nishes downloading the Downloader generates a Response (with that page) and sends it to the Engine, passing through the Downloader Middleware (response direction). 6. The Engine receives the Response from the Downloader and sends it to the Spider for processing, passing through the Spider Middleware (input direction). 7. The Spider processes the Response and returns scraped Items and new Requests (to follow) to the Engine. 8. The Engine sends scraped Items (returned by the Spider) to the Item Pipeline and Requests (returned by spider) to the Scheduler 9. The process repeats (from step 2) until there are no more requests from the Scheduler, and the Engine closes the domain.
The DOWNLOADER_MIDDLEWARES setting is merged with the DOWNLOADER_MIDDLEWARES_BASE setting dened in Scrapy (and not meant to be overridden) and then sorted by order to get the nal sorted list of enabled middlewares: the rst middleware is the one closer to the engine and the last is the one closer to the downloader. To decide which order to assign to your middleware see the DOWNLOADER_MIDDLEWARES_BASE setting and pick a value according to where you want to insert the middleware. The order does matter because each middleware performs a different action and your middleware could depend on some previous (or subsequent) middleware being applied. If you want to disable a built-in middleware (the ones dened in DOWNLOADER_MIDDLEWARES_BASE and enabled by default) you must dene it in your projects DOWNLOADER_MIDDLEWARES setting and assign None as its value. For example, if you want to disable the off-site middleware:
119
Finally, keep in mind that some middlewares may need to be enabled through a particular setting. See each middleware documentation for more info.
120
response (Response object) the response being processed spider (Spider object) the spider for which this response is intended process_exception(request, exception, spider) Scrapy calls process_exception() when a download handler or a process_request() (from a downloader middleware) raises an exception (including an IgnoreRequest exception) process_exception() should return: either None, a Response object, or a Request object. If it returns None, Scrapy will continue processing this exception, executing any other process_exception() methods of installed middleware, until no middleware is left and the default exception handling kicks in. If it returns a Response object, the process_response() method chain of installed middleware is started, and Scrapy wont bother calling any other process_exception() methods of middleware. If it returns a Request object, the returned request is rescheduled to be downloaded in the future. This stops the execution of process_exception() methods of the middleware the same as returning a response would. Parameters request (is a Request object) the request that generated the exception exception (an Exception object) the raised exception spider (Spider object) the spider for which this request is intended
New in version 0.15. There is support for keeping multiple cookie sessions per spider by using the cookiejar Request meta key. By default it uses a single cookie jar (session), but you can pass an identier to use different ones. For example:
121
Keep in mind that the cookiejar meta key is not sticky. You need to keep passing it along on subsequent requests. For example:
def parse_page(self, response): # do some processing return Request("http://www.example.com/otherpage", meta={cookiejar: response.meta[cookiejar]}, callback=self.parse_other_page)
COOKIES_ENABLED
Default: True Whether to enable the cookies middleware. If disabled, no cookies will be sent to web servers.
COOKIES_DEBUG
Default: False If enabled, Scrapy will log all cookies sent in requests (ie. Cookie header) and all cookies received in responses (ie. Set-Cookie header). Heres an example of a log with COOKIES_DEBUG enabled:
2011-04-06 14:35:10-0300 [diningcity] INFO: Spider opened 2011-04-06 14:35:10-0300 [diningcity] DEBUG: Sending cookies to: <GET http://www.diningcity.com/nethe Cookie: clientlanguage_nl=en_EN 2011-04-06 14:35:14-0300 [diningcity] DEBUG: Received cookies from: <200 http://www.diningcity.com/ne Set-Cookie: JSESSIONID=B~FA4DC0C496C8762AE4F1A620EAB34F38; Path=/ Set-Cookie: ip_isocode=US Set-Cookie: clientlanguage_nl=en_EN; Expires=Thu, 07-Apr-2011 21:21:34 GMT; Path=/ 2011-04-06 14:49:50-0300 [diningcity] DEBUG: Crawled (200) <GET http://www.diningcity.com/netherlands [...]
DefaultHeadersMiddleware class scrapy.contrib.downloadermiddleware.defaultheaders.DefaultHeadersMiddleware This middleware sets all default requests headers specied in the DEFAULT_REQUEST_HEADERS setting. DownloadTimeoutMiddleware class scrapy.contrib.downloadermiddleware.downloadtimeout.DownloadTimeoutMiddleware This middleware sets the download timeout for requests specied in the DOWNLOAD_TIMEOUT setting. HttpAuthMiddleware class scrapy.contrib.downloadermiddleware.httpauth.HttpAuthMiddleware This middleware authenticates all requests generated from certain spiders using Basic access authentication (aka. HTTP auth). 122 Chapter 6. Extending Scrapy
To enable HTTP authentication from certain spiders, set the http_user and http_pass attributes of those spiders. Example:
from scrapy.contrib.spiders import CrawlSpider class SomeIntranetSiteSpider(CrawlSpider): http_user = someuser http_pass = somepass name = intranet.example.com # .. rest of the spider code omitted ...
HttpCacheMiddleware class scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware This middleware provides low-level cache to all HTTP requests and responses. It has to be combined with a cache storage backend as well as a cache policy. Scrapy ships with two HTTP cache storage backends: Filesystem storage backend (default) DBM storage backend You can change the HTTP cache storage backend with the HTTPCACHE_STORAGE setting. Or you can also implement your own storage backend. Scrapy ships with two HTTP cache policies: RFC2616 policy Dummy policy (default) You can change the HTTP cache policy with the HTTPCACHE_POLICY setting. Or you can also implement your own policy.
Dummy policy (default)
This policy has no awareness of any HTTP Cache-Control directives. Every request and its corresponding response are cached. When the same request is seen again, the response is returned without transferring anything from the Internet. The Dummy policy is useful for testing spiders faster (without having to wait for downloads every time) and for trying your spider ofine, when an Internet connection is not available. The goal is to be able to replay a spider run exactly as it ran before. In order to use this policy, set: HTTPCACHE_POLICY to scrapy.contrib.httpcache.DummyPolicy
RFC2616 policy
This policy provides a RFC2616 compliant HTTP cache, i.e. with HTTP Cache-Control awareness, aimed at production and used in continuous runs to avoid downloading unmodied data (to save bandwidth and speed up crawls). what is implemented:
123
Do not attempt to store responses/requests with no-store cache-control directive set Do not serve responses from cache if no-cache cache-control directive is set even for fresh responses Compute freshness lifetime from max-age cache-control directive Compute freshness lifetime from Expires response header Compute freshness lifetime from Last-Modied response header (heuristic used by Firefox) Compute current age from Age response header Compute current age from Date header Revalidate stale responses based on Last-Modied response header Revalidate stale responses based on ETag response header Set Date header for any received response missing it what is missing: Pragma: no-cache support http://www.mnot.net/cache_docs/#PRAGMA Vary header support http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.6 Invalidation after updates or deletes http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.10 ... probably others .. In order to use this policy, set: HTTPCACHE_POLICY to scrapy.contrib.httpcache.RFC2616Policy
Filesystem storage backend (default)
File system storage backend is available for the HTTP cache middleware. In order to use this storage backend, set: HTTPCACHE_STORAGE to scrapy.contrib.httpcache.FilesystemCacheStorage Each request/response pair is stored in a different directory containing the following les: request_body - the plain request body request_headers - the request headers (in raw HTTP format) response_body - the plain response body response_headers - the request headers (in raw HTTP format) meta - some metadata of this cache resource in Python repr() format (grep-friendly format) pickled_meta - the same metadata in meta but pickled for more efcient deserialization The directory name is made from the request ngerprint (see scrapy.utils.request.fingerprint), and one level of subdirectories is used to avoid creating too many les into the same directory (which is inefcient in many le systems). An example directory could be:
/path/to/cache/dir/example.com/72/72811f648e718090f041317756c03adb0ada46c7
124
New in version 0.13. A DBM storage backend is also available for the HTTP cache middleware. By default, it uses the anydbm module, but you can change it with the HTTPCACHE_DBM_MODULE setting. In order to use this storage backend, set: HTTPCACHE_STORAGE to scrapy.contrib.httpcache.DbmCacheStorage
HTTPCache middleware settings
The HttpCacheMiddleware can be congured through the following settings: HTTPCACHE_ENABLED New in version 0.11. Default: False Whether the HTTP cache will be enabled. Changed in version 0.11: Before 0.11, HTTPCACHE_DIR was used to enable cache. HTTPCACHE_EXPIRATION_SECS Default: 0 Expiration time for cached requests, in seconds. Cached requests older than this time will be re-downloaded. If zero, cached requests will never expire. Changed in version 0.11: Before 0.11, zero meant cached requests always expire. HTTPCACHE_DIR Default: httpcache The directory to use for storing the (low-level) HTTP cache. If empty, the HTTP cache will be disabled. If a relative path is given, is taken relative to the project data dir. For more info see: Default structure of Scrapy projects. HTTPCACHE_IGNORE_HTTP_CODES New in version 0.10. Default: [] Dont cache response with these HTTP codes. HTTPCACHE_IGNORE_MISSING Default: False If enabled, requests not found in the cache will be ignored instead of downloaded. HTTPCACHE_IGNORE_SCHEMES New in version 0.10. Default: [file] Dont cache responses with these URI schemes. HTTPCACHE_STORAGE Default: scrapy.contrib.httpcache.FilesystemCacheStorage The class which implements the cache storage backend.
125
HTTPCACHE_DBM_MODULE New in version 0.13. Default: anydbm The database module to use in the DBM storage backend. This setting is specic to the DBM backend. HTTPCACHE_POLICY New in version 0.18. Default: scrapy.contrib.httpcache.DummyPolicy The class which implements the cache policy. HttpCompressionMiddleware class scrapy.contrib.downloadermiddleware.httpcompression.HttpCompressionMiddleware This middleware allows compressed (gzip, deate) trafc to be sent/received from web sites.
HttpCompressionMiddleware Settings
COMPRESSION_ENABLED Default: True Whether the Compression middleware will be enabled. ChunkedTransferMiddleware class scrapy.contrib.downloadermiddleware.chunked.ChunkedTransferMiddleware This middleware adds support for chunked transfer encoding HttpProxyMiddleware New in version 0.8. class scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware This middleware sets the HTTP proxy to use for requests, by setting the proxy meta value to Request objects. Like the Python standard library modules urllib and urllib2, it obeys the following environment variables: http_proxy https_proxy no_proxy RedirectMiddleware class scrapy.contrib.downloadermiddleware.redirect.RedirectMiddleware This middleware handles redirection of requests based on response status. The urls which the request goes through (while being redirected) can be found in the redirect_urls Request.meta key. The RedirectMiddleware can be congured through the following settings (see the settings documentation for more info): REDIRECT_ENABLED REDIRECT_MAX_TIMES 126 Chapter 6. Extending Scrapy
If Request.meta contains the dont_redirect key, the request will be ignored by this middleware.
RedirectMiddleware settings
REDIRECT_ENABLED New in version 0.13. Default: True Whether the Redirect middleware will be enabled. REDIRECT_MAX_TIMES Default: 20 The maximum number of redirections that will be follow for a single request. MetaRefreshMiddleware class scrapy.contrib.downloadermiddleware.redirect.MetaRefreshMiddleware This middleware handles redirection of requests based on meta-refresh html tag. The MetaRefreshMiddleware can be congured through the following settings (see the settings documentation for more info): METAREFRESH_ENABLED METAREFRESH_MAXDELAY This middleware obey REDIRECT_MAX_TIMES setting, dont_redirect and redirect_urls request meta keys as described for RedirectMiddleware
MetaRefreshMiddleware settings
METAREFRESH_ENABLED New in version 0.17. Default: True Whether the Meta Refresh middleware will be enabled. REDIRECT_MAX_METAREFRESH_DELAY Default: 100 The maximum meta-refresh delay (in seconds) to follow the redirection. RetryMiddleware class scrapy.contrib.downloadermiddleware.retry.RetryMiddleware A middlware to retry failed requests that are potentially caused by temporary problems such as a connection timeout or HTTP 500 error. Failed pages are collected on the scraping process and rescheduled at the end, once the spider has nished crawling all regular (non failed) pages. Once there are no more failed pages to retry, this middleware sends a signal (retry_complete), so other extensions could connect to that signal. The RetryMiddleware can be congured through the following settings (see the settings documentation for more info): RETRY_ENABLED
127
RETRY_TIMES RETRY_HTTP_CODES About HTTP errors to consider: You may want to remove 400 from RETRY_HTTP_CODES, if you stick to the HTTP protocol. Its included by default because its a common code used to indicate server overload, which would be something we want to retry. If Request.meta contains the dont_retry key, the request will be ignored by this middleware.
RetryMiddleware Settings
RETRY_ENABLED New in version 0.13. Default: True Whether the Retry middleware will be enabled. RETRY_TIMES Default: 2 Maximum number of times to retry, in addition to the rst download. RETRY_HTTP_CODES Default: [500, 502, 503, 504, 400, 408] Which HTTP response codes to retry. Other errors (DNS lookup issues, connections lost, etc) are always retried. RobotsTxtMiddleware class scrapy.contrib.downloadermiddleware.robotstxt.RobotsTxtMiddleware This middleware lters out requests forbidden by the robots.txt exclusion standard. To make sure Scrapy respects robots.txt make sure the middleware is enabled and the ROBOTSTXT_OBEY setting is enabled. Warning: Keep in mind that, if you crawl using multiple concurrent requests per domain, Scrapy could still download some forbidden pages if they were requested before the robots.txt le was downloaded. This is a known limitation of the current robots.txt middleware and will be xed in the future.
DownloaderStats class scrapy.contrib.downloadermiddleware.stats.DownloaderStats Middleware that stores stats of all requests, responses and exceptions that pass through it. To use this middleware you must enable the DOWNLOADER_STATS setting. UserAgentMiddleware class scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware Middleware that allows spiders to override the default user agent. In order for a spider to override the default user agent, its user_agent attribute must be set.
128
AjaxCrawlMiddleware class scrapy.contrib.downloadermiddleware.ajaxcrawl.AjaxCrawlMiddleware Middleware that nds AJAX crawlable page variants based on meta-fragment html tag. https://developers.google.com/webmasters/ajax-crawling/docs/getting-started for more info.
See
Note: Scrapy nds AJAX crawlable pages for URLs like http://example.com/!#foo=bar even without this middleware. AjaxCrawlMiddleware is necessary when URL doesnt contain !#. This is often a case for index or main website pages.
AjaxCrawlMiddleware Settings
AJAXCRAWL_ENABLED New in version 0.21. Default: False Whether the AjaxCrawlMiddleware will be enabled. You may want to enable it for broad crawls.
The SPIDER_MIDDLEWARES setting is merged with the SPIDER_MIDDLEWARES_BASE setting dened in Scrapy (and not meant to be overridden) and then sorted by order to get the nal sorted list of enabled middlewares: the rst middleware is the one closer to the engine and the last is the one closer to the spider. To decide which order to assign to your middleware see the SPIDER_MIDDLEWARES_BASE setting and pick a value according to where you want to insert the middleware. The order does matter because each middleware performs a different action and your middleware could depend on some previous (or subsequent) middleware being applied. If you want to disable a builtin middleware (the ones dened in SPIDER_MIDDLEWARES_BASE, and enabled by default) you must dene it in your project SPIDER_MIDDLEWARES setting and assign None as its value. For example, if you want to disable the off-site middleware:
SPIDER_MIDDLEWARES = { myproject.middlewares.CustomSpiderMiddleware: 543, scrapy.contrib.spidermiddleware.offsite.OffsiteMiddleware: None, }
Finally, keep in mind that some middlewares may need to be enabled through a particular setting. See each middleware documentation for more info.
129
130
process_start_requests(start_requests, spider) New in version 0.15. This method is called with the start requests of the spider, and works similarly to the process_spider_output() method, except that it doesnt have a response associated and must return only requests (not items). It receives an iterable (in the start_requests parameter) and must return another iterable of Request objects. Note: When implementing this method in your spider middleware, you should always return an iterable (that follows the input one) and not consume all start_requests iterator because it can be very large (or even unbounded) and cause a memory overow. The Scrapy engine is designed to pull start requests while it has capacity to process them, so the start requests iterator can be effectively endless where there is some other condition for stopping the spider (like a time limit or item/page count). Parameters start_requests (an iterable of Request) the start requests spider (Spider object) the spider to whom the start requests belong
131
The handle_httpstatus_list key of Request.meta can also be used to specify which response codes to allow on a per-request basis. Keep in mind, however, that its usually a bad idea to handle non-200 responses, unless you really know what youre doing. For more information see: HTTP Status Code Denitions.
HttpErrorMiddleware settings
HTTPERROR_ALLOWED_CODES Default: [] Pass all responses with non-200 status codes contained in this list. HTTPERROR_ALLOW_ALL Default: False Pass all responses, regardless of its status code. OffsiteMiddleware class scrapy.contrib.spidermiddleware.offsite.OffsiteMiddleware Filters out Requests for URLs outside the domains covered by the spider. This middleware lters out every request whose host names arent in the spiders allowed_domains attribute. When your spider returns a request for a domain not belonging to those covered by the spider, this middleware will log a debug message similar to this one:
To avoid lling the log with too much noise, it will only print one of these messages for each new domain ltered. So, for example, if another request for www.othersite.com is ltered, no log message will be printed. But if a request for someothersite.com is ltered, a message will be printed (but only for the rst request ltered). If the spider doesnt dene an allowed_domains attribute, or the attribute is empty, the offsite middleware will allow all requests. If the request has the dont_filter attribute set, the offsite middleware will allow the request even if its domain is not listed in allowed domains. RefererMiddleware class scrapy.contrib.spidermiddleware.referer.RefererMiddleware Populates Request Referer header, based on the URL of the Response which generated it.
RefererMiddleware settings
132
Whether to enable referer middleware. UrlLengthMiddleware class scrapy.contrib.spidermiddleware.urllength.UrlLengthMiddleware Filters out requests with URLs longer than URLLENGTH_LIMIT The UrlLengthMiddleware can be congured through the following settings (see the settings documentation for more info): URLLENGTH_LIMIT - The maximum URL length to allow for crawled URLs.
6.4 Extensions
The extensions framework provides a mechanism for inserting your own custom functionality into Scrapy. Extensions are just regular classes that are instantiated at Scrapy startup, when extensions are initialized.
As you can see, the EXTENSIONS setting is a dict where the keys are the extension paths, and their values are the orders, which dene the extension loading order. Extensions orders are not as important as middleware orders though, and they are typically irrelevant, ie. it doesnt matter in which order the extensions are loaded because they dont depend on each other [1]. However, this feature can be exploited if you need to add an extension which depends on other extensions already loaded. [1] This is is why the EXTENSIONS_BASE setting in Scrapy (which contains all built-in extensions enabled by default) denes all the extensions with the same order (500).
6.4. Extensions
133
134
# NotConfigured otherwise if not crawler.settings.getbool(MYEXT_ENABLED): raise NotConfigured # get the number of items from settings item_count = crawler.settings.getint(MYEXT_ITEMCOUNT, 1000) # instantiate the extension object ext = cls(item_count) # connect the extension object to signals crawler.signals.connect(ext.spider_opened, signal=signals.spider_opened) crawler.signals.connect(ext.spider_closed, signal=signals.spider_closed) crawler.signals.connect(ext.item_scraped, signal=signals.item_scraped) # return the extension object return ext def spider_opened(self, spider): spider.log("opened spider %s" % spider.name) def spider_closed(self, spider): spider.log("closed spider %s" % spider.name) def item_scraped(self, item, spider): self.items_scraped += 1 if self.items_scraped == self.item_count: spider.log("scraped %d items, resetting counter" % self.items_scraped) self.item_count = 0
class scrapy.contrib.logstats.LogStats Log basic stats like crawled pages and scraped items.
Core Stats extension
class scrapy.contrib.corestats.CoreStats Enable the collection of core statistics, provided the stats collection is enabled (see Stats Collection).
Web service extension
6.4. Extensions
135
class scrapy.telnet.TelnetConsole Provides a telnet console for getting into a Python interpreter inside the currently running Scrapy process, which can be very useful for debugging. The telnet console must be enabled by the TELNETCONSOLE_ENABLED setting, and the server will listen in the port specied in TELNETCONSOLE_PORT.
Memory usage extension
class scrapy.contrib.memusage.MemoryUsage Note: This extension does not work in Windows. Monitors the memory used by the Scrapy process that runs the spider and: 1, sends a notication e-mail when it exceeds a certain value 2. closes the spider when it exceeds a certain value The notication e-mails can be triggered when a certain warning value is reached (MEMUSAGE_WARNING_MB) and when the maximum value is reached (MEMUSAGE_LIMIT_MB) which will also cause the spider to be closed and the Scrapy process to be terminated. This extension is enabled by the MEMUSAGE_ENABLED setting and can be congured with the following settings: MEMUSAGE_LIMIT_MB MEMUSAGE_WARNING_MB MEMUSAGE_NOTIFY_MAIL MEMUSAGE_REPORT
Memory debugger extension
class scrapy.contrib.memdebug.MemoryDebugger An extension for debugging memory usage. It collects information about: objects uncollected by the Python garbage collector objects left alive that shouldnt. For more info, see Debugging memory leaks with trackref To enable this extension, turn on the MEMDEBUG_ENABLED setting. The info will be stored in the stats.
Close spider extension
class scrapy.contrib.closespider.CloseSpider Closes a spider automatically when some conditions are met, using a specic closing reason for each condition. The conditions for closing a spider can be congured through the following settings: CLOSESPIDER_TIMEOUT CLOSESPIDER_ITEMCOUNT CLOSESPIDER_PAGECOUNT
136
CLOSESPIDER_ERRORCOUNT CLOSESPIDER_TIMEOUT Default: 0 An integer which species a number of seconds. If the spider remains open for more than that number of second, it will be automatically closed with the reason closespider_timeout. If zero (or non set), spiders wont be closed by timeout. CLOSESPIDER_ITEMCOUNT Default: 0 An integer which species a number of items. If the spider scrapes more than that amount if items and those items are passed by the item pipeline, the spider will be closed with the reason closespider_itemcount. If zero (or non set), spiders wont be closed by number of passed items. CLOSESPIDER_PAGECOUNT New in version 0.11. Default: 0 An integer which species the maximum number of responses to crawl. If the spider crawls more than that, the spider will be closed with the reason closespider_pagecount. If zero (or non set), spiders wont be closed by number of crawled responses. CLOSESPIDER_ERRORCOUNT New in version 0.11. Default: 0 An integer which species the maximum number of errors to receive before closing the spider. If the spider generates more than that number of errors, it will be closed with the reason closespider_errorcount. If zero (or non set), spiders wont be closed by number of errors.
StatsMailer extension
class scrapy.contrib.statsmailer.StatsMailer This simple extension can be used to send a notication e-mail every time a domain has nished scraping, including the Scrapy stats collected. The email will be sent to all recipients specied in the STATSMAILER_RCPTS setting. Debugging extensions
Stack trace dump extension
class scrapy.contrib.debug.StackTraceDump Dumps information about the running process when a SIGQUIT or SIGUSR2 signal is received. The information dumped is the following: 1. engine status (using scrapy.utils.engine.get_engine_status()) 2. live references (see Debugging memory leaks with trackref ) 3. stack trace of all threads
6.4. Extensions
137
After the stack trace and engine status is dumped, the Scrapy process continues running normally. This extension only works on POSIX-compliant platforms (ie. not Windows), because the SIGQUIT and SIGUSR2 signals are not available on Windows. There are at least two ways to send Scrapy the SIGQUIT signal: 1. By pressing Ctrl-while a Scrapy process is running (Linux only?) 2. By running this command (assuming <pid> is the process id of the Scrapy process):
kill -QUIT <pid>
Debugger extension
class scrapy.contrib.debug.Debugger Invokes a Python debugger inside a running Scrapy process when a SIGUSR2 signal is received. After the debugger is exited, the Scrapy process continues running normally. For more info see Debugging in Python. This extension only works on POSIX-compliant platforms (ie. not Windows).
138
stats The stats collector of this crawler. This is used from extensions & middlewares to record stats of their behaviour, or access stats collected by other extensions. For an introduction on stats collection see Stats Collection. For the API see StatsCollector class. extensions The extension manager that keeps track of enabled extensions. Most extensions wont need to access this attribute. For an introduction on extensions and a list of available extensions on Scrapy see Extensions. spiders The spider manager which takes care of loading and instantiating spiders. Most extensions wont need to access this attribute. engine The execution engine, which coordinates the core crawling logic between the scheduler, downloader and spiders. Some extension may want to access the Scrapy engine, to modify inspect or modify the downloader and scheduler behaviour, although this is an advanced use and this API is not yet stable. configure() Congure the crawler. This loads extensions, middlewares and spiders, leaving the crawler ready to be started. It also congures the execution engine. start() Start the crawler. This calls configure() if it hasnt been called yet. Returns a deferred that is red when the crawl is nished.
139
getbool(name, default=False) Get a setting value as a boolean. For example, both 1 and 1, and True return True, while 0, 0, False and None return False For example, settings populated through environment variables set to 0 will return False when using this method. Parameters name (string) the setting name default (any) the value to return if no setting is found getint(name, default=0) Get a setting value as an int Parameters name (string) the setting name default (any) the value to return if no setting is found getfloat(name, default=0.0) Get a setting value as a oat Parameters name (string) the setting name default (any) the value to return if no setting is found getlist(name, default=None) Get a setting value as a list. If the setting original type is a list it will be returned verbatim. If its a string it will be split by ,. For example, settings populated through environment variables set to one,two will return a list [one, two] when using this method. Parameters name (string) the setting name default (any) the value to return if no setting is found
140
send_catch_log_deferred(signal, **kwargs) Like send_catch_log() but supports returning deferreds from signal handlers. Returns a deferred that gets red once all signal handlers deferreds were red. Send a signal, catch exceptions and log them. The keyword arguments are passed to the signal handlers (connected through the connect() method). disconnect(receiver, signal) Disconnect a receiver function from a signal. This has the opposite effect of the connect() method, and the arguments are the same. disconnect_all(signal) Disconnect all receivers from the given signal. Parameters signal (object) the signal to disconnect from
141
Downloader Middleware Customize how pages get requested and downloaded. Spider Middleware Customize the input and output of your spiders. Extensions Extend Scrapy with your custom functionality Core API Use it on extensions and middlewares to extend Scrapy functionality
142
CHAPTER 7
Reference
143
1. Using a dict:
request_with_cookies = Request(url="http://www.example.com", cookies={currency: USD, country: UY})
The latter form allows for customizing the domain and path attributes of the cookie. These is only useful if the cookies are saved for later requests. When some site returns cookies (in a response) those are stored in the cookies for that domain and will be sent again in future requests. Thats the typical behaviour of any regular web browser. However, if, for some reason, you want to avoid merging with existing cookies you can instruct Scrapy to do so by setting the dont_merge_cookies key in the Request.meta. Example of request without merging cookies:
request_with_cookies = Request(url="http://www.example.com", cookies={currency: USD, country: UY}, meta={dont_merge_cookies: True})
For more info see CookiesMiddleware. encoding (string) the encoding of this request (defaults to utf-8). This encoding will be used to percent-encode the URL and to convert the body to str (if given as unicode). priority (int) the priority of this request (defaults to 0). The priority is used by the scheduler to dene the order used to process requests. dont_lter (boolean) indicates that this request should not be ltered by the scheduler. This is used when you want to perform an identical request multiple times, to ignore the duplicates lter. Use it with care, or you will get into crawling loops. Default to False. errback (callable) a function that will be called if any exception was raised while processing the request. This includes pages that failed with 404 HTTP errors and such. It receives a Twisted Failure instance as rst parameter. url A string containing the URL of this request. Keep in mind that this attribute contains the escaped URL, so it can differ from the URL passed in the constructor. This attribute is read-only. To change the URL of a Request use replace(). method A string representing the HTTP method in the request. This is guaranteed to be uppercase. Example: "GET", "POST", "PUT", etc headers A dictionary-like object which contains the request headers. body A str that contains the request body. This attribute is read-only. To change the body of a Request use replace(). 144 Chapter 7. Reference
meta A dict that contains arbitrary metadata for this request. This dict is empty for new Requests, and is usually populated by different Scrapy components (extensions, middlewares, etc). So the data contained in this dict depends on the extensions you have enabled. See Request.meta special keys for a list of special meta keys recognized by Scrapy. This dict is shallow copied when the request is cloned using the copy() or replace() methods, and can also be accessed, in your spider, from the response.meta attribute. copy() Return a new Request which is a copy of this Request. See also: Passing additional data to callback functions. replace([url, method, headers, body, cookies, meta, encoding, dont_lter, callback, errback ]) Return a Request object with the same members, except for those members given new values by whichever keyword arguments are specied. The attribute Request.meta is copied by default (unless a new value is given in the meta argument). See also Passing additional data to callback functions. Passing additional data to callback functions The callback of a request is a function that will be called when the response of that request is downloaded. The callback function will be called with the downloaded Response object as its rst argument. Example:
def parse_page1(self, response): return Request("http://www.example.com/some_page.html", callback=self.parse_page2) def parse_page2(self, response): # this would log http://www.example.com/some_page.html self.log("Visited %s" % response.url)
In some cases you may be interested in passing arguments to those callback functions so you can receive the arguments later, in the second callback. You can use the Request.meta attribute for that. Heres an example of how to pass an item using this mechanism, to populate different elds from different pages:
def parse_page1(self, response): item = MyItem() item[main_url] = response.url request = Request("http://www.example.com/some_page.html", callback=self.parse_page2) request.meta[item] = item return request def parse_page2(self, response): item = response.meta[item] item[other_url] = response.url return item
145
dont_redirect dont_retry handle_httpstatus_list dont_merge_cookies (see cookies parameter of Request constructor) cookiejar redirect_urls bindaddress bindaddress The IP of the outgoing IP address to use for the performing the request.
146
Chapter 7. Reference
formnumber (integer) the number of form to use, when the response contains multiple forms. The rst one (and also the default) is 0. formdata (dict) elds to override in the form data. If a eld was already present in the response <form> element, its value is overridden by the one passed in this parameter. dont_click (boolean) If True, the form data will be submitted without clicking in any element. The other parameters of this class method are passed directly to the FormRequest constructor. New in version 0.10.3: The formname parameter. New in version 0.17: The formxpath parameter. Request usage examples
Using FormRequest to send data via HTTP POST
If you want to simulate a HTML Form POST in your spider and send a couple of key-value elds, you can return a FormRequest object (from your spider) like this:
return [FormRequest(url="http://www.example.com/post/action", formdata={name: John Doe, age: 27}, callback=self.after_post)]
It is usual for web sites to provide pre-populated form elds through <input type="hidden"> elements, such as session related data or authentication tokens (for login pages). When scraping, youll want these elds to be automatically pre-populated and only override a couple of them, such as the user name and password. You can use the FormRequest.from_response() method for this job. Heres an example spider which uses it:
class LoginSpider(Spider): name = example.com start_urls = [http://www.example.com/users/login.php] def parse(self, response): return [FormRequest.from_response(response, formdata={username: john, password: secret}, callback=self.after_login)] def after_login(self, response): # check login succeed before going on if "authentication failed" in response.body: self.log("Login failed", level=log.ERROR) return # continue scraping with authenticated session...
147
Parameters url (string) the URL of this response headers (dict) the headers of this response. The dict values can be strings (for single valued headers) or lists (for multi-valued headers). status (integer) the HTTP status of the response. Defaults to 200. body (str) the response body. It must be str, not unicode, unless youre using a encodingaware Response subclass, such as TextResponse. meta (dict) the initial values for the Response.meta attribute. If given, the dict will be shallow copied. ags (list) is a list containing the initial values for the Response.flags attribute. If given, the list will be shallow copied. url A string containing the URL of the response. This attribute is read-only. To change the URL of a Response use replace(). status An integer representing the HTTP status of the response. Example: 200, 404. headers A dictionary-like object which contains the response headers. body A str containing the body of this Response. Keep in mind that Reponse.body is always a str. If you want the unicode version use TextResponse.body_as_unicode() (only available in TextResponse and subclasses). This attribute is read-only. To change the body of a Response use replace(). request The Request object that generated this response. This attribute is assigned in the Scrapy engine, after the response and the request have passed through all Downloader Middlewares. In particular, this means that: HTTP redirections will cause the original request (to the URL before redirection) to be assigned to the redirected response (with the nal URL after redirection). Response.request.url doesnt always equal Response.url This attribute is only available in the spider code, and in the Spider Middlewares, but not in Downloader Middlewares (although you have the Request available there by other means) and handlers of the response_downloaded signal. meta A shortcut to the Request.meta self.request.meta). attribute of the Response.request object (ie.
Unlike the Response.request attribute, the Response.meta attribute is propagated along redirects and retries, so you will get the original Request.meta sent from your spider. See also: Request.meta attribute flags A list that contains ags for this response. Flags are labels used for tagging Responses. For example:
148
Chapter 7. Reference
cached, redirected, etc. And theyre shown on the string representation of the Response (__str__ method) which is used by the engine for logging. copy() Returns a new Response which is a copy of this Response. replace([url, status, headers, body, request, ags, cls ]) Returns a Response object with the same members, except for those members given new values by whichever keyword arguments are specied. The attribute Response.meta is copied by default.
Since, in the latter case, you would be using you system default encoding (typically ascii) to convert the body to unicode, instead of the response encoding.
149
HtmlResponse objects class scrapy.http.HtmlResponse(url[, ... ]) The HtmlResponse class is a subclass of TextResponse which adds encoding auto-discovering support by looking into the HTML meta http-equiv attribute. See TextResponse.encoding. XmlResponse objects class scrapy.http.XmlResponse(url[, ... ]) The XmlResponse class is a subclass of TextResponse which adds encoding auto-discovering support by looking into the XML declaration line. See TextResponse.encoding.
7.2 Settings
The Scrapy settings allows you to customize the behaviour of all Scrapy components, including the core, extensions, pipelines and spiders themselves. The infrastructure of the settings provides a global namespace of key-value mappings that the code can use to pull conguration values from. The settings can be populated through different mechanisms, which are described below. The settings are also the mechanism for selecting the currently active Scrapy project (in case you have many). For a list of available built-in settings see: Built-in settings reference.
150
Chapter 7. Reference
Example:
scrapy crawl myspider -s LOG_FILE=scrapy.log
2. Project settings module The project settings module is the standard conguration le for your Scrapy project. Its where most of your custom settings will be populated. For example:: myproject.settings. 3. Default settings per-command Each Scrapy tool command can have its own default settings, which override the global default settings. Those custom command settings are specied in the default_settings attribute of the command class. 4. Default global settings The global defaults are located in the scrapy.settings.default_settings module and documented in the Built-in settings reference section.
In other words, settings can be accessed like a dict, but its usually preferred to extract the setting in the format you need it to avoid type errors. In order to do that youll have to use one of the methods provided the Settings API.
7.2. Settings
151
AWS_ACCESS_KEY_ID Default: None The AWS access key used by code that requires access to Amazon Web services, such as the S3 feed storage backend.
AWS_SECRET_ACCESS_KEY Default: None The AWS secret key used by code that requires access to Amazon Web services, such as the S3 feed storage backend. BOT_NAME Default: scrapybot The name of the bot implemented by this Scrapy project (also known as the project name). This will be used to construct the User-Agent by default, and also for logging. Its automatically populated with your project name when you create your project with the startproject command. CONCURRENT_ITEMS Default: 100 Maximum number of concurrent items (per response) to process in parallel in the Item Processor (also known as the Item Pipeline). CONCURRENT_REQUESTS Default: 16 The maximum number of concurrent (ie. simultaneous) requests that will be performed by the Scrapy downloader. CONCURRENT_REQUESTS_PER_DOMAIN Default: 8 The maximum number of concurrent (ie. simultaneous) requests that will be performed to any single domain. CONCURRENT_REQUESTS_PER_IP Default: 0 The maximum number of concurrent (ie. simultaneous) requests that will be performed to any single IP. If nonzero, the CONCURRENT_REQUESTS_PER_DOMAIN setting is ignored, and this one is used instead. In other words, concurrency limits will be applied per IP, not per domain. This setting also affects DOWNLOAD_DELAY: if CONCURRENT_REQUESTS_PER_IP is non-zero, download delay is enforced per IP, not per domain.
152
Chapter 7. Reference
DEFAULT_ITEM_CLASS Default: scrapy.item.Item The default class that will be used for instantiating items in the the Scrapy shell. DEFAULT_REQUEST_HEADERS Default:
{ Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8, Accept-Language: en, }
The default headers used for Scrapy HTTP Requests. Theyre populated in the DefaultHeadersMiddleware. DEPTH_LIMIT Default: 0 The maximum depth that will be allowed to crawl for any site. If zero, no limit will be imposed. DEPTH_PRIORITY Default: 0 An integer that is used to adjust the request priority based on its depth. If zero, no priority adjustment is made from depth. DEPTH_STATS Default: True Whether to collect maximum depth stats. DEPTH_STATS_VERBOSE Default: False Whether to collect verbose depth stats. If this is enabled, the number of requests for each depth is collected in the stats. DNSCACHE_ENABLED Default: True Whether to enable DNS in-memory cache.
7.2. Settings
153
DOWNLOADER_DEBUG Default: False Whether to enable the Downloader debugging mode. DOWNLOADER_MIDDLEWARES Default:: {} A dict containing the downloader middlewares enabled in your project, and their orders. For more info see Activating a downloader middleware. DOWNLOADER_MIDDLEWARES_BASE Default:
{ scrapy.contrib.downloadermiddleware.robotstxt.RobotsTxtMiddleware: 100, scrapy.contrib.downloadermiddleware.httpauth.HttpAuthMiddleware: 300, scrapy.contrib.downloadermiddleware.downloadtimeout.DownloadTimeoutMiddleware: 350, scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware: 400, scrapy.contrib.downloadermiddleware.retry.RetryMiddleware: 500, scrapy.contrib.downloadermiddleware.defaultheaders.DefaultHeadersMiddleware: 550, scrapy.contrib.downloadermiddleware.redirect.MetaRefreshMiddleware: 580, scrapy.contrib.downloadermiddleware.httpcompression.HttpCompressionMiddleware: 590, scrapy.contrib.downloadermiddleware.redirect.RedirectMiddleware: 600, scrapy.contrib.downloadermiddleware.cookies.CookiesMiddleware: 700, scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware: 750, scrapy.contrib.downloadermiddleware.chunked.ChunkedTransferMiddleware: 830, scrapy.contrib.downloadermiddleware.stats.DownloaderStats: 850, scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware: 900, }
A dict containing the downloader middlewares enabled by default in Scrapy. You should never modify this setting in your project, modify DOWNLOADER_MIDDLEWARES instead. For more info see Activating a downloader middleware. DOWNLOADER_STATS Default: True Whether to enable downloader stats collection. DOWNLOAD_DELAY Default: 0 The amount of time (in secs) that the downloader should wait before downloading consecutive pages from the same website. This can be used to throttle the crawling speed to avoid hitting servers too hard. Decimal numbers are supported. Example:
DOWNLOAD_DELAY = 0.25 # 250 ms of delay
154
Chapter 7. Reference
This setting is also affected by the RANDOMIZE_DOWNLOAD_DELAY setting (which is enabled by default). By default, Scrapy doesnt wait a xed amount of time between requests, but uses a random interval between 0.5 and 1.5 * DOWNLOAD_DELAY. When CONCURRENT_REQUESTS_PER_IP is non-zero, delays are enforced per ip address instead of per domain. You can also change this setting per spider by setting download_delay spider attribute. DOWNLOAD_HANDLERS Default: {} A dict containing the request downloader handlers enabled in your project. See DOWNLOAD_HANDLERS_BASE for example format. DOWNLOAD_HANDLERS_BASE Default:
{ file: scrapy.core.downloader.handlers.file.FileDownloadHandler, http: scrapy.core.downloader.handlers.http.HttpDownloadHandler, https: scrapy.core.downloader.handlers.http.HttpDownloadHandler, s3: scrapy.core.downloader.handlers.s3.S3DownloadHandler, }
A dict containing the request download handlers enabled by default in Scrapy. You should never modify this setting in your project, modify DOWNLOAD_HANDLERS instead. DOWNLOAD_TIMEOUT Default: 180 The amount of time (in secs) that the downloader will wait before timing out. DUPEFILTER_CLASS Default: scrapy.dupefilter.RFPDupeFilter The class used to detect and lter duplicate requests. The default (RFPDupeFilter) lters based on scrapy.utils.request.request_fingerprint function. EDITOR Default: depends on the environment The editor to use for editing spiders with the edit command. It defaults to the EDITOR environment variable, if set. Otherwise, it defaults to vi (on Unix systems) or the IDLE editor (on Windows). request ngerprint using the
7.2. Settings
155
EXTENSIONS Default:: {} A dict containing the extensions enabled in your project, and their orders. EXTENSIONS_BASE Default:
{ scrapy.contrib.corestats.CoreStats: 0, scrapy.webservice.WebService: 0, scrapy.telnet.TelnetConsole: 0, scrapy.contrib.memusage.MemoryUsage: 0, scrapy.contrib.memdebug.MemoryDebugger: 0, scrapy.contrib.closespider.CloseSpider: 0, scrapy.contrib.feedexport.FeedExporter: 0, scrapy.contrib.logstats.LogStats: 0, scrapy.contrib.spiderstate.SpiderState: 0, scrapy.contrib.throttle.AutoThrottle: 0, }
The list of available extensions. Keep in mind that some of them need to be enabled through a setting. By default, this setting contains all stable built-in extensions. For more information See the extensions user guide and the list of available extensions. ITEM_PIPELINES Default: {} A dict containing the item pipelines to use, and their orders. The dict is empty by default order values are arbitrary but its customary to dene them in the 0-1000 range. Lists are supported in ITEM_PIPELINES for backwards compatibility, but they are deprecated. Example:
ITEM_PIPELINES = { mybot.pipeline.validate.ValidateMyItem: 300, mybot.pipeline.validate.StoreMyItem: 800, }
ITEM_PIPELINES_BASE Default: {} A dict containing the pipelines enabled by default in Scrapy. You should never modify this setting in your project, modify ITEM_PIPELINES instead. LOG_ENABLED Default: True Whether to enable logging.
156
Chapter 7. Reference
LOG_ENCODING Default: utf-8 The encoding to use for logging. LOG_FILE Default: None File name to use for logging output. If None, standard error will be used. LOG_LEVEL Default: DEBUG Minimum level to log. Available levels are: CRITICAL, ERROR, WARNING, INFO, DEBUG. For more info see Logging. LOG_STDOUT Default: False If True, all standard output (and error) of your process will be redirected to the log. For example if you print hello it will appear in the Scrapy log. MEMDEBUG_ENABLED Default: False Whether to enable memory debugging. MEMDEBUG_NOTIFY Default: [] When memory debugging is enabled a memory report will be sent to the specied addresses if this setting is not empty, otherwise the report will be written to the log. Example:
MEMDEBUG_NOTIFY = [user@example.com]
MEMUSAGE_ENABLED Default: False Scope: scrapy.contrib.memusage Whether to enable the memory usage extension that will shutdown the Scrapy process when it exceeds a memory limit, and also notify by email when that happened. See Memory usage extension.
7.2. Settings
157
MEMUSAGE_LIMIT_MB Default: 0 Scope: scrapy.contrib.memusage The maximum amount of memory to allow (in megabytes) before shutting down Scrapy (if MEMUSAGE_ENABLED is True). If zero, no check will be performed. See Memory usage extension. MEMUSAGE_NOTIFY_MAIL Default: False Scope: scrapy.contrib.memusage A list of emails to notify if the memory limit has been reached. Example:
MEMUSAGE_NOTIFY_MAIL = [user@example.com]
See Memory usage extension. MEMUSAGE_REPORT Default: False Scope: scrapy.contrib.memusage Whether to send a memory usage report after each spider has been closed. See Memory usage extension. MEMUSAGE_WARNING_MB Default: 0 Scope: scrapy.contrib.memusage The maximum amount of memory to allow (in megabytes) before sending a warning email notifying about it. If zero, no warning will be produced. NEWSPIDER_MODULE Default: Module where to create new spiders using the genspider command. Example:
NEWSPIDER_MODULE = mybot.spiders_dev
158
Chapter 7. Reference
RANDOMIZE_DOWNLOAD_DELAY Default: True If enabled, Scrapy will wait a random amount of time (between 0.5 and 1.5 * DOWNLOAD_DELAY) while fetching requests from the same website. This randomization decreases the chance of the crawler being detected (and subsequently blocked) by sites which analyze requests looking for statistically signicant similarities in the time between their requests. The randomization policy is the same used by wget --random-wait option. If DOWNLOAD_DELAY is zero (default) this option has no effect. REDIRECT_MAX_TIMES Default: 20 Denes the maximum times a request can be redirected. After this maximum the requests response is returned as is. We used Firefox default value for the same task. REDIRECT_MAX_METAREFRESH_DELAY Default: 100 Some sites use meta-refresh for redirecting to a session expired page, so we restrict automatic redirection to a maximum delay (in seconds) REDIRECT_PRIORITY_ADJUST Default: +2 Adjust redirect request priority relative to original request. A negative priority adjust means more priority. ROBOTSTXT_OBEY Default: False Scope: scrapy.contrib.downloadermiddleware.robotstxt If enabled, Scrapy will respect robots.txt policies. For more information see RobotsTxtMiddleware SCHEDULER Default: scrapy.core.scheduler.Scheduler The scheduler to use for crawling. SPIDER_CONTRACTS Default:: {} A dict containing the scrapy contracts enabled in your project, used for testing spiders. For more info see Spiders Contracts.
7.2. Settings
159
SPIDER_CONTRACTS_BASE Default:
{ scrapy.contracts.default.UrlContract : 1, scrapy.contracts.default.ReturnsContract: 2, scrapy.contracts.default.ScrapesContract: 3, }
A dict containing the scrapy contracts enabled by default in Scrapy. You should never modify this setting in your project, modify SPIDER_CONTRACTS instead. For more info see Spiders Contracts. SPIDER_MIDDLEWARES Default:: {} A dict containing the spider middlewares enabled in your project, and their orders. For more info see Activating a spider middleware. SPIDER_MIDDLEWARES_BASE Default:
{ scrapy.contrib.spidermiddleware.httperror.HttpErrorMiddleware: 50, scrapy.contrib.spidermiddleware.offsite.OffsiteMiddleware: 500, scrapy.contrib.spidermiddleware.referer.RefererMiddleware: 700, scrapy.contrib.spidermiddleware.urllength.UrlLengthMiddleware: 800, scrapy.contrib.spidermiddleware.depth.DepthMiddleware: 900, }
A dict containing the spider middlewares enabled by default in Scrapy. You should never modify this setting in your project, modify SPIDER_MIDDLEWARES instead. For more info see Activating a spider middleware. SPIDER_MODULES Default: [] A list of modules where Scrapy will look for spiders. Example:
SPIDER_MODULES = [mybot.spiders_prod, mybot.spiders_dev]
STATS_CLASS Default: scrapy.statscol.MemoryStatsCollector The class to use for collecting stats, who must implement the Stats Collector API .
160
Chapter 7. Reference
STATS_DUMP Default: True Dump the Scrapy stats (to the Scrapy log) once the spider nishes. For more info see: Stats Collection. STATSMAILER_RCPTS Default: [] (empty list) Send Scrapy stats after spiders nish scraping. See StatsMailer for more info. TELNETCONSOLE_ENABLED Default: True A boolean which species if the telnet console will be enabled (provided its extension is also enabled). TELNETCONSOLE_PORT Default: [6023, 6073] The port range to use for the telnet console. If set to None or 0, a dynamically assigned port is used. For more info see Telnet Console. TEMPLATES_DIR Default: templates dir inside scrapy module The directory where to look for templates when creating new projects with startproject command. URLLENGTH_LIMIT Default: 2083 Scope: contrib.spidermiddleware.urllength The maximum URL length to allow for crawled URLs. For more information about the default value for this setting see: http://www.boutell.com/newfaq/misc/urllength.html USER_AGENT Default: "Scrapy/VERSION (+http://scrapy.org)" The default User-Agent to use when crawling, unless overridden.
7.2. Settings
161
7.3 Signals
Scrapy uses signals extensively to notify when certain events occur. You can catch some of those signals in your Scrapy project (using an extension, for example) to perform additional tasks or extend Scrapy to add functionality not provided out of the box. Even though signals provide several arguments, the handlers that catch them dont need to accept all of them - the signal dispatching mechanism will only deliver the arguments that the handler receives. You can connect to signals (or send your own) through the Signals API .
engine_stopped scrapy.signals.engine_stopped() Sent when the Scrapy engine is stopped (for example, when a crawling process has nished). This signal supports returning deferreds from their handlers. item_scraped scrapy.signals.item_scraped(item, response, spider) Sent when an item has been scraped, after it has passed all the Item Pipeline stages (without being dropped). This signal supports returning deferreds from their handlers. Parameters item (Item object) the item scraped response (Response object) the response from where the item was scraped spider (Spider object) the spider which scraped the item
162
Chapter 7. Reference
item_dropped scrapy.signals.item_dropped(item, spider, exception) Sent after an item has been dropped from the Item Pipeline when some stage raised a DropItem exception. This signal supports returning deferreds from their handlers. Parameters item (Item object) the item dropped from the Item Pipeline spider (Spider object) the spider which scraped the item exception (DropItem exception) the exception (which must be a DropItem subclass) which caused the item to be dropped spider_closed scrapy.signals.spider_closed(spider, reason) Sent after a spider has been closed. This can be used to release per-spider resources reserved on spider_opened. This signal supports returning deferreds from their handlers. Parameters spider (Spider object) the spider which has been closed reason (str) a string which describes the reason why the spider was closed. If it was closed because the spider has completed scraping, the reason is finished. Otherwise, if the spider was manually closed by calling the close_spider engine method, then the reason is the one passed in the reason argument of that method (which defaults to cancelled). If the engine was shutdown (for example, by hitting Ctrl-C to stop it) the reason will be shutdown. spider_opened scrapy.signals.spider_opened(spider) Sent after a spider has been opened for crawling. This is typically used to reserve per-spider resources, but can be used for any task that needs to be performed when a spider is opened. This signal supports returning deferreds from their handlers. Parameters spider (Spider object) the spider which has been opened spider_idle scrapy.signals.spider_idle(spider) Sent when a spider has gone idle, which means the spider has no further: requests waiting to be downloaded requests scheduled items being processed in the item pipeline If the idle state persists after all handlers of this signal have nished, the engine starts closing the spider. After the spider has nished closing, the spider_closed signal is sent.
7.3. Signals
163
You can, for example, schedule some requests in your spider_idle handler to prevent the spider from being closed. This signal does not support returning deferreds from their handlers. Parameters spider (Spider object) the spider which has gone idle spider_error scrapy.signals.spider_error(failure, response, spider) Sent when a spider callback generates an error (ie. raises an exception). Parameters failure (Failure object) the exception raised as a Twisted Failure object response (Response object) the response being processed when the exception was raised spider (Spider object) the spider which raised the exception response_received scrapy.signals.response_received(response, request, spider) Sent when the engine receives a new Response from the downloader. This signal does not support returning deferreds from their handlers. Parameters response (Response object) the response received request (Request object) the request that generated the response spider (Spider object) the spider for which the response is intended response_downloaded scrapy.signals.response_downloaded(response, request, spider) Sent by the downloader right after a HTTPResponse is downloaded. This signal does not support returning deferreds from their handlers. Parameters response (Response object) the response downloaded request (Request object) the request that generated the response spider (Spider object) the spider for which the response is intended
7.4 Exceptions
7.4.1 Built-in Exceptions reference
Heres a list of all exceptions included in Scrapy and their usage.
164
Chapter 7. Reference
DropItem exception scrapy.exceptions.DropItem The exception that must be raised by item pipeline stages to stop processing an Item. For more information see Item Pipeline. CloseSpider exception scrapy.exceptions.CloseSpider(reason=cancelled) This exception can be raised from a spider callback to request the spider to be closed/stopped. Supported arguments: Parameters reason (str) the reason for closing For example:
def parse_page(self, response): if Bandwidth exceeded in response.body: raise CloseSpider(bandwidth_exceeded)
IgnoreRequest exception scrapy.exceptions.IgnoreRequest This exception can be raised by the Scheduler or any downloader middleware to indicate that the request should be ignored. NotCongured exception scrapy.exceptions.NotConfigured This exception can be raised by some components to indicate that they will remain disabled. Those components include: Extensions Item pipelines Downloader middlwares Spider middlewares The exception must be raised in the component constructor. NotSupported exception scrapy.exceptions.NotSupported This exception is raised to indicate an unsupported feature.
7.4. Exceptions
165
166
Chapter 7. Reference
2. Overriding the serialize_eld() method You can also override the serialize_field() method to customize how your eld value will be exported. Make sure you call the base class serialize_field() method after your custom code. Example:
from scrapy.contrib.exporter import XmlItemExporter class ProductXmlExporter(XmlItemExporter): def serialize_field(self, field, name, value): if field == price: return $ %s % str(value) return super(Product, self).serialize_field(field, name, value)
BaseItemExporter class scrapy.contrib.exporter.BaseItemExporter(elds_to_export=None, export_empty_elds=False, encoding=utf8) This is the (abstract) base class for all Item Exporters. It provides support for common features used by all 7.5. Item Exporters 167
(concrete) Item Exporters, such as dening what elds to export, whether to export empty elds, or which encoding to use. These features can be congured through the constructor arguments which populate their respective instance attributes: fields_to_export, export_empty_fields, encoding. export_item(item) Exports the given item. This method must be implemented in subclasses. serialize_field(eld, name, value) Return the serialized value for the given eld. You can override this method (in your custom Item Exporters) if you want to control how a particular eld or value will be serialized/exported. By default, this method looks for a serializer declared in the item eld and returns the result of applying that serializer to the value. If no serializer is found, it returns the value unchanged except for unicode values which are encoded to str using the encoding declared in the encoding attribute. Parameters eld (Field object) the eld being serialized name (str) the name of the eld being serialized value the value being serialized start_exporting() Signal the beginning of the exporting process. Some exporters may use this to generate some required header (for example, the XmlItemExporter). You must call this method before exporting any items. finish_exporting() Signal the end of the exporting process. Some exporters may use this to generate some required footer (for example, the XmlItemExporter). You must always call this method after you have no more items to export. fields_to_export A list with the name of the elds that will be exported, or None if you want to export all elds. Defaults to None. Some exporters (like CsvItemExporter) respect the order of the elds dened in this attribute. export_empty_fields Whether to include empty/unpopulated item elds in the exported data. Defaults to False. Some exporters (like CsvItemExporter) ignore this attribute and always export all empty elds. encoding The encoding that will be used to encode unicode values. This only affects unicode values (which are always serialized to str using this encoding). Other value types are passed unchanged to the specic serialization library. XmlItemExporter class scrapy.contrib.exporter.XmlItemExporter(le, item_element=item, root_element=items, **kwargs) Exports Items in XML format to the specied le object. Parameters le the le-like object to use for exporting the data. root_element (str) The name of root element in the exported XML. item_element (str) The name of each item element in the exported XML.
168
Chapter 7. Reference
The additional keyword arguments of this constructor are passed to the BaseItemExporter constructor. A typical output of this exporter would be:
<?xml version="1.0" encoding="utf-8"?> <items> <item> <name>Color TV</name> <price>1200</price> </item> <item> <name>DVD player</name> <price>200</price> </item> </items>
Unless overridden in the serialize_field() method, multi-valued elds are exported by serializing each value inside a <value> element. This is for convenience, as multi-valued elds are very common. For example, the item:
Item(name=[John, Doe], age=23)
CsvItemExporter class scrapy.contrib.exporter.CsvItemExporter(le, include_headers_line=True, join_multivalued=, , **kwargs) Exports Items in CSV format to the given le-like object. If the fields_to_export attribute is set, it will be used to dene the CSV columns and their order. The export_empty_fields attribute has no effect on this exporter. Parameters le the le-like object to use for exporting the data. include_headers_line (str) If enabled, makes the exporter output a header line with the eld names taken from BaseItemExporter.fields_to_export or the rst exported item elds. join_multivalued The char (or chars) that will be used for joining multi-valued elds, if found. The additional keyword arguments of this constructor are passed to the BaseItemExporter constructor, and the leftover arguments to the csv.writer constructor, so you can use any csv.writer constructor argument to customize this exporter. A typical output of this exporter would be:
169
PickleItemExporter class scrapy.contrib.exporter.PickleItemExporter(le, protocol=0, **kwargs) Exports Items in pickle format to the given le-like object. Parameters le the le-like object to use for exporting the data. protocol (int) The pickle protocol to use. For more information, refer to the pickle module documentation. The additional keyword arguments of this constructor are passed to the BaseItemExporter constructor. Pickle isnt a human readable format, so no output examples are provided. PprintItemExporter class scrapy.contrib.exporter.PprintItemExporter(le, **kwargs) Exports Items in pretty print format to the specied le object. Parameters le the le-like object to use for exporting the data. The additional keyword arguments of this constructor are passed to the BaseItemExporter constructor. A typical output of this exporter would be:
{name: Color TV, price: 1200} {name: DVD player, price: 200}
Longer lines (when present) are pretty-formatted. JsonItemExporter class scrapy.contrib.exporter.JsonItemExporter(le, **kwargs) Exports Items in JSON format to the specied le-like object, writing all objects as a list of objects. The additional constructor arguments are passed to the BaseItemExporter constructor, and the leftover arguments to the JSONEncoder constructor, so you can use any JSONEncoder constructor argument to customize this exporter. Parameters le the le-like object to use for exporting the data. A typical output of this exporter would be:
[{"name": "Color TV", "price": "1200"}, {"name": "DVD player", "price": "200"}]
Warning: JSON is very simple and exible serialization format, but it doesnt scale well for large amounts of data since incremental (aka. stream-mode) parsing is not well supported (if at all) among JSON parsers (on any language), and most of them just parse the entire object in memory. If you want the power and simplicity of JSON with a more stream-friendly format, consider using JsonLinesItemExporter instead, or splitting the output in multiple chunks.
170
Chapter 7. Reference
JsonLinesItemExporter class scrapy.contrib.exporter.JsonLinesItemExporter(le, **kwargs) Exports Items in JSON format to the specied le-like object, writing one JSON-encoded item per line. The additional constructor arguments are passed to the BaseItemExporter constructor, and the leftover arguments to the JSONEncoder constructor, so you can use any JSONEncoder constructor argument to customize this exporter. Parameters le the le-like object to use for exporting the data. A typical output of this exporter would be:
{"name": "Color TV", "price": "1200"} {"name": "DVD player", "price": "200"}
Unlike the one produced by JsonItemExporter, the format produced by this exporter is well suited for serializing large amounts of data. Command line tool Learn about the command-line tool and see all available commands. Requests and Responses Understand the classes used to represent HTTP requests and responses. Settings Learn how to congure Scrapy and see all available settings. Signals See all available signals and how to work with them. Exceptions See all available exceptions and their meaning. Item Exporters Quickly export your scraped items to a le (XML, CSV, etc).
171
172
Chapter 7. Reference
CHAPTER 8
modify the version of scrapy ubuntu package (commit 725900d) x 0.22.0 release date (commit af0219a) x typos in news.rst and remove (not released yet) header (commit b7f58f4)
174
Fix tests runner under pip 1.5 (issue 513) Fix logging error when spider name is unicode (issue 479)
175
Do not silence download errors when request errback raises an exception (commit 684cfc0) Bugxes Fix tests under Django 1.6 (commit b6bed44c) Lot of bugxes to retry middleware under disconnections using HTTP 1.1 download handler Fix inconsistencies among Twisted releases (issue 406) Fix scrapy shell bugs (issue 418, issue 407) Fix invalid variable name in setup.py (issue 429) Fix tutorial references (issue 387) Improve request-response docs (issue 391) Improve best practices docs (issue 399, issue 400, issue 401, issue 402) Improve django integration docs (issue 404) Document bindaddress request meta (commit 37c24e01d7) Improve Request class documentation (issue 226) Other Dropped Python 2.6 support (issue 448) Add cssselect python package as install dependency Drop libxml2 and multi selectors backend support, lxml is required from now on. Minimum Twisted version increased to 10.0.0, dropped Twisted 8.0 support. Running test suite now requires mock python library (issue 390) Thanks Thanks to everyone who contribute to this release! List of contributors sorted by number of commits:
69 37 13 9 9 8 8 6 3 2 2 2 2 2 1 1 1 Daniel Graa <dangra@...> Pablo Hoffman <pablo@...> Mikhail Korobov <kmike84@...> Alex Cepoi <alex.cepoi@...> alexanderlukanin13 <alexander.lukanin.13@...> Rolando Espinoza La fuente <darkrho@...> Lukasz Biedrycki <lukasz.biedrycki@...> Nicolas Ramirez <nramirez.uy@...> Paul Tremberth <paul.tremberth@...> Martin Olveyra <molveyra@...> Stefan <misc@...> Rolando Espinoza <darkrho@...> Loren Davie <loren@...> irgmedeiros <irgmedeiros@...> Stefan Koch <taikano@...> Stefan <cct@...> scraperdragon <dragon@...>
176
1 1 1 1 1 1 1
Kumara Tharmalingam <ktharmal@...> Francesco Piccinno <stack.box@...> Marcos Campal <duendex@...> Dragon Dave <dragon@...> Capi Etheriel <barraponto@...> cacovsky <amarquesferraz@...> Berend Iwema <berend@...>
177
x XmlItemExporter in Python 2.7.4 and 2.7.5 (commit de3e451) minor updates to 0.18 release notes (commit c45e5f1) x contributters list format (commit 0b60031)
178
Add scrapy commands using external libraries (issue 260) Added --pdb option to scrapy command line tool Added XPathSelector.remove_namespaces() which allows to remove all namespaces from XML documents for convenience (to work with namespace-less XPaths). Documented in Selectors. Several improvements to spider contracts New default middleware named MetaRefreshMiddldeware that handles meta-refresh html tag redirections, MetaRefreshMiddldeware and RedirectMiddleware have different priorities to address #62 added from_crawler method to spiders added system tests with mock server more improvements to Mac OS compatibility (thanks Alex Cepoi) several more cleanups to singletons and multi-spider support (thanks Nicolas Ramirez) support custom download slots added spider option to shell command. log overridden settings when scrapy starts Thanks to everyone who contribute to this release. Here is a list of contributors sorted by number of commits:
130 97 20 13 12 11 5 4 4 4 3 3 3 3 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Pablo Hoffman <pablo@...> Daniel Graa <dangra@...> Nicols Ramrez <nramirez.uy@...> Mikhail Korobov <kmike84@...> Pedro Faustino <pedrobandim@...> Steven Almeroth <sroth77@...> Rolando Espinoza La fuente <darkrho@...> Michal Danilak <mimino.coder@...> Alex Cepoi <alex.cepoi@...> Alexandr N Zamaraev (aka tonal) <tonal@...> paul <paul.tremberth@...> Martin Olveyra <molveyra@...> Jordi Llonch <llonchj@...> arijitchakraborty <myself.arijit@...> Shane Evans <shane.evans@...> joehillen <joehillen@...> Hart <HartSimha@...> Dan <ellisd23@...> Zuhao Wan <wanzuhao@...> whodatninja <blake@...> vkrest <v.krestiannykov@...> tpeng <pengtaoo@...> Tom Mortimer-Jones <tom@...> Rocio Aramberri <roschegel@...> Pedro <pedro@...> notsobad <wangxiaohugg@...> Natan L <kuyanatan.nlao@...> Mark Grey <mark.grey@...> Luan <luanpab@...> Libor Nenadl <libor.nenadal@...> Juan M Uys <opyate@...> Jonas Brunsgaard <jonas.brunsgaard@...> Ilya Baryshev <baryshev@...> Hasnain Lakhani <m.hasnain.lakhani@...>
179
1 1 1 1 1
Emanuel Schorsch <emschorsch@...> Chris Tilden <chris.tilden@...> Capi Etheriel <barraponto@...> cacovsky <amarquesferraz@...> Berend Iwema <berend@...>
180
StackTraceDump extension: also dump trackref live references (commit fe2ce93) nested items now fully supported in JSON and JSONLines exporters added cookiejar Request meta key to support multiple cookie sessions per spider decoupled encoding detection code to w3lib.encoding, and ported Scrapy code to use that mdule dropped support for Python 2.5. See http://blog.scrapy.org/scrapy-dropping-support-for-python-25 dropped support for Twisted 2.5 added REFERER_ENABLED setting, to control referer middleware changed default user agent to: Scrapy/VERSION (+http://scrapy.org) removed (undocumented) HTMLImageLinkExtractor class from scrapy.contrib.linkextractors.image removed per-spider settings (to be replaced by instantiating multiple crawler objects) USER_AGENT spider attribute will no longer work, use user_agent attribute instead DOWNLOAD_TIMEOUT spider attribute will no longer work, use download_timeout attribute instead removed ENCODING_ALIASES setting, as encoding auto-detection has been moved to the w3lib library promoted DjangoItem to main contrib LogFormatter method now return dicts(instead of strings) to support lazy formatting (issue 164, commit dcef7b0) downloader handlers (DOWNLOAD_HANDLERS setting) now receive settings as the rst argument of the constructor replaced memory usage acounting with (more portable) resource module, removed scrapy.utils.memory module removed signal: scrapy.mail.mail_sent removed TRACK_REFS setting, now trackrefs is always enabled DBM is now the default storage backend for HTTP cache middleware number of log messages (per level) are now tracked through Scrapy stats (stat name: log_count/LEVEL) number received responses are now tracked through Scrapy stats (stat name: response_received_count) removed scrapy.log.started attribute
8.1.17 0.14.4
added precise to supported ubuntu distros (commit b7e46df) xed bug in json-rpc webservice reported in https://groups.google.com/d/topic/scrapyusers/qgVBmFybNAQ/discussion. also removed no longer supported run command from extras/scrapy-ws.py (commit 340fbdb) meta tag attributes for content-type http equiv can be in any order. #123 (commit 0cb68af) replace import Image by more standard from PIL import Image. closes #88 (commit 4d17048) return trial status as bin/runtests.sh exit value. #118 (commit b7b2e7f)
182
8.1.18 0.14.3
forgot to include pydispatch license. #118 (commit fd85f9c) include egg les used by testsuite in source distribution. #118 (commit c897793) update docstring in project template to avoid confusion with genspider command, which may be considered as an advanced feature. refs #107 (commit 2548dcc) added note to docs/topics/rebug.rst about google directory being shut down (commit 668e352) dont discard slot when empty, just save in another dict in order to recycle if needed again. (commit 8e9f607) do not fail handling unicode xpaths in libxml2 backed selectors (commit b830e95) xed minor mistake in Request objects documentation (commit bf3c9ee) xed minor defect in link extractors documentation (commit ba14f38) removed some obsolete remaining code related to sqlite support in scrapy (commit 0665175)
8.1.19 0.14.2
move buffer pointing to start of le before computing checksum. refs #92 (commit 6a5bef2) Compute image checksum before persisting images. closes #92 (commit 9817df1) remove leaking references in cached failures (commit 673a120) xed bug in MemoryUsage extension: get_engine_status() takes exactly 1 argument (0 given) (commit 11133e9) xed struct.error on http compression middleware. closes #87 (commit 1423140) ajax crawling wasnt expanding for unicode urls (commit 0de3fb4) Catch start_requests iterator errors. refs #83 (commit 454a21d) Speed-up libxml2 XPathSelector (commit 2fbd662) updated versioning doc according to recent changes (commit 0a070f5) scrapyd: xed documentation link (commit 2b4e4c3) extras/makedeb.py: no longer obtaining version from git (commit caffe0e)
8.1.20 0.14.1
extras/makedeb.py: no longer obtaining version from git (commit caffe0e) bumped version to 0.14.1 (commit 6cb9e1c) xed reference to tutorial directory (commit 4b86bd6) doc: removed duplicated callback argument from Request.replace() (commit 1aeccdd) xed formatting of scrapyd doc (commit 8bf19e6) Dump stacks for all running threads and x engine status dumped by StackTraceDump extension (commit 14a8e6e) added comment about why we disable ssl on boto images upload (commit 5223575) SSL handshaking hangs when doing too many parallel connections to S3 (commit 63d583d) change tutorial to follow changes on dmoz site (commit bcb3198)
183
Avoid _disconnectedDeferred AttributeError exception in Twisted>=11.1.0 (commit 98f3f87) allow spider to set autothrottle max concurrency (commit 175a4b5)
8.1.21 0.14
New features and settings Support for AJAX crawleable urls New persistent scheduler that stores requests on disk, allowing to suspend and resume crawls (r2737) added -o option to scrapy crawl, a shortcut for dumping scraped items into a le (or standard output using -) Added support for passing custom settings to Scrapyd schedule.json api (r2779, r2783) New ChunkedTransferMiddleware (enabled by default) to support chunked transfer encoding (r2769) Add boto 2.0 support for S3 downloader handler (r2763) Added marshal to formats supported by feed exports (r2744) In request errbacks, offending requests are now received in failure.request attribute (r2738) Big downloader refactoring to support per domain/ip concurrency limits (r2732) CONCURRENT_REQUESTS_PER_SPIDER setting has been deprecated and replaced by: * CONCURRENT_REQUESTS, CONCURRENT_REQUESTS_PER_IP check the documentation for more details Added builtin caching DNS resolver (r2728) Moved Amazon AWS-related components/extensions (SQS spider queue, SimpleDB stats collector) to a separate project: [scaws](https://github.com/scrapinghub/scaws) (r2706, r2714) Moved spider queues to scrapyd: scrapy.spiderqueue -> scrapyd.spiderqueue (r2708) Moved sqlite utils to scrapyd: scrapy.utils.sqlite -> scrapyd.sqlite (r2781) Real support for returning iterators on start_requests() method. The iterator is now consumed during the crawl when the spider is getting idle (r2704) Added REDIRECT_ENABLED setting to quickly enable/disable the redirect middleware (r2697) Added RETRY_ENABLED setting to quickly enable/disable the retry middleware (r2694) Added CloseSpider exception to manually close spiders (r2691) Improved encoding detection by adding support for HTML5 meta charset declaration (r2690) Refactored close spider behavior to wait for all downloads to nish and be processed by spiders, before closing the spider (r2688) Added SitemapSpider (see documentation in Spiders page) (r2658) Added LogStats extension for periodically logging basic stats (like crawled pages and scraped items) (r2657) Make handling of gzipped responses more robust (#319, r2643). Now Scrapy will try and decompress as much as possible from a gzipped response, instead of failing with an IOError. Simplied !MemoryDebugger extension to use stats for dumping memory debugging info (r2639) CONCURRENT_REQUESTS_PER_DOMAIN,
184
Added new command to edit spiders: scrapy edit (r2636) and -e ag to genspider command that uses it (r2653) Changed default representation of items to pretty-printed dicts. (r2631). This improves default logging by making log more readable in the default case, for both Scraped and Dropped lines. Added spider_error signal (r2628) Added COOKIES_ENABLED setting (r2625) Stats are now dumped to Scrapy log (default value of STATS_DUMP setting has been changed to True). This is to make Scrapy users more aware of Scrapy stats and the data that is collected there. Added support for dynamically adjusting download delay and maximum concurrent requests (r2599) Added new DBM HTTP cache storage backend (r2576) Added listjobs.json API to Scrapyd (r2571) CsvItemExporter: added join_multivalued parameter (r2578) Added namespace support to xmliter_lxml (r2552) Improved cookies middleware by making COOKIES_DEBUG nicer and documenting it (r2579) Several improvements to Scrapyd and Link extractors Code rearranged and removed Merged item passed and item scraped concepts, as they have often proved confusing in the past. This means: (r2630) original item_scraped signal was removed original item_passed signal was renamed to item_scraped old log lines Scraped Item... were removed old log lines Passed Item... were renamed to Scraped Item... lines and downgraded to DEBUG level Reduced Scrapy codebase by striping part of Scrapy code into two new libraries: w3lib (several functions from scrapy.utils.{http,markup,multipart,response,url}, done in r2584) scrapely (was scrapy.contrib.ibl, done in r2586) Removed unused function: scrapy.utils.request.request_info() (r2577) Removed googledir project from examples/googledir. Theres now a new example project called dirbot available on github: https://github.com/scrapy/dirbot Removed support for default eld values in Scrapy items (r2616) Removed experimental crawlspider v2 (r2632) Removed scheduler middleware to simplify architecture. Duplicates lter is now done in the scheduler itself, using the same dupe tering class as before (DUPEFILTER_CLASS setting) (r2640) Removed support for passing urls to scrapy crawl command (use scrapy parse instead) (r2704) Removed deprecated Execution Queue (r2704) Removed (undocumented) spider context extension (from scrapy.contrib.spidercontext) (r2780) removed CONCURRENT_SPIDERS setting (use scrapyd maxproc instead) (r2789) 8.1. Release notes 185
Renamed attributes of core components: downloader.sites -> downloader.slots, scraper.sites -> scraper.slots (r2717, r2718) Renamed setting CLOSESPIDER_ITEMPASSED to CLOSESPIDER_ITEMCOUNT (r2655). Backwards compatibility kept.
8.1.22 0.12
The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available. New features and improvements Passed item is now sent in the item argument of the item_passed (#273) Added verbose option to scrapy version command, useful for bug reports (#298) HTTP cache now stored by default in the project data dir (#279) Added project data storage directory (#276, #277) Documented le structure of Scrapy projects (see command-line tool doc) New lxml backend for XPath selectors (#147) Per-spider settings (#245) Support exit codes to signal errors in Scrapy commands (#248) Added -c argument to scrapy shell command Made libxml2 optional (#260) New deploy command (#261) Added CLOSESPIDER_PAGECOUNT setting (#253) Added CLOSESPIDER_ERRORCOUNT setting (#254) Scrapyd changes Scrapyd now uses one process per spider It stores one log le per spider run, and rotate them keeping the lastest 5 logs per spider (by default) A minimal web ui was added, available at http://localhost:6800 by default There is now a scrapy server command to start a Scrapyd server of the current project Changes to settings added HTTPCACHE_ENABLED setting (False by default) to enable HTTP cache middleware changed HTTPCACHE_EXPIRATION_SECS semantics: now zero means never expire.
186
Deprecated/obsoleted functionality Deprecated runserver command in favor of server command which starts a Scrapyd server. See also: Scrapyd changes Deprecated queue command in favor of using Scrapyd schedule.json API. See also: Scrapyd changes Removed the !LxmlItemLoader (experimental contrib which never graduated to main contrib)
8.1.23 0.10
The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available. New features and improvements New Scrapy service called scrapyd for deploying Scrapy crawlers in production (#218) (documentation available) Simplied Images pipeline usage which doesnt require subclassing your own images pipeline now (#217) Scrapy shell now shows the Scrapy log by default (#206) Refactored execution queue in a common base code and pluggable backends called spider queues (#220) New persistent spider queue (based on SQLite) (#198), available by default, which allows to start Scrapy in server mode and then schedule spiders to run. Added documentation for Scrapy command-line tool and all its available sub-commands. (documentation available) Feed exporters with pluggable backends (#197) (documentation available) Deferred signals (#193) Added two new methods to item pipeline open_spider(), close_spider() with deferred support (#195) Support for overriding default request headers per spider (#181) Replaced default Spider Manager with one with similar functionality but not depending on Twisted Plugins (#186) Splitted Debian package into two packages - the library and the service (#187) Scrapy log refactoring (#188) New extension for keeping persistent spider contexts among different runs (#203) Added dont_redirect request.meta key for avoiding redirects (#233) Added dont_retry request.meta key for avoiding retries (#234) Command-line tool changes New scrapy command which replaces the old scrapy-ctl.py (#199) - there is only one global scrapy command now, instead of one scrapy-ctl.py per project - Added scrapy.bat script for running more conveniently from Windows Added bash completion to command-line tool (#210) Renamed command start to runserver (#209)
187
API changes url and body attributes of Request objects are now read-only (#230) Request.copy() and Request.replace() now also copies their callback and errback attributes (#231) Removed UrlFilterMiddleware from scrapy.contrib (already disabled by default) Offsite middelware doesnt lter out any request coming from a spider that doesnt have a allowed_domains attribute (#225) Removed Spider Manager load() method. Now spiders are loaded in the constructor itself. Changes to Scrapy Manager (now called Crawler): scrapy.core.manager.ScrapyManager class renamed to scrapy.crawler.Crawler scrapy.core.manager.scrapymanager scrapy.project.crawler singleton moved to
Moved module: scrapy.contrib.spidermanager to scrapy.spidermanager Spider Manager singleton moved from scrapy.spider.spiders to the spiders attribute of scrapy.project.crawler singleton. moved Stats Collector classes: (#204) scrapy.stats.collector.StatsCollector to scrapy.statscol.StatsCollector scrapy.stats.collector.SimpledbStatsCollector scrapy.contrib.statscol.SimpledbStatsCollector to
default per-command settings are now specied in the default_settings attribute of command object class (#201) changed arguments of Item pipeline process_item() method from (spider, item) to (item, spider) backwards compatibility kept (with deprecation warning) moved scrapy.core.signals module to scrapy.signals backwards compatibility kept (with deprecation warning) moved scrapy.core.exceptions module to scrapy.exceptions backwards compatibility kept (with deprecation warning) added handles_request() class method to BaseSpider dropped scrapy.log.exc() function (use scrapy.log.err() instead) dropped component argument of scrapy.log.msg() function dropped scrapy.log.log_level attribute Added from_settings() class methods to Spider Manager, and Item Pipeline Manager Changes to settings Added HTTPCACHE_IGNORE_SCHEMES setting to ignore certain schemes on !HttpCacheMiddleware (#225) Added SPIDER_QUEUE_CLASS setting which denes the spider queue to use (#220) Added KEEP_ALIVE setting (#220)
188
Removed SERVICE_QUEUE setting (#220) Removed COMMANDS_SETTINGS_MODULE setting (#201) Renamed REQUEST_HANDLERS to DOWNLOAD_HANDLERS and make download handlers classes (instead of functions)
8.1.24 0.9
The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available. New features and improvements Added SMTP-AUTH support to scrapy.mail New settings added: MAIL_USER, MAIL_PASS (r2065 | #149) Added new scrapy-ctl view command - To view URL in the browser, as seen by Scrapy (r2039) Added web service for controlling Scrapy process (this also deprecates the web console. (r2053 | #167) Support for running Scrapy as a service, for production systems (r1988, r2054, r2055, r2056, r2057 | #168) Added wrapper induction library (documentation only available in source code for now). (r2011) Simplied and improved response encoding support (r1961, r1969) Added LOG_ENCODING setting (r1956, documentation available) Added RANDOMIZE_DOWNLOAD_DELAY setting (enabled by default) (r1923, doc available) MailSender is no longer IO-blocking (r1955 | #146) Linkextractors and new Crawlspider now handle relative base tag urls (r1960 | #148) Several improvements to Item Loaders and processors (r2022, r2023, r2024, r2025, r2026, r2027, r2028, r2029, r2030) Added support for adding variables to telnet console (r2047 | #165) Support for requests without callbacks (r2050 | #166) API changes Change Spider.domain_name to Spider.name (SEP-012, r1975) Response.encoding is now the detected encoding (r1961) HttpErrorMiddleware now returns None or raises an exception (r2006 | #157) scrapy.command modules relocation (r2035, r2036, r2037) Added ExecutionQueue for feeding spiders to scrape (r2034) Removed ExecutionEngine singleton (r2039) Ported S3ImagesStore (images pipeline) to use boto and threads (r2033) Moved module: scrapy.management.telnet to scrapy.telnet (r2047)
189
8.1.25 0.8
The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available. New features Added DEFAULT_RESPONSE_ENCODING setting (r1809) Added dont_click argument to FormRequest.from_response() method (r1813, r1816) Added clickdata argument to FormRequest.from_response() method (r1802, r1803) Added support for HTTP proxies (HttpProxyMiddleware) (r1781, r1785) Ofste spider middleware now logs messages when ltering out requests (r1841) Backwards-incompatible changes Changed scrapy.utils.response.get_meta_refresh() signature (r1804) Removed deprecated scrapy.item.ScrapedItem class - use scrapy.item.Item instead (r1838) Removed deprecated scrapy.xpath module - use scrapy.selector instead. (r1836) Removed deprecated core.signals.domain_open signal - use core.signals.domain_opened instead (r1822) log.msg() now receives a spider argument (r1822) Old domain argument has been deprecated and will be removed in 0.9. For spiders, you should always use the spider argument and pass spider references. If you really want to pass a string, use the component argument instead. Changed core signals domain_opened, domain_closed, domain_idle Changed Item pipeline to use spiders instead of domains The domain argument of process_item() item pipeline method was changed to spider, the new signature is: process_item(spider, item) (r1827 | #105) To quickly port your code (to work with Scrapy 0.8) just use spider.domain_name where you previously used domain. Changed Stats API to use spiders instead of domains (r1849 | #113) StatsCollector was changed to receive spider references (instead of domains) in its methods (set_value, inc_value, etc). added StatsCollector.iter_spider_stats() method removed StatsCollector.list_domains() method Also, Stats signals were renamed and now pass around spider references (instead of domains). Heres a summary of the changes: To quickly port your code (to work with Scrapy 0.8) just use spider.domain_name where you previously used domain. spider_stats contains exactly the same data as domain_stats.
190
CloseDomain extension moved to scrapy.contrib.closespider.CloseSpider (r1833) Its settings were also renamed: * CLOSEDOMAIN_TIMEOUT to CLOSESPIDER_TIMEOUT * CLOSEDOMAIN_ITEMCOUNT to CLOSESPIDER_ITEMCOUNT Removed deprecated SCRAPYSETTINGS_MODULE SCRAPY_SETTINGS_MODULE instead (r1840) environment variable use
Renamed setting: REQUESTS_PER_DOMAIN to CONCURRENT_REQUESTS_PER_SPIDER (r1830, r1844) Renamed setting: CONCURRENT_DOMAINS to CONCURRENT_SPIDERS (r1830) Refactored HTTP Cache middleware HTTP Cache middleware has been heavilty refactored, retaining the same functionality except for the domain sectorization which was removed. (r1843 ) Renamed exception: DontCloseDomain to DontCloseSpider (r1859 | #120) Renamed extension: DelayedCloseDomain to SpiderCloseDelay (r1861 | #121) Removed obsolete scrapy.utils.markup.remove_escape_chars scrapy.utils.markup.replace_escape_chars instead (r1865) function use
8.1.26 0.7
First release of Scrapy.
191
write complete, reproducible, specic bug reports. The smaller the test case, the better. Remember that other developers wont have your project to reproduce the bug, so please include all relevant les required to reproduce it. include the output of scrapy version -v so developers working on your bug know exactly which version and platform it occurred on, which is often very helpful for reproducing it, or knowing if it was already xed.
192
8.2.7 Tests
Tests are implemented using the Twisted unit-testing framework called trial. Running tests To run all tests go to the root directory of Scrapy source code and run: bin/runtests.sh (on unix) bin\runtests.bat (on windows) To run a specic test (say scrapy.tests.test_contrib_loader) use: bin/runtests.sh scrapy.tests.test_contrib_loader (on unix) bin\runtests.bat scrapy.tests.test_contrib_loader (on windows) Writing tests All functionality (including new features and bug xes) must include a test case to check that it works as expected, so please include tests for your patches if you want them to get accepted sooner. Scrapy uses unit-tests, which are located in the scrapy.tests package (scrapy/tests directory). Their module name typically resembles the full path of the module theyre testing. For example, the item loaders code is in:
scrapy.contrib.loader
For example: 0.14.1 is the rst bugx release of the 0.14 series (safe to use in production)
Release notes See what has changed in recent Scrapy versions. Contributing to Scrapy Learn how to contribute to the Scrapy project. Versioning and API Stability Understand Scrapy versioning and API stability. Experimental features Learn about bleeding-edge features.
194
scrapy.contrib.logstats, 135 scrapy.contrib.memdebug, 136 scrapy.contracts, 91 scrapy.contrib.memusage, 136 scrapy.contracts.default, 90 scrapy.contrib.pipeline.images, 107 scrapy.contrib.closespider, 136 scrapy.contrib.spidermiddleware, 130 scrapy.contrib.corestats, 135 scrapy.contrib.spidermiddleware.depth, scrapy.contrib.debug, 137 131 scrapy.contrib.downloadermiddleware, 120 scrapy.contrib.spidermiddleware.httperror , scrapy.contrib.downloadermiddleware.ajaxcrawl , 131 129 scrapy.contrib.spidermiddleware.offsite , scrapy.contrib.downloadermiddleware.chunked , 132 126 scrapy.contrib.spidermiddleware.referer , scrapy.contrib.downloadermiddleware.cookies , 132 121 scrapy.contrib.spidermiddleware.urllength , scrapy.contrib.downloadermiddleware.defaultheaders , 133 122 scrapy.contrib.spiders , 33 scrapy.contrib.downloadermiddleware.downloadtimeout , scrapy.contrib.statsmailer, 137 122 scrapy.contrib.webservice , 77 scrapy.contrib.downloadermiddleware.httpauth , scrapy.contrib.webservice.crawler, 77 122 scrapy.contrib.webservice.enginestatus , scrapy.contrib.downloadermiddleware.httpcache , 77 123 scrapy.contrib.webservice.stats , 77 scrapy.contrib.downloadermiddleware.httpcompression , scrapy.crawler , 138 126 scrapy.exceptions , 164 scrapy.contrib.downloadermiddleware.httpproxy , scrapy.http , 143 126 scrapy.item , 26 scrapy.contrib.downloadermiddleware.redirect , scrapy.log , 70 126 scrapy.mail , 72 scrapy.contrib.downloadermiddleware.retry , scrapy.selector , 44 127 scrapy.settings , 139 scrapy.contrib.downloadermiddleware.robotstxt, scrapy.signalmanager , 140 128 scrapy.signals , 162 scrapy.contrib.downloadermiddleware.stats, scrapy.spider, 31 128 scrapy.statscol , 72 scrapy.contrib.downloadermiddleware.useragent , scrapy.telnet , 74 128 scrapy.utils.trackref, 102 scrapy.contrib.exporter, 166 scrapy.webservice, 135 scrapy.contrib.linkextractors, 65 scrapy.contrib.linkextractors.sgml, 65 scrapy.contrib.loader, 47 scrapy.contrib.loader.processor, 54
195