Scrapy is a high-level Python web crawling and web scraping framework, designed for fast extraction of structured data from websites. It's actively maintained with frequent releases, supporting applications from data mining to information processing and automated testing. The current version is 2.15.0.
pip install scrapyVerified import paths — ran on the pinned version, not inferred.
This quickstart demonstrates a basic Scrapy spider that crawls the 'quotes.toscrape.com' website, specifically the 'humor' tag. It extracts the author and text of each quote, then follows the 'Next Page' link to continue crawling. The `start_urls` attribute defines the initial URLs, and the `parse` method handles the response, extracting data and scheduling new requests using `response.follow` for pagination.
Upgrade to Python 3.10 or newer. Use a virtual environment for isolated Scrapy installations.
Migrate `def start_requests(self)` to `async def start(self)` if you are performing asynchronous operations (e.g., database calls) to generate your initial requests.
Review and update custom middlewares to support asynchronous spider output by defining `process_spider_output` as an asynchronous generator or implementing `process_spider_output_async`.
Do not rely on `Referrer-Policy` header values being executed as code. Be aware that POST requests resulting in 301 redirects will now be re-sent as GET requests.
Replace calls to `scrapy.utils.defer` functions with their `twisted.internet.defer` equivalents or appropriate coroutine patterns. For `maybeDeferred_coro()`, consider `twisted.internet.defer.maybeDeferred` if staying with Deferreds.
Store per-request/response data in the `request.meta` or `request.cb_kwargs` mappings instead of attaching new attributes to the objects.
Adjust `DOWNLOAD_DELAY` and `CONCURRENT_REQUESTS_PER_DOMAIN` in your `settings.py` if you require higher concurrency or a faster crawl rate for your specific use case.
Ensure Scrapy is installed with `pip install scrapy` and that your shell's PATH includes the directory where Scrapy executables are located, or activate your Python virtual environment.
Install Scrapy using pip: `pip install scrapy`.
Replace `from scrapy.spider import BaseSpider` with `from scrapy import Spider`.
Define a `parse` method in your spider class with `def parse(self, response):` and include your scraping logic there.
When embedding Scrapy, use `CrawlerProcess` or `CrawlerRunner` and ensure the reactor is managed properly; avoid calling `reactor.run()` explicitly after `process.start()` or for each spider run.