Scrapy on Renne Rocha

Dynamic rules for following links declaratively with Scrapy

blog@rocha.dev.br (Renne Rocha) — Fri, 14 Feb 2025 00:00:00 +0000

When using CrawlSpider, we have a fixed set of rules that declares how we should follow and process links extracted in the website.

But sometimes we don’t want the rules to be static. We need a certain level of dynamism, where the rules vary according to parameters provided as input in our spiders.

Consider that we are scraping product URLs from an ecommerce website, and we have the following patterns for category URLs:

https://store.example.com/ - main page of our store
https://store.example.com/electronics - list of products of electronics category
https://store.example.com/food - list of products of food category

We can notice the pattern https://store.example.com/<CATEGORY_SLUG> in our URLs. Using CrawlSpider as explained in my last post this set of rules can be defined as:

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class StoreSpider(CrawlSpider):
    name = "store"
    start_urls = ["https://store.example.com/"]

    rules = (
        Rule(
            LinkExtractor(allow=r"\/electronics$"),
            callback=self.parse_category,
        ),
        Rule(
            LinkExtractor(allow=r"\/electronics/ID\d+$"),
            callback=self.parse_product,
        ),
        Rule(
            LinkExtractor(allow=r"\/food$"),
            callback=self.parse_category,
        ),
        Rule(
            LinkExtractor(allow=r"\/food/ID\d+$"),
            callback=self.parse_product,
        ),
    )

    def parse_category(self, response):
        ...  # Code to parse a category

    def parse_product(self, response):
        ...  # Code to parse a product

There are a few potential problems with this approach:

We need to create a rule for each category we want to extract data from;
We need to change the code to add any new categories that we want to start processing;
If processing a particular category takes too long, we might want to run the spider in parallel so that each process extracts data from just one category.

What if we could send the name of the category that we want to process as an argument to our spider? We can do it using -a argument=value when calling scrapy crawl such as:

`1`	`scrapy crawl store -a category=food`

If we run the spider passing this argument, we now have access on the spider instance the attribute self.category with the value food which can be used to limit the links we want to be extracted.

Then we can filter our requests, preventing any page that is not in the desired category from being processed.

    def parse_category(self, response):
        if self.parse_category not in response.url:
            return

        ...  # Code to parse a category

The problem with this approach is that we send a real request to all links (even for the categories we don’t want to), no matter if the response will be discarded or not.

A better solution would be making the rules collection to be dynamically:

    rules = (
        Rule(
            LinkExtractor(allow=rf"\/{self.category}$"),
            callback=self.parse_category,
        ),
        Rule(
            LinkExtractor(allow=rf"\/{self.category}/ID\d+$"),
            callback=self.parse_product,
        ),
    )

Unfortunately this will not work, as the rules is a class attribute, so we don’t have an instance of the spider to get the content of our input.

Investigating Scrapy’s code, we found that the defined rules are processed in a call to the _compile_rule method.

Inside this method, each rule in self.rules is evaluated and then appended to self._rules attribute, which is where the spider decides which link to follow.

In this way, we can define our custom _compile_rules method, which will take the value passed as an argument to our spider and define the rules to only extract links that are of the desired category.

Our spider can look like this:

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class StoreSpider(CrawlSpider):
    name = "store"
    start_urls = ["https://store.example.com/"]

    def _compile_rules(self):
        self.rules = (
            Rule(
                LinkExtractor(allow=rf"\/{self.category}$"),
                callback=self.parse_category,
            ),
            Rule(
                LinkExtractor(allow=rf"\/{self.category}/ID\d+$"),
                callback=self.parse_product,
            ),
        )

        # After setting our rules, just use the existing _compile_rules() method
        super()._compile_rules()

    def parse_category(self, response):
        ...  # Code to parse a category

    def parse_product(self, response):
        ...  # Code to parse a product

We can now run a separate spider job for each category and extract product data from each category individually:

1
2

# Extracts data only of 'electronics' category
scrapy crawl store -a category=electronics

1
2

# Extracts data only of 'food' category
scrapy crawl store -a category=food

If we have new categories, we just pass them as a new value for the argument:

1
2

# Extracts data only of 'cars' category
scrapy crawl store -a category=cars

Following links declaratively with Scrapy

blog@rocha.dev.br (Renne Rocha) — Wed, 12 Feb 2025 00:00:00 +0000

This post assumes that you have a basic understanding of how Scrapy, a web scraping framework, works, discussing some lesser-known features. You can have an introduction to it in your documentation.

Extract links from a response, filtering by a specific pattern and following them is a common task that you will face when scraping a website.

As an example, suppose that you are scraping a store website and you want to process two different URLs:

Product URLs to gather product details
Category URLs to collect information about a category (e.g. number of items, subcategories, etc.) and/or more product URLs

We can implement our spider like the following:

class StoreSpider(scrapy.Spider):
    name = "store"
    start_urls = ["https://store.example.com/"]

    def parse(self, response):
        products = response.css('a.product::attr(href)')
        for link in links:
            yield scrapy.Request(link, callback=self.parse_product)

        categories = response.css('a.category::attr(href)')
        for link in links:
            yield scrapy.Request(link, callback=self.parse_category)

    def parse_product(self, response):
        ...  # Code to parse a product

    def parse_category(self, response):
        ...  # Code to parse a category

Only anchor elements with CSS classes product or category are of our interest. We have a different parse method based on the type of element being followed.

As soon you have more types of links and more complex rules to find them in the website response, your parsing methods will become more complicated and prone to have some code duplication.

For example, if inside your parse_category you also want to find more products, more categories and also subcategories, you will need to duplicate some code such as:

class StoreSpider(scrapy.Spider):
    name = "store"
    start_urls = ["https://store.example.com/"]

    # (...)

    def parse_category(self, response):
        # (...) Code to parse a category

        products = response.css('a.product::attr(href)')
        for link in links:
            yield scrapy.Request(link, callback=self.parse_product)

        categories = response.css('a.category::attr(href)')
        for link in links:
            yield scrapy.Request(link, callback=self.parse_category)

        sub_categories = response.css('a.sub_category::attr(href)')
        for link in links:
            yield scrapy.Request(link, callback=self.parse_subcategory)

To avoid all this repeated code, Scrapy comes with generic spiders providing special functionality for common scraping cases.

A CrawlSpider is a generic spider (inherited from the regular scrapy.Spider class) that provides a mechanism for following links by defining a set of rules (just like we did in our previous example).

Instead of actively looking for each link in its response, iterating over the results, and sending a request for each one, we can use a declarative pattern by providing a list of Rule objects with arguments that state which links we want to follow and what to do with the response.

Our previous spider can be rewritten as:

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class StoreSpider(CrawlSpider):
    name = "store"
    start_urls = ["https://store.example.com/"]

    rules = (
        Rule(
            LinkExtractor(restrict_css="a.product"),
            callback=self.parse_product
        ),
        Rule(
            LinkExtractor(restrict_css="a.category"),
            callback=self.parse_category
        ),
        Rule(
            LinkExtractor(
                restrict_css=".pagination",
                restrict_text="Next Page",
            ),
        ),
    )

    def parse_product(self, response):
        ...  # Code to parse a product

    def parse_category(self, response):
        ...  # Code to parse a category

You may notice that we now have a rules tuple. Each Rule is assigned a LinkExtractor object that defines how (and which) links will be extracted from each page.

In addition, we have a callback, which tells us which parse method should be used to process the return of the request from each of these links.

Given that rule as example:

Rule(
    LinkExtractor(restrict_css="a.product"),
    callback=self.parse_product
)

It extracts all the links with the CSS class product, requests them and processes the response in the spider’s method parse_product.

The following Rule extracts only links with the CSS class equal to next, but that contains the text Next Page on it. So <a class="next" href='https://store.example.com/page/2'>Next Page</a> will be extracted, but <a class="next" href='https://store.example.com/events/'>Future Events</a> will not.

Rule(
    LinkExtractor(
        restrict_css="a.next",
        restrict_text="Next Page",
    ),
),

Notice that we don’t have a callback provided for this Rule. If you don’t provide one, the link will still be requested and the response will be parsed using all the rules we have to find more links to follow.

You can see more ways to filter the links you want to extract in Link Extractor documentation.

Using CrawlSpider avoids some complexities in our parsing methods, keeping them focused only on how to scrape data into a page response, leaving the task of crawling the website (i.e., finding the following links) to a more declarative, easier-to-read and understand pattern.

Finding a job with web scraping

blog@rocha.dev.br (Renne Rocha) — Fri, 07 Feb 2025 00:00:00 +0000

A few months ago I was looking for a new job. This means spending hours browsing LinkedIn and/or job boards and drowning into outdated ads or positions that weren’t looking for someone with my skills.

After applying for some of these jobs, I realized that many companies around the world use recruitment platforms such as greenhouse.io and lever.co to receive applications. However, it is not possible through these platforms to search directly for the current openings. I was still relying on Linkedin ads or being lucky to see some of them being announced by a recruiter in some social network.

Most search engines allow us to search for keywords by limiting the results only within a specific domain. Usually adding site:<domain.com> together with our search terms will return only results in this domain. So if I search for the keywords related to the positions I want limiting my results to the recruitment platforms that I know would give me a list of job postings that probably I wouldn’t find easily and more likely to be still accepting applicants.

If I also gather information inside each job posting, I can filter them so I can apply only in the ones that are looking for someone with my skills, experience and location.

I didn’t want to do it manually, and given that I have a good experience with web scraping I decided to implement something that would help me to collect all this data.

Tools and Libraries

Scrapy

Scrapy is my default choice when performing web scraping projects. It is a simple and extensible framework that allows me to start to gather data from websites very quickly but powerful enough if I want to expand and build a more robust project.

Playwright

Although it is possible to scrape JavaScript-heavy websites without requiring a real browser to render the content, I decided to use Playwright, a tool for testing web applications that automate browser interactions but it also can be used in web scraping tasks. It also helps me to avoid beingh easily identified as a bot and being blocked to scrape the data.

scrapy-playwright

scrapy-playwright is a plugin that makes easier to integrate Playwright and make it to adhere to the regular Scrapy workflow.

Development

Preparing our environment

Project is a common Python project developed inside a virtualenv.

mkdir job_search
cd job_search
python -m venv .venv
source .venv/bin/activate
pip install scrapy scrapy-playwright

Then create a Scrapy project and configure scrapy-playwright following the installation and activation instructions available in the extension documentation.

1
2

scrapy startproject postings
cd postings

# postings/settings.py
# (...)

# Add the following to the existing file
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

Gathering URL of job postings

In Scrapy terminology, a Spider is a class which we define how to scrape and crawl information from a certain site. So let’s create one Spider to perform a search in DuckDuckGo passing as search parameters: (1) the domain we want the results and (2) keywords we want to be inside the results (so we can filter only the job postings of the technology/position we are interested).

# postings/spiders/duckduckgo.py
import itertools
from urllib.parse import urlparse
import scrapy
import w3lib.url

class DuckDuckGoSpider(scrapy.Spider):
    name = "duckduckgo"
    allowed_domains = ["duckduckgo.com"]

    def start_requests(self):
        ...

    def parse(self, response, keyword):
        ...

Inside start_request method, we schedule the initial requests sending search queries to DuckDuckGo passing domains and keywords we are interested.

Adding meta={"playwright": True} to the request ensures that a real browser (managed by playwright and scrapy-playwright) will be used. This will help us not to be easily identified as a bot (and blocked). Using a real browser (instead of sending plain requests only) will make our spider to be slower. Given that this spider will not be running frequently, we can accept the slowness of opening a real browser and perform the operations on it.

We also add cb_kwargs as a way to send some metadata to the request that will be used later to add extra information in the data returned.

    def start_requests(self):
        keywords = ["python", "django", "flask", "fastapi"]
        domains = ["jobs.lever.co", "boards.greenhouse.io"]

        for domain, keyword in itertools.product(domains, keywords):
            yield scrapy.Request(
                f"https://duckduckgo.com/?q=site%3A{domain}+{keyword}",
                meta={"playwright": True},
                cb_kwargs={"keyword": keyword},
            )

Next step is to implement parse method that will get the response from DuckDuckGo and gather the URLs of the job postings.

    def parse(self, response, keyword):
        for url in response.css("a::attr(href)").getall():
            parsed_url = urlparse(url)
            if parsed_url.netloc not in ["jobs.lever.co", "boards.greenhouse.io"]:
                # Ignore results not in the domains we are interested
                continue

            company_name = parsed_url.path.split("/")[1]

            yield {
                "company": company_name,
                "job_posting_url": url,
                "keyword": keyword,
            }

Running the Spider and exporting the results into a CSV file.

`1`	`scrapy crawl duckduckgo -o job_postings.csv`

And now we have an initial version of our spider and we are able to collect links of job postings that matches our search keywords.

Handling page pagination

This initial run of the spider probably will return just a few results (~10). Performing this search manually, we will notice that what we got until now is just the first page of the results. Looking at DuckDuckGo results page, we can notice in the bottom the following button that when clicked will load more results.

scrapy-playwright (and playwright) allows us to perform actions on pages such filling forms or clicking elements, so we can change our spider that when we perform the search, we will click in More Results button until we can’t find it in the page anymore (indicating that we don’t have more hidden results to be shown).

First we create our function that will get the page, and click the button as many times as needed. When the button is not visible anymore (we reach our maximum number of results), PlaywrightTimeoutError is raised stopping the interactions with the page and releasing it to be parsed.

# postings/spiders/duckduckgo.py
from playwright.async_api import TimeoutError as PlaywrightTimeoutError


async def more_results(page):
    while True:
        try:
            await page.locator(selector="#more-results").click()
        except PlaywrightTimeoutError:
            break
    return page.url

Then we add playwright_page_methods with the list of methods that we want to be called on the page.

# postings/spiders/duckduckgo.py

class DuckDuckGoSpider(scrapy.Spider):
    # (...)

    def start_requests(self):
        keywords = ["python", "django", "flask", "fastapi"]
        domains = ["jobs.lever.co", "boards.greenhouse.io"]

        for domain, keyword in itertools.product(domains, keywords):
            yield scrapy.Request(
                f"https://duckduckgo.com/?q=site%3A{domain}+{keyword}",
                meta={
                    "playwright": True,
                    "playwright_page_methods": [
                        PageMethod(more_results),
                    ],
                },
                cb_kwargs={"keyword": keyword},
            )

Running the Spider again, we will get all the results.

Removing duplicated results

After running the Spider, we will notice that some URLs are duplicated. One of the reasons is that more than one search keyword can return the same job posting result (we would expect a Django job posting also having Python keyword inside it).

To drop the duplicated values, we can create an Item Pipeline that will check, for each job posting returned whether it was returned before or not.

An item pipeline is a simple class that implements process_item method that receives items returned by the Spider, perform some processing on the item and then return it or drop it. When enabled all items returned by DuckDuckGoSpider will pass through it.

A seen_urls set is defined, so we can check if that particular URL was already returned (and then we can skip it) or if it is a new URL.

# postings/pipelines.py
from scrapy.exceptions import DropItem


class JobPostingDuplicatesPipeline:
    seen_urls = set()

    def process_item(self, item, spider):
        if item["job_posting_url"] in self.seen_urls:
            # We drop items which the URL was already returned before
            raise DropItem("Already returned")

        # Add the URL to the set when it was processed for the first time
        self.seen_urls.add(item["job_posting_url"])

        yield item

We need to enable this pipeline in our project.

# postings/settings.py
# (...)

ITEM_PIPELINES = {
   "postings.pipelines.JobPostingDuplicatesPipeline": 300,
}

We can run again the Spider, exporting the results into a CSV file. This time, we will have only unique job posting URLs.

`1`	`scrapy crawl duckduckgo -o job_postings.csv`

Possible improvements

This is the starting point of our job search. It was useful to me because it returned some companies that I have never heard about it, looking for professionals with the skills that I have. Certainly this narrowed more my options and helped me to apply to positions that made more sense to me.

A possible improvement would be create a specific spider for each recruitment platform to parse the content of the job postings. This would allow us to filter even more our data. For example, we can check for specific benefits or other keywords that would be useful for us to decide to apply or not to the job.

Summary

Here we have the complete code:

# postings/pipelines.py
import itertools
from urllib.parse import urlparse

import scrapy
import w3lib.url
from playwright.async_api import TimeoutError as PlaywrightTimeoutError
from scrapy_playwright.page import PageMethod


async def more_results(page):
    while True:
        try:
            await page.locator(selector="#more-results").click()
        except PlaywrightTimeoutError:
            break
    return page.url


class DuckDuckGoSpider(scrapy.Spider):

    def start_requests(self):
        keywords = ["python", "django", "flask", "fastapi"]
        domains = ["jobs.lever.co", "boards.greenhouse.io"]

        for domain, keyword in itertools.product(domains, keywords):
            yield scrapy.Request(
                f"https://duckduckgo.com/?q=site%3A{domain}+{keyword}",
                meta={
                    "playwright": True,
                    "playwright_page_methods": [
                        PageMethod(more_results),
                    ],
                },
                cb_kwargs={"keyword": keyword},
            )

    def parse(self, response, keyword):
        for url in response.css("a::attr(href)").getall():
            parsed_url = urlparse(url)
            if parsed_url.netloc not in ["jobs.lever.co", "boards.greenhouse.io"]:
                # Ignore results not in the domains we are interested
                continue

            company_name = parsed_url.path.split("/")[1]

            yield {
                "company": company_name,
                "job_posting_url": url,
                "keyword": keyword,
            }

# postings/settings.py
# (...)

# Add the following to the existing file
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

ITEM_PIPELINES = {
   "postings.pipelines.JobPostingDuplicatesPipeline": 300,
}

# postings/pipelines.py
from scrapy.exceptions import DropItem


class JobPostingDuplicatesPipeline:
    seen_urls = set()

    def process_item(self, item, spider):
        if item["job_posting_url"] in self.seen_urls:
            # We drop items which the URL was already returned before
            raise DropItem("Already returned")

        # Add the URL to the set when it was processed for the first time
        self.seen_urls.add(item["job_posting_url"])

        yield item

Good luck with your job hunt!

Revisão de Código: Extraindo dados do site 'Aos Fatos'

blog@rocha.dev.br (Renne Rocha) — Fri, 08 Apr 2022 00:00:00 +0000

Algumas semanas atrás, fiz a revisão de um código para extrair informações de campeonatos da CBF.

A experiência de fazer isso em uma live foi muito boa, pois consegui ajudar alguém compartilhando um pouco da minha experiência, mas também foi uma maneira de eu aprender mais, já que para comentar sobre algum assunto eu precisei ler e estudar (e relembrar) algumas coisas que já fazia algum tempo que eu não olhava.

Depois de algum tempo, encontrei outra pessoa que me autorizou a fazer essa revisão em uma live. Dessa vez, fiz a revisão do código que extraia informações de notícias do Aos Fatos.

O código é um projeto feito com o Scrapy, um framework Python para o desenvolvimento de raspadores de dados.

Se você quiser que eu faça uma revisão do seu código em um vídeo, é só entrar em contato comigo. Se eu achar que consigo ajudar de alguma maneira, combinamos uma nova transmissão.

Links de Referência

black - formatador de código automático
CrawlSpider - tipo de Spider que ajuda a escrever códigos mais organizados
dateparser - biblioteca para conversão de texto em data
LinkExtractor - class que auxilia a extração de links dentro de um HTML
Laboratório Hacker de Campinas - hackerspace localizado em Campinas
owncast - plataforma self-hosted por onde fiz a transmissão
PeerTube - plataforma de vídeos livre, decentralizada e federada
Meu PeerTube - instância do PeerTube onde armazeno todos os meus vídeos

Monitorando seus projetos de raspagem de dados

blog@rocha.dev.br (Renne Rocha) — Thu, 13 Jan 2022 00:00:00 +0000

Quando passamos a extrair dados de uma página com regularidade, com uma quantidade grande de dados, é muito importante monitorar a saúde dos seus raspadores e a qualidade das informações extraídas.

As páginas podem mudar (as vezes sutilmente), medidas anti-bot podem ser adicionadas e outros problemas temporários e inesperados podem ocorrer e, caso estejamos tratando de uma quantidade grande de scripts retornando milhares de itens, é praticamente impossível monitorar manualmente todo o nosso projeto.

Deste modo, devemos adicionar em nosso projeto, ferramentas que auxiliem no monitoramento.

Spidermon

O Scrapy é um dos principais frameworks para raspagem de dados em Python possuindo diversas bibliotecas e ferramentas que facilitam a criação de projetos de raspagem de dados.

O Spidermon é uma extensão do Scrapy que podemos adicionar ao nosso projeto para facilitar o monitoramento da execução dos seus spiders (como são chamados os scripts que realizam a extração de dados).

Com ela, é possível definir validar dados retornados, monitorar as estatísticas da execução (jobs) de seu spider e, a partir do resultadodessas validações, tomar ações como enviar notificações, gerar relatórios, abrir chamados em sistemas externos quando algo não esteja funcionando corretamente de uma maneira extensível.

O Spidermon pode ser instalado com o pip:

`1`	`pip install spidermon`

Uma vez instalado, precisamos habilitá-lo em nosso projeto Scrapy:

# myproject/settings.py
SPIDERMON_ENABLED = True
EXTENSIONS = {
    "spidermon.contrib.scrapy.extensions.Spidermon": 500,
}

Feito isso, podemos começar a pensar no que será monitorado.

Monitores

É no monitor onde a maioria das validações são definidas. Nele temos acesso as estatísticas e informações das execuções de nossos spiders e podemos definir as nossas regras de monitoramento.

Esse monitor precisa estar dentro de uma suíte de monitores, além de definir um conjunto de ações que devem ser executadas após a execução deles.

Como exemplo, queremos monitorar o número de mensagens de erro emitidas durante os nossos jobs. Isso pode ser feito verificando se o valor dentro da estatística log_count/ERROR é menor a um determinado valor que podemos, por exemplo, definir no arquivo de configuração do nosso projeto.

Um monitor que realiza essa tarefa pode ser definido como:

# myproject/monitors.py
from spidermon import Monitor

class JobErrorsMonitor(Monitor):
    def test_error_count(self):
        num_errors = self.data.stats.get("log_count/ERROR")
        num_errors_threshold = self.data.crawler.settings.getint(
            "MAX_NUM_ERRORS", 0
        )
        if num_errors:
            self.assertTrue(
                num_errors <= num_errors_threshold,
                msg=f"No more than {num_errors_threshold} allowed."
            )

Porém, só definir o monitor não é o suficiente para que esse teste seja rodado ao final da execução do seu spider. Precisamos adicioná-lo a uma suíte de monitores:

# myproject/monitors.py
from spidermon import Monitor, MonitorSuite

class JobErrorsMonitor(Monitor):
    # (...) O código do nosso monitor

class SpiderCloseMonitorSuite(MonitorSuite):
    monitors = [
        JobErrorsMonitor,
    ]

E definir o momento em que momento a suíte será executada, além do nosso valor de referência com o limite da quantidade de erros que aceitamos. Nesse caso, quando o spider terminar sua execução as verificações será feita.

# myproject/settings.py
SPIDERMON_ENABLED = True
EXTENSIONS = {
    "spidermon.contrib.scrapy.extensions.Spidermon": 500,
}
SPIDERMON_SPIDER_CLOSE_MONITORS = (
    "myproject.monitors.SpiderCloseMonitorSuite",
)
MAX_NUM_ERRORS = 10

Ao executar spider podemos ver os resultados desse monitor nos nossos logs:

[scrapy.core.engine] INFO: Closing spider (finished)
[myproject] INFO: [Spidermon] ---------------------- MONITORS ----------------------
[myproject] INFO: [Spidermon] JobErrorsMonitor/test_error_count... OK
[myproject] INFO: [Spidermon] ------------------------------------------------------
[myproject] INFO: [Spidermon] 1 monitor in 0.000s
[myproject] INFO: [Spidermon] OK
[myproject] INFO: [Spidermon] ------------------ FINISHED ACTIONS ------------------
[myproject] INFO: [Spidermon] ------------------------------------------------------
[myproject] INFO: [Spidermon] 0 actions in 0.000s
[myproject] INFO: [Spidermon] OK
[myproject] INFO: [Spidermon] ------------------- PASSED ACTIONS -------------------
[myproject] INFO: [Spidermon] ------------------------------------------------------
[myproject] INFO: [Spidermon] 0 actions in 0.000s
[myproject] INFO: [Spidermon] OK
[myproject] INFO: [Spidermon] ------------------- FAILED ACTIONS -------------------
[myproject] INFO: [Spidermon] ------------------------------------------------------
[myproject] INFO: [Spidermon] 1 action in 0.000s
[myproject] INFO: [Spidermon] OK (skipped=1)

Ou caso aconteça um erro:

[scrapy.core.engine] INFO: Closing spider (finished)
[myproject] INFO: [Spidermon] ---------------------- MONITORS ----------------------
[myproject] INFO: [Spidermon] JobErrorsMonitor/test_error_count... FAIL
[myproject] INFO: [Spidermon] ------------------------------------------------------
[myproject] ERROR: [Spidermon]
======================================================================
FAIL: JobErrorsMonitor/test_error_count
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/renne/projects/rennerocha/spidermon/examples/tutorial/tutorial/monitors.py", line 44, in test_error_count
    self.assertTrue(
AssertionError: False is not true : No more than 10 errors allowed.

[myproject] INFO: [Spidermon] 1 monitor in 0.001s
[myproject] INFO: [Spidermon] FAILED (failures=1)
[myproject] INFO: [Spidermon] ------------------ FINISHED ACTIONS ------------------
[myproject] INFO: [Spidermon] ------------------------------------------------------
[myproject] INFO: [Spidermon] 0 actions in 0.000s
[myproject] INFO: [Spidermon] OK
[myproject] INFO: [Spidermon] ------------------- PASSED ACTIONS -------------------
[myproject] INFO: [Spidermon] ------------------------------------------------------
[myproject] INFO: [Spidermon] 0 actions in 0.000s
[myproject] INFO: [Spidermon] OK
[myproject] INFO: [Spidermon] ------------------- FAILED ACTIONS -------------------

Simplificando o nosso monitor

Grande parte dos monitores que precisamos, verifica o valor de uma estatística do seu job com um valor pré-determinado. Por isso, podemos usar a classe BaseStatMonitor com um conjunto definido de atributos para simplificar o nosso código.

O mesmo monitor anterior, verificando se o valor de log_count/ERROR é menor ou igual ao valor definido em MAX_NUM_ERRORS nas configurações do projeto pode ser definido como:

# myproject/monitors.py
from spidermon.contrib.scrapy.monitors import BaseStatMonitor

class JobErrorsMonitor(BaseStatMonitor):
    stat_name = "log_count/ERROR"
    threshold_setting = "MAX_NUM_ERRORS"
    assert_type = "<="

O resultado da execução desse monitor é o mesmo do anterior, porém com um código muito mais simples.

Tornando o monitor mais dinâmico

Não precisamos nos restringir a testes de um valor nas estatísticas em relação a um valor estático definido nas configurações do projeto.

Para isso, ao invés de definir um valor para o atributo threshold_setting, podemos definir o método get_threshold que deverá retornar o valor que será usado como referência para o nosso monitor.

Isso é muito útil por exemplo, quando queremos verificar mais de uma estatística para definir o nosso valor de referência. No exemplo anterior, definimos uma valor fixo aceitável de mensagens de erro durante a execução do nosso spider. Porém termos 10 erros em uma execução que retorna 100k itens (0.001%) pode ser mais aceitável por exemplo do que 10 erros em uma execução que retornou 1k itens (1%).

Então podemos melhoramos o nosso monitor de erros, considerando além da quantidade de erros, a quantidade de itens retornados. Desse modo só haverá falha se o número de erros for menor do que 0.001% da quantidade de itens retornados.

# myproject/monitors.py
from spidermon.contrib.scrapy.monitors import BaseStatMonitor

class JobErrorsMonitor(BaseStatMonitor):
    stat_name = "log_count/ERROR"
    assert_type = "<="

    def get_threshold(self):
        item_scraped_count = self.data.stats.get(
            "item_scraped_count"
        )
        return item_scraped_count * 0.0001

Podemos também buscar dados de fontes externas, comparar com dados de execuções passadas que estejam armazenados em um banco de dados ou fazer cálculos mais complexos para definir o nosso valor de referência.

O importante é que sempre o retorno do método get_threshold seja um valor que possa ser usado na comparação com a estatística que queremos.

Na documentação é possível encontrar mais exemplo e opções para os nossos monitores, além de um conjunto de monitores pré-definidos para as validações mais comuns nos projetos diminuindo a quantidade de código que precisamos escrever para começar a monitorar nossos projetos.

Próximos passos

Ver o resultado dos nossos monitores apenas nos logs não é muito prático. O que precisamos fazer em seguida é configurar nosso projeto para que alguma ação seja tomada em caso de falha (ou de sucesso).

Para isso, precisamos definir actions. Na extensão temos integrações prontas para receber notificações por e-mail, Slack, Telegram ou Sentry, além de ser possível criar suas próprias ações de acordo com as necessidades do projeto.

Na documentação é possível encontrar mais informações sobre como defini-las e configurá-las.

Validando dados extraídos com Scrapy

blog@rocha.dev.br (Renne Rocha) — Fri, 08 Mar 2019 00:00:00 +0000

Como você garante a qualidade e a confiabilidade dos dados que você está extraindo em seu crawler?

Se o seu crawler foi desenvolvido com Scrapy, você tem a disposição a extensão Spidermon, que junto com a biblioteca Schematics permite que você inclua validações de dados de maneira simples e altamente customizável.

Definindo modelos de dados, é possível comparar os items retornados com uma estrutura pré-determinada, garantido que todos os campos contém dados no formato esperado.

Nosso exemplo

Para explicar o uso do Spidermon para realizar a validação dos dados, criamos um projeto de exemplo simples, que extrai citações do site http://quotes.toscrape.com/.

Começamos instalando as bibliotecas necessárias:

1
2
3

$ python3 -m venv .venv
$ source .venv/bin/activate
(.venv) $ pip install scrapy spidermon schematics

Em seguida criamos um novo projeto Scrapy:

1
2

$ scrapy startproject quotes_crawler
$ cd myproject & scrapy genspider quotes quotes.com

Definimos um item que será retornado pelo nosso spider:

# quotes_crawler/spiders/items.py
import scrapy

class QuoteItem(scrapy.Item):
    quote = scrapy.Field()
    author = scrapy.Field()
    author_url = scrapy.Field()
    tags = scrapy.Field()

E finalmente criamos o nosso spider:

# quotes_crawler/spiders/quotes.py
import scrapy
from myproject.items import QuoteItem

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    allowed_domains = ["quotes.toscrape.com"]
    start_urls = ["http://quotes.toscrape.com/"]

    def parse(self, response):
        for quote in response.css(".quote"):
            item = QuoteItem(
                quote=quote.css(".text::text").get(),
                author=quote.css(".author::text").get(),
                author_url=response.urljoin(
                    quote.css(".author a::attr(href)").get()),
                tags=quote.css(".tag *::text").getall(),
            )
            yield item
        yield scrapy.Request(
            response.urljoin(response.css(
                ".next a::attr(href)").get())
        )

Com isso já é possível executar o nosso crawler e obter os dados desejados:

`1`	`$ scrapy crawl quotes -o quotes.csv`

Definindo nosso modelo de validação

Schematics é uma biblioteca de validação de dados baseada em modelos (muito parecidos com modelos do Django). Esses modelos incluem alguns tipos de dados comuns e validadores, mas também é possível extendê-los e definir regras de validação customizadas.

Baseado no nosso exemplo anterior, vamos criar um modelo para o nosso item contendo algumas regras básicas de validação:

# quotes_crawler/spiders/validators.py
from schematics.models import Model
from schematics.types import URLType, StringType, ListType

class QuoteItem(Model):
    # String de preenchimento obrigatório
    quote = StringType(required=True)

    # String de preenchimento obrigatório
    author = StringType(required=True)

    # Uma string que represente uma URL de preenchimento obrigatório
    author_url = URLType(required=True)

    # Lista de Strings de preenchimento não obrigatório
    tags = ListType(StringType)

Precisamos agora habilitar o Spidermon e configurá-lo para que os campos retornados pelo nosso spider sejam validados contra esse modelo:

# quotes_crawler/settings.py
SPIDERMON_ENABLED = True

EXTENSIONS = {
    "spidermon.contrib.scrapy.extensions.Spidermon": 500,
}

ITEM_PIPELINES = {
    "spidermon.contrib.scrapy.pipelines.ItemValidationPipeline": 800
}

SPIDERMON_VALIDATION_MODELS = (
    "quotes_crawler.validators.QuoteItem",
)

# Inclui um campo '_validation' nos items que contenham erros
# com detalhes do problema encontrado
SPIDERMON_VALIDATION_ADD_ERRORS_TO_ITEMS = True

Conforme os dados forem extraidos, eles serão validados de acordo com as regras definidas no modelo e, caso algum erro seja encontrado, um novo campo _validation será incluído no item com detalhes do erro.

Caso um item receba uma URL inválida no campo author_url por exemplo, o resultado será:

{
    '_validation': defaultdict(
        <class 'list'>, {'author_url': ['Invalid URL']}),
    'author': 'C.S. Lewis',
    'author_url': 'invalid_url',
    'quote': 'Some day you will be old enough to start reading fairy tales '
        'again.',
    'tags': ['age', 'fairytales', 'growing-up']
}