Web Scraping on Renne Rocha

Dynamic rules for following links declaratively with Scrapy

blog@rocha.dev.br (Renne Rocha) — Fri, 14 Feb 2025 00:00:00 +0000

When using CrawlSpider, we have a fixed set of rules that declares how we should follow and process links extracted in the website.

But sometimes we don’t want the rules to be static. We need a certain level of dynamism, where the rules vary according to parameters provided as input in our spiders.

Consider that we are scraping product URLs from an ecommerce website, and we have the following patterns for category URLs:

https://store.example.com/ - main page of our store
https://store.example.com/electronics - list of products of electronics category
https://store.example.com/food - list of products of food category

We can notice the pattern https://store.example.com/<CATEGORY_SLUG> in our URLs. Using CrawlSpider as explained in my last post this set of rules can be defined as:

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class StoreSpider(CrawlSpider):
    name = "store"
    start_urls = ["https://store.example.com/"]

    rules = (
        Rule(
            LinkExtractor(allow=r"\/electronics$"),
            callback=self.parse_category,
        ),
        Rule(
            LinkExtractor(allow=r"\/electronics/ID\d+$"),
            callback=self.parse_product,
        ),
        Rule(
            LinkExtractor(allow=r"\/food$"),
            callback=self.parse_category,
        ),
        Rule(
            LinkExtractor(allow=r"\/food/ID\d+$"),
            callback=self.parse_product,
        ),
    )

    def parse_category(self, response):
        ...  # Code to parse a category

    def parse_product(self, response):
        ...  # Code to parse a product

There are a few potential problems with this approach:

We need to create a rule for each category we want to extract data from;
We need to change the code to add any new categories that we want to start processing;
If processing a particular category takes too long, we might want to run the spider in parallel so that each process extracts data from just one category.

What if we could send the name of the category that we want to process as an argument to our spider? We can do it using -a argument=value when calling scrapy crawl such as:

`1`	`scrapy crawl store -a category=food`

If we run the spider passing this argument, we now have access on the spider instance the attribute self.category with the value food which can be used to limit the links we want to be extracted.

Then we can filter our requests, preventing any page that is not in the desired category from being processed.

    def parse_category(self, response):
        if self.parse_category not in response.url:
            return

        ...  # Code to parse a category

The problem with this approach is that we send a real request to all links (even for the categories we don’t want to), no matter if the response will be discarded or not.

A better solution would be making the rules collection to be dynamically:

    rules = (
        Rule(
            LinkExtractor(allow=rf"\/{self.category}$"),
            callback=self.parse_category,
        ),
        Rule(
            LinkExtractor(allow=rf"\/{self.category}/ID\d+$"),
            callback=self.parse_product,
        ),
    )

Unfortunately this will not work, as the rules is a class attribute, so we don’t have an instance of the spider to get the content of our input.

Investigating Scrapy’s code, we found that the defined rules are processed in a call to the _compile_rule method.

Inside this method, each rule in self.rules is evaluated and then appended to self._rules attribute, which is where the spider decides which link to follow.

In this way, we can define our custom _compile_rules method, which will take the value passed as an argument to our spider and define the rules to only extract links that are of the desired category.

Our spider can look like this:

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class StoreSpider(CrawlSpider):
    name = "store"
    start_urls = ["https://store.example.com/"]

    def _compile_rules(self):
        self.rules = (
            Rule(
                LinkExtractor(allow=rf"\/{self.category}$"),
                callback=self.parse_category,
            ),
            Rule(
                LinkExtractor(allow=rf"\/{self.category}/ID\d+$"),
                callback=self.parse_product,
            ),
        )

        # After setting our rules, just use the existing _compile_rules() method
        super()._compile_rules()

    def parse_category(self, response):
        ...  # Code to parse a category

    def parse_product(self, response):
        ...  # Code to parse a product

We can now run a separate spider job for each category and extract product data from each category individually:

1
2

# Extracts data only of 'electronics' category
scrapy crawl store -a category=electronics

1
2

# Extracts data only of 'food' category
scrapy crawl store -a category=food

If we have new categories, we just pass them as a new value for the argument:

1
2

# Extracts data only of 'cars' category
scrapy crawl store -a category=cars

Following links declaratively with Scrapy

blog@rocha.dev.br (Renne Rocha) — Wed, 12 Feb 2025 00:00:00 +0000

This post assumes that you have a basic understanding of how Scrapy, a web scraping framework, works, discussing some lesser-known features. You can have an introduction to it in your documentation.

Extract links from a response, filtering by a specific pattern and following them is a common task that you will face when scraping a website.

As an example, suppose that you are scraping a store website and you want to process two different URLs:

Product URLs to gather product details
Category URLs to collect information about a category (e.g. number of items, subcategories, etc.) and/or more product URLs

We can implement our spider like the following:

class StoreSpider(scrapy.Spider):
    name = "store"
    start_urls = ["https://store.example.com/"]

    def parse(self, response):
        products = response.css('a.product::attr(href)')
        for link in links:
            yield scrapy.Request(link, callback=self.parse_product)

        categories = response.css('a.category::attr(href)')
        for link in links:
            yield scrapy.Request(link, callback=self.parse_category)

    def parse_product(self, response):
        ...  # Code to parse a product

    def parse_category(self, response):
        ...  # Code to parse a category

Only anchor elements with CSS classes product or category are of our interest. We have a different parse method based on the type of element being followed.

As soon you have more types of links and more complex rules to find them in the website response, your parsing methods will become more complicated and prone to have some code duplication.

For example, if inside your parse_category you also want to find more products, more categories and also subcategories, you will need to duplicate some code such as:

class StoreSpider(scrapy.Spider):
    name = "store"
    start_urls = ["https://store.example.com/"]

    # (...)

    def parse_category(self, response):
        # (...) Code to parse a category

        products = response.css('a.product::attr(href)')
        for link in links:
            yield scrapy.Request(link, callback=self.parse_product)

        categories = response.css('a.category::attr(href)')
        for link in links:
            yield scrapy.Request(link, callback=self.parse_category)

        sub_categories = response.css('a.sub_category::attr(href)')
        for link in links:
            yield scrapy.Request(link, callback=self.parse_subcategory)

To avoid all this repeated code, Scrapy comes with generic spiders providing special functionality for common scraping cases.

A CrawlSpider is a generic spider (inherited from the regular scrapy.Spider class) that provides a mechanism for following links by defining a set of rules (just like we did in our previous example).

Instead of actively looking for each link in its response, iterating over the results, and sending a request for each one, we can use a declarative pattern by providing a list of Rule objects with arguments that state which links we want to follow and what to do with the response.

Our previous spider can be rewritten as:

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class StoreSpider(CrawlSpider):
    name = "store"
    start_urls = ["https://store.example.com/"]

    rules = (
        Rule(
            LinkExtractor(restrict_css="a.product"),
            callback=self.parse_product
        ),
        Rule(
            LinkExtractor(restrict_css="a.category"),
            callback=self.parse_category
        ),
        Rule(
            LinkExtractor(
                restrict_css=".pagination",
                restrict_text="Next Page",
            ),
        ),
    )

    def parse_product(self, response):
        ...  # Code to parse a product

    def parse_category(self, response):
        ...  # Code to parse a category

You may notice that we now have a rules tuple. Each Rule is assigned a LinkExtractor object that defines how (and which) links will be extracted from each page.

In addition, we have a callback, which tells us which parse method should be used to process the return of the request from each of these links.

Given that rule as example:

Rule(
    LinkExtractor(restrict_css="a.product"),
    callback=self.parse_product
)

It extracts all the links with the CSS class product, requests them and processes the response in the spider’s method parse_product.

The following Rule extracts only links with the CSS class equal to next, but that contains the text Next Page on it. So <a class="next" href='https://store.example.com/page/2'>Next Page</a> will be extracted, but <a class="next" href='https://store.example.com/events/'>Future Events</a> will not.

Rule(
    LinkExtractor(
        restrict_css="a.next",
        restrict_text="Next Page",
    ),
),

Notice that we don’t have a callback provided for this Rule. If you don’t provide one, the link will still be requested and the response will be parsed using all the rules we have to find more links to follow.

You can see more ways to filter the links you want to extract in Link Extractor documentation.

Using CrawlSpider avoids some complexities in our parsing methods, keeping them focused only on how to scrape data into a page response, leaving the task of crawling the website (i.e., finding the following links) to a more declarative, easier-to-read and understand pattern.

Finding a job with web scraping

blog@rocha.dev.br (Renne Rocha) — Fri, 07 Feb 2025 00:00:00 +0000

A few months ago I was looking for a new job. This means spending hours browsing LinkedIn and/or job boards and drowning into outdated ads or positions that weren’t looking for someone with my skills.

After applying for some of these jobs, I realized that many companies around the world use recruitment platforms such as greenhouse.io and lever.co to receive applications. However, it is not possible through these platforms to search directly for the current openings. I was still relying on Linkedin ads or being lucky to see some of them being announced by a recruiter in some social network.

Most search engines allow us to search for keywords by limiting the results only within a specific domain. Usually adding site:<domain.com> together with our search terms will return only results in this domain. So if I search for the keywords related to the positions I want limiting my results to the recruitment platforms that I know would give me a list of job postings that probably I wouldn’t find easily and more likely to be still accepting applicants.

If I also gather information inside each job posting, I can filter them so I can apply only in the ones that are looking for someone with my skills, experience and location.

I didn’t want to do it manually, and given that I have a good experience with web scraping I decided to implement something that would help me to collect all this data.

Tools and Libraries

Scrapy

Scrapy is my default choice when performing web scraping projects. It is a simple and extensible framework that allows me to start to gather data from websites very quickly but powerful enough if I want to expand and build a more robust project.

Playwright

Although it is possible to scrape JavaScript-heavy websites without requiring a real browser to render the content, I decided to use Playwright, a tool for testing web applications that automate browser interactions but it also can be used in web scraping tasks. It also helps me to avoid beingh easily identified as a bot and being blocked to scrape the data.

scrapy-playwright

scrapy-playwright is a plugin that makes easier to integrate Playwright and make it to adhere to the regular Scrapy workflow.

Development

Preparing our environment

Project is a common Python project developed inside a virtualenv.

mkdir job_search
cd job_search
python -m venv .venv
source .venv/bin/activate
pip install scrapy scrapy-playwright

Then create a Scrapy project and configure scrapy-playwright following the installation and activation instructions available in the extension documentation.

1
2

scrapy startproject postings
cd postings

# postings/settings.py
# (...)

# Add the following to the existing file
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

Gathering URL of job postings

In Scrapy terminology, a Spider is a class which we define how to scrape and crawl information from a certain site. So let’s create one Spider to perform a search in DuckDuckGo passing as search parameters: (1) the domain we want the results and (2) keywords we want to be inside the results (so we can filter only the job postings of the technology/position we are interested).

# postings/spiders/duckduckgo.py
import itertools
from urllib.parse import urlparse
import scrapy
import w3lib.url

class DuckDuckGoSpider(scrapy.Spider):
    name = "duckduckgo"
    allowed_domains = ["duckduckgo.com"]

    def start_requests(self):
        ...

    def parse(self, response, keyword):
        ...

Inside start_request method, we schedule the initial requests sending search queries to DuckDuckGo passing domains and keywords we are interested.

Adding meta={"playwright": True} to the request ensures that a real browser (managed by playwright and scrapy-playwright) will be used. This will help us not to be easily identified as a bot (and blocked). Using a real browser (instead of sending plain requests only) will make our spider to be slower. Given that this spider will not be running frequently, we can accept the slowness of opening a real browser and perform the operations on it.

We also add cb_kwargs as a way to send some metadata to the request that will be used later to add extra information in the data returned.

    def start_requests(self):
        keywords = ["python", "django", "flask", "fastapi"]
        domains = ["jobs.lever.co", "boards.greenhouse.io"]

        for domain, keyword in itertools.product(domains, keywords):
            yield scrapy.Request(
                f"https://duckduckgo.com/?q=site%3A{domain}+{keyword}",
                meta={"playwright": True},
                cb_kwargs={"keyword": keyword},
            )

Next step is to implement parse method that will get the response from DuckDuckGo and gather the URLs of the job postings.

    def parse(self, response, keyword):
        for url in response.css("a::attr(href)").getall():
            parsed_url = urlparse(url)
            if parsed_url.netloc not in ["jobs.lever.co", "boards.greenhouse.io"]:
                # Ignore results not in the domains we are interested
                continue

            company_name = parsed_url.path.split("/")[1]

            yield {
                "company": company_name,
                "job_posting_url": url,
                "keyword": keyword,
            }

Running the Spider and exporting the results into a CSV file.

`1`	`scrapy crawl duckduckgo -o job_postings.csv`

And now we have an initial version of our spider and we are able to collect links of job postings that matches our search keywords.

Handling page pagination

This initial run of the spider probably will return just a few results (~10). Performing this search manually, we will notice that what we got until now is just the first page of the results. Looking at DuckDuckGo results page, we can notice in the bottom the following button that when clicked will load more results.

scrapy-playwright (and playwright) allows us to perform actions on pages such filling forms or clicking elements, so we can change our spider that when we perform the search, we will click in More Results button until we can’t find it in the page anymore (indicating that we don’t have more hidden results to be shown).

First we create our function that will get the page, and click the button as many times as needed. When the button is not visible anymore (we reach our maximum number of results), PlaywrightTimeoutError is raised stopping the interactions with the page and releasing it to be parsed.

# postings/spiders/duckduckgo.py
from playwright.async_api import TimeoutError as PlaywrightTimeoutError


async def more_results(page):
    while True:
        try:
            await page.locator(selector="#more-results").click()
        except PlaywrightTimeoutError:
            break
    return page.url

Then we add playwright_page_methods with the list of methods that we want to be called on the page.

# postings/spiders/duckduckgo.py

class DuckDuckGoSpider(scrapy.Spider):
    # (...)

    def start_requests(self):
        keywords = ["python", "django", "flask", "fastapi"]
        domains = ["jobs.lever.co", "boards.greenhouse.io"]

        for domain, keyword in itertools.product(domains, keywords):
            yield scrapy.Request(
                f"https://duckduckgo.com/?q=site%3A{domain}+{keyword}",
                meta={
                    "playwright": True,
                    "playwright_page_methods": [
                        PageMethod(more_results),
                    ],
                },
                cb_kwargs={"keyword": keyword},
            )

Running the Spider again, we will get all the results.

Removing duplicated results

After running the Spider, we will notice that some URLs are duplicated. One of the reasons is that more than one search keyword can return the same job posting result (we would expect a Django job posting also having Python keyword inside it).

To drop the duplicated values, we can create an Item Pipeline that will check, for each job posting returned whether it was returned before or not.

An item pipeline is a simple class that implements process_item method that receives items returned by the Spider, perform some processing on the item and then return it or drop it. When enabled all items returned by DuckDuckGoSpider will pass through it.

A seen_urls set is defined, so we can check if that particular URL was already returned (and then we can skip it) or if it is a new URL.

# postings/pipelines.py
from scrapy.exceptions import DropItem


class JobPostingDuplicatesPipeline:
    seen_urls = set()

    def process_item(self, item, spider):
        if item["job_posting_url"] in self.seen_urls:
            # We drop items which the URL was already returned before
            raise DropItem("Already returned")

        # Add the URL to the set when it was processed for the first time
        self.seen_urls.add(item["job_posting_url"])

        yield item

We need to enable this pipeline in our project.

# postings/settings.py
# (...)

ITEM_PIPELINES = {
   "postings.pipelines.JobPostingDuplicatesPipeline": 300,
}

We can run again the Spider, exporting the results into a CSV file. This time, we will have only unique job posting URLs.

`1`	`scrapy crawl duckduckgo -o job_postings.csv`

Possible improvements

This is the starting point of our job search. It was useful to me because it returned some companies that I have never heard about it, looking for professionals with the skills that I have. Certainly this narrowed more my options and helped me to apply to positions that made more sense to me.

A possible improvement would be create a specific spider for each recruitment platform to parse the content of the job postings. This would allow us to filter even more our data. For example, we can check for specific benefits or other keywords that would be useful for us to decide to apply or not to the job.

Summary

Here we have the complete code:

# postings/pipelines.py
import itertools
from urllib.parse import urlparse

import scrapy
import w3lib.url
from playwright.async_api import TimeoutError as PlaywrightTimeoutError
from scrapy_playwright.page import PageMethod


async def more_results(page):
    while True:
        try:
            await page.locator(selector="#more-results").click()
        except PlaywrightTimeoutError:
            break
    return page.url


class DuckDuckGoSpider(scrapy.Spider):

    def start_requests(self):
        keywords = ["python", "django", "flask", "fastapi"]
        domains = ["jobs.lever.co", "boards.greenhouse.io"]

        for domain, keyword in itertools.product(domains, keywords):
            yield scrapy.Request(
                f"https://duckduckgo.com/?q=site%3A{domain}+{keyword}",
                meta={
                    "playwright": True,
                    "playwright_page_methods": [
                        PageMethod(more_results),
                    ],
                },
                cb_kwargs={"keyword": keyword},
            )

    def parse(self, response, keyword):
        for url in response.css("a::attr(href)").getall():
            parsed_url = urlparse(url)
            if parsed_url.netloc not in ["jobs.lever.co", "boards.greenhouse.io"]:
                # Ignore results not in the domains we are interested
                continue

            company_name = parsed_url.path.split("/")[1]

            yield {
                "company": company_name,
                "job_posting_url": url,
                "keyword": keyword,
            }

# postings/settings.py
# (...)

# Add the following to the existing file
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

ITEM_PIPELINES = {
   "postings.pipelines.JobPostingDuplicatesPipeline": 300,
}

# postings/pipelines.py
from scrapy.exceptions import DropItem


class JobPostingDuplicatesPipeline:
    seen_urls = set()

    def process_item(self, item, spider):
        if item["job_posting_url"] in self.seen_urls:
            # We drop items which the URL was already returned before
            raise DropItem("Already returned")

        # Add the URL to the set when it was processed for the first time
        self.seen_urls.add(item["job_posting_url"])

        yield item

Good luck with your job hunt!