Allow SitemapRequestLoader to accept Request objects for sitemap_urls

[loic-bellinger]

Hi team,

First off, thanks for the great work on crawlee-python!

I'm proposing a small enhancement to make the SitemapRequestLoader more flexible: allowing the sitemap_urls parameter to accept a Sequence[str | Request].

This would allow users to attach metadata via user_data directly to sitemap sources, which is invaluable for data enrichment.

Use Case

A common task is scraping sites from different companies, where each company has its own metadata (e.g., a company_id). We need to attach this company_id to every page scraped from that company's sitemap.
Currently, this requires indirect workarounds because the context is lost when only URL strings are passed to the loader.

Proposed Solution

If the loader accepted Request objects, the process would be much cleaner:

sitemap_sources = [
    Request(
        url="https://company-a.com/sitemap.xml",
        user_data={"company_id": "A123", "source_type": "partner"}
    ),
    Request(
        url="https://company-b.com/sitemap.xml",
        user_data={"company_id": "B456", "source_type": "competitor"}
    )
]
loader = SitemapRequestLoader(sitemap_urls=sitemap_sources, ...)

The user_data could then be propagated to every URL discovered within that sitemap, making data enrichment direct and reliable.

This change would be fully backward-compatible. Thanks for considering it

[janbuchar]

Hello @loic-bellinger and thanks for using Crawlee! While I understand the use case, I'm afraid that implicitly propagating the user_data would be hard to discover for potential users and might even cause some confusion down the road.

However, it is true that this kind of behavior is pretty hard to achieve with the current sitemap loader. Paging @Mantisus as the author of the current implementation - any idea how to hack it together or how to improve the loader so that it's possible?

[Mantisus]

any idea how to hack it together or how to improve the loader so that it's possible?

I think we could add support for transform_request_function. It will accept RequestOptions with url. And this will allow creating Request with user parameters.

It is more transparent and is already used in EnqueueLinks.

[Mantisus]

And this transform_request_function would be a parameter of the SitemapRequestList, correct?

For SitemapRequestLoader, yes 🙂

But maybe it can be transformed into something useful?

Perhaps some custom middleware could be passed to RequestLoader.to_tandem, allowing custom code to be injected to modify or extend Request after RequestLoader before it is passed to the crawler.

However, this approach would require more code from users and may seem more complicated. And if it is only used to add user_data to Request, it seems like over-engineering. 😄

[loic-bellinger]

Based on the current RequestOptions signature (which already includes user_data), I’m trying to understand how a transform_request_function would enable per‑sitemap metadata propagation. My understanding is:

• A transform function (as used in EnqueueLinks) receives only the child RequestOptions (url, optional user_data, etc.), not the "origin sitemap" context.
• If a single SitemapRequestLoader processes multiple sitemap URLs, each with different metadata, the transform won’t know which sitemap a discovered URL came from, and therefore can’t reliably apply the correct metadata unless it receives that origin context.

Could you confirm whether this interpretation is correct?

But, if sitemap_urls accepted Sequence[str | Request], the loader could propagate request.user_data from the sitemap source to each discovered URL. That would be explicit and discoverable. It does raise one behavior question: if the same URL appears in multiple sitemaps with different metadata, should it be crawled again with the new metadata, or should there be a deduplication/precedence rule?

Given that my initial use case involves many different hosts, each with its own sitemap,, i'm starting to wonder if that would be simpler to create one crawler for each host and then rely on use_state to access the metadata i need. What do you think ?

[Mantisus]

Could you confirm whether this interpretation is correct?

That's right, but since it's a user function, you can do mapping. For example, based on the hostname of the input URL to the sitemap with the hostname in RequestOptions.url.

But, if sitemap_urls accepted Sequence[str | Request], the loader could propagate request.user_data from the sitemap source to each discovered URL

user_data is mutable and also stores some technical fields. Therefore, in certain cases, this approach can lead to problems that are difficult to diagnose. It is not desirable.

if the same URL appears in multiple sitemaps with different metadata, should it be crawled again with the new metadata

The appearance of the same URL in different sitemaps indicates a problem with the site's sitemap. This will not be deduplicated, as storing all received URLs in memory can be a problem for huge sitemaps with tens of millions of URLs.
The idea of completely processing the sitemap due to different metadata sounds like overhead to me. And this is something that needs to be processed before being sent to the crawler.

Given that my initial use case involves many different hosts, each with its own sitemap,, i'm starting to wonder if that would be simpler to create one crawler for each host and then rely on use_state to access the metadata i need. What do you think ?

This approach will indeed be simpler, both in terms of implementation and support.

[Mantisus]

This PR #1525 adds support transform_request_function and a documentation example for mapping a Request to a user function with the original sitemap URL.

[loic-bellinger]

Very interesting thanks. In the meantime, since i was doing a per-host crawling i overrided the SitemapRequestLoader:

from crawlee.request_loaders import SitemapRequestLoader
from crawlee import Request
from typing import override, Any

class UserDataSitemapRequestLoader(SitemapRequestLoader):
    """
    An extended version of SitemapRequestLoader that attaches a custom
    `user_data` dictionary to every Request object it yields.
    """
    def __init__(self, user_data: dict[str, Any], **kwargs):
        """Initializes the loader with the user_data to inject."""
        self._user_data = user_data
        super().__init__(**kwargs)

    @override
    async def fetch_next_request(self) -> Request | None:
        """
        Overrides the base method to fetch the next request and then inject
        the user_data into it before returning it.
        """
        request = await super().fetch_next_request()
        if request:
            if not request.user_data:
                request.user_data = {}
            request.user_data.update(self._user_data)
        return request

I guess now I can just use the transform_request_function parameter to pass my user_data.

[vdusek]

SitemapRequestLoader with transform_request_function was released in Crawlee 1.1. Closing the discussion.