Add option to respect robots.txt disallows

[tw4l]

Fixes #631

This PR introduces a --robots CLI flag to the crawler. When enabled, the crawler will attempt to fetch robots.txt for each host encountered during crawling, and if found will check if pages are disallowed before adding them to the crawl queue.

Fetched robots.txt bodies are cached by their URL in Redis using an LRU cache mechanism which retains the 100 most recently accessed ones in the cache to prevent memory usage from getting out of control.

Robots.txt bodies are parsed and checked for page allow/disallow status using the https://github.com/samclarke/robots-parser library, which is the most active and well-maintained implementation I could find with TypeScript types.

I have added basic tests that check the log lines to ensure robots.txt is being fetched and cached, but I haven't been able to find a robots.txt on a Webrecorder-managed domain with disallows to test that disallowed URLs are not actually queued. Maybe we could set up a single disallow on our website's robots.txt for this purpose?

Manual testing

1. In util/constants.ts, set ROBOTS_CACHE_LIMIT to 2
2. Re-build the crawler image and run a crawl across multiple hosts, with --robots and --logging debug enabled and a mixture of URLs that do and do not have a robots.txt specified, and for those that do, a mixture of URLs that will be allowed and disallowed, e.g.:

docker compose build
docker compose run crawler crawl --url https://forums.gentoo.org --url https://forums.gentoo.org/search.php --url https://forums.gentoo.org/statistics.php --url https://forums.gentoo.org/viewforum-f-33.html\?sid\=ed69503a6d2b866c8bcc28307f8e7d59 --url https://webrecorder.net/browsertrix --url https://webrecorder.net --url https://bl.uk --url https://bl.uk/collection --url https://bl.uk/visit --url https://www.kb.dk --url https://www.kb.dk/README.txt --url https://www.kb.dk/search/ --url https://www.kb.dk/en/find-materials --url https://bitarchivist.net --limit 20 --scopeType page --collection robots-cache --robots --generateWACZ --logging debug,errors

3. Inspect the crawl log for relevant messages with robots and links context. What should be present:

Log lines for fetching and caching robots.txt

{"timestamp":"2025-09-30T14:45:03.888Z","logLevel":"debug","context":"robots","message":"Fetching robots.txt","details":{"url":"https://forums.gentoo.org/robots.txt"}}
{"timestamp":"2025-09-30T14:45:04.226Z","logLevel":"debug","context":"robots","message":"Caching robots.txt body","details":{"url":"https://forums.gentoo.org/robots.txt"}}

Log lines for using cached robots.txt

{"timestamp":"2025-09-30T14:45:04.234Z","logLevel":"debug","context":"robots","message":"Using cached robots.txt body","details":{"url":"https://forums.gentoo.org/robots.txt"}}

Log lines for not queueing page URLs disallowed by robots

{"timestamp":"2025-09-30T14:45:04.235Z","logLevel":"debug","context":"links","message":"Page URL not queued, disallowed by robots.txt","details":{"url":"https://forums.gentoo.org/search.php"}}

Log lines for deleting least-recently accessed cached robots.txt when over cache limit

{"timestamp":"2025-09-30T14:45:04.843Z","logLevel":"debug","context":"robots","message":"Deleting cached robots.txt, over cache limit","details":{"url":"https://forums.gentoo.org/robots.txt"}}

Log lines for hosts where robots.txt cannot be found or fetched

{"timestamp":"2025-09-30T14:45:05.619Z","logLevel":"debug","context":"robots","message":"Robots.txt not fetched","details":{"url":"https://bitarchivist.net/robots.txt","status":404}}