Add support for non-default unnamed storages

[vdusek]

Problem

In local development, each run reuses the same default storages, which are wiped clean at the beginning. This behavior is intentional and should be preserved.

On the Apify platform, however, every Actor run automatically receives a new set of unnamed storages. Since these storages are unnamed, they are subject to the default 30-day retention policy.

The Apify Python client already exposes the ability to create new unnamed storages:

await DatasetCollectionClientAsync.get_or_create()

Each call creates a new unnamed dataset with a unique ID.

Crawlee (and the underlying SDK), by contrast, does not currently support this. For example, repeated calls to:

await Dataset.open()

always return the same default unnamed storage.

Motivation

In more complex scenarios (e.g., WCC), an Actor may require multiple storage instances within a single run (for example, multiple request queues in WCC). To support this, we must allow creating additional unnamed storages beyond the single default instance - non-default unnamed storages (NDU). This provides flexibility for advanced workflows and better alignment with the platform capabilities.

Goal state

Bring Crawlee storages (all storage clients, including ApifyStorageClient) to feature parity with the Apify platform by supporting non-default unnamed storages.

Preserve the current behavior of local storage clients: wipe storages at the start of each run and reuse the same default storage across runs.

Possible solution 1) - new argument

Introduce a new argument to the storage open constructor:

async def open(
    cls,
    name: str | None = None,
    id: str | None = None,
    scope: Literal['run', 'global'] = 'global',
) -> Dataset | KeyValueStore | RequestQueue:
    ...

• scope='run' indicates a non-default unnamed storage.
• scope='global' refers to globally named storages.
• The name parameter cannot be entirely removed for run scope storages, as it's needed for implementation:
  • For the filesystem storage: to use as a directory name.
  • For Apify platform storage: to store the mapping of name -> ID in the default key-value store.

Behavior matrix...

Open storage by ID and name

• Raise an exception (choose one of these).
• Scope argument is ignored.

Open storage by ID

• Opens an existing storage by ID.
• Scope?

Open storage by name

• Scope run:
  • Opens or creates a run-scope (non-default unnamed) storage.
    • name is used internally for reference-storage purposes but is not the actual storage's "name".
• Scope global:
  • Opens or creates a global named storage.

Open storage without args

• Opens the default unnamed storage.
• Scope argument is ignored.

Possible solution 2) - new constructor

Introduce an alternative constructor for non-default unnamed storages (e.g., open_by_alias or similar):

await Dataset.open_2(
    cls,
    alias: str,
)

• Each call to open_new() creates a new unnamed storage with a unique ID (mirroring the Apify API client).
• This also keeps the behavior of open() unchanged.

Another solution?

Discussion

• Both options have trade-offs:
  • Adding parameters to open() results in confusing combinations or a complex signature.
  • Alternative constructors may overwhelm users with choices from the start, and would need to be implemented for every storage type, and all related convenient helpers in crawlers & contexts.

[janbuchar]

When opening the storage by ID, the scope does not make sense. I think an exception would be appropriate.

[B4nan]

We should ask for more feedback, e.g. on slack

[metalwarrior665]

So to create a new persisted unnamed dataset, you would call Dataset.open(name='debug', scope='run') and then every time you call this (even after migration), it would return the same dataset, right?

Before releasing, I would have a short sync with the platform/output schema. There is e.g. this proposal so to make sure we don't use completely different terms apify/actor-whitepaper#25

[janbuchar]

So to create a new persisted unnamed dataset, you would call Dataset.open(name='debug', scope='run') and then every time you call this (even after migration), it would return the same dataset, right?

Yes, spot on. With the caveat that Dataset.open(name='debug') will open a different, global dataset. Perhaps we could just throw if multiple open calls share the same name but use different scopes.

Before releasing, I would have a short sync with the platform/output schema. There is e.g. this proposal so to make sure we don't use completely different terms apify/actor-whitepaper#25

The suggested implementation won't need any support from the platform side, but it's always a good idea to sync on terminology.

[vdusek]

Overview of the variants from the user experience perspective...

1) Scope version

• Scope global remains the default option.

Direct usage

• Same for all types of storages.

# Default dataset
default_dataset = await Dataset.open()

# Run scope dataset
dataset_run_scope = await Dataset.open(name='dataset_run_scope', scope='run')

# Global scope dataset
dataset_global_scope = await Dataset.open(name='dataset_global_scope', scope='global')

# And then all the methods remain the same…

Context helpers

• Helpers on the crawling context.

Push data

# Default dataset
await context.push_data(data)

# Global scope dataset
await context.push_data(data, dataset_name='dataset_global_scope', scope='global')

# Run scope dataset
await context.push_data(data, dataset_name='dataset_run_scope', scope='run')

Add requests

• Currently, there is no option for specifying the destination (always the default RQ); but we can add it.

# Default RQ
await context.add_requests(requests)

# Global scope RQ
await context.add_requests(requests, rq_name='rq_global_scope', scope='global')

# Run scope RQ
await context.add_requests(requests, rq_name='rq_run_scope', scope='run')

Enqueue links

• Currently, there is no option for specifying the destination (always the default RQ); but we can add it.

# Default RQ
await context.enqueue_links()

# Global scope RQ
await context.enqueue_links(rq_name='rq_global_scope', scope='global')

# Run scope RQ
await context.enqueue_links(rq_name='rq_run_scope', scope='run')

Get KVS

# Default KVS
kvs = await context.get_key_value_store()

# Global scope KVS
kvs_global = await context.get_key_value_store(name='kvs_global_scope', scope='global')

# Run scope KVS
kvs_run = await context.get_key_value_store(name='kvs_run_scope', scope='run')

Use state

• Should always use the default one.

Crawler helpers

• Helpers on the crawlers.

Export data

# Default dataset
data = await crawler.export_data()

# Global scope dataset
data = await crawler.export_data(dataset_name='dataset_global_scope', scope='global')

# Run scope dataset
data = await crawler.export_data(dataset_name='dataset_run_scope', scope='run')

Get dataset

# Default dataset
dataset = await crawler.get_dataset()

# Global scope dataset
dataset = await crawler.get_dataset(name='dataset_global_scope', scope='global')

# Run scope dataset
dataset = await crawler.get_dataset(name='dataset_run_scope', scope='run')

Get key value store

# Default KVS
kvs = await crawler.get_key_value_store()

# Global scope KVS
kvs = await crawler.get_key_value_store(name='kvs_global_scope', scope='global')

# Run scope KVS
kvs = await crawler.get_key_value_store(name='kvs_run_scope', scope='run')

Get request manager

• It returns the configured request manager, so it is not affected.

Add requests

• It uses the underlying request manager, so it is not affected.

2) Alias version

Direct usage

• Same for all types of storages.

# Default dataset
default_dataset = await Dataset.open()

# Global scope dataset (name)
dataset_global_scope = await Dataset.open(name='dataset_global_scope')

# Run scope dataset (alias)
dataset_run_scope = await Dataset.open(alias='dataset_run_scope')

# And then all the methods remain the same…

Context helpers

• Helpers on the crawling context.

Push data

# Default dataset
await context.push_data(data)

# Global scope dataset (name)
await context.push_data(data, dataset_name='dataset_global_scope')

# Run scope dataset (alias)
await context.push_data(data, dataset_alias='dataset_run_scope')

Add requests

Currently, there is no option for specifying the destination (always the default RQ); but we can add it.

# Default RQ
await context.add_requests(requests)

# Global scope RQ (name)
await context.add_requests(requests, rq_name='rq_global_scope')

# Run scope RQ (alias)
await context.add_requests(requests, rq_alias='rq_run_scope')

Enqueue links

Currently, there is no option for specifying the destination (always the default RQ); but we can add it.

# Default RQ
await context.enqueue_links()

# Global scope RQ (name)
await context.enqueue_links(rq_name='rq_global_scope')

# Run scope RQ (alias)
await context.enqueue_links(rq_alias='rq_run_scope')

Get KVS

# Default KVS
kvs = await context.get_key_value_store()

# Global scope KVS (name)
kvs_global = await context.get_key_value_store(name='kvs_global_scope')

# Run scope KVS (alias)
kvs_run = await context.get_key_value_store(alias='kvs_run_scope')

Use state

• Should always use the default one.

Crawler helpers

• Helpers on the crawlers.

Export data

# Default dataset
data = await crawler.export_data()

# Global scope dataset (name)
data = await crawler.export_data(dataset_name='dataset_global_scope')

# Run scope dataset (alias)
data = await crawler.export_data(dataset_alias='dataset_run_scope')

Get dataset

# Default dataset
dataset = await crawler.get_dataset()

# Global scope dataset (name)
dataset = await crawler.get_dataset(name='dataset_global_scope')

# Run scope dataset (alias)
dataset = await crawler.get_dataset(alias='dataset_run_scope')

Get key value store

# Default KVS
kvs = await crawler.get_key_value_store()

# Global scope KVS (name)
kvs = await crawler.get_key_value_store(name='kvs_global_scope')

# Run scope KVS (alias)
kvs = await crawler.get_key_value_store(alias='kvs_run_scope')

Get request manager

• It returns the configured request manager, so it is not affected.

Add requests

• It uses the underlying request manager, so it is not affected.