[core][datasets] Worker raylets OOM when using smaller machines and default object store memory

[stephanie-wang]

What happened + What you expected to happen

Currently the datasets_shuffle_* nightly tests use 32-vCPU large machines and limit the object store memory to only a fraction of the available RAM. This means that we're probably underutilizing the machines and it's not very representative to a real setup either. Previous attempts to use smaller instance types / larger object store memory cause worker raylet OOMs, such as in datasets_shuffle_random_shuffle_1tb.

Here's some example output from dmesg on a worker raylet.

Versions / Dependencies

2.0dev

Reproduction script

datasets_shuffle_* nightly tests, and changing the instance type in the cluster config.

Issue Severity

No response

[stephanie-wang]

A few odd things from the above dmesg output that we should look into, looking at the RSS column (multiply by 4096 to get # bytes):

• raylet memory usage is very high. anon-rss (so not including shared memory) at time of OOM is ~7GB.
• reduce tasks should be reading and writing 1GB partitions, but table shows rss is sometimes more than 10GB. Even including inputs/outputs in shared memory, this seems pretty high for heap memory usage (8x).
• unclear if I/O worker memory usage is as expected since there is no standard spill size

[mwtian]

I did not reproduce worker OOM yet, but managed to cause head raylet to exit unexpectedly with m5.2xlarge instances. We should first change the dataset_shuffle_* tests to use a smaller instance type like m5.4xlarge, and see if the tests can pass continuously. I'm investigating the head raylet issue with m5.2xlarge instances in parallel.

[mwtian]

Current status:

• [Core] Batch PinObjectIDs requests from Raylet client #24322 should decrease driver Raylet memory usage significantly, based on profiling and testing.
• For worker Raylet memory usage, it is observed that the memory usage can grow significantly during the reduce stage. Memory profiling is a bit harder to interpret in this case. I'm still investigating the source of memory usages.
• Python worker memory usage can be usually high (4GB~6GB) even when idle. Investigation has not started on this yet.

[mwtian]

5a82640 should reduce driver raylet memory usage. For Python workers, it seems their RSS memory is mostly shared memory usage, e.g. a Python worker with 6.6GB RSS uses about 6.4GB shared memory.