[data] Ray Autoscaling - Suboptimal Performance with Actors

[praateekmahajan]

Pipeline Description

Similar pipeline as #54433

1. We have a I/O stage which downloads from internet (~100 seconds)
2. We have an iterate stage (essentially unzips the file and parses it as pd.DataFrame; ~30 seconds)
3. We have an extract stage (that performs some expensive compute, super expensive ~700 seconds)
4. We have a write stage

Between each of the stages we pass a python object.

Setup

We can run this "pipeline" both as tasks or actors.

1. When using tasks, we don't specify concurrency. The runtime is 2x faster than using actors.
2. When using actors we specify (1, 16) as concurrency.

Problem

We have 16 CPUs and ~4 stages.
I only see ~6 cpus being used instead of all 16. For any of the stages the concurrency=(1,16) but none of the stages are at peak capacity.
Given the skew in time taken per stage, I'd imagine the slowest stage is running at peak capacity. But I'm interested in learning why / why not.

Screenshots

If you notice in the the following screenshot, Iterate already has produced 25 tasks, and naturally so it's been downscaled to 1 actor. However DocumentExtract (our slowest stage) is only at 4 actors. I'd imagine at this point since Extraction has a lot of pending work in its queue, we'll scale it up to max capacity but we didn't.

After some time, if you notice in the following screenshot, Iterate ends up producing all the 48 tasks (even with its single actor). However DocumentExtract still throughout the job never exceeded 4 actors. (I notice that at the time of this screenshot we have only 4 tasks left so 4 actors make sense, but throughout the job we didn't exceed 4 actors for this job)

Question

1. At any point, since Iterate has produced so much "output" which is in memory, won't we want to scale up Iterate+1'th stage to scale up, to start popping from the queue?
2. There is a 2x perf difference when comparing tasks vs actors. I understand its not a fair comparision since in Task world, Iterate/Extract/Write are all fused. But even then the workload distribution seems off

[alexeykudinkin]

There is a 2x perf difference when comparing tasks vs actors. I understand its not a fair comparision since in Task world, Iterate/Extract/Write are all fused. But even then the workload distribution seems off

As you already pointing out this is not an accurate comparison -- if you want to compare performance b/w Tasks vs Actors, you'd need to disable fusion first.

The reason why you'd see superior performance with 1 operator has many factors but the most important ones are:

• Avoids need to serialize/de blocks
• Avoids data transfer
• More straightforward resource allocation

[praateekmahajan]

Yup, here is our Actor only implementations repro in case others are also interested NVIDIA-NeMo/Curator#883 where we see the backpressure and "suboptimal" perf

[alexeykudinkin]

Summarizing observed issues here:

Download Stage Hogging all Resources

As highlighted in NVIDIA-NeMo/Curator#883, DocumentDownloadStageActor seems to hog all of the resources initially depriving the following stages of resources.

This is a known problem that has recently already been addressed by #55673 and will be released in 2.49