[ci][core] Perf regression on tasks_per_second, pgs_per_second

[rickyyx]

What happened + What you expected to happen

Regression on:

Versions / Dependencies

5a7071e [Core] Remove dead gcs monitor service (#41262)
1a986de [data] add support for multiple group keys in map_groups (#40778)
84c0ba0 [Data] Expose max_retry_cnt parameter for BigQuery Write (#41163)
25bee34 [GCS FT] Update Redis connection configs (#40860)
709dc1b [ci] build anyscale images for gcp (#41224)
31c6631 Make Ray compatible with pydantic>=2.5.0 (#40451)
9dec618 [ci] mark two core functions as flaky (#41237)
9a34839 [Doc][KubeRay]: Redis eviction suggestions when ENABLE_GCS_FT_REDIS_CLEANUP=false (#40949)
29dba63 [KubeRay] [Doc] Update the link of KubeRay API reference (#41236)
7c7eaf2 [Train][Tune] Support reading train result from cloud storage (#40622)
7a75d09 [core][gcs] Remove ByteSizeLong call from GcsTaskManager (#41108)
8209893 [core] make ProcessFD, [Client,Server]Connection non copyable. (#41106)
ca29fec [RLlib] New ConnectorV2 API #1: Some preparatory cleanups and fixes. (#41074)
0e2a523 Make Ray work on GH200 (#40816)

Reproduction script

NA

Issue Severity

None

[rickyyx]

Looks like #40860 introduces the regression (at least for tasks_per_second on many_tasks.aws)

Before (7a75d096a4): 518: https://buildkite.com/ray-project/release/builds/2052#018bf84d-2445-4bf2-aea1-c737f5f77a24

At 25bee34: 425: https://buildkite.com/ray-project/release/builds/2053#018bf84d-6da1-4c07-ae6b-2d9fb69b7623

[rickyyx]

cc @iycheng

[rickyyx]

Still bisecting - the previously thought PR already had regression in it. (Reverting doesn't work).

[rickyyx]

Ok this is the pr that introduces the regression: #40451

before this: https://buildkite.com/ray-project/release/builds/2308#018c1376-2be9-4fd0-ab4a-5d41e845e9c1 -> tasks_per_sec = 530
at this: https://buildkite.com/ray-project/release/builds/2306#018c137a-9694-455d-a113-cff39815fe87 -> tasks_per_sec = 423

[rkooo567]

Ok this is the pr that introduces the regression: #40451

@rickyyx since you fixed this issue, can we close it now?

[rickyyx]

Yeah.