Skip to content

Round-robin empty workers with queueing enabled #7222

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

Round-robin empty workers with queueing enabled #7222

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
545bdd0
Round-robin empty workers for queued tasks
gjoseph92 Oct 28, 2022
c578cb2
micro-optimization
gjoseph92 Oct 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 18 additions & 3 deletions distributed/scheduler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2124,9 +2124,24 @@ def decide_worker_rootish_queuing_enabled(self) -> WorkerState | None:
# All workers busy? Task gets/stays queued.
return None

# Just pick the least busy worker.
# NOTE: this will lead to worst-case scheduling with regards to co-assignment.
ws = min(self.idle.values(), key=lambda ws: len(ws.processing) / ws.nthreads)
# Pick the least-busy worker. If multiple workers have no tasks, round-robin
# them to ensure different workers get tasks on a quiet cluster.
empties: list[WorkerState] = []
min_ws: WorkerState | None = None
min_v: float | None = None
for cws in dict.values(self.idle):
# ^ micro-optimization: `SortedDict` inherits from plain `dict`; iterating
# in non-sorted order is 10x faster and order doesn't matter here.
Copy link
Collaborator Author

@gjoseph92 gjoseph92 Oct 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 10x number comes from here #4925 (comment)

Copy link
Collaborator Author

@gjoseph92 gjoseph92 Oct 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that if we do #7221 and/or #6974, then self.idle doesn't need to be a SortedDict anymore, so we'd then get rid of this micro-optimization. The only place using the sortedness of self.idle is the current round-robin logic.

Copy link
Collaborator Author

@gjoseph92 gjoseph92 Nov 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

xref #7245

v = len(cws.processing) / cws.nthreads
if min_v is None or v < min_v:
min_v = v
min_ws = cws
if v == 0:
empties.append(cws)

ws = empties[self.n_tasks % len(empties)] if empties else min_ws
assert ws

if self.validate:
assert not _worker_full(ws, self.WORKER_SATURATION), (
ws,
Expand Down