Implement an async execution mode to run Solid Queue workers and dispatchers in the same process
#269
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
That is, no more "work" or "dispatch". The behaviour is the same as "all", and to get work only or dispatch only it is enough with passing an empty array for dispatchers and workers.
And make Poller a class that the dispatcher and worker inherit from. It doesn't make sense to have a module there.
As it's just for the supervisor.
Modes will refer only to forked, async or inline. Stub fork mode for now.
As these won't necessarily be forks when introducing the async mode.
And affecting the result of `ReadyExecution.count`.
Also, rename SupervisorTest to ForkSupervisorTest, as that's the one being tested there.
No need to raise exceptions and rescue them, we can just check if a stop has been requested and terminate accordingly.
Set level to debug for successful enqueue + skipping, and error for errors, instead of info for everything.
Still not sold on that name, but Single is quite bad.
For worker and dispatcher, on errors outside their thread pool/tasks.
That won't run a loop for supervise, it'll just start the processes and the maintenance task.
… shutdown And add an observer for the Supervisor's maintenance task and the heartbeat task as well.
This is the first approach. Still needs refining.
Instead of periodically.
…r supervisor They don't claim executions, so no need to include that in the event.
That is, if there is some configuration not empty, do not fill missing dispatcher or worker with the default. Just assume no working/dispatching.
… Puma See #115 Co-authored-by: Ben Sheldon <[email protected]>
This was referenced Jul 29, 2024
`solid-queue-fork-supervisor` vs. `solid-queue-forksupervisor`.
|
We're now running this branch on HEY to ensure it still works well in the regular forked mode. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR builds on @bensheldon's initial spike in #115, to run workers and dispatchers in the same process as the supervisor instead of running them as forks. It also considerably refactors the way the supervisor works and it's started, to make room for this.
For now, running in this new
asyncmode is only supported via the Puma plugin but I plan to allow it also when running Solid Queue's supervisor alone in the near future, as a separate PR.This PR also introduces a breaking change in how Solid Queue is started. Prior this change, you could choose whether you wanted to run workers, dispatchers or both, by starting Solid Queue as
solid_queue:workorsolid_queue:dispatch. With this PR, the only option available issolid_queue:start. Whether the supervisor starts workers, dispatchers or both will depend on your configuration. For example, if you don't configure any dispatchers, only workers will be started. That is, with this configuration:the supervisor will run 3 workers, each one with 5 threads, and no supervisors.