-
Notifications
You must be signed in to change notification settings - Fork 837
Allow setting ring heartbeat timeout to zero to disable timeout check. #4342
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
566df03 to
16a505e
Compare
This change allows the various ring heartbeat timeouts to be configured with zero, as a means of disabling the timeout. This is expected to be used with a separate enhancement to allow disabling heartbeats. When the heartbeat timeout is disabled, instances will always appear as healthy in the ring. Signed-off-by: Steve Simpson <[email protected]>
16a505e to
e0bd75e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This change LGTM. I guess we'll need a follow up PR to also allow disabling heartbeating the ring at all (this PR just checks the timeout). What your plans on this?
About this PR: I would mention it as an experimental feature in docs/configuration/v1-guarantees.md
|
The two changes together were very confusing as they spread to so many bits of code in subtly different ways, so the disabling is here: #4344 |
Signed-off-by: Steve Simpson <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm - I don't think the word "experimental" is necessary but not going to argue.
cortexproject#4342) * Allow setting ring heartbeat timeout to zero to disable timeout check. This change allows the various ring heartbeat timeouts to be configured with zero, as a means of disabling the timeout. This is expected to be used with a separate enhancement to allow disabling heartbeats. When the heartbeat timeout is disabled, instances will always appear as healthy in the ring. Signed-off-by: Steve Simpson <[email protected]> * Review comments. Signed-off-by: Steve Simpson <[email protected]> Signed-off-by: Alvin Lin <[email protected]>
## What Removes the experimental feature that allowed setting heartbeat period or timeout to 0 to disable them. ## Why This feature was added in cortexproject/cortex#4342 but is not used in production. Removing unused experimental features simplifies the codebase and reduces maintenance burden. ## Changes - Remove "0 = disabled" from flag descriptions for `-*.ring.heartbeat-period` and `-*.ring.heartbeat-timeout` - Add validation to `LifecyclerConfig.Validate()` and `Config.Validate()` to reject zero values - Delete `TestRestartIngester_DisabledHeartbeat_unregister_on_shutdown_false` test - Update tests that were using zero heartbeat values to use non-zero values - Fix metric tests to use recent timestamps that work with non-zero heartbeat timeouts
## What Removes support for setting heartbeat period or timeout to 0 to disable them. ## Why This experimental feature was added in cortexproject/cortex#4342 but is not used in production. Removing unused experimental features simplifies the codebase. ## Changes - Remove "Hash ring" section from experimental features documentation listing the heartbeat disabling flags - Add CHANGELOG entry under `[CHANGE]` scope - Update dskit dependency to include validation changes (via replace directive for now) ## Dependencies This PR depends on grafana/dskit#759 which must be merged first. After that PR is merged, this PR will be updated to: 1. Remove the replace directive 2. Update dskit to the merged commit 3. Update the CHANGELOG with the actual PR number
What this PR does:
This change allows the various ring heartbeat timeouts to be configured
with zero, as a means of disabling the timeout. This is expected to be
used with a separate enhancement to allow disabling heartbeats. When the
heartbeat timeout is disabled, instances will always appear as healthy
in the ring.
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]