Ignore long-running requests on query-scheduler when checking for high latency. (#242)

pstibrany · web-flow · commit 8d1ecc7cefd4 · 2021-01-06T12:17:49.000+01:00
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,6 +7,7 @@
   - Cortex / Queries: added bucket index load operations and latency (available only when bucket index is enabled)
   - Alerts: added "CortexBucketIndexNotUpdated" (bucket index only) and "CortexTenantHasPartialBlocks"
 * [BUGFIX] Honor configured `per_instance_label` in all panels. #239
+* [BUGFIX] `CortexRequestLatency` alert now ignores long-running requests on query-scheduler. #242
 
 ## 1.6.0 / 2021-01-05
 
diff --git a/cortex-mixin/alerts/alerts.libsonnet b/cortex-mixin/alerts/alerts.libsonnet
@@ -39,7 +39,7 @@
         {
           alert: 'CortexRequestLatency',
           expr: |||
-            cluster_namespace_job_route:cortex_request_duration_seconds:99quantile{route!~"metrics|/frontend.Frontend/Process|ready"}
+            cluster_namespace_job_route:cortex_request_duration_seconds:99quantile{route!~"metrics|/frontend.Frontend/Process|ready|/schedulerpb.SchedulerForFrontend/FrontendLoop|/schedulerpb.SchedulerForQuerier/QuerierLoop"}
                >
             %(cortex_p99_latency_threshold_seconds)s
           ||| % $._config,

Original file line number	Diff line number	Diff line change
`@@ -39,7 +39,7 @@`
`39`	`39`	`{`
`40`	`40`	`alert: 'CortexRequestLatency',`
`41`	`41`	`expr: \|\|\|`
`42`		`- cluster_namespace_job_route:cortex_request_duration_seconds:99quantile{route!~"metrics\|/frontend.Frontend/Process\|ready"}`
	`42`	`+ cluster_namespace_job_route:cortex_request_duration_seconds:99quantile{route!~"metrics\|/frontend.Frontend/Process\|ready\|/schedulerpb.SchedulerForFrontend/FrontendLoop\|/schedulerpb.SchedulerForQuerier/QuerierLoop"}`
`43`	`43`	`>`
`44`	`44`	`%(cortex_p99_latency_threshold_seconds)s`
`45`	`45`	`\|\|\| % $._config,`