Issue on prometheus as a metrics provider for trimaran in openshift 4.7 cluster.

[sci110680]

Hi,
I am trying to use prometheus as a metrics provider for trimaran in openshift 4.7 cluster.
Getting below error in pod log.
Also i took secret from below command .

oc describe secret $(oc get secret -n openshift-monitoring|awk '{print $1}'|grep prometheus-k8s-token -m 1) -n openshift-monitoring|grep "token:"|cut -d: -f2|sed 's/^ *//g'

Error is :

ilocpsnd496.ocpd.corp.amdocs.com/api/v1/query\": x509: certificate signed by unknown authority\n"
time="2021-09-28T20:49:47Z" level=error msg="error querying Prometheus for query avg_over_time(instance:node_memory_utilisation:ratio[15m]): Post \"https://prometheus-k8s-openshift-monitoring.apps.ilocpsnd496.ocpd.corp.amdocs.com/api/v1/query\": x509: certificate signed by unknown authority\n"
time="2021-09-28T20:49:47Z" level=error msg="error querying Prometheus for query stddev_over_time(instance:node_cpu:ratio[15m]): Post \"https://prometheus-k8s-openshift-monitoring.apps.ilocpsnd496.ocpd.corp.amdocs.com/api/v1/query\": x509: certificate signed by unknown authority\n"
time="2021-09-28T20:49:47Z" level=error msg="error querying Prometheus for query stddev_over_time(instance:node_memory_utilisation:ratio[15m]): Post \"https://prometheus-k8s-openshift-monitoring.apps.ilocpsnd496.ocpd.corp.amdocs.com/api/v1/query\": x509: certificate signed by unknown authority\n"
time="2021-09-28T20:49:47Z" level=error msg="received error while fetching metrics: Post \"https://prometheus-k8s-openshift-monitoring.apps.ilocpsnd496.ocpd.corp.amdocs.com/api/v1/query\": x509: certificate signed by unknown authority"
I0928 20:49:59.764214       1 eventhandlers.go:283] delete event for scheduled pod openshift-marketplace/community-operators-jfhbr
W0928 20:50:09.102244       1 targetloadpacking.go:127] unable to update metrics: unable to get any latest metrics
I0928 20:50:17.017036       1 reflector.go:243] Listing and watching *v1.StorageClass from k8s.io/client-go/informers/factory.go:134
I0928 20:50:17.020044       1 round_trippers.go:444] GET https://172.30.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=99233057 200 OK in 2 milliseconds
I0928 20:50:17.022061       1 round_trippers.go:444] GET https://172.30.0.1:443/apis/storage.k8s.io/v1/storageclasses?allowWatchBookmarks=true&resourceVersion=99233057&timeout=8m10s&timeoutSeconds=490&watch=true 403 Forbidden in 1 milliseconds

[sci110680]

config.yaml.txt
Attaching file of configuration.

[lianghao208]

I think you may need to provide prometheus http address for the trimaran plugin config instead of https. Try to configure in-cluster http service address like http://prometheus-k8s.monitoring.svc.cluster.local:9090.

[Huang-Wei]

@wangchen615 could you give a hand on this?

[sci110680]

still below error after adding p8s svc

svc.cluster.local:9090/api/v1/query": dial tcp: lookup prometheus-k8s.monitoring.svc.cluster.local on 172.30.0.10:53: no such host\n"
time="2021-10-05T05:51:02Z" level=error msg="error querying Prometheus for query avg_over_time(instance:node_cpu:ratio[10m]): Post "http://prometheus-k8s.monitoring.svc.cluster.local:9090/api/v1/query\": dial tcp: lookup prometheus-k8s.monitoring.svc.cluster.local on 172.30.0.10:53: no such host\n"
time="2021-10-05T05:51:02Z" level=error msg="error querying Prometheus for query avg_over_time(instance:node_memory_utilisation:ratio[15m]): Post "http://prometheus-k8s.monitoring.svc.cluster.local:9090/api/v1/query\": dial tcp: lookup prometheus-k8s.monitoring.svc.cluster.local on 172.30.0.10:53: no such host\n"
time="2021-10-05T05:51:02Z" level=error msg="error querying Prometheus for query avg_over_time(instance:node_memory_utilisation:ratio[10m]): Post "http://prometheus-k8s.monitoring.svc.cluster.local:9090/api/v1/query\": dial tcp: lookup prometheus-k8s.monitoring.svc.cluster.local on 172.30.0.10:53: no such host\n"
time="2021-10-05T05:51:02Z" level=error msg="error querying Prometheus for query avg_over_time(instance:node_memory_utilisation:ratio[5m]): Post "http://prometheus-k8s.monitoring.svc.cluster.local:9090/api/v1/query\": dial tcp: lookup prometheus-k8s.monitoring.svc.cluster.local on 172.30.0.10:53: no such host\n"
time="2021-10-05T05:51:02Z" level=error msg="error querying Prometheus for query stddev_over_time(instance:node_cpu:ratio[5m]): Post "http://prometheus-k8s.monitoring.svc.cluster.local:9090/api/v1/query\": dial tcp: lookup prometheus-k8s.monitoring.svc.cluster.local on 172.30.0.10:53: no such host\n"
time="2021-10-05T05:51:02Z" level=error msg="error querying Prometheus for query stddev_over_time(instance:node_cpu:ratio[10m]): Post "http://prometheus-k8s.monitoring.svc.cluster.local:9090/api/v1/query\": dial tcp: lookup prometheus-k8s.monitoring.svc.cluster.local on 172.30.0.10:53: no such host\n"
time="2021-10-05T05:51:02Z" level=error msg="error querying Prometheus for query stddev_over_time(instance:node_cpu:ratio[15m]): Post "http://prometheus-k8s.monitoring.svc

[wangchen615]

@sci110680 , it is a known issue since OpenShift 4.7+. If you switch back to OpenShift 4.6 or 4.5, it works.

The workaround solution for this has been detailed here: https://cloud.redhat.com/blog/thanos-querier-versus-thanos-querier. We are going to test this solution as well and update the documentation.

Thanks so much for posting this issue. Let's keep it open until we update the doc.

Hi, I am trying to use prometheus as a metrics provider for trimaran in openshift 4.7 cluster. Getting below error in pod log. Also i took secret from below command .

oc describe secret $(oc get secret -n openshift-monitoring|awk '{print $1}'|grep prometheus-k8s-token -m 1) -n openshift-monitoring|grep "token:"|cut -d: -f2|sed 's/^ *//g'

Error is :

ilocpsnd496.ocpd.corp.amdocs.com/api/v1/query\": x509: certificate signed by unknown authority\n"
time="2021-09-28T20:49:47Z" level=error msg="error querying Prometheus for query avg_over_time(instance:node_memory_utilisation:ratio[15m]): Post \"https://prometheus-k8s-openshift-monitoring.apps.ilocpsnd496.ocpd.corp.amdocs.com/api/v1/query\": x509: certificate signed by unknown authority\n"
time="2021-09-28T20:49:47Z" level=error msg="error querying Prometheus for query stddev_over_time(instance:node_cpu:ratio[15m]): Post \"https://prometheus-k8s-openshift-monitoring.apps.ilocpsnd496.ocpd.corp.amdocs.com/api/v1/query\": x509: certificate signed by unknown authority\n"
time="2021-09-28T20:49:47Z" level=error msg="error querying Prometheus for query stddev_over_time(instance:node_memory_utilisation:ratio[15m]): Post \"https://prometheus-k8s-openshift-monitoring.apps.ilocpsnd496.ocpd.corp.amdocs.com/api/v1/query\": x509: certificate signed by unknown authority\n"
time="2021-09-28T20:49:47Z" level=error msg="received error while fetching metrics: Post \"https://prometheus-k8s-openshift-monitoring.apps.ilocpsnd496.ocpd.corp.amdocs.com/api/v1/query\": x509: certificate signed by unknown authority"
I0928 20:49:59.764214       1 eventhandlers.go:283] delete event for scheduled pod openshift-marketplace/community-operators-jfhbr
W0928 20:50:09.102244       1 targetloadpacking.go:127] unable to update metrics: unable to get any latest metrics
I0928 20:50:17.017036       1 reflector.go:243] Listing and watching *v1.StorageClass from k8s.io/client-go/informers/factory.go:134
I0928 20:50:17.020044       1 round_trippers.go:444] GET https://172.30.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=99233057 200 OK in 2 milliseconds
I0928 20:50:17.022061       1 round_trippers.go:444] GET https://172.30.0.1:443/apis/storage.k8s.io/v1/storageclasses?allowWatchBookmarks=true&resourceVersion=99233057&timeout=8m10s&timeoutSeconds=490&watch=true 403 Forbidden in 1 milliseconds

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.