Cannot install prometheus-operator chart with helmfile due to helm-diff error

[mojochao]

Versions used:

• helmfile v0.100.2
• helm v3.1.1
• databus23/helm-diff plugin v3.1.1

Problem observed:

I can successfully install the stable/prometheus-operator chart with helm directly.

$ helm install prometheus-operator stable/prometheus-operator -n monitoring
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
NAME: prometheus-operator
LAST DEPLOYED: Wed Feb 26 13:13:58 2020
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
NOTES:
The Prometheus Operator has been installed. Check its status by running:
  kubectl --namespace monitoring get pods -l "release=prometheus-operator"

Visit https://github.com/coreos/prometheus-operator for instructions on how
to create & configure Alertmanager and Prometheus instances using the Operator.

I cannot install the same chart with helmfile:

$ helmfile apply
Comparing release=prometheus-operator, chart=stable/prometheus-operator
in ./helmfile.yaml: in .helmfiles[0]: in helmfiles/prometheus-operator.yaml: failed processing release prometheus-operator: helm exited with status 1:
  Error: Failed to render chart: exit status 1: manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "Alertmanager" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "Prometheus" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"]

  Error: plugin "diff" exited with error

I see that the error is coming out of the helm-diff plugin. What command(s) is helmfile using that causes these errors to surface? I will open an issue in the databus23/helm-diff plugin repo once I understand how to repro this with the helm diff command directly.

Many thanks in advance!

[mojochao]

I made an educated guess as to the helm diff command:

$ helm diff --allow-unreleased upgrade -n monitoring prometheus-operator stable/prometheus-operator
********************

	Release was not present in Helm.  Diff will show entire contents as new.

********************
Error: Failed to render chart: exit status 1: manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "Alertmanager" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "Prometheus" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"]

Error: plugin "diff" exited with error

Hopefully that command is in the ballpark of the real command. My guess is that until the helm diff command succeeds, helmfile won't be able to successfully install that chart. :-) I'll report the Issue in their repo.

[mojochao]

Issue being tracked in databus23/helm-diff repo with databus23/helm-diff#189

[andrewnazarov]

What version of prometheus-operator are you trying to install?

[andrewnazarov]

Probably your version is a bit outdated. Helm 3 works differently with CRDs: https://helm.sh/docs/topics/chart_best_practices/custom_resource_definitions/

[mojochao]

I'm using helm v3.1.1, helm-diff plugin v3.1.1 trying to install `stable/prometheus-operator` chart. Does this chart not properly handle CRDs? Is there a different chart I should be using? Thanks for the assistance, Andrew!

…

On Thu, Feb 27, 2020 at 5:19 AM Andrew Nazarov ***@***.***> wrote: Probably your version is a bit outdated. Helm 3 works differently with CRDs: https://helm.sh/docs/topics/chart_best_practices/custom_resource_definitions/ — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1124?email_source=notifications&email_token=AAA2PMPIXVCGMEHUF6VAAATRE6OUVA5CNFSM4K4MVLUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEND7MNI#issuecomment-591918645>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA2PMK5DZYKD7MNSZ6IFALRE6OUVANCNFSM4K4MVLUA> .

[andrewnazarov]

We are successfully deploying version 8.7.0 with Helm 3.

Can you check the exact version of the prometheus-operator chart that is used?

[imtpot]

I'm using hooks to workaround

hooks:
  - events: ["prepare"]
    command: "/bin/sh"
    args: ["-c", "kubectl get crd prometheuses.monitoring.coreos.com >/dev/null 2>&1 || \
            kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml"]
  - events: ["prepare"]
    command: "/bin/sh"
    args: ["-c", "kubectl get crd alertmanagers.monitoring.coreos.com >/dev/null 2>&1 || \
            kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml"]
  - events: ["prepare"]
    command: "/bin/sh"
    args: ["-c", "kubectl get crd prometheusrules.monitoring.coreos.com >/dev/null 2>&1 || \
            kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml"]
  - events: ["prepare"]
    command: "/bin/sh"
    args: ["-c", "kubectl get crd servicemonitors.monitoring.coreos.com >/dev/null 2>&1 || \
            kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml"]
  - events: ["prepare"]
    command: "/bin/sh"
    args: ["-c", "kubectl get crd podmonitors.monitoring.coreos.com >/dev/null 2>&1 || \
        kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml"]
  - events: ["prepare"]
    command: "/bin/sh"
    args: ["-c", "kubectl get crd podmonitors.monitoring.coreos.com >/dev/null 2>&1 || \
        kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml"]  
  set:
  - name: prometheusOperator.createCustomResource
    value: false

[mojochao]

With helmfile (v0.104.1), helm-diff (v3.1.1), and helm (v3.1.2) I'm no longer having this issue.

…

On Tue, Mar 24, 2020 at 11:39 AM agmtr ***@***.***> wrote: I'am use hooks to workaround - events: ["prepare"] command: "/bin/sh" args: ["-c", "kubectl get crd prometheuses.monitoring.coreos.com >/dev/null 2>&1 || \ kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml"] - events: ["prepare"] command: "/bin/sh" args: ["-c", "kubectl get crd alertmanagers.monitoring.coreos.com >/dev/null 2>&1 || \ kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml"] - events: ["prepare"] command: "/bin/sh" args: ["-c", "kubectl get crd prometheusrules.monitoring.coreos.com >/dev/null 2>&1 || \ kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml"] - events: ["prepare"] command: "/bin/sh" args: ["-c", "kubectl get crd servicemonitors.monitoring.coreos.com >/dev/null 2>&1 || \ kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml"] - events: ["prepare"] command: "/bin/sh" args: ["-c", "kubectl get crd podmonitors.monitoring.coreos.com >/dev/null 2>&1 || \ kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml"] - events: ["prepare"] command: "/bin/sh" args: ["-c", "kubectl get crd podmonitors.monitoring.coreos.com >/dev/null 2>&1 || \ kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml"] set: - name: prometheusOperator.createCustomResource value: false``` — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1124 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA2PMIESYDOZ5SLQBIVS6LRJDO2LANCNFSM4K4MVLUA> .

[rchenzheng]

This is still happening as of helmfile (v0.106.3), helm-diff (v3.1.1), and helm (v3.1.2)

What prometheus-operator chart version are you using ?@mojochao

I seem to have the same issue with 8.12.3

[giannisbetas]

helmfile: v0.109.0
helm: v3.1.2
helm-diff: 3.1.1

Chart:
prometheus-operator: 8.12.13

The issue is not solved.
In order to work around it I had to install the CRDs manually

[logston]

Similar. But a different chart.

Bash:

$ helmfile --version
helmfile version v0.109.1
$ helm version
version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-eks-af3caf", GitCommit:"af3caf6136cd355f467083651cc1010a499f59b1", GitTreeState:"clean", BuildDate:"2020-03-27T21:51:36Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}
$ helm plugin list
NAME    VERSION DESCRIPTION                           
diff    3.1.1   Preview helm upgrade changes as a diff

helmfile:

repositories:
- name: dynatrace
  url: https://raw.githubusercontent.com/Dynatrace/helm-charts/master/repos/stable

releases:
  - name: dynatrace-oneagent-operator
    namespace: dynatrace
    chart: dynatrace/dynatrace-oneagent-operator
    version: ~0.7.0

[mumoshu]

Isn't this chart issue, rather than helm or helmfile's?

I mean, are those charts migrated to internally use the new crds directory added in Helm 3?

[Xartos]

I would rather say a helm 3 issue. Since doing helm install --dry-run don't work for charts with crds, see helm/helm#7449
And in the end that's why this fails as well, you can't verify objects that aren't yet defined in the API-server.

In my opinion the best solution would be the one suggested in that issue, but it doesn't seem to be a priority atm. So another solution would be to have a way to disable diff in helmfile when the release aren't present (when installing), like a --skip-diff-if-installing flag when doing apply or something.

[mumoshu]

Thanks. Interesting idea!

Just curious, but do you think how could Helmfile determines if it's an install or upgrade?