Request for strategy on rollout of CRP incase of diff in clusters

[roshair]

Hi Experts,

One of my customers is looking for a solution with fleet, a little different execution style than IfNotDrifted feature.

The customer wants, when Drift is detected on any member cluster, then no apply operation should happen on any of the member cluster.

When drift is not detected in any of the member cluster, then only the apply operation should happen, on all the clusters.

Could you please share some guidance on how we can achieve this using Staged Rollout?

[michaelawyu]

Hi roshair!

Thanks for creating the discussion thread.

If I understand the situation correctly, the use case you'd like to see is something like:

• when drifts occur on a member cluster, do not roll out any change from the hub cluster to the member clusters until the drifts are resolved

with the default rolling update strategy, this might not be possible at the moment. However, if the customer's willing to switch to the staged rollout strategy, they could simply:

• before triggering a staged rollout (i.e., creating a ClusterStagedUpdateRun), check if the CRP status is OK (i.e., if there are any apply failures that are drift related); do not create the ClusterStagedUpdateRun object if drifts are found

  e.g., they could run the command

  kubectl get clusterresourceplacement.v1beta1.placement.kubernetes-fleet.io crp-reportdiff-sample -o jsonpath='{.status.placementStatuses}' \
  | jq '[.[] | select (.diffedPlacements != null)] | map({clusterName, diffedPlacements})'

  If the output is non-empty there would be drifted clusters. Resolve the drifts manually and then re-trigger the rollout.

I totally understand the customer might want something more automatic. I will discuss with the team and see if we could offer some form of gate checks that can verify this status.

[jwtty]

Hi @roshair,

Thank you for being interested in kubefleet and contributing!
Just like @michaelawyu already pointed out, we currenly do not support reportig drifts for all the member clusters and then triggering rollout only after no drifts on ALL clusters automatically. This would require two rounds of rollout at the time:

1. First round of rollout with ApplyStrategy type set to ReportDiff mode: This would not do any write operation but only compare the desired manifests with the actual ones and report any differences observed.
2. Second round of rollout you can do the actually apply operation only after 1st round does not report any diff.

For both rounds of rollout, you may use staged updateRun. However, there's currently a known bug with staged updateRun rollout targeting a CRP with ReportDiff mode that I'm opening a PR for the fix. Before then, you may try running the 1st round of rollout with the default RollingUpdate strategy while 2nd round with actual placement still with Staged UpdateRun. But I assume the bug could be fixed very soon.

[roshair]

@jwtty and @michaelawyu Thanks for your response. I understood the first part, where we run the command

kubectl get clusterresourceplacement.v1beta1.placement.kubernetes-fleet.io crp-reportdiff-sample -o jsonpath='{.status.placementStatuses}'
| jq '[.[] | select (.diffedPlacements != null)] | map({clusterName, diffedPlacements})'

If non empty response, we fix the drift and then do stagedupdaterun.

However, for the second part, where we can run

First round of rollout with ApplyStrategy type set to ReportDiff mode: This would not do any write operation but only compare the desired manifests with the actual ones and report any differences observed.
Second round of rollout you can do the actually apply operation only after 1st round does not report any diff.

Where do i add the applystrategy? In the Strategy.yaml or CRP.yaml?

In crp.yaml, strategy has been set to external. If you could share bit more guidance around it, it will be helpful

[roshair]

I tried to add applystrategy to updatestrategy file under spec with reportdiff

piVersion: placement.kubernetes-fleet.io/v1beta1
kind: ClusterStagedUpdateStrategy
metadata:
name: example-staged-strategy
spec:
applyStrategy: ReportDiff
stages:
- name: staging
labelSelector:
matchLabels:
environment: staging
afterStageTasks:
- type: TimedWait
waitTime: 1h
- name: canary
labelSelector:
matchLabels:
environment: canary
afterStageTasks:
- type: Approval
- name: production
labelSelector:
matchLabels:
environment: production

i get the following error:
PS /home/mod> kubectl apply -f strategy.yaml
Error from server (BadRequest): error when creating "strategy.yaml": ClusterStagedUpdateStrategy in version "v1beta1" cannot be handled as a ClusterStagedUpdateStrategy: strict decoding error: unknown field "spec.applyStrategy"

[jwtty]

Hi @roshair, you need to set the applyStrategy on the CRP spec instead of the clusterStagedUpdateStrategy. It's under spec.Strategy.applyStrategy. You can find a concrete example in our doc here: https://kubefleet.dev/docs/how-tos/reportdiff/.

strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 100%
      unavailablePeriodSeconds: 1
    applyStrategy:
      type: ReportDiff