Pod Stuck in Terminating state if tainted node is power cycled....

[nmathew]

What happened?

If I power cycle the node after tainting the node, certain pods are stuck in "Terminating" state(these pods usually take more time to delete)

kubectl taint nodes control-plane-dc275-2-master-n5-1003 node-role.kubernetes.io/master:NoExecute
poweroff -f --reboot

pod is stuck in Terminating state.

But If I remove the taint immediately after poweroff, eventually pod will get removed.

If I untainted after the node comes up, the pod is stuck in terminating state.

Kubernetes Version: v1.27.1

What did you expect to happen?

Pod should get removed

How can we reproduce it (as minimally and precisely as possible)?

kubectl taint nodes control-plane-dc275-2-master-n5-1003 node-role.kubernetes.io/master:NoExecute
poweroff -f --reboot

Anything else we need to know?

Pods stuck in Terminated state usually take more time to get deleted

Kubernetes version

v1.27.1

Cloud provider

none

OS version

SLES 15 SP4

Install tools

none

Container runtime (CRI) and version (if applicable)

containerd

Related plugins (CNI, CSI, ...) and versions (if applicable)

No response

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nmathew]

Not seen in kubernetes 1.26.1

[nmathew]

/sig node

[nmathew]

@smarterclayton @wojtek-t @bobbypage any pointers?

[harche]

/cc

[SergeyKanzhelev]

/triage needs-information

Can you share the kubelet.log after restart? Maybe also before restart to see what state those Pods were at the moment of power cycle.

What is the time difference between immediately un-tainting or after node come up. What is immediately here?

[zyriljamez]

This issue is reproduced with k8s 1.27.2 as well.

After tainting the node, within 5 seconds the node is power cycled. Some of the pods are stuck in "Terminating" state.

Tried restarting kubelet multiple times, pods are not removed. But as soon as the taint is removed and the kubelet is restarted, the "Terminating" pods are removed.

[mboukhalfa]

Can you share the kubelet.log after restart? Maybe also before restart to see what state those Pods were at the moment of power cycle.

I reproduced this issue please find kubelet logs :

kubelet.txt

[bobbypage]

Can you please provide the kubectl get pod {SOME_POD} -o yaml of one the pods stuck in terminating? It would be good to see the state of it.

I tried to play around with a kind cluster to repro this but was unable to. Can you please provide the more detailed repro instructions?

In my repro attempt, the pod was in terminating until the kubelet restarted. After the kubelet restarted, the pod ends up getting deleted.