bug: new pod after a deployment gets terminated once before running stably

[kartik-579]

Description

When a Deployment is triggered, new pod goes into terminating state and old pod gets running for some time before a new pod is up again and remains stable.

This issue is intermittent but is observed for 70% of the cases.

To Reproduce

1. Start watching pods of an already deployed pipeline.
2. Trigger a new deployment from devtron with some of the configurations changed.
3. Monitor the pods watcher closely. In some cases, new pods starts creating but get terminated once and the old pod(with the old replica) is up instead. It takes some time for this old pod to get terminated permanently and the new replica pod to get up and running stable.

AB#114