[bpf] Connections dropped during felix restart

[sfudeus]

Expected Behavior

As usually observed in the iptables data plane, I'd expect a typha restart (as in a regular typha rolling redeployment) not having significant negative impact on the ebpf data plane.

Current Behavior

Using the ebpf data plane in a bigger cluster (280 nodes, 1900 services, 18k pods) causes hanging traffic after a typha restart. Seemingly, while a calico-node is reconnecting (despite having similar output as with the iptables data plane), the ebpf data plane is reconfigured and connot properly forward traffic until done.
I'm specifically seeing timeouts in our ingress controllers / gateways (traefik, istio) are not able to forward traffic to the workload pods.

In my last production test, I observed effects for up to 20m after the typha redeployment. Old typha pods took up to 5m to terminate (which is our termination grace period, but it seems all connections were handed off before termination). Typha metrics show that all connections were up at ~6m after the start of the shutdown - still I see application impact up to 20m after the restart.

Noteworthy as well: I see significant drop of conntrack table size during the typha reconnects, which I would not expect. Maybe this even is the core of the problem, that connectiontable state is (partially?) dropped?

Steps to Reproduce (for bugs)

1. kubectl rollout restart -n kube-system deployment calico-typha

Your Environment

• Calico version 3.30.1
• Calico dataplane (iptables, windows etc.) ebpf
• Orchestrator version (e.g. kubernetes, mesos, rkt): k8s v1.31.8
• Operating System and version: FlatCar ContainerLinux 4152.2.3

[fasaxc]

@sfudeus When typha restarts, that triggers felix to restart inside the calico-node pod. It may be important to know whether you were doing an upgrade of calico-node at the same time as typha or not. If only typha is restarted then felix will restart within the pod but that shouldn't cause calico-node as a whole to restart. If there's a pending upgrade of calico-node then the rollout of typha can trigger calico-node upgrades as each calico-node goes non-ready due to losing its connection to Typha.

Restarting only felix has much narrower scope to debug so it'd help to be sure that only typha was being restarted and calico-node stayed up.

[sfudeus]

Calico node stayed up. As far as I can deduce from the node logs, there are three independent processes in node which restart at different times based on a typha restart. Status-reporter, tunnel-ip-allocator and felix. After the Felix disconnect logs, with ebpf I see a lot more "noise" than after a Felix restart using iptables mode.

[sfudeus]

So maybe, my main concern is not with the typha restart, but with any felix restart. We observed as well a rather trivial change in felix configuration (flush iptables from true to false) caused a data plane disruption for several minutes.
Jut that changes to the FelixConfiguration are a rather rare event, whereas typha restarts are a common thing in our setup, with regular cluster redeployments under normal traffic.

[fasaxc]

Great, that narrows it down a lot. Felix restarts aren't supposed to cause any disruption, it should resync with dataplane and find that it's already in a good state. Please can you share logs from calico-node covering a felix restart with disruption (Calico Users slack might be the best place to share those directly with me or @tomastigera ).

If you want to restart felix in one pod only without any config change or typha restart, I think you can do that kubectl exec -n ... calico-node-xxxx -- sv hup felix

[sfudeus]

I sent logs from our integration and production systems to @tomastigera, and as well a screenshot of felix_bpf_conntrack_entries_seen metrics of the calico-node which I sent the sighup to. In production, it shows a drop of nat_forward and nat_reverse from 22k to 2k, attached here for demonstration as well.

[fasaxc]

@sfudeus is there a corresponding bump in felix_bpf_conntrack_expired or felix_bpf_conntrack_stale_nat or felix_bpf_conntrack_entries_deleted? If not, I'm wondering if we're nuking your conntrack map on restart (or if there's some problem with persisting the map in the first place).

[fasaxc]

Please can you crank the log level up in your logs. Warning logs will tell you if Calico thinks it hit a problem, but they're not useful for diagnosing an issue like this. We recommend turning on info logs all the time so that we have debugging information if you hit a problem. For this problem, we might need to selectively turn on debug logs with the logDebugFilenameRegex option.