Consider discontinuing the kubernetes-cni package

[xmudrii]

We should consider:

• removing dependency on the kubernetes-cni package from other packages
• completely discontinuing the kubernetes-cni package after at least 2-3 release cycles

Having containernetworking/plugins installed used to be a requirement in the past, however, with CNIs plugins becoming wide spread, that's not the case any longer. That said, most of users do not need containernetworking/plugins installed, hence we shouldn't have a dependency on that package.

After verifying that the dependency is not needed, we should stop building the package. We have a policy against repacking and hosting 3rd party tools which is recapped here: kubernetes/k8s.io#7708

/sig release
/area release-eng
/priority important-longterm

[afbjorklund]

This also applies to cri-tools, that is no longer required (since 1.32)

• stop shelling out to crictl from kubeadm kubeadm#3064

https://github.com/kubernetes-sigs/cri-tools

@xmudrii : Could be a separate issue though?

[afbjorklund]

That said, most of users do not need containernetworking/plugins installed, hence we shouldn't have a dependency on that package.

It is still needed, but it is the responsibility of the container runtime

And the project doesn't re-package those either (they are third party)

https://kubernetes.io/docs/setup/production-environment/container-runtimes/

https://github.com/containerd/containerd/blob/v2.0.0/docs/containerd-2.0.md#cri-containerd-cni--version-os-archtargz-release-bundles-have-been-removed

---

EDIT: We are using the flannel CNI, it might not be needed for other CNI
I think it is the /opt/cni/bin/portmap that is required (and br_netfilter)

image: ghcr.io/flannel-io/flannel-cni-plugin:v1.8.0-flannel1

https://github.com/lima-vm/lima/blob/master/templates/k8s.yaml

containerd:
  system: true
  user: false

    # cri-tools
    apt-get install -y cri-tools

    # cni-plugins
    apt-get install -y kubernetes-cni

[BenTheElder]

It's not necessarily needed even with the container runtime, there must be some CNI config and some binaries implementing that config if you don't only use hostNetwork, and those are commonly installed via a hostNetwork daemonset, which may use none of the plugins.

In the earlier days something like kubenet / dockershim may have needed these out of the box and had some default configuration.

cc @kubernetes/sig-network-leads

[afbjorklund]

Is there a good way to figure out what versions to use, preferably matching what Kubernetes was tested with?

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

Before this change: (the version is provided in the repository, for each kubernetes version)

sudo apt-get install -y kubernetes-cni

After this change: (the version is provided in the documentation, and not always updated)

CNI_PLUGINS_VERSION="v1.3.0"
case $(uname -m) in
      x86_64)   ARCH="amd64";;
      aarch64)  ARCH="arm64";;
esac
DEST="/opt/cni/bin"
sudo mkdir -p "$DEST"
curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_PLUGINS_VERSION}/cni-plugins-linux-${ARCH}-${CNI_PLUGINS_VERSION}.tgz" | sudo tar -C "$DEST" -xz

And the same for cri-tools, the major version should match k8s but the minor could be trickier to guess...

PS. This problem is not new, we had it also with the packages: (1.6.0 versus 1.7.1) #4100 (comment)

[BenTheElder]

Is there a good way to figure out what versions to use, preferably matching what Kubernetes was tested with?

We intentionally do not advertise what versions the jobs happen to test with because we don't need infighting over testing every CNI solution X N versions, every CRI version, etc ... advertising the versions we tests invites pushing for your preferred version instead of us just using something stable that allows us to test.

There wouldn't be one singular version anyhow, and the binaries Kubernetes releases do not use CNI.

CNI is versioned and your CNI config specifies the version, so you don't really need tight coupling to what the container runtime was tested with either.

And the same for cri-tools, the major version should match k8s but the minor could be trickier to guess...

https://github.com/kubernetes-sigs/cri-tools#compatibility-matrix-cri-tools--kubernetes

[afbjorklund]

It just felt like the installation instructions would grow even longer without the packages, but maybe it won't be so bad.

In my experience it is not tested at all, which was one of the reasons that I wanted to automate the README and get.k8s.io

[afbjorklund]

And the same for cri-tools, the major version should match k8s but the minor could be trickier to guess...

https://github.com/kubernetes-sigs/cri-tools#compatibility-matrix-cri-tools--kubernetes

The matrix just says to use "master", but then says to match version:

Kubernetes Version	cri-tools Version	cri-tools branch
≥ 1.27.x	≥ 1.27.x	master

It's recommended to use the same cri-tools and Kubernetes minor version, because new features added to the Container Runtime Interface (CRI) may not be fully supported if they diverge.

The documentation for Debugging Kubernetes nodes with crictl says something similar:

https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md

Download the version that corresponds to your version of Kubernetes.

And containerd just says to get it from the release (so I guess the above):

https://github.com/containerd/containerd/blob/main/docs/cri/crictl.md

If you are a user, your deployment should have installed crictl for you. If not, get it from your release tarball.

Making sure to configure it, to avoid running into the old "fallback" issue...

• Suggestion: Remove the fallback endpoints in a versioned release of crictl kubernetes-sigs/cri-tools#893

runtime-endpoint: unix:///run/containerd/containerd.sock

Adopted from: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-runtime

---

So I guess it will end something like:

KUBERNETES_VERSION=$(curl -L -s https://dl.k8s.io/release/stable.txt | cut -d'.' -f1-2)
CRICTL_VERSION=$(curl -fsSL https://api.github.com/repos/kubernetes-sigs/cri-tools/releases/latest | jq -r .tag_name)

For the CNI plugins, can use nerdctl-full:

/opt/cni -> /usr/local/libexec/cni
and either "bridge"* or "flannel" plugins

Same as in the containerd documentation:

https://github.com/containerd/containerd/blob/main/docs/getting-started.md

* the old single-node setup, like https://github.com/containerd/containerd/blob/main/script/setup/install-cni

{
  "cniVersion": "1.0.0",
  "name": "containerd-net",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni0",
      "isGateway": true,
      "ipMasq": true,
      "promiscMode": true,
      "ipam": {
        "type": "host-local",
        "ranges": [
          [{
            "subnet": "10.88.0.0/16"
          }],
          [{
            "subnet": "2001:4860:4860::/64"
          }]
        ],
        "routes": [
          { "dst": "0.0.0.0/0" },
          { "dst": "::/0" }
        ]
      }
    },
    {
      "type": "portmap",
      "capabilities": {"portMappings": true}
    }
  ]
}

Adopted from: https://kubernetes.io/docs/concepts/cluster-administration/addons/#networking-and-network-policy

---

And yes, I still think that it is more complicated with CRI and CNI than what it used to be (with Docker, that is). 😀

But it is more configurable this way, and goes with the rest of the removal of the "in-tree" ways of doing things...