Race condition in containerd restart when using --container-runtime=containerd and containerd was previously initialized

[krezovic]

I'm running minikube on SLES 12.5 on bare-metal machine, and I'm aware of the warning that containerd with --driver=none is experimental. However, it seemed to work once, but not anymore with 1.17.1. There seems to be a race condition where crictl command is called before containerd is fully started. Following command is ran as root user, hence no sudo (I'm fully aware of the issues that may arise).

KUBERNETES_VERSION=v1.20.2
SERVER_ADDRESS=redacted

minikube start --driver=none --apiserver-ips="${SERVER_ADDRESS}" --apiserver-port=6443 \
               --embed-certs=true --install-addons=false --kubernetes-version="${KUBERNETES_VERSION}" \
               --container-runtime=containerd --cni=calico

Full output of minikube start command used, if not already included:

😄  minikube v1.17.1 on Suse 12.5
✨  Using the none driver based on user configuration
❗  Using the 'containerd' runtime with the 'none' driver is an untested configuration!
👍  Starting control plane node minikube in cluster minikube
🤹  Running on localhost (CPUs=8, Memory=64159MB, Disk=40962MB) ...
ℹ️  OS release is SUSE Linux Enterprise Server 12 SP5
📦  Preparing Kubernetes v1.20.2 on containerd 1.4.3 ...

❌  Exiting due to K8S_INSTALL_FAILED: updating control plane: generating kubeadm cfg: getting cgroup driver: get cri info: sudo crictl info: exit status 1
stdout:

stderr:
time="2021-02-03T10:14:44+01:00" level=fatal msg="getting status of runtime: rpc error: code = Unknown desc = server is not initialized yet"


😿  If the above advice does not help, please let us know: 
👉  https://github.com/kubernetes/minikube/issues/new/choose

If I run sudo crictl info, I get the JSON output the command expects. So there seems to be a race condition between writing a containerd.toml file, restarting containerd and issuing crictl command to get the cgroup driver.

I have found out that this happens when containerd is previously initialized in /var/lib/containerd and has lots of data. If I wipe all containerd data, containerd will start up much faster, and the same command will succeed.

😄  minikube v1.17.1 on Suse 12.5
✨  Using the none driver based on user configuration
❗  Using the 'containerd' runtime with the 'none' driver is an untested configuration!
✅  Using image repository reg-hrpap.app.corpintra.net/infrastructure
👍  Starting control plane node minikube in cluster minikube
🤹  Running on localhost (CPUs=8, Memory=64159MB, Disk=40962MB) ...
ℹ️  OS release is SUSE Linux Enterprise Server 12 SP5
📦  Preparing Kubernetes v1.20.2 on containerd 1.4.3 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring Calico (Container Networking Interface) ...
🤹  Configuring local host environment ...

❗  The 'none' driver is designed for experts who need to integrate with an existing VM
💡  Most users should use the newer 'docker' driver instead, which does not require root!
📘  For more information, see: https://minikube.sigs.k8s.io/docs/reference/drivers/none/

❗  kubectl and minikube configuration will be stored in /root
❗  To use kubectl or minikube commands as your own user, you may need to relocate them. For example, to overwrite your own settings, run:

    ▪ sudo mv /root/.kube /root/.minikube $HOME
    ▪ sudo chown -R $USER $HOME/.kube $HOME/.minikube

💡  This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
🔎  Verifying Kubernetes components...
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

[afbjorklund]

Hmm, docker-containerd (as we can call it, the-containerd-started-by-docker) should store the data in /var/lib/docker

But like you say, it seems there is a race condition. Also, we shouldn't clobber configuration and restart, unless required.

[krezovic]

This is pure containerd, docker is not available on this machine. It's a bare-metal config, --driver=none.

[afbjorklund]

I was just wondering what had populated /var/lib/containerd, but maybe that was just minikube itself.

Hopefully there is some kind of containerd health-check, that can be waited upon after the Restart ?

        // Otherwise, containerd will fail API requests with 'Unimplemented'
        return r.Init.Restart("containerd")

[afbjorklund]

Probably this would be a good idea to have in the "cruntime", for all of the different CRI ?

        cr, err := cruntime.New(co)
        if err != nil {
                exit.Error(reason.InternalRuntime, "Failed runtime", err)
        }

        disableOthers := true
        if driver.BareMetal(cc.Driver) {
                disableOthers = false
        }

        err = cr.Enable(disableOthers, forceSystemd())
        if err != nil {
                exit.Error(reason.RuntimeEnable, "Failed to enable container runtime", err)
        }

        return cr

That is, before returning the container runtime instance - make sure that it is "healthy"

[krezovic]

From what I've seen in journal (gone now, not persisted across reboots), crictl complained something like "cannot open /run/containerd/containerd.sock: no such file or directory". So a possible liveness check is to check for existence of CRI socket?

[krezovic]

I was just wondering what had populated /var/lib/containerd, but maybe that was just minikube itself.

Hopefully there is some kind of containerd health-check, that can be waited upon after the Restart ?

        // Otherwise, containerd will fail API requests with 'Unimplemented'
        return r.Init.Restart("containerd")

Well, /var/lib/containerd is where all the images and overlays are stored, it's equivalent of /var/lib/docker for docker stuff. I had about 50 pods running at the time of upgrade.

[afbjorklund]

Yeah, checking that the unix socket file is actually there is cheap and easy - a good start if nothing else.

We are planning to move away from that pesky tcp port 2376 (docker) and varlink bridge (podman) anyway...
It won't work for the remote daemons (URL), but I guess we can fall back to calling "version" or whatever :-)
But that's just the general case, for the CRI there will be a local UNIX socket present - for all of them.

Just check that the SocketPath() exists, after calling the Enable(). And maybe a little wait/retry loop, on that.