Minikube loses track of nodes after second restart

[luispresuelgit]

When I create multi-node cluster using Minikube, on a second restart Minikube will lose track of the middle nodes e.g. I create 4 nodes: m1, m2, m3, m4; by some reason Minikube loses track of m2 and m3.

Taking this post example, creating a profile named vault-cluster

$ minikube start --nodes 4 -p vault-cluster --no-vtx-check
😄  [vault-cluster] minikube v1.20.0 on Microsoft Windows 10 Pro 10.0.19042 Build 19042
✨  Automatically selected the virtualbox driver
👍  Starting control plane node vault-cluster in cluster vault-cluster
🔥  Creating virtualbox VM (CPUs=2, Memory=2200MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.6 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass

👍  Starting node vault-cluster-m02 in cluster vault-cluster
🤷  virtualbox "vault-cluster-m02" VM is missing, will recreate.
🔥  Creating virtualbox VM (CPUs=2, Memory=2200MB, Disk=20000MB) ...
🌐  Found network options:
    ▪ NO_PROXY=192.168.99.124
    ▪ no_proxy=192.168.99.124
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.6 ...
    ▪ env NO_PROXY=192.168.99.124
E0617 13:40:17.898834    7160 node.go:128] unable to delete node "m02": nodes "vault-cluster-m02" not found
anyway: nodes "vault-cluster-m02" not found
🔎  Verifying Kubernetes components...

👍  Starting node vault-cluster-m03 in cluster vault-cluster
🤷  virtualbox "vault-cluster-m03" VM is missing, will recreate.
🌐  Found network options:
    ▪ NO_PROXY=192.168.99.124,192.168.99.125
    ▪ no_proxy=192.168.99.124,192.168.99.125
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.6 ...
    ▪ env NO_PROXY=192.168.99.124
    ▪ env NO_PROXY=192.168.99.124,192.168.99.125
E0617 13:41:59.753128    7160 node.go:128] unable to delete node "m03": nodes "vault-cluster-m03" not found
E0617 13:41:59.753128    7160 start.go:231] error removing existing worker node before rejoining cluster, will continue anyway: nodes "vault-cluster-m03" not found
🔎  Verifying Kubernetes components...

👍  Starting node vault-cluster-m04 in cluster vault-cluster
🔥  Creating virtualbox VM (CPUs=2, Memory=2200MB, Disk=20000MB) ...
🌐  Found network options:
    ▪ NO_PROXY=192.168.99.124,192.168.99.125,192.168.99.126
    ▪ no_proxy=192.168.99.124,192.168.99.125,192.168.99.126
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.6 ...
    ▪ env NO_PROXY=192.168.99.124
    ▪ env NO_PROXY=192.168.99.124,192.168.99.125
    ▪ env NO_PROXY=192.168.99.124,192.168.99.125,192.168.99.126
🔎  Verifying Kubernetes components...
🏄  Done! kubectl is now configured to use "vault-cluster" cluster and "default" namespace by default

Stopping them for first time:

$ minikube stop -p vault-cluster
✋  Stopping node "vault-cluster"  ...
✋  Stopping node "vault-cluster-m02"  ...
✋  Stopping node "vault-cluster-m03"  ...
✋  Stopping node "vault-cluster-m04"  ...

Restarting it again:

$ minikube start -p vault-cluster
😄  [vault-cluster] minikube v1.20.0 on Microsoft Windows 10 Pro 10.0.19042 Build 19042
✨  Using the virtualbox driver based on existing profile
👍  Starting control plane node vault-cluster in cluster vault-cluster
🔄  Restarting existing virtualbox VM for "vault-cluster" ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.6 ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
👍  Starting node vault-cluster-m02 in cluster vault-cluster
🔄  Restarting existing virtualbox VM for "vault-cluster-m02" ...
🌐  Found network options:
    ▪ NO_PROXY=192.168.99.124
    ▪ no_proxy=192.168.99.124
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.6 ...
    ▪ env NO_PROXY=192.168.99.124
🔎  Verifying Kubernetes components...
👍  Starting node vault-cluster-m03 in cluster vault-cluster
🔄  Restarting existing virtualbox VM for "vault-cluster-m03" ...
🌐  Found network options:
    ▪ NO_PROXY=192.168.99.124,192.168.99.125
    ▪ no_proxy=192.168.99.124,192.168.99.125
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.6 ...
    ▪ env NO_PROXY=192.168.99.124
    ▪ env NO_PROXY=192.168.99.124,192.168.99.125
🔎  Verifying Kubernetes components...
👍  Starting node vault-cluster-m04 in cluster vault-cluster
🔄  Restarting existing virtualbox VM for "vault-cluster-m04" ...
🌐  Found network options:
    ▪ NO_PROXY=192.168.99.124,192.168.99.125,192.168.99.126
    ▪ no_proxy=192.168.99.124,192.168.99.125,192.168.99.126
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.6 ...
    ▪ env NO_PROXY=192.168.99.124
    ▪ env NO_PROXY=192.168.99.124,192.168.99.125
    ▪ env NO_PROXY=192.168.99.124,192.168.99.125,192.168.99.126
🔎  Verifying Kubernetes components...
🏄  Done! kubectl is now configured to use "vault-cluster" cluster and "default" namespace by default

And then stopping it you can see that it only stopped the last node 4 times?

$ minikube stop -p vault-cluster
✋  Stopping node "vault-cluster"  ...
✋  Stopping node "vault-cluster-m04"  ...
✋  Stopping node "vault-cluster-m04"  ...
✋  Stopping node "vault-cluster-m04"  ...
🛑  4 nodes stopped.

And see that it actually lists:

$ minikube node list -p vault-cluster
vault-cluster   192.168.99.124
vault-cluster-m04       192.168.99.127
vault-cluster-m04       192.168.99.127
vault-cluster-m04       192.168.99.127

Environment:

Windows 10 Pro

Virtual Box 6.1

minikube version: v1.20.1 commit: c61663e

kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"windows/amd64"} Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

This also seems to be and issue in Minikube version 1.21.0: https://stackoverflow.com/a/68016356/8592680

[morhook]

I'm experiencing same behaviour on my Ubuntu with minikube v1.21.0, commit: 76d7419

I've found out if before stopping I fix the file $HOME/.minikube/profiles/minikube/config.json with the correct IPs and names it fixes the problem on the shutdown and the restart. For the person that opened this issue it would be fixing some duplicate corrupted Nodes on that file, that would end up like:

        {
            "Name": "m02",
            "IP": "192.168.99.125",
            "Port": 0,
            "KubernetesVersion": "v1.20.2",
            "ControlPlane": false,
            "Worker": true
        },
        {
            "Name": "m03",
            "IP": "192.168.99.126",
            "Port": 0,
            "KubernetesVersion": "v1.20.2",
            "ControlPlane": false,
            "Worker": true
        }

[andriyDev]

/assign

[andriyDev]

Faster reproduction:

1. minikube start -n 3
2. minikube stop
3. minikube start
4. minikube node list

[morhook]

I've tried to analyze what was going on, via this command watch diff -b /tmp/minikube_ok.json profiles/minikube/config.json. Not sure if the "breakup" is on the startup of the 2nd or the 3rd node.

[andriyDev]

It occurs on startup of 2nd node, it seems each time a node is brought up, the config is saved, and in this case saved with only ever the last node info.

I'm using VSCode to do my watching :P

[morhook]

Thanks for fixing this one @andriyDev ! Just tested with v1.22.0-beta.0