Deployment by README steps fails in 4-node cluster

[okartau]

Deployment as shown in README will succeed for sure only if attacher, provisioner, hostpathplugin and my-csi-app run in the same node. When I try the steps in 4-node cluster, the pods state will be:

NAME                         READY   STATUS              RESTARTS   AGE     IP            NODE   
csi-hostpath-attacher-0      1/1     Running             0          3m56s   10.244.1.59   host-1
csi-hostpath-provisioner-0   1/1     Running             0          3m56s   10.244.2.11   host-2 
csi-hostpathplugin-8knjt     2/2     Running             0          3m56s   192.168.8.4   host-1
csi-hostpathplugin-bw855     2/2     Running             0          3m56s   192.168.8.8   host-3
csi-hostpathplugin-dv8x7     2/2     Running             0          3m56s   192.168.8.6   host-2
my-csi-app                   0/1     ContainerCreating   0          3m3s    <none>        host-3

my-csi-app does not reach Running state as it can't mount:

mount: mounting /tmp/94131d40-35c8-11e9-946e-deadbeef0102 on
/var/lib/kubelet/pods/e884bdff-367e-11e9-8553-deadbeef0100/volumes/kubernetes.io~csi/pvc-8d479cfd-35c8-11e9-8553-deadbeef0100/mount
failed: No such file or directory

[okartau]

@pohly proposed these changes to get example deployment working:

• start one instance of csi-hostpathplugin
• use inter-pod affinity to land attacher, provisioner, hostpathplugin and my-csi-app on same node

If we agree on this, I will prepare a PR

[blaggacao]

I observe the same error:

Events:
  Type     Reason                  Age                From                     Message
  ----     ------                  ----               ----                     -------
  Warning  FailedScheduling        17s (x4 over 17s)  default-scheduler        pod has unbound immediate PersistentVolumeClaims (repeated 3 times)
  Normal   Scheduled               17s                default-scheduler        Successfully assigned gitops/atlantis-0 to 10.0.1.78
  Normal   SuccessfulAttachVolume  17s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-7b2a9f04-36ae-11e9-8852-52540082fa9b"
  Warning  FailedMount             1s (x2 over 1s)    kubelet, 10.0.1.78       MountVolume.SetUp failed for volume "pvc-7b2a9f04-36ae-11e9-8852-52540082fa9b" : rpc error: code = Unknown desc = mount failed: exit status 255
Mounting command: mount
Mounting arguments: -o bind /tmp/7b4198ab-36ae-11e9-8baf-5254003ed451 /opt/rke/var/lib/kubelet/pods/7b2de1a5-36ae-11e9-8852-52540082fa9b/volumes/kubernetes.io~csi/pvc-7b2a9f04-36ae-11e9-8852-52540082fa9b/mount
Output: mount: mounting /tmp/7b4198ab-36ae-11e9-8baf-5254003ed451 on /opt/rke/var/lib/kubelet/pods/7b2de1a5-36ae-11e9-8852-52540082fa9b/volumes/kubernetes.io~csi/pvc-7b2a9f04-36ae-11e9-8852-52540082fa9b/mount failed: No such file or directory

On this host 10.0.1.78, however, see:

[rancher@rancher ~]$ sudo ls -la /opt/rke/var/lib/kubelet/pods/7b2de1a5-36ae-11e9-8852-52540082fa9b/volumes/kubernetes.io~csi/pvc-7b2a9f04-36ae-11e9-8852-52540082fa9b
total 12
drwxr-x---    2 root     root          4096 Feb 22 14:47 .
drwxr-x---    3 root     root          4096 Feb 22 14:47 ..
-rw-r--r--    1 root     root           248 Feb 22 14:48 vol_data.json

Therefore, only the .../mount folder is missing. It's seems this comes from the request
however this path is at least not created by this csi-driver-driver-hostpath

[blaggacao]

In other words: How would a proper multi node example look like?

[pohly]

Hostpath by design can only work on a single node because it allocates volumes locally an a node. The solution I mentioned to @okartau ensures that a single hostpath driver runs and that everything else (provisioner, attacher, and the app) run on that same node.

This is not a good example because it forces a special deployment of the app. The advantage is that it works without depending on alpha features.

A better example would be to run provisioner, attacher and driver on one node and then force the app onto that node via topology. But that depends on alpha features in Kubernetes and is more complicated, therefore I think we should do the easy solution first and add topology support later.

[blaggacao]

@pohly Thanks for you comment.

I understand that the volume is bound to a host, and can only reasonably be consumed by apps deployed as a StatefulSet. However, I do not yet understand, why the provisioner and attacher cannot run successfully on multiple hosts, for example to cater a default StorageClass with provider: csi-hostpath.

Is that a limitation of the CSI spec? - I guess I'm still missing some bits, here.

[pohly]

Provisioner and attacher can only talk to a CSI driver that runs on the same host. That's because communication happens via a local Unix domain socket.

The usual model for other drivers that talk to some external storage backend is to run the CSI driver on all nodes and the attacher and provisioner on a few nodes (replicated, with leader election to determine which instance actually responds to requests).

Or at least that is the desirable model. In practice some deployments only start provisioner and attacher once, which causes delays during provisioning/attaching when those pods need to be restarted.

[blaggacao]

Thank you that clarifies a lot. In theory, host-path plugin - based on the request metadata - could determine on which node operations of provision / attach should ultimately take place.

That's unless, conceptually, it would be responsibility of a custom provisioner / attacher, which is actually what their substantivated names would indicate in my understanding.

Assuming I understood things the way they are: is supporting the mentioned* use case considerable for this project or completely out of scope?

* Create multi node cluster default StorageClass with provider: csi-hostpath to cater to PVCs

[pohly]

David Arnold <notifications@github.com> writes:

Thank you that clarifies a lot. In theory, host-path plugin, based on the request metadata could determine which node should ultimately provision / attach. That's unless, conceptually, it would be responsibility of a custom provisioner / attacher, which is precisely what their substantivated names would indicate.

Conceptually, provisioning a volume is independent of its usage on any particular node. The CSI spec is a bit biased towards volumes that are provided by a storage backend which then can be access from any node. The only situation where meta data has any information about the intended node is when: - WaitForFirstConsumer is used - topology support is enabled in the CSI driver Then the first entry in the topology information in CreateVolumeRequest is referring to the node on which the first consumer is about to run. See kubernetes-csi/external-provisioner#221

Assuming I understood things, the way they are, is supporting the mentioned* a use case considerable for this project or completely out of scope? \* Create c luster _default_ `StorageClass` with `provider: csi-hostpath`

csi-hostpath is an experimental driver whose main purpose is for testing the other Kubernetes-CSI components. I don't think it should be the default storage class for a real cluster because it is obviously much too limited.

[cwdsuzhou]

I think, this is mainly caused by controllerserver which connets to external-provisoner by local socket and only support create volume locally in csi-driver-host-path. However, nodeserver runs as a daemonset which may not run on the same node as external-provisoner.

[pohly]

@cwdsuzhou daemonset ensures that there is a hostpath driver instance on all nodes, so external-provisioner always has a local driver to talk to. The problem is that the app then also needs to run on that same node, which currently isn't enforced.

@okartau has a fix in #14

[cwdsuzhou]

@cwdsuzhou daemonset ensures that there is a hostpath driver instance on all nodes, so external-provisioner always has a local driver to talk to. The problem is that the app then also needs to run on that same node, which currently isn't enforced.

@okartau has a fix in #14

Thank you! I know making sure the apps run the same node can fix it. What I want to describe is that the nodeserver is actually working for the apps should run on the same node with the external-provsioner.

[pohly]

@cwdsuzhou what do you mean with "nodeserver is actually working for the apps"?

[cwdsuzhou]

@cwdsuzhou what do you mean with "nodeserver is actually working for the apps"?
It means the apps should run the same node with the external-provisoner and attacher