Snapshotter sidecar spams CreateSnapshot() forever

[bswartz]

I created a snapshot, and it succeeded as expected within a few seconds. However the sidecar continues to call CreateSnaphot() over and over every 60 seconds, despite my SP returning success every time. If I create multiple snapshots, each one causes an RPC call every 60 seconds to CreateSnapshot() as long as the snapshot exists. This will turn into a scalability problem very quickly with any reasonable number of snapshots.

Here is a log fragment illustrating the problem:
external-snapshotter.log

[xing-yang]

Is this happening with latest code in master?

[xing-yang]

/assign @xing-yang

[xing-yang]

@bswartz Does your driver support idempotency? If the controller calls CSI driver function CreateSnapshot periodically with the same input parameters, does it return information for the same snapshot? The sidecar controller will call CreateSnapshot to get status of a dynamically created snapshot. If the status is lost, we'll need to rebuild status based the CreateSnapshot call for dynamic provisioning.

[bswartz]

Yes the driver is idempotent. If you look at the log you can see exactly what gets returned from CreateSnapshot() each time, and you can see it doesn't change.

The image I'm using is: quay.io/k8scsi/csi-snapshotter:canary
I can try with any other version (or build my own) if that would help.

[xing-yang]

So it won't create multiple physical snapshots on your storage system for the same snapshot creation request, right? It just keeps calling CreateSnapshot to check status.

I think this came in after the splitting controller change. Before the split, snapshot controller will stop calling CreateSnapshot to check status when readyToUse is true. With the split, the common snapshot controller keeps the original behavior (it will stop when readyToUse is true), but the sidecar snapshotter lost that check. I think we should add a similar check in the sidecar.

[bswartz]

Yes I suspect the same. Polling would make sense if readyToUse was false, but once it's true, the sidecar must stop polling.

[xing-yang]

Can you give this a try?
#224

[bswartz]

That change does seem to fix this bug. There's another related bug that's less annoying but still problematic for me. I'll create another issue for that.