node processor's global HTTP request timeout frequently exceeded in large clusters #387
Description
Is this a BUG REPORT or FEATURE REQUEST?
BUG REPORT
Versions
CCM Version: latest
What happened?
In large clusters (250+ nodes), the global HTTP request timeout of 10 seconds in the CCM's node processor is frequently reaching the deadline before it can iterate though all the VNICs in the compartment.
What you expected to happen?
The default timeout should be based on the number of nodes, or it should be configurable - at least large enough to not cause erroneous context deadline exceeded errors (resulting in other problems like failing to process nodes).
How to reproduce it (as minimally and precisely as possible)?
Have a large cluster, try and use the OCI CCM.
Anything else we need to know?
We first bumped the timeout to 1 minute, but that was not enough. We ultimately had to bump the default timeout to 5 minutes to resolve the issue.
Error log
oci-cloud-controller-manager-2gh4z oci-cloud-controller-manager 2022-05-24T18:58:47.397Z ERROR oci/node_info_controller.go:227 Failed to map provider ID to instance ID {"component": "cloud-controller-manager", "node": "ashburn-ops-node-rtb-prd-ad2-public-vnic-180924", "error": "GetInstanceByNodeName: context deadline exceeded", "errorVerbose": "context deadline exceeded\ngithub.com/oracle/oci-cloud-controller-manager/pkg/oci/client.(*client).GetVNIC\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/oci/client/networking.go:54\ngithub.com/oracle/oci-cloud-controller-manager/pkg/oci/client.(*client).GetInstanceByNodeName\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/oci/client/compute.go:190\ngithub.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci.(*CloudProvider).InstanceID\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci/instances.go:171\ngithub.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci.getInstanceByNode\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci/node_info_controller.go:225\ngithub.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci.(*NodeInfoController).processItem\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci/node_info_controller.go:162\ngithub.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci.(*NodeInfoController).processNextItem\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci/node_info_controller.go:139\ngithub.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci.(*NodeInfoController).runWorker\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci/node_info_controller.go:124\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90\ngithub.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci.(*NodeInfoController).Run\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci/node_info_controller.go:119\nruntime.goexit\n\t/tools/go/src/runtime/asm_amd64.s:1374\nGetInstanceByNodeName\ngithub.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci.(*CloudProvider).InstanceID\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci/instances.go:176\ngithub.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci.getInstanceByNode\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci/node_info_controller.go:225\ngithub.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci.(*NodeInfoController).processItem\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci/node_info_controller.go:162\ngithub.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci.(*NodeInfoController).processNextItem\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci/node_info_controller.go:139\ngithub.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci.(*NodeInfoController).runWorker\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci/node_info_controller.go:124\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90\ngithub.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci.(*NodeInfoController).Run\n\t/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/cloudprovider/providers/oci/node_info_controller.go:119\nruntime.goexit\n\t/tools/go/src/runtime/asm_amd64.s:1374"}