aws-sdk-go-v2 route53 failing IMDS region discovery log messages #16837
Description
/kind bug
1. What kops version are you running? The command kops version, will display
this information.
Client version: 1.30.1 (git-v1.30.1)
2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
Server Version: v1.29.7
3. What cloud provider are you using?
AWS
4. What commands did you run? What is the simplest way to reproduce this issue?
When upgrading from kOps v1.29.2 -> v1.30.1 we are receiving some DEBUG messages from the aws-sdk when running command like kops update cluster from a laptop (outside of AWS).
5. What happened after the commands executed?
SDK 2024/09/16 08:54:03 DEBUG retrying request /, attempt 2
SDK 2024/09/16 08:54:04 DEBUG retrying request /, attempt 3
SDK 2024/09/16 08:54:05 DEBUG retrying request /, attempt 4
SDK 2024/09/16 08:54:06 DEBUG retrying request /, attempt 5
SDK 2024/09/16 08:54:06 DEBUG max retry attempts exhausted, max 5
SDK 2024/09/16 08:54:23 DEBUG retrying request /, attempt 2
SDK 2024/09/16 08:54:25 DEBUG retrying request /, attempt 3
SDK 2024/09/16 08:54:26 DEBUG retrying request /, attempt 4
SDK 2024/09/16 08:54:27 DEBUG retrying request /, attempt 5
SDK 2024/09/16 08:54:27 DEBUG request failed with unretryable error canceled, context deadline exceeded
SDK 2024/09/16 08:54:27 DEBUG request failed with unretryable error canceled, context deadline exceeded
I0916 08:55:26.635859 85354 executor.go:113] Tasks: 0 done / 211 total; 64 can run
I0916 08:55:29.085395 85354 executor.go:113] Tasks: 64 done / 211 total; 32 can run
I0916 08:55:29.706467 85354 executor.go:113] Tasks: 96 done / 211 total; 47 can run
I0916 08:55:30.975327 85354 executor.go:113] Tasks: 143 done / 211 total; 15 can run
I0916 08:55:33.756963 85354 executor.go:113] Tasks: 158 done / 211 total; 23 can run
I0916 08:55:35.424744 85354 executor.go:113] Tasks: 181 done / 211 total; 10 can run
I0916 08:55:36.377389 85354 executor.go:113] Tasks: 191 done / 211 total; 10 can run
I0916 08:55:36.549778 85354 executor.go:113] Tasks: 201 done / 211 total; 10 can run
I0916 08:55:36.704646 85354 executor.go:113] Tasks: 211 done / 211 total; 0 can run
SDK 2024/09/16 08:55:37 DEBUG retrying request /, attempt 2
SDK 2024/09/16 08:55:38 DEBUG retrying request /, attempt 3
SDK 2024/09/16 08:55:39 DEBUG retrying request /, attempt 4
SDK 2024/09/16 08:55:40 DEBUG retrying request /, attempt 5
SDK 2024/09/16 08:55:40 DEBUG max retry attempts exhausted, max 5
I0916 08:55:45.297521 85354 dns.go:235] Pre-creating DNS records
...
6. What did you expect to happen?
No failed requests, however it does not seem to be causing an error or preventing us from upgrading.
7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.
8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.
I0916 08:27:37.810079 77227 populate_cluster_spec.go:389] Defaulted PodCIDR to 100.96.0.0/11
I0916 08:27:37.810097 77227 populate_cluster_spec.go:398] Defaulted ServiceClusterIPRange to 100.64.0.0/13
I0916 08:27:37.810120 77227 subnets.go:51] All subnets have CIDRs; skipping assignment logic
I0916 08:27:37.810169 77227 defaults.go:262] Not setting up Proxy Excludes
I0916 08:27:37.810196 77227 clientset.go:114] Using keystore path: "s3://example-k8s-kops/example.k8s.kx.gd/pki"
I0916 08:27:37.810962 77227 middleware.go:44] AWS request: ?
SDK 2024/09/16 08:27:39 DEBUG retrying request /, attempt 2
I0916 08:27:39.064319 77227 middleware.go:44] AWS request: ?
SDK 2024/09/16 08:27:40 DEBUG retrying request /, attempt 3
I0916 08:27:40.068135 77227 middleware.go:44] AWS request: ?
SDK 2024/09/16 08:27:41 DEBUG retrying request /, attempt 4
I0916 08:27:41.071150 77227 middleware.go:44] AWS request: ?
SDK 2024/09/16 08:27:42 DEBUG retrying request /, attempt 5
I0916 08:27:42.073258 77227 middleware.go:44] AWS request: ?
SDK 2024/09/16 08:27:42 DEBUG max retry attempts exhausted, max 5
I0916 08:27:42.074464 77227 middleware.go:44] AWS request: ?
I0916 08:27:42.811986 77227 route53.go:69] Unable to discover region by IMDS, using SDK defaults: operation error ec2imds: GetRegion, request canceled, context deadline exceeded
9. Anything else do we need to know?
This looks related to the aws-sdk-go-v2 migration in #16424, and it seems like the fallback in #16778 is working as expected but it would be nice to suppress these messages.