Investigate AWS 5k Node e2e Scale costs

[BenTheElder]

We're spending around 50% of our budget on AWS on the 5k node scale tests, this is disproportionate.

We should investigate if there's knobs we can tune with the cluster configuration to bring this down, or otherwise reduce the interval again.

For a very rough comparison in November 2022 the 5k node project on GCP spend ~$22k, we spent > $142k in the 5k node AWS account for November 2023. These aren't equivalent, but I wouldn't expect this order of magnitude difference which suggests there's room to run the job more efficiently (e.g. perhaps we have public IPs on each worker node?), I know there was some tuning on the kube-up jobs.

/sig k8s-infra scalability
cc @dims

[ameukam]

cc @hakman @hakuna-matatah @upodroid

[ameukam]

AFAIK, we are talking about a single test: ci-kubernetes-e2e-kops-aws-scale-amazonvpc-using-cl2

[ameukam]

The test runs in a single account k8s-infra-e2e-boskos-scale-001.

Breaking down some services used:

Service	November 23
EC2-Instances	118,494.91
EC2-Other	15,299.39
Support (Business)	8,603.50
Elastic Load Balancing	258.91
Key Management Service	13.66
S3	3.16

[ameukam]

So what's EC2-Other ?

So EBS volumes on the EC2 instances and Data Transfer within the region.

[hakuna-matatah]

• One reason was that we were running 2 periodics vs 1 periodic on GCP.

• Other reason is that we kicked off one off tests using pre-submits as well while making code changes and fixing some issues before we saw successful runs.

• We are also running statefulsets in our cl2 tests, haven't looked into if GCP is running stateful sets as part of scale tests.

[BenTheElder]

One reason was that we were running 2 periodics vs 1 periodic on GCP.

I don't think that was true in November '22?
But either way that doesn't account for using ~50% of budget on AWS and a much smaller fraction on GCP.

EDIT: These two jobs run 5k node in k8s-infra-e2e-scale-5k-project
https://testgrid.k8s.io/sig-scalability-gce#gce-master-scale-correctness
https://testgrid.k8s.io/sig-scalability-gce#gce-master-scale-performance

Other reason is that we kicked off one off tests using pre-submits as well while making code changes and fixing some issues before we saw successful runs.

My understanding is that was true in the past, but in November we were not developing anymore?
This was discussed in today's meeting.

We are also running statefulsets in our cl2 tests, haven't looked into if GCP is running stateful sets as part of scale tests.

If this can account for a dramatic cost increase, we should consider not doing it.
I doubt it, since the GCP tests have been extremely long running (like > 10h), so it's mostly down to the size and shape of the cluster under test (or something else comparable like making sure we have reliable cleanup of resources when the test is down)

EDIT: I remember wrong, the GCE jobs take 3-4h.

I suspect we can run smaller worker nodes or something else along those lines?

[BenTheElder]

Aside: The GCP / AWS comparison is only to set a sense of scale by which this seems to be running much more expensively than expected, I don't expect identical costs and we're not attempting to compare platforms ... it will always be apples to oranges between kube-up on GCE and kops on EC2, but the difference is so large that I suspect we're missing something with running these new AWS scale test jobs cost effectively.

The more meaningful datapoint is consuming ~50% of budget on AWS just for this account, which is not expected.

So EBS volumes on the EC2 instances and Data Transfer within the region.

Maybe smaller worker node volumes are in order? Or a different volume type?

It looks like the spend on this account is by far mostly ec2 instance costs though, so probably we should revisit the node sizes / machine types?

[dims]

cc @hakuna-matatah @mengqiy

[hakuna-matatah]

I don't think that was true in November '22?
But either way that doesn't account for using ~50% of budget on AWS and a much smaller fraction on GCP.

I think we have been running 2 tests until 2 days back, this PR 2 days back moved it to one test for every 24 hrs.

My understanding is that was true in the past, but in November we were not developing anymore?
This was discussed in today's meeting.

I remember kicking off one off tests when working on experimenting with Networking SLOs in Novemeber.

Maybe smaller worker node volumes are in order? Or a different volume type?

We can definitely improve cost basis on the instance types we choose, we are having a discussion on slack around this here - https://kubernetes.slack.com/archives/C09QZTRH7/p1701906151681289?thread_ts=1701899708.933279&cid=C09QZTRH7

[hakman]

I think there are some aspects where kOps may be improved for reducing the cost and duration of the 5k tests:

• instance root volume size - the default is 48 GB, which seems way too much for this use case
• node registration - probably client-side throttling when so many nodes try to join at the same time
• collecting the logs and data from the nodes after tests - done sequentially, so we limited it to 500 nodes, but can probably do it in parallel
• resource cleanup - there is a lot of room for optimisation there, due to the many AWS API requests, they get throttled often

[hakuna-matatah]

resource cleanup - there is a lot of room for optimisation there, due to the many AWS API requests, they get throttled often

Do we know how long this step is taking today ? I want to compare it with internal EKS 5k test runs tear-down time. And what throttling are we experiencing in terms of APIs ? We can try to increase the limits if current limits are not enough at the time of deletion process.