iso: Improve build and clean up configuration

[nirs]

Previously making the iso-menuconfig-{aarch64,x86_64} or linux-menuconfig-{aarch64,x86_64} targets you had to build the entire iso, since the out/buildroot tree was created by the minikube-iso-{aarch64,x86_64} target.

This change fix the Makefile so you can run the iso-menuconfig-{aarch64,x86_64} or linux-menuconfig-{aarch64,x86_64} directly after checking the source. This makes it much easier to work on the iso configuration.

Previously the linux-menuconfig-{aarch64,x86_64} targets were not using the buildroot facilities, copying files manually using hard coded paths and kernel version. Fixing the target to use the buildroot way removed the unneeded KERNEL_VERSION variable from the Makefile, that made updating kernel version hard.

Running the iso-menuconfig-{aarch64,x86_64} and linux-menuconfig-{aarch64,x86_64} without any change revealed that we had manual changes in the configuration that were not created by config. I run all the targets to update the configuration to the output of kconfig, so future changes do not include unrelated changes.

For aarch64 the image is much larger (410 MiB vs 362 MiB). Turns out that removing unnecessary support for various Arm boards reduces the image to 392 MiB.

Status

• Building iso locally
• Testing with vfkit and qemu on arm64 using ramen rook testing environment https://github.com/RamenDR/ramen/blob/main/test/envs/rook.yaml
• Testing with kvm2 driver on amd64 using ramen regional dr environment https://github.com/RamenDR/ramen/blob/main/test/envs/regional-dr.yaml

[k8s-ci-robot]

Hi @nirs. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[nirs]

ok-to-build-iso

[minikube-bot]

Hi @nirs, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

[nirs]

@afbjorklund can you add ok-to-test? the .PHONY change does not affect the build and fixed now.

[nirs]

I added another commit removing unnecessary support for various Arm boards. This should not be needed since we run on in a vm.

[nirs]

ok-to-build-iso

[minikube-bot]

Hi @nirs, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

[ComradeProgrammer]

/ok-to-test

[nirs]

I checked KVM errors:

• KVM_Linux_containerd
  • volcano (known issue)
  • toomanyrequests: You have reached your unauthenticated pull rate limit (93 instances)
• KVM_Linux_crio
  • toomanyrequests: You have reached your unauthenticated pull rate limit (98 instances)

[medyagh]

/ok-to-test

[medyagh]

@nirs thank you for making it faster to iterate on the ISO build, do you mind sharing an example iteration of ISO build using this new target ? do you manually call make prepare-... ? or do you build the iso the way it was before and let it do the work underneath ?

if this new make target could help other developers, we could add it to the minikube website

[nirs]

@nirs thank you for making it faster to iterate on the ISO build, do you mind sharing an example iteration of ISO build using this new target ?

Without this change this will fail:

git clone https://github.com/kubernetes/minikube.git
cd minikube
make iso-menuconfig-{arch}

Because the out/buildroot is created by make minikube-iso-{arch}, which an take 2 hours to build.

Same for make linux-menuconfig-{arch}.

With this change we can do this to update kernel version:

git clone https://github.com/kubernetes/minikube.git
cd minikube

Update kernel version and headers version via kconfig and exit:

make iso-menuconfig-{arch}

Run to update linux config for the new kernel:

make linux-menuconfig-{arch}

do you manually call make prepare-... ?

No

or do you build the iso the way it was before and let it do the work underneath ?

Exactly, it is it internal target, this is also why it is not documented.

if this new make target could help other developers, we could add it to the minikube website

It is better not to complicate the docs, this target is relevant only if you wan to modify the makefile.

I think that the current docs need update for multiple architectures. Maybe they worked when minikube had only single iso.

• https://minikube.sigs.k8s.io/docs/contrib/building/iso/#modifying-buildroot-components
• https://minikube.sigs.k8s.io/docs/contrib/building/iso/#adding-kernel-modules

I will post another PR to describing updating kernel flow.

[minikube-bot]

Hi @nirs, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

[minikube-pr-bot]

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 20991) |
+----------------+----------+---------------------+
| minikube start | 52.5s    | 52.4s               |
| enable ingress | 16.4s    | 15.6s               |
+----------------+----------+---------------------+

Times for minikube (PR 20991) start: 53.4s 52.9s 52.1s 53.5s 50.0s
Times for minikube start: 53.6s 51.1s 53.7s 51.5s 52.8s

Times for minikube ingress: 15.1s 14.6s 15.1s 18.7s 18.6s
Times for minikube (PR 20991) ingress: 16.1s 15.1s 15.6s 16.1s 15.1s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 20991) |
+----------------+----------+---------------------+
| minikube start | 23.8s    | 25.0s               |
| enable ingress | 12.8s    | 13.0s               |
+----------------+----------+---------------------+

Times for minikube start: 23.5s 23.6s 24.4s 24.2s 23.5s
Times for minikube (PR 20991) start: 27.0s 23.5s 23.3s 23.6s 27.6s

Times for minikube ingress: 12.8s 12.8s 12.8s 13.3s 12.3s
Times for minikube (PR 20991) ingress: 12.3s 13.3s 13.9s 12.4s 13.3s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 20991) |
+----------------+----------+---------------------+
| minikube start | 23.0s    | 23.0s               |
| enable ingress | 26.4s    | 29.3s               |
+----------------+----------+---------------------+

Times for minikube start: 24.4s 22.1s 21.7s 21.6s 25.2s
Times for minikube (PR 20991) start: 22.1s 22.5s 21.6s 22.1s 26.6s

Times for minikube ingress: 22.8s 23.8s 22.8s 39.8s 22.8s
Times for minikube (PR 20991) ingress: 22.8s 22.8s 23.3s 38.8s 38.9s

[medyagh]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh, nirs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [medyagh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment