Allow usage of KubeletConfiguration next to JoinConfiguration on kubeadm join

[fabriziopandini]

as per @neolit123 comment

we need to start respecting KubeletConfiguration next to JoinConfiguration on kubeadm join.
last time i checked we weren't.

related to this KEP:
kubernetes/enhancements#1439

---

1.25 tracking of kubeadm patch support for kubeletconfiguration:

• KEP update
  sig-cl/kubeadm/1739: update KEP for support of patching kubelet config enhancements#3312
• k/k code PRs:
  kubeadm: add support for patching a "kubeletconfiguration" target kubernetes#110405
  kubeadm: ensure kubelet config patch results are in YAML kubernetes#110598
• k/kubeadm test update
  kinder: make sure worker nodes also get a patches dir created #2706
  kinder: add test for patching kubeletconfiguration; cleanup test jobs #2707
  kubeadm: remove jobs for the "patches" functionality for N-x versions test-infra#26557
• docs update for https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/control-plane-flags/ https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/kubelet-integration/
  kubeadm: document the option to use kubeletconfiguration patches website#34259

[fabriziopandini]

the tricky part here is how to preserve node-specific settings during upgrades...

[rosti]

Allowing for customization on a per-node basis - yes, allowing for a replacement - probably no.
Component configs are shared on a per-cluster basis and are stored in config maps.
Allowing for them to be patched with node-local settings is a must have, but allowing for their total replacement is probably not desired.

[fabriziopandini]

I'm +1 on merging this with the "Advanced configurations with kubeadm (Kustomize)" effort (customization on a per-node basis)

As an alternative, if we prefer to keep the scope of kustomize limited for the first iterations, it should be provided a way to specify the ConfigMap we should read from, and keep track of this with a new node annotation.

[rosti]

My bet is to push in the Kustomize direction. I don't think, that there are many users who push for such kind of feature and, therefore, having the delay (necessary for Kustomize) is acceptable.

[BenTheElder]

Without this you can customize kubelet per node but it requires using flags which is problematic given that many are deprecated.

[fabriziopandini]

Rif kubernetes/enhancements#1439