Kubernetes cluster creation Error - Kubernetes cluster kubeconfig not available currently in Isolated Network

[nxsbi]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Kubernetes Service

CLOUDSTACK VERSION

4.15.0

CONFIGURATION

Base install of 4.15.0 (upgraded from 4.11)
Kubernetes Service enabled
CoreOS template
community Kubernetes ISO (tried multiple from v 1.11.4 to 1.16.3)
Using Advanced Networking
User account uses Isolated Network (not L2 or Shared)
SSL is enabled for CS GUI, and System VMs

OS / ENVIRONMENT

CentOS 7 for Management Server

SUMMARY

On a freshly upgraded version to Cloudstack 4.15 (from 4.11), when I create Kubernetes Cluster (regardless of which version), the master and worker VMs are getting created and running successfully, but after the Timeout setting (default 3600 seconds) expires I see the state - "Error". Further more, under the "Access" tab, I see "Kubernetes cluster kubeconfig not available currently". I cannot download the config file/never becomes available.

This happens in Isolated Networks with source NAT enabled. I also tested on a Shared Network on a VLAN directly on the router.

It seems the VMs are getting setup but something is getting blocked when trying to check the status of the service. I have opened all ports for egress in the Isolated Network. ( I can see the data load of 200+MB taking place on the master and worker node via CS GUI)

STEPS TO REPRODUCE

EXPECTED RESULTS

Kubernetes Service should show as Active

ACTUAL RESULTS

Kubernetes Service Shows as Error

[nxsbi]

Additional Information:
I tried to get Config file via CMK.

I get error "HTTP 530 error code 9999"

Are there any workarounds to make Kubernetes work?

[nxsbi]

After further digging... I logged into the master node using SSH via Private Key, downloaded the /etc/kubernetes/kubectl.conf and saved it locally as kube.conf

I ran a few commands. However, the Dashboard is still not visible.

I saw the Issue #4146
It mentions about running -- curl -k on management server I think -- However in my case, management server is on a separate VLAN and the users will never have that kind of access. I ran the curl command inside the master node, and it does look like a certificate issue.. see screenshot below.

I do not know what else to check.

At this point, this is a show stopper for using Kubernetes Service for us, and is a critical bug in my opinion. I hope there is some workaround (that continues to work when new clusters get created)!!!

[shwstppr]

@weizhouapache @ravening can you please comment, this seems the same issue mentioned in #4146
@nxsbi I've tried to reproduce this in my test environment with SSL enabled for GUI but I'm not being able to reproduce. Must be the difference in certificates

The problem seems to be at https://github.com/apache/cloudstack/blob/master/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterUtil.java#L223 with server return 403 response.
Not sure if using HttpsURLConnection will help. I've created a PR (#4639) against 4.14, can you guys please help to test it

[nxsbi]

@shwstppr I asked on the other thread (#4639) - but how do I test this change in my environment?
I am not a developer, so I am not sure..
Please advise

PS> When you tried to reproduce, did you place the Management server on separate VLAN and the Kubernetes cluster in a separate VLAN such that they cannot talk to each other? When I tested with the 4.15 RC2, I did not put them on separate VLAN, and it worked fine. I am wondering if that has anything to do with it (my RC2 environment was already wiped).

[ravening]

@shwstppr Just to confirm the requirements, the mgt server should be able to ssh to vm's?
Im planning to test this again

[shwstppr]

@ravening yes, management server should be able to SSH to VMs through VR.
Service adds firewall and port forwarding rule for that

[nxsbi]

@shwstppr Per your last comment - Management Server needs to be able to SSH to VMs through VR.
That means Management Server needs to be able to connect to VR (and vice versa).
FYI - I have limited understanding of how that communication needs to happen, and I am trying to learn here, so excuse my silly question...

If the Management server sits on its own VLAN with a single NIC, the VR is on its own VLAN (its created as an Isolated Network by default, which gets its own VLAN) , Both have Internet connectivity, but the VR is not exposed (meaning the Public IP assigned is just another VLAN), how would they ever be able to communicate? Secondly, dosen't that introduce a huge security risk if the network is accessible from the VR (and hence any VM on that VR) to Management server?

Again, I do not know if my assumption here is completely off, so please correct/explain as needed

EDIT ---- I Think I answered my own question after some more research... The Management server connects to the Virtualization Host (XCP-ng in my case), and uses the "ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@LinkLocal" to get into the VR....

NOTE #2 -- In #4639 I added more details of my testing with the new build you provided. However it still failed. Here is a link for ease -- #4639 (comment)

[weizhouapache]

@shwstppr Per your last comment - Management Server needs to be able to SSH to VMs through VR.
That means Management Server needs to be able to connect to VR (and vice versa).
FYI - I have limited understanding of how that communication needs to happen, and I am trying to learn here, so excuse my silly question...

If the Management server sits on its own VLAN with a single NIC, the VR is on its own VLAN (its created as an Isolated Network by default, which gets its own VLAN) , Both have Internet connectivity, but the VR is not exposed (meaning the Public IP assigned is just another VLAN), how would they ever be able to communicate? Secondly, dosen't that introduce a huge security risk if the network is accessible from the VR (and hence any VM on that VR) to Management server?

Again, I do not know if my assumption here is completely off, so please correct/explain as needed

EDIT ---- I Think I answered my own question after some more research... The Management server connects to the Virtualization Host (XCP-ng in my case), and uses the "ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@LinkLocal" to get into the VR....

NOTE #2 -- In #4639 I added more details of my testing with the new build you provided. However it still failed. Here is a link for ease -- #4639 (comment)

@nxsbi as far as I know, when kubernetes cluster is created, some port forwarding rules are added for vms in the cluster. for example,
VR public IP:2222 -> master
VR public IP:2223 -> node-1
VR public IP:2224 -> node-2

mgt server connects to kubernetes master/nodes via the VR public IP and ports above, not linklocal IP.