Unable to acquire External IP using Kubernetes Nginx Ingress Controler

[tazouxme]

problem

Hi!

I installed a brand new CloudStack instance on my machine (4.20.1).
I managed to import the Kubernetes ISO version 1.33.1.
I was able to create a new Kubernetes cluster with 1 Controller and 3 Nodes.

Then, I wanted to install Nginx Ingress Controller and the service is stuck in state Pending (using this YAML manifest https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.13.1/deploy/static/provider/cloud/deploy.yaml). This version is compatible with Kubernetes 1.33.

kubectl get svc -n ingress-nginx

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.104.136.17 < pending > 80:30487/TCP,443:32395/TCP 8m
ingress-nginx-controller-admission ClusterIP 10.101.184.165 < none > 443/TCP 8m

The detailed error is the following

Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: Get "http://192.168.1.18:8080/client/api?apiKey=LVO-XvwBCdw0s_ES5s7f9eRkkfNSsSGa7vFXJCQsXhcTggOqo01QZpD7aQY8mtal7veZIxYKRHsErJTKt0_zdQ&command=listLoadBalancerRules&keyword=ac8f6b302d0274000bad294d69e64bd4&listall=true&response=json&signature=8BIFbTI5X6wAUt%2FLWC5brXJg%2Bfk%3D": dial tcp 192.168.1.18:8080: i/o timeout

Furthermore, when I check the logs, I see that "GET /client/api" never appears. Looks like the request never arrives to CloudStack.

Any idea how to resolve this issue?
Do I miss some configuration?

Thank you!

versions

ACS 4.20.1
CKS 1.33.1

The steps to reproduce the bug

1. Create a K8s cluster
2. Via manifest, install Nginx Ingress Controller (https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.13.1/deploy/static/provider/cloud/deploy.yaml)
3. State remains "Pending" and never gets an External IP
   ...

What to do about it?

No response

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[kiranchavala]

@tazouxme Can you try the following workaround mentioned in this PR

apache/cloudstack-kubernetes-provider#74

In "nginx-ingress-controller-patch.yml" set proxy-protocol: 'true'

[weizhouapache]

@tazouxme

if the ACS mgmt server reachable from the control node ?

ping -c3 192.168.1.18
curl http://192.168.1.18:8080/client/api

[tazouxme]

@tazouxme Can you try the following workaround mentioned in this PR

apache/cloudstack-kubernetes-provider#74

In "nginx-ingress-controller-patch.yml" set proxy-protocol: 'true'

I applied the patch with no effect...
Still the same issue.

[tazouxme]

@tazouxme

if the ACS mgmt server reachable from the control node ?

ping -c3 192.168.1.18
curl http://192.168.1.18:8080/client/api

Yes.

In the details: I'm using Lens. I can access my cluster. From there I open a shell command in the controller.
I can ping my CloudStack instance.

[weizhouapache]

@tazouxme
if the ACS mgmt server reachable from the control node ?

ping -c3 192.168.1.18
curl http://192.168.1.18:8080/client/api

Yes.

In the details: I'm using Lens. I can access my cluster. From there I open a shell command in the controller. I can ping my CloudStack instance.

is the CloudStack API endpoint reachable from the control node ?

btw: what ISO do you use ? did you create the ISO or download from somewhere ?

[tazouxme]

When I ping from the control node

ping 192.168.1.18
PING 192.168.1.18 (192.168.1.18): 56 data bytes
64 bytes from 192.168.1.18: icmp_seq=0 ttl=63 time=0.447 ms
64 bytes from 192.168.1.18: icmp_seq=1 ttl=63 time=0.347 ms
64 bytes from 192.168.1.18: icmp_seq=2 ttl=63 time=0.499 ms
64 bytes from 192.168.1.18: icmp_seq=3 ttl=63 time=0.466 ms
--- 192.168.1.18 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss

When I curl from the control node

curl http://192.168.1.18:8080/client/api
< ?xml version="1.0" encoding="UTF-8"?>< errorresponse cloud-stack-version="4.20.1.0">< errorcode>401< /errorcode>< errortext>unable to verify user credentials and/or request signature< /errortext>< /errorresponse>

I use the following Kubenetes ISO

http://download.cloudstack.org/cks/setup-v1.33.1-calico-x86_64.iso

[tazouxme]

I quickly tried to use Traefik. Same issue...

[tazouxme]

And even before installing Nginx, the cloud-controller-manager displays this error

E0826 21:23:11.537528 1 node_controller.go:249] Error getting instance metadata for node addresses: error retrieving instance type: Get "http://192.168.1.18:8080/client/api?apiKey=Z5VwKVsYUPwUjHYG7EjJJlzr6hM9wiQtd9sZgSOLyeLLMMacW8BOR3NPBeScZOSrrXjlz4GIlA1IJiIvI0K3dg&command=listVirtualMachines&name=k8s-kubernetessypt-control-198e802587b&response=json&signature=DNjfoTvi8WkoHyNIlzW14AstAdg%3D": dial tcp 192.168.1.18:8080: i/o timeout
E0826 21:23:41.538397 1 node_controller.go:249] Error getting instance metadata for node addresses: error retrieving instance type: Get "http://192.168.1.18:8080/client/api?apiKey=Z5VwKVsYUPwUjHYG7EjJJlzr6hM9wiQtd9sZgSOLyeLLMMacW8BOR3NPBeScZOSrrXjlz4GIlA1IJiIvI0K3dg&command=listVirtualMachines&name=k8s-kubernetessypt-node-198e8027f2d&response=json&signature=GLn7wQAD%2B4n3CtrE2MwW%2Bd9CqJA%3D": dial tcp 192.168.1.18:8080: i/o timeout
E0826 21:24:11.538733 1 node_controller.go:249] Error getting instance metadata for node addresses: error retrieving instance type: Get "http://192.168.1.18:8080/client/api?apiKey=Z5VwKVsYUPwUjHYG7EjJJlzr6hM9wiQtd9sZgSOLyeLLMMacW8BOR3NPBeScZOSrrXjlz4GIlA1IJiIvI0K3dg&command=listVirtualMachines&name=k8s-kubernetessypt-node-198e802a5e4&response=json&signature=YtXkNfGnkPg6FUqlB8I%2BQsDg1ik%3D": dial tcp 192.168.1.18:8080: i/o timeout

[weizhouapache]

@tazouxme
I think it is because the default pod CIDR of Calico is 192.168.0.0/16

root@cks-001-control-198eaf41116:~# kubectl get ippools.crd.projectcalico.org default-ipv4-ippool -o yaml
apiVersion: crd.projectcalico.org/v1
kind: IPPool
metadata:
  annotations:
    projectcalico.org/metadata: '{"creationTimestamp":"2025-08-27T09:58:22Z"}'
  creationTimestamp: "2025-08-27T09:58:22Z"
  generation: 1
  name: default-ipv4-ippool
  resourceVersion: "639"
  uid: 32a049ee-cf7d-44b4-b486-389ee831acf4
spec:
  allowedUses:
  - Workload
  - Tunnel
  assignmentMode: Automatic
  blockSize: 26
  cidr: 192.168.0.0/16
  ipipMode: Always
  natOutgoing: true
  nodeSelector: all()
  vxlanMode: Never

[weizhouapache]

I tested CKS on two networks

• network 1 with cidr 10.1.1.0/24, everything works
• network 1 with cidr 192.168.1.0/24, the external IP is stuck at Pending

[tazouxme]

So this is my Network.
Is this what you mean ?

[weizhouapache]

So this is my Network. Is this what you mean ?

in my testing, I just wanted to prove that it is an issue with the IP (192.168.x.x)

in your case, the issue is that kubernetes pods cannot reach the the mgmt server , because the mgmt server IP 192.168.1.18 is in the pod cidr 192.168.0.0/16.

[tazouxme]

Any advice / suggestion what I should do to make it work?

[tazouxme]

Oh I made it work.
I manually created a Network with Gateway 192.168.10.1. CIDR was then set to 192.168.10.0/24.

The LoadBalancer has now successfully acquires an External IP!
Thank you very much for your help and patience :)