CKS Enhancements - Custom CNI Documentation

[jdhirst]

problem

I have been recently testing out the CKS enhancements in Cloudstack and have been wondering if anyone happens to have an example CNI configuration for Cilium?

Additionally, the Calico example in the documentation doesnt appear to work (at least with the 1.33 prebuilt ISO I was using)

Calico example here:
https://docs.cloudstack.apache.org/en/latest/plugins/cloudstack-kubernetes-service.html

ISO I used:
https://download.cloudstack.org/cks/setup-v1.33.1-calico-x86_64.iso
(I have also tested by building a custom kubernetes image following the documentation with the same results)

I am also using the default CloudStack 4.21 system VM instead of a custom node image.

When using the custom CNI config, I get the following error in the control node's cloud-init-output.log:

2025-09-20 06:44:36,835 - util.py[WARNING]: Failed loading yaml blob. Invalid format at line 431 column 4: "while scanning for the next token
found character '%' that cannot start any token
  in "<unicode string>", line 431, column 4:
      {% if registry is defined %}
       ^"
2025-09-20 06:44:36,849 - util.py[WARNING]: Failed loading yaml blob. Invalid format at line 431 column 4: "while scanning for the next token
found character '%' that cannot start any token
  in "<unicode string>", line 431, column 4:
      {% if registry is defined %}

I pulled the userdata files from the control node in question and base64 decoded them:
userdata_0.txt
userdata_1.txt

I can see this if registry is defined thing, but am not sure whether that means something is not being templated correctly or not:

  {% if registry is defined %}
  - path: /opt/bin/setup-containerd
    permissions: '0755'
    owner: root:root
    content: |
      #!/bin/bash -e

      export registryConfig="\\        [plugins.\"io.containerd.grpc.v1.cri\".registry.mirrors.\"{{registry.url.endpoint}}\"]\n \\         endpoint = [\"{{registry.url}}\"]"
      export registryCredentials="\\      [plugins.\"io.containerd.grpc.v1.cri\".registry.configs.\"{{registry.url.endpoint}}\".auth]\n\tusername = \"{{registry.username}}\" \n\tpassword = \"{{registry.password}}\" \n\tidentitytoken = \"{{registry.token}}\""

      echo "creating config file for containerd"
      containerd config default > /etc/containerd/config.toml
      sed  -i '/\[plugins."io.containerd.grpc.v1.cri".registry\]/a '"${registryCredentials}"'' /etc/containerd/config.toml
      sed  -i '/\[plugins."io.containerd.grpc.v1.cri".registry.mirrors\]/a '"${registryConfig}"'' /etc/containerd/config.toml

      echo "Restarting containerd service"
      systemctl daemon-reload
      systemctl restart containerd
  {% endif %}

versions

cloudstack-management 4.21.0.0-1
cloudstack-agent 4.21.0.0-1

The steps to reproduce the bug

1. Either create a new kubernetes binaries ISO or use this one
2. Import the example calico CNI configuration from the documentation:

#cloud-config
- for i in {1..3}; do curl https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml -o /home/cloud/calico.yaml && break || sleep 5; done
- until [ -f /home/cloud/success ]; do sleep 5; done
- echo "Kubectl apply file"
- for i in {1..3}; do sudo /opt/bin/kubectl create -f /home/cloud/calico.yaml && break || sleep 5; done
- export PATH=$PATH:/home/cloud
- |
cat << 'EOF' > /home/cloud/create-configs.sh
#!/bin/bash
cat << 'EOL' > /home/cloud/bgp-config.yaml
apiVersion: crd.projectcalico.org/v1
kind: BGPConfiguration
metadata:
name: default
spec:
logSeverityScreen: Debug
asNumber: {{ AS_NUMBER }}
EOL
cat << 'EOL' > /home/cloud/bgp-peer.yaml
apiVersion: crd.projectcalico.org/v1
kind: BGPPeer
metadata:
name: bgp-peer-example
spec:
peerIP: {{ ds.meta_data.peer_ip_address }}
asNumber: {{ ds.meta_data.peer_as_number }}
EOL
EOF
- chmod +x /home/cloud/create-configs.sh
- /home/cloud/create-configs.sh
- for i in {1..3}; do sudo /opt/bin/kubectl apply -f /home/cloud/bgp-config.yaml && break || sleep 5; done
- for i in {1..3}; do sudo /opt/bin/kubectl apply -f /home/cloud/bgp-peer.yaml && break || sleep 5; done

3. Create a new cluster and define the custom CNI configuration. It will be stuck in Starting state forever and the control node will fail to render the cloud init scripts properly. To login to the node, use the debian user instead of cloud since it hasn't yet rebooted into the new user.
   ...

What to do about it?

Please provide a working example of a custom CNI configuration for CloudStack 4.21.0. Either calico or cilium (preferred since I am hoping to build clusters with cilium).

[prashanthr2]

@jdhirst

>>I can see this if registry is defined thing, but am not sure whether that means something is not being templated correctly or not:

This is the issue. Based on the userdata files you shared, they are clearly not in yaml format ( at the line where the failure is). Looks like the line where the failure is jinja2 template format. AFAIK this should be translated or rendered by Cloudstack.

Unfortunately, I don't have any sample custom CNI config. I will try to see if I can find any. Meanwhile, can you confirm on how debian user login worked for you? Was it through ssh or password based login? this is not working for me to check the cloud-init logs from my repro

[kiranchavala]

@jdhirst Have you specified BGP peer and AS in the cloudstack zone

https://docs.cloudstack.apache.org/en/4.21.0.0/adminguide/networking/dynamic_static_routing.html#manage-as-number-for-dynamic-routing

https://docs.cloudstack.apache.org/en/4.21.0.0/adminguide/networking/dynamic_static_routing.html#manage-bpg-peers-for-dynamic-routing

https://docs.cloudstack.apache.org/en/4.21.0.0/adminguide/networking/dynamic_static_routing.html

[prashanthr2]

@jdhirst As mentioned by @kiranchavala the failure when using example calico CNI could be as a result of missing Routed Network and BGP config.

Here is a sample configuration for the Cilium CNI network you were looking for. I tested this in my lab environment and was able to successfully deploy a CKS cluster. This configuration uses the VXLAN overlay mode for pod networking. As you can see, I have hardcoded the cilium version

Please treat this as a reference only. The configuration may require adjustments based on your environment, Kubernetes version, and operational constraints. You should thoroughly test in a non-production setup before rolling it out in production, as CNI changes can directly affect cluster networking and workload availability.

  - |
    cat >/home/cloud/cilium-install.sh <<'EOF'
    #!/bin/bash
    set -ex
    export KUBECONFIG=/etc/kubernetes/admin.conf
    export PATH=$PATH:/opt/bin:/usr/local/bin
    export HOME=/root   # fix for cilium cache issue

    # Wait until kube-apiserver is ready
    until kubectl get nodes >/dev/null 2>&1; do
      echo "Waiting for kube-apiserver..."
      sleep 5
    done

    # Install cilium-cli
    curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz
    tar xzf cilium-linux-amd64.tar.gz
    mv cilium /usr/local/bin/cilium

    # Deploy Cilium
    cilium install --version 1.18.2 --wait
    cilium status --wait
    EOF
  - chmod +x /home/cloud/cilium-install.sh
  - /home/cloud/cilium-install.sh || true