SSL - LetsEncrypt the Console Proxy

[asender]

ISSUE TYPE

• Feature Idea

COMPONENT NAME

Console Proxy VM

CLOUDSTACK VERSION

4.11.1+

CONFIGURATION

LetsEncrypt the Console Proxy

Many new applications are now integrating letsencrypt. This could replace the old retired realhostip.com DNS resolver . Noone would need to muck around with certs again on the console proxy.

SUMMARY

New Global Option For Letsencrypt enable on console proxy. Letsencrypt domain name option for letsencrypt ssl auto renew

[DaanHoogland]

any plans for this @asender ?

[DaanHoogland]

@davidjumani @weizhouapache can you advice how this would impact novnc? it seems not relevant to the type of console but i am not sure.

[weizhouapache]

@davidjumani @weizhouapache can you advice how this would impact novnc? it seems not relevant to the type of console but i am not sure.

@DaanHoogland it provides a default certificate (like old realhostip.com) to systemvms. It is an enhancement . ajax/novnc console will not be impacted.

[rohityadavcloud]

This is btw possible by using nginx proxy which is used for SSL termination. I've detailed how to do this here - https://markmail.org/message/ukueqxzrrv7g3nte

[lukasmrtvy]

@rohityadavcloud this will not work for systemvms deployed to public network, which is advanced setup and 99% of use cases, why whould I spend $ for another IP for additional reverse proxy? honestly dont know why its not possible to deploy systemvms to "private" public network and use reverse proxy... afaik its not possible to create another public network assigned only for systemvm from lets say cloudbr2 bridge...

[rohityadavcloud]

The email thread I've shared in fact saves you an IP address - typically how you can deploy a private ACS cloud behind a single public IP; you can deploy parts of the pod IP range in a private RFC1918 range (not public IP) and have the mgmt server and console proxy (maybe ssvm too) be LB'd by something like nginx in my example to do SSL termination.

[lukasmrtvy]

But Systemvms will have always ( in advanced mode ) IPs assigned from the Public network and private IPs from the Management network, its not possible to change this behavior. One can only assign a dedicated range for Systemvms from Public and Management network, but thats all..

I dont understand from Your email, in mentioned email list how to do it exactly.

Note:
I have /24 subnet of public IPs assigned to the Public network ( not RFC1918 )

Thanks

[rohityadavcloud]

My email/suggestion was to use RFC1918 networks for both public and management networks, of course this won't work for your use-case. You can use the traditional approach to setup/upload certificates (until something like this feature is fully automated).

[daviftorres]

Dear @asender,

I’d like to apologise for having a different view on the certificate strategy for the Console Proxy (and System VMs in general).

Since System VMs are ephemeral, they would request new certificates every time they are redeployed. This could quickly hit Let’s Encrypt rate limits (https://letsencrypt.org/docs/rate-limits/).

A better approach is to use a dedicated instance (or even a container) to request certificates, for example a wildcard via DNS challenge, about 30 days before expiry. The instance can then push the certs to CloudStack for use by the System VMs.

If you’re interested, I can also share how we set this up with HashiCorp Vault to securely store certificates and allow consumers to fetch only the ones they’re permitted to use.

[DaanHoogland]

@daviftorres , if you could share here, that would be great.

[daviftorres]

Sure!

This is my cheat sheet for deploying and performing the most common operations on HashiCorp Vault. I’ll leave the link here because I try to keep it always up to date: https://dft.wiki/?p=5070

Deployment

Deploying Vault is basically a one-liner with package managers like snap or apt.
Enabling the UI is optional, but honestly it’s very handy when showing around policies and vaults.

Unsealing

Vault does not store the keys to unseal (decrypt) the vaults. since it is always kept in memory, right after installing, and after each reboot, you need to unseal them manually. I haven’t figured out yet how to use HSM or vTPM to unseal automatically, but it’s on my radar.

Operation

Step 1 - Enable Key Vault (KV)

export VAULT_ADDR=http://127.0.0.1:8200
vault login
vault secrets enable -path=kv kv

Step 2 - Create Policies

Read+Write Policy - for issuing certificates

cat <<EOF > readwrite-policy.hcl
path "kv/*" {
    capabilities = ["create", "read", "update", "delete", "list"]
}
EOF
vault policy write readwrite readwrite-policy.hcl

Read-Only Policy - for consuming the secrets

cat <<EOF > readonly-policy.hcl
path "kv/*" {
    capabilities = ["read", "list"]
}
EOF
vault policy write readonly readonly-policy.hcl

Check

vault policy list

Note: The examples above allow access to all secrets in KV. In production, always create tailor-made policies per application+secret.

Step 3 - Generate Tokens

vault token create -policy=readonly -orphan -no-default-policy -ttl=0
vault token create -policy=readwrite -orphan -no-default-policy -ttl=0

Check

vault token lookup <TOKEN>

Note: These examples generate tokens that do not expire. In practice, use reasonable TTLs and set up automated re-issuance.

Step 4 - Put and Get Secrets

Basic put using vault command

vault kv put kv/example.com cert=TEST key=TEST fullchain=TEST

Put from files using vault command

vault kv put kv/example.com cert=@/etc/letsencrypt/live/example.com/cert.pem key=@/etc/letsencrypt/live/example.com/privkey.pem fullchain=@/etc/letsencrypt/live/example.com/fullchain.pem

Basic get using vault command

vault kv get kv/example.com

Basic put using curl command

curl --header "X-Vault-Token: TOKEN" --request POST --data '{"data": {"cert": "TEST", "key": "TEST", "fullchain": "TEST"}}' https://vault.example.com/v1/kv/data/example.com

Basic get using curl command

curl --header "X-Vault-Token: TOKEN" --request GET https://vault.example.com/v1/kv/data/example.com | jq '.data.data'

[DaanHoogland]

great @daviftorres , do you think we should automate this in ACS somehow? (or is it too much of a one time setup?)

[daviftorres]

@DaanHoogland , here is my take on all of this.

We currently have 4x ACS environments. 1x Prod and 3x Non-Prods.

All MS and SysVMs have a real but long-expired wildcard SSL/TLS certificate. The trick is that SSL/TLS is terminated at the edge on a cluster of NGINX servers. By default, NGINX does not check backend cert validity, and this check can be disabled if enabled.

Updating certificates on NGINX causes no downtime. It’s a millisecond operation that reloads the new certs into runtime.

We don’t update the certs on the backend side (MS and SysVMs) because doing so restarts them, causing 1~3 minutes of downtime (interrupting noVNC sessions, failing ISO/template transfers, logging out WebUI users, and stopping API responses). While some of these issues can be minimized, some interruptions are unavoidable.

In summary, I don’t recommend changing how ACS handles certificates, since managing their lifecycle on the reverse proxy (NGINX) is simpler and causes no downtime.

One thing I missed when starting with ACS was the lack of documentation on setting up a reverse proxy. It’s mentioned in discussions and conference talks, but the method is never demonstrated.

[DaanHoogland]

sounds like a good entry for “best practices” , cc @NuxRo

[abdessalllam]

Full guide on NGINX with CloudStack/SSL/Cloudflare for MGMT Server and SystemVMs: https://abdessal.am/cloudstack-part-4-first-vm-walkthrough-uefi-setup-domain-setup-w-sslcloudflare/