|
| 1 | +--- |
| 2 | +title: "Istio Mtls With Spire How It Works" |
| 3 | +date: 2025-10-17T11:45:22+08:00 |
| 4 | +draft: true |
| 5 | +description: "" |
| 6 | +tags: ["Istio", "SPIFFE"] |
| 7 | +categories: ["Security"] |
| 8 | +author: "Shawn Zhang" |
| 9 | +showToc: true |
| 10 | +TocOpen: false |
| 11 | +hidemeta: false |
| 12 | +comments: false |
| 13 | +disableHLJS: false |
| 14 | +disableShare: false |
| 15 | +searchHidden: false |
| 16 | +mermaid: true |
| 17 | +cover: |
| 18 | + image: "" |
| 19 | + alt: "" |
| 20 | + caption: "" |
| 21 | + relative: false |
| 22 | + hidden: true |
| 23 | +--- |
| 24 | + |
| 25 | +# Istio mTLS with SPIRE - How It Works |
| 26 | + |
| 27 | +## Overview |
| 28 | + |
| 29 | +When using Istio with SPIRE, applications communicate using plain HTTP, but the Istio sidecars automatically upgrade connections to mTLS using SPIRE-issued certificates. This provides transparent security without requiring application code changes. |
| 30 | + |
| 31 | +## Communication Flow |
| 32 | + |
| 33 | +{{< mermaid >}} |
| 34 | +sequenceDiagram |
| 35 | + participant Curl as curl container<br/>(Plain HTTP) |
| 36 | + participant CurlProxy as curl's istio-proxy<br/>SPIFFE: spiffe://foo.com/ns/default/sa/curl |
| 37 | + participant HttpbinProxy as httpbin's istio-proxy<br/>SPIFFE: spiffe://foo.com/ns/default/sa/httpbin |
| 38 | + participant Httpbin as httpbin container<br/>(Plain HTTP) |
| 39 | + |
| 40 | + Curl->>CurlProxy: 1. HTTP Request<br/>http://httpbin:8000/headers |
| 41 | + CurlProxy->>HttpbinProxy: 2. mTLS Handshake<br/>(mutual authentication) |
| 42 | + CurlProxy->>HttpbinProxy: 3. Encrypted mTLS Connection<br/>(SPIRE certificates) |
| 43 | + HttpbinProxy->>Httpbin: 4. HTTP Request<br/>(decrypted, localhost) |
| 44 | + Httpbin->>HttpbinProxy: HTTP Response |
| 45 | + HttpbinProxy->>CurlProxy: 5. Encrypted Response<br/>(adds X-Forwarded-Client-Cert) |
| 46 | + CurlProxy->>Curl: HTTP Response<br/>(decrypted) |
| 47 | +{{< /mermaid >}} |
| 48 | + |
| 49 | +## Step-by-Step Process |
| 50 | + |
| 51 | +### 1. Application Makes HTTP Request |
| 52 | +```bash |
| 53 | +curl http://httpbin:8000/headers |
| 54 | +``` |
| 55 | +- The curl container sends a plain HTTP request |
| 56 | +- No TLS, no certificates, no encryption at application level |
| 57 | + |
| 58 | +### 2. Sidecar Intercepts Request |
| 59 | +- curl's istio-proxy sidecar intercepts the outbound HTTP request |
| 60 | +- Determines the destination is httpbin service |
| 61 | + |
| 62 | +### 3. mTLS Handshake |
| 63 | +- curl's sidecar initiates mTLS connection to httpbin's sidecar |
| 64 | +- Both sidecars present their SPIRE-issued certificates: |
| 65 | + - **curl sidecar**: `spiffe://foo.com/ns/default/sa/curl` |
| 66 | + - **httpbin sidecar**: `spiffe://foo.com/ns/default/sa/httpbin` |
| 67 | +- Mutual authentication succeeds using SPIRE trust domain |
| 68 | + |
| 69 | +### 4. Encrypted Communication |
| 70 | +- HTTP request is encrypted and sent over mTLS connection |
| 71 | +- Only the sidecars handle encryption/decryption |
| 72 | +- Application containers remain unaware of TLS |
| 73 | + |
| 74 | +### 5. Sidecar Forwards to Application |
| 75 | +- httpbin's sidecar decrypts the request |
| 76 | +- Forwards plain HTTP to httpbin container on localhost |
| 77 | +- Adds `X-Forwarded-Client-Cert` header with client identity |
| 78 | + |
| 79 | +### 6. Response Returns |
| 80 | +- httpbin container sends HTTP response |
| 81 | +- httpbin's sidecar encrypts it with mTLS |
| 82 | +- curl's sidecar decrypts and forwards to curl container |
| 83 | + |
| 84 | +## Evidence of mTLS |
| 85 | + |
| 86 | +### X-Forwarded-Client-Cert Header |
| 87 | +```json |
| 88 | +{ |
| 89 | + "X-Forwarded-Client-Cert": [ |
| 90 | + "By=spiffe://foo.com/ns/default/sa/httpbin;Hash=...;Subject=\"O=SPIRE,C=US\";URI=spiffe://foo.com/ns/default/sa/curl" |
| 91 | + ] |
| 92 | +} |
| 93 | +``` |
| 94 | + |
| 95 | +This header proves: |
| 96 | +- **By**: Server identity (httpbin's sidecar) |
| 97 | +- **URI**: Client identity (curl's sidecar) |
| 98 | +- **Subject**: Certificate issued by SPIRE |
| 99 | +- **Hash**: Certificate fingerprint |
| 100 | + |
| 101 | +### Certificate Verification |
| 102 | +```bash |
| 103 | +# Check curl's certificate |
| 104 | +istioctl proxy-config secret curl-pod -o json | \ |
| 105 | + jq -r '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | \ |
| 106 | + base64 --decode | openssl x509 -text -noout |
| 107 | + |
| 108 | +# Shows: |
| 109 | +# Issuer: O=SPIRE |
| 110 | +# URI: spiffe://foo.com/ns/default/sa/curl |
| 111 | +``` |
| 112 | + |
| 113 | +## Why HTTP Instead of HTTPS? |
| 114 | + |
| 115 | +### Benefits of Transparent mTLS |
| 116 | + |
| 117 | +1. **Zero Application Changes** |
| 118 | + - No TLS libraries needed in application code |
| 119 | + - No certificate management in applications |
| 120 | + - Developers write simple HTTP code |
| 121 | + |
| 122 | +2. **Centralized Security** |
| 123 | + - Security policy managed by platform team |
| 124 | + - Consistent mTLS across all services |
| 125 | + - Certificate rotation handled automatically |
| 126 | + |
| 127 | +3. **Simplified Development** |
| 128 | + - Local development uses plain HTTP |
| 129 | + - Production gets automatic mTLS |
| 130 | + - No environment-specific code |
| 131 | + |
| 132 | +4. **Performance** |
| 133 | + - Sidecars handle TLS termination |
| 134 | + - Applications focus on business logic |
| 135 | + - Optimized TLS implementation in Envoy |
| 136 | + |
| 137 | +## Key Components |
| 138 | + |
| 139 | +### SPIRE |
| 140 | +- Issues X.509 certificates (SVIDs) to workloads |
| 141 | +- Provides SPIFFE IDs based on Kubernetes identity |
| 142 | +- Manages certificate lifecycle and rotation |
| 143 | + |
| 144 | +### SPIFFE CSI Driver |
| 145 | +- Mounts SPIRE socket into pods |
| 146 | +- Path: `/run/secrets/workload-spiffe-uds` |
| 147 | +- Enables Envoy to fetch certificates from SPIRE |
| 148 | + |
| 149 | +### Istio Sidecar (Envoy) |
| 150 | +- Intercepts all inbound/outbound traffic |
| 151 | +- Performs mTLS handshake with peer sidecars |
| 152 | +- Fetches certificates from SPIRE via CSI socket |
| 153 | +- Adds identity headers for authorization |
| 154 | + |
| 155 | +### Application Container |
| 156 | +- Sends/receives plain HTTP |
| 157 | +- Unaware of mTLS or certificates |
| 158 | +- Focuses on business logic only |
| 159 | + |
| 160 | +## Configuration Requirements |
| 161 | + |
| 162 | +### 1. SPIRE Installation |
| 163 | +```yaml |
| 164 | +global: |
| 165 | + spire: |
| 166 | + clusterName: foo-eks-cluster |
| 167 | + trustDomain: foo.com |
| 168 | +``` |
| 169 | +
|
| 170 | +### 2. Istio Configuration |
| 171 | +```yaml |
| 172 | +meshConfig: |
| 173 | + trustDomain: foo.com # Must match SPIRE |
| 174 | + |
| 175 | +values: |
| 176 | + sidecarInjectorWebhook: |
| 177 | + templates: |
| 178 | + spire: | |
| 179 | + labels: |
| 180 | + spiffe.io/spire-managed-identity: "true" |
| 181 | + spec: |
| 182 | + volumes: |
| 183 | + - name: workload-socket |
| 184 | + csi: |
| 185 | + driver: "csi.spiffe.io" |
| 186 | +``` |
| 187 | +
|
| 188 | +### 3. Workload Registration |
| 189 | +```yaml |
| 190 | +apiVersion: spire.spiffe.io/v1alpha1 |
| 191 | +kind: ClusterSPIFFEID |
| 192 | +metadata: |
| 193 | + name: istio-sidecar-reg |
| 194 | +spec: |
| 195 | + spiffeIDTemplate: "spiffe://{{ .TrustDomain }}/ns/{{ .PodMeta.Namespace }}/sa/{{ .PodSpec.ServiceAccountName }}" |
| 196 | + podSelector: |
| 197 | + matchLabels: |
| 198 | + spiffe.io/spire-managed-identity: "true" |
| 199 | +``` |
| 200 | +
|
| 201 | +### 4. Pod Deployment |
| 202 | +```yaml |
| 203 | +metadata: |
| 204 | + labels: |
| 205 | + spiffe.io/spire-managed-identity: "true" |
| 206 | + annotations: |
| 207 | + inject.istio.io/templates: "sidecar,spire" |
| 208 | +``` |
| 209 | +
|
| 210 | +## Verification Commands |
| 211 | +
|
| 212 | +### Test mTLS Communication |
| 213 | +```bash |
| 214 | +kubectl exec curl-pod -c curl -- curl http://httpbin:8000/headers |
| 215 | +``` |
| 216 | + |
| 217 | +### Check Certificate |
| 218 | +```bash |
| 219 | +istioctl proxy-config secret curl-pod -o json | \ |
| 220 | + jq -r '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | \ |
| 221 | + base64 --decode | openssl x509 -text -noout | grep "URI:spiffe" |
| 222 | +``` |
| 223 | + |
| 224 | +### Verify SPIRE Entries |
| 225 | +```bash |
| 226 | +kubectl exec -n spire-server spire-server-0 -c spire-server -- \ |
| 227 | + /opt/spire/bin/spire-server entry show -socketPath /tmp/spire-server/private/api.sock |
| 228 | +``` |
| 229 | + |
| 230 | +## Common Misconceptions |
| 231 | + |
| 232 | +### ❌ "Applications must use HTTPS" |
| 233 | +**Reality**: Applications use HTTP. Sidecars handle mTLS automatically. |
| 234 | + |
| 235 | +### ❌ "Need to manage certificates in application" |
| 236 | +**Reality**: SPIRE and Istio manage all certificates. Applications are unaware. |
| 237 | + |
| 238 | +### ❌ "mTLS requires code changes" |
| 239 | +**Reality**: Zero code changes. Just add labels and annotations to pods. |
| 240 | + |
| 241 | +### ❌ "HTTP is insecure in service mesh" |
| 242 | +**Reality**: HTTP between container and sidecar is on localhost. mTLS protects network traffic. |
| 243 | + |
| 244 | +## Security Guarantees |
| 245 | + |
| 246 | +1. **Mutual Authentication**: Both client and server verify each other's identity |
| 247 | +2. **Encryption**: All network traffic encrypted with TLS 1.3 |
| 248 | +3. **Identity-based Authorization**: Policies based on SPIFFE IDs |
| 249 | +4. **Automatic Rotation**: Certificates rotated without application restart |
| 250 | +5. **Zero Trust**: Every connection authenticated, even within cluster |
| 251 | + |
| 252 | +## Troubleshooting |
| 253 | + |
| 254 | +### Pod Stuck at 1/2 Ready |
| 255 | +- Check if pod has label: `spiffe.io/spire-managed-identity: "true"` |
| 256 | +- Verify ClusterSPIFFEID matches the pod |
| 257 | +- Check istio-proxy logs: `kubectl logs pod-name -c istio-proxy` |
| 258 | + |
| 259 | +### "workload is not authorized" Error |
| 260 | +- ClusterSPIFFEID selectors don't match the pod |
| 261 | +- SPIRE entry not created for the workload |
| 262 | +- Check: `kubectl get clusterspiffeid -o yaml` |
| 263 | + |
| 264 | +### No X-Forwarded-Client-Cert Header |
| 265 | +- mTLS not enabled or working |
| 266 | +- Check both pods have sidecars injected |
| 267 | +- Verify both pods have SPIRE certificates |
| 268 | + |
| 269 | +## References |
| 270 | + |
| 271 | +- [Istio SPIRE Integration](https://istio.io/latest/docs/ops/integrations/spire/) |
| 272 | +- [SPIFFE Specification](https://spiffe.io/docs/latest/spiffe-about/overview/) |
| 273 | +- [SPIRE Documentation](https://spiffe.io/docs/latest/spire-about/spire-concepts/) |
| 274 | +- [Envoy SDS API](https://www.envoyproxy.io/docs/envoy/latest/configuration/security/secret) |
0 commit comments