Skip to content

Commit 5b1ffe5

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #16920 from rsafonseca/apiserver_env
Add support for configuring environment variables on kube-apiserver
2 parents 63c7d83 + 8b89e82 commit 5b1ffe5

22 files changed

+1561
-1
lines changed

‎docs/cluster_spec.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -640,6 +640,17 @@ spec:
640640
logFormat: json
641641
```
642642

643+
### Environment Variables
644+
```yaml
645+
spec:
646+
kubeAPIServer:
647+
env:
648+
- name: GOMEMLIMIT
649+
value: "2750MiB"
650+
- name: GOGC
651+
value: 50
652+
```
653+
643654
## externalDns
644655

645656
This block contains configuration options for your `external-DNS` provider.

‎k8s/crds/kops.k8s.io_clusters.yaml‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1885,6 +1885,129 @@ spec:
18851885
description: EncryptionProviderConfig enables encryption at rest
18861886
for secrets.
18871887
type: string
1888+
env:
1889+
description: |-
1890+
Env allows users to pass in env variables to the apiserver container.
1891+
This can be useful to control some environment runtime settings, such as GOMEMLIMIT and GOCG to tweak the memory settings of the apiserver
1892+
This also allows the flexibility for adding any other variables for future use cases
1893+
items:
1894+
description: EnvVar represents an environment variable present
1895+
in a Container.
1896+
properties:
1897+
name:
1898+
description: Name of the environment variable. Must be a
1899+
C_IDENTIFIER.
1900+
type: string
1901+
value:
1902+
description: |-
1903+
Variable references $(VAR_NAME) are expanded
1904+
using the previously defined environment variables in the container and
1905+
any service environment variables. If a variable cannot be resolved,
1906+
the reference in the input string will be unchanged. Double $$ are reduced
1907+
to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e.
1908+
"$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)".
1909+
Escaped references will never be expanded, regardless of whether the variable
1910+
exists or not.
1911+
Defaults to "".
1912+
type: string
1913+
valueFrom:
1914+
description: Source for the environment variable's value.
1915+
Cannot be used if value is not empty.
1916+
properties:
1917+
configMapKeyRef:
1918+
description: Selects a key of a ConfigMap.
1919+
properties:
1920+
key:
1921+
description: The key to select.
1922+
type: string
1923+
name:
1924+
default: ""
1925+
description: |-
1926+
Name of the referent.
1927+
This field is effectively required, but due to backwards compatibility is
1928+
allowed to be empty. Instances of this type with an empty value here are
1929+
almost certainly wrong.
1930+
More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
1931+
type: string
1932+
optional:
1933+
description: Specify whether the ConfigMap or its
1934+
key must be defined
1935+
type: boolean
1936+
required:
1937+
- key
1938+
type: object
1939+
x-kubernetes-map-type: atomic
1940+
fieldRef:
1941+
description: |-
1942+
Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['<KEY>']`, `metadata.annotations['<KEY>']`,
1943+
spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs.
1944+
properties:
1945+
apiVersion:
1946+
description: Version of the schema the FieldPath
1947+
is written in terms of, defaults to "v1".
1948+
type: string
1949+
fieldPath:
1950+
description: Path of the field to select in the
1951+
specified API version.
1952+
type: string
1953+
required:
1954+
- fieldPath
1955+
type: object
1956+
x-kubernetes-map-type: atomic
1957+
resourceFieldRef:
1958+
description: |-
1959+
Selects a resource of the container: only resources limits and requests
1960+
(limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported.
1961+
properties:
1962+
containerName:
1963+
description: 'Container name: required for volumes,
1964+
optional for env vars'
1965+
type: string
1966+
divisor:
1967+
anyOf:
1968+
- type: integer
1969+
- type: string
1970+
description: Specifies the output format of the
1971+
exposed resources, defaults to "1"
1972+
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
1973+
x-kubernetes-int-or-string: true
1974+
resource:
1975+
description: 'Required: resource to select'
1976+
type: string
1977+
required:
1978+
- resource
1979+
type: object
1980+
x-kubernetes-map-type: atomic
1981+
secretKeyRef:
1982+
description: Selects a key of a secret in the pod's
1983+
namespace
1984+
properties:
1985+
key:
1986+
description: The key of the secret to select from. Must
1987+
be a valid secret key.
1988+
type: string
1989+
name:
1990+
default: ""
1991+
description: |-
1992+
Name of the referent.
1993+
This field is effectively required, but due to backwards compatibility is
1994+
allowed to be empty. Instances of this type with an empty value here are
1995+
almost certainly wrong.
1996+
More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
1997+
type: string
1998+
optional:
1999+
description: Specify whether the Secret or its key
2000+
must be defined
2001+
type: boolean
2002+
required:
2003+
- key
2004+
type: object
2005+
x-kubernetes-map-type: atomic
2006+
type: object
2007+
required:
2008+
- name
2009+
type: object
2010+
type: array
18882011
etcdCaFile:
18892012
description: EtcdCAFile is the path to a ca certificate
18902013
type: string

‎nodeup/pkg/model/kube_apiserver.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -699,7 +699,7 @@ func (b *KubeAPIServerBuilder) buildPod(ctx context.Context, kubeAPIServer *kops
699699
container := &v1.Container{
700700
Name: "kube-apiserver",
701701
Image: image,
702-
Env: proxy.GetProxyEnvVars(b.NodeupConfig.Networking.EgressProxy),
702+
Env: append(kubeAPIServer.Env, proxy.GetProxyEnvVars(b.NodeupConfig.Networking.EgressProxy)...),
703703
LivenessProbe: livenessProbe,
704704
ReadinessProbe: readinessProbe,
705705
StartupProbe: startupProbe,

‎nodeup/pkg/model/kube_apiserver_test.go‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -204,3 +204,10 @@ func TestKubeAPIServerBuilderARM64(t *testing.T) {
204204
return builder.Build(target)
205205
})
206206
}
207+
208+
func TestKubeAPIServerEnvBuilder(t *testing.T) {
209+
RunGoldenTest(t, "tests/golden/envvars", "kube-apiserver", func(nodeupModelContext *NodeupModelContext, target *fi.NodeupModelBuilderContext) error {
210+
builder := KubeAPIServerBuilder{NodeupModelContext: nodeupModelContext}
211+
return builder.Build(target)
212+
})
213+
}

‎nodeup/pkg/model/tests/golden/envvars/cluster.yaml‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
apiVersion: kops.k8s.io/v1alpha2
2+
kind: Cluster
3+
metadata:
4+
name: minimal.example.com
5+
spec:
6+
kubernetesApiAccess:
7+
- 0.0.0.0/0
8+
channel: stable
9+
cloudProvider: aws
10+
configBase: memfs://clusters.example.com/minimal.example.com
11+
etcdClusters:
12+
- cpuRequest: 200m
13+
etcdMembers:
14+
- instanceGroup: master-us-test-1a
15+
name: us-test-1a
16+
memoryRequest: 100Mi
17+
name: main
18+
provider: Manager
19+
backups:
20+
backupStore: memfs://clusters.example.com/minimal.example.com/backups/etcd-main
21+
- cpuRequest: 100m
22+
etcdMembers:
23+
- instanceGroup: master-us-test-1a
24+
name: us-test-1a
25+
memoryRequest: 100Mi
26+
name: events
27+
provider: Manager
28+
backups:
29+
backupStore: memfs://clusters.example.com/minimal.example.com/backups/etcd-events
30+
iam: {}
31+
kubeAPIServer:
32+
env:
33+
- name: GOMEMLIMIT
34+
valueFrom:
35+
resourceFieldRef:
36+
resource: limits.memory
37+
divisor: '1'
38+
- name: GOGC
39+
value: "50"
40+
kubelet:
41+
anonymousAuth: false
42+
kubernetesVersion: v1.28.0
43+
masterPublicName: api.minimal.example.com
44+
networkCIDR: 172.20.0.0/16
45+
networking:
46+
kubenet: {}
47+
nonMasqueradeCIDR: 100.64.0.0/10
48+
sshAccess:
49+
- 0.0.0.0/0
50+
subnets:
51+
- cidr: 172.20.32.0/19
52+
name: us-test-1a
53+
type: Public
54+
zone: us-test-1a
55+
56+
---
57+
58+
apiVersion: kops.k8s.io/v1alpha2
59+
kind: InstanceGroup
60+
metadata:
61+
name: master-us-test-1a
62+
labels:
63+
kops.k8s.io/cluster: minimal.example.com
64+
spec:
65+
associatePublicIp: true
66+
image: ami-1234
67+
machineType: m3.medium
68+
maxSize: 1
69+
minSize: 1
70+
role: Master
71+
subnets:
72+
- us-test-1a

‎nodeup/pkg/model/tests/golden/envvars/tasks-kops-controller.yaml‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
mode: "0755"
2+
path: /etc/kubernetes/kops-controller
3+
type: directory
4+
---
5+
contents: |
6+
kubernetes-ca: "3"
7+
service-account: "2"
8+
mode: "0600"
9+
owner: kops-controller
10+
path: /etc/kubernetes/kops-controller/keypair-ids.yaml
11+
type: file
12+
---
13+
contents:
14+
task:
15+
Name: kops-controller
16+
alternateNames:
17+
- kops-controller.internal.minimal.example.com
18+
keypairID: "3"
19+
signer: kubernetes-ca
20+
subject:
21+
CommonName: kops-controller
22+
type: server
23+
mode: "0644"
24+
owner: kops-controller
25+
path: /etc/kubernetes/kops-controller/kops-controller.crt
26+
type: file
27+
---
28+
contents:
29+
task:
30+
Name: kops-controller
31+
alternateNames:
32+
- kops-controller.internal.minimal.example.com
33+
keypairID: "3"
34+
signer: kubernetes-ca
35+
subject:
36+
CommonName: kops-controller
37+
type: server
38+
mode: "0600"
39+
owner: kops-controller
40+
path: /etc/kubernetes/kops-controller/kops-controller.key
41+
type: file
42+
---
43+
contents: |
44+
-----BEGIN CERTIFICATE-----
45+
MIIC2DCCAcCgAwIBAgIRALJXAkVj964tq67wMSI8oJQwDQYJKoZIhvcNAQELBQAw
46+
FTETMBEGA1UEAxMKa3ViZXJuZXRlczAeFw0xNzEyMjcyMzUyNDBaFw0yNzEyMjcy
47+
MzUyNDBaMBUxEzARBgNVBAMTCmt1YmVybmV0ZXMwggEiMA0GCSqGSIb3DQEBAQUA
48+
A4IBDwAwggEKAoIBAQDgnCkSmtnmfxEgS3qNPaUCH5QOBGDH/inHbWCODLBCK9gd
49+
XEcBl7FVv8T2kFr1DYb0HVDtMI7tixRVFDLgkwNlW34xwWdZXB7GeoFgU1xWOQSY
50+
OACC8JgYTQ/139HBEvgq4sej67p+/s/SNcw34Kk7HIuFhlk1rRk5kMexKIlJBKP1
51+
YYUYetsJ/QpUOkqJ5HW4GoetE76YtHnORfYvnybviSMrh2wGGaN6r/s4ChOaIbZC
52+
An8/YiPKGIDaZGpj6GXnmXARRX/TIdgSQkLwt0aTDBnPZ4XvtpI8aaL8DYJIqAzA
53+
NPH2b4/uNylat5jDo0b0G54agMi97+2AUrC9UUXpAgMBAAGjIzAhMA4GA1UdDwEB
54+
/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBVGR2r
55+
hzXzRMU5wriPQAJScszNORvoBpXfZoZ09FIupudFxBVU3d4hV9StKnQgPSGA5XQO
56+
HE97+BxJDuA/rB5oBUsMBjc7y1cde/T6hmi3rLoEYBSnSudCOXJE4G9/0f8byAJe
57+
rN8+No1r2VgZvZh6p74TEkXv/l3HBPWM7IdUV0HO9JDhSgOVF1fyQKJxRuLJR8jt
58+
O6mPH2UX0vMwVa4jvwtkddqk2OAdYQvH9rbDjjbzaiW0KnmdueRo92KHAN7BsDZy
59+
VpXHpqo1Kzg7D3fpaXCf5si7lqqrdJVXH4JC72zxsPehqgi8eIuqOBkiDWmRxAxh
60+
8yGeRx9AbknHh4Ia
61+
-----END CERTIFICATE-----
62+
mode: "0600"
63+
owner: kops-controller
64+
path: /etc/kubernetes/kops-controller/kubernetes-ca.crt
65+
type: file
66+
---
67+
contents: |
68+
-----BEGIN RSA PRIVATE KEY-----
69+
MIIEpAIBAAKCAQEA4JwpEprZ5n8RIEt6jT2lAh+UDgRgx/4px21gjgywQivYHVxH
70+
AZexVb/E9pBa9Q2G9B1Q7TCO7YsUVRQy4JMDZVt+McFnWVwexnqBYFNcVjkEmDgA
71+
gvCYGE0P9d/RwRL4KuLHo+u6fv7P0jXMN+CpOxyLhYZZNa0ZOZDHsSiJSQSj9WGF
72+
GHrbCf0KVDpKieR1uBqHrRO+mLR5zkX2L58m74kjK4dsBhmjeq/7OAoTmiG2QgJ/
73+
P2IjyhiA2mRqY+hl55lwEUV/0yHYEkJC8LdGkwwZz2eF77aSPGmi/A2CSKgMwDTx
74+
9m+P7jcpWreYw6NG9BueGoDIve/tgFKwvVFF6QIDAQABAoIBAA0ktjaTfyrAxsTI
75+
Bezb7Zr5NBW55dvuII299cd6MJo+rI/TRYhvUv48kY8IFXp/hyUjzgeDLunxmIf9
76+
/Zgsoic9Ol44/g45mMduhcGYPzAAeCdcJ5OB9rR9VfDCXyjYLlN8H8iU0734tTqM
77+
0V13tQ9zdSqkGPZOIcq/kR/pylbOZaQMe97BTlsAnOMSMKDgnftY4122Lq3GYy+t
78+
vpr+bKVaQZwvkLoSU3rECCaKaghgwCyX7jft9aEkhdJv+KlwbsGY6WErvxOaLWHd
79+
cuMQjGapY1Fa/4UD00mvrA260NyKfzrp6+P46RrVMwEYRJMIQ8YBAk6N6Hh7dc0G
80+
8Z6i1m0CgYEA9HeCJR0TSwbIQ1bDXUrzpftHuidG5BnSBtax/ND9qIPhR/FBW5nj
81+
22nwLc48KkyirlfIULd0ae4qVXJn7wfYcuX/cJMLDmSVtlM5Dzmi/91xRiFgIzx1
82+
AsbBzaFjISP2HpSgL+e9FtSXaaqeZVrflitVhYKUpI/AKV31qGHf04sCgYEA6zTV
83+
99Sb49Wdlns5IgsfnXl6ToRttB18lfEKcVfjAM4frnkk06JpFAZeR+9GGKUXZHqs
84+
z2qcplw4d/moCC6p3rYPBMLXsrGNEUFZqBlgz72QA6BBq3X0Cg1Bc2ZbK5VIzwkg
85+
ST2SSux6ccROfgULmN5ZiLOtdUKNEZpFF3i3qtsCgYADT/s7dYFlatobz3kmMnXK
86+
sfTu2MllHdRys0YGHu7Q8biDuQkhrJwhxPW0KS83g4JQym+0aEfzh36bWcl+u6R7
87+
KhKj+9oSf9pndgk345gJz35RbPJYh+EuAHNvzdgCAvK6x1jETWeKf6btj5pF1U1i
88+
Q4QNIw/QiwIXjWZeubTGsQKBgQCbduLu2rLnlyyAaJZM8DlHZyH2gAXbBZpxqU8T
89+
t9mtkJDUS/KRiEoYGFV9CqS0aXrayVMsDfXY6B/S/UuZjO5u7LtklDzqOf1aKG3Q
90+
dGXPKibknqqJYH+bnUNjuYYNerETV57lijMGHuSYCf8vwLn3oxBfERRX61M/DU8Z
91+
worz/QKBgQDCTJI2+jdXg26XuYUmM4XXfnocfzAXhXBULt1nENcogNf1fcptAVtu
92+
BAiz4/HipQKqoWVUYmxfgbbLRKKLK0s0lOWKbYdVjhEm/m2ZU8wtXTagNwkIGoyq
93+
Y/C1Lox4f1ROJnCjc/hfcOjcxX5M8A8peecHWlVtUPKTJgxQ7oMKcw==
94+
-----END RSA PRIVATE KEY-----
95+
mode: "0600"
96+
owner: kops-controller
97+
path: /etc/kubernetes/kops-controller/kubernetes-ca.key
98+
type: file
99+
---
100+
Name: kops-controller
101+
alternateNames:
102+
- kops-controller.internal.minimal.example.com
103+
keypairID: "3"
104+
signer: kubernetes-ca
105+
subject:
106+
CommonName: kops-controller
107+
type: server
108+
---
109+
Name: kops-controller
110+
home: ""
111+
shell: /sbin/nologin
112+
uid: 10011

0 commit comments

Comments
 (0)