Skip to content

Commit af2a287

Browse files
CaoShuFengCaoShuFeng
committed
advanced audit beta features
1 parent 7b692cd commit af2a287

File tree

1 file changed

+124
-65
lines changed
  • docs/tasks/debug-application-cluster

1 file changed

+124
-65
lines changed

‎docs/tasks/debug-application-cluster/audit.md‎

Lines changed: 124 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ answer the following questions:
2222
- from where was it initiated?
2323
- to where was it going?
2424

25-
## Audit logs
25+
## Legacy Audit
2626

2727
Kubernetes audit is part of [Kube-apiserver][kube-apiserver] logging all requests
2828
processed by the server. Each audit log entry contains two lines:
@@ -37,8 +37,13 @@ Example output for `admin` user listing pods in the `default` namespace:
3737
2017-03-21T03:57:09.108403639-04:00 AUDIT: id="c939d2a7-1c37-4ef1-b2f7-4ba9b1e43b53" response="200"
3838
```
3939

40-
Note that this format changes when enabling the `AdvancedAuditing` feature discussed
41-
later in this document.
40+
Note that Kubernetes 1.8 has switched to use the advanced structured audit log by default.
41+
To fallback to this legacy audit, disable the advanced auditing feature
42+
using the `AdvancedAuditing` feature gate on the [kube-apiserver][kube-apiserver]:
43+
44+
```
45+
--feature-gates=AdvancedAuditing=false
46+
```
4247

4348
### Configuration
4449

@@ -62,18 +67,12 @@ and `audit-log-maxage` options.
6267
## Advanced audit
6368

6469
Kubernetes 1.7 expands auditing with experimental functionality such as event
65-
filtering and a webhook for integration with external systems. The rest of this
66-
document covers features that are __alpha__ and may change in backward incompatible
67-
ways.
70+
filtering and a webhook for integration with external systems. Kubernetes 1.8
71+
upgrades the advanced audit feature to beta, and some backward incompatible changes
72+
have been committed.
6873

69-
Enable the alpha auditing features using the `AdvancedAuditing` feature gate on
70-
the [kube-apiserver][kube-apiserver]:
7174

72-
```
73-
--feature-gates=AdvancedAuditing=true
74-
```
75-
76-
`AdvancedAuditing`is customizable in two ways. Policy, which determines what's recorded,
75+
`AdvancedAuditing` is customizable in two ways. Policy, which determines what's recorded,
7776
and backends, which persist records. Backend implementations include logs files and
7877
webhooks.
7978

@@ -110,10 +109,17 @@ The policy file holds rules that determine the level of an event. Known audit le
110109
When an event is processed, it's compared against the list of rules in order.
111110
The first matching rule sets the audit level of the event. The audit policy is
112111
defined by the [`audit.k8s.io` API group][audit-api].
112+
Some new fields are supported in beta version, like `resourceNames` and `omitStages`.
113+
114+
In Kubernetes 1.8 `kind` and `apiVersion` along with `rules` __must__ be provided in
115+
the audit policy file. A policy file with 0 rules, or a policy file that doesn't provide
116+
a valid `apiVersion` and `kind` value will be treated as illgal.
113117

114-
An example audit policy file:
118+
Some example audit policy files:
115119

116120
```yaml
121+
apiVersion: audit.k8s.io/v1beta1 #this is required in Kubernetes 1.8
122+
kind: Policy
117123
rules:
118124
# Don't log watch requests by the "system:kube-proxy" on endpoints or services
119125
- level: None
@@ -155,10 +161,46 @@ rules:
155161
- level: Metadata
156162
```
157163
164+
The next audit policy file shows new features introduced in Kubernetes 1.8:
165+
166+
```yaml
167+
apiVersion: audit.k8s.io/v1beta1
168+
kind: Policy
169+
rules:
170+
# Log pod changes at Request level
171+
- level: Request
172+
resources:
173+
- group: ""
174+
# Resource "pods" no longer matches requests to any subresource of pods,
175+
# This behavior is consistent with the RBAC policy.
176+
resources: ["pods"]
177+
# Log "pods/log", "pods/status" at Metadata level
178+
- level: Metadata
179+
resources:
180+
- group: ""
181+
resources: ["pods/log", "pods/status"]
182+
183+
# Don't log requests to a configmap called "controller-leader"
184+
- level: None
185+
resources:
186+
- group: ""
187+
resources: ["configmaps"]
188+
resourceNames: ["controller-leader"]
189+
190+
# A catch-all rule to log all other requests at the Metadata level.
191+
# For this rule we use "omitStages" to omit events at "ReqeustReceived" stage.
192+
# Events in this stage will not be sent to backend.
193+
- level: Metadata
194+
omitStages:
195+
- "RequestReceived"
196+
```
197+
158198
You can use a minimal audit policy file to log all requests at the `Metadata` level:
159199

160200
```yaml
161201
# Log all requests at the Metadata level.
202+
apiVersion: audit.k8s.io/v1beta1
203+
kind: Policy
162204
rules:
163205
- level: Metadata
164206
```
@@ -181,33 +223,36 @@ API can be found [here][audit-api] with more details about the exact fields capt
181223
#### Log backend
182224

183225
The behavior of the `--audit-log-path` flag changes when enabling the `AdvancedAuditing`
184-
feature flag. This includes the cleanups discussed above, such as changes to the `method`
185-
values and the introduction of a "stage" for each event. As before, the `id` field of
186-
the log indicates which events were generated from the same request. With default legacy
187-
format, events are formatted as follows:
226+
feature flag. All generated events defined by `--audit-policy-file` are recorded in structured
227+
json format:
188228

189229
```
190-
2017-06-15T21:50:50.259470834Z AUDIT: id="591e9fde-6a98-46f6-b7bc-ec8ef575696d" stage="RequestReceived" ip="10.2.1.3" method="update" user="system:serviceaccount:kube-system:default" groups="\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="kube-system" uri="/api/v1/namespaces/kube-system/endpoints/kube-controller-manager" response="<deferred>"
191-
2017-06-15T21:50:50.259470834Z AUDIT: id="591e9fde-6a98-46f6-b7bc-ec8ef575696d" stage="ResponseComplete" ip="10.2.1.3" method="update" user="system:serviceaccount:kube-system:default" groups="\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="kube-system" uri="/api/v1/namespaces/kube-system/endpoints/kube-controller-manager" response="200"
230+
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"level":"Metadata","timestamp":"2017-09-05T10:04:55Z","auditID":"77e58433-d345-40ac-b2d8-9866bd355cea","stage":"RequestReceived","requestURI":"/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles","verb":"list","user":{"username":"kubecfg","groups":["system:masters","system:authenticated"]},"sourceIPs":["172.16.116.128"],"objectRef":{"resource":"roles","namespace":"default","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"}}
231+
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"level":"Metadata","timestamp":"2017-09-05T10:04:55Z","auditID":"77e58433-d345-40ac-b2d8-9866bd355cea","stage":"ResponseComplete","requestURI":"/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles","verb":"list","user":{"username":"kubecfg","groups":["system:masters","system:authenticated"]},"sourceIPs":["172.16.116.128"],"objectRef":{"resource":"roles","namespace":"default","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200}}
192232
```
193233

194-
Logged events omit the request and response bodies. The `Request` and
195-
`RequestResponse` levels are equivalent to `Metadata` for legacy format.
234+
In alpha version, objectRef.apiVersion holds both the api group and version.
235+
In beta version these were break out into objectRef.apiGroup and objectRef.apiVersion.
196236

197-
Since Kubernetes 1.8, structed json fromat is supported for log backend.
198-
Use the following option to switch log to json format:
237+
Starting from Kubernetes 1.8, structured json format is used for log backend by default.
238+
Use the following option to switch log to legacy format:
199239

200240
```
201-
--audit-log-format=json
241+
--audit-log-format=legacy
202242
```
203243

204-
With json format, events are formatted as follows:
244+
With legacy format, events are formatted as follows:
205245

206246
```
207-
{"kind":"Event","apiVersion":"audit.k8s.io/v1alpha1","metadata":{"creationTimestamp":null},"level":"Metadata","timestamp":"2017-07-12T11:02:43Z","auditID":"2e79f0c9-a941-45ae-a9ce-663a1b19ff14","stage":"RequestReceived","requestURI":"/api/v1/namespaces/default/pods","verb":"list","user":{"username":"kubecfg","groups":["system:masters","system:authenticated"]},"sourceIPs":["172.16.116.128"],"objectRef":{"resource":"pods","namespace":"default","apiVersion":"/v1"}}
208-
{"kind":"Event","apiVersion":"audit.k8s.io/v1alpha1","metadata":{"creationTimestamp":null},"level":"Metadata","timestamp":"2017-07-12T11:02:43Z","auditID":"2e79f0c9-a941-45ae-a9ce-663a1b19ff14","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/pods","verb":"list","user":{"username":"kubecfg","groups":["system:masters","system:authenticated"]},"sourceIPs":["172.16.116.128"],"objectRef":{"resource":"pods","namespace":"default","apiVersion":"/v1"},"responseStatus":{"metadata":{},"code":200}}
247+
2017-09-05T06:08:19.885328047-04:00 AUDIT: id="c28a95ad-f9dd-47e1-a617-b6dc152db95f" stage="RequestReceived" ip="172.16.116.128" method="list" user="kubecfg" groups="\"system:masters\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="default" uri="/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles" response="<deferred>"
248+
2017-09-05T06:08:19.885328047-04:00 AUDIT: id="c28a95ad-f9dd-47e1-a617-b6dc152db95f" stage="ResponseComplete" ip="172.16.116.128" method="list" user="kubecfg" groups="\"system:masters\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="default" uri="/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles" response="200"
209249
```
210250

251+
Logged events omit the request and response bodies. The `Request` and
252+
`RequestResponse` levels are equivalent to `Metadata` for legacy format. This legacy format
253+
of advanced audit is different from the [Legacy Audit](# Legacy Audit) discussed above, such
254+
as changes to the method values and the introduction of a "stage" for each event.
255+
211256
#### Webhook backend
212257

213258
The audit webhook backend can be used to have [kube-apiserver][kube-apiserver]
@@ -255,44 +300,54 @@ Events are POSTed as a JSON serialized `EventList`. An example payload:
255300
256301
```json
257302
{
258-
"kind": "EventList",
259-
"apiVersion": "audit.k8s.io/v1alpha1",
260-
"items": [
261-
{
262-
"metadata": {
263-
"creationTimestamp": null
264-
},
265-
"level": "Metadata",
266-
"timestamp": "2017-06-15T23:07:40Z",
267-
"auditID": "4faf711a-9094-400f-a876-d9188ceda548",
268-
"stage": "ResponseComplete",
269-
"requestURI": "/apis/rbac.authorization.k8s.io/v1beta1/namespaces/kube-public/rolebindings/system:controller:bootstrap-signer",
270-
"verb": "get",
271-
"user": {
272-
"username": "system:apiserver",
273-
"uid": "97a62906-e4d7-4048-8eda-4f0fb6ff8f1e",
274-
"groups": [
275-
"system:masters"
276-
]
277-
},
278-
"sourceIPs": [
279-
"127.0.0.1"
280-
],
281-
"objectRef": {
282-
"resource": "rolebindings",
283-
"namespace": "kube-public",
284-
"name": "system:controller:bootstrap-signer",
285-
"apiVersion": "rbac.authorization.k8s.io/v1beta1"
286-
},
287-
"responseStatus": {
288-
"metadata": {},
289-
"code": 200
290-
}
291-
}
292-
]
303+
"apiVersion": "audit.k8s.io/v1beta1",
304+
"items": [
305+
{
306+
"auditID": "24f30caf-d7d4-45d5-b7bd-e7af300d7886",
307+
"level": "Metadata",
308+
"metadata": {
309+
"creationTimestamp": null
310+
},
311+
"objectRef": {
312+
"apiGroup": "rbac.authorization.k8s.io",
313+
"apiVersion": "v1",
314+
"name": "jane",
315+
"namespace": "default",
316+
"resource": "roles"
317+
},
318+
"requestURI": "/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles/jane",
319+
"responseStatus": {
320+
"code": 200,
321+
"metadata": {}
322+
},
323+
"sourceIPs": [
324+
"172.16.116.128"
325+
],
326+
"stage": "ResponseComplete",
327+
"timestamp": "2017-09-05T10:20:24Z",
328+
"user": {
329+
"groups": [
330+
"system:masters",
331+
"system:authenticated"
332+
],
333+
"username": "kubecfg"
334+
},
335+
"verb": "get"
336+
}
337+
],
338+
"kind": "EventList",
339+
"metadata": {}
293340
}
294341
```
295342

343+
### Audit-Id
344+
345+
Audit-Id is a unique ID for each http request to kube-apiserver. The ID of events will be the
346+
same if they were generated from the same request. Starting from Kubernetes 1.8, if an audit
347+
event is generated for the request, kube-apiserver will respond with an Audit-Id in the HTTP header.
348+
Note that for some special requests like `kubectl exec`, `kubectl attach`, kube-apiserver works
349+
like a proxy, no Audit-Id will be returned even if audit events are recorded.
350+
296351
### Log Collector Examples
297352

298353
#### Use fluentd to collect and distribute audit events from log file
@@ -422,9 +477,13 @@ Note that in addition to file output plugin, logstash has a variety of outputs t
422477
let users route data where they want. For example, users can emit audit events to elasticsearch
423478
plugin which supports full-text search and analytics.
424479

425-
[audit-api]: https://github.com/kubernetes/kubernetes/blob/v1.7.0-rc.1/staging/src/k8s.io/apiserver/pkg/apis/audit/v1alpha1/types.go
480+
<!---
481+
herf of audit-api is not right, because https://github.com/kubernetes/kubernetes/pull/49280 is not included in this tag.
482+
Maybe it will show up in another tag, then I will update it.
483+
-->
484+
[audit-api]: https://github.com/kubernetes/kubernetes/blob/v1.8.0-beta.0/staging/src/k8s.io/apiserver/pkg/apis/audit/v1beta1/types.go
426485
[kube-apiserver]: /docs/admin/kube-apiserver
427-
[gce-audit-profile]: https://github.com/kubernetes/kubernetes/blob/v1.7.0/cluster/gce/gci/configure-helper.sh#L490
486+
[gce-audit-profile]: https://github.com/kubernetes/kubernetes/blob/v1.8.0-beta.0/cluster/gce/gci/configure-helper.sh#L532
428487
[fluentd]: http://www.fluentd.org/
429488
[fluentd_install_doc]: http://docs.fluentd.org/v0.12/articles/quickstart#step1-installing-fluentd
430489
[logstash]: https://www.elastic.co/products/logstash

0 commit comments

Comments
 (0)