Skip to content

Commit f38bf97

Browse files
nikhitanikhita
committed
Add proposal for SubResources for CustomResources
1 parent ff82a64 commit f38bf97

File tree

1 file changed

+203
-0
lines changed

1 file changed

+203
-0
lines changed

‎contributors/design-proposals/api-machinery/customresources-subresources.md‎

Lines changed: 203 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,203 @@
1+
# Subresources for CustomResources
2+
3+
Authors: @nikhita, @sttts
4+
5+
## Table of Contents
6+
7+
1. [Abstract](#abstract)
8+
2. [Goals](#goals)
9+
3. [Non-Goals](#non-goals)
10+
4. [Proposed Extension of CustomResourceDefinition](#proposed-extension-of-customresourcedefinition)
11+
1. [API Types](#api-types)
12+
2. [Feature Gate](#feature-gate)
13+
5. [Semantics](#semantics)
14+
1. [Validation Behavior](#validation-behavior)
15+
1. [Status](#status)
16+
2. [Scale](#scale)
17+
2. [Status Behavior](#status-behavior)
18+
3. [Scale Behavior](#scale-behavior)
19+
1. [Status Replicas Behavior](#status-replicas-behavior)
20+
2. [Selector Behavior](#selector-behavior)
21+
4. [Implementation Plan](#implementation-plan)
22+
5. [Alternatives](#alternatives)
23+
1. [Scope](#scope)
24+
25+
## Abstract
26+
27+
[CustomResourceDefinitions](https://github.com/kubernetes/community/pull/524) (CRDs) were introduced in 1.7. The objects defined by CRDs are called CustomResources (CRs). Currently, we do not provide subresources for CRs.
28+
29+
However, it is one of the [most requested features](https://github.com/kubernetes/kubernetes/issues/38113) and this proposal seeks to add `/status` and `/scale` subresources for CustomResources.
30+
31+
## Goals
32+
33+
1. Support status/spec split for CustomResources:
34+
1. Status changes are ignored on the main resource endpoint.
35+
2. Support a `/status` subresource HTTP path for status changes.
36+
3. `metadata.Generation` is increased only on spec changes.
37+
2. Support a `/scale` subresource for CustomResources.
38+
3. Maintain backward compatibility by allowing CRDs to opt-in to enable subresources.
39+
4. If a CustomResource is already structured using spec/status, allow it to easily transition to use the `/status` and `/scale` endpoint.
40+
5. Work seamlessly with [JSON Schema validation](https://github.com/kubernetes/community/pull/708).
41+
42+
## Non-Goals
43+
44+
1. Allow defining arbitrary subresources i.e. subresources except `/status` and `/scale`.
45+
46+
## Proposed Extension of CustomResourceDefinition
47+
48+
### API Types
49+
50+
The addition of the following external types in `apiextensions.k8s.io/v1beta1` is proposed:
51+
52+
```go
53+
type CustomResourceDefinitionSpec struct {
54+
...
55+
// SubResources describes the subresources for CustomResources
56+
// This field is alpha-level and should only be sent to servers that enable
57+
// subresources via the CurstomResourceSubResources feature gate.
58+
// +optional
59+
SubResources *CustomResourceSubResources `json:"subResources,omitempty"`
60+
}
61+
62+
// CustomResourceSubResources defines the status and scale subresources for CustomResources.
63+
type CustomResourceSubResources struct {
64+
// Status denotes the status subresource for CustomResources
65+
Status *CustomResourceSubResourceStatus `json:"status,omitempty"`
66+
// Scale denotes the scale subresource for CustomResources
67+
Scale *CustomResourceSubResourceScale `json:"scale,omitempty"`
68+
}
69+
70+
// CustomResourceSubResourceStatus defines how to serve the HTTP path <CR Name>/status.
71+
type CustomResourceSubResourceStatus struct {
72+
}
73+
74+
// CustomResourceSubResourceScale defines how to serve the HTTP path <CR name>/scale.
75+
type CustomResourceSubResourceScale struct {
76+
// required, e.g. “.spec.replicas”. Must be under `.spec`.
77+
// Only JSON paths without the array notation are allowed.
78+
SpecReplicasPath string `json:"specReplicasPath"`
79+
// optional, e.g. “.status.replicas”. Must be under `.status`.
80+
// Only JSON paths without the array notation are allowed.
81+
StatusReplicasPath string `json:"statusReplicasPath,omitempty"`
82+
// optional, e.g. “.spec.labelSelector”. Must be under `.spec`.
83+
// Only JSON paths without the array notation are allowed.
84+
LabelSelectorPath string `json:"labelSelectorPath,omitempty"`
85+
// ScaleGroupVersion denotes the GroupVersion of the Scale
86+
// object sent as the payload for /scale. It allows transition
87+
// to future versions easily.
88+
// Today only autoscaling/v1 is allowed.
89+
ScaleGroupVersion schema.GroupVersion `json:"groupVersion"`
90+
}
91+
```
92+
93+
### Feature Gate
94+
95+
The `SubResources` field in `CustomResourceDefinitionSpec` will be gated under the `CustomResourceSubResources` alpha feature gate.
96+
If the gate is not open, the value of the new field within `CustomResourceDefinitionSpec` is dropped on creation and updates of CRDs.
97+
98+
### Scale type
99+
100+
The `Scale` object is the payload sent over the wire for `/scale`. The [polymorphic `Scale` type](https://github.com/kubernetes/kubernetes/pull/53743) i.e. `autoscaling/v1.Scale` is used for the `Scale` object.
101+
102+
Since the GroupVersion of the `Scale` object is specified in `CustomResourceSubResourceScale`, transition to future versions (eg `autoscaling/v2.Scale`) can be done easily.
103+
104+
Note: If `autoscaling/v1.Scale` is deprecated, then it would be deprecated here as well.
105+
106+
## Semantics
107+
108+
### Validation Behavior
109+
110+
#### Status
111+
112+
The status endpoint of a CustomResource receives a full CR object. Changes outside of the `.status` subpath are ignored.
113+
For validation, the JSON Schema present in the CRD is validated only against the `.status` subpath.
114+
115+
To validate only against the schema for the `.status` subpath, `oneOf` and `anyOf` constructs are not allowed within the root of the schema, but only under a properties sub-schema (with this restriction, we can project a schema to a sub-path). The following is forbidden in the CRD spec:
116+
117+
```yaml
118+
validation:
119+
openAPIV3Schema:
120+
oneOf:
121+
...
122+
```
123+
124+
**Note**: The restriction for `oneOf` and `anyOf` allows us to write a projection function `ProjectJSONSchema(schema *JSONSchemaProps, path []string) (*JSONSchemaProps, error)` that can be used to apply a given schema for the whole object to only the sub-path `.status` or `.spec`.
125+
126+
#### Scale
127+
128+
Moreover, if the scale subresource is enabled:
129+
130+
On update, we copy the values from the `Scale` object into the specified paths in the CustomResource, if the path is set (`StatusReplicasPath` and `LabelSelectorPath` are optional).
131+
If `StatusReplicasPath` or `LabelSelectorPath` is not set, we validate that the value in `Scale` is also not specified and return an error otherwise.
132+
133+
On `get` and on `update` (after copying the values into the CustomResource as described above), we verify that:
134+
135+
- The value at the specified JSON Path `SpecReplicasPath` (e.g. `.spec.replicas`) is a non-negative integer value and is not empty.
136+
137+
- The value at the optional JSON Path `StatusReplicasPath` (e.g. `.status.replicas`) is an integer value if it exists (i.e. this can be empty).
138+
139+
- The value at the optional JSON Path `LabelSelectorPath` (e.g. `.spec.labelSelector`) is a valid label selector if it exists (i.e. this can be empty).
140+
141+
**Note**: The values at the JSON Paths specified by `SpecReplicasPath`, `LabelSelectorPath` and `StatusReplicasPath` are also validated with the same rules when the whole object or, in case the `/status` subresource is enabled, the `.status` sub-object is updated.
142+
143+
### Status Behavior
144+
145+
If the `/status` subresource is enabled, the following behaviors change:
146+
147+
- The main resource endpoint will ignore all changes in the status subpath.
148+
(note: it will **not** reject requests which try to change the status, following the existing semantics of other resources).
149+
150+
- The `.metadata.generation` field is updated if and only if the value at the `.spec` subpath changes.
151+
Additionally, if the spec does not change, `.metadata.generation` is not updated.
152+
153+
- The `/status` subresource receives a full resource object, but only considers the value at the `.status` subpath for the update.
154+
The value at the `.metadata` subpath is **not** considered for update as decided in https://github.com/kubernetes/kubernetes/issues/45539.
155+
156+
Both the status and the spec (and everything else if there is anything) of the object share the same key in the storage layer, i.e. the value at `.metadata.resourceVersion` is increased for any kind of change. There is no split of status and spec in the storage layer.
157+
158+
The `/status` endpoint supports both `get` and `update` verbs.
159+
160+
### Scale Behavior
161+
162+
The number of CustomResources can be easily scaled up or down depending on the replicas field present in the `.spec` subpath.
163+
164+
Only `ScaleSpec.Replicas` can be written. All other values are read-only and changes will be ignored. i.e. upon updating the scale subresource, two fields are modified:
165+
166+
1. The replicas field is copied back from the `Scale` object to the main resource as specified by `SpecReplicasPath` in the CRD, e.g. `.spec.replicas = scale.Spec.Replicas`.
167+
168+
2. The resource version is copied back from the `Scale` object to the main resource before writing to the storage: `.metadata.resourceVersion = scale.ResourceVersion`.
169+
In other words, the scale and the CustomResource share the resource version used for optimistic concurrency.
170+
Updates with outdated resource versions are rejected with a conflict error, read requests will return the resource version of the CustomResource.
171+
172+
The `/scale` endpoint supports both `get` and `update` verbs.
173+
174+
#### Status Replicas Behavior
175+
176+
As only the `scale.Spec.Replicas` field is to be written to by the CR user, the user-provided controller (not any generic CRD controller) counts its children and then updates the controlled object by writing to the `/status` subresource, i.e. the `scale.Status.Replicas` field is read-only.
177+
178+
#### Selector Behavior
179+
180+
`CustomResourceSubResourceScale.LabelSelectorPath` is the label selector over CustomResources that should match the replicas count.
181+
The value in the `Scale` object is one-to-one the value from the CustomResource if the label selector is non-empty.
182+
Intentionally we do not default it to another value from the CustomResource (e.g. `.spec.template.metadata.labels`) as this turned out to cause trouble (e.g. in `kubectl apply`) and it is generally seen as a wrong approach with existing resources.
183+
184+
## Implementation Plan
185+
186+
The `/scale` and `/status` subresources are mostly distinct. It is proposed to do the implementation in two phases (the order does not matter much):
187+
188+
1. `/status` subresource
189+
2. `/scale` subresource
190+
191+
## Alternatives
192+
193+
### Scope
194+
195+
In this proposal we opted for an opinionated concept of subresources i.e. we restrict the subresource spec to the two very specific subresources: `/status` and `/scale`.
196+
We do not aim for a more generic subresource concept. In Kubernetes there are a number of other subresources like `/log`, `/exec`, `/bind`. But their semantics is much more special than `/status` and `/scale`.
197+
Hence, we decided to leave those other subresources to the domain of User provided API Server (UAS) instead of inventing a more complex subresource concept for CustomResourceDefinitions.
198+
199+
**Note**: The types do not make the addition of other subresources impossible in the future.
200+
201+
We also restrict the JSON path for the status and the spec within the CustomResource.
202+
We could make them definable by the user and the proposed types actually allow us to open this up in the future.
203+
For the time being we decided to be opinionated as all status and spec subobjects in existing types live under `.status` and `.spec`. Keeping this pattern imposes consistency on user provided CustomResources as well.

0 commit comments

Comments
 (0)