Skip to content

Commit ff5d1de

Browse files
nikhitanikhita
committed
Proposal: SubResources for CustomResources
1 parent 8998246 commit ff5d1de

File tree

1 file changed

+221
-0
lines changed

1 file changed

+221
-0
lines changed

‎contributors/design-proposals/customresources-subresources.md‎

Lines changed: 221 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,221 @@
1+
# Subresources for CustomResources
2+
3+
Authors: @nikhita, @sttts
4+
5+
## Table of Contents
6+
7+
1. [Abstract](#abstract)
8+
2. [Goals](#goals)
9+
3. [Non-Goals](#non-goals)
10+
4. [Proposed Extension of CustomResourceDefinition](#proposed-extension-of-customresourcedefinition)
11+
1. [API Types](#api-types)
12+
2. [Feature Gate](#feature-gate)
13+
5. [Semantics](#semantics)
14+
1. [Validation Behavior](#validation-behavior)
15+
1. [Status](#status)
16+
2. [Scale](#scale)
17+
2. [Status Behavior](#status-behavior)
18+
3. [Scale Behavior](#scale-behavior)
19+
1. [Status Replicas Behavior](#status-replicas-behavior)
20+
2. [Selector Behavior](#selector-behavior)
21+
4. [Implementation Plan](#implementation-plan)
22+
5. [Alternatives](#alternatives)
23+
1. [Scope](#scope)
24+
25+
## Abstract
26+
27+
[CustomResourceDefinitions](https://github.com/kubernetes/community/pull/524) (CRDs) were introduced in 1.7. The objects defined by CRDs are called CustomResources (CRs). Currently, we do not provide subresources for CRs.
28+
29+
However, it is one of the [most requested features](https://github.com/kubernetes/kubernetes/issues/38113) and this proposal seeks to add `/status` and `/scale` subresources for CustomResources.
30+
31+
## Goals
32+
33+
1. Support status/spec split for CustomResources:
34+
1. Status changes are ignored on the main resource endpoint.
35+
2. Support a `/status` subresource HTTP path for status changes.
36+
3. `metadata.Generation` is increased only on spec changes.
37+
2. Support a `/scale` subresource for CustomResources.
38+
3. Maintain backward compatibility by allowing CRDs to opt-in to enable subresources.
39+
4. If a CustomResource is already structured using spec/status, allow it to easily transition to use the `/status` and `/scale` endpoint.
40+
5. Work seamlessly with [JSON Schema validation](https://github.com/kubernetes/community/pull/708).
41+
42+
## Non-Goals
43+
44+
1. Allow defining arbitrary subresources i.e. subresources except `/status` and `/scale`.
45+
2. Unify the many `Scale` types: as long as there is no generic `Scale` object in Kubernetes, we will propose to introduce yet another `Scale` type in the `apiextensions.k8s.io` api group. If Kubernetes switches to a generic `Scale` object, `apiextensions.k8s.io` will follow.
46+
47+
## Proposed Extension of CustomResourceDefinition
48+
49+
### API Types
50+
51+
The addition of the following external types in `apiextensions.k8s.io/v1beta1` is proposed:
52+
53+
```go
54+
type CustomResourceDefinitionSpec struct {
55+
...
56+
// SubResources describes the subresources for CustomResources
57+
// This field is alpha-level and should only be sent to servers that enable
58+
// subresources via the CurstomResourceSubResources feature gate.
59+
// +optional
60+
SubResources *CustomResourceSubResources `json:“subResources,omitempty”`
61+
}
62+
63+
// CustomResourceSubResources defines the status and scale subresources for CustomResources.
64+
type CustomResourceSubResources struct {
65+
// Status denotes the status subresource for CustomResources
66+
Status *CustomResourceSubResourceStatus `json:“status,omitempty”`
67+
// Scale denotes the scale subresource for CustomResources
68+
Scale *CustomResourceSubResourceScale `json:“scale,omitempty”`
69+
}
70+
71+
// CustomResourceSubResourceStatus defines how to serve the HTTP path <CR Name>/status.
72+
type CustomResourceSubResourceStatus struct {
73+
// The JSON path (e.g. “.status”) of the status of a CustomResource.
74+
// The whole object is transferred over the wire, but only the status can be mutated via the /status subresource.
75+
// StatusPath is restricted to be “.status” and defaults to “.status”.
76+
StatusPath string `json:“statusPath,omitempty”`
77+
// The JSON path (e.g. “.spec”) of the spec of a CustomResource.
78+
// SpecPath is restricted to be “.spec” and defaults to “.spec”.
79+
// Changes to the specified JSON path increase the “.metadata.generation” value.
80+
SpecPath string `json:“specPath,omitempty”`
81+
}
82+
83+
// CustomResourceSubResourceScale defines how to serve the HTTP path <CR name>/scale.
84+
type CustomResourceSubResourceScale struct {
85+
// required, e.g. “.spec.replicas”.
86+
// Only JSON paths without the array notation are allowed.
87+
SpecReplicasPath string `json:“specReplicasPath,omitempty”`
88+
// optional, e.g. “.status.replicas”.
89+
// Only JSON paths without the array notation are allowed.
90+
StatusReplicasPath string `json:“statusReplicasPath,omitempty”`
91+
// optional, e.g. “.spec.labelSelector”.
92+
// Only JSON paths without the array notation are allowed.
93+
LabelSelectorPath string `json:“labelSelectorPath,omitempty”`
94+
}
95+
96+
// The following is the payload to send over the wire for /scale. It happens
97+
// to be defined here in apiextensions.k8s.io because we don’t have a global
98+
// Scale type in meta/v1. Ref: https://github.com/kubernetes/kubernetes/issues/49504
99+
100+
// Scale represents a scaling request for a resource.
101+
type Scale struct {
102+
metav1.TypeMeta `json:",inline"`
103+
// Standard object metadata; More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata.
104+
// +optional
105+
metav1.ObjectMeta `json:"metadata,omitempty"`
106+
107+
// defines the behavior of the scale. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status.
108+
// +optional
109+
Spec ScaleSpec `json:"spec,omitempty"`
110+
111+
// current status of the scale. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status. Read-only.
112+
// +optional
113+
Status ScaleStatus `json:"status,omitempty"`
114+
}
115+
116+
// ScaleSpec describes the attributes of a scale subresource.
117+
type ScaleSpec struct {
118+
// desired number of instances for the scaled object.
119+
// +optional
120+
Replicas int32 `json:"replicas,omitempty"`
121+
}
122+
123+
// ScaleStatus represents the current status of a scale subresource.
124+
type ScaleStatus struct {
125+
// actual number of observed instances of the scaled object.
126+
Replicas int32 `json:"replicas"`
127+
128+
// label query that should match the replicas count. This is same
129+
// as the label selector but in the string format to avoid introspection
130+
// by clients. The string will be in the same format as the query-param syntax.
131+
// More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors
132+
// +optional
133+
Selector string `json:"selector"`
134+
}
135+
```
136+
137+
### Feature Gate
138+
139+
The `SubResources` field in `CustomResourceDefinitionSpec` will be gated under the `CustomResourceSubResources` alpha feature gate.
140+
If the gate is not open, the value of the new field within `CustomResourceDefinitionSpec` is dropped on creation and updates of CRDs.
141+
142+
## Semantics
143+
144+
### Validation Behavior
145+
146+
#### Status
147+
148+
The status endpoint of a CustomResource receives a full CR object. Changes outside of the status path are ignored.
149+
For validation everything outside of the JSON Path `StatusPath` can be reset to the existing values of the CustomResource. Then the JSON Schema presented in the CRD is validated against the whole object.
150+
151+
Note: one could extract the status subobject, filter the JSON schema by rules applied to the status and then validate the status subobject only.
152+
The filter step of the JSON schema though is not obvious in the case that there are status validations not below a properties JSON Schema construct. For that reason, the alternative implementation is preferred as its semantics are much simpler.
153+
154+
#### Scale
155+
156+
Moreover, if the scale subresource is enabled:
157+
158+
- The value at the specified JSON Path `SpecReplicasPath` (e.g. `.spec.replicas`) is validated to ensure that it contains non-negative integer value and is not empty.
159+
160+
- The value at the optional JSON Path `StatusReplicasPath` (e.g. `.status.replicas`) is validated to be a an integer number if it exists (i.e. this can be empty).
161+
162+
- The value at the optional JSON Path `LabelSelectorPath` (e.g. `.spec.labelSelector`) is validated to be a valid label selector if it exists (i.e. this can be empty).
163+
If it is invalid, an empty `LabelSelector` is used.
164+
165+
### Status Behavior
166+
167+
If the `/status` subresource is enabled, the following behaviors change:
168+
169+
- The main resource endpoint will ignore all changes in the specified status subpath.
170+
(note: it will **not** reject requests which try to change the status, following the existing semantics of other resources).
171+
172+
- The `.metadata.generation` field is updated if and only if the value at the specified `SpecPath` (e.g. `.spec`) changes.
173+
Additionally, if the spec does not change, `.metadata.generation` is not updated.
174+
175+
- The `/status` subresource receives a full resource object, but only considers the value at the specified `StatusPath` subpath (e.g. `.status`) for the update.
176+
The value at the `.metadata` subpath is **not** considered for update as decided in https://github.com/kubernetes/kubernetes/issues/45539.
177+
178+
Both the status and the spec (and everything else if there is anything) of the object share the same key in the storage layer, i.e. the value at `.metadata.resourceVersion` is increased for any kind of change. There is no split of status and spec in the storage layer.
179+
180+
### Scale Behavior
181+
182+
The number of CustomResources can be easily scaled up or down depending on the replicas field present in path specified by `SpecPath`.
183+
184+
Only `ScaleSpec.Replicas` can be written. All other values are read-only and changes will be ignored. i.e. upon updating the scale subresource, two fields are modified:
185+
186+
1. The replicas field is copied back from the `Scale` object to the main resource as specified by `SpecReplicasPath` in the CRD, e.g. `.spec.replicas = scale.Spec.Replicas`.
187+
188+
2. The resource version is copied back from the `Scale` object to the main resource before writing to the storage: `.metadata.resourceVersion = scale.ResourceVersion`.
189+
In other words, the scale and the CustomResource share the resource version used for optimistic concurrency.
190+
Updates with outdated resource versions are rejected with a conflict error, read requests will return the resource version of the CustomResource.
191+
192+
#### Status Replicas Behavior
193+
194+
As only the `scale.Spec.Replicas` field is to be written to by the CR user, the user-provided controller (not any generic CRD controller) counts its children and then updates the controlled object by writing to the `/status` subresource, i.e. the `scale.Status.Replicas` field is read-only.
195+
196+
#### Selector Behavior
197+
198+
`CustomResourceSubResourceScale.LabelSelectorPath` is the label selector over CustomResources that should match the replicas count.
199+
The value in the `Scale` object is one-to-one the value from the CustomResource if the label selector is non-empty.
200+
Intentionally we do not default it to another value from the CustomResource (e.g. `.spec.template.metadata.labels`) as this turned out to cause trouble (e.g. in `kubectl apply`) and it is generally seen as a wrong approach with existing resources.
201+
202+
## Implementation Plan
203+
204+
The `/scale` and `/status` subresources are mostly distinct. It is proposed to do the implementation in two phases (the order does not matter much):
205+
206+
1. `/status` subresource
207+
2. `/scale` subresource
208+
209+
## Alternatives
210+
211+
### Scope
212+
213+
In this proposal we opted for an opinionated concept of subresources i.e. we restrict the subresource spec to the two very specific subresources: `/status` and `/scale`.
214+
We do not aim for a more generic subresource concept. In Kubernetes there are a number of other subresources like `/log`, `/exec`, `/bind`. But their semantics is much more special than `/status` and `/scale`.
215+
Hence, we decided to leave those other subresources to the domain of User provided API Server (UAS) instead of inventing a more complex subresource concept for CustomResourceDefinitions.
216+
217+
**Note**: that the types do not make the addition of other subresources impossible in the future.
218+
219+
We also restrict the JSON path for the status and the spec within the CustomResource.
220+
We could make them definable by the user and the proposed types actually allow us to open this up in the future.
221+
For the time being we decided to be opinionated as all status and spec subobjects in existing types live under `.status` and `.spec`. Keeping this pattern imposes consistency on user provided CustomResources as well.

0 commit comments

Comments
 (0)