Skip to content

Commit c642c92

Browse files
nikhitanikhita
committed
Proposal: SubResources for CustomResources
1 parent 2754ac1 commit c642c92

File tree

1 file changed

+221
-0
lines changed

1 file changed

+221
-0
lines changed

‎contributors/design-proposals/customresources-subresources.md‎

Lines changed: 221 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,221 @@
1+
# Subresources for CustomResources
2+
3+
Authors: @nikhita, @sttts
4+
5+
## Table of Contents
6+
7+
1. [Abstract](#abstract)
8+
2. [Goals](#goals)
9+
3. [Non-Goals](#non-goals)
10+
4. [Proposed Extension of CustomResourceDefinition](#proposed-extension-of-customresourcedefinition)
11+
1. [API Types](#api-types)
12+
2. [Feature Gate](#feature-gate)
13+
5. [Semantics](#semantics)
14+
1. [Validation Behavior](#validation-behavior)
15+
1. [Status](#status)
16+
2. [Scale](#scale)
17+
2. [Status Behavior](#status-behavior)
18+
3. [Scale Behavior](#scale-behavior)
19+
1. [Status Replicas Behavior](#status-replicas-behavior)
20+
2. [Selector Behavior](#selector-behavior)
21+
4. [Implementation Plan](#implementation-plan)
22+
5. [Alternatives](#alternatives)
23+
1. [Scope](#scope)
24+
25+
## Abstract
26+
27+
[CustomResourceDefinitions](https://github.com/kubernetes/community/pull/524) (CRDs) were introduced in 1.7. The objects defined by CRDs are called CustomResources (CRs). Currently, we do not provide subresources for CRs.
28+
29+
However, it is one of the [most requested features](https://github.com/kubernetes/kubernetes/issues/38113) and this proposal seeks to add `/status` and `/scale` subresources for CustomResources.
30+
31+
## Goals
32+
33+
1. Support status/spec split for CustomResources:
34+
1. Status changes are ignored on the main resource endpoint.
35+
2. Support a `/status` subresource HTTP path for status changes.
36+
3. `metadata.Generation` is increased only on spec changes.
37+
2. Support a `/scale` subresource for CustomResources.
38+
3. Maintain backward compatibility by allowing CRDs to opt-in to enable subresources.
39+
4. If a CustomResource is already structured using spec/status, allow it to easily transition to use the `/status` and `/scale` endpoint.
40+
5. Work seamlessly with [JSON Schema validation](https://github.com/kubernetes/community/pull/708).
41+
42+
## Non-Goals
43+
44+
1. Allow defining arbitrary subresources i.e. subresources except `/status` and `/scale`.
45+
2. Unify the many `Scale` types: as long as there is no generic `Scale` object in Kubernetes, we will propose to introduce yet another `Scale` type in the `apiextensions.k8s.io` api group. If Kubernetes switches to a generic `Scale` object, `apiextensions.k8s.io` will follow.
46+
47+
## Proposed Extension of CustomResourceDefinition
48+
49+
### API Types
50+
51+
The addition of the following types in `apiextensions.k8s.io/v1beta1` is proposed:
52+
53+
```go
54+
type CustomResourceDefinitionSpec struct {
55+
Group string
56+
Version string
57+
Names CustomResourceDefintion
58+
Scope ResourceScope
59+
Validation *CustomResourceValidation
60+
// SubResources describes the subresources for CustomResources
61+
// This field is alpha-level and should only be sent to servers that enable
62+
// subresources via the CurstomResourceSubResources feature gate.
63+
// +optional
64+
SubResources *CustomResourceSubResources `json:“subResources,omitempty”`
65+
}
66+
67+
// CustomResourceSubResources defines the status and scale subresources for CustomResources.
68+
type CustomResourceSubResources {
69+
// Status denotes the status subresource for CustomResources
70+
Status *CustomResourceSubResourceStatus `json:“status,omitempty”`
71+
// Scale denotes the scale subresource for CustomResources
72+
Scale *CustomResourceSubResourceScale `json:“scale,omitempty”`
73+
}
74+
75+
// CustomResourceSubResourceStatus defines how to serve the HTTP path <CR Name>/status.
76+
type CustomResourceSubResourceStatus struct {
77+
// The JSON path (e.g. “.status”) of the status of a CustomResource.
78+
// The whole object is transferred over the wire, but only the status can be mutated via the /status subresource.
79+
// StatusPath is restricted to be “.status” and defaults to “.status”.
80+
StatusPath string `json:“statusPath,omitempty”`
81+
// The JSON path (e.g. “.spec”) of the spec of a CustomResource.
82+
// SpecPath is restricted to be “.spec” and defaults to “.spec”.
83+
// Changes to the specified JSON path increase the “.metadata.generation” value.
84+
SpecPath string `json:“specPath,omitempty”`
85+
}
86+
87+
// CustomResourceSubResourceScale defines how to serve the HTTP path <CR name>/scale.
88+
type CustomResourceSubResourceScale struct {
89+
// required, e.g. “.spec.replicas”.
90+
// Only JSON paths without the array notation are allowed.
91+
SpecReplicasPath string `json:“specReplicasPath,omitempty”`
92+
// optional, e.g. “.status.replicas”.
93+
// Only JSON paths without the array notation are allowed.
94+
StatusReplicasPath string `json:“statusReplicasPath,omitempty”`
95+
// optional, e.g. “.spec.labelSelector”.
96+
// Only JSON paths without the array notation are allowed.
97+
LabelSelectorPath string `json:“labelSelectorPath,omitempty”`
98+
}
99+
100+
// The following is the payload to send over the wire for /scale. It happens
101+
// to be defined here in apiextensions.k8s.io because we don’t have a global
102+
// Scale type in meta/v1. Ref: https://github.com/kubernetes/kubernetes/issues/49504
103+
104+
// Scale represents a scaling request for a resource.
105+
type Scale struct {
106+
metav1.TypeMeta `json:",inline"`
107+
// Standard object metadata; More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata.
108+
// +optional
109+
metav1.ObjectMeta `json:"metadata,omitempty"`
110+
111+
// defines the behavior of the scale. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status.
112+
// +optional
113+
Spec ScaleSpec `json:"spec,omitempty"`
114+
115+
// current status of the scale. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status. Read-only.
116+
// +optional
117+
Status ScaleStatus `json:"status,omitempty"`
118+
}
119+
120+
// ScaleSpec describes the attributes of a scale subresource.
121+
type ScaleSpec struct {
122+
// desired number of instances for the scaled object.
123+
// +optional
124+
Replicas int32 `json:"replicas,omitempty"`
125+
}
126+
127+
// ScaleStatus represents the current status of a scale subresource.
128+
type ScaleStatus struct {
129+
// actual number of observed instances of the scaled object.
130+
Replicas int32 `json:"replicas"`
131+
132+
// label query over pods that should match the replicas count.
133+
// More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors
134+
// +optional
135+
Selector *metav1.LabelSelector `json:"selector"`
136+
}
137+
```
138+
139+
### Feature Gate
140+
141+
The `SubResources` field in `CustomResourceDefinitionSpec` will be gated under the `CustomResourceSubResources` alpha feature gate.
142+
If the gate is not open, the value of the new field within `CustomResourceDefinitionSpec` is rejected on creation and updates of CRDs.
143+
144+
## Semantics
145+
146+
### Validation Behavior
147+
148+
#### Status
149+
150+
The status endpoint of a CustomResource receives a full CR object. Changes outside of the status path are ignored.
151+
For validation everything outside of the JSON Path `StatusPath` can be reset to the existing values of the CustomResource. Then the JSON Schema presented in the CRD is validated against the whole object.
152+
153+
Note: one could extract the status suboject, filter the JSON schema by rules applied to the status and then validate the status subobject only.
154+
The filter step of the JSON schema though is not obvious in the case that there are status validations not below a properties JSON Schema construct. For that reason, the alternative implementation is preferred as its semantics are much simpler.
155+
156+
#### Scale
157+
158+
Moreover, if the scale subresource is enabled:
159+
160+
- The value at the specified JSON Path `SpecReplicasPath` (e.g. `.spec.replicas`) is validated to ensure that it contains non-negative integer value and is not empty.
161+
162+
- The value at the optional JSON Path `StatusReplicasPath` (e.g. `.status.replicas`) is validated to be a an integer number if it exists (i.e. this can be empty).
163+
164+
- The value at the optional JSON Path `LabelSelectorPath` (e.g. `.spec.labelSelector`) is validated to be a valid label selector if it exists (i.e. this can be empty).
165+
166+
### Status Behavior
167+
168+
If the `/status` subresource is enabled, the following behaviors change:
169+
170+
- The main resource endpoint will ignore all changes in the specified status subpath.
171+
(note: it will **not** reject requests which try to change the status, following the existing semantics of other resources).
172+
173+
- The `.metadata.generation` field is updated if and only if the value at the specified `SpecPath` (e.g. `.spec`) changes.
174+
175+
- The `/status` subresource receives a full resource object, but only considers the value at the specified `StatusPath` subpath (e.g. `.status`) for the update.
176+
The value at the `.metadata` subpath is **not** considered for update as decided in https://github.com/kubernetes/kubernetes/issues/45539.
177+
178+
Both the status and the spec (and everything else if there is anything) of the object share the same key in the storage layer, i.e. the value at `.metadata.resourceVersion` is increased for any kind of change. There is no split of status and spec in the storage layer.
179+
180+
### Scale Behavior
181+
182+
The number of CustomResources can be easily scaled up or down depending on the replicas field present in path specified by `SpecPath`.
183+
184+
Only `ScaleSpec.Replicas` can be written. All other values are read-only and changes will be ignored. i.e. upon updating the scale subresource, two fields are modified:
185+
186+
1. The replicas field is copied back from the `Scale` object to the main resource as specified by `SpecReplicasPath` in the CRD, e.g. `.spec.replicas = scale.Spec.Replicas`.
187+
188+
2. The resource version is copied back from the `Scale` object to the main resource before writing to the storage: `.metadata.resourceVersion = scale.ResourceVersion`.
189+
In other words, the scale and the CustomResource share the resource version used for optimistic concurrency.
190+
Updates with outdated resource versions are rejected with a conflict error, read requests will return the resource version of the CustomResource.
191+
192+
#### Status Replicas Behavior
193+
194+
As only the `scale.Spec.Replicas` field is to be written to by the CR user, the user-provided controller (not any generic CRD controller) counts its children and then updates the controlled object by writing to the `/status` subresource, i.e. the `scale.Status.Replicas` field is read-only.
195+
196+
#### Selector Behavior
197+
198+
`CustomResourceSubResourceScale.LabelSelectorPath` is the label selector over CustomResources that should match the replicas count.
199+
The value in the `Scale` object is one-to-one the value from the CustomResource if the label selector is non-empty.
200+
Intentionally we do not default it to another value from the CustomResource (e.g. `.spec.template.metadata.labels`) as this turned out to cause trouble (e.g. in `kubectl apply`) and it is generally seen as a wrong approach with existing resources.
201+
202+
## Implementation Plan
203+
204+
The `/scale` and `/status` subresources are mostly distinct. It is proposed to do the implementation in two phases (the order does not matter much):
205+
206+
1. `/status` subresource
207+
2. `/scale` subresource
208+
209+
## Alternatives
210+
211+
### Scope
212+
213+
In this proposal we opted for an opinionated concept of subresources i.e. we restrict the subresource spec to the two very specific subresources: `/status` and `/scale`.
214+
We do not aim for a more generic subresource concept. In Kubernetes there are a number of other subresources like `/log`, `/exec`, `/bind`. But their semantics is much more special than `/status` and `/scale`.
215+
Hence, we decided to leave those other subresources to the domain of User provided API Server (UAS) instead of inventing a more complex subresource concept for CustomResourceDefinitions.
216+
217+
**Note**: that the types do not make the addition of other subresources impossible in the future.
218+
219+
We also restrict the JSON path for the status and the spec within the CustomResource.
220+
We could make them definable by the user and the proposed types actually allow us to open this up in the future.
221+
For the time being we decided to be opinionated as all status and spec subobjects in existing types live under `.status` and `.spec`. Keeping this pattern imposes consistency on user provided CustomResources as well.

0 commit comments

Comments
 (0)