CustomResourceDefinitions

[deads2k]

Enhancement Description

• One-line enhancement description (can be used as a release note): Custom Resource Definitions allow defining kubernetes-style APIs as add-ons
• Kubernetes Enhancement Proposal:
  • https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/20180415-crds-to-ga.md
• See also:
  • Moving ThirdPartyResources to beta
  • Versioning
  • Validation
  • Subresources
• Primary contact (assignee): @sttts, @jpbetz
• Responsible SIGs: api-machinery
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): Already released
  • Beta release target (x.y): v1.7
  • Stable release target (x.y): v1.16

Scope of work planned for v1.15

• Define allowed OpenAPI subset (sig-api-machinery KEP: Vanilla OpenAPI subset: structural schema #1002, Publish CRD OpenAPI Schema #692)
• Define and perform scale testing for CRD (sig-api-machinery: Add scale targets to CRDs to GA KEP #1015)
• Bring CRD conversion webhooks to beta (sig-api-machinery: port CRD conversion webhook design doc to KEP, add v1beta1 criteria #1004, Webhook Conversion for Custom Resources #598)

Scope of work planned for v1.11

• Implement first steps of CRD versioning. This has another features issue: Custom Resource Definition Versioning #544. Design proposal: CRD versioning design proposal community#2026.
• Add pruning and defaulting for custom resources.
• Support maps in the CRD validation schema. CustomResourceDefinitions should support maps in OpenAPI schema kubernetes#59485
• Implement quota support for custom resources (this can be considered as a bug fix too) quota support for custom resources kubernetes#53777

Scope of work planned for v1.10

• Finish CRD sub-resources as alpha Proposal: SubResources for CustomResources community#913
• Support categories for CRDs.

Scope of work planned for v1.9

• CRD validation to beta Requirements for ThirdPartyResource to graduate to beta kubernetes#22768 Getting CRD Validation to Beta kubernetes#53829

Scope of work planned for v1.8

• Remove deprecated ThirdPartyResource API.
• Add validation and defaulting for CustomResourceDefinition.
• Add subresources for CustomResourceDefinition.
  • Support Spec/Status split (/status subresource) on custom resources.
  • Support incrementing object Generation on custom resource data mutation (requires Spec/Status split).
• Support OwnerReference-based garbage collection with CRD.

Scope of work planned for v1.7

• Move TPR to a new API group (tentatively called apiextensions) to support deprecation of the extensions group
  • Ideally, implement the new TPR in a separate API server, to be integrated into kube-apiserver via API Aggregation.
• For now, only allow 1 version at a time per TPR. In the absence of conversion (which is out of scope for this release), this is necessary to remain consistent with the expectations of other components.
  • Support for multiple versions could be added (with or without conversion) in a later release.
• Fix name conflicts due to lossy conversion of TPR name into resource/kind.
• Allow TPRs to specify their own names for resources and kinds, rather than tying them to the TPR name.
• Allow TPRs to register short names that will be discoverable by kubectl.
• Allow TPRs to optionally be cluster-scoped rather than namespaced.
• Define and document a process to migrate from extensions/v1beta1 TPR, possibly requiring brief downtime for TPR custom controllers and operators.
  • Where possible, provide automated tools to help with migration.
• A finalizer ensures CR data is deleted if a CRD is deleted.
• Fix TPR/CRD data cleanup upon namespace deletion for the 3rd time, this time with a regression test.

Other plans not in scope for this release

• Support multiple versions at the same time for a given TPR.
  • Other components (e.g. GC, namespace finalizers) expect automatic conversion. TPR currently does not support that.
  • Note that it's possible to change the single registered version of a TPR, but it requires brief downtime for TPR custom controllers and operators.
  • The extensions/v1beta1 TPR gives the appearance of supporting multiple versions, but multiple version support was never implemented.
• Support customizing where TPR APIs appear in discovery, relative to other TPRs or other APIs.
• Support namespace-scoped CRD whose CRs are only visible in one namespace.

Plans with unclear status

Still investigating or TBD. Please comment/edit with any updates.

• Improve the display of TPRs in kubectl/dashboard.
  • There may be other feature trackers addressing this.

[deads2k]

@lavalamp I've created this to try to have a place where we can at least consolidate our thoughts and track progress on third party resources. I've tried to create a list of known shortcomings to be resolved before promotion to stable.

I don't have an owner in mind, but recognition of the problem seems like step 1.

[adohe-zz]

@deads2k I am learning third party resource recently, also wish to help with something.

[deads2k]

@deads2k I am learning third party resource recently, also wish to help with something.

I've re-ordered the list in terms of what I see as tactical priority. People are trying to use this now and these problems will burn them badly.

If you're comfortable taking the "multiple resources" item, that would be a great start. You could create a separate issue and we can talk about implementation in there.

[adohe-zz]

@deads2k I spent some time trying to reproduce the first issue:

Multiple Resources, single version, different add times - Adding resource A, waiting for it to appear, then adding resource B fails. Resource B is never added.

but with unluck. Below is my reproduce steps:

1. create a custom thirdparty resource&wait it to appear

[root@localhost kubernetes]# cat /home/tony/Desktop/debug/lbclaim.yaml
kind: ThirdPartyResource
apiVersion: extensions/v1beta1
metadata:
  name: loadbalancerclaim.k8s.io
description: "Allow user to claim a loadbalancer instance"
versions:
- name: v1
[root@localhost kubernetes]# kc create -f /home/tony/Desktop/debug/lbclaim.yaml
thirdpartyresource "loadbalancerclaim.k8s.io" created
[root@localhost kubernetes]# curl  http://localhost:8080/apis/extensions/v1beta1/thirdpartyresources/
{
  "kind": "ThirdPartyResourceList",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "selfLink": "/apis/extensions/v1beta1/thirdpartyresources/",
    "resourceVersion": "170"
  },
  "items": [
    {
      "metadata": {
        "name": "loadbalancerclaim.k8s.io",
        "selfLink": "/apis/extensions/v1beta1/thirdpartyresources/loadbalancerclaim.k8s.io",
        "uid": "dcb88b3a-9857-11e6-a19b-08002767e1f5",
        "resourceVersion": "146",
        "creationTimestamp": "2016-10-22T13:03:01Z"
      },
      "description": "Allow user to claim a loadbalancer instance",
      "versions": [
        {
          "name": "v1"
        }
      ]
    }
  ]
}

1. after a moment(more than 10s), create another custom thirdparty resource

[root@localhost kubernetes]# cat /home/tony/Desktop/debug/loadbalancer.yaml
kind: ThirdPartyResource
apiVersion: extensions/v1beta1
metadata:
  name: loadbalancer.k8s.io
description: "Allow user to curd a loadbalancer instance"
versions:
- name: v1
[root@localhost kubernetes]# kc create -f /home/tony/Desktop/debug/loadbalancer.yaml
thirdpartyresource "loadbalancer.k8s.io" created

1. verify both resources exist

[root@localhost kubernetes]# curl http://localhost:8080/apis/k8s.io/v1/
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "k8s.io/v1",
  "resources": [
    {
      "name": "loadbalancerclaims",
      "namespaced": true,
      "kind": "Loadbalancerclaim"
    },
    {
      "name": "loadbalancers",
      "namespaced": true,
      "kind": "Loadbalancer"
    }
  ]
}
[root@localhost kubernetes]# kc get loadbalancers
No resources found.
[root@localhost kubernetes]# kc get loadbalancerclaims
No resources found.

seems we already support multiple resources, single version.

[adohe-zz]

And I take a deep look at TPR related code. The thirdparty_controller will do periodically sync(every 10 seconds), it will install every new TPR, and also do some deletion job. The ThirdPartyResourceServer contains all installed TPR mappings. As we can see from SyncOneResource and InstallThirdPartyResource, even this this group exists, it will still update the group with the new API.

[adohe-zz]

Also I found that I am able to delete a TPR schema def even there are TPR instances in the system. I think this should not be allowed.

[deads2k]

@deads2k I spent some time trying to reproduce the first issue:

Try to enable this test: https://github.com/kubernetes/kubernetes/blob/master/test/integration/thirdparty/thirdparty_test.go#L137 . If it works, we're good. If it fails, something is wrong.

[adohe-zz]

@deads2k Hi David, please take a look at the message I sent on Slack. Besides, I add a fix to the failed integration test, the third party resource controller will remove the corresponding routes handler when a TPR get deleted, this will help with the integration test, but I am not sure whether this will bring in any other problems.

[brendandburns]

For problem #1, it was fixed here:

kubernetes/kubernetes#28414

[adohe-zz]

@brendandburns actually not, you can run the comment out integration test, and it will fail.

[adohe-zz]

@brendandburns More correctly, we did support multiple resources, single version, but the deletion logical has some problem.

[brendandburns]

@adohe did you file an issue? I can take a look.