Add KEP for volume scheduling limits

Author: jsafrane

keps/sig-storage/20190408-volume-scheduling-limits.md

@@ -0,0 +1,247 @@
+---
+title: KEP Template
+authors:
+  - "@jsafrane"
+owning-sig: sig-storage
+participating-sigs:
+  - sig-scheduling
+reviewers:
+  - "@bsalamat"
+  - "@gnufied"
+  - "@davidz627"
+approvers:
+  - "@bsalamat"
+  - "@davidz627"
+editor: TBD
+creation-date: 2019-04-08
+last-updated: 2019-04-08
+status: implementable
+see-also:
+ - https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20190129-csi-migration.md
+replaces: https://github.com/kubernetes/enhancements/pull/730
+superseded-by:
+---
+
+# Volume Scheduling Limits
+
+## Table of Contents
+
+- [Title](#title)
+  - [Table of Contents](#table-of-contents)
+  - [Release Signoff Checklist](#release-signoff-checklist)
+  - [Summary](#summary)
+  - [Motivation](#motivation)
+    - [Goals](#goals)
+    - [Non-Goals](#non-goals)
+  - [Proposal](#proposal)
+    - [User Stories [optional]](#user-stories-optional)
+      - [Story 1](#story-1)
+      - [Story 2](#story-2)
+    - [Implementation Details/Notes/Constraints [optional]](#implementation-detailsnotesconstraints-optional)
+    - [Risks and Mitigations](#risks-and-mitigations)
+  - [Design Details](#design-details)
+    - [Test Plan](#test-plan)
+    - [Graduation Criteria](#graduation-criteria)
+      - [Examples](#examples)
+        - [Alpha -> Beta Graduation](#alpha---beta-graduation)
+        - [Beta -> GA Graduation](#beta---ga-graduation)
+        - [Removing a deprecated flag](#removing-a-deprecated-flag)
+    - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
+    - [Version Skew Strategy](#version-skew-strategy)
+  - [Implementation History](#implementation-history)
+  - [Drawbacks [optional]](#drawbacks-optional)
+  - [Alternatives [optional]](#alternatives-optional)
+  - [Infrastructure Needed [optional]](#infrastructure-needed-optional)
+
+## Release Signoff Checklist
+
+- [ ] kubernetes/enhancements issue in release milestone, which links to KEP (this should be a link to the KEP location in kubernetes/enhancements, not the initial KEP PR)
+- [ ] KEP approvers have set the KEP status to `implementable`
+- [ ] Design details are appropriately documented
+- [ ] Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
+- [ ] Graduation criteria is in place
+- [ ] "Implementation History" section is up-to-date for milestone
+- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
+- [ ] Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
+
+**Note:** Any PRs to move a KEP to `implementable` or significant changes once it is marked `implementable` should be approved by each of the KEP approvers. If any of those approvers is no longer appropriate than changes to that list should be approved by the remaining approvers and/or the owning SIG (or SIG-arch for cross cutting KEPs).
+
+## Summary
+
+Number of volumes of certain type that can be attached to a node should be configurable easily and should be based on node type. This proposal implements dynamic attachable volume limits on a per-node basis rather than cluster global defaults that exist today. This proposal also implements a way of configuring volume limits for CSI volumes.
+
+This proposal replaces [#730](https://github.com/kubernetes/enhancements/pull/730) and integrates volume limits for in-tree volumes (AWS EBS, GCE PD, AZURE DD, OpenStack Cinder) and CSI into one predicate. As result, in-tree volumes and corresponding CSI driver can share the same volume limit. 
+
+## Motivation
+
+Current scheduler predicates for scheduling of pods with volumes is based on `node.status.capacity` and `node.status.allocatable`. It works well for hardcoded predicates for volume limits on AWS (`MaxEBSVolumeCount`), GCE(`MaxGCEPDVolumeCount`), Azure (`MaxAzureDiskVolumeCount`) and OpenStack (`MaxCinderVolumeCount`).
+
+It is problematic for CSI (`MaxCSIVolumeCountPred`) outlined in [#730](https://github.com/kubernetes/enhancements/pull/730)
+
+- `ResourceName` is limited to 63 characters. We must prefix `ResourceName` with unique string (such as `attachable-volumes-csi-<driver name>`) so it cannot collide with existing resources like `cpu` or `memory`. But `<driver name>` itself is up to 63 character long, so we ended up with using SHA-sums of driver name to keep the `ResourceName` unique, which is not user readable.
+- CSI driver cannot share its limits with in-tree volume plugin e.g. when running pods with AWS EBS in-tree volumes and `ebs.csi.aws.com` CSI driver on the same node.
+- `node.status` size increases with each installed CSI driver. Node objects is big enough already.
+
+### Goals
+
+- User can run use PVs both with in-tree volume plugins and CSI and they will share their limits. There is only one scheduler predicate that handles both kind of volumes.
+
+- Existing predicates for in-tree volumes `MaxEBSVolumeCount`, `MaxGCEPDVolumeCount`, `MaxAzureDiskVolumeCount` and `MaxCinderVolumeCount` are removed (with deprecation period).
+  - When both deprecated in-tree predicate and CSI predicate are enabled, only one of them does useful work and the other is NOOP to save CPU.
+
+- Increased CPU consumption as measured by [scheduler benchmark](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-scheduling/scheduler_benchmarking.md) is approved by sig-scheduling. There should be no performance regression in ideal case. 
+
+### Non-Goals
+
+
+## Proposal
+
+* Track volume limits for both in-tree volume plugins and CSI drivers in `CSINode` objects instead of `Node`.
+  * `Node` object is already big enough.
+  * Get rid of prefix + SHA for `ResourceName` of CSI volumes.
+  * In-tree volume plugin can share limits with CSI driver that uses the same storage backend.
+
+* Kubelet will create `CSINode` instance during initial node registration together with `Node` object.
+  * Limits of each in-tree volume plugin will be added to `CSINode.status.capacity` and `CSINode.status.allocatable`.
+    * Name of CSI driver corresponding to in-tree volume plugin will be used as `ResourceName` of these in-tree plugins, so the limits are shared between CSI driver and in-tree volume plugin in case both are running on the same node.
+    * If a CSI driver is registered for an in-tre volume plugin and it reports a different volume limit than in-tree volume plugin, the limit reported by CSI driver is used. (TODO: should kubelet `exit()`?)
+  * Kubelet will reconcile `CSINode.capacity`, overwriting any user's changes.
+  * User may change `CSINode.allocatable` to override volume plugin / CSI driver values, e.g. to "reserve" some attachment to the operating system.
+  * Kubelet will continue filling `Node.status.allocatable` and `Node.status.capacity` for both in-tree and CSI volumes during deprecation period. After the deprecation period, it will stop using them completely.
+    * User changes of `Node.status.allocatable` and `Node.status.capacity` will be ignored if `CSINode.status.allocatable` or `Node.status.capacity` is present.
+    * To support old kubelet, `MaxCSIVolumeCountPred` (or any deprecated volume limit predicate) falls back to `Node.status.allocatable` / `capacity` when `CSINode` does not contain any limits for a volume plugin.
+
+  CSINode example:
+
+  ```
+  apiVersion: storage.k8s.io/v1beta1
+  kind: CSINode
+  metadata:
+    name: ip-172-18-4-112.ec2.internal
+  spec:
+  status:
+    capacity:
+      "ebs.csi.aws.com": "39"
+    allocatable:
+      "ebs.csi.aws.com": "39"
+  ```
+
+* Existing scheduler predicates `MaxEBSVolumeCount`, `MaxGCEPDVolumeCount`, `MaxAzureDiskVolumeCount` and `MaxCinderVolumeCount` are already deprecated.
+  * If any of them is enabled together with `MaxCSIVolumeCountPred`, the deprecated predicate will do nothing (`MaxCSIVolumeCountPred` does the job of counting both in-tree and CSI volumes).
+  * The deprecated predicates will filter pods only when `MaxCSIVolumeCountPred` predicate is disabled.
+  * This way, we save CPU by running only one volume limit predicate during deprecation period.
+
+
+### New API
+
+TODO
+
+### User Stories
+
+TODO: fill? Is there any interesting user story?
+
+#### Story 1
+
+#### Story 2
+
+### Implementation Details/Notes/Constraints
+
+[CSI migration library](https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/csi-translation-lib) is used to find CSI driver name for in-tree volume plugins. This CSI driver name is used as key in `CSINode.status.capacity` and `CSINode.status.allocatable` lists.
+
+### Risks and Mitigations
+
+* This KEP depends on [CSI migration library](https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/csi-translation-lib). It can happen that CSI migration is redesigned / cancelled.
+  * Countermeasure: [CSI migration](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20190129-csi-migration.md) and this KEP should graduate together.
+
+## Design Details
+
+Existing feature gate `AttachVolumeLimit` will be re-used for implementation of this KEP. The feature is already beta and is enabled by default.
+
+### Test Plan
+
+**Note:** *Section not required until targeted at a release.*
+
+Consider the following in developing a test plan for this enhancement:
+- Will there be e2e and integration tests, in addition to unit tests?
+- How will it be tested in isolation vs with other components?
+
+No need to outline all of the test cases, just the general strategy.
+Anything that would count as tricky in the implementation and anything particularly challenging to test should be called out.
+
+All code is expected to have adequate tests (eventually with coverage expectations).
+Please adhere to the [Kubernetes testing guidelines][testing-guidelines] when drafting this test plan.
+
+[testing-guidelines]: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
+
+### Graduation Criteria
+
+**Note:** *Section not required until targeted at a release.*
+
+Define graduation milestones.
+
+These may be defined in terms of API maturity, or as something else. Initial KEP should keep
+this high-level with a focus on what signals will be looked at to determine graduation.
+
+Consider the following in developing the graduation criteria for this enhancement:
+- [Maturity levels (`alpha`, `beta`, `stable`)][maturity-levels]
+- [Deprecation policy][deprecation-policy]
+
+Clearly define what graduation means by either linking to the [API doc definition](https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-versioning),
+or by redefining what graduation means.
+
+In general, we try to use the same stages (alpha, beta, GA), regardless how the functionality is accessed.
+
+[maturity-levels]: https://git.k8s.io/community/contributors/devel/sig-architecture/api_changes.md#alpha-beta-and-stable-versions
+[deprecation-policy]: https://kubernetes.io/docs/reference/using-api/deprecation-policy/
+
+#### Examples
+
+TODO
+
+##### Alpha -> Beta Graduation
+
+N/A (`AttachVolumeLimit` feature is already beta).
+
+##### Beta -> GA Graduation
+
+TODO 
+
+##### Removing a deprecated flag
+
+- Announce deprecation and support policy of the existing flag
+- Two versions passed since introducing the functionality which deprecates the flag (to address version skew)
+- Address feedback on usage/changed behavior, provided on GitHub issues
+- Deprecate the flag
+
+**For non-optional features moving to GA, the graduation criteria must include [conformance tests].**
+
+[conformance tests]: https://github.com/kubernetes/community/blob/master/contributors/devel/conformance-tests.md
+
+### Upgrade / Downgrade Strategy
+
+TODO!
+
+* New scheduler and old kubelet / kubelet with `AttachVolumeLimit` feature disabled:
+  * `CSINode` does not contain `CSINode.status`: scheduler must fall back to `Node.status`.
+
+
+### Version Skew Strategy
+
+If applicable, how will the component handle version skew with other components? What are the guarantees? Make sure
+this is in the test plan.
+
+Consider the following in developing a version skew strategy for this enhancement:
+- Does this enhancement involve coordinating behavior in the control plane and in the kubelet? How does an n-2 kubelet without this feature available behave when this feature is used?
+- Will any other components on the node change? For example, changes to CSI, CRI or CNI may require updating that component before the kubelet.
+
+## Implementation History
+
+Major milestones in the life cycle of a KEP should be tracked in `Implementation History`.
+Major milestones might include
+
+- the `Summary` and `Motivation` sections being merged signaling SIG acceptance
+- the `Proposal` section being merged signaling agreement on a proposed design
+- the date implementation started
+- the first Kubernetes release where an initial version of the KEP was available
+- the version of Kubernetes where the KEP graduated to general availability
+- when the KEP was retired or superseded