Add cluster total CR limit, explain CR per CRD limits in more detail

Author: jpbetz

keps/sig-api-machinery/20180415-crds-to-ga.md

@@ -170,13 +170,53 @@ The targets are defined by the below suggested maximum limits, which are organiz
 
 - Since custom resources can be arbitrarily large, we have broken down the limit by custom resource object size.
 
+**Custom Resource Definitions:**
+
+| Suggested Maximum Limit: scope=cluster |
+| --- |
+| 500 |
+
+_Note: The Custom Resource Definition suggested maximum limit was selected not
+due to the above SLI/SLOs, but instead due to the latency OpenAPI publishing,
+which is a background process that occurs asychroniously each time a Custom
+Resource Definition schema is updated. For 500 Custom Resource Definitions it takes
+slightly over 35 seconds for a definition change to be visible via the OpenAPI
+spec endpoint._
+
+**Custom Resources, Cluster Wide:**
+
+Cluster wide limits for custom resources are storage bound and custom resources
+share the storage space with all other objects. While determining the
+appropriate storage limit for a cluster is out-of-scope for this document, once
+a etcd storage limit selected, suggested maximum limits for custom resources
+are:
+
+| etcd storage limit | Suggested Maximum Limit: scope=cluster |
+| --- | --- |
+| 4GB | 40000 |
+| 8GB | 80000 |
+
+These limits aim to keep custom resource storage usage to less than half of the
+total cluster storage capacity for custom resources of 50kb or less in size.
+
 **Custom Resources per Definition:**
 
+For each custom resource definition, the limit on the number of custom resources
+can be found by taking the (median) object size of the custom resource and finding
+the the matching row in this table:
+
 | Object size | Suggested Maximum Limit: scope=namespace (5s p99 SLO) | Suggested Maximum Limit: scope=cluster (30s p99 SLO) |
 | --- | --- | --- |
-| 10kb | 1500 | 10000 |
-| 25kb | 600 | 4000 |
-| 50kb | 300 | 2000 |
+| <=10kb | 1500 | 10000 |
+| (10kb - 25kb] | 600 | 4000 |
+| (25kb - 50kb] | 300 | 2000 |
+
+The cluster scope indicates the total number of custom resources for that
+definition allowed in the entire cluster.
+
+The namespace scope indicates the total number of custom resources for that
+definition allowed in any particular namespace. The cumulative count of the
+custom resource across all namespaces must not exceed the cluster limit.
 
 Since, in practice, custom resources scale farther without conversion webhooks
 within the SLI/SLOs (roughly 2x according to our scale tests), custom resource
@@ -190,19 +230,6 @@ and the scope=cluster suggested maximum limit indicates how many custom resource
 be in the cluster total. For custom resources of custom resource definitions using `scope: Cluster`: only
 the scope=cluster suggested maximum limit applies._
 
-**Custom Resource Definitions:**
-
-| Suggested Maximum Limit: scope=cluster |
-| --- |
-| 500 |
-
-_Note: The Custom Resource Definition suggested maximum limit was selected not
-due to the above SLI/SLOs, but instead due to the latency OpenAPI publishing,
-which is a background process that occurs asychroniously each time a Custom
-Resource Definition schema is updated. For 500 Custom Resource Definitions it takes
-slightly over 35 seconds for a definition change to be visible via the OpenAPI
-spec endpoint._
-
 **Conversion Webhooks:**
 
 Conversion Webhook SLOs are defined from the perspective of the conversion
@@ -211,14 +238,17 @@ making the request to the webhook, but it does include network latency.
 
 Given that the performance and scalability of conversion webhooks are the
 responsibility of their author, Custom resource scale targets are applied only for
-conversion webhooks that are within the follow latencies for the above suggested
+conversion webhooks that are within the following latencies for the above suggested
 maximum limits.
 
-| scope | Expected conversion Webhook SLO: p99 latency |
+| scope | object count limit | Expected conversion Webhook SLO: p99 latency |
 | --- | --- |
-| resource | 50ms |
-| namespace | 1 seconds |
-| cluster | 6 seconds |
+| resource | 1 | 50ms |
+| namespace | 1500 (<=10kb), 600 (10-25kb) or 300 (25-50kb) | 1 seconds |
+| cluster | 10000 (<=10kb), 4000 (10-25kb) or 2000 (25-50kb) | 6 seconds |
+
+The scope=resource's higher "per-object" latency (50ms vs ~1.5ms for namespace
+and cluster scope) it to accommodate for a request serving cost constant.
 
 The above object size and suggested maximum limits in the Custom Resources per
 Definition table applies to these conversion webhook SLOs. For example, for a