Skip to content

Commit a8ad9ca

Browse files
committed
add blog document for feature ServiceNodePortStaticSubrange
1 parent 5a97479 commit a8ad9ca

File tree

1 file changed

+152
-0
lines changed

1 file changed

+152
-0
lines changed
Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
---
2+
layout: blog
3+
title: "Kubernetes 1.27: Avoid Collisions Assigning ports to Nodeport Services"
4+
date: 2023-03-08
5+
slug: nodeport-dynamic-and-static-allocation
6+
---
7+
8+
**Author:** xuzhenglun (Alibaba)
9+
10+
In Kubernetes, a Service can be used to provide a unified traffic endpoint for
11+
applications running on a set of Pods. Clients can use the virtual IP address provided
12+
by the Service for access, and Kubernetes provides load balancing for traffic accessing
13+
different back-end Pods, but a ClusterIP type of Service is limited to providing access to
14+
nodes within the cluster, while traffic from outside the cluster cannot be routed.
15+
One way to solve this problem is to use a NodePort type of Service, which maps a VIP's port
16+
to a specific port of all nodes in the cluster, thus redirecting traffic from the
17+
outside to the inside of the cluster.
18+
19+
## How NodePort's ports are allocated?
20+
21+
When a NodePort Service is created, its corresponding port is allocated in two ways.
22+
23+
**Dynamic** : If the Service type is NodePort and the corresponding `service.spec.ports.nodePort`
24+
is not specified, the Kubernetes controlplane will automatically allocate an unused port
25+
to it at creation time.
26+
27+
**Static** : In addition to the dynamic auto-assignment described above, you can also
28+
explicitly assign a port that is within the nodeport port range configuration.
29+
30+
the `service.spec.ports.nodePort` must be unique for each NodePort Service across the cluster.
31+
Attempting to create a NodePort type of Service with a port already allocated will return an error.
32+
33+
## Why do you need to reserve ports of NodePort Service?
34+
Sometimes, you may want to have a NodePort Service running on well-known ports
35+
so that other components and users inside and outside the cluster can use them.
36+
37+
In some complex cluster deployments with a mix of K8S nodes and non-K8S nodes, it may be
38+
necessary to rely on some pre-defined port for communicating. In particular, some fundamental
39+
components cannot rely on the VIPs which provided by LoadBalancer services,
40+
because the load balancing itself that provides the VIPs may also rely on these fundamental components.
41+
42+
Now suppose we need to expose a Minio object storage service on K8S to services located on non-K8S nodes,
43+
and the agreed port is `30009`, we need to create a Service as follows:
44+
45+
```yaml
46+
apiVersion: v1
47+
kind: Service
48+
metadata:
49+
labels:
50+
k8s-app: minio
51+
name: minio
52+
namespace: kube-system
53+
spec:
54+
ports:
55+
- name: http
56+
nodePort: 30009
57+
port: 9000
58+
protocol: TCP
59+
targetPort: 9000
60+
selector:
61+
app: minio
62+
type: NodePort
63+
```
64+
65+
However, as mentioned before, if the port `30009` required for the minio Service is not reserved,
66+
and another NodePort Service is created and dynamically allocated before or concurrently with the minio
67+
Service, `30009` can be allocated to those NodePort Service, while the minio Service will fail to be
68+
created due to a port conflict.
69+
70+
## How can you avoid NodePort Service port conflicts?
71+
In the Kubernetes 1.24, Service ClusterIP had already divided the CIDRs into two blocks, using different
72+
allocation policies to reduce the risk of conflicts. In Kubernetes 1.27, a similar policy can be adopted
73+
for NodePort Services. You can enable a new feature gate `ServiceNodePortStaticSubrange`. Turning this on allows you
74+
to use a different port allocation strategy for NodePort Services, and reducing the risk of collision.
75+
76+
The port range for `NodePort` will be divided, based on the formula `min(max(16, nodeport-size / 32), 128)`,
77+
which can be described as _never less than 16 or more than 128 with a graduated step between them_.
78+
79+
Dynamic port assignment will use the upper band by default, once this has been exhausted it will use the lower range.
80+
This will allow users to use static allocations on the lower band with a low risk of collision.
81+
82+
## Examples:
83+
84+
### default range: 30000-32767
85+
service-node-port-range: 30000-32767
86+
Band Offset: `min(max(16, 2768/32), 128)` = `min(max(16, 86), 128)` = `min(86, 128)` = 86
87+
Static band start: 30000
88+
Static band end: 30085
89+
Range end: 32767
90+
91+
{{< mermaid >}}
92+
pie showData
93+
title 30000-32767
94+
"Static" : 86
95+
"Dynamic" : 2682
96+
{{< /mermaid >}}
97+
98+
### very small range: 30000-30015
99+
service-node-port-range: 30000-30015
100+
Band Offset: 0
101+
Static band start: 30000
102+
Static band end: 30000
103+
Range end: 30015
104+
105+
{{< mermaid >}}
106+
pie showData
107+
title 30000-30015
108+
"Static" : 0
109+
"Dynamic" : 16
110+
{{< /mermaid >}}
111+
112+
### small(lower boundary) range: 30000-30127
113+
service-node-port-range: 30000-30127
114+
Band Offset: `min(max(16, 128/32), 128)` = `min(max(16, 4), 128)` = `min(16, 128)` = 16
115+
Static band start: 30000
116+
Static band end: 30015
117+
Range end: 30127
118+
119+
{{< mermaid >}}
120+
pie showData
121+
title 30000-30127
122+
"Static" : 16
123+
"Dynamic" : 112
124+
{{< /mermaid >}}
125+
126+
### large(upper boundary) range: 30000-34095
127+
service-node-port-range: 30000-34095
128+
Band Offset: `min(max(16, 4096/32), 128)` = `min(max(16, 128), 128)` = `min(128, 128)` = 128
129+
Static band start: 30000
130+
Static band end: 30127
131+
Range end: 34095
132+
133+
{{< mermaid >}}
134+
pie showData
135+
title 30000-34095
136+
"Static" : 128
137+
"Dynamic" : 3968
138+
{{< /mermaid >}}
139+
140+
### very large range: 30000-38191
141+
service-node-port-range: 30000-38191
142+
Band Offset: `min(max(16, 8192/32), 128)` = `min(max(16, 256), 128)` =` min(256, 128)` = 128
143+
Static band start: 30000
144+
Static band end: 30127
145+
Range end: 38191
146+
147+
{{< mermaid >}}
148+
pie showData
149+
title 30000-38191
150+
"Static" : 128
151+
"Dynamic" : 3968
152+
{{< /mermaid >}}

0 commit comments

Comments
 (0)