Skip to content

Commit c0494d3

Browse files
Add post-release blog about KEP-5707 (#54690)
* Placeholder for KEP-5707 blog * wip * Add Gateway API words * WIP * More words and examples * Address direct feedback * Remove unneeded host from HTTPRoute * Make example seem real * Marketing words * Remove FIXME * Review tweaks * Style changes * Add comment explaining the loadBalancerClass value * More review feedback * Address more review --------- Co-authored-by: Dan Winship <danwinship@redhat.com>
1 parent 0eb1163 commit c0494d3

1 file changed

Lines changed: 230 additions & 0 deletions

File tree

Lines changed: 230 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,230 @@
1+
---
2+
layout: blog
3+
title: "Kubernetes v1.36: Deprecation and removal of Service ExternalIPs"
4+
draft: true # will be changed to date: YYYY-MM-DD before publication
5+
slug: kubernetes-v1-36-deprecation-and-removal-of-service-externalips # optional
6+
author: >
7+
Adrian Moisey (independent),
8+
Dan Winship (Red Hat),
9+
---
10+
11+
The `.spec.externalIPs` field for [Service](/docs/concepts/services-networking/service/) was an early attempt to provide
12+
cloud-load-balancer-like functionality for non-cloud clusters.
13+
Unfortunately, the API assumes that every user in the cluster is fully
14+
trusted, and in any situation where that is not the case, it enables
15+
various security exploits, as described in
16+
[CVE-2020-8554](https://www.cvedetails.com/cve/CVE-2020-8554/).
17+
18+
Since Kubernetes 1.21, the Kubernetes project has recommended that all users disable
19+
`.spec.externalIPs`. To make that easier, Kubernetes also added an admission controller
20+
(`DenyServiceExternalIPs`) that can be enabled to do this. At the time,
21+
SIG Network felt that blocking the functionality by default was too large a
22+
breaking change to consider.
23+
24+
However, the security problems are still there, and as a project we're increasingly
25+
unhappy with the "insecure by default" state of the feature.
26+
Additionally, there are now several better alternatives for non-cloud
27+
clusters wanting load-balancer-like functionality.
28+
29+
As a result, the `.spec.externalIPs` field for Service is now formally deprecated in Kubernetes 1.36.
30+
We expect that a future minor release of Kubernetes will drop
31+
implementation of the behavior from `kube-proxy`, and will update the
32+
Kubernetes [conformance](https://www.cncf.io/training/certification/software-conformance/) criteria to require that conforming implementations
33+
**do not** provide support.
34+
35+
## A note on terminology, and what hasn't been deprecated {#terminology}
36+
37+
The phrase _external IP_ is somewhat overloaded in Kubernetes:
38+
39+
- The Service API has a field `.spec.externalIPs` that can be used
40+
to add additional IP addresses that a Service will respond on.
41+
42+
- The Node API's `.status.addresses` field can list addresses of
43+
several different types, one of which is called `ExternalIP`.
44+
45+
- The `kubectl` tool, when displaying information about a Service of type
46+
LoadBalancer in the default output format, will show the load
47+
balancer IP address under the column heading `EXTERNAL-IP`.
48+
49+
This deprecation is about the first of those. If you are not setting
50+
the field `externalIPs` in any of your Services, then it does not
51+
apply to you.
52+
53+
That said, as a precaution, you may still want to enable the [DenyServiceExternalIPs](/docs/reference/access-authn-authz/admission-controllers/#denyserviceexternalips) admission controller to
54+
block any future use of the `externalIPs` field.
55+
56+
## Alternatives to `externalIPs` {#alternatives}
57+
58+
If you are using `.spec.externalIPs`, then there are several alternatives.
59+
60+
Consider a Service like the following:
61+
62+
```yaml
63+
apiVersion: v1
64+
kind: Service
65+
metadata:
66+
name: my-example-service
67+
spec:
68+
type: ClusterIP
69+
selector:
70+
app.kubernetes.io/name: my-example-app
71+
ports:
72+
- protocol: TCP
73+
port: 80
74+
targetPort: 8080
75+
externalIPs:
76+
- "192.0.2.4"
77+
```
78+
79+
### Using manually-managed LoadBalancer Services instead of `externalIPs` {#alternative-LoadBalancer}
80+
81+
The easiest (but also worst) option is to just switch from using
82+
`externalIPs` to using a `type: LoadBalancer` service, and assigning a
83+
load balancer IP by hand. This is, essentially, exactly the same as
84+
`externalIPs`, with one important difference: the load balancer IP is
85+
part of the Service's `.status`, not its `.spec`, and in a cluster
86+
with RBAC enabled, it can't be edited by ordinary users by default.
87+
Thus, this replacement for `externalIPs` would only be available to
88+
users who were given permission by the admins (although those users
89+
would then be fully empowered to replicate CVE-2020-8554; there would
90+
still not be any further checks to ensure that one user wasn't
91+
stealing another user's IPs, etc.)
92+
93+
Because of the way that `.status` works in Kubernetes, you must create the
94+
Service without a load balancer IP, and then add the IP as a second step:
95+
96+
```console
97+
$ cat loadbalancer-service.yaml
98+
apiVersion: v1
99+
kind: Service
100+
metadata:
101+
name: my-example-service
102+
spec:
103+
# prevent any real load balancer controllers from managing this service
104+
# by using a non-existent loadBalancerClass
105+
loadBalancerClass: non-existent-class
106+
type: LoadBalancer
107+
selector:
108+
app.kubernetes.io/name: my-example-app
109+
ports:
110+
- protocol: TCP
111+
port: 80
112+
targetPort: 8080
113+
$ kubectl apply -f loadbalancer-service.yaml
114+
service/my-example-service created
115+
$ kubectl patch service my-example-service --subresource=status --type=merge -p '{"status":{"loadBalancer":{"ingress":[{"ip":"192.0.2.4"}]}}}'
116+
```
117+
118+
### Using a non-cloud based load balancer controller {#alternative-load-balancer-controller}
119+
120+
Although `LoadBalancer` services were originally designed to be backed by
121+
cloud load balancers, Kubernetes can also support them on non-cloud platforms
122+
by using a third-party load balancer controller such as [MetalLB](https://metallb.io/).
123+
This solves the security problems associated with `externalIPs` because the
124+
administrator can configure what ranges of IP addresses the controller will assign
125+
to services, and the controller will ensure that two services can't both use the same
126+
IP.
127+
128+
So, for example, after [installing](https://metallb.io/installation/) and
129+
[configuring](https://metallb.io/configuration/) MetalLB, a cluster administrator
130+
could configure a pool of IP addresses for use in the cluster:
131+
132+
```yaml
133+
apiVersion: metallb.io/v1beta1
134+
kind: IPAddressPool
135+
metadata:
136+
name: production
137+
namespace: metallb-system
138+
spec:
139+
addresses:
140+
- 192.0.2.0/24
141+
autoAssign: true
142+
avoidBuggyIPs: false
143+
```
144+
145+
After which a user can create a `type: LoadBalancer` Service and MetalLB will handle the
146+
assignment of the IP address. MetalLB even supports the deprecated `loadBalancerIP`
147+
field in Service, so the end user can request a specific IP (assuming it is available)
148+
for backward-compatibility with the `externalIPs` approach, rather than being
149+
assigned one at random:
150+
151+
```yaml
152+
apiVersion: v1
153+
kind: Service
154+
metadata:
155+
name: my-example-service
156+
spec:
157+
type: LoadBalancer
158+
selector:
159+
app.kubernetes.io/name: my-example-app
160+
ports:
161+
- protocol: TCP
162+
port: 80
163+
targetPort: 8080
164+
loadBalancerIP: "192.0.2.4"
165+
```
166+
167+
Similar approaches would work with other load balancer controllers.
168+
This approach can allow cluster administrators to have control over which IP addresses are assigned,
169+
rather than users.
170+
171+
### Using Gateway API {#alternative-gateway-api}
172+
173+
Another potential solution is to use an implementation of the
174+
[Gateway API](https://gateway-api.sigs.k8s.io/).
175+
176+
Gateway API allows cluster administrators to define a Gateway resource, which can have an IP address
177+
attached to it via the `.spec.addresses` field. Since Gateway resources are designed to be managed by
178+
[cluster administrators](https://gateway-api.sigs.k8s.io/concepts/security/), RBAC rules can be put in place to only allow privileged users to manage them.
179+
180+
An example of how this could look is:
181+
182+
```yaml
183+
apiVersion: gateway.networking.k8s.io/v1
184+
kind: Gateway
185+
metadata:
186+
name: example-gateway
187+
spec:
188+
gatewayClassName: example-gateway-class
189+
addresses:
190+
- type: IPAddress
191+
value: "192.0.2.4"
192+
---
193+
apiVersion: gateway.networking.k8s.io/v1
194+
kind: HTTPRoute
195+
metadata:
196+
name: example-route
197+
spec:
198+
parentRefs:
199+
- name: example-gateway
200+
rules:
201+
- backendRefs:
202+
- name: example-svc
203+
port: 80
204+
---
205+
apiVersion: v1
206+
kind: Service
207+
metadata:
208+
name: example-svc
209+
spec:
210+
type: ClusterIP
211+
selector:
212+
app.kubernetes.io/name: example-app
213+
ports:
214+
- protocol: TCP
215+
port: 80
216+
targetPort: 8080
217+
```
218+
219+
The Gateway API project is the next generation of Kubernetes Ingress, Load Balancing, and Service Mesh APIs within Kubernetes.
220+
Gateway API was designed to fix the shortcomings of the Service and Ingress resource, making it a very reliable robust solution
221+
that is under active development.
222+
223+
## Timeline for `externalIPs` deprecation
224+
225+
The rough timeline for this deprecation is as follows:
226+
227+
1. With the release of Kubernetes 1.36, the field was deprecated;
228+
Kubernetes now emits [warnings](/blog/2020/09/03/warnings/) when a user uses this field
229+
2. About a year later (v1.40 at the earliest) support for `.spec.externalIPs` will be disabled in kube-proxy, but users will have a way to opt back in should they require more time to migrate away
230+
3. About another year later - (v1.43 at the earliest) support will be disabled completely; users won't have a way to opt back in

0 commit comments

Comments
 (0)