|
| 1 | +--- |
| 2 | +layout: blog |
| 3 | +title: "Kubernetes v1.36: Deprecation and removal of Service ExternalIPs" |
| 4 | +draft: true # will be changed to date: YYYY-MM-DD before publication |
| 5 | +slug: kubernetes-v1-36-deprecation-and-removal-of-service-externalips # optional |
| 6 | +author: > |
| 7 | + Adrian Moisey (independent), |
| 8 | + Dan Winship (Red Hat), |
| 9 | +--- |
| 10 | + |
| 11 | +The `.spec.externalIPs` field for [Service](/docs/concepts/services-networking/service/) was an early attempt to provide |
| 12 | +cloud-load-balancer-like functionality for non-cloud clusters. |
| 13 | +Unfortunately, the API assumes that every user in the cluster is fully |
| 14 | +trusted, and in any situation where that is not the case, it enables |
| 15 | +various security exploits, as described in |
| 16 | +[CVE-2020-8554](https://www.cvedetails.com/cve/CVE-2020-8554/). |
| 17 | + |
| 18 | +Since Kubernetes 1.21, the Kubernetes project has recommended that all users disable |
| 19 | +`.spec.externalIPs`. To make that easier, Kubernetes also added an admission controller |
| 20 | +(`DenyServiceExternalIPs`) that can be enabled to do this. At the time, |
| 21 | +SIG Network felt that blocking the functionality by default was too large a |
| 22 | +breaking change to consider. |
| 23 | + |
| 24 | +However, the security problems are still there, and as a project we're increasingly |
| 25 | +unhappy with the "insecure by default" state of the feature. |
| 26 | +Additionally, there are now several better alternatives for non-cloud |
| 27 | +clusters wanting load-balancer-like functionality. |
| 28 | + |
| 29 | +As a result, the `.spec.externalIPs` field for Service is now formally deprecated in Kubernetes 1.36. |
| 30 | +We expect that a future minor release of Kubernetes will drop |
| 31 | +implementation of the behavior from `kube-proxy`, and will update the |
| 32 | +Kubernetes [conformance](https://www.cncf.io/training/certification/software-conformance/) criteria to require that conforming implementations |
| 33 | +**do not** provide support. |
| 34 | + |
| 35 | +## A note on terminology, and what hasn't been deprecated {#terminology} |
| 36 | + |
| 37 | +The phrase _external IP_ is somewhat overloaded in Kubernetes: |
| 38 | + |
| 39 | + - The Service API has a field `.spec.externalIPs` that can be used |
| 40 | + to add additional IP addresses that a Service will respond on. |
| 41 | + |
| 42 | + - The Node API's `.status.addresses` field can list addresses of |
| 43 | + several different types, one of which is called `ExternalIP`. |
| 44 | + |
| 45 | + - The `kubectl` tool, when displaying information about a Service of type |
| 46 | + LoadBalancer in the default output format, will show the load |
| 47 | + balancer IP address under the column heading `EXTERNAL-IP`. |
| 48 | + |
| 49 | +This deprecation is about the first of those. If you are not setting |
| 50 | +the field `externalIPs` in any of your Services, then it does not |
| 51 | +apply to you. |
| 52 | + |
| 53 | +That said, as a precaution, you may still want to enable the [DenyServiceExternalIPs](/docs/reference/access-authn-authz/admission-controllers/#denyserviceexternalips) admission controller to |
| 54 | +block any future use of the `externalIPs` field. |
| 55 | + |
| 56 | +## Alternatives to `externalIPs` {#alternatives} |
| 57 | + |
| 58 | +If you are using `.spec.externalIPs`, then there are several alternatives. |
| 59 | + |
| 60 | +Consider a Service like the following: |
| 61 | + |
| 62 | +```yaml |
| 63 | +apiVersion: v1 |
| 64 | +kind: Service |
| 65 | +metadata: |
| 66 | + name: my-example-service |
| 67 | +spec: |
| 68 | + type: ClusterIP |
| 69 | + selector: |
| 70 | + app.kubernetes.io/name: my-example-app |
| 71 | + ports: |
| 72 | + - protocol: TCP |
| 73 | + port: 80 |
| 74 | + targetPort: 8080 |
| 75 | + externalIPs: |
| 76 | + - "192.0.2.4" |
| 77 | +``` |
| 78 | +
|
| 79 | +### Using manually-managed LoadBalancer Services instead of `externalIPs` {#alternative-LoadBalancer} |
| 80 | + |
| 81 | +The easiest (but also worst) option is to just switch from using |
| 82 | +`externalIPs` to using a `type: LoadBalancer` service, and assigning a |
| 83 | +load balancer IP by hand. This is, essentially, exactly the same as |
| 84 | +`externalIPs`, with one important difference: the load balancer IP is |
| 85 | +part of the Service's `.status`, not its `.spec`, and in a cluster |
| 86 | +with RBAC enabled, it can't be edited by ordinary users by default. |
| 87 | +Thus, this replacement for `externalIPs` would only be available to |
| 88 | +users who were given permission by the admins (although those users |
| 89 | +would then be fully empowered to replicate CVE-2020-8554; there would |
| 90 | +still not be any further checks to ensure that one user wasn't |
| 91 | +stealing another user's IPs, etc.) |
| 92 | + |
| 93 | +Because of the way that `.status` works in Kubernetes, you must create the |
| 94 | +Service without a load balancer IP, and then add the IP as a second step: |
| 95 | + |
| 96 | +```console |
| 97 | +$ cat loadbalancer-service.yaml |
| 98 | +apiVersion: v1 |
| 99 | +kind: Service |
| 100 | +metadata: |
| 101 | + name: my-example-service |
| 102 | +spec: |
| 103 | + # prevent any real load balancer controllers from managing this service |
| 104 | + # by using a non-existent loadBalancerClass |
| 105 | + loadBalancerClass: non-existent-class |
| 106 | + type: LoadBalancer |
| 107 | + selector: |
| 108 | + app.kubernetes.io/name: my-example-app |
| 109 | + ports: |
| 110 | + - protocol: TCP |
| 111 | + port: 80 |
| 112 | + targetPort: 8080 |
| 113 | +$ kubectl apply -f loadbalancer-service.yaml |
| 114 | +service/my-example-service created |
| 115 | +$ kubectl patch service my-example-service --subresource=status --type=merge -p '{"status":{"loadBalancer":{"ingress":[{"ip":"192.0.2.4"}]}}}' |
| 116 | +``` |
| 117 | + |
| 118 | +### Using a non-cloud based load balancer controller {#alternative-load-balancer-controller} |
| 119 | + |
| 120 | +Although `LoadBalancer` services were originally designed to be backed by |
| 121 | +cloud load balancers, Kubernetes can also support them on non-cloud platforms |
| 122 | +by using a third-party load balancer controller such as [MetalLB](https://metallb.io/). |
| 123 | +This solves the security problems associated with `externalIPs` because the |
| 124 | +administrator can configure what ranges of IP addresses the controller will assign |
| 125 | +to services, and the controller will ensure that two services can't both use the same |
| 126 | +IP. |
| 127 | + |
| 128 | +So, for example, after [installing](https://metallb.io/installation/) and |
| 129 | +[configuring](https://metallb.io/configuration/) MetalLB, a cluster administrator |
| 130 | +could configure a pool of IP addresses for use in the cluster: |
| 131 | + |
| 132 | +```yaml |
| 133 | +apiVersion: metallb.io/v1beta1 |
| 134 | +kind: IPAddressPool |
| 135 | +metadata: |
| 136 | + name: production |
| 137 | + namespace: metallb-system |
| 138 | +spec: |
| 139 | + addresses: |
| 140 | + - 192.0.2.0/24 |
| 141 | + autoAssign: true |
| 142 | + avoidBuggyIPs: false |
| 143 | +``` |
| 144 | + |
| 145 | +After which a user can create a `type: LoadBalancer` Service and MetalLB will handle the |
| 146 | +assignment of the IP address. MetalLB even supports the deprecated `loadBalancerIP` |
| 147 | +field in Service, so the end user can request a specific IP (assuming it is available) |
| 148 | +for backward-compatibility with the `externalIPs` approach, rather than being |
| 149 | +assigned one at random: |
| 150 | + |
| 151 | +```yaml |
| 152 | +apiVersion: v1 |
| 153 | +kind: Service |
| 154 | +metadata: |
| 155 | + name: my-example-service |
| 156 | +spec: |
| 157 | + type: LoadBalancer |
| 158 | + selector: |
| 159 | + app.kubernetes.io/name: my-example-app |
| 160 | + ports: |
| 161 | + - protocol: TCP |
| 162 | + port: 80 |
| 163 | + targetPort: 8080 |
| 164 | + loadBalancerIP: "192.0.2.4" |
| 165 | +``` |
| 166 | + |
| 167 | +Similar approaches would work with other load balancer controllers. |
| 168 | +This approach can allow cluster administrators to have control over which IP addresses are assigned, |
| 169 | +rather than users. |
| 170 | + |
| 171 | +### Using Gateway API {#alternative-gateway-api} |
| 172 | + |
| 173 | +Another potential solution is to use an implementation of the |
| 174 | +[Gateway API](https://gateway-api.sigs.k8s.io/). |
| 175 | + |
| 176 | +Gateway API allows cluster administrators to define a Gateway resource, which can have an IP address |
| 177 | +attached to it via the `.spec.addresses` field. Since Gateway resources are designed to be managed by |
| 178 | +[cluster administrators](https://gateway-api.sigs.k8s.io/concepts/security/), RBAC rules can be put in place to only allow privileged users to manage them. |
| 179 | + |
| 180 | +An example of how this could look is: |
| 181 | + |
| 182 | +```yaml |
| 183 | +apiVersion: gateway.networking.k8s.io/v1 |
| 184 | +kind: Gateway |
| 185 | +metadata: |
| 186 | + name: example-gateway |
| 187 | +spec: |
| 188 | + gatewayClassName: example-gateway-class |
| 189 | + addresses: |
| 190 | + - type: IPAddress |
| 191 | + value: "192.0.2.4" |
| 192 | +--- |
| 193 | +apiVersion: gateway.networking.k8s.io/v1 |
| 194 | +kind: HTTPRoute |
| 195 | +metadata: |
| 196 | + name: example-route |
| 197 | +spec: |
| 198 | + parentRefs: |
| 199 | + - name: example-gateway |
| 200 | + rules: |
| 201 | + - backendRefs: |
| 202 | + - name: example-svc |
| 203 | + port: 80 |
| 204 | +--- |
| 205 | +apiVersion: v1 |
| 206 | +kind: Service |
| 207 | +metadata: |
| 208 | + name: example-svc |
| 209 | +spec: |
| 210 | + type: ClusterIP |
| 211 | + selector: |
| 212 | + app.kubernetes.io/name: example-app |
| 213 | + ports: |
| 214 | + - protocol: TCP |
| 215 | + port: 80 |
| 216 | + targetPort: 8080 |
| 217 | +``` |
| 218 | + |
| 219 | +The Gateway API project is the next generation of Kubernetes Ingress, Load Balancing, and Service Mesh APIs within Kubernetes. |
| 220 | +Gateway API was designed to fix the shortcomings of the Service and Ingress resource, making it a very reliable robust solution |
| 221 | +that is under active development. |
| 222 | + |
| 223 | +## Timeline for `externalIPs` deprecation |
| 224 | + |
| 225 | +The rough timeline for this deprecation is as follows: |
| 226 | + |
| 227 | +1. With the release of Kubernetes 1.36, the field was deprecated; |
| 228 | + Kubernetes now emits [warnings](/blog/2020/09/03/warnings/) when a user uses this field |
| 229 | +2. About a year later (v1.40 at the earliest) support for `.spec.externalIPs` will be disabled in kube-proxy, but users will have a way to opt back in should they require more time to migrate away |
| 230 | +3. About another year later - (v1.43 at the earliest) support will be disabled completely; users won't have a way to opt back in |
0 commit comments