diff --git a/CHANGELOG.md b/CHANGELOG.md index 8b7a362..d1795ba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,16 @@ Gatekeeper is a standalone credential-injecting TLS-intercepting proxy. It trans Gatekeeper is pre-1.0. The configuration schema and credential source interface may change between minor versions. +## v0.18.0 — 2026-07-15 + +### Added + +- **The Postgres data-plane listener now supports PROXY protocol v1/v2** — until now, only the HTTP/CONNECT listener could recover the real client address behind a TCP-terminating load balancer; a load balancer fronting the Postgres port had no equivalent, so `client_ip` on postgres canonical log lines (and the `client_addr` field on a failed run-token-authentication log line) always showed the load balancer's own front-end address. Setting `postgres.proxy_protocol: true` wraps the Postgres listener with the same PROXY protocol handling the HTTP listener already had: fail-open `USE` connection policy (a header is honored when present; a connection that opens without one, or that doesn't send one within a 10s read timeout, falls back to its raw TCP peer address rather than being rejected, so load balancer health checks and direct TCP probes of the port keep working), and a single DEBUG log line for a connection whose header is present but fails to parse. The PROXY header is always the very first bytes on the wire — ahead of even the client's `SSLRequest` — so the listener must be PROXY-protocol-aware from the first `Accept`; `PostgresServer` gains a `StartListener(net.Listener)` method so the caller can wrap the listener before handing it off, and `gatekeeper.go`'s postgres wiring now binds, conditionally wraps, then calls it, mirroring the HTTP listener's own wrap-before-serve sequence. `enableKeepAlive` (`proxy/postgres.go`) unwraps a wrapped connection via a structural `Raw() net.Conn` accessor before its `*net.TCPConn` type assertion, so TCP keep-alives on the relayed connection keep working when the listener is PROXY-protocol-wrapped — unwrapping through `Raw()` is side-effect-free, unlike `RemoteAddr()`/`Read()`, which trigger the (potentially blocking) PROXY header parse + +### Changed + +- **PROXY protocol configuration moved from `network.proxy_protocol` to per-listener toggles: `proxy.proxy_protocol` (HTTP/CONNECT) and `postgres.proxy_protocol` (Postgres)** — `network.proxy_protocol` predates the Postgres listener supporting PROXY protocol at all, so it implicitly named the only listener capable of it; now that both listeners can, a single flag under `network` no longer has an unambiguous meaning, and `network` is otherwise about host allow/deny policy, not listener transport concerns. `network.proxy_protocol` is removed outright — gatekeeper is pre-1.0 and single-consumer, so this ships as a breaking config change rather than carrying a deprecated alias. `proxy.proxy_protocol` is the new home for the exact same HTTP-listener behavior (semantics, defaults, and the fail-open/malformed-header handling are unchanged — only the config path moved); existing configs setting `network.proxy_protocol: true` must move that line under `proxy:` before upgrading. Internally, both listeners now share one implementation: `proxy.WrapProxyProtocolListener` (new file, `proxy/proxyproto.go`) holds the `proxyproto.Listener` construction, the fail-open `USE` policy, the 10s `ReadHeaderTimeout`, and the malformed-header debug-log wrapper (`proxyProtoLogListener`/`proxyProtoLogConn`, moved here from `gatekeeper.go`, which previously duplicated this logic inline for the HTTP listener only) — `gatekeeper.go`'s HTTP and Postgres wiring both call this one helper, so the two listeners' PROXY protocol handling is byte-identical and can't drift + ## v0.17.3 — 2026-07-15 ### Changed diff --git a/README.md b/README.md index 58e936f..c7cc85d 100644 --- a/README.md +++ b/README.md @@ -53,6 +53,7 @@ cd examples && ./gen-ca.sh proxy: host: 127.0.0.1 port: 9080 + # proxy_protocol: true # behind a TCP-terminating LB, recover the real client_ip tls: ca_cert: ca.crt @@ -68,7 +69,6 @@ credentials: network: policy: permissive - # proxy_protocol: true # behind a TCP-terminating LB, recover the real client_ip log: level: info @@ -140,7 +140,9 @@ network: Policies: `permissive` (allow all), `strict` (deny all, allow listed). -Behind a TCP-terminating load balancer (e.g. GCP's global TCP Proxy LB), every connection's peer address is the load balancer's, not the real client — `network.proxy_protocol: true` parses a PROXY protocol v1/v2 header from the LB and uses its advertised source as the `client_ip` recorded in request logs. It's fail-open (headerless connections, like LB health checks, fall back to the raw peer address) and should only be enabled when the port is reachable solely through the load balancer, since the header is otherwise forgeable by any direct client. +## PROXY protocol (client IP behind a load balancer) + +Behind a TCP-terminating load balancer (e.g. GCP's global TCP Proxy LB), every connection's peer address is the load balancer's, not the real client. PROXY protocol parsing recovers the real client address, and it's configured per listener: `proxy.proxy_protocol: true` for the HTTP/CONNECT listener, and `postgres.proxy_protocol: true` for the Postgres data-plane listener. Each parses a PROXY protocol v1/v2 header from the LB and uses its advertised source as the `client_ip` recorded in request logs. Both are fail-open (headerless connections, like LB health checks, fall back to the raw peer address) and should only be enabled when that listener's port is reachable solely through the load balancer, since the header is otherwise forgeable by any direct client. See [Deploying behind a TCP load balancer](docs/content/guides/11-load-balancer-proxy-protocol.md). ## MCP relay diff --git a/config.go b/config.go index 7b11af6..8b40b47 100644 --- a/config.go +++ b/config.go @@ -22,6 +22,23 @@ type Config struct { type PostgresConfig struct { Port int `yaml:"port"` // listener port (e.g. 5432) Host string `yaml:"host,omitempty"` // bind address (default: same as proxy host) + + // ProxyProtocol enables PROXY protocol v1/v2 parsing on the Postgres + // data-plane listener, mirroring proxy.proxy_protocol: when true, each + // inbound connection is checked for a leading PROXY protocol header (as + // prepended by a TCP-terminating load balancer in front of the Postgres + // port) and, if present, the advertised source address replaces the raw + // TCP peer address as the connection's client address for logging. + // Connections that do not open with a PROXY header fall back to the raw + // TCP peer address (fail-open), so load balancer health checks and + // direct probes of the port keep working. Default false. + // + // Because headers are honored from any peer, a client that can reach + // this listener directly (bypassing the load balancer) can forge the + // logged client_ip by prepending its own PROXY header. Only enable this + // when the port is reachable solely through the load balancer, and never + // use client_ip for security decisions. + ProxyProtocol bool `yaml:"proxy_protocol,omitempty"` } // ProxyConfig configures the proxy listener. @@ -29,6 +46,23 @@ type ProxyConfig struct { Port int `yaml:"port"` Host string `yaml:"host"` AuthToken string `yaml:"auth_token,omitempty"` // Optional token clients must provide via Proxy-Authorization + + // ProxyProtocol enables PROXY protocol v1/v2 parsing on the HTTP/CONNECT + // proxy listener. When true, each inbound connection is checked for a + // leading PROXY protocol header (as prepended by a TCP load balancer, + // e.g. GCP's global TCP Proxy LB) and, if present, the advertised source + // address replaces the raw TCP peer address as the connection's client + // address for logging (the client_ip request-log attribute). Connections + // that do not open with a PROXY header fall back to the raw TCP peer + // address (fail-open), so load balancer health checks and direct probes + // of the port keep working. Default false. + // + // Because headers are honored from any peer, a client that can reach the + // listener directly (bypassing the load balancer) can forge the logged + // client_ip by prepending its own PROXY header. Only enable this when + // the port is reachable solely through the load balancer, and never use + // client_ip for security decisions. + ProxyProtocol bool `yaml:"proxy_protocol,omitempty"` } // TLSConfig configures the CA certificate used for TLS interception. @@ -110,23 +144,6 @@ type SourceConfig struct { type NetworkConfig struct { Policy string `yaml:"policy"` Allow []string `yaml:"allow,omitempty"` - - // ProxyProtocol enables PROXY protocol v1/v2 parsing on the proxy - // listener. When true, each inbound connection is checked for a leading - // PROXY protocol header (as prepended by a TCP load balancer, e.g. GCP's - // global TCP Proxy LB) and, if present, the advertised source address - // replaces the TCP peer address as the connection's client address for - // logging (the client_ip request-log attribute). Connections that do not - // open with a PROXY header fall back to the raw TCP peer address - // (fail-open), so load balancer health checks and direct probes of the - // port keep working. Default false. - // - // Because headers are honored from any peer, a client that can reach the - // listener directly (bypassing the load balancer) can forge the logged - // client_ip by prepending its own PROXY header. Only enable this when - // the port is reachable solely through the load balancer, and never use - // client_ip for security decisions. - ProxyProtocol bool `yaml:"proxy_protocol,omitempty"` } // LogConfig configures logging. diff --git a/config_test.go b/config_test.go index 132c879..6f167bc 100644 --- a/config_test.go +++ b/config_test.go @@ -15,6 +15,7 @@ func TestParseConfig_Full(t *testing.T) { proxy: port: 8080 host: 127.0.0.1 + proxy_protocol: true tls: ca_cert: /tmp/ca.crt ca_key: /tmp/ca.key @@ -42,7 +43,6 @@ network: allow: - "*.github.com" - api.anthropic.com - proxy_protocol: true log: level: debug format: json @@ -60,6 +60,9 @@ log: if cfg.Proxy.Host != "127.0.0.1" { t.Errorf("Proxy.Host = %q, want 127.0.0.1", cfg.Proxy.Host) } + if !cfg.Proxy.ProxyProtocol { + t.Error("Proxy.ProxyProtocol = false, want true") + } // TLS if cfg.TLS.CACert != "/tmp/ca.crt" { @@ -111,9 +114,6 @@ log: if cfg.Network.Allow[0] != "*.github.com" { t.Errorf("Network.Allow[0] = %q, want *.github.com", cfg.Network.Allow[0]) } - if !cfg.Network.ProxyProtocol { - t.Error("Network.ProxyProtocol = false, want true") - } // Log if cfg.Log.Level != "debug" { t.Errorf("Log.Level = %q, want debug", cfg.Log.Level) @@ -138,8 +138,11 @@ proxy: if len(cfg.Credentials) != 0 { t.Errorf("len(Credentials) = %d, want 0", len(cfg.Credentials)) } - if cfg.Network.ProxyProtocol { - t.Error("Network.ProxyProtocol = true, want false when absent from config") + if cfg.Proxy.ProxyProtocol { + t.Error("Proxy.ProxyProtocol = true, want false when absent from config") + } + if cfg.Postgres != nil && cfg.Postgres.ProxyProtocol { + t.Error("Postgres.ProxyProtocol = true, want false when absent from config") } } @@ -150,6 +153,7 @@ proxy: postgres: port: 5432 host: 0.0.0.0 + proxy_protocol: true credentials: - host: "*.neon.tech" postgres: @@ -173,6 +177,9 @@ credentials: if cfg.Postgres.Host != "0.0.0.0" { t.Errorf("Host = %q, want 0.0.0.0", cfg.Postgres.Host) } + if !cfg.Postgres.ProxyProtocol { + t.Error("Postgres.ProxyProtocol = false, want true") + } if len(cfg.Credentials) != 1 { t.Fatalf("credentials = %d, want 1", len(cfg.Credentials)) } diff --git a/docs/content/concepts/06-observability.md b/docs/content/concepts/06-observability.md index 691a8d1..e57c81b 100644 --- a/docs/content/concepts/06-observability.md +++ b/docs/content/concepts/06-observability.md @@ -69,11 +69,11 @@ Log level is determined by outcome: `ERROR` for server errors or transport failu ## Client IP attribution behind a load balancer -By default, `client_ip` comes from the raw TCP peer address of the proxy listener's accepted connection. That's accurate for a directly-exposed listener, but not for one that sits behind a TCP-terminating load balancer (e.g. GCP's global TCP Proxy load balancer) — every connection appears to originate from the load balancer's own front-end range (`35.191.0.0/16` for GCP), not the real client. +By default, `client_ip` comes from the raw TCP peer address of the accepted connection. That's accurate for a directly-exposed listener, but not for one that sits behind a TCP-terminating load balancer (e.g. GCP's global TCP Proxy load balancer) — every connection appears to originate from the load balancer's own front-end range (`35.191.0.0/16` for GCP), not the real client. -Setting `network.proxy_protocol: true` wraps the proxy listener with PROXY protocol v1/v2 parsing. When the load balancer prepends a PROXY header, gatekeeper uses its advertised source address as the connection's client address instead of the TCP peer, flowing through to `client_ip` on every request path, including CONNECT-intercepted inner requests. Connections opened without a header — load balancer health checks, direct probes of the port — fall back to the raw TCP peer address rather than being rejected. +PROXY protocol support is configured per listener: `proxy.proxy_protocol: true` wraps the HTTP/CONNECT proxy listener, and `postgres.proxy_protocol: true` wraps the [Postgres data-plane listener](./08-postgres-data-plane.md) — both parse PROXY protocol v1/v2 through the same shared logic. When the load balancer prepends a PROXY header, gatekeeper uses its advertised source address as the connection's client address instead of the TCP peer. On the HTTP listener this flows through to `client_ip` on every request path, including CONNECT-intercepted inner requests. On the Postgres listener it flows through to `client_ip` on the canonical log line and to the `client_addr` field on a failed run-token-authentication log line; the PROXY header is always the first bytes on the wire, so it's parsed before the client's `SSLRequest` and the TLS handshake. Connections opened without a header — load balancer health checks, direct probes of the port — fall back to the raw TCP peer address rather than being rejected, on either listener. -Because the header is honored from any peer that can reach the listener, only enable `proxy_protocol` when the port is reachable solely through the load balancer, and never use `client_ip` for security decisions — it's a logging convenience, not an authenticated identity. +Because the header is honored from any peer that can reach the listener, only enable `proxy_protocol` on a listener whose port is reachable solely through the load balancer, and never use `client_ip` for security decisions — it's a logging convenience, not an authenticated identity. See [Deploying behind a TCP load balancer](../guides/11-load-balancer-proxy-protocol.md) for the full setup, including the Postgres listener. ## Request ID tracking diff --git a/docs/content/guides/11-load-balancer-proxy-protocol.md b/docs/content/guides/11-load-balancer-proxy-protocol.md index b5030bf..9eea7fa 100644 --- a/docs/content/guides/11-load-balancer-proxy-protocol.md +++ b/docs/content/guides/11-load-balancer-proxy-protocol.md @@ -1,17 +1,19 @@ --- title: "Deploying Behind a TCP Load Balancer" description: "Recover the real client IP when gatekeeper runs behind a TCP-terminating load balancer, using PROXY protocol v1/v2." -keywords: ["gatekeeper", "load balancer", "PROXY protocol", "GCP", "client IP"] +keywords: ["gatekeeper", "load balancer", "PROXY protocol", "GCP", "client IP", "Postgres"] --- # Deploying behind a TCP load balancer Recover the real client IP when gatekeeper runs behind a TCP-terminating load balancer, such as GCP's global TCP Proxy load balancer. Without this, every request's `client_ip` log attribute shows the load balancer's hop instead of the actual client. +PROXY protocol support is configured per listener: `proxy.proxy_protocol` for the HTTP/CONNECT listener, and `postgres.proxy_protocol` for the [Postgres data-plane listener](../concepts/08-postgres-data-plane.md) (if configured). They are independent toggles — enabling one does not enable the other — because the two listeners are commonly fronted by different load balancers, or only one of them is exposed publicly at all. Everything below applies identically to both; the config examples call out where the field name differs. + ## Prerequisites - A working gatekeeper deployment (see [Config file](../reference/02-config-file.md)) -- A TCP-terminating load balancer in front of gatekeeper that supports PROXY protocol v1/v2 (for example, GCP's global TCP Proxy load balancer) +- A TCP-terminating load balancer in front of the listener you're enabling this for, supporting PROXY protocol v1/v2 (for example, GCP's global TCP Proxy load balancer) - `gcloud` configured against the target GCP project, if following the GCP walkthrough below ## Why client_ip shows the load balancer @@ -22,18 +24,19 @@ A TCP-terminating load balancer ends the client's TCP connection at its own edge 1. The client connects to the load balancer's public IP. 2. The load balancer terminates that connection and opens a new TCP connection to gatekeeper, prepending a PROXY protocol v1 or v2 header that names the original client address. -3. When `network.proxy_protocol: true`, gatekeeper wraps its proxy listener with PROXY protocol parsing. On each accepted connection, it reads the leading header (if present) and substitutes the header's advertised source address for the raw TCP peer address. -4. That substituted address flows through to the `client_ip` request-log attribute on every request path, including CONNECT-intercepted TLS traffic. +3. When `proxy.proxy_protocol: true` (HTTP/CONNECT listener) or `postgres.proxy_protocol: true` (Postgres listener) is set, gatekeeper wraps that listener with PROXY protocol parsing. On each accepted connection, it reads the leading header (if present) and substitutes the header's advertised source address for the raw TCP peer address. Both listeners share the exact same parsing logic — fail-open policy, timeout, and malformed-header handling are identical. +4. That substituted address flows through to the `client_ip` request-log attribute on every request path — including CONNECT-intercepted TLS traffic on the HTTP listener, and the canonical log line and the run-token-authentication-failure log line on the Postgres listener. On the Postgres listener, the PROXY header is always the very first bytes on the wire, so it's parsed before the client's `SSLRequest` and the TLS handshake that follows it. 5. A connection that opens without a PROXY header — or that doesn't send one within a 10-second read timeout — falls back to the raw TCP peer address instead of being rejected. This fail-open behavior keeps load balancer health checks and direct probes of the port working. ## Enabling proxy_protocol -Set `network.proxy_protocol: true` in gatekeeper.yaml: +Set `proxy.proxy_protocol: true` in gatekeeper.yaml for the HTTP/CONNECT listener: ```yaml proxy: host: 0.0.0.0 port: 9080 + proxy_protocol: true tls: ca_cert: ca.crt @@ -49,7 +52,6 @@ credentials: network: policy: permissive - proxy_protocol: true log: level: info @@ -58,9 +60,18 @@ log: `proxy.host` is bound to `0.0.0.0` here because the listener must be reachable from the load balancer, which is not running on the same host. See [Security Requirement](#security-requirement) below before deploying this configuration. +If gatekeeper's Postgres data-plane listener is also fronted by a TCP-terminating load balancer, enable it there independently with `postgres.proxy_protocol: true`: + +```yaml +postgres: + host: 0.0.0.0 + port: 5432 + proxy_protocol: true +``` + ## Configuring the GCP backend service -Enabling `proxy_protocol` on gatekeeper is only half the change — the load balancer must also be told to send the header. For a global TCP Proxy load balancer, this is the backend service's `proxyHeader` field. +Enabling `proxy_protocol` on gatekeeper is only half the change — the load balancer must also be told to send the header. For a global TCP Proxy load balancer, this is the backend service's `proxyHeader` field. Configure this on whichever backend service fronts the listener you enabled — the HTTP/CONNECT port, the Postgres port, or both, if both are load-balanced. Update an existing backend service: @@ -91,33 +102,37 @@ gcloud compute backend-services add-backend gatekeeper-backend \ `--proxy-header` accepts `NONE` (default) or `PROXY_V1`. Gatekeeper's PROXY protocol parser handles both v1 (text) and v2 (binary) headers, but GCP's global TCP Proxy load balancer sends v1. +The Postgres listener's backend service follows the same pattern, just pointed at the Postgres port (5432 by default) with a TCP health check instead of an HTTP one against `/healthz` — the Postgres listener has no HTTP health endpoint; a bare TCP connect check is sufficient, since the listener always answers the Postgres `SSLRequest` preamble. + ## Safe rollout ordering -Deploy in this order to avoid an outage: +Deploy in this order to avoid an outage, for each listener you're enabling: -1. **Enable `network.proxy_protocol: true` on gatekeeper first, and deploy it.** Because the listener fails open, connections without a PROXY header — which is every connection at this point, since the load balancer isn't sending one yet — fall back to the raw TCP peer address exactly as before. `/healthz` stays green throughout. +1. **Enable `proxy_protocol: true` (`proxy.proxy_protocol` or `postgres.proxy_protocol`) on gatekeeper first, and deploy it.** Because the listener fails open, connections without a PROXY header — which is every connection at this point, since the load balancer isn't sending one yet — fall back to the raw TCP peer address exactly as before. `/healthz` (HTTP listener) or a plain TCP connect (Postgres listener) stays green throughout. 2. **Confirm gatekeeper is healthy and serving traffic** with `proxy_protocol` enabled but no header arriving yet. 3. **Update the backend service's `proxyHeader` to `PROXY_V1`.** Once the load balancer starts sending headers, gatekeeper parses them and `client_ip` starts reflecting real clients. -Reversing this order — turning on `proxyHeader: PROXY_V1` before gatekeeper understands it — sends a raw PROXY header into a listener that doesn't parse it. Gatekeeper would treat the header bytes as the start of an HTTP request or TLS handshake and reject the connection, which is why gatekeeper goes first. +Reversing this order — turning on `proxyHeader: PROXY_V1` before gatekeeper understands it — sends a raw PROXY header into a listener that doesn't parse it. On the HTTP listener, gatekeeper would treat the header bytes as the start of an HTTP request and reject the connection; on the Postgres listener, it would corrupt the `SSLRequest` preamble and the connection would be dropped. Either way, gatekeeper goes first. + +If both listeners are load-balanced, roll them out independently, one at a time — enabling `proxy.proxy_protocol` has no effect on the Postgres listener's behavior, and vice versa. ## Security requirement -> **Warning:** `network.proxy_protocol` trusts the PROXY header from any peer that can reach the listener. A client that connects directly to gatekeeper's port — bypassing the load balancer — can prepend its own PROXY header and forge its logged `client_ip`. +> **Warning:** `proxy_protocol` trusts the PROXY header from any peer that can reach the listener. A client that connects directly to gatekeeper's port — bypassing the load balancer — can prepend its own PROXY header and forge its logged `client_ip`. -Only enable `proxy_protocol` when the port is reachable solely through the load balancer, and never use `client_ip` for security decisions — it's a logging convenience, not an authenticated identity. Restrict direct access to the port with a firewall rule that allows only the load balancer's and health check's source ranges: +Only enable `proxy_protocol` on a listener when its port is reachable solely through the load balancer, and never use `client_ip` for security decisions — it's a logging convenience, not an authenticated identity. Restrict direct access to the port with a firewall rule that allows only the load balancer's and health check's source ranges: ```bash gcloud compute firewall-rules create allow-gatekeeper-lb \ --network=default \ --direction=INGRESS \ --action=ALLOW \ - --rules=tcp:9080 \ + --rules=tcp:9080,tcp:5432 \ --source-ranges=130.211.0.0/22,35.191.0.0/16 \ --target-tags=gatekeeper ``` -The Postgres data-plane listener does not parse PROXY protocol headers — it is a separate listener and port, unaffected by this setting. +Adjust the `tcp:` ports to whichever listeners you've fronted with the load balancer — omit 5432 if the Postgres listener isn't load-balanced (or isn't configured at all). ## Verification @@ -127,13 +142,19 @@ Before enabling `proxy_protocol`, the canonical log line shows the load balancer level=INFO msg=request http_host=api.github.com client_ip=35.191.12.34 credential_injected=true ``` -After enabling `network.proxy_protocol: true` on gatekeeper and `--proxy-header=PROXY_V1` on the backend service, the same request logs the real client: +After enabling `proxy.proxy_protocol: true` on gatekeeper and `--proxy-header=PROXY_V1` on the backend service, the same request logs the real client: ```text level=INFO msg=request http_host=api.github.com client_ip=203.0.113.7 credential_injected=true ``` -A connection whose PROXY header fails to parse — a malformed v1 line or truncated v2 binary header — is dropped, and gatekeeper logs one line at `DEBUG`: +The Postgres listener's canonical log line behaves the same way once `postgres.proxy_protocol: true` is set and the fronting load balancer sends the header: + +```text +level=INFO msg=request proxy_type=postgres client_ip=203.0.113.7 credential_injected=true +``` + +A connection whose PROXY header fails to parse — a malformed v1 line or truncated v2 binary header — is dropped, and gatekeeper logs one line at `DEBUG`, on either listener: ```text level=DEBUG msg="dropping connection: malformed PROXY protocol header" peer=35.191.12.34:51422 err="proxyproto: ..." @@ -141,10 +162,11 @@ level=DEBUG msg="dropping connection: malformed PROXY protocol header" peer=35.1 ## Health check notes -Gatekeeper's fail-open policy means the load balancer's HTTP health check against `/healthz` keeps working whether or not it carries a PROXY header: a probe that includes one is parsed normally, and a probe that doesn't falls back to the raw TCP peer address. Neither case is rejected, so flipping `proxy_protocol` on and off does not require a matching change to the health check configuration. +Gatekeeper's fail-open policy means a load balancer health check keeps working whether or not it carries a PROXY header: a probe that includes one is parsed normally, and a probe that doesn't falls back to the raw TCP peer address. Neither case is rejected, so flipping `proxy_protocol` on and off does not require a matching change to the health check configuration. This holds for both the HTTP listener's `/healthz` endpoint and a bare TCP connect check against the Postgres listener. ## Next steps +- [Postgres data plane](../concepts/08-postgres-data-plane.md) — how the Postgres listener authenticates clients and resolves upstream credentials - [Observability](../concepts/06-observability.md) — how `client_ip` and other request attributes flow into logs, traces, and metrics -- [Config file](../reference/02-config-file.md) — full `network.proxy_protocol` field reference +- [Config file](../reference/02-config-file.md) — full `proxy.proxy_protocol` and `postgres.proxy_protocol` field reference - [OpenTelemetry](./08-opentelemetry.md) — correlate `client_ip` with traces and metrics diff --git a/docs/content/reference/02-config-file.md b/docs/content/reference/02-config-file.md index a51e1c7..a8f34fe 100644 --- a/docs/content/reference/02-config-file.md +++ b/docs/content/reference/02-config-file.md @@ -41,6 +41,8 @@ log: output: stderr ``` +PROXY protocol (recovering the real client IP behind a TCP load balancer) is configured per listener: `proxy.proxy_protocol` for the HTTP/CONNECT listener, `postgres.proxy_protocol` for the Postgres data-plane listener. See [proxy.proxy_protocol](#proxyproxy_protocol) and [postgres.proxy_protocol](#postgresproxy_protocol) below. + | Section | Description | |---------|-------------| | `proxy` | Proxy listener address and authentication | @@ -105,6 +107,27 @@ export HTTP_PROXY=http://user:my-secret-token@127.0.0.1:8080 The username portion is ignored. The token comparison is constant-time to prevent timing attacks. +### proxy.proxy_protocol + +Parse PROXY protocol v1/v2 headers on the HTTP/CONNECT proxy listener to recover the real client address behind a TCP-terminating load balancer. + +```yaml +proxy: + proxy_protocol: true +``` + +- **Type:** `bool` +- **Required:** No +- **Default:** `false` + +When enabled, each inbound connection to the proxy listener is checked for a leading PROXY protocol header — as prepended by a TCP load balancer such as GCP's global TCP Proxy load balancer — and, if present, the header's advertised source address replaces the raw TCP peer address as the connection's client address. This flows through to the `client_ip` request-log attribute on every request path, including CONNECT-intercepted TLS traffic. + +Connections that open without a PROXY header, or that don't send one within a 10s read timeout, fall back to the raw TCP peer address instead of being rejected (fail-open), so load balancer health checks and direct probes of the port keep working. + +Because the header is honored from any peer, a client that can reach the listener directly (bypassing the load balancer) can forge its logged `client_ip` by prepending its own PROXY header. Only enable this when the port is reachable solely through the load balancer, and never use `client_ip` for security decisions. + +The Postgres data-plane listener has its own, independent toggle — see [postgres.proxy_protocol](#postgresproxy_protocol). + --- ## tls @@ -177,6 +200,25 @@ postgres: - **Required:** No - **Default:** the `proxy.host` value +### postgres.proxy_protocol + +Parse PROXY protocol v1/v2 headers on the Postgres data-plane listener to recover the real client address behind a TCP-terminating load balancer, mirroring [proxy.proxy_protocol](#proxyproxy_protocol) for this listener. + +```yaml +postgres: + proxy_protocol: true +``` + +- **Type:** `bool` +- **Required:** No +- **Default:** `false` + +Semantics are identical to `proxy.proxy_protocol`, applied to this listener instead: each inbound connection is checked for a leading PROXY protocol header, and, if present, the header's advertised source address replaces the raw TCP peer address as the connection's client address (the `client_ip` request-log attribute and the `client_addr` field on a failed run-token authentication log line). Connections that open without a header, or that don't send one within a 10s read timeout, fall back to the raw TCP peer address instead of being rejected (fail-open). The PROXY header is always the first bytes on the wire, so it is parsed before the client's `SSLRequest` and the TLS handshake. + +Because the header is honored from any peer, only enable this when the Postgres port is reachable solely through the load balancer, and never use `client_ip` for security decisions. + +This is a separate toggle from `proxy.proxy_protocol` — enabling one does not enable the other, since the two listeners are typically fronted by different load balancers (or only one of them is exposed at all). + --- ## credentials @@ -374,25 +416,6 @@ network: Patterns support glob syntax. Port numbers are stripped before matching. -### network.proxy_protocol - -Parse PROXY protocol v1/v2 headers on the proxy listener to recover the real client address behind a TCP-terminating load balancer. - -```yaml -network: - proxy_protocol: true -``` - -- **Type:** `bool` -- **Required:** No -- **Default:** `false` - -When enabled, each inbound connection to the proxy listener is checked for a leading PROXY protocol header — as prepended by a TCP load balancer such as GCP's global TCP Proxy load balancer — and, if present, the header's advertised source address replaces the raw TCP peer address as the connection's client address. This flows through to the `client_ip` request-log attribute on every request path, including CONNECT-intercepted TLS traffic. - -Connections that open without a PROXY header, or that don't send one within a 10s read timeout, fall back to the raw TCP peer address instead of being rejected (fail-open), so load balancer health checks and direct probes of the port keep working. - -Because the header is honored from any peer, a client that can reach the listener directly (bypassing the load balancer) can forge its logged `client_ip` by prepending its own PROXY header. Only enable this when the port is reachable solely through the load balancer, and never use `client_ip` for security decisions. The postgres data-plane listener does not parse PROXY protocol headers — it is a separate listener and port. - --- ## log diff --git a/examples/gatekeeper-postgres.yaml b/examples/gatekeeper-postgres.yaml index 3b97aed..7180eef 100644 --- a/examples/gatekeeper-postgres.yaml +++ b/examples/gatekeeper-postgres.yaml @@ -43,6 +43,11 @@ proxy: postgres: host: 127.0.0.1 # optional; defaults to the proxy host port: 5432 + # Parse PROXY protocol v1/v2 headers from a fronting TCP load balancer so + # the client_ip log attribute records the real client address. Fail-open: + # connections without a header use the TCP peer address. Only enable when + # the port is reachable solely through the load balancer. + # proxy_protocol: true tls: ca_cert: ca.crt diff --git a/examples/gatekeeper.yaml b/examples/gatekeeper.yaml index 5847db0..754c58c 100644 --- a/examples/gatekeeper.yaml +++ b/examples/gatekeeper.yaml @@ -13,6 +13,11 @@ proxy: host: 127.0.0.1 port: 9080 + # Parse PROXY protocol v1/v2 headers from a fronting TCP load balancer so + # the client_ip log attribute records the real client address. Fail-open: + # connections without a header use the TCP peer address. Only enable when + # the port is reachable solely through the load balancer. + # proxy_protocol: true tls: ca_cert: ca.crt @@ -40,11 +45,6 @@ credentials: network: policy: permissive - # Parse PROXY protocol v1/v2 headers from a fronting TCP load balancer so - # the client_ip log attribute records the real client address. Fail-open: - # connections without a header use the TCP peer address. Only enable when - # the port is reachable solely through the load balancer. - # proxy_protocol: true log: level: info diff --git a/gatekeeper.go b/gatekeeper.go index 3af549b..e3295ec 100644 --- a/gatekeeper.go +++ b/gatekeeper.go @@ -29,7 +29,6 @@ import ( "github.com/majorcontext/gatekeeper/credentialsource" "github.com/majorcontext/gatekeeper/proxy" - "github.com/pires/go-proxyproto" "go.opentelemetry.io/contrib/bridges/otelslog" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/trace" @@ -815,30 +814,12 @@ func (s *Server) Start(ctx context.Context) error { // address; wrapping the listener here rewrites each accepted conn's // RemoteAddr() to that address before http.Server (and therefore every // request-logging path, including CONNECT-intercepted inner requests, - // which all log from the outer request's RemoteAddr) ever sees it. - // - // The policy is pinned to USE (rather than left at the library default) - // so a connection without a PROXY header still passes through using its - // real TCP peer address instead of being rejected — fail-open, so LB - // health checks and direct probes of the port keep working. - if s.cfg.Network.ProxyProtocol { - ln = &proxyproto.Listener{ - Listener: ln, - ReadHeaderTimeout: 10 * time.Second, - ConnPolicy: func(proxyproto.ConnPolicyOptions) (proxyproto.Policy, error) { - return proxyproto.USE, nil - }, - } - // go-proxyproto has no error-callback hook for header parse failures - // in this version: ValidateHeader only runs against a *successfully* - // parsed header, and header parsing itself is lazy — it happens - // inside the returned Conn on the first Read/RemoteAddr, not in - // Accept. A malformed header (as opposed to a merely absent one, - // which is the correct, silent USE-policy fallback) therefore - // surfaces only as an error from Conn.Read, which net/http treats as - // a dead connection and closes without a trace. Wrap the listener so - // that case gets one DEBUG log line instead of vanishing silently. - ln = &proxyProtoLogListener{Listener: ln} + // which all log from the outer request's RemoteAddr) ever sees it. See + // proxy.WrapProxyProtocolListener for the fail-open USE policy, the 10s + // header-read timeout, and the malformed-header debug log — the Postgres + // data-plane listener below shares this exact same helper. + if s.cfg.Proxy.ProxyProtocol { + ln = proxy.WrapProxyProtocolListener(ln) } slog.Info("gatekeeper listening", "addr", ln.Addr().String(), "version", s.version) @@ -865,9 +846,9 @@ func (s *Server) Start(ctx context.Context) error { if pgHost == "" { pgHost = host // same default the HTTP listener resolved } - pg := proxy.NewPostgresServer(s.proxy) pgAddr := fmt.Sprintf("%s:%d", pgHost, s.cfg.Postgres.Port) - if err := pg.Start(pgAddr); err != nil { + pgLn, err := net.Listen("tcp", pgAddr) + if err != nil { // Tear down the already-running HTTP server so a postgres bind // failure doesn't leak the HTTP listener. http.Server.Shutdown // closes the listener it was Serve-ing, so closing the server is @@ -877,6 +858,23 @@ func (s *Server) Start(ctx context.Context) error { cancel() return fmt.Errorf("starting postgres listener: %w", err) } + // Wrap before StartListener: the PROXY header arrives as the very + // first bytes on the wire, ahead of the client's SSLRequest, so the + // listener must be PROXY-protocol-aware from the first Accept. See + // proxy.WrapProxyProtocolListener — same fail-open USE policy, 10s + // header-read timeout, and malformed-header debug log as the HTTP + // listener above. + if s.cfg.Postgres.ProxyProtocol { + pgLn = proxy.WrapProxyProtocolListener(pgLn) + } + pg := proxy.NewPostgresServer(s.proxy) + if err := pg.StartListener(pgLn); err != nil { + _ = pgLn.Close() + shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + _ = s.proxyServer.Shutdown(shutdownCtx) + cancel() + return fmt.Errorf("starting postgres listener: %w", err) + } s.mu.Lock() s.pgServer = pg s.pgAddr = pg.Addr() @@ -900,52 +898,6 @@ func (s *Server) Start(ctx context.Context) error { return s.Stop(shutdownCtx) } -// proxyProtoLogListener wraps a *proxyproto.Listener so that a connection -// whose PROXY header fails to parse gets a single DEBUG log line before it's -// dropped. See the comment where this is constructed in Start for why the -// hook is needed at all. -type proxyProtoLogListener struct { - net.Listener -} - -func (l *proxyProtoLogListener) Accept() (net.Conn, error) { - conn, err := l.Listener.Accept() - if err != nil { - return nil, err - } - // Prefer the raw underlying TCP conn's address over conn.RemoteAddr(): - // on a *proxyproto.Conn, RemoteAddr() itself triggers header processing, - // which would force a blocking header read here in Accept before the - // peer has necessarily sent anything. Raw() has no such side effect. - peer := conn.RemoteAddr() - if pc, ok := conn.(*proxyproto.Conn); ok { - peer = pc.Raw().RemoteAddr() - } - return &proxyProtoLogConn{Conn: conn, peer: peer}, nil -} - -// proxyProtoLogConn wraps an accepted connection to detect and log genuine -// PROXY header parse failures. Header parsing is lazy: it happens inside the -// wrapped proxyproto.Conn on the first Read, not in Accept, and a parse -// failure surfaces only as an error from that Read. A connection that simply -// has no PROXY header at all is not an error here (proxyproto's USE policy -// falls back to the real peer address for it) and must stay quiet. -type proxyProtoLogConn struct { - net.Conn - peer net.Addr - once sync.Once -} - -func (c *proxyProtoLogConn) Read(b []byte) (int, error) { - n, err := c.Conn.Read(b) - if err != nil && !errors.Is(err, proxyproto.ErrNoProxyProtocol) && strings.HasPrefix(err.Error(), "proxyproto:") { - c.once.Do(func() { - slog.Debug("dropping connection: malformed PROXY protocol header", "peer", c.peer.String(), "err", err) - }) - } - return n, err -} - // Stop gracefully shuts down the proxy server and all background refresh goroutines. func (s *Server) Stop(ctx context.Context) error { if s.refreshCancel != nil { diff --git a/gatekeeper_test.go b/gatekeeper_test.go index dc149e8..d086981 100644 --- a/gatekeeper_test.go +++ b/gatekeeper_test.go @@ -703,7 +703,7 @@ func captureServerLog(t *testing.T, srv *Server) func() proxy.RequestLogData { // runs behind a GCP global TCP Proxy load balancer, which terminates the // client TCP connection and dials gatekeeper from its own front-end IP // (35.191.0.0/16), prepending a PROXY protocol v1 header naming the real -// client. With network.proxy_protocol enabled, the canonical log line's +// client. With proxy.proxy_protocol enabled, the canonical log line's // ClientAddr must reflect that real client — here a spoofed Modal egress IP, // 100.52.56.181 — not the load balancer's peer address. func TestServerProxyProtocol_V1(t *testing.T) { @@ -714,8 +714,8 @@ func TestServerProxyProtocol_V1(t *testing.T) { backendURL, _ := url.Parse(backend.URL) cfg := &Config{ - Proxy: ProxyConfig{Port: 0, Host: "127.0.0.1"}, - Network: NetworkConfig{Policy: "permissive", ProxyProtocol: true}, + Proxy: ProxyConfig{Port: 0, Host: "127.0.0.1", ProxyProtocol: true}, + Network: NetworkConfig{Policy: "permissive"}, } srv, err := New(context.Background(), cfg, "") if err != nil { @@ -774,8 +774,8 @@ func TestServerProxyProtocol_V2(t *testing.T) { backendURL, _ := url.Parse(backend.URL) cfg := &Config{ - Proxy: ProxyConfig{Port: 0, Host: "127.0.0.1"}, - Network: NetworkConfig{Policy: "permissive", ProxyProtocol: true}, + Proxy: ProxyConfig{Port: 0, Host: "127.0.0.1", ProxyProtocol: true}, + Network: NetworkConfig{Policy: "permissive"}, } srv, err := New(context.Background(), cfg, "") if err != nil { @@ -828,14 +828,14 @@ func TestServerProxyProtocol_V2(t *testing.T) { } // TestServerProxyProtocol_FailSafeNoHeader verifies that a connection with no -// PROXY header still succeeds when network.proxy_protocol is enabled — the LB's +// PROXY header still succeeds when proxy.proxy_protocol is enabled — the LB's // own health checks and any direct probe of the port do not send one, and // must not be rejected. func TestServerProxyProtocol_FailSafeNoHeader(t *testing.T) { logPath := filepath.Join(t.TempDir(), "gatekeeper.log") cfg := &Config{ - Proxy: ProxyConfig{Port: 0, Host: "127.0.0.1"}, - Network: NetworkConfig{Policy: "permissive", ProxyProtocol: true}, + Proxy: ProxyConfig{Port: 0, Host: "127.0.0.1", ProxyProtocol: true}, + Network: NetworkConfig{Policy: "permissive"}, Log: LogConfig{Level: "debug", Output: logPath}, } srv, err := New(context.Background(), cfg, "") @@ -878,7 +878,7 @@ func TestServerProxyProtocol_FailSafeNoHeader(t *testing.T) { } // TestServerProxyProtocol_MalformedHeader verifies that when -// network.proxy_protocol is enabled, a connection that opens with something +// proxy.proxy_protocol is enabled, a connection that opens with something // that looks like a PROXY header but fails to parse (as opposed to one that // simply lacks a header entirely — see TestServerProxyProtocol_FailSafeNoHeader, // which must stay quiet) is dropped AND logged at DEBUG level with the real @@ -890,8 +890,8 @@ func TestServerProxyProtocol_FailSafeNoHeader(t *testing.T) { func TestServerProxyProtocol_MalformedHeader(t *testing.T) { logPath := filepath.Join(t.TempDir(), "gatekeeper.log") cfg := &Config{ - Proxy: ProxyConfig{Port: 0, Host: "127.0.0.1"}, - Network: NetworkConfig{Policy: "permissive", ProxyProtocol: true}, + Proxy: ProxyConfig{Port: 0, Host: "127.0.0.1", ProxyProtocol: true}, + Network: NetworkConfig{Policy: "permissive"}, Log: LogConfig{Level: "debug", Output: logPath}, } srv, err := New(context.Background(), cfg, "") @@ -949,7 +949,7 @@ func TestServerProxyProtocol_MalformedHeader(t *testing.T) { } // TestServerProxyProtocol_DisabledDefault verifies that behavior is -// unchanged when network.proxy_protocol is left unset (the default): the +// unchanged when proxy.proxy_protocol is left unset (the default): the // canonical log line's ClientAddr is the raw TCP peer address, exactly as // before this feature existed. func TestServerProxyProtocol_DisabledDefault(t *testing.T) { @@ -1013,12 +1013,12 @@ func TestServerProxyProtocol_ConnectIntercepted(t *testing.T) { })) cfg := &Config{ - Proxy: ProxyConfig{Port: 0, Host: "127.0.0.1"}, + Proxy: ProxyConfig{Port: 0, Host: "127.0.0.1", ProxyProtocol: true}, TLS: TLSConfig{ CACert: filepath.Join(caDir, "ca.crt"), CAKey: filepath.Join(caDir, "ca.key"), }, - Network: NetworkConfig{Policy: "permissive", ProxyProtocol: true}, + Network: NetworkConfig{Policy: "permissive"}, } srv, err := New(context.Background(), cfg, "") if err != nil { @@ -2802,6 +2802,133 @@ func TestServerStartsPostgresListener(t *testing.T) { } } +// TestServerPostgresProxyProtocol verifies the postgres.proxy_protocol config +// field wires the Postgres data-plane listener through the same +// proxy.WrapProxyProtocolListener helper the HTTP listener uses: a PROXY +// protocol v1 header sent as the very first bytes — ahead of the client's +// SSLRequest — is parsed before the Postgres handshake begins, and the +// canonical log line's ClientAddr reflects the header's advertised address +// rather than the raw loopback peer the test dialed from. +func TestServerPostgresProxyProtocol(t *testing.T) { + caDir := t.TempDir() + ca, err := proxy.NewCA(caDir) + if err != nil { + t.Fatalf("NewCA: %v", err) + } + caPool := x509.NewCertPool() + if !caPool.AppendCertsFromPEM(ca.CertPEM()) { + t.Fatal("failed to add CA cert to pool") + } + + cfg := &Config{ + Proxy: ProxyConfig{Port: 0, Host: "127.0.0.1"}, + TLS: TLSConfig{ + CACert: filepath.Join(caDir, "ca.crt"), + CAKey: filepath.Join(caDir, "ca.key"), + }, + Postgres: &PostgresConfig{Port: 0, ProxyProtocol: true}, + Credentials: []CredentialConfig{ + { + Host: "*.neon.tech", + Postgres: &PostgresCredentialConfig{Resolver: "static"}, + Source: SourceConfig{Type: "static", Value: "pw"}, + }, + }, + } + + srv, err := New(context.Background(), cfg, "") + if err != nil { + t.Fatalf("New: %v", err) + } + waitLog := captureServerLog(t, srv) + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + go func() { _ = srv.Start(ctx) }() + + // Poll until the postgres listener has an address. + deadline := time.Now().Add(2 * time.Second) + var pgAddr string + for { + pgAddr = srv.PostgresAddr() + if pgAddr != "" { + break + } + if time.Now().After(deadline) { + t.Fatal("postgres listener did not start in time") + } + time.Sleep(10 * time.Millisecond) + } + + conn, err := net.DialTimeout("tcp", pgAddr, time.Second) + if err != nil { + t.Fatalf("dial postgres listener: %v", err) + } + defer conn.Close() + conn.SetDeadline(time.Now().Add(2 * time.Second)) + + if _, err := conn.Write([]byte("PROXY TCP4 100.52.56.181 10.0.0.1 51234 5432\r\n")); err != nil { + t.Fatalf("write PROXY header: %v", err) + } + + // The listener still speaks Postgres normally after the header: an + // SSLRequest gets a single 'S' byte back. + frontend := pgproto3.NewFrontend(conn, conn) + frontend.Send(&pgproto3.SSLRequest{}) + if err := frontend.Flush(); err != nil { + t.Fatalf("send SSLRequest: %v", err) + } + buf := make([]byte, 1) + if _, err := io.ReadFull(conn, buf); err != nil { + t.Fatalf("read SSLRequest response: %v", err) + } + if buf[0] != 'S' { + t.Fatalf("SSLRequest response = %q, want 'S'", buf[0]) + } + + // Complete the rest of the handshake so the connection reaches + // serveAuthenticated, which is where ClientAddr gets logged: TLS with the + // real CA-issued certificate for the SNI host, a startup message, then any + // password (standalone mode with no proxy.auth_token configured accepts + // every token — "localhost trust"). db.test.local matches no configured + // credential host, so the connection is denied for lack of a resolver — + // but only after the deny path logs ClientAddr, which is all this test + // needs. + tlsConn := tls.Client(conn, &tls.Config{ServerName: "db.test.local", RootCAs: caPool}) + if err := tlsConn.Handshake(); err != nil { + t.Fatalf("TLS handshake: %v", err) + } + defer tlsConn.Close() + + fe := pgproto3.NewFrontend(tlsConn, tlsConn) + fe.Send(&pgproto3.StartupMessage{ + ProtocolVersion: pgproto3.ProtocolVersionNumber, + Parameters: map[string]string{"user": "app", "database": "appdb"}, + }) + if err := fe.Flush(); err != nil { + t.Fatalf("send startup: %v", err) + } + if _, err := fe.Receive(); err != nil { + t.Fatalf("receive auth request: %v", err) + } + fe.Send(&pgproto3.PasswordMessage{Password: "any-token"}) + if err := fe.Flush(); err != nil { + t.Fatalf("send password: %v", err) + } + if _, err := fe.Receive(); err != nil { + t.Fatalf("receive auth result: %v", err) + } + + logged := waitLog() + host, _, err := net.SplitHostPort(logged.ClientAddr) + if err != nil { + t.Fatalf("ClientAddr = %q: SplitHostPort: %v", logged.ClientAddr, err) + } + if host != "100.52.56.181" { + t.Errorf("ClientAddr host = %q, want 100.52.56.181 (the PROXY-header source), not the raw TCP peer address", host) + } +} + func TestServerPostgresStartFailureCleansUpHTTP(t *testing.T) { // Occupy a port so the postgres listener fails to bind to it. occupied, err := net.Listen("tcp", "127.0.0.1:0") diff --git a/proxy/postgres.go b/proxy/postgres.go index b3ad7f1..5b656a0 100644 --- a/proxy/postgres.go +++ b/proxy/postgres.go @@ -31,7 +31,7 @@ const postgresKeepAlivePeriod = 30 * time.Second // enableKeepAlive turns on TCP keep-alives for a relayed connection. It is a // no-op for non-TCP connections (e.g. test pipes). func enableKeepAlive(conn net.Conn) { - tc, ok := conn.(*net.TCPConn) + tc, ok := underlyingTCPConn(conn) if !ok { return } @@ -39,6 +39,22 @@ func enableKeepAlive(conn net.Conn) { _ = tc.SetKeepAlivePeriod(postgresKeepAlivePeriod) } +// underlyingTCPConn returns the real *net.TCPConn beneath conn, unwrapping one +// layer through a Raw() accessor first. When the listener is wrapped with +// WrapProxyProtocolListener, an accepted conn is not itself a *net.TCPConn — +// it's a proxyProtoLogConn whose Raw() reaches down through the proxyproto.Conn +// to the transport socket. Raw() never touches the PROXY header (unlike +// RemoteAddr()/Read()), so unwrapping here is side-effect free and safe to call +// before any protocol bytes are read. Returns (nil, false) for a non-TCP conn +// (e.g. a test pipe) so keep-alive setup is skipped. +func underlyingTCPConn(conn net.Conn) (*net.TCPConn, bool) { + if rc, ok := conn.(interface{ Raw() net.Conn }); ok { + conn = rc.Raw() + } + tc, ok := conn.(*net.TCPConn) + return tc, ok +} + // postgresDefaultPort is the port used when matching Postgres host patterns. const postgresDefaultPort = 5432 @@ -452,6 +468,21 @@ func (s *PostgresServer) Start(addr string) error { if err != nil { return fmt.Errorf("postgres listen: %w", err) } + return s.startListener(ln) +} + +// StartListener begins accepting connections on a pre-created listener +// instead of binding one itself, so the caller can wrap it first — e.g. with +// WrapProxyProtocolListener, to add PROXY protocol support (see +// PostgresConfig.ProxyProtocol). It performs the same CA check as Start. +func (s *PostgresServer) StartListener(ln net.Listener) error { + if s.proxy.ca == nil { + return errors.New("postgres listener requires a CA for TLS termination") + } + return s.startListener(ln) +} + +func (s *PostgresServer) startListener(ln net.Listener) error { s.listener = ln go s.acceptLoop(ln) return nil diff --git a/proxy/postgres_test.go b/proxy/postgres_test.go index 206872c..ef92962 100644 --- a/proxy/postgres_test.go +++ b/proxy/postgres_test.go @@ -230,6 +230,25 @@ func newTestPostgresListener(t *testing.T, p *Proxy) *PostgresServer { return srv } +// newTestPostgresListenerWithProxyProtocol is newTestPostgresListener but +// wraps the listener with WrapProxyProtocolListener first, mirroring how +// gatekeeper.go wires postgres.proxy_protocol: true in production (bind, +// wrap, then StartListener on the wrapped listener). +func newTestPostgresListenerWithProxyProtocol(t *testing.T, p *Proxy) *PostgresServer { + t.Helper() + ln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatalf("listen: %v", err) + } + ln = WrapProxyProtocolListener(ln) + srv := NewPostgresServer(p) + if err := srv.StartListener(ln); err != nil { + t.Fatal(err) + } + t.Cleanup(srv.Stop) + return srv +} + // pgClientHandshake dials the listener, does SSLRequest+TLS, sends the startup // message, and answers AuthenticationCleartextPassword with password. It returns // the message received after sending the password (the auth result or an error) @@ -241,6 +260,16 @@ func pgClientHandshake(t *testing.T, addr, sniHost string, caPool *x509.CertPool if err != nil { t.Fatalf("dial: %v", err) } + return pgClientHandshakeOnConn(t, raw, sniHost, caPool, user, db, password) +} + +// pgClientHandshakeOnConn is pgClientHandshake but driven over an +// already-established raw connection, letting a caller write bytes ahead of +// the Postgres wire protocol — e.g. a PROXY protocol header — before the +// handshake begins. +func pgClientHandshakeOnConn(t *testing.T, raw net.Conn, sniHost string, caPool *x509.CertPool, user, db, password string) (pgproto3.BackendMessage, net.Conn) { + t.Helper() + _ = raw.SetDeadline(time.Now().Add(10 * time.Second)) // SSLRequest preamble. @@ -402,6 +431,111 @@ func TestPostgresListenerAcceptsGoodTokenButNoResolver(t *testing.T) { } } +// TestPostgresListenerProxyProtocolV1 verifies that when the Postgres +// listener is wrapped with WrapProxyProtocolListener (postgres.proxy_protocol: +// true in gatekeeper.yaml), a leading PROXY protocol v1 header — sent before +// the client's SSLRequest, since the header is always the very first bytes on +// the wire — is honored: the logged ClientAddr reflects the header's +// advertised source address, not the raw TCP loopback peer address the test +// actually dialed from. +func TestPostgresListenerProxyProtocolV1(t *testing.T) { + p, ca := newTestProxyWithCA(t) + p.SetContextResolver(func(token string) (*RunContextData, bool) { + if token == "good-token" { + return &RunContextData{}, true + } + return nil, false + }) + cap := &logCapture{} + p.SetLogger(cap.log) + srv := newTestPostgresListenerWithProxyProtocol(t, p) + + raw, err := net.Dial("tcp", srv.Addr()) + if err != nil { + t.Fatalf("dial: %v", err) + } + if _, err := raw.Write([]byte("PROXY TCP4 100.52.56.181 10.0.0.1 51234 5432\r\n")); err != nil { + raw.Close() + t.Fatalf("write PROXY header: %v", err) + } + + msg, conn := pgClientHandshakeOnConn(t, raw, "db.test.local", caTrustPool(t, ca), "app", "appdb", "good-token") + defer conn.Close() + + // Auth passed; serveAuthenticated denies for lack of a resolver, but it + // logs ClientAddr before that — which is all this test needs. + if _, ok := msg.(*pgproto3.ErrorResponse); !ok { + t.Fatalf("expected ErrorResponse, got %T", msg) + } + + var entries []RequestLogData + deadline := time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + entries = cap.snapshot() + if len(entries) >= 1 { + break + } + time.Sleep(20 * time.Millisecond) + } + if len(entries) != 1 { + t.Fatalf("got %d log entries, want exactly 1", len(entries)) + } + host, _, err := net.SplitHostPort(entries[0].ClientAddr) + if err != nil { + t.Fatalf("ClientAddr = %q: SplitHostPort: %v", entries[0].ClientAddr, err) + } + if host != "100.52.56.181" { + t.Errorf("ClientAddr host = %q, want 100.52.56.181 (the PROXY-header source), not the raw TCP peer address", host) + } +} + +// TestPostgresListenerProxyProtocolFailSafeNoHeader verifies that a +// connection with no PROXY header still succeeds when the Postgres listener +// is wrapped with WrapProxyProtocolListener: the fail-open USE policy falls +// back to the raw TCP peer address instead of rejecting the connection, so a +// direct probe or an LB health check that never speaks PROXY protocol still +// gets a normal Postgres handshake. +func TestPostgresListenerProxyProtocolFailSafeNoHeader(t *testing.T) { + p, ca := newTestProxyWithCA(t) + p.SetContextResolver(func(token string) (*RunContextData, bool) { + if token == "good-token" { + return &RunContextData{}, true + } + return nil, false + }) + cap := &logCapture{} + p.SetLogger(cap.log) + srv := newTestPostgresListenerWithProxyProtocol(t, p) + + // No PROXY header written — straight into the Postgres handshake. + msg, conn := pgClientHandshake(t, srv.Addr(), "db.test.local", caTrustPool(t, ca), "app", "appdb", "good-token") + defer conn.Close() + + if _, ok := msg.(*pgproto3.ErrorResponse); !ok { + t.Fatalf("expected ErrorResponse, got %T", msg) + } + + var entries []RequestLogData + deadline := time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + entries = cap.snapshot() + if len(entries) >= 1 { + break + } + time.Sleep(20 * time.Millisecond) + } + if len(entries) != 1 { + t.Fatalf("got %d log entries, want exactly 1", len(entries)) + } + host, _, err := net.SplitHostPort(entries[0].ClientAddr) + if err != nil { + t.Fatalf("ClientAddr = %q: SplitHostPort: %v", entries[0].ClientAddr, err) + } + if host != "127.0.0.1" { + t.Errorf("ClientAddr host = %q, want 127.0.0.1 (fail-open: no PROXY header present)", host) + } +} + func TestPostgresListenerStaticTokenAuth(t *testing.T) { p, ca := newTestProxyWithCA(t) p.SetAuthToken("static-token") diff --git a/proxy/proxyproto.go b/proxy/proxyproto.go new file mode 100644 index 0000000..caaa580 --- /dev/null +++ b/proxy/proxyproto.go @@ -0,0 +1,108 @@ +package proxy + +import ( + "errors" + "log/slog" + "net" + "strings" + "sync" + "time" + + "github.com/pires/go-proxyproto" +) + +// WrapProxyProtocolListener wraps ln with PROXY protocol v1/v2 parsing, +// matching HAProxy's per-bind `accept-proxy` semantics: a leading PROXY +// header is honored when present, and a connection that doesn't open with +// one falls back to its real TCP peer address (fail-open) rather than being +// rejected — so load balancer health checks and direct probes of the port +// keep working. The header-read is bounded by a 10s timeout, and a +// connection whose header is present but fails to parse is dropped and +// logged once at DEBUG (a connection with no header at all stays silent — +// that's the correct fail-open path, not an error). +// +// Both gatekeeper's HTTP/CONNECT listener and its Postgres data-plane +// listener call this helper so their PROXY protocol handling is +// byte-identical; see ProxyConfig.ProxyProtocol and +// PostgresConfig.ProxyProtocol. +func WrapProxyProtocolListener(ln net.Listener) net.Listener { + ln = &proxyproto.Listener{ + Listener: ln, + ReadHeaderTimeout: 10 * time.Second, + ConnPolicy: func(proxyproto.ConnPolicyOptions) (proxyproto.Policy, error) { + return proxyproto.USE, nil + }, + } + return &proxyProtoLogListener{Listener: ln} +} + +// proxyProtoLogListener wraps a *proxyproto.Listener so that a connection +// whose PROXY header fails to parse gets a single DEBUG log line before it's +// dropped. go-proxyproto has no error-callback hook for header parse +// failures in this version: ValidateHeader only runs against a +// *successfully* parsed header, and header parsing itself is lazy — it +// happens inside the returned Conn on the first Read/RemoteAddr, not in +// Accept. A malformed header (as opposed to a merely absent one, which is +// the correct, silent USE-policy fallback) therefore surfaces only as an +// error from Conn.Read, which callers otherwise treat as a dead connection +// and close without a trace. +type proxyProtoLogListener struct { + net.Listener +} + +func (l *proxyProtoLogListener) Accept() (net.Conn, error) { + conn, err := l.Listener.Accept() + if err != nil { + return nil, err + } + // Capture the raw peer address WITHOUT triggering the lazy PROXY header + // parse. On a *proxyproto.Conn, RemoteAddr() (like Read()) blocks on + // reading the header until the peer sends bytes or the 10s + // ReadHeaderTimeout fires — calling it here would stall the accept loop + // for every new connection behind a single silent client (a slow-loris, + // or an LB TCP health check that opens a socket and waits). Raw() has no + // such side effect, so read the address through it exclusively for a + // proxyproto conn. Only a non-proxyproto conn — which this listener never + // actually wraps, but guard defensively — needs the direct RemoteAddr(). + var peer net.Addr + if pc, ok := conn.(*proxyproto.Conn); ok { + peer = pc.Raw().RemoteAddr() + } else { + peer = conn.RemoteAddr() + } + return &proxyProtoLogConn{Conn: conn, peer: peer}, nil +} + +// proxyProtoLogConn wraps an accepted connection to detect and log genuine +// PROXY header parse failures. Header parsing is lazy: it happens inside the +// wrapped proxyproto.Conn on the first Read, not in Accept, and a parse +// failure surfaces only as an error from that Read. A connection that simply +// has no PROXY header at all is not an error here (proxyproto's USE policy +// falls back to the real peer address for it) and must stay quiet. +type proxyProtoLogConn struct { + net.Conn + peer net.Addr + once sync.Once +} + +func (c *proxyProtoLogConn) Read(b []byte) (int, error) { + n, err := c.Conn.Read(b) + if err != nil && !errors.Is(err, proxyproto.ErrNoProxyProtocol) && strings.HasPrefix(err.Error(), "proxyproto:") { + c.once.Do(func() { + slog.Debug("dropping connection: malformed PROXY protocol header", "peer", c.peer.String(), "err", err) + }) + } + return n, err +} + +// Raw returns the innermost non-PROXY-protocol connection, so a caller that +// needs the true transport conn (e.g. to enable TCP keep-alives via a +// *net.TCPConn type assertion) can get it without triggering a blocking +// PROXY header read: unlike RemoteAddr() or Read(), Raw() never touches the +// header. +func (c *proxyProtoLogConn) Raw() net.Conn { + if pc, ok := c.Conn.(*proxyproto.Conn); ok { + return pc.Raw() + } + return c.Conn +} diff --git a/proxy/proxyproto_test.go b/proxy/proxyproto_test.go new file mode 100644 index 0000000..e6ed4e4 --- /dev/null +++ b/proxy/proxyproto_test.go @@ -0,0 +1,118 @@ +package proxy + +import ( + "net" + "testing" + "time" +) + +// TestWrapProxyProtocolListenerAcceptDoesNotBlockOnSilentClient guards against +// a slow-loris stall of the whole accept loop. WrapProxyProtocolListener's +// Accept must NOT trigger the lazy PROXY header parse: on a *proxyproto.Conn, +// RemoteAddr() (and Read()) block on reading the header until bytes arrive or +// the 10s ReadHeaderTimeout fires. If Accept calls RemoteAddr() on the +// proxyproto conn, a single client that connects and sends nothing (a +// slow-loris, or just an LB TCP health check that opens a socket and waits) +// stalls Accept — and therefore every other pending connection on that +// listener — for the full timeout. Accept must capture the raw peer address +// via Raw().RemoteAddr(), which has no such side effect. +func TestWrapProxyProtocolListenerAcceptDoesNotBlockOnSilentClient(t *testing.T) { + base, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatalf("listen: %v", err) + } + ln := WrapProxyProtocolListener(base) + defer ln.Close() + + // Connect but send nothing — the silent-client / slow-loris case. + dialConn, err := net.Dial("tcp", ln.Addr().String()) + if err != nil { + t.Fatalf("dial: %v", err) + } + defer dialConn.Close() + + accepted := make(chan net.Conn, 1) + errCh := make(chan error, 1) + go func() { + c, err := ln.Accept() + if err != nil { + errCh <- err + return + } + accepted <- c + }() + + // The bug hangs Accept for the full 10s ReadHeaderTimeout; a 2s bound is + // well clear of that and non-flaky, since a correct Accept returns as soon + // as the connection is accepted (no header read at all). + select { + case c := <-accepted: + c.Close() + case err := <-errCh: + t.Fatalf("Accept returned an error: %v", err) + case <-time.After(2 * time.Second): + t.Fatal("Accept blocked on a silent client for >2s: the accept loop stalls until ReadHeaderTimeout because RemoteAddr() triggers the lazy blocking PROXY header read in Accept") + } +} + +// TestUnderlyingTCPConnUnwrapsWrappedListenerConn verifies that a connection +// accepted through WrapProxyProtocolListener — a proxyProtoLogConn over a +// proxyproto.Conn over the transport *net.TCPConn — can still be unwrapped to +// its underlying *net.TCPConn via the Raw() accessor, so enableKeepAlive's +// keep-alive settings reach the real socket. Without the unwrap, a direct +// *net.TCPConn assertion fails through the wrapper layers and keep-alive setup +// silently no-ops on a PROXY-protocol-wrapped listener. +func TestUnderlyingTCPConnUnwrapsWrappedListenerConn(t *testing.T) { + base, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatalf("listen: %v", err) + } + ln := WrapProxyProtocolListener(base) + defer ln.Close() + + dialConn, err := net.Dial("tcp", ln.Addr().String()) + if err != nil { + t.Fatalf("dial: %v", err) + } + defer dialConn.Close() + + acceptedCh := make(chan net.Conn, 1) + errCh := make(chan error, 1) + go func() { + c, err := ln.Accept() + if err != nil { + errCh <- err + return + } + acceptedCh <- c + }() + + var accepted net.Conn + select { + case accepted = <-acceptedCh: + case err := <-errCh: + t.Fatalf("Accept returned an error: %v", err) + case <-time.After(2 * time.Second): + t.Fatal("Accept timed out") + } + defer accepted.Close() + + // Precondition: the wrapper layers hide the *net.TCPConn, so a naive direct + // assertion (what enableKeepAlive would do without the unwrap) fails — which + // is exactly why the Raw() unwrap is load-bearing. + if _, ok := accepted.(*net.TCPConn); ok { + t.Fatal("expected the accepted conn to hide its *net.TCPConn behind the proxyproto wrappers") + } + + tc, ok := underlyingTCPConn(accepted) + if !ok { + t.Fatal("underlyingTCPConn did not reach the *net.TCPConn through proxyProtoLogConn + proxyproto.Conn; keep-alives would silently no-op on a PROXY-protocol-wrapped listener") + } + if tc == nil { + t.Fatal("underlyingTCPConn returned a nil *net.TCPConn with ok=true") + } + + // The whole point: enableKeepAlive must run against the real socket without + // panicking on the wrapped conn. + enableKeepAlive(accepted) +}