fix(postgres): surface upstream Postgres errors and log run-token auth failures

CHANGELOG.md

@@ -6,6 +6,10 @@ Gatekeeper is pre-1.0. The configuration schema and credential source interface
 
 ## v0.17.3 — 2026-07-15
 
+### Changed
+
+- **Postgres data-plane failures now leave a legible server-side trail** — the client always receives the same generic `FATAL: could not authenticate to upstream database` for any failure past the run-token boundary, by design (the underlying error could echo the upstream server's identifiers), but two gaps meant the operator had no better signal server-side either. First, when the upstream (Neon) rejected a connection with a Postgres wire `ErrorResponse` — an IP-allowlist denial, `28P01 invalid_password`, or anything else — gatekeeper collapsed it to either the bare `errUpstreamAuthFailed` sentinel or `"upstream error (SQLSTATE %s)"`, discarding the upstream server's own `Message` entirely; an IP-allowlist rejection and a bad password were indistinguishable in the log, and the message text (the most useful part of any Postgres error) never appeared at all. `upstreamErrorResponse` now folds the upstream's `Severity`, `Code`, and `Message` — the upstream *server's* own error fields, never a credential — into the error text at every point gatekeeper reads an `ErrorResponse` from the upstream (`authenticateSCRAM`, `collectPostAuthFrames`), so the existing DEBUG `postgres upstream connection failed` line now carries the real reason. Authentication-failure SQLSTATEs (`28P01`, `28000`) still wrap `errUpstreamAuthFailed` via `%w`, so `connectWithRetry`'s `errors.Is` check and its invalidate-cached-password-and-retry-once behavior are unaffected — only the message the sentinel carries got richer. That log line also gained a `stage` attribute naming which of three failure boundaries the error came from — `resolve` (the credential resolver, e.g. the Neon API, failed before the proxy reached the upstream: missing project, wrong scope, API error), `upstream_auth` (the upstream server rejected the presented credential), and `upstream_connect` (a dial/TLS/SCRAM/protocol failure reaching the upstream) — so a resolution failure, an auth rejection, and a network/TLS failure are each independently filterable rather than collapsed together. Each stage is classified from a wrapped sentinel via `errors.Is` (`errResolvePassword`, `errUpstreamAuthFailed`), not a string prefix; only `errUpstreamAuthFailed` drives `connectWithRetry`'s invalidate-and-retry, so a resolver failure never needlessly invalidates a cached password. Second, a client whose run token failed to authenticate produced zero log output server-side — the `handleConn` boundary just sent `28P01` and returned; a failed connection there was invisible to any dashboard or log search. It now logs a `WARN`-level `postgres run-token authentication failed` line (visible without enabling DEBUG) naming the client address and the SNI host — never the token itself, not even a prefix. Between this and the network-policy denial (already logged at `WARN` via the policy logger), the run-token, network-policy, credential-resolution, and upstream-connect boundaries are now each independently identifiable in the logs
+
 ### Fixed
 
 - **A portless `network.allow` pattern denied every Postgres data-plane connection under strict policy — including the shipped example** — `hostPattern.port == 0` (an unspecified pattern port) is documented and enforced by `matchesPattern` as "match only ports 80 and 443," an HTTP-scheme assumption. The Postgres data plane's policy check (`serveAuthenticated`, `proxy/postgres.go`) called that same HTTP-oriented `matchHost`/`checkNetworkPolicy` with the literal Postgres port (5432), so a pattern like `*.neon.tech` — which matches neither 80 nor 443 — could never match a live Postgres connection: every connection was denied with `FATAL: connection not allowed by network policy` (SQLSTATE 28000), regardless of how the allow list was written. This left the network-policy matcher inconsistent with the credential-resolver matcher for the identical connection: `postgresResolverFromEntries` already defaults an unspecified pattern port to 5432 when matching Postgres credential resolvers, so a host could be accepted for credential resolution and simultaneously rejected by policy. `examples/gatekeeper-postgres.yaml` and `docs/content/guides/13-postgres-neon.md` both document exactly this configuration (`policy: strict` + `allow: ["*.neon.tech"]`), so the shipped example was broken as written. Fixed by giving the Postgres data plane its own matcher, `matchHostPostgres` (`proxy/postgres.go`), and a corresponding `Proxy.checkNetworkPolicyPostgres` — both mirror the existing HTTP-facing functions exactly, but apply the same port-0-defaults-to-5432 override `postgresResolverFromEntries` already uses, so the two matchers can no longer disagree about the same connection. `matchHost`, `matchesPattern`, `checkNetworkPolicy`, and `checkNetworkPolicyForRequest` — the shared matcher and its HTTP/CONNECT callers — are untouched: a portless allow pattern still means "ports 80 and 443 only" for HTTP traffic, and an allow pattern pinned to an explicit non-5432 port still denies Postgres connections, exactly as before

proxy/pgtest_test.go

@@ -91,6 +91,10 @@ type fakePostgresServer struct {
 	// ErrorResponse with this SQLSTATE instead of AuthenticationOk after a
 	// successful SCRAM exchange. Used to exercise non-auth upstream errors.
 	failPostAuthWith string
+	// failPostAuthMessage is the ErrorResponse Message sent alongside
+	// failPostAuthWith. Defaults to a generic placeholder; tests that need to
+	// assert the message is surfaced in gatekeeper's logs set it explicitly.
+	failPostAuthMessage string
 
 	mu        sync.Mutex
 	authOK    int
@@ -110,7 +114,18 @@ func withAuthMechanisms(mechs ...string) fakePostgresOption {
 // withFailPostAuth makes the fake send a FATAL ErrorResponse with the given
 // SQLSTATE instead of AuthenticationOk once SCRAM succeeds.
 func withFailPostAuth(sqlState string) fakePostgresOption {
-	return func(f *fakePostgresServer) { f.failPostAuthWith = sqlState }
+	return withFailPostAuthMessage(sqlState, "simulated post-auth failure")
+}
+
+// withFailPostAuthMessage is like withFailPostAuth but also sets the
+// ErrorResponse Message, letting a test assert that gatekeeper surfaces the
+// upstream server's specific rejection reason (e.g. an IP-allowlist denial)
+// rather than a flattened, message-less error.
+func withFailPostAuthMessage(sqlState, message string) fakePostgresOption {
+	return func(f *fakePostgresServer) {
+		f.failPostAuthWith = sqlState
+		f.failPostAuthMessage = message
+	}
 }
 
 func (f *fakePostgresServer) counts() (authOK, authFail int) {
@@ -240,7 +255,7 @@ func (f *fakePostgresServer) handle(conn net.Conn) {
 		backend.Send(&pgproto3.ErrorResponse{
 			Severity: "FATAL",
 			Code:     f.failPostAuthWith,
-			Message:  "simulated post-auth failure",
+			Message:  f.failPostAuthMessage,
 		})
 		backend.Flush()
 		return

proxy/postgres.go

@@ -49,6 +49,14 @@ const postgresDialTimeout = 10 * time.Second
 // credentials the proxy presented (bad password or unknown role).
 var errUpstreamAuthFailed = errors.New("upstream authentication failed")
 
+// errResolvePassword indicates the credential resolver (e.g. the Neon API)
+// failed to produce an upstream password — the proxy never reached the
+// upstream server. It is deliberately distinct from errUpstreamAuthFailed so
+// the connect path can classify the failure stage in logs and, crucially, so
+// a resolution failure never triggers the invalidate-cached-password-and-retry
+// path reserved for an actual upstream credential rejection.
+var errResolvePassword = errors.New("resolve upstream password failed")
+
 // PostgresCredentialResolver resolves a Postgres password for a specific
 // upstream host, role, and database at connection time. Implementations
 // must never log password values.
@@ -378,13 +386,21 @@ func collectPostAuthFrames(frontend *pgproto3.Frontend) ([][]byte, error) {
 }
 
 // upstreamErrorResponse maps an upstream ErrorResponse to a proxy error.
-// Authentication-failure SQLSTATEs map to errUpstreamAuthFailed; everything
-// else reports the SQLSTATE code only (the message could echo identifiers).
+// Authentication-failure SQLSTATEs wrap errUpstreamAuthFailed (so
+// connectWithRetry's errors.Is check and its invalidate-and-retry-once
+// behavior keep working) with the upstream's severity, SQLSTATE, and
+// message attached; other SQLSTATEs get the same detail without the
+// sentinel. These are the UPSTREAM SERVER's own error fields — safe to log,
+// never a credential — and this is the only place gatekeeper preserves them
+// instead of discarding them, so the resulting error is never returned to
+// the client (see serveAuthenticated's sanitized "could not authenticate to
+// upstream database" reply) but is logged in full server-side.
 func upstreamErrorResponse(e *pgproto3.ErrorResponse) error {
+	detail := fmt.Sprintf("upstream error %s %s: %s", e.Severity, e.Code, e.Message)
 	if isAuthFailureCode(e.Code) {
-		return errUpstreamAuthFailed
+		return fmt.Errorf("%w: %s", errUpstreamAuthFailed, detail)
 	}
-	return fmt.Errorf("upstream error (SQLSTATE %s)", e.Code)
+	return errors.New(detail)
 }
 
 // isAuthFailureCode reports whether code is a SQLSTATE that indicates an
@@ -664,6 +680,13 @@ func (s *PostgresServer) handleConn(conn net.Conn) {
 
 	rc, ok := s.authenticate(pw.Password)
 	if !ok {
+		// The run token itself must never be logged, not even a prefix — only
+		// that authentication at this boundary failed, and where from.
+		slog.Warn("postgres run-token authentication failed",
+			"subsystem", "proxy",
+			"stage", "run_token_auth",
+			"client_addr", conn.RemoteAddr().String(),
+			"host", sniHost)
 		sendPGError(backend, "28P01", "password authentication failed")
 		return
 	}
@@ -780,10 +803,25 @@ func (s *PostgresServer) serveAuthenticated(ctx context.Context, clientConn net.
 	up, grants, err := s.connectWithRetry(ctx, resolver, sniHost, user, database, startupParams)
 	if err != nil {
 		// Never include the underlying error in the client-facing message: it
-		// could echo the upstream server's identifiers. The full error is only
-		// logged at debug level, never with credential values.
+		// could echo the upstream server's identifiers. The full error —
+		// including the upstream server's own ErrorResponse fields when the
+		// failure came from there (see upstreamErrorResponse) — is only logged
+		// at debug level, never with credential values. stage names which of the
+		// three failure boundaries the error came from, so they are never
+		// conflated in the log: resolve (the credential resolver, e.g. the Neon
+		// API, failed before the proxy reached the upstream), upstream_auth (the
+		// upstream server rejected the presented credential), and upstream_connect
+		// (dial/TLS/SCRAM/protocol failure reaching the upstream).
+		stage := "upstream_connect"
+		switch {
+		case errors.Is(err, errUpstreamAuthFailed):
+			stage = "upstream_auth"
+		case errors.Is(err, errResolvePassword):
+			stage = "resolve"
+		}
 		slog.Debug("postgres upstream connection failed",
 			"subsystem", "proxy",
+			"stage", stage,
 			"host", sniHost,
 			"user", user,
 			"error", err)
@@ -842,7 +880,11 @@ func (s *PostgresServer) connectWithRetry(ctx context.Context, resolver Postgres
 	connect := func() (*upstreamConn, error) {
 		password, err := resolver.ResolvePassword(ctx, host, user, database)
 		if err != nil {
-			return nil, fmt.Errorf("resolving postgres password: %w", err)
+			// Wrap both the errResolvePassword sentinel (so the connect path can
+			// classify this as stage=resolve via errors.Is, and so it is never
+			// mistaken for an upstream credential rejection that would invalidate
+			// and retry) and the human-readable context in one error.
+			return nil, fmt.Errorf("resolving postgres password: %w: %w", errResolvePassword, err)
 		}
 		return connectPostgresUpstream(ctx, upstreamParams{
 			dialAddr:          dialAddr,