Skip to content

fix: stop truncating proxied requests and harden proxy auth surface - #1

Merged
moveeeax merged 1 commit into
mainfrom
fix/proxy-request-limits-and-auth-hardening
Jul 25, 2026
Merged

fix: stop truncating proxied requests and harden proxy auth surface#1
moveeeax merged 1 commit into
mainfrom
fix/proxy-request-limits-and-auth-hardening

Conversation

@moveeeax

Copy link
Copy Markdown
Owner

This is an authenticating proxy, so the auth and forwarding paths are the product. Reading them turned up one outright bug and a set of hardening gaps on the same code.

The bug: the audit cap was truncating real traffic

audit.max_body_bytes is documented as "max body size stored per side". It was also being used as the limit on the request body actually forwarded to xAI:

reqBody, reqTrunc, err := readLimited(c.Request.Body, u.maxBody)   // audit cap
...
req, _ := http.NewRequestWithContext(..., bytes.NewReader(reqBody)) // forwarded

Any request over 64 KiB — a long conversation, a pasted document — was silently cut mid-JSON and relayed upstream, which answered with a parse error the client had no way to explain. It happened even with audit.enabled=false.

The two limits are now separate:

  • GAP_SERVER_MAX_REQUEST_BYTES (default 10 MiB) bounds what is proxied, and oversized requests get 413 instead of being truncated and forwarded.
  • GAP_AUDIT_MAX_BODY_BYTES clips only what is persisted, and sets request_truncated as before.

Side effect worth noting: the audit row now parses model and stream from the complete body rather than the clipped copy, so those columns stop coming back empty for large requests.

Hardening on the same paths

  • Timing-unsafe admin key compare. got != adminKey returns at the first differing byte, leaking the key through response latency. Both sides are now SHA-256'd and compared with crypto/subtle.ConstantTimeCompare — constant time and independent of key length. An unset or whitespace-only admin key now rejects everyone, where before a caller echoing that same whitespace back was authenticated.
  • Proxy credentials relayed upstream. X-Admin-Key, Proxy-Authorization and Cookie were forwarded to xAI. They are credentials scoped to this proxy; now stripped, along with the remaining RFC 9110 hop-by-hop headers (keep-alive, te, trailer, upgrade).
  • No read/idle timeouts. http.Server set only ReadHeaderTimeout, so a client could dribble out a request body or park idle keep-alive connections indefinitely — a trivial slowloris. ReadTimeout (60s) and IdleTimeout (120s) are now set and configurable, plus an explicit MaxHeaderBytes. WriteTimeout stays zero on purpose; a write deadline would cut SSE streams off mid-response, and that is now commented rather than implied.
  • Cleartext credential endpoints. auth.upstream_base and auth.issuer carry the Grok access token and refresh token respectively. Config now rejects non-HTTPS values for non-loopback hosts. Loopback stays permitted so local mock upstreams keep working.
  • Unbounded metric cardinality. The metrics middleware fell back to the raw URL path for unmatched routes, so any unauthenticated caller could mint one Prometheus series per 404'd URL and walk the process out of memory. Unmatched requests now share a single label value.
  • CORS cache poisoning. Vary: Origin was set only when the origin matched the allowlist, letting a shared cache replay an allowed origin's Access-Control-Allow-Origin to a disallowed one. It is now set on every response when the origin list is restricted.

TLS itself was already fine — no server-side TLS by design (ingress terminates), and the outbound client uses Go defaults with a floor of TLS 1.2.

Tests

internal/middleware had no tests at all; it now has the admin-auth accept/reject matrix and the CORS cases. Added proxy tests for the truncation regression, the 413 path, the audit-clipping split and header stripping, and config tests for the HTTPS enforcement and the new defaults.

Each new test was confirmed to fail with only its fix reverted:

Reverted fix Failure
request limit coupled to audit cap TestProxyRejectsOversizedRequestBody: status=200, want 413
header stripping TestProxyStripsClientCredentialHeaders: X-Admin-Key leaked upstream as "admin-secret"
== admin compare TestAdminAuthRejectsEverythingWhenKeyUnset: admin_key=" " sent=" ": expected 401, got 200
CORS Vary TestCORSAlwaysVariesOnOriginWhenRestricted: origin="https://evil.example.com": Vary="", want Origin
upstream scheme check TestValidateRejectsCleartextCredentialEndpoints: http_upstream_remote, bad_scheme, relative, http_issuer_remote

CI

Only the Docker image build ran on PRs — nothing executed the test suite. Added .github/workflows/ci.yml running gofmt, go mod tidy verification, build, vet, test and test -race. Three files were already not gofmt-clean and are formatted here so the gate passes (whitespace-only; struct field alignment in store.go, store_test.go, metrics.go).

Verified locally

Go 1.26.5, all in the foreground:

gofmt -l .        → (empty)
go build ./...    → OK
go vet ./...      → OK
go test ./...     → ok: auth, config, middleware, proxy, server, store
go test -race ./... → ok: auth 2.6s, config 1.9s, middleware 3.2s, proxy 3.9s, server 9.3s, store 8.3s
go mod tidy       → go.mod/go.sum unchanged
helm template deploy/helm/grok-auth-proxy --set admin.value=dummy → renders, new GAP_SERVER_* keys present

No breaking changes: the three new config keys are additive with defaults matching prior behaviour, and no exported function signature changed except proxy.Options, which gains an optional field.

The one behaviour change a consumer could notice is the HTTPS requirement on auth.upstream_base / auth.issuer — a deployment pointing at a remote plaintext HTTP upstream will now fail to start rather than leak its token on the wire. Loopback is exempt.

🤖 Generated with Claude Code

The audit body cap was doing double duty as the limit on the request body
actually forwarded upstream. Any request over GAP_AUDIT_MAX_BODY_BYTES
(default 64 KiB) was silently cut mid-JSON and relayed to xAI, which
answered with an unexplainable parse error. The two limits are now
separate: GAP_SERVER_MAX_REQUEST_BYTES (default 10 MiB) bounds what is
proxied and returns 413 above it, while the audit cap only clips what is
persisted. As a side effect the audit row's model/stream fields are now
parsed from the complete body instead of the clipped copy.

Security hardening on the same paths:

- AdminAuth compared the admin key with a byte-wise !=, which returns on
  the first differing byte and leaks the key through response latency.
  Both sides are now SHA-256'd and compared with
  crypto/subtle.ConstantTimeCompare, so the compare is constant time and
  length independent. An unset or whitespace-only admin key now rejects
  every caller instead of authenticating one that echoes it back.
- The proxy relayed X-Admin-Key, Proxy-Authorization and Cookie to xAI.
  Those are credentials scoped to this proxy and are now stripped, along
  with the remaining RFC 9110 hop-by-hop headers.
- http.Server had no ReadTimeout and no IdleTimeout, so a client could
  dribble out a body or park idle keep-alive connections forever. Both
  are now set and configurable. WriteTimeout stays zero on purpose for
  SSE streams.
- config rejects a non-HTTPS auth.upstream_base or auth.issuer for
  non-loopback hosts; those requests carry the Grok access token and
  refresh token, which must not cross the network in cleartext.
- The metrics middleware labelled unmatched routes with the raw URL
  path, letting any caller mint one Prometheus series per 404'd URL and
  exhaust process memory. Unmatched requests now share one label value.
- CORS sets Vary: Origin on every response when the origin list is
  restricted, not just on a match, so a shared cache cannot replay an
  allowed origin's Access-Control-Allow-Origin to a disallowed one.

Adds middleware tests (none existed) plus proxy and config tests; each
new test was confirmed to fail with its fix reverted. Adds a CI workflow
running gofmt/tidy/build/vet/test/test -race, since only the Docker image
build ran on PRs before, and gofmt-formats the three files that gate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@moveeeax
moveeeax merged commit 3c06952 into main Jul 25, 2026
2 checks passed
@moveeeax
moveeeax deleted the fix/proxy-request-limits-and-auth-hardening branch July 25, 2026 19:59
moveeeax added a commit that referenced this pull request Jul 26, 2026
The audit body cap was doing double duty as the limit on the request body
actually forwarded upstream. Any request over GAP_AUDIT_MAX_BODY_BYTES
(default 64 KiB) was silently cut mid-JSON and relayed to xAI, which
answered with an unexplainable parse error. The two limits are now
separate: GAP_SERVER_MAX_REQUEST_BYTES (default 10 MiB) bounds what is
proxied and returns 413 above it, while the audit cap only clips what is
persisted. As a side effect the audit row's model/stream fields are now
parsed from the complete body instead of the clipped copy.

Security hardening on the same paths:

- AdminAuth compared the admin key with a byte-wise !=, which returns on
  the first differing byte and leaks the key through response latency.
  Both sides are now SHA-256'd and compared with
  crypto/subtle.ConstantTimeCompare, so the compare is constant time and
  length independent. An unset or whitespace-only admin key now rejects
  every caller instead of authenticating one that echoes it back.
- The proxy relayed X-Admin-Key, Proxy-Authorization and Cookie to xAI.
  Those are credentials scoped to this proxy and are now stripped, along
  with the remaining RFC 9110 hop-by-hop headers.
- http.Server had no ReadTimeout and no IdleTimeout, so a client could
  dribble out a body or park idle keep-alive connections forever. Both
  are now set and configurable. WriteTimeout stays zero on purpose for
  SSE streams.
- config rejects a non-HTTPS auth.upstream_base or auth.issuer for
  non-loopback hosts; those requests carry the Grok access token and
  refresh token, which must not cross the network in cleartext.
- The metrics middleware labelled unmatched routes with the raw URL
  path, letting any caller mint one Prometheus series per 404'd URL and
  exhaust process memory. Unmatched requests now share one label value.
- CORS sets Vary: Origin on every response when the origin list is
  restricted, not just on a match, so a shared cache cannot replay an
  allowed origin's Access-Control-Allow-Origin to a disallowed one.

Adds middleware tests (none existed) plus proxy and config tests; each
new test was confirmed to fail with its fix reverted. Adds a CI workflow
running gofmt/tidy/build/vet/test/test -race, since only the Docker image
build ran on PRs before, and gofmt-formats the three files that gate.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant