feat: TrueFoundry resilience pivot — three fault-injection demo scenarios

[henryqingmo]

Summary

• Scenario 1 — MCP server crash: gateway now writes upstream_error to the audit log when the upstream MCP server is unreachable; eval runner accepts policyOutcome: upstream_error
• Scenario 2 — Budget limiter stops retry storm: demonstrated via direct curl loop in demo-resilience.sh; policy gate's existing budgetExceeded fires after 5 failed retries against a downed upstream
• Scenario 3 — Approval flow graceful degradation: gateway now writes expired audit record on approval timeout; APPROVAL_LOCK_TTL env var makes the timeout configurable (set to 15s in compose for demo); mock-slack added to compose stack so Slack outage can be simulated by stopping the service

Changes

• cmd/gateway/server.go — upstream_error audit write on forwarder failure
• cmd/gateway/policy_gate.go — expired audit write on approval timeout
• cmd/gateway/config.go + approval_bridge.go — APPROVAL_LOCK_TTL configurable (default 5m)
• docker-compose.yml + docker-compose.override.yml — mock-slack service, APPROVAL_LOCK_TTL: 15s, SLACK_API_BASE_URL
• evalsuite/resilience.yaml — two new eval cases
• scripts/demo-resilience.sh + Makefile — make demo-resilience target
• cmd/eval-runner/suite.go — upstream_error and required policyOutcome field fixes

Test Plan

• go test ./cmd/gateway/ ./cmd/eval-runner/ -short passes
• docker compose config --quiet parses cleanly
• bash -n scripts/demo-resilience.sh syntax OK
• Full end-to-end: make demo-resilience (requires Docker + ANTHROPIC_API_KEY)

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive resilience demo suite to ToolGate, showcasing fault-tolerance across the proxy layer, policy gate, and approval flows during infrastructure failures. Key changes include logging 'upstream_error' and 'expired' decisions to the audit trail, making the approval lock TTL configurable, adding a mock Slack service, and introducing a new resilience test suite with an orchestrating demo script. The review feedback highlights critical improvements: resolving an unsafe type assertion in the eval runner that could cause a panic, avoiding the global HTTP serve mux for security isolation, replacing a fragile hardcoded sleep in the demo script with dynamic health polling, and addressing macOS compatibility issues with the timeout command.

[gemini-code-assist]

Using http.HandleFunc registers handlers on the global http.DefaultServeMux, which is a security risk as any package in the dependency tree can register routes on it. Additionally, passing nil to http.ListenAndServe uses this global mux.

Consider using a local http.NewServeMux to isolate your routes.

	mux := http.NewServeMux()
	mux.HandleFunc("GET /", func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Content-Type", "text/html; charset=utf-8")
		_, _ = w.Write(uiHTML)
	})

	mux.HandleFunc("POST /run-eval", makeEvalHandler(runner, suite, pool))

	mux.HandleFunc("POST /run-eval/ai", func(w http.ResponseWriter, r *http.Request) {
		if aiRunner == nil {
			http.Error(w, `{"error":"AI_AGENT_URL not configured"}`, http.StatusServiceUnavailable)
			return
		}
		makeEvalHandler(aiRunner, aiSuite, pool)(w, r)
	})

	mux.HandleFunc("POST /run-eval/custom", makeCustomEvalHandler(pool))

	mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
	})

	slog.Info("eval server listening", "port", port)
	return http.ListenAndServe(":"+port, mux)

[Copilot]

Pull request overview

This PR pivots the ToolGate demo toward a “resilience / fault-injection” narrative by adding new gateway audit outcomes (upstream_error, expired), making the approval wait configurable via APPROVAL_LOCK_TTL, and wiring new Docker Compose + eval suites/scripts to demonstrate three failure scenarios (upstream down, retry storm budget limiting, Slack outage).

Changes:

• Gateway: audit upstream_error on pipeline failure during tools/call, and audit expired when approval wait times out; make approval wait duration configurable (APPROVAL_LOCK_TTL).
• Evals: add a resilience suite, extend eval-runner to accept upstream_error, and add an optional --serve mode with a small web UI.
• Demo/infra: add mock-slack + env wiring in compose and a make demo-resilience script/target to orchestrate the three scenarios.

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
scripts/demo-resilience.sh	New end-to-end fault-injection demo script (3 scenarios).
policy.yaml	Updates policy rules/tool set and keeps budgets/default deny.
Makefile	Adds demo-resilience target.
examples/support-agent/agent.py	Updates dispatch keywords/tool calls for the new demo flow.
evalsuite/resilience.yaml	Adds resilience eval cases (upstream down, approval timeout).
evalsuite/localstripe-agent.yaml	Adds localstripe-oriented eval suite.
evalsuite/default.yaml	Updates default eval suite cases to match new tool flow.
evalsuite/ai-agent.yaml	Adds AI-agent eval suite used by eval server.
docs/superpowers/specs/2026-05-27-truefoundry-resilience-pivot-design.md	Design spec for the resilience pivot.
docs/superpowers/plans/2026-05-27-truefoundry-resilience-pivot.md	Detailed implementation plan/checklist for the pivot.
docker-compose.yml	Adds mock-slack, approval timeout env, and other demo stack wiring.
docker-compose.override.yml	Overrides gateway upstream + depends_on/healthcheck sequencing for localstripe demo.
deploy/docker-compose.yml	Reworks deploy stack (localstripe seed, eval-trigger, eval-server, etc.).
cmd/gateway/server.go	Writes upstream_error audit record on tools/call pipeline error.
cmd/gateway/server_test.go	Adds test ensuring upstream_error audit write happens.
cmd/gateway/policy_gate.go	Writes expired audit record when approval wait times out.
cmd/gateway/policy_gate_test.go	Extends timeout test to assert expired audit record written.
cmd/gateway/main.go	Wires ApprovalLockTTL into approval bridge; wires server audit recorder.
cmd/gateway/config.go	Adds ApprovalLockTTL loaded from APPROVAL_LOCK_TTL (default 5m).
cmd/gateway/config_test.go	Adds tests for APPROVAL_LOCK_TTL parsing/defaulting.
cmd/gateway/approval_bridge.go	Makes approval timeout configurable via constructor parameter.
cmd/gateway/approval_bridge_integration_test.go	Updates bridge constructor call site to new signature.
cmd/eval-runner/ui.html	Adds a Tailwind-based UI for running eval suites via HTTP.
cmd/eval-runner/types.go	Adds JSON tags to API response structs used by the eval server.
cmd/eval-runner/suite.go	Adds upstream_error outcome and suite loading from io.Reader.
cmd/eval-runner/suite_test.go	Adds test for upstream_error suite acceptance.
cmd/eval-runner/serve.go	Adds --serve mode: HTTP endpoints + embedded UI + JSON results.
cmd/eval-runner/main.go	Adds --serve CLI entrypoint handling.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.