Add graceful degradation when dependent services are unavailable

[geoffjay]

Context

agentd services communicate via HTTP REST calls (e.g., ask → notify, orchestrator → wrap). Currently, if a downstream service is unavailable, requests fail with connection errors. There's no retry logic, circuit breaking, or graceful fallback behavior in the service layer.

Proposal

Implement resilient inter-service communication with retries, circuit breakers, and fallback behavior.

Acceptance Criteria

• Add retry logic with exponential backoff to all inter-service HTTP clients (notification_client.rs, client.rs in wrap/orchestrator)
• Implement circuit breaker pattern — after N consecutive failures, stop attempting calls for a cooldown period
• Services continue operating in degraded mode when dependencies are down (e.g., ask still answers questions but logs that notification delivery failed)
• Health endpoint reports dependency status (e.g., /health returns { "status": "degraded", "dependencies": { "notify": "unavailable" } })
• Startup does not fail if optional dependencies are unreachable — log warning and continue
• Add configurable timeout, retry count, and circuit breaker thresholds

Relevant Files

• crates/ask/src/notification_client.rs — calls notify service
• crates/orchestrator/src/manager.rs — calls wrap service
• crates/cli/src/client.rs — calls all services
• crates/notify/src/client.rs — HTTP client base
• crates/wrap/src/client.rs — HTTP client base

Notes

• Consider using tower middleware for retry/circuit-breaker on the client side
• The BAML client already has retry logic (crates/baml/src/client.rs) — use as reference pattern
• This pairs well with Add API authentication for inter-service communication #17 (API authentication) and Add integration tests for cross-service communication #7 (integration tests)

Dependencies

Benefits from: #49 (shared HTTP client would be the natural place to add retry/circuit-breaker logic, avoiding duplication across service clients)

Related: #17 (API authentication), #7 (integration tests for cross-service communication)

[geoffjay]

Note

BAML was removed and this issue still references it

[geoffjay]

✅ Triage Complete

Field	Value
Type	enhancement
Complexity	large
Milestone	v1.0.0 — Production Readiness (existing assignment retained)
Labels	enhancement, architecture, complexity:large, needs-tests

Notes:

• No duplicates found.
• enhancement retained — retry logic, circuit breakers, and degraded mode are new capability, not bug fixes.
• Milestone v1.0.0 kept: the milestone description explicitly lists "resilient communication" as a v1.0.0 goal. This issue is the primary implementation of that goal.
• architecture added: cross-service communication patterns, shared HTTP client design, and circuit breaker state management span multiple crates (ask, orchestrator, cli, notify, wrap). This is a cross-cutting infrastructure change.
• Complexity large: the implementation touches 5+ files across 4-5 crates — retry logic with exponential backoff, circuit breaker state machine (with N-failure threshold + cooldown), per-service degraded mode behavior, health endpoint dependency status reporting, and configurable thresholds. The design work alone (circuit breaker state transitions, per-service fallback decisions) is significant. Estimated 200–400+ lines.
• needs-tests added: 6 behavioral acceptance criteria with zero test requirements listed. Retry backoff timing, circuit breaker open/half-open/closed state transitions, degraded mode behavior, and health endpoint response format all need unit/integration test coverage to be trustworthy in production.
• Dependency ordering note: Extract reusable HTTP client pattern from notify crate #49 ("Extract reusable HTTP client pattern from notify crate", v0.4.0) is open and a natural predecessor. If retry/circuit-breaker middleware is implemented once in the shared client from Extract reusable HTTP client pattern from notify crate #49, it avoids duplicating the pattern across all 5 service clients. Recommend Extract reusable HTTP client pattern from notify crate #49 → Add graceful degradation when dependent services are unavailable #57 ordering even though Add graceful degradation when dependent services are unavailable #57 is not formally blocked.
• Acceptance criteria are sufficiently detailed; no clarification needed.