starknet_transaction_prover: /health returns 503 when service is saturated

[avi-starkware]

Adds SaturationMonitor (shared by ProvingRpcServerImpl and
HealthLayer) that tracks whether the concurrency semaphore has been
continuously rejecting proving requests. Once that has held for the
configured window (health_max_saturated_ms, default 10s), /health
returns 503 with an opaque body so load balancers can drain the pod
before in-flight requests start failing.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

[cursor]

PR Summary

Medium Risk
Changes load-balancer health semantics for pod draining; proving path only adds monitor hooks on existing reject/accept paths, but mis-tuned thresholds could drain pods early or late.

Overview
Adds saturation-aware /health so orchestrators can drain pods when the prover stays at its concurrency limit instead of only seeing always-200.

A shared SaturationMonitor records when prove_transaction starts rejecting on the semaphore (mark_rejected) and clears that window on a successful permit (mark_accepted). HealthLayer is wired with that monitor and health_max_saturated_ms (default 10s, overridable via config / --health-max-saturated-ms / HEALTH_MAX_SATURATED_MS); after continuous rejections for that duration, GET /health returns 503 with an opaque {"status":"unhealthy","reason":"saturated"} body. Startup passes a configured HealthLayer through HTTP and HTTPS middleware stacks into ProvingRpcServerImpl, with unit tests for the monitor and health behavior.

^{Reviewed by Cursor Bugbot for commit 5cd174d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[avi-starkware]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• starknet_transaction_prover: document observability surface in README #14203
• starknet_transaction_prover: origin-level breadcrumbs for validation errors #14172
• starknet_transaction_prover: /health returns 503 when service is saturated #14171 👈 (View in Graphite)
• starknet_transaction_prover: bump prover_panics_total on panic hook fire #14170
• starknet_transaction_prover: HTTP request count + latency + in-flight metrics #14169
• starknet_transaction_prover: proving-job duration + outcome metrics #14168
• starknet_transaction_prover: Prometheus /metrics endpoint with build_info #14167
• starknet_transaction_prover: global panic hook + graceful SIGTERM shutdown #14166
• starknet_transaction_prover,tower_ohttp: OHTTP-unlinkable request-id for decapsulated content #14222
• starknet_transaction_prover: per-request structured log with request-id #14165
• starknet_transaction_prover: unify HTTP middleware stack via macro #14286
• main-v0.14.3

This stack of pull requests is managed by Graphite. Learn more about stacking.

[reviewable-StarkWare]

This change is 

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d3f1139. Configure here.}