Skip to content

[5/6][rollout] feat: server admission control, rollout status, and cancel endpoints - #287

Merged
mathewjhan merged 1 commit into
mathew/harbor-backend-v2from
mathew/server-endpoints
Aug 6, 2026
Merged

[5/6][rollout] feat: server admission control, rollout status, and cancel endpoints#287
mathewjhan merged 1 commit into
mathew/harbor-backend-v2from
mathew/server-endpoints

Conversation

@mathewjhan

@mathewjhan mathewjhan commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Part of splitting #272 into reviewable pieces (5/6). Builds on #286 (backend capabilities).

What this adds

Three server-level behaviors that let a trainer manage load and track rollouts over plain HTTP:

  • Admission control on POST /rollout: when the backend's queue is at max_queue_depth, the server answers 429 with a Retry-After header instead of accepting work it cannot start. Accepted rollouts return 202, which states what actually happens: the rollout is queued and runs after the response is sent.
  • GET /rollout/{id}/status: reports queued, running, or grading for live rollouts, and success / failure / cancelled (with reward and error message) for recently finished ones, retained for a fixed window. Anything else returns unknown. A trainer polls this as a liveness signal instead of trusting a blind timeout.
  • POST /rollout/cancel: takes exactly one selector — {"ids": [...]}, {"prefix": "tenant-a::"}, or {"all": true} — and returns a disposition per rollout (cancelled_queued, cancelled_running, not_found). Prefix cancellation is what a controller uses to stop everything belonging to one adapter when it is deregistered. Cancelling an already-finished rollout returns not_found, so the call is safe to repeat.

Example

curl -X POST $SERVER/rollout -d '{"rollout_id": "tenant-a::r1", ...}'
# -> 202 (or 429 + Retry-After: 5 when the queue is full)

curl $SERVER/rollout/tenant-a::r1/status
# -> {"rollout_id": "tenant-a::r1", "status": "running"}

curl -X POST $SERVER/rollout/cancel -d '{"prefix": "tenant-a::"}'
# -> {"dispositions": {"tenant-a::r1": "cancelled_running"}}

@mathewjhan mathewjhan changed the title [rollout] feat: server admission control, rollout status, and cancel endpoints [5/6][rollout] feat: server admission control, rollout status, and cancel endpoints Aug 6, 2026
@mathewjhan
mathewjhan force-pushed the mathew/server-endpoints branch from 058c5de to dd1535a Compare August 6, 2026 22:29
@mathewjhan
mathewjhan force-pushed the mathew/server-endpoints branch from dd1535a to 21e5bdd Compare August 6, 2026 22:31
…endpoints

Co-authored-by: Cursor <cursoragent@cursor.com>
@mathewjhan
mathewjhan force-pushed the mathew/server-endpoints branch from 21e5bdd to eebfe35 Compare August 6, 2026 22:31
@mathewjhan
mathewjhan merged commit 025e637 into main Aug 6, 2026
1 check passed
@mathewjhan
mathewjhan deleted the mathew/server-endpoints branch August 6, 2026 22:33
@github-actions github-actions Bot added enhancement New feature or request rollout Remote Rollout module labels Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request rollout Remote Rollout module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants