Skip to content

[6/6][misc] feat: container lifecycle benchmark for harbor rollouts - #288

Merged
mathewjhan merged 1 commit into
mathew/server-endpointsfrom
mathew/container-lifecycle-bench
Aug 6, 2026
Merged

[6/6][misc] feat: container lifecycle benchmark for harbor rollouts#288
mathewjhan merged 1 commit into
mathew/server-endpointsfrom
mathew/container-lifecycle-bench

Conversation

@mathewjhan

@mathewjhan mathewjhan commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Part of splitting #272 into reviewable pieces (6/6). Exercises everything below it in the stack.

What this adds

A CLI that measures how long the full rollout lifecycle takes on the Harbor backend: submit, container build/start, agent run, verification, teardown. It reports cold and warm timings separately and per-rollout p50/p95/max, which is how regressions in container startup cost were found and fixed during development.

It runs against the bundled bench_harness project by default (a minimal workflow + grader), or against any Harbor task folder via --tasks-dir. Rollout ids carry a per-invocation nonce so concurrent bench runs cannot collide.

Example

# default bench task, 3 runs of 5 concurrent rollouts
uv run benchmarks/container_lifecycle/container_lifecycle_bench.py --runs 3 --concurrency 5

# against a real dataset task, without baking the SDK into the image
uv run benchmarks/container_lifecycle/container_lifecycle_bench.py \
    --tasks-dir harbor-datasets/datasets/algotune/algotune-aes-gcm-encryption \
    --runs 3 --concurrency 5 --no-patch-dockerfile-with-sdk

Output shows per-run wall time, success counts, rollouts/sec, and warm-run latency percentiles.

@mathewjhan mathewjhan changed the title [misc] feat: container lifecycle benchmark for harbor rollouts [6/6][misc] feat: container lifecycle benchmark for harbor rollouts Aug 6, 2026
@mathewjhan
mathewjhan force-pushed the mathew/container-lifecycle-bench branch from 6ecf59f to ea3583c Compare August 6, 2026 22:29
@mathewjhan
mathewjhan force-pushed the mathew/container-lifecycle-bench branch from ea3583c to beddf2c Compare August 6, 2026 22:31
Co-authored-by: Cursor <cursoragent@cursor.com>
@mathewjhan
mathewjhan force-pushed the mathew/container-lifecycle-bench branch from beddf2c to bb73391 Compare August 6, 2026 22:31
@mathewjhan
mathewjhan merged commit 96d2a58 into main Aug 6, 2026
1 check passed
@mathewjhan
mathewjhan deleted the mathew/container-lifecycle-bench branch August 6, 2026 22:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants