feat(tuner): trustworthy job statuses end-to-end — Ray ground truth in the API, finish-aware stop/delete in the UI - #130
Open
Fidasek009 wants to merge 9 commits into
Open
Conversation
Co-Authored-By: kimi-k3 <noreply@kiloc.ai>
Co-Authored-By: kimi-k3 <noreply@kiloc.ai>
Trials stay PENDING at submission; RUNNING is rendered at read time from Ray task state (State API) instead of an optimistic write, so queued trials no longer masquerade as running. Job status derives from effective trial statuses, and a 2h start watchdog fails batches whose trials never became schedulable instead of hanging forever. Spec: docs/superpowers/specs/2026-08-10-tuner-job-status-design.md Co-Authored-By: kimi-k3 <noreply@kiloc.ai>
With tuner-side statuses now trustworthy, a FINISHED job shows Delete instead of an inert Stop. Pruned (unmeasured) trials are hidden once the job completes, matching the stopped-job view. Co-Authored-By: kimi-k3 <noreply@kiloc.ai>
Fidasek009
changed the title
Fidasek009
commented
Aug 10, 2026
Fidasek009
left a comment
Fidasek009
left a comment
Collaborator
Author
There was a problem hiding this comment.
Review: Ray-ground-truth statuses + watchdog + finish-aware UI. See inline comments.
…y, stale-PENDING sweep - One filtered list_tasks query replaces per-trial get_task probes; query failure is distinguishable from 'nothing running', so a State API outage extends the wait instead of false-killing a healthy batch. - Terminal jobs sweep dangling PENDING trials to ERROR (stall, dead thread, and crash paths alike). - UI clears a selection whose trial is filtered out on job completion. - Probe failures log one warning line, no traceback. Co-Authored-By: kimi-k3 <noreply@kiloc.ai>
Ray SDK address resolution overrides any explicit address with the
RAY_ADDRESS env var, and a ray:// address triggers a full client
connect/disconnect cycle (ray_client_address_to_api_server_url). Those
cycles raced the job thread's ray.init in production: concurrent first
inits corrupt ClientContext ('NoneType' object has no attribute
'connection_info') and a second init hits 'Ray Client is already
connected'.
- Pin RAY_API_SERVER_ADDRESS to the dashboard HTTP URL so State API
queries stay pure-HTTP.
- Serialize ray.init under a lock and drop the unreliable
ray.is_initialized() fast-path in client mode.
- Document client-mode gotchas and e2e validation in tuner/AGENTS.md;
add the conventional CLAUDE.md symlink.
Co-Authored-By: kimi-k3 <noreply@kiloc.ai>
Co-Authored-By: kimi-k3 <noreply@kiloc.ai>
…sive A trial prunes only when it is both below EARLY_STOP_THRESHOLD of the fastest champion AND above EARLY_STOP_COST_RATIO x the cheapest in cost-per-step (footprint hourly rate / steps/sec), so the fastest and the most cost-efficient configs are always kept. Speed threshold hardened from 65% to 75% now that two axes must both fire. Dashboard: a tuner job reporting FINISHED is persisted as stopped (preserving its measured trials) and the tuner job is deleted, so dashboard never re-polls completed jobs. Co-Authored-By: kimi-k3 <noreply@kiloc.ai>
Co-Authored-By: kimi-k3 <noreply@kiloc.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the approved spec (docs/superpowers/specs/2026-08-10-tuner-job-status-design.md) from the tuner quality review, plus follow-ups surfaced by review and live e2e runs. Trial and job RUNNING status now reflect Ray task ground truth instead of optimistic guesses; a start watchdog bounds the one non-terminating code path (indefinite queue wait); early pruning uses two axes (speed AND cost-per-step) so both the fastest and the most cost-efficient config are always kept; and finished jobs are persisted dashboard-side instead of being polled forever.
Changes
_submit_trialsno longer writes RUNNING at submission. Trials stay PENDING until the read path renders them RUNNING — only while Ray's State API reports the task executing. Queued trials correctly show PENDING.ray.waitreturns nothing forTRIAL_START_TIMEOUT_SECONDS(2h, hardcoded), one filteredlist_tasksState API query decides: trials executing (or an HTTP outage — never a false kill) extends the wait; zero RUNNING on a successful query means the batch never became schedulable — futures cancelled, job fails with "cluster busy or unavailable - retry later". Terminal jobs never leave dangling PENDING trials:_error_pending_trialssweeps them to ERROR on every failure path.RAY_ADDRESS, and resolving aray://URL ran a full client connect/disconnect that raced job threads'ray.init. Fixed by pinningRAY_API_SERVER_ADDRESSto the dashboard HTTP URL (State API stays pure-HTTP) and serializing client init under a lock.EARLY_STOP_THRESHOLDof the fastest champion AND costlier per step thanEARLY_STOP_COST_RATIO× the cheapest champion (cost_per_step = footprint.hourly_cost() / steps_per_sec). The two champions are tracked independently, so a slow-but-cheap CPU config keeps exploring while a slow-and-expensive one stops early. Speed threshold hardened 65% → 75% since both axes must now fire.TunerView/AmberTunerView+ tables, unchanged enum/type surface):performance: null) hidden once the job finishes — consistent with the stopped-job view; dedicated empty state when nothing was measured.Testing
_should_early_stopboundaries, champion tracking, and dashboard FINISHED persistence.make fix,make type-check,make testall green (tuner 151, dashboard API 103, remaining harness 62).ruff format --diff/ruff checkclean (the exact CI lint-py commands).make -C tuner e2e ENV=dev): 3/3 passed (~13 min) against the pusheddevimage — real GMX+AMBER jobs through submission, dual-axis pruning paths, polling, and deletion, with zero client-mode errors in the pod logs.Not in scope