fix: name the real reason a session failed, and record it - #179
Merged
Conversation
Two sessions in the 2026-08-13 live run failed for two unrelated reasons and both were filed as SKIPPED_DATA_UNAVAILABLE: session-0014 main_agent reply was not JSON session-0019 Vertex AI 429 RESOURCE_EXHAUSTED Market data was fine in both cases. The exception text existed only on stdout, so anyone reading the memory bank afterwards would go and check Alpaca for an outage that never happened. An agent that was reached and gave an unusable answer is now SKIPPED_AGENT_ERROR, and SKIPPED_DATA_UNAVAILABLE means only what its name says. This is the same distinction _handle_overrun_candles already draws for SKIPPED_OVERRUN. The cause travels with the status as SessionSkip(status, detail) rather than a bare string, because keeping them apart is how they drifted in the first place. The detail is persisted to the session's warnings, so a post-mortem can read the real cause out of the bank alone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Two sessions in the 2026-08-13 live run (
run-4) failed, for two completelyunrelated reasons, and both were recorded under a status naming a third thing that
never happened:
run-4/session-0014SKIPPED_DATA_UNAVAILABLErun-4/session-0019429 RESOURCE_EXHAUSTEDSKIPPED_DATA_UNAVAILABLEMarket data was fine in both cases. Worse, the
warningscolumn held only theFR-005 feedback-block notes and a restatement of the status - the actual exception
text appeared nowhere in the memory bank, only on stdout. Anyone reading the
bank later would go check Alpaca for an outage that never happened.
The fix
SKIPPED_AGENT_ERRORis a new status for a failure inside the agent: anunparseable reply, or no reply at all because the provider refused.
SKIPPED_DATA_UNAVAILABLEnow means only what its name says.Cause: ...,so a post-mortem can read the real reason out of the bank alone.
This is the same distinction
_handle_overrun_candlesalready draws forSKIPPED_OVERRUN- three different problems with three different fixes, and filingone as another sends you to the wrong system.
Why
SessionSkiprather than a second return value_run_investigationused to return a bare status string. Status and cause nowtravel together in a frozen
SessionSkip(status, detail), because keeping the twoapart is exactly how they drifted: the status said one thing and the only real
explanation went to stdout and was lost.
Not in scope
The 429 itself deserves backoff, and the root cause of session-0014's unparseable
reply still needs the captured OTel bytes. Both are noted in the handoff as
follow-ups. This PR only stops the bank from lying about which of them happened.
Verification
ruff check alphoryn/ tests/cleanscheduler.py299 stmts / 84 branches, all covered)GOOGLE_APPLICATION_CREDENTIALS=/nonexistent/adc.json python -m pytest -q