feat: reconcile Alpaca against the memory bank at startup - #177
Merged
Conversation
The 2026-08-10 run surfaced 113 shares of QQQ held at Alpaca that the memory bank had no record of. Nothing noticed, and nothing could: - the monitor only ever iterates rows in the bank, so a position the bank has forgotten has no stop-loss, no exit window and no owner - FR-019 gates a ticker on its open rows, so it cannot block a ticker it has no row for - the next run would have opened a second QQQ position on top of the untracked 113 shares The two sides drift for ordinary reasons. Recreating the bank is the only way to apply a schema change, and it wipes local rows while the broker keeps holding the shares; a crash between placing an order and writing the row leaves the mirror image. `alphoryn run` now compares the two at startup (step 5, after the memory bank loads) and reports three kinds of disagreement: ORPHAN, PHANTOM and QUANTITY_DRIFT. It warns and continues rather than blocking - a check that could itself stop trading would be a worse problem than the drift it catches - and a broker that cannot be reached is reported, not fatal. `--reconcile` flattens the disagreement first. That is the only destructive path, and it is never a side effect of a plain run. Reconciled positions get status CLOSED_RECONCILED, deliberately outside both the feedback-blocking and feedback-due status sets: the ticker is freed, and the feedback agent never sees a position whose outcome nobody observed. exit_price is left NULL for the same reason - 0.0 would report a total loss and the entry price a flat trade, and both are inventions. The bank is only written after the broker confirms its close, so a failed close cannot manufacture the very drift this catches. Verified against the live account: the check reports the QQQ orphan and stays silent about SPY, which both sides agree on.
mohammadp1001
added a commit
that referenced
this pull request
Aug 13, 2026
The 2026-08-13 run was killed at 17:21 UTC holding 16 shares of XLE whose exit window had already expired. Nothing closed it, nothing warned, and nothing recorded it - the fact existed only at the broker until someone thought to look. The run before it died the same way and only escaped because it happened to be flat. There was no signal handling at all, so SIGTERM killed the process outright and run()'s finally block never executed. Termination now raises RunTerminatedError, which lets the shutdown path run. A terminated run does not trade on its way out - closing positions on a signal would realise them at whatever price the reaper happened to pick. It also does not drain, because draining waits candle by candle and a process being killed does not have that time. Instead it exits fast and says exactly what it is leaving behind, on stderr and as POSITIONS_ABANDONED telemetry: ticker, size, entry, stop, and window deadline - enough to act on by hand. Previous handlers are restored on the way out so embedding the scheduler does not permanently redirect the host process's signals. This cannot help against a hard kill. SIGKILL and TerminateProcess are not deliverable to any handler, and startup reconciliation (#177) remains the only net for that case. What it covers is every orderly stop: Ctrl-C, a supervisor's SIGTERM, a container shutdown.
mohammadp1001
added a commit
that referenced
this pull request
Aug 13, 2026
The 2026-08-13 run was killed at 17:21 UTC holding 16 shares of XLE whose exit window had already expired. Nothing closed it, nothing warned, and nothing recorded it - the fact existed only at the broker until someone thought to look. The run before it died the same way and only escaped because it happened to be flat. There was no signal handling at all, so SIGTERM killed the process outright and run()'s finally block never executed. Termination now raises RunTerminatedError, which lets the shutdown path run. A terminated run does not trade on its way out - closing positions on a signal would realise them at whatever price the reaper happened to pick. It also does not drain, because draining waits candle by candle and a process being killed does not have that time. Instead it exits fast and says exactly what it is leaving behind, on stderr and as POSITIONS_ABANDONED telemetry: ticker, size, entry, stop, and window deadline - enough to act on by hand. Previous handlers are restored on the way out so embedding the scheduler does not permanently redirect the host process's signals. This cannot help against a hard kill. SIGKILL and TerminateProcess are not deliverable to any handler, and startup reconciliation (#177) remains the only net for that case. What it covers is every orderly stop: Ctrl-C, a supervisor's SIGTERM, a container shutdown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
While clearing the stranded SPY position from the 2026-08-10 run, Alpaca turned out to
be holding 113 shares of QQQ ($81,553) that the memory bank had no record of at all.
Nothing noticed, and nothing could:
has no stop-loss, no exit window and no owner
The next run would have opened a second QQQ position on top of the untracked 113
shares
The two sides drift for ordinary reasons. Recreating the bank is the only way to apply a
schema change (there is no migration tooling), and it wipes local rows while the broker
keeps holding the shares. A crash between placing an order and writing the row leaves the
mirror image.
What this adds
alphoryn rungains a startup reconciliation, step 5, right after the memory bank loads.It classifies three disagreements:
ORPHANPHANTOMQUANTITY_DRIFTIt warns and continues; it does not block. A check that could itself stop trading
would be a worse problem than the drift it catches, so a broker that cannot be reached is
reported and the run proceeds.
--reconcileflattens the disagreement first. That is the only destructive path, and itis never a side effect of a plain run.
Design notes
CLOSED_RECONCILEDsits outside both the feedback-blocking and feedback-due statussets. The ticker is freed, and the feedback agent never sees the position. Its outcome
was never observed, so there is no honest thesis judgment to make.
exit_priceis left NULL, not given a value.0.0would report a total loss and theentry price would report a flat trade. Both are inventions; NULL says the one true thing,
which is that nobody knows. This needed a new
MemoryBank.mark_position_reconciledrather than reusing
update_position_close, which requires a price.The bank is written only after the broker confirms its close, so a failed close cannot
manufacture the very drift this exists to catch.
Verified against the live account
Read-only run of the new check against real Alpaca and the real memory bank:
It reports the real orphan and stays silent about SPY, which both sides agree on (1 share
each). Nothing was modified.
Tests
688 passed, 100% coverage,
ruff check alphoryn/ tests/clean. Verified under CI'scredential-less environment (
GOOGLE_APPLICATION_CREDENTIALS=/nonexistent/adc.json), notjust locally.
Three test modules that drive
alphoryn rungained an autouse fixture stubbing the newcheck, for the same reason the telemetry preflight needed one in #174: it builds a real
TradingClient, so leaving it live makes the suite pass or fail on whether the machinehappens to have credentials.
Note on a choice you may want to revisit
Warn-and-continue was chosen deliberately over blocking (exit 5). The tradeoff: an
operator who ignores the warning still runs with unmonitored exposure, and can still stack
a new position on top of an orphan. Blocking would prevent that outright, at the cost of
being unable to trade until the drift is cleared. The
--reconcileflag exists so the fixis one flag away either way.