Skip to content

feat: reconcile Alpaca against the memory bank at startup - #177

Merged
mohammadp1001 merged 1 commit into
mainfrom
feat/broker-state-reconciliation
Aug 11, 2026
Merged

feat: reconcile Alpaca against the memory bank at startup#177
mohammadp1001 merged 1 commit into
mainfrom
feat/broker-state-reconciliation

Conversation

@mohammadp1001

Copy link
Copy Markdown
Owner

Why

While clearing the stranded SPY position from the 2026-08-10 run, Alpaca turned out to
be holding 113 shares of QQQ ($81,553) that the memory bank had no record of at all.

Nothing noticed, and nothing could:

  • the monitor only ever iterates rows in the bank, so a position the bank has forgotten
    has no stop-loss, no exit window and no owner
  • FR-019 gates a ticker on its open rows, so it cannot block a ticker it has no row for.
    The next run would have opened a second QQQ position on top of the untracked 113
    shares

The two sides drift for ordinary reasons. Recreating the bank is the only way to apply a
schema change (there is no migration tooling), and it wipes local rows while the broker
keeps holding the shares. A crash between placing an order and writing the row leaves the
mirror image.

What this adds

alphoryn run gains a startup reconciliation, step 5, right after the memory bank loads.
It classifies three disagreements:

Kind Meaning
ORPHAN Broker holds it, bank has no open row. Unmonitored exposure.
PHANTOM Bank has an open row, broker holds nothing. The monitor can never close it.
QUANTITY_DRIFT Both hold it, in different sizes.

It warns and continues; it does not block. A check that could itself stop trading
would be a worse problem than the drift it catches, so a broker that cannot be reached is
reported and the run proceeds.

--reconcile flattens the disagreement first. That is the only destructive path, and it
is never a side effect of a plain run.

Design notes

CLOSED_RECONCILED sits outside both the feedback-blocking and feedback-due status
sets.
The ticker is freed, and the feedback agent never sees the position. Its outcome
was never observed, so there is no honest thesis judgment to make.

exit_price is left NULL, not given a value. 0.0 would report a total loss and the
entry price would report a flat trade. Both are inventions; NULL says the one true thing,
which is that nobody knows. This needed a new MemoryBank.mark_position_reconciled
rather than reusing update_position_close, which requires a price.

The bank is written only after the broker confirms its close, so a failed close cannot
manufacture the very drift this exists to catch.

Verified against the live account

Read-only run of the new check against real Alpaca and the real memory bank:

discrepancies: 1

  [ORPHAN] QQQ: Alpaca holds 113, the memory bank has no open position.
           It is unmonitored - no stop-loss, no exit window.

It reports the real orphan and stays silent about SPY, which both sides agree on (1 share
each). Nothing was modified.

Tests

688 passed, 100% coverage, ruff check alphoryn/ tests/ clean. Verified under CI's
credential-less environment (GOOGLE_APPLICATION_CREDENTIALS=/nonexistent/adc.json), not
just locally.

Three test modules that drive alphoryn run gained an autouse fixture stubbing the new
check, for the same reason the telemetry preflight needed one in #174: it builds a real
TradingClient, so leaving it live makes the suite pass or fail on whether the machine
happens to have credentials.

Note on a choice you may want to revisit

Warn-and-continue was chosen deliberately over blocking (exit 5). The tradeoff: an
operator who ignores the warning still runs with unmonitored exposure, and can still stack
a new position on top of an orphan. Blocking would prevent that outright, at the cost of
being unable to trade until the drift is cleared. The --reconcile flag exists so the fix
is one flag away either way.

The 2026-08-10 run surfaced 113 shares of QQQ held at Alpaca that the
memory bank had no record of. Nothing noticed, and nothing could:

- the monitor only ever iterates rows in the bank, so a position the bank
  has forgotten has no stop-loss, no exit window and no owner
- FR-019 gates a ticker on its open rows, so it cannot block a ticker it
  has no row for - the next run would have opened a second QQQ position
  on top of the untracked 113 shares

The two sides drift for ordinary reasons. Recreating the bank is the only
way to apply a schema change, and it wipes local rows while the broker
keeps holding the shares; a crash between placing an order and writing the
row leaves the mirror image.

`alphoryn run` now compares the two at startup (step 5, after the memory
bank loads) and reports three kinds of disagreement: ORPHAN, PHANTOM and
QUANTITY_DRIFT. It warns and continues rather than blocking - a check that
could itself stop trading would be a worse problem than the drift it
catches - and a broker that cannot be reached is reported, not fatal.

`--reconcile` flattens the disagreement first. That is the only
destructive path, and it is never a side effect of a plain run.

Reconciled positions get status CLOSED_RECONCILED, deliberately outside
both the feedback-blocking and feedback-due status sets: the ticker is
freed, and the feedback agent never sees a position whose outcome nobody
observed. exit_price is left NULL for the same reason - 0.0 would report a
total loss and the entry price a flat trade, and both are inventions.

The bank is only written after the broker confirms its close, so a failed
close cannot manufacture the very drift this catches.

Verified against the live account: the check reports the QQQ orphan and
stays silent about SPY, which both sides agree on.
@mohammadp1001
mohammadp1001 merged commit a7b1d55 into main Aug 11, 2026
3 checks passed
@mohammadp1001
mohammadp1001 deleted the feat/broker-state-reconciliation branch August 11, 2026 12:32
mohammadp1001 added a commit that referenced this pull request Aug 13, 2026
The 2026-08-13 run was killed at 17:21 UTC holding 16 shares of XLE whose exit
window had already expired. Nothing closed it, nothing warned, and nothing
recorded it - the fact existed only at the broker until someone thought to look.
The run before it died the same way and only escaped because it happened to be
flat.

There was no signal handling at all, so SIGTERM killed the process outright and
run()'s finally block never executed. Termination now raises RunTerminatedError,
which lets the shutdown path run.

A terminated run does not trade on its way out - closing positions on a signal
would realise them at whatever price the reaper happened to pick. It also does
not drain, because draining waits candle by candle and a process being killed
does not have that time. Instead it exits fast and says exactly what it is
leaving behind, on stderr and as POSITIONS_ABANDONED telemetry: ticker, size,
entry, stop, and window deadline - enough to act on by hand.

Previous handlers are restored on the way out so embedding the scheduler does
not permanently redirect the host process's signals.

This cannot help against a hard kill. SIGKILL and TerminateProcess are not
deliverable to any handler, and startup reconciliation (#177) remains the only
net for that case. What it covers is every orderly stop: Ctrl-C, a supervisor's
SIGTERM, a container shutdown.
mohammadp1001 added a commit that referenced this pull request Aug 13, 2026
The 2026-08-13 run was killed at 17:21 UTC holding 16 shares of XLE whose exit
window had already expired. Nothing closed it, nothing warned, and nothing
recorded it - the fact existed only at the broker until someone thought to look.
The run before it died the same way and only escaped because it happened to be
flat.

There was no signal handling at all, so SIGTERM killed the process outright and
run()'s finally block never executed. Termination now raises RunTerminatedError,
which lets the shutdown path run.

A terminated run does not trade on its way out - closing positions on a signal
would realise them at whatever price the reaper happened to pick. It also does
not drain, because draining waits candle by candle and a process being killed
does not have that time. Instead it exits fast and says exactly what it is
leaving behind, on stderr and as POSITIONS_ABANDONED telemetry: ticker, size,
entry, stop, and window deadline - enough to act on by hand.

Previous handlers are restored on the way out so embedding the scheduler does
not permanently redirect the host process's signals.

This cannot help against a hard kill. SIGKILL and TerminateProcess are not
deliverable to any handler, and startup reconciliation (#177) remains the only
net for that case. What it covers is every orderly stop: Ctrl-C, a supervisor's
SIGTERM, a container shutdown.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant