Screener live path can shrink the shared daily snapshot to one request's limit

[Kitkitkittt]

Problem Statement

A single screener request can shrink the Universe for every other reader of the Screener Snapshot, for the rest of the day.

The Screener Snapshot table is the api's materialized picture of the Universe — roughly 1,700 symbols, written each morning by the scheduled daily sync. It is not private to the screener. The heatmap, market breadth, and comparison surfaces all read the same table.

The live screener path also writes to it. When a request misses cache and falls through to a live vnstock fetch, whatever rows come back are upserted into the Screener Snapshot under today's date. But a live fetch is bounded by the caller's limit — 100 by default — so it writes at most 100 symbols.

Cache reads for today filter strictly on today's date and apply no minimum-row threshold. So if a live request lands before the daily sync has finished, the table holds only those symbols for today, and every subsequent request gets a cache hit on a Universe of 100. Not an error, not a timeout — a confident, well-formed answer computed over six percent of the market.

The failure is invisible from the outside. The heatmap renders. Breadth reports a number. Nothing indicates the Universe was truncated.

Evidence

Static reading only; none of this was executed.

• The upsert conflict key is (symbol, snapshot_date) and deliberately excludes source, so a live write lands on the same row the daily sync writes, and flips that row's source label as a side effect.
• The live path persists its pre-filter fetch result, which is bounded by the request limit (default 100, max 2000).
• The provider truncates the symbol list to limit before fetching, by plain head-slice of listing order — not by market cap or any other ranking. The 100 symbols written are an arbitrary prefix.
• The fresh-cache read matches on today's date with no row-count floor. A 100-row day looks exactly like a 1,700-row day.
• On fetch timeout the provider logs and returns whatever partial set completed rather than raising, so a degraded fetch writes an even smaller Universe by the same route.

The timeout makes it worse but is not required. A perfectly healthy limit=100 live request is sufficient to cause this.

Impact

Any reader of the Screener Snapshot can silently receive a truncated Universe: the screener itself, the heatmap, market breadth, and comparison. The window is from the first live-path write until the daily sync completes and rewrites the full set.

Severity comes from the blast radius rather than the trigger. One request's parameters degrade shared state that other, unrelated requests read as authoritative.

Suggested Direction

Not specified to implementation. Three directions worth weighing:

• Stop writing from the request path. Treat the Screener Snapshot as owned solely by the scheduled sync. The live path serves its response without persisting. Cleanest ownership story; loses the incidental cache-warming.
• Make partial writes unmistakable. Persist a completeness marker alongside the rows and have readers reject or flag a snapshot that does not cover the Universe. Keeps the warming, costs a schema change.
• Refuse to overwrite a fuller snapshot. Only write when the incoming set is at least as complete as what is already stored for that date. Narrowest change; leaves two writers of one table, which is the underlying discomfort.

The first is likely correct, but this needs a maintainer decision about who owns the Screener Snapshot before it is worth specifying.

Out of Scope

• The fundamental-fields defect tracked in Screener: fundamental fields silently null when filters arrive as serialized JSON #7. Same file, same pipeline, unrelated mechanism — that one is about enrichment ordering, this one is about write ownership.
• Provider batching, concurrency, and timeout tuning.
• Whether the heatmap and breadth surfaces should read this table at all.

Further Notes

Found while investigating #7. Not a regression from the current working-tree change — the write has been there. Filed separately to keep #7's scope intact.

Reproduction is a prediction from source, not an observed run: issue a limit=50 screener request with use_cache=false before the daily sync runs, then read the heatmap and count symbols. Confirm before acting.