This repository was archived by the owner on Jun 12, 2026. It is now read-only.
StorageIdb: align filter/update semantics with Knex canon - #151
Merged
sirdeggen merged 2 commits intoApr 27, 2026
Conversation
shruggr
force-pushed
the
fix/storage-provider-parity
branch
from
April 22, 2026 02:28
12518d0 to
90ea80f
Compare
Surfaced during a multi-account sync-failure investigation. A user's
IndexedDB held multiple accounts; Account B's output_tags_map rows
leaked into Account A's outbound sync chunk because filterOutputTagMaps
had a dead-code user-scope filter (guarded on a non-existent txid
column). Auditing that surfaced five related divergences from the
StorageKnex canon, all fixed in this commit.
All changes in src/storage/StorageIdb.ts (plus one small caller change
in src/storage/methods/listOutputsIdb.ts).
1. filterOutputTagMaps user-scope filter — CRIT
The guard was `if (userId !== undefined && r.txid)` but
TableOutputTagMap has no txid column — the check was always false
and the scope filter never ran. getOutputTagMapsForUser returned
rows across all users. Drop the bogus r.txid and run the sub-count
against output_tags unconditionally when userId is provided.
Matches Knex's WHERE EXISTS behavior.
2. filterOutputs / filterTransactions empty-array poisoning — HIGH
filterOutputs applied the tx-status sub-count on
`args.txStatus !== undefined`; filterTransactions used
`if (args.status && !args.status.includes(...))`. Since [] is
truthy, an empty array caused every row to be rejected on Idb while
Knex treats empty array as a no-op. Added .length > 0 guards
matching Knex/StorageBunSqlite.
3. HIGH-7: IDB listOutputs include 'sending' tx status
4. HIGH-8: IDB tag/label resolution tolerates orphan map rows
5. HIGH-11: IDB updateIdb silent-skip missing ids
6. MED-14: IDB allocateChangeInput hydrates chosen output's script
7. MED-16: IDB listOutputs supports negative offset (tail pagination)
8. MED-18: IDB findTransactions(noRawTx) keeps inputBEEF
9. MED-19: IDB insertCertificate strips non-schema logger field
10. LOW-28: IDB getRawTxOfKnownValidTransaction uses 'unfail'-inclusive
status set on slice path
Rebased onto 2.1.22. Previous branch history had a long sequence of
apply-then-revert commits that have been squashed into this single
commit against the current main.
shruggr
force-pushed
the
fix/storage-provider-parity
branch
from
April 23, 2026 19:34
02da237 to
5afac5d
Compare
shruggr
marked this pull request as ready for review
April 27, 2026 14:17
Sonar flagged: unused verifyOne import, and non-null assertions on offset/length already narrowed via the sliceRequested const.
|
sirdeggen
approved these changes
Apr 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Context
Surfaced during a sync-failure investigation in yours-wallet. A user's IndexedDB held multiple accounts; Account B's
output_tags_maprows leaked into Account A's outbound sync chunk and then exploded on the receiving storage (an mss1 wallet server in the 1sat-sdk monorepo) with "Result must be unique." The direct cause was a dead-code user-scope filter inStorageIdb.filterOutputTagMaps; auditing that surfaced five related divergences from theStorageKnexcanon, fixed in this PR.Full divergence audit (what matched, what drifted, severity) lives in
1sat-sdk/docs/plans/2026-04-21-storage-provider-divergences.md— happy to copy here on request.Changes (all in
src/storage/StorageIdb.ts)1.
filterOutputTagMapsuser-scope filter — CRITThe guard was
if (userId !== undefined && r.txid)butTableOutputTagMaphas notxidcolumn — the check was always false and the scope filter never ran.getOutputTagMapsForUserreturned rows across all users in the DB. Drop the bogusr.txidand run the sub-count againstoutput_tagsunconditionally when userId is provided. Matches Knex'sWHERE EXISTS (SELECT * FROM output_tags WHERE ...)behavior.2.
filterOutputs/filterTransactionsempty-arraytxStatus/statuspoisoning — HIGHfilterOutputsapplied the tx-status sub-count onargs.txStatus !== undefined;filterTransactions:1855usedif (args.status && !args.status.includes(...)). Since[]is truthy, an empty array caused every row to be rejected on Idb while Knex treats empty array as a no-op. Added.length > 0guards matching Knex/StorageBunSqlite.3. Undefined-partial silent over-match — HIGH
Knex throws
Undefined binding(s) detectedwhen a partial filter hasundefinedvalues.StorageIdbsilently skipped undefined keys via per-key truthiness guards, producing unintended over-matches — especially painful in sync'sEntityOutputTagMap.mergeFindpath where an unmapped idMap entry quietly propagatesundefineddown tofindOutputTagMaps({partial: {outputId: undefined, outputTagId: undefined}})and thenverifyOneOrNonethrows a misleading "Result must be unique." New privateassertNoUndefinedInPartialcalled at the top of everyfilter*method reproduces Knex's loud failure.4. Truthiness guards on integer-ID / string partial fields — MED
Sweep-replaced
if (args.partial.X && r.X !== args.partial.X)→if (args.partial.X !== undefined && r.X !== args.partial.X)across all filter methods (~106 replacements covering IDs, strings, date fields). Fixes silent filter skips forlockTime === 0,version === 0, empty-stringdescription/derivationPrefix/customInstructions, etc. Fields that were already!== undefined(isDeleted,satoshis, booleans) were left alone.5.
updateIdbreturn count — MEDWas unconditionally
return 1even for batch updates across an array of ids. Now tracks and returns the real updated count. (updateIdbKeyis single-record by composite key; itsreturn 1is correct and unchanged.)6.
filterProvenTxReqsuser-scope r.txid guard — LOWSame shape as finding 1.
TableProvenTxReq.txidis NOT NULL in the Knex canon schema, so&& r.txidis almost always truthy in practice — but IDB has no schema enforcement, and a row with empty/undefined txid would silently bypass user scoping. Knex'sWHERE EXISTS (... JOIN ON txid)would exclude such rows; Idb was including them. Drop the guard.What this does not touch
purgeDataIdbparity vsmethods/purgeData.ts— not audited in depth, flagged for follow-up.Rebase + packaging
Rebased onto
origin/master(2.1.21). Package name + version bumped to@bopen-io/wallet-toolbox@2.1.21-parity-fix.0(and the mirrored@bopen-io/wallet-toolbox-mobile@2.1.21-parity-fix.0) for immediate publication via the @bopen-io npm org. If this PR merges, a separate commit would revert the rename for@bsvpublication.Validation
npm run build— clean on 2.1.21 + my changes.npm test -- --testPathPattern="OutputTagMapTests|TxLabelMapTests|OutputTagTests|TxLabelTests|TransactionTests|OutputTests"— 61/61 passing.ChaintracksStorageIdb.test.ts(WoC file-hash mismatch, network-dependent, fails onmastertoo).Test plan for merge
getOutputTagMapsForUserwith a multi-user IDB returns only the requested user's maps.findOutputTagMaps({partial: {outputId: undefined}})throwsWERR_INVALID_PARAMETER.findTransactions({status: []})returns rows instead of nothing.findTransactions({partial: {lockTime: 0}})filters correctly.updateIdbreturns actual affected-row count on a batch update.