Symptom
Since 2025-11-11, the nightly onchain sync (processSyncOnchainTransactions, 02:00 UTC) fails every night with:
duplicate key value violates unique constraint "UQ_8a233973c49afbbc14a01ce076b"
The count is suspiciously constant — exactly 55 duplicate-key errors per night on PRD, 3 on DEV, every single night. That constant is the tell: 55/3 is the entire row count of transaction_onchain in each environment. The job collides with every row it just loaded, Promise.all rejects, the @Lock() wrapper logs Error during processSyncOnchainTransactions, and the job aborts. There are zero syncOnchainTransactions: runtime= success lines in the logs — the job has effectively never completed since November.
Root cause
Commit e18f05a (PR #84, 2025-11-11) bumped tsconfig.json target from es2017 to es2022:
- "target": "es2017",
+ "target": "es2022",
With target ≥ es2022, TypeScript's useDefineForClassFields default flips to true. Class-field declarations (like the entity base class's id) then become own properties initialized to undefined on every instance — including entities produced by repo.create({...}).
The update helpers merge a freshly-created entity into the loaded DB row with Object.assign:
|
private async doUpdateOnchainTransaction( |
|
updateOnchainTransactionEntity: TransactionOnchainEntity, |
|
): Promise<TransactionOnchainEntity> { |
|
let dbOnchainTransactionEntity = await this.transactionOnchainRepo.findOneBy({ |
|
transaction: Equal(updateOnchainTransactionEntity.transaction), |
|
}); |
|
|
|
if (!dbOnchainTransactionEntity) { |
|
dbOnchainTransactionEntity = updateOnchainTransactionEntity; |
|
} else { |
|
Object.assign(dbOnchainTransactionEntity, updateOnchainTransactionEntity); |
|
} |
|
|
|
return this.transactionOnchainRepo.save(dbOnchainTransactionEntity); |
|
} |
let dbOnchainTransactionEntity = await this.transactionOnchainRepo.findOneBy({
transaction: Equal(updateOnchainTransactionEntity.transaction),
});
if (!dbOnchainTransactionEntity) {
dbOnchainTransactionEntity = updateOnchainTransactionEntity;
} else {
Object.assign(dbOnchainTransactionEntity, updateOnchainTransactionEntity); // copies id: undefined!
}
return this.transactionOnchainRepo.save(dbOnchainTransactionEntity);
Pre-es2022, the fresh entity had no own id property, so Object.assign left the loaded row's id intact and save() issued an UPDATE. Post-es2022, the fresh entity carries id: undefined as an own property, Object.assign wipes the loaded row's id, and save() issues an INSERT — colliding with the very row the code just found.
The worse, silent half
transaction_onchain has UNIQUE ("transaction") (migration 1697006086810-Transactions.js), so its corruption is loud — the INSERT fails.
transaction_lightning has no unique constraint on (type, transaction) — only the PK. The identical pattern in doUpdateLightningTransaction:
|
private async doUpdateLightningTransaction( |
|
updateTransactionLightningEntity: TransactionLightningEntity, |
|
): Promise<TransactionLightningEntity> { |
|
let dbTransactionLightningEntity = await this.transactionLightningRepo.findOneBy({ |
|
type: Equal(updateTransactionLightningEntity.type), |
|
transaction: Equal(updateTransactionLightningEntity.transaction), |
|
}); |
|
|
|
if (!dbTransactionLightningEntity) { |
|
dbTransactionLightningEntity = updateTransactionLightningEntity; |
|
} else { |
|
Object.assign(dbTransactionLightningEntity, updateTransactionLightningEntity); |
|
} |
|
|
|
return this.transactionLightningRepo.save(dbTransactionLightningEntity); |
|
} |
…therefore succeeds in inserting duplicates. Consequences, ongoing since November 2025:
- The 03:00 UTC
processSyncLightningTransactions re-inserts its trailing-window candidate set as new rows every night (its logged entries= count has been growing, e.g. 369 → 1250/night over one recent week).
- The websocket live-update path turns every state transition (e.g. invoice OPEN → SETTLED) into a new row instead of an update, so a single invoice exists multiple times in different states.
Same mechanism likely affects the other Object.assign-onto-loaded-entity sites: user-boltcard.service.ts:62 and lightning-wallet.service.ts:286 — worth auditing all of them in the fix.
Note user_transaction.lightningTransactionId has an FK onto transaction_lightning.id, so any dedupe cleanup has to remap references, not just delete rows.
Suggested fix
Two options, not mutually exclusive:
"useDefineForClassFields": false in tsconfig — one line, restores the pre-November semantics for the entire codebase and kills the whole bug class at once.
- Replace the
Object.assign(dbEntity, freshEntity) merge sites with an id-safe merge (repo.merge() with an explicit field list, repo.update(id, partial), or explicit per-field assignment).
Option 1 is the smallest safe change; option 2 is more future-proof if a later target bump is ever wanted. Either way a regression test pinning "update does not change the row's id / does not insert a second row" would prevent a silent re-flip.
Optional hardening while in there: add the missing UNIQUE (type, transaction) constraint on transaction_lightning after the cleanup below, so this failure mode can never again be silent.
Follow-up: data cleanup (separate task)
After the code fix, transaction_lightning needs a dedupe pass (keep the newest row per (type, transaction), remap user_transaction.lightningTransactionId FKs to the surviving row, delete the rest) and the onchain sync needs one clean full run to verify. Sizing the duplicate population first (SELECT type, transaction, COUNT(*) ... GROUP BY ... HAVING COUNT(*) > 1) would tell us how big that task is.
Verification after deploy
- 02:00 UTC job:
Error during processSyncOnchainTransactions disappears; a syncOnchainTransactions: runtime= success line appears.
- Zero
UQ_8a233973 duplicate-key lines in the DB logs.
transaction_lightning row count stops growing at the nightly sync rate; invoice state transitions update rows in place.
Symptom
Since 2025-11-11, the nightly onchain sync (
processSyncOnchainTransactions, 02:00 UTC) fails every night with:The count is suspiciously constant — exactly 55 duplicate-key errors per night on PRD, 3 on DEV, every single night. That constant is the tell: 55/3 is the entire row count of
transaction_onchainin each environment. The job collides with every row it just loaded,Promise.allrejects, the@Lock()wrapper logsError during processSyncOnchainTransactions, and the job aborts. There are zerosyncOnchainTransactions: runtime=success lines in the logs — the job has effectively never completed since November.Root cause
Commit e18f05a (PR #84, 2025-11-11) bumped
tsconfig.jsontargetfromes2017toes2022:With target ≥ es2022, TypeScript's
useDefineForClassFieldsdefault flips totrue. Class-field declarations (like the entity base class'sid) then become own properties initialized toundefinedon every instance — including entities produced byrepo.create({...}).The update helpers merge a freshly-created entity into the loaded DB row with
Object.assign:api/src/subdomains/lightning/services/lightning-transaction.service.ts
Lines 308 to 322 in 67ce15f
Pre-es2022, the fresh entity had no own
idproperty, soObject.assignleft the loaded row'sidintact andsave()issued an UPDATE. Post-es2022, the fresh entity carriesid: undefinedas an own property,Object.assignwipes the loaded row's id, andsave()issues an INSERT — colliding with the very row the code just found.The worse, silent half
transaction_onchainhasUNIQUE ("transaction")(migration1697006086810-Transactions.js), so its corruption is loud — the INSERT fails.transaction_lightninghas no unique constraint on(type, transaction)— only the PK. The identical pattern indoUpdateLightningTransaction:api/src/subdomains/lightning/services/lightning-transaction.service.ts
Lines 409 to 424 in 67ce15f
…therefore succeeds in inserting duplicates. Consequences, ongoing since November 2025:
processSyncLightningTransactionsre-inserts its trailing-window candidate set as new rows every night (its loggedentries=count has been growing, e.g. 369 → 1250/night over one recent week).Same mechanism likely affects the other
Object.assign-onto-loaded-entity sites:user-boltcard.service.ts:62andlightning-wallet.service.ts:286— worth auditing all of them in the fix.Note
user_transaction.lightningTransactionIdhas an FK ontotransaction_lightning.id, so any dedupe cleanup has to remap references, not just delete rows.Suggested fix
Two options, not mutually exclusive:
"useDefineForClassFields": falsein tsconfig — one line, restores the pre-November semantics for the entire codebase and kills the whole bug class at once.Object.assign(dbEntity, freshEntity)merge sites with an id-safe merge (repo.merge()with an explicit field list,repo.update(id, partial), or explicit per-field assignment).Option 1 is the smallest safe change; option 2 is more future-proof if a later target bump is ever wanted. Either way a regression test pinning "update does not change the row's id / does not insert a second row" would prevent a silent re-flip.
Optional hardening while in there: add the missing
UNIQUE (type, transaction)constraint ontransaction_lightningafter the cleanup below, so this failure mode can never again be silent.Follow-up: data cleanup (separate task)
After the code fix,
transaction_lightningneeds a dedupe pass (keep the newest row per(type, transaction), remapuser_transaction.lightningTransactionIdFKs to the surviving row, delete the rest) and the onchain sync needs one clean full run to verify. Sizing the duplicate population first (SELECT type, transaction, COUNT(*) ... GROUP BY ... HAVING COUNT(*) > 1) would tell us how big that task is.Verification after deploy
Error during processSyncOnchainTransactionsdisappears; asyncOnchainTransactions: runtime=success line appears.UQ_8a233973duplicate-key lines in the DB logs.transaction_lightningrow count stops growing at the nightly sync rate; invoice state transitions update rows in place.