fix(stripe): read-only reconciliation CLI + ERROR-level logging for dropped entitlement webhooks#938
fix(stripe): read-only reconciliation CLI + ERROR-level logging for dropped entitlement webhooks#938FagnerMartinsBrack wants to merge 3 commits into
Conversation
|
@claude CI has FAILED. Use your Read tool to read the local file Context
Failed Jobs
|
|
Claude finished @FagnerMartinsBrack's task in 10m 42s —— View job CI Failure Analysis — Attempt 1 of 5
Conclusion: this failure is pre-existing on
|
| Check | Result |
|---|---|
| Files this PR changes | Stripe only — stripe-reconcile, stripe-subscriptions, subscription-providers, customer-subscription-deleted, provider-contracts, package.json. Zero bookmark/CSS/template/e2e files. |
| Where the test lives | Added by #936 (5b993302) — not in this PR branch. It runs because PR CI builds the merge into main. |
main's own CI |
Push run 28744076544 on 5b993302 (current main HEAD) fails identically — same test, same 145px → 153px, same ratio 0.12. |
So #936 introduced the visual test and baseline PNGs whose committed render (145px) doesn't match what the CI runner actually produces (153px). main is currently red, which blocks every open PR through the merge build — this Stripe PR just inherits it.
Why I did not change this PR
The CI‑failure‑fixer instructions say to fix only the CI failure and not make unrelated changes, and CLAUDE.md forbids reversing intentional changes. The broken baseline lives on main, not here; committing bookmark snapshot PNGs into a Stripe PR would be an out‑of‑scope, environment‑sensitive change — and blessing a fallback‑font render as the new baseline is a visual decision a human should confirm, not something to auto‑commit.
Recommended fix (on main, not this PR)
Reproduce on the CI runner image and decide:
- If the brand font is failing to load in the e2e server → fix font loading so the render matches the intended 145px design (keep the baseline).
- If the 153px render is acceptable → regenerate the baselines on the CI image and commit the two PNGs to
main:cd projects/hutch pnpm compile E2E_PORT=3100 CI=true pnpm exec playwright test \ --config playwright.config.local-dev.ts \ crawl-bookmark-visual --update-snapshots
Once main is green, re‑run CI on this PR (its merge build will pick up the corrected baseline) and it will pass with no changes needed here.
Summary: The web-tests failure is the crawl-bookmark-visual E2E snapshot mismatch (145px vs 153px) introduced by #936, already red on main (run 28744076544). It is independent of this Stripe PR, whose diff contains no bookmark/UI files. I made no changes — the fix belongs on main (fix e2e font loading or regenerate the baseline). This PR will go green once main is fixed and CI re‑runs.
· main red run · branch fix/stripe-reconciliation
|
@claude Review this PR. Use your Read tool to read the local file Context
|
|
Claude finished @FagnerMartinsBrack's task in 6m 22s —— View job Code Review — PR #938 (Stripe reconciliation CLI + ERROR-level dropped-webhook logging)I read the full diff against High Priority Issues (Must Fix)None found Medium Priority IssuesNone found Low Priority Suggestions1.
|
Code Review — PR #938 (Stripe reconciliation CLI + ERROR-level dropped-webhook logging)I read the full diff against High Priority Issues (Must Fix)None found Medium Priority IssuesNone found Low Priority Suggestions1.
|
|
@claude Resolve conflicts. Use your Read tool to read the local file Context
|
|
Claude finished @FagnerMartinsBrack's task in 16m 37s —— View job Conflicts resolved ✅Rebased the Stripe commit onto current
What conflicted & how it was resolvedBoth conflicts were additive collisions with #939 (pre-expiry trial reminder email), which merged into
The test-file conflict was tangled (git aligned the coincidentally-identical No Stripe logic was altered — the resolution is purely a re-integration of the two independent additions. VerificationRan
|
39ba2d1 to
a79b52d
Compare
|
@claude CI has FAILED. Use your Read tool to read the local file Context
Failed Jobs
|
|
Claude finished @FagnerMartinsBrack's task in 29m 43s —— View job CI Failure Analysis — Attempt 2 of 5 → ✅ CI now GREEN
Conclusion: the failure was a flaky firefox E2E timeout, unrelated to this Stripe PRThe failing task was Proof it's flaky, not caused by this PR:
The CI runner was momentarily under load (each failed attempt took ~57–59s vs. the passing OAuth test's ~30s), so browser ops blew the 30s email-field wait. The one server-path file this PR touches ( Action takenNo code fix was appropriate — a change to the firefox E2E infra would be an unrelated change the fixer instructions forbid. ResultRe-triggered run 28832738663 → |
|
@claude Review this PR. Use your Read tool to read the local file Context
|
|
Claude finished @FagnerMartinsBrack's task in 4m 44s —— View job Code Review — PR #938 (Stripe reconciliation CLI + ERROR-level dropped-webhook logging)I re-read the full diff against High Priority Issues (Must Fix)None found Medium Priority IssuesNone found Low Priority Suggestions(All three carried over from the prior review at 1.
|
Code Review — PR #938 (Stripe reconciliation CLI + ERROR-level dropped-webhook logging)I re-read the full diff against High Priority Issues (Must Fix)None found Medium Priority IssuesNone found Low Priority Suggestions(All three carried over from the prior review at 1.
|
|
@claude Resolve conflicts. Use your Read tool to read the local file Context
|
|
Claude finished @FagnerMartinsBrack's task in 16m 58s —— View job Conflicts resolved ✅Rebased onto current
What conflicted & how it was resolvedAll three conflicts were additive collisions with #940 (self-serve account deletion), which merged into
VerificationRan the full
No Stripe logic was touched — the resolution is purely a re-integration of the two independent additions. · branch |
7cbdce2 to
cec20d9
Compare
|
@claude Review this PR. Use your Read tool to read the local file Context
|
|
Claude finished @FagnerMartinsBrack's task in 5m 20s —— View job Code Review — PR #938 (Stripe reconciliation CLI + ERROR-level dropped-webhook logging)I re-read the full diff against High Priority Issues (Must Fix)None found Medium Priority IssuesNone found Low Priority Suggestions(All three carried over from the reviews at 1.
|
Code Review — PR #938 (Stripe reconciliation CLI + ERROR-level dropped-webhook logging)I re-read the full diff against High Priority Issues (Must Fix)None found Medium Priority IssuesNone found Low Priority Suggestions(All three carried over from the reviews at 1.
|
cec20d9 to
fa8c843
Compare
|
@claude Review this PR. Use your Read tool to read the local file Context
|
fa8c843 to
b331d8d
Compare
|
@claude Review this PR. Use your Read tool to read the local file Context
|
|
Claude finished @FagnerMartinsBrack's task in 3m 50s —— View job Code Review — PR #938 (Stripe reconciliation CLI + ERROR-level dropped-webhook logging)I read the full diff against High Priority Issues (Must Fix)None found Medium Priority IssuesNone found Low Priority Suggestions(All three carry over from the reviews at 1.
|
Code Review — PR #938 (Stripe reconciliation CLI + ERROR-level dropped-webhook logging)I read the full diff against High Priority Issues (Must Fix)None found Medium Priority IssuesNone found Low Priority Suggestions(All three carry over from the reviews at 1.
|
|
@claude Resolve conflicts. Use your Read tool to read the local file Context
|
|
Claude encountered an error —— View job Resolving merge conflicts — PR #938
|
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
2 similar comments
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
|
@claude Resolve conflicts. Use your Read tool to read the local file Context
|
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
3 similar comments
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
|
@claude Resolve conflicts. Use your Read tool to read the local file Context
|
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
3 similar comments
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
|
@claude Resolve conflicts. Use your Read tool to read the local file Context
|
|
Claude finished @FagnerMartinsBrack's task in 20m 43s —— View job Conflicts resolved ✅Rebased onto current
What actually conflicted & how it was resolvedThe prior conflict-fixer attempts errored on the agent-SDK startup crash (#852) and never resolved anything — so this was the first real pass. The rebase itself applied with no textual conflicts, but that hid a semantic break: main's inbox extraction (#950,
Git's rename detection correctly reapplied this PR's
No Stripe logic was altered — the resolution purely re-integrates the PR's additions with main's module move. The hutch composed test that drives VerificationRan the full
|
…ok logging Add a read-only Stripe-vs-DynamoDB reconciliation CLI (pnpm --dir projects/hutch stripe-reconcile) that pages through Stripe subscriptions and a full subscription-table scan, then reports five categories of drift via a pure, fully-tested reconcile() domain function. Customer emails are masked before they enter any finding. Also upgrade the customer.subscription.deleted "no subscription row found" log from WARN to a structured ERROR carrying subscriptionId, customerId, and eventType. The handler returns 200 so Stripe never retries and the Lambda-Errors alarm never fires — a dropped entitlement event is action-required, which is ERROR, not informational. Claude-Session: https://claude.ai/code/session_01En2uUiiwQoRQKWenCnkPi9
No-op commit to re-trigger CI. The failure was a flaky firefox-extension reader-link E2E timeout (30s waiting for the login #email field), not a defect in this Stripe-only diff: - The sibling "OAuth login flow" E2E passed in the same CI run using the same #email field, so the login page renders fine in this build. - The reader-link E2E passes locally in ~10s and nx auto-flagged firefox-extension:check as a flaky task (three separate local runs). - This PR changes only Stripe files; the one server-path change adds listAllSubscriptionRows, called exclusively by the reconcile CLI, never in the login/reader request path. Full `pnpm check` is green locally (37 projects, 48 tasks). No API permission to re-run the workflow, so re-triggering CI with this commit. Co-authored-by: Fayner Brack <FagnerMartinsBrack@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… read The rebase onto main picked up the inbox extraction (#950, 963345c), which moved initDynamoDbSubscriptionRead out of hutch into @packages/subscription-access. Git's rename detection reapplied this PR's listAllSubscriptionRows addition onto the relocated file, but left two loose ends the rebase could not see: - The reconcile CLI still imported initDynamoDbSubscriptionRead from the now-deleted hutch-local path. Repoint it to @packages/subscription-access, matching app.ts and the composed dynamodb-subscription-providers. - listAllSubscriptionRows coverage lived only in hutch's composed test, so the relocated function body was uncovered by its new package's own suite. Add the matching unit tests (single page, ExclusiveStartKey pagination, empty table) to subscription-access — restoring 100% coverage there. No Stripe logic changed; this only re-integrates the PR's additions with main's module move. Full `pnpm check` green (42 projects / 60 tasks). Co-authored-by: Fayner Brack <FagnerMartinsBrack@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
b331d8d to
0995a63
Compare
|
@claude Review this PR. Use your Read tool to read the local file Context
|
|
Claude finished @FagnerMartinsBrack's task in 6m 3s —— View job Code Review — PR #938 (Stripe reconciliation CLI + ERROR-level dropped-webhook logging)
Unlike the prior CI-retry commits, High Priority Issues (Must Fix)None found Medium Priority IssuesNone found Low Priority Suggestions1.
|
High Priority Issues (Must Fix)None found Medium Priority IssuesNone found Low Priority Suggestions1.
|

fix(stripe): read-only reconciliation CLI + ERROR-level logging for dropped entitlement webhooks
Hypothesis
A paying (or payment-attempting) customer can exist in Stripe with no matching app-side subscription row, and the app can hold entitlement rows Stripe no longer backs. Today the only signal is a WARN that nobody watches, on the only Stripe event we handle. Shipping a read-only diff tool + upgrading that WARN to a structured ERROR turns an invisible data-integrity failure into an actionable one. This PR is investigation tooling — it deliberately fixes nothing it hasn't measured.
Evidence (30d, Jun 4 – Jul 4 2026)
WARN "[stripe-webhook] no subscription row found — skipping event emission" subscriptionId=sub_1TSJTQBjoaOLiPzvOSCNP52H. The receiver then returned 200, so Stripe will never redeliver it.subscriptionId/customerId.3e9c7e5cJun 21 17:35,af194ac4Jun 23 12:53 — the latter after their Jun 18 cancellation).charge_succeededevents: 0 (by design — checkout success callsupsertActivedirectly, no event).StripeEventTypeunion and the infraevents:array both wire exactly one event —customer.subscription.deleted. Noinvoice.payment_failed, noinvoice.paid, no renewal signal of any kind reaches the app.Investigation narrative: the dropped webhook
Timeline:
3e9c7e5clands on/auth/checkout/success;upsertActivewrites an active row (this should be one of the 3 rows with a subscriptionId).customer.subscription.deletedforsub_1TSJTQBjoaOLiPzvOSCNP52H. The GSI lookup (subscriptionId-index) finds no row. Event dropped with a WARN; 200 returned.Two competing hypotheses, and the CLI distinguishes them:
incomplete_expired: the checkout's initial payment was asynchronous and failed; Stripe voids incomplete subscriptions (~23h window; async-failure paths fire sooner), emittingcustomer.subscription.deletedwith the sub inincomplete_expired. The 13.5h gap fits. If true,3e9c7e5csaw a success page, holds an active app row backed by a dead Stripe sub, and paid nothing — and because the webhook was dropped, nothing ever cancelled the row.upsertActivewrote a different id (or the write failed / was later overwritten by the cancellation flow, cf.af194ac4checking out after cancelling), so the GSI lookup missed a row that conceptually exists.The reconcile report answers it: if
sub_1TSJTQ…appears under “Stripe subs with no matching app row” withstripeStatus=incomplete_expiredand3e9c7e5cappears under “live app rows with no live Stripe sub”, it's H1 + drift; if the app row holds a different subscriptionId, it's H2.What this changes
pnpm --dir projects/hutch stripe-reconcile— a read-only CLI (zero writes; it composes only the read half of the subscription table) that pages throughstripe.subscriptions.list(status=all, expand data.customer)and a full table scan, then reports:stripeStatusand masked customer email, e.g.j***@g***, for manual correlation — raw emails never enter the findings object, enforced by test);active/pending_cancellationwith no live Stripe sub;cancelled— the “paying customer without entitlement” case;trialingrows with past (or absent)trialEndsAt— the 4 stuck rows;subscriptionIdentirely (expected ~42 of 45). App rows carry no email, so email-join is reported from the Stripe side (masked) rather than guessed.Diff logic is a pure, fully unit-tested
reconcile()inruntime/domain/stripe-reconcile/; the CLI is a thin composition root mirroringsend-checkout-recovery-emails.WARN → structured ERROR in the
customer.subscription.deletedhandler when no row matches: now includessubscriptionId,customerId,eventType, and a pointer to run the reconcile CLI. Why ERROR: the handler acknowledges with 200, so Stripe never retries and the Lambda-Errors alarm never fires — a dropped entitlement-affecting event is action-required, not informational. The remaining WARNs in the receiver (signature probes) are internet noise; the level split makes a future ERROR metric filter meaningful.What this deliberately does NOT change
invoice.payment_failed,invoice.paid,customer.subscription.updated) is real — the app currently has no way to learn a renewal charge failed — but wiring events before reconciliation data lands would be guessing. Future work, informed by the CLI's first prod run.Estimated result
The first prod run classifies all 45 rows and every Stripe sub: confirms/refutes H1 for
sub_1TSJTQ…, surfaces whether any live Stripe sub is unentitled (revenue-affecting), and inventories the 4 stuck trials. Ongoing: any future dropped webhook is an ERROR with enough context to act within one log line.How to measure
Owner commands (read-only; use prod AWS creds and the live secret key):
Watch for future drops — CloudWatch Logs Insights on
/aws/lambda/stripe-webhook-receiver-handler:(0 results after this lands = healthy; any result = run the CLI.)
Merge note
The analytics dashboard test asserts
toHaveLength(24)on widget count. This PR adds no analytics events and no widgets, so it does not touch that assertion — if sibling PRs #916–#918 bump it and conflict on rebase, take their side verbatim; nothing here depends on the count.https://claude.ai/code/session_01En2uUiiwQoRQKWenCnkPi9