Skip to content

keep external imports after connection loss - #45

Merged
matveynator merged 1 commit into
mainfrom
agent/fix-external-page-import
Jul 30, 2026
Merged

keep external imports after connection loss#45
matveynator merged 1 commit into
mainfrom
agent/fix-external-page-import

Conversation

@matveynator

Copy link
Copy Markdown
Owner

What changed

  • detach external-site import work from the lifetime of the initiating HTTP request
  • remove the ordinary response write deadline for the long-running import handler
  • use the durable import context for quota accounting, asset persistence, page storage, publication, and revisions
  • stop ignoring database write failures and return an explicit server error instead of reporting a false success
  • rebuild storage usage after storing an imported page
  • add a regression test based on https://kmv.ru/for-home.html that cancels the initiating request while a resource is downloading and verifies that both the page and its revision are stored

Root cause

The import pipeline and final database writes used r.Context(). When the browser, proxy, or progress connection disappeared during a long import, that context was cancelled. Database writes then failed, but their errors were discarded, allowing the UI to finish and redirect to a page that had never been created.

Impact

External page and whole-site imports now continue to completion after an incidental client/proxy connection loss. A successful response is sent only after the imported page and revision have actually been stored.

Validation

  • GOCACHE=/private/tmp/sitebrush-gocache go test . -run '^TestGrab(StoresPageAndRevisionAfterRequestConnectionIsLost|UsesRequestPathWhenPostedPathIsMissing)$' -count=1
  • git diff --check

The full go test ./... run still reports unrelated existing failures involving live DNS/mail relay behavior, platform-dependent disk usage, and stale static-asset expectations.

@matveynator
matveynator marked this pull request as ready for review July 30, 2026 21:11
@matveynator
matveynator merged commit 646d76d into main Jul 30, 2026
6 checks passed
@matveynator
matveynator deleted the agent/fix-external-page-import branch July 30, 2026 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant