Skip to content

fix: stream database dump exports#240

Open
akmittal006 wants to merge 1 commit into
outerbase:mainfrom
akmittal006:codex/stream-large-database-dumps
Open

fix: stream database dump exports#240
akmittal006 wants to merge 1 commit into
outerbase:mainfrom
akmittal006:codex/stream-large-database-dumps

Conversation

@akmittal006
Copy link
Copy Markdown

@akmittal006 akmittal006 commented May 22, 2026

Fixes #59

/claim #59

Purpose

Stream SQL database dump responses so large exports no longer build a single in-memory SQL string before returning a file. This directly addresses the current /export/dump memory pressure path while keeping the response-download behavior intact for bounded requests.

Tasks

  • Replace full SQL dump string accumulation with a ReadableStream response
  • Page table reads in 500-row batches instead of loading each table at once
  • Quote SQLite identifiers for schema lookups, SELECTs, and generated INSERT statements
  • Preserve SQL value escaping for strings, booleans, nulls, and binary values
  • Add regression coverage for batched export behavior

Verify

  • pnpm exec vitest run src/export/dump.test.ts src/export/index.test.ts src/export/csv.test.ts src/export/json.test.ts
  • pnpm exec prettier --check src/export/dump.ts src/export/dump.test.ts
  • git diff --check

Before

/export/dump accumulated the full dump in memory by appending table DDL and every row INSERT to one string before wrapping it in a Blob. Large databases could exhaust memory or block too long before a response could be streamed back.

After

/export/dump returns a ReadableStream, writes the SQLite header and table schema incrementally, and fetches row INSERTs in bounded batches with short yields between full batches.

@akmittal006
Copy link
Copy Markdown
Author

Added a short terminal recording showing the branch/commit and the focused dump export test passing, including coverage for exporting rows in batches without loading the whole table at once

starbasedb-59-test-run.mp4

@akmittal006 akmittal006 changed the title Stream database dump exports fix: stream database dump exports May 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Database dumps do not work on large databases

1 participant