Client-Side CSV/GeoJSON Bulk Export Pipeline with Streaming Chunked NDJSON and File System Access API

[elizabetheonoja-art]

Problem Statement / Feature Objective

Regulatory compliance and operational analysis require exporting filtered views of resource consumption data as CSV or GeoJSON files. The current export approach holds the entire dataset in memory, causing tab crashes for queries exceeding 500,000 rows. A client-side bulk export pipeline must stream data from the REST API in chunked requests, decompress gzipped NDJSON responses, transform rows into the target format, and write the result as a Blob download via the File System Access API or fallback to a download link. The pipeline must support CSV, GeoJSON, and shapefile (zipped) output with configurable column projection and row filtering.

Technical Invariants & Bounds

• Chunk size: 10,000 rows per REST request (GET /api/resources/export?offset={o}&limit=10000). Max 100 chunks = 1,000,000 rows total export limit.
• Compression: server sends gzipped NDJSON; decompress via DecompressionStream (browser-native) with polyfill fallback (pako).
• Transform throughput: target 50,000 rows/second for CSV, 20,000 rows/second for GeoJSON (requires geometry coordinate transformation).
• Memory guard: streaming pipeline must not accumulate more than 50 MB in memory at once. If the row buffer exceeds 50 MB, flush to disk via the File System Access API's WritableStream.
• Fallback: if File System Access API is unavailable, accumulate the first 100 MB in memory then fall back to Blob + download link with a warning.
• Coordinate precision: GeoJSON output truncates coordinates to 6 decimal places (~0.11m accuracy). Shapefiles must be in EPSG:4326.

Codebase Navigation Guide

• src/services/exportPipeline.ts - Streaming export orchestrator: manages chunk requests, transformers, and output writer.
• src/utils/ndjsonParser.ts - Streaming NDJSON parser that yields parsed rows as they become available.
• src/utils/formatTransformers.ts - Row-to-CSV, row-to-GeoJSONFeature, row-to-ShapefileRecord conversion functions.
• src/hooks/useExportProgress.ts - React hook exposing progress state: currentChunk, totalChunks, bytesDownloaded, rowsWritten, status.
• src/components/panels/ExportDialog.tsx - Dialog with format selector, column picker, filter builder, and progress bar + cancel button.
• src/utils/fileWriter.ts - Abstraction over File System Access API WritableStream and Blob fallback.

Implementation Blueprint

1. In src/utils/fileWriter.ts, create FileWriter interface with write(chunk: Uint8Array) and close(). Implementation 1: FileSystemWritableFileStream via showSaveFilePicker. Implementation 2: Blob aggregator with createObjectURL fallback.
2. Implement src/utils/ndjsonParser.ts as an async generator: create a TransformStream that pipes the fetch Response.body through DecompressionStream (or pako), splits on newlines, yields parsed JSON objects.
3. In src/utils/formatTransformers.ts: csvTransformer(row, columns) returns a CSV line with RFC 4180 escaping; geoJsonTransformer(row) returns a GeoJSON Feature with Point geometry from coordinates; shapefileTransformer(row) builds a record using the shapefile.js library.
4. Build the orchestrator src/services/exportPipeline.ts: accepts ExportConfig { format, columns, filters, maxRows }. Issues parallel prefetch of next chunk while processing current chunk (sliding window of 2 chunks). Each chunk's transformed rows are piped into fileWriter.write(). Tracks progress and emits events for the hook.
5. Create useExportProgress hook that subscribes to pipeline events: chunkStart, chunkComplete, error, complete.
6. ExportDialog provides: format dropdown, column multiselect (populated from schema endpoint GET /api/resources/schema), filter builder (field, operator, value), and a large progress bar with Cancel button. Cancel calls abortController.abort() on the fetch chain.
7. Write unit tests for formatTransformers with known input/output pairs, and a Playwright test that triggers an export and verifies the downloaded file has the expected row count.

[real-venus]

Hey, I've read through the issue and I understand it.

Could you assign it to me? I can finish this within 24 hrs.

[grantfox-oss]

🦊 GrantFox — @real-venus has been assigned to this issue as part of the Official Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #60)
2. Your PR will be reviewed by the Utility-Protocol maintainers

Good luck! Track your progress on GrantFox.

[grantfox-oss]

🎉 This issue has been marked as completed on GrantFox as part of the Official Campaign campaign!

@real-venus's PR #80 was approved and merged by @elizabetheonoja-art.

🏆 @real-venus: You earned 45 FoxPoints for this contribution! Your current tier: Specialist (3,571 total points). Track your full progress on GrantFox.

👏 Great work, @real-venus! Keep contributing to Utility-Protocol.