Skip to content

feat: cursor-based streaming for CSV export at scale (100K+ rows) #126

Description

@kvithayathil

Problem

_generate_csv_rows materializes all match results via list(yield_per(...)) before generating rows. This works at current scale (<10K) but will hit memory limits at 100K+ rows.

Approach

  • Replace materialized list with cursor-based pagination through _generate_csv_rows
  • Stream CSV rows via StreamingResponse with backpressure
  • Yield rows in batches to avoid unbounded memory growth
  • Preserve existing export format and behavior

Sub-tasks

  • Design streaming API contract (chunk size, cursor state)
  • Implement cursor-based iteration in _generate_csv_rows
  • Wire StreamingResponse in campaign router export endpoint
  • Integration test with 100K+ rows (in-memory SQLite benchmark)
  • Verify memory stays bounded under load

Context

Identified as deferred follow-up from PR #124 (cursor-based keyset pagination). Low priority — current scale is <10K rows.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions