Skip to content

Feat/streaming savings report - #4

Merged
Achal13jain merged 2 commits into
mainfrom
feat/streaming-savings-report
Aug 9, 2026
Merged

Feat/streaming savings report#4
Achal13jain merged 2 commits into
mainfrom
feat/streaming-savings-report

Conversation

@Achal13jain

Copy link
Copy Markdown
Owner

Summary

Adds streaming JSONL support for large record files and reports metadata reduction savings from fix.

This PR adds:

  • scan --stream for JSONL inputs.
  • validate --stream for JSONL inputs.
  • fix --stream --format jsonl for one-record-at-a-time cleanup.
  • Metadata reduction totals in fix output, including bytes removed and percentage reduction.
  • Aggregate report counts for streaming scan/validate without storing every record in memory.
  • Updated README, usage docs, changelog, and roadmap.

Testing

python -m pytest
ruff check .
ruff format --check .
mypy vectormeta

Results:

  • 60 passed
  • Ruff check passed
  • Ruff format check passed
  • Mypy passed

Checklist

  • Core logic remains independent from Typer/Rich where practical.
  • New behavior has focused tests.
  • Docs or examples are updated when user-facing behavior changes.
  • Vector database limit claims are cautious and sourced where needed.

@Achal13jain
Achal13jain merged commit fc514ff into main Aug 9, 2026
3 checks passed
@Achal13jain
Achal13jain deleted the feat/streaming-savings-report branch August 9, 2026 12:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant