Skip to content

feat(pipeline): fused template-gen to streaming dataset command#8

Open
tarekio wants to merge 5 commits into
mainfrom
pipeline
Open

feat(pipeline): fused template-gen to streaming dataset command#8
tarekio wants to merge 5 commits into
mainfrom
pipeline

Conversation

@tarekio

@tarekio tarekio commented Jul 6, 2026

Copy link
Copy Markdown
Owner

Adds pagen pipeline: one command that generates quality- and diversity-gated Arabic document templates via LLM, then streams one fresh fill per image straight into the render pipeline. pagen dataset is left byte-for-byte untouched.

  • pagen/diversity.py: diversity metrics promoted out of eval/ so the inline template gate and the offline sweeps run the same code (eval modules become thin re-export shims).
  • Template budgeting: plan_template_budget sizes doc-types by breadth from the requested image count, and invents new Arabic doc types via LLM when the built-in pool is exhausted (now spanning academic/scientific and literary genres too).
  • Streaming fills: one LLM call per image (no fill reuse), with a per-image debug.jsonl provenance sidecar (font, template, augment path, ground truth).
  • Graceful stop: a single Ctrl+C halts cleanly and leaves a valid partial labels.json; a populated output dir is refused unless --append.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant