An industrial-grade end-to-end pipeline for synthesizing high-quality multimodal (image + text) mathematics problems at scale. Built on a distributed multi-agent orchestration framework where specialized LLM agents operate in coordinated stages to generate, validate, and quality-gate problems.
Core workflow: Given a seed corpus of (image, problem) pairs, spawn 6–8 candidate problems per image via multi-agent generation, orchestrate cross-validation through a reasoning-intensive verification layer, and apply difficulty gating via a dedicated evaluation agent. Only problems that survive all gates reach production.
Three completely decoupled stages, organized by timestamp / pipeline phase / source image, enabling modular operation or full end-to-end orchestration:
Seed Corpus: multimodal_math_seed/（multimodal_math.json + images/）
│
▼
[Agent Layer 1: Generation] Synthesize problems: Each image → 6–8 variants via
▼ multi-template agent (Doubao-Seed-2-1-Turbo),
1_generate/<image>/ stochastically combining (transformation × type × style)
│
▼
[Agent Layer 2: Verification] Validate problem integrity & ground truth: GPT-5.5
▼ vision agent batch-processes same-image clusters,
2_verify/<image>/ validates problem statement viability, corrects answers
│
▼
[Agent Layer 3: Difficulty Gate] Evaluate solution complexity: Target agent
▼ (Doubao-Seed-2-0-Pro) attempts each problem in
3_difficulty/<image>/ isolation; failed attempts mark problems as
sufficiently challenging (qualified)
output/run_<timestamp>/
├── 1_generate/<image_id>/ # image_id = image filename without extension
│ ├── image_id.png # Original image copied from seed corpus
│ └── problems.jsonl # 6–8 generated candidate problems
├── 2_verify/<image_id>/
│ ├── image_id.png
│ ├─�� problems.jsonl # Post-verification: viable problems with
│ │ # answers corrected to ground truth
│ └── dropped.jsonl # Rejected problems (invalid statements)
└── 3_difficulty/<image_id>/
├── image_id.png
├── problems.jsonl # Complete problem set (with difficulty scores)
└── qualified.jsonl # Filtered set: target-agent-failed problems only
mathforge_multimodal/
├── README.md
├── requirements.txt
├── run_pipeline.py # Orchestration entrypoint
└── mmforge/
├── core/ # Infrastructure layer
│ ├── config.py # Endpoint routing / default agent configs /
│ │ # seed & image paths / API credentials
│ ├── llm.py # Async LLM client, retry logic, JSON parsing,
│ │ # multimodal image encoding, batch operations
│ ├── usage.py # Token accounting & cost estimation
│ └── layout.py # Directory path builders (timestamp/phase/image)
├── data/ # Data ingestion & management layer
│ ├── seeds.py # Seed corpus loading, sampling, base64 encoding
│ └── ingest.py # Heterogeneous input normalization: any format
│ │ # → standard schema, deduplication, cataloging
├── prompts/ # Agent prompt specifications (per stage)
│ ├── generate.py # Generation agent system prompt & templates
│ ├── batch_verify.py # Verification agent (batch semantics)
│ ├── difficulty.py # Difficulty evaluation agent
│ └── grade.py # Answer equivalence judgement
└── stages/ # Pipeline stages (agent orchestration)
├── generate.py # Stage 1: Image-to-problems synthesis
├── batch_verify.py # Stage 2: Batch validation & ground-truth
│ # correction
└── difficulty.py # Stage 3: Difficulty assessment & filtering
New datasets in any format (JSON/JSONL/CSV/TSV) with embedded images are automatically normalized to the standard schema via field-alias mapping, deduplicated by (problem_statement, answer), and appended to the corpus. Existing seeds need no modification.
Preview mode (validation without persistence):
python -m mmforge.data.ingest --input new.jsonl --images /path/to/imgs --dry-runIngest with custom field mapping:
python -m mmforge.data.ingest \
--input new.csv \
--images /path/to/imgs \
--map question=stem answer=gt image=fig \
--source dataset_v2Common aliases are auto-detected: stem/problem → question, gt/label → answer, fig/img/figure → image, subject → category.
Architecture principle: Data ingestion, problem synthesis, verification, and evaluation are cleanly separated. Update prompts in
prompts/, swap models viacore/config.py, modify orchestration instages/.
The generation agent executes a core competency (commanding) combined with stochastically sampled template axes, ensuring diversity across the 6–8 problems per image:
- Transformation Axis
TRANSFORM_TEMPLATES: Parameter generalization · Substituted unknowns · Constraint superposition · Multi-element coupling · Inverse reasoning · Extremization · Decomposition & counting - Problem Type Axis
TYPE_TEMPLATES: Metric measurement · Angular relationships · Ratio analysis · Cardinality · Optimization · Coordinate-algebraic quantities · Probabilistic inference - Presentation Axis
STYLE_TEMPLATES: Competition style (terse) · Contextual narratives · Deep reasoning · Minimal elegance
Each problem is tagged with its template composition in the gen_templates field for provenance tracking.
Multimodal requirement: Unlike generic problem synthesis, every generated problem must depend on the provided image. Problem statements cannot stand alone; they reference visual elements intrinsic to the source image.
Answer format: unique short-form answers only (numerical | expression | set | cardinality | angle), typeset in MathJax, excluding proofs and multiple-choice.
cd /path/to/mathforge_multimodal
pip install -r requirements.txtAPI credentials are passed via --api-key flag or AIHUBMIX_API_KEY environment variable. Never hardcode keys.
Full end-to-end pipeline (5 images × 6 problems each):
python run_pipeline.py --api-key $YOUR_API_KEY --num-images 5 --per-image 6Modular stage execution (all stages share --run-dir for intermediate outputs):
python -m mmforge.stages.generate --api-key $YOUR_API_KEY --num-images 5 --per-image 6 --run-dir output/run_x
python -m mmforge.stages.batch_verify --api-key $YOUR_API_KEY --run-dir output/run_x
python -m mmforge.stages.difficulty --api-key $YOUR_API_KEY --run-dir output/run_xQuick validation (skip verification, observe target-agent behavior):
python run_pipeline.py --api-key $YOUR_API_KEY --num-images 1 --skip-verifyConfigured in mmforge/core/config.py (all overridable via CLI flags):
| Stage | Agent | Role |
|---|---|---|
| ① Generation | doubao-seed-2-1-turbo |
Vision agent: reads image, synthesizes problem variants via template combinatorics |
| ② Verification | gpt-5.5 |
Reasoning agent: batch validates problem viability, extracts ground-truth answers |
| ③ Difficulty Gate · Target | doubao-seed-2-0-pro |
Vision agent: autonomous problem solving; failure → problem qualifies |
| ③ Difficulty Gate · Judge | doubao-seed-2-1-pro |
Equivalence agent: compares target agent's answer to ground truth |
Verification strategy: Batch verification uses a stronger, orthogonal model (GPT-5.5) to ensure both problem validity and trustworthy ground-truth answers. This establishes the baseline against which difficulty gating operates. One batch per image: all problems from the same image are evaluated in a single request, minimizing latency.
- Image Reuse: All downstream problems inherit the seed's image, passed as
image_urlto vision agents across all three stages. - Agent Specialization: Each stage isolates a single concern: generate focuses on coverage, verify ensures consistency, difficulty gates on complexity.
- Answer Normalization: Verified problems have their
answer/reference_answerfields overwritten with the ground-truth value from the verification agent. Corrections are logged (answer_corrected: true, original value preserved) for audit trails. - Modular Resumption: Any stage can be re-executed independently, enabling iterative refinement of agent prompts or model selection without re-running prior stages.
- Qualified Subset: Only problems that pass all three gates are marked as qualified; full problem lists (including unqualified) are preserved for analysis.
Stage 1 (1_generate/): Raw problem synthesis, unvalidated.
Stage 2 (2_verify/): Validated problems with corrected answers; rejected problems moved to dropped.jsonl.
Stage 3 (3_difficulty/): Complete problem set with difficulty scores; qualified.jsonl contains target-agent-failed problems (sufficiently challenging).