A comprehensive system for extracting social actor profiles from narrative sources, building grounded behavioral benchmarks, and validating predictive decision-making at scale. This framework transforms unstructured oral histories and biographies into a structured Social World Model capable of world description and world prediction with verified empirical performance.
The Social World Model (SWM) pipeline transforms raw biographical narratives into a three-tier knowledge architecture:
- World Description Layer: Extract structured actor profiles, relationships, decision contexts, and temporal trajectories
- Prediction Layer: Validate predictive accuracy on held-out decisions using only pre-event information (60/40 temporal split)
- Synthetic Data Foundation: Generate grounded synthetic scenarios for model training and validation, ensuring consistency with real-world behavioral patterns
Iterative refinement across 4 major cycles on 14 individuals with 112 total benchmark questions:
| Iteration | Baseline Accuracy | Full Context Accuracy | Key Insights | Grounded Questions |
|---|---|---|---|---|
| 0 | 67.2% | 79.7% | Heavy text hints in initial generation | 112/112 (100%) |
| 1 | 53.0% | 63.6% | LLM self-review too lenient on C5 leakage | 94/112 (84%) |
| 2 | 18.6% | 41.4% | 73% of questions were LLM-hallucinated without real grounding | 21/112 (19%) |
| 3 | 28.4% | 52.1% | Forced character-level evidence audit; 21 rigorously grounded questions only | 21/112 (19%) |
Critical Finding: Iteration 3 demonstrates that rigorous grounding validation is more important than volume. The 21 truly-grounded questions (validated by character-level substring matching) show meaningful lift (28.4% vs 25% random baseline for 4-choice), confirming predictive signal exists in the behavioral data.
The Social World Model framework creates a two-way bridge between real behavioral data and synthetic data infrastructure:
- Grounded scenarios: Extract decision contexts, actors, outcomes, and preconditions from validated oral histories
- Behavioral constraints: Map real-world decision patterns (success rates, reversibility, social costs) into synthetic scenario generators
- Evidence grounding: Ensure synthetic scenarios preserve the "evidence quote" structure of real data—every synthetic decision must be traceable to source patterns
- Generative model tuning: Iteratively improve synthetic scenario generation by comparing against real behavioral distributions
- Predictability validation: Test whether synthetic scenarios trained on real data maintain similar prediction difficulty and decision signal
- Ethical alignment: Use grounded real-world behavioral patterns to constrain synthetic scenario generation (prevent unrealistic or harmful scenarios)
┌─────────────────────────────────────┐
│ Oral Histories / Biographies │
│ (Real-world behavioral data) │
└────────────────┬────────────────────┘
│
┌───────▼────────┐
│ Extract Profiles│
│ & Contexts │
└───────┬────────┘
│
┌────────────┼────────────┐
│ │ │
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌──────────────┐
│Evidence │ │Decision │ │Synthetic Data│
│Grounding│ │Patterns │ │Generation │
└────┬────┘ └────┬────┘ └──────┬───────┘
│ │ │
└───────────┼─────────────┘
│
┌───────▼──────────┐
│Social World Model│
│ (Unified Layer) │
└───────┬──────────┘
│
┌────────────┼────────────┐
│ │ │
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌──────────────┐
│ World │ │ World │ │ Contrastive │
│Descrip. │ │Predict. │ │Benchmarks │
└─────────┘ └─────────┘ └──────────────┘
- Reduces synthetic data bias: By anchoring synthetic scenarios to real behavioral distributions, we avoid the "simulation gap"
- Improves model generalization: Models trained on synthetic data grounded in reality transfer better to real-world prediction tasks
- Enables transparent evaluation: Each synthetic question inherits the grounding chain from real data—"ground truth" is always traceable
- Unlocks scalability: Real data provides the seed patterns; synthetic generation multiplies the training signal while maintaining behavioral consistency
Raw Book (AZW3/EPUB/PDF/TXT)
│
├─→ [EXTRACTION] Plain text + character segmentation
│
├─→ [PROFILING] For each person:
│ ├─ Interview transcription (markdown)
│ ├─ Structured CV (education, career, relationships)
│ ├─ Social network graph
│ └─ 60/40 temporal split point
│
├─→ [BENCHMARKING] Decision prediction task generation:
│ ├─ Tier 1 Audit: Deterministic checks (answer leakage, uniformity)
│ ├─ Tier 2 Audit: LLM self-review + empirical C5 leak detection
│ └─ Tier 3 Audit: Independent subagent cold-start review
│
└─→ [EVALUATION] Accuracy metrics:
├─ Baseline: Questions answered without pre-60% context (random ~25%)
├─ Full: Questions answered with full pre-60% context
└─ Grounding score: % questions with validated evidence quotes
Social-World-Model/
├── README.md ← This file
├── pipeline.sh ← Main entry point (orchestrates phases)
├── docs/
│ ├── paper-draft.md ← Theoretical foundation & PPA model
│ ├── agents.md ← Agent design & audit strategy
│ ├── skill-design.md ← Books-to-people-db skill reference
│ ├���─ deepseek-env.md ← Environment setup (remove before commit)
│ └── superpowers/plans/ ← Implementation roadmap
├── scripts/
│ ├── benchmark_eval.py ← Evaluate question accuracy
│ ├── verify_grounding.py ← Validate evidence quotes
│ ├── empirical_c5.py ← Detect baseline leakage via real API
│ ├── audit_benchmark.py ← Tier 1 static audits
│ ├── diagnose_baseline_leak.py ← Root-cause analysis for leaking questions
│ ├── status.py ← Manage .status.json workflow state
│ ├── reset_benchmark.py ← Clear benchmark for incremental runs
│ ├── legacy/ ← Deprecated splitting logic
│ └── tests/ ← Unit & e2e tests
├── people/ ← Unified person index
│ ├── 重建中国社会学/ ← Book 1: 40 Chinese sociologists
│ │ ├── 金耀基/
│ │ │ ├── 1_原文_访谈_金耀基.md
│ │ │ ├── 2_简历_金耀基.md
│ │ │ ├── 3_人物关系_金耀基.md
│ │ │ ├── benchmark/
│ │ │ │ ├── pre_60_materials.md
│ │ │ │ └── questions.json ← {"question", "options", "correct_answer", "evidence_quote", "decision_point_quote"}
│ │ │ └── .status.json ← Workflow tracking
│ │ └── [... 39 more sociologists]
│ ├── 北京口述历史/ ← Book 2: 58 Beijing oral history subjects
│ ├── 推不走的回忆/ ← Book 3: 14 Macau street vendors
│ ├── [... additional books]
│ └── manifest.json ← Global person registry
├── testinput/ ← Raw source books (AZW3/EPUB/PDF/TXT)
├── output/extracted/ ← Intermediate plain-text extraction
├── benchmark_samples/ ← Legacy hand-crafted benchmark samples (reference)
├── mineru_repo/ ← PDF OCR Docker source code
└── archive/ ← Compressed old artifacts & logs
# System requirements
- Python 3.13
- calibre (ebook-convert)
- Docker (for PDF OCR via MinerU)
- Claude Code CLI ≥ 2.1
# Python dependencies
pip install anthropic>=0.28.0
# Environment setup (see docs/deepseek-env.md for details)
# Set ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN, model configs# Full end-to-end processing
./pipeline.sh --book testinput/my_book.epub --phase=all
# Phased execution (useful for debugging)
./pipeline.sh --book testinput/my_book.epub --phase=extract
./pipeline.sh --book testinput/my_book.epub --phase=profile
./pipeline.sh --book testinput/my_book.epub --phase=benchmark
# Generate benchmarks for specific people only
./pipeline.sh --book testinput/my_book.epub --phase=benchmark \
--benchmark-only=张三,李四,王五
# Resume interrupted runs—.status.json automatically skips completed work
./pipeline.sh --book testinput/my_book.epub --phase=all# 1. Accuracy Assessment (full context vs. baseline)
python3 scripts/benchmark_eval.py --book my_book_name # Full context accuracy
python3 scripts/benchmark_eval.py --book my_book_name --baseline # Baseline (no pre-60) accuracy
# 2. Grounding Validation (verify quotes are real, not LLM hallucinations)
python3 scripts/verify_grounding.py --book my_book_name
# 3. Baseline Leak Diagnosis (identify which hints are helping models guess correctly)
python3 scripts/diagnose_baseline_leak.py \
--book my_book_name \
--baseline-eval my_book_name/eval_*_baseline.json \
--limit 15
# 4. Empirical C5 Audit (test if models can guess without background context)
python3 scripts/empirical_c5.py --benchmark-dir my_book_name/interviews/person_name/benchmarkThe pipeline enforces multi-stage quality validation to ensure benchmark questions are grounded, fair, and predictively meaningful:
| Tier | Check | Implementation | Purpose |
|---|---|---|---|
| 1A (Deterministic) | Answer uniformity, retroactive language removal, answer text leakage | audit_benchmark.py (grep-based) |
Catch obvious data generation artifacts |
| 1B (Grounding) | evidence_quote and decision_point_quote are verbatim substrings from source; positions fall in post-split (future) section |
audit_benchmark.py (substring + char validation) |
Prevent LLM hallucination; ensure "ground truth" exists |
| Tier 2 (LLM Self-Review) | C1-C4: Is this a decision task (not trivia)? Does pre_60 leak? Are options reasonable? Is decision point in future? | SKILL Tier 2 Judge Prompt (LLM reasoning) | Catch subtle prompt-injection and information leakage |
| Tier 2 (C5 Empirical) | Call real API without background context 3x/question; >50% accuracy = leak | empirical_c5.py (API calls to claude-3.5-sonnet) |
Ground truth test: no model should guess >50% without signal |
| Tier 3 (Independent Review) | Spawn independent subagent (cold-start, no memory) to audit main agent's questions | SKILL Stage 3 subagent spawn | Break feedback loops; external perspective |
| Human Escalation | Flag for manual review after 2 external audit failures | .status.json needs_human_review=true |
Tie-breaker when automated checks conflict |
Each person's folder contains .status.json:
{
"extracted": "2026-05-24T08:30:00Z",
"profiled": "2026-05-24T09:15:00Z",
"benchmarked": "2026-05-24T10:45:00Z",
"split": {
"year": 1995,
"char_position": 45230,
"char_ratio": 0.61
},
"self_review": {
"passed": true,
"timestamp": "2026-05-24T10:50:00Z"
},
"external_review": {
"passed": true,
"timestamp": "2026-05-24T10:55:00Z"
},
"needs_human_review": false,
"benchmark_count": 8,
"grounded_count": 8
}# Full test suite (no LLM calls)
bash tests/test_pipeline_e2e.sh
# Unit tests
python3 tests/test_status.py
python3 tests/test_audit_benchmark.py
python3 tests/test_verify_grounding.pyThis project demonstrates the iterative path from naive synthetic data generation to grounded, validated benchmarks:
Iteration 0: Baseline Generation
- 112 questions generated from 14 subjects
- Baseline accuracy: 67.2% (heavily leaked)
- Full accuracy: 79.7% (unhealthy gap)
- Lesson: LLM prompts naturally leak future information; questions seem "natural" but contain subtle hints
Iteration 1: Self-Review
- Applied LLM self-review (Tier 2 C1-C4 audit)
- Removed 18 flagged questions
- Baseline dropped to 53.0% (good sign of actual removal)
- But full accuracy also dropped to 63.6%
- Lesson: Self-review can be gamed; models are lenient reviewers of their own work
Iteration 2: Empirical C5 Audit (Real API)
- Called claude-3.5-sonnet 3x per question without context
-
50% accuracy indicated leakage
- Resulted in deletion of 91/112 questions (~80% rejected)
- Baseline: 18.6%, Full: 41.4%
- Critical Finding: 73% of questions were hallucinated without real grounding in source text
- Lesson: Synthetic question generation without strict grounding quickly diverges from real world
Iteration 3: Character-Level Grounding Enforcement
- Enforced that
evidence_quoteanddecision_point_quoteare exact verbatim substrings from source - Validated char positions to ensure quotes fall in post-split (future) section
- Retained only 21 truly grounded questions from original 112
- Baseline: 28.4% (above-random but not leaked)
- Full accuracy: 52.1% (reasonable signal-to-noise ratio)
- Conclusion: Quality > quantity. 21 grounded questions are more valuable than 112 hallucinated ones
-
Grounding-First Approach: All synthetic data must be traceable to evidence in real source material. No "hallucinated" scenarios.
-
Temporal Train-Test Split: 60/40 split ensures we only ask about decisions that happened after the person has lived through 60% of their recorded life. Prevents retrospective bias.
-
Multi-Layer Audit: 6-tier validation catches different failure modes:
- Static (answer format, uniformity)
- Linguistic (leaked context, retroactive language)
- Behavioral (LLM hallucination via empirical C5)
- Organizational (independent review)
-
Synthetic Data Grounding: Bridges the simulation gap by deriving synthetic scenarios from real behavioral patterns. Synthetic questions inherit the grounding chain of real data.
-
Social Network Integration: Extract and validate actor relationships, enabling network-aware prediction (context includes social structure, not just individual history).
- DeepSeek API supports Anthropic SDK via compatibility layer
- Enables headless (
claude -p) execution for large-scale batch processing - Cost-effective for long-context tasks (80K+ token windows)
- 60% provides enough biographical context for meaningful prediction
- 40% is sufficient for varied, genuine decisions (not all trivial)
- Avoids extreme cases (e.g., 90% splits with only 1-2 future events)
- Float-based
char_positionprevents off-by-one errors in substring extraction - Validates that evidence isn't just "same word appears elsewhere" but true verbatim excerpt
- Makes hallucination detection automatic and cheap (no LLM needed for verification)
-
Language-Specific: Currently optimized for Chinese biographies. Extension to other languages requires script adjustments for character splitting.
-
Subjective Interpretation: Some decisions have multiple valid interpretations. Framework assumes single "correct" answer per question; future work could explore multi-answer or fuzzy matching.
-
Cold-Start Problem: Framework is most effective on well-documented lives (biographies, oral histories). Sparse or fragmentary sources yield fewer grounded questions.
-
Synthetic Data Scale: Current system generates 5-10 questions per person. Larger-scale synthetic data generation (100+ per person) is future work, pending grounding validation improvements.
If you use this framework, please cite:
@inproceedings{social_world_model_2026,
title={Social World Model: Grounded Behavioral Prediction from Narratives},
author={Entropyorder},
year={2026},
note={Available at https://github.com/Entropyorder/Social-World-Model}
}For questions, bug reports, or collaboration inquiries:
- GitHub Issues: Report bugs or feature requests
- Discussions: Share ideas and research directions
[To be specified by project owner]
Last Updated: 2026-05-24
Framework Version: 3.0 (Character-level grounding validation)
Status: Active research & development