CI: Add LLM-powered documentation sync job

[quinn-dougherty]

Summary

Add a CI job that uses an LLM to audit and synchronize documentation across the repository, ensuring consistency between code, book content, and README files.

Motivation

Documentation can drift from implementation as code evolves. An automated LLM-based sync job can:

• Catch discrepancies between book chapters and actual code
• Update outdated references and examples
• Ensure CLAUDE.md, READMEs, and book content stay aligned
• Open PRs with small fixes automatically

Proposed Implementation

Trigger Options (choose one or combine):

• On push to master (most responsive)
• Nightly (good balance)
• Weekly (less noisy, adequate for slower-moving projects)

Scope

The job should walk through:

• ./book/**/*.md - All book chapters
• ./evals/src/** - Implementation code
• ./README.md - Root documentation
• ./CLAUDE.md - Project instructions
• ./evals/README.md, ./book/README.md - Member READMEs
• ./scripts/README.md (if exists)

Verification Tasks

The LLM should check:

1. Code references: Do literalinclude directives point to correct line ranges?
2. API consistency: Do documented function signatures match implementation?
3. Command examples: Are CLI examples up-to-date with actual interface?
4. Cross-references: Do internal links between chapters work?
5. Dependency versions: Do installation instructions match pyproject.toml?
6. Configuration examples: Does sample config match actual config files?

Behavior

• If in sync: Job passes, no action
• If out of sync: Job opens a PR with:
  • Clear description of what was out of sync
  • SMALL edits only (documentation updates, not code changes)
  • Links to relevant issues/commits that caused drift
  • Passes all existing tests before opening PR

Constraints

• Small edits only: Documentation, comments, docstrings, book prose
• No implementation changes: Never modify actual code logic
• No new features: Only sync existing documentation
• Idempotent: Running twice should not create duplicate PRs

Technical Approach

Potential tools/frameworks:

• GitHub Actions workflow with cron schedule
• Anthropic Claude API (we're already using it for evals)
• Claude Code Agent SDK or inspect-ai for orchestration
• gh CLI for opening PRs programmatically

Example workflow structure:

name: Documentation Sync
on:
  push:
    branches: [master]
  schedule:
    - cron: '0 0 * * 0'  # Weekly on Sunday
  workflow_dispatch:  # Manual trigger

jobs:
  sync-docs:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.13'
      - name: Install dependencies
        run: |
          pip install uv
          uv sync
      - name: Run LLM documentation audit
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          uv run scripts/audit_docs.py
      - name: Create PR if changes
        if: ${{ steps.audit.outputs.has_changes }}
        run: |
          gh pr create --title "docs: Auto-sync from CI" --body "..."

Open Questions

1. Trigger frequency: Nightly vs weekly vs on-push?
2. Review requirements: Auto-merge if tests pass, or require human review?
3. Cost concerns: Claude API usage limits (probably negligible)
4. Failure handling: What if LLM makes a mistake? Manual review required?
5. Branch strategy: One long-lived docs-sync branch or new branch per run?

Success Criteria

• CI job runs on schedule/trigger without failing
• Job correctly identifies real documentation drift (no false positives)
• Opened PRs contain only small, relevant documentation updates
• Job doesn't create duplicate PRs for same issues
• Documentation stays < 1 week out of sync with code

Related

• See operations/ for existing CI/CD setup (deploy.yml)
• Budget: Claude API key needed in GitHub Secrets
• Consider expanding to check example outputs in book match current model behavior

[quinn-dougherty]

Actually: better than writing a script with the agent in it would just be to tag @-copilot