SE audit: decompose orchestrator, fix defects, 5.4x hot-path perf (byte-identical)

.github/workflows/monthly-refresh.yml

@@ -171,12 +171,12 @@ jobs:
           # Update README badges from badges.json
           if [ -f output/badges.json ]; then
             LAST_UPDATED=$(python3 -c "import json; print(json.load(open('output/badges.json'))['last_updated'])")
-            POS_HITS=$(python3 -c "import json; print(json.load(open('output/badges.json'))['cache_positive_hits'])")
-            NEG_HITS=$(python3 -c "import json; print(json.load(open('output/badges.json'))['cache_negative_hits'])")
+            TOTAL_QUERIES=$(python3 -c "import json; print(json.load(open('output/badges.json'))['total_queries'])")
+            HIT_RATE=$(python3 -c "import json; print(json.load(open('output/badges.json'))['hit_rate'])")
             LAST_UPDATED_ESC=$(echo "$LAST_UPDATED" | sed 's/-/--/g')
             sed -i "s|Updated-[0-9]*--[0-9]*-blue|Updated-${LAST_UPDATED_ESC}-blue|" README.md
-            sed -i "s|Cache_Hits-[0-9]*_positive-brightgreen|Cache_Hits-${POS_HITS}_positive-brightgreen|" README.md
-            sed -i "s|Cache_Misses-[0-9]*_negative-orange|Cache_Misses-${NEG_HITS}_negative-orange|" README.md
+            sed -i "s|Queries-[0-9]*-8A2BE2|Queries-${TOTAL_QUERIES}-8A2BE2|" README.md
+            sed -i "s|Cache_Hit_Rate-[0-9.]*%25-2ea44f|Cache_Hit_Rate-${HIT_RATE}%25-2ea44f|" README.md
           fi
 
           git add output/ README.md

.github/workflows/tests.yml

@@ -7,6 +7,36 @@ on:
     branches: [ main, master ]
 
 jobs:
+  lint:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: Set up Python
+      uses: actions/setup-python@v5
+      with:
+        python-version: "3.12"
+
+    - name: Cache pip dependencies
+      uses: actions/cache@v4
+      with:
+        path: ~/.cache/pip
+        key: ${{ runner.os }}-pip-lint-${{ hashFiles('pyproject.toml') }}
+        restore-keys: |
+          ${{ runner.os }}-pip-
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -e .[dev]
+
+    - name: Lint (ruff)
+      run: ruff check citeforge/ tests/ main.py
+
+    - name: Type check (mypy)
+      run: mypy citeforge/ main.py
+
   test:
     runs-on: ubuntu-latest
     strategy:
@@ -39,19 +69,29 @@ jobs:
 
     - name: Create API key files
       env:
+        SERPAPI_KEY: ${{ secrets.SERPAPI }}
         SERPLY_KEY: ${{ secrets.SERPLY }}
         SEMANTIC_KEY: ${{ secrets.SEMANTICSCHOLAR }}
         OPENREVIEW_KEY: ${{ secrets.OPENREVIEW }}
         GEMINI_KEY: ${{ secrets.GEMINI }}
       run: |
         mkdir -p keys
+        echo "$SERPAPI_KEY" > keys/SerpAPI.key
         echo "$SERPLY_KEY" > keys/Serply.key
         echo "$SEMANTIC_KEY" > keys/Semantic.key
         echo "$OPENREVIEW_KEY" > keys/OpenReview.key
         echo "$GEMINI_KEY" > keys/Gemini.key
 
-    - name: Run tests
-      run: pytest --tb=short
+    - name: Run tests with coverage
+      run: pytest --cov=citeforge --cov=main --cov-report=term-missing --cov-report=xml --cov-fail-under=55
+
+    - name: Upload coverage report
+      if: always()
+      uses: actions/upload-artifact@v4
+      with:
+        name: coverage-${{ matrix.python-version }}
+        path: coverage.xml
+        retention-days: 7
 
     - name: Upload test results on failure
       if: failure()

.gitignore

@@ -35,6 +35,7 @@ ENV/
 # Testing
 htmlcov/
 coverage.xml
+.coverage
 dmypy.json
 
 # User data and API keys
@@ -53,6 +54,10 @@ keys/
 temp/
 tmp/
 
+# Pipeline run artifacts (generated logs must never be committed)
+output/run.log
+output/**/*.log
+
 # Build artifacts
 third-party/*
 ruff-report.json

CLAUDE.md

@@ -0,0 +1,74 @@
+# CLAUDE.md
+
+Project overview and data sources: @README.md
+
+## Build & Run
+
+```bash
+pip install -e .          # Install (editable)
+pip install -e .[dev]     # Install with dev tools
+python3 main.py           # Run pipeline (default: data/input.csv)
+python3 main.py data/custom.csv   # Custom input
+python3 main.py --force   # Force re-enrichment (ignore cache completeness)
+```
+
+## Quality Gates
+
+All three must pass before merge:
+
+```bash
+ruff check citeforge/ tests/ main.py      # Lint
+mypy citeforge/ main.py                    # Type check (strict, ignore_missing_imports)
+pytest tests/ -v --tb=short          # Tests (full suite, Python 3.10-3.13)
+```
+
+Single test: `pytest tests/test_merge.py::test_function_name -v --tb=short`
+
+Ruff config: line-length 120, rules E/F/W/I/N/UP/B/C4/SIM/RUF/S (see pyproject.toml for ignores).
+
+## Architecture
+
+`main.py` is a thin command-line entry point (~120 LOC) that loads API keys, reads author records, and delegates to the `citeforge/pipeline/` package (`article.py` for per-article enrichment, `scheduler.py` for author-level scheduling, `postrun.py` for the post-run tail). Each article passes through Phase 1 (DOI validation) → Phase 2 (multi-API enrichment) → Phase 2.5 (SerpAPI publication string fallback) → Phase 3 (late DOI inference) → Phase 4 (trust-based merge + save). Post-run steps run in order: flush CSV → reconcile phantoms → remove orphans → year-window cleanup → post-run fixup → build a2i2 → rebuild baseline.json → rebuild badges.json.
+
+Trust hierarchy in `citeforge/merge_utils.py:merge_with_policy()` merges fields from 13 ranked sources with special override rules for DOI (published > preprint), journal (never downgrade to preprint), title (prefer longer), pages (reject invalid), and booktitle (upgrade generic series to conference name).
+
+## Three-Way Fix Pattern (CRITICAL)
+
+**Fixes for entry types, titles, and booktitles MUST be applied in three places or oscillation occurs between consecutive runs:**
+
+1. `_fixup_bib_entry()` — initial fixup on load
+2. Existing-file fixup in `process_article()` — before enrichment
+3. Phase 4 post-merge — after trust-based merge
+
+Consolidated helpers `_fix_title_text()` and `_apply_booktitle_fixups()` are called from all three. When adding a new text or type fix, grep for these functions and add the fix to all call sites.
+
+The following fixes are also applied in all 3 locations:
+- Abbreviated venue expansion (`ABBREVIATED_VENUE_MAP`)
+- Venue case correction (`VENUE_CASE_CORRECTIONS`)
+- Publisher-duplicate-container stripping (publisher == journal/booktitle → remove publisher)
+- `JOURNALS_NAMED_PROCEEDINGS` guard (conference-keyword suffix check before reclassifying)
+- `ACM_JOURNAL_PROCEEDINGS` guard (PACM journals excluded from conference-as-journal reclassification)
+
+## Key Conventions
+
+- **Config-driven**: All thresholds, API endpoints, trust order, rate limits, compound word dictionaries live in `citeforge/config.py`. Never hardcode these values elsewhere.
+- **Determinism**: Pipeline produces byte-identical output across consecutive cache-hit runs. Use `sorted()` for all directory/file iterations. No randomization in output-affecting code.
+- **DOI normalization**: Always use `_norm_doi()` from `citeforge/id_utils.py`. Always pair DOI matches with `title_similarity >= 0.55` check.
+- **Preprint detection**: Uses `PREPRINT_SERVERS`, `PREPRINT_DOI_PREFIXES`, and `PREPRINT_ONLY_PUBLISHERS`. Check all three for completeness.
+- **Container fields**: `@article` → `journal`, `@inproceedings`/`@incollection` → `booktitle`, `@misc` → `howpublished`. See `get_container_field()`.
+- **Repository guard**: `REPOSITORY_AS_JOURNAL` (Zenodo, OSTI, Figshare, etc.) prevents @misc→@inproceedings oscillation.
+- **Thesis detection**: @article with "university"/"institut" in journal → @phdthesis.
+- **Book-chapter DOI**: `.ch\d+` in DOI → Wiley book chapter → reclassify @article → @incollection.
+- **Content comparison guard**: Post-run fixup compares serialized output to existing file before writing, preventing phantom writes from serializer normalization.
+- **Cache defensive copy**: `ResponseCache.get()` returns `dict(...)` copy to prevent mutation.
+- **Orphan safety**: Never blindly delete orphan .bib files — verify as duplicates via `title_similarity >= 0.95`.
+- **CSV paths**: Relative to CWD (must run from project root).
+- **Fused compounds**: Never use `---` (em-dash) or accented characters in `FUSED_COMPOUND_WORDS` or `ABBREVIATED_VENUE_MAP` — serializer strips them.
+
+## Testing Patterns
+
+- Tests in `tests/` mirror `citeforge/` modules (e.g., `test_merge.py` tests `merge_utils.py`)
+- `tests/conftest.py` + `tests/fixtures.py` provide shared fixtures
+- Integration tests requiring API keys auto-skip when keys unavailable
+- Use `monkeypatch` for HTTP mocking; never make real API calls in unit tests
+- Do NOT create automated audit modules. Fix issues via pipeline code or direct .bib edits.

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2025 Gabriel Spadon
+Copyright (c) 2025-2026 Gabriel Spadon
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -6,8 +6,8 @@
   <a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License"></a>
   <br>
   <img src="https://img.shields.io/badge/Updated-2026--06-blue.svg" alt="Last Updated">
-  <img src="https://img.shields.io/badge/Queries-8%2C104-8A2BE2.svg" alt="Total Queries">
-  <img src="https://img.shields.io/badge/Cache_Hit_Rate-96.1%25-2ea44f.svg" alt="Cache Hit Rate">
+  <img src="https://img.shields.io/badge/Queries-8446-8A2BE2.svg" alt="Total Queries">
+  <img src="https://img.shields.io/badge/Cache_Hit_Rate-78.9%25-2ea44f.svg" alt="Cache Hit Rate">
 </p>
 
 <p align="center">
@@ -31,7 +31,7 @@ CiteForge takes care of this. Point it at a list of authors with their Google Sc
 - Merge fields using a **multi-level trust hierarchy** that prefers authoritative sources; and,
 - Output clean, LaTeX-ready `.bib` files organized by author.
 
-The result is deterministic — on cache-hit runs, CiteForge produces **byte-identical output** across consecutive runs, verified by SHA-256 checksums.
+The result is deterministic. On cache-hit runs, CiteForge produces **byte-identical output** across consecutive runs, verified by SHA-256 checksums.
 
 ## Getting Started
 
@@ -61,7 +61,7 @@ python3 main.py data/custom.csv  # Custom input
 python3 main.py --force          # Force re-enrichment
 ```
 
-The input CSV has three columns — name, Scholar link, and an optional DBLP link:
+The input CSV has three columns (name, Scholar link, and an optional DBLP link):
 
 ```csv
 Name,Scholar Link,DBLP Link
@@ -91,11 +91,11 @@ A trust-based consolidation stage merges the collected records according to sour
 
 The pipeline further corrects recurrent metadata issues, such as fragmented compound words, misclassified publication types, invalid page ranges, and titles written entirely in capital letters. Deterministic caching ensures that cache-hit executions produce byte-identical outputs, verified through SHA-256 checksums.
 
-To maintain efficiency and stability, author queries run in parallel while respecting per-API rate limits. All configurable parameters, including source trust order, similarity thresholds, rate limits, and venue mappings, are centralized in [`src/config.py`](src/config.py).
+To maintain efficiency and stability, author queries run in parallel while respecting per-API rate limits. All configurable parameters, including source trust order, similarity thresholds, rate limits, and venue mappings, are centralized in [`citeforge/config.py`](citeforge/config.py).
 
 ## Data Sources
 
-[SerpAPI](https://serpapi.com/) and [Serply](https://serply.io/) require keys — everything else is free or optional:
+[SerpAPI](https://serpapi.com/) and [Serply](https://serply.io/) require keys. Everything else is free or optional:
 
 - **Required:** [SerpAPI](https://serpapi.com/) (Google Scholar), [Serply](https://serply.io/) (citation details);
 - **Recommended:** [Semantic Scholar](https://www.semanticscholar.org/);
@@ -122,4 +122,4 @@ If you use CiteForge in your research or find it useful, please consider citing
 
 ## License
 
-This project is licensed under the **MIT [License](LICENSE)** — you're free to use, modify, and distribute it for any purpose.
+This project is licensed under the **MIT [License](LICENSE)**. You're free to use, modify, and distribute it for any purpose.