refactor: streamline, deduplicate, and optimize CiteForge end-to-end

CLAUDE.md

@@ -7,24 +7,23 @@ Project overview and data sources: @README.md
 ```bash
 pip install -e .          # Install (editable)
 pip install -e .[dev]     # Install with dev tools
-python3 main.py           # Run pipeline (default: data/input.csv)
-python3 main.py data/custom.csv   # Custom input
+python3 main.py           # Run pipeline (input: data/input.csv)
 python3 main.py --force   # Force re-enrichment (ignore cache completeness)
 ```
 
 ## Quality Gates
 
-All three must pass before merge:
+All three must pass before merge.
 
 ```bash
 ruff check citeforge/ tests/ main.py      # Lint
 mypy citeforge/ main.py                    # Type check (strict, ignore_missing_imports)
 pytest tests/ -v --tb=short          # Tests (full suite, Python 3.10-3.13)
 ```
 
-Single test: `pytest tests/test_merge.py::test_function_name -v --tb=short`
+Run a single test with `pytest tests/test_core.py::test_function_name -v --tb=short`.
 
-Ruff config: line-length 120, rules E/F/W/I/N/UP/B/C4/SIM/RUF/S (see pyproject.toml for ignores).
+Ruff uses line-length 120 and rules E/F/W/I/N/UP/B/C4/SIM/RUF/S (see pyproject.toml for ignores).
 
 ## Architecture
 
@@ -34,15 +33,15 @@ Trust hierarchy in `citeforge/merge_utils.py:merge_with_policy()` merges fields
 
 ## Three-Way Fix Pattern (CRITICAL)
 
-**Fixes for entry types, titles, and booktitles MUST be applied in three places or oscillation occurs between consecutive runs:**
+Fixes for entry types, titles, and booktitles MUST be applied in three places, or output oscillates between consecutive runs.
 
-1. `_fixup_bib_entry()` — initial fixup on load
-2. Existing-file fixup in `process_article()` — before enrichment
-3. Phase 4 post-merge — after trust-based merge
+1. `_fixup_bib_entry()` (initial fixup on load)
+2. Existing-file fixup in `process_article()` (before enrichment)
+3. Phase 4 post-merge (after trust-based merge)
 
 Consolidated helpers `_fix_title_text()` and `_apply_booktitle_fixups()` are called from all three. When adding a new text or type fix, grep for these functions and add the fix to all call sites.
 
-The following fixes are also applied in all 3 locations:
+The following fixes are also applied in all three locations.
 - Abbreviated venue expansion (`ABBREVIATED_VENUE_MAP`)
 - Venue case correction (`VENUE_CASE_CORRECTIONS`)
 - Publisher-duplicate-container stripping (publisher == journal/booktitle → remove publisher)
@@ -61,13 +60,13 @@ The following fixes are also applied in all 3 locations:
 - **Book-chapter DOI**: `.ch\d+` in DOI → Wiley book chapter → reclassify @article → @incollection.
 - **Content comparison guard**: Post-run fixup compares serialized output to existing file before writing, preventing phantom writes from serializer normalization.
 - **Cache defensive copy**: `ResponseCache.get()` returns `dict(...)` copy to prevent mutation.
-- **Orphan safety**: Never blindly delete orphan .bib files — verify as duplicates via `title_similarity >= 0.95`.
+- **Orphan safety**: Never blindly delete orphan .bib files; verify as duplicates via `title_similarity >= 0.95`.
 - **CSV paths**: Relative to CWD (must run from project root).
-- **Fused compounds**: Never use `---` (em-dash) or accented characters in `FUSED_COMPOUND_WORDS` or `ABBREVIATED_VENUE_MAP` — serializer strips them.
+- **Fused compounds**: Never use `---` (em-dash) or accented characters in `FUSED_COMPOUND_WORDS` or `ABBREVIATED_VENUE_MAP` (the serializer strips them).
 
 ## Testing Patterns
 
-- Tests in `tests/` mirror `citeforge/` modules (e.g., `test_merge.py` tests `merge_utils.py`)
+- Tests in `tests/` mostly mirror `citeforge/` modules (e.g., `test_canonicalize.py` tests `canonicalize.py`); merge coverage lives in `test_deduplication.py`, `test_save_entry.py`, and `test_core.py`
 - `tests/conftest.py` + `tests/fixtures.py` provide shared fixtures
 - Integration tests requiring API keys auto-skip when keys unavailable
 - Use `monkeypatch` for HTTP mocking; never make real API calls in unit tests

README.md

@@ -12,63 +12,61 @@
 
 <p align="center">
   <strong>Automated academic citation enrichment from multiple scholarly APIs.</strong><br>
-  CiteForge fetches, validates, deduplicates, and merges bibliographic<br>
-  metadata so you don't have to. Given a list of authors, it<br>
-  produces clean BibTeX files ready for LaTeX.
+  Given a list of authors, CiteForge fetches, validates, deduplicates,<br>
+  and merges bibliographic metadata into per-author BibTeX files.
 </p>
 
 ---
 
-## Why CiteForge?
+## Motivation
 
-If you've ever tried to build a comprehensive publication list for a research group, you know the pain. Google Scholar entries are often incomplete, missing DOIs, and have inconsistent formatting and broken author names. Manually cross-referencing Crossref, Semantic Scholar, arXiv, PubMed, and a handful of other databases gets tedious fast, especially when you're dealing with dozens of authors and hundreds of papers.
+Google Scholar profiles index an author's publications, but the entries are often incomplete, with missing DOIs, inconsistent venue names, and malformed author lists. Correcting them requires cross-referencing registries such as Crossref, Semantic Scholar, arXiv, and PubMed, which is impractical to do by hand at the scale of a research group. CiteForge automates this cross-referencing and consolidation.
 
-CiteForge takes care of this. Point it at a list of authors with their Google Scholar profiles, and it will:
+Given an input CSV of authors with Google Scholar profiles, CiteForge
 
-- Pull every publication from Scholar via [SerpAPI](https://serpapi.com/) and enrich citation details with [Serply](https://serply.io/);
-- Query **multiple academic APIs** for richer metadata on each paper;
-- Deduplicate results using fuzzy title matching, DOI normalization, and author overlap;
-- Merge fields using a **multi-level trust hierarchy** that prefers authoritative sources; and,
-- Output clean, LaTeX-ready `.bib` files organized by author.
+- pulls each author's publications from Scholar via [SerpAPI](https://serpapi.com/) and citation details via [Serply](https://serply.io/);
+- queries additional academic APIs for metadata on each paper;
+- deduplicates results using fuzzy title matching, DOI normalization, and author overlap;
+- merges fields using a multi-level trust hierarchy that prefers authoritative sources; and,
+- writes `.bib` files organized by author.
 
-The result is deterministic. On cache-hit runs, CiteForge produces **byte-identical output** across consecutive runs, verified by SHA-256 checksums.
+Output is deterministic on cache-hit runs.
 
 ## Getting Started
 
-You'll need **Python 3.10+** and a [SerpAPI](https://serpapi.com/) key (the free tier is sufficient for small runs).
+Requires **Python 3.10+** and a [SerpAPI](https://serpapi.com/) key (the free tier is sufficient for small runs).
 
 ```bash
 git clone https://github.com/gabrielspadon/CiteForge.git && cd CiteForge
 pip install -e .
 ```
 
-Drop your API keys into the `keys/` directory:
+Place API keys in the `keys/` directory.
 
 ```bash
 mkdir -p keys
 echo "your_serpapi_key" > keys/SerpAPI.key    # Required
-echo "your_serply_key" > keys/Serply.key      # Required
+echo "your_serply_key" > keys/Serply.key      # Recommended (citation detail skipped without it)
 echo "your_semantic_key" > keys/Semantic.key  # Recommended
 echo "your_gemini_key" > keys/Gemini.key      # Optional
 printf "user\npass" > keys/OpenReview.key     # Optional
 ```
 
-Then create an input CSV and run:
+Then create the input CSV and run the pipeline.
 
 ```bash
-python3 main.py                  # Default: data/input.csv
-python3 main.py data/custom.csv  # Custom input
-python3 main.py --force          # Force re-enrichment
+python3 main.py           # Input: data/input.csv
+python3 main.py --force   # Force re-enrichment (ignore cache completeness)
 ```
 
-The input CSV has three columns (name, Scholar link, and an optional DBLP link):
+The input CSV has three columns (name, Scholar link, and an optional DBLP link).
 
 ```csv
 Name,Scholar Link,DBLP Link
 Gabriel Spadon,https://scholar.google.com/citations?user=bfdGsGUAAAAJ,https://dblp.org/pid/192/1659
 ```
 
-Output is organized per author, with a shared summary and deduplication log:
+Output is organized per author, with a shared summary and run log.
 
 ```
 output/
@@ -85,28 +83,28 @@ API responses are cached under `data/api_cache/` with monthly expiry, so subsequ
 
 ## How It Works
 
-CiteForge begins by retrieving the complete list of publications from Google Scholar through SerpAPI. Each entry is then enriched by querying multiple scholarly services, including Semantic Scholar, Crossref, arXiv, OpenAlex, and PubMed, to gather complementary metadata.
+CiteForge retrieves each author's publication list from Google Scholar through SerpAPI, then enriches every entry by querying scholarly services including Semantic Scholar, Crossref, arXiv, OpenAlex, and PubMed.
 
-A trust-based consolidation stage merges the collected records according to source reliability, consistently prioritizing authoritative registries over scraped content. Duplicate detection combines DOI normalization, external identifier matching, and fuzzy title similarity to identify overlapping entries across sources.
+A trust-based consolidation stage merges the collected records according to source reliability, prioritizing authoritative registries over scraped content. Duplicate detection combines DOI normalization, external identifier matching, and fuzzy title similarity to identify overlapping entries across sources.
 
-The pipeline further corrects recurrent metadata issues, such as fragmented compound words, misclassified publication types, invalid page ranges, and titles written entirely in capital letters. Deterministic caching ensures that cache-hit executions produce byte-identical outputs, verified through SHA-256 checksums.
+The pipeline also corrects recurrent metadata issues such as fragmented compound words, misclassified publication types, invalid page ranges, and all-capitals titles. Deterministic caching ensures that cache-hit executions produce byte-identical outputs, verified through SHA-256 checksums.
 
-To maintain efficiency and stability, author queries run in parallel while respecting per-API rate limits. All configurable parameters, including source trust order, similarity thresholds, rate limits, and venue mappings, are centralized in [`citeforge/config.py`](citeforge/config.py).
+Author queries run in parallel under per-API rate limits. All configurable parameters, including source trust order, similarity thresholds, rate limits, and venue mappings, are centralized in [`citeforge/config.py`](citeforge/config.py).
 
 ## Data Sources
 
-[SerpAPI](https://serpapi.com/) and [Serply](https://serply.io/) require keys. Everything else is free or optional:
+SerpAPI requires a key; the remaining sources are keyless, recommended, or optional.
 
-- **Required:** [SerpAPI](https://serpapi.com/) (Google Scholar), [Serply](https://serply.io/) (citation details);
-- **Recommended:** [Semantic Scholar](https://www.semanticscholar.org/);
+- **Required (key needed):** [SerpAPI](https://serpapi.com/) (Google Scholar);
+- **Recommended (key needed):** [Serply](https://serply.io/) (citation details), [Semantic Scholar](https://www.semanticscholar.org/);
 - **Free (no key):** [Crossref](https://www.crossref.org/), [OpenAlex](https://openalex.org/), [arXiv](https://arxiv.org/), [PubMed](https://pubmed.ncbi.nlm.nih.gov/), [Europe PMC](https://europepmc.org/), [DataCite](https://datacite.org/), [ORCID](https://orcid.org/), [DBLP](https://dblp.org/); and,
 - **Optional (key needed):** [OpenReview](https://openreview.net/), [Google Gemini](https://ai.google.dev/).
 
-Set `CROSSREF_MAILTO` to get into Crossref's polite pool (faster responses).
+Set `CROSSREF_MAILTO` to join Crossref's polite pool (faster responses).
 
 ## Citation
 
-If you use CiteForge in your research or find it useful, please consider citing it:
+If you use CiteForge in your research, cite it with the entry below.
 
 ```bibtex
 @software{spadon_citeforge,
@@ -122,4 +120,4 @@ If you use CiteForge in your research or find it useful, please consider citing
 
 ## License
 
-This project is licensed under the **MIT [License](LICENSE)**. You're free to use, modify, and distribute it for any purpose.
+This project is licensed under the [MIT License](LICENSE).

citeforge/api_configs.py

@@ -1,9 +1,10 @@
 """Per-source API search configurations.
 
-Holds the `APISearchConfig` and `APIFieldMapping` instances for each scholarly
-source (Semantic Scholar, Crossref, OpenAlex, PubMed, Europe PMC, arXiv,
-OpenReview, DataCite) that `api_generics.py` consumes to run searches and build
-BibTeX entries.
+Holds the `APISearchConfig` and `APIFieldMapping` instances that
+`api_generics.py` consumes to run searches and build BibTeX entries. Sources
+whose protocol or output cannot be expressed here (PubMed's two-step lookup,
+Europe PMC and DataCite entry construction) stay hand-rolled in the client
+modules instead.
 """
 
 from __future__ import annotations
@@ -12,8 +13,7 @@
 from typing import Any
 
 from .api_generics import APIFieldMapping, APISearchConfig
-from .config import CROSSREF_BASE, EUROPEPMC_BASE, OPENALEX_BASE, PUBMED_BASE, S2_BASE
-from .text_utils import extract_year_from_any
+from .config import CROSSREF_BASE, EUROPEPMC_BASE, OPENALEX_BASE, S2_BASE
 
 
 def _year_from_date_parts(source: dict[str, Any]) -> int | None:
@@ -38,12 +38,6 @@ def _extract_crossref_year(item: dict[str, Any]) -> int:
     return 0
 
 
-def _extract_europepmc_year(article: dict[str, Any]) -> int:
-    """Extract year from Europe PMC pubYear field."""
-    year_str = article.get("pubYear") or ""
-    return int(year_str) if year_str.isdigit() else 0
-
-
 S2_SEARCH_CONFIG = APISearchConfig(
     api_name="semantic_scholar",
     base_url=f"{S2_BASE}/paper/search",
@@ -91,16 +85,6 @@ def _extract_europepmc_year(article: dict[str, Any]) -> int:
     ],
 )
 
-PUBMED_SEARCH_CONFIG = APISearchConfig(
-    api_name="pubmed",
-    base_url=f"{PUBMED_BASE}/esearch.fcgi",
-    query_param_name="term",
-    result_path=["esearchresult", "idlist"],  # Returns PMIDs, need second request
-    title_field="title",
-    author_field="authors",
-    additional_params={"db": "pubmed", "retmax": 10, "retmode": "json"},
-)
-
 EUROPEPMC_SEARCH_CONFIG = APISearchConfig(
     api_name="europepmc",
     base_url=f"{EUROPEPMC_BASE}/search",
@@ -171,40 +155,6 @@ def _extract_europepmc_year(article: dict[str, Any]) -> int:
     custom_year_extractor=lambda work: work.get("publication_year") or 0,
 )
 
-PUBMED_FIELD_MAPPING = APIFieldMapping(
-    api_name="pubmed",
-    title_fields=["title"],
-    author_fields=["authors"],
-    year_fields=["pubdate"],
-    venue_fields=["fulljournalname", "source"],
-    doi_fields=["articleids"],  # Special handling needed
-    url_fields=["uid", "pmid"],  # Will build URL from PMID
-    author_name_key="name",
-    venue_hints={"fulljournalname": "article", "source": "article"},
-    extra_field_mappings={"volume": "volume", "issue": "number", "pages": "pages"},
-    custom_author_extractor=lambda article: [
-        author.get("name", "").strip() for author in article.get("authors") or [] if author.get("name", "").strip()
-    ],
-    custom_year_extractor=lambda article: extract_year_from_any(article.get("pubdate"), fallback=0) or 0,
-)
-
-EUROPEPMC_FIELD_MAPPING = APIFieldMapping(
-    api_name="europepmc",
-    title_fields=["title"],
-    author_fields=["authorString"],
-    year_fields=["pubYear"],
-    venue_fields=["journalTitle", "bookTitle"],
-    doi_fields=["doi"],
-    url_fields=["pmcid", "pmid"],  # Will build URL from PMCID/PMID
-    entry_type_field="pubType",
-    venue_hints={"journalTitle": "article", "bookTitle": "inproceedings"},
-    extra_field_mappings={"journalVolume": "volume", "issue": "number", "pageInfo": "pages"},
-    custom_author_extractor=lambda article: [
-        name.strip() for name in (article.get("authorString") or "").split(",") if name.strip()
-    ],
-    custom_year_extractor=_extract_europepmc_year,
-)
-
 ARXIV_FIELD_MAPPING = APIFieldMapping(
     api_name="arxiv",
     title_fields=["title"],
@@ -236,18 +186,3 @@ def _extract_europepmc_year(article: dict[str, Any]) -> int:
         if str(a).strip()
     ],
 )
-
-DATACITE_FIELD_MAPPING = APIFieldMapping(
-    api_name="datacite",
-    title_fields=["attributes.titles"],
-    author_fields=["attributes.creators"],
-    year_fields=["attributes.publicationYear"],
-    venue_fields=["attributes.publisher"],
-    doi_fields=["attributes.doi"],
-    url_fields=["attributes.url"],
-    custom_author_extractor=lambda record: [
-        creator.get("name", "").strip()
-        for creator in (record.get("attributes") or {}).get("creators") or []
-        if creator.get("name", "").strip()
-    ],
-)