diff --git a/CLAUDE.md b/CLAUDE.md index 9136d2cc..6bc9d5d9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,14 +7,13 @@ Project overview and data sources: @README.md ```bash pip install -e . # Install (editable) pip install -e .[dev] # Install with dev tools -python3 main.py # Run pipeline (default: data/input.csv) -python3 main.py data/custom.csv # Custom input +python3 main.py # Run pipeline (input: data/input.csv) python3 main.py --force # Force re-enrichment (ignore cache completeness) ``` ## Quality Gates -All three must pass before merge: +All three must pass before merge. ```bash ruff check citeforge/ tests/ main.py # Lint @@ -22,9 +21,9 @@ mypy citeforge/ main.py # Type check (strict, ignore_missing_ pytest tests/ -v --tb=short # Tests (full suite, Python 3.10-3.13) ``` -Single test: `pytest tests/test_merge.py::test_function_name -v --tb=short` +Run a single test with `pytest tests/test_core.py::test_function_name -v --tb=short`. -Ruff config: line-length 120, rules E/F/W/I/N/UP/B/C4/SIM/RUF/S (see pyproject.toml for ignores). +Ruff uses line-length 120 and rules E/F/W/I/N/UP/B/C4/SIM/RUF/S (see pyproject.toml for ignores). ## Architecture @@ -34,15 +33,15 @@ Trust hierarchy in `citeforge/merge_utils.py:merge_with_policy()` merges fields ## Three-Way Fix Pattern (CRITICAL) -**Fixes for entry types, titles, and booktitles MUST be applied in three places or oscillation occurs between consecutive runs:** +Fixes for entry types, titles, and booktitles MUST be applied in three places, or output oscillates between consecutive runs. -1. `_fixup_bib_entry()` — initial fixup on load -2. Existing-file fixup in `process_article()` — before enrichment -3. Phase 4 post-merge — after trust-based merge +1. `_fixup_bib_entry()` (initial fixup on load) +2. Existing-file fixup in `process_article()` (before enrichment) +3. Phase 4 post-merge (after trust-based merge) Consolidated helpers `_fix_title_text()` and `_apply_booktitle_fixups()` are called from all three. When adding a new text or type fix, grep for these functions and add the fix to all call sites. -The following fixes are also applied in all 3 locations: +The following fixes are also applied in all three locations. - Abbreviated venue expansion (`ABBREVIATED_VENUE_MAP`) - Venue case correction (`VENUE_CASE_CORRECTIONS`) - Publisher-duplicate-container stripping (publisher == journal/booktitle → remove publisher) @@ -61,13 +60,13 @@ The following fixes are also applied in all 3 locations: - **Book-chapter DOI**: `.ch\d+` in DOI → Wiley book chapter → reclassify @article → @incollection. - **Content comparison guard**: Post-run fixup compares serialized output to existing file before writing, preventing phantom writes from serializer normalization. - **Cache defensive copy**: `ResponseCache.get()` returns `dict(...)` copy to prevent mutation. -- **Orphan safety**: Never blindly delete orphan .bib files — verify as duplicates via `title_similarity >= 0.95`. +- **Orphan safety**: Never blindly delete orphan .bib files; verify as duplicates via `title_similarity >= 0.95`. - **CSV paths**: Relative to CWD (must run from project root). -- **Fused compounds**: Never use `---` (em-dash) or accented characters in `FUSED_COMPOUND_WORDS` or `ABBREVIATED_VENUE_MAP` — serializer strips them. +- **Fused compounds**: Never use `---` (em-dash) or accented characters in `FUSED_COMPOUND_WORDS` or `ABBREVIATED_VENUE_MAP` (the serializer strips them). ## Testing Patterns -- Tests in `tests/` mirror `citeforge/` modules (e.g., `test_merge.py` tests `merge_utils.py`) +- Tests in `tests/` mostly mirror `citeforge/` modules (e.g., `test_canonicalize.py` tests `canonicalize.py`); merge coverage lives in `test_deduplication.py`, `test_save_entry.py`, and `test_core.py` - `tests/conftest.py` + `tests/fixtures.py` provide shared fixtures - Integration tests requiring API keys auto-skip when keys unavailable - Use `monkeypatch` for HTTP mocking; never make real API calls in unit tests diff --git a/README.md b/README.md index 8bfc766b..50279e22 100644 --- a/README.md +++ b/README.md @@ -12,63 +12,61 @@

Automated academic citation enrichment from multiple scholarly APIs.
- CiteForge fetches, validates, deduplicates, and merges bibliographic
- metadata so you don't have to. Given a list of authors, it
- produces clean BibTeX files ready for LaTeX. + Given a list of authors, CiteForge fetches, validates, deduplicates,
+ and merges bibliographic metadata into per-author BibTeX files.

--- -## Why CiteForge? +## Motivation -If you've ever tried to build a comprehensive publication list for a research group, you know the pain. Google Scholar entries are often incomplete, missing DOIs, and have inconsistent formatting and broken author names. Manually cross-referencing Crossref, Semantic Scholar, arXiv, PubMed, and a handful of other databases gets tedious fast, especially when you're dealing with dozens of authors and hundreds of papers. +Google Scholar profiles index an author's publications, but the entries are often incomplete, with missing DOIs, inconsistent venue names, and malformed author lists. Correcting them requires cross-referencing registries such as Crossref, Semantic Scholar, arXiv, and PubMed, which is impractical to do by hand at the scale of a research group. CiteForge automates this cross-referencing and consolidation. -CiteForge takes care of this. Point it at a list of authors with their Google Scholar profiles, and it will: +Given an input CSV of authors with Google Scholar profiles, CiteForge -- Pull every publication from Scholar via [SerpAPI](https://serpapi.com/) and enrich citation details with [Serply](https://serply.io/); -- Query **multiple academic APIs** for richer metadata on each paper; -- Deduplicate results using fuzzy title matching, DOI normalization, and author overlap; -- Merge fields using a **multi-level trust hierarchy** that prefers authoritative sources; and, -- Output clean, LaTeX-ready `.bib` files organized by author. +- pulls each author's publications from Scholar via [SerpAPI](https://serpapi.com/) and citation details via [Serply](https://serply.io/); +- queries additional academic APIs for metadata on each paper; +- deduplicates results using fuzzy title matching, DOI normalization, and author overlap; +- merges fields using a multi-level trust hierarchy that prefers authoritative sources; and, +- writes `.bib` files organized by author. -The result is deterministic. On cache-hit runs, CiteForge produces **byte-identical output** across consecutive runs, verified by SHA-256 checksums. +Output is deterministic on cache-hit runs. ## Getting Started -You'll need **Python 3.10+** and a [SerpAPI](https://serpapi.com/) key (the free tier is sufficient for small runs). +Requires **Python 3.10+** and a [SerpAPI](https://serpapi.com/) key (the free tier is sufficient for small runs). ```bash git clone https://github.com/gabrielspadon/CiteForge.git && cd CiteForge pip install -e . ``` -Drop your API keys into the `keys/` directory: +Place API keys in the `keys/` directory. ```bash mkdir -p keys echo "your_serpapi_key" > keys/SerpAPI.key # Required -echo "your_serply_key" > keys/Serply.key # Required +echo "your_serply_key" > keys/Serply.key # Recommended (citation detail skipped without it) echo "your_semantic_key" > keys/Semantic.key # Recommended echo "your_gemini_key" > keys/Gemini.key # Optional printf "user\npass" > keys/OpenReview.key # Optional ``` -Then create an input CSV and run: +Then create the input CSV and run the pipeline. ```bash -python3 main.py # Default: data/input.csv -python3 main.py data/custom.csv # Custom input -python3 main.py --force # Force re-enrichment +python3 main.py # Input: data/input.csv +python3 main.py --force # Force re-enrichment (ignore cache completeness) ``` -The input CSV has three columns (name, Scholar link, and an optional DBLP link): +The input CSV has three columns (name, Scholar link, and an optional DBLP link). ```csv Name,Scholar Link,DBLP Link Gabriel Spadon,https://scholar.google.com/citations?user=bfdGsGUAAAAJ,https://dblp.org/pid/192/1659 ``` -Output is organized per author, with a shared summary and deduplication log: +Output is organized per author, with a shared summary and run log. ``` output/ @@ -85,28 +83,28 @@ API responses are cached under `data/api_cache/` with monthly expiry, so subsequ ## How It Works -CiteForge begins by retrieving the complete list of publications from Google Scholar through SerpAPI. Each entry is then enriched by querying multiple scholarly services, including Semantic Scholar, Crossref, arXiv, OpenAlex, and PubMed, to gather complementary metadata. +CiteForge retrieves each author's publication list from Google Scholar through SerpAPI, then enriches every entry by querying scholarly services including Semantic Scholar, Crossref, arXiv, OpenAlex, and PubMed. -A trust-based consolidation stage merges the collected records according to source reliability, consistently prioritizing authoritative registries over scraped content. Duplicate detection combines DOI normalization, external identifier matching, and fuzzy title similarity to identify overlapping entries across sources. +A trust-based consolidation stage merges the collected records according to source reliability, prioritizing authoritative registries over scraped content. Duplicate detection combines DOI normalization, external identifier matching, and fuzzy title similarity to identify overlapping entries across sources. -The pipeline further corrects recurrent metadata issues, such as fragmented compound words, misclassified publication types, invalid page ranges, and titles written entirely in capital letters. Deterministic caching ensures that cache-hit executions produce byte-identical outputs, verified through SHA-256 checksums. +The pipeline also corrects recurrent metadata issues such as fragmented compound words, misclassified publication types, invalid page ranges, and all-capitals titles. Deterministic caching ensures that cache-hit executions produce byte-identical outputs, verified through SHA-256 checksums. -To maintain efficiency and stability, author queries run in parallel while respecting per-API rate limits. All configurable parameters, including source trust order, similarity thresholds, rate limits, and venue mappings, are centralized in [`citeforge/config.py`](citeforge/config.py). +Author queries run in parallel under per-API rate limits. All configurable parameters, including source trust order, similarity thresholds, rate limits, and venue mappings, are centralized in [`citeforge/config.py`](citeforge/config.py). ## Data Sources -[SerpAPI](https://serpapi.com/) and [Serply](https://serply.io/) require keys. Everything else is free or optional: +SerpAPI requires a key; the remaining sources are keyless, recommended, or optional. -- **Required:** [SerpAPI](https://serpapi.com/) (Google Scholar), [Serply](https://serply.io/) (citation details); -- **Recommended:** [Semantic Scholar](https://www.semanticscholar.org/); +- **Required (key needed):** [SerpAPI](https://serpapi.com/) (Google Scholar); +- **Recommended (key needed):** [Serply](https://serply.io/) (citation details), [Semantic Scholar](https://www.semanticscholar.org/); - **Free (no key):** [Crossref](https://www.crossref.org/), [OpenAlex](https://openalex.org/), [arXiv](https://arxiv.org/), [PubMed](https://pubmed.ncbi.nlm.nih.gov/), [Europe PMC](https://europepmc.org/), [DataCite](https://datacite.org/), [ORCID](https://orcid.org/), [DBLP](https://dblp.org/); and, - **Optional (key needed):** [OpenReview](https://openreview.net/), [Google Gemini](https://ai.google.dev/). -Set `CROSSREF_MAILTO` to get into Crossref's polite pool (faster responses). +Set `CROSSREF_MAILTO` to join Crossref's polite pool (faster responses). ## Citation -If you use CiteForge in your research or find it useful, please consider citing it: +If you use CiteForge in your research, cite it with the entry below. ```bibtex @software{spadon_citeforge, @@ -122,4 +120,4 @@ If you use CiteForge in your research or find it useful, please consider citing ## License -This project is licensed under the **MIT [License](LICENSE)**. You're free to use, modify, and distribute it for any purpose. +This project is licensed under the [MIT License](LICENSE). diff --git a/citeforge/api_configs.py b/citeforge/api_configs.py index 99b42a0e..c25af490 100644 --- a/citeforge/api_configs.py +++ b/citeforge/api_configs.py @@ -1,9 +1,10 @@ """Per-source API search configurations. -Holds the `APISearchConfig` and `APIFieldMapping` instances for each scholarly -source (Semantic Scholar, Crossref, OpenAlex, PubMed, Europe PMC, arXiv, -OpenReview, DataCite) that `api_generics.py` consumes to run searches and build -BibTeX entries. +Holds the `APISearchConfig` and `APIFieldMapping` instances that +`api_generics.py` consumes to run searches and build BibTeX entries. Sources +whose protocol or output cannot be expressed here (PubMed's two-step lookup, +Europe PMC and DataCite entry construction) stay hand-rolled in the client +modules instead. """ from __future__ import annotations @@ -12,8 +13,7 @@ from typing import Any from .api_generics import APIFieldMapping, APISearchConfig -from .config import CROSSREF_BASE, EUROPEPMC_BASE, OPENALEX_BASE, PUBMED_BASE, S2_BASE -from .text_utils import extract_year_from_any +from .config import CROSSREF_BASE, EUROPEPMC_BASE, OPENALEX_BASE, S2_BASE def _year_from_date_parts(source: dict[str, Any]) -> int | None: @@ -38,12 +38,6 @@ def _extract_crossref_year(item: dict[str, Any]) -> int: return 0 -def _extract_europepmc_year(article: dict[str, Any]) -> int: - """Extract year from Europe PMC pubYear field.""" - year_str = article.get("pubYear") or "" - return int(year_str) if year_str.isdigit() else 0 - - S2_SEARCH_CONFIG = APISearchConfig( api_name="semantic_scholar", base_url=f"{S2_BASE}/paper/search", @@ -91,16 +85,6 @@ def _extract_europepmc_year(article: dict[str, Any]) -> int: ], ) -PUBMED_SEARCH_CONFIG = APISearchConfig( - api_name="pubmed", - base_url=f"{PUBMED_BASE}/esearch.fcgi", - query_param_name="term", - result_path=["esearchresult", "idlist"], # Returns PMIDs, need second request - title_field="title", - author_field="authors", - additional_params={"db": "pubmed", "retmax": 10, "retmode": "json"}, -) - EUROPEPMC_SEARCH_CONFIG = APISearchConfig( api_name="europepmc", base_url=f"{EUROPEPMC_BASE}/search", @@ -171,40 +155,6 @@ def _extract_europepmc_year(article: dict[str, Any]) -> int: custom_year_extractor=lambda work: work.get("publication_year") or 0, ) -PUBMED_FIELD_MAPPING = APIFieldMapping( - api_name="pubmed", - title_fields=["title"], - author_fields=["authors"], - year_fields=["pubdate"], - venue_fields=["fulljournalname", "source"], - doi_fields=["articleids"], # Special handling needed - url_fields=["uid", "pmid"], # Will build URL from PMID - author_name_key="name", - venue_hints={"fulljournalname": "article", "source": "article"}, - extra_field_mappings={"volume": "volume", "issue": "number", "pages": "pages"}, - custom_author_extractor=lambda article: [ - author.get("name", "").strip() for author in article.get("authors") or [] if author.get("name", "").strip() - ], - custom_year_extractor=lambda article: extract_year_from_any(article.get("pubdate"), fallback=0) or 0, -) - -EUROPEPMC_FIELD_MAPPING = APIFieldMapping( - api_name="europepmc", - title_fields=["title"], - author_fields=["authorString"], - year_fields=["pubYear"], - venue_fields=["journalTitle", "bookTitle"], - doi_fields=["doi"], - url_fields=["pmcid", "pmid"], # Will build URL from PMCID/PMID - entry_type_field="pubType", - venue_hints={"journalTitle": "article", "bookTitle": "inproceedings"}, - extra_field_mappings={"journalVolume": "volume", "issue": "number", "pageInfo": "pages"}, - custom_author_extractor=lambda article: [ - name.strip() for name in (article.get("authorString") or "").split(",") if name.strip() - ], - custom_year_extractor=_extract_europepmc_year, -) - ARXIV_FIELD_MAPPING = APIFieldMapping( api_name="arxiv", title_fields=["title"], @@ -236,18 +186,3 @@ def _extract_europepmc_year(article: dict[str, Any]) -> int: if str(a).strip() ], ) - -DATACITE_FIELD_MAPPING = APIFieldMapping( - api_name="datacite", - title_fields=["attributes.titles"], - author_fields=["attributes.creators"], - year_fields=["attributes.publicationYear"], - venue_fields=["attributes.publisher"], - doi_fields=["attributes.doi"], - url_fields=["attributes.url"], - custom_author_extractor=lambda record: [ - creator.get("name", "").strip() - for creator in (record.get("attributes") or {}).get("creators") or [] - if creator.get("name", "").strip() - ], -) diff --git a/citeforge/api_generics.py b/citeforge/api_generics.py index d4930612..ea899746 100644 --- a/citeforge/api_generics.py +++ b/citeforge/api_generics.py @@ -13,6 +13,7 @@ from typing import Any from .cache import response_cache +from .clients.helpers import title_author_cache_key from .config import ( CACHE_TTL_SEARCH_DAYS, GENERIC_SERIES_NAMES, @@ -34,6 +35,7 @@ safe_get_field, safe_get_nested, ) +from .venue import first_non_generic_container def _resolve_dotted(obj: dict[str, Any], field: str) -> Any: @@ -75,10 +77,7 @@ def _resolve_dotted_str(obj: dict[str, Any], field: str, *, check_placeholder: b @dataclass class APISearchConfig: - """ - Configuration for API-specific search behavior including endpoint details, - query parameters, and custom field extractors. - """ + """Per-API search settings (endpoint, query parameters, response paths, extractors).""" api_name: str base_url: str @@ -105,10 +104,7 @@ class APISearchConfig: @dataclass class APIFieldMapping: - """ - Configuration for API-specific field mappings when building BibTeX entries, - translating diverse field names and structures to a unified BibTeX format. - """ + """Per-API field mappings used when building BibTeX entries from a response.""" api_name: str @@ -179,26 +175,19 @@ def _build_scoring_function( ) -def search_api_generic( - title: str, author_name: str | None, config: APISearchConfig, api_key: str | None = None -) -> dict[str, Any] | None: - """ - Search for academic publications across different API providers using a unified - interface with a two-pass matching strategy that attempts exact title matches first - and falls back to fuzzy matching when needed. +def _fetch_results( + title: str, + author_name: str | None, + config: APISearchConfig, + api_key: str | None, + cache_key: str, +) -> list[Any] | None: + """Issue the configured search request and extract the raw result list. + + Returns ``None`` both on HTTP failure (not cached, so transient errors are + retried on the next run) and on an empty result set (recorded as a negative + cache entry under *cache_key* before returning). """ - if not title: - return None - - cache_key = f"{normalize_title(title)}|{(author_name or '').strip().lower()}" - cached = response_cache.get(config.api_name, cache_key) - if cached is not None: - if cached.get("_negative"): - logger.debug(f"{config.api_name} | NEG_HIT | key={cache_key[:60]}", category=LogCategory.CACHE) - return None - logger.debug(f"{config.api_name} | HIT | key={cache_key[:60]}", category=LogCategory.CACHE) - return cached if cached else None - params = {config.query_param_name: title, **config.additional_params} if author_name and config.author_param_name: params[config.author_param_name] = author_name @@ -214,10 +203,37 @@ def search_api_generic( except ALL_API_ERRORS: return None - results = safe_get_nested(data, *config.result_path, default=[]) + results: list[Any] = safe_get_nested(data, *config.result_path, default=[]) if not results: response_cache.put_negative(config.api_name, cache_key) return None + return results + + +def search_api_generic( + title: str, author_name: str | None, config: APISearchConfig, api_key: str | None = None +) -> dict[str, Any] | None: + """Search one configured API for the best-matching publication. + + Two-pass matching. An exact normalized-title match (with author check) wins + outright; otherwise the highest-scoring fuzzy candidate above the pick + threshold is used. + """ + if not title: + return None + + cache_key = title_author_cache_key(title, author_name) + cached = response_cache.get(config.api_name, cache_key) + if cached is not None: + if cached.get("_negative"): + logger.debug(f"{config.api_name} | NEG_HIT | key={cache_key[:60]}", category=LogCategory.CACHE) + return None + logger.debug(f"{config.api_name} | HIT | key={cache_key[:60]}", category=LogCategory.CACHE) + return cached if cached else None + + results = _fetch_results(title, author_name, config, api_key, cache_key) + if results is None: + return None logger.debug( f"{config.api_name} | RESULTS | count={len(results)} | path={config.result_path}", @@ -281,16 +297,15 @@ def search_api_generic_multiple( max_results: int = 5, year_hint: int | None = None, ) -> list[dict[str, Any]]: - """ - Search for academic publications and return multiple candidates sorted by relevance. + """Search one configured API and return the top candidates sorted by score. - Similar to search_api_generic but returns a list of top candidates instead of just - the best match, enabling multiple candidates for validation. + Like ``search_api_generic`` but keeps every candidate above the tolerance + threshold (up to *max_results*) so callers can validate multiple options. """ if not title: return [] - cache_key = f"multi|{normalize_title(title)}|{(author_name or '').strip().lower()}" + cache_key = title_author_cache_key(title, author_name, prefix="multi|") cached = response_cache.get(config.api_name, cache_key) if cached is not None: if cached.get("_negative"): @@ -300,24 +315,8 @@ def search_api_generic_multiple( logger.debug(f"{config.api_name}_multi | HIT | key={cache_key[:60]}", category=LogCategory.CACHE) return cached_list - params = {config.query_param_name: title, **config.additional_params} - if author_name and config.author_param_name: - params[config.author_param_name] = author_name - - url = build_url(config.base_url, params) - logger.debug(f"{config.api_name} | HTTP_REQUEST | url={url[:80]}", category=LogCategory.SCORE) - - try: - if api_key and config.api_name == "semantic_scholar": - data = s2_http_get_json(url, api_key, timeout=config.timeout) - else: - data = http_get_json(url, timeout=config.timeout) - except ALL_API_ERRORS: - return [] - - results = safe_get_nested(data, *config.result_path, default=[]) - if not results: - response_cache.put_negative(config.api_name, cache_key) + results = _fetch_results(title, author_name, config, api_key, cache_key) + if results is None: return [] score_fn = _build_scoring_function(title, author_name, config, year_hint) @@ -393,14 +392,7 @@ def _extract_venue( # Crossref returns container-title as array: [series_name, conference_name] # Prefer the non-generic element over generic series names like LNCS if isinstance(raw_venue, list) and len(raw_venue) > 1: - non_generic = next( - ( - str(c).strip() - for c in raw_venue - if str(c).strip() and str(c).strip().lower() not in GENERIC_SERIES_NAMES - ), - None, - ) + non_generic = first_non_generic_container(raw_venue) venue = non_generic or _resolve_dotted_str(response, field_name) logger.debug( f"{mapping.api_name} | VENUE_ARRAY | elements={len(raw_venue)}" @@ -428,10 +420,7 @@ def _extract_venue( def build_bibtex_from_response(response: dict[str, Any], keyhint: str, mapping: APIFieldMapping) -> str | None: - """ - Build a BibTeX entry from an API response using configured field mappings to handle - diverse field naming conventions and data structures across different academic APIs. - """ + """Build a BibTeX entry from an API response using the configured field mappings.""" from .bibtex_build import build_bibtex_entry, determine_entry_type title = _first_resolved_str(response, mapping.title_fields, check_placeholder=True) diff --git a/citeforge/bibtex_build.py b/citeforge/bibtex_build.py index 4c3a4361..1ca00a9d 100644 --- a/citeforge/bibtex_build.py +++ b/citeforge/bibtex_build.py @@ -12,13 +12,19 @@ from collections.abc import Callable from typing import Any +from .bibtex_utils import bibtex_from_dict, make_bibkey +from .clients.helpers import _score_candidate_generic from .config import ( ABBREVIATED_VENUE_MAP, CONFERENCE_KEYWORDS, KNOWN_CONFERENCE_VENUES, SIM_TITLE_SIM_MIN, ) +from .id_utils import _norm_arxiv_id from .log_utils import LogCategory, logger +from .text_utils import author_name_matches, title_similarity + +_NON_WORD_RE = re.compile(r"\W+") _ARTICLE_TYPES = {"journal-article", "journal_article", "article"} _CONFERENCE_TYPES = {"proceedings-article", "paper-conference", "inproceedings", "conference"} @@ -38,11 +44,9 @@ def get_container_field(entry_type: str) -> str: - """ - Choose the BibTeX field that should store the venue for this entry type, - such as journal for articles, booktitle for conference papers and book - chapters, or howpublished for miscellaneous entries. - """ + """Choose the venue field for an entry type. journal for articles, + booktitle for conference papers and book chapters, howpublished + otherwise.""" if entry_type == "article": return "journal" if entry_type in ("inproceedings", "incollection"): @@ -51,10 +55,8 @@ def get_container_field(entry_type: str) -> str: def format_author_field(authors: list[str]) -> str | None: - """ - Combine a list of author names into the BibTeX author format using " and " - between names, or return None when the list is empty. - """ + """Join author names with " and " per BibTeX convention, or None when the + list is empty.""" return " and ".join(authors) if authors else None @@ -70,14 +72,9 @@ def build_bibtex_entry( arxiv_id: str | None = None, extra_fields: dict[str, str] | None = None, ) -> str: - """ - Build a complete BibTeX entry from the main publication details and optional - identifiers, skipping fields that are missing or empty. - """ - from .bibtex_utils import bibtex_from_dict, make_bibkey - from .id_utils import _norm_arxiv_id - - key = make_bibkey(title, authors, year, fallback=re.sub(r"\W+", "", keyhint) or "entry") + """Build a complete BibTeX entry from publication details and optional + identifiers, skipping missing or empty fields.""" + key = make_bibkey(title, authors, year, fallback=_NON_WORD_RE.sub("", keyhint) or "entry") container_field = get_container_field(entry_type) logger.debug( f"BUILD_ENTRY | type={entry_type} | key={key} | title={title[:60]}" @@ -115,20 +112,13 @@ def create_scoring_function( year_getter: Callable[[Any], int | None] | None = None, author_match_fn: Callable[[str, Any], bool] | None = None, ) -> Callable[[Any], float]: - """ - Create a scoring function that ranks search results against a target title, - author, and year using the supplied accessors and matching logic. - """ - from .clients.helpers import _score_candidate_generic - from .text_utils import author_name_matches, title_similarity - + """Create a scoring function that ranks search results against a target + title, author, and year using the supplied accessors and matching logic.""" author_match_fn = author_match_fn or author_name_matches def score_fn(candidate: Any) -> float: - """ - Compare a single candidate against the target description and return a - score that reflects how well title, author, and year agree. - """ + """Score how well a candidate's title, author, and year agree with + the target.""" cand_title = title_getter(candidate) tsim = title_similarity(title, cand_title) @@ -156,10 +146,8 @@ def score_fn(candidate: Any) -> float: def _classify_type_string(typ: str) -> str | None: - """ - Map a publication type string to a BibTeX entry type, returning None if - no match is found. - """ + """Map a publication type string to a BibTeX entry type, or None when no + match is found.""" if "journal" in typ or typ in _ARTICLE_TYPES: return "article" if "proceed" in typ or typ in _CONFERENCE_TYPES: @@ -188,11 +176,9 @@ def determine_entry_type( publication_types_field: str | None = None, venue_hints: dict[str, str] | None = None, ) -> str: - """ - Guess whether a publication should be treated as a journal article, - conference paper, book chapter, or miscellaneous entry by inspecting type - fields and venue hints. - """ + """Classify a publication as a journal article, conference paper, book + chapter, book, or miscellaneous entry by inspecting type fields and venue + hints.""" if obj is None: return "misc" diff --git a/citeforge/bibtex_utils.py b/citeforge/bibtex_utils.py index ae632b61..fc5b30d3 100644 --- a/citeforge/bibtex_utils.py +++ b/citeforge/bibtex_utils.py @@ -19,7 +19,6 @@ BIBTEX_KEY_MAX_WORDS, CACHE_TTL_GEMINI_DAYS, PREPRINT_DOI_PREFIXES, - PREPRINT_SERVERS, SIM_DEDUP_COMPOSITE_THRESHOLD, SIM_DEDUP_MULTI_SIGNAL_MIN, SIM_FILE_DUPLICATE_THRESHOLD, @@ -28,6 +27,7 @@ from .id_utils import _norm_doi, external_ids_match, extract_arxiv_eprint from .log_utils import LogCategory, logger from .text_utils import ( + _is_preprint_fields, author_overlap_ratio, authors_overlap, compute_dedup_score, @@ -61,29 +61,35 @@ } ) +# Compiled once at import; the parse/serialize/key helpers below run per entry. +_NON_ALNUM_RE = re.compile(r"[^A-Za-z0-9]") +_NON_WORD_RE = re.compile(r"\W+") +_ENTRY_HEAD_RE = re.compile(r"@\s*([a-zA-Z]+)\s*\{\s*([^,\s]+)\s*,") +_SINGLE_LINE_ENTRY_RE = re.compile(r"@\s*[a-zA-Z]+\s*\{\s*[^,\s]+\s*,\s*(.+)\s*\}\s*$", re.DOTALL) +_FIELD_ASSIGN_RE = re.compile(r"^\s*([a-zA-Z][a-zA-Z0-9_\-]*)\s*=\s*(.*)$") +_QUOTED_VALUE_RE = re.compile(r'^"([^"]*)"') +_FILENAME_SANITIZE_RE = re.compile(r"[^A-Za-z0-9_\-]+") +_TITLE_WORD_SPLIT_RE = re.compile(r"[^A-Za-z0-9]+") +_CONTROL_CHARS_RE = re.compile(r"[\n\r\t]") + def make_bibkey(title: str, authors: list[str], year: int, fallback: str = "entry") -> str: - """ - Build a compact BibTeX citation key using the first author's surname, the - publication year, and the first word of the title, falling back to a generic - label when needed. - """ - last = re.sub(r"[^A-Za-z0-9]", "", authors[0].split()[-1]) if authors and authors[0] else "" + """Build a compact citation key from the first author's surname, the year, + and the first title word, falling back to a generic label.""" + last = _NON_ALNUM_RE.sub("", authors[0].split()[-1]) if authors and authors[0] else "" title_words = title.split() - word = re.sub(r"[^A-Za-z0-9]", "", title_words[0]) if title_words else "" + word = _NON_ALNUM_RE.sub("", title_words[0]) if title_words else "" y = str(year) if year else "" parts = [p for p in [last, y, word] if p] base = "".join(parts) if parts else fallback - base = re.sub(r"\W+", "", base) + base = _NON_WORD_RE.sub("", base) return base or fallback def build_minimal_bibtex(title: str, authors: list[str], year: int, keyhint: str) -> str: - """ - Create a simple BibTeX @misc entry from a title, optional authors, and optional - year so that even sparse metadata can be stored consistently. - """ - key = make_bibkey(title, authors, year, fallback=re.sub(r"\W+", "", keyhint) or "entry") + """Create a minimal @misc entry from a title, optional authors, and + optional year.""" + key = make_bibkey(title, authors, year, fallback=_NON_WORD_RE.sub("", keyhint) or "entry") lines = [f"@misc{{{key},", f" title = {{{title}}},"] if authors: lines.append(f" author = {{{' and '.join(authors)}}},") @@ -96,22 +102,17 @@ def build_minimal_bibtex(title: str, authors: list[str], year: int, keyhint: str def _parse_bibtex_head(bibtex: str) -> dict[str, str] | None: - """ - Read the opening line of a BibTeX entry and pull out the entry type and - citation key if they follow the expected @type{key, pattern. - """ - m = re.search(r"@\s*([a-zA-Z]+)\s*\{\s*([^,\s]+)\s*,", bibtex) + """Pull the entry type and citation key from the @type{key, opening of a + BibTeX entry, or None when the pattern is absent.""" + m = _ENTRY_HEAD_RE.search(bibtex) if not m: return None return {"type": m.group(1).strip(), "key": m.group(2).strip()} def _extract_balanced_braces(text: str, start: int) -> str | None: - """ - Extract the text inside a balanced pair of braces starting at the given - position, keeping track of nested braces so inner blocks are preserved - correctly. - """ + """Extract the text inside a balanced brace pair starting at *start*, + preserving nested braces.""" if start >= len(text) or text[start] != "{": return None depth = 0 @@ -132,34 +133,28 @@ def _extract_balanced_braces(text: str, start: int) -> str | None: def _assign_field_value(fields: dict[str, str], field_name: str, full_value: str) -> None: - """ - Helper to assign a parsed value to fields based on whether the value is - brace-wrapped, quoted, or plain text. Keeps logic in one place to avoid - duplication. - """ + """Assign a parsed value to *fields*, handling brace-wrapped, quoted, and + plain text values in one place.""" if full_value.startswith("{"): val = _extract_balanced_braces(full_value, 0) fields[field_name] = val.strip() if val is not None else full_value.strip().strip("{}") elif full_value.startswith('"'): - m2 = re.match(r'^"([^"]*)"', full_value) + m2 = _QUOTED_VALUE_RE.match(full_value) fields[field_name] = m2.group(1).strip() if m2 else full_value.strip() else: fields[field_name] = full_value.strip() def parse_bibtex_to_dict(bibtex: str) -> dict[str, Any] | None: - """ - Turn a BibTeX string into a dictionary that separates the entry type, key, - and field values while handling nested braces and multi-line fields. - Also handles single-line BibTeX entries common in API responses. - """ + """Parse a BibTeX string into {type, key, fields}, handling nested braces, + multi-line fields, and the single-line entries common in API responses.""" head = _parse_bibtex_head(bibtex) if not head: logger.debug(f"header_fail | input={bibtex[:60]}", category=LogCategory.PARSE) return None fields: dict[str, str] = {} - single_line_pattern = re.search(r"@\s*[a-zA-Z]+\s*\{\s*[^,\s]+\s*,\s*(.+)\s*\}\s*$", bibtex, re.DOTALL) + single_line_pattern = _SINGLE_LINE_ENTRY_RE.search(bibtex) if single_line_pattern and "\n" not in bibtex.strip(): fields_text = single_line_pattern.group(1).strip() @@ -187,7 +182,7 @@ def parse_bibtex_to_dict(bibtex: str) -> dict[str, Any] | None: # Now parse each field for part in field_parts: - m = re.match(r"^\s*([a-zA-Z][a-zA-Z0-9_\-]*)\s*=\s*(.*)$", part) + m = _FIELD_ASSIGN_RE.match(part) if m: field_name = m.group(1).lower() field_value = m.group(2).strip() @@ -200,7 +195,7 @@ def parse_bibtex_to_dict(bibtex: str) -> dict[str, Any] | None: accumulator: list[str] = [] for line in bibtex.split("\n"): - m = re.match(r"^\s*([a-zA-Z][a-zA-Z0-9_\-]*)\s*=\s*(.*)$", line) + m = _FIELD_ASSIGN_RE.match(line) if m: if current_field and accumulator: @@ -218,7 +213,7 @@ def parse_bibtex_to_dict(bibtex: str) -> dict[str, Any] | None: current_field = None accumulator = [] elif rest.startswith('"'): - m2 = re.match(r'^"([^"]*)"', rest) + m2 = _QUOTED_VALUE_RE.match(rest) if m2: fields[current_field] = m2.group(1).strip() current_field = None @@ -265,175 +260,186 @@ def parse_bibtex_to_dict(bibtex: str) -> dict[str, Any] | None: ) -def bibtex_from_dict(entry: dict[str, Any]) -> str: - """ - Format a dictionary-based BibTeX entry back into text, listing common - citation fields first and writing remaining fields in a stable order. +# Serializer cleanup tables, compiled once at import. bibtex_from_dict runs +# per entry, so these must not be rebuilt inside the call. +_LATEX_FORMAT_CMD_PATTERNS: tuple[re.Pattern[str], ...] = tuple( + re.compile(r"\\" + cmd + r"\s*\{") + for cmd in ( + "textit", + "textbf", + "emph", + "textsc", + "texttt", + "textrm", + "textsf", + "underline", + "uppercase", + "lowercase", + "mbox", + "hbox", + "text", + ) +) +# For each old-style command, match {\xx content} first, then {\xx{content}}. +_OLD_STYLE_CMD_PATTERNS: tuple[tuple[re.Pattern[str], re.Pattern[str]], ...] = tuple( + ( + re.compile(r"\{\\" + cmd + r"\s+([^}]+)\}"), + re.compile(r"\{\\" + cmd + r"\s*\{([^}]+)\}\}"), + ) + for cmd in ("it", "bf", "em", "sc", "tt", "rm", "sf", "sl") +) +_LATEX_SPECIAL_CHARS = { + r"\&": "&", + r"\%": "%", + r"\$": "$", + r"\#": "#", + r"\_": "_", + r"\{": "{", + r"\}": "}", +} +_TILDE_RE = re.compile(r"(? str: + r"""Remove LaTeX formatting commands while preserving their content. + + Handles \command{...} (textit, textbf, emph, etc.), old-style {\xx ...} + commands (it, bf, em, etc.), escaped special characters + (\&, \%, \$, \#, \_, \{, \}), tildes, and dashes (-- / ---). """ + # Fast path: every transform below requires a backslash (commands and + # escaped specials), a tilde, a "--" run, or a double space. If none are + # present the function is a no-op, so skip the whole command scan. + if "\\" not in val and "~" not in val and "--" not in val and " " not in val: + return val - def _strip_latex_formatting(val: str) -> str: - r""" - Remove LaTeX formatting commands while preserving the content inside. - - Handles \command{...} (textit, textbf, emph, etc.), old-style - {\xx ...} commands (it, bf, em, etc.), escaped special characters - (\&, \%, \$, \#, \_, \{, \}), tildes, and dashes (-- / ---). - """ - # Fast path: every transform below requires a backslash (commands and - # escaped specials), a tilde, a "--" run, or a double space. If none are - # present the function is a no-op, so skip the whole command scan. - if "\\" not in val and "~" not in val and "--" not in val and " " not in val: - return val - - formatting_commands = [ - "textit", - "textbf", - "emph", - "textsc", - "texttt", - "textrm", - "textsf", - "underline", - "uppercase", - "lowercase", - "mbox", - "hbox", - "text", - ] - - prev_val = None - while prev_val != val: - prev_val = val - for cmd in formatting_commands: - # Match \command{...} with balanced braces - pattern = r"\\" + cmd + r"\s*\{" - while True: - match = re.search(pattern, val) - if not match: - break - # Find the matching closing brace - start = match.end() - 1 # Position of opening brace - depth = 0 - end = start - for i in range(start, len(val)): - if val[i] == "{": - depth += 1 - elif val[i] == "}": - depth -= 1 - if depth == 0: - end = i - break - if depth == 0: - # Extract content and replace - content = val[start + 1 : end] - val = val[: match.start()] + content + val[end + 1 :] - else: - # Unbalanced braces, skip this match - break - - old_style_commands = ["it", "bf", "em", "sc", "tt", "rm", "sf", "sl"] - for cmd in old_style_commands: - # Match {\xx content} or {\xx{content}} - pattern = r"\{\\" + cmd + r"\s+([^}]+)\}" - val = re.sub(pattern, r"\1", val) - # Also handle {\xx{content}} - pattern2 = r"\{\\" + cmd + r"\s*\{([^}]+)\}\}" - val = re.sub(pattern2, r"\1", val) - - special_chars = { - r"\&": "&", - r"\%": "%", - r"\$": "$", - r"\#": "#", - r"\_": "_", - r"\{": "{", - r"\}": "}", - } - for latex_char, plain_char in special_chars.items(): - val = val.replace(latex_char, plain_char) - - val = re.sub(r"(? str: + """Normalize Unicode to ASCII for BibTeX compatibility. + + Decodes HTML entities, strips LaTeX formatting, converts accented + characters via unidecode, and replaces curly quotes and dashes. + """ + # html.unescape only changes a string containing an '&' entity. + if "&" in val: + val = html.unescape(val) + val = _strip_latex_formatting(val) - return val + # strip_accents and every _UNICODE_TO_ASCII key are non-ASCII, so both + # are no-ops on an already-ASCII string; skip them in that common case. + if not val.isascii(): + val = strip_accents(val) + for unicode_char, ascii_char in _UNICODE_TO_ASCII.items(): + val = val.replace(unicode_char, ascii_char) - def _normalize_to_ascii(val: str) -> str: - """ - Normalize Unicode characters to ASCII equivalents for BibTeX compatibility. - - Decodes HTML entities, strips LaTeX formatting, converts accented - characters via unidecode, and replaces curly quotes / dashes. - """ - # html.unescape only changes a string containing an '&' entity. - if "&" in val: - val = html.unescape(val) - val = _strip_latex_formatting(val) - - # strip_accents and every replacement key below are non-ASCII, so both - # are no-ops on an already-ASCII string; skip them in that common case. - if not val.isascii(): - val = strip_accents(val) - replacements = { - "\u2019": "'", # Right single quotation mark → apostrophe - "\u2018": "'", # Left single quotation mark → apostrophe - "\u201c": '"', # Left double quotation mark → quote - "\u201d": '"', # Right double quotation mark → quote - "\u2013": "-", # En dash → hyphen - "\u2014": "--", # Em dash → double hyphen - "\u2026": "...", # Horizontal ellipsis → three dots - "\u00a0": " ", # Non-breaking space → regular space - } - for unicode_char, ascii_char in replacements.items(): - val = val.replace(unicode_char, ascii_char) - - # The apostrophe-year fixup requires a literal single quote to match. - if "'" in val: - val = re.sub(r"\s+'(\d{2})\b", r"'\1", val) + # The apostrophe-year fixup requires a literal single quote to match. + if "'" in val: + val = _APOS_YEAR_RE.sub(r"'\1", val) - return val + return val - def _sanitize_title(title_val: str | None) -> str | None: - if title_val is None: - return None - t = title_val.strip() - dup_suffix_removed = False - trailing_period = False - - # Remove duplicated suffix after colon - if ":" in t: - parts = t.split(":") - if len(parts) >= 3: # Has at least 2 colons - # Check if last two parts are the same (after stripping whitespace) - last_part = parts[-1].strip() - second_last_part = parts[-2].strip() - if last_part and last_part == second_last_part and len(last_part) > 15: - # Remove the duplicated last part - t = ":".join(parts[:-1]).strip() - dup_suffix_removed = True - - # trim trailing periods unless it's an ellipsis - if t.endswith("...") or t.endswith("\u2026"): - if dup_suffix_removed: - logger.debug( - "title_sanitize | dup_suffix_removed=True | trailing_period=False", - category=LogCategory.SERIAL, - ) - return t - if t.endswith("."): - trailing_period = True - t = t[:-1].rstrip() - if dup_suffix_removed or trailing_period: +def _sanitize_title(title_val: str | None) -> str | None: + """Drop a duplicated after-colon suffix and trailing periods (ellipses + are preserved) from a title at serialization time.""" + if title_val is None: + return None + t = title_val.strip() + dup_suffix_removed = False + trailing_period = False + + # Remove duplicated suffix after colon + if ":" in t: + parts = t.split(":") + if len(parts) >= 3: # Has at least 2 colons + # Check if last two parts are the same (after stripping whitespace) + last_part = parts[-1].strip() + second_last_part = parts[-2].strip() + if last_part and last_part == second_last_part and len(last_part) > 15: + # Remove the duplicated last part + t = ":".join(parts[:-1]).strip() + dup_suffix_removed = True + + # trim trailing periods unless it's an ellipsis + if t.endswith("...") or t.endswith("\u2026"): + if dup_suffix_removed: logger.debug( - f"title_sanitize | dup_suffix_removed={dup_suffix_removed} | trailing_period={trailing_period}", + "title_sanitize | dup_suffix_removed=True | trailing_period=False", category=LogCategory.SERIAL, ) return t + if t.endswith("."): + trailing_period = True + t = t[:-1].rstrip() + if dup_suffix_removed or trailing_period: + logger.debug( + f"title_sanitize | dup_suffix_removed={dup_suffix_removed} | trailing_period={trailing_period}", + category=LogCategory.SERIAL, + ) + return t + + +def bibtex_from_dict(entry: dict[str, Any]) -> str: + """Format a dict-based BibTeX entry back into text, listing common + citation fields first and remaining fields in a stable sorted order.""" etype = (entry.get("type") or "misc").lower() key = entry.get("key") or "entry" fields: dict[str, str] = entry.get("fields") or {} @@ -458,19 +464,16 @@ def _sanitize_title(title_val: str | None) -> str | None: def _short_title_for_key(title: str, max_words: int = BIBTEX_KEY_MAX_WORDS, gemini_api_key: str | None = None) -> str: - """ - Pick a few informative words from a title, skipping common stop words, and - join them into a compact phrase that works well in keys or filenames. - - If a Gemini API key is provided, this function will: - 1. Check the ResponseCache for a previously generated short title (only for default max_words) - 2. If not found, use the Gemini API to generate a short title - 3. Fall back to the original algorithm if Gemini fails or no API key is provided - 4. Save successful Gemini responses to the cache for future use - - Cache is only used when max_words equals the default (BIBTEX_KEY_MAX_WORDS). - When max_words is greater than default, we're disambiguating filename collisions, - so we bypass the cache and use the algorithmic approach to get more title words. + """Pick a few informative title words, skipping stop words, and join them + into a compact phrase for keys and filenames. + + With a Gemini API key, checks the ResponseCache first, calls the Gemini + API on a miss, caches successful responses, and falls back to the + algorithmic path on failure. + + The cache is used only when max_words equals BIBTEX_KEY_MAX_WORDS. A + larger max_words means a filename-collision disambiguation pass, which + bypasses the cache to pull more title words algorithmically. """ normalized_title = normalize_title(title) use_cache = max_words == BIBTEX_KEY_MAX_WORDS @@ -478,7 +481,9 @@ def _short_title_for_key(title: str, max_words: int = BIBTEX_KEY_MAX_WORDS, gemi if gemini_api_key and use_cache: cached = response_cache.get("gemini", normalized_title) if cached is not None: - saved_short = re.sub(r"[\n\r\t]", "", cached.get("short_title", "")) if not cached.get("_negative") else "" + saved_short = ( + _CONTROL_CHARS_RE.sub("", cached.get("short_title", "")) if not cached.get("_negative") else "" + ) if saved_short: return saved_short # Fall through to algorithmic path for negative/empty cache hits @@ -498,7 +503,7 @@ def _short_title_for_key(title: str, max_words: int = BIBTEX_KEY_MAX_WORDS, gemi ) return gemini_result - words = [w for w in re.split(r"[^A-Za-z0-9]+", title) if w] + words = [w for w in _TITLE_WORD_SPLIT_RE.split(title) if w] picks: list[str] = [] for w in words: if w.lower() not in _TITLE_STOP_WORDS: @@ -511,9 +516,8 @@ def _short_title_for_key(title: str, max_words: int = BIBTEX_KEY_MAX_WORDS, gemi def _first_author_lastname(authors_field: str | None) -> str | None: - """ - Derive the first author's last name from a BibTeX-style author field, - handling both "First Last" and "Last, First" name formats. + """Derive the first author's last name from a BibTeX-style author field, + handling "First Last" and "Last, First" formats. Strips academic suffixes (Jr, Sr, II, III, etc.) so that names like "Jose F. Rodrigues Jr" produce "rodrigues" instead of "jr". @@ -532,18 +536,16 @@ def _first_author_lastname(authors_field: str | None) -> str | None: while len(toks) > 1 and toks[-1].rstrip(".").lower() in AUTHOR_NAME_SUFFIXES: toks.pop() last = toks[-1] if toks else first - last = re.sub(r"[^a-zA-Z0-9]", "", strip_accents(last)).lower() + last = _NON_ALNUM_RE.sub("", strip_accents(last)).lower() return last or None def build_standard_citekey(entry: dict[str, Any], gemini_api_key: str | None = None) -> str | None: - """ - Build a human-readable citation key such as "Smith2024:MachineLearning" by - combining the first author's name, the year, and key title words. + """Build a citation key such as "Smith2024:MachineLearning" from the first + author's name, the year, and key title words. - Uses BIBTEX_KEY_MAX_WORDS (default 4) to generate more distinctive citation keys, - which helps avoid collisions for papers with similar titles like - "Dairy DigiD: keypoint..." vs "Dairy DigiD: Edge-Cloud..." + Uses BIBTEX_KEY_MAX_WORDS (default 4) title words so similar titles like + "Dairy DigiD: keypoint..." vs "Dairy DigiD: Edge-Cloud..." get distinct keys. """ fields = entry.get("fields") or {} title = (fields.get("title") or "").strip() @@ -562,11 +564,10 @@ def build_standard_citekey(entry: dict[str, Any], gemini_api_key: str | None = N def short_filename_for_entry( entry: dict[str, Any], gemini_api_key: str | None = None, existing_files: set[str] | None = None, max_words: int = 2 ) -> str: - """ - Construct a concise .bib filename from the first author's name, the year, - and a shortened title so that exported files are easy to identify. + """Construct a concise .bib filename from the first author's name, the + year, and a shortened title. - If existing_files is provided, appends more title words to resolve + When existing_files is provided, appends more title words to resolve filename collisions. """ fields = entry.get("fields") or {} @@ -580,7 +581,7 @@ def short_filename_for_entry( def _build_filename(num_words: int) -> str: short = _short_title_for_key(title, max_words=num_words, gemini_api_key=gemini_api_key) or "Title" - base = re.sub(r"[^A-Za-z0-9_\-]+", "", f"{last_cap}{y}-{short}")[:BIBTEX_FILENAME_MAX_LENGTH] + base = _FILENAME_SANITIZE_RE.sub("", f"{last_cap}{y}-{short}")[:BIBTEX_FILENAME_MAX_LENGTH] return f"{base}.bib" for num_words in range(max_words, 11): @@ -602,15 +603,6 @@ def _years_diverge(af: dict[str, Any], bf: dict[str, Any], max_gap: int = 3) -> return bool(a_year and b_year and abs(a_year - b_year) > max_gap) -def _is_preprint_entry(fields: dict[str, Any]) -> bool: - """Check if a BibTeX entry looks like a preprint based on DOI prefix or journal name.""" - doi = (fields.get("doi") or "").lower() - if any(doi.startswith(p) for p in PREPRINT_DOI_PREFIXES): - return True - journal = (fields.get("journal") or "").lower() - return any(ps in journal for ps in PREPRINT_SERVERS) - - def _identifier_title_conflict(af: dict[str, Any], bf: dict[str, Any]) -> bool: """Return True when two records carry clearly different titles. @@ -628,13 +620,12 @@ def _identifier_title_conflict(af: dict[str, Any], bf: dict[str, Any]) -> bool: def bibtex_entries_match_strict(entry_a: dict[str, Any], entry_b: dict[str, Any]) -> bool: - """ - Decide whether two BibTeX records refer to the same publication by comparing - DOI or arXiv identifiers first and then falling back to title, year, and - authors with fuzzy matching to handle formatting variations from different sources. + """Decide whether two BibTeX records refer to the same publication. - Uses a multi-signal composite score when title similarity alone is insufficient - (e.g., preprint/published pairs with rewritten titles). + Compares DOI or arXiv identifiers first, then falls back to fuzzy title, + year, and author matching. Uses a multi-signal composite score when title + similarity alone is insufficient (e.g., preprint/published pairs with + rewritten titles). """ if not entry_a or not entry_b: return False @@ -665,7 +656,7 @@ def bibtex_entries_match_strict(entry_a: dict[str, Any], entry_b: dict[str, Any] category=LogCategory.DEDUP, ) return False - # Exactly one DOI is a preprint — fall through to multi-signal scoring + # Exactly one DOI is a preprint, so fall through to multi-signal scoring preprint_doi = a_doi if a_is_preprint else b_doi published_doi = b_doi if a_is_preprint else a_doi logger.debug( @@ -742,8 +733,8 @@ def bibtex_entries_match_strict(entry_a: dict[str, Any], entry_b: dict[str, Any] logger.debug(f"ENTRY_REJECT | BELOW_MIN_SIM | sim={title_sim:.3f} | result=False", category=LogCategory.DEDUP) return False - a_preprint = _is_preprint_entry(af) - b_preprint = _is_preprint_entry(bf) + a_preprint = _is_preprint_fields(af) + b_preprint = _is_preprint_fields(bf) # Allow composite scoring for: preprint/published pairs, external ID matches, # or very strong multi-author overlap with moderate title similarity. diff --git a/citeforge/cache.py b/citeforge/cache.py index 340a7045..88c6c7d5 100644 --- a/citeforge/cache.py +++ b/citeforge/cache.py @@ -46,10 +46,11 @@ class ResponseCache: All entries expire on the 1st of each calendar month (AST/UTC-4), ensuring a full refresh of every API source at least once per month. - Negative cache entries use a three-tier confirmation system: + Negative cache entries use a three-tier confirmation system. + - Transient errors are never cached (callers simply skip the cache write). - Empty API results are stored as unconfirmed negatives (_confirmations < 3) - and are NOT served from get() — forcing a retry on the next pipeline run. + and are NOT served from get(), forcing a retry on the next pipeline run. - After 3 consecutive empty results (CACHE_NEGATIVE_CONFIRM_RUNS), the entry is promoted to a safe negative (_safe=True) and served until the next Monday or 1st of the following month, whichever comes first. @@ -86,7 +87,8 @@ def _entry_path(self, namespace: str, key: str) -> str: def _safe_negative_expired(entry_ts: float) -> bool: """Check if a safe negative has expired. - Safe negatives expire at the earlier of: + Safe negatives expire at the earlier of two boundaries. + - Next Monday 00:00 AST after the entry was created - 1st of the next month 00:00 AST after the entry was created """ @@ -146,7 +148,7 @@ def get(self, namespace: str, key: str) -> dict[str, Any] | None: _CACHE_MISSES += 1 return None ts = entry.get("timestamp", 0) - # Monthly boundary: all entries older than the 1st of this month are stale. + # All entries written before the 1st of this month are stale. if ts < self._month_boundary: with _CACHE_COUNTER_LOCK: _CACHE_MISSES += 1 @@ -154,11 +156,11 @@ def get(self, namespace: str, key: str) -> dict[str, Any] | None: data = entry.get("data", {}) if data.get("_negative"): if not data.get("_safe"): - # Unconfirmed negative — force API retry. + # Unconfirmed negative, force API retry. with _CACHE_COUNTER_LOCK: _CACHE_MISSES += 1 return None - # Safe negative — check Monday / month-boundary expiry. + # Safe negative, check Monday / month-boundary expiry. if self._safe_negative_expired(ts): with _CACHE_COUNTER_LOCK: _CACHE_MISSES += 1 @@ -190,11 +192,11 @@ def _write_entry( value: dict[str, Any], ttl_days: int, ) -> None: - """Atomic file write — must be called under the namespace lock.""" + """Atomic file write. Must be called under the namespace lock.""" ns_dir = self._ns_dir(namespace) os.makedirs(ns_dir, exist_ok=True) - # ttl_days is accepted for caller intent (and logged in put()), but is - # intentionally NOT stored or read back: positive-entry freshness is + # ttl_days is accepted for caller intent (and logged in put()) but is + # intentionally NOT stored or read back. Positive-entry freshness is # governed solely by the monthly boundary in get(), never a rolling # per-entry TTL. entry = {"timestamp": time.time(), "data": value} diff --git a/citeforge/canonicalize.py b/citeforge/canonicalize.py index c0748d28..883ce79c 100644 --- a/citeforge/canonicalize.py +++ b/citeforge/canonicalize.py @@ -124,7 +124,7 @@ def _rule_pacm_booktitle_to_article(entry: dict[str, Any], fields: dict[str, Any def _rule_named_proceedings_to_article(entry: dict[str, Any], fields: dict[str, Any]) -> bool: """@inproceedings with a PNAS/PVLDB-style journal as booktitle -> @article. - Guard: skip when the booktitle extends the journal name with conference keywords. + Skips when the booktitle extends the journal name with conference keywords. """ if entry.get("type") == "inproceedings" and fields.get("booktitle"): bt_lower = fields["booktitle"].strip().lower() @@ -476,22 +476,34 @@ def _rule_add_url_from_doi(entry: dict[str, Any], fields: dict[str, Any]) -> boo # --------------------------------------------------------------------------- # Site-B-only rules (load repair; existing-file fixup before enrichment) # --------------------------------------------------------------------------- +def _demote_preprint_journal_article(entry: dict[str, Any], fields: dict[str, Any]) -> None: + """Relabel an @article carrying a preprint-server journal as @misc. + + A genuine published DOI means the preprint journal is a stale enrichment + artifact, so it is dropped WITHOUT asserting a preprint (no + ``howpublished=``) and the published DOI is kept. Only when the DOI + is itself secondary (or absent) is the entry an actual preprint, and the + journal moves to howpublished. Shared by the Site B and Site C rules. + """ + doi = (fields.get("doi") or "").strip() + entry["type"] = "misc" + if doi and not idu.is_secondary_doi(doi): + fields.pop("journal", None) + else: + fields["howpublished"] = fields.pop("journal") + + def _rule_strip_preprint_journal_load(entry: dict[str, Any], fields: dict[str, Any]) -> bool: """Strip a preprint-server journal on load. - @article with a genuine published DOI: drop the stale preprint journal but keep it a - published work (no ``howpublished=``). @article without a published DOI: @misc - (journal -> howpublished). Any other type simply drops the journal. + An @article is demoted via ``_demote_preprint_journal_article`` (stale + journal dropped when the DOI is published, journal -> howpublished + otherwise). Any other type simply drops the journal. """ jnl = (fields.get("journal") or "").strip().lower() - if jnl and any(ps == jnl or ps in jnl for ps in PREPRINT_SERVERS): + if jnl and any(ps in jnl for ps in PREPRINT_SERVERS): if entry.get("type") == "article": - doi = (fields.get("doi") or "").strip() - entry["type"] = "misc" - if doi and not idu.is_secondary_doi(doi): - fields.pop("journal", None) - else: - fields["howpublished"] = fields.pop("journal") + _demote_preprint_journal_article(entry, fields) else: fields.pop("journal", None) return True @@ -581,7 +593,7 @@ def _rule_fix_author_casing_load(entry: dict[str, Any], fields: dict[str, Any]) # Site-C-only rules (Phase-4 post-merge; POST_MERGE is terminal) # --------------------------------------------------------------------------- def _rule_article_no_journal_to_misc(entry: dict[str, Any], fields: dict[str, Any]) -> bool: - """Terminal: @article with no journal (enrichment exhausted) -> @misc.""" + """Terminal rule. @article with no journal (enrichment exhausted) -> @misc.""" if entry.get("type") == "article" and not fields.get("journal"): entry["type"] = "misc" return True @@ -589,7 +601,7 @@ def _rule_article_no_journal_to_misc(entry: dict[str, Any], fields: dict[str, An def _rule_inproceedings_no_booktitle_to_misc(entry: dict[str, Any], fields: dict[str, Any]) -> bool: - """Terminal: @inproceedings without booktitle -> @misc (invalid BibTeX otherwise).""" + """Terminal rule. @inproceedings without booktitle -> @misc (invalid BibTeX otherwise).""" if entry.get("type") == "inproceedings" and not fields.get("booktitle"): entry["type"] = "misc" return True @@ -597,23 +609,16 @@ def _rule_inproceedings_no_booktitle_to_misc(entry: dict[str, Any], fields: dict def _rule_preprint_journal_to_misc(entry: dict[str, Any], fields: dict[str, Any]) -> bool: - """@article with a preprint-server journal. + """@article with a preprint-server journal -> @misc. - A genuine *published* DOI means the preprint journal is a stale enrichment artifact, - not evidence that the work is a preprint: drop the stale journal WITHOUT asserting a - preprint (no ``howpublished=``), keeping the published DOI. Only when the DOI - is itself secondary (or absent) is the entry an actual preprint, relabeled @misc with - journal -> howpublished. + Delegates to ``_demote_preprint_journal_article``, which keeps a published + DOI and drops the stale journal, or moves the journal to howpublished when + the DOI is secondary or absent. """ if entry.get("type") == "article": j_lower = (fields.get("journal") or "").lower().strip() - if j_lower and any(ps == j_lower or ps in j_lower for ps in PREPRINT_SERVERS): - doi = (fields.get("doi") or "").strip() - entry["type"] = "misc" - if doi and not idu.is_secondary_doi(doi): - fields.pop("journal", None) - else: - fields["howpublished"] = fields.pop("journal") + if j_lower and any(ps in j_lower for ps in PREPRINT_SERVERS): + _demote_preprint_journal_article(entry, fields) return True return False @@ -753,9 +758,7 @@ def _rule_howpublished_to_inproceedings(entry: dict[str, Any], fields: dict[str, if entry.get("type") == "misc" and fields.get("howpublished"): hp_val = (fields.get("howpublished") or "").strip() hp_lower = hp_val.lower() - is_preprint_hp = ( - any(ps == hp_lower or ps in hp_lower for ps in PREPRINT_SERVERS) or hp_lower in _R20_PREPRINT_HOWPUBLISHED - ) + is_preprint_hp = any(ps in hp_lower for ps in PREPRINT_SERVERS) or hp_lower in _R20_PREPRINT_HOWPUBLISHED is_repository_hp = any(rj in hp_lower for rj in REPOSITORY_AS_JOURNAL) doi = (fields.get("doi") or "").strip() inferred = mu.infer_howpublished_from_doi(doi) if doi else None diff --git a/citeforge/clients/helpers.py b/citeforge/clients/helpers.py index d254da0a..6a3bb952 100644 --- a/citeforge/clients/helpers.py +++ b/citeforge/clients/helpers.py @@ -9,8 +9,10 @@ import re from collections.abc import Callable +from datetime import datetime, timezone from typing import Any +from ..cache import response_cache from ..config import ( SIM_AUTHOR_BONUS, SIM_BEST_ITEM_THRESHOLD, @@ -19,7 +21,7 @@ SIM_YEAR_MATCH_WINDOW, ) from ..log_utils import LogCategory, logger -from ..text_utils import extract_year_from_any +from ..text_utils import extract_author_names, extract_year_from_any, normalize_title _HTML_BR_RE = re.compile(r"", re.IGNORECASE) _HTML_TAG_RE = re.compile(r"<[^>]+>") @@ -65,6 +67,32 @@ def _score_candidate_generic( return s +def title_author_cache_key(title: str, author_name: str | None, prefix: str = "") -> str: + """Build the shared search-cache key from a normalized title and lowercased author. + + This exact format is load-bearing. Existing entries under data/api_cache/ + were written with it, so any change would orphan the whole cache. + """ + return f"{prefix}{normalize_title(title)}|{(author_name or '').strip().lower()}" + + +def _doi_cache_lookup(namespace: str, doi_norm: str) -> tuple[dict[str, Any] | None, bool]: + """Check the response cache for a DOI-keyed entry, logging the outcome. + + Returns ``(entry, True)`` on a positive hit, ``(None, True)`` on a + confirmed negative, and ``(None, False)`` on a miss. + """ + cached = response_cache.get(namespace, doi_norm) + if cached is not None: + if cached.get("_negative"): + logger.debug(f"{namespace} | NEG_HIT | doi={doi_norm}", category=LogCategory.CACHE) + return None, True + logger.debug(f"{namespace} | HIT | doi={doi_norm}", category=LogCategory.CACHE) + return cached, True + logger.debug(f"{namespace} | MISS | doi={doi_norm}", category=LogCategory.CACHE) + return None, False + + def _best_item_by_score( items: list[Any], score_fn: Callable[[Any], float], @@ -92,8 +120,6 @@ def extract_authors_from_article(art: dict[str, Any]) -> list[str] | None: the truncation markers) instead of None so downstream code can still build a reasonable baseline entry. """ - from ..text_utils import extract_author_names - authors = art.get("authors") if not authors: return None @@ -143,6 +169,5 @@ def _sanitize_dblp_author(name: str) -> str: def get_current_year() -> int: - from datetime import datetime, timezone - + """Return the current UTC year.""" return datetime.now(timezone.utc).year diff --git a/citeforge/clients/search_apis.py b/citeforge/clients/search_apis.py index 94837ad3..258006da 100644 --- a/citeforge/clients/search_apis.py +++ b/citeforge/clients/search_apis.py @@ -17,7 +17,7 @@ import time import xml.etree.ElementTree as ElementTree from datetime import datetime, timezone -from typing import Any +from typing import TYPE_CHECKING, Any from ..cache import response_cache from ..config import ( @@ -64,7 +64,11 @@ safe_get_nested, trim_title_default, ) -from .helpers import _best_item_by_score, _sanitize_dblp_author +from ..venue import first_non_generic_container +from .helpers import _best_item_by_score, _doi_cache_lookup, _sanitize_dblp_author, title_author_cache_key + +if TYPE_CHECKING: + from ..api_generics import APISearchConfig _DBLP_ALLOWED_TAGS = frozenset({"article", "inproceedings", "incollection", "phdthesis", "mastersthesis"}) _DBLP_YEAR_RE = re.compile(r"^(19|20)\d{2}$") @@ -73,6 +77,28 @@ _QP_AUTHOR = "query.author" _QP_BIBLIOGRAPHIC = "query.bibliographic" + +def _get_cached_list( + namespace: str, + cache_key: str, + log_prefix: str, + list_key: str = "results", +) -> list[dict[str, Any]] | None: + """Return a cached candidate list, or ``None`` on a cache miss. + + A confirmed negative entry yields an empty list so callers can short-circuit + without re-querying the API. + """ + cached = response_cache.get(namespace, cache_key) + if cached is None: + return None + if cached.get("_negative"): + logger.debug(f"{log_prefix} | NEG_HIT | key={cache_key[:60]}", category=LogCategory.CACHE) + return [] + logger.debug(f"{log_prefix} | HIT | key={cache_key[:60]}", category=LogCategory.CACHE) + return list(cached.get(list_key, [])) + + # ============ Semantic Scholar ============ @@ -108,14 +134,10 @@ def s2_search_papers_multiple( """Search Semantic Scholar for multiple paper candidates.""" if not api_key or not title: return [] - cache_key = f"multi|{normalize_title(title)}|{(author_name or '').strip().lower()}" - cached = response_cache.get("semantic_scholar", cache_key) - if cached is not None: - if cached.get("_negative"): - logger.debug(f"s2_multi | NEG_HIT | key={cache_key[:60]}", category=LogCategory.CACHE) - return [] - logger.debug(f"s2_multi | HIT | key={cache_key[:60]}", category=LogCategory.CACHE) - return list(cached.get("results", [])) + cache_key = title_author_cache_key(title, author_name, prefix="multi|") + cached_list = _get_cached_list("semantic_scholar", cache_key, "s2_multi") + if cached_list is not None: + return cached_list query_parts = [f'"{title}"'] if author_name: query_parts.append(author_name) @@ -141,12 +163,14 @@ def s2_search_papers_multiple( # ============ Crossref ============ -def crossref_search(title: str, author_name: str | None) -> dict[str, Any] | None: - """Look up a publication in Crossref by title and optional author.""" - if not title: - return None +def _crossref_search_config(title: str, author_name: str | None) -> APISearchConfig: + """Return a Crossref search config with title/author query params applied. + + With an author, splits the query into ``query.title`` + ``query.author``; + without one, uses the combined ``query.bibliographic`` field. Adds the + polite-pool ``mailto`` when ``CROSSREF_MAILTO`` is set. + """ from ..api_configs import CROSSREF_SEARCH_CONFIG - from ..api_generics import search_api_generic config = copy.copy(CROSSREF_SEARCH_CONFIG) additional_params = dict(config.additional_params) @@ -159,7 +183,16 @@ def crossref_search(title: str, author_name: str | None) -> dict[str, Any] | Non if mailto: additional_params["mailto"] = mailto config.additional_params = additional_params - return search_api_generic(title, author_name, config) + return config + + +def crossref_search(title: str, author_name: str | None) -> dict[str, Any] | None: + """Look up a publication in Crossref by title and optional author.""" + if not title: + return None + from ..api_generics import search_api_generic + + return search_api_generic(title, author_name, _crossref_search_config(title, author_name)) def build_bibtex_from_crossref(item: dict[str, Any], keyhint: str) -> str | None: @@ -182,20 +215,9 @@ def crossref_search_multiple( """ if not title: return [] - from ..api_configs import CROSSREF_SEARCH_CONFIG from ..api_generics import search_api_generic_multiple - config = copy.copy(CROSSREF_SEARCH_CONFIG) - additional_params = dict(config.additional_params) - if author_name: - additional_params["query.title"] = title - additional_params[_QP_AUTHOR] = author_name - else: - additional_params[_QP_BIBLIOGRAPHIC] = title - mailto = os.getenv("CROSSREF_MAILTO") - if mailto: - additional_params["mailto"] = mailto - config.additional_params = additional_params + config = _crossref_search_config(title, author_name) return search_api_generic_multiple(title, author_name, config, None, max_results, year_hint) @@ -208,14 +230,9 @@ def fetch_csl_via_doi(doi: str, timeout: float = 20.0) -> dict[str, Any] | None: doi_norm = _norm_doi(doi) if not doi_norm: return None - cached = response_cache.get("doi_csl", doi_norm) - if cached is not None: - if cached.get("_negative"): - logger.debug(f"doi_csl | NEG_HIT | doi={doi_norm}", category=LogCategory.CACHE) - return None - logger.debug(f"doi_csl | HIT | doi={doi_norm}", category=LogCategory.CACHE) + cached, hit = _doi_cache_lookup("doi_csl", doi_norm) + if hit: return cached - logger.debug(f"doi_csl | MISS | doi={doi_norm}", category=LogCategory.CACHE) url = f"https://doi.org/{doi_norm}" headers = DEFAULT_JSON_HEADERS.copy() headers["Accept"] = "application/vnd.citationstyles.csl+json" @@ -234,14 +251,9 @@ def fetch_bibtex_via_doi(doi: str, timeout: float = 20.0) -> str | None: doi_norm = _norm_doi(doi) if not doi_norm: return None - cached = response_cache.get("doi_bibtex", doi_norm) - if cached is not None: - if cached.get("_negative"): - logger.debug(f"doi_bibtex | NEG_HIT | doi={doi_norm}", category=LogCategory.CACHE) - return None - logger.debug(f"doi_bibtex | HIT | doi={doi_norm}", category=LogCategory.CACHE) - return cached.get("bibtex") - logger.debug(f"doi_bibtex | MISS | doi={doi_norm}", category=LogCategory.CACHE) + cached, hit = _doi_cache_lookup("doi_bibtex", doi_norm) + if hit: + return cached.get("bibtex") if cached is not None else None url = f"https://doi.org/{doi_norm}" headers = DEFAULT_JSON_HEADERS.copy() headers["Accept"] = "application/x-bibtex" @@ -269,14 +281,7 @@ def bibtex_from_csl(csl: dict[str, Any], keyhint: str) -> str: year = extract_year_from_any(csl, fallback=0) or 0 container_raw = csl.get("container-title") if isinstance(container_raw, list) and len(container_raw) > 1: - container = None - for candidate in container_raw: - candidate_str = str(candidate).strip() - if candidate_str and candidate_str.lower() not in GENERIC_SERIES_NAMES: - container = candidate_str - break - if not container: - container = safe_get_field(csl, "container-title") + container = first_non_generic_container(container_raw) or safe_get_field(csl, "container-title") else: container = safe_get_field(csl, "container-title") if container and container.lower().strip() in GENERIC_SERIES_NAMES: @@ -329,14 +334,10 @@ def arxiv_search( """Search arXiv for papers matching the given title and optional author.""" if not title: return [] - cache_key = f"{normalize_title(title)}|{(author_name or '').strip().lower()}" - cached = response_cache.get("arxiv", cache_key) - if cached is not None: - if cached.get("_negative"): - logger.debug(f"arxiv | NEG_HIT | key={cache_key[:60]}", category=LogCategory.CACHE) - return [] - logger.debug(f"arxiv | HIT | key={cache_key[:60]}", category=LogCategory.CACHE) - return list(cached.get("entries", [])) + cache_key = title_author_cache_key(title, author_name) + cached_list = _get_cached_list("arxiv", cache_key, "arxiv", list_key="entries") + if cached_list is not None: + return cached_list q_parts = [f'ti:"{title}"'] if author_name: q_parts.append(f'au:"{author_name}"') @@ -593,7 +594,7 @@ def openreview_search_paper( """Query OpenReview for notes matching the requested paper.""" if not title: return None - cache_key = f"{normalize_title(title)}|{(author_name or '').strip().lower()}" + cache_key = title_author_cache_key(title, author_name) cached = response_cache.get("openreview", cache_key) if cached is not None: if cached.get("_negative"): @@ -650,14 +651,10 @@ def openreview_search_papers_multiple( """Query OpenReview for multiple candidate notes.""" if not title: return [] - cache_key = f"multi|{normalize_title(title)}|{(author_name or '').strip().lower()}" - cached = response_cache.get("openreview", cache_key) - if cached is not None: - if cached.get("_negative"): - logger.debug(f"openreview_multi | NEG_HIT | key={cache_key[:60]}", category=LogCategory.CACHE) - return [] - logger.debug(f"openreview_multi | HIT | key={cache_key[:60]}", category=LogCategory.CACHE) - return list(cached.get("results", [])) + cache_key = title_author_cache_key(title, author_name, prefix="multi|") + cached_list = _get_cached_list("openreview", cache_key, "openreview_multi") + if cached_list is not None: + return cached_list headers = openreview_login(creds) or DEFAULT_JSON_HEADERS.copy() candidates = _or_fetch_candidates(title, headers) if not candidates: @@ -887,44 +884,66 @@ def openalex_search_multiple( # ============ PubMed ============ -@handle_api_errors(default_return=None) -def pubmed_search_paper(title: str, author_name: str | None) -> dict[str, Any] | None: - """Search PubMed for a publication by title and optional author.""" - if not title: - return None - cache_key = f"{normalize_title(title)}|{(author_name or '').strip().lower()}" - cached = response_cache.get("pubmed", cache_key) - if cached is not None: - if cached.get("_negative"): - logger.debug(f"pubmed | NEG_HIT | key={cache_key[:60]}", category=LogCategory.CACHE) - return None - logger.debug(f"pubmed | HIT | key={cache_key[:60]}", category=LogCategory.CACHE) - return cached if cached else None +def _pubmed_query(title: str, author_name: str | None) -> str: + """Build a PubMed esearch term with title and optional author field tags.""" search_query = f"{title}[Title]" if author_name: search_query += f" AND {author_name}[Author]" + return search_query + + +def _pubmed_fetch_articles( + search_query: str, + retmax: int, + timeout: float, +) -> tuple[list[dict[str, Any]], int] | None: + """Run PubMed's two-step esearch/esummary lookup. + + Returns ``(articles, pmid_count)`` with an empty list when the search found + nothing, or ``None`` on HTTP failure so transient errors are never + negative-cached. + """ search_url = build_url( f"{PUBMED_BASE}/esearch.fcgi", - {"db": "pubmed", "term": search_query, "retmax": 10, "retmode": "json"}, + {"db": "pubmed", "term": search_query, "retmax": retmax, "retmode": "json"}, ) try: - search_data = http_get_json(search_url, timeout=15.0) + search_data = http_get_json(search_url, timeout=timeout) except NETWORK_ERRORS: return None - pmids = (search_data.get("esearchresult") or {}).get("idlist") or [] + pmids = (safe_get_nested(search_data, "esearchresult", "idlist", default=[]) or [])[:retmax] if not pmids: - response_cache.put_negative("pubmed", cache_key) - return None - fetch_url = build_url( + return [], 0 + summary_url = build_url( f"{PUBMED_BASE}/esummary.fcgi", {"db": "pubmed", "id": ",".join(pmids), "retmode": "json"}, ) try: - fetch_data = http_get_json(fetch_url, timeout=15.0) + summary_data = http_get_json(summary_url, timeout=timeout) except NETWORK_ERRORS: return None - result = fetch_data.get("result") or {} + result = safe_get_nested(summary_data, "result", default={}) or {} articles = [result[pmid] for pmid in pmids if pmid in result and isinstance(result[pmid], dict)] + return articles, len(pmids) + + +@handle_api_errors(default_return=None) +def pubmed_search_paper(title: str, author_name: str | None) -> dict[str, Any] | None: + """Search PubMed for a publication by title and optional author.""" + if not title: + return None + cache_key = title_author_cache_key(title, author_name) + cached = response_cache.get("pubmed", cache_key) + if cached is not None: + if cached.get("_negative"): + logger.debug(f"pubmed | NEG_HIT | key={cache_key[:60]}", category=LogCategory.CACHE) + return None + logger.debug(f"pubmed | HIT | key={cache_key[:60]}", category=LogCategory.CACHE) + return cached if cached else None + fetched = _pubmed_fetch_articles(_pubmed_query(title, author_name), retmax=10, timeout=15.0) + if fetched is None: + return None + articles, pmid_count = fetched if not articles: response_cache.put_negative("pubmed", cache_key) return None @@ -937,7 +956,7 @@ def pubmed_search_paper(title: str, author_name: str | None) -> dict[str, Any] | result = dict(article) response_cache.put("pubmed", cache_key, result, ttl_days=CACHE_TTL_SEARCH_DAYS) logger.debug( - f"pubmed | PUT | key={cache_key[:60]} | pmids={len(pmids)}", + f"pubmed | PUT | key={cache_key[:60]} | pmids={pmid_count}", category=LogCategory.CACHE, ) return result @@ -955,7 +974,7 @@ def pubmed_search_paper(title: str, author_name: str | None) -> dict[str, Any] | best = _best_item_by_score(articles, score_fn) if best is not None: response_cache.put("pubmed", cache_key, dict(best), ttl_days=CACHE_TTL_SEARCH_DAYS) - logger.debug(f"pubmed | PUT | key={cache_key[:60]} | pmids={len(pmids)}", category=LogCategory.CACHE) + logger.debug(f"pubmed | PUT | key={cache_key[:60]} | pmids={pmid_count}", category=LogCategory.CACHE) else: response_cache.put_negative("pubmed", cache_key) return best @@ -1006,39 +1025,14 @@ def pubmed_search_papers_multiple(title: str, author_name: str | None, max_resul """Search PubMed for multiple paper candidates.""" if not title: return [] - cache_key = f"multi|{normalize_title(title)}|{(author_name or '').strip().lower()}" - cached = response_cache.get("pubmed", cache_key) - if cached is not None: - if cached.get("_negative"): - logger.debug(f"pubmed_multi | NEG_HIT | key={cache_key[:60]}", category=LogCategory.CACHE) - return [] - logger.debug(f"pubmed_multi | HIT | key={cache_key[:60]}", category=LogCategory.CACHE) - return list(cached.get("results", [])) - search_query = f"{title}[Title]" - if author_name: - search_query += f" AND {author_name}[Author]" - search_url = build_url( - f"{PUBMED_BASE}/esearch.fcgi", - {"db": "pubmed", "term": search_query, "retmax": max_results, "retmode": "json"}, - ) - try: - search_data = http_get_json(search_url, timeout=20.0) - except NETWORK_ERRORS: - return [] - id_list = safe_get_nested(search_data, "esearchresult", "idlist", default=[]) - if not id_list: - response_cache.put_negative("pubmed", cache_key) - return [] - summary_url = build_url( - f"{PUBMED_BASE}/esummary.fcgi", - {"db": "pubmed", "id": ",".join(id_list[:max_results]), "retmode": "json"}, - ) - try: - summary_data = http_get_json(summary_url, timeout=20.0) - except NETWORK_ERRORS: + cache_key = title_author_cache_key(title, author_name, prefix="multi|") + cached_list = _get_cached_list("pubmed", cache_key, "pubmed_multi") + if cached_list is not None: + return cached_list + fetched = _pubmed_fetch_articles(_pubmed_query(title, author_name), retmax=max_results, timeout=20.0) + if fetched is None: return [] - result = safe_get_nested(summary_data, "result", default={}) - results_list = [result[uid] for uid in id_list[:max_results] if uid in result and isinstance(result[uid], dict)] + results_list, _ = fetched if results_list: response_cache.put("pubmed", cache_key, {"results": results_list}, ttl_days=CACHE_TTL_SEARCH_DAYS) logger.debug(f"pubmed_multi | PUT | key={cache_key[:60]}", category=LogCategory.CACHE) @@ -1050,6 +1044,15 @@ def pubmed_search_papers_multiple(title: str, author_name: str | None, max_resul # ============ Europe PMC ============ +def _europepmc_query(title: str, author_name: str | None) -> str: + """Build a Europe PMC fielded query, quoting the title and optional author.""" + safe_title = title.replace('"', "") + query = f'TITLE:"{safe_title}"' + if author_name: + query += f' AND AUTH:"{author_name}"' + return query + + def europepmc_search_paper(title: str, author_name: str | None) -> dict[str, Any] | None: """Search Europe PMC for a publication by title and optional author.""" if not title: @@ -1057,12 +1060,11 @@ def europepmc_search_paper(title: str, author_name: str | None) -> dict[str, Any from ..api_configs import EUROPEPMC_SEARCH_CONFIG from ..api_generics import search_api_generic - safe_title = title.replace('"', "") - query = f'TITLE:"{safe_title}"' - if author_name: - query += f' AND AUTH:"{author_name}"' config = copy.copy(EUROPEPMC_SEARCH_CONFIG) - config.additional_params = {**config.additional_params, config.query_param_name: query} + config.additional_params = { + **config.additional_params, + config.query_param_name: _europepmc_query(title, author_name), + } return search_api_generic(title, author_name, config) @@ -1122,22 +1124,18 @@ def europepmc_search_papers_multiple(title: str, author_name: str | None, max_re """Search Europe PMC for multiple paper candidates.""" if not title: return [] - cache_key = f"multi|{normalize_title(title)}|{(author_name or '').strip().lower()}" - cached = response_cache.get("europepmc", cache_key) - if cached is not None: - if cached.get("_negative"): - logger.debug(f"europepmc_multi | NEG_HIT | key={cache_key[:60]}", category=LogCategory.CACHE) - return [] - logger.debug(f"europepmc_multi | HIT | key={cache_key[:60]}", category=LogCategory.CACHE) - return list(cached.get("results", [])) + cache_key = title_author_cache_key(title, author_name, prefix="multi|") + cached_list = _get_cached_list("europepmc", cache_key, "europepmc_multi") + if cached_list is not None: + return cached_list from ..api_configs import EUROPEPMC_SEARCH_CONFIG - safe_title = title.replace('"', "") - query = f'TITLE:"{safe_title}"' - if author_name: - query += f' AND AUTH:"{author_name}"' config = copy.copy(EUROPEPMC_SEARCH_CONFIG) - config.additional_params = {**config.additional_params, "query": query, "pageSize": max_results} + config.additional_params = { + **config.additional_params, + "query": _europepmc_query(title, author_name), + "pageSize": max_results, + } url = build_url(config.base_url, config.additional_params) try: data = http_get_json(url, timeout=config.timeout) @@ -1155,49 +1153,30 @@ def europepmc_search_papers_multiple(title: str, author_name: str | None, max_re # ============ Venue-based searches (SerpAPI publication string) ============ -def crossref_search_by_venue( +def _venue_scored_search( + namespace: str, title: str, author_name: str | None, - container_title: str, - max_results: int = 5, + venue: str, + config: APISearchConfig, + params: dict[str, Any], + max_results: int, ) -> list[dict[str, Any]]: - """Search Crossref using venue metadata from a SerpAPI publication string. + """Shared fetch/score/cache scaffold for the venue-filtered searches. - Uses ``query.container-title`` for the journal/proceedings name and - ``query.bibliographic`` for the title, providing a different search vector - from the standard title-based search. + Candidates are scored against the target title/author and kept when they + clear ``SIM_BEST_ITEM_THRESHOLD``. Results are cached under *namespace* + with a ``venue|`` key; empty result sets are negative-cached. """ - if not title or not container_title: - return [] - - from ..api_configs import CROSSREF_SEARCH_CONFIG from ..api_generics import _build_scoring_function - cache_key = ( - f"venue|{normalize_title(title)}|{(author_name or '').strip().lower()}|{container_title.lower().strip()}" - ) - cached = response_cache.get("crossref_venue", cache_key) - if cached is not None: - if cached.get("_negative"): - logger.debug(f"crossref_venue | NEG_HIT | key={cache_key[:60]}", category=LogCategory.CACHE) - return [] - cached_list: list[dict[str, Any]] = cached.get("results", []) - logger.debug(f"crossref_venue | HIT | key={cache_key[:60]}", category=LogCategory.CACHE) + cache_key = f"venue|{title_author_cache_key(title, author_name)}|{venue.lower().strip()}" + cached_list = _get_cached_list(namespace, cache_key, namespace) + if cached_list is not None: return cached_list - config = copy.copy(CROSSREF_SEARCH_CONFIG) - params: dict[str, Any] = dict(config.additional_params) - params["query.container-title"] = container_title - params[_QP_BIBLIOGRAPHIC] = title - if author_name: - params[_QP_AUTHOR] = author_name - mailto = os.getenv("CROSSREF_MAILTO") - if mailto: - params["mailto"] = mailto - params["rows"] = max(max_results, 10) - url = build_url(config.base_url, params) - logger.debug(f"crossref_venue | HTTP_REQUEST | url={url[:80]}", category=LogCategory.SCORE) + logger.debug(f"{namespace} | HTTP_REQUEST | url={url[:80]}", category=LogCategory.SCORE) try: data = http_get_json(url, timeout=config.timeout) @@ -1206,7 +1185,7 @@ def crossref_search_by_venue( results = safe_get_nested(data, *config.result_path, default=[]) if not results: - response_cache.put_negative("crossref_venue", cache_key) + response_cache.put_negative(namespace, cache_key) return [] score_fn = _build_scoring_function(title, author_name, config) @@ -1222,13 +1201,44 @@ def crossref_search_by_venue( scored.sort(key=lambda x: x[0], reverse=True) top = [item for _, item in scored[:max_results]] if top: - cv = {"results": [dict(r) for r in top]} - response_cache.put("crossref_venue", cache_key, cv, ttl_days=CACHE_TTL_SEARCH_DAYS) + cache_value = {"results": [dict(r) for r in top]} + response_cache.put(namespace, cache_key, cache_value, ttl_days=CACHE_TTL_SEARCH_DAYS) else: - response_cache.put_negative("crossref_venue", cache_key) + response_cache.put_negative(namespace, cache_key) return top +def crossref_search_by_venue( + title: str, + author_name: str | None, + container_title: str, + max_results: int = 5, +) -> list[dict[str, Any]]: + """Search Crossref using venue metadata from a SerpAPI publication string. + + Uses ``query.container-title`` for the journal/proceedings name and + ``query.bibliographic`` for the title, providing a different search vector + from the standard title-based search. + """ + if not title or not container_title: + return [] + + from ..api_configs import CROSSREF_SEARCH_CONFIG + + config = copy.copy(CROSSREF_SEARCH_CONFIG) + params: dict[str, Any] = dict(config.additional_params) + params["query.container-title"] = container_title + params[_QP_BIBLIOGRAPHIC] = title + if author_name: + params[_QP_AUTHOR] = author_name + mailto = os.getenv("CROSSREF_MAILTO") + if mailto: + params["mailto"] = mailto + params["rows"] = max(max_results, 10) + + return _venue_scored_search("crossref_venue", title, author_name, container_title, config, params, max_results) + + def openalex_search_by_venue( title: str, author_name: str | None, @@ -1244,17 +1254,6 @@ def openalex_search_by_venue( return [] from ..api_configs import OPENALEX_SEARCH_CONFIG - from ..api_generics import _build_scoring_function - - cache_key = f"venue|{normalize_title(title)}|{(author_name or '').strip().lower()}|{venue_name.lower().strip()}" - cached = response_cache.get("openalex_venue", cache_key) - if cached is not None: - if cached.get("_negative"): - logger.debug(f"openalex_venue | NEG_HIT | key={cache_key[:60]}", category=LogCategory.CACHE) - return [] - cached_list: list[dict[str, Any]] = cached.get("results", []) - logger.debug(f"openalex_venue | HIT | key={cache_key[:60]}", category=LogCategory.CACHE) - return cached_list config = copy.copy(OPENALEX_SEARCH_CONFIG) params: dict[str, Any] = dict(config.additional_params) @@ -1262,34 +1261,4 @@ def openalex_search_by_venue( params["filter"] = f"primary_location.source.display_name.search:{venue_name}" params["per-page"] = max(max_results, 10) - url = build_url(config.base_url, params) - logger.debug(f"openalex_venue | HTTP_REQUEST | url={url[:80]}", category=LogCategory.SCORE) - - try: - data = http_get_json(url, timeout=config.timeout) - except ALL_API_ERRORS: - return [] - - results = safe_get_nested(data, *config.result_path, default=[]) - if not results: - response_cache.put_negative("openalex_venue", cache_key) - return [] - - score_fn = _build_scoring_function(title, author_name, config) - scored = [] - for item in results: - try: - score = score_fn(item) - if score is not None and score >= SIM_BEST_ITEM_THRESHOLD: - scored.append((score, item)) - except FIELD_ACCESS_ERRORS: - continue - - scored.sort(key=lambda x: x[0], reverse=True) - top = [item for _, item in scored[:max_results]] - if top: - ov = {"results": [dict(r) for r in top]} - response_cache.put("openalex_venue", cache_key, ov, ttl_days=CACHE_TTL_SEARCH_DAYS) - else: - response_cache.put_negative("openalex_venue", cache_key) - return top + return _venue_scored_search("openalex_venue", title, author_name, venue_name, config, params, max_results) diff --git a/citeforge/clients/serpapi_scholar.py b/citeforge/clients/serpapi_scholar.py index 831b9228..3cea816e 100644 --- a/citeforge/clients/serpapi_scholar.py +++ b/citeforge/clients/serpapi_scholar.py @@ -7,8 +7,8 @@ This module handles **author publication retrieval** only. Per-article citation detail lookups remain in ``serply_scholar.py`` (cheaper API). -Thread safety: all calls are stateless HTTP GETs through ``http_fetch_bytes``, -so no locking is required. +All calls are stateless HTTP GETs through ``http_fetch_bytes``, so no +locking is required. SerpAPI response structure (``/search?engine=google_scholar_author``):: @@ -57,7 +57,7 @@ def _serpapi_get( author_id: Google Scholar author profile ID. start: Result offset for pagination (0, 100, 200, ...). num: Results per page (max 100). - sort: Sort order — ``"pubdate"`` (newest first) or ``"citedby"``. + sort: Sort order, ``"pubdate"`` (newest first) or ``"citedby"``. Returns: Parsed JSON response dict, or empty dict on failure. @@ -147,7 +147,7 @@ def serpapi_fetch_author_publications( api_key: SerpAPI key. author_id: Google Scholar profile ID (e.g., ``"dg7f4K8AAAAJ"``). num: Maximum number of articles to return (hard safety cap). - sort: Sort order — ``"pubdate"`` or ``"citedby"``. + sort: Sort order, ``"pubdate"`` or ``"citedby"``. min_year: Minimum publication year to fetch. When > 0 and *sort* is ``"pubdate"``, pagination stops once a full page falls below this year. 0 disables year-bounded stopping. diff --git a/citeforge/clients/serply_scholar.py b/citeforge/clients/serply_scholar.py index 700dba70..7701c8b9 100644 --- a/citeforge/clients/serply_scholar.py +++ b/citeforge/clients/serply_scholar.py @@ -1,15 +1,14 @@ """Google Scholar access via the Serply REST API (api.serply.io). -A clean REST client for Google Scholar access through the Serply API. It exposes -two public functions: +Exposes two public functions. - ``serply_fetch_citation`` (title and author search for citation detail), the entry point used in production by ``scholar.py`` - ``serply_fetch_author_publications`` (keyword search by author name), a secondary entry point exercised by the tests -Thread safety: all calls are stateless HTTP GETs through ``http_fetch_bytes``, -so no locking is required. +All calls are stateless HTTP GETs through ``http_fetch_bytes``, so no +locking is required. Serply response structure (``/v1/scholar/{query}``):: diff --git a/citeforge/clients/utility_apis.py b/citeforge/clients/utility_apis.py index 7bca61e5..312d810b 100644 --- a/citeforge/clients/utility_apis.py +++ b/citeforge/clients/utility_apis.py @@ -21,7 +21,7 @@ from ..id_utils import _norm_doi from ..log_utils import LogCategory, LogSource, logger from ..text_utils import extract_year_from_any, normalize_title -from .helpers import _best_item_by_score +from .helpers import _best_item_by_score, _doi_cache_lookup # ============ Gemini ============ @@ -76,8 +76,14 @@ def gemini_generate_short_title(full_title: str, api_key: str, max_words: int | try: data = http_post_json(url, payload, headers=request_headers, timeout=15.0) break # success - except _requests.exceptions.HTTPError as e: - if e.response is not None and e.response.status_code == 429 and attempt < max_retries: + except (*ALL_API_ERRORS, ValueError) as e: + retryable_429 = ( + isinstance(e, _requests.exceptions.HTTPError) + and e.response is not None + and e.response.status_code == 429 + and attempt < max_retries + ) + if retryable_429: wait = (2**attempt) + random.uniform(0, 1) logger.debug( f"GEMINI_429 | attempt={attempt} | backoff={wait:.1f}s", @@ -97,14 +103,6 @@ def gemini_generate_short_title(full_title: str, api_key: str, max_words: int | source=LogSource.SYSTEM, ) return None - except (*ALL_API_ERRORS, ValueError) as e: - logger.debug(f"GEMINI_FAIL | error={type(e).__name__}", category=LogCategory.CITEKEY) - logger.warn( - f"API call failed: {type(e).__name__}", - category=LogCategory.ERROR, - source=LogSource.SYSTEM, - ) - return None if data is None: logger.debug("GEMINI_FAIL | reason=no_response", category=LogCategory.CITEKEY) @@ -156,14 +154,9 @@ def datacite_search_doi(doi: str) -> dict[str, Any] | None: if not doi_norm: return None - cached = response_cache.get("datacite", doi_norm) - if cached is not None: - if cached.get("_negative"): - logger.debug(f"datacite | NEG_HIT | doi={doi_norm}", category=LogCategory.CACHE) - return None - logger.debug(f"datacite | HIT | doi={doi_norm}", category=LogCategory.CACHE) + cached, hit = _doi_cache_lookup("datacite", doi_norm) + if hit: return cached - logger.debug(f"datacite | MISS | doi={doi_norm}", category=LogCategory.CACHE) encoded_doi = urllib.parse.quote(doi_norm, safe="") url = f"{DATACITE_BASE}/{encoded_doi}" diff --git a/citeforge/config.py b/citeforge/config.py index f2371ccc..f3f70f42 100644 --- a/citeforge/config.py +++ b/citeforge/config.py @@ -1,10 +1,8 @@ """Central configuration for CiteForge. -This module is the single source of truth for every tunable constant, including -the source trust hierarchy, similarity thresholds, HTTP rate limits and -timeouts, API endpoints, key-file paths, and the venue and compound-word -dictionaries. Per the config-driven convention these values live here and are -never hardcoded elsewhere. +Single source of truth for tunable constants: trust hierarchy, similarity +thresholds, HTTP rate limits and timeouts, API endpoints, key-file paths, and +venue/compound-word dictionaries. Never hardcode these values elsewhere. """ from __future__ import annotations @@ -71,9 +69,7 @@ def _current_year() -> int: # Skip Scholar citation fetch if BibTeX file already exists SKIP_SCHOLAR_FOR_EXISTING_FILES = True -# Trust hierarchy for merging metadata from different sources. -# Sources earlier in the list are more reliable than those later. -# This ordering reflects data quality, completeness, and standardization. +# Trust hierarchy for merging metadata; earlier sources win over later ones. TRUST_ORDER = [ "csl", # DOI → CSL-JSON (highest trust, structured metadata) "doi_bibtex", # DOI → BibTeX (direct from DOI resolver) @@ -100,14 +96,12 @@ def _current_year() -> int: SIM_TITLE_SIM_MIN = 0.8 # min title sim to consider a candidate SIM_EXACT_PICK_THRESHOLD = 0.9 # auto-accept single strong candidate SIM_BEST_ITEM_THRESHOLD = 0.8 # min score for best-of-N selection -SIM_SCHOLAR_FUZZY_ACCEPT = 0.9 # min sim for noisy Scholar data SIM_MERGE_DUPLICATE_THRESHOLD = 0.95 # threshold for merge-level dedup # DOI regex pattern _DOI_REGEX = r"\b(10\.\d{4,9}/[-._;()/:A-Za-z0-9]+)\b" -# arXiv DOI patterns -ARXIV_DOI_CHECK_PATTERN = r"10\.48550/arxiv" +# arXiv DOI extraction pattern ARXIV_DOI_EXTRACT_PATTERN = r"(?i)10\.48550/arxiv\.([0-9]{4}\.[0-9]{4,5})" # HTTP timeouts (seconds) @@ -152,12 +146,8 @@ def _current_year() -> int: # File-level dedup threshold (must be >= SIM_MERGE_DUPLICATE_THRESHOLD) SIM_FILE_DUPLICATE_THRESHOLD = 0.95 -# Minimum title similarity required to trust an EXACT DOI/arXiv identifier match. -# Guards against mislabeled identifiers: when two records share a DOI or arXiv id -# but their titles are clearly different papers, the identifier was attached to -# the wrong work (e.g., a Scholar entry pointing at the wrong arXiv id) and must -# not be treated as a match. The same paper reformatted across sources scores far -# above this; only genuinely different titles fall below it. +# Minimum title similarity required to trust an exact DOI/arXiv identifier match. +# Guards against mislabeled identifiers (same DOI attached to different papers). SIM_IDENTIFIER_TITLE_MIN = 0.55 # Preprint detection @@ -197,11 +187,9 @@ def _current_year() -> int: "10.64898/", # openRxiv "10.36227/techrxiv", # TechRxiv (IEEE preprints) "10.33774/", # Cambridge UP preprints (Authoria/MIIR) - # Grey-literature / preprint sub-prefixes. Keyed on the SPECIFIC sub-prefix - # (not the registrant) so a published journal DOI under the same registrant - # (e.g. 10.5194/acp) stays published. Kept in sync with - # venue._DOI_PREFIX_TO_HOWPUB so every DOI recognized for howpublished - # inference is also classified as secondary by is_secondary_doi. + # Grey-literature sub-prefixes: keyed on the SPECIFIC sub-prefix, not the + # registrant, so published DOIs under the same registrant stay published. + # MUST stay in sync with venue._DOI_PREFIX_TO_HOWPUB. "10.5194/egusphere", # EGU preprints (egusphere), NOT published EGU journals "10.2172/", # OSTI technical reports "10.31220/agrirxiv", # agriRxiv @@ -327,9 +315,8 @@ def _current_year() -> int: "Genome biology and evolution": "Genome Biology and Evolution", } -# Acronym case corrections for title fields (API sources sometimes return -# incorrect casing for well-known acronyms). Keys are the wrong form, values -# are the correct form. Applied via word-boundary regex on title fields. +# Acronym case corrections for titles: wrong form → correct form. +# Applied via word-boundary regex on title fields. ACRONYM_CASE_CORRECTIONS: dict[str, str] = { "Iot": "IoT", "Nims": "NIMS", @@ -365,7 +352,8 @@ def _current_year() -> int: _NIME_FULL = "New Interfaces for Musical Expression" -# Abbreviated venue names → full conference names (for S2/DBLP expansion) +# Abbreviated venue names → full conference names (for S2/DBLP expansion). +# WARNING: no em-dashes or accented characters in values; the serializer strips them. ABBREVIATED_VENUE_MAP: dict[str, str] = { "spire": "String Processing and Information Retrieval", "ircdl": "Italian Research Conference on Digital Libraries", @@ -451,12 +439,9 @@ def _current_year() -> int: # Author name suffixes to strip when extracting last names AUTHOR_NAME_SUFFIXES = frozenset({"jr", "sr", "ii", "iii", "iv", "v"}) -# Fused compound words: hyphens stripped by Google Scholar. -# Maps lowercased fused form → correctly hyphenated replacement. -# Suffixes that reliably form hyphenated compound adjectives in scientific text. -# Used by _fix_fused_compounds() as a fallback after dictionary lookup. -# The suffix approach matches words like "Knowledgedriven" → "Knowledge-Driven" -# when the prefix has ≥3 characters starting with an uppercase letter. +# Suffixes that form hyphenated compound adjectives ("Knowledgedriven" → +# "Knowledge-Driven"). Used by _fix_fused_compounds() as a fallback after +# FUSED_COMPOUND_WORDS lookup; requires a ≥3-char capitalized prefix. COMPOUND_SUFFIXES: tuple[str, ...] = ( "based", "driven", @@ -497,11 +482,10 @@ def _current_year() -> int: "aided", ) -# Dictionary of fused compound words for cases NOT caught by suffix-based detection: -# - Acronym prefixes (AI, FM, EEG, 6G, D2D, DNS, etc.) -# - Short prefixes (In, E, Low, etc.) -# - Irregular patterns (realtime, objectoriented, etc.) -# - Multi-word compounds (stateoftheart, endtoend, etc.) +# Fused compound words (hyphens stripped by Google Scholar) not caught by +# COMPOUND_SUFFIXES: acronym prefixes, short prefixes, irregular patterns, +# multi-word compounds. Maps lowercased fused form → hyphenated replacement. +# WARNING: no em-dashes or accented characters in values; the serializer strips them. FUSED_COMPOUND_WORDS: dict[str, str] = { # --- Multi-word compounds --- "stateoftheart": "State-of-the-Art", diff --git a/citeforge/doi_utils.py b/citeforge/doi_utils.py index 12c2fe85..f7704f6c 100644 --- a/citeforge/doi_utils.py +++ b/citeforge/doi_utils.py @@ -17,10 +17,41 @@ from .text_utils import normalize_title, title_similarity -def _validate_csl(doi: str, baseline_entry: dict[str, Any], result_id: str) -> tuple[bool, dict[str, Any] | None, Any]: - """ - Helper to validate DOI using CSL-JSON format. +def _parse_and_match( + raw_bib: str, + baseline_entry: dict[str, Any], + doi: str, + label: str, + format_name: str, +) -> tuple[bool, dict[str, Any] | None]: + """Parse a fetched BibTeX string and strict-match it against the baseline. + + Shared tail of the CSL and BibTeX validators. *label* prefixes the debug + log lines and *format_name* names the format in the success message. """ + entry = bt.parse_bibtex_to_dict(raw_bib) + logger.debug( + f"{label}_PARSE | entry_ok={entry is not None}", + category=LogCategory.DOI_VAL, + source=LogSource.DOI, + ) + if entry is None: + return False, None + + strict_match = bt.bibtex_entries_match_strict(baseline_entry, entry) + logger.debug(f"{label}_MATCH | strict_match={strict_match}", category=LogCategory.DOI_VAL, source=LogSource.DOI) + if strict_match: + logger.success( + f"{doi}: {format_name} format validated and added", + category=LogCategory.MATCH, + source=LogSource.DOI, + ) + return True, entry + return False, None + + +def _validate_csl(doi: str, baseline_entry: dict[str, Any], result_id: str) -> tuple[bool, dict[str, Any] | None, Any]: + """Validate a DOI through its CSL-JSON metadata.""" logger.debug(f"CSL_START | doi={doi}", category=LogCategory.DOI_VAL, source=LogSource.DOI) try: csl = search_apis.fetch_csl_via_doi(doi) @@ -37,19 +68,8 @@ def _validate_csl(doi: str, baseline_entry: dict[str, Any], result_id: str) -> t if not csl_bib: return False, None, None - csl_entry = bt.parse_bibtex_to_dict(csl_bib) - logger.debug( - f"CSL_PARSE | entry_ok={csl_entry is not None}", - category=LogCategory.DOI_VAL, - source=LogSource.DOI, - ) - if csl_entry is None: - return False, None, None - - strict_match = bt.bibtex_entries_match_strict(baseline_entry, csl_entry) - logger.debug(f"CSL_MATCH | strict_match={strict_match}", category=LogCategory.DOI_VAL, source=LogSource.DOI) - if strict_match: - logger.success(f"{doi}: CSL format validated and added", category=LogCategory.MATCH, source=LogSource.DOI) + matched, csl_entry = _parse_and_match(csl_bib, baseline_entry, doi, "CSL", "CSL") + if matched: return True, csl_entry, csl except ALL_API_ERRORS as e: @@ -64,9 +84,7 @@ def _validate_csl(doi: str, baseline_entry: dict[str, Any], result_id: str) -> t def _validate_bibtex(doi: str, baseline_entry: dict[str, Any]) -> tuple[bool, dict[str, Any] | None, Any]: - """ - Helper to validate DOI using BibTeX format. - """ + """Validate a DOI through its BibTeX metadata.""" logger.debug(f"BIBTEX_START | doi={doi}", category=LogCategory.DOI_VAL, source=LogSource.DOI) try: doi_bib = search_apis.fetch_bibtex_via_doi(doi) @@ -74,23 +92,8 @@ def _validate_bibtex(doi: str, baseline_entry: dict[str, Any]) -> tuple[bool, di if not doi_bib: return False, None, None - bibtex_entry = bt.parse_bibtex_to_dict(doi_bib) - logger.debug( - f"BIBTEX_PARSE | entry_ok={bibtex_entry is not None}", - category=LogCategory.DOI_VAL, - source=LogSource.DOI, - ) - if bibtex_entry is None: - return False, None, None - - strict_match = bt.bibtex_entries_match_strict(baseline_entry, bibtex_entry) - logger.debug(f"BIBTEX_MATCH | strict_match={strict_match}", category=LogCategory.DOI_VAL, source=LogSource.DOI) - if strict_match: - logger.success( - f"{doi}: BibTeX format validated and added", - category=LogCategory.MATCH, - source=LogSource.DOI, - ) + matched, bibtex_entry = _parse_and_match(doi_bib, baseline_entry, doi, "BIBTEX", "BibTeX") + if matched: return True, bibtex_entry, doi_bib except ALL_API_ERRORS as e: diff --git a/citeforge/fsscan.py b/citeforge/fsscan.py index 11f49907..26c6bf2b 100644 --- a/citeforge/fsscan.py +++ b/citeforge/fsscan.py @@ -1,15 +1,20 @@ """Centralized, deterministic directory scans. -Single source of truth for the two directory-scan shapes the pipeline relies -on. Both return sorted results so that determinism (byte-identical output on -cache-hit runs) is structural rather than duplicated at every call site. +Single source of truth for the directory-scan shapes the pipeline relies on. +All scans iterate in sorted order so that determinism (byte-identical output +on cache-hit runs) is structural rather than duplicated at every call site. -Leaf module: depends only on the standard library. +Near-leaf module: standard library plus the BibTeX parser (for the shared +scan+parse core used by the per-article duplicate scans). """ from __future__ import annotations import os +from collections.abc import Callable, Iterator +from typing import Any + +from .bibtex_utils import parse_bibtex_to_dict def iter_author_bibs(author_dir: str) -> list[str]: @@ -21,6 +26,48 @@ def iter_author_bibs(author_dir: str) -> list[str]: return sorted(f for f in os.listdir(author_dir) if f.endswith(".bib")) +def iter_parsed_author_bibs( + author_dir: str, + *, + skip_basename: str | None = None, + skip_path: str | None = None, + read_errors: tuple[type[Exception], ...] = (OSError,), + on_read_error: Callable[[str], None] | None = None, +) -> Iterator[tuple[str, str, dict[str, Any]]]: + """Yield ``(filename, path, entry)`` for each parseable ``.bib`` in *author_dir*. + + Shared scan+parse core for the per-article duplicate scans (Phase 4 + candidate-DOI dedup in ``pipeline.article`` and + ``merge_utils.save_entry_to_file``). Iterates in :func:`iter_author_bibs` + order (sorted), so determinism is structural. Match semantics stay at the + call sites; this helper only owns which files are opened and how read + failures are skipped. + + ``skip_basename`` skips a file by name and ``skip_path`` by absolute-path + identity, both before the file is opened. Exceptions in ``read_errors`` + raised while reading or parsing a file cause that file to be skipped, + after invoking ``on_read_error`` with its filename when provided; other + exceptions propagate. Files that parse to a falsy entry are skipped + silently. + """ + for filename in iter_author_bibs(author_dir): + if skip_basename is not None and filename == skip_basename: + continue + path = os.path.join(author_dir, filename) + if skip_path is not None and os.path.abspath(path) == os.path.abspath(skip_path): + continue + try: + with open(path, encoding="utf-8") as fh: + entry = parse_bibtex_to_dict(fh.read()) + except read_errors: + if on_read_error is not None: + on_read_error(filename) + continue + if not entry: + continue + yield filename, path, entry + + def iter_output_dirs(out_dir: str) -> list[str]: """Return the immediate subdirectory names of ``out_dir``, sorted. diff --git a/citeforge/http_utils.py b/citeforge/http_utils.py index 5cc7920e..04bcbc3d 100644 --- a/citeforge/http_utils.py +++ b/citeforge/http_utils.py @@ -288,9 +288,7 @@ def _get_session() -> requests.Session: def handle_api_errors(default_return: Any = None) -> Callable[[Callable[..., T]], Callable[..., T]]: - """ - Decorator to handle API errors consistently across all API client functions, returning a default value on error. - """ + """Decorator that returns *default_return* when the wrapped call raises an API error.""" def decorator(func: Callable[..., T]) -> Callable[..., T]: @wraps(func) @@ -337,15 +335,15 @@ def _http_request( timeout: float, json_payload: dict[str, Any] | None = None, ) -> bytes: - """Execute an HTTP request with the full CiteForge infrastructure. + """Execute an HTTP request through the shared client infrastructure. - Applies, in order: namespace classification, API call tracking, - per-API rate limiting, header randomization, global concurrency - gating, session management with rotation, Retry-After back-off, - and exponential retry on transient errors. + Applies namespace classification, API call tracking, per-API rate + limiting, header randomization, global concurrency gating, session + management with rotation, Retry-After back-off, and exponential retry + on transient errors, in that order. Args: - method: HTTP method -- ``"GET"`` or ``"POST"``. + method: HTTP method, ``"GET"`` or ``"POST"``. url: Target URL. headers: Base headers (will be copied and randomized). timeout: Read timeout in seconds; connect timeout is capped at 10 s. @@ -406,7 +404,7 @@ def _http_request( else: time.sleep((2**attempt) + random.uniform(0, 1)) - # Unreachable -- the loop always returns or raises -- but satisfies mypy. + # Unreachable (the loop always returns or raises) but satisfies mypy. raise requests.exceptions.RequestException(f"Failed to {method} {_scrub_secrets(url)}") @@ -432,7 +430,7 @@ def _decode_json_bytes(raw: bytes, url: str) -> dict[str, Any]: result: dict[str, Any] = json.loads(raw.decode("utf-8")) return result except json.JSONDecodeError as ex: - # include a preview for debugging (scrub the preview too — an upstream + # include a preview for debugging (scrub the preview too; an upstream # error body may echo request params carrying a key) preview = _scrub_secrets(raw[:256].decode("utf-8", errors="replace")) safe_url = _scrub_secrets(url) diff --git a/citeforge/id_utils.py b/citeforge/id_utils.py index c4eed873..f09493a6 100644 --- a/citeforge/id_utils.py +++ b/citeforge/id_utils.py @@ -42,10 +42,7 @@ def _norm_doi(doi: str | None) -> str | None: def normalize_doi(doi: str | None) -> str | None: - """ - Provide a public helper that normalizes DOIs into a consistent canonical - form suitable for comparison and lookups. - """ + """Normalize a DOI to its canonical comparison form (public alias of ``_norm_doi``).""" return _norm_doi(doi) diff --git a/citeforge/io_utils.py b/citeforge/io_utils.py index 789b933f..9eaf69cb 100644 --- a/citeforge/io_utils.py +++ b/citeforge/io_utils.py @@ -15,16 +15,24 @@ import threading from typing import Any +from . import bibtex_utils as bt +from .clients.helpers import get_current_year from .config import ( + A2I2_OUTPUT_DIR, DEFAULT_GEMINI_KEY_FILE, DEFAULT_INPUT, DEFAULT_OR_KEY_FILE, DEFAULT_S2_KEY_FILE, DEFAULT_SERPAPI_KEY_FILE, DEFAULT_SERPLY_KEY_FILE, + SIM_MERGE_DUPLICATE_THRESHOLD, + get_min_year, ) from .exceptions import CSV_ERRORS, FILE_READ_ERRORS +from .fsscan import iter_author_bibs, iter_output_dirs +from .id_utils import doi_bases_match, normalize_doi from .models import Record +from .text_utils import format_author_dirname, normalize_person_name, title_similarity _SUMMARY_CSV_FIELDNAMES = [ "file_path", @@ -48,17 +56,15 @@ def _project_root() -> str: - """ - Return the absolute path to the project root directory, inferred from the location of this module on disk. - """ + """Return the absolute project root path, inferred from this module's location on disk.""" return os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir)) def _candidate_paths(primary: str, legacy: str | None = None) -> list[str]: - """ - Build an ordered list of file paths to try for a given name, including the - original path, a project-root-relative variant, and an optional legacy - filename, while removing duplicates. + """Return the ordered, deduplicated paths to try for a file. + + Includes the original path, a project-root-relative variant, and an + optional legacy filename. """ candidates: list[str] = [primary] if not os.path.isabs(primary): @@ -73,10 +79,7 @@ def _candidate_paths(primary: str, legacy: str | None = None) -> list[str]: def _read_key_file( path: str, legacy: str | None = None, required: bool = True, expected_lines: int = 1 ) -> list[str] | None: - """ - Generic key file reader that handles common patterns for loading API keys and - credentials from configuration files with optional fallback to legacy filenames. - """ + """Read an API key or credentials file, with optional fallback to a legacy filename.""" candidates = _candidate_paths(path, legacy) last_err: Exception | None = None @@ -103,56 +106,37 @@ def _read_key_file( def read_semantic_api_key(path: str = DEFAULT_S2_KEY_FILE) -> str | None: - """ - Look for a Semantic Scholar API key in the usual locations and return it if - present, or None when no key file is found. - """ + """Return the Semantic Scholar API key, or None when no key file is found.""" lines = _read_key_file(path, required=False, expected_lines=1) return lines[0] if lines else None def read_openreview_credentials(path: str = DEFAULT_OR_KEY_FILE) -> tuple[str, str] | None: - """ - Read OpenReview credentials from a small text file where the first non-empty - line is the username and the second is the password, returning them as a - tuple. - """ + """Return the OpenReview (username, password) tuple, or None when no key file is found.""" lines = _read_key_file(path, legacy=None, required=False, expected_lines=2) return (lines[0], lines[1]) if lines else None def read_serpapi_api_key(path: str = DEFAULT_SERPAPI_KEY_FILE) -> str | None: - """ - Look for a SerpAPI key in the usual locations and return it if - present, or None when no key file is found. - """ + """Return the SerpAPI key, or None when no key file is found.""" lines = _read_key_file(path, required=False, expected_lines=1) return lines[0] if lines else None def read_serply_api_key(path: str = DEFAULT_SERPLY_KEY_FILE) -> str | None: - """ - Look for a Serply API key in the usual locations and return it if - present, or None when no key file is found. - """ + """Return the Serply API key, or None when no key file is found.""" lines = _read_key_file(path, required=False, expected_lines=1) return lines[0] if lines else None def read_gemini_api_key(path: str = DEFAULT_GEMINI_KEY_FILE) -> str | None: - """ - Look for a Gemini API key in the usual locations and return it if - present, or None when no key file is found. - """ + """Return the Gemini API key, or None when no key file is found.""" lines = _read_key_file(path, required=False, expected_lines=1) return lines[0] if lines else None def read_records(path: str = DEFAULT_INPUT) -> list[Record]: - """ - Load author records from a CSV file, skip empty rows, and keep only entries - with at least one valid identifier (Scholar or DBLP). - """ + """Load author records from a CSV file, keeping only rows with a Scholar or DBLP identifier.""" records: list[Record] = [] candidates = _candidate_paths(path) for p in candidates: @@ -206,9 +190,7 @@ def read_records(path: str = DEFAULT_INPUT) -> list[Record]: def safe_read_file(path: str, encoding: str = "utf-8") -> str | None: - """ - Safely read a file and return its contents, returning None on error. - """ + """Read a file and return its contents, or None on error.""" try: with open(path, encoding=encoding) as f: return f.read() @@ -217,9 +199,7 @@ def safe_read_file(path: str, encoding: str = "utf-8") -> str | None: def safe_read_json(path: str, default: Any = None) -> Any: - """ - Safely read a JSON file and return its parsed contents, returning a default value on error. - """ + """Read a JSON file and return its parsed contents, or *default* on error.""" try: with open(path, encoding="utf-8") as f: return json.load(f) @@ -239,9 +219,7 @@ def _ensure_parent_dirs(path: str) -> bool: def safe_write_file(path: str, content: str, encoding: str = "utf-8", makedirs: bool = True) -> bool: - """ - Safely write content to a file, optionally creating parent directories. - """ + """Write *content* to a file, optionally creating parent directories. Returns False on error.""" if makedirs and not _ensure_parent_dirs(path): return False @@ -254,9 +232,7 @@ def safe_write_file(path: str, content: str, encoding: str = "utf-8", makedirs: def safe_write_json(path: str, data: Any, makedirs: bool = True, indent: int | None = 2) -> bool: - """ - Safely write data to a JSON file, optionally creating parent directories. - """ + """Write *data* to a JSON file, optionally creating parent directories. Returns False on error.""" if makedirs and not _ensure_parent_dirs(path): return False @@ -273,8 +249,8 @@ def safe_write_json(path: str, data: Any, makedirs: bool = True, indent: int | N def init_summary_csv(csv_path: str, preserve_existing: bool = False) -> None: - """ - Initialize the summary CSV file with proper headers, creating the parent directory if needed. + """Initialize the summary CSV with headers, creating the parent directory if needed. + Loads existing entries into memory for O(1) dedup on appends. """ parent_dir = os.path.dirname(csv_path) @@ -308,10 +284,10 @@ def is_known_summary_path(file_path: str) -> bool: def append_summary_to_csv(csv_path: str, file_path: str, trust_hits: int, flags: dict[str, bool]) -> None: - """ - Append a summary row to the CSV file. New entries are appended in O(1). - Updated entries (same file_path) are tracked in memory and flushed at end of run. - Thread-safe via _CSV_LOCK. + """Append a summary row to the CSV file. + + New entries are appended in O(1). Updated entries (same file_path) are + tracked in memory and flushed at end of run. Thread-safe via _CSV_LOCK. """ new_row: dict[str, Any] = {"file_path": file_path, "trust_hits": trust_hits} new_row.update({f: int(bool(flags.get(f))) for f in _SUMMARY_CSV_FLAG_FIELDS}) @@ -330,9 +306,9 @@ def append_summary_to_csv(csv_path: str, file_path: str, trust_hits: int, flags: def flush_summary_csv(csv_path: str) -> None: - """ - Rewrite the summary CSV only if updates to existing entries occurred during the run. - Called once at the end of main(). + """Rewrite the summary CSV only when existing entries were updated during the run. + + Called once from the post-run tail. """ with _CSV_LOCK: if not _SUMMARY_UPDATES: @@ -360,14 +336,14 @@ def flush_summary_csv(csv_path: str) -> None: def collect_orphan_files(csv_path: str, output_dir: str) -> list[str]: - """ - Return absolute paths of .bib files on disk that have no entry in the - summary CSV. These are stale leftovers from previous runs where the same - article received a different citation key (e.g. Gemini returned a different + """Return absolute paths of .bib files on disk that have no entry in the summary CSV. + + These are stale leftovers from previous runs where the same article + received a different citation key (e.g. Gemini returned a different short title). Called after :func:`reconcile_summary_csv` so that phantom entries have - already been stripped -- any remaining CSV entry corresponds to a real file. + already been stripped, so any remaining CSV entry corresponds to a real file. """ # NOTE: CSV stores paths relative to CWD (e.g. "output/Author/file.bib"). # os.path.abspath() resolves them correctly when CWD is the project root. @@ -382,19 +358,16 @@ def collect_orphan_files(csv_path: str, output_dir: str) -> list[str]: orphans: list[str] = [] try: - subdirs = [ - os.path.join(output_dir, d) for d in os.listdir(output_dir) if os.path.isdir(os.path.join(output_dir, d)) - ] + subdirs = [os.path.join(output_dir, d) for d in iter_output_dirs(output_dir)] except OSError: return [] for subdir in subdirs: try: - for fname in os.listdir(subdir): - if fname.endswith(".bib"): - abs_path = os.path.abspath(os.path.join(subdir, fname)) - if abs_path not in csv_paths: - orphans.append(abs_path) + for fname in iter_author_bibs(subdir): + abs_path = os.path.abspath(os.path.join(subdir, fname)) + if abs_path not in csv_paths: + orphans.append(abs_path) except OSError: continue @@ -402,8 +375,7 @@ def collect_orphan_files(csv_path: str, output_dir: str) -> list[str]: def reconcile_summary_csv(csv_path: str) -> int: - """ - Remove CSV rows whose file no longer exists on disk (phantom entries). + """Remove CSV rows whose file no longer exists on disk (phantom entries). FILE_CLEANUP in save_entry_to_file can delete/rename files that already have CSV entries; the CSV is append-only so stale rows accumulate. @@ -457,12 +429,6 @@ def build_a2i2_folder( Returns the number of .bib files written, or 0 if ``a2i2_csv_path`` is missing. """ - from . import bibtex_utils as bt - from .clients.helpers import get_current_year - from .config import A2I2_OUTPUT_DIR, SIM_MERGE_DUPLICATE_THRESHOLD, get_min_year - from .id_utils import doi_bases_match, normalize_doi - from .text_utils import format_author_dirname, normalize_person_name, title_similarity - log = logging.getLogger("CiteForge.io") # --- Guard: skip if CSV missing ----------------------------------------- @@ -507,9 +473,7 @@ def build_a2i2_folder( if not os.path.isdir(author_dir): continue - for fname in sorted(os.listdir(author_dir)): - if not fname.endswith(".bib"): - continue + for fname in iter_author_bibs(author_dir): fpath = os.path.join(author_dir, fname) try: with open(fpath, encoding="utf-8") as bf: @@ -585,7 +549,7 @@ def _pick_richer( # --- Step 5: clear and rebuild a2i2 directory --------------------------- a2i2_dir = os.path.join(out_dir, A2I2_OUTPUT_DIR) os.makedirs(a2i2_dir, exist_ok=True) - for fname in os.listdir(a2i2_dir): + for fname in sorted(os.listdir(a2i2_dir)): fpath = os.path.join(a2i2_dir, fname) if os.path.isfile(fpath): os.remove(fpath) diff --git a/citeforge/log_utils.py b/citeforge/log_utils.py index 4417cbb0..7905a28d 100644 --- a/citeforge/log_utils.py +++ b/citeforge/log_utils.py @@ -221,7 +221,7 @@ def __init__(self) -> None: self._logger.propagate = False self._logger.handlers.clear() - # Console handler -- main thread only + # Console handler, main thread only self._console_handler = logging.StreamHandler(sys.stdout) self._console_handler.setLevel(logging.INFO) self._console_handler.addFilter(MainThreadFilter()) @@ -237,21 +237,6 @@ def __init__(self) -> None: self._logger.addHandler(self._tl_handler) self._adapter = CategoryAdapter(self._logger, {}) - self._add_custom_methods() - - def _add_custom_methods(self) -> None: - """Register STEP and SUCCESS level methods on the underlying logger.""" - - def step_method(msg: str, *args: Any, **kwargs: Any) -> None: - if self._logger.isEnabledFor(STEP_LEVEL): - self._logger._log(STEP_LEVEL, msg, args, **kwargs) - - def success_method(msg: str, *args: Any, **kwargs: Any) -> None: - if self._logger.isEnabledFor(SUCCESS_LEVEL): - self._logger._log(SUCCESS_LEVEL, msg, args, **kwargs) - - self._logger.step = step_method # type: ignore[attr-defined] - self._logger.success = success_method # type: ignore[attr-defined] def set_log_file(self, path: str) -> None: """Start mirroring all log messages to the specified file for the current thread.""" diff --git a/citeforge/merge_utils.py b/citeforge/merge_utils.py index fc424ed0..7a09bb40 100644 --- a/citeforge/merge_utils.py +++ b/citeforge/merge_utils.py @@ -13,6 +13,7 @@ import html import os import re +from collections.abc import Callable from typing import Any from .bibtex_build import determine_entry_type, get_container_field @@ -33,7 +34,7 @@ TRUST_DIFF_OVERRIDE_THRESHOLD, TRUST_ORDER, ) -from .fsscan import iter_author_bibs +from .fsscan import iter_author_bibs, iter_parsed_author_bibs from .id_utils import ( _norm_doi, allowlisted_url, @@ -55,11 +56,11 @@ title_is_truncated_match, title_similarity, ) + +# _matches_journal_named_proceedings and infer_howpublished_from_doi are +# re-exported for canonicalize.py and tests that access them as merge_utils.*. from .venue import ( # noqa: F401 _DAGSTUHL_DOI_RE, - _DOI_PREFIX_TO_HOWPUB, - _HOWPUB_CANONICAL, - _OSF_PREPRINTS, _is_conference_journal, _matches_journal_named_proceedings, _normalize_howpublished, @@ -70,6 +71,16 @@ _AUTHOR_PAREN_SUFFIX_RE = re.compile(r"\s*\(\d{1,4}\)\s*$") _AUTHOR_GLUED_DIGIT_RE = re.compile(r"(?<=[A-Za-z]{2})\d{1,4}$") _AUTHOR_INITIAL_RE = re.compile(r"^[A-Z]\.$") +_AND_AND_RE = re.compile(r"\band\s+And\b") + +_LEADING_DIGIT_RE = re.compile(r"^\d") +_PAGE_PART_SPLIT_RE = re.compile(r"[-\u2013\u2014,\s]+") +_NON_DIGIT_RE = re.compile(r"\D") +_PAGE_LEADING_ZERO_RE = re.compile(r"\b0+(\d)") +_WHITESPACE_RE = re.compile(r"\s+") +_HTML_TAG_RE = re.compile(r"<[^>]+>") +_PREPRINT_JOURNAL_SUFFIX_RE = re.compile(r"\s*:\s*the preprint server for [\w\s]+$", re.IGNORECASE) +_TITLE_CAP_WORD_RE = re.compile(r"[A-Z][a-z]+") _JOURNAL_URL_MAP: dict[str, str] = { "techrxiv.org": "TechRxiv", @@ -78,16 +89,32 @@ _AUTHOR_SEPARATOR = " and " +# URL substrings that mark a stale preprint URL on an entry that gained a +# published DOI. Checked in merge_with_policy when removing eprint fields. +_PREPRINT_URL_MARKERS = ( + "arxiv.org", + "biorxiv.org", + "medrxiv.org", + "ssrn.com", + "preprints.org", + "techrxiv.org", + "researchsquare.com", + "10.1101/", + "10.2139/", + "10.20944/", + "10.21203/", +) + def _fix_author_casing(author_val: str) -> tuple[str, bool]: - """Fix author name casing: capitalize all-lowercase tokens, convert - ALL-CAPS tokens (>2 chars) to title case, fix 2-char ALL-CAPS surnames - when preceded by a mixed-case given name, and fix capital 'And' separators. + """Fix author name casing. Capitalizes all-lowercase tokens, converts + ALL-CAPS tokens (>2 chars) to title case, fixes 2-char ALL-CAPS surnames + when preceded by a mixed-case given name, and fixes capital 'And' separators. Returns (fixed_string, was_modified). """ - # Fix capital "And" separator first (e.g. "and And Duncan" → "and Duncan") - val = re.sub(r"\band\s+And\b", "and", author_val) + # Fix capital "And" separator first (e.g. "and And Duncan" becomes "and Duncan") + val = _AND_AND_RE.sub("and", author_val) val = val.replace(" And ", _AUTHOR_SEPARATOR) and_was_fixed = val != author_val parts = [p.strip() for p in val.split(_AUTHOR_SEPARATOR)] @@ -146,11 +173,192 @@ def _is_preprint_doi(doi: str) -> bool: def _pop_fields(target: dict[str, Any], field_names: set[str] | frozenset[str], log_tag: str) -> None: """Remove *field_names* from *target*, logging any that were actually present.""" _sentinel = object() - removed = [f for f in field_names if target.pop(f, _sentinel) is not _sentinel] + removed = [f for f in sorted(field_names) if target.pop(f, _sentinel) is not _sentinel] if removed: logger.debug(f"{log_tag} | fields={removed}", category=LogCategory.CLEANUP) +def _invalid_pages_reason(pages_str: str) -> tuple[str, int] | None: + """Classify an invalid pages value, or return None when it looks like real pages. + + Pages must start with a digit, must not contain dots (manuscript IDs such as + "2025.11.07.685935"), and no component may exceed PAGES_MAX_DIGITS digits + (SAGE/Wiley article IDs masquerading as pages). Components are checked + individually so ranges like "13905-13917" pass. Returns (code, digits) with + code in {"no_leading_digit", "contains_dot", "overflow"}; digits is the + digit count of the first overflowing component (0 otherwise). + """ + if not _LEADING_DIGIT_RE.match(pages_str): + return ("no_leading_digit", 0) + if "." in pages_str: + return ("contains_dot", 0) + parts = _PAGE_PART_SPLIT_RE.split(pages_str) + overflow = [p for p in parts if p.strip() and len(_NON_DIGIT_RE.sub("", p)) > PAGES_MAX_DIGITS] + if overflow: + return ("overflow", len(_NON_DIGIT_RE.sub("", overflow[0]))) + return None + + +# Per-field merge guards. Each guard sees the incumbent value, the candidate +# value, both source names, and the trust ranking, and returns +# True -> accept the candidate regardless of rank, +# False -> reject the candidate regardless of rank, +# None -> fall through to the plain rank comparison. +def _guard_doi(cur: Any, v: Any, cur_src: str, src: str, type_rank: dict[str, int]) -> bool | None: + """Prefer published DOIs over preprint DOIs in both directions.""" + cur_is_preprint = _is_preprint_doi(str(cur)) + new_is_preprint = _is_preprint_doi(str(v)) + if cur_is_preprint and not new_is_preprint: + logger.debug( + f"DOI_UPGRADE | preprint->published | old={cur} | new={v} | src={src}", + category=LogCategory.MERGE, + ) + return True + if not cur_is_preprint and new_is_preprint: + logger.debug( + f"DOI_KEEP | published beats preprint | cur={cur} | rejected={v} | src={src}", + category=LogCategory.MERGE, + ) + return False + return None + + +def _guard_pages(cur: Any, v: Any, cur_src: str, src: str, type_rank: dict[str, int]) -> bool | None: + """Reject candidate pages that are not actual page numbers.""" + reason = _invalid_pages_reason(str(v).strip()) + if reason is not None: + code, digits = reason + msg = f"component_too_long({digits}digits)" if code == "overflow" else code + logger.debug(f"PAGES_REJECT | val={v} | reason={msg}", category=LogCategory.MERGE) + return False + return None + + +def _guard_journal(cur: Any, v: Any, cur_src: str, src: str, type_rank: dict[str, int]) -> bool | None: + """Never downgrade a published journal to a preprint server.""" + # Deliberately narrower than the canonical fields-level predicate + # text_utils._is_preprint_fields: this guard sees two bare venue strings, + # not fields dicts, and must not consult a DOI (DOI preference is + # _guard_doi's job; consulting it here would double-apply the DOI rule). + # Only the journal-substring branch of that predicate applies. + cur_is_preprint = any(ps in str(cur).lower() for ps in PREPRINT_SERVERS) + new_is_preprint = any(ps in str(v).lower() for ps in PREPRINT_SERVERS) + if not cur_is_preprint and new_is_preprint: + logger.debug( + f"JOURNAL_KEEP | published beats preprint | cur={cur} | rejected={v} | src={src}", + category=LogCategory.MERGE, + ) + return False + return None + + +def _guard_author(cur: Any, v: Any, cur_src: str, src: str, type_rank: dict[str, int]) -> bool | None: + """Prefer more complete (less abbreviated) author lists. + + Never drop authors. A truncated (shorter) list must not overwrite a more + complete one unless the new source is much more trusted (>= the title + rule's TRUST_DIFF_OVERRIDE_THRESHOLD). Mirrors the title-length guard so + a source that emits "A. Smith and others" cannot shrink a full list by rank. + """ + cur_parts = parse_authors_any(str(cur)) + new_parts = parse_authors_any(str(v)) + if cur_parts and new_parts and len(new_parts) < len(cur_parts): + trust_diff = type_rank.get(cur_src, 99) - type_rank.get(src, 99) + if trust_diff < TRUST_DIFF_OVERRIDE_THRESHOLD: + logger.debug( + f"AUTHOR_KEEP_SUPERSET | cur_n={len(cur_parts)} new_n={len(new_parts)} " + f"trust_diff={trust_diff} | src={src} | keeping more complete author list", + category=LogCategory.MERGE, + ) + return False + if cur_parts and new_parts and len(cur_parts) == len(new_parts): + # Count initials-only tokens (e.g. "J." but not "Jr.") + cur_inits = sum(1 for name in cur_parts for tok in name.split() if _AUTHOR_INITIAL_RE.match(tok)) + new_inits = sum(1 for name in new_parts for tok in name.split() if _AUTHOR_INITIAL_RE.match(tok)) + if new_inits > cur_inits: + logger.debug( + f"AUTHOR_KEEP_COMPLETE | cur_initials={cur_inits} new_initials={new_inits} " + f"| src={src} | keeping more complete names", + category=LogCategory.MERGE, + ) + return False + # Also prefer the version with longer total author text + # (catches "Samuel" vs "Sam", middle initials dropped, etc.) + if new_inits == cur_inits: + cur_len = sum(len(n) for n in cur_parts) + new_len = sum(len(n) for n in new_parts) + if new_len < cur_len: + logger.debug( + f"AUTHOR_KEEP_LONGER | cur_len={cur_len} new_len={new_len} " + f"| src={src} | keeping longer author names", + category=LogCategory.MERGE, + ) + return False + return None + + +def _guard_title(cur: Any, v: Any, cur_src: str, src: str, type_rank: dict[str, int]) -> bool | None: + """Prefer longer, more descriptive titles. + + Compares content length without whitespace so OCR artifacts (e.g. "Un met" + vs "Unmet") do not give the broken title a false length advantage. A + significantly shorter candidate (< TITLE_LENGTH_KEEP_RATIO of the current + length) only replaces when its source is at least + TRUST_DIFF_OVERRIDE_THRESHOLD positions higher in trust order. + """ + cur_len = len(_WHITESPACE_RE.sub("", str(cur))) + new_len = len(_WHITESPACE_RE.sub("", str(v))) + if cur_len > 0 and new_len < (cur_len * TITLE_LENGTH_KEEP_RATIO): + trust_diff = type_rank.get(cur_src, 99) - type_rank.get(src, 99) + if trust_diff < TRUST_DIFF_OVERRIDE_THRESHOLD: + logger.debug( + f"TITLE_KEEP_LONGER | cur_len={cur_len} new_len={new_len} " + f"ratio={new_len / cur_len:.2f} trust_diff={trust_diff}", + category=LogCategory.MERGE, + ) + return False + return None + + +def _guard_booktitle(cur: Any, v: Any, cur_src: str, src: str, type_rank: dict[str, int]) -> bool | None: + """Prefer a specific conference name over a generic series name. + + A generic-to-specific upgrade is accepted only when the specific value is + not much less trusted than the incumbent, so a low-trust source cannot + overwrite a trusted generic by specificity alone. When it is far less + trusted, fall through to the rank gate (which keeps the more-trusted + generic). The reverse (specific replaced by generic) is always rejected. + """ + cur_lower = str(cur).lower().strip() + new_lower = str(v).lower().strip() + if cur_lower in GENERIC_SERIES_NAMES and new_lower not in GENERIC_SERIES_NAMES: + trust_diff = type_rank.get(cur_src, 99) - type_rank.get(src, 99) + if trust_diff > -TRUST_DIFF_OVERRIDE_THRESHOLD: + logger.debug( + f"BOOKTITLE_UPGRADE | generic->specific | old={str(cur)[:60]} " + f"| new={str(v)[:60]} | src={src} | trust_diff={trust_diff}", + category=LogCategory.MERGE, + ) + return True + if cur_lower not in GENERIC_SERIES_NAMES and new_lower in GENERIC_SERIES_NAMES: + logger.debug( + f"BOOKTITLE_KEEP | specific beats generic | cur={str(cur)[:60]} | rejected={str(v)[:60]} | src={src}", + category=LogCategory.MERGE, + ) + return False + return None + + +_FIELD_GUARDS: dict[str, Callable[[Any, Any, str, str, dict[str, int]], bool | None]] = { + "author": _guard_author, + "booktitle": _guard_booktitle, + "doi": _guard_doi, + "journal": _guard_journal, + "pages": _guard_pages, + "title": _guard_title, +} + + def merge_with_policy(primary: dict[str, Any], enrichers: list[tuple[str, dict[str, Any]]]) -> dict[str, Any]: """ Combine a baseline BibTeX entry with metadata from multiple sources by @@ -218,156 +426,16 @@ def value_ok(val: str | None) -> bool: ) continue - # special handling for DOI field: prefer published DOIs over preprint DOIs - if k == "doi": - cur_is_preprint = _is_preprint_doi(str(cur)) - new_is_preprint = _is_preprint_doi(str(v)) - # if current is preprint DOI but new one isn't, always prefer published - if cur_is_preprint and not new_is_preprint: - logger.debug( - f"DOI_UPGRADE | preprint->published | old={cur} | new={v} | src={src}", - category=LogCategory.MERGE, - ) + # Field-specific guards (DOI, pages, journal, author, title, + # booktitle) may accept or reject the candidate regardless of rank. + guard = _FIELD_GUARDS.get(k) + if guard is not None: + decision = guard(cur, v, cur_src, src, type_rank) + if decision is True: merged[k] = v field_sources[k] = src continue - # if new is preprint DOI but current isn't, keep current - if not cur_is_preprint and new_is_preprint: - logger.debug( - f"DOI_KEEP | published beats preprint | cur={cur} | rejected={v} | src={src}", - category=LogCategory.MERGE, - ) - continue - - # special handling for pages field: must be actual page numbers only - if k == "pages": - new_str = str(v).strip() - # Validate: pages must start with a digit (page numbers only) - if not re.match(r"^\d", new_str): - logger.debug(f"PAGES_REJECT | val={v} | reason=no_leading_digit", category=LogCategory.MERGE) - continue - # Reject manuscript IDs containing dots (e.g., "2025.11.07.685935") - if "." in new_str: - logger.debug(f"PAGES_REJECT | val={v} | reason=contains_dot", category=LogCategory.MERGE) - continue - # Reject article IDs masquerading as pages (SAGE/Wiley use long numeric IDs) - # Check each page component individually so ranges like "13905-13917" pass - parts = re.split(r"[-\u2013\u2014,\s]+", new_str) - overflow = [p for p in parts if p.strip() and len(re.sub(r"\D", "", p)) > PAGES_MAX_DIGITS] - if overflow: - digits = len(re.sub(r"\D", "", overflow[0])) - logger.debug( - f"PAGES_REJECT | val={v} | reason=component_too_long({digits}digits)", - category=LogCategory.MERGE, - ) - continue - - # special handling for journal field: never downgrade from published journal to preprint server - if k == "journal": - cur_journal_lower = str(cur).lower() - new_journal_lower = str(v).lower() - - cur_is_preprint = any(ps in cur_journal_lower for ps in PREPRINT_SERVERS) - new_is_preprint = any(ps in new_journal_lower for ps in PREPRINT_SERVERS) - - # Never replace a published journal with a preprint server - if not cur_is_preprint and new_is_preprint: - logger.debug( - f"JOURNAL_KEEP | published beats preprint | cur={cur} | rejected={v} | src={src}", - category=LogCategory.MERGE, - ) - continue - - # special handling for author field: prefer more complete (less abbreviated) names - if k == "author": - cur_parts = parse_authors_any(str(cur)) - new_parts = parse_authors_any(str(v)) - # Never drop authors: a truncated (shorter) list must not overwrite a more - # complete one unless the new source is much more trusted (>= the title - # rule's TRUST_DIFF_OVERRIDE_THRESHOLD). Mirrors the title-length guard so - # a source that emits "A. Smith and others" cannot shrink a full list by rank. - if cur_parts and new_parts and len(new_parts) < len(cur_parts): - trust_diff = type_rank.get(cur_src, 99) - type_rank.get(src, 99) - if trust_diff < TRUST_DIFF_OVERRIDE_THRESHOLD: - logger.debug( - f"AUTHOR_KEEP_SUPERSET | cur_n={len(cur_parts)} new_n={len(new_parts)} " - f"trust_diff={trust_diff} | src={src} | keeping more complete author list", - category=LogCategory.MERGE, - ) - continue - if cur_parts and new_parts and len(cur_parts) == len(new_parts): - # Count initials-only tokens (e.g. "J." but not "Jr.") - cur_inits = sum(1 for name in cur_parts for tok in name.split() if _AUTHOR_INITIAL_RE.match(tok)) - new_inits = sum(1 for name in new_parts for tok in name.split() if _AUTHOR_INITIAL_RE.match(tok)) - if new_inits > cur_inits: - logger.debug( - f"AUTHOR_KEEP_COMPLETE | cur_initials={cur_inits} new_initials={new_inits} " - f"| src={src} | keeping more complete names", - category=LogCategory.MERGE, - ) - continue - # Also prefer the version with longer total author text - # (catches "Samuel" vs "Sam", middle initials dropped, etc.) - if new_inits == cur_inits: - cur_len = sum(len(n) for n in cur_parts) - new_len = sum(len(n) for n in new_parts) - if new_len < cur_len: - logger.debug( - f"AUTHOR_KEEP_LONGER | cur_len={cur_len} new_len={new_len} " - f"| src={src} | keeping longer author names", - category=LogCategory.MERGE, - ) - continue - - # special handling for title field: prefer longer, more descriptive titles - if k == "title": - # Compare content length without whitespace so OCR artifacts - # (e.g., "Un met" vs "Unmet") don't give the broken title a - # false length advantage. - cur_len = len(re.sub(r"\s+", "", str(cur))) - new_len = len(re.sub(r"\s+", "", str(v))) - - # If new title is significantly shorter (< 70% of current length), - # only replace if it comes from a MUCH more trusted source - # (at least 3 positions higher in trust order) - if cur_len > 0 and new_len < (cur_len * TITLE_LENGTH_KEEP_RATIO): - trust_diff = type_rank.get(cur_src, 99) - type_rank.get(src, 99) - if trust_diff < TRUST_DIFF_OVERRIDE_THRESHOLD: - # New source isn't significantly more trusted, keep longer title - logger.debug( - f"TITLE_KEEP_LONGER | cur_len={cur_len} new_len={new_len} " - f"ratio={new_len / cur_len:.2f} trust_diff={trust_diff}", - category=LogCategory.MERGE, - ) - continue - - # special handling for booktitle: prefer specific conference name over generic series - if k == "booktitle": - cur_lower = str(cur).lower().strip() - new_lower = str(v).lower().strip() - # If current is a generic series name and new is more specific, accept the - # upgrade -- but only when the specific value is not much less trusted than - # the incumbent, so a low-trust source cannot overwrite a trusted generic by - # specificity alone. When it is far less trusted, fall through to the rank - # gate (which keeps the more-trusted generic). - if cur_lower in GENERIC_SERIES_NAMES and new_lower not in GENERIC_SERIES_NAMES: - trust_diff = type_rank.get(cur_src, 99) - type_rank.get(src, 99) - if trust_diff > -TRUST_DIFF_OVERRIDE_THRESHOLD: - logger.debug( - f"BOOKTITLE_UPGRADE | generic->specific | old={str(cur)[:60]} " - f"| new={str(v)[:60]} | src={src} | trust_diff={trust_diff}", - category=LogCategory.MERGE, - ) - merged[k] = v - field_sources[k] = src - continue - # Never replace a specific conference name with a generic series - if cur_lower not in GENERIC_SERIES_NAMES and new_lower in GENERIC_SERIES_NAMES: - logger.debug( - f"BOOKTITLE_KEEP | specific beats generic | cur={str(cur)[:60]} " - f"| rejected={str(v)[:60]} | src={src}", - category=LogCategory.MERGE, - ) + if decision is False: continue # only replace if new source is more trustworthy @@ -398,7 +466,8 @@ def value_ok(val: str | None) -> bool: logger.debug("doi_remove | invalid after normalization", category=LogCategory.CLEANUP) merged.pop("doi", None) - # Validate DOI consistency: contradicting DOIs indicate different papers + # Contradicting primary/merged DOIs indicate different papers; keep the + # primary unless the merge upgraded a preprint DOI to a published one. primary_doi = _norm_doi(primary.get("fields", {}).get("doi")) has_doi_conflict = False merged_doi_norm = _norm_doi(merged.get("doi")) @@ -457,8 +526,8 @@ def value_ok(val: str | None) -> bool: merged = normalize_arxiv_metadata(merged) _pop_fields(merged, {"keywords", "copyright"}, "unwanted_removed") - # Sanitize author names: strip trailing digit suffixes (e.g., "Das1" -> "Das") - # that leak from Scholar/DBLP author disambiguation markers + # Strip trailing digit suffixes leaked from Scholar/DBLP author + # disambiguation markers (e.g., "Das1" becomes "Das"). author_val = merged.get("author", "") if author_val: author_parts = [p.strip() for p in str(author_val).split(_AUTHOR_SEPARATOR)] @@ -470,8 +539,8 @@ def value_ok(val: str | None) -> bool: category=LogCategory.CLEANUP, ) - # Fix author name casing: capitalize all-lowercase tokens AND convert - # ALL-CAPS tokens to title case (catches "darren steeves", "F VARNO"). + # Fix author name casing, covering all-lowercase tokens and ALL-CAPS + # tokens (catches "darren steeves", "F VARNO"). author_val = merged.get("author", "") if author_val: fixed_author, author_was_fixed = _fix_author_casing(str(author_val)) @@ -485,19 +554,18 @@ def value_ok(val: str | None) -> bool: pages_val = merged.get("pages", "") if pages_val: pages_str = str(pages_val).strip() - if not re.match(r"^\d", pages_str) or "." in pages_str: + invalid = _invalid_pages_reason(pages_str) + if invalid is not None and invalid[0] != "overflow": logger.debug(f"pages_remove | val={pages_str} | reason=invalid_format", category=LogCategory.CLEANUP) merged.pop("pages", None) + elif invalid is not None: + logger.debug(f"pages_remove | val={pages_str} | reason=digit_overflow", category=LogCategory.CLEANUP) + merged.pop("pages", None) else: - parts = re.split(r"[-\u2013\u2014,\s]+", pages_str) - if any(len(re.sub(r"\D", "", p)) > PAGES_MAX_DIGITS for p in parts if p.strip()): - logger.debug(f"pages_remove | val={pages_str} | reason=digit_overflow", category=LogCategory.CLEANUP) - merged.pop("pages", None) - else: - cleaned_pages = re.sub(r"\b0+(\d)", r"\1", pages_str) - if cleaned_pages != pages_str: - logger.debug(f"pages_leading_zeros | {pages_str}->{cleaned_pages}", category=LogCategory.CLEANUP) - merged["pages"] = cleaned_pages + cleaned_pages = _PAGE_LEADING_ZERO_RE.sub(r"\1", pages_str) + if cleaned_pages != pages_str: + logger.debug(f"pages_leading_zeros | {pages_str}->{cleaned_pages}", category=LogCategory.CLEANUP) + merged["pages"] = cleaned_pages # Remove volume if it equals year (common conference proceedings error) year_val = merged.get("year", "") @@ -512,7 +580,7 @@ def value_ok(val: str | None) -> bool: # Strip " : the preprint server for X" suffixes added by PubMed/Europe PMC journal_val = merged.get("journal", "") if journal_val: - journal_cleaned = re.sub(r"\s*:\s*the preprint server for [\w\s]+$", "", journal_val, flags=re.IGNORECASE) + journal_cleaned = _PREPRINT_JOURNAL_SUFFIX_RE.sub("", journal_val) if journal_cleaned != journal_val: logger.debug( f"journal_preprint_suffix | {journal_val}->{journal_cleaned.strip()}", @@ -543,7 +611,7 @@ def value_ok(val: str | None) -> bool: ) merged["publisher"] = "Cold Spring Harbor Laboratory" - # Apply journal-specific publisher corrections (e.g. SAGE → Mary Ann Liebert for JCB) + # Apply journal-specific publisher corrections (e.g. SAGE to Mary Ann Liebert for JCB) if journal_lower: for _jnl_key, _correct_pub in PUBLISHER_CORRECTIONS.items(): if _jnl_key in journal_lower: @@ -577,7 +645,7 @@ def value_ok(val: str | None) -> bool: field_val = merged.get(field, "") if field_val and isinstance(field_val, str): cleaned = html.unescape(field_val) - cleaned = re.sub(r"<[^>]+>", "", cleaned) + cleaned = _HTML_TAG_RE.sub("", cleaned) if cleaned != field_val: logger.debug(f"html_decode | field={field} | changed=True", category=LogCategory.CLEANUP) merged[field] = cleaned.strip() @@ -612,22 +680,7 @@ def value_ok(val: str | None) -> bool: merged.pop("primaryclass", None) # Update URL to match the published DOI (remove stale preprint URLs) _current_url = (merged.get("url") or "").lower() - if _current_url and any( - ps in _current_url - for ps in ( - "arxiv.org", - "biorxiv.org", - "medrxiv.org", - "ssrn.com", - "preprints.org", - "techrxiv.org", - "researchsquare.com", - "10.1101/", - "10.2139/", - "10.20944/", - "10.21203/", - ) - ): + if _current_url and any(ps in _current_url for ps in _PREPRINT_URL_MARKERS): merged["url"] = f"https://doi.org/{doi_val}" logger.debug( f"url_preprint_replaced | old={_current_url[:60]} | new={merged['url']}", @@ -643,8 +696,8 @@ def value_ok(val: str | None) -> bool: category=LogCategory.CLEANUP, ) merged.pop("journal", None) - # Backfill from enrichers when journal is missing: find the best-ranked - # source that carries a real (non-preprint) journal matching the published DOI. + # When the journal is missing, backfill it from the best-ranked source + # carrying a real (non-preprint) journal matching the published DOI. if not merged.get("journal"): doi_norm_check = _norm_doi(doi_val) best_journal: str | None = None @@ -705,7 +758,7 @@ def value_ok(val: str | None) -> bool: # Fix conference proceedings misclassified as @article with journal. # Crossref registers some conference proceedings (AAAI, EMNLP, PVLDB, # PACMHCI, etc.) as journal volumes. Any venue named "Proceedings of..." - # is proceedings, not a journal — reclassify as @inproceedings. + # is proceedings, not a journal, so reclassify as @inproceedings. if etype == "article" and merged.get("journal") and not merged.get("booktitle"): jnl_for_conf = merged["journal"].strip() if _is_conference_journal(jnl_for_conf): @@ -741,8 +794,8 @@ def value_ok(val: str | None) -> bool: category=LogCategory.CLEANUP, ) else: - # Festschrift or unknown series: conf abbrev not in map but DOI - # confirms it's a Dagstuhl proceedings publication (LIPIcs/OASIcs) + # Festschrift or unknown series. The conf abbrev is not in the map + # but the DOI confirms a Dagstuhl proceedings publication (LIPIcs/OASIcs) etype = "inproceedings" if old_journal and not merged.get("booktitle"): merged["booktitle"] = old_journal @@ -876,6 +929,205 @@ def value_ok(val: str | None) -> bool: return {"type": etype, "key": primary.get("key"), "fields": merged} +def _is_duplicate_bib_entry( + existing_entry: dict[str, Any], + entry: dict[str, Any], + new_doi: str, + existing_filename: str, +) -> bool: + """Decide whether *existing_entry* (from *existing_filename*) and *entry* + describe the same publication. + + Match ladder, in order. Exact DOI, DOI version variants, preprint/published + DOI pairs (XOR), external IDs, citation-key matches (title- or + author-verified), high title similarity, truncated-title prefix, strong + author overlap with composite score, and a relaxed preprint/published pair + with evidence on the published side. Differing-DOI entries that fail the + XOR pair test never fall through to the weaker text heuristics. + """ + new_fields = entry.get("fields", {}) + existing_fields = existing_entry.get("fields", {}) + existing_doi = _norm_doi(existing_fields.get("doi")) or "" + + if existing_doi and new_doi and existing_doi == new_doi: + logger.debug( + f"FILE_MATCH | DOI_EXACT | file={existing_filename} | doi={existing_doi}", + category=LogCategory.DEDUP, + ) + return True + + # DOI version variants (e.g. Preprints.org .v1 / .v2) + if existing_doi and new_doi and doi_bases_match(existing_doi, new_doi): + logger.debug( + f"FILE_MATCH | DOI_VERSION | file={existing_filename} | doi_a={existing_doi} | doi_b={new_doi}", + category=LogCategory.DEDUP, + ) + return True + + # Different DOIs only match as preprint/published pairs (XOR) + if existing_doi and new_doi and existing_doi != new_doi: + e_preprint = _is_preprint_doi(existing_doi) + n_preprint = _is_preprint_doi(new_doi) + if e_preprint != n_preprint: + # If both have distinct arXiv eprint IDs, they are + # different papers -- skip preprint pair matching. + e_eprint = extract_arxiv_eprint(existing_entry) + n_eprint = extract_arxiv_eprint(entry) + if e_eprint and n_eprint and e_eprint != n_eprint: + return False + e_title = existing_fields.get("title", "") + n_title = new_fields.get("title", "") + preprint_sim = title_similarity(e_title, n_title) + if preprint_sim >= SIM_PREPRINT_TITLE_THRESHOLD: + # The preprint/published (XOR) split is already the + # precondition here, so it is excluded from the composite + # to avoid double-counting. Excluding it keeps the score + # exact and predicate-independent, so a genuine twin whose + # published side still carries a leaked preprint journal is + # scored correctly. + effective_score = compute_dedup_score(existing_fields, new_fields, count_preprint_xor=False) + if effective_score >= SIM_DEDUP_COMPOSITE_THRESHOLD: + logger.debug( + f"FILE_MATCH | PREPRINT_PAIR | file={existing_filename}" + f" | sim={preprint_sim:.3f} | effective={effective_score:.3f}" + f" | e_preprint={e_preprint} n_preprint={n_preprint}", + category=LogCategory.DEDUP, + ) + return True + return False + + if external_ids_match(existing_fields, new_fields): + existing_title = existing_fields.get("title", "") + new_title = new_fields.get("title", "") + sim = title_similarity(existing_title, new_title) + if sim >= SIM_PREPRINT_TITLE_THRESHOLD: + logger.debug( + f"FILE_MATCH | EXTERNAL_ID | file={existing_filename} | sim={sim:.3f}", + category=LogCategory.DEDUP, + ) + return True + + existing_title = existing_fields.get("title", "") + new_title = new_fields.get("title", "") + + # Citation key match requires title verification to avoid + # Gemini generating identical short titles for different papers + existing_key = existing_entry.get("key", "").strip() + new_key = entry.get("key", "").strip() + if existing_key and new_key and existing_key == new_key: + key_title_sim = title_similarity(existing_title, new_title) + # Also check if shorter title is a prefix of longer (truncated stub) + _e_norm = normalize_title(existing_title) + _n_norm = normalize_title(new_title) + _is_prefix = (_e_norm.startswith(_n_norm) and len(_n_norm) > 20) or ( + _n_norm.startswith(_e_norm) and len(_e_norm) > 20 + ) + if key_title_sim >= SIM_FILE_DUPLICATE_THRESHOLD or _is_prefix: + logger.debug( + f"FILE_MATCH | KEY_TITLE | file={existing_filename} | key={existing_key} | sim={key_title_sim:.3f}", + category=LogCategory.DEDUP, + ) + return True + # Keys match but titles differ -- check author overlap. + # Same key + strong author overlap = same paper with + # title change between preprint and publication. + _key_author_overlap = author_overlap_ratio(existing_fields.get("author"), new_fields.get("author")) + if _key_author_overlap >= 0.8 and key_title_sim >= 0.55: + logger.debug( + f"FILE_MATCH | KEY_AUTHOR_OVERLAP | file={existing_filename} " + f"| key={existing_key} | sim={key_title_sim:.3f} " + f"| author_overlap={_key_author_overlap:.3f}", + category=LogCategory.DEDUP, + ) + return True + + # Keys match but titles differ significantly -- check if + # this is a preprint/published pair before giving up + if existing_doi and new_doi and existing_doi != new_doi: + e_preprint = _is_preprint_doi(existing_doi) + n_preprint = _is_preprint_doi(new_doi) + # Distinct arXiv eprint IDs -> different papers + ke_eprint = extract_arxiv_eprint(existing_entry) + kn_eprint = extract_arxiv_eprint(entry) + if ke_eprint and kn_eprint and ke_eprint != kn_eprint: + return False + if ( + (e_preprint ^ n_preprint) + and key_title_sim >= SIM_PREPRINT_TITLE_THRESHOLD + and authors_overlap(existing_fields.get("author"), new_fields.get("author")) + ): + key_preprint_score = compute_dedup_score(existing_fields, new_fields, count_preprint_xor=False) + logger.debug( + f"FILE_MATCH | KEY_PREPRINT_PAIR | file={existing_filename} " + f"| sim={key_title_sim:.3f} | composite={key_preprint_score:.3f}", + category=LogCategory.DEDUP, + ) + return True + return False + + # Compare by title similarity alone + sim = title_similarity(existing_title, new_title) + if sim >= SIM_FILE_DUPLICATE_THRESHOLD: + logger.debug( + f"FILE_MATCH | HIGH_TITLE_SIM | file={existing_filename} | sim={sim:.3f}", + category=LogCategory.DEDUP, + ) + return True + + # Truncated title fallback where one title is a prefix of the other + if title_is_truncated_match(existing_title, new_title) and authors_overlap( + existing_fields.get("author"), new_fields.get("author") + ): + logger.debug( + f"FILE_MATCH | TRUNCATED | file={existing_filename} | authors_overlap=True", + category=LogCategory.DEDUP, + ) + return True + + # Strong author overlap on a multi-author team with moderate title similarity + if sim >= 0.6: + e_authors = parse_authors_any(existing_fields.get("author", "")) + n_authors = parse_authors_any(new_fields.get("author", "")) + if len(e_authors) >= 2 and len(n_authors) >= 2: + overlap = author_overlap_ratio(existing_fields.get("author"), new_fields.get("author")) + if overlap >= 0.9: + score = compute_dedup_score(existing_fields, new_fields) + if score >= SIM_DEDUP_COMPOSITE_THRESHOLD: + logger.debug( + f"FILE_MATCH | STRONG_AUTHOR | file={existing_filename} " + f"| overlap={overlap:.3f} | sim={sim:.3f} " + f"| composite={score:.3f} " + f"| n_authors_a={len(e_authors)} n_authors_b={len(n_authors)}", + category=LogCategory.DEDUP, + ) + return True + + # Preprint/published pair with evidence on the published side + if sim >= SIM_PREPRINT_TITLE_THRESHOLD: + e_journal = existing_fields.get("journal", "").lower() + n_journal = new_fields.get("journal", "").lower() + # Deliberately broader than the canonical fields-level predicate + # text_utils._is_preprint_fields: _is_preprint_doi delegates to + # id_utils.is_secondary_doi, whose DOI branch also matches + # DATA_DOI_PREFIXES (Zenodo, Figshare), which _is_preprint_fields + # does not. The journal-substring branch is identical. + e_preprint = _is_preprint_doi(existing_doi) or any(ps in e_journal for ps in PREPRINT_SERVERS) + n_preprint = _is_preprint_doi(new_doi) or any(ps in n_journal for ps in PREPRINT_SERVERS) + if (e_preprint ^ n_preprint) and authors_overlap(existing_fields.get("author"), new_fields.get("author")): + published_has_evidence = (e_preprint and (new_doi or n_journal)) or ( + n_preprint and (existing_doi or e_journal) + ) + if published_has_evidence: + logger.debug( + f"FILE_MATCH | PREPRINT_RELAXED | file={existing_filename} | sim={sim:.3f} " + f"| evidence=preprint_published_pair", + category=LogCategory.DEDUP, + ) + return True + + return False + + def save_entry_to_file( out_dir: str, author_id: str, @@ -924,228 +1176,23 @@ def save_entry_to_file( category=LogCategory.DEDUP, ) - # Skip prefer_path in the duplicate scan: it's the file we're updating and - # is already handled by the while-loop's prefer_path check. Scanning it + # Skip prefer_path in the duplicate scan. It is the file being updated and + # is already handled by the while-loop's prefer_path check; scanning it # first would hide a real preprint/published duplicate sitting in another file. prefer_basename = os.path.basename(prefer_path) if prefer_path else None - for existing_filename in all_files: - if existing_filename == prefer_basename: - continue - existing_path = os.path.join(author_dir, existing_filename) - try: - with open(existing_path, encoding="utf-8") as ef: - existing_content = ef.read() - existing_entry = parse_bibtex_to_dict(existing_content) - - if existing_entry: - existing_fields = existing_entry.get("fields", {}) - existing_doi = _norm_doi(existing_fields.get("doi")) or "" + def _log_read_error(fname: str) -> None: + logger.debug(f"FILE_READ_ERROR | file={fname}", category=LogCategory.DEDUP) - if existing_doi and new_doi and existing_doi == new_doi: - logger.debug( - f"FILE_MATCH | DOI_EXACT | file={existing_filename} | doi={existing_doi}", - category=LogCategory.DEDUP, - ) - duplicate_found = True - duplicate_path = existing_path - break - - # DOI version variants (e.g. Preprints.org .v1 / .v2) - if existing_doi and new_doi and doi_bases_match(existing_doi, new_doi): - logger.debug( - f"FILE_MATCH | DOI_VERSION | file={existing_filename}" - f" | doi_a={existing_doi} | doi_b={new_doi}", - category=LogCategory.DEDUP, - ) - duplicate_found = True - duplicate_path = existing_path - break - - # Different DOIs: only match preprint/published pairs (XOR) - if existing_doi and new_doi and existing_doi != new_doi: - e_preprint = _is_preprint_doi(existing_doi) - n_preprint = _is_preprint_doi(new_doi) - if e_preprint != n_preprint: - # If both have distinct arXiv eprint IDs, they are - # different papers -- skip preprint pair matching. - e_eprint = extract_arxiv_eprint(existing_entry) - n_eprint = extract_arxiv_eprint(entry) - if e_eprint and n_eprint and e_eprint != n_eprint: - continue - e_title = existing_fields.get("title", "") - n_title = new_fields.get("title", "") - preprint_sim = title_similarity(e_title, n_title) - if preprint_sim >= SIM_PREPRINT_TITLE_THRESHOLD: - # The preprint/published (XOR) split is already the - # precondition here, so it is excluded from the composite - # to avoid double-counting. Excluding it keeps the score - # exact and predicate-independent, so a genuine twin whose - # published side still carries a leaked preprint journal is - # scored correctly. - effective_score = compute_dedup_score( - existing_fields, new_fields, count_preprint_xor=False - ) - if effective_score >= SIM_DEDUP_COMPOSITE_THRESHOLD: - logger.debug( - f"FILE_MATCH | PREPRINT_PAIR | file={existing_filename}" - f" | sim={preprint_sim:.3f} | effective={effective_score:.3f}" - f" | e_preprint={e_preprint} n_preprint={n_preprint}", - category=LogCategory.DEDUP, - ) - duplicate_found = True - duplicate_path = existing_path - break - continue - - if external_ids_match(existing_fields, new_fields): - existing_title = existing_fields.get("title", "") - new_title = new_fields.get("title", "") - sim = title_similarity(existing_title, new_title) - if sim >= SIM_PREPRINT_TITLE_THRESHOLD: - logger.debug( - f"FILE_MATCH | EXTERNAL_ID | file={existing_filename} | sim={sim:.3f}", - category=LogCategory.DEDUP, - ) - duplicate_found = True - duplicate_path = existing_path - break - - existing_title = existing_fields.get("title", "") - new_title = new_fields.get("title", "") - - # Citation key match requires title verification to avoid - # Gemini generating identical short titles for different papers - existing_key = existing_entry.get("key", "").strip() - new_key = entry.get("key", "").strip() - if existing_key and new_key and existing_key == new_key: - key_title_sim = title_similarity(existing_title, new_title) - # Also check if shorter title is a prefix of longer (truncated stub) - _e_norm = normalize_title(existing_title) - _n_norm = normalize_title(new_title) - _is_prefix = (_e_norm.startswith(_n_norm) and len(_n_norm) > 20) or ( - _n_norm.startswith(_e_norm) and len(_e_norm) > 20 - ) - if key_title_sim >= SIM_FILE_DUPLICATE_THRESHOLD or _is_prefix: - logger.debug( - f"FILE_MATCH | KEY_TITLE | file={existing_filename} " - f"| key={existing_key} | sim={key_title_sim:.3f}", - category=LogCategory.DEDUP, - ) - duplicate_found = True - duplicate_path = existing_path - break - # Keys match but titles differ -- check author overlap. - # Same key + strong author overlap = same paper with - # title change between preprint and publication. - _key_author_overlap = author_overlap_ratio( - existing_fields.get("author"), new_fields.get("author") - ) - if _key_author_overlap >= 0.8 and key_title_sim >= 0.55: - logger.debug( - f"FILE_MATCH | KEY_AUTHOR_OVERLAP | file={existing_filename} " - f"| key={existing_key} | sim={key_title_sim:.3f} " - f"| author_overlap={_key_author_overlap:.3f}", - category=LogCategory.DEDUP, - ) - duplicate_found = True - duplicate_path = existing_path - break - - # Keys match but titles differ significantly -- check if - # this is a preprint/published pair before giving up - if existing_doi and new_doi and existing_doi != new_doi: - e_preprint = _is_preprint_doi(existing_doi) - n_preprint = _is_preprint_doi(new_doi) - # Distinct arXiv eprint IDs -> different papers - ke_eprint = extract_arxiv_eprint(existing_entry) - kn_eprint = extract_arxiv_eprint(entry) - if ke_eprint and kn_eprint and ke_eprint != kn_eprint: - continue - if ( - (e_preprint ^ n_preprint) - and key_title_sim >= SIM_PREPRINT_TITLE_THRESHOLD - and authors_overlap(existing_fields.get("author"), new_fields.get("author")) - ): - key_preprint_score = compute_dedup_score( - existing_fields, new_fields, count_preprint_xor=False - ) - logger.debug( - f"FILE_MATCH | KEY_PREPRINT_PAIR | file={existing_filename} " - f"| sim={key_title_sim:.3f} | composite={key_preprint_score:.3f}", - category=LogCategory.DEDUP, - ) - duplicate_found = True - duplicate_path = existing_path - break - continue - - # Compare by title similarity alone - sim = title_similarity(existing_title, new_title) - if sim >= SIM_FILE_DUPLICATE_THRESHOLD: - logger.debug( - f"FILE_MATCH | HIGH_TITLE_SIM | file={existing_filename} | sim={sim:.3f}", - category=LogCategory.DEDUP, - ) - duplicate_found = True - duplicate_path = existing_path - break - - # Truncated title fallback: one title is a prefix of the other - if title_is_truncated_match(existing_title, new_title) and authors_overlap( - existing_fields.get("author"), new_fields.get("author") - ): - logger.debug( - f"FILE_MATCH | TRUNCATED | file={existing_filename} | authors_overlap=True", - category=LogCategory.DEDUP, - ) - duplicate_found = True - duplicate_path = existing_path - break - - # Strong author overlap: multi-author team with moderate title similarity - if sim >= 0.6: - e_authors = parse_authors_any(existing_fields.get("author", "")) - n_authors = parse_authors_any(new_fields.get("author", "")) - if len(e_authors) >= 2 and len(n_authors) >= 2: - overlap = author_overlap_ratio(existing_fields.get("author"), new_fields.get("author")) - if overlap >= 0.9: - score = compute_dedup_score(existing_fields, new_fields) - if score >= SIM_DEDUP_COMPOSITE_THRESHOLD: - logger.debug( - f"FILE_MATCH | STRONG_AUTHOR | file={existing_filename} " - f"| overlap={overlap:.3f} | sim={sim:.3f} " - f"| composite={score:.3f} " - f"| n_authors_a={len(e_authors)} n_authors_b={len(n_authors)}", - category=LogCategory.DEDUP, - ) - duplicate_found = True - duplicate_path = existing_path - break - - # Preprint/published pair with evidence on the published side - if sim >= SIM_PREPRINT_TITLE_THRESHOLD: - e_journal = existing_fields.get("journal", "").lower() - n_journal = new_fields.get("journal", "").lower() - e_preprint = _is_preprint_doi(existing_doi) or any(ps in e_journal for ps in PREPRINT_SERVERS) - n_preprint = _is_preprint_doi(new_doi) or any(ps in n_journal for ps in PREPRINT_SERVERS) - if (e_preprint ^ n_preprint) and authors_overlap( - existing_fields.get("author"), new_fields.get("author") - ): - published_has_evidence = (e_preprint and (new_doi or n_journal)) or ( - n_preprint and (existing_doi or e_journal) - ) - if published_has_evidence: - logger.debug( - f"FILE_MATCH | PREPRINT_RELAXED | file={existing_filename} | sim={sim:.3f} " - f"| evidence=preprint_published_pair", - category=LogCategory.DEDUP, - ) - duplicate_found = True - duplicate_path = existing_path - break - except OSError: - logger.debug(f"FILE_READ_ERROR | file={existing_filename}", category=LogCategory.DEDUP) + for existing_filename, existing_path, scanned_entry in iter_parsed_author_bibs( + author_dir, + skip_basename=prefer_basename, + on_read_error=_log_read_error, + ): + if _is_duplicate_bib_entry(scanned_entry, entry, new_doi, existing_filename): + duplicate_found = True + duplicate_path = existing_path + break dup_filename = os.path.basename(duplicate_path) if duplicate_path else "none" logger.debug( @@ -1157,7 +1204,7 @@ def save_entry_to_file( skip_write = False dedup_replaced = False if duplicate_found and duplicate_path: - # Default: reuse duplicate's filename (overridden below when parseable) + # Reuse the duplicate's filename by default (overridden below when parseable) filename = os.path.basename(duplicate_path) try: with open(duplicate_path, encoding="utf-8") as ef: @@ -1182,8 +1229,8 @@ def _replace_existing() -> None: dup_basename = os.path.basename(duplicate_path) if existing_doi and new_doi and existing_doi != new_doi: - # Different DOIs: one is preprint, one is published (dedup already confirmed match) - if not existing_is_preprint and (new_is_preprint or not new_doi): + # DOIs differ on a confirmed match, so one side is preprint + if not existing_is_preprint and new_is_preprint: # Existing is published, new is preprint -> keep published logger.debug( f"DECISION | KEEP_EXISTING | reason=existing_published_new_preprint | file={dup_basename}", @@ -1389,9 +1436,10 @@ def _replace_existing() -> None: if prefer_path and os.path.abspath(prefer_path) != os.path.abspath(path): try: if os.path.exists(prefer_path): - # Guard: don't remove prefer_path if it's more complete than what - # we're about to write (prevents enriched file from being replaced - # by an unenriched stub when Scholar returns duplicate entries) + # Guard against removing prefer_path when it is more complete + # than what we are about to write (prevents an enriched file from + # being replaced by an unenriched stub when Scholar returns + # duplicate entries) with open(prefer_path, encoding="utf-8") as pf: prefer_entry = parse_bibtex_to_dict(pf.read()) if prefer_entry: @@ -1410,8 +1458,8 @@ def _replace_existing() -> None: except OSError: pass - # Cross-file citation key collision check: scan other files in the - # directory for the same key. If found on a DIFFERENT paper, append + # Cross-file citation key collision check. Scan other files in the + # directory for the same key; if found on a DIFFERENT paper, append # a distinguishing suffix to the key to avoid LaTeX collisions. new_key = (entry.get("key") or "").strip() if should_write and new_key: @@ -1423,15 +1471,15 @@ def _replace_existing() -> None: with open(other_path, encoding="utf-8") as of: other_entry = parse_bibtex_to_dict(of.read()) if other_entry and other_entry.get("key", "").strip() == new_key: - # Same key in another file — check if genuinely different + # Same key in another file, so check if genuinely different other_doi = _norm_doi((other_entry.get("fields") or {}).get("doi")) this_doi = _norm_doi(new_fields.get("doi")) if other_doi and this_doi and other_doi == this_doi: continue # Same paper, key collision is fine - # Different paper — disambiguate key + # Different paper, so disambiguate the key _old_key = new_key # Use first significant title word not in the other key - _title_words = re.findall(r"[A-Z][a-z]+", new_fields.get("title", "")) + _title_words = _TITLE_CAP_WORD_RE.findall(new_fields.get("title", "")) _suffix = next((w for w in _title_words if w not in new_key), "B") entry["key"] = f"{new_key}{_suffix}" logger.debug( diff --git a/citeforge/models.py b/citeforge/models.py index 9e4a16b9..c8660cc7 100644 --- a/citeforge/models.py +++ b/citeforge/models.py @@ -11,11 +11,8 @@ @dataclass class Record: - """ - Store a single author's contact details together with their identifiers on - major academic platforms. This allows the rest of the pipeline to look up - publications and metadata in a consistent way. - """ + """One author row from the input CSV, with platform identifiers used by + the pipeline to look up publications and metadata.""" name: str scholar_id: str = "" # Google Scholar author ID (optional) diff --git a/citeforge/pipeline/article.py b/citeforge/pipeline/article.py index 0364b18e..d83cac01 100644 --- a/citeforge/pipeline/article.py +++ b/citeforge/pipeline/article.py @@ -61,7 +61,7 @@ ALL_API_ERRORS, PARSE_ERRORS, ) -from citeforge.fsscan import iter_author_bibs +from citeforge.fsscan import iter_author_bibs, iter_parsed_author_bibs from citeforge.http_utils import http_get_text from citeforge.io_utils import ( append_summary_to_csv, @@ -98,7 +98,7 @@ def _entry_is_complete(entry: dict[str, Any]) -> bool: doi = fields.get("doi") has_venue = any(fields.get(v) and not has_placeholder(str(fields.get(v))) for v in ("journal", "booktitle")) - # Determine completeness: check essential fields, venue, DOI, and preprint status + # Completeness requires the essential fields, a venue, and a non-preprint DOI doi_is_preprint = False journal_is_preprint = False @@ -183,11 +183,11 @@ def _try_multiple_candidates( """Try candidates from an API source in relevance order until one matches the baseline. When *seen_dois* is provided, every DOI encountered across all candidates - (matched or not) is collected. This enables downstream duplicate detection + (matched or not) is collected. This enables downstream duplicate detection against files already on disk even when the candidate was rejected by the matching gate. - Returns (matched, matched_candidate) tuple. + Returns a (matched, matched_candidate) tuple. """ if not candidates: return False, None @@ -231,6 +231,94 @@ def _try_multiple_candidates( return False, None +def _phase2_search( + debug_label: str | None, + source_name: str, + search_fn: Callable[[], Any], + build_func: Callable[..., str | None], + flag_key: str, + title: str, + baseline_entry: dict[str, Any], + result_id: str, + enr_list: list[tuple[str, dict[str, Any]]], + flags: dict[str, bool], + seen_dois: set[str], + err_template: str = "API error - {}", +) -> Any | None: + """Run one Phase 2 / 2.5 source search and validate candidates against the baseline. + + Logs SEARCH_START when *debug_label* is given, calls *search_fn*, and feeds + any candidates through :func:`_try_multiple_candidates`. API errors are + logged with *err_template* and swallowed. Returns the matched candidate or + None. + """ + if debug_label: + logger.debug(f"SEARCH_START | source={debug_label} | title={title[:60]}", category=LogCategory.AUDIT) + matched_candidate: Any | None = None + try: + candidates = search_fn() + if candidates: + _, matched_candidate = _try_multiple_candidates( + source_name, + candidates, + build_func, + baseline_entry, + result_id, + enr_list, + flags, + flag_key, + max_candidates=5, + seen_dois=seen_dois, + ) + except ALL_API_ERRORS as e: + logger.warn(err_template.format(e), category=LogCategory.ERROR, source=source_name) + return matched_candidate + + +def _stash_external_id(baseline_entry: dict[str, Any], api: str, field: str, value: Any) -> None: + """Store an external identifier from a matched candidate on the baseline entry.""" + if value: + baseline_entry["fields"][field] = str(value) + logger.debug( + f"ID_EXTRACT | api={api} | field={field} | value={value}", + category=LogCategory.AUDIT, + ) + + +def _relpath_or(path: str) -> str: + """Return *path* relative to CWD, or *path* unchanged when relpath fails.""" + try: + return os.path.relpath(path) + except (OSError, ValueError): + return path + + +def _record_skip_summary(summary_csv_path: str | None, path: str | None, flags: dict[str, bool]) -> None: + """Write a zero-hit summary row for a skipped article, unless a previous run already tracks it.""" + if not (summary_csv_path and path): + return + rel = _relpath_or(path) + if not is_known_summary_path(rel): + append_summary_to_csv(summary_csv_path, rel, 0, flags) + + +def _title_as_venue(fields: dict[str, Any]) -> str | None: + """Return the lowered title when it equals the journal or booktitle, else None. + + Marks corrupted Scholar data where the venue leaked into the title field. + Deliberately checked at two sites (existing-file fixup and Phase 4 + post-merge) so corrupt entries are deleted whichever path they take. + """ + title = (fields.get("title") or "").strip().lower() + if not title: + return None + journal = (fields.get("journal") or "").strip().lower() + booktitle = (fields.get("booktitle") or "").strip().lower() + if (journal and title == journal) or (booktitle and title == booktitle): + return title + return None + + def process_article( rec: Record, art: dict[str, Any], @@ -325,11 +413,10 @@ def process_article( baseline_entry = None existing_file_path = None - # Try to find existing BibTeX file to use as enrichment seed - # If found, load it and use as baseline - enrichment process will update/fix fields + # Look for an existing BibTeX file to use as the enrichment seed; when + # found it becomes the baseline and enrichment updates its fields if SKIP_SCHOLAR_FOR_EXISTING_FILES and os.path.exists(author_dir): - # Sort filenames for deterministic iteration order - bib_files = iter_author_bibs(author_dir) + bib_files = iter_author_bibs(author_dir) # sorted, for deterministic iteration logger.debug( f"EXISTING_FILE_SCAN | dir={author_dir} | files_checked={len(bib_files)}", category=LogCategory.AUDIT, @@ -384,20 +471,15 @@ def process_article( _bl_fields = baseline_entry.get("fields") or {} # Delete entries where title equals journal or booktitle (corrupted Scholar data) - _bl_title_venue = (_bl_fields.get("title") or "").strip().lower() - if _bl_title_venue: - _bl_journal_venue = (_bl_fields.get("journal") or "").strip().lower() - _bl_booktitle_venue = (_bl_fields.get("booktitle") or "").strip().lower() - if (_bl_journal_venue and _bl_title_venue == _bl_journal_venue) or ( - _bl_booktitle_venue and _bl_title_venue == _bl_booktitle_venue - ): - logger.debug( - f"EXISTING_FIXUP | title_is_venue | title={_bl_title_venue[:60]} | deleting", - category=LogCategory.CLEANUP, - ) - if existing_file_path and os.path.exists(existing_file_path): - os.remove(existing_file_path) - return 0 + _bl_title_venue = _title_as_venue(_bl_fields) + if _bl_title_venue is not None: + logger.debug( + f"EXISTING_FIXUP | title_is_venue | title={_bl_title_venue[:60]} | deleting", + category=LogCategory.CLEANUP, + ) + if existing_file_path and os.path.exists(existing_file_path): + os.remove(existing_file_path) + return 0 # Escape bare & in field values (bibtex_from_dict handles this on write, # but we need to trigger a rewrite for files that were never re-serialized) @@ -412,7 +494,7 @@ def process_article( # Skip enrichment entirely if entry is already complete (unless --force) if not force_enrich and existing_file_loaded and baseline_entry is not None and _entry_is_complete(baseline_entry): - # Quick fixup: strip preprint-only publishers from complete entries. + # Quick fixup that strips preprint-only publishers from complete entries. # Single-sourced in canonicalize() at the COMPLETE_SKIP_FINALIZE stage; the # debug log and write-gating stay here so the skip-path I/O is unchanged. bl_fields = baseline_entry.get("fields") or {} @@ -432,14 +514,7 @@ def process_article( # here. logger.info("Entry already complete; skipping enrichment", category=LogCategory.SKIP, source=LogSource.SYSTEM) - if summary_csv_path and existing_file_path: - try: - rel = os.path.relpath(existing_file_path) - except (OSError, ValueError): - rel = existing_file_path - # Only write a new CSV row if this file has no entry from a previous run - if not is_known_summary_path(rel): - append_summary_to_csv(summary_csv_path, rel, 0, flags) + _record_skip_summary(summary_csv_path, existing_file_path, flags) return 1 # If no existing file found, build minimal BibTeX baseline @@ -548,7 +623,7 @@ def process_article( if was_written: logger.success(f"Saved baseline: {path}", category=LogCategory.SAVE, source=LogSource.SYSTEM) else: - # save_entry_to_file found a duplicate and skipped writing — + # save_entry_to_file found a duplicate and skipped writing; # the article is already on disk under a different name. # Skip enrichment entirely to avoid churn. logger.info( @@ -556,19 +631,13 @@ def process_article( category=LogCategory.SKIP, source=LogSource.SYSTEM, ) - if summary_csv_path and path: - try: - rel = os.path.relpath(path) - except (OSError, ValueError): - rel = path - if not is_known_summary_path(rel): - append_summary_to_csv(summary_csv_path, rel, 0, flags) + _record_skip_summary(summary_csv_path, path, flags) return 1 enr_list: list[tuple[str, dict[str, Any]]] = [] # Collect DOIs from ALL Phase 2 candidates (matched or not) for - # deterministic dedup: if a candidate's DOI already exists on disk, - # we skip writing even when the candidate was rejected by the match gate. + # deterministic dedup; when a candidate's DOI already exists on disk the + # write is skipped even though the match gate rejected the candidate. all_candidate_dois: set[str] = set() # ===== PHASE 1: Early DOI Validation ===== @@ -651,162 +720,112 @@ def process_article( else: logger.info("No title available; skipped", category=LogCategory.SKIP, source=LogSource.SCHOLAR) - logger.debug(f"SEARCH_START | source=S2 | title={title[:60]}", category=LogCategory.AUDIT) - s2_paper = None - if s2_api_key: - try: - s2_papers = s2_search_papers_multiple(title, rec.name, s2_api_key, max_results=5) - if s2_papers: - _, s2_paper = _try_multiple_candidates( - LogSource.S2, - s2_papers, - build_bibtex_from_s2, - baseline_entry, - result_id, - enr_list, - flags, - "s2", - max_candidates=5, - seen_dois=all_candidate_dois, - ) - if s2_paper: - s2_id = s2_paper.get("paperId") - if s2_id: - baseline_entry["fields"]["x_s2_paper_id"] = str(s2_id) - logger.debug( - f"ID_EXTRACT | api=S2 | field=x_s2_paper_id | value={s2_id}", - category=LogCategory.AUDIT, - ) - except ALL_API_ERRORS as e: - logger.warn(f"API error - {e}", category=LogCategory.ERROR, source=LogSource.S2) + # Source order is fixed; each search resolves its client function at call + # time from module globals so tests can monkeypatch the entry points. + s2_paper = _phase2_search( + "S2", + LogSource.S2, + (lambda: s2_search_papers_multiple(title, rec.name, s2_api_key, max_results=5)) if s2_api_key else lambda: None, + build_bibtex_from_s2, + "s2", + title, + baseline_entry, + result_id, + enr_list, + flags, + all_candidate_dois, + ) + if s2_paper: + _stash_external_id(baseline_entry, "S2", "x_s2_paper_id", s2_paper.get("paperId")) + + cr_item = _phase2_search( + "Crossref", + LogSource.CROSSREF, + lambda: crossref_search_multiple(title, rec.name, max_results=5, year_hint=year_hint), + build_bibtex_from_crossref, + "crossref", + title, + baseline_entry, + result_id, + enr_list, + flags, + all_candidate_dois, + ) - logger.debug(f"SEARCH_START | source=Crossref | title={title[:60]}", category=LogCategory.AUDIT) - cr_item = None - try: - cr_items = crossref_search_multiple(title, rec.name, max_results=5, year_hint=year_hint) - if cr_items: - _, cr_item = _try_multiple_candidates( - LogSource.CROSSREF, - cr_items, - build_bibtex_from_crossref, - baseline_entry, - result_id, - enr_list, - flags, - "crossref", - max_candidates=5, - seen_dois=all_candidate_dois, - ) - except ALL_API_ERRORS as e: - logger.warn(f"API error - {e}", category=LogCategory.ERROR, source=LogSource.CROSSREF) + _phase2_search( + "OpenReview", + LogSource.OPENREVIEW, + lambda: openreview_search_papers_multiple(title, rec.name, or_creds, max_results=5), + build_bibtex_from_openreview, + "openreview", + title, + baseline_entry, + result_id, + enr_list, + flags, + all_candidate_dois, + ) - logger.debug(f"SEARCH_START | source=OpenReview | title={title[:60]}", category=LogCategory.AUDIT) - try: - or_notes = openreview_search_papers_multiple(title, rec.name, or_creds, max_results=5) - if or_notes: - _try_multiple_candidates( - LogSource.OPENREVIEW, - or_notes, - build_bibtex_from_openreview, - baseline_entry, - result_id, - enr_list, - flags, - "openreview", - max_candidates=5, - seen_dois=all_candidate_dois, - ) - except ALL_API_ERRORS as e: - logger.warn(f"API error - {e}", category=LogCategory.ERROR, source=LogSource.OPENREVIEW) + arxiv_entry = _phase2_search( + "arXiv", + LogSource.ARXIV, + lambda: arxiv_search(title, rec.name, year_hint), + build_bibtex_from_arxiv, + "arxiv", + title, + baseline_entry, + result_id, + enr_list, + flags, + all_candidate_dois, + ) - logger.debug(f"SEARCH_START | source=arXiv | title={title[:60]}", category=LogCategory.AUDIT) - arxiv_entry = None - try: - arxiv_entries = arxiv_search(title, rec.name, year_hint) - if arxiv_entries: - _, arxiv_entry = _try_multiple_candidates( - LogSource.ARXIV, - arxiv_entries, - build_bibtex_from_arxiv, - baseline_entry, - result_id, - enr_list, - flags, - "arxiv", - max_candidates=5, - seen_dois=all_candidate_dois, - ) - except ALL_API_ERRORS as e: - logger.warn(f"API error - {e}", category=LogCategory.ERROR, source=LogSource.ARXIV) + oa_work = _phase2_search( + "OpenAlex", + LogSource.OPENALEX, + lambda: openalex_search_multiple(title, rec.name, max_results=5, year_hint=year_hint), + build_bibtex_from_openalex, + "openalex", + title, + baseline_entry, + result_id, + enr_list, + flags, + all_candidate_dois, + ) + if oa_work: + _stash_external_id(baseline_entry, "OpenAlex", "x_openalex_id", oa_work.get("id")) + + pm_article = _phase2_search( + "PubMed", + LogSource.PUBMED, + lambda: pubmed_search_papers_multiple(title, rec.name, max_results=5), + build_bibtex_from_pubmed, + "pubmed", + title, + baseline_entry, + result_id, + enr_list, + flags, + all_candidate_dois, + ) - logger.debug(f"SEARCH_START | source=OpenAlex | title={title[:60]}", category=LogCategory.AUDIT) - oa_work = None - try: - oa_works = openalex_search_multiple(title, rec.name, max_results=5, year_hint=year_hint) - if oa_works: - _, oa_work = _try_multiple_candidates( - LogSource.OPENALEX, - oa_works, - build_bibtex_from_openalex, - baseline_entry, - result_id, - enr_list, - flags, - "openalex", - max_candidates=5, - seen_dois=all_candidate_dois, - ) - if oa_work: - oa_id = oa_work.get("id") - if oa_id: - baseline_entry["fields"]["x_openalex_id"] = str(oa_id) - logger.debug( - f"ID_EXTRACT | api=OpenAlex | field=x_openalex_id | value={oa_id}", - category=LogCategory.AUDIT, - ) - except ALL_API_ERRORS as e: - logger.warn(f"API error - {e}", category=LogCategory.ERROR, source=LogSource.OPENALEX) - logger.debug(f"SEARCH_START | source=PubMed | title={title[:60]}", category=LogCategory.AUDIT) - pm_article = None - try: - pm_articles = pubmed_search_papers_multiple(title, rec.name, max_results=5) - if pm_articles: - _, pm_article = _try_multiple_candidates( - LogSource.PUBMED, - pm_articles, - build_bibtex_from_pubmed, - baseline_entry, - result_id, - enr_list, - flags, - "pubmed", - max_candidates=5, - seen_dois=all_candidate_dois, - ) - except ALL_API_ERRORS as e: - logger.warn(f"API error - {e}", category=LogCategory.ERROR, source=LogSource.PUBMED) - logger.debug(f"SEARCH_START | source=EuropePMC | title={title[:60]}", category=LogCategory.AUDIT) - epmc_article = None - try: - epmc_articles = europepmc_search_papers_multiple(title, rec.name, max_results=5) - if epmc_articles: - _, epmc_article = _try_multiple_candidates( - LogSource.EUROPEPMC, - epmc_articles, - build_bibtex_from_europepmc, - baseline_entry, - result_id, - enr_list, - flags, - "europepmc", - max_candidates=5, - seen_dois=all_candidate_dois, - ) - except ALL_API_ERRORS as e: - logger.warn(f"API error - {e}", category=LogCategory.ERROR, source=LogSource.EUROPEPMC) + epmc_article = _phase2_search( + "EuropePMC", + LogSource.EUROPEPMC, + lambda: europepmc_search_papers_multiple(title, rec.name, max_results=5), + build_bibtex_from_europepmc, + "europepmc", + title, + baseline_entry, + result_id, + enr_list, + flags, + all_candidate_dois, + ) # ===== PHASE 2.5: Venue-Based Search (SerpAPI publication string) ===== - # Only attempt when no enrichment matched so far — avoids redundant API calls. + # Only attempted when no enrichment matched so far, avoiding redundant API calls. if not enr_list: pub_string = art.get("publication") or "" if pub_string: @@ -835,60 +854,37 @@ def process_article( # Tier 1: venue-based Crossref search (journal/conference only) if parsed_pub.venue_type in ("journal", "conference"): - try: - cr_venue_items = crossref_search_by_venue( - title, - rec.name, - container_title=parsed_pub.venue_name, - max_results=5, - ) - if cr_venue_items: - _try_multiple_candidates( - LogSource.CROSSREF, - cr_venue_items, - build_bibtex_from_crossref, - baseline_entry, - result_id, - enr_list, - flags, - "crossref", - max_candidates=5, - seen_dois=all_candidate_dois, - ) - except ALL_API_ERRORS as e: - logger.warn( - f"Venue-based Crossref error: {e}", - category=LogCategory.ERROR, - source=LogSource.CROSSREF, - ) + venue_name = parsed_pub.venue_name + _phase2_search( + None, + LogSource.CROSSREF, + lambda: crossref_search_by_venue(title, rec.name, container_title=venue_name, max_results=5), + build_bibtex_from_crossref, + "crossref", + title, + baseline_entry, + result_id, + enr_list, + flags, + all_candidate_dois, + err_template="Venue-based Crossref error: {}", + ) - # Tier 1: venue-based OpenAlex search (only if Crossref missed) - if not enr_list and parsed_pub.venue_type in ("journal", "conference"): - try: - oa_venue_items = openalex_search_by_venue( + # Tier 1: venue-based OpenAlex search (only if Crossref missed) + if not enr_list: + _phase2_search( + None, + LogSource.OPENALEX, + lambda: openalex_search_by_venue(title, rec.name, venue_name=venue_name, max_results=5), + build_bibtex_from_openalex, + "openalex", title, - rec.name, - venue_name=parsed_pub.venue_name, - max_results=5, - ) - if oa_venue_items: - _try_multiple_candidates( - LogSource.OPENALEX, - oa_venue_items, - build_bibtex_from_openalex, - baseline_entry, - result_id, - enr_list, - flags, - "openalex", - max_candidates=5, - seen_dois=all_candidate_dois, - ) - except ALL_API_ERRORS as e: - logger.warn( - f"Venue-based OpenAlex error: {e}", - category=LogCategory.ERROR, - source=LogSource.OPENALEX, + baseline_entry, + result_id, + enr_list, + flags, + all_candidate_dois, + err_template="Venue-based OpenAlex error: {}", ) # ===== PHASE 3: Late DOI Discovery ===== @@ -925,7 +921,7 @@ def _add_doi(source: str, doi: str | None) -> None: _add_doi("phase1_stash", unvalidated_doi) # Infer arXiv DOIs from eprint fields or URLs in baseline and enrichers - # (deterministic — no HTTP required) + # (deterministic, no HTTP required) _bl_eprint = idu.extract_arxiv_eprint(baseline_entry) if _bl_eprint: _add_doi("baseline_eprint", f"10.48550/arxiv.{_bl_eprint}") @@ -985,7 +981,7 @@ def _add_doi(source: str, doi: str | None) -> None: url_candidates.append(f"https://europepmc.org/article/PMC/{numeric_id}") # Deterministic DOI extraction from known URL patterns - # (no HTTP required — prevents network non-determinism) + # (no HTTP required, prevents network non-determinism) for u in filter(None, url_candidates): m = _ARXIV_ABS_RE.search(str(u)) if m: @@ -1115,13 +1111,13 @@ def _add_doi(source: str, doi: str | None) -> None: merged = mu.merge_with_policy(baseline_entry, enr_list) merged_fields = merged.get("fields") or {} - # Phase-4 post-merge canonicalization: entry-type reclassification and - # text/venue normalization, single-sourced in citeforge/canonicalize.py. + # Phase-4 post-merge canonicalization (entry-type reclassification and + # text/venue normalization), single-sourced in citeforge/canonicalize.py. # POST_MERGE is the terminal stage; absence rules (article/inproceedings # missing venue, preprint-DOI downgrade) fire here after enrichment. canonicalize(merged, stage=CanonicalStage.POST_MERGE) - # Annotate bare stubs: no enrichers, no DOI, no venue + # Annotate bare stubs (no enrichers, no DOI, no venue) is_bare_stub = ( not enr_list and not (merged_fields.get("doi") or "").strip() @@ -1190,22 +1186,17 @@ def _add_doi(source: str, doi: str | None) -> None: # Delete entries where title equals journal or booktitle (corrupted Scholar data). # Placed after Tier 2 filling so it catches entries populated from SerpAPI pub strings. - _p4_title_lower = (merged_fields.get("title") or "").strip().lower() - if _p4_title_lower: - _p4_journal_lower = (merged_fields.get("journal") or "").strip().lower() - _p4_booktitle_lower = (merged_fields.get("booktitle") or "").strip().lower() - if (_p4_journal_lower and _p4_title_lower == _p4_journal_lower) or ( - _p4_booktitle_lower and _p4_title_lower == _p4_booktitle_lower - ): - logger.debug( - f"TITLE_IS_VENUE | title={_p4_title_lower[:60]} | skipping entry", - category=LogCategory.AUDIT, - ) - if path and os.path.isfile(path): - os.remove(path) - return 0 + _p4_title_lower = _title_as_venue(merged_fields) + if _p4_title_lower is not None: + logger.debug( + f"TITLE_IS_VENUE | title={_p4_title_lower[:60]} | skipping entry", + category=LogCategory.AUDIT, + ) + if path and os.path.isfile(path): + os.remove(path) + return 0 - # Skip entries with type "book" (proceedings volumes, edited books — not individual papers) + # Skip entries with type "book" (proceedings volumes and edited books, not individual papers) if merged.get("type") == "book": has_file = bool(path and os.path.isfile(path)) logger.debug( @@ -1231,7 +1222,7 @@ def _add_doi(source: str, doi: str | None) -> None: ) if merged_authors and not author_found: # Check if enrichment corrupted the author field (misattributed DOI). - # If the ORIGINAL file had the correct author, keep it — don't delete. + # If the ORIGINAL file had the correct author, keep it rather than delete. if path and os.path.isfile(path) and baseline_entry: baseline_authors = bf.get("author", "") if baseline_authors and author_name_matches(rec.name, baseline_authors): @@ -1251,10 +1242,11 @@ def _add_doi(source: str, doi: str | None) -> None: os.remove(path) return 0 - # Deterministic dedup: check if any DOI (from Phase 2 candidates, Phase 3 + # Deterministic dedup. When any DOI (from Phase 2 candidates, Phase 3 # discovery, or the merged entry itself) already exists in a DIFFERENT file - # on disk. This prevents oscillation where a preprint/published pair creates - # a file under the preprint title that gets enriched and renamed every run. + # on disk, skip the write. This prevents oscillation where a preprint/ + # published pair creates a file under the preprint title that gets + # enriched and renamed every run. merged_doi = idu.normalize_doi(merged_fields.get("doi", "")) _merged_eprint = idu.extract_arxiv_eprint(merged) if _merged_eprint: @@ -1265,15 +1257,12 @@ def _add_doi(source: str, doi: str | None) -> None: prefer_doi = _read_doi_from_file(path) if path and os.path.isfile(path) else "" check_dois = all_candidate_dois - {prefer_doi} if prefer_doi else all_candidate_dois if check_dois: - for existing_bib in iter_author_bibs(author_dir): - epath = os.path.join(author_dir, existing_bib) - if path and os.path.abspath(epath) == os.path.abspath(path): - continue # skip self + for existing_bib, epath, edict in iter_parsed_author_bibs( + author_dir, + skip_path=path or None, # skip self + read_errors=(OSError, UnicodeDecodeError), + ): try: - with open(epath, encoding="utf-8") as ef: - edict = bt.parse_bibtex_to_dict(ef.read()) - if not edict: - continue edoi = idu.normalize_doi((edict.get("fields") or {}).get("doi", "")) if not edoi or edoi not in check_dois: continue @@ -1293,10 +1282,11 @@ def _add_doi(source: str, doi: str | None) -> None: ) _revert_misattributed_doi(merged_fields, edoi, doi_validated, doi_early) continue - # Published supersedes preprint: if the on-disk match is a preprint/ - # secondary DOI while the incoming entry carries a genuine published - # DOI, remove the on-disk preprint and keep the published entry rather - # than dropping the published file (mirrors the save-time tiebreak). + # Published supersedes preprint. When the on-disk match is a + # preprint/secondary DOI while the incoming entry carries a genuine + # published DOI, remove the on-disk preprint and keep the published + # entry rather than dropping the published file (mirrors the + # save-time tiebreak). merged_is_published = bool(merged_doi and not idu.is_secondary_doi(merged_doi)) if merged_is_published and idu.is_secondary_doi(edoi): logger.debug( @@ -1322,7 +1312,7 @@ def _add_doi(source: str, doi: str | None) -> None: merged["key"] = bt.build_standard_citekey(merged, gemini_api_key=gemini_api_key) or merged.get("key") or "Entry" - # Year-window guard: reject files whose enriched year falls outside the window + # Year-window guard; rejects entries whose enriched year falls below the window minimum if min_year > 0: final_year = extract_year_from_any(merged.get("fields", {}).get("year"), fallback=0) or 0 if 0 < final_year < min_year: @@ -1342,11 +1332,8 @@ def _add_doi(source: str, doi: str | None) -> None: logger.success(f"Enriched and renamed: {path2}", category=LogCategory.SAVE, source=LogSource.SYSTEM) else: logger.success(f"Enriched: {path2}", category=LogCategory.SAVE, source=LogSource.SYSTEM) - # Summary log: relative path and success flags - try: - rel = os.path.relpath(path2) - except (OSError, ValueError): - rel = path2 + # Summary row carries the relative path and per-source flags + rel = _relpath_or(path2) total_true = sum(1 for v in flags.values() if v) # ===== Enrichment Summary ===== diff --git a/citeforge/pipeline/postrun.py b/citeforge/pipeline/postrun.py index 3f33780b..555492de 100644 --- a/citeforge/pipeline/postrun.py +++ b/citeforge/pipeline/postrun.py @@ -9,7 +9,6 @@ from __future__ import annotations import csv -import json import os import re import time @@ -34,6 +33,7 @@ flush_summary_csv, reconcile_summary_csv, safe_write_file, + safe_write_json, ) from citeforge.log_utils import LogCategory, logger from citeforge.models import Record @@ -124,13 +124,11 @@ def finalize_run( # Remove .bib files outside the contribution window window_min = get_min_year() window_removed = 0 - for entry in os.listdir(out_dir): - d = os.path.join(out_dir, entry) - if not os.path.isdir(d) or entry == "a2i2": + for entry in iter_output_dirs(out_dir): + if entry == "a2i2": continue - for fname in os.listdir(d): - if not fname.endswith(".bib"): - continue + d = os.path.join(out_dir, entry) + for fname in iter_author_bibs(d): fpath = os.path.join(d, fname) # Try filename year first m = _FILENAME_YEAR_RE.search(f"/{fname}") @@ -143,7 +141,7 @@ def finalize_run( os.remove(fpath) window_removed += 1 continue - # Fallback: read BibTeX year field for non-standard filenames + # Fall back to the BibTeX year field for non-standard filenames try: with open(fpath, encoding="utf-8") as bf: parsed = bt.parse_bibtex_to_dict(bf.read()) @@ -163,17 +161,15 @@ def finalize_run( category=LogCategory.CLEANUP, ) - # Post-run fixup: apply entry type and field corrections to ALL .bib files - # This catches orphans (files not processed during enrichment) and any + # Post-run fixup applies entry type and field corrections to ALL .bib + # files. This catches orphans (files not processed during enrichment) and # entries where Phase 4 corrections were undone by Tier 2 filling. postrun_fixed = 0 for pr_entry_name in iter_output_dirs(out_dir): - pr_dir = os.path.join(out_dir, pr_entry_name) if pr_entry_name == "a2i2": continue - for pr_fname in sorted(os.listdir(pr_dir)): - if not pr_fname.endswith(".bib"): - continue + pr_dir = os.path.join(out_dir, pr_entry_name) + for pr_fname in iter_author_bibs(pr_dir): pr_fpath = os.path.join(pr_dir, pr_fname) try: with open(pr_fpath, encoding="utf-8") as prf: @@ -208,38 +204,30 @@ def finalize_run( category=LogCategory.CLEANUP, ) - # Write per-author baseline counts + # Write per-author baseline counts (a2i2 included by design; the + # baseline total must equal the on-disk .bib count) baseline: dict[str, int] = {} for entry in iter_output_dirs(out_dir): - d = os.path.join(out_dir, entry) - baseline[entry] = len(iter_author_bibs(d)) - baseline_path = os.path.join(out_dir, "baseline.json") - try: - with open(baseline_path, "w", encoding="utf-8") as bf: - json.dump({"total": sum(baseline.values()), "authors": baseline}, bf, indent=2) - except OSError: - pass + baseline[entry] = len(iter_author_bibs(os.path.join(out_dir, entry))) + safe_write_json( + os.path.join(out_dir, "baseline.json"), + {"total": sum(baseline.values()), "authors": baseline}, + ) # Write badge data for README workflow updates - badges_path = os.path.join(out_dir, "badges.json") - try: - with open(badges_path, "w", encoding="utf-8") as bf: - total = cache_counts["positive"] + cache_counts["negative"] + cache_counts["miss"] - hit_rate = ((cache_counts["positive"] + cache_counts["negative"]) / total * 100) if total else 0 - json.dump( - { - "last_updated": time.strftime("%Y-%m"), - "cache_positive_hits": cache_counts["positive"], - "cache_negative_hits": cache_counts["negative"], - "cache_misses": cache_counts["miss"], - "total_queries": total, - "hit_rate": round(hit_rate, 1), - }, - bf, - indent=2, - ) - except OSError: - pass + total = cache_counts["positive"] + cache_counts["negative"] + cache_counts["miss"] + hit_rate = ((cache_counts["positive"] + cache_counts["negative"]) / total * 100) if total else 0 + safe_write_json( + os.path.join(out_dir, "badges.json"), + { + "last_updated": time.strftime("%Y-%m"), + "cache_positive_hits": cache_counts["positive"], + "cache_negative_hits": cache_counts["negative"], + "cache_misses": cache_counts["miss"], + "total_queries": total, + "hit_rate": round(hit_rate, 1), + }, + ) logger.info(f"Summary CSV: {summary_csv_path}", category=LogCategory.PLAN) @@ -270,7 +258,7 @@ def _remove_superseded_preprints(out_dir: str) -> int: ``SIM_MERGE_DUPLICATE_THRESHOLD``. This generalizes the "published outranks a preprint" rule across every author and paper: a standalone preprint with no published counterpart is always retained, and published files are never - removed. Deterministic (sorted iteration) and idempotent -- once the preprint + removed. Deterministic (sorted iteration) and idempotent. Once the preprint is gone a later run finds no preprint+published pair and removes nothing. Returns the number of preprint files removed. diff --git a/citeforge/pipeline/scheduler.py b/citeforge/pipeline/scheduler.py index 4224c177..2b26f215 100644 --- a/citeforge/pipeline/scheduler.py +++ b/citeforge/pipeline/scheduler.py @@ -45,6 +45,11 @@ ) +def _author_dirname(rec: Record) -> str: + """Return the output directory name for *rec*, keyed by its Scholar or DBLP id.""" + return format_author_dirname(rec.name, rec.scholar_id or rec.dblp or "") + + def process_record( serpapi_key: str, serply_key: str | None, @@ -62,11 +67,8 @@ def process_record( Returns the number of BibTeX files successfully written. """ - # Setup thread-local logging for this author - effective_id = rec.scholar_id or rec.dblp or "" - author_dirname = format_author_dirname(rec.name, effective_id) - author_log_path = os.path.join(out_dir, author_dirname, "author.log") - + # Set up thread-local logging for this author + author_log_path = os.path.join(out_dir, _author_dirname(rec), "author.log") logger.set_log_file(author_log_path) try: @@ -85,7 +87,7 @@ def process_record( scholar_articles: list[dict[str, Any]] = [] max_fetch_retries = 3 - # SerpAPI call — pagination handled internally by serpapi_scholar + # SerpAPI call; pagination handled internally by serpapi_scholar data = {} for attempt in range(1, max_fetch_retries + 1): data = fetch_author_publications( @@ -96,7 +98,7 @@ def process_record( min_year=min_year, ) if data.get("articles"): - break # Got articles -- valid response + break # Got articles, valid response if attempt < max_fetch_retries: logger.warn( f"Scholar API returned empty (attempt {attempt}/{max_fetch_retries}), retrying...", @@ -226,11 +228,8 @@ def process_record( def count_existing_papers(rec: Record, out_dir: str) -> int: """Count existing .bib files in the author's output directory.""" - effective_id = rec.scholar_id or rec.dblp or "" - author_dirname = format_author_dirname(rec.name, effective_id) - author_dir = os.path.join(out_dir, author_dirname) try: - return len(iter_author_bibs(author_dir)) + return len(iter_author_bibs(os.path.join(out_dir, _author_dirname(rec)))) except OSError: return 0 @@ -280,11 +279,12 @@ def run_all( total_saved = 0 processed = 0 - # Prioritize new authors (no existing output dir) so they get browser/API - # resources first, before cached authors consume worker slots + # Prioritize new authors (no existing output dir) so they get API resources + # first, before cached authors consume worker slots. This intentionally + # overrides the count-descending order from prioritize_records for new + # authors only; the relative order within each group is preserved. def _has_output(r: Record) -> bool: - eid = r.scholar_id or r.dblp or "" - return os.path.isdir(os.path.join(out_dir, format_author_dirname(r.name, eid))) + return os.path.isdir(os.path.join(out_dir, _author_dirname(r))) records_sorted = [r for _, r in sorted(enumerate(records), key=lambda ir: (_has_output(ir[1]), ir[0]))] @@ -302,8 +302,8 @@ def _thread_excepthook(args: Any) -> None: threading.excepthook = _thread_excepthook - # Per-author timeout: 30 minutes per author to handle large publication lists - # Each article takes ~60-90s across all API calls, so 24 articles ≈ 36 minutes + # Per-author budget of 30 minutes, applied as one whole-pool deadline + # (author_timeout * len(records)) on as_completed, not per future author_timeout = 1800 # seconds with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor: diff --git a/citeforge/text_utils.py b/citeforge/text_utils.py index 9ab7b888..c82999f7 100644 --- a/citeforge/text_utils.py +++ b/citeforge/text_utils.py @@ -1,9 +1,8 @@ """Text and author-name normalization and similarity scoring. -Shared helpers for normalizing titles and author names and for scoring the -similarity between records. The merge, deduplication, and BibTeX-building layers -all compare records through these functions so the notion of the same paper -stays consistent across the pipeline. +Normalizes titles and author names and scores record similarity. The merge, +deduplication, and BibTeX-building layers all compare records through these +functions, so any output change here shifts dedup thresholds pipeline-wide. """ from __future__ import annotations @@ -12,6 +11,7 @@ import html as html_module import re import urllib.parse +from datetime import datetime, timezone from typing import Any from rapidfuzz.fuzz import ratio as fuzz_ratio @@ -63,11 +63,8 @@ def _name_from_dict(d: dict[str, Any]) -> str: - """ - Build a display name from a dictionary that may contain either a full - "name" field or separate given/family (first/last) components. - Returns an empty string if nothing usable is present. - """ + """Build a display name from a dict with a "name" field or given/family + (first/last) components. Returns "" when nothing usable is present.""" name = str(d.get("name") or "").strip() if name: return name @@ -77,18 +74,14 @@ def _name_from_dict(d: dict[str, Any]) -> str: def build_url(base: str, params: dict[str, Any]) -> str: - """ - Attach query parameters to a base URL and return the fully encoded address as a string. - """ + """Attach URL-encoded query parameters to a base URL.""" q = urllib.parse.urlencode(params) return f"{base}?{q}" def to_text(obj: Any) -> str: - """ - Convert an arbitrary value into a readable string, handling nested - dictionaries, lists of authors, and other common metadata shapes from APIs. - """ + """Convert an arbitrary value (nested dicts, author lists, other common + API metadata shapes) into a readable string.""" if obj is None: return "" if isinstance(obj, str): @@ -121,42 +114,43 @@ def to_text(obj: Any) -> str: def strip_accents(s: str) -> str: - """ - Remove accents and diacritics from a string so visually similar text from - different locales can be compared more reliably. - - Uses unidecode for Unicode-to-ASCII transliteration. - """ + """Transliterate Unicode to ASCII via unidecode, removing accents and + diacritics so text from different locales compares reliably.""" try: return unidecode(s) except PARSE_ERRORS + DECODE_ERRORS: return s +_MATH_DELIM_RE = re.compile(r"\$([^$]*)\$") +_LATEX_FRAC_RE = re.compile(r"\\frac\{([^}]*)\}\{([^}]*)\}") +_LATEX_CMD_ARG_RE = re.compile(r"\\[a-zA-Z]+\{([^}]*)}") +_LATEX_CMD_RE = re.compile(r"\\[a-zA-Z]+") +_TITLE_PUNCT_RE = re.compile(r"[,.;:!?\n\t\r'\"\-\(\)\[\]\{\}~]") + + @functools.lru_cache(maxsize=4096) def normalize_title(t: str | None) -> str: - """ - Normalize a title for comparison by stripping accents, lowercasing, removing - punctuation, brackets, LaTeX formatting, and collapsing repeated whitespace. - """ + """Normalize a title for comparison. Strips accents, lowercases, removes + punctuation, brackets, and LaTeX formatting, collapses whitespace.""" if not t: return "" t_str = str(t) t_str = html_module.unescape(t_str) - t_str = re.sub(r"\$([^$]*)\$", r"\1", t_str) - t_str = re.sub(r"\\frac\{([^}]*)\}\{([^}]*)\}", r"\1/\2", t_str) - t_str = re.sub(r"\\[a-zA-Z]+\{([^}]*)}", r"\1", t_str) - t_str = re.sub(r"\\[a-zA-Z]+", "", t_str) + t_str = _MATH_DELIM_RE.sub(r"\1", t_str) + t_str = _LATEX_FRAC_RE.sub(r"\1/\2", t_str) + t_str = _LATEX_CMD_ARG_RE.sub(r"\1", t_str) + t_str = _LATEX_CMD_RE.sub("", t_str) t2 = strip_accents(t_str).lower() - t2 = re.sub(r"[,.;:!?\n\t\r'\"\-\(\)\[\]\{\}~]", " ", t2) + t2 = _TITLE_PUNCT_RE.sub(" ", t2) result = " ".join(t2.split()) # If unidecode stripped everything (e.g., CJK-only title), fall back to # the original lowercased+collapsed string so similarity can still work. if not result and t_str.strip(): fallback = t_str.lower() - fallback = re.sub(r"[,.;:!?\n\t\r'\"\-\(\)\[\]\{\}~]", " ", fallback) + fallback = _TITLE_PUNCT_RE.sub(" ", fallback) result = " ".join(fallback.split()) return result @@ -246,10 +240,8 @@ def _fix_allcaps_title(s: str) -> str: def trim_title_default(t: str | None) -> str: - """ - Clean up a raw title by trimming whitespace, removing trailing full stops, - preserving genuine ellipses, and normalizing ALL-CAPS titles to title case. - """ + """Trim whitespace, remove trailing full stops (preserving genuine + ellipses), and normalize ALL-CAPS titles to title case.""" if t is None: return "" s = str(t).strip() @@ -273,10 +265,8 @@ def trim_title_default(t: str | None) -> str: def has_placeholder(s: str | None) -> bool: - """ - Detect whether a string looks like a placeholder value such as "n/a", - "unknown", "et al", or a run of dots instead of real content. - """ + """Detect placeholder values such as "n/a", "unknown", "et al", or a + short run of dots instead of real content.""" if s is None: return True s2 = str(s).strip() @@ -290,16 +280,17 @@ def has_placeholder(s: str | None) -> bool: return any(bad in low for bad in ("n/a", "tbd", "unknown", "placeholder")) +_PERSON_PUNCT_RE = re.compile(r"[^a-z0-9\s]") + + def normalize_person_name(n: Any | None) -> str: - """ - Normalize a person name for matching by lowercasing it, stripping accents - and punctuation, and collapsing extra spaces. - """ + """Normalize a person name for matching. Lowercases, strips accents and + punctuation, collapses spaces.""" if not n: return "" n_str = to_text(n) n2 = strip_accents(n_str).lower() - n2 = re.sub(r"[^a-z0-9\s]", " ", n2) + n2 = _PERSON_PUNCT_RE.sub(" ", n2) return " ".join(n2.split()) @@ -336,20 +327,37 @@ def _is_initials_token(token: str) -> bool: return 1 <= len(clean) <= 4 and clean.isalpha() and clean.isupper() and clean.lower() not in _INITIALS_EXCLUSIONS -_RE_NON_ALNUM = r"[^a-z0-9]" +_NON_ALNUM_RE = re.compile(r"[^a-z0-9]") +_NON_ALPHA_RE = re.compile(r"[^a-z]") def name_signature(n: Any | None) -> dict[str, Any] | None: - """ - Derive a compact signature for a person name that keeps the normalized last - name and initials, working with "Last, First", "First Last", and - "Lastname INITIALS" (PubMed/Europe PMC) formats. + """Derive a compact {last, initials} signature for a person name, handling + "Last, First", "First Last", and "Lastname INITIALS" (PubMed/Europe PMC) + formats. Noble particles (van, von, de, etc.) are included in the last name so that "Johan van der Waals" and "van der Waals, Johan" produce the same signature. + + String inputs are memoized (dedup loops resolve the same names O(n^2) + times); each call still returns a fresh dict. """ if not n: return None + if isinstance(n, str): + sig = _name_signature_cached(n) + return dict(sig) if sig is not None else None + return _compute_name_signature(n) + + +@functools.lru_cache(maxsize=4096) +def _name_signature_cached(n: str) -> dict[str, Any] | None: + """Cached name_signature core for hashable (string) names.""" + return _compute_name_signature(n) + + +def _compute_name_signature(n: Any) -> dict[str, Any] | None: + """Uncached name_signature core; see name_signature for the contract.""" n_clean = normalize_person_name(n) if not n_clean: return None @@ -359,7 +367,7 @@ def name_signature(n: Any | None) -> dict[str, Any] | None: rest = parts[1] if len(parts) > 1 else "" rest_tokens = [t for t in normalize_person_name(rest).split() if t] initials = "".join(t[0] for t in rest_tokens if t) - last_norm = re.sub(_RE_NON_ALNUM, "", normalize_person_name(last)) + last_norm = _NON_ALNUM_RE.sub("", normalize_person_name(last)) return {"last": last_norm, "initials": initials} tokens = n_clean.split() if not tokens: @@ -368,8 +376,8 @@ def name_signature(n: Any | None) -> dict[str, Any] | None: # by checking if the last token in the original text is all uppercase raw_tokens = to_text(n).strip().split() if len(tokens) == 2 and len(raw_tokens) == 2 and _is_initials_token(raw_tokens[-1]): - last_norm = re.sub(_RE_NON_ALNUM, "", tokens[0]) - initials = re.sub(r"[^a-z]", "", tokens[1]) + last_norm = _NON_ALNUM_RE.sub("", tokens[0]) + initials = _NON_ALPHA_RE.sub("", tokens[1]) return {"last": last_norm, "initials": initials} last_start = len(tokens) - 1 for i in range(len(tokens) - 2, -1, -1): @@ -379,17 +387,14 @@ def name_signature(n: Any | None) -> dict[str, Any] | None: break last_tokens = tokens[last_start:] first_tokens = tokens[:last_start] - last_norm = re.sub(_RE_NON_ALNUM, "", "".join(last_tokens)) + last_norm = _NON_ALNUM_RE.sub("", "".join(last_tokens)) initials = "".join(t[0] for t in first_tokens if t) return {"last": last_norm, "initials": initials} def extract_last_name(full_name: str | None) -> str: - """ - Extract the last name from a full name string, preserving original capitalization. - Handles both "First Last" and "Last, First" formats. - Returns the original name if extraction fails. - """ + """Extract the last name from "First Last" or "Last, First", preserving + capitalization. Returns the original name if extraction fails.""" if not full_name: return "Unknown" @@ -406,15 +411,16 @@ def extract_last_name(full_name: str | None) -> str: return tokens[-1] if tokens else name_str +_DIRNAME_SANITIZE_RE = re.compile(r'[/\\:*?"<>|]+') + + def format_author_dirname(author_name: str | None, author_id: str) -> str: - """ - Format author directory name as "LastName (author_id)". - Falls back to just author_id if name extraction fails. - If author_id is empty, uses LastName. - """ + """Format an author directory name as "LastName (author_id)". Falls back + to the id alone when name extraction fails, or to LastName when the id is + empty.""" last_name = extract_last_name(author_name) - sanitized_id = re.sub(r'[/\\:*?"<>|]+', "-", author_id) + sanitized_id = _DIRNAME_SANITIZE_RE.sub("-", author_id) if not sanitized_id: if last_name and last_name != "Unknown": @@ -428,36 +434,36 @@ def format_author_dirname(author_name: str | None, author_id: str) -> str: def parse_authors_any(authors: Any) -> list[str]: - """ - Pull author names out of flexible input formats such as lists, dictionaries - with given/family fields, and BibTeX-style strings with different separators. - - This is a convenience wrapper around extract_authors_from_any for simple use cases. - """ + """Extract author names from flexible input formats (lists, dicts with + given/family fields, BibTeX-style strings). Convenience wrapper around + extract_authors_from_any.""" return extract_authors_from_any(authors) -def title_similarity(a: str | None, b: str | None) -> float: - """ - Compute a similarity score between two titles after normalization, returning - a value between 0 and 1 where higher means more similar. +def _normalized_fuzz(a: str, b: str) -> float: + """Fuzzy-ratio score over normalized strings, 0-1, exactly 1.0 on equality. - Uses rapidfuzz for fast fuzzy-ratio scoring. + Shared core of title_similarity and venue_similarity; rapidfuzz returns + 0-100, rescaled here to 0-1. """ - norm_a = normalize_title(a or "") - norm_b = normalize_title(b or "") + norm_a = normalize_title(a) + norm_b = normalize_title(b) if norm_a == norm_b: return 1.0 - # rapidfuzz.fuzz.ratio returns 0-100, normalize to 0-1 return fuzz_ratio(norm_a, norm_b) / 100.0 +def title_similarity(a: str | None, b: str | None) -> float: + """Similarity score between two titles after normalization, 0-1, higher + means more similar.""" + return _normalized_fuzz(a or "", b or "") + + def title_is_truncated_match(a: str | None, b: str | None, min_length: int = 15) -> bool: - """ - Check if one title is a truncated version of the other (a strict prefix or - suffix after normalization). Scholar sometimes truncates long titles from - either end, producing entries like "Passive Co-presence" (prefix truncation) - or "Support Using Semantic GLEAN Workflows" when the full title starts with + """Check if one title is a truncated version of the other (a strict prefix + or suffix after normalization). Scholar truncates long titles from either + end, producing entries like "Passive Co-presence" (prefix truncation) or + "Support Using Semantic GLEAN Workflows" when the full title starts with "Decentralized Web-Based Clinical Decision" (suffix truncation). Requires the shorter title to be at least *min_length* characters to avoid @@ -472,10 +478,8 @@ def title_is_truncated_match(a: str | None, b: str | None, min_length: int = 15) def authors_overlap(authors_a: str | None, authors_b: str | None) -> bool: - """ - Check whether two author lists share at least one person in common by - comparing normalized last names and initials and allowing partial matches. - """ + """Check whether two author lists share at least one person, comparing + normalized last names and initials with partial matches allowed.""" names_a = parse_authors_any(authors_a or "") names_b = parse_authors_any(authors_b or "") if not names_a or not names_b: @@ -549,15 +553,18 @@ def venue_similarity(fields_a: dict[str, Any], fields_b: dict[str, Any]) -> floa b_venue = (fields_b.get("journal") or fields_b.get("booktitle") or fields_b.get("howpublished") or "").strip() if not a_venue or not b_venue: return 0.0 - a_norm = normalize_title(a_venue) - b_norm = normalize_title(b_venue) - if a_norm == b_norm: - return 1.0 - return fuzz_ratio(a_norm, b_norm) / 100.0 + return _normalized_fuzz(a_venue, b_venue) def _is_preprint_fields(fields: dict[str, Any]) -> bool: - """Check if fields look like a preprint based on DOI prefix or journal name.""" + """Check if fields look like a preprint based on DOI prefix or journal name. + + Canonical fields-level predicate. Not interchangeable with the DOI-level + predicate ``id_utils.is_secondary_doi`` (used by ``merge_utils._is_preprint_doi``): + that one also matches ``DATA_DOI_PREFIXES`` (Zenodo, Figshare), while this + one keys on ``PREPRINT_DOI_PREFIXES`` only and additionally consults the + journal name. + """ doi = str(fields.get("doi") or "").lower() if any(doi.startswith(p) for p in PREPRINT_DOI_PREFIXES): return True @@ -608,10 +615,8 @@ def compute_dedup_score(fields_a: dict[str, Any], fields_b: dict[str, Any], coun def author_name_matches(target_author: str | None, authors: Any) -> bool: - """ - Check whether a specific author appears in a candidate author list, preferring - last name plus initials and falling back to looser substring checks when needed. - """ + """Check whether a specific author appears in a candidate author list, + preferring last name plus initials with substring fallbacks.""" if not target_author: return False target_sig = name_signature(target_author) @@ -657,9 +662,8 @@ def author_name_matches(target_author: str | None, authors: Any) -> bool: def author_in_text(target_author: str | None, text: Any) -> bool: - """ - Check whether an author's normalized last name appears as a whole word inside a block of text. - """ + """Check whether an author's normalized last name appears as a whole word + in a block of text.""" if not target_author or not text: return False sig = name_signature(target_author) @@ -670,17 +674,18 @@ def author_in_text(target_author: str | None, text: Any) -> bool: return re.search(rf"\b{re.escape(last_tok)}\b", txt) is not None +_YEAR_RE = re.compile(r"(19|20)\d{2}") + + def extract_year_from_any(obj: Any, field_names: list[str] | None = None, fallback: int | None = None) -> int | None: - """ - Try to recover a four-digit publication year from many possible formats, - including integers, free text, date dictionaries, Crossref-style date parts, - and Unix timestamps, falling back when no plausible year is found. - """ + """Recover a four-digit publication year from integers, free text, date + dicts, Crossref-style date parts, or Unix timestamps, returning *fallback* + when no plausible year is found.""" if isinstance(obj, int): return obj if VALID_YEAR_MIN <= obj <= VALID_YEAR_MAX else fallback if isinstance(obj, str): - m = re.search(r"(19|20)\d{2}", obj) + m = _YEAR_RE.search(obj) if m: try: year = int(m.group(0)) @@ -719,8 +724,6 @@ def extract_year_from_any(obj: Any, field_names: list[str] | None = None, fallba ms = obj.get(fname) if isinstance(ms, (int, float)): try: - from datetime import datetime, timezone - year = datetime.fromtimestamp(float(ms) / 1000.0, timezone.utc).year if VALID_YEAR_MIN <= year <= VALID_YEAR_MAX: return year @@ -733,6 +736,13 @@ def extract_year_from_any(obj: Any, field_names: list[str] | None = None, fallba return fallback +def _join_given_family(d: dict[str, Any], given_key: str, family_key: str) -> str: + """Join explicit given/family fields into "Given Family", or "".""" + given = (d.get(given_key) or "").strip() + family = (d.get(family_key) or "").strip() + return f"{given} {family}".strip() if (given or family) else "" + + def extract_authors_from_any( obj: Any, field_names: list[str] | None = None, @@ -741,10 +751,8 @@ def extract_authors_from_any( given_key: str | None = None, family_key: str | None = None, ) -> list[str]: - """ - Extract a list of author names from flexible metadata structures such as lists, - dicts, and formatted strings, optionally cleaning DBLP-specific name artifacts. - """ + """Extract author names from flexible metadata structures (lists, dicts, + formatted strings), optionally cleaning DBLP-specific name artifacts.""" authors: list[str] = [] if obj is None: @@ -775,12 +783,7 @@ def extract_authors_from_any( if authors: return authors - if given_key and family_key: - given = (obj.get(given_key) or "").strip() - family = (obj.get(family_key) or "").strip() - nm = f"{given} {family}".strip() if (given or family) else "" - else: - nm = _name_from_dict(obj) + nm = _join_given_family(obj, given_key, family_key) if given_key and family_key else _name_from_dict(obj) if nm: nm = _sanitize(nm) if _sanitize else nm @@ -798,9 +801,7 @@ def extract_authors_from_any( authors.append(nm) elif isinstance(item, dict): if given_key and family_key: - given = (item.get(given_key) or "").strip() - family = (item.get(family_key) or "").strip() - nm = f"{given} {family}".strip() if (given or family) else "" + nm = _join_given_family(item, given_key, family_key) else: nm = (item.get(name_key) or "").strip() if not nm: @@ -861,10 +862,8 @@ def extract_authors_from_any( def extract_valid_title(obj: Any, field_names: list[str] | None = None, check_placeholder: bool = True) -> str | None: - """ - Pull a title from an object using common field names, discard placeholder-like - values, and return a trimmed version or None when no usable title is available. - """ + """Pull a title from an object by common field names, discard + placeholder-like values, and return a trimmed version or None.""" title = None if isinstance(obj, dict): @@ -891,10 +890,8 @@ def extract_valid_title(obj: Any, field_names: list[str] | None = None, check_pl def is_valid_value(val: Any, check_placeholder: bool = True) -> bool: - """ - Decide whether a value is worth keeping by rejecting None, empty containers, - and placeholder-like strings when placeholder checking is enabled. - """ + """Reject None, empty containers, and (optionally) placeholder-like + strings.""" if val is None: return False @@ -911,17 +908,13 @@ def is_valid_value(val: Any, check_placeholder: bool = True) -> bool: def filter_valid_fields(fields: dict[str, Any], check_placeholder: bool = True) -> dict[str, Any]: - """ - Remove keys whose values are empty, None, or placeholder-like so the - remaining dictionary contains only useful metadata fields. - """ + """Drop keys whose values are empty, None, or placeholder-like.""" return {k: v for k, v in fields.items() if is_valid_value(v, check_placeholder=check_placeholder)} def is_truncated(text: str | None) -> bool: - """ - Detect if text is truncated by checking for ellipsis, et al., or other truncation markers. - """ + """Detect truncated text via ellipsis, et al., other truncation markers, + or a dangling function-word ending.""" if not text or not isinstance(text, str): return False @@ -935,10 +928,8 @@ def is_truncated(text: str | None) -> bool: def get_truncation_score(article_data: dict[str, Any]) -> float: - """ - Calculate a truncation score for an article by checking key fields, returning - a value between 0.0 (complete) and 1.0 (fully truncated). - """ + """Fraction of key fields that look truncated, 0.0 (complete) to 1.0 + (fully truncated).""" candidates = [ article_data.get("title"), article_data.get("author_info"), @@ -959,10 +950,8 @@ def safe_get_field( required: bool = False, check_placeholder: bool = False, ) -> str | None: - """ - Safely extract and validate a string field from a dictionary, handling None values, - lists, whitespace, and optionally checking for placeholders. - """ + """Extract and validate a string field from a dict, handling None values, + lists, whitespace, and optional placeholder checks.""" value = obj.get(field) if value is None: @@ -988,10 +977,8 @@ def safe_get_field( def safe_get_nested(obj: Any, *keys: str, default: Any = None) -> Any: - """ - Safely get a nested dictionary value with null-safety, traversing multiple keys - and returning a default if any key is missing. - """ + """Traverse nested dict keys null-safely, returning *default* if any key + is missing.""" current = obj for key in keys: if not isinstance(current, dict): @@ -1005,8 +992,7 @@ def safe_get_nested(obj: Any, *keys: str, default: Any = None) -> Any: def extract_author_names( authors_field: Any, *, name_key: str = "name", given_key: str | None = None, family_key: str | None = None ) -> list[str]: - """ - Extract author names from various formats including list of dicts, list of strings, - comma-separated strings, and single dict or string. - """ + """Extract author names from lists of dicts or strings, comma-separated + strings, and single dicts or strings. Keyword wrapper around + extract_authors_from_any.""" return extract_authors_from_any(authors_field, name_key=name_key, given_key=given_key, family_key=family_key) diff --git a/citeforge/textnorm.py b/citeforge/textnorm.py index 20ed7dc3..f148ab98 100644 --- a/citeforge/textnorm.py +++ b/citeforge/textnorm.py @@ -1,9 +1,9 @@ """Title and booktitle text-normalization primitives. Pure functions over pre-compiled regex tables that repair recurrent metadata -defects in scholarly titles and booktitles: fused compound words (hyphens -stripped by Google Scholar), acronym casing, verbose conference metadata, -non-bibliographic "garbage" titles, and DBLP author-name corruption. +defects in scholarly titles and booktitles. Covered defects are fused compound +words (hyphens stripped by Google Scholar), acronym casing, verbose conference +metadata, non-bibliographic "garbage" titles, and DBLP author-name corruption. Depends only on stdlib ``re`` and ``citeforge.config`` (no dependency on ``main`` or the pipeline), so the canonicalization layer and the orchestrator can both build @@ -92,7 +92,7 @@ (re.compile(r"ITiCSE'\d{2}:\s*"), ""), # "SEET-Software" → "SEET - Software" (missing spaces around dash) (re.compile(r"^SEET-Software"), "SEET - Software"), - # Truncated SerpAPI booktitles — complete known conference name suffixes + # Truncated SerpAPI booktitles, completed to known conference names (re.compile(r"Conference on Innovation$"), "Conference on Innovation and Technology in Computer Science Education"), (re.compile(r"Applications of Computer$"), "Applications of Computer Vision"), (re.compile(r"Analyzing and Interpreting$"), "Analyzing and Interpreting Neural Networks for NLP"), @@ -174,7 +174,7 @@ def _fix_fused_compounds(title: str) -> str: 1. Dictionary lookup for special cases (acronyms, irregular patterns). 2. Suffix-based detection for common compound adjective suffixes (e.g. "Knowledgedriven" → "Knowledge-Driven"). - 3. Dictionary lookup again — catches entries newly exposed by the suffix + 3. Dictionary lookup again, catching entries newly exposed by the suffix pass (e.g. "Doubleedgeassisted" → suffix splits to "Doubleedge-Assisted" → dict converts "Doubleedge" to "Double-Edge"). """ diff --git a/citeforge/venue.py b/citeforge/venue.py index 09f8eaf7..8057888c 100644 --- a/citeforge/venue.py +++ b/citeforge/venue.py @@ -8,9 +8,10 @@ from __future__ import annotations import re +from collections.abc import Iterable from typing import Any -from .config import CONFERENCE_AS_JOURNAL, JOURNALS_NAMED_PROCEEDINGS +from .config import CONFERENCE_AS_JOURNAL, GENERIC_SERIES_NAMES, JOURNALS_NAMED_PROCEEDINGS _DAGSTUHL_DOI_RE = re.compile( r"^10\.4230/(lipics|oasics)\.([a-z0-9]+)\.(\d+)(?:\.\d+)?$", @@ -34,7 +35,7 @@ "openrxiv": "openRxiv", } -# Map preprint DOI prefixes → canonical howpublished value. +# Map preprint DOI prefixes to the canonical howpublished value. # Used to backfill howpublished on @misc entries with preprint DOIs. _DOI_PREFIX_TO_HOWPUB: tuple[tuple[str, str], ...] = ( ("10.48550/arxiv", "arXiv"), @@ -58,6 +59,24 @@ ) +def first_non_generic_container(values: Iterable[Any]) -> str | None: + """Return the first element that is non-empty and not a generic series name. + + Shared selection core for multi-element ``container-title`` arrays + (Crossref returns ``[series_name, conference_name]``). Elements are + coerced to ``str`` and stripped; membership in ``GENERIC_SERIES_NAMES`` + is tested case-insensitively. Returns ``None`` when every element is + empty or generic. Fallback resolution, event-name upgrades, and logging + stay at the call sites (``api_generics._extract_venue`` and + ``clients.search_apis.bibtex_from_csl``), whose semantics differ. + """ + for candidate in values: + text = str(candidate).strip() + if text and text.lower() not in GENERIC_SERIES_NAMES: + return text + return None + + def infer_howpublished_from_doi(doi: str) -> str | None: """Return canonical howpublished for a preprint DOI, or None.""" dl = doi.lower() diff --git a/main.py b/main.py index 8a0eb19e..8c282996 100644 --- a/main.py +++ b/main.py @@ -1,15 +1,17 @@ """CiteForge command-line entry point. -Loads the API keys and author records, runs the parallel enrichment scheduler -over every author, then finalizes the run. Accepts an optional input CSV path -(default ``data/input.csv``) and a ``--force`` flag that re-enriches every -record regardless of cache completeness. +Loads the API keys and author records from ``data/input.csv``, runs the +parallel enrichment scheduler over every author, then finalizes the run. +Accepts a ``--force`` flag that re-enriches every record regardless of cache +completeness. """ from __future__ import annotations import os import sys +from collections.abc import Callable +from typing import TypeVar from citeforge.canonicalize import _fixup_bib_entry # noqa: F401 # re-exported for test imports from citeforge.config import ( @@ -38,6 +40,18 @@ from citeforge.pipeline.scheduler import prioritize_records, run_all from citeforge.textnorm import _is_corrupted_title, _is_garbage_title # noqa: F401 # re-exported for test imports +T = TypeVar("T") + + +def _load_optional_key(reader: Callable[[], T], label: str, miss_note: str) -> T: + """Load an optional credential, logging success or the degradation on a miss.""" + value = reader() + if value: + logger.success(f"{label} loaded", category=LogCategory.PLAN) + else: + logger.warn(f"{label} not found; {miss_note}", category=LogCategory.PLAN) + return value + def main() -> int: """Set up the run, load API keys and author records, and process all authors in parallel. @@ -64,29 +78,26 @@ def main() -> int: return 2 logger.success("SerpAPI key loaded", category=LogCategory.PLAN) - serply_key = read_serply_api_key(DEFAULT_SERPLY_KEY_FILE) - if not serply_key: - logger.warn("Serply API key not found; Scholar citation detail will be skipped", category=LogCategory.PLAN) - else: - logger.success("Serply API key loaded", category=LogCategory.PLAN) - - s2_api_key = read_semantic_api_key(DEFAULT_S2_KEY_FILE) - if not s2_api_key: - logger.warn("Semantic Scholar key not found; S2 enrichment disabled", category=LogCategory.PLAN) - else: - logger.success("Semantic Scholar key loaded", category=LogCategory.PLAN) - - or_creds = read_openreview_credentials() - if not or_creds: - logger.warn("OpenReview credentials not found; OpenReview enrichment may be limited", category=LogCategory.PLAN) - else: - logger.success("OpenReview credentials loaded", category=LogCategory.PLAN) - - gemini_api_key = read_gemini_api_key() - if not gemini_api_key: - logger.warn("Gemini API key not found; short titles will use fallback algorithm", category=LogCategory.PLAN) - else: - logger.success("Gemini API key loaded", category=LogCategory.PLAN) + serply_key = _load_optional_key( + lambda: read_serply_api_key(DEFAULT_SERPLY_KEY_FILE), + "Serply API key", + "Scholar citation detail will be skipped", + ) + s2_api_key = _load_optional_key( + lambda: read_semantic_api_key(DEFAULT_S2_KEY_FILE), + "Semantic Scholar key", + "S2 enrichment disabled", + ) + or_creds = _load_optional_key( + read_openreview_credentials, + "OpenReview credentials", + "OpenReview enrichment may be limited", + ) + gemini_api_key = _load_optional_key( + read_gemini_api_key, + "Gemini API key", + "short titles will use fallback algorithm", + ) try: records = read_records(DEFAULT_INPUT) diff --git a/tests/test_fsscan.py b/tests/test_fsscan.py index 68c8aaeb..91a0045c 100644 --- a/tests/test_fsscan.py +++ b/tests/test_fsscan.py @@ -1,10 +1,27 @@ -"""Tests for citeforge.fsscan: deterministic directory-scan helpers.""" +"""Tests for citeforge.fsscan: deterministic directory-scan helpers. + +The ``iter_parsed_author_bibs`` section includes differential tests whose +oracles are verbatim replicas of the two inline scan loops the helper +replaced (``merge_utils.save_entry_to_file`` and the Phase 4 candidate-DOI +dedup in ``pipeline.article``), proving the shared scan+parse core visits the +same files, in the same order, with the same parse results and error +behavior. +""" from __future__ import annotations +import os import pathlib +from typing import Any + +import pytest + +from citeforge.bibtex_utils import parse_bibtex_to_dict +from citeforge.fsscan import iter_author_bibs, iter_output_dirs, iter_parsed_author_bibs -from citeforge.fsscan import iter_author_bibs, iter_output_dirs +_BIB_A = "@article{keyA,\n title = {Alpha Paper},\n doi = {10.1000/a},\n}\n" +_BIB_C = "@article{keyC,\n title = {Gamma Paper},\n doi = {10.1000/c},\n}\n" +_BIB_Z = "@article{keyZ,\n title = {Zeta Paper},\n doi = {10.1000/z},\n}\n" def test_iter_author_bibs_returns_sorted_bib_names_only(tmp_path: pathlib.Path) -> None: @@ -22,3 +39,129 @@ def test_iter_output_dirs_returns_sorted_subdir_names_excluding_files(tmp_path: (tmp_path / "baseline.json").write_text("{}", encoding="utf-8") assert iter_output_dirs(str(tmp_path)) == ["alpha", "zeta"] + + +def _write_scan_fixture(tmp_path: pathlib.Path) -> None: + """Author dir with parseable, unparseable, non-bib, and unreadable entries.""" + (tmp_path / "c.bib").write_text(_BIB_C, encoding="utf-8") + (tmp_path / "a.bib").write_text(_BIB_A, encoding="utf-8") + (tmp_path / "z.bib").write_text(_BIB_Z, encoding="utf-8") + (tmp_path / "garbage.bib").write_text("not bibtex at all", encoding="utf-8") + (tmp_path / "note.txt").write_text("ignored", encoding="utf-8") + # A directory with a .bib name: open() raises IsADirectoryError (OSError) + (tmp_path / "dir.bib").mkdir() + + +def test_iter_parsed_author_bibs_yields_sorted_parseable_entries(tmp_path: pathlib.Path) -> None: + _write_scan_fixture(tmp_path) + + result = list(iter_parsed_author_bibs(str(tmp_path))) + + assert [fname for fname, _, _ in result] == ["a.bib", "c.bib", "z.bib"] + assert [path for _, path, _ in result] == [os.path.join(str(tmp_path), f) for f, _, _ in result] + assert [entry["fields"]["doi"] for _, _, entry in result] == ["10.1000/a", "10.1000/c", "10.1000/z"] + + +def test_iter_parsed_author_bibs_skip_basename(tmp_path: pathlib.Path) -> None: + _write_scan_fixture(tmp_path) + + result = list(iter_parsed_author_bibs(str(tmp_path), skip_basename="c.bib")) + + assert [fname for fname, _, _ in result] == ["a.bib", "z.bib"] + + +def test_iter_parsed_author_bibs_skip_path_matches_by_absolute_identity(tmp_path: pathlib.Path) -> None: + _write_scan_fixture(tmp_path) + # Non-normalized spelling of the same file must still be skipped + dotted = os.path.join(str(tmp_path), ".", "c.bib") + + result = list(iter_parsed_author_bibs(str(tmp_path), skip_path=dotted)) + + assert [fname for fname, _, _ in result] == ["a.bib", "z.bib"] + + +def test_iter_parsed_author_bibs_read_error_invokes_callback_and_skips(tmp_path: pathlib.Path) -> None: + _write_scan_fixture(tmp_path) + seen: list[str] = [] + + result = list(iter_parsed_author_bibs(str(tmp_path), on_read_error=seen.append)) + + assert seen == ["dir.bib"] + assert [fname for fname, _, _ in result] == ["a.bib", "c.bib", "z.bib"] + + +def test_iter_parsed_author_bibs_default_read_errors_do_not_swallow_decode_errors(tmp_path: pathlib.Path) -> None: + (tmp_path / "bad.bib").write_bytes(b"@article{k,\n title = {\xff\xfe},\n}\n") + + with pytest.raises(UnicodeDecodeError): + list(iter_parsed_author_bibs(str(tmp_path))) + + result = list(iter_parsed_author_bibs(str(tmp_path), read_errors=(OSError, UnicodeDecodeError))) + assert result == [] + + +# --------------------------------------------------------------------------- +# Differential oracles: verbatim replicas of the inline loops the helper +# replaced. If these ever disagree with iter_parsed_author_bibs, the +# byte-identical-output invariant is at risk. +# --------------------------------------------------------------------------- + + +def _legacy_save_entry_scan(author_dir: str, prefer_basename: str | None) -> list[tuple[str, str, dict[str, Any]]]: + """Replica of the pre-consolidation loop in merge_utils.save_entry_to_file.""" + visited = [] + for existing_filename in iter_author_bibs(author_dir): + if existing_filename == prefer_basename: + continue + existing_path = os.path.join(author_dir, existing_filename) + try: + with open(existing_path, encoding="utf-8") as ef: + existing_entry = parse_bibtex_to_dict(ef.read()) + except OSError: + continue + if existing_entry: + visited.append((existing_filename, existing_path, existing_entry)) + return visited + + +def _legacy_phase4_scan(author_dir: str, path: str | None) -> list[tuple[str, str, dict[str, Any]]]: + """Replica of the pre-consolidation Phase 4 candidate-DOI loop in pipeline.article.""" + visited = [] + for existing_bib in iter_author_bibs(author_dir): + epath = os.path.join(author_dir, existing_bib) + if path and os.path.abspath(epath) == os.path.abspath(path): + continue + try: + with open(epath, encoding="utf-8") as ef: + edict = parse_bibtex_to_dict(ef.read()) + if not edict: + continue + except (OSError, UnicodeDecodeError): + continue + visited.append((existing_bib, epath, edict)) + return visited + + +@pytest.mark.parametrize("prefer_basename", [None, "c.bib", "missing.bib"]) +def test_iter_parsed_author_bibs_matches_legacy_save_entry_scan( + tmp_path: pathlib.Path, prefer_basename: str | None +) -> None: + _write_scan_fixture(tmp_path) + + legacy = _legacy_save_entry_scan(str(tmp_path), prefer_basename) + shared = list(iter_parsed_author_bibs(str(tmp_path), skip_basename=prefer_basename)) + + assert shared == legacy + + +@pytest.mark.parametrize("self_name", [None, "c.bib", "missing.bib"]) +def test_iter_parsed_author_bibs_matches_legacy_phase4_scan(tmp_path: pathlib.Path, self_name: str | None) -> None: + _write_scan_fixture(tmp_path) + path = os.path.join(str(tmp_path), self_name) if self_name else None + + legacy = _legacy_phase4_scan(str(tmp_path), path) + shared = list( + iter_parsed_author_bibs(str(tmp_path), skip_path=path or None, read_errors=(OSError, UnicodeDecodeError)) + ) + + assert shared == legacy diff --git a/tests/test_textnorm.py b/tests/test_textnorm.py index d2cfb34d..695a0a22 100644 --- a/tests/test_textnorm.py +++ b/tests/test_textnorm.py @@ -76,7 +76,7 @@ def test_apply_booktitle_fixups_idempotent(bt_in: str, bt_out: str) -> None: # --------------------------------------------------------------------------- -# _fix_title_text — colon/hyphen spacing and acronym casing +# _fix_title_text, colon/hyphen spacing and acronym casing # --------------------------------------------------------------------------- # (title_in, title_out) golden pairs captured live from _fix_title_text. diff --git a/tests/test_venue.py b/tests/test_venue.py new file mode 100644 index 00000000..76af4cf3 --- /dev/null +++ b/tests/test_venue.py @@ -0,0 +1,111 @@ +"""Tests for citeforge.venue.first_non_generic_container. + +Includes differential tests whose oracles are verbatim replicas of the two +inline selection expressions the helper replaced (the ``next(...)`` form in +``api_generics._extract_venue`` and the loop form in +``clients.search_apis.bibtex_from_csl``), plus behavior pins on +``bibtex_from_csl`` container selection over representative CSL payloads. +""" + +from __future__ import annotations + +from typing import Any + +import pytest + +from citeforge.clients.search_apis import bibtex_from_csl +from citeforge.config import GENERIC_SERIES_NAMES +from citeforge.venue import first_non_generic_container + +_GENERIC = "Lecture Notes in Computer Science" +_CONFERENCE = "International Conference on Machine Learning" + +_CASES: list[list[Any]] = [ + [_GENERIC, _CONFERENCE], # Crossref [series, conference] shape + [_CONFERENCE, _GENERIC], # non-generic first + [_GENERIC, _GENERIC.upper()], # generic-only (case-insensitive) + ["", " "], # empty and whitespace-only + ["", _CONFERENCE], # empty then usable + [None, 123, _CONFERENCE], # non-string elements coerced via str() + [" Padded Venue ", _GENERIC], # stripping applied to the winner + [], # empty list + [_GENERIC.title()], # single generic element +] + + +def _legacy_extract_venue_selection(raw_venue: list[Any]) -> str | None: + """Replica of the pre-consolidation next(...) in api_generics._extract_venue.""" + return next( + (str(c).strip() for c in raw_venue if str(c).strip() and str(c).strip().lower() not in GENERIC_SERIES_NAMES), + None, + ) + + +def _legacy_bibtex_from_csl_selection(container_raw: list[Any]) -> str | None: + """Replica of the pre-consolidation loop in clients.search_apis.bibtex_from_csl.""" + container = None + for candidate in container_raw: + candidate_str = str(candidate).strip() + if candidate_str and candidate_str.lower() not in GENERIC_SERIES_NAMES: + container = candidate_str + break + return container + + +@pytest.mark.parametrize("values", _CASES) +def test_first_non_generic_container_matches_both_legacy_selections(values: list[Any]) -> None: + result = first_non_generic_container(values) + + assert result == _legacy_extract_venue_selection(values) + assert result == _legacy_bibtex_from_csl_selection(values) + + +def test_first_non_generic_container_never_returns_empty_or_generic() -> None: + for values in _CASES: + result = first_non_generic_container(values) + if result is not None: + assert result == result.strip() + assert result + assert result.lower() not in GENERIC_SERIES_NAMES + + +def _csl(container_title: Any, event: Any = None) -> dict[str, Any]: + payload: dict[str, Any] = { + "title": "A Representative Paper", + "author": [{"given": "Ada", "family": "Lovelace"}], + "issued": {"date-parts": [[2024]]}, + "type": "paper-conference", + "DOI": "10.1000/xyz", + } + if container_title is not None: + payload["container-title"] = container_title + if event is not None: + payload["event"] = event + return payload + + +def test_bibtex_from_csl_prefers_non_generic_container_element() -> None: + bib = bibtex_from_csl(_csl([_GENERIC, _CONFERENCE]), keyhint="k") + + assert _CONFERENCE in bib + assert _GENERIC not in bib + + +def test_bibtex_from_csl_generic_container_falls_back_to_event_name() -> None: + bib = bibtex_from_csl(_csl(_GENERIC, event={"name": "MICCAI 2024"}), keyhint="k") + + assert "MICCAI 2024" in bib + + +def test_bibtex_from_csl_generic_only_array_keeps_first_element_without_event() -> None: + # All-generic array: selection yields None, safe_get_field falls back to + # the first element, and without an event name the generic value stays. + bib = bibtex_from_csl(_csl([_GENERIC, _GENERIC]), keyhint="k") + + assert _GENERIC in bib + + +def test_bibtex_from_csl_missing_container_omits_venue() -> None: + bib = bibtex_from_csl(_csl(None), keyhint="k") + + assert "booktitle" not in bib and "journal" not in bib