Skip to content

Fuzzy reference matching for DOI-less refs (citation graph robustness) #19

Description

@deOliveira-R

Spotted while smoke-testing GROBID on two papers as preconditions for #16:

  • Historical (1982 NSE OCR'd, 28 refs): 0 references have DOIs (pre-DOI era).
  • Modern (Raj et al. 2022, Annals of Nuclear Energy, 38 refs): only 1 reference has an extractable DOI.

Both cases break the planned DOI-only matching strategy in #16. Pre-DOI literature is unmatched by definition; modern papers fail because GROBID's reference parser doesn't reliably extract DOI strings from the printed bibliography even when the cited works do have DOIs (Crossref-style enrichment isn't part of processFulltextDocument).

Implication for #16

If we ship #16 with DOI-only matching, the citation graph will be sparse for both ends of the corpus — exactly the regimes where it would be most useful. ORPHEUS-relevant historical literature on collision probabilities, transport approximations, etc. won't link at all.

Sketch of fuzzy fallback

For each parsed reference:

  1. Try DOI match first (existing Citation graph: fill cites / cited_by from TEI references #16 path).
  2. Fallback: title + first-author-family + year tuple against the local items table.
    • Normalize title: lowercase, strip punctuation, collapse whitespace.
    • Token Jaccard ≥ 0.85 OR fuzz ratio ≥ 90 (whichever cheap match works).
    • Author family-name string-equality (case-insensitive) on first author.
    • Year exact match (or ±1 to account for online-first vs print).
  3. Fallback 2: venue + volume + page for refs that have a journal abbreviation but no clean title.
    • Venue normalized via the existing dedup venue-canonicalization (if any).
    • First-page exact match.

Acceptance

  • Add citations.fuzzy_match(ref, doi_to_id, title_to_id, ...) that returns an item_id or None.
  • Wire it into citations.build as a fallback when DOI lookup misses.
  • Confidence on fuzzy-matched edges is < 1.0 (e.g. 0.7) so downstream consumers can filter.
  • Tests: positive match for title+author+year, negative for near-misses, idempotency on re-run.
  • Re-run on the full library after Citation graph: fill cites / cited_by from TEI references #16 ships and report DOI-match vs fuzzy-match edge counts.

Notes

  • GROBID 0.8.1 has consistent per-paper output but the DOI-extraction recall in references is the documented weak point. Crossref reverse-lookup by reference text is an option (POST /works?query.bibliographic=...) but adds an external dependency and quota.
  • The TEI parser bug fixed in OCR fallback via kind='ocr_text' artifact (concrete path for #13 OCR item) #18-comment thread today (titles falling back to monogr/title when analytic/title is empty) makes title-based fuzzy matching meaningfully more robust on historical references.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions