diff --git a/.github/workflows/monthly-refresh.yml b/.github/workflows/monthly-refresh.yml index fbcb5b65..86071bcb 100644 --- a/.github/workflows/monthly-refresh.yml +++ b/.github/workflows/monthly-refresh.yml @@ -168,18 +168,7 @@ jobs: git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com" - # Update README badges from badges.json - if [ -f output/badges.json ]; then - LAST_UPDATED=$(python3 -c "import json; print(json.load(open('output/badges.json'))['last_updated'])") - TOTAL_QUERIES=$(python3 -c "import json; print(json.load(open('output/badges.json'))['total_queries'])") - HIT_RATE=$(python3 -c "import json; print(json.load(open('output/badges.json'))['hit_rate'])") - LAST_UPDATED_ESC=$(echo "$LAST_UPDATED" | sed 's/-/--/g') - sed -i "s|Updated-[0-9]*--[0-9]*-blue|Updated-${LAST_UPDATED_ESC}-blue|" README.md - sed -i "s|Queries-[0-9]*-8A2BE2|Queries-${TOTAL_QUERIES}-8A2BE2|" README.md - sed -i "s|Cache_Hit_Rate-[0-9.]*%25-2ea44f|Cache_Hit_Rate-${HIT_RATE}%25-2ea44f|" README.md - fi - - git add output/ README.md + git add output/ if ! git diff --cached --quiet; then git commit -m "data: monthly pipeline refresh $(date +%Y-%m)" git push origin main diff --git a/README.md b/README.md index d3feb7d2..e5263b54 100644 --- a/README.md +++ b/README.md @@ -1,40 +1,14 @@ -

CiteForge

+# CiteForge -

- Tests - Python 3.10+ - License -
- Last Updated - Total Queries - Cache Hit Rate -

+[![Tests](https://github.com/gabrielspadon/CiteForge/actions/workflows/tests.yml/badge.svg)](https://github.com/gabrielspadon/CiteForge/actions/workflows/tests.yml) -

- Automated academic citation enrichment from multiple scholarly APIs.
- Given a list of authors, CiteForge fetches, validates, deduplicates,
- and merges bibliographic metadata into per-author BibTeX files. -

+CiteForge builds per-author BibTeX files from scholarly APIs. Given a CSV of authors with Google Scholar profiles, it retrieves their publications, enriches each entry from multiple registries, deduplicates, and merges fields according to source trust. ---- +Google Scholar entries are often incomplete, with missing DOIs, inconsistent venue names, and malformed author lists. Correcting them requires cross-referencing registries such as Crossref, Semantic Scholar, arXiv, and PubMed, which does not scale to a research group. CiteForge automates that cross-referencing and consolidation. -## Motivation +## Installation -Google Scholar profiles index an author's publications, but the entries are often incomplete, with missing DOIs, inconsistent venue names, and malformed author lists. Correcting them requires cross-referencing registries such as Crossref, Semantic Scholar, arXiv, and PubMed, which is impractical to do by hand at the scale of a research group. CiteForge automates this cross-referencing and consolidation. - -Given an input CSV of authors with Google Scholar profiles, CiteForge - -- pulls each author's publications from Scholar via [SerpAPI](https://serpapi.com/) and citation details via [Serply](https://serply.io/); -- queries additional academic APIs for metadata on each paper; -- deduplicates results using fuzzy title matching, DOI normalization, and author overlap; -- merges fields using a multi-level trust hierarchy that prefers authoritative sources; and, -- writes `.bib` files organized by author. - -Output is deterministic on cache-hit runs. - -## Getting Started - -Requires **Python 3.10+** and a [SerpAPI](https://serpapi.com/) key (the free tier is sufficient for small runs). +Requires Python 3.10 or later. ```bash git clone https://github.com/gabrielspadon/CiteForge.git && cd CiteForge @@ -52,7 +26,9 @@ echo "your_gemini_key" > keys/Gemini.key # Optional printf "user\npass" > keys/OpenReview.key # Optional ``` -Then create the input CSV and run the pipeline. +## Usage + +Create the input CSV and run the pipeline from the project root. ```bash python3 main.py # Input: data/input.csv @@ -79,33 +55,31 @@ output/ └── ... ``` -API responses are cached under `data/api_cache/` with monthly expiry, so subsequent runs are fast and deterministic. +API responses are cached under `data/api_cache/` with monthly expiry. ## How It Works CiteForge retrieves each author's publication list from Google Scholar through SerpAPI, then enriches every entry by querying scholarly services including Semantic Scholar, Crossref, arXiv, OpenAlex, and PubMed. -A trust-based consolidation stage merges the collected records according to source reliability, prioritizing authoritative registries over scraped content. Duplicate detection combines DOI normalization, external identifier matching, and fuzzy title similarity to identify overlapping entries across sources. +A trust-based consolidation stage merges the collected records according to source reliability, prioritizing authoritative registries over scraped content. Duplicate detection combines DOI normalization, external identifier matching, and fuzzy title similarity. The pipeline also corrects recurrent metadata issues such as fragmented compound words, misclassified publication types, invalid page ranges, and all-capitals titles. -The pipeline also corrects recurrent metadata issues such as fragmented compound words, misclassified publication types, invalid page ranges, and all-capitals titles. Deterministic caching ensures that cache-hit executions produce byte-identical outputs, verified through SHA-256 checksums. - -Author queries run in parallel under per-API rate limits. All configurable parameters, including source trust order, similarity thresholds, rate limits, and venue mappings, are centralized in [`citeforge/config.py`](citeforge/config.py). +Cache-hit runs produce byte-identical output. Author queries run in parallel under per-API rate limits. Configurable parameters (source trust order, similarity thresholds, rate limits, venue mappings) are centralized in [`citeforge/config.py`](citeforge/config.py). ## Data Sources SerpAPI requires a key; the remaining sources are keyless, recommended, or optional. -- **Required (key needed):** [SerpAPI](https://serpapi.com/) (Google Scholar); -- **Recommended (key needed):** [Serply](https://serply.io/) (citation details), [Semantic Scholar](https://www.semanticscholar.org/); -- **Free (no key):** [Crossref](https://www.crossref.org/), [OpenAlex](https://openalex.org/), [arXiv](https://arxiv.org/), [PubMed](https://pubmed.ncbi.nlm.nih.gov/), [Europe PMC](https://europepmc.org/), [DataCite](https://datacite.org/), [ORCID](https://orcid.org/), [DBLP](https://dblp.org/); and, -- **Optional (key needed):** [OpenReview](https://openreview.net/), [Google Gemini](https://ai.google.dev/). +| Tier | Sources | +|------|---------| +| Required (key) | [SerpAPI](https://serpapi.com/) (Google Scholar) | +| Recommended (key) | [Serply](https://serply.io/) (citation details), [Semantic Scholar](https://www.semanticscholar.org/) | +| Free (no key) | [Crossref](https://www.crossref.org/), [OpenAlex](https://openalex.org/), [arXiv](https://arxiv.org/), [PubMed](https://pubmed.ncbi.nlm.nih.gov/), [Europe PMC](https://europepmc.org/), [DataCite](https://datacite.org/), [ORCID](https://orcid.org/), [DBLP](https://dblp.org/) | +| Optional (key) | [OpenReview](https://openreview.net/), [Google Gemini](https://ai.google.dev/) | -Set `CROSSREF_MAILTO` to join Crossref's polite pool (faster responses). +Set `CROSSREF_MAILTO` to join Crossref's polite pool. ## Citation -If you use CiteForge in your research, cite it with the entry below. - ```bibtex @software{spadon_citeforge, author = {Spadon, Gabriel}, @@ -120,4 +94,4 @@ If you use CiteForge in your research, cite it with the entry below. ## License -This project is licensed under the [MIT License](LICENSE). +[MIT](LICENSE).