Skip to content

docs: rewrite README plain and sober #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gabrielspadon merged 1 commit into from
Jul 5, 2026
Merged

docs: rewrite README plain and sober #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 1 addition & 12 deletions .github/workflows/monthly-refresh.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,18 +168,7 @@ jobs:
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"

# Update README badges from badges.json
if [ -f output/badges.json ]; then
LAST_UPDATED=$(python3 -c "import json; print(json.load(open('output/badges.json'))['last_updated'])")
TOTAL_QUERIES=$(python3 -c "import json; print(json.load(open('output/badges.json'))['total_queries'])")
HIT_RATE=$(python3 -c "import json; print(json.load(open('output/badges.json'))['hit_rate'])")
LAST_UPDATED_ESC=$(echo "$LAST_UPDATED" | sed 's/-/--/g')
sed -i "s|Updated-[0-9]*--[0-9]*-blue|Updated-${LAST_UPDATED_ESC}-blue|" README.md
sed -i "s|Queries-[0-9]*-8A2BE2|Queries-${TOTAL_QUERIES}-8A2BE2|" README.md
sed -i "s|Cache_Hit_Rate-[0-9.]*%25-2ea44f|Cache_Hit_Rate-${HIT_RATE}%25-2ea44f|" README.md
fi

git add output/ README.md
git add output/
if ! git diff --cached --quiet; then
git commit -m "data: monthly pipeline refresh $(date +%Y-%m)"
git push origin main
Expand Down
66 changes: 20 additions & 46 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,40 +1,14 @@
<h1 align="center">CiteForge</h1>
# CiteForge

<p align="center">
<a href="https://github.com/gabrielspadon/CiteForge/actions/workflows/tests.yml"><img src="https://github.com/gabrielspadon/CiteForge/actions/workflows/tests.yml/badge.svg" alt="Tests"></a>
<a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/Python-3.10+-blue.svg" alt="Python 3.10+"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License"></a>
<br>
<img src="https://img.shields.io/badge/Updated-2026--07-blue.svg" alt="Last Updated">
<img src="https://img.shields.io/badge/Queries-8658-8A2BE2.svg" alt="Total Queries">
<img src="https://img.shields.io/badge/Cache_Hit_Rate-77.2%25-2ea44f.svg" alt="Cache Hit Rate">
</p>
[![Tests](https://github.com/gabrielspadon/CiteForge/actions/workflows/tests.yml/badge.svg)](https://github.com/gabrielspadon/CiteForge/actions/workflows/tests.yml)

<p align="center">
<strong>Automated academic citation enrichment from multiple scholarly APIs.</strong><br>
Given a list of authors, CiteForge fetches, validates, deduplicates,<br>
and merges bibliographic metadata into per-author BibTeX files.
</p>
CiteForge builds per-author BibTeX files from scholarly APIs. Given a CSV of authors with Google Scholar profiles, it retrieves their publications, enriches each entry from multiple registries, deduplicates, and merges fields according to source trust.

---
Google Scholar entries are often incomplete, with missing DOIs, inconsistent venue names, and malformed author lists. Correcting them requires cross-referencing registries such as Crossref, Semantic Scholar, arXiv, and PubMed, which does not scale to a research group. CiteForge automates that cross-referencing and consolidation.

## Motivation
## Installation

Google Scholar profiles index an author's publications, but the entries are often incomplete, with missing DOIs, inconsistent venue names, and malformed author lists. Correcting them requires cross-referencing registries such as Crossref, Semantic Scholar, arXiv, and PubMed, which is impractical to do by hand at the scale of a research group. CiteForge automates this cross-referencing and consolidation.

Given an input CSV of authors with Google Scholar profiles, CiteForge

- pulls each author's publications from Scholar via [SerpAPI](https://serpapi.com/) and citation details via [Serply](https://serply.io/);
- queries additional academic APIs for metadata on each paper;
- deduplicates results using fuzzy title matching, DOI normalization, and author overlap;
- merges fields using a multi-level trust hierarchy that prefers authoritative sources; and,
- writes `.bib` files organized by author.

Output is deterministic on cache-hit runs.

## Getting Started

Requires **Python 3.10+** and a [SerpAPI](https://serpapi.com/) key (the free tier is sufficient for small runs).
Requires Python 3.10 or later.

```bash
git clone https://github.com/gabrielspadon/CiteForge.git && cd CiteForge
Expand All @@ -52,7 +26,9 @@ echo "your_gemini_key" > keys/Gemini.key # Optional
printf "user\npass" > keys/OpenReview.key # Optional
```

Then create the input CSV and run the pipeline.
## Usage

Create the input CSV and run the pipeline from the project root.

```bash
python3 main.py # Input: data/input.csv
Expand All @@ -79,33 +55,31 @@ output/
└── ...
```

API responses are cached under `data/api_cache/` with monthly expiry, so subsequent runs are fast and deterministic.
API responses are cached under `data/api_cache/` with monthly expiry.

## How It Works

CiteForge retrieves each author's publication list from Google Scholar through SerpAPI, then enriches every entry by querying scholarly services including Semantic Scholar, Crossref, arXiv, OpenAlex, and PubMed.

A trust-based consolidation stage merges the collected records according to source reliability, prioritizing authoritative registries over scraped content. Duplicate detection combines DOI normalization, external identifier matching, and fuzzy title similarity to identify overlapping entries across sources.
A trust-based consolidation stage merges the collected records according to source reliability, prioritizing authoritative registries over scraped content. Duplicate detection combines DOI normalization, external identifier matching, and fuzzy title similarity. The pipeline also corrects recurrent metadata issues such as fragmented compound words, misclassified publication types, invalid page ranges, and all-capitals titles.

The pipeline also corrects recurrent metadata issues such as fragmented compound words, misclassified publication types, invalid page ranges, and all-capitals titles. Deterministic caching ensures that cache-hit executions produce byte-identical outputs, verified through SHA-256 checksums.

Author queries run in parallel under per-API rate limits. All configurable parameters, including source trust order, similarity thresholds, rate limits, and venue mappings, are centralized in [`citeforge/config.py`](citeforge/config.py).
Cache-hit runs produce byte-identical output. Author queries run in parallel under per-API rate limits. Configurable parameters (source trust order, similarity thresholds, rate limits, venue mappings) are centralized in [`citeforge/config.py`](citeforge/config.py).

## Data Sources

SerpAPI requires a key; the remaining sources are keyless, recommended, or optional.

- **Required (key needed):** [SerpAPI](https://serpapi.com/) (Google Scholar);
- **Recommended (key needed):** [Serply](https://serply.io/) (citation details), [Semantic Scholar](https://www.semanticscholar.org/);
- **Free (no key):** [Crossref](https://www.crossref.org/), [OpenAlex](https://openalex.org/), [arXiv](https://arxiv.org/), [PubMed](https://pubmed.ncbi.nlm.nih.gov/), [Europe PMC](https://europepmc.org/), [DataCite](https://datacite.org/), [ORCID](https://orcid.org/), [DBLP](https://dblp.org/); and,
- **Optional (key needed):** [OpenReview](https://openreview.net/), [Google Gemini](https://ai.google.dev/).
| Tier | Sources |
|------|---------|
| Required (key) | [SerpAPI](https://serpapi.com/) (Google Scholar) |
| Recommended (key) | [Serply](https://serply.io/) (citation details), [Semantic Scholar](https://www.semanticscholar.org/) |
| Free (no key) | [Crossref](https://www.crossref.org/), [OpenAlex](https://openalex.org/), [arXiv](https://arxiv.org/), [PubMed](https://pubmed.ncbi.nlm.nih.gov/), [Europe PMC](https://europepmc.org/), [DataCite](https://datacite.org/), [ORCID](https://orcid.org/), [DBLP](https://dblp.org/) |
| Optional (key) | [OpenReview](https://openreview.net/), [Google Gemini](https://ai.google.dev/) |

Set `CROSSREF_MAILTO` to join Crossref's polite pool (faster responses).
Set `CROSSREF_MAILTO` to join Crossref's polite pool.

## Citation

If you use CiteForge in your research, cite it with the entry below.

```bibtex
@software{spadon_citeforge,
author = {Spadon, Gabriel},
Expand All @@ -120,4 +94,4 @@ If you use CiteForge in your research, cite it with the entry below.

## License

This project is licensed under the [MIT License](LICENSE).
[MIT](LICENSE).
Loading