Ingestion pipelines for external (non-Avanti) public datasets that we load into BigQuery for analysis. One folder per upstream source.
| Source | Folder | Status | Description |
|---|---|---|---|
| PLFS (India) | plfs/ |
Active | Periodic Labour Force Survey — MoSPI unit-level microdata across 11 releases (2018-19 → CY2025). Heavy local parsing → 6 plfs_* tables. |
| NIRF (India) | nirf/ |
Active | National Institutional Ranking Framework — rankings + admissions/placements/strength data, top-200 institutes across 9 disciplines, 2016-2025. Light pipeline (parquet → GCS → BQ) → 4 nirf_fact_* tables. |
| JNV (entrance + board) | jnv/ |
Active | JEE Mains + Advanced, NEET, JNVST selection test (2018), EI Asset Test, and CBSE 10th/12th board results for Jawahar Navodaya Vidyalaya students. → 6 tables: jnv_fact_jee_results, jnv_fact_neet_results, jnv_fact_selection_test_results, jnv_fact_ei_asset_test_results, jnv_fact_board_results_10th, jnv_fact_board_results_12th. |
| UDISE+ (India) | udise/ |
Active | Unified District Information System for Education — school enrolment by state × management × category × location × class × gender, AY 2024-25 (dashboard export, reshaped wide→long). Parse → GCS → BQ → 1 udise_fact_enrolment table. |
| Dakshana | dakshana/ |
Active | (1) Navodaya CoE Selection Test (NCST) results — scores, demographics, and coaching preferences for JNV students across 2022–2025, conducted jointly by Dakshana Foundation, ENF, and Avanti; (2) Dakshana's self-reported JEE-Main + NEET result sheets — the authoritative record of who was a Dakshana student and at which CoE. → 2 tables: dakshana_fact_ncst_results, dakshana_fact_reported_results. |
| NVS NCST | nvs/ |
Active | NCST results from 2026 onward — first year the test was conducted at national scale by NVS directly (Dakshana set the paper). ~43k students with richer data: per-subject raw scores, household wealth, academic history. Block 1 columns match dakshana_fact_ncst_results for cross-year UNIONs. → 1 table: nvs_fact_ncst_results. |
| UBSE (Uttarakhand) | ubse/ |
Active | Uttarakhand Board of School Education Grade 10 and Grade 12 board exam results (2026+). Long format — one row per student per subject. → 2 tables: ubse_fact_grade10_results, ubse_fact_grade12_results. |
| Punjab SCERT | pb_scert/ |
Active | SOE & RSMS Admission Test merit list for Punjab Schools of Eminence and Meritorious Schools (Class 11 admissions, AY 2024-25 to 2026-27, ~326k records). → 1 table: pb_scert_fact_soe_rsms_admission_merit_list. |
| JoSAA | josaa/ |
Active | JoSAA engineering seat-allotment cutoffs — opening and closing ranks for every IIT/NIT/IIIT/GFTI seat bucket, all rounds, 2016 → latest (464,496 rows). → 1 table: josaa_fact_cutoffs. |
| NAS (India) | nas/ |
Active | National Achievement Survey 2021 (NCERT) — student achievement performance and proficiency for grades 3/5/8/10 at state level, by management and location. Student-weighted national rollup included as state='All India'. → 1 table: nas_fact_state_proficiency. |
| AICTE (India) | aicte/ |
Active | AICTE technical-education intake data — approved intake, enrolment, passouts, placements, and institution counts for 6 streams (Engineering, Management, MCA, Pharmacy, Architecture, Hotel Management), AY 2012-13 → 2022-23. Three cuts: national, state, institution-type. → 1 table: aicte_fact_intake. |
| NMC (India) | nmc/ |
Active | National Medical Commission MBBS seat matrix 2024-25 — all ~780 MBBS medical colleges (incl. AIIMS & JIPMER) with state, district, university, management, year of inception, and annual UG seat intake. PDF parse → GCS → BQ. → 1 table: nmc_fact_mbbs_seats. |
| MoE Board Results | moe/ |
Active | Indian school board exam results (Class X + XII) from MoE RSHSE PDFs — pass figures across all 41–58 boards by year, gender, social category, and Class XII stream (2020–2024, no 2023 edition). PDF parse → GCS → BQ. → 1 table: moe_fact_board_exam_results. |
| AISHE (India) | aishe/ |
Active | All India Survey on Higher Education (MoE). Two pipelines: (1) Student enrolment + graduates from Final Report workbooks (Tables 33, 34a, 12+35) → aishe_fact_higher_ed_students; (2) Live institution directory from the AISHE HE Directory dashboard → 5 dim tables: aishe_dim_colleges (53k), aishe_dim_universities (1.4k), aishe_dim_standalone_institutions (17k), aishe_dim_research_institutions (279), aishe_dim_pm_vidyalaxmi_eligible_institutions (1k). |
All sources write into a single BigQuery dataset
avantifellows.external_data_sources (region asia-south1), with tables
prefixed by source: plfs_*, nirf_*, etc.
Future candidates: NFHS (health), ASER (learning levels), Census 2011 (and 2021 when released). Add a new folder per source.
Each <source>/ folder is self-contained: its own README, its own CLAUDE.md,
its own scripts, its own raw + clean data. No shared code across sources —
they vary too much for that to be useful at this scale.
Pick the closer skeleton:
plfs/— if upstream is custom-format and needs real parsing logic in this repo. Hasraw/,clean/,codemaps/, parsing scripts.nirf/— if upstream is already analyst-clean (parquet/CSV) and just needs staging through GCS. Smaller:raw/,schemas/,scripts/.
- One shared BigQuery dataset. All sources write into
avantifellows.external_data_sources.*(regionasia-south1), with tables prefixed by source:<source>_fact_*,<source>_dim_*. - GCS staging at
gs://avantifellows-external-data/<source>/for sources whose raw is already in a BQ-loadable format. Other sources keep raw + clean on local disk and load via pandas. - One-shot loader (
scripts/load_bq.py) — no orchestrator, no scheduler. These pipelines run on demand when upstream publishes a new release. - Raw data gitignored, reference docs committed. Public manuals/layouts belong in git; data files don't.
- Codemaps committed as CSV (when applicable). Even if generated by a script, the CSV is the consumed artifact and the common interface across sources.
- Schemas as YAML in
<source>/schemas/, one file per output table, named to match (nirf_fact_rankings.yaml→nirf_fact_rankings). - Python via
uvorvenvper source. Each source has its own dependencies; don't share a root.venv.
The repo root holds:
- this README,
CLAUDE.md(Claude orientation across the monorepo),- a thin
.gitignorefor cross-cutting noise (.DS_Store,__pycache__/,.venv/,.claude/).
Everything else lives under a source folder.