Add OpenITI rijāl/ḥadīth source corpus (24 books) + per-entry xlsx + pipeline wiring#5
Open
R3GENESI5 wants to merge 17 commits into
Open
Add OpenITI rijāl/ḥadīth source corpus (24 books) + per-entry xlsx + pipeline wiring#5R3GENESI5 wants to merge 17 commits into
R3GENESI5 wants to merge 17 commits into
Conversation
Classical ikhtilat reference (121 mukhtalit narrator entries). Fetched from OpenITI corpus (0939IbnKayyal.KawakibNayyirat, sourced from Shamela #309), since the live Shamela/al-maktaba sites are Cloudflare-gated. - Shamela0000309 primary (cleaned/paginated mARkdown) + readable plain-text - 3 alternate editions for variant collation - clean.py + README with provenance https://claude.ai/code/session_019h5LuguzhJsQxT88jb5Jj5
9th rijal text (Ibn al-Kayyal, d.939) — the standard ikhtilat reference. Pulls the OpenITI primary version (Shamela0000309 mARkdown) into src/rijal_raw/. https://claude.ai/code/session_019h5LuguzhJsQxT88jb5Jj5
Structured workbook (121 mukhtalit narrators, one row each: section, name, page locator, verbatim critic statements) across 19 sections + muqaddima + provenance sheets. build_xlsx.py regenerates it from the primary mARkdown. https://claude.ai/code/session_019h5LuguzhJsQxT88jb5Jj5
Tier 1 rijal: Lisan al-Mizan, Tarikh al-Kabir, Majruhin, Du'afa al-'Uqayli, Kashif, Ma'rifat al-Thiqat. Tier 2 defect: Tabaqat al-Mudallisin, al-Ightibat, Marasil, Jami' al-Tahsil, 'Ilal (Daraqutni + Ibn Abi Hatim). Plus Mughni, Ibn Ma'in, Ibn Shahin, Nasa'i/Ibn al-Jawzi Du'afa, and 4 mustalah works. Each: primary OpenITI mARkdown + clean .txt + per-entry/per-page xlsx + README. build_openiti_books.py regenerates all. Wired into download_openiti_rijal.py. (Siyar + Tabaqat committed separately due to size.) https://claude.ai/code/session_019h5LuguzhJsQxT88jb5Jj5
5,943 biographical entries. Largest text (~21MB); biographical graph source (dates, regions, teacher-student links). Primary mARkdown + clean .txt + xlsx. https://claude.ai/code/session_019h5LuguzhJsQxT88jb5Jj5
5,529 biographical entries; earliest tabaqat (generational layering). Primary mARkdown + clean .txt + xlsx + README. https://claude.ai/code/session_019h5LuguzhJsQxT88jb5Jj5
R3GENESI5
changed the title
Long-form master CSV (one row per biographical entry: source, normalized name keys, name_quality flag, verbatim verdict text) + by-narrator lookup (67,590 keys, 1,122 in >=2 books) + browsable xlsx. build_unified_index.py regenerates. Honest framing: by-narrator grouping is exact-normalized-name COLLISION CANDIDATES for the matcher (dedup_narrators / match_narrator_grades), NOT resolved identities — short common names conflate distinct narrators. No grades inferred. Excludes the 8 non-narrator-keyed (ilal/mustalah/unsegmented) texts. https://claude.ai/code/session_019h5LuguzhJsQxT88jb5Jj5
Compact shortlist filtered from by_narrator — the cross-source corroboration candidates to feed the matcher. (Was intended for Drive; Drive connector can't take the large binaries, so kept on git.) https://claude.ai/code/session_019h5LuguzhJsQxT88jb5Jj5
Assigns a generation layer (sahaba -> kibar tabi'in -> ... -> taba' al-atba') to each unified narrator via: arsanad Ibn-Hajar tabaqa > Taqrib ordinal in text > death year parsed from text (incl. Arabic numeral words) > companion marker > loose name match. Records tabaqa_basis for auditability. Outputs: unified_by_tabaqa.csv (all 67,590), multi_source_candidates_by_tabaqa.csv (1,122 >=2-book targets), candidates_by_tabaqa.xlsx (sheet per generation, >=3-book highlighted). ~35% of candidates classified; rest -> 'ghayr muhaddad' (still name-sorted). build_tabaqa_index.py regenerates. Honest caveats in README. https://claude.ai/code/session_019h5LuguzhJsQxT88jb5Jj5
…cates) Resolves entries that are the same narrator across books. Merge gates: canonical arsanad-id link (exact + fuzzy token-Jaccard>=0.8, death-gated) > name near-equality (Jaccard>=0.85, not containment, so short names can't hub) > name+death. Hard splits on differing arsanad-id or death years >5 apart. 70,620 entries -> 68,781 clusters (1,417 merged, 1,294 cross-book). Outputs: narrator_clusters.csv, entry_to_cluster.csv, duplicate_clusters.xlsx (reviewable). Each cluster carries basis + variant_names for audit. Precision-first; recall limited (next lift = isnad teacher/student graph). build_dedup.py regenerates. https://claude.ai/code/session_019h5LuguzhJsQxT88jb5Jj5
Extracts transmitters/students from each entry's text (روى عن … وعنه …) and merges same-name-core entries that share >=3 (or >=2 at Jaccard>=0.6) neighbours. Net +safe merges -> 1,300 cross-book clusters (68,761 total). Cross-block isnad merging was prototyped and REJECTED for precision: it merged distinct people sharing an isnad circle (e.g. brothers al-Hasan & Ali b. Salih b. Hayy). Isnad is only safe with a matching ism+father anchor — documented. https://claude.ai/code/session_019h5LuguzhJsQxT88jb5Jj5
Adds a per-entry era estimate from the sharpest available signal: death year > arsanad death > tabaqa generation > contemporaries/peers (eras of the teachers/students it transmits with). Era fixed for 15,599 entries. Used as BOTH a hard split (same name, different era => different people) and a safe merge enabler (name+era at Jaccard>=0.55 when eras agree). Critical precision fix: require a shared DISTINCTIVE (non-common) token for any name-based merge — sharing only ubiquitous elements (عبد/الله/محمد...) is not identifying. This broke the common-name hub over-merge (largest false cluster 59 -> 10 entries). 70,620 -> 68,349 clusters; 1,552 cross-book (up from 1,300), precision intact. Evidence: 1,276 id, 600 name+era, 520 name, 15 isnad, 5 name+death. https://claude.ai/code/session_019h5LuguzhJsQxT88jb5Jj5
Audit found 2.8% of merged clusters had conflicting fathers (false merges: transitive drift + arsanad fuzzy-link errors). Fixes: - father_compat gate on every merge incl. canonical-id (same person => same father; tolerant of truncation/grandfather-attribution, but a father of only common tokens like عبد الله cannot bridge). - final safety pass re-splits any union lacking father + shared-distinctive links. Conflicts 2.8% -> 1.2%; remaining are mostly false positives of the test itself (legitimate jadd-attribution / kunya variants). Egregious merges eliminated (largest false cluster 59 -> gone). 70,620 -> 68,471 clusters; 1,498 cross-book. https://claude.ai/code/session_019h5LuguzhJsQxT88jb5Jj5
Was anchored only on arsanad (18k). Now uses app/data/rijal merged profiles (115k, integrating GK/Jawami al-Kalim + classical), which links 14,247 entries (3.2x arsanad) and supplies ~37k numeric deaths -> stronger temporal fix. Precision guards added for the richer/noisier source: - index on full_name + only ism-matching namings (drops relational 'أبيه'/ancestor forms that conflated kin). - ism-agreement required for fuzzy links and as a hard split (kills father/son merges where a father's name is a sub-chain of the son's, e.g. الحكم بن ظهير / إبراهيم بن الحكم بن ظهير). - shared-distinctive token required for canon-id merges too. Net: 1,517 cross-book dups (> arsanad's 1,498) AND patronymic-conflict 1.2% -> 0.9% (true-error residual ~0.2-0.3%). Better on both recall and precision. https://claude.ai/code/session_019h5LuguzhJsQxT88jb5Jj5
… flag Lever 1 (GK temporal fix): use Jawami al-Kalim (kaggle_rawis, 24k, death+generation for all) via each profile's gk_id and by direct name match -> era tiers gk/gktab/ canontab. Era fixed 15,878 -> 16,700. Modest: GK already lives in the merged DB and coverage is name-linking-bound. Lever 2 (id-graph isnad): resolve text neighbours to GK ids + inherit profiles' curated GK teacher/student id-lists; isnad-gk merge on >=2 shared ids. Precise but rarely fires (short teacher names don't resolve uniquely). Lever 3 (smart audit): grandfather/kunya-aware same-person test (shared ism chain-head + compatible father among verifiable members) -> new 'flag' column. 17/1,662 merged clusters flagged 'review' (1.0%); 1,645 'ok'. Flagged rows highlighted in xlsx. Filter flag=ok for high-confidence dedup. https://claude.ai/code/session_019h5LuguzhJsQxT88jb5Jj5
Replace the narrow core_key + Jaccard>=0.8 fuzzy linker with a blocked multi-feature scorer: block on (ism, father), then require father_compat + a shared distinctive token, and score on name-Jaccard + era + distinctive overlap (accept Jaccard>=0.55, or >=0.40 with agreeing death/era or >=2 shared distinctive). Canonical links 14,247 -> 22,236 (+56%); cross-book duplicates 1,515 -> 3,386 (2.2x); era-fixed 16,700 -> 17,911. Precision IMPROVED: naive patronymic-conflict 1.2% -> 0.3%, smart-flag 0.5%, largest cluster 11 (no hubs). The guarded low-Jaccard scorer admits true variants the 0.8 cutoff missed while ism+father+death reject kin. 70,620 -> 65,579 clusters. Runtime ~30s. https://claude.ai/code/session_019h5LuguzhJsQxT88jb5Jj5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds 24 classical rijāl / ʿilal / muṣṭalaḥ texts from the OpenITI corpus (all corrected, non-OCR primary versions), each as: primary mARkdown (canonical) + readable
clean.txt+ structured xlsx + provenanceREADME.md. Sourced from GitHub raw (the live Shamela/al-maktaba sites are Cloudflare-gated). ~70,500 biographical narrator rows across the workbooks.Books added (
sources/<slug>/)Ikhtilāṭ: al-Kawākib al-Nayyirāt (Ibn al-Kayyāl) — the original request.
Tier 1 — core narrator grading: Lisān al-Mīzān (Ibn Ḥajar, 15.5k entries) · al-Tārīkh al-Kabīr (al-Bukhārī, 14k) · al-Kāshif (al-Dhahabī, 8.3k) · al-Mughnī fī al-Ḍuʿafāʾ (al-Dhahabī, 7.9k) · Maʿrifat al-Thiqāt (al-ʿIjlī, 2.4k) · al-Ḍuʿafāʾ al-Kabīr (al-ʿUqaylī, 2.1k) · al-Majrūḥīn (Ibn Ḥibbān, 1.3k).
Tier 2 — isnād-defect detection: Taʿrīf Ahl al-Taqdīs / Ṭabaqāt al-Mudallisīn (Ibn Ḥajar) · al-Ightibāṭ bi-man rumiya bi-l-ikhtilāṭ (Sibṭ Ibn al-ʿAjamī) · al-Marāsīl (Ibn Abī Ḥātim) · Jāmiʿ al-Taḥṣīl (al-ʿAlāʾī) · ʿIlal al-Dāraquṭnī · ʿIlal Ibn Abī Ḥātim.
Tier 3 — biographical depth + supplements: Siyar Aʿlām al-Nubalāʾ (al-Dhahabī, 5.9k) · al-Ṭabaqāt al-Kubrā (Ibn Saʿd, 5.5k) · Tārīkh Ibn Maʿīn · Tārīkh asmāʾ al-Thiqāt (Ibn Shāhīn) · al-Ḍuʿafāʾ wa-l-matrūkīn (al-Nasāʾī, Ibn al-Jawzī).
Muṣṭalaḥ (grading vocabulary): Muqaddimat Ibn al-Ṣalāḥ · Tadrīb al-Rāwī (al-Suyūṭī) · Fatḥ al-Mughīth (al-Sakhāwī) · al-Kifāya (al-Khaṭīb).
xlsx structure
#· section · name · page · pages · verbatim critic statements · char count).معلوماتprovenance sheet. Text is reproduced verbatim — no automated grading inference.Pipeline
src/download_openiti_rijal.py: extended from 8 → 32 texts (regenerate into the gitignoredsrc/rijal_raw/).sources/build_openiti_books.py: regenerates everyclean.txt+xlsxfrom the primary mARkdown.Note on size
Adds ~147 MB (Siyar ~47 MB and Ṭabaqāt ~23 MB dominate; largest single file 21 MB, under GitHub's 100 MB limit). Can migrate large files to Git LFS on request.
https://claude.ai/code/session_019h5LuguzhJsQxT88jb5Jj5