corpus-research skill + literature-CLI enrichment (snowball, provenance, rejected-stats sidecar)#85
Draft
cjer wants to merge 52 commits into
Draft
corpus-research skill + literature-CLI enrichment (snowball, provenance, rejected-stats sidecar)#85cjer wants to merge 52 commits into
cjer wants to merge 52 commits into
Conversation
Wrap the mabool headless snowball endpoint (POST /api/3/headless/snowball).
Given seed papers (corpus_id + relevance grade), promote candidates via:
- backward: references of the seeds (fills the backward-refs gap the CLI lacked)
- forward: papers that cite the seeds
- citances: papers referenced from the seeds' snippet citation contexts,
reranked against --query
Seeds via repeatable --seed corpusId:relevance and/or --seeds-file (JSON).
Results are emitted in the same shape as `literature find`. Verified
end-to-end against a local mabool dev server for all three modes; 45 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tools for comprehensive, trustworthy corpus-scale literature analysis Ships GENERIC capability only (no thread-fitted content — configs/codebooks/seeds are derived per-run by the session from the thread question): - SKILL.md: pipeline spine (thread-config -> acquire -> curate -> substrate -> coverage -> extract/answer), thread.json contract, principles - references/: coverage playbook (multi-modal acquisition, capture-recapture, canonical-vs-tail stratification, anchors), curation recipe, codebook derivation (grounded coding), fulltext-at-scale (two-tier evidence, cache-first map-reduce), data discipline (rings/gates/MANIFEST), answer format (how-performed notes, disagreement axes), principles P1-P7 - scripts/: s2.py (cached+rate-limited+serialized S2 client), fulltext.py (fetch+cache+section-digest), knowledge.py (query-own-store primitive), coverage_signals.py (estimators/anchors/localizers incl. relevance-weighted eigenvector centrality), substrate.py (observations+rings+stage-gates), relevance.py (standardized judgment currency) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cquire.py mechanics, interaction rhythm, retrieval division-of-labor, judge+open-code fusion, hard answer requirements (how-performed notes, CorpusId links) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…— CR overlap-gate (reliable flag), yield_by_frequency for deferred-slice residuals, triangulation transparency in report(), verdict discipline in playbook, link requirement generalized Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…knowledge.py named at coverage/answer steps (F14), living thread.json schema discipline (F13), positions-with-quotes + pre-synthesis [T] aggregation codified as the disagreement grounding pattern Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ate/gates/NEXT/map entry point, compaction-survival for long-running threads; keep batch files out of the run root Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ites a CLAUDE.md/AGENTS.md stub at session cwd (the auto-loaded surface); README.md alone has no special status, skills don't re-fire on resumption Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…D — an S2 failure is not evidence a work doesn't exist (retry errored; only unresolved = true no-match) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…re the population lives; web/registry enumeration (registries, leaderboards, list-articles) as first-class modality + recall anchor for industry-adjacent threads Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…efine the population; lab-page/arXiv-only reports get synthetic ids (web:<slug>/arxiv:<id>) + fetched evidence + URL, judged like any candidate Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ey, external-id lookup BEFORE minting synthetics, ids{} sidecar, promotion-with-aliasing (id-aliases.json), cross-idspace dedup order
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s-produced + skip-already-done (idempotent resume); a stalled write-at-end worker loses everything Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
From the A2-rerun + A1-clean retrospective: - SKILL: habitat-note divergence step (find the librarians; >=2 off-list modalities) — restores explore over checklist-diligence (a run missed the asker's #1 item without it) - SKILL: worker discipline HARD rule (append-as-produced + skip-done in every worker prompt) - SKILL: interactive re-scoped to precision/validation probe (measured: 44% precision, 0 unique-core; find-sweeps = 65% of core uniquely) - SKILL: scope.strata (include both != pool); defer scope-edge questions to the charter beat - references/report.md: the user-facing REPORT spec (10 requirements incl. data-generated charts, evidence in-body, self-contained rendering) — named at step 6 - codebook: canonicalization maps are data — attested-names-only + self-check + review queue - curation: graduated judging (prior-ordered queue + calibrated cheap-first cascade) - coverage-playbook: capture-vs-enumeration halves; exemplars+floor; completeness critic - acquire.py: universal candidates_from_asta (find/interactive/snowball, one envelope, keeps pf_grade/citationCount/citationContexts) + write_seeds_file - coverage_signals.py: gap_lift (local citers / log global) — automates hub-vs-gap triage Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…vidence) - SKILL: HARD cost gate before first acquisition fan-out — measured scale/time/token estimates (N=3: ~2-2.5h active, 2.5-7M out + 250-600M cache-read, ~$550-850 list), lighter alternatives offered, mid-run scope growth renews consent - playbook §6: calibrated verdict template — head/tail claimed separately (every tested verdict was overconfident; head ~85-95% is what canon-anchor licenses; tail is SAMPLED), Chao1 lower bounds only calibrated estimator, propagate single-modality smell, ingestion-loss + scope-drift as first-class terms - gap_lift DEMOTED to sort-order only: calibrated AUC 0.50 (chance) as hub-vs-gap triager vs a real hand triage — triage is a kind/scope judgment, use cheap-tier LLM classification of titles instead (playbook §5 + docstring) Calibration evidence: movement/eval/coverage-signal-calibration.md Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… sidecar, RJ compat
Pairs with the mabool feat/headless-enrichment stack (origins/total_hits/
relevance_judgement/include_rejected):
- COMPAT FIX: RelevanceJudgement.relevanceSummary was REQUIRED — parsing would
crash the moment an enriched server sends relevance_judgement on find results.
Models now tolerate both wire shapes (a2a artifact + headless) via AliasChoices;
criteria evidence accepted under evidence/relevant_snippets.
- Paper.origins (query_type/provider/variant/query/ranks/total_hits) — per-result
provenance; snowball origins carry seeds + pre-truncation candidate count.
- find --include-rejected {none,summary,sample} with SIDECAR discipline: drop
statistics go to <output>.rejected.json, one summary line to stderr, never into
the main results file (coverage-script input, not session reading material).
- corpus-research acquire.py adapter keeps origins on candidate rows.
All fields optional; old servers parse unchanged (smoke-tested both shapes).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every change carries a validation receipt (movement/eval/coverage-signal-calibration.md): - shared-filter honesty: 'modalities blind' → queries independent / FILTER shared, filter-lineage classification, named verdict term for the retrieval-filter FN rate (SKILL step 1 + playbook §1) - parametric family anchor = REQUIRED post-codebook gate (validated: caught 4 real families hidden in Other; correct null on a good codebook; charter-conditioning as FP control) — full protocol in codebook.md, named at SKILL step 3 - coverage reported as FAMILY×ERA MATRIX (misses concentrate 40-56% in ~20% of cells); era-dip + empty-cell = the two validated internal flags; quiet cells not claimable without an external volume prior (playbook §6) - shadow signals in playbook §5: mention-shadow sweep (validated — caught a run's #1 named gap + beyond-gold misses) + dangling-lineage pass-gate; extraction schemas gain a mentioned-entities field (SKILL step 6) - validate.py: [ingestion-loss] judged-relevant must hold a live ring (gold lost 85 silently); [canon-attestation] canonical names must be attested (sizes-as-versions shipped in a real run) — smoke-tested green on a real run - curation: verbatim-means-verbatim (audited 93% compliance, paraphrase the deviation) - report.md req 11: every prose aggregate has a data-file home - find --include-rejected sidecar named at the acquire step Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…analysis) PF-filtered vs filter-independent = best usable CR (still a lower bound, ~20% low); never pair two independent modalities (fame co-capture collapses N-hat); disjoint catchments are unfixable without exposure data. Receipt: eval/calibration/decorrelated-cr.md Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both were used pervasively but defined nowhere — an unstated scale isn't a shared currency: - SKILL.md: [T]/[J] now spelled out at first use (Tool/deterministic vs Judgment) with the never-fake-either rule and why the split IS the trust story - curation.md: the 0-3 per-criterion grade rubric (pinned to the paper-finder 4-level scale so grades are comparable across retrieval/Claude/audit records + across runs) and the grade->tier roll-up (in/relevant/maybe/not-relevant), flagged as a [J] not a threshold Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ed examples - models: Origin.query_type and RejectedItem.drop_stage optional; corpus_id coerced to str (one malformed origin must not fail the whole find result) - client: send include_rejected only when active (server back-compat, matches snowball() discipline) - curation.md: <axis>_guess derived from thread.json, not copied axis names - codebook.md: phantom-version example genericized 193/193 tests pass.
…n, not $550-850) Prior figures summed transcript lines (each API response logged once per content block, same usage repeated) — ~2.4x inflation. Measured deduped: 0.3-0.5M output + 100-160M cache-read per full run; cache-read ~60% of bill.
Sample rows are the filter-FN/truncation audit input; they live in the sidecar so the session context cost is zero. Summary-only starves the audit.
…ncurrency is small and timeouts count queue wait
…y + trigger broadening - SKILL.md acquire: staged fast->escalate policy with diligent anchor + [T] gate (receipts: diligent-replay, probe-2), charter-derived query floors + per-axis anchor enumeration (receipts: run3 recall scoring) - description: real-user phrasings + escalation-moment trigger (receipts: colleague kickoff + menu, 2 data points) - find-literature/literature-report: 'when the need outgrows this skill' cross-refs
…venue-completeness + two evolved anchor moves adopted Receipts: run3 85-90-vs-44 calibration gap, probe-2 (sidecars unread), BBNLP-2025 probe, run3+sa infrastructure-adjusted recall, sa external-enumeration anchor (38%).
…ys over compactions); support-gate floor concretized (>=2-3/side, receipt: cross-run axis replication)
…eviews as acquisition anchors; habitat note consumes prior rosters (user policy)
…nums (preserve-explore: prescribe the question, not the bucket list)
…easured), gap_lift (AUC 0.50), write_seeds_file, self-alias, n_strategies ~95 lines of code removed; capability cuts logged in movement design/forgone-capabilities.md; seeds-file format contract kept as one SKILL line.
…s) + staleness fixes Items 3,6,7,8,9,10,11,12,13,14,15,16: principles.md folded into SKILL and deleted; worker-discipline/answers-reqs/shared-filter/librarian/canon-anchor/strata/TAG_GATE each now stated once; concurrency folded into sweep sentence; interactive note compressed; stale sidecar conditional fixed; infra-note roadmap musing cut. (One regex mishap on fulltext-at-scale caught by diff review and redone surgically.)
…vance only); answers.md+report.md merged into deliverables.md (Part A/B, shared reqs deduped)
…ule, one war-story); retrieval tool detail moved to playbook §1, SKILL keeps policy + imperative pointer (224 lines, was 246+)
…ungs (C3), report_gate.py (C1), shards.py + workers.md (D) - view: canonical join tested on run3 snapshot (2263 rows, 493 extractions joined); grep logs to queries.log - fulltext: ACL rung fetches the PDF (cached for pdf-extraction) — landing pages must not masquerade as body text - report_gate: calibrated against both real reports (fails run3 on its two documented deficiencies; sa-run 89% tracing) — SVG/attr stripping, id-length exclusion - shards: interleave+salt+k-chunk builder + salt scorer (flags an inverted judge in test) - workers.md: the canonized contract (structure over prompts) - SKILL: view-first at answer phase, report gate HARD at report close, worker section points to contract
…ls/frameworks>' (data point 3: registry-native kickoff didn't trigger; description was literature-shaped)
…s ('sampled, not enumerated', 'do not read as complete') — found by the smoke run's verdict
…ipts/vault.py) - vault.md: layout, append-only growth model (canonical verbatim / caches fold-back binding SUBAGENTS / derived layers rebuilt-only), trust marks, round contract + round types, warm-workspace CLAUDE.md binding, manifest template - vault.py: init (run -> founding vault) + rebuild (registry-driven round discovery that survives moved/copied workspaces; whole-dir verbatim fold, no filename enums; deterministic derived view). Equivalence-tested against the eval-side builder: 0 differing rows on the live sa vault; idempotent on workspace copies. - SKILL.md: Long-running threads section pointing at the vault doctrine Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…on follow-ups are rounds; graduate at first round close Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…or regime, diversify-on-saturation) + row title + machinery-backstage + round/beat vocabulary - coverage-playbook §3: CR bias is one-directional+compounding (Brittain&Bohning 1/psi; Binette&Steorts heterogeneity; Chao lower bounds) -> two-enumerator proxies need MECHANISM diversity; same-model agents = worst quadrant (verbatim-verified, sa round-5 Q5.2) - coverage-playbook §2: diversify-on-saturation before concluding exhaustion (literature-derived, our-side check pending) - coverage-playbook §6: name the denominator regime (closed/samplable pool admits calibrated recall targets per TAR; open denominator = bounds only; mixed threads say which is which) - curation.md: title field in judgment rows (rows outlive their session; vault view metadata) - deliverables.md Part A: machinery stays backstage (user call) - vault.md: first build IS round 1; rounds compound, beats communicate Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…c-scholar; semantic-scholar named as lighter alternative in corpus-research Audit verdict (full record: movement eval/skill-ecosystem-audit.md): no redundancy, boundaries explicit, escalation paths already two-way; the one real hazard was unguarded bulk 'asta papers' looping vs the s2.py serialization doctrine — closed here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…fixes - deliverables.md: delivery is part of the report (deploy+URL on sharing-shaped asks; package must work off-machine; gate applies to updates) - report_gate.py: package-escape check + S2 canonical-link-form check (both user-caught) - vault.md: deliverable-gate as durable contract step; curation.md wins on row schema; manifest-prose ownership; deferral-with-declaration named; --amend documented - vault.py: rebuild --amend <latest-round> (post-close fixes refresh the canonical copy; earlier rounds refused — tested both ways) - coverage-playbook: denominator-regime line attaches to coverage CLAIMS, not only verdicts Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cjer
force-pushed
the
feat/corpus-research
branch
from
26b440a to
98cd050
Compare
- vault.py: resolved_latest overlay — newest opinion that POSTDATES an existing conflict = the thread's current call; agreement/history untouched; row-backed only. Registry newest-first INVARIANT documented (legacy oldest-first a2 registry produced bogus marks, fixed workspace-side). Verified: a2 = 222 resolved (57 round-5 deliberate re-judges + 165 later-charter calls), 525 DISPUTED preserved; sa = 0 as expected. - vault.md: DISPUTED-resolved trust mark documented - deliverables.md: living-report layering pattern (changelog + NEW/standing badges + bundled standing report), receipt = the two round-6 consolidations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…upted every cached PDF — bug found by a2 acquisition round-7) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… (same-culture pairs psi 1.8-2.6, recall +19-29% over-stated; diverse-mech pairs calibrated to +-3%) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…equence); any diverse pair may anchor recall Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cjer
force-pushed
the
feat/corpus-research
branch
from
2f7fc32 to
721b6fe
Compare
…iversity ADVANTAGE robust, absolute calibration not identifiable from inside own sources Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…spite doctrine — enforce mechanically); smoke 582/582 on oss corpus copy Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…both directions; diagnose shared-latent vs disjoint-reach; spread = uncertainty Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-7 convergence) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…receipt: a2 round-8) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…never-surfaced formal-expressivity spine) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…egisters (openreview probe receipt: intervention-identifiability axis was invisible to positives-only mining) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft for visibility/diff-review — the corpus-research work dogfooded across several full literature-research runs (two thread topics, validated against gold corpora with a calibration study).
What's here (15 commits, three layers)
1.
corpus-researchskill (plugins/asta-preview/skills/corpus-research/): methodology + deterministic tools for corpus-scale literature analysis — thread-config derivation, multi-modal acquisition (find/snowball/citances/web-enumeration), graded relevance curation, grounded codebooks, coverage estimation with calibrated honesty rules, per-paper extraction, trust-carrying answers. Includes a hard cost gate (measured: full runs ≈ $550–850 at list prices — the skill asks before committing), interaction beats, and validation gates throughout. 9 scripts + 8 reference docs.2.
asta literature snowballCLI — backward/forward/citances against the new headless endpoint (pairs with allenai/ai2i-agents#1977).3. Literature-CLI enrichment consumers (latest commit; pairs with the same PR):
RelevanceJudgement.relevanceSummarywas required — would crash on enriched-server responses. Models now tolerate both wire shapes.Paper.origins(per-result retrieval provenance incl. pre-truncation totals).find --include-rejected {none,summary,sample}with a sidecar discipline: drop statistics go to<output>.rejected.json(coverage-script input), one summary line to stderr, never into the main results file.Testing
193/193 repo tests pass; wire-shape smoke tests cover old-server, new-server, and absent-field payloads.
Note: the skill's per-thread configs deliberately do NOT ship (derived per run from the user's question — keeping the eval train/test boundary clean).
🤖 Generated with Claude Code