Verify pre-print licences instead of assuming CC-BY; make every shipped skill findable

[lfnothias]

Two follow-ups from #22, kept general so they work for any collection and any future expansion. Stacked on consolidate/2026-07-10; retarget to main once #22 lands.

1. Pre-prints are not always CC-BY

OPEN_ACCESS_POLICY.md asserted that pre-prints are "always CC-BY by repository policy". bioRxiv, medRxiv and arXiv all let the author choose, so the rule could admit text we have no right to redistribute.

scripts/preprint_license.py resolves the real posting licence. It reads pre-print-ness from the registry's own declaration — Crossref type: posted-content, DataCite resourceTypeGeneral: Preprint — never from a DOI prefix. A prefix names a registrant, not a work type: 10.1101 is Cold Spring Harbor, covering bioRxiv, medRxiv and CSHL journals, and bioRxiv now also registers under openRxiv's 10.64898. Any server either registry covers therefore resolves with no code change.

Where a registry declares no usable licence, it falls back to the server's own API, routed by the institution[].name the registry itself reports. Servers and licence tokens live in governance/preprint_servers.yaml; adding one is a data edit.

Three corrections to the guidance in #22, all established by calling the live APIs:

• arXiv's Atom API exposes no licence. Use DataCite (or OAI-PMH arXivRaw). arXiv's default nonexclusive-distrib grants no reuse rights.
• ChemRxiv's public API is behind Cloudflare and answers 403. Resolve it through Crossref, which carries its licence.
• Crossref sometimes points license[].URL at a FAQ page, not a deed. That is an unknown, not a licence.

A second licence axis, because the existing one asks a different question

license_tier answers "what may I do with the tool?" — open means commercial use is fine. A paper's licence answers "may we redistribute the source?". These disagree: CC-BY-ND permits commercial use but forbids derivative text, so reading it through the tool tier would call it open. Mapping paper licences onto license_tier, as originally proposed, is a category error.

So governance/license_tiers.yaml grows a second canonical table, source_reuse (full / limited / none). Adding CC-BY-ND-4.0 and CC-BY-NC-ND-4.0 to the tool map changes no behaviour — both previously reached the same answer through the fallback — it only makes them explicit.

An unlisted licence returns None, never a default. None (we failed to establish the rights) and none (the rights were withheld) are different answers and the code refuses to collapse them.

Audit of what is already shipped — reported, not dropped

Of 600 corpus DOIs, 56 are pre-prints. Only 15 are full-reuse; 32 are NC/ND; 9 grant no reuse at all (8 bioRxiv cc_no, 1 arXiv default) and are status: included. Full findings, the 9 entries, and 24 malformed DOIs the audit surfaced are in #24. No entry was removed and no gate was weakened — every one of the 56 was admitted on repo-oa, not on the pre-print rule, so this is the same access.type conflation tracked in #23.

A rate-limited registry used to be indistinguishable from a missing DOI, which silently marked resolvable pre-prints unresolved. fetch_json now retries 429/5xx with backoff and honours Retry-After; only 403/404/410 means "no record". That mistake inflated the first audit run by 2 entries.

2. A skill that ships must be findable

collections/metabolomics/v2/skills/masster/SKILL.md existed on disk but appeared in none of skills_index.json, kb_bundle.json or docs-site/search_index.json — invisible to search, the MCP server and the docs site.

The cause is structural. propagate_license_tiers.py joins tiers onto entries the index already has, keyed by DOI; it never adds one. Masster has no DOI and no corpus entry, so it could never enter by that route. And check_license_tiers.py only cross-checks skills already indexed, so CI never noticed.

Chosen fix: derive the index from the skills directory (option a), not admit non-OA tools into the paper corpus (option b). corpus.yaml is a paper corpus keyed by DOI and access.type; a tool with no paper has no source to redistribute, so gating its discoverability on an open-access field is wrong by construction. Option (b) would have coupled discovery to the legal gate and let non-OA content into the corpus verify-paper polices — weakening a gate the brief forbids weakening.

scripts/skill_index.py derives an entry from a skill's own frontmatter, and a new CI gate fails when a skill exists on disk but in no index. Skills are excluded by declarative property — an _-prefixed directory (the convention release_gate.py already uses) or metadata.role: meta — never by a hardcoded name list.

The acknowledgment gate is preserved: --fix refuses to index a skill with no metadata.license_tier, so a non-open skill can never become searchable while shedding the label that governs its use. Masster ships license_tier: noncommercial with requires_ack: true, and check_license_tiers.py now cross-checks it because it is finally visible.

A second orphan, found on the way — larger than it first looked

docs-site/build_search_index.py split frontmatter on the substring ---, so any skill whose frontmatter contained a rule was truncated. 41 of 7,722 skills parsed differently, not the 2 I first reported:

• 2 were dropped entirely — mzml-data-parsing-pymzml-pyopenms in v1 and v2, whose evidence span contains -----. YAML failed, and the skill vanished from the index.
• 39 parsed as valid dicts but with truncated frontmatter, silently losing keys like provenance, attribution and claims into the body — which then polluted their search_text and summary. Those 39 were mis-indexed, not missing, which is why a naive count found only 2.

Verified with a parser-vs-parser diff over all 7,722 files: 41 changed, zero regressions (no file lost a key or gained pollution). Both consumers now share one canonical split_frontmatter().

Search index: 7718 -> 7722, exactly the 7,722 SKILL.md on disk. The four new entries are the 2 recovered skills, masster, and asb-metabolomics (the committed index predated it). Zero removals; papers and tools unchanged.

Generality

The maintainer asked that these stay general for library and collection expansion. Enforced by tests, not by intent:

• test_no_doi_literals_in_scripts walks the AST of every scripts/*.py and fails on any DOI-shaped string constant outside a docstring.
• test_preprint_detection_never_dispatches_on_a_doi_prefix fails if a 10. literal re-enters the resolver.
• Both resolvers and the index gate are tested across four unrelated sciences — metabolomics, proteomics, genomics and astronomy — none of which the code may branch on.
• Every failure mode has a negative test: unknown licence, no licence declared, rate limit, absent DOI, journal article, untiered skill, meta skill, a collection publishing no index.

Each guard was mutation-tested: reintroducing the bug turns the test red.

Verification

• 285 passed (was 253; +32 new, no regressions)
• python -m scripts.skill_index exits 1 naming masster before the fix, 0 after
• check_license_tiers.py green; description-discipline lint green over 7,722 skills
• Index diffs are pure additions in the files' own indent — 26 inserted lines, 0 removed

Review findings folded in

Two independent adversarial reviews. What changed:

One blocking defect, in the direction that matters. _first_recognised_spdx returned the first licence a registry declared. Crossref lists the accepted manuscript and the version of record separately and does not guarantee their order, so a work declaring both CC-BY and CC-BY-NC-ND resolved to whichever came first — admissible_as_open_access flipped on array position. A content-version: tdm entry (a text-mining grant, not a redistribution grant) was also read as a licence to reuse. Both reproduced, then fixed: the most restrictive declared licence now governs, TDM grants are excluded, and a recognised SPDX with no source_reuse row blocks rather than falling through to a permissive sibling.

A licence deed quoted inside an unrelated URL was accepted. https://evil.example.com/redirect?to=creativecommons.org/licenses/by/4.0 resolved to CC-BY. The host is now checked before the path.

The gate could inspect nothing and pass. Run from the wrong directory, skill_index printed PASS having visited zero collections. It now fails when no skill directory matches.

packs/ was unguarded — 4,961 shipped skills with a kb_bundle.json the gate never visited, able to reproduce the exact bug this PR fixes. Now covered; the gate walks 12 directories.

The tier vocabulary had three copies (license_tiers.yaml, check_license_tiers.py, and the new skill_index.py). Both scripts now read the canonical map, so adding a tier is a data edit.

Also: kb_slugs are derived from a skill's DOIs instead of written empty; a malformed registry payload fails its own DOI rather than aborting a bulk audit; and unresolved is no longer cached, so a transient outage cannot freeze into a permanent verdict.

I did not accept one finding: 10.64898/… is a genuine bioRxiv DOI (prefix registered to openRxiv), verified against Crossref's prefix API and api.biorxiv.org.

Re-resolving all 15 previously-full pre-prints under most-restrictive-wins changed none of them, so the numbers in #24 stand.

Verification

• 303 passed (was 253 on the base; +50 new, no regressions)
• Every exploit above reproduced before the fix and closed after
• Every guard mutation-tested: reintroduce the bug, the test goes red
• skill_index exits 1 naming masster before the fix, 0 after, across 12 directories
• Index diffs are pure additions in each file's own indent — 26 inserted lines, 0 removed

CI note

This PR is based on consolidate/2026-07-10 (#22), and validate.yml / verify-paper.yml only trigger on pull_request: branches: [main]. No CI runs on this PR until #22 merges and this retargets to main. Both gates were therefore replayed locally by extracting and executing the workflows' own inline scripts: PASS: description discipline OK (7722 skill files checked) and PASS: Paper access-tier validation OK (684 included papers checked).