Skip to content

feat(kbc): coverage v2 — media assets auto-attach via citing docs#439

Open
LikiosSedo wants to merge 3 commits into
mainfrom
feat/kb-asset-provenance
Open

feat(kbc): coverage v2 — media assets auto-attach via citing docs#439
LikiosSedo wants to merge 3 commits into
mainfrom
feat/kb-asset-provenance

Conversation

@LikiosSedo

Copy link
Copy Markdown
Collaborator

What

Media assets (images under assets/) are document attachments, not first-class sources. Coverage v2 auto-accounts an image against any document that embeds it in its body, once that document is itself cited or excluded. compiled_from no longer needs a token per image and the exclusion ledger no longer needs a row per image.

Implements S1 (siclaw/kbc) of the raw reference contract v2. Design: DESIGN-kb-asset-provenance-2026-07-22.md §四 (normative).

Motivation

  • cf bloat: aggregated compiled_from drowns in image tokens (UHI 昇腾.md: 139 entries = 35 docs + 104 images), unreadable to a human.
  • exclusion-ledger noise: images as first-class sources force ~563 of UHI's 603 exclusion rows to be mechanical asset entries.
  • attribution gap: the schema could not express which document embeds this image, so mediaverify and the collapse UI each guessed with heuristics.

Math (v2)

auto        = { a ∈ media_assets | ∃d ∈ (cited ∪ excluded): edge(d→a) }
unaccounted = sources − cited − excluded − auto
  • Edges are derived from the raw tree — each document's body ![](…), [](…), and HTML <img src> links — resolved relative to the document + posixpath.normpath + URL-decoded. A link to a nonexistent / non-media path yields no edge, no error.
  • An orphan image (embedded by no accounted document — upload residue) is not auto-attached; it stays unaccounted and must be excluded with a reason, so a human still sees it (anti fail-open).
  • A directly-cited asset still counts as cited (v1 compatibility).

Monotonicity — old green libraries stay green

v2 only adds an accounting path, so unaccounted can only shrink. All pre-existing selfcheck tests pass unchanged — none asserted "unreferenced asset ⇒ unaccounted" over an assets/ path (the media-asset predicate requires an assets/ segment, which no existing test used, so no assertion needed updating).

Sheet-placeholder carve-out

assets/sheets/*.md table placeholders are content files, not media (is_media_asset requires an image extension). They stay first-class sources — auto-attaching them would launder "the table data was never compiled" into "covered".

Observability

coverage gains auto_attached (count) and auto_attached_sample (≤20 asset ← via edges), so auto-accounting is never a silent fail-open.

Two-repo fixture (byte-for-byte contract)

kbc/platform/pod/fixtures/asset-provenance/ = shared raw tree + candidate + EXCLUSIONS.json + expected.json. The sicore adoption ledger (S2) will assert the same expected.json. Cases: relative link, ../ cross-dir, HTML <img>, URL-encoded path, missing target, 0-byte download-failed placeholder, assets/sheets/*.md (not media), image shared by a cited + an unaccounted document, orphan image, image inheriting a document's exclusion, directly-cited asset.

S1b — compile discipline

The box constitution (prompts/{en,zh}/box_role.md) now tells the compile agent: cite documents only; images auto-attach — do not cite or exclude them one-by-one; orphan images still need a reasoned exclusion; assets/sheets/ placeholders handled as usual.

Test

python test_selfcheck.py — all green (4 new: is_media_asset, document_link_targets, coverage_v2_auto_attach, asset_provenance_fixture). test_mediaverify / test_incremental / test_redblue / test_compile_box also green.

Media assets (images under assets/) are document attachments, not first-class
sources. Coverage v2 auto-accounts an image against any document that embeds it
in its body, once that document is itself cited or excluded — so compiled_from
no longer needs a token per image and the exclusion ledger no longer needs a
row per image.

Motivation
- compiled_from bloat: aggregated cf drowned in image tokens (UHI 昇腾.md: 139
  entries = 35 docs + 104 images), unreadable to a human.
- exclusion-ledger noise: images as first-class sources forced ~563 of UHI's
  603 exclusion rows to be mechanical asset entries.
- attribution gap: the schema had no way to express "which document embeds this
  image", so mediaverify and the collapse UI each guessed with heuristics.

Math (v2)
  auto        = { a in media_assets | exists d in (cited | excluded): edge(d->a) }
  unaccounted = sources - cited - excluded - auto
- edges are derived from the raw tree (each document's body ![](), [](), and
  <img src> links), resolved relative to the document + posixpath.normpath +
  URL-decoded; a link to a nonexistent/non-media path yields no edge, no error.
- an ORPHAN image (embedded by no accounted document — upload residue) is NOT
  auto-attached; it stays unaccounted and must be excluded with a reason so a
  human still sees it (anti fail-open).
- a directly-cited asset still counts as cited (v1 compatibility).

Monotonicity (old green libraries stay green)
v2 only ADDS an accounting path, so unaccounted can only shrink. All pre-existing
selfcheck tests pass unchanged — none asserted "unreferenced asset => unaccounted"
over an assets/ path (the media-asset predicate requires an assets/ segment,
which no existing test used).

Sheet-placeholder carve-out
assets/sheets/*.md table placeholders are content files, not media
(is_media_asset requires an image extension). They stay first-class sources —
auto-attaching them would launder "the table data was never compiled" into
"covered".

Observability
coverage adds auto_attached (count) and auto_attached_sample (<=20 asset<-via
edges), so auto-accounting is never a silent fail-open.

Two-repo fixture
kbc/platform/pod/fixtures/asset-provenance/ is the shared, byte-for-byte
contract (raw tree + candidate + EXCLUSIONS + expected.json). sicore's adoption
ledger will assert the SAME expected.json. Cases: relative link, ../ cross-dir,
HTML <img>, URL-encoded path, missing target (no edge), 0-byte download-failed
placeholder, assets/sheets/*.md (not media), image shared by a cited and an
unaccounted document, orphan image, image inheriting a document's exclusion,
directly-cited asset.

S1b: box constitution now tells the compile agent to cite documents only —
images auto-attach; do not cite or exclude them one-by-one; orphan images still
need a reasoned exclusion; assets/sheets/ placeholders handled as usual.
Follow-up to coverage v2. media_citing_pages() gains a third discovery path:
a candidate page that cites a DOCUMENT d in its compiled_from now inherits the
image numeric-fidelity check of every image d embeds in its body (via
asset_attribution_edges), unioned with the existing two paths (directly-cited
compiled_from images + body (source: img) citations), which legacy pages keep
using.

Why: S1b tells the compile agent to stop citing images one-by-one — they
auto-attach. media_citing_pages() drove the blind fresh-eyes transcription check
off cited images only, so without this the numeric check would silently stop
covering embedded charts — a silent fail-open on the exact fidelity risk it
exists to catch, in the window before the image-audit dimension lands.

Edge assets are intersected with IMAGE_SOURCE_EXTS (the transcription surface),
so a media asset like .tiff is accounted by coverage but not numerically re-read.
Additive only: no existing media test used an assets/ raw path, so all prior
assertions pass unchanged.
Two implementation-level details locked byte-for-byte with the sicore adoption
ledger (shared fixture, DESIGN §六), after cross-checking the two implementations:

1. is_media_asset segment match is now case-INSENSITIVE (`Assets/`, `report.ASSETS/`
   count too). The platform always writes lowercase `assets/`, so this only
   affects hand-authored trees, but both repos must agree. Locked by a unit test,
   NOT the shared fixture: an uppercase directory is not portable on a
   case-insensitive filesystem (it collides with `assets/`).

2. document_link_targets truncates a `#fragment` AND `?query` from the target,
   and does so on the STILL-ENCODED string BEFORE percent-decoding (new helper
   _strip_fragment_query). This matches the ledger's parse order (strip angle →
   truncate #/? → unescape), so an encoded %23/%3F inside a real filename
   survives while an actual delimiter is removed.

Everything else already matched (ext set incl. tiff / no pdf, both quote styles
for HTML <img>, resolve-then-membership skip rule, single unquote, Clean(Join)
keeping leading ..). The shared fixture gains a `?query`-suffixed target case
(raw/extras.md → assets/q.png?v=2). Neither change alters any prior fixture
output; expected.json is regenerated from the reference implementation with an
in-generator sanity assert on the invariants. All selfcheck + mediaverify tests
green.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant