Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 49 additions & 8 deletions .github/scripts/conformance-red.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,15 +33,39 @@ done
# are ever shortened.
help_text=$("$PROVIDER" --help 2>&1 | sed $'s/\033\\[[0-9;]*m//g')

modes=$(printf '%s\n' "$help_text" |
sed -n 's/^[[:space:]]*-[[:space:]]\{1,\}\([a-z0-9][a-z0-9-]*\):.*/\1/p')
# Each mode's doc comment already names the check it is supposed to trip —
# "(trips `budget-honesty`)" — so the expected check is derived from the binary
# too, never hardcoded here. That closes the hole this script used to have: it
# asserted only that *some* check went red, so a mode could be "caught" by an
# unrelated check while the check that actually owns the guarantee silently
# stopped working. `crash-on-query` legitimately names two alternatives, so the
# contract is "at least one of the checks this mode claims to trip".
#
# Output is one `mode<TAB>expected1,expected2` record per line.
mode_records=$(printf '%s\n' "$help_text" | python3 -c '
import re, sys

if [[ -z "$modes" ]]; then
modes=$(printf '%s\n' "$help_text" | tr '\n' ' ' |
help_text = sys.stdin.read()
# The bulleted rendering clap uses when a variant carries a doc comment.
for mode, description in re.findall(
r"^\s*-\s+([a-z0-9][a-z0-9-]*):\s*(.*)$", help_text, re.MULTILINE
):
trips = re.search(r"\(trips ([^)]*)\)", description)
expected = re.findall(r"`([a-z][a-z0-9-]*)`", trips.group(1)) if trips else []
print(mode + "\t" + ",".join(expected))
')

if [[ -z "$mode_records" ]]; then
# Compact "[possible values: a, b]" rendering — no doc comments, so no
# expected-check information is available and every mode falls back to
# "caught by anything".
mode_records=$(printf '%s\n' "$help_text" | tr '\n' ' ' |
sed -n 's/.*\[possible values: \([^]]*\)\].*/\1/p' |
tr -d ' ' | tr ',' '\n' | sed '/^$/d')
tr -d ' ' | tr ',' '\n' | sed '/^$/d' | sed 's/$/\t/')
fi

modes=$(printf '%s\n' "$mode_records" | cut -f1)

if [[ -z "$modes" ]]; then
echo "::error::could not discover --misbehave modes from $PROVIDER --help"
exit 1
Expand All @@ -52,7 +76,7 @@ echo "$modes" | sed 's/^/ - /'
echo

failed=0
while read -r mode; do
while IFS=$'\t' read -r mode expected; do
[[ -z "$mode" ]] && continue
# `|| true` is load-bearing: inspect exits non-zero precisely when it catches
# a broken provider, which is the outcome this script is asserting. Without
Expand All @@ -75,10 +99,27 @@ print(",".join(c["name"] for c in report["checks"] if c["status"] != "pass"))
if [[ -z "$tripped" ]]; then
echo "::error::mode '$mode' passed every check — the suite does not catch it"
failed=1
continue
fi

if [[ -z "$expected" ]]; then
echo " ✓ $mode -> caught by: $tripped (no declared check to match against)"
continue
fi

# The mode must be caught by a check it actually claims to trip.
if MODE_TRIPPED="$tripped" MODE_EXPECTED="$expected" python3 -c '
import os, sys
tripped = {c for c in os.environ["MODE_TRIPPED"].split(",") if c}
expected = {c for c in os.environ["MODE_EXPECTED"].split(",") if c}
sys.exit(0 if tripped & expected else 1)
'; then
echo " ✓ $mode -> caught by: $tripped (expected: $expected)"
else
echo " ✓ $mode -> caught by: $tripped"
echo "::error::mode '$mode' was caught by [$tripped], but none of the checks it declares it trips [$expected] went red — the check that owns this guarantee has stopped catching it"
failed=1
fi
done <<<"$modes"
done <<<"$mode_records"

if [[ "$failed" -ne 0 ]]; then
echo
Expand Down
10 changes: 10 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -168,3 +168,13 @@ jobs:
working-directory: site
- run: pnpm build
working-directory: site
# The witness asserts the site still exposes the docs it claims to, on the
# neutral theme, with the logo wired up. It has lived in the repo passing
# locally and running nowhere, which makes it decoration rather than a
# gate — the same "checked in but never executed" gap this round is
# closing elsewhere (#51).
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Docs-site witness
run: python3 -m unittest discover -s tests -p 'docs_site_witness_test.py' -v
2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ edition = "2024"
rust-version = "1.90"
license = "MIT OR Apache-2.0"
repository = "https://github.com/macanderson/context-graph-protocol"
homepage = "https://github.com/macanderson/context-graph-protocol"
homepage = "https://contextgraphprotocol.org"
# Default: crates opt IN to publishing. The three Context Graph Protocol crates each set
# `publish = true`; any future internal/helper crate stays unpublished by default.
publish = false
Expand Down
106 changes: 98 additions & 8 deletions SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,30 @@ graph-capable provider **SHOULD** boost frames within a small number of relation
hops of an anchor. The ranking algorithm stays provider-private; the *contract*
is only that anchors bias relevance.

| # | Requirement | Verified by |
| - | ----------- | ----------- |
| **Q1** | When `kinds` is non-empty, a provider **MUST NOT** return a frame whose `kind` is outside it. Empty `kinds` means any kind. A provider serving none of the requested kinds returns zero frames, or replies `unsupported_kind`. | `kinds-filter` |

### 5.1 Why `kinds` binds (Q1)

`kinds` shipped as a request field with documented syntax and no stated
semantics, and not one implementation honored it — the reference provider and
all three SDKs declared `capabilities.query.kinds` and then ignored the filter,
returning whatever they had. That is the dead-capability surface
[ADR 0004](docs/adr/0004-dead-capability-surface.md) purged elsewhere, in its
subtler form: not an unreachable field, but a reachable one that silently does
nothing.

Specifying it rather than dropping it, because unlike `upsert`/`subscribe` the
surface is already load-bearing: `unsupported_kind` (§10) exists precisely to
answer "you asked for kinds I don't serve", which presupposes the filter binds.
A host that narrows to `["snippet"]` to keep prose out of a code-reasoning
prompt, and silently receives `doc` frames anyway, has had its budget spent on
content it explicitly excluded.

Q1 is a filter, not a ranking rule: it says which frames are *eligible*, and
leaves ordering provider-private like the rest of §5.

### 5.1 Embedding space (E1)

| # | Requirement |
Expand Down Expand Up @@ -416,7 +440,24 @@ content is what goes into a prompt.
| - | ----------- | ----------- |
| **G1** | Every `Relation` **MUST** carry a non-empty `display_name` — an edge is surfaced by human label, never a raw id. | `frame-validity` |
| **G2** | `target_uri` **MUST** be a non-empty URI. | `frame-validity` |
| **G3** | A provider declaring `capabilities.graph` **SHOULD** boost frames within a small number of relation hops of a query `anchor`. | advisory |
| **G3** | A provider declaring `capabilities.graph` **SHOULD** boost frames within a small number of relation hops of a query `anchor`. | `anchor-relevance` |
| **G4** | A frame is **anchored** by an anchor URI when its own `uri` equals that anchor (zero hops), or any of its `relations[].target_uri` does (one hop). A provider declaring `capabilities.graph` and given a non-empty `anchors` **MUST** return at least one anchored frame when it has one to serve, and **SHOULD** rank anchored frames above unanchored ones. | `anchor-relevance` |

### 8.2 Why anchoring needed a definition (G4)

G3 said providers should "boost frames within a small number of relation hops of
an anchor" and stopped there — it never said what an anchor is compared
*against*. Two conformant providers could reasonably match anchors against the
frame `uri`, against `relations[].target_uri`, or against neither, and no test
could distinguish a provider doing sophisticated graph traversal from one
ignoring `anchors` entirely. The reference fixture did the latter: it declared
`graph: false`, served frames with no relations at all, and every graph
requirement passed vacuously.

G4 gives "anchored" a decidable predicate — string equality on URIs, at zero or
one hop — so the SHOULD in G3 becomes something a suite can actually witness.
Deeper traversal stays provider-private: G4 is a floor on what must be *found*,
not a ceiling on how hard a provider may look.

### 8.1 Relation vocabulary (SHOULD)

Expand Down Expand Up @@ -548,11 +589,15 @@ What remains genuinely unchecked:
(C8) are properties of the host's HTTP client; exercising them needs a real
non-loopback, TLS network peer the in-process harness cannot stand up. They
remain the host-side harness's next increment.
- **R3 delimiting is checked; breakout-resistance is not.** The harness proves
`content` is fenced as quoted material, but the reference `compose_context`
does not escape a content-embedded fence token — hardened, injection-resistant
delimiting (an unguessable fence, escaping) is the composition module, issue
#15.
- **R3 breakout-resistance is now escaping, not an unguessable fence.** The
reference `compose_context` neutralizes a content-embedded `<frame`/`</frame>`
token and escapes fence attributes, so content cannot terminate the block that
quotes it or forge a sibling frame (issue #15). Escaping rather than a random
delimiter is deliberate: composition's contract is a byte-stable prompt prefix
(§1 of `docs/context-reuse.md`), and a per-turn nonce would forfeit the
provider prompt cache to buy a property escaping already provides. What
remains open is the *rest* of the composition module — global budget packing
and cross-provider dedup — still issue #15.
- **F5-bytes verifies a host-trusted source, not any provider-named `uri`.** The
verifier re-reads a path the host chooses to trust; automatically re-reading an
arbitrary `uri` a provider supplies is a capability decision (path confinement,
Expand Down Expand Up @@ -585,7 +630,7 @@ The freeze drops `-draft` without a flag day (§3.1) only if a `contextgraph/1.0
implementation can safely receive a message a later `1.x` peer emits. That
requires a stated rule for what "receive" does with surface the receiver was not
built to know about. These rules are normative; they are what make the additive
bias of §14 real rather than aspirational.
bias of §15 real rather than aspirational.

| # | Requirement |
| - | ----------- |
Expand All @@ -609,7 +654,52 @@ know only ever grew, and nothing it relied on was moved out from under it.

---

## 14. Changing this specification
## 14. Attribution

Provenance (§6.2) answers *where an item came from*. Attribution answers the
other half of the same question — *what it did* — so that including a frame is
an evaluable decision rather than an act of faith. Cost without outcome prompts
no decision ("this frame cost 400 tokens"), and outcome without cost prompts the
wrong one ("this frame was never cited" — it cost four).

| # | Requirement | Verified by |
| - | ----------- | ----------- |
| **A1** | A frame's attribution handle **is** its `FrameId` (§6.3) — the same `(provider id, frame id, content_digest)` triple used for composition, dedup, usage reports (§U1), and `verify` (§9). An implementation **MUST NOT** mint a separate attribution id. | `contextgraph-types::attribution` |
| **A2** | A host reporting attribution **MUST** report `selected`, `rendered`, and `cited` as independent observations, not a single score. `cited` **MUST** mean the model's output referred to the frame, an observable fact — never an inference that the frame *influenced* the output. | `contextgraph-types::attribution` |
| **A3** | An attribution record **MUST** be reconcilable: coherent (`cited` ⇒ `rendered` ⇒ `selected`) and naming a frame the paired usage report actually billed. | `AttributionReport::is_reconcilable` |

### 14.1 Why one id, and three booleans

**One id (A1).** A second identity would be free to disagree with the first, and
a disagreement between *the frame that was billed* and *the frame that was
cited* is precisely the confusion attribution exists to remove.

**Three booleans (A2).** They are separately observable and collapse badly. The
case that matters most is a frame that was `selected` and `rendered` but never
`cited`: the host paid its tokens, the model read it, and it changed nothing.
A `used`/`unused` flag cannot express that, and a 0–1 usefulness score would
invent a precision nobody measured. `selected` without `rendered` is a third
distinct state — ranked in, then dropped by budget packing — and it is neither
credit nor debit, because it was never shown.

Attribution is a **host self-report**. Unlike `token_cost`, which §B3 anchors to
a canonical rule anyone can recompute, there is no way to check a host's claim
that a frame was cited; the guarantee is scoped to hosts that want honest
measurement, not enforced against ones that don't.

**Not on the wire.** There is no `context/feedback` method and no
`Capabilities.feedback` in this revision. The vocabulary is specified because it
has to be shared for scores to be comparable across implementations; the
transport is deferred to a 1.x additive minor
(`docs/sketches/attribution-feedback.md`). Shipping a negotiated feedback method
with no provider consuming it would recreate exactly the dead capability surface
[ADR 0004](docs/adr/0004-dead-capability-surface.md) removed — and the asymmetry
favors waiting: adding the method later is family-safe, removing a dead one is
not.

---

## 15. Changing this specification

See [GOVERNANCE.md](./GOVERNANCE.md). A normative change needs an issue, a PR
updating this document and `CHANGELOG.md`, and a **witness** — a conformance
Expand Down
Loading
Loading