Skip to content

fix(harness): enforce real verifier gates#233

Open
EfeDurmaz16 wants to merge 41 commits into
fix/ruby-replay-store-capabilityfrom
fix/harness-adversarial-hardening
Open

fix(harness): enforce real verifier gates#233
EfeDurmaz16 wants to merge 41 commits into
fix/ruby-replay-store-capabilityfrom
fix/harness-adversarial-hardening

Conversation

@EfeDurmaz16

@EfeDurmaz16 EfeDurmaz16 commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • split shared CI and harness defenses from fix(x402): harden cross-SDK verifier and CI gates #226 into a harness-owned PR
  • require declared modes to call real SDK verifiers and add managed-signer drain vectors
  • strengthen test, vector, matrix, boot-policy, and payment-channels revision accounting

Verification

Stack

Targets #219 as requested. This PR should merge after the language fixes it exercises, or together with them, so the blocking gates never encode unavailable behavior.

@EfeDurmaz16

Copy link
Copy Markdown
Collaborator Author

@greptile-apps please review

@greptile-apps

greptile-apps Bot commented Jul 10, 2026

Copy link
Copy Markdown

Greptile Summary

This PR hardens the conformance harness by requiring runners that declare a verifier mode to actually execute it, and by extending the managed-signer transfer guard to cover multisig signer tails and Token-2022 ATA derivation. It also replaces several inline CI scripts with testable modules and tightens a number of accounting gates.

  • Declared-mode enforcement (mode-support.ts, conformance.test.ts): a runner that lists a mode in modesByIntent can no longer silently return unsupported-mode; strictModesByIntent additionally requires at least one accept and one reject vector per declared verifier mode.
  • assertNoManagedTransferFunding extraction (x402.ts): the inline authority+source check is promoted to an exported, testable function that also covers multisig signer positions and Token-2022 ATAs; three new unit tests pin each rejection path.
  • Infrastructure tightening: verify-routed-sdk-gate.mjs replaces 40 lines of inline CI JavaScript with a tested module; vector accounting migrates from directory-level to per-file FILE_CONSUMERS; Semgrep is pinned to a Docker SHA and flipped from advisory to blocking.

Confidence Score: 5/5

Safe to merge; the logic changes are well-tested and fail-closed, with only minor diagnostic-quality issues in a new shell script and a gate error message.

Every new enforcement path has a corresponding unit or integration test. The assertNoManagedTransferFunding refactoring is semantically equivalent for the single-authority case and strictly stronger for multisig tails. The two observations — a shared error message in verify-routed-sdk-gate.mjs and a whitespace-sensitive grep in check-payment-channels-revision.sh — affect diagnostic clarity only and do not change whether the gate fires or passes.

scripts/verify-routed-sdk-gate.mjs (error message ambiguity between the two mutual-exclusion failure modes) and scripts/check-payment-channels-revision.sh (4-space-hardcoded indent on the action.yml SHA check).

Important Files Changed

Filename Overview
harness/src/conformance/mode-support.ts New module enforcing declared mode coverage; assertDeclaredModeWasExecuted and assertStrictModeCoverage close the false-green gap for runners that skip verifier vectors.
harness/src/conformance/x402.ts Refactors inline managed-signer drain check into exported assertNoManagedTransferFunding; now covers multisig signer tails (positions 3+) and Token-2022 ATA derivation alongside direct source and authority.
harness/src/conformance/runners.ts Adds modesByIntent/strictModesByIntent to runner manifests with robust validation: known intents, allowed mode sets, non-empty arrays, no duplicates, and strict modes must subset declared modes.
scripts/verify-routed-sdk-gate.mjs Extracts inline CI gate logic into a testable module; introduces docs_only mutual-exclusion check but the shared error message covers two distinct failure modes, making diagnosis harder.
harness/test/conformance.test.ts Integrates mode-support enforcement into the per-language conformance loop; assertDeclaredModeWasExecuted fires per-vector and assertStrictModeCoverage fires in afterAll for a two-layer gate.
harness/test/vector-accounting.test.ts Migrates directory-level consumer map to per-file FILE_CONSUMERS and adds a stale-entry check; a new file beside an existing consumed vector now fails closed instead of inheriting its directory's coverage claim.
harness/test/ci-coverage-gate.test.ts Replaces regex text-scan with a proper YAML-parse + vitest-command extraction; only test/ paths explicitly passed to vitest run within a harness working-directory count as wired. Structured CI_EXEMPT entries now require owner/date/removalCondition.
.github/workflows/semgrep.yml Pins Semgrep to a Docker image SHA, removes pip install, and flips the source scan from advisory (continue-on-error) to blocking; scan step now passes SEMGREP_ERROR_ON_FINDINGS=1.
scripts/check-payment-channels-revision.sh New script asserting all workflows pin the same payment-channels SHA; the action.yml check uses grep -Fqx with a hardcoded 4-space indent, making it fragile to indentation changes.
scripts/select-pr-workflows.mjs Refactors path matching from flat prefix arrays to {directories, files} structure; any change under SHARED_HARNESS_PATHS now triggers all SDK workflows (previously only the harness job). Adds docs_only output for the gate.
harness/test/x402-exact-managed-signer.test.ts New test covering three rejection paths for managed-signer drain: multisig signer tail, direct source match, and Token-2022 ATA derivation.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Runner manifest JSON] -->|isRunnerManifest| B{Has modesByIntent?}
    B -->|No| C[Default: unsupported-mode allowed for all modes]
    B -->|Yes| D[isModeCapabilities validation\n- known intents only\n- non-empty arrays\n- no duplicates]
    D -->|Has strictModesByIntent?| E[strictModesByIntent must subset modesByIntent\n- only verifier modes allowed]
    D -->|No| F[Mode declared: unsupported-mode is a conformance error]
    E --> F
    F --> G[Per-vector: assertDeclaredModeWasExecuted]
    G -->|unsupported-mode returned| H[Test FAILS immediately]
    G -->|accept / reject| I[recordModeExecution]
    I --> J[afterAll: assertStrictModeCoverage]
    J -->|strict mode missing accept OR reject| K[Suite FAILS]
    J -->|both outcomes covered| L[Strict mode gate passes]
    C --> M[afterAll: UNSUPPORTED_MODE_EXEMPTIONS check]
    M -->|no exemption and 0 executions| N[Orphaned mode error]
    M -->|exemption present| O[OK]
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[Runner manifest JSON] -->|isRunnerManifest| B{Has modesByIntent?}
    B -->|No| C[Default: unsupported-mode allowed for all modes]
    B -->|Yes| D[isModeCapabilities validation\n- known intents only\n- non-empty arrays\n- no duplicates]
    D -->|Has strictModesByIntent?| E[strictModesByIntent must subset modesByIntent\n- only verifier modes allowed]
    D -->|No| F[Mode declared: unsupported-mode is a conformance error]
    E --> F
    F --> G[Per-vector: assertDeclaredModeWasExecuted]
    G -->|unsupported-mode returned| H[Test FAILS immediately]
    G -->|accept / reject| I[recordModeExecution]
    I --> J[afterAll: assertStrictModeCoverage]
    J -->|strict mode missing accept OR reject| K[Suite FAILS]
    J -->|both outcomes covered| L[Strict mode gate passes]
    C --> M[afterAll: UNSUPPORTED_MODE_EXEMPTIONS check]
    M -->|no exemption and 0 executions| N[Orphaned mode error]
    M -->|exemption present| O[OK]
Loading

Reviews (10): Last reviewed commit: "Merge remote-tracking branch 'origin/fix..." | Re-trigger Greptile

Comment thread .github/workflows/ci.yml Outdated
Comment thread .github/workflows/ci.yml Outdated
Comment thread harness/src/conformance/x402.ts
@EfeDurmaz16

Copy link
Copy Markdown
Collaborator Author

@greptile-apps please re-review. The router now classifies both sides of renames, validates every classifier output, requires selected workflows XOR explicit docs-only, and exercises missing/contradictory/skip cases in a blocking gate test; adversarial Sol review is clean.

@EfeDurmaz16

Copy link
Copy Markdown
Collaborator Author

@greptile-apps please review

Route x402 and session conformance through the real SDK verifiers and boot probes instead of accepting a fixture echo, so a verifier that never runs can no longer report a false pass. Adds the typescript adapter-boot fixture and expands the conformance, protocol, x402, session-voucher, value-binding, boot-policy and e2e tests to cover the routed path.
Add assertDeclaredModeWasExecuted and assertStrictModeCoverage: an SDK that declares support for a mode must execute both an accept and a reject vector for it, otherwise the run fails. Blocks an adapter from declaring a mode and silently returning unsupported-mode for every vector.
…ectors

Each runner manifest now declares the modes it supports per intent (for example go.json declares charge and session canonical-bytes and x402-exact verify-x402-transaction) so the mode-support gate can hold it to that contract. Adds the canonical-json-rejects-lone-surrogate-value and canonical-json-es6-number-boundaries vectors plus a session-voucher reject vector.
assert-run-count.mjs forces the settlement pair-test floor to >= 1 and fails when zero pair-tests execute or when vitest reports success=false, closing the passWithNoTests false-green. Covers it plus the coverage-shape, matrix-coverage, ci-coverage and vector-accounting gate tests; ci-coverage-gate.test.ts pulls in the yaml dev dependency.
verify-routed-sdk-gate.mjs fails when any routed SDK job finishes non-success and rejects the vacuous nothing-selected-nothing-ran case via docs_only === (selected.length > 0). Pins workflow selection through select-pr-workflows with a fixture-backed test and wires the gate into pr-routing.yml.
check-rust-coverage.py validates the llvm-cov JSON with a default floor of 90 on lines and regions, aggregate and per-file, over non-empty mpp and x402 scopes, and exits non-zero on any shortfall or malformed report. Ships an adversarial self-test.
failopen-default-store.yaml flags a replay or session store that silently falls back to an in-memory default on the go and python SDK server sources, while the good fixtures stay clean. Tightens the gated-optional-field rule and the semgrep runner and makes findings blocking.
Add the pr216 delivery-accounting ledger with validate, generate and reconcile scripts, a payment-channels revision gate, and publish-workflow and repo-hygiene guards. Remove the two vendored solana-mpp tarballs and reject tracked package tarballs going forward.
Invoke the run-count, Rust-coverage, delivery-ledger and payment-channels revision gates from ci.yml, pin the payment-channels checkout ref and add the focused Python session fault-injection leg in harness.yml, and build harness verifiers with cargo --locked in setup-harness. Bounds the lua coverage leg and refreshes the go and python legs.
Rebasing this leaf onto the harness base tip pulls in PR #214 (solana-go v2), which rewrites the four go dependency manifests (harness/go-{client,server}/go.{mod,sum}). Their delivery-head blob no longer matches the 216 source, so the ledger reconciles them from integrated to open_pr owned by #214 and repoints the surviving integrated evidence at the current delivery head. Produced by scripts/reconcile-pr216-open-deliveries.mjs; validate-pr216-ledger passes.
The native source-contract probes asserted Rust markers PR #227 never
ships (is_shared legacy capability, config.allow_unsafe_memory_store,
the "atomic durable shared replay store is required" message) and PHP
error text the php leaf never emits ("MPP requires an injected atomic
durable/shared replay store", "does not affirm durable/shared
capability"). Those probes were permanently red against the real SDK
sources.

Replace them with the same mechanism the #238 finalize agent uses: an
asserted-skip roster for SDKs that do not implement the shared
PAY_KIT_ALLOW_INMEMORY_REPLAY_STORE fail-closed contract, plus a
git-grep honest-roster test that REDs the moment any rostered SDK gains
an opt-in reference, forcing promotion to a live probe instead of a
silent skip. No probe now asserts a contract no leaf delivers.
This leaf shipped the identical missing-ATA blocking change (remove
continue-on-error from the session no-ATA leg, "This regression is
blocking" comment) that #228 already delivers, and #228 owns python.yml.
Revert python.yml to base so only #228 delivers it and the two leaves do
not conflict at integration.
…o boot-policy roster

cc93323 dropped the php/ruby/lua source-contract probes and mislabeled all
three in unimplementedProbes as having "no off-localnet fail-closed boot
guard". That is false: each enforces the same money-path policy natively.

  - php   src/Protocols/Mpp/Adapter.php + Server/SolanaChargeHandler.php reject
          an omitted or non-durable/non-shared replay store off localnet
  - ruby  lib/pay_kit/protocols/mpp/runtime.rb raises unless localnet or a
          durable replay_store is supplied
  - lua   protocols/mpp/init.lua + server/init.lua error unless localnet or a
          shared replay store is configured

Remove php/ruby/lua from unimplementedProbes and cover them with native
source-contract probes, gated by the same git-grep detection #238 uses
(sdkFilesReferencingOptIn kept byte-identical; a parallel marker helper carries
each SDK's native rejection string). A probe auto-requires when its guard is
in-tree and asserts-skip with an honest reason otherwise. Patterns are verified
against current source, unlike the stale strings cc93323 removed.

rust/kotlin/swift stay asserted-skip (verified: no such guard in-tree).
The go/ts/python fail-closed probes required their SDK guard, but those
remediations land in sibling leaves (#227/#238/#228), so in this
harness leaf they hard-failed. Resolve each covered probe through the
same sdkImplementsGuard git-grep mechanism the native php/ruby/lua
probes already use: a probe is required only when its SDK source
references the opt-in marker in-tree, else it asserts-skip with a loud
PENDING note and auto-promotes to required at integration. Moves the
REPO_ROOT const above the probe resolution to avoid a module-eval
temporal-dead-zone error.
python.yml's missing-ATA step is owned and gated by the Python leaf
(#228); this harness leaf asserts only the harness.yml step it modifies.
Both leaves make the step blocking; the step-name reconciliation lands
at #216 integration.
@EfeDurmaz16
EfeDurmaz16 force-pushed the fix/harness-adversarial-hardening branch from cda6f96 to 66964c3 Compare July 13, 2026 22:06
@EfeDurmaz16

Copy link
Copy Markdown
Collaborator Author

@greptileai review

python.yml checked out Moonsong-Labs/solana-payment-channels while
go.yml, harness.yml, and the build-payment-channels action all use
solana-foundation/payment-channels at the same ref. The
check-payment-channels-revision gate this leaf adds correctly flagged
the drift: a mismatched program checkout decodes the epoch-aware
openSlot bytes as a recipient count. Pin python.yml to the same
repository so every payment-channels build uses one program.
@EfeDurmaz16

Copy link
Copy Markdown
Collaborator Author

@greptileai review

@EfeDurmaz16
EfeDurmaz16 marked this pull request as ready for review July 13, 2026 23:36
@EfeDurmaz16
EfeDurmaz16 requested a review from lgalabru July 13, 2026 23:36
…216 equivalence ledger

Those five redelivery PRs landed on main rather than the #219 line, so their
open_pr followUps are corrected from 'Land PR #X' to note the merge and the
revalidation that follows when #219 rebases onto main. Validator + self-test
still pass (113 commits, 237 paths accounted for).
Rebuilds rehearsal/pr216-integration purely from the current leaf heads in
the canonical merge order. Conflicts abort loudly: the resolution belongs in
the later leaf of the conflicting pair, never in the rehearsal branch, so a
green rehearsal is green by construction.
…into fix/harness-adversarial-hardening

# Conflicts:
#	harness/test/boot-policy.test.ts
…nto fix/harness-adversarial-hardening

# Conflicts:
#	.github/workflows/ci.yml
#	.github/workflows/harness.yml
#	harness/test/boot-policy.test.ts
… into fix/harness-adversarial-hardening

# Conflicts:
#	harness/test/boot-policy.test.ts
#	harness/test/value-binding-verify.test.ts
…compare rule

typeof secret !== 'string' inspects the value's runtime type tag, not its
bytes, so it cannot leak match progress; the blocking scan flagged the
pay-kit challengeBindingSecret validator for it. Pin the class with a
known-good fixture so the self-test keeps the exclusion honest.
…o fix/harness-adversarial-hardening

# Conflicts:
#	scripts/check-rust-coverage.py
#	scripts/check-rust-coverage_test.py
…osed cascade

Re-run the reconcile against the tree that now carries every redelivery leaf
(the same tree #240 rebuilds from the leaf heads): two harness paths a sibling
leaf superseded move from integrated to open_pr evidence, and the delivery
baseline advances. Validator and self-test pass; the blocking radar scan on
this tree reports zero findings.
…hygiene gates

The ruby native-guard probe follows #235's durable_shared_replay_store?
predicate rename (the guard itself is unchanged and stronger), and the
missing-ATA hygiene gate matches the reconciled deliver-or-reject step name
while still asserting the step stays blocking with the red-fault env.
… fixtures

The session verifier now requires a valid context slot on confirmation
responses (it pins follow-up account reads via min_context_slot), so the
bound-tx mock reports one; and the close settle+distribute composer decodes
the signed wire to bind the broadcast signature before submitting, so the
fixture compiles the composed instructions into a real signed transaction and
asserts the reported signature is wire-derived, never the RPC response value.
… union

Order the mpp-session leaf (#239) ahead of the python leaf (#228) so their
overlapping edits to the shared ci.yml + boot-policy.test.ts interleave without
a phantom 3-way conflict (both are ultimately reconciled by the harness meta
leaf, which owns those paths and merges last). Add a pure-union assertion after
the merges: every non-merge commit the rehearsal introduces over base must be
contained in some leaf head, else it is a hand-authored patch on #240 and the
script refuses to push. This makes 'green by construction' mechanically checked,
not asserted.
… replay store

The mainnet-fork settlement suite injected `{ ...Store.memory(), isShared: true }`
into createPayKit. Once the MPP replay-store policy requires an atomic
putIfAbsent (a spread of the legacy get/put store has none), createPayKit fails
closed with a ConfigurationError before any settlement runs, so every gated
settlement assertion errored in beforeAll. This surfaces only in the full
cascade: the leaf that tightened the policy does not run the on-chain leg, and
the on-chain leg's leaves predate the tightening. Build a real single-process
store (Map-backed get/put/delete + atomic putIfAbsent, isShared+isDurable) and
declare it production, matching the SDK's own createSharedTestReplayStore
fixture. Drop mpp.allowUnsafeMemoryStore: a production store needs no unsafe
escape hatch, and keeping it would be a false 'allow unsafe' marker on a
money-path test.
…me rationale

mpp/client/subscription.rs and mpp/server/session.rs fall under the per-file
90% floor for the same reason the three existing exemptions do: their fail-closed
validation and reject paths are unit-covered (mock RPC), but the residual gap is
the on-chain RpcClient success paths (SubscriptionAuthority init broadcast,
fetch_channel_account + settle_and_seal assembly, get_transaction top-up fetch)
plus async state-machine desugaring regions that stay count-0 under llvm-cov,
none of which a unit run can exercise. Each exemption names its concrete
surfpool-integration follow-up. The session entry also records a real, still-open
reject-coverage gap (on-chain channel status/mint/payee/rent-payer mismatch
rejects are reachable via the SessionAccountRpc mock but not yet asserted) so it
is documented rather than hidden. Non-exempt aggregate stays 96.97%/96.66% (mpp)
and 94.46%/94.02% (x402), well above the 90% floor.
…e import)

The pre-merge of the shared blockhash cache left two artifacts that only the
integrated Rust gate catches (the leaf's own core/Rust job stops earlier): two
consecutive blank lines that cargo fmt --check rejects, and a second, redundant
`use super::*;` inside the test module (module-level import at the top already
covers it) that clippy -D warnings rejects as unused. Collapse the blanks and
drop the duplicate import; behavior is unchanged.
@EfeDurmaz16
EfeDurmaz16 force-pushed the fix/harness-adversarial-hardening branch from 9b1b881 to b9fe46b Compare July 14, 2026 16:29
…' into fix/harness-adversarial-hardening

# Conflicts:
#	harness/test/ci-coverage-gate.test.ts
…cascade root

Add the end-state seal: the cascade root lets the Rust conformance/coverage
steps and the python session leg wait on their subjects, so this leaf (the last
in the chain) asserts every such subject exists in the integrated tree — a
pending gate cannot silently outlive its purpose. Track the restructured
cascade in the rebuild script: the gate-activation root (#244) enters first and
the python leaf precedes the mpp-session leaf, mirroring the telescoped PR
chain.
@EfeDurmaz16
EfeDurmaz16 changed the base branch from split/pr216-ci-harness-mega to fix/ruby-replay-store-capability July 15, 2026 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant