M8 conformance baseline: 4 advisory failures — HARDENING.md fix work

[trendvidia]

ROADMAP M8 just landed cross-port. The new spec at docs/HARDENING.md defines the decoder-safety contract every port must enforce against untrusted input; the adversarial corpus tests it; the 9-port matrix workflow runs it on every protowire PR.

Companion check_decode reference for this port: scripts/check_decode.py (commit 51df4a6).

The workflow ships in advisory mode (continue-on-error: true) so failures do not block PRs. This issue tracks driving this port's regressions to zero so we can flip the gate to required for python.

Current baseline — 4 advisory failures (1 crash)

Corpus	Verdict	Reason
pxf/deep-nesting-200.pxf	FAIL_VERDICT	accepted; missing MaxNestingDepth cap (heavy lifting in C++ FFI)
pxf/deep-nesting-1000.pxf	FAIL_VERDICT	same
pxf/deep-nesting-100000.pxf	FAIL_CRASH	C++ FFI stack overflow leaks through to Python
pxf/invalid-utf8-string.pxf	FAIL_VERDICT	C++ FFI's PXF decoder doesn't enforce UTF-8 on proto3 string

The Python port passes most corpus inputs because:

• pb/deep-submessage-200.binpb rejected by google.protobuf (which has built-in recursion limits).
• pb/length-prefix-truncated.binpb rejected by google.protobuf.
• pxf/long-numeric.pxf rejected by Python's int(str) raising ValueError on the int64 parser path.
• pxf/lone-surrogate.pxf rejected at the lexer (already validates ValidRune).

What to fix

The 4 PXF failures are all on the C++ FFI side (the _protowire/module.cc extension and its protowire-cpp dependencies). Two HARDENING.md invariants in protowire-cpp will fix them upstream:

• §Recursion — MaxNestingDepth = 100 cap on the C++ PXF parser/decoder. Tracked in [protowire-cpp's M8 issue]; once that lands, this port inherits the fix automatically. The 100k-deep crash converts to a clean error returned through the FFI.
• §UTF-8 — strict UTF-8 enforcement on proto3 string populating in protowire-cpp's PXF decoder. Same upstream fix.

If the maintainers want a Python-side defense-in-depth in addition (e.g. setting sys.setrecursionlimit lower, or wrapping the FFI call in a stack-budget check), that's optional but not required to clear the corpus — the upstream C++ fixes are sufficient.

The pure-Python envelope.py decoder also has a couple of IndexError-vs-ValueError exception-type leaks the cross-port review flagged (envelope.py:131-132, 85-93); those aren't tested by the current corpus but are good hygiene fixes.

Reproduce locally

# In the venv with `pip install -e .` already done:
WALLCLOCK_SECONDS=10 bash ../protowire/scripts/cross_security_check.sh

# Or run a single corpus input:
python3 scripts/check_decode.py --format pxf \
  --schema adversarial.v1.Tree \
  --proto ../protowire/testdata/adversarial/adversarial.proto \
  --input ../protowire/testdata/adversarial/pxf/deep-nesting-100000.pxf
echo $?  # 0 = accepted, 1 = clean reject, 134/139 = crash via FFI

Convergence

When all 4 corpus inputs pass on this port, comment here. We'll flip continue-on-error: false for the python matrix entry in protowire/.github/workflows/security.yml. When all 9 ports converge, the workflow becomes a required check.

Cross-port context: 8 sibling per-port issues track the same convergence. The full cross-port matrix lives in protowire's ROADMAP § M8. Most of this port's findings are upstream of protowire-cpp — see that repo's M8 issue.

[trendvidia]

Cross-port convergence is tracked at trendvidia/protowire#39 — that issue holds the full 9-port verdict matrix and per-port convergence checklist.

[trendvidia]

M8 corpus convergence: all 4 advisory failures cleared on this port.

Root cause: wheels CI was pinned to protowire-cpp@v0.70.0 (SHA 0ac4067), which predates the M8 decoder-hardening commits (25b93ae PXF MaxNestingDepth + UTF-8, 329bdea SBE bounds) that landed on main via PR #2 (merge 7b712cd). The fixes are not in any tag yet.

Changes (this repo, on main):

• d86fc01 — bump PROTOWIRE_CPP_REF default v0.70.0 → main. Will bump to v0.71.0 once the C++ port cuts that tag.
• 4ed1684 — envelope.py hygiene: _dec_varint and _skip_field now raise ValueError on truncated input instead of leaking IndexError. Two regression tests added. (Cross-port review nit, not corpus-blocking — but in scope here so folded in.)

Verification (local rebuild against main):

=== deep-nesting-200       ===  reject: nesting depth exceeds 100  exit=1
=== deep-nesting-1000      ===  reject: nesting depth exceeds 100  exit=1
=== deep-nesting-100000    ===  reject: nesting depth exceeds 100  exit=1   (was FAIL_CRASH)
=== invalid-utf8-string    ===  reject (clean)                     exit=1

cross_security_check.sh shows zero python rows in the failure list. pytest: 51/51 pass.

Ready to flip continue-on-error: false for the python matrix entry in protowire/.github/workflows/security.yml whenever you're ready. Wheels CI on next push will rebuild against protowire-cpp@main.

[trendvidia]

Follow-up: pin bumped to protowire-cpp@v0.71.0 (commit df8e504, the first tag containing both M8 fixes).

1505bd3 — replaces the temporary main pin from d86fc01. Re-verified against the tag locally: all 4 adversarial PXF corpus inputs reject cleanly, pytest 51/51.

Wheels CI will rebuild against v0.71.0 on next push.

[trendvidia]

Released protowire-python@v0.71.0 (commit 69b4465, tag pushed) — package version aligned with protowire-cpp@v0.71.0. Wheels CI build queued for the tag push.

Final state of this port re: issue:

• Pinned to protowire-cpp@v0.71.0 (immutable tag).
• 4 adversarial corpus inputs reject cleanly (no acceptance, no crash).
• envelope.py cross-port ValueError contract enforced + tested.
• pytest 51/51, cross_security_check.sh shows zero python rows.

Ready to flip continue-on-error: false for the python matrix entry whenever you flip the gate.

[trendvidia]

Gate flip in flight: trendvidia/protowire#54 adds python to the required-port set in security.yml. CI on the PR will exercise the now-required gate against this port; once that's green and merged, HARDENING corpus regressions in protowire-python will hard-fail protowire PRs instead of being advisory.

[diegocoronado1416-collab]

ROADMAP M8 just landed cross-port. The new spec at docs/HARDENING.md defines the decoder-safety contract every port must enforce against untrusted input; the adversarial corpus tests it; the 9-port matrix workflow runs it on every protowire PR.

Companion check_decode reference for this port: scripts/check_decode.py (commit 51df4a6).

The workflow ships in advisory mode (continue-on-error: true) so failures do not block PRs. This issue tracks driving this port's regressions to zero so we can flip the gate to required for python.

Current baseline — 4 advisory failures (1 crash)

Corpus Verdict Reason
pxf/deep-nesting-200.pxf FAIL_VERDICT accepted; missing MaxNestingDepth cap (heavy lifting in C++ FFI)
pxf/deep-nesting-1000.pxf FAIL_VERDICT same
pxf/deep-nesting-100000.pxf FAIL_CRASH C++ FFI stack overflow leaks through to Python
pxf/invalid-utf8-string.pxf FAIL_VERDICT C++ FFI's PXF decoder doesn't enforce UTF-8 on proto3 string
The Python port passes most corpus inputs because:

• pb/deep-submessage-200.binpb rejected by google.protobuf (which has built-in recursion limits).
• pb/length-prefix-truncated.binpb rejected by google.protobuf.
• pxf/long-numeric.pxf rejected by Python's int(str) raising ValueError on the int64 parser path.
• pxf/lone-surrogate.pxf rejected at the lexer (already validates ValidRune).

What to fix

The 4 PXF failures are all on the C++ FFI side (the _protowire/module.cc extension and its protowire-cpp dependencies). Two HARDENING.md invariants in protowire-cpp will fix them upstream:

• §Recursion — MaxNestingDepth = 100 cap on the C++ PXF parser/decoder. Tracked in [protowire-cpp's M8 issue]; once that lands, this port inherits the fix automatically. The 100k-deep crash converts to a clean error returned through the FFI.
• §UTF-8 — strict UTF-8 enforcement on proto3 string populating in protowire-cpp's PXF decoder. Same upstream fix.

If the maintainers want a Python-side defense-in-depth in addition (e.g. setting sys.setrecursionlimit lower, or wrapping the FFI call in a stack-budget check), that's optional but not required to clear the corpus — the upstream C++ fixes are sufficient.

The pure-Python envelope.py decoder also has a couple of IndexError-vs-ValueError exception-type leaks the cross-port review flagged (envelope.py:131-132, 85-93); those aren't tested by the current corpus but are good hygiene fixes.

Reproduce locally

In the venv with pip install -e . already done:

WALLCLOCK_SECONDS=10 bash ../protowire/scripts/cross_security_check.sh

Or run a single corpus input:

python3 scripts/check_decode.py --format pxf
--schema adversarial.v1.Tree
--proto ../protowire/testdata/adversarial/adversarial.proto
--input ../protowire/testdata/adversarial/pxf/deep-nesting-100000.pxf
echo $? # 0 = accepted, 1 = clean reject, 134/139 = crash via FFI

Convergence

When all 4 corpus inputs pass on this port, comment here. We'll flip continue-on-error: false for the python matrix entry in protowire/.github/workflows/security.yml. When all 9 ports converge, the workflow becomes a required check.

Cross-port context: 8 sibling per-port issues track the same convergence. The full cross-port matrix lives in protowire's ROADMAP § M8. Most of this port's findings are upstream of protowire-cpp — see that repo's M8 issue.