π Problem Statement
bench/README.md documents:
"Current reruns must generate enough one-time signed payloads because verifier nonces are replay-protected."
This means anyone who wants to reproduce the committed benchmark results in bench/RESULTS-2026-05-13.txt must:
- Manually generate N signed payloads using the SDK or wallet
- Store them in the correct format for the bench harness
- Run the benchmark before any payload expires (5-minute TTL)
There is no automation for this. A contributor who wants to run make bench to verify their Rust changes didn't regress performance has no clear path to generating valid fresh payloads. The Makefile has no benchmark target at all. As a result:
- Benchmark results cannot be independently verified by contributors
- The committed
RESULTS-*.txt files cannot be reproduced without undocumented manual steps
- Performance regressions from Rust verifier changes go undetected
Proposed Fix
Add a make bench-generate target that uses the existing TypeScript SDK to generate a configurable number of fresh signed payloads and stores them in bench/payloads.json:
# In Makefile
BENCH_PAYLOAD_COUNT ?= 2000
BENCH_PAYLOAD_FILE := bench/payloads.json
.PHONY: bench-generate
bench-generate: ## Generate fresh signed payloads for benchmarking
@echo "Generating $(BENCH_PAYLOAD_COUNT) signed payloads..."
@cd sdk/typescript && PAYGATE_GATEWAY_URL=http://localhost:3000 \
EVM_PRIVATE_KEY=$(SERVER_WALLET_PRIVATE_KEY) \
bun run scripts/generate-bench-payloads.ts \
--count $(BENCH_PAYLOAD_COUNT) \
--output ../../$(BENCH_PAYLOAD_FILE)
@echo "Payloads written to $(BENCH_PAYLOAD_FILE)"
.PHONY: bench
bench: ## Run verifier benchmark (requires bench-generate first, or BENCH_PAYLOAD_FILE set)
@if [ ! -f "$(BENCH_PAYLOAD_FILE)" ]; then \
echo "Error: $(BENCH_PAYLOAD_FILE) not found. Run 'make bench-generate' first."; \
exit 1; \
fi
@cd bench && cargo bench --bench verifier_bench
@cp bench/target/criterion/*/base/estimates.json bench/RESULTS-$(shell date +%Y-%m-%d).json
And sdk/typescript/scripts/generate-bench-payloads.ts that uses the existing PaygateClient to hit POST /api/ai/summarize, collect the 402 payment contexts, sign them all, and write to the output JSON.
Add a CI gate in .github/workflows/bench.yml that:
- Runs on pushes to
verifier/ directory
- Generates 100 payloads (sufficient for a regression check, not a full benchmark)
- Runs the bench
- Fails if p99 latency regresses > 20% from the baseline in
bench/RESULTS-*.txt
Files to Create/Modify
| File |
Change |
Makefile |
Add bench-generate and bench targets |
sdk/typescript/scripts/generate-bench-payloads.ts |
New payload generator script |
.github/workflows/bench.yml |
New CI workflow for regression detection |
bench/README.md |
Update to document make bench-generate workflow |
Suggested labels: enhancement, developer-experience, bench, level: intermediate
I would like to work on this. Could you please assign it to me?
π Problem Statement
bench/README.mddocuments:This means anyone who wants to reproduce the committed benchmark results in
bench/RESULTS-2026-05-13.txtmust:There is no automation for this. A contributor who wants to run
make benchto verify their Rust changes didn't regress performance has no clear path to generating valid fresh payloads. TheMakefilehas no benchmark target at all. As a result:RESULTS-*.txtfiles cannot be reproduced without undocumented manual stepsProposed Fix
Add a
make bench-generatetarget that uses the existing TypeScript SDK to generate a configurable number of fresh signed payloads and stores them inbench/payloads.json:And
sdk/typescript/scripts/generate-bench-payloads.tsthat uses the existingPaygateClientto hitPOST /api/ai/summarize, collect the 402 payment contexts, sign them all, and write to the output JSON.Add a CI gate in
.github/workflows/bench.ymlthat:verifier/directorybench/RESULTS-*.txtFiles to Create/Modify
Makefilebench-generateandbenchtargetssdk/typescript/scripts/generate-bench-payloads.ts.github/workflows/bench.ymlbench/README.mdmake bench-generateworkflowSuggested labels:
enhancement,developer-experience,bench,level: intermediateI would like to work on this. Could you please assign it to me?