Skip to content

feat: bench/ micro-benchmark requires pre-generating one-time signed payloads manually β€” add a make bench-generate target that automates payload generation and prevents stale benchmark resultsΒ #255

Description

@divyanshim27

πŸš€ Problem Statement

bench/README.md documents:

"Current reruns must generate enough one-time signed payloads because verifier nonces are replay-protected."

This means anyone who wants to reproduce the committed benchmark results in bench/RESULTS-2026-05-13.txt must:

  1. Manually generate N signed payloads using the SDK or wallet
  2. Store them in the correct format for the bench harness
  3. Run the benchmark before any payload expires (5-minute TTL)

There is no automation for this. A contributor who wants to run make bench to verify their Rust changes didn't regress performance has no clear path to generating valid fresh payloads. The Makefile has no benchmark target at all. As a result:

  • Benchmark results cannot be independently verified by contributors
  • The committed RESULTS-*.txt files cannot be reproduced without undocumented manual steps
  • Performance regressions from Rust verifier changes go undetected

Proposed Fix

Add a make bench-generate target that uses the existing TypeScript SDK to generate a configurable number of fresh signed payloads and stores them in bench/payloads.json:

# In Makefile

BENCH_PAYLOAD_COUNT ?= 2000
BENCH_PAYLOAD_FILE := bench/payloads.json

.PHONY: bench-generate
bench-generate: ## Generate fresh signed payloads for benchmarking
    @echo "Generating $(BENCH_PAYLOAD_COUNT) signed payloads..."
    @cd sdk/typescript && PAYGATE_GATEWAY_URL=http://localhost:3000 \
        EVM_PRIVATE_KEY=$(SERVER_WALLET_PRIVATE_KEY) \
        bun run scripts/generate-bench-payloads.ts \
        --count $(BENCH_PAYLOAD_COUNT) \
        --output ../../$(BENCH_PAYLOAD_FILE)
    @echo "Payloads written to $(BENCH_PAYLOAD_FILE)"

.PHONY: bench
bench: ## Run verifier benchmark (requires bench-generate first, or BENCH_PAYLOAD_FILE set)
    @if [ ! -f "$(BENCH_PAYLOAD_FILE)" ]; then \
        echo "Error: $(BENCH_PAYLOAD_FILE) not found. Run 'make bench-generate' first."; \
        exit 1; \
    fi
    @cd bench && cargo bench --bench verifier_bench
    @cp bench/target/criterion/*/base/estimates.json bench/RESULTS-$(shell date +%Y-%m-%d).json

And sdk/typescript/scripts/generate-bench-payloads.ts that uses the existing PaygateClient to hit POST /api/ai/summarize, collect the 402 payment contexts, sign them all, and write to the output JSON.

Add a CI gate in .github/workflows/bench.yml that:

  • Runs on pushes to verifier/ directory
  • Generates 100 payloads (sufficient for a regression check, not a full benchmark)
  • Runs the bench
  • Fails if p99 latency regresses > 20% from the baseline in bench/RESULTS-*.txt

Files to Create/Modify

File Change
Makefile Add bench-generate and bench targets
sdk/typescript/scripts/generate-bench-payloads.ts New payload generator script
.github/workflows/bench.yml New CI workflow for regression detection
bench/README.md Update to document make bench-generate workflow

Suggested labels: enhancement, developer-experience, bench, level: intermediate

I would like to work on this. Could you please assign it to me?

Metadata

Metadata

Assignees

Labels

documentationImprovements or additions to documentationrustPull requests that update rust codetriageNeeds maintainer triage.type:devopsDeployment, observability, operations, CI, or infrastructure work.type:docsDocumentation, API docs, examples, or contributor docs.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions