ci: automated performance regression detection

[membphis]

Motivation

The benchmark suite (benches/lua_bench.lua) is run manually via make bench and results are committed to docs/benchmarks.md. There is no automated gate that catches performance regressions on PRs.

This means a change that halves parse throughput (e.g. disabling AVX2 dispatch, breaking the skip-cache hot path) would merge silently.

Proposed approach

Add a CI job that runs a short, deterministic benchmark on every PR and posts a comparison comment:

1. Checkout base branch → build → record ops/s for small and medium scenarios.
2. Checkout PR branch → build → record ops/s for same scenarios.
3. If PR ops/s < base × 0.95 (5% regression threshold), fail the job and post a comment with the delta table.

Implementation options

• criterion Rust benchmarks — deterministic, CI-friendly, produces JSON output. Requires porting the Lua bench logic to Rust (measures Rust-side throughput, not LuaJIT JIT-compiled throughput).
• bencher GitHub Action — wraps arbitrary benchmark output, stores historical data, posts PR comments. Can wrap the existing lua_bench.lua output.
• Simple shell diff — run lua_bench.lua twice, compare with awk. Fragile on shared runners due to noisy neighbors.

Recommended: bencher GitHub Action pointing at the existing Lua bench, filtering for small and github-100k scenarios (fast, representative).

Affected files

• .github/workflows/ci.yml — add bench job
• benches/lua_bench.lua — ensure machine-parseable output (already has structured output lines)

Related

• Issue bench: add large_dump.json (~20 MB) and deep_nest.json fixtures #24 (add large fixtures) — larger fixtures will also benefit from regression tracking once they land.

[coderabbitai]

🔗 Related PRs

#43 - test: add cjson and simdjson fixture coverage [merged]
#45 - feat: release v0.1.0 [merged]
#54 - perf: Lua encode + cache + scanner + validation stack optimizations [merged]
#58 - docs: fix install instructions and qualify ARM64 performance claims [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[membphis]

Closing as not planned for now. GitHub-hosted Actions runners do not provide stable enough CPU performance to make a reliable perf-regression gate, and a noisy threshold would be more distracting than useful. Revisit this as either advisory-only reporting or a blocking gate on dedicated/self-hosted benchmark hardware.

[membphis]

补充关闭原因：这个 issue 原本希望在 CI 中自动发现性能回退，但 GitHub-hosted Actions 使用共享 runner，CPU 型号、负载、调度、turbo/频率状态都不稳定，无法保证 base branch 和 PR branch 前后两次 benchmark 的性能可比性。把这种结果做成 blocking gate 很容易产生误报，反而会降低 CI 信号质量。

后续如果还需要性能回归检测，建议重新开一个范围更明确的 issue，二选一：

1. advisory-only：只在 PR 上发布 benchmark 对比结果或 artifact，不阻断合并；
2. dedicated benchmark runner：使用固定/自托管机器，再考虑阈值和 blocking gate。