How should runtime benchmark artifacts be interpreted?

[techleadevelopers]

The repo includes baseline and adversarial runtime load-test artifacts.

What is the intended interpretation difference between hot-gate benchmark latency and broader operational pipeline latency?

[simonbcross]

The hot-gate benchmark and the broader operational pipeline measure different parts of the system.

The hot-gate benchmark is intentionally narrow. It measures only the synchronous decision path:

• decode
• fast preflight
• adaptive preflight
• adaptive quote
• total hot gate

It excludes RPC calls, payload lookup, relay submission, dashboard IO, persistence, receipt polling, and realized PnL accounting. Because of that, it is useful for evaluating CPU-side latency, rejection quality, and adversarial input handling.

The broader operational pipeline includes more runtime overhead:

• async scheduling
• mempool/network IO
• pool/cache lookups
• payload construction
• relay/RPC submission
• persistence
• finalization observation

So the hot-gate report can show microsecond-level latency, while the operational pipeline can remain millisecond-scale. They should not be compared as the same metric.

A good interpretation is:

• use runtime_baseline_report.json to measure clean hot-path latency
• use runtime_adversarial_report.json to measure hostile-input rejection and tail latency
• use replay/live telemetry to measure realized PnL, gas avoided, inclusion quality, and revert avoidance

In short: hot-gate benchmarks prove fast rejection behavior; operational pipeline metrics prove production behavior.