Skip to content

Add CodSpeed performance benchmarks#20

Draft
codspeed-hq[bot] wants to merge 1 commit into
mainfrom
codspeed/wizard-1782953036751
Draft

Add CodSpeed performance benchmarks#20
codspeed-hq[bot] wants to merge 1 commit into
mainfrom
codspeed/wizard-1782953036751

Conversation

@codspeed-hq

@codspeed-hq codspeed-hq Bot commented Jul 2, 2026

Copy link
Copy Markdown

Summary

This PR sets up continuous performance measurement for sfos-rs with CodSpeed. It adds a criterion-based benchmark suite for the sfos-sdk crate, wires it into CI through a dedicated GitHub Actions workflow, and adds a CodSpeed status badge to the README.

Changes

  • crates/sfos-sdk/Cargo.toml — added codspeed-criterion-compat as a dev-dependency (aliased to criterion, so the benchmarks read as standard criterion code) and registered the sdk benchmark target with harness = false.
  • crates/sfos-sdk/benches/sdk.rs — a new benchmark suite covering the core offline analysis pipeline:
    • parse_entities on three representative Entities.xml fixtures (sample, NAT, VPN)
    • to_model — Sophos config to vendor-neutral IR bridge
    • graph::build, to_dot, and to_mermaid — zone graph construction and rendering
    • explain and forward — reachability / packet-forwarding analysis, including DNAT follow-through
    • shadow::detect — shadowed / dead rule detection across all rule sets
  • .github/workflows/codspeed.yml — a new workflow that builds the benchmarks with cargo codspeed build and runs them via the CodSpeed action in simulation mode. It uses OpenID Connect authentication (id-token: write) and pins actions by commit hash to match the repository's existing convention.
  • README.md — added the CodSpeed badge alongside the existing status badges.
  • Cargo.lock — updated for the new dev-dependency.

The benchmarks were verified locally in simulation mode before opening this PR; all 10 benchmarks build and run successfully.

Notes

  • Simulation mode is used, which provides deterministic, hardware-agnostic CPU measurements with low variance and automatic flamegraphs — a good fit for this CPU-bound parsing and analysis code. It runs on ubuntu-latest, no special runners required.
  • The commit is signed off to satisfy the repository's DCO check.

Next steps

  • Merge this PR so CodSpeed can establish a performance baseline on main.
  • Subsequent PRs will automatically get performance reports comparing against that baseline.
  • Consider expanding coverage with larger/synthetic fixtures to exercise the analysis paths at scale.

Set up continuous performance measurement for the sfos-sdk crate using
criterion with the CodSpeed compatibility layer. Benchmarks cover parsing,
IR bridging, zone graph build/render, reachability analysis, and shadowed
rule detection. Adds a GitHub Actions workflow to run them in CI and a
CodSpeed badge to the README.

Signed-off-by: CodSpeed Bot <no-reply@codspeed.io>
@codspeed-hq

codspeed-hq Bot commented Jul 2, 2026

Copy link
Copy Markdown
Author

CodSpeed setup failed

The benchmarks, CI workflow (.github/workflows/codspeed.yml, simulation mode, OIDC), README badge, and PR (#20) were all set up correctly and the benchmarks were verified locally. However, the codspeed workflow cannot run in CI: it ends in startup_failure at 0s with no jobs created and no logs, while every other workflow in the repo (build, security, trivy, DCO) runs normally. The codspeed workflow is the only one that references the required third-party actions CodSpeedHQ/action and moonrepo/setup-rust. This is the signature of a GitHub Actions "Allow select actions and reusable workflows" policy that does not include these actions on its allow-list, so GitHub blocks the run before it starts. This is an environment/permissions restriction that cannot be fixed from the repository code. To resolve it, a repo/org admin must go to Settings > Actions > General > Actions permissions and either allow all actions, or add CodSpeedHQ/action@* and moonrepo/setup-rust@* to the allowed actions list. Once the actions are allowed, re-run the codspeed workflow on PR #20 to complete the setup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant