Skip to content

feat(scenarios): assert-based scenarios + 5 new scenarios + report comparison#101

Merged
sepehr-safari merged 1 commit into
mainfrom
feat/scenarios-assertions
Jul 9, 2026
Merged

feat(scenarios): assert-based scenarios + 5 new scenarios + report comparison#101
sepehr-safari merged 1 commit into
mainfrom
feat/scenarios-assertions

Conversation

@sepehr-safari

Copy link
Copy Markdown
Member

Summary

Adds 5 new scenarios that use rich assertions (from Issue #86), bringing the total to 15 scenarios. Also adds compareScenarioReports() for regression testing.

New Scenarios

  1. slow-csms-response — CSMS takes 15s to respond to BootNotification. Uses timing assertion.
  2. meter-anomaly — Meter values decrease (non-monotonic). Uses payload_field and event_count assertions.
  3. short-session — Session lasts 10s. Uses session_state and event_count assertions.
  4. heartbeat-irregular — Heartbeats at 600s vs 300s expected. Uses event_count assertion.
  5. unresponsive-csms — CSMS never responds to BootNotification. Uses failure_severity and failure_count assertions.

Report Comparison

export function compareScenarioReports(a: ScenarioEvalResult, b: ScenarioEvalResult): ScenarioComparison;

Compares two scenario evaluation results — useful for regression testing.

Files Changed

  • packages/toolkit/src/scenarios/__scenarios__/slow-csms-response.ts — new
  • packages/toolkit/src/scenarios/__scenarios__/meter-anomaly.ts — new
  • packages/toolkit/src/scenarios/__scenarios__/short-session.ts — new
  • packages/toolkit/src/scenarios/__scenarios__/heartbeat-irregular.ts — new
  • packages/toolkit/src/scenarios/__scenarios__/unresponsive-csms.ts — new
  • packages/toolkit/src/scenarios/compare.ts — new comparison module
  • packages/toolkit/src/scenarios/compare.test.ts — 6 tests
  • packages/toolkit/src/scenarios/index.ts — register 5 new scenarios + export compareScenarioReports
  • packages/toolkit/src/scenarios/index.test.ts — updated for 15 scenarios + v0.3 failure codes
  • tests/external-fixture/test.mjs — updated scenario count to 15
  • .changeset/v03-scenarios-assertions.md — minor bump changeset
  • CURRENT_STATE.md — updated with v0.3.0 progress

Closes #87

Checklist

  • Scenario type extended with optional assertions field (done in feat(core): rich scenario assertions #86)
  • 5 new scenarios using rich assertions
  • All existing 10 scenarios still work (backward compat)
  • compareScenarioReports() implemented with ScenarioComparison return type
  • Unit tests for new scenarios and comparison function
  • All scenario data is synthetic
  • External fixture test updated
  • Changeset added
  • CURRENT_STATE.md updated
  • Lint, typecheck, test, format all pass (365 tests)

@sepehr-safari sepehr-safari merged commit 8db857e into main Jul 9, 2026
2 checks passed
@sepehr-safari sepehr-safari deleted the feat/scenarios-assertions branch July 9, 2026 06:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(scenarios): assert-based scenario format + report comparison

1 participant