Skip to content

perf(engine): O(n) detection rules for dataset-scale traces #36

Description

@sepehr-safari

Summary

Several detection rules are O(n²) in the event count — they match events to
each other by scanning (e.g. detectFailedAuthorization scans all events for each
result; detectConnectorFault, detectUnexpectedStart, detectSlowResponse,
detectUnresponsiveCsms similarly). They mirror the toolkit's algorithms and are
fine at the browser-scale 10k-event cap, but stall at Studio's trusted capacity.

As a result, the workspace currently skips detection above
detection.max_events_for_detection (50 000) — see ADR-0007. A 500k-event trace
opens, correlates, and is fully inspectable, but shows no detected failures.

Goal

Make the hot rules O(n) (or O(n log n)) so detection runs across the full
trusted capacity, then raise/remove the cap.

Approach

  • Index once, look up O(1): build StringHashMap maps from message-id → call
    event (Authorize/response matching) and pair StartTransaction ↔ StopTransaction
    in a single pass, instead of nested scans.
  • Keep output bit-identical: the contract-v1 conformance harness (15/15
    goldens) is the safety net — the optimization must not change a single detected
    code. Add a large-trace timing guard.

Acceptance criteria

  • The O(n²) rules are O(n) / O(n log n); a 500k-event trace detects in well
    under a second.
  • All 15 conformance goldens still match exactly.
  • detection.max_events_for_detection raised to (or past) the trusted
    capacity, and the workspace no longer skips detection at 500k.

Context

Follow-up from #29 (trusted ingestion). Deferred deliberately: optimizing six
rules bit-exactly is a distinct, conformance-gated effort, separate from the
capacity/ingestion work.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions