PRD: Add Disruptor utility for real-device network impairment

[rishabh-slamcore]

Problem Statement

Slamcore engineers need a way to validate Aware update delivery against real devices under controlled network impairment without depending on the earlier container-only test-farm model. The current invocation flow can replay toy client scenarios and apply fleet-wide impairment inside a router container, but the practical test setup now uses real Aware devices, a real updater, and a disruptor machine sitting in the network path. Engineers need a reproducible, scenario-driven utility that impairs device-bound traffic while the updater starts downloads and owns success validation.

Solution

Add a Disruptor utility to test-farm. The utility reads a Disruptor Scenario File, discovers Slamcore Aware devices through mDNS, resolves the effective impairment policy for each discovered device, renders a per-device tc plan for the explicit client-facing interface, applies the plan, blocks while impairment is active, and tears the plan down when stopped. It also supports a dry-run mode that performs parsing, discovery, policy resolution, and plan rendering without applying tc.

The first implementation slice is static. Timed and repeating schedules are reserved in the product direction but not implemented in v1.

User Stories

1. As a Slamcore engineer, I want to run disruptor against real Aware devices, so that I can validate update delivery without building Aware-like containers.
2. As a Slamcore engineer, I want a YAML Disruptor Scenario File, so that disruption settings can be named, reviewed, and replayed.
3. As a Slamcore engineer, I want the default impairment to apply to every discovered Aware device, so that broad network degradation is easy to express.
4. As a Slamcore engineer, I want ordered device-specific overrides, so that selected devices can receive different impairment from the discovered fleet default.
5. As a Slamcore engineer, I want first-match-wins override behavior, so that override order is predictable and author controlled.
6. As a Slamcore engineer, I want overrides to replace the whole default policy, so that a device-specific policy has no hidden inherited impairment.
7. As a Slamcore engineer, I want to set an override to no impairment, so that selected devices can remain controls while the rest of the discovered fleet is disrupted.
8. As a Slamcore engineer, I want scenario authors to use mDNS-resolved device identities rather than IP literals, so that scenarios are not coupled to transient DHCP addresses.
9. As a Slamcore engineer, I want unresolved selectors to be best effort warnings, so that disruptor can still impair matched devices found on the network.
10. As a Slamcore engineer, I want unresolved selectors represented structurally, so that dry-run output and tests do not depend on log text.
11. As a Slamcore engineer, I want the effective policy per discovered device to be visible in dry-run, so that I can confirm scenario intent before changing network state.
12. As a Slamcore engineer, I want dry-run to render the planned tc tree, so that I can inspect the commands without requiring root or changing the interface.
13. As a Slamcore engineer, I want disruptor to require the client-facing interface explicitly, so that it does not guess the wrong physical NIC.
14. As a Slamcore engineer, I want disruptor to clear stale tc state on the explicit interface before setup, so that repeated runs recover cleanly from prior crashes.
15. As a Slamcore engineer, I want disruptor to tear down its root qdisc when stopped, so that the lab machine returns to normal networking after a scenario.
16. As a Slamcore engineer, I want disruptor to block while impairment is active, so that Ctrl-C or process supervision defines the active disruption window.
17. As a Slamcore engineer, I want v1 impairment fields to include delay, loss, and bandwidth limit, so that the feature reuses the impairment model already proven in test-farm.
18. As a Slamcore engineer, I want bandwidth limit and netem impairment to compose correctly, so that rate limiting happens before delay or loss.
19. As a Slamcore engineer, I want device-bound traffic to be impaired only on disruptor egress, so that unrelated routes and device-to-updater traffic are not claimed as controlled by v1.
20. As a Slamcore engineer, I want the updater to remain responsible for starting downloads and validating delivery, so that disruptor stays focused on network conditions.
21. As a Slamcore engineer, I want the utility to log matched devices, unmatched selectors, and effective policies, so that real lab runs are auditable.
22. As a maintainer, I want scenario parsing isolated from invocation scenario parsing, so that the existing test-farm invocation contract remains stable.
23. As a maintainer, I want policy resolution isolated from tc execution, so that matching and fallback semantics can be tested without network privileges.
24. As a maintainer, I want the tc plan represented as an object before command rendering, so that tests assert behavior rather than log text.
25. As a maintainer, I want fake-discovery integration tests, so that scenario-to-plan behavior can be tested without real mDNS or real tc.
26. As a future maintainer, I want the scenario direction to reserve timed and repeating schedules, so that later work can add duration-based disruption without redesigning the schema.
27. As a future maintainer, I want scheduled override fallback semantics documented as defaulting to no impairment, so that future scheduling work has a clear product contract.
28. As a future maintainer, I want dry-run scheduling behavior defined as one-cycle preview when scheduling arrives, so that the future feature remains inspectable without adding flags.

Implementation Decisions

• Add a new Disruptor command under the existing test-farm CLI surface.
• The command shape is test-farm disrupt <scenario> --interface <client-facing-nic>, with an optional --dry-run flag.
• The interface option is mandatory in v1. No automatic NIC detection is included.
• The command blocks while impairment is active and tears down impairment on normal stop.
• Startup clears any root qdisc on the explicit target interface before applying the fresh plan.
• The Disruptor Scenario File is distinct from the existing invocation Scenario File.
• The v1 Disruptor Scenario File has a default impairment policy plus ordered overrides.
• The default impairment applies to every discovered Slamcore Aware device.
• Overrides are evaluated in order. The first matching override determines the device policy.
• An override replaces the entire default policy rather than merging field by field.
• An impairment policy can be none, which means the matched device is classified but intentionally unimpaired.
• Scenario authors do not provide explicit IP addresses. Device selection is based on mDNS-resolved device identities.
• Exact mDNS record details are deferred until the existing device-discovery information from the other project is available.
• Unmatched selectors do not fail the run. They are returned as structured warnings and surfaced to operators.
• v1 impairment fields are delay, loss, and bandwidth limit.
• Later impairment extensions such as jitter, reordering, duplication, corruption, and correlated or bursty loss are not part of v1.
• The policy resolver is a deep module: it accepts discovered devices and a parsed scenario, then returns effective per-device policies plus structured warnings.
• Device discovery is a separate module behind a simple interface, so fake discovery can drive tests while real mDNS support is refined.
• The tc plan renderer is a deep module: it converts effective policies and device IPs into a typed plan object independent of command execution.
• The tc command renderer converts the typed plan object into CLI commands.
• The initial runtime approach should rebuild the whole qdisc tree when applying a plan rather than performing fine-grained branch mutations.
• Dry-run parses, discovers, resolves, and renders the plan, but it does not clear or apply tc state.
• Timed and repeating schedules are reserved for a later implementation. The intended future shape is active duration, repeat interval, and fallback policy.
• Future scheduled overrides fall back to none outside their active window unless the scenario explicitly says otherwise.
• Future dry-run scheduling preview should show one cycle and should not add extra flags in the first scheduling slice.

Testing Decisions

• Scenario parsing should have focused unit tests for valid and invalid schema, unknown fields, supported impairment fields, ordered overrides, and none policies.
• Tests should not assert against captured log output.
• Best-effort unresolved selectors should be tested through structured resolver output.
• Policy resolution should be tested with fake discovered devices, including default policy application, first-match-wins behavior, override replacement, none, and unresolved selectors.
• Plan rendering should be tested by asserting the typed plan object, including classes, filters, destination IP matching, and leaf qdisc choices.
• Command rendering should be tested separately from plan construction.
• Dry-run should be tested behaviorally with fake discovery and a fake executor, asserting that no tc application occurs and that a plan object is produced.
• Integration tests should prefer the flow from Disruptor Scenario File to fake discovery to resolved policy to plan object to rendered commands.
• Real mDNS integration tests are deferred until the expected mDNS records and naming scheme are available.
• Real tc integration tests are deferred until there is a safe interface strategy that does not require accidental mutation of developer networking.
• Lifecycle tests should use fakes to assert that the explicit interface is used, stale state cleanup is requested, apply is requested, and teardown is requested.
• Existing invocation scenario tests should remain valid; disruptor parsing should not weaken the current Scenario File contract.

Out of Scope

• Triggering downloads from the updater.
• Validating successful update delivery.
• Reading updater or Hawkbit verdicts.
• Explicit IP literals in Disruptor Scenario Files.
• Automatic client-facing NIC detection.
• Ingress or device-to-updater impairment.
• Real mDNS implementation details until the available record structure is provided.
• Timed or repeating schedule execution in the first implementation slice.
• Jitter, reorder, duplicate, corrupt, correlated loss, and bursty loss in v1.
• Real tc host-level integration tests in the first slice.
• A long-running daemon or background service model beyond the blocking CLI command.

Further Notes

The Disruptor utility changes the practical test-farm shape: test-farm provides controlled network conditions for real devices, while the external updater owns the download workflow and success criteria. The first slice should prove the static end-to-end path from scenario to effective per-device tc plan before adding time-varying schedules or richer mDNS matching.

The current glossary has been updated with Disruptor and Disruptor Scenario File terminology so later issues and PRDs can distinguish real-device disruption policy from the existing invocation Scenario File.