Automated Regression Test Suite for Validator Consensus Failure Modes

[JamesEjembi]

Problem Statement

The consensus layer has no automated regression suite that simulates specific failure modes (network partition, Byzantine equivocation, message timeout, view-change deadlock). Failures discovered in production require manual reproduction. Build a test harness that injects controlled faults and verifies protocol-level recovery within bounded time.

Technical Bounds

• Must support at least 8 concurrent validators in simulation.
• Fault injection: network partition, message delay, equivocation, timeout, duplicate messages.
• Test scenarios: minimum 95% coverage of known failure modes.
• Run complete suite within 10 minutes on CI (4-core machine).
• Assertions: liveness (block committed within N rounds), safety (no conflicting blocks), recovery (partition healed within M rounds).

Steps

1. Design fault injection API: InjectFault { fault_type, target_nodes, duration, params }.
2. Implement simulation runner with configurable validator sets; use tokio::time::pause() for deterministic timing.
3. Write test scenarios for each failure mode category: partition, equivocation, timeout, reorg, cache invalidation.
4. Implement safety and liveness assertions: assert_no_conflict(blocks), assert_eventually_commit(block, max_rounds).
5. Add CI pipeline step that runs simulation suite; output JUnit XML for test reporting.
6. Add chaos test: randomly inject 5 simultaneous faults, verify protocol recovers within 20 rounds.