feat(bench): soak harness — sustained load + chaos + continuity verdict#46
Merged
Conversation
New benches/src/bin/soak_bench.rs against the 3-broker compose:
- Paced idempotent producer (acks=all) stamps per-partition sequence
numbers; a consumer group (KACRAB_SOAK_CONSUMERS) reads them back.
- Continuity verdict distinguishes the three ways a sequence stream can
misbehave: duplicates (at-least-once re-reads), reordered (a forward
jump whose gap a racing handover stream later refills — two consumers'
poll streams interleave at the tracker, not a loss), and LOSSES (gaps
never refilled — records no consumer ever delivered). Verdict fails on
losses or an undrained tail; unfilled gap ranges are dumped in the
report for forensics.
- Chaos: broker stop/start rotation via the docker CLI
(KACRAB_SOAK_CHAOS_*), consumer bounce (close + recreate), and a
wedge watchdog that detects 90s of zero consumption progress with
backlog, force-bounces all consumers, and counts occurrences.
- Every KACRAB_SOAK_SAMPLE_SECS: counters, window latency percentiles,
RSS, producer/consumer lib metrics -> soak.csv; chaos timeline ->
events.log; final report.md + exit code carries the verdict.
docker-compose.cluster.yml host ports are now parameterized
(KAFKA{1,2,3}_HOST_PORT, defaults unchanged) so the cluster can run
beside a native broker owning 9092.
Shakedown findings (compressed chaos: kill every 75s, bounce every
100s): producer delivered 480k/480k acked through repeated broker
kills; consumer group once wedged permanently (every poll
Wire(Timeout)) after a coordinator-broker kill + member bounce, and a
later run left 239 sequences undelivered to any consumer — both queued
for investigation with the overnight run's data.
New advisory published today against crossbeam-epoch 0.9.18 (invalid pointer dereference in fmt::Display of Atomic/Shared null pointers), reached transitively via criterion -> rayon. Not triggered by anything in this workspace, but it fails the cargo-deny advisories gate on every CI run from now on — bumping the lockfile per the advisory's fix.
SOAK-REPORT.md carries the 2026-07-07 overnight run: 4h17m healthy phase (15.2M records through ~25 rotating broker kills, 0.002% loud errors, flat RSS), the OOM-triggered cascade, and the four queued findings (F1 consumer-group wedge, F2 unfilled gaps / at-least-once suspect, F3 delivery timeout not enforced under prolonged outage, F4 suspected connection leak across consumer restarts).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A soak/chaos harness (
benches/src/bin/soak_bench.rs) answering the README's remaining production-readiness question — behavior under sustained load — plus the first overnight run's results.docker-compose.cluster.ymlhost ports are now parameterized (KAFKA{1,2,3}_HOST_PORT, defaults unchanged) so the cluster can run beside a native broker owning 9092.Harness: paced idempotent producer stamps per-partition sequences; a consumer group reads them back through a continuity tracker with gap-refill semantics — duplicates (at-least-once re-reads), reordered (forward gap later refilled by a racing handover stream), and losses (never refilled) are separated, because a naive tracker false-positives on two members' poll streams interleaving at handover. Chaos: rotating broker
docker stop/start, consumer close/recreate cycles, and a watchdog that detects 90 s of zero progress and force-bounces the group. 10 s samples →soak.csv; verdict →report.md+ exit code.First overnight run (4 h 46 m, 2026-07-07)
Healthy phase — 4 h 17 m, ~25 rotating broker kills:
Cascade phase: kafka1 OOM-killed by the undersized VM at 4 h 17 m (infra, not chaos) — staging an unplanned prolonged-outage drill that the client failed. Findings queued:
Wire(Timeout)~45 s cycle) after coordinator-broker kill + member change; client recreation does not always heal itdelivery.timeout.ms=120 s; no produce resume after full cluster restorationFull report with methodology and evidence pointers:
SOAK-REPORT.mdat the repo root.Validation
Baseline (no chaos): PASS, losses 0. Compressed-chaos smokes (kill/75 s + bounce/100 s): reproduced F1 (intermittent) and F2 (239 unfilled gaps with a fully drained tail). Workspace clippy
--all-targetsclean, nightly fmt applied; CI does not run the harness.