diff --git a/docs/specs/blocksync/congestion_control.md b/docs/specs/blocksync/congestion_control.md new file mode 100644 index 00000000000..731d71ab1fb --- /dev/null +++ b/docs/specs/blocksync/congestion_control.md @@ -0,0 +1,183 @@ +# Block-Sync BBR Congestion Control — Specification + +## Overview + +The controller runs **per peer** and is **byte-denominated**: it measures how fast a +peer delivers block bodies (bytes/second) and how quickly it answers (round-trip +time), and from those it sizes a per-peer **in-flight window** — the amount of +outstanding body bytes we allow that peer at any moment. One request fetches one +block body, so the in-flight _request count_ is simply `window ÷ body size`. + +It pursues three goals: + +1. **Responsiveness** — when we must re-request ("rescue") a block from a different + peer because its current carrier is slow, that peer can serve it almost + immediately. This only holds if the peer's queue is shallow: a freshly issued + request sits near the front instead of behind a long backlog. +2. **Throughput** — every peer's link stays full. +3. **Bounded memory** — total outstanding data is capped no matter how peers behave. + +The key insight is that responsiveness and throughput are **not** in tension at the +right window size. The smallest window that still keeps a link full is exactly **one +BDP** (byte delivery rate × base round-trip). At that point the pipe is saturated +(full throughput) and almost nothing is queued (full responsiveness); any larger +window adds only queue — latency — for zero extra throughput. + +So the controller: + +- **maximizes throughput** by growing the window toward `BDP × gain`, probing just + above the measured pipe so it finds and fills available bandwidth; +- **maximizes responsiveness** by never growing it further — periodically draining + the queue (ProbeRtt) to re-confirm the true base round-trip, and trimming the + window the instant a standing queue starts to form (the delay gradient); and +- **bounds memory** with a global in-flight byte budget and per-peer caps, kept + independent of the window logic. + +## Glossary + +The names below are the plain-language terms used throughout this document; the +BBR/code identifier follows in parentheses. + +- **In-flight window** (`cwnd`, "congestion window") — the maximum body bytes a single + peer may have outstanding to us at once. A larger window means more parallel + requests to that peer. +- **Base round-trip** (`RTprop`) — the _minimum_ recent round-trip time to a peer: + how long send-then-receive takes when nothing is queued. It is a latency, **not** a + window size, but it sizes the window (see BDP). "RT" = round trip. +- **BDR — byte delivery rate** (`BtlBw`, "bottleneck bandwidth") — the _maximum_ + recent rate, in bytes/second, at which a peer delivers bodies: the speed of the + bottleneck link to that peer. +- **BDP — bandwidth-delay product** = `BDR × base round-trip` — the amount of + in-flight data that exactly fills the pipe to a peer. Hold this much outstanding and + the link is busy with no standing queue; hold less and it idles, hold more and the + excess only sits in a queue. This is the window's natural target. +- **Gain** — a multiplier above 1 applied to the BDP so the window probes slightly + past the measured capacity, letting it discover newly available bandwidth. +- **Measurement horizon** — the recent time span (10 s) over which we take the min + (base round-trip) and max (BDR). The `*_window` config knobs set these. +- **ProbeBw / ProbeRtt** — the two operating modes. ProbeBw is steady state (window ≈ + BDP). ProbeRtt briefly drains the queue to re-measure an honest base round-trip. +- **Delay gradient** — the signal that a queue is forming: the recent round-trip has + risen above the base round-trip by more than a set ratio. It trims the window down. +- **Floor** — the lowest block height we still need; the chain cannot commit past it. + A "floor request" fetches it, and if its carrier is slow the height is _rescued_ — + re-requested from a faster peer. + +## Measured signals (per peer) + +- **Base round-trip** — windowed _min_ of the raw request round-trip + (`bbr_rtprop_window`, 10 s). The propagation floor; never collapses to zero. +- **BDR** — windowed _max_ of the per-response delivery rate in bytes/s + (`bbr_delivery_rate_window`, 10 s). +- **BDP** = `BDR × base round-trip` (bytes). Window target = + `max(min window, BDP × gain)` (`gain = 200%`). +- **Delay gradient** — a smoothed round-trip compared against a size-aware healthy + baseline (`base round-trip + bytes/BDR`), used to detect a building queue. + +In-flight admission compares a peer's **reserved body bytes** against its window; the +in-flight _request count_ falls out as `window ÷ body size`. + +## Control law — MUST + +- A peer's outstanding reserved bytes MUST NOT exceed its window (plus the bounded + floor bypass below). +- **ProbeBw** (steady state): the window tracks `BDP × gain`, clamped above by the + delay-gradient ceiling and below by the minimum window. +- **ProbeRtt**: every `bbr_probe_rtt_interval` (10 s) the window MUST drain to the + minimum and hold for `bbr_probe_rtt_duration` (200 ms) so one uncontended request + yields a fresh base round-trip. Without this, a sustained queue inflates the + base-round-trip min and the window never collapses for a genuinely slow peer. + +## Control law — SHOULD + +- On a real request timeout the window SHOULD take one multiplicative dip (`×0.85`, + bounded by the minimum) and pull the delay ceiling down to the dipped value. A + timeout is merely congestion evidence, it should not trigger backoff. +- When the smoothed round-trip exceeds `base round-trip × delay_gradient` (150%) the + queue is building, so the window ceiling SHOULD ratchet down (`×0.9`); when there is + headroom it SHOULD relax up (~12% per response, saturating) so a cleared queue + re-probes for bandwidth. + +--- + +## Edge cases and security bounds + +### Problem: a fast peer's base round-trip can collapse toward zero and void the BDP + +On a fast link, subtracting a body's transmission time from its round-trip leaves +almost nothing, which would zero `BDR × base round-trip` and pin the window at its +floor. + +- The BDP MUST be sized from the **raw** round-trip minimum, never a + transmission-stripped residual (the residual is used only by the delay gate, so a + big body's honest transfer time is not mistaken for a queue). +- The window MUST be floored at `bbr_min_cwnd_bytes` (4 MiB, the primary tuning + lever) so a near-zero BDP still keeps the pipe primed. + +### Problem: a burst of buffered bodies inflates the BDR + +Many bodies arriving in one tick would read as an impossibly high rate. + +- The delivery-rate interval MUST be floored at the previous base round-trip, so a + single-tick burst cannot inflate the BDR max. + +### Problem: a slow peer holding the contiguous floor stalls the whole sync + +The lowest missing height gates commit; one slow carrier must not pin it. This is the +responsiveness goal made concrete — a shallow per-peer window is what lets a rescue +land fast. + +- A **floor** request MUST carry a short fixed leash (`floor_rescue_timeout`, 2 s); + on expiry its height MUST be returned to the queue and the peer retry-avoided — + rescued to a faster carrier, **not** disconnected (record-only). +- The floor MAY borrow up to `floor_bypass_slots` (2) representative bodies beyond a + saturated window so it is fetched even when every peer is at its window; the borrow + MUST stay within the advertised request-count cap and reserve real budget. +- **Above-floor** speculation SHOULD use a patient, size-aware deadline + (`request_timeout + estimated_bytes ÷ BDR`) and MUST NOT gate the floor. + +### Problem: unbounded memory under attacker-controlled bodies or stalls + +- Total in-flight + reorder + applying bytes MUST be bounded by the global + `max_inflight_block_bytes` budget (6 GiB). Concurrent reservations MUST NOT + over-commit it. +- Every per-request size estimate MUST be clamped to `[floor, MAX_BLOCK_BYTES]`; + untrusted header size hints MUST NOT exceed the per-block worst case. +- The advertised request-count cap (≤ `MAX_BS_INFLIGHT_REQUESTS = 32 768`) MUST bind + even when byte headroom remains, so a peer serving tiny bodies cannot be issued an + unbounded request count. +- The reorder look-ahead and the serving-request heap MUST be bounded. + +### Problem: an unbounded wait wedges a peer + +- Every outbound request MUST have a network deadline — the **only** sanctioned timer + (and itself an event). The above-floor deadline assumes a minimum delivery rate when + a peer's measured BDR is still near zero, so the deadline is finite (~16 s worst + case), never unbounded. + +### Numeric safety — MUST + +- Arithmetic over external/untrusted values MUST saturate or be checked — never wrap + or panic. +- Rates and BDP products MUST be clamped to finite, non-negative values before they + size a window. + +--- + +## Defaults + +| Knob | Default | Meaning | +| --- | --- | --- | +| `bbr_cwnd_unit` | `bytes` | window budgets header-hinted body bytes | +| `bbr_cwnd_gain_percent` | 200 | window target = 2 × BDP | +| `bbr_min_cwnd_bytes` | 4 MiB | window floor / cold-start (primary lever) | +| `bbr_min_cwnd` | 4 | window floor in blocks (A/B baseline unit) | +| `bbr_rtprop_window` | 10 s | base-round-trip measurement horizon | +| `bbr_delivery_rate_window` | 10 s | BDR measurement horizon | +| `bbr_probe_rtt_interval` | 10 s | ProbeRtt cadence | +| `bbr_probe_rtt_duration` | 200 ms | drained hold to refresh base round-trip | +| `bbr_delay_gradient_percent` | 150 | queue-building round-trip ratio | +| `floor_rescue_timeout` | 2 s | floor leash before rescue | +| `floor_bypass_slots` | 2 | floor borrow beyond the window | +| `max_inflight_block_bytes` | 6 GiB | global in-flight byte ceiling | +| timeout dip / delay-cap down / EWMA α | ×0.85 / ×0.9 / 0.25 | (constants) | diff --git a/zebra-network/src/zakura/block_sync/admission.rs b/zebra-network/src/zakura/block_sync/admission.rs index b8e4d2032d9..aa96485c722 100644 --- a/zebra-network/src/zakura/block_sync/admission.rs +++ b/zebra-network/src/zakura/block_sync/admission.rs @@ -1,7 +1,16 @@ +use std::time::{Duration, Instant}; + use zebra_chain::block; use super::{config::ZakuraBlockSyncConfig, state::next_height}; +/// Delivery rate assumed when sizing an above-floor deadline for a peer whose +/// measured BtlBw is still near zero, so the patience window is bounded rather than +/// unbounded. A worst-case `MAX_BLOCK_BYTES` body at this rate transfers in ~8 s, so +/// with the `request_timeout` base the above-floor deadline tops out near 16 s — the +/// "a block every ~16 s is fine" tolerance the directive sets for speculative work. +const ABOVE_FLOOR_DEADLINE_MIN_BYTES_PER_SEC: u64 = 256 * 1024; + /// Pure inputs for deciding whether a block request may consume budget. #[derive(Copy, Clone, Debug, Eq, PartialEq)] pub(super) struct AdmissionSnapshot { @@ -55,6 +64,39 @@ pub(super) fn request_priority( /// /// Returns `None` when no bytes can be admitted, or when an above-floor request would /// exceed the lookahead limits. + +/// The per-request network deadline (the one sanctioned timer), set by priority: +/// +/// - **Floor**: a short fixed leash. On expiry the lowest missing height is rescued +/// to a faster carrier (returned to the queue + the peer retry-avoided), so the +/// contiguous floor never waits on a slow peer — and the peer is *not* disconnected. +/// - **Above-floor**: the base `request_timeout` plus the size-expected transfer time +/// (`estimated_bytes / BtlBw`), so a legitimately slow large-body fetch runs to +/// completion. These deadlines never gate the floor, so they can afford to be +/// patient; `btlbw_bytes_per_sec` is the peer's measured rate (`None` cold-start), +/// floored at [`ABOVE_FLOOR_DEADLINE_MIN_BYTES_PER_SEC`]. +pub(super) fn request_deadline( + priority: RequestPriority, + queued_at: Instant, + request_timeout: Duration, + floor_rescue_timeout: Duration, + estimated_bytes: u64, + btlbw_bytes_per_sec: Option, +) -> Instant { + match priority { + RequestPriority::Floor => queued_at + floor_rescue_timeout, + RequestPriority::AboveFloor => { + let rate = btlbw_bytes_per_sec + .unwrap_or(0) + .max(ABOVE_FLOOR_DEADLINE_MIN_BYTES_PER_SEC); + // One body per request, so `estimated_bytes / rate` is at most + // `MAX_BLOCK_BYTES / rate` (~8 s): finite and non-negative. + let transfer = Duration::from_secs_f64(estimated_bytes as f64 / rate as f64); + queued_at + request_timeout + transfer + } + } +} + pub(super) fn admission_decision( config: &ZakuraBlockSyncConfig, snapshot: AdmissionSnapshot, @@ -98,3 +140,69 @@ pub(super) fn admission_decision( max_request_bytes, }) } + +#[cfg(test)] +mod tests { + use super::*; + + const TIMEOUT: Duration = Duration::from_secs(8); + const RESCUE: Duration = Duration::from_secs(2); + + #[test] + fn floor_request_uses_the_short_rescue_leash() { + let now = Instant::now(); + let deadline = request_deadline( + RequestPriority::Floor, + now, + TIMEOUT, + RESCUE, + 2_000_000, + None, + ); + // The floor is rescued on the fixed leash regardless of size or measured rate. + assert_eq!(deadline, now + RESCUE); + } + + #[test] + fn above_floor_deadline_grows_with_body_size() { + let now = Instant::now(); + // No measured rate: the min-rate floor (256 KiB/s) sizes the transfer term, so a + // 256 KiB body adds ~1 s and a 2 MiB body adds ~8 s on top of the base timeout. + let small = request_deadline( + RequestPriority::AboveFloor, + now, + TIMEOUT, + RESCUE, + 256 * 1024, + None, + ); + let large = request_deadline( + RequestPriority::AboveFloor, + now, + TIMEOUT, + RESCUE, + 2 * 1024 * 1024, + None, + ); + assert_eq!(small, now + TIMEOUT + Duration::from_secs(1)); + assert_eq!(large, now + TIMEOUT + Duration::from_secs(8)); + assert!(large > small); + } + + #[test] + fn above_floor_deadline_shrinks_as_measured_rate_rises() { + let now = Instant::now(); + // A fast peer transfers the body quickly, so its above-floor deadline collapses + // toward the base timeout — the size term is negligible at high BtlBw. + let fast = request_deadline( + RequestPriority::AboveFloor, + now, + TIMEOUT, + RESCUE, + 2 * 1024 * 1024, + Some(64 * 1024 * 1024), + ); + assert!(fast > now + TIMEOUT); + assert!(fast < now + TIMEOUT + Duration::from_millis(100)); + } +} diff --git a/zebra-network/src/zakura/block_sync/bbr.rs b/zebra-network/src/zakura/block_sync/bbr.rs new file mode 100644 index 00000000000..ddbec8b40ab --- /dev/null +++ b/zebra-network/src/zakura/block_sync/bbr.rs @@ -0,0 +1,1043 @@ +use std::time::{Duration, Instant}; + +use super::{ + config::{CwndUnit, ZakuraBlockSyncConfig}, + state::DeliverySnapshot, +}; + +/// BBR-lite multiplicative cwnd dip applied on a real request timeout, +/// bounded below by `bbr_min_cwnd`. +const BBR_TIMEOUT_DIP: f64 = 0.85; +/// EWMA weight for the smoothed request round-trip the delay-gradient compares against +/// RTprop (higher = more responsive, noisier). +const BBR_DELAY_EWMA_ALPHA: f64 = 0.25; +/// Multiplicative shrink applied to the delay-gradient ceiling on each delivery whose +/// smoothed round-trip exceeds `RTprop × delay_gradient` (queue building). +const BBR_DELAY_CAP_DOWN: f64 = 0.9; + +/// A time-windowed set of `f64` samples supporting `min` (RTprop) and `max` (BtlBw) +/// filters — the BBR-lite estimators. Samples older than `horizon` are pruned on +/// insert; the windows are small (seconds of per-request samples) so the linear +/// scan is cheap and runs once per completed request. +#[derive(Clone, Debug)] +struct WindowedSamples { + horizon: Duration, + samples: Vec<(Instant, f64)>, +} + +impl WindowedSamples { + fn new(horizon: Duration) -> Self { + Self { + horizon, + samples: Vec::new(), + } + } + + fn observe(&mut self, now: Instant, value: f64) { + self.samples.push((now, value)); + if let Some(cutoff) = now.checked_sub(self.horizon) { + self.samples.retain(|(at, _)| *at >= cutoff); + } + } + + fn min(&self) -> Option { + self.samples + .iter() + .map(|(_, value)| *value) + .reduce(f64::min) + } + + fn max(&self) -> Option { + self.samples + .iter() + .map(|(_, value)| *value) + .reduce(f64::max) + } +} + +/// Per-peer BBR-lite control parameters extracted from config (Copy, lock-free). +#[derive(Copy, Clone, Debug)] +struct BbrParams { + /// Unit the cwnd/BtlBw/`delivered` are denominated in. `Blocks` keeps the + /// request-counting controller (the A/B baseline); `Bytes` makes the controller + /// reason in header-hinted body bytes so the in-flight request count falls out as + /// `cwnd_bytes / advertised_block_size`. + unit: CwndUnit, + cwnd_gain: f64, + /// Minimum / cold-start cwnd, in the active unit (`bbr_min_cwnd` blocks or + /// `bbr_min_cwnd_bytes` bytes). + min_cwnd: usize, + startup_cwnd: usize, + rtprop_window: Duration, + delivery_rate_window: Duration, + /// How long between ProbeRTT drains (the cadence at which RTprop is refreshed). + probe_rtt_interval: Duration, + /// How long to hold the cwnd at `min_cwnd` once the queue has drained, so at + /// least one uncontended request completes and yields a clean RTprop sample. + probe_rtt_duration: Duration, + /// Smoothed-RTT / RTprop ratio above which the queue is judged to be building and + /// the delay-gradient ceiling ratchets the cwnd down (e.g. 1.5 = shrink once the + /// recent round-trip runs 50% over the uncontended minimum). + delay_gradient: f64, +} + +impl BbrParams { + fn from_config(config: &ZakuraBlockSyncConfig) -> Self { + let (min_cwnd, startup_cwnd) = match config.bbr_cwnd_unit { + CwndUnit::Blocks => { + let min = usize::try_from(config.bbr_min_cwnd).unwrap_or(1).max(1); + // Cold start opens at the configured initial window until the first + // BDP sample. + let startup = usize::try_from(config.initial_inflight_requests) + .unwrap_or(min) + .max(min); + (min, startup) + } + CwndUnit::Bytes => { + // Byte denomination: the floor (and cold-start window) is the + // configured minimum byte cwnd. The BDP estimate takes over once the + // first delivery sample arrives; until then `bbr_min_cwnd_bytes` + // primes the pipe with a few bodies' worth of in-flight budget. + let min = usize::try_from(config.bbr_min_cwnd_bytes) + .unwrap_or(usize::MAX) + .max(1); + (min, min) + } + }; + Self { + unit: config.bbr_cwnd_unit, + cwnd_gain: f64::from(config.bbr_cwnd_gain_percent) / 100.0, + min_cwnd, + startup_cwnd, + rtprop_window: config.bbr_rtprop_window, + delivery_rate_window: config.bbr_delivery_rate_window, + probe_rtt_interval: config.bbr_probe_rtt_interval, + probe_rtt_duration: config.bbr_probe_rtt_duration, + delay_gradient: f64::from(config.bbr_delay_gradient_percent.max(100)) / 100.0, + } + } +} + +/// BBR-lite control phase. `ProbeBw` is the steady state (cwnd tracks BDP × gain); +/// `ProbeRtt` periodically drains the queue to `min_cwnd` to take a fresh, uncontended +/// RTprop sample. Without ProbeRtt, a peer's RTprop min-filter stays inflated under a +/// sustained queue (the round-trip we measure is queue + serve + RTT), so the cwnd never +/// collapses for a genuinely slow peer. +#[derive(Copy, Clone, Debug, Eq, PartialEq)] +enum BbrPhase { + ProbeBw, + ProbeRtt, +} + +impl BbrPhase { + /// Numeric code for the JSONL trace (0 = ProbeBw, 1 = ProbeRtt). + fn trace_code(self) -> u64 { + match self { + BbrPhase::ProbeBw => 0, + BbrPhase::ProbeRtt => 1, + } + } +} + +fn rounded_usize(value: f64, fallback: usize) -> usize { + // BBR rates, windows, and gains are non-negative in normal operation. Keep a + // fallback for NaN/inf or defensive underflow before casting. + if value.is_finite() && value >= 0.0 { + value.round() as usize + } else { + fallback + } +} + +fn secs_to_ms(secs: f64) -> u64 { + // Round-trip samples are non-negative and finite for real requests. + (secs * 1000.0).round() as u64 +} + +/// Per-peer BBR-lite estimators: an RTprop min-filter over request round-trips, a +/// BtlBw max-filter over per-ack delivery rate, and a delivered-block counter. The +/// owning routine samples these lock-free on each completed request. Stage 1 measures +/// and traces them; the control law (`available_slots`) consumes them in a later stage. +#[derive(Clone, Debug)] +pub(super) struct BbrState { + params: BbrParams, + /// Windowed-min of the **raw request round-trip** (seconds) — the BDP's RTprop term + /// (`bdp = BtlBw × RTprop`) under both units: the genuine fastest observed round trip, + /// which never collapses to zero. (The earlier byte-unit model fed this the + /// *size-residual* `elapsed − bytes/BtlBw`; on a high-BtlBw carrier the fastest + /// delivery's residual is ≈0, which zeroed the BDP and pinned the cwnd at the floor. + /// The size-residual now lives in [`rtprop_residual_secs`](Self::rtprop_residual_secs), + /// used only by the size-aware delay gate.) + rtprop_secs: WindowedSamples, + /// Windowed-min of the **size-residual** round-trip (`elapsed − bytes/BtlBw` under + /// `Bytes`; the raw round-trip under `Blocks`) — the transmission-stripped propagation + /// latency. Used **only** as the delay gate's healthy-round-trip base, so a big block's + /// honest transfer time is not mistaken for a standing queue. It never feeds the BDP, + /// which must reflect real in-flight depth rather than a residual that collapses to ~0 + /// on a fast carrier. + rtprop_residual_secs: WindowedSamples, + /// Max-filter over per-ack delivery rate, in **units per second** (blocks/s under + /// [`CwndUnit::Blocks`], bytes/s under [`CwndUnit::Bytes`]). The byte denomination + /// makes `BtlBw × RTprop` a true bandwidth-delay product over heterogeneous body + /// sizes; the block denomination is the A/B baseline. + btlbw_per_sec: WindowedSamples, + /// Cumulative delivered amount in the active unit (blocks or bytes), used as the + /// per-ack delivery-rate numerator via [`DeliverySnapshot`]. + delivered: u64, + delivered_at: Option, + /// Effective cwnd in blocks currently applied by `available_slots`: the + /// BDP-derived target once measured, the startup window before that, dipped on + /// timeouts. Never below `min_cwnd`. Ignored while in `ProbeRtt` (which forces + /// `min_cwnd`) but preserved so the cwnd restores on exit. + cwnd_cap: usize, + /// Current control phase. + phase: BbrPhase, + /// When the last ProbeRtt completed (or the first delivery, to anchor the first + /// probe one interval out). `None` until the first delivery is recorded. + last_probe_rtt_at: Option, + /// Set the moment the queue first drains to `min_cwnd` during a ProbeRtt; the + /// `probe_rtt_duration` hold timer runs from here. + probe_rtt_drained_at: Option, + /// EWMA of the request round-trip, compared against RTprop by the delay-gradient. + smoothed_elapsed_secs: Option, + /// Delay-gradient ceiling on the effective cwnd. Starts unbounded (`usize::MAX`) so + /// it never limits an uncongested peer; ratchets down toward the true operating + /// point whenever the smoothed round-trip rises above `RTprop × delay_gradient`, and + /// relaxes back up when the queue clears. Guards against a `BtlBw × RTprop` BDP that + /// overshoots the sustainable rate (max-rate and min-RTT can come from different + /// samples under variable queueing), which would otherwise inflate the cwnd. + delay_cap: usize, +} + +impl BbrState { + pub(super) fn new(config: &ZakuraBlockSyncConfig) -> Self { + let params = BbrParams::from_config(config); + Self { + rtprop_secs: WindowedSamples::new(params.rtprop_window), + rtprop_residual_secs: WindowedSamples::new(params.rtprop_window), + btlbw_per_sec: WindowedSamples::new(params.delivery_rate_window), + delivered: 0, + delivered_at: None, + cwnd_cap: params.startup_cwnd, + phase: BbrPhase::ProbeBw, + last_probe_rtt_at: None, + probe_rtt_drained_at: None, + smoothed_elapsed_secs: None, + delay_cap: usize::MAX, + params, + } + } + + pub(super) fn delivery_snapshot(&self, now: Instant) -> DeliverySnapshot { + DeliverySnapshot { + delivered: self.delivered, + delivered_at: self.delivered_at.unwrap_or(now), + } + } + + /// Record a completed request: `elapsed` from send to the final body, `blocks` in + /// it, and `inflight` = the work still outstanding to this peer *after* this + /// completion, **denominated in the cwnd's unit** (request count under `Blocks`, + /// reserved body bytes under `Bytes`) so the ProbeRtt drain check can compare it + /// against `min_cwnd` consistently. The RTprop sample is the request round-trip. + /// The BtlBw sample is measured over the request's pipe interval + /// (`delivered_delta / elapsed_since_snapshot`), so one-block responses can still + /// observe concurrent completions while the request was in flight. The interval is + /// floored at the previous RTprop so a burst of buffered bodies arriving within one + /// tick cannot inflate the bandwidth estimate. Re-derives the applied cwnd from the + /// fresh BDP estimate, then advances the ProbeBw/ProbeRtt phase machine. + pub(super) fn record_delivery( + &mut self, + now: Instant, + elapsed: Duration, + blocks: u32, + delivered_bytes: u64, + inflight: u64, + snapshot: DeliverySnapshot, + ) { + let rtt_secs = elapsed.as_secs_f64(); + // Floor the delivery-rate interval at the *previous* RTprop min (captured + // before this sample is observed) so a burst of buffered bodies arriving within + // one tick cannot inflate the bandwidth estimate. + let rate_floor = self.rtprop_secs.min().unwrap_or(rtt_secs).max(1e-4); + + // Accumulate the delivered amount in the active unit and push a per-ack rate + // sample into the BtlBw max-filter (blocks/s under `Blocks`, bytes/s under + // `Bytes`). + let delivered_amount = match self.params.unit { + CwndUnit::Blocks => u64::from(blocks), + CwndUnit::Bytes => delivered_bytes, + }; + let delivered_after = self.delivered.saturating_add(delivered_amount); + let delivered_delta = delivered_after.saturating_sub(snapshot.delivered).max(1); + let interval = now.saturating_duration_since(snapshot.delivered_at); + // `delivered_delta` is a count/byte total over a short sampling window; + // converting it to `f64` is exact for the operating ranges this controller sees. + let rate = delivered_delta as f64 / interval.as_secs_f64().max(rate_floor); + self.btlbw_per_sec.observe(now, rate); + self.delivered = delivered_after; + self.delivered_at = Some(now); + + // Observe the BDP's RTprop sample: the **raw** round trip under both units. Its + // windowed min ≈ the base round trip of the fastest deliveries, which is the real + // in-flight depth the BDP needs. Feeding the BDP the size residual instead would + // collapse it to ~0 on a high-BtlBw carrier (the fastest delivery's residual + // `elapsed − bytes/BtlBw` ≈ 0), pinning the cwnd at the floor. + self.rtprop_secs.observe(now, rtt_secs); + // Observe the size-residual separately, for the delay gate only: under `Bytes` it + // strips the body's transmission time so a big block's honest transfer is not read + // as a standing queue; under `Blocks` it is the raw round trip (A/B baseline). + let residual_sample = match self.params.unit { + CwndUnit::Blocks => rtt_secs, + CwndUnit::Bytes => self.size_residual_rtprop(rtt_secs, delivered_bytes), + }; + self.rtprop_residual_secs.observe(now, residual_sample); + + if let Some(target) = self.cwnd_target() { + self.cwnd_cap = target; + } + // Delay-gradient runs in ProbeBw only: the drained round-trips ProbeRtt produces + // are artificially short and would spuriously relax the ceiling. `phase` here is + // still the pre-`advance_phase` value, so a tick that flips into ProbeRtt this + // call last updated the ceiling under genuine ProbeBw conditions. + if self.phase == BbrPhase::ProbeBw { + self.update_delay_cap(rtt_secs, delivered_bytes); + } + self.advance_phase(now, inflight); + } + + /// Size-residual RTprop sample (`Bytes` unit): subtract the body's transmission + /// time at the bottleneck rate from the round trip, leaving the fixed-latency + /// component. Falls back to the raw round trip before any rate is known, and is + /// clamped to `[ε, elapsed]` (the residual can never exceed the time elapsed, and a + /// tiny positive floor keeps the byte-BDP well-defined). + fn size_residual_rtprop(&self, rtt_secs: f64, delivered_bytes: u64) -> f64 { + let btlbw = self.btlbw_per_sec.max().unwrap_or(0.0); + let residual = if btlbw > 0.0 { + // `delivered_bytes as f64` is exact for real body sizes. + rtt_secs - delivered_bytes as f64 / btlbw + } else { + rtt_secs + }; + residual.clamp(1e-4, rtt_secs.max(1e-4)) + } + + /// Update the delay-gradient ceiling from this delivery's round-trip. When the + /// smoothed round-trip rises above `RTprop × delay_gradient` the queue is building, + /// so ratchet the ceiling down from the current operating cwnd; otherwise relax it + /// back up so a cleared queue lets the cwnd re-probe for bandwidth. + fn update_delay_cap(&mut self, rtt_secs: f64, delivered_bytes: u64) { + let smoothed = match self.smoothed_elapsed_secs { + Some(prev) => prev * (1.0 - BBR_DELAY_EWMA_ALPHA) + rtt_secs * BBR_DELAY_EWMA_ALPHA, + None => rtt_secs, + }; + self.smoothed_elapsed_secs = Some(smoothed); + // The delay gate's base is the *residual* RTprop (transmission stripped), not the + // raw round trip the BDP uses: the size-aware `expected` below adds the body's + // transmission back, so basing it on the raw round trip would double-count it. + let rtprop = self + .rtprop_residual_secs + .min() + .unwrap_or(rtt_secs) + .max(1e-4); + // The expected round trip for a healthy (unqueued) delivery. Under `Bytes` it is + // size-aware — `RTprop + transmission time` — so a big block's honest transfer + // time is not mistaken for a standing queue; under `Blocks` it is just RTprop + // (the A/B baseline). + let expected = match self.params.unit { + CwndUnit::Blocks => rtprop, + CwndUnit::Bytes => { + let btlbw = self.btlbw_per_sec.max().unwrap_or(0.0); + let transmit = if btlbw > 0.0 { + delivered_bytes as f64 / btlbw + } else { + 0.0 + }; + rtprop + transmit + } + }; + if smoothed > expected * self.params.delay_gradient { + // Queue building: shrink the ceiling relative to the current operating cwnd. + let operating = self.cwnd_cap.min(self.delay_cap).max(self.params.min_cwnd); + // `operating` is a non-negative cwnd; f64 precision is enough for tuning math. + let operating_f64 = operating as f64; + let shrunk = rounded_usize(operating_f64 * BBR_DELAY_CAP_DOWN, self.params.min_cwnd); + self.delay_cap = shrunk.max(self.params.min_cwnd); + } else { + // Headroom: relax the ceiling up (~12%/delivery), saturating so an + // uncongested peer's ceiling stays effectively unbounded. + let grow = (self.delay_cap / 8).max(1); + self.delay_cap = self.delay_cap.saturating_add(grow); + } + } + + /// Drive the ProbeBw/ProbeRtt cycle off completed deliveries (the only event that + /// carries both a fresh timestamp and the current inflight measure). `inflight` is in + /// the cwnd's unit (request count under `Blocks`, reserved bytes under `Bytes`) so it + /// is comparable to `min_cwnd`. ProbeRtt forces the cwnd to `min_cwnd`, which drains + /// the queue; once drained, it holds for `probe_rtt_duration` so an uncontended + /// request completes and refreshes RTprop. + fn advance_phase(&mut self, now: Instant, inflight: u64) { + // Anchor the first probe one interval after the first delivery. + let anchor = *self.last_probe_rtt_at.get_or_insert(now); + match self.phase { + BbrPhase::ProbeBw => { + if now.saturating_duration_since(anchor) >= self.params.probe_rtt_interval { + self.phase = BbrPhase::ProbeRtt; + self.probe_rtt_drained_at = None; + } + } + BbrPhase::ProbeRtt => { + // Start the hold timer the moment the queue first reaches the floor. + // `inflight` and `min_cwnd` are in the same unit; widen `min_cwnd` + // (`usize`) to `u64` for the comparison (lossless on supported targets). + if self.probe_rtt_drained_at.is_none() && inflight <= self.params.min_cwnd as u64 { + self.probe_rtt_drained_at = Some(now); + } + let Some(drained_at) = self.probe_rtt_drained_at else { + return; + }; + if now.saturating_duration_since(drained_at) < self.params.probe_rtt_duration { + return; + } + + // Exit: a clean RTprop sample has been taken at low queue depth. + self.phase = BbrPhase::ProbeBw; + self.last_probe_rtt_at = Some(now); + self.probe_rtt_drained_at = None; + if let Some(target) = self.cwnd_target() { + self.cwnd_cap = target; + } + } + } + } + + /// The effective cwnd in blocks currently applied (never below `min_cwnd`). During + /// ProbeRtt the cwnd is pinned to `min_cwnd` to drain the queue; in ProbeBw it is the + /// BDP-derived cwnd capped by the delay-gradient ceiling. + pub(super) fn effective_cwnd(&self) -> usize { + match self.phase { + BbrPhase::ProbeRtt => self.params.min_cwnd, + BbrPhase::ProbeBw => self.cwnd_cap.min(self.delay_cap).max(self.params.min_cwnd), + } + } + + /// Apply one multiplicative dip on a real timeout (BBR-style), bounded by the + /// minimum cwnd. Suppressed during ProbeRtt, + /// where the cwnd is already pinned to `min_cwnd` and timeouts are an expected + /// consequence of the drain, not congestion signal. A timeout is strong congestion + /// evidence, so it also ratchets the delay-gradient ceiling down to the dipped cwnd. + pub(super) fn dip_on_timeout(&mut self) { + if self.phase == BbrPhase::ProbeRtt { + return; + } + // `cwnd_cap` is a non-negative cwnd; f64 precision is enough for tuning math. + let cwnd_cap = self.cwnd_cap as f64; + let dipped = rounded_usize(cwnd_cap * BBR_TIMEOUT_DIP, self.params.min_cwnd); + self.cwnd_cap = dipped.max(self.params.min_cwnd); + self.delay_cap = self.delay_cap.min(self.cwnd_cap); + } + + /// Bandwidth-delay product in the active unit: BtlBw (units/s) × RTprop (s) — blocks + /// under `Blocks`, bytes under `Bytes`. `None` until at least one delivery sample + /// exists (cold start). + fn bdp(&self) -> Option { + match (self.btlbw_per_sec.max(), self.rtprop_secs.min()) { + (Some(rate), Some(rtprop)) => Some(rate * rtprop), + _ => None, + } + } + + /// Target cwnd in the active unit = `max(min_cwnd, BDP × gain)`. `None` until the + /// first delivery sample exists, so the cwnd stays at the cold-start value until then. + fn cwnd_target(&self) -> Option { + let bdp = self.bdp()?; + let cwnd = rounded_usize(bdp * self.params.cwnd_gain, self.params.min_cwnd); + Some(cwnd.max(self.params.min_cwnd)) + } + + pub(super) fn rtprop_ms(&self) -> Option { + self.rtprop_secs.min().map(secs_to_ms) + } + + /// Raw BtlBw max-filter value in the active unit per second (`None` cold-start). + pub(super) fn btlbw_units_per_sec(&self) -> Option { + self.btlbw_per_sec.max() + } + + pub(super) fn btlbw_milliblocks_per_sec(&self) -> Option { + // A rounded non-negative rate scaled by 1000 fits u64 for any real rate. Only + // meaningful under `Blocks`; the byte trace path reports bytes/sec instead. + self.btlbw_per_sec + .max() + .map(|rate| (rate * 1000.0).round() as u64) + } + + pub(super) fn delivered(&self) -> u64 { + self.delivered + } + + /// Numeric phase code for the trace (0 = ProbeBw, 1 = ProbeRtt). + pub(super) fn phase_code(&self) -> u64 { + self.phase.trace_code() + } + + /// The smoothed request round-trip in milliseconds, for tracing the delay-gradient. + pub(super) fn smoothed_elapsed_ms(&self) -> Option { + self.smoothed_elapsed_secs.map(secs_to_ms) + } + + /// The delay-gradient ceiling in blocks once it has bound the cwnd (`None` while + /// still unbounded), for tracing. + pub(super) fn delay_cap(&self) -> Option { + (self.delay_cap != usize::MAX).then_some(self.delay_cap) + } +} + +#[cfg(test)] +mod bbr_tests { + use super::super::{ + request::{BlockRangeRequest, ExpectedBlock}, + state::{DownloadWindow, OutstandingBlockRange, ReceivedBlockTracker}, + }; + use super::*; + use zebra_chain::block; + + /// A config with a short ProbeRTT cadence and predictable cwnd math for the unit + /// tests below. The probe interval/duration are scaled down so a handful of + /// deliveries crosses a full ProbeBw → ProbeRtt → ProbeBw cycle. + fn bbr_test_config() -> ZakuraBlockSyncConfig { + ZakuraBlockSyncConfig { + // These tests assert blocks-slot semantics; pin the unit so the production + // default flip to `Bytes` does not change them. + bbr_cwnd_unit: CwndUnit::Blocks, + bbr_min_cwnd: 4, + bbr_cwnd_gain_percent: 200, + bbr_probe_rtt_interval: Duration::from_secs(1), + bbr_probe_rtt_duration: Duration::from_millis(200), + bbr_rtprop_window: Duration::from_secs(10), + bbr_delivery_rate_window: Duration::from_secs(10), + initial_inflight_requests: 16, + ..Default::default() + } + } + + /// A clean delivery: 40 blocks in 10 ms ⇒ rate 4000 blk/s, RTprop 0.01 s, + /// BDP 40 blocks, ×2 gain ⇒ cwnd target 80. + const CLEAN_ELAPSED: Duration = Duration::from_millis(10); + const CLEAN_BLOCKS: u32 = 40; + const EXPECTED_CWND: usize = 80; + + /// Blocks-mode delivery helper (the `delivered_bytes` arg is ignored under + /// `CwndUnit::Blocks`, so it passes 0). `inflight` is the request count, which is the + /// Blocks-unit in-flight measure. + fn record_delivery( + bbr: &mut BbrState, + now: Instant, + elapsed: Duration, + blocks: u32, + inflight: usize, + ) { + let snapshot = DeliverySnapshot { + delivered: bbr.delivered, + delivered_at: now - elapsed, + }; + // Blocks unit: the in-flight measure is the request count. + bbr.record_delivery(now, elapsed, blocks, 0, inflight as u64, snapshot); + } + + #[test] + fn cwnd_tracks_bdp_after_first_delivery() { + let mut bbr = BbrState::new(&bbr_test_config()); + let t0 = Instant::now(); + // Cold start: the configured initial window until the first BDP sample. + assert_eq!(bbr.effective_cwnd(), 16); + record_delivery(&mut bbr, t0, CLEAN_ELAPSED, CLEAN_BLOCKS, 50); + assert_eq!(bbr.effective_cwnd(), EXPECTED_CWND); + assert_eq!(bbr.phase, BbrPhase::ProbeBw); + } + + #[test] + fn one_block_responses_observe_pipe_delivery_rate() { + let mut bbr = BbrState::new(&bbr_test_config()); + let t0 = Instant::now(); + let rtprop = Duration::from_millis(100); + let sent_at = t0 - rtprop; + let snapshots: Vec<_> = (0..16).map(|_| bbr.delivery_snapshot(sent_at)).collect(); + + for snapshot in snapshots { + bbr.record_delivery(t0, rtprop, 1, 0, 16, snapshot); + } + + // Sixteen one-block responses completed during the same request interval: + // BtlBw = 16 / 100 ms, BDP = 16, cwnd gain = 2. + assert_eq!(bbr.effective_cwnd(), 32); + assert_eq!(bbr.btlbw_milliblocks_per_sec(), Some(160_000)); + } + + #[test] + fn delivery_rate_floor_uses_previous_rtprop_sample() { + let mut bbr = BbrState::new(&bbr_test_config()); + let t0 = Instant::now(); + + // Establish a 100 ms RTprop and 100 blocks/s BtlBw sample. + record_delivery(&mut bbr, t0, Duration::from_millis(100), 10, 10); + assert_eq!(bbr.btlbw_milliblocks_per_sec(), Some(100_000)); + + // A later 1 ms request is also the new RTprop, but it must not remove the + // floor for its own delivery-rate sample. With the old ordering this sample + // was 10 / 1 ms = 10_000 blocks/s and inflated BtlBw by 100x. + record_delivery( + &mut bbr, + t0 + Duration::from_millis(10), + Duration::from_millis(1), + 10, + 10, + ); + assert_eq!(bbr.rtprop_ms(), Some(1)); + assert_eq!(bbr.btlbw_milliblocks_per_sec(), Some(100_000)); + } + + #[test] + fn probe_rtt_pins_min_cwnd_then_drains_and_exits() { + let cfg = bbr_test_config(); + let min_cwnd = usize::try_from(cfg.bbr_min_cwnd).unwrap(); + let mut bbr = BbrState::new(&cfg); + let t0 = Instant::now(); + + // Establish a healthy cwnd; anchors the first probe at t0. + record_delivery(&mut bbr, t0, CLEAN_ELAPSED, CLEAN_BLOCKS, 50); + assert_eq!(bbr.effective_cwnd(), EXPECTED_CWND); + + // One interval later, a delivery trips ProbeRtt: cwnd pins to min_cwnd even + // though the BDP estimate is unchanged. + let t1 = t0 + Duration::from_millis(1_100); + record_delivery(&mut bbr, t1, CLEAN_ELAPSED, CLEAN_BLOCKS, 50); + assert_eq!(bbr.phase, BbrPhase::ProbeRtt); + assert_eq!(bbr.effective_cwnd(), min_cwnd); + + // Queue not yet drained (inflight still above min): hold ProbeRtt, no timer. + let t2 = t1 + Duration::from_millis(50); + record_delivery(&mut bbr, t2, CLEAN_ELAPSED, 10, min_cwnd + 5); + assert_eq!(bbr.phase, BbrPhase::ProbeRtt); + assert!(bbr.probe_rtt_drained_at.is_none()); + + // Queue drains to the floor: the hold timer starts here. + let t3 = t2 + Duration::from_millis(20); + record_delivery(&mut bbr, t3, CLEAN_ELAPSED, 10, min_cwnd - 1); + assert_eq!(bbr.phase, BbrPhase::ProbeRtt); + assert_eq!(bbr.probe_rtt_drained_at, Some(t3)); + + // Before the hold elapses, still draining. + let t4 = t3 + Duration::from_millis(100); + record_delivery(&mut bbr, t4, CLEAN_ELAPSED, 10, 1); + assert_eq!(bbr.phase, BbrPhase::ProbeRtt); + + // After probe_rtt_duration past the drain, exit to ProbeBw and restore cwnd. + let t5 = t3 + Duration::from_millis(200); + record_delivery(&mut bbr, t5, CLEAN_ELAPSED, 10, 1); + assert_eq!(bbr.phase, BbrPhase::ProbeBw); + assert_eq!(bbr.effective_cwnd(), EXPECTED_CWND); + assert_eq!(bbr.last_probe_rtt_at, Some(t5)); + } + + #[test] + fn probe_rtt_collapses_cwnd_for_a_slow_peer() { + // The headline case: a peer whose RTprop inflated under a deep queue. ProbeRtt + // forces the cwnd to min_cwnd while it drains, regardless of the (stale, large) + // BDP estimate — this is the slow-peer collapse the trace analysis motivated. + let cfg = bbr_test_config(); + let min_cwnd = usize::try_from(cfg.bbr_min_cwnd).unwrap(); + let mut bbr = BbrState::new(&cfg); + let t0 = Instant::now(); + record_delivery(&mut bbr, t0, CLEAN_ELAPSED, CLEAN_BLOCKS, 50); + let t1 = t0 + Duration::from_millis(1_100); + record_delivery(&mut bbr, t1, CLEAN_ELAPSED, CLEAN_BLOCKS, 50); + assert_eq!(bbr.effective_cwnd(), min_cwnd); + } + + #[test] + fn timeout_dip_applies_in_probe_bw_but_is_suppressed_in_probe_rtt() { + let cfg = bbr_test_config(); + let min_cwnd = usize::try_from(cfg.bbr_min_cwnd).unwrap(); + let mut bbr = BbrState::new(&cfg); + let t0 = Instant::now(); + record_delivery(&mut bbr, t0, CLEAN_ELAPSED, CLEAN_BLOCKS, 50); + assert_eq!(bbr.effective_cwnd(), EXPECTED_CWND); + + // In ProbeBw a timeout dips the cwnd by the multiplicative factor. + bbr.dip_on_timeout(); + let expected_dip = (EXPECTED_CWND as f64 * BBR_TIMEOUT_DIP).round() as usize; + assert_eq!(bbr.effective_cwnd(), expected_dip); + + // Enter ProbeRtt; a timeout there is an expected drain consequence, not + // congestion signal, so cwnd_cap is left untouched. + let t1 = t0 + Duration::from_millis(1_100); + record_delivery(&mut bbr, t1, CLEAN_ELAPSED, CLEAN_BLOCKS, 50); + assert_eq!(bbr.phase, BbrPhase::ProbeRtt); + let cap_before = bbr.cwnd_cap; + bbr.dip_on_timeout(); + assert_eq!(bbr.cwnd_cap, cap_before); + assert_eq!(bbr.effective_cwnd(), min_cwnd); + } + + /// Push `n` placeholder outstanding requests onto a window to drive its slot count. + fn fill_outstanding(window: &mut DownloadWindow, n: usize) { + let now = Instant::now(); + for _ in 0..n { + window.outstanding.push(OutstandingBlockRange { + request: BlockRangeRequest { + start_height: block::Height(0), + count: 1, + anchor_hash: block::Hash([0; 32]), + estimated_bytes: 0, + expected_blocks: Vec::new(), + }, + queued_at: now, + deadline: now, + delivery_snapshot: window.delivery_snapshot(now), + delivered_bytes: 0, + received: ReceivedBlockTracker::default(), + }); + } + } + + #[test] + fn floor_bypass_grants_bonus_slots_only_when_cwnd_is_saturated() { + // Cold-start cwnd 8, hard cap well above it so the bonus is not clamped. + let cfg = ZakuraBlockSyncConfig { + initial_inflight_requests: 8, + max_inflight_requests: 256, + ..bbr_test_config() + }; + let mut window = DownloadWindow::new(&cfg); + assert_eq!(window.bbr_effective_cwnd(), 8); + + // Below cwnd: normal capacity already covers the floor, bonus adds nothing extra + // beyond the same headroom. + fill_outstanding(&mut window, 6); + assert_eq!(window.available_slots(), 2); + assert_eq!(window.available_slots_with_bonus(2), 4); + + // Saturated at cwnd: normal capacity is 0 but the floor may borrow the bonus. + fill_outstanding(&mut window, 2); + assert_eq!(window.available_slots(), 0); + assert_eq!(window.available_slots_with_bonus(2), 2); + + // Saturated even into the bonus region: nothing left for anyone. + fill_outstanding(&mut window, 2); + assert_eq!(window.available_slots_with_bonus(2), 0); + } + + #[test] + fn delay_gradient_does_not_bind_an_uncongested_peer() { + // Every delivery's round-trip equals RTprop (no queue), so the delay ceiling + // stays unbounded and the cwnd tracks the full BDP target. + let mut bbr = BbrState::new(&bbr_test_config()); + let mut now = Instant::now(); + for _ in 0..20 { + record_delivery(&mut bbr, now, CLEAN_ELAPSED, CLEAN_BLOCKS, 50); + now += Duration::from_millis(5); + } + assert_eq!(bbr.effective_cwnd(), EXPECTED_CWND); + assert!(bbr.delay_cap().is_none(), "ceiling should stay unbounded"); + } + + #[test] + fn delay_gradient_caps_cwnd_when_the_round_trip_inflates() { + // RTprop is established low (10 ms), then every round-trip runs far above it + // (queue building) while the BtlBw×RTprop target stays high — exactly the cwnd + // overshoot the delay-gradient must contain. The ceiling ratchets the effective + // cwnd well below the (inflated) BDP target. + let cfg = bbr_test_config(); + let mut bbr = BbrState::new(&cfg); + let t0 = Instant::now(); + // One clean delivery anchors RTprop at 10 ms and the BDP target at 80. + record_delivery(&mut bbr, t0, CLEAN_ELAPSED, CLEAN_BLOCKS, 50); + assert_eq!(bbr.effective_cwnd(), EXPECTED_CWND); + + // Now deliveries keep arriving at the same low RTprop sample for the min-filter + // (so the target stays 80) but with long *smoothed* round-trips — model that with + // a low-elapsed sample to hold RTprop and the cwnd target, interleaved with the + // queue signal. Here we simply feed inflated round-trips: RTprop min stays 10 ms + // (the first sample is in-window), smoothed climbs, the ceiling ratchets down. + let inflated = Duration::from_millis(120); + let mut now = t0; + for _ in 0..40 { + now += Duration::from_millis(5); + record_delivery(&mut bbr, now, inflated, CLEAN_BLOCKS, 50); + } + assert_eq!( + bbr.phase, + BbrPhase::ProbeBw, + "stay in ProbeBw for this test" + ); + assert!( + bbr.effective_cwnd() < EXPECTED_CWND, + "delay-gradient should cap the cwnd below the BDP target, got {}", + bbr.effective_cwnd(), + ); + assert!( + bbr.delay_cap().is_some(), + "the ceiling should have bound the cwnd", + ); + } + + /// Push `count` single-height requests each reserving `bytes_each` estimated bytes. + fn push_outstanding_bytes(window: &mut DownloadWindow, count: usize, bytes_each: u64) { + let now = Instant::now(); + for i in 0..count { + // A `u32` index; the test count is tiny so the cast is safe. + let height = block::Height(1 + i as u32); + window.outstanding.push(OutstandingBlockRange { + request: BlockRangeRequest { + start_height: height, + count: 1, + anchor_hash: block::Hash([0; 32]), + estimated_bytes: bytes_each, + expected_blocks: vec![ExpectedBlock { + height, + hash: block::Hash([0; 32]), + estimated_bytes: bytes_each, + }], + }, + queued_at: now, + deadline: now, + delivery_snapshot: window.delivery_snapshot(now), + delivered_bytes: 0, + received: ReceivedBlockTracker::default(), + }); + } + } + + /// A byte-unit config whose cold-start byte cwnd is exactly `min_cwnd_bytes` (the + /// floor doubles as the cold-start window) with a request-count cap well above it. + fn byte_test_config(min_cwnd_bytes: u64, max_inflight: u32) -> ZakuraBlockSyncConfig { + ZakuraBlockSyncConfig { + bbr_cwnd_unit: CwndUnit::Bytes, + bbr_min_cwnd_bytes: min_cwnd_bytes, + max_inflight_requests: max_inflight, + ..bbr_test_config() + } + } + + #[test] + fn cwnd_unit_bytes_budgets_in_flight_by_reserved_bytes() { + // The byte cwnd is the byte floor at cold start: an 8000 B in-flight budget, + // sourced from the controller's byte denomination — independent of how many + // *requests* that is. + let cfg = byte_test_config(8000, 256); + let mut window = DownloadWindow::new(&cfg); + assert_eq!(window.available_slots(), 8000); + + // Six 1000 B requests (their header-hinted `estimated_bytes`) leave 2000 B... + push_outstanding_bytes(&mut window, 6, 1000); + assert_eq!(window.available_slots(), 2000); + // ...and two more exhaust the byte budget. + push_outstanding_bytes(&mut window, 2, 1000); + assert_eq!(window.available_slots(), 0); + + // A peer serving 4 KB bodies fills the same byte cwnd with far fewer requests — + // the point of the byte unit. Two 4000 B requests already saturate the 8000 B + // budget, so the in-flight request count self-adjusts to the body size. + let mut big = DownloadWindow::new(&cfg); + push_outstanding_bytes(&mut big, 2, 4000); + assert_eq!(big.available_slots(), 0); + } + + #[test] + fn cwnd_unit_bytes_enforces_the_request_count_hard_cap() { + // A peer advertising a small inflight cap but serving tiny bodies must not be + // issued more *requests* than it will service, however much byte headroom the + // cwnd still shows — the advertised request-count cap binds first (review fix F2). + let cfg = byte_test_config(400_000, 4); // 400 KB byte cwnd, hard cap 4 requests + let mut window = DownloadWindow::new(&cfg); + assert_eq!(window.hard_outbound_capacity(), 4); + // 400_000 B of byte headroom — room for many tiny bodies. + assert!(window.available_slots() > 0); + + // Four tiny (10 B) requests reach the request-count hard cap. The byte budget is + // nowhere near exhausted (40 B of 400_000 B), but the advertised cap must bind: + // no further request may be issued. + push_outstanding_bytes(&mut window, 4, 10); + assert_eq!( + window.available_slots(), + 0, + "the advertised request-count cap must bind even with byte headroom left", + ); + // The floor bypass must not breach the advertised cap either. + assert_eq!(window.available_slots_with_bonus(2), 0); + } + + #[test] + fn probe_rtt_drain_respects_reserved_bytes_under_byte_unit() { + // Regression for the unit-inconsistent ProbeRtt drain gate under `CwndUnit::Bytes`. + // The drain check compares the in-flight measure against `min_cwnd`, which under + // `Bytes` is `min_cwnd_bytes`. The window must therefore feed the controller its + // reserved *bytes*, not a request count — otherwise a small count is always below + // the multi-KiB byte floor, the hold timer starts before the byte queue has + // drained, and ProbeRtt exits while still contended. Here a still-full byte queue + // must hold ProbeRtt open until the reserved bytes actually fall to the floor. + let cfg = byte_test_config(8_000, 256); // 8 KB byte floor + let mut window = DownloadWindow::new(&cfg); + let t0 = Instant::now(); + + // Five 4 KB reservations ⇒ 20 KB in flight, well above the 8 KB floor. + push_outstanding_bytes(&mut window, 5, 4_000); + let deliver = |window: &mut DownloadWindow, now: Instant| { + let snapshot = window.delivery_snapshot(now - Duration::from_millis(10)); + window.record_delivery(now, Duration::from_millis(10), 1, 4_000, snapshot); + }; + + // First delivery anchors the probe at t0; stays in ProbeBw. + deliver(&mut window, t0); + assert_eq!(window.bbr_phase_code(), 0); + + // One interval later ProbeRtt trips. + let t1 = t0 + Duration::from_millis(1_100); + deliver(&mut window, t1); + assert_eq!(window.bbr_phase_code(), 1, "ProbeRtt should have tripped"); + + // The byte queue is still 20 KB (above the floor), so even well past + // `probe_rtt_duration` the drain is NOT detected and ProbeRtt holds. The buggy + // count-vs-bytes gate would have stamped the drain immediately and exited here. + let t2 = t1 + Duration::from_millis(250); + deliver(&mut window, t2); + let t3 = t2 + Duration::from_millis(250); + deliver(&mut window, t3); + assert_eq!( + window.bbr_phase_code(), + 1, + "the byte queue never drained, so ProbeRtt must stay pinned to the floor", + ); + + // Drain the byte queue to the floor (4 KB <= 8 KB): the hold timer starts now... + window.outstanding.truncate(1); + let t4 = t3 + Duration::from_millis(10); + deliver(&mut window, t4); + assert_eq!( + window.bbr_phase_code(), + 1, + "draining, hold timer just started" + ); + + // ...and only after `probe_rtt_duration` past the drain does ProbeRtt exit. + let t5 = t4 + Duration::from_millis(250); + deliver(&mut window, t5); + assert_eq!( + window.bbr_phase_code(), + 0, + "byte queue drained and held, ProbeRtt should exit to ProbeBw", + ); + } + + #[test] + fn byte_mode_btlbw_is_bytes_per_sec_and_floor_binds_at_low_bdp() { + // A 20 KB body served in 10 ms: BtlBw = 2 MB/s, raw round trip 10 ms ⇒ a genuine + // byte-BDP of 20 KB, ×2 gain = 40 KB, below the 100 KB `min_cwnd_bytes` floor. So + // the floor is the binding operating window — the low-BDP regime the floor exists + // for. (Unlike the old size-residual model, this binds because the *real* BDP is + // small, not because the residual spuriously collapsed to ~0.) + let cfg = byte_test_config(100_000, 256); + let mut bbr = BbrState::new(&cfg); + let t0 = Instant::now(); + let snapshot = DeliverySnapshot { + delivered: 0, + delivered_at: t0 - Duration::from_millis(10), + }; + bbr.record_delivery(t0, Duration::from_millis(10), 1, 20_000, 50, snapshot); + // BtlBw is denominated in bytes/sec now, not blocks/sec. + assert_eq!(bbr.btlbw_units_per_sec(), Some(2_000_000.0)); + // The byte floor binds because BDP×gain (40 KB) < floor (100 KB). + assert_eq!(bbr.effective_cwnd(), 100_000); + } + + #[test] + fn byte_bdp_uses_raw_rtt_so_a_fast_carrier_lifts_off_the_floor() { + // The regression guard for the floor-pin fix. An 800 KB body served in 20 ms: + // BtlBw = 40 MB/s, raw round trip 20 ms ⇒ byte-BDP 800 KB, ×2 gain = 1.6 MB, well + // above the 256 KB floor. The cwnd lifts off the floor. + // + // The *size residual* of this same delivery collapses to the ε floor (its implied + // transmission 800 KB / 40 MB/s = 20 ms equals the whole round trip), so the old + // model would have computed BDP ≈ 0 and pinned the cwnd at 256 KB. Using the raw + // round trip for the BDP is what keeps a genuinely fast carrier from being + // under-pipelined. + let cfg = byte_test_config(256_000, 256); + let mut bbr = BbrState::new(&cfg); + let t0 = Instant::now(); + let snapshot = DeliverySnapshot { + delivered: 0, + delivered_at: t0 - Duration::from_millis(20), + }; + bbr.record_delivery(t0, Duration::from_millis(20), 1, 800_000, 50, snapshot); + assert_eq!(bbr.btlbw_units_per_sec(), Some(40_000_000.0)); + // The residual would have zeroed the BDP; the raw round trip does not. + assert_eq!(bbr.size_residual_rtprop(0.02, 800_000), 1e-4); + assert_eq!(bbr.effective_cwnd(), 1_600_000); + } + + #[test] + fn byte_residual_rtprop_subtracts_transmission_time() { + // With an established 1 MB/s BtlBw, a 100 ms round trip that carried 50 KB has a + // residual RTprop of 100 ms − 50 ms = 50 ms (the fixed-latency component), while a + // round trip whose implied transmission exceeds it clamps to the positive floor. + let cfg = byte_test_config(1, 256); + let mut bbr = BbrState::new(&cfg); + let now = Instant::now(); + bbr.btlbw_per_sec.observe(now, 1_000_000.0); + let residual = bbr.size_residual_rtprop(0.1, 50_000); + assert!( + (residual - 0.05).abs() < 1e-9, + "residual should subtract 50 ms of transmission, got {residual}", + ); + // 200 KB at 1 MB/s implies 200 ms of transmission > the 100 ms round trip: clamp. + assert_eq!(bbr.size_residual_rtprop(0.1, 200_000), 1e-4); + } + + #[test] + fn byte_size_aware_delay_gate_does_not_ratchet_a_big_block() { + // A long smoothed round-trip that is fully explained by a big block's transmission + // time must NOT ratchet the delay ceiling under `Bytes` (size-aware expected RT), + // whereas the identical round trip WOULD ratchet under `Blocks` (RTprop-only). + let now = Instant::now(); + + let bytes_cfg = byte_test_config(1, 256); + let mut bytes = BbrState::new(&bytes_cfg); + bytes.btlbw_per_sec.observe(now, 1_000_000.0); // 1 MB/s + // The delay gate's base is the *residual* RTprop estimator (10 ms base RTT here). + bytes.rtprop_residual_secs.observe(now, 0.01); + // 200 ms round trip carrying a 190 KB body: expected ≈ 10 ms + 190 ms = 200 ms. + bytes.update_delay_cap(0.2, 190_000); + assert!( + bytes.delay_cap().is_none(), + "a big block's honest transfer time must not look like a standing queue", + ); + + let blocks_cfg = bbr_test_config(); + let mut blocks = BbrState::new(&blocks_cfg); + blocks.rtprop_residual_secs.observe(now, 0.01); + // Same 200 ms round trip, blocks mode: expected = RTprop (10 ms) → ratchets. + blocks.update_delay_cap(0.2, 190_000); + assert!( + blocks.delay_cap().is_some(), + "blocks mode treats the inflated round trip as a queue and ratchets down", + ); + } + + #[test] + fn floor_bypass_never_exceeds_the_advertised_hard_cap() { + // cwnd == hard cap (8): the bypass must not push in-flight past what the peer + // advertised it will service. + let cfg = ZakuraBlockSyncConfig { + initial_inflight_requests: 8, + max_inflight_requests: 8, + ..bbr_test_config() + }; + let mut window = DownloadWindow::new(&cfg); + assert_eq!(window.hard_outbound_capacity(), 8); + fill_outstanding(&mut window, 8); + assert_eq!(window.available_slots(), 0); + assert_eq!(window.available_slots_with_bonus(2), 0); + } +} diff --git a/zebra-network/src/zakura/block_sync/config.rs b/zebra-network/src/zakura/block_sync/config.rs index dc9322b2b4a..06b80a271eb 100644 --- a/zebra-network/src/zakura/block_sync/config.rs +++ b/zebra-network/src/zakura/block_sync/config.rs @@ -7,19 +7,17 @@ use super::{error::*, wire::*, *}; pub const DEFAULT_BS_BLOCKS_PER_RESPONSE: u32 = 1; /// Default advertised hard cap on concurrent in-flight block requests per peer. /// -/// This is the ceiling the adaptive per-peer window grows *toward*, **not** the -/// opening window. Scheduling starts at [`DEFAULT_BS_INITIAL_INFLIGHT`] and ramps -/// up to the peer-advertised value (this default, clamped to -/// [`MAX_BS_INFLIGHT_REQUESTS`]) only after sustained error-free responses (see -/// the streak-gated cubic ramp on `DownloadWindow`). In a homogeneous fleet this -/// is the per-peer concurrency ceiling every peer offers. +/// This is the safety ceiling the BBR-lite cwnd is clamped to, **not** the +/// operating point: the binding per-peer concurrency is the measured +/// bandwidth-delay product (see `DownloadWindow`'s BBR controller), which is +/// normally far below this. In a homogeneous fleet this is the per-peer +/// concurrency ceiling every peer offers. pub const DEFAULT_BS_MAX_INFLIGHT: u32 = 32000; -/// Initial per-peer outbound request window (slow-start point). +/// Initial per-peer cwnd (cold-start point), in blocks. /// -/// The adaptive window starts here and grows toward the peer-advertised hard cap -/// on successful responses, rather than opening at the full `max_inflight`. This -/// keeps the opening burst modest so a peer is not flooded before its latency is -/// known. +/// The BBR cwnd opens here and converges to the BDP-derived target once the first +/// delivery sample arrives, rather than opening at the full `max_inflight`. This +/// keeps the opening burst modest before a peer's rate/latency is known. pub const DEFAULT_BS_INITIAL_INFLIGHT: u32 = 64; /// Maximum peer-advertised in-flight request count accepted by this node. /// @@ -76,13 +74,24 @@ pub const BS_CHECKPOINT_RANGE_BYTE_FLOOR: u64 = MIN_BS_CHECKPOINT_SUBMITTED_BLOCK_APPLIES as u64 * BS_PER_BLOCK_WORST_CASE_BYTES; /// Default block-sync request timeout. pub const DEFAULT_BS_REQUEST_TIMEOUT: Duration = Duration::from_secs(8); +/// Default short leash on a floor (lowest-missing-height) request. +/// +/// A floor request that has not been served within this window is rescued to a +/// faster carrier (returned to the queue + the peer retry-avoided), never letting +/// the contiguous download floor wait on a slow peer. Far tighter than the base +/// `request_timeout`, which governs patient above-floor speculation instead. +pub const DEFAULT_BS_FLOOR_RESCUE_TIMEOUT: Duration = Duration::from_secs(2); +/// Request-timeout windows allowed before block-progress liveness disconnects. +const BLOCK_PROGRESS_TIMEOUT_REQUESTS: u32 = 4; +/// Default cooldown before a no-progress peer may be admitted again. +pub const DEFAULT_BS_NO_PROGRESS_PEER_COOLDOWN: Duration = Duration::from_secs(180); /// Default hard floor-peer avoid cooldown after a watchdog cancellation. pub const DEFAULT_BS_FLOOR_PEER_AVOID_COOLDOWN: Duration = DEFAULT_BS_REQUEST_TIMEOUT; -/// Default block-sync status refresh interval reserved for later advertisement. +/// Default block-sync status refresh interval after local frontier changes. pub const DEFAULT_BS_STATUS_REFRESH_INTERVAL: Duration = Duration::from_secs(30); -/// Default tolerated size-hint deviation percentage reserved for later soft scoring. +/// Default tolerated size-hint deviation percentage before a peer is reported. pub const DEFAULT_BS_SIZE_DEVIATION_TOLERANCE: u32 = 200; -/// Default block-sync peer fanout for the same requested range. +/// Default legacy range-fanout reservation multiplier. pub const DEFAULT_BS_FANOUT: usize = 1; /// Maximum peer-advertised aggregate byte target accepted per requested range. /// @@ -91,6 +100,59 @@ pub const DEFAULT_BS_FANOUT: usize = 1; /// only controls how many bounded body frames a server sends before `BlocksDone`. pub const MAX_BS_RESPONSE_BYTES: u32 = DEFAULT_BS_MAX_RESPONSE_BYTES; +/// Default steady-state cwnd gain, as a percent of the bandwidth-delay product. +pub const DEFAULT_BS_BBR_CWND_GAIN_PERCENT: u32 = 200; +/// Default ProbeBW up-probe pacing gain, percent. +pub const DEFAULT_BS_BBR_PROBE_BW_GAIN_PERCENT: u32 = 125; +/// Default ProbeRTT cadence: how often to drain to re-measure the min-RTT. +pub const DEFAULT_BS_BBR_PROBE_RTT_INTERVAL: Duration = Duration::from_secs(10); +/// Default ProbeRTT hold time at the drained cwnd. +pub const DEFAULT_BS_BBR_PROBE_RTT_DURATION: Duration = Duration::from_millis(200); +/// Default windowed-min horizon for the RTprop (min-RTT) estimate. Kept equal to +/// the ProbeRTT interval so a stale min is always re-probed before it expires. +pub const DEFAULT_BS_BBR_RTPROP_WINDOW: Duration = Duration::from_secs(10); +/// Default max-filter horizon for the BtlBw (delivery-rate) estimate. +pub const DEFAULT_BS_BBR_DELIVERY_RATE_WINDOW: Duration = Duration::from_secs(10); +/// Default per-RTT Startup growth (percent ⇒ ≈2×/RTT exponential ramp). +pub const DEFAULT_BS_BBR_STARTUP_GROWTH_PERCENT: u32 = 200; +/// Default minimum cwnd in blocks — keeps the pipe primed and lets ProbeRTT send. +pub const DEFAULT_BS_BBR_MIN_CWND: u32 = 4; +/// Default minimum cwnd in **bytes** (the floor under [`CwndUnit::Bytes`]). +/// +/// Under byte denomination the steady cwnd is `BtlBw_bytes × RTprop × gain`. For a +/// low-latency peer that product is small (a fast link with ~1 ms base RTT needs +/// little in flight to stay busy), so this floor — not the BDP — is the binding +/// operating window most of the time. It is sized to keep enough concurrent +/// single-block requests in flight to actually pipeline a server that has spare +/// capacity (the trace showed peers 60–86% idle at a pinned cwnd of 4), while the +/// size-aware delay-gradient still shrinks it back if a real standing queue forms. +/// **This is the primary live-A/B tuning lever** (`byte-cwnd-1`): raise it to push +/// more concurrency, lower it if floor head-of-line latency regresses. +pub const DEFAULT_BS_BBR_MIN_CWND_BYTES: u64 = 4 * 1024 * 1024; +/// Default delay-gradient down-adjust threshold, percent of RTprop. +pub const DEFAULT_BS_BBR_DELAY_GRADIENT_PERCENT: u32 = 150; +/// Default number of slots the floor request may borrow beyond the BBR cwnd, so the +/// lowest missing height is fetched even when every servable peer is at its cwnd. +pub const DEFAULT_BS_FLOOR_BYPASS_SLOTS: u32 = 2; + +/// Unit the per-peer BBR cwnd budgets in-flight work against. The controller itself is +/// unit-agnostic (it sizes a cwnd from measured delivery rate × RTprop); the unit only +/// changes how outstanding work is counted against that cwnd in `available_slots`. +#[derive(Copy, Clone, Debug, Default, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum CwndUnit { + /// Count outstanding *requests* against the cwnd (one request ≈ one slot), equal + /// weight regardless of body size. The A/B baseline — retained for comparison + /// against the byte controller and for tests. + Blocks, + /// Count reserved body *bytes* against the cwnd, where the cwnd is itself a byte + /// bandwidth-delay product sourced from the header size hints (`BtlBw_bytes × + /// RTprop × gain`), so a peer serving large bodies holds fewer in flight and one + /// serving small bodies holds many. The shipped default. + #[default] + Bytes, +} + /// Block-sync peer status advertisement. #[derive(Copy, Clone, Debug, Eq, PartialEq)] pub struct BlockSyncStatus { @@ -163,8 +225,8 @@ pub struct ZakuraBlockSyncConfig { pub max_blocks_per_response: u32, /// Maximum concurrent `GetBlocks` requests this node advertises per peer. pub max_inflight_requests: u32, - /// Initial per-peer outbound request window (slow-start point); grows toward - /// the advertised hard cap on success. Clamped to `[1, max_inflight_requests]`. + /// Initial per-peer BBR cwnd (cold-start point), in blocks; converges to the + /// BDP-derived target once the first delivery is measured. pub initial_inflight_requests: u32, /// Maximum total response bytes this node advertises per `GetBlocks` response. pub max_response_bytes: u32, @@ -177,19 +239,67 @@ pub struct ZakuraBlockSyncConfig { /// How long to avoid reassigning an expired floor height to the same peer. #[serde(with = "humantime_serde")] pub floor_peer_avoid_cooldown: Duration, - /// Maximum block bodies submitted to the verifier before completed applies - /// release more submission slots. + /// Depth for block-sync action/body channels, clamped to at least one full + /// checkpoint range. pub max_submitted_block_applies: usize, /// Timeout for an outstanding block-body range request. #[serde(with = "humantime_serde")] pub request_timeout: Duration, + /// Short leash on a floor request before its height is rescued to a faster + /// carrier. Clamped positive and never above `request_timeout`. + #[serde(with = "humantime_serde")] + pub floor_rescue_timeout: Duration, + /// How long to keep a peer disconnected after it makes no accepted block progress. + #[serde(with = "humantime_serde")] + pub no_progress_peer_cooldown: Duration, /// How often this node sends unsolicited status refreshes after local frontier changes. #[serde(with = "humantime_serde")] pub status_refresh_interval: Duration, /// Percentage deviation from advertised body-size hints tolerated before soft scoring. pub size_deviation_tolerance: u32, - /// Number of peers later range scheduling may fan out to for the same body gap. + /// Legacy range-fanout reservation multiplier. + /// + /// No active scheduler currently sends the same range to multiple peers; this + /// only keeps old configs parsing and sizes the validation floor for one + /// worst-case floor request. pub fanout: usize, + /// Steady-state cwnd as a percent of the measured bandwidth-delay product. + pub bbr_cwnd_gain_percent: u32, + /// ProbeBW up-probe pacing gain, percent. Reserved: the ProbeBW gain cycle is not + /// yet wired into the controller, so this knob is currently inert (the BtlBw + /// max-filter already adopts higher delivery rates without an explicit up-probe). + pub bbr_probe_bw_gain_percent: u32, + /// How often to enter ProbeRTT to refresh the min-RTT estimate. + #[serde(with = "humantime_serde")] + pub bbr_probe_rtt_interval: Duration, + /// How long to hold the drained cwnd during ProbeRTT. + #[serde(with = "humantime_serde")] + pub bbr_probe_rtt_duration: Duration, + /// Windowed-min horizon for the RTprop (min-RTT) estimate. + #[serde(with = "humantime_serde")] + pub bbr_rtprop_window: Duration, + /// Max-filter horizon for the delivery-rate (BtlBw) estimate. + #[serde(with = "humantime_serde")] + pub bbr_delivery_rate_window: Duration, + /// Per-RTT Startup cwnd growth, percent. Reserved: there is no separate Startup ramp + /// phase yet, so this knob is currently inert (cold start uses + /// `initial_inflight_requests` and the BDP estimate takes over once samples arrive). + pub bbr_startup_growth_percent: u32, + /// Minimum cwnd, in blocks (the floor under [`CwndUnit::Blocks`]). + pub bbr_min_cwnd: u32, + /// Minimum cwnd, in bytes (the floor under [`CwndUnit::Bytes`]). Doubles as the + /// cold-start byte window before the first delivery sample, and as the binding + /// operating window for low-latency peers whose byte-BDP is below it. + pub bbr_min_cwnd_bytes: u64, + /// Delay-gradient down-adjust threshold, percent of RTprop. + pub bbr_delay_gradient_percent: u32, + /// Unit the BBR cwnd budgets in-flight work against (`bytes` = header-hinted + /// reserved body bytes, default; `blocks` = request count, the A/B baseline). + pub bbr_cwnd_unit: CwndUnit, + /// Slots a floor (lowest-missing-height) request may borrow beyond the BBR cwnd, up + /// to the peer's advertised hard cap. Lets the floor be fetched even when every + /// servable peer is saturated at its cwnd; `0` disables the bypass. + pub floor_bypass_slots: u32, /// Block-sync peer caps and queue limits owned by this service. pub peer_limits: ServicePeerLimits, } @@ -216,9 +326,23 @@ impl Default for ZakuraBlockSyncConfig { floor_peer_avoid_cooldown: DEFAULT_BS_FLOOR_PEER_AVOID_COOLDOWN, max_submitted_block_applies: DEFAULT_BS_MAX_SUBMITTED_BLOCK_APPLIES, request_timeout: DEFAULT_BS_REQUEST_TIMEOUT, + floor_rescue_timeout: DEFAULT_BS_FLOOR_RESCUE_TIMEOUT, + no_progress_peer_cooldown: DEFAULT_BS_NO_PROGRESS_PEER_COOLDOWN, status_refresh_interval: DEFAULT_BS_STATUS_REFRESH_INTERVAL, size_deviation_tolerance: DEFAULT_BS_SIZE_DEVIATION_TOLERANCE, fanout: DEFAULT_BS_FANOUT, + bbr_cwnd_gain_percent: DEFAULT_BS_BBR_CWND_GAIN_PERCENT, + bbr_probe_bw_gain_percent: DEFAULT_BS_BBR_PROBE_BW_GAIN_PERCENT, + bbr_probe_rtt_interval: DEFAULT_BS_BBR_PROBE_RTT_INTERVAL, + bbr_probe_rtt_duration: DEFAULT_BS_BBR_PROBE_RTT_DURATION, + bbr_rtprop_window: DEFAULT_BS_BBR_RTPROP_WINDOW, + bbr_delivery_rate_window: DEFAULT_BS_BBR_DELIVERY_RATE_WINDOW, + bbr_startup_growth_percent: DEFAULT_BS_BBR_STARTUP_GROWTH_PERCENT, + bbr_min_cwnd: DEFAULT_BS_BBR_MIN_CWND, + bbr_min_cwnd_bytes: DEFAULT_BS_BBR_MIN_CWND_BYTES, + bbr_delay_gradient_percent: DEFAULT_BS_BBR_DELAY_GRADIENT_PERCENT, + bbr_cwnd_unit: CwndUnit::Bytes, + floor_bypass_slots: DEFAULT_BS_FLOOR_BYPASS_SLOTS, peer_limits: ServicePeerLimits::default(), } } @@ -240,7 +364,7 @@ impl ZakuraBlockSyncConfig { clamp_advertised_response_bytes(self.max_response_bytes) } - /// Return the non-zero verifier submission cap. + /// Return the non-zero block-sync action/body channel depth. pub fn submitted_apply_limit(&self) -> usize { self.max_submitted_block_applies .max(MIN_BS_CHECKPOINT_SUBMITTED_BLOCK_APPLIES) @@ -257,7 +381,29 @@ impl ZakuraBlockSyncConfig { self.floor_peer_avoid_cooldown.max(Duration::from_millis(1)) } + /// Return the maximum time an active block-sync peer may go without serving + /// an accepted full block body. + pub(super) fn effective_liveness_timeout(&self) -> Duration { + self.request_timeout + .saturating_mul(BLOCK_PROGRESS_TIMEOUT_REQUESTS) + } + + /// Return the no-progress peer cooldown clamped to a positive duration. + pub(super) fn effective_no_progress_peer_cooldown(&self) -> Duration { + self.no_progress_peer_cooldown.max(Duration::from_millis(1)) + } + + /// Return the floor-rescue leash, clamped positive and no looser than the base + /// request timeout (a floor request is never more patient than a normal one). + pub(super) fn effective_floor_rescue_timeout(&self) -> Duration { + self.floor_rescue_timeout + .clamp(Duration::from_millis(1), self.request_timeout) + } + /// Return the largest byte reservation a single floor request can need. + /// + /// `fanout` is only a compatibility reservation multiplier; it does not mean + /// current scheduling sends the same floor request to multiple peers. pub fn floor_request_byte_reservation(&self) -> u64 { let fanout = u64::try_from(self.fanout.max(1)).unwrap_or(u64::MAX); let worst_case_blocks = u64::from(self.advertised_max_blocks_per_response()) @@ -280,6 +426,25 @@ impl ZakuraBlockSyncConfig { if self.max_inflight_block_bytes <= self.floor_request_byte_reservation() { return Err("max_inflight_block_bytes must exceed one floor request"); } + if self.request_timeout < Duration::from_millis(1) { + return Err("request_timeout must be at least 1ms"); + } + if self.bbr_min_cwnd == 0 { + return Err("bbr_min_cwnd must be greater than zero"); + } + if self.bbr_min_cwnd_bytes == 0 { + return Err("bbr_min_cwnd_bytes must be greater than zero"); + } + if self.bbr_cwnd_gain_percent < 100 + || self.bbr_probe_bw_gain_percent < 100 + || self.bbr_startup_growth_percent < 100 + || self.bbr_delay_gradient_percent < 100 + { + return Err("bbr gain/threshold percentages must be at least 100"); + } + if self.bbr_probe_rtt_interval <= self.bbr_probe_rtt_duration { + return Err("bbr_probe_rtt_interval must exceed bbr_probe_rtt_duration"); + } Ok(()) } diff --git a/zebra-network/src/zakura/block_sync/mod.rs b/zebra-network/src/zakura/block_sync/mod.rs index d5e9cbd8f07..30dc4eecf53 100644 --- a/zebra-network/src/zakura/block_sync/mod.rs +++ b/zebra-network/src/zakura/block_sync/mod.rs @@ -31,6 +31,7 @@ use super::{ }; mod admission; +mod bbr; #[cfg(feature = "internal-bench")] mod bench; mod config; @@ -56,7 +57,10 @@ pub use bench::{ spawn_bench_sequencer, BenchBodyFeeder, BenchCommitter, BenchSequencerHandle, BenchSubmissions, BenchSubmit, SequencerProgress, }; -pub use config::{BlockSyncStatus, ZakuraBlockSyncConfig, MAX_BS_RESPONSE_BYTES}; +pub use config::{ + BlockSyncStatus, CwndUnit, ZakuraBlockSyncConfig, DEFAULT_BS_MAX_SUBMITTED_BLOCK_APPLIES, + MAX_BS_RESPONSE_BYTES, +}; pub use error::BlockSyncWireError; pub use events::{ BlockApplyResult, BlockApplyToken, BlockSyncAction, BlockSyncBlockMeta, BlockSyncEvent, diff --git a/zebra-network/src/zakura/block_sync/peer_registry.rs b/zebra-network/src/zakura/block_sync/peer_registry.rs index 241779ea0e1..f397e9a2a52 100644 --- a/zebra-network/src/zakura/block_sync/peer_registry.rs +++ b/zebra-network/src/zakura/block_sync/peer_registry.rs @@ -15,9 +15,10 @@ //! `received_status` (when it decodes a `Status` frame in its own task), //! `outstanding` (on issue/finish/timeout/disconnect — per *request*, never per //! *body*), slot diagnostics, and download-side misbehavior. The **reactor** owns -//! only entry insert/remove (admission/teardown) and reports serving-side -//! misbehavior. Misbehavior is record-only: it is observed and traced but never -//! drives a disconnect, so the registry keeps no per-peer misbehavior state. +//! entry insert/remove (admission/teardown), serving-side misbehavior, and +//! floor-watchdog hard excludes. Misbehavior is record-only: it is observed and +//! traced but never drives a disconnect, so the registry keeps no per-peer +//! misbehavior state. use std::{ collections::{BTreeMap, HashMap}, @@ -50,12 +51,13 @@ pub(super) struct Entry { /// independent of `work.in_flight`, so it structurally closes the /// reject-rollback window. pub(super) outstanding: BTreeMap, - /// Routine-published slot diagnostics (trace only): the per-peer download - /// window state the reactor reads for the periodic `BLOCK_SYNC_STATE` row. - /// Updated whenever the routine issues/finishes/times out a request. + /// Routine-published slot and BBR diagnostics. The reactor summarizes this for + /// the periodic `BLOCK_SYNC_STATE` row, and peer routines read it for cross-peer + /// floor-bias decisions. Updated whenever the routine issues/finishes/times out + /// a request. pub(super) slots: SlotDiagnostics, - /// Heights this peer may not re-take until the given instant. - pub(super) retry_avoid: BTreeMap, + /// Heights this peer may not re-take after a floor-watchdog cancellation. + pub(super) floor_watchdog_avoid: BTreeMap, /// Monotonic generation bumped each time a routine is (re)spawned for this /// peer. A cancelled routine's async `Drop` only clears outstanding when the /// generation still matches, so an old Drop racing a reset respawn cannot wipe @@ -79,21 +81,21 @@ impl Entry { max_response_bytes: config.advertised_max_response_bytes(), outstanding: BTreeMap::new(), slots: SlotDiagnostics::default(), - retry_avoid: BTreeMap::new(), + floor_watchdog_avoid: BTreeMap::new(), generation, } } } -/// Per-peer download window slot diagnostics published by the routine for the -/// reactor's periodic `BLOCK_SYNC_STATE` trace row. +/// Per-peer download window diagnostics published by the routine for trace +/// summaries and cross-peer floor-bias decisions. #[derive(Copy, Clone, Debug, Default)] pub(super) struct SlotDiagnostics { pub(super) hard_capacity: usize, pub(super) effective_window: usize, pub(super) available_slots: usize, - pub(super) timeout_recovery_slots: usize, pub(super) outstanding_requests: usize, + pub(super) bbr_rtprop_ms: Option, } /// Published metadata for one unreceived outstanding height. @@ -118,6 +120,7 @@ pub(super) struct OutstandingClaim { #[derive(Debug)] pub(super) struct PeerRegistry { peers: StdMutex>, + parked_peers: StdMutex>, /// Source of monotonically-increasing routine generations. next_generation: std::sync::atomic::AtomicU64, } @@ -132,6 +135,7 @@ impl PeerRegistry { pub(super) fn new() -> Self { Self { peers: StdMutex::new(HashMap::new()), + parked_peers: StdMutex::new(HashMap::new()), next_generation: std::sync::atomic::AtomicU64::new(1), } } @@ -142,6 +146,24 @@ impl PeerRegistry { .expect("peer registry mutex is never poisoned") } + fn lock_parked(&self) -> std::sync::MutexGuard<'_, HashMap> { + self.parked_peers + .lock() + .expect("peer registry parked-peer mutex is never poisoned") + } + + /// Refuse this peer at block-sync admission until `until`. + pub(super) fn park_peer_until(&self, peer: &ZakuraPeerId, until: Instant) { + self.lock_parked().insert(peer.clone(), until); + } + + /// Whether the peer is still in its no-progress reconnect cooldown. + pub(super) fn is_peer_parked(&self, peer: &ZakuraPeerId, now: Instant) -> bool { + let mut parked_peers = self.lock_parked(); + parked_peers.retain(|_, until| *until > now); + parked_peers.get(peer).is_some_and(|until| *until > now) + } + /// Admit (or re-admit) a peer and allocate a fresh routine generation. /// /// On a genuinely new peer this inserts a default entry; on a respawn (reset) @@ -164,7 +186,7 @@ impl PeerRegistry { .and_modify(|entry| { entry.direction = direction; entry.outstanding.clear(); - entry.retry_avoid.clear(); + entry.floor_watchdog_avoid.clear(); entry.generation = generation; }) .or_insert_with(|| Entry::new(direction, config, generation)); @@ -229,8 +251,8 @@ impl PeerRegistry { } } - /// Publish the routine's download-window slot diagnostics (trace only), - /// generation-gated like the outstanding writers. + /// Publish the routine's download-window diagnostics, generation-gated like the + /// outstanding writers. These feed both trace summaries and floor-bias decisions. pub(super) fn publish_slots( &self, peer: &ZakuraPeerId, @@ -263,9 +285,6 @@ impl PeerRegistry { summary.available = summary .available .saturating_add(entry.slots.available_slots); - summary.timeout_recovery = summary - .timeout_recovery - .saturating_add(entry.slots.timeout_recovery_slots); if entry.slots.available_slots == 0 { summary.saturated_peers = summary.saturated_peers.saturating_add(1); } @@ -439,6 +458,43 @@ impl PeerRegistry { .min() } + /// Whether some peer other than `self_peer` is a preferred floor server for + /// `height`: servable for it, holding a free normal (non-bypass) slot, and a + /// better floor server by RTprop. "Better" is strictly lower RTprop, or — when + /// `allow_equal_score` — equal-or-lower. + /// + /// The floor rides the fastest servable carrier. The normal take path passes + /// `allow_equal_score = false`, so this peer defers the floor only to a strictly + /// faster carrier; equal-RTprop carriers all stay eligible and the single-owner + /// work queue assigns one of them. The floor-bypass path passes + /// `allow_equal_score = true`, so a peer whose cwnd is saturated yields its scarce + /// bypass slot to an equal-or-faster peer that can take the floor through normal + /// capacity. Deadlock-free either way: the unique fastest unsaturated server is + /// never preferred over (nothing beats it), and if every servable peer is + /// saturated this returns false and the floor still moves. Unknown RTprop is + /// treated as worst, so a measured peer is never deferred to an unmeasured one. + pub(super) fn floor_has_preferred_unsaturated_server( + &self, + height: block::Height, + self_peer: &ZakuraPeerId, + self_rtprop_ms: Option, + allow_equal_score: bool, + ) -> bool { + let self_score = self_rtprop_ms.unwrap_or(u64::MAX); + let peers = self.lock(); + peers.iter().any(|(peer, entry)| { + if peer == self_peer || !entry.can_serve_with_room(height) { + return false; + } + let other_score = entry.slots.bbr_rtprop_ms.unwrap_or(u64::MAX); + if allow_equal_score { + other_score <= self_score + } else { + other_score < self_score + } + }) + } + /// Snapshot all peer claims for one height. pub(super) fn outstanding_claims_at(&self, height: block::Height) -> Vec { let peers = self.lock(); @@ -462,8 +518,9 @@ impl PeerRegistry { } } - /// Hard-exclude this peer from re-taking `height` until `until`. - pub(super) fn avoid_height_until( + /// Hard-exclude this peer from re-taking `height` until `until` after the + /// floor watchdog force-cancels its stale claim. + pub(super) fn avoid_floor_height_until( &self, peer: &ZakuraPeerId, height: block::Height, @@ -471,12 +528,12 @@ impl PeerRegistry { ) { let mut peers = self.lock(); if let Some(entry) = peers.get_mut(peer) { - entry.retry_avoid.insert(height, until); + entry.floor_watchdog_avoid.insert(height, until); } } - /// Whether this peer is still hard-excluded from `height`. - pub(super) fn is_avoiding_height( + /// Whether the floor watchdog still hard-excludes this peer from `height`. + pub(super) fn is_floor_height_avoided( &self, peer: &ZakuraPeerId, height: block::Height, @@ -486,12 +543,34 @@ impl PeerRegistry { let Some(entry) = peers.get_mut(peer) else { return false; }; - entry.retry_avoid.retain(|_, until| *until > now); + entry.floor_watchdog_avoid.retain(|_, until| *until > now); entry - .retry_avoid + .floor_watchdog_avoid .get(&height) .is_some_and(|until| *until > now) } + + /// The next floor-watchdog hard-exclude expiry for this peer, if any. The + /// routine uses this to wake itself when a registry-owned avoid expires. + pub(super) fn next_floor_avoid_deadline( + &self, + peer: &ZakuraPeerId, + now: Instant, + ) -> Option { + let mut peers = self.lock(); + let entry = peers.get_mut(peer)?; + entry.floor_watchdog_avoid.retain(|_, until| *until > now); + entry.floor_watchdog_avoid.values().min().copied() + } +} + +impl Entry { + fn can_serve_with_room(&self, height: block::Height) -> bool { + self.received_status + && self.servable_low <= height + && height <= self.servable_high + && self.slots.available_slots > 0 + } } /// Aggregated slot diagnostics across peers for the periodic trace row. @@ -500,7 +579,6 @@ pub(super) struct SlotSummary { pub(super) capacity: usize, pub(super) effective_window: usize, pub(super) available: usize, - pub(super) timeout_recovery: usize, pub(super) saturated_peers: usize, pub(super) outstanding_requests: usize, } @@ -521,3 +599,203 @@ pub(super) fn hard_outbound_capacity(max_inflight_requests: u32) -> usize { .expect("u32 max inflight requests fits in usize on supported targets") .min(EFFECTIVE_BS_OUTBOUND_INFLIGHT_PER_PEER) } + +#[cfg(test)] +mod floor_bias_tests { + use super::*; + + fn peer(byte: u8) -> ZakuraPeerId { + ZakuraPeerId::new(vec![byte; 32]).expect("32-byte test peer id is valid") + } + + /// Register `peer` as servable for `[low, high]` with `available` free slots. + fn register_with_rtprop( + reg: &PeerRegistry, + config: &super::super::ZakuraBlockSyncConfig, + peer: &ZakuraPeerId, + low: u32, + high: u32, + available: usize, + bbr_rtprop_ms: Option, + ) { + let generation = reg.admit(peer, ServicePeerDirection::Outbound, config); + reg.upsert_status( + peer, + generation, + BlockSyncStatus { + servable_low: block::Height(low), + servable_high: block::Height(high), + ..BlockSyncStatus::default() + }, + ); + reg.publish_slots( + peer, + generation, + SlotDiagnostics { + available_slots: available, + bbr_rtprop_ms, + ..SlotDiagnostics::default() + }, + ); + } + + fn register( + reg: &PeerRegistry, + config: &super::super::ZakuraBlockSyncConfig, + peer: &ZakuraPeerId, + low: u32, + high: u32, + available: usize, + ) { + register_with_rtprop(reg, config, peer, low, high, available, None); + } + + #[test] + fn bypass_defers_to_an_equal_or_faster_unsaturated_other_server() { + let config = super::super::ZakuraBlockSyncConfig::default(); + let reg = PeerRegistry::new(); + let (a, b) = (peer(1), peer(2)); + // A is saturated; B serves the floor and has a free slot at an equal RTprop. + register_with_rtprop(®, &config, &a, 0, 1000, 0, Some(50)); + register_with_rtprop(®, &config, &b, 0, 1000, 3, Some(50)); + // In the bypass region (include_equal) A defers — B can take the floor through + // its normal capacity, so A keeps its scarce bypass slot… + assert!(reg.floor_has_preferred_unsaturated_server(block::Height(100), &a, Some(50), true)); + // …but B itself has no other unsaturated server (A is saturated), so B bypasses. + assert!(!reg.floor_has_preferred_unsaturated_server( + block::Height(100), + &b, + Some(50), + true + )); + } + + #[test] + fn normal_path_defers_only_to_a_strictly_faster_server() { + let config = super::super::ZakuraBlockSyncConfig::default(); + let reg = PeerRegistry::new(); + let (slow, fast) = (peer(1), peer(2)); + // Both unsaturated; the normal take path (include_equal = false). + register_with_rtprop(®, &config, &slow, 0, 1000, 3, Some(120)); + register_with_rtprop(®, &config, &fast, 0, 1000, 3, Some(40)); + // The slow peer hands the floor up to the strictly-faster carrier… + assert!(reg.floor_has_preferred_unsaturated_server( + block::Height(100), + &slow, + Some(120), + false + )); + // …and the fastest carrier never defers, so the floor always lands somewhere. + assert!(!reg.floor_has_preferred_unsaturated_server( + block::Height(100), + &fast, + Some(40), + false + )); + } + + #[test] + fn normal_path_keeps_equal_carriers_eligible() { + let config = super::super::ZakuraBlockSyncConfig::default(); + let reg = PeerRegistry::new(); + let (a, b) = (peer(1), peer(2)); + // Two equal-RTprop unsaturated carriers: neither defers (strict <), so both stay + // eligible and the single-owner work queue assigns the floor to one of them — + // they never both defer and wedge the floor. + register_with_rtprop(®, &config, &a, 0, 1000, 3, Some(50)); + register_with_rtprop(®, &config, &b, 0, 1000, 3, Some(50)); + assert!(!reg.floor_has_preferred_unsaturated_server( + block::Height(100), + &a, + Some(50), + false + )); + assert!(!reg.floor_has_preferred_unsaturated_server( + block::Height(100), + &b, + Some(50), + false + )); + } + + #[test] + fn saturated_fast_peer_does_not_defer_to_slower_unsaturated_peer() { + let config = super::super::ZakuraBlockSyncConfig::default(); + let reg = PeerRegistry::new(); + let (fast, slow) = (peer(1), peer(2)); + register_with_rtprop(®, &config, &fast, 0, 1000, 0, Some(40)); + register_with_rtprop(®, &config, &slow, 0, 1000, 3, Some(120)); + assert!(!reg.floor_has_preferred_unsaturated_server( + block::Height(100), + &fast, + Some(40), + true + )); + } + + #[test] + fn bypasses_when_every_server_is_saturated() { + let config = super::super::ZakuraBlockSyncConfig::default(); + let reg = PeerRegistry::new(); + let (a, b) = (peer(1), peer(2)); + register(®, &config, &a, 0, 1000, 0); + register(®, &config, &b, 0, 1000, 0); + assert!(!reg.floor_has_preferred_unsaturated_server(block::Height(100), &a, None, true)); + } + + #[test] + fn ignores_an_unsaturated_peer_that_cannot_serve_the_floor() { + let config = super::super::ZakuraBlockSyncConfig::default(); + let reg = PeerRegistry::new(); + let (a, b) = (peer(1), peer(2)); + register(®, &config, &a, 0, 1000, 0); + // B has a free slot but only serves heights 500..=1000 — it cannot take a floor + // request at height 100, so A must still bypass. + register(®, &config, &b, 500, 1000, 3); + assert!(!reg.floor_has_preferred_unsaturated_server(block::Height(100), &a, None, true)); + } + + #[test] + fn floor_avoid_deadline_prunes_expired_entries_and_returns_next_wake() { + let config = super::super::ZakuraBlockSyncConfig::default(); + let reg = PeerRegistry::new(); + let peer = peer(1); + reg.admit(&peer, ServicePeerDirection::Outbound, &config); + let now = Instant::now(); + + reg.avoid_floor_height_until( + &peer, + block::Height(1), + now - std::time::Duration::from_secs(1), + ); + reg.avoid_floor_height_until( + &peer, + block::Height(2), + now + std::time::Duration::from_secs(2), + ); + reg.avoid_floor_height_until( + &peer, + block::Height(3), + now + std::time::Duration::from_secs(1), + ); + + assert_eq!( + reg.next_floor_avoid_deadline(&peer, now), + Some(now + std::time::Duration::from_secs(1)), + ); + assert!(!reg.is_floor_height_avoided(&peer, block::Height(1), now)); + assert!(reg.is_floor_height_avoided(&peer, block::Height(2), now)); + } + + #[test] + fn parked_peer_expires_after_cooldown() { + let reg = PeerRegistry::new(); + let peer = peer(1); + let now = Instant::now(); + + reg.park_peer_until(&peer, now + std::time::Duration::from_secs(1)); + + assert!(reg.is_peer_parked(&peer, now)); + assert!(!reg.is_peer_parked(&peer, now + std::time::Duration::from_secs(2))); + } +} diff --git a/zebra-network/src/zakura/block_sync/peer_routine.rs b/zebra-network/src/zakura/block_sync/peer_routine.rs index 195e8da4754..5824f1bb3e3 100644 --- a/zebra-network/src/zakura/block_sync/peer_routine.rs +++ b/zebra-network/src/zakura/block_sync/peer_routine.rs @@ -28,18 +28,21 @@ use tokio_util::sync::CancellationToken; use super::events::RoutineToReactor; use super::{ - admission::{admission_decision, floor_rescue_high, AdmissionSnapshot, RequestPriority}, + admission::{ + admission_decision, floor_rescue_high, request_deadline, AdmissionSnapshot, RequestPriority, + }, peer_registry::{hard_outbound_capacity, PeerRegistry}, pipe::block_sync_guard, reactor::{ - block_sync_message_label, bs_insert_height, bs_insert_peer, bs_insert_u64, tolerated_bytes, + block_sync_message_label, bs_insert_height, bs_insert_peer, bs_insert_str, bs_insert_u64, + tolerated_bytes, }, reorder::BufferedBlockBody, request::{BlockRangeRequest, ExpectedBlock}, sequencer_task::{SequencedBody, SequencerControlInput, SequencerView}, state::{ - next_height, DownloadWindow, OutstandingBlockRange, ReceivedBlockTracker, ThroughputMeter, - TimeoutBackoffOutcome, + next_height, DownloadWindow, LivenessOutcome, OutstandingBlockRange, ReceivedBlockTracker, + ThroughputMeter, }, work_queue::{WorkItem, WorkQueue}, BlockSyncAction, BlockSyncMessage, BlockSyncMisbehavior, BlockSyncPeerSession, BlockSyncStatus, @@ -64,6 +67,7 @@ use zebra_chain::{block, serialization::ZcashSerialize}; const RETRY_AVOID_BACKOFF: Duration = Duration::from_millis(50); /// Poll interval while this peer's outbound stream queue is full. const OUTBOUND_FULL_POLL_INTERVAL: Duration = Duration::from_millis(10); +const CLOSE_BLOCK_SYNC_NO_BLOCK_PROGRESS: &str = "block_sync_no_block_progress"; fn is_block_frame(frame: &crate::zakura::Frame) -> bool { frame.payload.first().copied() == Some(MSG_BS_BLOCK) @@ -255,18 +259,18 @@ impl PeerRoutine { if self.session.outbound_capacity() > 0 { self.try_fill().await; } - let outbound_has_capacity = self.session.outbound_capacity() > 0; + let outbound_queue_has_capacity = self.session.outbound_capacity() > 0; // Sleep until the earliest outstanding deadline (own-timeout arm). let timeout = self.earliest_deadline_sleep(); tokio::pin!(timeout); - let outbound_capacity = time::sleep(OUTBOUND_FULL_POLL_INTERVAL); - tokio::pin!(outbound_capacity); + let outbound_queue_poll = time::sleep(OUTBOUND_FULL_POLL_INTERVAL); + tokio::pin!(outbound_queue_poll); tokio::select! { biased; _ = self.cancel.cancelled() => return Ok(()), - frame = self.recv.recv(), if outbound_has_capacity => { + frame = self.recv.recv(), if outbound_queue_has_capacity => { match frame { // Decode the frame and run the download/serving dispatch // in this same task. A protocol reject propagates out so @@ -285,14 +289,14 @@ impl PeerRoutine { Err(_) => return Ok(()), } } - _ = &mut timeout => self.expire_due_timeouts(Instant::now())?, + _ = &mut timeout => self.handle_deadlines(Instant::now()).await?, _ = &mut capacity => { self.trace_wake("budget_capacity"); } _ = &mut available => { self.trace_wake("work_added"); } - _ = &mut outbound_capacity, if !outbound_has_capacity => {} + _ = &mut outbound_queue_poll, if !outbound_queue_has_capacity => {} } } } @@ -478,15 +482,17 @@ impl PeerRoutine { self.retry_avoid.clear(); // Clear our (now-empty) registry outstanding and refresh slot diagnostics. self.publish_outstanding(); + self.window.disarm_liveness_if_idle(); // The want-work loop re-fans from the queue at the top of the next // iteration (the `reset_above` + producer re-query repopulate `pending`). } /// Sleep future resolving at the earliest wake the routine schedules for - /// itself: the soonest outstanding request deadline (own-timeout) **or** the - /// soonest retry-avoid expiry (so a routine that quiet-returned its only work - /// re-runs want-work once the bias lifts, even if no external event arrives). - /// Defaults to a long idle sleep when neither exists. + /// itself: the soonest outstanding request deadline (own-timeout), block + /// liveness deadline, **or** the soonest retry-avoid expiry (local failure bias + /// or registry-owned floor-watchdog hard exclude), so a routine that quiet-returned + /// its only work re-runs want-work once the bias lifts even if no external event + /// arrives. Defaults to a long idle sleep when none exists. fn earliest_deadline_sleep(&self) -> time::Sleep { let now = Instant::now(); let earliest_deadline = self @@ -495,11 +501,18 @@ impl PeerRoutine { .iter() .map(|outstanding| outstanding.deadline) .min(); - let earliest_avoid = self.retry_avoid.values().min().copied(); - let earliest = match (earliest_deadline, earliest_avoid) { - (Some(a), Some(b)) => Some(a.min(b)), - (only, None) | (None, only) => only, - }; + let liveness_deadline = self.window.block_liveness_deadline; + let local_retry_avoid = self.retry_avoid.values().min().copied(); + let floor_watchdog_avoid = self.registry.next_floor_avoid_deadline(&self.peer, now); + let earliest = [ + earliest_deadline, + liveness_deadline, + local_retry_avoid, + floor_watchdog_avoid, + ] + .into_iter() + .flatten() + .min(); match earliest { // Floor the wait at the deadline so a far-future request still wakes // promptly; an already-due deadline wakes immediately. @@ -517,15 +530,11 @@ impl PeerRoutine { /// There is no floor gate: downloads are governed by the byte budget and /// per-peer slots, never floor-distance / near-tip lag. async fn try_fill(&mut self) { - // Reconcile the adaptive window's hard cap with the peer's currently - // advertised `max_inflight_requests` (it may have grown/shrunk via a - // `Status`; `handle_status` set `window.max_inflight_requests`), clamping - // the window / recovery slots to the new hard capacity. - let hard = self.window.hard_outbound_capacity(); - self.window.outbound_request_window = self.window.outbound_request_window.min(hard).max(1); - self.window.timeout_recovery_slots = self.window.timeout_recovery_slots.min(hard); - // GC this routine's own fully-covered outstanding requests: when the - // download floor passes the end of a request, its bodies are no longer + // The BBR cwnd is clamped to the peer's advertised hard cap inside + // `available_slots`, so there is no separate window to reconcile on a + // `Status` change. + // GC this routine's own fully-committed outstanding requests: when the + // committed floor passes the end of a request, its bodies are no longer // needed, so release its reservation and free its slot promptly rather // than waiting for the request's own timeout. This GCs *our own* covered // requests; it is never a fetch throttle and never churns other peers (a @@ -536,102 +545,110 @@ impl PeerRoutine { // routine again. let now = Instant::now(); self.retry_avoid.retain(|_, until| *until > now); - loop { - if !self.received_status || self.window.available_slots() == 0 { - break; + // Count requests issued this pass and capture *why* the fill loop stops, so a + // trace can attribute carrier idle ("bubble") time to a cause. The loop yields a + // `&'static str` reason via `break`; a pass that issues nothing (`fill_sent == 0`) + // is a candidate bubble. + let mut fill_sent = 0u32; + let fill_stop: &'static str = loop { + let floor_bonus = usize::try_from(self.config.floor_bypass_slots).unwrap_or(0); + let normal_slots = self.window.available_slots(); + let floor_slots = self.window.available_slots_with_bonus(floor_bonus); + // Break only when even a bypassed floor request has no slot. A cwnd that is + // saturated for above-floor work (`normal_slots == 0`) still leaves up to + // `floor_bonus` slots so the lowest missing height keeps moving. + if !self.received_status { + break "no_status"; + } + if floor_slots == 0 { + break "cwnd_saturated"; } - // One contiguous chunk up to the peer's per-request count cap; the - // outer loop fills the rest of the peer's slots. - let local_peer_count_cap = usize::try_from( - self.max_blocks_per_response - .min(self.config.advertised_max_blocks_per_response()) - .max(1), - ) - .unwrap_or(usize::MAX); + let in_bypass = normal_slots == 0; let (servable_low, servable_high) = (self.servable_low, self.servable_high); // Compute this chunk's count and byte ceiling before taking any work. // The count cap is the peer/request cap; the byte cap is enforced by // the budgeted work-queue take and then by the reservation below. - let max_count = local_peer_count_cap; + let max_count = self.request_count_cap(); let response_byte_cap = u64::from(self.max_response_bytes.max(1)); let view = *self.sequencer_view.borrow(); let floor_high = floor_rescue_high(view.download_floor); let mut request_priority = RequestPriority::Floor; - let (reserved_above_floor_bytes, reserved_above_floor_blocks) = - self.work.reserved_above(view.download_floor); - let mut items = if servable_low <= floor_high + let reserved_above_floor = self.work.reserved_above(view.download_floor); + // The floor rides the fastest servable carrier: defer it whenever a + // preferred peer can take it. Outside the bypass region only a strictly + // faster carrier makes this peer defer (equal carriers stay eligible, so a + // slow peer hands the floor up but two equal carriers both contest it); in + // the bypass region (this peer's cwnd is saturated) an equal-RTprop peer is + // preferred too, so a scarce bypass slot is only spent when no + // equal-or-faster peer can take the floor normally. If every servable peer + // is saturated this is `false` and the floor still moves. + let floor_arm_allowed = !self.registry.floor_has_preferred_unsaturated_server( + view.download_floor, + &self.peer, + self.window.bbr_rtprop_ms(), + in_bypass, + ); + let mut items = if floor_arm_allowed + && servable_low <= floor_high && self .work .first_pending_in_range(servable_low, servable_high.min(floor_high)) .is_some() { - let floor_available = self.budget.available().min(response_byte_cap); - if floor_available == 0 { - Vec::new() - } else { - let floor_take_high = match next_height(floor_high).and_then(|tail_start| { - admission_decision( - &self.config, - AdmissionSnapshot { - download_floor: view.download_floor, - reorder_buffered_bytes: view.reorder_buffered_bytes, - reorder_buffered_blocks: view.reorder_len, - applying_buffered_bytes: view.applying_buffered_bytes, - applying_buffered_blocks: view.applying_len, - sequencer_input_queued_bytes: self - .sequencer_input_bytes - .load(std::sync::atomic::Ordering::Relaxed), - reserved_above_floor_bytes, - reserved_above_floor_blocks, - budget_available: self.budget.available(), - }, - tail_start, - response_byte_cap, - ) - }) { - Some(_) => servable_high, - None => servable_high.min(floor_high), - }; - self.work.take_in_range_budgeted( - servable_low, - floor_take_high, - max_count, - floor_available, + // Size the floor take by the live budget as usual, but never below one + // byte. `take_in_range_budgeted` always takes its first item regardless of + // the byte cap, so a `>= 1` cap guarantees the floor block itself is taken + // even when the budget is exactly full — which reaches + // `reserve_request_budget`'s floor path, whose `FundFloorReservation` sheds + // an above-floor reorder body to fund the floor. The bare `budget.available()` + // cap collapsed to an *empty* take at `available() == 0`, breaking the fill + // loop before the funding path and wedging the floor permanently. The + // `.min(response_byte_cap)` still bounds the speculative above-floor tail to + // the live budget, so a funded floor request never over-commits. + let floor_available = self.budget.available().min(response_byte_cap).max(1); + let floor_take_high = match next_height(floor_high).and_then(|tail_start| { + admission_decision( + &self.config, + self.admission_snapshot(view, reserved_above_floor), + tail_start, + response_byte_cap, ) - } + }) { + Some(_) => servable_high, + None => servable_high.min(floor_high), + }; + self.work.take_in_range_budgeted( + servable_low, + floor_take_high, + max_count, + floor_available, + ) } else { Vec::new() }; if items.is_empty() { + if in_bypass { + // Saturated cwnd: the floor bypass funds the floor only, never a + // speculative above-floor fetch. Nothing more to take this pass. + break "cwnd_saturated"; + } let Some(start_height) = self .work .first_pending_in_range(servable_low, servable_high) else { - break; + break "no_work"; }; let Some(decision) = admission_decision( &self.config, - AdmissionSnapshot { - download_floor: view.download_floor, - reorder_buffered_bytes: view.reorder_buffered_bytes, - reorder_buffered_blocks: view.reorder_len, - applying_buffered_bytes: view.applying_buffered_bytes, - applying_buffered_blocks: view.applying_len, - sequencer_input_queued_bytes: self - .sequencer_input_bytes - .load(std::sync::atomic::Ordering::Relaxed), - reserved_above_floor_bytes, - reserved_above_floor_blocks, - budget_available: self.budget.available(), - }, + self.admission_snapshot(view, reserved_above_floor), start_height, response_byte_cap, ) else { metrics::gauge!("sync.block.backlog.at_cap").set(1.0); - break; + break "lookahead_cap"; }; if decision.priority == RequestPriority::AboveFloor { @@ -646,7 +663,7 @@ impl PeerRoutine { } } if items.is_empty() { - break; + break "no_work"; } // Peer-local retry bias: if the contiguous chunk we just took leads // with heights this routine recently *failed* (RangeUnavailable / @@ -660,7 +677,9 @@ impl PeerRoutine { { let keep_from = items.iter().position(|(height, _)| { !self.retry_avoid.contains_key(height) - && !self.registry.is_avoiding_height(&self.peer, *height, now) + && !self + .registry + .is_floor_height_avoided(&self.peer, *height, now) }); match keep_from { Some(0) => {} @@ -671,7 +690,7 @@ impl PeerRoutine { None => { let avoided: Vec<_> = items.iter().map(|(h, _)| *h).collect(); self.work.return_items_quiet(avoided); - break; + break "retry_avoid"; } } } @@ -691,7 +710,7 @@ impl PeerRoutine { .await { self.return_taken_items(&items); - break; + break "budget"; } let marked = self .work @@ -701,7 +720,7 @@ impl PeerRoutine { let _ = self .work .release_and_return_items(items.iter().map(|(height, _)| *height)); - break; + break "internal"; } let count = match u32::try_from(kept_count) { @@ -711,7 +730,7 @@ impl PeerRoutine { .work .release_and_return_items(items.iter().map(|(height, _)| *height)); self.budget.release(released); - break; + break "internal"; } }; let request = BlockRangeRequest { @@ -750,15 +769,25 @@ impl PeerRoutine { .release_and_return_items(items.iter().map(|(height, _)| *height)); self.budget.release(released); if matches!(error, OrderedSendError::Full) { - break; + break "outbound_full"; } self.session.cancel_token().cancel(); - break; + break "send_error"; } - let deadline = queued_at + self.config.request_timeout; + let deadline = request_deadline( + request_priority, + queued_at, + self.config.request_timeout, + self.config.effective_floor_rescue_timeout(), + reserved_bytes, + self.window.bbr_btlbw_bytes_per_sec(), + ); metrics::counter!("sync.block.request.sent").increment(1); - self.window.record_outbound_request_scheduled(); + if in_bypass { + // A floor request borrowed a bypass slot while the cwnd was saturated. + metrics::counter!("sync.block.request.floor_bypass").increment(1); + } let request_start_height = request.start_height; let request_count = request.count; let request_estimated_bytes = request.estimated_bytes; @@ -766,14 +795,28 @@ impl PeerRoutine { request, queued_at, deadline, + delivery_snapshot: self.window.delivery_snapshot(queued_at), + delivered_bytes: 0, received: ReceivedBlockTracker::default(), }); + self.window + .arm_liveness(queued_at, self.config.effective_liveness_timeout()); self.publish_outstanding(); self.trace_get_blocks_sent( request_start_height, request_count, request_estimated_bytes, + in_bypass, ); + fill_sent = fill_sent.saturating_add(1); + }; + // Attribute this pass's stop. A pass that issued nothing is a candidate bubble; + // the reason + the live slot/budget/work snapshot let a trace tell a legitimate + // stop (no_work with empty queue, cwnd_saturated) from a recoverable one (slots + + // budget + work all free, stopped anyway). + metrics::counter!("sync.block.fill_stop", "reason" => fill_stop).increment(1); + if fill_sent == 0 { + self.trace_fill_stop(fill_stop); } // If pending work is running low, ping the reactor to re-query (the @@ -785,6 +828,36 @@ impl PeerRoutine { } } + fn admission_snapshot( + &self, + view: SequencerView, + reserved_above_floor: (u64, u64), + ) -> AdmissionSnapshot { + let (reserved_above_floor_bytes, reserved_above_floor_blocks) = reserved_above_floor; + AdmissionSnapshot { + download_floor: view.download_floor, + reorder_buffered_bytes: view.reorder_buffered_bytes, + reorder_buffered_blocks: view.reorder_len, + applying_buffered_bytes: view.applying_buffered_bytes, + applying_buffered_blocks: view.applying_len, + sequencer_input_queued_bytes: self + .sequencer_input_bytes + .load(std::sync::atomic::Ordering::Relaxed), + reserved_above_floor_bytes, + reserved_above_floor_blocks, + budget_available: self.budget.available(), + } + } + + fn request_count_cap(&self) -> usize { + usize::try_from( + self.max_blocks_per_response + .min(self.config.advertised_max_blocks_per_response()) + .max(1), + ) + .unwrap_or(usize::MAX) + } + async fn reserve_request_budget( &mut self, priority: RequestPriority, @@ -849,7 +922,15 @@ impl PeerRoutine { // ===================== own-timeout arm (ports `expire_due_timeouts`) ===== - fn expire_due_timeouts(&mut self, now: Instant) -> Result<(), SinkReject> { + async fn handle_deadlines(&mut self, now: Instant) -> Result<(), SinkReject> { + let rescued_timed_out = self.expire_due_timeouts(now); + if rescued_timed_out && self.session.outbound_capacity() > 0 { + self.try_fill().await; + } + self.check_block_liveness(now) + } + + fn expire_due_timeouts(&mut self, now: Instant) -> bool { let mut timed_out = Vec::new(); let mut index = 0; while index < self.window.outstanding.len() { @@ -860,9 +941,9 @@ impl PeerRoutine { } } if timed_out.is_empty() { - return Ok(()); + return false; } - let backoff = self.window.reduce_outbound_window_after_timeout(); + self.window.record_timeout(); for outstanding in &timed_out { // Return only the unreceived heights — received ones are buffered (in // `in_flight` until committed); re-queuing them would re-fetch a body @@ -875,16 +956,41 @@ impl PeerRoutine { let timed_out_heights: Vec<_> = timed_out.iter().flat_map(unreceived_heights).collect(); self.note_retry_avoid(timed_out_heights); self.publish_outstanding(); - if backoff == TimeoutBackoffOutcome::DisconnectPeer { - tracing::debug!( - peer = ?self.peer, - "disconnecting Zakura block-sync peer after repeated timeouts at minimum window" - ); - return Err(SinkReject::protocol( - "block-sync peer repeatedly timed out at minimum window", - )); + true + } + + fn check_block_liveness(&mut self, now: Instant) -> Result<(), SinkReject> { + match self.window.check_liveness(now) { + LivenessOutcome::Ok => Ok(()), + LivenessOutcome::Disarm => { + self.window.disarm_liveness_if_idle(); + Ok(()) + } + LivenessOutcome::Disconnect if self.session.outbound_capacity() == 0 => { + // This is local ordered-stream backpressure, not the peer's request + // window. While the outbound queue is full, the select loop above + // does not drain inbound frames, so a useful block may already be + // waiting behind our own write-side congestion. + self.window.block_liveness_deadline = + Some(now + self.config.effective_liveness_timeout()); + Ok(()) + } + LivenessOutcome::Disconnect => { + let error = + "block-sync peer made no accepted block progress before liveness deadline"; + self.registry.park_peer_until( + &self.peer, + now + self.config.effective_no_progress_peer_cooldown(), + ); + self.trace_protocol_reject_liveness(error); + tracing::debug!( + peer = ?self.peer, + outstanding = self.window.outstanding.len(), + "disconnecting Zakura block-sync peer after no accepted block progress" + ); + Err(SinkReject::protocol(error)) + } } - Ok(()) } /// Drop this routine's outstanding requests whose whole range is at or below @@ -915,6 +1021,7 @@ impl PeerRoutine { } if removed { self.publish_outstanding(); + self.window.disarm_liveness_if_idle(); } } @@ -966,6 +1073,7 @@ impl PeerRoutine { return; }; let outstanding = &self.window.outstanding[index]; + let delivery_snapshot = outstanding.delivery_snapshot; if outstanding.has_received(height) { tracing::debug!(peer = ?self.peer, ?height, "ignoring duplicate block-sync body frame"); return; @@ -990,7 +1098,8 @@ impl PeerRoutine { let estimated_bytes = outstanding.estimated_bytes_for_height(height).unwrap_or(0); let request_start_height = outstanding.request.start_height; let request_range_count = outstanding.request.count; - let request_elapsed_ms = elapsed_ms_u64(outstanding.queued_at.elapsed()); + let request_elapsed = outstanding.queued_at.elapsed(); + let request_elapsed_ms = elapsed_ms_u64(request_elapsed); // The body's transactions are not validated against the header here; // consensus does it on apply (`handle_block_apply_finished` attributes a @@ -1050,14 +1159,28 @@ impl PeerRoutine { Some(request_elapsed_ms), ); + self.window + .note_block_progress(Instant::now(), self.config.effective_liveness_timeout()); let mut completed = None; if let Some(outstanding) = self.window.outstanding.get_mut(index) { + outstanding.record_body_bytes(serialized_bytes); outstanding.mark_received(height); if outstanding.is_complete() { completed = Some(self.window.outstanding.remove(index)); - self.window.increase_outbound_window_after_success(); } } + if let Some(outstanding) = &completed { + // Feed the BBR estimators on request completion: the round-trip (RTprop) + // and the per-ack delivery rate (BtlBw) for this request's block count and + // delivered bytes. + self.window.record_delivery( + Instant::now(), + request_elapsed, + request_range_count, + outstanding.delivered_bytes, + delivery_snapshot, + ); + } if let Some(outstanding) = completed { self.finish_detached(outstanding, Disposition::Satisfied); } else { @@ -1419,6 +1542,7 @@ impl PeerRoutine { } } self.publish_outstanding(); + self.window.disarm_liveness_if_idle(); } fn return_unreceived_to_queue(&self, outstanding: &OutstandingBlockRange) -> u64 { @@ -1467,18 +1591,18 @@ impl PeerRoutine { self.registry .set_outstanding(&self.peer, self.generation, map); } - // Publish the window slot diagnostics for the reactor's periodic trace row - // (trace only; the reactor lost direct visibility into the routine window). + // Publish the window diagnostics for the reactor's periodic trace row and + // for other routines' cross-peer floor-bias decisions. let hard_capacity = hard_outbound_capacity(self.window.max_inflight_requests); self.registry.publish_slots( &self.peer, self.generation, super::peer_registry::SlotDiagnostics { hard_capacity, - effective_window: hard_capacity.min(self.window.outbound_request_window), + effective_window: self.window.bbr_effective_cwnd().min(hard_capacity), available_slots: self.window.available_slots(), - timeout_recovery_slots: self.window.timeout_recovery_slots, outstanding_requests: self.window.outstanding.len(), + bbr_rtprop_ms: self.window.bbr_rtprop_ms(), }, ); } @@ -1537,9 +1661,45 @@ impl PeerRoutine { }); } - /// Trace a decoded inbound message in the routine that decoded it. Records the - /// message kind only; the per-variant field detail lives on the reactor's - /// heavier trace path. + fn trace_protocol_reject_liveness(&self, error: &str) { + self.emit(bs_trace::BLOCK_PEER_PROTOCOL_REJECT, |row| { + bs_insert_peer(row, bs_trace::PEER, &self.peer); + row.insert( + bs_trace::REASON.to_string(), + serde_json::Value::String(CLOSE_BLOCK_SYNC_NO_BLOCK_PROGRESS.to_string()), + ); + row.insert( + bs_trace::ERROR.to_string(), + serde_json::Value::String(error.to_string()), + ); + bs_insert_u64( + row, + bs_trace::OUTSTANDING, + u64::try_from(self.window.outstanding.len()).unwrap_or(u64::MAX), + ); + bs_insert_u64( + row, + "bbr_cwnd", + u64::try_from(self.window.bbr_effective_cwnd()).unwrap_or(u64::MAX), + ); + bs_insert_u64( + row, + "available_slots", + u64::try_from(self.window.available_slots()).unwrap_or(u64::MAX), + ); + if let Some(last_block_at) = self.window.last_block_at { + bs_insert_u64( + row, + "last_block_age_ms", + elapsed_ms_u64(last_block_at.elapsed()), + ); + } + }); + } + + /// Trace a decoded inbound message (the previous reactor's `trace_message_received`, + /// now emitted in the routine that decoded it). Records the message kind only; + /// the per-variant field detail lives on the reactor's heavier trace path. fn trace_message_received(&self, msg: &BlockSyncMessage) { self.emit(bs_trace::BLOCK_MESSAGE_RECEIVED, |row| { row.insert( @@ -1551,6 +1711,7 @@ impl PeerRoutine { fn trace_status_received(&self, status: BlockSyncStatus) { self.emit(bs_trace::BLOCK_STATUS_RECEIVED, |row| { + bs_insert_peer(row, bs_trace::PEER, &self.peer); bs_insert_height(row, "servable_low", status.servable_low); bs_insert_height(row, "servable_high", status.servable_high); }); @@ -1564,7 +1725,35 @@ impl PeerRoutine { }); } - fn trace_get_blocks_sent(&self, start_height: block::Height, count: u32, estimated_bytes: u64) { + /// Emitted when a `try_fill` pass issued no request (a candidate carrier "bubble"). + /// The reason plus the live slot/budget/work snapshot let a trace tell a legitimate + /// idle (`no_work` with an empty queue, `cwnd_saturated`) from a recoverable one + /// (slots + budget + work all free yet stopped — a wakeup gap to fix). + fn trace_fill_stop(&self, reason: &'static str) { + let floor_bonus = usize::try_from(self.config.floor_bypass_slots).unwrap_or(0); + self.emit(bs_trace::BLOCK_FILL_STOP, |row| { + bs_insert_peer(row, bs_trace::PEER, &self.peer); + bs_insert_str(row, bs_trace::FILL_STOP_REASON, reason); + bs_insert_u64(row, bs_trace::FILL_SENT, 0); + bs_insert_u64(row, "normal_slots", self.window.available_slots() as u64); + bs_insert_u64( + row, + "floor_slots", + self.window.available_slots_with_bonus(floor_bonus) as u64, + ); + bs_insert_u64(row, "budget_available", self.budget.available()); + bs_insert_u64(row, "pending_work", self.work.pending_len() as u64); + bs_insert_u64(row, "received_status", u64::from(self.received_status)); + }); + } + + fn trace_get_blocks_sent( + &self, + start_height: block::Height, + count: u32, + estimated_bytes: u64, + floor_bypass: bool, + ) { self.emit(bs_trace::BLOCK_GET_BLOCKS_SENT, |row| { bs_insert_peer(row, bs_trace::PEER, &self.peer); bs_insert_height(row, bs_trace::RANGE_START, start_height); @@ -1576,6 +1765,9 @@ impl PeerRoutine { "peer_outstanding", self.window.outstanding.len() as u64, ); + // A floor request issued while the peer was saturated at its cwnd — borrowed + // a floor-bypass slot. Lets the analysis confirm the bypass actually fired. + bs_insert_u64(row, "floor_bypass", u64::from(floor_bypass)); }); } @@ -1611,6 +1803,36 @@ impl PeerRoutine { if let Some(request_elapsed_ms) = request_elapsed_ms { bs_insert_u64(row, "request_elapsed_ms", request_elapsed_ms); } + bs_insert_u64( + row, + "bbr_cwnd", + u64::try_from(self.window.bbr_effective_cwnd()).unwrap_or(u64::MAX), + ); + if let Some(rtprop_ms) = self.window.bbr_rtprop_ms() { + bs_insert_u64(row, "bbr_rtprop_ms", rtprop_ms); + } + if let Some(btlbw) = self.window.bbr_btlbw_milliblocks() { + bs_insert_u64(row, "bbr_btlbw_milliblocks_per_sec", btlbw); + } + // Byte-denomination fields (emitted only under `CwndUnit::Bytes`): the byte + // cwnd, the bytes/sec BtlBw, and the in-flight reserved bytes. `bbr_cwnd` + // above stays the derived in-flight *request* count so existing analysis + // scripts keep working in either unit. + if let Some(cwnd_bytes) = self.window.bbr_effective_cwnd_bytes() { + bs_insert_u64(row, "bbr_cwnd_bytes", cwnd_bytes); + bs_insert_u64(row, "bbr_inflight_bytes", self.window.bbr_inflight_bytes()); + } + if let Some(btlbw_bytes) = self.window.bbr_btlbw_bytes_per_sec() { + bs_insert_u64(row, "bbr_btlbw_bytes_per_sec", btlbw_bytes); + } + bs_insert_u64(row, "bbr_delivered", self.window.bbr_delivered()); + bs_insert_u64(row, "bbr_phase", self.window.bbr_phase_code()); + if let Some(smoothed_ms) = self.window.bbr_smoothed_elapsed_ms() { + bs_insert_u64(row, "bbr_smoothed_elapsed_ms", smoothed_ms); + } + if let Some(delay_cap) = self.window.bbr_delay_cap() { + bs_insert_u64(row, "bbr_delay_cap", delay_cap); + } }); } @@ -1728,3 +1950,127 @@ impl Drop for PeerRoutine { self.registry.clear_outstanding(&self.peer, self.generation); } } + +#[cfg(test)] +mod tests { + use std::sync::atomic::AtomicU64; + use std::sync::{Arc, Mutex}; + use std::time::{Duration, Instant}; + + use tokio::sync::{mpsc, watch}; + use tokio::time::timeout; + use tokio_util::sync::CancellationToken; + use zebra_chain::block; + + use super::super::peer_registry::PeerRegistry; + use super::super::request::BlockSizeEstimate; + use super::super::sequencer_task::{initial_view, SequencerControlInput}; + use super::super::state::{ByteBudget, ThroughputMeter}; + use super::super::work_queue::WorkQueue; + use super::super::{BlockSyncFrontiers, BlockSyncPeerSession, ZakuraBlockSyncConfig}; + use super::PeerRoutine; + use crate::zakura::framed_channel; + use crate::zakura::trace::ZakuraTrace; + use crate::zakura::ZakuraPeerId; + + /// A floor request whose byte reservation cannot be met must still reach the + /// sequencer's floor-funding path so the rescue shed can free room — even when the + /// byte budget is *exactly* full. + /// + /// Regression guard for the wedge where `try_fill`'s floor arm sized its take by + /// `budget.available()`: at `available() == 0` the take came back empty, the fill + /// loop broke, and `reserve_request_budget` (the only caller that emits + /// `FundFloorReservation`) was never reached — so the shed that would rescue the + /// floor never fired and the floor wedged permanently. The fix sizes the floor take + /// by one response and lets the reservation shed; here we assert the funding request + /// is emitted with a non-zero need. + #[tokio::test] + async fn exhausted_budget_floor_request_still_reaches_the_funding_path() { + let config = ZakuraBlockSyncConfig::default(); + + // A byte budget reserved down to exactly zero free: the case that used to wedge. + let mut budget = ByteBudget::new(8_192); + assert!(budget.try_reserve(8_192)); + assert_eq!(budget.available(), 0, "the budget is exactly full"); + + // The floor height (1) is pending and servable by this peer; the download floor + // is 0 so height 1 is the floor. + let work = Arc::new(WorkQueue::new(block::Height(0))); + assert_eq!( + work.extend([( + block::Height(1), + block::Hash([1; 32]), + BlockSizeEstimate::Advertised(1_000), + )]), + 1, + ); + + let cancel = CancellationToken::new(); + let (out_send, _out_recv) = framed_channel(16); + let (_in_send, in_recv) = framed_channel(16); + let peer = ZakuraPeerId::new(vec![7u8; 32]).expect("test peer id is within bounds"); + let session = BlockSyncPeerSession::for_test(peer.clone(), out_send, cancel.clone()); + + let (sequencer_input_tx, _sequencer_input_rx) = mpsc::channel(16); + let (control_tx, mut control_rx) = mpsc::unbounded_channel(); + let (actions_tx, _actions_rx) = mpsc::channel(16); + let (routine_to_reactor_tx, _routine_to_reactor_rx) = mpsc::channel(16); + let (_view_tx, view_rx) = watch::channel(initial_view(BlockSyncFrontiers { + finalized_height: block::Height(0), + verified_block_tip: block::Height(0), + verified_block_hash: block::Hash([0; 32]), + })); + + let mut routine = PeerRoutine::new( + peer, + session, + in_recv, + config, + 0, + budget, + work, + Arc::new(PeerRegistry::new()), + Arc::new(Mutex::new(ThroughputMeter::new(Instant::now()))), + sequencer_input_tx, + Arc::new(AtomicU64::new(0)), + control_tx, + actions_tx, + routine_to_reactor_tx, + view_rx, + cancel, + ZakuraTrace::noop(), + ); + // The routine learns these from a `Status` frame in production; set them directly + // so a single `try_fill` pass exercises the floor arm. + routine.received_status = true; + routine.servable_low = block::Height(1); + routine.servable_high = block::Height(10); + + let fill = tokio::spawn(async move { + routine.try_fill().await; + }); + + let message = timeout(Duration::from_secs(5), control_rx.recv()) + .await + .expect("an exhausted floor must request funding within the timeout") + .expect("the sequencer-control channel stays open"); + match message { + SequencerControlInput::FundFloorReservation { + needed_bytes, + reply, + } => { + assert!( + needed_bytes > 0, + "the floor reservation funds a non-zero request", + ); + // No reorder body to shed in this unit-level test: deny the funding. The + // routine returns the taken floor height and exits the pass cleanly. + let _ = reply.send(false); + } + other => panic!("expected FundFloorReservation, got {other:?}"), + } + + fill.await + .expect("try_fill completes after the funding decision"); + } +} diff --git a/zebra-network/src/zakura/block_sync/reactor.rs b/zebra-network/src/zakura/block_sync/reactor.rs index 61b8494ed11..3a9ac9bed38 100644 --- a/zebra-network/src/zakura/block_sync/reactor.rs +++ b/zebra-network/src/zakura/block_sync/reactor.rs @@ -385,7 +385,7 @@ impl BlockSyncReactor { self.registry .clear_outstanding_height(&claim.peer, claim.height); if servable_peers > 2 { - self.registry.avoid_height_until( + self.registry.avoid_floor_height_until( &claim.peer, claim.height, now + self.startup.config.effective_floor_peer_avoid_cooldown(), @@ -1465,7 +1465,6 @@ impl BlockSyncReactor { let slot_capacity = slots.capacity; let slot_effective_window = slots.effective_window; let slot_available = slots.available; - let slot_timeout_recovery = slots.timeout_recovery; let slot_saturated_peers = slots.saturated_peers; let counts = self.registry.direction_status_counts(); let inbound_peers = counts.inbound; @@ -1631,11 +1630,6 @@ impl BlockSyncReactor { slot_effective_window as u64, ); bs_insert_u64(row, "request_slot_available", slot_available as u64); - bs_insert_u64( - row, - "request_slot_timeout_recovery", - slot_timeout_recovery as u64, - ); bs_insert_u64( row, "request_slot_saturated_peers", @@ -2215,7 +2209,7 @@ fn block_misbehavior_label(reason: BlockSyncMisbehavior) -> &'static str { } } -fn bs_insert_str( +pub(super) fn bs_insert_str( row: &mut serde_json::Map, key: &'static str, value: &str, diff --git a/zebra-network/src/zakura/block_sync/service.rs b/zebra-network/src/zakura/block_sync/service.rs index 3506a6bd9dd..11f342f3706 100644 --- a/zebra-network/src/zakura/block_sync/service.rs +++ b/zebra-network/src/zakura/block_sync/service.rs @@ -3,7 +3,10 @@ use crate::zakura::{ handle_pipe_exit, spawn_supervised_pipe, FramedRecv, FramedSend, OrderedSendError, Peer, PeerStreamSession, Service, SinkReject, Stream, StreamMode, ZakuraPeerId, FRAME_HEADER_BYTES, }; -use std::sync::atomic::{AtomicU64, Ordering}; +use std::{ + sync::atomic::{AtomicU64, Ordering}, + time::Instant, +}; /// Maximum frame bytes for one stream-6 body frame plus protocol framing. /// @@ -48,6 +51,23 @@ impl BlockSyncPeerSession { } } + /// Build a session directly from a `FramedSend` for routine-level unit tests, + /// bypassing a full `PeerStreamSession`. The `send` half feeds a `framed_channel` + /// the test reads, and `cancel_token` lets the test tear the routine down. + #[cfg(test)] + pub(super) fn for_test( + peer_id: ZakuraPeerId, + send: FramedSend, + cancel_token: CancellationToken, + ) -> Self { + Self { + peer_id, + direction: ServicePeerDirection::Outbound, + send, + cancel_token, + } + } + /// Authenticated peer identity for this block-sync session. pub fn peer_id(&self) -> &ZakuraPeerId { &self.peer_id @@ -319,6 +339,13 @@ impl BlockSyncService { count < cap } + fn peer_is_parked(&self, peer_id: &ZakuraPeerId) -> bool { + self.inner + .routine_wiring + .as_ref() + .is_some_and(|wiring| wiring.registry.is_peer_parked(peer_id, Instant::now())) + } + /// Whether `add_peer` may install a session for this peer. A peer that is /// already registered may always *replace* its session (the /// connection-symmetry collision where both sides opened a block-sync stream @@ -356,14 +383,19 @@ impl Service for BlockSyncService { fn wants_peer( &self, - _peer: &ZakuraPeerId, + peer: &ZakuraPeerId, _negotiated: u64, direction: ServicePeerDirection, ) -> bool { - self.peer_slots_free(direction) + !self.peer_is_parked(peer) && self.peer_slots_free(direction) } fn add_peer(&self, mut peer: Peer) { + if self.peer_is_parked(&peer.id) { + peer.service_cancel_token().cancel(); + return; + } + if !self.can_admit_peer(&peer.id, peer.direction) { return; } diff --git a/zebra-network/src/zakura/block_sync/state.rs b/zebra-network/src/zakura/block_sync/state.rs index 260c2817396..2ec887720ca 100644 --- a/zebra-network/src/zakura/block_sync/state.rs +++ b/zebra-network/src/zakura/block_sync/state.rs @@ -1,4 +1,4 @@ -use super::{config::*, request::*, work_queue::WorkQueue, *}; +use super::{bbr::BbrState, config::*, request::*, work_queue::WorkQueue, *}; use crate::zakura::{ chain_frontier_from_parts, Frontier, FrontierUpdate, ServicePeerDirection, ServicePeerSnapshot, ZakuraBlockSyncCandidateState, @@ -11,38 +11,6 @@ use crate::zakura::{ /// [`MAX_BS_INFLIGHT_REQUESTS`]). // `MAX_BS_INFLIGHT_REQUESTS` is a `u32`, which fits in `usize` on supported targets. pub(super) const EFFECTIVE_BS_OUTBOUND_INFLIGHT_PER_PEER: usize = MAX_BS_INFLIGHT_REQUESTS as usize; -/// Consecutive error-free responses that make up one window-growth epoch. -/// -/// The adaptive window holds flat for a full epoch of successes before each -/// growth step, so it never opens faster than the peer has demonstrably served. -const OUTBOUND_WINDOW_GROWTH_EPOCH_SUCCESSES: usize = 16; -/// Cubic coefficient for the streak-gated window ramp. -/// -/// Target window = `growth_base + COEFF * epoch^3`, capped at the hard cap, where -/// `epoch` counts completed [`OUTBOUND_WINDOW_GROWTH_EPOCH_SUCCESSES`]-success -/// runs since the last timeout. Growth is gentle for the first few epochs and -/// accelerates the longer the peer serves without an error, then resets on the -/// next timeout. With the default base of 64 and a 16-success epoch, the window -/// ramps cubically toward the peer's advertised hard cap (locally clamped to -/// [`MAX_BS_INFLIGHT_REQUESTS`] = 32,768; the default advertisement is 32,000), -/// reaching the default 32,000 ceiling after 16 epochs (256 consecutive -/// error-free successes). -const OUTBOUND_WINDOW_GROWTH_CUBIC_COEFF: usize = 8; -/// Cubic coefficient for the streak-gated timeout backoff. -const OUTBOUND_WINDOW_REDUCTION_CUBIC_COEFF: usize = OUTBOUND_WINDOW_GROWTH_CUBIC_COEFF; -/// Consecutive timeout batches that make up one window-reduction epoch. -const OUTBOUND_WINDOW_REDUCTION_EPOCH_TIMEOUTS: usize = OUTBOUND_WINDOW_GROWTH_EPOCH_SUCCESSES; -/// Timeouts tolerated after the adaptive window has already reached its floor of -/// one in-flight request, before the peer is disconnected. -/// -/// Set to two full reduction epochs. Once a peer has been backed off all the way -/// down to a single in-flight request, we keep probing it for -/// `2 * OUTBOUND_WINDOW_REDUCTION_EPOCH_TIMEOUTS` consecutive timeouts before -/// giving up — at the default 8s request timeout that is ~256s of uninterrupted -/// failure at the floor. Any successful response (even a single block) resets the -/// streak, so only a peer that serves nothing across the whole window is dropped. -const OUTBOUND_WINDOW_FLOOR_TIMEOUTS_BEFORE_DISCONNECT: usize = - 2 * OUTBOUND_WINDOW_REDUCTION_EPOCH_TIMEOUTS; /// Cached chain frontiers used by the block-sync reactor. #[derive(Copy, Clone, Debug, Eq, PartialEq)] @@ -327,167 +295,265 @@ impl BlockSyncState { } } -/// Adaptive per-peer outbound request window + outstanding requests. -/// -/// Kept as a standalone type so the window math stays unit-testable; the per-peer -/// download state lives in the spawned [`PeerRoutine`](super::peer_routine), which -/// embeds one of these. +/// Carved out of `PeerBlockState` so the window math stays unit-testable +/// while the per-peer download state moves into the spawned +/// [`PeerRoutine`](super::peer_routine) (per-peer routines). The routine embeds one of these. #[derive(Clone, Debug)] pub(super) struct DownloadWindow { pub(super) max_inflight_requests: u32, - pub(super) outbound_request_window: usize, - pub(super) timeout_recovery_slots: usize, pub(super) outstanding: Vec, - /// Completed error-free responses since the last timeout-driven reduction. - /// Drives the streak-gated cubic ramp in - /// [`increase_outbound_window_after_success`](Self::increase_outbound_window_after_success). - consecutive_successes: usize, - /// Consecutive timeout batches since the last successful response. - /// - /// Drives the same streak-gated cubic shape as successful response growth, - /// but downward. The window holds flat for a full timeout epoch, then lowers - /// faster as the consecutive timeout streak grows. - consecutive_timeouts: usize, - /// Window value at the last reduction — the floor the cubic ramp grows back - /// up from, so probing resumes from the post-backoff window rather than the - /// original slow-start point. - growth_base: usize, - /// Window value when the current timeout streak began. - reduction_base: usize, - /// Consecutive timeout batches observed while the window was already at the - /// minimum. Once this crosses the threshold, the caller should disconnect the - /// peer rather than retrying indefinitely at one request. - floor_timeouts: usize, + /// Per-peer BBR-lite estimators + cwnd — the sole congestion controller. Under + /// [`CwndUnit::Bytes`] the cwnd is itself a byte budget sourced from header size + /// hints (no fixed per-request byte weight), so there is no `nominal_request_bytes`. + bbr: BbrState, + /// Whether the cwnd budgets outstanding work in request slots or reserved bytes. + cwnd_unit: CwndUnit, + /// Deadline by which an active peer must send another accepted full block. + pub(super) block_liveness_deadline: Option, + /// Last time this peer sent an accepted full block body. + pub(super) last_block_at: Option, } #[derive(Copy, Clone, Debug, Eq, PartialEq)] -pub(super) enum TimeoutBackoffOutcome { - KeepPeer, - DisconnectPeer, +pub(super) enum LivenessOutcome { + Ok, + Disarm, + Disconnect, } impl DownloadWindow { pub(super) fn new(config: &ZakuraBlockSyncConfig) -> Self { - let max_inflight_requests = config.advertised_max_inflight_requests(); - // Slow-start: open at the configured initial window (clamped to the - // advertised hard cap) and grow toward the cap on success, rather than - // opening at the full `max_inflight`. - let initial_window = config - .initial_inflight_requests - .clamp(1, max_inflight_requests); - let initial_window = usize::try_from(initial_window) - .expect("u32 initial inflight requests fits in usize on supported targets"); Self { - max_inflight_requests, - outbound_request_window: initial_window, - timeout_recovery_slots: 0, + max_inflight_requests: config.advertised_max_inflight_requests(), outstanding: Vec::new(), - consecutive_successes: 0, - consecutive_timeouts: 0, - growth_base: initial_window, - reduction_base: initial_window, - floor_timeouts: 0, + bbr: BbrState::new(config), + cwnd_unit: config.bbr_cwnd_unit, + block_liveness_deadline: None, + last_block_at: None, } } - pub(super) fn available_slots(&self) -> usize { - let hard_capacity = self.hard_outbound_capacity(); - let adaptive_limit = hard_capacity.min(self.outbound_request_window); - let adaptive_slots = adaptive_limit.saturating_sub(self.outstanding.len()); - if adaptive_slots > 0 { - return adaptive_slots; + pub(super) fn delivery_snapshot(&self, now: Instant) -> DeliverySnapshot { + self.bbr.delivery_snapshot(now) + } + + /// Record a completed request into the BBR estimators (RTprop / BtlBw / delivered) + /// and advance the ProbeRtt phase machine. `delivered_bytes` is the request's total + /// delivered body bytes — under the single-block-per-request invariant + /// (`DEFAULT_BS_BLOCKS_PER_RESPONSE = 1`) this is the completing body's + /// `serialized_bytes`. Call after removing the completed request from `outstanding`, + /// so the in-flight measure reflects the post-completion queue depth. + pub(super) fn record_delivery( + &mut self, + now: Instant, + elapsed: Duration, + blocks: u32, + delivered_bytes: u64, + snapshot: DeliverySnapshot, + ) { + // The ProbeRtt drain check compares this against `min_cwnd`, so the in-flight + // measure MUST be in the cwnd's unit: request count under `Blocks`, reserved + // body bytes under `Bytes`. Passing the raw request count under `Bytes` made the + // drain check (`count <= min_cwnd_bytes`) trivially true, so the hold timer + // started before the byte queue had actually drained and the RTprop sample could + // still be contended. + let inflight = match self.cwnd_unit { + // `outstanding.len()` (a `usize` request count) widens to `u64` losslessly. + CwndUnit::Blocks => self.outstanding.len() as u64, + CwndUnit::Bytes => self.outstanding_reserved_bytes(), + }; + self.bbr + .record_delivery(now, elapsed, blocks, delivered_bytes, inflight, snapshot); + } + + /// The effective BBR cwnd as a **request count**, for diagnostics that compare + /// against the request-count hard cap (the periodic slot trace, cross-peer floor + /// bias). Under `Blocks` this is the cwnd directly; under `Bytes` it is the byte + /// cwnd divided by a representative body size, so it reads as "requests this peer's + /// byte window admits". The byte cwnd itself is available via + /// [`bbr_effective_cwnd_bytes`](Self::bbr_effective_cwnd_bytes). + pub(super) fn bbr_effective_cwnd(&self) -> usize { + match self.cwnd_unit { + CwndUnit::Blocks => self.bbr.effective_cwnd(), + CwndUnit::Bytes => { + let cwnd_bytes = self.bbr.effective_cwnd() as u64; + let rep = self.representative_body_bytes(); + usize::try_from((cwnd_bytes / rep.max(1)).max(1)).unwrap_or(usize::MAX) + } } + } - self.timeout_recovery_slots - .min(hard_capacity.saturating_sub(self.outstanding.len())) + /// The effective byte cwnd under `Bytes` (`None` under `Blocks`), for tracing. + pub(super) fn bbr_effective_cwnd_bytes(&self) -> Option { + matches!(self.cwnd_unit, CwndUnit::Bytes).then(|| self.bbr.effective_cwnd() as u64) } - // reduce_outbound_window_after_timeout is the per-peer backoff path for block-sync - // downloads. It is called when a peer times out, and it shrinks that peer's adaptive - // outbound request window so Zebra asks that peer for fewer block ranges - // concurrently. - pub(super) fn reduce_outbound_window_after_timeout(&mut self) -> TimeoutBackoffOutcome { - // If this is the first timeout in a row, reset the reduction base to the current window. - if self.consecutive_timeouts == 0 { - self.reduction_base = self.outbound_request_window; + /// A representative body size in bytes for converting a byte cwnd into a request + /// count: the mean reserved bytes across in-flight requests, falling back to the + /// per-block worst case when nothing is outstanding. Used only for diagnostics and + /// the floor-bypass byte bonus, never for admission. + fn representative_body_bytes(&self) -> u64 { + let outstanding = self.outstanding.len() as u64; + if outstanding == 0 { + return block::MAX_BLOCK_BYTES; } + (self.outstanding_reserved_bytes() / outstanding).max(1) + } - // Increment the consecutive timeout streak. - self.consecutive_timeouts = self.consecutive_timeouts.saturating_add(1); - - // Check if the window is at the minimum. - let was_at_floor = self.outbound_request_window == 1; - - // Calculate the epoch of the timeout streak. - let epoch = self.consecutive_timeouts / OUTBOUND_WINDOW_REDUCTION_EPOCH_TIMEOUTS; - if epoch > 0 { - let cubic_reduction = epoch - .saturating_mul(epoch) - .saturating_mul(epoch) - .saturating_mul(OUTBOUND_WINDOW_REDUCTION_CUBIC_COEFF); - let target = self.reduction_base.saturating_sub(cubic_reduction).max(1); - // Reduction only: never grow the window on a timeout. - self.outbound_request_window = self.outbound_request_window.min(target).max(1); - } + /// The current RTprop estimate in milliseconds, for tracing. + pub(super) fn bbr_rtprop_ms(&self) -> Option { + self.bbr.rtprop_ms() + } - // A timeout ends the current success streak; the cubic ramp restarts from - // the reduced window, so growth probes back up from the post-backoff floor. - self.consecutive_successes = 0; - self.growth_base = self.outbound_request_window; - self.timeout_recovery_slots = self - .timeout_recovery_slots - .saturating_add(1) - .min(self.hard_outbound_capacity()); - - if was_at_floor { - self.floor_timeouts = self.floor_timeouts.saturating_add(1); - } else { - self.floor_timeouts = 0; + /// The current BtlBw estimate in milli-blocks/sec (blocks/sec × 1000), for tracing. + /// `None` under `Bytes`, where [`bbr_btlbw_bytes_per_sec`](Self::bbr_btlbw_bytes_per_sec) + /// is the meaningful rate. + pub(super) fn bbr_btlbw_milliblocks(&self) -> Option { + matches!(self.cwnd_unit, CwndUnit::Blocks) + .then(|| self.bbr.btlbw_milliblocks_per_sec()) + .flatten() + } + + /// The current BtlBw estimate in bytes/sec under `Bytes` (`None` under `Blocks`). + pub(super) fn bbr_btlbw_bytes_per_sec(&self) -> Option { + if !matches!(self.cwnd_unit, CwndUnit::Bytes) { + return None; } + self.bbr + .btlbw_units_per_sec() + // A non-negative finite bytes/sec rate rounds into u64 for any real link. + .map(|rate| rate.round() as u64) + } - if self.floor_timeouts >= OUTBOUND_WINDOW_FLOOR_TIMEOUTS_BEFORE_DISCONNECT { - TimeoutBackoffOutcome::DisconnectPeer - } else { - TimeoutBackoffOutcome::KeepPeer + /// Bytes reserved across this peer's in-flight requests, for tracing the byte window + /// occupancy. + pub(super) fn bbr_inflight_bytes(&self) -> u64 { + self.outstanding_reserved_bytes() + } + + /// Total delivered through this peer's completed requests, for tracing — blocks + /// under `Blocks`, bytes under `Bytes`. + pub(super) fn bbr_delivered(&self) -> u64 { + self.bbr.delivered() + } + + /// The current BBR phase as a numeric code (0 = ProbeBw, 1 = ProbeRtt), for tracing. + pub(super) fn bbr_phase_code(&self) -> u64 { + self.bbr.phase_code() + } + + /// The smoothed request round-trip in milliseconds the delay-gradient tracks. + pub(super) fn bbr_smoothed_elapsed_ms(&self) -> Option { + self.bbr.smoothed_elapsed_ms() + } + + /// The delay-gradient cwnd ceiling in blocks once it binds (`None` while unbounded). + pub(super) fn bbr_delay_cap(&self) -> Option { + self.bbr + .delay_cap() + .map(|cap| u64::try_from(cap).unwrap_or(u64::MAX)) + } + + pub(super) fn available_slots(&self) -> usize { + self.available_slots_with_bonus(0) + } + + /// Available headroom allowing `bonus` extra in-flight requests beyond the BBR cwnd, + /// still clamped to the peer's advertised hard cap. `bonus == 0` is the normal + /// (above-floor) capacity used by [`available_slots`]; a small positive `bonus` is + /// the floor bypass — it lets the lowest missing height be fetched even when the + /// peer is saturated at its cwnd, without ever exceeding the advertised inflight. + /// + /// The return value is non-zero exactly when there is room for at least one more + /// request; callers use it as a gate, not an absolute count. Under + /// [`CwndUnit::Bytes`] the cwnd is itself a byte budget (`BtlBw_bytes × RTprop × + /// gain`, from header size hints) compared against reserved body bytes, so a peer + /// serving large bodies holds fewer in flight and a peer serving small bodies holds + /// many — the in-flight *request* count falls out of `cwnd_bytes / body_size`. The + /// controller is unit-agnostic; only this comparison differs — the seam that makes + /// switching units a small change. + pub(super) fn available_slots_with_bonus(&self, bonus: usize) -> usize { + // BBR-lite is the sole congestion controller: cap in-flight at the BDP-derived + // cwnd so a peer's queue stays at ~one BDP and head-of-line latency tracks + // RTprop. The floor bypass adds `bonus` on top. + let hard_cap = self.hard_outbound_capacity(); + match self.cwnd_unit { + CwndUnit::Blocks => { + let cwnd_slots = self + .bbr + .effective_cwnd() + .saturating_add(bonus) + .min(hard_cap); + cwnd_slots.saturating_sub(self.outstanding.len()) + } + CwndUnit::Bytes => { + // The peer's advertised request-count cap still binds in byte mode: a peer + // serving tiny bodies must never be issued more in-flight *requests* than it + // advertised it will service, however much byte headroom the cwnd still + // shows. Once the request count reaches the hard cap there is no slot, + // regardless of bytes — mirroring the blocks-unit ceiling (review fix F2). + let outstanding = self.outstanding.len(); + if outstanding >= hard_cap { + return 0; + } + // The cwnd is already a byte budget. The floor bypass grants `bonus` + // *representative* bodies of extra byte headroom — sized to the recent + // per-request reservation, NOT the 2 MB worst case — so a starved floor + // can still be fetched when the byte window is full without ballooning + // the in-flight bytes far past the cwnd (which would defeat the byte + // denomination's head-of-line bound). The take is still count-capped to + // one block and passes the real `ByteBudget` reservation. + let reserved = self.outstanding_reserved_bytes(); + let representative = self.representative_body_bytes(); + let bonus_bytes = (bonus as u64).saturating_mul(representative); + let cwnd_bytes = (self.bbr.effective_cwnd() as u64).saturating_add(bonus_bytes); + usize::try_from(cwnd_bytes.saturating_sub(reserved)).unwrap_or(usize::MAX) + } } } - /// Grow the adaptive window on a successful response using a streak-gated - /// cubic ramp: hold the window flat until a full epoch of consecutive - /// successes, then raise it to `growth_base + COEFF * epoch^3` (capped at the - /// hard cap). The step grows cubically with the no-error streak, so the window - /// opens gently at first and accelerates the longer the peer serves without a - /// timeout. Any timeout resets the streak and the base (see - /// [`reduce_outbound_window_after_timeout`](Self::reduce_outbound_window_after_timeout)). - pub(super) fn increase_outbound_window_after_success(&mut self) { - self.consecutive_timeouts = 0; - self.reduction_base = self.outbound_request_window; - self.floor_timeouts = 0; - self.consecutive_successes = self.consecutive_successes.saturating_add(1); - let max_window = self.hard_outbound_capacity(); - if self.outbound_request_window >= max_window { - return; + /// Bytes reserved across this peer's in-flight requests (the per-request size + /// estimates of heights not yet received). Recomputed on demand — the byte unit is + /// experimental; a hot path would maintain a running counter instead. + fn outstanding_reserved_bytes(&self) -> u64 { + self.outstanding.iter().fold(0u64, |acc, range| { + acc.saturating_add(range.reserved_bytes()) + }) + } + + /// Apply the BBR cwnd dip on a real request timeout (one multiplicative dip, + /// bounded by the minimum cwnd). + pub(super) fn record_timeout(&mut self) { + self.bbr.dip_on_timeout(); + } + + pub(super) fn arm_liveness(&mut self, now: Instant, timeout: Duration) { + if self.block_liveness_deadline.is_none() { + self.block_liveness_deadline = Some(now + timeout); } - let epoch = self.consecutive_successes / OUTBOUND_WINDOW_GROWTH_EPOCH_SUCCESSES; - if epoch == 0 { - // Still inside the first flat epoch: do not open the window yet. - return; + } + + pub(super) fn note_block_progress(&mut self, now: Instant, timeout: Duration) { + self.last_block_at = Some(now); + self.block_liveness_deadline = if self.outstanding.is_empty() { + None + } else { + Some(now + timeout) + }; + } + + pub(super) fn disarm_liveness_if_idle(&mut self) { + if self.outstanding.is_empty() { + self.block_liveness_deadline = None; } - let cubic = epoch - .saturating_mul(epoch) - .saturating_mul(epoch) - .saturating_mul(OUTBOUND_WINDOW_GROWTH_CUBIC_COEFF); - let target = self.growth_base.saturating_add(cubic).min(max_window); - // Growth only: never shrink the window on a success. - self.outbound_request_window = self.outbound_request_window.max(target); - } - - pub(super) fn record_outbound_request_scheduled(&mut self) { - let adaptive_limit = self - .hard_outbound_capacity() - .min(self.outbound_request_window); - if self.outstanding.len() >= adaptive_limit && self.timeout_recovery_slots > 0 { - self.timeout_recovery_slots = self.timeout_recovery_slots.saturating_sub(1); + } + + pub(super) fn check_liveness(&self, now: Instant) -> LivenessOutcome { + match self.block_liveness_deadline { + None => LivenessOutcome::Ok, + Some(deadline) if now < deadline => LivenessOutcome::Ok, + Some(_) if self.outstanding.is_empty() => LivenessOutcome::Disarm, + Some(_) => LivenessOutcome::Disconnect, } } @@ -579,16 +645,23 @@ pub(super) struct OutstandingBlockRange { pub(super) request: BlockRangeRequest, pub(super) queued_at: Instant, pub(super) deadline: Instant, + pub(super) delivery_snapshot: DeliverySnapshot, + pub(super) delivered_bytes: u64, pub(super) received: ReceivedBlockTracker, } +#[derive(Copy, Clone, Debug)] +pub(super) struct DeliverySnapshot { + pub(super) delivered: u64, + pub(super) delivered_at: Instant, +} + impl OutstandingBlockRange { /// Bytes still reserved for this request: the sum of the per-height size /// estimates for every requested height not yet received. Each received body /// shrinks its estimate toward the actual size, so releasing this (on /// timeout/disconnect/short response) never over-releases bytes already handed /// to the reorder buffer. - #[cfg(test)] pub(super) fn reserved_bytes(&self) -> u64 { self.request .expected_blocks @@ -615,6 +688,10 @@ impl OutstandingBlockRange { } } + pub(super) fn record_body_bytes(&mut self, bytes: u64) { + self.delivered_bytes = self.delivered_bytes.saturating_add(bytes); + } + /// Mark every requested height at or below `tip` as received and return the /// sum of the per-height size estimates those newly-received heights still /// held, so the caller releases exactly the reservation those heights held. @@ -705,6 +782,18 @@ impl BlockBudgetLedger { } } +/// Number of distinct request offsets the [`ReceivedBlockTracker`] bitset can hold — +/// one per bit of its `u128`. +const RECEIVED_TRACKER_OFFSET_CAPACITY: u32 = u128::BITS; + +// A request range carries one received-offset bit per requested height (offsets +// `0..count`). If the advertised block-count cap ever exceeded the bitset width, +// `bit_for_offset` would return `None` for the overflowing heights, so they could +// never be marked received, `is_complete()` would be unreachable, and the range would +// wedge (its reservation never released). Couple the two so a future cap bump that +// outgrows the bitset fails to compile instead of silently wedging. +const _: () = assert!(MAX_BS_BLOCKS_PER_REQUEST <= RECEIVED_TRACKER_OFFSET_CAPACITY); + #[derive(Clone, Debug, Default)] pub(super) struct ReceivedBlockTracker { bits: u128, diff --git a/zebra-network/src/zakura/block_sync/tests.rs b/zebra-network/src/zakura/block_sync/tests.rs index e1e18690c5c..d0213c1f849 100644 --- a/zebra-network/src/zakura/block_sync/tests.rs +++ b/zebra-network/src/zakura/block_sync/tests.rs @@ -9,7 +9,8 @@ use super::{ DEFAULT_BS_FLOOR_PEER_AVOID_COOLDOWN, DEFAULT_BS_MAX_INFLIGHT_BLOCK_BYTES, DEFAULT_BS_MAX_REORDER_LOOKAHEAD_BLOCKS, DEFAULT_BS_MAX_REORDER_LOOKAHEAD_BYTES, DEFAULT_BS_MAX_RESPONSE_BYTES, DEFAULT_BS_MAX_SUBMITTED_BLOCK_APPLIES, - DEFAULT_BS_REQUEST_TIMEOUT, MAX_BS_INFLIGHT_REQUESTS, MAX_BS_RESPONSE_BYTES, + DEFAULT_BS_NO_PROGRESS_PEER_COOLDOWN, DEFAULT_BS_REQUEST_TIMEOUT, MAX_BS_INFLIGHT_REQUESTS, + MAX_BS_RESPONSE_BYTES, }, reactor::node_id_from_block_peer_id, reorder::*, @@ -18,9 +19,11 @@ use super::{ state::*, }; use crate::zakura::{ - framed_channel, ChainFrontier, FramedRecv, FramedSend, Frontier, FrontierChange, - FrontierUpdate, Peer, Service, ServicePeerSnapshot, ServiceRegistry, StreamMode, - ZakuraBlockSyncCandidateState, ZakuraSyncExchange, + framed_channel, + testkit::{TraceCapture, TraceValue}, + ChainFrontier, FramedRecv, FramedSend, Frontier, FrontierChange, FrontierUpdate, Peer, Service, + ServicePeerSnapshot, ServiceRegistry, StreamMode, ZakuraBlockSyncCandidateState, + ZakuraSyncExchange, }; use zebra_chain::{ fmt::HexDebug, @@ -423,8 +426,16 @@ fn download_window() -> DownloadWindow { DownloadWindow::new(&ZakuraBlockSyncConfig::default()) } +fn test_delivery_snapshot(now: Instant) -> DeliverySnapshot { + DeliverySnapshot { + delivered: 0, + delivered_at: now, + } +} + fn window_request(height: u32) -> OutstandingBlockRange { let byte = u8::try_from(height).expect("test heights fit in u8"); + let now = Instant::now(); OutstandingBlockRange { request: BlockRangeRequest { start_height: block::Height(height), @@ -437,143 +448,143 @@ fn window_request(height: u32) -> OutstandingBlockRange { estimated_bytes: 1, }], }, - queued_at: Instant::now(), - deadline: Instant::now(), + queued_at: now, + deadline: now, + delivery_snapshot: test_delivery_snapshot(now), + delivered_bytes: 0, + received: ReceivedBlockTracker::default(), + } +} + +fn window_request_range(start: u32, count: u32) -> OutstandingBlockRange { + let byte = u8::try_from(start).expect("test heights fit in u8"); + let now = Instant::now(); + OutstandingBlockRange { + request: BlockRangeRequest { + start_height: block::Height(start), + count, + anchor_hash: block::Hash([byte; 32]), + estimated_bytes: u64::from(count), + expected_blocks: (start..start + count) + .map(|height| ExpectedBlock { + height: block::Height(height), + hash: block::Hash([u8::try_from(height).expect("test heights fit in u8"); 32]), + estimated_bytes: 1, + }) + .collect(), + }, + queued_at: now, + deadline: now, + delivery_snapshot: test_delivery_snapshot(now), + delivered_bytes: 0, received: ReceivedBlockTracker::default(), } } #[test] -fn peer_outbound_request_window_backs_off_and_grows_with_streaks() { +fn block_liveness_disconnects_silent_active_peer_after_default_timeout() { + let config = ZakuraBlockSyncConfig::default(); + let timeout = config.effective_liveness_timeout(); + assert_eq!(timeout, Duration::from_secs(32)); + + let now = Instant::now(); let mut window = download_window(); - let max_inflight = - usize::try_from(MAX_BS_INFLIGHT_REQUESTS).expect("test max inflight fits usize"); - window.max_inflight_requests = MAX_BS_INFLIGHT_REQUESTS; - window.outbound_request_window = max_inflight; window.outstanding.push(window_request(1)); - assert_eq!(window.available_slots(), max_inflight - 1); + window.arm_liveness(now, timeout); - for _ in 0..15 { - assert_eq!( - window.reduce_outbound_window_after_timeout(), - TimeoutBackoffOutcome::KeepPeer - ); - } assert_eq!( - window.outbound_request_window, max_inflight, - "window holds flat within the first timeout epoch" + window.check_liveness(now + timeout - Duration::from_millis(1)), + LivenessOutcome::Ok ); assert_eq!( - window.reduce_outbound_window_after_timeout(), - TimeoutBackoffOutcome::KeepPeer + window.check_liveness(now + timeout), + LivenessOutcome::Disconnect ); - assert_eq!(window.outbound_request_window, max_inflight - 8); +} - for _ in 0..16 { - assert_eq!( - window.reduce_outbound_window_after_timeout(), - TimeoutBackoffOutcome::KeepPeer - ); - } - assert_eq!(window.outbound_request_window, max_inflight - 64); +#[test] +fn block_liveness_never_disconnects_idle_peer() { + let now = Instant::now(); + let mut window = download_window(); - for _ in 0..(16 * 16) { - assert_eq!( - window.reduce_outbound_window_after_timeout(), - TimeoutBackoffOutcome::KeepPeer - ); - if window.outbound_request_window == 1 { - break; - } - } - assert_eq!(window.outbound_request_window, 1); - assert_eq!(window.available_slots(), window.timeout_recovery_slots); - // Pinned at the floor, the peer is tolerated for two full reduction epochs - // (2 * 16 = 32 consecutive timeouts, ~256s at the 8s request timeout) before - // being disconnected. - for _ in 0..31 { - assert_eq!( - window.reduce_outbound_window_after_timeout(), - TimeoutBackoffOutcome::KeepPeer - ); + assert_eq!(window.check_liveness(now), LivenessOutcome::Ok); + + window.block_liveness_deadline = Some(now); + assert_eq!(window.check_liveness(now), LivenessOutcome::Disarm); + window.disarm_liveness_if_idle(); + assert_eq!(window.block_liveness_deadline, None); + assert_eq!(window.check_liveness(now), LivenessOutcome::Ok); +} + +#[test] +fn block_liveness_progress_before_deadline_keeps_peer_alive() { + let timeout = ZakuraBlockSyncConfig::default().effective_liveness_timeout(); + let mut now = Instant::now(); + let mut window = download_window(); + window.outstanding.push(window_request(1)); + window.arm_liveness(now, timeout); + + for _ in 0..4 { + now += timeout - Duration::from_millis(1); + assert_eq!(window.check_liveness(now), LivenessOutcome::Ok); + window.note_block_progress(now, timeout); + assert_eq!(window.block_liveness_deadline, Some(now + timeout)); } - assert_eq!( - window.reduce_outbound_window_after_timeout(), - TimeoutBackoffOutcome::DisconnectPeer - ); +} + +#[test] +fn block_liveness_disarms_when_outstanding_drains() { + let timeout = ZakuraBlockSyncConfig::default().effective_liveness_timeout(); + let now = Instant::now(); + let mut window = download_window(); + window.outstanding.push(window_request(1)); + window.arm_liveness(now, timeout); - // Streak-gated cubic ramp: the window holds flat for a full epoch of - // consecutive successes, then steps up. From the reduced base of 1, the first - // epoch (16 successes) raises it to base + COEFF * 1^3 = 1 + 8 = 9. window.outstanding.clear(); - window.timeout_recovery_slots = 0; - for _ in 0..15 { - window.increase_outbound_window_after_success(); - } - assert_eq!( - window.outbound_request_window, 1, - "window holds flat within the first success epoch" - ); - window.increase_outbound_window_after_success(); - assert_eq!( - window.outbound_request_window, 9, - "the first full success epoch steps the window up by COEFF * 1^3" - ); - // A second epoch accelerates cubically: base + COEFF * 2^3 = 1 + 64 = 65. - for _ in 0..16 { - window.increase_outbound_window_after_success(); - } - assert_eq!( - window.outbound_request_window, 65, - "the second epoch grows cubically faster than the first" - ); + window.disarm_liveness_if_idle(); - window.outbound_request_window = max_inflight; - window.increase_outbound_window_after_success(); - assert_eq!(window.outbound_request_window, max_inflight); + assert_eq!(window.block_liveness_deadline, None); + assert_eq!(window.check_liveness(now + timeout), LivenessOutcome::Ok); } #[test] -fn peer_timeout_recovery_slot_replaces_timed_out_request_above_reduced_window() { +fn block_liveness_resuming_after_idle_gets_fresh_deadline() { + let timeout = ZakuraBlockSyncConfig::default().effective_liveness_timeout(); + let now = Instant::now(); let mut window = download_window(); - window.max_inflight_requests = 8; - window.outbound_request_window = 8; + window.outstanding.push(window_request(1)); + window.arm_liveness(now, timeout); + window.outstanding.clear(); + window.disarm_liveness_if_idle(); - for height in 1u32..=8 { - window.outstanding.push(window_request(height)); - } + let resumed = now + Duration::from_secs(60); + window.outstanding.push(window_request(2)); + window.arm_liveness(resumed, timeout); - assert_eq!(window.available_slots(), 0); + assert_eq!(window.block_liveness_deadline, Some(resumed + timeout)); +} - for _ in 0..16 { - assert_eq!( - window.reduce_outbound_window_after_timeout(), - TimeoutBackoffOutcome::KeepPeer - ); - } - assert_eq!(window.outbound_request_window, 1); - assert_eq!(window.timeout_recovery_slots, 8); - assert_eq!( - window.available_slots(), - 0, - "the advertised hard cap is still full until the timed-out request is removed" - ); +#[test] +fn block_liveness_multi_block_range_progress_resets_each_body() { + let timeout = ZakuraBlockSyncConfig::default().effective_liveness_timeout(); + let start = Instant::now(); + let mut window = download_window(); + window.outstanding.push(window_request_range(1, 3)); + window.arm_liveness(start, timeout); - window.outstanding.remove(0); - assert_eq!( - window.available_slots(), - 1, - "a timeout recovery slot lets the retry replace the timed-out request" - ); + let first = start + Duration::from_secs(4); + window.note_block_progress(first, timeout); + assert_eq!(window.block_liveness_deadline, Some(first + timeout)); - window.record_outbound_request_scheduled(); - assert_eq!(window.timeout_recovery_slots, 7); - window.outstanding.push(window_request(9)); - assert_eq!( - window.available_slots(), - 0, - "regular scheduling remains held below the reduced adaptive window" - ); + let second = first + Duration::from_secs(4); + assert_eq!(window.check_liveness(second), LivenessOutcome::Ok); + window.note_block_progress(second, timeout); + assert_eq!(window.block_liveness_deadline, Some(second + timeout)); + + let third = second + Duration::from_secs(4); + assert_eq!(window.check_liveness(third), LivenessOutcome::Ok); + window.note_block_progress(third, timeout); + assert_eq!(window.block_liveness_deadline, Some(third + timeout)); } // The old `BlockRangeScheduler` single-pass timeout-retry bias @@ -581,7 +592,7 @@ fn peer_timeout_recovery_slot_replaces_timed_out_request_above_reduced_window() // per-peer assignment to bias, so a returned height is simply contestable by any // servable peer. The peer-local timeout bias is re-introduced in per-peer routines. The // reactor-level locality property is still covered by -// `reactor_timeout_backoff_is_local_and_healthy_peer_keeps_filling`. +// `reactor_timeout_recovery_is_local_and_healthy_peer_keeps_filling`. #[test] fn work_queue_returned_height_is_contestable_by_any_peer() { let queue = work_queue_with(0, [needed(1, BlockSizeEstimate::Advertised(100))]); @@ -711,6 +722,10 @@ fn block_sync_config_defaults_and_round_trips() { default.floor_peer_avoid_cooldown, DEFAULT_BS_FLOOR_PEER_AVOID_COOLDOWN ); + assert_eq!( + default.no_progress_peer_cooldown, + DEFAULT_BS_NO_PROGRESS_PEER_COOLDOWN + ); assert_eq!( default.effective_max_reorder_lookahead_bytes(), DEFAULT_BS_MAX_REORDER_LOOKAHEAD_BYTES @@ -723,6 +738,10 @@ fn block_sync_config_defaults_and_round_trips() { default.effective_floor_peer_avoid_cooldown(), DEFAULT_BS_FLOOR_PEER_AVOID_COOLDOWN ); + assert_eq!( + default.effective_no_progress_peer_cooldown(), + DEFAULT_BS_NO_PROGRESS_PEER_COOLDOWN + ); assert!(default.validate().is_ok()); assert_eq!( default.max_submitted_block_applies, @@ -790,6 +809,12 @@ fn config_validate_rejects_degenerate_values() { ..ZakuraBlockSyncConfig::default() }; assert!(config.validate().is_ok()); + + config = ZakuraBlockSyncConfig { + request_timeout: Duration::ZERO, + ..ZakuraBlockSyncConfig::default() + }; + assert!(config.validate().is_err()); } #[test] @@ -2049,15 +2074,15 @@ async fn reactor_budget_constrained_issuance_rotates_across_peers() { reactor_task.abort(); } -/// One peer whose request times out backs off (its outbound window halves) and -/// must not block the other peers from being filled out of the same shared work. +/// One peer whose request times out enters local recovery and must not block the +/// other peers from being filled out of the same shared work. /// /// This is the timeout-locality invariant: a slow peer's recovery is local to /// that peer. The retry path re-queues the timed-out range to a *different* /// servable peer, so the healthy peer keeps making progress while the slow peer /// is in recovery rather than the whole download stalling behind one straggler. #[tokio::test] -async fn reactor_timeout_backoff_is_local_and_healthy_peer_keeps_filling() { +async fn reactor_timeout_recovery_is_local_and_healthy_peer_keeps_filling() { let mut config = immediate_body_download_config(); config.fanout = 1; // A request timeout long enough that the opening pass fans both heights out @@ -2143,7 +2168,7 @@ async fn reactor_timeout_backoff_is_local_and_healthy_peer_keeps_filling() { // Pick one peer to be the straggler (it never answers) and the other to be // healthy. The healthy peer answers `RangeUnavailable` for its own range so // it frees its slot without committing anything; the straggler's range then - // times out and re-queues. Because the timeout backoff is local to the + // times out and re-queues. Because timeout recovery is local to the // straggler, the healthy peer must keep being offered the re-queued shared // work rather than the whole download stalling behind the straggler. let healthy = peer_b.clone(); @@ -2183,7 +2208,101 @@ async fn reactor_timeout_backoff_is_local_and_healthy_peer_keeps_filling() { ); assert!( healthy_offers >= 2, - "the healthy peer was filled repeatedly despite the slow peer's timeout backoff" + "the healthy peer was filled repeatedly despite the slow peer's timeout recovery" + ); + + reactor_task.abort(); +} + +#[tokio::test] +async fn block_liveness_disconnects_silent_peer_and_traces_reason() { + let mut capture = + TraceCapture::for_test("block_liveness_disconnects_silent_peer_and_traces_reason") + .expect("trace capture initializes"); + let mut config = immediate_body_download_config(); + config.fanout = 1; + config.request_timeout = Duration::from_millis(400); + config.max_inflight_block_bytes = BS_PER_BLOCK_WORST_CASE_BYTES * 64; + + let (tip_tx, tip_rx) = watch::channel((block::Height(0), block::Hash([0; 32]))); + let mut startup = BlockSyncStartup::new( + BlockSyncFrontiers { + finalized_height: block::Height(0), + verified_block_tip: block::Height(0), + verified_block_hash: block::Hash([0; 32]), + }, + (block::Height(0), block::Hash([0; 32])), + tip_rx, + config.clone(), + ); + startup.trace = ZakuraTrace::new(capture.tracer(), "01"); + let (handle, mut actions, reactor_task) = spawn_block_sync_reactor(startup); + let service = BlockSyncService::new_with_handle_for_test(config, handle.clone()); + + let peer = peer(0x51); + let (inbound_tx, inbound_rx) = framed_channel(16); + let (outbound_tx, mut outbound_rx) = framed_channel(16); + let streams = HashMap::from([(ZAKURA_STREAM_BLOCK_SYNC, (inbound_rx, outbound_tx))]); + let connection_cancel = CancellationToken::new(); + service.add_peer(Peer::new_with_direction( + peer.clone(), + None, + ZAKURA_CAP_BLOCK_SYNC, + ServicePeerDirection::Outbound, + streams, + connection_cancel.clone(), + )); + wait_for_outbound_status(&mut outbound_rx).await; + + inbound_tx + .send( + BlockSyncMessage::Status(BlockSyncStatus { + servable_low: block::Height(1), + servable_high: block::Height(1), + tip_hash: block::Hash([1; 32]), + max_blocks_per_response: 1, + max_inflight_requests: 1, + max_response_bytes: MAX_BS_RESPONSE_BYTES, + }) + .encode_frame() + .expect("status encodes"), + ) + .await + .expect("status frame queues"); + + tip_tx + .send((block::Height(1), block::Hash([1; 32]))) + .expect("tip watch is live"); + wait_for_query_needed_blocks(&mut actions, block::Height(0), block::Height(1)).await; + + handle + .send(BlockSyncEvent::NeededBlocks(vec![BlockSyncBlockMeta { + height: block::Height(1), + hash: block::Hash([1; 32]), + size: BlockSizeEstimate::Advertised(1_000), + }])) + .await + .expect("needed metadata queues"); + + let (start_height, count) = wait_for_outbound_getblocks(&mut outbound_rx).await; + assert_eq!(start_height, block::Height(1)); + assert_eq!(count, 1); + + tokio::time::timeout(Duration::from_secs(3), connection_cancel.cancelled()) + .await + .expect("silent active peer is disconnected by block-progress liveness"); + + capture.flush().await; + let reader = capture.reader().expect("trace rows load"); + reader.table("block_sync").assert_row( + bs_trace::BLOCK_PEER_PROTOCOL_REJECT, + &[ + ( + bs_trace::REASON, + TraceValue::Str("block_sync_no_block_progress"), + ), + (bs_trace::OUTSTANDING, TraceValue::U64(1)), + ], ); reactor_task.abort(); @@ -2563,51 +2682,6 @@ fn shed_top_until_available_self_funds_floor_reservation() { ); } -#[test] -fn window_reduction_uses_consecutive_timeout_streak() { - let mut window = download_window(); - window.max_inflight_requests = 256; - window.outbound_request_window = 256; - - for _ in 0..15 { - assert_eq!( - window.reduce_outbound_window_after_timeout(), - TimeoutBackoffOutcome::KeepPeer - ); - } - assert_eq!(window.outbound_request_window, 256); - assert_eq!( - window.reduce_outbound_window_after_timeout(), - TimeoutBackoffOutcome::KeepPeer - ); - assert_eq!(window.outbound_request_window, 248); - - for _ in 0..16 { - assert_eq!( - window.reduce_outbound_window_after_timeout(), - TimeoutBackoffOutcome::KeepPeer - ); - } - assert_eq!(window.outbound_request_window, 192); - - // A successful response resets the timeout streak, so the next timeout starts - // a fresh cubic backoff from the current window instead of continuing the old - // streak. - window.increase_outbound_window_after_success(); - for _ in 0..15 { - assert_eq!( - window.reduce_outbound_window_after_timeout(), - TimeoutBackoffOutcome::KeepPeer - ); - } - assert_eq!(window.outbound_request_window, 192); - assert_eq!( - window.reduce_outbound_window_after_timeout(), - TimeoutBackoffOutcome::KeepPeer - ); - assert_eq!(window.outbound_request_window, 184); -} - // ---- Sequencer commit pipeline ---- fn test_sequencer(verified_tip: u32, submitted_apply_limit: usize) -> Sequencer { @@ -2956,14 +3030,35 @@ fn outstanding_three_block_range(budget: &mut ByteBudget) -> OutstandingBlockRan ], }; assert!(budget.try_reserve(request.estimated_bytes)); + let now = Instant::now(); OutstandingBlockRange { request, - queued_at: Instant::now(), - deadline: Instant::now(), + queued_at: now, + deadline: now, + delivery_snapshot: test_delivery_snapshot(now), + delivered_bytes: 0, received: ReceivedBlockTracker::default(), } } +#[test] +fn outstanding_range_accumulates_delivered_bytes_for_bbr_sample() { + let mut budget = ByteBudget::new(THREE_BLOCK_ESTIMATE * 3); + let mut outstanding = outstanding_three_block_range(&mut budget); + + for (height, bytes) in [ + (block::Height(1), 700), + (block::Height(2), 800), + (block::Height(3), 900), + ] { + outstanding.record_body_bytes(bytes); + outstanding.mark_received(height); + } + + assert!(outstanding.is_complete()); + assert_eq!(outstanding.delivered_bytes, 700 + 800 + 900); +} + #[test] fn block_budget_ledger_settles_and_releases_current_charge() { let mut under = BlockBudgetLedger::reserved(1_000); @@ -3204,6 +3299,44 @@ fn budget_reservation_never_exceeds_max_and_only_shrinks_per_block() { } } +/// A request range at the maximum advertised block count (128) fills the +/// `ReceivedBlockTracker`'s `u128` bitset exactly (offsets `0..=127`): every height — +/// including the top-bit height 128 — must be markable, complete, and fully released, +/// so no height silently falls off the end of the bitset. Guards the boundary that the +/// `MAX_BS_BLOCKS_PER_REQUEST <= u128::BITS` const assertion in `state.rs` protects. +#[test] +fn received_tracker_handles_a_full_range_at_the_bitset_boundary() { + let count = MAX_BS_BLOCKS_PER_REQUEST; + assert_eq!(count, u128::BITS, "the cap is sized to the bitset width"); + // Heights `1..=128`, offsets `0..=127`; the helper keeps heights within `u8`. + let mut outstanding = window_request_range(1, count); + assert_eq!(outstanding.reserved_bytes(), u64::from(count)); + + for height in 1..=count { + assert!( + !outstanding.has_received(block::Height(height)), + "height {height} should start unreceived", + ); + outstanding.mark_received(block::Height(height)); + assert!( + outstanding.has_received(block::Height(height)), + "height {height} (offset {}) must be markable — the bitset must cover the \ + whole range", + height - 1, + ); + } + + assert!( + outstanding.is_complete(), + "a fully-received {count}-block range must report complete", + ); + assert_eq!( + outstanding.reserved_bytes(), + 0, + "every height received ⇒ no reserved bytes remain (no offset fell off the bitset)", + ); +} + /// A body whose actual serialized size exceeds its advertised size hint is still /// accepted and buffered, and the byte budget charges the overshoot so it cannot /// issue more work while under-counting held bodies. @@ -3227,10 +3360,13 @@ fn underestimated_body_is_buffered_and_charges_budget_delta() { }], }; assert!(budget.try_reserve(request.estimated_bytes)); + let now = Instant::now(); let mut outstanding = OutstandingBlockRange { request, - queued_at: Instant::now(), - deadline: Instant::now(), + queued_at: now, + deadline: now, + delivery_snapshot: test_delivery_snapshot(now), + delivered_bytes: 0, received: ReceivedBlockTracker::default(), }; assert_eq!(budget.reserved(), hint); diff --git a/zebra-network/src/zakura/testkit/blocksync_fuzz/invariants.rs b/zebra-network/src/zakura/testkit/blocksync_fuzz/invariants.rs index b8870a58212..213cf5590f7 100644 --- a/zebra-network/src/zakura/testkit/blocksync_fuzz/invariants.rs +++ b/zebra-network/src/zakura/testkit/blocksync_fuzz/invariants.rs @@ -25,6 +25,20 @@ pub(crate) struct InvariantReport { pub(crate) final_budget_reserved: u64, /// Liveness-reaper / protocol-reject disconnects observed. pub(crate) protocol_rejects: usize, + /// `block_get_blocks_sent` requests issued via the floor bypass (a floor request + /// sent while the peer was saturated at its BBR cwnd). + pub(crate) floor_bypass_requests: usize, + /// Peak per-peer byte cwnd observed on a `block_body_received` row (`bbr_cwnd_bytes`, + /// emitted only under the byte unit). `0` means the field never appeared (blocks + /// unit, or no completed deliveries). + pub(crate) peak_cwnd_bytes: u64, + /// Peak per-peer in-flight reserved bytes observed (`bbr_inflight_bytes`). + pub(crate) peak_inflight_bytes: u64, + /// Peak per-peer derived byte→request capacity observed (`bbr_cwnd`, the byte cwnd + /// divided by a representative body). Under the byte unit this scales as + /// `cwnd_bytes / body_size`, so it is the clean signal that request depth tracks + /// the inverse of body size. + pub(crate) peak_cwnd_requests: u64, } /// Extract the report from a flushed trace reader. @@ -54,6 +68,34 @@ pub(crate) fn report(reader: &TraceReader) -> InvariantReport { let protocol_rejects = reader .table("block_sync") .count("block_peer_protocol_reject"); + let body_rows: Vec<&Value> = reader + .table("block_sync") + .rows() + .into_iter() + .filter(|row| event(row) == Some("block_body_received")) + .collect(); + let floor_bypass_requests = reader + .table("block_sync") + .rows() + .into_iter() + .filter(|row| event(row) == Some("block_get_blocks_sent")) + .filter(|row| u64_field(row, "floor_bypass") == Some(1)) + .count(); + let peak_cwnd_bytes = body_rows + .iter() + .filter_map(|row| u64_field(row, "bbr_cwnd_bytes")) + .max() + .unwrap_or(0); + let peak_inflight_bytes = body_rows + .iter() + .filter_map(|row| u64_field(row, "bbr_inflight_bytes")) + .max() + .unwrap_or(0); + let peak_cwnd_requests = body_rows + .iter() + .filter_map(|row| u64_field(row, "bbr_cwnd")) + .max() + .unwrap_or(0); InvariantReport { state_samples: state_rows.len(), @@ -61,6 +103,10 @@ pub(crate) fn report(reader: &TraceReader) -> InvariantReport { peak_budget_reserved, final_budget_reserved, protocol_rejects, + floor_bypass_requests, + peak_cwnd_bytes, + peak_inflight_bytes, + peak_cwnd_requests, } } @@ -104,6 +150,19 @@ pub(crate) fn assert_core( report.max_outstanding, outstanding_bound, ); + + // The global byte budget is never over-committed: peak reserved download bytes + // (in-flight + reorder + applying) must stay within the configured ceiling. Every + // per-peer routine reserves against the same CAS-guarded `ByteBudget`, so this must + // hold no matter how many peers race — the memory bound the spec requires. Vacuous + // only for scenarios that set an effectively unbounded budget (`u64::MAX`); the + // tight-ceiling scenarios make it bite. + assert!( + report.peak_budget_reserved <= scenario.config.max_inflight_block_bytes, + "peak reserved bytes {} exceeded the global in-flight byte budget {}", + report.peak_budget_reserved, + scenario.config.max_inflight_block_bytes, + ); } fn event(row: &Value) -> Option<&str> { diff --git a/zebra-network/src/zakura/testkit/blocksync_fuzz/mod.rs b/zebra-network/src/zakura/testkit/blocksync_fuzz/mod.rs index 9e706efe25f..99641cc0b51 100644 --- a/zebra-network/src/zakura/testkit/blocksync_fuzz/mod.rs +++ b/zebra-network/src/zakura/testkit/blocksync_fuzz/mod.rs @@ -104,6 +104,7 @@ pub(crate) async fn run_scenario( corpus.clone(), target, apply.clone(), + scenario.commit, committed_tx, shutdown.clone(), )); @@ -135,7 +136,7 @@ pub(crate) async fn run_scenario( )); } - let _running = RunningHarness { + let running = RunningHarness { shutdown: shutdown.clone(), reactor_task, tasks, @@ -154,6 +155,7 @@ pub(crate) async fn run_scenario( .and_then(|result| result.ok()) .map(|height| *height); let committed_tip = reached.unwrap_or_else(|| *committed_rx.borrow()); + running.stop().await; Ok(FuzzOutcome { committed_tip, @@ -181,6 +183,26 @@ impl Drop for RunningHarness { } } +impl RunningHarness { + async fn stop(mut self) { + self.shutdown.cancel(); + stop_task(&mut self.reactor_task).await; + for task in &mut self.tasks { + stop_task(task).await; + } + } +} + +async fn stop_task(task: &mut JoinHandle<()>) { + if tokio::time::timeout(Duration::from_secs(2), &mut *task) + .await + .is_err() + { + task.abort(); + let _ = task.await; + } +} + /// Answers the reactor's actions from the corpus and mock apply frontier. fn spawn_action_driver( handle: BlockSyncHandle, @@ -188,10 +210,12 @@ fn spawn_action_driver( corpus: SyntheticBlockCorpus, target: block::Height, apply: MockApplyFrontier, + commit: CommitProfile, committed_tx: watch::Sender, shutdown: CancellationToken, ) -> JoinHandle<()> { tokio::spawn(async move { + let mut applied = 0u64; loop { let action = tokio::select! { _ = shutdown.cancelled() => break, @@ -236,6 +260,15 @@ fn spawn_action_driver( } } BlockSyncAction::SubmitBlock { token, block } => { + // Model a slow/bursty commit drain: hold the submitted body before + // applying so its reserved bytes stay held until + // `BlockApplyFinished`, letting the apply backlog build against the + // byte budget. + if !commit.per_commit_delay.is_zero() + && sleep_or_cancel(&shutdown, commit.per_commit_delay).await + { + break; + } let height = block .coinbase_height() .expect("synthetic submitted block has height"); @@ -256,6 +289,16 @@ fn spawn_action_driver( { break; } + applied = applied.saturating_add(1); + if let Some(burst) = commit.burst { + if burst.every_commits > 0 + && applied.is_multiple_of(burst.every_commits) + && !burst.duration.is_zero() + && sleep_or_cancel(&shutdown, burst.duration).await + { + break; + } + } } BlockSyncAction::Misbehavior { .. } => {} } diff --git a/zebra-network/src/zakura/testkit/blocksync_fuzz/peer.rs b/zebra-network/src/zakura/testkit/blocksync_fuzz/peer.rs index 45e8def8bbd..edfcf33e686 100644 --- a/zebra-network/src/zakura/testkit/blocksync_fuzz/peer.rs +++ b/zebra-network/src/zakura/testkit/blocksync_fuzz/peer.rs @@ -148,12 +148,18 @@ async fn serve_loop( } let mut returned = 0u32; - for (_, block, _) in &blocks { - if !spec.serve.per_block_is_zero() { - let delay = spec.serve.per_block_latency.sample(&mut rng); - if sleep_or_cancel(shutdown, delay).await { - return; + for (_, block, block_bytes) in &blocks { + // Byte-accurate serve when a bandwidth is set: the block's transmission time + // is `bytes / bandwidth`. Otherwise fall back to the fixed per-block latency. + let per_block_delay = match spec.serve.bandwidth_bytes_per_sec { + Some(bandwidth) => Duration::from_secs_f64(*block_bytes as f64 / bandwidth as f64), + None if !spec.serve.per_block_is_zero() => { + spec.serve.per_block_latency.sample(&mut rng) } + None => Duration::ZERO, + }; + if !per_block_delay.is_zero() && sleep_or_cancel(shutdown, per_block_delay).await { + return; } if peer .send(BlockSyncMessage::Block(block.clone())) diff --git a/zebra-network/src/zakura/testkit/blocksync_fuzz/scenario.rs b/zebra-network/src/zakura/testkit/blocksync_fuzz/scenario.rs index 3801fad3570..dafef9ca52d 100644 --- a/zebra-network/src/zakura/testkit/blocksync_fuzz/scenario.rs +++ b/zebra-network/src/zakura/testkit/blocksync_fuzz/scenario.rs @@ -66,8 +66,15 @@ pub(crate) struct ServeProfile { /// trace-proven head-of-line effect emerges from queue depth on top of this. pub(crate) first_block_latency: LatencyDist, /// Delay between blocks inside a response (models a rate-limited serve path; - /// effective serve rate ≈ `1 / per_block_latency`). + /// effective serve rate ≈ `1 / per_block_latency`). Ignored when + /// [`bandwidth_bytes_per_sec`](Self::bandwidth_bytes_per_sec) is set. pub(crate) per_block_latency: LatencyDist, + /// Optional **byte-accurate** serve bandwidth. When set, each block's serve delay is + /// `block_bytes / bandwidth` instead of the fixed `per_block_latency`, so a response + /// takes `first_block_latency + Σ bytes / bandwidth` — the realistic model where a + /// big block genuinely takes longer to transmit. This is what lets the byte-cwnd + /// controller observe a true bytes/sec BtlBw and size-dependent transfer time. + pub(crate) bandwidth_bytes_per_sec: Option, /// Optional periodic stall. pub(crate) idle_gap: Option, /// Probability in `[0, 1]` that a request is silently dropped (no response), @@ -86,6 +93,7 @@ impl ServeProfile { Self { first_block_latency: LatencyDist::zero(), per_block_latency: LatencyDist::zero(), + bandwidth_bytes_per_sec: None, idle_gap: None, drop_probability: 0.0, withhold: None, @@ -102,6 +110,17 @@ impl ServeProfile { } } + /// A byte-accurate peer: a fixed base RTT plus a finite serve `bandwidth` (bytes/sec), + /// so each block takes `bytes / bandwidth` to transmit. This is the model the + /// byte-cwnd controller is meant to track — `elapsed ≈ base_rtt + bytes / bandwidth`. + pub(crate) fn byte_rate(base_rtt: Duration, bandwidth_bytes_per_sec: u64) -> Self { + Self { + first_block_latency: LatencyDist::Fixed(base_rtt), + bandwidth_bytes_per_sec: Some(bandwidth_bytes_per_sec.max(1)), + ..Self::fast() + } + } + pub(crate) fn first_block_is_zero(&self) -> bool { self.first_block_latency.is_zero() } @@ -111,6 +130,32 @@ impl ServeProfile { } } +/// How the harness's mock commit pipeline drains the applyQ. +/// +/// The default applies each contiguous body instantly. A stall profile injects a steady +/// per-commit delay and/or a periodic burst stall, modelling a slow/bursty commit drain +/// (the trace-proven 27–53 s `commit_finish` tails). Because the commit driver only +/// releases the byte budget once it reports the durable frontier *after* applying, a slow +/// drain lets the apply backlog (and the reserved bytes that bound it) build — so a run +/// can prove the queue is bounded by the memory ceiling, not by throttling download. +#[derive(Clone, Copy, Debug, Default)] +pub(crate) struct CommitProfile { + /// Fixed delay applied before each body is committed (steady slow commit). + pub(crate) per_commit_delay: Duration, + /// Optional periodic burst stall: every `every_commits` applied bodies, pause for + /// `duration` before continuing (the sawtooth the durable-watch must absorb). + pub(crate) burst: Option, +} + +/// A periodic burst stall in the mock commit pipeline. +#[derive(Clone, Copy, Debug)] +pub(crate) struct CommitBurstStall { + /// Pause after every this-many applied bodies. + pub(crate) every_commits: u64, + /// How long each pause lasts. + pub(crate) duration: Duration, +} + /// One synthetic peer the node downloads from. #[derive(Clone, Copy, Debug)] pub(crate) struct PeerSpec { @@ -218,6 +263,8 @@ pub(crate) struct Scenario { pub(crate) peers: Vec, /// Timed frontier changes (header growth, reanchor, verified reset). pub(crate) timeline: Vec, + /// How the mock commit pipeline drains the applyQ (default: instant). + pub(crate) commit: CommitProfile, /// Wall-clock bound for the run. pub(crate) deadline: Duration, } @@ -239,6 +286,7 @@ impl Scenario { config, peers, timeline: Vec::new(), + commit: CommitProfile::default(), deadline: Duration::from_secs(30), } } @@ -261,12 +309,21 @@ impl FuzzOutcome { /// A default block-sync config for harness runs: generous byte budget (memory is not /// the constraint under test by default), moderate per-response/inflight caps. +/// +/// The BBR ProbeRTT cadence is scaled down to sub-second so the mechanism is exercised +/// within a fuzzer run's compressed wall-clock (production defaults are 10 s / 200 ms, +/// which never fire in a ~1 s run). `rtprop_window` matches the probe interval so a +/// stale (queue-inflated) RTprop sample ages out one interval after the probe that +/// replaced it. pub(crate) fn fuzz_config() -> ZakuraBlockSyncConfig { ZakuraBlockSyncConfig { max_blocks_per_response: 16, max_inflight_requests: 256, max_inflight_block_bytes: u64::MAX, request_timeout: Duration::from_secs(30), + bbr_probe_rtt_interval: Duration::from_millis(150), + bbr_probe_rtt_duration: Duration::from_millis(30), + bbr_rtprop_window: Duration::from_millis(150), ..ZakuraBlockSyncConfig::default() } } diff --git a/zebra-network/src/zakura/testkit/blocksync_fuzz/tests.rs b/zebra-network/src/zakura/testkit/blocksync_fuzz/tests.rs index c087e988812..c9daccb7c5c 100644 --- a/zebra-network/src/zakura/testkit/blocksync_fuzz/tests.rs +++ b/zebra-network/src/zakura/testkit/blocksync_fuzz/tests.rs @@ -10,9 +10,9 @@ use std::time::Duration; use zebra_chain::block; use super::{ - assert_core_invariants, fuzz_config, invariant_report, run_scenario, run_trace, FuzzOutcome, - IdleGap, InvariantReport, LatencyDist, PeerSpec, Scenario, ServeProfile, TipEvent, - TipEventKind, + assert_core_invariants, fuzz_config, invariant_report, run_scenario, run_trace, + CommitBurstStall, CommitProfile, FuzzOutcome, IdleGap, InvariantReport, LatencyDist, PeerSpec, + Scenario, ServeProfile, TipEvent, TipEventKind, }; use crate::zakura::ZakuraBlockSyncConfig; @@ -41,6 +41,7 @@ async fn run_checked( peak_budget_reserved = report.peak_budget_reserved, final_budget_reserved = report.final_budget_reserved, protocol_rejects = report.protocol_rejects, + floor_bypass_requests = report.floor_bypass_requests, "blocksync fuzz scenario complete", ); assert_core_invariants(&scenario, &outcome, &report, outstanding_slack); @@ -78,29 +79,73 @@ async fn fuzz_steady() { run_checked("fuzz_steady", scenario, 32).await; } -/// Head-of-line: one slow, high-latency peer alongside fast peers. The trace-proven -/// regime the BBR work targets. For now we only require convergence; once BBR lands we -/// compare the per-peer queue depth / HoL latency in the report across controllers. +/// Steady state under the byte cwnd unit: the controller budgets in-flight +/// work by reserved body bytes instead of request count. End-to-end seam check — the +/// byte-denominated `available_slots` gate must still drive the real reactor to the tip +/// without stalling. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn fuzz_steady_bytes_unit() { + let blocks = 300; + let config = ZakuraBlockSyncConfig { + bbr_cwnd_unit: crate::zakura::CwndUnit::Bytes, + ..fuzz_config() + }; + let scenario = Scenario::new( + blocks, + 0x57ea_0008, + config, + vec![ + PeerSpec::fast(1, target(blocks)), + PeerSpec::fast(2, target(blocks)), + PeerSpec::fast(3, target(blocks)), + ], + ); + run_checked("fuzz_steady_bytes_unit", scenario, 32).await; +} + +/// Head-of-line: one genuinely slow, high-RTprop peer alongside two fast byte-accurate +/// carriers, in the single-block-per-request production regime. The floor must ride the +/// carriers (the slow peer's higher RTprop defers it off the floor on the normal take +/// path) while the slow peer is **kept, not reaped** — it serves a body roughly every +/// 0.5 s, well inside the liveness window, so disconnecting it would throw away real +/// bandwidth for no floor benefit. Asserts convergence and zero reaper disconnects; the +/// floor-HoL p99 itself is the live-trace metric. #[tokio::test(flavor = "multi_thread", worker_threads = 4)] async fn fuzz_one_slow_peer_hol() { let blocks = 300; + let config = ZakuraBlockSyncConfig { + max_blocks_per_response: 1, + ..fuzz_config() + }; + // 64 KiB/s with an 80 ms base RTT ⇒ ~0.5 s per 32 KiB body and a measured RTprop far + // above the carriers', so the normal-path floor preference defers the floor to them. let slow = PeerSpec::with_serve( 1, target(blocks), - ServeProfile::slow(Duration::from_millis(20), Duration::from_millis(5)), + ServeProfile::byte_rate(Duration::from_millis(80), 64 * 1024), ); + let carrier = |id| { + PeerSpec::with_serve( + id, + target(blocks), + ServeProfile::byte_rate(Duration::from_millis(2), 50 * 1024 * 1024), + ) + }; let mut scenario = Scenario::new( blocks, 0x57ea_0002, - fuzz_config(), - vec![ - slow, - PeerSpec::fast(2, target(blocks)), - PeerSpec::fast(3, target(blocks)), - ], + config, + vec![slow, carrier(2), carrier(3)], ); + scenario.target_block_bytes = Some(32 * 1024); scenario.deadline = Duration::from_secs(60); - run_checked("fuzz_one_slow_peer_hol", scenario, 32).await; + let (_, report) = run_checked("fuzz_one_slow_peer_hol", scenario, 32).await; + + // Directive #1: a peer delivering at a slow-but-steady cadence is never kicked. + assert_eq!( + report.protocol_rejects, 0, + "the slow-but-progressing peer must not be reaped (it serves ~every 0.5 s)", + ); } /// Reorg: a mid-sync verified-tip reset, then sync resumes to the target. Stretched @@ -233,3 +278,321 @@ async fn fuzz_large_to_small() { scenario.deadline = Duration::from_secs(60); run_checked("fuzz_large_to_small", scenario, 32).await; } + +/// A byte-cwnd config whose window binds on reserved body bytes rather than the +/// request-count cap: a `min_cwnd_bytes` floor chosen below `max_inflight × body`, with +/// the generous `fuzz_config` byte budget. +/// +/// Forces **one block per request** (`max_blocks_per_response = 1`), the regime the byte +/// controller is designed for and the production default — with multi-block ranges a +/// single request can reserve many bodies at once, so the per-request byte cwnd would no +/// longer bound the in-flight bytes (the request *count* is what the cwnd gates). +fn byte_window_config(min_cwnd_bytes: u64) -> ZakuraBlockSyncConfig { + ZakuraBlockSyncConfig { + bbr_cwnd_unit: crate::zakura::CwndUnit::Bytes, + bbr_min_cwnd_bytes: min_cwnd_bytes, + max_blocks_per_response: 1, + ..fuzz_config() + } +} + +/// Run one byte-unit scenario with a fixed body size and two byte-accurate peers, +/// returning its invariant report. +async fn run_byte_size_run( + name: &str, + seed: u64, + blocks: u32, + body_bytes: usize, + config: ZakuraBlockSyncConfig, + serve: ServeProfile, +) -> InvariantReport { + let mut scenario = Scenario::new( + blocks, + seed, + config, + vec![ + PeerSpec::with_serve(1, target(blocks), serve), + PeerSpec::with_serve(2, target(blocks), serve), + ], + ); + scenario.target_block_bytes = Some(body_bytes); + scenario.deadline = Duration::from_secs(30); + let (_, report) = run_checked(name, scenario, 64).await; + report +} + +/// Mixed block sizes: the byte cwnd holds ~the same reserved bytes in flight regardless +/// of body size, so the in-flight *request* depth must scale inversely with the body +/// size — small bodies pack many requests into the byte window, large bodies few. Two +/// runs that differ only in body size make the headline byte-cwnd property a direct, +/// deterministic comparison (the blocks unit, by contrast, would hold the same request +/// count in both). +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn fuzz_mixed_block_sizes() { + let blocks = 400; + let config = byte_window_config(512 * 1024); + // Byte-accurate serve: each block takes `bytes / 20 MiB/s`, so a big block genuinely + // takes longer and the controller sees a real bytes/sec BtlBw. + let serve = ServeProfile::byte_rate(Duration::from_millis(2), 20 * 1024 * 1024); + + let small = run_byte_size_run( + "fuzz_mixed_block_sizes_small", + 0x57ea_0009, + blocks, + 8 * 1024, + config.clone(), + serve, + ) + .await; + let large = run_byte_size_run( + "fuzz_mixed_block_sizes_large", + 0x57ea_000a, + blocks, + 64 * 1024, + config, + serve, + ) + .await; + + // The byte cwnd (a similar order of magnitude in both runs — the budget is in bytes, + // not requests) maps to far more in-flight *requests* for small bodies than for large: + // request depth ∝ 1 / body_size, the headline byte-cwnd property the blocks unit + // cannot express. (Here ~8× the body size ⇒ a multiple-× request-depth gap.) + assert!( + small.peak_cwnd_requests > large.peak_cwnd_requests.saturating_mul(2), + "small bodies should admit many more single-block requests in a comparable byte \ + window (small={} reqs @ {} B cwnd, large={} reqs @ {} B cwnd)", + small.peak_cwnd_requests, + small.peak_cwnd_bytes, + large.peak_cwnd_requests, + large.peak_cwnd_bytes, + ); + // The byte window bounds in-flight memory: with one block per request the peak + // reserved bytes track the byte cwnd (plus a small floor-bypass margin), never + // ballooning to many multiples of it — the head-of-line bound byte denomination + // exists to provide. (A multi-block-range regression would push this well past 2×.) + for (label, run) in [("small", &small), ("large", &large)] { + let bound = run.peak_cwnd_bytes.saturating_mul(2); + assert!( + run.peak_inflight_bytes <= bound, + "{label}: in-flight reserved bytes {} must stay bounded by ~the byte cwnd \ + (cwnd={} B, bound={} B)", + run.peak_inflight_bytes, + run.peak_cwnd_bytes, + bound, + ); + } +} + +/// Commit stall: the mock commit pipeline drains slowly and in bursts while fast peers +/// keep serving. This is the `BLOCKSYNC_BYTE_CWND_PLAN` step-3 system check — the apply +/// backlog must be bounded **only** by the byte budget (download parks on the memory +/// ceiling, not on a commit-coupled throttle), and the verified tip must resume the +/// instant each stall clears so the run still reaches the target. +/// +/// `run_checked` already asserts the target is reached (vtip resumes; no commit-induced +/// wedge). On top of that we assert the peak reserved bytes stayed within the configured +/// ceiling (the queue did not grow toward the full chain) yet the ceiling was actually +/// exercised (the stall created real backpressure — the bound is not vacuous). +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn fuzz_commit_stall() { + let blocks = 400; + let body_bytes = 32 * 1024usize; + // A small resident download budget so the commit stall makes it bind: the chain is + // ~12.8 MB (400 × 32 KiB) against a 2 MiB ceiling, so the budget must recycle ~6× and + // download genuinely waits on commit. Kept below the request-count cap's byte + // equivalent (2 peers × 64 reqs × 32 KiB = 4 MiB) so the *byte budget*, not the + // request count, is the binding constraint. (The fuzzer reactor path does not call + // `validate()`, which would otherwise require a full checkpoint range; the synthetic + // bodies are tiny and the mock committer needs no checkpoint batch, so a sub-floor + // ceiling is the right knob to exercise byte-budget backpressure here.) + let byte_ceiling: u64 = 2 * 1024 * 1024; + let config = ZakuraBlockSyncConfig { + max_inflight_block_bytes: byte_ceiling, + max_blocks_per_response: 1, + ..fuzz_config() + }; + let mut scenario = Scenario::new( + blocks, + 0x57ea_000c, + config, + vec![ + PeerSpec::fast(1, target(blocks)), + PeerSpec::fast(2, target(blocks)), + ], + ); + scenario.target_block_bytes = Some(body_bytes); + // Steady 1 ms/commit plus a 120 ms burst every 40 commits — a slow, sawtoothing drain + // that holds the byte budget full between bursts without ever permanently wedging. + scenario.commit = CommitProfile { + per_commit_delay: Duration::from_millis(1), + burst: Some(CommitBurstStall { + every_commits: 40, + duration: Duration::from_millis(120), + }), + }; + scenario.deadline = Duration::from_secs(60); + let (_, report) = run_checked("fuzz_commit_stall", scenario, 64).await; + + // The byte budget is the only bound on the apply backlog: peak reserved bytes stay + // within the ceiling plus a small floor-bypass / request-boundary margin — i.e. the + // queue did not grow toward the 12.8 MB chain despite the commit stall. + let margin = (body_bytes as u64).saturating_mul(16); + let bound = byte_ceiling.saturating_add(margin); + assert!( + report.peak_budget_reserved <= bound, + "reserved bytes {} must stay within the {} B memory ceiling (+{} B margin); the \ + apply backlog is bounded by the byte budget, not unbounded by the commit stall", + report.peak_budget_reserved, + byte_ceiling, + margin, + ); + // Non-vacuous: the stall actually filled the budget (otherwise the bound proves + // nothing about backpressure). + assert!( + report.peak_budget_reserved >= byte_ceiling / 2, + "the commit stall should have created real byte-budget backpressure (reserved \ + {} of the {} B ceiling)", + report.peak_budget_reserved, + byte_ceiling, + ); + tracing::info!( + peak_budget_reserved = report.peak_budget_reserved, + final_budget_reserved = report.final_budget_reserved, + byte_ceiling, + "commit_stall byte-budget backpressure observation", + ); +} + +/// A single high-bandwidth peer with real headroom, served byte-accurately, under the +/// byte unit. The controller must drive a clean sync to the tip while keeping the byte +/// window the binding constraint — a per-peer byte cwnd is traced and the in-flight +/// reserved bytes track it (the controller reasons in bytes, not request slots). +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn fuzz_high_bw_fast_peer() { + let blocks = 500; + let config = byte_window_config(256 * 1024); + let peer = PeerSpec::with_serve( + 1, + target(blocks), + // 100 MiB/s link with a 20 ms base RTT: enough headroom that the byte-BDP can + // exceed the floor once concurrency builds. + ServeProfile::byte_rate(Duration::from_millis(20), 100 * 1024 * 1024), + ); + let mut scenario = Scenario::new(blocks, 0x57ea_000b, config, vec![peer]); + scenario.target_block_bytes = Some(16 * 1024); + scenario.deadline = Duration::from_secs(30); + let (_, report) = run_checked("fuzz_high_bw_fast_peer", scenario, 64).await; + + // A per-peer byte cwnd was traced (byte denomination is live) and never dipped below + // its floor; the in-flight reserved bytes were tracked too. + assert!( + report.peak_cwnd_bytes >= 256 * 1024, + "byte cwnd should be traced at or above the floor, got {} B", + report.peak_cwnd_bytes, + ); + assert!( + report.peak_inflight_bytes > 0, + "byte in-flight occupancy should be traced", + ); + tracing::info!( + peak_cwnd_bytes = report.peak_cwnd_bytes, + peak_inflight_bytes = report.peak_inflight_bytes, + max_outstanding = report.max_outstanding, + "high_bw_fast_peer byte-window observation", + ); +} + +/// Lossy peer: a peer silently drops ~30% of requests (no response at all), forcing the +/// node's request-timeout / re-request path, while a covering fast peer can serve every +/// height. The node must route around the drops and still commit a contiguous, correct +/// prefix to the target. Drives the `drop_probability` serve knob no other scenario sets. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn fuzz_lossy_peer() { + let blocks = 300; + let lossy = PeerSpec::with_serve( + 1, + target(blocks), + ServeProfile { + drop_probability: 0.3, + ..ServeProfile::fast() + }, + ); + let mut scenario = Scenario::new( + blocks, + 0x57ea_000d, + // Short request timeout so dropped requests are re-requested well within the run. + retry_config(), + vec![lossy, PeerSpec::fast(2, target(blocks))], + ); + scenario.deadline = Duration::from_secs(60); + run_checked("fuzz_lossy_peer", scenario, 32).await; +} + +/// Reverse-order serving: peers return the blocks of each multi-block response high→low, +/// exercising out-of-order body arrival and the reorder buffer. `fuzz_config`'s +/// `max_blocks_per_response = 16` lets the node issue multi-block ranges, so the reversal +/// is non-trivial. The node must still commit a contiguous, hash-correct prefix to the +/// target. Drives the `reorder` serve knob no other scenario sets. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn fuzz_reorder() { + let blocks = 300; + let reorder_serve = ServeProfile { + reorder: true, + ..ServeProfile::fast() + }; + let scenario = Scenario::new( + blocks, + 0x57ea_000e, + fuzz_config(), + vec![ + PeerSpec::with_serve(1, target(blocks), reorder_serve), + PeerSpec::with_serve(2, target(blocks), reorder_serve), + ], + ); + run_checked("fuzz_reorder", scenario, 32).await; +} + +/// Many peers racing against a tight global byte budget: every per-peer routine reserves +/// against the one shared `ByteBudget`, so this stresses the concurrent reservation path +/// end-to-end. A steady slow commit keeps the shared budget full so it actually binds +/// while eight routines reserve concurrently. `assert_core` asserts +/// `peak_budget_reserved` never exceeds the configured ceiling (the global memory bound +/// under multi-peer contention — the spec's "concurrent reservations MUST NOT +/// over-commit"); here we additionally assert the ceiling was genuinely approached, so +/// that bound is not vacuous. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn fuzz_multi_peer_tight_budget() { + let blocks = 400; + let body_bytes = 32 * 1024usize; + // ~12.8 MB chain against a 3 MiB ceiling ⇒ the shared budget recycles ~4× and binds. + let byte_ceiling: u64 = 3 * 1024 * 1024; + let config = ZakuraBlockSyncConfig { + max_inflight_block_bytes: byte_ceiling, + max_blocks_per_response: 1, + ..fuzz_config() + }; + let peers: Vec<_> = (1u8..=8) + .map(|id| PeerSpec::fast(id, target(blocks))) + .collect(); + let mut scenario = Scenario::new(blocks, 0x57ea_000f, config, peers); + scenario.target_block_bytes = Some(body_bytes); + // A steady slow commit so the budget stays full and binds; instant commit would keep + // it nearly empty and make the bound vacuous. + scenario.commit = CommitProfile { + per_commit_delay: Duration::from_millis(1), + burst: None, + }; + scenario.deadline = Duration::from_secs(60); + let (_, report) = run_checked("fuzz_multi_peer_tight_budget", scenario, 128).await; + + // Non-vacuous: the tight ceiling was genuinely approached under 8-peer contention, so + // `assert_core`'s `peak_budget_reserved <= max_inflight_block_bytes` proves a real + // bound rather than an idle budget. + assert!( + report.peak_budget_reserved >= byte_ceiling / 2, + "the tight budget should bind under 8-peer contention (reserved {} of {} B)", + report.peak_budget_reserved, + byte_ceiling, + ); +} diff --git a/zebra-network/src/zakura/trace.rs b/zebra-network/src/zakura/trace.rs index afd6904eb76..eb41822dcb4 100644 --- a/zebra-network/src/zakura/trace.rs +++ b/zebra-network/src/zakura/trace.rs @@ -131,10 +131,12 @@ pub mod block_sync_trace { pub const SEND_ELAPSED_MS: &str = "send_elapsed_ms"; /// Commit result label (`committed`, `duplicate`, `rejected`, `timed_out`). pub const RESULT: &str = "result"; - /// Reactor-local verifier submission token. - pub const APPLY_TOKEN: &str = "apply_token"; /// Bounded reason field. pub const REASON: &str = "reason"; + /// Error detail field. + pub const ERROR: &str = "error"; + /// Block apply token field. + pub const APPLY_TOKEN: &str = "apply_token"; /// Scheduler/query lower bound for block body requests. pub const REQUEST_FLOOR: &str = "request_floor"; /// Highest contiguous body height already submitted for apply. @@ -220,6 +222,8 @@ pub mod block_sync_trace { pub const BLOCK_PEER_CONNECTED: &str = "block_peer_connected"; /// Block-sync peer disconnected from the reactor. pub const BLOCK_PEER_DISCONNECTED: &str = "block_peer_disconnected"; + /// Block-sync peer rejected for a protocol/liveness reason. + pub const BLOCK_PEER_PROTOCOL_REJECT: &str = "block_peer_protocol_reject"; /// Body range request sent to a peer. pub const BLOCK_GET_BLOCKS_SENT: &str = "block_get_blocks_sent"; /// Reactor accepted an inbound event. @@ -252,6 +256,16 @@ pub mod block_sync_trace { pub const BLOCK_WORK_EXTENDED: &str = "block_work_extended"; /// A peer claimed a contiguous chunk of pending work for issuance. pub const BLOCK_WORK_TAKEN: &str = "block_work_taken"; + /// A `try_fill` pass ended having issued no request: the routine went idle this wake. + /// Carries [`FILL_STOP_REASON`] plus the slot/budget/work snapshot so a trace can + /// attribute carrier idle ("bubble") time to a cause rather than inferring it. + pub const BLOCK_FILL_STOP: &str = "block_fill_stop"; + /// Why a `try_fill` pass stopped issuing requests (`no_status` / `cwnd_saturated` / + /// `no_work` / `lookahead_cap` / `retry_avoid` / `budget` / `outbound_full` / + /// `send_error` / `internal`). + pub const FILL_STOP_REASON: &str = "fill_stop_reason"; + /// Requests this `try_fill` pass issued before stopping (0 = a candidate bubble). + pub const FILL_SENT: &str = "fill_sent"; /// Verified body frontier advanced from state. pub const BLOCK_FRONTIERS_CHANGED: &str = "block_frontiers_changed"; /// Chain tip reset rolled the body frontier back. @@ -392,6 +406,18 @@ pub mod commit_state_trace { pub const BEST_HEADER_TIP: &str = "best_header_tip"; /// Elapsed milliseconds field. pub const ELAPSED_MS: &str = "elapsed_ms"; + /// Diagnostic attribution of why a commit was still pending at the stall deadline + /// (see [`COMMIT_STALL_CONTIGUOUS_HEAD`] / [`COMMIT_STALL_BEHIND_PREFIX`]). Purely a + /// trace label — it never gates the commit. + pub const COMMIT_STALL_REASON: &str = "commit_stall_reason"; + /// Highest contiguously-committed height the committer had reached at the stall. + pub const COMMITTED_MARKER: &str = "committed_marker"; + /// Highest height the committer had fired a commit for at the stall (submission + /// high-water — lets analysis see whether the range above the stalled block was + /// even submitted to the verifier yet). + pub const FIRED_HIGH_WATER: &str = "fired_high_water"; + /// Number of fired-but-unresolved commits at the stall (concurrency / batch depth). + pub const COMMITS_IN_FLIGHT: &str = "commits_in_flight"; /// Peer field. pub const PEER: &str = "peer"; /// Queue length field. @@ -419,6 +445,15 @@ pub mod commit_state_trace { pub const COMMIT_START: &str = "commit_start"; /// Verifier commit exceeded the driver timeout but is still being awaited. pub const COMMIT_STALLED: &str = "commit_stalled"; + /// [`COMMIT_STALL_REASON`] value: the committed tip sat immediately below the stalled + /// block, so it was the contiguous head — the gate is downstream of submission (the + /// checkpoint batch above it is still filling in the verifier, or verify+persist on + /// the head is slow). This is the "make commit faster" signal. + pub const COMMIT_STALL_CONTIGUOUS_HEAD: &str = "contiguous_head"; + /// [`COMMIT_STALL_REASON`] value: a lower contiguous block had not committed yet, so + /// the stalled block was blocked behind the un-committed prefix (floor / range + /// head-of-line). + pub const COMMIT_STALL_BEHIND_PREFIX: &str = "behind_committed_prefix"; /// Verifier commit finished. pub const COMMIT_FINISH: &str = "commit_finish"; /// Post-commit frontier query started. diff --git a/zebra-network/src/zakura/transport/guard.rs b/zebra-network/src/zakura/transport/guard.rs index 57d84fa507e..0a8a673e228 100644 --- a/zebra-network/src/zakura/transport/guard.rs +++ b/zebra-network/src/zakura/transport/guard.rs @@ -355,6 +355,88 @@ mod tests { assert_eq!(budget.available(), 50); } + // A `ByteBudget` is cloned and shared across the block-sync Sequencer and every + // per-peer routine, all reserving concurrently against one counter. The reservation + // path must never over-commit the max — the property the CAS loop in `try_reserve` + // exists for. This drives many reservers at a tight budget and asserts the shared + // counter never exceeds the max; a regression to a non-atomic load-modify-store + // would over-commit under this contention. + #[test] + fn byte_budget_concurrent_reservations_never_over_commit() { + use std::sync::atomic::{AtomicUsize, Ordering as AtomicOrdering}; + use std::sync::{Arc, Barrier}; + use std::thread; + + const CHUNK: u64 = 4_096; + const CAP: u64 = 8; + const THREADS: usize = 16; // deliberately oversubscribed: 16 reservers, 8 slots + let max = CHUNK * CAP; + let budget = ByteBudget::new(max); + + // Phase 1 — deterministic oversubscription. Every thread tries to reserve one + // CHUNK simultaneously and holds it across a barrier, so the counter sits at its + // peak with all reservations decided. Exactly CAP may be admitted; never more. + let start = Arc::new(Barrier::new(THREADS)); + let all_reserved = Arc::new(Barrier::new(THREADS)); + let successes = Arc::new(AtomicUsize::new(0)); + let mut handles = Vec::new(); + for _ in 0..THREADS { + let mut budget = budget.clone(); + let start = start.clone(); + let all_reserved = all_reserved.clone(); + let successes = successes.clone(); + handles.push(thread::spawn(move || { + start.wait(); + let ok = budget.try_reserve(CHUNK); + if ok { + successes.fetch_add(1, AtomicOrdering::AcqRel); + } + // All reservations are decided and still held: the counter is at its peak. + all_reserved.wait(); + assert!( + budget.reserved() <= max, + "reserved {} exceeded the budget max {max} (over-commit)", + budget.reserved(), + ); + if ok { + budget.release(CHUNK); + } + })); + } + for handle in handles { + handle.join().expect("reserver thread panicked"); + } + assert_eq!( + u64::try_from(successes.load(AtomicOrdering::Acquire)).unwrap(), + CAP, + "exactly CAP reservations may be admitted; the rest must be rejected", + ); + assert_eq!( + budget.reserved(), + 0, + "every admitted reservation was released" + ); + + // Phase 2 — sustained churn. Many reserve/release rounds under contention, each + // asserting the counter is never over budget. + let mut handles = Vec::new(); + for _ in 0..THREADS { + let mut budget = budget.clone(); + handles.push(thread::spawn(move || { + for _ in 0..5_000 { + if budget.try_reserve(CHUNK) { + assert!(budget.reserved() <= max, "over-commit during churn"); + budget.release(CHUNK); + } + } + })); + } + for handle in handles { + handle.join().expect("churn thread panicked"); + } + assert_eq!(budget.reserved(), 0); + } + #[test] fn admit_rejects_disallowed_type() { let mut guard = SessionGuard::new(ALLOWED, 1_024, None);