WSPR/FT8 dt walks ~0.2 s/hr (recorder RTP→UTC anchor slide) — radiod output proven clean

[rrobinett]

Summary

On B4-100 the WSPR/FT8 decode-window offset (dt, averaged over each cycle's spots) walks monotonically away from true UTC at ~0.2 s/hr while radiod's output stream is provably clean. The dt-guard (0e50f8c) catches it at 1.25 s and re-anchors, so it self-recovers — but the underlying slide means the recorder's RTP→UTC anchor is drifting relative to radiod, and every walk costs a re-anchor + partial cycle.

What it is NOT (ruled out with live data, 2026-07-20)

Measured on the live 20 m + fleet of 38 output channels:

• Not input sample loss — radiod fork RTP↔GPS step monitor: 0 STEPPED events. frontend.ad_over moves (ADC railing) but samples_since_over shows no gaps.
• Not output frame drops — st.filter_drops (metadump [77] "block drops") = 0 new drops/hr fleet-wide over a 4-min window while dt actively walked −0.95 → −1.06. (147 cumulative drops are historical, from a pre-CPU-isolation contention period; radiod stopped dropping once it had its cache pair to itself.)
• Not sample-rate / clock error — output RTP timestamp tracks GPS at 12000.09 Hz vs 12000 nominal (~7.5 ppm ≈ 27 ms/hr), an order of magnitude too small to explain ~0.2 s/hr.
• output_errors = 0, maximum buffered pkts = 0 on all channels; the (rtp_timesnap − output_samples) gap does not drift.

So radiod's output stream — RTP timestamps, GPS discipline, sample counts — is internally consistent and correct. The ~0.2 s/hr walk is introduced downstream, in the recorder's RTP→UTC anchor.

Where it likely lives

RtpSyncStrategy anchors (rtp_timesnap ↔ gps_time via ka9q ChannelInfo) and projects each output sample's UTC from that anchor. The internal guards (abs-divergence check, anchor_epoch step detector) don't catch a slow slide:

• the abs-divergence check compares against the same StatusListener-refreshed ChannelInfo that places the windows, so a co-drifting reference reads ~0 divergence until the slide is large;
• anchor_epoch step detection was removed in ka9q-python 2026-06-28 (vestigial).

Candidate mechanisms to investigate:

1. Stale anchor: is the StatusListener actually re-reading radiod's periodically-republished (gps_time, rtp_timesnap) and re-projecting, or does the recorder anchor once and extrapolate? If it extrapolates on a fixed anchor, any small systematic error accumulates — but note the measured rate error (7.5 ppm) is too small alone, so this can't be the whole story.
2. Window-extraction phase: does the per-period slice re-pin to radiod's live mapping each period, or accumulate a rounding/phase error across periods (samples-to-boundary arithmetic)?
3. A second, larger effective rate mismatch somewhere between the anchor's assumed rate and the RTP advance the extractor uses.

Impact

• Every walk consumes a dt-guard re-anchor (~1 partial cycle × all bands) roughly every few hours.
• Below the guard threshold, spots are uploaded with a growing dt bias (cosmetic for WSPR reporting, but it is a real timing error).

Asks

1. Confirm whether RtpSyncStrategy re-reads/re-projects radiod's current anchor per period vs extrapolating from a fixed one.
2. Instrument the projected-anchor-UTC vs radiod's fresh (gps_time, rtp_timesnap) each period and log the divergence — that will show exactly where the 0.2 s/hr enters.
3. Note the dt-guard fault message currently says "investigate radiod sample loss / output stalls" — misleading for this case (radiod is clean); worth softening to "recorder anchor slide OR radiod output stall."

Context

First real-world dt-guard fire happened while diagnosing this (−1.41 → −1.57 → re-anchor → +0.19). New filter_drops per-cycle logging landed in wspr-recorder 8744cd6 so radiod-side output drops are now visible in the journal if contention ever returns. Full measurement scripts / numbers available on request.