RTP unwrap slips whole 2^32 wraps after radiod discontinuities — spots written up to +14 days in the future

[rrobinett]

Summary

Across a week on B4-100, psk-recorder (via ka9q-python's RTP unwrap / SlotClock path) wrote 1,191,598 FT8 rows with spot times up to +14 days in the future into the shared sink. The offsets cluster near integer multiples of the 32-bit RTP wrap period at 12 kHz (2^32 / 12000 ≈ 99.4 h ≈ 4.14 days) — i.e. unwrap slips of +1…+3 wraps, not clock drift.

Timeline / forensics (B4-100)

• 2026-07-09 01:15Z — writing starts. This is the era of repeated radiod RX888 sample-loss glitches on this host (fork radiod's RTP↔GPS step monitor logged bursts on Jul 9/10 and a +2.64 s loss on Jul 15/16).
• 2026-07-09 → 2026-07-16: ~1.19M FT8 rows written with $.time between +1 and +14 days ahead (max seen: 2026-07-30T02:29:15), avg dt ≈ -1.9, spot volume otherwise normal — the streams kept decoding, so nothing looked wrong from throughput.
• 2026-07-16 02:04Z — writing stops instantly at a psk-recorder restart (fresh SlotClock anchors). Strong evidence the wrong-time state lived in the long-running process's unwrap/anchor state, seeded by the radiod discontinuities.
• Distribution of the future offsets is quantized ≈ N × 4.14 days.

Impact

• The spots were direct-posted to PSKReporter with future timestamps for a week.
• They sat in pending_uploads where any consumer selecting by spot time (e.g. the wsprdaemon per-cycle tar source) would have shipped them as "fresh" once their fake dates arrived. (B4's rows are backed up and quarantined: B4:/home/rob/psk-future-rows-backup-20260716.tsv.gz.)
• Downstream health tooling that windows by spot time was silently poisoned (sigmond 723bda2 switched the timing watchdog to queued_at + id-window for this reason).

Suspected mechanism

A large forward/backward RTP discontinuity (radiod sample loss / output stall / restart artifacts) reaching the 32-bit unwrap logic gets interpreted as a wrap, permanently advancing the unwrapped counter by ~2^32 samples. Every subsequent RTP→UTC conversion is then +4.14 days (per slip). Nothing re-validates the unwrapped time against wallclock, so the state persists for the life of the process and can compound (+1, +2, +3 wraps observed).

Suggested directions

1. Sanity-gate the unwrap: an unwrap step that implies a jump of more than (say) half a wrap period against the wallclock/status anchor should be treated as a discontinuity → re-anchor, not a wrap.
2. SlotClock/ChannelInfo cross-check: periodically compare SlotClock-derived UTC against the status-stream GPS anchor; divergence beyond a threshold ⇒ LOUD fault + re-anchor (detect + alarm + recover, never silent — same principle as wspr-recorder's new cycle-dt guard, 0e50f8c).
3. At minimum, refuse to emit spots whose computed time is > a few minutes in the future — that's always a bug, never physics.

Related context

• ka9q-python removed the per-update anchor_epoch step detection on 2026-06-28 (status-jitter storms), so consumers currently have no discontinuity signal from the library at all.
• Single-instance hosts also can't lean on the sigmond timing-watchdog's peer-comparison detector; its absolute gate (TWD_DT_ABS_SEC=2.5) sat just above the −2.0 s offset this bug family produces (observed again on 2026-07-16, without future-dating).

Forensic queries, radiod step logs, and the quarantined row backup are available on B4 on request.