Scale telemetry: SoC temp, weight-stall watchdog, reset reason

[skialpine]

Summary

Diagnostics for field reports of "weight stops being collected under sustained multi-protocol load" — the only recovery seen was a long battery-out cooldown (a quick power-cycle didn't help), pointing at a thermal/analog failure rather than firmware state. Adds telemetry to confirm/rule it out, with no behavior change to the weight/WiFi/BLE paths.

New fields in the /snapshot WS status frame (and serial logs):

• soc_temp_c / soc_temp_max_c — live + peak ESP32-S3 die temp (temperatureRead()), sampled every 2 s.
• weight_stalled / stall_count / last_stall_ms / last_stall_temp_c — a watchdog in pureScale() that flags when the ADS1232 raw value is frozen/railed >8 s (a live cell dithers every sample), recording the die temp at failure to correlate stalls with heat. Skipped during the deliberate ADC power-cycle recovery; throttled to 250 ms.
• reset_reason — esp_reset_reason() at boot, so a brownout/panic/WDT reset is attributable.

Plus tools/thermal_load_test.sh: a 1-hour USB+WiFi+churn+mDNS soak that polls the telemetry (BT driven externally).

Threading: new cross-task scalars are volatile per CLAUDE.md; the status frame only reads them.

Test plan

• Builds for esp32s3; flashed; status frame reports the new fields.
• 1-hour multi-protocol soak to capture peak temp + any stall and the die temp at which it occurs.

🤖 Generated with Claude Code

[skialpine]

Code review

Reviewed the telemetry diff (deep bug scan + threading/CLAUDE.md check). Found one real issue, now fixed in this PR (commit 8e345d9):

• Stall watchdog false-trip during ADC recovery — the watchdog read getDebugInfo().rawValue, which is frozen by definition during the firmware's own powerDown()/powerUp() recovery, so a genuine signal-timeout recovery would be miscounted as a railed/frozen stall (corrupting the very metric this adds). Fixed: skip the check while b_adc_recovery_active and re-seed the window on resume.
• Also (cost): it ran getDebugInfo() (which does a sqrt + dataset passes) every loop iteration though only rawValue is used — wasteful on a chip we're characterizing for heat. Now throttled to 250 ms (the ADC only samples ~10/s).

Verified clean: printf format/arg pairing in both status frames; cross-task reads are volatile (benign torn read only); temperatureRead()/resetReasonStr() safe; at-rest false-positive risk is low (24-bit raw at SAMPLES=1 dithers every sample, so 8 s of byte-identical raw is a genuine freeze).

🤖 Generated with Claude Code

[skialpine]

Force-pushed: rebased onto current main (post #58 + #60) and restructured telemetry delivery.

What changed in the protocol: the 8 telemetry fields no longer ride on the hot status broadcast. They move out into three new frame shapes:

• session_info — sent once per client on WS connect. Carries protocol_version, firmware_version, reset_reason (the connection-immutable fields).
• debug events — broadcast on actual diagnostic changes: stall_start / stall_end, adc_recovery (count incremented), temp_peak (new SoC max). Subscribers get the event the moment it happens; non-subscribers ignore the unknown type.
• debug reply — per-client on-request snapshot via {\"command\":\"debug\"}.

Status frame stays at its current 16 fields (~310 B payload) instead of bloating to ~470 B. Apps that don't care about diagnostics get back ~21% per broadcast.

Why now: the 8-hour soak validating #61's heap-defense knobs (cap=4 + 32K floor) showed that under marginal RSSI (-83) and multi-client load, status broadcasts were the dominant heap-pressure source — gate fired 5,000+ times. Slimming the status payload at the source is structurally better than just gating harder.

Backward compat: the previous status frame shape (which had the 8 telemetry fields appended) had not shipped on any merged release, so there's nothing to deprecate. Clients written against the README's old example will fall back to ignoring unknown fields cleanly anyway.

Build clean, validated diff against current main. Squashed to one commit since the original 7-commit history was reshuffled enough to make rebase-without-conflict impossible — happy to break it back into reviewable chunks if that's preferred.

[skialpine]

Code review

Found 3 issues:

1. tools/thermal_load_test.sh reads telemetry fields exclusively from status frames, but commit f72d6e2 moved them all to session_info and debug frames. The monitor never subscribes to debug events nor reads session_info, so peak stays at its -999 sentinel and the verdict path forces RESULT=FAIL on every run regardless of device behavior. Fix: handle all three frame types in the Python monitor (the script also unconditionally sends events on but never debug subscribe). The script's header comment at line 5 still says "WS status-frame telemetry" — also stale.

openscale/tools/thermal_load_test.sh

Lines 67 to 96 in f72d6e2

	try:
	d=json.loads(w.recv())
	if d.get("type")=="status": st=d
	except Exception: pass
	return st
	ws=connect()
	peak=-999.0; total_stalls=0; reboots=0; max_recov=0
	prev_stalls=None; prev_max=None; last_reset="?"
	no_status_streak=0; max_no_status_streak=0; total_no_status=0
	first=True
	while time.time()<end:
	st = first_status(ws, 9 if first else 2.5); first=False
	if st:
	soc=st.get('soc_temp_c'); mx=st.get('soc_temp_max_c'); sc=st.get('stall_count',0) or 0
	recov=st.get('adc_recovery_count',0) or 0; rr=st.get('reset_reason','?')
	last_reset=rr; no_status_streak=0
	if isinstance(mx,(int,float)) and mx>peak: peak=mx
	if recov>max_recov: max_recov=recov
	# reboot heuristic: since-boot counters or peak dropped vs last frame
	if prev_stalls is not None and (sc<prev_stalls or (isinstance(mx,(int,float)) and isinstance(prev_max,(int,float)) and mx<prev_max-3)):
	reboots+=1
	print("[%s] *** REBOOT detected (counters reset; reset_reason=%s) ***"%(time.strftime('%H:%M:%S'),rr),flush=True)
	total_stalls=max(total_stalls, sc)
	prev_stalls=sc; prev_max=mx
	flag=" *** STALL ***" if st.get('weight_stalled') else ""
	print("[%s] soc=%5sC max=%5sC stalled=%-5s stalls=%s recov=%s last_stall_ms=%s stall_temp=%s reset=%s grams=%s chg=%s%s"%(
	time.strftime('%H:%M:%S'), soc, mx, st.get('weight_stalled'), sc, recov,
	st.get('last_stall_ms'), st.get('last_stall_temp_c'), rr, st.get('grams'),
	st.get('charging'), flag), flush=True)
	else:

2. Debug event broadcasts (sendWebsocketDebugStall, sendWebsocketDebugAdcRecovery, sendWebsocketDebugTempPeak) gate only on !b_wifiEnabled || websocket.count() == 0. They skip the b_websocketEventsEnabled check that every other event broadcast in this file uses (sendWebsocketButton L188, sendWebsocketPowerOff L199, sendWebsocketStatusAll L246). README says "A client must send events on before periodic status, local scale button presses, or power-off notifications are emitted." — new debug frames break this opt-in contract. Either add !b_websocketEventsEnabled to the early-return, or document the always-on choice explicitly in the README's debug-frame section.

openscale/include/websocket.h

Lines 317 to 350 in f72d6e2

	// keep their own running snapshot from session_info + the on-request debug reply,
	// and update it from these deltas. Keeps each event small (~80-140 B).
	void sendWebsocketDebugStall(bool started) {
	if (!b_wifiEnabled || websocket.count() == 0) return;
	if (!wsBroadcastHeapOk()) return;
	if (started) {
	websocket.printfAll("{\"type\":\"debug\",\"event\":\"stall_start\",\"stall_count\":%lu,\"last_stall_ms\":%lu,\"last_stall_temp_c\":%.1f,\"ms\":%lu}",
	(unsigned long)g_stallCount,
	g_lastStallMs,
	g_lastStallTempC,
	millis());
	} else {
	websocket.printfAll("{\"type\":\"debug\",\"event\":\"stall_end\",\"ms\":%lu}", millis());
	}
	}
	void sendWebsocketDebugAdcRecovery() {
	if (!b_wifiEnabled || websocket.count() == 0) return;
	if (!wsBroadcastHeapOk()) return;
	websocket.printfAll("{\"type\":\"debug\",\"event\":\"adc_recovery\",\"adc_recovery_count\":%lu,\"ms\":%lu}",
	(unsigned long)i_adc_recovery_count,
	millis());
	}
	// temp_peak is broadcast when g_socTempMaxC ticks up. Rate-limited at the call
	// site (main loop) -- not here -- since the temp sampler already throttles.
	void sendWebsocketDebugTempPeak(float maxC) {
	if (!b_wifiEnabled || websocket.count() == 0) return;
	if (!wsBroadcastHeapOk()) return;
	websocket.printfAll("{\"type\":\"debug\",\"event\":\"temp_peak\",\"soc_temp_max_c\":%.1f,\"ms\":%lu}",
	maxC,
	millis());
	}

3. i_adc_recovery_count documentation contradicts the implementation. include/parameter.h:197-200 says "counts truthfully ... over a long soak instead of saturating at 255" and README:256 says "number of ADC power-cycle recoveries since boot". But src/hds.ino:1011 calls resetAdcRecoveryState() after every successful scale.update(), which zeroes the counter. The actual semantics — confirmed by the display-logic use at src/hds.ino:1830 (>= ADC_ERROR_RECOVERY_COUNT ? "ADC ERROR" : "ADC RECOVER") — is "consecutive failed recoveries in the current episode," not a lifetime total. A client tracking adc_recovery_count expecting monotonic growth will see it ping-pong to 1 and back to 0. Either fix the docs (parameter.h block-comment + README field description) or rename for a true lifetime counter.

openscale/include/parameter.h

Lines 195 to 201 in f72d6e2

	static bool b_adc_recovery_active = false;
	// volatile: written on the main loop -- incremented on each ADC power-cycle
	// recovery, reset to 0 by resetAdcRecoveryState() -- and read in the WS status
	// frame (which can be built on the AsyncTCP task). uint32_t (not uint8_t) so a
	// *perpetual* recovery loop -- the one failure mode the stall watchdog is blind
	// to -- keeps counting truthfully over a long soak instead of saturating at 255.
	static volatile uint32_t i_adc_recovery_count = 0;

---

1 below-threshold issue (50-74)

• [Worth a look] src/hds.ino:22 comment says adsDebugCallback is "enabled during soak tests" — the actual enablement is via the USB binary protocol in include/usbcomm.h, not anything soak-test-specific. One-line clarification (score: 50).

🤖 Generated with Claude Code

_{If this code review was useful, please react with 👍. Otherwise, react with 👎.}