Skip to content

Add behavioral readiness and Jetson observability#2

Merged
cezarc1 merged 4 commits into
mainfrom
codex/v02-observability
Jul 14, 2026
Merged

Add behavioral readiness and Jetson observability#2
cezarc1 merged 4 commits into
mainfrom
codex/v02-observability

Conversation

@cezarc1

@cezarc1 cezarc1 commented Jul 14, 2026

Copy link
Copy Markdown
Owner

What changed

  • enforce NightJet minFrames, minFps, and frame-sample freshness in controller readiness
  • add stable task/runtime/model/parent-CR labels and controller reconciliation/readiness metrics
  • add ASR latency, audio-duration, and realtime-factor histograms
  • promote the ARM64 tegrastats exporter with its existing jetson_* metric contract
  • add standalone Prometheus configuration, optional ServiceMonitor/PrometheusRule resources, and a Grafana dashboard
  • add opt-in experimental Nsight Operator pod labeling and a pinned Tegra-binary override path
  • add deterministic fake-vision/kind failure and recovery scenarios
  • record the full real-Orin pre-release evidence and restoration proof

Root cause and impact

The controller previously treated reachable NightJet metrics with sufficient frames as ready but did not evaluate the configured FPS threshold or sample age. A live workload requiring 25 FPS therefore reported CaptureReady=True while producing about 15 FPS. Readiness now fails with specific reasons for low FPS, stale samples, and unavailable metrics, then recovers without restarting the runtime pod.

Companion runtime PR

  • depends on NightJet #5
  • tested runtime digest: ghcr.io/cezarc1/nightjet:0.2.0-rc.20260713.b305045-orin@sha256:9007d79bcb6a782618bf87385d1bb0c4c37544e1951d706373c6bc169d2d6e59
  • tested controller digest: ghcr.io/cezarc1/kubejet-controller:0.2.0-rc.20260713.76e2083@sha256:57226d1624e720c700886f81e127fa7fbcba956f25fcd4d9355b0a6fe0df3a6d

Verification

  • just validate: 97 Python tests passed (5 skipped), 13 Rust tests passed; formatting, Ruff, strict mypy, clippy, Helm, kubeconform, dependency/security scans all passed
  • just verify-observability-kind: 5 passed in 272.06 seconds; kind cluster deleted cleanly
  • real Orin: default -> detail -> default swaps; increasing frames; correct immutable image/engine/model identity; one engine mount; HTTP 200 readiness; zero errors
  • live readiness red/green on the same pod UID: 19.120/25 FPS -> False/FrameRateBelowThreshold; 19+/10 FPS -> True/RuntimeMetricsReady
  • existing Prometheus: NightJet and tegrastats targets up=1; current generic pod scrape still lacks node and parent-CR labels, documented as a pre-release gap
  • post-test restoration: original CRD and CR specs matched their saved/GitOps sources exactly; original controller/runtime restored; frames increased 61 in four seconds

See docs/acceptance/2026-07-13-v0.2-orin-pre-release.md for commands, digests, condition transitions, model identities, and skipped release gates.

Scope

This is a pre-release integration PR, not a v0.2 release. Live Grafana alert firing, ASR off-node Prometheus scraping, and Nsight Operator/Tegra report capture remain explicit release gates.

@cezarc1

cezarc1 commented Jul 14, 2026

Copy link
Copy Markdown
Owner Author

CI follow-up: the first validate run caught newly-yanked transitive crate spin 0.9.8 through multer. Commit 4938c24 refreshes only that lockfile entry to non-yanked spin 0.9.9; strict cargo-deny policy remains enabled. A fresh isolated-index cargo-deny red/green check, full local just validate, and the replacement GitHub validate/e2e jobs all passed.

@cezarc1 cezarc1 marked this pull request as ready for review July 14, 2026 03:13
@cezarc1 cezarc1 merged commit 1786577 into main Jul 14, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant