KubeJet is a Kubernetes-native inference operator for NVIDIA Jetson devices.
It turns low-level Jetson inference work into agent-readable Kubernetes APIs: runtime compatibility checks, TensorRT-Edge-LLM deployment, memory budgets, OpenAI-compatible contract validation, Prometheus metrics, and GitOps-friendly rollbacks.
# Five minutes, no GPU, no Jetson required:
just verify-observability-kind # ASR + behavioral vision conformance on kindSee docs/quickstart-kind.md for the step-by-step
version, and kubectl get eis afterwards for the point of it all:
NAME READY REASON AGE
conformance-stub True RuntimeReady 2m
over-budget False SpecInvalid 1m
KubeJet ships as an OCI Helm chart. Install the controller and CRDs into any
Kubernetes cluster with Helm 4. The 0.1.1 release is tested with Helm
v4.2.2:
helm upgrade --install kubejet oci://ghcr.io/cezarc1/charts/kubejet \
--version 0.1.1 \
--namespace kubejet \
--create-namespace \
--waitHelm 4 uses server-side apply for new releases by default. Existing releases created by Helm 3 may keep their previous apply mode; KubeJet's chart is tested with Helm 4 for both fresh installs and upgrades.
Verify the controller is running:
kubectl get deploy -n kubejet
kubectl get crd edgeinferenceservices.kubejet.dev edgevisionpipelines.kubejet.devTry the GPU-free conformance service:
kubectl apply -f \
https://raw.githubusercontent.com/cezarc1/KubeJet/main/examples/kind-quickstart/asr-stub.yaml
kubectl wait --for=condition=ServingReady=True \
edgeinferenceservice/conformance-stub -n kubejet --timeout=300sOn a real Jetson cluster, install the same controller first, then apply a device-specific manifest such as examples/qwen3-asr-orin-nano.yaml or examples/nightjet-tensorrt.yaml after the referenced model artifact or TensorRT engine exists on the target node.
Manual TensorRT deployment is powerful, but it is hard for humans and coding agents to operate safely:
- artifacts depend on JetPack, L4T, CUDA, TensorRT, architecture, and batch shape;
- a pod can be healthy while model outputs are wrong;
- Jetson memory pressure can wedge the node before Kubernetes reports a useful error;
- runtime flags, engine build flags, and model compatibility are spread across shell scripts and logs.
KubeJet makes those constraints declarative. A service is Ready only after
its runtime rolls out and an OpenAI-compatible contract smoke test passes
against it — because a healthy pod serving corrupted output is the failure
Kubernetes cannot see (we learned this from maxBatch>1 engines that built,
ran, and produced WER 1.0 garbage; see docs/v0.1-design.md).
No Kubernetes-native control plane for Jetson inference exists today:
- The NVIDIA GPU Operator does not support Jetson/Tegra (still requested in 2025+); the baseline is hand-configuring nvidia-container-toolkit, CDI, and the standalone device plugin against k3s.
- The NIM Operator targets datacenter GPUs (Hopper/Blackwell/Ada) with an enterprise license, not an 8 GB Orin Nano.
- Triton on Jetson is a reduced-feature special build, and deploying it on k3s is tutorial-grade DIY: bare Deployments, no validation, no Jetson-aware status.
- KServe assumes cluster-scale infrastructure and has no Jetson awareness.
Where KubeJet sits:
| Layer | Job | Examples |
|---|---|---|
| Inference engine | Execute a compiled graph on one device | TensorRT, TensorRT-Edge-LLM, llama.cpp |
| Inference server | Wrap engines in a network service | Triton (Dynamo-Triton), vLLM |
| Serving control plane | Kubernetes lifecycle: CRDs, validation, readiness, status | KServe, NIM Operator, KubeJet (Jetson) |
Given a k3s Jetson node and a TensorRT-Edge-LLM ASR artifact, KubeJet should:
- reject incompatible model/runtime/batch/memory configs;
- deploy a TensorRT-Edge-LLM-backed service on the Jetson;
- expose an OpenAI-compatible transcription endpoint;
- run an OpenAI-compatible smoke request through a Kubernetes Job;
- publish Prometheus request metrics from the runtime wrapper;
- leave enough Kubernetes status for agents to debug failures with
kubectl.
The kind conformance suite (tests/e2e/) executes this promise on every PR
with the stub runtime; the Jetson path executes it with the real engine.
KubeJet v0.1 uses a split control-plane/data-plane design:
- controller: Python + Kopf for fast Kubernetes reconciliation, validation, and status-condition iteration;
- runtime wrapper: Rust + Tokio for the hot request path on the edge device;
- backend bridge: a runtime-image script that preprocesses uploaded audio and
invokes TensorRT-Edge-LLM
llm_inference; - transport: HTTP first, with HTTP/2-capable plumbing and gRPC reserved for a later stable transport;
- artifact source: the existing proven PVC-backed TensorRT-Edge-LLM bundle before OCI model artifacts are introduced.
The first target is intentionally narrow:
- device: Jetson Orin Nano 8 GB running k3s;
- task: ASR/transcription;
- runtime: TensorRT-Edge-LLM;
- model path: Qwen3-ASR-0.6B INT8;
- serving shape: persistent warm runtime with
batch_size: 1; - API:
/v1/audio/transcriptions; - observability: tegrastats plus per-request inference metrics.
The first vision extension is a narrow EdgeVisionPipeline path for the
NightJet passive night-vision demo. DeepStream production vision, Riva speech,
vLLM/SGLang adapters, and Thor support remain future adapters after the narrow
paths are proven.
The v0.2 observability work makes that vision readiness behavioral: configured frame count, rolling FPS, and last-frame freshness all affect CR status. It also ships native ASR/NightJet metrics, an ARM64 tegrastats exporter, optional Prometheus Operator resources, a Grafana dashboard, and an explicitly experimental Nsight Operator integration. See observability and verification.
KubeJet's first vision path is wired around the public NightJet model stack:
| Project | Role |
|---|---|
| nightjet | Low-light model, weights, ONNX export, and public Orin runtime artifacts |
| kubejet (here) | Kubernetes operator that reconciles the Jetson runtime and readiness checks |
The EdgeVisionPipeline example deploys the public NightJet runtime image. It
requires a Jetson Orin, a local TensorRT engine built for that runtime, and an
Arducam-compatible /dev/video0 camera path.
api/ API notes; canonical CRDs live in the chart
charts/kubejet/ Helm chart and CRDs
controllers/ Kubernetes controller implementation
docs/ design notes, quickstart, agent workflows
examples/ example EdgeInferenceService / EdgeVisionPipeline manifests
runtimes/trt-edgellm/ TensorRT-Edge-LLM adapter (Rust wrapper + bridge script)
runtimes/fake/ stub runtime image for GPU-free conformance
observability/ tegrastats exporter, Prometheus, Grafana, Nsight assets
hack/ release and e2e scripts
tests/ unit tests and kind conformance suite