feat(runtime): add SGLang runtime backend (#974)

[joryirving]

What

Adds sglang as a first-class InferenceService.Spec.Runtime value,
alongside llamacpp / vllm / tgi / personaplex / generic.

A new typed SGLangConfig (with nested SGLangSpeculativeConfig) covers
sharding, memory, batching, quantization/KV-cache, attention, agentic glue
(tool/reasoning parsers, chat template), speculative decoding (EAGLE /
EAGLE3), and LoRA basics — mirroring the vLLM pattern. GPU vendor (NVIDIA
vs AMD) drives image selection, so the AMD ROCm image is picked automatically
when model.spec.hardware.gpu.vendor is amd.

SGLang launches via python3 -m sglang.launch_server (mirroring how
personaplex wraps a Python module). Probes target /health_generate for
startup + readiness (runs a token, accurate) and /health for liveness
(cheap). Startup tolerates 180 failures (~30 min) for cold-start model load.

Why

Fixes #974

Headline motivation is SGLang's RadixAttention automatic prefix caching.
A foreman agent's tool loop re-sends the same large shared prefix every turn
(system prompt + tool definitions + issue/repo context); SGLang serves that
shared prefix from cache instead of reprocessing it, which materially cuts
latency and raises throughput for the multi-turn, high-concurrency pattern
the foreman-agent generates. Today an operator who wants SGLang has to fall
back to the generic runtime and hand-write container config — losing the
auto-generated args, mode handling, and health wiring the first-class
backends get.

How

• api/v1alpha1/inferenceservice_types.go: SGLangConfig +
  SGLangSpeculativeConfig structs, SGLangConfig field on
  InferenceServiceSpec, sglang added to the runtime enum.
• internal/controller/runtime_sglang.go: SGLangBackend implementing
  RuntimeBackend + CommandBuilder + HPAMetricProvider +
  EnvBuilder. BuildArgs emits base flags + --served-model-name +
  typed fields in declaration order + auto-derived --tp + --is-embedding
  for mode + extraArgs last.
• internal/controller/runtime_sglang_args.go: free-function helpers,
  one per flag. Mirrors runtime_vllm_args.go structure.
• internal/controller/runtime.go: case RuntimeSGLANG in
  resolveBackend.
• internal/controller/deployment_builder.go: resolveRuntimeImage
  picks ROCm image for AMD GPUs; otherwise CUDA.
• make manifests + make chart-crds regenerated CRDs.
• docs/contributors/adding-a-runtime.md, docs/site/concepts/comparison.md,
  docs/site/guides/model-matrix.md updated.

Out of scope (B3 followup): LoRA hot-loading, speculative perf tuning
(--speculative-accept-threshold-*), data-parallel rendezvous,
PodMonitor for /metrics, and remaining minor SGLang flags. Tracked as a
separate issue.

Checklist

• Tests added/updated (TestSGLangBuildArgs table-driven +
  TestSGLangBuildCommand + TestSGLangProbes + TestSGLangBuildEnv +
  TestValidateSGLangConfig + dispatch + image-resolution tests)
• make test passes locally
• make lint passes locally
• make lint-all passes locally (cross-arch)
• Commit messages follow conventional commits
• All commits are signed off (git commit -s)
• AI assistance disclosed: this PR was produced by an AI coding agent
  working with a repo collaborator. The agent followed the
  docs/contributors/adding-a-runtime.md pattern, the design spec at
  docs/superpowers/specs/2026-07-10-sglang-runtime-backend-design.md
  (local agent scratch, not committed), and AGENTS.md conventions.
• Documentation updated

[codecov]

Codecov Report

❌ Patch coverage is 70.71651% with 94 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
api/v1alpha1/zz_generated.deepcopy.go	0.00%	91 Missing and 1 partial ⚠️
internal/controller/runtime_sglang.go	98.07%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[Defilan]

This is excellent work, Jory, one of the cleanest runtime additions we have had. It mirrors the vLLM pattern faithfully, the arg emission is deterministic (nice TestSGLangBuildArgsDeterministic), the probe split (/health for liveness vs /health_generate for readiness) is a step up from vLLM, and both CRD copies regenerate with zero drift, which fully addresses the earlier #1042 feedback. I ran make manifests && make chart-crds, the SGLang/image/dispatch tests, and cross-arch lint locally: all green.

Not blocking from my side, approving in spirit. One thing worth resolving so the runtime is truly first-class: SGLang does not expose /metrics unless it is launched with --enable-metrics (per the SGLang production-metrics docs). Right now nothing injects that flag, so the runtime ships with no metrics endpoint at all, yet DefaultHPAMetric() (runtime_sglang.go:62) advertises sglang:num_requests_running and the docs table lists it, so HPA (and the future #1060 PodMonitor) silently get nothing. vLLM exposes metrics with no flag and llama.cpp always passes --metrics, so SGLang is the odd one out. I would add --enable-metrics to the base args (around runtime_sglang.go:83), mirroring llama.cpp's always-on convention:

"--host", "::",
"--port", fmt.Sprintf("%d", port),
// Expose SGLang's Prometheus /metrics endpoint (off by default in SGLang).
// Mirrors llama.cpp's always-on --metrics so PodMonitor scraping and the
// advertised HPA metric actually work.
"--enable-metrics",

If you would rather keep metrics fully deferred to #1060, that is fine too, just have DefaultHPAMetric() and the docs table reflect that it is not functional yet.

Two tiny nits, both optional:

• runtime_sglang.go:133 uses the "embedding" string literal; the package already defines servingModeEmbedding, so prefer the constant to keep a future rename from drifting. (The rest of the codebase also resolves mode via resolveServingMode(isvc), which infers embedding from the endpoint path, worth considering for parity.)
• The --is-embedding / --mem-fraction-static GPU check keys off the device-plugin count, so a DRA-only model (count 0) would skip --mem-fraction-static with a spurious "no GPU" log. This mirrors vLLM's existing behavior, so it is a shared limitation rather than a regression, just noting for a future cleanup.

Everything else is ready to merge. Thanks for the great contribution and the thorough disclosure.