feat(terminal): Kubernetes execution backend (run commands in isolated session pods)

[Tombar]

Summary

Adds a kubernetes execution backend to the terminal / code-execution tools — a peer of the existing ssh / docker / modal / daytona backends. With TERMINAL_ENV=kubernetes, each agent shell command runs by exec-ing into a dedicated session pod in the same cluster, instead of in the agent's own process/container.

Rebased onto current upstream main and squashed to a single commit. Note: upstream removed the standalone vercel python extra since this work began, so this branch adds only the kubernetes extra/lazy-dep and does not reintroduce vercel.

Motivation

When Hermes runs inside Kubernetes, the local backend executes commands in the agent's own container — sharing its filesystem, secrets, and ServiceAccount token. The Kubernetes backend isolates execution into a separate pod, so untrusted or heavy commands can't touch the agent's home, memory, or credentials. It's the K8s-native counterpart to the Docker/SSH sandboxes.

Context

We built this backend to pair with our companion Kubernetes controller, hermes-operator, which manages Hermes deployments as custom resources and can reconcile the RBAC + admission policy automatically. But it's intentionally operator-agnostic — namespace, ServiceAccount, image, persistence, deadline, and image-pull-secrets are all config/env-driven, and the k8s/ samples run it standalone with no operator involved. The operator is an optional convenience, not a dependency.

How it works

New tools/environments/kubernetes.py:

• KubernetesEnvironment(BaseEnvironment) — runs each command via kubernetes.stream exec into the session pod, wrapped in the existing _ThreadedProcessHandle (same pattern as the Daytona/Modal backends). Inherits execute(), the session snapshot, CWD tracking, and interrupt handling unchanged.
• WorkspaceProvisioner interface + DirectProvisioner — create the session Pod (and a PVC when persistent) via the in-cluster API (load_incluster_config(), with a kubeconfig fallback for out-of-cluster use).
• Locked-down pod shape: runAsNonRoot, non-root runAsUser/fsGroup, drop ALL caps, allowPrivilegeEscalation: false, no host namespaces, automountServiceAccountToken: false.
• Clean isolated workspace: the session pod starts empty at /workspace; no host file sync — write_file/patch route through the exec channel.

Lifecycle

• Ephemeral (default) — Pod with an emptyDir, deleted on cleanup, with an activeDeadlineSeconds ceiling as a leak backstop.
• Persistent (opt-in, TERMINAL_KUBERNETES_PERSISTENT=true) — Pod mounts a retained PVC hermes-ws-<task_id> at /workspace; on cleanup the pod is deleted but the PVC is kept, so the next session resumes the filesystem (mirrors Daytona's persistent_filesystem contract).
• Crash safety — the session pod carries an ownerReference to the agent's own pod (via downward-API HERMES_POD_NAME / HERMES_POD_UID), so Kubernetes garbage-collects it if the agent dies.

Configuration

TERMINAL_ENV=kubernetes
TERMINAL_KUBERNETES_NAMESPACE                 # default: in-cluster SA namespace
TERMINAL_KUBERNETES_IMAGE                     # session-pod image (default: ubuntu:22.04)
TERMINAL_KUBERNETES_POD_SA                    # no-perms SA for session pods
TERMINAL_KUBERNETES_PERSISTENT               # default: false (ephemeral)
TERMINAL_KUBERNETES_ACTIVE_DEADLINE_SECONDS  # default: 14400
TERMINAL_KUBERNETES_PULL_SECRETS

The kubernetes client ships as a named optional extra plus a LAZY_DEPS entry (kept out of [all], per the existing terminal-backend policy), and is baked into the image via the Dockerfile.

Testing

• Unit (mocked client): manifest shape / security context, ensure/destroy, exec loop, factory wiring, requirements gate — 23 tests (23 passed on the rebased branch).
• Integration tests/integration/test_kubernetes_terminal.py (kind, @pytest.mark.integration, env-gated): real exec roundtrip + PVC persistence across pod recreation.
• CI .github/workflows/test-k8s.yml: spins up kind and runs the integration tests.
• Validated end-to-end on a real cluster: real model → terminal tool → session pod → output, including ownerReference garbage-collection.

Deployment manifests

Cluster operators need two things for this backend to be safe: a scoped RBAC grant for the agent SA, and an admission policy that constrains the shape of session pods (so a granted pods/create cannot mean "create a privileged pod"). Both ship in k8s/:

• rbac.yaml — namespace-scoped Role (pods, pods/exec, pods/log, persistentvolumeclaims) + RoleBinding to the agent SA, plus a no-perms session ServiceAccount (automountServiceAccountToken: false).
• validatingadmissionpolicy.yaml — CEL policy (Kubernetes ≥ 1.30) that denies session pods running as the default SA, using host namespaces, with allowPrivilegeEscalation true, without drop: ALL, or mounting hostPath. Scoped via matchConditions to pod-creates by a ServiceAccount carrying the backend's app.kubernetes.io/managed-by: hermes-agent label.
• README.md — apply/verify steps, hardening notes, and a pointer to an operator-managed alternative.

Repo docs

• .env.example — new KUBERNETES BACKEND section with the full env-var surface.
• cli-config.yaml.example — new OPTION 7: Kubernetes session pod.
• README.md — terminal-backend count bumped to "Seven" with a link to k8s/.

🤖 Generated with Claude Code

[github-actions]

🔎 Lint report: feat/k8s-exec-backend vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 9667 on HEAD, 9637 on base (🆕 +30)

🆕 New issues (27):

Rule	Count
unresolved-attribute	15
unresolved-import	9
invalid-assignment	2
invalid-argument-type	1

First entries

tools/environments/kubernetes.py:202: [invalid-assignment] invalid-assignment: Invalid subscript assignment with key of type `Literal["activeDeadlineSeconds"]` and value of type `int` on object of type `dict[str, str | bool | dict[str, int] | ... omitted 3 union elements]`
tools/environments/kubernetes.py:313: [unresolved-import] unresolved-import: Cannot resolve imported module `kubernetes.stream`
tests/integration/test_kubernetes_terminal.py:12: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
tests/tools/test_kubernetes_environment.py:19: [unresolved-attribute] unresolved-attribute: Unresolved attribute `config` on type `ModuleType`
tools/environments/kubernetes.py:325: [unresolved-attribute] unresolved-attribute: Attribute `namespace` is not defined on `None` in union `PodRef | None`
tools/environments/kubernetes.py:270: [unresolved-attribute] unresolved-attribute: Attribute `delete_namespaced_pod` is not defined on `None` in union `Unknown | None`
tests/tools/test_kubernetes_environment.py:18: [unresolved-attribute] unresolved-attribute: Unresolved attribute `client` on type `ModuleType`
tools/environments/kubernetes.py:224: [unresolved-attribute] unresolved-attribute: Attribute `create_namespaced_persistent_volume_claim` is not defined on `None` in union `Unknown | None`
tests/tools/test_kubernetes_environment.py:33: [unresolved-attribute] unresolved-attribute: Unresolved attribute `stream` on type `ModuleType`
tools/terminal_tool.py:1328: [unresolved-import] unresolved-import: Cannot resolve imported module `kubernetes`
tools/environments/kubernetes.py:230: [unresolved-attribute] unresolved-attribute: Attribute `create_namespaced_pod` is not defined on `None` in union `Unknown | None`
tests/tools/test_kubernetes_environment.py:49: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
tests/tools/test_kubernetes_environment.py:32: [unresolved-attribute] unresolved-attribute: Unresolved attribute `load_incluster_config` on type `ModuleType`
tools/environments/kubernetes.py:324: [unresolved-attribute] unresolved-attribute: Attribute `pod_name` is not defined on `None` in union `PodRef | None`
tests/tools/test_kubernetes_environment.py:180: [unresolved-import] unresolved-import: Cannot resolve imported module `kubernetes.client.exceptions`
tools/environments/kubernetes.py:248: [unresolved-attribute] unresolved-attribute: Attribute `read_namespaced_pod` is not defined on `None` in union `Unknown | None`
tests/tools/test_kubernetes_environment.py:30: [unresolved-attribute] unresolved-attribute: Unresolved attribute `exceptions` on type `ModuleType`
tools/terminal_tool.py:1329: [unresolved-import] unresolved-import: Cannot resolve imported module `kubernetes.client`
tests/integration/test_kubernetes_terminal.py:42: [unresolved-import] unresolved-import: Cannot resolve imported module `kubernetes`
tools/environments/kubernetes.py:326: [unresolved-attribute] unresolved-attribute: Attribute `container` is not defined on `None` in union `PodRef | None`
tests/tools/test_kubernetes_environment.py:31: [unresolved-attribute] unresolved-attribute: Unresolved attribute `CoreV1Api` on type `ModuleType`
tools/environments/kubernetes.py:314: [unresolved-import] unresolved-import: Cannot resolve imported module `kubernetes.client`
tools/environments/kubernetes.py:150: [invalid-assignment] invalid-assignment: Invalid subscript assignment with key of type `Literal["ownerReferences"]` and value of type `list[dict[Unknown, Unknown]]` on object of type `dict[str, str | dict[str, str]]`
tools/environments/kubernetes.py:356: [invalid-argument-type] invalid-argument-type: Argument to bound method `WorkspaceProvisioner.destroy` is incorrect: Expected `PodRef`, found `PodRef | None`
tools/environments/kubernetes.py:218: [unresolved-attribute] unresolved-attribute: Attribute `read_namespaced_persistent_volume_claim` is not defined on `None` in union `Unknown | None`
... and 2 more

✅ Fixed issues: none

Unchanged: 4991 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.