From 84e701f01d3d4dee2fab1cdd7185c4b4744625c6 Mon Sep 17 00:00:00 2001 From: Tony Meyer Date: Fri, 22 May 2026 19:35:30 +1200 Subject: [PATCH 1/9] docs: add how-to guide for debugging Kubernetes charms --- docs/.custom_wordlist.txt | 1 + docs/howto/debug-a-kubernetes-charm.md | 183 +++++++++++++++++++++++++ docs/howto/debug-your-charm.md | 4 + docs/howto/index.md | 1 + 4 files changed, 189 insertions(+) create mode 100644 docs/howto/debug-a-kubernetes-charm.md diff --git a/docs/.custom_wordlist.txt b/docs/.custom_wordlist.txt index 7c8f1c564..d71e99691 100644 --- a/docs/.custom_wordlist.txt +++ b/docs/.custom_wordlist.txt @@ -29,6 +29,7 @@ replan repo requirers reusability +sidecar snappass stderr stdout diff --git a/docs/howto/debug-a-kubernetes-charm.md b/docs/howto/debug-a-kubernetes-charm.md new file mode 100644 index 000000000..55c44c42e --- /dev/null +++ b/docs/howto/debug-a-kubernetes-charm.md @@ -0,0 +1,183 @@ +--- +myst: + html_meta: + description: Debug Kubernetes charms and their Pebble-managed workloads -- inspect the charm and workload containers, read Pebble services, logs, plans, changes, checks, and notices, and diagnose common sidecar failure modes. +--- + +(debug-a-kubernetes-charm)= +# How to debug a Kubernetes charm + +> See first: {ref}`debug-your-charm`, {ref}`workload-containers`, {external+juju:ref}`Juju | How to manage logs ` + +A Kubernetes charm runs as a sidecar: each unit is a pod with a *charm container* (running your charm code) alongside zero or more *workload containers*. Juju injects {external+pebble:doc}`Pebble ` into every workload container as its service manager, and your charm talks to each workload's Pebble over an HTTP-on-Unix-socket API. + +This split is what makes debugging a Kubernetes charm different from debugging a machine charm: a problem can live in the charm code, in the Pebble configuration, in the workload process itself, or at the Kubernetes layer below all of them. This guide covers the Kubernetes- and Pebble-specific tools for narrowing that down. For the substrate-agnostic tools (`juju debug-log`, `juju debug-hooks`, `juju debug-code`, jhack, and remote debugging with VS Code), see {ref}`debug-your-charm`. + +(k8s-two-container-model)= +## Know which container you're looking at + +Each piece of the system lives in a specific place, and reaching for the wrong one is the most common way to waste time: + +| What | Where it runs | How to reach it | +| --- | --- | --- | +| Your charm code (`src/charm.py`, hooks, logs) | charm container (named `charm`) | `juju ssh `, `juju debug-log` | +| Pebble and the workload process | workload container (named after the `containers` entry in `charmcraft.yaml`) | `juju ssh --container `, the Pebble CLI | +| The pod, image pulls, scheduling | Kubernetes | `kubectl` | + +The charm and workload containers each have their own filesystem and process space. The charm reaches a workload's Pebble through a Unix socket that Juju mounts into both containers: + +- In the **workload container**, the socket is at `/var/lib/pebble/default/pebble.sock`. +- In the **charm container**, the same socket is mounted at `/charm//pebble.sock`, and the Pebble CLI binary is at `/charm/bin/pebble`. + +```{tip} +If [`Container.can_connect()`](ops.Container.can_connect) returns `False` or your charm raises [`ops.pebble.ConnectionError`](ops.pebble.ConnectionError), the charm container cannot reach the workload's Pebble over that socket. This usually means the workload container hasn't started yet (no [`PebbleReadyEvent`](ops.PebbleReadyEvent) has fired) -- look at the pod first (see [](#k8s-inspect-the-pod)), not at your charm code. +``` + +(k8s-pebble-cli)= +## Inspect the workload with the Pebble CLI + +SSH into the workload container and use the Pebble CLI to see what Pebble thinks is going on. The {ref}`debug-your-charm` guide covers `pebble services`, `pebble logs`, `pebble exec`, `pebble plan`, and `pebble checks`. The commands below go deeper and are the ones you'll reach for when a workload won't start or keeps crashing. + +```shell +juju ssh --container myapp myapp/0 +``` + +All of the examples below assume you're running them inside the workload container, where `pebble` is on the `PATH`. (To run them from the charm container instead, see [](#k8s-debug-from-charm-container).) + +### Read service states + +`pebble services` reports a `Current` state for each service. The state tells you most of what you need to know: + +| State | Meaning | +| --- | --- | +| `active` | The service is running normally. | +| `inactive` | The service is not running. It was never started (`startup: disabled`), was stopped, or its command could not be executed at all (a wrong path or a binary missing from the image). | +| `backoff` | The service started but exited, and Pebble is restarting it on a backoff schedule -- the workload is crash-looping. | +| `error` | The service failed and Pebble has stopped trying to restart it. | + +`backoff` and `error` mean the process ran and then died -- look at the workload itself with `pebble logs` next. An unexpected `inactive` (a service you expected to be running) usually means the command never executed; the reason is in `pebble changes` / `pebble tasks` (see below) rather than in the service's logs. + +### Trace what Pebble did with changes and tasks + +When a service won't start, or a `replan` from your charm seems to have done nothing, the *change log* tells you what Pebble actually attempted and why it failed. This is the single most useful Pebble debugging command and is not obvious from the service list alone. + +`pebble changes` lists recent operations -- each replan, service start, service stop, and check change: + +```text +$ pebble changes +ID Status Spawn Ready Summary +1 Done today at 02:05 UTC today at 02:05 UTC Autostart service "myapp" +2 Error today at 02:09 UTC today at 02:09 UTC Start service "myapp" +``` + +A change in `Error` is your lead. Drill into its tasks to see the failure detail and the captured logs: + +```text +$ pebble tasks 2 +Status Spawn Ready Summary +Error today at 02:09 UTC today at 02:09 UTC Start service "myapp" + +...................................................................... +Start service "myapp" + +2026-05-22T02:09:01Z INFO Most recent service output: + Traceback (most recent call last): + ... + KeyError: 'DATABASE_URL' +2026-05-22T02:09:01Z ERROR service start attempt: exited quickly with code 1, will restart +``` + +The captured "Most recent service output" is the workload's own stdout/stderr, so a stack trace, a missing-config error, or a permission error shows up right here. A service whose command can't be executed at all (a wrong path, or a binary missing from the image) fails differently -- `cannot start service: fork/exec ...: no such file or directory`. Use `pebble tasks --last=start` to jump straight to the most recent service start without looking up its ID. + +### Verify the effective plan + +`pebble plan` prints the *merged* plan -- the result of combining every layer Pebble knows about. If a service is missing or has the wrong command, compare this against what your charm intended: + +```text +$ pebble plan +services: + myapp: + summary: my application + startup: enabled + override: replace + command: /bin/myapp --port 8080 +``` + +`pebble plan` is the right tool for "did my charm's configuration take effect?" -- it shows the live, in-memory result. A charm adds its configuration at runtime with [`Container.add_layer()`](ops.Container.add_layer), which sends the layer to Pebble over the API; those layers are held in memory and are *not* written to `/var/lib/pebble/default/layers/`. That directory contains only the layers baked into the container image (a rock built with Rockcraft may ship some), which Pebble reads once at startup -- so an empty or sparse layers directory is normal and doesn't mean your charm's `add_layer()` call failed. Trust `pebble plan`, not the directory listing. + +If you change a service's configuration but the running process doesn't change, the usual cause is a missing [`replan`](#run-workloads-with-a-charm-kubernetes-replan): adding a layer updates the plan but does not restart services on its own. + +### Check health checks + +If your charm configures {ref}`Pebble health checks `, a check that has failed its threshold can be the hidden cause of a problem you're chasing. Depending on how the check is configured, going "down" can restart the service (via `on-check-failure`), and a `level: alive` or `level: ready` check is wired to the container's Kubernetes liveness or readiness probe -- so a failing `alive` check makes Kubernetes restart the container, while a failing `ready` check marks the pod not-ready and removes it from its Service's endpoints. Inspect checks with: + +```shell +pebble checks # status of all checks +pebble check myapp-ready # full detail for one check, in YAML +pebble check myapp-ready --refresh # run it now instead of waiting for the next interval +pebble health # exit code 0 if all checks healthy, 1 otherwise +``` + +`pebble check ` shows the failure count, the threshold, and the error from the most recent run -- useful for distinguishing "the check is configured incorrectly" from "the workload is genuinely unhealthy". + +### Read notices and warnings + +If your charm responds to {ref}`custom notices ` and an expected `pebble_custom_notice` event never fires, check whether the notice was actually recorded: + +```shell +pebble notices # notices not yet acknowledged +pebble notice 4 # detail for one notice by ID +pebble warnings --all # Pebble's own warnings (deprecations, config issues) +``` + +(k8s-debug-from-charm-container)= +## Debug from the charm container + +Many production workload images are stripped down to just the application -- with no shell or utilities -- so `juju ssh --container` lands you nowhere useful. You can still run Pebble commands against that workload from the charm container, because the workload's socket is mounted there: + +```shell +juju ssh myapp/0 # connects to the charm container by default +export PEBBLE_SOCKET=/charm/myapp/pebble.sock # point at the workload's Pebble +/charm/bin/pebble services +/charm/bin/pebble logs +/charm/bin/pebble exec -- cat /etc/myapp/config.yaml +``` + +This is also the most faithful way to reproduce what your charm sees, since your charm talks to exactly this socket. If a Pebble command works here but your charm raises an error, the problem is in the charm code rather than the Pebble configuration. + +(k8s-inspect-the-pod)= +## Inspect the pod at the Kubernetes layer + +When a unit is stuck before Pebble is even reachable -- the container is `waiting`, the image won't pull, or the pod won't schedule -- the answer is below Juju, at the Kubernetes layer. Juju puts each model in its own namespace, and names each unit's pod `-`. + +```shell +kubectl -n get pods +kubectl -n describe pod myapp-0 # events: image pulls, scheduling, OOM kills, restarts +kubectl -n logs myapp-0 -c charm # charm container stdout +kubectl -n logs myapp-0 -c myapp # workload container stdout (Pebble's own output) +kubectl -n logs myapp-0 -c myapp --previous # output from the last crashed instance +``` + +The `Events` section of `describe pod` is where you'll find `ImagePullBackOff`, `CreateContainerConfigError`, failed liveness probes, and out-of-memory kills -- none of which appear in `juju debug-log` or the Pebble logs. + +```{note} +Reach for `kubectl` when the problem is the *container or pod* not coming up. Once the workload container is running and Pebble is responding, switch back to the Pebble CLI and `juju debug-log`, which give you a workload- and charm-aware view that raw `kubectl logs` does not. +``` + +(k8s-common-failure-modes)= +## Common failure modes + +| Symptom | Where to look | +| --- | --- | +| Charm stuck in `maintenance`/`waiting`; `can_connect()` is `False` | The workload container hasn't started -- `kubectl describe pod` for image-pull or scheduling errors ([](#k8s-inspect-the-pod)). | +| Service shows `backoff` or `error` | `pebble logs` for the crash output, then `pebble changes` / `pebble tasks` for the start failure ([](#k8s-pebble-cli)). | +| Config change has no effect on the running process | The charm added a layer but didn't [`replan`](#run-workloads-with-a-charm-kubernetes-replan); confirm with `pebble plan` and `pebble services`. | +| Charm raises `ConnectionError` mid-handler | The workload's Pebble became unreachable -- guard Pebble calls with `try`/`except` rather than `can_connect()` ([](ops.Container.can_connect)). | +| `pebble_custom_notice` never fires | Confirm the notice was recorded with `pebble notices`; check the `key` your handler matches on ([](#k8s-pebble-cli)). | +| Workload won't go ready despite running | A health check is failing -- `pebble checks` and `pebble check --refresh` ([](#k8s-pebble-cli)). | + +## Reproduce workload behaviour without deploying + +You don't always need a live model. [State-transition tests](/explanation/state-transition-testing) let you simulate a workload container, its Pebble plan, services, and `can_connect` state, and assert on what your charm does -- including the Pebble API calls it makes. This is the fastest loop for charm-side container logic, and it runs in CI. Use a live deployment and the tools above to confirm behaviour against the real workload image. + +> See more: {ref}`write-unit-tests-for-a-charm`, {external+pebble:doc}`Pebble | CLI commands ` diff --git a/docs/howto/debug-your-charm.md b/docs/howto/debug-your-charm.md index e93123bc9..675d09fbd 100644 --- a/docs/howto/debug-your-charm.md +++ b/docs/howto/debug-your-charm.md @@ -63,6 +63,10 @@ juju ssh --container Then use the Pebble CLI (available at `/charm/bin/pebble`) to inspect the workload. We've included several examples below. +```{tip} +For Kubernetes- and Pebble-specific debugging in more depth -- tracing Pebble changes and tasks, working with minimal workload images, and diagnosing common sidecar failure modes -- see {ref}`debug-a-kubernetes-charm`. +``` + > See also: {external+pebble:doc}`Pebble | CLI commands ` ### Check service status diff --git a/docs/howto/index.md b/docs/howto/index.md index 3dec8ecbf..27c54f4c8 100644 --- a/docs/howto/index.md +++ b/docs/howto/index.md @@ -70,6 +70,7 @@ Juju provides a variety of debugging tools, which Ops integrates with. Log from your charm Debug your charm +Debug a Kubernetes charm ``` ## Managing machine workloads From 7477801740e7656d8bd89f453749ffd831b0214c Mon Sep 17 00:00:00 2001 From: Tony Meyer Date: Tue, 26 May 2026 10:13:26 +1200 Subject: [PATCH 2/9] Apply suggestion from @tonyandrewmeyer --- docs/howto/debug-a-kubernetes-charm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto/debug-a-kubernetes-charm.md b/docs/howto/debug-a-kubernetes-charm.md index 55c44c42e..4a6d7aa35 100644 --- a/docs/howto/debug-a-kubernetes-charm.md +++ b/docs/howto/debug-a-kubernetes-charm.md @@ -9,7 +9,7 @@ myst: > See first: {ref}`debug-your-charm`, {ref}`workload-containers`, {external+juju:ref}`Juju | How to manage logs ` -A Kubernetes charm runs as a sidecar: each unit is a pod with a *charm container* (running your charm code) alongside zero or more *workload containers*. Juju injects {external+pebble:doc}`Pebble ` into every workload container as its service manager, and your charm talks to each workload's Pebble over an HTTP-on-Unix-socket API. +A Kubernetes charm runs as a sidecar: each unit is a pod with a *charm container* (running your charm code) alongside zero or more *workload containers*. Juju injects {external+pebble:doc}`Pebble` into every workload container as its service manager, and your charm talks to each workload's Pebble over an HTTP-on-Unix-socket API. This split is what makes debugging a Kubernetes charm different from debugging a machine charm: a problem can live in the charm code, in the Pebble configuration, in the workload process itself, or at the Kubernetes layer below all of them. This guide covers the Kubernetes- and Pebble-specific tools for narrowing that down. For the substrate-agnostic tools (`juju debug-log`, `juju debug-hooks`, `juju debug-code`, jhack, and remote debugging with VS Code), see {ref}`debug-your-charm`. From cd404ada654ff3fe44e981a96843e8dfe4b98e45 Mon Sep 17 00:00:00 2001 From: Tony Meyer Date: Fri, 29 May 2026 12:40:29 +1200 Subject: [PATCH 3/9] Shift content around based on review feedback. --- docs/howto/debug-a-kubernetes-charm.md | 35 +++++++++++++------------- 1 file changed, 17 insertions(+), 18 deletions(-) diff --git a/docs/howto/debug-a-kubernetes-charm.md b/docs/howto/debug-a-kubernetes-charm.md index 4a6d7aa35..ee9921156 100644 --- a/docs/howto/debug-a-kubernetes-charm.md +++ b/docs/howto/debug-a-kubernetes-charm.md @@ -29,9 +29,20 @@ The charm and workload containers each have their own filesystem and process spa - In the **workload container**, the socket is at `/var/lib/pebble/default/pebble.sock`. - In the **charm container**, the same socket is mounted at `/charm//pebble.sock`, and the Pebble CLI binary is at `/charm/bin/pebble`. -```{tip} -If [`Container.can_connect()`](ops.Container.can_connect) returns `False` or your charm raises [`ops.pebble.ConnectionError`](ops.pebble.ConnectionError), the charm container cannot reach the workload's Pebble over that socket. This usually means the workload container hasn't started yet (no [`PebbleReadyEvent`](ops.PebbleReadyEvent) has fired) -- look at the pod first (see [](#k8s-inspect-the-pod)), not at your charm code. -``` +(k8s-common-failure-modes)= +## Common failure modes + +If you're not sure where to start, find your symptom here and jump to the section that covers it: + +| Symptom | Where to look | +| --- | --- | +| Charm stuck in `maintenance`/`waiting`; `can_connect()` is `False` (or a Pebble call raises `ConnectionError`) at startup | The charm container can't reach the workload's Pebble -- usually the workload container hasn't started yet (no [`PebbleReadyEvent`](ops.PebbleReadyEvent) has fired). Look at the pod, not your charm code -- `kubectl describe pod` for image-pull or scheduling errors ([](#k8s-inspect-the-pod)). | +| Service shows `backoff` or `error` | `pebble logs` for the crash output, then `pebble changes` / `pebble tasks` for the start failure ([](#k8s-pebble-cli)). | +| Config change has no effect on the running process | The charm added a layer but didn't [`replan`](#run-workloads-with-a-charm-kubernetes-replan); confirm with `pebble plan` and `pebble services`. | +| Charm raises `ConnectionError` mid-handler | The workload's Pebble became unreachable -- guard Pebble calls with `try`/`except` rather than `can_connect()` ([](ops.Container.can_connect)). | +| `pebble_custom_notice` never fires | Confirm the notice was recorded with `pebble notices`; check the `key` your handler matches on ([](#k8s-pebble-cli)). | +| Workload won't go ready despite running | A health check is failing -- `pebble checks` and `pebble check --refresh` ([](#k8s-pebble-cli)). | +| `juju ssh --container` lands in an image with no shell or tools | The workload image is stripped down -- run Pebble against it from the charm container instead, where the socket is mounted ([](#k8s-debug-from-charm-container)). | (k8s-pebble-cli)= ## Inspect the workload with the Pebble CLI @@ -109,7 +120,7 @@ If you change a service's configuration but the running process doesn't change, ### Check health checks -If your charm configures {ref}`Pebble health checks `, a check that has failed its threshold can be the hidden cause of a problem you're chasing. Depending on how the check is configured, going "down" can restart the service (via `on-check-failure`), and a `level: alive` or `level: ready` check is wired to the container's Kubernetes liveness or readiness probe -- so a failing `alive` check makes Kubernetes restart the container, while a failing `ready` check marks the pod not-ready and removes it from its Service's endpoints. Inspect checks with: +A failed {ref}`Pebble health check ` behaves differently depending on how it's configured: going "down" can restart the service (via `on-check-failure`), and a `level: alive` or `level: ready` check is wired to the container's Kubernetes liveness or readiness probe -- so a failing `alive` check makes Kubernetes restart the container, while a failing `ready` check marks the pod not-ready and removes it from its Service's endpoints. Inspect checks with: ```shell pebble checks # status of all checks @@ -133,7 +144,7 @@ pebble warnings --all # Pebble's own warnings (deprecations, config issu (k8s-debug-from-charm-container)= ## Debug from the charm container -Many production workload images are stripped down to just the application -- with no shell or utilities -- so `juju ssh --container` lands you nowhere useful. You can still run Pebble commands against that workload from the charm container, because the workload's socket is mounted there: +Run Pebble commands against a workload from the charm container, where the workload's socket is mounted: ```shell juju ssh myapp/0 # connects to the charm container by default @@ -148,7 +159,7 @@ This is also the most faithful way to reproduce what your charm sees, since your (k8s-inspect-the-pod)= ## Inspect the pod at the Kubernetes layer -When a unit is stuck before Pebble is even reachable -- the container is `waiting`, the image won't pull, or the pod won't schedule -- the answer is below Juju, at the Kubernetes layer. Juju puts each model in its own namespace, and names each unit's pod `-`. +When a unit is stuck before Pebble is even reachable, drop below Juju to the Kubernetes layer. Juju puts each model in its own namespace, and names each unit's pod `-`. ```shell kubectl -n get pods @@ -164,18 +175,6 @@ The `Events` section of `describe pod` is where you'll find `ImagePullBackOff`, Reach for `kubectl` when the problem is the *container or pod* not coming up. Once the workload container is running and Pebble is responding, switch back to the Pebble CLI and `juju debug-log`, which give you a workload- and charm-aware view that raw `kubectl logs` does not. ``` -(k8s-common-failure-modes)= -## Common failure modes - -| Symptom | Where to look | -| --- | --- | -| Charm stuck in `maintenance`/`waiting`; `can_connect()` is `False` | The workload container hasn't started -- `kubectl describe pod` for image-pull or scheduling errors ([](#k8s-inspect-the-pod)). | -| Service shows `backoff` or `error` | `pebble logs` for the crash output, then `pebble changes` / `pebble tasks` for the start failure ([](#k8s-pebble-cli)). | -| Config change has no effect on the running process | The charm added a layer but didn't [`replan`](#run-workloads-with-a-charm-kubernetes-replan); confirm with `pebble plan` and `pebble services`. | -| Charm raises `ConnectionError` mid-handler | The workload's Pebble became unreachable -- guard Pebble calls with `try`/`except` rather than `can_connect()` ([](ops.Container.can_connect)). | -| `pebble_custom_notice` never fires | Confirm the notice was recorded with `pebble notices`; check the `key` your handler matches on ([](#k8s-pebble-cli)). | -| Workload won't go ready despite running | A health check is failing -- `pebble checks` and `pebble check --refresh` ([](#k8s-pebble-cli)). | - ## Reproduce workload behaviour without deploying You don't always need a live model. [State-transition tests](/explanation/state-transition-testing) let you simulate a workload container, its Pebble plan, services, and `can_connect` state, and assert on what your charm does -- including the Pebble API calls it makes. This is the fastest loop for charm-side container logic, and it runs in CI. Use a live deployment and the tools above to confirm behaviour against the real workload image. From 921d8bc1bbc3a4b2ebb5187c1e480b6337a04f9e Mon Sep 17 00:00:00 2001 From: Tony Meyer Date: Fri, 29 May 2026 12:50:40 +1200 Subject: [PATCH 4/9] Fix broken link. --- docs/howto/debug-a-kubernetes-charm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto/debug-a-kubernetes-charm.md b/docs/howto/debug-a-kubernetes-charm.md index ee9921156..f24508c54 100644 --- a/docs/howto/debug-a-kubernetes-charm.md +++ b/docs/howto/debug-a-kubernetes-charm.md @@ -9,7 +9,7 @@ myst: > See first: {ref}`debug-your-charm`, {ref}`workload-containers`, {external+juju:ref}`Juju | How to manage logs ` -A Kubernetes charm runs as a sidecar: each unit is a pod with a *charm container* (running your charm code) alongside zero or more *workload containers*. Juju injects {external+pebble:doc}`Pebble` into every workload container as its service manager, and your charm talks to each workload's Pebble over an HTTP-on-Unix-socket API. +A Kubernetes charm runs as a sidecar: each unit is a pod with a *charm container* (running your charm code) alongside zero or more *workload containers*. Juju injects {external+pebble:doc}`Pebble ` into every workload container as its service manager, and your charm talks to each workload's Pebble over an HTTP-on-Unix-socket API. This split is what makes debugging a Kubernetes charm different from debugging a machine charm: a problem can live in the charm code, in the Pebble configuration, in the workload process itself, or at the Kubernetes layer below all of them. This guide covers the Kubernetes- and Pebble-specific tools for narrowing that down. For the substrate-agnostic tools (`juju debug-log`, `juju debug-hooks`, `juju debug-code`, jhack, and remote debugging with VS Code), see {ref}`debug-your-charm`. From c2973113e3c2b913adf68135786014d34489248f Mon Sep 17 00:00:00 2001 From: Tony Meyer Date: Fri, 29 May 2026 15:53:36 +1200 Subject: [PATCH 5/9] Drop the uni ttest bit. --- docs/howto/debug-a-kubernetes-charm.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/docs/howto/debug-a-kubernetes-charm.md b/docs/howto/debug-a-kubernetes-charm.md index f24508c54..d50663840 100644 --- a/docs/howto/debug-a-kubernetes-charm.md +++ b/docs/howto/debug-a-kubernetes-charm.md @@ -174,9 +174,3 @@ The `Events` section of `describe pod` is where you'll find `ImagePullBackOff`, ```{note} Reach for `kubectl` when the problem is the *container or pod* not coming up. Once the workload container is running and Pebble is responding, switch back to the Pebble CLI and `juju debug-log`, which give you a workload- and charm-aware view that raw `kubectl logs` does not. ``` - -## Reproduce workload behaviour without deploying - -You don't always need a live model. [State-transition tests](/explanation/state-transition-testing) let you simulate a workload container, its Pebble plan, services, and `can_connect` state, and assert on what your charm does -- including the Pebble API calls it makes. This is the fastest loop for charm-side container logic, and it runs in CI. Use a live deployment and the tools above to confirm behaviour against the real workload image. - -> See more: {ref}`write-unit-tests-for-a-charm`, {external+pebble:doc}`Pebble | CLI commands ` From d16ba5b354433832278e663d46fd15861a24152e Mon Sep 17 00:00:00 2001 From: Tony Meyer Date: Fri, 5 Jun 2026 21:21:15 +1200 Subject: [PATCH 6/9] Apply suggestions from code review Co-authored-by: Dave Wilding --- docs/howto/debug-a-kubernetes-charm.md | 52 ++++++++++++++++---------- docs/howto/debug-your-charm.md | 7 ++-- 2 files changed, 36 insertions(+), 23 deletions(-) diff --git a/docs/howto/debug-a-kubernetes-charm.md b/docs/howto/debug-a-kubernetes-charm.md index d50663840..cd8acea16 100644 --- a/docs/howto/debug-a-kubernetes-charm.md +++ b/docs/howto/debug-a-kubernetes-charm.md @@ -9,25 +9,27 @@ myst: > See first: {ref}`debug-your-charm`, {ref}`workload-containers`, {external+juju:ref}`Juju | How to manage logs ` -A Kubernetes charm runs as a sidecar: each unit is a pod with a *charm container* (running your charm code) alongside zero or more *workload containers*. Juju injects {external+pebble:doc}`Pebble ` into every workload container as its service manager, and your charm talks to each workload's Pebble over an HTTP-on-Unix-socket API. +A Kubernetes charm runs as a sidecar: each unit is a pod with a *charm container* (running your charm code) alongside zero or more *workload containers*. Juju injects {external+pebble:doc}`Pebble ` into each workload container as its service manager. Your charm talks to each workload's Pebble over an HTTP-on-Unix-socket API. -This split is what makes debugging a Kubernetes charm different from debugging a machine charm: a problem can live in the charm code, in the Pebble configuration, in the workload process itself, or at the Kubernetes layer below all of them. This guide covers the Kubernetes- and Pebble-specific tools for narrowing that down. For the substrate-agnostic tools (`juju debug-log`, `juju debug-hooks`, `juju debug-code`, jhack, and remote debugging with VS Code), see {ref}`debug-your-charm`. +This split is what makes debugging a Kubernetes charm different from debugging a machine charm: a problem can live in the charm code, in the Pebble configuration, in the workload process itself, or at the Kubernetes layer below all of them. This guide covers the Kubernetes-specific tools for narrowing that down. + +For substrate-agnostic tools (`juju debug-log`, `juju debug-hooks`, `juju debug-code`, jhack, and remote debugging with VS Code), see {ref}`debug-your-charm`. (k8s-two-container-model)= ## Know which container you're looking at -Each piece of the system lives in a specific place, and reaching for the wrong one is the most common way to waste time: +Each piece of the system lives in a specific place. Reaching for the wrong one is the most common way to waste time. | What | Where it runs | How to reach it | | --- | --- | --- | -| Your charm code (`src/charm.py`, hooks, logs) | charm container (named `charm`) | `juju ssh `, `juju debug-log` | +| Your charm code (`src/charm.py`), logs, hooks | charm container (named `charm`) | `juju ssh `, `juju debug-log`, `juju debug-hooks` | | Pebble and the workload process | workload container (named after the `containers` entry in `charmcraft.yaml`) | `juju ssh --container `, the Pebble CLI | | The pod, image pulls, scheduling | Kubernetes | `kubectl` | The charm and workload containers each have their own filesystem and process space. The charm reaches a workload's Pebble through a Unix socket that Juju mounts into both containers: -- In the **workload container**, the socket is at `/var/lib/pebble/default/pebble.sock`. -- In the **charm container**, the same socket is mounted at `/charm//pebble.sock`, and the Pebble CLI binary is at `/charm/bin/pebble`. +- **Workload container** -- The socket is at `/var/lib/pebble/default/pebble.sock`. +- **Charm container** -- The same socket is mounted at `/charm//pebble.sock`. The Pebble CLI binary is at `/charm/bin/pebble`. (k8s-common-failure-modes)= ## Common failure modes @@ -36,24 +38,24 @@ If you're not sure where to start, find your symptom here and jump to the sectio | Symptom | Where to look | | --- | --- | -| Charm stuck in `maintenance`/`waiting`; `can_connect()` is `False` (or a Pebble call raises `ConnectionError`) at startup | The charm container can't reach the workload's Pebble -- usually the workload container hasn't started yet (no [`PebbleReadyEvent`](ops.PebbleReadyEvent) has fired). Look at the pod, not your charm code -- `kubectl describe pod` for image-pull or scheduling errors ([](#k8s-inspect-the-pod)). | +| Charm stuck in `maintenance`/`waiting`; `can_connect()` is `False` (or a Pebble call raises `ConnectionError`) at startup | The charm container can't reach the workload's Pebble. Usually the workload container hasn't started yet (no [`PebbleReadyEvent`](ops.PebbleReadyEvent) has fired). Look at the pod, not your charm code -- `kubectl describe pod` for image-pull or scheduling errors. See [](#k8s-inspect-the-pod) | | Service shows `backoff` or `error` | `pebble logs` for the crash output, then `pebble changes` / `pebble tasks` for the start failure ([](#k8s-pebble-cli)). | -| Config change has no effect on the running process | The charm added a layer but didn't [`replan`](#run-workloads-with-a-charm-kubernetes-replan); confirm with `pebble plan` and `pebble services`. | -| Charm raises `ConnectionError` mid-handler | The workload's Pebble became unreachable -- guard Pebble calls with `try`/`except` rather than `can_connect()` ([](ops.Container.can_connect)). | +| Config change has no effect on the running process | The charm added a layer but didn't [`replan`](#run-workloads-with-a-charm-kubernetes-replan). Confirm with `pebble plan` and `pebble services`. | +| Charm raises `ConnectionError` mid-handler | The workload's Pebble became unreachable. Guard Pebble calls with `try`/`except` rather than `can_connect()` ([](ops.Container.can_connect)). | | `pebble_custom_notice` never fires | Confirm the notice was recorded with `pebble notices`; check the `key` your handler matches on ([](#k8s-pebble-cli)). | -| Workload won't go ready despite running | A health check is failing -- `pebble checks` and `pebble check --refresh` ([](#k8s-pebble-cli)). | +| Workload isn't ready despite running | A health check is failing -- `pebble checks` and `pebble check --refresh` ([](#k8s-pebble-cli)). | | `juju ssh --container` lands in an image with no shell or tools | The workload image is stripped down -- run Pebble against it from the charm container instead, where the socket is mounted ([](#k8s-debug-from-charm-container)). | (k8s-pebble-cli)= ## Inspect the workload with the Pebble CLI -SSH into the workload container and use the Pebble CLI to see what Pebble thinks is going on. The {ref}`debug-your-charm` guide covers `pebble services`, `pebble logs`, `pebble exec`, `pebble plan`, and `pebble checks`. The commands below go deeper and are the ones you'll reach for when a workload won't start or keeps crashing. +SSH into the workload container and use the Pebble CLI to see what Pebble thinks is going on. {ref}`debug-your-charm` covers `pebble services`, `pebble logs`, `pebble exec`, `pebble plan`, and `pebble checks`. The commands below go deeper and are the ones you'll reach for when a workload won't start or keeps crashing. ```shell juju ssh --container myapp myapp/0 ``` -All of the examples below assume you're running them inside the workload container, where `pebble` is on the `PATH`. (To run them from the charm container instead, see [](#k8s-debug-from-charm-container).) +All of the examples below assume you're running them inside the workload container, where `pebble` is on the `PATH`. To run them from the charm container instead, see [](#k8s-debug-from-charm-container). ### Read service states @@ -63,10 +65,12 @@ All of the examples below assume you're running them inside the workload contain | --- | --- | | `active` | The service is running normally. | | `inactive` | The service is not running. It was never started (`startup: disabled`), was stopped, or its command could not be executed at all (a wrong path or a binary missing from the image). | -| `backoff` | The service started but exited, and Pebble is restarting it on a backoff schedule -- the workload is crash-looping. | +| `backoff` | The service started but exited, and Pebble is restarting it on a backoff schedule. The workload is crashing each time it starts. | | `error` | The service failed and Pebble has stopped trying to restart it. | -`backoff` and `error` mean the process ran and then died -- look at the workload itself with `pebble logs` next. An unexpected `inactive` (a service you expected to be running) usually means the command never executed; the reason is in `pebble changes` / `pebble tasks` (see below) rather than in the service's logs. +`backoff` and `error` mean the process ran and then died. Look at the workload itself with `pebble logs`. + +An unexpected `inactive` (a service you expected to be running) usually means the command never executed. The reason is in `pebble changes` / `pebble tasks`, rather than in the service's logs. We'll look at that next. ### Trace what Pebble did with changes and tasks @@ -98,7 +102,11 @@ Start service "myapp" 2026-05-22T02:09:01Z ERROR service start attempt: exited quickly with code 1, will restart ``` -The captured "Most recent service output" is the workload's own stdout/stderr, so a stack trace, a missing-config error, or a permission error shows up right here. A service whose command can't be executed at all (a wrong path, or a binary missing from the image) fails differently -- `cannot start service: fork/exec ...: no such file or directory`. Use `pebble tasks --last=start` to jump straight to the most recent service start without looking up its ID. +Instead of `pebble tasks `, use `pebble tasks --last=start` to jump straight to the most recent service start attempt. + +The captured "Most recent service output" is the workload's own stdout/stderr, so a stack trace, a missing-config error, or a permission error shows up right here. + +A service whose command can't be executed at all (a wrong path, or a binary missing from the image) fails differently. For example, `cannot start service: fork/exec ...: no such file or directory`. ### Verify the effective plan @@ -114,13 +122,19 @@ services: command: /bin/myapp --port 8080 ``` -`pebble plan` is the right tool for "did my charm's configuration take effect?" -- it shows the live, in-memory result. A charm adds its configuration at runtime with [`Container.add_layer()`](ops.Container.add_layer), which sends the layer to Pebble over the API; those layers are held in memory and are *not* written to `/var/lib/pebble/default/layers/`. That directory contains only the layers baked into the container image (a rock built with Rockcraft may ship some), which Pebble reads once at startup -- so an empty or sparse layers directory is normal and doesn't mean your charm's `add_layer()` call failed. Trust `pebble plan`, not the directory listing. +`pebble plan` is the right tool for "did my charm's configuration take effect?" -- it shows the live, in-memory result. + +A charm adds its configuration at runtime with [`Container.add_layer()`](ops.Container.add_layer), which sends the layer to Pebble over the API. Added layers are held in memory and are *not* written to `/var/lib/pebble/default/layers/`. That directory contains only the layers baked into the container image (a rock built with Rockcraft may ship some), which Pebble reads once at startup. An empty or sparse layers directory is normal and doesn't mean your charm's `add_layer()` call failed. Trust `pebble plan`, not the directory listing. -If you change a service's configuration but the running process doesn't change, the usual cause is a missing [`replan`](#run-workloads-with-a-charm-kubernetes-replan): adding a layer updates the plan but does not restart services on its own. +If you change a service's configuration but the running process doesn't change, the usual cause is a missing [`replan`](#run-workloads-with-a-charm-kubernetes-replan): Pebble updates the plan when you add a layer, but doesn't restart services until you call `replan`. ### Check health checks -A failed {ref}`Pebble health check ` behaves differently depending on how it's configured: going "down" can restart the service (via `on-check-failure`), and a `level: alive` or `level: ready` check is wired to the container's Kubernetes liveness or readiness probe -- so a failing `alive` check makes Kubernetes restart the container, while a failing `ready` check marks the pod not-ready and removes it from its Service's endpoints. Inspect checks with: +A failed {ref}`Pebble health check ` behaves differently depending on how it's configured: +- Going "down" can restart the service (set with `on-check-failure`). +- A `level: alive` or `level: ready` check is wired to the container's Kubernetes liveness or readiness probe. This means that a failing `alive` check makes Kubernetes restart the container, while a failing `ready` check marks the pod not-ready and removes it from its Service's endpoints. + +Inspect checks with: ```shell pebble checks # status of all checks @@ -129,7 +143,7 @@ pebble check myapp-ready --refresh # run it now instead of waiting for the nex pebble health # exit code 0 if all checks healthy, 1 otherwise ``` -`pebble check ` shows the failure count, the threshold, and the error from the most recent run -- useful for distinguishing "the check is configured incorrectly" from "the workload is genuinely unhealthy". +`pebble check ` shows the failure count, the threshold, and the error from the most recent run. This is useful for distinguishing "the check is configured incorrectly" from "the workload is genuinely unhealthy". ### Read notices and warnings diff --git a/docs/howto/debug-your-charm.md b/docs/howto/debug-your-charm.md index 675d09fbd..fec381b48 100644 --- a/docs/howto/debug-your-charm.md +++ b/docs/howto/debug-your-charm.md @@ -63,11 +63,10 @@ juju ssh --container Then use the Pebble CLI (available at `/charm/bin/pebble`) to inspect the workload. We've included several examples below. -```{tip} -For Kubernetes- and Pebble-specific debugging in more depth -- tracing Pebble changes and tasks, working with minimal workload images, and diagnosing common sidecar failure modes -- see {ref}`debug-a-kubernetes-charm`. -``` +See also: -> See also: {external+pebble:doc}`Pebble | CLI commands ` +- {ref}`debug-a-kubernetes-charm`, which covers Kubernetes charm failure modes and debugging tools in more depth +- {external+pebble:doc}`Pebble | CLI commands ` ### Check service status From 14d8bd932c356af202019621eb84b677957cd702 Mon Sep 17 00:00:00 2001 From: Tony Meyer Date: Fri, 5 Jun 2026 21:22:48 +1200 Subject: [PATCH 7/9] Apply suggestion from @tonyandrewmeyer --- docs/howto/debug-a-kubernetes-charm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto/debug-a-kubernetes-charm.md b/docs/howto/debug-a-kubernetes-charm.md index cd8acea16..d63e4131f 100644 --- a/docs/howto/debug-a-kubernetes-charm.md +++ b/docs/howto/debug-a-kubernetes-charm.md @@ -185,6 +185,6 @@ kubectl -n logs myapp-0 -c myapp --previous # output from the last cra The `Events` section of `describe pod` is where you'll find `ImagePullBackOff`, `CreateContainerConfigError`, failed liveness probes, and out-of-memory kills -- none of which appear in `juju debug-log` or the Pebble logs. -```{note} +```{tip} Reach for `kubectl` when the problem is the *container or pod* not coming up. Once the workload container is running and Pebble is responding, switch back to the Pebble CLI and `juju debug-log`, which give you a workload- and charm-aware view that raw `kubectl logs` does not. ``` From 4354705cf8ff78dbb4ee1498fd4362e27ed1033a Mon Sep 17 00:00:00 2001 From: Tony Meyer Date: Fri, 5 Jun 2026 21:30:02 +1200 Subject: [PATCH 8/9] Address review: consistent connectives, SSH subsection - Replace `--`/`;` connectives with full stops in the symptoms table. - Promote the workload-container SSH command to its own subsection. Co-Authored-By: Claude Opus 4.7 --- docs/howto/debug-a-kubernetes-charm.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/howto/debug-a-kubernetes-charm.md b/docs/howto/debug-a-kubernetes-charm.md index d63e4131f..502361e29 100644 --- a/docs/howto/debug-a-kubernetes-charm.md +++ b/docs/howto/debug-a-kubernetes-charm.md @@ -38,19 +38,21 @@ If you're not sure where to start, find your symptom here and jump to the sectio | Symptom | Where to look | | --- | --- | -| Charm stuck in `maintenance`/`waiting`; `can_connect()` is `False` (or a Pebble call raises `ConnectionError`) at startup | The charm container can't reach the workload's Pebble. Usually the workload container hasn't started yet (no [`PebbleReadyEvent`](ops.PebbleReadyEvent) has fired). Look at the pod, not your charm code -- `kubectl describe pod` for image-pull or scheduling errors. See [](#k8s-inspect-the-pod) | +| Charm stuck in `maintenance`/`waiting`; `can_connect()` is `False` (or a Pebble call raises `ConnectionError`) at startup | The charm container can't reach the workload's Pebble. Usually the workload container hasn't started yet (no [`PebbleReadyEvent`](ops.PebbleReadyEvent) has fired). Look at the pod, not your charm code. Run `kubectl describe pod` for image-pull or scheduling errors. See [](#k8s-inspect-the-pod) | | Service shows `backoff` or `error` | `pebble logs` for the crash output, then `pebble changes` / `pebble tasks` for the start failure ([](#k8s-pebble-cli)). | | Config change has no effect on the running process | The charm added a layer but didn't [`replan`](#run-workloads-with-a-charm-kubernetes-replan). Confirm with `pebble plan` and `pebble services`. | | Charm raises `ConnectionError` mid-handler | The workload's Pebble became unreachable. Guard Pebble calls with `try`/`except` rather than `can_connect()` ([](ops.Container.can_connect)). | -| `pebble_custom_notice` never fires | Confirm the notice was recorded with `pebble notices`; check the `key` your handler matches on ([](#k8s-pebble-cli)). | -| Workload isn't ready despite running | A health check is failing -- `pebble checks` and `pebble check --refresh` ([](#k8s-pebble-cli)). | -| `juju ssh --container` lands in an image with no shell or tools | The workload image is stripped down -- run Pebble against it from the charm container instead, where the socket is mounted ([](#k8s-debug-from-charm-container)). | +| `pebble_custom_notice` never fires | Confirm the notice was recorded with `pebble notices`. Check the `key` your handler matches on ([](#k8s-pebble-cli)). | +| Workload isn't ready despite running | A health check is failing. Run `pebble checks` and `pebble check --refresh` ([](#k8s-pebble-cli)). | +| `juju ssh --container` lands in an image with no shell or tools | The workload image is stripped down. Run Pebble against it from the charm container instead, where the socket is mounted ([](#k8s-debug-from-charm-container)). | (k8s-pebble-cli)= ## Inspect the workload with the Pebble CLI SSH into the workload container and use the Pebble CLI to see what Pebble thinks is going on. {ref}`debug-your-charm` covers `pebble services`, `pebble logs`, `pebble exec`, `pebble plan`, and `pebble checks`. The commands below go deeper and are the ones you'll reach for when a workload won't start or keeps crashing. +### SSH into the workload container + ```shell juju ssh --container myapp myapp/0 ``` From 16c01cbf6e326dfbf1370cd451a43610e83f1e1f Mon Sep 17 00:00:00 2001 From: Tony Meyer Date: Fri, 5 Jun 2026 22:32:26 +1200 Subject: [PATCH 9/9] Don't tell people the charm container is named 'charm'. --- docs/howto/debug-a-kubernetes-charm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto/debug-a-kubernetes-charm.md b/docs/howto/debug-a-kubernetes-charm.md index 502361e29..79a57a767 100644 --- a/docs/howto/debug-a-kubernetes-charm.md +++ b/docs/howto/debug-a-kubernetes-charm.md @@ -22,7 +22,7 @@ Each piece of the system lives in a specific place. Reaching for the wrong one i | What | Where it runs | How to reach it | | --- | --- | --- | -| Your charm code (`src/charm.py`), logs, hooks | charm container (named `charm`) | `juju ssh `, `juju debug-log`, `juju debug-hooks` | +| Your charm code (`src/charm.py`), logs, hooks | charm container | `juju ssh `, `juju debug-log`, `juju debug-hooks` | | Pebble and the workload process | workload container (named after the `containers` entry in `charmcraft.yaml`) | `juju ssh --container `, the Pebble CLI | | The pod, image pulls, scheduling | Kubernetes | `kubectl` |