Skip to content

docker: keep metadata when containerd socket is unavailable#3910

Open
U-S-jun wants to merge 1 commit into
google:masterfrom
U-S-jun:codex/docker-metadata-fallback
Open

docker: keep metadata when containerd socket is unavailable#3910
U-S-jun wants to merge 1 commit into
google:masterfrom
U-S-jun:codex/docker-metadata-fallback

Conversation

@U-S-jun

@U-S-jun U-S-jun commented Jul 10, 2026

Copy link
Copy Markdown

Summary

Keep Docker metadata collection available when cAdvisor detects the
containerd snapshotter storage driver (overlayfs) but cannot connect to
the containerd socket.

This is a follow-up to #3709.

Problem

In the reproduced Docker Desktop environment, the Docker API is available
through /var/run/docker.sock and reports overlayfs as the storage
driver, but /run/containerd/containerd.sock is not exposed to the
cAdvisor container.

cAdvisor currently treats failure to create the containerd client as a
Docker factory registration failure:

Registration of the docker container factory failed:
unable to create containerd client:
containerd: cannot unix dial containerd api service:
dial unix /run/containerd/containerd.sock:
connect: no such file or directory

The raw cgroup handler continues to expose resource metrics, but Docker
metadata is unavailable because the Docker handler is never created.

Before this change, metrics only contained the cgroup path:

container_last_seen{id="/docker/<container-id>"}

The container name, image, Docker labels, and Compose labels were missing.

Fix

  • Only create the containerd client when Docker disk usage metrics are enabled.
  • If the containerd socket is unavailable, log a warning and disable
    DiskUsageMetrics for the Docker factory instead of aborting registration.
  • Only resolve the container root filesystem when disk usage metrics are enabled.
  • Continue creating Docker handlers so metadata can be collected through
    Docker ContainerInspect.
  • Add a regression test covering an overlayfs Docker handler without a
    containerd client or disk usage metrics.

After this change, Docker factory registration succeeds and metadata is
restored:

container_last_seen{
  id="/docker/<container-id>",
  name="cadvisor_label_test_backend",
  container_label_com_docker_compose_service="backend",
  container_label_test_role="backend"
}

Degraded behavior

When the containerd socket is unavailable, per-container disk usage metrics
such as container_fs_usage_bytes, filesystem limits, and inode usage are
not reported.

CPU, memory, disk I/O, container names, images, and Docker/Compose labels
remain available.

Testing

  • go test -count=1 ./container/docker
  • cd lib && go test -count=1 ./container/... ./manager ./metrics
  • Built the cAdvisor image with deploy/Dockerfile
  • Verified with Docker Desktop 28.5.1, cgroup v2, and the overlayfs
    storage driver
  • Confirmed Docker factory registration succeeds after the containerd
    connection warning
  • Confirmed name, image, Compose service, and custom Docker labels are
    present in Prometheus metrics
  • Confirmed no panic or repeated stats collection errors

Keep the Docker factory registered when the containerd snapshotter socket cannot be reached. Disable Docker disk usage metrics for that factory so container names and labels remain available through Docker inspect.
@U-S-jun U-S-jun changed the title docker: keep metadata when containerd is unavailable docker: keep metadata when containerd socket is unavailable Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant