docker: keep metadata when containerd socket is unavailable#3910
Open
U-S-jun wants to merge 1 commit into
Open
docker: keep metadata when containerd socket is unavailable#3910U-S-jun wants to merge 1 commit into
U-S-jun wants to merge 1 commit into
Conversation
Keep the Docker factory registered when the containerd snapshotter socket cannot be reached. Disable Docker disk usage metrics for that factory so container names and labels remain available through Docker inspect.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Keep Docker metadata collection available when cAdvisor detects the
containerd snapshotter storage driver (
overlayfs) but cannot connect tothe containerd socket.
This is a follow-up to #3709.
Problem
In the reproduced Docker Desktop environment, the Docker API is available
through
/var/run/docker.sockand reportsoverlayfsas the storagedriver, but
/run/containerd/containerd.sockis not exposed to thecAdvisor container.
cAdvisor currently treats failure to create the containerd client as a
Docker factory registration failure:
The raw cgroup handler continues to expose resource metrics, but Docker
metadata is unavailable because the Docker handler is never created.
Before this change, metrics only contained the cgroup path:
The container name, image, Docker labels, and Compose labels were missing.
Fix
DiskUsageMetricsfor the Docker factory instead of aborting registration.Docker
ContainerInspect.overlayfsDocker handler without acontainerd client or disk usage metrics.
After this change, Docker factory registration succeeds and metadata is
restored:
Degraded behavior
When the containerd socket is unavailable, per-container disk usage metrics
such as
container_fs_usage_bytes, filesystem limits, and inode usage arenot reported.
CPU, memory, disk I/O, container names, images, and Docker/Compose labels
remain available.
Testing
go test -count=1 ./container/dockercd lib && go test -count=1 ./container/... ./manager ./metricsdeploy/Dockerfileoverlayfsstorage driver
connection warning
name, image, Compose service, and custom Docker labels arepresent in Prometheus metrics