Description:
When cadvisor supports collecting disk usage metrics of containerd containers, it uses the root directory of the container, causing the value of the container_fs_usage_bytes metric to include the size of the read-only layer.
// container/containerd/handler.go L152:156
// Add the name and bare ID as aliases of the container. handler.image = cntr.Image if handler.includedMetrics.Has(container.DiskUsageMetrics) { handler.rootfsStorageDir = path.Join(rootfsDir, *ArgContainerdNamespace, id, "rootfs") handler.fsHandler = common.NewFsHandler(common.DefaultPeriod, handler.rootfsStorageDir, "", fsInfo) }
The rootfsStorageDir value constructed here is: /run/containerd/io.containerd.runtime.v2.task/k8s.io/{container id}/roofts, which is the container root directory. Then, GetDirUsage of fs.go is used to calculate the disk usage of its directory and subdirectories.
We all know that the container_fs_usage_bytes metric indicates the usage of the container's writable layer. Therefore, this value is inaccurate, and I have actually verified this result.
Steps to Reproduce:
- Deploy this code independently
- Query the container_fs_usage_bytes indicator of the container on the node
- Log in to the node and view the writable layer directory of the container on the host
- Found that the value of the container_fs_usage_bytes indicator > the size of the writable layer directory
Expected:
handler.fsHandler = common.NewFsHandler(common.DefaultPeriod, handler.rootfsStorageDir, "", fsInfo)
The rootfsStorageDir used should be the container's writable layer directory. Similar to docker factory, the writable layer directory is used when constructing fsHandler.
ENV:
Description:
When cadvisor supports collecting disk usage metrics of containerd containers, it uses the root directory of the container, causing the value of the container_fs_usage_bytes metric to include the size of the read-only layer.
// container/containerd/handler.go L152:156
// Add the name and bare ID as aliases of the container. handler.image = cntr.Image if handler.includedMetrics.Has(container.DiskUsageMetrics) { handler.rootfsStorageDir = path.Join(rootfsDir, *ArgContainerdNamespace, id, "rootfs") handler.fsHandler = common.NewFsHandler(common.DefaultPeriod, handler.rootfsStorageDir, "", fsInfo) }The rootfsStorageDir value constructed here is: /run/containerd/io.containerd.runtime.v2.task/k8s.io/{container id}/roofts, which is the container root directory. Then, GetDirUsage of fs.go is used to calculate the disk usage of its directory and subdirectories.
We all know that the container_fs_usage_bytes metric indicates the usage of the container's writable layer. Therefore, this value is inaccurate, and I have actually verified this result.
Steps to Reproduce:
Expected:
handler.fsHandler = common.NewFsHandler(common.DefaultPeriod, handler.rootfsStorageDir, "", fsInfo)The rootfsStorageDir used should be the container's writable layer directory. Similar to docker factory, the writable layer directory is used when constructing fsHandler.
ENV:
Related PR: support disk usage metrics for containerd factory #3502