container_fs metrics {device} has no label or info-metric to associate it with a volumeattachment or persistentvolumeclaim

[ringerc]

Problem

There appears to be no label or info-metric to associate cadvisor's container_fs_* metrics with a PersistentVolume attachment or PersistentVolumeClaim, or with the mount-point of the fs within the container. There is only a device label, for the device-node path within the OS.

This makes it seemingly impossible to determine which meaningful volume a container's I/O is associated with; for example, if a database container has two PVs mounted, one for the main DB and one for WAL, and it also has an ephemeral volume for tempfiles and sorts, there seems to be no way to tell which container_fs_writes_bytes_total metric is for which of the volumes.

Proposed feature

At would be enormously helpful if cadvsor added a label or exposed an info metric associating (device,container) label-pairs from container_fs_* metrics with the k8s volume attachment name or persistent volume claim.

It'd also be great to have an info-metric exposing the volume mount path within the container to the (container,device). This can't be done as an extra label on the container_fs_* metrics because one device node can be mounted multiple times within one container (bind mounts, subvolume mounts, btrfs submounts, etc). This would make it possible to see the container-path the volume is mounted on in monitoring. It would also then be possible to associate the persistent volume by exposing the volumeMount paths for a Pod in kube-state-metrics.

Exposing the filesystem uuid would also be helpful.

Alternatives considered

kube-state-metrics cannot provide this because it has no insight into the device node a container volumeMount path is associated with. There's nothing usable in PersistentVolume or PersistentVolumeClaim's .spec or .status. cadvisor doesn't appear to expose the csi info, pvc uid which could be used to associate these. There is a VolumeAttachment CR with .status.metadata.devicePath (for some CSIs) which k-s-m exposes as kube_volumeattachment_status_attachment_metadata but this only seems to be provided by the AWS EKS CSI, and the volume paths differ to those seen within the container e.g. an in-container /dev/dm-0 is exposed as /dev/xvdaa in the attachment metadata. Thus this is not usable for volume associations.

node-exporter recently gained filesystem_mount_info (prometheus/node_exporter#2970) which maps device to mountpoint - but it isn't container-scoped and doesn't expose the volume attachment so it's not usable for associating the device with a PV. Its older node_filesystem_avail_bytes{device,mountpoint} similarly exposes host-path mount points under /run/containerd/io.containerd.grpc.v1.cri/ and has no info that could be used to associate with a PV, PV attachment or volumeMount. (Due to mount-scoping rules it cannot see some mounts anyway).

kubelet metrics don't appear to expose the needed info either, and there's nothing apparent in the main k8s metrics docs either. Querying kubectl get --raw "/api/v1/nodes/NODENAME/proxy/metrics" and kubectl get --raw "/api/v1/nodes/NODENAME/proxy/metrics/resource" didn't reveal anything promising.

Kubelet /stats/summary is (a) deprecated and (b) exposes the volume's name as listed in Pod.spec.volumes and any pvcRef but not the device-node or mount-path so it cannot be used to associate metrics. It doesn't have I/O stats so it's not an alternative data source either.

So I didn't find any way to associate the cadvisor metrics to the pv attachment, pvc, and pv by fs uuid, pv uuid, pvc uuid, data exposed by kube apiserver, other existing metrics api servers, etc.

Benefits

If a volumeattachment label was available directly or via an info-metric, this could be joined on kube_volumeattachment_spec_source_persistentvolume from kube-state-metrics to find the kube_persistentvolumeclaim_info, kube_persisistentvolume_info, kube_persistentvolumeclaim_labels, etc.

If a mapping of volumeMount paths to volumes and devices was available, I/O could be associated with a specific container path in reporting and dashboards, e.g. "100MiB/s on /postgres/data, 200 MiB/s on /postgres/pg_wal, 500MiB/s on /postgres/ephemeral_store_tablespace".

Details

cadvisor exposes some useful container-level filesystem I/O metrics:

• container_fs_reads_bytes_total
• container_fs_reads_total
• container_fs_writes_bytes_total
• container_fs_writes_total

which are exposed with labels including device (device node path the filesystem is mounted from) and name (container-id without containerd:// prefix), e.g.

container_fs_reads_bytes_total{container="...", device="/dev/dm-0", job="kubelet", metrics_path="/metrics/cadvisor", name="...", pod="...", ...}

There is nothing here, or in any of the other cadvisor metrics I found, that would allow this to be associated with a persistent volume claim. kube-state-metrics cannot expose this information because it does not have access to the device-node paths from which volumes are mounted within containers. See kubernetes/kube-state-metrics#1701

container_fs_writes_bytes_total and container_fs_reads_bytes_total get their data from /sys/fs/cgroup/blkio/blkio.throttle.io_service_bytes on cgroups-v1 nodes, or /sys/fs/cgroup/[...]/io.stat on cgroups-v2 nodes. This file offers device-node level info only, so the data source itself is can't be directly used to enrich with the needed info.

Looking at the cadvisor source:

• there's FsInfo in MachineInfo which knows the device-node path, but not any volume attachment or persistent volume info.
• there's PerDiskStats in DiskIoStats in ContainerStats, but nothing there associates with a volume attachment or a mount path. There's FsStats in the same file, which again is only keyed by Device and filesystem type, it doesn't have path or attachment info.
• in metrics/prometheus.go the metrics with a "device" label don't appear to have anything else to associate with an attachment or claim; I didn't find likely keywords like "vol", "attach", "mount" or "path" anywhere.
• There's a GetFsInfoByFsUUID function, and fs/fs.go uses https://pkg.go.dev/github.com/moby/sys/mountinfo#Info so it has access to the mount uid, but the FS UUID isn't exposed as a label, and even if it was there doesn't seem to be anything elsewhere to use the fs uuid to join on for a volume attachment etc.

There's container_blkio_device_usage_total with major minor and operation but that doesn't provide any association; the rest only have device as a label. It can be joined on node_disk_info by (major,minor) but that doesn't do anything to help map it to a workload, as the one disk is typically shared by many workloads.

It looks like cadvisor could expose an info-metric on device -> mount points using Mountpoint from https://pkg.go.dev/github.com/moby/sys/mountinfo#Info, and expose the filesystem uid too. This doesn't provide a way to associate with a PV or PVC directly, but might be usable indirectly via pod metadata from kube-state-metrics etc since volume and volumeMount on a Pod are exposed in the API.

Ideally kubelet could expose this mapping instead. Perhaps via https://kubernetes.io/docs/reference/instrumentation/cri-pod-container-metrics/ . There's no sign it does so though.

---

Related: #1702

[ringerc]

Some deeper digging into this suggests a reason why cadvisor does not expose per-PV I/O metrics: the kernel doesn't expose them in a way it can access, or at least not necessarily in a consistent and guaranteed manner that's independent of the CSI driver.

It'd still be great to have a mapping of the block device in the container <-> the fs mount <-> the block device <-> the pvc <-> the PV, but for some CSI drivers this mapping would be non-unique, with multiple mounts in one container mapped to one block device, limiting the utility of the statistics in some cases.

It cannot be assumed that there's a 1:1 mapping of block device (major:minor) to a PVC mount in a container. This is up to the CSI. Exposing the mapping of mount path -> PVC would still be extremely useful.

The association asked for in this ticket could only reliably be determined when using a CSI driver that has a 1:1 mapping of Linux kernel block devices <--> kube PersistentVolumes, since I/O statistics are tracked in /sys/fs/cgroup/**/io.stat where io.stat is a mapping of block device major:minor -> I/O stats..

The following shows that it's not really possible on kind with the default storageclass/standard csi driver rancher.io/local-path for example.

Lets look at this workload as an example, running on kind on a Ubuntu host (cgroups v2, 6.8.0-64-generic):

$ kubectl get pod -n p-epbf9ucd3x-a p-epbf9ucd3x-a-1-1 -o yaml | yq '[.spec.volumes[]|select(.persistentVolumeClaim!=null)]'
- name: pgdata
  persistentVolumeClaim:
    claimName: p-epbf9ucd3x-a-1-1
- name: pg-wal
  persistentVolumeClaim:
    claimName: p-epbf9ucd3x-a-1-1-wal

So we can see it has 2 PVCs associated.

Here's the running container:

$ kubectl get pod -n p-epbf9ucd3x-a p-epbf9ucd3x-a-1-1 -o yaml | yq '.status.containerStatuses[]|select(.name=="postgres")|.containerID'
containerd://0403f5d781e17afb7c6adf03cc8c8d6cf0eef9fd23b6a53c1e3e02ab9a047dff

Here's the relevant metric series returned by max without (id, endpoint, service, image, instance, metrics_path, node, name) (container_fs_reads_bytes_total{name="0403f5d781e17afb7c6adf03cc8c8d6cf0eef9fd23b6a53c1e3e02ab9a047dff"}):

{container="postgres", device="/dev/dm-0", job="kubelet", namespace="p-epbf9ucd3x-a", pod="p-epbf9ucd3x-a-1-1"} => 316071936 @[1753669820.387]
{container="postgres", device="/dev/dm-1", job="kubelet", namespace="p-epbf9ucd3x-a", pod="p-epbf9ucd3x-a-1-1"} => 316071936 @[1753669820.387]
{container="postgres", device="/dev/nvme1n1", job="kubelet", namespace="p-epbf9ucd3x-a", pod="p-epbf9ucd3x-a-1-1"} => 316071936 @[1753669820.387]

Now lets look at the workload low level stats:

/ # nsenter -t 1 -m -p -u bash -i
/ # PS1="# "
# container_cg_scope="$(find  /sys/fs/cgroup/ -name \*0403f5d781e17afb7c6adf03cc8c8d6cf0eef9fd23b6a53c1e3e02ab9a047dff\*)"
# echo "${container_cg_scope}"
/sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/kubelet-kubepods-burstable-pod62d531d3_968f_42c1_8093_c2fa79554302.slice/cri-containerd-0403f5d781e17afb7c6adf03cc8c8d6cf0eef9fd23b6a53c1e3e02ab9a047dff.scope

# cat $container_cg_scope/io.stat 
259:5 rbytes=316071936 wbytes=0 rios=2544 wios=2601 dbytes=0 dios=0
252:0 rbytes=316071936 wbytes=67080192 rios=2544 wios=3255 dbytes=0 dios=0
252:1 rbytes=316071936 wbytes=136794112 rios=2544 wios=3454 dbytes=0 dios=0

What about mapping to block devices?

#  lsblk --include 259
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme0n1     259:0    0 476.9G  0 disk 
|-nvme0n1p1 259:1    0  1000M  0 part 
|-nvme0n1p2 259:2    0    16M  0 part 
|-nvme0n1p3 259:3    0 475.3G  0 part 
`-nvme0n1p4 259:4    0   651M  0 part 
nvme1n1     259:5    0   1.8T  0 disk 
|-nvme1n1p1 259:6    0   512M  0 part 
|-nvme1n1p2 259:7    0   1.7G  0 part 
`-nvme1n1p3 259:8    0   1.8T  0 part 
#  lsblk --include 252
# ls -l /dev/dm-*
brw-rw---- 1 root disk 252, 0 Jul 24 23:19 /dev/dm-0
brw-rw---- 1 root disk 252, 1 Jul 24 23:19 /dev/dm-1
brw-rw---- 1 root disk 252, 2 Jul 24 23:19 /dev/dm-2
# cat /sys/dev/block/252:0/uevent
MAJOR=252
MINOR=0
DEVNAME=dm-0
DEVTYPE=disk
DISKSEQ=13
# cat /sys/dev/block/252:1/uevent
MAJOR=252
MINOR=1
DEVNAME=dm-1
DEVTYPE=disk
DISKSEQ=14
# cat /sys/dev/block/252:0/dm/name
nvme0n1p3_crypt
# cat /sys/dev/block/252:1/dm/name
vgubuntu-root

So we can map them to block devices. But what about mapping the mount points within the container to the block device?

#  nsenter -t $(head -1 "${container_cg_scope}/cgroup.procs") -a findmnt --evaluate --nofsroot -t ext4 -M /var/lib/postgresql/data
TARGET                   SOURCE                    FSTYPE OPTIONS
/var/lib/postgresql/data /dev/mapper/vgubuntu-root ext4   rw,relatime,errors=remount-ro

#  nsenter -t $(head -1 "${container_cg_scope}/cgroup.procs") -a findmnt --evaluate --nofsroot -t ext4 -M /var/lib/postgresql/wal 
TARGET                  SOURCE                    FSTYPE OPTIONS
/var/lib/postgresql/wal /dev/mapper/vgubuntu-root ext4   rw,relatime,errors=remount-ro

the mtab actually shows the fs mount indirected via the host's LVM, and both mounted volumes have the same block device. Both are on the DM device dm-1.

So - cadvisor could probably tell us that the container has /var/lib/postgresql/data mounted from /dev/mapper/vgubuntu-root and that /dev/mapper/vgubuntu-root, or perhaps resolve it to the lower level /dev/dm-1.

But both would still point to the same block device, since the underlying kernel I/O stats are for the block device not the FS.

(With all this said, in the most common cases of real CSIs such a 1:1 mapping exists, and would be extremely useful to expose)

node-exporter can't expose this: it cannot see inside the container's mounts.

kube-state-metrics cannot expose this: it cannot map the container mount point and volume to a block device.

Only cadvisor or the kubelet itself has the access to expose info on which block device the container's pvc mount has mapped to.

[tiemma]

Thanks for the details @ringerc, did you later find any solutions to this? From your notes, it appears this is still not possible.

[ringerc]

@tiemma Correct.

It would make sense to add an info-metric series to cadvisor to expose a mapping of mount path -> PVC.

And it would also make sense to expose a mapping of PVC -> block device path from cadvisor - but with the caveat that there is no guarantee this is a 1:1 mapping; it's possible for multiple PVCs to map to one block device.

I don't expect to have time to try to add these info metrics to cadvisor any time soon, as I'm currently working on improving peak memory use for high cardinality queries in Thanos. I suspect it's not especially hard to add, just a matter of digging through the cadvisor structure and helpers.

[tiemma]

I'm working on a fix in #3844, hopefully the reviewers agree with the approach.