Can we do self monitoring for grafana mimir without grafana-agent as we are not able to install grafana-agent on out k8s cluster.

[Shindek77]

In our customer k8s cluster currently we are not able install grafana-agent due crds present inside it. So can we tset self monitoring dashbaordsof grafana-mimir with promethues. If yes then what we have to do configuration for same.

[sj14]

You say it's due to the CRDs. You can also install the agent without the operator: https://grafana.com/docs/grafana-cloud/kubernetes-monitoring/other-methods/k8s-agent-metrics/

[Shindek77]

@sj14 , Thank you for your response. Is there anything other than grafana-agent we can use to scrape metrics from grafana-mimir for self monitoring. If yes then what is tool and how to configure.

[dimitarvdimitrov]

You can scrape Prometheus metrics from the Mimir pods like from any other pod. The pods expose a /metrics endpoint on port 8080 that Prometheus can scrape. For logs, I suspect you should be able to use whatever setup you have for the rest of your kubernetes cluster.

There are some labels that should be correctly added to metrics and logs. Please see the rules and dashboards requirements. If you follow these, then you can easily use the prebuilt rules and dashboards. Mind that in order to use the "Slow Queries" dashboard the logs should be available in a Loki datasource in Grafana. For the rest of the dashboards the metrics should be available in any Prometheus-compatible store (e.g. Prometheus or Mimir itself)