mimirtool is not loading rules alerts giving 500 internal error

[muwiess]

We are trying to use the binaries mimirtool-windows-amd64.exe/mimirtool-linux-amd64. We are getting the following error when we use load rules:

./mimirtool-linux-amd64 rules load alerts.yaml --address=http://localhost:8080 --id=test-tenatnid
group: 'mimir-group', ns: 'mimirt'
/prometheus/config/v1/rules/mimirt/mimir-group
ERRO[0000] server returned HTTP status 500 Internal Server Error: mkdir /ruler: read-only file system fields.msg="mkdir /ruler: read-only file system" status="500 Internal Server Error"
ERRO[0000] unable to load rule group error="POST request to http://localhost:8080/prometheus/config/v1/rules/tview-mimir-test failed: server returned HTTP status 500 Internal Server Error: mkdir /ruler: read-only file system" group=mimir-group namespace=mimirt
mimirtool-linux-amd64: error: load operation unsuccessful, try --help

The tool is trying to create a /ruler directory in the ruler container where the filesytem is read-only.

We also are getting errors when we are trying to load Alertmanager configurations.

Is there another way that we can load up the alert rules or alertmanager configurations , similar like the Prometheus loads them up within configuration files under serverFiles.recording_rules.yml: {} serverFiles.alerting_rules.yml: {} ?

[muwiess]

Helm Chart mimir-distributed-3.1.0
Grafana Mimir 2.3.1
mimirtool binaries, latest version

[muwiess]

It comes by default to /data in the Helm values.yaml file:

    ruler:
      alertmanager_url: dnssrvnoa+http://_http-metrics._tcp.{{ template "mimir.fullname" . }}-alertmanager-headless.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}/alertmanager
      enable_api: true
      rule_path: /data

I changed to /ruler and tested but the ruler container is erroring:

kubectl logs tviewmimir-mimir01test-ruler-84ff5b6ccd-x7gcj -n tview-mimir-test
error validating config: the configured ruler data directory "/ruler" cannot overlap with the configured ruler storage filesystem directory "ruler"; please set different paths, also ensuring one is not a subdirectory of the other one

So the helm chart default value it's fine to /data but the binary is expecting /ruler being created

Can you point me to the mimirtool repo to compile a new binary?
We have been working on Mimir's deployment for almost a month and now this is the last issue that we need to overcome.

[muwiess]

I fix it by adding a new volume to the container:

    - mountPath: /ruler
      name: ruler

in the deployment file.

Can this be fix for any subsequent release?

[dimitarvdimitrov]

what is the reason for using rule_path: /ruler in the first place? Why doesn't the default /data work for you?

[muwiess]

That is a good question, I am not using rulePath: /ruler, as default my deployment uses /data as the original helm chart defaults. In the other hand the binary requires to use /ruler and fails when is not present.

[Logiraptor]

@muwiess Are you using the filesystem backend for the ruler_storage? I think the error you're getting is related to the ruler object storage, not the rule path. The helm chart will create a volume for the rule path, but not for the object storage filesystem backend because it expects you to either use the provided minio or an external object storage.