Grafana - Mimir- Query only cluster

[velu4689]

Hi Team,

I have a separate write only cluster which uses S3 as backend storage. I would like setup a query only cluster which will read/query the data from the S3 alone and not from the ingester (reason: we do not want to query any live data & just rely on Storage only data)

I would like to get your thoughts/opinions on the query only cluster being setup using the following configuration:

###################################################################
target: read,backend,alertmanager,overrides-exporter

common:
  storage:
    backend: s3
    s3:
      endpoint: s3.us-west-2.amazonaws.com
      region: us-west-2
      bucket_name: <bucket_name>

blocks_storage:
  backend: s3
  storage_prefix: blocks
  bucket_store:
    sync_interval: 15m
  tsdb:
    retention_period: 2h
    ship_interval: 5m
    dir: /data/tsdb

memberlist:
  join_members: [<EC2_IP1>, <EC2_IP2>, <EC2_IP3>]

compactor:
  data_dir: /data/compactor
  compaction_interval: 15m
  cleanup_interval: 15m
  sharding_ring:
    instance_addr: <EC2_IP1>
    kvstore:
      store: memberlist

querier:
  query_ingesters_within: 15s
  query_store_after: 0
  max_query_into_future: 12h

frontend:
 align_queries_with_step: true
 cache_results: true
 log_queries_longer_than: 10s

#distributor:
#  ring:
#    instance_addr: <EC2_IP1>
#    kvstore:
#      store: memberlist
#  ha_tracker:
#    enable_ha_tracker: true

#ingester:
#  ring:
#    instance_addr: <EC2_IP1>
#    kvstore:
#      store: memberlist
#   replication_factor: 3

alertmanager:
  data_dir: /data/alertmanager
  external_url: http://<EC2_IP1>:9009/alertmanager

ruler:
  rule_path: /data/ruler
  alertmanager_url: http://<EC2_IP1>:8080/alertmanager
  ring:
    # Quickly detect unhealthy rulers to speed up the tutorial.
    heartbeat_period: 15s
    heartbeat_timeout: 1m

ruler_storage:
  backend: s3
  storage_prefix: ruler

server:
  http_listen_address: "<EC2_IP1>"
  http_listen_port: 9009
  log_level: info

store_gateway:
  sharding_ring:
    instance_addr:<EC2_IP1>
    kvstore:
      store: memberlist
      prefix: "/data/store_collectors"
    replication_factor: 3

limits:
  max_fetched_chunks_per_query: 50000000000
  max_label_names_per_series: 200
  max_global_series_per_user: 500000000000
  ingestion_rate: 1500000000
  ingestion_burst_size: 4000000000

[pstibrany]

I would like to get your thoughts/opinions on the query only cluster being setup using the following configuration:

In general, this should work. In "query-only cluster" I would avoid running compactor, ruler and alertmanager:

• not running compactor avoids conflicts with compactor in "write-only cluster". Querier/store-gateway will rely on bucket-index being updated by compactor in "write-only cluster".
• ruler -- because rules typically query most-recent data, and write back to ingesters. It's not clear to me if this "read-only cluster" will see ingesters or not, but in any case -- with default evaluation delay being 1m, rules will return no results (because that data is only in ingesters, but those are not queried thanks to query_ingesters_within: 15s setting).

[velu4689]

Thanks @pstibrany. I tried to remove compactor, ruler, ruler storage & alertmanager. Am still facing "Validation failed. Ring Failed" when I configured the query.

[velu4689]

Yellow is the error that I get from the Grafana Dashboard for the respective data source I added. the screenshot later shows the logs am seeing from the mimir side

[pstibrany]

I think the reason why you see that error is because requests for list of label names and values will always try to use ingesters. That's because query_ingesters_within: 15s setting... these requests always use "now" as a timestamp, and therefore fall within 15 seconds window. You could try to use some tiny value like 1ns instead (that may still not work though). Unfortunately using 0 would disable this check and always use ingesters again.

[velu4689]

Got it ! Is there a reference config that I could start using for read-only cluster ?

It will be really helpful. I did try with static time period and still getting the same issue(example: May 30 12 PM - 2 PM, so that it doesn't poll for current or within 15secs window)

[pstibrany]

Is there a reference config that I could start using for read-only cluster ?

There's no reference config for "read-only cluster" that I'm aware of.

I did try with static time period and still getting the same issue

I believe Grafana will not respect selected time period when asking for label names or values (but I haven't verified this). However running the query itself will use selected time period.

[pstibrany]

Note that Grafana shows the problem as a "warning". You can still proceed and run your query. As long as selected time period doesn't fall within last 15 seconds, it should still work, despite the warning displayed by Grafana.

[velu4689]

@pstibrany : Yes, but not able to load the metrics :(