Skip to content

Bug: MimirStrongConsistencyOffsetMissing alert raised in ingest-storage mode #16122

Description

@antonincms

What is the bug?

On my mimir cluster deployed with ingest storage, the MimirStrongConsistencyOffsetMissing alert is always raised.

Maybe it is because I encountered a bug or I misconfigured my cluster, but I wonder if it is because strong consistency is disabled by default while the alert is enabled when ingest storage is used:

# (experimental) The default consistency level to enforce for queries when using
# the ingest storage. Supports values: strong, eventual.
# CLI flag: -ingest-storage.read-consistency
[ingest_storage_read_consistency: <string> | default = "eventual"]

How to reproduce it?

Install mimir in ingest storage mode, value example:

Details

mimir:
  structuredConfig:
    alertmanager:
      sharding_ring:
        instance_enable_ipv6: true
    alertmanager_storage:
      s3:
        bucket_name: <bucket-name>
    blocks_storage:
      s3:
        bucket_name: <bucket-name>
    common:
      storage:
        backend: s3
        s3:
          endpoint: s3.<region>.amazonaws.com
          region: <region>
    compactor:
      sharding_ring:
        instance_enable_ipv6: true
    distributor:
      ring:
        instance_enable_ipv6: true
    frontend:
      instance_enable_ipv6: true
    ingest_storage:
      kafka:
        address: <kafka-address>
    ingester:
      ring:
        instance_enable_ipv6: true
    limits:
      align_queries_with_step: true
      cardinality_analysis_enabled: true
      compactor_blocks_retention_period: 30d
      out_of_order_time_window: 30m
    memberlist:
      bind_addr:
        - '::'
    querier:
      ring:
        instance_enable_ipv6: true
    query_scheduler:
      ring:
        instance_enable_ipv6: true
    ruler:
      ring:
        instance_enable_ipv6: true
    ruler_storage:
      s3:
        bucket_name: <bucket-name>
    store_gateway:
      sharding_ring:
        instance_enable_ipv6: true
alertmanager:
  persistentVolume:
    enabled: true
  replicas: 3
  statefulSet:
    enabled: true
distributor:
  replicas: 2
gateway:
  enabledNonEnterprise: true
  replicas: 2
ingester:
  replicas: 4
  zoneAwareReplication:
    enabled: true
    topologyKey: kubernetes.io/hostname
    zones:
      - name: zone-a
        prepareDownscale: true
      - name: zone-b
        prepareDownscale: true
overrides_exporter:
  replicas: 1
querier:
  replicas: 2
query_frontend:
  replicas: 2
query_scheduler:
  replicas: 2
rollout_operator:
  enabled: true
ruler:
  replicas: 2
serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: <role-arn>
store_gateway:
  replicas: 3
  zoneAwareReplication:
    enabled: false
kafka:
  enabled: false
minio:
  enabled: false
nginx:
  enabled: false

What did you think would happen?

The alert would not be raised with the default configuration.

What was your environment?

Mimir 3.1.2 (deployed with corresponding mixins) deployed with the chart 6.1.0
Kafka 4.3.0
Kubernetes 1.35.

Any additional context to share?

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions