Ops Agent 2.50.0+: Excessive disk reads due to file_cache_advise default

[marek-knappe]

Problem

Starting with Ops Agent 2.50.0, fluent-bit's tail input plugin causes 8-9x read amplification on disk due to the file_cache_advise feature being enabled by default.

Impact

• Working: Ops Agent 2.49.0 (fluent-bit v3.0.4) → ~0 KB/s disk reads
• Broken: Ops Agent 2.50.0+ (fluent-bit v3.1.6) → 4-5 MB/s sustained disk reads
• Disk utilization jumps from 0.3% to 84-90%

Root Cause

Fluent-bit commit 8e0941e introduced file_cache_advise (defaults to true) which calls posix_fadvise(POSIX_FADV_DONTNEED) after reading, evicting pages from kernel cache.

With multiple pipelines reading the same files + WAL buffering, data is read multiple times. Cache eviction forces every re-read to hit disk instead of being served from RAM.

Reproduction

1. Deploy Ops Agent 2.50.0 on a VM with file-based logging receivers
2. Configure 2+ pipelines reading the same log files (common pattern)
3. Observe sustained high disk reads even when fluent-bit is caught up

Tested on:

• GCP VMs running Ubuntu 22.04
• /data/logs/*/output.log files rotating every 2-3 hours (2 GB each)
• Two pipelines: default_pipeline and http_request_pipeline

Proposed Solution

Expose file_cache_advise as a user-configurable option in the Ops Agent YAML config, defaulting to false to match pre-2.50.0 behavior.

logging:
  receivers:
    my-logs:
      type: files
      include_paths:
        - /var/log/*.log
      file_cache_advise: false  # New option

Workaround

Pin to Ops Agent 2.49.0:

sudo apt-mark hold google-cloud-ops-agent
sudo apt-get install google-cloud-ops-agent=2.49.0~ubuntu22.04

One server upgraded to 2.50.0 one downgraded to 2.49.0