Skip to content

feat: add expression-based log filtering and sampling#32

Merged
peterborodatyy merged 8 commits into
masterfrom
feat/filter-sampling
Mar 29, 2026
Merged

feat: add expression-based log filtering and sampling#32
peterborodatyy merged 8 commits into
masterfrom
feat/filter-sampling

Conversation

@peterborodatyy

@peterborodatyy peterborodatyy commented Mar 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add configurable filter/sampling layer using expr-lang expressions, evaluated post-parse against structured log fields
  • Support drop (remove matches) and keep (retain only matches) actions with optional sample_rate (0-1) per rule
  • Auto-coerce numeric NGINX fields (status, request_time, etc.) so users write status >= 500 not int(status) >= 500
  • Rules configurable via YAML (settings.filters) or env var (FILTER_RULES)
  • Add nginx_clickhouse_lines_filtered_total Prometheus metric
  • Validate filter expressions at startup and via -check flag

Example

settings:
  filters:
    - expr: 'request contains "/health"'
      action: drop
    - expr: 'status >= 200 && status < 300'
      action: drop
      sample_rate: 0.9   # drop 90% of 2xx, keep 10%
    - expr: 'request_time == 0'
      action: drop
    - expr: 'status >= 500'
      action: keep

Motivation

Users frequently need to reduce log volume before it reaches ClickHouse — dropping health checks, sampling high-volume 2xx traffic, filtering cached responses with zero latency. Previously this required an extra hop through Vector/Fluent Bit. This feature handles it natively with a small expression DSL.

Changes

  • New package: filter/ — expr compilation, drop/keep/sampling logic, numeric field coercion
  • Modified: config/config.go — FilterRule struct, FILTER_RULES env var parsing
  • Modified: main.go — filter chain init, wired into flush(), flushLoop(), replay, --check
  • Modified: config-sample.yml — documented filter examples
  • Modified: README.md — new Filtering & Sampling section, updated metrics table, env vars, --check output
  • Modified: CLAUDE.md — architecture, deps, test counts

Test plan

  • 21 filter tests: drop, keep, sampling (deterministic + statistical), missing fields, non-numeric fallback, empty entries, multiple rules, rule ordering, regex, contains, unknown fields, invalid
    expr/action/sample_rate
  • 5 config tests: YAML parsing, env var parsing, edge cases (empty, trailing ;, no colon), env replaces YAML
  • Full suite: 145 tests across 8 packages, all pass with -race
  • gofmt, go vet clean

peterborodatyy and others added 8 commits March 29, 2026 23:17
Add FilterRule type with Expr, Action, and SampleRate fields plus
Filters []FilterRule to SettingsConfig for YAML-based filter rules.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Parse FILTER_RULES env var in SetEnvVariables() using
expr:action[:sample_rate] format separated by semicolons,
populating Settings.Filters to allow runtime rule injection.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wire the filter.Chain into main.go's flush pipeline between parsing and
saving. Adds nginx_clickhouse_lines_filtered_total Prometheus counter.
Filter also applied during crash recovery replay.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Validate SampleRate is within [0, 1] in NewChain
- Fix linesProcessed/linesNotProcessed to count post-filter entries
- Document FILTER_RULES colon delimiter limitation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add Filtering & Sampling section with config, expression examples,
  available fields, env var usage, and validation
- Add lines_filtered_total to metrics table
- Add FILTER_RULES to env var table
- Update CLAUDE.md: architecture, dependencies, test counts

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 13 new tests covering: fractional sampling (drop + keep), missing
fields, non-numeric values, empty entries, multiple drop rules, rule
ordering, unknown fields as strings, FILTER_RULES edge cases (empty,
trailing semicolons, invalid entries), env var replacing YAML filters.

Update README: document sample_rate on keep rules, add filters to
--check output example. Update CLAUDE.md test counts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@peterborodatyy peterborodatyy changed the title Feat/filter sampling feat: add expression-based log filtering and sampling Mar 29, 2026
@peterborodatyy
peterborodatyy merged commit 62e352b into master Mar 29, 2026
5 checks passed
@peterborodatyy
peterborodatyy deleted the feat/filter-sampling branch March 29, 2026 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant