feat: add expression-based log filtering and sampling#32
Merged
Conversation
Add FilterRule type with Expr, Action, and SampleRate fields plus Filters []FilterRule to SettingsConfig for YAML-based filter rules. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Parse FILTER_RULES env var in SetEnvVariables() using expr:action[:sample_rate] format separated by semicolons, populating Settings.Filters to allow runtime rule injection. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wire the filter.Chain into main.go's flush pipeline between parsing and saving. Adds nginx_clickhouse_lines_filtered_total Prometheus counter. Filter also applied during crash recovery replay. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Validate SampleRate is within [0, 1] in NewChain - Fix linesProcessed/linesNotProcessed to count post-filter entries - Document FILTER_RULES colon delimiter limitation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add Filtering & Sampling section with config, expression examples, available fields, env var usage, and validation - Add lines_filtered_total to metrics table - Add FILTER_RULES to env var table - Update CLAUDE.md: architecture, dependencies, test counts Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 13 new tests covering: fractional sampling (drop + keep), missing fields, non-numeric values, empty entries, multiple drop rules, rule ordering, unknown fields as strings, FILTER_RULES edge cases (empty, trailing semicolons, invalid entries), env var replacing YAML filters. Update README: document sample_rate on keep rules, add filters to --check output example. Update CLAUDE.md test counts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
drop(remove matches) andkeep(retain only matches) actions with optionalsample_rate(0-1) per rulestatus,request_time, etc.) so users writestatus >= 500notint(status) >= 500settings.filters) or env var (FILTER_RULES)nginx_clickhouse_lines_filtered_totalPrometheus metric-checkflagExample
Motivation
Users frequently need to reduce log volume before it reaches ClickHouse — dropping health checks, sampling high-volume 2xx traffic, filtering cached responses with zero latency. Previously this required an extra hop through Vector/Fluent Bit. This feature handles it natively with a small expression DSL.
Changes
Test plan
expr/action/sample_rate