feat: add server-side batching, -once/-stdin input modes, and optimized schema#30
Merged
Conversation
Add use_server_side_batching toggle to ClickHouseConfig. When enabled, sets async_insert=1 and wait_for_async_insert=1 on the PrepareBatch context using the idiomatic clickhouse-go/v2 WithSettings API, preserving at-least-once delivery guarantees. Logs a warning if disk buffer is also enabled since ClickHouse handles durability via async inserts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add feature bullet, env var, config example, and Reliability subsection explaining async insert behavior and interaction with disk buffer. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Verify Save() with UseServerSideBatching=true inserts rows correctly via ClickHouse async inserts (async_insert=1, wait_for_async_insert=1). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add async inserts to project overview, architecture, and config sections. Correct test counts to reflect current state across all packages. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add optimized CREATE TABLE with Delta/ZSTD codecs, LowCardinality for enrichment columns, IPv4 native type, monthly partitioning, 180-day TTL, codec rationale table, and ALTER TABLE migration from basic schema. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove basic schema and migration SQL. Keep one schema with codecs, partitioning, TTL, and rationale table. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This reverts commit 6dbb7eb.
Add -once flag to read a log file from start to EOF and exit, and -stdin flag to read log lines from standard input (supports both pipes and streaming). Both modes reuse the existing buffer/flush pipeline. Extract line source abstraction (scanLines, openLineSource) so all three modes (tail, once, stdin) feed lines into the same processing loop. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Go's flag package uses single dash by convention. Replace --check with -check for consistency with -once, -stdin, -config_path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Build the binary, run it with each flag against a real ClickHouse, and verify results. Tests cover: - -once: reads file, inserts 3 rows, exits - -stdin: reads piped input, inserts 3 rows, exits - -check: validates config, expects success - -check with bad config: expects failure Also update CI workflow and Makefile to run root integration tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rename integration_test.go to e2e_test.go with build tag "e2e". Integration tests (clickhouse/) test the Go API directly. E2e tests (root) build the binary and test flags end-to-end. Add make test-e2e target and separate CI step. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Increase bufio.Scanner max line size to 1MB (default 64KB truncates long URLs/user agents) - Guard shutdown with sync.Once to prevent race between signal handler and EOF-triggered shutdown - Use strings.Contains instead of bytes.Contains in e2e test Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
peterborodatyy
changed the title
Replace inline commands with make lint, make test, make test-integration, and make test-e2e. Align Makefile lint target with CI (fail on unformatted files instead of just listing them). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
delivery
Test plan