Skip to content

Modernize library: context support, connection pooling, compression, …#2

Merged
peterborodatyy merged 8 commits into
masterfrom
modernize-library
Mar 20, 2026
Merged

Modernize library: context support, connection pooling, compression, …#2
peterborodatyy merged 8 commits into
masterfrom
modernize-library

Conversation

@peterborodatyy

@peterborodatyy peterborodatyy commented Mar 20, 2026

Copy link
Copy Markdown

Summary

Modernize the library to align with the current ClickHouse HTTP API, follow Go style guidelines, and add CI/CD.

API Improvements

  • context.Context support on all public methods (Exec, Iter, Ping, CheckCtx) for cancellation and timeouts
  • Connection pooling — reuse http.Client with pooled connections instead of creating one per request
  • Gzip compression via HTTPTransport{Compression: true}
  • HTTP status code checking — non-200 responses return structured *DBError or descriptive errors
  • Database selection via X-ClickHouse-Database header and ?database= URL param
  • Query/Session ID support (Query.QueryID, Query.SessionID) for tracking, cancellation, and session persistence
  • Per-query settings via Query.SetSetting() (e.g., max_rows_to_read, max_execution_time)
  • NewConnWithOptions constructor accepting ConnOptions{Host, User, Password, Database}

Type System

  • Uint unmarshal: uint, uint8, uint16, uint32, uint64
  • Bool unmarshal: 1/0/true/false
  • Nullable: \N handling with pointer types (*string, *int64, *float64)
  • Date format: unmarshal tries DateTime first, then Date
  • DateTime marshal: time.Time with non-zero time marshals as 2006-01-02 15:04:05

Bug Fixes

  • Multiline error parsing — (?s) regex flag for errors spanning multiple lines
  • Error regex compiled once at package level (regexp.MustCompile)

Go Style Refactoring

  • Naming: HttpTransport → HTTPTransport, DbError → DBError, prepareHttp → prepareHTTP (Go initialisms must be all-caps)
  • interface{} → any throughout (Go 1.18+)
  • Doc comments on all exported types and functions
  • doc.go with package-level documentation and usage example
  • Compile-time interface check: var _ Transport = (*HTTPTransport)(nil)
  • Error strings lowercased per Go convention
  • Extracted helpers: normalizeHost(), buildMultipartPostRequest(), buildSimplePostRequest(), unmarshalIntSlice(), unmarshalStringSlice(), unmarshalArray() to reduce function complexity
  • Eliminated unsafe intermediate type assertions in unmarshal — use direct two-value assignment
  • http.MethodGet/http.MethodPost instead of string literals
  • sync.Mutex → sync.RWMutex for Cluster read operations
  • Table-driven tests, descriptive test names, assert.NoError/assert.ErrorAs/assert.Empty
  • Proper import grouping (stdlib, then third-party)
  • Preallocated slices where size is known

Build & CI

  • Go modules: added go.mod (Go 1.21+), removed GOPATH dependency
  • Removed .travis.yml (targeted Go 1.3–1.9)
  • Replaced deprecated io/ioutil → io, strings.Replace → strings.ReplaceAll, strings.Index → strings.HasPrefix
  • GitHub Actions — Unit Tests: runs on Go 1.21/1.22/1.23 with race detector and coverage
  • GitHub Actions — Integration Tests: spins up clickhouse/clickhouse-server:latest and runs 26 integration tests
  • GitHub Actions — Release: auto-creates a GitHub release with version bump on push to master

Integration Tests (26 tests against real ClickHouse)

  • Ping, context cancellation, timeouts
  • SELECT with all types (int, uint, float, string, bool, DateTime, Date, Nullable, arrays)
  • CREATE TABLE → INSERT → SELECT round-trip
  • Placeholder substitution (int and string)
  • Database selection via ConnOptions
  • Query ID tracking (verified via system.query_log)
  • Session ID (SET + getSetting across same session)
  • Per-query settings enforcement (max_rows_to_read)
  • Gzip compression
  • External data tables
  • Cluster health checks
  • Special character escaping
  • Error handling (bad SQL, nonexistent tables)

Docs

  • README updated with all new features, correct API names, current examples
  • CONTRIBUTING.md with setup, testing, code style, and PR guidelines

Test plan

  • 59 unit tests pass with -race flag
  • 26 integration tests pass against real ClickHouse
  • go vet clean
  • gofmt clean
  • Verify GitHub Actions workflows trigger on merge
  • Smoke test against a production ClickHouse instance

peterborodatyy and others added 8 commits March 20, 2026 19:12
…and more

- Add context.Context to all public methods (Exec, Iter, Ping, CheckCtx)
- Reuse http.Client with connection pooling instead of creating per-request
- Add gzip compression support (HttpTransport.Compression)
- Check HTTP status codes and return structured errors
- Add database selection via header and URL param
- Add query_id, session_id, and per-query settings support
- Add NewConnWithOptions constructor
- Add uint, bool, Nullable, and Date unmarshal support
- Fix multiline error parsing with (?s) regex flag
- Compile error regex once at package level
- Replace deprecated io/ioutil, strings.Replace, strings.Index
- Upgrade Cluster from sync.Mutex to sync.RWMutex
- Add go.mod (Go 1.21+) with testify dependency
- Replace Travis CI with GitHub Actions (test + release workflows)
- Expand tests from 30 to 58

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
26 integration tests running against clickhouse/clickhouse-server:
- Ping, context cancellation, and timeouts
- SELECT with all supported types (int, uint, float, string, bool, DateTime, Date)
- Nullable type support (\N handling)
- Array types (int, string)
- CREATE TABLE, INSERT, SELECT round-trip
- BuildInsert and BuildMultiInsert
- Placeholder substitution (int and string)
- Database selection via ConnOptions
- Query ID tracking (verified via system.query_log)
- Session ID (SET + getSetting across same session)
- Per-query settings (max_rows_to_read enforcement)
- Gzip compression
- External data tables
- Cluster health checks
- Special character escaping
- Error handling (bad SQL, nonexistent tables)

Tests use //go:build integration tag so they only run when
explicitly requested. GitHub Actions runs them in a separate
job with a ClickHouse service container.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace Travis/roistat badges with GitHub Actions/mintance
- Update all examples to use context.Context
- Add sections: auth, database, query ID, session ID, settings, compression, context timeout
- Update ClickHouse branding and links
- Document NewConnWithOptions, SetSetting, CheckCtx

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Naming (Go initialisms must be all caps):
- HttpTransport → HTTPTransport
- NewHttpTransport → NewHTTPTransport
- DbError → DBError
- prepareHttp → prepareHTTP

Documentation:
- Add doc.go with package-level documentation and usage example
- Add doc comments to all exported types and functions
- Add compile-time interface check: var _ Transport = (*HTTPTransport)(nil)

Error strings (lowercase, no punctuation per Go convention):
- "Connection pointer is nil" → "connection pointer is nil"
- "Clickhouse host response was..." → "clickhouse: unexpected ping response..."
- "Amount of row items..." → "row item count..."

Code style:
- interface{} → any (Go 1.18+)
- Extract normalizeHost() to deduplicate host normalization
- Extract buildMultipartPostRequest/buildSimplePostRequest from monolithic function
- Extract unmarshalIntSlice/unmarshalStringSlice/unmarshalArray from giant switch
- Eliminate intermediate variable m in unmarshal (use two-value form of type assertion)
- Iter receiver r → it (first letter of type name)
- Remove unnecessary named returns
- Use http.MethodGet/http.MethodPost instead of string literals
- Use assert.NoError/require.NoError instead of assert.Equal(t, nil, err)
- Use assert.ErrorAs instead of type assertion for error checks
- Use assert.Empty instead of assert.Equal(t, "", ...)
- Rename numbered tests to descriptive names (Ping2 → PingBadResponse, etc.)
- Convert TestConnect to table-driven test
- Lowercase test handler types (not exported)
- Group imports: stdlib, then third-party
- Preallocate args slice in BuildMultiInsert with append
- Use range index in benchmarks

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add Quick Start section with full working example including Ping
- Add GoDoc badge
- Add Nullable types section with pointer scanning example
- Add Error handling section showing DBError and HTTP error patterns
- Add Supported types table with marshal/unmarshal matrix
- Add Connection pooling section documenting defaults
- Expand Cluster section with IsDown and ActiveConn nil check
- Add context to all examples that were missing it
- Add ClickHouse settings docs link
- Show proper error handling (if err != nil) instead of if err == nil
- Expand Features list with specific type ranges and HTTP status checking

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@peterborodatyy
peterborodatyy merged commit d180e6e into master Mar 20, 2026
5 checks passed
@peterborodatyy
peterborodatyy deleted the modernize-library branch March 20, 2026 23:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant