Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 1 addition & 9 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: test test-no-cache test-stores test-stores-no-cache test-integration-all test-integration-all-no-cache test-integration-stores test-integration-stores-no-cache test-integration-postgres test-integration-postgres-no-cache test-integration-mssql test-integration-mssql-no-cache test-integration-msql
.PHONY: test test-no-cache test-stores test-stores-no-cache test-integration-all test-integration-all-no-cache test-integration-stores test-integration-stores-no-cache test-integration-postgres test-integration-postgres-no-cache

GO ?= go
GOTOOLCHAIN ?= local
Expand Down Expand Up @@ -37,11 +37,3 @@ test-integration-postgres:

test-integration-postgres-no-cache:
$(GO_ENV) $(GO) test -count=1 -mod=$(MOD_MODE) -tags=$(INTEGRATION_TAG) -timeout=$(TEST_TIMEOUT) $(TEST_FLAGS) ./stores/postgres

test-integration-mssql:
$(GO_ENV) $(GO) test -mod=$(MOD_MODE) -tags=$(INTEGRATION_TAG) -timeout=$(TEST_TIMEOUT) $(TEST_FLAGS) ./stores/mssql

test-integration-mssql-no-cache:
$(GO_ENV) $(GO) test -count=1 -mod=$(MOD_MODE) -tags=$(INTEGRATION_TAG) -timeout=$(TEST_TIMEOUT) $(TEST_FLAGS) ./stores/mssql

test-integration-msql: test-integration-mssql
21 changes: 5 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ A lightweight Go vector-store library inspired by `Microsoft.Extensions.VectorDa
MVP scope:
- Go 1.24.x
- Postgres + `pgvector`
- MSSQL connector
- Record-based core API with optional typed codec wrapper

This library can be used to build retrieval systems such as:
Expand All @@ -22,14 +21,12 @@ This library can be used to build retrieval systems such as:

- Go 1.24.x
- PostgreSQL 16+ with `pgvector` extension
- SQL Server 2022+ (for `stores/mssql`)
- Docker (optional, for integration tests and sample compose flows)

## Project layout

- `vectordata`: backend-agnostic core interfaces, record model, filters, typed wrapper
- `stores/postgres`: Postgres implementation with `pgxpool`
- `stores/mssql`: SQL Server implementation with `database/sql`
- `samples`: runnable demos (see `samples/README.md`)
- `docs`: architecture and implementation notes

Expand Down Expand Up @@ -197,25 +194,15 @@ go test ./...
```

Notes:
- Integration tests start Postgres/pgvector and SQL Server automatically via Testcontainers
- Integration tests start Postgres/pgvector automatically via Testcontainers
- Docker daemon must be available when running integration tests
- Optional override: set `PGVECTOR_TEST_DSN` to use an existing Postgres instance instead of starting a container
- Optional override: set `MSSQL_TEST_DSN` to use an existing SQL Server instance instead of starting a container

Run MSSQL integration tests against root compose service:

```bash
docker compose up -d mssql
MSSQL_TEST_DSN="sqlserver://sa:YourStrong%21Passw0rd@localhost:14339?database=master&encrypt=disable" \
go test -tags=integration ./stores/mssql
```

## Docker Compose (optional)

`docker-compose.yml` at the repository root is kept for manual local runs.

- Use root `docker-compose.yml` when you want a persistent local Postgres+pgvector instance outside tests
- Root compose also includes SQL Server (`mssql`) for local MSSQL connector validation
- Use Testcontainers (`go test -tags=integration ./...`) for integration tests
- Sample apps have their own compose files at `samples/semantic-search/docker-compose.yml` and `samples/ragrimosa/docker-compose.yml`
- Sample Dockerfiles use `golang:1.24-alpine` to match the repo Go version (`1.24.x`)
Expand All @@ -233,8 +220,9 @@ Release options:
2. Tag-driven: push a semver tag and the workflow publishes release notes automatically:

```bash
git tag v0.2.3
git push origin v0.2.3
VERSION="$(cat VERSION)"
git tag "v${VERSION}"
git push origin "v${VERSION}"
```

## Samples
Expand All @@ -247,6 +235,7 @@ git push origin v0.2.3

- Docs index: [`docs/README.md`](docs/README.md)
- Internals and architecture: [`docs/architecture.md`](docs/architecture.md)
- Connector development guide: [`docs/connector-development.md`](docs/connector-development.md)

## License

Expand Down
1 change: 1 addition & 0 deletions VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0.2.3
10 changes: 0 additions & 10 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,3 @@ services:
interval: 2s
timeout: 2s
retries: 30

mssql:
image: mcr.microsoft.com/mssql/server:2022-latest
container_name: go-vectorstore-mssql
environment:
ACCEPT_EULA: "Y"
MSSQL_PID: "Developer"
MSSQL_SA_PASSWORD: "YourStrong!Passw0rd"
ports:
- "14339:1433"
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ This directory contains internal design and implementation notes for `go-vectors

- [`architecture.md`](architecture.md): architecture, request flow, schema behavior, filter system, and index model
- [`stores-implementation.md`](stores-implementation.md): detailed store-backend design, component responsibilities, flows, invariants, and backend differences
- [`connector-development.md`](connector-development.md): step-by-step checklist for implementing a new backend connector
28 changes: 0 additions & 28 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ The project is split into two layers:

- `vectordata`: backend-agnostic contracts and primitives
- `stores/postgres`: PostgreSQL + pgvector implementation
- `stores/mssql`: SQL Server implementation

This keeps the public API stable while allowing additional storage engines later.

Expand Down Expand Up @@ -58,11 +57,6 @@ All methods require `context.Context`.
- `EnsureExtension`: `true`
- `StrictByDefault`: `true`

`mssql.StoreOptions` defaults (`mssql.DefaultStoreOptions()`):

- `Schema`: `dbo`
- `StrictByDefault`: `true`

Collection defaults:

- Metric defaults to cosine when omitted
Expand Down Expand Up @@ -127,23 +121,6 @@ Each record is validated before sending:

The pgvector operators are documented in the [pgvector README](https://github.com/pgvector/pgvector#querying).

### MSSQL: Ensure / Search

`MSSQLVectorStore.EnsureCollection`:

1. Ensures target schema exists
2. Ensures internal collection metadata table exists
3. Creates or validates the collection table
4. Persists and validates dimension/metric metadata

`MSSQLCollection.SearchByVector`:

1. Streams records from SQL Server
2. Evaluates filters against records in-process
3. Computes distance in-process (cosine/l2/inner product)
4. Applies threshold and keeps a bounded in-memory top-k heap
5. Returns top-k sorted by distance

## 6) Filter System (AST -> SQL)

Filters are represented as an AST in `vectordata`:
Expand All @@ -170,8 +147,6 @@ Behavior details:

JSON path extraction behavior comes from PostgreSQL [JSON/JSONB functions and operators](https://www.postgresql.org/docs/current/functions-json.html).

For MSSQL in this MVP, the same AST is evaluated in-process against loaded records.

## 7) Schema Safety Modes

`CollectionSpec.Mode` controls ensure behavior:
Expand Down Expand Up @@ -264,7 +239,6 @@ Errors are wrapped with context so callers can use `errors.Is(...)` against base
Current MVP scope:

- Postgres + pgvector backend
- MSSQL backend (vectors stored as JSON payloads)
- single-vector column per collection
- metadata filtering through a focused AST

Expand All @@ -273,10 +247,8 @@ Future extension points:
- additional backends under `stores/`
- richer filter operators
- optional reranking strategies
- native MSSQL vector indexing/query pushdown

## 13) References

- pgvector project docs: https://github.com/pgvector/pgvector
- PostgreSQL docs: https://www.postgresql.org/docs/current/index.html
- SQL Server docs: https://learn.microsoft.com/sql/sql-server/
159 changes: 159 additions & 0 deletions docs/connector-development.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
# Connector Development Guide

This guide explains how to add a new backend connector under `stores/<backend>`.

## 1) Scope and Contract

Every connector must implement the shared interfaces in `vectordata`:

- `vectordata.VectorStore`
- `vectordata.Collection`

The connector must preserve shared behavior:

- validate collection specs (`name`, `dimension`, `metric`, `mode`)
- enforce vector dimension on write and search
- return `vectordata.ErrNotFound` for missing records
- return `vectordata.ErrDimensionMismatch`, `vectordata.ErrSchemaMismatch`, and `vectordata.ErrInvalidFilter` where applicable
- compute score via `vectordata.ScoreFromDistance`

## 2) Recommended Layout

Create a new directory:

- `stores/<backend>/doc.go`
- `stores/<backend>/store.go`
- `stores/<backend>/collection.go`
- `stores/<backend>/schema.go`
- `stores/<backend>/helpers.go`
- `stores/<backend>/<backend>_integration_test.go`

Add extra files only when needed (for example filter compiler/evaluator files).

## 3) Store Implementation Checklist

Implement a store type that owns connection/resources and options:

1. Add `StoreOptions` and `DefaultStoreOptions()`.
2. Add `NewVectorStore(...)` constructor with option normalization.
3. Implement `EnsureCollection(ctx, spec)`:
- normalize and validate spec
- ensure schema/table/metadata structures
- create or validate collection storage
4. Implement `Collection(name, dimension, metric)` as a lightweight handle constructor.

## 4) Collection Implementation Checklist

Implement collection operations with strict validation:

1. `Insert` and `Upsert`
- validate ID and vector dimension
- normalize nil metadata to empty object
- use batching/chunking for bulk writes
2. `Get` and `Delete`
- return `ErrNotFound` from `Get` when missing
3. `Count`
- support nil filter and filter predicates
4. `SearchByVector`
- validate `topK > 0`
- validate query vector dimension
- apply filter (SQL pushdown or in-process evaluation)
- apply optional threshold
- honor projection (`Metadata`, `Content`, `Vector`)
- return results ordered by best match first
- compute `Score` from `Distance`
5. `EnsureIndexes`
- create backend-specific indexes when supported
- return explicit error if unsupported options are requested

## 5) Filter Strategy

Choose one strategy:

- SQL pushdown: compile `vectordata.Filter` to parameterized backend SQL
- In-process: load records and evaluate filter AST in Go

Requirements:

- never interpolate raw values into SQL
- return `vectordata.ErrInvalidFilter` for invalid AST/input

## 6) Schema Safety Modes

Respect `CollectionSpec.Mode`:

- `EnsureStrict`: fail on mismatches
- `EnsureAutoMigrate`: add/fix optional schema parts where possible

When mode is unset, use connector defaults (`StrictByDefault` behavior).

## 7) Testing Requirements

Add both unit and integration coverage.

Unit tests:

- spec validation
- schema mismatch behavior
- filter behavior (happy path + invalid filters)
- dimension mismatch and error mapping
- projection and threshold behavior in search

Integration tests:

- put in `stores/<backend>/<backend>_integration_test.go`
- use `//go:build integration`
- start backend with Testcontainers when DSN env var is absent
- allow DSN override via `<BACKEND>_TEST_DSN`

Run commands:

```bash
go test ./...
go test -tags=integration ./stores/<backend>
```

## 8) Repository Wiring

When adding a connector, also update:

1. `README.md`:
- scope
- requirements
- project layout
- integration test notes
2. `docs/architecture.md` and `docs/stores-implementation.md`
3. `docs/README.md`
4. `Makefile` integration targets (if per-backend targets are used)
5. `docker-compose.yml` only if manual local backend compose is required
6. `go.mod` and `go.sum` dependencies

## 9) Minimal Skeleton

```go
package mybackend

import (
"context"

"github.com/gabisonia/go-vectorstore/vectordata"
)

type StoreOptions struct{}

func DefaultStoreOptions() StoreOptions { return StoreOptions{} }

type VectorStore struct{}

func NewVectorStore(opts StoreOptions) (*VectorStore, error) {
return &VectorStore{}, nil
}

func (s *VectorStore) EnsureCollection(ctx context.Context, spec vectordata.CollectionSpec) (vectordata.Collection, error) {
return nil, nil
}

func (s *VectorStore) Collection(name string, dimension int, metric vectordata.DistanceMetric) vectordata.Collection {
return nil
}
```
Loading