Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
name: CI

on:
push:
branches:
- master
- main
pull_request:

jobs:
test:
runs-on: ubuntu-latest

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Setup Go
uses: actions/setup-go@v5
with:
go-version-file: go.mod

- name: Run unit tests
run: go test ./...
79 changes: 79 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
name: Release

on:
push:
tags:
- "v*"
workflow_dispatch:
inputs:
version:
description: "Semver version (e.g. 0.2.0 or v0.2.0)"
required: true
type: string

permissions:
contents: write

jobs:
release:
runs-on: ubuntu-latest

steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Setup Go
uses: actions/setup-go@v5
with:
go-version-file: go.mod

- name: Run unit tests
run: go test ./...

- name: Resolve release tag
id: tag
shell: bash
run: |
set -euo pipefail

if [[ "${GITHUB_EVENT_NAME}" == "workflow_dispatch" ]]; then
raw="${{ inputs.version }}"
raw="${raw#v}"
if [[ ! "$raw" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?(\+[0-9A-Za-z.-]+)?$ ]]; then
echo "Invalid version: $raw"
exit 1
fi
tag="v${raw}"
else
tag="${GITHUB_REF_NAME}"
fi

echo "tag_name=$tag" >> "$GITHUB_OUTPUT"

- name: Create git tag (manual release)
if: github.event_name == 'workflow_dispatch'
shell: bash
run: |
set -euo pipefail

tag="${{ steps.tag.outputs.tag_name }}"

if git rev-parse "$tag" >/dev/null 2>&1; then
echo "Tag $tag already exists locally"
else
git tag "$tag"
fi

if git ls-remote --tags origin "refs/tags/$tag" | grep -q "$tag"; then
echo "Tag $tag already exists on origin"
else
git push origin "$tag"
fi

- name: Publish GitHub release
uses: softprops/action-gh-release@v2
with:
tag_name: ${{ steps.tag.outputs.tag_name }}
generate_release_notes: true
34 changes: 32 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ A lightweight Go vector-store library inspired by `Microsoft.Extensions.VectorDa

MVP scope:
- Go 1.24.x
- Postgres + `pgvector` only
- Postgres + `pgvector`
- MSSQL connector
- Record-based core API with optional typed codec wrapper

This library can be used to build retrieval systems such as:
Expand All @@ -21,12 +22,14 @@ This library can be used to build retrieval systems such as:

- Go 1.24.x
- PostgreSQL 16+ with `pgvector` extension
- SQL Server 2022+ (for `stores/mssql`)
- Docker (optional, for integration tests and sample compose flows)

## Project layout

- `vectordata`: backend-agnostic core interfaces, record model, filters, typed wrapper
- `stores/postgres`: Postgres implementation with `pgxpool`
- `stores/mssql`: SQL Server implementation with `database/sql`
- `samples`: runnable demos (see `samples/README.md`)
- `docs`: architecture and implementation notes

Expand Down Expand Up @@ -193,19 +196,46 @@ go test ./...
```

Notes:
- Integration tests start Postgres + pgvector automatically via Testcontainers
- Integration tests start Postgres/pgvector and SQL Server automatically via Testcontainers
- Docker daemon must be available when running integration tests
- Optional override: set `PGVECTOR_TEST_DSN` to use an existing Postgres instance instead of starting a container
- Optional override: set `MSSQL_TEST_DSN` to use an existing SQL Server instance instead of starting a container

Run MSSQL integration tests against root compose service:

```bash
docker compose up -d mssql
MSSQL_TEST_DSN="sqlserver://sa:YourStrong%21Passw0rd@localhost:14339?database=master&encrypt=disable" \
go test -tags=integration ./stores/mssql
```

## Docker Compose (optional)

`docker-compose.yml` at the repository root is kept for manual local runs.

- Use root `docker-compose.yml` when you want a persistent local Postgres+pgvector instance outside tests
- Root compose also includes SQL Server (`mssql`) for local MSSQL connector validation
- Use Testcontainers (`go test -tags=integration ./...`) for integration tests
- Sample apps have their own compose files at `samples/semantic-search/docker-compose.yml` and `samples/ragrimosa/docker-compose.yml`
- Sample Dockerfiles use `golang:1.24-alpine` to match the repo Go version (`1.24.x`)

## Release Automation

GitHub Actions workflows are configured for:

- CI on pushes/PRs (`.github/workflows/ci.yml`)
- release publishing (`.github/workflows/release.yml`)

Release options:

1. Manual (recommended): run `Release` workflow via GitHub UI with `version` input (`0.2.0` or `v0.2.0`).
2. Tag-driven: push a semver tag and the workflow publishes release notes automatically:

```bash
git tag v0.2.0
git push origin v0.2.0
```

## Samples

- Overview and entry points: [`samples/README.md`](samples/README.md)
Expand Down
10 changes: 10 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,13 @@ services:
interval: 2s
timeout: 2s
retries: 30

mssql:
image: mcr.microsoft.com/mssql/server:2022-latest
container_name: go-vectorstore-mssql
environment:
ACCEPT_EULA: "Y"
MSSQL_PID: "Developer"
MSSQL_SA_PASSWORD: "YourStrong!Passw0rd"
ports:
- "14339:1433"
37 changes: 32 additions & 5 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ The project is split into two layers:

- `vectordata`: backend-agnostic contracts and primitives
- `stores/postgres`: PostgreSQL + pgvector implementation
- `stores/mssql`: SQL Server implementation

This keeps the public API stable while allowing additional storage engines later.

Expand All @@ -23,7 +24,7 @@ The base record type is `vectordata.Record`:
Search returns `vectordata.SearchResult`:

- `Record`: matched item
- `Distance`: raw pgvector distance
- `Distance`: backend-computed distance value
- `Score`: normalized score derived from distance

Score normalization:
Expand Down Expand Up @@ -57,6 +58,11 @@ All methods require `context.Context`.
- `EnsureExtension`: `true`
- `StrictByDefault`: `true`

`mssql.StoreOptions` defaults (`mssql.DefaultStoreOptions()`):

- `Schema`: `dbo`
- `StrictByDefault`: `true`

Collection defaults:

- Metric defaults to cosine when omitted
Expand All @@ -71,7 +77,7 @@ Other operational defaults:

## 5) Request Flow

### Ensure collection
### Postgres: Ensure collection

`PostgresVectorStore.EnsureCollection`:

Expand All @@ -90,7 +96,7 @@ metadata jsonb not null default '{}'::jsonb,
content text
```

### Insert / Upsert
### Postgres: Insert / Upsert

Bulk writes are chunked and executed as parameterized SQL:

Expand All @@ -103,7 +109,7 @@ Each record is validated before sending:
- vector dimension match
- metadata JSON serialization

### Search
### Postgres: Search

`SearchByVector` builds a query plan, then executes it:

Expand All @@ -121,6 +127,22 @@ Each record is validated before sending:

The pgvector operators are documented in the [pgvector README](https://github.com/pgvector/pgvector#querying).

### MSSQL: Ensure / Search

`MSSQLVectorStore.EnsureCollection`:

1. Ensures target schema exists
2. Ensures internal collection metadata table exists
3. Creates or validates the collection table
4. Persists and validates dimension/metric metadata

`MSSQLCollection.SearchByVector`:

1. Loads records from SQL Server
2. Evaluates filters against records in-process
3. Computes distance in-process (cosine/l2/inner product)
4. Applies threshold, sorts by distance, returns top-k

## 6) Filter System (AST -> SQL)

Filters are represented as an AST in `vectordata`:
Expand All @@ -147,6 +169,8 @@ Behavior details:

JSON path extraction behavior comes from PostgreSQL [JSON/JSONB functions and operators](https://www.postgresql.org/docs/current/functions-json.html).

For MSSQL in this MVP, the same AST is evaluated in-process against loaded records.

## 7) Schema Safety Modes

`CollectionSpec.Mode` controls ensure behavior:
Expand Down Expand Up @@ -238,7 +262,8 @@ Errors are wrapped with context so callers can use `errors.Is(...)` against base

Current MVP scope:

- Postgres + pgvector only
- Postgres + pgvector backend
- MSSQL backend (vectors stored as JSON payloads)
- single-vector column per collection
- metadata filtering through a focused AST

Expand All @@ -247,8 +272,10 @@ Future extension points:
- additional backends under `stores/`
- richer filter operators
- optional reranking strategies
- native MSSQL vector indexing/query pushdown

## 13) References

- pgvector project docs: https://github.com/pgvector/pgvector
- PostgreSQL docs: https://www.postgresql.org/docs/current/index.html
- SQL Server docs: https://learn.microsoft.com/sql/sql-server/
7 changes: 5 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,11 @@ go 1.24.0

toolchain go1.24.9

require github.com/jackc/pgx/v5 v5.7.2
require (
github.com/jackc/pgx/v5 v5.7.2
github.com/microsoft/go-mssqldb v1.8.0
github.com/testcontainers/testcontainers-go v0.33.0
)

require (
dario.cat/mergo v1.0.0 // indirect
Expand Down Expand Up @@ -45,7 +49,6 @@ require (
github.com/shirou/gopsutil/v3 v3.23.12 // indirect
github.com/shoenig/go-m1cpu v0.1.6 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/testcontainers/testcontainers-go v0.33.0 // indirect
github.com/tklauser/go-sysconf v0.3.12 // indirect
github.com/tklauser/numcpus v0.6.1 // indirect
github.com/yusufpapurcu/wmi v1.2.3 // indirect
Expand Down
6 changes: 6 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,10 @@ github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY=
github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9 h1:C/Fq8+Vm3x8+3xB3SO2/3A8Q+7WmfYI8MViMbum/v8A=
github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9/go.mod h1:9x5EikdFyA8Kj4m2Vj93QShv2N6Yf4H6J5D9Q1QzM4Y=
github.com/golang-sql/sqlexp v0.1.0 h1:DKM0nqnvWFfB6K6x2zI2Qf8F9Pwc7P0W8o2JQ7QY9MM=
github.com/golang-sql/sqlexp v0.1.0/go.mod h1:hxXw1K3Zk7K8Y6nG5N7f6d4rV1E8j9a7q6x5y4z3w2Q=
github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
Expand All @@ -57,6 +61,8 @@ github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I=
github.com/magiconair/properties v1.8.7 h1:IeQXZAiQcpL9mgcAe1Nu6cX9LLw6ExEHKjN0VQdvPDY=
github.com/magiconair/properties v1.8.7/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
github.com/microsoft/go-mssqldb v1.8.0 h1:7cyZ/AT7ycDsEoWPIXibd+aVKFtteUNhDGf3aobP+tw=
github.com/microsoft/go-mssqldb v1.8.0/go.mod h1:6znkekS3T2vp0waiMhen4GPU1BiAsrP+iXHcE7a7rFo=
github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0=
github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo=
github.com/moby/patternmatcher v0.6.0 h1:GmP9lR19aU5GqSSFko+5pRqHi+Ohk1O69aFiKkVGiPk=
Expand Down
Loading