Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,8 @@ upkeep and don't absorb the rest of your company's knowledge.
- You only need one ad-hoc query - `psql` or a notebook will do

Works with PostgreSQL, Snowflake, BigQuery, ClickHouse, MySQL, SQL Server,
SQLite, DuckDB, Amazon Athena, and MongoDB. Integrates with dbt, MetricFlow,
LookML, Looker, Metabase, Sigma, Notion, and Google Drive.
SQLite, DuckDB, Databricks, Amazon Athena, and MongoDB. Integrates with dbt,
MetricFlow, LookML, Looker, Metabase, Sigma, Notion, and Google Drive.

## Quick Start

Expand Down
10 changes: 5 additions & 5 deletions docs-site/content/docs/cli-reference/ktx-setup.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ runtime features are missing.

| Flag | Description |
|------|-------------|
| `--database <driver>` | Database driver to configure; repeatable. Choices: `sqlite`, `duckdb`, `postgres`, `mysql`, `clickhouse`, `sqlserver`, `bigquery`, `snowflake` |
| `--database <driver>` | Database driver to configure; repeatable. Choices: `sqlite`, `duckdb`, `postgres`, `mysql`, `clickhouse`, `sqlserver`, `bigquery`, `snowflake`, `databricks`, `athena` |
| `--database-connection-id <id>` | Existing selected connection id; repeatable. With `--database` or `--database-url`, connection id for the new connection. |
| `--database-url <url>` | URL, `env:NAME`, or `file:/path` for one new URL-style database connection; also used as the SQLite or DuckDB path |
| `--database-schema <schema>` | Database schema or dataset to include; repeatable |
Expand All @@ -131,8 +131,8 @@ context. Use `--skip-databases` only when intentionally leaving the project
incomplete.

`--database-schema` maps to the driver's scope field: `schemas` for PostgreSQL,
MySQL, and SQL Server; `schema_names` for Snowflake; `dataset_ids` for
BigQuery; and `databases` for ClickHouse.
MySQL, and SQL Server; `schema_names` for Snowflake and Databricks;
`dataset_ids` for BigQuery; and `databases` for ClickHouse.

A BigQuery `--database-schema` value may be qualified as `project.dataset` to
scan a dataset hosted in another project (such as
Expand All @@ -142,8 +142,8 @@ entries explicitly. See
[Primary sources → BigQuery](/docs/integrations/primary-sources#cross-project-datasets).

With `--no-input`, scope for a scope-bearing driver (PostgreSQL, MySQL,
ClickHouse, SQL Server, BigQuery, Snowflake) must come from `--database-schema`
or from existing connection config in `ktx.yaml` (for example
ClickHouse, SQL Server, BigQuery, Snowflake, Databricks) must come from
`--database-schema` or from existing connection config in `ktx.yaml` (for example
`connections.<id>.dataset_ids`). When neither is set, the database step fails
fast and prints the missing scope flag and config key — non-interactive setup
never auto-discovers and scans every schema. SQLite has no scope and is
Expand Down
10 changes: 9 additions & 1 deletion docs-site/content/docs/configuration/ktx-yaml.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,7 @@ context-source drivers share the map.
| `sqlserver` | Warehouse | `driver` | `url`, `enabled_tables` |
| `bigquery` | Warehouse | `driver` | `credentials_json`, `dataset_ids`, `enabled_tables`, `historicSql` |
| `snowflake` | Warehouse | `driver` | `schema_names`, `enabled_tables`, `historicSql` |
| `databricks` | Warehouse | `driver`, `server_hostname`, `http_path`, `catalog` | `schema_names`, `enabled_tables` |
| `clickhouse` | Warehouse | `driver` | `url`, `database`, `databases`, `enabled_tables` |
| `metabase` | Context source | `driver`, `api_url` | `api_key_ref`, `mappings` |
| `looker` | Context source | `driver`, `base_url`, `client_id` | `client_secret_ref`, `mappings` |
Expand Down Expand Up @@ -172,6 +173,12 @@ connections:
credentials_json: file:./service-account.json
location: US
dataset_ids: [analytics, mart]
databricks-warehouse:
driver: databricks
server_hostname: dbc-example.cloud.databricks.com
http_path: /sql/1.0/warehouses/abc123
catalog: main
schema_names: [analytics, mart]
```

A BigQuery `dataset_ids` / `dataset_id` entry may be written `project.dataset`
Expand All @@ -187,7 +194,8 @@ defaults to `4`. This caps all concurrent SQL work for that connector instance,
including schema introspection, table sampling, relationship profiling,
relationship validation, and read-only SQL execution. BigQuery and ClickHouse
do not expose `maxConnections` because their connectors don't use client-side
connection pools.
connection pools. Databricks opens short-lived SQL sessions per operation and
also does not expose `maxConnections`.

For Postgres, BigQuery, and Snowflake, `historicSql` and `context.queryHistory`
toggle query-history ingest. The shape is connector-specific; the setup wizard
Expand Down
2 changes: 1 addition & 1 deletion docs-site/content/docs/getting-started/quickstart.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,7 @@ The wizard walks you through everything **ktx** needs in one pass:
embeddings or `sentence-transformers` to run locally without an API key.
4. **Database** - adds at least one primary connection. Supported drivers:
PostgreSQL, Snowflake, BigQuery, MySQL, ClickHouse, SQL Server, SQLite, and
DuckDB.
DuckDB, Databricks, and Amazon Athena.
5. **Context sources** - optionally adds dbt, MetricFlow, LookML, Looker,
Metabase, or Notion. You can skip and add them later.
6. **Build** - offers to run the first ingest so semantic sources and wiki
Expand Down
59 changes: 57 additions & 2 deletions docs-site/content/docs/integrations/primary-sources.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Primary Sources
description: Connect ktx to PostgreSQL, Snowflake, BigQuery, MySQL, ClickHouse, SQL Server, SQLite, DuckDB, MongoDB, or Amazon Athena.
description: Connect ktx to PostgreSQL, Snowflake, BigQuery, Databricks, MySQL, ClickHouse, SQL Server, SQLite, DuckDB, MongoDB, or Amazon Athena.
---

**ktx** connects to your data warehouse or database to build schema context,
Expand All @@ -26,7 +26,7 @@ Agents should prefer environment or file references over literal secrets.

| Field | Required | Applies to | Description |
|-------|----------|------------|-------------|
| `driver` | Yes | all connections | Connector driver such as `postgres`, `snowflake`, `bigquery`, `mysql`, `clickhouse`, `sqlserver`, `sqlite`, `duckdb`, `mongodb`, or `athena` |
| `driver` | Yes | all connections | Connector driver such as `postgres`, `snowflake`, `bigquery`, `databricks`, `mysql`, `clickhouse`, `sqlserver`, `sqlite`, `duckdb`, `mongodb`, or `athena` |
| `url` | One of the connection methods | URL-style connectors | Database URL, `env:NAME`, or `file:/path/to/secret` |
| `host`, `port`, `database`, `username`, `password` | One of the connection methods | PostgreSQL, MySQL, SQL Server | Field-by-field connection values |
| `schema` or `schemas` | No | schema-aware warehouses | Single schema or list of schemas to scan |
Expand Down Expand Up @@ -198,6 +198,61 @@ artifact shape as Postgres and BigQuery.

---

## Databricks

Connects via the Databricks SQL Driver for Node.js. Supports Unity Catalog
three-level namespaces, schema introspection, table sampling, column sampling,
and read-only SQL execution.

### Connection config

```yaml title="ktx.yaml"
connections:
my-databricks:
driver: databricks
server_hostname: dbc-example.cloud.databricks.com
http_path: /sql/1.0/warehouses/abc123
catalog: main
schema_names:
- sales
- finance
authMethod: pat
token: env:DATABRICKS_TOKEN
```

`ktx setup` discovers schemas after the connection is verified and writes the
selected list to `schema_names`. For hand-written config, prefer
`schema_names` even for a single schema. `schema_name: sales` is accepted as a
single-schema shorthand only when `schema_names` is not set.

### Authentication

| Method | Config |
|--------|--------|
| Personal Access Token | `authMethod: pat`, `token: env:DATABRICKS_TOKEN` |
| OAuth machine-to-machine | `authMethod: oauth-m2m`, `client_id`, `client_secret: env:DATABRICKS_CLIENT_SECRET` |

### Features

| Feature | Supported | Notes |
|---------|-----------|-------|
| Tables & views | Yes | Via Unity Catalog `INFORMATION_SCHEMA.TABLES` |
| Primary keys | Yes | Via table constraints when available to the service principal |
| Foreign keys | No | Not emitted by the connector |
| Row count estimates | No | - |
| Column statistics | No | - |
| Query history | No | Databricks query-history ingest is not implemented |
| Table sampling | Yes | Uses read-only `SELECT ... LIMIT` sampling |

### Dialect notes

- Unity Catalog table names use `catalog.schema.table`
- Identifiers are quoted with backticks
- Parameter binding uses positional `?` placeholders
- Read-only SQL is validated before execution

---

## BigQuery

Authenticates via GCP service account credentials. Supports multi-dataset scanning and query-history configuration for `INFORMATION_SCHEMA.JOBS_BY_PROJECT`.
Expand Down
1 change: 1 addition & 0 deletions packages/cli/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@
"@clack/prompts": "1.4.0",
"@clickhouse/client": "^1.18.5",
"@commander-js/extra-typings": "14.0.0",
"@databricks/sql": "^1.16.0",
"@duckdb/node-api": "1.5.3-r.3",
"@google-cloud/bigquery": "^8.3.1",
"google-auth-library": "10.6.2",
Expand Down
3 changes: 2 additions & 1 deletion packages/cli/src/commands/setup-commands.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,8 @@ function databaseDriver(value: string): KtxSetupDatabaseDriver {
value === 'clickhouse' ||
value === 'sqlserver' ||
value === 'bigquery' ||
value === 'snowflake'
value === 'snowflake' ||
value === 'databricks'
) {
return value;
}
Expand Down
1 change: 1 addition & 0 deletions packages/cli/src/connection-drivers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ export const KTX_DATABASE_DRIVER_IDS = [
'sqlserver',
'bigquery',
'snowflake',
'databricks',
'athena',
] as const;

Expand Down
Loading
Loading