Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs-site/content/docs/cli-reference/ktx-setup.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,11 @@ prompts.
| `--vertex-location <location>` | Vertex AI location, `env:NAME`, or `file:/path` reference |
| `--skip-llm` | Leave LLM setup incomplete |

Non-interactive setup (`--no-input`) requires an explicit `--llm-backend`; it
does not assume a default. `--llm-backend` is independent of `--target`: the
target selects which agent client receives integration, while `--llm-backend`
selects the provider **ktx** itself uses.

Choose only one Anthropic credential source. Anthropic credential flags are only
valid with the Anthropic backend; Vertex flags are only valid with the Vertex
backend. The `claude-code` and `codex` backends use local authentication instead
Expand Down Expand Up @@ -276,6 +281,7 @@ Use `ktx status` for repeatable readiness checks after setup exits.
|-------|-------|----------|
| Setup resumes an unexpected project | `KTX_PROJECT_DIR` or nearest `ktx.yaml` points to another directory | Pass `--project-dir <path>` explicitly |
| Setup cannot run in CI | Required values are missing and `--no-input` disables prompts | Provide the relevant automation flags or create a fixture `ktx.yaml` |
| `Missing LLM backend` | `--no-input` without `--llm-backend` | Pass `--llm-backend` with `anthropic`, `vertex`, `claude-code`, or `codex` (the latter two need no API key) |
| Provider health check fails | Provider key, model id, Vertex project, or Vertex location is invalid | Fix the `env:` or `file:` reference and rerun setup |
| Python runtime is missing | The selected setup needs runtime-backed agent, query-history, Looker, or local embedding features | Accept the interactive prompt, rerun with `--yes`, or run the suggested `ktx admin runtime install` command |
| `--enable-query-history` is rejected | The selected database driver does not support query history | Use Postgres, BigQuery, or Snowflake, or rerun without query-history flags |
Expand Down
4 changes: 4 additions & 0 deletions docs-site/content/docs/cli-reference/ktx-sql.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ Use `ktx sql` with a required connection id and positional SQL text.
ktx sql --connection <id> [options] <sql...>
```

`ktx sql` runs against any configured connection, whether or not it is a scan or
ingest target. Connections marked `scan_enabled: false` (execute-only) work here
too — see [execute-only connections](/docs/configuration/ktx-yaml#execute-only-connections).

## Options

Use output flags to choose between terminal display, TSV rows, and structured
Expand Down
29 changes: 27 additions & 2 deletions docs-site/content/docs/configuration/ktx-yaml.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,31 @@ connections:
dataset_ids: [analytics, mart]
```

#### Execute-only connections

Set `scan_enabled: false` to register a warehouse for SQL execution only. The
connection is usable by `ktx sql` and the agent `sql_execution` tool, but **ktx**
never introspects, scans, or ingests it: automatic ingest skips it, `ktx setup`
validates the credential without discovering or scanning its schemas, and even an
explicit `ktx scan <id>` or `ktx ingest <id>` is refused with guidance. This is
the supported way to run read-only queries against shared or public data (for
example a BigQuery billing project full of unrelated datasets) without making it
a context source.

```yaml
connections:
public_bq:
driver: bigquery
credentials_json: file:./service-account.json
scan_enabled: false
```

Without `scan_enabled`, a warehouse is a scan target. In scripted setup
(`--no-input`) with no `--database-schema` and no `dataset_ids`/`schemas`, **ktx**
scopes the scan to every schema or dataset the credential can see and prints a
warning naming the count; pass `--database-schema` to narrow it, or
`scan_enabled: false` to register it for execution only.

For Postgres, MySQL, SQL Server, and Snowflake connections, set
`maxConnections` when scan or ingest work needs to stay below the target's
connection cap. Postgres, MySQL, and SQL Server default to `10`; Snowflake
Expand Down Expand Up @@ -360,7 +385,7 @@ storage:
|-------|------|---------|---------|
| `state` | `sqlite` \| `postgres` | `sqlite` | Backend for ktx state. `sqlite` uses `.ktx/db.sqlite`; `postgres` expects a configured Postgres connection. |
| `search` | `sqlite-fts5` \| `postgres-hybrid` | `sqlite-fts5` | Backend for search indexes. `postgres-hybrid` combines lexical and vector search in Postgres. |
| `git.auto_commit` | `boolean` | `true` | When `true`, ktx auto-commits changes to the git-backed state store. |
| `git.auto_commit` | `boolean` | `true` | When `true`, a context-source ingest run commits its changes to the git-backed state store. When `false`, the changes are applied to the working tree and left staged for you to commit. |
| `git.author` | `string` | `ktx <ktx@example.com>` | Git author identity for auto-commits. Standard `Name <email>` form. |

## `llm`
Expand Down Expand Up @@ -619,7 +644,7 @@ memory:

| Field | Type | Default | Purpose |
|-------|------|---------|---------|
| `auto_commit` | `boolean` | `true` | When `true`, ktx auto-commits memory updates to the git-backed store. |
| `auto_commit` | `boolean` | `true` | When `true`, a memory/wiki ingest run commits its updates to the git-backed store. When `false`, the updates are applied to the working tree and left staged for you to commit. |

## A full example

Expand Down
8 changes: 8 additions & 0 deletions docs-site/content/docs/guides/reviewing-context.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,14 @@ and replay; it belongs in `.gitignore`. If your team wants a record of *why* a
change happened, link the transcript path in the PR description rather than
committing the file.

**ktx** maintains its own git repository at the project directory. When the
project lives inside an existing repository (for example a `data/ktx`
subdirectory of your application repo), **ktx** initializes a dedicated
repository at the project directory rather than committing into the enclosing
one — its ingest commits stay isolated, and writes and reindexing always share
the same working-tree root. Track the project directory in your outer repo as a
nested checkout (or keep it separate) depending on how you want to review it.

## A typical review session

The loop above describes the shape. In practice, one review session looks like
Expand Down
26 changes: 10 additions & 16 deletions packages/cli/src/commands/setup-commands.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,20 +21,6 @@ function positiveInteger(value: string): number {
return parsed;
}

function embeddingBackend(value: string): 'openai' | 'sentence-transformers' {
if (value === 'openai' || value === 'sentence-transformers') {
return value;
}
throw new InvalidArgumentError(`invalid choice '${value}'`);
}

function llmBackend(value: string): KtxSetupLlmBackend {
if (value === 'anthropic' || value === 'vertex' || value === 'claude-code' || value === 'codex') {
return value;
}
throw new InvalidArgumentError(`invalid choice '${value}'`);
}

function databaseDriver(value: string): KtxSetupDatabaseDriver {
if (
value === 'sqlite' ||
Expand Down Expand Up @@ -220,7 +206,11 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo
.addOption(new Option('--skip-agents', 'Leave agent integration incomplete for now').hideHelp().default(false))
.option('--yes', 'Accept project creation and runtime install defaults where setup confirms', false)
.option('--no-input', 'Disable interactive terminal input')
.addOption(new Option('--llm-backend <backend>', 'LLM backend').argParser(llmBackend).hideHelp())
.addOption(
new Option('--llm-backend <backend>', 'LLM backend')
.choices(['anthropic', 'vertex', 'claude-code', 'codex'])
.hideHelp(),
)
.addOption(
new Option('--anthropic-api-key-env <name>', 'Environment variable containing the Anthropic API key').hideHelp(),
)
Expand All @@ -230,7 +220,11 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo
.addOption(new Option('--vertex-project <project>', 'Google Vertex AI project ID, env:NAME, or file:/path').hideHelp())
.addOption(new Option('--vertex-location <location>', 'Google Vertex AI location, env:NAME, or file:/path').hideHelp())
.addOption(new Option('--skip-llm', 'Leave LLM setup incomplete for now').hideHelp().default(false))
.addOption(new Option('--embedding-backend <backend>', 'Embedding backend').argParser(embeddingBackend).hideHelp())
.addOption(
new Option('--embedding-backend <backend>', 'Embedding backend')
.choices(['openai', 'sentence-transformers'])
.hideHelp(),
)
.addOption(
new Option(
'--embedding-api-key-env <name>',
Expand Down
18 changes: 18 additions & 0 deletions packages/cli/src/context/connections/local-warehouse-descriptor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,24 @@ export function localConnectionToWarehouseDescriptor(
return info;
}

/**
* True when the connection is registered for SQL execution only (`scan_enabled: false`) and
* must never be used as a scan/ingest target. Execution paths (`ktx sql`, `sql_execution`) are
* unaffected — they resolve the warehouse via {@link localConnectionToWarehouseDescriptor}.
*/
export function isExecuteOnlyConnection(connection: KtxProjectConnectionConfig | undefined): boolean {
return (connection as { scan_enabled?: boolean } | undefined)?.scan_enabled === false;
}

/**
* True when the connection is a warehouse AND eligible to be scanned/ingested. This is the single
* predicate every scan-target selection path routes through, so execute-only connections are
* excluded consistently — including the "fall back to all warehouses" path.
*/
export function isScanTargetWarehouse(id: string, connection: KtxProjectConnectionConfig | undefined): boolean {
return localConnectionToWarehouseDescriptor(id, connection) !== null && !isExecuteOnlyConnection(connection);
}

export function localConnectionTypeForConfig(id: string, connection: KtxProjectConnectionConfig | undefined): string {
const descriptor = localConnectionToWarehouseDescriptor(id, connection);
if (descriptor) {
Expand Down
117 changes: 105 additions & 12 deletions packages/cli/src/context/core/git.service.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { promises as fs } from 'node:fs';
import { dirname, join } from 'node:path';
import type { SimpleGit } from 'simple-git';
import { CheckRepoActions, type SimpleGit } from 'simple-git';
import { noopLogger, resolveConfigDir, type KtxCoreConfig, type KtxLogger } from './config.js';
import { createSimpleGit } from './git-env.js';

Expand All @@ -27,9 +27,21 @@ export interface WorktreeEntry {
head: string | null;
}

/**
* `dirty` means the target (main) working tree had uncommitted tracked changes before the
* merge — typically residue from a prior `auto_commit: false` run that was never committed.
* Squashing into it would either commit those unrelated changes or, on conflict cleanup,
* `reset --hard` them away, so the merge is refused and the tree is left untouched.
*/
export type SquashMergeResult =
| { ok: true; squashSha: string; touchedPaths: string[] }
| { ok: false; conflict: true; conflictPaths: string[] };
| { ok: false; conflict: true; conflictPaths: string[] }
| { ok: false; dirty: true; dirtyPaths: string[] };

export type StageSquashResult =
| { ok: true; touchedPaths: string[]; stagedTree: string }
| { ok: false; conflict: true; conflictPaths: string[] }
| { ok: false; dirty: true; dirtyPaths: string[] };

function mergeErrorMessage(error: unknown): string {
if (error instanceof Error) {
Expand Down Expand Up @@ -94,15 +106,29 @@ export class GitService {

private async initialize(): Promise<void> {
try {
// Check if already initialized
const isRepo = await this.git.checkIsRepo();

if (!isRepo) {
// The ktx store assumes configDir is its own git working-tree root: writes, session
// worktrees, squash-merges, and reindex scans are all resolved relative to it. The
// default checkIsRepo() (IN_TREE) is also satisfied by an *enclosing* repository, so a
// project nested inside another git working tree would silently operate against that
// outer repo — ingest-written files land at the outer root while reindex scans configDir,
// leaving e.g. the wiki seeded but unindexed. Require configDir to be the repo root
// itself; otherwise initialize a dedicated repository here.
const isOwnRepoRoot = await this.git.checkIsRepo(CheckRepoActions.IS_REPO_ROOT);

if (!isOwnRepoRoot) {
const insideEnclosingRepo = await this.git.checkIsRepo(CheckRepoActions.IN_TREE);
await this.git.init();
const gitConfig = this.config.git;
await this.git.addConfig('user.name', gitConfig.userName);
await this.git.addConfig('user.email', gitConfig.userEmail);
this.logger.log('Initialized git repository');
if (insideEnclosingRepo) {
this.logger.log(
`Initialized a dedicated ktx git repository at ${this.configDir} (nested inside an existing ` +
'git repository); ktx commits stay in this repository and do not touch the outer one.',
);
} else {
this.logger.log('Initialized git repository');
}
}

// Keep any auto-maintenance triggered by writes in-process. Detached maintenance can
Expand Down Expand Up @@ -667,6 +693,76 @@ export class GitService {
authorEmail: string,
commitMessage: string,
): Promise<SquashMergeResult> {
const applied = await this.applySquashToIndex(branch);
if (!applied.ok) {
return applied;
}
if (applied.touchedPaths.length === 0) {
const head = (await this.git.revparse(['HEAD'])).trim();
return { ok: true, squashSha: head, touchedPaths: [] };
}

await this.git.commit(commitMessage, { '--author': `${author} <${authorEmail}>` });
const squashSha = (await this.git.revparse(['HEAD'])).trim();
return { ok: true, squashSha, touchedPaths: applied.touchedPaths };
}

/**
* Like {@link squashMergeIntoMain} but stops before committing: applies `branch` onto the
* current branch's index + working tree and leaves the result staged for the user to commit.
* Returns the staged tree's SHA, which is a valid diff/read ref (`git diff A..<tree>`,
* `git show <tree>:path`) so callers can reconcile derived indexes without a commit.
*
* This backs the `auto_commit: false` ingest path: changes still reach the working tree (so
* the run is not silently discarded), they are just not committed automatically.
*
* Caller must hold the `config:repo` lock, as with {@link squashMergeIntoMain}.
*/
async stageSquashMergeIntoMain(branch: string): Promise<StageSquashResult> {
return this.withMutationQueue(() => this.stageSquashMergeIntoMainUnlocked(branch));
}

private async stageSquashMergeIntoMainUnlocked(branch: string): Promise<StageSquashResult> {
const applied = await this.applySquashToIndex(branch);
if (!applied.ok) {
return applied;
}
const stagedTree = (await this.git.raw(['write-tree'])).trim();
return { ok: true, touchedPaths: applied.touchedPaths, stagedTree };
}

/**
* Shared core of the squash-merge paths: applies `branch` onto the current branch's index +
* working tree via `git merge --squash` WITHOUT committing, leaving the caller to either
* commit (auto-commit on) or stage (auto-commit off). Returns the touched paths, or conflict
* info after restoring a clean tree.
*/
private async applySquashToIndex(
branch: string,
): Promise<
| { ok: true; touchedPaths: string[] }
| { ok: false; conflict: true; conflictPaths: string[] }
| { ok: false; dirty: true; dirtyPaths: string[] }
> {
// `git commit` after the squash commits the WHOLE index, and conflict cleanup `reset
// --hard`s the tree. So pre-existing *staged* changes are the hazard: they would be
// committed under this run's message, or discarded on conflict. The residue from a prior
// `auto_commit: false` run is exactly that — `git merge --squash` stages without
// committing. Refuse when the index already differs from HEAD. Unstaged working-tree edits
// (e.g. a `ktx.yaml` written by setup and committed later, or user edits) are NOT captured
// by `git commit`, so they must not block the squash; untracked files are likewise ignored.
const dirtyPaths = (await this.git.raw(['diff', '--cached', '--name-only']))
.split('\n')
.map((line) => line.trim())
.filter(Boolean);
if (dirtyPaths.length > 0) {
this.logger.warn(
`applySquashToIndex: target index has ${dirtyPaths.length} staged change(s) ` +
`(${dirtyPaths.slice(0, 5).join(', ')}); refusing to squash ${branch} to avoid committing or discarding them`,
);
return { ok: false, dirty: true, dirtyPaths };
}

// Diff of HEAD..branch (two dots) lists commits/files reachable from `branch` that
// aren't on HEAD — i.e. exactly what the squash would apply. Three dots (HEAD...branch)
// is symmetric difference and would mis-classify cases where main moved ahead.
Expand All @@ -676,8 +772,7 @@ export class GitService {
.map((l) => l.trim())
.filter(Boolean);
if (touchedPaths.length === 0) {
const head = (await this.git.revparse(['HEAD'])).trim();
return { ok: true, squashSha: head, touchedPaths: [] };
return { ok: true, touchedPaths: [] };
}

// `git merge --squash` may NOT throw on a textual conflict — it stages the clean
Expand Down Expand Up @@ -710,9 +805,7 @@ export class GitService {
return { ok: false, conflict: true, conflictPaths };
}

await this.git.commit(commitMessage, { '--author': `${author} <${authorEmail}>` });
const squashSha = (await this.git.revparse(['HEAD'])).trim();
return { ok: true, squashSha, touchedPaths };
return { ok: true, touchedPaths };
}

/**
Expand Down
Loading