diff --git a/docs-site/content/docs/cli-reference/ktx-setup.mdx b/docs-site/content/docs/cli-reference/ktx-setup.mdx index 8ae4469d3..f5b5cf266 100644 --- a/docs-site/content/docs/cli-reference/ktx-setup.mdx +++ b/docs-site/content/docs/cli-reference/ktx-setup.mdx @@ -60,6 +60,11 @@ prompts. | `--vertex-location ` | Vertex AI location, `env:NAME`, or `file:/path` reference | | `--skip-llm` | Leave LLM setup incomplete | +Non-interactive setup (`--no-input`) requires an explicit `--llm-backend`; it +does not assume a default. `--llm-backend` is independent of `--target`: the +target selects which agent client receives integration, while `--llm-backend` +selects the provider **ktx** itself uses. + Choose only one Anthropic credential source. Anthropic credential flags are only valid with the Anthropic backend; Vertex flags are only valid with the Vertex backend. The `claude-code` and `codex` backends use local authentication instead @@ -276,6 +281,7 @@ Use `ktx status` for repeatable readiness checks after setup exits. |-------|-------|----------| | Setup resumes an unexpected project | `KTX_PROJECT_DIR` or nearest `ktx.yaml` points to another directory | Pass `--project-dir ` explicitly | | Setup cannot run in CI | Required values are missing and `--no-input` disables prompts | Provide the relevant automation flags or create a fixture `ktx.yaml` | +| `Missing LLM backend` | `--no-input` without `--llm-backend` | Pass `--llm-backend` with `anthropic`, `vertex`, `claude-code`, or `codex` (the latter two need no API key) | | Provider health check fails | Provider key, model id, Vertex project, or Vertex location is invalid | Fix the `env:` or `file:` reference and rerun setup | | Python runtime is missing | The selected setup needs runtime-backed agent, query-history, Looker, or local embedding features | Accept the interactive prompt, rerun with `--yes`, or run the suggested `ktx admin runtime install` command | | `--enable-query-history` is rejected | The selected database driver does not support query history | Use Postgres, BigQuery, or Snowflake, or rerun without query-history flags | diff --git a/docs-site/content/docs/cli-reference/ktx-sql.mdx b/docs-site/content/docs/cli-reference/ktx-sql.mdx index d78864e2a..9c8f540a2 100644 --- a/docs-site/content/docs/cli-reference/ktx-sql.mdx +++ b/docs-site/content/docs/cli-reference/ktx-sql.mdx @@ -15,6 +15,10 @@ Use `ktx sql` with a required connection id and positional SQL text. ktx sql --connection [options] ``` +`ktx sql` runs against any configured connection, whether or not it is a scan or +ingest target. Connections marked `scan_enabled: false` (execute-only) work here +too — see [execute-only connections](/docs/configuration/ktx-yaml#execute-only-connections). + ## Options Use output flags to choose between terminal display, TSV rows, and structured diff --git a/docs-site/content/docs/configuration/ktx-yaml.mdx b/docs-site/content/docs/configuration/ktx-yaml.mdx index db74ffa78..182a4bea0 100644 --- a/docs-site/content/docs/configuration/ktx-yaml.mdx +++ b/docs-site/content/docs/configuration/ktx-yaml.mdx @@ -158,6 +158,31 @@ connections: dataset_ids: [analytics, mart] ``` +#### Execute-only connections + +Set `scan_enabled: false` to register a warehouse for SQL execution only. The +connection is usable by `ktx sql` and the agent `sql_execution` tool, but **ktx** +never introspects, scans, or ingests it: automatic ingest skips it, `ktx setup` +validates the credential without discovering or scanning its schemas, and even an +explicit `ktx scan ` or `ktx ingest ` is refused with guidance. This is +the supported way to run read-only queries against shared or public data (for +example a BigQuery billing project full of unrelated datasets) without making it +a context source. + +```yaml +connections: + public_bq: + driver: bigquery + credentials_json: file:./service-account.json + scan_enabled: false +``` + +Without `scan_enabled`, a warehouse is a scan target. In scripted setup +(`--no-input`) with no `--database-schema` and no `dataset_ids`/`schemas`, **ktx** +scopes the scan to every schema or dataset the credential can see and prints a +warning naming the count; pass `--database-schema` to narrow it, or +`scan_enabled: false` to register it for execution only. + For Postgres, MySQL, SQL Server, and Snowflake connections, set `maxConnections` when scan or ingest work needs to stay below the target's connection cap. Postgres, MySQL, and SQL Server default to `10`; Snowflake @@ -360,7 +385,7 @@ storage: |-------|------|---------|---------| | `state` | `sqlite` \| `postgres` | `sqlite` | Backend for ktx state. `sqlite` uses `.ktx/db.sqlite`; `postgres` expects a configured Postgres connection. | | `search` | `sqlite-fts5` \| `postgres-hybrid` | `sqlite-fts5` | Backend for search indexes. `postgres-hybrid` combines lexical and vector search in Postgres. | -| `git.auto_commit` | `boolean` | `true` | When `true`, ktx auto-commits changes to the git-backed state store. | +| `git.auto_commit` | `boolean` | `true` | When `true`, a context-source ingest run commits its changes to the git-backed state store. When `false`, the changes are applied to the working tree and left staged for you to commit. | | `git.author` | `string` | `ktx ` | Git author identity for auto-commits. Standard `Name ` form. | ## `llm` @@ -619,7 +644,7 @@ memory: | Field | Type | Default | Purpose | |-------|------|---------|---------| -| `auto_commit` | `boolean` | `true` | When `true`, ktx auto-commits memory updates to the git-backed store. | +| `auto_commit` | `boolean` | `true` | When `true`, a memory/wiki ingest run commits its updates to the git-backed store. When `false`, the updates are applied to the working tree and left staged for you to commit. | ## A full example diff --git a/docs-site/content/docs/guides/reviewing-context.mdx b/docs-site/content/docs/guides/reviewing-context.mdx index 63d4fceba..f60adbf3d 100644 --- a/docs-site/content/docs/guides/reviewing-context.mdx +++ b/docs-site/content/docs/guides/reviewing-context.mdx @@ -59,6 +59,14 @@ and replay; it belongs in `.gitignore`. If your team wants a record of *why* a change happened, link the transcript path in the PR description rather than committing the file. +**ktx** maintains its own git repository at the project directory. When the +project lives inside an existing repository (for example a `data/ktx` +subdirectory of your application repo), **ktx** initializes a dedicated +repository at the project directory rather than committing into the enclosing +one — its ingest commits stay isolated, and writes and reindexing always share +the same working-tree root. Track the project directory in your outer repo as a +nested checkout (or keep it separate) depending on how you want to review it. + ## A typical review session The loop above describes the shape. In practice, one review session looks like diff --git a/packages/cli/src/commands/setup-commands.ts b/packages/cli/src/commands/setup-commands.ts index 418b27f96..a51ac72e9 100644 --- a/packages/cli/src/commands/setup-commands.ts +++ b/packages/cli/src/commands/setup-commands.ts @@ -21,20 +21,6 @@ function positiveInteger(value: string): number { return parsed; } -function embeddingBackend(value: string): 'openai' | 'sentence-transformers' { - if (value === 'openai' || value === 'sentence-transformers') { - return value; - } - throw new InvalidArgumentError(`invalid choice '${value}'`); -} - -function llmBackend(value: string): KtxSetupLlmBackend { - if (value === 'anthropic' || value === 'vertex' || value === 'claude-code' || value === 'codex') { - return value; - } - throw new InvalidArgumentError(`invalid choice '${value}'`); -} - function databaseDriver(value: string): KtxSetupDatabaseDriver { if ( value === 'sqlite' || @@ -220,7 +206,11 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo .addOption(new Option('--skip-agents', 'Leave agent integration incomplete for now').hideHelp().default(false)) .option('--yes', 'Accept project creation and runtime install defaults where setup confirms', false) .option('--no-input', 'Disable interactive terminal input') - .addOption(new Option('--llm-backend ', 'LLM backend').argParser(llmBackend).hideHelp()) + .addOption( + new Option('--llm-backend ', 'LLM backend') + .choices(['anthropic', 'vertex', 'claude-code', 'codex']) + .hideHelp(), + ) .addOption( new Option('--anthropic-api-key-env ', 'Environment variable containing the Anthropic API key').hideHelp(), ) @@ -230,7 +220,11 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo .addOption(new Option('--vertex-project ', 'Google Vertex AI project ID, env:NAME, or file:/path').hideHelp()) .addOption(new Option('--vertex-location ', 'Google Vertex AI location, env:NAME, or file:/path').hideHelp()) .addOption(new Option('--skip-llm', 'Leave LLM setup incomplete for now').hideHelp().default(false)) - .addOption(new Option('--embedding-backend ', 'Embedding backend').argParser(embeddingBackend).hideHelp()) + .addOption( + new Option('--embedding-backend ', 'Embedding backend') + .choices(['openai', 'sentence-transformers']) + .hideHelp(), + ) .addOption( new Option( '--embedding-api-key-env ', diff --git a/packages/cli/src/context/connections/local-warehouse-descriptor.ts b/packages/cli/src/context/connections/local-warehouse-descriptor.ts index 4ad926df1..674a7033b 100644 --- a/packages/cli/src/context/connections/local-warehouse-descriptor.ts +++ b/packages/cli/src/context/connections/local-warehouse-descriptor.ts @@ -72,6 +72,24 @@ export function localConnectionToWarehouseDescriptor( return info; } +/** + * True when the connection is registered for SQL execution only (`scan_enabled: false`) and + * must never be used as a scan/ingest target. Execution paths (`ktx sql`, `sql_execution`) are + * unaffected — they resolve the warehouse via {@link localConnectionToWarehouseDescriptor}. + */ +export function isExecuteOnlyConnection(connection: KtxProjectConnectionConfig | undefined): boolean { + return (connection as { scan_enabled?: boolean } | undefined)?.scan_enabled === false; +} + +/** + * True when the connection is a warehouse AND eligible to be scanned/ingested. This is the single + * predicate every scan-target selection path routes through, so execute-only connections are + * excluded consistently — including the "fall back to all warehouses" path. + */ +export function isScanTargetWarehouse(id: string, connection: KtxProjectConnectionConfig | undefined): boolean { + return localConnectionToWarehouseDescriptor(id, connection) !== null && !isExecuteOnlyConnection(connection); +} + export function localConnectionTypeForConfig(id: string, connection: KtxProjectConnectionConfig | undefined): string { const descriptor = localConnectionToWarehouseDescriptor(id, connection); if (descriptor) { diff --git a/packages/cli/src/context/core/git.service.ts b/packages/cli/src/context/core/git.service.ts index 216183ff5..9343eb611 100644 --- a/packages/cli/src/context/core/git.service.ts +++ b/packages/cli/src/context/core/git.service.ts @@ -1,6 +1,6 @@ import { promises as fs } from 'node:fs'; import { dirname, join } from 'node:path'; -import type { SimpleGit } from 'simple-git'; +import { CheckRepoActions, type SimpleGit } from 'simple-git'; import { noopLogger, resolveConfigDir, type KtxCoreConfig, type KtxLogger } from './config.js'; import { createSimpleGit } from './git-env.js'; @@ -27,9 +27,21 @@ export interface WorktreeEntry { head: string | null; } +/** + * `dirty` means the target (main) working tree had uncommitted tracked changes before the + * merge — typically residue from a prior `auto_commit: false` run that was never committed. + * Squashing into it would either commit those unrelated changes or, on conflict cleanup, + * `reset --hard` them away, so the merge is refused and the tree is left untouched. + */ export type SquashMergeResult = | { ok: true; squashSha: string; touchedPaths: string[] } - | { ok: false; conflict: true; conflictPaths: string[] }; + | { ok: false; conflict: true; conflictPaths: string[] } + | { ok: false; dirty: true; dirtyPaths: string[] }; + +export type StageSquashResult = + | { ok: true; touchedPaths: string[]; stagedTree: string } + | { ok: false; conflict: true; conflictPaths: string[] } + | { ok: false; dirty: true; dirtyPaths: string[] }; function mergeErrorMessage(error: unknown): string { if (error instanceof Error) { @@ -94,15 +106,29 @@ export class GitService { private async initialize(): Promise { try { - // Check if already initialized - const isRepo = await this.git.checkIsRepo(); - - if (!isRepo) { + // The ktx store assumes configDir is its own git working-tree root: writes, session + // worktrees, squash-merges, and reindex scans are all resolved relative to it. The + // default checkIsRepo() (IN_TREE) is also satisfied by an *enclosing* repository, so a + // project nested inside another git working tree would silently operate against that + // outer repo — ingest-written files land at the outer root while reindex scans configDir, + // leaving e.g. the wiki seeded but unindexed. Require configDir to be the repo root + // itself; otherwise initialize a dedicated repository here. + const isOwnRepoRoot = await this.git.checkIsRepo(CheckRepoActions.IS_REPO_ROOT); + + if (!isOwnRepoRoot) { + const insideEnclosingRepo = await this.git.checkIsRepo(CheckRepoActions.IN_TREE); await this.git.init(); const gitConfig = this.config.git; await this.git.addConfig('user.name', gitConfig.userName); await this.git.addConfig('user.email', gitConfig.userEmail); - this.logger.log('Initialized git repository'); + if (insideEnclosingRepo) { + this.logger.log( + `Initialized a dedicated ktx git repository at ${this.configDir} (nested inside an existing ` + + 'git repository); ktx commits stay in this repository and do not touch the outer one.', + ); + } else { + this.logger.log('Initialized git repository'); + } } // Keep any auto-maintenance triggered by writes in-process. Detached maintenance can @@ -667,6 +693,76 @@ export class GitService { authorEmail: string, commitMessage: string, ): Promise { + const applied = await this.applySquashToIndex(branch); + if (!applied.ok) { + return applied; + } + if (applied.touchedPaths.length === 0) { + const head = (await this.git.revparse(['HEAD'])).trim(); + return { ok: true, squashSha: head, touchedPaths: [] }; + } + + await this.git.commit(commitMessage, { '--author': `${author} <${authorEmail}>` }); + const squashSha = (await this.git.revparse(['HEAD'])).trim(); + return { ok: true, squashSha, touchedPaths: applied.touchedPaths }; + } + + /** + * Like {@link squashMergeIntoMain} but stops before committing: applies `branch` onto the + * current branch's index + working tree and leaves the result staged for the user to commit. + * Returns the staged tree's SHA, which is a valid diff/read ref (`git diff A..`, + * `git show :path`) so callers can reconcile derived indexes without a commit. + * + * This backs the `auto_commit: false` ingest path: changes still reach the working tree (so + * the run is not silently discarded), they are just not committed automatically. + * + * Caller must hold the `config:repo` lock, as with {@link squashMergeIntoMain}. + */ + async stageSquashMergeIntoMain(branch: string): Promise { + return this.withMutationQueue(() => this.stageSquashMergeIntoMainUnlocked(branch)); + } + + private async stageSquashMergeIntoMainUnlocked(branch: string): Promise { + const applied = await this.applySquashToIndex(branch); + if (!applied.ok) { + return applied; + } + const stagedTree = (await this.git.raw(['write-tree'])).trim(); + return { ok: true, touchedPaths: applied.touchedPaths, stagedTree }; + } + + /** + * Shared core of the squash-merge paths: applies `branch` onto the current branch's index + + * working tree via `git merge --squash` WITHOUT committing, leaving the caller to either + * commit (auto-commit on) or stage (auto-commit off). Returns the touched paths, or conflict + * info after restoring a clean tree. + */ + private async applySquashToIndex( + branch: string, + ): Promise< + | { ok: true; touchedPaths: string[] } + | { ok: false; conflict: true; conflictPaths: string[] } + | { ok: false; dirty: true; dirtyPaths: string[] } + > { + // `git commit` after the squash commits the WHOLE index, and conflict cleanup `reset + // --hard`s the tree. So pre-existing *staged* changes are the hazard: they would be + // committed under this run's message, or discarded on conflict. The residue from a prior + // `auto_commit: false` run is exactly that — `git merge --squash` stages without + // committing. Refuse when the index already differs from HEAD. Unstaged working-tree edits + // (e.g. a `ktx.yaml` written by setup and committed later, or user edits) are NOT captured + // by `git commit`, so they must not block the squash; untracked files are likewise ignored. + const dirtyPaths = (await this.git.raw(['diff', '--cached', '--name-only'])) + .split('\n') + .map((line) => line.trim()) + .filter(Boolean); + if (dirtyPaths.length > 0) { + this.logger.warn( + `applySquashToIndex: target index has ${dirtyPaths.length} staged change(s) ` + + `(${dirtyPaths.slice(0, 5).join(', ')}); refusing to squash ${branch} to avoid committing or discarding them`, + ); + return { ok: false, dirty: true, dirtyPaths }; + } + // Diff of HEAD..branch (two dots) lists commits/files reachable from `branch` that // aren't on HEAD — i.e. exactly what the squash would apply. Three dots (HEAD...branch) // is symmetric difference and would mis-classify cases where main moved ahead. @@ -676,8 +772,7 @@ export class GitService { .map((l) => l.trim()) .filter(Boolean); if (touchedPaths.length === 0) { - const head = (await this.git.revparse(['HEAD'])).trim(); - return { ok: true, squashSha: head, touchedPaths: [] }; + return { ok: true, touchedPaths: [] }; } // `git merge --squash` may NOT throw on a textual conflict — it stages the clean @@ -710,9 +805,7 @@ export class GitService { return { ok: false, conflict: true, conflictPaths }; } - await this.git.commit(commitMessage, { '--author': `${author} <${authorEmail}>` }); - const squashSha = (await this.git.revparse(['HEAD'])).trim(); - return { ok: true, squashSha, touchedPaths }; + return { ok: true, touchedPaths }; } /** diff --git a/packages/cli/src/context/ingest/ingest-bundle.runner.ts b/packages/cli/src/context/ingest/ingest-bundle.runner.ts index 6f5372d27..90207c5ef 100644 --- a/packages/cli/src/context/ingest/ingest-bundle.runner.ts +++ b/packages/cli/src/context/ingest/ingest-bundle.runner.ts @@ -2677,29 +2677,58 @@ export class IngestBundleRunner { throw error; } const commitMessage = this.buildCommitMessage(job, syncId, diffSummary, failedWorkUnits); + // With auto-commit disabled, apply the run onto main's working tree and leave it staged + // rather than committing. The wiki index is reconciled from the staged tree (a valid + // diff/read ref), so search stays consistent with the staged files; only the git commit + // and its message-enhancement job are skipped. + const autoCommit = this.deps.storage.autoCommit; const squashResult = await this.deps.lockingService.withLock('config:repo', async () => { const preSquashSha = await this.deps.gitService.revParseHead(); - const merge = await this.deps.gitService.squashMergeIntoMain( - sessionWorktree.branch, - this.deps.storage.systemGitAuthor.name, - this.deps.storage.systemGitAuthor.email, - commitMessage, - ); - return { preSquashSha, merge }; + if (autoCommit) { + const merge = await this.deps.gitService.squashMergeIntoMain( + sessionWorktree.branch, + this.deps.storage.systemGitAuthor.name, + this.deps.storage.systemGitAuthor.email, + commitMessage, + ); + return { preSquashSha, committed: true as const, merge }; + } + const merge = await this.deps.gitService.stageSquashMergeIntoMain(sessionWorktree.branch); + return { preSquashSha, committed: false as const, merge }; }); - const mergeResult = squashResult.merge; - if (!mergeResult.ok) { + if (!squashResult.merge.ok) { await this.deps.runs.markFailed(runRow.id); - throw new Error(`squash merge conflict: ${mergeResult.conflictPaths.join(', ')}`); + if ('dirty' in squashResult.merge) { + throw new Error( + 'The project has staged but uncommitted changes ' + + `(${squashResult.merge.dirtyPaths.slice(0, 5).join(', ')}); commit or unstage them before ingesting ` + + '(this typically means a previous run with storage.git.auto_commit: false was left staged).', + ); + } + throw new Error(`squash merge conflict: ${squashResult.merge.conflictPaths.join(', ')}`); + } + const touchedPaths = squashResult.merge.touchedPaths; + const hasChanges = touchedPaths.length > 0; + // `syncRef` is the tree-ish to diff/read when reconciling the wiki index: the new commit + // SHA when committed, the staged tree SHA when staging. `commitSha` is only set when an + // actual commit was created (it surfaces in the report and progress UI). + let commitSha: string | null = null; + let syncRef: string | null = null; + if (hasChanges) { + if (squashResult.committed) { + commitSha = squashResult.merge.squashSha; + syncRef = commitSha; + } else { + syncRef = squashResult.merge.stagedTree; + } } - const commitSha = mergeResult.touchedPaths.length === 0 ? null : mergeResult.squashSha; await runTrace.event( 'debug', 'squash', 'squash_finished', { commitSha, - touchedPaths: mergeResult.touchedPaths, + touchedPaths, }, undefined, Date.now() - squashStartedAt, @@ -2714,18 +2743,28 @@ export class IngestBundleRunner { wikiCount: countMemoryFlowActions(memoryFlowSavedActions, 'wiki'), slCount: countMemoryFlowActions(memoryFlowSavedActions, 'sl'), }); - await stage6?.updateProgress(1.0, commitSha ? `Saved changes (${commitSha.slice(0, 8)})` : 'No changes to save'); + await stage6?.updateProgress( + 1.0, + commitSha + ? `Saved changes (${commitSha.slice(0, 8)})` + : hasChanges + ? 'Staged changes (auto-commit disabled)' + : 'No changes to save', + ); // Sync the shared `knowledge` index from the squashed diff in a single // transaction. If this throws, the run fails and no partial index state // survives (thanks to the transactional upsert in applyDiffTransactional). - if (commitSha) { + // `syncRef` is the new commit when committed, or the staged tree when staging. + if (syncRef) { const indexSyncStartedAt = Date.now(); // Multi-file squash → omit path so the handler diffs the whole commit // (a comma-joined pathspec would match nothing and the job would no-op). - const pathFilter = mergeResult.touchedPaths.length === 1 ? mergeResult.touchedPaths[0] : ''; - await this.deps.commitMessages.enqueueForExternalCommit({ commitHash: commitSha }, commitMessage, pathFilter); - await this.deps.wikiService.syncFromCommit(squashResult.preSquashSha, commitSha, runRow.id); + const pathFilter = touchedPaths.length === 1 ? touchedPaths[0] : ''; + if (squashResult.committed) { + await this.deps.commitMessages.enqueueForExternalCommit({ commitHash: syncRef }, commitMessage, pathFilter); + } + await this.deps.wikiService.syncFromCommit(squashResult.preSquashSha, syncRef, runRow.id); await this.syncKnowledgeSlRefsFromActions(job.connectionId, memoryFlowSavedActions); const touchedConnections = [ ...new Set( diff --git a/packages/cli/src/context/ingest/local-adapters.ts b/packages/cli/src/context/ingest/local-adapters.ts index 3cd8a9989..3d3e69d10 100644 --- a/packages/cli/src/context/ingest/local-adapters.ts +++ b/packages/cli/src/context/ingest/local-adapters.ts @@ -1,5 +1,5 @@ import { join } from 'node:path'; -import { localConnectionToWarehouseDescriptor } from '../../context/connections/local-warehouse-descriptor.js'; +import { isScanTargetWarehouse, localConnectionToWarehouseDescriptor } from '../../context/connections/local-warehouse-descriptor.js'; import { notionConnectionToPullConfig, parseNotionConnectionConfig } from '../../context/connections/notion-config.js'; import { resolveKtxConfigReference } from '../core/config-reference.js'; import { ktxLocalStateDbPath } from '../../context/project/local-state-db.js'; @@ -147,14 +147,14 @@ export function createDefaultLocalIngestAdapters( function primaryWarehouseConnectionIds(project: KtxLocalProject): string[] { const configuredPrimaryIds = project.config.setup?.database_connection_ids ?? []; const configured = configuredPrimaryIds.filter((connectionId) => - Boolean(localConnectionToWarehouseDescriptor(connectionId, project.config.connections[connectionId])), + isScanTargetWarehouse(connectionId, project.config.connections[connectionId]), ); if (configured.length > 0) { return [...new Set(configured)]; } return Object.entries(project.config.connections) - .filter(([connectionId, connection]) => Boolean(localConnectionToWarehouseDescriptor(connectionId, connection))) + .filter(([connectionId, connection]) => isScanTargetWarehouse(connectionId, connection)) .map(([connectionId]) => connectionId) .sort((left, right) => left.localeCompare(right)); } diff --git a/packages/cli/src/context/ingest/local-bundle-runtime.ts b/packages/cli/src/context/ingest/local-bundle-runtime.ts index 69b9159a0..3f6c3381f 100644 --- a/packages/cli/src/context/ingest/local-bundle-runtime.ts +++ b/packages/cli/src/context/ingest/local-bundle-runtime.ts @@ -129,6 +129,10 @@ class LocalIngestStorage implements IngestStoragePort { this.homeDir = join(project.projectDir, '.ktx'); } + get autoCommit(): boolean { + return this.project.config.storage.git.auto_commit; + } + resolveUploadDir(uploadId: string): string { return join(this.project.projectDir, '.ktx/cache/local-ingest', uploadId, 'upload'); } diff --git a/packages/cli/src/context/ingest/ports.ts b/packages/cli/src/context/ingest/ports.ts index 88294f59b..d68984781 100644 --- a/packages/cli/src/context/ingest/ports.ts +++ b/packages/cli/src/context/ingest/ports.ts @@ -159,6 +159,11 @@ interface IngestGitAuthor { export interface IngestStoragePort { homeDir: string; systemGitAuthor: IngestGitAuthor; + /** + * Mirror of config `storage.git.auto_commit`. When false, an ingest run applies its squash + * onto the project's working tree and leaves it staged instead of committing it. + */ + autoCommit: boolean; resolveUploadDir(uploadId: string): string; resolvePullDir(jobId: string): string; resolveTranscriptDir(jobId: string): string; diff --git a/packages/cli/src/context/memory/local-memory.ts b/packages/cli/src/context/memory/local-memory.ts index b72bc9ce7..82805427d 100644 --- a/packages/cli/src/context/memory/local-memory.ts +++ b/packages/cli/src/context/memory/local-memory.ts @@ -116,6 +116,7 @@ export function createLocalProjectMemoryIngest( knowledge: { userScopedKnowledgeEnabled: false }, slValidation: { probeRowCount: 0 }, llm: { memoryIngestionModel: project.config.llm.models.default ?? 'local-memory-model' }, + autoCommit: project.config.memory.auto_commit, }, promptService: new PromptService({ promptsDir, partials: [] }), skillsRegistry: new SkillsRegistryService({ skillsDir }), diff --git a/packages/cli/src/context/memory/memory-agent.service.ts b/packages/cli/src/context/memory/memory-agent.service.ts index 7f6438d30..6d649203a 100644 --- a/packages/cli/src/context/memory/memory-agent.service.ts +++ b/packages/cli/src/context/memory/memory-agent.service.ts @@ -4,6 +4,7 @@ import { join } from 'node:path'; import * as YAML from 'yaml'; import { z } from 'zod'; import { type KtxLogger, noopLogger } from '../../context/core/config.js'; +import type { SquashMergeResult, StageSquashResult } from '../../context/core/git.service.js'; import type { KtxRuntimeToolSet } from '../../context/llm/runtime-port.js'; import { revertSourceToPreHead, type SlValidationDeps } from '../../context/sl/tools/sl-warehouse-validation.js'; import type { SemanticLayerSource } from '../../context/sl/types.js'; @@ -253,33 +254,52 @@ export class MemoryAgentService { reconciledCrossRefs, gateRevertedSources, ); - const mergeResult = await this.deps.lockingService.withLock('config:repo', () => - this.deps.gitService.squashMergeIntoMain( - sessionWorktree.branch, - SYSTEM_GIT_AUTHOR.name, - SYSTEM_GIT_AUTHOR.email, - squashMessage, - ), + // With auto-commit disabled, apply the session to main's working tree and leave it + // staged rather than committing. The knowledge/SL DB is written eagerly during the + // session (and rolled back on conflict below), so search stays consistent with the + // staged files; we only skip the git commit and its message-enhancement job. + const autoCommit = this.deps.settings.autoCommit; + const mergeResult = await this.deps.lockingService.withLock('config:repo', () => + autoCommit + ? this.deps.gitService.squashMergeIntoMain( + sessionWorktree.branch, + SYSTEM_GIT_AUTHOR.name, + SYSTEM_GIT_AUTHOR.email, + squashMessage, + ) + : this.deps.gitService.stageSquashMergeIntoMain(sessionWorktree.branch), ); if (!mergeResult.ok) { + // Both conflict and dirty-target mean "not landed" — roll back the eager session + // writes so the DB doesn't get ahead of main, and leave main untouched. sessionOutcome = 'conflict'; - sessionConflictPaths = mergeResult.conflictPaths; + if ('dirty' in mergeResult) { + this.logger.warn( + `[memory-agent] chat=${chatId} not landed: project has staged but uncommitted changes ` + + `(${mergeResult.dirtyPaths.slice(0, 5).join(', ')}); commit or unstage them and re-run ` + + '(a previous run with auto_commit disabled may have been left staged).', + ); + } else { + sessionConflictPaths = mergeResult.conflictPaths; + } await this.rollbackDbForAbortedSession(session, actions); } else if (mergeResult.touchedPaths.length === 0) { sessionOutcome = 'empty'; } else { - squashSha = mergeResult.squashSha; touchedPaths = mergeResult.touchedPaths; - // Single-file commits: pass the path so the handler diff is path-scoped. - // Multi-file commits: omit path so the handler grabs the full commit diff - // (a comma-joined pathspec would match nothing). - const pathFilter = touchedPaths.length === 1 ? touchedPaths[0] : ''; - await this.deps.rootFileStore.enqueueCommitMessageJobForExternalCommit( - { commitHash: squashSha }, - squashMessage, - pathFilter, - ); + if ('squashSha' in mergeResult) { + squashSha = mergeResult.squashSha; + // Single-file commits: pass the path so the handler diff is path-scoped. + // Multi-file commits: omit path so the handler grabs the full commit diff + // (a comma-joined pathspec would match nothing). + const pathFilter = touchedPaths.length === 1 ? touchedPaths[0] : ''; + await this.deps.rootFileStore.enqueueCommitMessageJobForExternalCommit( + { commitHash: squashSha }, + squashMessage, + pathFilter, + ); + } } } catch (error) { sessionCrashed = true; diff --git a/packages/cli/src/context/memory/types.ts b/packages/cli/src/context/memory/types.ts index 5cc7dab23..90df9f58b 100644 --- a/packages/cli/src/context/memory/types.ts +++ b/packages/cli/src/context/memory/types.ts @@ -74,6 +74,11 @@ interface MemoryAgentSettings { llm: { memoryIngestionModel: string; }; + /** + * When false (config `memory.auto_commit: false`), a completed session is applied to the + * project's working tree and left staged instead of committed, so the user commits it. + */ + autoCommit: boolean; } interface MemoryTelemetryPort { diff --git a/packages/cli/src/context/project/config.ts b/packages/cli/src/context/project/config.ts index fd7f482c2..18c87626b 100644 --- a/packages/cli/src/context/project/config.ts +++ b/packages/cli/src/context/project/config.ts @@ -230,7 +230,12 @@ const setupSchema = z const storageGitSchema = z .strictObject({ - auto_commit: z.boolean().default(true).describe('When true, KTX automatically commits state changes to the local Git-backed store.'), + auto_commit: z + .boolean() + .default(true) + .describe( + 'When true, a context-source ingest run (`ktx ingest `) commits its changes to the local Git-backed store. When false, the changes are applied to the working tree and left staged for you to commit.', + ), author: z .string() .min(1) @@ -278,7 +283,12 @@ const agentSchema = z const memorySchema = z .strictObject({ - auto_commit: z.boolean().default(true).describe('When true, KTX automatically commits memory updates to the Git-backed store.'), + auto_commit: z + .boolean() + .default(true) + .describe( + 'When true, a memory/wiki ingest run commits its updates to the Git-backed store. When false, the updates are applied to the working tree and left staged for you to commit.', + ), }) .describe('Memory subsystem configuration.'); diff --git a/packages/cli/src/context/project/driver-schemas.ts b/packages/cli/src/context/project/driver-schemas.ts index f9a3639fe..12505b662 100644 --- a/packages/cli/src/context/project/driver-schemas.ts +++ b/packages/cli/src/context/project/driver-schemas.ts @@ -32,6 +32,12 @@ function warehouseConnectionSchema(driver: .describe( 'Optional allowlist of fully-qualified table names ("schema.table") to ingest. When set, live-database ingest discards any table whose schema-qualified name is not in this list. Useful for smoke-testing ingest on a single table.', ), + scan_enabled: z + .boolean() + .optional() + .describe( + 'When false, this connection is registered for SQL execution only (ktx sql / sql_execution) and is never used as a scan/ingest target. Omit (or true) to scan and ingest it as a primary warehouse.', + ), }) .describe( `${driver} warehouse connection. Additional driver-tunable fields (e.g. context.queryHistory) are accepted and passed through.`, diff --git a/packages/cli/src/ingest.ts b/packages/cli/src/ingest.ts index 233b1b6e4..46b04692a 100644 --- a/packages/cli/src/ingest.ts +++ b/packages/cli/src/ingest.ts @@ -8,6 +8,7 @@ import type { MemoryFlowEvent, MemoryFlowReplayInput } from './context/ingest/me import { renderMemoryFlowReplay } from './context/ingest/memory-flow/render.js'; import type { KtxSqlQueryExecutorPort } from './context/connections/query-executor.js'; import { loadKtxProject, type KtxLocalProject } from './context/project/project.js'; +import { isExecuteOnlyConnection } from './context/connections/local-warehouse-descriptor.js'; import { getKtxCliPackageInfo } from './cli-runtime.js'; import { resolveProjectEmbeddingProvider } from './embedding-resolution.js'; import { createKtxCliIngestQueryExecutor } from './ingest-query-executor.js'; @@ -695,6 +696,13 @@ export async function runKtxIngest( const project = await loadKtxProject({ projectDir: args.projectDir }); const env = deps.env ?? process.env; if (args.command === 'run') { + if (isExecuteOnlyConnection(project.config.connections[args.connectionId])) { + io.stderr.write( + `Connection '${args.connectionId}' is registered for SQL execution only (scan_enabled: false) and ` + + 'cannot be ingested. Remove scan_enabled: false to make it a scan/ingest target, or use `ktx sql` to query it.\n', + ); + return 1; + } const resolveEmbeddingProvider = deps.resolveEmbeddingProvider ?? resolveProjectEmbeddingProvider; const resolution = await resolveEmbeddingProvider(project, { mode: 'ensure', diff --git a/packages/cli/src/scan.ts b/packages/cli/src/scan.ts index 5961e3f12..8169cf081 100644 --- a/packages/cli/src/scan.ts +++ b/packages/cli/src/scan.ts @@ -1,6 +1,7 @@ import type { KtxProgressPort, KtxScanMode, KtxScanReport, KtxScanWarning } from './context/scan/types.js'; import { runLocalScan } from './context/scan/local-scan.js'; import { loadKtxProject, type KtxLocalProject } from './context/project/project.js'; +import { isExecuteOnlyConnection } from './context/connections/local-warehouse-descriptor.js'; import { getKtxCliPackageInfo } from './cli-runtime.js'; import { resolveProjectEmbeddingProvider } from './embedding-resolution.js'; import type { KtxCliIo } from './index.js'; @@ -326,6 +327,13 @@ export async function runKtxScan(args: KtxScanArgs, io: KtxCliIo = process, deps let project: KtxLocalProject | undefined; try { project = await loadKtxProject({ projectDir: args.projectDir }); + if (isExecuteOnlyConnection(project.config.connections[args.connectionId])) { + io.stderr.write( + `Connection '${args.connectionId}' is registered for SQL execution only (scan_enabled: false) and ` + + 'cannot be scanned. Remove scan_enabled: false to make it a scan target, or use `ktx sql` to query it.\n', + ); + return 1; + } const resolveEmbeddingProvider = deps.resolveEmbeddingProvider ?? resolveProjectEmbeddingProvider; const resolution = await resolveEmbeddingProvider(project, { mode: 'ensure', diff --git a/packages/cli/src/setup-databases.ts b/packages/cli/src/setup-databases.ts index 002ead30e..9a22c6430 100644 --- a/packages/cli/src/setup-databases.ts +++ b/packages/cli/src/setup-databases.ts @@ -4,6 +4,7 @@ import { delimiter, dirname, join } from 'node:path'; import { fileURLToPath } from 'node:url'; import { promisify } from 'node:util'; import { getDriverRegistration } from './context/connections/drivers.js'; +import { isExecuteOnlyConnection } from './context/connections/local-warehouse-descriptor.js'; import { createLocalKtxLlmRuntimeFromConfig } from './context/llm/local-config.js'; import type { KtxLlmRuntimePort } from './context/llm/runtime-port.js'; import { queryHistoryDialectForConnection } from './context/ingest/adapters/historic-sql/connection-dialect.js'; @@ -459,12 +460,14 @@ function configuredPrimaryConnectionIds( const configuredIds = setupConnectionIds ?.filter((connectionId) => normalizeDriver(connections[connectionId]?.driver) !== null) + .filter((connectionId) => !isExecuteOnlyConnection(connections[connectionId])) .filter((connectionId, index, ids) => ids.indexOf(connectionId) === index) ?? []; if (configuredIds.length > 0) { return configuredIds; } return Object.entries(connections) .filter(([, connection]) => normalizeDriver(connection.driver) !== null) + .filter(([, connection]) => !isExecuteOnlyConnection(connection)) .map(([connectionId]) => connectionId) .sort((left, right) => left.localeCompare(right)); } @@ -1384,11 +1387,13 @@ async function maybeConfigureDatabaseScope(input: { if (input.args.inputMode === 'disabled') { if (spec) { let scopeToWrite: string[] = cliSchemas; + let scopedFromDiscovery = false; if (scopeToWrite.length === 0) { try { scopeToWrite = unique( await (input.deps.listSchemas ?? defaultListSchemas)(input.projectDir, input.connectionId), ); + scopedFromDiscovery = true; } catch (error) { const detail = error instanceof Error ? error.message : String(error); input.io.stderr.write( @@ -1397,6 +1402,18 @@ async function maybeConfigureDatabaseScope(input: { return okValidateResult(); } } + // Scripted setup with no explicit scope would otherwise silently scan every discovered + // schema/dataset the credential can see — including unrelated ones on a shared billing + // account. Surface that so the operator can narrow it or register the connection as + // execute-only instead of discovering it as a silent full-warehouse scan. + if (scopedFromDiscovery && scopeToWrite.length > 1) { + input.io.stderr.write( + `No --database-schema given for ${input.connectionId}; scanning all ${scopeToWrite.length} ` + + `discovered ${spec.nounPlural} (${scopeToWrite.join(', ')}). Pass --database-schema to narrow ` + + 'the scan, or set connections.' + + `${input.connectionId}.scan_enabled: false to register it for SQL execution only.\n`, + ); + } if (scopeToWrite.length > 0) { await writeScopeConfig({ projectDir: input.projectDir, @@ -1894,6 +1911,15 @@ async function validateAndScanConnection(input: { const testLines = ['✓ Connection test passed', `Driver: ${driverDisplay}`]; writeSetupSection(input.io, `Testing ${input.connectionId}`, testLines); + // Execute-only connections (scan_enabled: false) are registered for SQL execution only: + // the credential is validated above, but ktx never introspects/scans the warehouse. + if (isExecuteOnlyConnection(project.config.connections[input.connectionId])) { + writeSetupSection(input.io, `Registering ${input.connectionId}`, [ + 'Execute-only connection (scan_enabled: false) — skipping schema scan.', + ]); + return okValidateResult(); + } + const scopeStatus = await maybeConfigureDatabaseScope({ ...input, forcePrompt: input.forceScopeAndTables }); if (scopeStatus.status !== 'ok') { return scopeStatus; diff --git a/packages/cli/src/setup-embeddings.ts b/packages/cli/src/setup-embeddings.ts index 5d02e3e48..92be5d259 100644 --- a/packages/cli/src/setup-embeddings.ts +++ b/packages/cli/src/setup-embeddings.ts @@ -213,7 +213,10 @@ async function chooseCredentialRef( return { status: 'ready', ref, value }; } if (args.inputMode === 'disabled') { - io.stderr.write('Missing embedding API key: pass --embedding-api-key-env or --embedding-api-key-file.\n'); + io.stderr.write( + 'Missing embedding API key for --embedding-backend openai: pass --embedding-api-key-env or --embedding-api-key-file ' + + '(or use --embedding-backend sentence-transformers for local embeddings).\n', + ); return { status: 'missing-input' }; } diff --git a/packages/cli/src/setup-models.ts b/packages/cli/src/setup-models.ts index 911579a9c..890b81ef9 100644 --- a/packages/cli/src/setup-models.ts +++ b/packages/cli/src/setup-models.ts @@ -135,7 +135,8 @@ const execFileAsync = promisify(execFile); type ChooseBackendResult = | { status: 'ready'; backend: KtxSetupLlmBackend; prompted: boolean } - | { status: 'back' }; + | { status: 'back' } + | { status: 'missing-input' }; type VertexConfigChoice = | { @@ -372,7 +373,10 @@ async function chooseCredentialRef( return { status: 'ready', ref, value }; } if (args.inputMode === 'disabled') { - io.stderr.write('Missing Anthropic API key: pass --anthropic-api-key-env or --anthropic-api-key-file.\n'); + io.stderr.write( + 'Missing Anthropic API key for --llm-backend anthropic: pass --anthropic-api-key-env or --anthropic-api-key-file ' + + '(or use --llm-backend claude-code or --llm-backend codex for local subscription auth).\n', + ); return { status: 'missing-input' }; } @@ -446,7 +450,16 @@ async function chooseBackend( return { status: 'ready', backend: explicit, prompted: false }; } if (args.inputMode === 'disabled') { - return { status: 'ready', backend: 'anthropic', prompted: false }; + // No safe default exists: anthropic/vertex need credentials and claude-code/codex + // need local auth, so non-interactive setup must be told which backend to use rather + // than silently picking one that cannot self-configure. + io.stderr.write( + 'Missing LLM backend: pass --llm-backend with one of anthropic, vertex, claude-code, codex.\n' + + ' claude-code, codex — use your local subscription auth (no API key)\n' + + ' anthropic — also pass --anthropic-api-key-env or --anthropic-api-key-file\n' + + ' vertex — also pass --vertex-project (and optionally --vertex-location)\n', + ); + return { status: 'missing-input' }; } const prompts = deps.prompts ?? createPromptAdapter(); diff --git a/packages/cli/test/context/connections/local-warehouse-descriptor.test.ts b/packages/cli/test/context/connections/local-warehouse-descriptor.test.ts index e0a285a97..bf218bb89 100644 --- a/packages/cli/test/context/connections/local-warehouse-descriptor.test.ts +++ b/packages/cli/test/context/connections/local-warehouse-descriptor.test.ts @@ -1,10 +1,32 @@ import { describe, expect, it } from 'vitest'; import { + isExecuteOnlyConnection, + isScanTargetWarehouse, localConnectionInfoFromConfig, localConnectionToWarehouseDescriptor, localConnectionTypeForConfig, } from '../../../src/context/connections/local-warehouse-descriptor.js'; +describe('execute-only warehouse connections', () => { + it('treats a warehouse without scan_enabled as a scan target', () => { + const connection = { driver: 'postgres', url: 'postgresql://db/a' } as const; + expect(isExecuteOnlyConnection(connection)).toBe(false); + expect(isScanTargetWarehouse('w', connection)).toBe(true); + }); + + it('excludes a warehouse with scan_enabled: false from scan targets but still resolves it as a warehouse', () => { + const connection = { driver: 'postgres', url: 'postgresql://db/a', scan_enabled: false } as const; + expect(isExecuteOnlyConnection(connection)).toBe(true); + expect(isScanTargetWarehouse('w', connection)).toBe(false); + // Execution paths must still see it as a warehouse so `ktx sql` works. + expect(localConnectionToWarehouseDescriptor('w', connection)).not.toBeNull(); + }); + + it('does not treat non-warehouse connections as scan targets', () => { + expect(isScanTargetWarehouse('n', { driver: 'notion', auth_token: 'x' } as never)).toBe(false); + }); +}); + describe('localConnectionToWarehouseDescriptor', () => { it('maps local Postgres URLs to canonical warehouse descriptors', () => { expect( diff --git a/packages/cli/test/context/core/git.service.test.ts b/packages/cli/test/context/core/git.service.test.ts index 8b19ed09b..89b2f080e 100644 --- a/packages/cli/test/context/core/git.service.test.ts +++ b/packages/cli/test/context/core/git.service.test.ts @@ -1,8 +1,9 @@ -import { mkdtemp, readFile, realpath, rm, writeFile } from 'node:fs/promises'; +import { mkdir, mkdtemp, readFile, realpath, rm, stat, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { afterEach, beforeEach, describe, expect, it } from 'vitest'; import type { KtxCoreConfig } from '../../../src/context/core/config.js'; +import { createSimpleGit } from '../../../src/context/core/git-env.js'; import { GitService } from '../../../src/context/core/git.service.js'; // These tests drive a real git repo inside a temp directory — simple-git shells out to the @@ -318,6 +319,238 @@ describe('GitService', () => { }); }); + describe('nested inside an enclosing git repository', () => { + let enclosingRoot: string; + let nestedDir: string; + let nested: GitService; + + const makeConfig = (dir: string): KtxCoreConfig => ({ + storage: { configDir: dir, homeDir: dir }, + git: { + userName: 'Test User', + userEmail: 'test@example.com', + bootstrapMessage: 'Initialize test config repo', + bootstrapAuthor: 'test-system', + bootstrapAuthorEmail: 'system@example.com', + }, + }); + + beforeEach(async () => { + // An enclosing repo, like a user's application repository. + enclosingRoot = await mkdtemp(join(tmpdir(), 'git-service-enclosing-')); + const enclosing = new GitService(makeConfig(enclosingRoot)); + await enclosing.onModuleInit(); + + // A ktx project living in a subdirectory of that repo. + nestedDir = join(enclosingRoot, 'subproj'); + await mkdir(nestedDir, { recursive: true }); + nested = new GitService(makeConfig(nestedDir)); + await nested.onModuleInit(); + }); + + afterEach(async () => { + await rm(enclosingRoot, { recursive: true, force: true }); + }); + + it('initializes a dedicated repo at the config dir rather than using the enclosing repo', async () => { + const hasOwnGitDir = await stat(join(nestedDir, '.git')) + .then(() => true) + .catch(() => false); + expect(hasOwnGitDir).toBe(true); + + const toplevel = (await createSimpleGit(nestedDir).revparse(['--show-toplevel'])).trim(); + expect(await realpath(toplevel)).toBe(await realpath(nestedDir)); + }); + + it('lands worktree squash-merges under the config dir, not the enclosing root', async () => { + // Seed so HEAD exists, then ingest a wiki page through the worktree+squash path + // exactly like memory/wiki ingest does. + await writeFile(join(nestedDir, 'seed.md'), 'seed', 'utf-8'); + const { commitHash: baseSha } = await nested.commitFile('seed.md', 'seed', 'Test', 'test@example.com'); + + const wtParent = await realpath(join(enclosingRoot, '..')); + const wtDir = join(wtParent, `wt-${Date.now()}-nested`); + await nested.addWorktree(wtDir, 'session/wiki', baseSha); + const scoped = nested.forWorktree(wtDir); + await mkdir(join(wtDir, 'wiki', 'global'), { recursive: true }); + await writeFile(join(wtDir, 'wiki', 'global', 'page.md'), '# Page\n', 'utf-8'); + await scoped.commitFile('wiki/global/page.md', 'wip wiki', 'System User', 'system@example.com'); + + const result = await nested.squashMergeIntoMain( + 'session/wiki', + 'System User', + 'system@example.com', + 'Memory ingest (external_ingest): 1 wiki [chat=test1234]', + ); + expect(result.ok).toBe(true); + + // The page must materialize where reindex scans (under the project dir), + // not one level up at the enclosing repo root. + const underNested = await stat(join(nestedDir, 'wiki', 'global', 'page.md')) + .then(() => true) + .catch(() => false); + const underEnclosing = await stat(join(enclosingRoot, 'wiki', 'global', 'page.md')) + .then(() => true) + .catch(() => false); + expect(underNested).toBe(true); + expect(underEnclosing).toBe(false); + + await nested.removeWorktree(wtDir).catch(() => undefined); + await rm(wtDir, { recursive: true, force: true }).catch(() => undefined); + }); + }); + + describe('stageSquashMergeIntoMain', () => { + it('applies the branch to main without committing, leaving the changes staged', async () => { + const { commitHash: baseSha } = await writeAndCommit('seed.md', 'seed'); + const parent = await realpath(join(tempDir, '..')); + const wtDir = join(parent, `wt-${Date.now()}-stage`); + await service.addWorktree(wtDir, 'session/stage', baseSha); + + const scoped = service.forWorktree(wtDir); + await writeFile(join(wtDir, 'a.yaml'), 'one: 1\n', 'utf-8'); + await scoped.commitFile('a.yaml', 'wip a', 'System User', 'system@example.com'); + + const result = await service.stageSquashMergeIntoMain('session/stage'); + expect(result.ok).toBe(true); + if (!result.ok) { + throw new Error('unreachable'); + } + expect(result.touchedPaths).toEqual(['a.yaml']); + expect(result.stagedTree).toMatch(/^[0-9a-f]{40}$/); + + // HEAD did not advance: no commit was created. + expect(await service.revParseHead()).toBe(baseSha); + // The change is in main's working tree... + await expect(readFile(join(tempDir, 'a.yaml'), 'utf-8')).resolves.toBe('one: 1\n'); + // ...and staged in the index for the user to commit. + const stagedNames = await createSimpleGit(tempDir).raw(['diff', '--cached', '--name-only']); + expect(stagedNames).toContain('a.yaml'); + // The staged tree is usable as a diff/read ref for DB sync. + const treeListing = await createSimpleGit(tempDir).raw(['ls-tree', '-r', '--name-only', result.stagedTree]); + expect(treeListing).toContain('a.yaml'); + + await service.removeWorktree(wtDir).catch(() => undefined); + await rm(wtDir, { recursive: true, force: true }).catch(() => undefined); + }); + + it('reports conflicts without committing or mutating main', async () => { + const { commitHash: baseSha } = await writeAndCommit('conflict.md', 'base\n'); + const parent = await realpath(join(tempDir, '..')); + const wtDir = join(parent, `wt-${Date.now()}-stage-conflict`); + await service.addWorktree(wtDir, 'session/stage-conflict', baseSha); + const scoped = service.forWorktree(wtDir); + await writeFile(join(wtDir, 'conflict.md'), 'from-branch\n', 'utf-8'); + await scoped.commitFile('conflict.md', 'branch edit', 'System User', 'system@example.com'); + + // Move main ahead with a conflicting change. + await writeAndCommit('conflict.md', 'from-main\n'); + const mainHead = await service.revParseHead(); + + const result = await service.stageSquashMergeIntoMain('session/stage-conflict'); + expect(result.ok).toBe(false); + if (result.ok || !('conflict' in result)) { + throw new Error('expected a conflict'); + } + expect(result.conflictPaths).toContain('conflict.md'); + expect(await service.revParseHead()).toBe(mainHead); + + await service.removeWorktree(wtDir).catch(() => undefined); + await rm(wtDir, { recursive: true, force: true }).catch(() => undefined); + }); + }); + + describe('refuses to squash into a dirty main worktree', () => { + it('reports dirty without committing, merging, or discarding pre-existing staged changes', async () => { + const { commitHash: baseSha } = await writeAndCommit('seed.md', 'seed'); + const parent = await realpath(join(tempDir, '..')); + const wtDir = join(parent, `wt-${Date.now()}-dirty`); + await service.addWorktree(wtDir, 'session/dirty', baseSha); + const scoped = service.forWorktree(wtDir); + await writeFile(join(wtDir, 'from-branch.yaml'), 'b: 1\n', 'utf-8'); + await scoped.commitFile('from-branch.yaml', 'wip', 'System User', 'system@example.com'); + + // Residue from a prior auto_commit:false run: a staged change on main. + await writeFile(join(tempDir, 'pending.md'), 'pending work\n', 'utf-8'); + await createSimpleGit(tempDir).add('pending.md'); + + const result = await service.squashMergeIntoMain('session/dirty', 'System User', 'system@example.com', 'msg'); + + expect(result.ok).toBe(false); + if (result.ok || !('dirty' in result)) { + throw new Error('expected a dirty refusal'); + } + expect(result.dirtyPaths).toContain('pending.md'); + + // Main is untouched: HEAD unchanged, branch not merged, pending change preserved. + expect(await service.revParseHead()).toBe(baseSha); + await expect(readFile(join(tempDir, 'pending.md'), 'utf-8')).resolves.toBe('pending work\n'); + const staged = await createSimpleGit(tempDir).raw(['diff', '--cached', '--name-only']); + expect(staged).toContain('pending.md'); + const branchFileLanded = await stat(join(tempDir, 'from-branch.yaml')) + .then(() => true) + .catch(() => false); + expect(branchFileLanded).toBe(false); + + await service.removeWorktree(wtDir).catch(() => undefined); + await rm(wtDir, { recursive: true, force: true }).catch(() => undefined); + }); + + it('allows the squash when main has only unstaged (not staged) changes', async () => { + // e.g. setup writes ktx.yaml during the flow and commits it only after the context + // build, leaving it modified-but-unstaged. `git commit` never captures unstaged changes, + // so the squash must proceed and leave them untouched. + const { commitHash: baseSha } = await writeAndCommit('config.yaml', 'a: 1\n'); + const parent = await realpath(join(tempDir, '..')); + const wtDir = join(parent, `wt-${Date.now()}-unstaged`); + await service.addWorktree(wtDir, 'session/unstaged', baseSha); + const scoped = service.forWorktree(wtDir); + await writeFile(join(wtDir, 'from-branch.yaml'), 'b: 1\n', 'utf-8'); + await scoped.commitFile('from-branch.yaml', 'wip', 'System User', 'system@example.com'); + + // Modify a tracked file on main WITHOUT staging it. + await writeFile(join(tempDir, 'config.yaml'), 'a: 2\n', 'utf-8'); + + const result = await service.squashMergeIntoMain('session/unstaged', 'System User', 'system@example.com', 'msg'); + + expect(result.ok).toBe(true); + if (!result.ok || !('squashSha' in result)) { + throw new Error('expected the squash to land'); + } + expect(result.touchedPaths).toEqual(['from-branch.yaml']); + // The branch landed, and the unstaged edit was neither committed nor discarded. + await expect(readFile(join(tempDir, 'from-branch.yaml'), 'utf-8')).resolves.toBe('b: 1\n'); + await expect(readFile(join(tempDir, 'config.yaml'), 'utf-8')).resolves.toBe('a: 2\n'); + + await service.removeWorktree(wtDir).catch(() => undefined); + await rm(wtDir, { recursive: true, force: true }).catch(() => undefined); + }); + + it('stageSquashMergeIntoMain also refuses on a dirty main', async () => { + const { commitHash: baseSha } = await writeAndCommit('seed.md', 'seed'); + const parent = await realpath(join(tempDir, '..')); + const wtDir = join(parent, `wt-${Date.now()}-dirty-stage`); + await service.addWorktree(wtDir, 'session/dirty-stage', baseSha); + const scoped = service.forWorktree(wtDir); + await writeFile(join(wtDir, 'from-branch.yaml'), 'b: 1\n', 'utf-8'); + await scoped.commitFile('from-branch.yaml', 'wip', 'System User', 'system@example.com'); + + await writeFile(join(tempDir, 'pending.md'), 'pending work\n', 'utf-8'); + await createSimpleGit(tempDir).add('pending.md'); + + const result = await service.stageSquashMergeIntoMain('session/dirty-stage'); + expect(result.ok).toBe(false); + if (result.ok || !('dirty' in result)) { + throw new Error('expected a dirty refusal'); + } + expect(result.dirtyPaths).toContain('pending.md'); + expect(await service.revParseHead()).toBe(baseSha); + + await service.removeWorktree(wtDir).catch(() => undefined); + await rm(wtDir, { recursive: true, force: true }).catch(() => undefined); + }); + }); + describe('squashMergeIntoMain', () => { it('merges a session branch as one commit on main, returning the new SHA + touched paths', async () => { const { commitHash: baseSha } = await writeAndCommit('seed.md', 'seed'); @@ -402,8 +635,8 @@ describe('GitService', () => { ); expect(result.ok).toBe(false); - if (result.ok) { - throw new Error('unreachable'); + if (result.ok || !('conflict' in result)) { + throw new Error('expected a conflict'); } expect(result.conflict).toBe(true); expect(result.conflictPaths).toContain('shared.yaml'); @@ -434,8 +667,8 @@ describe('GitService', () => { ); expect(result.ok).toBe(false); - if (result.ok) { - throw new Error('unreachable'); + if (result.ok || !('conflict' in result)) { + throw new Error('expected a conflict'); } expect(result.conflict).toBe(true); expect(result.conflictPaths).toEqual(['knowledge.md']); diff --git a/packages/cli/test/context/ingest/ingest-bundle.runner.isolated-diff.test.ts b/packages/cli/test/context/ingest/ingest-bundle.runner.isolated-diff.test.ts index bad400981..b4505c4b0 100644 --- a/packages/cli/test/context/ingest/ingest-bundle.runner.isolated-diff.test.ts +++ b/packages/cli/test/context/ingest/ingest-bundle.runner.isolated-diff.test.ts @@ -199,6 +199,7 @@ function makeDeps( storage: { homeDir: join(runtime.configDir, '.ktx'), systemGitAuthor: { name: 'KTX Test', email: 'system@ktx.local' }, + autoCommit: true, resolveUploadDir: (id) => join(runtime.homeDir, 'upload', id), resolvePullDir: (id) => join(runtime.homeDir, 'pull', id), resolveTranscriptDir: (id) => join(runtime.configDir, '.ktx/ingest-transcripts', id), diff --git a/packages/cli/test/context/ingest/ingest-bundle.runner.test.ts b/packages/cli/test/context/ingest/ingest-bundle.runner.test.ts index 688147927..f22c53bc1 100644 --- a/packages/cli/test/context/ingest/ingest-bundle.runner.test.ts +++ b/packages/cli/test/context/ingest/ingest-bundle.runner.test.ts @@ -262,6 +262,7 @@ const buildRunner = (deps: ReturnType = makeDeps(), overrides: storage: { homeDir: '/tmp/ktx-test', systemGitAuthor: { name: 'KTX Test', email: 'system@ktx.local' }, + autoCommit: true, resolveUploadDir: (uploadId) => `/tmp/ktx-test/ingest-uploads/${uploadId}`, resolvePullDir: (jobId) => `/tmp/ktx-test/ingest-pulls/${jobId}`, resolveTranscriptDir: (jobId) => `/tmp/ktx-test/run/wu-transcripts/${jobId}`, @@ -1519,6 +1520,7 @@ describe('IngestBundleRunner — Stages 1 → 7', () => { storage: { homeDir: tempRoot, systemGitAuthor: { name: 'KTX Test', email: 'system@ktx.local' }, + autoCommit: true, resolveUploadDir: (uploadId: string) => join(tempRoot, 'ingest-uploads', uploadId), resolvePullDir: (jobId: string) => join(tempRoot, 'ingest-pulls', jobId), resolveTranscriptDir: (jobId: string) => join(tempRoot, 'run', 'wu-transcripts', jobId), diff --git a/packages/cli/test/context/ingest/local-adapters.test.ts b/packages/cli/test/context/ingest/local-adapters.test.ts index a8799cee5..c0834676c 100644 --- a/packages/cli/test/context/ingest/local-adapters.test.ts +++ b/packages/cli/test/context/ingest/local-adapters.test.ts @@ -634,6 +634,21 @@ describe('local ingest adapters', () => { await expect(adapter?.listTargetConnectionIds?.('/tmp/staged-dbt')).resolves.toEqual(['warehouse']); }); + it('excludes execute-only (scan_enabled: false) warehouses from primary scan targets', async () => { + const adapters = createDefaultLocalIngestAdapters( + projectWithConnections({ + scannable: { driver: 'postgres', url: 'postgresql://db/a' }, + executeonly: { driver: 'postgres', url: 'postgresql://db/b', scan_enabled: false }, + docs: { driver: 'dbt', source_dir: './dbt' }, + } as never), + ); + + // No setup.database_connection_ids → falls back to "all warehouses", which must now + // skip the execute-only connection rather than re-including it. + const dbt = adapters.find((adapter) => adapter.source === 'dbt'); + await expect(dbt?.listTargetConnectionIds?.('/tmp/staged-dbt')).resolves.toEqual(['scannable']); + }); + it('passes primary warehouse connection ids to the local Notion adapter', async () => { const adapters = createDefaultLocalIngestAdapters( projectWithConnections({ diff --git a/packages/cli/test/context/ingest/local-bundle-ingest.test.ts b/packages/cli/test/context/ingest/local-bundle-ingest.test.ts index 6140dd104..1d8819e2c 100644 --- a/packages/cli/test/context/ingest/local-bundle-ingest.test.ts +++ b/packages/cli/test/context/ingest/local-bundle-ingest.test.ts @@ -424,6 +424,60 @@ describe('canonical local ingest', () => { } }); + it('with auto_commit disabled, stages ingest changes and indexes the wiki without committing', async () => { + const projectDir = join(tempDir, 'no-autocommit-project'); + await initKtxProject({ projectDir }); + await writeFile( + join(projectDir, 'ktx.yaml'), + [ + 'connections:', + ' warehouse:', + ' driver: postgres', + 'ingest:', + ' adapters:', + ' - fake', + ' embeddings:', + ' backend: none', + 'storage:', + ' git:', + ' auto_commit: false', + '', + ].join('\n'), + 'utf-8', + ); + const stagedProject = await loadKtxProject({ projectDir }); + const preHead = await stagedProject.git.revParseHead(); + + const sourceDir = join(tempDir, 'no-autocommit-source'); + await mkdir(join(sourceDir, 'orders'), { recursive: true }); + await writeFile(join(sourceDir, 'orders', 'orders.json'), '{"name":"orders"}\n', 'utf-8'); + + const result = await runLocalIngest({ + project: stagedProject, + adapters: [new FakeSourceAdapter()], + adapter: 'fake', + connectionId: 'warehouse', + sourceDir, + jobId: 'wiki-staged-1', + agentRunner: new WikiWritingAgentRunner(), + }); + + expect(result.result.failedWorkUnits).toEqual([]); + // No commit was created: HEAD is unchanged. + expect(await stagedProject.git.revParseHead()).toBe(preHead); + // ...yet the wiki page is on disk (staged) and indexed for search, reconciled from the + // staged tree rather than a commit. + await expect(readFile(join(projectDir, 'wiki', 'global', 'orders_context.md'), 'utf-8')).resolves.toContain('Orders'); + const db = new Database(join(projectDir, '.ktx', 'db.sqlite'), { readonly: true }); + try { + expect(db.prepare('SELECT key, summary FROM knowledge_pages ORDER BY key').all()).toEqual([ + { key: 'orders_context', summary: 'Orders source context' }, + ]); + } finally { + db.close(); + } + }); + it('does not persist noop embedding vectors when local embeddings are disabled', async () => { await writeFile( join(project.projectDir, 'ktx.yaml'), diff --git a/packages/cli/test/context/memory/memory-agent.service.ingest.test.ts b/packages/cli/test/context/memory/memory-agent.service.ingest.test.ts index acb1c2f8d..72878ebe9 100644 --- a/packages/cli/test/context/memory/memory-agent.service.ingest.test.ts +++ b/packages/cli/test/context/memory/memory-agent.service.ingest.test.ts @@ -40,6 +40,7 @@ interface BuiltMocks { slValidator: any; toolsetFactory: any; logger: any; + autoCommit: boolean; } const buildMocks = (overrides: Partial = {}): BuiltMocks => { @@ -111,6 +112,9 @@ const buildMocks = (overrides: Partial = {}): BuiltMocks => { gitService: { revParseHead: vi.fn().mockResolvedValue('basesha'), squashMergeIntoMain: vi.fn().mockResolvedValue({ ok: true, squashSha: 'cafebabe', touchedPaths: ['a.yaml'] }), + stageSquashMergeIntoMain: vi + .fn() + .mockResolvedValue({ ok: true, touchedPaths: ['a.yaml'], stagedTree: 'deadbeeftree' }), }, lockingService: { withLock: vi.fn().mockImplementation((_key: string, fn: () => Promise) => fn()), @@ -134,6 +138,7 @@ const buildMocks = (overrides: Partial = {}): BuiltMocks => { }), }, logger: { log: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() }, + autoCommit: true, }; return { ...defaults, ...overrides }; @@ -151,6 +156,7 @@ const buildService = (mocks: BuiltMocks): MemoryAgentService => llm: { memoryIngestionModel: mocks.appSettings.settings.llm.memoryIngestionModel, }, + autoCommit: mocks.autoCommit, }, promptService: mocks.prompt, skillsRegistry: mocks.skillsRegistry, @@ -242,6 +248,26 @@ describe('MemoryAgentService.ingest — session-branch orchestration', () => { expect(result.commitHash).toBe('cafebabe'); }); + it('with auto_commit disabled, stages the session on main without committing or enqueuing a note', async () => { + const mocks = buildMocks({ autoCommit: false }); + const svc = buildService(mocks); + + const result = await svc.ingest(baseInput); + + // Applied to main via the staging path, never the committing path. + expect(mocks.gitService.stageSquashMergeIntoMain).toHaveBeenCalledWith('session/chat-1'); + expect(mocks.gitService.squashMergeIntoMain).not.toHaveBeenCalled(); + // No commit means no commit-message enhancement job. + expect(mocks.configService.enqueueCommitMessageJobForExternalCommit).not.toHaveBeenCalled(); + // The session still applied successfully; there is just no commit hash. + expect(result.commitHash).toBeNull(); + expect(mocks.sessionWorktreeService.cleanup).toHaveBeenCalledWith( + expect.objectContaining({ chatId: 'chat-1' }), + 'success', + expect.any(Object), + ); + }); + it('normalizes load_skill output to markdown while preserving structured payload', async () => { const tempDir = await mkdtemp(join(tmpdir(), 'ktx-memory-skill-')); const skillDir = join(tempDir, 'memory_agent'); @@ -339,6 +365,24 @@ describe('MemoryAgentService.ingest — session-branch orchestration', () => { expect(result.actions).toEqual([]); }); + it('dirty-target path: rolls back DB and does not land when main has uncommitted changes', async () => { + const mocks = buildMocks(); + mocks.gitService.squashMergeIntoMain.mockResolvedValue({ + ok: false, + dirty: true, + dirtyPaths: ['pending.md'], + }); + const svc = buildService(mocks); + + const result = await svc.ingest(baseInput); + + // Treated as a not-landed abort: rolled back, no commit, no message-enhancement job. + expect(mocks.sessionWorktreeService.cleanup).toHaveBeenCalledWith(expect.any(Object), 'conflict', expect.any(Object)); + expect(mocks.configService.enqueueCommitMessageJobForExternalCommit).not.toHaveBeenCalled(); + expect(result.commitHash).toBeNull(); + expect(result.actions).toEqual([]); + }); + it('crash path: post-loop step throws → cleanup(crash), commitHash=null', async () => { const mocks = buildMocks(); // Force the cross-ref reconciler to throw, escaping into the outer try/catch and diff --git a/packages/cli/test/context/memory/memory-agent.service.test.ts b/packages/cli/test/context/memory/memory-agent.service.test.ts index ba91444ec..8e44f5fa3 100644 --- a/packages/cli/test/context/memory/memory-agent.service.test.ts +++ b/packages/cli/test/context/memory/memory-agent.service.test.ts @@ -193,6 +193,7 @@ describe('MemoryAgentService.reconcileCrossRefs', () => { knowledge: { userScopedKnowledgeEnabled: false }, slValidation: { probeRowCount: 1 }, llm: { memoryIngestionModel: 'test-model' }, + autoCommit: true, }, promptService: undefined as never, skillsRegistry: undefined as never, @@ -369,6 +370,7 @@ describe('MemoryAgentService.gateRevertInvalidSources (J3)', () => { knowledge: { userScopedKnowledgeEnabled: false }, slValidation: { probeRowCount: 1 }, llm: { memoryIngestionModel: 'test-model' }, + autoCommit: true, }, promptService: undefined as never, skillsRegistry: undefined as never, diff --git a/packages/cli/test/context/project/config.test.ts b/packages/cli/test/context/project/config.test.ts index e5911a250..99912ffd3 100644 --- a/packages/cli/test/context/project/config.test.ts +++ b/packages/cli/test/context/project/config.test.ts @@ -129,6 +129,18 @@ connections: expect(serialized).not.toContain('completed_steps:'); }); + it('parses and serializes a warehouse connection marked execute-only (scan_enabled: false)', () => { + const config = parseKtxProjectConfig(` +connections: + public_bq: + driver: bigquery + scan_enabled: false +`); + + expect(config.connections.public_bq).toMatchObject({ driver: 'bigquery', scan_enabled: false }); + expect(serializeKtxProjectConfig(config)).toContain('scan_enabled: false'); + }); + it('parses global direct Anthropic LLM config', () => { const config = parseKtxProjectConfig(` llm: diff --git a/packages/cli/test/context/project/project.test.ts b/packages/cli/test/context/project/project.test.ts index 668fa264c..8cd49337c 100644 --- a/packages/cli/test/context/project/project.test.ts +++ b/packages/cli/test/context/project/project.test.ts @@ -1,7 +1,8 @@ -import { mkdtemp, readFile, rm, stat } from 'node:fs/promises'; +import { mkdtemp, readFile, realpath, rm, stat } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { createSimpleGit } from '../../../src/context/core/git-env.js'; import { initKtxProject, loadKtxProject } from '../../../src/context/project/project.js'; describe('KTX local project runtime', () => { @@ -42,6 +43,24 @@ describe('KTX local project runtime', () => { await expect(stat(join(projectDir, '.git'))).resolves.toBeDefined(); }); + it('creates its own git repo when the project dir is nested inside another repository', async () => { + // An enclosing repository, like a user's application repo. + const enclosing = createSimpleGit(tempDir); + await enclosing.init(); + await enclosing.addConfig('user.name', 'Enclosing'); + await enclosing.addConfig('user.email', 'enclosing@example.com'); + await enclosing.commit('enclosing root', { '--allow-empty': null }); + + const projectDir = join(tempDir, 'subproj'); + await initKtxProject({ projectDir, authorName: 'Agent', authorEmail: 'agent@example.com' }); + + // ktx must own a dedicated repo at the project dir rather than committing into the + // enclosing repo, so writes and reindex scans share one working-tree root. + await expect(stat(join(projectDir, '.git'))).resolves.toBeDefined(); + const toplevel = (await createSimpleGit(projectDir).revparse(['--show-toplevel'])).trim(); + expect(await realpath(toplevel)).toBe(await realpath(projectDir)); + }); + it('loads an initialized project with a working file store', async () => { const projectDir = join(tempDir, 'warehouse'); await initKtxProject({ projectDir }); diff --git a/packages/cli/test/index.test.ts b/packages/cli/test/index.test.ts index eef511fb0..c0c285ca7 100644 --- a/packages/cli/test/index.test.ts +++ b/packages/cli/test/index.test.ts @@ -1522,7 +1522,8 @@ describe('runKtxCli', () => { ).resolves.toBe(1); expect(setup).not.toHaveBeenCalled(); - expect(setupIo.stderr()).toContain("invalid choice 'deterministic'"); + expect(setupIo.stderr()).toContain("argument 'deterministic' is invalid"); + expect(setupIo.stderr()).toContain('Allowed choices are openai, sentence-transformers'); }); it('rejects gateway as a setup embedding backend', async () => { @@ -1534,7 +1535,8 @@ describe('runKtxCli', () => { ).resolves.toBe(1); expect(setup).not.toHaveBeenCalled(); - expect(setupIo.stderr()).toContain("invalid choice 'gateway'"); + expect(setupIo.stderr()).toContain("argument 'gateway' is invalid"); + expect(setupIo.stderr()).toContain('Allowed choices are openai, sentence-transformers'); }); it('rejects conflicting embedding credential setup flags', async () => { diff --git a/packages/cli/test/ingest.test.ts b/packages/cli/test/ingest.test.ts index c1abfe8b5..e2e965d3b 100644 --- a/packages/cli/test/ingest.test.ts +++ b/packages/cli/test/ingest.test.ts @@ -54,6 +54,40 @@ describe('runKtxIngest', () => { await rm(tempDir, { recursive: true, force: true }); }); + it('refuses to ingest a connection marked execute-only (scan_enabled: false)', async () => { + const projectDir = join(tempDir, 'project'); + await initKtxProject({ projectDir }); + await writeFile( + join(projectDir, 'ktx.yaml'), + [ + 'connections:', + ' public_bq:', + ' driver: bigquery', + ' scan_enabled: false', + 'ingest:', + ' adapters:', + ' - fake', + ' embeddings:', + ' backend: none', + '', + ].join('\n'), + 'utf-8', + ); + const runLocal = vi.fn(); + const io = makeIo(); + + await expect( + runKtxIngest( + { command: 'run', projectDir, connectionId: 'public_bq', adapter: 'fake', outputMode: 'plain' }, + io.io, + { runLocalIngest: runLocal }, + ), + ).resolves.toBe(1); + + expect(runLocal).not.toHaveBeenCalled(); + expect(io.stderr()).toContain('scan_enabled: false'); + }); + it('runs local ingest and reads status', async () => { const projectDir = join(tempDir, 'project'); await writeWarehouseConfig(projectDir); diff --git a/packages/cli/test/scan.test.ts b/packages/cli/test/scan.test.ts index 51c55498b..2396dc7b3 100644 --- a/packages/cli/test/scan.test.ts +++ b/packages/cli/test/scan.test.ts @@ -332,6 +332,35 @@ describe('runKtxScan', () => { await rm(tempDir, { recursive: true, force: true }); }); + it('refuses to scan a connection marked execute-only (scan_enabled: false)', async () => { + await initKtxProject({ projectDir: tempDir }); + await writeFile( + join(tempDir, 'ktx.yaml'), + ['connections:', ' public_bq:', ' driver: bigquery', ' scan_enabled: false', ''].join('\n'), + 'utf-8', + ); + const runLocalScan = vi.fn(); + const io = makeIo(); + + await expect( + runKtxScan( + { + command: 'run', + projectDir: tempDir, + connectionId: 'public_bq', + mode: 'structural', + detectRelationships: false, + dryRun: false, + }, + io.io, + { runLocalScan, createLocalIngestAdapters: noLocalIngestAdapters }, + ), + ).resolves.toBe(1); + + expect(runLocalScan).not.toHaveBeenCalled(); + expect(io.stderr()).toContain('scan_enabled: false'); + }); + it('runs structural scans and prints a dev-friendly plain summary', async () => { await initKtxProject({ projectDir: tempDir }); const runLocalScan = vi.fn( diff --git a/packages/cli/test/setup-databases.test.ts b/packages/cli/test/setup-databases.test.ts index 957dfdb2f..f3029021c 100644 --- a/packages/cli/test/setup-databases.test.ts +++ b/packages/cli/test/setup-databases.test.ts @@ -1586,6 +1586,64 @@ describe('setup databases step', () => { }); }); + it('registers an execute-only connection (scan_enabled: false) without scanning it', async () => { + await writeFile( + join(tempDir, 'ktx.yaml'), + ['connections:', ' public_bq:', ' driver: bigquery', ' scan_enabled: false', ''].join('\n'), + 'utf-8', + ); + const io = makeIo(); + const testConnection = vi.fn(async () => 0); + const scanConnection = vi.fn(async () => 0); + + const result = await runKtxSetupDatabasesStep( + { + projectDir: tempDir, + inputMode: 'disabled', + databaseConnectionIds: ['public_bq'], + databaseSchemas: [], + skipDatabases: false, + }, + io.io, + { testConnection, scanConnection, listSchemas: vi.fn(async () => ['a', 'b', 'c']) }, + ); + + expect(result.status).toBe('ready'); + // The credential is validated, but the warehouse is never introspected/scanned. + expect(testConnection).toHaveBeenCalledWith(tempDir, 'public_bq', expect.anything()); + expect(scanConnection).not.toHaveBeenCalled(); + }); + + it('warns instead of silently scanning every discovered dataset when scripted setup has no scope', async () => { + await writeFile( + join(tempDir, 'ktx.yaml'), + ['connections:', ' warehouse:', ' driver: bigquery', ''].join('\n'), + 'utf-8', + ); + const io = makeIo(); + + const result = await runKtxSetupDatabasesStep( + { + projectDir: tempDir, + inputMode: 'disabled', + databaseConnectionIds: ['warehouse'], + databaseSchemas: [], + skipDatabases: false, + }, + io.io, + { + testConnection: vi.fn(async () => 0), + scanConnection: vi.fn(async () => 0), + listSchemas: vi.fn(async () => ['stripe', 'posthog', 'linear']), + listTables: vi.fn(async () => []), + }, + ); + + expect(result.status).toBe('ready'); + expect(io.stderr()).toContain('No --database-schema given for warehouse'); + expect(io.stderr()).toContain('scan_enabled: false'); + }); + it('keeps scripted database ids fail-fast even when input mode is auto', async () => { await writeFile( join(tempDir, 'ktx.yaml'), diff --git a/packages/cli/test/setup-models.test.ts b/packages/cli/test/setup-models.test.ts index f09691e0f..d79287a79 100644 --- a/packages/cli/test/setup-models.test.ts +++ b/packages/cli/test/setup-models.test.ts @@ -814,7 +814,7 @@ describe('setup Anthropic model step', () => { expect(io.stderr()).toContain(`Missing Anthropic API key file: ${missingSecretPath}`); }); - it('does not recommend skipping when non-interactive setup is missing an Anthropic credential source', async () => { + it('requires an explicit LLM backend in non-interactive setup instead of silently defaulting to anthropic', async () => { const io = makeIo(); const result = await runKtxSetupAnthropicModelStep( @@ -823,10 +823,33 @@ describe('setup Anthropic model step', () => { ); expect(result.status).toBe('missing-input'); - expect(io.stderr()).toContain( - 'Missing Anthropic API key: pass --anthropic-api-key-env or --anthropic-api-key-file.', + const err = io.stderr(); + // Names the flag the user actually needs and lists every valid backend. + expect(err).toContain('--llm-backend'); + expect(err).toContain('anthropic'); + expect(err).toContain('vertex'); + expect(err).toContain('claude-code'); + expect(err).toContain('codex'); + // No silent anthropic default, so the bare api-key error is not the reason shown. + expect(err).not.toContain('Missing Anthropic API key: pass --anthropic-api-key-env'); + expect(err).not.toContain('--skip-llm'); + }); + + it('names --llm-backend when an explicit anthropic backend is missing its API key in non-interactive setup', async () => { + const io = makeIo(); + + const result = await runKtxSetupAnthropicModelStep( + { projectDir: tempDir, inputMode: 'disabled', llmBackend: 'anthropic', skipLlm: false }, + io.io, ); - expect(io.stderr()).not.toContain('--skip-llm'); + + expect(result.status).toBe('missing-input'); + const err = io.stderr(); + expect(err).toContain('--anthropic-api-key-env'); + // Reveals that the backend itself is selectable, so a user who wanted a keyless + // backend (claude-code/codex) can discover it from the error. + expect(err).toContain('--llm-backend'); + expect(err).not.toContain('--skip-llm'); }); it('writes pasted keys to .ktx/secrets and never prints the key', async () => { diff --git a/uv.lock b/uv.lock index 40553e46e..1f35cc3e9 100644 --- a/uv.lock +++ b/uv.lock @@ -466,7 +466,7 @@ wheels = [ [[package]] name = "ktx-daemon" -version = "0.9.0" +version = "0.10.0" source = { editable = "python/ktx-daemon" } dependencies = [ { name = "fastapi" }, @@ -523,7 +523,7 @@ dev = [ [[package]] name = "ktx-sl" -version = "0.9.0" +version = "0.10.0" source = { editable = "python/ktx-sl" } dependencies = [ { name = "pydantic" },