Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ upkeep and don't absorb the rest of your company's knowledge.

Works with PostgreSQL, Snowflake, BigQuery, ClickHouse, MySQL, SQL Server,
SQLite, DuckDB, Amazon Athena, and MongoDB. Integrates with dbt, MetricFlow,
LookML, Looker, Metabase, Sigma, Notion, and Google Drive.
LookML, Looker, Metabase, Sigma, Tableau, Notion, and Google Drive.

## Quick Start

Expand Down
80 changes: 78 additions & 2 deletions docs-site/content/docs/integrations/context-sources.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Context Sources
description: Ingest semantic context from dbt, MetricFlow, LookML, Metabase, Looker, Notion, Sigma, and Google Drive.
description: Ingest semantic context from dbt, MetricFlow, LookML, Metabase, Looker, Notion, Sigma, Tableau, and Google Drive.
---

Context sources feed your existing analytics tooling into **ktx**. During ingestion, **ktx** extracts metadata from each source and uses a reconciliation agent to reconcile it with your existing semantic layer and knowledge base - preserving accepted edits rather than overwriting.
Expand All @@ -27,7 +27,7 @@ LookML uses top-level `repoUrl`, and MetricFlow uses nested

| Field | Required | Description |
|-------|----------|-------------|
| `driver` | Yes | Source connector: `dbt`, `metricflow`, `lookml`, `metabase`, `looker`, `notion`, `sigma`, or `gdrive` |
| `driver` | Yes | Source connector: `dbt`, `metricflow`, `lookml`, `metabase`, `looker`, `notion`, `sigma`, `tableau`, or `gdrive` |
| `source_dir` | For local file sources | Absolute or project-relative source directory |
| `repo_url` | For Git-hosted dbt sources | Git repository URL |
| `repoUrl` | For Git-hosted LookML sources | Git repository URL |
Expand Down Expand Up @@ -473,6 +473,82 @@ connections:

---

## Tableau

Ingests published data sources, calculated fields, and workbook metadata from **Tableau Cloud** or **Tableau Server** as semantic context. Uses the Tableau REST API for authentication (Personal Access Token) and the Tableau **Metadata API** (GraphQL) to fetch data source definitions and lineage.

### What it provides

- Published data source names, projects, and descriptions
- Field definitions — dimensions and measures, with data types
- **Calculated fields** with their formulas (captured as candidate metric definitions)
- Upstream (physical) warehouse tables each data source draws from
- Workbook names, projects, and descriptions

### Connection config

```yaml title="ktx.yaml"
connections:
tableau-main:
driver: tableau
host: https://us-west-2b.online.tableau.com # Tableau Cloud pod, or your Server host
site_content_url: "your-site" # site subpath; "" for the Default site on Server
personal_access_token_name: "ktx-ingest"
personal_access_token_secret_ref: env:TABLEAU_PAT_SECRET
```

**Tableau Cloud** uses your pod hostname (e.g. `https://10ax.online.tableau.com`); **Tableau Server** uses your own host (e.g. `https://tableau.mycompany.com`). On Cloud a non-default `site_content_url` is almost always required. Override the REST API version with `api_version` if needed (defaults to `3.29`).

### Authentication

| Method | Config |
|--------|--------|
| Personal Access Token | `personal_access_token_name` + `personal_access_token_secret_ref: env:TABLEAU_PAT_SECRET` |

Create a PAT in Tableau: **My Account Settings → Personal Access Tokens → Create Token**. The secret is shown only once at creation. Provide it to ktx via an `env:` or `file:` reference — never inline the literal secret in `ktx.yaml`:

```yaml title="ktx.yaml"
personal_access_token_secret_ref: file:secrets/tableau-pat # or env:TABLEAU_PAT_SECRET
```

Don't have an instance to test with? Join the free [Tableau Developer Program](https://www.tableau.com/developer) for a personal Tableau Cloud sandbox (the Metadata API is enabled by default) and create a PAT there.

### What gets ingested

- Published data sources, organized into work units, each with its fields (including calculated-field formulas) and upstream tables
- Workbook metadata (name, project, description)
- Ingest is incremental: items whose `updatedAt` timestamp is unchanged since the last run are skipped

### Data source and workbook filters

At large scale, limit which items are fetched during ingest with `datasourceFilter` / `workbookFilter`:

```yaml title="ktx.yaml"
connections:
tableau-main:
driver: tableau
host: https://us-west-2b.online.tableau.com
site_content_url: "your-site"
personal_access_token_name: "ktx-ingest"
personal_access_token_secret_ref: env:TABLEAU_PAT_SECRET
datasourceFilter:
updatedSince: "2026-01-01T00:00:00Z" # only recently updated data sources
workbookFilter:
updatedSince: "2026-01-01T00:00:00Z" # only recently updated workbooks
```

| Field | Default | Description |
|-------|---------|-------------|
| `updatedSince` | — | ISO 8601 date; only items updated on or after this date are fetched |

### Notes

- This release is **wiki-only**: Tableau ingest produces ktx wiki knowledge (metric definitions from calculated fields, business concepts from data source and workbook names). Deterministic semantic-layer sources under `semantic-layer/<connection-id>/` are not written yet — warehouse-table projection is planned for a follow-up release.
- Context ingest (`ktx ingest tableau-main`) fetches from the Tableau APIs directly.
- The PAT secret must be provided via an `env:` or `file:` reference; the token authenticates both the REST sign-in and the Metadata API calls.

---

## Google Drive

Ingests Google Docs from a shared Google Drive folder as wiki-ready knowledge content. This v1 implementation is knowledge-only and ingests Google Docs MIME types only.
Expand Down
1 change: 1 addition & 0 deletions packages/cli/src/commands/setup-commands.ts
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ function sourceType(value: string): KtxSetupSourceType {
value === 'lookml' ||
value === 'notion' ||
value === 'sigma' ||
value === 'tableau' ||
value === 'gdrive'
) {
return value;
Expand Down
148 changes: 148 additions & 0 deletions packages/cli/src/context/ingest/adapters/tableau/chunk.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
import { readdir, readFile } from 'node:fs/promises';
import { join, relative } from 'node:path';
import type { ChunkResult, DiffSet, WorkUnit } from '../../types.js';
import {
type StagedDatasourceFile,
type StagedWorkbookFile,
type TableauManifest,
stagedDatasourceFileSchema,
stagedWorkbookFileSchema,
tableauManifestSchema,
STAGED_FILES,
} from './types.js';

interface LoadedBundle {
manifest: TableauManifest | null;
datasourcesByPath: Map<string, StagedDatasourceFile>;
workbooksByPath: Map<string, StagedWorkbookFile>;
allPaths: string[];
}

async function walkStagedDir(stagedDir: string): Promise<string[]> {
let entries;
try {
entries = await readdir(stagedDir, { withFileTypes: true, recursive: true });
} catch (err) {
if ((err as NodeJS.ErrnoException).code === 'ENOENT') return [];
throw err;
}
const paths: string[] = [];
for (const entry of entries) {
if (!entry.isFile()) continue;
const abs = join(entry.parentPath, entry.name);
paths.push(relative(stagedDir, abs).replace(/\\/g, '/'));
}
paths.sort();
return paths;
}

async function loadTyped<T>(
stagedDir: string,
allPaths: string[],
prefix: string,
parse: (raw: unknown) => T,
): Promise<Map<string, T>> {
const byPath = new Map<string, T>();
for (const path of allPaths) {
if (!path.startsWith(prefix) || !path.endsWith('.json')) continue;
try {
const body = await readFile(join(stagedDir, path), 'utf-8');
byPath.set(path, parse(JSON.parse(body)));
} catch {
// Malformed file — skip.
}
}
return byPath;
}

async function loadBundle(stagedDir: string): Promise<LoadedBundle> {
const allPaths = await walkStagedDir(stagedDir);
let manifest: TableauManifest | null = null;
try {
const body = await readFile(join(stagedDir, STAGED_FILES.manifest), 'utf-8');
manifest = tableauManifestSchema.parse(JSON.parse(body));
} catch {
manifest = null;
}

const datasourcesByPath = await loadTyped(stagedDir, allPaths, `${STAGED_FILES.datasourcesDir}/`, (raw) =>
stagedDatasourceFileSchema.parse(raw),
);
const workbooksByPath = await loadTyped(stagedDir, allPaths, `${STAGED_FILES.workbooksDir}/`, (raw) =>
stagedWorkbookFileSchema.parse(raw),
);

return { manifest, datasourcesByPath, workbooksByPath, allPaths };
}

/** Max data sources per LLM work unit. Controls parallel processing granularity. */
const DATASOURCES_PER_UNIT = 50;
/** Max workbooks per LLM work unit. Controls incremental re-sync granularity. */
const WORKBOOKS_PER_UNIT = 2000;

function emitBatches(
paths: string[],
perUnit: number,
unitKeyBase: string,
labelBase: string,
noun: string,
allPaths: string[],
): WorkUnit[] {
const batches = Math.ceil(paths.length / perUnit) || 0;
const units: WorkUnit[] = [];
for (let i = 0; i < batches; i++) {
const batch = paths.slice(i * perUnit, (i + 1) * perUnit);
const rawFiles = [...batch].sort();
const rawFilesSet = new Set(rawFiles);
const suffix = batches > 1 ? `-${i}` : '';
units.push({
unitKey: `${unitKeyBase}${suffix}`,
displayLabel: batches > 1 ? `${labelBase} (${i + 1}/${batches})` : labelBase,
rawFiles,
peerFileIndex: allPaths.filter((p) => !rawFilesSet.has(p)).sort(),
dependencyPaths: [],
notes: `${batch.length} ${noun}${batch.length === 1 ? '' : 's'}`,
});
}
return units;
}

function emitWorkUnits(bundle: LoadedBundle): WorkUnit[] {
if (!bundle.manifest) return [];
const dsPaths = [...bundle.datasourcesByPath.keys()].sort();
const wbPaths = [...bundle.workbooksByPath.keys()].sort();
return [
...emitBatches(dsPaths, DATASOURCES_PER_UNIT, 'tableau-datasources', 'Tableau: data sources', 'data source', bundle.allPaths),
...emitBatches(wbPaths, WORKBOOKS_PER_UNIT, 'tableau-workbooks', 'Tableau: workbooks', 'workbook', bundle.allPaths),
];
}

interface ChunkOptions {
diffSet?: DiffSet;
}

export async function chunkTableauStagedDir(stagedDir: string, opts: ChunkOptions = {}): Promise<ChunkResult> {
const bundle = await loadBundle(stagedDir);
if (!bundle.manifest) {
return { workUnits: [] };
}

const firstRunUnits = emitWorkUnits(bundle);
if (!opts.diffSet) {
return { workUnits: firstRunUnits };
}

const touched = new Set([...opts.diffSet.added, ...opts.diffSet.modified]);
const kept: WorkUnit[] = [];
for (const wu of firstRunUnits) {
const anyTouched = wu.rawFiles.some((p) => touched.has(p));
if (!anyTouched) continue;
const changedFiles = wu.rawFiles.filter((p) => touched.has(p));
const unchangedFiles = wu.rawFiles.filter((p) => !touched.has(p));
const deps = new Set([...wu.dependencyPaths, ...unchangedFiles]);
kept.push({ ...wu, rawFiles: changedFiles.sort(), dependencyPaths: [...deps].sort() });
}
const eviction =
opts.diffSet.deleted.length > 0 ? { deletedRawPaths: [...opts.diffSet.deleted].sort() } : undefined;
return { workUnits: kept, eviction };
}
62 changes: 62 additions & 0 deletions packages/cli/src/context/ingest/adapters/tableau/client-port.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
import type { FetchContext } from '../../types.js';
import type { DatasourceFilterInput, TableauPullConfig, WorkbookFilterInput } from './types.js';

export interface TableauTestConnectionResult {
success: boolean;
message?: string;
error?: string;
}

/** A field record as returned by the Metadata API for a published data source. */
export interface TableauFieldRecord {
name: string;
role?: string;
dataType?: string;
/** Present for a CalculatedField; absent for a plain ColumnField. */
formula?: string;
description?: string;
}

/** An upstream (physical) table record from the Metadata API lineage. */
export interface TableauUpstreamTableRecord {
luid?: string;
name: string;
schema?: string;
fullName?: string;
}

/** A published data source with its fields and upstream tables, from the Metadata API. */
export interface TableauDatasourceRecord {
luid: string;
name: string;
projectName?: string;
updatedAt?: string;
hasExtracts?: boolean;
description?: string;
fields: TableauFieldRecord[];
upstreamTables: TableauUpstreamTableRecord[];
}

/** Workbook summary metadata from the Metadata API. */
export interface TableauWorkbookRecord {
luid: string;
name: string;
projectName?: string;
description?: string;
updatedAt?: string;
}

/** Re-exported so callers can reference the option types without importing from types.ts directly. */
export type { DatasourceFilterInput as ListDatasourcesOptions } from './types.js';
export type { WorkbookFilterInput as ListWorkbooksOptions } from './types.js';

export interface TableauRuntimeClient {
testConnection(): Promise<TableauTestConnectionResult>;
listDatasources(opts?: DatasourceFilterInput): Promise<TableauDatasourceRecord[]>;
listWorkbooks(opts?: WorkbookFilterInput): Promise<TableauWorkbookRecord[]>;
cleanup(): Promise<void>;
}

export interface TableauClientFactory {
createClient(config: TableauPullConfig, ctx: FetchContext): Promise<TableauRuntimeClient> | TableauRuntimeClient;
}
Loading