Context
During indexing, FilesystemRepositoryIndexer._get_repository_file_paths() applies four filters before any file is parsed:
- Whitelist — include-only glob patterns (e.g.
*.py, *.ts, *.md)
- Blacklist — exclude glob patterns (e.g.
node_modules/, dist/, *.min.js)
- Size filter — skip files larger than a configured byte threshold (~200 KB)
- Depth filter — configurable max directory depth
None of these are currently exposed in the wizard UI. Users have no way to tell the indexer to ignore vendored deps, generated files, or focus on a specific subdirectory.
Scope
Expose these controls in the wizard's Configure step for all three local-ish connectors:
- Git (
GitConfigure.tsx)
- Local FS (
LocalFsConfigure.tsx)
- Archive (
ArchiveConfigure.tsx)
Confluence and Jira have different indexing paths and are out of scope.
UI / UX
Suggested placement: collapsible "Advanced options" panel in Step 2 (Configure), below the main repo/path input.
Fields:
- Whitelist — comma-separated glob patterns (default: empty = all supported extensions)
- Blacklist — comma-separated glob patterns (default:
node_modules/, .git/, dist/, *.min.js, *.lock)
- Max file size (KB) — number input (default: 200)
- Max depth — number input (default: unlimited)
Backend wiring
GenerateWikiRequest (and ScanRequest) need new optional fields:
class IndexerOptions(BaseModel):
whitelist: list[str] = []
blacklist: list[str] = []
max_file_size_kb: int = 200
max_depth: int | None = None
Pass through to FilesystemRepositoryIndexer constructor.
Acceptance
- Wizard shows "Advanced options" expander on all three connectors
- Defaults match current implicit behaviour (no regression on existing wikis)
- A repo with a custom blacklist (
vendor/) correctly skips that directory during indexing
- Settings round-trip through the scan + generate API
Notes
The Scan step (Step 3) already previews what files will be indexed — it should reflect the filter settings so the user sees the effect before committing.
Context
During indexing,
FilesystemRepositoryIndexer._get_repository_file_paths()applies four filters before any file is parsed:*.py,*.ts,*.md)node_modules/,dist/,*.min.js)None of these are currently exposed in the wizard UI. Users have no way to tell the indexer to ignore vendored deps, generated files, or focus on a specific subdirectory.
Scope
Expose these controls in the wizard's Configure step for all three local-ish connectors:
GitConfigure.tsx)LocalFsConfigure.tsx)ArchiveConfigure.tsx)Confluence and Jira have different indexing paths and are out of scope.
UI / UX
Suggested placement: collapsible "Advanced options" panel in Step 2 (Configure), below the main repo/path input.
Fields:
node_modules/, .git/, dist/, *.min.js, *.lock)Backend wiring
GenerateWikiRequest(andScanRequest) need new optional fields:Pass through to
FilesystemRepositoryIndexerconstructor.Acceptance
vendor/) correctly skips that directory during indexingNotes
The Scan step (Step 3) already previews what files will be indexed — it should reflect the filter settings so the user sees the effect before committing.