Skip to content

Expose indexer file filters in wizard (whitelist / blacklist / size / depth) for Git, Archive, Local FS #308

Description

@arozumenko

Context

During indexing, FilesystemRepositoryIndexer._get_repository_file_paths() applies four filters before any file is parsed:

  1. Whitelist — include-only glob patterns (e.g. *.py, *.ts, *.md)
  2. Blacklist — exclude glob patterns (e.g. node_modules/, dist/, *.min.js)
  3. Size filter — skip files larger than a configured byte threshold (~200 KB)
  4. Depth filter — configurable max directory depth

None of these are currently exposed in the wizard UI. Users have no way to tell the indexer to ignore vendored deps, generated files, or focus on a specific subdirectory.

Scope

Expose these controls in the wizard's Configure step for all three local-ish connectors:

  • Git (GitConfigure.tsx)
  • Local FS (LocalFsConfigure.tsx)
  • Archive (ArchiveConfigure.tsx)

Confluence and Jira have different indexing paths and are out of scope.

UI / UX

Suggested placement: collapsible "Advanced options" panel in Step 2 (Configure), below the main repo/path input.

Fields:

  • Whitelist — comma-separated glob patterns (default: empty = all supported extensions)
  • Blacklist — comma-separated glob patterns (default: node_modules/, .git/, dist/, *.min.js, *.lock)
  • Max file size (KB) — number input (default: 200)
  • Max depth — number input (default: unlimited)

Backend wiring

GenerateWikiRequest (and ScanRequest) need new optional fields:

class IndexerOptions(BaseModel):
    whitelist: list[str] = []
    blacklist: list[str] = []
    max_file_size_kb: int = 200
    max_depth: int | None = None

Pass through to FilesystemRepositoryIndexer constructor.

Acceptance

  • Wizard shows "Advanced options" expander on all three connectors
  • Defaults match current implicit behaviour (no regression on existing wikis)
  • A repo with a custom blacklist (vendor/) correctly skips that directory during indexing
  • Settings round-trip through the scan + generate API

Notes

The Scan step (Step 3) already previews what files will be indexed — it should reflect the filter settings so the user sees the effect before committing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions