Flexible KB Source Resolution

[rucka]

Resolution

Delivered by PR #133 (merged). Board: Done.

feat: flexible KB source resolution.

Story Statement

As a CLI user
I want flexible KB source installation from remote URLs, local ZIPs, local directories, or default sources
So that I can install/update KB content from any location without workarounds and work offline when network unavailable

Where: Command-line interface (pair install, pair update commands)

Epic Context

Parent Epic: #65 Enhanced CLI Packaging & Distribution
Status: Refined
Priority: P1 (Should-Have)

Status Workflow

• Refined: Story is detailed, estimated, and ready for development ✅ Current
• In Progress: Story is actively being developed
• Done: Story delivered and accepted

Acceptance Criteria

Functional Requirements

Given-When-Then Format:

1. Given developer in monorepo development mode runs pair install
   When no flags provided
   Then CLI installs KB from packages/knowledge-hub/dataset/ with success message

2. Given developer with released CLI runs pair install
   When no flags provided
   Then CLI downloads KB from GitHub release ZIP to ~/.pair/kb/{version}/ and installs with success message

3. Given developer runs pair install --source https://example.com/releases/kb-v1.0.0.zip
   When network available
   Then CLI downloads ZIP from URL, extracts to temp location, validates, installs to target with progress indicator

4. Given developer runs pair install --source /absolute/path/to/kb-dataset
   When directory exists and valid
   Then CLI validates KB structure, copies content to target, displays success message

5. Given developer runs pair install --source ./relative/path/to/kb.zip
   When ZIP file exists and valid
   Then CLI extracts ZIP, validates KB structure, installs to target

6. Given developer runs pair install --source ../outside-repo/shared-kb/
   When directory exists outside current repo
   Then CLI validates and installs KB from external directory

7. Given developer runs pair install --offline
   When no --source provided
   Then CLI exits with error "Offline mode requires explicit --source with local path"

8. Given developer runs pair install --offline --source /local/kb-dataset
   When directory exists
   Then CLI installs from local source without network access, skips download attempts

9. Given developer runs pair update --source https://example.com/kb-v2.0.0.zip
   When target already has KB installed
   Then CLI downloads new KB version, validates, replaces existing KB content, preserves target config

10. Given developer runs pair install --source https://example.com/kb.zip
    When network unavailable
    Then CLI displays error "Failed to download KB from URL: network unavailable. Try --offline with local --source"

11. Given developer runs pair install --source /non-existent/path
    When path doesn't exist
    Then CLI exits with error "KB source path not found: /non-existent/path"

12. Given developer runs pair install --source /path/with/invalid-kb-structure
    When directory missing required .pair/ structure
    Then CLI exits with error "Invalid KB structure: missing required directories/files" with validation details

Business Rules

• Dataset root concept ELIMINATED - no mandatory dataset root configuration
• Default source (monorepo): packages/knowledge-hub/dataset/ relative to repo root
• Default source (release/npm): GitHub release ZIP URL from package metadata
• --source accepts: http(s) URLs, absolute paths, relative paths (resolved from CWD)
• Relative paths resolved from: current working directory (process.cwd()), not project root
• --offline mode: prevents all network access, requires local --source
• KB validation: all sources validated before installation (structure, manifest, required files)
• Progress indicators: display for download operations (URL sources)
• Atomic installation: partial installs cleaned up on failure
• Version coordination: release mode KB version matches CLI version (default source only)

Edge Cases and Error Handling

• --offline with remote URL: Error "Cannot use --offline with remote URL source"
• Invalid ZIP format: Error "Failed to extract KB ZIP: corrupted or invalid format"
• Partial download: Cleanup temp files, error "KB download incomplete, retrying..."
• Insufficient permissions: Error "Cannot write to target directory: permission denied"
• Invalid KB structure: Detailed validation error listing missing/invalid components
• Network timeout: Error with retry suggestion "Download timeout, retry or use --offline with local source"
• Large KB warning: Warning if KB size >100MB during download/install
• Concurrent installs: Lock mechanism prevents conflicts or clear error message

Definition of Done Checklist

Development Completion

• Source resolution service implemented (resolveKBSource)
• Download manager enhanced with progress indicator
• Extraction service implemented (ZIP support)
• KB validator integration (US Validate KB Content #74 module)
• Installation service refactored for atomic operation
• Default source detection (monorepo vs release)
• --source override with all path types (URL, absolute, relative)
• --offline mode enforcement (prevents network, requires local source)
• Relative path resolution from CWD
• Progress indicators for download operations
• Retry logic for network failures
• Checksum verification for downloads
• Atomic installation with rollback
• Temp file cleanup on success and failure
• Lock mechanism for concurrent install prevention
• Code follows project standards (TypeScript strict, no any)
• Code review completed and approved
• Unit tests for source resolution (10+ tests)
• Unit tests for download manager (8+ tests)
• Unit tests for extraction service (6+ tests)
• Integration tests for full install flow (15+ tests covering all source types)
• Regression tests for existing behavior (5+ tests)
• E2E tests for install/update commands (8+ tests)

Quality Assurance

• All acceptance criteria tested and verified
• Edge cases tested (network failures, invalid sources, permissions)
• Backward compatibility verified (default install/update unchanged)
• Performance tested (KB install <60s for typical KB)
• Download progress indicator works correctly
• Offline mode prevents network access
• All source types work (URL, absolute, relative inside/outside repo)
• Validation errors clear and actionable
• Atomic installation verified (no partial installs)
• Cleanup verified on failure scenarios
• Concurrent install handling tested
• Cross-platform testing (Linux/macOS minimum)
• Code coverage ≥85% for new modules

Deployment and Release

• Feature works in both dev and release modes
• Default source determination correct for both modes
• Manual testing in monorepo environment
• Manual testing in release environment (npm package)
• Documentation (US KB Source Documentation & CLI Reference #90) matches implementation
• Smoke tests created for all source types
• Demo-ready: show install from URL, local directory, offline mode

Story Sizing and Sprint Readiness

Refined Story Points

Final Story Points: 8
Confidence Level: High
Sizing Justification:

• Source resolution implementation (~1 pt)
• Download manager enhancement (~1.5 pt)
• Extraction service (~1 pt)
• Validator integration (~0.5 pt)
• Installation service refactor (~1.5 pt)
• Progress indicators and UX (~0.5 pt)
• Comprehensive testing (~2 pt)
• Well-defined scope with clear specification, builds on existing infrastructure

Sprint Capacity Validation

Sprint Fit Assessment: Yes - fits in single sprint with focused effort
Development Time Estimate: 4-5 days (implementation across modules)
Testing Time Estimate: 2-3 days (unit, integration, E2E, regression)
Total Effort Assessment: 6-8 days total - fits within 2-week sprint if focused, may benefit from pairing

Story Splitting Recommendations

If needed, split into:

1. Story A (P0): Core flexible source support (URL + local paths, validation, basic offline)
   • Acceptance Criteria: AC1-6, AC11-12
   • Size Estimate: 5 pts
2. Story B (P1): Enhanced UX (progress indicators, retry logic, advanced offline)
   • Acceptance Criteria: AC7-10 (offline enhancements)
   • Dependencies: Story A
   • Size Estimate: 3 pts

Split Rationale: Maintains incremental delivery - core functionality first, UX enhancements second

Dependencies and Coordination

Story Dependencies

Prerequisite Stories:

• US KB Source Documentation & CLI Reference #90 (KB Source Documentation) - provides specification ✅
• US CommandConfig Contract & Parser Refactoring #91 (CommandConfig Parser) - provides input types ✅
  Dependent Stories: None (delivers complete user value)
  Shared Components:
• kb-manager.ts (enhance existing download logic)
• config-utils.ts (path resolution)
• KB validator from US Validate KB Content #74 (integration point)

Team Coordination

Development Roles Involved:

• Backend/CLI Developer: Implementation across multiple modules
• QA: Comprehensive testing including network scenarios, cross-platform

External Dependencies

Infrastructure Requirements:

• GitHub releases for default remote KB source (already exists)
• Network access for URL-based installs (user-controlled)

Validation and Testing Strategy

Acceptance Testing Approach

Testing Methods:

1. Unit tests for source resolution algorithm (all CommandConfig types)
2. Unit tests for download manager (mocked network, progress tracking)
3. Unit tests for extraction service (sample ZIPs)
4. Integration tests for full install flow (all source types)
5. E2E tests with real KB sources (local repo dataset, mock remote)
6. Regression tests for existing behavior (default install/update)
7. Manual testing in monorepo and release environments
8. Network failure simulation tests (offline scenarios)

Test Data Requirements:

• Sample KB dataset directory (valid structure)
• Sample KB ZIP file (valid format)
• Invalid KB sources (missing .pair/, malformed structure)
• Mock GitHub release URLs (test environment)
• Large KB dataset for performance testing

Environment Requirements:

• Monorepo dev environment
• Simulated release environment (npm package)
• Network simulation (online/offline/slow)
• Multiple platforms (Linux, macOS minimum)

User Validation

User Feedback Collection: Sprint review demo + early adopter beta testing
Success Metrics:

• Install from URL completes in <60s for typical KB
• Install from local directory completes in <10s
• Offline mode prevents network access successfully
• All documented source types work without workarounds
• Validation errors actionable (users can fix issues)

Rollback Plan: Feature behind flag (--source override) - existing default behavior unchanged, can disable --source flag if critical issues

Notes and Additional Context

Refinement Session Insights:

• Dataset root ELIMINATED - no mandatory dataset root concept
• Relative paths from CWD - not project root (clear specification)
• Offline mode explicit - requires --source, prevents network access
• Progress indicators required - download operations need user feedback
• Atomic installation critical - no partial installs allowed
• Validation before install - integrate US Validate KB Content #74 validator

Technical Decisions:

• Use adm-zip (existing catalog dependency) for extraction
• Simple console progress (not fancy spinners/bars initially)
• Retry logic: 3 attempts, exponential backoff
• Temp directory: use os.tmpdir() with unique subdirectory
• Lock file: .pair/.install-lock at target
• Checksum verification: optional for local sources, required for remote

Implementation Notes:

• Source resolution algorithm follows spec from US KB Source Documentation & CLI Reference #90
• CommandConfig types from US CommandConfig Contract & Parser Refactoring #91 provide type-safe input
• Validator from US Validate KB Content #74 provides pre-installation validation
• Existing kb-manager.ts has download logic to enhance
• Path resolution uses Node.js path.resolve with process.cwd()

Future Considerations:

• Resume support for interrupted downloads (not in this story)
• Advanced progress indicators (fancy spinners, ETA) (not in this story)
• Multiple KB version management (future epic)
• KB source caching beyond current version (future enhancement)

Technical Analysis

Implementation Strategy

Current State: Stories #90 (KB Packaging) and #91 (CommandConfig Parser) delivered the foundational infrastructure. The codebase has been significantly refactored since the original task breakdown — command pattern (commands/{command}/parser.ts+handler.ts+metadata.ts), content-ops service layer (FileSystemService, HttpClientService), and smoke test framework (scripts/smoke-tests/) are now in place.

Approach: Fix gaps in existing pipeline rather than creating new services. Move generic logic to content-ops, fix local resolution bugs, thread download progress end-to-end, add retry logic, extend smoke tests.

Existing Infrastructure (already implemented)

Component	Location	Status
CommandConfig discriminated union (install + update)	commands/install/parser.ts, commands/update/parser.ts	Done
resolveDatasetRoot (default/remote/local strategies)	config/kb-resolver.ts	Done (gaps in local)
detectSourceType + SourceType enum	content-ops/path-resolution/source-detector.ts	Done (uses bare fs)
validateCommandOptions (offline validation)	config/cli.ts	Done
Download with progress infrastructure	content-ops/http/download-manager.ts (ProgressReporter)	Done (not wired through)
KB download + extract + validate	kb-manager/kb-installer.ts	Done
KB error formatting	kb-manager/error-formatter.ts (KBDownloadError, formatDownloadError)	Done
Smoke test framework	scripts/smoke-tests/run-all.sh + 6 scenarios	Done (no source scenarios)

Gaps to Address

Gap	Impact	Fix
detectSourceType uses bare fs module, not FileSystemService	Cannot test with InMemoryFileSystemService	T-1
validateKBStructure, normalizeExtractedKB live in CLI-only kb-installer.ts	Generic logic not reusable from content-ops	T-2
ProgressReporter exists in content-ops but DatasetResolveOptions doesn't accept progressWriter/isTTY	Download progress not visible to user through install/update flow	T-3
No retry with backoff in downloadFile	Transient network failures are fatal	T-3
resolveDatasetRoot local directory returns config.path as-is (no relative resolution)	AC-5 broken for relative paths	T-4
Parsers don't throw on INVALID source type from detectSourceType	Non-existent paths silently become local config	T-4
No KB validation for local directory path in resolveDatasetRoot	AC-12 not enforced for local dirs	T-4
installKBFromLocalDirectory uses process.cwd() instead of fs.currentWorkingDirectory()	Not testable with InMemoryFileSystemService	T-4
No smoke test scenario for --source or --offline	No integration coverage for source resolution	T-5
No --offline-only flag in run-all.sh	Cannot filter offline-safe tests in CI	T-5

Data Flow (current pipeline)

CLI args → parseInstallCommand() → InstallCommandConfig (discriminated union)
  ↓
handleInstallCommand(config, fs, options)
  ↓
resolveDatasetRoot(fs, config, options) → datasetRoot path
  ├── 'default' → getKnowledgeHubDatasetPath(fs) [monorepo/release detection]
  ├── 'remote'  → getKnowledgeHubDatasetPathWithFallback() → ensureKBAvailable() → installKB()
  └── 'local'   → installKBFromLocalZip() [.zip] or return path [dir] ← GAPS HERE
  ↓
extractRegistries → forEachRegistry → doCopyAndUpdateLinks → postCopyOps

Key Risks

Risk	Mitigation
detectSourceType signature change breaks consumers	Optional FileSystemService param with default (backward compat)
Moving KB validation to content-ops breaks imports	Re-export from old location, thin wrappers
Retry logic adds latency on real failures	Only retry transient errors (ECONNRESET, ETIMEDOUT), not 404/403

---

Task Breakdown

• T-1: Refactor detectSourceType to accept FileSystemService (2h)
• T-2: Extract KB validation utilities to content-ops (3h)
• T-3: Thread download progress + retry with backoff (3h)
• T-4: Harden local resolution, validation & error handling (3h)
• T-5: Source resolution smoke tests + --offline-only flag (3h)
• T-6: CLI UX improvements — CliPresenter + help formatting (3h)

Total Hours: 17h | Story Estimate: 8 pts

Dependency Graph

T-1 ──────┐
T-2 ──────┼── T-4 ── T-5
T-3 ──────┘

T-6 (independent)

T-1, T-2, T-3 are independent (parallel). T-4 depends on all three. T-5 depends on T-4. T-6 is independent (UX track).

AC Coverage

AC	Brief	Tasks	Status
AC-1 (default monorepo)	pair install in monorepo	Existing code	Already works
AC-2 (default release)	pair install from release	Existing code	Already works
AC-3 (--source URL)	Remote URL download + install	T-3 (progress+retry)	Partially works, needs progress/retry
AC-4 (--source absolute)	Local absolute directory	T-2, T-4 (validation)	Works, needs KB validation
AC-5 (--source relative)	Relative path resolution	T-4	Bug: not resolved from CWD
AC-6 (--source outside repo)	External directory	T-4, T-5	Needs validation + smoke test
AC-7 (--offline no source)	Error on offline without source	Existing code	Already works
AC-8 (--offline + local)	Offline with local source	T-5	Works, needs smoke coverage
AC-9 (update --source)	Update mirrors install	T-4	Parser fix for INVALID
AC-10 (network failure)	Clear error on network fail	T-3 (retry)	Needs retry + improved messages
AC-11 (non-existent path)	Error on missing path	T-1, T-4	Bug: parser ignores INVALID
AC-12 (invalid KB structure)	Error on bad structure	T-2, T-4	Missing for local dirs

---

T-1: Refactor detectSourceType to accept FileSystemService

Priority: P0 | Estimated Hours: 2h | Bounded Context: content-ops / path-resolution

Summary: Refactor detectSourceType in content-ops to accept an optional FileSystemService parameter instead of using bare fs module and process.cwd(), enabling testability with InMemoryFileSystemService.

Type: Refactoring

Description:
detectSourceType in packages/content-ops/src/path-resolution/source-detector.ts currently imports fs and path directly and uses process.cwd() for relative path resolution. This prevents testing with InMemoryFileSystemService. Refactor to accept an optional FileSystemService parameter (defaulting to real FS for backward compatibility). The function must preserve its current behavior for all callers.

Acceptance Criteria:

• Primary deliverable: detectSourceType(source, fs?) accepting optional FileSystemService
• Quality standard: All existing tests pass, new tests use InMemoryFileSystemService
• Integration requirement: Callers (install/parser.ts, update/parser.ts, config/cli.ts) continue working without changes
• Verification method: Unit tests with InMemoryFileSystemService for all source types

Technical Requirements:

• Functionality: Detect REMOTE_URL, LOCAL_ZIP, LOCAL_DIRECTORY, INVALID with injectable FS
• Performance: No overhead when using default FS
• Security: Maintain unsafe protocol rejection (file://, ftp://)

Implementation Approach:

• Technical Design: Add optional FileSystemService param. Use fs.existsSync/fs.stat from service instead of bare fs module. Use fs.currentWorkingDirectory() instead of process.cwd() for relative path resolution. Default param to production fileSystemService for backward compat.
• Bounded Context & Modules: content-ops package, path-resolution module
• Files to Modify/Create:
  • packages/content-ops/src/path-resolution/source-detector.ts — add FileSystemService param
  • packages/content-ops/src/path-resolution/source-detector.test.ts — rewrite tests with InMemoryFileSystemService (remove real FS tmpdir setup)

Dependencies:

• Technical: FileSystemService interface (already in content-ops)
• Tasks: None (independent)

Implementation Steps:

1. RED: Write tests using InMemoryFileSystemService for all source types (URL, ZIP, directory, invalid, unsafe protocols)
2. GREEN: Refactor detectSourceType signature to (source: string, fsService?: FileSystemService)
3. Replace fs.existsSync/fs.statSync with service calls, process.cwd() with fsService.currentWorkingDirectory()
4. Default fsService to production fileSystemService singleton
5. Verify all existing callers still work (no signature break)
6. REFACTOR: Remove beforeAll/afterAll tmpdir setup from test file

Testing Strategy:

• Unit Tests: 8+ tests with InMemoryFS — remote URL, unsafe protocols, local ZIP (abs+rel), local directory (abs+rel), non-existent path, INVALID fallback
• Integration Tests: Verify parseInstallCommand and parseUpdateCommand still work (existing tests pass)

Notes: Backward-compatible change. Optional param means zero impact on existing callers.

---

T-2: Extract KB validation utilities to content-ops

Priority: P0 | Estimated Hours: 3h | Bounded Context: content-ops / file-system

Summary: Move generic KB validation functions (validateKBStructure, normalizeExtractedKB, helpers) from CLI's kb-installer.ts to content-ops package, making them reusable and testable with InMemoryFileSystemService.

Type: Refactoring

Description:
validateKBStructure, normalizeExtractedKB, findKBStructureInSubdirectories, moveDirectoryContents, and copyDirectoryInMemory in apps/pair-cli/src/kb-manager/kb-installer.ts are generic file operations with no CLI dependency. They should live in content-ops for reuse (e.g., by resolveDatasetRoot for local directory validation in T-4). CLI's kb-installer.ts will import from content-ops instead.

Acceptance Criteria:

• Primary deliverable: New content-ops/src/file-system/kb-validation.ts module with exported functions
• Quality standard: 1:1 test file with InMemoryFS, all existing kb-installer tests still pass
• Integration requirement: CLI's kb-installer.ts imports from @pair/content-ops instead of local definitions
• Verification method: pnpm quality-gate passes across both packages

Technical Requirements:

• Functionality: validateKBStructure(path, fs), normalizeExtractedKB(path, fs) available from content-ops
• Performance: No change (same logic, different location)
• Security: No change

Implementation Approach:

• Technical Design: Create packages/content-ops/src/file-system/kb-validation.ts. Move functions. Export from content-ops public API. Update kb-installer.ts imports.
• Bounded Context & Modules: content-ops file-system module, pair-cli kb-manager module
• Files to Modify/Create:
  • packages/content-ops/src/file-system/kb-validation.ts — NEW: moved functions
  • packages/content-ops/src/file-system/kb-validation.test.ts — NEW: tests with InMemoryFS
  • packages/content-ops/src/file-system/index.ts — export new module
  • packages/content-ops/src/index.ts — export new functions
  • apps/pair-cli/src/kb-manager/kb-installer.ts — replace local functions with imports from @pair/content-ops

Dependencies:

• Technical: FileSystemService, InMemoryFileSystemService (already in content-ops)
• Tasks: None (independent)

Implementation Steps:

1. RED: Write tests for validateKBStructure and normalizeExtractedKB in content-ops using InMemoryFS
2. GREEN: Move functions from kb-installer.ts to kb-validation.ts
3. Export from content-ops barrel files (file-system/index.ts, src/index.ts)
4. Update kb-installer.ts to import from @pair/content-ops
5. Remove logDirEntriesIfDebug from moved functions (debug concern stays in CLI)
6. REFACTOR: Verify all kb-installer tests still pass
7. Run pnpm quality-gate across both packages

Testing Strategy:

• Unit Tests: 10+ tests — valid KB (with .pair/), valid KB (with AGENTS.md), invalid (empty), normalize (nested single dir), normalize (zip-temp dir), moveDirectoryContents, copyDirectoryInMemory
• Regression: All existing kb-installer.test.ts tests pass unchanged

Notes: logDirEntriesIfDebug is a CLI debug concern — keep it in CLI, don't move. The moved functions accept FileSystemService already (no signature change needed).

---

T-3: Thread download progress + retry with backoff

Priority: P0 | Estimated Hours: 3h | Bounded Context: content-ops / http + pair-cli / config

Summary: Wire download progress reporting through the full resolution pipeline (handler → resolveDatasetRoot → download) and add retry with exponential backoff for transient network failures in content-ops downloadFile.

Type: Feature Implementation

Description:
Two gaps: (1) ProgressReporter infrastructure exists in content-ops downloadFile but DatasetResolveOptions and getKnowledgeHubDatasetPathWithFallback don't accept progressWriter/isTTY, so download progress is invisible to users. (2) downloadFile has resume support but no retry — transient network errors (ECONNRESET, ETIMEDOUT) are fatal. Fix both by threading progress options through the pipeline and adding a retry wrapper.

Acceptance Criteria:

• Primary deliverable: (1) Progress visible during pair install --source <url>. (2) Transient failures retried 3 times with backoff.
• Quality standard: Tests use MockHttpClientService, no real network calls
• Integration requirement: Existing download behavior preserved, progress/retry are additive
• Verification method: Unit tests simulating progress and transient failures

Technical Requirements:

• Functionality: Progress bar during remote downloads, retry 3x with 1s/2s/4s backoff
• Performance: No overhead for local/offline operations
• Security: No change

Implementation Approach:

• Technical Design:
  • Progress threading: Add progressWriter? and isTTY? to DatasetResolveOptions in config/kb-resolver.ts. Thread through getKnowledgeHubDatasetPathWithFallback → downloadKBIfNeeded → ensureKBAvailable → installKB → downloadFile. Install/update handlers pass process.stdout as progressWriter.
  • Retry logic: Add downloadWithRetry wrapper in content-ops download-manager.ts (or retryable-download.ts). Wraps downloadFile with configurable maxRetries (default 3) and delays (default [1000, 2000, 4000]). Only retries on transient errors (isRetryableError check: ECONNRESET, ETIMEDOUT, ECONNREFUSED — NOT 404/403).
• Bounded Context & Modules: content-ops http module, pair-cli config module
• Files to Modify/Create:
  • packages/content-ops/src/http/retryable-download.ts — NEW: downloadWithRetry wrapper
  • packages/content-ops/src/http/retryable-download.test.ts — NEW: tests with MockHttpClient
  • packages/content-ops/src/http/index.ts — export new module
  • apps/pair-cli/src/config/kb-resolver.ts — add progressWriter/isTTY to DatasetResolveOptions, thread through
  • apps/pair-cli/src/kb-manager/kb-installer.ts — use downloadWithRetry instead of downloadFile
  • apps/pair-cli/src/commands/install/handler.ts — pass progressWriter: process.stdout in options
  • apps/pair-cli/src/commands/update/handler.ts — same

Dependencies:

• Technical: MockHttpClientService, ProgressReporter (already in content-ops)
• Tasks: None (independent of T-1, T-2)

Implementation Steps:

1. RED: Write tests for downloadWithRetry — success on first try, success on retry, failure after max retries, non-retryable error not retried
2. GREEN: Implement downloadWithRetry in content-ops with isRetryableError check
3. RED: Write test for progress threading — verify progressWriter reaches downloadFile
4. GREEN: Add progressWriter/isTTY to DatasetResolveOptions, thread through getKnowledgeHubDatasetPathWithFallback
5. Update kb-installer.ts installKB to call downloadWithRetry
6. Update install/update handlers to pass process.stdout as progressWriter
7. REFACTOR: Run pnpm quality-gate

Testing Strategy:

• Unit Tests: 8+ tests — retry success, retry exhaustion, non-retryable skip, progress threading verification, backoff timing
• Manual Testing: pair install --source <url> shows progress bar, retry on simulated failure

Notes: isRetryableError checks error message for econnreset, etimedout, econnrefused, socket hang up. HTTP 404/403 are NOT retried. Resume (byte-level) is orthogonal to retry (request-level) — both can coexist.

---

T-4: Harden local resolution, validation & error handling

Priority: P0 | Estimated Hours: 3h | Bounded Context: pair-cli / config + commands

Summary: Fix local directory resolution bugs in resolveDatasetRoot (relative paths, existence validation, KB structure validation) and add INVALID source type error in parsers. Covers AC-5, AC-6, AC-9, AC-11, AC-12.

Type: Bug Fix + Feature Implementation

Description:
Three related bugs/gaps: (1) resolveDatasetRoot case 'local' for directories returns config.path as-is without resolving relative paths or validating existence. (2) parseInstallCommand and parseUpdateCommand don't throw when detectSourceType returns INVALID — non-existent paths silently become local config. (3) Local directory sources skip KB structure validation (ZIP goes through normalizeExtractedKB but directory doesn't). Also fix installKBFromLocalDirectory using process.cwd() instead of fs.currentWorkingDirectory().

Acceptance Criteria:

• Primary deliverable: Relative paths resolved, non-existent paths error early, KB structure validated for all source types
• Quality standard: Tests use InMemoryFileSystemService for all FS operations
• Integration requirement: Existing behavior for valid paths unchanged
• Verification method: Unit tests covering AC-5, AC-11, AC-12 scenarios

Technical Requirements:

• Functionality: Resolve relative paths from CWD, validate existence, validate KB structure
• Performance: Validation adds minimal overhead (single FS stat + structure check)
• Security: Path traversal prevented by detectSourceType INVALID check

Implementation Approach:

• Technical Design:
  • Parser fix: After detectSourceType call, if result is INVALID, throw Error('KB source path not found: <source>'). Both install and update parsers.
  • resolveDatasetRoot fix: For local non-ZIP: resolve relative path via path.resolve(fs.currentWorkingDirectory(), config.path), validate existence with fs.exists(), validate KB structure via validateKBStructure (from content-ops after T-2), throw clear errors.
  • installKBFromLocalDirectory fix: Replace process.cwd() with fs.currentWorkingDirectory().
• Bounded Context & Modules: pair-cli config + commands modules
• Files to Modify/Create:
  • apps/pair-cli/src/config/kb-resolver.ts — fix local directory resolution: resolve relative, validate existence, validate KB structure
  • apps/pair-cli/src/config/kb-resolver.test.ts — add tests for relative paths, non-existent, invalid structure
  • apps/pair-cli/src/commands/install/parser.ts — throw on INVALID source type
  • apps/pair-cli/src/commands/install/parser.test.ts — add INVALID source type test
  • apps/pair-cli/src/commands/update/parser.ts — throw on INVALID source type
  • apps/pair-cli/src/commands/update/parser.test.ts — add INVALID source type test
  • apps/pair-cli/src/kb-manager/kb-installer.ts — fix process.cwd() usage

Dependencies:

• Technical: validateKBStructure from content-ops (T-2)
• Tasks: T-1 (detectSourceType with FileSystemService for testable parser), T-2 (KB validation in content-ops), T-3 (progress threading through resolveDatasetRoot options)

Implementation Steps:

1. RED: Write parser test — detectSourceType returns INVALID → parser throws
2. GREEN: Add INVALID check in parseInstallCommand and parseUpdateCommand
3. RED: Write resolveDatasetRoot test — relative local dir path → resolved absolute path
4. GREEN: Fix local directory case: path.resolve(fs.currentWorkingDirectory(), config.path)
5. RED: Write test — non-existent local dir → throws "KB source path not found"
6. GREEN: Add fs.exists() check, throw if missing
7. RED: Write test — local dir with invalid KB structure → throws "Invalid KB structure"
8. GREEN: Import validateKBStructure from @pair/content-ops, call after existence check
9. Fix installKBFromLocalDirectory to use fs.currentWorkingDirectory()
10. REFACTOR: Run pnpm quality-gate

Testing Strategy:

• Unit Tests: 12+ tests — parser INVALID throw, relative path resolution, absolute path passthrough, non-existent path error, invalid KB structure error, ZIP path unchanged, update parser INVALID throw, installKBFromLocalDirectory with InMemoryFS
• Regression: All existing kb-resolver.test.ts and parser.test.ts tests pass

Notes: The parser INVALID check uses the current (non-injected) detectSourceType call — this is fine for production but parser tests may need to create real tmp dirs OR rely on the refactored detectSourceType from T-1 with InMemoryFS. Prefer InMemoryFS approach after T-1.

---

T-5: Source resolution smoke tests + --offline-only flag

Priority: P0 | Estimated Hours: 3h | Bounded Context: scripts / smoke-tests

Summary: Add new smoke test scenario source-resolution.sh covering --source (local dir, local zip) and --offline modes. Add --offline-only flag to run-all.sh for CI filtering.

Type: Testing

Description:
The smoke test framework has 6 scenarios but none test --source or --offline flags. Add source-resolution.sh scenario testing local directory source, local ZIP source, offline mode, and error cases (non-existent path, invalid structure). Also add --offline-only flag to run-all.sh that filters to scenarios tagged as offline-safe (all current scenarios are offline, future remote-URL scenarios would be online-only).

Acceptance Criteria:

• Primary deliverable: source-resolution.sh scenario with 6+ test cases, --offline-only flag in run-all.sh
• Quality standard: All tests pass in CI mode, follow existing scenario patterns (lib/utils.sh helpers)
• Integration requirement: pnpm smoke-tests includes new scenario, pnpm smoke-tests -- --offline-only filters correctly
• Verification method: Manual run of pnpm smoke-tests

Technical Requirements:

• Functionality: Test --source (dir, zip), --offline, error cases; offline filtering
• Performance: New scenario <30s
• Compatibility: Bash, Linux/macOS, CI-safe (no network)

Implementation Approach:

• Technical Design:
  • New scenario: scripts/smoke-tests/scenarios/source-resolution.sh — uses setup_workspace, run_pair, assert_* helpers from lib/utils.sh. Tests: (1) pair install --source <dataset-dir> (2) pair install --source <kb.zip> created by 00-create-install-package.sh (3) pair install --offline --source <dataset-dir> (4) pair install --source /non-existent → assert_failure (5) pair install --offline without source → assert_failure (6) pair update --source <dataset-dir>
  • Offline flag: Add --offline-only flag to run-all.sh. Each scenario file exports an OFFLINE_SAFE=true|false variable (default true). When --offline-only passed, skip scenarios with OFFLINE_SAFE=false.
• Bounded Context & Modules: scripts/smoke-tests/
• Files to Modify/Create:
  • scripts/smoke-tests/scenarios/source-resolution.sh — NEW scenario
  • scripts/smoke-tests/run-all.sh — add --offline-only flag and filtering logic
  • scripts/smoke-tests/scenarios/install-basic.sh — add OFFLINE_SAFE=true tag (example for pattern)

Dependencies:

• Technical: Smoke test framework (lib/utils.sh), packaged CLI from 00-create-install-package.sh
• Tasks: T-4 (source resolution bugs fixed)

Implementation Steps:

1. Add OFFLINE_SAFE=true variable convention to existing scenarios (all are offline)
2. Add --offline-only flag parsing to run-all.sh
3. Add filtering logic: when --offline-only, check scenario's OFFLINE_SAFE before running
4. Create source-resolution.sh scenario with test cases
5. Add to CI_TESTS list in run-all.sh
6. Run pnpm smoke-tests to verify all pass
7. Run pnpm smoke-tests -- --offline-only to verify filtering

Testing Strategy:

• Manual Testing: Run full smoke suite, verify new scenario passes
• CI Validation: Verify --offline-only flag works correctly

Notes: All current scenarios are offline-safe. The --offline-only flag is forward-looking — when remote URL smoke tests are added later, they'll be tagged OFFLINE_SAFE=false and skipped in offline-only mode.

---

T-6: CLI UX improvements — CliPresenter + help formatting

Priority: P0 | Estimated Hours: 3h | Bounded Context: pair-cli / ui + commands

Summary: Create a CliPresenter module for structured CLI output with progress tracking, operation summaries, and improved help formatting using chalk.

Type: Feature Implementation

Description:
The CLI currently has minimal user feedback: sparse pushLog('info', ...) messages, no progress tracking (e.g., "Registry 2/5"), no summary at completion, and basic Commander.js help output. chalk is already a dependency but barely used. Create a CliPresenter abstraction that wraps pushLog with chalk-formatted console output, preserving backward compatibility with existing log accumulation and test patterns. Integrate into install/update handlers and enhance Commander.js help formatting.

Acceptance Criteria:

• Primary deliverable: CliPresenter module with createCliPresenter(pushLog) and createSilentPresenter(pushLog) factories
• Quality standard: Unit tests for formatting logic, existing handler tests unbroken
• Integration requirement: Presenter wraps existing pushLog — all log accumulation continues working
• Verification method: pnpm quality-gate passes, manual verification of install/update/help output

Technical Requirements:

• Functionality: Progress per registry ([1/4] name source → target), operation summary with timing, phase messages for backup/rollback, improved help with chalk
• Performance: No overhead — chalk formatting is synchronous string operations
• Security: No change

Implementation Approach:

• Technical Design: Create CliPresenter interface with startOperation, registryStart, registryDone, registryError, phase, summary methods. Two factories: createCliPresenter(pushLog) (chalk + console + pushLog) and createSilentPresenter(pushLog) (pushLog only). Integrate into install/update handlers via injectable presenter option. Enhance Commander.js help via addHelpText and configureHelp.
• Bounded Context & Modules: pair-cli ui module (new), commands/install, commands/update, cli.ts
• Files to Modify/Create:
  • apps/pair-cli/src/ui/presenter.ts — NEW: CliPresenter with chalk formatting
  • apps/pair-cli/src/ui/index.ts — NEW: barrel export
  • apps/pair-cli/src/ui/presenter.test.ts — NEW: unit tests
  • apps/pair-cli/src/commands/install/handler.ts — integrate presenter
  • apps/pair-cli/src/commands/update/handler.ts — integrate presenter
  • apps/pair-cli/src/cli.ts — improved help formatting
  • apps/pair-cli/tsconfig.json — add #ui path alias
  • apps/pair-cli/package.json — add #ui import mapping

Dependencies:

• Technical: chalk (already in deps), Commander.js (already in deps)
• Tasks: None (independent of T-1 through T-5)

Implementation Steps:

1. Setup #ui path alias in tsconfig.json and package.json
2. Create presenter.ts with CliPresenter interface, createCliPresenter, createSilentPresenter
3. Create presenter.test.ts with unit tests for formatting
4. Integrate into install handler — replace pushLog calls with presenter methods
5. Integrate into update handler — same pattern plus phase messages
6. Enhance Commander.js help in cli.ts — header, footer, chalk formatting
7. Run pnpm quality-gate

Testing Strategy:

• Unit Tests: presenter.test.ts — verify formatted output for each method, verify pushLog still called, verify silent presenter behavior
• Integration Tests: Existing handler tests pass unchanged (they check FS state, not log strings)
• Manual Testing: pair install, pair update, pair --help, pair install --help

Notes: forEachRegistry already provides index parameter — no signature change needed. createSilentPresenter allows tests to suppress console output while preserving pushLog accumulation.

---

Refinement Completed By: Product Manager (AI-assisted)
Refinement Date: 2025-12-09
Task Breakdown Date: 2026-02-14
Task Breakdown Review: Pending developer confirmation