Skip to content

Add network-dependent smoke tests for remote URL KB sources #136

Description

@rucka

Context

Follow-up from PR #133 (Story #92) code review. Remote URL smoke tests were deferred because they require network access and are not suitable for offline CI environments.

Problem

Current smoke tests (scripts/smoke-tests/scenarios/source-resolution.sh) only cover local sources (directories, offline mode). Remote URL scenarios are not tested end-to-end:

  • Install from remote HTTPS URL
  • Update from remote HTTPS URL
  • Network failure handling (timeout, connection refused)
  • Download retry with exponential backoff
  • Progress indicator during download

Proposed Solution

Create new smoke test scenario for network-dependent tests:

File: scripts/smoke-tests/scenarios/remote-source-resolution.sh

Tag: OFFLINE_SAFE=false (filtered out by --offline-only flag)

Test scenarios:

  1. Install from remote HTTPS URL (GitHub release artifact)
  2. Update from remote HTTPS URL
  3. Download progress indicator verification (TTY output check)
  4. Error: Invalid URL (malformed)
  5. Error: 404 Not Found
  6. Error: Network timeout (simulated with bad URL)

Mock setup:

  • Use real GitHub release URL for testing (predictable, stable)
  • Alternative: Set up local HTTP server with test fixtures

Acceptance Criteria

  • remote-source-resolution.sh created with OFFLINE_SAFE=false tag
  • Test 1: Install from HTTPS URL succeeds
  • Test 2: Update from HTTPS URL succeeds
  • Test 3: Progress indicator shows during download
  • Test 4: Malformed URL rejected with clear error
  • Test 5: 404 error handled gracefully
  • Test 6: Network timeout triggers retry logic
  • Tests skipped when --offline-only flag used
  • Tests run in CI environment with network access
  • Update run-all.sh to include new scenario (but exclude with --offline-only)

Technical Notes

Retry verification:

  • Simulate transient failure (connection refused)
  • Verify retry attempts with exponential backoff (1s, 2s, 4s)
  • Verify max 3 retries before giving up

Progress verification:

  • Capture stdout during download
  • Verify progress indicator format: [N/M] registry-name
  • Verify operation summary at completion

Priority

Low - Unit tests already cover retry logic and error handling. Smoke tests provide additional E2E confidence but are not critical for correctness.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions