Claude/markdown review qa report a hho y

PRD.md

@@ -172,9 +172,38 @@
   - What if config file is malformed?
   - No documented recovery procedures
 
+  ## Resolution Tracking
+
+  | Issue | Section | Priority | Status | Addressed By |
+  |-------|---------|----------|--------|-------------|
+  | Notification blocking monitoring | 1 | Critical | Open | Feature 007 (pending) |
+  | Process timeout | 1 | Critical | **Resolved** | Feature 016 |
+  | Missing retry logic (TCP/HTTP) | 1 | Medium | Open | — |
+  | Threshold config conflicts | 2 | Medium | Open | Needs config validation |
+  | Race condition in periodic traceroute | 2 | High | Open | Needs audit in Feature 010 code |
+  | Flapping detection | 2 | Medium | Open | — |
+  | No transaction management | 3 | High | Open | — |
+  | Unbounded traceroute storage | 3 | Medium | Partial | Config exists, enforcement needs audit |
+  | Database lock contention | 3 | Medium | Open | — |
+  | Missing config validation | 4 | Medium | Open | — |
+  | Gateway auto-detection fallback | 4 | Low | Open | — |
+  | ICMP vs TCP method mismatch | 5 | Medium | Open | — |
+  | HTTP latency thresholds | 5 | Medium | Open | Feature 009 (pending) |
+  | No rate limiting on concurrency | 6 | Medium | Open | — |
+  | Log rotation timing | 6 | Low | Partial | Feature 006 (daily rotation) |
+  | Memory leak in quality metrics | 6 | Low | N/A | Feature 008 not implemented |
+  | macOS command parsing fragility | 7 | Medium | Open | — |
+  | launchd service race condition | 7 | Low | Open | Feature 006 |
+  | Command injection risk | 8 | **Highest** | Open | Needs audit of ping/traceroute shell-out |
+  | openssl shell-out for certs | 8 | Medium | N/A | Feature 009 not implemented |
+  | No rollback strategy | 9 | Low | Open | — |
+  | Backup without verification | 9 | Low | Open | Feature 011 |
+  | Error code standards | 10 | Low | Open | — |
+  | Disaster recovery docs | 10 | Low | Open | — |
+
   ## Summary of Critical Issues
 
-  1. Highest Priority: Command injection vulnerability needs immediate audit
+  1. **Highest Priority**: Command injection vulnerability needs immediate audit
   2. High Priority: Database transaction safety, state machine race conditions
   3. Medium Priority: Configuration validation, retry logic, method-specific thresholds
   4. Low Priority: Documentation gaps, error codes, rollback strategy

TEST.md

@@ -1,6 +1,31 @@
-# Manual Testing Guide
+# Testing Guide
 
-This guide covers manual testing for Feature 010 (Enhanced Culprit Tracking).
+## Overview
+
+This guide covers automated and manual testing for Vigil.
+
+### Automated Tests
+
+```bash
+cargo test                      # Run all unit and integration tests
+cargo test -- --nocapture       # Run with stdout visible
+./scripts/qa.sh                 # Full QA: fmt, clippy, test, doc, build
+VIGIL_ENV=test cargo test       # Run with isolated test database
+```
+
+### Test Environments
+
+| Environment | Database Path | Usage |
+|-------------|--------------|-------|
+| Production | `~/Library/Application Support/ch.kapptec.vigil/monitor.db` | Live monitoring |
+| Development | `.../ch.kapptec.vigil/dev/monitor.db` | `--dev` flag or `VIGIL_ENV=dev` |
+| Test | `.../ch.kapptec.vigil/test/monitor.db` | `VIGIL_ENV=test` |
+
+---
+
+## Manual Testing: Feature 010 (Enhanced Culprit Tracking)
+
+This section covers manual testing for Feature 010.
 
 ## 1. Setup Development Environment
 

claude.md

@@ -20,9 +20,12 @@ This file provides context for Claude Code sessions working on this project.
 | 004 Hop Analysis | Done | `src/monitor/traceroute.rs` |
 | 005 CLI Reporting | Done | `src/cli/helpers.rs`, `status.rs`, `outages.rs`, `stats.rs` |
 | 006 Polish & Service | Done | `src/cli/service.rs`, log rotation in `lib.rs` |
-| 009 HTTP Endpoint Monitoring | Pending | Timing breakdown, cert tracking, `vigil http`/`vigil certs` commands |
+| 007 Alerts & Notifications | Pending | Desktop, webhook, command notifications |
+| 008 Latency Quality Metrics | Pending | Jitter, packet loss, MOS score |
+| 009 HTTP Endpoint Monitoring | Partial | Basic HTTP in 014; remaining: timing breakdown, cert tracking, `vigil http`/`vigil certs` |
 | 010 Enhanced Culprit Tracking | Done | `src/db.rs` (migrate_v2), `src/monitor/state.rs`, `src/cli/outage_detail.rs` |
 | 011 Dev Environment | Done | `config.rs` (Environment), `main.rs` (--dev flag) |
+| 012 CI Pipeline Fix | Done | `.github/workflows/ci.yml` |
 | 013 Gateway-First Diagnosis | Done | `src/monitor/traceroute.rs`, `main.rs`, `cli/outage_detail.rs` |
 | 014 TCP/HTTP Connectivity | Done | `src/monitor/tcp.rs`, `src/monitor/http.rs`, `models.rs` |
 | 015 Version Information | Done | `build.rs`, `src/lib.rs`, `src/cli/version.rs` |
@@ -87,9 +90,10 @@ route -n get default | grep gateway
 ## Testing
 
 ```bash
-cargo test           # Run all tests
-cargo run -- init    # Initialize config/db
-cargo run -- start   # Start monitoring (placeholder until 002)
+cargo test                      # Run all tests
+cargo run -- --dev init         # Initialize dev config/db
+cargo run -- --dev start -f     # Start monitoring (dev mode, foreground)
+./scripts/qa.sh                 # Full QA: fmt, clippy, test, doc, build
 ```
 
 ## Dependencies
@@ -108,12 +112,16 @@ Display:
 - `tabled` - Table formatting
 - `indicatif` - Progress bars
 
-## Database Schema
+## Database Schema (v3)
 
 ```sql
 outages(id, start_time, end_time, duration_secs, affected_targets, failing_hop, failing_hop_ip, notes)
 ping_log(id, timestamp, target, target_name, latency_ms, success)
-traceroutes(id, outage_id, timestamp, target, hops, success)
+traceroutes(id, outage_id, degraded_event_id, trace_trigger, gateway_reachable,
+            gateway_latency_ms, diagnosis, timestamp, target, hops, success)
+degraded_events(id, start_time, end_time, duration_secs, escalated_to_outage_id,
+                affected_targets, notes)
+schema_version(version, applied_at, description)
 ```
 
 ## Installation

docs/README.md

@@ -19,8 +19,18 @@ Implementation is organized into sequentially numbered features:
 | [002](./features/002-ping-monitor.md) | Ping Monitor | Done | Continuous multi-target ping monitoring |
 | [003](./features/003-outage-detection.md) | Outage Detection | Done | State machine for connectivity tracking |
 | [004](./features/004-hop-analysis.md) | Hop Analysis | Done | Traceroute integration for fault isolation |
-| [005](./features/005-cli-reporting.md) | CLI & Reporting | Pending | Status display, outage history, statistics |
-| [006](./features/006-polish-service.md) | Polish & Service | Pending | Launchd, graceful shutdown, log rotation |
+| [005](./features/005-cli-reporting.md) | CLI & Reporting | Done | Status display, outage history, statistics |
+| [006](./features/006-polish-service.md) | Polish & Service | Done | Launchd, graceful shutdown, log rotation |
+| [007](./features/007-alerts-notifications.md) | Alerts & Notifications | Pending | Desktop, webhook, command notifications |
+| [008](./features/008-latency-quality-metrics.md) | Latency Quality Metrics | Pending | Jitter, packet loss, MOS score |
+| [009](./features/009-http-endpoint-monitoring.md) | HTTP Endpoint Monitoring | Pending | Timing breakdown, cert tracking, HTTP CLI |
+| [010](./features/010-enhanced-culprit-tracking.md) | Enhanced Culprit Tracking | Done | Periodic traceroutes, degraded events |
+| [011](./features/011-dev-environment-upgrade-strategy.md) | Dev Environment & Upgrades | Done | Environment isolation, DB migrations |
+| [012](./features/012-ci-pipeline-fix.md) | CI Pipeline Fix | Done | GitHub Actions fix and simplification |
+| [013](./features/013-gateway-first-diagnosis.md) | Gateway-First Diagnosis | Done | Gateway ping before traceroute |
+| [014](./features/014-tcp-connectivity-monitoring.md) | TCP/HTTP Connectivity | Done | TCP and HTTP monitoring methods |
+| [015](./features/015-version-info.md) | Version Information | Done | Build info, schema status, JSON output |
+| [016](./features/016-process-timeout.md) | Process Timeout | Done | Subprocess hard timeout, slow ping detection |
 
 ## Quick Start
 
@@ -29,13 +39,16 @@ Implementation is organized into sequentially numbered features:
 vigil init
 
 # Start monitoring
-vigil start
+vigil start --foreground
 
 # Check status
 vigil status
 
 # View recent outages
-vigil outages --last 24h
+vigil outages -l 24h
+
+# View version and build info
+vigil version
 ```
 
 ## Problem Statement

docs/architecture.md

@@ -3,21 +3,27 @@
 ## System Overview
 
 ```
-┌─────────────────────────────────────────────────────────────────┐
-│                     Network Monitor                              │
-├─────────────────────────────────────────────────────────────────┤
-│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────────┐  │
-│  │ Ping Monitor │  │ Hop Analyzer │  │ Outage Detector      │  │
-│  │ (continuous) │  │ (on-demand)  │  │ (state machine)      │  │
-│  └──────┬───────┘  └──────┬───────┘  └──────────┬───────────┘  │
-│         │                 │                      │              │
-│         └─────────────────┼──────────────────────┘              │
-│                           ▼                                      │
-│                   ┌───────────────┐                              │
-│                   │ Event Logger  │                              │
-│                   │ (SQLite)      │                              │
-│                   └───────────────┘                              │
-└─────────────────────────────────────────────────────────────────┘
+┌──────────────────────────────────────────────────────────────────┐
+│                      Vigil Network Monitor                        │
+├──────────────────────────────────────────────────────────────────┤
+│  ┌───────────────────────┐  ┌──────────────────────────────────┐ │
+│  │ Connectivity Checker  │  │ Outage Detector (state machine)  │ │
+│  │ ┌───────┐ ┌─────┐    │  │ ONLINE → DEGRADED → OFFLINE      │ │
+│  │ │ Ping  │ │ TCP │    │  └──────────────┬───────────────────┘ │
+│  │ └───────┘ └─────┘    │                 │                     │
+│  │ ┌───────┐             │                 ▼                     │
+│  │ │ HTTP  │             │  ┌──────────────────────────────────┐ │
+│  │ └───────┘             │  │ Hop Analyzer (gateway-first)     │ │
+│  └───────────┬───────────┘  │ gateway ping → traceroute        │ │
+│              │               └──────────────┬───────────────────┘ │
+│              └──────────────────────────────┘                     │
+│                              │                                    │
+│                              ▼                                    │
+│                      ┌───────────────┐                            │
+│                      │ Event Logger  │                            │
+│                      │ (SQLite v3)   │                            │
+│                      └───────────────┘                            │
+└──────────────────────────────────────────────────────────────────┘
 ```
 
 ## Network Topology
@@ -33,18 +39,26 @@ By monitoring multiple targets and running traceroute during outages, we can ide
 
 ## Components
 
-### 1. Ping Monitor (`src/monitor/ping.rs`)
+### 1. Connectivity Checker (`src/monitor/`)
 
-- Continuously pings multiple targets at configurable intervals
-- Uses macOS `ping` command via shell-out
-- Parses output for latency and success/failure
+Unified dispatcher (`mod.rs`) routes checks to the appropriate method:
+
+| Method | File | Protocol | Use Case |
+|--------|------|----------|----------|
+| Ping | `ping.rs` | ICMP | Traditional, may be rate-limited |
+| TCP | `tcp.rs` | TCP SYN | Default — accurate for real-world connectivity |
+| HTTP | `http.rs` | HTTP HEAD | Full HTTP check with response validation |
+
+- Continuously checks multiple targets at configurable intervals (default 2s)
+- Hard process timeout (default 6s) kills hung subprocesses
+- Adaptive polling: 4x faster during DEGRADED state (500ms)
 - Runs concurrently using tokio tasks
 
 **Targets monitored:**
 
 - Local gateway (auto-detected or configured)
-- External DNS servers (8.8.8.8, 1.1.1.1)
-- Custom targets (user-configured)
+- External DNS servers (8.8.8.8, 1.1.1.1) via TCP:443 by default
+- Custom targets (user-configured, any method)
 
 ### 2. State Machine (`src/monitor/state.rs`)
 
@@ -68,42 +82,55 @@ Tracks connectivity state with hysteresis to avoid flapping:
 
 **Thresholds (configurable):**
 
-- `degraded_threshold`: 3 consecutive failures → DEGRADED
-- `offline_threshold`: 5 consecutive failures → OFFLINE
-- `recovery_threshold`: 2 consecutive successes → ONLINE
+- `degraded_threshold`: 2 consecutive failures → DEGRADED
+- `offline_threshold`: 3 consecutive failures → OFFLINE
+- `recovery_threshold`: 3 consecutive successes → ONLINE
 
 ### 3. Hop Analyzer (`src/monitor/traceroute.rs`)
 
-- Triggered when entering OFFLINE state
+- Gateway-first diagnosis: pings gateway before running traceroute
+- Triggered when entering DEGRADED or OFFLINE state
+- Periodic traceroutes during ongoing outages (configurable interval)
 - Runs macOS `traceroute` command
 - Parses output to identify failing hop
-- Stores results linked to outage events
+- Stores results linked to outage or degraded events
+- Diagnosis output: `LocalNetworkDown`, `IspIssue`, `Healthy`, `Intermittent`, `Unknown`
 
 ### 4. Database (`src/db.rs`)
 
-SQLite database with three tables:
+SQLite database (schema v3) with five tables:
 
-**outages** - Outage events
+**outages** - Outage events (OFFLINE state)
 
-```sql
-- id, start_time, end_time, duration_secs
-- affected_targets (JSON array)
-- failing_hop, failing_hop_ip
-- notes
+```
+id, start_time, end_time, duration_secs, affected_targets (JSON),
+failing_hop, failing_hop_ip, notes
+```
+
+**degraded_events** - Degraded state transitions (pre-outage)
+
+```
+id, start_time, end_time, duration_secs, escalated_to_outage_id,
+affected_targets (JSON), notes
 ```
 
 **ping_log** - Individual ping results (sampled)
 
-```sql
-- id, timestamp, target, target_name
-- latency_ms, success
+```
+id, timestamp, target, target_name, latency_ms, success
 ```
 
-**traceroutes** - Traceroute snapshots
+**traceroutes** - Traceroute snapshots linked to outages or degraded events
 
-```sql
-- id, outage_id, timestamp, target
-- hops (JSON array), success
+```
+id, outage_id, degraded_event_id, trace_trigger, gateway_reachable,
+gateway_latency_ms, diagnosis, timestamp, target, hops (JSON), success
+```
+
+**schema_version** - Migration tracking
+
+```
+version, applied_at, description
 ```
 
 ### 5. Configuration (`src/config.rs`)
@@ -122,48 +149,61 @@ Supports:
 ## Data Flow
 
 ```
-1. Ping Monitor sends pings every 1 second
+1. Connectivity Checker runs checks every 2 seconds (ping/TCP/HTTP)
                     │
                     ▼
 2. Results fed to State Machine
                     │
                     ├── State unchanged → Log ping result
                     │
-                    ├── State → OFFLINE
+                    ├── State → DEGRADED
                     │       │
                     │       ▼
-                    │   Trigger Hop Analyzer
+                    │   Create DegradedEvent + run gateway-first diagnosis
+                    │
+                    ├── State → OFFLINE (from DEGRADED)
+                    │       │
+                    │       ▼
+                    │   Create Outage record + run diagnosis
+                    │   Periodic traceroutes every N seconds
+                    │
+                    ├── State → ONLINE (from DEGRADED)
                     │       │
                     │       ▼
-                    │   Create Outage record
+                    │   End DegradedEvent (no outage created)
                     │
                     └── State → ONLINE (from OFFLINE)
                             │
                             ▼
-                        End Outage record
-                        (set end_time, duration)
+                        End Outage record (set end_time, duration)
 ```
 
 ## File Structure
 
 ```
 src/
 ├── main.rs              # CLI entry point (clap)
-├── lib.rs               # Library root, logging init
-├── config.rs            # Configuration management
-├── db.rs                # SQLite operations
-├── models.rs            # Data structures
+├── lib.rs               # Library root, logging init, version constants
+├── config.rs            # Configuration management, environment support
+├── db.rs                # SQLite operations, migrations (v1→v3)
+├── models.rs            # Data structures (Target, Outage, PingResult, etc.)
 ├── monitor/
-│   ├── mod.rs
-│   ├── ping.rs          # Ping implementation
-│   ├── state.rs         # State machine
-│   └── traceroute.rs    # Traceroute implementation
+│   ├── mod.rs           # Unified connectivity dispatcher
+│   ├── ping.rs          # ICMP ping with process timeout
+│   ├── tcp.rs           # TCP connectivity checks
+│   ├── http.rs          # HTTP endpoint checks
+│   ├── state.rs         # State machine (ONLINE/DEGRADED/OFFLINE)
+│   └── traceroute.rs    # Traceroute + gateway-first diagnosis
 └── cli/
     ├── mod.rs
-    ├── start.rs         # Start command
-    ├── status.rs        # Status command
-    ├── outages.rs       # Outages command
-    └── stats.rs         # Stats command
+    ├── start.rs         # Start monitor daemon
+    ├── status.rs        # Current status display
+    ├── outages.rs       # List outages
+    ├── outage_detail.rs # Detailed outage view with traceroutes
+    ├── stats.rs         # Statistics reporting
+    ├── service.rs       # macOS launchd service management
+    ├── version.rs       # Version and build info
+    └── helpers.rs       # Shared CLI utilities
 ```
 
 ## macOS Integration