Dashboard Enhancement: Trend Charts and Analytics

[ojfbot]

Overview

Add trend charts and analytics to the visual regression dashboard to track test health over time and identify patterns in failures.

Goal: Make sense of the noise by visualizing trends and identifying patterns in test results.

---

Features

1. Failure Trend Chart (Priority: High)

Display: Line chart showing failure rate over time

• X-axis: Time (last 30 days, configurable)
• Y-axis: Failure percentage
• Data points: Test runs with hover details

Metrics:

• Overall pass/fail rate
• Failure rate by component/diagram
• Average diff percentage over time
• Test duration trends

Implementation:

• Use Chart.js (already in dependencies)
• Data source: Aggregate from index.json and manifests
• Update on dashboard data refresh

2. Component Health Dashboard

Display: Grid of components with health status

• Component name
• Pass rate (last 30 days)
• Latest status (✓/✗)
• Trend indicator (↑/→/↓)

Filters:

• Show only failing components
• Sort by failure rate
• Group by diagram

3. Flaky Test Detection

Identify tests that flip between pass/fail:

• Calculate flip rate (% of runs with status change)
• Flag tests with >20% flip rate as "flaky"
• Show flaky badge on test cards
• Dedicated "Flaky Tests" filter

4. Historical Comparison

Compare two test runs side-by-side:

• Select two runs from timeline
• Show diff statistics
• Highlight changed components
• Visual diff of screenshots

---

Acceptance Criteria

• Trend chart displays failure rate over time
• Component health grid shows pass rates
• Flaky tests are automatically detected and flagged
• Historical comparison works for any two runs
• Analytics data updates with dashboard refresh
• Charts are responsive (mobile-friendly)
• Performance: < 100ms to render charts

---

Technical Notes

• Chart.js already included in dependencies
• Use memoization for expensive calculations
• Cache analytics data in index.json
• Update analytics during CI/CD run

---

Part of #71 (Visual Regression Testing Pipeline)
Follows #76 (GitHub Pages Dashboard)

[ojfbot]

📊 Dashboard Enhancement: Trend Charts and Analytics - Status Update

Parent: #71 (Visual Regression Testing Pipeline)
Status: OPEN - Planned enhancement
Priority: High
Estimated Effort: 2-3 days

---

Current Status

Foundation Complete: ✅

• Chart.js dependency added to visual-dashboard package
• Data pipeline established (index.json, manifests)
• Dashboard infrastructure ready for analytics features

Implementation Status: 🔜 Not started
This issue tracks enhancements beyond the MVP dashboard delivered in #76.

---

Implementation Readiness

Ready to Implement

• ✅ Chart.js already integrated in dashboard
• ✅ Data aggregation infrastructure in place (generate-index.ts)
• ✅ Historical data available in manifests
• ✅ React components ready for chart integration

Prerequisites Met

• ✅ Test run index with timestamps
• ✅ Pass/fail status per run
• ✅ Component-level diff statistics
• ✅ Test duration tracking (in manifests)

---

Feature Breakdown

1. Failure Trend Chart (Priority: High)

Complexity: Medium
Implementation Plan:

• Add <FailureTrendChart> component
• Aggregate pass/fail rates from index.json
• Use Chart.js Line chart
• Time range selector (7d/30d/90d)

Data Source: index.json runs array

2. Component Health Dashboard (Priority: High)

Complexity: Medium
Implementation Plan:

• Calculate pass rates by component
• Add trend indicators (↑/→/↓)
• Grid layout with health badges
• Filter by failing components

Data Source: Manifest interactions array

3. Flaky Test Detection (Priority: High)

Complexity: High
Algorithm: Track status changes across runs

• Calculate flip rate: (status_changes / total_runs) * 100
• Flag tests with >20% flip rate
• Store flip history in separate index

Data Source: Historical manifest comparison

4. Historical Comparison (Priority: Medium)

Complexity: Medium
UI: Side-by-side diff view

• Select two runs from timeline
• Compare diff statistics
• Highlight changed components
• Visual screenshot comparison

---

Technical Approach

Data Aggregation

// Add to generate-index.ts
interface AnalyticsData {
  failureRate: number;
  componentHealth: Record<string, ComponentHealth>;
  flakyTests: FlakyTest[];
  trends: TrendData[];
}

function aggregateAnalytics(runs: TestRun[]): AnalyticsData {
  // Calculate failure rates
  // Detect component health
  // Identify flaky tests
  // Generate trend data
}

Chart Integration

// src/components/FailureTrendChart.tsx
import { Line } from 'react-chartjs-2';

export function FailureTrendChart({ data }: Props) {
  const chartData = {
    labels: data.map(d => d.date),
    datasets: [{
      label: 'Failure Rate',
      data: data.map(d => d.failureRate),
      borderColor: 'rgb(255, 99, 132)',
    }]
  };
  
  return <Line data={chartData} options={chartOptions} />;
}

---

Acceptance Criteria Checklist

• Trend chart displays failure rate over time
• Component health grid shows pass rates
• Flaky tests are automatically detected and flagged
• Historical comparison works for any two runs
• Analytics data updates with dashboard refresh
• Charts are responsive (mobile-friendly)
• Performance: < 100ms to render charts

---

Dependencies

Blocks: None
Blocked by: None (ready to implement)
Related: #80 (Advanced filtering will enhance analytics)

---

Recommendation

✅ Ready to implement - All prerequisites are in place. This enhancement will significantly improve the developer experience by providing insights into test health trends.

Suggested Next Steps:

1. Start with Failure Trend Chart (highest value, medium complexity)
2. Add Component Health Dashboard
3. Implement Flaky Test Detection (requires historical data)
4. Add Historical Comparison

---

🤖 Generated with Claude Code