Feature: Add analytics and telemetry for observability

[ojfbot]

Summary

Add comprehensive analytics and telemetry to track user behavior, performance metrics, errors, and usage patterns for better product insights and debugging.

Motivation

Currently, the application lacks visibility into:

• How users interact with features (file uploads, chat, summaries)
• Performance bottlenecks (PDF parsing times, LLM response latency)
• Error rates and types
• Feature adoption and usage patterns
• User journey and conversion funnels

Proposed Implementation

1. Event Tracking

Track key user actions:

• File uploads (type, size, success/failure)
• Chat interactions (message sent, streaming started/completed)
• Modal opens/closes (preview, chat)
• Navigation between tabs
• Feature usage (summarize, preview, download)

2. Performance Metrics

Measure and log:

• PDF parsing duration
• LLM response time (time to first token, total time)
• File upload latency
• API request/response times

3. Error Tracking

Capture and report:

• API errors with context (endpoint, status code, user action)
• PDF parsing failures (file type, size, error message)
• Chat streaming errors
• Client-side exceptions with stack traces

4. Usage Analytics

Aggregate data for:

• Daily/weekly active users (browser local tracking)
• Feature usage frequency
• Session duration
• File type distribution
• Average file sizes processed

Technical Approach

Option 1: Simple Console Logging (Phase 1)

Start with structured console logging for development:

// packages/agent-core/src/utils/telemetry.ts
export const telemetry = {
  trackEvent: (name: string, properties?: Record<string, any>) => {
    console.log('[TELEMETRY]', { name, properties, timestamp: new Date().toISOString() })
  },
  trackPerformance: (name: string, duration: number, metadata?: Record<string, any>) => {
    console.log('[PERFORMANCE]', { name, duration, metadata, timestamp: new Date().toISOString() })
  },
  trackError: (error: Error, context?: Record<string, any>) => {
    console.error('[ERROR]', { error: error.message, stack: error.stack, context })
  }
}

Option 2: Integration with Analytics Service (Phase 2)

Consider privacy-focused options:

• Posthog (self-hosted or cloud) - Open source, feature flags + analytics
• Plausible (self-hosted or cloud) - Privacy-focused, GDPR compliant
• Simple file-based logging - Write to local files for self-hosted deployments

Implementation Locations

Frontend (packages/browser-app/src/):

• src/utils/telemetry.ts - Client-side telemetry wrapper
• Track UI interactions, navigation, errors
• Respect user privacy preferences

Backend (packages/api/src/):

• src/middleware/telemetry.ts - Request/response timing
• src/services/telemetry-service.ts - Aggregate metrics
• Track API performance, errors, resource usage

Agent Core (packages/agent-core/src/):

• src/utils/telemetry.ts - Shared telemetry utilities
• Track PDF parsing, LLM calls, data operations

Privacy Considerations

• No PII tracking: Never log personal information, file contents, or chat messages
• Opt-in/opt-out: Provide user setting to disable telemetry
• Local-only option: Support fully local deployments with file-based logging
• Data retention: Clear retention policy (e.g., 30 days)
• Transparency: Document what data is collected in privacy policy

Example Usage

// Track file upload
telemetry.trackEvent('file_upload', {
  fileType: file.type,
  fileSize: file.size,
  success: true
})

// Track performance
const start = Date.now()
const result = await parsePDF(buffer)
telemetry.trackPerformance('pdf_parse', Date.now() - start, {
  pageCount: result.pageCount,
  wordCount: result.wordCount
})

// Track errors
try {
  await bioFilesApi.getSummary(fileId)
} catch (error) {
  telemetry.trackError(error, {
    action: 'get_summary',
    fileId
  })
  throw error
}

Implementation Phases

Phase 1: Foundation (this PR)

• Create telemetry utility module
• Add structured console logging
• Track file uploads and parsing
• Track chat interactions
• Track errors

Phase 2: Performance Monitoring

• Add performance timing for all async operations
• Create performance dashboard/logs
• Set up alerts for slow operations

Phase 3: Analytics Integration (optional)

• Evaluate analytics providers (Posthog, Plausible)
• Implement privacy controls
• Add user consent UI
• Create analytics dashboard

Success Metrics

• All critical user actions tracked (upload, chat, preview, download)
• Performance metrics available for all LLM calls and PDF parsing
• Error rate and types visible in logs
• No PII or sensitive data logged
• Telemetry overhead < 5ms per operation

Related Issues

Part of PR #57 review feedback: #57

Priority

Low-Medium - Important for production readiness but not blocking core features