Add chapters to video transcripts

.gitignore

@@ -306,6 +306,15 @@ paket-files/
 __pycache__/
 *.pyc
 
+# Python virtual environments
+venv/
+.venv/
+env/
+.env
+ENV/
+env.bak/
+venv.bak/
+
 # Cake - Uncomment if you are using it
 # tools/**
 # !tools/packages.config

video-transcripts/README.md

@@ -1,7 +1,11 @@
 # YouTube Transcripts
 
-This directory contains a small script which fetches transcripts from the YouTube API, so that we can 
-keep text/markdown corresponding to the transcripts of the videos that the community produces.
+This directory contains a script that fetches transcripts from YouTube videos and generates markdown files with:
+- Video metadata (title, description, publish date)
+- AI-generated summary of the content
+- Chapters/timestamps for easy navigation
+- Cleaned up transcript with proper formatting
+- Raw YouTube transcript for reference
 
 ## Setup
 
@@ -11,21 +15,117 @@ keep text/markdown corresponding to the transcripts of the videos that the commu
 
 ## Configure Environment
 
-Edit `.env` file and set `API_KEY` to the correct value for YouTube
+Copy the example environment file and configure your API keys:
+
+```bash
+cp env.example .env
+```
+
+Edit `.env` file and set `API_KEY` to the correct value for YouTube.
 Optionally, add `OPENAI_API_KEY` if you intend to use the categorizer
 
-## Run & Pull Transcripts!
+- **`API_KEY`** (required): YouTube Data API v3 key for fetching video data and transcripts
+- **`OPENAI_API_KEY`** (optional): OpenAI API key for AI-enhanced features:
+  - Generates concise summaries of video content
+  - Creates YouTube-style chapter timestamps (e.g., `00:00:00 Introduction`)
+  - Cleans up raw transcripts into readable markdown format
+
+If `OPENAI_API_KEY` is not provided, the script will still fetch and save raw transcripts, but summaries and chapters will be unavailable.
 
-Typical usage will involve a YouTube Channel ID, which is in the [URL for the channel](https://www.youtube.com/channel/UCHZDBZTIfdy94xMjMKz-_MA)
+## Usage
+
+### Basic Usage - Channel
+
+Fetch transcripts from a YouTube channel (defaults to OpenTelemetry channel):
+
+```bash
+python3 transcripts.py --channel UCHZDBZTIfdy94xMjMKz-_MA --path ./transcripts
+```
 
-From within this directory, we'd place transcripts in that directory off of the root.
+### Fetch from Playlist
 
+```bash
+python3 transcripts.py --playlist PLDGkOdUX1UjrEOz4fOB4UZW8m-hx8_mtb --path ./transcripts
 ```
+
+### Command-Line Options
+
+- `-c, --channel`: YouTube Channel ID (default: OpenTelemetry channel)
+- `-p, --playlist`: YouTube Playlist ID (no default)
+- `-s, --start`: Start date for videos in ISO 8601 format (default: `2023-01-01T00:00:00Z`)
+- `-e, --end`: End date for videos in ISO 8601 format (default: `2030-12-31T00:00:00Z`)
+- `-d, --path`: Directory to write markdown files to (default: `./transcripts/`)
+- `-l, --limit`: Limit the number of videos to process (useful for testing)
+
+### Example with Date Range and Limit
+
+```bash
 python3 transcripts.py \
     --channel UCHZDBZTIfdy94xMjMKz-_MA \
+    --start 2024-01-01T00:00:00Z \
+    --end 2024-12-31T00:00:00Z \
+    --limit 5 \
     --path ./transcripts
 ```
 
+## Output Format
+
+Each video generates a markdown file named: `{publish_date}-{slugified-title}.md`
+
+The generated markdown includes:
+- **Title and Metadata**: Video title, publish date, description, and YouTube URL
+- **Summary**: AI-generated one-paragraph overview of the video content
+- **Chapters**: YouTube-style timestamps identifying key moments (e.g., `00:00:00 Introduction`)
+- **Cleaned Transcript**: Readable, formatted transcript with proper paragraphs
+- **Raw Transcript**: Original YouTube auto-generated transcript for reference
+
+## Features & Reliability
+
+### Rate Limiting
+The script includes comprehensive rate limiting to avoid YouTube API quota issues:
+- Small delays between video transcript fetches (2-5 seconds) to prevent rate limiting
+- Automatic 60-120 second wait and retry when rate limits (429 errors) are hit
+- Exponential backoff for transient errors
+- Detects and reports IP blocking vs. temporary rate limiting
+
+### Error Handling
+- Exponential backoff with up to 3 retry attempts for transcript fetching
+- Graceful handling of videos with disabled transcripts
+- Validates transcript quality (minimum length, filters music/applause markers)
+- Skips videos that already have transcripts (resumes interrupted runs)
+
+### Language Support
+Attempts to fetch transcripts in multiple languages: English, Spanish, French, German, Japanese
+
+## Troubleshooting
+
+### YouTube IP Blocking
+
+If you encounter "YouTube is blocking requests from your IP" errors, this means YouTube has blocked your IP address from accessing their transcript API. This is different from rate limiting and won't clear quickly.
+
+**Common causes:**
+- Repeated testing/debugging over multiple days
+- Using a cloud provider IP (AWS, GCP, Azure, etc.)
+- Using certain VPN services
+- Your ISP's IP range being flagged
+
+**Solutions:**
+
+1. **Wait 24-48 hours** - IP blocks usually clear after 1-2 days of no activity
+
+2. **Use a different network** - Switch to mobile hotspot, different WiFi, or different location
+
+3. **Cookie-based authentication** (Advanced):
+   - Export cookies from a logged-in YouTube session in your browser
+   - Use a browser extension like "Get cookies.txt LOCALLY" (Chrome/Firefox)
+   - Save cookies.txt in the video-transcripts directory
+   - Modify the script to use cookies (requires code changes)
+   - See [youtube-transcript-api documentation](https://github.com/jdepoix/youtube-transcript-api#cookies) for details
+
+4. **Use a residential proxy** - Cloud/datacenter IPs are often blocked, but residential IPs work better
+
+5. **Run from a different machine** - If possible, run the script from a home computer instead of a server
+
 ## Need a YouTube Key?
 
 * Have a GCP project

video-transcripts/env.example

@@ -0,0 +1,11 @@
+# YouTube API Configuration
+# Get your API key from: https://console.cloud.google.com/
+# 1. Go to a GCP project
+# 2. Make sure YouTube Data API v3 is enabled
+# 3. Create an API key in that project
+API_KEY=your_youtube_api_key_here
+
+# OpenAI API Configuration (optional)
+# Only needed if you want AI-powered summaries and cleanup
+# Get your API key from: https://platform.openai.com/api-keys
+OPENAI_API_KEY=your_openai_api_key_here

video-transcripts/requirements.txt

@@ -35,7 +35,7 @@ protobuf==5.29.5
 pyasn1==0.6.1
 pyasn1_modules==0.4.2
 pydantic==2.11.9
-pydantic_core==2.39.0
+pydantic_core==2.33.2
 pyparsing==3.2.5
 python-dateutil==2.9.0.post0
 python-dotenv==1.1.1
@@ -55,4 +55,4 @@ typing_extensions==4.15.0
 tzlocal==5.3.1
 uritemplate==4.2.0
 urllib3==2.5.0
-youtube-transcript-api==0.6.3
+youtube-transcript-api==1.2.3

video-transcripts/test_transcripts.py

@@ -0,0 +1,163 @@
+import unittest
+from transcripts import build_timeline_from_transcript, parse_timestamp_to_seconds, format_duration
+
+
+class TestTimelineBuilder(unittest.TestCase):
+    """Test the timeline building functionality for chapter generation."""
+
+    def test_build_timeline_basic(self):
+        """Test that timeline is built correctly from raw transcript data."""
+        raw_transcript = [
+            {'text': 'Hello everyone', 'start': 0, 'duration': 2},
+            {'text': 'Welcome to the show', 'start': 2, 'duration': 3},
+            {'text': 'Today we are talking about OpenTelemetry', 'start': 35, 'duration': 4},
+            {'text': 'It is very exciting', 'start': 39, 'duration': 2},
+        ]
+
+        timeline = build_timeline_from_transcript(raw_transcript, sample_interval=30)
+
+        # Should have entries at 00:00:00 and 00:00:30
+        self.assertIn('[00:00:00]', timeline)
+        self.assertIn('[00:00:30]', timeline)
+        self.assertIn('Hello everyone', timeline)
+        self.assertIn('OpenTelemetry', timeline)
+
+    def test_build_timeline_filters_music(self):
+        """Test that [Music] and similar markers are filtered out."""
+        raw_transcript = [
+            {'text': 'Hello everyone', 'start': 0, 'duration': 2},
+            {'text': '[Music]', 'start': 2, 'duration': 5},
+            {'text': '[Applause]', 'start': 7, 'duration': 3},
+            {'text': 'Welcome back', 'start': 28, 'duration': 2},  # Within ±5 window of 30s
+        ]
+
+        timeline = build_timeline_from_transcript(raw_transcript, sample_interval=30)
+
+        # Should not contain filtered markers
+        self.assertNotIn('[Music]', timeline)
+        self.assertNotIn('[Applause]', timeline)
+        # Should contain actual speech
+        self.assertIn('Hello everyone', timeline)
+        self.assertIn('Welcome back', timeline)
+
+    def test_build_timeline_empty_transcript(self):
+        """Test that empty transcript returns empty timeline."""
+        timeline = build_timeline_from_transcript([], sample_interval=30)
+        self.assertEqual(timeline, "")
+
+        timeline = build_timeline_from_transcript(None, sample_interval=30)
+        self.assertEqual(timeline, "")
+
+    def test_build_timeline_custom_interval(self):
+        """Test that custom sample intervals work correctly."""
+        raw_transcript = [
+            {'text': 'Start', 'start': 0, 'duration': 1},
+            {'text': 'At 10 seconds', 'start': 10, 'duration': 2},
+            {'text': 'At 20 seconds', 'start': 20, 'duration': 2},
+            {'text': 'At 30 seconds', 'start': 30, 'duration': 2},
+        ]
+
+        # Sample every 10 seconds
+        timeline = build_timeline_from_transcript(raw_transcript, sample_interval=10)
+
+        # Should have entries at 0, 10, 20, 30
+        self.assertIn('[00:00:00]', timeline)
+        self.assertIn('[00:00:10]', timeline)
+        self.assertIn('[00:00:20]', timeline)
+        self.assertIn('[00:00:30]', timeline)
+
+
+    def test_build_timeline_preserves_precision(self):
+        """Test that timeline captures precise timestamps, not just rounded intervals."""
+        raw_transcript = [
+            {'text': 'Starting at 3 seconds', 'start': 3, 'duration': 2},
+            {'text': 'Now at 32 seconds', 'start': 32, 'duration': 2},
+            {'text': 'And at 91 seconds', 'start': 91, 'duration': 2},
+        ]
+
+        # Sample every 30 seconds with ±5 second window
+        # 0s sample (window 0-5) captures text at 3s
+        # 30s sample (window 25-35) captures text at 32s
+        # 90s sample (window 85-95) captures text at 91s
+        timeline = build_timeline_from_transcript(raw_transcript, sample_interval=30)
+
+        # Should capture text from the sampling points
+        self.assertIn('[00:00:00]', timeline)
+        self.assertIn('[00:00:30]', timeline)
+        self.assertIn('[00:01:30]', timeline)
+        # Verify text is captured at each sample point
+        self.assertIn('3 seconds', timeline)
+        self.assertIn('32 seconds', timeline)
+        self.assertIn('91 seconds', timeline)
+
+
+class TestYouTubeAPIErrorHandling(unittest.TestCase):
+    """Test that YouTube API exceptions work as expected with our pinned version."""
+
+    def test_youtube_api_exceptions_importable(self):
+        """Test that all YouTube API exceptions we depend on can be imported.
+
+        This will fail if the API changes in a way that breaks our imports.
+        """
+        # Import all exceptions we use
+        from youtube_transcript_api import TranscriptsDisabled, NoTranscriptFound, VideoUnavailable
+        from youtube_transcript_api._errors import YouTubeRequestFailed
+
+        # Verify they're actual exception classes
+        self.assertTrue(issubclass(TranscriptsDisabled, Exception))
+        self.assertTrue(issubclass(NoTranscriptFound, Exception))
+        self.assertTrue(issubclass(VideoUnavailable, Exception))
+        self.assertTrue(issubclass(YouTubeRequestFailed, Exception))
+
+    def test_youtube_api_exceptions_catchable(self):
+        """Test that YouTube API exceptions can be caught.
+
+        This validates that exceptions raised by the library can be caught
+        and their error messages can be inspected (as our code does).
+        """
+        from youtube_transcript_api import TranscriptsDisabled, NoTranscriptFound
+
+        # Test we can catch TranscriptsDisabled
+        with self.assertRaises(TranscriptsDisabled):
+            raise TranscriptsDisabled("video_id")
+
+        # Test we can catch NoTranscriptFound
+        with self.assertRaises(NoTranscriptFound):
+            raise NoTranscriptFound("video_id", [], "en")
+
+
+class TestTimestampParsing(unittest.TestCase):
+    """Test timestamp parsing and formatting functions."""
+
+    def test_parse_timestamp_to_seconds(self):
+        """Test converting timestamp strings to seconds."""
+        self.assertEqual(parse_timestamp_to_seconds('00:00:00'), 0)
+        self.assertEqual(parse_timestamp_to_seconds('00:01:00'), 60)
+        self.assertEqual(parse_timestamp_to_seconds('00:05:30'), 330)
+        self.assertEqual(parse_timestamp_to_seconds('01:00:00'), 3600)
+        self.assertEqual(parse_timestamp_to_seconds('01:23:45'), 5025)
+
+        # Test MM:SS format
+        self.assertEqual(parse_timestamp_to_seconds('05:30'), 330)
+        self.assertEqual(parse_timestamp_to_seconds('1:00'), 60)
+
+    def test_format_duration(self):
+        """Test formatting seconds into HH:MM:SS."""
+        self.assertEqual(format_duration(0), '00:00:00')
+        self.assertEqual(format_duration(60), '00:01:00')
+        self.assertEqual(format_duration(330), '00:05:30')
+        self.assertEqual(format_duration(3600), '01:00:00')
+        self.assertEqual(format_duration(5025), '01:23:45')
+
+    def test_timestamp_roundtrip(self):
+        """Test that parsing and formatting are inverse operations."""
+        timestamps = ['00:00:00', '00:05:30', '01:23:45', '02:00:00']
+        for ts in timestamps:
+            seconds = parse_timestamp_to_seconds(ts)
+            formatted = format_duration(seconds)
+            self.assertEqual(formatted, ts)
+
+
+if __name__ == '__main__':
+    unittest.main()
+