diff --git a/CLAUDE.md b/CLAUDE.md index 8671043..21d412f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,16 +2,26 @@ This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. +> **Loading discipline (ADR-0081):** this root file is the always-loaded Layer 0 — repo identity, data-safety/fair-use invariants, and the command quick-start. Deeper, path-conditional material is routed out: forum-scraper internals → `deakins_forums/CLAUDE.md` (loads when editing that subtree); record schemas → `documentation/data-model-reference.md`; structure / podcast-ingest / testing reference → `documentation/repo-reference.md`. + ## Overview This is a **Team Deakins Podcast and Forum Knowledge Base** project. It contains: 1. **Podcast Episode Downloader & Ingest**: Scripts to download MP3s from RSS feeds and organize them into structured episode directories with metadata and transcripts. -2. **Forum Scraper (`deakins_forums`)**: A modular, ultra-structured Python package for scraping rogerdeakins.com forums into a searchable JSON-based knowledge base. -3. **TypeScript UI Layer (`packages/`)**: A Module Federation remote (React/Vite micro-frontend on port 3020 + Express API on port 3021) for exploring podcast and forum data locally. UI components are imported from `@ojfbot/frame-ui-components` (npm `^1.0.1`) (DashboardLayout, ChatShell, ChatMessage, MarkdownMessage, ErrorBoundary, ThreadSidebar, CondensedChat). Exposes a `GET /api/beads` endpoint per the fleet-wide ADR-0016 bead projection contract. Architecture decisions documented in ADR-006 through ADR-009, ADR-0016 (bead projection), and ADR-0030 (shared component architecture). +2. **Forum Scraper (`deakins_forums`)**: A modular, ultra-structured Python package for scraping rogerdeakins.com forums into a searchable JSON-based knowledge base. (Design + CLI: `deakins_forums/CLAUDE.md`.) +3. **TypeScript UI Layer (`packages/`)**: A Module Federation remote (React/Vite micro-frontend on port 3020 + Express API on port 3021) for exploring podcast and forum data locally. Exposes `GET /api/beads` per the fleet-wide ADR-0016 bead projection contract. Architecture decisions: ADR-006 through ADR-009, ADR-0016, ADR-0030. 4. **Standalone Flask UI (`app.py`)**: A single-file dark-theme knowledge browser at `localhost:5050`, reading directly from `downloads/` and `library/forums/`. Zero dependency on the Module Federation stack — used for local debugging against the raw corpus. + +## Fair Use & Privacy (must-keep) + All content is for **personal research and educational purposes only** under fair use principles. Commercial use requires explicit permission from copyright holders. +- **Scraped data stays off-git** — the repo is public, but scraped forum content (`library/`), podcast audio (`downloads/`), and exports (`analysis/`) are gitignored and never committed. Only source code and illustrative schema examples are public. See Git Data Policy below. +- **Personal/educational use only** — no commercial redistribution. Contact rogerdeakins.com / Team Deakins for commercial licensing. +- Scripts identify research purpose via User-Agent headers; rate limiting is enforced (3s default); respect `robots.txt` and server resources. +- All content remains property of original copyright holders. + ## Git Data Policy **What is committed to git:** @@ -28,50 +38,6 @@ All content is for **personal research and educational purposes only** under fai **Schema examples vs. actual data:** The `documentation/examples-schemas/` directory is the git-committed representation of the data layer. Each file shows the shape of a record type with fictional illustrative content — not real scraped data. See `documentation/examples-schemas/README.md` for field-level documentation. -## Project Structure - -``` -purefoy/ -├── deakins_forums/ # Forum scraper package (structured, MCP-ready) -│ ├── cli.py # Command-line interface -│ ├── pipeline.py # High-level scraping orchestration -│ ├── models.py # Pydantic data models (PostLeaf, TopicLeaf, etc.) -│ ├── parser_bbpress.py # HTML parsing logic for bbPress forums -│ ├── http_client.py # Rate-limited HTTP client with ETag support -│ ├── store_json.py # JSON "leaf" persistence layer -│ ├── index_sqlite.py # SQLite FTS5 search index -│ ├── normalize.py # Content extraction (links, quotes, blocks) -│ ├── export.py # Export utilities (CSV, text, by-author, etc.) -│ ├── coverage.py # Coverage tracking for incremental scrapes -│ ├── query_tracker.py # Query provenance/lineage tracking -│ ├── report.py # Reporting utilities -│ └── config.py # Settings with env var overrides -├── packages/ # TypeScript UI layer (pnpm workspace) -│ ├── browser-app/ # React/Vite micro-frontend remote (port 3020) — UI via @ojfbot/frame-ui-components (npm ^1.0.1) -│ ├── browser-automation/ # Visual regression testing via browser automation (screenshot capture, CI integration) -│ ├── api/ # Express API over flat JSON + SQLite (port 3021) — includes GET /api/beads (ADR-0016) -│ └── shared/ # @purefoy/shared — generated OpenAPI schema + API type contracts (incl. GoalManifest, ReviewProgress) -├── app.py # Standalone Flask UI — dark-theme knowledge browser (port 5050) -├── download_episodes.py # Podcast RSS feed downloader (standalone) -├── ingest_teamdeakins_downloads.py # Episode ingest with metadata & transcripts -├── scrape_forum.py # Legacy forum scraper (standalone) -├── library/ # Output directory for scraped data -│ └── forums/ # Forum JSON leafs organized by type -│ ├── posts/ # Individual post JSON files -│ ├── topics/ # Topic index files -│ ├── forums/ # Forum metadata files -│ └── _site/ # Sqlite index, HTTP state, coverage -├── downloads/ # Podcast episode directories (348/348 episodes transcribed) -│ └── S02E176__2026-01-14__guest-name__libsyn_abc123/ -│ ├── audio.mp3 -│ ├── metadata.json -│ └── transcript/ -├── analysis/ # Export output (CSV, text dumps, stats) -├── package.json # pnpm workspace root -└── pyproject.toml # Python package metadata -``` -``` - ## Deployment **NEVER deploy directly to production** via CLI (`vercel deploy --prod`, `aws s3 sync` to prod paths, etc.). @@ -96,57 +62,6 @@ pip install -r requirements.txt pip install -e . ``` -### Podcast Episode Management - -```bash -# Download MP3s from RSS feed -.venv/bin/python3 download_episodes.py - -# Organize downloads into episode directories (dry-run) -.venv/bin/python3 ingest_teamdeakins_downloads.py \ - --rss "https://rss.libsyn.com/shows/265448/destinations/2018942.xml" \ - --downloads "./downloads" \ - --dry-run - -# Actually ingest with transcript fetching -.venv/bin/python3 ingest_teamdeakins_downloads.py \ - --rss "https://rss.libsyn.com/shows/265448/destinations/2018942.xml" \ - --downloads "./downloads" \ - --fill-transcripts \ - --tapesearch -``` - -### Forum Scraping (deakins_forums package) - -```bash -# Using the CLI entry point (if installed) -deakins-forums scrape-all --max-topics 1000 --build-index -deakins-forums scrape-forum team-deakins --max-pages 20 --build-index -deakins-forums scrape-topic https://rogerdeakins.com/forums/topic/... - -# Or using python -m -python -m deakins_forums.cli scrape-forum team-deakins --max-pages 20 -python -m deakins_forums.cli build-index -python -m deakins_forums.cli search "lighting techniques" --limit 20 -python -m deakins_forums.cli stats - -# Export data -python -m deakins_forums.cli export all-text -o analysis/all_posts.txt -python -m deakins_forums.cli export csv -o analysis/posts.csv -python -m deakins_forums.cli export roger-only -o analysis/roger_posts.txt - -# Query tracking (provenance) -python -m deakins_forums.cli scrape-forum team-deakins \ - --query-name "Research natural lighting techniques" \ - --querier-role "Cinematographer" \ - --querier-department "Camera" \ - --query-context "TV miniseries prep" \ - --query-intent "Find Team Deakins insights on natural light" - -python -m deakins_forums.cli provenance --topic "team-deakins__topic-slug" -python -m deakins_forums.cli coverage --format full -``` - ### Code Quality (optional-dependencies) ```bash @@ -160,218 +75,11 @@ ruff check . pytest ``` -## Architecture - -### Forum Scraper Design - -The `deakins_forums` package follows a **composable pipeline architecture**: - -1. **HttpClient** (`http_client.py`): Rate-limited HTTP layer with conditional GET (ETag/Last-Modified) support for efficient incremental updates. - -2. **Parser** (`parser_bbpress.py`): BeautifulSoup-based HTML extraction for bbPress forum structure. Extracts forums index, forum pages, topic pages, and posts. - -3. **Models** (`models.py`): Pydantic models defining the "leaf" JSON structure (v2 schema): - - `PostLeaf`: Individual post with content, author, timestamps, structured blocks, quotes, links, media, **plus threading fields** (post_type, parent_post_id, parent_type, position) - - `TopicLeaf`: Topic metadata + post ID index (posts are separate files) **plus reply tree** (recursive nested structure showing conversation hierarchy with thread statistics) - - `ForumLeaf`: Forum metadata + topic reference list - - `PostType` enum: Distinguishes "topic" (starter) from "reply" posts - - All include `Provenance` (source URL, scrape timestamp, HTTP headers) and `Integrity` (content hash for change detection, parser_version tracks schema) - -4. **Pipeline** (`pipeline.py`): High-level orchestration gluing fetch → parse → normalize → store. Handles pagination, deduplication, incremental refresh. - -5. **Storage** (`store_json.py`): Persists structured JSON "leafs" with deterministic file paths (e.g., `posts/{post_id}.json`). Enables version control and incremental updates. - -6. **Index** (`index_sqlite.py`): SQLite FTS5 full-text search index for fast queries across posts. Rebuilt from JSON leafs on demand. - -7. **Normalize** (`normalize.py`): Extracts structured content elements (links, media, quotes, content blocks) from raw HTML/text for agent-friendly consumption. - -8. **Query Tracking** (`query_tracker.py`): Records provenance of each scrape query (who, why, when, what was accessed) for research auditability. - -### Key Design Principles - -- **Incremental & Resumable**: HTTP ETag caching, content hashing, and visited URL tracking allow efficient re-scraping -- **Deterministic Paths**: Slug-based file naming enables easy navigation and version control -- **Agent-Friendly**: Ultra-structured JSON with normalized content blocks, quotes, links for downstream AI/MCP tools -- **Provenance-First**: Every leaf includes source URL, scrape timestamp, HTTP headers, content hash -- **Rate-Limited & Respectful**: Configurable delays (default 3s), User-Agent identification, robots.txt compliance - -### Episode Ingest Design - -The `ingest_teamdeakins_downloads.py` script: -- Parses RSS XML to extract episode metadata (title, date, duration, iTunes tags, description) -- Matches downloaded MP3s to RSS entries via URL -- Creates structured episode directories with naming: `S{season}E{episode}__{date}__{guest}__{source_id}/` -- Generates `metadata.json` with comprehensive provenance -- Creates `transcript/` scaffold with `sources.json` and `transcript.txt` (placeholder or fetched) -- Supports Podcasting 2.0 `` URLs and optional Tapesearch fallback - -## Environment Variables - -Configure via environment variables (all optional): - -```bash -export DEAKINS_BASE_URL="https://rogerdeakins.com" -export DEAKINS_USER_AGENT="Your Custom User Agent" -export DEAKINS_DELAY_S="3.0" # Seconds between requests -export DEAKINS_TIMEOUT_S="30" # Request timeout -export DEAKINS_MAX_RETRIES="3" -export DEAKINS_OUT_DIR="./library/forums" -``` - -## Data Model Reference - -### PostLeaf Structure (v2 Schema) - -**NEW in v2:** Threading support with parent-child relationships and post types. - -```json -{ - "ids": { - "post_id": "175763", - "forum_slug": "team-deakins", - "topic_slug": "lighting-discussion", - "reply_permalink": "https://...", - "parent_post_id": "175760", - "parent_type": "topic", - "position": 1 - }, - "post_type": "reply", - "author": { - "display_name": "Roger A. Deakins", - "role": "Keymaster" - }, - "timestamps": { - "raw": "January 18, 2023 at 3:32 pm #175763", - "parsed_iso": "2023-01-18T15:32:00", - "parse_confidence": "medium" - }, - "content_text": "...", - "content_html": "...", - "blocks": [...], - "quotes": [...], - "links": [...], - "media": [...], - "provenance": { - "source_url": "...", - "scraped_at": "2026-01-15T...", - "http": {"etag": "...", "last_modified": "..."} - }, - "integrity": { - "content_hash": "sha256:...", - "parser_version": "bbpress-v2" - } -} -``` - -**Threading Fields:** -- `post_type`: "topic" (starter) or "reply" -- `parent_post_id`: ID of parent post (null for topics) -- `parent_type`: "topic" or "reply" (null for topics) -- `position`: Sequential position in reply list (1-indexed) - -### TopicLeaf Structure (v2 Schema) - -**NEW in v2:** Hierarchical reply tree with thread statistics. - -```json -{ - "topic_url": "https://...", - "topic_slug": "lighting-discussion", - "title": "Lighting Discussion", - "post_ids": ["175760", "175763", "175764"], - "reply_tree": { - "post_id": "175760", - "post_type": "topic", - "author": "John Doe", - "timestamp": "2023-01-18T15:00:00", - "children": [ - { - "post_id": "175763", - "post_type": "reply", - "author": "Roger A. Deakins", - "timestamp": "2023-01-18T15:32:00", - "children": [] - }, - { - "post_id": "175764", - "post_type": "reply", - "author": "Jane Smith", - "timestamp": "2023-01-18T16:00:00", - "children": [] - } - ] - }, - "reply_count": 2, - "max_depth": 1, - "provenance": {...}, - "integrity": { - "content_hash": "sha256:...", - "parser_version": "bbpress-v2" - } -} -``` - -**Reply Tree Fields:** -- `reply_tree`: Recursive nested structure showing conversation flow -- `reply_count`: Total number of replies (excluding topic starter) -- `max_depth`: Maximum nesting depth (0 = no replies, 1 = direct replies only) -- `post_ids`: Flat list for backward compatibility - -**Migration Note:** Use `python -m deakins_forums.migrate --rebuild-trees` to rebuild reply trees for existing data. - -### Episode metadata.json Structure - -```json -{ - "title": "SEASON 2 - EPISODE 176 - Chris Lowe...", - "guid": "...", - "pubDate_iso": "2026-01-14", - "itunes": { - "season": 2, - "episode": 176, - "duration_s": 5400, - "summary": "..." - }, - "rss_description": "...", - "rss_content_encoded": "...", - "audio": { - "url": "https://...", - "mime": "audio/mpeg", - "size_bytes": 123456789, - "bitrate_kbps": 128, - "duration_s": 5400, - "format": "MP3" - }, - "provenance": { - "rss_url": "...", - "scraped_at": "2026-01-15T...", - "script_version": "ingest-v2" - } -} -``` - -## Testing Notes - -- **TypeScript layer**: CI runs TypeScript type-check, test, and codegen drift guard jobs (see `packages/`). Fleet-wide visual regression tests run via the `browser-automation` package (screenshots capture the composed shell, not standalone apps; waits for a Frame sentinel rather than `networkidle`). -- **Python layer**: Manual testing via CLI commands and dry-run flags; no formal Python test suite yet -- Incremental updates can be verified by running scrapes twice and checking for "Not modified" messages - -## Important Fair Use Reminders - -- **Personal/educational use only** - no commercial redistribution -- **Keep repository PRIVATE** - scraped forum data is included for research -- Scripts include User-Agent headers identifying research purpose -- Rate limiting is enforced (3s default delay between requests) -- Respect robots.txt and server resources -- All content remains property of original copyright holders -- Contact rogerdeakins.com / Team Deakins for commercial licensing - -## Git Repository Notes +**Forum scraping commands** (scrape, index, search, export, provenance) → `deakins_forums/CLAUDE.md`. +**Podcast download & ingest commands** → `documentation/repo-reference.md`. -This repository previously included scraped forum data (`library/forums/`) but that directory is now gitignored (see Git Data Policy above). On-disk data is for private research use: -- Forum content is publicly accessible (not behind authentication) -- Structured JSON format is git-friendly -- Enables incremental updates and change tracking -- **Keep this repository PRIVATE** - do not make public +## Architecture & data model -Audio files (`.mp3`) and episode metadata are excluded via `.gitignore`. +- **Forum scraper pipeline + design principles** → `deakins_forums/CLAUDE.md` +- **Record schemas** (PostLeaf, TopicLeaf, Episode metadata; authoritative source `deakins_forums/models.py`) → `documentation/data-model-reference.md` +- **Project structure, episode-ingest design, testing notes** → `documentation/repo-reference.md` diff --git a/GIT_READY.md b/GIT_READY.md index ab9248a..793ccf8 100644 --- a/GIT_READY.md +++ b/GIT_READY.md @@ -104,7 +104,7 @@ purefoy/ - `PROJECT_STRUCTURE.md` - Structure guide - `GIT_READY.md` - This file -**Data (Private Repo):** +**Data (on-disk only — gitignored, not committed):** - `library/forums/` - 3,075 post JSONs, 691 topic JSONs (24MB) - `library/articles/` - Sample article JSONs @@ -181,20 +181,18 @@ purefoy/ ## Git Repository Configuration ### Repository Type -**Recommended:** Private repository +**Adopted:** Public repository, code only — scraped data kept off-git. + +**Configuration:** +1. `library/`, `downloads/`, and `analysis/` are in `.gitignore` +2. Only source code, documentation, and illustrative schema examples are committed +3. All ~24MB of scraped data stays on-disk (never committed) +4. Content copyright disclaimer in README / CLAUDE.md **Reasoning:** - Forum content is publicly accessible but should not be redistributed - Personal research and educational use only -- Includes 24MB of scraped JSON data (library/) -- Respects copyright while enabling version control - -### Alternative: Public Code Only -If making code public: -1. Add `library/` to `.gitignore` -2. Only commit source code and documentation -3. Exclude all scraped data -4. Add prominent disclaimer about content copyright +- Respects copyright while keeping the codebase open --- @@ -282,7 +280,7 @@ git push -u origin main ## Next Steps After Git Init 1. **Create Remote Repository** - - GitHub, GitLab, or Gitea (private) + - GitHub, GitLab, or Gitea - Add remote: `git remote add origin ` 2. **Push Initial Commit** diff --git a/PROJECT_STRUCTURE.md b/PROJECT_STRUCTURE.md index 2ba0148..a03f3a2 100644 --- a/PROJECT_STRUCTURE.md +++ b/PROJECT_STRUCTURE.md @@ -78,7 +78,7 @@ __pycache__/ # Python bytecode - **`deakins_articles/`**: Articles scraper for rogerdeakins.com/articles ### Data Storage -- **`library/`**: Primary data storage with JSON "leafs" (TRACKED in git for private repo) +- **`library/`**: Primary data storage with JSON "leafs" (gitignored — on-disk only; public repo keeps scraped data off-git) - Structured, version-controllable JSON files - Enables incremental updates via content hashing - Ready for AI/MCP consumption diff --git a/README.md b/README.md index 79640ca..cd68e1d 100644 --- a/README.md +++ b/README.md @@ -126,7 +126,7 @@ Scrapes technical articles from rogerdeakins.com/articles: ### Installation ```bash -# Clone repository (private repo assumed) +# Clone repository git clone cd purefoy @@ -312,9 +312,13 @@ http_state.json # HTTP cache (ephemeral) **What IS tracked:** - ✅ All source code (`deakins_forums/`, `deakins_articles/`) -- ✅ Library data (`library/`) - Structured JSON for private repo - ✅ Documentation and guides - ✅ Configuration files +- ✅ Illustrative schema examples (`documentation/examples-schemas/`) + +**What is NOT tracked (gitignored, on-disk only):** +- ❌ Scraped forum content (`library/`) — public repo, so data stays off-git +- ❌ Podcast audio + episode dirs (`downloads/`), exports (`analysis/`) --- @@ -364,7 +368,7 @@ See [CLAUDE.md](CLAUDE.md) for detailed architecture documentation. **Permitted Use:** - ✅ Personal research and education -- ✅ Private knowledge base for learning +- ✅ Personal knowledge base for learning - ✅ Academic study of cinematography **Prohibited Use:** @@ -377,21 +381,14 @@ See [CLAUDE.md](CLAUDE.md) for detailed architecture documentation. --- -### Git Repository Privacy - -**⚠️ This MUST be a PRIVATE repository** - -The project is configured for a private git repository: +### Git Data Policy -- `library/` directory contains JSON data (forum content) tracked in the private repo -- Forum content is publicly accessible but should not be publicly redistributed -- Use for personal/educational research only -- Do not push to public GitHub/GitLab +This is a **public repository** — scraped data is kept off-git, not behind repo visibility. -**If making code public:** -- Exclude `library/` directory entirely -- Include only source code and documentation -- Add clear disclaimers about content copyright +- `library/` (forum content), `downloads/` (podcast audio), and `analysis/` (exports) are **gitignored** and never committed. +- Only source code, documentation, and illustrative schema examples (`documentation/examples-schemas/`) are public. +- Forum content is publicly accessible but should not be redistributed; use for personal/educational research only. +- Content remains copyright of the original holders — see Copyright & Usage above. --- diff --git a/deakins_forums/CLAUDE.md b/deakins_forums/CLAUDE.md new file mode 100644 index 0000000..33ce83e --- /dev/null +++ b/deakins_forums/CLAUDE.md @@ -0,0 +1,83 @@ +# deakins_forums — scraper subtree guidance + +Path-conditional guidance (ADR-0081 Layer 1): this file loads when you edit the +`deakins_forums/` forum-scraper package. Repo-wide policy lives in the root `CLAUDE.md`; +the leaf/record schemas live in `documentation/data-model-reference.md` (authoritative +source: `deakins_forums/models.py`). + +## Forum Scraper Design + +The `deakins_forums` package follows a **composable pipeline architecture**: + +1. **HttpClient** (`http_client.py`): Rate-limited HTTP layer with conditional GET (ETag/Last-Modified) support for efficient incremental updates. + +2. **Parser** (`parser_bbpress.py`): BeautifulSoup-based HTML extraction for bbPress forum structure. Extracts forums index, forum pages, topic pages, and posts. + +3. **Models** (`models.py`): Pydantic models defining the "leaf" JSON structure (v2 schema): + - `PostLeaf`: Individual post with content, author, timestamps, structured blocks, quotes, links, media, **plus threading fields** (post_type, parent_post_id, parent_type, position) + - `TopicLeaf`: Topic metadata + post ID index (posts are separate files) **plus reply tree** (recursive nested structure showing conversation hierarchy with thread statistics) + - `ForumLeaf`: Forum metadata + topic reference list + - `PostType` enum: Distinguishes "topic" (starter) from "reply" posts + - All include `Provenance` (source URL, scrape timestamp, HTTP headers) and `Integrity` (content hash for change detection, parser_version tracks schema) + +4. **Pipeline** (`pipeline.py`): High-level orchestration gluing fetch → parse → normalize → store. Handles pagination, deduplication, incremental refresh. + +5. **Storage** (`store_json.py`): Persists structured JSON "leafs" with deterministic file paths (e.g., `posts/{post_id}.json`). Enables version control and incremental updates. + +6. **Index** (`index_sqlite.py`): SQLite FTS5 full-text search index for fast queries across posts. Rebuilt from JSON leafs on demand. + +7. **Normalize** (`normalize.py`): Extracts structured content elements (links, media, quotes, content blocks) from raw HTML/text for agent-friendly consumption. + +8. **Query Tracking** (`query_tracker.py`): Records provenance of each scrape query (who, why, when, what was accessed) for research auditability. + +### Key Design Principles + +- **Incremental & Resumable**: HTTP ETag caching, content hashing, and visited URL tracking allow efficient re-scraping +- **Deterministic Paths**: Slug-based file naming enables easy navigation and version control +- **Agent-Friendly**: Ultra-structured JSON with normalized content blocks, quotes, links for downstream AI/MCP tools +- **Provenance-First**: Every leaf includes source URL, scrape timestamp, HTTP headers, content hash +- **Rate-Limited & Respectful**: Configurable delays (default 3s), User-Agent identification, robots.txt compliance + +## Environment Variables + +Configure via environment variables (all optional): + +```bash +export DEAKINS_BASE_URL="https://rogerdeakins.com" +export DEAKINS_USER_AGENT="Your Custom User Agent" +export DEAKINS_DELAY_S="3.0" # Seconds between requests +export DEAKINS_TIMEOUT_S="30" # Request timeout +export DEAKINS_MAX_RETRIES="3" +export DEAKINS_OUT_DIR="./library/forums" +``` + +## Forum Scraping commands + +```bash +# Using the CLI entry point (if installed) +deakins-forums scrape-all --max-topics 1000 --build-index +deakins-forums scrape-forum team-deakins --max-pages 20 --build-index +deakins-forums scrape-topic https://rogerdeakins.com/forums/topic/... + +# Or using python -m +python -m deakins_forums.cli scrape-forum team-deakins --max-pages 20 +python -m deakins_forums.cli build-index +python -m deakins_forums.cli search "lighting techniques" --limit 20 +python -m deakins_forums.cli stats + +# Export data +python -m deakins_forums.cli export all-text -o analysis/all_posts.txt +python -m deakins_forums.cli export csv -o analysis/posts.csv +python -m deakins_forums.cli export roger-only -o analysis/roger_posts.txt + +# Query tracking (provenance) +python -m deakins_forums.cli scrape-forum team-deakins \ + --query-name "Research natural lighting techniques" \ + --querier-role "Cinematographer" \ + --querier-department "Camera" \ + --query-context "TV miniseries prep" \ + --query-intent "Find Team Deakins insights on natural light" + +python -m deakins_forums.cli provenance --topic "team-deakins__topic-slug" +python -m deakins_forums.cli coverage --format full +``` diff --git a/documentation/GIT_INIT_CHECKLIST.md b/documentation/GIT_INIT_CHECKLIST.md index 9397326..c2b02c9 100644 --- a/documentation/GIT_INIT_CHECKLIST.md +++ b/documentation/GIT_INIT_CHECKLIST.md @@ -94,18 +94,13 @@ git ls-files --others --exclude-standard **Schema Examples:** - `schema_examples/**/*` -**Forum Scraped Data (for private repo):** -- `library/forums/posts/*.json` -- `library/forums/topics/*.json` -- `library/forums/forums/*.json` -- `library/forums/_site/forums_index.json` - **Placeholder files:** - `downloads/.gitkeep` - `analysis/.gitkeep` ### Files that should NOT be tracked: +❌ `library/**` (scraped forum/article data — public repo keeps it off-git) ❌ `downloads/S*/**` (episode directories with audio) ❌ `*.mp3`, `*.m4a`, `*.wav` (audio files) ❌ `*.sqlite`, `*.db` (search index - can be rebuilt) @@ -124,7 +119,7 @@ git add . git diff --cached --name-only | grep -E "(\.mp3|\.sqlite|\.env|downloads/S)" # If the above command returns nothing, proceed -# (Note: library/forums IS tracked - that's expected) +# (Note: library/ scraped data is gitignored — NOT committed) git commit -m "Initial commit: Team Deakins knowledge base toolkit - Forum scraper with structured JSON storage @@ -132,7 +127,7 @@ git commit -m "Initial commit: Team Deakins knowledge base toolkit - Schema examples and documentation - Comprehensive .gitignore for copyright/privacy -Forum data included for private research use. +Public repo, code only: scraped forum data is gitignored (on-disk). Audio files and SQLite indices excluded." ``` @@ -148,8 +143,8 @@ du -sh .git git ls-files | wc -l git ls-files | head -20 -# Verify forum data IS tracked (should find files) -git ls-files | grep "library/forums" && echo "✓ Forum data is tracked" || echo "Warning: No forum data found" +# Verify scraped forum data is NOT tracked (gitignored — should find nothing) +git ls-files | grep "library/" && echo "ERROR: scraped data tracked!" || echo "✓ Forum data correctly gitignored" # Verify excluded files are NOT tracked git ls-files | grep -E "(downloads/S|\.mp3|\.sqlite|analysis/.*\.csv)" && echo "ERROR: Excluded files tracked!" || echo "✓ Excluded files not tracked" @@ -235,7 +230,7 @@ git push origin --force --all ✅ Documentation tracked ✅ Schema examples tracked ✅ Configuration tracked -✅ **Forum scraped data tracked** (for private repo) +❌ **Forum scraped data excluded** (`library/` gitignored — public repo, off-git) ❌ Audio files excluded (.mp3, .m4a, etc.) ❌ Episode metadata excluded (downloads/) ❌ SQLite indices excluded (can rebuild) @@ -243,7 +238,7 @@ git push origin --force --all ❌ Sensitive data excluded This ensures: -- **Privacy**: Keep repository **PRIVATE** (contains scraped forum data) +- **Privacy**: scraped forum data stays off-git via `.gitignore` (repo is public, code only) - **Security**: No credentials or API keys - **Copyright**: For personal/educational use only - **Size**: Audio files excluded to manage repository size diff --git a/documentation/data-model-reference.md b/documentation/data-model-reference.md new file mode 100644 index 0000000..00364cd --- /dev/null +++ b/documentation/data-model-reference.md @@ -0,0 +1,137 @@ +# Data Model Reference + +Deep schema reference for the forum/podcast knowledge base (ADR-0081 Layer 2 — task +reference, not always-loaded). **Authoritative source:** `deakins_forums/models.py` +(Pydantic). Git-committed illustrative examples: `documentation/examples-schemas/` +(see its `README.md` for field-level docs). This file mirrors those shapes for quick +reference when working on the data layer. + +## PostLeaf Structure (v2 Schema) + +**NEW in v2:** Threading support with parent-child relationships and post types. + +```json +{ + "ids": { + "post_id": "175763", + "forum_slug": "team-deakins", + "topic_slug": "lighting-discussion", + "reply_permalink": "https://...", + "parent_post_id": "175760", + "parent_type": "topic", + "position": 1 + }, + "post_type": "reply", + "author": { + "display_name": "Roger A. Deakins", + "role": "Keymaster" + }, + "timestamps": { + "raw": "January 18, 2023 at 3:32 pm #175763", + "parsed_iso": "2023-01-18T15:32:00", + "parse_confidence": "medium" + }, + "content_text": "...", + "content_html": "...", + "blocks": [...], + "quotes": [...], + "links": [...], + "media": [...], + "provenance": { + "source_url": "...", + "scraped_at": "2026-01-15T...", + "http": {"etag": "...", "last_modified": "..."} + }, + "integrity": { + "content_hash": "sha256:...", + "parser_version": "bbpress-v2" + } +} +``` + +**Threading Fields:** +- `post_type`: "topic" (starter) or "reply" +- `parent_post_id`: ID of parent post (null for topics) +- `parent_type`: "topic" or "reply" (null for topics) +- `position`: Sequential position in reply list (1-indexed) + +## TopicLeaf Structure (v2 Schema) + +**NEW in v2:** Hierarchical reply tree with thread statistics. + +```json +{ + "topic_url": "https://...", + "topic_slug": "lighting-discussion", + "title": "Lighting Discussion", + "post_ids": ["175760", "175763", "175764"], + "reply_tree": { + "post_id": "175760", + "post_type": "topic", + "author": "John Doe", + "timestamp": "2023-01-18T15:00:00", + "children": [ + { + "post_id": "175763", + "post_type": "reply", + "author": "Roger A. Deakins", + "timestamp": "2023-01-18T15:32:00", + "children": [] + }, + { + "post_id": "175764", + "post_type": "reply", + "author": "Jane Smith", + "timestamp": "2023-01-18T16:00:00", + "children": [] + } + ] + }, + "reply_count": 2, + "max_depth": 1, + "provenance": {...}, + "integrity": { + "content_hash": "sha256:...", + "parser_version": "bbpress-v2" + } +} +``` + +**Reply Tree Fields:** +- `reply_tree`: Recursive nested structure showing conversation flow +- `reply_count`: Total number of replies (excluding topic starter) +- `max_depth`: Maximum nesting depth (0 = no replies, 1 = direct replies only) +- `post_ids`: Flat list for backward compatibility + +**Migration Note:** Use `python -m deakins_forums.migrate --rebuild-trees` to rebuild reply trees for existing data. + +## Episode metadata.json Structure + +```json +{ + "title": "SEASON 2 - EPISODE 176 - Chris Lowe...", + "guid": "...", + "pubDate_iso": "2026-01-14", + "itunes": { + "season": 2, + "episode": 176, + "duration_s": 5400, + "summary": "..." + }, + "rss_description": "...", + "rss_content_encoded": "...", + "audio": { + "url": "https://...", + "mime": "audio/mpeg", + "size_bytes": 123456789, + "bitrate_kbps": 128, + "duration_s": 5400, + "format": "MP3" + }, + "provenance": { + "rss_url": "...", + "scraped_at": "2026-01-15T...", + "script_version": "ingest-v2" + } +} +``` diff --git a/documentation/development/PROJECT_STATUS.md b/documentation/development/PROJECT_STATUS.md index 8702979..9b02574 100644 --- a/documentation/development/PROJECT_STATUS.md +++ b/documentation/development/PROJECT_STATUS.md @@ -63,12 +63,12 @@ purefoy/ └── README.md ``` -### Will be included in git ✅ (Forum Data) +### Excluded from git ❌ (Forum Data — gitignored, on-disk only) -**Note: This is for a PRIVATE repository** +The repo is public, so scraped forum data is never committed — it lives on-disk only: ``` -├── library/forums/ # Scraped forum data +├── library/forums/ # Scraped forum data (gitignored) │ ├── posts/*.json # Individual post files │ ├── topics/*.json # Topic metadata files │ ├── forums/*.json # Forum metadata files @@ -165,20 +165,19 @@ git ls-files | head -20 ## Privacy & Copyright Protection -**⚠️ IMPORTANT: Keep this repository PRIVATE** +**This is a public repository — scraped data is kept off-git via `.gitignore`.** +✅ **Forum data excluded** - `library/` gitignored; scraped JSON stays on-disk only ✅ **Audio files excluded** - No .mp3, .m4a, etc. in git -✅ **Forum data INCLUDED** - JSON forum data tracked for private research ✅ **Episode metadata excluded** - No episode directories tracked ✅ **SQLite indices excluded** - Can be rebuilt from JSON ✅ **Analysis exports excluded** - Can be regenerated ✅ **Sensitive data excluded** - No .env, credentials, or API keys in git -**Why include forum data?** -- Private repository for personal/educational research -- Forum content is publicly accessible (not behind authentication) -- Structured JSON is git-friendly -- Enables incremental updates and version control +**Why exclude forum data?** +- Public repository — scraped content should not be redistributed +- Keeps the codebase open while respecting copyright +- Data is regenerable on-disk via the scraper; no need to commit it ## If You Need Help @@ -192,15 +191,14 @@ git ls-files | head -20 🎉 **Project is ready for git initialization!** Your .gitignore is properly configured to: -- **Include forum data** for private research use +- **Exclude forum data** (`library/`) — kept off-git on a public repo - **Exclude audio files** to manage repository size - **Exclude SQLite indices** (can be rebuilt) - **Exclude episode metadata** (contains audio files) - **Secure credentials** (no sensitive data) -**CRITICAL: Keep repository PRIVATE** -- Forum data is included for research purposes -- Personal/educational use only -- Do not make this repository public +**Data handling (public repo):** +- Scraped forum data is gitignored — never committed +- Personal/educational use only; do not redistribute scraped content Proceed with `git init` when ready. diff --git a/documentation/repo-reference.md b/documentation/repo-reference.md new file mode 100644 index 0000000..50095c8 --- /dev/null +++ b/documentation/repo-reference.md @@ -0,0 +1,85 @@ +# Repo Reference + +Deep reference for the purefoy knowledge base (ADR-0081 Layer 2 — task reference, not +always-loaded). Repo-wide policy and the command quick-start live in the root `CLAUDE.md`; +the forum-scraper internals live in `deakins_forums/CLAUDE.md`; record schemas in +`documentation/data-model-reference.md`. + +## Project Structure + +``` +purefoy/ +├── deakins_forums/ # Forum scraper package (structured, MCP-ready) +│ ├── cli.py # Command-line interface +│ ├── pipeline.py # High-level scraping orchestration +│ ├── models.py # Pydantic data models (PostLeaf, TopicLeaf, etc.) +│ ├── parser_bbpress.py # HTML parsing logic for bbPress forums +│ ├── http_client.py # Rate-limited HTTP client with ETag support +│ ├── store_json.py # JSON "leaf" persistence layer +│ ├── index_sqlite.py # SQLite FTS5 search index +│ ├── normalize.py # Content extraction (links, quotes, blocks) +│ ├── export.py # Export utilities (CSV, text, by-author, etc.) +│ ├── coverage.py # Coverage tracking for incremental scrapes +│ ├── query_tracker.py # Query provenance/lineage tracking +│ ├── report.py # Reporting utilities +│ └── config.py # Settings with env var overrides +├── packages/ # TypeScript UI layer (pnpm workspace) +│ ├── browser-app/ # React/Vite micro-frontend remote (port 3020) — UI via @ojfbot/frame-ui-components (npm ^1.0.1) +│ ├── browser-automation/ # Visual regression testing via browser automation (screenshot capture, CI integration) +│ ├── api/ # Express API over flat JSON + SQLite (port 3021) — includes GET /api/beads (ADR-0016) +│ └── shared/ # @purefoy/shared — generated OpenAPI schema + API type contracts (incl. GoalManifest, ReviewProgress) +├── app.py # Standalone Flask UI — dark-theme knowledge browser (port 5050) +├── download_episodes.py # Podcast RSS feed downloader (standalone) +├── ingest_teamdeakins_downloads.py # Episode ingest with metadata & transcripts +├── scrape_forum.py # Legacy forum scraper (standalone) +├── library/ # Output directory for scraped data +│ └── forums/ # Forum JSON leafs organized by type +│ ├── posts/ # Individual post JSON files +│ ├── topics/ # Topic index files +│ ├── forums/ # Forum metadata files +│ └── _site/ # Sqlite index, HTTP state, coverage +├── downloads/ # Podcast episode directories (348/348 episodes transcribed) +│ └── S02E176__2026-01-14__guest-name__libsyn_abc123/ +│ ├── audio.mp3 +│ ├── metadata.json +│ └── transcript/ +├── analysis/ # Export output (CSV, text dumps, stats) +├── package.json # pnpm workspace root +└── pyproject.toml # Python package metadata +``` + +## Podcast Episode Management + +```bash +# Download MP3s from RSS feed +.venv/bin/python3 download_episodes.py + +# Organize downloads into episode directories (dry-run) +.venv/bin/python3 ingest_teamdeakins_downloads.py \ + --rss "https://rss.libsyn.com/shows/265448/destinations/2018942.xml" \ + --downloads "./downloads" \ + --dry-run + +# Actually ingest with transcript fetching +.venv/bin/python3 ingest_teamdeakins_downloads.py \ + --rss "https://rss.libsyn.com/shows/265448/destinations/2018942.xml" \ + --downloads "./downloads" \ + --fill-transcripts \ + --tapesearch +``` + +### Episode Ingest Design + +The `ingest_teamdeakins_downloads.py` script: +- Parses RSS XML to extract episode metadata (title, date, duration, iTunes tags, description) +- Matches downloaded MP3s to RSS entries via URL +- Creates structured episode directories with naming: `S{season}E{episode}__{date}__{guest}__{source_id}/` +- Generates `metadata.json` with comprehensive provenance +- Creates `transcript/` scaffold with `sources.json` and `transcript.txt` (placeholder or fetched) +- Supports Podcasting 2.0 `` URLs and optional Tapesearch fallback + +## Testing Notes + +- **TypeScript layer**: CI runs TypeScript type-check, test, and codegen drift guard jobs (see `packages/`). Fleet-wide visual regression tests run via the `browser-automation` package (screenshots capture the composed shell, not standalone apps; waits for a Frame sentinel rather than `networkidle`). +- **Python layer**: Manual testing via CLI commands and dry-run flags; no formal Python test suite yet +- Incremental updates can be verified by running scrapes twice and checking for "Not modified" messages diff --git a/documentation/roadmap/ROADMAP.md b/documentation/roadmap/ROADMAP.md index f6d8c92..3bb00fa 100644 --- a/documentation/roadmap/ROADMAP.md +++ b/documentation/roadmap/ROADMAP.md @@ -370,15 +370,15 @@ python -m deakins_forums.cli tree team-deakins__bikes --format html -o bikes_tre ### Open Source Considerations -#### 4.6 Public vs. Private Decision +#### 4.6 Public vs. Private Decision — ✅ RESOLVED **Priority:** Critical (before git init) -**Options:** +**Options considered:** 1. **Private Repository**: Keep scraped data private, respect forum ToS -2. **Public Code, Private Data**: Open-source scrapers, private data repo +2. **Public Code, Private Data**: Open-source scrapers, scraped data off-git 3. **Fully Public**: Open everything (requires legal review) -**Current:** Private repository assumed +**Decision:** Option 2 — **public repository, code only**. Scrapers and docs are public; scraped forum content (`library/`), audio (`downloads/`), and exports (`analysis/`) are gitignored and never committed. Respects forum ToS/copyright while keeping the codebase open. ---