docs(claude-md): loading-discipline routing + drop stale private-repo framing

GIT_READY.md

@@ -104,7 +104,7 @@ purefoy/
 - `PROJECT_STRUCTURE.md` - Structure guide
 - `GIT_READY.md` - This file
 
-**Data (Private Repo):**
+**Data (on-disk only — gitignored, not committed):**
 - `library/forums/` - 3,075 post JSONs, 691 topic JSONs (24MB)
 - `library/articles/` - Sample article JSONs
 
@@ -181,20 +181,18 @@ purefoy/
 ## Git Repository Configuration
 
 ### Repository Type
-**Recommended:** Private repository
+**Adopted:** Public repository, code only — scraped data kept off-git.
+
+**Configuration:**
+1. `library/`, `downloads/`, and `analysis/` are in `.gitignore`
+2. Only source code, documentation, and illustrative schema examples are committed
+3. All ~24MB of scraped data stays on-disk (never committed)
+4. Content copyright disclaimer in README / CLAUDE.md
 
 **Reasoning:**
 - Forum content is publicly accessible but should not be redistributed
 - Personal research and educational use only
-- Includes 24MB of scraped JSON data (library/)
-- Respects copyright while enabling version control
-
-### Alternative: Public Code Only
-If making code public:
-1. Add `library/` to `.gitignore`
-2. Only commit source code and documentation
-3. Exclude all scraped data
-4. Add prominent disclaimer about content copyright
+- Respects copyright while keeping the codebase open
 
 ---
 
@@ -282,7 +280,7 @@ git push -u origin main
 ## Next Steps After Git Init
 
 1. **Create Remote Repository**
-   - GitHub, GitLab, or Gitea (private)
+   - GitHub, GitLab, or Gitea
    - Add remote: `git remote add origin <url>`
 
 2. **Push Initial Commit**

PROJECT_STRUCTURE.md

@@ -78,7 +78,7 @@ __pycache__/               # Python bytecode
 - **`deakins_articles/`**: Articles scraper for rogerdeakins.com/articles
 
 ### Data Storage
-- **`library/`**: Primary data storage with JSON "leafs" (TRACKED in git for private repo)
+- **`library/`**: Primary data storage with JSON "leafs" (gitignored — on-disk only; public repo keeps scraped data off-git)
   - Structured, version-controllable JSON files
   - Enables incremental updates via content hashing
   - Ready for AI/MCP consumption

README.md

@@ -126,7 +126,7 @@ Scrapes technical articles from rogerdeakins.com/articles:
 ### Installation
 
 ```bash
-# Clone repository (private repo assumed)
+# Clone repository
 git clone <repo-url>
 cd purefoy
 
@@ -312,9 +312,13 @@ http_state.json            # HTTP cache (ephemeral)
 
 **What IS tracked:**
 - ✅ All source code (`deakins_forums/`, `deakins_articles/`)
-- ✅ Library data (`library/`) - Structured JSON for private repo
 - ✅ Documentation and guides
 - ✅ Configuration files
+- ✅ Illustrative schema examples (`documentation/examples-schemas/`)
+
+**What is NOT tracked (gitignored, on-disk only):**
+- ❌ Scraped forum content (`library/`) — public repo, so data stays off-git
+- ❌ Podcast audio + episode dirs (`downloads/`), exports (`analysis/`)
 
 ---
 
@@ -364,7 +368,7 @@ See [CLAUDE.md](CLAUDE.md) for detailed architecture documentation.
 
 **Permitted Use:**
 - ✅ Personal research and education
-- ✅ Private knowledge base for learning
+- ✅ Personal knowledge base for learning
 - ✅ Academic study of cinematography
 
 **Prohibited Use:**
@@ -377,21 +381,14 @@ See [CLAUDE.md](CLAUDE.md) for detailed architecture documentation.
 
 ---
 
-### Git Repository Privacy
-
-**⚠️ This MUST be a PRIVATE repository**
-
-The project is configured for a private git repository:
+### Git Data Policy
 
-- `library/` directory contains JSON data (forum content) tracked in the private repo
-- Forum content is publicly accessible but should not be publicly redistributed
-- Use for personal/educational research only
-- Do not push to public GitHub/GitLab
+This is a **public repository** — scraped data is kept off-git, not behind repo visibility.
 
-**If making code public:**
-- Exclude `library/` directory entirely
-- Include only source code and documentation
-- Add clear disclaimers about content copyright
+- `library/` (forum content), `downloads/` (podcast audio), and `analysis/` (exports) are **gitignored** and never committed.
+- Only source code, documentation, and illustrative schema examples (`documentation/examples-schemas/`) are public.
+- Forum content is publicly accessible but should not be redistributed; use for personal/educational research only.
+- Content remains copyright of the original holders — see Copyright & Usage above.
 
 ---
 

deakins_forums/CLAUDE.md

@@ -0,0 +1,83 @@
+# deakins_forums — scraper subtree guidance
+
+Path-conditional guidance (ADR-0081 Layer 1): this file loads when you edit the
+`deakins_forums/` forum-scraper package. Repo-wide policy lives in the root `CLAUDE.md`;
+the leaf/record schemas live in `documentation/data-model-reference.md` (authoritative
+source: `deakins_forums/models.py`).
+
+## Forum Scraper Design
+
+The `deakins_forums` package follows a **composable pipeline architecture**:
+
+1. **HttpClient** (`http_client.py`): Rate-limited HTTP layer with conditional GET (ETag/Last-Modified) support for efficient incremental updates.
+
+2. **Parser** (`parser_bbpress.py`): BeautifulSoup-based HTML extraction for bbPress forum structure. Extracts forums index, forum pages, topic pages, and posts.
+
+3. **Models** (`models.py`): Pydantic models defining the "leaf" JSON structure (v2 schema):
+   - `PostLeaf`: Individual post with content, author, timestamps, structured blocks, quotes, links, media, **plus threading fields** (post_type, parent_post_id, parent_type, position)
+   - `TopicLeaf`: Topic metadata + post ID index (posts are separate files) **plus reply tree** (recursive nested structure showing conversation hierarchy with thread statistics)
+   - `ForumLeaf`: Forum metadata + topic reference list
+   - `PostType` enum: Distinguishes "topic" (starter) from "reply" posts
+   - All include `Provenance` (source URL, scrape timestamp, HTTP headers) and `Integrity` (content hash for change detection, parser_version tracks schema)
+
+4. **Pipeline** (`pipeline.py`): High-level orchestration gluing fetch → parse → normalize → store. Handles pagination, deduplication, incremental refresh.
+
+5. **Storage** (`store_json.py`): Persists structured JSON "leafs" with deterministic file paths (e.g., `posts/{post_id}.json`). Enables version control and incremental updates.
+
+6. **Index** (`index_sqlite.py`): SQLite FTS5 full-text search index for fast queries across posts. Rebuilt from JSON leafs on demand.
+
+7. **Normalize** (`normalize.py`): Extracts structured content elements (links, media, quotes, content blocks) from raw HTML/text for agent-friendly consumption.
+
+8. **Query Tracking** (`query_tracker.py`): Records provenance of each scrape query (who, why, when, what was accessed) for research auditability.
+
+### Key Design Principles
+
+- **Incremental & Resumable**: HTTP ETag caching, content hashing, and visited URL tracking allow efficient re-scraping
+- **Deterministic Paths**: Slug-based file naming enables easy navigation and version control
+- **Agent-Friendly**: Ultra-structured JSON with normalized content blocks, quotes, links for downstream AI/MCP tools
+- **Provenance-First**: Every leaf includes source URL, scrape timestamp, HTTP headers, content hash
+- **Rate-Limited & Respectful**: Configurable delays (default 3s), User-Agent identification, robots.txt compliance
+
+## Environment Variables
+
+Configure via environment variables (all optional):
+
+```bash
+export DEAKINS_BASE_URL="https://rogerdeakins.com"
+export DEAKINS_USER_AGENT="Your Custom User Agent"
+export DEAKINS_DELAY_S="3.0"          # Seconds between requests
+export DEAKINS_TIMEOUT_S="30"         # Request timeout
+export DEAKINS_MAX_RETRIES="3"
+export DEAKINS_OUT_DIR="./library/forums"
+```
+
+## Forum Scraping commands
+
+```bash
+# Using the CLI entry point (if installed)
+deakins-forums scrape-all --max-topics 1000 --build-index
+deakins-forums scrape-forum team-deakins --max-pages 20 --build-index
+deakins-forums scrape-topic https://rogerdeakins.com/forums/topic/...
+
+# Or using python -m
+python -m deakins_forums.cli scrape-forum team-deakins --max-pages 20
+python -m deakins_forums.cli build-index
+python -m deakins_forums.cli search "lighting techniques" --limit 20
+python -m deakins_forums.cli stats
+
+# Export data
+python -m deakins_forums.cli export all-text -o analysis/all_posts.txt
+python -m deakins_forums.cli export csv -o analysis/posts.csv
+python -m deakins_forums.cli export roger-only -o analysis/roger_posts.txt
+
+# Query tracking (provenance)
+python -m deakins_forums.cli scrape-forum team-deakins \
+  --query-name "Research natural lighting techniques" \
+  --querier-role "Cinematographer" \
+  --querier-department "Camera" \
+  --query-context "TV miniseries prep" \
+  --query-intent "Find Team Deakins insights on natural light"
+
+python -m deakins_forums.cli provenance --topic "team-deakins__topic-slug"
+python -m deakins_forums.cli coverage --format full
+```

documentation/GIT_INIT_CHECKLIST.md

@@ -94,18 +94,13 @@ git ls-files --others --exclude-standard
 **Schema Examples:**
 - `schema_examples/**/*`
 
-**Forum Scraped Data (for private repo):**
-- `library/forums/posts/*.json`
-- `library/forums/topics/*.json`
-- `library/forums/forums/*.json`
-- `library/forums/_site/forums_index.json`
-
 **Placeholder files:**
 - `downloads/.gitkeep`
 - `analysis/.gitkeep`
 
 ### Files that should NOT be tracked:
 
+❌ `library/**` (scraped forum/article data — public repo keeps it off-git)
 ❌ `downloads/S*/**` (episode directories with audio)
 ❌ `*.mp3`, `*.m4a`, `*.wav` (audio files)
 ❌ `*.sqlite`, `*.db` (search index - can be rebuilt)
@@ -124,15 +119,15 @@ git add .
 git diff --cached --name-only | grep -E "(\.mp3|\.sqlite|\.env|downloads/S)"
 
 # If the above command returns nothing, proceed
-# (Note: library/forums IS tracked - that's expected)
+# (Note: library/ scraped data is gitignored — NOT committed)
 git commit -m "Initial commit: Team Deakins knowledge base toolkit
 
 - Forum scraper with structured JSON storage
 - Podcast episode downloader and ingest scripts
 - Schema examples and documentation
 - Comprehensive .gitignore for copyright/privacy
 
-Forum data included for private research use.
+Public repo, code only: scraped forum data is gitignored (on-disk).
 Audio files and SQLite indices excluded."
 ```
 
@@ -148,8 +143,8 @@ du -sh .git
 git ls-files | wc -l
 git ls-files | head -20
 
-# Verify forum data IS tracked (should find files)
-git ls-files | grep "library/forums" && echo "✓ Forum data is tracked" || echo "Warning: No forum data found"
+# Verify scraped forum data is NOT tracked (gitignored — should find nothing)
+git ls-files | grep "library/" && echo "ERROR: scraped data tracked!" || echo "✓ Forum data correctly gitignored"
 
 # Verify excluded files are NOT tracked
 git ls-files | grep -E "(downloads/S|\.mp3|\.sqlite|analysis/.*\.csv)" && echo "ERROR: Excluded files tracked!" || echo "✓ Excluded files not tracked"
@@ -235,15 +230,15 @@ git push origin --force --all
 ✅ Documentation tracked
 ✅ Schema examples tracked
 ✅ Configuration tracked
-✅ **Forum scraped data tracked** (for private repo)
+❌ **Forum scraped data excluded** (`library/` gitignored — public repo, off-git)
 ❌ Audio files excluded (.mp3, .m4a, etc.)
 ❌ Episode metadata excluded (downloads/)
 ❌ SQLite indices excluded (can rebuild)
 ❌ Analysis exports excluded (can regenerate)
 ❌ Sensitive data excluded
 
 This ensures:
-- **Privacy**: Keep repository **PRIVATE** (contains scraped forum data)
+- **Privacy**: scraped forum data stays off-git via `.gitignore` (repo is public, code only)
 - **Security**: No credentials or API keys
 - **Copyright**: For personal/educational use only
 - **Size**: Audio files excluded to manage repository size