feat(aggregator): scraper daemon + dual-mode write-policy (#1046)

[c03rad0r]

What

Adds an active scraper daemon to the aggregator relay from #1066, so it continuously mirrors marketplace events from upstream relays into one fast local relay — instead of only storing events explicitly published to it. The market app then reads from a single, pre-populated relay. Also adds a write-policy that accepts all marketplace event kinds.

Stacked on #1066 (the aggregator relay), which is still OPEN. Once #1066 merges, GitHub collapses this diff to just the scraper changes.

Why

The aggregator relay deployed in #1066 only holds events explicitly published to it. Without active scraping it would miss most marketplace activity — listings, profiles, ratings, etc. live spread across many public relays. The scraper pre-fetches and caches them so the app's reads resolve against one healthy local relay instead of fanning out across potentially-slow public relays.

Refs #1046. Depends on #1066.

Changes

File	Change
scraper.py (new)	Long-lived daemon. Bootstraps from the configured seed npub's kind 3 + 10002, discovers each contact's relay list, opens a worker per relay holding chunked-author + #p subscriptions, and republishes every matching event into strfry. Tracks discovered pubkeys (capped via MAX_PUBKEYS) and prunes stale entries hourly.
write-policy.py	Accepts all public marketplace event kinds (0/1/3/7/9735/1985/10000/10002, 1023–1026, 30402/30405/06/08, 30440–30442, 31555/31990) so scraped market data is stored; rejects non-market kinds. Hot-reloads on config change.
tests/ (new)	pytest suite — 108 tests. write-policy: market-kind acceptance + the full stdin/stdout strfry plugin protocol (case-insensitive seed, garbage rejection) exercised via subprocess. scraper: pubkey tracking/harvest with MAX_PUBKEYS cap, bounded-LRU event dedup, kind-3 follow + kind-10002 relay-list parsing, pruning, discovery, and the relay_worker mirror path (subscription construction + deduped republish) with a stubbed websocket.
Dockerfile	Add py3-websocket-client (the websocket-client lib scraper.py imports).
docker-compose.yml	Add scraper service (depends on strfry-market-agg, read-write state volume, internal ws://strfry-market-agg:7777 URL).
README.md	Document scraper architecture, data flow, and configuration.
strfry.conf / .env.example	Scraper + relay configuration.

Architecture

[Market App] --reads--> [strfry :7777] <--write-policy
                              ^
                              | republish
                              |
                     [scraper daemon] --scrape--> damus / nos.lol / plebeian + discovered relays

Test plan

• pytest deploy-simple/aggregator/ -q → 108 passed — write-policy market-kind acceptance (via a real subprocess exercising the stdin/stdout plugin protocol) and scraper logic + relay_worker mirror path with a stubbed websocket module (no network).
• Both .py files py_compile clean; docker-compose.yml validated.
• Live (post-merge): scraper connects to seed relays, fetches the seed npub's network (kind 3 + 10002), republishes marketplace events to strfry, and stays current on the maintain timer.

[c03rad0r]

ℹ️ Status note: This PR is stacked on #1066 (feat/market-agg-relay). CI does not run for stacked branches. Once #1066 merges, this will be retargeted to master and CI will trigger automatically. Reviewers can review the diff now — the scraper daemon and write-policy logic are self-contained.

[c03rad0r]

📋 Stacking chain reference: Chain B (Relay/Aggregator): #1066 ← this PR

Full review-order map with all stacking chains: #1088

(Automated tag — see issue for review priority recommendations.)