diff --git a/CHANGELOG.md b/CHANGELOG.md index c745704..aa6ea99 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,24 @@ All notable changes to this project are documented here. Format loosely follows [Keep a Changelog](https://keepachangelog.com/); versions follow [SemVer](https://semver.org/). +## [0.6.0] — 2026-07-07 + +### Added +- `-q/--quiet` flag for `podpull get`: suppresses the spinner/progress UI and prints + only saved file paths to stdout — for scripting/piping. (#1, thanks @adjenk!) +- Robust feed parsing: RSS 2.0 / RSS 1.0 (RDF) / Atom, any-namespace matching, + `media:content` + Atom-enclosure fallbacks, dirty-XML sanitize-and-retry + (undefined entities, bare `&`, control chars), ISO-8601 dates. +- Verified against Chinese-market hosts: xiaoyuzhou, Ximalaya, SoundOn, Firstory, + WavPub, Typlog, Fireside, Lizhi (offline fixtures + `pytest -m network` live suite). +- `ximalaya.com/album/` links are now accepted directly. +- Optional Podcast Index support (BYOK: `PODCASTINDEX_API_KEY`/`_SECRET`): + enriches `search` and adds a feed-resolution fallback when iTunes has no feed. + +### Fixed +- Download guard: `text/*` responses (stale enclosures, e.g. Ximalaya CDN) now + error instead of writing a garbage audio file. + ## [0.5.1] — 2026-06-30 ### Fixed diff --git a/README.md b/README.md index 5b62574..c5a139f 100644 --- a/README.md +++ b/README.md @@ -144,7 +144,21 @@ Drive, OneDrive, Dropbox, iCloud, etc. (CJK and ordinary text are kept). When yo **multiple** episodes at once, they're placed in a sub-folder named after the show. `` accepts: an Apple show URL, a bare Apple ID, a raw RSS feed URL, an Apple -episode URL (`?i=`), or a xiaoyuzhou episode URL. +episode URL (`?i=`), a xiaoyuzhou episode URL, or a Ximalaya album URL +(`ximalaya.com/album/`). + +### Podcast Index (optional) + +podpull can enrich `search` and feed resolution with the open +[Podcast Index](https://podcastindex.org) directory. Get a free API key at +[api.podcastindex.org](https://api.podcastindex.org/signup) and set: + +```bash +export PODCASTINDEX_API_KEY=... +export PODCASTINDEX_API_SECRET=... +``` + +Without these, podpull behaves exactly as before (iTunes only). ## Roadmap @@ -153,8 +167,11 @@ episode URL (`?i=`), or a xiaoyuzhou episode URL. colored help, scriptable stdout. Adds `rich` + `questionary`. - **v0.3**: renamed `podget` → `podpull`. - **v0.4**: cloud-safe filename normalization; multi-episode downloads grouped into a per-show folder. -- **v0.5** (current): `pull` alias for `get`; `podpull skills install` sets up integrations for Claude Code, Codex, OpenCode, and Cursor. -- **next**: more robust feed parsing, tests on more hosts, Podcast Index support. +- **v0.5**: `pull` alias for `get`; `podpull skills install` sets up integrations for Claude Code, Codex, OpenCode, and Cursor. +- **v0.6** (current): robust feed parsing (RSS 2.0 / RSS 1.0 / Atom, dirty-XML recovery), + verified against Chinese-market hosts, `ximalaya.com/album/` links, optional + Podcast Index (BYOK) search + feed-resolution fallback. +- **next**: `--json` output mode for scripting. - **v1+ (`podpull[ai]`)**: opt-in **BYOK summarization** — local transcription (faster-whisper) + your own LLM key (Anthropic/OpenAI). Fully local, private, no subscription. Cleanly isolated from the core. diff --git a/docs/superpowers/plans/2026-07-07-feed-robustness-podcastindex.md b/docs/superpowers/plans/2026-07-07-feed-robustness-podcastindex.md new file mode 100644 index 0000000..2739ab6 --- /dev/null +++ b/docs/superpowers/plans/2026-07-07-feed-robustness-podcastindex.md @@ -0,0 +1,1514 @@ +# Feed Robustness + Podcast Index Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make `core.parse_feed` survive real-world feeds (CN + global hosts, Atom/RDF, dirty XML), add optional BYOK Podcast Index search/resolution, and back it all with offline fixtures — shipping as v0.6.0. + +**Architecture:** All logic changes live in `src/podpull/core.py` (pure stdlib — CLAUDE.md invariant #1); the only `cli.py` change is `cmd_search` merging Podcast Index results. Tests are offline fixtures under `tests/fixtures/feeds/` driven by one parametrized case table; live-network checks go in a `network`-marked module excluded by default. + +**Tech Stack:** Python ≥3.9 stdlib (`xml.etree`, `html.entities`, `hashlib`, `urllib`); pytest; no new runtime dependencies. + +**Spec:** `docs/superpowers/specs/2026-07-07-feed-robustness-podcastindex-design.md` (approved 2026-07-07) + +**Working branch:** `feed-robustness-podcastindex` (already exists; all commits go here) + +**Conventions the engineer must know (from CLAUDE.md):** +- `core.py` may import ONLY the standard library. UI/rich stays in `cli.py`. +- stdout is machine output; all human messages go to stderr (`cli.ui` / `_err`). +- Default test suite must never touch the network. +- Run everything from the repo root with the venv: `. .venv/bin/activate` (or use `python3 -m pytest`). Full suite: `pytest -q`; lint: `ruff check src`. +- Version lives in BOTH `src/podpull/__init__.py` and `pyproject.toml`. + +--- + +### Task 1: pytest `network` marker configuration + +**Files:** +- Modify: `pyproject.toml` (add `[tool.pytest.ini_options]` at end of file) + +- [ ] **Step 1: Add pytest config** + +Append to `pyproject.toml`: + +```toml +[tool.pytest.ini_options] +markers = [ + "network: hits live podcast hosts; excluded by default (run: pytest -m network)", +] +addopts = "-m 'not network'" +``` + +- [ ] **Step 2: Verify the suite still passes and the marker is honored** + +Run: `pytest -q` +Expected: all existing tests pass (24 passed at time of writing; count may differ after PR #3 merges), `no tests ran` is a FAILURE. + +Run: `pytest -q -m network` +Expected: `no tests ran` / all deselected (no network tests exist yet — that's fine). + +- [ ] **Step 3: Commit** + +```bash +git add pyproject.toml +git commit -m "test: add network marker, excluded by default" +``` + +--- + +### Task 2: `Episode.date` ISO-8601 fallback + +**Files:** +- Modify: `src/podpull/core.py` (imports + `Episode.date`, currently lines 15–43) +- Test: `tests/test_core.py` (extend `test_episode_date`) + +- [ ] **Step 1: Write the failing test** + +In `tests/test_core.py`, replace `test_episode_date` with: + +```python +def test_episode_date(): + assert _eps()[0].date == "2026-06-27" + assert Episode(title="x", pub="garbage", url="u").date == "0000-00-00" + # RFC-822 with non-zero-padded day (Lizhi) — parsedate handles it + assert Episode(title="x", pub="Sun, 5 Jul 2026 21:00:00 +0800", url="u").date == "2026-07-05" + # ISO-8601 (Atom published / dc:date), with and without Z + assert Episode(title="x", pub="2026-07-01T08:30:00Z", url="u").date == "2026-07-01" + assert Episode(title="x", pub="2026-07-01T08:30:00+08:00", url="u").date == "2026-07-01" + assert Episode(title="x", pub="", url="u").date == "0000-00-00" +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `pytest tests/test_core.py::test_episode_date -v` +Expected: FAIL — the ISO strings return `"0000-00-00"`. + +- [ ] **Step 3: Implement** + +In `src/podpull/core.py`, add to imports (after `from email.utils import parsedate_to_datetime`): + +```python +from datetime import datetime +``` + +Replace the `date` property of `Episode`: + +```python + @property + def date(self) -> str: + for parse in ( + lambda s: parsedate_to_datetime(s), # RFC-822 + lambda s: datetime.fromisoformat(s.strip().replace("Z", "+00:00")), # ISO-8601 + ): + try: + return parse(self.pub).strftime("%Y-%m-%d") + except Exception: + continue + return "0000-00-00" +``` + +- [ ] **Step 4: Run tests** + +Run: `pytest tests/test_core.py -q` +Expected: PASS (all). + +- [ ] **Step 5: Commit** + +```bash +git add src/podpull/core.py tests/test_core.py +git commit -m "feat: parse ISO-8601 pub dates (Atom/dc:date) in Episode.date" +``` + +--- + +### Task 3: namespace-agnostic feed parsing (RSS 2.0 / RSS 1.0 RDF / Atom) + +**Files:** +- Modify: `src/podpull/core.py` — replace `parse_feed` (currently lines 114–139) with helpers + new implementation +- Create: `tests/fixtures/feeds/rss2_plain.xml`, `tests/fixtures/feeds/rss10_rdf.xml`, `tests/fixtures/feeds/atom.xml` +- Test: `tests/test_core.py` — add fixture harness + parametrized test + +- [ ] **Step 1: Create three fixture files** + +`tests/fixtures/feeds/rss2_plain.xml` (regression: today's happy path — itunes author, enclosure): + +```xml + + + + Plain RSS2 Show + Plain Author + + EP2 + Fri, 03 Jul 2026 08:00:00 GMT + plain-ep2 + https://example.test/ep2 + + + + EP1 + Wed, 01 Jul 2026 08:00:00 GMT + plain-ep1 + + + + +``` + +`tests/fixtures/feeds/rss10_rdf.xml` (RSS 1.0: default namespace on every element, items OUTSIDE channel — today's parser finds zero items because `.//item` misses namespaced tags): + +```xml + + + + RDF Show + RDF Author + + + RDF EP1 + 2026-06-30T10:00:00Z + + + +``` + +`tests/fixtures/feeds/atom.xml` (Atom: `feed`/`entry`, `link rel="enclosure"`, `author/name`, ISO dates): + +```xml + + + Atom Show + Atom Author + + Atom EP1 + atom-ep1 + 2026-07-01T08:30:00Z + + + + +``` + +- [ ] **Step 2: Add the fixture harness + failing parametrized test** + +In `tests/test_core.py`, add after the imports: + +```python +import io +from pathlib import Path + +FIXTURES = Path(__file__).parent / "fixtures" / "feeds" + +# (fixture, show_title, show_author, n_episodes, first_url, first_date) +FEED_CASES = [ + ("rss2_plain.xml", "Plain RSS2 Show", "Plain Author", 2, + "https://cdn.example.test/ep2.mp3", "2026-07-03"), + ("rss10_rdf.xml", "RDF Show", "RDF Author", 1, + "https://cdn.example.test/rdf1.mp3", "2026-06-30"), + ("atom.xml", "Atom Show", "Atom Author", 1, + "https://cdn.example.test/atom1.mp3", "2026-07-01"), +] + + +def _parse_fixture(monkeypatch, fname): + raw = (FIXTURES / fname).read_bytes() + monkeypatch.setattr(core, "fetch", lambda url, timeout=45: io.BytesIO(raw)) + return core.parse_feed("https://example.test/feed") + + +@pytest.mark.parametrize("fname,title,author,count,first_url,first_date", FEED_CASES) +def test_parse_feed_fixture(monkeypatch, fname, title, author, count, first_url, first_date): + t, a, eps = _parse_fixture(monkeypatch, fname) + assert t == title + assert a == author + assert len(eps) == count + assert eps[0].url == first_url + assert eps[0].date == first_date +``` + +- [ ] **Step 3: Run to verify failure** + +Run: `pytest tests/test_core.py::test_parse_feed_fixture -v` +Expected: `rss2_plain.xml` PASSES (regression guard); `rss10_rdf.xml` and `atom.xml` FAIL (0 episodes / empty title). + +- [ ] **Step 4: Implement** + +In `src/podpull/core.py`, replace the whole `parse_feed` function (keep its position in the file) with: + +```python +def _localname(tag) -> str: + """'{ns}Tag' -> 'tag'. Comments/PIs have non-str tags -> ''.""" + if not isinstance(tag, str): + return "" + return tag.rsplit("}", 1)[-1].lower() + + +def _clean(text) -> str: + return re.sub(r"\s+", " ", (text or "").strip()) + + +def _first_text(el, *names) -> str: + """First non-empty direct-child text whose localname is in names.""" + wanted = {n.lower() for n in names} + for child in el: + if _localname(child.tag) in wanted and _clean(child.text): + return _clean(child.text) + return "" + + +def _author(el) -> str: + """itunes:author / atom author>name -> managingEditor -> dc:creator.""" + for name in ("author", "managingeditor", "creator"): + for child in el: + if _localname(child.tag) == name: + txt = _first_text(child, "name") or _clean(child.text) + if txt: + return txt + return "" + + +def _pub(item) -> str: + """pubDate -> dc:date -> atom published -> atom updated (raw string).""" + for name in ("pubdate", "date", "published", "updated"): + for child in item: + if _localname(child.tag) == name and _clean(child.text): + return child.text.strip() + return "" + + +def _find_enclosure(item) -> tuple[str, str]: + """-> (audio_url, mime) or ('', ''). Priority: enclosure > media:content + (audio) > atom link rel=enclosure. First match wins — WavPub/Omny items + carry BOTH enclosure and media:content; one item must yield one URL.""" + for child in item: + if _localname(child.tag) == "enclosure" and child.get("url"): + return child.get("url"), child.get("type") or "" + for child in item: + if _localname(child.tag) == "content" and child.get("url"): + mime = child.get("type") or "" + if mime.startswith("audio/") or child.get("medium") == "audio": + return child.get("url"), mime + for child in item: + if (_localname(child.tag) == "link" and child.get("rel") == "enclosure" + and child.get("href")): + return child.get("href"), child.get("type") or "" + return "", "" + + +def _item_link(item) -> str: + link = _first_text(item, "link") + if link: + return link + for child in item: # atom: with no/alternate rel + if (_localname(child.tag) == "link" and child.get("href") + and child.get("rel") in (None, "alternate")): + return child.get("href") + return "" + + +def parse_feed(feed_url: str) -> tuple[str, str, list[Episode]]: + """-> (show_title, show_author, episodes). Handles RSS 2.0, RSS 1.0 (RDF) + and Atom, with any namespace layout (matching by localname).""" + root = ET.fromstring(fetch(feed_url).read()) + chan = next((el for el in root.iter() + if _localname(el.tag) in ("channel", "feed")), root) + eps: list[Episode] = [] + for it in root.iter(): + if _localname(it.tag) not in ("item", "entry"): + continue + url, mime = _find_enclosure(it) + if not url: + continue + eps.append(Episode( + title=_first_text(it, "title"), + pub=_pub(it), + url=url, + mime=mime, + guid=_first_text(it, "guid", "id"), + link=_item_link(it), + )) + return _first_text(chan, "title"), _author(chan), eps +``` + +Note: the old `itunes = "{http://www.itunes.com/dtds/podcast-1.0.dtd}"` constant inside `parse_feed` disappears — `_author`'s localname matching covers both `http://` and `https://` itunes namespace variants for free. + +- [ ] **Step 5: Run tests** + +Run: `pytest tests/test_core.py -q` +Expected: PASS (all, including all three fixture rows). + +- [ ] **Step 6: Commit** + +```bash +git add src/podpull/core.py tests/test_core.py tests/fixtures/feeds/ +git commit -m "feat: namespace-agnostic feed parsing — RSS 2.0, RSS 1.0/RDF, Atom" +``` + +--- + +### Task 4: enclosure fallbacks + duplicate dedupe + +**Files:** +- Create: `tests/fixtures/feeds/media_content_only.xml`, `tests/fixtures/feeds/wavpub.xml`, `tests/fixtures/feeds/omny.xml` +- Test: `tests/test_core.py` (extend `FEED_CASES`) + +The implementation already landed in Task 3 (`_find_enclosure`); this task pins its behavior with fixtures. If any row fails, fix `_find_enclosure`, not the fixture. + +- [ ] **Step 1: Create fixtures** + +`tests/fixtures/feeds/media_content_only.xml` (MRSS only, no `` — today's parser yields zero episodes; also one non-audio `media:content` that must be skipped): + +```xml + + + + MRSS Show + editor@example.test (MRSS Author) + + MRSS EP1 + Thu, 02 Jul 2026 08:00:00 GMT + + + + + +``` + +`tests/fixtures/feeds/wavpub.xml` (半拿铁-shaped: duplicate `enclosure` + `media:content` per item — must yield ONE episode with the `enclosure` URL; note enclosure and media URLs differ so the test catches wrong-priority bugs): + +```xml + + + + 半拿铁 | 商业沉浮录 + 潇磊布道 + + <![CDATA[91. 蜜雪冰城：贫穷限制了它的想象]]> + Wed, 01 Jul 2026 22:00:00 +0800 + wavpub-91 + + + + + +``` + +`tests/fixtures/feeds/omny.xml` (Omny-shaped: enclosure + TWO `media:content` per item): + +```xml + + + + 馬力歐陪你喝一杯 + 鏡好聽 + + EP388 對談 + Tue, 30 Jun 2026 20:00:00 +0800 + + + + + + +``` + +- [ ] **Step 2: Add the (partly failing) case rows** + +Append to `FEED_CASES` in `tests/test_core.py`: + +```python + ("media_content_only.xml", "MRSS Show", "editor@example.test (MRSS Author)", 1, + "https://cdn.example.test/mrss1.mp3", "2026-07-02"), + ("wavpub.xml", "半拿铁 | 商业沉浮录", "潇磊布道", 1, + "https://tk.wavpub.com/track/caffebreve/91.m4a", "2026-07-01"), + ("omny.xml", "馬力歐陪你喝一杯", "鏡好聽", 1, + "https://omny.example.test/ep388.mp3", "2026-06-30"), +``` + +- [ ] **Step 3: Run tests** + +Run: `pytest tests/test_core.py::test_parse_feed_fixture -v` +Expected: PASS for all rows (Task 3's `_find_enclosure` already implements the behavior; treat any failure as an implementation bug to fix in `core.py`). + +- [ ] **Step 4: Commit** + +```bash +git add tests/ +git commit -m "test: enclosure fallback + duplicate media:content dedupe fixtures" +``` + +--- + +### Task 5: dirty-XML recovery (`_sanitize_xml`) + +**Files:** +- Modify: `src/podpull/core.py` (imports; new `_parse_xml` + `_sanitize_xml`; `parse_feed` first line) +- Create: `tests/fixtures/feeds/dirty_entities.xml`, `tests/fixtures/feeds/utf16_bom.xml` +- Test: `tests/test_core.py` + +- [ ] **Step 1: Create fixtures** + +`tests/fixtures/feeds/dirty_entities.xml` — undefined HTML entity, bare `&`, control char, leading blank line. **Write this file exactly as shown** (the blank first line and the literal ` ` / bare `&` are the point; the `\x0b` control char is written via the printf command below since editors strip it): + +```xml + + + + + Dirty Show + Dirty & Sons + + EP1 A & B + Mon, 29 Jun 2026 08:00:00 GMT + + + + +``` + +Then inject a control character into the title (after writing the file): + +```bash +python3 - <<'EOF' +from pathlib import Path +p = Path("tests/fixtures/feeds/dirty_entities.xml") +p.write_bytes(p.read_bytes().replace(b"Dirty Show", b"Dirty \x0bShow")) +EOF +``` + +`tests/fixtures/feeds/utf16_bom.xml` — UTF-16 with BOM. Generate it (don't hand-write): + +```bash +python3 - <<'EOF' +from pathlib import Path +xml = ''' +UTF16 Show +U16 EP1Sun, 28 Jun 2026 08:00:00 GMT + +''' +Path("tests/fixtures/feeds/utf16_bom.xml").write_bytes(b"\xff\xfe" + xml.encode("utf-16-le")) +EOF +``` + +- [ ] **Step 2: Add failing case rows + a fail-loudly test** + +Append to `FEED_CASES`: + +```python + ("dirty_entities.xml", "Dirty Show", "Dirty & Sons", 1, + "https://cdn.example.test/dirty1.mp3?a=1&b=2", "2026-06-29"), + ("utf16_bom.xml", "UTF16 Show", "", 1, + "https://cdn.example.test/u16.mp3", "2026-06-28"), +``` + +And add a standalone test (still-broken XML must raise, not return empty): + +```python +def test_parse_feed_hopeless_xml_raises(monkeypatch): + monkeypatch.setattr(core, "fetch", lambda url, timeout=45: io.BytesIO(b"")) + with pytest.raises(core.ET.ParseError): + core.parse_feed("https://example.test/feed") +``` + +Note on expectations: ` ` becomes a non-breaking space which `_clean` collapses to a regular space → title `"Dirty Show"`; the `\x0b` control char is dropped; `&b=2` keeps its literal `&` after ET unescapes the sanitized `&`. + +- [ ] **Step 3: Run to verify failure** + +Run: `pytest tests/test_core.py -q` +Expected: the two new fixture rows FAIL with `xml.etree.ElementTree.ParseError` (undefined entity / junk before declaration). (`utf16_bom` may pass — expat handles UTF-16 BOMs natively; if so it's a cheap regression guard, keep it.) + +- [ ] **Step 4: Implement** + +In `src/podpull/core.py` add `import html.entities` to the stdlib imports block, in alphabetical order (immediately before `import json`). + +Add above `parse_feed`: + +```python +_XML_PREDEFINED = frozenset({"amp", "lt", "gt", "quot", "apos"}) + + +def _sanitize_xml(raw: bytes) -> bytes: + """Best-effort repair of common real-world feed dirt: junk before the + declaration, control chars, undefined HTML entities, bare '&'.""" + m = re.search(rb'<\?xml[^>]*encoding=["\']([A-Za-z0-9._-]+)["\']', raw[:200]) + enc = m.group(1).decode("ascii", "replace") if m else "utf-8" + try: + text = raw.decode(enc, "replace") + except LookupError: # unknown codec name in declaration + text = raw.decode("utf-8", "replace") + text = text.lstrip("\ufeff\x00 \t\r\n") + # we re-encode as UTF-8 below, so the declared encoding must not disagree + text = re.sub(r'(<\?xml[^>]*?)\s+encoding=["\'][^"\']*["\']', r"\1", text, count=1) + text = re.sub(r"[\x00-\x08\x0b\x0c\x0e-\x1f]", "", text) + + def _entity(mm): + name = mm.group(1) + if name in _XML_PREDEFINED: + return mm.group(0) + ch = html.entities.html5.get(name + ";") + return "".join(f"&#{ord(c)};" for c in ch) if ch else "" + text = re.sub(r"&([A-Za-z][A-Za-z0-9]{1,31});", _entity, text) + text = re.sub(r"&(?![A-Za-z][A-Za-z0-9]{1,31};|#\d+;|#x[0-9A-Fa-f]+;)", "&", text) + return text.encode("utf-8") + + +def _parse_xml(raw: bytes) -> "ET.Element": + try: + return ET.fromstring(raw) + except ET.ParseError as err: + try: + return ET.fromstring(_sanitize_xml(raw)) + except ET.ParseError: + raise err from None # fail loudly with the original error +``` + +In `parse_feed`, change the first line to: + +```python + root = _parse_xml(fetch(feed_url).read()) +``` + +- [ ] **Step 5: Run tests** + +Run: `pytest tests/test_core.py -q` +Expected: PASS (all rows + `test_parse_feed_hopeless_xml_raises`). + +- [ ] **Step 6: Commit** + +```bash +git add src/podpull/core.py tests/ +git commit -m "feat: sanitize-and-retry recovery for dirty feed XML" +``` + +--- + +### Task 6: Chinese-host fixtures (xiaoyuzhou, Ximalaya, SoundOn, Firstory) + +**Files:** +- Create: `tests/fixtures/feeds/xiaoyuzhou.xml`, `ximalaya.xml`, `soundon.xml`, `firstory.xml` +- Test: `tests/test_core.py` (extend `FEED_CASES`) + +Pure regression pinning — parser code from Tasks 3–5 should already handle these. Any failure = fix `core.py`. + +- [ ] **Step 1: Create fixtures** + +`tests/fixtures/feeds/xiaoyuzhou.xml` (忽左忽右-shaped: CDATA, dts-api tracking enclosure with the target host embedded in the path): + +```xml + + + + <![CDATA[忽左忽右]]> + + + + <![CDATA[380 一战爆发110周年：欧洲旧秩序的黄昏]]> + Thu, 02 Jul 2026 20:00:00 +0800 + xyz-380 + https://www.xiaoyuzhoufm.com/episode/abc380 + + 本期节目…

]]> + + + +``` + +`tests/fixtures/feeds/ximalaya.xml` (声动早咖啡-shaped: `//` double-slash enclosure path, `&`-escaped query with nested percent-encoded URL in `jt=`): + +```xml + + + + 声动早咖啡 + 声动活泼 + + 硅谷巨头抢购核电，AI耗电有多猛？ + Mon, 06 Jul 2026 07:30:00 +0800 + xmly-1 + + + + +``` + +`tests/fixtures/feeds/soundon.xml` (股癌-shaped: custom `soundon:` namespace, self-referencing `itunes:new-feed-url`, `?times=` querystring): + +```xml + + + + 股癌 + 謝孟恭 + https://feeds.soundon.fm/podcasts/954689a5-3096-43a4-a80b-7810b219cef3.xml + 954689a5-3096-43a4-a80b-7810b219cef3 + + EP560 | 台股創高 + Sun, 05 Jul 2026 10:00:00 +0800 + soundon-560 + + + + +``` + +`tests/fixtures/feeds/firstory.xml` (百靈果-shaped: EVERYTHING CDATA-wrapped including ``, stale channel-level `pubDate` that must NOT leak into episodes, percent-encoded URL embedded in enclosure path): + +```xml + + + + <![CDATA[百靈果 News]]> + + Mon, 01 Dec 2025 00:00:00 +0000 + + + <![CDATA[EP500 國際新聞]]> + + + + + + +``` + +- [ ] **Step 2: Add case rows** + +Append to `FEED_CASES`: + +```python + ("xiaoyuzhou.xml", "忽左忽右", "JustPod", 1, + "https://dts-api.xiaoyuzhoufm.com/track/cv4bkgpuglwp/xyz380/media.xyzcdn.net/lvJzoGJDVn.m4a", + "2026-07-02"), + ("ximalaya.xml", "声动早咖啡", "声动活泼", 1, + "https://jt.ximalaya.com//GKwRIW8LWq_ZAX-DKgJcpzTV.m4a?channel=rss&jt=https%3A%2F%2Faod.cos.tx.xmcdn.com%2Fstorages%2Fabc.m4a", + "2026-07-06"), + ("soundon.xml", "股癌", "謝孟恭", 1, + "https://rss.soundon.fm/rssf/954689a5/ep560/rssFileVip.mp3?times=1751700000", "2026-07-05"), + ("firstory.xml", "百靈果 News", "百靈果 News", 1, + "https://m.cdn.firstory.me/track/cmjaz594i/https%3A%2F%2Ffile.cdn.firstory.me%2Fstory%2Fep500.mp3", + "2026-07-04"), +``` + +- [ ] **Step 3: Run tests** + +Run: `pytest tests/test_core.py::test_parse_feed_fixture -v` +Expected: PASS for all. (Firstory's `2026-07-04` proves the item's CDATA pubDate is used, not the stale channel one — a wrong implementation that reads channel-level pubDate yields `2025-12-01`.) + +- [ ] **Step 4: Commit** + +```bash +git add tests/ +git commit -m "test: CN host fixtures — xiaoyuzhou, ximalaya, soundon, firstory" +``` + +--- + +### Task 7: remaining host fixtures (Lizhi, Typlog, Fireside, Anchor, Acast, Libsyn, Transistor) + +**Files:** +- Create: `tests/fixtures/feeds/lizhi.xml`, `typlog.xml`, `fireside.xml`, `anchor.xml`, `acast.xml`, `libsyn.xml`, `transistor.xml` +- Test: `tests/test_core.py` (extend `FEED_CASES`) + +- [ ] **Step 1: Create fixtures** + +`tests/fixtures/feeds/lizhi.xml` (大内密谈-shaped: non-zero-padded RFC-822 date): + +```xml + + + + 大内密谈 + 大内密谈 + + vol.1200 深夜谈谈 + Sun, 5 Jul 2026 21:00:00 +0800 + + + + +``` + +`tests/fixtures/feeds/typlog.xml` (疯投圈-shaped: zero CDATA, entity-escaped text, minimal namespaces): + +```xml + + + + 疯投圈 + 黄海、Rio + + 76. 咖啡 & 茶饮：新消费的陈年老酒 + Fri, 03 Jul 2026 12:00:00 GMT + + + + +``` + +`tests/fixtures/feeds/fireside.xml` (科技早知道-shaped: `+0800` dates, Podcasting 2.0 namespace): + +```xml + + + + 科技早知道 + 声动活泼 + no + + S8E20 硅谷现场 + Thu, 02 Jul 2026 08:00:00 +0800 + + + + +``` + +`tests/fixtures/feeds/anchor.xml` (台灣通勤第一品牌-shaped: percent-encoded CloudFront URL inside the enclosure path): + +```xml + + + + 台灣通勤第一品牌 + 台通 + + EP600 通勤路上 + Wed, 01 Jul 2026 16:00:00 GMT + + + + +``` + +`tests/fixtures/feeds/acast.xml` (不明白播客-shaped): + +```xml + + + + 不明白播客 + 袁莉和她的朋友们 + 68004395b4ef799a7a410371 + + EP-100 访谈 + Tue, 30 Jun 2026 22:00:00 GMT + + + + +``` + +`tests/fixtures/feeds/libsyn.xml`: + +```xml + + + + Libsyn Show + Libsyn Author + + Libsyn EP1 + Mon, 29 Jun 2026 09:00:00 +0000 + + + + +``` + +`tests/fixtures/feeds/transistor.xml`: + +```xml + + + + Transistor Show + Transistor Author + + Transistor EP1 + Sun, 28 Jun 2026 09:00:00 +0000 + + + + +``` + +- [ ] **Step 2: Add case rows** + +Append to `FEED_CASES`: + +```python + ("lizhi.xml", "大内密谈", "大内密谈", 1, + "http://cdn.lizhi.fm/audio/2026/07/05/1200.mp3", "2026-07-05"), + ("typlog.xml", "疯投圈", "黄海、Rio", 1, + "https://rio.xyzcdn.net/crazy-capital/ep76.m4a", "2026-07-03"), + ("fireside.xml", "科技早知道", "声动活泼", 1, + "https://aphid.fireside.fm/d/1437767933/s8e20.mp3", "2026-07-02"), + ("anchor.xml", "台灣通勤第一品牌", "台通", 1, + "https://anchor.fm/s/1ea77470/podcast/play/999/https%3A%2F%2Fd3ctxlq1ktw2nl.cloudfront.net%2Fstaging%2Fep600.m4a", + "2026-07-01"), + ("acast.xml", "不明白播客", "袁莉和她的朋友们", 1, + "https://sphinx.acast.com/p/open/s/68004395/e/100/media.mp3", "2026-06-30"), + ("libsyn.xml", "Libsyn Show", "Libsyn Author", 1, + "https://traffic.libsyn.com/secure/example/ep1.mp3?dest-id=1", "2026-06-29"), + ("transistor.xml", "Transistor Show", "Transistor Author", 1, + "https://media.transistor.fm/abc123/ep1.mp3", "2026-06-28"), +``` + +- [ ] **Step 3: Run tests, then the whole suite** + +Run: `pytest tests/test_core.py::test_parse_feed_fixture -v` → PASS (all 18 rows) +Run: `pytest -q` → PASS + +- [ ] **Step 4: Commit** + +```bash +git add tests/ +git commit -m "test: host fixtures — lizhi, typlog, fireside, anchor, acast, libsyn, transistor" +``` + +--- + +### Task 8: browser-UA regression test + +**Files:** +- Test: `tests/test_core.py` + +Four researched hosts (SoundOn, Typlog, Fireside, Lizhi CDN) 403 `Python-urllib` UAs; this pins CLAUDE.md invariant #5 at the code level. + +- [ ] **Step 1: Write the test** + +```python +def test_fetch_and_download_send_browser_ua(monkeypatch, tmp_path): + seen = [] + + class _Resp(io.BytesIO): + status = 200 + length = 2 + headers = {"Content-Type": "audio/mpeg"} + + def __init__(self): + super().__init__(b"ok") + + def fake_urlopen(req, timeout=None): + seen.append(req.get_header("User-agent")) + return _Resp() + + monkeypatch.setattr(core.urllib.request, "urlopen", fake_urlopen) + core.fetch("https://feeds.soundon.fm/x.xml") + core.download_url("https://cdn.lizhi.fm/a.mp3", str(tmp_path / "a.mp3")) + assert seen == [core.UA, core.UA] + assert all("mozilla" in ua.lower() and "python" not in ua.lower() for ua in seen) +``` + +Note: `_Resp.headers` being a plain dict is fine — both `resp.headers.get(...)` call sites only use `.get`. + +- [ ] **Step 2: Run it** + +Run: `pytest tests/test_core.py::test_fetch_and_download_send_browser_ua -v` +Expected: PASS immediately (behavior already exists — this is a pin, not a change). If it fails, `core.fetch`/`download_url` stopped sending `core.UA`: fix core, not the test. + +- [ ] **Step 3: Commit** + +```bash +git add tests/test_core.py +git commit -m "test: pin browser User-Agent on fetch and download (CLAUDE.md invariant 5)" +``` + +--- + +### Task 9: `classify()` accepts Ximalaya album links + +**Files:** +- Modify: `src/podpull/core.py` (`classify`, currently lines 70–84) +- Test: `tests/test_core.py` (`test_classify`) + +- [ ] **Step 1: Extend the test (failing)** + +Add to `test_classify` in `tests/test_core.py`: + +```python + assert core.classify("https://www.ximalaya.com/album/51076156") == \ + ("rss", "https://www.ximalaya.com/album/51076156.xml") + assert core.classify("https://www.ximalaya.com/album/51076156.xml") == \ + ("rss", "https://www.ximalaya.com/album/51076156.xml") +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `pytest tests/test_core.py::test_classify -v` +Expected: FAIL — the album page URL classifies as plain `rss` with the original (non-.xml) URL. + +- [ ] **Step 3: Implement** + +In `core.classify`, insert before the `if s.startswith("http"):` line: + +```python + m = re.search(r"ximalaya\.com/album/(\d+)", s) + if m: # Ximalaya Podcast托管 albums expose RSS at album/.xml + return "rss", f"https://www.ximalaya.com/album/{m.group(1)}.xml" +``` + +Update the docstring's kind list comment to mention that ximalaya album links normalize to `rss`. + +- [ ] **Step 4: Run tests** + +Run: `pytest tests/test_core.py -q` → PASS + +- [ ] **Step 5: Commit** + +```bash +git add src/podpull/core.py tests/test_core.py +git commit -m "feat: accept ximalaya.com/album/ links (normalize to RSS .xml)" +``` + +--- + +### Task 10: download sanity guard (reject `text/*` responses) + +**Files:** +- Modify: `src/podpull/core.py` (`download_url`, currently lines 240–273) +- Test: `tests/test_core.py` + +Motivation: Ximalaya's CDN answers stale enclosure queries with `HTTP 200`, `Content-Type: text/plain`, 7-byte body — podpull would write a 7-byte ".m4a" and exit 0. + +- [ ] **Step 1: Write the failing test** + +```python +def _fake_response(body: bytes, ctype: str): + class _Resp(io.BytesIO): + status = 200 + headers = {"Content-Type": ctype} + + def __init__(self): + super().__init__(body) + self.length = len(body) + return _Resp() + + +def test_download_url_rejects_text_response(monkeypatch, tmp_path): + monkeypatch.setattr(core.urllib.request, "urlopen", + lambda req, timeout=None: _fake_response(b"deleted", "text/plain; charset=utf-8")) + dest = tmp_path / "ep.m4a" + with pytest.raises(ValueError, match="not audio"): + core.download_url("https://jt.ximalaya.com//x.m4a?bad=1", str(dest)) + assert not dest.exists() + + +def test_download_url_accepts_audio_and_octet_stream(monkeypatch, tmp_path): + for ctype in ("audio/mpeg", "application/octet-stream", ""): + monkeypatch.setattr(core.urllib.request, "urlopen", + lambda req, timeout=None, c=ctype: _fake_response(b"AUDIO", c)) + dest = tmp_path / f"ok-{ctype.replace('/', '_') or 'none'}.mp3" + assert core.download_url("https://cdn.example.test/a.mp3", str(dest)) == str(dest) + assert dest.read_bytes() == b"AUDIO" +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `pytest tests/test_core.py::test_download_url_rejects_text_response -v` +Expected: FAIL — no ValueError; the 7-byte file gets written. + +- [ ] **Step 3: Implement** + +In `core.download_url`, right after the `if existing and getattr(resp, "status", 200) == 200:` block and before `mode = "ab" if existing else "wb"`, insert: + +```python + ctype = (resp.headers.get("Content-Type") or "").split(";")[0].strip().lower() + if ctype.startswith("text/"): + # e.g. Ximalaya's CDN answers a stale enclosure query with 200 text/plain + raise ValueError(f"server returned {ctype}, not audio — " + "the feed's enclosure URL may be stale") +``` + +- [ ] **Step 4: Run tests** + +Run: `pytest tests/test_core.py -q` → PASS + +- [ ] **Step 5: Commit** + +```bash +git add src/podpull/core.py tests/test_core.py +git commit -m "feat: reject text/* responses in download_url (stale-enclosure guard)" +``` + +--- + +### Task 11: Podcast Index core functions + +**Files:** +- Modify: `src/podpull/core.py` (imports + new section between "search / resolve" and "direct-episode resolvers") +- Test: `tests/test_core.py` + +- [ ] **Step 1: Write the failing tests** + +```python +def test_pi_credentials(monkeypatch): + monkeypatch.delenv("PODCASTINDEX_API_KEY", raising=False) + monkeypatch.delenv("PODCASTINDEX_API_SECRET", raising=False) + assert core.pi_credentials() is None + monkeypatch.setenv("PODCASTINDEX_API_KEY", "k") + assert core.pi_credentials() is None # secret still missing + monkeypatch.setenv("PODCASTINDEX_API_SECRET", "s") + assert core.pi_credentials() == ("k", "s") + + +def test_pi_headers_deterministic(): + h = core._pi_headers("key", "secret", now=1751900000) + assert h["X-Auth-Key"] == "key" + assert h["X-Auth-Date"] == "1751900000" + import hashlib + assert h["Authorization"] == hashlib.sha1(b"keysecret1751900000").hexdigest() + assert h["User-Agent"] == core.UA + + +def test_pi_search_shows_normalizes(monkeypatch): + monkeypatch.setenv("PODCASTINDEX_API_KEY", "k") + monkeypatch.setenv("PODCASTINDEX_API_SECRET", "s") + monkeypatch.setattr(core, "_pi_get", lambda path, params: { + "feeds": [{"id": 887080, "title": "忽左忽右", "url": "https://feed.xyzfm.space/cv4bkgpuglwp", + "author": "JustPod", "itunesId": 1478791559, "episodeCount": 380}], + }) + rows = core.pi_search_shows("忽左忽右", limit=5) + assert rows == [{"collectionId": 1478791559, "collectionName": "忽左忽右", + "artistName": "JustPod", "feedUrl": "https://feed.xyzfm.space/cv4bkgpuglwp", + "trackCount": 380}] + + +def test_pi_feed_by_itunes_id(monkeypatch): + monkeypatch.setattr(core, "_pi_get", + lambda path, params: {"feed": {"url": "https://feed.example.test/x"}}) + assert core.pi_feed_by_itunes_id("123") == "https://feed.example.test/x" + monkeypatch.setattr(core, "_pi_get", lambda path, params: {"feed": []}) + assert core.pi_feed_by_itunes_id("123") is None + + def _boom(path, params): + raise OSError("network down") + monkeypatch.setattr(core, "_pi_get", _boom) + assert core.pi_feed_by_itunes_id("123") is None # degrades, never raises +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `pytest tests/test_core.py -q -k pi_` +Expected: FAIL with `AttributeError: module 'podpull.core' has no attribute ...` + +- [ ] **Step 3: Implement** + +In `src/podpull/core.py` imports, add `import hashlib` and `import time` (alphabetical order within the stdlib block). Then add a new section after `apple_show_to_feed` (before `parse_feed`'s helpers): + +```python +# --------------------------------------------------------------------------- # +# Podcast Index (optional, BYOK) — free keys at https://api.podcastindex.org +# Active only when both env vars are set; otherwise podpull never contacts PI. +# --------------------------------------------------------------------------- # +PODCASTINDEX_API = "https://api.podcastindex.org/api/1.0" + + +def pi_credentials() -> "tuple[str, str] | None": + key = os.environ.get("PODCASTINDEX_API_KEY", "").strip() + secret = os.environ.get("PODCASTINDEX_API_SECRET", "").strip() + return (key, secret) if key and secret else None + + +def _pi_headers(key: str, secret: str, now: "int | None" = None) -> dict: + ts = str(int(time.time()) if now is None else now) + auth = hashlib.sha1((key + secret + ts).encode()).hexdigest() + return {"User-Agent": UA, "X-Auth-Key": key, "X-Auth-Date": ts, + "Authorization": auth} + + +def _pi_get(path: str, params: dict) -> dict: + creds = pi_credentials() + if not creds: + raise ValueError("Podcast Index credentials not set " + "(PODCASTINDEX_API_KEY / PODCASTINDEX_API_SECRET)") + q = urllib.parse.urlencode(params) + req = urllib.request.Request(f"{PODCASTINDEX_API}{path}?{q}", + headers=_pi_headers(*creds)) + return json.load(urllib.request.urlopen(req, timeout=45)) + + +def pi_search_shows(term: str, limit: int = 10) -> list: + """Search Podcast Index; rows use the same keys as iTunes search results + so the CLI table code needs no changes.""" + data = _pi_get("/search/byterm", {"q": term, "max": limit}) + return [{"collectionId": f.get("itunesId") or "", + "collectionName": f.get("title") or "", + "artistName": f.get("author") or "", + "feedUrl": f.get("url") or "", + "trackCount": f.get("episodeCount") or 0} + for f in data.get("feeds", [])] + + +def pi_feed_by_itunes_id(pid: str) -> "str | None": + """Second-directory feed lookup by Apple ID. Never raises — returns None + so callers degrade gracefully when PI is down or has no entry.""" + try: + feed = _pi_get("/podcasts/byitunesid", {"id": pid}).get("feed") or {} + return (feed.get("url") or None) if isinstance(feed, dict) else None + except Exception: + return None +``` + +- [ ] **Step 4: Run tests** + +Run: `pytest tests/test_core.py -q` → PASS + +- [ ] **Step 5: Commit** + +```bash +git add src/podpull/core.py tests/test_core.py +git commit -m "feat: Podcast Index core — BYOK auth, search, by-itunes-id lookup" +``` + +--- + +### Task 12: `apple_show_to_feed` falls back to Podcast Index + +**Files:** +- Modify: `src/podpull/core.py` (`apple_show_to_feed`, currently lines 98–111) +- Test: `tests/test_core.py` + +- [ ] **Step 1: Write the failing tests** + +```python +def test_apple_show_to_feed_pi_fallback(monkeypatch): + monkeypatch.setenv("PODCASTINDEX_API_KEY", "k") + monkeypatch.setenv("PODCASTINDEX_API_SECRET", "s") + monkeypatch.setattr(core, "fetch_json", lambda url: {"results": []}) # iTunes empty + monkeypatch.setattr(core, "pi_feed_by_itunes_id", + lambda pid: "https://feed.example.test/fallback") + feed, name, author, pid = core.apple_show_to_feed("1478791559") + assert feed == "https://feed.example.test/fallback" + assert pid == "1478791559" + assert name == "" and author == "" + + +def test_apple_show_to_feed_no_creds_still_raises(monkeypatch): + monkeypatch.delenv("PODCASTINDEX_API_KEY", raising=False) + monkeypatch.delenv("PODCASTINDEX_API_SECRET", raising=False) + monkeypatch.setattr(core, "fetch_json", lambda url: {"results": []}) + calls = [] + monkeypatch.setattr(core, "pi_feed_by_itunes_id", + lambda pid: calls.append(pid)) + with pytest.raises(ValueError): + core.apple_show_to_feed("123") + assert calls == [] # PI never contacted without keys +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `pytest tests/test_core.py -q -k apple_show` +Expected: `test_apple_show_to_feed_pi_fallback` FAILS (raises "iTunes lookup returned nothing"). + +- [ ] **Step 3: Implement** + +Replace the body of `apple_show_to_feed` after `pid = m.group(1)`: + +```python + results = fetch_json(f"{ITUNES_LOOKUP}?id={pid}").get("results", []) + r = results[0] if results else {} + feed = r.get("feedUrl") + if not feed and pi_credentials(): # second directory, BYOK-only + feed = pi_feed_by_itunes_id(pid) + if not feed: + if not results: + raise ValueError(f"iTunes lookup returned nothing for id={pid}") + raise ValueError(f"No feedUrl for id={pid} (not a podcast?)") + return feed, r.get("collectionName", ""), r.get("artistName", ""), pid +``` + +- [ ] **Step 4: Run tests** + +Run: `pytest -q` → PASS (full suite: the two new tests plus no regressions). + +- [ ] **Step 5: Commit** + +```bash +git add src/podpull/core.py tests/test_core.py +git commit -m "feat: fall back to Podcast Index when iTunes lookup has no feed" +``` + +--- + +### Task 13: `cmd_search` merges Podcast Index results + +**Files:** +- Modify: `src/podpull/cli.py` (`cmd_search`, currently lines 93–109; `EXAMPLES`, lines 296–310) +- Test: `tests/test_cli.py` + +- [ ] **Step 1: Write the failing tests** + +Add to `tests/test_cli.py` (it already imports `argparse`, `cli`, `core` — follow the file's existing conventions; add imports only if missing): + +```python +def _search_args(term="故事"): + return argparse.Namespace(term=term, limit=10, country="US") + + +def test_search_without_keys_never_calls_pi(monkeypatch): + monkeypatch.setattr(core, "pi_credentials", lambda: None) + monkeypatch.setattr(core, "search_shows", lambda term, limit, country: [ + {"collectionId": 1, "collectionName": "A", "artistName": "x", + "feedUrl": "https://f/a", "trackCount": 3}]) + monkeypatch.setattr(core, "pi_search_shows", + lambda *a, **k: pytest.fail("PI must not be called without keys")) + assert cli.cmd_search(_search_args()) == 0 + + +def test_search_merges_and_dedupes_pi(monkeypatch): + monkeypatch.setattr(core, "pi_credentials", lambda: ("k", "s")) + monkeypatch.setattr(core, "search_shows", lambda term, limit, country: [ + {"collectionId": 1, "collectionName": "A", "artistName": "x", + "feedUrl": "https://f/a", "trackCount": 3}]) + monkeypatch.setattr(core, "pi_search_shows", lambda term, limit: [ + {"collectionId": 1, "collectionName": "A (PI)", "artistName": "x", + "feedUrl": "https://f/a/", "trackCount": 3}, # dup (trailing slash) + {"collectionId": 2, "collectionName": "B", "artistName": "y", + "feedUrl": "https://f/b", "trackCount": 9}]) # new + seen = [] + monkeypatch.setattr(cli, "_render_search_table", lambda term, rows: seen.extend(rows)) + assert cli.cmd_search(_search_args()) == 0 + assert [r["collectionName"] for r in seen] == ["A", "B"] # iTunes wins the dup + + +def test_search_survives_one_backend_failing(monkeypatch): + monkeypatch.setattr(core, "pi_credentials", lambda: ("k", "s")) + + def _boom(term, limit, country): + raise OSError("itunes down") + monkeypatch.setattr(core, "search_shows", _boom) + monkeypatch.setattr(core, "pi_search_shows", lambda term, limit: [ + {"collectionId": 2, "collectionName": "B", "artistName": "y", + "feedUrl": "https://f/b", "trackCount": 9}]) + assert cli.cmd_search(_search_args()) == 0 # PI results still shown + + monkeypatch.setattr(core, "pi_credentials", lambda: None) + with pytest.raises(OSError): # no keys -> unchanged behavior + cli.cmd_search(_search_args()) +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `pytest tests/test_cli.py -q -k search` +Expected: FAIL (`_render_search_table` doesn't exist; merge behavior missing). + +- [ ] **Step 3: Implement** + +In `src/podpull/cli.py`, replace `cmd_search` with: + +```python +def _render_search_table(term: str, results: list) -> None: + table = Table(title=f"Podcasts matching “{term}”", header_style="bold", expand=False) + table.add_column("Apple ID", style="cyan", no_wrap=True) + table.add_column("Eps", justify="right", style="magenta") + table.add_column("Show") + table.add_column("Author", style="dim") + for r in results: + table.add_row(str(r.get("collectionId") or "—"), str(r.get("trackCount") or "?"), + r.get("collectionName") or "", r.get("artistName") or "") + ui.print(table) + ui.print("[dim]Next:[/] podpull list • podpull get ") + + +def _merge_results(primary: list, extra: list) -> list: + seen = {(r.get("feedUrl") or "").strip().rstrip("/") for r in primary} + seen.discard("") + merged = list(primary) + for r in extra: + key = (r.get("feedUrl") or "").strip().rstrip("/") + if key and key in seen: + continue + seen.add(key) + merged.append(r) + return merged + + +def cmd_search(args) -> int: + results, warnings = [], [] + with ui.status(f"[cyan]Searching for “{args.term}”…"): + if core.pi_credentials() is None: + # no PI keys -> exactly the old behavior (errors propagate to main) + results = core.search_shows(args.term, limit=args.limit, country=args.country) + else: + try: + results = core.search_shows(args.term, limit=args.limit, country=args.country) + except Exception as e: + warnings.append(f"iTunes search failed: {e}") + try: + results = _merge_results(results, core.pi_search_shows(args.term, limit=args.limit)) + except Exception as e: + warnings.append(f"Podcast Index search failed: {e}") + for w in warnings: + _err(w) + if not results: + _err("no shows found") + return 1 + _render_search_table(args.term, results) + return 0 +``` + +In `EXAMPLES` (the help epilog), add before the final `"""`: + +``` +[dim]Optional: set PODCASTINDEX_API_KEY + PODCASTINDEX_API_SECRET (free — podcastindex.org)[/] +[dim]to enrich search results and add a feed-resolution fallback.[/] +``` + +- [ ] **Step 4: Run tests** + +Run: `pytest -q` → PASS (full suite). + +- [ ] **Step 5: Commit** + +```bash +git add src/podpull/cli.py tests/test_cli.py +git commit -m "feat: merge Podcast Index results into search (BYOK, graceful degradation)" +``` + +--- + +### Task 14: network-marked integration tests + +**Files:** +- Create: `tests/test_network.py` + +- [ ] **Step 1: Write the file** + +```python +"""Live-network integration tests. EXCLUDED from the default suite. + +Run explicitly: pytest -m network +These hit real hosts and will flake if a show migrates feeds — that's the +point: they detect real-world drift the offline fixtures can't. +""" +import pytest + +from podpull import core + +pytestmark = pytest.mark.network + +LIVE_FEEDS = [ + # (label, feed_url) + ("xiaoyuzhou 忽左忽右", "https://feed.xyzfm.space/cv4bkgpuglwp"), + ("ximalaya 声动早咖啡", "https://www.ximalaya.com/album/51076156.xml"), + ("soundon 股癌", "https://feeds.soundon.fm/podcasts/954689a5-3096-43a4-a80b-7810b219cef3.xml"), + ("firstory 百靈果", "https://feed.firstory.me/rss/user/cmjaz594i0000hdvpdpnd4fw8"), +] + + +@pytest.mark.parametrize("label,feed", LIVE_FEEDS, ids=[f[0] for f in LIVE_FEEDS]) +def test_live_feed_parses(label, feed): + title, _author, eps = core.parse_feed(feed) + assert title, f"{label}: empty show title" + assert eps, f"{label}: no episodes parsed" + first = eps[0] + assert first.url.startswith("http"), f"{label}: bad enclosure {first.url!r}" + assert first.date != "0000-00-00", f"{label}: unparseable pubDate {first.pub!r}" + + +def test_live_itunes_resolution(): + feed, name, _author, pid = core.apple_show_to_feed("1493503146") # 忽左忽右 + assert feed.startswith("http") and pid == "1493503146" and name +``` + +- [ ] **Step 2: Verify exclusion and (optionally) run live** + +Run: `pytest -q` → the new tests are DESELECTED (count unchanged). +Run: `pytest -m network -q` → should pass with live network; if a host is down or a show migrated, note it in the commit message but don't block the branch on it. + +- [ ] **Step 3: Commit** + +```bash +git add tests/test_network.py +git commit -m "test: live-network integration tests (pytest -m network)" +``` + +--- + +### Task 15: docs, integrations note, version bump, final verification + +**Files:** +- Modify: `src/podpull/__init__.py` (version), `pyproject.toml` (version), `CHANGELOG.md`, `README.md`, `src/podpull/integrations/SKILL.md`, `src/podpull/integrations/opencode_command.md`, `src/podpull/integrations/cursor_rule.mdc` + +- [ ] **Step 1: Version bump — BOTH files** + +`src/podpull/__init__.py`: `__version__ = "0.6.0"` +`pyproject.toml`: `version = "0.6.0"` + +- [ ] **Step 2: CHANGELOG entry** + +Add at the top of `CHANGELOG.md` (match the existing entry format in that file): + +```markdown +## 0.6.0 — 2026-07-07 + +- Robust feed parsing: RSS 2.0 / RSS 1.0 (RDF) / Atom, any-namespace matching, + `media:content` + Atom-enclosure fallbacks, dirty-XML sanitize-and-retry + (undefined entities, bare `&`, control chars), ISO-8601 dates. +- Verified against Chinese-market hosts: xiaoyuzhou, Ximalaya, SoundOn, Firstory, + WavPub, Typlog, Fireside, Lizhi (offline fixtures + `pytest -m network` live suite). +- `ximalaya.com/album/` links are now accepted directly. +- Download guard: `text/*` responses (stale enclosures, e.g. Ximalaya CDN) now + error instead of writing a garbage audio file. +- Optional Podcast Index support (BYOK: `PODCASTINDEX_API_KEY`/`_SECRET`): + enriches `search` and adds a feed-resolution fallback when iTunes has no feed. +``` + +(If the tag is cut on a later day, update the date in the same commit as the tag.) + +- [ ] **Step 3: README — add an optional-PI section + ximalaya mention** + +In `README.md`: add `ximalaya.com/album/` to wherever supported inputs are listed, and add a short section: + +```markdown +### Podcast Index (optional) + +podpull can enrich `search` and feed resolution with the open +[Podcast Index](https://podcastindex.org) directory. Get a free API key at +[api.podcastindex.org](https://api.podcastindex.org/signup) and set: + +```bash +export PODCASTINDEX_API_KEY=... +export PODCASTINDEX_API_SECRET=... +``` + +Without these, podpull behaves exactly as before (iTunes only). +``` + +- [ ] **Step 4: Integrations one-liners** + +Add one line to each of `src/podpull/integrations/SKILL.md`, `opencode_command.md`, `cursor_rule.mdc`, in their existing style, near their command/rules list: + +``` +- Optional: if PODCASTINDEX_API_KEY/PODCASTINDEX_API_SECRET are set, `search` also queries Podcast Index and feed resolution gains a fallback; ximalaya.com/album/ links work as a source. +``` + +- [ ] **Step 5: Full verification** + +Run: `pytest -q` → ALL PASS +Run: `ruff check src` → clean +Run: `python -m build && unzip -l dist/*.whl | grep integrations` → integration files present in the wheel (`pip install build` first if missing; then delete `dist/` — it's gitignored but don't commit it) +Run: `podpull --help` → epilog shows the PI note; `podpull search --help` still renders. + +- [ ] **Step 6: Commit** + +```bash +git add src/podpull/__init__.py pyproject.toml CHANGELOG.md README.md src/podpull/integrations/ +git commit -m "docs: v0.6.0 — changelog, README Podcast Index section, integrations note" +``` + +--- + +## Post-plan notes for the executor + +- **Do not** merge or rebase against PR #3 (`--quiet`) mid-plan; if it lands on main first, rebasing this branch is the integrator's job at the end (conflicts, if any, are confined to `cli.py`). +- The release itself (tag `v0.6.0`, PyPI, Homebrew bump) is NOT part of this plan — the user decides when to cut it. No dependency changes were made, so no `brew update-python-resources` run is needed. +- If a live `pytest -m network` run fails because a show migrated its feed, update `LIVE_FEEDS`, don't weaken assertions. diff --git a/docs/superpowers/specs/2026-07-07-feed-robustness-podcastindex-design.md b/docs/superpowers/specs/2026-07-07-feed-robustness-podcastindex-design.md new file mode 100644 index 0000000..04c1203 --- /dev/null +++ b/docs/superpowers/specs/2026-07-07-feed-robustness-podcastindex-design.md @@ -0,0 +1,161 @@ +# Design: robust feed parsing, multi-host tests, Podcast Index support + +Date: 2026-07-07 · Target release: **v0.6.0** · Status: awaiting user review + +Backlog source: Obsidian `Projects/Podpull/跟进 Follow-ups.md` → "later hardening": +robust parsing across arbitrary hosts, tests vs more hosts + a network-marked integ +test, Podcast Index API ("richer search than iTunes"). `--json` mode is **out of +scope** for this iteration (not requested). + +## Decisions (made while user was AFK — please confirm) + +1. **No `feedparser`; harden the stdlib parser instead.** CLAUDE.md invariant #1 + (`core.py` stays pure stdlib) outweighs the backlog's "consider feedparser". + podpull needs only title/author/enclosure/date per item — a narrow slice that + stdlib `ElementTree` + targeted fallbacks covers. Avoids optional-dep matrix, + Homebrew resource churn, and a heavyweight dependency. +2. **Podcast Index = search enrichment + resolve fallback**, BYOK via env vars + `PODCASTINDEX_API_KEY` / `PODCASTINDEX_API_SECRET`. With no keys set, behavior + is byte-for-byte unchanged. Not a first-class input source (no PI URLs/IDs as + `get` arguments) — YAGNI, can be added later. +3. **Host coverage** (revised 2026-07-07 after Chinese-market research, per user + request) = the hosts that actually carry mainstream Chinese-language podcasts — + xiaoyuzhou/xyzfm, Ximalaya RSS export, SoundOn, Firstory, WavPub/播客公社, + Typlog, Fireside, Lizhi — plus global hosts (Anchor/Spotify, Acast, Omny, + Libsyn, Transistor) and synthetic edge-case fixtures (Atom, RSS 1.0/RDF, + dirty entities, media:content-only, UTF-16/BOM). Closed CN platforms + (蜻蜓FM, 网易云音乐播客, Ximalaya's non-enrolled/paid catalog) stay out of + scope — no public RSS; supporting them means scraping, a different project. + +## 1. Robust feed parsing (`core.py`) + +`parse_feed(feed_url) -> (title, author, episodes)` keeps its signature; all +changes are internal. New small helpers (all stdlib): + +- **`_localname(tag)`** — strip `{namespace}` so traversal matches elements by + local name regardless of namespace or prefix. This one change makes RSS 2.0, + namespaced RSS 1.0 (RDF), and Atom traversable with shared code. +- **Feed shapes**: RSS 2.0 `channel/item` (current), RSS 1.0 `{rss-1.0}item`, + Atom `feed/entry`. +- **`_find_enclosure(item)`** — per-item audio URL discovery, priority order: + 1. `` (any namespace) + 2. `` (MRSS ns) with `type` audio/* or `medium="audio"` — + first match wins + 3. Atom `` + Items with no audio URL are skipped, as today. First match wins **and stops** — + WavPub and Omny feeds carry a duplicate `media:content` alongside `enclosure` + per item; the priority order must yield exactly one URL, never two episodes. +- **Metadata fallbacks** (first non-empty wins): + - show author: `itunes:author` (accept both `http://` and `https://` namespace + variants) → `managingEditor` → `dc:creator` → Atom `author/name` + - episode pub date: `pubDate` → `dc:date` → Atom `published`/`updated` + - title/guid/link: by localname (ET already folds CDATA to text); collapse + whitespace +- **`Episode.date`** — RFC-822 parse (current) then ISO-8601 fallback + (`datetime.fromisoformat`, tolerating trailing `Z`); `0000-00-00` as last resort. +- **`_sanitize_xml(raw: bytes)` recovery path** — `ET.fromstring` is tried on the + raw bytes first (honors XML encoding declarations; strip UTF-8/16 BOM first). + Only on `ParseError` do we sanitize and retry once: + - undefined named HTML entities (` ` etc.) → numeric refs via + `html.entities.html5` (XML's five predefined entities left untouched) + - bare `&` not starting a valid entity → `&` + - stray control chars (except tab/newline/CR) → dropped + If the retry still fails, raise the original `ParseError` (fail loudly, not + silently-empty). + +- **Input classification**: `classify()` learns one new pattern — + `ximalaya.com/album/` (album page or `.xml` link) → kind `rss` normalized + to `https://www.ximalaya.com/album/.xml`. Ximalaya's Podcast托管 product + issues public RSS at exactly that URL; for non-enrolled albums the fetch 404s + and the existing error path reports it. (No page scraping.) +- **Download sanity guard** (`download_url`) — Ximalaya's CDN answers a + malformed/stale enclosure query with **HTTP 200, `Content-Type: text/plain`, + a 7-byte body** — today podpull would write a 7-byte ".m4a" and exit 0. New + guard: when the destination is an audio download and the final response's + `Content-Type` starts with `text/`, raise `ValueError` ("server returned text, + not audio — feed enclosure may be stale") instead of writing the file. + `application/octet-stream` and all `audio/*` types remain accepted. + +Explicitly **not** doing: full HTML-in-XML soup recovery, itunes:duration/episode +numbers (no consumer yet), paged feeds (RFC 5005), non-audio enclosure handling, +scraping closed CN platforms (蜻蜓FM / 网易云音乐 / Lizhi-app-only content). + +## 2. Podcast Index support (`core.py` + `cli.py`) + +Core (pure stdlib — auth is `hashlib.sha1`): + +- `PODCASTINDEX_API = "https://api.podcastindex.org/api/1.0"` +- `pi_credentials() -> tuple[str, str] | None` — reads the two env vars; `None` + disables everything PI-related. +- `_pi_get(path, params)` — GET with headers `X-Auth-Key`, `X-Auth-Date` (unix ts), + `Authorization: sha1(key + secret + ts)` hexdigest, and the existing `UA`. +- `pi_search_shows(term, limit) -> list[dict]` — `/search/byterm`; results + normalized to the **same dict keys the iTunes search path yields** + (`collectionName`, `artistName`, `feedUrl`, `collectionId`) so `cli.cmd_search`'s + table code is reused untouched. +- `pi_feed_by_itunes_id(pid) -> str | None` — `/podcasts/byitunesid`; used by + `apple_show_to_feed` as a fallback **only when** iTunes lookup returns no + result or no `feedUrl` and credentials exist. +- **Chinese-content caveat (from 2026-07-07 research):** Podcast Index does + index CN/TW shows (incl. `feed.xyzfm.space` feeds) but lags feed migrations — + observed stale feed URLs (故事FM) and duplicate entries (声动早咖啡 on both its + old Fireside and new xyzfm feeds). iTunes therefore stays the primary + directory; PI is enrichment/fallback only — which is exactly this design. + Related: keep `search_shows`'s `country=US` default; the US storefront indexes + every sampled CN/TW show while `country=CN` is censored (e.g. 不明白播客 absent). + +CLI (`cmd_search`): when credentials exist, query iTunes then Podcast Index, +merge and dedupe by normalized `feedUrl` (iTunes result wins ties), render the +same table. Either backend failing degrades gracefully: warning to stderr, the +other backend's results still shown (search fails only when both fail — and +without keys, iTunes failure behaves exactly as today). No new CLI flags. Help epilog + README document the env +vars; integrations files get a one-line note (env vars only — no flag/UX change). + +## 3. Tests + +- **Offline fixtures** (default suite, no network — CLAUDE.md rule): + `tests/fixtures/feeds/*.xml`, each a trimmed 2–3-item sample reproducing the + host's observed quirks (verified live 2026-07-07): + + | Fixture | Modeled on | Quirk the fixture must reproduce | + |---|---|---| + | `xiaoyuzhou` | 忽左忽右 (`feed.xyzfm.space/cv4bkgpuglwp`) | CDATA-heavy; `dts-api.xiaoyuzhoufm.com/track/...` tracking enclosure with nested host-in-path | + | `ximalaya` | 声动早咖啡 (`ximalaya.com/album/51076156.xml`) | `//` double-slash in enclosure path; `&`-escaped query with nested full URL in `jt=` param | + | `soundon` | 股癌 (`feeds.soundon.fm/podcasts/.xml`) | `soundon:` custom namespace; self-referencing `itunes:new-feed-url`; `?times=` querystring enclosure | + | `firstory` | 百靈果 News (`feed.firstory.me/rss/user/`) | **every** field CDATA-wrapped incl. ``; stale channel-level `pubDate`; percent-encoded URL embedded in enclosure path | + | `wavpub` | 半拿铁 (`proxy.wavpub.com/caffebreve.xml`) | duplicate `enclosure` **and** `media:content` per item (must yield 1 episode, not 2); 15 declared namespaces | + | `typlog` | 疯投圈 (`crazy.capital/feed`) | zero CDATA, fully entity-escaped text (counter-fixture) | + | `fireside` | 科技早知道 (`feeds.fireside.fm/guiguzaozhidao/rss`) | `+0800` timezone pubDates; `podcast:` (Podcasting 2.0) namespace | + | `lizhi` | 大内密谈 (`rss.lizhi.fm/rss/14275.xml`) | non-zero-padded RFC-822 dates (`Sun, 5 Jul 2026`); plain-`http` feed URL | + | `anchor` | 台灣通勤第一品牌 | percent-encoded CloudFront URL inside enclosure path | + | `acast`, `omny`, `libsyn`, `transistor` | 不明白播客 / 馬力歐陪你喝一杯 / — | Omny: 2× `media:content` + enclosure per item; others: mainstream-global regression | + | `atom`, `rss10_rdf`, `dirty_entities`, `media_content_only`, `utf16_bom` | synthetic | Atom feed/entry; RSS 1.0 namespaced items; undefined ` ` + bare `&`; MRSS-only items; UTF-16 + BOM | + + A parametrized `test_core` case asserts per fixture: parse succeeds, expected + show title/author, episode count (dedupe!), first episode URL + date. Plus a + unit test that `fetch`/`download_url` send `core.UA` — four of the CN hosts + (SoundOn, Typlog, Fireside, Lizhi CDN) 403 `Python-urllib`, so the browser-UA + invariant (CLAUDE.md #5) now has teeth beyond xiaoyuzhou. +- **Podcast Index tests** (offline): auth-header construction with a frozen + timestamp; env-var gating (no keys → PI functions never called); result + normalization; `apple_show_to_feed` fallback path; search merge/dedupe in + `cli` with mocked core. +- **Network integ test**: `tests/test_network.py`, `@pytest.mark.network`, + excluded by default via `pyproject.toml` (`markers` + `addopts = -m "not network"`). + Hits ~4 live feeds end-to-end (resolve → parse → enclosure URL sanity): + xiaoyuzhou (忽左忽右), Ximalaya (声动早咖啡), SoundOn (股癌), Firstory (百靈果). + Run manually with `pytest -m network`; not wired into CI in this iteration. + +## Error handling summary + +- Malformed XML: one sanitize-retry, then original `ParseError` propagates (cli + already prints errors to stderr and exits non-zero). +- PI unavailable/misconfigured: stderr warning, feature degrades to iTunes-only. +- No credentials: PI code paths are dead — zero behavioral or network change. + +## Release notes / docs to touch + +- `CHANGELOG.md` v0.6.0; version bump in `__init__.py` + `pyproject.toml` (both). +- README: "Podcast Index (optional)" section — free key signup, env vars. +- `src/podpull/integrations/*`: one-line env-var note. +- No dependency changes → no Homebrew resource work. diff --git a/pyproject.toml b/pyproject.toml index 512f895..3655165 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "podpull" -version = "0.5.1" +version = "0.6.0" description = "Download specific podcast episode audio from Apple Podcasts, RSS feeds, or xiaoyuzhou links — with an interactive picker." readme = "README.md" requires-python = ">=3.9" @@ -40,3 +40,9 @@ packages = ["src/podpull"] [tool.ruff] line-length = 100 target-version = "py39" + +[tool.pytest.ini_options] +markers = [ + "network: hits live podcast hosts; excluded by default (run: pytest -m network)", +] +addopts = "-m 'not network'" diff --git a/src/podpull/__init__.py b/src/podpull/__init__.py index 9932c67..bb0052b 100644 --- a/src/podpull/__init__.py +++ b/src/podpull/__init__.py @@ -6,4 +6,4 @@ the file. Stdlib only. """ -__version__ = "0.5.1" +__version__ = "0.6.0" diff --git a/src/podpull/cli.py b/src/podpull/cli.py index fd2dd24..0dbe1de 100644 --- a/src/podpull/cli.py +++ b/src/podpull/cli.py @@ -112,22 +112,53 @@ def cb(done: int, total: int, _t=task) -> None: # --------------------------------------------------------------------------- # # commands # --------------------------------------------------------------------------- # -def cmd_search(args) -> int: - with ui.status(f"[cyan]Searching iTunes for “{args.term}”…"): - results = core.search_shows(args.term, limit=args.limit, country=args.country) - if not results: - _err("no shows found") - return 1 - table = Table(title=f"Podcasts matching “{args.term}”", header_style="bold", expand=False) +def _render_search_table(term: str, results: list) -> None: + table = Table(title=f"Podcasts matching “{term}”", header_style="bold", expand=False) table.add_column("Apple ID", style="cyan", no_wrap=True) table.add_column("Eps", justify="right", style="magenta") table.add_column("Show") table.add_column("Author", style="dim") for r in results: - table.add_row(str(r.get("collectionId")), str(r.get("trackCount") or "?"), + table.add_row(str(r.get("collectionId") or "—"), str(r.get("trackCount") or "?"), r.get("collectionName") or "", r.get("artistName") or "") ui.print(table) ui.print("[dim]Next:[/] podpull list • podpull get ") + + +def _merge_results(primary: list, extra: list) -> list: + seen = {(r.get("feedUrl") or "").strip().rstrip("/") for r in primary} + seen.discard("") + merged = list(primary) + for r in extra: + key = (r.get("feedUrl") or "").strip().rstrip("/") + if key and key in seen: + continue + seen.add(key) + merged.append(r) + return merged + + +def cmd_search(args) -> int: + results, warnings = [], [] + with ui.status(f"[cyan]Searching for “{args.term}”…"): + if core.pi_credentials() is None: + # no PI keys -> exactly the old behavior (errors propagate to main) + results = core.search_shows(args.term, limit=args.limit, country=args.country) + else: + try: + results = core.search_shows(args.term, limit=args.limit, country=args.country) + except Exception as e: + warnings.append(f"iTunes search failed: {e}") + try: + results = _merge_results(results, core.pi_search_shows(args.term, limit=args.limit)) + except Exception as e: + warnings.append(f"Podcast Index search failed: {e}") + for w in warnings: + _err(w) + if not results: + _err("no shows found") + return 1 + _render_search_table(args.term, results) return 0 @@ -336,6 +367,9 @@ def cmd_skills_uninstall(args) -> int: [dim] = Apple show URL · bare Apple ID · RSS feed URL · Apple ?i= episode URL · xiaoyuzhou link[/] [dim]Downloads default to ~/Downloads/Podcasts (override with --out).[/] + +[dim]Optional: set PODCASTINDEX_API_KEY + PODCASTINDEX_API_SECRET (free — podcastindex.org)[/] +[dim]to enrich search results and add a feed-resolution fallback.[/] """ diff --git a/src/podpull/core.py b/src/podpull/core.py index 558c3b0..e75e4c7 100644 --- a/src/podpull/core.py +++ b/src/podpull/core.py @@ -4,15 +4,19 @@ """ from __future__ import annotations +import hashlib +import html.entities import json import os import re +import time import unicodedata import urllib.error import urllib.parse import urllib.request import xml.etree.ElementTree as ET from dataclasses import dataclass, field +from datetime import datetime from email.utils import parsedate_to_datetime # A plain browser User-Agent. Some podcast CDNs (e.g. xiaoyuzhou's feed.xyzfm.space) @@ -37,10 +41,16 @@ class Episode: @property def date(self) -> str: - try: - return parsedate_to_datetime(self.pub).strftime("%Y-%m-%d") - except Exception: - return "0000-00-00" + for parse in ( + lambda s: parsedate_to_datetime(s), # RFC-822 + lambda s: datetime.fromisoformat( # ISO-8601 + re.sub(r"([+-]\d{2})(\d{2})$", r"\1:\2", s.strip().replace("Z", "+00:00"))), + ): + try: + return parse(self.pub).strftime("%Y-%m-%d") + except Exception: + continue + return "0000-00-00" @dataclass @@ -69,7 +79,7 @@ def fetch_json(url: str): # --------------------------------------------------------------------------- # def classify(src: str) -> tuple[str, str]: """Return (kind, normalized_src). kind ∈ {apple_show, apple_episode, - xyz_episode, rss}.""" + xyz_episode, rss}. Ximalaya album links normalize to rss with .xml extension.""" s = src.strip() if re.fullmatch(r"\d+", s): return "apple_show", s @@ -79,6 +89,9 @@ def classify(src: str) -> tuple[str, str]: return "apple_episode", s if "podcasts.apple.com" in s: return "apple_show", s + m = re.search(r"ximalaya\.com/album/(\d+)", s) + if m: # Ximalaya Podcast托管 albums expose RSS at album/.xml + return "rss", f"https://www.ximalaya.com/album/{m.group(1)}.xml" if s.startswith("http"): return "rss", s raise ValueError(f"Cannot classify input: {src!r}") @@ -102,41 +115,220 @@ def apple_show_to_feed(src: str) -> tuple[str, str, str, str]: raise ValueError(f"No Apple podcast id (idNNN) found in: {src}") pid = m.group(1) results = fetch_json(f"{ITUNES_LOOKUP}?id={pid}").get("results", []) - if not results: - raise ValueError(f"iTunes lookup returned nothing for id={pid}") - r = results[0] + r = results[0] if results else {} feed = r.get("feedUrl") + if not feed and pi_credentials(): # second directory, BYOK-only + feed = pi_feed_by_itunes_id(pid) if not feed: + if not results: + raise ValueError(f"iTunes lookup returned nothing for id={pid}") raise ValueError(f"No feedUrl for id={pid} (not a podcast?)") return feed, r.get("collectionName", ""), r.get("artistName", ""), pid +# --------------------------------------------------------------------------- # +# Podcast Index (optional, BYOK) — free keys at https://api.podcastindex.org +# Active only when both env vars are set; otherwise podpull never contacts PI. +# --------------------------------------------------------------------------- # +PODCASTINDEX_API = "https://api.podcastindex.org/api/1.0" + + +def pi_credentials() -> "tuple[str, str] | None": + key = os.environ.get("PODCASTINDEX_API_KEY", "").strip() + secret = os.environ.get("PODCASTINDEX_API_SECRET", "").strip() + return (key, secret) if key and secret else None + + +def _pi_headers(key: str, secret: str, now: "int | None" = None) -> dict: + ts = str(int(time.time()) if now is None else now) + auth = hashlib.sha1((key + secret + ts).encode()).hexdigest() + return {"User-Agent": UA, "X-Auth-Key": key, "X-Auth-Date": ts, + "Authorization": auth} + + +def _pi_get(path: str, params: dict) -> dict: + creds = pi_credentials() + if not creds: + raise ValueError("Podcast Index credentials not set " + "(PODCASTINDEX_API_KEY / PODCASTINDEX_API_SECRET)") + q = urllib.parse.urlencode(params) + req = urllib.request.Request(f"{PODCASTINDEX_API}{path}?{q}", + headers=_pi_headers(*creds)) + return json.load(urllib.request.urlopen(req, timeout=45)) + + +def pi_search_shows(term: str, limit: int = 10) -> list: + """Search Podcast Index; rows use the same keys as iTunes search results + so the CLI table code needs no changes.""" + data = _pi_get("/search/byterm", {"q": term, "max": limit}) + return [{"collectionId": f.get("itunesId") or "", + "collectionName": f.get("title") or "", + "artistName": f.get("author") or "", + "feedUrl": f.get("url") or "", + "trackCount": f.get("episodeCount") or 0} + for f in data.get("feeds", [])] + + +def pi_feed_by_itunes_id(pid: str) -> "str | None": + """Second-directory feed lookup by Apple ID. Never raises — returns None + so callers degrade gracefully when PI is down or has no entry.""" + try: + feed = _pi_get("/podcasts/byitunesid", {"id": pid}).get("feed") or {} + return (feed.get("url") or None) if isinstance(feed, dict) else None + except Exception: + return None + + +def _localname(tag) -> str: + """'{ns}Tag' -> 'tag'. Comments/PIs have non-str tags -> ''.""" + if not isinstance(tag, str): + return "" + return tag.rsplit("}", 1)[-1].lower() + + +def _clean(text) -> str: + return re.sub(r"\s+", " ", (text or "").strip()) + + +def _first_text(el, *names) -> str: + """First non-empty direct-child text whose localname is in names. + Bare (un-namespaced) tags win over namespaced synonyms, so a real + beats <itunes:title> regardless of document order.""" + wanted = {n.lower() for n in names} + candidates = [c for c in el if _localname(c.tag) in wanted] + for c in candidates: + if isinstance(c.tag, str) and "}" not in c.tag and _clean(c.text): + return _clean(c.text) + for c in candidates: + if _clean(c.text): + return _clean(c.text) + return "" + + +def _author(el) -> str: + """itunes:author / atom author>name -> managingEditor -> dc:creator.""" + for name in ("author", "managingeditor", "creator"): + for child in el: + if _localname(child.tag) == name: + txt = _first_text(child, "name") or _clean(child.text) + if txt: + return txt + return "" + + +def _pub(item) -> str: + """pubDate -> dc:date -> atom published -> atom updated (raw string).""" + for name in ("pubdate", "date", "published", "updated"): + for child in item: + if _localname(child.tag) == name and _clean(child.text): + return child.text.strip() + return "" + + +def _find_enclosure(item) -> tuple[str, str]: + """-> (audio_url, mime) or ('', ''). Priority: enclosure > media:content + (audio) > atom link rel=enclosure. First match wins — WavPub/Omny items + carry BOTH enclosure and media:content; one item must yield one URL.""" + for child in item: + if _localname(child.tag) == "enclosure" and child.get("url"): + return child.get("url"), child.get("type") or "" + for child in item: + if _localname(child.tag) == "content" and child.get("url"): + mime = child.get("type") or "" + if mime.startswith("audio/") or child.get("medium") == "audio": + return child.get("url"), mime + for child in item: + if (_localname(child.tag) == "link" and child.get("rel") == "enclosure" + and child.get("href")): + return child.get("href"), child.get("type") or "" + return "", "" + + +def _item_link(item) -> str: + link = _first_text(item, "link") + if link: + return link + for child in item: # atom: <link href=…/> with no/alternate rel + if (_localname(child.tag) == "link" and child.get("href") + and child.get("rel") in (None, "alternate")): + return child.get("href") + return "" + + +_XML_PREDEFINED = frozenset({"amp", "lt", "gt", "quot", "apos"}) + + +def _repair_entities(text: str) -> str: + def _entity(mm): + name = mm.group(1) + if name in _XML_PREDEFINED: + return mm.group(0) + ch = html.entities.html5.get(name + ";") + # unknown entities stay visible as literal text, never dropped + return "".join(f"&#{ord(c)};" for c in ch) if ch else "&" + name + ";" + text = re.sub(r"&([A-Za-z][A-Za-z0-9]{1,31});", _entity, text) + return re.sub(r"&(?![A-Za-z][A-Za-z0-9]{1,31};|#\d+;|#x[0-9A-Fa-f]+;)", "&", text) + + +def _sanitize_xml(raw: bytes) -> bytes: + """Best-effort repair of common real-world feed dirt: junk before the + declaration, control chars, undefined HTML entities, bare '&'. CDATA + sections pass through untouched. Known limitation: a bare '&' that spells + an HTML entity name inside a URL query (\u2026?id=1§ion=2) is converted + like an entity \u2014 unfixable without a full parser.""" + head = raw[:64] + if head[:2] in (b"\xff\xfe", b"\xfe\xff"): + enc = "utf-16" # BOM decides the byte order + elif b"\x00" in head: # null-interleaved: BOM-less UTF-16 + enc = "utf-16-be" if head.startswith(b"\x00") else "utf-16-le" + else: + m = re.search(rb'<\?xml[^>]*encoding=["\']([A-Za-z0-9._-]+)["\']', raw[:200]) + enc = m.group(1).decode("ascii", "replace") if m else "utf-8" + try: + text = raw.decode(enc, "replace") + except LookupError: # unknown codec name in declaration + text = raw.decode("utf-8", "replace") + text = text.lstrip("\ufeff\x00 \t\r\n") + # we re-encode as UTF-8 below, so the declared encoding must not disagree + text = re.sub(r'(<\?xml[^>]*?)\s+encoding=["\'][^"\']*["\']', r"\1", text, count=1) + text = re.sub(r"[\x00-\x08\x0b\x0c\x0e-\x1f]", "", text) + parts = re.split(r"(<!\[CDATA\[.*?\]\]>)", text, flags=re.S) + return "".join(p if p.startswith("<![CDATA[") else _repair_entities(p) + for p in parts).encode("utf-8") + + +def _parse_xml(raw: bytes) -> "ET.Element": + try: + return ET.fromstring(raw) + except ET.ParseError as err: + try: + return ET.fromstring(_sanitize_xml(raw)) + except ET.ParseError: + raise err from None # fail loudly with the original error + + def parse_feed(feed_url: str) -> tuple[str, str, list[Episode]]: - """-> (show_title, show_author, episodes).""" - root = ET.fromstring(fetch(feed_url).read()) - chan = root.find(".//channel") - title = (chan.findtext("title") if chan is not None else "") or "" - itunes = "{http://www.itunes.com/dtds/podcast-1.0.dtd}" - author = "" - if chan is not None: - author = (chan.findtext(f"{itunes}author") or "").strip() + """-> (show_title, show_author, episodes). Handles RSS 2.0, RSS 1.0 (RDF) + and Atom, with any namespace layout (matching by localname).""" + root = _parse_xml(fetch(feed_url).read()) + chan = next((el for el in root.iter() + if _localname(el.tag) in ("channel", "feed")), root) eps: list[Episode] = [] - for it in root.findall(".//item"): - enc = it.find("enclosure") - if enc is None: + for it in root.iter(): + if _localname(it.tag) not in ("item", "entry"): continue - enc_url = enc.get("url") - if not enc_url: + url, mime = _find_enclosure(it) + if not url: continue eps.append(Episode( - title=(it.findtext("title") or "").strip(), - pub=(it.findtext("pubDate") or "").strip(), - url=enc_url, - mime=enc.get("type") or "", - guid=(it.findtext("guid") or "").strip(), - link=(it.findtext("link") or "").strip(), + title=_first_text(it, "title"), + pub=_pub(it), + url=url, + mime=mime, + guid=_first_text(it, "guid", "id"), + link=_item_link(it), )) - return title.strip(), author, eps + return _first_text(chan, "title"), _author(chan), eps def resolve_show(src: str) -> Show: @@ -257,6 +449,11 @@ def download_url(url: str, dest: str, *, resume: bool = True, on_progress=None) raise if existing and getattr(resp, "status", 200) == 200: existing = 0 # server ignored Range; restart cleanly + ctype = (resp.headers.get("Content-Type") or "").split(";")[0].strip().lower() + if ctype.startswith("text/"): + # e.g. Ximalaya's CDN answers a stale enclosure query with 200 text/plain + raise ValueError(f"server returned {ctype}, not audio — " + "the feed's enclosure URL may be stale") mode = "ab" if existing else "wb" remaining = resp.length or 0 total = (remaining + existing) if remaining else 0 diff --git a/src/podpull/integrations/SKILL.md b/src/podpull/integrations/SKILL.md index eae0995..56aeffe 100644 --- a/src/podpull/integrations/SKILL.md +++ b/src/podpull/integrations/SKILL.md @@ -48,6 +48,7 @@ podpull get <src> ... -q / --quiet # suppress spinner/progress bar - Downloads default to `~/Downloads/Podcasts`; pass `--out DIR` to change. - Selecting **multiple** episodes creates a per-show sub-folder. Filenames are normalized to be cloud-storage-safe (emoji/illegal characters removed; CJK kept). +- Optional: if PODCASTINDEX_API_KEY/PODCASTINDEX_API_SECRET are set, `search` also queries Podcast Index and feed resolution gains a fallback; ximalaya.com/album/<id> links work as a source. ## Examples diff --git a/src/podpull/integrations/cursor_rule.mdc b/src/podpull/integrations/cursor_rule.mdc index 8711b5f..7b25b54 100644 --- a/src/podpull/integrations/cursor_rule.mdc +++ b/src/podpull/integrations/cursor_rule.mdc @@ -24,3 +24,4 @@ Rules: output (the saved file path) is unaffected. - File path prints to stdout, progress to stderr. Default output `~/Downloads/Podcasts` (`--out` to change). - Multiple episodes → per-show folder; filenames are normalized to be cloud-storage-safe (CJK kept). +- Optional: if PODCASTINDEX_API_KEY/PODCASTINDEX_API_SECRET are set, `search` also queries Podcast Index and feed resolution gains a fallback; ximalaya.com/album/<id> links work as a source. diff --git a/src/podpull/integrations/opencode_command.md b/src/podpull/integrations/opencode_command.md index a436b3c..4f1a01f 100644 --- a/src/podpull/integrations/opencode_command.md +++ b/src/podpull/integrations/opencode_command.md @@ -22,5 +22,6 @@ Rules: meaningfully outside a live terminal anyway. The saved file path still prints to stdout regardless. - The saved file path is printed to stdout; progress to stderr. Default output dir is `~/Downloads/Podcasts` (`--out DIR` to change). Multiple episodes go into a per-show folder; filenames are cloud-safe. +- Optional: if PODCASTINDEX_API_KEY/PODCASTINDEX_API_SECRET are set, `search` also queries Podcast Index and feed resolution gains a fallback; ximalaya.com/album/<id> links work as a source. Steps: figure out the show (search/list if needed), then download with the right selector, then report the saved path(s). diff --git a/tests/fixtures/feeds/acast.xml b/tests/fixtures/feeds/acast.xml new file mode 100644 index 0000000..defa959 --- /dev/null +++ b/tests/fixtures/feeds/acast.xml @@ -0,0 +1,14 @@ +<?xml version="1.0" encoding="UTF-8"?> +<rss version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" + xmlns:acast="https://schema.acast.com/1.0/"> + <channel> + <title>不明白播客 + 袁莉和她的朋友们 + 68004395b4ef799a7a410371 + + EP-100 访谈 + Tue, 30 Jun 2026 22:00:00 GMT + + + + diff --git a/tests/fixtures/feeds/anchor.xml b/tests/fixtures/feeds/anchor.xml new file mode 100644 index 0000000..efec475 --- /dev/null +++ b/tests/fixtures/feeds/anchor.xml @@ -0,0 +1,12 @@ + + + + 台灣通勤第一品牌 + 台通 + + EP600 通勤路上 + Wed, 01 Jul 2026 16:00:00 GMT + + + + diff --git a/tests/fixtures/feeds/atom.xml b/tests/fixtures/feeds/atom.xml new file mode 100644 index 0000000..4183b58 --- /dev/null +++ b/tests/fixtures/feeds/atom.xml @@ -0,0 +1,12 @@ + + + Atom Show + Atom Author + + Atom EP1 + atom-ep1 + 2026-07-01T08:30:00Z + + + + diff --git a/tests/fixtures/feeds/dirty_entities.xml b/tests/fixtures/feeds/dirty_entities.xml new file mode 100644 index 0000000..0c7636a --- /dev/null +++ b/tests/fixtures/feeds/dirty_entities.xml @@ -0,0 +1,13 @@ + + + + + Dirty  Show + Dirty & Sons + + EP1 A & B + Mon, 29 Jun 2026 08:00:00 GMT + + + + diff --git a/tests/fixtures/feeds/fireside.xml b/tests/fixtures/feeds/fireside.xml new file mode 100644 index 0000000..3cdb690 --- /dev/null +++ b/tests/fixtures/feeds/fireside.xml @@ -0,0 +1,14 @@ + + + + 科技早知道 + 声动活泼 + no + + S8E20 硅谷现场 + Thu, 02 Jul 2026 08:00:00 +0800 + + + + diff --git a/tests/fixtures/feeds/firstory.xml b/tests/fixtures/feeds/firstory.xml new file mode 100644 index 0000000..c1e6ad2 --- /dev/null +++ b/tests/fixtures/feeds/firstory.xml @@ -0,0 +1,15 @@ + + + + <![CDATA[百靈果 News]]> + + Mon, 01 Dec 2025 00:00:00 +0000 + + + <![CDATA[EP500 國際新聞]]> + + + + + + \ No newline at end of file diff --git a/tests/fixtures/feeds/itunes_title_order.xml b/tests/fixtures/feeds/itunes_title_order.xml new file mode 100644 index 0000000..6662e82 --- /dev/null +++ b/tests/fixtures/feeds/itunes_title_order.xml @@ -0,0 +1,14 @@ + + + + Short Channel + Real Channel Title + Order Author + + Short EP + Real Episode Title + Sat, 04 Jul 2026 08:00:00 GMT + + + + diff --git a/tests/fixtures/feeds/libsyn.xml b/tests/fixtures/feeds/libsyn.xml new file mode 100644 index 0000000..70aec92 --- /dev/null +++ b/tests/fixtures/feeds/libsyn.xml @@ -0,0 +1,12 @@ + + + + Libsyn Show + Libsyn Author + + Libsyn EP1 + Mon, 29 Jun 2026 09:00:00 +0000 + + + + diff --git a/tests/fixtures/feeds/lizhi.xml b/tests/fixtures/feeds/lizhi.xml new file mode 100644 index 0000000..78037f2 --- /dev/null +++ b/tests/fixtures/feeds/lizhi.xml @@ -0,0 +1,12 @@ + + + + 大内密谈 + 大内密谈 + + vol.1200 深夜谈谈 + Sun, 5 Jul 2026 21:00:00 +0800 + + + + diff --git a/tests/fixtures/feeds/media_content_only.xml b/tests/fixtures/feeds/media_content_only.xml new file mode 100644 index 0000000..182c454 --- /dev/null +++ b/tests/fixtures/feeds/media_content_only.xml @@ -0,0 +1,13 @@ + + + + MRSS Show + editor@example.test (MRSS Author) + + MRSS EP1 + Thu, 02 Jul 2026 08:00:00 GMT + + + + + diff --git a/tests/fixtures/feeds/omny.xml b/tests/fixtures/feeds/omny.xml new file mode 100644 index 0000000..663a967 --- /dev/null +++ b/tests/fixtures/feeds/omny.xml @@ -0,0 +1,15 @@ + + + + 馬力歐陪你喝一杯 + 鏡好聽 + + EP388 對談 + Tue, 30 Jun 2026 20:00:00 +0800 + + + + + + diff --git a/tests/fixtures/feeds/rss10_rdf.xml b/tests/fixtures/feeds/rss10_rdf.xml new file mode 100644 index 0000000..16c2e46 --- /dev/null +++ b/tests/fixtures/feeds/rss10_rdf.xml @@ -0,0 +1,14 @@ + + + + RDF Show + RDF Author + + + RDF EP1 + 2026-06-30T10:00:00Z + + + diff --git a/tests/fixtures/feeds/rss2_plain.xml b/tests/fixtures/feeds/rss2_plain.xml new file mode 100644 index 0000000..0f7ff5e --- /dev/null +++ b/tests/fixtures/feeds/rss2_plain.xml @@ -0,0 +1,20 @@ + + + + Plain RSS2 Show + Plain Author + + EP2 + Fri, 03 Jul 2026 08:00:00 GMT + plain-ep2 + https://example.test/ep2 + + + + EP1 + Wed, 01 Jul 2026 08:00:00 GMT + plain-ep1 + + + + diff --git a/tests/fixtures/feeds/soundon.xml b/tests/fixtures/feeds/soundon.xml new file mode 100644 index 0000000..512c843 --- /dev/null +++ b/tests/fixtures/feeds/soundon.xml @@ -0,0 +1,16 @@ + + + + 股癌 + 謝孟恭 + https://feeds.soundon.fm/podcasts/954689a5-3096-43a4-a80b-7810b219cef3.xml + 954689a5-3096-43a4-a80b-7810b219cef3 + + EP560 | 台股創高 + Sun, 05 Jul 2026 10:00:00 +0800 + soundon-560 + + + + \ No newline at end of file diff --git a/tests/fixtures/feeds/transistor.xml b/tests/fixtures/feeds/transistor.xml new file mode 100644 index 0000000..f5439f5 --- /dev/null +++ b/tests/fixtures/feeds/transistor.xml @@ -0,0 +1,12 @@ + + + + Transistor Show + Transistor Author + + Transistor EP1 + Sun, 28 Jun 2026 09:00:00 +0000 + + + + diff --git a/tests/fixtures/feeds/typlog.xml b/tests/fixtures/feeds/typlog.xml new file mode 100644 index 0000000..c20945e --- /dev/null +++ b/tests/fixtures/feeds/typlog.xml @@ -0,0 +1,12 @@ + + + + 疯投圈 + 黄海、Rio + + 76. 咖啡 & 茶饮：新消费的陈年老酒 + Fri, 03 Jul 2026 12:00:00 GMT + + + + diff --git a/tests/fixtures/feeds/utf16_bom.xml b/tests/fixtures/feeds/utf16_bom.xml new file mode 100644 index 0000000..20ff29f Binary files /dev/null and b/tests/fixtures/feeds/utf16_bom.xml differ diff --git a/tests/fixtures/feeds/wavpub.xml b/tests/fixtures/feeds/wavpub.xml new file mode 100644 index 0000000..c056ad4 --- /dev/null +++ b/tests/fixtures/feeds/wavpub.xml @@ -0,0 +1,15 @@ + + + + 半拿铁 | 商业沉浮录 + 潇磊布道 + + <![CDATA[91. 蜜雪冰城：贫穷限制了它的想象]]> + Wed, 01 Jul 2026 22:00:00 +0800 + wavpub-91 + + + + + diff --git a/tests/fixtures/feeds/xiaoyuzhou.xml b/tests/fixtures/feeds/xiaoyuzhou.xml new file mode 100644 index 0000000..5ec0736 --- /dev/null +++ b/tests/fixtures/feeds/xiaoyuzhou.xml @@ -0,0 +1,17 @@ + + + + <![CDATA[忽左忽右]]> + + + + <![CDATA[380 一战爆发110周年：欧洲旧秩序的黄昏]]> + Thu, 02 Jul 2026 20:00:00 +0800 + xyz-380 + https://www.xiaoyuzhoufm.com/episode/abc380 + + 本期节目…

]]> + + + \ No newline at end of file diff --git a/tests/fixtures/feeds/ximalaya.xml b/tests/fixtures/feeds/ximalaya.xml new file mode 100644 index 0000000..781f7cc --- /dev/null +++ b/tests/fixtures/feeds/ximalaya.xml @@ -0,0 +1,13 @@ + + + + 声动早咖啡 + 声动活泼 + + 硅谷巨头抢购核电，AI耗电有多猛？ + Mon, 06 Jul 2026 07:30:00 +0800 + xmly-1 + + + + \ No newline at end of file diff --git a/tests/test_cli.py b/tests/test_cli.py index a22ae7b..be9a58f 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -5,6 +5,8 @@ import sys import types +import pytest + from podpull import cli, core from rich.console import Console @@ -168,3 +170,49 @@ def fake_dl(ep, out, **kw): assert os.path.exists(os.path.join(show_dir, "EP0.mp3")) assert os.path.exists(os.path.join(show_dir, "EP2.mp3")) assert not os.path.exists(os.path.join(show_dir, "EP1.mp3")) + + +def _search_args(term="故事"): + return argparse.Namespace(term=term, limit=10, country="US") + + +def test_search_without_keys_never_calls_pi(monkeypatch): + monkeypatch.setattr(core, "pi_credentials", lambda: None) + monkeypatch.setattr(core, "search_shows", lambda term, limit, country: [ + {"collectionId": 1, "collectionName": "A", "artistName": "x", + "feedUrl": "https://f/a", "trackCount": 3}]) + monkeypatch.setattr(core, "pi_search_shows", + lambda *a, **k: pytest.fail("PI must not be called without keys")) + assert cli.cmd_search(_search_args()) == 0 + + +def test_search_merges_and_dedupes_pi(monkeypatch): + monkeypatch.setattr(core, "pi_credentials", lambda: ("k", "s")) + monkeypatch.setattr(core, "search_shows", lambda term, limit, country: [ + {"collectionId": 1, "collectionName": "A", "artistName": "x", + "feedUrl": "https://f/a", "trackCount": 3}]) + monkeypatch.setattr(core, "pi_search_shows", lambda term, limit: [ + {"collectionId": 1, "collectionName": "A (PI)", "artistName": "x", + "feedUrl": "https://f/a/", "trackCount": 3}, # dup (trailing slash) + {"collectionId": 2, "collectionName": "B", "artistName": "y", + "feedUrl": "https://f/b", "trackCount": 9}]) # new + seen = [] + monkeypatch.setattr(cli, "_render_search_table", lambda term, rows: seen.extend(rows)) + assert cli.cmd_search(_search_args()) == 0 + assert [r["collectionName"] for r in seen] == ["A", "B"] # iTunes wins the dup + + +def test_search_survives_one_backend_failing(monkeypatch): + monkeypatch.setattr(core, "pi_credentials", lambda: ("k", "s")) + + def _boom(term, limit, country): + raise OSError("itunes down") + monkeypatch.setattr(core, "search_shows", _boom) + monkeypatch.setattr(core, "pi_search_shows", lambda term, limit: [ + {"collectionId": 2, "collectionName": "B", "artistName": "y", + "feedUrl": "https://f/b", "trackCount": 9}]) + assert cli.cmd_search(_search_args()) == 0 # PI results still shown + + monkeypatch.setattr(core, "pi_credentials", lambda: None) + with pytest.raises(OSError): # no keys -> unchanged behavior + cli.cmd_search(_search_args()) diff --git a/tests/test_core.py b/tests/test_core.py index ca6ecb5..e87aacd 100644 --- a/tests/test_core.py +++ b/tests/test_core.py @@ -1,9 +1,117 @@ """Pure-logic tests (no network).""" +import io +from pathlib import Path + import pytest from podpull import core from podpull.core import Episode +FIXTURES = Path(__file__).parent / "fixtures" / "feeds" + +# (fixture, show_title, show_author, n_episodes, first_url, first_date) +FEED_CASES = [ + ("rss2_plain.xml", "Plain RSS2 Show", "Plain Author", 2, + "https://cdn.example.test/ep2.mp3", "2026-07-03"), + ("rss10_rdf.xml", "RDF Show", "RDF Author", 1, + "https://cdn.example.test/rdf1.mp3", "2026-06-30"), + ("atom.xml", "Atom Show", "Atom Author", 1, + "https://cdn.example.test/atom1.mp3", "2026-07-01"), + ("itunes_title_order.xml", "Real Channel Title", "Order Author", 1, + "https://cdn.example.test/order1.mp3", "2026-07-04"), + ("media_content_only.xml", "MRSS Show", "editor@example.test (MRSS Author)", 1, + "https://cdn.example.test/mrss1.mp3", "2026-07-02"), + ("wavpub.xml", "半拿铁 | 商业沉浮录", "潇磊布道", 1, + "https://tk.wavpub.com/track/caffebreve/91.m4a", "2026-07-01"), + ("omny.xml", "馬力歐陪你喝一杯", "鏡好聽", 1, + "https://omny.example.test/ep388.mp3", "2026-06-30"), + ("dirty_entities.xml", "Dirty Show", "Dirty & Sons", 1, + "https://cdn.example.test/dirty1.mp3?a=1&b=2", "2026-06-29"), + ("utf16_bom.xml", "UTF16 Show", "", 1, + "https://cdn.example.test/u16.mp3", "2026-06-28"), + ("xiaoyuzhou.xml", "忽左忽右", "JustPod", 1, + "https://dts-api.xiaoyuzhoufm.com/track/cv4bkgpuglwp/xyz380/media.xyzcdn.net/lvJzoGJDVn.m4a", + "2026-07-02"), + ("ximalaya.xml", "声动早咖啡", "声动活泼", 1, + "https://jt.ximalaya.com//GKwRIW8LWq_ZAX-DKgJcpzTV.m4a?channel=rss&jt=https%3A%2F%2Faod.cos.tx.xmcdn.com%2Fstorages%2Fabc.m4a", + "2026-07-06"), + ("soundon.xml", "股癌", "謝孟恭", 1, + "https://rss.soundon.fm/rssf/954689a5/ep560/rssFileVip.mp3?times=1751700000", "2026-07-05"), + ("firstory.xml", "百靈果 News", "百靈果 News", 1, + "https://m.cdn.firstory.me/track/cmjaz594i/https%3A%2F%2Ffile.cdn.firstory.me%2Fstory%2Fep500.mp3", + "2026-07-04"), + ("lizhi.xml", "大内密谈", "大内密谈", 1, + "http://cdn.lizhi.fm/audio/2026/07/05/1200.mp3", "2026-07-05"), + ("typlog.xml", "疯投圈", "黄海、Rio", 1, + "https://rio.xyzcdn.net/crazy-capital/ep76.m4a", "2026-07-03"), + ("fireside.xml", "科技早知道", "声动活泼", 1, + "https://aphid.fireside.fm/d/1437767933/s8e20.mp3", "2026-07-02"), + ("anchor.xml", "台灣通勤第一品牌", "台通", 1, + "https://anchor.fm/s/1ea77470/podcast/play/999/https%3A%2F%2Fd3ctxlq1ktw2nl.cloudfront.net%2Fstaging%2Fep600.m4a", + "2026-07-01"), + ("acast.xml", "不明白播客", "袁莉和她的朋友们", 1, + "https://sphinx.acast.com/p/open/s/68004395/e/100/media.mp3", "2026-06-30"), + ("libsyn.xml", "Libsyn Show", "Libsyn Author", 1, + "https://traffic.libsyn.com/secure/example/ep1.mp3?dest-id=1", "2026-06-29"), + ("transistor.xml", "Transistor Show", "Transistor Author", 1, + "https://media.transistor.fm/abc123/ep1.mp3", "2026-06-28"), +] + + +def _parse_fixture(monkeypatch, fname): + raw = (FIXTURES / fname).read_bytes() + monkeypatch.setattr(core, "fetch", lambda url, timeout=45: io.BytesIO(raw)) + return core.parse_feed("https://example.test/feed") + + +@pytest.mark.parametrize("fname,title,author,count,first_url,first_date", FEED_CASES) +def test_parse_feed_fixture(monkeypatch, fname, title, author, count, first_url, first_date): + t, a, eps = _parse_fixture(monkeypatch, fname) + assert t == title + assert a == author + assert len(eps) == count + assert eps[0].url == first_url + assert eps[0].date == first_date + if fname == "itunes_title_order.xml": + assert eps[0].title == "Real Episode Title" + + +def test_parse_feed_hopeless_xml_raises(monkeypatch): + monkeypatch.setattr(core, "fetch", lambda url, timeout=45: io.BytesIO(b"")) + with pytest.raises(core.ET.ParseError): + core.parse_feed("https://example.test/feed") + + +def test_sanitize_preserves_cdata_and_unknown_entities(monkeypatch): + raw = ( + b'\n' + b'\n' + b'<![CDATA[AT&T R&D Show]]>\n' + b'Dirty & Sons\n' + b'A &bogus123; B\n' + b'Wed, 01 Jul 2026 08:00:00 GMT\n' + b'\n' + b'' + ) + monkeypatch.setattr(core, "fetch", lambda url, timeout=45: io.BytesIO(raw)) + title, _a, eps = core.parse_feed("https://example.test/feed") + assert title == "AT&T R&D Show" # CDATA content untouched + assert eps[0].title == "A &bogus123; B" # unknown entity preserved, not dropped + + +def test_sanitize_handles_utf16_without_bom(monkeypatch): + xml = ('' + '中文播客 标题' + 'EP 一' + 'Wed, 01 Jul 2026 08:00:00 GMT' + '' + '') + raw = xml.encode("utf-16-le") # deliberately NO BOM + monkeypatch.setattr(core, "fetch", lambda url, timeout=45: io.BytesIO(raw)) + title, _a, eps = core.parse_feed("https://example.test/feed") + assert title == "中文播客 标题" + assert eps[0].title == "EP 一" + def test_classify(): assert core.classify("1532755821") == ("apple_show", "1532755821") @@ -11,6 +119,10 @@ def test_classify(): assert core.classify("https://podcasts.apple.com/us/podcast/x/id123?i=456")[0] == "apple_episode" assert core.classify("https://www.xiaoyuzhoufm.com/episode/abc")[0] == "xyz_episode" assert core.classify("https://feed.firstory.me/rss/user/xyz")[0] == "rss" + assert core.classify("https://www.ximalaya.com/album/51076156") == \ + ("rss", "https://www.ximalaya.com/album/51076156.xml") + assert core.classify("https://www.ximalaya.com/album/51076156.xml") == \ + ("rss", "https://www.ximalaya.com/album/51076156.xml") with pytest.raises(ValueError): core.classify("not a url") @@ -54,6 +166,14 @@ def _eps(): def test_episode_date(): assert _eps()[0].date == "2026-06-27" assert Episode(title="x", pub="garbage", url="u").date == "0000-00-00" + # RFC-822 with non-zero-padded day (Lizhi) — parsedate handles it + assert Episode(title="x", pub="Sun, 5 Jul 2026 21:00:00 +0800", url="u").date == "2026-07-05" + # ISO-8601 (Atom published / dc:date), with and without Z + assert Episode(title="x", pub="2026-07-01T08:30:00Z", url="u").date == "2026-07-01" + assert Episode(title="x", pub="2026-07-01T08:30:00+08:00", url="u").date == "2026-07-01" + # ISO-8601 with colon-less offset (py3.9's fromisoformat rejects it raw) + assert Episode(title="x", pub="2026-07-01T08:30:00+0800", url="u").date == "2026-07-01" + assert Episode(title="x", pub="", url="u").date == "0000-00-00" def test_select_match(): @@ -72,3 +192,135 @@ def test_select_index(): def test_select_none(): assert core.select(_eps()) == [] + + +def _fake_response(body: bytes, ctype: str): + class _Resp(io.BytesIO): + status = 200 + headers = {"Content-Type": ctype} + + def __init__(self): + super().__init__(body) + self.length = len(body) + return _Resp() + + +def test_download_url_rejects_text_response(monkeypatch, tmp_path): + monkeypatch.setattr(core.urllib.request, "urlopen", + lambda req, timeout=None: _fake_response(b"deleted", "text/plain; charset=utf-8")) + dest = tmp_path / "ep.m4a" + with pytest.raises(ValueError, match="not audio"): + core.download_url("https://jt.ximalaya.com//x.m4a?bad=1", str(dest)) + assert not dest.exists() + + +def test_download_url_accepts_audio_and_octet_stream(monkeypatch, tmp_path): + for ctype in ("audio/mpeg", "application/octet-stream", ""): + monkeypatch.setattr(core.urllib.request, "urlopen", + lambda req, timeout=None, c=ctype: _fake_response(b"AUDIO", c)) + dest = tmp_path / f"ok-{ctype.replace('/', '_') or 'none'}.mp3" + assert core.download_url("https://cdn.example.test/a.mp3", str(dest)) == str(dest) + assert dest.read_bytes() == b"AUDIO" + + +def test_fetch_and_download_send_browser_ua(monkeypatch, tmp_path): + seen = [] + + class _Resp(io.BytesIO): + status = 200 + length = 2 + headers = {"Content-Type": "audio/mpeg"} + + def __init__(self): + super().__init__(b"ok") + + def fake_urlopen(req, timeout=None): + seen.append(req.get_header("User-agent")) + return _Resp() + + monkeypatch.setattr(core.urllib.request, "urlopen", fake_urlopen) + core.fetch("https://feeds.soundon.fm/x.xml") + core.download_url("https://cdn.lizhi.fm/a.mp3", str(tmp_path / "a.mp3")) + assert seen == [core.UA, core.UA] + assert all("mozilla" in ua.lower() and "python" not in ua.lower() for ua in seen) + + +def test_pi_credentials(monkeypatch): + monkeypatch.delenv("PODCASTINDEX_API_KEY", raising=False) + monkeypatch.delenv("PODCASTINDEX_API_SECRET", raising=False) + assert core.pi_credentials() is None + monkeypatch.setenv("PODCASTINDEX_API_KEY", "k") + assert core.pi_credentials() is None # secret still missing + monkeypatch.setenv("PODCASTINDEX_API_SECRET", "s") + assert core.pi_credentials() == ("k", "s") + + +def test_pi_headers_deterministic(): + h = core._pi_headers("key", "secret", now=1751900000) + assert h["X-Auth-Key"] == "key" + assert h["X-Auth-Date"] == "1751900000" + import hashlib + assert h["Authorization"] == hashlib.sha1(b"keysecret1751900000").hexdigest() + assert h["User-Agent"] == core.UA + + +def test_pi_search_shows_normalizes(monkeypatch): + monkeypatch.setenv("PODCASTINDEX_API_KEY", "k") + monkeypatch.setenv("PODCASTINDEX_API_SECRET", "s") + monkeypatch.setattr(core, "_pi_get", lambda path, params: { + "feeds": [{"id": 887080, "title": "忽左忽右", "url": "https://feed.xyzfm.space/cv4bkgpuglwp", + "author": "JustPod", "itunesId": 1478791559, "episodeCount": 380}], + }) + rows = core.pi_search_shows("忽左忽右", limit=5) + assert rows == [{"collectionId": 1478791559, "collectionName": "忽左忽右", + "artistName": "JustPod", "feedUrl": "https://feed.xyzfm.space/cv4bkgpuglwp", + "trackCount": 380}] + + +def test_pi_feed_by_itunes_id(monkeypatch): + monkeypatch.setattr(core, "_pi_get", + lambda path, params: {"feed": {"url": "https://feed.example.test/x"}}) + assert core.pi_feed_by_itunes_id("123") == "https://feed.example.test/x" + monkeypatch.setattr(core, "_pi_get", lambda path, params: {"feed": []}) + assert core.pi_feed_by_itunes_id("123") is None + + def _boom(path, params): + raise OSError("network down") + monkeypatch.setattr(core, "_pi_get", _boom) + assert core.pi_feed_by_itunes_id("123") is None # degrades, never raises + + +def test_pi_get_never_touches_network_without_creds(monkeypatch): + monkeypatch.delenv("PODCASTINDEX_API_KEY", raising=False) + monkeypatch.delenv("PODCASTINDEX_API_SECRET", raising=False) + + def _fail(*a, **k): + pytest.fail("network must not be touched without PI credentials") + monkeypatch.setattr(core.urllib.request, "urlopen", _fail) + with pytest.raises(ValueError, match="credentials not set"): + core._pi_get("/search/byterm", {"q": "x"}) + assert core.pi_feed_by_itunes_id("123") is None # degrades, still no network + + +def test_apple_show_to_feed_pi_fallback(monkeypatch): + monkeypatch.setenv("PODCASTINDEX_API_KEY", "k") + monkeypatch.setenv("PODCASTINDEX_API_SECRET", "s") + monkeypatch.setattr(core, "fetch_json", lambda url: {"results": []}) # iTunes empty + monkeypatch.setattr(core, "pi_feed_by_itunes_id", + lambda pid: "https://feed.example.test/fallback") + feed, name, author, pid = core.apple_show_to_feed("1478791559") + assert feed == "https://feed.example.test/fallback" + assert pid == "1478791559" + assert name == "" and author == "" + + +def test_apple_show_to_feed_no_creds_still_raises(monkeypatch): + monkeypatch.delenv("PODCASTINDEX_API_KEY", raising=False) + monkeypatch.delenv("PODCASTINDEX_API_SECRET", raising=False) + monkeypatch.setattr(core, "fetch_json", lambda url: {"results": []}) + calls = [] + monkeypatch.setattr(core, "pi_feed_by_itunes_id", + lambda pid: calls.append(pid)) + with pytest.raises(ValueError): + core.apple_show_to_feed("123") + assert calls == [] # PI never contacted without keys diff --git a/tests/test_network.py b/tests/test_network.py new file mode 100644 index 0000000..3cb559a --- /dev/null +++ b/tests/test_network.py @@ -0,0 +1,34 @@ +"""Live-network integration tests. EXCLUDED from the default suite. + +Run explicitly: pytest -m network +These hit real hosts and will flake if a show migrates feeds — that's the +point: they detect real-world drift the offline fixtures can't. +""" +import pytest + +from podpull import core + +pytestmark = pytest.mark.network + +LIVE_FEEDS = [ + # (label, feed_url) + ("xiaoyuzhou 忽左忽右", "https://feed.xyzfm.space/cv4bkgpuglwp"), + ("ximalaya 声动早咖啡", "https://www.ximalaya.com/album/51076156.xml"), + ("soundon 股癌", "https://feeds.soundon.fm/podcasts/954689a5-3096-43a4-a80b-7810b219cef3.xml"), + ("firstory 百靈果", "https://feed.firstory.me/rss/user/cmjaz594i0000hdvpdpnd4fw8"), +] + + +@pytest.mark.parametrize("label,feed", LIVE_FEEDS, ids=[f[0] for f in LIVE_FEEDS]) +def test_live_feed_parses(label, feed): + title, _author, eps = core.parse_feed(feed) + assert title, f"{label}: empty show title" + assert eps, f"{label}: no episodes parsed" + first = eps[0] + assert first.url.startswith("http"), f"{label}: bad enclosure {first.url!r}" + assert first.date != "0000-00-00", f"{label}: unparseable pubDate {first.pub!r}" + + +def test_live_itunes_resolution(): + feed, name, _author, pid = core.apple_show_to_feed("1493503146") # 忽左忽右 + assert feed.startswith("http") and pid == "1493503146" and name