Skip to content

Master implementation plan: Phase 0 → 8 (full v1 + AniDB) #1

Description

@narugo1992

This issue is the master tracking document for the full implementation of animedex from the current scaffold to a v1.0 release that covers every phase in plans/04-roadmap-and-mvp.md, including the heavy AniDB track. Sub-issues and PRs should reference this issue as Refs #N. The checklists below are the source of truth for "what still needs to land".

The design rationale is fixed in plans/01..05 and AGENTS.md; this issue does not re-litigate it. It only enumerates the executable checklist.


0. Tech stack snapshot

0.1 Locked-in (do not re-discuss)

Decision Choice Source
Project shape read-only multi-source CLI for anime/manga plan 03
Command flavor gh-flavored (backend verb + api passthrough) plan 03
Policy carrier docstring (Backend: / Rate limit: / Agent Guidance triplet) plan 02
Source attribution mandatory _source / [src: ...] annotation everywhere plan 03 §5
CLI framework Click 8 requirements.txt
Sync HTTP requests 2.28+ requirements.txt
Packaging PyInstaller 6 (spec-based) + sdist + wheel landed
Style ruff (120-col) + flake8 + reST docstrings landed
Docs Sphinx + RTD theme + auto_rst pipeline landed
Token storage OS keyring plan 02 P1
Repo language English-only repo content AGENTS §1
Identity model per-repo git config + per-invocation GH_TOKEN AGENTS §2-3

0.2 Decided this round

Decision Choice Side effect
Models layer pydantic v2 BaseModel adds runtime dep; PyInstaller bundle grows ~4-5 MB; cache uses model_dump_json() directly
Python floor 3.9+ (drop 3.7 and 3.8) update setup.py, python_requires, classifiers, CI matrix

0.3 Still open (default answers given; resolve before they bite)

#3 corrects several plan-01 facts that have drifted by 2026-05-07 (kitsu.app/.io parity, shikimori.one still serving, MangaDex Via no longer enforced, Trace.moe anonymous quota dropped from 1000 to 100/month, no rate-limit headers exposed by Jikan/MangaDex/Danbooru). Read #3's drift register before touching plan 01.

  • MCP registration timing: eager (import-time side effect) vs lazy. Default applied (PR Phase 0: substrate (models / transport / cache / auth / render / policy / mcp) #2): lazy via explicit register_animedex_tools(server) entry point — animedex/mcp/register.py has no import-time side effects, confirmed by its selftest().
  • animedex.aio async layer launch: Default applied: deferred until after v1.0. All shipped code (Phase 0+1+2 + jq wheel) is sync; no animedex.aio module exists in the tree.
  • MyAnimeList official API v2 backend fate: read-only constraint makes Jikan a full replacement. Default applied: explicitly dropped — the package ships no MAL backend, Jikan covers every MAL anonymous read.
  • AnimeChan filtered quote access: Default updated by PR Add Ghibli and quote backends #14: the high-level AnimeChan v1 surface exposes random, random-by-anime, random-by-character, quotes-by-anime, quotes-by-character, and anime; the older premium-only flag framing is no longer the project shape.

1. Roadmap (bird's-eye)

Phase 0   substrate            4-5 d   ─── critical path; everything depends on it
Phase 1   api passthrough      2 d         ─── 8 backends, validates substrate
Phase 2   MVP commands         2-3 d           ─── anilist/jikan/trace/nekos
Phase 3   mid-tier read        3 d                 ─── kitsu/mangadex(no reader)/danbooru/waifu
Phase 4   cal/news/trivia      2 d                       ─── shikimori/ann/ghibli/quote
Phase 5   aggregate            2 d                             ─── search/show/crossref/season/schedule
Phase 6   mangadex reader      1.5 d                                 ─── pages subcommand

Phase 7   AniDB heavy track    5-8 d   ─── independent track; can run in parallel with 1-6
Phase 8   polish               2 d         ─── completion/alias/--web/MCP server/docs build

Release cadence:

  • 0.1.x = MVP (Phase 0+1+2 done)
  • 0.2.x = mid-tier + aggregate (Phase 3+4+5)
  • 0.3.x = mangadex reader (Phase 6)
  • 0.x.x+anidb build metadata = AniDB landing (Phase 7)
  • 1.0.0 = docstring lint green + Linux/macOS smoke matrix green + JSON schema stable

2. Phase 0 — Substrate (4-5 d) [the highest-leverage phase]

Each sub-piece is written once and reused by every subsequent phase. Cutting corners here costs us in every later phase.

2.1 Stack-level prerequisites

  • Bump python_requires to >=3.9 in setup.py; drop 3.7/3.8 classifiers.
  • Add pydantic>=2,<3 to requirements.txt.
  • Add platformdirs>=3 to requirements.txt (cache + token-store paths).
  • Add keyring>=24 to requirements.txt.
  • Update CI matrix to remove 3.7/3.8 and add 3.9..3.13.

2.2 animedex/models/ (pydantic v2 BaseModel layer)

  • models/common.py: SourceTag(backend, fetched_at, cached, rate_limited), Pagination, RateLimit, ApiError.
  • models/anime.py: AnimeTitle, AnimeRating, AnimeStreamingLink, Anime(source: SourceTag, ids: dict[str, str], ...).
  • models/manga.py: Manga, Chapter, AtHomeServer.
  • models/character.py: Character, Staff, Studio.
  • Shared AnimedexModel(BaseModel) base with populate_by_name=True, extra='ignore', frozen=True.
  • selftest() instantiates each model from a fixture and verifies schema.

2.3 animedex/transport/ (HTTP client + rate limiting + UA)

  • transport/http.py: HttpClient wrapping requests.Session per backend.
  • transport/ratelimit.py: in-memory token-bucket; one bucket per backend.
  • transport/useragent.py: single source of UA string with version + contact email.
  • transport/read_only.py: advisory known-read classifier for cache eligibility; raw passthrough no longer rejects caller-supplied methods. (Updated by PR Add raw API universal flags #17.)
  • selftest() exercises the advisory known-read classifier + UA injection with mocks. (Updated by PR Add raw API universal flags #17.)

2.4 animedex/cache/ (SQLite TTL cache)

  • cache/sqlite.py: keyed by (backend, request_signature); rows store (response_bytes, fetched_at, ttl).
  • Default TTL table: metadata 72 h, list 24 h, schedule 1 h, offline dump 30 d.
  • Path resolution via platformdirs.user_cache_dir("animedex").
  • pydantic models serialize via model_dump_json(); loads via Model.model_validate_json().
  • selftest() opens a tmp DB, writes one row, reads it back, expires it.

2.5 animedex/auth/ (OS keyring token store)

  • auth/keyring_store.py: real keyring backend, namespaced per backend.
  • auth/inmemory_store.py: in-memory backend for tests and headless CI.
  • Config.token_store plug point (interface defined in 2.8).
  • CLI commands: animedex auth status / auth login <backend> / auth logout / auth token.
  • selftest() does NOT touch the real keyring; only verifies the in-memory backend round-trip.

2.6 animedex/render/ (source-attributed renderer)

  • render/tty.py: human-friendly table renderer with [src: ...] suffixes.
  • render/json_renderer.py: full schema with per-field _source.
  • render/field_projection.py: --json field1,field2,... projection.
  • render/jq_filter.py: --jq <expr> via jq subprocess; gracefully degrades when jq is absent.
  • render/template.py: --template <jinja2> (optional).
  • render/attrib_off.py: --source-attribution=off (JSON only; TTY always shows source).
  • TTY-vs-pipe auto switch via sys.stdout.isatty().
  • selftest() round-trips a fixture Anime instance through both default renderers.

2.7 animedex/policy/ (docstring lint + --agent-guide)

  • policy/lint.py: walks animedex/backends/** and animedex.api.call, asserts every @cli.command(...) and @mcp.tool(...) docstring contains Backend:, Rate limit:, --- LLM Agent Guidance ---, --- End ---.
  • Wire into make lint.
  • animedex --agent-guide CLI: concatenates all Agent Guidance blocks across registered commands.
  • selftest() runs lint with no backends (Phase 0 state); confirms the runner returns 0.

2.8 animedex/config/profile.py (programmatic flag stack)

  • Config(BaseModel) with one field per CLI flag (cache_ttl_seconds, no_cache, rate, source_attribution, user_agent, timeout_seconds, token_store).
  • Config() (no args) reproduces unflagged CLI behavior.
  • Each public API function accepts config: Config | None = None.
  • selftest() instantiates default and fully-populated Config.

2.9 animedex/mcp/ (MCP server scaffolding, lazy)

  • mcp/register.py: register_animedex_tools(server) entry point, no import-time side effects.
  • mcp/tool_decorator.py: @mcp.tool lightweight wrapper that turns the docstring's Agent Guidance block into the MCP tool description.
  • MCP deps go in requirements-mcp.txt extras, not the runtime baseline.
  • animedex mcp serve CLI is stubbed in Phase 0; wired in Phase 8.
  • selftest() imports register.py and confirms it has no side effects.

2.10 Phase 0 gating criteria

  • make format && make test && make build && make test_cli all green.
  • python -m animedex.policy.lint animedex/ passes (vacuously, no backends yet).
  • animedex --agent-guide outputs an empty table.
  • animedex selftest reports every Phase 0 module under "smoke" not "import only".
  • A throwaway _example backend exercises the full substrate end-to-end (committed and removed in the same PR or kept as a template).

2.11 Phase 0 risks (carry forward)

Risk Likelihood Mitigation
pydantic-core fails to import in PyInstaller frozen binary medium early make build && make test_cli; list pydantic_core._pydantic_core in HIDDEN_IMPORTS
keyring has no backend on headless Linux CI high InMemoryStore + monkeypatch in tests; CI never touches the real keyring
--jq subprocess fails on Windows (no jq) high feature-detect; emit a friendly "jq not found" message; do not fail CI
pydantic frozen=True conflicts with backend-side scratch construction medium keep mutable scratch step before final model_validate

3. Phase 1 — animedex api passthrough (2 d)

See #3 for the Phase 1 runbook: live-validated endpoint catalog, drift register (kitsu.app + shikimori.one + Trace.moe quota + MangaDex Via), default UA convention (animedex/<version>), UA enforcement matrix, and the auth-fetch recipes that are deferred from this phase. Phase 1 is anonymous-only; no auth flows are wired here. The checklists below stay green-tickable in this issue, but their detail lives in #3.

3.1 Implementation order

(AniDB is excluded from Phase 1 by design; it lands in Phase 7.)

3.2 Universal flags (per plan 03 §7)

3.3 Raw passthrough method and cache semantics

The old local read-only firewall has been retired. Raw animedex api forwards caller-supplied methods and paths verbatim; animedex informs through docs and stderr warnings where needed, but does not reject raw method/path choices on the user's behalf. The remaining classifier in animedex/transport/read_only.py is advisory: it marks known-read requests as cache-eligible and makes mutating-looking or unknown raw requests bypass cache so they reach the upstream every time.

Default read shapes still matter for convenience behavior:

  • AniList: defaults to GraphQL POST / only when the user did not explicitly pass -X/--method.

  • Shikimori GraphQL: defaults to POST /api/graphql only when the user did not explicitly pass -X/--method.

  • Trace.moe: defaults to byte-body POST /search for upload search only when the user did not explicitly pass -X/--method.

  • Jikan / Kitsu / MangaDex / Danbooru / Shikimori REST / ANN / Ghibli / Quote: raw REST passthrough defaults to GET.

  • Implement and unit-test each per-backend advisory known-read/cache-eligibility rule.

  • Remove local raw method/path rejection; explicit POST, PUT, PATCH, DELETE, or any other caller-supplied method is forwarded to the upstream, with cache bypass for requests that are not classified as known reads. (Landed via PR Add raw API universal flags #17.)

3.4 Phase 1 gating

  • All eight animedex api <backend> invocations succeed against real upstreams in a one-off integration check.
  • --paginate works for the original three paginating raw backends and the later Shikimori/Quote paginated surfaces. (Landed via PR Add raw API universal flags #17.)
  • selftest() for each backend passes mock-only.

4. Phase 2 — MVP commands (2-3 d) — closes 0.1.x

4.1 Command tree

4.2 Notes

  • Every command's docstring must satisfy the policy lint.
  • Renderer is consumed verbatim from substrate; do not re-implement.
  • Trace.moe anonymous quota is 100/month; selftests must never hit the live API.

4.3 Phase 2 gating = 0.1.0 release


5. Phase 3 — Mid-tier read backends (3 d)

Notes:

  • The Danbooru tag DSL is fully documented in the docstring; rating:g is not auto-injected (plan 02).
  • MangaDex enforces UA injection and forbids Via header.
  • Waifu.im exposes --include-tag/--exclude-tag/--is-nsfw; each tag class is named in the docstring.

6. Phase 4 — Calendar / news / trivia (2 d)

  • animedex shikimori calendar/search/show/screenshots/videos/characters/staff/similar/related/external-links/topics/studios/genres/manga-search/manga-show/ranobe-search/ranobe-show/club-search/club-show/publishers/people-search/person (UA default-injected by the shared transport). (Landed via PR dev(narugo1992): add shikimori and ann high-level backends #15: high-level REST CLI/Python API with fixtures, rich models, docs, and GIF demo.)
  • animedex ann show/search/reports (XML adapter via xml.etree.ElementTree). (Landed via PR dev(narugo1992): add shikimori and ann high-level backends #15: generic lossless XML adapter, typed ANN models, warning-bearing 200 responses modeled as data, fixtures, docs, and GIF demo.)
  • animedex ghibli films/people/locations/vehicles/species (offline; bundled animedex/data/ghibli.json). (Landed via PR Add Ghibli and quote backends #14: offline snapshot-backed high-level CLI/Python API plus raw live passthrough.)
  • animedex quote random/random-by-anime/random-by-character/quotes-by-anime/quotes-by-character/anime (5 req/h, local SQLite cache, lazy-fill). (Landed via PR Add Ghibli and quote backends #14: AnimeChan v1 anonymous read surface, including paginated quotes-by-anime and quotes-by-character, with cache-before-rate-limit behaviour.)
  • selftest() for the Ghibli backend loads and validates the bundled JSON. (Landed via PR Add Ghibli and quote backends #14; Quote selftest also smokes default SQLite cache path and schema creation.)

7. Phase 5 — Aggregate commands (2 d) — closes 0.2.x

Notes:

  • The aggregate substrate is shared across all five commands: animedex/agg/_fanout.py (concurrent fan-out helper + per-source partial-failure status), animedex/models/aggregate.py (AggregateResult / AggregateSourceStatus / MergedAnime / merge_diagnostics), animedex/entry/aggregate.py (_report_failures + _report_merge_diagnostics stderr inform + _emit / _finish), animedex/utils/timezone.py. The remaining Phase 5 PRs reuse these directly rather than re-implementing.
  • Cross-source merging is the user-visible default on season (and will be on search once the follow-up merge slice lands). Matching is conservative: shared external IDs first, then a deterministic fuzzy comparison using anyascii / jaconv / unidecode for CJK/Hangul recall. The threshold (_MERGE_THRESHOLD = 70) is calibrated against the 2010-2025 adjudicated baseline at test/fixtures/aggregate/season_matrix/expected_matches.json; lowering increases recall, raising increases precision. Re-run tools/merge_eval/evaluate_rule.py after any threshold change.
  • Score merging keeps both upstream values under MergedAnime.source_details and source_payloads; do not average. Source attribution is the contract — the merged row exposes every upstream record under the sources / records map so no upstream-visible field disappears.
  • crossref consults a vendored static map first (animedex/data/crossref.json, snapshot of nattadasu/animeApi); --deep falls back to live AniDB. Until Phase 7 lands, --deep raises a typed error with a pointer to install the AniDB extra. Do not break the command surface.
  • The substrate adds runtime dependencies anyascii, jaconv, unidecode, tzdata, and python-dateutil; each is justified in PR Add calendar aggregate commands #21's commit body. The PyInstaller spec hides the lazy-load data tables for anyascii and unidecode; the animedex selftest runner now smokes both transliterators against representative CJK/kana input.
  • Tag 0.2.0 once all five commands plus the follow-up search-merge slice are green.

8. Phase 6 — MangaDex At-Home reader (1.5 d) — closes 0.3.x

  • animedex mangadex pages <chapter-id> [--save-to <dir>].

Notes:

  • Two-step flow: GET /at-home/server/{chapterId} returns a short-lived base URL; pages are fetched from <base>/data/<hash>/<file>. Do not cache the base URL across chapters.
  • HTTP/2 same-host concurrency cap of 6.
  • File naming: <page-no>.<ext> by default; user can override via flag.
  • Docstring states the DMCA reality and the agent-guidance note ("invoke only when the user explicitly asks to read manga").
  • Tag 0.3.0 once green.

9. Phase 7 — AniDB heavy track (5-8 d, independent)

Justified only because AniDB is the only source for ed2k file fingerprinting and the broadest cross-service ID map (<resources>).

Sub-tasks in dependency order:

  • Persistent rate-limit scheduler — file-backed token bucket at ~/.cache/animedex/anidb-rate-state.json. Window survives process exit. Ship this before any AniDB client code.
  • HTTP API client (gzip + XML), with retry and back-off honouring the scheduler.
  • anime-titles dump parser for offline title lookup.
  • XML→JSON adaptor that preserves source attribution.
  • UDP socket client: AUTH, ENCRYPT, sequence numbers, PING-keepalive, thread-safe in-flight map.
  • ed2k computation utility (4 KiB-block streaming hash; pure Python).
  • AniDB credential storage: keyring namespace separate from any HTTP-only client name.
  • CLI: animedex anidb show/crossref/dump-titles/fingerprint.
  • selftest() (offline) for ed2k correctness against a known-hash fixture; selftest_online() (opt-in) for HTTP liveness only.

Gating:

  • Continuous 30 min run of dump-titles + show + crossref against real AniDB without any ban.
  • Ship 0.x.x+anidb build metadata.

10. Phase 8 — Polish + Auth (3-4 d) — closes 1.0.0

OAuth UX decision (recorded 2026-05-08, see #6 for full rationale):

  • AniList OAuth Client ships with redirect_uri = https://anilist.co/api/v2/oauth/pin. Default flow is PIN (out-of-band): browser → Authorize → AniList shows code → user pastes into CLI. Zero-infrastructure on user side; cross-platform (Windows / macOS / Linux / SSH / containers).
  • client_secret ships with the binary / sdist / wheel. AniList does not support PKCE or device flow; the secret is a public client identifier with bounded leak risk (an attacker with the secret can still only issue tokens for their own account).
  • --auth-mode loopback advanced flag (http://localhost:53682/callback): smoother UX for desktop users, breaks for SSH-remote / containers; explicitly opt-in.
  • animedex auth login anilist — runs the PIN flow, stores token via animedex.auth.keyring_store.KeyringTokenStore.
  • animedex auth logout anilist — revokes via the AniList revoke endpoint, clears keyring entry.
  • animedex auth status — lists which backends have tokens stored.
  • animedex anilist viewer / notification / markdown / ani-chart-user — replace the auth-required stubs from Phase 2 with real implementations that read the token from keyring and pass it through _dispatch.call.
  • Same flow for Kitsu (password grant), Danbooru (api_key), Shikimori (OAuth code), MangaDex (Personal Client) per their per-backend recipes in Phase 1: animedex api passthrough — runbook, endpoint catalog, drift register #3 §6.

Polish (original Phase 8 scope):

  • animedex completion bash | zsh | fish (Click built-in export).
  • animedex alias set / unset / list.
  • animedex extension ... is deferred beyond v1.0; reserve the command name only.
  • --web flag for the five mappings in plan 03 §9.
  • animedex mcp serve (real MCP server entry; uses 2.9 scaffolding).
  • Sphinx docs full build, including the auto-extracted Agents Reference page.
  • pydantic JSON schema export to docs/source/schema/.

Gating = 1.0.0:

  • Three-platform make format && make test && make build && make test_cli all green.
  • python -m animedex.policy.lint green for every command.
  • Docs site live on RTD, Agents Reference page populated.
  • JSON schema documented as stable.

11. Cross-cutting concerns

11.1 Test strategy

  • test/ mirrors animedex/ exactly (one test_*.py per module).
  • Mock HTTP via responses; mock byte streams for Trace.moe.
  • make test INTEGRATION=1 opt-in for live API tests; CI does not run by default.
  • Every backend ships selftest() (offline) and may ship selftest_online() (opt-in).
  • make test_cli runs the PyInstaller artifact via subprocess.

11.2 CI matrix

  • Unit tests on Linux × Python {3.9, 3.13}; macOS × {3.11}; Windows × {3.11} (lighter coverage on the harder platforms).
  • Frozen smoke on Linux + macOS each release.
  • Windows frozen smoke deferred (PyInstaller on Windows is intermittently flaky); tracked as a separate issue.
  • make format-check + python -m animedex.policy.lint as a dedicated job.
  • make docs job that fails on Sphinx warnings.
  • Tag-triggered release job builds sdist/wheel + Linux/macOS frozen binaries and uploads to GitHub Releases + PyPI.

11.3 Documentation

  • docs/source/ is end-user-facing; it does not mention plans/ or "Top rule: human agency". The agency principle binds contributors via AGENTS.md, not library users. (Verified post-PR feat: nekos.best v2 backend + docs front-face overhaul #9: the new tutorials only carry the user-facing read of the principle — no plans/ references.)
  • docs/source/api_doc/ is regenerated via make rst_auto. (Wired in Phase 0; landing page renamed "API Reference" via PR feat: nekos.best v2 backend + docs front-face overhaul #9's auto_rst.py change.)
  • Tutorials: quickstart (4 commands) → advanced (passthrough / jq / output modes / Python library) → agent guide. (Landed via PR feat: nekos.best v2 backend + docs front-face overhaul #9: rewritten quickstart.rst with five progressive examples, tutorials/index.rst directory with per-backend deep-dives + raw_passthrough.rst + output_modes.rst + python_library.rst + agent_guide.rst. Each backend page carries its own demo GIF, References table, detailed examples, and a list-table of every endpoint with :func: cross-refs.)
  • docs/source/agents.rst extracts every --- LLM Agent Guidance --- block automatically at build time (plan 02 §4). (tutorials/agent_guide.rst documents the animedex --agent-guide flag manually; the auto-extraction directive is still TODO.)
  • docs/source/schema/ exposes the pydantic JSON schemas.

11.4 Release process

  • semver per plan 05 §7.
  • Each minor release ships a GitHub Release with sdist/wheel + Linux/macOS frozen binaries.
  • Changelog auto-derived from dev(...): summary commit messages.

12. Risk register (cross-phase)

Phase Risk Likelihood Mitigation
0 pydantic-core fails in PyInstaller medium early frozen smoke; HIDDEN_IMPORTS list
0 jq missing on Windows high graceful degradation; do not fail CI
1 AniList "degraded" 30 req/min stays permanent high conservative limiter; long cache TTL
3 MangaDex DMCA churn changes search surface medium OpenAPI regenerated each release; assert key endpoints
6 At-Home base URLs become shorter-lived medium never cache base; re-resolve per chapter
7 AniDB IP ban during development low (with scheduler) scheduler ships before client; integration tests behind opt-in flag
8 MCP protocol evolves medium use stable schema; keep API decoupled from MCP DTOs

13. First-week day-by-day (Phase 0)

Day Work
D1 AM Update setup.py / requirements.txt / classifiers / CI matrix for Python 3.9+ + pydantic v2; commit.
D1 PM Implement models/common.py + models/anime.py; selftest.
D2 AM Implement transport/http.py + transport/ratelimit.py + transport/useragent.py.
D2 PM Implement transport/read_only.py + unit tests.
D3 AM Implement cache/sqlite.py + unit tests.
D3 PM Implement auth/keyring_store.py + auth/inmemory_store.py.
D4 AM Implement render/tty.py + render/json_renderer.py + field_projection.py.
D4 PM Implement policy/lint.py + --agent-guide command.
D5 AM Implement mcp/register.py lazy stub; add Phase 0 dummy _example backend.
D5 PM Phase 0 gating: full make format && make test && make build && make test_cli; commit + tag phase-0-done.
D6-7 Phase 1: animedex api passthrough across the eight backends.

Subsequent phases will be expanded into day-by-day tracking when the previous phase's PRs are in review.


14. How to use this issue

  • The checklists above are the source of truth for "what's done vs not". Do not delete a checkbox; leave it unchecked or check it as work lands.
  • New scope additions append to the relevant phase; do not silently rewrite the gating criteria.
  • Each phase ends with a tagged release. The tag name is the phase exit criterion in §1.
  • Open questions in §0.3 must be resolved before they block the phase that needs them; otherwise, the listed default applies.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions