ingest: pure `classify_url` — tell a git repo URL from a hosted docs page from an unsupported input (first slice of #67)

[alphacrack]

User story

As a contributor picking up the docs-site ingestion epic, I want a single pure function that says what a given URL is — a git repository, a hosted docs page, or something we cannot ingest — so that #67 can build fetching on top of a decided, tested vocabulary instead of inventing URL rules inline while also writing HTTP code.

What

One pure, side-effect-free classifier plus table-driven tests. Nothing else.

1. A verdict type. A Literal alias next to the existing ones in src/readme2demo/types.py, suggested:

SourceKind = Literal["git", "docs", "unsupported"]

2. A classifier in src/readme2demo/ingest.py, suggested shape:

def classify_url(url: str) -> UrlVerdict: ...

returning a small immutable result (pydantic v2 BaseModel in types.py, or a frozen dataclass — implementer's call, but be consistent with the repo's pydantic-v2 convention) carrying:

• kind: SourceKind
• repo_url: str | None — for kind == "git", the cloneable repository root. For a plain https://github.com/owner/repo this is the input (normalized: trailing slash dropped). For a GitHub deep link (/tree/..., /blob/...) this is the root the deep link points into. None for docs / unsupported.
• reason: str — a short human-readable explanation, suitable for a future error message ("plain http is not supported — use https", "no owner/repo path segment", …).

3. Classification rules (deliberately conservative — this slice sets policy, later slices consume it):

Input	kind
https://github.com/owner/repo, with or without trailing / or .git	git
https://gitlab.com/group/subgroup/repo (subgroups stay legal)	git
https://github.com/owner/repo/tree/main/examples	git, repo_url = https://github.com/owner/repo
https://github.com/owner/repo/blob/main/README.md	git, repo_url = repo root
https://github.com/owner, https://github.com/	unsupported
https://docs.example.com/quickstart, https://example.com	docs
https://owner.github.io/project/ (Pages site, not a repo)	docs
http://github.com/owner/repo (plain http)	unsupported
git@github.com:owner/repo.git, ssh://git@github.com/owner/repo.git	unsupported
/local/path/to/repo, file:///tmp/repo, ""	unsupported
https://evil.example/https://github.com/owner/repo	must not be git

That last row is the one to be careful about: today clone_repo rejects it and tests/test_ingest.py:51 pins that. Under a naive "any other https is docs" rule it lands in docs, which is harmless in this slice (nothing fetches anything), but the test must assert kind != "git" explicitly so a future refactor cannot quietly promote it.

Out of scope — do not touch any of this:

• Any change to clone_repo, ingest(), cli.py, or the orchestrator. The function is intentionally unused by the pipeline in this slice — Docs-site / URL ingestion: point readme2demo at a hosted docs page, not just a repo #67 wires it up. A PR that also changes ingestion behavior will be asked to split.
• Fetching, HTTP, crawling, markdown extraction, JS rendering — all Docs-site / URL ingestion: point readme2demo at a hosted docs page, not just a repo #67.
• The targeted deep-link error message — that is ingest: GitHub deep links (…/tree/main/x) pass URL validation, then die with a raw git error #109 (see below).
• Local-path acceptance — Local path ingestion: readme2demo run ./my-repo #74 (roadmap:later) may later reclassify /local/path as supported; here it is unsupported, and a comment saying so is welcome.
• Provenance / manifest fields for non-git sources — Docs-site / URL ingestion: point readme2demo at a hosted docs page, not just a repo #67 item 4.
• Adding new allowed git hosts (bitbucket, codeberg, self-hosted) — keep the existing github/gitlab allowlist; widening it is its own conversation.

Why

#67 ("Docs-site / URL ingestion", roadmap:next, from ROADMAP.md:21 "point it at a hosted docs page, not just a repo") opens with exactly this step: "detect non-git URLs in ingest.py (today it validates github.com/gitlab.com patterns and rejects the rest)". That detection is pure string logic with a large edge-case surface; the rest of #67 is network I/O, content extraction, and provenance plumbing. Bundling them means the URL rules get written hastily inside a much larger PR. Carving the classifier out makes it a genuine good first issue — pure Python, no Docker, no API key — and gives #67 a settled vocabulary to build on. This mirrors how #173 was carved out of #114: a pure helper module lands first, consumers follow.

Relationship to #109 — read this carefully, and do not close #109. #109 is the bug where _URL_RE accepts a pasted browser deep link like https://github.com/owner/repo/tree/main/examples, which then dies with a raw git clone error. This classifier would help #109 by giving it the reusable piece it needs — the deep-link recognition and root extraction — so that whoever fixes #109 only has to decide the policy (strip-and-note vs. targeted IngestError) and wire it into clone_repo. But this issue does not fix #109: with no call site changed, a deep link still reaches git clone and still fails exactly as it does today. Cross-reference, don't claim the fix.

Grounding implications: none in this slice, and it must stay that way. The classifier is pure, has no call site, and produces no content — it cannot reach tutorial.md, step_by_step.md, commands.sh, or demo.tape. Worth stating for whoever picks up #67 next: a docs page classified docs is input, never evidence. The fresh-container replay in verify.py remains the only source of "verified", and no classifier verdict may ever be used to mark anything verified.

Pointers

All verified by reading the current tree (main @ 64a3cbd):

• src/readme2demo/ingest.py:27-30 — the whole of today's URL policy:

  _URL_RE = re.compile(
      r"^https://(?:github|gitlab)\.com/"  # allowed hosts only (MVP)
      r"[A-Za-z0-9_.-]+(?:/[A-Za-z0-9_.-]+)+/?$"  # owner/repo (+ gitlab subgroups)
  )

  Note the (?:/…)+ quantifier: one or more extra segments. That is why github.com/owner-only is rejected (needs at least two segments) and why github.com/owner/repo/tree/main/examples is accepted — ingest: GitHub deep links (…/tree/main/x) pass URL validation, then die with a raw git error #109's claim checks out. The permissiveness is deliberate, for GitLab subgroups.
• src/readme2demo/ingest.py:54-65 — clone_repo's validation. It is a bare if not _URL_RE.match(repo_url): raise IngestError(...) with a single generic message naming the two expected forms; there is no host-specific handling anywhere. Everything after line 66 is subprocess. Your function must not go near it.
• src/readme2demo/ingest.py:79-82 — the raw git clone failed ({returncode}): {stderr} passthrough that a deep link currently hits (context for ingest: GitHub deep links (…/tree/main/x) pass URL validation, then die with a raw git error #109 only).
• src/readme2demo/types.py:31-32 — prior art for the verdict alias: Phase = Literal[...] / Outcome = Literal[...], plain module-level aliases above the models. Put SourceKind alongside them.
• tests/test_ingest.py:40-58 — test_clone_repo_rejects_bad_urls is already a @pytest.mark.parametrize table over ten bad URLs, and asserts rejection happens before any git call (assert not (tmp_path / "repo").exists()). Extend the file with your own parametrized table in the same style; leave that test itself untouched, since it pins current clone_repo behavior that this issue does not change.
• tests/test_ingest.py:1 — module docstring: "No network, no docker, no API calls." Your tests inherit that constraint; a classifier test that performs a DNS lookup or an HTTP request is a bug.
• Header comment tests/test_ingest.py:37 — # -- clone_repo URL validation --- is the existing section-comment convention; add a matching one for the classifier.

Acceptance criteria

• SourceKind (or equivalently-named) Literal alias added to src/readme2demo/types.py beside Phase / Outcome
• Result model carrying at least kind, repo_url, reason, with a docstring and full type hints (pydantic v2 if a model)
• classify_url added to src/readme2demo/ingest.py, public, documented, and pure: no subprocess, no filesystem access, no network, no LLM call
• Table-driven @pytest.mark.parametrize tests covering every row of the table above, including empty string
• A test asserting https://evil.example/https://github.com/owner/repo does not classify as git
• A test asserting deep links (/tree/…, /blob/…) yield repo_url equal to the repository root
• git ls-files -m shows no change to clone_repo, ingest(), cli.py, or orchestrator.py; grep -rn "classify_url" src/ finds only the definition — no pipeline call site
• tests/test_ingest.py:40-58 (test_clone_repo_rejects_bad_urls) still passes unmodified
• Full suite green: python -m pytest tests/ -q
• ruff check . and ruff format --check . clean

Notes for contributors

• Pure Python. No Docker, no API key, no network, no LLM credits. Setup is pip install -e ".[dev]" and the whole suite runs with python -m pytest tests/ -q — CONTRIBUTING.md:36 puts it at "under a second", so iterate freely.
• Everything you need is in two files you can read end to end in a few minutes: src/readme2demo/ingest.py (287 lines) and tests/test_ingest.py (387 lines).
• Prerequisite reading: Docs-site / URL ingestion: point readme2demo at a hosted docs page, not just a repo #67 for where this is heading, ingest: GitHub deep links (…/tree/main/x) pass URL validation, then die with a raw git error #109 for the neighbouring bug you are not fixing.
• This repo's rule is that bug fixes carry a test with a """Regression: …""" docstring. This is a new feature, not a bug fix, so plain descriptive test docstrings are correct here — but if you find that a case in your table also reveals a real misclassification risk worth pinning forever, say so in the PR.
• The strongest thing you can bring is edge cases. The table above is a floor, not a ceiling: URLs with query strings or fragments, uppercase hosts, www.github.com, trailing whitespace, unicode. Decide, document the decision in the docstring, and pin it with a row. Discussing a debatable case in the PR is welcome and expected.
• Resist the pull to wire it in. The value of this slice is that it is reviewable in one sitting and cannot break a run.