fix: auto-download URL-sourced documents on view/crop with notice and progress

README.md

@@ -122,7 +122,7 @@ open it exits immediately, naming all three paths.
 |---|---|
 | `ade parse -d invoice.pdf` | ensure parsed; artifacts land in `~/.ade` |
 | `ade parse --document-url https://…/doc.pdf` | server fetches the URL |
-| `ade parse --document-url https://… --keep-copy` | also store the document bytes locally, so page previews and crops render |
+| `ade parse --document-url https://… --keep-copy` | also store the document bytes locally, while the (often pre-signed) URL still works |
 | `ade parse -d doc.pdf --tier standard --wait 0` | cheap lane, submit-and-return |
 | `ade parse -d doc.pdf --env eu` | run in the EU region (or `export ADE_ENV=eu`) |
 | `ade parse -d doc.pdf --include markdown --json` | carry the markdown in the payload |
@@ -203,12 +203,14 @@ On a terminal, `view` opens the viewer in your browser by default
 (`--no-open` suppresses it); `--json` runs and piped output never
 launch a browser — the artifact path is in the output either way.
 Without a JOB_ITEM_ID, `view` targets the latest viewable job item.
-URL-parsed items have no local document bytes, so page previews can't
-render until you fetch them: `ade view <id> --download` attaches a copy
-of the document to the job item (plain HTTP, no API credits — but note
-pre-signed URLs expire, so `parse --keep-copy` at parse time is the
-reliable way), after which previews and crops render from it with an
-"unverified against the parsed run" caveat.
+URL-parsed items have no local document bytes, so on first `view` or
+`crop` the CLI fetches a copy of the document into the job item
+automatically — announced on stderr with a progress line, plain HTTP,
+no API credits (`--no-download` skips the fetch; previews then stay
+empty). Note pre-signed URLs expire, so `parse --keep-copy` at parse
+time is the reliable way to secure the bytes. Previews and crops render
+from the attached copy with an "unverified against the parsed run"
+caveat.
 
 Every job item gets a **self-contained `view.html`**: page images with
 bounding-box overlays beside the parsed markdown (or extraction JSON),

SKILL.md

@@ -184,8 +184,10 @@ clearing a parse item cascades to the extractions referencing it.
 - **`--markdown` extractions have no page evidence** (there is no parse
   to join against) — evidence degrades to spans-only, and `view`
   renders the markdown pane alone.
-- **URL parses have no local bytes**, so `view`/`crop` cannot render
-  page imagery until you attach a copy: `parse --document-url …
-  --keep-copy` at parse time (reliable — pre-signed URLs expire), or
-  `ade view <id> --download` after the fact. Markdown, elements, and
-  extractions work either way.
+- **URL parses have no local bytes**, so page imagery renders from an
+  attached copy: `parse --document-url … --keep-copy` fetches it at
+  parse time (reliable — pre-signed URLs expire); otherwise the first
+  `view`/`crop` downloads it automatically (`--no-download` skips;
+  the payload records `downloaded`, and on a failed fetch `view`
+  degrades to an empty preview with `download_error` while `crop`
+  errors). Markdown, elements, and extractions work either way.

docs/reference/help.json

@@ -472,7 +472,7 @@
           "metavar": null,
           "required": false,
           "default": null,
-          "help": "--document-url only: also download the document into the job item (plain HTTP, no API credits) so page previews and crops render locally \u2014 fetched now, while the URL (often pre-signed) still works. Without it, URL parses have no local bytes and the viewer explains how to fetch them later (`ade view <id> --download`)."
+          "help": "--document-url only: also download the document into the job item (plain HTTP, no API credits) so page previews and crops render locally \u2014 fetched now, while the URL (often pre-signed) still works. Without it, the first `view`/`crop` fetches the copy instead, by which time a pre-signed URL may have expired."
         },
         {
           "flags": "--include",
@@ -999,11 +999,11 @@
           "help": "Pages to embed images for, 1-indexed, e.g. '1,3-5'."
         },
         {
-          "flags": "--download",
+          "flags": "--download, --no-download",
           "metavar": null,
           "required": false,
           "default": null,
-          "help": "URL-parsed items: fetch the document from its recorded URL into the job item and render page previews from that copy \u2014 the parse itself never gives the CLI the bytes (#169). Plain HTTP, no API credits; the copy is unverified against the parsed run. Also works on an extract item id (fetches into its referenced parse item)."
+          "help": "URL-parsed items: fetch the document from its recorded URL into the job item and render page previews from that copy \u2014 the parse itself never gives the CLI the bytes (#169). This happens automatically when no copy is attached yet (a notice and progress line land on stderr); --no-download skips the fetch and previews stay empty. Explicit --download makes a failed fetch an error instead of a warning. Plain HTTP, no API credits; the copy is unverified against the parsed run. Also works on an extract item id (fetches into its referenced parse item)."
         },
         {
           "flags": "--no-sidebar-sync",
@@ -1061,7 +1061,11 @@
           },
           {
             "key": "downloaded",
-            "what": "with --download: true when this run fetched the URL document into the job item (false: already attached)"
+            "what": "URL items: true when this run fetched the document into the job item (automatic on first view). false when the automatic fetch failed (see download_error) or explicit --download found the copy already attached; absent when nothing needed fetching"
+          },
+          {
+            "key": "download_error",
+            "what": "why the automatic fetch failed, when it did (the viewer still builds, previews empty; else absent)"
           },
           {
             "key": "deep_link",
@@ -1139,6 +1143,13 @@
           "required": false,
           "default": null,
           "help": "Open the crop (or the directory holding them)."
+        },
+        {
+          "flags": "--download, --no-download",
+          "metavar": null,
+          "required": false,
+          "default": null,
+          "help": "URL-parsed items: fetch the document from its recorded URL into the job item and crop from that copy \u2014 the parse itself never gives the CLI the bytes (#169). This happens automatically when no copy is attached yet (a notice and progress line land on stderr); --no-download skips the fetch, and the crop then fails honestly (a crop has no empty-imagery fallback)."
         }
       ],
       "supports_json": true,
@@ -1164,6 +1175,10 @@
           {
             "key": "crops",
             "what": "one record per PNG (element_id, type, page, box, dpi, path, width, height)"
+          },
+          {
+            "key": "downloaded",
+            "what": "URL items: true when this run fetched the document into the parse item before cropping (absent when nothing needed fetching)"
           }
         ],
         "note": "One shape whatever matched: a single --element-id is count 1 with one crops[] record; a filter (--type/--page/--all) matching nothing is count 0 with crops []."
@@ -1331,7 +1346,7 @@
       },
       {
         "path": "  document.<ext>",
-        "what": "URL parses: the attached document copy (`parse --keep-copy` / `view --download`) page previews and crops render from \u2014 unverified against the parsed run"
+        "what": "URL parses: the attached document copy (`parse --keep-copy`, or fetched automatically on first `view`/`crop`) page previews and crops render from \u2014 unverified against the parsed run"
       },
       {
         "path": "  view.html / crops/",

src/ade_cli/attach.py

@@ -2,19 +2,22 @@
 
 ``parse --document-url`` never hands the CLI the document bytes — the
 server fetches the URL — so page previews and crops have nothing local
-to render from. An *attached copy* closes that gap on explicit consent:
-``parse --keep-copy`` downloads the document at parse time (while the
-URL — often pre-signed — still works), and ``view --download`` fetches
-it after the fact. The copy lives inside the job item
-(``jobs/<id>/document.<ext>``) and is recorded on meta.json as auxiliary
-metadata: the recorded ``source`` stays the URL (provenance truth) and
-the item id never moves. The raster layer falls back to the copy via
-``renderable_source``.
-
-The CLI still never fetches a URL without one of these explicit flags —
-and because it never saw the original bytes, an attached copy is not
-verifiable against the parsed generation: renders from it carry the
-``caveat`` note rather than posing as ground truth.
+to render from. An *attached copy* closes that gap: ``parse
+--keep-copy`` downloads the document at parse time (while the URL —
+often pre-signed — still works), and ``view`` / ``crop`` fetch it on
+first use — automatically when the item is URL-sourced and no copy is
+attached yet, announced by a stderr notice + progress line
+(``download_with_notice``), suppressible with ``--no-download``. The
+copy lives inside the job item (``jobs/<id>/document.<ext>``) and is
+recorded on meta.json as auxiliary metadata: the recorded ``source``
+stays the URL (provenance truth) and the item id never moves. The
+raster layer falls back to the copy via ``renderable_source``.
+
+A URL fetch is therefore never *silent* — it is either asked for at
+parse time or said out loud before it starts — and because the CLI
+never saw the original bytes, an attached copy is not verifiable
+against the parsed generation: renders from it carry the ``caveat``
+note rather than posing as ground truth.
 """
 
 from __future__ import annotations
@@ -25,6 +28,7 @@
 from urllib.parse import urlparse
 
 import httpx
+import typer
 
 from .store import JobStore, replace_with_retry
 
@@ -118,13 +122,19 @@ def download(
     *,
     transport: httpx.BaseTransport,
     now: float,
+    progress=None,
 ) -> tuple[str, int]:
     """Fetch the item's URL source and attach the copy; returns
     ``(filename, bytes)``. Raises AttachError with the remediation —
     pre-signed URLs expire, so a late fetch failing is the expected
-    failure mode, not a surprise."""
+    failure mode, not a surprise. ``progress`` (a ``guarantee.Progress``)
+    reports received/total per chunk; a body without Content-Length
+    still shows the moving label, just without a percentage."""
     url = meta.get("source") or ""
     name = copy_name(url)
+    label = f"downloading {name}"
+    if progress is not None:
+        progress.update(label=label)
     target = jobs.item_dir(item_id) / name
     target.parent.mkdir(parents=True, exist_ok=True)
     tmp = target.with_name(f".{name}.tmp-{os.getpid()}")
@@ -156,6 +166,7 @@ def download(
                         f"{MAX_COPY_BYTES}-byte attach cap; parse the local "
                         "file instead (ade parse -d <file>).",
                     )
+                total = int(declared) if declared.isdigit() else 0
                 with tmp.open("wb") as sink:
                     for chunk in response.iter_bytes():
                         received += len(chunk)
@@ -168,6 +179,10 @@ def download(
                             )
                         digest.update(chunk)
                         sink.write(chunk)
+                        if progress is not None and total:
+                            progress.update(
+                                label=label, fraction=received / total
+                            )
     except httpx.HTTPError as error:
         tmp.unlink(missing_ok=True)
         raise AttachError(
@@ -198,3 +213,44 @@ def download(
         )
         jobs.write_json(item_id, "meta.json", current)
     return name, received
+
+
+def download_with_notice(
+    jobs: JobStore,
+    item_id: str,
+    meta: dict,
+    *,
+    ports,
+    as_json: bool,
+) -> tuple[str, int]:
+    """``download`` with the say-it-out-loud surface around it: a stderr
+    notice naming why the network is about to be touched and how to
+    suppress it, then a progress line while the body streams. stderr
+    only, fully silent under ``--json`` (the payload carries the
+    receipt); raises AttachError exactly like ``download``."""
+    # Lazy import mirrors update.py's: guarantee.py imports update, which
+    # would make a top-level import here a needlessly heavy chain.
+    from .guarantee import Progress
+
+    if not as_json:
+        typer.echo(
+            f"note: job item {item_id} was parsed from a URL and no local "
+            "copy is attached — downloading it so page imagery can render "
+            "(skip with --no-download)",
+            err=True,
+        )
+    progress = Progress(
+        ports.clock,
+        "off" if as_json else ("tty" if ports.stderr_is_tty() else "plain"),
+    )
+    try:
+        return download(
+            jobs,
+            item_id,
+            meta,
+            transport=ports.transport,
+            now=ports.clock.now(),
+            progress=progress,
+        )
+    finally:
+        progress.close()

src/ade_cli/crop.py

@@ -161,6 +161,7 @@ def find_element_or_exit(
 
 
 def crop(
+    ctx: typer.Context,
     job_id_token: str | None = typer.Argument(
         None, metavar="[JOB_ITEM_ID]", help="Job item id or unambiguous prefix."
     ),
@@ -194,6 +195,16 @@ def crop(
     open_image: bool = typer.Option(
         False, "--open", help="Open the crop (or the directory holding them)."
     ),
+    download: bool | None = typer.Option(
+        None, "--download/--no-download",
+        help="URL-parsed items: fetch the document from its recorded URL "
+        "into the job item and crop from that copy — the parse itself "
+        "never gives the CLI the bytes (#169). This happens "
+        "automatically when no copy is attached yet (a notice and "
+        "progress line land on stderr); --no-download skips the fetch, "
+        "and the crop then fails honestly (a crop has no empty-imagery "
+        "fallback).",
+    ),
     as_json: bool = JSON_FLAG,
 ) -> None:
     """Crop element regions from the source document into PNGs: one
@@ -234,11 +245,61 @@ def crop(
     selected = elements.select(
         records, element_type=element_type, page=page, element_ids=element_ids
     )
+    parse_meta = jobs.read_json(parse_item_id, "meta.json")
+    if download and not attach.is_url_source(parse_meta):
+        message = (
+            f"Job item {item_id} has no URL source to download: "
+            "--download applies to items parsed from --document-url "
+            "(local parses crop from their file directly)."
+        )
+        exit_with(
+            {
+                "error": "not_a_url_source",
+                "job_item_id": item_id,
+                "message": message,
+            },
+            message,
+            as_json=as_json,
+            code=EXIT_USAGE,
+        )
+    # A URL parse without an attached copy has nothing local to crop
+    # from, so the copy fetches now by default — announced on stderr,
+    # suppressible with --no-download (#169 follow-up, mirroring `view`).
+    # Unlike view there is no degraded render to fall back to (a crop is
+    # never served from missing imagery), so a failed fetch is the
+    # command's failure, exactly as the no-copy state already was.
+    downloaded = None
+    if (
+        download is not False
+        and selected
+        and attach.is_url_source(parse_meta)
+        and attach.attached_file(jobs, parse_item_id, parse_meta) is None
+    ):
+        try:
+            attach.download_with_notice(
+                jobs,
+                parse_item_id,
+                parse_meta or {},
+                ports=ctx.obj,
+                as_json=as_json,
+            )
+        except attach.AttachError as error:
+            exit_with(
+                {
+                    "error": error.kind,
+                    "job_item_id": parse_item_id,
+                    "message": error.message,
+                },
+                error.message,
+                as_json=as_json,
+                code=EXIT_FAILED,
+            )
+        downloaded = True
+        parse_meta = jobs.read_json(parse_item_id, "meta.json")
     # One drift check per invocation, not per element: the batch renders
     # from a single recorded source, and hashing it once is the whole cost.
     # URL items have no drift check (no recorded content hash); a render
     # from their attached copy carries the unverified-bytes caveat instead.
-    parse_meta = jobs.read_json(parse_item_id, "meta.json")
     drift = source_drift_note(parse_meta) or attach.caveat(
         jobs, parse_item_id, parse_meta
     )
@@ -271,8 +332,9 @@ def crop(
             message = error.message
             if "parsed from a URL" in message:
                 message += (
-                    f" Fetch it with `ade view {parse_item_id} --download`, "
-                    "then re-run this crop."
+                    " Re-run without --no-download to fetch the document, "
+                    f"or fetch it with `ade view {parse_item_id} "
+                    "--download` first."
                 )
                 tail = ""
             else:
@@ -312,6 +374,7 @@ def crop(
         "count": len(crops),
         "directory": str(landed),
         "crops": crops,
+        **({"downloaded": downloaded} if downloaded is not None else {}),
         **({"warning": drift} if drift else {}),
     }
     if not batch:

src/ade_cli/help.py

@@ -237,6 +237,9 @@
             ("directory", "where they landed"),
             ("crops", "one record per PNG (element_id, type, "
              "page, box, dpi, path, width, height)"),
+            ("downloaded", "URL items: true when this run fetched the "
+             "document into the parse item before cropping (absent when "
+             "nothing needed fetching)"),
         ],
         "note": "One shape whatever matched: a single --element-id is "
         "count 1 with one crops[] record; a filter (--type/--page/--all) "
@@ -252,8 +255,13 @@
             ("built", "true when this run rebuilt the artifact"),
             ("pages_embedded", "pages inlined; the rest load from sidecars"),
             ("note", "why the render weakened, when it did (else null)"),
-            ("downloaded", "with --download: true when this run fetched "
-             "the URL document into the job item (false: already attached)"),
+            ("downloaded", "URL items: true when this run fetched the "
+             "document into the job item (automatic on first view). false "
+             "when the automatic fetch failed (see download_error) or "
+             "explicit --download found the copy already attached; absent "
+             "when nothing needed fetching"),
+            ("download_error", "why the automatic fetch failed, when it "
+             "did (the viewer still builds, previews empty; else absent)"),
             ("deep_link", "view.html#element=... when --element-id was given"),
             ("history_items", "items in the rebuilt sidebar read model"),
             ("sidebar_sync", "true when sibling viewers build in the background"),
@@ -490,8 +498,9 @@
     {
         "path": "  document.<ext>",
         "what": "URL parses: the attached document copy (`parse "
-        "--keep-copy` / `view --download`) page previews and crops render "
-        "from — unverified against the parsed run",
+        "--keep-copy`, or fetched automatically on first `view`/`crop`) "
+        "page previews and crops render from — unverified against the "
+        "parsed run",
     },
     {
         "path": "  view.html / crops/",