Hive virtual store: moczarr-backed /gridlook/hive/ (phase 6d of issue #1)

jupyter/README.md

@@ -44,7 +44,8 @@ e.g. on a hub: `https://hub.example.org/user/<you>/gridlook/`.
 | `/gridlook/`                  | the static gridlook SPA                                                                                                                 |
 | `/gridlook/api/health`        | tiny JSON probe (`{"extension": "gridlook-jupyter", ...}`)                                                                              |
 | `/gridlook/s3/<bucket>/<key>` | streaming S3 proxy — GET/HEAD only, `Range` pass-through (206), no LIST                                                                 |
-| `/gridlook/hive/…`            | **reserved** for the phase-6 moczarr virtual-store endpoint (an `open_hive()` AOI served as one flat zarr store) — not implemented here |
+| `/gridlook/hive/open`         | open (or LRU-refresh) a **morton-hive virtual-store view** via moczarr — see below                                                      |
+| `/gridlook/hive/<view>/<key>` | serve one zarr object (metadata / whole chunk) of an open view                                                                          |
 
 ## Configuration
 
@@ -58,6 +59,12 @@ Via traitlets (`jupyter_server_config.py`, or `--GridlookProxy.…` on the comma
 c.GridlookProxy.allowed_buckets = ["my-zagg-outputs"]
 c.GridlookProxy.region = "us-west-2"          # optional; ambient AWS config otherwise
 c.GridlookProxy.static_dir = "/path/to/dist"  # optional; dev override for the SPA files
+
+# /gridlook/hive/ knobs (phase 6d; defaults shown)
+c.GridlookProxy.hive_max_views = 8            # LRU bound on materialized views
+c.GridlookProxy.hive_max_cells = 500_000      # per-view cell bound; 413 beyond
+c.GridlookProxy.hive_max_concurrent_builds = 2  # concurrent materializations; over-limit opens queue
+c.GridlookProxy.local_hive_store_roots = []   # allowed roots for local-path stores (dev only)
 ```
 
 Or environment variables (used only when the trait is not configured):
@@ -72,6 +79,58 @@ the standard hub setup. The proxy streams responses chunk-by-chunk and never buf
 objects; there are no presigned URLs, so nothing credential-shaped is ever exposed to the
 browser.
 
+## Morton-hive virtual store (`/gridlook/hive/`)
+
+Phase 6d of the viewer plan: a zagg **morton-hive** store is many leaf zarrs, but gridlook
+expects one zarr source — and post-englacial/zagg#314 stores are **morton-only**, so gridlook's
+existing HEALPix path (which consumes NESTED `cell_ids`) cannot read a leaf directly. The hive
+endpoint closes both gaps hub-side with [moczarr](https://github.com/espg/moczarr):
+`open_hive()` selects a product/AOI/window, **fabricates the exact NESTED `cell_ids`
+coordinate**, and the extension serves the result as **one flat zarr v3 store** the browser's
+unmodified zarrita reads.
+
+```
+GET /gridlook/hive/open?store=<url>[&product=<name>][&aoi=<decimal,csv>][&window=<label>]
+  -> {"view": "<id>", "url": ".../gridlook/hive/<id>", "cells": N, "cell_order": K, "cached": false}
+GET /gridlook/hive/<view>/<zarr-key>      # zarr.json documents and whole chunks
+```
+
+- `store` — hive store root: `s3://bucket/prefix` (bucket must be allowlisted, same posture as
+  the S3 proxy) or a local path whose realpath is contained in one of
+  `GridlookProxy.local_hive_store_roots` (empty by default — local stores disabled; dev/tests).
+- `product` — named product root under a multi-product store (zagg D19).
+- `aoi` — comma-separated morton decimals (mixed orders fine); omit for the whole store.
+- `window` — window label on a time-windowed (`morton-hive/2`) store.
+
+Views are **materialized on open** into an in-memory zarr store and held in a per-server LRU
+cache: `GridlookProxy.hive_max_views` (default 8) bounds the cache, and
+`GridlookProxy.hive_max_cells` (default 500 000) bounds a single view — an over-size selection
+gets a 413 telling you to narrow the AOI. Re-opening the same selection refreshes the existing
+view; evicted view URLs 404 until re-opened. Materialize-on-open keeps the serve path free of
+zarr chunk/codec arithmetic; a streaming/virtual encoding is the future optimization if views
+outgrow memory.
+
+The served attrs carry a **pre-6c compatibility shim**: gridlook's grid detector currently
+accepts only `dggs.name == "healpix"`, so the view advertises the healpix-shaped block with
+`coordinate: "cell_ids"` + `refinement_level` — the fabricated NESTED ids are plain HEALPix
+NESTED indices, exactly what the existing sparse-HEALPix render path consumes. When phase 6c
+teaches the detector the morton convention entry, the shim goes away.
+
+### CryoCloud MVP recipe: render a hive store
+
+```bash
+# 1. install the extension wheel plus the hive extra (moczarr; git source until PyPI)
+pip install gridlook_jupyter-<version>-py3-none-any.whl "gridlook-jupyter[hive]"
+# 2. allowlist the bucket holding the store (or use env GRIDLOOK_ALLOWED_BUCKETS)
+#    e.g. in jupyter_server_config.py: c.GridlookProxy.allowed_buckets = ["my-zagg-outputs"]
+#    ...then restart your server so the extension picks it up.
+```
+
+3. Open a view (in a notebook, `requests.get(...)`, or just a browser tab):
+   `https://<hub>/user/<you>/gridlook/hive/open?store=s3://my-zagg-outputs/store-root&aoi=4331422`
+4. Copy the returned `url`, open the app at `https://<hub>/user/<you>/gridlook/`, and paste the
+   view URL as the dataset source. The view renders through the existing HEALPix path.
+
 ## `s3://` inputs in the app
 
 When the SPA is served under `/gridlook/` (it detects this by probing `api/health`), pasting an
@@ -93,3 +152,8 @@ jupyter server --GridlookProxy.static_dir="$(pwd)/../dist" --GridlookProxy.allow
 
 Tests exercise the proxy against an obstore `LocalStore` via the `GridlookProxy.store_factory`
 seam — no real S3 needed.
+
+The hive tests run against moczarr's committed SERC fixture and need moczarr importable from a
+**repo checkout** (so the fixture sits next to the package): `uv pip install -e
+/path/to/moczarr` (or set `GRIDLOOK_MOCZARR_TESTDATA` to a checkout's `tests/data`). Without
+moczarr the hive suite skips.

jupyter/gridlook_jupyter/config.py

@@ -3,7 +3,7 @@
 import os
 from pathlib import Path
 
-from traitlets import List, Unicode, default
+from traitlets import Int, List, Unicode, default
 from traitlets.config import Configurable
 
 #: Bounded per-process cache of bucket -> store (buckets come from the allowlist,
@@ -47,6 +47,54 @@ class GridlookProxy(Configurable):
         ),
     ).tag(config=True)
 
+    local_hive_store_roots = List(
+        Unicode(),
+        help=(
+            "Root directories under which /gridlook/hive/open may open local "
+            "filesystem hive stores (development/tests). Empty (the default) "
+            "disables local stores entirely; a requested path is accepted only "
+            "when its realpath is contained in one of these (also realpath'd) "
+            "roots, so 'dev' can't become whole-filesystem reach. s3:// stores "
+            "are always gated by allowed_buckets instead. Env fallback: "
+            "GRIDLOOK_LOCAL_HIVE_STORE_ROOTS (comma-separated)."
+        ),
+    ).tag(config=True)
+
+    hive_max_views = Int(
+        8,
+        help=(
+            "Maximum materialized hive views held per server process; opening "
+            "beyond it evicts the least-recently-used view (its /gridlook/hive/ "
+            "URLs then 404 until re-opened)."
+        ),
+    ).tag(config=True)
+
+    hive_max_cells = Int(
+        500_000,
+        help=(
+            "Maximum cells a single hive view may materialize; /gridlook/hive/open "
+            "returns 413 beyond it. Views are held in memory (~100 B/cell at "
+            "typical zagg variable counts), so hive_max_views * hive_max_cells "
+            "bounds the RESIDENT cache footprint."
+        ),
+    ).tag(config=True)
+
+    hive_max_concurrent_builds = Int(
+        2,
+        help=(
+            "Maximum hive views materialized concurrently. Each in-flight open "
+            "holds its xarray dataset plus a MemoryStore copy transiently, ON TOP "
+            "of the resident cache, so a burst of opens can exceed the resident "
+            "bound; this caps that transient peak. Over-limit opens QUEUE (they "
+            "wait, they are not rejected)."
+        ),
+    ).tag(config=True)
+
+    @default("local_hive_store_roots")
+    def _default_local_hive_store_roots(self):
+        env = os.environ.get("GRIDLOOK_LOCAL_HIVE_STORE_ROOTS", "")
+        return [r.strip() for r in env.split(",") if r.strip()]
+
     @default("allowed_buckets")
     def _default_allowed_buckets(self):
         env = os.environ.get("GRIDLOOK_ALLOWED_BUCKETS", "")

jupyter/gridlook_jupyter/extension.py

@@ -9,6 +9,7 @@
 
 from .config import GridlookProxy
 from .handlers import HealthHandler, S3ProxyHandler
+from .hive import HiveOpenHandler, HiveViewCache, HiveViewHandler
 
 
 def load_extension(serverapp):
@@ -26,9 +27,11 @@ def load_extension(serverapp):
     handlers = [
         (escaped + r"/api/health", HealthHandler),
         (escaped + r"/s3/([^/]+)/(.+)", S3ProxyHandler),
-        # /gridlook/hive/... is RESERVED: phase 6 mounts the moczarr virtual-store
-        # endpoint there (an open_hive() AOI served as one flat zarr store).
-        # Do not claim that namespace with other routes.
+        # Phase 6d: the moczarr virtual store on the namespace phase 4 reserved —
+        # an open_hive() product/AOI/window selection served as ONE flat zarr
+        # store (fabricated NESTED cell_ids included; see hive.py).
+        (escaped + r"/hive/open", HiveOpenHandler),
+        (escaped + r"/hive/([0-9a-f]{16})/(.+)", HiveViewHandler),
         (escaped + r"$", web.RedirectHandler, {"url": base + "/"}),
         (
             escaped + r"/(.*)",
@@ -37,6 +40,7 @@ def load_extension(serverapp):
         ),
     ]
     serverapp.web_app.settings["gridlook_proxy"] = proxy
+    serverapp.web_app.settings["gridlook_hive_views"] = HiveViewCache(proxy)
     serverapp.web_app.add_handlers(".*$", handlers)
     serverapp.log.info(
         "gridlook-jupyter loaded: app at %s/, proxy %s",

jupyter/gridlook_jupyter/handlers.py

@@ -77,10 +77,8 @@ async def get(self):
         )
 
 
-class S3ProxyHandler(JupyterHandler):
-    """Streaming byte-range proxy: GET/HEAD only, no LIST, hub-side credentials only."""
-
-    SUPPORTED_METHODS = ("GET", "HEAD")
+class PlainTextErrorMixin:
+    """Surface HTTPError log messages as plain-text bodies (pointed, curl-readable)."""
 
     def write_error(self, status_code, **kwargs):
         exc = kwargs.get("exc_info", (None, None, None))[1]
@@ -90,6 +88,12 @@ def write_error(self, status_code, **kwargs):
         self.set_header("Content-Type", "text/plain; charset=utf-8")
         self.finish(message or f"error {status_code}")
 
+
+class S3ProxyHandler(PlainTextErrorMixin, JupyterHandler):
+    """Streaming byte-range proxy: GET/HEAD only, no LIST, hub-side credentials only."""
+
+    SUPPORTED_METHODS = ("GET", "HEAD")
+
     def _store_for(self, bucket: str):
         proxy: GridlookProxy = self.settings["gridlook_proxy"]
         if not proxy.enabled: