Plan: shard-vector + output-grid globe viewer, packaged for JupyterHub (cryocloud)

[espg]

🤖 from Claude

Plan: shard-vector + output-grid globe viewer on gridlook, packaged for JupyterHub (cryocloud)

The Phase-B viewer from englacial/zagg#74 (globe-view shardmap viewer, one extension also serving the output-zarr-grid view). Decisions ratified on that thread and in-session (2026-07-17):

• Build on gridlook (this fork; upstream d70-t/gridlook, MIT, three.js + Vue 3 + Vite — not deck.gl).
• One shell, two layer families: zagg's shard/granule vector polygons + gridlook's existing zarr datacube globe.
• Landing target: cryocloud (2i2c) as an installable extension. NASA policy: no public buckets — output stores are reachable from cryocloud, not from the public internet, which forces the data path through a hub-side proxy (Phase 4).
• Upstream policy (ratified): offer generic pieces (filled-polygon layer, vector layer kind, jupyter packaging) to d70-t/gridlook per-piece, opportunistically — never gating our timeline.
• Pip wrapper name (ratified): gridlook-jupyter. Proxy shape (ratified): one custom jupyter-server extension serving both the static app and the S3 byte-range proxy (fewer moving parts than jupyter-server-proxy + a second handler).

Architecture facts this plan builds on (verified against this fork at 1.4.1):

• The overlay system is now a first-class, user-manipulable layer stack (store.layerStack, LAYER_KINDS: grid/coastlines/graticules/mask/texture) with add/remove/reorder/visibility/opacity and a LayerPanel.vue UI. TEXTURE layers are already dynamically user-injected (id-keyed, syncTextureLayers in useGridOverlays.ts) — the injection risk flagged on zagg#74 is retired; a vector kind follows this exact pattern.
• The vector gap is unchanged: geojson.ts renders LineString/MultiLineString only (GPU-projected line segments); Polygon/MultiPolygon are rejected; three-conic-polygon-geometry / three-geojson-geometry are declared in package.json but unimported.
• House style for anything on the globe: geometry built in lat/lon, projection applied in-shader via ProjectionHelper (gpuProjectedLines, gridShaders) so layers track gridlook's globe↔flat projection switching. The polygon layer should follow this (triangulate rings, project in shader), not bake globe-space geometry.
• vite.config.ts has base: "./" — the built app serves under any path prefix (a Jupyter URL) without rebuild.
• Input surface: zagg already emits the vector data (shard_outlines / granule_footprints WGS84 GeoJSON FeatureCollections, antimeridian-split). GeoJSON is the only interchange — same surface as zagg's Phase-C ipyleaflet viewer, no second tessellation (the zagg#74 constraint).

Phases

• Phase 1 — filled-polygon rendering primitive. Extend the geojson layer path with Polygon/MultiPolygon support: triangulate rings in lat/lon (three's Earcut / ShapeUtils, or wire the already-declared three-conic-polygon-geometry — implementer decides with a note on projection-switching behavior), GPU-projected fill material following the line-layer house style, plus polygon outlines through the existing line builder. A LAYER_KINDS.VECTOR entry rendered from a store-held FeatureCollection. Acceptance: zagg shard_outlines GeoJSON renders filled+stroked on globe and flat projections; vitest unit tests for the triangulation/geometry builder; lint-ci + typecheck green.
• Phase 2 — user injection + layer UI. Load vector layers from URL or file upload (mirroring texture-layer injection), multiple id-keyed vector layers, LayerPanel entries with fill/stroke/opacity styling controls, hover/picking with a feature-properties readout (shard id, granule id). Acceptance: drag in / point at a zagg GeoJSON with zero code.
• Phase 3 — shardmap presentation. Default styling for shard_outlines / granule_footprints (property-driven choropleth, e.g. granule count per shard), a legend, and URL-param wiring so a hub launcher can deep-link a specific shardmap + catalog view.
• Phase 4 — gridlook-jupyter (in-repo jupyter/ dir). Python package (own pyproject; wheel built from this repo) containing one custom jupyter-server extension: (a) serves the built Vite app on a hub URL, (b) proxies byte-range GETs to S3 hub-side (obstore/boto3; bucket allowlist via config/env) so the browser reads non-public buckets through the hub — the NASA-policy data path. Acceptance: pip install . on a plain JupyterLab, app opens, zarr loads from an allowlisted private bucket through the proxy.
• Phase 5 — cryocloud/2i2c integration + launcher polish. Launcher card (minimal prebuilt Lab extension or jupyter-app-launcher entry) opening the app in a Lab tab; deployment recipe for the 2i2c image (docs); creds/allowlist configuration documented.
• Phase 6 — morton-hive store awareness ("moczarr-lite" TS). v1: open a single hive leaf zarr (vanilla v3 — works with gridlook today). v2: parse morton_hive.json + root coverage.moc ranges, compute leaf paths arithmetically (BigInt for u64 words; box tier only — no zstd bitmap in the browser), and enumerate/open leaves from the UI. Golden vectors ported from moczarr (-5112333, its hive path, the ranges envelope) pin TS↔Python convention parity. Refs Design: sparse coverage convention + cross-resolution read path (MOC domain, hive stores, xdggs extension) englacial/zagg#198 §5/§6.

Phases 1–3 are pure frontend (testable with npm run dev + zagg-emitted GeoJSON, no hub). Phase 4 unblocks cryocloud; 6 can interleave after 3.

Upstream tracking

Candidates to offer d70-t/gridlook once stable, per-piece: the polygon rendering primitive + VECTOR layer kind (phases 1–2, most generic), the jupyter packaging (phase 4, if upstream wants it). zagg-specific presentation (phase 3 styling defaults) and morton-hive awareness (phase 6) likely stay fork-side unless upstream asks.

Open questions (non-blocking for phase 1)

1. Picking tech (phase 2): three.js raycaster against triangulated meshes vs GPU id-picking — decide when hover lands; raycaster is likely fine at shard counts (10²–10⁴ features).
2. Proxy auth model (phase 4): bucket allowlist only, or also per-user scoping via hub auth state? Start allowlist-only.
3. Node floor: repo engines wants node ≥24.16; local dev currently has 23.5 (npm warns, doesn't fail). Align dev env when convenient.

Working conventions: branches claude/<issue#>-<topic>, draft PRs with phased checklists, one commit per phase, adversarial review + fold per phase push — same workflow as moczarr/zagg. Keep the fork synced with upstream main regularly to keep upstream PRs cheap.

[espg]

🤖 from Claude

Amendment: native HEALPix/morton data support — verified, and it changes phase 4/6 shape

Question from espg (in-session): does gridlook plot actual morton/healpix data, and is that rolled into the plan? Verified against this fork's source — yes, natively, including the sparse form zagg writes, with two integration specifics now folded into the plan:

What gridlook already supports (verified)

• HEALPix is a first-class grid type, and detection reads the zarr-dggs convention — gridTypeDetector.ts accepts attrs.dggs.name == "healpix", and Healpix.vue reads refinement_level → nside and coordinate → the cell-id coordinate name (getNside/getCells). That is the exact attrs block zagg leaves already carry (verified on the moczarr side: leaf groups declare name: healpix, refinement_level, coordinate: cell_ids).
• Sparse / limited-area data is the supported case, not a gap: fillLimitedAreaHealpixChunkData maps an explicit cell-id coordinate array to global pixel positions, NaN-fills elsewhere, fetches only relevant slices, and hover works through a cell→index map. zagg's sparse leaves are exactly this shape.
• Morton words never need to reach the browser. zagg's dual-coordinate design (Upstream (xdggs): support MortonIndexDtype as a native DGGS coordinate englacial/zagg#72/Graphical distortions and fragment in flat views d70-t/gridlook#135: NESTED cell_ids as the dggs-convention coordinate, morton as the index coordinate) means gridlook consumes cell_ids. Its Number(cell) conversion is safe for NESTED ids through order ≤ 24 (12·4²⁴ < 2⁵³); production cell order 19 clears that with room. The >2⁵³ morton words stay Python/moczarr-side — this validates the dual-coordinate design.

Plan adjustments

1. Phase 6 v1 gets cheaper: pointing gridlook at a single hive leaf should light up the existing sparse-HEALPix path essentially as-is; v1 is an integration verification (group-path resolution to the data group), not new rendering code.
2. Phase 4 ↔ 6 synergy (new option, likely preferred): gridlook expects ONE zarr source per dataset, but a hive store is many leaves. Instead of merging N sparse leaves browser-side (TS concat), the gridlook-jupyter server extension can use moczarr's open_hive() hub-side to expose an AOI as a single virtual zarr store over HTTP — the browser then sees exactly the limited-area shape gridlook already renders, zarrita unmodified. This reuses the Python reader (one convention implementation, moczarr's golden vectors and all), and it composes with the S3 proxy that phase 4 needs anyway under the no-public-buckets policy. Browser-side moczarr-lite (manifest arithmetic) remains useful for leaf enumeration UI, but the data plane can stay hub-side.
3. Scale/LOD, explicit: the limited-area path does per-chunk linear scans over the cell coordinate and per-cell geometry — right-sized for leaf/AOI scale (10³–10⁵ cells), not order-19 continental extents. The LOD story is Pyramid/overview generation sweep (second pass): overview zarrs, MOC regen, optional interop materialization englacial/zagg#201's pyramid sweep: coarse-order overview zarrs feed this same rendering path densely. Recorded as the standing dependency, not a phase here.

Shard-vector polygons (phases 1–3) are unaffected — that remains the genuinely missing primitive.

[espg]

🤖 from Claude

Amendment 2: the morton-only storage decision (englacial/zagg#262, espg-ratified 2026-07-17) — plan impact

zagg stores will stop carrying the NESTED cell_ids array ("NESTED is fabricated, never stored" — morton becomes the only stored coordinate). Impact on this plan:

Already-coded work: no changes needed. Phase 1 (PR #2, the polygon vector layer) is pure GeoJSON — no cell-id coordinate involvement whatsoever. Nothing shipped so far touches the decision.

Planning-level changes:

1. The cryocloud path is unchanged — the phase-4 proxy (moczarr hub-side) now fabricates the NESTED cell_ids view at serve time in addition to flattening hive→single-store; the browser keeps seeing exactly the limited-area NESTED shape gridlook already renders. This was already the preferred phase 4↔6 shape; the decision just makes the fabrication mandatory rather than pass-through.
2. Phase 6 grows a real browser-direct arm (upgraded from optional): espg wants a public source.coop example that renders browser-direct — no proxy in that path, so the fork needs native TS morton support: BigInt word handling (words exceed 2⁵³ — Number is unsafe for morton, unlike NESTED), a gridTypeDetector branch for the morton convention entry (the detector currently rejects dggs.name != "healpix"), morton→NESTED conversion (or direct morton rendering) in TS, and the order-29 rule from zagg#262: unknown-resolution-encoded-at-29 ids clip on the fly to order 24 (float64-safe); genuinely-finer data takes the proxy path. moczarr's golden vectors remain the TS parity fixtures.
3. Phase 6 v1 ordering flips slightly: "point gridlook at a leaf as-is" keeps working only while zagg dual-writes cell_ids (it will, until moczarr's fabrication layer ships — the zagg#262 gate). Fine for development; the morton detector branch is what makes it durable.

[espg]

🤖 from Claude

Phase 4 plan: gridlook-jupyter — posted for ratification before implementation

Per the in-session agreement, phases 4+ get a plan-per-phase pass (real design surface, unlike the frontend phases). Phase 3 is implementing in parallel. Ratified inputs carried in: one custom jupyter-server extension (static app + S3 proxy), pip name gridlook-jupyter, in-fork jupyter/ dir, allowlist-only auth to start, NASA no-public-buckets (browser cannot reach the buckets; hub can).

Proposed shape

• jupyter/ directory with its own pyproject.toml (hatchling). The wheel embeds the built Vite dist/ as package data via a build hook that runs npm ci && npm run build — dist is never committed; sdist/wheel builds happen in CI where node is available. A new package.yml-style workflow job builds the wheel on PRs touching jupyter/ or src/.
• One server extension (gridlook_jupyter, jupyter_server extension points) with two route families:
  • /gridlook/ → serves the static SPA (works under any base path — the app already ships base: "./").
  • /gridlook/s3/<bucket>/<key...> → streaming byte-range proxy: forwards Range headers to S3 via obstore (dep parity with moczarr/zagg), returns 206s, hub-side credentials only — creds never reach the browser. GET/HEAD only, no LIST in v1 (zarr/zarrita don't need it; the manifest/MOC pattern exists precisely to avoid LIST).
  • Route namespace /gridlook/hive/... reserved, not implemented — this is where the phase-6 moczarr virtual-store endpoint (serve an open_hive() AOI as one flat zarr store) lands later, per amendment 2.
• Config surface (traitlets, standard jupyter-server config + env fallback): GridlookProxy.allowed_buckets (list; empty = proxy disabled), GridlookProxy.region, credentials from the ambient chain (the cryocloud case). Requests outside the allowlist → 403 with a pointed message.
• v1 launcher = a URL (<hub-base>/gridlook/), documented; the launcher card (tiny labextension or jupyter-app-launcher entry) is phase 5, per the plan split.
• Deps: jupyter-server, obstore. moczarr only arrives with the phase-6 hive endpoint.
• Tests: pytest against a running jupyter-server (jupyter_server's pytest fixtures) — static serving, range-request round-trip against a LocalStore-backed fake bucket, allowlist 403, HEAD support. CI job runs them with node present for the build.

Design decisions needing your call (leans included)

1. How the SPA addresses proxied data. (a) v1: users paste the proxy URL directly (/gridlook/s3/bucket/prefix) — zero frontend change, works today; (b) fast-follow: the app recognizes s3://bucket/prefix input and rewrites to the proxy path when served under /gridlook/ (small fork feature, much better UX). Lean: ship (a) in phase 4, (b) as a follow-up commit in the same PR if trivial — ratify (b)'s rewrite rule now?
2. Proxy semantics: streaming vs presigned-redirect. Streaming keeps creds fully hub-side and Range support simple; presigned redirects offload bandwidth to S3 but leak time-limited URLs to the browser and complicate CORS. Lean: streaming, firmly.
3. Packaging trigger: build hook running npm at wheel-build time (standard-ish, needs node in the build env) vs a committed-dist release branch (never). Lean: build hook + CI; local pip install ./jupyter requires node, documented.
4. Auth beyond the allowlist: v1 allowlist-only (ratified). Flag for later: per-user scoping via the hub's auth state if cryocloud ever needs it — out of scope now, named so it's not forgotten.

Say the word on (1)-(4) (or just "leans approved") and phase 4 implements against this plan.

[espg]

Lean: ship (a) in phase 4, (b) as a follow-up commit in the same PR if trivial — ratify (b)'s rewrite rule now?

agree

Auth beyond the allowlist: v1 allowlist-only (ratified). Flag for later: per-user scoping via the hub's auth state if cryocloud ever needs it — out of scope now, named so it's not forgotten.

yeah, we'll want per user scoping pretty quickly so we can enable during dev (this may get relaxed to all users later, but starts opt in)...

leans approved

[espg]

🤖 from Claude

Phase-6 amendment from the updated zagg#299/#300 discussion: path_grouping

zagg's manifest gains a path_grouping field in the #299 window (existing stores retroactively 1; grouped prefixes become a default flip later, not a schema change — ratified on the zagg#300 thread). Binding for the phase-6 TS "moczarr-lite": path construction chunks the morton digit string per the manifest's path_grouping as the only code path — no hardcoded one-digit assumption with a parameter bolted on later. The TS parity fixtures should include a path_grouping: 3 case alongside the default, mirroring the moczarr golden-vector requirement (espg/moczarr#11). Also inherited from those ratifications: readers are MOC-first, always — recursive enumeration is out-of-contract (audit tooling only), which simplifies the phase-6 browser story (manifest + root MOC + arithmetic paths; no LIST fallback in the viewer). Phase 6 remains blocked on zagg#262's convention entry + zagg#299 landing; blast radius tracked at espg/moczarr#11.