Skip to content

Add implementation plan: review + phased build plan#2

Open
Keeeeeeeks wants to merge 2 commits into
add-pipeline-specfrom
add-implementation-plan
Open

Add implementation plan: review + phased build plan#2
Keeeeeeeks wants to merge 2 commits into
add-pipeline-specfrom
add-implementation-plan

Conversation

@Keeeeeeeks

@Keeeeeeeks Keeeeeeeks commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds pipeline-spec/05-implementation-plan.md — turns the v5 spec into a runnable, cluster-scale plan.

  • Adversarial review of the five pipeline-spec docs, verified against the actual v4 code (false-match test, is_cached resume, create_repo-only upload, --gpus=1 ghx4 for a CPU job).
  • Gap analysis vs PI feedback (false-match, consistency, CPU/GPU, DONE markers, dataset availability).
  • Data-access matrix + HF/MMU inventory aligned to the UniverseTBD "Multimodal Universe HATS" collection; corrected for org namespaces (UniverseTBD/ vs hugging-science/ vs LSDB/), with APOGEE DR17 and Legacy-south now on HF, and a flag to verify mmu_gaia_gaia full-vs-preview.
  • Reuses Smith42 pre-cross-matched sets where they cover a pairing, then lsdb's built-in crossmatch for HATS catalogs; convert non-HATS sources to HATS with hats-import and flag any we can't (sec 4.5).
  • Locked decisions: galaxies-first / quality-gated volume; data-gen on x86 Delta cpu + analysis on DeltaAI (shared /taiga); Parquet+fixed_size_list.
  • Fixes: DeltaAI no-CPU-partition (C1), apply_space_motion distance trap (H1), 2-D work model + finalize barrier (H2), manifest build-or-attach (H3), Lustre small-file storm (H4).
  • Phased P0->P2 plan with modules, scripts, and per-phase acceptance criteria; open decisions for PI sign-off.

Notes

Test plan (Phase 0, before the expensive full run)

  • PI reviews the three open decisions: galaxy-image source (HSC-only vs +Legacy), volume ambition, match strictness.
  • Confirm the compute nodes (not just the login node) can reach the internet (HuggingFace / S3). If not, switch to pre-staging data via Globus.
  • On one small patch of sky, confirm our galaxy image-to-spectrum matches agree with Smith42's already-matched catalog for that patch (proves the matching is correct before committing cluster hours).

Turns the v5 spec into a runnable, cluster-scale plan:
- adversarial review of the five pipeline-spec docs (verified vs v4 code)
- gap analysis against PI feedback
- corrected data-access matrix + HF/MMU inventory (preview vs full HATS)
- galaxy-first, quality-gated objective; Delta(CPU) data-gen + DeltaAI(GPU) analysis
- consistency/false-match/typing fixes; P0->P2 plan with scripts + acceptance criteria
- open decisions for PI sign-off
- cite the UniverseTBD "Multimodal Universe HATS" collection as canonical source list
- fix org namespaces (UniverseTBD / hugging-science / LSDB)
- APOGEE DR17 + Legacy-south now on HF (lsdb), drop local-FITS assumption; flag gaia preview
- crossmatch: prefer Smith42 pre-matched + lsdb built-in crossmatch; custom only where needed
- add sec 4.5: convert non-HATS sources to HATS via hats-import, flag any we can't
@Keeeeeeeks Keeeeeeeks changed the title Add implementation plan: adversarial review + phased build plan Add implementation plan: review + phased build plan Jun 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant