Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
db97c6a
Add Claude Code skill for examples/e2e_template
Varfalamei Jun 30, 2026
a98cc92
Add Claude Code skills for embedder_fiftyone and sam_cvat examples
Varfalamei Jun 30, 2026
44bb9eb
Point each example README at its Claude Code setup skill
Varfalamei Jun 30, 2026
4f8b4e1
Document skills the conventional way: root table with direct-invoke n…
Varfalamei Jun 30, 2026
4d34996
Correct e2e skill against master: working-dir overlap is already fixe…
Varfalamei Jun 30, 2026
c964870
Re-verify e2e troubleshooting against master: fix misattributed metri…
Varfalamei Jun 30, 2026
5c0a024
Drop train-race note: it's auto-handled on master, no need to documen…
Varfalamei Jun 30, 2026
268dd48
Drop Claude Code skills section from root README; keep concise per-ex…
Varfalamei Jun 30, 2026
ae6d418
Skills: ask more up front (which DB, reuse env, logs) and work plan-f…
Varfalamei Jul 1, 2026
8aa72b6
Skills: show normal logs; on failure escalate to datapipe --debug/--d…
Varfalamei Jul 1, 2026
b5d34d4
tags-addon: add a reproducible demo-scenario recipe for agents
Varfalamei Jul 1, 2026
1344049
Add datapipe-examples router skill
Varfalamei Jul 2, 2026
0b5b122
e2e_template: bump default seed to 120 cat/dog images; finalize tag d…
Varfalamei Jul 2, 2026
1bdc53b
e2e skill: ask up front whether to annotate in Label Studio or inject…
Varfalamei Jul 6, 2026
a7f8f46
tags-addon: document wiring tags into the pipeline for the UI graph
Varfalamei Jul 6, 2026
a77e2fb
tags-addon: frame pipeline wiring as a per-consumer extension in thei…
Varfalamei Jul 6, 2026
d4428e6
e2e skill: document injecting ground truth to skip Label Studio annot…
Varfalamei Jul 6, 2026
787d1f1
tags-addon: add live demo walkthrough (skill -> inject GT -> tags -> …
Varfalamei Jul 6, 2026
7d716a5
tags-addon: add 'clean slate before demo' step (test data only; clear…
Varfalamei Jul 6, 2026
b2676ef
Add detection_tags example (tags-native, no LS/FiftyOne) + skill; low…
Varfalamei Jul 6, 2026
81ff19f
detection_tags: load data via a datapipe step (load_request table + l…
Varfalamei Jul 6, 2026
96a1916
detection_tags: fix stale load_batch.py references in docs
Varfalamei Jul 6, 2026
28a39e6
detection_tags skill: fix stale list_s3_images note (no such step in …
Varfalamei Jul 6, 2026
4472b8d
detection_tags: cache-first load step (pre-staged images + gt.json to…
Varfalamei Jul 6, 2026
196a90c
detection_tags skill: note transitive polars must be forced to polars…
Varfalamei Jul 6, 2026
8041325
detection_tags skill: branch on demo(test) vs real data; epochs 50->1…
Varfalamei Jul 6, 2026
697d13c
Lock validated demo params: detection_tags base 450 + night 50 (500 t…
Varfalamei Jul 6, 2026
39508f2
detection_tags: document creating the DB schema before db create-all …
Varfalamei Jul 6, 2026
9ebdce3
detection_tags: expose TP/FP/FN in tag_metrics
Varfalamei Jul 6, 2026
37f5ab5
detection_tags skill: surface baseline tag metric before retraining (…
Varfalamei Jul 6, 2026
70c8c74
detection_tags: fixed-val demo + FiftyOne + two-part checkpoint flow
Varfalamei Jul 7, 2026
91927e6
detection_tags: split demo data synthesis out of the pipeline
Varfalamei Jul 7, 2026
d37ac04
Merge remote-tracking branch 'origin/add-e2e-template-claude-skill' i…
Varfalamei Jul 7, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions .claude/skills/datapipe-examples/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
---
name: datapipe-examples
description: >
Use when working in the epoch8 datapipe repo with the examples/* folders, when asked to run or
debug any datapipe example pipeline on your own data, or when unsure which example or setup skill
applies.
---

# datapipe examples — router

Every example here is meant to run on YOUR data; each bundled demo dataset is only a smoke-test. Pick the example, then its setup skill, then set that skill's "Run on YOUR data" knobs first.

Each `examples/*` pipeline has a dedicated setup skill — pick by what it does:

| Example | What it does | Skill |
|---|---|---|
| `embedder_fiftyone` | DINOv2/DINOv3 image embeddings → FiftyOne UMAP + similarity | **setup-embedder-fiftyone** |
| `e2e_template/image_detection` | YOLO detection + Label Studio human-in-the-loop → train → FiftyOne | **setup-e2e-template** |
| `e2e_template/image_keypoints` | YOLO-pose keypoints + Label Studio → train → FiftyOne | **setup-e2e-template** |
| `sam_cvat` | SAM3 text-prompt boxes+masks → CVAT pre-annotations | **setup-sam-cvat** |
| `detection_tags` | YOLO detection + **tags** (per-scenario metrics), no Label Studio / FiftyOne, GT injected | **setup-detection-tags** |

## Ask first — don't assume (only the unresolved)
Clarify what's not obvious before acting — don't spin up services or pull data you don't need. Ask only the unresolved:
- **Demo or your own data?** · **Provision Postgres/services or reuse existing?** (e2e ships `docker compose`; embedder/sam need external Postgres)
- **Which Postgres + which database** for `DB_URL`? Never point it at an existing DB or drop in a `localhost` default without confirming.
- **Reuse an existing venv / `uv` env, or create a fresh one?** · **Which GPU?** (SAM3 >8 GB; DINOv2/YOLO ~8 GB) · **Annotation backend up?** (LS for e2e / external CVAT for sam)
- **Surface stage logs or run quiet?** · **Per-tag scenario metrics** (retrain new case, old vs new)? → the `detection_tags` example / **setup-detection-tags**

## How to work
Read the setup, then propose a short plan and get a go-ahead before touching anything. Prepare `.env` and **pause for the user to verify it** before running. Run each stage with its logs shown and, after each, say what you did and what changed — don't run the pipeline silently. Trust the status table (`*_training_status`/`brain_status`), not the exit code. If a stage fails and the cause isn't clear from the normal logs, re-run it with `datapipe --debug … run` (or `--debug-sql` for SQL errors) sent to a file and `grep`ped, rather than dumping the verbose output inline.

## Run on YOUR data (universal rule)
When you swap in your own subject, **align your class name across every place it appears** — text
prompt / label config, the label field, and any class filter. A mismatch runs fine but yields 0
useful results. The per-example skill lists the exact knobs.

## Universal prerequisites (every example)
1. **PostgreSQL** at `DB_URL`. embedder + sam_cvat need an EXTERNAL Postgres (they don't start one);
e2e_template bundles Postgres in its `docker compose`. Empty DB is fine; tables auto-create via
`datapipe db create-all`. `.env.example` default `...postgres:postgres@localhost:5432/postgres`.
External quick start: `docker run -d -e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres -e POSTGRES_DB=postgres -p 5432:5432 postgres:16`
(k8s pod / no docker → conda postgres, `initdb` as non-root with `--locale=C`.)
2. **A GPU big enough** (`nvidia-smi`): DINOv2/YOLO fit ~8 GB; SAM3 wants >8 GB (OOM'd on an 8 GB Pascal in our tests).
3. **Annotation backend, if used:** e2e_template ships Label Studio in its `docker compose`; sam_cvat
needs an external CVAT you provide (URL + creds + a project whose labels match its config).
4. **`uv` + Python ≥3.10,<3.13.** Each example has a `pyproject.toml` → `uv sync` (cu124 torch pinned,
CUDA OOTB; no manual venv/pip). Read-only/small `$HOME`: `export UV_CACHE_DIR=/tmp/uvcache HF_HOME=/tmp/hf`.
5. **Human-in-the-loop** (e2e_template, sam_cvat): a human reviews and marks tasks completed before
the pipeline advances.

## HF auth (only for GATED models)
Needed for `dinov3_*` (embedder, off by default) and `sam3` (sam_cvat). Accept the model license on
its HF page, then `export HF_TOKEN=...` with **gated-repo read access**. A plain token → `403`, often
masked as `LocalEntryNotFoundError: check your connection`. Public models (dinov2, YOLO) need no token.

## Universal run shape
```bash
cp .env.example .env # set DB_URL and the example's data + backend vars
uv sync
datapipe db create-all # auto-creates all tables
datapipe run # or: datapipe step --labels stage=<s> run
```
Datapipe caches by content: re-running does little work unless inputs or tracked config change.
Then follow the per-example skill — set its **Run on YOUR data** knobs first.
Loading
Loading