From db97c6a8ed9e862d92d48b5eb78ae42985768736 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Tue, 30 Jun 2026 17:18:04 +0300 Subject: [PATCH 01/32] Add Claude Code skill for examples/e2e_template Project skill under .claude/skills/setup-e2e-template covering setup, run, training and troubleshooting of the e2e_template pipeline on your own data, plus an optional tags-addon recipe for per-scenario metrics. README documents how it is auto-discovered and used. --- .claude/skills/setup-e2e-template/SKILL.md | 63 +++++++++++++++++ .../skills/setup-e2e-template/tags-addon.md | 68 +++++++++++++++++++ README.md | 15 ++++ 3 files changed, 146 insertions(+) create mode 100644 .claude/skills/setup-e2e-template/SKILL.md create mode 100644 .claude/skills/setup-e2e-template/tags-addon.md diff --git a/.claude/skills/setup-e2e-template/SKILL.md b/.claude/skills/setup-e2e-template/SKILL.md new file mode 100644 index 00000000..de18be3d --- /dev/null +++ b/.claude/skills/setup-e2e-template/SKILL.md @@ -0,0 +1,63 @@ +--- +name: setup-e2e-template +description: > + Use when setting up or running examples/e2e_template (image_detection or image_keypoints) — + datapipe YOLO/Label Studio pipelines — or debugging their install, stages, training, or FiftyOne output. +--- + +# e2e_template (detection + keypoints YOLO/Label-Studio pipeline) + +This skill = run the YOLO/Label-Studio detection/keypoints pipeline on YOUR images. The COCO seed sample is just a smoke-test; the real goal is your own images — set the knobs below first. + +**Before starting, if not already provided, ask the user:** demo or your own data? bring up the `docker compose` services or reuse existing? GPU available? per-tag scenario metrics ([tags-addon.md](tags-addon.md))? — ask only what's unresolved, then do only what's needed. + +## Run on YOUR data +- **Align your class everywhere** (mismatch → 0 useful results): `LABEL_CONFIG` names == `CLASSES_TO_KEEP`. Detection also `COCO_CLASSES`/`DETECTION_MODEL_CONFIG`; keypoints also `KEYPOINTS_LABELS` (order matters), `COCO_PERSON_KEYPOINT_FLIP_IDX`, `KEYPOINTS_MODEL_CONFIG`. +- **Working-dir overlap (fix this):** shipped `S3_PREFIX=images` + `S3_DATAPIPE_PREFIX=images/datapipe` is nested inside the input, and `list_s3_images` recurses `{bucket}/images` → pipeline re-ingests its own outputs. Set `S3_DATAPIPE_PREFIX` (or `DATAPIPE_E2E_DIR`) **outside** `S3_PREFIX`. +- **Own S3 (where YOUR images go):** set `AWS_*`, `S3_BUCKET`, `S3_PREFIX`, `S3_PUBLIC_URL` (browser-reachable; LS loads from `$S3_PUBLIC_URL/$S3_BUCKET/`). Two endpoints: `S3_ENDPOINT_URL` (host) vs `LABEL_STUDIO_S3_ENDPOINT_URL` (`minio:9000`). + +## Prerequisites +- **Python 3.10–3.12** (hard pin `>=3.10,<3.13`). **Install:** `cd examples/e2e_template && uv sync` + (`pyproject.toml` declares editable workspace libs + cu124 `torch==2.6.0`). +- **Services:** `docker compose up` → Postgres 5432, MinIO (9000/9001, bucket `datapipe-e2e`, + anon-download for browser images), MongoDB 27017, Label Studio :8080. +- **Env:** `cp .env.example .env` then `set -a && source .env && set +a` before any `datapipe` command + (`config.py` raises if `DB_URL` unset). Detection/keypoints use separate schemas. +- **LS API token:** :8080 → Account & Settings → Access Token → `LABEL_STUDIO_API_KEY=...` in `.env` + (or `scripts/label_studio_token.py`). +- **GPU for training** (no CPU knob). **≥10 annotated images** to freeze a dataset (`min_delta=10`). +- Read-only/small `/home`: `export UV_CACHE_DIR=/tmp/uvcache HF_HOME=/tmp/hf` before `uv sync`. +- Stages: `annotation`, `ls-sync`, `train`, `fiftyone`. + +## Quick demo to verify setup +Skip if you have data: `uv run python scripts/seed_sample_data.py` makes schemas, caches COCO (~241MB, override `DATAPIPE_CACHE_DIR`), uploads ~20 JPEGs to MinIO; then run §Run as-is. + +## Run (from the project subdir) +```bash +cd image_detection # or image_keypoints +source ../.venv/bin/activate # else prefix every command with `uv run` +set -a && source ../.env && set +a # config raises w/o DB_URL +datapipe db create-all +datapipe step --labels=stage=annotation run # LS tasks + pre-annotations +# → annotate ≥10 images in LS (:8080), mark completed → +datapipe step --labels=stage=ls-sync run # → image__ground_truth +datapipe step --labels=stage=train run # freeze + train YOLO + metrics + best +datapipe step --labels=stage=fiftyone run +fiftyone app launch datapipe_detection_e2e # or datapipe_keypoints_e2e +``` +Train uses `yolov8n*.pt` (imgsz 320, 30 ep); pre-annotation fallback `yolo11n*.pt`; best on `subset_id=val`. + +## Add-on upgrade — tags for per-scenario metrics +An **optional layer on top of the base pipeline** (not needed to run it). New case (e.g. dark-room +pallets) → tag those images, let part flow into training, and measure the model **on that scenario +separately** (old vs new) without touching the split. Bolt-on recipe (catalog `tag`/`image__tag` + +one `tag_metrics` step; example logic unchanged): [tags-addon.md](tags-addon.md). Verified end-to-end — +`tag_metrics` shows the retrained model gaining recall on the tagged scenario. + +## Troubleshooting (may already be fixed — verify against current files) +- **Exit 0 but no model trained** → datapipe swallows step errors; check the `*_training_status` table, not the exit code. +- **First `train` fails: `No weight file found for best epoch N`** → S3 sync/rename race (`best.pt` still `.tmp`). Just **re-run** `stage=train` — it finalizes from the synced checkpoint. (keypoints avoid it via `save_period:1`.) +- **`detection` metrics all 0 / no best model** → the shipped `DETECTION_MODEL_CONFIG` is a smoke config (`input_size:[16,16]`, `score_threshold:0.01`) → model predicts nothing. For real metrics use `input_size:[320,320]`+`score_threshold:~0.25` and enough data. +- **Dataset silently grows** → working-dir nested in input prefix (see "Working-dir overlap"). +- **`cv_pipeliner` keypoint pre-annotation is broken** (pinned rev): inferencer drops keypoints, LS parser doesn't apply them → keypoint `train` needs real keypoint GT injected. +- **Offline / restricted-network / pre-AVX2 nodes:** the shipped `polars-lts-cpu` pin breaks on pre-AVX2 (`ImportError`, mixed runtimes) → `uv pip install "polars[rtcompat]==1.42.0"` (the `rtcompat` extra; bare `polars-runtime-compat` isn't directly installable). AMP-check hangs downloading `yolo26n.pt` (pre-place it from the **`v8.4.0`** assets tag + the YOLO weights). `/home` read-only → `UV_*`/`UV_UNMANAGED_INSTALL` on `/tmp`. `seed_sample_data.py` aborts on a single COCO timeout on a flaky link → per-image retry. diff --git a/.claude/skills/setup-e2e-template/tags-addon.md b/.claude/skills/setup-e2e-template/tags-addon.md new file mode 100644 index 00000000..e309a855 --- /dev/null +++ b/.claude/skills/setup-e2e-template/tags-addon.md @@ -0,0 +1,68 @@ +# e2e_template — add-on: tags for per-scenario metrics + +**An optional layer on top of the base pipeline** (not needed to run it — a bolt-on you add when you +want it). + +**When you need it.** A new case shows up (e.g. "the model misses pallets in dark rooms"). You want +to: (1) label those images with a **tag**, (2) let part of them flow into training, (3) **measure the +model on that scenario separately** and compare old vs new — without touching the normal train/val. + +This is a **datapipe-native recipe**. Below: what to add and how. It was run end-to-end on e2e +detection and checks out. + +## Principle +- **Don't touch the split.** Scenario images get a tag and flow through the normal `image__subset` → + part lands in `train` (the model learns them = "added to training"), part in the holdout. + **"Add to training" = just tag the new images** — no extra logic. +- **A tag is just a label** (`image__tag`, many tags per image); it does NOT affect split/training. +- **Slice metrics by `(tag_id, subset_id)`** in a separate `tag_metrics` table; standard metrics untouched. +- **Old vs new model** — compare `tag_metrics` rows by eye (no automated gate). +- ⚠️ In the example the split yields only **`train`/`val`** (no `test`): holdout = `val` (or do a 3-way split). + +## What to add (catalog + one step; example logic unchanged) +- `tag(tag_id PK, name)` — tag dictionary. +- `image__tag(image_name PK, tag_id PK)` — tags per image, **many per image**. Key = `image_name` + (as in every e2e table). Both read via the standard `UpdateExternalTable`; who writes them is out of + scope (UI / manual INSERT / script). +- `tag_metrics(detection_model_id PK, tag_id PK, subset_id PK, calc__precision, calc__recall, + calc__f1_score, support, images_support)` — a new `BatchTransform` step. + **Source — NOT the raw `detection_predictions`** (that's pre-annotation; the trained model's output is + in `detection_prediction_train`). Cleanest: take the already-computed **per-image** + `detection_model_train__metrics_on_image` (`image_name, detection_model_id, subset_id, + calc__TP/FP/FN/support`), join `image__tag` on `image_name`, and **re-aggregate by + `(detection_model_id, tag_id, subset_id)`** — same as `count_..._metrics_on_subset`, just with + `tag_id` in the GROUP BY. +- **Leave alone:** `image__subset`/split, `..._metrics_on_subset`, `FindBestModel(subset_id="val")`, training. + +## How to use +1. Label the scenario images → rows in `image__tag` (a tag, e.g. `dark_room`). +2. Run the pipeline **as usual** — tagged images go through the normal split, part to `train` + (learned), part to `val` (holdout). +3. Read `tag_metrics`: the row `model=new, tag=dark_room, subset=val` against the same row for the old + model (several versions live in `detection_model_train`, per-image metrics computed for each). + +## Aggregation (core of the `tag_metrics` step) +The same can be checked one-off in SQL. ⚠️ Columns `calc__TP/FP/FN` were created in **mixed case** → +in Postgres quote them (`"calc__TP"`), otherwise `column ... does not exist`. +```sql +SELECT m.detection_model_id, t.tag_id, m.subset_id, + count(*) AS images_support, + sum(m."calc__TP")+sum(m."calc__FN") AS support, + sum(m."calc__TP")::numeric / NULLIF(sum(m."calc__TP")+sum(m."calc__FP"),0) AS precision, + sum(m."calc__TP")::numeric / NULLIF(sum(m."calc__TP")+sum(m."calc__FN"),0) AS recall +FROM detection_model_train__metrics_on_image m +JOIN image__tag t USING (image_name) +WHERE t.tag_id = 'dark_room' +GROUP BY 1,2,3; +``` + +## Gotchas (datapipe-ml behavior) +- **Any custom script that drives datapipe-ml training must be guarded with `if __name__ == "__main__":`** + — datapipe-ml launches YOLO training via multiprocessing **spawn**, so without the guard the child + re-imports the module and crashes with `RuntimeError: An attempt has been made to start a new process + before ... bootstrapping`. The stock `datapipe step ... stage=train run` is already guarded. +- **Post-training is not instant:** datapipe-ml syncs every epoch checkpoint to S3/MinIO and + `collect_results`/select-best reads them back, so on a slow link this takes minutes — normal, not a + hang (the process sits in `Sl`/`Dl`). +- **Don't carve up `image__subset`** to fake a "scenario" — the tag IS the slice; slice metrics by the + tag and keep the split full, otherwise old and new models get measured on different holdouts. diff --git a/README.md b/README.md index 244e3b44..9712ea4a 100644 --- a/README.md +++ b/README.md @@ -53,6 +53,21 @@ uv run python -c "import datapipe_cvat.cvat_step, datapipe_cvat.utils" Documentation lives in `libs/datapipe-core/docs`. Design notes live in `libs/datapipe-core/design-docs`. +## Claude Code skill: e2e_template setup + +`.claude/skills/setup-e2e-template/` is a [Claude Code skill](https://code.claude.com/docs/en/skills) +for setting up and running `examples/e2e_template` (YOLO detection / keypoints + Label Studio → +train → FiftyOne) on your own data — external prerequisites, env knobs, and the data-alignment +gotchas are baked in. + +**Adding it:** nothing to install. It's a project skill committed to the repo, so anyone who clones +this repo and opens it in Claude Code gets it auto-discovered. + +**Using it:** just describe the task — e.g. *"set up the e2e_template detection example on my images"* +or *"the model misses pallets in dark rooms — add a tag and measure it separately"*. Claude loads the +skill when the request matches its triggers. The optional `setup-e2e-template/tags-addon.md` covers +per-scenario tag metrics (tag images → part flows into training → a separate `tag_metrics` table). + ## Version Compatibility * `master` — current development state, will become the `0.15.x` release From a98cc92ba7318fde703ec8b0486184abd471d29a Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Tue, 30 Jun 2026 17:39:03 +0300 Subject: [PATCH 02/32] Add Claude Code skills for embedder_fiftyone and sam_cvat examples Adds project skills setup-embedder-fiftyone and setup-sam-cvat alongside the e2e_template skill, and generalizes the README section to list all three. --- .../skills/setup-embedder-fiftyone/SKILL.md | 66 +++++++++++++++++++ .claude/skills/setup-sam-cvat/SKILL.md | 60 +++++++++++++++++ README.md | 26 ++++---- 3 files changed, 141 insertions(+), 11 deletions(-) create mode 100644 .claude/skills/setup-embedder-fiftyone/SKILL.md create mode 100644 .claude/skills/setup-sam-cvat/SKILL.md diff --git a/.claude/skills/setup-embedder-fiftyone/SKILL.md b/.claude/skills/setup-embedder-fiftyone/SKILL.md new file mode 100644 index 00000000..7c028a1a --- /dev/null +++ b/.claude/skills/setup-embedder-fiftyone/SKILL.md @@ -0,0 +1,66 @@ +--- +name: setup-embedder-fiftyone +description: > + Use when working in examples/embedder_fiftyone, or running / debugging the embedder→FiftyOne + datapipe example (DINOv2/DINOv3 image embeddings, UMAP, similarity search). +--- + +# embedder_fiftyone (DINO → FiftyOne) + +This skill = run the DINO→FiftyOne embedder on YOUR images. The FiftyOne-zoo dataset (caltech101) is just a smoke-test; the real goal is your own images — set the knobs below first. + +**Before starting, if not already provided, ask the user:** demo (zoo) or your own data? external Postgres up or provision it? GPU? — ask only what's unresolved, then do only what's needed. + +## Run on YOUR data +- **Align `LABELS_JSON` keys to the file stem** (optional) — a mismatch silently yields an unlabeled + sample (`None`) / 0 useful results. +- **Your images:** `LOCAL_IMAGES_DIR=/path` (`.jpg/.jpeg/.png`; file stem → `image_name`). +- **Pick models:** `config.py` → `EMBEDDERS` (default dinov2_base + dinov2_large). + +## Prerequisites +Stages (labels match `app.py`): `source` (local folder or zoo fallback) → `embedder` → `fiftyone` +(publish samples) → `embeddings` (per-image `.npy`) → `fiftyone-brain` (UMAP + cosine sim per embedder). +- **External PostgreSQL** at `DB_URL` (SQLAlchemy URL). Not started by the example; an empty DB is fine — + `datapipe db create-all` auto-creates tables. `.env.example` default `...postgres@localhost:5432/postgres`. +- **FiftyOne must run on the SAME machine** — samples/brain live in local FiftyOne/MongoDB; remote → + empty panels / `no_matching_samples`. +- **GPU** ~8 GB plenty for dinov2_base/large (CPU works, slower). **`uv` + Python 3.10–3.12** → `uv sync` + (pins cu124 `torch==2.6.0`, `transformers`, `fiftyone`). +- **HF token only for DINOv3.** Defaults (dinov2_base/large) are public; the `dinov3_vitl16` entry is + **commented out** — to use it, uncomment, accept the `facebook/dinov3-*` license, set `HF_TOKEN` with + gated-repo read access (else the whole embeddings batch fails). + +## Quick demo to verify setup +Skip if you have data. Leave `LOCAL_IMAGES_DIR` unset → zoo fallback (`ZOO_DATASET`=caltech101, +`ZOO_NUM_CLASSES`=10, keeps the **last N** classes); run §Run as-is to confirm install/DB/GPU. First run +downloads it, then auto-deletes the source zoo dataset; `image_name` rewritten `/`→`__` (`steps.py:83`). + +## Run (from examples/embedder_fiftyone) +```bash +cp .env.example .env # DB_URL, LOCAL_IMAGES_DIR or ZOO_*, FIFTYONE_DATASET_NAME (+ FIFTYONE_DATABASE_DIR, see traps) +uv sync && source .venv/bin/activate # else prefix each command with `uv run` +datapipe db create-all && datapipe run +# one stage: datapipe step --labels=stage=source run (source|embedder|fiftyone|embeddings|fiftyone-brain) +``` + +## Caching +Embeddings cache as `.npy` in `EMBEDDINGS_DIR/{embedder_id}/` (default `./data/embeddings`); delete to recompute. + +## View in FiftyOne (same machine) +```bash +fiftyone app launch --remote --address 0.0.0.0 --port 5151 --wait -1 # venv active; SSH-forward 5151 +``` +Open `FIFTYONE_DATASET_NAME`. **Embeddings** panel → key `_umap` (single `_`), color by +`ground_truth`. **Similarity** → `__sim` (double `_`) — don't transpose. + +## Success criteria +`brain_status.status = 'ok'` per embedder — **trust the table, not the exit code**: datapipe swallows +step errors and still exits 0. Plus: Embeddings panel shows clusters, similarity returns neighbors. + +## Troubleshooting (may already be fixed — verify against current files) +- **`brain_status` empty / 0 images though run exited 0** → FiftyOne mongod failed to start (old + `~/.fiftyone` datadir: FCV mismatch → `Wrong mongod version` / `failed to bind to port`). Use a fresh + dir: `FIFTYONE_DATABASE_DIR=/tmp/fo_db`. +- **`no_matching_samples`** → no sample `image_name` matched an embedding key: check the `fiftyone` + stage published samples, pipeline + FiftyOne on the same machine, and (zoo) the `__`-rewrite. +- **DINOv3 403 / `LocalEntryNotFoundError`** → needs a license-accepted gated `HF_TOKEN` (see Prerequisites). diff --git a/.claude/skills/setup-sam-cvat/SKILL.md b/.claude/skills/setup-sam-cvat/SKILL.md new file mode 100644 index 00000000..49b173ba --- /dev/null +++ b/.claude/skills/setup-sam-cvat/SKILL.md @@ -0,0 +1,60 @@ +--- +name: setup-sam-cvat +description: > + Use when working in examples/sam_cvat, or when setting up / running / debugging the datapipe + SAM3→CVAT pre-annotation example, on the cats-n-dogs demo data or on your own images. +--- + +# sam_cvat (SAM3 + CVAT) + +This skill = run SAM3→CVAT pre-annotation on YOUR images. The HZMD/cats-n-dogs demo is just a smoke-test; the real goal is your own images — set the knobs below first. + +**Before starting, if not already provided, ask the user:** demo (cats-n-dogs) or your own data? external Postgres + CVAT ready or provision/point at them? GPU >8 GB? — ask only what's unresolved, then do only what's needed. + +## Run on YOUR data +- **Align across all three:** `SAM_TEXT_PROMPT` == the CVAT labels (`CVAT_BOX_LABEL` / `CVAT_POLYGON_LABEL`) == `HF_DATASET_LABEL` must all mean the same class — a mismatch runs clean but yields 0 useful results. +- **Point at your images:** `INPUT_DIR=/path` (`.jpg/.jpeg/.png`; stem → `image_id`). An existing-but-EMPTY `INPUT_DIR` yields 0 images (no HF fallback). +- **Levers:** `SAM_SCORE_THRESHOLD` (0.5), `SAM_MAX_DETECTIONS` (10). + +## Prerequisites +- **GPU >8 GB with FlashAttention.** *Field note:* ~5.5 GB on a 16 GB RTX 4060 Ti, but OOM on an 8 GB + GTX 1070 (Pascal: no FlashAttention → math-attention, O(n²) memory). +- **HF token — SAM3 is gated.** Accept the license at huggingface.co/facebook/sam3, set `HF_TOKEN` + with gated-repo read access (a non-gated token → download/load failure). +- **External PostgreSQL** at `DB_URL` (not started here; tables auto-create via `datapipe db create-all`). +- **External CVAT** (not started here): set `CVAT_URL`, `CVAT_USERNAME`, `CVAT_PASSWORD`, optional + `CVAT_ORGANIZATION`, and `CVAT_PROJECT_ID` for a project whose label names match `CVAT_BOX_LABEL` / + `CVAT_POLYGON_LABEL`. +- **`uv` + Python ≥3.10,<3.13** → `uv sync`. `pyproject.toml` pins cu124 torch and editable local libs + (`../../libs/datapipe-*`, resolves only inside the monorepo checkout), and builds `sam3` from a pinned + git source (needs git, a compiler, `nvcc`/`CUDA_HOME`). If `uv sync` fails, check the `sam3` rev in `[tool.uv.sources]`. +- Stages: `ingest` (local folder or HF-dataset fallback) → `sam` (text-prompt inference → boxes + mask + polygons) → `cvat` (upload images + pre-annotations; parse edits back into `image__annotations`). + +## Quick demo to verify setup +Skip if you have data. Leave `INPUT_DIR` unset → HF fallback; shipped `.env.example` is self-consistent +(`a cat` / `cat_box`/`cat_mask` / `HF_DATASET_LABEL=0`=cats) → first run on `HZMD/cats-n-dogs` yields +detections, confirming install/DB/GPU/CVAT. + +## Run +```bash +cp .env.example .env # DB_URL, HF_TOKEN, CVAT_*, CVAT_PROJECT_ID, SAM_TEXT_PROMPT, labels +uv sync && source .venv/bin/activate # else prefix each command with `uv run` +datapipe db create-all && datapipe run +# or by stage: datapipe step --labels stage=sam run +``` +Run from `examples/sam_cvat/` (`app.py` `load_dotenv()`s before importing config — wrong cwd breaks `.env`). + +## Caching gotcha +`SAM_TEXT_PROMPT` IS tracked (→ `sam_config` table) so changing it retriggers inference — **but +`CVATStep` does NOT push new pre-annotations to EXISTING tasks** (only on image add/remove/path change). +Treat the prompt as fixed per run; to change it, wipe the old CVAT tasks + datapipe CVAT tables, re-run. + +## Annotate (human-in-the-loop) +In CVAT fix preannotations, **mark the task completed**, re-run `datapipe run` → edits land in +`image__annotations`. CVAT tables (for wipes): `image_batches`, `cvat_task`, `cvat_images` (not "cvat_files"), `cvat_task_sync_table`. + +## Troubleshooting (may already be fixed — verify against current files) +- **0 detections** → `.env` class misaligned; re-align all three. +- **CVAT rejects label** → project needs labels named `CVAT_BOX_LABEL`/`CVAT_POLYGON_LABEL`. +- **Prompt change not in CVAT** → expected (see Caching). diff --git a/README.md b/README.md index 9712ea4a..73d1fd96 100644 --- a/README.md +++ b/README.md @@ -53,20 +53,24 @@ uv run python -c "import datapipe_cvat.cvat_step, datapipe_cvat.utils" Documentation lives in `libs/datapipe-core/docs`. Design notes live in `libs/datapipe-core/design-docs`. -## Claude Code skill: e2e_template setup +## Claude Code skills for the examples -`.claude/skills/setup-e2e-template/` is a [Claude Code skill](https://code.claude.com/docs/en/skills) -for setting up and running `examples/e2e_template` (YOLO detection / keypoints + Label Studio → -train → FiftyOne) on your own data — external prerequisites, env knobs, and the data-alignment -gotchas are baked in. +`.claude/skills/` ships [Claude Code skills](https://code.claude.com/docs/en/skills) that help you set +up and run the example pipelines on your own data — external prerequisites, env knobs, and the +data-alignment gotchas are baked in: -**Adding it:** nothing to install. It's a project skill committed to the repo, so anyone who clones -this repo and opens it in Claude Code gets it auto-discovered. +- `setup-e2e-template` — `examples/e2e_template`: YOLO detection / keypoints + Label Studio → + train → FiftyOne. Includes an optional `tags-addon.md` recipe for per-scenario tag metrics. +- `setup-embedder-fiftyone` — `examples/embedder_fiftyone`: DINOv2/DINOv3 embeddings → FiftyOne + UMAP + similarity search. +- `setup-sam-cvat` — `examples/sam_cvat`: SAM3 text-prompt boxes + masks → CVAT pre-annotations. -**Using it:** just describe the task — e.g. *"set up the e2e_template detection example on my images"* -or *"the model misses pallets in dark rooms — add a tag and measure it separately"*. Claude loads the -skill when the request matches its triggers. The optional `setup-e2e-template/tags-addon.md` covers -per-scenario tag metrics (tag images → part flows into training → a separate `tag_metrics` table). +**Adding them:** nothing to install. They are project skills committed to the repo, so anyone who +clones it and opens it in Claude Code gets them auto-discovered. + +**Using them:** just describe the task — e.g. *"set up the e2e_template detection example on my +images"* or *"the model misses pallets in dark rooms — add a tag and measure it separately"*. Claude +loads the matching skill when the request matches its triggers. ## Version Compatibility From 44bb9ebb927b394ad472dc313d22f10cdce82852 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Tue, 30 Jun 2026 18:23:58 +0300 Subject: [PATCH 03/32] Point each example README at its Claude Code setup skill --- examples/e2e_template/README.md | 4 ++++ examples/embedder_fiftyone/README.md | 3 +++ examples/sam_cvat/README.md | 3 +++ 3 files changed, 10 insertions(+) diff --git a/examples/e2e_template/README.md b/examples/e2e_template/README.md index cd1b5ed1..586614ad 100644 --- a/examples/e2e_template/README.md +++ b/examples/e2e_template/README.md @@ -1,5 +1,9 @@ # Datapipe E2E Template +> **Using Claude Code?** This repo ships a setup skill (`.claude/skills/setup-e2e-template`) — just +> describe your task (e.g. "run image_detection on my own images") and it's auto-loaded with the +> prerequisites, env knobs, and gotchas. See the repo root README. + This example contains end-to-end Datapipe pipeline templates: - `image_detection` for bbox detection. diff --git a/examples/embedder_fiftyone/README.md b/examples/embedder_fiftyone/README.md index 3d3d5d20..c72a64b0 100644 --- a/examples/embedder_fiftyone/README.md +++ b/examples/embedder_fiftyone/README.md @@ -1,5 +1,8 @@ # Embedder + FiftyOne Pipeline +> **Using Claude Code?** This repo ships a setup skill (`.claude/skills/setup-embedder-fiftyone`) — +> just describe your task and it's auto-loaded with the prerequisites and gotchas. See the repo root README. + Datapipe example for: - loading images + class labels (local folder or FiftyOne zoo fallback) - running multiple embedders diff --git a/examples/sam_cvat/README.md b/examples/sam_cvat/README.md index 61ea0556..e9054a0d 100644 --- a/examples/sam_cvat/README.md +++ b/examples/sam_cvat/README.md @@ -1,5 +1,8 @@ # SAM3 + CVAT Pipeline +> **Using Claude Code?** This repo ships a setup skill (`.claude/skills/setup-sam-cvat`) — just +> describe your task and it's auto-loaded with the prerequisites and gotchas. See the repo root README. + Datapipe example for: - loading images (local folder or Hugging Face dataset fallback) From 4f8b4e15f962d444ac05f8c5d8a7feb05a41a432 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Tue, 30 Jun 2026 18:28:27 +0300 Subject: [PATCH 04/32] Document skills the conventional way: root table with direct-invoke names, concise per-example pointers --- README.md | 27 ++++++++++++--------------- examples/e2e_template/README.md | 5 ++--- examples/embedder_fiftyone/README.md | 4 ++-- examples/sam_cvat/README.md | 4 ++-- 4 files changed, 18 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 73d1fd96..57865fea 100644 --- a/README.md +++ b/README.md @@ -53,24 +53,21 @@ uv run python -c "import datapipe_cvat.cvat_step, datapipe_cvat.utils" Documentation lives in `libs/datapipe-core/docs`. Design notes live in `libs/datapipe-core/design-docs`. -## Claude Code skills for the examples +## Claude Code skills -`.claude/skills/` ships [Claude Code skills](https://code.claude.com/docs/en/skills) that help you set -up and run the example pipelines on your own data — external prerequisites, env knobs, and the -data-alignment gotchas are baked in: +This repo ships [Claude Code skills](https://code.claude.com/docs/en/skills) for the example pipelines +under `.claude/skills/`. They are **project skills**: open the repo in Claude Code and they are +auto-discovered — no install. Claude loads the relevant one when your request matches it, or you can +invoke it directly by name. -- `setup-e2e-template` — `examples/e2e_template`: YOLO detection / keypoints + Label Studio → - train → FiftyOne. Includes an optional `tags-addon.md` recipe for per-scenario tag metrics. -- `setup-embedder-fiftyone` — `examples/embedder_fiftyone`: DINOv2/DINOv3 embeddings → FiftyOne - UMAP + similarity search. -- `setup-sam-cvat` — `examples/sam_cvat`: SAM3 text-prompt boxes + masks → CVAT pre-annotations. +| Skill | Example | Invoke | +|---|---|---| +| `setup-e2e-template` | `examples/e2e_template` — YOLO detection / keypoints + Label Studio → train → FiftyOne | `/setup-e2e-template` | +| `setup-embedder-fiftyone` | `examples/embedder_fiftyone` — DINOv2/DINOv3 embeddings → FiftyOne UMAP + similarity | `/setup-embedder-fiftyone` | +| `setup-sam-cvat` | `examples/sam_cvat` — SAM3 text-prompt boxes + masks → CVAT pre-annotations | `/setup-sam-cvat` | -**Adding them:** nothing to install. They are project skills committed to the repo, so anyone who -clones it and opens it in Claude Code gets them auto-discovered. - -**Using them:** just describe the task — e.g. *"set up the e2e_template detection example on my -images"* or *"the model misses pallets in dark rooms — add a tag and measure it separately"*. Claude -loads the matching skill when the request matches its triggers. +Each skill carries the example's external prerequisites, env knobs, and data-alignment gotchas. +`setup-e2e-template` also bundles `tags-addon.md`, a recipe for per-scenario tag metrics. ## Version Compatibility diff --git a/examples/e2e_template/README.md b/examples/e2e_template/README.md index 586614ad..42b10f23 100644 --- a/examples/e2e_template/README.md +++ b/examples/e2e_template/README.md @@ -1,8 +1,7 @@ # Datapipe E2E Template -> **Using Claude Code?** This repo ships a setup skill (`.claude/skills/setup-e2e-template`) — just -> describe your task (e.g. "run image_detection on my own images") and it's auto-loaded with the -> prerequisites, env knobs, and gotchas. See the repo root README. +**Claude Code skill:** `/setup-e2e-template` — auto-loads when relevant, or invoke it directly. It +carries the prerequisites, env knobs, and gotchas below (see the repo root README). This example contains end-to-end Datapipe pipeline templates: diff --git a/examples/embedder_fiftyone/README.md b/examples/embedder_fiftyone/README.md index c72a64b0..f45df075 100644 --- a/examples/embedder_fiftyone/README.md +++ b/examples/embedder_fiftyone/README.md @@ -1,7 +1,7 @@ # Embedder + FiftyOne Pipeline -> **Using Claude Code?** This repo ships a setup skill (`.claude/skills/setup-embedder-fiftyone`) — -> just describe your task and it's auto-loaded with the prerequisites and gotchas. See the repo root README. +**Claude Code skill:** `/setup-embedder-fiftyone` — auto-loads when relevant, or invoke it directly +(see the repo root README). Datapipe example for: - loading images + class labels (local folder or FiftyOne zoo fallback) diff --git a/examples/sam_cvat/README.md b/examples/sam_cvat/README.md index e9054a0d..0a516bec 100644 --- a/examples/sam_cvat/README.md +++ b/examples/sam_cvat/README.md @@ -1,7 +1,7 @@ # SAM3 + CVAT Pipeline -> **Using Claude Code?** This repo ships a setup skill (`.claude/skills/setup-sam-cvat`) — just -> describe your task and it's auto-loaded with the prerequisites and gotchas. See the repo root README. +**Claude Code skill:** `/setup-sam-cvat` — auto-loads when relevant, or invoke it directly (see the +repo root README). Datapipe example for: From 4d34996c757d55bacbddeb4977c6606724a76e0a Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Tue, 30 Jun 2026 18:46:50 +0300 Subject: [PATCH 05/32] Correct e2e skill against master: working-dir overlap is already fixed, fix S3/demo knobs master's config separates input ($DATAPIPE_E2E_DIR/images) from the working dir ($DATAPIPE_E2E_DIR/datapipe), so the working-dir-overlap and dataset-grows notes no longer apply; rewrote the 'where your images go' knobs to the env vars master actually ships, corrected the seed-demo description, and dropped the node-specific offline/pre-AVX2 note (polars-lts-cpu is pinned for old CPUs and the seed already retries per image). --- .claude/skills/setup-e2e-template/SKILL.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/.claude/skills/setup-e2e-template/SKILL.md b/.claude/skills/setup-e2e-template/SKILL.md index de18be3d..48c19648 100644 --- a/.claude/skills/setup-e2e-template/SKILL.md +++ b/.claude/skills/setup-e2e-template/SKILL.md @@ -13,8 +13,7 @@ This skill = run the YOLO/Label-Studio detection/keypoints pipeline on YOUR imag ## Run on YOUR data - **Align your class everywhere** (mismatch → 0 useful results): `LABEL_CONFIG` names == `CLASSES_TO_KEEP`. Detection also `COCO_CLASSES`/`DETECTION_MODEL_CONFIG`; keypoints also `KEYPOINTS_LABELS` (order matters), `COCO_PERSON_KEYPOINT_FLIP_IDX`, `KEYPOINTS_MODEL_CONFIG`. -- **Working-dir overlap (fix this):** shipped `S3_PREFIX=images` + `S3_DATAPIPE_PREFIX=images/datapipe` is nested inside the input, and `list_s3_images` recurses `{bucket}/images` → pipeline re-ingests its own outputs. Set `S3_DATAPIPE_PREFIX` (or `DATAPIPE_E2E_DIR`) **outside** `S3_PREFIX`. -- **Own S3 (where YOUR images go):** set `AWS_*`, `S3_BUCKET`, `S3_PREFIX`, `S3_PUBLIC_URL` (browser-reachable; LS loads from `$S3_PUBLIC_URL/$S3_BUCKET/`). Two endpoints: `S3_ENDPOINT_URL` (host) vs `LABEL_STUDIO_S3_ENDPOINT_URL` (`minio:9000`). +- **Where YOUR images go:** put them under `$DATAPIPE_E2E_DIR/images` (`DATAPIPE_E2E_DIR` defaults to `s3://datapipe-e2e`; the pipeline writes its own artifacts under `$DATAPIPE_E2E_DIR/datapipe`, a sibling — input and working dir don't overlap). Set `AWS_*`, `S3_ENDPOINT_URL`, and `S3_PUBLIC_URL` (browser-reachable — Label Studio loads images from it). Label Studio reaches S3 via its own `LABEL_STUDIO_S3_ENDPOINT_URL` (`minio:9000`), not `S3_ENDPOINT_URL`. ## Prerequisites - **Python 3.10–3.12** (hard pin `>=3.10,<3.13`). **Install:** `cd examples/e2e_template && uv sync` @@ -30,7 +29,7 @@ This skill = run the YOLO/Label-Studio detection/keypoints pipeline on YOUR imag - Stages: `annotation`, `ls-sync`, `train`, `fiftyone`. ## Quick demo to verify setup -Skip if you have data: `uv run python scripts/seed_sample_data.py` makes schemas, caches COCO (~241MB, override `DATAPIPE_CACHE_DIR`), uploads ~20 JPEGs to MinIO; then run §Run as-is. +Skip if you have data: `uv run python scripts/seed_sample_data.py` downloads ~20 COCO images (10 cat/dog + 10 person keypoints; `--detection-limit`/`--keypoints-limit` to change) and uploads them to MinIO; then run §Run as-is. ## Run (from the project subdir) ```bash @@ -58,6 +57,4 @@ one `tag_metrics` step; example logic unchanged): [tags-addon.md](tags-addon.md) - **Exit 0 but no model trained** → datapipe swallows step errors; check the `*_training_status` table, not the exit code. - **First `train` fails: `No weight file found for best epoch N`** → S3 sync/rename race (`best.pt` still `.tmp`). Just **re-run** `stage=train` — it finalizes from the synced checkpoint. (keypoints avoid it via `save_period:1`.) - **`detection` metrics all 0 / no best model** → the shipped `DETECTION_MODEL_CONFIG` is a smoke config (`input_size:[16,16]`, `score_threshold:0.01`) → model predicts nothing. For real metrics use `input_size:[320,320]`+`score_threshold:~0.25` and enough data. -- **Dataset silently grows** → working-dir nested in input prefix (see "Working-dir overlap"). - **`cv_pipeliner` keypoint pre-annotation is broken** (pinned rev): inferencer drops keypoints, LS parser doesn't apply them → keypoint `train` needs real keypoint GT injected. -- **Offline / restricted-network / pre-AVX2 nodes:** the shipped `polars-lts-cpu` pin breaks on pre-AVX2 (`ImportError`, mixed runtimes) → `uv pip install "polars[rtcompat]==1.42.0"` (the `rtcompat` extra; bare `polars-runtime-compat` isn't directly installable). AMP-check hangs downloading `yolo26n.pt` (pre-place it from the **`v8.4.0`** assets tag + the YOLO weights). `/home` read-only → `UV_*`/`UV_UNMANAGED_INSTALL` on `/tmp`. `seed_sample_data.py` aborts on a single COCO timeout on a flaky link → per-image retry. From c964870be37ffcd4b4cdb2e3eb38653028b9a09a Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Tue, 30 Jun 2026 19:13:00 +0300 Subject: [PATCH 06/32] Re-verify e2e troubleshooting against master: fix misattributed metrics-0 note and stale train-race advice DETECTION_MODEL_CONFIG ([16,16]) is the pre-annotation fallback model, not the training config (training is yolov8n imgsz 320), so metrics~0 on the seed set is a data-size issue, not that config. master also ships resume_config that auto-retries failed/raced training attempts, so the manual re-run note is stale. --- .claude/skills/setup-e2e-template/SKILL.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.claude/skills/setup-e2e-template/SKILL.md b/.claude/skills/setup-e2e-template/SKILL.md index 48c19648..8671fd0d 100644 --- a/.claude/skills/setup-e2e-template/SKILL.md +++ b/.claude/skills/setup-e2e-template/SKILL.md @@ -54,7 +54,7 @@ one `tag_metrics` step; example logic unchanged): [tags-addon.md](tags-addon.md) `tag_metrics` shows the retrained model gaining recall on the tagged scenario. ## Troubleshooting (may already be fixed — verify against current files) -- **Exit 0 but no model trained** → datapipe swallows step errors; check the `*_training_status` table, not the exit code. -- **First `train` fails: `No weight file found for best epoch N`** → S3 sync/rename race (`best.pt` still `.tmp`). Just **re-run** `stage=train` — it finalizes from the synced checkpoint. (keypoints avoid it via `save_period:1`.) -- **`detection` metrics all 0 / no best model** → the shipped `DETECTION_MODEL_CONFIG` is a smoke config (`input_size:[16,16]`, `score_threshold:0.01`) → model predicts nothing. For real metrics use `input_size:[320,320]`+`score_threshold:~0.25` and enough data. +- **No model after `train`, exit 0** → datapipe swallows step errors; check `detection_training_status`, not the exit code. Training auto-retries failed/raced attempts (`resume_config` resumes from `last` up to `max_attempts`), so a transient first-attempt checkpoint-sync hiccup recovers on its own. +- **Demo pre-annotations are empty** → the fallback `DETECTION_MODEL_CONFIG` is a smoke model (`yolo11n`, `input_size:[16,16]`, `score_threshold:0.01`) used until a trained model exists, so it detects almost nothing. Expected for the demo; for useful pre-annotations set a real model + `input_size`/`score_threshold`. +- **Training metrics ~0** → not a config bug: the trained model is `yolov8n` (imgsz 320, 30 ep), and the seed sample (~20 images) is simply too small to learn from. Real metrics need enough annotated data. - **`cv_pipeliner` keypoint pre-annotation is broken** (pinned rev): inferencer drops keypoints, LS parser doesn't apply them → keypoint `train` needs real keypoint GT injected. From 5c0a0243f51660aea31e6f48f4664a92389083ed Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Tue, 30 Jun 2026 19:25:33 +0300 Subject: [PATCH 07/32] Drop train-race note: it's auto-handled on master, no need to document a fixed problem --- .claude/skills/setup-e2e-template/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.claude/skills/setup-e2e-template/SKILL.md b/.claude/skills/setup-e2e-template/SKILL.md index 8671fd0d..85c45e46 100644 --- a/.claude/skills/setup-e2e-template/SKILL.md +++ b/.claude/skills/setup-e2e-template/SKILL.md @@ -54,7 +54,7 @@ one `tag_metrics` step; example logic unchanged): [tags-addon.md](tags-addon.md) `tag_metrics` shows the retrained model gaining recall on the tagged scenario. ## Troubleshooting (may already be fixed — verify against current files) -- **No model after `train`, exit 0** → datapipe swallows step errors; check `detection_training_status`, not the exit code. Training auto-retries failed/raced attempts (`resume_config` resumes from `last` up to `max_attempts`), so a transient first-attempt checkpoint-sync hiccup recovers on its own. +- **No model after `train`, exit 0** → datapipe swallows step errors; check `detection_training_status`, not the exit code. - **Demo pre-annotations are empty** → the fallback `DETECTION_MODEL_CONFIG` is a smoke model (`yolo11n`, `input_size:[16,16]`, `score_threshold:0.01`) used until a trained model exists, so it detects almost nothing. Expected for the demo; for useful pre-annotations set a real model + `input_size`/`score_threshold`. - **Training metrics ~0** → not a config bug: the trained model is `yolov8n` (imgsz 320, 30 ep), and the seed sample (~20 images) is simply too small to learn from. Real metrics need enough annotated data. - **`cv_pipeliner` keypoint pre-annotation is broken** (pinned rev): inferencer drops keypoints, LS parser doesn't apply them → keypoint `train` needs real keypoint GT injected. From 268dd48586986d43005c53bd7fb12c6402462b48 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Tue, 30 Jun 2026 23:44:34 +0300 Subject: [PATCH 08/32] Drop Claude Code skills section from root README; keep concise per-example pointers --- README.md | 16 ---------------- examples/e2e_template/README.md | 2 +- examples/embedder_fiftyone/README.md | 3 +-- examples/sam_cvat/README.md | 3 +-- 4 files changed, 3 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index 57865fea..244e3b44 100644 --- a/README.md +++ b/README.md @@ -53,22 +53,6 @@ uv run python -c "import datapipe_cvat.cvat_step, datapipe_cvat.utils" Documentation lives in `libs/datapipe-core/docs`. Design notes live in `libs/datapipe-core/design-docs`. -## Claude Code skills - -This repo ships [Claude Code skills](https://code.claude.com/docs/en/skills) for the example pipelines -under `.claude/skills/`. They are **project skills**: open the repo in Claude Code and they are -auto-discovered — no install. Claude loads the relevant one when your request matches it, or you can -invoke it directly by name. - -| Skill | Example | Invoke | -|---|---|---| -| `setup-e2e-template` | `examples/e2e_template` — YOLO detection / keypoints + Label Studio → train → FiftyOne | `/setup-e2e-template` | -| `setup-embedder-fiftyone` | `examples/embedder_fiftyone` — DINOv2/DINOv3 embeddings → FiftyOne UMAP + similarity | `/setup-embedder-fiftyone` | -| `setup-sam-cvat` | `examples/sam_cvat` — SAM3 text-prompt boxes + masks → CVAT pre-annotations | `/setup-sam-cvat` | - -Each skill carries the example's external prerequisites, env knobs, and data-alignment gotchas. -`setup-e2e-template` also bundles `tags-addon.md`, a recipe for per-scenario tag metrics. - ## Version Compatibility * `master` — current development state, will become the `0.15.x` release diff --git a/examples/e2e_template/README.md b/examples/e2e_template/README.md index 42b10f23..6ba40e87 100644 --- a/examples/e2e_template/README.md +++ b/examples/e2e_template/README.md @@ -1,7 +1,7 @@ # Datapipe E2E Template **Claude Code skill:** `/setup-e2e-template` — auto-loads when relevant, or invoke it directly. It -carries the prerequisites, env knobs, and gotchas below (see the repo root README). +carries the prerequisites, env knobs, and gotchas below. This example contains end-to-end Datapipe pipeline templates: diff --git a/examples/embedder_fiftyone/README.md b/examples/embedder_fiftyone/README.md index f45df075..2cbd3ce5 100644 --- a/examples/embedder_fiftyone/README.md +++ b/examples/embedder_fiftyone/README.md @@ -1,7 +1,6 @@ # Embedder + FiftyOne Pipeline -**Claude Code skill:** `/setup-embedder-fiftyone` — auto-loads when relevant, or invoke it directly -(see the repo root README). +**Claude Code skill:** `/setup-embedder-fiftyone` — auto-loads when relevant, or invoke it directly. Datapipe example for: - loading images + class labels (local folder or FiftyOne zoo fallback) diff --git a/examples/sam_cvat/README.md b/examples/sam_cvat/README.md index 0a516bec..5401f67e 100644 --- a/examples/sam_cvat/README.md +++ b/examples/sam_cvat/README.md @@ -1,7 +1,6 @@ # SAM3 + CVAT Pipeline -**Claude Code skill:** `/setup-sam-cvat` — auto-loads when relevant, or invoke it directly (see the -repo root README). +**Claude Code skill:** `/setup-sam-cvat` — auto-loads when relevant, or invoke it directly. Datapipe example for: From ae6d4180d236c54aa3ad6b229e94bb4004cdb90e Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Wed, 1 Jul 2026 13:29:57 +0300 Subject: [PATCH 09/32] Skills: ask more up front (which DB, reuse env, logs) and work plan-first with an .env checkpoint From real-run feedback: ask which Postgres/database to use (don't silently write a localhost default or target an existing DB), whether to reuse an existing venv, and whether to surface stage logs; propose a plan before acting, prepare .env and pause for the user to verify it, and report what changed after each stage instead of running the pipeline silently. --- .claude/skills/setup-e2e-template/SKILL.md | 4 +++- .claude/skills/setup-embedder-fiftyone/SKILL.md | 4 +++- .claude/skills/setup-sam-cvat/SKILL.md | 4 +++- 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/.claude/skills/setup-e2e-template/SKILL.md b/.claude/skills/setup-e2e-template/SKILL.md index 85c45e46..3a9aeb97 100644 --- a/.claude/skills/setup-e2e-template/SKILL.md +++ b/.claude/skills/setup-e2e-template/SKILL.md @@ -9,7 +9,9 @@ description: > This skill = run the YOLO/Label-Studio detection/keypoints pipeline on YOUR images. The COCO seed sample is just a smoke-test; the real goal is your own images — set the knobs below first. -**Before starting, if not already provided, ask the user:** demo or your own data? bring up the `docker compose` services or reuse existing? GPU available? per-tag scenario metrics ([tags-addon.md](tags-addon.md))? — ask only what's unresolved, then do only what's needed. +**Ask first — don't assume (only the unresolved):** demo or your own data? bring up the bundled `docker compose` services or reuse existing? **which Postgres + which database** for `DB_URL` — never point it at an existing DB without confirming; reuse an existing venv / `uv` env or create a fresh one? GPU? per-tag scenario metrics ([tags-addon.md](tags-addon.md))? surface stage logs or run quiet? + +**How to work:** read the setup, then propose a short plan and get a go-ahead before touching anything. Prepare `.env` and **pause for the user to verify it** before running. Run each stage with logs surfaced (e.g. `datapipe run 2>&1 | tail -60`) unless told otherwise, and after each stage say what you did and what changed — don't run the whole pipeline silently. ## Run on YOUR data - **Align your class everywhere** (mismatch → 0 useful results): `LABEL_CONFIG` names == `CLASSES_TO_KEEP`. Detection also `COCO_CLASSES`/`DETECTION_MODEL_CONFIG`; keypoints also `KEYPOINTS_LABELS` (order matters), `COCO_PERSON_KEYPOINT_FLIP_IDX`, `KEYPOINTS_MODEL_CONFIG`. diff --git a/.claude/skills/setup-embedder-fiftyone/SKILL.md b/.claude/skills/setup-embedder-fiftyone/SKILL.md index 7c028a1a..208c0075 100644 --- a/.claude/skills/setup-embedder-fiftyone/SKILL.md +++ b/.claude/skills/setup-embedder-fiftyone/SKILL.md @@ -9,7 +9,9 @@ description: > This skill = run the DINO→FiftyOne embedder on YOUR images. The FiftyOne-zoo dataset (caltech101) is just a smoke-test; the real goal is your own images — set the knobs below first. -**Before starting, if not already provided, ask the user:** demo (zoo) or your own data? external Postgres up or provision it? GPU? — ask only what's unresolved, then do only what's needed. +**Ask first — don't assume (only the unresolved):** demo (zoo) or your own data? **which Postgres + which database** for `DB_URL` — never point it at an existing DB or drop in a `localhost` default without confirming; reuse an existing venv / `uv` env or create a fresh one? GPU? surface stage logs or run quiet? + +**How to work:** read the setup, then propose a short plan and get a go-ahead before touching anything. Prepare `.env` and **pause for the user to verify it** before running. Run each stage with logs surfaced (e.g. `datapipe run 2>&1 | tail -60`) unless told otherwise, and after each stage say what you did and what changed — don't run the whole pipeline silently. ## Run on YOUR data - **Align `LABELS_JSON` keys to the file stem** (optional) — a mismatch silently yields an unlabeled diff --git a/.claude/skills/setup-sam-cvat/SKILL.md b/.claude/skills/setup-sam-cvat/SKILL.md index 49b173ba..fff11b62 100644 --- a/.claude/skills/setup-sam-cvat/SKILL.md +++ b/.claude/skills/setup-sam-cvat/SKILL.md @@ -9,7 +9,9 @@ description: > This skill = run SAM3→CVAT pre-annotation on YOUR images. The HZMD/cats-n-dogs demo is just a smoke-test; the real goal is your own images — set the knobs below first. -**Before starting, if not already provided, ask the user:** demo (cats-n-dogs) or your own data? external Postgres + CVAT ready or provision/point at them? GPU >8 GB? — ask only what's unresolved, then do only what's needed. +**Ask first — don't assume (only the unresolved):** demo (cats-n-dogs) or your own data? **which Postgres + which database** for `DB_URL` — never point it at an existing DB or drop in a default without confirming; external CVAT ready or provision/point at it? reuse an existing venv / `uv` env or create a fresh one? GPU >8 GB? surface stage logs or run quiet? + +**How to work:** read the setup, then propose a short plan and get a go-ahead before touching anything. Prepare `.env` and **pause for the user to verify it** before running. Run each stage with logs surfaced (e.g. `datapipe run 2>&1 | tail -60`) unless told otherwise, and after each stage say what you did and what changed — don't run the whole pipeline silently. ## Run on YOUR data - **Align across all three:** `SAM_TEXT_PROMPT` == the CVAT labels (`CVAT_BOX_LABEL` / `CVAT_POLYGON_LABEL`) == `HF_DATASET_LABEL` must all mean the same class — a mismatch runs clean but yields 0 useful results. From 8aa72b66cca06ae9543c345cff627a9ada442abf Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Wed, 1 Jul 2026 17:45:15 +0300 Subject: [PATCH 10/32] Skills: show normal logs; on failure escalate to datapipe --debug/--debug-sql into a file + grep Replaces the tail-60 guidance (which hid progress and still cost tokens): run stages with their normal INFO logs shown, and only when a failure's cause is unclear re-run with datapipe --debug (or --debug-sql) written to a file and grepped, instead of dumping the verbose debug firehose into context. --- .claude/skills/setup-e2e-template/SKILL.md | 2 +- .claude/skills/setup-embedder-fiftyone/SKILL.md | 2 +- .claude/skills/setup-sam-cvat/SKILL.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.claude/skills/setup-e2e-template/SKILL.md b/.claude/skills/setup-e2e-template/SKILL.md index 3a9aeb97..3e1997ad 100644 --- a/.claude/skills/setup-e2e-template/SKILL.md +++ b/.claude/skills/setup-e2e-template/SKILL.md @@ -11,7 +11,7 @@ This skill = run the YOLO/Label-Studio detection/keypoints pipeline on YOUR imag **Ask first — don't assume (only the unresolved):** demo or your own data? bring up the bundled `docker compose` services or reuse existing? **which Postgres + which database** for `DB_URL` — never point it at an existing DB without confirming; reuse an existing venv / `uv` env or create a fresh one? GPU? per-tag scenario metrics ([tags-addon.md](tags-addon.md))? surface stage logs or run quiet? -**How to work:** read the setup, then propose a short plan and get a go-ahead before touching anything. Prepare `.env` and **pause for the user to verify it** before running. Run each stage with logs surfaced (e.g. `datapipe run 2>&1 | tail -60`) unless told otherwise, and after each stage say what you did and what changed — don't run the whole pipeline silently. +**How to work:** read the setup, then propose a short plan and get a go-ahead before touching anything. Prepare `.env` and **pause for the user to verify it** before running. Run each stage with its logs shown and, after each, say what you did and what changed — don't run the pipeline silently. If a stage fails and the cause isn't clear from the normal logs, re-run it with `datapipe --debug … run` (or `--debug-sql` for SQL errors); debug is very verbose, so send it to a file and `grep` it (e.g. `datapipe --debug run > /tmp/dp_debug.log 2>&1; grep -nEi "error|traceback" /tmp/dp_debug.log`) rather than dumping it inline. ## Run on YOUR data - **Align your class everywhere** (mismatch → 0 useful results): `LABEL_CONFIG` names == `CLASSES_TO_KEEP`. Detection also `COCO_CLASSES`/`DETECTION_MODEL_CONFIG`; keypoints also `KEYPOINTS_LABELS` (order matters), `COCO_PERSON_KEYPOINT_FLIP_IDX`, `KEYPOINTS_MODEL_CONFIG`. diff --git a/.claude/skills/setup-embedder-fiftyone/SKILL.md b/.claude/skills/setup-embedder-fiftyone/SKILL.md index 208c0075..d526710c 100644 --- a/.claude/skills/setup-embedder-fiftyone/SKILL.md +++ b/.claude/skills/setup-embedder-fiftyone/SKILL.md @@ -11,7 +11,7 @@ This skill = run the DINO→FiftyOne embedder on YOUR images. The FiftyOne-zoo d **Ask first — don't assume (only the unresolved):** demo (zoo) or your own data? **which Postgres + which database** for `DB_URL` — never point it at an existing DB or drop in a `localhost` default without confirming; reuse an existing venv / `uv` env or create a fresh one? GPU? surface stage logs or run quiet? -**How to work:** read the setup, then propose a short plan and get a go-ahead before touching anything. Prepare `.env` and **pause for the user to verify it** before running. Run each stage with logs surfaced (e.g. `datapipe run 2>&1 | tail -60`) unless told otherwise, and after each stage say what you did and what changed — don't run the whole pipeline silently. +**How to work:** read the setup, then propose a short plan and get a go-ahead before touching anything. Prepare `.env` and **pause for the user to verify it** before running. Run each stage with its logs shown and, after each, say what you did and what changed — don't run the pipeline silently. If a stage fails and the cause isn't clear from the normal logs, re-run it with `datapipe --debug … run` (or `--debug-sql` for SQL errors); debug is very verbose, so send it to a file and `grep` it (e.g. `datapipe --debug run > /tmp/dp_debug.log 2>&1; grep -nEi "error|traceback" /tmp/dp_debug.log`) rather than dumping it inline. ## Run on YOUR data - **Align `LABELS_JSON` keys to the file stem** (optional) — a mismatch silently yields an unlabeled diff --git a/.claude/skills/setup-sam-cvat/SKILL.md b/.claude/skills/setup-sam-cvat/SKILL.md index fff11b62..9d01ec15 100644 --- a/.claude/skills/setup-sam-cvat/SKILL.md +++ b/.claude/skills/setup-sam-cvat/SKILL.md @@ -11,7 +11,7 @@ This skill = run SAM3→CVAT pre-annotation on YOUR images. The HZMD/cats-n-dogs **Ask first — don't assume (only the unresolved):** demo (cats-n-dogs) or your own data? **which Postgres + which database** for `DB_URL` — never point it at an existing DB or drop in a default without confirming; external CVAT ready or provision/point at it? reuse an existing venv / `uv` env or create a fresh one? GPU >8 GB? surface stage logs or run quiet? -**How to work:** read the setup, then propose a short plan and get a go-ahead before touching anything. Prepare `.env` and **pause for the user to verify it** before running. Run each stage with logs surfaced (e.g. `datapipe run 2>&1 | tail -60`) unless told otherwise, and after each stage say what you did and what changed — don't run the whole pipeline silently. +**How to work:** read the setup, then propose a short plan and get a go-ahead before touching anything. Prepare `.env` and **pause for the user to verify it** before running. Run each stage with its logs shown and, after each, say what you did and what changed — don't run the pipeline silently. If a stage fails and the cause isn't clear from the normal logs, re-run it with `datapipe --debug … run` (or `--debug-sql` for SQL errors); debug is very verbose, so send it to a file and `grep` it (e.g. `datapipe --debug run > /tmp/dp_debug.log 2>&1; grep -nEi "error|traceback" /tmp/dp_debug.log`) rather than dumping it inline. ## Run on YOUR data - **Align across all three:** `SAM_TEXT_PROMPT` == the CVAT labels (`CVAT_BOX_LABEL` / `CVAT_POLYGON_LABEL`) == `HF_DATASET_LABEL` must all mean the same class — a mismatch runs clean but yields 0 useful results. From b5d34d450c1b2b132a9832b9ca818607db93714a Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Wed, 1 Jul 2026 21:27:50 +0300 Subject: [PATCH 11/32] tags-addon: add a reproducible demo-scenario recipe for agents Documents how to recreate the tag demo end-to-end: seed a non-trivial cat/dog set, build a darkened tagged scenario with inherited GT, train a baseline with the scenario excluded, retrain with it included, compute tag_metrics, and compare on the val holdout. Notes evaluating on final-epoch weights. --- .../skills/setup-e2e-template/tags-addon.md | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/.claude/skills/setup-e2e-template/tags-addon.md b/.claude/skills/setup-e2e-template/tags-addon.md index e309a855..1daa034a 100644 --- a/.claude/skills/setup-e2e-template/tags-addon.md +++ b/.claude/skills/setup-e2e-template/tags-addon.md @@ -66,3 +66,35 @@ GROUP BY 1,2,3; hang (the process sits in `Sl`/`Dl`). - **Don't carve up `image__subset`** to fake a "scenario" — the tag IS the slice; slice metrics by the tag and keep the split full, otherwise old and new models get measured on different holdouts. + +## Demo scenario — how to reproduce (for an agent) + +Goal: show that a model scores near-0 on a tagged scenario and, after the tagged images are added to +training, the metric on that scenario rises. Steps to recreate it end-to-end: + +1. **Base data.** Seed a non-trivial cat/dog set (not the 10-image smoke default): + `uv run python scripts/seed_sample_data.py --detection-limit --keypoints-limit 0`. Images are + downloaded from COCO — needs outbound internet; on a restricted node, fetch them elsewhere and + upload to MinIO. Read-only `/home` → set `DATAPIPE_CACHE_DIR` and `UV_*` under `/tmp`. +2. **Build the scenario.** Take a subset of the already-labelled cat/dog images, make **darkened + copies** (gamma ≈ 0.10 — 0.25 is too mild to actually stump the model), upload them, and **inherit + the GT boxes/labels from each source image** (identical pixel size → identical boxes, so no + annotation). Insert one `tag` row (e.g. `night`) and one `image__tag` row per darkened image; split + them ~¾ train / ¼ val. +3. **Baseline A — scenario NOT in training.** Keep the scenario's val images in `image__subset` as + `val`, but exclude its train images from `train`. Train through the repo's train flow → A scores + near-0 on the tag. +4. **Model B — scenario IN training.** Add the scenario's train images to `image__subset` `train`, then + retrain a fresh model. (Freeze is delta-gated: if it no-ops, bump those images' GT `update_ts` so it + re-cuts a dataset.) +5. **Metrics.** Compute per-image TP/FP/FN on val (predict, match GT at IoU≥0.5 per class), join + `image__tag`, and aggregate by `(detection_model_id, tag_id, subset_id)` into `tag_metrics`. +6. **Compare.** Read `tag_metrics` rows for `model=A` vs `model=B` at the **holdout** subset — B is + higher. The e2e split yields only `train`/`val`, so the holdout is `val` (include `test` too if your + split has one); ignore the `train` row (a model scores high on its own training data). For a visual, + open FiftyOne filtered to the tag and show B detecting where A is empty. + +**Evaluate on the trained (final-epoch) weights.** On a small validation set the pipeline's +"best epoch" selection latches onto an early, noisy metric peak and can publish an under-trained +checkpoint that predicts nothing — use the final weights when computing the demo metrics. If the +built-in metrics step errors on the injected data, compute the per-image metrics directly as in step 5. From 134404930b52581dfd4aef0c127d7da122ab0b8d Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Thu, 2 Jul 2026 10:37:53 +0300 Subject: [PATCH 12/32] Add datapipe-examples router skill Router that points to the right example + setup skill and lists the universal prerequisites (Postgres, GPU, annotation backend, uv env, HF auth). Carries the same ask-first / plan-first / logs / --debug guidance as the setup skills. --- .claude/skills/datapipe-examples/SKILL.md | 64 +++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 .claude/skills/datapipe-examples/SKILL.md diff --git a/.claude/skills/datapipe-examples/SKILL.md b/.claude/skills/datapipe-examples/SKILL.md new file mode 100644 index 00000000..166278e9 --- /dev/null +++ b/.claude/skills/datapipe-examples/SKILL.md @@ -0,0 +1,64 @@ +--- +name: datapipe-examples +description: > + Use when working in the epoch8 datapipe repo with the examples/* folders, when asked to run or + debug any datapipe example pipeline on your own data, or when unsure which example or setup skill + applies. +--- + +# datapipe examples — router + +Every example here is meant to run on YOUR data; each bundled demo dataset is only a smoke-test. Pick the example, then its setup skill, then set that skill's "Run on YOUR data" knobs first. + +Each `examples/*` pipeline has a dedicated setup skill — pick by what it does: + +| Example | What it does | Skill | +|---|---|---| +| `embedder_fiftyone` | DINOv2/DINOv3 image embeddings → FiftyOne UMAP + similarity | **setup-embedder-fiftyone** | +| `e2e_template/image_detection` | YOLO detection + Label Studio human-in-the-loop → train → FiftyOne | **setup-e2e-template** | +| `e2e_template/image_keypoints` | YOLO-pose keypoints + Label Studio → train → FiftyOne | **setup-e2e-template** | +| `sam_cvat` | SAM3 text-prompt boxes+masks → CVAT pre-annotations | **setup-sam-cvat** | + +## Ask first — don't assume (only the unresolved) +Clarify what's not obvious before acting — don't spin up services or pull data you don't need. Ask only the unresolved: +- **Demo or your own data?** · **Provision Postgres/services or reuse existing?** (e2e ships `docker compose`; embedder/sam need external Postgres) +- **Which Postgres + which database** for `DB_URL`? Never point it at an existing DB or drop in a `localhost` default without confirming. +- **Reuse an existing venv / `uv` env, or create a fresh one?** · **Which GPU?** (SAM3 >8 GB; DINOv2/YOLO ~8 GB) · **Annotation backend up?** (LS for e2e / external CVAT for sam) +- **Surface stage logs or run quiet?** · **(e2e) per-tag scenario metrics** (retrain new case, old vs new)? → `setup-e2e-template/tags-addon.md` + +## How to work +Read the setup, then propose a short plan and get a go-ahead before touching anything. Prepare `.env` and **pause for the user to verify it** before running. Run each stage with its logs shown and, after each, say what you did and what changed — don't run the pipeline silently. Trust the status table (`*_training_status`/`brain_status`), not the exit code. If a stage fails and the cause isn't clear from the normal logs, re-run it with `datapipe --debug … run` (or `--debug-sql` for SQL errors) sent to a file and `grep`ped, rather than dumping the verbose output inline. + +## Run on YOUR data (universal rule) +When you swap in your own subject, **align your class name across every place it appears** — text +prompt / label config, the label field, and any class filter. A mismatch runs fine but yields 0 +useful results. The per-example skill lists the exact knobs. + +## Universal prerequisites (every example) +1. **PostgreSQL** at `DB_URL`. embedder + sam_cvat need an EXTERNAL Postgres (they don't start one); + e2e_template bundles Postgres in its `docker compose`. Empty DB is fine; tables auto-create via + `datapipe db create-all`. `.env.example` default `...postgres:postgres@localhost:5432/postgres`. + External quick start: `docker run -d -e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres -e POSTGRES_DB=postgres -p 5432:5432 postgres:16` + (k8s pod / no docker → conda postgres, `initdb` as non-root with `--locale=C`.) +2. **A GPU big enough** (`nvidia-smi`): DINOv2/YOLO fit ~8 GB; SAM3 wants >8 GB (OOM'd on an 8 GB Pascal in our tests). +3. **Annotation backend, if used:** e2e_template ships Label Studio in its `docker compose`; sam_cvat + needs an external CVAT you provide (URL + creds + a project whose labels match its config). +4. **`uv` + Python ≥3.10,<3.13.** Each example has a `pyproject.toml` → `uv sync` (cu124 torch pinned, + CUDA OOTB; no manual venv/pip). Read-only/small `$HOME`: `export UV_CACHE_DIR=/tmp/uvcache HF_HOME=/tmp/hf`. +5. **Human-in-the-loop** (e2e_template, sam_cvat): a human reviews and marks tasks completed before + the pipeline advances. + +## HF auth (only for GATED models) +Needed for `dinov3_*` (embedder, off by default) and `sam3` (sam_cvat). Accept the model license on +its HF page, then `export HF_TOKEN=...` with **gated-repo read access**. A plain token → `403`, often +masked as `LocalEntryNotFoundError: check your connection`. Public models (dinov2, YOLO) need no token. + +## Universal run shape +```bash +cp .env.example .env # set DB_URL and the example's data + backend vars +uv sync +datapipe db create-all # auto-creates all tables +datapipe run # or: datapipe step --labels stage= run +``` +Datapipe caches by content: re-running does little work unless inputs or tracked config change. +Then follow the per-example skill — set its **Run on YOUR data** knobs first. From 0b5b122c2e9e51f57f0987f9873c171731e85d70 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Thu, 2 Jul 2026 13:30:16 +0300 Subject: [PATCH 13/32] e2e_template: bump default seed to 120 cat/dog images; finalize tag demo recipe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 10-image smoke default is too small — the pipeline's val metric is noisy and its built-in metrics/checkpoint selection misbehave. Default --detection-limit to 120, at which a clean from-scratch run trains, computes metrics natively, and shows the tag arc (baseline low on the tagged scenario, higher after the tag is added to training). Pin the demo-scenario counts in tags-addon.md (120 base, ~40 night 30/10, gamma 0.10) and soften the checkpoint-selection note now that native metrics work. --- .../skills/setup-e2e-template/tags-addon.md | 27 ++++++++++--------- .../e2e_template/scripts/seed_sample_data.py | 2 +- 2 files changed, 15 insertions(+), 14 deletions(-) diff --git a/.claude/skills/setup-e2e-template/tags-addon.md b/.claude/skills/setup-e2e-template/tags-addon.md index 1daa034a..8281ff0a 100644 --- a/.claude/skills/setup-e2e-template/tags-addon.md +++ b/.claude/skills/setup-e2e-template/tags-addon.md @@ -72,15 +72,16 @@ GROUP BY 1,2,3; Goal: show that a model scores near-0 on a tagged scenario and, after the tagged images are added to training, the metric on that scenario rises. Steps to recreate it end-to-end: -1. **Base data.** Seed a non-trivial cat/dog set (not the 10-image smoke default): - `uv run python scripts/seed_sample_data.py --detection-limit --keypoints-limit 0`. Images are - downloaded from COCO — needs outbound internet; on a restricted node, fetch them elsewhere and - upload to MinIO. Read-only `/home` → set `DATAPIPE_CACHE_DIR` and `UV_*` under `/tmp`. -2. **Build the scenario.** Take a subset of the already-labelled cat/dog images, make **darkened - copies** (gamma ≈ 0.10 — 0.25 is too mild to actually stump the model), upload them, and **inherit - the GT boxes/labels from each source image** (identical pixel size → identical boxes, so no - annotation). Insert one `tag` row (e.g. `night`) and one `image__tag` row per darkened image; split - them ~¾ train / ¼ val. +1. **Base data.** Seed a non-trivial cat/dog set (the 10-image smoke default is too small — the + metric on a tiny val is noisy). `scripts/seed_sample_data.py` defaults to `--detection-limit 120`, + which is enough for the pipeline's own metrics to work; `--keypoints-limit 0` for a detection-only + demo. Images download from COCO — needs outbound internet; on a restricted node fetch them + elsewhere and upload to MinIO. Read-only `/home` → set `DATAPIPE_CACHE_DIR` and `UV_*` under `/tmp`. +2. **Build the scenario.** Take ~40 of the already-labelled cat/dog images, make **darkened copies** + (gamma ≈ 0.10 — 0.25 is too mild to actually stump the model), upload them, and **inherit the GT + boxes/labels from each source image** (identical pixel size → identical boxes, so no annotation). + Insert one `tag` row (e.g. `night`) and one `image__tag` row per darkened image; split them ~30 + train / ~10 val. 3. **Baseline A — scenario NOT in training.** Keep the scenario's val images in `image__subset` as `val`, but exclude its train images from `train`. Train through the repo's train flow → A scores near-0 on the tag. @@ -94,7 +95,7 @@ training, the metric on that scenario rises. Steps to recreate it end-to-end: split has one); ignore the `train` row (a model scores high on its own training data). For a visual, open FiftyOne filtered to the tag and show B detecting where A is empty. -**Evaluate on the trained (final-epoch) weights.** On a small validation set the pipeline's -"best epoch" selection latches onto an early, noisy metric peak and can publish an under-trained -checkpoint that predicts nothing — use the final weights when computing the demo metrics. If the -built-in metrics step errors on the injected data, compute the per-image metrics directly as in step 5. +**Note on checkpoint selection.** With enough data the pipeline's built-in metrics compute fine and the +tag arc shows up in `detection_model_train__metrics_on_subset` natively. On a *small* validation set the +"best epoch" pick can latch onto an early, noisy metric peak and publish an under-trained checkpoint — if +a model looks suspiciously empty, sanity-check against the final-epoch weights. diff --git a/examples/e2e_template/scripts/seed_sample_data.py b/examples/e2e_template/scripts/seed_sample_data.py index b25df7ef..f8d6b910 100644 --- a/examples/e2e_template/scripts/seed_sample_data.py +++ b/examples/e2e_template/scripts/seed_sample_data.py @@ -162,7 +162,7 @@ def upload_images(local_paths: list[Path]) -> None: def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser(description="Download sample COCO images and upload them to local MinIO.") - parser.add_argument("--detection-limit", type=int, default=10, help="Number of cat/dog images") + parser.add_argument("--detection-limit", type=int, default=120, help="Number of cat/dog images") parser.add_argument("--keypoints-limit", type=int, default=10, help="Number of person keypoint images") parser.add_argument("--skip-download", action="store_true", help="Only upload files already in sample_data/") parser.add_argument("--skip-upload", action="store_true", help="Only download images locally") From 1bdc53b5faa952b9d2cb683d6970a70f0700f4b4 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 6 Jul 2026 08:11:11 +0300 Subject: [PATCH 14/32] e2e skill: ask up front whether to annotate in Label Studio or inject ready-made GT Skipping the human annotation step (injecting COCO/inherited ground truth) vs a real Label Studio pass is a demo-shaping choice, so surface it in the Ask-first block and flag it in the tag demo recipe rather than deciding silently. --- .claude/skills/setup-e2e-template/SKILL.md | 2 +- .claude/skills/setup-e2e-template/tags-addon.md | 7 ++++++- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/.claude/skills/setup-e2e-template/SKILL.md b/.claude/skills/setup-e2e-template/SKILL.md index 3e1997ad..a625a904 100644 --- a/.claude/skills/setup-e2e-template/SKILL.md +++ b/.claude/skills/setup-e2e-template/SKILL.md @@ -9,7 +9,7 @@ description: > This skill = run the YOLO/Label-Studio detection/keypoints pipeline on YOUR images. The COCO seed sample is just a smoke-test; the real goal is your own images — set the knobs below first. -**Ask first — don't assume (only the unresolved):** demo or your own data? bring up the bundled `docker compose` services or reuse existing? **which Postgres + which database** for `DB_URL` — never point it at an existing DB without confirming; reuse an existing venv / `uv` env or create a fresh one? GPU? per-tag scenario metrics ([tags-addon.md](tags-addon.md))? surface stage logs or run quiet? +**Ask first — don't assume (only the unresolved):** demo or your own data? bring up the bundled `docker compose` services or reuse existing? **which Postgres + which database** for `DB_URL` — never point it at an existing DB without confirming; reuse an existing venv / `uv` env or create a fresh one? GPU? **annotate for real in Label Studio, or inject ready-made ground truth** (reuse existing labels / a labelled dataset) to skip the human `annotation` step? per-tag scenario metrics ([tags-addon.md](tags-addon.md))? surface stage logs or run quiet? **How to work:** read the setup, then propose a short plan and get a go-ahead before touching anything. Prepare `.env` and **pause for the user to verify it** before running. Run each stage with its logs shown and, after each, say what you did and what changed — don't run the pipeline silently. If a stage fails and the cause isn't clear from the normal logs, re-run it with `datapipe --debug … run` (or `--debug-sql` for SQL errors); debug is very verbose, so send it to a file and `grep` it (e.g. `datapipe --debug run > /tmp/dp_debug.log 2>&1; grep -nEi "error|traceback" /tmp/dp_debug.log`) rather than dumping it inline. diff --git a/.claude/skills/setup-e2e-template/tags-addon.md b/.claude/skills/setup-e2e-template/tags-addon.md index 8281ff0a..ee6a1196 100644 --- a/.claude/skills/setup-e2e-template/tags-addon.md +++ b/.claude/skills/setup-e2e-template/tags-addon.md @@ -70,7 +70,12 @@ GROUP BY 1,2,3; ## Demo scenario — how to reproduce (for an agent) Goal: show that a model scores near-0 on a tagged scenario and, after the tagged images are added to -training, the metric on that scenario rises. Steps to recreate it end-to-end: +training, the metric on that scenario rises. Steps to recreate it end-to-end. + +> This recipe **injects ready-made ground truth** (COCO labels / inherited boxes) and **skips the +> Label Studio `annotation` step** — so it runs unattended. Confirm this is acceptable up front (see +> the skill's "Ask first" line): if the demo must show real human annotation in Label Studio, that +> stays a manual step and isn't automated here. 1. **Base data.** Seed a non-trivial cat/dog set (the 10-image smoke default is too small — the metric on a tiny val is noisy). `scripts/seed_sample_data.py` defaults to `--detection-limit 120`, From a7f8f4617587fd1fc1b2ca4661a479ebc0ef70cb Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 6 Jul 2026 08:25:27 +0300 Subject: [PATCH 15/32] tags-addon: document wiring tags into the pipeline for the UI graph Add the concrete recipe to surface tags as a graph node in datapipe-app: catalog entries (tag/image__tag/tag_metrics) + an aggregation step, and note it MUST be a DatatableBatchTransform (whole-table SQL group-by), not a chunked BatchTransform, since it aggregates many images into one (model,tag,subset) row. These pipeline edits stay docs-only / applied physically for the demo, not committed to the example. --- .../skills/setup-e2e-template/tags-addon.md | 26 +++++++++++++++---- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/.claude/skills/setup-e2e-template/tags-addon.md b/.claude/skills/setup-e2e-template/tags-addon.md index ee6a1196..42201835 100644 --- a/.claude/skills/setup-e2e-template/tags-addon.md +++ b/.claude/skills/setup-e2e-template/tags-addon.md @@ -25,15 +25,31 @@ detection and checks out. (as in every e2e table). Both read via the standard `UpdateExternalTable`; who writes them is out of scope (UI / manual INSERT / script). - `tag_metrics(detection_model_id PK, tag_id PK, subset_id PK, calc__precision, calc__recall, - calc__f1_score, support, images_support)` — a new `BatchTransform` step. - **Source — NOT the raw `detection_predictions`** (that's pre-annotation; the trained model's output is - in `detection_prediction_train`). Cleanest: take the already-computed **per-image** - `detection_model_train__metrics_on_image` (`image_name, detection_model_id, subset_id, + calc__f1_score, support, images_support)` — a new aggregation step. + **Source — NOT the raw predictions** (that's pre-annotation). Take the already-computed **per-image** + metrics table (`…__metrics_on_image`: `image_name, detection_model_id, subset_id, calc__TP/FP/FN/support`), join `image__tag` on `image_name`, and **re-aggregate by - `(detection_model_id, tag_id, subset_id)`** — same as `count_..._metrics_on_subset`, just with + `(detection_model_id, tag_id, subset_id)`** — same as `count_…_metrics_on_subset`, just with `tag_id` in the GROUP BY. +- ⚠️ **Use `DatatableBatchTransform` (whole-table SQL group-by), not a plain `BatchTransform`.** This is + an aggregation (many images → one `(model,tag,subset)` row); a chunked `BatchTransform` splits a tag's + images across chunks and computes wrong sums. Mirror the existing `count_…_metrics_on_subset` step + (it's a `DatatableBatchTransform`) and add `tag_id` to its grouping. - **Leave alone:** `image__subset`/split, `..._metrics_on_subset`, `FindBestModel(subset_id="val")`, training. +## Wire it into the pipeline (so it shows in the datapipe-app UI) +To make tags a node in the UI graph (not just a side script), add to the example code: +1. **`data.py` → `catalog`:** add `tag`, `image__tag`, `tag_metrics` as `Table(TableStoreDB(...))` (same + pattern as the other tables). `tag`/`image__tag` are external inputs (written by UI / INSERT / script); + `tag_metrics` is the step's output. +2. **`app.py` → `pipeline`:** add the aggregation step (see ⚠️ above) reading `…__metrics_on_image` + + `image__tag`, writing `tag_metrics`, `labels=[("stage","count-metrics")]` so it runs with metrics. +3. Populate `image__tag`, then run `stage=train` (or `stage=count-metrics`) — the UI graph then shows + the tag tables + step, and `tag_metrics` fills per model. + +> Keep these pipeline edits **out of the committed example** (docs-only, per this recipe) — apply them +> physically where you run the demo, but the repo's example code stays unchanged. + ## How to use 1. Label the scenario images → rows in `image__tag` (a tag, e.g. `dark_room`). 2. Run the pipeline **as usual** — tagged images go through the normal split, part to `train` From a77e2fb303b5a10dd465ef9e6ef7a374397fc59c Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 6 Jul 2026 08:27:06 +0300 Subject: [PATCH 16/32] tags-addon: frame pipeline wiring as a per-consumer extension in their own repo --- .claude/skills/setup-e2e-template/tags-addon.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.claude/skills/setup-e2e-template/tags-addon.md b/.claude/skills/setup-e2e-template/tags-addon.md index 42201835..8ebbb6f2 100644 --- a/.claude/skills/setup-e2e-template/tags-addon.md +++ b/.claude/skills/setup-e2e-template/tags-addon.md @@ -47,8 +47,9 @@ To make tags a node in the UI graph (not just a side script), add to the example 3. Populate `image__tag`, then run `stage=train` (or `stage=count-metrics`) — the UI graph then shows the tag tables + step, and `tag_metrics` fills per model. -> Keep these pipeline edits **out of the committed example** (docs-only, per this recipe) — apply them -> physically where you run the demo, but the repo's example code stays unchanged. +> This is a **per-consumer** extension: the upstream datapipe example stays unchanged; whoever needs +> tags applies these edits **in their own project/repo** (following this recipe) and commits them there. +> Keep them out of the committed example. ## How to use 1. Label the scenario images → rows in `image__tag` (a tag, e.g. `dark_room`). From d4428e6156454c0d6ac8fc3aa92ad8e66629bc7a Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 6 Jul 2026 11:27:43 +0300 Subject: [PATCH 17/32] e2e skill: document injecting ground truth to skip Label Studio annotation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit For unattended/demo runs without a human annotator: write image__ground_truth + image__subset directly, but following the pipeline conventions — image_name as list_s3_images emits it (basename relative to INPUT_IMAGES_DIR, no invented prefix), labels matching DETECTION_CLASSES casing (Cat/Dog not cat/dog), via DataStore so _meta stays consistent. Captures the two mismatches that silently break freeze/metrics. --- .claude/skills/setup-e2e-template/SKILL.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/.claude/skills/setup-e2e-template/SKILL.md b/.claude/skills/setup-e2e-template/SKILL.md index a625a904..7dd2bb38 100644 --- a/.claude/skills/setup-e2e-template/SKILL.md +++ b/.claude/skills/setup-e2e-template/SKILL.md @@ -48,6 +48,20 @@ fiftyone app launch datapipe_detection_e2e # or datapipe_keypoints_e2e ``` Train uses `yolov8n*.pt` (imgsz 320, 30 ep); pre-annotation fallback `yolo11n*.pt`; best on `subset_id=val`. +## Skip annotation — inject ground truth (unattended / demo) +No human in Label Studio? You can bypass `annotation`/`ls-sync` by writing `image__ground_truth` + +`image__subset` directly, then run `stage=train`. **The injected rows MUST follow the pipeline's own +conventions, or the freeze join silently yields nothing / classes don't match:** +- **`image_name` must equal what `list_s3_images` emits** — the key **relative to `INPUT_IMAGES_DIR` + (`$DATAPIPE_E2E_DIR/images`)**, i.e. the plain object name (`000000008458.jpg`). Don't invent a + prefixed form — if `s3_images` has `X` and your GT has `pfx___X`, the `GT ⋈ subset ⋈ s3_images` join + is empty → `freeze_dataset` fails with `No ground truth`. +- **`labels` must match `DETECTION_CLASSES` casing** (e.g. `Cat`/`Dog`, not COCO lowercase `cat`/`dog`) + — otherwise predictions and GT land in *different* classes and every metric is 0. +- **`bboxes`** = pixel `[x1,y1,x2,y2]`; assign `image__subset.subset_id` (`train`/`val`) yourself. +- Write via datapipe `DataStore`/`UpdateExternalTable` (keeps `*_meta` in sync) — not raw SQL `UPDATE` + on PK columns. Then `datapipe step --labels=stage=train run` (skip `annotation`/`ls-sync`). + ## Add-on upgrade — tags for per-scenario metrics An **optional layer on top of the base pipeline** (not needed to run it). New case (e.g. dark-room pallets) → tag those images, let part flow into training, and measure the model **on that scenario From 787d1f161dfb1b007a2602bbe55c86a94506b0cd Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 6 Jul 2026 11:32:15 +0300 Subject: [PATCH 18/32] tags-addon: add live demo walkthrough (skill -> inject GT -> tags -> UI -> train x2 -> 2 models + metrics) --- .../skills/setup-e2e-template/tags-addon.md | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/.claude/skills/setup-e2e-template/tags-addon.md b/.claude/skills/setup-e2e-template/tags-addon.md index 8ebbb6f2..c27511ed 100644 --- a/.claude/skills/setup-e2e-template/tags-addon.md +++ b/.claude/skills/setup-e2e-template/tags-addon.md @@ -121,3 +121,27 @@ training, the metric on that scenario rises. Steps to recreate it end-to-end. tag arc shows up in `detection_model_train__metrics_on_subset` natively. On a *small* validation set the "best epoch" pick can latch onto an early, noisy metric peak and publish an under-trained checkpoint — if a model looks suspiciously empty, sanity-check against the final-epoch weights. + +## Live demo walkthrough (with the datapipe-app UI) +The narrated flow for showing this to someone — you drive the setup by chatting, then click in the UI: + +1. **Invoke the skill.** `/setup-e2e-template`, and say *"this is a demo"*. It brings up the services + and seeds demo cat/dog data (COCO → MinIO), i.e. the images that would be annotated in Label Studio. +2. **Skill asks the key question:** *annotate for real in Label Studio, or inject ready-made GT?* For a + live demo pick **inject** — it writes GT directly (following the conventions in the skill's "Skip + annotation" section: basename `image_name`, `Cat`/`Dog` casing) so no human labelling is needed. +3. **Set up tags (don't skip this).** Before training, define the scenario: darken a subset → `tag` + (`night`) + `image__tag`, and add the `tag_metrics` step (see this file's "Wire it into the + pipeline"). Now tags are nodes in the UI graph. +4. **Bring up the UI:** `datapipe --pipeline app:app api` (agent mode + `DATAPIPE_APP_PIPELINE_ID`), + port-forward, open it. Show the pipeline graph including the tag tables/step. +5. **Train model A (baseline)** from the UI: run the **`train`** stage (not the whole pipeline — that + hits Label Studio). This is the model *without* the tagged scenario in training. +6. **Add the scenario to training** (tag-train rows into `image__subset` `train`) and run **`train`** + again → **model B**. Then run **`count-metrics`** / the tag step. +7. **Payoff:** the UI now shows **two models** and their metrics; `tag_metrics` on the tag's `val` + holdout is ~0 for A and higher for B. Optionally FiftyOne: B detects on the tagged images where A is empty. + +> Trigger runs **by stage** from the UI (there's a per-stage run menu / graph-node run), never "Run +> pipeline" (that runs `annotation` → Label Studio). Training runs on the API process; on this WIP +> branch it needs `pi_heif` and a pre-AVX2-safe `polars-lts-cpu` in the venv (see the api issues log). From 7d716a597c460e3d308859e661bd5eb608bc2f84 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 6 Jul 2026 12:08:31 +0300 Subject: [PATCH 19/32] tags-addon: add 'clean slate before demo' step (test data only; clear pipeline schema AND public observability run history separately) --- .claude/skills/setup-e2e-template/tags-addon.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/.claude/skills/setup-e2e-template/tags-addon.md b/.claude/skills/setup-e2e-template/tags-addon.md index c27511ed..d0378bb0 100644 --- a/.claude/skills/setup-e2e-template/tags-addon.md +++ b/.claude/skills/setup-e2e-template/tags-addon.md @@ -125,6 +125,17 @@ a model looks suspiciously empty, sanity-check against the final-epoch weights. ## Live demo walkthrough (with the datapipe-app UI) The narrated flow for showing this to someone — you drive the setup by chatting, then click in the UI: +0. **Start from a clean slate (TEST DATA ONLY).** Before demoing, wipe prior state so the graph, runs + and metrics are pristine — otherwise old models and "Recent runs" from earlier attempts show up and + confuse the story. Two independent places to clear: + - **pipeline data** — the schema (e.g. `datapipe_e2e_detection`) + MinIO objects + local train + outputs, then `datapipe db create-all`; + - **observability run history** — lives **separately** in the `public` schema + (`pipeline_runs`, `pipeline_run_steps`, `pipeline_run_logs`, `analytics_training_runs`); dropping + the pipeline schema does NOT clear it. Truncate these to empty "Recent runs". + ⚠️ Do this **only on a throwaway/demo database**, never on real data. (Note: this Postgres is shared + with Label Studio — scope truncates to the datapipe-app tables above; don't `CASCADE` into LS tables.) + 1. **Invoke the skill.** `/setup-e2e-template`, and say *"this is a demo"*. It brings up the services and seeds demo cat/dog data (COCO → MinIO), i.e. the images that would be annotated in Label Studio. 2. **Skill asks the key question:** *annotate for real in Label Studio, or inject ready-made GT?* For a From b2676ef16565d1ec9d8ee57e44952cd79cb70b12 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 6 Jul 2026 13:14:36 +0300 Subject: [PATCH 20/32] Add detection_tags example (tags-native, no LS/FiftyOne) + skill; lowercase classes; drop tags-addon New self-contained example examples/detection_tags: YOLO detection with tags built into the pipeline (tag/image__tag/tag_metrics + a compute_tag_metrics BatchTransform), NO Label Studio and NO FiftyOne, ground truth injected (COCO labels, lowercase cat/dog), postgres+minio only, deps pin polars-lts-cpu + pi-heif so it runs from scratch. Two-step data load (base batch, tagged scenario batch) via scripts/load_batch.py. New setup-detection-tags skill + router row. e2e_template LABEL_CONFIG lowercased (cat/dog) to match COCO; the old tags-addon.md recipe is removed in favour of the real example. --- .claude/skills/datapipe-examples/SKILL.md | 3 +- .claude/skills/setup-detection-tags/SKILL.md | 67 ++++++++ .claude/skills/setup-e2e-template/SKILL.md | 13 +- .../skills/setup-e2e-template/tags-addon.md | 158 ------------------ examples/detection_tags/.env.example | 10 ++ examples/detection_tags/README.md | 46 +++++ examples/detection_tags/detection/__init__.py | 0 examples/detection_tags/detection/app.py | 130 ++++++++++++++ examples/detection_tags/detection/config.py | 85 ++++++++++ examples/detection_tags/detection/data.py | 65 +++++++ examples/detection_tags/detection/steps.py | 64 +++++++ examples/detection_tags/docker-compose.yml | 42 +++++ examples/detection_tags/pyproject.toml | 33 ++++ examples/detection_tags/scripts/load_batch.py | 143 ++++++++++++++++ .../e2e_template/image_detection/config.py | 4 +- 15 files changed, 695 insertions(+), 168 deletions(-) create mode 100644 .claude/skills/setup-detection-tags/SKILL.md delete mode 100644 .claude/skills/setup-e2e-template/tags-addon.md create mode 100644 examples/detection_tags/.env.example create mode 100644 examples/detection_tags/README.md create mode 100644 examples/detection_tags/detection/__init__.py create mode 100644 examples/detection_tags/detection/app.py create mode 100644 examples/detection_tags/detection/config.py create mode 100644 examples/detection_tags/detection/data.py create mode 100644 examples/detection_tags/detection/steps.py create mode 100644 examples/detection_tags/docker-compose.yml create mode 100644 examples/detection_tags/pyproject.toml create mode 100644 examples/detection_tags/scripts/load_batch.py diff --git a/.claude/skills/datapipe-examples/SKILL.md b/.claude/skills/datapipe-examples/SKILL.md index 166278e9..7e43b7c7 100644 --- a/.claude/skills/datapipe-examples/SKILL.md +++ b/.claude/skills/datapipe-examples/SKILL.md @@ -18,13 +18,14 @@ Each `examples/*` pipeline has a dedicated setup skill — pick by what it does: | `e2e_template/image_detection` | YOLO detection + Label Studio human-in-the-loop → train → FiftyOne | **setup-e2e-template** | | `e2e_template/image_keypoints` | YOLO-pose keypoints + Label Studio → train → FiftyOne | **setup-e2e-template** | | `sam_cvat` | SAM3 text-prompt boxes+masks → CVAT pre-annotations | **setup-sam-cvat** | +| `detection_tags` | YOLO detection + **tags** (per-scenario metrics), no Label Studio / FiftyOne, GT injected | **setup-detection-tags** | ## Ask first — don't assume (only the unresolved) Clarify what's not obvious before acting — don't spin up services or pull data you don't need. Ask only the unresolved: - **Demo or your own data?** · **Provision Postgres/services or reuse existing?** (e2e ships `docker compose`; embedder/sam need external Postgres) - **Which Postgres + which database** for `DB_URL`? Never point it at an existing DB or drop in a `localhost` default without confirming. - **Reuse an existing venv / `uv` env, or create a fresh one?** · **Which GPU?** (SAM3 >8 GB; DINOv2/YOLO ~8 GB) · **Annotation backend up?** (LS for e2e / external CVAT for sam) -- **Surface stage logs or run quiet?** · **(e2e) per-tag scenario metrics** (retrain new case, old vs new)? → `setup-e2e-template/tags-addon.md` +- **Surface stage logs or run quiet?** · **Per-tag scenario metrics** (retrain new case, old vs new)? → the `detection_tags` example / **setup-detection-tags** ## How to work Read the setup, then propose a short plan and get a go-ahead before touching anything. Prepare `.env` and **pause for the user to verify it** before running. Run each stage with its logs shown and, after each, say what you did and what changed — don't run the pipeline silently. Trust the status table (`*_training_status`/`brain_status`), not the exit code. If a stage fails and the cause isn't clear from the normal logs, re-run it with `datapipe --debug … run` (or `--debug-sql` for SQL errors) sent to a file and `grep`ped, rather than dumping the verbose output inline. diff --git a/.claude/skills/setup-detection-tags/SKILL.md b/.claude/skills/setup-detection-tags/SKILL.md new file mode 100644 index 00000000..b0483d13 --- /dev/null +++ b/.claude/skills/setup-detection-tags/SKILL.md @@ -0,0 +1,67 @@ +--- +name: setup-detection-tags +description: > + Use when setting up or running examples/detection_tags — a self-contained datapipe detection + demo built around tags (per-scenario metrics), with no Label Studio and no FiftyOne, deployed + from scratch; or when demoing "add a tagged batch, retrain, watch the tag metric rise". +--- + +# detection_tags (tags demo — no Label Studio, no FiftyOne) + +Minimal detection pipeline whose whole point is **tags**: train a baseline, load a **tagged scenario +batch**, retrain, and watch the metric on that tag rise in a `tag_metrics` table. Ground truth is +**injected** (COCO labels, lowercase `cat`/`dog`) — no human annotation — so it runs unattended. + +**Ask first — don't assume (only the unresolved):** which Postgres + which database for `DB_URL` +(don't target an existing DB or drop a `localhost` default without confirming); reuse an existing +venv/`uv` env or a fresh one? GPU available (training is GPU)? surface stage logs or run quiet? + +**How to work:** propose a short plan and get a go-ahead; show each stage's logs and report what +changed; trust the `*_training_status` table, not the exit code. On an unclear failure re-run with +`datapipe --debug … run` sent to a file + `grep`, not inline. + +## Deploy from scratch +```bash +cp .env.example .env && set -a && source .env && set +a # DB_URL, S3/MinIO, DATAPIPE_TAGS_DIR +docker compose up -d # postgres + minio ONLY (no mongo, no Label Studio) +uv sync # cu124 torch + polars-lts-cpu + pi-heif baked in +cd detection && datapipe db create-all +``` + +## Two-step data load (no annotation) +```bash +# from examples/detection_tags/detection +python ../scripts/load_batch.py --n 120 # batch 1: base cat/dog +python ../scripts/load_batch.py --n 40 --tag night --darken 0.1 # batch 2: tagged scenario +``` +`load_batch.py` downloads COCO cat/dog, uploads to MinIO, injects GT (+ a tag for batch 2). Labels are +lowercase `cat`/`dog`; `image_name` = the object basename (what `list_s3_images` emits) so the joins line up. + +## Run +```bash +datapipe run # ingest → split → freeze → train → inference → metrics → tag_metrics +# or by stage: datapipe step --labels=stage=train run +``` + +## The payoff +`tag_metrics(detection_model_id, tag_id, subset_id)` → precision/recall/f1. Compare the baseline +(trained before batch 2) vs the retrained model at `tag=night, subset=val` — recall on the tag rises +once the tagged batch is in training. `tag`/`image__tag` are external inputs (from `load_batch.py`); +`tag_metrics` is a real pipeline step (`steps.compute_tag_metrics`, `transform_keys=["detection_model_id"]` +so the aggregation is correct). + +## Two-model demo (baseline vs retrained) +1. Load batch 1 → `datapipe step --labels=stage=train run` → model A (no tag in training). +2. Load batch 2 (`--tag night --darken 0.1`) → `datapipe step --labels=stage=train run` → model B. +3. Read `tag_metrics`: `night/val` ≈ low for A, higher for B. + +## Troubleshooting (may already be fixed — verify against current files) +- **`SIGILL` / `Illegal instruction` in the training subprocess** → `polars` built for a CPU newer + than the host (pre-AVX2). This example pins `polars-lts-cpu`; if it still happens, reinstall it. +- **`No labels found` / every image "corrupt: No module named 'pi_heif'`** → ultralytics image + verification needs `pi_heif` (pinned here); reinstall if missing. +- **`No ground truth` at freeze** → injected `image__ground_truth.image_name` must equal what + `list_s3_images` emits (object basename); a prefixed key makes the join empty. +- **Metrics 0 on a trained model** → tiny/noisy val makes "best epoch" latch onto an early + checkpoint; use enough data (default batches) and the shipped 50-epoch config. +- **Training exits 0 but no model** → datapipe swallows step errors; check `detection_training_status`. diff --git a/.claude/skills/setup-e2e-template/SKILL.md b/.claude/skills/setup-e2e-template/SKILL.md index 7dd2bb38..112cc961 100644 --- a/.claude/skills/setup-e2e-template/SKILL.md +++ b/.claude/skills/setup-e2e-template/SKILL.md @@ -9,7 +9,7 @@ description: > This skill = run the YOLO/Label-Studio detection/keypoints pipeline on YOUR images. The COCO seed sample is just a smoke-test; the real goal is your own images — set the knobs below first. -**Ask first — don't assume (only the unresolved):** demo or your own data? bring up the bundled `docker compose` services or reuse existing? **which Postgres + which database** for `DB_URL` — never point it at an existing DB without confirming; reuse an existing venv / `uv` env or create a fresh one? GPU? **annotate for real in Label Studio, or inject ready-made ground truth** (reuse existing labels / a labelled dataset) to skip the human `annotation` step? per-tag scenario metrics ([tags-addon.md](tags-addon.md))? surface stage logs or run quiet? +**Ask first — don't assume (only the unresolved):** demo or your own data? bring up the bundled `docker compose` services or reuse existing? **which Postgres + which database** for `DB_URL` — never point it at an existing DB without confirming; reuse an existing venv / `uv` env or create a fresh one? GPU? **annotate for real in Label Studio, or inject ready-made ground truth** (reuse existing labels / a labelled dataset) to skip the human `annotation` step? surface stage logs or run quiet? (per-tag scenario metrics live in a separate example → **setup-detection-tags**) **How to work:** read the setup, then propose a short plan and get a go-ahead before touching anything. Prepare `.env` and **pause for the user to verify it** before running. Run each stage with its logs shown and, after each, say what you did and what changed — don't run the pipeline silently. If a stage fails and the cause isn't clear from the normal logs, re-run it with `datapipe --debug … run` (or `--debug-sql` for SQL errors); debug is very verbose, so send it to a file and `grep` it (e.g. `datapipe --debug run > /tmp/dp_debug.log 2>&1; grep -nEi "error|traceback" /tmp/dp_debug.log`) rather than dumping it inline. @@ -62,12 +62,11 @@ conventions, or the freeze join silently yields nothing / classes don't match:** - Write via datapipe `DataStore`/`UpdateExternalTable` (keeps `*_meta` in sync) — not raw SQL `UPDATE` on PK columns. Then `datapipe step --labels=stage=train run` (skip `annotation`/`ls-sync`). -## Add-on upgrade — tags for per-scenario metrics -An **optional layer on top of the base pipeline** (not needed to run it). New case (e.g. dark-room -pallets) → tag those images, let part flow into training, and measure the model **on that scenario -separately** (old vs new) without touching the split. Bolt-on recipe (catalog `tag`/`image__tag` + -one `tag_metrics` step; example logic unchanged): [tags-addon.md](tags-addon.md). Verified end-to-end — -`tag_metrics` shows the retrained model gaining recall on the tagged scenario. +## Per-scenario tag metrics → see the dedicated example +Want to tag a scenario (e.g. dark-room pallets), add it to training, and measure the model on that +scenario separately (baseline vs retrained)? That lives as its own self-contained example — +`examples/detection_tags` (`tag`/`image__tag`/`tag_metrics` built into the pipeline, **no Label Studio +or FiftyOne**, ground truth injected). Use the **setup-detection-tags** skill. ## Troubleshooting (may already be fixed — verify against current files) - **No model after `train`, exit 0** → datapipe swallows step errors; check `detection_training_status`, not the exit code. diff --git a/.claude/skills/setup-e2e-template/tags-addon.md b/.claude/skills/setup-e2e-template/tags-addon.md deleted file mode 100644 index d0378bb0..00000000 --- a/.claude/skills/setup-e2e-template/tags-addon.md +++ /dev/null @@ -1,158 +0,0 @@ -# e2e_template — add-on: tags for per-scenario metrics - -**An optional layer on top of the base pipeline** (not needed to run it — a bolt-on you add when you -want it). - -**When you need it.** A new case shows up (e.g. "the model misses pallets in dark rooms"). You want -to: (1) label those images with a **tag**, (2) let part of them flow into training, (3) **measure the -model on that scenario separately** and compare old vs new — without touching the normal train/val. - -This is a **datapipe-native recipe**. Below: what to add and how. It was run end-to-end on e2e -detection and checks out. - -## Principle -- **Don't touch the split.** Scenario images get a tag and flow through the normal `image__subset` → - part lands in `train` (the model learns them = "added to training"), part in the holdout. - **"Add to training" = just tag the new images** — no extra logic. -- **A tag is just a label** (`image__tag`, many tags per image); it does NOT affect split/training. -- **Slice metrics by `(tag_id, subset_id)`** in a separate `tag_metrics` table; standard metrics untouched. -- **Old vs new model** — compare `tag_metrics` rows by eye (no automated gate). -- ⚠️ In the example the split yields only **`train`/`val`** (no `test`): holdout = `val` (or do a 3-way split). - -## What to add (catalog + one step; example logic unchanged) -- `tag(tag_id PK, name)` — tag dictionary. -- `image__tag(image_name PK, tag_id PK)` — tags per image, **many per image**. Key = `image_name` - (as in every e2e table). Both read via the standard `UpdateExternalTable`; who writes them is out of - scope (UI / manual INSERT / script). -- `tag_metrics(detection_model_id PK, tag_id PK, subset_id PK, calc__precision, calc__recall, - calc__f1_score, support, images_support)` — a new aggregation step. - **Source — NOT the raw predictions** (that's pre-annotation). Take the already-computed **per-image** - metrics table (`…__metrics_on_image`: `image_name, detection_model_id, subset_id, - calc__TP/FP/FN/support`), join `image__tag` on `image_name`, and **re-aggregate by - `(detection_model_id, tag_id, subset_id)`** — same as `count_…_metrics_on_subset`, just with - `tag_id` in the GROUP BY. -- ⚠️ **Use `DatatableBatchTransform` (whole-table SQL group-by), not a plain `BatchTransform`.** This is - an aggregation (many images → one `(model,tag,subset)` row); a chunked `BatchTransform` splits a tag's - images across chunks and computes wrong sums. Mirror the existing `count_…_metrics_on_subset` step - (it's a `DatatableBatchTransform`) and add `tag_id` to its grouping. -- **Leave alone:** `image__subset`/split, `..._metrics_on_subset`, `FindBestModel(subset_id="val")`, training. - -## Wire it into the pipeline (so it shows in the datapipe-app UI) -To make tags a node in the UI graph (not just a side script), add to the example code: -1. **`data.py` → `catalog`:** add `tag`, `image__tag`, `tag_metrics` as `Table(TableStoreDB(...))` (same - pattern as the other tables). `tag`/`image__tag` are external inputs (written by UI / INSERT / script); - `tag_metrics` is the step's output. -2. **`app.py` → `pipeline`:** add the aggregation step (see ⚠️ above) reading `…__metrics_on_image` + - `image__tag`, writing `tag_metrics`, `labels=[("stage","count-metrics")]` so it runs with metrics. -3. Populate `image__tag`, then run `stage=train` (or `stage=count-metrics`) — the UI graph then shows - the tag tables + step, and `tag_metrics` fills per model. - -> This is a **per-consumer** extension: the upstream datapipe example stays unchanged; whoever needs -> tags applies these edits **in their own project/repo** (following this recipe) and commits them there. -> Keep them out of the committed example. - -## How to use -1. Label the scenario images → rows in `image__tag` (a tag, e.g. `dark_room`). -2. Run the pipeline **as usual** — tagged images go through the normal split, part to `train` - (learned), part to `val` (holdout). -3. Read `tag_metrics`: the row `model=new, tag=dark_room, subset=val` against the same row for the old - model (several versions live in `detection_model_train`, per-image metrics computed for each). - -## Aggregation (core of the `tag_metrics` step) -The same can be checked one-off in SQL. ⚠️ Columns `calc__TP/FP/FN` were created in **mixed case** → -in Postgres quote them (`"calc__TP"`), otherwise `column ... does not exist`. -```sql -SELECT m.detection_model_id, t.tag_id, m.subset_id, - count(*) AS images_support, - sum(m."calc__TP")+sum(m."calc__FN") AS support, - sum(m."calc__TP")::numeric / NULLIF(sum(m."calc__TP")+sum(m."calc__FP"),0) AS precision, - sum(m."calc__TP")::numeric / NULLIF(sum(m."calc__TP")+sum(m."calc__FN"),0) AS recall -FROM detection_model_train__metrics_on_image m -JOIN image__tag t USING (image_name) -WHERE t.tag_id = 'dark_room' -GROUP BY 1,2,3; -``` - -## Gotchas (datapipe-ml behavior) -- **Any custom script that drives datapipe-ml training must be guarded with `if __name__ == "__main__":`** - — datapipe-ml launches YOLO training via multiprocessing **spawn**, so without the guard the child - re-imports the module and crashes with `RuntimeError: An attempt has been made to start a new process - before ... bootstrapping`. The stock `datapipe step ... stage=train run` is already guarded. -- **Post-training is not instant:** datapipe-ml syncs every epoch checkpoint to S3/MinIO and - `collect_results`/select-best reads them back, so on a slow link this takes minutes — normal, not a - hang (the process sits in `Sl`/`Dl`). -- **Don't carve up `image__subset`** to fake a "scenario" — the tag IS the slice; slice metrics by the - tag and keep the split full, otherwise old and new models get measured on different holdouts. - -## Demo scenario — how to reproduce (for an agent) - -Goal: show that a model scores near-0 on a tagged scenario and, after the tagged images are added to -training, the metric on that scenario rises. Steps to recreate it end-to-end. - -> This recipe **injects ready-made ground truth** (COCO labels / inherited boxes) and **skips the -> Label Studio `annotation` step** — so it runs unattended. Confirm this is acceptable up front (see -> the skill's "Ask first" line): if the demo must show real human annotation in Label Studio, that -> stays a manual step and isn't automated here. - -1. **Base data.** Seed a non-trivial cat/dog set (the 10-image smoke default is too small — the - metric on a tiny val is noisy). `scripts/seed_sample_data.py` defaults to `--detection-limit 120`, - which is enough for the pipeline's own metrics to work; `--keypoints-limit 0` for a detection-only - demo. Images download from COCO — needs outbound internet; on a restricted node fetch them - elsewhere and upload to MinIO. Read-only `/home` → set `DATAPIPE_CACHE_DIR` and `UV_*` under `/tmp`. -2. **Build the scenario.** Take ~40 of the already-labelled cat/dog images, make **darkened copies** - (gamma ≈ 0.10 — 0.25 is too mild to actually stump the model), upload them, and **inherit the GT - boxes/labels from each source image** (identical pixel size → identical boxes, so no annotation). - Insert one `tag` row (e.g. `night`) and one `image__tag` row per darkened image; split them ~30 - train / ~10 val. -3. **Baseline A — scenario NOT in training.** Keep the scenario's val images in `image__subset` as - `val`, but exclude its train images from `train`. Train through the repo's train flow → A scores - near-0 on the tag. -4. **Model B — scenario IN training.** Add the scenario's train images to `image__subset` `train`, then - retrain a fresh model. (Freeze is delta-gated: if it no-ops, bump those images' GT `update_ts` so it - re-cuts a dataset.) -5. **Metrics.** Compute per-image TP/FP/FN on val (predict, match GT at IoU≥0.5 per class), join - `image__tag`, and aggregate by `(detection_model_id, tag_id, subset_id)` into `tag_metrics`. -6. **Compare.** Read `tag_metrics` rows for `model=A` vs `model=B` at the **holdout** subset — B is - higher. The e2e split yields only `train`/`val`, so the holdout is `val` (include `test` too if your - split has one); ignore the `train` row (a model scores high on its own training data). For a visual, - open FiftyOne filtered to the tag and show B detecting where A is empty. - -**Note on checkpoint selection.** With enough data the pipeline's built-in metrics compute fine and the -tag arc shows up in `detection_model_train__metrics_on_subset` natively. On a *small* validation set the -"best epoch" pick can latch onto an early, noisy metric peak and publish an under-trained checkpoint — if -a model looks suspiciously empty, sanity-check against the final-epoch weights. - -## Live demo walkthrough (with the datapipe-app UI) -The narrated flow for showing this to someone — you drive the setup by chatting, then click in the UI: - -0. **Start from a clean slate (TEST DATA ONLY).** Before demoing, wipe prior state so the graph, runs - and metrics are pristine — otherwise old models and "Recent runs" from earlier attempts show up and - confuse the story. Two independent places to clear: - - **pipeline data** — the schema (e.g. `datapipe_e2e_detection`) + MinIO objects + local train - outputs, then `datapipe db create-all`; - - **observability run history** — lives **separately** in the `public` schema - (`pipeline_runs`, `pipeline_run_steps`, `pipeline_run_logs`, `analytics_training_runs`); dropping - the pipeline schema does NOT clear it. Truncate these to empty "Recent runs". - ⚠️ Do this **only on a throwaway/demo database**, never on real data. (Note: this Postgres is shared - with Label Studio — scope truncates to the datapipe-app tables above; don't `CASCADE` into LS tables.) - -1. **Invoke the skill.** `/setup-e2e-template`, and say *"this is a demo"*. It brings up the services - and seeds demo cat/dog data (COCO → MinIO), i.e. the images that would be annotated in Label Studio. -2. **Skill asks the key question:** *annotate for real in Label Studio, or inject ready-made GT?* For a - live demo pick **inject** — it writes GT directly (following the conventions in the skill's "Skip - annotation" section: basename `image_name`, `Cat`/`Dog` casing) so no human labelling is needed. -3. **Set up tags (don't skip this).** Before training, define the scenario: darken a subset → `tag` - (`night`) + `image__tag`, and add the `tag_metrics` step (see this file's "Wire it into the - pipeline"). Now tags are nodes in the UI graph. -4. **Bring up the UI:** `datapipe --pipeline app:app api` (agent mode + `DATAPIPE_APP_PIPELINE_ID`), - port-forward, open it. Show the pipeline graph including the tag tables/step. -5. **Train model A (baseline)** from the UI: run the **`train`** stage (not the whole pipeline — that - hits Label Studio). This is the model *without* the tagged scenario in training. -6. **Add the scenario to training** (tag-train rows into `image__subset` `train`) and run **`train`** - again → **model B**. Then run **`count-metrics`** / the tag step. -7. **Payoff:** the UI now shows **two models** and their metrics; `tag_metrics` on the tag's `val` - holdout is ~0 for A and higher for B. Optionally FiftyOne: B detects on the tagged images where A is empty. - -> Trigger runs **by stage** from the UI (there's a per-stage run menu / graph-node run), never "Run -> pipeline" (that runs `annotation` → Label Studio). Training runs on the API process; on this WIP -> branch it needs `pi_heif` and a pre-AVX2-safe `polars-lts-cpu` in the venv (see the api issues log). diff --git a/examples/detection_tags/.env.example b/examples/detection_tags/.env.example new file mode 100644 index 00000000..ba970ba4 --- /dev/null +++ b/examples/detection_tags/.env.example @@ -0,0 +1,10 @@ +export AWS_ACCESS_KEY_ID=minioadmin +export AWS_SECRET_ACCESS_KEY=minioadmin +export AWS_REGION=us-east-1 +export S3_ENDPOINT_URL=http://localhost:9000 +export FSSPEC_S3_ENDPOINT_URL="${S3_ENDPOINT_URL}" +export S3_PUBLIC_URL=http://localhost:9000 +export DATAPIPE_TAGS_DIR=s3://datapipe-tags +export DATAPIPE_TAGS_TMP_DIR=/tmp/datapipe-tags +export DB_URL=postgresql://postgres:password@localhost:5432/postgres +export DB_SCHEMA=datapipe_tags diff --git a/examples/detection_tags/README.md b/examples/detection_tags/README.md new file mode 100644 index 00000000..d1d0864a --- /dev/null +++ b/examples/detection_tags/README.md @@ -0,0 +1,46 @@ +# detection_tags + +A minimal, self-contained datapipe detection example built around **tags** — with **no Label +Studio and no FiftyOne**. Ground truth is injected directly (COCO labels, lowercase `cat`/`dog`), +so the whole thing runs unattended from scratch. + +The point: train a baseline, then add a **tagged scenario batch**, retrain, and watch the metric on +that tag rise — visible in a dedicated `tag_metrics` table (a real pipeline step). + +## Deploy from scratch +```bash +cp .env.example .env && set -a && source .env && set +a +docker compose up -d # postgres + minio only +uv sync # cu124 torch, polars-lts-cpu, pi-heif baked in +cd detection +datapipe db create-all +``` + +## Two-step data load (no annotation) +```bash +# from examples/detection_tags/detection +python ../scripts/load_batch.py --n 120 # batch 1: base cat/dog +python ../scripts/load_batch.py --n 40 --tag night --darken 0.1 # batch 2: tagged low-light scenario +``` +`load_batch.py` downloads COCO cat/dog images, uploads them to MinIO, and injects ground truth +(+ a tag for batch 2). No human labelling. + +## Run the pipeline +```bash +datapipe run # ingest -> split -> freeze -> train -> inference -> metrics -> tag_metrics +# or by stage: datapipe step --labels=stage=train run +``` + +## What you get +- `detection_model_train` — trained model(s). +- `detection_model_train__metrics_on_subset` — overall metrics per model. +- **`tag_metrics`** — `(detection_model_id, tag_id, subset_id)` → precision/recall/f1. Compare the + baseline (trained before batch 2) vs the retrained model on `tag=night, subset=val`: the tag recall + rises once the tagged batch is in training. + +## Notes +- Classes are lowercase (`cat`/`dog`) to match COCO, so injected GT and predictions align. +- On a tiny validation set the "best epoch" pick can latch onto an early checkpoint; this example + uses 50 epochs and a non-trivial batch size to keep metrics meaningful. +- Tables `tag` / `image__tag` are external inputs (written by `load_batch.py`); `tag_metrics` is + produced by a pipeline `BatchTransform` (`steps.compute_tag_metrics`). diff --git a/examples/detection_tags/detection/__init__.py b/examples/detection_tags/detection/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/examples/detection_tags/detection/app.py b/examples/detection_tags/detection/app.py new file mode 100644 index 00000000..47403051 --- /dev/null +++ b/examples/detection_tags/detection/app.py @@ -0,0 +1,130 @@ +from __future__ import annotations + +from datapipe.compute import DatapipeApp, Pipeline +from datapipe.datatable import DataStore +from datapipe.step.batch_generate import BatchGenerate +from datapipe.step.batch_transform import BatchTransform +from datapipe_ml.metrics.model_selection import FindBestModel +from datapipe_ml.tasks.detection.freeze import DetectionFreezeDataset +from datapipe_ml.tasks.detection.inference import Inference_DetectionModel +from datapipe_ml.tasks.detection.metrics import CountMetrics_Subset_DetectionModel +from datapipe_ml.tasks.detection.train.yolov8 import Train_YoloV8_DetectionModel, YoloV8_TrainingConfig +from datapipe_ml.training.specs import TrainingResumeConfig, TrainingSyncConfig + +import steps +from config import DATAPIPE_DIR, DBCONN, datapipe_tmp_folder +from data import catalog + +# Ground truth (image__ground_truth) and tags (tag / image__tag) are loaded by +# scripts/load_batch.py — there is no Label Studio annotation stage in this example. + +pipeline = Pipeline( + [ + BatchGenerate( + steps.list_s3_images, + outputs=["s3_images"], + labels=[("stage", "ingest")], + ), + BatchTransform( + func=steps.split_df_train_val, + inputs=["image__ground_truth", "image__subset"], + outputs=["image__subset"], + transform_keys=["image_name"], + kwargs=dict(primary_keys=["image_name"], val_perc=0.25, random_seed=42), + labels=[("stage", "train"), ("stage", "train-prepare")], + ), + DetectionFreezeDataset( # type: ignore[list-item] + input__image="s3_images", + input__image__ground_truth="image__ground_truth", + input__subset__has__image="image__subset", + output__detection_frozen_dataset="detection_frozen_dataset", + output__detection_frozen_dataset__has__image_gt="detection_frozen_dataset__has__image_gt", + working_dir=str(DATAPIPE_DIR), + min_within_time="1s", + min_delta=10, + primary_keys=["image_name"], + bbox_id__name=None, + image__image_path__name="image_url", + labels=[("stage", "train"), ("stage", "train-prepare")], + create_table=True, + ), + Train_YoloV8_DetectionModel( # type: ignore[list-item] + input__detection_frozen_dataset="detection_frozen_dataset", + input__detection_frozen_dataset__has__image_gt="detection_frozen_dataset__has__image_gt", + output__yolov8_train_config="yolov8_train_config", + output__detection_size_for_resize="detection_size_for_resize", + output__detection_frozen_dataset__resized_image_file="detection_frozen_dataset__resized_image_file", + output__detection_frozen_dataset__yolo_txt="detection_frozen_dataset__yolo_txt", + output__detection_model="detection_model_train", + output__detection_model_is_trained_on_detection_frozen_dataset=( + "detection_model_is_trained_on_detection_frozen_dataset" + ), + output__training_status="detection_training_status", + output__detection_frozen_dataset__class_names="detection_frozen_dataset__class_names", + max_within_time="1w", + working_dir=str(DATAPIPE_DIR), + tmp_folder=datapipe_tmp_folder(), + yolov8_train_configs=[ + YoloV8_TrainingConfig(model="yolov8n.pt", imgsz=320, batch=10, epochs=50, exist_ok=True) + ], + sync_config=TrainingSyncConfig(enabled=True, interval_s=30, retries=3, retry_sleep_s=30), + resume_config=TrainingResumeConfig( + continue_train_failed_models=True, min_completed_epochs=1, checkpoint="last", + max_attempts=10, reset_attempts_after="10m", lease_ttl_s=60, heartbeat_interval_s=10, + ), + primary_keys=["image_name"], + bbox_id__name=None, + labels=[("stage", "train")], + create_table=True, + allow_sample_size_mismatch=True, + model_suffix="_tags", + ), + Inference_DetectionModel( + input__image="s3_images", + input__detection_model="detection_model_train", + output__detection_prediction="detection_prediction_train", + primary_keys=["image_name"], + bbox_id__name=None, + image__image_path__name="image_url", + batch_size_default=1, + labels=[("stage", "train"), ("stage", "inference")], + create_table=True, + ), + CountMetrics_Subset_DetectionModel( + input__image__ground_truth="image__ground_truth", + input__subset__has__image="image__subset", + input__detection_prediction="detection_prediction_train", + output__detection_model__metrics__on__image="detection_model_train__metrics_on_image", + output__detection_model__metrics__on__subset="detection_model_train__metrics_on_subset", + primary_keys=["image_name"], + bbox_id__name=None, + minimum_iou=0.5, + labels=[("stage", "train"), ("stage", "count-metrics")], + create_table=True, + ), + FindBestModel( + input__model="detection_model_train", + input__model__metrics_on__subset="detection_model_train__metrics_on_subset", + output__attr__model__is_best="attr__detection_model__is_best", + output__best_model="best_detection_model", + subset_id="val", + is_best__name="detection_model__is_best", + primary_keys=["detection_model_id"], + metric__name="calc__f1_score", + func="max", + group_by=None, + labels=[("stage", "train"), ("stage", "count-metrics")], + ), + # tag arc: aggregate per-image metrics by (model, tag, subset) + BatchTransform( + func=steps.compute_tag_metrics, + inputs=["detection_model_train__metrics_on_image", "image__tag"], + outputs=["tag_metrics"], + transform_keys=["detection_model_id"], + labels=[("stage", "train"), ("stage", "count-metrics"), ("stage", "tag-metrics")], + ), + ] +) + +ds = DataStore(DBCONN, create_meta_table=True) +app = DatapipeApp(ds, catalog, pipeline) diff --git a/examples/detection_tags/detection/config.py b/examples/detection_tags/detection/config.py new file mode 100644 index 00000000..b4a871e0 --- /dev/null +++ b/examples/detection_tags/detection/config.py @@ -0,0 +1,85 @@ +from __future__ import annotations + +import os +from pathlib import Path +from typing import Optional + +import fsspec +from datapipe.store.database import DBConn +from pathy import Pathy + +# --- object storage (MinIO / S3) ------------------------------------------------ +AWS_KEY = os.environ.get("AWS_ACCESS_KEY_ID", "") +AWS_SECRET = os.environ.get("AWS_SECRET_ACCESS_KEY", "") +AWS_REGION = os.environ.get("AWS_REGION", "us-east-1") +S3_ENDPOINT_URL = os.environ.get("S3_ENDPOINT_URL") +S3_PUBLIC_URL = os.environ.get("S3_PUBLIC_URL", S3_ENDPOINT_URL or "http://localhost:9000") + +# --- classes (lowercase, as in COCO) ------------------------------------------- +# detection classes for this demo; keep lowercase to match COCO category names so +# injected ground truth and model predictions use identical label strings. +DETECTION_CLASSES = ["cat", "dog"] +COCO_CAT_IDS = {17: "cat", 18: "dog"} # COCO category_id -> class name + +# --- single storage root -------------------------------------------------------- +# Input images live under /images; the pipeline working_dir under /datapipe +# (siblings) so listing input images never re-ingests training artifacts. +DATAPIPE_DIR_ROOT = os.environ.get("DATAPIPE_TAGS_DIR", "s3://datapipe-tags").rstrip("/") +INPUT_IMAGES_DIR = f"{DATAPIPE_DIR_ROOT}/images" + + +def _is_cloud_path(path: str) -> bool: + protocol, _ = fsspec.core.split_protocol(path) + return protocol not in (None, "file") + + +def datapipe_working_dir() -> str: + if _is_cloud_path(DATAPIPE_DIR_ROOT): + return f"{DATAPIPE_DIR_ROOT}/datapipe" + local = Path(DATAPIPE_DIR_ROOT).resolve() / "datapipe" + local.mkdir(parents=True, exist_ok=True) + return str(local) + + +DATAPIPE_DIR = datapipe_working_dir() + + +def input_storage_options() -> dict: + if not _is_cloud_path(INPUT_IMAGES_DIR): + return {} + client_kwargs: dict = {"region_name": AWS_REGION} + if S3_ENDPOINT_URL: + client_kwargs["endpoint_url"] = S3_ENDPOINT_URL + return {"key": AWS_KEY, "secret": AWS_SECRET, "client_kwargs": client_kwargs} + + +def input_bucket() -> Optional[str]: + if not _is_cloud_path(DATAPIPE_DIR_ROOT): + return None + return Pathy(DATAPIPE_DIR_ROOT).root + + +def input_image_url(rel_key: str) -> str: + """Browser- and fsspec-readable URL for an input image relative to INPUT_IMAGES_DIR.""" + if _is_cloud_path(INPUT_IMAGES_DIR): + target = Pathy(INPUT_IMAGES_DIR) / rel_key + return f"{S3_PUBLIC_URL.rstrip('/')}/{target.root}/{target.key}" + return (Path(INPUT_IMAGES_DIR) / rel_key).resolve().as_uri() + + +def datapipe_tmp_folder() -> str: + if _is_cloud_path(DATAPIPE_DIR): + return str(Path(os.environ.get("DATAPIPE_TAGS_TMP_DIR", "/tmp/datapipe-tags")).resolve()) + return str(Path(DATAPIPE_DIR) / "tmp") + + +# --- database ------------------------------------------------------------------- +DB_URL = os.environ.get("DB_URL") +if not DB_URL: + raise RuntimeError( + "DB_URL is required. Copy examples/detection_tags/.env.example to .env, " + "start docker compose, and run: set -a && source .env && set +a" + ) + +DB_SCHEMA = os.environ.get("DB_SCHEMA", "datapipe_tags") +DBCONN = DBConn(DB_URL, DB_SCHEMA) diff --git a/examples/detection_tags/detection/data.py b/examples/detection_tags/detection/data.py new file mode 100644 index 00000000..88df8a64 --- /dev/null +++ b/examples/detection_tags/detection/data.py @@ -0,0 +1,65 @@ +from __future__ import annotations + +from datapipe.compute import Catalog, Table +from datapipe.store.database import TableStoreDB +from sqlalchemy import Column, Float, Integer, JSON, String + +from config import DBCONN + + +def _t(name, schema): + return Table(TableStoreDB(dbconn=DBCONN, name=name, data_sql_schema=schema, create_table=True)) + + +catalog = Catalog( + { + # images listed from object storage + "s3_images": _t("s3_images", [ + Column("image_name", String, primary_key=True), + Column("image_url", String), + ]), + # ground truth injected by the data-loading script (no Label Studio) + "image__ground_truth": _t("image__ground_truth", [ + Column("image_name", String, primary_key=True), + Column("bboxes", JSON), + Column("labels", JSON), + ]), + # train/val split + "image__subset": _t("image__subset", [ + Column("image_name", String, primary_key=True), + Column("subset_id", String, primary_key=True), + ]), + # --- tags ------------------------------------------------------------- + "tag": _t("tag", [ + Column("tag_id", String, primary_key=True), + Column("name", String), + ]), + "image__tag": _t("image__tag", [ + Column("image_name", String, primary_key=True), + Column("tag_id", String, primary_key=True), + ]), + # per-image metrics produced by the CountMetrics step (declared so the + # tag-metrics transform can read it as an input) + "detection_model_train__metrics_on_image": _t("detection_model_train__metrics_on_image", [ + Column("image_name", String, primary_key=True), + Column("detection_model_id", String, primary_key=True), + Column("subset_id", String, primary_key=True), + Column("calc__support", Integer), + Column("calc__TP", Integer), + Column("calc__FP", Integer), + Column("calc__FN", Integer), + Column("calc__iou_mean", Float), + ]), + # per-(model, tag, subset) metrics — the tag arc lives here + "tag_metrics": _t("tag_metrics", [ + Column("detection_model_id", String, primary_key=True), + Column("tag_id", String, primary_key=True), + Column("subset_id", String, primary_key=True), + Column("calc__images_support", Integer), + Column("calc__support", Integer), + Column("calc__precision", Float), + Column("calc__recall", Float), + Column("calc__f1_score", Float), + ]), + } +) diff --git a/examples/detection_tags/detection/steps.py b/examples/detection_tags/detection/steps.py new file mode 100644 index 00000000..ce802052 --- /dev/null +++ b/examples/detection_tags/detection/steps.py @@ -0,0 +1,64 @@ +from __future__ import annotations + +from typing import Iterator + +import fsspec +import pandas as pd + +from config import INPUT_IMAGES_DIR, input_image_url, input_storage_options + + +def list_s3_images() -> Iterator[pd.DataFrame]: + """List input images from object storage; image_name = key relative to INPUT_IMAGES_DIR.""" + fs, root = fsspec.core.url_to_fs(INPUT_IMAGES_DIR, **input_storage_options()) + keys, urls = [], [] + for path in fs.find(root): + if not path.lower().endswith((".jpg", ".jpeg", ".png", ".webp")): + continue + rel_key = path[len(root):].lstrip("/") + keys.append(rel_key.replace("/", "___")) + urls.append(input_image_url(rel_key)) + yield pd.DataFrame({"image_name": keys, "image_url": urls}) + + +def split_df_train_val( + df: pd.DataFrame, + subset_df: pd.DataFrame, + primary_keys: list[str], + val_perc: float = 0.25, + random_seed: int = 42, +) -> pd.DataFrame: + """Assign a train/val subset to any image that doesn't have one yet (stable).""" + df__merged = pd.merge(df, subset_df, on=primary_keys, how="outer") + df__missing = df__merged[df__merged["subset_id"].isna()].copy() + df__val = df__missing.sample(frac=val_perc, random_state=random_seed) + df__missing["subset_id"] = "train" + df__missing.loc[df__val.index, "subset_id"] = "val" + return pd.concat([subset_df, df__missing], ignore_index=True)[primary_keys + ["subset_id"]] + + +def compute_tag_metrics(df_metrics_on_image: pd.DataFrame, df_image_tag: pd.DataFrame) -> pd.DataFrame: + """Aggregate per-image detection metrics by (model, tag, subset). + + transform_keys=['detection_model_id'] groups one model's whole per-image table into a + single call, so the aggregation is correct (no cross-chunk splitting). df_image_tag has + no detection_model_id, so datapipe broadcasts it in full. + """ + cols = ["detection_model_id", "tag_id", "subset_id", "calc__images_support", + "calc__support", "calc__precision", "calc__recall", "calc__f1_score"] + if df_metrics_on_image.empty or df_image_tag.empty: + return pd.DataFrame(columns=cols) + m = df_metrics_on_image.merge(df_image_tag[["image_name", "tag_id"]], on="image_name") + if m.empty: + return pd.DataFrame(columns=cols) + g = (m.groupby(["detection_model_id", "tag_id", "subset_id"], as_index=False) + .agg(images_support=("image_name", "count"), + tp=("calc__TP", "sum"), fp=("calc__FP", "sum"), fn=("calc__FN", "sum"))) + tp, fp, fn = g["tp"], g["fp"], g["fn"] + g["calc__precision"] = (tp / (tp + fp)).where((tp + fp) > 0, 0.0) + g["calc__recall"] = (tp / (tp + fn)).where((tp + fn) > 0, 0.0) + p, r = g["calc__precision"], g["calc__recall"] + g["calc__f1_score"] = (2 * p * r / (p + r)).where((p + r) > 0, 0.0) + g["calc__support"] = (tp + fn).astype(int) + g["calc__images_support"] = g["images_support"].astype(int) + return g[cols] diff --git a/examples/detection_tags/docker-compose.yml b/examples/detection_tags/docker-compose.yml new file mode 100644 index 00000000..d1788ffa --- /dev/null +++ b/examples/detection_tags/docker-compose.yml @@ -0,0 +1,42 @@ +services: + postgres: + image: postgres:14 + environment: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: password + POSTGRES_DB: postgres + POSTGRES_HOST_AUTH_METHOD: trust + ports: + - "5432:5432" + healthcheck: + test: ["CMD-SHELL", "pg_isready -U postgres"] + interval: 10s + timeout: 5s + retries: 5 + + minio: + image: minio/minio:latest + command: server /data --console-address ":9001" + environment: + MINIO_ROOT_USER: minioadmin + MINIO_ROOT_PASSWORD: minioadmin + ports: + - "9000:9000" + - "9001:9001" + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] + interval: 10s + timeout: 5s + retries: 5 + + minio-init: + image: minio/mc:latest + depends_on: + minio: + condition: service_healthy + entrypoint: > + /bin/sh -c " + mc alias set local http://minio:9000 minioadmin minioadmin && + mc mb --ignore-existing local/datapipe-tags && + mc anonymous set download local/datapipe-tags + " diff --git a/examples/detection_tags/pyproject.toml b/examples/detection_tags/pyproject.toml new file mode 100644 index 00000000..edb38bc3 --- /dev/null +++ b/examples/detection_tags/pyproject.toml @@ -0,0 +1,33 @@ +[project] +name = "datapipe-detection-tags-example" +version = "0" +requires-python = ">=3.10,<3.13" +dependencies = [ + "datapipe-app", + "datapipe-ml[torch]", + "torch==2.6.0", + "torchvision==0.21.0", + "stringzilla==4.4.0", # needed by albumentations + "polars-lts-cpu==1.33.1", # ultralytics/datapipe on pre-AVX2 CPUs (avoids SIGILL) + "pi-heif", # ultralytics image verification imports it +] + +[[tool.uv.index]] +name = "pytorch-cu124" +url = "https://download.pytorch.org/whl/cu124" +explicit = true + +[tool.uv.sources] +torch = { index = "pytorch-cu124" } +torchvision = { index = "pytorch-cu124" } +datapipe-app = { path = "../../libs/datapipe-app", editable = true } +datapipe-ml = { path = "../../libs/datapipe-ml", editable = true } +datapipe-core = { path = "../../libs/datapipe-core", editable = true } + +[dependency-groups] +dev = [ + "pytest<8", + "requests", + "tqdm", + "boto3", +] diff --git a/examples/detection_tags/scripts/load_batch.py b/examples/detection_tags/scripts/load_batch.py new file mode 100644 index 00000000..e8221a35 --- /dev/null +++ b/examples/detection_tags/scripts/load_batch.py @@ -0,0 +1,143 @@ +"""Load a batch of cat/dog images into the demo (no Label Studio). + +Downloads COCO cat/dog images, uploads them to object storage, and injects ground truth +directly (labels lowercase, as in COCO). Optionally darkens the images and attaches a tag — +this is how you add a "scenario" batch that the baseline model hasn't seen. + +Two-step demo: + # batch 1 — base data + python scripts/load_batch.py --n 120 + # batch 2 — a tagged low-light scenario + python scripts/load_batch.py --n 40 --tag night --darken 0.1 + +Then run the pipeline (ingest -> split -> freeze -> train -> metrics -> tag_metrics). +Run from examples/detection_tags/detection (so `import config` resolves). +""" +from __future__ import annotations + +import argparse +import io +import os +import random +import sys +import time +import zipfile +from pathlib import Path + +import fsspec +import numpy as np +import pandas as pd +import requests +from PIL import Image +from sqlalchemy import Column, JSON, String + +sys.path.insert(0, ".") +import config # noqa: E402 +from datapipe.datatable import DataStore # noqa: E402 +from datapipe.store.database import TableStoreDB # noqa: E402 + +COCO_IMG_BASE = "http://images.cocodataset.org/train2017/" +COCO_ANN_URL = "http://images.cocodataset.org/annotations/annotations_trainval2017.zip" +CACHE = Path(os.environ.get("DATAPIPE_TAGS_CACHE_DIR", "/tmp/datapipe-tags-cache")) +RNG_SEED = 1234 + + +def _get(url: str, attempts: int = 4) -> requests.Response: + last = None + for a in range(attempts): + try: + r = requests.get(url, timeout=60, stream=True) + r.raise_for_status() + return r + except requests.RequestException as e: + last = e + time.sleep(min(2 ** a, 20)) + raise last + + +def coco_instances() -> dict: + import json + CACHE.mkdir(parents=True, exist_ok=True) + zip_path = CACHE / "annotations_trainval2017.zip" + if not zip_path.exists() or zip_path.stat().st_size < 1_000_000: + print(f"downloading COCO annotations (~241MB) to {zip_path} ...", flush=True) + with _get(COCO_ANN_URL) as r, open(zip_path, "wb") as f: + for chunk in r.iter_content(1 << 20): + f.write(chunk) + with zipfile.ZipFile(zip_path) as zf, zf.open("annotations/instances_train2017.json") as h: + return json.load(h) + + +def darken(im: Image.Image, gamma: float) -> Image.Image: + a = np.asarray(im.convert("RGB")).astype(np.float32) / 255.0 + return Image.fromarray((np.power(a, 1.0 / gamma) * 255).clip(0, 255).astype(np.uint8)) + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--n", type=int, default=120, help="number of cat/dog images") + ap.add_argument("--tag", type=str, default=None, help="tag id to attach (e.g. night)") + ap.add_argument("--darken", type=float, default=None, help="gamma < 1 darkens (e.g. 0.1)") + ap.add_argument("--offset", type=int, default=0, help="skip the first OFFSET picked images") + args = ap.parse_args() + + inst = coco_instances() + id_to_img = {im["id"]: im for im in inst["images"]} + anns: dict[int, list] = {} + for a in inst["annotations"]: + if a.get("category_id") in config.COCO_CAT_IDS and not a.get("iscrowd", 0) and a.get("bbox"): + anns.setdefault(a["image_id"], []).append(a) + cat_dog_ids = sorted(anns.keys()) + random.seed(RNG_SEED) + random.shuffle(cat_dog_ids) + picked = cat_dog_ids[args.offset: args.offset + args.n] + + bucket = config.input_bucket() + fs = fsspec.filesystem("s3", **{k: v for k, v in config.input_storage_options().items()}) + ds = DataStore(config.DBCONN, create_meta_table=True) + + s3_rows, gt_rows, tag_rows = [], [], [] + for i, iid in enumerate(picked): + fn = id_to_img[iid]["file_name"] # e.g. 000000123456.jpg + stem, ext = os.path.splitext(fn) + raw = _get(COCO_IMG_BASE + fn).content + if args.darken is not None: + name = f"{stem}__{args.tag or 'dark'}{ext}" + buf = io.BytesIO() + darken(Image.open(io.BytesIO(raw)), args.darken).save(buf, format="JPEG", quality=92) + data = buf.getvalue() + else: + name = fn + data = raw + fs.pipe(f"{bucket}/images/{name}", data) + boxes, labels = [], [] + for a in anns[iid]: + x, y, w, h = a["bbox"] + boxes.append([int(x), int(y), int(x + w), int(y + h)]) + labels.append(config.COCO_CAT_IDS[a["category_id"]]) + s3_rows.append({"image_name": name, "image_url": config.input_image_url(name)}) + gt_rows.append({"image_name": name, "bboxes": boxes, "labels": labels}) + if args.tag: + tag_rows.append({"image_name": name, "tag_id": args.tag}) + if (i + 1) % 20 == 0: + print(f" {i + 1}/{len(picked)}", flush=True) + + def tbl(nm, sch): + return ds.get_or_create_table(nm, TableStoreDB(dbconn=config.DBCONN, name=nm, data_sql_schema=sch, create_table=True)) + + # s3_images gets refreshed by the ingest stage too, but write it here so the batch is + # usable immediately; image_name matches what list_s3_images emits (basename). + tbl("s3_images", [Column("image_name", String, primary_key=True), Column("image_url", String)]).store_chunk(pd.DataFrame(s3_rows)) + tbl("image__ground_truth", [Column("image_name", String, primary_key=True), Column("bboxes", JSON), Column("labels", JSON)]).store_chunk(pd.DataFrame(gt_rows)) + if args.tag: + tbl("tag", [Column("tag_id", String, primary_key=True), Column("name", String)]).store_chunk( + pd.DataFrame([{"tag_id": args.tag, "name": args.tag}])) + tbl("image__tag", [Column("image_name", String, primary_key=True), Column("tag_id", String, primary_key=True)]).store_chunk(pd.DataFrame(tag_rows)) + print(f"loaded {len(s3_rows)} images" + + (f", tag={args.tag}" if args.tag else "") + + (f", darken={args.darken}" if args.darken is not None else ""), flush=True) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/examples/e2e_template/image_detection/config.py b/examples/e2e_template/image_detection/config.py index f71e07a1..7e348998 100644 --- a/examples/e2e_template/image_detection/config.py +++ b/examples/e2e_template/image_detection/config.py @@ -24,8 +24,8 @@ - """ From 81ff19f46729a8abc6d84740da6465895894e647 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 6 Jul 2026 14:02:05 +0300 Subject: [PATCH 21/32] detection_tags: load data via a datapipe step (load_request table + load BatchTransform) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the standalone load_batch.py with a load stage in the pipeline: a load_request table (request_id/n/offset/tag/darken) feeds a load BatchTransform that downloads COCO cat/dog, uploads to MinIO and emits s3_images + ground truth (+ tag/image__tag). Loading a batch = add a request row (scripts/add_request.py) then datapipe step --labels=stage=load run — so ingestion is datapipe-native and shows in the graph. Two requests (base, tagged night) drive the two-model demo. --- .claude/skills/setup-detection-tags/SKILL.md | 15 +- examples/detection_tags/README.md | 21 +-- examples/detection_tags/detection/app.py | 17 +- examples/detection_tags/detection/data.py | 10 +- examples/detection_tags/detection/steps.py | 155 ++++++++++++++---- .../detection_tags/scripts/add_request.py | 49 ++++++ examples/detection_tags/scripts/load_batch.py | 143 ---------------- 7 files changed, 212 insertions(+), 198 deletions(-) create mode 100644 examples/detection_tags/scripts/add_request.py delete mode 100644 examples/detection_tags/scripts/load_batch.py diff --git a/.claude/skills/setup-detection-tags/SKILL.md b/.claude/skills/setup-detection-tags/SKILL.md index b0483d13..047eec07 100644 --- a/.claude/skills/setup-detection-tags/SKILL.md +++ b/.claude/skills/setup-detection-tags/SKILL.md @@ -28,18 +28,21 @@ uv sync # cu124 torch + polars-lts-cpu + pi-heif baked in cd detection && datapipe db create-all ``` -## Two-step data load (no annotation) +## Two-step data load — via datapipe steps (no annotation) +Loading is a pipeline step (`stage=load`) driven by rows in the `load_request` table. Add a request, +run the load step; it downloads COCO cat/dog, uploads to MinIO, and emits `s3_images` + ground truth +(+ tag). Labels are lowercase `cat`/`dog`; `image_name` = object basename so all joins line up. ```bash # from examples/detection_tags/detection -python ../scripts/load_batch.py --n 120 # batch 1: base cat/dog -python ../scripts/load_batch.py --n 40 --tag night --darken 0.1 # batch 2: tagged scenario +python ../scripts/add_request.py --id base --n 120 +datapipe step --labels=stage=load run # batch 1: base cat/dog +python ../scripts/add_request.py --id night --n 40 --offset 120 --tag night --darken 0.1 +datapipe step --labels=stage=load run # batch 2: tagged scenario ``` -`load_batch.py` downloads COCO cat/dog, uploads to MinIO, injects GT (+ a tag for batch 2). Labels are -lowercase `cat`/`dog`; `image_name` = the object basename (what `list_s3_images` emits) so the joins line up. ## Run ```bash -datapipe run # ingest → split → freeze → train → inference → metrics → tag_metrics +datapipe run # load → split → freeze → train → inference → metrics → tag_metrics # or by stage: datapipe step --labels=stage=train run ``` diff --git a/examples/detection_tags/README.md b/examples/detection_tags/README.md index d1d0864a..bb6f6f17 100644 --- a/examples/detection_tags/README.md +++ b/examples/detection_tags/README.md @@ -16,20 +16,21 @@ cd detection datapipe db create-all ``` -## Two-step data load (no annotation) +## Two-step data load — via datapipe steps (no annotation) +Loading is a real pipeline step (`stage=load`): add a request row, then run the step. It downloads +COCO cat/dog, uploads to MinIO, and produces `s3_images` + ground truth (+ a tag) — no human labelling. ```bash # from examples/detection_tags/detection -python ../scripts/load_batch.py --n 120 # batch 1: base cat/dog -python ../scripts/load_batch.py --n 40 --tag night --darken 0.1 # batch 2: tagged low-light scenario -``` -`load_batch.py` downloads COCO cat/dog images, uploads them to MinIO, and injects ground truth -(+ a tag for batch 2). No human labelling. +python ../scripts/add_request.py --id base --n 120 +datapipe step --labels=stage=load run # batch 1: base cat/dog +datapipe step --labels=stage=train run # -> model A (baseline, no tag in training) -## Run the pipeline -```bash -datapipe run # ingest -> split -> freeze -> train -> inference -> metrics -> tag_metrics -# or by stage: datapipe step --labels=stage=train run +python ../scripts/add_request.py --id night --n 40 --offset 120 --tag night --darken 0.1 +datapipe step --labels=stage=load run # batch 2: tagged low-light scenario +datapipe step --labels=stage=train run # -> model B (tag in training) +datapipe step --labels=stage=count-metrics run # (re)compute metrics incl. tag_metrics ``` +Or just `datapipe run` after adding request rows to run every stage. ## What you get - `detection_model_train` — trained model(s). diff --git a/examples/detection_tags/detection/app.py b/examples/detection_tags/detection/app.py index 47403051..97f9ecb4 100644 --- a/examples/detection_tags/detection/app.py +++ b/examples/detection_tags/detection/app.py @@ -2,7 +2,6 @@ from datapipe.compute import DatapipeApp, Pipeline from datapipe.datatable import DataStore -from datapipe.step.batch_generate import BatchGenerate from datapipe.step.batch_transform import BatchTransform from datapipe_ml.metrics.model_selection import FindBestModel from datapipe_ml.tasks.detection.freeze import DetectionFreezeDataset @@ -15,15 +14,19 @@ from config import DATAPIPE_DIR, DBCONN, datapipe_tmp_folder from data import catalog -# Ground truth (image__ground_truth) and tags (tag / image__tag) are loaded by -# scripts/load_batch.py — there is no Label Studio annotation stage in this example. +# Data is loaded via the `load` step: add a row to `load_request` (request_id, n, offset, tag, +# darken) and run `datapipe step --labels=stage=load run`. It downloads COCO cat/dog images, +# uploads them to object storage, and produces s3_images + ground truth (+ tag) directly — there +# is no Label Studio annotation stage. pipeline = Pipeline( [ - BatchGenerate( - steps.list_s3_images, - outputs=["s3_images"], - labels=[("stage", "ingest")], + BatchTransform( + func=steps.load_batch, + inputs=["load_request"], + outputs=["s3_images", "image__ground_truth", "tag", "image__tag"], + transform_keys=["request_id"], + labels=[("stage", "load")], ), BatchTransform( func=steps.split_df_train_val, diff --git a/examples/detection_tags/detection/data.py b/examples/detection_tags/detection/data.py index 88df8a64..4fce06e8 100644 --- a/examples/detection_tags/detection/data.py +++ b/examples/detection_tags/detection/data.py @@ -13,7 +13,15 @@ def _t(name, schema): catalog = Catalog( { - # images listed from object storage + # a load request: add a row here, then run `datapipe step --labels=stage=load run` + "load_request": _t("load_request", [ + Column("request_id", String, primary_key=True), + Column("n", Integer), + Column("offset", Integer), + Column("tag", String), + Column("darken", Float), + ]), + # images uploaded by the load step (image_name = object basename) "s3_images": _t("s3_images", [ Column("image_name", String, primary_key=True), Column("image_url", String), diff --git a/examples/detection_tags/detection/steps.py b/examples/detection_tags/detection/steps.py index ce802052..dde2571e 100644 --- a/examples/detection_tags/detection/steps.py +++ b/examples/detection_tags/detection/steps.py @@ -1,34 +1,132 @@ from __future__ import annotations -from typing import Iterator +import io +import json +import os +import random +import time +import zipfile +from pathlib import Path +from typing import Optional import fsspec +import numpy as np import pandas as pd +import requests +from PIL import Image -from config import INPUT_IMAGES_DIR, input_image_url, input_storage_options - - -def list_s3_images() -> Iterator[pd.DataFrame]: - """List input images from object storage; image_name = key relative to INPUT_IMAGES_DIR.""" - fs, root = fsspec.core.url_to_fs(INPUT_IMAGES_DIR, **input_storage_options()) - keys, urls = [], [] - for path in fs.find(root): - if not path.lower().endswith((".jpg", ".jpeg", ".png", ".webp")): - continue - rel_key = path[len(root):].lstrip("/") - keys.append(rel_key.replace("/", "___")) - urls.append(input_image_url(rel_key)) - yield pd.DataFrame({"image_name": keys, "image_url": urls}) - - -def split_df_train_val( - df: pd.DataFrame, - subset_df: pd.DataFrame, - primary_keys: list[str], - val_perc: float = 0.25, - random_seed: int = 42, -) -> pd.DataFrame: - """Assign a train/val subset to any image that doesn't have one yet (stable).""" +from config import ( + COCO_CAT_IDS, + DBCONN, + input_bucket, + input_image_url, + input_storage_options, +) + +COCO_IMG_BASE = "http://images.cocodataset.org/train2017/" +COCO_ANN_URL = "http://images.cocodataset.org/annotations/annotations_trainval2017.zip" +CACHE = Path(os.environ.get("DATAPIPE_TAGS_CACHE_DIR", "/tmp/datapipe-tags-cache")) +RNG_SEED = 1234 + + +# --- COCO helpers --------------------------------------------------------------- +def _get(url: str, attempts: int = 4) -> requests.Response: + last: Optional[Exception] = None + for a in range(attempts): + try: + r = requests.get(url, timeout=60, stream=True) + r.raise_for_status() + return r + except requests.RequestException as e: + last = e + time.sleep(min(2 ** a, 20)) + assert last is not None + raise last + + +def _coco_annotations_by_image() -> tuple[dict, dict]: + """Return (id_to_image_meta, {image_id: [cat/dog annotations]}), cached on disk.""" + CACHE.mkdir(parents=True, exist_ok=True) + zip_path = CACHE / "annotations_trainval2017.zip" + if not zip_path.exists() or zip_path.stat().st_size < 1_000_000: + with _get(COCO_ANN_URL) as r, open(zip_path, "wb") as f: + for chunk in r.iter_content(1 << 20): + f.write(chunk) + with zipfile.ZipFile(zip_path) as zf, zf.open("annotations/instances_train2017.json") as h: + inst = json.load(h) + id_to_img = {im["id"]: im for im in inst["images"]} + anns: dict[int, list] = {} + for a in inst["annotations"]: + if a.get("category_id") in COCO_CAT_IDS and not a.get("iscrowd", 0) and a.get("bbox"): + anns.setdefault(a["image_id"], []).append(a) + return id_to_img, anns + + +def _darken(im: Image.Image, gamma: float) -> Image.Image: + a = np.asarray(im.convert("RGB")).astype(np.float32) / 255.0 + return Image.fromarray((np.power(a, 1.0 / gamma) * 255).clip(0, 255).astype(np.uint8)) + + +# --- load step ------------------------------------------------------------------ +def load_batch(df_request: pd.DataFrame): + """For each load_request row: download COCO cat/dog images, upload to object storage, and + return (s3_images, image__ground_truth, tag, image__tag). Ground truth uses lowercase COCO + labels; image_name is the object basename (what a listing would produce).""" + s3_cols = ["image_name", "image_url"] + gt_cols = ["image_name", "bboxes", "labels"] + tag_cols = ["tag_id", "name"] + it_cols = ["image_name", "tag_id"] + if df_request.empty: + return (pd.DataFrame(columns=s3_cols), pd.DataFrame(columns=gt_cols), + pd.DataFrame(columns=tag_cols), pd.DataFrame(columns=it_cols)) + + id_to_img, anns = _coco_annotations_by_image() + cat_dog_ids = sorted(anns.keys()) + random.seed(RNG_SEED) + random.shuffle(cat_dog_ids) + + fs = fsspec.filesystem("s3", **input_storage_options()) + bucket = input_bucket() + + s3_rows, gt_rows, tag_rows, tag_defs = [], [], [], {} + for _, req in df_request.iterrows(): + n = int(req["n"]) + offset = int(req.get("offset") or 0) + tag = req.get("tag") or None + darken = req.get("darken") + darken = float(darken) if darken is not None and str(darken) != "" else None + for iid in cat_dog_ids[offset: offset + n]: + fn = id_to_img[iid]["file_name"] + stem, ext = os.path.splitext(fn) + raw = _get(COCO_IMG_BASE + fn).content + if darken is not None: + name = f"{stem}__{tag or 'dark'}{ext}" + buf = io.BytesIO() + _darken(Image.open(io.BytesIO(raw)), darken).save(buf, format="JPEG", quality=92) + data = buf.getvalue() + else: + name, data = fn, raw + fs.pipe(f"{bucket}/images/{name}", data) + boxes = [[int(a["bbox"][0]), int(a["bbox"][1]), + int(a["bbox"][0] + a["bbox"][2]), int(a["bbox"][1] + a["bbox"][3])] for a in anns[iid]] + labels = [COCO_CAT_IDS[a["category_id"]] for a in anns[iid]] + s3_rows.append({"image_name": name, "image_url": input_image_url(name)}) + gt_rows.append({"image_name": name, "bboxes": boxes, "labels": labels}) + if tag: + tag_rows.append({"image_name": name, "tag_id": tag}) + tag_defs[tag] = tag + + return ( + pd.DataFrame(s3_rows, columns=s3_cols), + pd.DataFrame(gt_rows, columns=gt_cols), + pd.DataFrame([{"tag_id": t, "name": n} for t, n in tag_defs.items()], columns=tag_cols), + pd.DataFrame(tag_rows, columns=it_cols), + ) + + +# --- train/val split ------------------------------------------------------------ +def split_df_train_val(df: pd.DataFrame, subset_df: pd.DataFrame, primary_keys: list[str], + val_perc: float = 0.25, random_seed: int = 42) -> pd.DataFrame: df__merged = pd.merge(df, subset_df, on=primary_keys, how="outer") df__missing = df__merged[df__merged["subset_id"].isna()].copy() df__val = df__missing.sample(frac=val_perc, random_state=random_seed) @@ -37,13 +135,8 @@ def split_df_train_val( return pd.concat([subset_df, df__missing], ignore_index=True)[primary_keys + ["subset_id"]] +# --- tag metrics ---------------------------------------------------------------- def compute_tag_metrics(df_metrics_on_image: pd.DataFrame, df_image_tag: pd.DataFrame) -> pd.DataFrame: - """Aggregate per-image detection metrics by (model, tag, subset). - - transform_keys=['detection_model_id'] groups one model's whole per-image table into a - single call, so the aggregation is correct (no cross-chunk splitting). df_image_tag has - no detection_model_id, so datapipe broadcasts it in full. - """ cols = ["detection_model_id", "tag_id", "subset_id", "calc__images_support", "calc__support", "calc__precision", "calc__recall", "calc__f1_score"] if df_metrics_on_image.empty or df_image_tag.empty: diff --git a/examples/detection_tags/scripts/add_request.py b/examples/detection_tags/scripts/add_request.py new file mode 100644 index 00000000..43a08cdb --- /dev/null +++ b/examples/detection_tags/scripts/add_request.py @@ -0,0 +1,49 @@ +"""Add a load request, then run the load step: datapipe step --labels=stage=load run + + # batch 1 — base cat/dog + python ../scripts/add_request.py --id base --n 120 + # batch 2 — tagged low-light scenario + python ../scripts/add_request.py --id night --n 40 --offset 120 --tag night --darken 0.1 + +Run from examples/detection_tags/detection (so `import config` resolves).""" +from __future__ import annotations + +import argparse +import sys + +import pandas as pd +from sqlalchemy import Column, Float, Integer, String + +sys.path.insert(0, ".") +import config # noqa: E402 +from datapipe.datatable import DataStore # noqa: E402 +from datapipe.store.database import TableStoreDB # noqa: E402 + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--id", required=True, help="request_id (unique per batch)") + ap.add_argument("--n", type=int, default=120) + ap.add_argument("--offset", type=int, default=0, help="skip the first OFFSET picked COCO images") + ap.add_argument("--tag", default=None) + ap.add_argument("--darken", type=float, default=None, help="gamma < 1 darkens (e.g. 0.1)") + a = ap.parse_args() + + ds = DataStore(config.DBCONN, create_meta_table=True) + dt = ds.get_or_create_table("load_request", TableStoreDB( + dbconn=config.DBCONN, name="load_request", + data_sql_schema=[ + Column("request_id", String, primary_key=True), + Column("n", Integer), Column("offset", Integer), + Column("tag", String), Column("darken", Float), + ], create_table=True)) + dt.store_chunk(pd.DataFrame([{ + "request_id": a.id, "n": a.n, "offset": a.offset, "tag": a.tag, "darken": a.darken, + }])) + print(f"added request {a.id}: n={a.n} offset={a.offset} tag={a.tag} darken={a.darken}") + print("now run: datapipe step --labels=stage=load run") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/examples/detection_tags/scripts/load_batch.py b/examples/detection_tags/scripts/load_batch.py deleted file mode 100644 index e8221a35..00000000 --- a/examples/detection_tags/scripts/load_batch.py +++ /dev/null @@ -1,143 +0,0 @@ -"""Load a batch of cat/dog images into the demo (no Label Studio). - -Downloads COCO cat/dog images, uploads them to object storage, and injects ground truth -directly (labels lowercase, as in COCO). Optionally darkens the images and attaches a tag — -this is how you add a "scenario" batch that the baseline model hasn't seen. - -Two-step demo: - # batch 1 — base data - python scripts/load_batch.py --n 120 - # batch 2 — a tagged low-light scenario - python scripts/load_batch.py --n 40 --tag night --darken 0.1 - -Then run the pipeline (ingest -> split -> freeze -> train -> metrics -> tag_metrics). -Run from examples/detection_tags/detection (so `import config` resolves). -""" -from __future__ import annotations - -import argparse -import io -import os -import random -import sys -import time -import zipfile -from pathlib import Path - -import fsspec -import numpy as np -import pandas as pd -import requests -from PIL import Image -from sqlalchemy import Column, JSON, String - -sys.path.insert(0, ".") -import config # noqa: E402 -from datapipe.datatable import DataStore # noqa: E402 -from datapipe.store.database import TableStoreDB # noqa: E402 - -COCO_IMG_BASE = "http://images.cocodataset.org/train2017/" -COCO_ANN_URL = "http://images.cocodataset.org/annotations/annotations_trainval2017.zip" -CACHE = Path(os.environ.get("DATAPIPE_TAGS_CACHE_DIR", "/tmp/datapipe-tags-cache")) -RNG_SEED = 1234 - - -def _get(url: str, attempts: int = 4) -> requests.Response: - last = None - for a in range(attempts): - try: - r = requests.get(url, timeout=60, stream=True) - r.raise_for_status() - return r - except requests.RequestException as e: - last = e - time.sleep(min(2 ** a, 20)) - raise last - - -def coco_instances() -> dict: - import json - CACHE.mkdir(parents=True, exist_ok=True) - zip_path = CACHE / "annotations_trainval2017.zip" - if not zip_path.exists() or zip_path.stat().st_size < 1_000_000: - print(f"downloading COCO annotations (~241MB) to {zip_path} ...", flush=True) - with _get(COCO_ANN_URL) as r, open(zip_path, "wb") as f: - for chunk in r.iter_content(1 << 20): - f.write(chunk) - with zipfile.ZipFile(zip_path) as zf, zf.open("annotations/instances_train2017.json") as h: - return json.load(h) - - -def darken(im: Image.Image, gamma: float) -> Image.Image: - a = np.asarray(im.convert("RGB")).astype(np.float32) / 255.0 - return Image.fromarray((np.power(a, 1.0 / gamma) * 255).clip(0, 255).astype(np.uint8)) - - -def main() -> int: - ap = argparse.ArgumentParser() - ap.add_argument("--n", type=int, default=120, help="number of cat/dog images") - ap.add_argument("--tag", type=str, default=None, help="tag id to attach (e.g. night)") - ap.add_argument("--darken", type=float, default=None, help="gamma < 1 darkens (e.g. 0.1)") - ap.add_argument("--offset", type=int, default=0, help="skip the first OFFSET picked images") - args = ap.parse_args() - - inst = coco_instances() - id_to_img = {im["id"]: im for im in inst["images"]} - anns: dict[int, list] = {} - for a in inst["annotations"]: - if a.get("category_id") in config.COCO_CAT_IDS and not a.get("iscrowd", 0) and a.get("bbox"): - anns.setdefault(a["image_id"], []).append(a) - cat_dog_ids = sorted(anns.keys()) - random.seed(RNG_SEED) - random.shuffle(cat_dog_ids) - picked = cat_dog_ids[args.offset: args.offset + args.n] - - bucket = config.input_bucket() - fs = fsspec.filesystem("s3", **{k: v for k, v in config.input_storage_options().items()}) - ds = DataStore(config.DBCONN, create_meta_table=True) - - s3_rows, gt_rows, tag_rows = [], [], [] - for i, iid in enumerate(picked): - fn = id_to_img[iid]["file_name"] # e.g. 000000123456.jpg - stem, ext = os.path.splitext(fn) - raw = _get(COCO_IMG_BASE + fn).content - if args.darken is not None: - name = f"{stem}__{args.tag or 'dark'}{ext}" - buf = io.BytesIO() - darken(Image.open(io.BytesIO(raw)), args.darken).save(buf, format="JPEG", quality=92) - data = buf.getvalue() - else: - name = fn - data = raw - fs.pipe(f"{bucket}/images/{name}", data) - boxes, labels = [], [] - for a in anns[iid]: - x, y, w, h = a["bbox"] - boxes.append([int(x), int(y), int(x + w), int(y + h)]) - labels.append(config.COCO_CAT_IDS[a["category_id"]]) - s3_rows.append({"image_name": name, "image_url": config.input_image_url(name)}) - gt_rows.append({"image_name": name, "bboxes": boxes, "labels": labels}) - if args.tag: - tag_rows.append({"image_name": name, "tag_id": args.tag}) - if (i + 1) % 20 == 0: - print(f" {i + 1}/{len(picked)}", flush=True) - - def tbl(nm, sch): - return ds.get_or_create_table(nm, TableStoreDB(dbconn=config.DBCONN, name=nm, data_sql_schema=sch, create_table=True)) - - # s3_images gets refreshed by the ingest stage too, but write it here so the batch is - # usable immediately; image_name matches what list_s3_images emits (basename). - tbl("s3_images", [Column("image_name", String, primary_key=True), Column("image_url", String)]).store_chunk(pd.DataFrame(s3_rows)) - tbl("image__ground_truth", [Column("image_name", String, primary_key=True), Column("bboxes", JSON), Column("labels", JSON)]).store_chunk(pd.DataFrame(gt_rows)) - if args.tag: - tbl("tag", [Column("tag_id", String, primary_key=True), Column("name", String)]).store_chunk( - pd.DataFrame([{"tag_id": args.tag, "name": args.tag}])) - tbl("image__tag", [Column("image_name", String, primary_key=True), Column("tag_id", String, primary_key=True)]).store_chunk(pd.DataFrame(tag_rows)) - print(f"loaded {len(s3_rows)} images" - + (f", tag={args.tag}" if args.tag else "") - + (f", darken={args.darken}" if args.darken is not None else ""), flush=True) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) From 96a1916e4d295e9daf2955d63c916d868b424194 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 6 Jul 2026 14:02:20 +0300 Subject: [PATCH 22/32] detection_tags: fix stale load_batch.py references in docs --- .claude/skills/setup-detection-tags/SKILL.md | 2 +- examples/detection_tags/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.claude/skills/setup-detection-tags/SKILL.md b/.claude/skills/setup-detection-tags/SKILL.md index 047eec07..67630728 100644 --- a/.claude/skills/setup-detection-tags/SKILL.md +++ b/.claude/skills/setup-detection-tags/SKILL.md @@ -49,7 +49,7 @@ datapipe run # load → split → freeze → train → inferenc ## The payoff `tag_metrics(detection_model_id, tag_id, subset_id)` → precision/recall/f1. Compare the baseline (trained before batch 2) vs the retrained model at `tag=night, subset=val` — recall on the tag rises -once the tagged batch is in training. `tag`/`image__tag` are external inputs (from `load_batch.py`); +once the tagged batch is in training. `tag`/`image__tag` are external inputs (produced by the `load` step); `tag_metrics` is a real pipeline step (`steps.compute_tag_metrics`, `transform_keys=["detection_model_id"]` so the aggregation is correct). diff --git a/examples/detection_tags/README.md b/examples/detection_tags/README.md index bb6f6f17..72aeb5d2 100644 --- a/examples/detection_tags/README.md +++ b/examples/detection_tags/README.md @@ -43,5 +43,5 @@ Or just `datapipe run` after adding request rows to run every stage. - Classes are lowercase (`cat`/`dog`) to match COCO, so injected GT and predictions align. - On a tiny validation set the "best epoch" pick can latch onto an early checkpoint; this example uses 50 epochs and a non-trivial batch size to keep metrics meaningful. -- Tables `tag` / `image__tag` are external inputs (written by `load_batch.py`); `tag_metrics` is +- Tables `tag` / `image__tag` are external inputs (produced by the `load` step); `tag_metrics` is produced by a pipeline `BatchTransform` (`steps.compute_tag_metrics`). From 28a39e67eef4d50550171f69989c4f43dc9f1fc9 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 6 Jul 2026 14:02:48 +0300 Subject: [PATCH 23/32] detection_tags skill: fix stale list_s3_images note (no such step in this example) --- .claude/skills/setup-detection-tags/SKILL.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.claude/skills/setup-detection-tags/SKILL.md b/.claude/skills/setup-detection-tags/SKILL.md index 67630728..f409aa67 100644 --- a/.claude/skills/setup-detection-tags/SKILL.md +++ b/.claude/skills/setup-detection-tags/SKILL.md @@ -63,8 +63,8 @@ so the aggregation is correct). than the host (pre-AVX2). This example pins `polars-lts-cpu`; if it still happens, reinstall it. - **`No labels found` / every image "corrupt: No module named 'pi_heif'`** → ultralytics image verification needs `pi_heif` (pinned here); reinstall if missing. -- **`No ground truth` at freeze** → injected `image__ground_truth.image_name` must equal what - `list_s3_images` emits (object basename); a prefixed key makes the join empty. +- **`No ground truth` at freeze** → `image__ground_truth.image_name` must match `s3_images.image_name` + (the load step uses the object basename for both); a mismatched key makes the join empty. - **Metrics 0 on a trained model** → tiny/noisy val makes "best epoch" latch onto an early checkpoint; use enough data (default batches) and the shipped 50-epoch config. - **Training exits 0 but no model** → datapipe swallows step errors; check `detection_training_status`. From 4472b8d84aced29ba753cf580652f9d28bd7ba80 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 6 Jul 2026 14:19:21 +0300 Subject: [PATCH 24/32] detection_tags: cache-first load step (pre-staged images + gt.json to skip COCO on slow networks) --- examples/detection_tags/README.md | 9 +++++ examples/detection_tags/detection/steps.py | 45 +++++++++++++++++----- 2 files changed, 44 insertions(+), 10 deletions(-) diff --git a/examples/detection_tags/README.md b/examples/detection_tags/README.md index 72aeb5d2..b15fd3f7 100644 --- a/examples/detection_tags/README.md +++ b/examples/detection_tags/README.md @@ -32,6 +32,15 @@ datapipe step --labels=stage=count-metrics run # (re)compute metrics incl. tag_m ``` Or just `datapipe run` after adding request rows to run every stage. +### Fast data on a slow/blocked network (pre-staged cache) +The `load` step downloads images from COCO per request, which is slow on a high-latency or +throttled link. To avoid it, pre-stage a cache once from a fast machine: +`DATAPIPE_TAGS_CACHE_DIR/images/.jpg` + `DATAPIPE_TAGS_CACHE_DIR/gt.json` +(`{ ".jpg": {"bboxes": [[x1,y1,x2,y2]], "labels": ["cat"|"dog"]}, ... }`). If that cache is +present, the load step reads images + ground truth from it (no COCO fetches at all). Build it by +downloading the COCO cat/dog subset elsewhere and copying the folder to `DATAPIPE_TAGS_CACHE_DIR` +(default `/tmp/datapipe-tags-cache`). + ## What you get - `detection_model_train` — trained model(s). - `detection_model_train__metrics_on_subset` — overall metrics per model. diff --git a/examples/detection_tags/detection/steps.py b/examples/detection_tags/detection/steps.py index dde2571e..e9649472 100644 --- a/examples/detection_tags/detection/steps.py +++ b/examples/detection_tags/detection/steps.py @@ -80,14 +80,43 @@ def load_batch(df_request: pd.DataFrame): return (pd.DataFrame(columns=s3_cols), pd.DataFrame(columns=gt_cols), pd.DataFrame(columns=tag_cols), pd.DataFrame(columns=it_cols)) - id_to_img, anns = _coco_annotations_by_image() - cat_dog_ids = sorted(anns.keys()) - random.seed(RNG_SEED) - random.shuffle(cat_dog_ids) + # Prefer a pre-staged local cache (fast, no COCO): CACHE/images/ + CACHE/gt.json. + # Download the cache once on a fast machine and copy it here — avoids per-image COCO fetches. + cache_gt_path = CACHE / "gt.json" + use_cache = cache_gt_path.exists() and (CACHE / "images").is_dir() + if use_cache: + cache_gt = json.loads(cache_gt_path.read_text()) + pool = sorted(cache_gt.keys()) # deterministic order + random.seed(RNG_SEED) + random.shuffle(pool) + anns = None + else: + id_to_img, anns = _coco_annotations_by_image() + by_id = sorted(anns.keys()) + random.seed(RNG_SEED) + random.shuffle(by_id) + pool = [id_to_img[i]["file_name"] for i in by_id] # file names, same order semantics fs = fsspec.filesystem("s3", **input_storage_options()) bucket = input_bucket() + def raw_bytes_and_gt(fn: str): + if use_cache: + data = (CACHE / "images" / fn).read_bytes() + g = cache_gt[fn] + return data, g["bboxes"], g["labels"] + # COCO fallback + # map filename back to id via anns lookup (filename -> id): rebuild once + raw = _get(COCO_IMG_BASE + fn).content + iid = _fn_to_id[fn] + boxes = [[int(a["bbox"][0]), int(a["bbox"][1]), + int(a["bbox"][0] + a["bbox"][2]), int(a["bbox"][1] + a["bbox"][3])] for a in anns[iid]] + labels = [COCO_CAT_IDS[a["category_id"]] for a in anns[iid]] + return raw, boxes, labels + + if not use_cache: + _fn_to_id = {id_to_img[i]["file_name"]: i for i in anns.keys()} + s3_rows, gt_rows, tag_rows, tag_defs = [], [], [], {} for _, req in df_request.iterrows(): n = int(req["n"]) @@ -95,10 +124,9 @@ def load_batch(df_request: pd.DataFrame): tag = req.get("tag") or None darken = req.get("darken") darken = float(darken) if darken is not None and str(darken) != "" else None - for iid in cat_dog_ids[offset: offset + n]: - fn = id_to_img[iid]["file_name"] + for fn in pool[offset: offset + n]: stem, ext = os.path.splitext(fn) - raw = _get(COCO_IMG_BASE + fn).content + raw, boxes, labels = raw_bytes_and_gt(fn) if darken is not None: name = f"{stem}__{tag or 'dark'}{ext}" buf = io.BytesIO() @@ -107,9 +135,6 @@ def load_batch(df_request: pd.DataFrame): else: name, data = fn, raw fs.pipe(f"{bucket}/images/{name}", data) - boxes = [[int(a["bbox"][0]), int(a["bbox"][1]), - int(a["bbox"][0] + a["bbox"][2]), int(a["bbox"][1] + a["bbox"][3])] for a in anns[iid]] - labels = [COCO_CAT_IDS[a["category_id"]] for a in anns[iid]] s3_rows.append({"image_name": name, "image_url": input_image_url(name)}) gt_rows.append({"image_name": name, "bboxes": boxes, "labels": labels}) if tag: From 196a90c771cc1b39a9332ba69a0c52a777625b95 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 6 Jul 2026 15:26:20 +0300 Subject: [PATCH 25/32] detection_tags skill: note transitive polars must be forced to polars-lts-cpu (pin alone insufficient) --- .claude/skills/setup-detection-tags/SKILL.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.claude/skills/setup-detection-tags/SKILL.md b/.claude/skills/setup-detection-tags/SKILL.md index f409aa67..40692b10 100644 --- a/.claude/skills/setup-detection-tags/SKILL.md +++ b/.claude/skills/setup-detection-tags/SKILL.md @@ -60,7 +60,9 @@ so the aggregation is correct). ## Troubleshooting (may already be fixed — verify against current files) - **`SIGILL` / `Illegal instruction` in the training subprocess** → `polars` built for a CPU newer - than the host (pre-AVX2). This example pins `polars-lts-cpu`; if it still happens, reinstall it. + than the host (pre-AVX2). The `polars-lts-cpu` pin is **not enough** on its own: the regular `polars` + comes in transitively (datapipe-ml/core) and both install the same `polars` module. After `uv sync` + force lts-cpu to win: `uv pip uninstall polars polars-lts-cpu && uv pip install polars-lts-cpu==1.33.1`. - **`No labels found` / every image "corrupt: No module named 'pi_heif'`** → ultralytics image verification needs `pi_heif` (pinned here); reinstall if missing. - **`No ground truth` at freeze** → `image__ground_truth.image_name` must match `s3_images.image_name` From 804132501d2513c25e8297e6b294785573df3080 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 6 Jul 2026 15:56:11 +0300 Subject: [PATCH 26/32] detection_tags skill: branch on demo(test) vs real data; epochs 50->10 for faster demo iteration --- .claude/skills/setup-detection-tags/SKILL.md | 20 ++++++++++++++++++++ examples/detection_tags/detection/app.py | 2 +- 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/.claude/skills/setup-detection-tags/SKILL.md b/.claude/skills/setup-detection-tags/SKILL.md index 40692b10..b907880d 100644 --- a/.claude/skills/setup-detection-tags/SKILL.md +++ b/.claude/skills/setup-detection-tags/SKILL.md @@ -12,6 +12,26 @@ Minimal detection pipeline whose whole point is **tags**: train a baseline, load batch**, retrain, and watch the metric on that tag rise in a `tag_metrics` table. Ground truth is **injected** (COCO labels, lowercase `cat`/`dog`) — no human annotation — so it runs unattended. +**First, ask: demo (test) or real data?** — the whole flow branches on this. + +- **Demo / test** → run it unattended on the built-in COCO cat/dog data. Do it in this order and + *narrate each step*: + 1. deploy (services + `uv sync` + `db create-all`), load the **base** batch, run `stage=train`; + 2. **show the metrics** (`detection_model_train__metrics_on_subset`) for the baseline model; + 3. say: *"there's a `night` low-light tagged scenario — want to add it and retrain, or stop here?"* + 4. if yes → load the **night** batch (`--tag night --darken 0.25`), run `stage=train` again; + 5. after the run, **show the metrics again** — both `metrics_on_subset` and **`tag_metrics`** (the + `night/val` recall for the baseline vs the retrained model). + +- **Real data** → don't guess; gather everything up front and fill it in explicitly: + - **images + ground truth**: where are the images, and where do the boxes/labels come from — + a labelled dataset to inject, or real annotation? (this example has no Label Studio; GT is injected) + - **classes**: the real class names → set `DETECTION_CLASSES` / the GT labels to match exactly (casing!) + - **the tag/scenario**: what is the tagged case (e.g. night, occluded, a site) and which images carry it + - **storage + DB**: which S3/MinIO bucket (`DATAPIPE_TAGS_DIR`) and which Postgres/schema (`DB_URL`/`DB_SCHEMA`) + - **GPU + training size**: enough data and epochs that metrics are meaningful (see the note below) + Then wire the loader to the real source instead of the COCO `load_batch` step. + **Ask first — don't assume (only the unresolved):** which Postgres + which database for `DB_URL` (don't target an existing DB or drop a `localhost` default without confirming); reuse an existing venv/`uv` env or a fresh one? GPU available (training is GPU)? surface stage logs or run quiet? diff --git a/examples/detection_tags/detection/app.py b/examples/detection_tags/detection/app.py index 97f9ecb4..bfc00fb8 100644 --- a/examples/detection_tags/detection/app.py +++ b/examples/detection_tags/detection/app.py @@ -68,7 +68,7 @@ working_dir=str(DATAPIPE_DIR), tmp_folder=datapipe_tmp_folder(), yolov8_train_configs=[ - YoloV8_TrainingConfig(model="yolov8n.pt", imgsz=320, batch=10, epochs=50, exist_ok=True) + YoloV8_TrainingConfig(model="yolov8n.pt", imgsz=320, batch=10, epochs=10, exist_ok=True) ], sync_config=TrainingSyncConfig(enabled=True, interval_s=30, retries=3, retry_sleep_s=30), resume_config=TrainingResumeConfig( From 697d13ce1e898d66412eb13195a5af40e4385473 Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 6 Jul 2026 16:11:51 +0300 Subject: [PATCH 27/32] Lock validated demo params: detection_tags base 450 + night 50 (500 total), gamma 0.25, epochs 10; add pi-heif to e2e deps --- .claude/skills/setup-detection-tags/SKILL.md | 8 ++++---- examples/detection_tags/README.md | 4 ++-- examples/detection_tags/scripts/add_request.py | 2 +- examples/e2e_template/pyproject.toml | 1 + 4 files changed, 8 insertions(+), 7 deletions(-) diff --git a/.claude/skills/setup-detection-tags/SKILL.md b/.claude/skills/setup-detection-tags/SKILL.md index b907880d..d97daff5 100644 --- a/.claude/skills/setup-detection-tags/SKILL.md +++ b/.claude/skills/setup-detection-tags/SKILL.md @@ -54,9 +54,9 @@ run the load step; it downloads COCO cat/dog, uploads to MinIO, and emits `s3_im (+ tag). Labels are lowercase `cat`/`dog`; `image_name` = object basename so all joins line up. ```bash # from examples/detection_tags/detection -python ../scripts/add_request.py --id base --n 120 +python ../scripts/add_request.py --id base --n 450 datapipe step --labels=stage=load run # batch 1: base cat/dog -python ../scripts/add_request.py --id night --n 40 --offset 120 --tag night --darken 0.1 +python ../scripts/add_request.py --id night --n 50 --offset 450 --tag night --darken 0.25 datapipe step --labels=stage=load run # batch 2: tagged scenario ``` @@ -75,7 +75,7 @@ so the aggregation is correct). ## Two-model demo (baseline vs retrained) 1. Load batch 1 → `datapipe step --labels=stage=train run` → model A (no tag in training). -2. Load batch 2 (`--tag night --darken 0.1`) → `datapipe step --labels=stage=train run` → model B. +2. Load batch 2 (`--tag night --darken 0.25`) → `datapipe step --labels=stage=train run` → model B. 3. Read `tag_metrics`: `night/val` ≈ low for A, higher for B. ## Troubleshooting (may already be fixed — verify against current files) @@ -88,5 +88,5 @@ so the aggregation is correct). - **`No ground truth` at freeze** → `image__ground_truth.image_name` must match `s3_images.image_name` (the load step uses the object basename for both); a mismatched key makes the join empty. - **Metrics 0 on a trained model** → tiny/noisy val makes "best epoch" latch onto an early - checkpoint; use enough data (default batches) and the shipped 50-epoch config. + checkpoint; use enough data (~500 total via the default batches) and the shipped epoch config. - **Training exits 0 but no model** → datapipe swallows step errors; check `detection_training_status`. diff --git a/examples/detection_tags/README.md b/examples/detection_tags/README.md index b15fd3f7..7001c383 100644 --- a/examples/detection_tags/README.md +++ b/examples/detection_tags/README.md @@ -21,11 +21,11 @@ Loading is a real pipeline step (`stage=load`): add a request row, then run the COCO cat/dog, uploads to MinIO, and produces `s3_images` + ground truth (+ a tag) — no human labelling. ```bash # from examples/detection_tags/detection -python ../scripts/add_request.py --id base --n 120 +python ../scripts/add_request.py --id base --n 450 datapipe step --labels=stage=load run # batch 1: base cat/dog datapipe step --labels=stage=train run # -> model A (baseline, no tag in training) -python ../scripts/add_request.py --id night --n 40 --offset 120 --tag night --darken 0.1 +python ../scripts/add_request.py --id night --n 50 --offset 450 --tag night --darken 0.25 datapipe step --labels=stage=load run # batch 2: tagged low-light scenario datapipe step --labels=stage=train run # -> model B (tag in training) datapipe step --labels=stage=count-metrics run # (re)compute metrics incl. tag_metrics diff --git a/examples/detection_tags/scripts/add_request.py b/examples/detection_tags/scripts/add_request.py index 43a08cdb..56539876 100644 --- a/examples/detection_tags/scripts/add_request.py +++ b/examples/detection_tags/scripts/add_request.py @@ -23,7 +23,7 @@ def main() -> int: ap = argparse.ArgumentParser() ap.add_argument("--id", required=True, help="request_id (unique per batch)") - ap.add_argument("--n", type=int, default=120) + ap.add_argument("--n", type=int, default=450) ap.add_argument("--offset", type=int, default=0, help="skip the first OFFSET picked COCO images") ap.add_argument("--tag", default=None) ap.add_argument("--darken", type=float, default=None, help="gamma < 1 darkens (e.g. 0.1)") diff --git a/examples/e2e_template/pyproject.toml b/examples/e2e_template/pyproject.toml index f3a39ebd..02b9ca70 100644 --- a/examples/e2e_template/pyproject.toml +++ b/examples/e2e_template/pyproject.toml @@ -10,6 +10,7 @@ dependencies = [ "stringzilla==4.4.0", # needed for albumentations "fiftyone==1.17.0", "polars-lts-cpu==1.33.1", # for ultralytics on old CPUs + "pi-heif", # ultralytics image verification imports it ] [[tool.uv.index]] From 39508f2495b5e095e6472634acbd9b2e38e5ac7c Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Mon, 6 Jul 2026 16:14:14 +0300 Subject: [PATCH 28/32] detection_tags: document creating the DB schema before db create-all (fresh deploy) --- .claude/skills/setup-detection-tags/SKILL.md | 5 ++++- examples/detection_tags/README.md | 1 + 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/.claude/skills/setup-detection-tags/SKILL.md b/.claude/skills/setup-detection-tags/SKILL.md index d97daff5..38f0d8e7 100644 --- a/.claude/skills/setup-detection-tags/SKILL.md +++ b/.claude/skills/setup-detection-tags/SKILL.md @@ -45,7 +45,10 @@ changed; trust the `*_training_status` table, not the exit code. On an unclear f cp .env.example .env && set -a && source .env && set +a # DB_URL, S3/MinIO, DATAPIPE_TAGS_DIR docker compose up -d # postgres + minio ONLY (no mongo, no Label Studio) uv sync # cu124 torch + polars-lts-cpu + pi-heif baked in -cd detection && datapipe db create-all +cd detection +# db create-all makes the tables, NOT the schema — create it first: +psql "$DB_URL" -c "CREATE SCHEMA IF NOT EXISTS $DB_SCHEMA" # or: docker exec psql -U postgres -c "CREATE SCHEMA IF NOT EXISTS datapipe_tags" +datapipe db create-all ``` ## Two-step data load — via datapipe steps (no annotation) diff --git a/examples/detection_tags/README.md b/examples/detection_tags/README.md index 7001c383..8356a89b 100644 --- a/examples/detection_tags/README.md +++ b/examples/detection_tags/README.md @@ -13,6 +13,7 @@ cp .env.example .env && set -a && source .env && set +a docker compose up -d # postgres + minio only uv sync # cu124 torch, polars-lts-cpu, pi-heif baked in cd detection +psql "$DB_URL" -c "CREATE SCHEMA IF NOT EXISTS $DB_SCHEMA" # db create-all makes tables, not the schema datapipe db create-all ``` From 9ebdce3cf8989575012561a8db472411ef9c831e Mon Sep 17 00:00:00 2001 From: Renat Shakirov Date: Mon, 6 Jul 2026 14:07:45 +0000 Subject: [PATCH 29/32] detection_tags: expose TP/FP/FN in tag_metrics compute_tag_metrics already summed per-image TP/FP/FN to derive precision/recall/f1 but did not return them; surface them as calc__TP/FP/FN and add the columns to the tag_metrics table. Co-Authored-By: Claude Opus 4.8 (1M context) --- examples/detection_tags/detection/data.py | 3 +++ examples/detection_tags/detection/steps.py | 6 +++++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/examples/detection_tags/detection/data.py b/examples/detection_tags/detection/data.py index 4fce06e8..9ab7da82 100644 --- a/examples/detection_tags/detection/data.py +++ b/examples/detection_tags/detection/data.py @@ -65,6 +65,9 @@ def _t(name, schema): Column("subset_id", String, primary_key=True), Column("calc__images_support", Integer), Column("calc__support", Integer), + Column("calc__TP", Integer), + Column("calc__FP", Integer), + Column("calc__FN", Integer), Column("calc__precision", Float), Column("calc__recall", Float), Column("calc__f1_score", Float), diff --git a/examples/detection_tags/detection/steps.py b/examples/detection_tags/detection/steps.py index e9649472..fc7d37ff 100644 --- a/examples/detection_tags/detection/steps.py +++ b/examples/detection_tags/detection/steps.py @@ -163,7 +163,8 @@ def split_df_train_val(df: pd.DataFrame, subset_df: pd.DataFrame, primary_keys: # --- tag metrics ---------------------------------------------------------------- def compute_tag_metrics(df_metrics_on_image: pd.DataFrame, df_image_tag: pd.DataFrame) -> pd.DataFrame: cols = ["detection_model_id", "tag_id", "subset_id", "calc__images_support", - "calc__support", "calc__precision", "calc__recall", "calc__f1_score"] + "calc__support", "calc__TP", "calc__FP", "calc__FN", + "calc__precision", "calc__recall", "calc__f1_score"] if df_metrics_on_image.empty or df_image_tag.empty: return pd.DataFrame(columns=cols) m = df_metrics_on_image.merge(df_image_tag[["image_name", "tag_id"]], on="image_name") @@ -179,4 +180,7 @@ def compute_tag_metrics(df_metrics_on_image: pd.DataFrame, df_image_tag: pd.Data g["calc__f1_score"] = (2 * p * r / (p + r)).where((p + r) > 0, 0.0) g["calc__support"] = (tp + fn).astype(int) g["calc__images_support"] = g["images_support"].astype(int) + g["calc__TP"] = tp.astype(int) + g["calc__FP"] = fp.astype(int) + g["calc__FN"] = fn.astype(int) return g[cols] From 37f5ab5e6df8871337036d77dda17a34a26d7738 Mon Sep 17 00:00:00 2001 From: Renat Shakirov Date: Mon, 6 Jul 2026 15:09:45 +0000 Subject: [PATCH 30/32] detection_tags skill: surface baseline tag metric before retraining (problem->fix flow) Reorder the demo so the low night/val tag metric is shown on the baseline model first (metric stages run on model A over the new night batch, no training) and framed as the problem, then retraining closes it. Note the inference->count-metrics->tag-metrics ordering and that night/val is small so the rise is directional. Co-Authored-By: Claude Opus 4.8 (1M context) --- .claude/skills/setup-detection-tags/SKILL.md | 31 +++++++++++++++----- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/.claude/skills/setup-detection-tags/SKILL.md b/.claude/skills/setup-detection-tags/SKILL.md index 38f0d8e7..dbca4bb7 100644 --- a/.claude/skills/setup-detection-tags/SKILL.md +++ b/.claude/skills/setup-detection-tags/SKILL.md @@ -15,13 +15,21 @@ batch**, retrain, and watch the metric on that tag rise in a `tag_metrics` table **First, ask: demo (test) or real data?** — the whole flow branches on this. - **Demo / test** → run it unattended on the built-in COCO cat/dog data. Do it in this order and - *narrate each step*: - 1. deploy (services + `uv sync` + `db create-all`), load the **base** batch, run `stage=train`; + *narrate each step*. The arc is deliberately **problem → fix**: first make the baseline's weakness + on the tag *visible*, then close it by retraining. + 1. deploy (services + `uv sync` + `db create-all`), load the **base** batch, run `stage=train` + → model A; 2. **show the metrics** (`detection_model_train__metrics_on_subset`) for the baseline model; 3. say: *"there's a `night` low-light tagged scenario — want to add it and retrain, or stop here?"* - 4. if yes → load the **night** batch (`--tag night --darken 0.25`), run `stage=train` again; - 5. after the run, **show the metrics again** — both `metrics_on_subset` and **`tag_metrics`** (the - `night/val` recall for the baseline vs the retrained model). + 4. if yes → load the **night** batch (`--tag night --darken 0.25`); + 5. **before retraining, measure the baseline on the tag** — run the metric stages on model A over + the new night data *without* training: `stage=train-prepare` (split) → `stage=inference` → + `stage=count-metrics` → `stage=tag-metrics`. **Show `tag_metrics`**: `night/val` recall is **low** + — say plainly *"the baseline barely sees in the dark — that's the problem we'll fix"* + (this works because inference/metrics/tag_metrics depend only on the *existing* model, not on training); + 6. **now fix it** → run `stage=train` again → model B (night is now in training); + 7. **show the metrics again** — both `metrics_on_subset` and **`tag_metrics`**: the `night/val` recall + rises from the baseline (A) to the retrained model (B). That rise is the payoff. - **Real data** → don't guess; gather everything up front and fill it in explicitly: - **images + ground truth**: where are the images, and where do the boxes/labels come from — @@ -78,8 +86,17 @@ so the aggregation is correct). ## Two-model demo (baseline vs retrained) 1. Load batch 1 → `datapipe step --labels=stage=train run` → model A (no tag in training). -2. Load batch 2 (`--tag night --darken 0.25`) → `datapipe step --labels=stage=train run` → model B. -3. Read `tag_metrics`: `night/val` ≈ low for A, higher for B. +2. Load batch 2 (`--tag night --darken 0.25`). **Before retraining**, measure the baseline on the + tag *without training* — run only the metric stages on model A over the new data: + `stage=train-prepare` → `stage=inference` → `stage=count-metrics` → `stage=tag-metrics`, then read + `tag_metrics`: `night/val` is **low** (A never trained on night). This is the "problem" the demo surfaces. +3. Retrain → `datapipe step --labels=stage=train run` → model B (night now in training). +4. Read `tag_metrics` again: `night/val` recall rises from A to B — the tagged batch in training fixed it. + +> `tag_metrics` cannot be computed straight after `inference`: it aggregates from +> `metrics_on_image`, which the `count-metrics` stage produces — so `inference → count-metrics → +> tag-metrics`, in that order. The `night/val` set is small, so treat the rise as **directional**; +> enlarge the night batch (or hold night out as a fixed val set) for a more dramatic gap. ## Troubleshooting (may already be fixed — verify against current files) - **`SIGILL` / `Illegal instruction` in the training subprocess** → `polars` built for a CPU newer From 70c8c74c33575ebe24b6b4adbfb773f426aeaa3d Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Tue, 7 Jul 2026 10:13:09 +0300 Subject: [PATCH 31/32] detection_tags: fixed-val demo + FiftyOne + two-part checkpoint flow Fixed val (correct A/B metric comparison): - add_request.py --subset pins a batch to train/val; load_batch emits image__subset_hint; split honors it (random split only fills un-pinned images). Load base-val + night-val up front so val is frozen; add night-train only for the retrain. FiftyOne (ported from e2e_template/image_detection): - stage=fiftyone: download_images + publish GT and per-model predictions (predictions_model_a=baseline, predictions_model_b=retrained; slot by sorted model id); tag/subset as sample fields. mongo added to docker-compose; datapipe-ml[fiftyone]+fiftyone dep. Tag dimension: numeric tag_id (deterministic via config.TAG_IDS) + tag_name + tag_description; image__tag/tag_metrics key on tag_id, join tag for readable name. Bug fixes: - darken=NaN: a NULL numeric column arrives as NaN when several requests load in one batch; pd.isna guards so base batches are not darkened with gamma=NaN into garbage. - download_images: don't pass S3 storage options to fsspec.open of a public HTTP URL. Defaults/robustness: DB_SCHEMA defaults to public; standard ports. SKILL.md: two-part checkpoint flow (demo-only), fixed-val rationale, pre-flight port/schema checks, always-print-full-table rule, service-watch guidance, troubleshooting (polars-lts on pre-AVX2, count-metrics ordering, double-train, port/tunnel/proxy pitfalls); snapshot/rehearse as inline pg_dump/psql (no committed demo scripts). Co-Authored-By: Claude Opus 4.8 --- .claude/skills/setup-detection-tags/SKILL.md | 333 ++++++++++++++---- examples/detection_tags/.env.example | 8 +- examples/detection_tags/README.md | 97 +++-- examples/detection_tags/detection/app.py | 37 +- examples/detection_tags/detection/config.py | 25 +- examples/detection_tags/detection/data.py | 50 ++- examples/detection_tags/detection/steps.py | 127 ++++++- examples/detection_tags/docker-compose.yml | 11 + examples/detection_tags/pyproject.toml | 3 +- .../detection_tags/scripts/add_request.py | 22 +- 10 files changed, 569 insertions(+), 144 deletions(-) diff --git a/.claude/skills/setup-detection-tags/SKILL.md b/.claude/skills/setup-detection-tags/SKILL.md index 38f0d8e7..088b899b 100644 --- a/.claude/skills/setup-detection-tags/SKILL.md +++ b/.claude/skills/setup-detection-tags/SKILL.md @@ -2,94 +2,273 @@ name: setup-detection-tags description: > Use when setting up or running examples/detection_tags — a self-contained datapipe detection - demo built around tags (per-scenario metrics), with no Label Studio and no FiftyOne, deployed - from scratch; or when demoing "add a tagged batch, retrain, watch the tag metric rise". + demo built around tags (per-scenario metrics), split into two parts around a checkpoint (train a + baseline, then add a tagged TRAIN batch and retrain, watching the tag metric rise), with a + FiftyOne view and injected ground truth (no Label Studio). Also for "add a tagged batch, retrain, + watch the tag metric rise" and for rehearsing that retraining demo from a saved checkpoint. --- -# detection_tags (tags demo — no Label Studio, no FiftyOne) - -Minimal detection pipeline whose whole point is **tags**: train a baseline, load a **tagged scenario -batch**, retrain, and watch the metric on that tag rise in a `tag_metrics` table. Ground truth is -**injected** (COCO labels, lowercase `cat`/`dog`) — no human annotation — so it runs unattended. - -**First, ask: demo (test) or real data?** — the whole flow branches on this. - -- **Demo / test** → run it unattended on the built-in COCO cat/dog data. Do it in this order and - *narrate each step*: - 1. deploy (services + `uv sync` + `db create-all`), load the **base** batch, run `stage=train`; - 2. **show the metrics** (`detection_model_train__metrics_on_subset`) for the baseline model; - 3. say: *"there's a `night` low-light tagged scenario — want to add it and retrain, or stop here?"* - 4. if yes → load the **night** batch (`--tag night --darken 0.25`), run `stage=train` again; - 5. after the run, **show the metrics again** — both `metrics_on_subset` and **`tag_metrics`** (the - `night/val` recall for the baseline vs the retrained model). - -- **Real data** → don't guess; gather everything up front and fill it in explicitly: - - **images + ground truth**: where are the images, and where do the boxes/labels come from — - a labelled dataset to inject, or real annotation? (this example has no Label Studio; GT is injected) - - **classes**: the real class names → set `DETECTION_CLASSES` / the GT labels to match exactly (casing!) - - **the tag/scenario**: what is the tagged case (e.g. night, occluded, a site) and which images carry it - - **storage + DB**: which S3/MinIO bucket (`DATAPIPE_TAGS_DIR`) and which Postgres/schema (`DB_URL`/`DB_SCHEMA`) - - **GPU + training size**: enough data and epochs that metrics are meaningful (see the note below) - Then wire the loader to the real source instead of the COCO `load_batch` step. - -**Ask first — don't assume (only the unresolved):** which Postgres + which database for `DB_URL` -(don't target an existing DB or drop a `localhost` default without confirming); reuse an existing -venv/`uv` env or a fresh one? GPU available (training is GPU)? surface stage logs or run quiet? - -**How to work:** propose a short plan and get a go-ahead; show each stage's logs and report what -changed; trust the `*_training_status` table, not the exit code. On an unclear failure re-run with -`datapipe --debug … run` sent to a file + `grep`, not inline. - -## Deploy from scratch +# detection_tags (tags demo — two-part, FiftyOne, no Label Studio) + +Detection pipeline whose whole point is **tags**: train a baseline (model A), then add a **tagged +scenario TRAIN batch** and retrain (model B), and watch recall on that tag rise in a `tag_metrics` +table. Ground truth is **injected** (COCO labels, lowercase `cat`/`dog`) — no human annotation. + +The two-part / checkpoint split below is a **presentation device for the live demo only** — it lets +you prep part 1 ahead of time, present part 2, and rehearse it. It is NOT required to use the pipeline: +for real work you just run the stages end to end (the frozen-val data layout still matters — that's +about metric correctness, not the demo). See "Real data" at the end. + +The demo is deliberately **two parts around a checkpoint**, so you can prepare part 1 ahead of time +and present/rehearse part 2 (the retraining) as often as you like: + +- **Part 1 (prep, before the audience):** deploy → load `base-train` + `base-val` + `night-val` + (val is frozen up front) → train **model A** → compute all metrics → **take a checkpoint** → build + the FiftyOne view. Stop here. +- **Part 2 (the live demo):** show model A's metrics + FiftyOne → *ask "ready to retrain?"* → load + the tagged TRAIN batch `night-train` → retrain **model B** → show the metrics again: `night` recall + rises from A to B. Rehearse by resetting to the checkpoint. + +## Ask first — don't assume (only the unresolved) + +1. **Demo (test) or real data?** — the whole flow branches on this (see "Real data" at the end). +2. **Is an env/compose already up?** Check before deploying: `docker compose ls` / + `docker ps` on the target host, and whether a `.venv` already has torch+fiftyone. **Reuse it** + (skip `uv sync` / `compose up` / `db create-all` as appropriate) rather than redeploying — only + deploy the pieces that are missing. Don't drop a `localhost` DB default or target an existing DB + without confirming. +3. **GPU available?** Training is GPU. On a remote cluster (e.g. epoch8 gpu5) run the whole stack + there over SSH; note the host may have a **read-only home** and **no AVX2** (see Troubleshooting). +4. **Surface stage logs or run quiet?** Default: show each stage and report what changed. + +## Frozen val — why the data is loaded in this order (READ THIS) + +A metric on `val` is an **aggregate over the set of images currently in val**. Change that set and +the number moves *even if the model is byte-identical*. If you load the tagged batch as one blob and +let the random split scatter it across train/val at retrain time, model A's val (and tag/val) numbers +shift between the two measurements — an apples-to-oranges comparison that confuses everyone. + +Fix: **freeze val up front.** Load `base-val` and `night-val` (subset pinned to `val`) *before* +training model A, and only add `night-train` (pinned to `train`) later. Then val never changes; model +A is measured once on the full frozen val and its numbers are stable, and B is compared on the exact +same val. `add_request.py --subset train|val` pins a batch; the load step emits `image__subset_hint` +and the split step honors it (random split only fills images without a hint). + +Batches (COCO cat/dog; the pre-staged cache holds **500 images**, so keep the total ≤ 500 or expand +the cache — see Troubleshooting): + +| batch | n | offset | subset | tag | darken | when | +|-------|---|--------|--------|-----|--------|------| +| `base-train` | 325 | 0 | train | — | — | part 1 | +| `base-val` | 100 | 325 | val | — | — | part 1 | +| `night-val` | 25 | 425 | val | night | 0.25 | part 1 | +| `night-train`| 50 | 450 | train | night | 0.25 | part 2 | + +Validated on gpu5: model A → train recall ~0.95, night/val recall ~0.09 (blind in the dark); after +part 2, model B → night/train recall ~0.86. Keep night/val small but not tiny (25 imgs / ~45 GT) — +below ~20 the val payoff is pure ±1-box noise and can even go the wrong way. + +## Pre-flight — CHECK what's already there, then ASK (do this before deploying) + +Never assume a clean host. Before `docker compose up` / loading / training, check each resource and, +if it's occupied, ASK the user what to do (reuse / different port / different schema or DB / wipe) — +don't silently claim it or clobber someone else's work. + ```bash -cp .env.example .env && set -a && source .env && set +a # DB_URL, S3/MinIO, DATAPIPE_TAGS_DIR -docker compose up -d # postgres + minio ONLY (no mongo, no Label Studio) -uv sync # cu124 torch + polars-lts-cpu + pi-heif baked in +# 1) ports free? (host) — postgres 5432, minio 9000/9001, mongo 27017, fiftyone 5151, app 8000 +ss -ltn | grep -E ':(5432|9000|9001|27017|5151|8000)\b' || echo "all free" +# 2) is a stack already up? +docker compose ls ; docker ps --format '{{.Names}}\t{{.Ports}}' +# 3) does the target DB/schema already hold this pipeline's tables? +psql "$DB_URL" -c "\dt $DB_SCHEMA.*" 2>/dev/null | grep -qi detection && echo "SCHEMA IN USE" || echo "schema clean" +# 4) is a venv already built (torch+fiftyone)? reuse vs fresh uv sync +``` + +Findings → questions to ask: a **busy port** means another stack (reuse it, or remap ports?); a +**populated schema** means a prior run (resume it, wipe it, or use a different `DB_SCHEMA`/DB?); a +**running compose** (reuse or `down`?). The same check applies on your laptop when a **local tunnel +port is busy** — pick a different left-hand port (`-L 5433:localhost:5432`), don't kill blindly. + +## Deploy from scratch (standard ports; skip pieces already up) + +```bash +cp .env.example .env && set -a && source .env && set +a # DB_URL, S3/MinIO, FIFTYONE_DATABASE_URI +docker compose up -d # postgres + minio + mongo (mongo is for FiftyOne). No Label Studio. +uv sync # cu124 torch + datapipe-ml[torch,fiftyone] + fiftyone + pi-heif +# On a pre-AVX2 host, force the lts polars to win (else the training subprocess SIGILLs): +uv pip uninstall polars polars-lts-cpu && uv pip install polars-lts-cpu==1.33.1 cd detection -# db create-all makes the tables, NOT the schema — create it first: -psql "$DB_URL" -c "CREATE SCHEMA IF NOT EXISTS $DB_SCHEMA" # or: docker exec psql -U postgres -c "CREATE SCHEMA IF NOT EXISTS datapipe_tags" +# DB_SCHEMA defaults to `public` (already exists). Only if you set a dedicated schema (to share the +# Postgres with other pipelines) create it first: psql "$DB_URL" -c "CREATE SCHEMA IF NOT EXISTS $DB_SCHEMA" datapipe db create-all ``` -## Two-step data load — via datapipe steps (no annotation) -Loading is a pipeline step (`stage=load`) driven by rows in the `load_request` table. Add a request, -run the load step; it downloads COCO cat/dog, uploads to MinIO, and emits `s3_images` + ground truth -(+ tag). Labels are lowercase `cat`/`dog`; `image_name` = object basename so all joins line up. +## Part 1 — baseline to checkpoint + +Run `stage=train` as ONE step (it internally does split→freeze→train→inference); surface each stage +from its log and by querying the tables after — do NOT invoke `stage=train-prepare` separately before +`stage=train`, or the intermediate re-freeze retrains the model twice (see Troubleshooting). + ```bash -# from examples/detection_tags/detection -python ../scripts/add_request.py --id base --n 450 -datapipe step --labels=stage=load run # batch 1: base cat/dog -python ../scripts/add_request.py --id night --n 50 --offset 450 --tag night --darken 0.25 -datapipe step --labels=stage=load run # batch 2: tagged scenario +# from examples/detection_tags/detection, with .env sourced +python ../scripts/add_request.py --id base-train --n 325 --offset 0 --subset train +python ../scripts/add_request.py --id base-val --n 100 --offset 325 --subset val +python ../scripts/add_request.py --id night-val --n 25 --offset 425 --subset val --tag night --darken 0.25 +datapipe step --labels=stage=load run # 450 images; val frozen (100 base + 25 night) +# show the frozen split: SELECT subset_id, count(*) FROM image__subset_hint GROUP BY subset_id + +datapipe step --labels=stage=train run # split -> freeze -> train model A -> inference (SHOW every epoch) +datapipe step --labels=stage=count-metrics run # metrics_on_image/subset + tag_metrics +# count-metrics can print "Batches to process 0" right after training (it hasn't seen the fresh +# predictions yet) — RE-RUN it once; then the tables fill. (See Troubleshooting.) + +# SHOW THE FULL TABLES (see the Rule below). To rehearse part 2 later, snapshot the post-A state +# BEFORE the fiftyone stage (demo-only convenience; skip for real use): +docker exec pg_dump -U postgres -n "$DB_SCHEMA" postgres > /tmp/checkpoint.sql + +datapipe step --labels=stage=fiftyone run # download_images -> GT + model-A predictions ``` -## Run +**Stop here (end of stage 1) and hand the baseline over to the user:** +- show the full training log (all epochs) and the **full** metric tables (`metrics_on_subset` + + `tag_metrics`) — `tag_metrics` for model A at `night/val` is **low** (baseline blind in the dark); +- bring up the **FiftyOne App** service so they can browse GT vs model-A predictions, and give them + its address / how to reach it (see "Let the user watch" — specifics depend on the setup); +- then ask whether to proceed to part 2 (retrain). That's the problem part 2 fixes. + +Surface every epoch from the training log (it streams to a file, e.g. `/tmp/train_A.log`): ```bash -datapipe run # load → split → freeze → train → inference → metrics → tag_metrics -# or by stage: datapipe step --labels=stage=train run +grep -oE "[0-9]+/[0-9]+ +[0-9.]+G +[0-9.]+ +[0-9.]+ +[0-9.]+" /tmp/train_A.log # per-epoch box/cls/dfl loss +grep -E "^ +all +[0-9]+ +[0-9]+" /tmp/train_A.log # per-epoch val P/R/mAP50/mAP50-95 ``` -## The payoff -`tag_metrics(detection_model_id, tag_id, subset_id)` → precision/recall/f1. Compare the baseline -(trained before batch 2) vs the retrained model at `tag=night, subset=val` — recall on the tag rises -once the tagged batch is in training. `tag`/`image__tag` are external inputs (produced by the `load` step); -`tag_metrics` is a real pipeline step (`steps.compute_tag_metrics`, `transform_keys=["detection_model_id"]` -so the aggregation is correct). - -## Two-model demo (baseline vs retrained) -1. Load batch 1 → `datapipe step --labels=stage=train run` → model A (no tag in training). -2. Load batch 2 (`--tag night --darken 0.25`) → `datapipe step --labels=stage=train run` → model B. -3. Read `tag_metrics`: `night/val` ≈ low for A, higher for B. - -## Troubleshooting (may already be fixed — verify against current files) -- **`SIGILL` / `Illegal instruction` in the training subprocess** → `polars` built for a CPU newer - than the host (pre-AVX2). The `polars-lts-cpu` pin is **not enough** on its own: the regular `polars` - comes in transitively (datapipe-ml/core) and both install the same `polars` module. After `uv sync` - force lts-cpu to win: `uv pip uninstall polars polars-lts-cpu && uv pip install polars-lts-cpu==1.33.1`. -- **`No labels found` / every image "corrupt: No module named 'pi_heif'`** → ultralytics image - verification needs `pi_heif` (pinned here); reinstall if missing. -- **`No ground truth` at freeze** → `image__ground_truth.image_name` must match `s3_images.image_name` - (the load step uses the object basename for both); a mismatched key makes the join empty. -- **Metrics 0 on a trained model** → tiny/noisy val makes "best epoch" latch onto an early - checkpoint; use enough data (~500 total via the default batches) and the shipped epoch config. +## Part 2 — retrain and watch the tag metric rise + +```bash +python ../scripts/add_request.py --id night-train --n 50 --offset 450 --subset train --tag night --darken 0.25 +datapipe step --labels=stage=load run # 500 images total +datapipe step --labels=stage=train run # split (night -> TRAIN, val UNCHANGED) + train model B (SHOW every epoch) +# verify val stayed frozen after the split: +# SELECT subset_id, count(*) FILTER (WHERE image_name LIKE '%night%') night, count(*) total +# FROM image__subset s JOIN image__ground_truth USING(image_name) GROUP BY subset_id +datapipe step --labels=stage=count-metrics run # re-run if "0 batches" +datapipe step --labels=stage=fiftyone run # adds predictions_model_b +``` + +`night/val` recall rises from model A to model B — the payoff. Show the full tables again. + +## Rehearse part 2 from the snapshot (demo-only) + +Restore the post-A snapshot and wipe the FiftyOne db, then re-run Part 2 — no retraining of model A, +no image re-download: + +```bash +docker exec psql -U postgres -c "DROP SCHEMA IF EXISTS $DB_SCHEMA CASCADE; CREATE SCHEMA $DB_SCHEMA" +docker exec -i psql -U postgres < /tmp/checkpoint.sql +docker exec mongosh --quiet --eval "db.getSiblingDB('fiftyone').dropDatabase()" +``` + +## RULE: always print the FULL metrics table + +When you show metrics, dump the **whole** table, not a truncated view. Both of these, every time: + +```bash +docker exec psql -U postgres -x -c \ + "SELECT * FROM $DB_SCHEMA.detection_model_train__metrics_on_subset ORDER BY detection_model_id, subset_id" +docker exec psql -U postgres -x -c \ + "SELECT tm.*, t.tag_name FROM $DB_SCHEMA.tag_metrics tm JOIN $DB_SCHEMA.tag t USING(tag_id) \ + ORDER BY tm.detection_model_id, tm.tag_id, tm.subset_id" +``` + +`tag_id` is a numeric surrogate key (deterministic map in `config.TAG_IDS`, e.g. `night`→1); the +`tag` dimension holds `tag_name`/`tag_description`, so join it to show the readable name. + +The payoff comparison: `tag_metrics` at `tag=night, subset=val`, model A vs model B — recall rises. +The night/val set is small (25 images), so treat the rise as **directional**. + +## Let the user watch (offer it — work out the specifics per setup) + +The user can watch three things; make them available and hand over whatever access they need — don't +prescribe fixed commands here and don't leave a committed file behind (it would bake in one machine's +host/ports/paths and mislead on the next). Compute the specifics for the ACTUAL setup at runtime +(local vs remote host, which ports are free, tunnels if remote) and give the user the commands +directly in chat: + +- **tables** in DBeaver (Postgres → the `$DB_SCHEMA` schema): `tag_metrics`, + `detection_model_train__metrics_on_subset`, `detection_training_status`. +- **images** in the FiftyOne App: `ground_truth` vs `predictions_model_a` vs `predictions_model_b`, + filterable by `tag`/`subset`. +- **training progress**: the run streams to a log file — tail it; trust + `detection_training_status.status`, not exit codes. + +**When you START a training run, immediately tell the user how to follow it** (before it finishes), +so they can watch the epochs live — give the log path and the follow command, e.g. +`tail -f /tmp/train_A.log` (or the per-epoch `grep` from Part 1). Do this the moment training kicks +off, for model A and again for model B — not only after it completes. + +When the host is remote these are reached over SSH tunnels; if a local tunnel port is busy, pick a +different left-hand port rather than killing whatever holds it. + +## FiftyOne + +`stage=fiftyone` downloads images locally then publishes to one dataset (`FIFTYONE_DATASET_NAME`, +metadata in MongoDB): fields `ground_truth`, `predictions_model_a` (baseline), `predictions_model_b` +(retrained), plus sample fields `tag` and `subset`. Ported straight from +`examples/e2e_template/image_detection` (same `download_images` + `publish_to_fiftyone` + +`FiftyOneImagesDataTableStore`); the two model fields are assigned by sorted model-id (earliest = +`model_a`). Launch: `fiftyone app launch "$FIFTYONE_DATASET_NAME" --port 5151 --address 0.0.0.0`, +tunnel 5151 (local port MUST equal remote port — the App's WebSocket calls the same port). + +`tag` and `subset` are ordinary **sample fields** (set in-pipeline by `publish_gt_to_fiftyone`, no +scripts) — NOT FiftyOne's native "sample tags". Filter by the **fields** in the sidebar: pick +`tag = night` and `subset = val` — different fields AND together, so you get exactly the night-val +set. (Selecting multiple values inside FiftyOne's native TAGS panel ORs them, which is why we do NOT +mirror tag/subset into native tags.) That isolates where model A misses in the dark; after part 2, +`predictions_model_b` shows model B catching it. Fields stay `predictions_model_a` (baseline, +earliest model id) / `predictions_model_b` (retrained); the exact model ids are in the DB tables. + +## How to work + +Propose a short plan and get a go-ahead; show each stage's logs and report what changed. Trust the +`*_training_status` table, not the exit code. On an unclear failure re-run with `datapipe --debug … run` +to a file + `grep`, not inline. + +## Troubleshooting (verify against current files) + +- **Pre-staged cache is exactly 500 images** → `gt.json` in the cache dir caps the pool. An `offset` + past 500 yields an empty batch silently. Keep the batch total ≤ 500, or rebuild the cache larger + (or point `DATAPIPE_TAGS_CACHE_DIR` at a fresh dir to force a full COCO download). +- **`SIGILL` / `Illegal instruction` in training** → `polars` built for a newer CPU than the host + (pre-AVX2). The `polars-lts-cpu` pin isn't enough alone; the regular `polars` comes in transitively. + After `uv sync`: `uv pip uninstall polars polars-lts-cpu && uv pip install polars-lts-cpu==1.33.1`, + then verify `python -c "import polars; print(polars.__version__)"` is the lts one. +- **`No labels found` / every image "corrupt: No module named 'pi_heif'`** → reinstall `pi-heif`. +- **count-metrics prints "Batches to process 0" right after training** → datapipe hasn't propagated + the fresh predictions in the same pass. Re-run `count-metrics` once; then `metrics_on_image/subset` + and `tag_metrics` fill. +- **Two identical models get trained** → you ran `stage=train-prepare` and then `stage=train` + separately; the split between them changes `image__subset`, which re-freezes the dataset and + retrains. Run `stage=train` as one step (it includes prepare), and show the split with a query. +- **`kill -0 ` waiters, not `pgrep -f ""`** → a `pgrep`/`until` loop whose own + command line contains the pattern (e.g. `stage=load`, `uv sync`, `bin/datapipe step`) matches + ITSELF and never exits. Wait on the captured PID instead. +- **Read-only home on the cluster** (e.g. gpu5 `ml`) → put the repo, `HOME`, uv cache, and FiftyOne + local images under a writable path like `/var/tmp`; `/tmp` and `/var/tmp` are writable. +- **Flaky link to the cluster** (Moscow/RU hosts) → SSH may time out transiently; retry. +- **Metrics 0 on a trained model** → tiny/noisy val latches an early "best" checkpoint; use enough + data (~450 total) and the shipped epoch config. - **Training exits 0 but no model** → datapipe swallows step errors; check `detection_training_status`. + +## Real data + +**Skip the two-part / checkpoint / rehearse machinery — that's demo-only.** For real data just run the +stages end to end (`load` → `train` → `count-metrics` → optionally `fiftyone`); no artificial stop, no +snapshot/restore. `--darken` is also demo-only (it synthesizes the low-light scenario from normal COCO +images — real data has real tagged images, so drop it). Still keep **val frozen** (pin `--subset`) — +that's about metric correctness, not the demo. + +Don't guess; gather up front and wire the loader to the real source instead of the COCO `load_batch`: +images + where boxes/labels come from (a labelled set to inject, or real annotation); real class +names (set `DETECTION_CLASSES` / GT labels to match exactly, casing!); the tag/scenario and which +images carry it; storage + DB (`DATAPIPE_TAGS_DIR`, `DB_URL`/`DB_SCHEMA`); GPU + enough data/epochs +that metrics are meaningful. diff --git a/examples/detection_tags/.env.example b/examples/detection_tags/.env.example index ba970ba4..a10f4deb 100644 --- a/examples/detection_tags/.env.example +++ b/examples/detection_tags/.env.example @@ -7,4 +7,10 @@ export S3_PUBLIC_URL=http://localhost:9000 export DATAPIPE_TAGS_DIR=s3://datapipe-tags export DATAPIPE_TAGS_TMP_DIR=/tmp/datapipe-tags export DB_URL=postgresql://postgres:password@localhost:5432/postgres -export DB_SCHEMA=datapipe_tags +# default schema; set a dedicated one (e.g. datapipe_tags) only if you SHARE this Postgres with +# other pipelines — then `db create-all` needs the schema created first (see README). +export DB_SCHEMA=public +# FiftyOne (stage=fiftyone): dataset metadata in MongoDB, images rendered from local files +export FIFTYONE_DATABASE_URI=mongodb://localhost:27017 +export FIFTYONE_DATASET_NAME=datapipe_detection_tags +export LOCAL_IMAGES_DIR=/tmp/datapipe-tags-fiftyone-images diff --git a/examples/detection_tags/README.md b/examples/detection_tags/README.md index 8356a89b..8c805639 100644 --- a/examples/detection_tags/README.md +++ b/examples/detection_tags/README.md @@ -1,57 +1,78 @@ # detection_tags -A minimal, self-contained datapipe detection example built around **tags** — with **no Label -Studio and no FiftyOne**. Ground truth is injected directly (COCO labels, lowercase `cat`/`dog`), -so the whole thing runs unattended from scratch. +A self-contained datapipe detection example built around **tags** (per-scenario metrics), with a +**FiftyOne** view and **no Label Studio**. Ground truth is injected directly (COCO labels, lowercase +`cat`/`dog`), so it runs unattended from scratch. -The point: train a baseline, then add a **tagged scenario batch**, retrain, and watch the metric on -that tag rise — visible in a dedicated `tag_metrics` table (a real pipeline step). +The point: train a baseline (model A), then add a **tagged scenario TRAIN batch** and retrain +(model B), and watch recall on that tag rise — visible in a dedicated `tag_metrics` table. The demo +is split into **two parts around a checkpoint** so you can prep part 1 and rehearse part 2. ## Deploy from scratch ```bash cp .env.example .env && set -a && source .env && set +a -docker compose up -d # postgres + minio only -uv sync # cu124 torch, polars-lts-cpu, pi-heif baked in +docker compose up -d # postgres + minio + mongo (mongo backs FiftyOne) +uv sync # cu124 torch + datapipe-ml[torch,fiftyone] + fiftyone + pi-heif +# pre-AVX2 host only: force the lts polars to win (else training SIGILLs) +uv pip uninstall polars polars-lts-cpu && uv pip install polars-lts-cpu==1.33.1 cd detection -psql "$DB_URL" -c "CREATE SCHEMA IF NOT EXISTS $DB_SCHEMA" # db create-all makes tables, not the schema +# DB_SCHEMA defaults to `public` (exists already). For a dedicated schema (sharing the Postgres with +# other pipelines) create it first: psql "$DB_URL" -c "CREATE SCHEMA IF NOT EXISTS $DB_SCHEMA" datapipe db create-all ``` -## Two-step data load — via datapipe steps (no annotation) -Loading is a real pipeline step (`stage=load`): add a request row, then run the step. It downloads -COCO cat/dog, uploads to MinIO, and produces `s3_images` + ground truth (+ a tag) — no human labelling. +## Frozen val (why the load order matters) +A `val` metric is an aggregate over whatever images are in val; if you add the tagged batch to val at +retrain time, model A's numbers move and the A-vs-B comparison is invalid. So **freeze val up front**: +load `base-val` + `night-val` (pinned `--subset val`) before training A, and add `night-train` +(`--subset train`) only for part 2. `add_request.py --subset` pins a batch (`image__subset_hint`), +the split step honors it. The pre-staged cache holds **500 images**, so keep the total ≤ 500. + +## Part 1 — baseline to checkpoint ```bash # from examples/detection_tags/detection -python ../scripts/add_request.py --id base --n 450 -datapipe step --labels=stage=load run # batch 1: base cat/dog -datapipe step --labels=stage=train run # -> model A (baseline, no tag in training) - -python ../scripts/add_request.py --id night --n 50 --offset 450 --tag night --darken 0.25 -datapipe step --labels=stage=load run # batch 2: tagged low-light scenario -datapipe step --labels=stage=train run # -> model B (tag in training) -datapipe step --labels=stage=count-metrics run # (re)compute metrics incl. tag_metrics +python ../scripts/add_request.py --id base-train --n 325 --offset 0 --subset train +python ../scripts/add_request.py --id base-val --n 100 --offset 325 --subset val +python ../scripts/add_request.py --id night-val --n 25 --offset 425 --subset val --tag night --darken 0.25 +datapipe step --labels=stage=load run +datapipe step --labels=stage=train run # model A +datapipe step --labels=stage=count-metrics run # re-run once if it prints "Batches to process 0" +# (demo-only) snapshot the post-A state to rehearse part 2 later, before the fiftyone stage: +docker exec pg_dump -U postgres -n "$DB_SCHEMA" postgres > /tmp/checkpoint.sql +datapipe step --labels=stage=fiftyone run # GT + model-A predictions into FiftyOne ``` -Or just `datapipe run` after adding request rows to run every stage. -### Fast data on a slow/blocked network (pre-staged cache) -The `load` step downloads images from COCO per request, which is slow on a high-latency or -throttled link. To avoid it, pre-stage a cache once from a fast machine: -`DATAPIPE_TAGS_CACHE_DIR/images/.jpg` + `DATAPIPE_TAGS_CACHE_DIR/gt.json` -(`{ ".jpg": {"bboxes": [[x1,y1,x2,y2]], "labels": ["cat"|"dog"]}, ... }`). If that cache is -present, the load step reads images + ground truth from it (no COCO fetches at all). Build it by -downloading the COCO cat/dog subset elsewhere and copying the folder to `DATAPIPE_TAGS_CACHE_DIR` -(default `/tmp/datapipe-tags-cache`). +## Part 2 — retrain and watch the tag metric rise +```bash +python ../scripts/add_request.py --id night-train --n 50 --offset 450 --subset train --tag night --darken 0.25 +datapipe step --labels=stage=load run +datapipe step --labels=stage=train run # model B (night now in training) +datapipe step --labels=stage=count-metrics run # re-run once if "0 batches" +datapipe step --labels=stage=fiftyone run # adds predictions_model_b +``` +Rehearse (demo-only): restore the snapshot + wipe the FiftyOne db, then re-run part 2 — no retraining +of model A: +```bash +docker exec psql -U postgres -c "DROP SCHEMA IF EXISTS $DB_SCHEMA CASCADE; CREATE SCHEMA $DB_SCHEMA" +docker exec -i psql -U postgres < /tmp/checkpoint.sql +docker exec mongosh --quiet --eval "db.getSiblingDB('fiftyone').dropDatabase()" +``` ## What you get -- `detection_model_train` — trained model(s). -- `detection_model_train__metrics_on_subset` — overall metrics per model. -- **`tag_metrics`** — `(detection_model_id, tag_id, subset_id)` → precision/recall/f1. Compare the - baseline (trained before batch 2) vs the retrained model on `tag=night, subset=val`: the tag recall - rises once the tagged batch is in training. +- `detection_model_train__metrics_on_subset` — overall metrics per (model, subset). +- **`tag_metrics`** — `(detection_model_id, tag_id, subset_id)` → precision/recall/f1. Compare model + A vs model B at `tag=night, subset=val`: recall rises once the tagged batch is in training. +- **FiftyOne** dataset `$FIFTYONE_DATASET_NAME`: `ground_truth`, `predictions_model_a`, + `predictions_model_b`, plus `tag`/`subset` sample fields. Launch: + `fiftyone app launch "$FIFTYONE_DATASET_NAME" --port 5151 --address 0.0.0.0` (tunnel that port if + remote — local port must equal the remote port; filter by the `tag`/`subset` fields). + +### Pre-staged cache (fast loads) +If `DATAPIPE_TAGS_CACHE_DIR/gt.json` + `DATAPIPE_TAGS_CACHE_DIR/images/.jpg` exist, the load +step reads from them instead of fetching COCO. Default dir `/tmp/datapipe-tags-cache`. It caps the +image pool at its size (500 here) — expand it or point at a fresh dir to download more from COCO. ## Notes -- Classes are lowercase (`cat`/`dog`) to match COCO, so injected GT and predictions align. -- On a tiny validation set the "best epoch" pick can latch onto an early checkpoint; this example - uses 50 epochs and a non-trivial batch size to keep metrics meaningful. -- Tables `tag` / `image__tag` are external inputs (produced by the `load` step); `tag_metrics` is - produced by a pipeline `BatchTransform` (`steps.compute_tag_metrics`). +- Classes are lowercase (`cat`/`dog`) to match COCO so injected GT and predictions align. +- Trust `detection_training_status.status`, not exit codes; `count-metrics` may need a second run. +- FiftyOne integration is ported from `examples/e2e_template/image_detection`. diff --git a/examples/detection_tags/detection/app.py b/examples/detection_tags/detection/app.py index bfc00fb8..07ead686 100644 --- a/examples/detection_tags/detection/app.py +++ b/examples/detection_tags/detection/app.py @@ -24,13 +24,13 @@ BatchTransform( func=steps.load_batch, inputs=["load_request"], - outputs=["s3_images", "image__ground_truth", "tag", "image__tag"], + outputs=["s3_images", "image__ground_truth", "tag", "image__tag", "image__subset_hint"], transform_keys=["request_id"], labels=[("stage", "load")], ), BatchTransform( func=steps.split_df_train_val, - inputs=["image__ground_truth", "image__subset"], + inputs=["image__ground_truth", "image__subset", "image__subset_hint"], outputs=["image__subset"], transform_keys=["image_name"], kwargs=dict(primary_keys=["image_name"], val_perc=0.25, random_seed=42), @@ -126,6 +126,39 @@ transform_keys=["detection_model_id"], labels=[("stage", "train"), ("stage", "count-metrics"), ("stage", "tag-metrics")], ), + # --- FiftyOne (stage=fiftyone): browse GT + both models' predictions, filter by tag --- + BatchTransform( + func=steps.download_images, + inputs=["s3_images"], + outputs=["local_images"], + transform_keys=["image_name"], + kwargs=dict(image__image_path__name="image_url", image__local_image_path__name="local_path"), + labels=[("stage", "fiftyone")], + ), + BatchTransform( + func=steps.publish_gt_to_fiftyone, + inputs=["local_images", "image__ground_truth", "image__subset", "image__tag"], + outputs=["fiftyone_annotations"], + transform_keys=["image_name"], + kwargs=dict(primary_keys=["image_name"], image__image_path__name="local_path"), + labels=[("stage", "fiftyone")], + ), + BatchTransform( + func=steps.publish_predictions_to_fiftyone, + inputs=["local_images", "detection_prediction_train"], + outputs=["fiftyone_predictions_model_a"], + transform_keys=["image_name"], + kwargs=dict(slot="model_a", primary_keys=["image_name"], image__image_path__name="local_path"), + labels=[("stage", "fiftyone")], + ), + BatchTransform( + func=steps.publish_predictions_to_fiftyone, + inputs=["local_images", "detection_prediction_train"], + outputs=["fiftyone_predictions_model_b"], + transform_keys=["image_name"], + kwargs=dict(slot="model_b", primary_keys=["image_name"], image__image_path__name="local_path"), + labels=[("stage", "fiftyone")], + ), ] ) diff --git a/examples/detection_tags/detection/config.py b/examples/detection_tags/detection/config.py index b4a871e0..610295fe 100644 --- a/examples/detection_tags/detection/config.py +++ b/examples/detection_tags/detection/config.py @@ -21,6 +21,22 @@ DETECTION_CLASSES = ["cat", "dog"] COCO_CAT_IDS = {17: "cat", 18: "dog"} # COCO category_id -> class name +# --- tags ----------------------------------------------------------------------- +# Numeric tag ids must be DETERMINISTIC (datapipe re-runs), so map known tag names to +# fixed ids; unknown names fall back to a stable crc32-derived number. +import zlib # noqa: E402 + +TAG_IDS = {"night": 1} +TAG_NAMES = {v: k for k, v in TAG_IDS.items()} + + +def tag_id_for(name: str) -> int: + return TAG_IDS.get(name) or (zlib.crc32(name.encode()) % 100000) + + +def tag_name_for(tag_id) -> str: + return TAG_NAMES.get(tag_id, str(tag_id)) + # --- single storage root -------------------------------------------------------- # Input images live under /images; the pipeline working_dir under /datapipe # (siblings) so listing input images never re-ingests training artifacts. @@ -81,5 +97,12 @@ def datapipe_tmp_folder() -> str: "start docker compose, and run: set -a && source .env && set +a" ) -DB_SCHEMA = os.environ.get("DB_SCHEMA", "datapipe_tags") +DB_SCHEMA = os.environ.get("DB_SCHEMA", "public") DBCONN = DBConn(DB_URL, DB_SCHEMA) + +# --- FiftyOne ------------------------------------------------------------------- +# FiftyOne stores dataset metadata in MongoDB (FIFTYONE_DATABASE_URI, brought up by +# docker compose) and renders samples from LOCAL image files, so the fiftyone stage +# downloads S3 images to LOCAL_IMAGES_DIR first. +FIFTYONE_DATASET_NAME = os.environ.get("FIFTYONE_DATASET_NAME", "datapipe_detection_tags") +LOCAL_IMAGES_DIR = Path(os.environ.get("LOCAL_IMAGES_DIR", "/tmp/datapipe-tags-fiftyone-images")) diff --git a/examples/detection_tags/detection/data.py b/examples/detection_tags/detection/data.py index 4fce06e8..c7184a99 100644 --- a/examples/detection_tags/detection/data.py +++ b/examples/detection_tags/detection/data.py @@ -1,16 +1,34 @@ from __future__ import annotations +from cv_pipeliner.utils.fiftyone import FifyOneSession from datapipe.compute import Catalog, Table from datapipe.store.database import TableStoreDB +from datapipe_ml.utils.image_data_stores import FiftyOneImagesDataTableStore from sqlalchemy import Column, Float, Integer, JSON, String -from config import DBCONN +from config import DBCONN, FIFTYONE_DATASET_NAME + +# one FiftyOne session/dataset shared by the ground-truth + per-model prediction stores; +# each store writes its own field (ground_truth / predictions_model_a / predictions_model_b) +# into the SAME dataset, so a sample shows GT and both models side by side. +fo_session = FifyOneSession() def _t(name, schema): return Table(TableStoreDB(dbconn=DBCONN, name=name, data_sql_schema=schema, create_table=True)) +def _fo(field, additional_info_keys_in_sample=()): + return Table(FiftyOneImagesDataTableStore( + dataset=FIFTYONE_DATASET_NAME, + fo_session=fo_session, + fo_detections_label=field, + rm_only_fo_fields=True, + additional_info_keys_in_sample=list(additional_info_keys_in_sample), + primary_schema=[Column("image_name", String(255), primary_key=True)], + )) + + catalog = Catalog( { # a load request: add a row here, then run `datapipe step --labels=stage=load run` @@ -20,6 +38,9 @@ def _t(name, schema): Column("offset", Integer), Column("tag", String), Column("darken", Float), + # pin every image of this batch to a subset (train/val); NULL = let the + # random split decide. Pinning is how the demo freezes val. + Column("subset", String), ]), # images uploaded by the load step (image_name = object basename) "s3_images": _t("s3_images", [ @@ -37,14 +58,22 @@ def _t(name, schema): Column("image_name", String, primary_key=True), Column("subset_id", String, primary_key=True), ]), + # per-batch pinned subset emitted by the load step (--subset); the split step + # honors these and only randomizes images without a hint. Freezes val. + "image__subset_hint": _t("image__subset_hint", [ + Column("image_name", String, primary_key=True), + Column("subset_id", String), + ]), # --- tags ------------------------------------------------------------- + # tag dimension: numeric tag_id (surrogate key), tag_name (slug), tag_description (readable) "tag": _t("tag", [ - Column("tag_id", String, primary_key=True), - Column("name", String), + Column("tag_id", Integer, primary_key=True), + Column("tag_name", String), + Column("tag_description", String), ]), "image__tag": _t("image__tag", [ Column("image_name", String, primary_key=True), - Column("tag_id", String, primary_key=True), + Column("tag_id", Integer, primary_key=True), ]), # per-image metrics produced by the CountMetrics step (declared so the # tag-metrics transform can read it as an input) @@ -61,7 +90,7 @@ def _t(name, schema): # per-(model, tag, subset) metrics — the tag arc lives here "tag_metrics": _t("tag_metrics", [ Column("detection_model_id", String, primary_key=True), - Column("tag_id", String, primary_key=True), + Column("tag_id", Integer, primary_key=True), Column("subset_id", String, primary_key=True), Column("calc__images_support", Integer), Column("calc__support", Integer), @@ -69,5 +98,16 @@ def _t(name, schema): Column("calc__recall", Float), Column("calc__f1_score", Float), ]), + # --- FiftyOne (stage=fiftyone) --------------------------------------- + # images downloaded locally (FiftyOne renders from local files) + "local_images": _t("local_images", [ + Column("image_name", String(255), primary_key=True), + Column("local_path", String(1024)), + ]), + # GT boxes + per-sample subset/tag fields (filter tag=night in the FO App) + "fiftyone_annotations": _fo("ground_truth", additional_info_keys_in_sample=["subset", "tag"]), + # predictions of the baseline (model A) and the retrained model (model B) + "fiftyone_predictions_model_a": _fo("predictions_model_a"), + "fiftyone_predictions_model_b": _fo("predictions_model_b"), } ) diff --git a/examples/detection_tags/detection/steps.py b/examples/detection_tags/detection/steps.py index e9649472..f61ad680 100644 --- a/examples/detection_tags/detection/steps.py +++ b/examples/detection_tags/detection/steps.py @@ -13,14 +13,18 @@ import numpy as np import pandas as pd import requests +from datapipe_ml.core.image_data import convert_df_with_bbox_to_df_with_image_data from PIL import Image from config import ( COCO_CAT_IDS, DBCONN, + LOCAL_IMAGES_DIR, input_bucket, input_image_url, input_storage_options, + tag_id_for, + tag_name_for, ) COCO_IMG_BASE = "http://images.cocodataset.org/train2017/" @@ -74,11 +78,13 @@ def load_batch(df_request: pd.DataFrame): labels; image_name is the object basename (what a listing would produce).""" s3_cols = ["image_name", "image_url"] gt_cols = ["image_name", "bboxes", "labels"] - tag_cols = ["tag_id", "name"] + tag_cols = ["tag_id", "tag_name", "tag_description"] it_cols = ["image_name", "tag_id"] + hint_cols = ["image_name", "subset_id"] if df_request.empty: return (pd.DataFrame(columns=s3_cols), pd.DataFrame(columns=gt_cols), - pd.DataFrame(columns=tag_cols), pd.DataFrame(columns=it_cols)) + pd.DataFrame(columns=tag_cols), pd.DataFrame(columns=it_cols), + pd.DataFrame(columns=hint_cols)) # Prefer a pre-staged local cache (fast, no COCO): CACHE/images/ + CACHE/gt.json. # Download the cache once on a fast machine and copy it here — avoids per-image COCO fetches. @@ -117,13 +123,19 @@ def raw_bytes_and_gt(fn: str): if not use_cache: _fn_to_id = {id_to_img[i]["file_name"]: i for i in anns.keys()} - s3_rows, gt_rows, tag_rows, tag_defs = [], [], [], {} + s3_rows, gt_rows, tag_rows, tag_defs, hint_rows = [], [], [], {}, [] for _, req in df_request.iterrows(): n = int(req["n"]) offset = int(req.get("offset") or 0) - tag = req.get("tag") or None + # NB: when several requests load in one batch, a NULL in a numeric column arrives as NaN + # (not None) because pandas upcasts the column — so guard with pd.isna, else a base batch + # with darken=NULL would be "darkened" with gamma=NaN and produce garbage images. + tag = req.get("tag") + tag = None if pd.isna(tag) or str(tag) == "" else str(tag) + subset = req.get("subset") + subset = None if pd.isna(subset) or str(subset) == "" else str(subset) darken = req.get("darken") - darken = float(darken) if darken is not None and str(darken) != "" else None + darken = None if pd.isna(darken) or str(darken) == "" else float(darken) for fn in pool[offset: offset + n]: stem, ext = os.path.splitext(fn) raw, boxes, labels = raw_bytes_and_gt(fn) @@ -138,24 +150,44 @@ def raw_bytes_and_gt(fn: str): s3_rows.append({"image_name": name, "image_url": input_image_url(name)}) gt_rows.append({"image_name": name, "bboxes": boxes, "labels": labels}) if tag: - tag_rows.append({"image_name": name, "tag_id": tag}) - tag_defs[tag] = tag + tid = tag_id_for(tag) + tag_rows.append({"image_name": name, "tag_id": tid}) + # tag_defs[name] = (numeric id, readable description); note darkening if any + desc = tag if darken is None else f"{tag} (low-light, gamma {darken})" + tag_defs[tag] = (tid, desc) + if subset: + hint_rows.append({"image_name": name, "subset_id": subset}) return ( pd.DataFrame(s3_rows, columns=s3_cols), pd.DataFrame(gt_rows, columns=gt_cols), - pd.DataFrame([{"tag_id": t, "name": n} for t, n in tag_defs.items()], columns=tag_cols), + pd.DataFrame([{"tag_id": tid, "tag_name": nm, "tag_description": d} + for nm, (tid, d) in tag_defs.items()], columns=tag_cols), pd.DataFrame(tag_rows, columns=it_cols), + pd.DataFrame(hint_rows, columns=hint_cols), ) # --- train/val split ------------------------------------------------------------ -def split_df_train_val(df: pd.DataFrame, subset_df: pd.DataFrame, primary_keys: list[str], - val_perc: float = 0.25, random_seed: int = 42) -> pd.DataFrame: +def split_df_train_val(df: pd.DataFrame, subset_df: pd.DataFrame, hint_df: pd.DataFrame, + primary_keys: list[str], val_perc: float = 0.25, + random_seed: int = 42) -> pd.DataFrame: + """Assign a subset to every image that doesn't have one yet. An image whose subset the load + step pinned via ``image__subset_hint`` (batches loaded with ``--subset``) keeps that pinned + value; the rest are split train/val at random. Honoring the hint is what lets the demo FREEZE + val — load base-val + night-val up front (subset=val) and their assignment never changes when + train batches are added later, so a model's val metrics are stable across retrainings.""" df__merged = pd.merge(df, subset_df, on=primary_keys, how="outer") - df__missing = df__merged[df__merged["subset_id"].isna()].copy() - df__val = df__missing.sample(frac=val_perc, random_state=random_seed) - df__missing["subset_id"] = "train" + df__missing = df__merged[df__merged["subset_id"].isna()][primary_keys].copy() + if hint_df is not None and not hint_df.empty: + hint = hint_df[primary_keys + ["subset_id"]].drop_duplicates(subset=primary_keys) + df__missing = df__missing.merge(hint, on=primary_keys, how="left") + else: + df__missing["subset_id"] = np.nan + df__missing = df__missing.reset_index(drop=True) + unpinned = df__missing[df__missing["subset_id"].isna()] + df__val = unpinned.sample(frac=val_perc, random_state=random_seed) + df__missing.loc[unpinned.index, "subset_id"] = "train" df__missing.loc[df__val.index, "subset_id"] = "val" return pd.concat([subset_df, df__missing], ignore_index=True)[primary_keys + ["subset_id"]] @@ -180,3 +212,72 @@ def compute_tag_metrics(df_metrics_on_image: pd.DataFrame, df_image_tag: pd.Data g["calc__support"] = (tp + fn).astype(int) g["calc__images_support"] = g["images_support"].astype(int) return g[cols] + + +# --- FiftyOne (stage=fiftyone) -------------------------------------------------- +# Same pattern as examples/e2e_template/image_detection: download images locally, +# then publish bbox rows into a FiftyOne dataset via FiftyOneImagesDataTableStore. +def download_images(s3_images_df: pd.DataFrame, image__image_path__name: str, + image__local_image_path__name: str) -> pd.DataFrame: + LOCAL_IMAGES_DIR.mkdir(parents=True, exist_ok=True) + local_paths = [] + for _, row in s3_images_df.iterrows(): + s3_path = row[image__image_path__name] + local_path = LOCAL_IMAGES_DIR / os.path.basename(str(s3_path)) + if not local_path.exists(): + # image_url is the public HTTP URL of the MinIO object (anonymous download), so open it + # plainly — passing S3 storage options here would hit the http filesystem with s3 kwargs. + with fsspec.open(s3_path, "rb") as f_src: + local_path.write_bytes(f_src.read()) + local_paths.append(str(local_path.resolve())) + s3_images_df[image__local_image_path__name] = local_paths + return s3_images_df[["image_name", image__local_image_path__name]] + + +def publish_gt_to_fiftyone(images_df: pd.DataFrame, gt_df: pd.DataFrame, subset_df: pd.DataFrame, + tag_df: pd.DataFrame, primary_keys: list[str], + image__image_path__name: str) -> pd.DataFrame: + """Publish ground-truth boxes and attach per-sample subset/tag (so you can filter tag=night).""" + pk = primary_keys[0] + df = convert_df_with_bbox_to_df_with_image_data( + df__with_bbox=pd.merge(gt_df, images_df, on=pk), + primary_keys=primary_keys, + image__image_path__name=image__image_path__name, + ) + subset_by_image = (subset_df.drop_duplicates(subset=[pk]).set_index(pk)["subset_id"].to_dict() + if not subset_df.empty else {}) + tagid_by_image = (tag_df.drop_duplicates(subset=[pk]).set_index(pk)["tag_id"].to_dict() + if not tag_df.empty else {}) + for _, row in df.iterrows(): + tid = tagid_by_image.get(row[pk]) + # FiftyOne sample fields must be strings; write the tag NAME (not the numeric id) + row["image_data"].additional_info = { + "subset": subset_by_image.get(row[pk]) or "none", + "tag": tag_name_for(tid) if tid is not None else "none", + } + return df + + +def publish_predictions_to_fiftyone(images_df: pd.DataFrame, predictions_df: pd.DataFrame, + slot: str, primary_keys: list[str], + image__image_path__name: str) -> pd.DataFrame: + """Publish one model's predictions into the slot's FiftyOne field. slot='model_a' is the + earliest-trained model (baseline), 'model_b' the next (retrained). Model ids are + timestamp-prefixed, so sorting them yields a stable A/B assignment across all images.""" + pk = primary_keys[0] + empty = pd.DataFrame(columns=primary_keys + ["image_data"]) + if predictions_df.empty: + return empty + ids = sorted(predictions_df["detection_model_id"].dropna().unique().tolist()) + if not ids: + return empty + slot_of = {mid: ("model_a" if i == 0 else "model_b") for i, mid in enumerate(ids)} + target_ids = [mid for mid, s in slot_of.items() if s == slot] + pred = predictions_df[predictions_df["detection_model_id"].isin(target_ids)] + if pred.empty: + return empty + return convert_df_with_bbox_to_df_with_image_data( + df__with_bbox=pd.merge(pred, images_df, on=pk), + primary_keys=primary_keys, + image__image_path__name=image__image_path__name, + ) diff --git a/examples/detection_tags/docker-compose.yml b/examples/detection_tags/docker-compose.yml index d1788ffa..a99d2d0e 100644 --- a/examples/detection_tags/docker-compose.yml +++ b/examples/detection_tags/docker-compose.yml @@ -40,3 +40,14 @@ services: mc mb --ignore-existing local/datapipe-tags && mc anonymous set download local/datapipe-tags " + + # FiftyOne dataset metadata store (the fiftyone stage). Point FIFTYONE_DATABASE_URI at it. + mongo: + image: mongo:7 + ports: + - "27017:27017" + healthcheck: + test: ["CMD", "mongosh", "--quiet", "--eval", "db.adminCommand('ping')"] + interval: 10s + timeout: 5s + retries: 5 diff --git a/examples/detection_tags/pyproject.toml b/examples/detection_tags/pyproject.toml index edb38bc3..a4b989f2 100644 --- a/examples/detection_tags/pyproject.toml +++ b/examples/detection_tags/pyproject.toml @@ -4,12 +4,13 @@ version = "0" requires-python = ">=3.10,<3.13" dependencies = [ "datapipe-app", - "datapipe-ml[torch]", + "datapipe-ml[torch,fiftyone]", # fiftyone extra pulls FiftyOneImagesDataTableStore + cv_pipeliner.fiftyone "torch==2.6.0", "torchvision==0.21.0", "stringzilla==4.4.0", # needed by albumentations "polars-lts-cpu==1.33.1", # ultralytics/datapipe on pre-AVX2 CPUs (avoids SIGILL) "pi-heif", # ultralytics image verification imports it + "fiftyone==1.17.0", # FiftyOne App + dataset (metadata in MongoDB) ] [[tool.uv.index]] diff --git a/examples/detection_tags/scripts/add_request.py b/examples/detection_tags/scripts/add_request.py index 56539876..790c3b4a 100644 --- a/examples/detection_tags/scripts/add_request.py +++ b/examples/detection_tags/scripts/add_request.py @@ -1,10 +1,15 @@ """Add a load request, then run the load step: datapipe step --labels=stage=load run - # batch 1 — base cat/dog - python ../scripts/add_request.py --id base --n 120 - # batch 2 — tagged low-light scenario - python ../scripts/add_request.py --id night --n 40 --offset 120 --tag night --darken 0.1 +Fixed-val demo (freeze val up front, add the tagged train batch later): + # loaded BEFORE training model A — val is frozen from the start + python ../scripts/add_request.py --id base-train --n 400 --offset 0 --subset train + python ../scripts/add_request.py --id base-val --n 100 --offset 400 --subset val + python ../scripts/add_request.py --id night-val --n 50 --offset 500 --subset val --tag night --darken 0.25 + # loaded BEFORE retraining model B (the tagged TRAIN batch — the fix) + python ../scripts/add_request.py --id night-train --n 50 --offset 550 --subset train --tag night --darken 0.25 + +--subset train|val pins every image of the batch to that subset so the random split never moves it. Run from examples/detection_tags/detection (so `import config` resolves).""" from __future__ import annotations @@ -27,6 +32,8 @@ def main() -> int: ap.add_argument("--offset", type=int, default=0, help="skip the first OFFSET picked COCO images") ap.add_argument("--tag", default=None) ap.add_argument("--darken", type=float, default=None, help="gamma < 1 darkens (e.g. 0.1)") + ap.add_argument("--subset", default=None, choices=["train", "val"], + help="pin every image of this batch to a subset (freezes val); omit to random-split") a = ap.parse_args() ds = DataStore(config.DBCONN, create_meta_table=True) @@ -36,11 +43,14 @@ def main() -> int: Column("request_id", String, primary_key=True), Column("n", Integer), Column("offset", Integer), Column("tag", String), Column("darken", Float), + Column("subset", String), ], create_table=True)) dt.store_chunk(pd.DataFrame([{ - "request_id": a.id, "n": a.n, "offset": a.offset, "tag": a.tag, "darken": a.darken, + "request_id": a.id, "n": a.n, "offset": a.offset, "tag": a.tag, + "darken": a.darken, "subset": a.subset, }])) - print(f"added request {a.id}: n={a.n} offset={a.offset} tag={a.tag} darken={a.darken}") + print(f"added request {a.id}: n={a.n} offset={a.offset} tag={a.tag} " + f"darken={a.darken} subset={a.subset}") print("now run: datapipe step --labels=stage=load run") return 0 From 91927e6a4288a97a2a4232e70f669973e4e3936a Mon Sep 17 00:00:00 2001 From: Shakirov Renat Date: Tue, 7 Jul 2026 10:21:11 +0300 Subject: [PATCH 32/32] detection_tags: split demo data synthesis out of the pipeline Move all COCO/demo-only code (constants, download+cache, gamma darkening) into a dedicated detection/coco_demo.py (CocoDemoSource + darken). steps.load_batch is now source-agnostic plumbing: it pulls (bytes, boxes, labels) from the source and lays them into the pipeline tables. For real data you replace one file (coco_demo.py); steps.py/app.py are untouched. Co-Authored-By: Claude Opus 4.8 --- .../detection_tags/detection/coco_demo.py | 105 +++++++++++++++++ examples/detection_tags/detection/steps.py | 108 ++---------------- 2 files changed, 115 insertions(+), 98 deletions(-) create mode 100644 examples/detection_tags/detection/coco_demo.py diff --git a/examples/detection_tags/detection/coco_demo.py b/examples/detection_tags/detection/coco_demo.py new file mode 100644 index 00000000..fd24eb2b --- /dev/null +++ b/examples/detection_tags/detection/coco_demo.py @@ -0,0 +1,105 @@ +"""DEMO DATA SYNTHESIS — COCO cat/dog (replace this module for real data). + +Everything here exists only to fabricate the demo dataset: download the COCO cat/dog subset, inject +ground truth, and darken images to fake a low-light "night" scenario. None of it is part of the tag +pipeline itself — for REAL data, drop this module and feed the `load` step from your own source of +(image bytes, boxes, labels). The pipeline (steps.py / app.py) doesn't care where images come from. +""" +from __future__ import annotations + +import io +import json +import os +import random +import time +import zipfile +from pathlib import Path +from typing import Optional + +import numpy as np +import requests +from PIL import Image + +from config import COCO_CAT_IDS + +COCO_IMG_BASE = "http://images.cocodataset.org/train2017/" +COCO_ANN_URL = "http://images.cocodataset.org/annotations/annotations_trainval2017.zip" +CACHE = Path(os.environ.get("DATAPIPE_TAGS_CACHE_DIR", "/tmp/datapipe-tags-cache")) +RNG_SEED = 1234 + + +def _get(url: str, attempts: int = 4) -> requests.Response: + last: Optional[Exception] = None + for a in range(attempts): + try: + r = requests.get(url, timeout=60, stream=True) + r.raise_for_status() + return r + except requests.RequestException as e: + last = e + time.sleep(min(2 ** a, 20)) + assert last is not None + raise last + + +def _coco_annotations_by_image() -> tuple[dict, dict]: + """Return (id_to_image_meta, {image_id: [cat/dog annotations]}), cached on disk.""" + CACHE.mkdir(parents=True, exist_ok=True) + zip_path = CACHE / "annotations_trainval2017.zip" + if not zip_path.exists() or zip_path.stat().st_size < 1_000_000: + with _get(COCO_ANN_URL) as r, open(zip_path, "wb") as f: + for chunk in r.iter_content(1 << 20): + f.write(chunk) + with zipfile.ZipFile(zip_path) as zf, zf.open("annotations/instances_train2017.json") as h: + inst = json.load(h) + id_to_img = {im["id"]: im for im in inst["images"]} + anns: dict[int, list] = {} + for a in inst["annotations"]: + if a.get("category_id") in COCO_CAT_IDS and not a.get("iscrowd", 0) and a.get("bbox"): + anns.setdefault(a["image_id"], []).append(a) + return id_to_img, anns + + +def darken(raw: bytes, gamma: float) -> bytes: + """Gamma-darken JPEG bytes to fake a low-light scene (gamma < 1 darkens). Demo-only.""" + a = np.asarray(Image.open(io.BytesIO(raw)).convert("RGB")).astype(np.float32) / 255.0 + out = Image.fromarray((np.power(a, 1.0 / gamma) * 255).clip(0, 255).astype(np.uint8)) + buf = io.BytesIO() + out.save(buf, format="JPEG", quality=92) + return buf.getvalue() + + +class CocoDemoSource: + """Deterministic shuffled pool of COCO cat/dog filenames + per-file (bytes, boxes, labels). + + Prefers a pre-staged cache (`$DATAPIPE_TAGS_CACHE_DIR/gt.json` + `images/`) so loads are fast + and offline; falls back to fetching from COCO. Order is a fixed shuffle (RNG_SEED) so `offset`/`n` + slices are reproducible across runs. + """ + + def __init__(self) -> None: + cache_gt_path = CACHE / "gt.json" + self._use_cache = cache_gt_path.exists() and (CACHE / "images").is_dir() + if self._use_cache: + self._cache_gt = json.loads(cache_gt_path.read_text()) + pool = sorted(self._cache_gt.keys()) + else: + self._id_to_img, self._anns = _coco_annotations_by_image() + pool = [self._id_to_img[i]["file_name"] for i in sorted(self._anns.keys())] + self._fn_to_id = {self._id_to_img[i]["file_name"]: i for i in self._anns.keys()} + random.seed(RNG_SEED) + random.shuffle(pool) + self.pool = pool + + def fetch(self, fn: str) -> tuple[bytes, list, list]: + """Return (image_bytes, bboxes[[x1,y1,x2,y2]], labels[lowercase]) for a pool filename.""" + if self._use_cache: + data = (CACHE / "images" / fn).read_bytes() + g = self._cache_gt[fn] + return data, g["bboxes"], g["labels"] + raw = _get(COCO_IMG_BASE + fn).content + anns = self._anns[self._fn_to_id[fn]] + boxes = [[int(a["bbox"][0]), int(a["bbox"][1]), + int(a["bbox"][0] + a["bbox"][2]), int(a["bbox"][1] + a["bbox"][3])] for a in anns] + labels = [COCO_CAT_IDS[a["category_id"]] for a in anns] + return raw, boxes, labels diff --git a/examples/detection_tags/detection/steps.py b/examples/detection_tags/detection/steps.py index f61ad680..8b927fdf 100644 --- a/examples/detection_tags/detection/steps.py +++ b/examples/detection_tags/detection/steps.py @@ -1,24 +1,14 @@ from __future__ import annotations -import io -import json import os -import random -import time -import zipfile -from pathlib import Path -from typing import Optional import fsspec import numpy as np import pandas as pd -import requests from datapipe_ml.core.image_data import convert_df_with_bbox_to_df_with_image_data -from PIL import Image +from coco_demo import CocoDemoSource, darken as demo_darken from config import ( - COCO_CAT_IDS, - DBCONN, LOCAL_IMAGES_DIR, input_bucket, input_image_url, @@ -27,55 +17,12 @@ tag_name_for, ) -COCO_IMG_BASE = "http://images.cocodataset.org/train2017/" -COCO_ANN_URL = "http://images.cocodataset.org/annotations/annotations_trainval2017.zip" -CACHE = Path(os.environ.get("DATAPIPE_TAGS_CACHE_DIR", "/tmp/datapipe-tags-cache")) -RNG_SEED = 1234 - - -# --- COCO helpers --------------------------------------------------------------- -def _get(url: str, attempts: int = 4) -> requests.Response: - last: Optional[Exception] = None - for a in range(attempts): - try: - r = requests.get(url, timeout=60, stream=True) - r.raise_for_status() - return r - except requests.RequestException as e: - last = e - time.sleep(min(2 ** a, 20)) - assert last is not None - raise last - - -def _coco_annotations_by_image() -> tuple[dict, dict]: - """Return (id_to_image_meta, {image_id: [cat/dog annotations]}), cached on disk.""" - CACHE.mkdir(parents=True, exist_ok=True) - zip_path = CACHE / "annotations_trainval2017.zip" - if not zip_path.exists() or zip_path.stat().st_size < 1_000_000: - with _get(COCO_ANN_URL) as r, open(zip_path, "wb") as f: - for chunk in r.iter_content(1 << 20): - f.write(chunk) - with zipfile.ZipFile(zip_path) as zf, zf.open("annotations/instances_train2017.json") as h: - inst = json.load(h) - id_to_img = {im["id"]: im for im in inst["images"]} - anns: dict[int, list] = {} - for a in inst["annotations"]: - if a.get("category_id") in COCO_CAT_IDS and not a.get("iscrowd", 0) and a.get("bbox"): - anns.setdefault(a["image_id"], []).append(a) - return id_to_img, anns - - -def _darken(im: Image.Image, gamma: float) -> Image.Image: - a = np.asarray(im.convert("RGB")).astype(np.float32) / 255.0 - return Image.fromarray((np.power(a, 1.0 / gamma) * 255).clip(0, 255).astype(np.uint8)) - - -# --- load step ------------------------------------------------------------------ +# --- load step (demo: pulls images/GT from coco_demo; swap that module for real data) ----------- def load_batch(df_request: pd.DataFrame): - """For each load_request row: download COCO cat/dog images, upload to object storage, and - return (s3_images, image__ground_truth, tag, image__tag). Ground truth uses lowercase COCO - labels; image_name is the object basename (what a listing would produce).""" + """For each load_request row: pull images + ground truth from the demo source (coco_demo), + optionally darken (a tagged scenario), upload to object storage, and return (s3_images, + image__ground_truth, tag, image__tag, image__subset_hint). image_name is the object basename. + For REAL data, replace coco_demo — this step itself is source-agnostic plumbing.""" s3_cols = ["image_name", "image_url"] gt_cols = ["image_name", "bboxes", "labels"] tag_cols = ["tag_id", "tag_name", "tag_description"] @@ -86,43 +33,10 @@ def load_batch(df_request: pd.DataFrame): pd.DataFrame(columns=tag_cols), pd.DataFrame(columns=it_cols), pd.DataFrame(columns=hint_cols)) - # Prefer a pre-staged local cache (fast, no COCO): CACHE/images/ + CACHE/gt.json. - # Download the cache once on a fast machine and copy it here — avoids per-image COCO fetches. - cache_gt_path = CACHE / "gt.json" - use_cache = cache_gt_path.exists() and (CACHE / "images").is_dir() - if use_cache: - cache_gt = json.loads(cache_gt_path.read_text()) - pool = sorted(cache_gt.keys()) # deterministic order - random.seed(RNG_SEED) - random.shuffle(pool) - anns = None - else: - id_to_img, anns = _coco_annotations_by_image() - by_id = sorted(anns.keys()) - random.seed(RNG_SEED) - random.shuffle(by_id) - pool = [id_to_img[i]["file_name"] for i in by_id] # file names, same order semantics - + source = CocoDemoSource() # demo data (COCO cat/dog); for real data swap coco_demo.py fs = fsspec.filesystem("s3", **input_storage_options()) bucket = input_bucket() - def raw_bytes_and_gt(fn: str): - if use_cache: - data = (CACHE / "images" / fn).read_bytes() - g = cache_gt[fn] - return data, g["bboxes"], g["labels"] - # COCO fallback - # map filename back to id via anns lookup (filename -> id): rebuild once - raw = _get(COCO_IMG_BASE + fn).content - iid = _fn_to_id[fn] - boxes = [[int(a["bbox"][0]), int(a["bbox"][1]), - int(a["bbox"][0] + a["bbox"][2]), int(a["bbox"][1] + a["bbox"][3])] for a in anns[iid]] - labels = [COCO_CAT_IDS[a["category_id"]] for a in anns[iid]] - return raw, boxes, labels - - if not use_cache: - _fn_to_id = {id_to_img[i]["file_name"]: i for i in anns.keys()} - s3_rows, gt_rows, tag_rows, tag_defs, hint_rows = [], [], [], {}, [] for _, req in df_request.iterrows(): n = int(req["n"]) @@ -136,14 +50,12 @@ def raw_bytes_and_gt(fn: str): subset = None if pd.isna(subset) or str(subset) == "" else str(subset) darken = req.get("darken") darken = None if pd.isna(darken) or str(darken) == "" else float(darken) - for fn in pool[offset: offset + n]: + for fn in source.pool[offset: offset + n]: stem, ext = os.path.splitext(fn) - raw, boxes, labels = raw_bytes_and_gt(fn) + raw, boxes, labels = source.fetch(fn) if darken is not None: name = f"{stem}__{tag or 'dark'}{ext}" - buf = io.BytesIO() - _darken(Image.open(io.BytesIO(raw)), darken).save(buf, format="JPEG", quality=92) - data = buf.getvalue() + data = demo_darken(raw, darken) else: name, data = fn, raw fs.pipe(f"{bucket}/images/{name}", data)