Detection tags frozenval UI

.claude/skills/datapipe-examples/SKILL.md

@@ -18,7 +18,7 @@ Each `examples/*` pipeline has a dedicated setup skill — pick by what it does:
 | `e2e_template/image_detection` | YOLO detection + Label Studio human-in-the-loop → train → FiftyOne | **setup-e2e-template** |
 | `e2e_template/image_keypoints` | YOLO-pose keypoints + Label Studio → train → FiftyOne | **setup-e2e-template** |
 | `sam_cvat` | SAM3 text-prompt boxes+masks → CVAT pre-annotations | **setup-sam-cvat** |
-| `detection_tags` | YOLO detection + **tags** (per-scenario metrics), no Label Studio / FiftyOne, GT injected | **setup-detection-tags** |
+| `detection_tags` | YOLO detection + **tags** (per-scenario metrics), injected GT, FiftyOne A/B view | **setup-detection-tags** |
 
 ## Ask first — don't assume (only the unresolved)
 Clarify what's not obvious before acting — don't spin up services or pull data you don't need. Ask only the unresolved:

.claude/skills/setup-e2e-template/SKILL.md

@@ -65,8 +65,9 @@ conventions, or the freeze join silently yields nothing / classes don't match:**
 ## Per-scenario tag metrics → see the dedicated example
 Want to tag a scenario (e.g. dark-room pallets), add it to training, and measure the model on that
 scenario separately (baseline vs retrained)? That lives as its own self-contained example —
-`examples/detection_tags` (`tag`/`image__tag`/`tag_metrics` built into the pipeline, **no Label Studio
-or FiftyOne**, ground truth injected). Use the **setup-detection-tags** skill.
+`examples/detection_tags` (`tag`/`image__tag`, `CountMetrics_Subset_PipelineModel` tag arc,
+**no Label Studio**, ground truth injected, **FiftyOne** baseline vs retrained). Use the
+**setup-detection-tags** skill.
 
 ## Troubleshooting (may already be fixed — verify against current files)
 - **No model after `train`, exit 0** → datapipe swallows step errors; check `detection_training_status`, not the exit code.

.gitignore

@@ -23,6 +23,9 @@ dist/
 # Lock files (deps resolved inside the image)
 poetry.lock
 uv.lock
+# ...except detection_tags: its whole point is reproducible cross-machine runs — the lock pins the
+# training stack (ultralytics etc.); without it every machine resolves different versions.
+!examples/detection_tags/uv.lock
 
 # ML models & checkpoints
 *.pt

examples/detection_tags/.env.example

@@ -5,6 +5,14 @@ export S3_ENDPOINT_URL=http://localhost:9000
 export FSSPEC_S3_ENDPOINT_URL="${S3_ENDPOINT_URL}"
 export S3_PUBLIC_URL=http://localhost:9000
 export DATAPIPE_TAGS_DIR=s3://datapipe-tags
+# Local cache for training artifacts and images downloaded for FiftyOne (mounted into compose fiftyone service)
 export DATAPIPE_TAGS_TMP_DIR=/tmp/datapipe-tags
 export DB_URL=postgresql://postgres:password@localhost:5432/postgres
-export DB_SCHEMA=datapipe_tags
+# default schema; set a dedicated one (e.g. datapipe_tags) only if you SHARE this Postgres with
+# other pipelines — then `db create-all` needs the schema created first (see README).
+export DB_SCHEMA=public
+# FiftyOne (stage=fiftyone): metadata in MongoDB; App UI in docker compose (:5151)
+export FIFTYONE_DATABASE_URI=mongodb://localhost:27017
+export FIFTYONE_DATASET_NAME=datapipe_detection_tags
+# optional override; default is $DATAPIPE_TAGS_TMP_DIR/local_images (see detection/config.py)
+# export LOCAL_IMAGES_DIR=/tmp/datapipe-tags/local_images

examples/detection_tags/.gitignore

@@ -0,0 +1 @@
+.docker-data/

examples/detection_tags/.python-version

@@ -0,0 +1 @@
+3.10

examples/detection_tags/README.md

@@ -1,64 +1,98 @@
 # detection_tags
 
-A minimal, self-contained datapipe detection example built around **tags** — with **no Label
-Studio and no FiftyOne**. Ground truth is injected directly (COCO labels, lowercase `cat`/`dog`),
-so the whole thing runs unattended from scratch.
+A self-contained datapipe detection example built around **tags** (per-scenario metrics), with a
+**FiftyOne** view and **no Label Studio**. Ground truth is injected directly (COCO labels, lowercase
+`cat`/`dog`), so it runs unattended from scratch.
 
-The point: train a baseline, then add a **tagged scenario batch**, retrain, and watch the metric on
-that tag rise — visible in `pipeline_model__metrics_by_tag_on_subset` (a real pipeline step).
+The point: train a baseline (model A), then add a **tagged scenario TRAIN batch** and retrain
+(model B), and watch recall on that tag rise — visible in a dedicated `tag_metrics` table. The demo
+is split into **two parts around a checkpoint** so you can prep part 1 and rehearse part 2.
 
 ## Deploy from scratch
 ```bash
 cp .env.example .env && set -a && source .env && set +a
-docker compose up -d            # postgres + minio only
-uv sync                         # cu124 torch, polars-lts-cpu, pi-heif baked in
+HOST_UID=$(id -u) HOST_GID=$(id -g) docker compose up -d   # postgres + minio + mongo + fiftyone (:5151)
+uv sync                         # modern hosts: as-is. Do NOT edit deps/re-lock (drifts lib versions
+                                # across machines and changes training results — reproducibility!)
+# legacy pre-AVX2 host (e.g. epoch8 gpu5) only:
+#   uv sync --extra old-cpu && uv pip uninstall polars polars-lts-cpu && uv pip install polars-lts-cpu==1.33.1
 cd detection
-psql "$DB_URL" -c "CREATE SCHEMA IF NOT EXISTS $DB_SCHEMA"   # create-all makes tables, not the schema
-uv run datapipe db create-all   # required on first run (and after schema-breaking changes)
+# DB_SCHEMA defaults to `public` (exists already). For a dedicated schema (sharing the Postgres with
+# other pipelines) create it first: psql "$DB_URL" -c "CREATE SCHEMA IF NOT EXISTS $DB_SCHEMA"
+datapipe db create-all
 ```
 
-## Ops smoke data (optional)
+## Frozen val (why the load order matters)
+A `val` metric is an aggregate over whatever images are in val; if you add the tagged batch to val at
+retrain time, model A's numbers move and the A-vs-B comparison is invalid. So **freeze val up front**:
+load `base-val` + `night-val` (pinned `--subset val`) before training A, and add `night-train`
+(`--subset train`) only for part 2. `add_request.py --subset` pins a batch (`image__subset_hint`),
+the split step honors it. The pre-staged cache holds **500 images**, so keep the total ≤ 500.
+
+## Identical metrics on any machine — demo seed (no training)
+
+Live-training numbers are bit-stable on one GPU but DIFFER across GPU models (proven — different CUDA
+kernels; no software setting fixes it). For the same numbers everywhere, restore the reference state
+instead of training (~2 min): models A/B, all metrics and run history land exactly as validated.
+
 ```bash
-uv run python ../scripts/seed_ops_smoke.py
+# stack up + .env sourced + venv built + image cache present (scripts/build_cache.py)
+bash scripts/restore_demo_seed.sh        # wipes the public schema, restores demo_seed/db.sql.gz,
+                                         # re-uploads images byte-identically from the cache
 ```
 
-## Two-step data load — via datapipe steps (no annotation)
-Loading is a real pipeline step (`stage=load`): add a request row, then run the step. It downloads
-COCO cat/dog, uploads to MinIO, and produces `s3_images` + ground truth (+ a tag) — no human labelling.
+Then open the front — `cat_dog` shows the reference metrics (A val recall 0.324 / B 0.415 etc.).
+Live (re)training stays available; treat its numbers as per-GPU.
+
+## Part 1 — baseline to checkpoint
 ```bash
 # from examples/detection_tags/detection
-python ../scripts/add_request.py --id base --n 450
-datapipe step --labels=stage=load run          # batch 1: base cat/dog
-datapipe step --labels=stage=train run         # -> model A (baseline, no tag in training)
-
-python ../scripts/add_request.py --id night --n 50 --offset 450 --tag night --darken 0.25
-datapipe step --labels=stage=load run          # batch 2: tagged low-light scenario
-datapipe step --labels=stage=train run         # -> model B (tag in training)
-datapipe step --labels=stage=count-metrics run # (re)compute overall + tag metrics
+python ../scripts/add_request.py --id base-train --n 200 --offset 0   --subset train
+python ../scripts/add_request.py --id base-val   --n 75  --offset 200 --subset val
+python ../scripts/add_request.py --id night-val  --n 75  --offset 275 --subset val --tag night --darken 0.40
+datapipe step --labels=stage=load run
+datapipe step --labels=stage=train run            # model A
+datapipe step --labels=stage=count-metrics run    # re-run once if it prints "Batches to process 0"
+# (demo-only) snapshot the post-A state to rehearse part 2 later, before the fiftyone stage:
+docker exec <pg> pg_dump -U postgres -n "$DB_SCHEMA" postgres > /tmp/checkpoint.sql
+datapipe step --labels=stage=fiftyone run         # GT + model-A predictions into FiftyOne
 ```
-Or just `datapipe run` after adding request rows to run every stage.
 
-### Fast data on a slow/blocked network (pre-staged cache)
-The `load` step downloads images from COCO per request, which is slow on a high-latency or
-throttled link. To avoid it, pre-stage a cache once from a fast machine:
-`DATAPIPE_TAGS_CACHE_DIR/images/<file>.jpg` + `DATAPIPE_TAGS_CACHE_DIR/gt.json`
-(`{ "<file>.jpg": {"bboxes": [[x1,y1,x2,y2]], "labels": ["cat"|"dog"]}, ... }`). If that cache is
-present, the load step reads images + ground truth from it (no COCO fetches at all). Build it by
-downloading the COCO cat/dog subset elsewhere and copying the folder to `DATAPIPE_TAGS_CACHE_DIR`
-(default `/tmp/datapipe-tags-cache`).
+## Part 2 — retrain and watch the tag metric rise
+```bash
+python ../scripts/add_request.py --id night-train-a --n 50 --offset 350 --subset train --tag night --darken 0.30
+python ../scripts/add_request.py --id night-train-b --n 50 --offset 400 --subset train --tag night --darken 0.40
+python ../scripts/add_request.py --id night-train-c --n 50 --offset 450 --subset train --tag night --darken 0.55
+datapipe step --labels=stage=load run
+datapipe step --labels=stage=train run            # model B (night now in training)
+datapipe step --labels=stage=count-metrics run    # re-run once if "0 batches"
+datapipe step --labels=stage=fiftyone run         # adds predictions_model_b
+```
+Rehearse (demo-only): restore the snapshot + wipe the FiftyOne db, then re-run part 2 — no retraining
+of model A:
+```bash
+docker exec <pg> psql -U postgres -c "DROP SCHEMA IF EXISTS $DB_SCHEMA CASCADE; CREATE SCHEMA $DB_SCHEMA"
+docker exec -i <pg> psql -U postgres < /tmp/checkpoint.sql
+docker exec <mongo> mongosh --quiet --eval "db.getSiblingDB('fiftyone').dropDatabase()"
+```
 
 ## What you get
-- `detection_model_train` — trained model(s).
-- `pipeline_model__metrics_on_subset` — overall weighted/macro metrics per model.
-- `pipeline_model__metrics_by_cls_on_subset` — per-class metrics per model/subset.
-- **`pipeline_model__metrics_by_tag_on_subset`** — `(detection_model_id, tag_id, subset_id)` metrics.
-  The same tag can appear for multiple models (compare baseline vs retrained on `batch_night`).
-- `pipeline_model__metrics_by_tag_by_cls_on_subset` — per-class metrics within each tag slice.
+- `pipeline_model__metrics_on_subset` — overall metrics per (model, subset).
+- **`pipeline_model__metrics_by_tag_on_subset`** — per `(detection_model_id, tag_id, subset_id)`.
+  Compare model A vs B at `tag_id=night, subset_id=val`: weighted recall/F1 rise after retraining.
+- Class tables: `pipeline_model__metrics_by_cls_on_subset`, `pipeline_model__metrics_by_tag_by_cls_on_subset`.
+- **FiftyOne** dataset `$FIFTYONE_DATASET_NAME`: fields `annotations`, `predictions_model_a`,
+  `predictions_model_b`; sample fields `tag_id` / `subset_id`. After `stage=fiftyone`, open the App
+  from docker compose: **http://localhost:5151** (remote → SSH tunnel `-L 5151:localhost:5151`).
+  Compose mounts `DATAPIPE_TAGS_TMP_DIR` (default `/tmp/datapipe-tags`); local images land in
+  `$DATAPIPE_TAGS_TMP_DIR/local_images`.
+
+### Pre-staged cache (fast loads)
+If `DATAPIPE_TAGS_CACHE_DIR/gt.json` + `DATAPIPE_TAGS_CACHE_DIR/images/<file>.jpg` exist, the load
+step reads from them instead of fetching COCO. Default dir `/tmp/datapipe-tags-cache`. It caps the
+image pool at its size (500 here) — expand it or point at a fresh dir to download more from COCO.
 
 ## Notes
-- Classes are lowercase (`cat`/`dog`) to match COCO, so injected GT and predictions align.
-- On a tiny validation set the "best epoch" pick can latch onto an early checkpoint; this example
-  uses 50 epochs and a non-trivial batch size to keep metrics meaningful.
-- Tables `tag` / `image__tag` are external inputs (produced by the `load` step). Metric tables are
-  created by `CountMetrics_Subset_PipelineModel` — run `datapipe db create-all` before the first
-  pipeline run.
+- Classes are lowercase (`cat`/`dog`) to match COCO so injected GT and predictions align.
+- Trust `detection_training_status.status`, not exit codes; `count-metrics` may need a second run.
+- FiftyOne integration is ported from `examples/e2e_template/image_detection`.

examples/detection_tags/detection/app.py

@@ -1,8 +1,16 @@
 from __future__ import annotations
 
+import os
+
+# Deterministic training: cuBLAS needs CUBLAS_WORKSPACE_CONFIG set BEFORE torch/CUDA init, so set it
+# here at the entrypoint — the train subprocess inherits it. Pairs with seed + deterministic=True in
+# YoloV8_TrainingConfig to keep model-A / model-B metrics reproducible run-to-run on the same GPU.
+os.environ.setdefault("CUBLAS_WORKSPACE_CONFIG", ":4096:8")
+
 from datapipe.compute import Pipeline
 from datapipe.datatable import DataStore
 from datapipe.step.batch_transform import BatchTransform
+from datapipe.types import Required
 from datapipe_app import (
     DatapipeApp,
     DatapipeOpsSpec,
@@ -36,21 +44,19 @@
 # darken) and run `datapipe step --labels=stage=load run`. It downloads COCO cat/dog images,
 # uploads them to object storage, and produces s3_images + ground truth (+ tag) directly — there
 # is no Label Studio annotation stage.
-#
-# First-time setup: `datapipe db create-all` (see README).
 
 pipeline = Pipeline(
     [
         BatchTransform(
             func=steps.load_batch,
             inputs=["load_request"],
-            outputs=["s3_images", "image__ground_truth", "tag", "image__tag"],
+            outputs=["s3_images", "image__ground_truth", "tag", "image__tag", "image__subset_hint"],
             transform_keys=["request_id"],
             labels=[("stage", "load")],
         ),
         BatchTransform(
             func=steps.split_df_train_val,
-            inputs=["image__ground_truth", "image__subset"],
+            inputs=["image__ground_truth", "image__subset", "image__subset_hint"],
             outputs=["image__subset"],
             transform_keys=["image_name"],
             kwargs=dict(primary_keys=["image_name"], val_perc=0.25, random_seed=42),
@@ -68,7 +74,7 @@
             primary_keys=["image_name"],
             bbox_id__name=None,
             image__image_path__name="image_url",
-            labels=[("stage", "train"), ("stage", "train-prepare")],
+            labels=[("stage", "train"), ("stage", "train-prepare")]
         ),
         Train_YoloV8_DetectionModel(  # type: ignore[list-item]
             input__detection_frozen_dataset="detection_frozen_dataset",
@@ -87,7 +93,7 @@
             working_dir=str(DATAPIPE_DIR),
             tmp_folder=datapipe_tmp_folder(),
             yolov8_train_configs=[
-                YoloV8_TrainingConfig(model="yolov8n.pt", imgsz=320, batch=10, epochs=10, exist_ok=True)
+                YoloV8_TrainingConfig(model="yolov8n.pt", imgsz=320, batch=10, epochs=10, exist_ok=True, seed=42, workers=0)
             ],
             sync_config=TrainingSyncConfig(enabled=True, interval_s=30, retries=3, retry_sleep_s=30),
             resume_config=TrainingResumeConfig(
@@ -149,6 +155,65 @@
             minimum_iou=0.5,
             labels=[("stage", "train"), ("stage", "count-metrics"), ("stage", "tag-metrics")],
         ),
+        # --- FiftyOne (stage=fiftyone): GT + baseline/retrained predictions, filter by tag_id ---
+        BatchTransform(
+            func=steps.download_images,
+            inputs=["s3_images"],
+            outputs=["local_images"],
+            transform_keys=["image_name"],
+            labels=[("stage", "fiftyone")],
+            kwargs=dict(
+                image__image_path__name="image_url",
+                image__local_image_path__name="local_path",
+            ),
+        ),
+        BatchTransform(
+            func=steps.publish_to_fiftyone,
+            inputs=["local_images"],
+            outputs=["fiftyone_images"],
+            labels=[("stage", "fiftyone")],
+            kwargs=dict(
+                primary_keys=["image_name"],
+                image__image_path__name="local_path",
+            ),
+        ),
+        BatchTransform(
+            func=steps.publish_to_fiftyone_ground_truth,
+            inputs=[
+                "local_images",
+                Required("image__ground_truth"),
+                Required("image__subset"),
+                Required("image__tag"),
+            ],
+            outputs=["fiftyone_annotations"],
+            labels=[("stage", "fiftyone")],
+            kwargs=dict(
+                primary_keys=["image_name"],
+                image__image_path__name="local_path",
+            ),
+        ),
+        BatchTransform(
+            func=steps.publish_to_fiftyone_predictions_baseline,
+            inputs=[Required("local_images"), Required("detection_prediction_train")],
+            outputs=["fiftyone_predictions_model_a"],
+            transform_keys=["image_name", "detection_model_id"],
+            labels=[("stage", "fiftyone")],
+            kwargs=dict(
+                primary_keys=["image_name"],
+                image__image_path__name="local_path",
+            ),
+        ),
+        BatchTransform(
+            func=steps.publish_to_fiftyone_predictions_retrained,
+            inputs=[Required("local_images"), Required("detection_prediction_train")],
+            outputs=["fiftyone_predictions_model_b"],
+            transform_keys=["image_name", "detection_model_id"],
+            labels=[("stage", "fiftyone")],
+            kwargs=dict(
+                primary_keys=["image_name"],
+                image__image_path__name="local_path",
+            ),
+        ),
     ]
 )