WIP: Features/UI

.claude/skills/datapipe-examples/SKILL.md

@@ -18,7 +18,7 @@ Each `examples/*` pipeline has a dedicated setup skill — pick by what it does:
 | `e2e_template/image_detection` | YOLO detection + Label Studio human-in-the-loop → train → FiftyOne | **setup-e2e-template** |
 | `e2e_template/image_keypoints` | YOLO-pose keypoints + Label Studio → train → FiftyOne | **setup-e2e-template** |
 | `sam_cvat` | SAM3 text-prompt boxes+masks → CVAT pre-annotations | **setup-sam-cvat** |
-| `detection_tags` | YOLO detection + **tags** (per-scenario metrics), no Label Studio / FiftyOne, GT injected | **setup-detection-tags** |
+| `detection_tags` | YOLO detection + **tags** (per-scenario metrics), injected GT, FiftyOne A/B view | **setup-detection-tags** |
 
 ## Ask first — don't assume (only the unresolved)
 Clarify what's not obvious before acting — don't spin up services or pull data you don't need. Ask only the unresolved:

.claude/skills/setup-e2e-template/SKILL.md

@@ -65,8 +65,9 @@ conventions, or the freeze join silently yields nothing / classes don't match:**
 ## Per-scenario tag metrics → see the dedicated example
 Want to tag a scenario (e.g. dark-room pallets), add it to training, and measure the model on that
 scenario separately (baseline vs retrained)? That lives as its own self-contained example —
-`examples/detection_tags` (`tag`/`image__tag`/`tag_metrics` built into the pipeline, **no Label Studio
-or FiftyOne**, ground truth injected). Use the **setup-detection-tags** skill.
+`examples/detection_tags` (`tag`/`image__tag`, `CountMetrics_Subset_PipelineModel` tag arc,
+**no Label Studio**, ground truth injected, **FiftyOne** baseline vs retrained). Use the
+**setup-detection-tags** skill.
 
 ## Troubleshooting (may already be fixed — verify against current files)
 - **No model after `train`, exit 0** → datapipe swallows step errors; check `detection_training_status`, not the exit code.

.github/workflows/lib-datapipe-app.yml

@@ -86,6 +86,12 @@ jobs:
         run: |
           pytest libs/datapipe-app/tests
 
+      - name: Test ML observability integration
+        run: |
+          uv pip install --system -e "libs/datapipe-ml[sqlite,observability]" --no-deps
+          uv pip install --system -e "libs/datapipe-core[sqlite]"
+          pytest libs/datapipe-app/tests/test_observability_ml_integration.py
+
   test-example:
     runs-on: ubuntu-latest
     strategy:

.gitignore

@@ -1,19 +1,84 @@
+# Secrets & local env
 .env
+.env.*
 
+# VCS & IDE
+.git/
+.github/
 .DS_Store
-.idea
+.idea/
+
+# Python
 .mypy_cache/
 .pytest_cache/
+.ruff_cache/
 .venv/
 venv/
 __pycache__/
 *.egg-info
 dist/
+*.pyc
 .vscode/sftp.json
 
-test_data/
-
+# Lock files (deps resolved inside the image)
 poetry.lock
 uv.lock
+# ...except detection_tags: its whole point is reproducible cross-machine runs — the lock pins the
+# training stack (ultralytics etc.); without it every machine resolves different versions.
+!examples/detection_tags/uv.lock
+
+# ML models & checkpoints
 *.pt
-*.sqlite
+*.pth
+*.onnx
+*.ckpt
+*.weights
+*.safetensors
+*.h5
+*.pb
+*.tflite
+
+# Databases & local stores
+*.sqlite
+*.sqlite3
+*.db
+
+# Images & media (keep .svg — used in datapipe-app frontend src)
+*.jpg
+*.jpeg
+*.png
+*.gif
+*.webp
+*.bmp
+*.tiff
+*.tif
+*.ico
+*.mp4
+*.mov
+*.avi
+*.mkv
+
+# Tabular / binary datasets
+*.parquet
+*.feather
+*.arrow
+*.npy
+*.npz
+
+# Archives
+*.zip
+*.tar
+*.tar.gz
+*.tgz
+*.7z
+
+# Node
+**/node_modules/
+
+
+# Test & sample data
+test_data/
+**/sample_data/
+datapipe-examples/
+examples/**/input/
+examples/**/output/

examples/datapipe_app/Dockerfile

@@ -11,7 +11,7 @@ ENV PYTHONUNBUFFERED=1
 
 WORKDIR /app
 
-COPY pyproject.toml uv.lock ./
+COPY pyproject.toml README.md ./
 COPY libs/datapipe-core ./libs/datapipe-core
 COPY libs/datapipe-app ./libs/datapipe-app
 COPY examples/datapipe_app ./examples/datapipe_app

examples/datapipe_app/alembic/env.py

@@ -12,10 +12,6 @@
 if config.config_file_name is not None:
     fileConfig(config.config_file_name)
 
-# add your model's MetaData object here
-# for 'autogenerate' support
-# from myapp import mymodel
-# target_metadata = mymodel.Base.metadata
 from app import app
 
 target_metadata = [app.ds.meta_dbconn.sqla_metadata]

examples/datapipe_core/neo4j_pipeline/docker-compose.yml

@@ -1,3 +1,8 @@
+# Neo4j keeps image defaults (cannot run as arbitrary host uid).
+#   HOST_UID=$(id -u) HOST_GID=$(id -g) docker compose up -d
+x-host-user: &host-user
+  user: "${HOST_UID:-1000}:${HOST_GID:-1000}"
+
 services:
   neo4j:
     image: neo4j:latest

examples/detection_tags/.env.example

@@ -5,12 +5,14 @@ export S3_ENDPOINT_URL=http://localhost:9000
 export FSSPEC_S3_ENDPOINT_URL="${S3_ENDPOINT_URL}"
 export S3_PUBLIC_URL=http://localhost:9000
 export DATAPIPE_TAGS_DIR=s3://datapipe-tags
+# Local cache for training artifacts and images downloaded for FiftyOne (mounted into compose fiftyone service)
 export DATAPIPE_TAGS_TMP_DIR=/tmp/datapipe-tags
 export DB_URL=postgresql://postgres:password@localhost:5432/postgres
 # default schema; set a dedicated one (e.g. datapipe_tags) only if you SHARE this Postgres with
 # other pipelines — then `db create-all` needs the schema created first (see README).
 export DB_SCHEMA=public
-# FiftyOne (stage=fiftyone): dataset metadata in MongoDB, images rendered from local files
+# FiftyOne (stage=fiftyone): metadata in MongoDB; App UI in docker compose (:5151)
 export FIFTYONE_DATABASE_URI=mongodb://localhost:27017
 export FIFTYONE_DATASET_NAME=datapipe_detection_tags
-export LOCAL_IMAGES_DIR=/tmp/datapipe-tags-fiftyone-images
+# optional override; default is $DATAPIPE_TAGS_TMP_DIR/local_images (see detection/config.py)
+# export LOCAL_IMAGES_DIR=/tmp/datapipe-tags/local_images

examples/detection_tags/.gitignore

@@ -0,0 +1 @@
+.docker-data/

examples/detection_tags/.python-version

@@ -0,0 +1 @@
+3.10

examples/detection_tags/README.md

@@ -11,10 +11,11 @@ is split into **two parts around a checkpoint** so you can prep part 1 and rehea
 ## Deploy from scratch
 ```bash
 cp .env.example .env && set -a && source .env && set +a
-docker compose up -d            # postgres + minio + mongo (mongo backs FiftyOne)
-uv sync                         # cu124 torch + datapipe-ml[torch,fiftyone] + fiftyone + pi-heif
-# pre-AVX2 host only: force the lts polars to win (else training SIGILLs)
-uv pip uninstall polars polars-lts-cpu && uv pip install polars-lts-cpu==1.33.1
+HOST_UID=$(id -u) HOST_GID=$(id -g) docker compose up -d   # postgres + minio + mongo + fiftyone (:5151)
+uv sync                         # modern hosts: as-is. Do NOT edit deps/re-lock (drifts lib versions
+                                # across machines and changes training results — reproducibility!)
+# legacy pre-AVX2 host (e.g. epoch8 gpu5) only:
+#   uv sync --extra old-cpu && uv pip uninstall polars polars-lts-cpu && uv pip install polars-lts-cpu==1.33.1
 cd detection
 # DB_SCHEMA defaults to `public` (exists already). For a dedicated schema (sharing the Postgres with
 # other pipelines) create it first: psql "$DB_URL" -c "CREATE SCHEMA IF NOT EXISTS $DB_SCHEMA"
@@ -31,9 +32,9 @@ the split step honors it. The pre-staged cache holds **500 images**, so keep the
 ## Part 1 — baseline to checkpoint
 ```bash
 # from examples/detection_tags/detection
-python ../scripts/add_request.py --id base-train --n 325 --offset 0   --subset train
-python ../scripts/add_request.py --id base-val   --n 100 --offset 325 --subset val
-python ../scripts/add_request.py --id night-val  --n 25  --offset 425 --subset val --tag night --darken 0.25
+python ../scripts/add_request.py --id base-train --n 200 --offset 0   --subset train
+python ../scripts/add_request.py --id base-val   --n 75  --offset 200 --subset val
+python ../scripts/add_request.py --id night-val  --n 75  --offset 275 --subset val --tag night --darken 0.40
 datapipe step --labels=stage=load run
 datapipe step --labels=stage=train run            # model A
 datapipe step --labels=stage=count-metrics run    # re-run once if it prints "Batches to process 0"
@@ -44,7 +45,9 @@ datapipe step --labels=stage=fiftyone run         # GT + model-A predictions int
 
 ## Part 2 — retrain and watch the tag metric rise
 ```bash
-python ../scripts/add_request.py --id night-train --n 50 --offset 450 --subset train --tag night --darken 0.25
+python ../scripts/add_request.py --id night-train-a --n 50 --offset 350 --subset train --tag night --darken 0.30
+python ../scripts/add_request.py --id night-train-b --n 50 --offset 400 --subset train --tag night --darken 0.40
+python ../scripts/add_request.py --id night-train-c --n 50 --offset 450 --subset train --tag night --darken 0.55
 datapipe step --labels=stage=load run
 datapipe step --labels=stage=train run            # model B (night now in training)
 datapipe step --labels=stage=count-metrics run    # re-run once if "0 batches"
@@ -59,13 +62,15 @@ docker exec <mongo> mongosh --quiet --eval "db.getSiblingDB('fiftyone').dropData
 ```
 
 ## What you get
-- `detection_model_train__metrics_on_subset` — overall metrics per (model, subset).
-- **`tag_metrics`** — `(detection_model_id, tag_id, subset_id)` → precision/recall/f1. Compare model
-  A vs model B at `tag=night, subset=val`: recall rises once the tagged batch is in training.
-- **FiftyOne** dataset `$FIFTYONE_DATASET_NAME`: `ground_truth`, `predictions_model_a`,
-  `predictions_model_b`, plus `tag`/`subset` sample fields. Launch:
-  `fiftyone app launch "$FIFTYONE_DATASET_NAME" --port 5151 --address 0.0.0.0` (tunnel that port if
-  remote — local port must equal the remote port; filter by the `tag`/`subset` fields).
+- `pipeline_model__metrics_on_subset` — overall metrics per (model, subset).
+- **`pipeline_model__metrics_by_tag_on_subset`** — per `(detection_model_id, tag_id, subset_id)`.
+  Compare model A vs B at `tag_id=night, subset_id=val`: weighted recall/F1 rise after retraining.
+- Class tables: `pipeline_model__metrics_by_cls_on_subset`, `pipeline_model__metrics_by_tag_by_cls_on_subset`.
+- **FiftyOne** dataset `$FIFTYONE_DATASET_NAME`: fields `annotations`, `predictions_model_a`,
+  `predictions_model_b`; sample fields `tag_id` / `subset_id`. After `stage=fiftyone`, open the App
+  from docker compose: **http://localhost:5151** (remote → SSH tunnel `-L 5151:localhost:5151`).
+  Compose mounts `DATAPIPE_TAGS_TMP_DIR` (default `/tmp/datapipe-tags`); local images land in
+  `$DATAPIPE_TAGS_TMP_DIR/local_images`.
 
 ### Pre-staged cache (fast loads)
 If `DATAPIPE_TAGS_CACHE_DIR/gt.json` + `DATAPIPE_TAGS_CACHE_DIR/images/<file>.jpg` exist, the load