[FEATURE] Implementing statistics based on exploration from #216

[tsenoner]

Feature Request

Implementing statistics based on exploration from #216

Code of Conduct

• I agree to follow this project's Code of Conduct

[tsenoner]

Elbow method — used to find the optimal number of clusters
Silhouette score
Davies–Bouldin index
Calinski–Harabasz index

[jcoludar]

Projection-statistics MVP — engine + web PRs up for review.

Implemented as two PRs (engine first, web consumes it):

• Engine — feat(stats): projection statistics (cluster-validity + faithfulness) protspace#61: new protspace.stats subsystem.
• Web — feat(statistics): consume projection statistics in the bundle pipeline #295: prep service produces it, bundle reader tolerates it.

Cluster validity (the metrics from the comment above) — all per projection:

• Elbow → optimal K (KMeans sweep; knee = max perpendicular distance to the first→last inertia chord)
• Silhouette, Davies–Bouldin, Calinski–Harabasz at that K

Projection faithfulness — added so a user can tell whether to trust the map (the #216 "competitive" framing):

• kNN-overlap@k and trustworthiness / continuity between the source embedding and the projection (how much the reduction distorted the neighbourhood structure).

How it's delivered: computed at prep time and baked into the .parquetbundle as an optional fifth statistics.parquet part — additive and backward compatible (existing 3-/4-part bundles unchanged). The table is a tidy 8-column long format, so new statistics later add rows, not columns. CLI: protspace stats -i emb.h5 -p project_dir -o statistics.parquet, plus prepare --stats/--no-stats; the web prep service folds it in best-effort, outside the job timeout, and feature-probes the subcommand so it no-ops on older protspace.

Deliberately not in this MVP (follow-ups):

• Frontend rendering — the bundle now carries the statistics, but the UI doesn't display them yet (separate, committed follow-up).
• Embedding-space cluster-validity, annotation/feature label sources + ARI/NMI external validation, and the broader pair/edge/set analyses catalogued in [FEATURE] Explore statistics that are usefull to interpret the projection #216 (query↔reference proximity, cross-method consensus, Top-N) — future typed bundle parts, reusing the same registry.

Both PRs are green locally (engine full suite 534; web bundle 24 + prep 19) and ready for review.

[jcoludar]

How the projection statistics are implemented

A follow-up to the PRs above, documenting the design end-to-end (engine tsenoner/protspace#61 + web #295).

What the statistics are

Two families, computed per projection (per UMAP / PCA / t-SNE / … reduction in a project):

1. cluster_validity — does the projection form clean clusters? (no source embedding required)

• KMeans is run on the 2D/3D projection coordinates with an elbow-chosen K, and that labelling is scored by silhouette, Davies-Bouldin, and Calinski-Harabasz.
• An n_clusters meta row records the achieved K alongside the requested K, knee confidence, and the silhouette-optimal K (a cross-check).

2. faithfulness — does the projection preserve the high-dimensional structure? (requires the source embedding)

• kNN-overlap, trustworthiness, and continuity, each comparing the projection against its source pLM embedding using the reducer's own high-dim metric (euclidean by default, or e.g. cosine if that's what built the projection).

This is the elbow + silhouette/DB/CH cluster-validity set plus faithfulness discussed earlier in this thread.

How they're computed (engine — src/protspace/stats/)

• A lazy registry (stats/__init__.py → get_statistics()) returns [ClusterValidity, Faithfulness]. scikit-learn imports are function-local so import protspace / CLI startup stays fast.
• The driver (stats/driver.py) loops over each reduction: it picks the source embedding (explicit source → sole available embedding → best id-set cover), id-joins embedding rows to projection coordinates, reads the high-dim metric from the reducer params, then runs each statistic. Cluster-validity scores the full projection; faithfulness scores the aligned id-intersection subset. Every statistic and every projection is isolated in try/except, so one bad projection or one failing metric never sinks the report — it returns partial/empty rather than raising.
• Elbow selection (stats/cluster/kmeans_elbow.py): sweep K = 2…√n, pick the K of maximum perpendicular deviation of the (normalised) inertia curve from its first-to-last chord, with a silhouette-optimal-K cross-check and a high/low knee-confidence flag.
• Scale guards (all deterministic, rng_seed=42): faithfulness subsamples with a fixed seed above 5000 points and skips entirely above a 20000 hard ceiling (recording a skipped marker); silhouette samples above 5000; k is clamped to (n-1)//2 so sklearn's trustworthiness won't raise on small n.
• Output: one tidy long-format, 8-column Arrow table — space_kind, space_name, stat_family, label_kind, metric, metric_kind, value, extra_json. The schema is frozen: new statistics add rows, never columns; per-statistic detail (seed, sampling, requested K, metric used, skip reasons) lives in extra_json.

How you get them

CLI (manual):

protspace stats  -i embeddings.h5 -p <projections_dir> -o statistics.parquet
protspace bundle -p <projections_dir> -a annotations.parquet -s statistics.parquet -o out.parquetbundle

The new -s slot makes statistics the optional 5th part of the .parquetbundle (parts 1–4 being annotations, projection metadata, projection data, settings).

Web (automatic — prep pipeline): after the core bundle is built and shipped, a best-effort _maybe_add_statistics step runs outside the main timeout budget:

1. Feature-probes whether the installed protspace even has a stats subcommand (so the web side is safe to merge ahead of the stats-bearing engine release).
2. Runs protspace stats, then re-bundles to a temp file with -s and os.replaces it over the core bundle — atomic, so a mid-write timeout/kill can never corrupt the already-shipped bundle.
3. Emits a computing_statistics SSE stage for UI progress. Any failure or timeout → the bundle simply ships without statistics.

So the statistics travel inside the bundle end-to-end.

Known follow-ups

• Frontend rendering: the 5th bundle part is currently parsed-but-ignored on the web side — surfacing it in the UI is the remaining work.
• The prep service's protspace dependency floor still needs bumping to the stats-bearing release once engine Refactor/style: data loader functionality refactored, website responsiveness styles fixed.  #61 ships.