Spatial Transcriptomics: Cellpose + Cosine Prototype Pipeline

This project builds a baseline pipeline for spatial transcriptomics on a human breast cancer Xenium dataset. Cellpose segments cells from morphology images; cosine similarity against scRNA-seq-derived prototypes assigns each cell a biological identity.

In practice this separation breaks down: segmentation errors propagate directly into misclassification, and a sparse gene panel compresses all similarity scores toward zero, making assignments systematically ambiguous. We show this empirically and implement a post-processing ablation that uses biological signal as feedback to identify and selectively re-segment low-confidence cells.

The assignment problem is framed as a segmentation quality problem, not a cosine-similarity optimisation. A low or ambiguous similarity score is treated as a symptom that Cellpose captured the wrong transcript mix for that cell — cosine similarity is a proxy for boundary quality, not an end goal. A gene-subset experiment (NB04) confirms this: restricting the comparison to the top 100 highest-variance genes across the scRNA-seq prototypes substantially increases similarity scores and assignment margins, demonstrating that the full 5,054-gene space dilutes discriminative signal with noise. Cells that show low confidence even in this focused subset are used as high-priority re-segmentation candidates in the ablation (NB05).

Pipeline overview

breast_reference.h5ad ──► 01_build_prototypes  ──► prototypes_normalized.csv
                                                            │
morphology_focus/     ──► 02_assign_transcripts ──► cell_expression_aligned.csv
transcripts.zarr      ──►  (Cellpose + transcript assignment)
                                                            │
                                                            ▼
                          03_cosine_similarity  ──► cell_type_predictions.csv
                                                            │
                                                            ▼
                          04_gene_subset_cosine_similarity
                               ──► subset predictions + gene-subset confidence signals
                                                            │
                                                            ▼
                          05_evaluation_and_ablation
                               ──► refined predictions + QC figures + ablation metrics

Key findings

Gene space: Of 8,456 Xenium panel genes and 37,361 scRNA-seq genes, 5,054 were shared — the working gene space for all downstream comparisons. This overlap sounds large in absolute terms, but represents only ~14% of the scRNA-seq transcriptome, stripping out most of the discriminative signal that distinguishes cell types in the reference.

Reference cell types: 8 cell types were retained from the scRNA-seq reference (those with ≥ 1,000 cells): endothelial cell, endothelial cell of artery, endothelial cell of vascular tree, fibroblast, myofibroblast cell, pericyte, vascular associated smooth muscle cell, and vein endothelial cell.

Segmentation: Cellpose detected 82 cells in the 512×512 centre crop. Of 116,364 transcripts in the crop region, 78,588 (67.5%) were assigned to a cell; 37,776 fell outside all segmented masks. After filtering to shared genes, 74,005 transcripts were used to build the cell × gene expression matrix.

Cosine similarity: Best-match scores ranged from roughly 0.07–0.15, with a median of 0.104. This is far below what typical dense-panel pipelines achieve. The margin between the top two cell-type matches was near zero for most cells, making assignments inherently ambiguous regardless of threshold.

Gene subset: Restricting cosine similarity to the top 100 highest-variance genes across the scRNA-seq prototype profiles — a fully data-driven selection requiring no external annotation — substantially increased similarity scores and assignment margins relative to using all 5,054 shared genes. A marker-panel variant (21 genes) is also computed for comparison but is not used to drive the ablation, as the gene selection lacks a citable source.

Predicted cell type distribution:

Predicted cell type	Cells
endothelial cell	30
vein endothelial cell	16
fibroblast	12
pericyte	8
endothelial cell of artery	5
myofibroblast cell	5
endothelial cell of vascular tree	4
vascular associated smooth muscle cell	2

The heavy skew toward endothelial subtypes likely reflects both the stromal-vascular composition of the breast tissue region captured in the crop and the limited discriminative power of the sparse panel between closely related cell types.

Ablation (post-processing): Seven re-segmentation strategies were tested on low-confidence cells. Five use the full-gene confidence criterion (cosine score and margin below calibrated 25th-percentile thresholds); two new experiments use a combined criterion that additionally flags cells flagged by the top-100 variance subset, directly connecting the gene-subset analysis to the correction loop. Each experiment re-ran Cellpose locally around flagged cells using a smaller diameter (20 px vs 30 px) and accepted the new segmentation only if cosine similarity improved. Results varied across patch size, acceptance tolerance, and cell selection strategy — see the Results section for detail. The global shift in mean cosine similarity was modest across all experiments, consistent with a fundamental signal-limitation rather than a correctable segmentation problem.

Results

Segmentation

Cellpose was run on a 512×512 centre crop of the morphology image (morphology_focus_0000.ome.tif) with CELL_DIAMETER=30. 82 cells were detected. Transcript coordinates (stored in microns) were converted to pixel space using the physical pixel size (0.2125 µm/px) before assignment.

Marker gene validation

The heatmap shows mean expression of canonical marker groups per predicted cell type. A well-calibrated classifier would show a diagonal pattern — each marker group peaking in its corresponding predicted type. The eight marker groups evaluated were: fibroblast (COL1A1, COL1A2, DCN, LUM, PDGFRA), myofibroblast (ACTA2, TAGLN, MYL9, COL3A1), endothelial (PECAM1, VWF, KDR, RAMP2, ENG), pericyte (RGS5, PDGFRB, MCAM, CSPG4), vascular smooth muscle (ACTA2, MYH11, TAGLN, MYLK), cycling (MKI67, TOP2A, UBE2C, CENPF), immune (PTPRC, CD3D, CD3E, MS4A1, LYZ), and epithelial (EPCAM, KRT8, KRT18, KRT19).

Several canonical markers — including COL1A1, ACTA2, and VWF — were absent from the shared gene set, limiting what the heatmap can confirm.

###The degree to which the expected diagonal pattern appears, and which cell types show the clearest enrichment in their expected marker groups, should be described here after running notebook 05.

Gene subset analysis

NB04 tests whether restricting cosine similarity to a focused gene subset improves segmentation-quality assessment. The primary approach selects the top 100 genes by variance across the scRNA-seq prototype profiles — genes that vary most between cell-type means are, by definition, the most discriminative for this dataset. This selection is entirely data-driven and requires no external annotation or prior biological knowledge. A marker-panel variant (21 canonical marker genes) is also computed for comparison; both modes write separate output files and neither overwrites the full-gene baseline from NB03. The top-100 variance subset is the one used to drive the ablation in NB05.

Confidence and spatial distribution

The spatial overlay uses three colours: cells meeting the full-gene confidence threshold (green), cells flagged by the full-gene criterion alone (red), and cells additionally flagged by the top-100 variance subset criterion (orange). Orange cells are the re-segmentation targets unique to the subset-driven ablation experiments.

Cosine similarity before vs. after correction

---

Repo structure

Computational-Genomics-Project/
├── notebooks/
│   ├── 01_build_prototypes.ipynb              # scRNA-seq → per-cell-type prototype profiles
│   ├── 02_assign_transcripts.ipynb            # Cellpose segmentation + transcript assignment
│   ├── 03_cosine_similarity.ipynb             # cosine similarity + cell-type labelling
│   ├── 04_gene_subset_cosine_similarity.ipynb # focused gene subset + confidence signals
│   └── 05_evaluation_and_ablation.ipynb       # validation, marker eval, ablation study
├── data/                                      # all data files — git-ignored
├── outputs/                                   # figures and QC plots — git-ignored
├── requirements.txt
└── README.md

Data files required

Place the following files in data/ before running. Data files are not tracked by git.

File	Description	Used by
breast_reference.h5ad	scRNA-seq reference (AnnData, human breast cancer, 104,032 cells × 37,361 genes)	01
transcripts.zarr	Xenium transcript table — 386M transcripts, 8,455 unique genes	02
morphology_focus/	Xenium morphology images (OME-TIFF, 74,945 × 51,265 px)	02, 05

All intermediate files (prototypes_normalized.csv, cell_expression_aligned.csv, masks_center.npy, etc.) are generated by running the notebooks in order.

Setup

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Tested on Python 3.10–3.13. Cellpose GPU support requires a CUDA environment (e.g. Google Colab with GPU runtime).

Each notebook has a Config cell near the top — set paths there before running.

Key parameters

Notebook	Parameter	Value	Why
01	CELL_TYPE_COL	cell_type	Column name in breast_reference.h5ad obs
01	MIN_CELLS_PER_TYPE	1,000	Removes rare or poorly-sampled cell types; retains 8 of the original types
01	EXCLUDE_LABELS	unknown, unclassified, ambiguous, unclear, etc.	QC artifacts — would create spurious prototypes that steal assignments from real cell types
02	CROP_SIZE	512	Centre crop in pixels
02	CELL_DIAMETER	30	Expected cell diameter for Cellpose (pixels)
02	MICRONS_PER_PIXEL	0.2125	Xenium physical pixel size — required to align transcript coordinates (microns) with image coordinates (pixels)
03	HEATMAP_N_CELLS	50	Number of cells shown in the similarity heatmap
04	MODES_TO_RUN	["marker_panel", "top_reference_variance"]	Both subset strategies run back-to-back; outputs are saved separately and neither overwrites NB03 results
04	TOP_N_REFERENCE_GENES	100	Number of genes selected by the variance-based mode
05	SUBSET_TAG	top100_variance	Selects the data-driven top-100 variance subset from NB04 as the confidence criterion; change to marker_subset to use the marker panel instead
05	CONFIDENCE_THRESHOLD	0.5	Absolute threshold — flags 100% of cells under this panel; kept for reference only
05	MARGIN_THRESHOLD	0.05	Gap between top two prototype scores — the primary confidence signal
05	RETRY_DIAMETER	20	Smaller cell diameter used when re-segmenting low-confidence cells

Ablation experiment configurations

Experiment	Patch size	Retry diameter	Accept tolerance	Cell selection
baseline_no_correction	80	20	—	25th-percentile low-confidence (no re-seg)
strict_patch80_diam20	80	20	1e-4	25th-percentile low-confidence
relaxed_patch80_diam20	80	20	0.0	25th-percentile low-confidence
larger_patch150_diam20	150	20	0.0	25th-percentile low-confidence
worst25_patch150_diam20	150	20	0.0	Bottom 25% of scores only
subset_driven_patch80_diam20	80	20	0.0	Combined: low-confidence OR no subset signal
subset_driven_patch150_diam20	150	20	0.0	Combined: low-confidence OR no subset signal

The two subset-driven experiments extend the candidate pool by including cells flagged as low-confidence by the top-100 variance subset criterion, even if their full-gene cosine score alone did not trigger re-segmentation. This directly applies the NB04 findings to the correction loop.

A note on cosine similarity scores

Absolute cosine scores in this pipeline are low (≈0.07–0.15, median 0.104). The shared gene space covers only 5,054 of the 37,361 genes in the scRNA-seq reference. With most transcriptomic signal absent, even a correct cell-type match produces a low score in absolute terms. Relative comparisons (margin, rank) are more informative than absolute thresholds here. The gene-subset experiment in NB04 isolates this effect directly: restricting to the top 100 highest-variance genes raises both scores and margins substantially, confirming that the full-gene compression is a gene-space problem rather than a typing failure.

Limitations

Sparse gene panel. 5,054 shared genes represent ~14% of the scRNA-seq transcriptome. Many canonical markers (COL1A1, ACTA2, VWF) are absent, compressing all similarity scores and limiting biological validation.

No ground truth. Without experimentally confirmed cell identities, evaluation is indirect (marker enrichment, confidence distributions). The ablation measures internal consistency, not accuracy against a gold standard.

Single crop. Results are from a 512×512 centre crop — 82 cells from a dataset spanning tens of thousands. Representativeness of the broader tissue is unknown.

Segmentation–expression coupling. Any segmentation error directly distorts the expression profile of the affected cell. This motivates the post-processing step but also limits how much it can recover. The gene-subset analysis suggests that for many cells the issue is incomplete transcript capture rather than a misidentified boundary — a distinction that the ablation approach cannot fully resolve at this panel depth.

Background

Cellpose is a deep learning segmentation model that predicts a flow field pointing each pixel toward its nearest cell centre, then traces boundaries from those flows. This handles touching and overlapping cells without requiring manual parameter tuning per tissue type.

Cosine similarity measures the angle between two gene expression vectors, independent of their magnitude. It is the standard metric for comparing sparse, high-dimensional expression profiles.

Coordinate system alignment. Xenium transcript coordinates are in microns; the OME-TIFF image is in pixels. These are reconciled using the physical pixel size (0.2125 µm/pixel) before transcript assignment. Skipping this step results in zero assignments.