Use MathBERT (or similar math-aware embedding model) to embed concept text and cluster related items. MathBERT returns a vector for arbitrary text, so we can run any of our items through it.
Use cases
- Filter non-mathematical records. Pass non-mathematical items through to get clusters; helps find what to drop.
- Group related concepts. Clusters of mathematical items as candidate links or as structure for the UI.
Method
- Embed each item's text (name + description).
- Cluster in the embedding space.
- For each cluster, compute a medoid as representative — the item closest to all others by the same metric. (Centroid is min sum of squared distances; not meaningful here since it isn't an actual item.)
- Optionally consult an LLM to generate a description of each cluster.
Quality measures
Silhouette and Dunn indices, both of which only need a distance measure between items.
Questions
- Does MathBERT actually capture the distinctions we care about, or do we need a different embedding model?
- Best clustering algorithm (k-means, HDBSCAN, agglomerative)?
- Cluster on items, or on concepts (Union-Find merged)?
Use MathBERT (or similar math-aware embedding model) to embed concept text and cluster related items. MathBERT returns a vector for arbitrary text, so we can run any of our items through it.
Use cases
Method
Quality measures
Silhouette and Dunn indices, both of which only need a distance measure between items.
Questions