svsal-bertopic

Experiment for analysis of the School of Salamanca corpus with BERTopic.

Developed together by @jcarrill718 and @awagner-mainz, the idea was to do dynamic topic modelling over the corpus. Unfortunately, we got stuck when evaluating the topics BERTopic suggested. Remnants of that endeavour can still be consulted in the bak directory, e.g. a notebook or a couple of visuals.

Since the beginning in 2024, the creation of text resources from the Salamanca corpus has been substantially reworked, and new opportunities for working with embeddings have appeared. Beginning in May 2025, the notebooks have thus been re-organized by @awagner-mainz. Different tasks -- embeddings creation, upload to a vector Db, dimensionality reduction/projection, analysis (Topic Modeling) -- have been separated and emphasis was put on creation of embeddings and uploading them to a vector Db that has been developed alongside (EmbAPI).

A 3D projection was computed and the School of Salamanca website has been enhanced with similarity search/RAG and visualization features (beta mode). For now, documentation is currently still pending, but you can get a glimpse on this (German-language) Blogpost by @tdiepenb and Tom Hunze.

Currently (Feb 2026) new embeddings have been created for an expanded set of texts (new texts have been released in the Salamanca project since 2024...). They will be uploaded to the vector Db next and the spatial projection(s) for the Salamanca website will be created. Hopefully, it will be possible also to return to the analysis part soon (TM). If that is of interest to you, you may want to consider following the repository.

Embedding Atlas export pipeline

For Embedding Atlas-friendly data export (2D projection + BERTopic metadata), run:

python 05-embeddings-export-atlas.py --input-dir ./out-data

The script reads the newest *_all_docs.parquet from out-data, selects a single embeddings_* column, runs UMAP (2D) for Atlas coordinates, fits BERTopic with its own UMAP reduction (default: 5 dimensions), and writes:

• out-data/embeddings_atlas_<provider>.parquet
• out-data/embeddings_atlas_<provider>_topics.parquet

Selecting an embedding provider/column:

• If multiple embeddings_* columns are present, the script uses the first one.
• To choose a specific provider, pass --embedding-column with the exact column name.

python 05-embeddings-export-atlas.py \
  --input-dir ./out-data \
  --embedding-column embeddings_openai_text-embedding-3-small

Optional output formats:

python 05-embeddings-export-atlas.py --also-jsonl --also-csv

Override BERTopic UMAP dimensionality (if needed):

python 05-embeddings-export-atlas.py --topic-umap-components 5

Optional local viewer launch (if installed):

pip install embedding-atlas
python 05-embeddings-export-atlas.py --launch-atlas

Atlas dataset columns

The output parquet keeps all existing non-embedding metadata columns from *_all_docs.parquet and adds:

• atlas_x, atlas_y (2D UMAP coordinates)
• text_snippet (truncated from text column)
• topic_id (BERTopic integer topic)
• topic_label (BERTopic topic name)
• topic_keywords (comma-separated top words)
• topic_probability (BERTopic confidence for assigned topic)
• embedding_provider (provider/model identifier from embedding column)

License

Documentation is published under Creative Commons Attribution 4.0 International License. Software code is published under MIT license. See LICENSE.