word2vec

Custom Skip-Gram and Bag-of-Words implementations trained on a $30$ MB Gutenberg subset, built with PyTorch and scikit-learn.

Dataset

Training data was drawn from $46$ books sourced from Project Gutenberg, producing a $33$ MB corpus with a vocabulary of $28,890$ tokens and $98%$ coverage of the WordSim-353 benchmark. Four additional Gutenberg books form the test set - see dataset.py for the full list.

Models

Skip-Gram

skip_gram.py contains a PyTorch implementation of the Skip-Gram model. Training configuration:

• Embedding dimension: 128
• Optimizer: Adam with default settings, lr=0.001
• Epochs: 30
• Context window radius: 4
• Negative sampling: 20 negative token pairs per batch

TensorBoard was used for experiment tracking and training was run locally on an NVIDIA GeForce GTX 1650.

Bag of Words (BoW)

bag_of_words.py uses a scikit-learn TF-IDF vectorizer. The corpus is split into consecutive, non-overlapping windows of 256 tokens, each treated as a document, yielding word vectors of dimension 11029.

Evaluation

Both models were evaluated on the WordSim-353 human-annotated word-pair similarity benchmark. Cosine similarity scores were rescaled to $[0, 1]$, and the dataset's original $[0, 10]$ annotations were normalised by dividing by $10$. Performance was measured using $\mathrm{MSE}$.

Results

Model	MSE	RMSE
Bag of Words	0.05451	0.23347
Skip-Gram	0.04049	0.20122