Classification of arXiv research articles into major arXiv categories using transformer-based models.
This project addresses the classification of research articles from the arXiv preprint repository into major academic categories. ArXiv is one of the largest open-access repositories of scientific preprints, hosting over 3 million articles spanning fields from physics and mathematics to computer science and economics.
The goal is to automatically assign each article to a major academic category using only the abstract as input. This is framed as a single-label text classification problem, where the first listed category is used as the primary label, consistent with arXiv's own convention.
Three transformer-based models are evaluated:
- SPECTER2 - a scientific document embedding model evaluated in a zero-shot setting as a domain-specific baseline requiring no training
- BERT-base - a general-purpose transformer model fine-tuned on the classification task
- SciBERT - a BERT-variant pre-trained on scientific text, fine-tuned on the same task.
This setup allows two key questions to be answered:
- How much does fine-tuning improve over zero-shot classification?
- Does domain-specific pre-training provide meaningful gains over general-purpose BERT?
The following three papers directly motivated the implementation choices in this project.
BERT introduced bidirectional transformer pre-training on large general-domain corpora,
demonstrating state-of-the-art performance across a range of NLP tasks including text classification.
The model uses a [CLS] token representation fed into a linear classification layer
for sequence classification tasks. This project uses its uncased version as a general-purpose
fine-tuning baseline to establish a general-domain performance reference for arXiv abstract
classification.
Beltagy et al. (2019) - SciBERT: A Pretrained Language Model for Scientific Text
SciBERT extends BERT by pre-training on 1.14 million scientific papers from Semantic Scholar, introducing a domain-specific vocabulary with only 42% token overlap with BERT's general vocabulary, illustrating the substantial linguistic difference between scientific and general domain-text. The authors demonstrate that SciBERT outperforms BERT-base by +2.11 F1 on scientific NLP tasks with fine-tuning. This project uses its uncased version as a domain-specific challenger to BERT-base, directly testing whether scientific pre-training improves classification of arXiv abstracts.
Singh et al. (2023) - SciRepEval: A Multi-Format Benchmark for Scientific Document Representations
SciRepEval introduces a comprehensive benchmark of 24 tasks across four formats (classification, regression, proximity, and search) for evaluating scientific document representations, and proposes SPECTER2, a family of multi-format models that learn a distinct document embedding per task format using task-format-specific adapters. The authors show that format-specific embeddings outperform single-embedding approaches. This project evaluates SPECTER2 with its classification adapter in a zero-shot setting to quantify the benefit of fine-tuning over a strong domain-specific baseline requiring no task-specific training.
The dataset used in this project is the arXiv metadata dataset published by Cornell University, available via Kaggle. For this project, only three fields are used:
- id - unique paper identifier used for aggregating model predictions
- abstract - the text input for classification
- categories - the raw arXiv category labels used to derive ground truth labels.
The dataset is split as follows:
| Split | Size | Purpose |
|---|---|---|
| Train | ~218k | Fine-tuning BERT and SciBERT |
| Validation | ~54k | Hyperparameter selection |
| Test | ~302k | Final evaluation of all models |
The training and validation splits are derived from a single stratified sample of ~272k papers, divided 80/20 into train (~218k) and validation sets (~54k). This sample size was chosen to fit within a 3-hour computational window on an NVIDIA H100 SXM GPU. Fine-tuning pre-trained transformer models is known to exhibit diminishing returns beyond tens of thousands of samples, making larger samples unlikely to yield meaningful improvements in classification performance. A held-out test set of ~302k papers is used for final evaluation of all models, with the validation set used exclusively for hyperparameter selection. All the splits are stratified by target category to preserve the original class distribution. Full details of the sampling and splitting procedure are documented in first three notebooks.
Full exploratory data analysis is available in EDA notebook. The key findings are summarized below.
The number of categories was treated as unknown at the outset. EDA revealed that mapping the raw arXiv categories to the 8-tier modern arXiv category taxonomy offered the most natural and interpretable structure for this task, resulting in 8 major categories used for all subsequent modeling.
The raw categories field contains arXiv category labels in two formats: (1) subcategory
format (e.g. cs.LG, math.CO) and (2) legacy format (e.g. hep-ph, cmp-lg) used
before the current taxonomy was introduced. All labels are mapped to 8 major categories
consistent with the current arXiv taxonomy,
with legacy categories mapped to their modern equivalents. The 8 major categories are:
physics, computer science, math, statistics, electrical engineering, biology, finance, and economics.
The resulting category distribution is heavily imbalanced, with physics alone accounting for ~50% of all papers, while top 3 categories (physics, computer science, and mathematics) covering ~88% of the dataset. This directly informs the choice of macro-F1 as the primary evaluation metric.
83% of papers carry a single major category. For the remaining 17%, co-occurrence analysis reveals that multi-label papers are concentrated in predictable domain overlaps (computer science/mathematics, physics/mathematics, and computer science/electrical engineering). Since arXiv convention treats the first listed category as primary, single-label classification is adopted using the first listed major category as ground truth. This covers 83% of papers exactly and applies the same rule consistently to the remaining 17%.
Abstract lengths are consistently short, with approximately 188 tokens on average. Fewer than 0.01% abstracts exceed BERT's 512-token limit, confirming that no truncation strategy or longer-context model variant is needed.
32.6% of abstracts contain LaTeX notation, heavily concentrated in mathematics (51.8%)
and physics (37.3%), and 55.3% of abstracts contain special characters. Since BERT's
tokenizer treats unrecognized symbols as [UNK] tokens, leaving LaTex and special characters
unprocessed would hurt model performance most in the two largest categories.
Preprocessing is therefore applied before tokenization.
The following preprocessing steps are applied to all abstracts before tokenization:
- LaTeX normalization - display math, inline math, and LaTeX commands are removed.
- Special character cleaning - non-ASCII characters and symbols are removed, while basic punctuation is retained as BERT's pre-training corpus includes punctuation and its removal would differ from the text the model was pre-trained on.
- Whitespace normalization - multiple spaces are collapsed and leading/trailing whitespace is stripped.
Lowercasing is intentionally omitted, as BERT's tokenizer handles casing internally. The
cleaned dataset is exported as three stratified splits containing id, abstract, and
target_category for all subsequent modeling steps.
Three models are evaluated, covering zero-shot and fine-tuning approaches:
| Model | HuggingFace ID | Type | Approach |
|---|---|---|---|
| SPECTER2 | allenai/specter2_base |
Scientific embedding | Zero-shot |
| BERT-base | bert-base-uncased |
General purpose | Fine-tuned |
| SciBERT | allenai/scibert_scivocab_uncased |
Scientific domain | Fine-tuned |
All models are loaded and fine-tuned using the HuggingFace Transformers library.
SPECTER2 is evaluated without any task-specific training. Each abstract is embedded using the classification adapter, and assigned to the category whose description embedding has the highest cosine similarity. One description embedding is constructed per major category using a short representative text. This serves as a domain-specific baseline requiring no labelled training data, making it possible to directly measure how much fine-tuning helps.
Both BERT and SciBERT are fine-tuned on the training split using the AdamW optimizer with 0.01 weight decay. Maximum sequence length is set to 256 tokens, covering the 75th percentile of abstract lengths while avoiding the memory overhead of the full 512-token limit. Class weights are applied inversely proportional to class frequency during fine-tuning to prevent the model from ignoring minority categories during training (physics representing ~50% of training data vs economics at ~0.4%).
Both BERT and SciBERT authors independently recommend fine-tuning with learning rates 2e-5, 3e-5, or 5e-5, batch sizes 16 or 32, and 2-4 training epochs. Due to computational and time constraints, the full grid over all values was not feasible. Learning rate 5e-5 was dropped as it is generally considered too aggressive for fine-tuning on well-structured text classification tasks and risks destabilizing the pre-trained weights. Training for 4 epochs was dropped as BERT and SciBERT authors themselves note that 2 or 4 epochs with a learning rate of 2e-5 works best across most datasets, suggesting 4 epochs adds little beyond 3 in most cases. Batch size 16 was later dropped in favour of 32, as the dataset size of ~218k samples makes the difference between the two negligible at this dataset size while batch size 32 better utilizes available hardware.
The resulting grid covers learning rates [2e-5, 3e-5] and epochs [2, 3] with batch size fixed at 32, producing 4 combinations per model evaluated on the validation set by macro-F1. The best configuration per model is used for final test set evaluation.
| Model | Best LR | Best Epochs | Val Macro-F1 |
|---|---|---|---|
| BERT | 3e-5 | 3 | 0.72 |
| SciBERT | 2e-5 | 3 | 0.75 |
Full evaluation results are available in Evaluation notebook.
| Model | Approach | Macro-F1 |
|---|---|---|
| SPECTER2 | Zero-shot | 0.44 |
| BERT | Fine-tuning | 0.73 |
| SciBERT | Fine-tuning | 0.75 |
Fine-tuning provides a substantial improvement over zero-shot classification, with BERT achieving macro-F1 0.73 compared to 0.44 for SPECTER2. SciBERT further outperforms BERT by +0.02 macro-F1, suggesting that domain-specific pre-training on scientific text provides a consistent but modest benefit over general-purpose BERT for this task.
| Category | SPECTER2 F1 | BERT F1 | SciBERT F1 |
|---|---|---|---|
| biology | 0.31 | 0.64 | 0.66 |
| computer_science | 0.63 | 0.88 | 0.90 |
| economics | 0.14 | 0.57 | 0.62 |
| electrical_engineering | 0.17 | 0.54 | 0.56 |
| finance | 0.48 | 0.71 | 0.70 |
| math | 0.66 | 0.90 | 0.91 |
| physics | 0.78 | 0.97 | 0.97 |
| statistics | 0.34 | 0.61 | 0.65 |
Per-class results follow the pattern expected from the EDA findings. Physics and mathematics, the two largest and most linguistically distinct categories, achieve the highest F1 scores across all models. Economics, electrical engineering, and statistics are the most challenging categories across all models, consistent with their smaller training set sizes and significant overlap with adjacent fields identified during EDA.
SciBERT's most notable improvement over BERT is on economics (+0.05 F1), suggesting that domain-specific pre-training is most beneficial for categories with highly specialised vocabulary.
Error analysis is conducted on SciBERT, the best performing model. The per-category error rates are as follows:
| Category | Error Rate |
|---|---|
| economics | 33.6% |
| statistics | 29.0% |
| electrical_engineering | 28.7% |
| finance | 20.9% |
| biology | 20.1% |
| computer_science | 13.3% |
| math | 7.7% |
| physics | 4.2% |
Error rates correlate strongly with category size and inter-domain overlap, as the three most error-prone categories (economics, statistics, electrical engineering) are also the three smallest categories with the strongest co-occurrence signals identified during EDA.
Confusion matrix analysis reveals that misclassifications are concentrated along domain boundaries identified during EDA. Electrical engineering is consistently confused with computer science across all models (15-20% of EE papers predicted as CS), reflecting the genuine overlap between signal processing and machine learning research. The statistics/computer science and finance/economics confusion similarly align with co-occurrence patterns observed in EDA, suggesting these errors reflect inherent label ambiguity rather than model failure.
This project addressed the classification of arXiv research articles into 8 major academic categories using transformer-based NLP models. EDA revealed 8 major categories as the optimal classification structure. Three models were evaluated (SPECTER2 in a zero-shot setting, and BERT and SciBERT with fine-tuning) across a stratified test set of ~302k papers.
Fine-tuning provides a substantial improvement over zero-shot classification. BERT achieves a macro-F1 of 0.73 compared to 0.44 for SPECTER2, a 66% relative improvement that confirms task-specific training is essential for reliable classification of scientific abstracts.
Domain-specific pre-training provides a consistent but modest benefit. SciBERT outperforms BERT by +0.02 macro-F1, suggesting that the quality of fine-tuning data is more impactful than pre-training domain for this task. The most notable per-class improvement is on economics (+0.05 F1), where domain-specific vocabulary appears to give SciBERT the greatest advantage.
Error analysis reveals that the misclassifications are concentrated along domain boundaries identified during EDA, most notably electrical engineering/computer science and statistics/computer science. These errors reflect inherent label ambiguity in the arXiv taxonomy rather than model failure, as many papers genuinely span multiple fields.
Future work could explore several directions. Fine-tuning SPECTER2 with its classification adapter would provide a fairer comparison against BERT and SciBERT and better quantify the true contribution of domain-specific pre-training. Multi-label classification could be adopted to handle the 17% of papers carrying multiple categories. Larger fine-tuning datasets for minority classes such as economics and finance could reduce the high error rates observed in these categories. Finally, more recent architectures such as ModernBERT and instruction-tuned models such as Flan-T5 are worth exploring as next steps beyond BERT-class models.
arxiv-classification/
├── data/ # train, val, and test splits
├── notebooks/ # data, EDA and metrics notebooks
├── results/ # model predictions and experiment outputs
├── src/
│ ├── data.py # data loading and prediction saving
│ ├── device.py # device detection (CUDA, MPS, CPU)
│ ├── embeddings.py # embedding generation
│ ├── engine.py # WeightedTrainer and class weight computation
│ ├── fine_tune.py # fine-tuning entry point
│ ├── grid_search.py # hyperparameter grid search
│ ├── labels.py # category definitions and label mappings
│ ├── run_experiments.sh # runs all experiments sequentially
│ └── zero_shot.py # SPECTER2 zero-shot classification
Python 3.12+pip
- Clone the repository
cd arxiv-classification- Create virtual environment:
python -m venv env - Activate virtual environment:
source env/bin/activate - Install dependencies:
pip install -r requirements
Download the arXiv dataset from Kaggle
and place the raw (unzipped) file in the data/ directory.
Then run 01_, 02_ and 03_ notebooks to generate train, val, and test splits:
cd notebooks/
jupyter-labThis will produce arxiv-train.json, arxiv-val.json, and arxiv-test.json in data/,
and metrics.json in results/.
Run the full experiment pipeline sequentially (SPECTER2 zero-shot, followed by
BERT and SciBERT fine-tuning):
cd src/
chmod +x run_expeirments.sh
bash run_expeirments.shLogs are written to logs/ for each model.
To run overnight without keeping the terminal open:
nohup bash run_experiments.sh &cd src/
python zero_shot.pycd src/
python fine_tune.py --model bertcd src/
python fine_tune.py --model scibertOnce all experiments are complete, open the 04_Evaluation.ipynb notebook to compute
and visualize results:
cd notebooks/
jupyter-lab