This project implements language models in dependency-free C++, eliminating the need for PyTorch or Python to train a transformer locally. The core implementation is a decoder-only GPT architecture featuring custom tensors, embeddings, multi-head causal self-attention, layer normalization, cross-entropy loss, and an analytical backward pass with the AdamW optimizer - all contained within main.cpp ,llm.mm and the include/ directory also a token level [BPE] tokenizer.h implementation inside include. With no autograd engine or external frameworks, every gradient is explicitly derived and written out. The model achieves a validation loss of 1.6371 nats after 76 minutes of CPU training on 31.4 million characters, demonstrating that character-level language modeling at this scale is highly tractable on commodity hardware without external dependencies. On a GPU (CUDA/bfloat16), a validation loss of 2.3918 is reached in under 83 minutes, achieving a peak throughput of 19.6k tokens per second.
| S.No. | time | val_bpb / Metric | scale | Date | Contributors |
|---|---|---|---|---|---|
| 1 | 168 hours | 29.41 PPL (~0.93 BPB) | 124M (32x TPU v3) | Feb 2019 | OpenAI (GPT-2 Small) |
| 2 | 45 min | 3.28 Val Loss (~0.748 BPB) | 124M (8x H100) | May 2024 | Andrej Karpathy (llm.c) |
| 3 | 2.98 min | 3.28 Val Loss (~0.748 BPB) | 124M (8x H100) | Feb 2025 | Keller Jordan et al. (Modded-NanoGPT) |
| 4 | 72 hours | 22.7 PPL (~0.85 BPB) | 125M (32x A100) | Feb 2024 | Meta (MobileLLM-125M) |
| 5 | ~24 hours | ~1.02 BPB | 135M (64x H100) | Jul 2024 | Hugging Face (SmolLM-135M) |
| 6 | 61.3 min | 0.7176 | 10.82M (T4) | Mar 2026 | @Eamon2009 |
| 7 | 6.1 min | 0.9250 | 1.99M (T4) | Feb 2026 | @Eamon2009 |
| 8 | 39.4 min | 1.3145 | 0.82M (CPU) | July 2026 | @Eamon2009 |
| 9 | 76.2 min | 1.6371 | 0.82M (CPU) | Jan 2026 | @Eamon2009 |
More broadly, the primary contribution of this work lies in its absolute transparency. Every gradient in the backward pass is explicitly written and readable, and every tensor operation is a standard C++ function. By exposing exactly what frameworks like PyTorch compute under the hood, this implementation provides a clear educational pathway. We believe that this fundamental understanding is the true foundation of genuine expertise in deep learning.
Alongside it sits a parallel PyTorch implementation in engine/main.py and engine/inference.py, so you can train and generate the same architecture with torch + tiktoken when you want speed instead of transparency. There's also an experimental integrated-GPU path in iGPU/. The point of this repo is the C++ core. The PyTorch exist to make the model usable, but if you're here to train a GPT without a framework doing the work for you, include/backward.h is where to start reading.
The custom C++ backend is transparent but slow: a CPU executes scalar matrix multiplication at roughly 1-10 GFLOP/s. An NVIDIA RTX 4090 delivers ∼80 TFLOP/s-an 8,000–80,000× speedup for the same computation.The LibTorch port replaces the custom backend with PyTorch’s C++ API, gaining cuBLAS-accelerated matrix operations. The transformer architecture remains unchanged only the compute layer is modified. A single line migrates the model to GPU:
model->to(torch::kCUDA)
which transfers all parameters to GPU memory; all subsequent torch::matmul calls dispatch to cuBLAS automatically.
technical notes: docs
The fastest way to see the whole pipeline - tokenize, train, checkpoint, generate - using the bundled character-level corpus:
get the data set first :
cd data # you set the file size for data set
python data_set.py # also get any dataset from hugging face datasets # run this
g++ -std=c++17 -O3 -march=native -fopenmp -I. -Iinclude -o llm.exe main.cpp
./llm.exe data/input.txtshould see something like this
llm.cpp
[BPE] Text length: 31456820 characters
[BPE] Target vocab size: 2048
[BPE] Running 1944 merges...
[BPE] step 100/1944 vocab=204
...
[BPE] Done. Final vocab size: 2048
[DATA] Total tokens : 9950307
[DATA] Train tokens : 8955276
[DATA] Val tokens : 995031
+-----------------------------------------------------------------------------+
| LLM.cpp |
|======================================+======================================|
| Parameter / Spec | Value |
|--------------------------------------+--------------------------------------|
| Host CPU Device | AMD Ryzen 5 PRO 3500U w/ Radeon... |
| Max Sequence Length | 64 |
| Vocab Size (BPE Merges) | 2048 |
| Number of Layers | 4 |
| Number of Heads | 2 |
| Channels (Embeddings) | 128 |
| Number of Parameters | 1326336 |
| Repetition Penalty | 10 |
| Repetition Window | 10 |
+--------------------------------------+--------------------------------------+
step 1/5000 | loss 7.662571 | lr 5.00e-04 | 5456.40 ms | 375 tok/s
step 2/5000 | loss 7.580595 | lr 5.00e-04 | 4994.96 ms | 410 tok/s
step 3/5000 | loss 7.540967 | lr 5.00e-04 | 4747.15 ms | 431 tok/s
step 4/5000 | loss 7.503835 | lr 5.00e-04 | 4489.64 ms | 456 tok/s
step 5/5000 | loss 7.464558 | lr 5.00e-04 | 4701.70 ms | 435 tok/s
step 6/5000 | loss 7.422751 | lr 5.00e-04 | 4645.18 ms | 440 tok/s
step 7/5000 | loss 7.387443 | lr 5.00e-04 | 4677.04 ms | 437 tok/s
step 8/5000 | loss 7.343823 | lr 5.00e-04 | 4753.86 ms | 430 tok/s
step 9/5000 | loss 7.327121 | lr 5.00e-04 | 4874.67 ms | 420 tok/s
step 10/5000 | loss 7.279384 | lr 5.00e-04 | 4939.59 ms | 414 tok/s
step 11/5000 | loss 7.264264 | lr 5.00e-04 | 4968.59 ms | 412 tok/s
...
This trains from scratch on data/input.txt and writes the best checkpoint to best_model.bin. Once you have a checkpoint, generate or chat with it:
./llm.exe data/input.txt --generate
./llm.exe data/input.txt --chat --chat-tokens 300debugging tip: drop -O2 for -g when compiling if you want to step through include/backward.h or include/gpt.h in a debugger — the manual backward pass is much easier to follow one breakpoint at a time.
llm.exe [data_path] [--generate] [--chat] [--chat-tokens N]# to train
cd engine
python main.py
# for inference stay in engine/
python inference.pySingle GPU or CPU Mode (Default Fallback) If you do not pass any distributed environment parameters, the script automatically defaults to running locally on a single GPU (if available) or CPU.
cd engine
cd distributed
python train.pyTo train using multiple GPUs on a single machine, use PyTorch's torchrun launcher. Replace --nproc_per_node with the number of GPUs available on your machine (e.g., 2, 4, or 8).
cd engine
cd distributed
# Example: Running on 4 GPUs on a single machine
torchrun --standalone --nproc_per_node=4 train.pyOptimized for any hardware - from your laptop CPU to a GPU.
 Apple Silicon M1–M4 · Metal |
 M Pro 19-core GPU · Metal |
 M Max 40-core GPU · Metal |
 M Ultra 76-core GPU · Metal |
 H100 SXM5 · NVLink · 80 GB HBM3 |
 A100 SXM4 · 80 GB HBM2e |
 B200 Blackwell · 192 GB HBM3e |
 T4 CUDA · Cloud · Colab |
 Radeon RX ROCm · RDNA3 |
 MI300X ROCm · 192 GB HBM3 |
 CPU (x86-64) OpenMP · AVX · Zero install |
|
Note: No over-promising of anything
## Structure
```text
.
├── .devops/ # DevOps, Docker, & proxy configurations
│ ├── docker-compose.dev.yml # Local development compose setup
│ ├── docker-compose.gpu.yml # Compose setup with GPU acceleration
│ ├── docker-compose.yml # Base Docker Compose file
│ ├── Dockerfile # Primary build setup
│ ├── Dockerfile.backend # Backend service Docker setup
│ ├── Dockerfile.cpp # Native C++ build container setup
│ ├── Dockerfile.frontend # Frontend UI Docker setup
│ └── nginx.conf # Reverse proxy configuration
│
├── .github/ # GitHub Actions CI/CD workflows & repository templates
│ ├── ISSUE_TEMPLATE/ # Bug report & feature request templates
│ │ ├── bug_report.md
│ │ ├── config.yml
│ │ └── feature_request.md
│ ├── workflows/ # CI/CD automation pipelines
│ │ ├── check.yml
│ │ ├── ci-approval.yml
│ │ ├── ci.yml
│ │ ├── docker-publish.yml
│ │ ├── jekyll-gh-pages.yml
│ │ ├── main.yml
│ │ ├── pr-check.yml
│ │ ├── release.yml
│ │ └── test.yml
│ ├── dependabot.yml # Automated dependency update configuration
│ └── pull_request_template.md # Pull request contribution template
│
├── assets/ # Visual execution benchmarks & screenshots
│ ├── run_2026-07-16 165731.png
│ ├── run_20260430_192930.png
│ ├── run_20260508_110726.png
│ └── run_20260530_165216 (1).png
│
├── benches/ # Performance benchmarking suite
│ └── bench.cpp # C++ benchmark execution script
│
├── config/ # Global project configurations
│ └── config.h # C++ global configuration header
│
├── data/ # Dataset ingestion scripts & raw samples
│ ├── data_set.py # Data loader preparation script
│ └── input.txt # Sample raw dataset text
│
├── docs/ # Documentation media & report graphics
│ └── training_report.png # Model training performance chart
│
├── engine/ # Core execution & inference engines
│ ├── distributed/ # Multi-node / distributed training & inference
│ │ ├── infer.py # Distributed inference pipeline
│ │ └── train.py # Distributed training pipeline
│ ├── llm.cpp/ # Low-level CUDA/C++ runtime engine
│ │ ├── config/ # Engine-specific configurations
│ │ ├── include/ # CUDA kernels & C++ architecture headers
│ │ ├── best_model.bin # Trained binary weights checkpoint
│ │ ├── llm.cu # CUDA GPU execution source
│ │ ├── llm.exe # Compiled engine binary executable
│ │ ├── llm.py # Engine Python bindings/wrapper
│ │ ├── Makefile # Engine build compilation setup
│ │ └── train.mm # Metal training harness
│ ├── logs/ # Execution log files
│ ├── inference.py # Python model inference entry point
│ ├── main.py # Primary Python execution entry point
│ └── llm.pt # Pretrained PyTorch model checkpoint
│
├── include/ # Core C++ neural network architecture headers
│ ├── attention.h # Multi-head attention implementation
│ ├── backward.h # Backpropagation & gradient calculation utilities
│ ├── block.h # Transformer block assembly
│ ├── embedding.h # Token & positional embedding logic
│ ├── feedforward.h # Feed-forward layer implementation
│ ├── gpt.h # Full GPT transformer architecture layout
│ ├── layernorm.h # Layer normalization operations
│ ├── linear.h # Fully connected dense layer
│ ├── lm.h # High-level language model interface
│ ├── sampler.h # Token sampling routines (Top-K, Top-P, Temperature)
│ ├── tensor.h # Multidimensional array data structures
│ ├── tokenizer.h # Text tokenization logic
│ └── torch_bridge.h # PyTorch interoperability layer
│
├── scripts/ # Automation & compilation scripts
│
| Argument | Description |
|---|---|
data_path |
Plain-text corpus used to build the tokenizer and train/validation split |
--generate |
Load weights and continuously generate text |
--chat |
Load weights and start interactive terminal chat |
--chat-tokens N |
Max generated tokens per chat response |
| Env var | Default | Description |
|---|---|---|
GPT_DATA_PATH |
data/input.txt |
Override the default training corpus |
GPT_MODEL_PATH |
best_model.bin |
Override the checkpoint path |
No third-party runtime dependency - it builds from main.cpp, config/config.h, and include/*.h alone.
- Byte Pair Encoding (BPE) Tokenizer - Built entirely from scratch. Compiles a custom vocabulary directly from the training corpus by running iterative token-pair merges until it hits a targeted vocabulary threshold.
- Train/validation split via
DataLoader - Token + positional embeddings
- Multi-head causal self-attention with explicit QKV projections
- Pre-layer-norm residual transformer blocks
- Feed-forward MLP with ReLU
- Cross-entropy loss
- Fully analytical backward pass - every gradient (attention, layer norm, MLP, embeddings) is derived mathematically and coded explicitly in
include/backward.h, not autograd - AdamW optimizer (first/second moment estimates, weight decay)
- Checkpoint save/load
- Autoregressive generation and terminal chat mode
Hyperparameters live in engine/llm.cpp/config/config.h and require a rebuild to take effect:
// note: The c++ version only runs on cpu not on GPU
static const unsigned int SEED = 1337;
static const double TRAIN_SPLIT = 0.9;
static const int BATCH_SIZE = 32;
static const int BLOCK_SIZE = 64;
static const int MAX_ITERS = 5000;
static const int EVAL_INTERVAL = 500;
static const float LEARNING_RATE = 5e-4f;
static const int EVAL_ITERS = 25;
static const int N_EMBD = 128;
static const int N_HEAD = 2;
static const int N_LAYER = 4;
static const float DROPOUT = 0.05f;
static const int BPE_VOCAB_SIZE = 2048; engine/main.py trains the same architectural idea with torch, torch.nn, and GPT-4 BPE tokenization via tiktoken, useful when you want to scale past what C++ loops can comfortably train on CPU.
cd engine
python fineweb_dataset.py # you can also use data/input.txt also
python main.pyIt looks for engine/input.txt by default; point it elsewhere with QUADTRIX_TRAIN_DATA if needed. Run inference against a saved checkpoint:
python engine/inference.py --checkpoint engine/best_model.pt --prompt "Once upon a time" --max-new-tokens 100| Metric | Character-Level | Small Scale | Large Scale |
|---|---|---|---|
| Parameters | 0.83M | 2.00M | 19.17M |
| Layers | 4 | 4 | 4 |
| Embedding dim | 128 | 200 | 200 |
| Attention heads | 4 | 4 | 4 |
| Context length | 64 | 200 | 200 |
| Vocab | 105 char | 110 char | ~50K BPE |
| Corpus | TinyStories | TinyStories | Children's Stories |
| Iterations | 3,000 | 5,000 | 5,000 |
| Train loss | 1.5632 | 0.9045 | — |
| Val loss | 1.6371 | 0.9301 | — |
| Gen. gap | 0.0739 | 0.0256 | — |
See run.md and the leaderboard in the full docs for more configurations.
| Project | Focus | Language | Autograd |
|---|---|---|---|
| nanoGPT / minGPT | Minimal, educational GPT training | Python | PyTorch |
| llama2.c | Inference-only | C | None |
| llm.cpp | Training and inference, manual backward pass | C++ / Python | Manual (C++) + PyTorch |
I'd like the C++ core (main.cpp, include/, config/) to stay dependency-free and to stay the part of this repo that explin transformer internals directly. The PyTorch engine, include, and ci are welcome to grow more features, integrations, and CI polish. If you build a port to another language or framework, I'm happy to link to it from a notable-forks section; just open an issue or PR.
- Vaswani et al., "Attention Is All You Need", 2017
- Radford et al., "Language Models are Unsupervised Multitask Learners" (GPT-2 technical work), 2019
- Brown et al., "Language Models are Few-Shot Learners" (GPT-3 paper), 2020
- Meta AI, "The Llama 3 Herd of Models" (Llama 3 paper), 2024
- Andrej Karpathy, nanoGPT repository as an educational reference point
- HuggingFace Datasets for FineWeb and other pretraining/fine-tuning datasets
If you find llm.cpp helpful in your research cite as:
@misc{llm.cpp,
author = {Eamon Sippy},
title = {llm.cpp: LLM training in C++ \& Python},
year = {2026},
publisher = {GitHub},
journal = {GitHub repository},
url = {https://github.com/LMGNU/llm.cpp}
}
GPL-3.0