This is a 28.9 million parameter language model that generates text on an ESP32-S3 microcontroller. It runs on the chip itself, with nothing sent to a server, and it displays generated text at 9.88 tokens per second on a small screen wired to the chip. It fits because most of the model lives in flash instead of RAM, using Per-Layer Embeddings, an idea from Google's Gemma 3n.
| Parameters | 28.9M stored (25M of them in a flash lookup table) |
| Chip | ESP32-S3, 512KB SRAM, 8MB PSRAM and 16MB flash |
| Speed | 9.88 tok/s end to end, 94.9 ms/token of compute |
| Connectivity | none, everything runs on the device |
| Model size | 14.9MB at 4-bit |
A microcontroller has very little fast memory. The ESP32-S3 gives you 512KB of SRAM, and only the values touched many times per token can live there: activations and norm weights. The dense core and output head, scanned once per position, sit in PSRAM. What is left is the embedding tables, and their size is what normally decides how big a model can be.
In this model, most parameters sit in an embedding table, which the model reads from rather than computes on. So that 25-million-parameter table stays in slow flash, and only the few rows each token needs are pulled from it, about 450 bytes. Most of the model is therefore never loaded to run it: it sits in flash and is sampled a little at a time.
That idea is Google's Per-Layer Embeddings, from Gemma 3n. Here it runs on the memory layout of a microcontroller instead of a phone or a GPU.
Each tier holds whatever is read at its own frequency:
SRAM (fast, tiny) activations and norm weights, touched many times a token
PSRAM (medium) the core and output head, read once per position
FLASH (huge, slow) the 25M-param table, about 6 rows read per token (~450 B)
The model was trained on TinyStories, so it writes short, simple stories and mostly keeps them coherent. It will not answer questions, follow instructions, write code, or know facts. That limit comes from the small part of the model that does the reasoning, and the memory trick does not change it. What is interesting here is the architecture, fitting a large model onto a tiny chip, rather than what a 28.9 million parameter model can say.
- Barista - espresso question answering
- TinyStories - story generation
Download and deployment are separate operations: one reaches the network, the other touches the board.
scripts/fetch_model.sh <tinystories|barista> # download, verify, install into artifacts/
scripts/deploy.sh <tinystories|barista> # generate headers, run gates, compile, flashfetch_model.sh checks the inference assets against a SHA-256 and byte size
pinned in the script, and cross-checks the release's own metadata.json against
those same pins. It installs nothing unless every check passes, so a failed
download leaves what you already have untouched. deploy.sh never reaches the
network; it works from whatever is in artifacts/<model>/. Both require the model
to be named, because the board holds one at a time and deploying replaces it.
The firmware details and the boot output to expect live in
firmware/esp32_tinystories/README.md. The reusable
architecture is in src/; the training, ablation and quantization code that
reproduces the published numbers is in research/tinystories/. The full method,
the ablations, and the on-chip measurements are written up in
RESULTS.md.
TinyStories is the dataset this trains on: short synthetic stories simple enough that a small model can still learn to write coherently (Ronen Eldan and Yuanzhi Li, Microsoft Research, arXiv:2305.07759). The other half is Per-Layer Embeddings, Google's design from Gemma 3n, which is what lets a big model fit on a small chip.
Andrej Karpathy's llama2.c is the reference for training a small language model and running it in plain C.
Detailed measurements and ablations are documented in RESULTS.md.