Gaon (가온)

Real LLMs, built from scratch. Gaon — pure Korean for "center/core." Rung 1 of the ladder to a frontier lab.

— train on a free GPU.

📦 Models on HF Hub: gaon-1.7b-v2-instruct · gaon-1.7b-v2-translate · GGUF — or run locally in one line: ollama run hf.co/k08200/gaon-1.7b-v2-instruct-GGUF

Community GGUF quants (12 sizes each, 516 MB – 3.4 GB, via @mradermacher): instruct static · imatrix — translate static · imatrix

🌐 Live demo: KO↔EN translator running in your browser (WebGPU, no server, nothing leaves your machine — currently validating with prebuilt Qwen3-1.7B weights; Gaon's own translation weights swap in once the upstream MLC toolchain skew is fixed).

Done: Qwen3-architecture models trained from scratch — Gaon-0.6B and Gaon-1.7B (v1 + a Chinchilla-scale v2 with code data), base + instruction-tuned, Korean + English. Same codebase, single-GPU to 4-GPU FSDP. Endgame: frontier. This repo is where the credibility + skill to get there is earned.

Results

Model	Params	Tokens	Compute	Final loss	Post-training
Gaon-0.6B	596M	12B	1× B200	2.48	instruct (SFT 2.12)
Gaon-1.7B	1.72B	12B	4× B200 (FSDP)	2.37	instruct (SFT 1.53)
Gaon-1.7B v2	1.72B	34B (+code)	4× B200 (FSDP)	1.96	instruct (SFT 1.26) · translate KO↔EN (SFT 1.35)

**Same data + same code, 3× params → lower loss (2.48 → 2.37); same params, 3× tokens

• code mixture → 2.37 → 1.96.** The from-scratch pipeline scales with both params and tokens as expected. Full write-up: docs/TECH_REPORT.md — architecture, training recipe, multi-GPU FSDP engineering, distillation, honest limits (including why the code mixture didn't fix coding — see §6).

Why not bigger: the 235B-class "full Qwen level" needs thousands of GPUs and tens of trillions of tokens — the capital game. The small models you can match on your own GPUs now, while learning every stage end to end. Same architecture, same pipeline; only scale (and money) differ.

Architecture (faithful to Qwen3)

Decoder-only · RMSNorm (pre-norm) · RoPE · Grouped-Query Attention · QK-Norm (Qwen3-specific per-head q/k RMSNorm) · SwiGLU · tied embeddings. See src/model/gaon.py. Bit-compatible with HuggingFace Qwen3 (round-trip logit diff 0.0), so we reuse its tokenizer and the HF post-training stack.

Layout

configs/            training configs (0.6b single-GPU, 1.7b 4-GPU FSDP, 1.7b v2)
src/model/          Gaon model — Qwen3-compatible architecture (config.py, gaon.py)
src/data/           prepare.py (download+tokenize+pack), loader.py (mmap batches)
src/train/          train.py (single-GPU + FSDP via torchrun, resume, disk-safe ckpts)
src/posttrain/      sft.py (TRL SFT + our->HF weight map), distill.py, dpo.py
src/eval/           generate.py (sampling), chat.py (REPL), benchmark.py
scripts/            sanity_check.py, verify_ckpt.py, test_chat.py, prepare_mixture.sh

Usage (chat with a trained model)

Point src/eval/chat.py at any instruction-tuned checkpoint directory — a local HF-format folder, e.g. checkpoints/gaon-1.7b-instruct/. Works on CPU, Apple Silicon (MPS), or CUDA; picks the device automatically.

pip install -r requirements.txt

# interactive REPL — type a message, get a reply, 'exit' to quit (best chat model)
python -m src.eval.chat --model checkpoints/gaon-1.7b-v2-instruct

# KO<->EN translation model — prompt like "다음 문장을 영어로 번역해줘: ..."
python -m src.eval.chat --model checkpoints/gaon-1.7b-v2-translate

# smaller/faster model
python -m src.eval.chat --model checkpoints/gaon-0.6b-instruct

# tune sampling (translation works best at low temperature)
python -m src.eval.chat --model checkpoints/gaon-1.7b-v2-translate \
    --max-new 256 --temperature 0.3

# non-interactive batch of test prompts (no REPL)
python -m scripts.test_chat checkpoints/gaon-1.7b-v2-instruct

# sanity-check a raw pretraining checkpoint (reload + score, not chat-ready)
python -m scripts.verify_ckpt checkpoints/gaon_1.7b_v2/latest.pt

Each turn is independent (no conversation history) — small models drift on long context, so keep prompts self-contained.

Reproduce (train from scratch)

pip install -r requirements.txt
python scripts/sanity_check.py                                        # correctness (seconds)
python -m src.data.prepare --dataset fineweb-edu --out data/fineweb_edu
python -m src.data.prepare --dataset codeparrot  --out data/codeparrot
torchrun --standalone --nproc_per_node=4 -m src.train.train \
    --config configs/gaon_1.7b_v2.yaml                                # pretrain (FSDP)
python -m src.posttrain.sft --ckpt checkpoints/.../ckpt_30000.pt \
    --data-jsonl data/distill.jsonl --out checkpoints/sft             # instruction tune
python -m src.eval.chat --model checkpoints/sft                       # chat with it

Roadmap

• Architecture + HF bit-compat correctness tests
• Data packing + mmap loader (EN + Korean + code)
• Pretraining loop (single-GPU and multi-GPU FSDP, resume, disk-safe)
• Post-training: distillation SFT → instruct (KO/EN chat)
• Gaon-0.6B base + instruct (loss 2.48)
• Gaon-1.7B base + instruct (loss 2.37) — scaling demonstrated
• Gaon-1.7B v2: Chinchilla-scale (34B tokens) + code mixture (loss 1.96)
• Gaon-1.7B-Translate: KO↔EN translation SFT (open parallel corpora + chat mix)
• Eval vs Qwen3-1.7B-Base (MMLU 25.1 vs 62.6, KMMLU 22.3 vs 35.5, HAERAE 19.9 vs 46.8) — fluent but chance-level knowledge at a 34B-token budget; the cleanest quantification of what tokens buy (see TECH_REPORT §6)
• Browser translator demo page (WebLLM/WebGPU, docs/demo) — currently validating with prebuilt Qwen3-1.7B weights; Gaon weight swap pending upstream MLC toolchain fix
• Scale same code to 7B (when compute/capital allow)

Compute reality (so the plan is honest)

Training FLOPs ≈ 6 × params × tokens. Gaon-1.7B (12B tokens) ≈ 1.2e20 FLOPs, done in ~26h on 4× B200. A Chinchilla-optimal 7B (~140B tokens) is 5.9e21 FLOPs — roughly 6,500 GPU-hours ($13k rented). Frontier is orders of magnitude beyond that; this repo is the on-ramp, not the destination.