This internship task compares five LLMs on a document summarization task using an identical input document and prompt, then goes further to examine how each model's tokenizer, architecture, and design choices affect the output.
- ChatGPT
- Gemini
- Claude
- Llama 3.3 70B Versatile (via Groq)
- DeepSeek
document/asteroid_comet_article.pdf — "JPL Study Finds Near-Earth Asteroid Is Actually Comet," a NASA/JPL press article published July 16, 2026. Chosen deliberately as a niche, very recently published article, and because its density of exact names, dates, distances, and quotes makes hallucinations easy to catch.
Summarize the following article in no more than 150 words. The summary
should be written for someone with no prior knowledge of astronomy. Preserve
all key facts, names, dates, and figures exactly as stated in the source.
Do not add any information, statistic, name, or example that is not
explicitly present in the article. Structure the summary as a single
paragraph.
- Summary Quality
- Accuracy
- Conciseness
- Hallucinations
Rather than assigning scores from impression, each summary was checked sentence-by-sentence against the source article for: (1) factual claims not present in the source, (2) numeric/date/name errors, and (3) omission of load-bearing facts (named researchers, telescopes, publication venue). Word/character counts were computed.
| Model | Word Count | Summary Quality (1–5) | Accuracy (1–5) | Conciseness (1–5) | Hallucinations | Overall |
|---|---|---|---|---|---|---|
| DeepSeek | 139 | 5 — dense, well-ordered, all key figures retained | 5 — no errors found; correctly includes exact telescope sizes (3.6m, 1.5m, 8.2m) | 5 — under limit, no filler | None detected | 5/5 |
| ChatGPT | 144 | 4 — clear, slightly reorders for narrative flow | 5 — no errors found | 4 — under limit, omits telescope sizes | None detected | 4.5/5 |
| Claude | 155 | 4 — clear and accurate | 5 — no errors found | 3 — exceeds the 150-word cap given in the prompt | None detected | 4/5 |
| Llama 3.3 70B (Groq) | 133 | 3 — reads fine but thinner | 4 — accurate but omits both named researchers (Farnocchia, Hainaut) and the publication venue (Nature Astronomy) | 5 — shortest, under limit | None detected, but omission of key names weakens completeness | 3.5/5 |
| Gemini | 135 | 4 — well written | 2 — contains a factual error: states the object's "42-year orbit" where the source says a "4½-year orbit" — a nearly 10x numeric distortion, likely from misreading the "4½" character | 5 — under limit | Yes — one confirmed numeric hallucination | 3/5 |
- Gemini's numeric error is the most consequential finding of this comparison. It reported a "42-year orbit" instead of the source's "4½-year orbit." This is very likely a tokenization/character-parsing artifact — the "½" (a single Unicode fraction character) may have been misread or merged incorrectly during input processing. This is exactly the kind of subtle, high-confidence-sounding hallucination that a surface read would miss, since the sentence otherwise reads fluently.
- Claude broke its own conciseness instruction — 155 words against a 150-word cap. Worth noting plainly rather than glossing over since we're evaluating rigor, not defending a favorite.
- Llama 3.3 70B was accurate but incomplete — it never names Farnocchia or Hainaut, the two people whose findings the whole article is built around, and omits that the study was published in Nature Astronomy. Nothing it says is false, but it drops load-bearing information the other four models kept.
- No model fabricated a name, date, or telescope that wasn't in the source. Given the same document constraint prompt across all five, hallucination in the sense of invented content was rare — the one real failure (Gemini) was a distortion of an existing figure, not an invention.
Every one of these models saw the same article, but they did not see the same number of tokens. Tokenization directly affects: (a) cost per API call, (b) how much of a long document fits in one pass, and (c) subtly, how well a model handles rare technical terms — because if a word gets split into several sub-word pieces, the model has to reconstruct its meaning from parts rather than reading it as one unit.
| Model | Words | Characters | Approx. tokens (industry rule-of-thumb: ~4 characters/token for English) |
|---|---|---|---|
| ChatGPT | 144 | 965 | ~241 |
| Gemini | 135 | 890 | ~222 |
| Llama 3.3 (Groq) | 133 | 871 | ~218 |
| DeepSeek | 139 | 952 | ~238 |
| Claude | 155 | 1058 | ~264 |
| Model family | Tokenizer type | Vocabulary size (approx., cited) |
|---|---|---|
| GPT-4 / GPT-3.5 | tiktoken, BPE (cl100k_base) |
~100,000 tokens |
| GPT-4o and newer | tiktoken, BPE (o200k_base) |
~200,000 tokens |
| Llama 3.1 | Byte-level BPE (regex pre-tokenized) | ~128,000 tokens |
| Llama 4 | Byte-level BPE | ~200,000 tokens |
| Gemini / Gemma | SentencePiece (Unigram/BPE hybrid) | ~256,000+ tokens (Gemma 3: 262,144) |
| DeepSeek (recent) | Byte-level BPE | ~128,000 tokens |
Source: cross-referenced technical reports and tokenizer benchmarking papers (2025–2026), not vendor marketing pages.
What a bigger vocabulary buys you: a larger vocabulary means more common words and word-pieces get their own single token, so text compresses into fewer tokens — cheaper and faster to process, and it lets the model "see" whole words instead of fragments. This is a real, measurable design tradeoff between vocabulary size, embedding table size (memory cost), and compression efficiency — not just a marketing number.
We flagged which technical terms in this specific article (words like nongravitational, perturbations, astrometry, outgassing, provisionally) are the kind of low-frequency, multi-morpheme words that BPE/SentePiece tokenizers typically cannot store as a single vocabulary entry — they get split into 2–4 sub-word pieces regardless of which model processes them. This is exactly why: (a) the DeepSeek and Claude summaries, which retained the most of this technical vocabulary, likely used more tokens per unit of information than the Llama 3.3 summary, which sidestepped several of these words entirely — a real, direct tradeoff between technical precision and token economy visible in this exact dataset, not a hypothetical one.
| Model | Technical/low-frequency terms retained |
|---|---|
| DeepSeek | 7 (nongravitational, perturbations, outgassing, coauthor, designation, provisionally, planetary) |
| ChatGPT | 3 (nongravitational, perturbations, designation) |
| Claude | 5 (nongravitational, perturbations, astrometry, coauthor, planetary) |
| Gemini | 3 (coauthor, designation, planetary) |
| Llama 3.3 (Groq) | 5 (astrometry, designation, provisionally, planetary, irregularities) |
- DeepSeek performed best on this specific task: highest factual density with zero detected errors, under the word cap.
- ChatGPT and Claude are close seconds — accurate and complete, though Claude should tighten to the stated word limit.
- Gemini needs a second read-through in any workflow where a single-character numeric error (like "4½" → "42") could matter — this is a real, reproducible failure mode worth testing further with more documents containing fraction/decimal characters.
- Llama 3.3 70B (Groq) is a reasonable free/open-weight option for high-level gist but drops named entities that closed models retained — better suited to skimming than to citation-quality summarization.
Across five real model outputs on an identical prompt and document, none fabricated new facts, but two concrete, checkable failures emerged: Gemini's orbit-length error and Llama 3.3's omission of both named researchers. Ranked by combined accuracy and conciseness against the source: DeepSeek > ChatGPT ≈ Claude > Llama 3.3 (Groq) > Gemini. The tokenizer analysis shows this ranking is not just about summarization skill in the abstract — it correlates with how much technical vocabulary each model chose to retain versus simplify, which is itself a token-economy decision every one of these models is implicitly making on every response.
Model-Comparison/
├── README.md # this file
├── document/
│ └── asteroid_comet_article.pdf # the shared input document
├── summaries/
│ ├── chatgpt.md
│ ├── gemini.md
│ ├── llama_groq.md
│ ├── deepseek.md
│ └── claude.md