pith:6IXVWWO4
Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking
Language models learn to generate rationales before each token during pretraining to improve future predictions.
arxiv:2403.09629 v2 · 2024-03-14 · cs.CL · cs.AI · cs.LG
Add to your LaTeX paper
\usepackage{pith}
\pithnumber{6IXVWWO4WRT5VT36XR6GW7BB26}
Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge
Record completeness
Claims
after continued pretraining of an LM on a corpus of internet text with Quiet-STaR, we find zero-shot improvements on GSM8K (5.9%→10.9%) and CommonsenseQA (36.3%→47.2%) and observe a perplexity improvement of difficult tokens in natural text. Crucially, these improvements require no fine-tuning on these tasks.
that language models can initially learn to generate and effectively use internal rationales at each token to improve future text predictions, despite starting without knowledge of how to produce or apply such thoughts.
Quiet-STaR lets language models learn token-level rationales from general text, producing zero-shot gains on GSM8K and CommonsenseQA after continued pretraining.
References
Cited by
Receipt and verification
| First computed | 2026-05-17T23:38:48.938887Z |
|---|---|
| Builder | pith-number-builder-2026-05-17-v1 |
| Signature | Pith Ed25519
(pith-v1-2026-05) · public key |
| Schema | pith-number/v1.0 |
Canonical hash
f22f5b59dcb467dacf7ebc7c6b7c21d7923b2c8f7ab649fb3b91f13d073fad22
Aliases
· · · · ·Agent API
Verify this Pith Number yourself
curl -sH 'Accept: application/ld+json' https://pith.science/pith/6IXVWWO4WRT5VT36XR6GW7BB26 \
| jq -c '.canonical_record' \
| python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: f22f5b59dcb467dacf7ebc7c6b7c21d7923b2c8f7ab649fb3b91f13d073fad22
Canonical record JSON
{
"metadata": {
"abstract_canon_sha256": "f4409e4df469e5e7ae3e3127a2895fc4e2c1d886a715f729ec104cf299962786",
"cross_cats_sorted": [
"cs.AI",
"cs.LG"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"primary_cat": "cs.CL",
"submitted_at": "2024-03-14T17:58:16Z",
"title_canon_sha256": "58f96fea8ecf1be0ee9111de3692519516364c20b44332de0796546b9a2d27b3"
},
"schema_version": "1.0",
"source": {
"id": "2403.09629",
"kind": "arxiv",
"version": 2
}
}