Continual pretraining on texts augmented with LLM-generated "hidden thoughts" improves Gemma2-9B's MMLU accuracy more than standard continual pretraining, with the largest gains on harder questions and across domains.
Openwebmath: An open dataset of high-quality mathematical web text
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Mining Hidden Thoughts from Texts: Evaluating Continual Pretraining with Synthetic Data for LLM Reasoning
Continual pretraining on texts augmented with LLM-generated "hidden thoughts" improves Gemma2-9B's MMLU accuracy more than standard continual pretraining, with the largest gains on harder questions and across domains.