For 3B-7B LLMs, larger batch sizes with lower learning rates improve instruction-tuning benchmarks, early gradient and loss signals predict final quality, and stacked training matches phased training with fewer samples.
Phase Description # Samples Phase 00 Instruction following warmup: simple, template-based instruction-response pairs to transition the base models to instruction-following behavior
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs
For 3B-7B LLMs, larger batch sizes with lower learning rates improve instruction-tuning benchmarks, early gradient and loss signals predict final quality, and stacked training matches phased training with fewer samples.