Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Bigger batches and lower learning rates win for small-LLM tuning.

desk verdict Useful SFT recipe map for small LLMs, but the headline batch-size effect is confounded with data volume and the early-stopping predictor is post-hoc. read the letter →

arxiv 2412.13337 v1 pith:3UIWKHMN submitted 2024-12-17 cs.LG cs.AIstat.ML

Aldo Pareja , Nikhil Shivakumar Nayak , Hao Wang , Krishnateja Killamsetty , Shivchander Sudalairaj , Wenlong Zhao , Seungwook Han , Abhishek Bhandwaldar
show 5 more authors
Guangxuan Xu Kai Xu Ligong Han Luke Inglis Akash Srivastava
This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords supervisedfine-tuningsmalllanguagemodelsinstructiontuningbatchsizelearningratetrainingdynamicsgradientnormMMLU
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to turn supervised fine-tuning of small language models (3B–7B parameters) from folklore defaults into an evidence-based recipe. It systematically varies batch size, learning rate, warmup steps, schedule, and training strategy across four open base models and several instruction datasets, evaluating on MMLU, MTBench, and the Open LLM Leaderboard v2. The central claim: larger effective batches (roughly 4,000–8,000 samples) paired with lower learning rates beat the widely used TULU settings; warmup and cosine decay can be dropped; and stacked training matches or beats phased training while being more sample efficient. It also claims that early training dynamics—low gradient norm and high training loss—predict final benchmark scores well enough to stop unpromising runs early. If right, this gives researchers with modest compute a simpler, cheaper fine-tuning pipeline and a way to know early whether a run will pay off.

What carries the argument

The load-bearing instrument is the controlled sweep: one factor at a time (batch size, learning rate, warmup steps, schedule, training strategy) is varied while everything else is held fixed, under three named configurations—LAB, TULU, and the paper's TULU++—and evaluated on MMLU and MTBench, with Leaderboard v2, ARC, and GSM8K as secondary checks. The mechanism they invoke for the batch-size effect is gradient-noise reduction: averaging over more samples cuts gradient variance roughly as $1/\sqrt{n}$, producing steadier updates that keep the model near its pretrained parameters. The early-dynamics predictor is the other central object: plotted curves of log-transformed gradient norm and total training loss over samples, which the paper reads as a proxy for whether a run will land in a flat, generalizing region.

What would settle it

Re-run the headline Granite 7B stacked comparison—batch sizes 128, 3,840, and 7,680 at learning rate 2e-5—with at least ten seeds and report per-seed MMLU and MTBench. If the seed-to-seed spread (say ±0.01 MMLU, ±0.3 MTBench) overlaps the claimed gaps, the ordering and the early-dynamics correlation are not established.

Watch

Extended reading notes

Core claim

The paper's discovery is that, for instruction-tuning small LLMs, the usual cautious recipe is wrong in both directions: you want a much larger batch and a lower learning rate than the defaults promoted by TULU. On the Granite 7B model in stacked training, raising the effective batch from 128 to 3,840 to 7,680 samples improved MMLU from 0.516 to 0.526 to 0.529 and MTBench from 6.406 to 6.768 to 6.831, with larger batches needing more samples to peak but reaching higher final scores. Lower learning rates (2e-5 for Granite, 1e-6 for Mistral) consistently beat higher ones, and this did not interact with batch size as some scaling heuristics predict. The paper further claims that omitting warmup and using a constant learning rate rather than cosine decay does not hurt performance, that stacked training is at least as good as sequential phased training while using fewer samples, and that a run's eventual quality can be read from early training dynamics: the best final models show lower gradient norms and higher training loss from the start. The same pattern held when they retested on the TULU dataset, on a math–reasoning–code dataset, and on Granite 3B, LLaMA 3B, and Mistral 7B.

Load-bearing premise

Every ranking in this study comes from a single run per configuration, so the claimed orderings—including differences as small as 0.01 MMLU points—depend on run-to-run noise being smaller than those margins.

Editorial extensions

If this is right

  • Practitioners can drop warmup and cosine decay: a constant learning rate matches or beats cosine decay on MMLU and MTBench, simplifying the training loop.
  • Larger effective batches (4k–8k) are worth the extra samples when compute allows; when it does not, a 4k batch gives most of the gain and peaks earlier.
  • Stacked training can replace phased training, eliminating phase-checkpoint selection while matching or slightly exceeding its scores and using fewer samples.
  • Gradient accumulation on a single node reproduces multi-node full-batch training, so the large-batch recipe does not require a cluster.
  • Early gradient-norm and loss curves can serve as a cheap early-stopping signal, saving the compute that would be spent on sub-optimal runs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the early-dynamics correlation is causal, the same low-gradnorm/high-loss heuristic should transfer to larger instruction-tuned models and to parameter-efficient methods such as LoRA; the paper only tests dense 3B–7B models, so this is a testable extension.
  • The seemingly paradoxical 'higher training loss is better' result suggests the benchmarks reward staying close to the pretrained weights rather than minimizing SFT loss; a testable consequence is that early-dynamics-selected checkpoints should also do better on held-out instruction-following evaluations, not just MMLU and MTBench.
  • Because most experiments use taxonomy-generated synthetic instruction data, the recipe's advantage over small batches may shrink on human-written instruction data with different noise structure; re-running the batch-size sweep on such data would show whether the guidance is universal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports an empirical study of supervised fine-tuning for small (3B-7B) language models on instruction-following, knowledge, and skills data. The authors compare stacked versus phased training, sweep batch sizes (128/4K/8K), learning rates, warmup steps, and learning-rate schedules, and compare the TULU and LAB hyperparameter configurations across Granite, LLaMA, and Mistral models on benchmarks including MMLU, MTBench, and Open LLM Leaderboard v2. They claim that (i) larger batch sizes paired with lower learning rates improve downstream performance, (ii) early training dynamics such as lower gradient norms and higher training loss predict final performance, (iii) warmup steps and cosine decay are unnecessary, and (iv) stacked training matches phased training while being simpler and more sample efficient.

Significance. If the claims held, the paper would provide a valuable, actionable recipe for resource-constrained practitioners and would challenge widely cited defaults such as TULU's hyperparameters and Orca-style phased training. The study is unusually thorough in documenting hyperparameter sweeps, including negative results, infrastructure details, and cross-model checks on Granite, LLaMA, and Mistral. However, the central causal attributions are currently not established: the batch-size comparison is confounded with total sample count, the TULU-vs-LAB comparison varies multiple factors simultaneously, and all runs are single-seed with no significance testing. The early-dynamics 'predictor' is a post-hoc description of the same sweep rather than an out-of-sample prediction. The paper has clear practical value as a documentation resource, but the headline empirical claims need stronger experimental support.

major comments (4)
  1. [§3.2, Table 3] The batch-size comparison holds the number of gradient steps approximately constant, but the number of samples consumed differs by a factor of four: the stacked MMLU runs use 2,099,328 (batch 128), 3,694,080 (4K), and 8,885,760 (8K) samples. The claim that 'larger batch sizes lead to better final performance' is therefore confounded with total training data and effective number of epochs; a same-sample or same-epoch control is needed before attributing the gains to batch size. Appendix A.5.2 shows smaller batches reaching higher scores early and plateauing, which is the expected signature of training on fewer samples, not decisive evidence of a batch-size effect.
  2. [§3.4, Table 5 and Table 1] The TULU-vs-LAB comparison changes batch size, learning rate, warmup, scheduling, and number of epochs simultaneously (Table 1: batch 128 versus 3,840/7,680; LR 2e-5 versus 3e-5; warmup ratio 0.03 versus 0.01; linear decay versus constant; 3 versus 10 epochs). The paper states that 'LAB outperforms TULU across all benchmarks,' but this is a composite comparison and cannot be attributed to batch size. The cross-dataset experiment in Table 5 also varies the full configuration, not just batch size, so the claim that larger batch sizes generalize across datasets is not isolated.
  3. [§3.7, Figures 1-2, Appendix A.5.9] The early-training-dynamics predictor is identified post hoc from the same runs used to select the best hyperparameters. The pattern 'lower gradient norms and higher loss' is correlated with larger batch size and lower learning rate in the sweep, so it has not been shown to predict performance on unseen configurations. To support the claimed early-termination capability, the authors should demonstrate that this criterion predicts performance on held-out hyperparameter settings or on a second model/dataset not used in the analysis.
  4. [§3.1, §3.2, and Limitations] The abstract and Section 3.1 report 'no significant difference' between phased and stacked training and rank configurations with differences as small as 0.01 on MMLU and MTBench (e.g., Table 3: stacked MMLU 0.516 vs 0.526; MTBench 6.406 vs 6.768). All reported runs appear to be single-seed, and no variance estimates or significance tests are provided. The Limitations section acknowledges that single-seed runs 'may introduce some noise into the observations,' but if run-to-run noise is comparable to these margins, the ordering of batch sizes, learning rates, and training strategies is not statistically established.
minor comments (5)
  1. [Abstract and §4] The abstract claims the findings hold 'robustly across datasets as well as model families and sizes,' but the main text reports single-seed runs and the Limitations acknowledge potential noise; please temper this claim or provide replication statistics.
  2. [§3.2] 'Approximately the same number of gradient steps' is not quantified; please state the exact step counts for the runs in Table 3 so readers can verify the intended comparison.
  3. [Figure 2 and surrounding text] The score annotations in Figure 2 appear garbled ('6.7566.6446.747'); please fix the formatting and ensure the match to the learning-rate sweep results.
  4. [Appendix A.5.7, Table 10] In Table 10, the MATH Lvl 5 score for LAB is 0.04, identical to TULU, but the text says LAB 'outperforms TULU across all evaluation metrics'; please reconcile this statement with the reported numbers.
  5. [§2.3 and Appendix A.5.1] The medium batch size is referred to as both '3,840' and '4,000' in different places; please standardize the notation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claims are empirical comparisons against external benchmarks, and the self-cited LAB dataset/configurations are experimental inputs rather than derived conclusions.

full rationale

The paper's main claims—larger batch sizes with lower learning rates improve downstream performance, warmup and cosine decay are unnecessary, and stacked training matches phased training—are direct empirical comparisons across multiple models and external benchmarks (MMLU, MTBench, Open LLM Leaderboard). None of these findings is derived from its own inputs by definition: batch size and learning rate are varied independently of the outcome metric, and the sample-count metric is an accounting definition, not a target being predicted. The self-citations to the authors' prior LAB work provide the dataset and a hyperparameter configuration used as experimental inputs; they do not themselves assert or prove the paper's conclusions, and the TULU comparison uses an external dataset and benchmark suite. The early-training-dynamics 'predictor' is a post-hoc correlation observed on the same runs used to rank configurations, which limits its predictive strength and is appropriately flagged as a methodological limitation by the single-seed acknowledgment; however, the correlation direction is not entailed by the definitions of gradient norm or loss, so this is not a circular derivation. No load-bearing step reduces to its own inputs, and no fitted parameter is renamed as a prediction. Therefore the circularity score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper's recommendations rest on hand-selected hyperparameters, single-seed benchmark comparisons, and the authors' own LAB dataset/config; no new physical entities are introduced.

free parameters (6)
  • Learning rate for Granite/Llama = 2e-5
    Selected as best in a sweep from 2e-5 to 1e-4 on Granite 7B; the paper recommends this value and the conclusions depend on it.
  • Learning rate for Mistral = 1e-6
    Best among 1e-6, 5e-6, and 2e-5 for Mistral 7B; presented as architecture-specific guidance.
  • Effective batch size = 3,840 or 7,680 samples
    The 'large batch' settings are hand-picked based on the LAB config and 64-GPU infrastructure; findings apply to these values.
  • Number of epochs = 3 (TULU) vs 10 (LAB)
    Epoch count differs between the compared configurations, confounding the attribution of gains to batch size.
  • Warmup steps = 25
    LAB config uses 25 warmup steps; the finding that warmup is unnecessary is tested around this value.
  • Adam beta2 = 0.95
    Chosen to reduce emphasis on past gradient variance for large batches; all runs use this optimizer setting.
assumptions (5)
  • domain assumption Benchmark scores on MMLU, MTBench, and Open LLM Leaderboard v2 are valid proxies for fine-tuning quality.
    Section 2.4 defines these as the evaluation metrics; all conclusions are drawn from them.
  • domain assumption Single-seed training is representative enough to rank configurations.
    Limitations: 'experiments were conducted using a single seed due to computational constraints.' Without this, the small benchmark differences cannot be interpreted.
  • domain assumption Synthetic datasets generated via a taxonomy and teacher models are representative of real instruction-tuning data.
    Appendix A.2; the main testbed is the authors' own LAB dataset, so conclusions are conditional on this data type.
  • domain assumption Gradient accumulation exactly replicates multi-node full-batch training.
    Appendix A.5.10 reports equivalence on one setup; this underpins all large-batch experiments across hardware configurations.
  • ad hoc to paper Differences between TULU and LAB configurations can be attributed to batch size.
    Section 3.4 compares TULU (batch 128, 3 epochs, linear decay) with LAB (batch 4k, 10 epochs, constant LR) without isolating variables.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs." pith.science (2026). https://pith.science/paper/3UIWKHMN

@misc{pith2026241213337,
  author       = {Pith},
  title        = {Pith review of: Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3UIWKHMN}},
  note         = {Machine review of arXiv:2412.13337}
}
read the original abstract

The rise of large language models (LLMs) has created a significant disparity: industrial research labs with their computational resources, expert teams, and advanced infrastructures, can effectively fine-tune LLMs, while individual developers and small organizations face barriers due to limited resources. In this paper, we aim to bridge this gap by presenting a comprehensive study on supervised fine-tuning of LLMs using instruction-tuning datasets spanning diverse knowledge domains and skills. We focus on small-sized LLMs (3B to 7B parameters) for their cost-efficiency and accessibility. We explore various training configurations and strategies across four open-source pre-trained models. We provide detailed documentation of these configurations, revealing findings that challenge several common training practices, including hyperparameter recommendations from TULU and phased training recommended by Orca. Key insights from our work include: (i) larger batch sizes paired with lower learning rates lead to improved model performance on benchmarks such as MMLU, MTBench, and Open LLM Leaderboard; (ii) early-stage training dynamics, such as lower gradient norms and higher loss values, are strong indicators of better final model performance, enabling early termination of sub-optimal runs and significant computational savings; (iii) through a thorough exploration of hyperparameters like warmup steps and learning rate schedules, we provide guidance for practitioners and find that certain simplifications do not compromise performance; and (iv) we observed no significant difference in performance between phased and stacked training strategies, but stacked training is simpler and more sample efficient. With these findings holding robustly across datasets and models, we hope this study serves as a guide for practitioners fine-tuning small LLMs and promotes a more inclusive environment for LLM research.

Figures

Figures reproduced from arXiv: 2412.13337 by the authors.

Figure 1
Figure 1. Correlation between early training dynamics and final performance on MMLU and MT [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. LAB Learning Rate (LR) Sweep: Training Dynamics and MTBench Performance. MMLU [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Comparison of stacked and phased training strategies on MTBench using LAB hyperpa [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Final MMLU Performance comparison using LAB hyperparameters: stacked vs. phased [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]
Figure 5
Figure 5. Figure 5: MMLU Sample efficiency comparison between stacked and phased training. [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]
Figure 6
Figure 6. Figure 6: Performance comparison of stacked vs. phased training on difficulty-partitioned data (by [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Impact of batch size on model performance in stacked and phased training on MMLU [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Impact of batch size on model performance in stacked and phased training on MTBench [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: Comparison of learning rate schedules with a batch size of 3,840 samples on MMLU and [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: Comparison of TULU vs. LAB on memorization and generalization. [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: MTBench performance after Phase 10 training with different learning rates. [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Performance comparison across different learning rates and batch sizes on MMLU and [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: MMLU and MTBench performance of the Granite 7B LAB model with varying warmup [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: Benchmark performance comparison of different batch sizes for the Mistral 7B model. [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]
Figure 15
Figure 15. Figure 15: Training dynamics for Mistral 7B with different learning rates, and their final perfor [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]
Figure 16
Figure 16. Figure 16: Training dynamics for Granite 3B with different batch sizes and training strategies (8k [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]
Figure 17
Figure 17. Figure 17: LAB Learning Rate Sweep: Impact on Training Dynamics (Grad Norm, Loss) and Final [PITH_FULL_IMAGE:figures/full_fig_p030_17.png]
Figure 18
Figure 18. Figure 18: Effect of Batch Size (4k vs 8k) on Training Dynamics and Final Performance on MMLU [PITH_FULL_IMAGE:figures/full_fig_p031_18.png]
Figure 19
Figure 19. Figure 19: Effect of Constant Learning Rate vs. Cosine Decay Across Different Learning Rates on [PITH_FULL_IMAGE:figures/full_fig_p032_19.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Probabilistic Concept-Aware Steering for Trustworthy LLM Inference

    cs.AI 2026-05 reject novelty 4.0 of 10

    PCS improves steering direction accuracy by adaptively sampling the intervention coefficient from a cosine-similarity-conditioned Gaussian, but its evaluation is partly circular because the optimal coefficient is chos...

  2. How Reliable are LLMs for Reasoning on the Re-ranking task?

    cs.CL 2025-08 reject novelty 4.0 of 10

    In a small Earth-science reranking dataset, DPO-trained LLMs rank best and SHAP attribution scores help a general LLM explain why items were selected, but the explanation claim rests on only two examples.

  3. Filter-And-Refine: A MLLM Based Cascade System for Industrial-Scale Video Content Moderation

    cs.LG 2025-07 conditional novelty 4.0 of 10

    A cascade of an embedding router and a fine-tuned multimodal LLM ranker is claimed to improve content moderation F1 by 66.5% while using 1.5% of the compute of direct LLM deployment.

  4. Too Big to Think: Capacity, Memorization, and Generalization in Pre-Trained Transformers

    cs.LG 2025-06 conditional novelty 4.0 of 10

    Capacity-limited Transformers generalize on held-out single-digit arithmetic while larger models memorize facts; joint training suppresses extrapolation in all tested sizes.

Reference graph

Works this paper leans on

58 extracted references · 9 canonical work pages · cited by 4 Pith papers

  1. [1]

    Physics of language models: Part 3.1, knowledge storage and extraction

    Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint arXiv:2309.14316,

    arXiv

  2. [2]

    arXiv preprint arXiv:2311.10702,

    arXiv

  3. [3]

    In- struction pre-training: Language models are supervised multitask learners

    Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. In- struction pre-training: Language models are supervised multitask learners. arXiv preprint arXiv:2406.14491,

    arXiv

  4. [5]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    arXiv

  5. [6]

    Both performed similarly, with stacked training slightly outperforming phased training across all bench- marks

    showed no significant difference between phased and stacked training in this setting. Both performed similarly, with stacked training slightly outperforming phased training across all bench- marks. This suggests that even when the data is carefully partitioned based on difficulty, phased training does not improve model performance over stacked training. M...

    work page arXiv

  6. [7]

    Phase Description # Samples Phase 00 Instruction following warmup: simple, template-based instruction-response pairs to transition the base models to instruction-following behavior

    Table 7: Summary of datasets used in different phases. Phase Description # Samples Phase 00 Instruction following warmup: simple, template-based instruction-response pairs to transition the base models to instruction-following behavior. 308343 Phase 05 Foundational knowledge acquisition: synthetically gen- erated question-answer pairs from textbooks cover...

    work page 2024

  7. [10]

    Apple intelligence foundation language models

    Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang, Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Cheng Chiu, David Qiu, et al. Apple intelligence foundation language models. arXiv preprint arXiv:2407.21075,

    arXiv

  8. [11]

    Scaling laws and compute-optimal training beyond fixed training durations

    12 Alexander H¨agele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro V on Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. arXiv preprint arXiv:2405.18392,

    arXiv

Show all 58 references
  1. [12]

    Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detec- tion

    Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detec- tion. arXiv preprint arXiv:2203.09509,

    arXiv

  2. [13]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300,

    2009 arXiv

  3. [14]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zhen Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chaochao Jia, Guoyang Zeng, Daha...

    arXiv

  4. [15]

    Adam Ibrahim, Benjamin Th ´erien, Kshitij Gupta, Mats L Richter, Quentin Anthony, Timoth ´ee Lesort, Eugene Belilovsky, and Irina Rish

    URL https: //api.semanticscholar.org/CorpusID:269009975. Adam Ibrahim, Benjamin Th ´erien, Kshitij Gupta, Mats L Richter, Quentin Anthony, Timoth ´ee Lesort, Eugene Belilovsky, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. arXiv...

    arXiv

  5. [16]

    This diversity reduces gradient variance, promoting stable updates and helping the model retain pre-trained knowledge without significant forgetting

    The larger batch size likely improves performance by increasing data diversity within each batch, covering a range of tasks, skills, and knowledge. This diversity reduces gradient variance, promoting stable updates and helping the model retain pre-trained knowledge without sig...

    2023

  6. [18]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  7. [19]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088,

    arXiv

  8. [20]

    Fantastic generalization measures and where to find them

    Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint arXiv:1912.02178,

    1912 arXiv

  9. [23]

    Timoth´ee Lesort, Massimo Caccia, and Irina Rish

    URL https: //www.salesforce.com/blog/small-language-models/. Timoth´ee Lesort, Massimo Caccia, and Irina Rish. Understanding continual learning settings with data distribution drift analysis. ArXiv, abs/2104.01678,

    arXiv

  10. [25]

    From quantity to quality: Boosting llm performance with self- guided data selection for instruction tuning

    Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting llm performance with self- guided data selection for instruction tuning. ArXiv, abs/2308.12032,

    arXiv

  11. [26]

    Stephanie Lin, Jacob Hilton, and Owain Evans

    URL https: //api.semanticscholar.org/CorpusID:261076515. Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958,

    arXiv

  12. [27]

    What makes good data for alignment? a comprehensive study of automatic data selection in instruction tun- ing

    Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tun- ing. ArXiv, abs/2312.15685,

    arXiv

  13. [28]

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al

    URL https://api.semanticscholar.org/ CorpusID:266551413. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprin...

    arXiv

  14. [29]

    Gran- ite code models: A family of open foundation models for code intelligence

    Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza So- ria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, et al. Gran- ite code models: A family of open foundation models for code intelligence. arXiv preprint arXiv:24...

    arXiv

  15. [30]

    Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, Dany Rouhana, Andres Co- das, Yadong Lu, Wei-ge Chen, Olga Vrousgos, Corby Rosset, et al

    URL https://arxiv.org/abs/2311.11045. Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, Dany Rouhana, Andres Co- das, Yadong Lu, Wei-ge Chen, Olga Vrousgos, Corby Rosset, et al. Agentinstruct: Toward gen- erative teaching with agentic flows. arXiv preprint arXiv...

    arXiv

  16. [31]

    Orca: Progressive learning from complex explanation traces of gpt-4

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Hassan Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. ArXiv, abs/2306.02707,

    arXiv

  17. [32]

    ISBN 9781450384421

    Associa- tion for Computing Machinery. ISBN 9781450384421. doi: 10.1145/3458817.3476209. URL https://doi.org/10.1145/3458817.3476209. 14 OpenAI. Chatgpt: Optimizing language models for dialogue. OpenAI Blog,

  18. [33]

    URL https: //openai.com/blog/chatgpt/. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  19. [34]

    Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke E

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, J...

    arXiv

  20. [35]

    Wei Pang, Chuan Zhou, Xiao-Hua Zhou, and Xiaojie Wang

    URL https://api.semanticscholar.org/ CorpusID:246426909. Wei Pang, Chuan Zhou, Xiao-Hua Zhou, and Xiaojie Wang. Phased instruction fine-tuning for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Ling...

    2024

  21. [36]

    doi: 10.18653/v1/2024

    Association for Computational Linguistics. doi: 10.18653/v1/2024. findings-acl.341. URL https://aclanthology.org/2024.findings-acl.341. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. ArXiv, abs/2304.03277,

    2024 arXiv

  22. [37]

    Jonathan S

    URL https://www.redhat.com/en/topics/ai/ llm-vs-slm. Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. ArXiv, abs/1909.12673,

    1909 arXiv

  23. [38]

    semanticscholar.org/CorpusID:203592013

    URL https://api. semanticscholar.org/CorpusID:203592013. Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, An- toine Chung, Yacine Jernite, Jared Launay, Shaden Bari, et al. Multitask prompted training en- ables zero-shot task generaliz...

    arXiv

  24. [39]

    Teven Le Scao et al

    URL https://openreview.net/forum?id=0bmXrtTDUu. Teven Le Scao et al. Bloom: A 176b-parameter open-access multilingual language model. ArXiv, abs/2301.16790,

  25. [40]

    SL Smith

    URL https://api.semanticscholar.org/CorpusID:252815378. SL Smith. Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489,

    arXiv

  26. [43]

    Challenging big-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Sch¨arli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261,

    arXiv

  27. [44]

    Llama 2: Open founda- tion and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288,

    arXiv

  28. [45]

    Code alpaca: Code instruction data from community contributions

    Haonan Wang et al. Code alpaca: Code instruction data from community contributions. ArXiv, abs/2303.08388, 2023a. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-g...

    arXiv

  29. [46]

    How far can camels go? exploring the state of instruction tuning on open resources

    Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing...

    arXiv

  30. [47]

    Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652,

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652,

    arXiv

  31. [48]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei et al. Chain-of-thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903,

    arXiv

  32. [49]

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang

    URL https://api.semanticscholar.org/CorpusID:265213147. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244,

    arXiv

  33. [50]

    Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer

    Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ry- der, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466,

    arXiv

  34. [51]

    Synthetic continued pretraining

    Zitong Yang, Neil Band, Shuangping Li, Emmanuel Cand `es, and Tatsunori Hashimoto. Synthetic continued pretraining. arXiv preprint arXiv:2409.07431,

    arXiv

  35. [52]

    Lima: Less is more for alignment

    Ming Zhou et al. Lima: Less is more for alignment. ArXiv, abs/2305.11206,

    arXiv

  36. [55]

    from the pre-trained parameters, potentially overshooting the minima. 25 A.5.6 E FFECT OF WARMUP STEPS Figure 13 shows the performance comparison with different warmup steps: 0, 25, and 100 warmup steps, on the MMLU and MTBench benchmarks, respectively. The model trained witho...

  37. [56]

    0.02 0.04 0.04 9,980,259 3,468,664 Leaderboard (MuSR) 0.05 0.08 0.04 16,966,128 2,973,753 ARC 0.78 0.75 0.68 2,745,290 247,372 GSM8K 0.27 0.69 0.66 12,225,143 5,450,009 26 Additionally, we fine-tuned the LLaMA 3B model using both the stacked and sequential phased training stra...

    2024

  38. [58]

    0.01 0.02 0.01 250,089 741,924 Leaderboard (MuSR) 0.05 0.22 0.15 10,979,309 1,731,217 MMLU 0.56 0.57 0.55 6,986,437 2,473,477 ARC 0.78 0.78 0.74 2,744,559 2,473,477 GSM8K 0.27 0.51 0.49 3,742,399 2,473,477 MTBench - 5.00 4.97 9,232,227 2,473,477 The results, depicted in Table ...

  39. [1966]

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C´esar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al

    doi: 10.1002/j.1538-7305.1966.tb01709.x. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C´esar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644,

    1966

  40. [2014]

    Lab: Large-scale alignment for chatbots

    Shivchander Sudalairaj, Abhishek Bhandwaldar, Aldo Pareja, Kai Xu, David D Cox, and Akash Srivastava. Lab: Large-scale alignment for chatbots. arXiv preprint arXiv:2403.01081,

    arXiv

  41. [2015]

    Scaling laws for downstream task performance of large language models

    Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance of large language models. InICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models,

    2024

  42. [2016]

    Accurate, large minibatch sgd: Training imagenet in 1 hour

    Priya Goyal, Piotr Doll´ar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677,

    arXiv

  43. [2017]

    Dropout: a simple way to prevent neural networks from overfitting

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958,

    1929

  44. [2018]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

  45. [2019]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv, abs/2001.08361,

    2001 arXiv

  46. [2020]

    13 Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang

    URL https://api.semanticscholar.org/ CorpusID:210861095. 13 Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836,

    arXiv

  47. [2021]

    semanticscholar.org/CorpusID:233024916

    URL https://api. semanticscholar.org/CorpusID:233024916. Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al. Synthetic data (almost) from scratch: Generalized instruction tun...

    arXiv

  48. [2022]

    Enhancing chat language models by scaling high-quality instructional conversations

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Conference on Empirical Methods in Natural Language Processing , 2023a. URL htt...

    arXiv

  49. [2023]

    Deepseek llm: Scaling open-source language models with longtermism

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954,

    arXiv

  50. [2024]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, et al. Scaling instruction-finetuned language models. ArXiv, abs/2210.11416,

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.