{"id":"36d27fd7-aa32-40f8-912f-4e7e85a10a65","arxiv_id":"2412.13337","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"For 3B-7B LLMs, larger batch sizes with lower learning rates improve instruction-tuning benchmarks, early gradient and loss signals predict final quality, and stacked training matches phased training with fewer samples.","lead":"This paper is a large empirical study of how to fine-tune small open-source language models (3B to 7B parameters) for instruction following. It reports that larger batch sizes with lower learning rates improve benchmark scores, that early training signals predict final quality, and that simpler stacked training is as good as multi-phase training.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline batch-size finding is confounded: Section 3.2 compares batch sizes at equal gradient steps, so 4K/8K runs consume 2–4x more samples than 128-batch runs, and single-seed margins of 0.01–0.02 could be noise.","rationale":"The reader's weakest assumption is single-seed noise, which is valid and explicitly acknowledged. I identify a more specific, load-bearing confound that operates even before considering seed noise: the batch-size comparison fixes gradient steps, so larger batches see much more training data. This directly affects the first pillar of the recipe ('large effective batch') and also contaminates the early-dynamics predictor, since the dynamics are measured on the same runs. The concern is not that the paper is dishonest; it is transparent about the sample-efficiency tradeoff and reports many configurations across four models and multiple datasets. Rather, the causal attribution 'larger batch sizes lead to better final performance' is not supported by the experimental design as reported. A same-sample, multi-seed control would settle the question. Because the reader already recommended CONDITIONAL and my concern reinforces that condition rather than overturning the paper, the verdict should remain UNCHANGED.","tokens_in":30334,"tokens_out":6226,"duration_ms":61318,"concrete_test":"Train Granite 7B stacked with LAB hyperparameters at batch sizes 128, 3,840, and 7,680, matching total samples (e.g., 3.7M and 8.9M) by scaling gradient-steps inversely with batch size, with 3 seeds per cell. If the 8K/4K advantage over 128 disappears or falls within seed noise at fixed sample budget, batch size per se is not the cause; the reported benefit is a longer-training/data-volume artifact. Also report seed-level standard deviations for MMLU and MTBench.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 reports that larger batch sizes improve final MMLU/MTBench, but the experiments hold gradient steps approximately constant rather than holding total samples constant. Table 3 shows the stacked MMLU runs use 2,099,328 (128), 3,694,080 (4K), and 8,885,760 (8K) samples; MTBench uses 1,799,424 vs 4,392,960 vs 8,586,240. The 8K model therefore sees roughly 4x as much training data as the 128 model and 2.4x as much as the 4K model. The gains claimed for batch size could be a data-volume or epoch effect. Appendix A.5.2 shows smaller batches reach higher scores early and plateau, with larger batches overtaking only after many more samples, which is exactly the signature of training longer on more data. The paper acknowledges larger batches need more samples, but the recipe still asserts batch size as the cause; without a same-sample control the causal attribution is unsupported. This is compounded by the single-seed design acknowledged in the Limitations: differences as small as MMLU 0.516 vs 0.526 and MTBench 6.406 vs 6.768 may be within run-to-run noise. The early-dynamics predictor inherits the same problem: 'lower gradient norms and higher loss' is observed on the same runs and is correlated with lower LR/larger batch, so it has not been shown to predict across unseen configurations.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical study of supervised fine-tuning for small (3B-7B) language models on instruction-following, knowledge, and skills data. The authors compare stacked versus phased training, sweep batch sizes (128/4K/8K), learning rates, warmup steps, and learning-rate schedules, and compare the TULU and LAB hyperparameter configurations across Granite, LLaMA, and Mistral models on benchmarks including MMLU, MTBench, and Open LLM Leaderboard v2. They claim that (i) larger batch sizes paired with lower learning rates improve downstream performance, (ii) early training dynamics such as lower gradient norms and higher training loss predict final performance, (iii) warmup steps and cosine decay are unnecessary, and (iv) stacked training matches phased training while being simpler and more sample efficient.","tokens_in":30698,"tokens_out":4381,"duration_ms":37092,"significance":"If the claims held, the paper would provide a valuable, actionable recipe for resource-constrained practitioners and would challenge widely cited defaults such as TULU's hyperparameters and Orca-style phased training. The study is unusually thorough in documenting hyperparameter sweeps, including negative results, infrastructure details, and cross-model checks on Granite, LLaMA, and Mistral. However, the central causal attributions are currently not established: the batch-size comparison is confounded with total sample count, the TULU-vs-LAB comparison varies multiple factors simultaneously, and all runs are single-seed with no significance testing. The early-dynamics 'predictor' is a post-hoc description of the same sweep rather than an out-of-sample prediction. The paper has clear practical value as a documentation resource, but the headline empirical claims need stronger experimental support.","major_comments":[{"comment":"The batch-size comparison holds the number of gradient steps approximately constant, but the number of samples consumed differs by a factor of four: the stacked MMLU runs use 2,099,328 (batch 128), 3,694,080 (4K), and 8,885,760 (8K) samples. The claim that 'larger batch sizes lead to better final performance' is therefore confounded with total training data and effective number of epochs; a same-sample or same-epoch control is needed before attributing the gains to batch size. Appendix A.5.2 shows smaller batches reaching higher scores early and plateauing, which is the expected signature of training on fewer samples, not decisive evidence of a batch-size effect.","section":"§3.2, Table 3"},{"comment":"The TULU-vs-LAB comparison changes batch size, learning rate, warmup, scheduling, and number of epochs simultaneously (Table 1: batch 128 versus 3,840/7,680; LR 2e-5 versus 3e-5; warmup ratio 0.03 versus 0.01; linear decay versus constant; 3 versus 10 epochs). The paper states that 'LAB outperforms TULU across all benchmarks,' but this is a composite comparison and cannot be attributed to batch size. The cross-dataset experiment in Table 5 also varies the full configuration, not just batch size, so the claim that larger batch sizes generalize across datasets is not isolated.","section":"§3.4, Table 5 and Table 1"},{"comment":"The early-training-dynamics predictor is identified post hoc from the same runs used to select the best hyperparameters. The pattern 'lower gradient norms and higher loss' is correlated with larger batch size and lower learning rate in the sweep, so it has not been shown to predict performance on unseen configurations. To support the claimed early-termination capability, the authors should demonstrate that this criterion predicts performance on held-out hyperparameter settings or on a second model/dataset not used in the analysis.","section":"§3.7, Figures 1-2, Appendix A.5.9"},{"comment":"The abstract and Section 3.1 report 'no significant difference' between phased and stacked training and rank configurations with differences as small as 0.01 on MMLU and MTBench (e.g., Table 3: stacked MMLU 0.516 vs 0.526; MTBench 6.406 vs 6.768). All reported runs appear to be single-seed, and no variance estimates or significance tests are provided. The Limitations section acknowledges that single-seed runs 'may introduce some noise into the observations,' but if run-to-run noise is comparable to these margins, the ordering of batch sizes, learning rates, and training strategies is not statistically established.","section":"§3.1, §3.2, and Limitations"}],"minor_comments":[{"comment":"The abstract claims the findings hold 'robustly across datasets as well as model families and sizes,' but the main text reports single-seed runs and the Limitations acknowledge potential noise; please temper this claim or provide replication statistics.","section":"Abstract and §4"},{"comment":"'Approximately the same number of gradient steps' is not quantified; please state the exact step counts for the runs in Table 3 so readers can verify the intended comparison.","section":"§3.2"},{"comment":"The score annotations in Figure 2 appear garbled ('6.7566.6446.747'); please fix the formatting and ensure the match to the learning-rate sweep results.","section":"Figure 2 and surrounding text"},{"comment":"In Table 10, the MATH Lvl 5 score for LAB is 0.04, identical to TULU, but the text says LAB 'outperforms TULU across all evaluation metrics'; please reconcile this statement with the reported numbers.","section":"Appendix A.5.7, Table 10"},{"comment":"The medium batch size is referred to as both '3,840' and '4,000' in different places; please standardize the notation.","section":"§2.3 and Appendix A.5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a serious empirical contribution with strong documentation value, but the main causal claims are not yet supported because of the sample-count confound in the batch-size experiment, the multi-factor confound in the TULU-vs-LAB comparison, and the lack of repeated seeds or significance testing. These issues are fixable with additional experiments or a substantially more cautious framing, so I recommend major revision rather than rejection. If the authors can add a same-sample or same-epoch control and report variance estimates (even on a subset of configurations), the paper would be much stronger."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This one is worth reading but with a skeptical eye. The paper is a systematic measurement campaign on supervised fine-tuning of 3B-7B models, and the practical payoff is real: they document that you can drop warmup, use a constant LR, and stack all data instead of phasing, without hurting performance. Those results come from a wide set of models and datasets, and the paper is admirably upfront about what they did and did not test. If you do SFT of small models, the config tables alone are a useful starting point.\n\nThe soft spots are in the headline claims. The biggest issue is the batch-size result. The paper says they ran experiments for approximately the same number of gradient steps, but Table 3 itself shows the 8K run consumed 8.9M samples versus 2.1M for the 128-batch run, and the 4K run consumed 3.7M. So the larger-batch runs also saw two to four times more data. The curves in Appendix A.5.2 show small batches plateau early and large batches overtake only later, which is exactly the signature of training on more data, not a batch-size effect. Without a same-sample control (or at least an iso-compute comparison), the causal claim is unsupported.\n\nThe TULU vs LAB comparison is also confounded: batch size, learning rate, warmup, schedule, and number of epochs all change at once. The 'no significant difference' between stacked and phased is asserted without any significance testing, and some of the reported margins (0.516 vs 0.526 MMLU) are close enough that single-seed noise could flip the ordering. The Limitations section concedes the single-seed issue.\n\nThe early-dynamics predictor is the weakest section. 'Lower gradient norms and higher loss means better final performance' is a post-hoc description of the runs that produced the best configurations. They never show it predicting performance on a held-out configuration, so the early-stopping advice is plausible but unvalidated. Note the stress-test's claim about 'equal gradient steps' is wrong—the paper's own table contradicts it—but the underlying confound is real and actually worse than the stress-test states.\n\nAll that said, the paper is honest, internally consistent, and the negative results (warmup not needed, constant LR fine) are worth having. It is the kind of study that should get refereed, but a careful referee needs to demand multi-seed runs, unconfounded comparisons, and a proper test of the early-dynamics rule before the headline claims are accepted. For a practitioner who wants a starting point for SFT configs, this is a useful guide; for a researcher who wants causal evidence, it is a list of hypotheses to test properly.","headline":"Useful SFT recipe map for small LLMs, but the headline batch-size effect is confounded with data volume and the early-stopping predictor is post-hoc.","tokens_in":31263,"tokens_out":4591,"would_cite":true,"duration_ms":39113,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bigger batches and lower learning rates win for small-LLM tuning.","keywords":["supervised fine-tuning","small language models","instruction tuning","batch size","learning rate","training dynamics","gradient norm","MMLU"],"falsifier":"Re-run the headline Granite 7B stacked comparison—batch sizes 128, 3,840, and 7,680 at learning rate 2e-5—with at least ten seeds and report per-seed MMLU and MTBench. If the seed-to-seed spread (say ±0.01 MMLU, ±0.3 MTBench) overlaps the claimed gaps, the ordering and the early-dynamics correlation are not established.","tokens_in":30165,"feed_emoji":"🧠","tokens_out":8923,"duration_ms":77105,"temperature":0.7,"pith_summary":"This paper tries to turn supervised fine-tuning of small language models (3B–7B parameters) from folklore defaults into an evidence-based recipe. It systematically varies batch size, learning rate, warmup steps, schedule, and training strategy across four open base models and several instruction datasets, evaluating on MMLU, MTBench, and the Open LLM Leaderboard v2. The central claim: larger effective batches (roughly 4,000–8,000 samples) paired with lower learning rates beat the widely used TULU settings; warmup and cosine decay can be dropped; and stacked training matches or beats phased training while being more sample efficient. It also claims that early training dynamics—low gradient norm and high training loss—predict final benchmark scores well enough to stop unpromising runs early. If right, this gives researchers with modest compute a simpler, cheaper fine-tuning pipeline and a way to know early whether a run will pay off.","feed_headline":"Bigger batches, lower learning rates win for small-LLM tuning","feed_subtitle":"An empirical guide finds simple settings—stacked data, no warmup, constant LR—beat the old defaults on 3B–7B models.","key_machinery":"The load-bearing instrument is the controlled sweep: one factor at a time (batch size, learning rate, warmup steps, schedule, training strategy) is varied while everything else is held fixed, under three named configurations—LAB, TULU, and the paper's TULU++—and evaluated on MMLU and MTBench, with Leaderboard v2, ARC, and GSM8K as secondary checks. The mechanism they invoke for the batch-size effect is gradient-noise reduction: averaging over more samples cuts gradient variance roughly as $1/\\sqrt{n}$, producing steadier updates that keep the model near its pretrained parameters. The early-dynamics predictor is the other central object: plotted curves of log-transformed gradient norm and total training loss over samples, which the paper reads as a proxy for whether a run will land in a flat, generalizing region.","core_discovery":"The paper's discovery is that, for instruction-tuning small LLMs, the usual cautious recipe is wrong in both directions: you want a much larger batch and a lower learning rate than the defaults promoted by TULU. On the Granite 7B model in stacked training, raising the effective batch from 128 to 3,840 to 7,680 samples improved MMLU from 0.516 to 0.526 to 0.529 and MTBench from 6.406 to 6.768 to 6.831, with larger batches needing more samples to peak but reaching higher final scores. Lower learning rates (2e-5 for Granite, 1e-6 for Mistral) consistently beat higher ones, and this did not interact with batch size as some scaling heuristics predict. The paper further claims that omitting warmup and using a constant learning rate rather than cosine decay does not hurt performance, that stacked training is at least as good as sequential phased training while using fewer samples, and that a run's eventual quality can be read from early training dynamics: the best final models show lower gradient norms and higher training loss from the start. The same pattern held when they retested on the TULU dataset, on a math–reasoning–code dataset, and on Granite 3B, LLaMA 3B, and Mistral 7B.","pith_inferences":["If the early-dynamics correlation is causal, the same low-gradnorm/high-loss heuristic should transfer to larger instruction-tuned models and to parameter-efficient methods such as LoRA; the paper only tests dense 3B–7B models, so this is a testable extension.","The seemingly paradoxical 'higher training loss is better' result suggests the benchmarks reward staying close to the pretrained weights rather than minimizing SFT loss; a testable consequence is that early-dynamics-selected checkpoints should also do better on held-out instruction-following evaluations, not just MMLU and MTBench.","Because most experiments use taxonomy-generated synthetic instruction data, the recipe's advantage over small batches may shrink on human-written instruction data with different noise structure; re-running the batch-size sweep on such data would show whether the guidance is universal."],"forward_implications":["Practitioners can drop warmup and cosine decay: a constant learning rate matches or beats cosine decay on MMLU and MTBench, simplifying the training loop.","Larger effective batches (4k–8k) are worth the extra samples when compute allows; when it does not, a 4k batch gives most of the gain and peaks earlier.","Stacked training can replace phased training, eliminating phase-checkpoint selection while matching or slightly exceeding its scores and using fewer samples.","Gradient accumulation on a single node reproduces multi-node full-batch training, so the large-batch recipe does not require a cluster.","Early gradient-norm and loss curves can serve as a cheap early-stopping signal, saving the compute that would be spent on sub-optimal runs."],"supporting_citations":[{"why":"Defines the LAB taxonomy-driven instruction datasets and phased tuning framework whose hyperparameters serve as the paper's baseline configuration.","marker":"(Sudalairaj et al., 2024)"},{"why":"Provides the TULU recipe and mixture v2 dataset that the paper directly challenges, supplying the batch-128/warmup/decay baseline and the cross-dataset test.","marker":"(Wang et al., 2023b; Ivison et al., 2023)"},{"why":"Orca's phased progressive learning is the training strategy the paper compares stacked training against and finds unnecessary.","marker":"(Mitra et al., 2023)"},{"why":"MMLU is the primary knowledge benchmark used to measure final performance in every sweep.","marker":"(Hendrycks et al., 2020)"},{"why":"MTBench is the primary conversational and skill generalization benchmark used in every sweep.","marker":"(Zheng et al., 2023)"},{"why":"Supplies the mechanism the paper invokes: large batches keep weights closer to initialization, reducing forgetting.","marker":"(Hoffer et al., 2017)"},{"why":"Documents the earlier large-batch generalization gap that the paper's opposite finding is explicitly contrasted with.","marker":"(Keskar et al., 2016)"},{"why":"Origin of the warmup practice that the paper tests and finds unnecessary for fine-tuning small LLMs.","marker":"(Goyal et al., 2017)"}],"fun_headline_variants":["Bigger batches, lower LR: the real recipe for small LLMs","Small LLM tuning: stacked data, no warmup, constant LR works","Predict final LLM quality from early loss—cut bad runs short","TULU and Orca got it wrong: small LLM tuning revised"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every ranking in this study comes from a single run per configuration, so the claimed orderings—including differences as small as 0.01 MMLU points—depend on run-to-run noise being smaller than those margins.","fun_headline_variants_meta":{"raw":{"variants":["Bigger batches, lower LR: the real recipe for small LLMs","Small LLM tuning: stacked data, no warmup, constant LR works","Predict final LLM quality from early loss—cut bad runs short","TULU and Orca got it wrong: small LLM tuning revised"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000271,"raw_usage":{"total_tokens":1733,"prompt_tokens":1151,"completion_tokens":582,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":767,"completion_tokens_details":{"reasoning_tokens":514}},"tokens_in":767,"tokens_out":582,"duration_ms":6106,"temperature":1.0,"reasoning_tokens":514,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:14:35.399546+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the headline Granite 7B stacked comparison—batch sizes 128, 3,840, and 7,680 at learning rate 2e-5—with at least ten seeds and report per-seed MMLU and MTBench. If the seed-to-seed spread (say ±0.01 MMLU, ±0.3 MTBench) overlaps the claimed gaps, the ordering and the early-dynamics correlation are not established.","supporting_citations":[],"review_version":1}