{"id":"b45758dc-4fcb-44a4-b1f0-f45ea30f619d","arxiv_id":"2412.11231","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"In three instruction-evolution scenarios, 7B-8B models produced instruction data that led to better downstream fine-tuning performance than data from 70B+ models.","lead":"This paper tests whether small language models (around 8 billion parameters) generate better instruction-tuning data than 70-billion-parameter models, and finds they often do. The result could lower the cost of building fine-tuning datasets and change how synthetic instruction data is produced.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Scaling experiment's full-tuning/LoRA split confounds size with tuning method; evidence for a parametrization-independent 'smaller is better' claim is not yet established.","rationale":"The reader's weakest_assumption identifies exactly the same confound: model size is not cleanly separated from other differences. The scaling experiment is the single place where the paper tries to vary size systematically, and Section A.1/Table 7 reveal that tuning method and learning rate change at the same point as size (full FT for 0.5B-7B, LoRA for 14B-72B). This is not an internal inconsistency, but it is a genuine correctness risk for the title-level claim 'smaller models are better,' which is stated parametrization-free. The paper deserves credit for consistent results across Evol-Instruct, AutoIF, and Auto Evol-Instruct with two model families, and the IC-IFD contribution is separable, but the causal-load-bearing part of the title is the weak link. Hence CONDITIONAL rather than REJECT: a matched-tuning scaling run could easily save the claim, and the current evidence is suggestive but not sufficient. My concrete test is a matched-method, multi-seed scaling experiment; if the inversion between 7B and 14B is explained away by the LoRA/full-FT switch, the paper should soften the title and claim to 'the specific 7-8B models we tested,' which is a meaningful but weaker statement.","tokens_in":23815,"tokens_out":1519,"duration_ms":12890,"concrete_test":"Re-run the Qwen-2.5 scaling experiment using a single, consistent adaptation method across all sizes (e.g., LoRA at rank 8/alpha 8 for every model, holding LR, epochs, and data volume constant), with at least 3 seeds per size. If the SLM over LLM gap disappears or the 7B-to-14B inversion vanishes under matched tuning, the size-based conclusion fails; if the inversion persists, the size-attribution concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that model size, not other properties, causes the instruction-evolution advantage. The scaling experiment (Figure 3, Table 11) is the only direct evidence for a size trend, but it is confounded: models 0.5B-7B are full fine-tuned while 14B-72B use LoRA with a different learning rate (5e-5 vs 1e-5/7e-6, Table 7), so the visible inversion between 7B and 14B could reflect that the larger models were adapted with a lower-capacity method and a higher LR, not their size. The paper also never reports variance or multiple seeds anywhere, so the 0.5B/1.5B and 7B/14B differences in the scaling curves may be within run-to-run noise. Because every other comparison (Tables 1-4) contrasts only one 7-8B model against one 70B+ model, it cannot separate the size variable from pretraining data, alignment procedure, or the fixed prompt method described in A.1; the title-size attribution rests almost entirely on the confounded scaling run.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether smaller language models (SLMs, 7-8B) synthesize more effective instruction-tuning data than larger models (LLMs, 70B+) under three instruction-evolution pipelines: Evol-Instruct, AutoIF, and Auto Evol-Instruct. Using Llama-3.1 and Qwen-2 families as instruction evolvers and a range of backbone models as downstream fine-tuning targets, the authors report that SLM-evolved instructions generally produce better or comparable downstream performance than LLM-evolved instructions. They connect this to a broader output space of SLMs via lower top-1 token probabilities, and they introduce an instruction-complexity-penalized version of the IFD score (IC-IFD) as a training-free way to evaluate instruction data. The paper includes a scaling study across Qwen-2.5 downstream models from 0.5B to 72B.","tokens_in":24077,"tokens_out":11858,"duration_ms":97047,"significance":"If the main finding is robust, the paper is practically significant: it would imply that instruction-evolution pipelines can be run with much smaller and cheaper models without sacrificing data quality, and it would challenge the common practice of delegating instruction synthesis to frontier or 70B+ models. The study has genuine strengths: it evaluates multiple downstream backbones, uses two model families for the main Evol-Instruct comparison, fixes the evolution prompt sequence to reduce confounds within a pair, holds the response generator constant, and releases code. The token-probability analysis is a plausible and testable mechanism. However, the causal attribution to 'size' is currently not established as cleanly as the title suggests, and the IC-IFD validation is partially circular. With matched training setups, additional evolver-size points, and repeated runs, the evidence could be made convincing.","major_comments":[{"comment":"The scaling experiment varies the size of the downstream fine-tuned model while keeping the instruction-evolution models fixed (Llama-3.1-8B vs Llama-3.1-70B), and the downstream training setup is confounded: 0.5B-7B use full fine-tuning (LR 1e-5 or 7e-6) while 14B-72B use LoRA (LR 5e-5). The step from 7B to 14B in Figure 3 could reflect the change in tuning method and learning rate rather than model size. As a consequence, the statement that the findings 'hold across models of different sizes' is not yet established. Please provide a matched training setup and seed variance for at least the 7B vs 14B comparison.","section":"§3.1, 'Scaling Experiments', Figure 3, Table 7, Table 11"},{"comment":"The direct evidence for the title claim is a single 7-8B vs 70B+ comparison in each of two model families (plus single-family AutoIF and Auto Evol-Instruct comparisons). Because the compared supervisors differ not only in size but also in pretraining data, alignment procedure, and possibly instruction-following strength, the pairwise results do not establish a general size law; they establish that the two specific SLMs are more effective than their two specific larger counterparts in these three scenarios. The scaling experiment in Figure 3 does not vary the evolver size, so it cannot fill this gap. A size sweep of evolution models within one family, with fixed prompt and data construction, would let the title claim be made without overreach.","section":"§3.1 and Tables 1-4"},{"comment":"IC-IFD is designed from the observed performance degradation on Alpaca-iter3 (Section 3.1, Figure 1) and then validated by filtering exactly that dataset (Table 5). This is a post hoc evaluation on the same data that motivated the metric. The AlpacaFarm comparisons in Appendix A.4 (Figures 6-7) are a step in the right direction, but they use the original Alpaca data rather than a different evolution round or dataset. Please validate IC-IFD on held-out instruction sets (e.g., GSM8K or Code Alpaca iter-3, or LLM-evolved iter-3) to support the claim that it 'more accurately' assesses instruction data.","section":"§5, 'Performance of IC-IFD', Table 5, and Limitation (3)"},{"comment":"No repeated runs or significance tests are reported anywhere, and not every cell favors SLM-INST: for example, in Table 2 the Llama-3-8B model reaches 63.76 on GSM8K with Qwen-2-72B-evolved instructions versus 63.91 with Qwen-2-7B-evolved instructions. Many other differences are small enough that they could be within run-to-run noise. The headline conclusion relies on point estimates. Please report at least three independent seeds with means and standard deviations, or paired significance tests, for the main SLM-vs-LLM comparisons and for the scaling curves.","section":"Throughout §3, especially Table 2"}],"minor_comments":[{"comment":"The heading reads 'Why do SLMs Outerperform LLMs' and the word should be 'Outperform'.","section":"Section 1, RQ2 heading"},{"comment":"The denominator introduces LΘ(Q) but the text describes it as 'perplexity'; please clarify whether LΘ(Q) is the average cross-entropy loss and how it is normalized by sequence length, since average cross-entropy loss is not perplexity unless exponentiated.","section":"Equation (2)"},{"comment":"The caption and legend should explicitly state that the x-axis is the size of the fine-tuned downstream model, and that the two curves correspond to instructions evolved by the Llama-3.1-8B and Llama-3.1-70B supervisors; the full-fine-tuning vs LoRA split should be marked in the figure.","section":"Figure 3"},{"comment":"The learning-rate rows do not indicate which training regime they belong to; given the full fine-tuning/LoRA split, please add a column or note so the reader can see which learning rate applies to each size and method.","section":"Table 7"},{"comment":"The statement that SLMs generate 6.9% more trajectory keywords is used to support Finding 3, but the underlying counts are not reported in any table or appendix; please include them.","section":"Section 3.3"},{"comment":"The token-probability analysis is only reported for one model pair (Llama-3.1-8B vs Llama-3.1-70B) and one scenario; please state whether the distribution difference is statistically significant and whether it is also observed in the Qwen family.","section":"Section 4 and Figure 5"},{"comment":"AutoIF and Auto Evol-Instruct report only the Llama-3.1 family; the abstract's claim of consistency across scenarios would benefit from an explicit statement that these two scenarios are currently single-family demonstrations.","section":"Sections 3.2 and 3.3"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper is within scope and likely to attract attention. In my view, the current evidence supports a weaker statement than the title: specific 7-8B models can outperform specific 70B+ models as instruction evolvers in the tested pipelines, but the general 'smaller is better' size-scaling claim is not yet established. The missing size sweep of instruction-evolver models and the confounded downstream scaling study should be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful finding here is real: across Evol-Instruct, AutoIF, and Auto Evol-Instruct, instruction data evolved by 7-8B models consistently fine-tunes backbones as well as or better than data from 70B+ models. That is new and worth knowing even if the mechanism is wrong. The paper is also honest in its limitations and releases code, which helps.\n\nWhat is actually new: this is the first systematic SLM-vs-LLM comparison for instruction evolution, not response generation. It uses two model families (Llama-3.1 and Qwen-2) and three scenarios, with evaluation grounded in external benchmarks. The output-space analysis (lower top-1 token probabilities for SLMs) is a plausible and testable hypothesis, and the IC-IFD metric is a reasonable attempt to fix a known blind spot in IFD. The temperature robustness check in the appendix is a nice touch.\n\nThe soft spots are real but not fatal. The scaling experiment (Figure 3, Table 11) is the only direct evidence for a size trend, and it is confounded: models 0.5-7B get full fine-tuning, while 14-72B get LoRA with a different learning rate. So the visible inversion around 7B/14B could be a tuning-method artifact, not a size effect. There are also no repeated runs or variances anywhere, so some reported differences (0.5B/1.5B exceptions, 7B vs 14B gaps) may be within run-to-run noise. The IC-IFD was designed after observing the iter3 degradation and validated on that same dataset; the Appendix tests on Alpaca are broader, but the core validation still has that post hoc feel. The output-space explanation is correlational and tested on only one SLM/LLM pair, though the main result does hold across two families.\n\nThe central empirical claim—SLM-evolved data is not worse, often better, than LLM-evolved data in these settings—holds up as an empirical statement. The stronger claim that smaller models are better because of their size is not established. The title overclaims, and the reader's stress-test note is on target.\n\nWho is this for? Anyone building synthetic instruction data pipelines or studying data quality. It is a legitimate subfield contribution, not a field-reshaping result. I would send it to peer review, with a request for repeated seeds, error bars, and a cleaner scaling comparison (e.g., LoRA for all sizes, or at least a control for tuning method). I would also ask for IC-IFD validation on a dataset it was not designed on. A serious referee can extract a good paper from this version.","headline":"SLM-evolved instructions consistently beat LLM-evolved ones across several setups, but the paper's size-based explanation is not clean and the scaling evidence is confounded.","tokens_in":24559,"tokens_out":1926,"would_cite":true,"duration_ms":18184,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Smaller 7B–8B models evolve instruction-tuning data that outperforms data evolved by 70B+ models across three pipelines.","keywords":["instruction evolution","instruction tuning","data synthesis","model scaling","output diversity","language models","data selection"],"falsifier":"Run the three evolution scenarios with a matched pair of models that differ only in scale (same architecture, tokenizer, pretraining corpus, and alignment recipe), then fine-tune the same backbones on both instruction sets; if the larger model's data matches or beats the smaller model's data, the size-based claim fails. A second check: if the top-1 token probability distributions are equalized (e.g., by temperature scaling), the performance gap should disappear if the output-space explanation is the active cause.","tokens_in":23661,"feed_emoji":"🧠","tokens_out":4184,"duration_ms":34857,"temperature":0.7,"pith_summary":"The paper challenges the working assumption that bigger models are better at rewriting and generating training instructions. Across Evol-Instruct, AutoIF, and Auto Evol-Instruct, instruction data evolved by 7B–8B models outperformed data evolved by 70B+ models when either set was used to fine-tune the same backbone models. The authors attribute the advantage to a broader output space: smaller models assign lower probability to their top token, producing more complex and diverse instructions. If correct, this would make instruction-data synthesis cheaper and shift the scaling logic of data generation away from the largest available models.","feed_headline":"Smaller models beat 70B models at instruction evolution","feed_subtitle":"Data evolved by 7B–8B models fine-tunes better on math, code, and instruction following, plus a metric to pick hard data.","key_machinery":"The central mechanism is the output space of the instruction-evolving model, measured by the distribution of top-1 token probabilities during generation: lower top-1 probability means a broader, less overconfident output distribution, which yields more complex and diverse instructions. The supporting object is the Instruction Complex-Aware IFD (IC-IFD) score, defined as $$\\text{IC-IFD}_\\Theta(Q,A) = \\frac{\\mathcal{L}_\\Theta(A|Q)}{\\mathcal{L}_\\Theta(Q)\\cdot\\mathcal{L}_\\Theta(A)},$$ which introduces instruction perplexity as a penalty into the original IFD score so that extremely hard instructions do not receive inflated quality estimates.","core_discovery":"The central claim is that, for the purpose of evolving instruction-tuning data, smaller language models are more effective than their larger counterparts. In three instruction-evolution scenarios, datasets produced by 7B–8B models consistently led to better fine-tuned performance on instruction following, math reasoning, and code generation than datasets produced by 70B+ models from the same model families. The paper explains this through the distribution of top-1 token probabilities: the larger models are more confident in each generated token, narrowing their output space, whereas smaller models sample from a wider distribution and therefore generate instructions that are more complex and diverse. The paper also introduces IC-IFD, a data-quality score that penalizes the original IFD score by instruction complexity, and shows it predicts downstream fine-tuning gains better when selecting subsets of evolved instructions.","pith_inferences":["A testable extension beyond the paper: match the output entropy or sampling temperature of the 70B model to that of the 8B model; if the performance gap closes, the output-space explanation, rather than model size per se, is the active variable.","The result suggests that instruction evolution is a regime where teaching ability does not monotonically track the teacher's own capability, analogous to findings for response generation, so data-synthesis pipelines should benchmark the data product rather than the generator's benchmark scores.","The IC-IFD penalty could plausibly be applied beyond Evol-Instruct-style pipelines, for example to filter any instruction dataset where difficulty is skewed, though the paper only demonstrates it on Alpaca and evolved Alpaca data."],"forward_implications":["Data evolved by 7B–8B models can replace data evolved by 70B+ models without a loss in fine-tuned performance, cutting the compute and API cost of instruction data construction.","Evolving with smaller models can reach the same or better instruction complexity in fewer iterations than evolving with larger models.","High-complexity instructions are not always better: after two rounds of SLM evolution, a third round can overshoot and degrade performance, which IC-IFD flags by penalizing instruction difficulty.","IC-IFD can be used to filter instruction data without running instruction tuning, outperforming instruction length, instruction perplexity, and the original IFD at the tested selection ratios."],"supporting_citations":[{"why":"introduces Evol-Instruct, the base pipeline the paper adapts for the first scenario","marker":"Xu et al., 2024a"},{"why":"provides AutoIF, the second comparison scenario with verification functions","marker":"Dong et al., 2024"},{"why":"defines Auto Evol-Instruct, the third scenario where models design their own evolution trajectories","marker":"Zeng et al., 2024"},{"why":"supplies the IFD score that IC-IFD modifies with an instruction-complexity penalty","marker":"Li et al., 2024"},{"why":"establishes the seed-instruction expansion paradigm used across the evolution scenarios","marker":"Wang et al., 2023"},{"why":"documents the Larger Models' Paradox for response generation that this paper extends to instruction evolution","marker":"Xu et al., 2024c"}],"fun_headline_variants":["Smaller models beat 70B+ in instruction evolution","7B-8B models outperform 70B+ for evolving instruction data","For instruction evolution, smaller LLMs are more effective","Why 8B models produce more diverse instruction variants than 70B","IC-IFD: a better metric for selecting evolved instruction data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that smaller size causes better instruction evolution assumes the measured advantage comes from model size rather than from other differences between the compared models, such as pretraining data, alignment recipe, or fine-tuning method.","fun_headline_variants_meta":{"raw":{"variants":["Smaller models beat 70B+ in instruction evolution","7B-8B models outperform 70B+ for evolving instruction data","For instruction evolution, smaller LLMs are more effective","Why 8B models produce more diverse instruction variants than 70B","IC-IFD: a better metric for selecting evolved instruction data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000326,"raw_usage":{"total_tokens":1820,"prompt_tokens":934,"completion_tokens":886,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":797}},"tokens_in":550,"tokens_out":886,"duration_ms":8145,"temperature":1.0,"reasoning_tokens":797,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:08:59.812039+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the three evolution scenarios with a matched pair of models that differ only in scale (same architecture, tokenizer, pretraining corpus, and alignment recipe), then fine-tune the same backbones on both instruction sets; if the larger model's data matches or beats the smaller model's data, the size-based claim fails. A second check: if the top-1 token probability distributions are equalized (e.g., by temperature scaling), the performance gap should disappear if the output-space explanation is the active cause.","supporting_citations":[],"review_version":1}