{"id":"391864ea-26a1-4140-a76f-c9598fca8371","arxiv_id":"2506.21545","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Ordering training data by a gradient-based score, using a folding scheme that interleaves multiple curriculum passes, improves small-scale LM accuracy by roughly 1.5 to 2 points on average benchmarks.","lead":"This paper proposes DELT, a framework for improving language model training by scoring, selecting, and ordering training data, and introduces two components: LQS scoring and Folding ordering. The authors report consistent but small accuracy gains, about 1.5 to 2 points on average, across model sizes from 160M to 1B and token budgets from 1B to 50B, without adding data or compute.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposed-method gains are reported without variance estimates while only the baseline is averaged over three seeds; the 'consistently outperforms' claim may be seed noise.","rationale":"The paper's intended contribution is an empirical one: ordering training data with the DELT pipeline improves final LM accuracy at fixed data scale and model size. The most load-bearing requirement is therefore that the reported improvements are real and not artifacts of run-to-run variance. The text only averages the conventional baseline over three seeds and gives no variance information for the proposed method, so Table 1a's +1.7-point gain is the key quantity to establish. I focused on this rather than the proxy-transfer assumption in the reader's weakest_assumption because even a perfectly transferable scoring method would not support the headline claim if the measured gains are within noise; conversely, if the gains survive a multi-seed test, the proxy-transfer question becomes a secondary mechanistic concern. The appendix's derivation of LQS also has an algebraic problem (Eq. 12–14 multiply the summed Learnability and Quality scores but then collapse the product into a single sum, which is not valid for T>2), and the reliability weight R cancels exactly; these weaken the paper's explanation of LQS but do not by themselves falsify the empirical ordering effect. I credit the paper for releasing code, reporting multi-model and multi-scale experiments, and providing a concrete paradigm, but those strengths do not substitute for a significance check on the central claim.","tokens_in":23517,"tokens_out":10398,"duration_ms":116366,"concrete_test":"Run the main 160M/1B-token configuration with at least five independent random seeds for both Conventional random ordering and LQS+Folding (same scores, same L=3, same training hyperparameters). Report per-seed 8-benchmark averages and the DELT minus Conventional difference with a paired or two-sample 95% confidence interval, for example via bootstrap. If the interval includes 0, the central claim should be weakened to preliminary evidence; if the interval excludes 0 by a margin at least as large as the reported +1.7 points, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that the LQS+Folding pipeline improves performance over conventional random ordering without changing data scale or model size—is not yet supported by the reported numbers, because the proposed pipeline is presented without any variance estimate. The Table 1 caption states that \"the conventional method presents the average result over three random seeds,\" but says nothing similar for \"Ours\"; the DELT rows appear to be single runs. The headline gain is about +1.7 average points at 160M/1B tokens (Table 1a: 36.37 to 38.08), and several individual benchmark moves are small or negative (e.g., Wino 50.72 to 53.04 at 160M but 52.07 to 51.28 at 1B; ARC-e 41.88 to 41.07 at 50B in Table 9b). Training a 160M–1B model on 1B tokens with different random seeds can easily produce run-to-run variation of this magnitude on eight small multiple-choice benchmarks. Because only the conventional baseline is averaged, the comparison is asymmetric: a lucky single seed for DELT would be compared against a stabilized baseline, inflating the apparent effect. Without standard deviations, confidence intervals, or a paired test across seeds, the consistently-outperforms statement is indistinguishable from seed noise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines \"data efficacy\" as improving language model performance by reorganizing training data without changing dataset content or model architecture, and proposes a DELT paradigm with three components: data scoring, optional data selection, and data ordering. It introduces Learnability-Quality Scoring (LQS), which combines per-sample gradient-magnitude ratios and gradient alignment with a target vector, and Folding Ordering (FO), which interleaves curriculum-sorted data over L folds. The authors report pre-training experiments on Mistral-style models from 160M to 1B parameters on RedPajama subsets, post-training experiments on OpenWebMath and The-Stack-v2, and evaluations on OLMo, code, and math benchmarks. They claim DELT consistently outperforms baselines, and that data efficacy and data efficiency are compatible.","tokens_in":23785,"tokens_out":4276,"duration_ms":50116,"significance":"If the reported effect is real, the claim that reordering existing training data alone improves downstream LM performance across model sizes and data scales is significant and practically valuable, since it is almost cost-free relative to data collection or architecture changes. The paper also proposes a general paradigm that could unify curriculum learning, annealing, and data selection. Strengths include public code, a broad evaluation suite, multiple model sizes and data scales, and the use of held-out evaluation benchmarks distinct from the reference set used for scoring. The significance is conditional, however, on resolving the variance and hyperparameter-selection issues detailed below; the current evidence does not yet support the central \"consistently outperforms\" claim.","major_comments":[{"comment":"The proposed DELT rows are reported without any variance estimate, while only the conventional baseline is averaged over three random seeds. For example, at 160M/1B tokens the average gain is about 1.7 points (36.37 to 38.08), and individual benchmarks move in opposite directions across settings (e.g., Wino improves from 50.72 to 53.04 at 160M but declines from 52.07 to 51.28 at 1B; ARC-e declines from 41.88 to 41.07 at 50B in Table 9b). Run-to-run variation at this scale on small multiple-choice benchmarks can easily be of this magnitude. Because the baseline is stabilized by averaging but the proposed method is not, the comparison is asymmetric and the \"consistently outperforms\" statement is not distinguishable from seed noise. The authors should report means and standard deviations over at least three seeds for the proposed pipeline, or provide confidence intervals and paired tests.","section":"§5.2, Table 1 caption and Table 9"},{"comment":"The ordering direction (ascending vs. descending) and the folding layer L are selected using the same OLMo evaluation benchmarks on which final results are reported. Table 4 shows that ascending sorting helps while descending sorting hurts, and Figure 7/Table 12 show that L=3 is chosen because it peaks on those benchmarks. With gains of roughly 1-2 average points, selecting these free hyperparameters on the test benchmarks can inflate the apparent effect. The paper should either prespecify these choices, select them on a held-out validation split, or demonstrate that the results are stable across a range of L and both directions.","section":"§5.3, Table 4 and Figure 7"},{"comment":"The reliability score R(θ_{t+1}) = ||λ_{t+1}|| is introduced as a weighting that reflects model capability, but it cancels exactly in the derivation: Eq. (12) simplifies to Eq. (14), which is the same as Eq. (9) in the main text. Consequently, the stated motivation that \"a stronger model will assign more weight to the scores\" is not implemented by the final scoring function. This is not a numerical error, but it is a mismatch between the narrative and the method. Either R should be retained as an explicit weighting factor, or the claim that reliability is part of LQS should be removed.","section":"Appendix D, Eqs. (11)-(14)"},{"comment":"The entire scoring pipeline rests on a 125M proxy model and a tiny reference set (1,030 LIMA examples for general data), and the resulting scores are used to order training for models up to 1B parameters. The paper does not provide any sensitivity analysis with respect to proxy model size, reference set size, or training trajectory length T. Since the central claim is that gradient-geometry scores transfer across model scales and downstream tasks, the authors should at least report an ablation varying proxy capacity or reference data to show that the ordering benefit is not an artifact of a particular proxy choice.","section":"§5.1 and Appendix E"}],"minor_comments":[{"comment":"The sentence \"This area complements data efficacy\" appears to be a typo; it should read \"data efficiency,\" since the surrounding text contrasts with data efficiency.","section":"§1, Introduction"},{"comment":"The phrase \"which is three times the random average results\" is unclear; it likely means the conventional baseline is averaged over three random seeds. Please rephrase.","section":"Table 12 caption"},{"comment":"The rows for \"LQS - Folding\" and \"LQS ✓ Folding\" are numerically identical across all eight benchmarks. If the selection row reports the best result over selection ratios and that best occurs at r=1.0 (i.e., no actual selection), this should be stated explicitly; otherwise the duplicate rows look like an error.","section":"Table 2"},{"comment":"Figure 5 is never referenced in the main text and no numeric values or selection ratios are given in the caption. Please add a pointer in the text and summarize the shown ratios.","section":"Figure 5"},{"comment":"The target-vector recursion involves a Hessian-vector product ∇²L(θ_t, γ)λ_{t+1}; the paper does not describe how these products are computed or approximated in practice, nor how the computational cost compares with PDS [10].","section":"Eq. (8)"}],"recommendation":"major_revision","confidential_remarks":"The topic is timely and the paradigm framing is useful, but the empirical foundation needs strengthening before publication. If the authors cannot provide multi-seed variance estimates for the proposed pipeline or a validation-based selection of L and ordering direction, the \"consistently outperforms\" claim should be softened substantially. The identical LQS/Folding rows in Table 2 should also be checked for a possible reporting artifact."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper's most defensible new result is Folding Ordering—a deterministic interleaving of a curriculum sort—which beats both random shuffle and plain sorting across every scorer they test (KenLM, PDS, and their own LQS). That pattern is visible in Table 2 and it is not circular: Folding helps even with the baseline scorers. The second thing: the paper's headline claim, that DELT \"consistently outperforms\" across models and data sizes, is not yet fully supported by the numbers, because the conventional baseline is averaged over three seeds while the DELT rows appear to be single runs. On eight small multiple-choice benchmarks with 160M-1B models, that difference can easily be within run-to-run noise. This is the paper's main soft spot, and it is fixable with seed averaging or error bars on every row.\n\nWhat is genuinely good: the paradigm itself is clean and useful for thinking about data organization, the code is public, and the experimental sweep is wide—multiple model sizes, data scales, math/code post-training, and multi-epoch training. The limitations section is candid about the need for a small high-quality reference set. LQS is, however, a modest variant of the authors' earlier PDS method: the scoring denominator moves from ∇ℓ(x_n, θ_t) to ∇ℓ(x_n, θ_{t+1}), which changes the weighting but not the core idea. Also, in the appendix the \"reliability score\" R(θ_{t+1}) is introduced as motivation, but it cancels algebraically in Equation 13, so the stated motivation does not actually influence the score. A careful reader will also note that the folding layer L and the ordering direction are selected on the same evaluation benchmarks, which slightly inflates the apparent advantage.\n\nFor whom: anyone doing data-centric LM training or curriculum learning will get value from the Folding result and the paradigm. I would send it to a serious referee, but with the clear expectation that the authors add seed-level variance estimates and either pre-register or hold out their hyperparameter choices. The central claim that data ordering matters is probably true—the effects are consistent across many independent conditions—but the magnitude is not yet nailed down.","headline":"Folding ordering is a solid, non-circular contribution, but the paper's headline gains are undercut by single-run DELT numbers against a seed-averaged baseline.","tokens_in":24345,"tokens_out":4119,"would_cite":true,"duration_ms":45252,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reordering a fixed training corpus by gradient-derived scores improves language model accuracy with no added data or parameters.","keywords":["data efficacy","data ordering","data scoring","data selection","learnability-quality scoring","folding ordering","curriculum learning","language model pretraining"],"falsifier":"Train one fixed corpus and model with LQS+Folding and with random shuffling, repeated over at least three seeds; the claim predicts a clear average gain on the OLMo benchmarks. The claim is falsified if the gain disappears within seed noise, or if the score ranking computed by the 125M proxy model fails to transfer when the same ordering is applied to a 1B target model in a controlled head-to-head.","tokens_in":23328,"feed_emoji":"🔀","tokens_out":8715,"duration_ms":93876,"temperature":0.7,"pith_summary":"The paper defines 'data efficacy' as improving language models by changing how fixed training data is organized, rather than by choosing which data to keep or adding more data. It proposes a three-component pipeline it calls DELT — data scoring, optional data selection, and data ordering — and designs two new pieces: Learnability-Quality Scoring (LQS), which rates each sample by the way its gradient changes across training steps, and Folding Ordering (FO), which interleaves a score-sorted list in several passes. On the OLMo benchmark suite, the LQS+FO combination beats random shuffling, plain curriculum sorting, and the KenLM and PDS baselines across models from 160M to 1B parameters and training sets from 1B to 50B tokens, and it also improves math and code post-training. The same pipeline applied on top of data selection improves both accuracy and data efficiency, showing that ordering and selection are compatible rather than competing.","feed_headline":"Reordering training data lifts LM scores with no extra tokens","feed_subtitle":"A three-stage scoring-and-folding pipeline beats random shuffling across model sizes and data scales.","key_machinery":"The machinery that carries the argument is the LQS score of Equation 9: $$\\gamma_n = \\sum_{t=1}^{T-1} \\frac{\\lambda_{t+1}^\\top \\nabla \\ell(x_n, \\theta_t)}{\\|\\nabla \\ell(x_n, \\theta_{t+1})\\|},$$ where $\\lambda_t$ is the target gradient vector of Equation 8, obtained by propagating the downstream loss $J(\\theta)$ backward through the training trajectory. The numerator charges a sample for aligning with the direction that reduces the downstream loss; the denominator charges it for remaining hard, so high-scoring samples are both high-quality and highly learnable. Because computing this on the full corpus is expensive, the authors annotate a small proxy subset with these scores, train a small scorer to predict them, and then score the whole corpus. Folding Ordering then converts the score vector into a training order: sort ascending and take every $L$-th element in $L$ consecutive passes, with $L=3$ chosen as the default, so the model gets a repeated easy-to-hard progression without duplicating data.","core_discovery":"On its own terms, the paper's claim is that the presentation order of training data is a first-class source of language model performance. Reordering the same corpus — no samples added, no parameters changed — yields measurable gains on downstream reasoning benchmarks, with the largest gains coming from combining a gradient-consistency scorer with a multi-pass interleaved ordering. The authors argue that pure score-sorted curriculum learning suffers from forgetting and distribution bias because the model sees all easy examples first and all hard examples last, and their folding schedule answers that by cycling through the sorted list, so every part of the score range appears in each pass. They also claim this ordering benefit survives when data selection is applied, meaning a user can simultaneously train on fewer tokens and get better results.","pith_inferences":["Inference: because the score is computed from gradients of a small proxy model, the transfer claim could be tested directly by re-running the ordering with the target model's own gradients; if the gap widens, the scoring step is even more useful than the paper shows.","Inference: the reference-set choice (LIMA for general data) is likely a sensitive knob; swapping the 1,030-example reference set for another small set of equal quality would show whether ordering gains are robust or reference-specific.","Inference: folding layer count $L$ is empirically peaked at 3 for these sizes, but the mechanism suggests $L$ should roughly track the number of distinct difficulty levels in the corpus; an adaptive $L$ could be the natural scaling extension to terabyte-scale pretraining.","Inference: the same learnability signal that orders data could be reused mid-run to identify samples that have already been mastered, connecting data efficacy to dynamic truncation or on-the-fly data selection during training."],"forward_implications":["Any existing pretraining run can be improved by reordering its own data; the only extra cost is a one-time scoring pass with a small model.","The ordering gain does not wash out as the model or corpus grows: the paper reports the effect at 160M, 470M, and 1B parameters and at 1B, 10B, and 50B tokens.","Ordering and selection are compatible: putting DELT on top of KenLM, PDS, or the proposed scorer improves both accuracy and the fraction of the corpus that must be kept.","The method transfers to domain-specific continued training, with gains reported on math and code benchmarks when ordering is applied after pretraining.","The default of $L=3$ folding layers outperforms $L=1$ (pure curriculum) on the reported benchmarks, so even the ordering component alone is an improvement over sorting."],"supporting_citations":[{"why":"Defines the optimal-control-style gradient reference direction that LQS extends, and serves as the main data-selection baseline.","marker":"[10]"},{"why":"Provides the KenLM perplexity scorer used as a baseline data-scoring method and as an input to DELT comparisons.","marker":"[19]"},{"why":"Establishes curriculum learning, the baseline ascending-sort ordering that Folding Ordering builds on and compares against.","marker":"[11]"},{"why":"Supplies the small high-quality reference set (LIMA) whose downstream loss drives the LQS gradient scores.","marker":"[39]"},{"why":"Provides the RedPajama pretraining corpus used as the general-domain training data in the main experiments.","marker":"[28]"},{"why":"Defines the OLMo benchmark suite used to evaluate the pretrained models in the main results.","marker":"[46]"}],"fun_headline_variants":["Reorder training data for free LM performance gains","Data ordering matters: folding beats random shuffling","No new tokens, just better order: LM gains from reordering","Folding scheduling lifts LM scores without extra data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the gradient geometry of a small 125M-parameter proxy model, measured against a small curated reference set, predicts which ordering helps much larger models on unrelated benchmarks; if that transfer fails, the scores carry no ordering information.","fun_headline_variants_meta":{"raw":{"variants":["Reorder training data for free LM performance gains","Data ordering matters: folding beats random shuffling","No new tokens, just better order: LM gains from reordering","Folding scheduling lifts LM scores without extra data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000151,"raw_usage":{"total_tokens":1203,"prompt_tokens":953,"completion_tokens":250,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":188}},"tokens_in":569,"tokens_out":250,"duration_ms":3545,"temperature":1.0,"reasoning_tokens":188,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:23:08.197079+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train one fixed corpus and model with LQS+Folding and with random shuffling, repeated over at least three seeds; the claim predicts a clear average gain on the OLMo benchmarks. The claim is falsified if the gain disappears within seed noise, or if the score ranking computed by the 125M proxy model fails to transfer when the same ordering is applied to a 1B target model in a controlled head-to-head.","supporting_citations":[{"cited_title":"KenLM: Faster and smaller language model queries","cited_arxiv_id":null,"evidence_quote":"Provides the KenLM perplexity scorer used as a baseline data-scoring method and as an input to DELT comparisons."},{"cited_title":"Lima: Less is more for alignment","cited_arxiv_id":null,"evidence_quote":"Supplies the small high-quality reference set (LIMA) whose downstream loss drives the LQS gradient scores."}],"review_version":1}