{"id":"78039639-4e52-430f-9e0e-2679c2677a21","arxiv_id":"2504.20644","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DiSF selects LLM pre-training files that are maximally decorrelated in a pretrained text embedding space, improving downstream accuracy while using only 1.5% of SlimPajama files.","lead":"This paper finds that selecting LLM pre-training files by similarity to high-quality domains causes dimensional collapse, and proposes DiSF, a method that selects the most decorrelated files in a text feature space. If it holds up, it suggests large fractions of pre-training data can be discarded with better downstream performance, a practical result for anyone training LLMs on a budget.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 1.5% budget and the resulting 5x data-efficiency number are chosen post hoc on the evaluation benchmark, and the margins over Full Data are single-run, so the headline efficiency claim is not yet established.","rationale":"The reader's stated weakest assumption is the proxy validity of Contriever-feature diversity, and that concern is real but partially mitigated by ablations across feature extractors, model architectures, and tasks. The more load-bearing issue is the empirical protocol behind the headline numbers: the paper explicitly chooses the 1.5% budget because it achieves the desired comparison against Full Data, and the main tables and figures are single-seed. If the budget were fixed a priori or if seed averaging were applied, the small margins over Full Data could easily vanish, which would directly invalidate the central quantitative claim of 5x data efficiency and 1.5x training efficiency. The theoretical weak-submodularity analysis is also incomplete (constants epsilon and mu are never computed, and the guarantee applies to global greedy, not the batch-wise Algorithm 1), but this is secondary because the empirical claim is the headline. The appropriate verdict remains CONDITIONAL: the method is plausible and the experiments are suggestive, but the central efficiency claim needs a fixed protocol and error bars before it can be accepted as stated.","tokens_in":26580,"tokens_out":11433,"duration_ms":117819,"concrete_test":"Fix the selection budget before any evaluation (1.5%, plus 3% as a robustness check), and train TinyLlama 1.1B for 50B tokens with three independent seeds for DiSF and Full Data, using identical tokenization, data order, and random initializations. Report the mean and standard deviation of the 7-task Harness average. If the 1.5% DiSF mean is not above Full Data by more than the pooled standard error, or if 3% is needed to beat it, the headline 5x data-efficiency claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central efficiency claim rests on a condition that was not fixed in advance. Section 4.1 states: \"we choose to report performance with a 1.5% selection budget, since it achieves comparable performance compared to Full Data pre-training under 50B pre-training budget on TinyLlama 1.1B.\" The 1.5% budget is therefore selected after looking at the same Harness tasks used for the headline result. Figure 8 shows the budget-performance curve is non-monotonic (the 120M model peaks at 3% and declines), so the reported \"98.5% file savings\" is an optimized point on a curve, not a protocol-independent property of DiSF. In addition, Table 2 and Figures 5-6 report one training run per condition; the 1.1B/50B margin of DiSF over Full Data is roughly 0.5-1 point on the 7-task average, which is within typical seed-to-seed variation for small-model pretraining. Only Table 3 reports confidence intervals, and those are evaluation-level CIs, not training-seed CIs. Thus the headline \"outperforms full data\" and the derived 1.5x/5x efficiencies may shrink or disappear under a pre-registered budget or seed averaging. This is a correctness risk in the empirical protocol, independent of whether the diversity proxy itself is right.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes DiSF, a file-selection method for LLM pretraining that greedily chooses text files to minimize the Frobenius norm of the feature covariance matrix in a frozen Contriever embedding space, framed as γ-weakly submodular maximization. Experiments on TinyLlama (120M/560M/1.1B) with 10B and 50B token budgets on SlimPajama claim consistent average improvements over baseline selectors on nine Harness tasks, and that selecting 1.5% of files (about 9B tokens) outperforms full-data pretraining under a 50B budget, yielding approximately 1.5x training efficiency and 5x data efficiency.","tokens_in":26868,"tokens_out":7721,"duration_ms":70946,"significance":"If established, the efficiency gains would make DiSF a practical and inexpensive addition to LLM data curation: it uses an off-the-shelf feature extractor, its selection cost is moderate, and the code is provided. The paper also offers a benchmark across model sizes and ablations over architectures and feature extractors. However, the central efficiency claim is not yet secured, because the budget was chosen post hoc on the evaluation tasks and the main comparisons are single runs; the theoretical analysis in the appendix is also incomplete. These issues are fixable and do not invalidate the core idea, but they must be addressed before the paper's headline claims can be accepted.","major_comments":[{"comment":"Section 4.1 states: \"we choose to report performance with a 1.5% selection budget, since it achieves comparable performance compared to Full Data pre-training under 50B pre-training budget on TinyLlama 1.1B.\" This openly acknowledges that the 1.5% budget was selected after inspecting the evaluation results. Because Figure 8 shows that the budget-performance curve is non-monotonic (the 120M model peaks at 3% and declines), the \"5x data efficiency\" and \"1.5x training efficiency\" claims in Section 4.3 and the Abstract are not properties of a fixed selection protocol; they are the result of an oracle choice on the evaluation tasks. Please report results for a pre-registered budget or for a range of budgets with error bars, and derive the efficiency claims from that protocol.","section":"Section 4.1, Section 4.3, Figure 8"},{"comment":"All main results in Table 2 and Figures 5-6 are single training runs without seed variance. The headline comparison to Full Data relies on differences of roughly 0.5-1 point on the 7-task average for TinyLlama 1.1B with 50B tokens (e.g., DiSF 45.2 vs. QuRating-A 44.8 and Doremi 44.5 in Table 2). These margins are comparable to typical seed-to-seed variation in small-model pretraining, and the confidence intervals in Table 3 are evaluation-level only. Please provide seed-averaged results for the main table and the efficiency curves, or otherwise justify why single runs are sufficient for the claimed superiority.","section":"Table 2, Figures 5-6"},{"comment":"The derivations leading to the claimed weak-submodularity lower bound appear to have a direction error. Assumption 2 gives e^{-Δ(e|B)} ≤ e^{2μ}; since the function (e^{-Δ-ε}-1)/(e^{-Δ}-1) is increasing in e^{-Δ} for e^{-Δ}<1, the substitution yields an upper bound rather than the claimed lower bound e^{-2μ}(e^{2μ-ε}-1)/(e^{2μ}-1). Moreover, the constants ε and μ are never estimated, and the result is conditioned on \"positive gain\" without a precise characterization. Please correct the derivation or explicitly state that the theoretical guarantee rests on unvalidated assumptions; if the bound cannot be repaired, consider removing the approximation guarantee from the main text.","section":"Section 3.2, Appendix A.5.3"},{"comment":"The selection objective replaces the true objective in Eq. (1) with the Frobenius norm of the covariance matrix in a fixed Contriever feature space, but the paper does not establish that this proxy is aligned with downstream performance beyond the qualitative correlation in Figure 4. This is a load-bearing assumption for the method's validity. Please add a limitation paragraph discussing this gap, or provide an analysis (e.g., rank correlation between proxy values and held-out task performance across several feature extractors) to justify the proxy.","section":"Section 3.1, Eq. (5)"}],"minor_comments":[{"comment":"The citation \"Renduchintala et al., 1991\" is incorrect; INGENIOUS was published at EMNLP 2023, and the reference entry should be updated accordingly.","section":"Sections 2.2, 3.1, 4.1, and References"},{"comment":"The phrase \"a element x\" should read \"an element x\", and the domain of the set function should be written consistently as 2^Ω rather than 2^N when the ground set is denoted Ω.","section":"Section 3.2"},{"comment":"In the pseudocode, the removal step \"bi←bi\\{x*}\" is placed after the while-condition check, so the initial random sample is removed only in the first loop iteration; please reorder the lines to make the loop invariant explicit.","section":"Algorithm 1"},{"comment":"The proof uses the letter e both for the added sample and for Euler's number, which makes the formulas difficult to follow; please use a different symbol (e.g., s) for the added sample.","section":"Appendix A.5.3"},{"comment":"The caption promises color highlighting (\"bold blue, black, and red\") that is not visible in the monochrome text; please use textual markers or an accessible color scheme instead.","section":"Table 2 caption"},{"comment":"The sentence explaining the choice of 1.5% selection budget should be accompanied by a discussion of the risk of post-hoc selection, or moved to a limitations paragraph, since the same evaluation tasks are used for the headline result.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of the journal and the proposed method is interesting. The empirical protocol for the headline efficiency claim needs strengthening (pre-specified budget, seed variance), and the theoretical bound in Appendix A.5.3 appears to have a sign/direction error. I recommend major revision rather than rejection because the core idea is defensible and the issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the new idea is real. DiSF minimizes the Frobenius norm of the feature covariance on a fixed off-the-shelf encoder (Contriever) and does it greedily in batches. That is a different selection objective from D4 or INGENIOUS, and the paper shows it spreads eigenvalues more uniformly and consistently beats a range of baselines on TinyLlama at 120M–1.1B, across three architectures. The evaluation is honest in the important sense: selection uses external features and eval uses separate Harness tasks, so no circular fitting. The paper also ships code, reports selection costs, and includes ablations on feature extractor, selection scale, and budget. That is a solid empirical contribution at the subfield level.\n\nThe soft spot is exactly where the reader and stress-test put it. The 1.5% budget is not a protocol-independent property of the method. The paper says it chose 1.5% because that budget achieved comparable performance to full data on TinyLlama 1.1B. Figure 8 shows the budget-perf curve peaks at 3% for 120M and then declines, so 1.5% is an optimized point on the curve, not a finding that DiSF always gives 5x data efficiency. And the main results are one run per condition; the margin over full data is about 0.5–1 point on a 7-task average, which is within typical seed noise for this scale. The MMLU/BBH table has CIs but those are eval-set CIs, not training-seed CIs. So the headline \"outperforms full data with 98.5% file savings\" is plausible but not yet established.\n\nThe theory section is honest but thin: the weak-submodularity bound depends on Assumptions 1 and 2 with constants epsilon and mu that are never computed, so the guarantee is conditional. Not a fatal flaw—it is labeled as such and the empirical monotonicity plot helps—but it means the theoretical contribution is illustrative, not load-bearing.\n\nOverall: the central idea—decorrelation in a frozen feature space is a useful diversity proxy for LLM pretraining—holds up reasonably well at this scale. The paper deserves a serious referee. For the revision I would ask for multiple training seeds, a pre-registered or at least explained selection budget, and an attempt to estimate epsilon and mu. I would not desk-reject this.","headline":"Solid idea, honestly reported; the headline 5x data-efficiency number is not yet established because the 1.5% budget was chosen post hoc and the main runs have no seed variance.","tokens_in":27413,"tokens_out":1868,"would_cite":true,"duration_ms":18030,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Selecting pre-training files to maximize embedding decorrelation, not domain similarity, lets 1.5% of SlimPajama (about 9B tokens) beat full-data pre-training within a 50B-token budget on nine tasks.","keywords":["data selection","pre-training data curation","dimensional collapse","feature diversity","Frobenius norm decorrelation","weakly submodular optimization","data efficiency","large language models"],"falsifier":"Run DiSF's selection pipeline after destroying the geometry the criterion relies on: apply a random fixed rotation or projection to the Contriever embeddings before computing the covariance, so that decorrelation in the scrambled space no longer corresponds to any real semantic diversity. If the scrambled-feature selection still matches or beats full-data pre-training on the nine tasks, the decorrelation criterion itself is not carrying the gains; if performance collapses toward random-selection levels, the frozen feature space is confirmed as load-bearing. A complementary check is to compare DiSF with a deliberate non-uniform-spectrum diversity objective (for example, greedy log-determinant selection, the variant the paper calls DiSF-LD) at identical budgets: if that variant matches DiSF, generic diversity rather than eigenvalue uniformity is the active mechanism.","tokens_in":26370,"feed_emoji":"🎯","tokens_out":18138,"duration_ms":145253,"temperature":0.7,"pith_summary":"The paper claims that the right criterion for choosing LLM pre-training data is not similarity to a 'high-quality' domain but diversity: pick text files whose embeddings are decorrelated from one another. Domain-targeted selectors such as DSIR and QuRating, the paper argues, suffer a diversity dilemma — dimensional collapse in feature space that helps domain-like tasks while hurting general performance. DiSF selects files by minimizing the Frobenius norm of the feature covariance matrix, which by the paper's Lemma 1 is the same as flattening the eigenvalue spectrum toward a uniform distribution. Trained on just 1.5% of SlimPajama's roughly 590M files (about 9B tokens), TinyLlama models from 120M to 1.1B parameters outperform full-data pre-training within a 50B-token budget across nine evaluation tasks, gaining about 1.5x training efficiency and 5x data efficiency. A sympathetic reader should care because this suggests redundancy, not source quality, is the dominant correctable cost in pre-training data.","feed_headline":"Only 1.5% of training files beats full-data LLM pre-training","feed_subtitle":"A decorrelation-based pick of files gives ~1.5x training and ~5x data efficiency versus full-data pre-training.","key_machinery":"The carrying mechanism is the covariance-Frobenius proxy $F^{\\text{DiSF}}_M(U) = e^{-\\|C(U,M)\\|_F}$, computed from Contriever embeddings that are standardized per dimension. Lemma 1 is the hinge: because standardized features have trace $d$, the eigenvalue spread equals $\\sum_i(\\lambda_i - \\bar{\\lambda})^2 = \\|C(U,M)\\|_F^2 - d$, so minimizing the Frobenius norm is exactly flattening the eigenvalues — decorrelation and dimensional diversity become one objective. A classical greedy algorithm maximizes the proxy at batch scale (the paper uses a selection scale of $b = 1024$), and the analysis bounds the weak-submodularity ratio below by $e^{-2\\mu}\\frac{e^{2\\mu-\\epsilon}-1}{e^{2\\mu}-1}$, which yields the greedy approximation guarantee. The same machinery diagnoses the baselines: the dominance score $\\sum_{i=1}^k \\lambda_i / \\sum_j \\lambda_j$ quantifies how collapsed each method's selected feature distribution is, and DiSF attains the most uniform spectrum of all methods compared.","core_discovery":"The paper's central discovery is a diagnosis paired with a cure: file selection driven by similarity to a target domain (heuristic classification, DSIR, QuRating) concentrates the embeddings of chosen files in a thin, lower-dimensional slice of feature space, and this dimensional collapse is what inflates domain-like task scores while dragging down general performance. The proposed remedy, DiSF, treats a set of files as good when its feature covariance matrix $C(U,M)$ has minimal Frobenius norm $\\|C(U,M)\\|_F$, because for standardized features $\\sum_i (\\lambda_i - 1)^2 = \\|C(U,M)\\|_F^2 - d$: uniform eigenvalues and a small Frobenius norm are the same target. Maximizing the non-negative proxy $F^{\\text{DiSF}}_M(U) = e^{-\\|C(U,M)\\|_F}$ with a batch greedy algorithm yields, the paper proves via $\\gamma$-weak submodularity, a $(1-e^{-\\gamma})$-approximation to the optimal selection; empirically, the chosen 1.5% slice outperforms full-data pre-training at every model size tested. Stated as the authors would state it: decorrelating selected files in a frozen, off-the-shelf feature space is a sufficient proxy for the diversity that improves general LLM performance under a fixed token budget.","pith_inferences":["Because the criterion is defined on any embedding space, a natural untested extension is an adaptive feature extractor — for instance, re-selecting files from the model's own hidden states during training — which would tighten the loop between the diversity proxy and the representation actually being learned.","If spectrum-flatness is the right lens for diversity, scaling-law practice should treat redundancy as a first-class variable: the marginal value of an extra pre-training token should shrink as the selected corpus's covariance spectrum approaches uniformity, making the optimal corpus size a function of the compute budget rather than only of the data source.","The paper's own peak at a 3% selection ratio hints at a coupling the authors leave implicit: the optimal selection ratio and the token budget are linked, since a larger budget can absorb more decorrelated files before marginal returns vanish; mapping DiSF's optimum at other budgets (for instance 100B tokens) would chart this trade-off.","A direct extension to domain-mixture weighting suggests itself: apply the same covariance-Frobenius objective at the domain level to set mixture proportions, which would connect DiSF to mixture-optimization methods while keeping its freedom from trained proxies."],"forward_implications":["DiSF reaches or exceeds full-data pre-training performance using 1.5% of SlimPajama's files (about 9B tokens) inside a 50B-token budget, yielding about 5x data efficiency and about 1.5x training efficiency; the appendix quantifies the token share needed as 0.15 to 0.20 of the full-data corpus across model sizes.","The diversity dilemma is demonstrably avoidable: unlike DSIR, QuRating-W, and heuristic filtering, DiSF does not trade reading-comprehension gains for losses on physical-world tasks, and its average advantage over DSIR grows with training budget (from 2.1% to 3.4% on the 1.1B model) and with model scale (from 2.8% to 3.4% from 120M to 1.1B parameters).","The selection transfers across architectures (TinyLlama, Pythia, OPT), to a larger 3B model, and to a code-augmented corpus (SlimPajama plus StarcoderData), where domain-targeted selection visibly degrades code ability.","Selection ratio is a tunable dial with a peak near 3% of files (about 20B tokens) for TinyLlama 120M at a 50B budget; beyond that, performance declines toward full-data levels, so more selected data is not monotonically better.","No trained proxy or target domain is needed: a frozen off-the-shelf encoder (Contriever) suffices, and the selection cost is $O(|S|^2 b d^2)$, independent of total corpus size."],"supporting_citations":[{"why":"Supplies the frozen Contriever text encoder whose embedding space defines the decorrelation criterion for selection.","marker":"Izacard et al., 2021"},{"why":"The domain-resampling baseline (DSIR) whose selected files exhibit dimensional collapse; the main method DiSF must beat.","marker":"Xie et al., 2023b"},{"why":"Provides the QuRating judge-model baselines and the per-file quality judgments on SlimPajama reused in the benchmark.","marker":"Wettig et al., 2024"},{"why":"The D4 diversity-via-feature-distance baseline that DiSF compares against and surpasses in eigenvalue uniformity.","marker":"Tirumala et al., 2023"},{"why":"Supplies the redundancy-reduction principle linking covariance decorrelation to uniform eigenvalues, which motivates the Frobenius-norm objective.","marker":"Zbontar et al., 2021"},{"why":"Gives the greedy approximation guarantee for γ-weakly submodular maximization that the selection analysis invokes.","marker":"DAS, 2011"},{"why":"Defines the TinyLlama architecture and training recipe used for all pre-training runs in the benchmark.","marker":"Zhang et al., 2024a"},{"why":"The Harness framework supplies the nine evaluation tasks used to measure general performance and the efficiency curves.","marker":"Gao et al., 2024"},{"why":"SlimPajama is the corpus on which selection runs and against which full-data pre-training is compared.","marker":"Computer, 2023"}],"fun_headline_variants":["1.5% of data beats full training via decorrelated files","DiSF stops dimensional collapse, beats full-data LLM training","Diverse file selection: 5x data efficiency without collapse","Decorrelation in file pick prevents collapse, boosts general tasks","Skip 98.5% of files: DiSF outperforms full pre-training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire method rests on assuming that 'diverse' text, as measured by how evenly its Contriever embeddings spread across feature dimensions, is the same thing as text that improves LLM pre-training; the paper replaces the true objective of final task performance with this proxy without proving they coincide, and its formal guarantee for greedy selection depends on constants $\\epsilon$ and $\\mu$ (Assumptions 1 and 2 in Appendix A.5.3) that are left uncomputed.","fun_headline_variants_meta":{"raw":{"variants":["1.5% of data beats full training via decorrelated files","DiSF stops dimensional collapse, beats full-data LLM training","Diverse file selection: 5x data efficiency without collapse","Decorrelation in file pick prevents collapse, boosts general tasks","Skip 98.5% of files: DiSF outperforms full pre-training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000285,"raw_usage":{"total_tokens":1762,"prompt_tokens":1109,"completion_tokens":653,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":725,"completion_tokens_details":{"reasoning_tokens":563}},"tokens_in":725,"tokens_out":653,"duration_ms":6776,"temperature":1.0,"reasoning_tokens":563,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:23:19.836649+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DiSF's selection pipeline after destroying the geometry the criterion relies on: apply a random fixed rotation or projection to the Contriever embeddings before computing the covariance, so that decorrelation in the scrambled space no longer corresponds to any real semantic diversity. If the scrambled-feature selection still matches or beats full-data pre-training on the nine tasks, the decorrelation criterion itself is not carrying the gains; if performance collapses toward random-selection levels, the frozen feature space is confirmed as load-bearing. A complementary check is to compare DiSF with a deliberate non-uniform-spectrum diversity objective (for example, greedy log-determinant selection, the variant the paper calls DiSF-LD) at identical budgets: if that variant matches DiSF, generic diversity rather than eigenvalue uniformity is the active mechanism.","supporting_citations":[{"cited_title":"Qurating: Selecting high-quality data for training language models","cited_arxiv_id":null,"evidence_quote":"Provides the QuRating judge-model baselines and the per-file quality judgments on SlimPajama reused in the benchmark."},{"cited_title":"Submodular meets spectral: Greedy algorithms for subset selection, sparse approximation and dictionary selection","cited_arxiv_id":null,"evidence_quote":"Gives the greedy approximation guarantee for γ-weakly submodular maximization that the selection analysis invokes."}],"review_version":1}