{"id":"7eb67164-09ce-4747-b4f0-39add87b80ef","arxiv_id":"2602.00747","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Weighted averaging of component models trained on individual data sources can serve as a cheap, faithful proxy for training on arbitrary data mixtures, enabling cheaper data-mix search for LLM pre-training.","lead":"The paper introduces DeMix, a way to search for the best data mix when pre-training large language models: instead of training a fresh model for each candidate mix, it trains one model per data source and then averages their weights to estimate how any mix would perform. The authors report that this shortcut finds better mixes at much lower compute cost than existing methods, and they release a 22-trillion-token dataset with the recommended mix.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported optimal mixture lies outside the subspace the merged proxies can represent, so the central claim is not yet supported.","rationale":"The reader identified the same core weakness: the proxy's implied mixture (0.5·Σα_iD_i + 0.5·D_general) does not match the final reported mixture (General=0.218, domains=0.782). My reading strengthens this by locating an explicit admission in Appendix A.1 that DeMix 'can only search within the subspace where non-general data accounts for less than 50%', while Table 9 reports a final mixture with 78.2% non-general. This is not merely a possible extrapolation risk; it is an internal contradiction between the method's stated searchable space and the reported output. The central claim — that DeMix efficiently discovers an optimal data mixture via model merging — is therefore unsupported for the specific mixture that drives the headline results. The proxy-consistency evidence in Table 2 is also affected, because if the 96 reference mixtures are sampled from the full 7-simplex, the proxies and references have systematically different domain proportions, so high Spearman ρ may reflect monotonicity rather than faithful approximation. These issues are addressable in revision (e.g., by constraining the final mixture to the proxy's valid subspace, retraining components without the 50% general admixture, or validating extrapolation with high-domain reference models), so a conditional acceptance is appropriate rather than outright rejection.","tokens_in":20231,"tokens_out":8040,"duration_ms":95119,"concrete_test":"Train reference models on 50B-token mixtures spanning non-general share 50–80%, including the exact DeMix-224 mixture from Table 9, and compare their benchmark ranks to the DeMix merged-proxy ranks for the same α. If the Spearman ρ drops substantially below the Table 2 macro value (0.81), the proxy does not rank high-domain mixtures correctly, confirming that the reported optimal mixture is outside the validated search space. Also train a final model on the best mixture constrained to non-general ≤50%; if it achieves a comparable or better rank than 24.00, the Table 3 result does not require the out-of-distribution 78.2% mixture.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The load-bearing gap is that the merged proxy's training distribution is not the mixture being optimized. Each component is trained on 0.5·D_i + 0.5·D_general (Section 3, β=0.5; Appendix A.1). Therefore M = Σα_iΘ_i approximates training on (α_G + 0.5(1−α_G))·D_general + 0.5Σ_{i≠G}α_iD_i, so the proxy can only represent mixtures with at most 50% non-general data. The paper itself concedes this: 'Although we can only search within the subspace where non-general data accounts for less than 50%...' (Appendix A.1). Yet the headline DeMix-224 mixture in Table 9 is General=0.218, Math+Code=0.782 — 78.2% non-general, far outside that subspace. The same mismatch applies to the proxy-consistency evaluation: if the 96 reference mixtures are sampled from the 7-simplex, their mean non-general share is ~86%, while the merged proxies contain ≤50%, so the reported ρ=0.81 measures correlation across systematically different training distributions, not fidelity of the mapping in Eqs. 5–6. Unless the predictor's extrapolation to >50% non-general is separately validated, the Table 3 result (rank 24.00) cannot be attributed to DeMix's search.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DeMix, a data-mixture search framework for LLM pre-training. Instead of training a new proxy model for every sampled mixture, DeMix trains seven component models, each on one candidate dataset mixed 50/50 with general data, and then synthesizes proxy models for arbitrary mixture weights by weighted linear merging. A LightGBM predictor is fit to benchmark rankings of these merged proxies and used, with iterative resampling, to select a final mixture, which is then validated by training a 1.7B model on 50B tokens. The paper reports that merged proxies achieve macro-average Spearman ρ=0.81 against 96 reference models at a total budget of ~212B tokens, and that the final DeMix mixture achieves a better macro-average rank (24.00) than RegMix and CLIMB baselines. The paper also releases DeMix Corpora, a 22T-token dataset with stage-wise validated mixtures.","tokens_in":20557,"tokens_out":4650,"duration_ms":52820,"significance":"If the proxy-fidelity claim were established, DeMix would be a meaningful advance: it decouples the cost of mixture search from the number of sampled mixtures, turning an expensive search into a fixed-cost component-training phase plus negligible merging/benchmarking. The paper is transparent about experimental settings, compares several merging methods, and releases code and data, which are concrete strengths. The central concern is that the proxy models and the final searched mixture live in different distributional subspaces; the paper's own appendix acknowledges that only mixtures with less than 50% non-general data are searchable, yet the headline final mixture is 78.2% non-general. This mismatch affects both the proxy-consistency validation and the attribution of the final mixture's performance to the search procedure. The issue is load-bearing but appears fixable within the manuscript's scope.","major_comments":[{"comment":"The derivation of the merged proxy assumes additivity of weight deltas for models trained on pure datasets: Δ(D_i ∪ D_j) ≈ Δ(D_i) + Δ(D_j), leading to M = Σα_iΘ_i approximating training on Σα_iD_i. However, each component model is trained on 0.5·D_i + 0.5·D_general (Section 3, β=0.5). Therefore M ≈ Θ_base + Δ(0.5·Σα_iD_i + 0.5·D_general), i.e., the proxy represents an affine-shifted mixture with at most 50% non-general data. This is not the mixture Σα_iD_i that the paper claims to optimize. Appendix A.1 concedes this: \"Although we can only search within the subspace where non-general data accounts for less than 50%...\" This concession should appear in the main text, and the derivation should be corrected or the search explicitly restricted to the affine subspace that merged proxies can represent.","section":"Section 2.3, Eq. (5)-(6); Section 3; Appendix A.1"},{"comment":"The proxy-consistency evaluation samples 96 reference mixtures from the full 7-simplex. Under uniform sampling, the expected non-general share of a reference mixture is 6/7 ≈ 86%, while the merged proxies always contain at most 50% non-general data. Thus the reported ρ=0.81 measures rank correlation between two systematically different training distributions, not the fidelity of the mapping in Eq. (5)-(6) for the mixtures of interest. To support the claim that merging preserves ranking of 'models trained on real data mixtures', the evaluation should restrict the reference mixtures to the proxy-representable subspace (or otherwise correct for the affine shift), and report ρ separately within that subspace.","section":"Section 4.1; Section 3.3.1"},{"comment":"The final DeMix-224 mixture selected by the predictor has General=0.218 and non-general=0.782, i.e., 78.2% non-general data. This lies far outside the subspace of mixtures that merged proxies can represent (non-general ≤50%). The LightGBM predictor is trained on proxy scores from the ≤50% subspace and then used to select mixtures in the >50% region, but no out-of-subspace validation is provided. Consequently, the reported rank of 24.00 for the final mixture cannot be attributed to DeMix's search; it could reflect the extrapolation behavior of the regressor rather than the quality of the proxy signal. The authors should either validate extrapolation with reference models trained at high non-general shares, or change the search/proxy construction so that the final mixture lies inside the validated region.","section":"Table 3; Appendix C, Table 9"},{"comment":"The Capability Recovery Rate compares average benchmark scores of the merged proxy and the reference model. Because the proxy's effective mixture is shifted toward general data (at least 50% general by construction), a high recovery rate may simply reflect that general benchmarks dominate the average, rather than that the proxy preserves the absolute performance of the reference mixture. At minimum, capability recovery should be reported per domain and conditioned on the actual mixture represented by the proxy, not on the nominal α.","section":"Section 4.1, Capability Recovery"}],"minor_comments":[{"comment":"The y-axis label \"Proxy Accuracy (Spearman's)\" is incomplete; it should read \"Spearman's ρ\".","section":"Figure 1"},{"comment":"The paper states that δ is approximately 10% but does not show how this is measured or for which dataset/model. A brief empirical derivation or a reference to an appendix would help.","section":"Section 2.3"},{"comment":"The concession about the searchable subspace is placed in the data-curation appendix. Since it directly limits the central claim, it should be moved or at least summarized in Section 2.3 or Section 3.","section":"Appendix A.1"},{"comment":"The abstract and Section 5 describe \"22T-token\" DeMix Corpora, while Table 8 lists stage totals (14.4T, 5.8T, 2.1T) that sum to about 22T; this is consistent but should be stated explicitly in the main text to avoid confusion.","section":"Table 8 / Section 5"},{"comment":"For the DeMix rows, \"Pre-Cost\" is not clearly defined. The reader has to infer that it is the cost of training the base model; please define this column header in the table caption.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read DeMix. The core idea is worth a second look: train a handful of component models, then synthesize arbitrarily many proxy models by weighted averaging of weights, and use those proxies to search the data-mixing simplex. That is a genuine new idea, and the paper backs it with a lot of work: a 22T-token corpus release, 96 reference models, ablations over merging methods and mixing ratios. The observation that merged proxies correlate with reference models (ρ=0.81 macro) is interesting, and the cost savings over training-based proxies are real if the proxies are faithful.\n\nBut there is a load-bearing mismatch that the paper itself concedes in Appendix A.1. Each component is trained on 0.5·D_i + 0.5·D_general. So a weighted merge of components is, under the paper's own additivity assumption, a proxy for a mixture with at most 50% non-general data. The paper acknowledges: \"we can only search within the subspace where non-general data accounts for less than 50%.\" Yet the headline DeMix-224 mixture in Table 9 has General=0.218, i.e. 78.2% non-general. That is far outside the subspace the proxies can represent. The final mixture was selected by a LightGBM predictor that extrapolated beyond its training domain, and the paper provides no validation of that extrapolation.\n\nThis undermines the central claim. The reported ρ=0.81 is computed against 96 reference mixtures sampled from the full 7-simplex, whose mean non-general share is ~86%. The merged proxies, by construction, contain ≤50% non-general data. So the correlation is between models trained on systematically different distributions, not a test of the mapping in Eqs. 5–6. The top-25% ρ of 0.59 is modest and does not rescue the point. Without separate validation of extrapolation to high domain share, the Table 3 result (rank 24) cannot be attributed to DeMix's search. It could be a lucky extrapolation, or the general-heavy proxy may still carry some signal, but the paper doesn't show that.\n\nOther soft spots: no error bars anywhere; the comparison with RegMix/CLIMB is not apples-to-apples because the trained proxies are trained on the actual sampled mixtures while merged proxies are not; and the \"breaks the trade-off\" framing overstates what is demonstrated.\n\nStill, this is a promising research direction, and the flaw is addressable: restrict the search to the representable subspace and report a final mixture inside it, or develop a correction for the fixed general-data injection. The corpus release is a real contribution. I'd send this to peer review with a request for major revision, but I would not cite the current version's empirical claims.","headline":"Clever proxy idea, but the final mixture is outside the representable subspace, so the central validation doesn't hold as written.","tokens_in":21059,"tokens_out":4490,"would_cite":false,"duration_ms":49978,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Weighted averaging of separately trained component models preserves the benchmark ranking of models trained directly on the same data mixture, letting LLM pre-training data-mixture search run at a fraction of the compute.","keywords":["data mixture optimization","model merging","LLM pre-training","proxy models","weighted parameter averaging","benchmark ranking","pre-training corpus","DeMix"],"falsifier":"A direct test: sample several ratios with 70-90% domain weight, train 50B-token reference models on each, and compare merged-proxy rankings with the real trained-model rankings; if Spearman ρ in this region is much lower than the reported 0.81, the central claim fails where the final mixture lives.","tokens_in":20118,"feed_emoji":"🧠","tokens_out":9038,"duration_ms":91261,"temperature":0.7,"pith_summary":"Choosing the right mix of pre-training data is one of the strongest levers on a language model's final abilities, but testing a mixture normally means training a whole model on it, which is too expensive for broad search. The paper proposes DeMix: train one component model per candidate dataset, then synthesize a proxy for any mixture by a weighted average of component weights. The central evidence is that these merged proxies preserve the ranking of reference models actually trained on those mixtures—macro-average Spearman rank correlation 0.81 at a total budget of about 212B tokens, versus 0.53 for 2B-token trained proxies at comparable cost. Because each proxy is a merge rather than a training run, the method can evaluate far more candidate mixtures and still find a better final mixture at lower compute. A 1.7B-parameter model trained on the selected mixture outperforms mixtures from trained-proxy search methods across general, math, and code benchmarks.","feed_headline":"Merged models find LLM data mixes 6x cheaper than trained proxies","feed_subtitle":"Weighted averaging of trained components preserves mixture ranking, making LLM data-mix search cheap and exhaustive.","key_machinery":"The load-bearing identity is approximate linear additivity of parameter updates: if training on a dataset changes model weights by a small delta (about 10% of total weight magnitude here), then the delta from training on a union of datasets is close to the sum of the individual deltas. Consequently, merging the individually trained component models with weights {α_i} approximates the parameters of a model trained on the same mixture {α_i}. Around this identity, DeMix wraps an iterative search: sample ratios uniformly, construct merged proxies, rank them on a benchmark suite, fit a regression predictor, and resample near the top-ranked ratios.","core_discovery":"The core discovery is that a weighted linear merge of component models—each trained once on one candidate data source mixed with 50% general data—behaves like a faithful proxy for a model trained directly on the corresponding weighted mixture. In the paper's evaluation, 96 random mixtures were scored both by merged proxies and by reference models trained on 50B tokens; the merged proxies reached macro-average Spearman ρ = 0.81 and top-25% ρ = 0.59 at a total search budget of 212B tokens, while 2B-token trained proxies under a comparable budget reached only 0.53 and 0.20. The paper also reports a capability recovery rate of up to 0.85, meaning the absolute benchmark scores of merged proxies a","pith_inferences":["Editorial inference: the paper validates proxy fidelity inside the 50%-general subspace, but the final selected mixture (e.g., 21.8% general, 78.2% domain) sits far outside it, so the ranking transfer that matters for the final choice is an extrapolation, not a measured fact.","Editorial inference: the small-update additivity premise (δ≈10%) means merged proxies should become less faithful as components train longer or scale up; at frontier training scales, components may need to be re-trained more often, which would erode the claimed cost advantage.","Editorial inference: because the search objective is defined by a fixed benchmark suite, the chosen mixture is optimal for those benchmarks; a capability absent from the suite would not be optimized, and mixture choices might shift if new benchmarks are added."],"forward_implications":["Data-mixture search becomes essentially free after one training pass per candidate dataset: any number of sampled ratios can be evaluated by merging the same set of component models, so the number of search trials is no longer bounded by training budget.","Under a comparable token budget, merged proxies are far more reliable ranking signals than small trained proxies (macro ρ 0.81 vs 0.53), and the gap is especially stark on general benchmarks, where 2B-token proxies show near-zero correlation.","Increasing the number of merged proxies improves the quality of the final mixture up to a point: the paper observes rank improvement from 29.33 with 56 proxies to 24.00 with 224 proxies, and degradation at 448, suggesting overfitting to proxy noise.","Mixing each candidate dataset with at least 50% general data is necessary for proxy fidelity: dropping to 25% or 0% general data lowers Spearman ρ from about 0.79 to 0.67/0.65.","The released 22T-token corpus with validated mixtures can be used directly for pre-training, and a 1.7B model trained on it achieves better average benchmark rank than comparable mixed corpora."],"fun_headline_variants":["Merged models proxy LLM data mixes at a fraction of cost","Model merging predicts optimal data ratios without extra training","Data mix search via merged models, not extra training","Cheap LLM data-mix search via model merging","Merged models decouple LLM data-mix search from training"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a weighted average of separately trained component models approximates a model trained directly on that weighted mixture—so merged proxies rank mixtures the way real training would, even for domain-heavy mixtures far from the components' own 50%-general training distribution.","fun_headline_variants_meta":{"raw":{"variants":["Merged models proxy LLM data mixes at a fraction of cost","Model merging predicts optimal data ratios without extra training","Data mix search via merged models, not extra training","Cheap LLM data-mix search via model merging","Merged models decouple LLM data-mix search from training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000852,"raw_usage":{"total_tokens":3556,"prompt_tokens":772,"completion_tokens":2784,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":2700}},"tokens_in":516,"tokens_out":2784,"duration_ms":23038,"temperature":1.0,"reasoning_tokens":2700,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:54:22.903058+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test: sample several ratios with 70-90% domain weight, train 50B-token reference models on each, and compare merged-proxy rankings with the real trained-model rankings; if Spearman ρ in this region is much lower than the reported 0.81, the central claim fails where the final mixture lives.","supporting_citations":[],"review_version":2}