{"id":"8a0359a7-03f5-4d28-85af-7eb9edbefb47","arxiv_id":"2606.18663","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"RegMix-D fits regression models to proxy loss trajectories to produce dynamic data mixture schedules that outperform static RegMix and DoReMi on 25B-token Pile pretraining with a 1B model.","lead":"RegMix-D extends static data mixture selection by fitting regression models to full loss trajectories from small proxy runs, allowing prediction of optimal mixtures at multiple training stages. A smart generalist might read it because data mixture choice is a major cost driver in LLM pretraining and dynamic schedules could reduce wasted compute.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"Proxy-to-target transfer of trajectory-based mixture predictions lacks direct validation","rationale":"The reader's weakest assumption is exactly the transferability step required for the regression to be useful; the abstract-only review correctly flags it, and the full-text claim does not add independent evidence (such as proxy-target loss correlation plots or scaling ablations) that would remove the concern.","tokens_in":1693,"tokens_out":313,"duration_ms":9649,"concrete_test":"Train the 1B target once with the offline RegMix-D schedule and once with a static RegMix baseline; at 5B, 10B, 15B, and 20B tokens record per-domain losses on held-out validation sets. If the dynamic schedule does not produce measurably lower losses than the static one at multiple checkpoints, the trajectory regression has not transferred.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The method trains a regressor on full loss trajectories from small proxy runs to output time-varying mixture weights for a 1B target. This is only valid if the mapping from mixture to loss trajectory is sufficiently preserved across model scale. The abstract reports downstream gains on the Pile with 25B tokens, but provides no ablation measuring how closely the proxy-derived schedule matches the actual loss surface experienced by the target at corresponding token counts, nor any scaling check (e.g., proxy size vs. prediction error). The online mode still initializes from the proxy regressor, so the same transfer assumption remains load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces RegMix-D as a dynamic extension of RegMix for LLM pretraining data mixture selection. It trains regression models on full loss trajectories (rather than endpoint losses) from small proxy runs to predict time-varying optimal mixtures at multiple training stages. The method offers offline (precomputed schedule) and online (adaptive during target training) modes. Experiments train a 1B target model on 25B tokens from the Pile and report consistent gains over RegMix and DoReMi across 13 downstream tasks while using only 25% of RegMix's proxy compute budget (128 proxy models).","tokens_in":1829,"tokens_out":487,"duration_ms":18213,"significance":"If the proxy-to-target trajectory transfer holds, RegMix-D would provide a practical route to dynamic data mixing that improves downstream performance with substantially lower proxy overhead than static regression baselines. The proxy-efficiency result and the use of trajectory information rather than single-point losses are the primary contributions.","major_comments":[{"comment":"Abstract and Experiments: The central claim that proxy-derived dynamic mixtures improve over RegMix relies on the untested assumption that loss trajectories observed on small proxies are sufficiently predictive of the loss surface experienced by the 1B target at corresponding token counts. No ablation or scaling check measuring prediction error or schedule fidelity between proxy and target is reported.","section":"Abstract / Experiments"},{"comment":"Method: The regression model choice, feature construction from trajectories, and how targets are defined at multiple stages are not described. Without these details it is impossible to assess whether the reported gains are robust or sensitive to modeling decisions.","section":"Method"},{"comment":"Experiments: The claim of consistent gains \"across 13 downstream tasks\" and the proxy-efficiency comparison (128 vs. RegMix's budget) lack reported statistical significance tests, variance across runs, or controls for multiple testing, weakening the strength of the empirical conclusion.","section":"Experiments"}],"minor_comments":[{"comment":"The distinction between offline and online deployment modes is described at a high level; a concrete pseudocode or diagram would clarify how the online mode uses observed loss to adapt the regressor output.","section":"Method"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed and constructive review. We address each major comment below, indicating where we agree revisions are needed and what changes will be made in the revised manuscript.","responses":[{"response":"We agree that the manuscript does not include an explicit ablation or scaling study that directly measures prediction error or schedule fidelity between proxy trajectories and the 1B target. The reported gains on the target model provide indirect support for the transfer, but this does not substitute for a direct check. In the revised version we will add an analysis (new subsection or appendix) that evaluates proxy-target fidelity, for example by comparing mixtures predicted from proxies against those that would be optimal on partial target runs or additional proxy scales. This addresses the concern while preserving the core empirical results.","revision_made":"yes","referee_comment":"[Abstract / Experiments] Abstract and Experiments: The central claim that proxy-derived dynamic mixtures improve over RegMix relies on the untested assumption that loss trajectories observed on small proxies are sufficiently predictive of the loss surface experienced by the 1B target at corresponding token counts. No ablation or scaling check measuring prediction error or schedule fidelity between proxy and target is reported."},{"response":"The referee correctly identifies that the Method section omits key implementation details. We will expand this section to specify the regression model (including type and hyperparameters), the exact feature construction process from loss trajectories (e.g., which time points or summary statistics are used), and the procedure for defining targets at multiple training stages. These additions will allow assessment of robustness and will be placed in the main text or a dedicated subsection.","revision_made":"yes","referee_comment":"[Method] Method: The regression model choice, feature construction from trajectories, and how targets are defined at multiple stages are not described. Without these details it is impossible to assess whether the reported gains are robust or sensitive to modeling decisions."},{"response":"We acknowledge the absence of statistical significance tests, run-to-run variance, and multiple-testing controls in the current Experiments section. In the revision we will add error bars or standard deviations (where multiple runs exist), perform appropriate significance tests on the downstream improvements, and apply a correction for multiple comparisons. The proxy-efficiency comparison will be clarified with explicit compute accounting. Full variance reporting on all 1B runs may be limited by compute cost, so we will note this limitation and supplement with proxy-run statistics where possible.","revision_made":"partial","referee_comment":"[Experiments] Experiments: The claim of consistent gains \"across 13 downstream tasks\" and the proxy-efficiency comparison (128 vs. RegMix's budget) lack reported statistical significance tests, variance across runs, or controls for multiple testing, weakening the strength of the empirical conclusion."}],"tokens_in":1366,"tokens_out":596,"duration_ms":23081,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The key point here is that RegMix-D extends the original RegMix by regressing on full loss trajectories from proxy models instead of just endpoint losses. This lets it output mixture weights that change across training stages, either as a fixed offline schedule or adapted online based on observed losses during the main run.\n\nThe paper demonstrates this on 25 billion tokens from the Pile using a 1 billion parameter target model. It reports consistent improvements over both the static RegMix and DoReMi baselines across 13 downstream tasks. Notably, one variant achieves this with only 128 proxy models, which is 25 percent of the compute used by the original RegMix. That efficiency angle is the practical win if the numbers check out.\n\nOn the positive side, the approach is a natural extension and the two modes give flexibility for different training setups. The claim of remaining proxy-efficient while improving performance is worth attention for labs that already invest in proxy runs.\n\nThe main concern is the lack of direct evidence for the proxy-to-target transfer. The method assumes that loss trajectories seen on small proxies will guide good mixtures for the larger target at corresponding points in training. The reported results show downstream gains, but without an ablation comparing the proxy-predicted schedule to the actual loss behavior on the target, it's unclear how much of the improvement comes from the dynamic aspect versus other variables. The abstract also doesn't specify the regression model used, how features are built from the trajectories, or any significance testing, which leaves the strength of the evidence open.\n\nThis work is for people in the data mixture selection area of LLM pretraining. Someone already running RegMix-style experiments would find the extension straightforward to understand and potentially build on.\n\nOverall, the paper engages honestly with the literature and presents a testable idea with empirical support, so it should go to peer review for a closer look at the methods and results.","headline":"RegMix-D adds dynamic stage-wise mixing from proxy trajectories and reports efficiency gains over static RegMix, but the proxy-to-target transfer assumption lacks direct checks.","tokens_in":2333,"tokens_out":457,"would_cite":false,"duration_ms":23499,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Loss trajectories from small proxy models enable dynamic data mixing that outperforms static mixtures for LLM pretraining.","keywords":["data mixture selection","proxy training","loss trajectories","dynamic mixing","large language models","pretraining","regression models","Pile dataset"],"falsifier":"Run a large target model with the dynamic mixture schedule predicted from proxy trajectories and compare its downstream scores to a model trained with the static mixture from the same proxies; absence of improvement or lack of correlation between proxy and target trajectories would falsify the claim.","tokens_in":2589,"feed_emoji":"","tokens_out":732,"duration_ms":18376,"temperature":0.7,"pith_summary":"The paper aims to establish that proxy training runs yield full loss trajectories rather than just final losses, and that these trajectories can be fed into a regression model to predict optimal data mixtures at multiple stages of pretraining. This produces a dynamic mixing schedule instead of the single static mixture chosen by prior methods. Experiments train a 1B-parameter model on 25B tokens from the Pile and report gains across 13 downstream tasks while using fewer proxy runs than the static baseline. A sympathetic reader would care because the choice of data mixture directly shapes what a large language model learns from heterogeneous sources during pretraining, and a more efficient way to tune it could lower overall compute.","feed_headline":"Proxy loss trajectories predict dynamic data mixes for LLMs","feed_subtitle":"RegMix-D fits regression models to full proxy curves to set stage-specific mixtures, beating static baselines on 13 tasks with 25 percent of","key_machinery":"Regression model trained on proxy loss trajectories to predict stage-specific optimal data mixtures.","core_discovery":"RegMix-D trains a regression model on the complete loss trajectories observed in small proxy runs, rather than endpoint losses alone. This model then predicts the data mixture that minimizes loss at each stage of target-model training. The approach supports an offline mode that outputs a full schedule before target training begins and an online mode that adjusts the mixture on the fly using observed losses. On the Pile dataset with a 1B target model, the resulting schedules improve downstream performance over both the static RegMix baseline and DoReMi while requiring only 25 percent of the proxy compute budget used by RegMix.","pith_inferences":["Trajectory-based regression could be applied to other training decisions such as learning-rate schedules or model-size scaling.","If proxy trajectories remain predictive across model scales, the approach might shrink the compute needed for hyperparameter search in general.","Online adaptation could serve as a safeguard that corrects an initial mixture once early target losses deviate from proxy predictions.","Splitting trajectories into finer-grained stages or predicting continuous mixture weights might further tighten the schedule."],"forward_implications":["Dynamic mixtures yield higher downstream accuracy than a single static mixture chosen from the same proxies.","The method remains proxy-efficient, delivering gains even when the number of proxy runs is cut to 25 percent of the static baseline budget.","Both a fixed schedule computed in advance and an adaptive schedule updated during target training are viable.","The same regression-on-trajectories idea extends the original RegMix framework from one-time selection to time-varying selection."],"fun_headline_variants":["Proxy trajectories enable dynamic data mixing in RegMix-D","Full proxy loss curves predict stage-specific mixtures","RegMix-D uses trajectory regression for dynamic LLM mixes","Loss trajectories from proxies set optimal data schedules"],"cache_read_input_tokens":64,"weakest_assumption_plain":"Loss trajectories seen on small proxy models are sufficiently predictive of the loss surface that the same mixtures will produce on a large target model at corresponding stages.","fun_headline_variants_meta":{"raw":{"variants":["Proxy trajectories enable dynamic data mixing in RegMix-D","Full proxy loss curves predict stage-specific mixtures","RegMix-D uses trajectory regression for dynamic LLM mixes","Loss trajectories from proxies set optimal data schedules"]},"model":"grok-4.3","cost_usd":0.004529,"raw_usage":{"total_tokens":2252,"prompt_tokens":666,"num_sources_used":0,"completion_tokens":57,"cost_in_usd_ticks":45287000,"prompt_tokens_details":{"text_tokens":666,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1529,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":666,"tokens_out":57,"duration_ms":12650,"temperature":1.0,"reasoning_tokens":1529,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T20:59:53.284960+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Run a large target model with the dynamic mixture schedule predicted from proxy trajectories and compare its downstream scores to a model trained with the static mixture from the same proxies; absence of improvement or lack of correlation between proxy and target trajectories would falsify the claim.","supporting_citations":[],"review_version":1}