{"id":"894784dc-c73d-48dd-972a-965c2139a16e","arxiv_id":"2508.12116","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DynamixSFT adaptively reweights instruction-tuning datasets during training using a prior-anchored bandit and one-step loss-reduction reward, giving small but consistent benchmark gains.","lead":"This paper presents a dynamic data-mixing method for instruction tuning, treating each dataset as a slot-machine arm and reweighting datasets during training based on how much they reduce model loss. A 1B and a 3B language model improved by 1.5 percent to 2.2 percent on average across 10 benchmarks versus static dataset proportions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing no-reward control: reported gains may come from the static prior+uniform floor in Eq. 3, not from the 1-Step Look-ahead Reward.","rationale":"Good-faith read: the paper proposes a lightweight bandit scheduler and the ablation suite is informative. However, the central claim is that the adaptive reward-driven mixture is what improves over static baselines. The design confounds the reward with a static prior-scaled mixture: Eq. (3) always contains (1-gamma)*p0 plus gamma/K. The W/O PRIOR ablation removes the prior but leaves the reward, and its distribution remains near uniform—strong evidence that Q alone does not move the policy much. The missing control is a training run with Q frozen at 0; it directly separates the adaptive mechanism from the static interpolation and is much cheaper than a full reward-correlation study. This is the single most load-bearing gap because if the control matches the reported averages, the method's 'dynamic' contribution is not demonstrated. I do not see an internal inconsistency; the concern is about omitted evidence. The reader's proxy concern is real but secondary: it only matters if the reward is actually changing the mixture. The final verdict remains CONDITIONAL; the condition should include publishing the no-reward control and ideally multiple seeds.","tokens_in":13172,"tokens_out":8001,"duration_ms":98011,"concrete_test":"Run Algorithm 1 exactly as in the paper on LLaMA3.2 1B and Qwen2.5 3B, but fix Q_k=0 for all k (equivalently set beta=0), so the sampler is the static mixture p_k=(1-gamma)*p0_k + gamma/K with gamma=0.3, same update interval, batch size, and epochs. Evaluate on the same 10 benchmarks. If the average is approximately 27.40 (1B) or 52.97 (3B) within run-to-run noise, the reward contributes nothing and the headline claim must be reframed; if the average falls back toward Full Coverage/Uniform values (26.81/52.21 or below), the reward is the active ingredient and the current claim survives this test.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The method's only adaptive component is the reward Q in Eqs. (4)-(5), but the paper never isolates it. The sampling rule (Eq. 3) already contains a strong static component: with gamma=0.3, 30% of the mass is uniform and the prior p0 anchors the rest. The W/O PRIOR row (Table 1) is not this control: it removes p0, not Q. In fact, the paper states that without p0 the mixture 'remains close to its initial uniform distribution throughout training,' which suggests the reward alone moves the policy very little. A no-reward control with Q=0 (i.e., static p=(1-gamma)*p0 + gamma/K) is absent. If this static mixture matches DynamixSFT's 27.40/52.97 averages, then the advertised +2.2%/+1.5% gains are due to the prior and floor, not to dynamic reweighting, and the central claim of adaptive mixture optimization is unsupported. If the static control is clearly worse, the reward is doing real work and the proxy concern becomes secondary. This control costs one training run per model and should be reported before claiming that the reward drives the gains.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DynamixSFT, a dynamic dataset-mixture optimization method for instruction tuning. Datasets are treated as arms in a multi-armed bandit; at each update interval the sampling distribution is recomputed with a Prior-scaled Boltzmann Exploration (Eq. 3) that combines a prior proportional to the original mixture, a uniform floor controlled by gamma, and an exponential-weight reward term. The reward is the relative one-step loss reduction of a virtual gradient step on a fresh mini-batch (Eq. 4), smoothed by an EMA (Eq. 5). The method is evaluated on the Tulu-v2 mixture with LLaMA3.2 1B and Qwen2.5 3B across 10 benchmarks, reporting up to +2.2% and +1.5% average gains over the static full-coverage mixture, and is compared against MultiDDS, MultiUAT, and ablations.","tokens_in":13450,"tokens_out":6377,"duration_ms":69117,"significance":"If the reported gains are real, DynamixSFT is a useful practical contribution: it is lightweight, requires no proxy model or validation-set labels, preserves coverage through the prior anchor and floor, and provides interpretable mixture trajectories. The entropy-reward variant in Figure 4 and the dynamics visualizations are strengths. However, the current empirical support is incomplete in a way that directly affects the central claim: the adaptive reward is never isolated from the static prior-plus-floor component, and hyperparameters appear to be selected on the same evaluation benchmarks. The contribution is a plausible incremental improvement, not yet a fully supported one.","major_comments":[{"comment":"The paper never isolates the adaptive reward Q. In Eq. (3), the sampling rule contains a large static component: with gamma=0.3, 30% of the mass is uniform, and the remaining mass is anchored to p0 by the prior-scaled Boltzmann term. Since all Q are initialized to zero, the Q=0 policy is p=(1-gamma)*p0 + gamma/K. This static mixture is not reported. The 'W/O PRIOR' row in Table 1 removes p0, not Q, and the text's observation that without the prior the distribution 'remains close to its initial uniform distribution' suggests the reward moves the policy very little. If the Q=0 static mixture matches the 27.40/52.97 averages, the advertised gains come from the prior and floor, not from dynamic reweighting. I request this control for both models; it costs one run each and is necessary to support the central claim of adaptive mixture optimization.","section":"§3.2, Eq. (3)–(5)"},{"comment":"The final hyperparameters (gamma=0.3, alpha=0.95, beta=4 for 1B and beta=5 for 3B; T_update=50) appear to be selected from sweeps over update interval, beta, gamma, and alpha on the 1B model's AVG score over the same 10 benchmarks. This is selection on the evaluation set, and no validation split is used anywhere in the pipeline. The reported +2.2%/+1.5% gains are therefore likely optimistic. Additionally, Table 1 reports single runs with no variance or multiple seeds; the within-row differences among Full Coverage, MultiDDS, MultiUAT, and DynamixSFT are often less than 1 point. Please provide a protocol with hyperparameters fixed before seeing the benchmarks, or at least report the full sweep grid and use a hold-out validation set for model selection, and include multiple seeds with standard deviations.","section":"§4.2/Table 1; Appendix Implementation Details; Figures 3, 5, 6"},{"comment":"The reward definition assumes that one-step relative loss reduction on a fresh mini-batch is a faithful proxy for a dataset's contribution to final benchmark performance. No experiment tests this assumption. The Delta-Entropy comparison in Figure 4 shows the framework is flexible, but it does not validate the proxy. A direct test would be to compute the correlation between per-dataset rewards and per-dataset held-out performance, or to compare DynamixSFT against a reward that is shuffled/randomized while keeping the same sampling rule. Without such evidence, the mechanism driving the gains remains unverified, even if the no-reward control shows the reward has some effect.","section":"§3.2, Eq. (4) and §4.3"}],"minor_comments":[{"comment":"The text says the collection has 16 datasets and K=16, but the dataset list in §2 contains 15 named entries and the Appendix refers to a combined 'science-related dataset' of five sub-datasets. Figure 2 also shows SciERC_NER and SciERC_Relation separately. Please reconcile the arm count and the dataset list.","section":"§2 / Appendix Details of Dataset Collection"},{"comment":"The panel label for DynamixSFT is corrupted ('D/y.pc/n.pc/a.pc/m.pc/i.pc/x.pcSFT'); it should read DynamixSFT.","section":"Figure 2"},{"comment":"The caption contains raw LaTeX control sequences ('/uni0394', 'Best baseline /uni0394-Entropy /uni0394-Loss'); the rendering should be fixed.","section":"Figure 4"},{"comment":"Please define the value of epsilon in Eq. (4) and state how the virtual one-step update interacts with the optimizer state and learning-rate schedule; the reward may depend on the current LR and batch composition.","section":"Algorithm 1 / Eq. (4)"},{"comment":"The adaptations of MultiDDS and MultiUAT to instruction tuning are not described (e.g., how validation examples are selected, how gradient similarity is computed across heterogeneous instruction datasets). Add implementation details for reproducibility.","section":"§4.1 Baselines"}],"recommendation":"major_revision","confidential_remarks":"The decisive experiment is the Q=0 static control. If it matches DynamixSFT, the paper's title and abstract overstate the contribution and the appropriate outcome would be rejection or a major reframing; if it is clearly worse, the method is likely sound. I would also ask the editor to require the authors to clarify the hyperparameter selection protocol, since the current appendix and figures are consistent with tuning on the test benchmarks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper proposes a clean, lightweight bandit-based method for dynamically mixing instruction-tuning datasets, with the specific twist of a prior-scaled Boltzmann policy anchored to the original dataset proportions and a one-step loss-reduction reward. That combination is genuinely new in the SFT setting—no proxy model, no validation set, just periodic virtual gradient steps. The ablations are thorough: sweeps over sharpness, exploration, EMA smoothing, update interval, and even an entropy-based reward. The visualizations of mixture dynamics over training are helpful. It is well written and honestly scoped to small models.\n\nThe central empirical claim, however, is not yet fully nailed down. The most important gap is the missing no-reward control. With the sampling rule in Eq. 3, setting Q = 0 gives a static mixture of (1-gamma) times the prior plus a uniform floor. The paper never reports this baseline. The W/O PRIOR ablation removes the prior, not the reward, so it does not isolate the contribution of the look-ahead signal. The paper's own observation that without the prior the mixture stays near uniform actually suggests the reward alone moves the policy very little. This control costs one run per model and should be mandatory before claiming the adaptive component drives the +2.2%/+1.5% gains.\n\nAlso soft: hyperparameters (beta, gamma, alpha) are selected from sweeps on the evaluation benchmarks themselves, and all numbers are single runs without error bars. That can easily inflate small gains. The abstract mentions Tulu-3 experiments, but the paper only reports Tulu-v2 results—sloppy, and should be fixed. The reward proxy itself (one-step loss reduction as a proxy for downstream utility) is untested, though the fact that entropy-based rewards give similar results is mildly reassuring.\n\nThese are all addressable. The method is plausible and the paper is a reasonable contribution to LLM post-training data mixing. But as written, the load-bearing claim that the reward does the work is not isolated. I would send it to peer review with a request for the no-reward control, multiple seeds, and a corrected abstract. For my own work, I'd wait for the revised version before citing.","headline":"Reasonable dynamic SFT mixture method, but the adaptive reward's contribution is not isolated by a control, and hyperparameter selection on eval plus single runs leave the +2.2% claim under-supported.","tokens_in":13955,"tokens_out":3243,"would_cite":false,"duration_ms":36974,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a bandit scheduler using a one-step loss-reduction reward can dynamically reweight instruction-tuning datasets, lifting average benchmark performance by up to 2.2% over static mixtures without extra parameters or a va","keywords":["instruction tuning","dataset mixture optimization","multi-armed bandit","dynamic sampling","Boltzmann exploration","one-step look-ahead reward","LLM post-training","Tulu-v2 mixture"],"falsifier":"Direct test: for the 16 Tulu-v2 datasets, compute each dataset's one-step relative loss-reduction reward at the first update and the held-out benchmark change caused by training on that dataset alone; if the rank correlation is not clearly positive, the reward is not tracking final utility. Simpler version: rerun DynamixSFT with per-dataset rewards randomly shuffled or negated; if average benchmark performance still keeps the reported gain, the claimed mechanism is not what is doing the work.","tokens_in":13073,"feed_emoji":"🎰","tokens_out":9943,"duration_ms":96201,"temperature":0.7,"pith_summary":"DynamixSFT claims that an instruction-tuning data mixture should not be a fixed recipe but a trajectory that tracks the model's current learning state. It treats each dataset as an arm of a multi-armed bandit and updates the sampling distribution with a Prior-scaled Boltzmann Exploration, which keeps the learned proportions softly anchored to the original dataset ratios so diversity and coverage are not lost. The utility signal is a lightweight one-step look-ahead reward: the relative drop in training loss on a fresh batch after a single temporary gradient step. On the 16-dataset Tulu-v2-mixture, the paper reports up to a 2.2% relative gain in average performance over static proportional sampling across 10 benchmarks, with LLaMA3.2 1B and Qwen2.5 3B, and better results than MultiDDS and MultiUAT, all without a validation set or extra trainable parameters. If the claim holds, post-training data selection can be automated and self-evolving rather than manually curated.","feed_headline":"Bandit scheduler lifts instruction-tuning mixes by up to 2.2%","feed_subtitle":"It reweights 16 instruction-tuning datasets on the fly from a one-step loss drop, needing no validation set or extra model.","key_machinery":"The load-bearing pair is the Prior-scaled Boltzmann policy plus the 1-Step Look-ahead reward. The policy (Equation 3) blends the original mixture proportions $p^{(0)}$ with a reward-sharpened softmax $\\exp(\\beta Q_k)$ and a uniform floor $\\gamma/K$, so dynamic weights can drift only as far as the reward justifies while every dataset keeps a nonzero chance. The reward (Equation 4) measures, for each dataset, the average relative loss drop from one temporary gradient step on a fresh mini-batch; this is the signal that tells the scheduler which datasets are currently useful. An exponential moving average (Equation 5) smooths the rewards to handle the non-stationary nature of training.","core_discovery":"The paper's central claim is that the right mixture for instruction tuning is a moving target: the optimal proportions change as the model learns, and the model itself can discover them. DynamixSFT defines the sampling probability for dataset $k$ as a convex blend of a prior-scaled softmax and a uniform floor, $p_k = (1-\\gamma)\\frac{\\exp(\\beta Q_k)\\,p_k^{(0)}}{\\sum_j \\exp(\\beta Q_j)\\,p_j^{(0)}} + \\gamma/K$, where $p^{(0)}$ is the original dataset distribution, $\\beta$ controls exploitation sharpness, $\\gamma$ sets a minimum exploration floor, and $Q_k$ is an EMA-smoothed reward estimate. The reward is $r_k = \\frac{1}{|B|}\\sum_{x\\in B} \\frac{L_{\\mathrm{pre}}(x)-L_{\\mathrm{post}}(x)}{L_{\\mathr","pith_inferences":["A consequence the paper leaves implicit: the one-step loss-reduction reward will tend to favor data the model can already make immediate progress on, so datasets that pay off only after sustained training may be downweighted; on harder or noisier collections this could cap or reverse the gains.","Because the learned distribution is anchored to the original proportions, the method inherits the strengths and weaknesses of the initial recipe; if the base mixture is badly composed, the bandit can only correct it within the anchor's pull.","A testable extension would be to compare the one-step loss reward against a leave-one-dataset-out benchmark contribution for each of the 16 datasets; a positive rank correlation would confirm the reward is tracking final utility, while a null or negative correlation would show the gains come from something else.","The method could naturally be extended from dataset-level arms to instance- or cluster-level arms; the paper itself flags instance-level weighting as open, and the bandit machinery does not depend on the granularity."],"forward_implications":["If correct, static, hand-designed mixture recipes in instruction tuning can be replaced by a scheduler that reweights datasets as training proceeds.","A model can allocate data according to its own current weaknesses, meaning the order and composition of data matter even within a fixed, uncurated collection.","Because the reward needs only one temporary gradient step, mixture optimization adds only modest compute over naive sampling, so it is feasible in post-training at 1B-3B scale.","The ablation suggests that the original dataset proportions are a crucial anchor; a uniform starting mixture does not recover the gains even when the same reward updates are applied.","The entropy-difference variant shows the framework tolerates reward definitions other than loss reduction, so the bandit layer is separable from the choice of utility signal."],"supporting_citations":[{"why":"Supplies the 16-dataset Tulu-v2-mixture collection and the training recipe that DynamixSFT optimizes and compares against.","marker":"(Ivison et al. 2023)"},{"why":"Supplies the 10-benchmark evaluation setup and the Tulu-3 mixture context the paper builds on.","marker":"(Lambert et al. 2025)"},{"why":"Source of the Boltzmann exploration strategy that the prior-scaled policy extends.","marker":"(Sutton, Barto et al. 1998)"},{"why":"Grounds Boltzmann exploration as a non-stationary bandit strategy the method adapts.","marker":"(Cesa-Bianchi et al. 2017)"},{"why":"Defines the MultiDDS dynamic-reweighting baseline DynamixSFT must outperform.","marker":"(Wang, Tsvetkov, and Neubig 2020)"},{"why":"Defines the MultiUAT uncertainty-based baseline used in the comparison.","marker":"(Wu et al. 2021)"},{"why":"Representative proxy-model mixture optimizer whose cost motivates the no-proxy design.","marker":"(Xie et al. 2023)"}],"fun_headline_variants":["Bandit scheduler adapts instruction data mix on the fly","Dynamic bandit reweights datasets during training","Instruction tuning: Self-tuned mixture via bandit","One-step reward guides adaptive data mixing"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that a dataset's one-step relative loss reduction on a fresh mini-batch is a faithful proxy for how much that dataset will improve the final benchmark performance; if the most valuable datasets happen to show slower immediate loss drops, the reward signal will steer sampling toward the wrong data.","fun_headline_variants_meta":{"raw":{"variants":["Bandit scheduler adapts instruction data mix on the fly","Dynamic bandit reweights datasets during training","Instruction tuning: Self-tuned mixture via bandit","One-step reward guides adaptive data mixing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000703,"raw_usage":{"total_tokens":3007,"prompt_tokens":743,"completion_tokens":2264,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":2205}},"tokens_in":487,"tokens_out":2264,"duration_ms":18783,"temperature":1.0,"reasoning_tokens":2205,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T19:33:53.295620+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Direct test: for the 16 Tulu-v2 datasets, compute each dataset's one-step relative loss-reduction reward at the first update and the held-out benchmark change caused by training on that dataset alone; if the rank correlation is not clearly positive, the reward is not tracking final utility. Simpler version: rerun DynamixSFT with per-dataset rewards randomly shuffled or negated; if average benchmark performance still keeps the reported gain, the claimed mechanism is not what is doing the work.","supporting_citations":[],"review_version":1}