{"id":"b9c3cb5a-6810-499b-987f-4397d5ef0890","arxiv_id":"2505.12762","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"IDEAL tunes SFT data mixture proportions per domain with influence-function gradients, claiming about 7% average benchmark improvement over uniform mixing.","lead":"The paper proposes IDEAL, an iterative method that adjusts how many training examples come from each capability domain during supervised fine-tuning, using gradient and Hessian information to reweight the mixture. It matters because multi-task SFT data proportions are a practical bottleneck, but the reported gains are computed by optimizing against the evaluation benchmarks themselves.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training sets for math and reasoning are regenerated from the evaluation benchmarks themselves, and Dref is never shown disjoint from them; the reported 7% gain may be test-set optimization rather than generalization.","rationale":"The reader's weakest assumption was the optimality requirement for Lemma 1. I agree that is a serious flaw: Appendix C explicitly concedes that models are not trained to optimality, so Eq. (5) is not the true influence and Algorithm 1 is heuristic. However, the most load-bearing concern for the paper's headline claim is the unresolved overlap between the data used to drive optimization and the data used to measure success. The paper's own Appendix A shows the math and reasoning training sets are generated from the official GSM8K and BBH datasets—the same benchmarks reported in Table 1. If the reference set Dref is also drawn from these benchmarks, which the paper neither states nor rules out, then the bi-level objective in Eq. (2) is essentially minimizing loss on a subset of the test set. The resulting ~7% average gain is then a measurement of test-set fitting, not of improved multi-capability alignment. This concern is more fundamental than the optimality gap: the optimality gap challenges the derivation of the update rule, whereas the contamination challenge invalidates the empirical verification itself. Section 5.2's volume-matching control does not address this, since it only shows IDEAL beats volume-matched DoReMi/DOGE, not that the evaluation is unbiased. The extended experiment in Sec. 4.3 adds MATH/ARC_C as OOD benchmarks and shows a smaller gain, which slightly mitigates the concern but does not resolve it, because the training sets are still derived from GSM8K/BBH and the Dref construction remains unspecified. A clean evaluation with fully disjoint training, reference, and test data is required to establish the claim.","tokens_in":17072,"tokens_out":7417,"duration_ms":70925,"concrete_test":"Perform a data-lineage audit between the released/described training sets and the evaluation benchmarks: (1) Check whether the 'official GSM8K dataset' and 'official BBH dataset' used to build the math and reasoning training sets include the exact questions used in the GSM8K and BBH evaluation splits; (2) determine how Dref was constructed and whether it samples from those same splits. Concretely, run an n-gram overlap analysis (e.g., 8-gram) between the training set entries (after GPT-4 regeneration) and the test prompts of GSM8K/BBH/IFEval; if any evaluation question appears in training or in Dref, the 7% improvement is attributable to memorization/test-set fitting rather than IDEAL's distribution optimization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that IDEAL's optimized data distribution, not extra data volume, causes the ~7% multi-task improvement (Abstract, Sec. 4.2)—requires the evaluation to be unbiased. That condition is not established. Appendix A states that the mathematics training set is built by taking 'the official GSM8K dataset' and using GPT-4 to regenerate chain-of-thought solutions, and the reasoning training set is built from 'the official BBH dataset' with GPT-4-regenerated answers. Thus the model is trained on data derived from the exact benchmarks on which it is evaluated. Section 3.1 introduces a 'small, independent reference dataset Dref' that is 'strictly excluded from the training process,' but the paper never specifies Dref's source or proves it is disjoint from GSM8K/BBH/IFEval. Eq. (2) optimizes beta by minimizing L(Dref, theta*). If Dref is a held-out portion of these same benchmarks—or if the training sets themselves contain test questions—then IDEAL is directly fitting the evaluation metric, and the reported gains are not evidence of generalization. The optimality assumption in Lemma 1 (Eq. 10) is a real theoretical gap, but it is secondary: even a correct influence calculation would not make a contaminated evaluation valid.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IDEAL, an iterative framework that adjusts per-domain data volumes in multi-capability supervised fine-tuning (SFT). A domain-level weight beta controls upsampling or downsampling of each domain's training data, and an influence-function-style derivation (Lemma 1) is used to compute the gradient of a reference loss L(Dref, theta*) with respect to beta. The Hessian is approximated with K-FAC, and the resulting beta updates are applied over two iterations. Experiments on Llama3.1-8B cover mathematics, coding, reasoning, instruction-following, and (in an extended setting) trustworthiness, claiming an average improvement of about 7% over uniform data mixing and about 2.1% in the extended five-domain setting.","tokens_in":17303,"tokens_out":3983,"duration_ms":45224,"significance":"If the central claim were valid, IDEAL would be a useful contribution: a principled, gradient-guided alternative to heuristic data mixing for SFT, with a clearly stated objective (minimizing reference loss) and a scalable Hessian approximation. The paper includes a formal derivation, an explicit limitation appendix, and sensitivity analyses, and it reports repeated runs. However, the significance is undermined by two load-bearing problems. First, the training data for mathematics and reasoning are generated from the official GSM8K and BBH datasets, which are also the evaluation benchmarks, and the reference dataset Dref is never specified or shown to be disjoint from the training/evaluation data; the reported gains may therefore reflect test-set optimization rather than generalization. Second, Lemma 1's derivation assumes the model is trained to optimality, while the experiments use 1 or 3 epochs and Appendix C concedes the resulting sub-optimality; the computed influence gradient is thus not the true influence of beta on the reference loss. These issues are not local presentation defects: they affect the validity of the main empirical and theoretical claims.","major_comments":[{"comment":"The evaluation protocol is compromised by training/evaluation overlap. Appendix A states that the mathematics training set is built by starting with 'the official GSM8K dataset' and using GPT-4 to generate chain-of-thought solutions, and that the reasoning training set is built by using 'the official BBH dataset as a foundation' with GPT-4-regenerated answers. Section 4.1 lists GSM8K and BBH as the evaluation benchmarks for mathematics and reasoning. The paper does not report a train/test split for these datasets, nor does it describe any deduplication between the training set, Dref, and the test sets. Section 3.1 introduces Dref as a 'small, independent reference dataset' that is 'strictly excluded from the training process,' but its source is never given and its disjointness from GSM8K, BBH, IFEval, and the derived training sets is never established. Since Eq. (2) optimizes beta by minimizing L(Dref, theta*), and the reported improvements in Table 1 are computed on those same benchmarks, the claimed ~7% improvement cannot be interpreted as evidence of generalization. This is a load-bearing flaw in the central empirical claim.","section":"Appendix A, Section 4.1, Algorithm 1"},{"comment":"The derivation of the influence gradient in Eq. (5) relies on Eq. (10), which assumes that theta* is a local optimum of the weighted training objective in Eq. (1), so that the implicit function theorem can be applied. The experimental protocol, however, trains all models for exactly 1 or 3 epochs (Section 4.2), which is far from convergence, and the number of epochs is a fixed experimental choice, not a convergence criterion. Appendix C explicitly concedes that 'models are typically not trained to reach the globally optimal parameters' and that the gradients and Hessian-related calculations 'may not accurately represent the true behavior of the model at its optimal state.' Consequently, the quantity computed in Eq. (5) is not the true derivative of L(Dref, theta*) with respect to beta, and the updates in Algorithm 1 are not justified by Lemma 1 as written. This is not a minor approximation: the beta update rule is the core mechanism of the method, and its validity under the actual training regime is not established.","section":"Lemma 1, Appendix B.1, Appendix C, Section 4.2"},{"comment":"The method's key hyperparameter m is selected on the same evaluation benchmarks used to report the headline improvement. Section 5.1 compares m in {0.1, 0.15, 0.3} using the D1(IDEAL) setting of Section 4.2, whose performance is measured on GSM8K, HumanEval, BBH, and IFEval -- the same benchmarks that produce the ~7% claim in the abstract. Choosing m=0.15 because it 'achieves the highest average performance' is a form of test-set model selection. The introduction states that the framework 'operates without costly hyperparameter sweeps,' but the method has several free parameters (m, sigma, the number of iterations T, the K-FAC layer-selection threshold, and the damping lambda), and at least m is tuned on the evaluation benchmarks. This compounds the circularity identified above and further weakens the claim that the reported gains are attributable to the optimized data distribution rather than to selection on the test set.","section":"Section 5.1, Eq. (9), Introduction"}],"minor_comments":[{"comment":"Line 2 says 'Train M0 on Dtr,t until optimal,' but the experiments in Section 4.2 train for a fixed 1 or 3 epochs; the pseudocode should state the actual stopping condition used in the experiments, or the experiments should be reconciled with the convergence assumption in Lemma 1.","section":"Algorithm 1, Line 2"},{"comment":"The 'Specific' rows report averages that include zero-size datasets for the other three domains, so the Overall column for those rows is an average over the single nonzero domain's score and three zero scores; this presentation makes the single-task baselines look much weaker than they are and should be clarified with a footnote or a different aggregation.","section":"Table 1, 'Specific' rows"},{"comment":"The statement that a nonconvex problem 'can be locally approximated by a strictly convex optimization problem' near a local minimum is handwaving; a strict local minimum only guarantees positive semidefiniteness of the Hessian, not invertibility, and the damping argument is introduced without tying it to the actual loss used in training.","section":"Appendix B.2"},{"comment":"The definition of Lambda_ii as the variance of the projected pseudo-gradient is not clearly connected to the eigendecompositions in Eq. (7); please spell out how Lambda relates to the eigenvalues of X_l and Delta_l, since the current notation is ambiguous.","section":"Eq. (8)"}],"recommendation":"reject","confidential_remarks":"The evaluation contamination is severe enough that the headline empirical claim cannot be trusted, and the theoretical derivation does not apply to the actual training regime. A revision would need a completely new experimental protocol with training data demonstrably disjoint from the evaluation benchmarks, a specified and disjoint Dref, and either convergence-based training or a rigorously justified finite-horizon variant of the influence calculation. Those are substantial changes rather than local fixes, so I recommend rejection. I do not question the authors' intent; the issues are methodological."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a plausible incremental extension of influence functions to domain-level data mixing for SFT, but its headline result—the ~7% average gain—is not trustworthy as reported. The math and reasoning training sets are built from the same benchmarks used for evaluation (GSM8K and BBH, with GPT-4-regenerated answers), and the reference set Dref is never shown to be disjoint from those test sets. So the optimized proportions are at least partially fitting the evaluation metric, not predicting generalization.\n\nWhat is actually new: applying influence functions at the domain level rather than sample level, with an iterative normalized update and K-FAC approximation to make the Hessian inversion tractable. That is a legitimate contribution, modest but useful. The paper is clearly written, the experiments are extensive, and the limitations section honestly concedes the sub-optimality issue in training. The K-FAC details are standard but well-explained.\n\nThe soft spots are serious. First, the contamination issue is load-bearing. Even if the influence calculation were exact, training on regenerated versions of the eval questions means the eval scores are inflated. Second, the theory assumes the model reaches the optimum of the weighted objective; experiments use 1–3 epochs, far from convergence. The appendix admits this, which means the gradient in Eq. (5) is at best a heuristic. That gap is real but secondary—the evaluation contamination alone invalidates the empirical claim. There are also a few free parameters (m, sigma, T, damping) tuned on the same benchmarks, adding to the circularity.\n\nWho gets value: people working on automatic data mixture optimization for SFT, as a cautionary example and a starting point. The method could be salvageable with clean train/validation/test separation and code release. As presented, the evidence does not support the headline.\n\nI'd send it to peer review, not desk reject—it’s an honest and technically competent attempt, and a good referee can force the authors to re-run with proper separation. But I would not cite it in its current form.","headline":"A reasonable domain-level influence-function idea, undercut by training on the evaluation benchmarks; the reported 7% gain is contaminated.","tokens_in":17892,"tokens_out":1916,"would_cite":false,"duration_ms":20140,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Per-domain data reweighting guided by second-order influence lifts multi-capability SFT scores by roughly 7% over uniform mixing, reaching its best balance in two iterations.","keywords":["data mixture optimization","supervised fine-tuning","multi-capability alignment","influence function","bi-level optimization","K-FAC","data reweighting","LLM alignment"],"falsifier":"Take a small model and the same four domains, and train it to genuine convergence instead of one epoch: compute the exact influence of Eq. (5) and the K-FAC approximation, then compare both with the best mixture found by exhaustive search over domain shares. If the exact influence points away from the empirical optimum, the lemma's optimality premise is the failure point; if the directions agree but two IDEAL updates still do not beat uniform mixing, the step size or stopping rule deserves the blame.","tokens_in":16830,"feed_emoji":"⚖️","tokens_out":7692,"duration_ms":75535,"temperature":0.7,"pith_summary":"This paper claims that the composition of a multi-domain supervised fine-tuning dataset—how many examples each capability contributes—can be optimized by a short, gradient-guided iterative procedure instead of uniform mixing or hand-tuning. The procedure, called IDEAL, measures the influence of each domain's data volume on a small reference loss, then upsamples or downsamples domains in the direction that most reduces that loss, repeating for two rounds. In experiments across mathematics, coding, reasoning, and instruction-following, the reweighted mixtures beat uniform data blending by roughly 7% on average, and the paper argues the gain comes from the mixture distribution rather than from adding data. The practical point is that SFT data preparation currently relies on trial and error, and the paper proposes a principled, cheaply computable update to replace that guesswork.","feed_headline":"7% multi-task gain from influence-guided SFT data mixing","feed_subtitle":"Second-order influence rebalances per-domain data volumes, lifting multi-task scores in two rounds.","key_machinery":"The load-bearing object is the influence of a per-domain volume parameter $\\beta$ on the reference loss, $\\alpha_j = -(\\partial L(D_{\\mathrm{ref}},\\theta^*)/\\partial \\theta^*)^\\top \\nabla^2 L(D_{\\mathrm{tr}},\\theta^*)^{-1} \\nabla L(D_j,\\theta^*)$, derived by differentiating the first-order stationarity condition of the weighted training objective through the implicit function theorem. The update rule $\\beta = -\\gamma \\odot \\alpha$ converts this influence into concrete upsample and downsample decisions, with a dynamic scaling vector capping the step size so that each round makes a moderate, stable adjustment. A K-FAC Kronecker-factorized, eigendecomposed approximation of the Hessian is what makes the inverse-Hessian-vector product tractable at the scale of an 8-billion-parameter model. Together these pieces turn dataset composition into a quantity that can be optimized by gradient descent rather than guessed.","core_discovery":"On its own terms, the paper's central claim is that the optimal SFT data mixture is a differentiable object: the best per-domain data volume is found by differentiating the trained model's reference loss with respect to a volume-controlling parameter $\\beta$, evaluated at the optimum of the weighted training objective. Lemma 1 gives the derivative of the optimal parameters with respect to each $\\beta_j$ as $-H^{-1}\\nabla L(D_j,\\theta^*)$, and substituting this into the chain rule yields the influence of each domain's share on the reference loss. A K-FAC block-diagonal approximation makes the inverse Hessian computable for an 8-billion-parameter model, and a scaling vector turns the influence into a concrete update $D_i \\leftarrow (1+\\beta_i)D_i$. Two such updates suffice in the reported settings: IDEAL reaches an average of 57.9–59.2 across the four benchmarks versus 54.8–55.4 for joint SFT, and the controlled comparison that equalizes data volume shows the advantage comes from the distribution rather than the amount of data.","pith_inferences":["If the update direction from Lemma 1 is roughly correct even when training stops far short of convergence (as the results hint, though the derivation assumes optimality), the same two-step influence update should transfer to other mixture-tuning settings where volumes are currently hand-set, such as preference-data blending for alignment or continual fine-tuning.","The Hessian approximation is replaceable: a damped diagonal or low-rank curvature estimate would probe whether the update direction, rather than the K-FAC precision, does the work, and would make the method cheaper still.","A small-scale replica where a model can be trained to genuine convergence could separate the lemma from the approximations: compute the exact inverse-Hessian influence and check whether it points toward the empirically best mixture found by exhaustive search.","The 'optimal in two iterations' result is tied to the step-size cap $m=0.15$; running more iterations or an adaptive $m$ would reveal whether two is a genuine fixed point or just the first two safe steps."],"forward_implications":["Uniform or hand-set data mixtures are a costly default: the influence-guided update raises average multi-task scores by about 7% over uniform blending with no new data and no hyperparameter sweeps.","Two iterations of reweighting are sufficient in the tested settings, and the method is stable enough that the second-round model substantially improves coding performance without sacrificing the other domains.","Data volume is not the lever: the extended experiment shows that the full 66k-sample joint SFT does not beat the balanced 25k-sample mixture, and equalizing volume across reweighting baselines leaves IDEAL ahead.","The mechanism carries over to a five-domain, eight-benchmark setup with still-positive average gains (about 2.1%), indicating the effect is not an artifact of the four-domain configuration.","Because each pass only needs the trained model, a reference set, and K-FAC Hessian computations, the procedure can replace manual reweighting in routine SFT pipelines."],"supporting_citations":[{"why":"Supplies the K-FAC Kronecker-factored curvature approximation that makes the inverse-Hessian computation in Eq. (5) tractable.","marker":"[40]"},{"why":"Provides the DoReMi group-DRO reweighting baseline that IDEAL is compared against and that it extends from pretraining to SFT.","marker":"[62]"},{"why":"Provides the DOGE gradient-discrepancy reweighting baseline used in the iterative data evolution chain comparison.","marker":"[16]"},{"why":"Gives the finding that repeating data up to four times approximates new data, which justifies the $\\beta$-driven upsampling mechanism.","marker":"[43]"},{"why":"Establishes the influence-function formalism that underlies Eqs. (3)–(5).","marker":"[32]"},{"why":"Presents the data mixing laws approach that IDEAL contrasts with as costly and pretraining-focused.","marker":"[64]"},{"why":"Supplies the Kronecker-factored influence computation for large models that IDEAL adapts for efficiency.","marker":"[66]"}],"fun_headline_variants":["Gradient-guided data rebalancing lifts multi-task scores 7%","Influence-based data mixing for SFT yields 7% across tasks","Data equilibrium: adaptive SFT mixes gain 7% multi-task","Optimal per-domain data volumes via gradient: 7% gain","Two-round influence tuning of SFT data: 7% improvement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The influence calculation assumes the fine-tuned model has reached the optimum of its weighted training objective, but the experiments train for only one or three epochs—far short of that—so the gradient that drives each reweighting step may not be the true influence of a domain's data share on the reference loss.","fun_headline_variants_meta":{"raw":{"variants":["Gradient-guided data rebalancing lifts multi-task scores 7%","Influence-based data mixing for SFT yields 7% across tasks","Data equilibrium: adaptive SFT mixes gain 7% multi-task","Optimal per-domain data volumes via gradient: 7% gain","Two-round influence tuning of SFT data: 7% improvement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000338,"raw_usage":{"total_tokens":1901,"prompt_tokens":1013,"completion_tokens":888,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":795}},"tokens_in":629,"tokens_out":888,"duration_ms":9171,"temperature":1.0,"reasoning_tokens":795,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:27:24.498076+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a small model and the same four domains, and train it to genuine convergence instead of one epoch: compute the exact influence of Eq. (5) and the K-FAC approximation, then compare both with the best mixture found by exhaustive search over domain shares. If the exact influence points away from the empirical optimum, the lemma's optimality premise is the failure point; if the directions agree but two IDEAL updates still do not beat uniform mixing, the step size or stopping rule deserves the blame.","supporting_citations":[{"cited_title":"Martens and R","cited_arxiv_id":null,"evidence_quote":"Supplies the K-FAC Kronecker-factored curvature approximation that makes the inverse-Hessian computation in Eq. (5) tractable."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the DoReMi group-DRO reweighting baseline that IDEAL is compared against and that it extends from pretraining to SFT."},{"cited_title":"Muennighoff, A","cited_arxiv_id":null,"evidence_quote":"Gives the finding that repeating data up to four times approximates new data, which justifies the $\\beta$-driven upsampling mechanism."}],"review_version":1}