{"id":"9eb665ff-e1b4-453d-9ccf-07d91ac94fdd","arxiv_id":"2504.13227","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"DIDS dynamically reweights training domains using gradient clustering and a Fisher Information-guided KL metric, and reports matching or better LLM benchmark scores with 10% of the data.","lead":"DIDS is a method for choosing how much of each data domain to feed a large language model during training, using gradient patterns and a Fisher Information metric to decide which domains matter most for target tasks. The authors report that training on 10% of the data with DIDS matches or beats training on the full dataset across nine benchmark tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The FIM-guided impact in Eq. (12) is a positive quadratic form in (∇ℓ_Sj−∇ℓ_Di), so it is largest when domain and task gradients oppose; Eq. (16) multiplies this by positive loss improvement and upweights those domains, the opposite of the prose's alignment story.","rationale":"The reader's weakest_assumption identifies exactly the same concern: the FIM metric as written rewards divergence between domain and task gradients, while the prose claims alignment-based impact, and no derivation connects Eq. (16) to the Eq. (4) objective. I agree that this is the most load-bearing issue because it determines whether the proposed mechanism, not just the empirical recipe, is coherent. I also agree that the concern is addressable rather than automatically fatal: the paper provides a code link, ablations showing each component contributes, multiple model-dataset combinations, and an OOD check, so the empirical claim might survive with a corrected sign convention or a clarified derivation. However, as written, the update rule cannot be justified from the stated objective, and the positive loss-improvement term is task-level rather than domain-specific, so it cannot attribute improvement to any particular domain. Other weaknesses, such as the absence of error bars and the possible overlap between the observable validation subset and the evaluation set, are real but secondary; the sign/derivation issue directly targets the central mechanism. The reader's CONDITIONAL verdict is therefore appropriate, and my analysis does not move it. The concrete test—running the sign-flipped variant on the same setup—would settle whether the reported gains come from the stated mechanism, from the opposite mechanism, or from an undocumented implementation detail.","tokens_in":22791,"tokens_out":5279,"duration_ms":60154,"concrete_test":"Run the public repository's DIDS training on Llama-3.1-8B with Tulu-3 and the 100k sample budget, comparing three variants: (A) the update exactly as written in Eq. (16) and Algorithm 1; (B) the same update with I replaced by -I, which upweights domains whose updates align with the task update; and (C) a gradient-alignment baseline such as DGA. If variant (B) matches or exceeds the reported 62.3 multi-task / 63.7 single-task averages, then the sign and mechanism in the paper are not the source of the gains. If variant (A) alone reproduces the numbers, the implemented metric must differ from the text (e.g., an undocumented sign convention or normalization), and the paper must state that explicitly. A cheaper complementary check is to independently re-derive Eq. (16) from the Eq.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism is internally inconsistent as written. In Section 4.2, Eq. (12) sets I(D_i,S_j) = (1/2)(∇ℓ_Sj−∇ℓ_Di)^T F (∇ℓ_Sj−∇ℓ_Di), with F the diagonal Fisher information matrix, which is positive semidefinite. Consequently I >= 0, I = 0 only when ∇ℓ_Di equals ∇ℓ_Sj, and I grows monotonically as the domain update and the task update move in opposite directions. The prose in Sections 4.2 and 4.3 claims the metric measures how domain updates affect the model's predictive distribution on downstream tasks, and that DIDS upweights impactful domains; a natural reading is that a useful domain should move the model toward the task update, which would make I small, not large. Algorithm 1 (line 30) then computes U(D_i) = Σ_j I(D_i,S_j)·(ΔL(S_j)+L_p(S_j)) / p_{t-1,i}, where ΔL(S_j)=L_{t-1}(S_j)-L_t(S_j) and L_p(S_j) are nonnegative when task loss is improving and future loss is predicted to be lower. Softmax over U therefore assigns the largest sampling probability to domains whose gradient updates are most different from the task gradient. No derivation from the objective in Eq. (4) is supplied; in particular, task-level loss improvement ΔL(S_j) is the same scalar for every domain, so it cannot attribute the observed improvement to domain D_i. The reported empirical gains (62.3 multi-task, 63.7 single-task) could still be real, but if the implementation follows the text, the stated mechanism would not explain those gains. This is the load-bearing concern because it targets the proposed algorithm's own update rule rather than external baselines or dataset choices.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DIDS, a domain-level data sampling method for LLM post-training. It has three components: (1) gradient-based domain repartitioning using a small proxy model, top-k gradient retention, and Johnson-Lindenstrauss random projection followed by k-means clustering; (2) a Fisher Information Matrix (FIM) guided impact metric, obtained as a second-order Taylor approximation of the KL divergence between model predictions after a domain update and after a downstream-task update; and (3) a periodic sampling-probability update that combines the FIM impact scores with current loss improvement and a predicted future loss improvement, smoothed by an exponential moving average. Experiments on Llama-3.1-8B and Mixtral-7B with Tulu-3 and OpenHermes-2.5 report that 100k samples selected by DIDS outperform full-data training and several baselines on a nine-task suite, with additional ablations, OOD evaluations, and computational cost analyses in the appendices.","tokens_in":23126,"tokens_out":5324,"duration_ms":58178,"significance":"If the reported results hold, DIDS would be a practical data-efficiency contribution: it reduces post-training data by roughly an order of magnitude while matching or exceeding full-data performance, and the public code would support reproducibility. The paper also contains a genuine theoretical fragment, the KL/FIM Taylor expansion in Section 4.2, and a commendably extensive experimental appendix covering multiple models, datasets, ablations, OOD generalization, and cost breakdowns. However, the central mechanism as written has an internal inconsistency that must be resolved before the empirical claims can be attributed to the proposed algorithm.","major_comments":[{"comment":"The core mechanism is internally inconsistent as written. Eq. (12) defines I(D_i,S_j) = (1/2)(∇ℓ_Sj − ∇ℓ_Di)^T F (∇ℓ_Sj − ∇ℓ_Di). Since F is positive semidefinite, I is zero when the domain gradient equals the task gradient and grows as the two gradients diverge or oppose. Eq. (16) then multiplies this quantity by (l_c + l_p), both of which are nonnegative when the task loss is improving, and adds the product to the sampling weight. The update therefore upweights domains whose gradient updates are most different from the task update, which is the opposite of the alignment story in the prose. No derivation is given showing that this update optimizes the Eq. (4) bi-level objective, and l_c = ΔL(S_j) is a task-level scalar that is identical for every domain, so it cannot attribute the observed task improvement to a specific domain. If the implementation follows the text, the reported gains cannot be explained by the stated mechanism; the authors need to either correct the sign/direction of the metric, replace it with an alignment-based measure, or supply a derivation connecting Eq. (16) to Eq. (4).","section":"§4.2–4.3, Eqs. (12) and (16)"},{"comment":"There is a potential evaluation circularity that needs to be resolved. The downstream tasks S_j are used in Algorithm 1 to compute per-domain gradients, the FIM, and the loss trajectories that drive the sampling update, and the same nine tasks are then reported as the evaluation scores in Table 1. Section 5.1.2 mentions partitioning a validation subset for the baselines, but it does not clearly state whether DIDS itself also restricts its observable task data to a separate held-out subset. If the reweighting uses any of the evaluation data or the same task distributions as the final evaluation, the reported gains partly reflect fitting to the evaluation suite. The OOD results in Appendix J are helpful, but the main claim in Section 5.2 still needs a clean separation between what is observed for sampling and what is reported as performance.","section":"§5.1.2 and Algorithm 1"},{"comment":"The paper claims 'theoretical guarantees' and 'theoretical foundations' for the FIM-guided impact metric, but the only derivation is the Taylor approximation of KL divergence in Section 4.2 and Appendix E. There is no theorem showing that the sampling update in Eq. (16) optimizes the stated objective in Eq. (4), nor any convergence or optimality guarantee for the resulting mixture. The word 'guarantees' should be reserved for what is actually proven, or the claims should be toned down to describe the metric as a heuristic with a Taylor-based justification.","section":"Abstract and §4.3"}],"minor_comments":[{"comment":"The sample counts and percentages in Table 4 do not sum to the claimed totals: the listed components sum to 889,344 samples rather than 939,344, and the percentages sum to roughly 94.5% rather than 100%. The table should be reconciled with the dataset manifest.","section":"Table 4"},{"comment":"The notation for the Fisher matrix is inconsistent: Eq. (11) uses the elementwise product E[∇log p ⊙ ∇log p], the figure uses a similar form, and Eq. (17) in Appendix E uses the outer product ∇log p (∇log p)^T. The diagonal approximation should be stated explicitly and used consistently throughout.","section":"§4.2, Eqs. (10)–(12) and Appendix E"},{"comment":"Appendix E.2 says the KL approximation is 'positive definite and symmetric (when properly scaled).' With a diagonal F the quadratic form 1/2 Δ^T F Δ is positive semidefinite, not necessarily positive definite, since F can have zero diagonal entries.","section":"Appendix E.2"},{"comment":"The sentence describing the 5-update setting says DIDS achieves 58.2, 'comparable to the random sampling baseline at 58.9,' which is a roundabout way of saying it underperforms random sampling by 0.7 points; the text should state this directly so readers can assess the sensitivity of the method.","section":"§5.5.1 and Figure 2a"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the sign/direction inconsistency between the FIM metric in Eq. (12) and the sampling update in Eq. (16). If the released code follows the text, the empirical gains are likely being driven by something other than the stated mechanism; if the code uses the opposite convention, the paper must state this and correct the equations. I would also want the authors to clearly document the train/validation split for the downstream tasks used in reweighting versus evaluation before the results can be taken at face value. The experimental breadth is a strength, and the issues appear fixable, so I do not recommend rejection, but the revision needs to be substantive rather than cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's what you should know about this one. The paper reports a clear empirical win—DIDS, a dynamic domain-sampling method, beats full-data training on 100k samples across four model/dataset combos—but the update rule in the paper's equations appears to do the opposite of what the prose claims. Eq. (12) defines domain impact as a positive quadratic form in (∇ℓ_Sj − ∇ℓ_Di), so it is zero when the domain and task gradients match and grows as they diverge. Eq. (16) then multiplies this impact by loss improvement (a positive scalar when things are going well) and upweights the domain. So the algorithm, as written, boosts exactly the domains that push the model away from the task update. No derivation connects this to the bi-level objective in Eq. (4). That is not a minor typo; it is the load-bearing mechanism.\n\nWhat is genuinely new and worth credit: the gradient-based domain repartitioning. Instead of using hand-defined or BERT-semantic clusters, they project per-sample gradients to a low dimension (top-k layer gradients, random projection) and k-means them, which is a sensible way to define trainable units that share learning behavior. The FIM-weighted KL quadratic is a textbook local approximation, but using it as a domain-impact signal in online mixing is new, and the derivation itself is correct. The experimental effort is also unusually thorough: four model/dataset combinations, multi-task and single-task optimization, ablations, update-frequency and proxy-size sweeps, an OOD generalization check, and a static-ratio grid. If the numbers survive scrutiny, a 10% subset beating full data is a real result.\n\nSoft spots beyond the equation inconsistency: no seeds or error bars anywhere, which matters when the headline advantage is about 2 points over the best baseline; the observation set used to drive the reweighting is taken from the same downstream task suite that is reported, and the paper never explicitly says it is disjoint from the evaluation set; the code is linked but unverified. These are addressable, but the sign problem is the one that must be fixed first—or at least explained. If the implementation actually uses a negative I or some other inversion, the text needs to match it.\n\nBottom line: this is a paper for the data-mixing community, and it deserves a serious referee because the empirical claims are important and the flaw is fixable. But as written, I would not accept it: I would send it back asking for a corrected derivation, an explicit statement on evaluation/validation disjointness, and a basic variance estimate. If those come back clean, this could be a useful contribution.\n\nRecommendation: engage, but send it back with major revision rather than accepting the headline result on faith.","headline":"Empirically broad, but the central update rule as written appears to reward domain gradients that oppose the task gradient—the opposite of the paper's narrative; fix that and this could be worth publishing.","tokens_in":23791,"tokens_out":5197,"would_cite":false,"duration_ms":54884,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Domain-impact-aware data sampling outperforms full-data LLM training using only 100k examples, according to experiments on Llama-3.1-8B and nine downstream tasks.","keywords":["domain-level data sampling","data mixing","gradient clustering","Fisher information matrix","large language model training","downstream task impact","loss trajectory weighting","training data efficiency"],"falsifier":"Inspect the released code: for one domain-task pair where training on the domain clearly improves task loss, compute $\\Delta=\\nabla\\ell_{S_j}-\\nabla\\ell_{D_i}$ and the resulting sampling-weight change; if the weight rises when $\\Delta^\\top F \\Delta$ is large, the mechanism is inverted and the gains would need another explanation. Alternatively, run DIDS with the impact term's sign flipped and compare average scores.","tokens_in":22531,"feed_emoji":"🎯","tokens_out":8110,"duration_ms":77816,"temperature":0.7,"pith_summary":"This paper argues that the biggest untapped lever in LLM post-training is not adding data but deciding which data to sample, and proposes Domain Impact-aware Data Sampling (DIDS) to make that choice online. DIDS first repartitions the training corpus by clustering proxy-model gradients, so each \"domain\" is a set of examples with similar learning effects rather than similar topics. It then measures each domain's impact on each downstream task with a Fisher Information Matrix (FIM) guided quadratic form $I(D_i,S_j) = \\tfrac12(\\nabla\\ell_{S_j}-\\nabla\\ell_{D_i})^\\top F (\\nabla\\ell_{S_j}-\\nabla\\ell_{D_i})$, and updates sampling probabilities by combining that impact with loss-improvement trajectories. On Llama-3.1-8B and nine tasks, DIDS reports average scores of 62.3 (multi-task) and 63.7 (single-task) using only 100k samples, versus 61.2 for training on the full 929k corpus, with roughly 3.4% higher average performance claimed. The point of interest is practical: if it holds, downstream accuracy can be improved while spending about 10% of the data and a small compute overhead.","feed_headline":"Impact sampling lets 100k examples beat full-data LLM training","feed_subtitle":"An impact-aware mixing rule reports 3.4% higher average scores on nine tasks while using about one-tenth of the data.","key_machinery":"The load-bearing object is the FIM-guided domain impact metric, a quadratic form $I(D_i,S_j) = \\tfrac12(\\nabla\\ell_{S_j}-\\nabla\\ell_{D_i})^\\top F (\\nabla\\ell_{S_j}-\\nabla\\ell_{D_i})$ over the difference between a training-domain gradient and a downstream-task gradient, with $F$ the diagonal Fisher Information Matrix $F \\approx \\mathbb{E}[\\nabla\\log p(y|\\theta)\\odot \\nabla\\log p(y|\\theta)]$. It is the second-order Taylor approximation of the KL divergence between the model's predictive distribution after a domain update and after a task update. This metric feeds the sampling update in Eq. (16), where impact is multiplied by per-task loss improvements and potential, divided by previous sampling weight, then smoothed by EMA; the accompanying gradient-clustering step supplies the \"domains\" that make the metric meaningful.","core_discovery":"The paper's central claim is that a dynamically updated domain sampling distribution, computed from gradient-based clusters and an FIM-guided measure of how each cluster shifts the model's output distribution on target tasks, optimizes the bi-level objective in Eq. (4) better than uniform, random, or gradient-alignment baselines, and even better than full-data training at one-tenth the budget. Concretely, DIDS reports 62.3 multi-task and 63.7 single-task averages on Llama-3.1-8B across BBH, BoolQ, GSM8K, MathQA, IFEval, MMLU, PIQA, PubMedQA, and TruthfulQA, against 61.2 for full-data training and 60.2 for the best multi-task baseline. The result extends to Mixtral-7B and the OpenHermes-2.5 dataset in the appendix, where DIDS again matches or exceeds full-data training with 100k samples.","pith_inferences":["Beyond the paper: the sign of the impact term deserves a direct ablation, because $I(D_i,S_j)$ vanishes when the domain and task gradients agree and grows when they oppose; as written, Equation (16) would reward opposing updates, so the reported gains may be carried by the clustering, loss-trajectory, or smoothing components rather than by the FIM term.","Beyond the paper: the diagonal-FIM approximation and the exponential-decay loss model are both local assumptions that the appendices acknowledge; the method's transfer to out-of-distribution tasks is consistent with the sampling acting as an implicit regularizer, which could be tested by comparing validation-loss trajectories.","Beyond the paper: because the impact metric is a local quadratic form, it should become less reliable exactly when updates are large; a practical extension would be to recompute the FIM on the updated model more often or to blend in direct task-accuracy estimates."],"forward_implications":["If the central claim holds, a 100k-sample impact-weighted subset produces higher average downstream accuracy than the full 929k-corpus training run, so post-training budgets can drop by roughly an order of magnitude.","The appendix replicates the pattern on Mixtral-7B and on the OpenHermes-2.5 dataset, which suggests the sampling rule transfers across model families and data distributions rather than memorizing one benchmark suite.","The gradient-based domain repartition places algebra and code together by learning effect, and replacing that repartition with BERT semantic clustering costs over a point of average accuracy; this makes behavioral clustering a load-bearing part of the method.","Dynamic updates that combine impact with loss trajectories beat every static mixing ratio in the grid-search comparison, so the optimal mixture is not constant over training.","DIDS retains competitive scores on four unseen out-of-distribution tasks (WMT16, TLDR, ARC-Challenge, MBPP), indicating that the optimized sampling distribution does not merely overfit the nine observed evaluation tasks."],"supporting_citations":[{"why":"Provides the DGA baseline and the gradient-alignment definition in Eq. (6) that DIDS replaces with the FIM-guided metric.","marker":"Fan et al., 2024a"},{"why":"Doge, the domain-reweighting baseline whose gradient-alignment objective DIDS extends and outperforms.","marker":"Fan et al., 2024b"},{"why":"Doremi, the group-DRO baseline that motivates excess-loss weighting and is compared against in the main experiments.","marker":"Xie et al., 2024"},{"why":"Velocitune, the velocity-based online reweighting baseline that DIDS's loss-trajectory term is designed to improve upon.","marker":"Luo et al., 2024"},{"why":"Supplies the Tulu-3 post-training dataset of 939,344 samples from 18 sources used for all main experiments.","marker":"Lambert et al., 2024"},{"why":"Provides the Johnson-Lindenstrauss random projection method used to reduce gradient vectors for clustering.","marker":"Park et al., 2023"}],"fun_headline_variants":["DIDS sampling: 100k examples outperform full-data training","Impact-aware data sampling boosts LLM scores 3.4%","Sampling by domain impact beats full-data LLM training","DIDS: One-tenth data, 3.4% better LLM performance","Impact-guided sampling: full-data LLM training beaten"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains depend on the sampling-update formula meaning what the prose says; as written, the formula appears to upweight domains whose updates oppose the downstream task, so the implementation, not the metric, is the load-bearing premise.","fun_headline_variants_meta":{"raw":{"variants":["DIDS sampling: 100k examples outperform full-data training","Impact-aware data sampling boosts LLM scores 3.4%","Sampling by domain impact beats full-data LLM training","DIDS: One-tenth data, 3.4% better LLM performance","Impact-guided sampling: full-data LLM training beaten"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1302,"prompt_tokens":939,"completion_tokens":363,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":274}},"tokens_in":555,"tokens_out":363,"duration_ms":3860,"temperature":1.0,"reasoning_tokens":274,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:20:37.419750+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the released code: for one domain-task pair where training on the domain clearly improves task loss, compute $\\Delta=\\nabla\\ell_{S_j}-\\nabla\\ell_{D_i}$ and the resulting sampling-weight change; if the weight rises when $\\Delta^\\top F \\Delta$ is large, the mechanism is inverted and the gains would need another explanation. Alternatively, run DIDS with the impact term's sign flipped and compare average scores.","supporting_citations":[],"review_version":1}