REVIEW 3 major objections 5 minor 1 cited by
Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilities
T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A 15% subset, chosen by a normalize-then-rebalance influence algorithm, outperforms full instruction tuning across seven benchmarks.
desk verdict BIDS is a clean, likely useful balancing fix for influence-based data selection, but the marquee claim that a 15% subset beats full training is compromised by test-split leakage in the selection signal. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Attribution Matrix A, whose rows are training examples and whose columns are validation instances. BIDS standardizes each column to zero mean and unit variance, then runs a greedy loop: with selected set T, the utility of candidate t_i is Δ^(i) = max_{1≤j≤|V|} (A_{ij} − (1/|T|)Σ_{k:t_k∈T} A_{kj}), and the candidate with the highest utility is added. Because the largest component of that difference points to the validation instance where the current subset is weakest, the loop mechanically favors the most underrepresented task at every step.
What would settle it
Take the same UltraInteract pool and the same seven benchmarks, draw the 50 validation instances per task strictly from splits disjoint from the test sets, and rerun the 15% BIDS-versus-full comparison; if the macro-average advantage disappears, the current result depends on the shared test instances. Separately, a retraining probe that adds the highest-gap candidate to a fixed subset and measures the actual marginal performance change would directly test whether the componentwise-gap identity is what drives the gains.
Extended reading notes
Core claim
LESS, the influence-based selector used as the starting point, computes pairwise gradient similarities between training and validation data and then picks the training examples with the highest raw influence scores. When the tasks are diverse, this recipe is biased: columns of the influence matrix that correspond to certain tasks carry larger values by construction, so top-k selection oversamples those tasks and undersamples others, and the paper shows that even the oversampled tasks do not improve. BIDS corrects the recipe with two changes: it standardizes each validation column of the influence matrix to zero mean and unit variance, and then, instead of scoring examples once, it builds the subset greedily by comparing each candidate's influence distribution with the average influence distribution of the current subset and choosing the example whose largest componentwise gap is greatest. The reported consequence is that selected subsets become balanced in influence while remaining influential, and a 15% subset trained for four epochs surpasses full-data training in average performance.
Load-bearing premise
The load-bearing premise is that influence scores are additive in the sense that the average influence vector of the selected set is a valid baseline and the example with the largest single-column gap delivers the largest real improvement, an assumption the paper itself flags as first-order linearity; a second premise is that the 50 validation instances per task, sampled from the test split when no validation split exists, do not unfairly leak evaluation information into selection.
Editorial extensions
If this is right
- Column-wise normalization alone already improves the instance-wise max selector under 5%, 10%, and 15% budgets, and adding iterative selection raises the average across benchmarks further while narrowing cross-task performance gaps.
- A 15% subset chosen by BIDS and trained for four epochs beats both one-epoch and four-epoch full-data training in average performance, with its clearest gains on instruction-following and balanced gains on reasoning tasks.
- The same pattern holds with a second base model family, so the balancing effect is not an artifact of Llama-3-8B's influence estimates.
- Analyzing the selected subsets shows that BIDS lowers the disparity of average influence across tasks and instances, whereas top-k selection raises average influence without raising performance—evidence that high influence does not linearly add up.
Reading between the lines
- The paper does not test BIDS on other influence estimators, but the failure it identifies lives in the raw influence matrix and the top-k scoring rule, not in LESS's specific gradient computation; plugging column normalization and gap-based greedy selection into any other attribution-matrix method is a direct testable extension.
- The same scale-mismatch diagnosis likely applies to non-influence utility scores: RDS in the paper also over-selects coding tasks, so normalizing columns and selecting by largest underrepresented-gap could rebalance representation-similarity selection as well.
- The headline comparison replays the 15% subset over four epochs, so the claimed efficiency is about unique examples rather than total training tokens; comparing equal total tokens or equal compute would put the '15% beats full data' statement on a stricter footing.
- The greedy rule optimizes a first-order additivity objective; if influence interactions matter at larger budgets, BIDS's advantage should saturate or reverse somewhere above 15%, which a budget sweep would reveal.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes BIDS, an influence-based data selection algorithm for multi-task instruction tuning. Given an Attribution Matrix of pairwise influence scores between training examples and validation instances, BIDS first applies column-wise normalization to place influence scores of different validation instances (tasks) on a common scale, then iteratively selects training examples that maximize, over validation instances, the componentwise gap between the candidate's influence distribution and the average influence distribution of the already-selected set. This procedure favors training data that most increase influence on currently underrepresented tasks. Experiments on Llama-3-8B and Mistral-7B-v0.3 across seven benchmarks (coding, math, logical inference, world knowledge, instruction following) show that BIDS-selected subsets at 5%, 10%, and 15% budgets achieve higher macro-average performance than the LESS task-wise max baseline, other influence-based heuristics, random selection, and a representation-based method (RDS). A 15% BIDS subset trained for four epochs is reported to outperform full-dataset training. The paper also provides ablations and analyses (THI, AID) of the influence distribution of selected data.
Significance. If the reported results hold, BIDS would be a simple, computationally cheap, and effective method for balanced multi-capability instruction tuning, addressing a real shortcoming of current influence-based selection methods. The paper includes several strengths: a clear problem diagnosis (cross-task influence scale bias is documented in Figures 1 and 2), a lightweight algorithm whose overhead is quantified in Appendix A.7, consistent improvements over the considered baselines under equal budgets across two model families (Tables 2 and 4), and ablations that support the contribution of each design component (Table 3). The full-dataset-outperforming claim is potentially significant, but it is currently confounded by the use of test-split instances during selection, as detailed in the major comments. The core methodological contribution—balanced influence-based selection relative to other influence methods—is defensible, but the headline claim against full training needs a protocol fix before the paper's central message can be fully accepted.
major comments (3)
- [Appendix A.2 and Table 2] The selection procedure uses test-split instances as validation data. Appendix A.2 states that for each of the seven evaluation tasks, the authors "uniformly sample 50 validation instances from either the validation or the test split (when there is not a separate validation split)". For HumanEval, MBPP, GSM-Plus, MATH, BBH, MMLU, and IFEval, no separate validation split is standard, so the 350 validation instances are drawn from the exact test sets on which the final models are evaluated. This gives BIDS (and all influence-based selection methods, including LESS) access to the evaluation distribution and to the specific test items during the selection step. The full-dataset and random baselines never see test labels. Consequently, the abstract's claim that "training on a 15% subset selected by BIDS can even outperform full-dataset training" is not an apples-to-apples comparison under a standard held-out evaluation protocol. The relative comparison to LESS and other influence baselines is less affected because all influence methods share the same leakage, but the full-training comparison is confounded. To support the headline claim, the authors should either (a) use a validation set fully disjoint from the test splits (e.g., sample development examples from the training set or create a separate held-out split), or (b) explicitly reframe the result as conditional on access to the test distribution and remove or heavily qualify the full-training comparison. This is a load-bearing issue because the abstract, introduction, and conclusion all emphasize the full-dataset-outperforming result.
- [Table 2 and Section 5.2] The claim that BIDS outperforms full-dataset training rests on a fragile and nonstandard experimental comparison. In Table 2, BIDS at 15% with the default two epochs gives a macro-average of 46.1, which is lower than Full (epochs=1) at 46.9 and Full (epochs=4) at 46.7. The paper's positive result uses "BIDS (epochs=4)" (47.1), a configuration where the subset is trained for twice as many epochs as the default, while the full-dataset baselines are trained for one or four epochs without reporting any analogous search over epochs. The macro-average differences between BIDS (epochs=4), Full (epochs=1), and Full (epochs=4) are 0.2–0.4 points, which is within typical run-to-run variance for instruction tuning of 8B-scale models, yet no standard errors or multiple-seed results are reported for these key comparisons. To make the "outperforms full training" claim credible, the authors should report means and standard deviations over at least three random seeds, and should compare the best subset-trained model against a full-dataset model trained under an equally tuned epoch budget (or show that additional epochs do not improve the full-data baseline). Without this, the central headline is not statistically supported.
- [Section 6.2 and Algorithm 1] The paper's design relies on a first-order additivity assumption that is acknowledged as a limitation in Section 6.2, but the acknowledgment does not fully address the algorithm's dependence on it. The utility function in Algorithm 1 (A_ij minus the average of already-selected rows) assumes that the average influence distribution of a selected subset is a valid proxy for the subset's actual effect on validation performance. The paper itself notes that "high-influence points selected by top-k increase average influence but do not add up to performance." Since BIDS is explicitly optimizing the balance of average influence rather than the balance of task accuracy, the relationship between the two should be tested more directly. For example, the authors could compare the utility score ∆(i) of a candidate with the actual change in validation accuracy when that example is added to a small selected set. The ablations in Table 3 provide indirect evidence that the heuristic works, but given that the algorithm's entire greedy rule is built on this assumption, a direct test or a more circumspect claim about what BIDS optimizes would strengthen the paper. The already-stated limitation is appreciated, but it should be elevated to a clearly stated caveat in the Results and Conclusion.
minor comments (5)
- [Algorithm 1] When the selected set T is empty, the expression (1/|T|) sum_{k in T} A_kj involves division by zero. The paper does not specify how the first iteration is handled. Please clarify the initialization (e.g., select the first example with the largest max_j A_ij, or start with a random example, or define the average over an empty set as zero).
- [Table 2 and Section 5.2] The row "BIDS (epochs=4)" uses a different training regime than the other rows in the same table, but this is not highlighted in the methodology. Please explicitly state in Section 5.2 that the four-epoch setting is a separate protocol and explain why this is the appropriate comparison to full-dataset training.
- [Section 6.2] The phrases "The maximums of AID decrease" and "The minimums of AID increase" are awkward and could be misread. Please rephrase to clarify that these are per-task (or per-validation-instance) maxima and minima of the average influence distribution.
- [Figures 4, 5, 9, 10] The y-axis labels in the AID plots are not clearly described. Please specify the units and explain what the points represent (e.g., mean normalized influence per validation instance). It would also help to mark the per-task boundaries clearly in the figures.
- [Appendix A.2] The evaluation protocol for IFEval (temperature=0.7, median over three seeds) differs from the greedy decoding used for the other tasks. This is disclosed, but it would be good to confirm that the same protocol is used for all models and all budgets, and to state the standard deviation or range of the three IFEval runs.
Circularity Check
No significant circularity: BIDS's performance claims are empirical and not derived from its own selection objective.
full rationale
The central claims of the paper—that BIDS outperforms baselines and can beat full-dataset training—are empirical results obtained by training models on selected subsets and evaluating on standard benchmarks. BIDS operates on an externally computed influence matrix (the LESS gradient-similarity estimator) and does not fit any parameter to the reported benchmark accuracies. The iterative rule in Algorithm 1 maximizes a component-wise gap relative to the selected-set average, and the Appendix A.2 practice of sampling validation instances from test splits when no separate validation split exists is a genuine evaluation-confounding concern; however, that is a soundness/selection-bias issue, not a definitional equivalence, because the influence scores are gradient-based and the paper itself notes that high average influence does not linearly translate into performance (Section 6.2). No self-citation chain or fitted-parameter renaming is present, so no circular step can be exhibited. The AID/THI analysis in Section 6.2 partly restates the balancing objective of BIDS, but it is used as post-hoc explanation rather than as the load-bearing derivation of the headline result. Therefore the paper receives a 0 for circularity.
Assumptions & free parameters
free parameters (5)
- validation instances per task =
50
- warmup epochs for influence estimation =
4
- projection dimension =
8192
- LoRA rank and alpha =
rank 128, alpha 512
- training epochs for selected subsets =
2 normally; 4 for headline full-data comparison
assumptions (3)
- domain assumption LESS influence scores, computed as LoRA-gradient cosine similarity accumulated over warmup checkpoints, are a valid signal for training data value in instruction tuning.
- ad hoc to paper Influence contributions are first-order additive: the average influence distribution of a selected subset approximates the subset's effect on validation performance.
- domain assumption Column-wise z-score normalization makes influence scores across validation instances commensurable, and each column's influence distribution is approximately normal.
Cite this review
Pith. "Pith review of Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilities." pith.science (2026). https://pith.science/paper/H34GUXUU
@misc{pith2026250112147,
author = {Pith},
title = {Pith review of: Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilities},
year = {2026},
howpublished = {\url{https://pith.science/paper/H34GUXUU}},
note = {Machine review of arXiv:2501.12147}
}
read the original abstract
Selecting appropriate training data is crucial for instruction fine-tuning of large language models (LLMs), which aims to (1) elicit strong capabilities, and (2) achieve balanced performance across different tasks. Influence-based methods show promise in achieving (1), by estimating the contribution of each training example to the model's predictions, but often struggle with (2). Our systematic investigation reveals that this underperformance can be attributed to an inherent bias, where some tasks intrinsically have greater influence than others. As a result, data selection is often biased towards these tasks, not only hurting the model's performance on others but also, counterintuitively, harming performance on these high-influence tasks themselves. To address this, we propose BIDS, a Balanced and Influential Data Selection algorithm. BIDS first normalizes influence scores of the training data, and then iteratively chooses the training example with the highest influence on the most underrepresented task. Experiments with both Llama-3 and Mistral-v0.3 on seven benchmarks spanning five diverse capabilities show that BIDS consistently outperforms both state-of-the-art influence-based algorithms and other non-influence-based frameworks. Surprisingly, training on a 15% subset selected by BIDS can even outperform full-dataset training with a much more balanced performance. Our analysis highlights the importance of both instance-level normalization and iterative optimization of selected data for balanced learning of diverse capabilities.
Figures
Figures from the paper (5 more)
Forward citations
Cited by 1 Pith paper
-
Learning Task Mixtures from Task Affinities: A Probabilistic Graphical Model for Supervised Fine-Tuning
TaskPGM optimizes a quadratic energy over task mixtures using PMI/JSD behavioral affinities, yielding mixtures that outperform naive sampling on several 7B LLM fine-tuning benchmarks.
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.