{"id":"b18b99a1-393b-4047-8ab1-379c6a2cc916","arxiv_id":"2501.12147","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"BIDS, a balanced influence-based data selection algorithm using per-task normalization and iterative greedy selection, improves balanced multi-capability instruction tuning and can outperform full-dataset training on a 15% subset.","lead":"Instruction tuning data for large language models is often chosen with influence scores that favor some tasks over others. This paper proposes BIDS, a selection algorithm that normalizes influence scores and iteratively favors underrepresented tasks, and reports that a 15% BIDS-selected subset can match or beat full-dataset training on seven benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Selection uses test-split instances to compute influence, so the claim that BIDS beats full-dataset training is confounded by evaluation leakage.","rationale":"The reader's verdict is already CONDITIONAL and explicitly cites test-split leakage as one of the load-bearing weaknesses, so my read does not move the verdict. However, I elevate this issue above the additivity concern as the single most load-bearing threat to the central claim. The additivity assumption is a modeling limitation that the authors themselves acknowledge, and the paper's analysis in Section 6.2 even uses the failure of first-order additivity as an explanatory observation. By contrast, the test-split leakage is an experimental design flaw that directly undermines the fairness of the headline comparison against full-dataset training. Full training and random selection have no access to test examples, while BIDS's selection signal is computed from gradients of test instances, so the comparison is not controlled. This is not a claim of dishonesty; it is a standard methodological red flag that can be resolved by rerunning selection with a properly disjoint validation set and reporting multiple seeds. I partially agree with the reader because they identified both additivity and leakage as load-bearing, whereas I focus on leakage as the decisive issue for the paper's strongest claim.","tokens_in":21404,"tokens_out":5477,"duration_ms":62950,"concrete_test":"Re-run the Llama-3-8B 15% budget experiment (and if possible the 5% and 10% budgets) with the 350 influence-validation instances drawn from a held-out split that is disjoint from all seven evaluation benchmarks, for example from UltraInteract's own validation portion or from a randomly reserved subset of the training pool, and then evaluate on the official test splits. If BIDS no longer exceeds full-dataset training by at least 0.5 macro-average points, the test-split leakage is material to the central claim and the headline should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is in Appendix A.2: 'we uniformly sample 50 validation instances from either the validation or the test split (when there is not a separate validation split) of each of the seven evaluation tasks, leading to a total of 350 validation instances.' For HumanEval, MBPP, GSM-Plus, MATH, BBH, MMLU, and IFEval, no separate validation split is standard, so the 350 instances are drawn from the same test sets used for evaluation. The entire Attribution Matrix A is therefore computed against test-set gradients, and BIDS's iterative selection explicitly optimizes the selected subset's influence on these exact test instances. This gives BIDS an information advantage over the full-dataset and random baselines, which never see test labels during training or selection. The central claim that a 15% BIDS subset 'can even outperform full-dataset training' is thus not an apples-to-apples comparison under standard evaluation protocol: the selection criterion is partially optimized for the evaluation distribution and even for the specific sampled test items. The relative comparison BIDS vs. LESS is less affected because both use the same leakage, but the headline claim against full training is not interpretable until this is controlled. This concern is independent of the additivity limitation the paper acknowledges in Section 6.2; even if the selection objective were exactly correct, the experimental comparison would still be confounded.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BIDS, an influence-based data selection algorithm for multi-task instruction tuning. Given an Attribution Matrix of pairwise influence scores between training examples and validation instances, BIDS first applies column-wise normalization to place influence scores of different validation instances (tasks) on a common scale, then iteratively selects training examples that maximize, over validation instances, the componentwise gap between the candidate's influence distribution and the average influence distribution of the already-selected set. This procedure favors training data that most increase influence on currently underrepresented tasks. Experiments on Llama-3-8B and Mistral-7B-v0.3 across seven benchmarks (coding, math, logical inference, world knowledge, instruction following) show that BIDS-selected subsets at 5%, 10%, and 15% budgets achieve higher macro-average performance than the LESS task-wise max baseline, other influence-based heuristics, random selection, and a representation-based method (RDS). A 15% BIDS subset trained for four epochs is reported to outperform full-dataset training. The paper also provides ablations and analyses (THI, AID) of the influence distribution of selected data.","tokens_in":21596,"tokens_out":7184,"duration_ms":77662,"significance":"If the reported results hold, BIDS would be a simple, computationally cheap, and effective method for balanced multi-capability instruction tuning, addressing a real shortcoming of current influence-based selection methods. The paper includes several strengths: a clear problem diagnosis (cross-task influence scale bias is documented in Figures 1 and 2), a lightweight algorithm whose overhead is quantified in Appendix A.7, consistent improvements over the considered baselines under equal budgets across two model families (Tables 2 and 4), and ablations that support the contribution of each design component (Table 3). The full-dataset-outperforming claim is potentially significant, but it is currently confounded by the use of test-split instances during selection, as detailed in the major comments. The core methodological contribution—balanced influence-based selection relative to other influence methods—is defensible, but the headline claim against full training needs a protocol fix before the paper's central message can be fully accepted.","major_comments":[{"comment":"The selection procedure uses test-split instances as validation data. Appendix A.2 states that for each of the seven evaluation tasks, the authors \"uniformly sample 50 validation instances from either the validation or the test split (when there is not a separate validation split)\". For HumanEval, MBPP, GSM-Plus, MATH, BBH, MMLU, and IFEval, no separate validation split is standard, so the 350 validation instances are drawn from the exact test sets on which the final models are evaluated. This gives BIDS (and all influence-based selection methods, including LESS) access to the evaluation distribution and to the specific test items during the selection step. The full-dataset and random baselines never see test labels. Consequently, the abstract's claim that \"training on a 15% subset selected by BIDS can even outperform full-dataset training\" is not an apples-to-apples comparison under a standard held-out evaluation protocol. The relative comparison to LESS and other influence baselines is less affected because all influence methods share the same leakage, but the full-training comparison is confounded. To support the headline claim, the authors should either (a) use a validation set fully disjoint from the test splits (e.g., sample development examples from the training set or create a separate held-out split), or (b) explicitly reframe the result as conditional on access to the test distribution and remove or heavily qualify the full-training comparison. This is a load-bearing issue because the abstract, introduction, and conclusion all emphasize the full-dataset-outperforming result.","section":"Appendix A.2 and Table 2"},{"comment":"The claim that BIDS outperforms full-dataset training rests on a fragile and nonstandard experimental comparison. In Table 2, BIDS at 15% with the default two epochs gives a macro-average of 46.1, which is lower than Full (epochs=1) at 46.9 and Full (epochs=4) at 46.7. The paper's positive result uses \"BIDS (epochs=4)\" (47.1), a configuration where the subset is trained for twice as many epochs as the default, while the full-dataset baselines are trained for one or four epochs without reporting any analogous search over epochs. The macro-average differences between BIDS (epochs=4), Full (epochs=1), and Full (epochs=4) are 0.2–0.4 points, which is within typical run-to-run variance for instruction tuning of 8B-scale models, yet no standard errors or multiple-seed results are reported for these key comparisons. To make the \"outperforms full training\" claim credible, the authors should report means and standard deviations over at least three random seeds, and should compare the best subset-trained model against a full-dataset model trained under an equally tuned epoch budget (or show that additional epochs do not improve the full-data baseline). Without this, the central headline is not statistically supported.","section":"Table 2 and Section 5.2"},{"comment":"The paper's design relies on a first-order additivity assumption that is acknowledged as a limitation in Section 6.2, but the acknowledgment does not fully address the algorithm's dependence on it. The utility function in Algorithm 1 (A_ij minus the average of already-selected rows) assumes that the average influence distribution of a selected subset is a valid proxy for the subset's actual effect on validation performance. The paper itself notes that \"high-influence points selected by top-k increase average influence but do not add up to performance.\" Since BIDS is explicitly optimizing the balance of average influence rather than the balance of task accuracy, the relationship between the two should be tested more directly. For example, the authors could compare the utility score ∆(i) of a candidate with the actual change in validation accuracy when that example is added to a small selected set. The ablations in Table 3 provide indirect evidence that the heuristic works, but given that the algorithm's entire greedy rule is built on this assumption, a direct test or a more circumspect claim about what BIDS optimizes would strengthen the paper. The already-stated limitation is appreciated, but it should be elevated to a clearly stated caveat in the Results and Conclusion.","section":"Section 6.2 and Algorithm 1"}],"minor_comments":[{"comment":"When the selected set T is empty, the expression (1/|T|) sum_{k in T} A_kj involves division by zero. The paper does not specify how the first iteration is handled. Please clarify the initialization (e.g., select the first example with the largest max_j A_ij, or start with a random example, or define the average over an empty set as zero).","section":"Algorithm 1"},{"comment":"The row \"BIDS (epochs=4)\" uses a different training regime than the other rows in the same table, but this is not highlighted in the methodology. Please explicitly state in Section 5.2 that the four-epoch setting is a separate protocol and explain why this is the appropriate comparison to full-dataset training.","section":"Table 2 and Section 5.2"},{"comment":"The phrases \"The maximums of AID decrease\" and \"The minimums of AID increase\" are awkward and could be misread. Please rephrase to clarify that these are per-task (or per-validation-instance) maxima and minima of the average influence distribution.","section":"Section 6.2"},{"comment":"The y-axis labels in the AID plots are not clearly described. Please specify the units and explain what the points represent (e.g., mean normalized influence per validation instance). It would also help to mark the per-task boundaries clearly in the figures.","section":"Figures 4, 5, 9, 10"},{"comment":"The evaluation protocol for IFEval (temperature=0.7, median over three seeds) differs from the greedy decoding used for the other tasks. This is disclosed, but it would be good to confirm that the same protocol is used for all models and all budgets, and to state the standard deviation or range of the three IFEval runs.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a genuine problem and the core idea (column-wise normalization plus iterative selection for underrepresented tasks) is simple, well-motivated, and computationally cheap. The equal-budget comparisons to LESS and other selection baselines are likely to hold up under scrutiny, and the ablations are clean. However, the headline claim of outperforming full-dataset training is currently not interpretable because of the test-split leakage in the selection pipeline, and even setting that aside, the full-training comparison is based on a single very small macro-average gap with no statistical grounding. Both issues are fixable within the scope of a revision: the authors can rerun the key comparisons with a disjoint validation set for selection, or at minimum, they can remove or heavily qualify the full-training claim. I would also encourage them to strengthen the epoch comparison and report variance. The additivity limitation is self-acknowledged but deserves a more prominent caveat. Given the paper's potential impact if the results are confirmed, I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. The method is simple, well-motivated, and likely a real improvement over LESS for balanced instruction tuning. The headline claim — that a 15% subset selected by BIDS beats full-dataset training — is confounded by test-split leakage in the influence computation, so treat that specific claim as unverified.\n\nWhat is genuinely new is the diagnosis. The paper shows that LESS-style influence scores carry a cross-task scale bias: MMLU's influence values swamp the others, so selection oversamples MMLU and, paradoxically, does not even help MMLU much. That is a solid observation, backed by the AID/THI analysis. The fix is two clean steps: column-wise z-score normalization of the Attribution Matrix, and iterative greedy selection that picks the example with the largest component-wise gap against the current subset's average influence. The Table 3 ablation supports both components, and the Llama-3 gains are consistent across the 5/10/15% budgets. BIDS is also cheap — sub-800MB matrix, under a minute of selection time. Credit where due: the paper is honest about its first-order additivity assumption in Section 6.2, and the writeup is transparent.\n\nThe soft spots, in proportion. The biggest one is load-bearing. Appendix A.2 samples 50 validation instances per task 'from either the validation or the test split (when there is not a separate validation split).' Most of these benchmarks — HumanEval, MBPP, GSM-Plus, MATH, BBH, IFEval — have no standard validation split, so those instances are effectively test data. BIDS then optimizes influence on exactly those instances, while the full-dataset and random baselines never see test labels. The relative BIDS-vs-LESS comparison is still interpretable because both use the same instances, but the full-training comparison is not apples-to-apples until re-run with a proper held-out validation set.\n\nSmaller issues: the headline rests on the 4-epoch 15% subset (47.1) beating full training at either one epoch (46.9) or four (46.7); the default two-epoch 15% subset (46.1) does not beat full training. The Mistral check is close to a tie (43.3 vs 43.2 at 15%, 4 epochs). No error bars, no released code or data.\n\nBottom line: useful for people working on instruction-tuning data selection and influence estimation. It deserves a serious referee; the central flaw is fixable. Send it to review and ask for a leakage-free re-run before the full-training claim is accepted.","headline":"BIDS is a clean, likely useful balancing fix for influence-based data selection, but the marquee claim that a 15% subset beats full training is compromised by test-split leakage in the selection signal.","tokens_in":22181,"tokens_out":7618,"would_cite":true,"duration_ms":70983,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 15% subset, chosen by a normalize-then-rebalance influence algorithm, outperforms full instruction tuning across seven benchmarks.","keywords":["influence functions","instruction tuning","data selection","balanced multi-task learning","large language models","LESS","BIDS","gradient similarity"],"falsifier":"Take the same UltraInteract pool and the same seven benchmarks, draw the 50 validation instances per task strictly from splits disjoint from the test sets, and rerun the 15% BIDS-versus-full comparison; if the macro-average advantage disappears, the current result depends on the shared test instances. Separately, a retraining probe that adds the highest-gap candidate to a fixed subset and measures the actual marginal performance change would directly test whether the componentwise-gap identity is what drives the gains.","tokens_in":21102,"feed_emoji":"⚖️","tokens_out":10178,"duration_ms":100467,"temperature":0.7,"pith_summary":"Influence-based instruction data selection aims to pick training examples that most improve an LLM's predictions, but existing methods like LESS—a gradient-similarity influence selector—systematically over-select data for tasks whose influence scores are intrinsically large. This paper claims that this cross-task scale bias is why LESS produces unbalanced and sometimes weaker multi-capability performance, and that the bias can be removed by normalizing influence scores and by selecting iteratively. On seven benchmarks spanning coding, math, logical inference, world knowledge, and instruction following, with Llama-3-8B and Mistral-7B-v0.3, the proposed BIDS algorithm consistently raises the average across all seven benchmarks compared with both influence-based and non-influence-based selectors. The paper's headline result is that training on a 15% BIDS-selected subset outperforms full-dataset training on the same pool, with a much more balanced distribution of task performances. If that holds, selective tuning is not only cheaper but better than training on everything.","feed_headline":"A 15% subset beats full-dataset LLM training across five capabilities","feed_subtitle":"Normalizing influence scores and favoring the most underrepresented task lifts average accuracy and instruction-following.","key_machinery":"The central object is the Attribution Matrix A, whose rows are training examples and whose columns are validation instances. BIDS standardizes each column to zero mean and unit variance, then runs a greedy loop: with selected set T, the utility of candidate t_i is Δ^(i) = max_{1≤j≤|V|} (A_{ij} − (1/|T|)Σ_{k:t_k∈T} A_{kj}), and the candidate with the highest utility is added. Because the largest component of that difference points to the validation instance where the current subset is weakest, the loop mechanically favors the most underrepresented task at every step.","core_discovery":"LESS, the influence-based selector used as the starting point, computes pairwise gradient similarities between training and validation data and then picks the training examples with the highest raw influence scores. When the tasks are diverse, this recipe is biased: columns of the influence matrix that correspond to certain tasks carry larger values by construction, so top-k selection oversamples those tasks and undersamples others, and the paper shows that even the oversampled tasks do not improve. BIDS corrects the recipe with two changes: it standardizes each validation column of the influence matrix to zero mean and unit variance, and then, instead of scoring examples once, it builds the subset greedily by comparing each candidate's influence distribution with the average influence distribution of the current subset and choosing the example whose largest componentwise gap is greatest. The reported consequence is that selected subsets become balanced in influence while remaining influential, and a 15% subset trained for four epochs surpasses full-data training in average performance.","pith_inferences":["The paper does not test BIDS on other influence estimators, but the failure it identifies lives in the raw influence matrix and the top-k scoring rule, not in LESS's specific gradient computation; plugging column normalization and gap-based greedy selection into any other attribution-matrix method is a direct testable extension.","The same scale-mismatch diagnosis likely applies to non-influence utility scores: RDS in the paper also over-selects coding tasks, so normalizing columns and selecting by largest underrepresented-gap could rebalance representation-similarity selection as well.","The headline comparison replays the 15% subset over four epochs, so the claimed efficiency is about unique examples rather than total training tokens; comparing equal total tokens or equal compute would put the '15% beats full data' statement on a stricter footing.","The greedy rule optimizes a first-order additivity objective; if influence interactions matter at larger budgets, BIDS's advantage should saturate or reverse somewhere above 15%, which a budget sweep would reveal."],"forward_implications":["Column-wise normalization alone already improves the instance-wise max selector under 5%, 10%, and 15% budgets, and adding iterative selection raises the average across benchmarks further while narrowing cross-task performance gaps.","A 15% subset chosen by BIDS and trained for four epochs beats both one-epoch and four-epoch full-data training in average performance, with its clearest gains on instruction-following and balanced gains on reasoning tasks.","The same pattern holds with a second base model family, so the balancing effect is not an artifact of Llama-3-8B's influence estimates.","Analyzing the selected subsets shows that BIDS lowers the disparity of average influence across tasks and instances, whereas top-k selection raises average influence without raising performance—evidence that high influence does not linearly add up."],"supporting_citations":[{"why":"LESS supplies the influence estimation pipeline and the task-wise max baseline that BIDS rebalances.","marker":"Xia et al., 2024"},{"why":"UltraInteract provides the training pool and the seven benchmarks across five capabilities used in all experiments.","marker":"Yuan et al., 2024"},{"why":"LoRA is the parameter-efficient training method used for influence warmup and for training selected subsets.","marker":"Hu et al., 2021"},{"why":"Representation-based data selection (RDS) baseline; its similarity scores are compared against BIDS.","marker":"Zhang et al., 2018"},{"why":"The other half of the RDS baseline definition, used for the representation-similarity selection comparison.","marker":"Hanawa et al., 2020"},{"why":"Origin of influence-function estimation, the conceptual basis for gradient-matching influence scores.","marker":"Koh and Liang, 2017"},{"why":"Gradient-tracing influence method cited as the line of work LESS scales to LLMs.","marker":"Pruthi et al., 2020"}],"fun_headline_variants":["Balanced influence beats full dataset with just 15% of data","BIDS: balanced influence selection lifts all five capabilities","Normalize influence, rebalance tasks, then 15% beats 100% of data","LLM fine-tuning: 15% balanced subset outperforms full dataset","Forget full dataset: influence-balanced 15% wins across tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that influence scores are additive in the sense that the average influence vector of the selected set is a valid baseline and the example with the largest single-column gap delivers the largest real improvement, an assumption the paper itself flags as first-order linearity; a second premise is that the 50 validation instances per task, sampled from the test split when no validation split exists, do not unfairly leak evaluation information into selection.","fun_headline_variants_meta":{"raw":{"variants":["Balanced influence beats full dataset with just 15% of data","BIDS: balanced influence selection lifts all five capabilities","Normalize influence, rebalance tasks, then 15% beats 100% of data","LLM fine-tuning: 15% balanced subset outperforms full dataset","Forget full dataset: influence-balanced 15% wins across tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00023,"raw_usage":{"total_tokens":1509,"prompt_tokens":996,"completion_tokens":513,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":419}},"tokens_in":612,"tokens_out":513,"duration_ms":5722,"temperature":1.0,"reasoning_tokens":419,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:28:07.280681+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same UltraInteract pool and the same seven benchmarks, draw the 50 validation instances per task strictly from splits disjoint from the test sets, and rerun the 15% BIDS-versus-full comparison; if the macro-average advantage disappears, the current result depends on the shared test instances. Separately, a retraining probe that adds the highest-gap candidate to a fixed subset and measures the actual marginal performance change would directly test whether the componentwise-gap identity is what drives the gains.","supporting_citations":[{"cited_title":"Evaluation of Similarity-based Explanations","cited_arxiv_id":"2006.04528","evidence_quote":"The other half of the RDS baseline definition, used for the representation-similarity selection comparison."}],"review_version":1}