{"id":"e1bf5339-a355-429e-b1d0-466aae50eea6","arxiv_id":"2505.05327","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RICo scores instruction examples by their in-context perplexity effect on an assessment set, then trains a lightweight selector to pick top-scoring data, achieving better benchmark results from 5% to 15% of the original data.","lead":"This paper introduces RICo, a scoring method that estimates how much each instruction-tuning example improves a language model by inserting the example into the model's context and measuring perplexity changes. RICo selects small high-value data subsets that, in the reported experiments, outperform training on the full dataset and on several existing selection baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gains are computed at the peak of a data-ratio curve fitted to the same benchmarks, biasing the 5.42-point and 2.06-point claims.","rationale":"The reader's verdict (CONDITIONAL) already captures the main risks, and our concern reinforces it rather than changing it. We partially agree with the reader's weakest-assumption choice: the ICL-to-finetuning proxy is indeed unverified and load-bearing, but the more immediately checkable weakness is that the headline numbers come from a post hoc selection of the 15% ratio on the same benchmarks used for evaluation. The proxy concern is fundamental, yet the paper's empirical results are the only evidence supporting it, and those results are optimistically biased. A validation-based re-evaluation would settle whether the method's apparent advantage is real. Until then, the paper should not be accepted as stating the claimed magnitudes; the conditional verdict is appropriate.","tokens_in":18624,"tokens_out":11053,"duration_ms":112736,"concrete_test":"Pre-register a single selection ratio without access to the 12 benchmark scores. For example, fix 15% a priori or choose it on a held-out validation split (e.g., 10% of Alpaca instructions). Then, with at least 3 random seeds, train LLaMA3.1-8B on the top 15% selected by RICo and by each baseline, and report means and standard errors on the 12 benchmarks. If RICo's mean advantage over the best baseline is less than 1 point or the 95% confidence intervals overlap, the claimed 2.06-point advantage is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the empirical superiority of RICo's 15% subset, but that ratio was not pre-specified. In Section 6 (Optimal Data Scale), the authors evaluate RICo at ten scales (1% to 100%) on the same 12 benchmarks used for the headline, observe that the average peaks at 15%, and then write: 'Based on this observation, we select the model trained on 15% of RICo-selected data as the representative model for comparisons.' The reported RICO(15%) average of 43.37 (+5.42 over FULL) in Table 1 and the 2.06-point advantage over the best baseline in Table 2 are therefore the maximum of a curve fit to the evaluation set. Baselines are compared only at this same 15% budget, which is RICo's empirically chosen optimum; if a baseline has a different optimal scale, the comparison is biased. No held-out validation split, pre-registered ratio, or multiple-comparison correction is described. This does not refute the method, but it means the claimed magnitude of the central result is an optimistic upper bound, and the evidence that RICo 'accurately identifies high-contribution samples' is weaker than the headline suggests.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RICo, a gradient-free method for instruction-tuning data selection. RICo scores each candidate training sample by the change in perplexity it induces when inserted into the context of an assessment set, normalized by a length-controlled random baseline and task difficulty (Eqs. 1-5). A lightweight selection model is then trained with LoRA on the RICo labels so that the deployed selection requires one forward pass per candidate. Experiments on LLaMA3.1-8B, Qwen2.5-3B, and LLaMA2-7B, using Alpaca and WizardLM training data and 12 benchmarks plus 5 pairwise evaluation sets, support the headline claims: models trained on 15% of RICo-selected Alpaca data improve LLaMA3.1-8B's average benchmark score by 5.42 points over full-data training and by 2.06 points over the best prior selection method.","tokens_in":18843,"tokens_out":6205,"duration_ms":62537,"significance":"If the results hold, RICo is a practically valuable contribution: it avoids gradients, provides a fairness adjustment for length bias, reports consistent gains across three model sizes, includes a cross-dataset transfer experiment, and reduces training FLOPs by an order of magnitude. The design is sensible, and the evaluation is broader than in many prior data-selection papers. The main quantitative claims, however, are weakened by two protocol issues: the 15% ratio is chosen after observing the peak of the accuracy-versus-scale curve on the same evaluation benchmarks, and all comparisons rest on single runs without variance estimates. These issues affect the magnitude and reliability of the headline numbers rather than the basic plausibility of the method.","major_comments":[{"comment":"The 15% selection ratio used for the headline comparison is chosen after inspecting the same evaluation benchmarks on which the gains are reported. Section 6 states that the average score 'generally rises and then declines, peaking at 15%' and that 'based on this observation, we select the model trained on 15% of RICo-selected data as the representative model for comparisons.' The +5.42-point gain over FULL (Table 1) and the +2.06-point gain over the best baseline (Table 2) are therefore the maximum of the scale curve in Table 6, and all baselines are evaluated only at this empirically chosen optimum. This makes the headline magnitude an optimistic upper bound and biases the comparison against baselines whose optimal scale may differ. Please add a validation split (e.g., hold out several benchmarks for choosing the ratio), pre-register the ratio, or report the full scale curve for all baselines; at minimum, the paper should state explicitly that the reported gain is the best of ten tested scales.","section":"Section 6; Tables 1, 2, 6"},{"comment":"All reported numbers are single runs with no variance estimates. The central comparison (RICo 15% vs FULL on LLaMA3.1-8B, 43.37 vs 37.95) and the 2.06-point advantage over the best baseline are differences between individual runs. For Qwen2.5-3B the gain is 1.24 points, and for LLaMA2-7B the 15% run is 0.09 points below FULL while the 1% run is +0.65 points, so results are unstable across models and scales. Please report standard deviations over at least three seeds or bootstrap confidence intervals over benchmarks, and discuss whether the headline differences are within noise.","section":"Section 5.1 and Table 1"},{"comment":"The abstract and Section 3.3 describe 'strictly linear inference complexity,' but this applies only to applying the already-trained selection model. Computing the global-RICo labels requires O(n x m) forward passes over n assessment samples and m candidates, and the paper does not state how many samples are used to train the selection model, what top-K fraction is used for labeling, or how the classifier's training-set distribution relates to the full candidate pool. Please report these details and clarify that the linear-complexity claim covers only the deployment phase; otherwise the scalability claim is incomplete.","section":"Section 3.3"},{"comment":"The Limitations section concedes that in-context learning 'does not replicate the exact dynamics of full batch training with gradient updates,' yet Eq. (4) assumes the perplexity delta caused by inserting a candidate into the context is a measure of that candidate's contribution to instruction tuning. This proxy is load-bearing for the claim that RICo 'accurately identifies high-contribution data.' I am not asking for a theoretical proof, but the paper should provide a direct empirical check on at least one model/dataset: correlate the RICo score ranking with influence estimates from leave-one-out retraining or a gradient-based method on a small candidate pool, or show that top-RICo subsets outperform random subsets across several independent assessment sets. Without such a check, the mechanism claim is supported only indirectly by the end-task results.","section":"Limitations and Eq. (4)"}],"minor_comments":[{"comment":"The word 'gredient-free' should be 'gradient-free', and the method name is typeset as 'RIC O' throughout; please unify the notation, e.g., 'RICo'.","section":"Introduction"},{"comment":"The sentence 'These sets contain 218, 252, 80 and 300 human-curated instruction data' gives only four numbers for five test sets; please map each number to its named test set (Vicuna, WizardLM, LIMA, SInstruct, Koala).","section":"Section 4.3.2"},{"comment":"The text preceding Table 2 says the table reports 13 evaluation benchmarks, but Table 2 (and Table 5) contain 12 benchmarks; please correct the count.","section":"Section 5.2"},{"comment":"The description 'generally improves and then declines' is not supported by Table 6, where 1% (42.91) is higher than 5% (42.09) and 10% (42.08); the curve is non-monotonic before peaking at 15%, so please describe it more precisely.","section":"Section 6 and Figure 3"},{"comment":"The t-test citation to Kendall (1937) is inaccurate for a Student t-test; please cite the appropriate statistical source or remove the citation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the core method is plausible and the empirical scope is appropriate, but the headline numerical claims should not be taken at face value until the post-hoc ratio selection and the absence of variance estimates are addressed. I would also encourage the authors to release the RICo scores and training scripts, since the current project page alone makes independent verification difficult."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look. RICo is a real step beyond the binary ICL comparisons in Nuggets and the heuristic scores in Deita/Superfilter: it computes a fine-grained, length-adjusted perplexity difference per candidate sample against an assessment set, and then trains a lightweight linear-time selector on those scores. That combination is sensible, the experiments are broad (three models, twelve benchmarks, pairwise comparisons), and the cross-dataset transfer to WizardLM is a nice robustness check. The central direction of the result, that small selected subsets can beat full-data training on Alpaca, is probably real.\n\nThe soft spots are mostly about magnitude, not existence. First, the headline 15% ratio is the peak of a data-scale sweep on the very same evaluation benchmarks used for the headline comparison (Section 6: 'Based on this observation, we select the model trained on 15%...'). That is post hoc selection on the test set, and it inflates both the 5.42-point gain over FULL and the 2.06-point edge over the best baseline, since baselines are only compared at RICo's chosen optimum. Second, there are no error bars anywhere; every number looks like a single run, so it is hard to know how much of these point differences is noise. Third, the intro says LLaMA2-7B gains 0.65 points using 5%, but Table 1 shows that gain comes from 1%, and the 5% run is +0.01. That is a small inconsistency, but it should be fixed. Fourth, the ICL-to-finetuning proxy is load-bearing; the Limitations section admits ICL does not replicate gradient dynamics. That is honest, but it means the ranking method rests on an assumption that is not directly validated.\n\nThe reader's stress-test note is fair, but it does not kill the paper. The post hoc 15% selection is a real bias, yet the pattern across scales and models (and the WizardLM transfer) suggests the method is doing something useful. The better fix is to report results at multiple fixed budgets with variance, or at least to present the 15% number as part of a curve rather than as a single pre-specified comparison.\n\nWho is this for? People working on instruction-tuning data selection or efficient alignment. I would bring it to a reading group and cite it, though with a caveat about the ratio selection.\n\nRecommendation: send it to peer review. It deserves a serious referee, and the requests should be for variance estimates, a validation split for choosing the budget, and a clearer statement that the headline ratio was not fixed in advance.","headline":"Solid, genuinely new data-selection method with a plausible scoring mechanism, but the headline gains are likely optimistic because the 15% ratio is chosen post hoc on the same benchmarks and there are no variance estimates.","tokens_in":19401,"tokens_out":1313,"would_cite":true,"duration_ms":13213,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RICo ranks instruction samples by in-context contribution, and 15% of selected Alpaca data outperforms the full dataset on LLaMA3.1-8B.","keywords":["instruction tuning","data selection","in-context learning","perplexity","length-bias correction","Alpaca","LLaMA","gradient-free selection"],"falsifier":"Take a fixed set of candidate samples and measure, for each, the actual change in assessment-set perplexity after finetuning on that sample alone. If those measured changes do not rank-order samples the way RICo scores do, or if top-RICo subsets fail to beat full-data training on new benchmarks, the in-context proxy is not carrying the central claim.","tokens_in":18419,"feed_emoji":"🎯","tokens_out":10397,"duration_ms":85426,"temperature":0.7,"pith_summary":"The paper proposes RICo, a gradient-free method for choosing which instruction-tuning samples to train on. RICo measures each candidate sample's contribution by inserting it into the model's context and recording how much assessment-set perplexity drops, relative to a same-length meaningless sequence. The paper's central claim is that this in-context score ranks samples by their true value for finetuning, so that training on a small top fraction outperforms training on the full dataset. On LLaMA3.1-8B, 15% of Alpaca selected by RICo reaches an average benchmark score of 43.37 versus 37.95 for full-data training, and RICo beats the best prior selection method by 2.06 points. The paper also reports consistent gains on Qwen2.5-3B and LLaMA2-7B, and that the learned selector transfers from Alpaca to WizardLM.","feed_headline":"One perplexity score finds the 15% of data that beats 100%","feed_subtitle":"RICo ranks instruction samples by in-context gain, creating smaller datasets that train better models.","key_machinery":"The central object is the global-RICo score: the average, over a diverse assessment set $\\mathcal{D}_a$, of task-level scores that measure how much a candidate training sample $T_j$ reduces the model's perplexity on assessment sample $S_i$ when $T_j$ is inserted into the context. Each task-level score is $\\frac{\\mathrm{PPL}_\\theta(S_i \\mid T_j^{\\mathrm{rand}}) - \\mathrm{PPL}_\\theta(S_i \\mid T_j)}{\\mathrm{PPL}_\\theta(S_i) + \\epsilon}$, where $T_j^{\\mathrm{rand}}$ is a randomly generated sequence of the same length as $T_j$; subtracting this baseline is the fairness adjustment that removes length bias. The global score assigns equal weight to every assessment task, so a sample ranks high only if it helps broadly rather than specializing in one task type. A second component, a LoRA-trained classifier trained on top-scoring samples, reduces the cost of selecting from a large pool from $O(nm)$ to $O(m)$ inference calls.","core_discovery":"RICo's central claim is that a fine-grained, length-corrected perplexity difference measured in-context is a reliable estimate of a sample's contribution to gradient-based instruction tuning. For a training candidate $T_j$, the task-level score on assessment sample $S_i$ is $\\frac{\\mathrm{PPL}_\\theta(S_i \\mid T_j^{\\mathrm{rand}}) - \\mathrm{PPL}_\\theta(S_i \\mid T_j)}{\\mathrm{PPL}_\\theta(S_i) + \\epsilon}$, and the global score averages this over a diverse assessment set. The paper argues that a higher score means greater contribution and a negative score means harm, and that selecting the top fraction by this score finds samples that are diverse and moderately difficult rather than merely hardest. The empirical assertion is that models trained on such subsets match or beat full-data models: on LLaMA3.1-8B, 15% of RICo-selected Alpaca data gives 43.37 average benchmark score versus 37.95 for full-data training, and exceeds the best prior selection method by 2.06 points.","pith_inferences":["If the RICo score truly approximates finetuning influence, it could serve as an attribution tool: the same assessment set could localize which training samples drive failures on specific benchmarks, not just rank a corpus.","The length-bias correction is a separable component; a controlled ablation that removes only the random-sequence baseline would show how much of the gain over prior perplexity-based methods comes from fairness adjustment rather than from fine-grained task averaging.","Because the selection model is trained on Alpaca and transfers to WizardLM, the approach may scale to much larger candidate pools, but the LoRA classifier's calibration on out-of-distribution instructions would need to be checked before trusting rankings in production-scale corpora.","The reported optimal scale at 15% suggests that comparisons among selection methods should be made at each method's best scale; a fixed 15% comparison may favor methods whose optimum happens to fall there."],"forward_implications":["Training on 15% of RICo-selected Alpaca data outperforms full-data training on LLaMA3.1-8B by 5.42 points and on Qwen2.5-3B by 1.24 points, while using roughly one-fifth to one-hundredth of the FLOPs.","RICo outperforms established selection methods, including random, low and high perplexity, Alpagasus, Deita, Superfilter, and Nuggets, in both benchmark averages and GPT-4 pairwise comparisons.","The selection paradigm transfers across datasets: a selector trained on Alpaca scores directly filters WizardLM and improves average scores by up to 4.54 points at 5% selected data.","High-contribution samples tend to be diverse and of moderate difficulty, with extreme-difficulty samples avoided, suggesting an optimal data scale around 15% in this setting."],"supporting_citations":[{"why":"Supplies the Alpaca training corpus and the self-instruction generation setup on which all main experiments are run.","marker":"(Taori et al., 2023)"},{"why":"Baseline Nuggets, an in-context probing method using binary comparisons that RICo refines into fine-grained scores.","marker":"(Li et al., 2023c)"},{"why":"Provides the feasibility analysis of in-context probing that motivates using in-context learning as a gradient-free contribution signal.","marker":"(Jiao et al., 2025)"},{"why":"Documents perplexity's length sensitivity, motivating the random-sequence fairness adjustment in the RICo score.","marker":"(Wang et al., 2022a)"},{"why":"Alpagasus baseline based on GPT-3.5-Turbo response scoring, one of the widely used selection methods RICo must beat.","marker":"(Chen et al., 2023)"},{"why":"Deita baseline and the diversity and quality scoring framework used in related work and in the selected-sample analysis.","marker":"(Liu et al., 2023c)"},{"why":"Superfilter baseline and the IFD difficulty metric used to characterize high-contribution samples.","marker":"(Li et al., 2024b)"},{"why":"LoRA training of the lightweight selection paradigm that reduces selection complexity to linear inference calls.","marker":"(Hu et al., 2021)"},{"why":"LESS, a gradient-based selection method whose computational cost motivates RICo's gradient-free design.","marker":"(Xia et al., 2024)"}],"fun_headline_variants":["15% of data beats 100% when picked by RICo","In-context perplexity shift selects 15% that beats full data","RICo's 15% data selection outperforms full dataset","Smaller data wins: RICo's 15% beats 100% in tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that a training sample's effect on the model when placed in the prompt at inference time predicts its effect when the model is actually trained on that sample with gradient updates.","fun_headline_variants_meta":{"raw":{"variants":["15% of data beats 100% when picked by RICo","In-context perplexity shift selects 15% that beats full data","RICo's 15% data selection outperforms full dataset","Smaller data wins: RICo's 15% beats 100% in tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00063,"raw_usage":{"total_tokens":2921,"prompt_tokens":964,"completion_tokens":1957,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":1876}},"tokens_in":580,"tokens_out":1957,"duration_ms":15551,"temperature":1.0,"reasoning_tokens":1876,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:06:43.880409+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed set of candidate samples and measure, for each, the actual change in assessment-set perplexity after finetuning on that sample alone. If those measured changes do not rank-order samples the way RICo scores do, or if top-RICo subsets fail to beat full-data training on new benchmarks, the in-context proxy is not carrying the central claim.","supporting_citations":[],"review_version":1}