{"id":"28b13e88-b102-447e-87a7-e109e7662b6f","arxiv_id":"2412.00631","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A reward-oriented influence scoring method that uses pairwise preference loss to select 5% of instruction-tuning data, outperforming similarity-based selection baselines on SHP, SE, and HH-RLHF.","lead":"ROSE picks useful training examples for fine-tuning a language model by estimating which examples best improve a preference-based reward, rather than which examples reduce prediction loss. On three preference benchmarks, selecting just 5% of the data with ROSE often matches or outperforms fine-tuning on the full dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ROSE's influence score (Eq. 7) is never directly validated; Fig. 4 correlations do not establish the mechanism.","rationale":"The reader's weakest_assumption identifies the general area of concern: the influence score's validity as a reward proxy. I agree that the DPO-loss-to-win-rate correlation evidence is thin (only four points). However, I do not think the 'cross-loss mix' itself is the issue; the Taylor expansion in Eq. 1-2 is valid for any validation loss when the training step is on the training loss. The more serious gap is that the paper provides no direct validation of the influence approximation itself—only end-to-end selection results. The transfer results in Table 8 show that the method is fragile when the selection model differs from the target model, which weakens the claim of generalizability. These issues are addressable with a focused empirical test. Because the paper's main empirical claims are still supported by multiple models and benchmarks, and the concerns are about internal mechanism rather than reproducibility, the conditional verdict remains appropriate.","tokens_in":23213,"tokens_out":4592,"duration_ms":43902,"concrete_test":"Compute ROSE scores S(z) for a random subset of ~50 training examples. For each example, fine-tune a model from the same warm-up checkpoint on a fixed base set of, say, 1% of D plus that single example (and a control with only the base set), then measure the change in DPO loss on D'_val and the change in test win rate. Compute the Spearman rank correlation between S(z) and the actual change in win rate. Also, verify the single-step Taylor approximation by comparing Eq. 7's predicted change in DPO loss for one example to the actual change after one SGD step on that example. A significant positive correlation (e.g., rho>0.3) would support the mechanism; a null result would indicate the selection gains arise from the gradient-similarity heuristic rather than validated influence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise of ROSE is that the influence score in Eq. 7 (inner product of the DPO validation gradient and the SFT training gradient) identifies training examples that, when used for SFT fine-tuning, increase the model's downstream reward/win rate. The first-order Taylor argument in Sec. 3 justifies using the inner product to approximate the change in a validation loss caused by a single gradient step on the training loss; that argument does not require the two losses to be identical, so the 'cross-loss' mix is not itself a flaw. What is missing is any direct evidence that this approximation is reliable for DPO validation loss and SFT training gradients over the full LoRA training trajectory. Fig. 4 only shows that the DPO *loss* on the few-shot validation set correlates with test win rate across 4 checkpoints (n=4; SHP r=0.66, SE r=0.92, HH r=0.93). It does not show that Eq. 7's *scores* correlate with the actual per-example impact on win rate, nor that the approximation survives the multi-step, batched AdamW optimization used to train the final model. The transfer experiments (Table 8) underscore the fragility: when the selection model differs from the target model, ROSE-T is often no better than random (e.g., SHP on Llama-3.1-8B: ROSE-T 29.7 vs random 37.4), and the method's shot counts are tuned per benchmark, with the HH ablation showing strong fluctuation. If the influence scores are only a coarse similarity metric rather than a valid counterfactual estimate, the paper's central claim reduces to an empirical comparison of heuristics, with no mechanism explaining why the reward-oriented gradient should transfer across tasks.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ROSE, a data selection method for task-specific instruction tuning. Instead of minimizing next-token prediction loss on a validation set, ROSE constructs a few-shot preference validation set and uses the DPO pairwise loss as a reward signal. Influence scores are computed as the inner product between DPO validation gradients and SFT training gradients across LoRA checkpoints (Eq. 7), and the top 5% of training examples are selected for final SFT. Experiments on SHP, SE, and HH-RLHF with Llama-2, Llama-3.1, and Mistral models report that ROSE outperforms several baselines and is competitive with full-data fine-tuning. The paper also includes ablations on data percentage, checkpoint count, shot count, and transferability.","tokens_in":23531,"tokens_out":5573,"duration_ms":48666,"significance":"If the mechanism holds, the paper offers a practical recipe: a small number of preference pairs can guide the selection of a small SFT subset that beats full-data tuning. The evaluation is broad and mostly careful, with three trials per configuration, standard deviations, six model variants, multiple baselines, and a released code repository. The main unresolved issue is theoretical: Eq. 7 mixes two different losses without a derivation, and the paper's evidence for the reward-loss correlation (Fig. 4, four checkpoints per dataset) does not directly validate the influence score. Because the main in-distribution win rates are consistent and large, the result is worth pursuing, but the current manuscript needs additional validation of the selection mechanism before the central claim is fully supported.","major_comments":[{"comment":"The load-bearing score in Eq. (7) is the inner product between the DPO validation gradient ∇θ L_ROSE(D'_val; θ_i) and the SFT training gradient ∇̄θ l(z; θ_i). The Taylor argument in Sec. 3 (Eqs. (1)-(2)) justifies the inner-product approximation for a single loss used for both training and validation. ROSE does not provide a derivation or a formal argument for why the cross-loss inner product approximates the effect of an SFT training step on the DPO reward. Figure 4 is not direct evidence: it shows that the DPO validation loss correlates with test win rate over four checkpoints (SHP r=0.66, SE r=0.92, HH r=0.93), but it does not show that the scores from Eq. (7) rank individual examples by their effect on win rate, nor that the approximation survives multi-step AdamW optimization with LoRA and batch size 128. Please add either a derivation with explicit assumptions or a direct validation, e.g., comparing Eq. (7) scores against leave-one-out or top-k/bottom-k selection effects on win rate in a controlled SFT setting.","section":"§4.2, Eq. (7)"},{"comment":"The transfer experiments show that ROSE-T (selection with Llama-2-7B, training on the target model) is often worse than random selection. For example, on SHP, ROSE-T is 29.6 vs. random 42.1 for Llama-2-13B, 29.7 vs. 37.4 for Llama-3.1-8B, 37.7 vs. 42.3 for Llama-3.1-8B-INS., 36.2 vs. 53.3 for Mistral-7B, and 53.9 vs. 58.4 for Mistral-7B-INS.-V0.3; on SE, ROSE-T is below random in all five transferred settings. The appendix text acknowledges this ('comparable or inferior to random selection'), but the main text says ROSE-T 'consistently outperforms LESS-T' without noting the failure against random. This is load-bearing for the generalizability claim, because selecting with a weaker model is the practical scenario when the target model is large. Please either temper the claim or provide an analysis of why transfer fails and under what conditions the selection score transfers.","section":"§5.2, Table 8"},{"comment":"The method's default configuration uses per-benchmark tuned shot counts (5 for SHP, 2 for SE, 1 for HH), and Figure 5 shows considerable fluctuation with shot count, especially on HH. The paper's own Limitations section states that specific shot numbers are tailored to individual datasets. As a result, the reported 5% advantage is not shown to be robust to this hyperparameter: without a principled way to choose the shot count or a fixed-shot ablation, a user of ROSE on a new task does not know whether the method will be effective. Please report results with a fixed shot count across benchmarks and/or an automatic criterion for choosing the shot count. In addition, the few-shot preference validation set is sampled from the same benchmark dataset as the test set, so the validation/test distribution overlap is a confound; please discuss how this affects the 'few-shot' claim.","section":"Appendix B.1, Table 5; Limitations"}],"minor_comments":[{"comment":"The text says that using 5% data 'achieves performance that surpasses full-data fine-tuning,' but Table 2 shows several counterexamples (e.g., Llama-2-13B SHP: 44.3 vs. 48.4; Llama-3.1-8B HH: 58.6 vs. 60.6). Please qualify the claim to 'competitive with, and sometimes better than, full-data fine-tuning.'","section":"§5.2, paragraph after Figure 3"},{"comment":"The correlations are computed from only four checkpoints per dataset; please report confidence intervals or a permutation test, and note the small sample size in the caption. Also, the next-token prediction loss correlation is negative for SHP but positive for SE and HH, which is not discussed.","section":"Figure 4"},{"comment":"Please proofread for typos and formatting issues: Table 1 caption has 'beselines' for 'baselines,' the title and abstract contain 'inStruction,' and Section 5.2 has 'funetuning' for 'fine-tuning.' The column headers in Table 8 are also difficult to parse because repeated groups are not visually separated.","section":"Global"},{"comment":"The RDS baseline is cited to Zhang et al. (2018), which appears to be the deep-features perceptual metric paper; please verify that this is the correct source for representation-based data selection. In Algorithm 1, the output is Γ′ but the training step says 'Train model Γ on Dtrain' without defining how Γ′ is obtained from Γ.","section":"References and Algorithm 1"},{"comment":"Equation (7) defines S(z, D′_val^{(j)}) for each subtask, but the text says ROSE takes the maximum score across subtasks without defining the operator formally. Please make the aggregation step explicit.","section":"§4.2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely a good fit for an applied ML venue. The main issue is the missing direct validation of the influence mechanism; a revision with a targeted ablation (e.g., comparing Eq. (7) scores against actual per-example impact on win rate) would substantially strengthen the paper. The transfer and shot-count issues also need to be addressed honestly, as the appended Limitations section already indicates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plainly: this is a good empirical paper with a real contribution. ROSE takes the LESS influence-function machinery and swaps the next-token validation loss for a DPO preference loss on a few-shot preference set. That is a simple, non-obvious move, and the paper executes it thoroughly: six models, three benchmarks, standard deviations over seeds, and a fair set of baselines. The headline result -- 5% of data beating full fine-tuning on several model/benchmark combinations -- is repeated consistently enough that I believe it is real.\n\nThe paper earns credit for the ablation work. The checkpoint-number study, the shot-count analysis, and the transfer experiments are exactly the things you'd want to see. The transfer table is honest: ROSE-T is often no better than random when the selection model is much weaker than the target model. That limits the claim that selection transfers across model families, but it doesn't undermine the main result.\n\nThe soft spot is the mechanism. Eq. 7 mixes DPO validation gradients with SFT training gradients, and the Taylor expansion in Sec. 3 justifies an inner product for a single loss; the cross-loss version is asserted, not derived. That alone isn't fatal -- first-order influence heuristics work all the time -- but the paper never directly validates that the ROSE scores track actual per-example impact. Fig. 4 only shows that the DPO validation loss correlates with test win rate over four checkpoints (n=4), which supports the reward signal, not the influence scores. The shot counts are tuned per benchmark, and the HH ablation shows real fluctuation. These are fixable issues, not load-bearing flaws.\n\nThe limitations section is refreshingly frank about the shot-number selection and the dependence on preference validation data quality. I'd like to see code and data released, and a direct test of the influence scores -- e.g., comparing win-rate impact of top- vs bottom-ranked examples, or a leave-one-out check on a small model.\n\nBottom line: this is a paper worth a serious referee. Send it to review; ask for the influence-score validation and transfer caveats to be tightened. I'd cite it if I worked on data selection, and I'd bring it to reading group.","headline":"Solid empirical contribution with a real idea; the mechanism is under-validated but the main result is believable and the paper deserves a serious referee.","tokens_in":24102,"tokens_out":2309,"would_cite":true,"duration_ms":23277,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Selecting 5% of instruction data by preference-gradient influence scores beats full-data fine-tuning on win rate.","keywords":["instruction tuning","data selection","direct preference optimization","influence functions","reward maximization","win rate","LoRA","task-specific fine-tuning"],"falsifier":"Compute ROSE scores for every training point on one benchmark, fine-tune with the top 5%, the bottom 5%, and a random 5%, and compare win rates; the central claim predicts top beats random and random beats bottom. To test the score's ranking directly, measure leave-one-out win-rate impact for a few hundred scored examples and check for a strong positive correlation between ROSE score and measured impact.","tokens_in":22996,"feed_emoji":"🎯","tokens_out":10069,"duration_ms":88837,"temperature":0.7,"pith_summary":"The paper argues that existing data-selection methods for task-specific instruction tuning pick data by similarity to a validation set, which amounts to minimizing next-token prediction loss on that set—and next-token loss does not track real task performance. ROSE instead selects the training examples whose gradients align with the gradient of a pairwise preference loss on a small set of preferred and dispreferred responses. Using that alignment score, selecting just 5% of a general instruction corpus produces higher judged win rates than fine-tuning on the full corpus across the SHP, Stack Exchange, and HH-RLHF benchmarks, and higher win rates than prior selection methods. The useful consequence, if the claim holds, is that task-specific instruction tuning can be made much cheaper by optimizing for reward rather than token-matching similarity.","feed_headline":"ROSE picks 5% of data, beats full fine-tuning","feed_subtitle":"Why it matters: task-specific tuning could use a 5% subset chosen by preference-reward gradients instead of the full corpus.","key_machinery":"The load-bearing identity is Eq. 7, a checkpoint-weighted inner product between Adam gradients of the training loss and SGD gradients of the DPO-style reward loss on a few-shot preference validation set. A first-order Taylor expansion of validation-loss change under one SGD step gives the usual influence inner product; ROSE keeps that machinery but swaps the validation loss for pairwise preference loss, so the score estimates whether training on an example increases preference reward rather than decreases next-token perplexity. The implementation uses warm-up LoRA training, random projection of LoRA gradients to 8192 dimensions, and aggregation over four checkpoints to approximate the trajectory of training.","core_discovery":"On its own terms, ROSE claims that the right objective for instruction-data selection is reward, not distributional similarity: choose the subset that maximises expected reward on a few-shot preference validation set. ROSE converts that objective into the influence score $S(z, D'_{\\text{val}}) = \\sum_i \\eta_i \\langle \\nabla_{\\theta_i} L_{\\text{ROSE}}(D'_{\\text{val}}; \\theta_i), \\bar{\\nabla}_{\\theta_i} l(z; \\theta_i) \\rangle$, where $l$ is the SFT next-token loss, $L_{\\text{ROSE}}$ is the DPO-style pairwise preference loss computed with a reference model, and the sum runs over warm-up checkpoints. The paper reports that with Llama-2-7B, 5% of the corpus selected this way reaches win rates of 32.0 on SHP, 26.2 on SE, and 51.0 on HH, versus 22.7, 17.6, and 44.2 for full-data fine-tuning, and it shows the same pattern across six model variants. The mechanism is that preference loss, unlike next-token loss, is roughly monotonic in win rate across checkpoints, so the gradient inner product selects data that raise reward.","pith_inferences":["A natural extension the paper does not test is iterative selection: warm up on ROSE-selected data, recompute Eq. 7 scores, and select again, which could sharpen the correlation between score and downstream win rate.","The same swap—using DPO-style validation loss instead of next-token loss inside influence scoring—could apply to other influence-function uses such as mislabeled-sample detection or data removal whenever the target metric is preference rather than perplexity.","The paper's four-checkpoint correlations show that preference loss tracks win rate at the checkpoint level, but they do not directly show that individual examples are ranked correctly; a leave-one-out win-rate study over hundreds of scored examples would settle that."],"forward_implications":["A 5% subset selected by ROSE can replace full-data fine-tuning for a target task, cutting training cost by roughly 95% while improving win rate.","The non-monotonicity of next-token validation loss becomes a solvable problem: pairwise preference loss is the selection signal, and ROSE's checkpoint correlations suggest it tracks win rate far more closely.","Selection scores transfer across model families and sizes, so the expensive scoring pass on a small model can inform data selection for larger and instruction-tuned models.","Because the preference validation set is only a few shots per subtask, a practitioner can supply a handful of preferred and dispreferred responses and obtain task-specific selection without large human-labeled preference data.","ROSE fits into the existing gradient-based selection pipeline, so the extra computational cost over methods like LESS comes mainly from the preference-loss gradient on the validation side."],"supporting_citations":[{"why":"This paper supplies the gradient-influence selection recipe (warm-up, checkpoint averaging, Adam/SGD gradient features) that ROSE adapts, and it provides the LESS baseline that ROSE is compared against.","marker":"(Xia et al., 2024)"},{"why":"It provides the Direct Preference Optimization loss and closed-form reward that ROSE uses to turn the few-shot validation set into a preference signal.","marker":"(Rafailov et al., 2024)"},{"why":"It gives the TRAK random-projection method used to compress LoRA gradients into low-dimensional influence features.","marker":"(Park et al., 2023)"},{"why":"It supplies LoRA, the low-rank training scheme that makes warm-up training and gradient extraction on 7B-13B models affordable.","marker":"(Hu et al., 2021)"},{"why":"It provides the influence-function formulation that Eq. 2's Taylor expansion adapts to LLM training.","marker":"(Koh and Liang, 2017)"},{"why":"It underlies the Bradley-Terry preference model used in the DPO-style validation loss in Eq. 4.","marker":"(Bradley and Terry, 1952)"},{"why":"It is cited as evidence that next-token prediction loss is not monotonically related to task performance, motivating the reward-oriented objective.","marker":"(Zhou et al., 2024)"}],"fun_headline_variants":["ROSE: reward-based data selection cuts training data to 5%","ROSE selects data by reward, not similarity, and wins","5% of data chosen by reward signal beats full tuning","ROSE uses preference loss to pick 5% of data, beats full","Reward-oriented subset selection: 5% beats 100% for tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method stands or falls on the premise that a few dozen preferred- versus dispreferred-response pairs capture the task's true reward, so the gradient-alignment score in Eq. 7 correctly ranks which training examples will raise the model's real win rate.","fun_headline_variants_meta":{"raw":{"variants":["ROSE: reward-based data selection cuts training data to 5%","ROSE selects data by reward, not similarity, and wins","5% of data chosen by reward signal beats full tuning","ROSE uses preference loss to pick 5% of data, beats full","Reward-oriented subset selection: 5% beats 100% for tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00074,"raw_usage":{"total_tokens":3380,"prompt_tokens":1096,"completion_tokens":2284,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":712,"completion_tokens_details":{"reasoning_tokens":2191}},"tokens_in":712,"tokens_out":2284,"duration_ms":15461,"temperature":1.0,"reasoning_tokens":2191,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:09:40.165370+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute ROSE scores for every training point on one benchmark, fine-tune with the top 5%, the bottom 5%, and a random 5%, and compare win rates; the central claim predicts top beats random and random beats bottom. To test the score's ranking directly, measure leave-one-out win-rate impact for a few hundred scored examples and check for a strong positive correlation between ROSE score and measured impact.","supporting_citations":[{"cited_title":"Trak: Attributing model behavior at scale","cited_arxiv_id":null,"evidence_quote":"It gives the TRAK random-projection method used to compress LoRA gradients into low-dimensional influence features."}],"review_version":1}