Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuning

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Selecting 5% of instruction data by preference-gradient influence scores beats full-data fine-tuning on win rate.

desk verdict Solid empirical contribution with a real idea; the mechanism is under-validated but the main result is believable and the paper deserves a serious referee. read the letter →

arxiv 2412.00631 v2 pith:IX7NAH52 submitted 2024-12-01 cs.LG cs.AIcs.CL

Yang Wu , Huayi Zhang , Yizheng Jiao , Lin Ma , Xiaozhong Liu , Jinhong Yu , Dongyu Zhang , Dezhi Yu
show 1 more author
Wei Xu
This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords instructiontuningdataselectiondirectpreferenceoptimizationinfluencefunctionsrewardmaximizationwinrateLoRAtask-specificfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that existing data-selection methods for task-specific instruction tuning pick data by similarity to a validation set, which amounts to minimizing next-token prediction loss on that set—and next-token loss does not track real task performance. ROSE instead selects the training examples whose gradients align with the gradient of a pairwise preference loss on a small set of preferred and dispreferred responses. Using that alignment score, selecting just 5% of a general instruction corpus produces higher judged win rates than fine-tuning on the full corpus across the SHP, Stack Exchange, and HH-RLHF benchmarks, and higher win rates than prior selection methods. The useful consequence, if the claim holds, is that task-specific instruction tuning can be made much cheaper by optimizing for reward rather than token-matching similarity.

What carries the argument

The load-bearing identity is Eq. 7, a checkpoint-weighted inner product between Adam gradients of the training loss and SGD gradients of the DPO-style reward loss on a few-shot preference validation set. A first-order Taylor expansion of validation-loss change under one SGD step gives the usual influence inner product; ROSE keeps that machinery but swaps the validation loss for pairwise preference loss, so the score estimates whether training on an example increases preference reward rather than decreases next-token perplexity. The implementation uses warm-up LoRA training, random projection of LoRA gradients to 8192 dimensions, and aggregation over four checkpoints to approximate the trajectory of training.

What would settle it

Compute ROSE scores for every training point on one benchmark, fine-tune with the top 5%, the bottom 5%, and a random 5%, and compare win rates; the central claim predicts top beats random and random beats bottom. To test the score's ranking directly, measure leave-one-out win-rate impact for a few hundred scored examples and check for a strong positive correlation between ROSE score and measured impact.

Watch

Extended reading notes

Core claim

On its own terms, ROSE claims that the right objective for instruction-data selection is reward, not distributional similarity: choose the subset that maximises expected reward on a few-shot preference validation set. ROSE converts that objective into the influence score $S(z, D'_{\text{val}}) = \sum_i \eta_i \langle \nabla_{\theta_i} L_{\text{ROSE}}(D'_{\text{val}}; \theta_i), \bar{\nabla}_{\theta_i} l(z; \theta_i) \rangle$, where $l$ is the SFT next-token loss, $L_{\text{ROSE}}$ is the DPO-style pairwise preference loss computed with a reference model, and the sum runs over warm-up checkpoints. The paper reports that with Llama-2-7B, 5% of the corpus selected this way reaches win rates of 32.0 on SHP, 26.2 on SE, and 51.0 on HH, versus 22.7, 17.6, and 44.2 for full-data fine-tuning, and it shows the same pattern across six model variants. The mechanism is that preference loss, unlike next-token loss, is roughly monotonic in win rate across checkpoints, so the gradient inner product selects data that raise reward.

Load-bearing premise

The method stands or falls on the premise that a few dozen preferred- versus dispreferred-response pairs capture the task's true reward, so the gradient-alignment score in Eq. 7 correctly ranks which training examples will raise the model's real win rate.

Editorial extensions

If this is right

  • A 5% subset selected by ROSE can replace full-data fine-tuning for a target task, cutting training cost by roughly 95% while improving win rate.
  • The non-monotonicity of next-token validation loss becomes a solvable problem: pairwise preference loss is the selection signal, and ROSE's checkpoint correlations suggest it tracks win rate far more closely.
  • Selection scores transfer across model families and sizes, so the expensive scoring pass on a small model can inform data selection for larger and instruction-tuned models.
  • Because the preference validation set is only a few shots per subtask, a practitioner can supply a handful of preferred and dispreferred responses and obtain task-specific selection without large human-labeled preference data.
  • ROSE fits into the existing gradient-based selection pipeline, so the extra computational cost over methods like LESS comes mainly from the preference-loss gradient on the validation side.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not test is iterative selection: warm up on ROSE-selected data, recompute Eq. 7 scores, and select again, which could sharpen the correlation between score and downstream win rate.
  • The same swap—using DPO-style validation loss instead of next-token loss inside influence scoring—could apply to other influence-function uses such as mislabeled-sample detection or data removal whenever the target metric is preference rather than perplexity.
  • The paper's four-checkpoint correlations show that preference loss tracks win rate at the checkpoint level, but they do not directly show that individual examples are ranked correctly; a leave-one-out win-rate study over hundreds of scored examples would settle that.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ROSE, a data selection method for task-specific instruction tuning. Instead of minimizing next-token prediction loss on a validation set, ROSE constructs a few-shot preference validation set and uses the DPO pairwise loss as a reward signal. Influence scores are computed as the inner product between DPO validation gradients and SFT training gradients across LoRA checkpoints (Eq. 7), and the top 5% of training examples are selected for final SFT. Experiments on SHP, SE, and HH-RLHF with Llama-2, Llama-3.1, and Mistral models report that ROSE outperforms several baselines and is competitive with full-data fine-tuning. The paper also includes ablations on data percentage, checkpoint count, shot count, and transferability.

Significance. If the mechanism holds, the paper offers a practical recipe: a small number of preference pairs can guide the selection of a small SFT subset that beats full-data tuning. The evaluation is broad and mostly careful, with three trials per configuration, standard deviations, six model variants, multiple baselines, and a released code repository. The main unresolved issue is theoretical: Eq. 7 mixes two different losses without a derivation, and the paper's evidence for the reward-loss correlation (Fig. 4, four checkpoints per dataset) does not directly validate the influence score. Because the main in-distribution win rates are consistent and large, the result is worth pursuing, but the current manuscript needs additional validation of the selection mechanism before the central claim is fully supported.

major comments (3)
  1. [§4.2, Eq. (7)] The load-bearing score in Eq. (7) is the inner product between the DPO validation gradient ∇θ L_ROSE(D'_val; θ_i) and the SFT training gradient ∇̄θ l(z; θ_i). The Taylor argument in Sec. 3 (Eqs. (1)-(2)) justifies the inner-product approximation for a single loss used for both training and validation. ROSE does not provide a derivation or a formal argument for why the cross-loss inner product approximates the effect of an SFT training step on the DPO reward. Figure 4 is not direct evidence: it shows that the DPO validation loss correlates with test win rate over four checkpoints (SHP r=0.66, SE r=0.92, HH r=0.93), but it does not show that the scores from Eq. (7) rank individual examples by their effect on win rate, nor that the approximation survives multi-step AdamW optimization with LoRA and batch size 128. Please add either a derivation with explicit assumptions or a direct validation, e.g., comparing Eq. (7) scores against leave-one-out or top-k/bottom-k selection effects on win rate in a controlled SFT setting.
  2. [§5.2, Table 8] The transfer experiments show that ROSE-T (selection with Llama-2-7B, training on the target model) is often worse than random selection. For example, on SHP, ROSE-T is 29.6 vs. random 42.1 for Llama-2-13B, 29.7 vs. 37.4 for Llama-3.1-8B, 37.7 vs. 42.3 for Llama-3.1-8B-INS., 36.2 vs. 53.3 for Mistral-7B, and 53.9 vs. 58.4 for Mistral-7B-INS.-V0.3; on SE, ROSE-T is below random in all five transferred settings. The appendix text acknowledges this ('comparable or inferior to random selection'), but the main text says ROSE-T 'consistently outperforms LESS-T' without noting the failure against random. This is load-bearing for the generalizability claim, because selecting with a weaker model is the practical scenario when the target model is large. Please either temper the claim or provide an analysis of why transfer fails and under what conditions the selection score transfers.
  3. [Appendix B.1, Table 5; Limitations] The method's default configuration uses per-benchmark tuned shot counts (5 for SHP, 2 for SE, 1 for HH), and Figure 5 shows considerable fluctuation with shot count, especially on HH. The paper's own Limitations section states that specific shot numbers are tailored to individual datasets. As a result, the reported 5% advantage is not shown to be robust to this hyperparameter: without a principled way to choose the shot count or a fixed-shot ablation, a user of ROSE on a new task does not know whether the method will be effective. Please report results with a fixed shot count across benchmarks and/or an automatic criterion for choosing the shot count. In addition, the few-shot preference validation set is sampled from the same benchmark dataset as the test set, so the validation/test distribution overlap is a confound; please discuss how this affects the 'few-shot' claim.
minor comments (5)
  1. [§5.2, paragraph after Figure 3] The text says that using 5% data 'achieves performance that surpasses full-data fine-tuning,' but Table 2 shows several counterexamples (e.g., Llama-2-13B SHP: 44.3 vs. 48.4; Llama-3.1-8B HH: 58.6 vs. 60.6). Please qualify the claim to 'competitive with, and sometimes better than, full-data fine-tuning.'
  2. [Figure 4] The correlations are computed from only four checkpoints per dataset; please report confidence intervals or a permutation test, and note the small sample size in the caption. Also, the next-token prediction loss correlation is negative for SHP but positive for SE and HH, which is not discussed.
  3. [Global] Please proofread for typos and formatting issues: Table 1 caption has 'beselines' for 'baselines,' the title and abstract contain 'inStruction,' and Section 5.2 has 'funetuning' for 'fine-tuning.' The column headers in Table 8 are also difficult to parse because repeated groups are not visually separated.
  4. [References and Algorithm 1] The RDS baseline is cited to Zhang et al. (2018), which appears to be the deep-features perceptual metric paper; please verify that this is the correct source for representation-based data selection. In Algorithm 1, the output is Γ′ but the training step says 'Train model Γ on Dtrain' without defining how Γ′ is obtained from Γ.
  5. [§4.2.3] Equation (7) defines S(z, D′_val^{(j)}) for each subtask, but the text says ROSE takes the maximum score across subtasks without defining the operator formally. Please make the aggregation step explicit.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: ROSE's reward signal is defined externally by DPO, its selection score is a first-order influence approximation, and its claims are evaluated against held-out GPT-4-judged win rates rather than the quantities used for selection.

full rationale

The paper's derivation chain is self-contained and non-circular. The selection objective in Eq. 7 is an inner product between DPO validation gradients and SFT training gradients, derived from a first-order Taylor expansion in Section 3. That expansion is an approximation argument, not a restatement of the target quantity; the target is test win rate, which is measured with GPT-4 on separate test sets, not by the DPO loss used in selection. The few-shot preference validation set is constructed from open preference datasets, and although it is drawn from the same benchmark families as the test sets, the test data are fixed, held-out samples and the selection scores are not fitted to those test labels. The empirical correlation analysis in Figure 4 supports the choice of DPO loss as a proxy, but it is not a circular step: it is an external validity check, and its small sample size (four checkpoints per dataset) is a robustness limitation, not a definitional reduction. The transfer results in Table 8 are weaker in some settings, but weakened transfer is an empirical outcome, not evidence that the prediction is equivalent to its inputs. Self-citations in the related work are for context, and the load-bearing methodological citations (LESS, TRAK, DPO) are external prior work, not the present authors' own unverified results. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from self-citations, and no ansatz is smuggled in via self-citation. The paper's own Limitations section notes dependence on preference validation quality and per-benchmark shot tuning, but those are dataset-sensitivity concerns rather than circularity. Therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central method rests on several unproved or borrowed premises: DPO's implicit reward formula, the first-order influence approximation, the cross-loss gradient inner product heuristic, and the transferability of a small preference validation set. These are not free-standing derivations; they are assumptions imported from LESS/DPO/TRAK or asserted in Sec. 4. The free parameters are mostly standard hyperparameters plus per-benchmark shot counts and the unreported DPO beta, which are tuned or left unspecified.

free parameters (5)
  • beta (DPO temperature)
    In Eqs. 3-4, beta controls the deviation from the reference model and scales the reward. The paper never reports its value, yet it enters every validation gradient (Eq. 5) and therefore every influence score.
  • few-shot shot counts per benchmark = SHP=5, SE=2, HH=1
    Selected per dataset to maximize performance (Appendix B.1); the paper's limitation section admits these are tailored rather than universal.
  • number of checkpoints N = 4
    Default N=4 chosen because Table 3 shows it outperforms N=1; affects gradient feature richness and final selection.
  • LoRA rank and alpha = rank=128, alpha=512
    Fixed across all models for training efficiency; not searched but hand-chosen and affects gradient geometry.
  • warmup data percentage = 5% randomly selected
    The selection model is initially trained on a random 5% subset; this choice shapes all subsequent gradient features and checkpoints.
assumptions (5)
  • standard math DPO implicit reward characterization: r(x,y) = beta log(pi_theta(y|x)/pi_ref(y|x)) + beta log Z(x)
    Taken as a theorem from Rafailov et al. (2024); used without proof as the basis for Eq. 3.
  • domain assumption First-order Taylor expansion approximates the effect of an SGD step on validation loss via gradient inner product (Eqs. 1-2)
    Standard influence-function approximation, valid only for small steps and for the same loss; imported from Koh and Liang and LESS.
  • ad hoc to paper Cross-loss inner product between DPO validation gradients and SFT training gradients approximates influence on reward
    The core heuristic of ROSE; Eq. 7 is asserted after Eq. 2 without derivation for mixed losses or Adam/TRAK projections.
  • domain assumption A few-shot preference validation set sampled from the benchmark reflects true task performance
    Needed so selection on D'_val transfers to D_test; the authors note in Limitations that the quality of preference data is crucial.
  • domain assumption LoRA fine-tuning and TRAK projection preserve the gradient information needed for influence ranking
    Borrowed from LESS/TRAK; storage and compute constraints force this approximation, but it is not separately validated here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuning." pith.science (2026). https://pith.science/paper/IX7NAH52

@misc{pith2026241200631,
  author       = {Pith},
  title        = {Pith review of: ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IX7NAH52}},
  note         = {Machine review of arXiv:2412.00631}
}
read the original abstract

Instruction tuning has underscored the significant potential of large language models (LLMs) in producing more human controllable and effective outputs in various domains. In this work, we focus on the data selection problem for task-specific instruction tuning of LLMs. Prevailing methods primarily rely on the crafted similarity metrics to select training data that aligns with the test data distribution. The goal is to minimize instruction tuning loss on the test data, ultimately improving performance on the target task. However, it has been widely observed that instruction tuning loss (i.e., cross-entropy loss for next token prediction) in LLMs often fails to exhibit a monotonic relationship with actual task performance. This misalignment undermines the effectiveness of current data selection methods for task-specific instruction tuning. To address this issue, we introduce ROSE, a novel Reward-Oriented inStruction data sElection method which leverages pairwise preference loss as a reward signal to optimize data selection for task-specific instruction tuning. Specifically, ROSE adapts an influence formulation to approximate the influence of training data points relative to a few-shot preference validation set to select the most task-related training data points. Experimental results show that by selecting just 5\% of the training data using ROSE, our approach can achieve competitive results compared to fine-tuning with the full training dataset, and it surpasses other state-of-the-art data selection methods for task-specific instruction tuning. Our qualitative analysis further confirms the robust generalizability of our method across multiple benchmark datasets and diverse model architectures.

Figures

Figures reproduced from arXiv: 2412.00631 by the authors.

Figure 2
Figure 2. Illustration of ROSE. We generate suboptimal responses to create a preference validation set, then use [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Results of different data selection percent [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Comparison of our method with traditional data selection methods for LLM instruction tuning. We [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Comparison of our method with LESS and random data selection across different numbers of shots in the [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CLID-MU: Cross-Layer Information Divergence Based Meta Update Strategy for Learning with Noisy Labels

    cs.LG 2025-07 conditional novelty 6.0 of 10

    CLID-MU replaces the clean meta-dataset in meta-learning with an unsupervised cross-layer divergence metric, improving noisy-label and semi-supervised results on several benchmarks.

  2. CLaDMoP: Learning Transferrable Models from Successful Clinical Trials via LLMs

    cs.LG 2025-05 conditional novelty 6.0 of 10

    CLaDMoP pre-trains a two-branch model on successful clinical trials with a pair-matching objective, then fine-tunes it to predict trial success, outperforming prior models on the TOP benchmark.

  3. Active Domain Knowledge Acquisition with 100-Dollar Budget: Enhancing LLMs via Cost-Efficient, Expert-Involved Interaction in Sensitive Domains

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    A budget-aware framework (PU-ADKA) selects which domain expert an LLM should query under a fixed $100 budget, improving specialized-domain answers at low cost.

Reference graph

Works this paper leans on

59 extracted references · 17 canonical work pages · cited by 3 Pith papers

  1. [1]

    Llama model card, 2024

    AI@Meta . Llama model card, 2024

    work page 2024

  2. [2]

    The fellowship of the llms: Multi-agent workflows for synthetic preference optimization dataset generation

    Samee Arif, Sualeha Farid, Abdul Hameed Azeemi, Awais Athar, and Agha Ali Raza. The fellowship of the llms: Multi-agent workflows for synthetic preference optimization dataset generation. arXiv preprint arXiv:2408.08688, 2024

    arXiv 2024

  3. [3]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  4. [4]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  5. [5]

    Automated data curation for robust language model fine-tuning

    Jiuhai Chen and Jonas Mueller. Automated data curation for robust language model fine-tuning. arXiv preprint arXiv:2403.12776, 2024

    arXiv 2024

  6. [6]

    Free dolly: Introducing the world’s first truly open instruction-tuned llm

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned llm. Company Blog of Databricks, 2023

    2023

  7. [7]

    Free dolly: Introducing the world’s first truly open instruction-tuned llm

    Databricks. Free dolly: Introducing the world’s first truly open instruction-tuned llm. https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm, 2023. Blog post

    work page 2023

  8. [8]

    Misdetect: Iterative mislabel detection using early loss

    Yuhao Deng, Chengliang Chai, Lei Cao, Nan Tang, Jiayi Wang, Ju Fan, Ye Yuan, and Guoren Wang. Misdetect: Iterative mislabel detection using early loss. Association for Computing Machinery (ACM), 2024

    work page 2024

Show all 59 references
  1. [9]

    Mods: Model-oriented data selection for instruction tuning

    Qianlong Du, Chengqing Zong, and Jiajun Zhang. Mods: Model-oriented data selection for instruction tuning. arXiv preprint arXiv:2311.15653, 2023

    2023 arXiv

  2. [10]

    Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. In Forty-first International Conference on Machine Learning

  3. [11]

    What neural networks memorize and why: Discovering the long tail via influence estimation

    Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33: 0 2881--2891, 2020

    2020

  4. [12]

    Shapley values for feature selection: The good, the bad, and the axioms

    Daniel Fryer, Inga Str \"u mke, and Hien Nguyen. Shapley values for feature selection: The good, the bad, and the axioms. Ieee Access, 9: 0 144352--144360, 2021

    2021

  5. [13]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022

    2022 arXiv

  6. [14]

    Teaching large language models to reason with reinforcement learning

    Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.04642, 2024

    2024 arXiv

  7. [15]

    A demonstration of autood: a self-tuning anomaly detection system

    Dennis Hofmann, Peter VanNostrand, Huayi Zhang, Yizhou Yan, Lei Cao, Samuel Madden, and Elke Rundensteiner. A demonstration of autood: a self-tuning anomaly detection system. Proceedings of the VLDB Endowment, 15 0 (12): 0 3706--3709, 2022

    2022

  8. [16]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  9. [17]

    Uce-fid: Using large unlabeled, medium crowdsourced-labeled, and small expert-labeled tweets for foodborne illness detection

    Ruofan Hu, Dongyu Zhang, Dandan Tao, Huayi Zhang, Hao Feng, and Elke Rundensteiner. Uce-fid: Using large unlabeled, medium crowdsourced-labeled, and small expert-labeled tweets for foodborne illness detection. In 2023 IEEE International Conference on Big Data (BigData), pp.\ 5...

    2023

  10. [18]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  11. [19]

    Aligning large language models with self-generated preference data

    Dongyoung Kim, Kimin Lee, Jinwoo Shin, and Jaehyung Kim. Aligning large language models with self-generated preference data. arXiv preprint arXiv:2406.04412, 2024 a

    2024 arXiv

  12. [20]

    Health-llm: Large language models for health prediction via wearable sensor data

    Yubin Kim, Xuhai Xu, Daniel McDuff, Cynthia Breazeal, and Hae Won Park. Health-llm: Large language models for health prediction via wearable sensor data. arXiv preprint arXiv:2401.06866, 2024 b

    2024 arXiv

  13. [21]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  14. [22]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp.\ 1885--1894. PMLR, 2017

    2017

  15. [23]

    o pf, Yannic Kilcher, Dimitri von R \

    Andreas K \"o pf, Yannic Kilcher, Dimitri von R \"u tte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Rich \'a rd Nagyfi, et al. Openassistant conversations-democratizing large language model alignment. Advances in Neural Info...

    2024

  16. [24]

    Pretraining language models with human preferences

    Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning, pp.\ 17506--17533. PMLR, 2023

    2023

  17. [25]

    Huggingface h4 stack exchange preference dataset, 2023

    Nathan Lambert, Lewis Tunstall, Nazneen Rajani, and Tristan Thrush. Huggingface h4 stack exchange preference dataset, 2023. URL https://huggingface.co/datasets/HuggingFaceH4/stack-exchange-preferences

    2023

  18. [26]

    Adapting large language models for education: Foundational capabilities, potentials, and challenges

    Qingyao Li, Lingyue Fu, Weiming Zhang, Xianyu Chen, Jingwei Yu, Wei Xia, Weinan Zhang, Ruiming Tang, and Yong Yu. Adapting large language models for education: Foundational capabilities, potentials, and challenges. arXiv preprint arXiv:2401.08664, 2023

    2023 arXiv

  19. [27]

    Lipo: Listwise preference optimization through learning-to-rank

    Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, et al. Lipo: Listwise preference optimization through learning-to-rank. arXiv preprint arXiv:2402.01878, 2024

    2024 arXiv

  20. [28]

    The flan collection: Designing data and methods for effective instruction tuning

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pp.\ 22631--22648. PMLR, 2023

    2023

  21. [29]

    Post-hoc interpretability for neural nlp: A survey

    Andreas Madsen, Siva Reddy, and Sarath Chandar. Post-hoc interpretability for neural nlp: A survey. ACM Computing Surveys, 55 0 (8): 0 1--42, 2022

    2022

  22. [30]

    Smaller language models are capable of selecting instruction-tuning training data for larger language models

    Dheeraj Mekala, Alex Nguyen, and Jingbo Shang. Smaller language models are capable of selecting instruction-tuning training data for larger language models. arXiv preprint arXiv:2402.10430, 2024

    2024 arXiv

  23. [31]

    Simpo: Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024

    2024 arXiv

  24. [32]

    Gpt model card, 2024

    OPENAI . Gpt model card, 2024

    2024

  25. [33]

    Trak: Attributing model behavior at scale

    Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. In International Conference on Machine Learning (ICML), 2023

    2023

  26. [34]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  27. [35]

    The probabilistic relevance framework: Bm25 and beyond

    Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3 0 (4): 0 333--389, 2009

    2009

  28. [36]

    Scale efficiently: Insights from pre-training and fine-tuning transformers

    Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers. arXiv preprint arXiv:2109.10686, 2021

    2021 arXiv

  29. [37]

    Fine-tuning language models for factuality

    Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Fine-tuning language models for factuality. arXiv preprint arXiv:2311.08401, 2023

    2023 arXiv

  30. [38]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  31. [39]

    Facet: Robust counterfactual explanation analytics

    Peter M VanNostrand, Huayi Zhang, Dennis M Hofmann, and Elke A Rundensteiner. Facet: Robust counterfactual explanation analytics. Proceedings of the ACM on Management of Data, 1 0 (4): 0 1--27, 2023

    2023

  32. [40]

    A survey on data selection for llm instruction tuning

    Jiahao Wang, Bolin Zhang, Qianlong Du, Jiajun Zhang, and Dianhui Chu. A survey on data selection for llm instruction tuning. arXiv preprint arXiv:2402.05123, 2024

    2024 arXiv

  33. [41]

    How far can camels go? exploring the state of instruction tuning on open resources

    Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing...

    2023

  34. [42]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  35. [43]

    Instructiongpt-4: A 200-instruction paradigm for fine-tuning minigpt-4

    Lai Wei, Zihao Jiang, Weiran Huang, and Lichao Sun. Instructiongpt-4: A 200-instruction paradigm for fine-tuning minigpt-4. arXiv preprint arXiv:2308.12067, 2023

    2023 arXiv

  36. [44]

    Community-based hierarchical positive-unlabeled (pu) model fusion for chronic disease prediction

    Yang Wu, Xurui Li, Xuhong Zhang, Yangyang Kang, Changlong Sun, and Xiaozhong Liu. Community-based hierarchical positive-unlabeled (pu) model fusion for chronic disease prediction. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, ...

    2023

  37. [45]

    Knowledge-infused legal wisdom: Navigating llm consultation through the lens of diagnostics and positive-unlabeled reinforcement learning

    Yang Wu, Chenghao Wang, Ece Gumusel, and Xiaozhong Liu. Knowledge-infused legal wisdom: Navigating llm consultation through the lens of diagnostics and positive-unlabeled reinforcement learning. arXiv preprint arXiv:2406.03600, 2024

    2024 arXiv

  38. [46]

    Less: Selecting influential data for targeted instruction tuning

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333, 2024

    2024 arXiv

  39. [47]

    Data selection for language models via importance resampling

    Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 0 34201--34227, 2023

    2023

  40. [48]

    Some things are more cringe than others: Preference optimization with the pairwise cringe loss

    Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682, 2023

    2023 arXiv

  41. [49]

    Rrhf: Rank responses to align language models with human feedback

    Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems, 36, 2024

    2024

  42. [50]

    Lancet: labeling complex data at scale

    Huayi Zhang, Lei Cao, Samuel Madden, and Elke Rundensteiner. Lancet: labeling complex data at scale. Proceedings of the VLDB Endowment, 14 0 (11), 2021 a

    2021

  43. [51]

    Elite: Robust deep anomaly detection with meta gradient

    Huayi Zhang, Lei Cao, Peter VanNostrand, Samuel Madden, and Elke A Rundensteiner. Elite: Robust deep anomaly detection with meta gradient. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp.\ 2174--2182, 2021 b

    2021

  44. [52]

    Metastore: Analyzing deep learning meta-data at scale

    Huayi Zhang, Binwei Yan, Lei Cao, Samuel Madden, and Elke Rundensteiner. Metastore: Analyzing deep learning meta-data at scale. Proceedings of the VLDB Endowment, 17 0 (6): 0 1446--1459, 2024

    2024

  45. [53]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 586--595, 2018

    2018

  46. [54]

    Lima: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024

    2024

  47. [55]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

  48. [56]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  49. [57]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  50. [58]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  51. [59]

    over night\

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.