Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Sequential preference optimization makes Shapley-style data valuation for LLM alignment affordable by reconstructing any coalition policy from singleton models alone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 16:52 UTC pith:BL232D7U

load-bearing objection The algebra is correct; the application is timely, but the paper changes the game without validating it. the 3 major comments →

arxiv 2512.15765 v3 pith:BL232D7U submitted 2025-12-12 cs.LG cs.GTstat.ML

Shapley-based Data Valuation for LLM Alignment via Sequential Preference Optimization

M\'elissa Tamine , Otmane Sakhi , Benjamin Heymann , Maxime Vono , Patrick Loiseau This is my paper
classification cs.LG cs.GTstat.ML
keywords Shapley valuedata valuationLLM alignmentDirect Preference Optimizationlanguage model arithmeticsequential DPOpreference optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that Shapley data valuation for LLM preference alignment can be made affordable by using a sequential training procedure that exposes an algebraic structure in Direct Preference Optimization and related objectives. Applied source by source, DPO at convergence makes the policy aligned on any coalition of datasets exactly recoverable from the individually aligned models alone, through arithmetic on log-probabilities. That identity converts the exponential number of coalition fine-tunings required by the Shapley value into a linear number of singleton fine-tunings, with all coalition utilities evaluated at inference time. The paper demonstrates the resulting per-source scores on four preference-dataset subsets under helpfulness and harmlessness rewards, showing that different sources have distinct contribution profiles. A sympathetic reader would care because it turns an otherwise prohibitive attribution problem into a tractable one, conditional on the convergence idealization the authors themselves flag.

Core claim

Under exact optimization, sequential DPO on a coalition S converges to the closed-form policy π*_S(y|x) ∝ exp((1/β) Σ_{ℓ∈S} r̂_ℓ(x,y)) π0(y|x), where each r̂_ℓ is the reward model learned from source ℓ alone. Rewriting log-probabilities as scores, this is equivalent to π*_S(y|x) ∝ exp(Σ_{ℓ∈S} s_ℓ(x,y) + (1−|S|) s0(x,y)), with s_ℓ and s0 the log-probabilities of the singleton-aligned models and the reference model. Therefore any coalition policy can be obtained exactly from the singleton models and the reference, with no coalition-specific training. The paper presents this as a formal instance of language model arithmetic: dataset-coalition union maps to reward-model addition and to a policy-

What carries the argument

Sequential Preference Optimization. Algorithm 1 feeds the current policy into the next source's DPO step, iterating until all sources in a coalition are seen. At convergence each step contracts to the KL-regularized optimal policy, and the composition collapses to Equation (6), the product form exp((1/β)Σ r̂_ℓ) π0. The companion identity in log-probability space — subtracting (|S|−1) copies of the reference log-probabilities — is the exact policy-reconstruction formula used to evaluate every coalition's utility. This identity is load-bearing: it converts an exponential enumeration of fine-tunings into n trainings plus 2^n inference-time evaluations, and makes the coalition-agnostic data sign

Load-bearing premise

The construction stands on the premise that sequential DPO at convergence produces exactly the coalition policy whose implicit reward is the sum of the per-source rewards, and that finite-sample LoRA-trained models stay close enough to that ideal; the paper itself flags convergence-not-reached as an open practical question.

What would settle it

Train a single model with standard DPO on the union of two sources, and on the side train each source separately and reconstruct the coalition by the arithmetic composition; compare their implicit rewards on held-out prompts or their preference accuracy. If the union-trained model's behavior deviates systematically and predictably from the reconstruction beyond sampling noise, the exact-recovery claim fails in realistic conditions, and the Shapley values computed from reconstructed policies are not the values of actual coalition-aligned models.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Shapley data valuation for DPO-style alignment needs only n fine-tunings, not 2^n; all coalition utilities are obtained by combining adapters at inference time.
  • The same additive-reward property covers formal preference objectives based on log-policy ratios (DPO, IPO), so the acceleration is not tied to a single algorithm.
  • Evaluating every coalition model on multiple rewards yields a per-source signature that can separate helpfulness-contributors from harmlessness-contributors, and negative values flag sources that systematically hurt a given objective.
  • When the number of sources grows, the cheap coalition utilities can be subsampled with Shapley approximation schemes such as permutation sampling or regression, trading inference calls instead of training runs.
  • Because the coalition policy depends only on the set S and not on the order of the sequential updates, the procedure is naturally order-invariant under the convergence assumption.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: one testable extension is to measure how the reconstructed coalition policy diverges from a standard DPO-trained union model as a function of LoRA rank, number of epochs, and β; the smaller the divergence, the wider the regime where these Shapley scores are faithful.
  • Beyond the paper: if the additive-reward identity holds only approximately in practice, the resulting numbers are better described as arithmetic-approximation Shapley values; the exact Shapley value would require true coalition training, which the method no longer performs.
  • Beyond the paper: the morphism viewpoint suggests that other preference aggregation rules might admit similar commutative compositions; the paper flags this as an open question worth probing with both theory and experiments.
  • Beyond the paper: in a data-market setting, the inference-time reconstruction means a server could train singleton adapters once and answer many clients' coalition-valuation queries without retraining — but only if clients accept the convergence idealization in the product-form reward.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes an efficient Shapley-based data-valuation method for LLM preference alignment. It introduces Sequential DPO, an offline procedure in which DPO is applied to each preference dataset in sequence. The central observation is that, under exact optimization, the resulting coalition policy has the closed form pi*_S(y|x) proportional to exp((1/beta) * sum_{l in S} rhat_l(x,y)) * pi0(y|x) (Eq. 6), and can therefore be reconstructed from the individually aligned policies via a simple arithmetic combination rule. This reduces the number of required fine-tunings from exponential to linear in the number of data sources. The paper demonstrates the approach on four UltraFeedback subsets using SmolLM-135M with LoRA adapters, computing approximate Shapley values under helpfulness and harmlessness reward models (Eq. 7, Figure 2). Section 5 frames the relation between sequential DPO and standard DPO-on-union as future work.

Significance. If the paper's underlying game is taken to be Sequential DPO, the composition identity is elegant and the derivation is largely self-contained; the reduction in the number of fine-tunings is a genuine practical advantage, and the paper is explicit in Section 5 that key approximation questions remain open. However, the paper currently blurs the distinction between standard DPO-on-union alignment, used to define the utility in Section 2.1, and the Sequential DPO game for which Eq. (7) is exact. The empirical section validates only the arithmetic reconstructions and never trains a coalition model by either standard DPO or Sequential DPO, so it cannot support the claim that the result approximates the Shapley values of standard DPO alignment. The contribution is therefore a promising but as-yet-unvalidated redefinition of the valuation game rather than a demonstrated approximation of the original one.

major comments (3)
  1. [§2.1, §3, Eq. (6)] The valuation game is changed without establishing equivalence. Equation (2) defines u(S) through a policy pi_S obtained by a fixed alignment procedure (e.g., DPO) on the union of datasets. Section 3 replaces this with Sequential DPO, whose exact optimum is Eq. (6), pi*_S ∝ exp((1/beta) sum_{l in S} rhat_l) pi0. For |S|>1, this is not generally the optimum of DPO applied to the union: the Bradley-Terry MLE reward on the pooled data is not the sum of per-source MLE rewards. Thus Eq. (7) is exact for a different game, and the paper's own Section 5 lists 'how the sequential resolution of an optimization problem approximates the original problem' as future work. As submitted, the abstract and Section 1 overstate the connection to standard DPO alignment. A revision should either explicitly define the valuation game as Sequential DPO (and qualify the claims accordingly) or provide an empirical
  2. [§4, Figure 2] The empirical demonstration validates only the arithmetic construction. No coalition model is trained by standard DPO or by Sequential DPO, so Figure 2 cannot detect bias introduced by the reconstruction and cannot support the claim that the computed values approximate the Shapley values of DPO alignment. In addition, utilities are estimated from 128 prompts with no repeated seeds or confidence intervals, so statements such as 'evol_instruct provides helpfulness at the cost of the most significant negative harmless value' are not supported with uncertainty quantification. I request a comparison of pi-hat_S with actual coalition-training baselines on at least singleton and pairwise coalitions, and error bars on the Shapley estimates.
  3. [§4, adapter combination] The adapter-combination rule is not specified. The text says only that coalition policies pi-hat_S are built by 'combining the corresponding LoRA adapters.' Equation (7) is exact only given the reconstructed policies, so the implementation must state which arithmetic operation is used (e.g., adding adapter weights, interpolating, or combining output log-probabilities) and ideally verify that it matches the product-form policy in Eq. (6). Without this, the experiment is neither reproducible nor connected to the theoretical derivation.
minor comments (6)
  1. [Algorithm 1] The output index is off by one. The loop starts with k=0 and increments k after computing pi_{k+1}; after the loop k=|S|, so the output pi_{k+1} is pi_{|S|+1}, while the last computed policy is pi_{|S|}. Initialize k=1 or output pi_k.
  2. [References] References [9] and [10] appear to be the same paper (Dekoninck et al., Controlled Text Generation via Language Model Arithmetic). Keep one citation.
  3. [Title] The arXiv listing title ('Shapley-based Data Valuation for LLM Alignment via Sequential Preference Optimization') and the full-text title ('Data Valuation for LLM Fine-Tuning: Efficient Shapley Value Approximation via Language Model Arithmetic') should be aligned.
  4. [§4] The number of preference pairs per source and the exact construction of the evaluation prompts are not reported. The 'unused portion of UltraFeedback' should be checked for overlap with the four training subsets.
  5. [Figure 2] The statement that the diagonal y=x indicates 'perfect agreement' is only meaningful if the two reward models produce values on the same scale; please clarify or rescale.
  6. [Figure 1 / §2.1] The text alternates between 'group' and 'semi-group'. Since the operations do not necessarily have inverses, use 'semi-group' consistently.

Circularity Check

1 steps flagged

Coalition utility is redefined by Sequential DPO, so the efficient Shapley is exact for the new game; the original DPO-on-union game is left as future work.

specific steps
  1. self definitional [Section 2.1 (Eq. 2) vs Section 3 (Eq. 6) and Section 4 (Eq. 7)]
    "let π_S denote the model obtained by applying a fixed alignment procedure (e.g., DPO)... to the union of the corresponding datasets ... at convergence, the aligned coalition policy satisfies the closed form π*_S(y|x)∝exp((1/β)Σ_{ℓ∈S} r̂_ℓ(x,y))π0(y|x),(6) ... for any coalition S, the coalition-aligned model π*_S can be recovered exactly using only the individually aligned models {π*_ℓ}_ℓ∈S and the reference model π0."

    Equation (2) defines coalition utility for the original game, with π_S produced by DPO on the union of datasets. Section 3 replaces the coalition procedure with Sequential DPO and calls its optimum (Eq. 6) 'the aligned coalition policy'. The reconstruction identity that enables the whole method is exactly Eq. (6), so the linear-cost Shapley formula (Eq. 7) is exact for the Sequential-DPO game by construction. The paper's own Section 5 lists 'how the sequential resolution of an optimization problem approximates the original problem' as future work, confirming that the original DPO-on-union game is not shown to have this product optimum. Thus the central 'recovery' reduces to a redefinition of the coalition-alignment game, not an external prediction.

full rationale

The derivation of the additive composition rule is not fitted: Eq. (6) follows algebraically from the DPO/RLHF closed-form optimal policy and the sequential update rule. That part is self-contained and not circular. However, the paper begins by defining u(S)=v(π_S) with π_S from a fixed alignment procedure on the union of datasets, then silently substitutes Sequential DPO's optimum for π_S. The efficient Shapley computation is therefore exact for the redefined Sequential-DPO game, not for the standard DPO-on-union game. The paper explicitly acknowledges this gap in Section 5 as future work. The empirical section also constructs coalition policies from the same arithmetic formula whose validity is at issue, so it cannot independently confirm equivalence to true coalition alignments. There is no load-bearing self-citation, no fitted parameter renamed as a prediction, and no externally forced uniqueness claim; the circularity is a redefinition of the characteristic function combined with a self-referential validation. Hence a moderate score of 3 is appropriate.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

Central claim depends on 3 free hyperparameters and 5 assumptions; no new entities.

free parameters (3)
  • DPO inverse temperature beta = 0.1
    Chosen as TRL default; enters the reward-composition exponent and thus the reconstructed coalition policies and Shapley values; no sensitivity analysis.
  • LoRA rank r and scaling alpha = r=8, alpha=16
    Chosen hyperparameters for adapters; the arithmetic combination of adapters is assumed to approximate sequential DPO, but rank/scale effects are not studied.
  • DPO training schedule = 4 epochs, batch size 32, grad accum 4, lr 2e-5
    DPO fine-tuning is stopped early, far from the exact optimum invoked by the theory; these choices determine the actual per-source policies.
axioms (5)
  • domain assumption DPO's KL-regularized RL equivalence yields optimal policy pi* proportional to exp(r/beta) pi0 (Eq. 5).
    Standard DPO result; the paper builds on it without re-deriving.
  • ad hoc to paper Sequential DPO converges to a policy independent of dataset order, with reward equal to sum of per-dataset rewards (Eq. 6).
    This is the load-bearing premise; the paper states 'assuming convergence' but does not prove or empirically verify order-independence in finite training.
  • ad hoc to paper A coalition's utility should be defined by the arithmetic-combined policy (LM arithmetic) rather than by standard DPO on the union.
    The Shapley values are computed for this redefined alignment procedure; no baseline shows it equals standard DPO-on-union.
  • ad hoc to paper LoRA adapters trained per source can be combined by exponentiating/log-adding to approximate the sequential coalition policy.
    Used in Section 4; no verification that adapter arithmetic matches full sequential fine-tuning.
  • domain assumption Helpfulness/harmlessness reward models accurately measure alignment value.
    Evaluation uses reward models from prior RLHF work; their fidelity is taken for granted.

pith-pipeline@v1.3.0-alltime-deepseek · 10394 in / 13191 out tokens · 116175 ms · 2026-08-03T16:52:02.962508+00:00 · methodology

0 comments
read the original abstract

Data valuation is a natural framework for understanding which preference datasets matter most when aligning a Large Language Model (LLM) using multiple sources. The standard game-theoretic approach assigns each dataset a contribution score via the Shapley value. In practice, however, Shapley-based valuation is computationally prohibitive because it requires fine-tuning a separate model for every possible coalition of preference datasets, i.e., an exponential number of alignments. We address this challenge for a broad family of preference-optimization objectives, including DPO and IPO, that learn directly from log-policy ratios with respect to a reference policy. We introduce Sequential Preference Optimization, an offline procedure that applies existing preference optimization methods sequentially, source by source, updating the current policy after each dataset. Under exact optimization, this procedure yields an additive composition rule in reward space and an equivalent arithmetic composition rule in policy space. This observation enables an efficient approximation of the Shapley value: we train one model per preference dataset and reconstruct coalition policies at inference time from the singleton models, reducing the required alignments from exponential to linear in the number of sources. Leveraging this property, we compute Shapley values for several real-world preference datasets and reveal how each source drives model alignment.

Figures

Figures reproduced from arXiv: 2512.15765 by Benjamin Heymann, Maxime Vono, M\'elissa Tamine, Otmane Sakhi, Patrick Loiseau.

Figure 1
Figure 1. Figure 1: We can equip the collections of datasets, the set of reward models, and the set of policies with binary operators to give them [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Approximate Shapley values of 4 UltraFeedback data sources (flan_v2_niv2, sharegpt, evol_instruct, ultrachat) under two rewards. Each point corresponds to a data source. The x-axis shows its Shapley value for the helpfulness reward, and the y-axis for the harmlessness reward. The diagonal 𝑦 = 𝑥 indicates perfect agreement between the two rewards on the relative importance of each source. method makes it pr… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. An Asymptotic Analysis of the Shapley Value for Dataset Valuation

    cs.GT 2026-07 conditional novelty 7.0

    Under smooth RKHS embedding utilities, a fixed owner's Shapley value is O(1/I)-close in L1 to an explicit leading term of scale (log I)/I driven by a first-order population signal.

Reference graph

Works this paper leans on

46 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [1]

    Anish Agarwal, Munther Dahleh, and Tuhin Sarkar. 2019. A Marketplace for Data: An Algorithmic Solution. InProceedings of the 2019 ACM Conference on Economics and Computation(Phoenix, AZ, USA)(EC ’19). Association for Computing Machinery, New York, NY, USA, 701–726. https://doi.org/10.1145/3328526.3329589

    arXiv 2019

  2. [2]

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. 2024. Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, ...

    doi:10.18653/v 2024

  3. [3]

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Leandro von Werra, and Thomas Wolf. 2024. SmolLM - blazingly fast and remarkably powerful

    2024

  4. [4]

    Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeffrey Wu. 2024. Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. InProceedings of the 41st International Conference on Machine Learning (Proceeding...

    2024

  5. [5]

    Kash, Tal Moran, and Salil Vadhan

    Yiling Chen, Stephen Chong, Ian A. Kash, Tal Moran, and Salil Vadhan. 2016. Truthful Mechanisms for Agents That Value Privacy.ACM Trans. Econ. Comput.4, 3, Article 13 (March 2016), 30 pages. https://doi.org/10.1145/2892555

    work page doi:10.1145/2892555 2016

  6. [6]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep Reinforcement Learning from Human Preferences. InAdvances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc. https://proceedings.neurips...

    2017

  7. [7]

    Holliday, Bob M

    Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H. Holliday, Bob M. Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, Emanuel Tewolde, and William S. Zwicker. 2024. Position: social choice should guide AI alignment in dealing with diverse human feedback. InProceedings of the 41st International Conference on Mach...

    2024

  8. [8]

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. UltraFeedback: Boosting Language Models with High-quality Feedback. arXiv:2310.01377 [cs.CL]

    Pith/arXiv arXiv 2023

  9. [9]

    Jasper Dekoninck, Marc Fischer, Luca Beurer-Kellner, and Martin Vechev. 2024. Controlled Text Generation via Language Model Arithmetic. In International Conference on Representation Learning, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024. 35011–35038. https://proceedings.iclr.cc/paper_files/paper/2024/file/96aad3299d1...

    2024

  10. [10]

    Jasper Dekoninck, Marc Fischer, Luca Beurer-Kellner, and Martin Vechev. 2024. Controlled Text Generation via Language Model Arithmetic. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=SLw9fp4yI6

    2024

  11. [11]

    Luise Ge, Daniel Halpern, Evi Micha, Ariel D Procaccia, Itai Shapira, Yevgeniy Vorobeychik, and Junlin Wu. 2024. Axioms for ai alignment from human feedback.Advances in Neural Information Processing Systems37 (2024), 80439–80465

    2024

  12. [12]

    Amirata Ghorbani and James Zou. 2019. Data Shapley: Equitable Valuation of Data for Machine Learning. InProceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 2242–2251. https://proceedings.mlr.press/v97/ghorbani19c.html

    2019

  13. [13]

    Yexiao He, Ziyao Wang, Zheyu Shen, Guoheng Sun, Yucong Dai, Yongkai Wu, Hongyi Wang, and Ang Li. 2024. SHED: shapley-based automated dataset refinement for instruction fine-tuning. InProceedings of the 38th International Conference on Neural Information Processing Systems(Vancouver, BC, Canada)(NIPS ’24). Curran Associates Inc., Red Hook, NY, USA, Article...

    2024

  14. [14]

    Benjamin Heymann. 2025. Adaptive Preference Aggregation.arXiv preprint arXiv:2503.10215(2025)

    Pith/arXiv arXiv 2025

  15. [15]

    Rae, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre...

    2022

  16. [16]

    Miriam Horovicz and Roni Goldshmidt. 2024. TokenSHAP: Interpreting Large Language Models with Monte Carlo Shapley Value Estimation. InProceedings of the 1st Workshop on NLP for Science (NLP4Science), Lotem Peled-Cohen, Nitay Calderon, Shir Lissak, and Roi Reichart (Eds.). Association for Computational Linguistics, Miami, FL, USA, 1–8. https://doi.org/10.1...

    doi:10.18653/v1/2024.nlp4science-1.1 2024

  17. [17]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv 2021.arXiv preprint arXiv:2106.0968510 (2021)

    Pith/arXiv arXiv 2021

  18. [18]

    Turner, and Douglas Eck

    Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E. Turner, and Douglas Eck. 2017. Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control. InProceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 70), Doina Precup and Yee Whye ...

    2017

  19. [19]

    Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas Spanos, and Dawn Song. 2019. Efficient task- specific data valuation for nearest neighbor algorithms.Proc. VLDB Endow.12, 11 (July 2019), 1610–1623. https://doi.org/10.14778/3342263.3342637

    arXiv 2019

  20. [20]

    Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J. Spanos

  21. [21]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models.CoRRabs/2001.08361 (2020). https://arxiv.org/pdf/2001.08361.pdf

    Pith/arXiv arXiv 2020

  22. [22]

    Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. InProceedings of the 34th International Conference on Machine Learning - Volume 70(Sydney, NSW, Australia)(ICML’17). JMLR.org, 1885–1894

    2017

  23. [23]

    Lundberg and Su-In Lee

    Scott M. Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. InAdvances in Neural Information Processing Systems, Vol. 30. https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf

    2017

  24. [24]

    Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions.Advances in neural information processing systems30 (2017)

    2017

  25. [25]

    Sasan Maleki, Long Tran-Thanh, Greg Hines, Talal Rahwan, and Alex Rogers. 2013. Bounding the Estimation Error of Sampling-based Shapley Value Approximation With/Without Stratifying.CoRRabs/1306.4265 (2013). arXiv:1306.4265 http://arxiv.org/abs/1306.4265

    Pith/arXiv arXiv 2013

  26. [26]

    Roberto-Rafael Maura-Rivero, Marc Lanctot, Francesco Visin, and Kate Larson. 2025. Jackpot! alignment as a maximal lottery.arXiv preprint arXiv:2501.19266(2025)

    Pith/arXiv arXiv 2025

  27. [27]

    Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher Manning. 2024. An Emulator for Fine-tuning Large Language Models using Small Language Models. InInternational Conference on Representation Learning, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024. 13229–13244. https://proceedings.iclr.cc/paper...

    2024

  28. [28]

    Hyeonseok Moon, Jaehyung Seo, Seonmin Koo, Jinsung Kim, Young-kyoung Ham, Jiwon Moon, and Heuiseok Lim. 2025. LimaCost: Data Valuation for Instruction Tuning of Large Language Models. InFindings of the Association for Computational Linguistics: EMNLP 2025, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association f...

    work page doi:10.18653/v1/2025.findings-emnlp.688 2025

  29. [29]

    Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, et al. 2024. Nash learning from human feedback. InForty-first International Conference on Machine Learning

    2024

  30. [30]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human f...

    2022

  31. [31]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. InAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 53728...

    2023

  32. [32]

    Stephanie Schoch, Ritwick Mishra, and Yangfeng Ji. 2023. Data Selection for Fine-tuning Large Language Models Using Transferred Shapley Values. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), Vishakh Padmakumar, Gisela Vallejo, and Yao Fu (Eds.). Association for Computational...

    work page doi:10.18653/v1/2023.acl-srw.37 2023

  33. [33]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs.LG] https://arxiv.org/abs/1707.06347

    Pith/arXiv arXiv 2017

  34. [34]

    Lloyd S Shapley. 1953. A Value for n-Person Games. InContributions to the Theory of Games II, Harold W. Kuhn and Albert W. Tucker (Eds.). Princeton University Press, Princeton, 307–317

    1953

  35. [35]

    Rachael Hwee Ling Sim, Yehong Zhang, Mun Choon Chan, and Bryan Kian Hsiang Low. 2020. Collaborative machine learning with incentive-aware model rewards. InProceedings of the 37th International Conference on Machine Learning (ICML’20). JMLR.org, Article 828, 10 pages

    2020

  36. [36]

    Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell. 2024. Distributional preference learning: Understanding and accounting for hidden context in rlhf.ICLR(2024)

    2024

  37. [37]

    Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. 2024. A minimaximalist approach to reinforcement learning from human feedback. InProceedings of the 41st International Conference on Machine Learning(Vienna, Austria)(ICML’24). JMLR.org, Article 1929, 33 pages. 10

    2024

  38. [38]

    Mélissa Tamine, Benjamin Heymann, Patrick Loiseau, and Maxime Vono. 2025. On the Impact of the Utility in Semivalue-based Data Valuation. arXiv:2502.06574 [cs.AI] https://arxiv.org/abs/2502.06574

    arXiv 2025

  39. [39]

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. 2020. TRL: Transformer Reinforcement Learning. https://github.com/huggingface/trl

    2020

  40. [40]

    Jingtan Wang, Xiaoqiang Lin, Rui Qiao, Chuan-Sheng Foo, and Bryan Kian Hsiang Low. 2024. Helpful or harmful data? fine-tuning-free shapley attribution for explaining language model predictions. InProceedings of the 41st International Conference on Machine Learning(Vienna, Austria) (ICML’24). JMLR.org, Article 2089, 32 pages

    2024

  41. [41]

    Jiachen (Tianhao) Wang, Prateek Mittal, Dawn Song, and Ruoxi Jia. 2025. Data Shapley in One Training Run. InInternational Conference on Representation Learning, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025. 12358–12395. https://proceedings.iclr.cc/paper_files/paper/ 2025/file/20fdaf67581e6d7157376d1ed584040a-Paper-Conference.pdf

    2025

  42. [42]

    Wang, and Jian Du

    Yingtai Xiao, Yuqing Zhu, Sirat Samyoun, Wanrong Zhang, Jiachen T. Wang, and Jian Du. 2025. TokenShapley: Token Level Context Attribution with Shapley Value. InFindings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, V...

    doi:10.18653/v1/2025.findings- 2025

  43. [43]

    Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. 2024. Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment.International Conference on Machine Learning(2024)

    2024

  44. [44]

    Zikun Ye and Hema Yoganarasimhan. 2025. Fair Document Valuation in LLM Summaries via Shapley Values. arXiv:2505.23842 [cs.CL] https: //arxiv.org/abs/2505.23842

    Pith/arXiv arXiv 2025

  45. [45]

    Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B

    Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-Tuning Language Models from Human Preferences.arXiv preprint arXiv:1909.08593(2019). https://arxiv.org/abs/1909.08593 11

    Pith/arXiv arXiv 2019

  46. [2019]

    InProceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research, Vol

    Towards Efficient Data Valuation Based on the Shapley Value. InProceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research, Vol. 89), Kamalika Chaudhuri and Masashi Sugiyama (Eds.). PMLR, 1167–1176. https://proceedings.mlr.press/v89/jia19a.html