Pith. sign in

REVIEW 3 major objections 5 minor 104 references

Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Swapping χPO's log loss for a bounded square loss yields offline preference alignment guarantees that stay optimal under label privacy and label corruption for general function classes.

desk verdict Solid theory paper: square-loss χPO gets clean DP/robustness guarantees for general function classes, but the uniform Vmax assumption is the real cost and should be flagged prominently. read the letter →

arxiv 2505.21395 v1 pith:H3IFIZDI submitted 2025-05-27 cs.LG

Xingyu Zhou , Yulian Wu , Wenqian Weng , Francesco Orabona This is my paper
classification cs.LG
keywords offlinepreferencealignmentdifferentialprivacyrandomizedresponseHubercorruptionsquarelossoverprobabilitiessingle-policyconcentrabilityleast-squaresregressionboundsgeneralmodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a single, simple modification to χPO — replacing the standard log loss with a square loss over the predicted win probability, the Brier score — is enough to make offline preference alignment simultaneously private and robust to corrupted labels, without sacrificing the optimal $1/\sqrt{n}$ sample complexity that χPO obtains in the clean setting. If true, this would be the first time a direct alignment method (no separate reward model, no RL loop) attains optimal rates under local differential privacy of labels with general function approximation, and the first meaningful theoretical guarantee for Huber-corrupted labels in that setting. The paper also claims the same loss yields the first pure central-DP result for general function classes, and the first private-and-robust results for general non-transitive preference models, all through one unified analysis. The proof engine is a new generalization bound for least-squares regression with corrupted and privatized binary responses.

What carries the argument

The carrying mechanism is a square loss over probabilities: for each sample the algorithm minimizes $\sum_i \bigl(2\sigma(\mathrm{clip}_{2R_{\max}}[\beta h_{\chi\mathrm{PO},i}]) - 1 - c(\varepsilon)\bar z_i\bigr)^2$ with $h_{\chi\mathrm{PO},i} = \phi(\pi(a^1_i\mid x_i)/\pi_{\mathrm{ref}}(a^1_i\mid x_i)) - \phi(\pi(a^0_i\mid x_i)/\pi_{\mathrm{ref}}(a^0_i\mid x_i))$, $\phi(u) = u + \log u$, and $c(\varepsilon) = (e^\varepsilon+1)/(e^\varepsilon-1)$ the randomized-response debiasing factor. This loss is the Brier score: bounded in $[0,1]$, so a corrupted label can move the objective by at most a constant, unlike log loss. The sigmoid-plus-clipping maps implicit reward differences into a fixed range, which keeps sensitivity bounded for the exponential mechanism and lets the mean-value theorem translate least-squares regression error into suboptimality gap. The analysis then runs through a meta theorem that converts the least-squares generalization error of the implicit reward into a suboptimality gap scaled by single-policy concentrability.

What would settle it

On a synthetic preference dataset, choose a reference policy with near-zero probability on an action whose true preference probability is extreme, so the optimal implicit reward difference exceeds the clip range $2R_{\max}$; if the empirical suboptimality gap of SquareχPO then stops decaying at the $1/\sqrt{n}$ rate as $n$ grows, the clipping-preserves-realizability step fails.

Watch

Extended reading notes

Core claim

The central claim is Theorem 3.5: for any comparator policy $\pi^\star$, under policy realizability and a bounded implicit reward difference, the policy output by SquareχPO (Algorithm 1) has suboptimality gap $\mathrm{SG}_{\mathrm{CTL}}(\hat\pi; \pi^\star) \lesssim \kappa(\pi^\star)\bigl(c(\varepsilon)\sqrt{\log(|\Pi|/\zeta)/n} + \sqrt{\alpha}\bigr)$ under corruption-then-LDP, and $\mathrm{SG}_{\mathrm{LTC}}(\hat\pi; \pi^\star) \lesssim \kappa(\pi^\star)\bigl(c(\varepsilon)\sqrt{\log(|\Pi|/\zeta)/n} + \sqrt{\alpha c(\varepsilon)}\bigr)$ when LDP happens first, where $\kappa(\pi^\star) = e^{2R_{\max}}(V_{\max}/R_{\max})\sqrt{C_{\pi^\star}}$ is a single-policy concentrability factor. The same machinery gives a pure-DP central-model bound with privacy cost $(1 + 1/\sqrt{\varepsilon})$ instead of $c(\varepsilon)$, and extends to general preference models with duality-gap bounds. All bounds are driven by the same least-squares regression analysis, so the paper's contribution is identifying the square loss as the right loss to make privacy and corruption costs additive under the benign order (CTL) and to expose the extra $\sqrt{c(\varepsilon)}$ factor when corruption follows privacy (LTC).

Load-bearing premise

The proof assumes a single constant $V_{\max}$ bounds the implicit reward difference $\beta(\phi(\pi(a\mid x)/\pi_{\mathrm{ref}}(a\mid x)) - \phi(\pi(b\mid x)/\pi_{\mathrm{ref}}(b\mid x)))$ for every policy in the class, so for large-norm neural policies the guarantee scales linearly in $V_{\max}$ and can become vacuous.

Editorial extensions

If this is right

  • SquareχPO achieves $c(\varepsilon)\sqrt{\log(|\Pi|/\zeta)/n}$ suboptimality under $\varepsilon$-LDP label privacy with general function classes, matching the optimal privacy cost and $1/\sqrt{n}$ rate for offline direct alignment.
  • Under $\alpha$-Huber corruption alone, the method keeps a $1/\sqrt{n}$ rate with an additive $\sqrt{\alpha}$ cost, improving on prior robust DPO-style analyses that only reach $n^{-1/4}$ in the linear case and a non-vanishing gap in general.
  • In the central model, the exponential mechanism with the square loss gives pure $\varepsilon$-DP protecting prompts, responses, and labels, with corruption cost additive in $\sqrt{\alpha}$.
  • The order of privacy and corruption matters: corruption then LDP is separable and additive, while LDP then corruption carries an extra $\sqrt{c(\varepsilon)}$ factor, showing that privatizing first makes subsequent corruption harder to absorb.
  • An iterative version of SquareχPO extends the guarantees to general non-transitive preference models, giving the first private-and-robust duality-gap bounds in that setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This suggests the bounded-loss-plus-debiasing recipe could be dropped into other DPO-style losses to inherit private-and-robust guarantees, since the analysis only seems to need a bounded loss, realizability, and the same implicit-reward reparameterization.
  • The CTL-vs-LTC separation is proven only as an upper bound; a direct test would be to run the two orders on a compact mean-estimation benchmark and see whether the $\sqrt{\alpha}$ versus $\sqrt{\alpha c(\varepsilon)}$ gap is real, as tight bounds in bandit settings suggest.
  • For large neural policies, making the guarantee non-vacuous likely requires either controlling output norms during training or replacing the fixed $2R_{\max}$ clip with a norm-dependent clip, an extension the paper does not explore.
  • The generalized least-squares bound is written for binary responses; the paper notes random rounding could extend it to continuous ratings, which would immediately give private-and-robust guarantees for scalar preference strengths.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SquareχPO, an offline direct alignment algorithm that replaces the log-loss of χPO with a squared Brier-type loss over sigmoid probabilities, and scales the private label by c(ε) = (e^ε+1)/(e^ε-1) under randomized response. The main results are suboptimality-gap bounds for the Bradley-Terry preference model under three data-generating settings: corruption-then-LDP (CTL), LDP-then-corruption (LTC), and central DP with corruption (cDP). Theorem 3.5 gives rates of order κ(π*)( c(ε)√(log|Π|/ζ / n) + √α ) for CTL and κ(π*)( c(ε)√(log|Π|/ζ / n) + √(α c(ε)) ) for LTC, with κ(π*) = e^{2Rmax} (Vmax/Rmax) √Cπ*. Theorem 3.7 gives the analogous central-DP bound with privacy cost (1+1/√ε)√(log|Π|/ζ / n). Section 4 extends the approach to a general preference model via an iterative self-play algorithm, with duality-gap bounds in Theorem 4.4. The technical core is a unified generalization bound for least-squares regression under privacy and corruption (Lemma B.1), which is then combined with a meta theorem adapted from Huang et al. (2024).

Significance. If the results hold as stated, they would represent a meaningful advance: SquareχPO appears to be the first offline direct alignment method achieving 1/√n rates with single-policy concentrability under local label privacy and Huber corruption for non-linear function classes, and the first pure-DP guarantee for general function classes in this setting. The modular proof structure is a genuine strength: Lemma B.1 cleanly separates the regression analysis from the alignment-specific reductions, and the c(ε) scaling is derived from the unbiasedness of randomized response rather than fitted to the target rates, so I see no circularity in the argument. The generalization bound may indeed be of independent interest for private and robust RL. However, the advertised scope of 'general function approximations' is substantially qualified by Assumption 3.3, which requires a uniform bound Vmax on the implicit reward difference over the entire policy class; as discussed in the major comments, this is a load-bearing restriction that should be confronted explicitly in the paper's claims.

major comments (3)
  1. Assumption 3.3 is not a benign bounded-reward condition: it requires a uniform constant Vmax such that |βϕ(π(a|x)/πref(a|x)) − βϕ(π(b|x)/πref(b|x))| ≤ Vmax for every π ∈ Π, x, a, b. For the softmax policy class over a single context with two actions and a uniform reference policy, Vmax(θ) diverges as θ → ∞, so any finite Vmax excludes policies that approach determinism, which are precisely the natural targets of alignment. Because κ(π*) is linear in Vmax, the suboptimality bound in Theorem 3.5 degrades linearly in Vmax and can be vacuous even for fixed n. The abstract and Section 1 advertise 'general function approximations' without flagging this restriction; the authors should either qualify the claim to policy classes with uniformly bounded likelihood ratios, exhibit a nontrivial function class (e.g., neural softmax with a bounded parameter norm) for which a finite Vmax provably holds, or add an explicit discussion of the regimes where the bound is meaningful. The same issue applies to Assumption 4.3 in the general-preference setting.
  2. The stated choice of β in Theorem C.1 appears to be a typo. The text reads β = sqrt(2/(Cπ* · Vmax errstat/Rmax)), but minimizing β Cπ* + 2β^{-1} Vmax^2 err2_stat^2 / Rmax^2 with respect to β yields β ∝ Vmax err2_stat / (Rmax sqrt(Cπ*)). As printed, the dependence on errstat is inverted and dimensionally inconsistent, so the displayed derivation of the final bound does not follow. Please correct the formula and verify the resulting constants.
  3. The claim that Algorithm 2 provides a 'first' pure-DP result for general function approximations should be reconciled with the fact that the exponential mechanism over an arbitrary policy class is not computationally efficient, a point the paper acknowledges only in Remark 3.8. More importantly, the utility analysis for the exponential mechanism in the proof of Lemma B.1 writes '+ log(|H|/ζ)/ε' after applying the exponential mechanism, but with the mechanism's exponent ε/8 the correct additive term is 8 log(|H|/ζ)/ε (up to constants). The final rate is unaffected because the extra constant is absorbed, but the proof step should be stated accurately.
minor comments (5)
  1. The repeated 'Evaluation. Evaluation.' at the start of the evaluation paragraph should be fixed.
  2. The informal note 'In fact, we are working on a new paper that will have a more thorough discussion. Stay tuned.' is not appropriate for a formal archival paper and should be removed or replaced with a neutral pointer.
  3. The definition is stated as '(ε, δ)-DP' but the surrounding text says 'We say A satisfies ε-DP'; the wording should be aligned to avoid confusion.
  4. In the CTL and LTC cases, the variance term after applying Freedman's inequality is written as (1/2)∑ E[(h(ui) − h*(ui))^2] with the c(ε)^2 factor apparently absent; this is correct only if η is chosen of order 1/c(ε)^2, and the write-up should state that choice explicitly.
  5. The repeated references to 'concurrent work' (Zhou et al., 2025) should be updated to the final publication status, and the various 'first' claims should be checked for accuracy in the current literature.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the SquareχPO bounds follow from a least-squares generalization lemma with explicit realizability assumptions; self-citations are comparative only.

full rationale

The paper's central claims (Theorems 3.5, 3.7, 4.4) are derived by reducing the square-loss objective to a least-squares regression problem (Lemma B.1) and then applying concentration (Freedman's inequality) plus a meta-theorem adapted from Huang et al. (2024). No parameter is fitted to the target suboptimality gap; the c(ε) rescaling is fixed by the randomized-response mechanism and appears in the bound through variance calculations, not by construction of the theorem statement. The realizability assumptions (3.2, 3.3, 4.1-4.3) are stated assumptions, not consequences of the claims, and the paper verifies realizability of the clipped sigmoid reparameterization explicitly in the proof of Lemma C.2. Citations to Zhou et al. (2025), Zhou & Zhang (2024), and Wu et al. (2024b) occur only in comparisons or as supporting intuition for the privacy-corruption interplay, and are not load-bearing for the rates proved. The fragility of Assumption 3.3 for large policy classes is a limitation of the theorem's applicability, not circularity. Footnote 3 in Section 6 explicitly states that an improved α^2 dependence is not yet established; this is a disclosed limitation rather than a hidden circular step.

Assumptions & free parameters 3 free parameters · 8 assumptions · 0 invented entities

The theoretical claims rest on standard realizability and boundedness assumptions inherited from χPO and offline RL, plus the new square-loss regression analysis. No invented entities and no data-fitted constants are introduced; the hyperparameters β, η, T are chosen in the analysis rather than fitted to data.

free parameters (3)
  • β (regularization coefficient) = chosen in Theorem C.1 as approximately sqrt(2/(C_π* V_max err_stat / R_max))
    A hyperparameter of the algorithm whose value is selected in the analysis to balance terms; it is not fitted to data but is required to instantiate the method.
  • η (step size) and T (number of iterations) in Algorithm 3 = η = 1/T and T chosen depending on n, m, ε, α
    Hyperparameters for the iterative general-preference algorithm; chosen in Theorem D.3 to optimize the bound.
  • Clipping threshold R = 2R_max = 2R_max
    Derived from the assumed reward bound R_max; not a free parameter but a modeling choice.
assumptions (8)
  • domain assumption Bradley-Terry preference model: P*(a1≻a0|x) = exp(r*(x,a1))/(exp(r*(x,a1))+exp(r*(x,a0))), with r* in [0,R_max]
    Defines the preference generation process; used throughout Section 3 and in the realizability verification in Lemma C.2.
  • domain assumption Policy realizability: π*_β ∈ Π (Assumption 3.2)
    Ensures the implicit reward reparameterization is realizable in the policy class; the proof maps the true reward to an element of the class.
  • domain assumption Bounded implicit reward difference: |βφ(π(a|x)/πref(a|x)) - βφ(π(b|x)/πref(b|x))| ≤ V_max for all π, x, a, b (Assumption 3.3)
    Controls the scale of the reparameterized reward; appears multiplicatively in the final suboptimality bound.
  • domain assumption Finite single-policy concentrability: C_π* = E_π*[π*(a|x)/πref(a|x)] < ∞ (Definition 3.4)
    The bound in Theorem 3.5 scales with sqrt(C_π*); if the reference policy does not cover the comparator, the guarantee is vacuous.
  • domain assumption Preference function realizability for the general model: ℓ* ∈ L (Assumption 4.1)
    Required for the iterative algorithm's preference-model estimation step in Section 4.
  • domain assumption Bounded implicit reward difference for general preferences (Assumption 4.3)
    Analogous to Assumption 3.3; needed for the self-play policy optimization analysis.
  • standard math Standard concentration inequalities (Freedman's inequality, Lemma B.3)
    Used in the proof of the key regression bound Lemma B.1.
  • standard math Exponential mechanism privacy and utility guarantees (McSherry & Talwar 2007)
    Used to derive the central-DP results in Theorem 3.7 and Lemma B.1(cDP).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment." pith.science (2026). https://pith.science/paper/H3IFIZDI

@misc{pith2026250521395,
  author       = {Pith},
  title        = {Pith review of: Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H3IFIZDI}},
  note         = {Machine review of arXiv:2505.21395}
}
abstract

In this paper, we theoretically study the offline alignment of language models with human preference feedback, under both preference label corruption and privacy protections. To this end, we propose Square$\chi$PO, a simple one-line change to $\chi$PO where the standard log-loss is replaced by a new square loss over probability. Thanks to the inherent properties of this new loss, we have advanced the state-of-the-art of differentially private and robust offline direct alignment. Specifically, for the local model of label privacy, Square$\chi$PO is the first algorithm that attains an optimal rate based on single-policy concentrability even with general function approximations. It also gives the first result under the central model of privacy protection over both prompts (responses) and labels. On the robustness side against Huber label corruption, Square$\chi$PO is the first alignment method that has a meaningful theoretical guarantee under general function approximations. More importantly, Square$\chi$PO can address privacy protection and corruption simultaneously, where an interesting separation is observed, implying that the order of privacy and corruption matters. Furthermore, we show that Square$\chi$PO can also be easily extended to handle the scenario of the general preference model with state-of-the-art guarantees under corruption and privacy. Last but not least, all of our theoretical guarantees enjoy a unified analysis, building upon a new result on the generalization error bounds of least-square regression under corruption and privacy constraints, which we believe is of independent interest to the community.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

104 extracted references · 35 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    I., Jacobs, S

    Abdin, M. I., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H. S., Benhaim, A., Bilenko, M., Bjorck, J., Bubeck, S., Cai, M., Mendes, C. C. T., Chen, W., Chaudhary, V., Chopra, P., Giorno, A. D., de Rosa, G., Dixon, M., Eldan, R., Iter, D., Garg, A., Goswami, A., Gunasekar, S., Haider, E., H...

    arXiv 2024

  3. [3]

    M., and Sun, W

    Agarwal, A., Jiang, N., Kakade, S. M., and Sun, W. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 32: 0 96, 2019

    2019

  4. [4]

    J., Jiang, N., Sekhari, A., and Xie, T

    Amortila, P., Foster, D. J., Jiang, N., Sekhari, A., and Xie, T. Harnessing density ratios for online reinforcement learning. arXiv preprint arXiv:2401.09681, 2024 a

    arXiv 2024

  5. [5]

    Scalable Online Exploration via Coverability

    Amortila, P., Foster, D. J., and Krishnamurthy, A. Scalable online exploration via coverability. arXiv preprint arXiv:2403.06571, 2024 b

    work page Pith review arXiv 2024

  6. [6]

    G., Guo, Z

    Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp.\ 4447--4455. PMLR, 2024

    2024

  7. [7]

    M., Schneider, J., and Ng, A

    Bagnell, J., Kakade, S. M., Schneider, J., and Ng, A. Policy search by dynamic programming. Advances in neural information processing systems, 16, 2003

    2003

  8. [8]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, ...

    arXiv 2022

Show all 104 references
  1. [9]

    E., Fort, S., Lanham, T., Telleen-Lawton, T., Conerly, T., Henighan, T., Hume, T., Bowman, S

    Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., ...

    2022 arXiv

  2. [10]

    Contextual bandit algorithms with supervised learning guarantees

    Beygelzimer, A., Langford, J., Li, L., Reyzin, L., and Schapire, R. Contextual bandit algorithms with supervised learning guarantees. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp.\ 19--26. JMLR Workshop and Conference ...

    2011

  3. [11]

    Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  4. [12]

    Brier, G. W. Verification of forecasts expressed in terms of probability. Monthly weather review, 78 0 (1): 0 1--3, 1950

    1950

  5. [13]

    Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., Wang, T., Marks, S., Segerie, C.-R., Carroll, M., Peng, A., Christoffersen, P., Damani, M., Slocum, S., Anwar, U., Siththaranjan, A., Nadeau, M., Michau...

    2023 arXiv

  6. [14]

    D., Zhan, W., Oertell, O., Brantley, K., Misra, D., Lee, J

    Chang, J. D., Zhan, W., Oertell, O., Brantley, K., Misra, D., Lee, J. D., and Sun, W. Dataset reset policy optimization for RLHF . arXiv preprint arXiv:2404.08495, 2024

    2024 arXiv

  7. [15]

    Robust and private stochastic linear bandits

    Charisopoulos, V., Esfandiari, H., and Mirrokni, V. Robust and private stochastic linear bandits. In International Conference on Machine Learning, pp.\ 4096--4115. PMLR, 2023

    2023

  8. [16]

    and Hsu, D

    Chaudhuri, K. and Hsu, D. Sample complexity bounds for differentially private learning. In Proceedings of the 24th Annual Conference on Learning Theory, pp.\ 155--186. JMLR Workshop and Conference Proceedings, 2011

    2011

  9. [17]

    and Jiang, N

    Chen, J. and Jiang, N. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning, pp.\ 1042--1051. PMLR, 2019

    2019

  10. [18]

    and Sentenac, F

    Chhor, J. and Sentenac, F. Robust estimation of discrete distributions under local differential privacy. In International Conference on Algorithmic Learning Theory, pp.\ 411--446. PMLR, 2023

    2023

  11. [19]

    Chowdhury, S. R. and Zhou, X. Differentially private regret minimization in episodic markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 6375--6383, 2022 a

    2022

  12. [20]

    Chowdhury, S. R. and Zhou, X. Distributed differential privacy in multi-armed bandits. arXiv preprint arXiv:2206.05772, 2022 b

    2022 arXiv

  13. [21]

    Chowdhury, S. R. and Zhou, X. Shuffle private linear contextual bandits. arXiv preprint arXiv:2202.05567, 2022 c

    2022 arXiv

  14. [22]

    R., Kini, A., and Natarajan, N

    Chowdhury, S. R., Kini, A., and Natarajan, N. Provably robust DPO : Aligning language models with noisy feedback. arXiv preprint arXiv:2403.00409, 2024 a

    2024 arXiv

  15. [23]

    R., Zhou, X., and Natarajan, N

    Chowdhury, S. R., Zhou, X., and Natarajan, N. Differentially private reward estimation with preference feedback. In International Conference on Artificial Intelligence and Statistics, pp.\ 4843--4851. PMLR, 2024 b

    2024

  16. [24]

    F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D

    Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  17. [25]

    and Du, S

    Cui, Q. and Du, S. S. When are offline two-player zero-sum markov games solvable? Advances in Neural Information Processing Systems, 35: 0 25779--25791, 2022

    2022

  18. [26]

    Minimax-optimal off-policy evaluation with linear function approximation

    Duan, Y., Jia, Z., and Wang, M. Minimax-optimal off-policy evaluation with linear function approximation. In International Conference on Machine Learning, pp.\ 2701--2709. PMLR, 2020

    2020

  19. [27]

    Calibrating noise to sensitivity in private data analysis

    Dwork, C., McSherry, F., Nissim, K., and Smith, A. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pp.\ 265--284. Springer, 2006

    2006

  20. [28]

    R., Yun, H., Teo, C

    Feng, Q., Kasa, S. R., Yun, H., Teo, C. H., and Bodapati, S. B. Exposing privacy gaps: Membership inference attack on preference data for LLM alignment. arXiv preprint arXiv:2407.06443, 2024

    2024 arXiv

  21. [29]

    Importance-weighted offline learning done right

    Gabbianelli, G., Neu, G., and Papini, M. Importance-weighted offline learning done right. In International Conference on Algorithmic Learning Theory, pp.\ 614--634. PMLR, 2024

    2024

  22. [30]

    D., Zhan, W., Oertell, O., Swamy, G., Brantley, K., Joachims, T., Bagnell, J

    Gao, Z., Chang, J. D., Zhan, W., Oertell, O., Swamy, G., Brantley, K., Joachims, T., Bagnell, J. A., Lee, J. D., and Sun, W. Rebel: Reinforcement learning via regressing relative rewards. arXiv preprint arXiv:2404.16767, 2024

    2024 arXiv

  23. [31]

    Local differential privacy for regret minimization in reinforcement learning

    Garcelon, E., Perchet, V., Pike-Burke, C., and Pirotta, M. Local differential privacy for regret minimization in reinforcement learning. Advances in Neural Information Processing Systems, 34: 0 10561--10573, 2021

    2021

  24. [32]

    and Hopkins, S

    Georgiev, K. and Hopkins, S. Privacy induces robustness: Information-computation gaps and sparse mean estimation. Advances in neural information processing systems, 35: 0 6829--6842, 2022

    2022

  25. [33]

    B., Kamath, G., Majid, M., and Narayanan, S

    Hopkins, S. B., Kamath, G., Majid, M., and Narayanan, S. Robustness implies privacy in statistical estimation. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing, pp.\ 497--506, 2023

    2023

  26. [34]

    D., Sun, W., Krishnamurthy, A., and Foster, D

    Huang, A., Zhan, W., Xie, T., Lee, J. D., Sun, W., Krishnamurthy, A., and Foster, D. J. Correcting the mythos of KL -regularization: Direct alignment without overparameterization via Chi -squared preference optimization. arXiv preprint arXiv:2407.13399, 2024

    2024 arXiv

  27. [35]

    Huber, P. J. Robust estimation of a location parameter. The Annals of Mathematical Statistics, 35 0 (1): 0 73--101, 1964

    1964

  28. [36]

    Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms

    Jin, C., Liu, Q., and Miryoosefi, S. Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems, 34: 0 13406--13418, 2021 a

    2021

  29. [37]

    Is pessimism provably efficient for offline rl? In International Conference on Machine Learning, pp.\ 5084--5096

    Jin, Y., Yang, Z., and Wang, Z. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning, pp.\ 5084--5096. PMLR, 2021 b

    2021

  30. [38]

    and Langford, J

    Kakade, S. and Langford, J. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pp.\ 267--274, 2002

    2002

  31. [39]

    The broader landscape of robustness in algorithmic statistics, 2024

    Kamath, G. The broader landscape of robustness in algorithmic statistics, 2024. URL https://arxiv.org/abs/2412.02670

    2024 arXiv

  32. [40]

    P., Lee, H

    Kasiviswanathan, S. P., Lee, H. K., Nissim, K., Raskhodnikova, S., and Smith, A. What can we learn privately? SIAM Journal on Computing, 40 0 (3): 0 793--826, 2011

    2011

  33. [41]

    and Brown-Cohen, J

    Korkmaz, E. and Brown-Cohen, J. Learning differentially private rewards from human feedback. https://openreview.net/pdf?id=reBq1gmlhS, 2024

    2024

  34. [42]

    Optidice: Offline policy optimization via stationary distribution correction estimation

    Lee, J., Jeon, W., Lee, B., Pineau, J., and Kim, K.-E. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pp.\ 6120--6130. PMLR, 2021

    2021

  35. [43]

    Differentially private linear bandits with partial distributed feedback

    Li, F., Zhou, X., and Ji, B. Differentially private linear bandits with partial distributed feedback. In 2022 20th International Symposium on Modeling and Optimization in Mobile, Ad hoc, and Wireless Networks (WiOpt), pp.\ 41--48. IEEE, 2022

    2022

  36. [44]

    B., and Yu, Y

    Li, M., Berrett, T. B., and Yu, Y. On robustness and local differential privacy. The Annals of Statistics, 51 0 (2): 0 717--737, 2023

    2023

  37. [45]

    J., and Liu, J

    Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M., Liu, P. J., and Liu, J. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657, 2023

    2023 arXiv

  38. [46]

    Provably mitigating overoptimization in RLHF : Your SFT loss is implicitly an adversarial regularizer

    Liu, Z., Lu, M., Zhang, S., Liu, B., Guo, H., Yang, Y., Blanchet, J., and Wang, Z. Provably mitigating overoptimization in RLHF : Your SFT loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405.16436, 2024

    2024 arXiv

  39. [47]

    Y., Yan, J., Jayaraman, D., and Bastani, O

    Ma, J. Y., Yan, J., Jayaraman, D., and Bastani, O. Offline goal-conditioned reinforcement learning via f -advantage regression. Advances in neural information processing systems, 35: 0 310--323, 2022 a

    2022

  40. [48]

    J., Shen, A., Jayaraman, D., and Bastani, O

    Ma, Y. J., Shen, A., Jayaraman, D., and Bastani, O. Smodice: Versatile offline imitation learning via state occupancy matching. arXiv preprint arXiv:2202.02433, 1 0 (2): 0 3, 2022 b

    2022 arXiv

  41. [49]

    Corruption robust offline reinforcement learning with human feedback

    Mandal, D., Nika, A., Kamalaruban, P., Singla, A., and Radanovi \'c , G. Corruption robust offline reinforcement learning with human feedback. arXiv preprint arXiv:2402.06734, 2024

    2024 arXiv

  42. [50]

    and Talwar, K

    McSherry, F. and Talwar, K. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07), pp.\ 94--103. IEEE, 2007

    2007

  43. [51]

    and Thakurta, A

    Mishra, N. and Thakurta, A. (nearly) optimal differentially private stochastic multi-arm bandits. In Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence, pp.\ 592--601, 2015

    2015

  44. [52]

    and Szepesv \'a ri, C

    Munos, R. and Szepesv \'a ri, C. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9 0 (5), 2008

    2008

  45. [53]

    G., Rowland, M., Guo, Z

    Munos, R., Valko, M., Calandriello, D., Azar, M. G., Rowland, M., Guo, Z. D., Tang, Y., Geist, M., Mesnard, T., Michi, A., et al. Nash learning from human feedback. arXiv preprint arXiv:2312.00886, 2023

    2023 arXiv

  46. [54]

    Nemirovskij, A. S. and Yudin, D. Problem complexity and method efficiency in optimization. Wiley, New York, NY, USA, 1983

    1983

  47. [55]

    ChatGPT : Optimizing language models for dialogue

    OpenAI , T. ChatGPT : Optimizing language models for dialogue. OpenAI , 2022

    2022

  48. [56]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to foll...

    2022

  49. [57]

    and Wang, Y.-X

    Qiao, D. and Wang, Y.-X. Near-optimal differentially private reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp.\ 9914--9940. PMLR, 2023

    2023

  50. [58]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2023

    2023

  51. [59]

    Bridging offline reinforcement learning and imitation learning: A tale of pessimism

    Rashidinejad, P., Zhu, B., Ma, C., Jiao, J., and Russell, S. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 34: 0 11702--11716, 2021

    2021

  52. [60]

    Ren, W., Zhou, X., Liu, J., and Shroff, N. B. Multi-armed bandits with local differential privacy. arXiv preprint arXiv:2007.03121, 2020

    2007 arXiv

  53. [61]

    and Bagnell, J

    Ross, S. and Bagnell, J. A. Agnostic system identification for model-based reinforcement learning. arXiv preprint arXiv:1203.1007, 2012

    2012 arXiv

  54. [62]

    Direct Nash optimization: Teaching language models to self-improve with general preferences

    Rosset, C., Cheng, C.-A., Mitra, A., Santacroce, M., Awadallah, A., and Xie, T. Direct Nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715, 2024

    2024 arXiv

  55. [63]

    and Sheffet, O

    Sajed, T. and Sheffet, O. An optimal private stochastic-mab algorithm based on optimal private stopping rule. In International Conference on Machine Learning, pp.\ 5579--5588. PMLR, 2019

    2019

  56. [64]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  57. [65]

    and Sheffet, O

    Shariff, R. and Sheffet, O. Differentially private contextual linear bandits. Advances in Neural Information Processing Systems, 31, 2018

    2018

  58. [66]

    D., and Brown, D

    Shin, D., Dragan, A. D., and Brown, D. S. Benchmarks and algorithms for offline preference-based reward learning. arXiv preprint arXiv:2301.01392, 2023

    2023 arXiv

  59. [67]

    A., Krishnamurthy, A., and Sun, W

    Song, Y., Zhou, Y., Sekhari, A., Bagnell, J. A., Krishnamurthy, A., and Sun, W. Hybrid RL : Using both offline and online data can make RL efficient. arXiv preprint arXiv:2210.06718, 2022

    2022 arXiv

  60. [68]

    The importance of online data: Understanding preference fine-tuning via coverage

    Song, Y., Swamy, G., Singh, A., Bagnell, D., and Sun, W. The importance of online data: Understanding preference fine-tuning via coverage. In ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2024

    2024

  61. [69]

    Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33: 0 3008--3021, 2020

    2020

  62. [70]

    Sun, L., Huang, Y., Wang, H., Wu, S., Zhang, Q., Gao, C., Huang, Y., Lyu, W., Zhang, Y., Li, X., Liu, Z., Liu, Y., Wang, Y., Zhang, Z., Vidgen, B., Kailkhura, B., Xiong, C., Xiao, C., Li, C., Xing, E., Huang, F., Liu, H., Ji, H., Wang, H., Zhang, H., Yao, H., Kellis, M., Zitni...

    2024 arXiv

  63. [71]

    Principle-driven self-alignment of language models from scratch with minimal human supervision

    Sun, Z., Shen, Y., Zhou, Q., Zhang, H., Chen, Z., Cox, D., Yang, Y., and Gan, C. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2024 b

    2024

  64. [72]

    S., and Agarwal, A

    Swamy, G., Dann, C., Kidambi, R., Wu, Z. S., and Agarwal, A. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.04056, 2024

    2024 arXiv

  65. [73]

    D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P

    Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Pires, B. \'A ., and Piot, B. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749, 2024

    2024 arXiv

  66. [74]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  67. [75]

    and Sun, W

    Uehara, M. and Sun, W. Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06226, 2021

    2021 arXiv

  68. [76]

    Private reinforcement learning with pac and regret guarantees

    Vietri, G., Balle, B., Krishnamurthy, A., and Wu, S. Private reinforcement learning with pac and regret guarantees. In International Conference on Machine Learning, pp.\ 9754--9764. PMLR, 2020

    2020

  69. [77]

    TRL : T ransformer R einforcement L earning

    von Werra, L., Belkada, Y., Tunstall, L., Beeching, E., Thrush, T., Lambert, N., Huang, S., Rasul, K., and Gallouédec, Q. TRL : T ransformer R einforcement L earning. https://github.com/huggingface/trl, 2020

    2020

  70. [78]

    Beyond reverse KL : Generalizing direct preference optimization with diverse divergence constraints

    Wang, C., Jiang, Y., Yang, C., Liu, H., and Chen, Y. Beyond reverse KL : Generalizing direct preference optimization with diverse divergence constraints. arXiv preprint arXiv:2309.16240, 2023 a

    2023 arXiv

  71. [79]

    The central role of the loss function in reinforcement learning

    Wang, K., Kallus, N., and Sun, W. The central role of the loss function in reinforcement learning. arXiv preprint arXiv:2409.12799, 2024 a

    2024 arXiv

  72. [80]

    Oracle-efficient pessimism: Offline policy optimization in contextual bandits

    Wang, L., Krishnamurthy, A., and Slivkins, A. Oracle-efficient pessimism: Offline policy optimization in contextual bandits. In International Conference on Artificial Intelligence and Statistics, pp.\ 766--774. PMLR, 2024 b

    2024

  73. [81]

    Is RLHF more difficult than standard RL ? arXiv preprint arXiv:2306.14111, 2023 b

    Wang, Y., Liu, Q., and Jin, C. Is RLHF more difficult than standard RL ? arXiv preprint arXiv:2306.14111, 2023 b

    2023 arXiv

  74. [82]

    Warner, S. L. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American statistical association, 60 0 (309): 0 63--69, 1965

    1965

  75. [83]

    On private and robust bandits

    Wu, Y., Zhou, X., Tao, Y., and Wang, D. On private and robust bandits. Advances in Neural Information Processing Systems, 36: 0 34778--34790, 2023

    2023

  76. [84]

    Self-play preference optimization for language model alignment

    Wu, Y., Sun, Z., Yuan, H., Ji, K., Yang, Y., and Gu, Q. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675, 2024 a

    2024 arXiv

  77. [85]

    On private and robust bandits

    Wu, Y., Zhou, X., Tao, Y., and Wang, D. On private and robust bandits. Advances in Neural Information Processing Systems, 36, 2024 b

    2024

  78. [86]

    Xiao, J., Li, Z., Xie, X., Getzen, E., Fang, C., Long, Q., and Su, W. J. On the algorithmic bias of aligning large language models with RLHF : Preference collapse and matching regularization. arXiv preprint arXiv:2405.16455, 2024

    2024 arXiv

  79. [87]

    and Zhu, J

    Xiao, T. and Zhu, J. Foundations of large language models. arXiv preprint arXiv:2501.09223, 2025

    2025 arXiv

  80. [88]

    Bellman-consistent pessimism for offline reinforcement learning

    Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34: 0 6683--6694, 2021 a

    2021

  81. [89]

    Policy finetuning: Bridging sample-efficient offline and online reinforcement learning

    Xie, T., Jiang, N., Wang, H., Xiong, C., and Bai, Y. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34: 0 27395--27407, 2021 b

    2021

  82. [90]

    J., Bai, Y., Jiang, N., and Kakade, S

    Xie, T., Foster, D. J., Bai, Y., Jiang, N., and Kakade, S. M. The role of coverage in online reinforcement learning. arXiv preprint arXiv:2210.04157, 2022

    2022 arXiv

  83. [91]

    J., Krishnamurthy, A., Rosset, C., Awadallah, A., and Rakhlin, A

    Xie, T., Foster, D. J., Krishnamurthy, A., Rosset, C., Awadallah, A., and Rakhlin, A. Exploratory preference optimization: Harnessing implicit Q* -approximation for sample-efficient RLHF . arXiv preprint arXiv:2405.21046, 2024

    2024 arXiv

  84. [92]

    A., Kamath, G., Kulkarni, J., Lee, Y

    Yu, D., Naik, S., Backurs, A., Gopi, S., Inan, H. A., Kamath, G., Kulkarni, J., Lee, Y. T., Manoel, A., Wutschitz, L., Yekhanin, S., and Zhang, H. Differentially private fine-tuning of language models. arXiv preprint arXiv:2110.06500, 2021

    arXiv

  85. [93]

    Offline reinforcement learning with realizability and single-policy concentrability

    Zhan, W., Huang, B., Huang, A., Jiang, N., and Lee, J. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory, pp.\ 2730--2775. PMLR, 2022

    2022

  86. [94]

    D., and Sun, W

    Zhan, W., Uehara, M., Kallus, N., Lee, J. D., and Sun, W. Provable offline reinforcement learning with human feedback. In ICML 2023 Workshop The Many Facets of Preference-Based Learning, 2023

    2023

  87. [95]

    Corruption-robust offline reinforcement learning

    Zhang, X., Chen, Y., Zhu, X., and Sun, W. Corruption-robust offline reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp.\ 5757--5773. PMLR, 2022

    2022

  88. [96]

    Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P. J. SLiC-HF : Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023

    2023 arXiv

  89. [97]

    Locally differentially private (contextual) bandits learning

    Zheng, K., Cai, T., Huang, W., Li, Z., and Wang, L. Locally differentially private (contextual) bandits learning. Advances in Neural Information Processing Systems, 33: 0 12300--12310, 2020

    2020

  90. [98]

    Differentially private reinforcement learning with linear function approximation

    Zhou, X. Differentially private reinforcement learning with linear function approximation. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 6 0 (1): 0 1--27, 2022

    2022

  91. [99]

    and Tan, J

    Zhou, X. and Tan, J. Local differential privacy for bayesian optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 11152--11159, 2021

    2021

  92. [100]

    and Zhang, W

    Zhou, X. and Zhang, W. Locally private and robust multi-armed bandits. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  93. [101]

    A unified theoretical analysis of private and robust offline alignment: from rlhf to dpo

    Zhou, X., Wu, Y., and Orabona, F. A unified theoretical analysis of private and robust offline alignment: from rlhf to dpo. arXiv preprint arXiv:2505.15694, 2025

    2025 arXiv

  94. [102]

    Principled reinforcement learning with human feedback from pairwise or K -wise comparisons

    Zhu, B., Jordan, M., and Jiao, J. Principled reinforcement learning with human feedback from pairwise or K -wise comparisons. In International Conference on Machine Learning, pp.\ 43037--43067. PMLR, 2023

    2023

  95. [103]

    and Zhang, A

    Zhu, H. and Zhang, A. Provably efficient offline goal-conditioned reinforcement learning with general function approximation and single-policy concentrability. Advances in Neural Information Processing Systems, 36, 2024

    2024

  96. [104]

    M., Stiennon, N., Wu, J., Brown, T

    Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.