{"id":"89f70130-070e-4f7d-a3d2-8873838efccd","arxiv_id":"2505.15694","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Under linear-model assumptions, offline RLHF and DPO both reduce to logistic regression, and privatizing labels before corruption (LTC) carries an extra c(ε) factor in the error bounds compared to corrupting before privatizing (CTL).","lead":"This paper analyzes how two types of label noise, local privacy and adversarial corruption, affect offline RLHF and DPO, and shows the order of the two noise processes changes the theoretical guarantees. It may matter because it provides a unified framework and the first O(1/√n) rates for robust or private DPO under linear-model assumptions.","discovery_kind":"unification","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unstated action-sampling assumption is load-bearing: without a^0_i,a^1_i drawn i.i.d. from πsft, the identification of bΣ with Σdiff_{πsft,πsft} in Corollary 4.4 and Proposition 4.6 fails, so the κ-based suboptimality bounds in Theorems 6.1 and 6.3 do not follow as stated.","rationale":"The paper's main contributions are the reduction framework and the CTL/LTC separation. Theorem 5.4's estimation bounds appear internally consistent: the gradient decompositions in Appendix E.4 are correct, and the claimed sign-direction error in Mandal et al. (Appendix G) is plausible. The load-bearing fragility is at the interface between Theorem 5.4's estimation error in the empirical norm and the suboptimality reductions in Corollary 4.4 and Proposition 4.6, where bΣ is identified with Σdiff_{πsft,πsft}. This identification is an unstated data-generation assumption: nothing in Assumptions 4.1/4.5 or the Section 3 setup guarantees that a^0_i and a^1_i are drawn from πsft. If they are drawn from a different behavior policy, the covariance can be arbitrarily misaligned with the one used in the relative condition number, and the claimed rates do not follow. The fix is straightforward—state the sampling assumption explicitly—so the paper remains conditionally acceptable, consistent with the reader's verdict. I do not elevate to REJECT because the estimation theory and the qualitative separation are independent of this gap and appear sound. I would not ACCEPT as-is because a central proof step relies on a non-obvious, unstated condition. The reader identified precisely this assumption; my analysis concurs with that identification and with the CONDITIONAL verdict.","tokens_in":26897,"tokens_out":8860,"duration_ms":71465,"concrete_test":"Construct a behavior-policy counterexample: choose πb ≠ πsft such that Σdiff_{πb,πb} is not proportional to Σdiff_{πsft,πsft} (e.g., πb concentrates on a low-variance subset of actions). Generate n i.i.d. preference samples from πb and check (i) whether the empirical bΣ satisfies ‖bΣ − Σdiff_{πb,πb}‖ = O(√(d/n)) while ‖bΣ − Σdiff_{πsft,πsft}‖ stays bounded away from zero, and (ii) whether the suboptimality bound from Corollary 4.4/Proposition 4.6 with κ defined via Σdiff_{πsft,πsft} is violated by the observed suboptimality. The analytical version: re-run the proof with Lemma H.1 replaced by concentration around Σdiff_{πb,πb} and verify that step (a) in Corollary 4.4 cannot be salvaged unless κ is redefined with Σdiff_{πb,πb}.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reduction to logistic regression (Propositions 4.2 and 4.6) holds for any action distribution, but the conversion of the estimation error from the empirical norm ∥·∥_{bΣ+λI} into a suboptimality bound in terms of the relative condition number κ requires bΣ to be concentrated around Σdiff_{πsft,πsft}. This is asserted in the proof of Proposition 4.6 ('Σdiff_{πsft,πsft} is the corresponding population matrix of bΣ') and used in step (a) of Corollary 4.4 via Lemma H.1. The dataset description in Section 3 and Assumptions 4.1 and 4.5 do not state that the two actions in each preference sample are sampled i.i.d. from the SFT/reference policy πsft. If the behavior policy is some πb ≠ πsft, then bΣ concentrates around Σdiff_{πb,πb} instead, so the covariance replacement in Lemma H.1 is invalid and the displayed bounds with κ(π†,πref) or κΠ do not follow; the suboptimality may depend on a different and potentially much larger concentrability coefficient. The CTL/LTC separation in Theorem 5.4 is independent of this issue and appears sound, but the paper's headline suboptimality guarantees for both RLHF and DPO are not yet proven under the assumptions as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops a unified reduction of offline RLHF and DPO, under a linear reward model (Assumption 4.1) and a log-linear policy class (Assumption 4.5), to parameter estimation in logistic regression with privately and adversarially corrupted labels. It proposes a single analyzer based on a shifted-and-scaled log-loss paired with randomized response, proves weighted-norm estimation error bounds under CTL and LTC (Theorem 5.4) and an L2-norm bound under uniform coverage (Theorem 5.7), and converts these into suboptimality bounds for RLHF (Theorems 6.1, 6.2) and DPO (Theorem 6.3). The paper's headline results include a separation between LTC and CTL in which LTC carries an extra c(epsilon) factor, improved root-alpha corruption dependence for robust RLHF, and the first O(1/sqrt(n)) rates for private or robust DPO under the log-linear policy class. The appendix contains detailed proofs and a claimed diagnosis of a sign-direction error in Mandal et al. (2024).","tokens_in":27178,"tokens_out":16384,"duration_ms":134017,"significance":"If the results hold as stated, the reduction framework provides an elegant unifying perspective on private and robust offline alignment, and the CTL/LTC separation is conceptually valuable. The paper ships detailed appendix proofs, a clean decomposition of the gradient into privacy, corruption, and standard logistic-regression terms, and a concrete, apparently sound identification of a sign-direction gap in prior work on robust RLHF. The improved corruption dependence for RLHF and the new O(1/sqrt(n)) DPO rates are meaningful advances over the existing state of the art. However, the headline suboptimality guarantees currently rest on an unstated data-generation assumption; this is a load-bearing gap rather than a presentation issue, and it must be addressed before the central claims can be accepted as proven.","major_comments":[{"comment":"The suboptimality bounds in Corollary 4.4, Proposition 4.6, and hence Theorems 6.1 and 6.3 require that the empirical covariance bSigma concentrates around the population matrix Sigma^diff_{pi_sft,pi_sft} used in the relative condition numbers kappa(pi_dagger,pi_ref) and kappa_Pi. This is used in step (a) of Corollary 4.4 via Lemma H.1 and asserted in the proof of Proposition 4.6 with the sentence 'Sigma^diff_{pi_sft,pi_sft} is the corresponding population matrix of bSigma'. However, Section 3 only specifies that states s_i are i.i.d. from rho; no assumption is made about how the two actions a_i^0, a_i^1 are drawn, and neither Assumption 4.1 nor Assumption 4.5 states that they are sampled from pi_sft. If the actions are generated by a different behavior policy pi_b, then bSigma concentrates around Sigma^diff_{pi_b,pi_b}, the covariance replacement in Lemma H.1 is invalid, and the displayed bounds with kappa(pi_dagger,pi_ref) or kappa_Pi do not follow; the suboptimality may depend on a different, potentially much larger concentrability coefficient. This issue is load-bearing for the paper's main suboptimality claims. Please add an explicit data-generation assumption (e.g., a_i^0, a_i^1 ~ pi_sft(·|s_i)) to Section 3 or to Assumptions 4.1/4.5, and verify that all applications of Lemma H.1 are then justified.","section":"Section 3, Corollary 4.4, Proposition 4.6, Theorems 6.1 and 6.3"},{"comment":"The reduction for DPO relies on the exact log-linear realizability of both pi_star and pi_sft (Assumption 4.5). The equality P(y_i=1|s_i,a_i^0,a_i^1) = sigma(beta(theta_star - theta_sft)^T x_i) holds only when both policies lie exactly in the class Pi with the known feature map phi; under misspecification, the preference labels do not follow the logistic regression model (3), and the estimation error bounds of Theorem 5.4 no longer connect to the DPO suboptimality. While this is an explicit assumption rather than an internal inconsistency, the paper should state the limitation prominently and clarify that Theorem 6.3 and the claimed O(1/sqrt(n)) rate are contingent on exact realizability. The current text sometimes reads as if the reduction applies to general policy classes, e.g., in the discussion of the rDPO implementation in Appendix D.","section":"Section 4.2, Proposition 4.6, Theorem 6.3"}],"minor_comments":[{"comment":"There is a typo in the theorem statement and in Definition 5.1: 'alpha, in [0, 1/2)' should be 'alpha in [0, 1/2)'.","section":"Theorem 5.4 and Definition 5.1"},{"comment":"The notation eO and eTheta is used without definition; please define soft-O notation when first used.","section":"Section 6.1 and Appendix C"},{"comment":"The experiments introduce corruption as randomly flipped labels with probability alpha, whereas the theoretical results in Theorems 5.4, 6.1, and 6.3 are for the strong adversarial corruption model of Definition 3.3. The paper should state explicitly that the experiments serve only as a sanity check for the qualitative separation and do not directly test the adversarial-corruption bounds.","section":"Appendix D"},{"comment":"The comparison with Mandal et al. (2024) claims their corruption dependence is O(alpha^{1/4}) when reduced from MDP to bandit, while the present label-only setting achieves O(sqrt(alpha)). Since Mandal et al. consider corruption of both features and labels in a linear MDP, the claim that the present setting is 'easier' and the reduction of their bound to the bandit setting deserve a more careful statement; as written, the comparison could be read as comparing incomparable settings.","section":"Section 6.1, comparison with Mandal et al."},{"comment":"The remark contrasts single-policy and all-policy concentrability, but the distinction is only meaningful after the missing action-sampling assumption is added; the wording should be updated to reference the assumed data-generating distribution.","section":"Section 4.2, Remark 4.8"}],"recommendation":"major_revision","confidential_remarks":"The paper is a substantial theoretical contribution with detailed appendix proofs and a plausible diagnosis of an error in prior work. The main blocking issue is the unstated action-sampling assumption for the covariance concentration; this is fixable by an explicit data-generation assumption. I recommend that the editor ask an independent expert to verify the claimed sign-direction error in Mandal et al. (2024) before the paper's comparative claims are relied upon. The paper is within the scope of the journal and raises no ethical concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid theory paper. The unified reduction of offline RLHF and DPO to private/corrupted logistic regression is real, the CTL/LTC separation in Theorem 5.4 is clean and sound, and the first O(1/sqrt(n)) rate for robust DPO is a genuine advance. The claimed sign-direction error in Mandal et al. (Appendix G) looks correct on inspection. Eq. (14) is a nice adaptive loss that handles CTL, LTC, and CLC without knowing the setting. You should send this out.\n\nThe soft spot that matters: the suboptimality bounds in Corollary 4.4, Proposition 4.6, and Theorems 6.1-6.3 convert the empirical-norm estimation error into kappa-based suboptimality using the identification of bSigma with Sigma_diff_{pi_sft,pi_sft}. That identification is asserted in the proof of Prop 4.6 but never stated as an assumption about how the preference data are collected. If the two actions in each sample come from some behavior policy pi_b != pi_sft, the concentration step (Lemma H.1) gives you Sigma_diff_{pi_b,pi_b}, and the displayed kappa(pi_dagger,pi_ref)/kappa_Pi bounds do not follow as written. This is fixable - add the i.i.d.-from-pi_sft sampling condition to Assumptions 4.1/4.5 and the chain closes - but as written it is load-bearing. The estimation bounds in Theorem 5.4 do not depend on this, so the core reduction survives.\n\nTwo smaller things. The CTL/LTC separation is proven for the upper bounds; the matching lower bound is only sketched/conjectured in Appendix C. The paper is honest about this, so it's a minor issue, just don't cite it as a proven gap. The experiments are supportive but not decisive: they use random-flipping corruption, not the strong adversary of the theory, and no code or data is shipped.\n\nVerdict: deserves a serious referee. I'd suggest the authors add the data-generation assumption, and ideally a matching lower bound for the separation or at least a precise conjecture. For your reading group: yes, worth the hour.","headline":"A genuinely useful reduction paper with a clean CTL/LTC separation; the suboptimality bounds need one more assumption stated explicitly before the claims as written hold.","tokens_in":27810,"tokens_out":1970,"would_cite":true,"duration_ms":16150,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Offline RLHF and DPO reduce to one noisy logistic-regression problem, and the order of privacy and corruption determines how hard that problem is.","keywords":["offline alignment","RLHF","DPO","local differential privacy","adversarial label corruption","logistic regression","Bradley-Terry model","relative condition number"],"falsifier":"Simulate logistic-regression labels with $d=20$, features $x_i$ drawn from a behavior policy different from $\\pi_{\\mathrm{sft}}$, apply randomized response and then adversarial corruption, and run Algorithm 1; if $\\|\\hat{\\theta}-\\theta_{\\mathrm{true}}\\|_2$ keeps growing or fails to match the predicted $1/\\sqrt{n}$ concentration as $n$ grows, the covariance identification premise is false. Alternatively, compare Algorithm 1 under LTC and CTL with $\\varepsilon$ fixed: if the empirical suboptimality gap does not grow with $c(\\varepsilon)$, the separation claim fails.","tokens_in":26595,"feed_emoji":"🔒","tokens_out":6518,"duration_ms":53583,"temperature":0.7,"pith_summary":"This paper claims that offline alignment of language models—whether done through reward-then-optimize RLHF or through direct DPO—is at bottom one statistical problem: estimating a logistic-regression parameter from preference labels that have been both privatized and adversarially corrupted. Under linear reward or log-linear policy assumptions, the preference labels are exactly logistic-regression labels, so suboptimality bounds for both paradigms follow from a single estimation bound. The paper's central discovery is that the order of operations matters: if labels are privatized first and corrupted second (LTC), the error carries an extra privacy-dependent factor $c(\\varepsilon)$ compared with corruption-then-privacy (CTL), making LTC the harder setting. The same reduction yields the first $O(1/\\sqrt{n})$ suboptimality rate for private or corrupted DPO, and $O(\\sqrt{\\alpha})$ robustness for RLHF under label corruption.","feed_headline":"Privacy before corruption makes alignment provably harder","feed_subtitle":"A single reduction shows LTC costs an extra c(ε) and yields the first 1/sqrt(n) robust DPO rate.","key_machinery":"The load-bearing mechanism is the reduction of preference learning to logistic regression. Under the Bradley-Terry model with a linear reward, RLHF labels satisfy the logistic model with $\\theta_{\\mathrm{true}}=\\theta^\\star$ and $x_i=\\phi(s_i,a_i^1)-\\phi(s_i,a_i^0)$; under the log-linear policy class, DPO labels satisfy it with $\\theta_{\\mathrm{true}}=\\beta(\\theta^\\star-\\theta_{\\mathrm{sft}})$. The estimation side is carried by a new loss $\\tilde{\\ell}_i(\\theta)=\\ln(1-\\sigma(\\theta^\\top x_i))+(z_i+\\sigma(\\varepsilon)-1)c(\\varepsilon)\\theta^\\top x_i$ with $c(\\varepsilon)=(e^\\varepsilon+1)/(e^\\varepsilon-1)$, which debiases randomized-response labels and is agnostic to which noise order generated them. The suboptimality translation is controlled by the relative condition number $\\kappa(\\pi^\\dagger,\\pi_{\\mathrm{ref}})$ for RLHF and its all-policy analogue $\\kappa_{\\Pi}$ for DPO, which measure how well the comparator or the whole policy class is covered by the SFT sampling distribution.","core_discovery":"On the paper's own terms, the discovery is that privacy and corruption in preference labels do not need separate theories: both RLHF and DPO, under Assumptions 4.1 and 4.5, reduce to learning $\\theta_{\\mathrm{true}}$ in the logistic model $P(y=1\\mid x)=\\sigma(\\langle\\theta_{\\mathrm{true}},x\\rangle)$, where $x_i=\\phi(s_i,a_i^1)-\\phi(s_i,a_i^0)$. Algorithm 1, a randomized-response privatizer followed by a shifted-and-scaled logistic loss, estimates this parameter with the same algorithm whether corruption precedes privacy (CTL) or follows it (LTC). The resulting bounds show LTC pays an extra $c(\\varepsilon)$ factor in the corruption and bias terms, establishing a separation between the two noise orders for both RLHF and DPO. As corollaries, the paper obtains explicit suboptimality guarantees in terms of relative condition numbers, the first $O(1/\\sqrt{n})$ rate for robust or private DPO, and robustness rates matching the best known in standard offline reinforcement learning.","pith_inferences":["The covariance identification step implicitly assumes the two actions in each preference pair are drawn from $\\pi_{\\mathrm{sft}}$; if deployment data are collected under a different behavior policy, the empirical covariance may not concentrate around $\\Sigma^{\\mathrm{diff}}_{\\pi_{\\mathrm{sft}},\\pi_{\\mathrm{sft}}}$, and the relative-condition-number bounds would not follow.","The logistic-regression reduction hinges on exact log-linear realizability of both $\\pi^\\star$ and $\\pi_{\\mathrm{sft}}$; without that realizability, the misspecification error falls outside the analysis, although the same shifted-and-scaled loss is still a sensible heuristic, as the paper's neural-net experiments indicate.","The separation between LTC and CTL is not an artifact of the linear model: the same ordering appears in the paper's GPT2-large experiments, suggesting it may be a general phenomenon worth testing in larger preference datasets.","A sharper editor-level conjecture, which the paper itself raises as an open direction, is that the $c(\\varepsilon)$ separation under the $\\sqrt{\\alpha}$ regime may be improvable to $\\sqrt{c(\\varepsilon)}$."],"forward_implications":["If the reduction is correct, RLHF under strong label corruption achieves a degradation of order $O(\\sqrt{\\alpha})$ (and $O(\\alpha/\\xi)$ under uniform coverage), matching the best known corruption-robust offline reinforcement learning bounds.","If the reduction is correct, private or corrupted DPO attains an $O(1/\\sqrt{n})$ suboptimality rate, closing the gap to the noise-free rate and improving the previously known $O(1/n^{1/4})$.","If the reduction is correct, LTC is strictly harder than CTL by a factor $c(\\varepsilon)$ in the corruption term, so privacy protection should be applied after data cleaning whenever feasible.","If the reduction is correct, a single algorithm that does not know whether privacy or corruption came first achieves the stated bounds in all scenarios, because its loss is an unbiased proxy for the standard log-loss under randomized response."],"supporting_citations":[{"why":"Supplies the Bradley-Terry preference model that turns pairwise reward comparisons into binary labels, the starting point of the reduction.","marker":"Bradley & Terry, 1952"},{"why":"Defines the DPO objective and the KL-regularized reward reparameterization that the paper analyzes under log-linear policies.","marker":"Rafailov et al., 2023"},{"why":"Provides the randomized response mechanism used as the local randomizer for label privacy.","marker":"Warner, 1965"},{"why":"Establishes the logistic-regression reduction for RLHF and the greedy/pessimistic suboptimality decomposition that this paper extends to private and corrupted labels.","marker":"Zhu et al., 2023"},{"why":"Gives the earlier locally private RLHF parameter-estimation guarantee that the paper recovers and reformulates in terms of a relative condition number.","marker":"Chowdhury et al., 2023"},{"why":"Proposes rDPO for robust DPO and provides the $O(1/n^{1/4})$ baseline that the paper improves to $O(1/\\sqrt{n})$.","marker":"Chowdhury et al., 2024"},{"why":"Provides the previous best corruption-robust RLHF analysis whose suboptimal $\\alpha^{1/4}$ dependence and proof gap the paper discusses.","marker":"Mandal et al., 2024"},{"why":"Establishes corruption-robust offline RL rates and lower bounds that the paper aligns with for the reward-based analogue.","marker":"Zhang et al., 2022"},{"why":"Supplies the total-variation contraction lemma used to argue that LTC incurs an additional privacy-related factor in the lower-bound discussion.","marker":"Kairouz et al., 2014"}],"fun_headline_variants":["Noise order decides alignment cost: LTC beats CTL","Unified reduction shows LTC pays extra c(ε) factor","First 1/√n robust DPO rate from one reduction","One logistic model unifies private, robust alignment","Privacy-before-corruption is provably tougher for alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument collapses if the preference data are not generated the way the proof assumes: the two actions in each sample must be drawn from the SFT or reference policy $\\pi_{\\mathrm{sft}}$ so the empirical covariance equals the population matrix $\\Sigma^{\\mathrm{diff}}_{\\pi_{\\mathrm{sft}},\\pi_{\\mathrm{sft}}}$, and both $\\pi^\\star$ and $\\pi_{\\mathrm{sft}}$ must be exactly representable in the log-linear class.","fun_headline_variants_meta":{"raw":{"variants":["Noise order decides alignment cost: LTC beats CTL","Unified reduction shows LTC pays extra c(ε) factor","First 1/√n robust DPO rate from one reduction","One logistic model unifies private, robust alignment","Privacy-before-corruption is provably tougher for alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000648,"raw_usage":{"total_tokens":2976,"prompt_tokens":950,"completion_tokens":2026,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":1942}},"tokens_in":566,"tokens_out":2026,"duration_ms":12936,"temperature":1.0,"reasoning_tokens":1942,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:13:27.458828+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate logistic-regression labels with $d=20$, features $x_i$ drawn from a behavior policy different from $\\pi_{\\mathrm{sft}}$, apply randomized response and then adversarial corruption, and run Algorithm 1; if $\\|\\hat{\\theta}-\\theta_{\\mathrm{true}}\\|_2$ keeps growing or fails to match the predicted $1/\\sqrt{n}$ concentration as $n$ grows, the covariance identification premise is false. Alternatively, compare Algorithm 1 under LTC and CTL with $\\varepsilon$ fixed: if the empirical suboptimality gap does not grow with $c(\\varepsilon)$, the separation claim fails.","supporting_citations":[],"review_version":1}