{"id":"adf7afde-507d-43df-84be-4bb3e57b3ee6","arxiv_id":"2505.06601","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Non-asymptotic regret bounds for deep-network reward estimators from pairwise comparisons, with a Tsybakov-style margin condition that accelerates the rate.","lead":"This paper derives statistical learning bounds for reward models trained from pairwise human preferences with deep neural networks, including a sharper regret rate when preferences are clear. It also runs synthetic experiments showing that network architecture and label noise affect regret as the theory predicts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.8/3.9 hinges on Appendix D.4 replacing the population action-pair distribution by empirical counts n_ij/N in the Hessian of the population log-likelihood; since no concentration is shown, the claimed L2 and regret rates are not established.","rationale":"The paper's headline contribution is the architecture-dependent regret bound in Theorem 3.9, whose proof reduces to Theorem 3.8. The load-bearing step is the lower bound in Appendix D.4, where the population log-likelihood's Hessian is replaced by an empirical average over the observed comparison counts n_ij/N. Under the random-design model of Section 2, this equality is not justified: the action pairs are sampled from P(a1,a0|s), and the empirical counts are a random proxy. Assumption 3.7 only assumes the empirical Laplacian has a spectral gap; it says nothing about the population pair distribution, and the paper does not show that the empirical gap concentrates or that the population gap is bounded. The reader's weakest assumption identifies this same empirical-versus-population gap, and I agree. The proof's displayed Taylor expansion also appears to omit the 1/2 factor from the quadratic term and the expectation over the comparison outcome, further indicating that the step is not a routine calculation. Since this inequality is the only bridge from excess likelihood to ||r-hat-r*||_{L2}, the main theorem is unsupported as written. A revision would need either an explicit fixed-design assumption with n_ij fixed, or a concentration argument showing lambda_2 of the empirical comparison graph is bounded below with high probability and that the population Hessian can be lower bounded through the population pair probabilities. The concern is substantive, not stylistic, so the rejection verdict stands.","tokens_in":21127,"tokens_out":6911,"duration_ms":76321,"concrete_test":"Take |A|=2, a Bradley-Terry model, states uniform, P(a1=1,a0=0|s)=p=1/2 and P(a1=0,a0=1|s)=1/2. Let the realized dataset have all N comparisons ordered (1,0), so n_01/N=1 and lambda_2(Lambda)=2. Compute the population Hessian term E[partial^2 log g(y;zeta)(h_{a1}-h_{a0})^2] and compare it with the expression used in D.4, (1/N) sum_{i<j} n_ij(...); they differ by the factor p=1/2, and the Taylor coefficient 1/2 is also absent. Recompute the claimed lower bound with r-hat chosen as a constant shift; if the bound fails, the D.4 step is invalid.","verdict_should_be":"REJECT","load_bearing_attack":"The central result Theorem 3.9 inherits from Theorem 3.8, whose proof in Appendix D.4 contains the step l(r*)-l(r-hat) = integral over S of (1/N) sum_{i<j} n_ij partial^2/partial u^2 log g(y; zeta) (r-hat(s,a_i)-r-hat(s,a_j)-(r*(s,a_i)-r*(s,a_j)))^2 d rho_s >= kappa_2 kappa_Lambda ||r-hat-r*||^2_{L2(S,ell2)}. For the population log-likelihood l(r) = E log g(y; r(s,a1)-r(s,a0)) defined in Section 3, the Hessian is an expectation over the action-pair distribution P(a1,a0|s), not over the realized comparison counts n_ij/N. Those counts are random and are not equal to the population pair probabilities, pointwise or in expectation, and no concentration argument is supplied. Assumption 3.7 controls only the empirical Laplacian of n_ij/N, so it does not justify the inequality unless one conditions on a fixed design with exactly those counts. This lower bound is the only link from excess risk to the L2 error used in Theorem 3.8, so the advertised non-asymptotic rate and its margin-accelerated version in Theorem 3.9 are not proven as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies reward modeling from pairwise comparison data using deep ReLU neural networks in a non-parametric setting. It defines an MLE over a DNN function class, decomposes the excess risk into stochastic and approximation error, and derives non-asymptotic bounds on the L2 estimation error (Theorem 3.8) and on the regret of the induced policy (Theorem 3.9). A margin-type condition (Assumption 2.1) is introduced to model 'clear human preferences', and the paper claims a faster regret rate O(N^{-β/[(d+2β)(3-2α)]}) under this condition, with numerical experiments illustrating the architecture trade-off and the effect of label noise.","tokens_in":21377,"tokens_out":5447,"duration_ms":56994,"significance":"If the main results were valid, the paper would make a useful contribution: it is one of the first non-parametric analyses of DNN-based reward modeling, it makes the dependence on network architecture explicit, and it connects the quality of pairwise comparison data to sample efficiency through a margin condition. The experimental section also provides a reasonable sanity check of the claimed approximation-stochastic trade-off. However, the central proof step connecting the excess likelihood risk to the L2 error is not justified, and the advertised regret rates in Theorems 3.8 and 3.9 are not established as written.","major_comments":[{"comment":"The Taylor expansion after 'the first-order optimality condition' replaces the population Hessian of l(r) with the empirical expression (1/N)Σ_{i<j} n_{ij} ∂²/∂u² log g(y;ζ) (r̂(s,a_i)-r̂(s,a_j)-(r*(s,a_i)-r*(s,a_j)))², but the population log-likelihood l(r)=E[log g(y;r(s,a_1)-r(s,a_0))] is an expectation over the random design P(a_1,a_0|s)ρ_s, not over the realized comparison counts n_{ij}/N. No concentration argument is supplied linking n_{ij}/N to the population pair probabilities, and even in expectation the pooled counts do not equal the state-dependent pair probabilities that would be needed inside the integral over S. Since this inequality is the only link from excess risk to the L2 error, Theorem 3.8, and hence Theorem 3.9, are not proven.","section":"Appendix D.4 (proof of Theorem 3.8)"},{"comment":"The lower bound used at the end of Appendix D.4, κ₂ ∫_S Σ_{i<j} (n_{ij}/N)(v_i-v_j)² dρ_s ≥ κ₂ κ_Λ ||v||²_{L2(S,ℓ2)} with v_a = r̂(s,a)-r*(s,a), requires control of the comparison graph at each state s: for fixed s the quadratic form is Σ_{i<j} w_{ij}(s)(v_i(s)-v_j(s))² with weights proportional to the state-conditional probability P(a_i,a_j|s) or to state-specific empirical counts. Assumption 3.7 only lower-bounds the spectral gap of the aggregate Laplacian built from pooled counts n_{ij}/N. The aggregate spectral gap does not control the statewise quadratic form, so the claimed L2(S,ℓ2) bound does not follow even if the counts were concentrated around their expectations.","section":"Assumption 3.7 and the last inequality of Appendix D.4"}],"minor_comments":[{"comment":"The sentence 'We denoted as d the dimension' is grammatically incomplete; it should read 'We denote by d the dimension of the input for the reward function r.'","section":"Section 2, after Eq. (1)"},{"comment":"In the first displayed equation of the proof, the integrand is written as r*(s,πr*(s))-r*(s,πr*(s)); the second occurrence should be r*(s,π_hat_r(s)).","section":"Appendix C, proof of Theorem 2.5"},{"comment":"The final line contains 'E(Cr)' which should be 'E(r)'.","section":"Appendix C.1, proof of Corollary 2.6"},{"comment":"The quantities n_{ij} are used in Assumption 3.7 but are never formally defined in the main text; they should be defined when the dataset is introduced in Section 2.","section":"Assumption 3.7"},{"comment":"The statement that the depth is O(√N) is loose: with M₂=N^{d/(2d+4β)} the depth is O(N^{d/(2d+4β)} polylog(N)), which equals O(√N) only when d≫β; the text should state the exact growth condition rather than asserting d≫β without justification.","section":"Appendix D.4"}],"recommendation":"reject","confidential_remarks":"The gap in Appendix D.4 is not a local technical omission but a mismatch between the random-design data model in Section 2 and the use of pooled empirical comparison counts in a population-level Taylor expansion. This affects the main advertised theorem. If the authors resubmit with a fixed comparison-design formulation, or with a state-dependent coverage assumption and a real concentration argument, the paper could be worth reconsidering; the experimental study and the margin-condition idea are potentially useful."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading for its basic idea: it brings nonparametric DNN approximation rates into reward modeling from pairwise comparisons and adds a margin-type condition to accelerate regret. That combination is genuinely new, and if the results were fully established, the practical guidance (depth growing with sqrt N, filtering ambiguous comparisons) would be useful.\n\nWhat works well: Theorem 2.5 is a clean margin-based regret bound, and the proof in Appendix C is logically sound. The stochastic and approximation error bounds in Propositions 3.5 and 3.6 are standard but carefully assembled, and the emphasis on deep narrow networks rather than wide shallow ones is sensible.\n\nThe problem is the load-bearing step in Appendix D.4. To convert excess likelihood into L2 error, the proof replaces the population Hessian of l(r*)-l(r) with the empirical comparison graph through the term (1/N) sum_{i<j} n_ij ... . That is not the Hessian of the population log-likelihood; l(r) is an expectation over the action-pair distribution, not over realized counts. No concentration argument links the empirical Laplacian to the population pair probabilities. Assumption 3.7 controls only the empirical Laplacian, so the inequality connecting excess risk to ||r_hat - r*||^2_{L2} is not justified. This gap is exactly where Theorems 3.8 and 3.9 get their advertised rates, so the main DNN guarantee is not established as written.\n\nTwo smaller issues: the Taylor expansion in the same step appeals to a zeta that may fall outside the bounded interval used to define kappa2; and the abstract's claim that the improved rate is 'independent of the specific estimators used' is only true for the margin-to-regret step, not for the overall bound.\n\nThis paper should not be desk rejected. The core idea is plausible, the flaw is technical and probably fixable by either moving to a fixed design with known comparison topology or proving concentration of the empirical Laplacian under the random design. A serious referee can insist on that fix. I would not cite it in this form, but I would bring it to the reading group to see how the proof issue plays out.\n\nRecommendation: send to peer review with a clear request to redo the proof of Theorem 3.8.","headline":"A promising combination of DNN approximation and margin conditions for pairwise reward modeling, but the main theorem hinges on an unjustified swap of empirical Laplacian for population risk in Appendix D.4.","tokens_in":21925,"tokens_out":3146,"would_cite":false,"duration_ms":30876,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G05","62G08","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that when human preferences satisfy a margin condition, deep-network reward estimators achieve regret $O(N^{-\\beta/[(d+2\\beta)(3-2\\alpha)]})$ with high probability.","keywords":["reward modeling","pairwise comparisons","deep ReLU networks","margin condition","regret bound","nonparametric estimation","Reinforcement Learning from Human Feedback","Hölder smoothness"],"falsifier":"Repeatedly draw the random action-pair design from Section 2 at a fixed sample size over a modest action set, form the empirical comparison Laplacian, and measure its second-smallest eigenvalue: if it falls below any fixed positive $\\kappa_\\Lambda$ with non-negligible frequency, then the sampling model does not populate the theorem's coverage assumption, and the stated high-probability regret bound cannot follow from the proof as written.","tokens_in":20903,"feed_emoji":"🎯","tokens_out":18050,"duration_ms":148830,"temperature":0.7,"pith_summary":"This paper tries to establish that deep ReLU reward models trained by maximum likelihood on pairwise comparison data carry explicit nonparametric regret guarantees, and that the guarantees improve when human preferences are unambiguous. The central bound is probabilistic: with a properly sized network the greedy policy induced by the estimated reward suffers regret of order $O(N^{-\\beta/[(d+2\\beta)(3-2\\alpha)]})$ up to logarithms, where $\\beta$ is the Hölder smoothness of the true reward and $\\alpha$ measures how often the optimal action wins by a clear margin. Without that margin condition the same machinery gives the slower rate $O(N^{-\\beta/[3(d+2\\beta)]})$, so the paper's point is that clear human beliefs are a quantifiable source of sample efficiency. The improvement is claimed to be independent of the specific estimator, which is why a sympathetic reader would care: it connects the empirical folklore about clean RLHF data to concrete rates.","feed_headline":"Clear human feedback provably speeds up deep reward learning","feed_subtitle":"Clear human preferences yield regret of order O(N^{-β/((d+2β)(3-2α))}), beating the margin-free rate.","key_machinery":"The argument rides on two named objects. The first is the margin condition (Assumption 2.1), which bounds the state mass where the optimal action's winning probability lies within $t$ of $1/2$ by $c\\,t^{\\alpha/(1-\\alpha)}$; Lemma 2.4 converts it into a control on the reward gap, and Theorem 2.5 turns the squared $L^2$ error $\\|\\hat r-r^*\\|^2_{L^2(S,\\ell^2)}$ into regret at the $1/(3-2\\alpha)$ power of that error. The second is the comparison graph Laplacian $\\Lambda$ and its spectral gap $\\lambda_2(\\Lambda)$ (Assumption 3.7), which is what lets the proof convert excess likelihood into the functional error $\\|\\hat r-r^*\\|^2_{L^2(S,\\ell^2)}$; without a positive $\\kappa_\\Lambda$, the maximum-likelihood estimator can have vanishing excess risk yet nonzero estimation error. Around these, the proof balances the ReLU network's approximation error against a covering-number-based stochastic error, producing the architecture-dependent width and depth prescriptions of Theorem 3.9.","core_discovery":"On its own terms, the paper's central discovery is a rate identity. For a deep ReLU network of width $O(d^{\\lfloor\\beta\\rfloor+1})$ and depth $O(\\sqrt{N})$, the maximum-likelihood reward estimator $\\hat r$ satisfies, with probability at least $1-\\delta$, that the regret $E(\\hat r)$ is of order $N^{-\\beta/[(d+2\\beta)(3-2\\alpha)]}$, where $\\alpha$ is the margin exponent from Assumption 2.1 and $\\beta$ is the Hölder smoothness of the true reward $r^*$. The bound balances a deterministic approximation error that decreases like $(M_1M_2)^{-2\\beta/d}$ against a stochastic error controlled through the network's covering number. Functional convergence of $\\hat r$ to $r^*$ in squared $L^2(S,\\ell^2)$ norm is inherited from excess likelihood only through the spectral gap $\\kappa_\\Lambda$ of the comparison graph Laplacian, and the margin exponent converts that $L^2$ error into regret at the power $1/(3-2\\alpha)$. The no-margin case $\\alpha=0$ recovers the slower regret exponent $1/3$.","pith_inferences":["The paper's proof as written requires the empirical comparison Laplacian to have a spectral gap bounded below, but gives no concentration argument showing the random action-pair design produces such a gap; until that step is supplied, Theorem 3.9 is best read as conditional on the realized dataset's comparison graph.","Because the margin condition is stated on observables rather than on the unknown reward, a practical extension is to estimate $\\alpha$ from the empirical distribution of winning probabilities and use it to pre-screen datasets for fast learning.","The spectral-gap condition implies a data-collection design principle the authors leave implicit: compare actions in a balanced, well-connected pattern rather than concentrating queries on popular pairs, which should matter as much as sample size and is testable by comparing regret under star-shaped versus balanced comparison graphs.","The margin-condition machinery parallels fast-rate classification under label noise, which suggests testable extensions with adaptive margin exponents and active querying that selects the most informative action pairs."],"forward_implications":["Datasets with clear preferences converge provably faster: the margin exponent $\\alpha$ improves the regret rate from $N^{-\\beta/[3(d+2\\beta)]}$ to $N^{-\\beta/[(d+2\\beta)(3-2\\alpha)]}$.","Network shape matters in finite samples: width should scale like $d^{\\lfloor\\beta\\rfloor+1}$ and depth like $\\sqrt{N}$, and over-wide networks inflate stochastic error and degrade regret.","The margin bonus is estimator-agnostic: any reward estimator using pairwise comparisons inherits the same acceleration, so the conclusion transfers to other comparison-based fine-tuning algorithms.","The probability of selecting a suboptimal action decays like $(\\|\\hat r-r^*\\|^2_{L^2(S,\\ell^2)})^{\\alpha/(3-2\\alpha)}$, approaching the fastest rate as preferences become nearly deterministic.","The guarantees are nonparametric: they apply to Hölder-smooth reward functions rather than linear or otherwise parametric classes, removing a misspecification assumption common in earlier comparison-based analyses."],"supporting_citations":[{"why":"Supplies the comparison-based RLHF regret setup, the Laplacian coverage condition, and the linear-model baseline that this paper extends to a nonparametric deep-network setting.","marker":"Zhu et al. (2023)"},{"why":"Source of the margin-condition fast-rate technique that Assumption 2.1 adapts to reward modeling.","marker":"Tsybakov (2004)"},{"why":"Provides the plug-in classifier fast-rate argument behind Theorem 2.5's regret derivation.","marker":"Audibert and Tsybakov (2007)"},{"why":"Supplies the ReLU network approximation theorem for Hölder functions used to bound the approximation error in Proposition 3.6.","marker":"Jiao et al. (2023)"},{"why":"Supplies the covering-number bound for deep ReLU networks used in the stochastic error estimate of Proposition 3.5.","marker":"Shen (2024)"},{"why":"Justifies the role of the comparison graph's spectral gap and the zero-sum identifiability constraint in converting likelihood loss into $L^2$ estimation error.","marker":"Shah et al. (2016)"},{"why":"Provides the logistic comparison model used as the canonical example of the margin condition.","marker":"Bradley and Terry (1952)"},{"why":"Documents the empirical importance of label quality in RLHF, motivating the margin-type assumption and the data-corruption experiments.","marker":"Wang et al. (2024)"}],"fun_headline_variants":["Clear human preferences sharpen deep reward learning rates","Margin condition tightens deep reward regret bounds","Deep reward learning rate improves with clear feedback","Clear beliefs yield sharper regret for deep reward models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the realized set of action pairs actually compared in the data forms a well-connected network, with the second-smallest eigenvalue of the comparison Laplacian bounded below by a fixed positive constant; the sampling model for action pairs is never shown to guarantee this, and if some actions are compared only rarely the proof's conversion from likelihood loss to estimation error collapses.","fun_headline_variants_meta":{"raw":{"variants":["Clear human preferences sharpen deep reward learning rates","Margin condition tightens deep reward regret bounds","Deep reward learning rate improves with clear feedback","Clear beliefs yield sharper regret for deep reward models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000324,"raw_usage":{"total_tokens":1810,"prompt_tokens":926,"completion_tokens":884,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":828}},"tokens_in":542,"tokens_out":884,"duration_ms":9436,"temperature":1.0,"reasoning_tokens":828,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:39:27.625317+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeatedly draw the random action-pair design from Section 2 at a fixed sample size over a modest action set, form the empirical comparison Laplacian, and measure its second-smallest eigenvalue: if it falls below any fixed positive $\\kappa_\\Lambda$ with non-negligible frequency, then the sampling model does not populate the theorem's coverage assumption, and the stated high-probability regret bound cannot follow from the proof as written.","supporting_citations":[],"review_version":1}