{"id":"dde9f08a-3ccc-4f6d-8a92-2562aa6ae9a8","arxiv_id":"2506.08266","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"HC-RLHF returns an aligned language model only after a held-out safety test certifies, with probability at least 1-delta, that expected harm (as judged by a learned cost model) is below a chosen threshold.","lead":"This paper adds a statistical safety test to reinforcement learning from human feedback, so a model is only released if a held-out check confirms its expected harmfulness is below a user-set limit with high confidence. The method targets high-stakes uses like medical or legal advice, where a harmful answer is costly.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The safety-test estimator in Algorithm 1 is never shown to be on-policy; Theorem 4.2's unbiasedness claim is false if D_s contains stored responses, so the central guarantee is unsupported as written.","rationale":"The reader's weakest_assumption was normality, but that is a residual, explicitly acknowledged approximation: the paper itself offers Hoeffding's inequality as a distribution-free alternative, and Theorem 4.2 is conditional on Assumption 4.1. The sampling mismatch is more fundamental because it invalidates the estimator itself. The proof's unbiasedness bullet in Section 4 says 'by definition of g, E[g-hat_i] = g(theta_c),' but g is defined over pi_{theta_c}; no step establishes that D_s pairs come from that policy. Under Algorithm 1 as printed, D_s is a partition of the input dataset, so the natural reading is off-policy. This is an internal inconsistency, not a disagreement with consensus. The paper's own note that the guarantee assumes a stationary prompt distribution does not address this: stationarity of D_x does not make stored responses y_i equal in distribution to pi_{theta_c}(.|x). The released code is a suitable arbiter, so I recommend REJECT for the current version: the central high-confidence guarantee does not follow as written, although a revision that samples safety-test responses from theta_c (or applies off-policy corrections) and restates Assumption 4.1 would put it back to CONDITIONAL.","tokens_in":19792,"tokens_out":10576,"duration_ms":139514,"concrete_test":"Inspect the released repository's safety-test implementation and then run a controlled comparison: with the same trained theta_c, construct D_s twice for the same 4,000 prompts—once using the stored responses used in the paper's pipeline, once by sampling y_i ~ pi_{theta_c}(.|x_i) at safety-test time. Compute U_ttest and the pass/fail decision for both. If the stored-response version differs materially from the on-policy version, the unbiasedness assertion is empirically violated; if the implementation already samples from pi_{theta_c}, the gap is notational and can be fixed by amending Algorithm 1 line 4.","verdict_should_be":"REJECT","load_bearing_attack":"The load-bearing step in Theorem 4.2 is the claim in Section 4 that each g-hat_i = C_psi(x_i, y_i), for (x_i, y_i) in D_s, is an unbiased estimator of g(theta_c) = E_{x~D_x, y~pi_{theta_c}(.|x)}[C_psi(x,y)] - tau. That equality holds only when the response y_i is sampled from the candidate policy pi_{theta_c}. Algorithm 1, however, partitions a fixed input dataset D into D_c and D_s and then evaluates C_psi on the stored pairs; it never resamples y_i from theta_c and provides no importance-weighting or other off-policy correction. If D_s was generated by SFT, the reward-model data, or any policy other than theta_c, then E[g-hat_i] is the expected cost under that behavior policy, not under theta_c, and the t-test bounds the wrong quantity. This breaks the proof before Student's t or the normality assumption is ever invoked. A reader cannot verify the guarantee from the paper: the distinction between 'held-out prompts' and 'held-out responses generated by the candidate' is exactly what the theorem requires, and it is absent from both the pseudocode and the proof.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HC-RLHF, a Seldonian-style algorithm for RLHF that decouples helpfulness and harmlessness by training separate reward and cost models, then adds a held-out safety test: after candidate selection under a pessimistic cost constraint, the algorithm computes a Student-t high-confidence upper bound on expected cost and returns the candidate only if that bound is at or below zero, otherwise returning NSF. The main theoretical claim (Theorem 4.2) is that, under Assumption 4.1, the probability that HC-RLHF returns an unsafe policy is at most delta. The experiments compare HC-RLHF with Safe RLHF on Qwen2-1.5B, Qwen2.5-3B, and Llama3.2-3B, reporting win rates, scatter plots of reward versus cost, a 30-trial failure-rate study for one base model, and a single-trial threshold sweep.","tokens_in":20016,"tokens_out":6727,"duration_ms":81076,"significance":"If the guarantee can be stated and proved correctly, the paper would make a meaningful contribution: it is the first RLHF method, to my knowledge, that attempts to provide a Seldonian high-confidence safety guarantee on expected harm while maximizing helpfulness. The Seldonian meta-architecture is standard, the decision to keep the safety test independent of candidate selection is principled, and the code is publicly available. The paper also correctly identifies that the candidate-selection penalty need not be valid for the final safety guarantee to hold, since the safety test uses the held-out split. However, the central theorem is not valid as written because the safety-test estimator is not shown to be on-policy and the threshold tau is omitted from the test statistic; these issues are load-bearing and must be repaired before the contribution can be assessed.","major_comments":[{"comment":"The unbiasedness claim in the proof of Theorem 4.2 is unsupported as written. The theorem requires E[g_hat_i] = g(theta_c) = E_{x~D_x, y~pi_{theta_c}(.|x)}[C_psi(x,y)] - tau, but Algorithm 1 partitions a fixed input dataset into D_c and D_s and sets g_hat_i = C_psi(x_i,y_i) for the stored pairs. The paper never states that the response y_i is sampled from the candidate policy pi_{theta_c} for the safety test, nor does it provide an importance-weighting or other off-policy correction. If D_s was generated by the SFT policy, the preference dataset, or any policy other than theta_c, then E[g_hat_i] is the expected cost under that behavior policy, not under the candidate policy, and the t-test bounds the wrong quantity. Please specify the generative process for D_s (e.g., draw prompts from a held-out prompt pool and sample responses from the candidate after candidate selection) and adjust the algorithm and proof accordingly.","section":"Algorithm 1, line 4; Section 4 proof"},{"comment":"Even if the responses in D_s were on-policy, the estimator in line 4 is g_hat_i = C_psi(x_i,y_i), whose expectation is E[C_psi], whereas the performance function g in Eq. (8) is E[C_psi] - tau. The safety test U_ttest(g_hat) <= 0 therefore certifies E[C_psi] <= 0, not E[C_psi] - tau <= 0. For the negative thresholds tau in Table 2, this is a strictly weaker condition than g(theta_c) <= 0, so passing the test does not imply the policy is safe according to the paper's own definition. The proof's equality E[g_hat_i] = g(theta_c) is inconsistent with the definition in line 4. The fix is to define g_hat_i = C_psi(x_i,y_i) - tau and test U_ttest(g_hat) <= 0, or to restrict the theorem to tau = 0.","section":"Algorithm 1, line 4; Eq. (8)"},{"comment":"The abstract advertises an unconditional probabilistic guarantee: 'proof that it will not return an unsafe solution with a probability greater than a user-specified threshold.' This is stronger than what the paper proves. Theorem 4.2 depends on Assumption 4.1, which requires the estimates g_hat to be normally distributed (or at least the sample mean to be normal), and Section 3.1 explicitly concedes that Student's t-test is exact only under normality and relies on the CLT as a large-m approximation. For the finite sample sizes and non-normal cost-model outputs used in the experiments, the advertised 1-delta confidence is not exact. The abstract should either state the normality assumption or the paper should use the Hoeffding bound from Appendix B with a stated boundedness assumption on C_psi. This is a central claim and should be reported accurately.","section":"Abstract; Section 3.1; Assumption 4.1"}],"minor_comments":[{"comment":"Section 2.3 says that the training dataset D 'consists of prompts sampled from D_x', but Algorithm 1 iterates over (x_i,y_i) pairs in D_s. Please clarify what D contains and where the responses y_i come from; this is part of the under-specification of the safety-test sampling.","section":"Section 2.3; Algorithm 1"},{"comment":"The statement that the failure rate 'was observed to be 0, with a standard deviation of 0' is misleading. With 30 trials and zero failures, the standard deviation of the observed proportion is not the right uncertainty measure; a binomial confidence interval (e.g., approximate 95% CI of [0, 0.116]) would be more informative. Also, this experiment uses only Qwen2-1.5b, and the Llama3.2-3b threshold sweep in Table 2 is a single trial, so the empirical support for the probabilistic guarantee across models is currently thin.","section":"Section 5.1, Seldonian Guarantee"},{"comment":"Table 2 reports one trial per threshold. A single trial cannot demonstrate a 1-delta property, and the table should be explicitly labeled as an illustrative run rather than as validation of the Seldonian guarantee.","section":"Table 2"},{"comment":"There is a typo in the REINFORCE review: 'cummulative rewrad' should be 'cumulative reward'.","section":"Appendix A"},{"comment":"The candidate-selection penalty K(delta) includes free coefficients rho_1 and rho_2, but the paper does not state explicitly that these coefficients affect only the candidate-selection stage and do not enter the validity of the final safety test. Since the safety test uses only D_s, this is not a threat to the theorem, but stating it would prevent confusion.","section":"Section 3.2, K(delta)"}],"recommendation":"major_revision","confidential_remarks":"The core Seldonian idea is sound and can be repaired, so the paper should not be rejected outright. The two load-bearing technical issues are the missing on-policy sampling specification for D_s and the omitted tau in the safety-test estimator; both are fixable within the scope of the manuscript. I would also ask the authors to align the abstract with the assumptions actually used in Theorem 4.2. The paper's heavy reliance on the authors' own prior work in the related-work discussion is noticeable but does not affect my recommendation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: the paper combines the Seldonian safety test with a decoupled reward/cost RLHF pipeline, which is a genuinely useful new combination, but the proof of the main guarantee has a hole that's easy to miss. Algorithm 1 evaluates the safety test on stored (x_i, y_i) pairs from dataset D_s, and the proof assumes those y_i are sampled from the candidate policy pi_theta_c. The pseudocode never says that. If D_s contains responses from SFT or the preference data, the expected cost is under the behavior policy, not theta_c, so the t-test bounds the wrong quantity. The unbiasedness claim in Theorem 4.2 doesn't follow from anything the paper actually states. This is the load-bearing step, so as written the guarantee is unsupported.\n\nThe paper does a lot right. The Seldonian framework is a good fit for RLHF safety, and the idea of using a pessimistic constraint during candidate selection to improve the chance of passing the safety test is sensible. The writing is clear, they make code available, and the empirical section at least attempts to validate the guarantee. The theoretical argument up to the unbiasedness step is the standard Seldonian proof, and it would work if the sampling were specified correctly.\n\nThe other soft spots are smaller. Assumption 4.1 (normality) is conceded to be an approximation via the CLT; that's fine if stated as an approximation. The empirical validation is thin—30 trials for one small model, and one trial for the threshold sweep—so the 'high-probability' claim rests mostly on the math, which is why the sampling gap matters so much. The cost model defining harm is a modeling choice, not a flaw.\n\nWho's this for? People working on safe RLHF or formal guarantees for LLM alignment. It deserves to go to peer review, because the idea is good and the fix is likely small: specify that D_s contains prompts only, and that responses are freshly sampled from theta_c after training. But a referee should insist on that change, and on re-deriving the proof with that explicit step. If the authors can't make that sampling explicit, the guarantee does not hold.\n\nMy recommendation: send it to review, but flag the safety-test sampling as a major revision.","headline":"Seldonian-style safety guarantee for RLHF that is close, but the proof has a real gap: the safety test must sample responses from the candidate policy, and Algorithm 1 never says it does.","tokens_in":20575,"tokens_out":3706,"would_cite":false,"duration_ms":39179,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HC-RLHF trains language models under a deliberately pessimistic safety constraint and only returns a model if a held-out statistical test certifies that its expected harm is below tolerance with probability at least 1−δ.","keywords":["reinforcement learning from human feedback","LLM alignment","safety constraints","high-confidence guarantees","cost model","harmlessness","Student's t-test","safety test"],"falsifier":"Run HC-RLHF with a cost model whose harm scores are heavily skewed and bounded, fixing δ = 0.1 and a small safety-test size such as m = 50, then repeatedly measure the true expected cost of returned policies; if the empirical failure rate clearly exceeds 10%, the advertised 1−δ guarantee does not hold outside the normality assumption.","tokens_in":19589,"feed_emoji":"🛡️","tokens_out":6110,"duration_ms":73340,"temperature":0.7,"pith_summary":"This paper proposes High-Confidence RLHF (HC-RLHF), a way to align language models with human preferences while providing a statistical safety certificate. It separates helpfulness from harmlessness by training separate reward and cost models, then runs a two-step pipeline: first it optimizes under an intentionally pessimistic safety constraint, then it tests the resulting policy on held-out data. The algorithm returns the policy only when a Student's t upper-confidence bound on expected harm is at or below a user-chosen tolerance; otherwise it returns a no-solution outcome that is treated as safe. The central result is a bound on the probability of ever returning an unsafe policy, and experiments on three language models show fewer harmful responses and improved helpfulness compared with an existing constrained-RLHF baseline.","feed_headline":"Safety test certifies LLMs as harmless with probability 1−δ","feed_subtitle":"HC-RLHF only deploys a model when a held-out test certifies its expected harm is below tolerance.","key_machinery":"The load-bearing object is the high-confidence upper bound on expected cost, U_ttest(ĝ) = sample mean + (sample standard deviation / √m) · t_{1−δ,m−1}, built from Student's t quantiles. This bound appears twice: first in candidate selection, where the training constraint inflates the empirical mean cost by a standard-deviation term K(δ) to make the optimization pessimistic about safety, and second in the held-out safety test that decides whether to return the candidate or to return no solution. The policy-gradient update implementing this pessimistic constraint uses an augmented reward containing the empirical mean and standard deviation of costs, with RLOO variance reduction.","core_discovery":"The paper's central claim is Theorem 4.2: for Algorithm 1, with g defined as the expected cost under the learned cost model minus a tolerance τ, and under the assumption that the mean of the safety-test estimates is normally distributed, Pr(g(alg(D)) ≤ 0) ≥ 1−δ. In other words, the algorithm either returns a candidate policy whose expected harm is certified by a Student's t upper-confidence bound to be at most τ with confidence at least 1−δ, or it returns a no-solution outcome whose harm is defined as zero. The proof shows that the safety-test estimates are unbiased and i.i.d., making the Student's t bound valid, and that the algorithm returns a policy only when that bound is non-positive.","pith_inferences":["The certificate applies to expected cost under the learned cost model, not directly to real-world harm; if the cost model mislabels harmful content, the guarantee does not cover that error.","The exact 1−δ claim depends on the sample mean of safety-test estimates being normally distributed; with heavily skewed or heavy-tailed harm scores and small safety sets, the Student's t bound is only an approximation and the stated confidence may be optimistic.","Treating the no-solution outcome as safe shifts the practical burden: an algorithm that frequently returns no solution is formally safe but may be unusable, so the data size and threshold needed to avoid frequent no-solution outcomes become important operational questions.","The guarantee assumes a stationary prompt distribution between training and deployment; adversarial rephrasing or shifts in prompt frequency are outside the certified claim, as the paper notes."],"forward_implications":["If the theorem holds, a deployed model is either certified by the safety test to respect the harm tolerance with confidence at least 1−δ, or the algorithm returns no solution and no unsafe model is released.","Because the final decision is made on held-out data by the safety test, the method cannot silently return an over-optimized policy that only looks safe under the training objective.","The same safety-test machinery can use distribution-free bounds such as Hoeffding's inequality, extending the high-confidence guarantee to settings where harm scores are known to be bounded.","Experiments indicate that HC-RLHF stays safe across stricter tolerance settings (more negative τ), whereas the constrained baseline without the safety test fails at stricter thresholds.","Users can choose the confidence level δ and the harm tolerance τ ahead of time, so the safety guarantee is tuned to the risk profile of the application."],"supporting_citations":[{"why":"Supplies the Student's t-test high-confidence upper bound used in both the safety test and the pessimistic candidate-selection constraint.","marker":"[Student, 1908]"},{"why":"Provides the high-confidence constraint framework whose probabilistic guarantee HC-RLHF instantiates for harmlessness.","marker":"[Thomas et al., 2019]"},{"why":"Contributes the decoupled helpfulness and harmlessness preference modeling and the constrained RLHF setting that HC-RLHF builds on.","marker":"[Dai et al., 2023]"},{"why":"Offers the distribution-free alternative upper bound that the paper says can replace Student's t-test in the same safety-test structure.","marker":"[Hoeffding, 1963]"},{"why":"Defines the paired-comparison preference model used to train both the reward model and the cost model.","marker":"[Bradley and Terry, 1952]"},{"why":"Supplies the RLOO estimator used as the low-variance policy-gradient baseline for the HC-RLHF optimization objective.","marker":"[Kool et al., 2019]"}],"fun_headline_variants":["Certify LLM harmlessness with probability 1−δ","High-confidence safety test for RLHF-aligned LLMs","RLHF with a statistical safety guarantee","Only deploy LLMs that pass a harmlessness test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The certificate is exact only if the safety-test cost estimates are unbiased i.i.d. draws from the candidate policy's response distribution and their sample mean is normally distributed, so that Student's t-test gives the claimed confidence.","fun_headline_variants_meta":{"raw":{"variants":["Certify LLM harmlessness with probability 1−δ","High-confidence safety test for RLHF-aligned LLMs","RLHF with a statistical safety guarantee","Only deploy LLMs that pass a harmlessness test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000252,"raw_usage":{"total_tokens":1562,"prompt_tokens":950,"completion_tokens":612,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":549}},"tokens_in":566,"tokens_out":612,"duration_ms":8270,"temperature":1.0,"reasoning_tokens":549,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:15:28.256458+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run HC-RLHF with a cost model whose harm scores are heavily skewed and bounded, fixing δ = 0.1 and a small safety-test size such as m = 50, then repeatedly measure the true expected cost of returned policies; if the empirical failure rate clearly exceeds 10%, the advertised 1−δ guarantee does not hold outside the normality assumption.","supporting_citations":[{"cited_title":"The probable error of a mean","cited_arxiv_id":null,"evidence_quote":"Supplies the Student's t-test high-confidence upper bound used in both the safety test and the pessimistic candidate-selection constraint."},{"cited_title":"Rank analysis of incomplete block designs: I","cited_arxiv_id":null,"evidence_quote":"Defines the paired-comparison preference model used to train both the reward model and the cost model."}],"review_version":1}