{"id":"3c1ee665-9098-4a66-8c41-bfcc7ada285b","arxiv_id":"2502.02921","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"HSBC learns rewards from preference batches by conservatively cutting the hypothesis space with a voting threshold, tolerating erroneous labels up to a bound set by a parameter gamma.","lead":"This paper introduces HSBC, a method that learns a reward function from human preference comparisons by repeatedly cutting away parts of the space of possible reward functions, using a voting threshold to tolerate wrong labels. It is a candidate approach for human-in-the-loop robotics and control when human feedback is noisy or unreliable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The PAC bound (Theorem 4.2) assumes queries are i.i.d. from P_XY conditional on DIS(V_i), while Algorithm 1 (Section 6.3) generates pairs via MPC under the current reward ensemble, so the implemented query process is neither i.i.d. nor fixed; Appendix E.4 does not close this gap.","rationale":"The reader's weakest-assumption diagnosis is correct and is the most load-bearing gap. The paper's headline contributions are (i) a robust containment guarantee under a bounded fraction of false labels (Lemma 5.2) and (ii) a query-complexity certificate (Theorem 4.2). Lemma 5.2 is solid and distribution-free; it deserves credit. The experimental evaluation is extensive and shows strong empirical gains, and the ablation of gamma suggests some robustness to overestimating the false rate. The weakness is that Theorem 4.2 and its appendix proof assume an i.i.d. active-learning protocol that the implemented system does not use. This severs the formal link between the main theory and the experiments. The fix is not necessarily difficult: either change the query protocol to sample from a fixed pool conditional on disagreement, or state Theorem 4.2 as applying only to that protocol and treat the online MPC results as empirical. A stronger alternative would be to prove a distribution-shift or regret-style bound for the self-generated query process, but that is a substantial new result. Given the strength of the experiments and the correctness of the robust containment lemma, conditional acceptance remains appropriate, with the theory/experiment gap made explicit and either closed or qualified.","tokens_in":21811,"tokens_out":6174,"duration_ms":61765,"concrete_test":"Re-run the Walker-Walk experiment under two query protocols: (A) the implemented online MPC protocol of Section 6.3, and (B) a fixed offline pool of trajectory pairs drawn once from a fixed P_XY, with queries selected by disagreement filtering, matching the protocol assumed in Theorem 4.2. Hold batch size, ensemble size, disagreement threshold, and false rate fixed, and compare reward error and total queries needed to reach the same performance. If protocol B's empirical error/query trade-off materially differs from protocol A, the theorem cannot be cited for the implemented algorithm; if they match, the mismatch is harmless in practice. An additional analytical check is to derive the sampling distribution induced by Algorithm 1 and verify whether it equals P_XY(·|DIS(V_i)); it does only in the degenerate case where MPC rollouts are exact i.i.d. draws from P_XY.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing theoretical result is Theorem 4.2: with all-true preferences, after K = O(zeta(d log zeta + log(log(1/epsilon)/delta)) log(1/epsilon)) disagreement-based queries, the current hypothesis space contains only rewards with preference-prediction error at most epsilon, with probability at least 1 - delta. The proof in Appendix E.3/E.4 applies the passive PAC bound to each batch B_i and uses it to show the disagreement region shrinks geometrically (Eq. 49). This reasoning is valid only if the N queried pairs in each batch are independent draws from P_XY conditioned on DIS(V_i). In the implemented HSBC (Algorithm 1, Section 6.3), trajectory pairs are produced by sampling-based MPC using the current reward ensemble (Eq. 21), with exploration noise and a disagreement filter; they are not samples from any fixed, unknown P_XY. The selection distribution therefore depends on the current ensemble E_i and changes as Theta_i changes. Consequently, conditional on being queried, the pairs are not i.i.d. from P_X|DIS(V_i), the conditional-error argument in Eq. 43 has no basis, and the halving inequality in Eq. 49 is not established for the actual algorithm. This is not a minor technicality: the advertised query-complexity certificate is a central contribution, and the experiments use the online MPC protocol, not the i.i.d. active-learning protocol assumed by the theorem. Lemma 5.2 only provides distribution-free containment of theta_H; it supplies no PAC convergence, so without Theorem 4.2 the paper has no formal query-efficiency guarantee for the method as implemented. The same gap would affect any extension of the PAC argument to noisy preferences, which the paper does not provide.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Hypothesis Space Batch Cutting (HSBC), an iterative reward-alignment method that maintains a hypothesis space of reward parameters and cuts it using batches of preference labels. For clean labels, it claims a PAC bound with active query complexity K = O(ζ(d log ζ + log(log(1/ε)/δ)) log(1/ε)) (Theorem 4.2). For noisy labels, it replaces the strict intersection cut with a thresholded vote Ci = {θ | Vi(θ) > ⌊(1−γ)N⌋ − 0.5} and proves that the true reward remains in the hypothesis space whenever each batch contains at most ⌈γN⌉ false labels (Lemma 5.2). Experiments in an MPC setting across six tasks compare HSBC with PEBBLE, RIME, SURF, MAE, and t-CE, showing competitive clean-label performance and improved robustness at 10–30% false-preference rates, plus ablations and real-human-data tests.","tokens_in":22159,"tokens_out":5542,"duration_ms":51035,"significance":"If the theoretical claims were established for the implemented algorithm, the paper would make a useful contribution: it provides a geometrically interpretable, query-efficient active-learning framework for preference-based reward learning, a conservative voting cut with a distribution-free containment guarantee under a false-label budget, and a broad empirical comparison including real human feedback. The code and website are advertised, which is helpful for reproducibility. However, the central PAC certificate is proved for a disagreement-based active-learning protocol that differs from the implemented MPC-based query process, and the robust guarantee is only a containment result, not a convergence or error bound. These gaps are load-bearing for the paper's headline claims, so the contribution is not yet fully established.","major_comments":[{"comment":"Theorem 4.2's proof requires each batch Bi to consist of independent draws from P_XY conditioned on the disagreement region DIS(Vi) of the current version space. In the implemented HSBC (Algorithm 1, Section 6.3), trajectory pairs are produced by sampling-based MPC under the current reward ensemble (Eq. 21), with exploration noise and a disagreement filter (Eq. 22). This query process is neither i.i.d. nor drawn from a fixed P_XY: it depends on the current ensemble Ei, which changes with i and with the policy induced by the learned reward. Consequently, conditional on passing the disagreement filter, the queried pairs are not samples from P_X|DIS(Vi), so the conditional-error bound in Eq. (43) has no basis and the halving inequality in Eq. (49) is not established for the actual algorithm. Appendix E.4 does not close this gap. The advertised query-complexity certificate is a central contribution, and all experiments use the online MPC protocol, so this mismatch needs to be resolved either by changing the analysis to fit the protocol or by changing the protocol to satisfy the theorem's assumptions.","section":"§6.3, Algorithm 1; Appendix E.4, Eqs. (43)–(49)"},{"comment":"The robust guarantee of Lemma 5.2 is conditional on a per-batch false-label budget of at most ⌈γN⌉. In the simulated-human experiments, the conservativeness level is set equal to the actual false rate (Fig. 5 caption), and false labels are generated by 'a random selection of human preference labels is flipped' (Section 7). For N = 10 and a 30% false rate, the probability that a batch contains more than 3 false labels is about 0.35, so Lemma 5.2's condition is violated on a substantial fraction of batches even though the average rate matches γ. The experiments therefore validate robustness under a favorable calibration of γ, not under the unknown-false-preference scenario advertised in the abstract. Please either set γ as a true upper bound in the experiments, report the empirical maximum number of false labels per batch, or provide results with γ strictly above the unknown rate across all tasks.","section":"§7, Fig. 5 caption; §5.1, Lemma 5.2"},{"comment":"The proof of Theorem 4.2 asserts log(N/d) ≤ log ζ 'with proper choice of c, c′'. From N = c ζ(d log ζ + log(1/δ′)), we have N/d ≈ c ζ log ζ, so log(N/d) ≤ log ζ would require c ζ log ζ ≤ ζ, i.e., c log ζ ≤ 1, which is false for ζ < e^{1/c}. Unless additional assumptions on ζ or a different choice of N are supplied, the step from Eq. (44) to the halving inequality Eq. (49) is not justified for all ζ > 0. This is load-bearing for the query-complexity bound, so the proof needs a corrected bound or an explicit restriction on the disagreement coefficient.","section":"Appendix E.4, Eq. (44)"},{"comment":"The robust section establishes only that the true reward θH remains in every hypothesis space when the false-label budget holds; it does not provide any PAC-style guarantee that the returned ensemble has low preference-prediction error err(rθ) under false labels. The problem statement Q2 asks whether the agent can 'still learn θH regardless of false preferences', but the theory never bounds err(rθ) for the robust procedure, nor does it show that the hypothesis space shrinks to a small error region in the presence of false labels. The empirical results suggest graceful degradation, but the 'provable robustness' claim in the abstract is stronger than the proven containment result. Please state this limitation explicitly or add an error bound for the robust algorithm.","section":"§5, Lemma 5.2; §3, Q2"}],"minor_comments":[{"comment":"Line 3 contains a typo: 'Sample an assemble Ei' should read 'Sample an ensemble Ei'.","section":"Algorithm 1"},{"comment":"The disagreement score formula is written as '4n+Ei n−Ei /N^2', which is ambiguous; it should be typeset as 4 n+_{Ei} n−_{Ei} / N^2, with n+_{Ei} and n−_{Ei} defined explicitly.","section":"Eq. (22)"},{"comment":"The passive PAC bound uses the symbol 'LP ASS' and later switches to 'M' and 'N' without defining the relationship; please use the batch size N consistently throughout the proof.","section":"Appendix E.3"},{"comment":"In the Walker row, the correlation at 0% false rate (0.584±0.035) is lower than at 10% (0.636±0.060), which is inconsistent with the text's claim of strong correlations across all rates; this deserves a comment or a corrected interpretation.","section":"Section 7.4, Table 3"},{"comment":"The Uni-RLHF reference 'Yuan et al.' lacks a year, venue, and page/arXiv information; please complete it.","section":"References"},{"comment":"The caption states that the conservativeness level equals the actual false rate, but Section 5.1 frames γ as an unknown-rate hyperparameter; please reconcile this in the main text and discuss the limitation that this favorable calibration does not test unknown false rates.","section":"Fig. 5 caption, Section 7.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the geometric framing is appealing. My main concern is that the theoretical certificates are proven for a disagreement-based active-learning protocol that differs from the implemented MPC-based query process, and that the robust guarantee is a containment result rather than a convergence or error bound. The gamma-calibration issue in the experiments further weakens the empirical support for the unknown-false-rate claim. These are fixable with a substantial revision: aligning the theorem with the protocol, correcting the ζ-dependent inequality, and re-running experiments with conservative γ choices. I see no grounds for rejection at this stage, but the revision should be treated as substantive rather than cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful practical paper, and the core geometric idea is genuinely nice. The robust cut threshold in (16) is a simple, plausible device that I haven't seen in the prior work they cite. The experiments are thorough: six tasks, comparisons to RIME, SURF, MAE, t-CE, B-Pref teachers, and four real human volunteers. The method works as advertised when the conservativeness level gamma matches the actual false rate, and the ablation suggests it is tolerant to overestimating gamma. That is real value.\n\nThe soft spots are in the theory. Lemma 5.2 is correct but essentially a restatement of the budget: if at most ceil(gamma N) labels in a batch are wrong, theta_H receives at least floor((1-gamma)N) votes, and the threshold in (16) keeps it. That is fine, but it is a conditional containment guarantee, not a learning guarantee.\n\nThe bigger problem is Theorem 4.2. The proof in Appendix E.4 starts from a PAC bound for i.i.d. samples from a fixed P_XY, conditions on DIS(V_i), and derives a halving inequality. The implemented Algorithm 1 generates trajectory pairs using sampling-based MPC with the current reward ensemble, so the pair distribution is not fixed and not independent of the hypothesis space evolution. The authors never connect the two protocols. So the advertised query-complexity certificate does not apply to the method as implemented. This is a substantial gap, though not a fatal one for the practical value: the experiments stand on their own.\n\nThe other soft spot is the false-rate assumption. The main experiments set gamma equal to the true false rate, which is oracle information. The real-human experiments set gamma to 40% as a guess, and the B-Pref section fixes gamma at 20% regardless of teacher, which is more honest. But the headline robustness claim about unknown false preferences is only conditionally supported. The paper should either provide an adaptive or validated gamma procedure, or carefully reframe the claim as conditional on the budget.\n\nWho this is for: people doing preference-based RL or human-in-the-loop RL, especially with noisy labels. It is a good read for them. Would I send it to review? Yes—the idea is original, the experiments are useful, and the theory, while overstated, is worth checking and repairing. I would accept it and ask for a revision that addresses the protocol mismatch.","headline":"A useful practical method for noisy preference learning, but the headline PAC guarantee does not apply to the implemented query process and the robustness claim leans on an oracle-tuned gamma.","tokens_in":22712,"tokens_out":3472,"would_cite":true,"duration_ms":30771,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By voting on each batch of preferences, an agent can keep the true reward inside its hypothesis space even when some labels are wrong, and with clean labels the method PAC-learns the reward in a bounded number of queries.","keywords":["preference-based reinforcement learning","reward alignment","hypothesis space cutting","voting function","noisy human feedback","active learning","disagreement-based query","model predictive control"],"falsifier":"Construct a single batch with $\\lceil \\gamma N \\rceil + 1$ false labels arranged against the true reward: then $V_i(\\theta_H) \\le \\lfloor (1-\\gamma)N \\rfloor - 1$, which is below the threshold $\\lfloor (1-\\gamma)N \\rfloor - 0.5$, so $\\theta_H$ is removed from the hypothesis space and the claimed invariance fails exactly beyond its stated budget.","tokens_in":2053,"feed_emoji":"🗳","tokens_out":6859,"duration_ms":184693,"temperature":0.7,"pith_summary":"The paper is trying to establish that a preference-based reward learner can tolerate a bounded fraction of wrong human labels without ever needing to identify which labels are wrong. The mechanism is hypothesis-space batch cutting: each batch of preferences defines a voting function over candidate rewards, and the learner keeps only rewards that receive enough votes, where the vote threshold is set by a conservativeness level $\\gamma$. The paper proves that if each batch contains at most $\\lceil \\gamma N \\rceil$ false labels, the true reward stays in every hypothesis space, so downstream planning and control remain aligned. For the all-true case it gives a probably approximately correct guarantee with a bounded query count. This matters because human feedback in real settings is noisy, and avoiding an explicit false-label filtering step keeps the learning procedure simple and geometrically interpretable.","feed_headline":"Majority-vote cuts let reward learning survive false human labels","feed_subtitle":"Batch preferences vote on which reward hypotheses survive, so wrong labels never cut out the true reward.","key_machinery":"The load-bearing object is the voting function $V_i(\\theta) = \\sum_{j=1}^{N} H\\bigl(f(\\theta, \\xi^0_{i,j}, \\xi^1_{i,j}, y_{i,j})\\bigr)$, where $H$ is the Heaviside step function and $f$ records whether reward parameter $\\theta$ agrees with the preference label $y_{i,j}$ on a trajectory pair. It turns a batch of preferences into a geometric cut through the thresholded indicator $\\mathbf{1}_{C_i}(\\theta) = H\\bigl(V_i(\\theta) - \\lfloor(1-\\gamma)N\\rfloor + 0.5\\bigr)$. The threshold is chosen so that the true reward, which receives one vote from every true label, is preserved exactly when false labels are budgeted by $\\gamma$. In the analysis, the disagreement coefficient and the VC-dimension of the reward model convert disagreement-based active sampling into a label-complexity bound.","core_discovery":"The central discovery is that the strict intersection cut of a preference batch, which fails on a single wrong label, can be replaced by a thresholded vote $C_i = \\{\\theta \\mid V_i(\\theta) > \\lfloor (1-\\gamma)N \\rfloor - 0.5\\}$, where $V_i(\\theta)$ counts how many labels in the batch agree that $\\theta$ is consistent with them. Because true labels always vote for $\\theta_H$ and false labels vote against it, $\\theta_H$ collects at least $\\lfloor (1-\\gamma)N \\rfloor$ votes whenever the batch has at most $\\lceil \\gamma N \\rceil$ false labels; the threshold is set just below that number, so $\\theta_H$ is never cut out. With all-true preferences, $\\gamma=0$ reduces the thresholded cut to the intersection, and disagreement-based queries give the PAC bound $K = O\\bigl(\\zeta(d\\log\\zeta + \\log(\\log(1/\\epsilon)/\\delta))\\log(1/\\epsilon)\\bigr)$ with error at most $\\epsilon$ and confidence at least $1-\\delta$. The paper further reports that in six simulated control and manipulation tasks the method matches a standard preference-learning baseline at zero false rate and outperforms it and several robust variants at 10% to 30% false rates.","pith_inferences":["The PAC theorem assumes trajectory pairs are drawn independently from a fixed distribution $P_{XY}$, but the implemented agent generates trajectories from its own evolving model-predictive controller under the current reward ensemble; the paper does not close this gap, so the stated bound likely does not directly govern the online experiments. A concrete test would run HSBC on a fixed offline pref","The vote threshold implies a robustness-conservativeness trade-off: setting $\\gamma$ above the true false rate preserves $\\theta_H$ but cuts less volume per batch. The paper notes that a higher $\\gamma$ increases query complexity, and one could quantify this by measuring hypothesis-space volume removed per query.","A reported weakness concerns a stochastic teacher that labels nearly tied trajectories inconsistently; this suggests that label noise concentrated inside the disagreement region can exceed a batch-level budget. An adaptive $\\gamma$ or a disagreement-weighted voting scheme is a testable remedy the paper leaves implicit.","The geometric picture aligns with standard active learning, where the disagreement coefficient controls label complexity; HSBC can therefore be read as a label-noise-tolerant form of disagreement-based active learning rather than an entirely new statistical model."],"forward_implications":["With error-free preferences, HSBC learns a reward whose preference-prediction error is at most $\\epsilon$ with probability at least $1-\\delta$ using $K = O(\\zeta(d\\log\\zeta + \\log(\\log(1/\\epsilon)/\\delta))\\log(1/\\epsilon))$ queries.","When every batch honors at most $\\lceil \\gamma N \\rceil$ false labels, the true reward hypothesis is never removed, so planning stays aligned even at the 20% and 30% label-error rates reported in the experiments.","The method needs no separate classifier, no label-flipping step, and no assumption about the distribution of false preferences; robustness comes from the vote threshold alone.","Setting $\\gamma=0$ recovers exact intersection cutting, so the robust procedure is a strict generalization of the clean-label version.","In the reported experiments, performance degrades only mildly from 0% to 30% false labels while a standard preference-learning baseline collapses, and the method also learns from real human volunteers with a conservativeness level of 40%."],"supporting_citations":[{"why":"Supplies the disagreement coefficient $\\zeta$ and the active-learning halving argument used to bound the number of batches in Theorem 4.2.","marker":"Hanneke (2007)"},{"why":"Supplies the version-space, disagreement-region, and passive PAC analysis template that the proof of Theorem 4.2 follows.","marker":"Settles (2012)"},{"why":"Supplies the VC-dimension notion and the passive PAC sample-complexity bound used to set the batch size in the proof.","marker":"Vapnik (1998)"},{"why":"Establishes the deep preference-based reward learning setting and the disagreement-based query strategy that HSBC extends.","marker":"Christiano et al. (2017)"},{"why":"Provides the baseline reward-learning algorithm and the simulated-human preference collection protocol used in the experiments.","marker":"Lee et al. (2021b)"},{"why":"Provides the teacher models (stochastic, mistake, myopic) used in Section 7.5 and evidence that standard preference learning degrades under label noise.","marker":"Lee et al. (2021a)"},{"why":"Shows ReLU multilayer perceptrons have finite VC-dimension, making Theorem 4.2 applicable to the neural network rewards tested.","marker":"Bartlett et al. (2019)"},{"why":"Provides RIME, a noisy-preference filtering baseline against which HSBC is compared at high false rates.","marker":"Cheng et al. (2024)"}],"fun_headline_variants":["Batch votes cut reward hypotheses, surviving false labels","Wrong labels can't cut out the true reward: batch voting","Thresholded batch votes keep reward learning robust to noise","Vote-based hypothesis cuts: robust reward alignment","Majority-vote cuts beat noisy labels in reward learning"],"cache_read_input_tokens":24704,"weakest_assumption_plain":"The load-bearing premise is that every batch contains no more than $\\lceil \\gamma N \\rceil$ false labels and that trajectory pairs are drawn independently from one fixed distribution; the implemented online algorithm generates its own trajectories from the evolving reward, so the fixed-distribution assumption is not met.","fun_headline_variants_meta":{"raw":{"variants":["Batch votes cut reward hypotheses, surviving false labels","Wrong labels can't cut out the true reward: batch voting","Thresholded batch votes keep reward learning robust to noise","Vote-based hypothesis cuts: robust reward alignment","Majority-vote cuts beat noisy labels in reward learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00051,"raw_usage":{"total_tokens":2516,"prompt_tokens":1015,"completion_tokens":1501,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":631,"completion_tokens_details":{"reasoning_tokens":1434}},"tokens_in":631,"tokens_out":1501,"duration_ms":9874,"temperature":1.0,"reasoning_tokens":1434,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T10:36:30.301741+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a single batch with $\\lceil \\gamma N \\rceil + 1$ false labels arranged against the true reward: then $V_i(\\theta_H) \\le \\lfloor (1-\\gamma)N \\rfloor - 1$, which is below the threshold $\\lfloor (1-\\gamma)N \\rfloor - 0.5$, so $\\theta_H$ is removed from the hypothesis space and the claimed invariance fails exactly beyond its stated budget.","supporting_citations":[{"cited_title":"A bound on the label complexity of agnostic active learning","cited_arxiv_id":null,"evidence_quote":"Supplies the disagreement coefficient $\\zeta$ and the active-learning halving argument used to bound the number of batches in Theorem 4.2."},{"cited_title":"Active Learning","cited_arxiv_id":null,"evidence_quote":"Supplies the version-space, disagreement-region, and passive PAC analysis template that the proof of Theorem 4.2 follows."},{"cited_title":"Statistical learning theory","cited_arxiv_id":null,"evidence_quote":"Supplies the VC-dimension notion and the passive PAC sample-complexity bound used to set the batch size in the proof."},{"cited_title":"Rime: Robust preference-based reinforcement learning with noisy preferences","cited_arxiv_id":null,"evidence_quote":"Provides RIME, a noisy-preference filtering baseline against which HSBC is compared at high false rates."}],"review_version":1}