{"id":"255a6688-ae0b-4498-ac77-d1dea50854a4","arxiv_id":"2506.03568","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"C-HAC combines human demonstrations and reward-based RL for driving, using distributional return estimates to decide when the agent should follow the human-guided policy versus its self-learned policy.","lead":"This paper proposes C-HAC, a two-stage method for autonomous driving that first learns from human demonstrations via distributional proxy value propagation inside the DSAC reinforcement learning algorithm, then refines the policy with reward-based self-learning while using confidence scores to switch between the human-guided and self-learned policies.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's guarantee rests on inequality (36), which is applied in the wrong regime: the term it bounds appears only when T_c=0, but (36) is asserted from the T_c=1 condition, and the confidence critic Z^c estimates the self-learning policy's return, not Q^g.","rationale":"The paper's headline contribution is the claimed safety and performance guarantee, not merely the empirical method. Theorem 1 is the only formal support for that guarantee, and the proof's decisive step, inequality (36), is not derivable from the stated confidence condition in the regime where it is used: the term it bounds in Eq. (34) is multiplied by (1−T_c), so it matters only when T_c=0, whereas (36) is asserted from the T_c=1 condition. The two conditions are mutually exclusive, and for δ=0.15 they yield inequalities of opposite sign. Independently, the confidence score uses Z^c, a critic trained toward the self-learning policy's return, while the advantage decomposition requires the human-guided policy's true Q^g. Both defects point to the same step the reader identified, so the reader's reject verdict is confirmed. The empirical results may be valid as demonstrations, but they cannot establish the formal bound; the theorem needs either a corrected derivation or a substantially different intervention analysis.","tokens_in":23640,"tokens_out":10762,"duration_ms":115292,"concrete_test":"Re-derive Eq. (36) from Eqs. (29)–(30) and check the conditioning event against Eq. (34). Analytically, when T_c=0, (30) gives P(Q^r_ζ>Q^g_ζ)>1−δ, which yields E_{a∼π_r}Q^g − E_{a∼π_g}Q^g ≥ −√(σ_r²+σ_g²)·Φ^{-1}(δ), the opposite direction of (36) for δ=0.15. Then instantiate a minimal two-state MDP in which Z^c is updated by (26)–(27) so it approximates Q^{π_r}, set δ=0.15, and choose Q^g(s,a_r)−Q^g(s,a_g) large while P(Q^c_r>Q^c_g)≤1−δ; verify whether inequality (31) still holds. A counterexample settles that Theorem 1 is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central safety/performance guarantee in Theorem 1 is unsupported because the key step, inequality (36), is not only underived but is applied to the wrong conditioning event. In the proof, the term E_{a∼π_r}Q^g(s,a) − E_{a∼π_g}Q^g(s,a) appears in Eq. (34) multiplied by (1−T_c(s)), so it contributes only in states where T_c(s)=0 and the self-learning policy is selected. By the intervention rule (30), T_c(s)=0 implies P(Q^r_ζ>Q^g_ζ)>1−δ. Inequality (36), however, is asserted from the opposite condition P(Q^r_ζ>Q^g_ζ)≤1−δ, which is exactly the T_c(s)=1 regime where the term is multiplied by zero. For the experimental setting δ=0.15, the two conditions even give inequalities of opposite sign. Separately, the confidence evaluation uses Z^c_ζ, which is updated in (26)–(27) toward the self-learning policy's return distribution, whereas the advantage decomposition (34)–(36) requires Q^g, the true value of the human-guided policy. No argument shows that comparing Z^c values bounds the true Q^g advantage. Thus the promised lower bound J(π_b) ≥ J(π_g) − (1−β)·√2·σ_max·Φ^{-1}(δ)/(1−γ) does not follow from the stated assumptions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage confidence-guided human-AI collaboration (C-HAC) framework for autonomous driving. In the first stage, distributional proxy value propagation (D-PVP) extends PVP to the distributional soft actor-critic (DSAC) framework to learn a human-guided policy from human demonstrations. In the second stage, a shared control mechanism combines this human-guided policy with a self-learning policy via a confidence-based intervention function, with the aim of continuing to improve performance while retaining human-like safety. The central theoretical claim is Theorem 1, which asserts a lower bound on the return of the combined behavior policy in terms of the human-guided policy's return, the variance bound σ_max, the confidence threshold δ, and the expected intervention rate β. The paper reports extensive simulation experiments in MetaDrive and a qualitative real-world UGV demonstration.","tokens_in":24024,"tokens_out":8817,"duration_ms":82551,"significance":"If the performance guarantee in Theorem 1 were valid, the paper would make a useful contribution: a principled way to interleave a human-learned policy with a reward-maximizing policy while bounding performance degradation. The empirical study is extensive for a robotics venue, covering multiple baselines, an ablation of the shared-control and confidence components, and a real-world deployment with code and videos offered. However, the theoretical guarantee is load-bearing for the advertised 'safety and performance guarantees' and the proof as written is not sound. Specifically, the key inequality (36) is applied to the wrong conditioning event, and the confidence evaluation uses a critic that does not estimate the Q-function of the human-guided policy as required by the proof. These are not local fixable gaps; they invalidate the central claim. The empirical results may still be of interest, but the paper's main advertised contribution is not established.","major_comments":[{"comment":"Inequality (36) is asserted under the condition P(Q^g(s,a_r) > Q^g(s,a_g)) ≤ 1-δ, which corresponds to T_c(s)=1 by the intervention rule (30). However, in the proof, the term E_{a∼π_r}Q^g(s,a) − V^g(s) is multiplied by (1−T_c(s)) in Eq. (34), so it is only present when T_c(s)=0. The bound is therefore applied to the opposite conditioning event from the one stated. For the experimental value δ=0.15, the two events give inequalities of opposite sign, so the lower bound in Eq. (31) does not follow from the given derivation. This is a load-bearing error: without (36), the proof of Theorem 1 collapses.","section":"Section II-C, Theorem 1 proof, Eq. (36)"},{"comment":"The confidence evaluation in Eqs. (29)–(30) uses the distributions Z^c_ζ, which are updated in Eqs. (26)–(27) toward the self-learning policy's return distribution. The advantage decomposition in Eq. (34), however, requires Q^g, the true value function of the human-guided policy π_g. Evaluating Z^c_ζ at the action a_g does not produce Q^g(s,a_g), and no argument is given that the confidence probability P(Q^r_ζ > Q^g_ζ) bounds the true Q^g advantage. The notational shift from Q^g in Eq. (36) to the confidence critic's outputs is not justified, so the link between the switching rule and the performance bound is missing.","section":"Section II-C, Eqs. (28)–(30) vs. Eqs. (34)–(36)"},{"comment":"The proof replaces the state-dependent factor (1−T_c(s)) inside the expectation with the constant (1−β). The quantity β defined in Eq. (32) is a particular weighted average over the state distribution under π_b, and the proof does not establish that E[(1−T_c(s)) f(s)] = (1−β) E[f(s)] for f(s) = E_{a∼π_r} Q^g(s,a) − V^g(s). This step is also unsupported and is load-bearing for the final bound.","section":"Section II-C, Eq. (34)"},{"comment":"Inequality (36) relies on the Gaussian CDF Φ and a specific relationship between the probability P(Q^r > Q^g) and the difference of means. No Gaussianity of the return distributions is established; Assumption 1 only bounds variances by σ_max. Moreover, σ_max and β are never estimated from the policies in the experiments, so even if the proof were corrected, the bound in Eq. (31) could not be evaluated or certified for the proposed method.","section":"Section II-C, Assumption 1 and Eq. (36)"}],"minor_comments":[{"comment":"The comparison is not matched on total interaction budget: C-HAC uses 1M total steps (0.05M demonstration plus 0.95M RL enhancement) while PVP and D-PVP are reported at 0.05M total steps. Please report all methods at matched total interactions or clearly justify why the budget difference does not affect the conclusions.","section":"Section III-C, Table I"},{"comment":"The real-world validation is qualitative only; please provide quantitative results such as success rate, intervention rate, and number of safety violations on Route 1 and the generalization Route 2.","section":"Section III-E"},{"comment":"The failure-probability bound is imported from reference [45] without derivation in this paper; please clarify whether the assumptions behind that bound (e.g., TD3-based PVP) carry over to the DSAC-based D-PVP setting presented here.","section":"Section II-B, Eq. (23)"},{"comment":"The text before Eq. (34) says 'the proof of Theorem 2' but the statement being proved is Theorem 1; this should be corrected.","section":"Section II-C, proof of Theorem 1"},{"comment":"The simulator name is spelled 'Metadrive' in several places; the correct name is 'MetaDrive'. Figure 11 references panel '(g)' but the figure only shows panels (a)–(f). Several figure captions contain broken unicode escape sequences that should be repaired.","section":"Throughout"},{"comment":"The reference list contains duplicates: reference [8] and reference [30] are the same source, and references [1] and [24] overlap; please de-duplicate.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper has a substantial empirical component and the code is promised, but the advertised theoretical guarantee is invalid as written. The proof of Theorem 1 applies inequality (36) to the wrong conditioning event and confuses the confidence critic's learned distribution with the true Q-function of the human-guided policy. These are load-bearing errors that cannot be addressed by local edits; the theorem would need to be re-proved under the actual switching rule, or the guarantee would need to be withdrawn. Since the guarantee is a central contribution, I recommend rejection. A resubmission that either fixes the proof or reframes the contribution around the empirical findings without the theoretical claim could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid engineering paper with an unsupported theory section. The combination is new—D-PVP ports PVP's proxy value labeling into DSAC, and the shared-control/confidence-switching layer on top is a reasonable way to cut human effort. The MetaDrive comparisons are extensive and the C-HAC numbers look plausible; code and videos are linked, which is real. If the paper were just 'here is a method that works in simulation and on a campus UGV', I'd be fairly positive.\n\nThe soft spot is Theorem 1. The proof's Eq. (36) is asserted without derivation, and the conditioning is backwards. The term that needs bounding appears only when T_c(s)=0, i.e. when the self-learning policy is chosen, but (36) is stated from the T_c=1 condition. Also, the confidence function uses the learned Z^c return distributions, while the advantage decomposition needs Q^g, the true value of the human-guided policy; the paper never connects these. So the promised lower bound J(π_b) ≥ J(π_g) − ... does not follow. That matters because the abstract sells 'safety and performance guarantees.' sigma_max and beta are not estimated from the learned policies either, so even the final bound can't be checked empirically.\n\nMinor: real-world validation is qualitative snapshots; no quantitative success/failure counts or intervention statistics on Routes 1/2. Fine as a demo, not as evidence of safety.\n\nWho's this for? Someone building human-in-the-loop RL for driving will find the two-stage recipe and the confidence-switching idea useful. The empirical comparison with PVP, HG-DAgger, IWR, and offline/IL baselines is a useful data point. But no one should cite Theorem 1 as a guarantee.\n\nRecommendation: send it to peer review—the method and experiments deserve referee attention—but the referee should require a corrected or removed Theorem 1, along with either an empirical estimate of the bound or an explicit statement that it's a heuristic. Right now the central theory is a liability, not a contribution.","headline":"A useful empirical integration of PVP and DSAC with serious experiments, but the Theorem 1 safety guarantee does not follow from the proof; treat the method as a heuristic.","tokens_in":24505,"tokens_out":4370,"would_cite":false,"duration_ms":50449,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Human demonstrations seed a driving policy that then self-improves with a confidence-gated safety switch.","keywords":["autonomous driving","human-AI collaboration","distributional reinforcement learning","proxy value propagation","shared control","confidence-based intervention","safe exploration","distributional soft actor-critic"],"falsifier":"Compute, on the trained reward critic and both policies, $E_{a \\sim \\pi_r}Q^g(s,a) - E_{a \\sim \\pi_g}Q^g(s,a)$ over states where the confidence condition $P(Q^g(s,a_r) > Q^g(s,a_g)) \\leq 1-\\delta$ holds; if any observed value exceeds $\\sqrt{\\sigma_r^2 + \\sigma_g^2} \\, \\Phi^{-1}(\\delta)$, then inequality (36) is false and the bound of Theorem 1 is not established. A second check is to repeat the proof using the reward critic $Z^c$ wherever the confidence function actually evaluates it and see whether the chain still closes.","tokens_in":23470,"feed_emoji":"🚗","tokens_out":7774,"duration_ms":72258,"temperature":0.7,"pith_summary":"The paper proposes C-HAC, a two-stage strategy for training autonomous driving policies with minimal human involvement. In the first stage, a distributional version of proxy value propagation (D-PVP) labels human actions with value +1 and agent actions with -1 inside a distributional soft actor-critic, so the return distribution itself encodes human intentions. In the second stage, a shared control mechanism blends the human-guided policy with a reward-maximizing self-learning policy, and a confidence-based intervention function computed from the return distributions decides which policy acts at each step. The paper claims that this yields a theoretical performance floor, with Theorem 1 bounding the loss of the mixed behavior policy relative to the human-guided policy, and shows empirically that C-HAC reaches higher returns, lower collision costs, and higher success rates than standard RL, safe RL, imitation learning, and prior human-AI collaboration baselines, including on a real vehicle.","feed_headline":"Driving agent learns from a human, then surpasses it","feed_subtitle":"Confidence-gated switching lifts a human-seeded driving policy to 91% success in simulation.","key_machinery":"The load-bearing object is the distributional critic: a neural network that outputs a Gaussian over soft state-action returns rather than a single number. D-PVP labels human and novice actions with Dirac deltas at +1 and -1, and the reward-free TD update propagates these proxy values through transitions. The confidence function (30) then compares the two return distributions $\\mathcal{N}(Q_r, \\sigma_r^2)$ and $\\mathcal{N}(Q_g, \\sigma_g^2)$ and switches to the human-guided policy when $P(Q_r > Q_g) \\leq 1-\\delta$. This single distributional representation does double duty: it carries human intention during demonstration learning and carries the uncertainty estimate that gates the safety-preserving switch.","core_discovery":"On its own terms, the paper's central claim is that encoding human preference directly into a distribution over returns, rather than into a scalar reward or an action label, lets an agent internalize a safe driving style from very few interventions and then safely exceed it. The distributional proxy value network $Z^g$ is trained with a proxy-value loss that pushes human actions toward the return distribution $\\delta_1$ and novice actions toward $\\delta_{-1}$, while a reward-free TD update spreads those values through the state space. A second distributional critic $Z^c$ estimates cumulative rewards for both policies and feeds the confidence function $T_c(s)$, which selects the human-guided policy whenever the self-learning policy's advantage is not confident at level $1-\\delta$. Theorem 1 states that the behavior policy's return is no worse than $J(\\pi_g) - (1-\\beta)\\sqrt{2}\\sigma_{\\max}\\Phi^{-1}(\\delta)/(1-\\gamma)$. In the MetaDrive benchmark the method achieves an average return of 392.92, an episodic safety cost of 0.16, and a 91% success rate, and it completes a real-world campus route with a human-intervention switch.","pith_inferences":["The confidence-gated switch is a generic mechanism: any human-AI collaboration setup with a distributional value function could replace the driving-specific policy pair with task-specific policies, turning human oversight into automatic, variance-based arbitration.","The threshold $\\delta$ offers an explicit, tunable trade-off between safety and autonomy; setting $\\delta$ lower would gate the self-learning policy more conservatively, an option the paper does not explore.","The guarantee's practical validity rests on inequality (36), which the paper asserts without proof; measuring its left-hand side on the trained critics would show whether the safety floor actually holds in the reported runs.","A testable extension is to replace the Gaussian assumption in (29) with a non-parametric confidence estimate, preserving the arbitration rule when return distributions are skewed."],"forward_implications":["A competent driving policy can be learned from about 15,000 human-guided steps (roughly an hour of simulator time) and then continue improving without further human input.","The confidence-based switch limits the performance drop when the reward signal is added: the full method shows about a 20% decline versus 70% without shared control and 50% without confidence evaluation.","Because the safety floor in Theorem 1 is tied to the bounds on return-distribution variance, the same recipe could certify a minimum performance for a deployed policy.","The method transfers from simulation to a real vehicle with fused LiDAR, camera, and radar observations, including generalization to a second route.","Compared with offline methods trained on the same 50K human dataset, C-HAC's online trajectory-based learning avoids the distribution-shift failure that keeps BC, GAIL, and CQL below 10% success."],"supporting_citations":[{"why":"Supplies the original proxy value propagation method that D-PVP extends into the distributional setting.","marker":"[41]"},{"why":"Provides the distributional soft actor-critic framework whose return distributions carry proxy values and confidence estimates.","marker":"[53]"},{"why":"Provides the MetaDrive safety benchmark and scenario suite used for all simulation comparisons.","marker":"[48]"},{"why":"Supplies the state-distribution discrepancy bound used as Lemma 1 in the proof of Theorem 1.","marker":"[46]"},{"why":"Supplies the performance-difference identity used as Lemma 2 in the proof of Theorem 1.","marker":"[47]"},{"why":"Provides the upper bound on the probability of failure for the human-intervention behavior policy in the demonstration stage.","marker":"[45]"},{"why":"Interactive imitation baseline (HG-DAgger) that C-HAC must beat in the human-AI collaboration comparison.","marker":"[33]"},{"why":"Intervention-weighted regression baseline (IWR) that C-HAC must beat in the human-AI collaboration comparison.","marker":"[36]"}],"fun_headline_variants":["Human-guided AI driving: few interventions, then it exceeds you","Driving policy learns from human, then outdrives human","Confidence-based switch lets AI outdrive its human teacher","Distributional proxy values cut human input, boost safety","Human-AI driving: minimal guidance, maximal performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The safety guarantee in Theorem 1 collapses if inequality (36) does not hold: the step that turns a confidence threshold into a bound on expected advantage is asserted, not derived, and it mixes the reward critic with the true value function.","fun_headline_variants_meta":{"raw":{"variants":["Human-guided AI driving: few interventions, then it exceeds you","Driving policy learns from human, then outdrives human","Confidence-based switch lets AI outdrive its human teacher","Distributional proxy values cut human input, boost safety","Human-AI driving: minimal guidance, maximal performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000717,"raw_usage":{"total_tokens":3277,"prompt_tokens":1057,"completion_tokens":2220,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":673,"completion_tokens_details":{"reasoning_tokens":2139}},"tokens_in":673,"tokens_out":2220,"duration_ms":18696,"temperature":1.0,"reasoning_tokens":2139,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:59:18.018773+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, on the trained reward critic and both policies, $E_{a \\sim \\pi_r}Q^g(s,a) - E_{a \\sim \\pi_g}Q^g(s,a)$ over states where the confidence condition $P(Q^g(s,a_r) > Q^g(s,a_g)) \\leq 1-\\delta$ holds; if any observed value exceeds $\\sqrt{\\sigma_r^2 + \\sigma_g^2} \\, \\Phi^{-1}(\\delta)$, then inequality (36) is false and the bound of Theorem 1 is not established. A second check is to repeat the proof using the reward critic $Z^c$ wherever the confidence function actually evaluates it and see whether the chain still closes.","supporting_citations":[{"cited_title":"Learning from active human involvement through proxy value propagation,","cited_arxiv_id":null,"evidence_quote":"Supplies the original proxy value propagation method that D-PVP extends into the distributional setting."},{"cited_title":"Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors,","cited_arxiv_id":null,"evidence_quote":"Provides the distributional soft actor-critic framework whose return distributions carry proxy values and confidence estimates."},{"cited_title":"Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Provides the MetaDrive safety benchmark and scenario suite used for all simulation comparisons."},{"cited_title":"Guarded policy optimization with imperfect online demonstrations,","cited_arxiv_id":null,"evidence_quote":"Supplies the state-distribution discrepancy bound used as Lemma 1 in the proof of Theorem 1."},{"cited_title":"Trust region policy optimization,","cited_arxiv_id":null,"evidence_quote":"Supplies the performance-difference identity used as Lemma 2 in the proof of Theorem 1."},{"cited_title":"Human as AI mentor: Enhanced human-in-the-loop reinforcement learning for safe and effi- cient autonomous driving,","cited_arxiv_id":null,"evidence_quote":"Provides the upper bound on the probability of failure for the human-intervention behavior policy in the demonstration stage."}],"review_version":1}