Pith. sign in

REVIEW 4 major objections 6 minor 55 references

Confidence-Guided Human-AI Collaboration: Reinforcement Learning with Distributional Proxy Value Propagation for Autonomous Driving

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Human demonstrations seed a driving policy that then self-improves with a confidence-gated safety switch.

desk verdict A useful empirical integration of PVP and DSAC with serious experiments, but the Theorem 1 safety guarantee does not follow from the proof; treat the method as a heuristic. read the letter →

arxiv 2506.03568 v2 pith:3ERSEYNI submitted 2025-06-04 cs.RO cs.AI

Li Zeqiao , Wang Yijing , Wang Haoyu , Li Zheng , Li Peng , Zuo zhiqiang , Hu Chuan This is my paper
classification cs.ROcs.AI
keywords autonomousdrivinghuman-AIcollaborationdistributionalreinforcementlearningproxyvaluepropagationsharedcontrolconfidence-basedinterventionsafeexplorationsoftactor-critic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes C-HAC, a two-stage strategy for training autonomous driving policies with minimal human involvement. In the first stage, a distributional version of proxy value propagation (D-PVP) labels human actions with value +1 and agent actions with -1 inside a distributional soft actor-critic, so the return distribution itself encodes human intentions. In the second stage, a shared control mechanism blends the human-guided policy with a reward-maximizing self-learning policy, and a confidence-based intervention function computed from the return distributions decides which policy acts at each step. The paper claims that this yields a theoretical performance floor, with Theorem 1 bounding the loss of the mixed behavior policy relative to the human-guided policy, and shows empirically that C-HAC reaches higher returns, lower collision costs, and higher success rates than standard RL, safe RL, imitation learning, and prior human-AI collaboration baselines, including on a real vehicle.

What carries the argument

The load-bearing object is the distributional critic: a neural network that outputs a Gaussian over soft state-action returns rather than a single number. D-PVP labels human and novice actions with Dirac deltas at +1 and -1, and the reward-free TD update propagates these proxy values through transitions. The confidence function (30) then compares the two return distributions $\mathcal{N}(Q_r, \sigma_r^2)$ and $\mathcal{N}(Q_g, \sigma_g^2)$ and switches to the human-guided policy when $P(Q_r > Q_g) \leq 1-\delta$. This single distributional representation does double duty: it carries human intention during demonstration learning and carries the uncertainty estimate that gates the safety-preserving switch.

What would settle it

Compute, on the trained reward critic and both policies, $E_{a \sim \pi_r}Q^g(s,a) - E_{a \sim \pi_g}Q^g(s,a)$ over states where the confidence condition $P(Q^g(s,a_r) > Q^g(s,a_g)) \leq 1-\delta$ holds; if any observed value exceeds $\sqrt{\sigma_r^2 + \sigma_g^2} \, \Phi^{-1}(\delta)$, then inequality (36) is false and the bound of Theorem 1 is not established. A second check is to repeat the proof using the reward critic $Z^c$ wherever the confidence function actually evaluates it and see whether the chain still closes.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that encoding human preference directly into a distribution over returns, rather than into a scalar reward or an action label, lets an agent internalize a safe driving style from very few interventions and then safely exceed it. The distributional proxy value network $Z^g$ is trained with a proxy-value loss that pushes human actions toward the return distribution $\delta_1$ and novice actions toward $\delta_{-1}$, while a reward-free TD update spreads those values through the state space. A second distributional critic $Z^c$ estimates cumulative rewards for both policies and feeds the confidence function $T_c(s)$, which selects the human-guided policy whenever the self-learning policy's advantage is not confident at level $1-\delta$. Theorem 1 states that the behavior policy's return is no worse than $J(\pi_g) - (1-\beta)\sqrt{2}\sigma_{\max}\Phi^{-1}(\delta)/(1-\gamma)$. In the MetaDrive benchmark the method achieves an average return of 392.92, an episodic safety cost of 0.16, and a 91% success rate, and it completes a real-world campus route with a human-intervention switch.

Load-bearing premise

The safety guarantee in Theorem 1 collapses if inequality (36) does not hold: the step that turns a confidence threshold into a bound on expected advantage is asserted, not derived, and it mixes the reward critic with the true value function.

Editorial extensions

If this is right

  • A competent driving policy can be learned from about 15,000 human-guided steps (roughly an hour of simulator time) and then continue improving without further human input.
  • The confidence-based switch limits the performance drop when the reward signal is added: the full method shows about a 20% decline versus 70% without shared control and 50% without confidence evaluation.
  • Because the safety floor in Theorem 1 is tied to the bounds on return-distribution variance, the same recipe could certify a minimum performance for a deployed policy.
  • The method transfers from simulation to a real vehicle with fused LiDAR, camera, and radar observations, including generalization to a second route.
  • Compared with offline methods trained on the same 50K human dataset, C-HAC's online trajectory-based learning avoids the distribution-shift failure that keeps BC, GAIL, and CQL below 10% success.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The confidence-gated switch is a generic mechanism: any human-AI collaboration setup with a distributional value function could replace the driving-specific policy pair with task-specific policies, turning human oversight into automatic, variance-based arbitration.
  • The threshold $\delta$ offers an explicit, tunable trade-off between safety and autonomy; setting $\delta$ lower would gate the self-learning policy more conservatively, an option the paper does not explore.
  • The guarantee's practical validity rests on inequality (36), which the paper asserts without proof; measuring its left-hand side on the trained critics would show whether the safety floor actually holds in the reported runs.
  • A testable extension is to replace the Gaussian assumption in (29) with a non-parametric confidence estimate, preserving the arbitration rule when return distributions are skewed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a two-stage confidence-guided human-AI collaboration (C-HAC) framework for autonomous driving. In the first stage, distributional proxy value propagation (D-PVP) extends PVP to the distributional soft actor-critic (DSAC) framework to learn a human-guided policy from human demonstrations. In the second stage, a shared control mechanism combines this human-guided policy with a self-learning policy via a confidence-based intervention function, with the aim of continuing to improve performance while retaining human-like safety. The central theoretical claim is Theorem 1, which asserts a lower bound on the return of the combined behavior policy in terms of the human-guided policy's return, the variance bound σ_max, the confidence threshold δ, and the expected intervention rate β. The paper reports extensive simulation experiments in MetaDrive and a qualitative real-world UGV demonstration.

Significance. If the performance guarantee in Theorem 1 were valid, the paper would make a useful contribution: a principled way to interleave a human-learned policy with a reward-maximizing policy while bounding performance degradation. The empirical study is extensive for a robotics venue, covering multiple baselines, an ablation of the shared-control and confidence components, and a real-world deployment with code and videos offered. However, the theoretical guarantee is load-bearing for the advertised 'safety and performance guarantees' and the proof as written is not sound. Specifically, the key inequality (36) is applied to the wrong conditioning event, and the confidence evaluation uses a critic that does not estimate the Q-function of the human-guided policy as required by the proof. These are not local fixable gaps; they invalidate the central claim. The empirical results may still be of interest, but the paper's main advertised contribution is not established.

major comments (4)
  1. [Section II-C, Theorem 1 proof, Eq. (36)] Inequality (36) is asserted under the condition P(Q^g(s,a_r) > Q^g(s,a_g)) ≤ 1-δ, which corresponds to T_c(s)=1 by the intervention rule (30). However, in the proof, the term E_{a∼π_r}Q^g(s,a) − V^g(s) is multiplied by (1−T_c(s)) in Eq. (34), so it is only present when T_c(s)=0. The bound is therefore applied to the opposite conditioning event from the one stated. For the experimental value δ=0.15, the two events give inequalities of opposite sign, so the lower bound in Eq. (31) does not follow from the given derivation. This is a load-bearing error: without (36), the proof of Theorem 1 collapses.
  2. [Section II-C, Eqs. (28)–(30) vs. Eqs. (34)–(36)] The confidence evaluation in Eqs. (29)–(30) uses the distributions Z^c_ζ, which are updated in Eqs. (26)–(27) toward the self-learning policy's return distribution. The advantage decomposition in Eq. (34), however, requires Q^g, the true value function of the human-guided policy π_g. Evaluating Z^c_ζ at the action a_g does not produce Q^g(s,a_g), and no argument is given that the confidence probability P(Q^r_ζ > Q^g_ζ) bounds the true Q^g advantage. The notational shift from Q^g in Eq. (36) to the confidence critic's outputs is not justified, so the link between the switching rule and the performance bound is missing.
  3. [Section II-C, Eq. (34)] The proof replaces the state-dependent factor (1−T_c(s)) inside the expectation with the constant (1−β). The quantity β defined in Eq. (32) is a particular weighted average over the state distribution under π_b, and the proof does not establish that E[(1−T_c(s)) f(s)] = (1−β) E[f(s)] for f(s) = E_{a∼π_r} Q^g(s,a) − V^g(s). This step is also unsupported and is load-bearing for the final bound.
  4. [Section II-C, Assumption 1 and Eq. (36)] Inequality (36) relies on the Gaussian CDF Φ and a specific relationship between the probability P(Q^r > Q^g) and the difference of means. No Gaussianity of the return distributions is established; Assumption 1 only bounds variances by σ_max. Moreover, σ_max and β are never estimated from the policies in the experiments, so even if the proof were corrected, the bound in Eq. (31) could not be evaluated or certified for the proposed method.
minor comments (6)
  1. [Section III-C, Table I] The comparison is not matched on total interaction budget: C-HAC uses 1M total steps (0.05M demonstration plus 0.95M RL enhancement) while PVP and D-PVP are reported at 0.05M total steps. Please report all methods at matched total interactions or clearly justify why the budget difference does not affect the conclusions.
  2. [Section III-E] The real-world validation is qualitative only; please provide quantitative results such as success rate, intervention rate, and number of safety violations on Route 1 and the generalization Route 2.
  3. [Section II-B, Eq. (23)] The failure-probability bound is imported from reference [45] without derivation in this paper; please clarify whether the assumptions behind that bound (e.g., TD3-based PVP) carry over to the DSAC-based D-PVP setting presented here.
  4. [Section II-C, proof of Theorem 1] The text before Eq. (34) says 'the proof of Theorem 2' but the statement being proved is Theorem 1; this should be corrected.
  5. [Throughout] The simulator name is spelled 'Metadrive' in several places; the correct name is 'MetaDrive'. Figure 11 references panel '(g)' but the figure only shows panels (a)–(f). Several figure captions contain broken unicode escape sequences that should be repaired.
  6. [References] The reference list contains duplicates: reference [8] and reference [30] are the same source, and references [1] and [24] overlap; please de-duplicate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the C-HAC guarantee is a conditional proof attempt, not a fitted input renamed as a prediction; the key weakness is an unproven and misapplied inequality, which is a correctness gap rather than a definitional loop.

full rationale

Walking the claimed derivation chain: D-PVP trains the proxy value distribution Z^g_theta and the confidence critic Z^c_zeta by TD losses; T_c(s) is defined in (30) by comparing learned return-distribution means; Theorem 1 then attempts to bound J(pi_b) using Lemmas from [46] and [47], Assumption 1 on sigma_max, and inequality (36). None of these ingredients is a fitted parameter renamed as a prediction: beta is defined as a weighted intervention rate, sigma_max is an assumption, and the bound is conditional on those quantities. The best candidate for circularity is that T_c uses the learned Z^c distributions while the proof of Theorem 1 uses the true Q^g advantage. However, the theorem does not define Q^g in terms of T_c, and the conclusion J(pi_b) >= J(pi_g) - (1-beta)*sqrt(2)*sigma_max*Phi^{-1}(delta)/(1-gamma) does not reduce to equation (30) by construction. The proof instead rests on inequality (36), which is asserted without derivation and appears to invoke the wrong branch of the confidence condition: the term multiplied by (1 - T_c(s)) is active only when T_c(s)=0, whereas (36) is asserted from the T_c(s)=1 condition, and it conflates the learned Z^c_zeta with Q^g. That is an unproven and possibly misapplied lemma, i.e., a correctness risk, not a circular reduction. The paper is benchmarked against external baselines and relies on external sources for the PVP construction and the distribution-shape bound; no load-bearing self-citation chain is present. Therefore the appropriate circularity finding is no significant circularity, score 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a small set of hand-chosen hyperparameters (delta, eta, theta_c, kappa, N_g), several domain assumptions about human and return-distribution behavior, and external lemmas from [46] and [47]. No entirely new physical or algorithmic entity is introduced; the proxy value network is inherited from PVP.

free parameters (5)
  • delta (confidence threshold) = 0.15 (full) vs 0.5 (ablation)
    Chosen by hand; controls how often self-learning policy takes over and appears in Eq. (30) and Theorem 1.
  • eta (variance convergence rate) = not reported
    Appears in the proxy value gradient updates, Eqs. (14) and (15); no value or tuning procedure given.
  • theta_c (variance transition threshold) = not reported
    Used in transition condition (40) to decide when stage 1 ends.
  • kappa (log-probability transition threshold) = not reported
    Used in transition condition (40); distinct from the kappa safety constant in Eq. (23).
  • N_g (minimum training steps before transition) = not reported
    Used in transition condition (40).
assumptions (5)
  • domain assumption Return distributions are approximately Gaussian (used in Eq. (29) to compute a normal CDF confidence).
    Z^c and Z^g networks output Gaussian diagonal distributions; no justification that the true return distributions are Gaussian.
  • domain assumption Human expert provides unsafe actions with probability at most epsilon and fails to intervene on unsafe novice actions with probability at most kappa.
    Safety bound in Eq. (23) is credited to prior work [45] rather than derived.
  • ad hoc to paper Variances of return distributions are bounded by sigma_max (Assumption 1, Section II-C).
    Assumed for Theorem 1; no mechanism ensures this bound, and sigma_max is not estimated.
  • standard math Lemma 1 (state-distribution discrepancy bound from [46]) and Lemma 2 (advantage performance identity from [47]) hold as cited.
    Both lemmas are borrowed from external references and are standard tools for policy performance bounds.
  • domain assumption The reward-free TD target in Eq. (18) propagates proxy values without accumulating reward bias.
    Unstated; the reward-free Bellman operator is defined in Eq. (9), but its interaction with the PV loss is not analyzed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Confidence-Guided Human-AI Collaboration: Reinforcement Learning with Distributional Proxy Value Propagation for Autonomous Driving." pith.science (2026). https://pith.science/paper/3ERSEYNI

@misc{pith2026250603568,
  author       = {Pith},
  title        = {Pith review of: Confidence-Guided Human-AI Collaboration: Reinforcement Learning with Distributional Proxy Value Propagation for Autonomous Driving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3ERSEYNI}},
  note         = {Machine review of arXiv:2506.03568}
}
read the original abstract

Autonomous driving promises significant advancements in mobility, road safety and traffic efficiency, yet reinforcement learning and imitation learning face safe-exploration and distribution-shift challenges. Although human-AI collaboration alleviates these issues, it often relies heavily on extensive human intervention, which increases costs and reduces efficiency. This paper develops a confidence-guided human-AI collaboration (C-HAC) strategy to overcome these limitations. First, C-HAC employs a distributional proxy value propagation method within the distributional soft actor-critic (DSAC) framework. By leveraging return distributions to represent human intentions C-HAC achieves rapid and stable learning of human-guided policies with minimal human interaction. Subsequently, a shared control mechanism is activated to integrate the learned human-guided policy with a self-learning policy that maximizes cumulative rewards. This enables the agent to explore independently and continuously enhance its performance beyond human guidance. Finally, a policy confidence evaluation algorithm capitalizes on DSAC's return distribution networks to facilitate dynamic switching between human-guided and self-learning policies via a confidence-based intervention function. This ensures the agent can pursue optimal policies while maintaining safety and performance guarantees. Extensive experiments across diverse driving scenarios reveal that C-HAC significantly outperforms conventional methods in terms of safety, efficiency, and overall performance, achieving state-of-the-art results. The effectiveness of the proposed method is further validated through real-world road tests in complex traffic conditions. The videos and code are available at: https://github.com/lzqw/C-HAC.

Figures

Figures reproduced from arXiv: 2506.03568 by the authors.

Figure 1
Figure 1. Overall framework of C-HAC The soft Q value is given by Q π (st, at) = rt + γ E (si>t,ai>t)∼ρπ "X∞ i=t γ i−t [ri − α log π (ai | si)]# , (3) which delineates the expected soft return for choosing at at state st under policy π. The soft Q value can be updated using the soft Bellman operator T π T πQ π (s, a) = r + γ E s ′∼p,a′∼π [Q π (s ′ , a′ ) − α log π (a ′ | s ′ )] . (4) Meanwhile, the policy π is updated by maxi… view at source ↗
Figure 2
Figure 2. Illustration of Distributional Proxy Value Propagation. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Simulation environment and human interfaces. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Performance of different baselines. measuring the occurrence of dangerous situations during the training process. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Performance comparison of C-HAC with HAC methods. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 7
Figure 7. Figure 7: The action sequences generated by C-HAC and PVP agents in the [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 9
Figure 9. Figure 9: Routes for training and testing in real-world experiments [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 8
Figure 8. Figure 8: Comparative visualizations of D-PVP and C-HAC across two driving [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 10
Figure 10. Figure 10: Architecture of UGV setup with human intervention. [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Real-world driving performance on Route 1 and Route 2. [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Action outputs and speeds of the vehicle in real-world scenarios [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

55 extracted references · 37 canonical work pages

  1. [1]

    Learning to drive in a day,

    A. Kendall, J. Hawke, D. Janz, P. Mazur, D. Reda, J. M. Allen, V . D. Lam, A. Bewley, and A. Shah, “Learning to drive in a day,” in 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 8248–8254

    work page 2019

  2. [2]

    End to end learning for self-driving cars,

    M. Bojarski, D. W. del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba, “End to end learning for self-driving cars,” ArXiv, vol. abs/1604.07316, 2016

    arXiv 2016

  3. [3]

    Dense reinforcement learning for safety validation of autonomous vehicles,

    S. Feng, H. Sun, X. Yan, H. Zhu, Z. Zou, S. Shen, and H. X. Liu, “Dense reinforcement learning for safety validation of autonomous vehicles,” Nature, vol. 615, no. 7953, pp. 620–627, 2023

    2023

  4. [5]

    A survey of deep RL and IL for autonomous driving policy learning,

    Z. Zhu and H. Zhao, “A survey of deep RL and IL for autonomous driving policy learning,” IEEE Transactions on Intelligent Transporta- tion Systems, vol. 23, no. 9, pp. 4043–4065, 2022

    work page 2022

  5. [6]

    A survey on autonomous vehicle control in the era of mixed-autonomy: From physics-based to ai-guided driving policy learning,

    X. Di and R. Shi, “A survey on autonomous vehicle control in the era of mixed-autonomy: From physics-based to ai-guided driving policy learning,” Transportation Research Part C: Emerging Technologies, vol. 125, pp. 3008–3048, 2021

    work page 2021

  6. [7]

    Deep learning for safe autonomous driving: Current chal- lenges and future directions,

    K. Muhammad, A. Ullah, J. Lloret, J. D. Ser, and V . H. C. de Al- buquerque, “Deep learning for safe autonomous driving: Current chal- lenges and future directions,” IEEE Transactions on Intelligent Trans- portation Systems, vol. 22, no. 7, pp. 4316–4336, 2021

    2021

  7. [8]

    Survey of deep reinforcement learning for motion planning of autonomous vehicles,

    S. Aradi, “Survey of deep reinforcement learning for motion planning of autonomous vehicles,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 2, pp. 740–759, 2022

    2022

  8. [9]

    A general reinforcement learning algorithm that masters chess, shogi, and go through self-play,

    D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis, “A general reinforcement learning algorithm that masters chess, shogi, and go through self-play,” Science, vol. 362, no. 6419, pp. 1140–1144, 2018

    2018

Show all 55 references
  1. [10]

    Deep reinforcement learning for autonomous driving: A survey,

    B. R. Kiran, I. Sobh, V . Talpaert, P. Mannion, A. A. A. Sallab, S. Yo- gamani, and P. P ´erez, “Deep reinforcement learning for autonomous driving: A survey,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 6, pp. 4909–4926, 2022

    2022

  2. [11]

    End-to-end urban driving by imitating a reinforcement learning coach,

    Z. Zhang, A. Liniger, D. Dai, F. Yu, and L. Van Gool, “End-to-end urban driving by imitating a reinforcement learning coach,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , 2021, pp. 5202–5212

    2021

  3. [12]

    Reward misdesign for autonomous driving,

    W. B. Knox, A. Allievi, H. Banzhaf, F. Schmitt, and P. Stone, “Reward misdesign for autonomous driving,” Artif. Intell., vol. 316, no. 103829, Mar. 2023

    2023

  4. [13]

    Scalable agent alignment via reward modeling: A research direction. arxiv 2018,

    J. Leike, D. Krueger, T. Everitt, M. Martic, V . Maini, and S. Legg, “Scalable agent alignment via reward modeling: A research direction. arxiv 2018,” arXiv preprint arXiv:1811.07871 , 1811

    2018 arXiv

  5. [14]

    Demonstrating spec- ification gaming in reasoning models,

    A. Bondarenko, D. V olk, D. V olkov, and J. Ladish, “Demonstrating spec- ification gaming in reasoning models,” arXiv preprint arXiv:2502.13295, 2025

    2025 arXiv

  6. [15]

    Toward human-in-the-loop AI: Enhancing deep reinforcement learning via real-time human guidance for autonomous driving,

    J. Wu, Z. Huang, Z. Hu, and C. Lv, “Toward human-in-the-loop AI: Enhancing deep reinforcement learning via real-time human guidance for autonomous driving,” Engineering, vol. 21, pp. 75–91, 2023

    2023

  7. [16]

    Hindsight credit assignment,

    A. Harutyunyan, W. Dabney, T. Mesnard, M. Gheshlaghi Azar, B. Piot, N. Heess, H. P. van Hasselt, G. Wayne, S. Singh, D. Precup et al. , “Hindsight credit assignment,” vol. 32, 2019, pp. 167–175

    2019

  8. [17]

    Trial without error: Towards safe reinforcement learning via human intervention,

    W. Saunders, G. Sastry, A. Stuhlmueller, and O. Evans, “Trial without error: Towards safe reinforcement learning via human intervention,” arXiv preprint arXiv:1707.05173 , 2017

    2017 arXiv

  9. [18]

    A survey on imitation learning techniques for end-to-end autonomous vehicles,

    L. Le Mero, D. Yi, M. Dianati, and A. Mouzakitis, “A survey on imitation learning techniques for end-to-end autonomous vehicles,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 9, pp. 4128–4147, 2022

    2022

  10. [19]

    Conditional predictive behavior planning with inverse reinforcement learning for human-like autonomous driving,

    Z. Huang, H. Liu, J. Wu, and C. Lv, “Conditional predictive behavior planning with inverse reinforcement learning for human-like autonomous driving,” IEEE Transactions on Intelligent Transportation Systems , vol. 24, no. 7, pp. 7244–7258, 2023

    2023

  11. [20]

    Pattern recognition and adaptive control,

    B. Widrow, “Pattern recognition and adaptive control,” IEEE Transac- tions on Applications and Industry , vol. 83, no. 74, pp. 269–277, 1964

    1964

  12. [21]

    An algorithmic perspective on imitation learning,

    T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, and J. Peters, “An algorithmic perspective on imitation learning,” ArXiv, vol. abs/1811.06711, 2018

    2018 arXiv

  13. [22]

    Learning a decision module by imitating driver’s control behaviors,

    J. Huang, S. Xie, J. Sun, Q. Ma, C. Liu, D. Lin, and B. Zhou, “Learning a decision module by imitating driver’s control behaviors,” in Proceedings of the 2020 Conference on Robot Learning , J. Kober, F. Ramos, and C. Tomlin, Eds., 2021, pp. 1–10

    2020

  14. [23]

    Conservative safety critics for exploration,

    H. Bharadhwaj, A. Kumar, N. Rhinehart, S. Levine, F. Shkurti, and A. Garg, “Conservative safety critics for exploration,” ArXiv, vol. abs/2010.14497, 2020

    2010 arXiv

  15. [24]

    Behavior regularized offline reinforcement learning,

    Y . Wu, G. Tucker, and O. Nachum, “Behavior regularized offline reinforcement learning,” ArXiv, vol. abs/1911.11361, 2019

    1911 arXiv

  16. [25]

    Off-policy deep reinforcement learning without exploration,

    S. Fujimoto, D. Meger, and D. Precup, “Off-policy deep reinforcement learning without exploration,” in Proceedings of the 36th International Conference on Machine Learning , vol. 97, 2019, pp. 2052–2062

    2019

  17. [26]

    Adversarial inverse rein- forcement learning with self-attention dynamics model,

    J. Sun, L. Yu, P. Dong, B. Lu, and B. Zhou, “Adversarial inverse rein- forcement learning with self-attention dynamics model,” IEEE Robotics and Automation Letters , vol. 6, no. 2, pp. 1880–1886, 2021

    2021

  18. [27]

    Efficient reductions for imitation learning,

    S. Ross and D. Bagnell, “Efficient reductions for imitation learning,” in Proceedings of the thirteenth international conference on artificial intel- ligence and statistics . JMLR Workshop and Conference Proceedings, 2010, pp. 661–668

    2010

  19. [28]

    Exploring the limi- tations of behavior cloning for autonomous driving,

    F. Codevilla, E. Santana, A. Lopez, and A. Gaidon, “Exploring the limi- tations of behavior cloning for autonomous driving,” in 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , 2019, pp. 9328– 9337

    2019

  20. [29]

    Behavioral cloning a correction,

    R. Camacho and D. Michie, “Behavioral cloning a correction,” AI Mag., vol. 16, pp. 92–101, 1995

    1995

  21. [30]

    A general reinforcement learning algorithm that masters chess, shogi, and go through self-play,

    D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. P. Lillicrap, K. Si- monyan, and D. Hassabis, “A general reinforcement learning algorithm that masters chess, shogi, and go through self-play,” Science, vol...

    2018

  22. [31]

    A reduction of imitation learning and structured prediction to no-regret online learning,

    S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in International Conference on Artificial Intelligence and Statistics , 2011, pp. 627–635

    2011

  23. [32]

    Query-efficient imitation learning for end-to-end autonomous driving,

    J. Zhang and K. Cho, “Query-efficient imitation learning for end-to-end autonomous driving,” ArXiv, vol. abs/1605.06450, 2016

    2016 arXiv

  24. [33]

    Hg-dagger: Interactive imitation learning with human experts,

    M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer, “Hg-dagger: Interactive imitation learning with human experts,” in 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 8077–8083

    2019

  25. [34]

    Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning,

    R. Hoque, A. Balakrishna, E. Novoseller, A. Wilcox, D. S. Brown, and K. Goldberg, “Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning,” in Proceedings of the 5th Conference on Robot Learning, vol. 164, 2022, pp. 598–608

    2022

  26. [35]

    Expert intervention learning,

    J. Spencer, S. Choudhury, M. Barnes, M. Schmittle, M. Chiang, P. J. Ra- madge, and S. S. Srinivasa, “Expert intervention learning,” Autonomous Robots, vol. 46, pp. 99–113, 2021

    2021

  27. [36]

    Human-in-the-loop imitation learning using remote tele- operation,

    A. Mandlekar, D. Xu, R. Mart’in-Mart’in, Y . Zhu, F. F. Li, and S. Savarese, “Human-in-the-loop imitation learning using remote tele- operation,” ArXiv, vol. abs/2012.06733, 2020

    2012 arXiv

  28. [37]

    Deep reinforcement learning from human preferences,

    P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” in Advances in Neural Information Processing Systems , 2017, pp. 4299–4307

    2017

  29. [38]

    Batch active preference-based learning of reward functions,

    E. Biyik and D. Sadigh, “Batch active preference-based learning of reward functions,” in Proceedings of The 2nd Conference on Robot Learning, vol. 87, 2018, pp. 519–528

    2018

  30. [39]

    Learning reward functions by integrating human demonstrations and preferences,

    M. Palan, N. C. Landolfi, G. Shevchuk, and D. Sadigh, “Learning reward functions by integrating human demonstrations and preferences,” ArXiv, vol. abs/1906.08928, 2019

    1906 arXiv

  31. [40]

    Efficient learning of safe driving policy via human-AI copilot optimization,

    Q. Li, Z. Peng, and B. Zhou, “Efficient learning of safe driving policy via human-AI copilot optimization,” in International Conference on Learning Representations, 2022, pp. 1–19

    2022

  32. [41]

    Learning from active human involvement through proxy value propagation,

    Z. Peng, W. Mo, C. Duan, Q. Li, and B. Zhou, “Learning from active human involvement through proxy value propagation,” in Advances in Neural Information Processing Systems , 2023, pp. 7969–7992

    2023

  33. [42]

    Meta-world: A benchmark and evaluation for multi- task and meta reinforcement learning,

    T. Yu, Z. He, D. Quillen, R. Julian, K. Hausman, C. Finn, and S. Levine, “Meta-world: A benchmark and evaluation for multi- task and meta reinforcement learning,” 2019. [Online]. Available: https://arxiv.org/abs/1910.10897

    2019 arXiv

  34. [43]

    Socially situated artificial intelligence enables learning from human interaction,

    R. Krishna, D. Lee, L. Fei-Fei, and M. S. Bernstein, “Socially situated artificial intelligence enables learning from human interaction,” vol. 119, no. 39, 2022, pp. 1157–1169. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 13

    2022

  35. [44]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,” in International Conference on Machine Learning , 2018, pp. 1861–1870

    2018

  36. [45]

    Human as AI mentor: Enhanced human-in-the-loop reinforcement learning for safe and effi- cient autonomous driving,

    Z. Huang, Z. Sheng, C. Ma, and S. Chen, “Human as AI mentor: Enhanced human-in-the-loop reinforcement learning for safe and effi- cient autonomous driving,”Communications in Transportation Research, vol. 4, pp. 100–127, 2024

    2024

  37. [46]

    Guarded policy optimization with imperfect online demonstrations,

    Z. Xue, Z. Peng, Q. Li, Z. Liu, and B. Zhou, “Guarded policy optimization with imperfect online demonstrations,” 2023

    2023

  38. [47]

    Trust region policy optimization,

    J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel, “Trust region policy optimization,” 2017

    2017

  39. [48]

    Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning,

    Q. Li, Z. Peng, L. Feng, Q. Zhang, Z. Xue, and B. Zhou, “Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning,” IEEE Transactions on Pattern Analysis and Machine Intel- ligence, vol. 45, no. 3, pp. 3461–3475, 2023

    2023

  40. [49]

    Responsive safety in reinforce- ment learning by pid lagrangian methods,

    A. Stooke, J. Achiam, and P. Abbeel, “Responsive safety in reinforce- ment learning by pid lagrangian methods,” in International Conference on Machine Learning , 2020, pp. 9133–9143

    2020

  41. [50]

    Learning to walk in the real world with minimal human effort,

    S. Ha, P. Xu, Z. Tan, S. Levine, and J. Tan, “Learning to walk in the real world with minimal human effort,” arXiv preprint arXiv:2002.08550 , 2020

    2002 arXiv

  42. [51]

    Conservative Q- learning for offline reinforcement learning,

    A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conservative Q- learning for offline reinforcement learning,” in Proceedings of the 34th International Conference on Neural Information Processing Systems , 2020, pp. 1179–1191

    2020

  43. [52]

    Prox- imal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  44. [53]

    Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors,

    J. Duan, Y . Guan, S. E. Li, Y . Ren, Q. Sun, and B. Cheng, “Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 11, pp. 6584–6598, 2022

    2022

  45. [54]

    A framework for behavioural cloning,

    M. Bain and C. Sammut, “A framework for behavioural cloning,” in Machine Intelligence, 1999, pp. 103–129

    1999

  46. [55]

    Generative adversarial imitation learning,

    J. Ho and S. Ermon, “Generative adversarial imitation learning,” in Ad- vances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, Eds., vol. 29, 2016, pp. 1–9. Zeqiao Li received the B.S. degree in intelligent science and techn...

    2016

  47. [2022]

    His research interests include the perception, decisionmaking,path planning, and motion control of intelligent and connected vehicles (ICVs), autonomous driving, eco-driving, human–machine trust an cooperation, shared control, and machine-learning applications in ICVs

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.