Pith. sign in

REVIEW 3 major objections 6 minor 55 references

Surrounding traffic controlled by instruction-following language models creates interactive long-tail tests that current planners still fail.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 20:00 UTC pith:LQQQJ252

load-bearing objection Useful LLM-agent + SemanticPlan long-tail suite on nuPlan; main multi-planner safety table is only partially interactive because agents are frozen overlays. the 3 major comments →

arxiv 2607.04331 v1 pith:LQQQJ252 submitted 2026-07-05 cs.RO cs.CV

Agent-driven Long-tail Simulation for Autonomous Driving

Junru Gu , Lijin Yang , Jianing Huang , Shu Liu , Zhongzhan Huang , Hang Zhao This is my paper
classification cs.RO cs.CV
keywords autonomous drivingclosed-loop simulationLLM agentslong-tail scenariosSemanticPlanmotion planninginteractive traffic
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Closed-loop evaluation of self-driving systems needs other road users that react with intent, not only log replay or simple rules. This paper argues that large language models, given role instructions and a constrained action interface, can control pedestrians and vehicles so their behavior stays intentional, reactive, and physically plausible. On top of real driving logs the authors build SemanticPlan: more than 230 long-tail scenarios in which multiple agents follow diverse language instructions. When strong planners are run zero-shot on these scenes, even the best still leave large safety and task-completion gaps. A sympathetic reader cares because the work claims a practical way to stress-test planning where ordinary benchmarks look solved.

Core claim

Instruction-following agents that output high-level structured actions—executed by the simulator with route planning and flow-matching motion—produce intentional, reactive surrounding traffic in real-map closed-loop simulation. The SemanticPlan scenarios built this way remain hard: state-of-the-art planners do not consistently finish safely or respect semantic constraints such as honking appropriateness and penalty regions.

What carries the argument

Agent-driven simulation: controlled road users are queried with role instructions, simulator feedback, local text-and-view observations, and chat history; they return constrained actions (WASD-style motion plus pick-up/enter for humans; lane-change/park/honk-style maneuvers for vehicles) that the simulator validates and executes with physical constraints.

Load-bearing premise

On the main collision track, agent motions are pre-generated once and replayed against every planner, so surrounding agents cannot replan against each planner’s own path.

What would settle it

Re-run the full collision-prone track with live agent re-querying against each planner’s online ego trajectory and check whether safety and overall scores drop relative to the reported pre-generated-overlay results (best overall about 0.65).

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes an agent-driven closed-loop simulation framework in which selected surrounding road participants are controlled by instruction-following VLMs/LLMs through a constrained structured action interface (human WASD+high-level actions and soft signals; vehicle high-level maneuvers executed by a route planner), with short-horizon human motion realized by a conditional flow-matching trajectory generator. Building on nuPlan, the authors introduce SemanticPlan: >50 long-tail scenario types and >230 scenarios that augment real logs with multi-agent language-instructed behaviors, split into a collision-prone track and a semantic track (region/traffic-police constraints and honk-required/honk-penalized cases). Zero-shot evaluation reports that popular planners achieve limited safety/progress on the collision-prone track (best overall 0.651, Table 2) and that IDM-family LLM-augmented planners remain weak on the semantic track (overall ≤0.389, Table 3), with ablations supporting flow matching, high-level vehicle actions, and honking trade-offs.

Significance. If the framework and benchmark hold up under fully interactive evaluation, this is a useful contribution to closed-loop AD evaluation: it targets intentional, instruction-conditioned long-tail interaction beyond log replay and IDM, and SemanticPlan adds semantic decision axes (penalty regions, traffic-police compliance, role-weighted honking) that standard progress/collision metrics miss. Strengths include a carefully constrained agent interface rather than free-form trajectory dumping, explicit simulator feedback for closed-loop correction, quantitative simulation-quality checks (Table 4), human-motion statistics closer to nuPlan logs with flow matching (Table 5), and a clear vehicle-interface ablation (Table 6). The work is primarily empirical systems/benchmark engineering; its lasting value depends on whether the multi-planner safety conclusions truly reflect reactive multi-agent interaction rather than fixed overlays.

major comments (3)
  1. §4.2 and Table 1 state that the collision-prone track uses pre-generated agent trajectories (K=3 rollouts) saved as overlays and replayed for every planner, so “planner evaluation no longer performs online agent inference.” Under this protocol, agents cannot replan against each planner’s distinct ego motion. Table 2 is the main multi-planner evidence for the Abstract/§4.3 claim that SOTA planners “still struggle” in interactive long-tail closed loop. Frozen overlays mix non-reactive stress with true interaction; comparative safety/progress gaps may understate (or misattribute) interactive difficulty. Please either (i) re-run at least a subset of Table 2 with online agents for all planners, or (ii) substantially reframe claims for that track as partially non-reactive / open-loop agent motion, and quantify the gap between overlay and online protocols on a shared planner set.
  2. The broad “state-of-the-art planners” claim is uneven across tracks. Learning-based and hybrid planners (UrbanDriver, PlanTF, Diffusion Planner, PLUTO, PDM Hybrid, etc.) appear only on the collision-prone track (Table 2), while the fully interactive semantic track evaluates only IDM, IDM+LLM, and stop-prioritized IDM+LLM (Table 3). Semantic decisions (honking, region avoidance, police compliance) are central to SemanticPlan’s novelty, yet the strongest planners are not tested there. Either extend Table 3 to at least one strong hybrid/learning planner under real-time agents, or narrow the abstract/conclusion language so that interactive semantic difficulty is not attributed to the full SOTA set.
  3. §4.2 notes that collision-prone rollouts are generated once (with agent interaction in the generation context) then frozen. The manuscript does not specify the ego policy used during generation of those overlays, nor whether generation used IDM (as in Table 1’s runtime measurement) versus a stronger planner. If overlays were produced against a weak ego, measured collisions for strong planners may partly reflect agent trajectories optimized against a different partner. Please document the generation ego policy, and report sensitivity of Table 2 scores to the ego used when producing the K=3 overlays.
minor comments (6)
  1. Figure 2 caption and §3.2–3.3 are clear, but the main text never fully specifies how multi-agent soft signals and honking are ordered within a single 0.1 s step when multiple agents query every 2 s; a short timing diagram or pseudocode in the appendix would help reproducibility.
  2. Table 5 reports acceleration/jerk closer to nuPlan logs with flow matching, but “Turn Direction deg” and the Straight/L/R fractions are not defined in the main text; add definitions and sample sizes.
  3. Appendix B.3 metrics (Scoll, Sregion, Shonk) are well specified, yet Table 2/3 column names (Prog., Safe., Gen. Sem. Penalty, etc.) are not explicitly mapped to those formulas in the main body; a one-line mapping would reduce ambiguity.
  4. Related Work cites HumanSim and CitySim; a short qualitative comparison of action interface design (structured maneuvers vs freer language control) would better position the contribution without claiming novelty of “LLM agents” alone.
  5. Typo/consistency: human output schema uses “brief_visable_action” (Appendix B.4); fix spelling. Also arXiv id in the prompt (2607.04331) vs typical 2025/2026 dating is fine for review but ensure camera-ready metadata matches.
  6. §4.1 says “over 230 scenarios” and “3 to 5” base scenes per type, while Appendix B.1 says “3 to 8”; reconcile the construction numbers.

Circularity Check

0 steps flagged

No circular derivation: empirical systems/benchmark paper with independently defined metrics and zero-shot evaluation.

full rationale

This paper proposes an agent-driven simulation framework and the SemanticPlan benchmark, then reports closed-loop planner scores. There is no claimed first-principles derivation whose conclusion is algebraically forced by its inputs. Collision-prone scores are defined from route progress, ego-at-fault collisions, and drivable-area compliance against simulator logs (Appendix B.3); semantic scores use truncated progress, region-overlap penalties, and role-weighted honk penalties—none of which are fitted free parameters renamed as predictions. Flow-matching trajectory generation is trained on held-out nuPlan pedestrian trajectories with separate ADE/FDE and direction-error metrics (Appendix A.1, Table 8), not used to tautologically force planner rankings. Planners are evaluated zero-shot on unaugmented nuPlan training data without fine-tuning on SemanticPlan. Self-citations, if any, are not load-bearing uniqueness theorems that forbid alternatives. The skeptic concern about pre-generated agent overlays on the collision-prone track is a methodological validity issue (partial non-reactivity), not circularity: measured scores still depend on planner behavior against fixed trajectories rather than reducing by construction to the agent-generation inputs. No self-definitional loop, fitted-input-as-prediction, or ansatz-smuggled uniqueness chain is present.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

Load-bearing content is mostly engineering assumptions and hand-chosen evaluation knobs, not mathematical axioms. The central empirical claim rests on: (i) LLM agents plus structured executors being an adequate proxy for intentional human road behavior; (ii) manual long-tail scenario design on nuPlan validation logs; (iii) frozen agent overlays for multi-planner collision evaluation; (iv) metric weights and tolerances for semantic/honk scoring. Free parameters are evaluation and simulation hyperparameters rather than fitted physical constants. Invented entities are software constructs (SemanticPlan, structured action interface, soft signals), not new physical objects.

free parameters (5)
  • Agent query period / planning horizon = Δt=0.1 s; query every 20 steps; H=2 s
    Agents queried every 2 s with H=2 s, L=20 waypoints at 0.1 s; chosen for practicality and not derived from data.
  • LLM sampling temperature and model choice = temp=0.7; K=3; Qwen3.6-27B
    Default Qwen3.6-27B FP8, temperature 0.7, K=3 rollouts; ablations show model size matters but values are hand-selected.
  • Honk penalty decay γ and role weights w_j = γ=0.7; w_j in [0,1] by role
    Honk-sensitive score uses γ=0.7 and role-dependent weights; these set the progress–appropriateness trade-off by design.
  • Region penalty tolerance τ = scenario-specific
    Scenario-specific clip tolerance for ego–penalty-region overlap; larger τ forgives brief intrusion.
  • Flow-matching trajectory generator training set size/balancing = ~1.4M trajectories; 2.0 s history/horizon
    ~1.4M balanced pedestrian trajectories from nuPlan validation; conditioning and horizon choices define the motion prior used as 'physical plausibility'.
axioms (5)
  • domain assumption Structured high-level actions executed by deterministic simulator executors preserve physical plausibility better than free-form LLM trajectories or low-level controls.
    Core design claim in §3.3–3.5 and Table 6; supported empirically for vehicles but assumed for the overall framework.
  • domain assumption Language role instructions plus local text/image observations and feedback suffice for intentional, reactive multi-agent road behavior in long-tail scenes.
    Stated throughout §3.2 and dataset construction; simulation-quality success is only measured on 10 predicate-checkable types (Table 4).
  • domain assumption Zero-shot evaluation on SemanticPlan (planners trained only on standard nuPlan) measures generalization relevant to real long-tail deployment risk.
    §4.2 experimental goal; assumes the hand-built augmentations are representative stress cases rather than adversarial artifacts.
  • ad hoc to paper Pre-generated stochastic agent rollouts can be reused across planners for the collision-prone track without invalidating comparative safety conclusions.
    Explicit protocol in §4.2/Table 1 to control cost; load-bearing for Table 2 comparisons.
  • domain assumption nuPlan map, dynamics, and observation stack plus IDM route execution are an adequate substrate for closed-loop planning scores.
    Framework extends nuPlan throughout §3 and Appendix A; standard in the subfield but still an external modeling premise.
invented entities (3)
  • SemanticPlan benchmark no independent evidence
    purpose: Provide closed-loop long-tail and semantic planning scenarios by placing instruction-following agents into real nuPlan scenes.
    New dataset/suite introduced by the paper; independent evidence is limited to internal metrics and qualitative examples, not external multi-lab adoption yet.
  • Structured agent action interface (human WASD+actions; vehicle high-level maneuvers; soft signals; honk channel) no independent evidence
    purpose: Constrain LLM outputs so the simulator can enforce feasibility, map compliance, and feedback-driven correction.
    Software interface defined in §3.3 and Appendix B.4; validated partly by Tables 4–6 but not an externally measured natural phenomenon.
  • Soft agent-agent interaction / stimulus events no independent evidence
    purpose: Propagate gestures, utterances, and honks as non-physical stimuli that condition other agents without direct state overwrite.
    Introduced in §3.3 and Appendix A; behavior depends on prompt compliance rather than measured human signaling data.

pith-pipeline@v1.1.0-grok45 · 20055 in / 4153 out tokens · 39383 ms · 2026-07-11T20:00:45.030521+00:00 · methodology

0 comments
read the original abstract

Evaluating autonomous driving systems in closed-loop settings requires realistic and interactive simulation, yet existing simulators largely rely on log replay or rule-based agents, limiting behavioral diversity and long-tail coverage. We propose an agent-driven simulation framework in which surrounding road participants are controlled by instruction-following large language models through a structured action interface, enabling intentional and reactive behaviors while preserving physical plausibility. Furthermore, we introduce SemanticPlan, a benchmark of closed-loop planning in long-tail and semantically rich scenarios that augment real nuPlan scenes with multiple interactive agents following diverse language instructions. Evaluation results show that state-of-the-art planners still struggle to consistently achieve safe and effective task completion, suggesting that these long-tail scenarios remain challenging.

Figures

Figures reproduced from arXiv: 2607.04331 by Hang Zhao, Jianing Huang, Junru Gu, Lijin Yang, Shu Liu, Zhongzhan Huang.

Figure 1
Figure 1. Figure 1: Overview of long-tail agents and scenario types in SemanticPlan. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our agent-driven simulation framework. At each query step, the simulator [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 5
Figure 5. Figure 5: Representative scenarios in the seman￾tic track. duces a nonzero penalty in honk-penalized scenarios; the resulting Shonk captures this progress– appropriateness trade-off. 4.7 Qualitative Examples Collision-prone scenarios. We present qualitative visualizations of collision-prone scenarios using IDM as the planner to illustrate the safety challenges faced by ego planners in closed-loop evaluation, as show… view at source ↗
Figure 6
Figure 6. Figure 6: Dataset construction pipeline. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 12 linked inside Pith

  1. [1]

    Y . Li, W. Yuan, S. Zhang, W. Yan, Q. Shen, C. Wang, and M. Yang. Choose your simulator wisely: A review on open-source simulators for autonomous driving.IEEE Transactions on Intelligent Vehicles, 9(5):4861–4876, 2024

    2024

  2. [2]

    Caesar, J

    H. Caesar, J. Kabzan, K. S. Tan, W. K. Fong, E. Wolff, A. Lang, L. Fletcher, O. Beijbom, and S. Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles.arXiv preprint arXiv:2106.11810, 2021

    Pith/arXiv arXiv 2021

  3. [3]

    Karnchanachari, D

    N. Karnchanachari, D. Geromichalos, K. S. Tan, N. Li, C. Eriksen, S. Yaghoubi, N. Mehdipour, G. Bernasconi, W. K. Fong, Y . Guo, et al. Towards learning-based planning: The nuplan benchmark for real-world autonomous driving. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 629–636. IEEE, 2024

    2024

  4. [4]

    Treiber, A

    M. Treiber, A. Hennecke, and D. Helbing. Congested traffic states in empirical observations and microscopic simulations.Physical review E, 62(2):1805, 2000

    2000

  5. [5]

    Dosovitskiy, G

    A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V . Koltun. Carla: An open urban driving simulator. InConference on robot learning, pages 1–16. PMLR, 2017

    2017

  6. [6]

    Hallgarten, J

    M. Hallgarten, J. Zapata, M. Stoll, K. Renz, and A. Zell. Can vehicle motion planning generalize to realistic long-tail scenarios? In2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5388–5395. IEEE, 2024

    2024

  7. [7]

    Gulino, J

    C. Gulino, J. Fu, W. Luo, G. Tucker, E. Bronstein, Y . Lu, J. Harb, X. Pan, Y . Wang, X. Chen, et al. Waymax: An accelerated, data-driven simulator for large-scale autonomous driving research.Advances in Neural Information Processing Systems, 36:7730–7742, 2023

    2023

  8. [8]

    Q. Li, Z. M. Peng, L. Feng, Z. Liu, C. Duan, W. Mo, and B. Zhou. Scenarionet: Open- source platform for large-scale traffic scenario simulation and modeling.Advances in neural information processing systems, 36:3894–3920, 2023

    2023

  9. [9]

    M. Peng, R. Yao, X. Guo, and J. Ma. nuplan-r: A closed-loop planning benchmark for autonomous driving via reactive multi-agent simulation.arXiv preprint arXiv:2511.10403, 2025

    arXiv 2025

  10. [10]

    Dauner, M

    D. Dauner, M. Hallgarten, T. Li, X. Weng, Z. Huang, Z. Yang, H. Li, I. Gilitschenski, B. Ivanovic, M. Pavone, et al. Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking.Advances in Neural Information Processing Systems, 37:28706–28719, 2024

    2024

  11. [11]

    W. Cao, M. Hallgarten, T. Li, D. Dauner, X. Gu, C. Wang, Y . Miron, M. Aiello, H. Li, I. Gilitschenski, et al. Pseudo-simulation for autonomous driving.arXiv preprint arXiv:2506.04218, 2025

    arXiv 2025

  12. [12]

    X. Jia, Z. Yang, Q. Li, Z. Zhang, and J. Yan. Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving.Advances in Neural Information Processing Systems, 37:819–844, 2024

    2024

  13. [13]

    J. You, X. Jia, Z. Zhang, Y . Zhu, and J. Yan. Bench2drive-r: Turning real world data into reactive closed-loop autonomous driving benchmark by generative model.arXiv preprint arXiv:2412.09647, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Montali, J

    N. Montali, J. Lambert, P. Mougin, A. Kuefler, N. Rhinehart, M. Li, C. Gulino, T. Emrich, Z. Yang, S. Whiteson, et al. The waymo open sim agents challenge.Advances in Neural Information Processing Systems, 36:59151–59171, 2023

    2023

  15. [15]

    Y . Li, C. Fan, S. Z. Zhao, C. Li, C. Xu, H. Yao, M. Tomizuka, B. Zhou, C. Tang, M. Ding, et al. Womd-reasoning: A large-scale dataset for interaction reasoning in driving. InForty-second International Conference on Machine Learning. 10

  16. [16]

    R. Xu, H. Lin, W. Jeon, H. Feng, Y . Zou, L. Sun, J. Gorman, E. Tolstaya, S. Tang, B. White, et al. Wod-e2e: Waymo open dataset for end-to-end driving in challenging long-tail scenarios. arXiv preprint arXiv:2510.26125, 2025

    arXiv 2025

  17. [17]

    H. Yu, W. Yang, H. Ruan, Z. Yang, Y . Tang, X. Gao, X. Hao, Y . Shi, Y . Pan, N. Sun, et al. V2x-seq: A large-scale sequential dataset for vehicle-infrastructure cooperative perception and forecasting. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5486–5495, 2023

    2023

  18. [18]

    X. Yang, L. Wen, T. Wei, Y . Ma, J. Mei, X. Li, W. Lei, D. Fu, P. Cai, M. Dou, et al. Drivearena: A closed-loop generative simulation platform for autonomous driving. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 26933–26943, 2025

    2025

  19. [19]

    H. Yu, W. Yang, R. Hao, C. Wang, J. Zhong, P. Luo, and Z. Nie. Drivee2e: Closed-loop benchmark for end-to-end autonomous driving through real-to-simulation.arXiv preprint arXiv:2509.23922, 2025

    arXiv 2025

  20. [20]

    Y . Wang, K. Cheng, J. He, Q. Wang, H. Dai, Y . Chen, F. Xia, and Z.-X. Zhang. Drivingdojo dataset: Advancing interactive and knowledge-enriched driving world model.Advances in Neural Information Processing Systems, 37:13020–13034, 2024

    2024

  21. [21]

    W. Wu, X. Feng, Z. Gao, and Y . Kan. Smart: Scalable multi-agent real-time motion generation via next-token prediction.Advances in Neural Information Processing Systems, 37:114048– 114071, 2024

    2024

  22. [22]

    Zhang, X

    Z. Zhang, X. Jia, G. Chen, Q. Li, and J. Yan. Trajtok: Technical report for 2025 waymo open sim agents challenge.arXiv preprint arXiv:2506.21618, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Z. Peng, Y . Liu, and B. Zhou. Infgen: Scenario generation as next token group prediction.arXiv preprint arXiv:2506.23316, 2025

    arXiv 2025

  24. [24]

    L. Rowe, R. Girgis, A. Gosselin, L. Paull, C. Pal, and F. Heide. Scenario dreamer: Vectorized latent diffusion for generating driving simulation environments. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 17207–17218, 2025

    2025

  25. [25]

    S. Tan, J. Lambert, H. Jeon, S. Kulshrestha, Y . Bai, J. Luo, D. Anguelov, M. Tan, and C. M. Jiang. Scenediffuser++: City-scale traffic simulation via a generative world model. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 1570–1580, 2025

    2025

  26. [26]

    Ransiek, P

    J. Ransiek, P. Reis, T. Sch ¨urmann, and E. Sax. Adversarial and reactive traffic entities for behavior-realistic driving simulation: A review.arXiv preprint arXiv:2409.14196, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    L. Rowe, R. Girgis, A. Gosselin, B. Carrez, F. Golemo, F. Heide, L. Paull, and C. Pal. Ctrl-sim: Reactive and controllable driving agents with offline reinforcement learning.arXiv preprint arXiv:2403.19918, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    S. Tan, B. Ivanovic, Y . Chen, B. Li, X. Weng, Y . Cao, P. Kr¨ahenb¨uhl, and M. Pavone. Promptable closed-loop traffic simulation.arXiv preprint arXiv:2409.05863, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    L. Zhou, M. Jiang, and D. Wang. Humansim: Human-like multi-agent novel driving simulation for corner case generation. InEuropean Conference on Computer Vision, pages 287–304. Springer, 2024

    2024

  30. [30]

    Bougie and N

    N. Bougie and N. Watanabe. Citysim: Modeling urban behaviors and city dynamics with large-scale llm-driven agent simulation. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 215–229, 2025

    2025

  31. [31]

    M. Li, W. Ding, H. Lin, Y . Lyu, Y . Yao, Y . Zhang, and D. Zhao. Crashagent: Crash scenario generation via multi-modal reasoning.arXiv preprint arXiv:2505.18341, 2025. 11

    Pith/arXiv arXiv 2025

  32. [32]

    K. Xie, I. Yang, J. Gunerli, and M. Riedl. Making large language models into world models with precondition and effect knowledge. InProceedings of the 31st International Conference on Computational Linguistics, pages 7532–7545, 2025

    2025

  33. [33]

    J. Xie, K. Zhang, J. Chen, T. Zhu, R. Lou, Y . Tian, Y . Xiao, and Y . Su. Travelplanner: A benchmark for real-world planning with language agents. InForty-first International Conference on Machine Learning

  34. [34]

    Dauner, M

    D. Dauner, M. Hallgarten, A. Geiger, and K. Chitta. Parting with misconceptions about learning- based vehicle motion planning. InConference on Robot Learning, pages 1268–1281. PMLR, 2023

    2023

  35. [35]

    Scheel, L

    O. Scheel, L. Bergamini, M. Wolczyk, B. Osi´nski, and P. Ondruska. Urban driver: Learning to drive from real-world demonstrations using policy gradients. InConference on Robot Learning, pages 718–728. PMLR, 2022

    2022

  36. [36]

    Cheng, Y

    J. Cheng, Y . Chen, X. Mei, B. Yang, B. Li, and M. Liu. Rethinking imitation-based planners for autonomous driving. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 14123–14130. IEEE, 2024

    2024

  37. [37]

    Cheng, Y

    J. Cheng, Y . Chen, and Q. Chen. Pluto: Pushing the limit of imitation learning-based planning for autonomous driving.arXiv preprint arXiv:2404.14327, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    K. Renz, K. Chitta, O.-B. Mercea, A. Koepke, Z. Akata, and A. Geiger. Plant: Explainable planning transformers via object-level representations.arXiv preprint arXiv:2210.14222, 2022

    Pith/arXiv arXiv 2022

  39. [39]

    Hallgarten, M

    M. Hallgarten, M. Stoll, and A. Zell. From prediction to planning with goal conditioned lane graph traversals. In2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), pages 951–958. IEEE, 2023

    2023

  40. [40]

    Huang, H

    Z. Huang, H. Liu, and C. Lv. Gameformer: Game-theoretic modeling and learning of transformer-based interactive prediction and planning for autonomous driving. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 3903–3913, 2023

    2023

  41. [41]

    H. Liu, L. Chen, Y . Qiao, C. Lv, and H. Li. Reasoning multi-agent behavioral topology for interactive autonomous driving.Advances in Neural Information Processing Systems, 37: 92605–92637, 2024

    2024

  42. [42]

    Y . Chen, S. Veer, P. Karkus, and M. Pavone. Interactive joint planning for autonomous vehicles. IEEE Robotics and Automation Letters, 9(2):987–994, 2023

    2023

  43. [43]

    Zhang, Z

    L. Zhang, Z. Peng, Q. Li, and B. Zhou. Cat: Closed-loop adversarial training for safe end-to-end driving. InConference on Robot Learning, pages 2357–2372. PMLR, 2023

    2023

  44. [44]

    Y . Lu, J. Fu, G. Tucker, X. Pan, E. Bronstein, R. Roelofs, B. Sapp, B. White, A. Faust, S. Whiteson, et al. Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios. In2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7553–7560. IEEE, 2023

    2023

  45. [45]

    Zheng, R

    Y . Zheng, R. Liang, K. ZHENG, J. Zheng, L. Mao, J. Li, W. Gu, R. Ai, S. E. Li, X. Zhan, et al. Diffusion-based planning for autonomous driving with flexible guidance. InThe Thirteenth International Conference on Learning Representations

  46. [46]

    Y . Hu, S. Chai, Z. Yang, J. Qian, K. Li, W. Shao, H. Zhang, W. Xu, and Q. Liu. Solving motion planning tasks with a scalable generative model. InEuropean Conference on Computer Vision, pages 386–404. Springer, 2024

    2024

  47. [47]

    Zheng, Z

    Y . Zheng, Z. Xing, Q. Zhang, B. Jin, P. Li, Y . Zheng, Z. Xia, K. Zhan, X. Lang, Y . Chen, et al. Planagent: A multi-modal large language agent for closed-loop vehicle motion planning.arXiv preprint arXiv:2406.01587, 2024. 12

    Pith/arXiv arXiv 2024

  48. [48]

    Chen, Z.-h

    Y . Chen, Z.-h. Ding, Z. Wang, Y . Wang, L. Zhang, and S. Liu. Asynchronous large language model enhanced planner for autonomous driving. InEuropean Conference on Computer Vision, pages 22–38. Springer, 2024

    2024

  49. [49]

    Sharan, F

    S. Sharan, F. Pittaluga, M. Chandraker, et al. Llm-assist: Enhancing closed-loop planning with language-based reasoning.arXiv preprint arXiv:2401.00125, 2023

    Pith/arXiv arXiv 2023

  50. [50]

    C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, J. Beißwenger, P. Luo, A. Geiger, and H. Li. Drivelm: Driving with graph visual question answering. InEuropean conference on computer vision, pages 256–274. Springer, 2024

    2024

  51. [51]

    Huang, E

    X. Huang, E. M. Wolff, P. Vernaza, T. Phan-Minh, H. Chen, D. S. Hayden, M. Edmonds, B. Pierce, X. Chen, P. E. Jacob, et al. Drivegpt: Scaling autoregressive behavior models for driving. InForty-second International Conference on Machine Learning

  52. [52]

    Y . Chen, Y . Wang, and Z. Zhang. Drivinggpt: Unifying driving world modeling and planning with multi-modal autoregressive transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 26890–26900, 2025

    2025

  53. [53]

    X. Li, Y . Bai, P. Cai, L. Wen, D. Fu, B. Zhang, X. Yang, X. Cai, T. Ma, J. Guo, et al. Towards knowledge-driven autonomous driving.arXiv preprint arXiv:2312.04316, 2023

    Pith/arXiv arXiv 2023

  54. [54]

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626, 2023

    2023

  55. [55]

    waving to stop a car

    T. Tan, Y . Zheng, R. Liang, Z. Wang, K. Zheng, J. Zheng, J. Li, X. Zhan, and J. Liu. Flow matching-based autonomous driving planning with advanced interactive behavior modeling. Advances in Neural Information Processing Systems, 38:38310–38335, 2026. 13 A Simulation Details Ego observation.For ego planning, we preserve the default nuPlan observation, inc...

    2026