Pith. sign in

REVIEW 4 major objections 4 minor 62 references

GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving

T0 review · 4 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Language-based driving models trained with serialized traffic scene graphs drive better, even with graphs removed from the prompt at test time.

desk verdict Useful systematic study of scene-graph prompting for language-based driving, but the headline SG10 result is oversold: no error bars, model-dependent effects, and an abstract that overclaims. read the letter →

arxiv 2511.11266 v4 pith:DCMQM62V submitted 2025-11-14 cs.CV

Fabian Schmidt , Markus Enzweiler , Abhinav Valada This is my paper · ORCID
classification cs.CV
keywords autonomousdrivingscenegraphsvision-languagemodelspromptconditioningrelationalsupervisionclosed-loopLangAutobenchmarkgraphserialization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Language-based driving models are vision-language agents that map camera and LiDAR input plus a natural-language navigation command to a trajectory; today they infer traffic interactions implicitly from dense visual features. This paper proposes injecting explicit relational structure instead: a per-timestep traffic scene graph, serialized into text and placed into the model's prompt. The central claim is that training with these serialized graphs yields consistent driving-score gains on the LangAuto benchmark for two different state-of-the-art agents, and the gains persist when the graph is omitted at test time. The authors interpret this as the model internalizing relational priors during training, which matters because it means the complex and error-prone step of generating scene graphs at runtime can be skipped. The effect holds across abstraction levels and formats, with the leanest 'actor-only' representation — just agents and their pairwise relations — giving the best performance per token.

What carries the argument

The central object is the traffic scene graph: a directed, labeled multigraph whose nodes are actors (vehicles, pedestrians, the ego car), traffic objects (traffic lights, signs), and structural elements (lanes, roads, junctions), and whose edges encode proximity, directional, lateral, hierarchical, topological, and regulatory relations. The mechanism that carries the argument is its serialization — into compact text, JSON, or YAML — and insertion into the language-model prompt through one of three templates, leaving the model architecture and loss untouched. What isolates the effect is the four-way experimental contrast (no graphs; test-time-only; training-only; train-and-test), with the tr

What would settle it

Fine-tune the same models in the SG10 condition but replace the semantic relation labels in the serialized graphs with random labels drawn from the same vocabulary, preserving token counts and graph structure. If the random-label models match the semantic-graph models in driving score, the claimed relational grounding is a token-count artifact; if semantic graphs clearly outperform scrambled ones, the relational explanation is supported.

Watch

Extended reading notes

Core claim

The central discovery is that prompt-level relational supervision during training is sufficient to make language-based driving models plan better, and that the benefit survives removing the graph at inference. In the authors' SG10 setting, models are fine-tuned with serialized scene graphs in the prompt and then tested without any graph; on two different driving VLMs this setting matches or exceeds the 'graphs at train and test' setting and clearly outperforms both the pretrained baselines and fine-tuning without graphs. The best configuration raises mean driving score from 42.95 to 51.8 for LMDrive and from 44.70 to 56.1 for BEVDriver on LangAuto, while roughly halving collisions with vehic

Load-bearing premise

The load-bearing premise is that the scene graphs used for training are accurate, complete representations of the traffic scene; the paper adopts an existing scene-graph extraction tool without evaluating graph quality, so the measured gains could in principle come from added structured tokens rather than from genuine relational information, and could disappear when graphs are produced by noisy real-world perception.

Editorial extensions

If this is right

  • Test-time scene graph generation can be omitted entirely: the training-only (SG10) configuration matches or beats the full-supervision one, removing a complex, error-prone runtime module and shortening prompts.
  • The method is model-agnostic: it improves both LMDrive and BEVDriver, which use different language backbones and prompt conventions, so prompt-level relational supervision appears to transfer across architectures.
  • Lean actor-only graphs (about 70 tokens on average, versus 400+ for full graphs) deliver the best driving scores, pointing to inter-actor relations as the most planning-relevant part of the scene structure.
  • Scene-graph-conditioned training roughly halves collisions with vehicles and layout elements, nearly eliminates pedestrian collisions, and cuts route deviations, so the gains are not just higher travel distance but safer behavior.
  • Even without any training exposure (test-time-only injection), several prompt configurations beat the baseline, so pretrained driving VLMs can already exploit relational text context.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A cheap falsification of the 'internalization' explanation would be to scramble the relation labels in the SG10 training graphs (keeping token counts and structure identical); if random-label graphs match semantic graphs, the effect is token-level regularization, not relational grounding.
  • The SG10 result suggests the visual encoder is reshaped during training to extract the very relations the graph spells out; inspecting attention or activation patterns before and after SG10 fine-tuning would test this directly.
  • The training graphs come from simulator ground truth, so the method's real-world transfer hinges on whether automatically detected, noisy scene graphs still provide the same training signal; that is the most direct next experiment.
  • Since actor-only graphs capture most of the benefit, a simpler auxiliary objective such as predicting pairwise relation labels from visual features might achieve similar grounding with even less prompt overhead — the serialization approach is one implementation of a broader relational-supervision principle.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. GraphPilot proposes a model-agnostic method for conditioning language-based autonomous driving agents on serialized traffic scene graphs. The method constructs scene graphs at three abstraction levels (Full, Road-Level, Actor-Only), serializes them in Text/JSON/YAML, and injects them via one of three prompt templates. Experiments with LMDrive and BEVDriver on the LangAuto benchmark evaluate four usage settings: SG00 (no graphs), SG01 (test-time only), SG10 (training only), and SG11 (train and test). The central claims are that scene graph conditioning during training improves driving score and that these gains persist when graphs are omitted at test time (SG10 matches/exceeds SG11). The paper also reports token statistics, training GPU hours, and inference latency/memory.

Significance. If the central claims held, this would be a useful contribution: a simple, architecture-agnostic way to inject relational structure into VLM planners, with a practical deployment benefit for SG10 (no test-time scene graph generation). The extensive ablations across abstraction levels, serialization formats, and templates, plus the promise of released code, models, and dataset, are commendable. However, the evidence in Table 4 does not consistently support the SG10-vs-SG11 claim, and the absence of variance/confidence intervals makes it difficult to evaluate the reliability of the reported gains. The direction is promising, but the current manuscript overstates the strength of the findings.

major comments (4)
  1. [Sec. 4.6, Table 4] The sentence 'SG10 again matches or exceeds SG11' is contradicted by the table for BEVDriver. In four of six cells SG11 has a higher DS than SG10 (JSON V2: 53.2 vs 47.3; JSON V3: 43.8 vs 42.2; YAML V3: 52.8 vs 48.2; Text V3: 52.2 vs 51.5), one cell is tied, and only YAML V2 favors SG10. The mean row for BEVDriver also favors SG11 (V2: 52.5 vs 51.7; V3: 49.6 vs 47.3). For the persistence claim, three SG10 cells (JSON V2 47.3, JSON V3 42.2, YAML V3 48.2) fall below the BEVDriver SG00-FT baseline of 48.94. The conclusion that SG10 provides 'large and persistent improvements' across both models is therefore not supported by the reported means.
  2. [Sec. 4.1 and all result tables] The paper states that each configuration is executed three times and results are averaged, but no variance, confidence interval, or paired route-level significance test is reported anywhere. With three runs, differences such as LMDrive Text V2 SG10 (46.8) vs SG11 (46.4) or JSON V3 SG10 (45.5) vs SG11 (48.8) are within typical noise. Without uncertainty estimates, the claims that SG10 'matches or exceeds' SG11 and that gains are 'significant' cannot be evaluated. Please report standard deviations or confidence intervals for the headline driving scores and perform paired route-level comparisons (or a sign test across configurations) before making statements about relative superiority.
  3. [Abstract vs. body] The abstract asserts 'Extensive evaluations on the LangAuto and Bench2Drive benchmarks' and mentions SimLingo baselines, but the paper only reports LangAuto experiments with LMDrive and BEVDriver. There are no Bench2Drive or SimLingo results anywhere in the main text or supplementary. This is an unsupported claim in the abstract and must be corrected, or the missing evaluations must be included.
  4. [Sec. 4.6, Tables 2 and 4] The fine-tuning configurations (Actor-Only, V2, V3) are selected based on SG01 results on the same benchmark and then used for the SG10/SG11 comparison. Because the same test set is used for selection and evaluation, the reported best numbers are maxima over a searched grid and may overstate the effect. To make the central conclusion robust, report results for all fine-tuned configurations (including V1 and other abstractions) or use a held-out validation split for configuration selection and clearly state the selection protocol.
minor comments (4)
  1. [Abstract] The percentages in the abstract (15.6% for LMDrive, 17.5% for BEVDriver) are not directly derivable from the reported baseline and best DS values in Table 4. Please specify the reference baseline (e.g., SG00-PT or SG00-FT) and the exact computation.
  2. [Supplementary, Tables 12/13] The supplementary track-level tables report SG11 and SG01 but omit SG10, so the reader cannot inspect the persistence claim per track. Please add the SG10 track-level breakdowns.
  3. [Sec. 4.6] The text says SG10 and SG11 'significantly outperform SG01 (44.7)', but SG01 was only evaluated for LMDrive. For BEVDriver no SG01 result is reported, so this comparison is incomplete and should be qualified.
  4. [Throughout] The notation 'Y AML' appears with a space in several places; please standardize to 'YAML'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper reports empirical measurements with an externally sourced scene-graph tool; self-citations are contextual and non-load-bearing.

full rationale

The paper contains no formal derivation chain whose conclusion could reduce to its inputs. Its central claims are empirical: conditioning LMDrive and BEVDriver on serialized traffic scene graphs during training yields higher LangAuto driving scores, and SG10 (train-time graphs only) performs comparably to SG11 (graphs at train and test time). These are measurements reported in Tables 2, 4, and the supplementary tables, not quantities derived from definitions. Scene graphs are produced by an external tool, roadscene2vec [22], which is a cited external system, not a self-citation. The method injects graphs via prompt templates (Section 3.3) without modifying architectures or losses, so there is no fitted parameter that is later renamed as a prediction. The only configuration selection is the choice of Actor-Only abstraction and templates V2/V3 after the SG01 sweep in Table 2; this is ordinary model selection on the same benchmark and, while it may affect generalizability or statistical support, it is not a definitional or constructional circularity. The paper explicitly acknowledges its limitation that accurate scene graphs are required during training, which is an external dependency rather than circular reasoning. Some self-citations appear in Related Work (e.g., Valada co-authored scene-graph references), but none is load-bearing: the paper does not invoke a uniqueness theorem, an ansatz, or an unverified prior result to force its conclusions. The skeptical concerns about SG10-vs-SG11 parity and missing variance are statistical-evidence concerns, not circularity, and under the instructions such concerns belong to correctness risk rather than the circularity score. Therefore no circular step is identified and the score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No new physical entities or fitted numeric parameters. The central claim rests on the accuracy of the scene-graph extraction pipeline and the reliability of the benchmark.

free parameters (1)
  • Best configuration selection (Actor-Only + V2/V3)
    The best-performing abstraction and template are chosen based on the authors' SG01 experiments; no numeric parameter is fitted, but the headline results come from a post hoc selection.
assumptions (3)
  • domain assumption roadscene2vec (or an extension) produces accurate scene graphs for each frame.
    The entire method relies on the quality of extracted nodes and relations; the paper does not evaluate scene graph accuracy and acknowledges this in Limitations (Sec. 5).
  • domain assumption LangAuto benchmark metrics (RC, IS, DS) are reliable indicators of driving quality in CARLA.
    Standard benchmark assumption; not independently verified in this work.
  • domain assumption LoRA fine-tuning on the extended LMDrive dataset preserves baseline behavior except for the introduced prompt context.
    The SG00-FT comparison aims to isolate the effect of scene graphs, but dataset shift and fine-tuning dynamics could confound the comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving." pith.science (2026). https://pith.science/paper/DCMQM62V

@misc{pith2026251111266,
  author       = {Pith},
  title        = {Pith review of: GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DCMQM62V}},
  note         = {Machine review of arXiv:2511.11266}
}
read the original abstract

Vision-language models have recently emerged as promising planners for autonomous driving, where success hinges on topology-aware reasoning over spatial structure and dynamic interactions from multimodal input. However, existing models are typically trained without supervision that explicitly encodes these relational dependencies, limiting their ability to infer how agents and other traffic entities influence one another from raw sensor data. In this work, we bridge this gap with a novel model-agnostic method that conditions language-based driving models on structured relational context in the form of traffic scene graphs. We serialize scene graphs at various abstraction levels and formats, and incorporate them into models via structured prompt templates, enabling systematic analysis of when and how relational supervision is most beneficial and computationally efficient. Extensive evaluations on the LangAuto and Bench2Drive benchmarks show that scene graph conditioning yields large and persistent improvements. We observe a substantial performance increase in the Driving Score of our proposed approach versus competitive LMDrive, BEVDriver, and SimLingo baselines. These results indicate that diverse architectures can effectively internalize and ground relational priors through scene graph-conditioned training, even without requiring scene graph input at test-time. Code, fine-tuned models, and our scene graph dataset are publicly available at https://github.com/iis-esslingen/GraphPilot.

Figures

Figures reproduced from arXiv: 2511.11266 by the authors.

Figure 1
Figure 1. Explicit relational grounding through scene graph conditioning. We visualize four setups, where two binary digits indicate scene graph usage during training and testing: SG00-PT (baseline pretrained, no scene graphs), SG00-FT (baseline fine￾tuned without scene graphs), SG10 (scene graphs only during training), and SG11 (scene graphs during training and testing). Models trained under SG10 perform largely on par with … view at source ↗
Figure 2
Figure 2. Scene graph construction. Each traffic scene is represented as a structured, labeled graph capturing entities (actors, objects, structure) and their relations. We define three levels of abstraction: Full (all node types and relations), Road-Level (collapsed structural detail), and Actor-Only (actors and their pairwise interactions), enabling analysis of the trade-off between relational fidelity and prompt efficiency… view at source ↗
Figure 3
Figure 3. Serialization formats. We serialize scene graphs as Text, JSON, or YAML, each encoding subject-predicate-object triplets. Text uses compact natural-language packing for brevity, JSON provides a structured and parser-friendly representation, and YAML achieves lower token counts through minimal syntax. 3.2. Scene Graph Serialization We serialize each graph as subject-predicate-object state￾ments using a fixed, human-r… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Prompt templates. Three prompt templates combine scene graphs with navigation commands: V1 uses direct concate￾nation, V2 adds ego-role framing and section headers, and V3 in￾troduces a structured preamble with markdown-style fencing for consistent formatting. 3.3. Sce…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 2 linked inside Pith

  1. [1]

    nuscenes: A mul- timodal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A mul- timodal dataset for autonomous driving. InCVPR, pages 11621–11631, 2020. 2

    2020

  2. [2]

    Solve: Synergy of language-vision and end-to-end networks for autonomous driving

    Xuesong Chen, Linjiang Huang, Tao Ma, Rongyao Fang, Shaoshuai Shi, and Hongsheng Li. Solve: Synergy of language-vision and end-to-end networks for autonomous driving. InCVPR, pages 12068–12077, 2025. 2

    2025

  3. [3]

    From static to dynamic: a survey of topology-aware perception in autonomous driving

    Yixiao Chen, Ruining Yang, Xin Chen, Jia He, Dongliang Xu, and Yue Yao. From static to dynamic: a survey of topology-aware perception in autonomous driving. InICCV, pages 4511–4523, 2025. 1

    2025

  4. [4]

    A survey on multimodal large language models for autonomous driving

    Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al. A survey on multimodal large language models for autonomous driving. InWACV, pages 958–979,

  5. [5]

    Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking.NeurIPS, 37:28706–28719, 2024

    Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, et al. Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking.NeurIPS, 37:28706–28719, 2024. 2

    2024

  6. [6]

    CARLA: An open urban driving simulator

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. InCoRL, pages 1–16, 2017. 2

    2017

  7. [7]

    Orion: A holistic end-to- end autonomous driving framework by vision-language in- structed action generation

    Haoyu Fu, Diankun Zhang, Zongchuang Zhao, Jianfeng Cui, Dingkang Liang, Chong Zhang, Dingyuan Zhang, Hongwei Xie, Bing Wang, and Xiang Bai. Orion: A holistic end-to- end autonomous driving framework by vision-language in- structed action generation. InICCV, pages 24823–24834,

  8. [8]

    Collaborative dynamic 3d scene graphs for automated driving

    Elias Greve, Martin B ¨uchner, Niclas V¨odisch, Wolfram Bur- gard, and Abhinav Valada. Collaborative dynamic 3d scene graphs for automated driving. InICRA, pages 11118–11124. IEEE, 2024. 3

    2024

Show all 62 references
  1. [9]

    Conceptgraphs: Open-vocabulary 3d scene graphs for per- ception and planning

    Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for per- ception and planning. InICRA, pages 5021–5028. IEEE,

  2. [10]

    Dme-driver: Integrating human decision logic and 3d scene perception in autonomous driving

    Wencheng Han, Dongqian Guo, Cheng-Zhong Xu, and Jian- bing Shen. Dme-driver: Integrating human decision logic and 3d scene perception in autonomous driving. InAAAI, pages 3347–3355, 2025. 2

    2025

  3. [11]

    Dis- tilling multi-modal large language models for autonomous driving

    Deepti Hegde, Rajeev Yasarla, Hong Cai, Shizhong Han, Apratim Bhattacharyya, Shweta Mahajan, Litian Liu, Risheek Garrepalli, Vishal M Patel, and Fatih Porikli. Dis- tilling multi-modal large language models for autonomous driving. InCVPR, pages 27575–27585, 2025. 2

    2025

  4. [12]

    Language-grounded dy- namic scene graphs for interactive object search with mobile manipulation.IEEE Robot

    Daniel Honerkamp, Martin B ¨uchner, Fabien Despinoy, Tim Welschehold, and Abhinav Valada. Language-grounded dy- namic scene graphs for interactive object search with mobile manipulation.IEEE Robot. Autom. Lett., 2024. 1

    2024

  5. [13]

    EMMA: End-to-end multimodal model for autonomous driving.TMLR, 2025

    Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, Yin Zhou, James Guo, Dragomir Anguelov, and Mingxing Tan. EMMA: End-to-end multimodal model for autonomous driving.TMLR, 2025. 2

    2025

  6. [14]

    Hdgt: Heterogeneous driving graph trans- former for multi-agent trajectory prediction via scene encod- ing.IEEE TPAMI, 45(11):13860–13875, 2023

    Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, and Junchi Yan. Hdgt: Heterogeneous driving graph trans- former for multi-agent trajectory prediction via scene encod- ing.IEEE TPAMI, 45(11):13860–13875, 2023. 2, 3

    2023

  7. [15]

    Image retrieval using scene graphs

    Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. InCVPR, pages 3668–3678,

  8. [16]

    The bare necessities: Designing simple, effective open- vocabulary scene graphs.arXiv preprint arXiv:2412.01539,

    Christina Kassab, Mat ´ıas Mattamala, Sacha Morin, Martin B¨uchner, Abhinav Valada, Liam Paull, and Maurice Fal- lon. The bare necessities: Designing simple, effective open- vocabulary scene graphs.arXiv preprint arXiv:2412.01539,

    arXiv

  9. [17]

    Vlr-driver: Large vision-language-reasoning models for embodied autonomous driving

    Fanjie Kong, Yitong Li, Weihuang Chen, Chen Min, Yizhe Li, Zhiqiang Gao, Haoyang Li, Zhongyu Guo, and Hongbin Sun. Vlr-driver: Large vision-language-reasoning models for embodied autonomous driving. InICCV, pages 26966– 26976, 2025. 2

    2025

  10. [18]

    Work zones challenge vlm tra- jectory planning: Toward mitigation and robust autonomous driving.arXiv preprint arXiv:2510.02803, 2025

    Yifan Liao, Zhen Sun, Xiaoyun Qiu, Zixiao Zhao, Wen- bing Tang, Xinlei He, Xinhu Zheng, Tianwei Zhang, Xinyi Huang, and Xingshuo Han. Work zones challenge vlm tra- jectory planning: Toward mitigation and robust autonomous driving.arXiv preprint arXiv:2510.02803, 2025. 3

    2025

  11. [19]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InCVPR, pages 26296–26306, 2024. 5

    2024

  12. [20]

    Learning from interaction-enhanced scene graph for pedestrian collision risk assessment.IEEE T-IV, 8(9):4237–4248, 2023

    Xinxin Liu, Yuchen Zhou, and Chao Gou. Learning from interaction-enhanced scene graph for pedestrian collision risk assessment.IEEE T-IV, 8(9):4237–4248, 2023. 2, 3

    2023

  13. [21]

    T2sg: Traffic topology scene graph for topology reasoning in autonomous driving

    Changsheng Lv, Mengshi Qi, Liang Liu, and Huadong Ma. T2sg: Traffic topology scene graph for topology reasoning in autonomous driving. InCVPR, pages 17197–17206, 2025. 3

    2025

  14. [22]

    roadscene2vec: A tool for extracting and em- bedding road scene-graphs.Knowledge-Based Systems, 242: 108245, 2022

    Arnav Vaibhav Malawade, Shih-Yuan Yu, Brandon Hsu, Harsimrat Kaeley, Anurag Karra, and Mohammad Abdullah Al Faruque. roadscene2vec: A tool for extracting and em- bedding road scene-graphs.Knowledge-Based Systems, 242: 108245, 2022. 3

    2022

  15. [23]

    Spatiotemporal scene-graph em- bedding for autonomous vehicle collision prediction.IEEE Internet of Things Journal, 9(12):9379–9388, 2022

    Arnav Vaibhav Malawade, Shih-Yuan Yu, Brandon Hsu, Deepan Muthirayan, Pramod P Khargonekar, and Moham- mad Abdullah Al Faruque. Spatiotemporal scene-graph em- bedding for autonomous vehicle collision prediction.IEEE Internet of Things Journal, 9(12):9379–9388, 2022. 2, 3

    2022

  16. [24]

    A language agent for autonomous driving

    Jiageng Mao, Junjie Ye, Yuxi Qian, Marco Pavone, and Yue Wang. A language agent for autonomous driving. InFirst Conference on Language Modeling, 2024. 2

    2024

  17. [25]

    Continuously learning, adapting, and improving: A dual- process approach to autonomous driving

    Jianbiao Mei, Yukai Ma, Xuemeng Yang, Licheng Wen, Xinyu Cai, Xin Li, Daocheng Fu, Bo Zhang, Pinlong Cai, Min Dou, Botian Shi, Liang He, Yong Liu, and Yu Qiao. Continuously learning, adapting, and improving: A dual- process approach to autonomous driving. InNeurIPS, 2024. 2

    2024

  18. [26]

    Multi-agent trajectory prediction with heterogeneous edge- 9 enhanced graph attention network.IEEE Transactions on Intelligent Transportation Systems, 23(7):9554–9567, 2022

    Xiaoyu Mo, Zhiyu Huang, Yang Xing, and Chen Lv. Multi-agent trajectory prediction with heterogeneous edge- 9 enhanced graph attention network.IEEE Transactions on Intelligent Transportation Systems, 23(7):9554–9567, 2022. 3

    2022

  19. [27]

    More: Mo- bile manipulation rearrangement through grounded language reasoning.IROS, 2025

    Mohammad Mohammadi, Daniel Honerkamp, Martin B¨uchner, Matteo Cassinelli, Tim Welschehold, Fabien De- spinoy, Igor Gilitschenski, and Abhinav Valada. More: Mo- bile manipulation rearrangement through grounded language reasoning.IROS, 2025. 1

    2025

  20. [28]

    Vlp: Vision language planning for autonomous driving

    Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. Vlp: Vision language planning for autonomous driving. InCVPR, pages 14760–14769, 2024. 2

    2024

  21. [29]

    Sayplan: Ground- ing large language models using 3d scene graphs for scalable robot task planning

    Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou- Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Ground- ing large language models using 3d scene graphs for scalable robot task planning. InCoRL, 2023. 1

    2023

  22. [30]

    Simlingo: Vision-only closed-loop autonomous driving with language-action alignment

    Katrin Renz, Long Chen, Elahe Arani, and Oleg Sinavski. Simlingo: Vision-only closed-loop autonomous driving with language-action alignment. InCVPR, pages 11993–12003,

  23. [31]

    Lmdrive: Closed-loop end-to-end driving with large language models

    Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. InCVPR, pages 15120–15130, 2024. 1, 2, 5

    2024

  24. [32]

    Collaborative dynamic 3d scene graphs for open- vocabulary urban scene understanding

    Tim Steinke, Martin B ¨uchner, Niclas V¨odisch, and Abhinav Valada. Collaborative dynamic 3d scene graphs for open- vocabulary urban scene understanding. InIROS, 2025. 3

    2025

  25. [33]

    Drivevlm: The convergence of autonomous driving and large vision-language models

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. InCoRL, 2024. 2

    2024

  26. [34]

    Rsg-gcn: Predicting semantic relationships in urban traffic scene with map geometric prior.IEEE Open Journal of Intelligent Transportation Systems, 4:244–260, 2023

    Yafu Tian, Alexander Carballo, Ruifeng Li, and Kazuya Takeda. Rsg-gcn: Predicting semantic relationships in urban traffic scene with map geometric prior.IEEE Open Journal of Intelligent Transportation Systems, 4:244–260, 2023. 3

    2023

  27. [35]

    Rsg-search: semantic traffic scene retrieval using graph-based scene representation

    Yafu Tian, Alexander Carballo, Ruifeng Li, and Kazuya Takeda. Rsg-search: semantic traffic scene retrieval using graph-based scene representation. In2023 IEEE Intelligent Vehicles Symposium (IV), pages 1–8. IEEE, 2023. 3

    2023

  28. [36]

    Rsg-search plus: An advanced traffic scene retrieval methods based on road scene graph

    Yafu Tian, Alexander Carballo, Ruifeng Li, Simon Thomp- son, and Kazuya Takeda. Rsg-search plus: An advanced traffic scene retrieval methods based on road scene graph. In 2024 IEEE Intelligent Vehicles Symposium (IV), pages 1171–

    2024

  29. [37]

    Query by example: Semantic traf- fic scene retrieval using llm-based scene graph representa- tion.Sensors, 25(8):2546, 2025

    Yafu Tian, Alexander Carballo, Ruifeng Li, Simon Thomp- son, and Kazuya Takeda. Query by example: Semantic traf- fic scene retrieval using llm-based scene graph representa- tion.Sensors, 25(8):2546, 2025. 3

    2025

  30. [38]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 5

    2023 arXiv

  31. [39]

    Rs2g: Data-driven scene-graph extraction and embedding for ro- bust autonomous perception and scenario understanding

    Junyao Wang, Arnav Vaibhav Malawade, Junhong Zhou, Shih-Yuan Yu, and Mohammad Abdullah Al Faruque. Rs2g: Data-driven scene-graph extraction and embedding for ro- bust autonomous perception and scenario understanding. In WACV, pages 7493–7502, 2024. 3

    2024

  32. [40]

    Rad: Retrieval-augmented decision-making of meta- actions with vision-language models in autonomous driving

    Yujin Wang, Quanfeng Liu, Zhengxin Jiang, Tianyi Wang, Junfeng Jiao, Hongqing Chu, Bingzhao Gao, and Hong Chen. Rad: Retrieval-augmented decision-making of meta- actions with vision-language models in autonomous driving. InCVPR, pages 3838–3848, 2025. 2

    2025

  33. [41]

    Dilu: A knowledge-driven approach to autonomous driving with large language models

    Licheng Wen, Daocheng Fu, Xin Li, Xinyu Cai, Tao MA, Pinlong Cai, Min Dou, Botian Shi, Liang He, and Yu Qiao. Dilu: A knowledge-driven approach to autonomous driving with large language models. InICLR, 2024. 2

    2024

  34. [42]

    Hierarchical open- vocabulary 3d scene graphs for language-grounded robot navigation

    Abdelrhman Werby, Chenguang Huang, Martin B ¨uchner, Abhinav Valada, and Wolfram Burgard. Hierarchical open- vocabulary 3d scene graphs for language-grounded robot navigation. InRSS, 2024. 1

    2024

  35. [43]

    Bev- driver: Leveraging bev maps in llms for robust closed-loop driving

    Katharina Winter, Mark Azer, and Fabian B Flohr. Bev- driver: Leveraging bev maps in llms for robust closed-loop driving. InIROS, 2025. 1, 2, 5

    2025

  36. [44]

    Are vlms ready for autonomous driving? an empirical study from the reliability, data and metric perspectives

    Shaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima, Wenwei Zhang, Qi Alfred Chen, Ziwei Liu, and Liang Pan. Are vlms ready for autonomous driving? an empirical study from the reliability, data and metric perspectives. In ICCV, pages 6585–6597, 2025. 1

    2025

  37. [45]

    S4-driver: Scalable self-supervised driving mul- timodal large language model with spatio-temporal visual representation

    Yichen Xie, Runsheng Xu, Tong He, Jyh-Jing Hwang, Katie Luo, Jingwei Ji, Hubert Lin, Letian Chen, Yiren Lu, Zhaoqi Leng, et al. S4-driver: Scalable self-supervised driving mul- timodal large language model with spatio-temporal visual representation. InCVPR, pages 1622–1632, 2025. 2

    2025

  38. [46]

    Drivegpt4-v2: Harnessing large language model capabilities for enhanced closed-loop autonomous driving

    Zhenhua Xu, Yan Bai, Yujia Zhang, Zhuoling Li, Fei Xia, Kwan-Yee K Wong, Jianqiang Wang, and Hengshuang Zhao. Drivegpt4-v2: Harnessing large language model capabilities for enhanced closed-loop autonomous driving. InCVPR, pages 17261–17270, 2025. 2

    2025

  39. [47]

    Sg-nav: Online 3d scene graph prompting for llm-based zero-shot object navigation.NeurIPS, 37:5285–5307, 2024

    Hang Yin, Xiuwei Xu, Zhenyu Wu, Jie Zhou, and Jiwen Lu. Sg-nav: Online 3d scene graph prompting for llm-based zero-shot object navigation.NeurIPS, 37:5285–5307, 2024. 1

    2024

  40. [48]

    Rag-driver: Gen- eralisable driving explanations with retrieval-augmented in- context multi-modal large language model learning

    Jianhao Yuan, Shuyang Sun, Daniel Omeiza, Bo Zhao, Paul Newman, Lars Kunze, and Matthew Gadd. Rag-driver: Gen- eralisable driving explanations with retrieval-augmented in- context multi-modal large language model learning. InRSS,

  41. [49]

    Futuresight- drive: Thinking visually with spatio-temporal cot for au- tonomous driving

    Shuang Zeng, Xinyuan Chang, Mengwei Xie, Xinran Liu, Yifan Bai, Zheng Pan, Mu Xu, and Xing Wei. Futuresight- drive: Thinking visually with spatio-temporal cot for au- tonomous driving. InNeurIPS, 2025. 2

    2025

  42. [50]

    Feedback-guided autonomous driving

    Jimuyang Zhang, Zanming Huang, Arijit Ray, and Eshed Ohn-Bar. Feedback-guided autonomous driving. InCVPR, pages 15000–15011, 2024. 2

    2024

  43. [51]

    Safeauto: Knowledge-enhanced safe autonomous driving with multimodal foundation models

    Jiawei Zhang, Xuan Yang, Taiqi Wang, Yu Yao, Aleksandr Petiushko, and Bo Li. Safeauto: Knowledge-enhanced safe autonomous driving with multimodal foundation models. In ICML, 2025. 1, 2

    2025

  44. [52]

    Adadrive: Self-adaptive slow-fast system for language-grounded autonomous driv- ing

    Ruifei Zhang, Junlin Xie, Wei Zhang, Weikai Chen, Xiao Tan, Xiang Wan, and Guanbin Li. Adadrive: Self-adaptive slow-fast system for language-grounded autonomous driv- ing. InICCV, pages 5112–5121, 2025. 1, 2 10

    2025

  45. [53]

    Vldrive: Vision-augmented lightweight mllms for efficient language- grounded autonomous driving

    Ruifei Zhang, Wei Zhang, Xiao Tan, Sibei Yang, Xi- ang Wan, Xiaonan Luo, and Guanbin Li. Vldrive: Vision-augmented lightweight mllms for efficient language- grounded autonomous driving. InICCV, pages 5923–5933,

  46. [54]

    Graphad: Interaction scene graph for end-to-end autonomous driving

    Yunpeng Zhang, Deheng Qian, Ding Li, Yifeng Pan, Yong Chen, Zhenbao Liang, Zhiyao Zhang, Yingzong Liu, Jianhui Mei, Maolei Fu, Yun Ye, Zhujin Liang, Yi Shan, and Da- long Du. Graphad: Interaction scene graph for end-to-end autonomous driving. InIJCAI, pages 2422–2430. Interna-...

    2025

  47. [55]

    Parking-sg: Open-vocabulary hierarchical 3d scene graph representation for open parking environments

    Yaowen Zhang, Yi Ruan, Miaoxin Pan, Yi Yang, and Mengyin Fu. Parking-sg: Open-vocabulary hierarchical 3d scene graph representation for open parking environments. InICRA, pages 7291–7297. IEEE, 2025. 3

    2025

  48. [56]

    Hktsg: A hierarchical knowledge-guided traffic scene graph representation learning framework for intelligent vehi- cles.IEEE T-IV, pages 1–12, 2024

    Yuchen Zhou, Xinxin Liu, Zipeng Guo, Ming Cai, and Chao Gou. Hktsg: A hierarchical knowledge-guided traffic scene graph representation learning framework for intelligent vehi- cles.IEEE T-IV, pages 1–12, 2024. 3

    2024

  49. [57]

    Hivt: Hierarchical vector transformer for multi-agent motion prediction

    Zikang Zhou, Luyao Ye, Jianping Wang, Kui Wu, and Kejie Lu. Hivt: Hierarchical vector transformer for multi-agent motion prediction. InCVPR, pages 8823–8833, 2022. 3 11 GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving Supplementary Material T...

    2022

  50. [58]

    These settings remain fixed across all experiments unless explicitly men- tioned

    Implementation Details This section provides additional details on model configu- ration, dataset preprocessing, and training hyperparameters that apply to both LMDrive and BEVDriver. These settings remain fixed across all experiments unless explicitly men- tioned. Tab. 6 summ...

  51. [59]

    All model, dataset, and training specific hyperparameters are shared across experiments

    Training Time Our training-time analysis covers the combinations of graph abstraction level (Full, Road-Level, Actor-Only), serializa- tion format (Text, JSON, Y AML), and template version Table 6.Shared Implementation Details for LMDrive and BEVDriver.Both methods use their b...

  52. [60]

    Computational Analysis All experiments in this section were executed on a single NVIDIA L40S GPU node (see Sec. 6). To quantify the com- putational overhead introduced by scene graph processing, we measure the runtime and resource usage of each compo- nent: generation, abstrac...

  53. [61]

    The main paper reports aver- aged or representative values due to space constraints; here, we include all combinations for completeness

    Detailed Quantitative Results This section provides full numerical results for all configu- rations evaluated in the paper. The main paper reports aver- aged or representative values due to space constraints; here, we include all combinations for completeness. Extended Baselin...

  54. [62]

    Qualitative Results Alongside this supplementary material, we include a video showcasing qualitative results of our approach, as well as failure cases of the LMDrive and BEVDriver baselines in which our method demonstrates superior driving per- formance. The failure scenarios ...

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.