Pith. sign in

REVIEW 5 major objections 4 minor 54 references

DriveSOTIF: Advancing Perception SOTIF Through Multimodal Large Language Models

T0 review · 5 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning a small multimodal language model on a SOTIF-specific VQA dataset lets it detect, explain, and recommend responses to perception-related driving hazards in near real time, with a measured 11.8% gain on close-ended and 12.0% on…

desk verdict A useful new SOTIF VQA dataset and a reasonable fine-tuning study, but the headline gains are cherry-picked and the open-ended evaluation is GPT judging GPT. read the letter →

arxiv 2505.07084 v3 pith:AUPXFN5T submitted 2025-05-11 cs.RO

Shucheng Huang , Freda Shi , Chen Sun , Jiaming Zhong , Minghao Ning , Yufeng Yang , Yukun Lu , Hong Wang
show 1 more author
Amir Khajepour
This is my paper · ORCID
classification cs.RO
keywords SOTIFSafetyoftheIntendedFunctionalitymultimodallargelanguagemodelsvisualquestionansweringautonomousdrivingLoRAfine-tuningLLM-as-judgelong-tailtrafficscenarios
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a general-purpose multimodal large language model, fine-tuned on a purpose-built visual question-answering and captioning dataset, can take on the perception-related Safety of the Intended Functionality (SOTIF) reasoning that autonomous vehicles currently lack. The authors introduce DriveSOTIF, the first SOTIF-specific VQA dataset, and report that fine-tuning lifts close-ended accuracy by 11.8% and open-ended rubric scores by 12.0% over baselines, at 0.59 seconds per image on a consumer GPU. Real-world tests in Canada and China show the model identifying snowy-road, merging-truck, and headlight-glare hazards, though one extremely subtle case (a child hidden in a box) is missed. If the result holds, a small onboard model could act as a near-real-time, explainable SOTIF risk-assessment layer for autonomous driving.

What carries the argument

The central object is DriveSOTIF, a dataset of 1,114 images, 1,114 captions, and 5,570 question-answer pairs built from a long-tail perception-SOTIF image collection. The generative machinery is a multi-agent pipeline in which vision-language models alternate roles for captioning, question generation, answer generation, and validation, with failures regenerated and a second model checking image relevance, question suitability, and answer correctness. The training machinery is LoRA-based supervised fine-tuning of open-source multimodal models, and the evaluation machinery for open-ended answers is an LLM-as-judge scoring relevance, trustworthiness, clarity, and coherence. Together these parts turn a generic visual question-answering model into a domain-specific SOTIF risk assessor.

What would settle it

Have a panel of human driving-safety experts label several hundred unseen SOTIF images for risk presence, cause, and recommended action, then compare the fine-tuned model's answers to those human labels rather than to LLM-generated references; if agreement is near chance on cases where experts agree, the claim collapses. The paper's own child-hidden-in-a-box case is a partial falsifier already, since the model misses the hazard.

Watch

Extended reading notes

Core claim

The paper's central claim is that domain-specific supervised fine-tuning gives multimodal LLMs the spatial and causal intelligence that human drivers use to judge safety: open-world generalization to unseen hazards, causal reasoning about combinations of conditions such as rain, night, and glare, and contextual understanding of scene actors. Fine-tuned models produce longer, situation-aware captions and more detailed, scenario-specific visual question-answering responses than their baselines. The largest gains appear in small models, with the 1-billion-parameter model's accuracy rising from 55.7% to 63.5% and its overall judge score from 3.84 to 4.30, making the capability compatible with onboard, resource-limited deployment. The paper also reports a limit: the model misses a child hidden in a box, which it attributes to the visual encoder's capacity.

Load-bearing premise

The model's notion of a correct SOTIF risk assessment is inherited from the large language models that wrote the answers and from the language model that scores them, so the reported gains could measure agreement with those models' opinions rather than with safety truth.

Editorial extensions

If this is right

  • A model with roughly 1 billion parameters and about 2 GB of GPU memory can serve as an onboard, near-real-time SOTIF risk assessment module instead of a cloud call.
  • Fine-tuning lifts both captioning and visual question answering across model sizes, with the largest gains in small models, so resource-constrained vehicles benefit most.
  • Learned SOTIF reasoning transfers across countries and weather conditions, suggesting a single fine-tuned model can cover diverse operational design domains.
  • Connecting the model to a decision-making layer would let an autonomous vehicle factor perception-risk explanations into real-time planning, with continuous improvement via human-in-the-loop data.
  • The approach does not yet handle extremely subtle hazards, as the hidden-child case shows, so a safety case would need to bound the miss rate on such scenarios.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because both training answers and evaluation scores come from large language models, part of the measured gain may be alignment with those models' prior opinions rather than with safety truth; a deployment-grade claim needs human-expert or physical ground truth.
  • Beyond the paper: the same generation-and-fine-tuning pipeline should transfer to LiDAR, radar, and bird's-eye-view inputs; a multi-sensor SOTIF risk dataset is a natural next test and would show whether the approach scales beyond camera images.
  • Beyond the paper: evaluating a DriveSOTIF-fine-tuned model on an independent corner-case benchmark for risk localization would separate genuine SOTIF reasoning from dataset-specific phrasing.
  • Beyond the paper: the reported trade-off between response time and concurrency suggests a fast-slow architecture in practice, with lightweight perception running continuously and the MLLM triggered for semantic risk assessment rather than invoked on every frame at high speed.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces DriveSOTIF, a visual question answering (VQA) and image captioning dataset for perception-related Safety of the Intended Functionality (SOTIF) in autonomous driving. The dataset is generated by a multi-agent pipeline using GPT-4v, GPT-4o, and Claude 3 Opus, with GPT-4.1-based validation, and contains 1,114 images, 1,114 captions, and 5,570 question-answer pairs. The authors benchmark several open-source multimodal LLMs before and after LoRA-based supervised fine-tuning, reporting gains in close-ended VQA accuracy and in GPT-4.1-judged open-ended VQA scores, along with deployment latency measurements on RTX 3090 and Jetson Orin platforms and qualitative real-world case studies from Canada and China. The paper claims to be the first application of domain-specific MLLM fine-tuning to the SOTIF domain.

Significance. If the reported results are taken at face value, the paper offers a useful new resource: DriveSOTIF is, to my knowledge, the first VQA and captioning dataset focused on perception-related SOTIF, and the authors make the dataset and code publicly available. The multi-agent generation pipeline with validation and the careful deployment study are pragmatic contributions, and the close-ended accuracy improvements are consistent across all eight evaluated models. The paper also contains an honest discussion of hallucination and of the hidden-child failure case. However, the central open-ended evaluation is circular because the annotation, validation, and judging all come from the same LLM family, and the paper's universal-improvement claim is contradicted by its own Table VII. These issues affect the main quantitative claims and need to be addressed before the paper can be accepted.

major comments (5)
  1. [Section IV.C and Table VII] The claim that "Fine-tuning improves performance across all evaluation metrics" is directly contradicted by Table VII. For example, Qwen2.5-VL-7B relevance drops from 4.57 to 4.13 and overall from 4.76 to 4.61; InternVL3-2B relevance drops from 4.22 to 3.92 and overall from 4.48 to 4.47; InternVL3-8B relevance drops from 4.56 to 4.12, trustworthiness from 4.70 to 4.53, and overall from 4.79 to 4.62. The abstract's headline 11.8% and 12.0% figures are also selected best-model gains (InternVL3-2B for close-ended accuracy and InternVL3-1B for overall open-ended score), not aggregate results across the eight models. Please report aggregate statistics with per-model effect sizes, and either substantiate the universal claim or replace it with a more precise statement about which metrics and model sizes improve.
  2. [Sections III.B, III.D, and IV.C] The open-ended evaluation is circular. The ground-truth answers are generated by GPT-4o and Claude 3 Opus and validated by GPT-4.1 (Section III.B), the open-ended responses are scored by GPT-4.1 as an LLM judge (Section III.D), and the fine-tuned models are trained to reproduce GPT-style answers. The reported 12.0% open-ended gain may therefore reflect alignment with the judge's stylistic preferences rather than improved safety-relevant reasoning. The 3-4% human-review error rate on 595 samples does not bound errors across the full training set. Please add a human evaluation of a held-out sample of open-ended responses, report judge-human agreement, and/or use an independent judge model from a different model family than the one used for annotation.
  3. [Section V.A and Abstract] The abstract states that fine-tuned models maintain "real-time performance with a 0.59-second average inference time per image," but Table VIII shows that 0.59 s is specifically the InternVL3-1B result on an RTX 3090. Other configurations are substantially slower, e.g., Qwen2-VL-2B ranges from 0.74 to 0.87 s on the RTX 3090 and 2.93 to 3.70 s on Jetson Orin, and Qwen2.5-VL-3B takes 1.01 to 1.39 s on RTX 3090 and 5.63 to 6.86 s on Orin. Please report latency per model, size, quantization, and platform, and avoid implying that 0.59 s is a general average across the evaluated systems. In addition, the continuous-inference experiments at 30 Hz show many requests queued or dropped, so the claim of real-time suitability should be qualified relative to the actual frame rate and timeout policy.
  4. [Sections V.B, V.C, and Abstract] The real-world case-study evaluation is qualitative and lacks a systematic comparison: there are no baseline model outputs, no quantitative risk-detection or false-alarm rates, and no scoring rubric for the four scenarios. Moreover, the hidden-child scenario is reported as a failure (the model "did not accurately capture and interpret the partially or fully obscured child"), which is inconsistent with the abstract's claim that fine-tuned models "correctly identify safety risks that challenge even experienced human drivers." Please either provide quantitative evaluation on a labeled real-world set with baseline comparison, or substantially soften the abstract and conclusion in line with the mixed results.
  5. [Tables VI and VII] No confidence intervals, error bars, or significance tests are reported for any of the quantitative comparisons. The test set is small (555 questions, 111 images), and several reported differences are only one or two points on the LLM-judge scales, so the improvements may not be statistically reliable. Please report confidence intervals or bootstrap estimates, and paired significance tests where appropriate, especially for the close-ended accuracy and captioning metrics.
minor comments (4)
  1. [Section VI.B] The sentence "Results in Table III show that proprietary LLM models can provide accurate and contextually relevant answers" appears to cite the wrong table, since Table III shows sample dataset annotations rather than benchmark results; please re-check the reference.
  2. [Section III.C] The paragraph beginning "The multi-agent system employed varying temperature settings" ends with "format standardization." and is followed by a fragment beginning "with close-ended and open-ended questions automatically categorized". Please complete the sentence or merge it with the preceding one.
  3. [Section V.A] The conclusion repeats "average inference time of 0.59 seconds per image" without noting that this is the best-case InternVL3-1B result on a specific GPU; please qualify this statement as in the major comment above.
  4. [Section IV.B] The hyperparameter details are deferred to the supplemental material; given that LoRA rank and fine-tuning hyperparameters directly affect the reported gains, please state at least the rank and learning rate in the main text.

Circularity Check

1 steps flagged · score 6.0 of 10

Open-ended VQA evaluation is self-referential: GPT-4.1 both validates the GPT-generated ground truth and scores fine-tuned outputs against it.

  1. self definitional [Sec. III.B 'Dataset Generation through Collaborative LLM-Agents'; Sec. III.D 'Evaluation Metrics'; Sec. IV.B 'Model Fine-tuning']
    "For open-ended VQA, where no ground truth exists, we follow a rubric-based approach, leveraging an LLM-as-judge [34] to score answers. Specifically, we use GPT-4.1 to evaluate responses based on the question, ground-truth answer, input image, and the model's output. ... This validation process leverages OpenAI's GPT-4.1 as the LLM backend to ensure high-quality outputs that align with the perception SOTIF context."

    The open-ended 'ground-truth answers' used by the judge are generated by GPT-4o and Claude 3 Opus and validated by GPT-4.1 (Sec. III.B). Supervised fine-tuning then trains the target models to imitate those same GPT-generated answers (Sec. IV.B). The open-ended evaluator is the same GPT-4.1 model, scoring model outputs against those GPT-generated references (Sec. III.D). Consequently, the reported 12.0% open-ended VQA gain measures how closely the fine-tuned outputs match GPT-4.1-validated GPT-style text, not an independently grounded SOTIF reasoning ability. The evaluation target is defined by the same LLM family that produces the training target, so the open-ended improvement is partially forced by construction and cannot serve as independent validation of safety reasoning.

full rationale

The main circularity is confined to the open-ended VQA evaluation loop: GPT-4.1 validates the LLM-generated ground truth used in training and then acts as the judge that scores fine-tuned model outputs against that same ground truth. This makes the open-ended improvement metric self-referential, because the model is rewarded for reproducing the stylistic and semantic patterns of the GPT-family annotations on which it was trained. The close-ended accuracy results, captioning metrics, deployment latency measurements, and the real-world case studies are external to this loop and give the paper independent empirical content. The self-citations to prior SOTIF work by the same research group are background context and are not load-bearing for the central derivation. The paper's claim that fine-tuning improves performance across all evaluation metrics is additionally contradicted by several rows of Table VII (e.g., Qwen2.5-VL-7B relevance 4.57 to 4.13; InternVL3-8B relevance 4.56 to 4.12 and overall 4.79 to 4.62), but that is a correctness and reporting issue rather than a circularity issue. Overall, the open-ended evaluation is partially circular while the remaining contributions retain independent content, so the appropriate score is 6.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central quantitative claims rely on a small number of externally sourced and internally chosen premises. The visual source is PeSOTIF's first batch. The semantic ground truth is produced by proprietary LLM agents and only sample-checked by humans. The open-ended evaluation is produced by GPT-4.1 without human calibration. These are not fitted constants in the usual sense, but the engineering choices above behave like adjustable settings that shape all reported gains: the question mix, the LoRA setup described only in the supplement, and the judge temperature fixed at 0.7.

free parameters (3)
  • LoRA rank and SFT hyperparameters
    Sec. IV.B says configurations were kept consistent across models with details in supplemental S.II; no values appear in the main text, so the results are not independently reproducible from the paper alone.
  • LLM-as-judge temperature = 0.7
    Sec. III.D fixes temperature at 0.7 for the three repeated GPT-4.1 evaluations; the reported open-ended scores depend on this choice.
  • Per-image question mix = 5 questions per image, 2-3 closed-ended and 2-3 open-ended
    Sec. III.A fixes five questions per image and a target balance; this design choice shapes the benchmark and the measured accuracy gains.
assumptions (5)
  • domain assumption PeSOTIF first-batch images are representative of perception-related SOTIF long-tail traffic scenarios.
    Dataset construction in Sec. III.A relies on PeSOTIF as the visual source; if the image distribution does not cover relevant SOTIF cases, the fine-tuning evidence is unrepresentative.
  • ad hoc to paper LLM-generated captions, questions, and answers (GPT-4v, GPT-4o, Claude 3 Opus) are factually grounded and correctly describe SOTIF risks.
    Sec. III.B answer generation and validation are performed by proprietary LLMs; only a 595-sample human review with a reported 3-4% error rate is described. The full dataset's correctness rests on this automated annotation assumption.
  • ad hoc to paper GPT-4.1 LLM-as-judge scores correlate with true quality of open-ended SOTIF answers.
    Sec. III.D scores open-ended responses with GPT-4.1 over three runs at temperature 0.7, without human-annotated calibration on the DriveSOTIF test set.
  • domain assumption Fine-tuning on VQA data transfers to real-world SOTIF risk assessment.
    Sec. V.B-C presents four qualitative case studies; no quantitative detection rate or comparison against non-fine-tuned baselines on these real-world scenarios is provided.
  • domain assumption A human review of 595 finite-population-sampled entries bounds the dataset error rate at 3-4%.
    Sec. III.C reports this error rate; sampling details are deferred to supplementary S.IV, and the authors do not state whether errors were corrected before release or how they affect benchmark numbers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DriveSOTIF: Advancing Perception SOTIF Through Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/AUPXFN5T

@misc{pith2026250507084,
  author       = {Pith},
  title        = {Pith review of: DriveSOTIF: Advancing Perception SOTIF Through Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AUPXFN5T}},
  note         = {Machine review of arXiv:2505.07084}
}
read the original abstract

Human drivers possess spatial and causal intelligence, enabling them to perceive driving scenarios, anticipate hazards, and react to dynamic environments. In contrast, autonomous vehicles lack these abilities, making it challenging to manage perception-related Safety of the Intended Functionality (SOTIF) risks, especially under complex or unpredictable driving conditions. To address this gap, we propose fine-tuning multimodal large language models (MLLMs) on a customized dataset specifically designed to capture perception-related SOTIF scenarios. Benchmarking results show that fine-tuned MLLMs achieve an 11.8\% improvement in close-ended VQA accuracy and a 12.0\% increase in open-ended VQA scores compared to baseline models, while maintaining real-time performance with a 0.59-second average inference time per image. We validate our approach through real-world case studies in Canada and China, where fine-tuned models correctly identify safety risks that challenge even experienced human drivers. This work represents the first application of domain-specific MLLM fine-tuning for SOTIF domain in autonomous driving. The dataset and related resources are available at github.com/s95huang/DriveSOTIF.git

Figures

Figures reproduced from arXiv: 2505.07084 by the authors.

Figure 1
Figure 1. Overview of the proposed methodology for supervised fine-tuning of pre-trained MLLMs on DriveSOTIF dataset. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Proposed dataset generation pipeline through collaborative LLM Agents. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Tradeoff between API response time and queue waiting [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figures from the paper (1 more)
Figure 3
Figure 3. Figure 3: Average generation time per image in continuous [PITH_FULL_IMAGE:figures/full_fig_p009_3.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 27 canonical work pages

  1. [1]

    A holistic robust motion control framework for autonomous platooning,

    H. Wang, L.-M. Peng, Z. Wei, K. Yang, L. Jiang, E. Hashemiet al., “A holistic robust motion control framework for autonomous platooning,” IEEE Transactions on Vehicular Technology, vol. 72, no. 12, pp. 15 213– 15 226, 2023

    work page 2023

  2. [2]

    A survey on an emerging safety challenge for autonomous vehicles: Safety of the intended functionality,

    H. Wang, W. Shao, C. Sun, K. Yang, D. Cao, and J. Li, “A survey on an emerging safety challenge for autonomous vehicles: Safety of the intended functionality,”Engineering, 2024

    work page 2024

  3. [3]

    Sotif entropy: Online sotif risk quantification and mitigation for autonomous driving,

    L. Peng, B. Li, W. Yu, K. Yang, W. Shao, and H. Wang, “Sotif entropy: Online sotif risk quantification and mitigation for autonomous driving,” IEEE Transactions on Intelligent Transportation Systems, 2023

    work page 2023

  4. [4]

    Pesotif: A challenging visual dataset for perception sotif problems in long-tail traffic scenarios,

    L. Peng, J. Li, W. Shao, and H. Wang, “Pesotif: A challenging visual dataset for perception sotif problems in long-tail traffic scenarios,” in 2023 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2023, pp. 1–8

    work page 2023

  5. [5]

    Multi-modal answer validation for knowledge-based vqa,

    J. Wu, J. Lu, A. Sabharwal, and R. Mottaghi, “Multi-modal answer validation for knowledge-based vqa,” inProceedings of the AAAI conference on artificial intelligence, vol. 36, no. 3, 2022, pp. 2712– 2721

    work page 2022

  6. [6]

    Context-vqa: Towards context- aware and purposeful visual question answering,

    N. Naik, C. Potts, and E. Kreiss, “Context-vqa: Towards context- aware and purposeful visual question answering,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 2821–2825

    work page 2023

  7. [7]

    Coda: A real-world road corner case dataset for object detection in autonomous driving,

    K. Li, K. Chen, H. Wang, L. Hong, C. Ye, J. Han, Y . Chen, W. Zhang, C. Xu, D.-Y . Yeunget al., “Coda: A real-world road corner case dataset for object detection in autonomous driving,”arXiv preprint arXiv:2203.07724, 2022

    arXiv 2022

  8. [8]

    Simulation-based performance evaluation of 3d object detection methods with deep learning for a lidar point cloud dataset in a sotif-related use case,

    M. Patel and R. Jung, “Simulation-based performance evaluation of 3d object detection methods with deep learning for a lidar point cloud dataset in a sotif-related use case,” inProceedings of the 10th Inter- national Conference on Vehicle Technology and Intelligent Transport Systems - VEHITS, INSTICC. SciTePress, 2024, pp. 415–426

    work page 2024

Show all 54 references
  1. [9]

    Automated evaluation of large vision-language models on self-driving corner cases,

    Y . Li, W. Zhang, K. Chen, Y . Liu, P. Li, R. Gao, L. Hong, M. Tian, X. Zhao, Z. Liet al., “Automated evaluation of large vision-language models on self-driving corner cases,”arXiv preprint arXiv:2404.10595, 2024

    2024 arXiv

  2. [10]

    Enhancing autonomous vehicle safety based on operational design domain defi- nition, monitoring, and functional degradation: A case study on lane keeping system,

    Z. Jiang, W. Pan, J. Liu, Y . Han, Z. Pan, H. Li, and Y . Pan, “Enhancing autonomous vehicle safety based on operational design domain defi- nition, monitoring, and functional degradation: A case study on lane keeping system,”IEEE Transactions on Intelligent Vehicles, 2024

    2024

  3. [11]

    Sotif-oriented percep- tion evaluation method for forward obstacle detection of autonomous vehicles,

    J. Chu, T. Zhao, J. Jiao, Y . Yuan, and Y . Jing, “Sotif-oriented percep- tion evaluation method for forward obstacle detection of autonomous vehicles,”IEEE Systems Journal, vol. 17, pp. 2319–2330, 2023

    2023

  4. [12]

    The sotif meta-algorithm: Quantitative analyses of the safety of autonomous behaviors,

    C. C ˆarlan, N. Carlson, C. Dwyer, M. Hirannaiah, and M. Wagner, “The sotif meta-algorithm: Quantitative analyses of the safety of autonomous behaviors,” in2024 IEEE 35th International Symposium on Software Reliability Engineering Workshops (ISSREW). IEEE, 2024, pp. 191– 198

    2024

  5. [13]

    A hazard analysis approach for the sotif in intelligent railway driving assistance systems using stpa and complex network,

    S. Zhang, T. Tang, and J. Liu, “A hazard analysis approach for the sotif in intelligent railway driving assistance systems using stpa and complex network,”Applied Sciences, vol. 11, no. 16, p. 7714, 2021

    2021

  6. [14]

    Safety of the intended functionality (sotif) based on system theoretic process analysis (stpa): Study for specific control action in blind spot detection (bsd),

    M. D. Meneks ¸e, O. ¨Ozc ¸etin, T. E. Ercan, and K. F. Do ˘gan, “Safety of the intended functionality (sotif) based on system theoretic process analysis (stpa): Study for specific control action in blind spot detection (bsd),” in2024 8th International Symposium on Innovative A...

    2024

  7. [15]

    Formal cer- tification methods for automated vehicle safety assessment,

    T. Zhao, E. Yurtsever, J. A. Paulson, and G. Rizzoni, “Formal cer- tification methods for automated vehicle safety assessment,”IEEE Transactions on Intelligent Vehicles, vol. 8, no. 1, pp. 232–249, 2022

    2022

  8. [16]

    Systematic modeling ap- proach for environmental perception limitations in automated driving,

    A. Adee, R. Gansch, and P. Liggesmeyer, “Systematic modeling ap- proach for environmental perception limitations in automated driving,” in2021 17th European Dependable Computing Conference (EDCC). IEEE, 2021, pp. 103–110

    2021

  9. [17]

    Decomposition and quan- tification of sotif requirements for perception systems of autonomous vehicles,

    R. Yu, C. Wang, Y . Sui, and Y . Zhang, “Decomposition and quan- tification of sotif requirements for perception systems of autonomous vehicles,”Available at SSRN 4598460, 2025

    2025

  10. [18]

    Online quantitative analysis of perception uncertainty based on high- definition map,

    M. Yang, X. Jiao, K. Jiang, Q. Cheng, Y . Yang, M. Yang, and D. Yang, “Online quantitative analysis of perception uncertainty based on high- definition map,”Sensors, vol. 23, no. 24, p. 9876, 2023

    2023

  11. [19]

    Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing,

    V . Agarwal, R. Shetty, and M. Fritz, “Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9690–9698

    2020

  12. [20]

    Drivellm: Charting the path toward full autonomous driving with large language models,

    Y . Cui, S. Huang, J. Zhong, Z. Liu, Y . Wang, C. Sun, B. Li, X. Wang, and A. Khajepour, “Drivellm: Charting the path toward full autonomous driving with large language models,”IEEE Transactions on Intelligent Vehicles, 2023. 13

    2023

  13. [21]

    Driving with llms: Fusing object- level vector modality for explainable autonomous driving,

    L. Chen, O. Sinavski, J. H ¨unermann, A. Karnsund, A. J. Willmott, D. Birch, D. Maund, and J. Shotton, “Driving with llms: Fusing object- level vector modality for explainable autonomous driving,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 20...

    2024

  14. [22]

    Vlm-mpc: Vision language foundation model (vlm)-guided model predictive controller (mpc) for autonomous driving,

    K. Long, H. Shi, J. Liu, and X. Li, “Vlm-mpc: Vision language foundation model (vlm)-guided model predictive controller (mpc) for autonomous driving,”arXiv preprint arXiv:2408.04821, 2024

    2024 arXiv

  15. [23]

    Scene understanding for autonomous driving using visual question answering,

    A. Wantiez, T. Qiu, S. Matthes, and H. Shen, “Scene understanding for autonomous driving using visual question answering,” in2023 International Joint Conference on Neural Networks (IJCNN). IEEE, 2023, pp. 1–7

    2023

  16. [24]

    Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario,

    T. Qian, J. Chen, L. Zhuo, Y . Jiao, and Y .-G. Jiang, “Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 5, 2024, pp. 4542–4550

    2024

  17. [25]

    Explaining autonomous driving actions with visual question answering,

    S. Atakishiyev, M. Salameh, H. Babiker, and R. Goebel, “Explaining autonomous driving actions with visual question answering,” in2023 IEEE 26th International Conference on Intelligent Transportation Sys- tems (ITSC). IEEE, 2023, pp. 1207–1214

    2023

  18. [26]

    Gpt-4v (ision) is a generalist web agent, if grounded,

    B. Zheng, B. Gou, J. Kil, H. Sun, and Y . Su, “Gpt-4v (ision) is a generalist web agent, if grounded,”arXiv preprint arXiv:2401.01614, 2024

    2024 arXiv

  19. [27]

    Multi-frame, lightweight & efficient vision-language models for question answering in autonomous driving,

    A. Gopalkrishnan, R. Greer, and M. Trivedi, “Multi-frame, lightweight & efficient vision-language models for question answering in autonomous driving,”arXiv preprint arXiv:2403.19838, 2024

    2024 arXiv

  20. [28]

    Drama: Joint risk localization and captioning in driving,

    S. Malla, C. Choi, I. Dwivedi, J. H. Choi, and J. Li, “Drama: Joint risk localization and captioning in driving,” inProceedings of the IEEE/CVF winter conference on applications of computer vision, 2023, pp. 1043– 1052

    2023

  21. [29]

    Referring multi-object tracking,

    D. Wu, W. Han, T. Wang, X. Dong, X. Zhang, and J. Shen, “Referring multi-object tracking,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 14 633–14 642

    2023

  22. [30]

    Microsoft coco captions: Data collection and evaluation server,

    X. Chen, H. Fang, T.-Y . Lin, R. Vedantam, S. Gupta, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco captions: Data collection and evaluation server,”arXiv preprint arXiv:1504.00325, 2015

    2015 arXiv

  23. [31]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering,

    Y . Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the v in vqa matter: Elevating the role of image understanding in visual question answering,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 6904–6913

    2017

  24. [32]

    Cider: Consensus- based image description evaluation,

    R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus- based image description evaluation,” inProceedings of the IEEE confer- ence on computer vision and pattern recognition, 2015, pp. 4566–4575

    2015

  25. [33]

    Spice: Semantic propositional image caption evaluation,

    P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” inComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14. Springer, 2016, pp. 382–398

    2016

  26. [34]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xinget al., “Judging llm-as-a-judge with mt-bench and chatbot arena,”Advances in Neural Information Processing Systems, vol. 36, pp. 46 595–46 623, 2023

    2023

  27. [35]

    Judge anything: Mllm as a judge across any modality,

    S. Pu, Y . Wang, D. Chen, Y . Chen, G. Wang, Q. Qin, Z. Zhang, Z. Zhang, Z. Zhou, S. Gong, Y . Gui, Y . Wan, and P. S. Yu, “Judge anything: Mllm as a judge across any modality,” 2025. [Online]. Available: https://arxiv.org/abs/2503.17489

    2025 arXiv

  28. [36]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” inInternational Conference on Machine Learning. PMLR, 2022, pp. 12 888–12 900

    2022

  29. [37]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” inInternational conference on machine learning. PMLR, 2023, pp. 19 730–19 742

    2023

  30. [38]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” 2023

    2023

  31. [39]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Geet al., “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,”arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  32. [40]

    Qwen2. 5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tanget al., “Qwen2. 5-vl technical report,”arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  33. [41]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,

    J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y . Duan, W. Su, J. Shaoet al., “Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,”arXiv preprint arXiv:2504.10479, 2025

    2025 arXiv

  34. [42]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  35. [43]

    LA VIS: A one-stop library for language-vision intelligence,

    D. Li, J. Li, H. Le, G. Wang, S. Savarese, and S. C. Hoi, “LA VIS: A one-stop library for language-vision intelligence,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). Toronto, Canada: Association fo...

    2023

  36. [44]

    Llamafactory: Unified efficient fine-tuning of 100+ language models,

    Y . Zheng, R. Zhang, J. Zhang, Y . Ye, Z. Luo, Z. Feng, and Y . Ma, “Llamafactory: Unified efficient fine-tuning of 100+ language models,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). Bangkok, Thai...

    2024 arXiv

  37. [45]

    TensorRT-LLM,

    NVIDIA, “TensorRT-LLM,” Jan. 2024. [Online]. Available: github. com/NVIDIA/TensorRT-LLM

    2024

  38. [46]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  39. [47]

    Towards human-centric autonomous driving: A fast-slow architecture integrating large language model guidance with reinforcement learning,

    C. Xu, J. Liu, Y . Guo, Y . Zhang, P. Hang, and J. Sun, “Towards human-centric autonomous driving: A fast-slow architecture integrating large language model guidance with reinforcement learning,” 2025. [Online]. Available: https://arxiv.org/abs/2505.06875

    2025 arXiv

  40. [48]

    Vlmplanner: Integrating visual language models with motion planning,

    Z. Tang, S. Zhang, J. Deng, C. Wang, G. You, Y . Huang, X. Lin, and Y . Zhang, “Vlmplanner: Integrating visual language models with motion planning,” 2025. [Online]. Available: https://arxiv.org/abs/2507.20342

    2025 arXiv

  41. [49]

    Canadian adverse driving conditions dataset,

    M. Pitropov, D. E. Garcia, J. Rebello, M. Smart, C. Wang, K. Czarnecki, and S. Waslander, “Canadian adverse driving conditions dataset,”The International Journal of Robotics Research, vol. 40, no. 4-5, pp. 681– 690, 2021

    2021

  42. [50]

    Safe rlhf: Safe reinforcement learning from human feedback,

    J. Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y . Wang, and Y . Yang, “Safe rlhf: Safe reinforcement learning from human feedback,”arXiv preprint arXiv:2310.12773, 2023

    2023 arXiv

  43. [51]

    Siren’s song in the ai ocean: a survey on hallucination in large language models,

    Y . Zhang, Y . Li, L. Cui, D. Cai, L. Liuet al., “Siren’s song in the ai ocean: a survey on hallucination in large language models,”arXiv preprint arXiv:2309.01219, 2023

    2023 arXiv

  44. [52]

    Retrieval- augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschelet al., “Retrieval- augmented generation for knowledge-intensive nlp tasks,”Advances in Neural Information Processing Systems, vol. 33, pp. 9459–9474, 2020

    2020

  45. [53]

    Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback,

    H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. Lu, C. Bishop, E. Hall, V . Carbune, A. Rastogi, and S. Prakash, “Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback,” 2024

    2024

  46. [54]

    Reducing hallucinations in vision-language models via latent space steering,

    S. Liu, H. Ye, and J. Zou, “Reducing hallucinations in vision-language models via latent space steering,”arXiv preprint arXiv:2410.15778, 2024. Shucheng Huangreceived the B.S. degree in me- chanical engineering from Pennsylvania State Uni- versity, State College, USA, in 2018,...

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.