Pith. sign in

REVIEW 15 cited by

HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Driving

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.05186 v2 pith:G4PKEYNA submitted 2023-09-11 cs.CV

Xinpeng Ding , Jianhua Han , Hang Xu , Wei Zhang , Xiaomeng Li This is my paper · ORCID
classification cs.CV
keywords mllmsperceptiondrivinghilm-dinformationvisualautonomoushigh-resolution
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Recent efforts to use natural language for interpretable driving focus mainly on planning, neglecting perception tasks. In this paper, we address this gap by introducing ROLISP (Risk Object Localization and Intention and Suggestion Prediction), which towards interpretable risk object detection and suggestion for ego car motions. Accurate ROLISP implementation requires extensive reasoning to identify critical traffic objects and infer their intentions, prompting us to explore the capabilities of multimodal large language models (MLLMs). However, the limited perception performance of CLIP-ViT vision encoders in existing MLLMs struggles with capturing essential visual perception information, e.g., high-resolution, multi-scale and visual-related inductive biases, which are important for autonomous driving. Addressing these challenges, we introduce HiLM-D, a resource-efficient framework that enhances visual information processing in MLLMs for ROLISP. Our method is motivated by the fact that the primary variations in autonomous driving scenarios are the motion trajectories rather than the semantic or appearance information (e.g., the shapes and colors) of objects. Hence, the visual process of HiLM-D is a two-stream framework: (i) a temporal reasoning stream, receiving low-resolution dynamic video content, to capture temporal semantics, and (ii) a spatial perception stream, receiving a single high-resolution frame, to capture holistic visual perception-related information. The spatial perception stream can be made very lightweight by a well-designed P-Adapter, which is lightweight, training-efficient, and easily integrated into existing MLLMs. Experiments on the DRAMA-ROLISP dataset show HiLM-D's significant improvements over current MLLMs, with a 3.7% in BLEU-4 for captioning and 8.7% in mIoU for detection.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VADER: Adaptive Debiasing for Hallucination Mitigation in Video Large Language Models

    cs.CV 2026-08 conditional novelty 7.0 of 10

    A training-free decoding framework that adaptively reweights attention toward video tokens and erases key visual evidence per frame to suppress hallucinated predictions, achieving 72.60% accuracy on EventHallusion wit...

  2. Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sink

    cs.LG 2025-01 conditional novelty 7.0 of 10

    Adversarial images optimized to induce attention sink behavior increase hallucination rates in multiple MLLMs, including commercial APIs, without visibly degrading response quality.

  3. CoopReflect: Towards Natural Language Communication for Cooperative Autonomous Driving via Multi-Agent Learning

    cs.RO 2025-05 conditional novelty 6.0 of 10

    Post-episode multi-agent debriefing lets LLM driving agents learn concise natural-language coordination protocols that avoid collisions and merge traffic, and distillation makes the policy fast enough for near-real-time use.

  4. Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Driving

    cs.CV 2025-05 conditional novelty 6.0 of 10

    DriveMonkey, which bridges a vision-language model to a pre-trained 3D detector with learnable queries, improves 3D visual grounding mAP by 9.86 percentage points over general LVLMs on the new NuInteract benchmark.

  5. Distilling Multi-modal Large Language Models for Autonomous Driving

    cs.CV 2025-01 conditional novelty 6.0 of 10

    DiMA jointly trains a vision-only planner with an LLM and uses auxiliary language, reconstruction, and scene editing tasks to improve planning on nuScenes while dropping the LLM at inference.

  6. Embodied Scene Understanding for Vision Language Models via MetaVQA

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Fine-tuning on the auto-generated MetaVQA VQA corpus improves VLMs' spatial reasoning accuracy and partially improves their closed-loop driving safety in simulation.

  7. MLLM-SUL: Multimodal Large Language Model for Semantic Scene Understanding and Localization in Traffic Scenarios

    cs.CV 2024-12 conditional novelty 6.0 of 10

    The paper proposes MLLM-SUL, an image-based multimodal language model that jointly generates driving-scene captions and localizes risk objects, reporting state-of-the-art scores on DRAMA-ROLISP and an extended DRAMA-S...

  8. SAFLITE: Fuzzing Autonomous Systems via Large Language Models

    cs.SE 2024-12 conditional novelty 6.0 of 10

    An LLM-based test-case relevance scorer, SaFliTe, increases the number of collision and policy-violation test cases found by several drone fuzzing tools.

  9. [CLS] Token Tells Everything Needed for Training-free Efficient MLLMs

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Using the CLIP [CLS] token's attention to prune image tokens before the LLM runs outperforms prior training-free pruning methods on LLaVA-1.5-7B and speeds up inference.

  10. Task-Oriented Semantic Communication in Large Multimodal Models-based Vehicle Networks

    cs.AI 2025-05 conditional novelty 5.0 of 10

    A task-oriented semantic communication framework for LLaVA-based vehicle assistants that crops image slices matched to the user's question and allocates transmission power by fused objective and subjective attention, ...

  11. H-MBA: Hierarchical MamBa Adaptation for Multi-Modal Video Understanding in Autonomous Driving

    cs.CV 2025-01 conditional novelty 5.0 of 10

    A hierarchical Mamba adapter (C-Mamba and Q-Mamba) improves multimodal LLM video understanding in autonomous driving, achieving SOTA 66.9% mIoU on DRAMA risk localization.

  12. World knowledge-enhanced Reasoning Using Instruction-guided Interactor in Autonomous Driving

    cs.CV 2024-12 conditional novelty 5.0 of 10

    An instruction-guided token selection and cross-attention module improves MLLM performance on autonomous driving QA and planning benchmarks, trained with a new GPT-generated object-level risk assessment dataset.

  13. SD++: Enhancing Standard Definition Maps by Incorporating Road Knowledge using LLMs

    cs.RO 2025-02 conditional novelty 4.0 of 10

    SD++ enhances OpenStreetMap road centerlines by extracting lane and shoulder parameters from road manuals with LLMs and generating lane geometry algorithmically.

  14. Visual Large Language Models for Generalized and Specialized Applications

    cs.CV 2025-01 conditional novelty 3.0 of 10

    This paper reviews and taxonomizes VLLM applications into vision-to-text, vision-to-action, and text-to-vision, adding ethics and future-work discussion.

  15. A Novel MLLM-based Approach for Autonomous Driving in Different Weather Conditions

    cs.RO 2024-11 reject novelty 3.0 of 10

    A closed-loop study in CARLA/LimSim++ evaluating a GPT-4o prompt-based driving agent across five weather conditions and several camera/LiDAR configurations.

Pith tools