Pith. sign in

REVIEW 9 cited by

VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Driving

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.15544 v1 pith:O2HFSJPR submitted 2024-12-20 cs.RO cs.AIcs.CV

Zilin Huang , Zihao Sheng , Yansong Qu , Junwei You , Sikai Chen This is my paper
classification cs.ROcs.AIcs.CV
keywords drivingrewardvlm-rllanguagelearningrewardsautonomousextensive
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In recent years, reinforcement learning (RL)-based methods for learning driving policies have gained increasing attention in the autonomous driving community and have achieved remarkable progress in various driving scenarios. However, traditional RL approaches rely on manually engineered rewards, which require extensive human effort and often lack generalizability. To address these limitations, we propose \textbf{VLM-RL}, a unified framework that integrates pre-trained Vision-Language Models (VLMs) with RL to generate reward signals using image observation and natural language goals. The core of VLM-RL is the contrasting language goal (CLG)-as-reward paradigm, which uses positive and negative language goals to generate semantic rewards. We further introduce a hierarchical reward synthesis approach that combines CLG-based semantic rewards with vehicle state information, improving reward stability and offering a more comprehensive reward signal. Additionally, a batch-processing technique is employed to optimize computational efficiency during training. Extensive experiments in the CARLA simulator demonstrate that VLM-RL outperforms state-of-the-art baselines, achieving a 10.5\% reduction in collision rate, a 104.6\% increase in route completion rate, and robust generalization to unseen driving scenarios. Furthermore, VLM-RL can seamlessly integrate almost any standard RL algorithms, potentially revolutionizing the existing RL paradigm that relies on manual reward engineering and enabling continuous performance improvements. The demo video and code can be accessed at: https://zilin-huang.github.io/VLM-RL-website.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Steadily moving semi-infinite fracture in plane poroelasticity

    physics.geo-ph 2026-04 unverdicted novelty 7.0 of 10

    XEmbodied achieves SOTA on 18 embodied VQA benchmarks by fusing 3D geometric tokens and distilled physical cues into a 30B VLM with progressive curriculum training.

  2. AnySlot: Goal-Conditioned Vision-Language-Action Policies for Zero-Shot Slot-Level Placement

    cs.RO 2026-04 unverdicted novelty 6.0 of 10

    AnySlot decouples language slot selection from control via an explicit visual goal marker and reports large zero-shot gains on SlotBench, but the provided body text does not match the abstract.

  3. AutoNeural: Co-Designing Vision-Language Models for NPU Inference

    cs.CL 2025-12 conditional novelty 6.0 of 10

    A NPU-native VLM combining a MobileNet-style encoder with a hybrid Transformer-SSM backbone claims 14x lower latency and 7x lower quantization error over ViT-Transformer baselines, though quantized accuracy is not reported.

  4. One Object, Multiple Lies: A Benchmark for Cross-task Adversarial Attack on Unified Vision-Language Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A single adversarial image can make a unified vision-language model misclassify the same object across captioning, detection, region classification, and localization, and the new CrossVLAD benchmark and CRAFT attack m...

  5. SEAL: Vision-Language Model-Based Safe End-to-End Cooperative Autonomous Driving with Adaptive Long-Tail Modeling

    cs.RO 2025-06 reject novelty 6.0 of 10

    SEAL extends a V2X vision-language driving model with GPT-4o-generated snow/fog data, gated scenario attention, and contrastive learning, reporting improved planning accuracy on synthetic long-tail tests.

  6. ROAD: Responsibility-Oriented Reward Design for Reinforcement Learning in Autonomous Driving

    cs.LG 2025-05 reject novelty 6.0 of 10

    A responsibility-aware crash penalty, built from a traffic-law knowledge graph and a vision-language blame classifier, improves MetaDrive success rates and shifts reported collision blame away from the ego vehicle.

  7. Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidance

    cs.AI 2025-08 reject novelty 4.0 of 10

    A lightweight vision-language model on an edge device fuses roadside hazard alerts with onboard camera views to adjust trajectories, and the authors report a 77% simulated collision reduction over a vision-only baseline.

  8. A Survey on Vision-Language-Action Models for Autonomous Driving

    cs.CV 2025-06 conditional novelty 4.0 of 10

    A survey organizes vision-language-action models for autonomous driving into four stages, compares over 20 systems, and catalogs datasets, benchmarks, and open challenges.

  9. Simulating the Unseen: Crash Prediction Must Learn from What Did Not Happen

    cs.LG 2025-05 conditional novelty 4.0 of 10

    Crash prediction should learn from near-miss events and synthetic counterfactual scenarios, not just recorded crashes.

Pith tools