Pith. sign in

REVIEW 19 cited by

Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.07865 v2 pith:K7VE3ENF submitted 2024-02-12 cs.CV cs.AIcs.CLcs.LG

Siddharth Karamcheti , Suraj Nair , Ashwin Balakrishna , Percy Liang , Thomas Kollar , Dorsa Sadigh This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.LG
keywords vlmsmodelsdesignevaluationslanguagevisualadoptionchallenge
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

Visually-conditioned language models (VLMs) have seen growing adoption in applications such as visual dialogue, scene understanding, and robotic task planning; adoption that has fueled a wealth of new models such as LLaVa, InstructBLIP, and PaLI-3. Despite the volume of new releases, key design decisions around image preprocessing, architecture, and optimization are under-explored, making it challenging to understand what factors account for model performance $-$ a challenge further complicated by the lack of objective, consistent evaluations. To address these gaps, we first compile a suite of standardized evaluations spanning visual question answering, object localization, and challenge sets that probe properties such as hallucination; evaluations that provide fine-grained insight VLM capabilities. Second, we rigorously investigate VLMs along key design axes, including pretrained visual representations and training from base vs. instruct-tuned language models, amongst others. We couple our analysis with three resource contributions: (1) a unified framework for evaluating VLMs, (2) optimized, flexible training code, and (3) checkpoints for all models, including a family of VLMs at the 7-13B scale that strictly outperform InstructBLIP and LLaVa v1.5, the state-of-the-art in open VLMs.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 19 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. An Exam for Active Observers

    cs.CV 2026-07 conditional novelty 7.0 of 10

    On a new 17-task benchmark of active visual observation, the best frontier multimodal model solves 10.6% of items and humans solve 96.1%.

  2. RoboVista: Evaluating Vision Language Models for Diverse Robot Applications

    cs.RO 2026-07 accept novelty 6.5 of 10

    Expert-curated modular Robot-VQA benchmark of 474 questions across 39 robot tasks shows SOTA VLMs have large gaps that correlate with physical robot execution.

  3. Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

    cs.RO 2026-07 unverdicted novelty 6.0 of 10

    A memory-guided LLM planner composes a frozen VLA as a contact-rich primitive with fixed analytic controllers, lifting perturbed manipulation success without VLA finetuning.

  4. SQAP-VLA: A Synergistic Quantization-Aware Pruning Framework for High-Performance Vision-Language-Action Models

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A training-free framework jointly quantizes a VLA model to 4 bits and prunes visual tokens, recovering or exceeding full-precision success rates at 1.93x speedup.

  5. Hidden in plain sight: VLMs overlook their visual representations

    cs.CV 2025-06 conditional novelty 6.0 of 10

    VLMs perform far worse than their own visual encoders on vision-centric tasks because the language model fails to use accessible visual information and instead follows its language priors.

  6. Diffusion Instruction Tuning

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Lavender fine-tunes vision-language models by aligning their attention maps with Stable Diffusion's attention targets, improving accuracy on 20 benchmarks with as few as 0.13 million training examples.

  7. Mordal: Automated Pretrained Model Selection for Vision Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Mordal automates the selection of pretrained vision encoder and LLM pairs for VLMs, using representation clustering and partial-training scaling predictions to cut search cost by about an order of magnitude.

  8. Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Eagle2-9B matches or outperforms much larger vision-language models on many benchmarks through a carefully constructed post-training data strategy.

  9. NoisyEQA: Benchmarking Embodied Question Answering Against Noisy Queries

    cs.CV 2024-12 conditional novelty 6.0 of 10

    NoisyEQA benchmarks embodied QA agents against four noise types and claims a self-correcting prompt (NACoT) markedly improves noisy-question accuracy as scored by GPT-4.

  10. How to Merge Your Multimodal Models Over Time?

    cs.LG 2024-12 conditional novelty 6.0 of 10

    A systematic study of temporal model merging shows that initialization and deployment choices matter far more than the merging technique, with EMA-style weight interpolation as the best practice.

  11. Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Robotics

    cs.RO 2024-11 conditional novelty 6.0 of 10

    Adversarial patches optimized against OpenVLA raise manipulation failure rates from about 23% to 100% in LIBERO simulation and disrupt a physical robot arm in 43% of trials.

  12. Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Structural pruning with finetuning plus hidden-state distillation recovers most performance in multimodal LLMs, with 5% of training data sufficient at moderate compression levels.

  13. EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models

    cs.CV 2025-06 conditional novelty 5.0 of 10

    EfficientVLA combines LLM layer pruning, task-aware visual token selection, and diffusion-head feature caching to cut CogACT's inference cost to 28.9% of baseline FLOPs with a 0.6% SIMPLER success drop.

  14. Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoning

    cs.RO 2025-06 conditional novelty 5.0 of 10

    FiS-VLA embeds a diffusion-based action module into the final transformer blocks of a vision-language model, achieving 69% mean success on RLBench and a claimed 117.7 Hz control frequency.

  15. Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A new patch-aligned pretraining loss improves fine-grained vision-language alignment and grounding in multimodal LLMs.

  16. An Atomic Skill Library Construction Method for Data-Efficient Embodied Manipulation

    cs.RO 2025-01 conditional novelty 5.0 of 10

    A data-driven framework decomposes manipulation tasks into reusable atomic skills, fine-tunes a VLA model per skill, and reports reduced data needs with comparable or better real-robot success rates.

  17. LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering

    cs.CV 2024-12 conditional novelty 5.0 of 10

    A frozen-LLM method that steers visual token representations in low-rank subspaces achieves benchmark scores close to LoRA with about 500x fewer trainable parameters.

  18. StemVLA:An Open-Source Vision-Language-Action Model with Future 3D Spatial Geometry Knowledge and 4D Historical Representation

    cs.RO 2026-02 reject novelty 4.0 of 10

    StemVLA supervises a GPT-2-based VLA with predicted future 3D-geometry features (VGGT) and temporally aggregated history, reporting 86.0% on LIBERO-Long - but its CALVIN results and equations are placeholders.

  19. Expand VSR Benchmark for VLLM to Expertize in Spatial Rules

    cs.CV 2024-12 conditional novelty 4.0 of 10

    The authors expand the VSR benchmark with 50 question templates and diffusion-augmented images, fuse four vision encoders, and report a VSR-specialist VLLM (VSRE) that substantially outperforms LLaVA1.5 on spatial rea...

Pith tools