Pith. sign in

REVIEW 5 cited by

Benchmarking Vision, Language, & Action Models on Robotic Learning Tasks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.05821 v2 pith:CFJUE25W submitted 2024-11-04 cs.RO cs.CVcs.LG

Pranav Guruprasad , Harshvardhan Sikka , Jaewoo Song , Yangyue Wang , Paul Pu Liang This is my paper
classification cs.ROcs.CVcs.LG
keywords modelstasksperformanceroboticacrossactionevaluationdemonstrating
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Vision-language-action (VLA) models represent a promising direction for developing general-purpose robotic systems, demonstrating the ability to combine visual understanding, language comprehension, and action generation. However, systematic evaluation of these models across diverse robotic tasks remains limited. In this work, we present a comprehensive evaluation framework and benchmark suite for assessing VLA models. We profile three state-of-the-art VLM and VLAs - GPT-4o, OpenVLA, and JAT - across 20 diverse datasets from the Open-X-Embodiment collection, evaluating their performance on various manipulation tasks. Our analysis reveals several key insights: 1. current VLA models show significant variation in performance across different tasks and robot platforms, with GPT-4o demonstrating the most consistent performance through sophisticated prompt engineering, 2. all models struggle with complex manipulation tasks requiring multi-step planning, and 3. model performance is notably sensitive to action space characteristics and environmental factors. We release our evaluation framework and findings to facilitate systematic assessment of future VLA models and identify critical areas for improvement in the development of general purpose robotic systems.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization

    cs.CV 2025-10 conditional novelty 6.0 of 10

    SOTA VLA models like OpenVLA and pi0 collapse when object positions are perturbed, indicating that standard LIBERO scores reward memorization; LIBERO-PRO provides a systematic perturbed evaluation suite.

  2. Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A dexterous VLA pretrained on a 2.5M-instance human hand motion dataset transfers skills to a real robot hand, outperforming baselines in manipulation tasks.

  3. HRIBench: Benchmarking Vision-Language Models for Real-Time Human Perception in Human-Robot Interaction

    cs.RO 2025-06 conditional novelty 6.0 of 10

    HRIBench is a new 1,000-question VQA benchmark for five HRI perception domains; state-of-the-art vision-language models are neither accurate enough nor fast enough for real-time human-robot interaction.

  4. BadVLA: Towards Backdoor Attacks on Vision-Language-Action Models via Objective-Decoupled Optimization

    cs.CR 2025-05 conditional novelty 6.0 of 10

    A two-stage, objective-decoupled training method embeds visual backdoors into VLA robot policies, achieving near-100% trigger-induced task failure with minimal clean-performance loss in simulation.

  5. An Open-Source Software Toolkit & Benchmark Suite for the Evaluation and Adaptation of Multimodal Action Models

    cs.LG 2025-06 conditional novelty 4.0 of 10

    MultiNet provides an open-source benchmark, data SDK, evaluation harness, and adapted VLA models for assessing generalization across vision, language, and action tasks.

Pith tools