REVIEW 21 cited by
AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Robotic manipulation in open-world settings requires not only task execution but also the ability to detect and learn from failures. While recent advances in vision-language models (VLMs) and large language models (LLMs) have improved robots' spatial reasoning and problem-solving abilities, they still struggle with failure recognition, limiting their real-world applicability. We introduce AHA, an open-source VLM designed to detect and reason about failures in robotic manipulation using natural language. By framing failure detection as a free-form reasoning task, AHA identifies failures and provides detailed, adaptable explanations across different robots, tasks, and environments. We fine-tuned AHA using FailGen, a scalable framework that generates the first large-scale dataset of robotic failure trajectories, the AHA dataset. FailGen achieves this by procedurally perturbing successful demonstrations from simulation. Despite being trained solely on the AHA dataset, AHA generalizes effectively to real-world failure datasets, robotic systems, and unseen tasks. It surpasses the second-best model (GPT-4o in-context learning) by 10.3% and exceeds the average performance of six compared models including five state-of-the-art VLMs by 35.3% across multiple metrics and datasets. We integrate AHA into three manipulation frameworks that utilize LLMs/VLMs for reinforcement learning, task and motion planning, and zero-shot trajectory generation. AHA's failure feedback enhances these policies' performances by refining dense reward functions, optimizing task planning, and improving sub-task verification, boosting task success rates by an average of 21.4% across all three tasks compared to GPT-4 models.
Forward citations
Cited by 21 Pith papers
-
ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
ContactGuard uses an action-conditioned latent world model to predict a post-contact visual embedding from a planned action chunk and abort before the gripper closes, improving failure prediction over current-latent a...
-
ValueFormer: A Causal Transformer Value Function with Stage-Aware Labels for Semi-Autonomous Vision-Language-Action Policies
ValueFormer trains a causal transformer on stage-aware success-then-decay Monte Carlo labels to give robot policies a per-frame progress value and binary mistake alert in one forward pass.
-
Learning Panorama-Aware VLA for Mobile Manipulation with Whole-Body Teleoperation
Adding a panoramic camera feed to a vision-language-action policy raises end-to-end success on four real-world mobile two-arm tasks from 30% to 73%.
-
Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
A taxonomy of robot learning on a weights-versus-skills axis, with a five-rung self-improvement ladder whose top cell (feedback plus memory plus search) holds only a few recent systems.
-
LabRobFail: A Benchmark for Robotic Failure Analysis in Chemical Self-driving Laboratory
A chemical-lab failure sim, 20K-trajectory dataset, six-axis benchmark, and specialized VLM raise failure detection to 90.8% on seen scenes and lift downstream policy success by 4–16 points.
-
FORGE-plus: Force-Budgeted Recovery for Contact-Rich Assembly with a Frozen LLM Supervisor
With a hidden per-episode breaking force, an LLM-set force ceiling plus force-signature recovery achieves 256/256 clean insertions on fragile and robust parts and resolves 40–64% of injected jams in simulation.
-
From Sign Language Generation to Humanoid Execution: Vision-Language Guided Retargeting with Collision Mitigation
A humanoid signing pipeline that first cleans SMPL-X self-collisions and then uses a VLM visual critic to refine IK retargeting; collision energy drops on 9 sequences, but VLM benefits are only qualitatively demonstrated.
-
TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics
The log-probability a VLM assigns to 'True' for 'does this video prefix complete the task?' is used as a zero-shot dense progress reward that outperforms GVL on open-source models.
-
World-VLA-Loop: Closed-Loop Learning of Video World Model and VLA Policy
World-VLA-Loop alternately fine-tunes a video world model and a VLA policy, using RL inside the simulator to boost real-world success rates by up to 36.7 percentage points over two iterations.
-
FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models
A VLM trained on auto-generated failure trajectories with executable correction actions helps VLA models detect and fix manipulation errors, lifting success rates by up to 22.6 percentage points.
-
CRAFT: Coaching Reinforcement Learning Autonomously using Foundation Models for Multi-Robot Coordination Tasks
An LLM/VLM coaching loop that generates curricula and reward functions enabled MARL agents to learn coordinated gate passing, seesaw balancing, and bimanual pot lifting, with one policy transferred to real quadrupeds.
-
CorrectNav: Self-Correction Flywheel Empowers Vision-Language-Action Navigation Model
By iteratively retraining on automatically generated corrective examples derived from its own wrong paths, CorrectNav reports new state-of-the-art success rates of 65.1% (R2R-CE) and 69.3% (RxR-CE).
-
Unsupervised Discovery of Failure Taxonomies from Deployment Logs
An unsupervised pipeline converts robot failure videos into natural language explanations, clusters them into recurring failure types, and uses those types to guide data collection and runtime monitoring.
-
Humanoid World Models: Open World Foundation Models for Humanoid Robotics
Masked-transformers trained on humanoid video forecast future frames with better FID than flow-matching models, and parameter sharing cut model size 33-53% with minimal quality loss.
-
A Real-to-Sim-to-Real Approach to Robotic Manipulation with VLM-Generated Iterative Keypoint Rewards
IKER uses VLM-generated keypoint rewards to train manipulation policies in simulation that transfer to a real robot, enabling multi-step tasks and replanning.
-
SAFECAST: Robust Failure Detection for VLA Policies with Contrast-Set Training and Calibration
SAFECAST augments hidden-state failure-probe training and conformal calibration with visual and language contrast sets, improving VLA failure detection under distribution shift in several tested settings.
-
GeoManip: Geometric Constraints as General Interfaces for Robot Manipulation
GeoManip uses large vision-language models to turn task descriptions into geometric constraints and cost functions, then solves for robot trajectories without training, reporting state-of-the-art success rates on simu...
-
Enhanced Vision-Language Models for Diverse Sensor Understanding: Cost-Efficient Optimization and Benchmarking
A small dataset of sensor images plus positive and negative answer examples markedly improves VLM performance on thermal, depth, and X-ray understanding without retraining the model architecture.
-
3D-CDRGP: Towards Cross-Device Robotic Grasping Policy in 3D Open World
A training-free cross-device robotic grasping policy uses background-class softmax and geometry clustering to detect and grasp open-world objects.
-
Data Pyramid for Embodied Manipulation: A Survey
Embodied training data form a five-layer pyramid—real-robot, UMI, ego/exo, simulation, general V–L—ordered by the trade-off between scale and robot alignment, and model capabilities track how those layers are mixed.
-
A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
A survey that catalogs large vision-language models, their alignment methods, benchmarks, and challenges, but is compromised by inconsistent counts and misclassified entries.
Discussion (0). Continue with ORCID to comment.