Pith. sign in

REVIEW 5 major objections 9 minor 59 references

Controllable synthetic lab failures can train a vision-language supervisor that detects, localizes, and corrects robotic errors well enough to raise closed-loop task success.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 15:19 UTC pith:PAB4NTA5

load-bearing objection Solid lab-robotics failure benchmark with real engineering value; headline Seen numbers are soft from leakage risk, but Unseen and recovery results still carry weight. the 5 major comments →

arxiv 2607.23704 v2 pith:PAB4NTA5 submitted 2026-07-26 cs.RO cs.CV

LabRobFail: A Benchmark for Robotic Failure Analysis in Chemical Self-driving Laboratory

Haobo Wang , Baoli Sun , Anqi Zou , Dongsheng Huang , Zelin Lv , Ning Wang , Rui Li , Dongzhan Zhou
show 3 more authors
Weiyu Guo Zhihui Wang Wanli Ouyang
This is my paper
classification cs.RO cs.CV
keywords robotic failure analysisself-driving laboratoriesvision-language modelsfailure injectionclosed-loop recoverychemical laboratory roboticstemporal localizationseverity assessment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Self-driving chemical laboratories need robots that can catch and fix mistakes, because experiments are often irreversible and safety-critical. Progress has been blocked by scarce failure data and by benchmarks that only ask for coarse detection. This paper builds a simulator that injects failures at the control, physics, and task-logic levels, yielding more than 20,000 trajectories across 70-plus lab scenarios, five failure families, and eleven fine-grained types. It pairs that data with a six-part evaluation of task understanding, detection, temporal localization, severity, classification, and actionable correction, and trains a specialized vision-language model on the resulting structured labels. On familiar settings the model substantially beats general-purpose vision-language models on detection and frame-level localization, and when used as a real-time supervisor it improves downstream manipulation success by several percentage points.

Core claim

A domain-specialized vision-language model trained on multi-level synthetic laboratory failures can produce structured diagnoses—presence, frame, type, severity, and recovery instructions—that outperform general-purpose models on lab failure analysis and, when wired as an external supervisor, raise closed-loop task success rates by 4 to 16 percentage points on seen settings.

What carries the argument

LabRobFail-Sim’s multi-level failure generator: systematic perturbations of control signals (pose and gripper noise), physics parameters (friction, mass, viscosity scaling), and semantic structure (step order and target swaps), which produce labeled failure trajectories for structured training and six-dimensional evaluation.

Load-bearing premise

Failures made by control noise, physics scaling, and step rearrangements in simulation are representative enough of real chemical-lab failures that the reported detection accuracy and recovery gains will still matter on physical hardware.

What would settle it

Mount the same supervisor on a physical lab robot, inject real instances of the paper’s failure types (object slip, wrong step order, incomplete door close, mis-aimed pour), and measure whether detection, frame localization, and the 4–16 point success lift still beat unadapted general vision-language models.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Fine-grained labels (frame, type, severity, correction) can drive recovery without full policy retraining when mapped through a fixed action dictionary.
  • Paired success–failure trajectories under matched conditions sharpen the boundary between normal and anomalous lab motion.
  • Six-dimensional lab benchmarks make binary household-style failure detection insufficient as a standard.
  • Domain-adapted vision-language supervisors can lift success on hard lab skills such as pour and oven open/close for existing manipulation policies.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Closing the stated sim-to-real visual gap is the main gate between this benchmark and routine safety layers in irreversible self-driving labs.
  • The five-category taxonomy (perception, grasp, motion, logic, safety) is a natural template for other high-stakes embodied settings with irreversible steps.
  • Replacing the deterministic action dictionary with policies that consume open-ended correction language would test whether structured diagnosis can drive recovery end-to-end.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 9 minor

Summary. The paper introduces LabRobFail, a failure-centric framework for robotic manipulation in chemical self-driving laboratories. It consists of (i) LabRobFail-Sim, which extends the LabUtopia simulator with automated failure injection at control (Gaussian/Lie-algebra keyframe noise, gripper faults), physics (uniform scaling of friction/mass/viscosity), and semantic (sequence permutation, target substitution) levels; (ii) LabRobFail-Data, 20K+ trajectories over 70+ task scenarios with paired success/failure rollouts, annotated by a GPT-5.4 VQA pipeline driven by simulation metadata; (iii) LabRobFail-Bench, a six-question evaluation (task decomposition, detection, temporal localization, severity, classification, correction); and (iv) LabRobFail-VLM, a hybrid fine-tuned Qwen3-VL-8B (full FT of vision tower + LoRA on the LLM) over a spatio-temporal keyframe grid. On the Seen split it reports 90.83% detection and 77.21% localization accuracy, large margins over general-purpose VLMs, retained but degraded performance under object/scene shifts, and 4–16pp downstream success gains when used as a supervisor for OpenVLA/ACT via a deterministic Action Dictionary.

Significance. If the evaluation issues are addressed, this is a useful contribution to an underserved niche: the first large-scale failure-centric dataset/benchmark for chemical laboratory manipulation, with automated multi-level failure injection, paired success/failure contrastive data, a six-dimensional evaluation protocol going beyond binary detection (localization, severity, correction), and demonstrated downstream recovery gains on two VLA policies. The authors also ship an unseen-environment generalization split, ablations of the hybrid fine-tuning design, a transfer experiment to AHA, and a public code/data release, and they report the compound-shift degradation and sim-only limitation honestly. The benchmark itself is likely to be used by the SDL/embodied-AI community even if the specific VLM results are revised.

major comments (5)
  1. [Experiment Details; Table 2] Experiment Details (Supplementary), 'We split LabRobFail-Data into training, validation, and test sets with a ratio of 8:1:1... For the Seen subset, we employ random splitting': this split is not adequate for the headline claim. LabRobFail-Data is explicitly 'paired contrastive data' — success and failure trajectories generated under identical conditions from only 70+ task configurations for 20K+ trajectories (§LabRobFail-Data). Random trajectory-level splitting therefore places near-duplicates of each test episode (same scene, objects, task, nominal keyframe sequence, differing only in the injected perturbation) into the training set, and the labels themselves are deterministic functions of the same injection metadata used to generate the trajectories (Φctrl/Φphy/Φsem + rule-based filtering 'consistent with the simulation metadata'). The Seen-split numbers in Table 2 (90.83% Q2, 77.21%
  2. [Table 2 (Q6); Supplementary Table 9] Q6 evaluation (Table 2, Q6 columns; Table 9): the correction ground truth is drawn from a fixed set of template instructions (Table 9 lists the complete set, with placeholders filled from task context), and the metric is BLEU-n/ROUGE-L against those references. A model fine-tuned on this distribution can achieve the reported 0.74 BLEU-4 largely by identifying the failure type and emitting the memorized template — the score is closer to template classification than to evidence of 'actionable, fine-grained correction' as claimed in the Abstract and §LabRobFail-Bench. Compounding this, GPT-5.4 both generates the reference corrections (Supplementary, VQA Generation Module) and appears as a baseline in Table 2, so the comparison partially measures distillation of the annotation pipeline. Please add at least one of: (a) an execution-based evaluation of corrections (you already have the Action
  3. [Evaluation Metrics; Table 2 (Q3)] Q3 metric definition (§Experiment, Evaluation Metrics; Table 2, Q3): 'Top-1 Accuracy' for temporal localization is never operationalized. Is exact frame-index match required, or is a tolerance window (e.g., ±1 keyframe) used? This matters because general-purpose baselines score 5–16% — near zero — which suggests a protocol mismatch (e.g., baselines failing to emit valid frame indices, or not receiving the same temporally indexed keyframe grid) rather than a pure capability gap. Please state: the exact matching criterion, whether all baselines received the identical rendered grid with temporal indices (if not, the comparison conflates input representation with model capability), and chance-level accuracy given the M×K grid so readers can calibrate the 77.21% figure.
  4. [Downstream Policy Recovery; Table 5] Table 5 (downstream policy recovery): each cell is n=25 trials with no variance, confidence interval, or significance statement; with n=25, the +4pp cells (e.g., Pick/ACT, 64%→68%) are well within binomial noise. The reporting convention is also unclear: '21/25' — are these 25 episodes with injected failures only, or mixed success/failure episodes? If mixed, what happens when the supervisor fires on a successful trajectory (false-positive cost is never reported)? Given the supervisor's corrections are produced by the same taxonomy/Action-Dictionary pipeline that defines the failures, the closed-loop gains are partly by construction; a false-positive-rate analysis and at least Wilson confidence intervals (or more trials for the marginal cells) are needed before the '4–16 percentage points' claim in the Abstract is well supported.
  5. [§LabRobFail-Bench (Q4); Supplementary Table 8] Q4 severity dimension (§LabRobFail-Bench L3 vs. Supplementary Table 8): the main text defines four ordinal severity levels (Minor, Recoverable, Critical, Catastrophic), but Table 8's actual annotation options are '(1) Risk Level (2) Dangerous Level (3) Fatal Level (4) Management Incident' with definitions that are not ordinal — option (4) is a compliance category ('unclosed cabinet'), not a severity grade, and options (1)–(3) are defined by event type (collision/spillage vs. equipment damage) rather than by a consistent severity scale. Q4 'accuracy' therefore conflates event-type recognition with severity assessment, and the two descriptions of the label space contradict each other. Please reconcile the taxonomy (either make Q4 genuinely ordinal with a stated rubric for assigning levels, or rename the dimension to reflect what is actually measured) and clarify which label set the Table 2
minor comments (9)
  1. [Supplementary, VQA Generation Module] Inconsistent naming of the annotation model: 'GPT-5.4', 'GPT5.4', and 'GPT-5.4 Thinking' appear interchangeably (e.g., §LLM-driven semantic annotation vs. Supplementary). Please standardize, and state the exact model version and access date, since annotation quality depends on it.
  2. [Supplementary, VQA Generation Module] The system prompt tells the annotator to answer 'based on the visual observation and metadata', but it is not stated whether GPT-5.4 actually receives the keyframe-grid image as input or only the text metadata. This distinction matters for how much of the label is visually grounded vs. metadata-derived; please clarify.
  3. [Supplementary, LLM Annotation Pipeline] The human-inspection step in the annotation validation is described only as 'a random sample... is manually inspected'. Please report the sample size/fraction, the rejection rate, and inter-annotator procedure, so readers can estimate label noise.
  4. [Figure 3] Figure 3(b): the per-type trajectory counts along the x-axis are unreadable in the rendered figure, and the normal/abnormal balance per task level is hard to extract. Please enlarge or tabulate these counts in the supplementary.
  5. [§LabRobFail-Data; §LabRobFail-Sim] PF category definition (§LabRobFail-Data, Fine-grained Failure Taxonomy): 'failures caused by transparent glassware or reflective fluid surfaces' is listed as a failure type, but this is an environmental condition, not a failure event — the injection pipeline (§LabRobFail-Sim) does not describe perturbing optical properties. Please clarify how Perception failures are actually injected, since Φctrl/Φphy/Φsem do not obviously cover them.
  6. [Ablation Studies; Table 4] Ablation (Table 4): the 'Frozen vision' variant is described as 'freezing both the vision encoder and projector', which contradicts the text's phrasing 'frozen vision' vs. the hybrid strategy — please make the three ablation variants' trainable parameter sets explicit (a small table of ΘV/ΘL trainable flags would suffice). Also report variance over seeds for the ablation; the 6.63-point Q3 drop attributed to temporal indices is within plausible seed noise for a single run.
  7. [Table 1] Table 1: the column alignment in the '# Eval Dims / # Traj.' entries is confusing (LabRobFail row renders as '11 6 20K'), and RLBench-Fail/BridgeDataV2-Fail/UR5-Fail are all attributed to the same Guardian citation (Pacaud et al. 2025) without distinguishing them. Please clean up the table formatting and clarify provenance.
  8. [Abstract] The code/data link (github.com/Su-ISE-2001/SciRobo) uses a repository name unrelated to the paper title; please confirm this is the intended release repository and state a release timeline in the camera-ready.
  9. [§LabRobFail-Bench, Eq. (3); §LabRobFail-Sim] Notation: Eq. (3) uses y_risk for severity while Q4 is called 'Severity Assessment' and Table 8 uses 'Risk Level'; y_dete/yloca abbreviations are nonstandard. Consider consistent naming (y_sev, y_det, y_loc). Also 'trajectory˜τ = Φsem(Φctrl(τ))' omits Φphy from the composition despite the text saying the trajectory is 'executed under perturbed dynamics Φphy(Ψ)' — please make the composition consistent.

Circularity Check

0 steps flagged

No derivation-chain circularity: empirical sim benchmark with metadata GT and fine-tuned VLM metrics, not inputs renamed as predictions.

full rationale

LabRobFail is an empirical systems/benchmark paper, not a first-principles derivation. Failures are produced by explicit multi-level injection (Φctrl/Φphy/Φsem) on LabUtopia; detection/localization/type labels are simulation metadata (standard synthetic GT), not quantities defined as the model outputs they later claim to predict. LabRobFail-VLM is supervised on that labeled set and scored against held-out trajectories and external generalist VLMs; downstream gains are measured as policy success-rate deltas under an Action Dictionary supervisor. GPT-5.4 template annotation and the presence of GPT-5.4 as a zero-shot baseline create a mild labeler–eval family resemblance on semantic Q4/Q6 text, and random Seen splits of paired rollouts raise leakage/validity questions—but those are evaluation-soundness issues, not self-definitional reductions, fitted parameters rebranded as predictions, load-bearing self-cited uniqueness theorems, or renamed known laws. No step reduces a claimed prediction to its inputs by construction. Score 0.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 3 invented entities

The central empirical claims rest on accepting LabUtopia-class physics as a stand-in for lab contact/liquid dynamics, treating scripted multi-level perturbations as coverage of real failure modes, trusting GPT-5.4+rules as annotation ground truth for structured diagnoses, and treating BLEU/ROUGE alignment to templated corrections plus dictionary-mapped primitives as evidence of actionable recovery. Hyperparameters and perturbation scales are chosen by authors; no new physical entities are postulated.

free parameters (5)
  • Control perturbation covariances Σ and gripper fault probability λ_grip = Not numerically reported; ξ ~ N(0, Σ), λ_grip unspecified
    Hand-chosen noise scales for Φ_ctrl that define how often and how severely pose/gripper failures appear in LabRobFail-Data.
  • Physics scaling range α for Ψ={μ,m,ν,...} = δ ~ U(-α, α); α not given a single global value in text
    Uniform multiplicative perturbation magnitude that induces slippage and related physics failures; directly shapes GF/MF prevalence.
  • Hybrid fine-tuning hyperparameters = as in Implementation Details
    Vision LRs 2e-6 / 1e-5, LoRA r=64 α=128 dropout 0.05, LM LR 1e-4, 3 epochs, batch 256—chosen to produce reported accuracies.
  • Keyframe grid size M×K and action-transition sampling = grid resolution 1536×768; M,K task-dependent
    Determines what temporal evidence the VLM sees for localization; critical to Q3 performance.
  • Severity rubric bins L1–L4 / Risk–Management options = 4 levels (Table 8)
    Author-defined discrete severity ontology used as classification targets for Q4; not externally standardized.
axioms (6)
  • domain assumption LabUtopia-level simulated RGB multi-view trajectories plus rigid/soft contact and liquid approximations are adequate proxies for chemical-lab manipulation perception and failure cues.
    Entire dataset and bench are built on this sim stack (LabRobFail-Sim section).
  • ad hoc to paper Failures factor into control-, physics-, and semantic-level perturbations that can be injected independently and labeled from metadata.
    Core generator design Φ_ctrl, Φ_phy, Φ_sem; real failures may entangle these levels.
  • ad hoc to paper The five-category / eleven-type taxonomy (PF/GF/MF/LF/SF and subtypes) is an appropriate complete-enough ontology for lab robotic failures.
    Defines Q5 targets and data balance (Fig. 3b).
  • domain assumption GPT-5.4 VQA instantiation from sim metadata, after rule filters and light human sampling, yields reliable structured answers for training and metrics.
    LLM-driven semantic annotation pipeline; residual label noise unquantified.
  • ad hoc to paper Lexical overlap (BLEU/ROUGE) with reference correction templates plus deterministic Action Dictionary execution measures actionable recovery quality.
    Q6 protocol and Table 5 supervisor setup; open-language execution is left to future work.
  • domain assumption Standard VLM transfer learning assumptions: full FT of vision + LoRA on language preserves reasoning while adapting to glassware/liquid cues.
    Hybrid fine-tuning strategy and Eq. (6).
invented entities (3)
  • LabRobFail-Sim multi-level Failure Generator (Φ_ctrl/Φ_phy/Φ_sem) no independent evidence
    purpose: Scalable automated synthesis of annotated lab failure trajectories.
    Engineering construct; validated only by internal dataset statistics and downstream metrics, not external failure corpora.
  • LabRobFail-Bench six-dimensional composite diagnostic state Y no independent evidence
    purpose: Unified evaluation target for task parse, detect, localize, severity, type, correction.
    New benchmark schema; usefulness shown only within this paper’s splits.
  • LabRobFail-VLM structured supervisor + Action Dictionary no independent evidence
    purpose: Map video+instruction to recovery primitives for closed-loop policy help.
    System composition; gains reported in-sim on OpenVLA/ACT only.

pith-pipeline@v1.2.0-grok45-kimik3 · 22063 in / 4217 out tokens · 87894 ms · 2026-07-30T15:19:20.843705+00:00 · methodology

0 comments
read the original abstract

The deployment of embodied agents in self-driving laboratories could accelerate scientific discovery, yet their reliability is constrained by the irreversible and safety-critical nature of chemical experiments. Progress is further hindered by scarce failure data and the lack of fine-grained evaluation protocols. To address these challenges, we introduce LabRobFail, a failure-centric framework for learning and evaluating robotic failure analysis in chemical laboratories. LabRobFail-Sim injects controllable failures at the control, physics, and semantic levels, enabling the construction of LabRobFail-Data, which contains over 20,000 trajectories across 70+ task scenarios, five failure categories, and 11 fine-grained failure types. LabRobFail-Bench evaluates six capabilities spanning task understanding, failure detection, temporal localization, severity assessment, failure classification, and actionable correction. We further develop LabRobFail-VLM, a domain-specialized vision-language model that generates structured failure diagnoses and recovery instructions. On seen environments, it achieves 90.83% failure-detection accuracy and 77.21% temporal-localization accuracy, substantially outperforming general-purpose VLMs. When integrated as a real-time supervisor, it improves downstream task success rates by 4-16 percentage points, demonstrating the value of fine-grained failure understanding for closed-loop recovery and reliable laboratory autonomy. Our code and data are available at https://github.com/Su-ISE-2001/SciRobo

Figures

Figures reproduced from arXiv: 2607.23704 by Anqi Zou, Baoli Sun, Dongsheng Huang, Dongzhan Zhou, Haobo Wang, Ning Wang, Rui Li, Wanli Ouyang, Weiyu Guo, Zelin Lv, Zhihui Wang.

Figure 1
Figure 1. Figure 1: LabRobFail Framework for Chemical Laboratory Failure Analysis. A high-fidelity simulation platform automat￾ically injects failures across five categories: Perception (PF), Grasping (GF), Motion (MF), Logic (LF), and Safety (SF). The resulting data support a six-dimensional benchmark for failure detection, localization, diagnosis, and fine-grained correction. Abstract The deployment of embodied agents in se… view at source ↗
Figure 2
Figure 2. Figure 2: The LabRobFail-Sim Framework. (a) The multi-level failure generator injects faults via Control, Physics, and Semantic perturbations. (b) The LLM-driven semantic annotation pipeline leverages simulation metadata and GPT-5.4 to automatically generate instruction-answer pairs. Sh. Task: Atomic Actions Mh. Task: Chained Actions Lh. Task: Hierarchical Planning (b) close pick stir pre ss 29.30% place open 26.12%… view at source ↗
Figure 3
Figure 3. Figure 3: LabRobFail-Data and LabRobFail-Bench. (a) Distribution of short-, medium-, and long-horizon tasks, correspond￾ing to atomic actions, chained actions, and hierarchical planning. (b) Distribution of normal and failed trajectories across tasks (left), and 11 fine-grained failure types grouped into five categories (right). (c) Six-dimensional benchmark covering task un￾derstanding, failure detection and locali… view at source ↗
Figure 4
Figure 4. Figure 4: Overview of LabRobFail-VLM. Temporally indexed multi-view keyframes are processed by a hybrid-tuned Qwen3- VL to jointly perform failure detection, localization, diagnosis, and correction. types ( [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Examples of unseen objects and scenes. 2024) and ACT (Zhao et al. 2023). It generates fine-grained corrections for arm motion, gripper state, and execution con￾ditions, which a deterministic Action Dictionary converts into control primitives without additional fine-tuning. Ta￾ble 5 shows gains of 4–16 percentage points across all eight settings, with the largest improvement on Pour under ACT (32% to 48%). … view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of Protocol Violation failure. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Visualization of Sequence Reversal failure. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Visualization of Pose Control Error failure. [PITH_FULL_IMAGE:figures/full_fig_p012_10.png] view at source ↗
Figure 12
Figure 12. Figure 12: Visualization of Gripper Control Failure. [PITH_FULL_IMAGE:figures/full_fig_p013_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Visualization of Object Slippage failure. [PITH_FULL_IMAGE:figures/full_fig_p013_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Visualization of Target Positioning Deviation fail [PITH_FULL_IMAGE:figures/full_fig_p013_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Visualization of Operation Position Error failure. [PITH_FULL_IMAGE:figures/full_fig_p013_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

59 extracted references · 14 linked inside Pith

  1. [1]

    Langley , title =

    P. Langley , title =. Proceedings of the 17th International Conference on Machine Learning (ICML 2000) , address =. 2000 , pages =

    2000

  2. [2]

    T. M. Mitchell. The Need for Biases in Learning Generalizations. 1980

    1980

  3. [3]

    M. J. Kearns , title =

  4. [4]

    Machine Learning: An Artificial Intelligence Approach, Vol. I. 1983

    1983

  5. [5]

    R. O. Duda and P. E. Hart and D. G. Stork. Pattern Classification. 2000

    2000

  6. [6]

    Suppressed for Anonymity , author=

  7. [7]

    Newell and P

    A. Newell and P. S. Rosenbloom. Mechanisms of Skill Acquisition and the Law of Practice. Cognitive Skills and Their Acquisition. 1981

    1981

  8. [8]

    A. L. Samuel. Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development. 1959

    1959

  9. [9]

    Advanced Materials , volume=

    Artificial chemist: an autonomous quantum dot synthesis bot , author=. Advanced Materials , volume=. 2020 , publisher=

    2020

  10. [10]

    Science , volume=

    Organic synthesis in a modular robotic system driven by a chemical programming language , author=. Science , volume=. 2019 , publisher=

    2019

  11. [17]

    IEEE Robotics and Automation Letters , volume=

    Rlbench: The robot learning benchmark & learning environment , author=. IEEE Robotics and Automation Letters , volume=. 2020 , publisher=

    2020

  12. [18]

    Conference on Robot Learning , pages=

    Bridgedata v2: A dataset for robot learning at scale , author=. Conference on Robot Learning , pages=. 2023 , organization=

    2023

  13. [21]

    2024 IEEE International Conference on Robotics and Automation (ICRA) , pages=

    A multimodal handover failure detection dataset and baselines , author=. 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages=. 2024 , organization=

    2024

  14. [25]

    Conference on Robot Learning , pages=

    Rt-2: Vision-language-action models transfer web knowledge to robotic control , author=. Conference on Robot Learning , pages=. 2023 , organization=

    2023

  15. [26]

    Black, Kevin and Brown, Noah and Darpinian, James and Dhabalia, Karan and Driess, Danny and Esmail, Adnan and Equi, Michael and Finn, Chelsea and Fusai, Niccolo and others , journal=

  16. [27]

    Nature , volume=

    An autonomous laboratory for the accelerated synthesis of novel materials , author=. Nature , volume=. 2023 , publisher=

    2023

  17. [28]

    Nature Synthesis , volume=

    The rise of self-driving labs in chemical and materials sciences , author=. Nature Synthesis , volume=. 2023 , publisher=

    2023

  18. [30]

    Forty-first International Conference on Machine Learning , year=

    Position: LLMs can’t plan, but can help planning in LLM-modulo frameworks , author=. Forty-first International Conference on Machine Learning , year=

  19. [31]

    ICML 2024 Workshop on Structured Probabilistic Inference \ & \ Generative Modeling , year=

    Recursive introspection: Teaching LLM agents how to self-improve , author=. ICML 2024 Workshop on Structured Probabilistic Inference \ & \ Generative Modeling , year=

    2024

  20. [32]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Code-as-monitor: Constraint-aware visual programming for reactive and proactive robotic failure detection , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  21. [33]

    2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages=

    Doremi: Grounding language model by detecting and recovering from plan-execution misalignment , author=. 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages=. 2024 , organization=

    2024

  22. [34]

    arXiv preprint arXiv:2303.07280 , year=

    Vision-language models as success detectors , author=. arXiv preprint arXiv:2303.07280 , year=

    Pith/arXiv arXiv

  23. [36]

    Liu, Haotian and Li, Chunyuan and Li, Yuheng and Li, Bo and Zhang, Yuanhan and Shen, Sheng and Lee, Yong Jae , year=

  24. [39]

    2026 , howpublished=

    2026

  25. [40]

    arXiv preprint arXiv:2403.05530 , year=

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context , author=. arXiv preprint arXiv:2403.05530 , year=

    Pith/arXiv arXiv

  26. [41]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  27. [42]

    Hu, Edward J and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , booktitle=

  28. [43]

    Proceedings of Robotics: Science and Systems , year=

    Learning fine-grained bimanual manipulation with low-cost hardware , author=. Proceedings of Robotics: Science and Systems , year=

  29. [45]

    Bai, S.; Cai, Y.; Chen, R.; Chen, K.; Chen, X.; Cheng, Z.; Deng, L.; Ding, W.; Gao, C.; Ge, C.; et al. 2025. Qwen3-VL Technical Report. arXiv preprint arXiv:2511.21631

    Pith/arXiv arXiv 2025

  30. [46]

    Black, K.; Brown, N.; Darpinian, J.; Dhabalia, K.; Driess, D.; Esmail, A.; Equi, M.; Finn, C.; Fusai, N.; et al. 2025. _ 0.5 : A Vision-Language-Action Model with Open-World Generalization. arXiv preprint arXiv:2504.16054

    Pith/arXiv arXiv 2025

  31. [47]

    Chen, Z.; Wang, W.; Cao, Y.; Liu, Y.; Gao, Z.; Cui, E.; Zhu, J.; Ye, S.; Tian, H.; Liu, Z.; et al. 2024. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling. arXiv preprint arXiv:2412.05271

    Pith/arXiv arXiv 2024

  32. [48]

    C.; Jacobsson, H.; Szpektor, I.; Jiang, N.-J.; Haridasan, K.; Omran, A.; et al

    Comanici, G.; Bieber, E.; Schaekermann, M.; Pasupat, I.; Sachdeva, N.; Dhillon, I.; Blistein, M.; Ram, O.; Zhang, D.; Rosen, E.; Marris, L.; Petulla, S.; Gaffney, C.; Aharoni, A.; Lintz, N.; Pais, T. C.; Jacobsson, H.; Szpektor, I.; Jiang, N.-J.; Haridasan, K.; Omran, A.; et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality...

    Pith/arXiv arXiv 2025

  33. [49]

    R.; Tian, S.; Yuan, W.; Krishna, R.; Fox, D.; Mandlekar, A.; and Guo, Y

    Duan, J.; Pumacay, W.; Kumar, N.; Wang, Y. R.; Tian, S.; Yuan, W.; Krishna, R.; Fox, D.; Mandlekar, A.; and Guo, Y. 2024. Aha: A vision-language-model for detecting and reasoning over failures in robotic manipulation. arXiv preprint arXiv:2410.00371

    Pith/arXiv arXiv 2024

  34. [50]

    W.; Bowen, M

    Epps, R. W.; Bowen, M. S.; Volk, A. A.; Abdel-Latif, K.; Han, S.; Reyes, K. G.; Amassian, A.; and Abolhasani, M. 2020. Artificial chemist: an autonomous quantum dot synthesis bot. Advanced Materials, 32(30): 2001626

    2020

  35. [51]

    Grislain, C.; Rahimi, H.; Sigaud, O.; and Chetouani, M. 2025. I-FailSense: Towards General Robotic Failure Detection with Vision-Language Models. arXiv preprint arXiv:2509.16072

    arXiv 2025

  36. [52]

    Guo, Y.; Wang, Y.-J.; Zha, L.; and Chen, J. 2024. Doremi: Grounding language model by detecting and recovering from plan-execution misalignment. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 12124--12131. IEEE

    2024

  37. [53]

    J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

    Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022. LoRA : Low-rank adaptation of large language models. In International Conference on Learning Representations

    2022

  38. [54]

    R.; and Davison, A

    James, S.; Ma, Z.; Arrojo, D. R.; and Davison, A. J. 2020. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2): 3019--3026

    2020

  39. [55]

    P.; and Murthy, A

    Kambhampati, S.; Valmeekam, K.; Guan, L.; Verma, M.; Stechly, K.; Bhambri, S.; Saldyt, L. P.; and Murthy, A. B. 2024. Position: LLMs can’t plan, but can help planning in LLM-modulo frameworks. In Forty-first International Conference on Machine Learning

    2024

  40. [56]

    J.; Pertsch, K.; Karamcheti, S.; Xiao, T.; Balakrishna, A.; Nair, S.; Rafailov, R.; Foster, E.; Lam, G.; Sanketi, P.; et al

    Kim, M. J.; Pertsch, K.; Karamcheti, S.; Xiao, T.; Balakrishna, A.; Nair, S.; Rafailov, R.; Foster, E.; Lam, G.; Sanketi, P.; et al. 2024. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246

    Pith/arXiv arXiv 2024

  41. [57]

    Lan, Z.; Jiang, Y.; Wang, R.; Xie, X.; Zhang, R.; Zhu, Y.; Li, P.; Yang, T.; Chen, T.; Gao, H.; et al. 2025. Autobio: A simulation and benchmark for robotic automation in digital biology laboratory. arXiv preprint arXiv:2505.14030

    Pith/arXiv arXiv 2025

  42. [58]

    Li, R.; Hu, Z.; Qu, W.; Zhang, J.; Yin, Z.; Zhang, S.; Huang, X.; Wang, H.; Wang, T.; Pang, J.; et al. 2025. LabUtopia: High-Fidelity Simulation and Hierarchical Benchmark for Scientific Embodied Agents. arXiv preprint arXiv:2505.22634

    arXiv 2025

  43. [59]

    Lin, J.; Du, Y.; Watkins, O.; Hafner, D.; Abbeel, P.; Klein, D.; and Dragan, A. 2023. Learning to model the world with language. arXiv preprint arXiv:2308.01399

    Pith/arXiv arXiv 2023

  44. [60]

    Liu, H.; Li, C.; Li, Y.; Li, B.; Zhang, Y.; Shen, S.; and Lee, Y. J. 2024. LLaVA-NeXT : Improved reasoning, OCR , and world knowledge. https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  45. [61]

    Liu, Z.; Bahety, A.; and Song, S. 2023. Reflect: Summarizing robot experiences for failure explanation and correction. arXiv preprint arXiv:2306.15724

    Pith/arXiv arXiv 2023

  46. [62]

    Lu, W.; Ye, M.; Ye, Z.; Tao, R.; Yang, S.; and Zhao, B. 2025. RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correction. arXiv preprint arXiv:2505.12224

    arXiv 2025

  47. [63]

    Mitash, C.; Wang, F.; Lu, S.; Terhuja, V.; Garaas, T.; Polido, F.; and Nambi, M. 2023. Armbench: An object-centric benchmark dataset for robotic manipulation. arXiv preprint arXiv:2303.16382

    Pith/arXiv arXiv 2023

  48. [64]

    OpenAI . 2026. GPT-5.4 Thinking system card. Technical report, OpenAI. https://openai.com/index/gpt-5-4-thinking-system-card/

    2026

  49. [65]

    Pacaud, P.; Garcia, R.; Chen, S.; and Schmid, C. 2025. Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models. arXiv preprint arXiv:2512.01946

    arXiv 2025

  50. [66]

    Qu, Y.; Zhang, T.; Garg, N.; and Kumar, A. 2024. Recursive introspection: Teaching LLM agents how to self-improve. In ICML 2024 Workshop on Structured Probabilistic Inference \ & \ Generative Modeling

    2024

  51. [67]

    M.; Keenan, G.; Hinkley, T.; Aragon-Camarasa, G.; Kitson, P

    Steiner, S.; Wolf, J.; Glatzel, S.; Andreou, A.; Granda, J. M.; Keenan, G.; Hinkley, T.; Aragon-Camarasa, G.; Kitson, P. J.; Angelone, D.; et al. 2019. Organic synthesis in a modular robotic system driven by a chemical programming language. Science, 363(6423): eaav2211

    2019

  52. [68]

    J.; Rendy, B.; Fei, Y.; Kumar, R

    Szymanski, N. J.; Rendy, B.; Fei, Y.; Kumar, R. E.; He, T.; Milsted, D.; McDermott, M. J.; Gallant, M.; Cubuk, E. D.; Merchant, A.; et al. 2023. An autonomous laboratory for the accelerated synthesis of novel materials. Nature, 624(7990): 86--91

    2023

  53. [69]

    Thoduka, S.; Hochgeschwender, N.; Gall, J.; and Pl \"o ger, P. G. 2024. A multimodal handover failure detection dataset and baselines. In 2024 IEEE International Conference on Robotics and Automation (ICRA), 17013--17019. IEEE

    2024

  54. [70]

    R.; Black, K.; Zhao, T

    Walke, H. R.; Black, K.; Zhao, T. Z.; Vuong, Q.; Zheng, C.; Hansen-Estruch, P.; He, A. W.; Myers, V.; Kim, M. J.; Du, M.; et al. 2023. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning, 1723--1736. PMLR

    2023

  55. [71]

    Wu, Z.; Chen, X.; Pan, Z.; Liu, X.; Liu, W.; Dai, D.; Gao, H.; Ma, Y.; Wu, C.; Wang, B.; et al. 2024. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302

    Pith/arXiv arXiv 2024

  56. [72]

    Zeng, X.; Zhou, X.; Li, Y.; Shi, J.; Li, T.; Chen, L.; Ren, L.; and Li, Y.-L. 2025. Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbols. arXiv preprint arXiv:2512.02787

    arXiv 2025

  57. [73]

    Z.; Kumar, V.; Levine, S.; and Finn, C

    Zhao, T. Z.; Kumar, V.; Levine, S.; and Finn, C. 2023. Learning fine-grained bimanual manipulation with low-cost hardware. In Proceedings of Robotics: Science and Systems

    2023

  58. [74]

    Zhen, H.; Qiu, X.; Chen, P.; Yang, J.; Yan, X.; Du, Y.; Hong, Y.; and Gan, C. 2024. 3d-vla: A 3d vision-language-action generative world model. arXiv preprint arXiv:2403.09631

    Pith/arXiv arXiv 2024

  59. [75]

    Zhou, E.; Su, Q.; Chi, C.; Zhang, Z.; Wang, Z.; Huang, T.; Sheng, L.; and Wang, H. 2025. Code-as-monitor: Constraint-aware visual programming for reactive and proactive robotic failure detection. In Proceedings of the Computer Vision and Pattern Recognition Conference, 6919--6929

    2025