Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

Fail2Progress claims that generating a small simulation dataset that reproduces a robot's real-world failure, then fine-tuning its skill effect model on that dataset, sharply reduces future failures in long-horizon manipulation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Fail2Progress generates failure-targeted simulation data via Stein variational inference and fine-tunes skill effect models, improving long-horizon manipulation success rates and generalizing to unseen object counts and viewpoints.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A genuinely useful failure-driven data generation method with honest limits; the main risk is that the failure-classification step rests on bounding-box Real2Sim fidelity that is never directly validated. the 4 major comments →

arxiv 2509.01746 v1 pith:7OWIGGEO submitted 2025-09-01 cs.RO

Fail2Progress: Learning from Real-World Robot Failures with Stein Variational Inference

Yixuan Huang , Novella Alvina , Mohanraj Devendran Shanthi , Tucker Hermans This is my paper
classification cs.RO
keywords Learning from failuresSkill effect modelsStein variational inferenceLong-horizon manipulationReal-to-simActive learningFailure classificationMobile manipulation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Fail2Progress claims that a robot can learn from its own real-world mistakes by generating a small, targeted simulation dataset that mirrors the failure, then fine-tuning its skill effect model on that data. The paper formalizes this as a variational inference problem: find simulator states that reproduce the failed relations and actions that most reduce the model's uncertainty, using Stein variational inference to keep the samples diverse. On a hierarchical tabletop organization task, a model refined this way succeeds in 86% of trials with the Points2Plans skill model, versus 11% without fine-tuning and 53% for the best of six baselines. The result is evidence that the quality and targeting of fine-tuning data, not its volume, is what lets long-horizon manipulation recover from out-of-distribution failures.

Core claim

The paper's central claim is that failures of learned skill effect models in real-world manipulation can be converted into a training signal: detect a symbolic prediction failure, classify it as distinct from a Sim2Real gap, generate a parallel set of low-fidelity simulation scenes that replay the failure relations and push the model toward informative actions, and fine-tune on that set. Fail2Progress frames dataset generation as a constrained optimization that maximizes expected information gain of the updated model subject to states matching the observed failure, then solves it by approximating two posteriors with Stein variational gradient descent: one over simulator states matching the f

What carries the argument

Stein variational inference (SVI) is the mechanism: a set of particles is evolved by Stein variational gradient descent so that the particles approximate a posterior distribution while a repulsive kernel term keeps them from collapsing to one mode. Fail2Progress runs SVI twice — first to place object poses so that rendered point clouds satisfy the relations observed at the moment of failure, then to choose skill parameters that maximize entropy of the current skill effect model's predictions. This dual use supplies both the 'similar to the failure' constraint and the 'informative for learning' objective, and the parallel particle updates are what make generation efficient.

Load-bearing premise

The bounding-box reconstruction of real scenes in simulation is accurate enough that failures are classified as symbolic-prediction errors rather than Sim2Real gaps, and the simplified simulated physics transfers to the real robot during fine-tuning.

What would settle it

Run Fail2Progress on a failure caused by a property the bounding-box scene cannot capture, such as a deformable object, a transparent object, or an object whose friction or center of mass dominates the skill outcome, and check whether the reconstructed simulation reproduces the observed failure relations. If the simulated effect matches the real effect but the fine-tuned model still fails at similar scenes, the Real2Sim approximation is the broken link; if the simulated effect diverges, the two-category failure classifier mislabels the failure as a Sim2Real gap.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A skill effect model fine-tuned on failure-conditioned generated data outperforms models given 100 times more unfiltered random data, so data selection can dominate data volume.
  • Replanning alone is not enough for long-horizon recovery when the dynamics model is wrong; updating the model itself is necessary.
  • SVI's particle diversity matters: sequential gradient descent and rejection sampling both lag behind in fine-tuned success and efficiency, especially as object count rises.
  • Fine-tuning on as few as 20 targeted simulation scenarios transfers to unseen object counts, shapes, and viewpoints, suggesting the method learns generalizable relational dynamics rather than memorizing failure scenes.
  • When point-cloud noise grows, performance degrades, showing the method's ceiling is set by the size of the Sim2Real gap and by how faithfully the reconstructed simulation captures the real scene.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because Fail2Progress only needs low-fidelity scenes, plugging it into a high-fidelity Real2Sim pipeline could improve failure classification for geometrically complex or deformable objects without changing the learning loop.
  • The same two-stage Stein formulation could be pointed at failures caused by Sim2Real gaps by treating simulator physics parameters, such as friction or center of mass, as latent states to infer alongside poses, letting the model learn from failures it currently discards.
  • The one-shot fine-tuning framing suggests an online loop: as failures are detected and classified, the system can accumulate targeted datasets across days, so success rates should rise until they hit the ceiling set by unmodeled disturbances.
  • Adding failure causes beyond the two-category scheme, such as other agents, hardware drift, or broken relations, would require enriching the failure event representation, but the variational dataset-generation core would survive.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper addresses the problem of improving learned skill effect models for long-horizon manipulation after deployment, specifically when the robot encounters out-of-distribution situations that cause symbolic-level failures. The proposed system, Fail2Progress, detects a failure when the predicted relational state after skill execution does not match the observed relations, classifies the failure as either an incorrect symbolic prediction or a Sim2Real gap by comparing real-world outcomes with outcomes in a bounding-box-based Real2Sim reconstruction, and—for failures attributed to incorrect symbolic predictions—generates a targeted simulation dataset using Stein variational inference (SVI). The dataset generation objective seeks states and actions that satisfy the failure relations while maximizing the current model's entropy, thereby acting as an active-learning scheme. The skill effect model is then fine-tuned on this dataset. Experiments are reported for three skill effect model architectures (Points2Plans, Stow-GNN, Binary-Pred) on hierarchical tabletop organization, multi-object transport, and constrained packing, with simulation results reporting large gains over baselines (e.g., 86% vs. 53% for Points2Plans on Hierarchical Tabletop Organization) and real-world experiments showing consistent improvements over Gradient and Sampling baselines. The paper also includes generalization tests and ablations on the number of SVI particles.

Significance. If the central claim holds, Fail2Progress makes a valuable contribution: it turns a single real-world failure into a small, targeted simulation dataset that can materially improve a deployed skill effect model, with 20 samples being sufficient in the reported tasks. The idea of formulating failure-driven data collection as variational inference and using SVI to obtain diverse, parallel samples is novel and well-motivated. The paper is also commendable for the breadth of its evaluation: three distinct skill effect models, 300-trial simulation conditions, generalization to unseen object counts and viewpoints, ablations, and real-world deployment. The authors are explicit about limitations, including the reliance on Real2Sim fidelity and the two-category failure classification scheme. However, the load-bearing claim depends on the correctness of the failure classification step, whose accuracy is never directly measured, and the reported quantitative comparisons are missing uncertainty quantification, which tempers enthusiasm for the otherwise large observed margins.

major comments (4)
  1. [Sec. 4 and Sec. 5.3] The failure classification step is load-bearing but is never directly validated. The robot classifies a failure as 'incorrect symbolic predictions' iff the bounding-box Real2Sim reconstruction reproduces the real-world relational outcome; otherwise it is attributed to a Sim2Real gap. This dichotomy is only valid if the low-fidelity reconstruction—cuboid/open-box shape priors, fixed friction and center-of-mass values—is faithful enough for the comparison to be correct. The paper acknowledges this risk (Sec. 5.3, Sec. 8) but never measures classification accuracy against ground-truth failure labels. End-to-end success rates (Fig. 3) and relation-detection F1=0.92 do not isolate the classification error. If a real Sim2Real-gap failure is misclassified as symbolic, the generated D+ conditions on the wrong failure mode and the claimed improvement for the intended class is not established; if
  2. [Tables 1, 3 and Fig. 3] The quantitative claims are reported without any measure of uncertainty. Table 1 reports aggregate success percentages from 300 trials per condition but no variance across seeds, no confidence intervals, and no statistical tests; Table 3 shows the same for per-object-count conditions. The real-world results in Fig. 3a are based on 10 trials per approach per object count with no error bars or tests, making it difficult to assess whether the differences among Fail2Progress, Gradient, and Sampling are reliable. The claim that Fail2Progress 'consistently outperforms' baselines would be substantially strengthened by reporting seed-level standard deviations, confidence intervals, or paired significance tests. This is particularly important because the main simulation result (e.g., 86% vs. 53%) is large but could still be affected by implementation choices or evaluation protocol.
  3. [Eq. 2, Eq. 3b, and Sec. 6 (Q6)] There is a self-referential element in the evaluation: the data-generation objective uses the current model's entropy and relation predictions to select states and actions, and the main evaluation measures success on the same failure distribution (e.g., Hierarchical Tabletop Organization) from which the failure was drawn. While Table 2 provides some generalization evidence, the test scenarios remain in the same task family and failure mode. The paper would be strengthened by a test on a mismatched failure mode (e.g., fine-tuning on failures from one relational structure and evaluating on a different relational structure) or by ablating the entropy term to show that the targeted failure-conditioning, rather than the model's own uncertainty, drives the gains. As written, the reader cannot fully rule out that the method is primarily exploiting the constraint in Eq. 3b, with the entropy obje
  4. [Sec. 5.1 and Sec. 5.3] The approximation in Eq. 3b replaces the true observation likelihood P(OF|S) with the rigid segment-wise transformation ξ(S)OF, and the subsequent Real2Sim construction uses only bounding-box dimensions and pre-defined physical parameters. This is a pragmatic choice, and the paper is transparent about it, but the validity of this approximation is never demonstrated. In particular, the claim that 'Real2Sim is accurate enough for our tasks' (Q7) is supported only by the end-to-end success rate and the noisy-input degradation in Fig. 3b. Please provide a quantitative measure of how often the simulated relational outcome R''_k matches the real-world outcome for a set of diverse real-world scenes, and how sensitive the classification results are to the fixed friction and center-of-mass parameters.
minor comments (5)
  1. [Eq. 3a and Eq. 6] The objective defined in Eq. 3a uses a product of entropies over r in RF', while Eq. 6 uses a sum of negative entropies inside an expectation with a product over r. Maximizing a product is equivalent to maximizing a sum only after log-transformation, so the notation is inconsistent. Please clarify whether the objective is a product, a sum, or a sum of logs, and make Eq. 6 consistent with Eq. 3a.
  2. [Appx. A.13.2, Eq. 7] The Stein update for action particles is written with ∇ ln H(Gamma(...)), but Eq. 6 minimizes −H(...). If the entropy can be zero or negative gradients are intended, the sign and the logarithm are unclear. Please align the gradient expression with the loss being optimized.
  3. [Appx. A.6, Table 3] The table includes a 'Base' column that is not defined in the main text or in the appendix. Please explain what 'Base' refers to and how it differs from 'Original'.
  4. [Fig. 3] The figure labels contain rendering artifacts (e.g., 'Fail2Progress' rendered as 'Fail2Progress'). Please regenerate the figure with clean fonts so the labels and the meaning of the two panels are unambiguous.
  5. [Appx. A.11.3, training details] The fine-tuning uses 200 epochs on only 20 samples. This is a very high epoch-to-sample ratio and could lead to overfitting. Please report the fine-tuning loss curve or validation performance, or justify the choice of 200 epochs.

Circularity Check

0 steps flagged

No significant circularity; simulator labels and external success metrics anchor the loop.

full rationale

Fail2Progress's derivation is not circular. The fine-tuning dataset D+ is labeled by IsaacGym simulation rollouts (s++ = f(s+, φF, a+)), not by the model's own predictions, so the loop is grounded in external physics. The Eq. 1a information-gain objective is approximated by the standard entropy heuristic in Eq. 3a; this is an active-learning approximation, not a hidden reuse of the evaluation target. The state constraint Eq. 3b uses the current model Γ to render and evaluate relations, but the actual samples are physical simulator states, and the same objective is used by the Gradient and Sampling baselines, isolating the SVI contribution. The central comparison (Table 1, Fig. 3) tests execution success on the target tasks, an external metric, and the generalization experiments (Table 2) test unseen object counts and viewpoints, providing non-circular evidence. The paper's own limitations (Sec. 5.3 and Sec. 8) concede that the bounding-box Real2Sim may cause incorrect failure classification and that classification accuracy is not directly measured; this is an unvalidated auxiliary assumption and a correctness risk, not a circular reduction. Self-citations to [2] and [19] refer to prior base models and standard SVGD implementation details, and are not load-bearing. No equation reduces to its input by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 7 axioms · 0 invented entities

The central claim depends on a chain of modeling choices: the active-learning proxy, the uniform state prior, the xi(S) point-cloud transformation in place of a full observation likelihood, and the bounding-box Real2Sim. The model parameters themselves are trained, not free constants, but the algorithmic choices above are not derived from first principles.

free parameters (5)
  • Number of SVI particles M = 20
    Selected via ablation on the Hierarchical Tabletop Organization task (Appx. A.5) using success rate as the criterion; the main results reuse this value, so the reported performance is contingent on this choice.
  • Boundary relation threshold epsilon_boundary = 0.1 m
    Hand-set threshold in the relation definition (Appx. A.9); affects the symbolic state used in failure detection and the constraint in Eq. 3b.
  • Bottom dimension threshold epsilon_bottom = 0.2 m
    Hand-set threshold in the relation definition (Appx. A.9).
  • Loss balancing constants a and b = a=12, b=5
    Fixed constants in the position loss (Appx. A.11.2), inherited from [2]; they shape model training but are not central to the failure-learning claim.
  • Fine-tuning epochs = 200
    Hyperparameter (Appx. A.11.2) used for all fine-tuning; chosen without a reported sweep.
axioms (7)
  • standard math SVGD converges to a valid approximation of the target posterior for the state and action distributions.
    The method relies on Stein variational gradient descent (Sec. 5.2, Eqs. 4-7) to approximate multi-modal posteriors; this is a background result from Liu and Wang 2016, not proved in the paper.
  • domain assumption Entropy of the current model is a valid proxy for expected information gain in active learning.
    Eq. 3a replaces the intractable expected information gain in Eq. 1a with the entropy of the current model's predictive distribution over the failed relations, citing standard active learning practice [54,55].
  • domain assumption The skill effect model Gamma is differentiable with respect to continuous action parameters and object poses.
    The SVGD updates (Eqs. 5 and 7) require gradients of Gamma w.r.t. states and actions; the paper does not discuss how these gradients are obtained through the point-cloud transformation xi(S) and the dynamics model, but assumes they exist.
  • domain assumption Segmented point clouds with correct semantics are available from the perception pipeline.
    Sec. 3 states the model requires segmented point clouds and assumes a perception pipeline with semantics and segmentation, using SAM and Grounding DINO.
  • domain assumption The low-fidelity bounding-box Real2Sim scene is accurate enough for failure classification and fine-tuning.
    Sec. 5.3 builds scenes from bounding boxes of object segments; the paper acknowledges this can cause incorrect failure classification and poor data generation (Sec. 5.3 and Limitations).
  • domain assumption Failure causes can be partitioned into incorrect symbolic predictions and Sim2Real gap, with no other causes.
    Sec. 4 classifies failures into two categories, assuming no hardware breakage or disturbance by other agents; the Limitations section lists these exclusions.
  • ad hoc to paper The transformation xi(S)OF, which rigidly transposes observed point cloud segments, is a sufficient stand-in for the full observation likelihood P(OF|S).
    Eq. 3b simplifies the constraint in Eq. 2 by searching over poses of the segmented point clouds instead of full object geometry; this is a tractability approximation specific to this paper.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Fail2Progress: Learning from Real-World Robot Failures with Stein Variational Inference." pith.science (2026). https://pith.science/paper/7OWIGGEO

@misc{pith2026250901746,
  author       = {Pith},
  title        = {Pith review of: Fail2Progress: Learning from Real-World Robot Failures with Stein Variational Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7OWIGGEO}},
  note         = {Machine review of arXiv:2509.01746}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Skill effect models for long-horizon manipulation tasks are prone to failures in conditions not covered by training data distributions. Therefore, enabling robots to reason about and learn from failures is necessary. We investigate the problem of efficiently generating a dataset targeted to observed failures. After fine-tuning a skill effect model on this dataset, we evaluate the extent to which the model can recover from failures and minimize future failures. We propose Fail2Progress, an approach that leverages Stein variational inference to generate multiple simulation environments in parallel, enabling efficient data sample generation similar to observed failures. Our method is capable of handling several challenging mobile manipulation tasks, including transporting multiple objects, organizing a constrained shelf, and tabletop organization. Through large-scale simulation and real-world experiments, we demonstrate that our approach excels at learning from failures across different numbers of objects. Furthermore, we show that Fail2Progress outperforms several baselines.

Figures

Figures reproduced from arXiv: 2509.01746 by Mohanraj Devendran Shanthi, Novella Alvina, Tucker Hermans, Yixuan Huang.

Figure 1
Figure 1. Figure 1: Overview of Failure Case Reasoning. Top: Based on failure cases due to incorrect symbolic predictions, our approach generates targeted, diverse simulation data in parallel to fine-tune the robot’s model.Bottom: The fine￾tuned model successfully performs diverse long-horizon manipulation tasks in challenging real-world scenarios. Skill-effects models predict the change in world state when running a skill gi… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of Fail2Progress. Our approach first detects the real-world failure and classifies it as incorrect symbolic predictions. Based on the real-world failure, Fail2Progress generates particles representing simulation states and actions to approximate posterior distributions. The state posterior distribution captures failure rela￾tions (e.g., objects inside the box), while the action posterior distribut… view at source ↗
Figure 4
Figure 4. Figure 4: Rollouts of real-world evaluations and corresponding failure cases. A detailed explanation of this figure is provided in Sec. A.1. 3objs (Seen) 5objs (Unseen) 4objs (Unseen) 6objs (Unseen) [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Real-world generalization visualizations. We show how Fail2Progress generalizes to different numbers of objects (3-6), different object shapes, and different tables. We present qualitative results in [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Efficiency experiments show that Fail2Progress is comparable to Gradient and more efficient than Sampling. The optimiza￾tion time is presented on a logarithmic scale. We compare Fail2Progress with the two best-performing baselines, Gradient and Sampling, to assess opti￾mization efficiency using the best-performing architec￾ture (Points2Plans). Error bars in the figure represent stan￾dard deviations across … view at source ↗
Figure 7
Figure 7. Figure 7: An ablation study for different par￾ticles. To determine the best fine-tuning dataset size, measured by the number of particles, we conduct an ablation study presented in [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Visualizations of simulation generalization scenarios. Fail2Progress, fine-tuned on a dataset with 3 objects, successfully generalizes to scenes with 5 and 7 objects. Additionally, Fail2Progress demonstrates generalization to two unseen viewpoints. A.15 Experimental Details We now provide further details regarding the datasets. We first report the failure relations satisfaction score, which measures the pe… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Distributionally Robust Control via Stein Variational Inference for Contact-Rich Manipulation

    cs.RO 2026-05 unverdicted novelty 6.0

    Introduces a Stein variational inference-based deterministic formulation for distributionally robust control in contact-rich robotic manipulation, reporting up to 3x improved robustness under parametric uncertainty.

  2. Distributionally Robust Control via Stein Variational Inference for Contact-Rich Manipulation

    cs.RO 2026-05 conditional novelty 6.0

    SV-DRO evolves parameter particles via task-optimality-gap Stein gradients inside DRO-MPC, yielding up to 3× higher success on contact-rich manipulation under parametric uncertainty.

  3. Grasp-Then-Plan with Failure Attribution: A Closed Two-Stage Framework for Precise and Generalizable Robotic Manipulation

    cs.RO 2026-06 unverdicted novelty 5.0

    GTP-FA is a grasp-then-plan framework with failure attribution that diagnoses errors to optimize grasping priors and planning data collection, raising success rates across RL, IL, diffusion, and VLA methods in simulat...

  4. Hierarchical DLO Routing with Reinforcement Learning and In-Context Vision-language Models

    cs.RO 2025-10 unverdicted novelty 5.0

    Hierarchical framework pairs in-context VLMs for high-level plan synthesis with RL-trained low-level skills and failure recovery to reach 92% success on long-horizon DLO routing across varied scenes and language inputs.

Reference graph

Works this paper leans on

82 extracted references · 49 canonical work pages · cited by 3 Pith papers · 7 internal anchors

  1. [1]

    Search-Based Task Planning with Learned Skill Effect Models for Lifelong Robotic Manipulation

    J. Liang, M. Sharma, A. LaGrassa, S. Vats, S. Saxena, and O. Kroemer. Search-Based Task Planning with Learned Skill Effect Models for Lifelong Robotic Manipulation. In IEEE International Conference on Robotics and Automation (ICRA) , 2022. URL https: //arxiv.org/abs/2109.08771

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  2. [2]

    Huang, C

    Y . Huang, C. Agia, J. Wu, T. Hermans, and J. Bohg. Points2plans: From point clouds to long- horizon plans with composable relational dynamics.arXiv preprint arXiv:2408.14769, 2024. 9

    Pith/arXiv arXiv 2024

  3. [3]

    H. Chen, Y . Niu, K. Hong, S. Liu, Y . Wang, Y . Li, and K. R. Driggs-Campbell. Predicting object interactions with behavior primitives: An application in stowing tasks. In7th Annual Conference on Robot Learning, 2023. URL https://openreview.net/forum?id=VH6WIPF4Sj

    2023

  4. [4]

    Paxton, C

    C. Paxton, C. Xie, T. Hermans, and D. Fox. Predicting Stable Configurations for Semantic Placement of Novel Objects. InConference on Robot Learning (CoRL), 11 2021. URL https: //arxiv.org/abs/2108.12062

    Pith/arXiv arXiv 2021

  5. [5]

    C. Agia, T. Migimatsu, J. Wu, and J. Bohg. STAP: Sequencing task-agnostic policies. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7951–7958. IEEE, 2023

    2023

  6. [6]

    Tremblay, A

    J. Tremblay, A. Prakash, D. Acuna, M. Brophy, V . Jampani, C. Anil, T. To, E. Cameracci, S. Boochoon, and S. Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. InProceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 969–977, 2018

    work page 2018

  7. [7]

    Tobin, R

    J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23–30. IEEE, 2017

    2017

  8. [8]

    Ramos, R

    F. Ramos, R. C. Possas, and D. Fox. Bayessim: adaptive domain randomization via probabilistic inference for robotics simulators. arXiv preprint arXiv:1906.01728, 2019

    Pith/arXiv arXiv 1906

  9. [9]

    O. X.-E. Collaboration, A. O’Neill, A. Rehman, A. Gupta, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, A. Tung, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Gupta, A. Wang, A. Kolobov, A. Singh, A. Garg, A. Kembhavi, A. Xie, A. Brohan, A. Raffin, A. Sharma, A. Yavary, A. Jain, A. Balakrishna, A. Wahid...

    Pith/arXiv arXiv 2023

  10. [10]

    Khazatsky, K

    A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y . J. Ma, P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y . Lee, M. Memmel, S. Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. ...

    2024

  11. [11]

    M. Li, T. Zhang, Y . Chen, and A. J. Smola. Efficient mini-batch training for stochastic optimization. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 661–670, 2014

    work page 2014

  12. [12]

    Kumar, T

    N. Kumar, T. Silver, W. McClinton, L. Zhao, S. Proulx, T. Lozano-P´erez, L. P. Kaelbling, and J. Barry. Practice makes perfect: Planning to learn skill parameter policies. InRobotics: Science and Systems (RSS), 2024

    work page 2024

  13. [13]

    Smith, Y

    L. Smith, Y . Cao, and S. Levine. Grow your limits: Continuous improvement with real-world rl for robotic locomotion. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 10829–10836. IEEE, 2024

    work page 2024

  14. [14]

    Mandi, Y

    Z. Mandi, Y . Weng, D. Bauer, and S. Song. Real2code: Reconstruct articulated objects via code generation. arXiv preprint arXiv:2406.08474, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Z. Chen, A. Walsman, M. Memmel, K. Mo, A. Fang, K. Vemuri, A. Wu, D. Fox, and A. Gupta. Urdformer: A pipeline for constructing articulated simulation environments from real-world images. arXiv preprint arXiv:2405.11656, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Torne, A

    M. Torne, A. Simeonov, Z. Li, A. Chan, T. Chen, A. Gupta, and P. Agrawal. Reconciling reality through simulation: A real-to-sim-to-real approach for robust manipulation. arXiv preprint arXiv:2403.03949, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Makoviychuk, L

    V . Makoviychuk, L. Wawrzyniak, Y . Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. In Advances in Neural Information Processing Systems, 2021. URL https: //sites.google.com/view/isaacgym-nvidia

    work page 2021

  18. [18]

    Mittal, C

    M. Mittal, C. Yu, Q. Yu, J. Liu, N. Rudin, D. Hoeller, J. L. Yuan, R. Singh, Y . Guo, H. Mazhar, A. Mandlekar, B. Babich, G. State, M. Hutter, and A. Garg. Orbit: A unified simulation framework for interactive robot learning environments.IEEE Robotics and Automation Letters, 8(6):3740– 3747, 2023. doi:10.1109/LRA.2023.3270034

    arXiv 2023

  19. [19]

    Pavlasek, S

    J. Pavlasek, S. R. Lewis, B. Sundaralingam, F. Ramos, and T. Hermans. Ready, set, plan! planning to goal sets using generalized bayesian inference. In7th Annual Conference on Robot Learning,

    work page

  20. [20]

    F. B. Smith, A. Kirsch, S. Farquhar, Y . Gal, A. Foster, and T. Rainforth. Prediction-oriented bayesian active learning. In International Conference on Artificial Intelligence and Statistics, pages 7331–7348. PMLR, 2023

    work page 2023

  21. [21]

    Farid, D

    A. Farid, D. Snyder, A. Z. Ren, and A. Majumdar. Failure prediction with statistical guarantees for vision-based robot control. arXiv preprint arXiv:2202.05894, 2022. 11

    Pith/arXiv arXiv 2022

  22. [22]

    Inceoglu, E

    A. Inceoglu, E. E. Aksoy, A. C. Ak, and S. Sariel. Fino-net: A deep multimodal sensor fusion framework for manipulation failure detection. In2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6841–6847. IEEE, 2021

    work page 2021

  23. [23]

    C. Agia, R. Sinha, J. Yang, Z. Cao, R. Antonova, M. Pavone, and J. Bohg. Unpacking failure modes of generative policies: Runtime monitoring of consistency and progress. In8th Annual Conference on Robot Learning, 2024. URL https://openreview.net/forum?id=yqLFb0RnDW

    work page 2024

  24. [24]

    Sharma, N

    A. Sharma, N. Azizan, and M. Pavone. Sketching curvature for efficient out-of-distribution detection for deep neural networks. InUncertainty in artificial intelligence, pages 1958–1967. PMLR, 2021

    work page 1958

  25. [25]

    Antonante, D

    P. Antonante, D. I. Spivak, and L. Carlone. Monitoring and diagnosability of perception systems. In 2021 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 168–175. IEEE, 2021

    work page 2021

  26. [26]

    S. Vats, M. Likhachev, and O. Kroemer. Efficient recovery learning using model predictive meta-reasoning. In 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023

    work page 2023

  27. [27]

    Namasivayam, A

    K. Namasivayam, A. Tuli, V . Bindal, H. Singh, P. Singla, and R. Paul. Learning to recover from plan execution errors during robot manipulation: A neuro-symbolic approach. In2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 12632–12639. IEEE, 2024

    work page 2024

  28. [28]

    S. Vats, D. K. Jha, M. Likhachev, O. Kroemer, and D. Romeres. Recoverychaining: Learning local recovery policies for robust manipulation.arXiv preprint arXiv:2410.13979, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Thomason and H

    W. Thomason and H. Kress-Gazit. Counterexample-guided repair for symbolic-geometric action abstractions. IEEE Transactions on Robotics, 39(5):4152–4165, 2023

    work page 2023

  30. [30]

    Torne, A

    M. Torne, A. Jain, J. Yuan, V . Macha, L. Ankile, A. Simeonov, P. Agrawal, and A. Gupta. Robot learning with super-linear scaling. arXiv preprint arXiv:2412.01770, 2024

    arXiv 2024

  31. [31]

    L. Wang, R. Guo, Q. Vuong, Y . Qin, H. Su, and H. Christensen. A real2sim2real method for robust object grasping with neural surface reconstruction. In2023 IEEE 19th International Conference on Automation Science and Engineering (CASE), pages 1–8. IEEE, 2023

    work page 2023

  32. [32]

    V . Lim, H. Huang, L. Y . Chen, J. Wang, J. Ichnowski, D. Seita, M. Laskey, and K. Goldberg. Planar robot casting with real2sim2real self-supervised learning. arXiv preprint arXiv:2111.04814, 2021

    Pith/arXiv arXiv 2021

  33. [33]

    Memmel, A

    M. Memmel, A. Wagenmaker, C. Zhu, P. Yin, D. Fox, and A. Gupta. Asid: Active exploration for system identification in robotic manipulation.arXiv preprint arXiv:2404.12308, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    Chebotar, A

    Y . Chebotar, A. Handa, V . Makoviychuk, M. Macklin, J. Issac, N. Ratliff, and D. Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In2019 International Conference on Robotics and Automation (ICRA), pages 8973–8979. IEEE, 2019

    2019

  35. [35]

    L. Ma, J. Meng, S. Liu, W. Chen, J. Xu, and R. Chen. Sim2real 2: Actively building explicit physics model for precise articulated object manipulation. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11698–11704. IEEE, 2023

    work page 2023

  36. [36]

    Antonova, J

    R. Antonova, J. Yang, P. Sundaresan, D. Fox, F. Ramos, and J. Bohg. A bayesian treatment of real-to-sim for deformable object manipulation.IEEE Robotics and Automation Letters, 7(3): 5819–5826, 2022

    work page 2022

  37. [37]

    S. Qian, L. Jin, C. Rockwell, S. Chen, and D. F. Fouhey. Understanding 3d object articulation in internet videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1599–1609, 2022. 12

    work page 2022

  38. [38]

    Jiang, C.-C

    Z. Jiang, C.-C. Hsu, and Y . Zhu. Ditto: Building digital twins of articulated objects from interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5616–5626, 2022

    work page 2022

  39. [39]

    Heiden, Z

    E. Heiden, Z. Liu, V . Vineet, E. Coumans, and G. S. Sukhatme. Inferring articulated rigid body dynamics from rgbd video. In2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8383–8390. IEEE, 2022

    work page 2022

  40. [40]

    Y . Mao, Y . Zhang, H. Jiang, A. Chang, and M. Savva. Multiscan: Scalable rgbd scanning for 3d environments with articulated objects.Advances in neural information processing systems, 35: 9058–9071, 2022

    work page 2022

  41. [41]

    Barcelos, A

    L. Barcelos, A. Lambert, R. Oliveira, P. Borges, B. Boots, and F. Ramos. Dual Online Stein Variational Inference for Control and Dynamics. InProceedings of Robotics: Science and Systems, Virtual, July 2021. doi:10.15607/RSS.2021.XVII.068

    work page doi:10.15607/rss.2021.xvii.068 2021

  42. [42]

    Y . Lee, A. Z. Li, P. Huang, E. Heiden, K. M. Jatavallabhula, F. Damken, K. Smith, D. Nowrouzezahrai, F. Ramos, and F. Shkurti. Stamp: Differentiable task and motion plan- ning via stein variational gradient descent.arXiv preprint arXiv:2310.01775, 2023

    Pith/arXiv arXiv 2023

  43. [43]

    Power and D

    T. Power and D. Berenson. Constrained stein variational trajectory optimization.IEEE Transac- tions on Robotics, 2024

    work page 2024

  44. [44]

    Entropy Regularized Motion Planning via Stein Variational Inference

    A. Lambert and B. Boots. Entropy regularized motion planning via stein variational inference. arXiv preprint arXiv:2107.05146, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  45. [45]

    Lambert, B

    A. Lambert, B. Hou, R. Scalise, S. S. Srinivasa, and B. Boots. Stein variational probabilistic roadmaps. In 2022 International Conference on Robotics and Automation (ICRA), pages 11094– 11101. IEEE, 2022

    work page 2022

  46. [46]

    Honda, N

    K. Honda, N. Akai, K. Suzuki, M. Aoki, H. Hosogaya, H. Okuda, and T. Suzuki. Stein variational guided model predictive path integral control: Proposal and experiments with fast maneuvering vehicles. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 7020–7026. IEEE, 2024

    work page 2024

  47. [47]

    Kirillov, E

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, et al. Segment anything. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023

    2023

  48. [48]

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023

    Pith/arXiv arXiv 2023

  49. [49]

    K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: From natural language instructions to feasible plans. Autonomous Robots, 47(8):1345–1365, 2023

    work page 2023

  50. [50]

    Huang, N

    Y . Huang, N. C. Taylor, A. Conkey, W. Liu, and T. Hermans. Latent Space Planning for Multi- Object Manipulation with Environment-Aware Relational Classifiers. IEEE Transactions on Robotics (T-RO), 2024. URL https://arxiv.org/pdf/2305.10857.pdf

    Pith/arXiv arXiv 2024

  51. [51]

    C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-P´erez. Integrated task and motion planning.Annual review of control, robotics, and autonomous systems, 4:265–293, 2021. URL https://arxiv.org/abs/2010.01083

    Pith/arXiv arXiv 2021

  52. [52]

    Huang, A

    Y . Huang, A. Conkey, and T. Hermans. Planning for Multi-Object Manipulation with Graph Neural Network Relational Classifiers. In IEEE International Conference on Robotics and Automation (ICRA), 2023. URL https://arxiv.org/abs/2209.11943. 13

    Pith/arXiv arXiv 2023

  53. [53]

    Huang, J

    Y . Huang, J. Yuan, C. Kim, P. Pradhan, B. Chen, L. Fuxin, and T. Hermans. Out of Sight, Still in Mind: Reasoning and Planning about Unobserved Objects with Video Tracking Enabled Memory Models. In IEEE International Conference on Robotics and Automation (ICRA), 2024

    work page 2024

  54. [54]

    B. Settles. Active learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 6 (1):1–114, 6 2012

    work page 2012

  55. [55]

    Active Learning of Probabilistic Movement Primitives

    A. Conkey and T. Hermans. Active Learning of Probabilistic Movement Primitives. InIEEE- RAS International Conference on Humanoid Robotics (Humanoids), 10 2019. URL https: //arxiv.org/abs/1907.00277

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  56. [56]

    Q. Lu, M. V . der Merwe, and T. Hermans. Multi-Fingered Active Grasp Learning. InIEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 10 2020. URL https: //arxiv.org/abs/2006.05264

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  57. [57]

    H¨ullermeier and W

    E. H¨ullermeier and W. Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods.Machine learning, 110(3):457–506, 2021

    work page 2021

  58. [58]

    Kendall and Y

    A. Kendall and Y . Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017

    work page 2017

  59. [59]

    Matsubara, J

    T. Matsubara, J. Knoblauch, F.-X. Briol, and C. J. Oates. Robust generalised bayesian inference for intractable likelihoods. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3):997–1022, 2022

    work page 2022

  60. [60]

    D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859–877, apr 2017. doi:10.1080/ 01621459.2017.1285773. URL https://doi.org/10.1080%2F01621459.2017.1285773

    arXiv 2017

  61. [61]

    Liu and D

    Q. Liu and D. Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems, 29, 2016

    work page 2016

  62. [62]

    Garreau, W

    D. Garreau, W. Jitkrittum, and M. Kanagawa. Large sample analysis of the median heuristic. arXiv preprint arXiv:1707.07269, 2017

    Pith/arXiv arXiv 2017

  63. [63]

    W. R. Gilks and P. Wild. Adaptive rejection sampling for gibbs sampling.Journal of the Royal Statistical Society: Series C (Applied Statistics), 41(2):337–348, 1992

    work page 1992

  64. [64]

    Z. Liu, A. Bahety, and S. Song. Reflect: Summarizing robot experiences for failure explanation and correction. arXiv preprint arXiv:2306.15724, 2023

    Pith/arXiv arXiv 2023

  65. [65]

    N. Shah, J. Nagpal, P. Verma, and S. Srivastava. From reals to logic and back: Inventing symbolic vocabularies, actions and models for planning from raw data.arXiv preprint arXiv:2402.11871, 2024

    arXiv 2024

  66. [66]

    Ahmetoglu, B

    A. Ahmetoglu, B. Celik, E. Oztop, and E. Ugur. Discovering predictive relational object symbols with symbolic attentive layers.IEEE Robotics and Automation Letters, 2024

    work page 2024

  67. [67]

    R. Shah, A. Yu, Y . Zhu, Y . Zhu, and R. Mart´ın-Mart´ın. Bumble: Unifying reasoning and acting with vision-language models for building-wide mobile manipulation. arXiv preprint arXiv:2410.06237, 2024

    Pith/arXiv arXiv 2024

  68. [68]

    P. Liu, Z. Guo, M. Warke, S. Chintala, C. Paxton, N. M. M. Shafiullah, and L. Pinto. Dynamem: Online dynamic spatio-semantic memory for open world mobile manipulation.arXiv preprint arXiv:2411.04999, 2024

    Pith/arXiv arXiv 2024

  69. [69]

    Y . Tang, M. Wang, Y . Deng, Z. Zheng, J. Deng, and Y . Yue. Openin: Open-vocabulary instance- oriented navigation in dynamic domestic environments.arXiv preprint arXiv:2501.04279, 2025. 14

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  70. [70]

    K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. Reid, and N. Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable task planning. In7th Annual Conference on Robot Learning, 2023. URL https://openreview.net/forum?id=wMpOMO0Ss7a

    work page 2023

  71. [71]

    C. Agia, K. Jatavallabhula, M. Khodeir, O. Miksik, V . Vineet, M. Mukadam, L. Paull, and F. Shkurti. Taskography: Evaluating robot task planning over large 3d scene graphs. InConference on Robot Learning, pages 46–58. PMLR, 2022

    work page 2022

  72. [72]

    C. R. Garrett, T. Lozano-P´erez, and L. P. Kaelbling. Pddlstream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 30, pages 440–448, 2020. URL https://arxiv.org/abs/1802.08705

    Pith/arXiv arXiv 2020

  73. [73]

    C. R. Garrett, T. Lozano-P´erez, and L. P. Kaelbling. Sample-based methods for factored task and motion planning. In Robotics: Science and Systems, 2017. URL https://dspace.mit.edu/ bitstream/handle/1721.1/137701/garrett-rss17.pdf?sequence=2&isAllowed=y

    work page 2017

  74. [74]

    B. Kim, Z. Wang, L. P. Kaelbling, and T. Lozano-P´erez. Learning to guide task and motion planning using score-space representation. The International Journal of Robotics Research, 38 (7):793–812, 2019. URL https://arxiv.org/abs/1807.09962

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  75. [75]

    Curtis, X

    A. Curtis, X. Fang, L. P. Kaelbling, T. Lozano-P´erez, and C. R. Garrett. Long-horizon manip- ulation of unknown objects via task and motion planning with estimated affordances. In2022 International Conference on Robotics and Automation (ICRA), pages 1940–1946. IEEE, 2022

    work page 1940

  76. [76]

    Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Image

    D. Driess, J.-S. Ha, and M. Toussaint. Deep visual reasoning: Learning to predict action sequences for task and motion planning from an initial scene image. InProceedings of Robotics: Science and Systems, 2020. URL https://arxiv.org/abs/2006.05398

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  77. [77]

    U. A. Mishra, S. Xue, Y . Chen, and D. Xu. Generative skill chaining: Long-horizon skill planning with diffusion models. In Conference on Robot Learning, pages 2905–2925. PMLR, 2023

    work page 2023

  78. [78]

    Cheng and D

    S. Cheng and D. Xu. League: Guided skill learning and abstraction for long-horizon manipulation. IEEE Robotics and Automation Letters, 2023

    work page 2023

  79. [79]

    W. Wu, Z. Qi, and L. Fuxin. PointConv: Deep Convolutional Networks on 3D Point Clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9621–9630, 2019. URL https://arxiv.org/abs/1811.07246

    Pith/arXiv arXiv 2019

  80. [80]

    Paszke, S

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Sys...

    work page 2019

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.