Pith. sign in

REVIEW 3 major objections 6 minor 66 references

A Bayesian active-testing loop maps a robot policy's failures and strengths across task factors with 20–40% fewer real-world trials than random sampling.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 02:04 UTC pith:F6MMTLMG

load-bearing objection Solid empirical active-testing paper for real-robot evaluation, with a big ground-truth dataset, but the 20–40% savings claim rests on an undemonstrated 'negligible stochasticity' assumption. the 3 major comments →

arxiv 2607.14439 v1 pith:F6MMTLMG submitted 2026-07-16 cs.LG cs.RO

Active Real-World Factor-Based Evaluation for Generalist Robot Policies

Andrew Liao , Hanchen Cui , Karthik Desingh , Aryan Deshwal This is my paper
classification cs.LG cs.RO
keywords robot policy evaluationactive testingBayesian active learninggeneralist robot policiessurrogate modelsGaussian processesout-of-distribution generalizationdata curation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Generalist robot policies are usually judged by a handful of random real-hardware trials collapsed into a single success rate, a practice that can miss the failure modes that matter for deployment. This paper treats policy evaluation as an active-learning problem: fit a probabilistic surrogate that predicts performance across task factors (object position, table height, camera viewpoint), then run each next real-world trial where it would reduce uncertainty most. Across three tabletop manipulation tasks and 2,331 real executions, the active procedure reproduces the performance map that random testing gets in 100 trials using 20–40 fewer trials, and matches its uncertainty-calibration quality in 50–65 fewer trials. If the claim holds, evaluators can characterize how a policy generalizes, where it fails, and where extra demonstrations will help, at a fraction of typical evaluation cost. The object that carries the argument is the surrogate's uncertainty: it is what makes the next trial informative rather than arbitrary.

Core claim

Bayesian active testing — a probabilistic surrogate choosing each next configuration to run — estimates a generalist robot policy's performance distribution more efficiently than uniform random testing. With a 100-trial budget, the active surrogate matches random testing's best RMSE in 20–40 fewer trials and its best log-likelihood in 50–65 fewer trials across three tasks. The same surrogate predicts out-of-distribution performance, ranks sensitivity object position > camera viewpoint > table height, and yields lower run-to-run variance. The best pairing is a deep ensemble with BALD, an acquisition function targeting genuine model disagreement.

What carries the argument

The engine is a Bayesian active-testing loop over a pool of task configurations defined by three factors: object position on an 11×11 grid, table height, and scene camera viewpoint. A probabilistic surrogate — a Gaussian process that learns per-factor relevance, a mixture density network, or a deep ensemble — maps each configuration to a predicted score with an uncertainty estimate. An acquisition function ranks unevaluated configurations by expected information gain; the top one runs on real hardware, its graded outcome joins the training data, and the loop repeats to a 100-trial budget. The best combination is a deep ensemble with BALD (Bayesian Active Learning by Disagreement), which pref

Load-bearing premise

The load-bearing premise, stated in Section IV.A, is that offline replay from a ground-truth dataset with a single evaluation per configuration — assuming the policy's outcome at a fixed configuration is effectively deterministic — faithfully represents online sequential evaluation; if real executions are noisy, the reported 20–40% savings may not transfer to live testing.

What would settle it

Run the selection loop live on the robot with no pre-collected answer key: choose a configuration by the acquisition function, execute it, feed the outcome back, and repeat for 100 trials, comparing the surrogate's RMSE against a live random-testing baseline while repeating several configurations multiple times to measure outcome noise. If the live active curve still beats random by 20–40 trials, the claim holds; if the gap shrinks or vanishes once noise is present, the offline-replay design produced the savings.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • An evaluator with a 100-trial budget reaches the same accuracy of performance-distribution estimates in roughly 60–80 trials, cutting real-hardware evaluation time by 20–40%.
  • Because the surrogate predicts performance at out-of-distribution factor values, a policy's generalization can be assessed from actively chosen trials instead of exhaustively covering unseen factor combinations.
  • The measured sensitivity ordering — object position matters most, table height least — tells evaluators which factors deserve denser test coverage and where narrow test suites are most misleading.
  • Lower run-to-run variance under active selection makes the result of a single evaluation campaign more reproducible, which matters because real-world evaluation is usually run once.
  • In the data-curation extension, demonstrations at actually observed failure points improved overall performance more than demonstrations at surrogate-predicted lowest-outcome points, marking observed failure points as more influential.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 20–40% savings are measured offline: active and random selection are replayed from a dataset with one evaluation per configuration. If repeated live executions are noisy, the live savings will likely be smaller — a gap the paper's own noise-floor ablation suggests exists but is bounded.
  • The same loop transfers naturally to sim-to-real evaluation: run the surrogate cheaply over simulated configurations and let it flag which ones deserve expensive hardware confirmation, concentrating real-robot effort where the sim-to-real gap is most uncertain.
  • The advantage over random testing likely shrinks if outcomes are binary rather than graded, because the surrogate needs a smooth signal to interpolate, and likely grows as more factors are added — a prediction a rerun with a binary scoring rule could test.
  • The appendix's finding that adding demonstrations in one factor region can lower outcomes in another hints that the performance map could warn against negative-interference regions before a data-collection budget is spent; the paper records the effect but leaves this use undeveloped.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes an active evaluation framework for generalist robot manipulation policies. The method models policy performance as a function over task factors (object position, table height, camera viewpoint) using a probabilistic surrogate (GP, mixture density network, or deep ensemble) and selects the next evaluation configuration via an acquisition function (PSD, NIPV, BALD, or EPIG). The authors collected a full ground-truth dataset of 2,331 real-world evaluations across three manipulation tasks, then replayed active and random testing offline from these results. They report that active testing matches the random-testing surrogate's RMSE in 20–40 fewer trials and its log-likelihood in 50–65 fewer trials, and they also study generalization to out-of-distribution factor values and targeted data collection for fine-tuning.

Significance. The paper's main strength is its large real-world evaluation dataset: 2,331 configurations with full ground truth over the factor space is a substantial resource for studying evaluation methodology. The offline replay of active vs. random testing is a sound first-order comparison when the policy is deterministic, and the inclusion of six surrogate/acquisition combinations is useful. The generalization analysis and the data curation experiment are additional contributions. If the reported trial savings transfer to genuinely online sequential evaluation, the framework would be practically valuable for reducing the cost of evaluating generalist policies. However, the central quantitative claim depends on an unquantified assumption about policy stochasticity and on a loosely defined savings metric, which currently limits confidence in the headline number.

major comments (3)
  1. [§IV.A, §IV.B] The assumption that 'the stochasticity of the robot policy’s outputs is assumed to be negligible' is load-bearing for the paper's central claim, but no supporting repeated-evaluation data are provided. The paper itself acknowledges aleatoric noise in §IV.B (nonzero asymptotic RMSE) and Appendix VI.B.2 (GP noise floor of 10^-1 due to 'possible aleatoric noise'). In the offline replay, the active learner observes the exact ground-truth outcome at each selected configuration, whereas a real evaluator would observe a noisy draw. If outcome noise is non-negligible, active acquisition can mistake aleatoric noise for epistemic uncertainty, and the reported 20–40% trial savings may shrink or vanish. Please provide repeated evaluations on a subset of configurations (e.g., 50–100 configurations, 3–5 repetitions each) to quantify outcome noise, or run a genuinely online active evaluation on at leas
  2. [§IV.B, Figure 4] The metric 'matches the best value achieved by the random-testing surrogate in 20–40 fewer trials' is under-specified. Does 'best value' mean the random surrogate's RMSE at trial 100, or the minimum RMSE along its learning curve? How is 'typically' quantified across tasks and random seeds? Without a precise definition of the crossing-time or threshold protocol, the headline savings are not uniquely verifiable. Please define the savings measure explicitly (e.g., the trial at which the active method reaches the random method's final RMSE, averaged over runs) and report the standard deviation across seeds.
  3. [§IV.B, Appendix VI.B.1] The paper selects 'Deep Ensemble + BALD' as the best combination after comparing all six surrogate/acquisition combinations on the same data that are later used to support the conclusions in §V. This post-hoc selection can inflate apparent performance due to multiple comparisons. The abstract and introduction phrase the claim as 'our approach typically saves... 20–40% of trials,' which is ambiguous about whether this is the best variant or the ensemble of active variants. Please clarify whether the reported savings are for the selected best combination only; if so, discuss the selection bias and consider validating the selected combination on a held-out task or reporting a selection-adjusted estimate.
minor comments (6)
  1. [Abstract, §I] The phrase 'at least 20-40%' is logically imprecise; it should be '20–40%' or 'at least 20%'. Similarly, 'typically saves' in §IV.B should be accompanied by quantitative confidence intervals or a distribution over runs.
  2. [Eq. (11)] The EPIG formula has a notational issue: the expectation $E_{p(y|x,D_t)}$ should make clear that it is over the predictive distribution at the candidate point x, and the integral over the target input $x^*$ should be explicit. Please clean up the notation.
  3. [Algorithm 1] Line 4 says 'Fit GP surrogate model,' but the framework also includes MDN and deep ensembles. Change to 'Fit surrogate model' to match the general formulation.
  4. [Figure 5] The generalization claim is based on visual comparison of ground truth and surrogate predictions. Add quantitative error metrics (e.g., RMSE or MAE on ID/OOD subsets) to support the statement that the surrogate 'can accurately predict both ID and OOD performance.'
  5. [§VI.A] Reproducibility would be improved by pinning the exact version/commit of the 'openpi' repository and reporting the LoRA hyperparameters beyond the number of steps.
  6. [References] Several references contain formatting artifacts (e.g., 'Y .' in refs [52]–[57]); please ensure consistent and correct rendering.

Circularity Check

0 steps flagged

No significant circularity: the efficiency result is an empirical offline comparison against a fixed ground-truth set, not a derivation reducing to its inputs.

full rationale

The central efficiency claim (Sec. IV.B) is an empirical comparison, not a first-principles derivation. Active and random testing are simulated offline by sampling from a precomputed full-evaluation ground truth (Sec. IV.A), the surrogate is refit on the acquired subset, and RMSE/log-likelihood are measured as a function of trial count. The active method does not receive the target outcomes as fitted parameters; the random baseline is evaluated under an identical protocol; and the reported 20-40% savings is a measured difference in trial counts, not a quantity equal to the method's inputs by construction. No load-bearing step relies on a self-citation chain or on a uniqueness theorem from the authors. The paper's own caveats — single-trial ground truth, the assumption that policy stochasticity is negligible, and selection of the Deep Ensemble + BALD variant after inspecting all combinations on the same data — are threats to external validity and to transferability of the 20-40% figure, but they do not make the comparison definitionally circular. The GP noise floor and kernel choices are standard modeling choices, not fitted versions of the reported outcome. Accordingly, no circular step is identified.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper's contribution is an empirical comparison, not a derivation, so there are no invented entities. The central result rests on modeling assumptions (smooth performance landscape), an experimental assumption (deterministic outcomes), and a protocol assumption (offline replay equals online testing). The listed free parameters are the budget, warm-start size, noise floor, and neural architecture choices, which were chosen by hand or ablation.

free parameters (4)
  • Evaluation budget and initial sample size = T=100 trials, N0=30 warm-start points
    The headline saving is measured at this budget; the advantage of active testing could change with different budgets.
  • GP noise floor = 10^-1
    Selected from an ablation (Figure 8); it affects calibration and convergence but is not derived from first principles.
  • MDN and Deep Ensemble architecture hyperparameters
    2 mixture components, 4 hidden layers of width 32, dropout 0.1, 200 epochs; chosen by hand and affect acquisition quality.
  • GP kernel hyperparameters (output variance and per-factor lengthscales)
    Learned by maximizing marginal likelihood on accumulated evaluation data; the surrogate's predictive quality depends on them.
axioms (4)
  • domain assumption Policy performance is smooth over the factor design space
    The GP/neural surrogates assume correlated outcomes across nearby configurations; if the performance landscape is highly discontinuous, surrogate predictions and acquisition values lose validity (Section III.B.1).
  • domain assumption Policy outcomes are deterministic and repeatable
    Each configuration is evaluated once and policy stochasticity is "assumed to be negligible based on empirical observations" (Section IV.A); if false, the ground-truth map is one noisy sample.
  • domain assumption Offline replay from full ground truth is equivalent to online active evaluation
    Active and random testing are simulated by sampling from the precomputed 2,331 evaluations (Section IV.A); this assumes no distribution shift or time-correlated effects occur when the same evaluations are run online.
  • domain assumption Factors outside the design space are held constant and do not affect relative ranking
    Lighting, object orientations, and initial arm positions are fixed (Section IV.A.4), so conclusions about factor sensitivity apply only within that fixed background.

pith-pipeline@v1.3.0-alltime-deepseek · 17199 in / 12731 out tokens · 127778 ms · 2026-08-02T02:04:07.373880+00:00 · methodology

0 comments
read the original abstract

Generalist robot manipulation policies trained on large, diverse datasets have shown remarkable promise across a wide range of tasks. However, rigorously evaluating these policies remains a fundamental challenge. Real-world performance depends on a large combinatorial space of task factors including object poses and camera viewpoints, making full, exhaustive evaluation intractable. Additionally, real hardware evaluation is slow and resource-intensive, so current practice is to use narrow test suites that can miss critical failure modes and misrepresent true deployment readiness. We propose an active evaluation framework that addresses this challenge by treating policy evaluation as a sequential experimental design problem. Our approach fits a probabilistic surrogate model over a structured space of task factors and adaptively selects evaluation configurations to maximize information gain over the policy's performance distribution, allowing for sample-efficient characterization of policy behavior across unseen conditions and a systematic identification of failure-prone regions. We conduct 2331 real-world evaluations across 3 tasks with 3 factor variations and find that our approach typically saves the evaluator at least 20-40% of trials compared to typical random testing.

Figures

Figures reproduced from arXiv: 2607.14439 by Andrew Liao, Aryan Deshwal, Hanchen Cui, Karthik Desingh.

Figure 1
Figure 1. Figure 1: Illustration of our proposed approach which uses Bayesian active [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of evaluation strategies: full evaluation covers the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Our evaluation suite: 3 tasks, 3 table heights, 3 scene camera positions, and 121 object positions on table. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Active testing methods versus uniform-random testing across 3 tasks. Shaded regions indicate [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ground truth and surrogate model-predicted mean outcome for ID & [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of different acquisition functions using the same surrogate model (Gaussian Process). BALD is superior in log-likelihood and [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of different surrogate models using the same acquisition function (Bayesian Active Learning by Disagreement). Deep Ensemble [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Several Gaussian Process design choices: 1) standard (noise floor of [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Mean continuous outcomes after using either a surrogate prediction [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 12
Figure 12. Figure 12: Baseline outcome (before data collection), number of points [PITH_FULL_IMAGE:figures/full_fig_p015_12.png] view at source ↗
Figure 11
Figure 11. Figure 11: Change in outcomes of all points in the factor spaces for each [PITH_FULL_IMAGE:figures/full_fig_p015_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 17 linked inside Pith

  1. [1]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PmLR, 2021, pp. 8748–8763

    2021

  2. [2]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei...

    2020

  3. [3]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” inAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 34 892–34 916. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2023/file/6dcf277ea32ce3288914fa...

    2023

  4. [4]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control,

    B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahidet al., “Rt-2: Vision-language-action models transfer web knowledge to robotic control,” inConference on Robot Learning. PMLR, 2023, pp. 2165–2183

    2023

  5. [5]

    Openvla: An open-source vision-language-action model,

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn, “Openvla: An open-source vision-language-action model,” inProceedings of The 8th Conference on Robot Learning, ser. Proceedings of Machine Learning Res...

    2025

  6. [6]

    π0: A vision-language-action flow model for general robot control,

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky, “π0: A vision-language-action flow model for general robot control,”

  7. [7]

    Libero-plus: In-depth robustness analysis of vision-language-action models,

    S. Fei, S. Wang, J. Shi, Z. Dai, J. Cai, P. Qian, L. Ji, X. He, S. Zhang, Z. Fei, J. Fu, J. Gong, and X. Qiu, “Libero-plus: In-depth robustness analysis of vision-language-action models,” 2025. [Online]. Available: https://arxiv.org/abs/2510.13626

    Pith/arXiv arXiv 2025

  8. [8]

    Shortcut learning in generalist robot policies: The role of dataset diversity and fragmentation,

    Y . Xing, X. Luo, J. Xie, L. Gao, H. T. Shen, and J. Song, “Shortcut learning in generalist robot policies: The role of dataset diversity and fragmentation,” inProceedings of The 9th Conference on Robot Learning, ser. Proceedings of Machine Learning Research, J. Lim, S. Song, and H.-W. Park, Eds., vol. 305. PMLR, 27–30 Sep 2025, pp. 3239–3266. [Online]. A...

    2025

  9. [9]

    Robot learning as an empirical science: Best practices for policy evaluation,

    H. Kress-Gazit, K. Hashimoto, N. Kuppuswamy, P. Shah, P. Horgan, G. Richardson, S. Feng, and B. Burchfiel, “Robot learning as an empirical science: Best practices for policy evaluation,” 2024. [Online]. Available: https://arxiv.org/abs/2409.09491

    Pith/arXiv arXiv 2024

  10. [10]

    Isaac gym: High performance gpu-based physics simulation for robot learning,

    V . Makoviychuk, L. Wawrzyniak, Y . Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State, “Isaac gym: High performance gpu-based physics simulation for robot learning,” 2021. [Online]. Available: https://arxiv.org/abs/2108.10470

    Pith/arXiv arXiv 2021

  11. [11]

    Mujoco: A physics engine for model-based control,

    E. Todorov, T. Erez, and Y . Tassa, “Mujoco: A physics engine for model-based control,”2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026–5033, 2012. [Online]. Available: https://api.semanticscholar.org/CorpusID:5230692

    2012

  12. [12]

    A survey on vision–language–action models for embodied ai,

    Y . Ma, Z. Song, Y . Zhuang, J. Hao, and I. King, “A survey on vision–language–action models for embodied ai,”IEEE Transactions on Neural Networks and Learning Systems, 2026

    2026

  13. [13]

    Bc-z: Zero-shot task generalization with robotic imitation learning,

    E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, “Bc-z: Zero-shot task generalization with robotic imitation learning,” inconference on Robot Learning. PMLR, 2022, pp. 991–1002

    2022

  14. [14]

    A generalist agent,

    S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y . Sulsky, J. Kay, J. T. Springenberg, T. Eccles, J. Bruce, A. Razavi, A. Edwards, N. Heess, Y . Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas, “A generalist agent,” 2022. [Online]. Available: https://arxiv.org/abs/2205.06175

    Pith/arXiv arXiv 2022

  15. [15]

    Rt-1: Robotics transformer for real-world control at scale,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, I. Leal, K.-H. Lee, S. Levine, Y . Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J...

    Pith/arXiv arXiv 2023

  16. [16]

    Open x-embodiment: Robotic learning datasets and rt-x models: Open x- embodiment collaboration 0,

    A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jainet al., “Open x-embodiment: Robotic learning datasets and rt-x models: Open x- embodiment collaboration 0,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 6892–6903

    2024

  17. [17]

    Droid: A large-scale in-the-wild robot manipulation dataset,

    A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y . J. Ma, P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y . Lee, M. Memmel, S. Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. ...

    Pith/arXiv arXiv 2025

  18. [18]

    A taxonomy for evaluating generalist robot manipulation policies,

    J. Gao, S. Belkhale, S. Dasari, A. Balakrishna, D. Shah, and D. Sadigh, “A taxonomy for evaluating generalist robot manipulation policies,” IEEE Robotics and Automation Letters, 2026

    2026

  19. [19]

    Investigating the role of instruction variety and task difficulty in robotic manipulation tasks,

    A. Parekh, N. Vitsakis, A. Suglia, and I. Konstas, “Investigating the role of instruction variety and task difficulty in robotic manipulation tasks,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 19 389–19 424

    2024

  20. [20]

    Libero: Benchmarking knowledge transfer for lifelong robot learn- ing,

    B. Liu, Y . Zhu, C. Gao, Y . Feng, Q. Liu, Y . Zhu, and P. Stone, “Libero: Benchmarking knowledge transfer for lifelong robot learn- ing,”Advances in Neural Information Processing Systems, vol. 36, pp. 44 776–44 791, 2023

    2023

  21. [21]

    Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks,

    O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard, “Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks,”IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 7327–7334, 2022

    2022

  22. [22]

    The colosseum: A benchmark for evaluating generalization for robotic manipulation,

    W. Pumacay, I. Singh, J. Duan, R. Krishna, J. Thomason, and D. Fox, “The colosseum: A benchmark for evaluating generalization for robotic manipulation,” 2024. [Online]. Available: https://arxiv.org/abs/2402.08191

    Pith/arXiv arXiv 2024

  23. [23]

    Decomposing the generalization gap in imitation learning for visual robotic manipulation,

    A. Xie, L. Lee, T. Xiao, and C. Finn, “Decomposing the generalization gap in imitation learning for visual robotic manipulation,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 3153–3160

    2024

  24. [24]

    Vlabench: A large-scale benchmark for language- conditioned robotics manipulation with long-horizon reasoning tasks,

    S. Zhang, Z. Xu, P. Liu, X. Yu, Y . Li, Q. Gao, Z. Fei, Z. Yin, Z. Wu, Y .-G. Jianget al., “Vlabench: A large-scale benchmark for language- conditioned robotics manipulation with long-horizon reasoning tasks,” inProceedings of the IEEE/CVF International Conference on Com- puter Vision, 2025, pp. 11 142–11 152

    2025

  25. [25]

    Design and use paradigms for gazebo, an open-source multi-robot simulator,

    N. P. Koenig and A. Howard, “Design and use paradigms for gazebo, an open-source multi-robot simulator,”2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566), vol. 3, pp. 2149–2154 vol.3, 2004. [Online]. Available: https://api.semanticscholar.org/CorpusID:206941306

    2004

  26. [26]

    Evaluating real-world robot manipulation policies in simulation,

    X. Li, K. Hsu, J. Gu, O. Mees, K. Pertsch, H. R. Walke, C. Fu, I. Lunawat, I. Sieh, S. Kirmani, S. Levine, J. Wu, C. Finn, H. Su, Q. Vuong, and T. Xiao, “Evaluating real-world robot manipulation policies in simulation,” inProceedings of The 8th Conference on Robot Learning, ser. Proceedings of Machine Learning Research, P. Agrawal, O. Kroemer, and W. Burg...

  27. [27]

    Reconciling reality through simulation: A real-to-sim- to-real approach for robust manipulation,

    M. Torne, A. Simeonov, Z. Li, A. Chan, T. Chen, A. Gupta, and P. Agrawal, “Reconciling reality through simulation: A real-to-sim- to-real approach for robust manipulation,” 2024. [Online]. Available: https://arxiv.org/abs/2403.03949

    Pith/arXiv arXiv 2024

  28. [28]

    Real-to-sim robot policy evaluation with gaussian splatting simulation of soft-body interactions,

    K. Zhang, S. Sha, H. Jiang, M. Loper, H. Song, G. Cai, Z. Xu, X. Hu, C. Zheng, and Y . Li, “Real-to-sim robot policy evaluation with gaussian splatting simulation of soft-body interactions,” 2025. [Online]. Available: https://arxiv.org/abs/2511.04665

    arXiv 2025

  29. [29]

    Realm: A real-to-sim validated benchmark for generalization in robotic ma- nipulation,

    M. Sedlacek, P. Yefanov, G. Ponimatkin, J. Bardhan, S. Pilc, M. Fourmy, E. Kazakos, C. G. Snoek, J. Sivic, and V . Petrik, “Realm: A real-to-sim validated benchmark for generalization in robotic ma- nipulation,”IEEE Robotics and Automation Letters, 2026

    2026

  30. [30]

    Polaris: Scalable real-to-sim evaluations for generalist robot policies,

    A. Jain, M. Zhang, K. Arora, W. Chen, M. Torne, M. Z. Irshad, S. Zakharov, Y . Wang, S. Levine, C. Finn, W.-C. Ma, D. Shah, A. Gupta, and K. Pertsch, “Polaris: Scalable real-to-sim evaluations for generalist robot policies,” 2025. [Online]. Available: https://arxiv.org/abs/2512.16881

    arXiv 2025

  31. [31]

    Real-is-sim: Bridging the sim-to-real gap with a dynamic digital twin,

    J. Abou-Chakra, L. Sun, K. Rana, B. May, K. Schmeckpeper, N. Suenderhauf, M. V . Minniti, and L. Herlant, “Real-is-sim: Bridging the sim-to-real gap with a dynamic digital twin,” 2025. [Online]. Available: https://arxiv.org/abs/2504.03597

    Pith/arXiv arXiv 2025

  32. [32]

    Real2render2real: Scaling robot data without dynamics simulation or robot hardware,

    J. Yu, L. Fu, H. Huang, K. El-Refai, R. A. Ambrus, R. Cheng, M. Z. Irshad, and K. Goldberg, “Real2render2real: Scaling robot data without dynamics simulation or robot hardware,” 2025. [Online]. Available: https://arxiv.org/abs/2505.09601

    Pith/arXiv arXiv 2025

  33. [33]

    Robotarena ∞: Scalable robot benchmarking via real-to-sim translation,

    Y . Jangir, Y . Zhang, P.-C. Lo, K. Yamazaki, C. Zhang, K.-H. Tu, T.-W. Ke, L. Ke, Y . Bisk, and K. Fragkiadaki, “Robotarena ∞: Scalable robot benchmarking via real-to-sim translation,” 2026. [Online]. Available: https://arxiv.org/abs/2510.23571

    arXiv 2026

  34. [34]

    Contrast sets for evaluating language-guided robot policies,

    A. Anwar, R. Gupta, and J. Thomason, “Contrast sets for evaluating language-guided robot policies,” inProceedings of The 8th Conference on Robot Learning, ser. Proceedings of Machine Learning Research, P. Agrawal, O. Kroemer, and W. Burgard, Eds., vol

  35. [35]

    Reliable and scalable robot policy evaluation with imperfect simulators,

    A. Badithela, D. Snyder, L. Zha, J. Mikhail, M. O’Kelly, A. Dixit, and A. Majumdar, “Reliable and scalable robot policy evaluation with imperfect simulators,” 2025. [Online]. Available: https://arxiv.org/abs/2510.04354

    arXiv 2025

  36. [36]

    Is your imitation learning policy better than mine? policy comparison with near-optimal stopping,

    D. Snyder, A. J. Hancock, A. Badithela, E. Dixon, P. Miller, R. A. Ambrus, A. Majumdar, M. Itkina, and H. Nishimura, “Is your imitation learning policy better than mine? policy comparison with near-optimal stopping,” 2025. [Online]. Available: https://arxiv.org/abs/2503.10966

    Pith/arXiv arXiv 2025

  37. [37]

    2205–2219

    PMLR, 06–09 Nov 2025, pp. 2205–2219. [Online]. Available: https://proceedings.mlr.press/v270/anwar25a.html

    2025

  38. [38]

    Gaussian pro- cess regression: Active data selection and test point rejection,

    S. Seo, M. Wallat, T. Graepel, and K. Obermayer, “Gaussian pro- cess regression: Active data selection and test point rejection,” in Proceedings of the IEEE-INNS-ENNS international joint conference on neural networks. IJCNN 2000. Neural computing: New challenges and perspectives for the New Millennium, vol. 3. IEEE, 2000, pp. 241–246

    2000

  39. [39]

    Active risk estimation,

    C. Sawade, N. Landwehr, S. Bickel, and T. Scheffer, “Active risk estimation,” inInternational Conference on Machine Learning,

  40. [40]

    Bayesian experimental design: A review,

    K. Chaloner and I. Verdinelli, “Bayesian experimental design: A review,”Statistical science, pp. 273–304, 1995

    1995

  41. [41]

    Active surrogate estimators: An active learning approach to label-efficient model evaluation,

    ——, “Active surrogate estimators: An active learning approach to label-efficient model evaluation,”Advances in Neural Information Processing Systems, vol. 35, pp. 24 557–24 570, 2022

    2022

  42. [42]

    Active testing: An unbiased evaluation method for distantly supervised relation extraction,

    P. Li, X. Zhang, W. Jia, and W. Zhao, “Active testing: An unbiased evaluation method for distantly supervised relation extraction,” in Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 204–211

    2020

  43. [43]

    Efficient evaluation of multi-task robot policies with active experiment selection,

    A. Anwar, R. Gupta, Z. Merchant, S. Ghosh, W. Neiswanger, and J. Thomason, “Efficient evaluation of multi-task robot policies with active experiment selection,” inProceedings of The 9th Conference on Robot Learning, ser. Proceedings of Machine Learning Research, J. Lim, S. Song, and H.-W. Park, Eds., vol. 305. PMLR, 27–30 Sep 2025, pp. 1636–1653. [Online]...

    2025

  44. [44]

    Active testing: Sample-efficient model evaluation,

    J. Kossen, S. Farquhar, Y . Gal, and T. Rainforth, “Active testing: Sample-efficient model evaluation,” inInternational Conference on Machine Learning. PMLR, 2021, pp. 5753–5763

    2021

  45. [45]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning,

    Y . Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” ininternational conference on machine learning. PMLR, 2016, pp. 1050–1059

    2016

  46. [46]

    Gaussian process regression: active data selection and test point rejection,

    S. Seo, M. Wallat, T. Graepel, and K. Obermayer, “Gaussian process regression: active data selection and test point rejection,” inPro- ceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium, vol. 3, 2000, pp. 241–246 vol.3

    2000

  47. [47]

    Bayesian active learning for classification and preference learning,

    N. Houlsby, F. Husz ´ar, Z. Ghahramani, and M. Lengyel, “Bayesian active learning for classification and preference learning,” 2011. [Online]. Available: https://arxiv.org/abs/1112.5745

    Pith/arXiv arXiv 2011

  48. [48]

    A. J. Smola and B. Sch ¨olkopf,Learning with kernels. GMD- Forschungszentrum Informationstechnik Berlin, Germany, 1998, vol. 4

    1998

  49. [49]

    What matters in learning from large-scale datasets for robot manipulation,

    V . Saxena, M. Bronars, N. R. Arachchige, K. Wang, W. C. Shin, S. Nasiriany, A. Mandlekar, and D. Xu, “What matters in learning from large-scale datasets for robot manipulation,” inInternational Conference on Learning Representations 2025, 2025

    2025

  50. [50]

    Efficient data collection for robotic manipulation via compositional generalization,

    J. Gao, A. Xie, T. Xiao, C. Finn, and D. Sadigh, “Efficient data collection for robotic manipulation via compositional generalization,”

  51. [51]

    CUPID: Curating data your robot loves with influence functions,

    C. Agia, R. Sinha, J. Yang, R. Antonova, M. Pavone, H. Nishimura, M. Itkina, and J. Bohg, “CUPID: Curating data your robot loves with influence functions,” in9th Annual Conference on Robot Learning, 2025. [Online]. Available: https://openreview.net/forum?id= TqevdDMqrK

    2025

  52. [52]

    Prediction-oriented bayesian active learning,

    F. B. Smith, A. Kirsch, S. Farquhar, Y . Gal, A. Foster, and T. Rain- forth, “Prediction-oriented bayesian active learning,” inInternational conference on artificial intelligence and statistics. PMLR, 2023, pp. 7331–7348

    2023

  53. [53]

    Curating demonstrations using online experience,

    A. S. Chen, A. M. Lessing, Y . Liu, and C. Finn, “Curating demonstrations using online experience,” 2025. [Online]. Available: https://arxiv.org/abs/2503.03707

    Pith/arXiv arXiv 2025

  54. [54]

    Is diversity all you need for scalable robotic manipulation?

    M. Shi, L. Chen, J. Chen, Y . Lu, C. Liu, G. Ren, P. Luo, D. Huang, M. Yao, and H. Li, “Is diversity all you need for scalable robotic manipulation?”IEEE Transactions on Robotics, 2026

    2026

  55. [55]

    Guiding data collection via factored scaling curves,

    L. Zha, A. Badithela, M. Zhang, J. Lidard, J. Bao, E. Zhou, D. Snyder, A. Z. Ren, D. Shah, and A. Majumdar, “Guiding data collection via factored scaling curves,” 2025. [Online]. Available: https://arxiv.org/abs/2505.07728

    Pith/arXiv arXiv 2025

  56. [56]

    Robot data curation with mutual information estimators,

    J. Hejna, S. Mirchandani, A. Balakrishna, A. Xie, A. Wahid, J. Tompson, P. Sanketi, D. Shah, C. Devin, and D. Sadigh, “Robot data curation with mutual information estimators,” 2025. [Online]. Available: https://arxiv.org/abs/2502.08623

    Pith/arXiv arXiv 2025

  57. [57]

    Actively testing your model while it learns: Realizing label-efficient learning in practice,

    D. Yu, W. Shi, and Q. Yu, “Actively testing your model while it learns: Realizing label-efficient learning in practice,” inThirty- seventh Conference on Neural Information Processing Systems, 2023. [Online]. Available: https://openreview.net/forum?id=du0hvEpgj8

    2023

  58. [62]

    DataMIL: Selecting data for robot imitation learning with datamodels,

    S. Dass, A. Khaddaj, L. Engstrom, A. Madry, A. Ilyas, and R. Mart ´ın- Mart´ın, “DataMIL: Selecting data for robot imitation learning with datamodels,” inThe Fourteenth International Conference on Learning Representations, 2026. [Online]. Available: https: //openreview.net/forum?id=AcTsKglDdh VI. APPENDIX A. Policy Finetuning •The following numbers of dem...

    2026

  59. [63]

    Here, we show ablations of either only acquisition functions or only surrogate models to highlight the best combination for sample-efficient evaluation

    Best Combinations:The full comparison of all com- binations of surrogate models and acquisition functions can be found in Figure 4. Here, we show ablations of either only acquisition functions or only surrogate models to highlight the best combination for sample-efficient evaluation. Figure 6 shows an ablation of acquisition functions, keeping the surroga...

  60. [64]

    Figure 8 shows the ablation of several model design choices: 1) no noise floor,

    Surrogate Model: •Gaussian Process:For the Gaussian Process (”Single- TaskGP” on BoTorch), we use a noise floor of10 −1 because of possible aleatoric noise (e.g., human labeling inconsistency, policy stochasticity). Figure 8 shows the ablation of several model design choices: 1) no noise floor,

  61. [65]

    Using a noise floor of 10−1 converges the fastest

    warm start (when refitting at trialn, start from the previous trial’s model parameters), and 3) refitting every three trials instead of every trial. Using a noise floor of 10−1 converges the fastest. •Fully Bayesian Gaussian Process:The Fully Bayesian Gaussian Process surrogate (”FullyBayesianSingle- TaskGP” on BoTorch) uses Markov Chain Monte Carlo (MCMC...

  62. [66]

    However, XWED is inappropriate for our problem for two reasons: –The output of the generalist model (actions) differs from the output of the surrogate model (outcomes)

    Acquisition Function: •Expected Weighted Disagreement (XWED):XWED, introduced as an alternative to BALD for the active testing problem [41], weights each BALD disagreement term by the loss of the original modelfat the inputsxcompared to the ground truthy: αXWED(x) =E Y∼π(·|x) [−L(Y, f(x)) logπ(Y|x)] −E Θ∼π(·) h EY∼π(·|x,Θ) [−L(Y, f(x)) logπ(Y|x,Θ)] i The ...

  63. [270]

    3705–3728

    PMLR, 06–09 Nov 2025, pp. 3705–3728. [Online]. Available: https://proceedings.mlr.press/v270/li25c.html

    2025

  64. [2010]

    Available: https://api.semanticscholar.org/CorpusID: 16144243

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 16144243

  65. [2024]

    Available: https://arxiv.org/abs/2403.05110

    [Online]. Available: https://arxiv.org/abs/2403.05110

    Pith/arXiv arXiv

  66. [2026]

    Available: https://arxiv.org/abs/2410.24164

    [Online]. Available: https://arxiv.org/abs/2410.24164

    Pith/arXiv arXiv