Pith. sign in

REVIEW 3 major objections 5 minor 52 references

Efficient Evaluation of Multi-Task Robot Policies With Active Experiment Selection

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Selecting the most informative trials lets a language-guided surrogate model estimate robot policy performance across many tasks at lower cost than random sampling.

desk verdict Useful active-testing framework for robot evaluation, but the EIG engine is a heuristic dressed as Bayesian, and the paper's own Limitations concede the point. read the letter →

arxiv 2502.09829 v1 pith:ANEYAZUF submitted 2025-02-14 cs.RO cs.AIcs.LG

Abrar Anwar , Rohan Gupta , Zain Merchant , Sayan Ghosh , Willie Neiswanger , Jesse Thomason This is my paper · ORCID
classification cs.ROcs.AIcs.LG
keywords activetestingrobotpolicyevaluationexpectedinformationgaincost-awaresamplinglanguage-basedtaskembeddingssurrogatemodelmulti-taskpopulationparameterestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Evaluating a robot policy on every task is expensive: each trial needs a human reset, and each task change may require rearranging objects or swapping robots. This paper reframes evaluation as an active testing problem: run a few well-chosen trials, train a surrogate model that predicts the outcome distribution for every policy-task pair, and use the model to decide which experiment to run next. The paper's central claim is that natural-language task descriptions provide a useful prior for sharing information across tasks, and that a cost-aware expected-information-gain acquisition rule estimates policy performance at lower cost than random sampling. The method handles both binary success and continuous reward outcomes, and experiments on existing real-robot and simulation evaluation data support the cost reduction.

What carries the argument

The central object is a learned surrogate parameter-estimation model combined with a cost-aware acquisition function. For a policy $\pi_i$ and task $T_j$, an MLP maps policy and task embeddings to the parameters of the outcome distribution; these parameters are a Bernoulli probability for binary success, or the mixture coefficients, means, and standard deviations of a two-component Gaussian mixture for continuous outcomes, trained with a mixture density loss. The acquisition function is $\mathrm{EIG}(\pi_i, T_j)/(\lambda c_{\mathrm{switch}}(T_{\mathrm{current}}, T_j) + 1)$, where EIG is the BALD disagreement between ten Monte Carlo dropout samples of the predicted distribution, and $c_{\mathrm{switch}}$ is the cost of switching from the current task. The framework uses an epsilon-greedy exploration rate of 0.1 and evaluates each selected experiment three times.

What would settle it

Run the framework on a benchmark with a held-out set of tasks whose instructions share vocabulary with training tasks but whose success profiles are anticorrelated, and compare the surrogate's mean estimates against a constant global-mean prior at matched trial cost; if the language-conditioned surrogate does not beat that trivial baseline, the claimed transfer from language embeddings is falsified.

Watch

Extended reading notes

Core claim

The paper claims that multi-task robot evaluation can be cast as population parameter estimation over a matrix of policy-task performance distributions, and that a surrogate model $f(\pi_i, T_j) = \mathrm{MLP}(e_{\pi_i}, e_{T_j})$ trained on a small set of trials estimates the parameters $\theta_{ij}$ of every cell. Task embeddings formed as $e_{T_j} = 0.8\,e_{\mathrm{verb}} + 0.2\,e_{\mathrm{task}} + 0.1\,\mathcal{N}(0,1)$ from PCA-reduced MiniLMv2 embeddings of the instruction text and its verbs let the model transfer information from evaluated to unevaluated tasks. The next experiment is chosen by maximizing expected information gain, computed from ten Monte Carlo dropout samples of the surrogate, divided by a switching-cost penalty. On offline datasets from real robots and simulation, cost-aware EIG sampling estimates the means of Bernoulli and Gaussian-mixture outcomes more efficiently than random baselines, with task-based sampling often more cost-effective than policy-task sampling.

Load-bearing premise

The load-bearing premise is that a policy's performance on an unevaluated task is predictable from a language-based embedding of the task description, so if that embedding fails to align with true performance correlations, the shared information vanishes and active selection gains disappear.

Editorial extensions

If this is right

  • Experimenters can skip exhaustive policy-task testing and instead estimate a full performance matrix from a budgeted number of informative trials.
  • Because the surrogate is trained only on outcome samples, the method applies to closed models and engineered systems without access to policy weights.
  • Cost-aware selection matters most when task switching is expensive; the paper shows this by adding switching-cost terms on top of execution costs.
  • The same protocol supports both binary success metrics and continuous rewards, so it covers the two most common robot evaluation settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open multi-step lookahead: choosing a short sequence of experiments at once could cut switching costs further when transitions dominate the budget, since a one-step greedy rule cannot plan across multiple switches.
  • The gap the authors report between language embeddings and outcome-trained embeddings suggests a testable extension: incorporating initial scene observations or object affordance descriptors into the task representation could close part of that gap.
  • A prospective validation would run the acquisition rule live on a physical robot, rather than replaying offline datasets, and compare total cost against a random baseline for a target precision on the estimated means.
  • The fixed execution-cost assumption is a limitation the authors acknowledge; modelling failure-dependent costs, such as cleaning up a spilled object, would change which experiments are selected.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper frames multi-task robot policy evaluation as an active testing problem. A surrogate MLP predicts each policy-task cell's outcome distribution (Gaussian mixture for continuous rewards, Bernoulli for success) from a policy embedding and a task embedding; task embeddings combine a MiniLM verb-phrase embedding, the full language embedding, and noise. The acquisition function is a cost-aware expected information gain (BALD) computed from test-time Monte Carlo dropout samples, with switching costs between tasks. The authors evaluate the model on offline datasets from HAMSTER, OpenVLA, and MetaWorld, comparing random, EIG, and cost-aware EIG selection, at both the policy-task and task level, using average log likelihood and L1 error of the predicted mean.

Significance. If its central claim held, the paper would address a real and growing bottleneck in robot evaluation: the high experimenter cost of testing many policy-task pairs. The manuscript makes a useful step by formalizing evaluation as population parameter estimation and by showing, on several real and simulated datasets, that language-based task embeddings carry transferable information. The paper is also transparent about its negative results, explicitly noting that EIG-based approaches struggle to model the full outcome distribution. However, the headline contribution---cost reduction via informative trial selection---is only clearly supported for estimating the mean, not the full distribution, and the uncertainty estimator that drives the acquisition function is an unvalidated approximation of posterior sampling. The framework is best read, as presented, as an empirical heuristic whose theoretical grounding is incomplete.

major comments (3)
  1. [Section IV-B, Eq. (6)] The BALD acquisition score is defined as an expectation over θij ∼ f(θij|D), yet the samples are obtained by applying dropout only at test time to a single MLP trained without dropout. The Gal & Ghahramani Monte Carlo dropout approximation, on which the method is based, requires dropout to be active during training; otherwise stochastic forward passes are not draws from an approximate posterior. The Limitations section concedes that this approach "has not been rigorously tested by the Bayesian optimization community." Because the central cost-reduction claim is built on selecting experiments by expected information gain, the paper must either validate this uncertainty estimator (e.g., by comparing with a properly trained MC-dropout model or an ensemble on the acquisition scores) or reframe the acquisition as a heuristic and remove the information-theoretic interpretation.
  2. [Abstract and Section VII-B (Figs. 5 and 6)] The claim that the framework "reduces the cost of calculating evaluation metrics" is broader than the evidence. Figure 5 shows that EIG-based acquisition functions fit the full outcome distribution only marginally better than random baselines, and the paper itself states that the improvement is not consistent across datasets. The clear advantage appears only in the L1 error of the mean (Figure 6). The conclusion should be scoped to mean estimation, or the paper should provide distribution-level evidence (e.g., calibrated negative log likelihood with confidence intervals) to support the general cost-reduction claim.
  3. [Section IV-A Eq. (3) and Section IV-B (Algorithm 1, line 10)] The task-embedding weights (0.8 verb, 0.2 full text, 0.1 noise) and the choice of three trials per selected experiment are described as having been selected in preliminary experiments on the same offline datasets subsequently used for the final evaluation. Because the active-selection results use the Verb representation and the three-trial protocol, this selection on the evaluation data can inflate the apparent advantage of the method. The manuscript should report a sensitivity analysis on held-out tasks or policies, or explicitly acknowledge that these hyperparameters were tuned on the evaluation datasets and that the reported improvements may not generalize.
minor comments (5)
  1. [Section IV-A (before Eq. 3)] The sentence introducing the task embedding refers to "the task embedding eπi", but the notation eπi is elsewhere used for the policy embedding; this should read eTj.
  2. [Section IV-B] The reference to Monte Carlo dropout [12] should explicitly state that the theoretical guarantee that makes dropout a posterior approximation requires dropout to be active during training, not only at test time; as written, the text conflates the general technique with the specific conditions under which it is justified.
  3. [Section VII-A] The sentence "The task-based sampling strategies is more realistic to how experimenters evaluate their robots today" has a subject-verb agreement error and should be revised.
  4. [Section VI-A and Section VII-A] The representation experiments use 750 evaluation steps while the acquisition experiments use 1500 steps; the text should explain why the protocols differ and whether this affects comparability of the results across the two sections.
  5. [Section IV-A, Eq. (4)] The symbol π is used both for policies/sampling probabilities and for the mixture component weights in Eq. (4); using a different symbol, such as α_k, would avoid ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: active-testing results are benchmarked against external random baselines; cited limitations affect validity, not circularity.

full rationale

This paper does not contain a derivation that reduces to its own inputs. It is an empirical active-testing framework: a surrogate model (Eq. 2) is trained on observed outcome data, and the claimed cost reductions are measured against random-sampling baselines on existing offline evaluation datasets (HAMSTER, OpenVLA, MetaWorld). The language task embedding in Eq. (3) and the choice of three trials per experiment are justified by exploratory statements, but the paper does not show that these choices are equivalent to the evaluation metric or that the central comparison is forced by construction. The Limitations passage conceding that test-time MC dropout has not been rigorously tested by the Bayesian optimization community is a validity caveat about the uncertainty estimator, not a circularity: the EIG scores could be imperfect without making the evaluation a restatement of the framework's inputs. The paper's self-citations are background references and are not load-bearing for the central claim. No uniqueness theorem is imported, no fitted parameter is renamed as a prediction, and no known result is repackaged as a new derivation. The main correctness risks concern the statistical validity of the uncertainty estimate, not circular reasoning.

Assumptions & free parameters 9 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several hand-fitted hyperparameters (embedding weights, noise, trials, lambda) and on domain assumptions about MC dropout, dataset fidelity, and cost models. None of these is a new invented entity, but together they define the operating regime of the method.

free parameters (9)
  • Task embedding weights (0.8 verb, 0.2 full text, 0.1 noise) = 0.8, 0.2, 0.1
    Chosen by hand in Section IV-A Eq. 3 to compensate for language models overfocusing on nouns; no search reported.
  • Noise term std for task embedding = 1.0 (N(0,1))
    Added to separate close task embeddings, selected after observing embeddings too close (Section IV-A).
  • Lambda cost sensitivity = not reported
    Hyperparameter in Eq. 7 controlling trade-off between EIG and switching cost; no value or sensitivity analysis given.
  • Epsilon exploration rate = 0.1
    Set in Algorithm 1 for epsilon-greedy exploration.
  • Trials per selected experiment = 3
    Chosen after preliminary experiments as better for cost-efficient estimation (Algorithm 1, Section IV-B).
  • Number of Gaussian components K = 2
    Assumed bimodal robot outcomes (Section IV-A).
  • Entropy discretization bins = 25
    Number of bins used to estimate mixture entropy (Section IV-B).
  • MC dropout samples = 10
    Number of sampled distributions per policy-task pair for EIG (Algorithm 1).
  • PCA dimension for language embeddings = 32
    Reduced MiniLMv2 embeddings to 32 dimensions (Section IV-A).
assumptions (4)
  • ad hoc to paper MC dropout at test time provides a valid posterior approximation for BALD acquisition
    Used in Section IV-B; authors note in Limitations that this approach has not been rigorously tested by the Bayesian optimization community.
  • domain assumption The offline evaluation datasets faithfully represent the true outcome distributions of policies on tasks
    The active testing simulation samples from these datasets; for HAMSTER, single evaluations are treated as Gaussian means with fixed standard deviation (Section V).
  • domain assumption The hand-assigned switching costs reflect real experimenter effort
    Costs in Appendix A are manually set (e.g., +1 or +2 for object/task changes) and the authors note in Limitations that execution costs are represented naively at a fixed cost.
  • domain assumption Language embeddings encode task similarity relevant to policy performance
    Central modeling premise of Section IV-A and Section VI; results show it holds partially, as language embeddings underperform optimal embeddings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Evaluation of Multi-Task Robot Policies With Active Experiment Selection." pith.science (2026). https://pith.science/paper/ANEYAZUF

@misc{pith2026250209829,
  author       = {Pith},
  title        = {Pith review of: Efficient Evaluation of Multi-Task Robot Policies With Active Experiment Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ANEYAZUF}},
  note         = {Machine review of arXiv:2502.09829}
}
read the original abstract

Evaluating learned robot control policies to determine their physical task-level capabilities costs experimenter time and effort. The growing number of policies and tasks exacerbates this issue. It is impractical to test every policy on every task multiple times; each trial requires a manual environment reset, and each task change involves re-arranging objects or even changing robots. Naively selecting a random subset of tasks and policies to evaluate is a high-cost solution with unreliable, incomplete results. In this work, we formulate robot evaluation as an active testing problem. We propose to model the distribution of robot performance across all tasks and policies as we sequentially execute experiments. Tasks often share similarities that can reveal potential relationships in policy behavior, and we show that natural language is a useful prior in modeling these relationships between tasks. We then leverage this formulation to reduce the experimenter effort by using a cost-aware expected information gain heuristic to efficiently select informative trials. Our framework accommodates both continuous and discrete performance outcomes. We conduct experiments on existing evaluation data from real robots and simulations. By prioritizing informative trials, our framework reduces the cost of calculating evaluation metrics for robot policies across many tasks.

Figures

Figures reproduced from arXiv: 2502.09829 by the authors.

Figure 1
Figure 1. Overview. Exhaustively evaluating multiple robot policies across various tasks has high experimenter cost. In this work, we leverage latent relationships between tasks and policies to model performance distributions across all tasks and policies. These estimates are updated sequentially and used to implement cost-aware active experiment selection strategies. exhaustive evaluation even more impractical. As such, ther… view at source ↗
Figure 2
Figure 2. Method. We build a surrogate parameter estimation model that learns task and policy embeddings to predict the outcome performance distribution of a task and policy combination. We use Bernoulli distributions for binary outcomes or a bimodal Gaussian for continuous outcomes. Given this parameter estimation model, we develop an active testing strategy with cost-aware sampling based on expected information gain. matrix… view at source ↗
Figure 3
Figure 3. Offline Datasets used for Experiments. We consider 4 settings: (1) evaluations from HAMSTER [29], (2) evaluations from the OpenVLA paper [23], (3) MetaWorld [52] where we evaluate different policies, and (4) MetaWorld where we evaluate multiple checkpoints of a single policy. For the MetaWorld evaluations, we can model the performance distributions of success rate or continuous rewards. For OpenVLA, the outcomes are… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Task and Policy Representation Experiments. We compute the average log likelihood of all outcomes under probability distribution represented by the predicted population parameters across various policy and task representations. We evaluate these methods over the HAMSTE…
Figure 5
Figure 5. Figure 5: Average Log Likelihood Over Cost. We show the average log likelihood of all the outcomes in our offline dataset against the cost of evaluation for MetaWorld Policies, MetaWorld Checkpoints, HAMSTER, and OpenVLA over continuous and binary performance distributions. Each…
Figure 6
Figure 6. Figure 6: Average L1 Error of the Mean Over Cost. Instead of computing the average log likelihood of the data as in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Predicted Mean Distributions. We provide a visualization of the means for the predicted continuous and binary distributions over 0, 150, and 750 sampled queries. We use random sampling with 3 evaluations per policy-task pair to show that our surrogate model can activel…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 47 canonical work pages

  1. [1]

    Sim-to-real transfer for vision-and-language navigation

    Peter Anderson, Ayush Shrivastava, Joanne Truong, Ar- jun Majumdar, Devi Parikh, Dhruv Batra, and Stefan Lee. Sim-to-real transfer for vision-and-language navigation. Conference on Robot Learning (CoRL) , 2021

    work page 2021

  2. [2]

    Con- trast sets for evaluating language-guided robot policies

    Abrar Anwar, Rohan Gupta, and Jesse Thomason. Con- trast sets for evaluating language-guided robot policies. Conference on Robot Learning (CoRL) , 2024

    work page 2024

  3. [3]

    Remembr: Building and reasoning over long-horizon spatio-temporal memory for robot navigation

    Abrar Anwar, John Welsh, Joydeep Biswas, Soha Pouya, and Yan Chang. Remembr: Building and reasoning over long-horizon spatio-temporal memory for robot navigation. International Conference on Robotics and Automation (ICRA), 2025

    work page 2025

  4. [4]

    Surrogate assisted generation of human-robot interaction scenarios

    Varun Bhatt, Heramb Nemlekar, Matthew C Fontaine, Bryon Tjanaka, Hejia Zhang, Ya-Chuan Hsu, and Ste- fanos Nikolaidis. Surrogate assisted generation of human-robot interaction scenarios. Conference on Robot Learning (CoRL), 2023

    work page 2023

  5. [5]

    Mixture density networks

    Christopher M Bishop. Mixture density networks. Tech- nical Report, 1994

    work page 1994

  6. [6]

    π0: A vision- language-action flow model for general robot control

    Kevin Black, Noah Brown, Danny Driess, Adnan Es- mail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision- language-action flow model for general robot control. arXiv preprint arXiv:2410.24164 , 2024

    arXiv

  7. [7]

    A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning

    Eric Brochu, Vlad M Cora, and Nando De Freitas. A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.2599, 2010

    arXiv 2010

  8. [8]

    A survey on evaluation of large language models

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxi- ang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology (TIST) , 2024

    work page 2024

Show all 52 references
  1. [9]

    Learning surrogate models for simulation-based opti- mization

    Alison Cozad, Nikolaos V Sahinidis, and David C Miller. Learning surrogate models for simulation-based opti- mization. AIChE Journal, 60(6):2211–2227, 2014

    2014

  2. [10]

    RoboTHOR: An Open Simulation-to-Real Embodied AI Platform

    Matt Deitke, Winson Han, Alvaro Herrasti, Anirud- dha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, Luca Weihs, Mark Yatskar, and Ali Farhadi. RoboTHOR: An Open Simulation-to-Real Embodied AI Platform. Conference o...

    2020

  3. [11]

    Efficient benchmarking of hyper- parameter optimizers via surrogates

    Katharina Eggensperger, Frank Hutter, Holger Hoos, and Kevin Leyton-Brown. Efficient benchmarking of hyper- parameter optimizers via surrogates. In Proceedings of the AAAI conference on artificial intelligence , 2015

    2015

  4. [12]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. International Conference on Machine Learning (ICML), 2016

    2016

  5. [13]

    Efficient Data Collection for Robotic Manipulation via Compositional Generalization

    Jensen Gao, Annie Xie, Ted Xiao, Chelsea Finn, and Dorsa Sadigh. Efficient Data Collection for Robotic Manipulation via Compositional Generalization. Pro- ceedings of Robotics: Science and Systems (RSS) , 2024

    2024

  6. [14]

    Liu, Phoebe Mul- caire, Qiang Ning, Sameer Singh, Noah A

    Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mul- caire, Qi...

    2020

  7. [15]

    Navigating to objects in the real world

    Theophile Gervet, Soumith Chintala, Dhruv Batra, Jiten- dra Malik, and Devendra Singh Chaplot. Navigating to objects in the real world. Science Robotics, 2023

    2023

  8. [16]

    Minillm: Knowledge distillation of large language mod- els

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language mod- els. In International Conference on Learning Represen- tations (ICLR), 2024

    2024

  9. [17]

    World models

    David Ha and J ¨urgen Schmidhuber. World models. Conference on Neural Information Processing System (NeurIPS), 2018

    2018

  10. [18]

    Benchmarking neural network robustness to common corruptions and perturbations

    Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. International Conference on Learning Representations (ICLR), 2019

    2019

  11. [19]

    Pretrained transformers improve out-of-distribution robustness

    Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve out-of-distribution robustness. As- sociation for Computational Linguistics (ACL) , 2020

    2020

  12. [20]

    Bayesian active learning for classification and preference learning

    Neil Houlsby, Ferenc Husz ´ar, Zoubin Ghahramani, and M´at´e Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745, 2011

    2011 arXiv

  13. [21]

    Deploying and Evaluating LLMs to Program Service Mobile Robots

    Zichao Hu, Francesca Lucchetti, Claire Schlesinger, Yash Saxena, Anders Freeman, Sadanand Modak, Arjun Guha, and Joydeep Biswas. Deploying and Evaluating LLMs to Program Service Mobile Robots. IEEE Robotics and Automation Letters (RA-L) , 2024

    2024

  14. [22]

    Sim2Real Predictivity: Does Evaluation in Simulation Predict Real- World Performance? IEEE Robotics and Automation Letters (RA-L), 2020

    Abhishek Kadian, Joanne Truong, Aaron Gokaslan, Alexander Clegg, Erik Wijmans, Stefan Lee, Manolis Savva, Sonia Chernova, and Dhruv Batra. Sim2Real Predictivity: Does Evaluation in Simulation Predict Real- World Performance? IEEE Robotics and Automation Letters (RA-L), 2020

    2020

  15. [23]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla...

    2024

  16. [24]

    Active testing: Sample-efficient model eval- uation

    Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sample-efficient model eval- uation. International Conference on Machine Learning (ICML), 2021

    2021

  17. [25]

    Robot learn- ing as an empirical science: Best practices for policy evaluation

    Hadas Kress-Gazit, Kunimatsu Hashimoto, Naveen Kup- puswamy, Paarth Shah, Phoebe Horgan, Gordon Richard- son, Siyuan Feng, and Benjamin Burchfiel. Robot learn- ing as an empirical science: Best practices for policy evaluation. arXiv, 2024

    2024

  18. [26]

    Dropout injection at test time for post hoc uncertainty quantification in neural networks

    Emanuele Ledda, Giorgio Fumera, and Fabio Roli. Dropout injection at test time for post hoc uncertainty quantification in neural networks. Information Sciences, 2023

    2023

  19. [27]

    Cost-aware bayesian optimization

    Eric Hans Lee, Valerio Perrone, Cedric Archambeau, and Matthias Seeger. Cost-aware bayesian optimization. arXiv preprint arXiv:2003.10870 , 2020

    2003 arXiv

  20. [28]

    Evaluating real-world robot manipulation policies in sim- ulation

    Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. Evaluating real-world robot manipulation policies in sim- ulation. Con...

    2024

  21. [29]

    Ham- ster: Hierarchical action models for open-world robot manipulation

    Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Caelan Reed Garrett, Fabio Ramos, Dieter Fox, Anqi Li, Abhishek Gupta, and Ankit Goyal. Ham- ster: Hierarchical action models for open-world robot manipulation. International Conference on Learning Representations (ICLR), 2025

    2025

  22. [30]

    Holistic evaluation of language models

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. Transactions on Machine Learning Research (TLMR) , 2022

    2022

  23. [31]

    A general framework for uncertainty estimation in deep learning

    Antonio Loquercio, Mattia Segu, and Davide Scara- muzza. A general framework for uncertainty estimation in deep learning. IEEE Robotics and Automation Letters (RA-L), 2020

    2020

  24. [32]

    Probabilistic matrix factorization

    Andriy Mnih and Russ R Salakhutdinov. Probabilistic matrix factorization. Conference on Neural Information Processing Systems (NeurIPS) , 2007

    2007

  25. [33]

    Differential assessment of black-box ai agents

    Rashmeet Kaur Nayyar, Pulkit Verma, and Siddharth Sri- vastava. Differential assessment of black-box ai agents. AAAI Conference on Artificial Intelligence , 2022

    2022

  26. [34]

    Octo: An open-source generalist robot policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo...

    2024

  27. [35]

    Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Tasks

    Amit Parekh, Nikolas Vitsakis, Alessandro Suglia, and Ioannis Konstas. Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Tasks. Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2024

    2024

  28. [36]

    Cost-aware bayesian optimization via information directed sampling

    Biswajit Paria, Willie Neiswanger, Ramina Ghods, Jeff Schneider, and Barnab ´as P ´oczos. Cost-aware bayesian optimization via information directed sampling. In Adap- tive Experimental Design and Active Learning in the Real World Workshop at ICML, 2020

    2020

  29. [37]

    THE COLOS- SEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation

    Wilbert Pumacay, Ishika Singh, Jiafei Duan, Ranjay Krishna, Jesse Thomason, and Dieter Fox. THE COLOS- SEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation. Robotics: Science and Systems (RSS), 2024

    2024

  30. [38]

    Building surrogate models based on detailed and approximate simulations

    Zhiguang Qian, Carolyn Conner Seepersad, V Roshan Joseph, Janet K Allen, and CF Jeff Wu. Building surrogate models based on detailed and approximate simulations. Journal of Mechanical Design , 2006

    2006

  31. [39]

    Modern bayesian experimental design

    Tom Rainforth, Adam Foster, Desi R Ivanova, and Fred- die Bickford Smith. Modern bayesian experimental design. Statistical Science, 39(1):100–114, 2024

    2024

  32. [40]

    Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning (ICML), 2019

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning (ICML), 2019

    2019

  33. [41]

    Active risk estimation

    Christoph Sawade, Niels Landwehr, Steffen Bickel, and Tobias Scheffer. Active risk estimation. InProceedings of the 27th International Conference on Machine Learning (ICML-10), pages 951–958, 2010

    2010

  34. [42]

    Vint: A foundation model for visual navigation

    Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kyle Sta- chowicz, Kevin Black, Noriaki Hirose, and Sergey Levine. Vint: A foundation model for visual navigation. Conference on Robot Learning (CoRL) , 2022

    2022

  35. [43]

    Lm- nav: Robotic navigation with large pre-trained models of language, vision, and action

    Dhruv Shah, Bła ˙zej Osi ´nski, Sergey Levine, et al. Lm- nav: Robotic navigation with large pre-trained models of language, vision, and action. Conference on Robot Learning (CoRL), 2023

    2023

  36. [44]

    Taking the human out of the loop: A review of bayesian optimization

    Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE , 104(1):148–175, 2015

    2015

  37. [45]

    Targeted active learning for probabilistic models

    Christopher Tosh, Mauricio Tec, and Wesley Tansey. Targeted active learning for probabilistic models. arXiv preprint arXiv:2210.12122, 2022

    2022 arXiv

  38. [46]

    Discovering user-interpretable capabilities of black-box planning agents

    Pulkit Verma, Shashank Rao Marpally, and Siddharth Srivastava. Discovering user-interpretable capabilities of black-box planning agents. International Conference on Principles of Knowledge Representation and Reasoning (KR), 2021

    2021

  39. [47]

    Autonomous capability assessment of sequen- tial decision-making systems in stochastic settings

    Pulkit Verma, Rushang Karia, and Siddharth Srivas- tava. Autonomous capability assessment of sequen- tial decision-making systems in stochastic settings. Conference on Neural Information Processing Systems (NeurIPS), 2023

    2023

  40. [48]

    How Generalizable Is My Behavior Cloning Policy? A Statis- tical Approach to Trustworthy Performance Evaluation

    Joseph A Vincent, Haruki Nishimura, Masha Itkina, Paarth Shah, Mac Schwager, and Thomas Kollar. How Generalizable Is My Behavior Cloning Policy? A Statis- tical Approach to Trustworthy Performance Evaluation. IEEE Robotics and Automation Letters (RA-L) , 2024

    2024

  41. [49]

    CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding

    Dong Wang, Ning Ding, Piji Li, and Hai-Tao Zheng. CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding. Asso- ciation for Computational Linguistics (ACL) , 2021

    2021

  42. [50]

    Decomposing the generalization gap in imitation learning for visual robotic manipulation

    Annie Xie, Lisa Lee, Ted Xiao, and Chelsea Finn. Decomposing the generalization gap in imitation learning for visual robotic manipulation. International Conference on Robotics and Automation (ICRA) , 2024

    2024

  43. [51]

    Sample efficient model evaluation

    Emine Yilmaz, Peter Hayes, Raza Habib, Jordan Burgess, and David Barber. Sample efficient model evaluation. arXiv preprint arXiv:2109.12043 , 2021

    2021 arXiv

  44. [52]

    Meta- world: A benchmark and evaluation for multi-task and meta reinforcement learning

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta- world: A benchmark and evaluation for multi-task and meta reinforcement learning. Conference on Robot Learning (CoRL), 2020. APPENDIX A OFFLINE DATASET DETAILS A. HAMSTE...

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.