{"id":"7c6be217-688a-45f9-b90f-e06be708615b","arxiv_id":"2607.14439","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An active evaluation framework selects the most informative task configurations for real-robot tests, matching random testing's accuracy in 20-40% fewer trials.","lead":"Researchers propose a method to choose which real-robot test setups to run next, using a model of the robot's performance to pick the most informative trials. In tests across three manipulation tasks, the method reached the same accuracy as random testing in 20-40% fewer trials.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Offline replay's 'negligible stochasticity' assumption is not established for a flow-based VLA policy; single-trial ground truth may make reported 20–40% trial savings optimistic.","rationale":"I agree with the reader's identification of the offline replay assumption as the weakest point, and I sharpen it: the policy is a flow model, so stochasticity is not a generic concern but a structural one. The paper's own noise floor is an admission. The other candidate concerns—selection of the best surrogate/acquisition combination from the same experiments and lack of code/data—affect reproducibility and overfitting risk, but they are less central: the paper shows most active variants beat random, so the headline is not solely due to cherry-picking. The offline replay concern directly threatens the quantitative transfer of the headline claim to real sequential testing. If noise is present, the simulation is optimistic because it gives the active learner the noise-free outcome at each queried point. A concrete repeated-trial experiment plus a noise-injected replay would settle this. Since the reader already assigned CONDITIONAL, my stress test does not change the verdict; it reinforces it.","tokens_in":17465,"tokens_out":7158,"duration_ms":71101,"concrete_test":"Select 50–100 configurations per task from the ground-truth set and repeat each evaluation 5–10 times on the real robot, recording per-configuration outcome variance. If the standard deviation is non-negligible (e.g., >0.3 on the 0–6 outcome scale), re-run the offline active-vs-random comparison but sample the observed outcome at each selected configuration from a distribution with that empirical variance (instead of using the fixed ground-truth value). If the active method no longer matches random's RMSE in 20–40 fewer trials, the headline claim needs to be qualified to the noiseless/deterministic setting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim (§IV.B) is that active testing matches the random-testing surrogate's RMSE in 20–40 fewer trials. This is established only through the offline replay in §IV.A: 'Active and random testing are done offline, sampling from these ground truth results to simulate the sequential testing process... The stochasticity of the robot policy's outputs is assumed to be negligible based on empirical observations.' The load-bearing premise is that a single deterministic outcome per configuration is a faithful stand-in for the real sequential evaluation. This is especially fragile because the policy under test is π0, a flow-based VLA (ref [6]) whose actions are generated by iterative denoising/sampling; repeated executions at a fixed configuration can differ. The paper itself acknowledges noise: §IV.B attributes nonzero asymptotic RMSE partly to 'aleatoric noise, irreducible noise from the data-generating process (e.g., inconsistencies with labeling, stochasticity of the policy),' and Appendix VI.B.2 sets a GP noise floor of 10^-1 because of 'possible aleatoric noise.' If single-trial outcomes are noisy, the offline replay provides the active learner with the exact true value at each selected point, whereas a real evaluator observes a noisy draw. Active acquisition can mistake aleatoric noise for epistemic uncertainty and select points that do not generalize; the 20–40% savings may shrink or vanish. No repeated-evaluation data are shown to support the 'negligible' assumption. This is not an internal inconsistency, but the headline requires an empirical anchor before the savings claim can be accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an active evaluation framework for generalist robot manipulation policies. The method models policy performance as a function over task factors (object position, table height, camera viewpoint) using a probabilistic surrogate (GP, mixture density network, or deep ensemble) and selects the next evaluation configuration via an acquisition function (PSD, NIPV, BALD, or EPIG). The authors collected a full ground-truth dataset of 2,331 real-world evaluations across three manipulation tasks, then replayed active and random testing offline from these results. They report that active testing matches the random-testing surrogate's RMSE in 20–40 fewer trials and its log-likelihood in 50–65 fewer trials, and they also study generalization to out-of-distribution factor values and targeted data collection for fine-tuning.","tokens_in":17814,"tokens_out":4970,"duration_ms":52202,"significance":"The paper's main strength is its large real-world evaluation dataset: 2,331 configurations with full ground truth over the factor space is a substantial resource for studying evaluation methodology. The offline replay of active vs. random testing is a sound first-order comparison when the policy is deterministic, and the inclusion of six surrogate/acquisition combinations is useful. The generalization analysis and the data curation experiment are additional contributions. If the reported trial savings transfer to genuinely online sequential evaluation, the framework would be practically valuable for reducing the cost of evaluating generalist policies. However, the central quantitative claim depends on an unquantified assumption about policy stochasticity and on a loosely defined savings metric, which currently limits confidence in the headline number.","major_comments":[{"comment":"The assumption that 'the stochasticity of the robot policy’s outputs is assumed to be negligible' is load-bearing for the paper's central claim, but no supporting repeated-evaluation data are provided. The paper itself acknowledges aleatoric noise in §IV.B (nonzero asymptotic RMSE) and Appendix VI.B.2 (GP noise floor of 10^-1 due to 'possible aleatoric noise'). In the offline replay, the active learner observes the exact ground-truth outcome at each selected configuration, whereas a real evaluator would observe a noisy draw. If outcome noise is non-negligible, active acquisition can mistake aleatoric noise for epistemic uncertainty, and the reported 20–40% trial savings may shrink or vanish. Please provide repeated evaluations on a subset of configurations (e.g., 50–100 configurations, 3–5 repetitions each) to quantify outcome noise, or run a genuinely online active evaluation on at leas","section":"§IV.A, §IV.B"},{"comment":"The metric 'matches the best value achieved by the random-testing surrogate in 20–40 fewer trials' is under-specified. Does 'best value' mean the random surrogate's RMSE at trial 100, or the minimum RMSE along its learning curve? How is 'typically' quantified across tasks and random seeds? Without a precise definition of the crossing-time or threshold protocol, the headline savings are not uniquely verifiable. Please define the savings measure explicitly (e.g., the trial at which the active method reaches the random method's final RMSE, averaged over runs) and report the standard deviation across seeds.","section":"§IV.B, Figure 4"},{"comment":"The paper selects 'Deep Ensemble + BALD' as the best combination after comparing all six surrogate/acquisition combinations on the same data that are later used to support the conclusions in §V. This post-hoc selection can inflate apparent performance due to multiple comparisons. The abstract and introduction phrase the claim as 'our approach typically saves... 20–40% of trials,' which is ambiguous about whether this is the best variant or the ensemble of active variants. Please clarify whether the reported savings are for the selected best combination only; if so, discuss the selection bias and consider validating the selected combination on a held-out task or reporting a selection-adjusted estimate.","section":"§IV.B, Appendix VI.B.1"}],"minor_comments":[{"comment":"The phrase 'at least 20-40%' is logically imprecise; it should be '20–40%' or 'at least 20%'. Similarly, 'typically saves' in §IV.B should be accompanied by quantitative confidence intervals or a distribution over runs.","section":"Abstract, §I"},{"comment":"The EPIG formula has a notational issue: the expectation $E_{p(y|x,D_t)}$ should make clear that it is over the predictive distribution at the candidate point x, and the integral over the target input $x^*$ should be explicit. Please clean up the notation.","section":"Eq. (11)"},{"comment":"Line 4 says 'Fit GP surrogate model,' but the framework also includes MDN and deep ensembles. Change to 'Fit surrogate model' to match the general formulation.","section":"Algorithm 1"},{"comment":"The generalization claim is based on visual comparison of ground truth and surrogate predictions. Add quantitative error metrics (e.g., RMSE or MAE on ID/OOD subsets) to support the statement that the surrogate 'can accurately predict both ID and OOD performance.'","section":"Figure 5"},{"comment":"Reproducibility would be improved by pinning the exact version/commit of the 'openpi' repository and reporting the LoRA hyperparameters beyond the number of steps.","section":"§VI.A"},{"comment":"Several references contain formatting artifacts (e.g., 'Y .' in refs [52]–[57]); please ensure consistent and correct rendering.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the unquantified policy stochasticity assumption. If the authors can supply repeated-trial data showing the policy is indeed deterministic, or a re-run of the offline replay with added noise matched to empirical measurements, the paper could become acceptable. Also, the post-hoc selection of the best surrogate/acquisition combination should be addressed, as it affects the strength of the headline claim. The paper's scope fits cs.LG well; the real-world dataset is a valuable contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful, honest empirical paper on active testing for real-robot evaluation. The big asset is a 2,331-trial full ground-truth dataset across three tasks and three factors, and a fair offline comparison of active vs random with several surrogate/acquisition combos. The claim that active testing saves 20–40% of trials is plausible but hinges on a load-bearing assumption the paper doesn't back up: that stochasticity in the policy's outputs is negligible. π0 is a flow-based VLA, so repeated executions at the same configuration can differ; the paper says 'assumed to be negligible based on empirical observations' but shows no repeated-run data. The offline replay gives the active learner the exact outcome at each selected point, whereas a real evaluator sees a noisy draw. That can make the savings look better than they would be online. The paper itself acknowledges aleatoric noise in the RMSE floor and the 10^-1 noise floor, so this is not a straw-man.\n\nAlso, the 20–40% number comes from the best surrogate/acquisition combination (Deep Ensemble + BALD), chosen after looking at all six on the same data. That's a mild selection-bias issue; the ablations in the appendix help, but the headline is for the winner, not the average.\n\nWhat the paper does well: the experimental design is transparent — full ground truth, replayed active vs random with the same surrogates and metrics, and useful ablations. The finding that active testing gives lower variance across runs is a nice practical point. The data-curation appendix is a separate good contribution, even if a bit tangential.\n\nWho's it for: anyone thinking about efficient evaluation of robot policies on real hardware. The dataset and methodology are worth engaging with even if the headline isn't fully nailed down.\n\nMy recommendation: send it to peer review, but request (a) repeated evaluations on a subset of configurations to test the stochasticity assumption, (b) report per-combination results without cherry-picking, and (c) release the dataset. If the assumption holds, this becomes a solid, citable result.","headline":"Solid empirical active-testing paper for real-robot evaluation, with a big ground-truth dataset, but the 20–40% savings claim rests on an undemonstrated 'negligible stochasticity' assumption.","tokens_in":18316,"tokens_out":3495,"would_cite":true,"duration_ms":29183,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Bayesian active-testing loop maps a robot policy's failures and strengths across task factors with 20–40% fewer real-world trials than random sampling.","keywords":["robot policy evaluation","active testing","Bayesian active learning","generalist robot policies","surrogate models","Gaussian processes","out-of-distribution generalization","data curation"],"falsifier":"Run the selection loop live on the robot with no pre-collected answer key: choose a configuration by the acquisition function, execute it, feed the outcome back, and repeat for 100 trials, comparing the surrogate's RMSE against a live random-testing baseline while repeating several configurations multiple times to measure outcome noise. If the live active curve still beats random by 20–40 trials, the claim holds; if the gap shrinks or vanishes once noise is present, the offline-replay design produced the savings.","tokens_in":17366,"feed_emoji":"🤖","tokens_out":18409,"duration_ms":152067,"temperature":0.7,"pith_summary":"Generalist robot policies are usually judged by a handful of random real-hardware trials collapsed into a single success rate, a practice that can miss the failure modes that matter for deployment. This paper treats policy evaluation as an active-learning problem: fit a probabilistic surrogate that predicts performance across task factors (object position, table height, camera viewpoint), then run each next real-world trial where it would reduce uncertainty most. Across three tabletop manipulation tasks and 2,331 real executions, the active procedure reproduces the performance map that random testing gets in 100 trials using 20–40 fewer trials, and matches its uncertainty-calibration quality in 50–65 fewer trials. If the claim holds, evaluators can characterize how a policy generalizes, where it fails, and where extra demonstrations will help, at a fraction of typical evaluation cost. The object that carries the argument is the surrogate's uncertainty: it is what makes the next trial informative rather than arbitrary.","feed_headline":"Save 20–40% of robot evaluation trials with active testing","feed_subtitle":"Choosing each robot test with a Bayesian surrogate maps performance and failure zones using far fewer hardware runs.","key_machinery":"The engine is a Bayesian active-testing loop over a pool of task configurations defined by three factors: object position on an 11×11 grid, table height, and scene camera viewpoint. A probabilistic surrogate — a Gaussian process that learns per-factor relevance, a mixture density network, or a deep ensemble — maps each configuration to a predicted score with an uncertainty estimate. An acquisition function ranks unevaluated configurations by expected information gain; the top one runs on real hardware, its graded outcome joins the training data, and the loop repeats to a 100-trial budget. The best combination is a deep ensemble with BALD (Bayesian Active Learning by Disagreement), which pref","core_discovery":"Bayesian active testing — a probabilistic surrogate choosing each next configuration to run — estimates a generalist robot policy's performance distribution more efficiently than uniform random testing. With a 100-trial budget, the active surrogate matches random testing's best RMSE in 20–40 fewer trials and its best log-likelihood in 50–65 fewer trials across three tasks. The same surrogate predicts out-of-distribution performance, ranks sensitivity object position > camera viewpoint > table height, and yields lower run-to-run variance. The best pairing is a deep ensemble with BALD, an acquisition function targeting genuine model disagreement.","pith_inferences":["The 20–40% savings are measured offline: active and random selection are replayed from a dataset with one evaluation per configuration. If repeated live executions are noisy, the live savings will likely be smaller — a gap the paper's own noise-floor ablation suggests exists but is bounded.","The same loop transfers naturally to sim-to-real evaluation: run the surrogate cheaply over simulated configurations and let it flag which ones deserve expensive hardware confirmation, concentrating real-robot effort where the sim-to-real gap is most uncertain.","The advantage over random testing likely shrinks if outcomes are binary rather than graded, because the surrogate needs a smooth signal to interpolate, and likely grows as more factors are added — a prediction a rerun with a binary scoring rule could test.","The appendix's finding that adding demonstrations in one factor region can lower outcomes in another hints that the performance map could warn against negative-interference regions before a data-collection budget is spent; the paper records the effect but leaves this use undeveloped."],"forward_implications":["An evaluator with a 100-trial budget reaches the same accuracy of performance-distribution estimates in roughly 60–80 trials, cutting real-hardware evaluation time by 20–40%.","Because the surrogate predicts performance at out-of-distribution factor values, a policy's generalization can be assessed from actively chosen trials instead of exhaustively covering unseen factor combinations.","The measured sensitivity ordering — object position matters most, table height least — tells evaluators which factors deserve denser test coverage and where narrow test suites are most misleading.","Lower run-to-run variance under active selection makes the result of a single evaluation campaign more reproducible, which matters because real-world evaluation is usually run once.","In the data-curation extension, demonstrations at actually observed failure points improved overall performance more than demonstrations at surrogate-predicted lowest-outcome points, marking observed failure points as more influential."],"fun_headline_variants":["Active testing saves 20-40% of robot evaluation trials","Bayesian test selection maps robot failure zones efficiently","Smarter test choices: active evaluation cuts robot trials by 20-40%","Adaptive robot evaluation finds performance limits with fewer runs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise, stated in Section IV.A, is that offline replay from a ground-truth dataset with a single evaluation per configuration — assuming the policy's outcome at a fixed configuration is effectively deterministic — faithfully represents online sequential evaluation; if real executions are noisy, the reported 20–40% savings may not transfer to live testing.","fun_headline_variants_meta":{"raw":{"variants":["Active testing saves 20-40% of robot evaluation trials","Bayesian test selection maps robot failure zones efficiently","Smarter test choices: active evaluation cuts robot trials by 20-40%","Adaptive robot evaluation finds performance limits with fewer runs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000567,"raw_usage":{"total_tokens":2493,"prompt_tokens":682,"completion_tokens":1811,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":426,"completion_tokens_details":{"reasoning_tokens":1741}},"tokens_in":426,"tokens_out":1811,"duration_ms":13020,"temperature":1.0,"reasoning_tokens":1741,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T02:04:07.373880+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the selection loop live on the robot with no pre-collected answer key: choose a configuration by the acquisition function, execute it, feed the outcome back, and repeat for 100 trials, comparing the surrogate's RMSE against a live random-testing baseline while repeating several configurations multiple times to measure outcome noise. If the live active curve still beats random by 20–40 trials, the claim holds; if the gap shrinks or vanishes once noise is present, the offline-replay design produced the savings.","supporting_citations":[],"review_version":1}