Robotics · arXiv:2610.24801 v1 · submitted 17 Oct 2026 · 25 references
When Better Policies Leave Harder Failures: Coverage Shift in VLM Success Monitoring
The paper says
“Can a better robot policy become harder to monitor? We study a fixed vision-language model (VLM) that judges success while a policy improves. In a controlled 20-seed tray-placement experiment, training selects only simulator-verified successes.”
more of the abstract
Success rises from 66.8% to 97.0%, and accepted hover failures fall from 5.0% to 0.9% of executions. Yet the judge accepts 50.0% of remaining hovers, versus 31.1% under untrained sampling. These residual failures more often lie below the tray rim, which can conceal the gap beneath a grasped cube. A separate boundary probe accepts 148/160 below-rim hovers and 0/160 above-rim hovers. We call this change in conditional monitoring sensitivity coverage shift: learning changes which failures reach a fixed judge. A broader diagnosis covers nine judges and 1,392 stimuli. Judge-filtered experiments across eight settings further show heterogeneous effects of state-dependent acceptance. The results distinguish task improvement from failure detectability and motivate reporting absolute misses, conditional misses, and audit coverage together.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 20 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. A clean, honest demonstration of a real monitoring failure mode: when the policy improves, residual failures shift into a region the VLM misses. The seed-reuse concern is real but acknowledged, and the paper's intellectual honesty earns it a send to review. This line is wrong
2 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 25 references say.
The paper says, at Sec. 5.1, Table 2, line b267
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.86):
“Seed reuse and imperfect historical reproduction limit confirma- tion.”
Pith says: Objection 1 of 2 · major
The central monitoring comparison (C1 vs N0) uses 20 seeds that are reused from the earlier F1 follow-up (seeds 5–24). As the paper acknowledges, this is not an independent confirmatory replication. The evidence for the 18.9 pp increase in conditional hover acceptance therefore comes from the same scene configurations used during the exploratory phase that motivated the study.
more of objection 1
The protocol was likely frozen before execution, but the non-independence of seeds means that the observed effect may be tied to those specific scenes. The paper should discuss whether any independent evidence (e.g., from scene seeds not used in F1) supports the same conclusion, or, if none exists, state explicitly that the central result currently rests on a single non-independent seed set. This is a load-bearing limitation for the paper's main quantitative claim.
The paper says, at Sec. 5.1, Table 2, Fig. 5
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.84):
“Inde- pendent prospective prediction, neural-policy confirmation, real-image boundary tests, and equal-label-budget auditing have no completed formal evidence here.”
Pith says: Objection 2 of 2 · major
The central monitoring result is established in a single judge–task–view setting (M1, tray, V1). While the paper acknowledges this scope limitation (b264, b275), the broader claim of 'coverage shift' as a general phenomenon is supported only by this one setting. The paper would benefit from explicitly stating that generalization to other judges, tasks, views, and neural policies remains untested and that the strength of the evidence is limited to the specific controlled conditions studied.
5 minor objections
The paper says, at Sec. 3.2, Eq. (3)
Pith says: minor objection 1 of 5
The refitting objective in Eq. (3) is described as the 'objective, not an exact unconstrained density transformation.' The subsequent paragraph mentions smoothing, clipping, and constraining. It would be helpful to state more directly that the update is not a pure maximum-likelihood fit and that the fitted family is misspecified relative to the initial uniform distribution, as this affects the interpretation of the R0 control.
The paper says, at Sec. 5.1, Table 2
Pith says: minor objection 2 of 5
Table 2 reports count-pooled values (e.g., 86/175 = 49.14%) alongside seed-mean conditional acceptance (49.98%). The discrepancy illustrates the point in Eq. (1) about seed-level versus pool-level ratios, but the presentation is clear. Consider adding a footnote explaining that the seed-mean is the inferential target and the pooled ratio is descriptive.
show 3 more
The paper says, at Sec. 5.2, Table 3
Pith says: minor objection 3 of 5
Table 3 reports C3–C2* differences with 95% paired-bootstrap intervals that are not multiplicity-adjusted. The paper notes this. Given eight related cells, a brief justification for reporting unadjusted intervals (e.g., the contrasts are treated as descriptive rather than hypothesis tests) would aid the reader.
The paper says, at Fig. 1 caption
Pith says: minor objection 4 of 5
The figure caption mentions 'Revised common-reset run: 20 seeds, rounds 1–5, pooled height-bin ratios.' It would be clearer to state that the confidence intervals in subfigure (c) are equal-seed paired t intervals, as reported in the text.
The paper says, at Sec. 6, Discussion
Pith says: minor objection 5 of 5
The Discussion says 'A useful audit should give task success, failure incidence, absolute accepted failures, and conditional acceptance for specified failure sets, together with counts and the states sampled.' This is a practical recommendation, but it is not accompanied by any formal validation. Consider qualifying it as a suggestion based on the paper's results rather than a proven prescription.
What would settle it
Repeat the common-reset protocol (N0/R0/C1, M1/tray/V1, 20 seeds, 100 rollouts per arm per round, rounds 1–5) using entirely new scene seeds (e.g., 100–119) that have never been used in any prior experiment in this project. Compute the paired difference in conditional hover acceptance (C1 – N0) and its 95% confidence interval. If the interval excludes zero and the point estimate lies within 10 percentage points of 18.9, the coverage-shift result is supported; otherwise, the claim weakens.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 20 Oct 2026
- Reader holds up
- Skeptic agrees
- Editor wrote the verdict above
- Referee minor revision: 2 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Isabel and coauthors show something intuitive that hasn't been cleanly shown before: a better robot policy can leave fewer failures but make those failures harder for a fixed VLM judge to detect. The core experiment (truth-filtered training against untrained sampling, 20 seeds, paired comparisons, honest intervals) is well designed and executed. The boundary probe is a nice mechanistic check: below-rim hovers pass 148/160, above-rim hovers pass 0/160. The paper also does the right thing by reporting null and negative results (static prediction fails, repair criterion fails, historical discrepancies) instead of sweeping them under the rug.
The soft spot that matters is the seed reuse. The central 20-seed monitoring comparison uses the same scene seeds (5–24) as the earlier F1 follow-up, so it is not an independent confirmatory replication. The authors say this clearly (b267). It means the headline effect—18.9 percentage point increase in conditional hover acceptance—could in principle depend on those particular scenes. That said, the mechanism is plausible, the boundary probe is separate evidence, and the paper does not claim generality beyond its tested conditions. I would have liked to see a replication on a fresh set of scene seeds, but given the paper's own transparent limits, this is a proportionate concern, not a fatal flaw.
Who is this for? Anyone working on robot learning with VLM-based monitoring or success detection, and anyone interested in evaluation pitfalls under distribution shift. It is a solid empirical contribution that earns its place. I would send this to peer review without hesitation. The authors have done the hard work of designing controlled comparisons and reporting honestly what they found, including what they did not find.
The strongest objection
The core monitoring comparison (C1 vs N0, 20 seeds, M1/tray/V1) is conducted with the same scene seeds (5–24) used in the earlier F1 follow-up. Although the paper acknowledges this reuse (b267), it means the evidence is not an independent confirmatory replication. The central claim—that truth-filtered training increases conditional hover acceptance by 18.9 percentage points—may depend on the particular scene configurations realized by those seeds. A new set of scene seeds (e.g., 100–119) would test whether the coverage shift generalizes beyond the original scenes, or whether the observed effect is tied to idiosyncratic scene geometry.
The reader's weakest assumption already flags the single-setting and seed-reuse limitation. My proposed test directly addresses that limitation. The paper is transparent and has strong internal validity (boundary probe, consistent height-strata counts, 20 seeds). Thus the ACCEPT verdict remains appropriate; no adjustment is needed.
The referee's report
If the central finding stands, it provides a concrete counterexample to the assumption that improving a policy necessarily makes failures easier for a fixed monitor to detect. The paper isolates a mechanism—geometric coverage shift near a task boundary—and supports it with both an executed-state audit and a controlled boundary probe (F1). The experiments are well-executed: 20 seeds, common reset, paired comparisons, pre-specified endpoints (or frozen protocols), honest confidence intervals, and thorough reporting of limitations. The permutation intervention (C2*) further demonstrates that the assignment of judge errors to individual states can affect learning outcomes, with substantial variation across settings. The paper's contribution is empirical and diagnostic rather than algorithmic, and it responsibly distinguishes the central truth-filtered monitoring result from the subsequent judge-filtered experiments.
Pith's reading of the paper (a machine's summary, not the authors')
The paper establishes a phenomenon called coverage shift: when a robot policy improves through truth-filtered training, its failures become less frequent but also less detectable by a fixed vision-language model (VLM) judge. In the central tray-placement experiment, success rises from 66.8% to 97.0%, and accepted hovers per execution fall from 5.0% to 0.9%. Yet the judge accepts 50.0% of remaining hovers, up from 31.1% under untrained sampling. The residual failures concentrate below the tray rim, where a separate probe shows the judge accepts 148/160 hovers, versus 0/160 above the rim. The authors argue that reporting both absolute and conditional misses, along with the states the policy visits, is necessary for truthful monitoring.
What carries the argument. Coverage shift, quantified by conditional acceptance $c_t = \int_B a_j(x) \, d_t(x \mid B)$, where learning alters the distribution $d_t$ of physical outcomes within a failure set $B$, while the judge's acceptance function $a_j$ remains fixed. The boundary probe (F1) exposes a missing region of state space (below the tray rim) where visual evidence of the failure is occluded. The paper also uses a permutation intervention (C2*) that randomizes which states receive judge acceptance within truth strata to isolate effects of state assignment.
Core claim. The central discovery is that truth-filtered training (C1) in a tray-placement task, compared to untrained sampling (N0), raises the judge's acceptance conditional on a hover from 31.1% to 50.0% (paired difference 18.9 pp, 95% CI [9.7, 28.1]) while absolute accepted hovers per execution fall from 5.0% to 0.9%. The residual hovers shift below the tray rim, where a boundary probe accepts 148/160 below-rim hovers versus 0/160 above-rim hovers. This 'coverage shift' occurs because learning changes the mixture of failure states evaluated by a fixed judge, without any change in the judge itself. The paper also shows via permutation experiments that state-dependent judge acceptance can affect subsequent learning outcomes.
The premise it rests on. The central truth-filtered effect (increased conditional acceptance despite lower absolute accepted failures) is established in only one judge–task–view setting (M1/tray/V1), using a revised protocol developed after earlier observations that reused seed numbers; the generalization to other judges, tasks, views, and neural policies has not been demonstrated.
If the paper is right (Pith's inferences, not the authors')
Evaluating a VLM judge on a static diagnostic set can miss distribution changes; monitoring reports should always include the actual failure states the current policy visits.
Reporting only absolute accepted failures per execution can obscure a rising conditional miss rate; both denominators are necessary.
A judge with a low overall false positive rate on a diagnostic set may still have critical blind spots if residual failures concentrate there.
Checking views and stricter thresholds can help but may not resolve the coverage gap and can even harm learning by discarding useful successes.
The state-dependent error structure of a judge affects downstream learning, not just static prediction accuracy.
The same coverage shift mechanism may apply to any learned system where a fixed monitor (reward model, classifier, human reviewer) evaluates a shifting data distribution—for example, in language model alignment with reward models that are evaluated on static test sets.
The boundary probe methodology (identifying a success/failure boundary region where visual evidence is ambiguous) could be systematized as an auditing tool for VLM judges before deployment.
The paper's failed T1 transfer test (static error map does not predict learned policy failures) suggests that reliance on pre-training static evaluations for monitoring readiness is insufficient; prospective evaluation on the actual failing states is needed.
The permutation experiment (C2*) shows that state-dependent acceptance has causal effects on learning, implying that coverage shift is not just a statistical observation but a design concern for self-improving systems.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Can a better robot policy become harder to monitor? We study a fixed vision-language model (VLM) that judges success while a policy improves. In a controlled 20-seed tray-placement experiment, training selects only simulator-verified successes. Success rises from 66.8% to 97.0%, and accepted hover failures fall from 5.0% to 0.9% of executions. Yet the judge accepts 50.0% of remaining hovers, versus 31.1% under untrained sampling. These residual failures more often lie below the tray rim, which can conceal the gap beneath a grasped cube. A separate boundary probe accepts 148/160 below-rim hovers and 0/160 above-rim hovers. We call this change in conditional monitoring sensitivity coverage shift: learning changes which failures reach a fixed judge. A broader diagnosis covers nine judges and 1,392 stimuli. Judge-filtered experiments across eight settings further show heterogeneous effects of state-dependent acceptance. The results distinguish task improvement from failure detectability and motivate reporting absolute misses, conditional misses, and audit coverage together.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-20
The authors' main claim, as the classifier picked it out 0.83
“We call this change in conditional monitoring sensitivity coverage shift: learning changes which failures reach a fixed judge.”
How the main claim is stated: asserted 0.99
What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.67 · human participants 0.03 · controlled comparison 0.89 · randomized assignment 0.06 · states quantitative result 0.99 · compares with prior results 0.77 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Can a better robot policy become harder to monitor? | background 0.69 | 0.06 | 0.05 | unclear 0.56 |
| We study a fixed vision-language model (VLM) that judges success while a policy improves. | contribution 0.84 | 0.77 | 0.03 | asserted 0.92 |
| In a controlled 20-seed tray-placement experiment, training selects only simulator-verified successes. | method 1.00 | 0.91 | 0.03 | asserted 1.00 |
| Success rises from 66.8% to 97.0%, and accepted hover failures fall from 5.0% to 0.9% of executions. | result 1.00 | 0.94 | 0.07 | asserted 1.00 |
| Yet the judge accepts 50.0% of remaining hovers, versus 31.1% under untrained sampling. | result 1.00 | 0.94 | 0.13 | asserted 1.00 |
| These residual failures more often lie below the tray rim, which can conceal the gap beneath a grasped cube. | interpretation 0.68 | 0.88 | 0.06 | hedged possibility 0.54 |
| A separate boundary probe accepts 148/160 below-rim hovers and 0/160 above-rim hovers. | result 0.89 | 0.93 | 0.06 | asserted 1.00 |
| We call this change in conditional monitoring sensitivity coverage shift: learning changes which failures reach a fixed judge. | interpretation 0.81 | 0.83 | 0.69 | asserted 0.99 |
| A broader diagnosis covers nine judges and 1,392 stimuli. | method 0.66 | 0.92 | 0.05 | asserted 0.98 |
| Judge-filtered experiments across eight settings further show heterogeneous effects of state-dependent acceptance. | result 0.59 | 0.92 | 0.06 | asserted 1.00 |
| The results distinguish task improvement from failure detectability and motivate reporting absolute misses, conditional misses, and audit coverage together. | interpretation 0.84 | 0.77 | 0.68 | asserted 0.98 |
What kind of paper the classifier reports this is: empirical study 0.93 · evidence from observation or experiment 0.86
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-20: 1 request, 7,984 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (9)
Figures are the authors', reproduced from the paper.
References (25)
- [1] Eric Arazo, Diego Ortego, Paul Albert, Noel E. O’Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In International Joint Conference on Neural Networks, pages 1–8, 2020 on Pith
- [2] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhao- hai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Jun... on PithPith's reviewarXiv
- [3] Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Co- line Devin, Alex X. Lee, Maria Bauza, Todor Davchev, Yux- iang Zhou, Agrim Gupta, et al. RoboCat: A self-improving generalist agent for robotic manipulation. Transactions on Machine Learning Research, 2023 on Pith
- [4] Pieter-Tjerk de Boer, Dirk P . Kroese, Shie Mannor, and Reuven Y . Rubinstein. A tutorial on the cross-entropy method. Annals of Operations Research , 134(1):19–67, 2005 on Pith
- [5] Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision-Language Models as Success Detectors. arXiv preprint arXiv:2303.07280, 2023 on PithPith's reviewarXiv
- [6] Jiafei Duan, Wilbert Pumacay, Nishanth Kumar, Yi Ru Wang, Shulin Tian, Wentao Yuan, Ranjay Krishna, Dieter Fox, Ajay Mandlekar, and Yijie Guo. AHA: A vision-language-model for detecting and reasoning over failures in robotic manipu- lation. In International Conference on Learn... on Pith
- [7] Kazuki Egashira, Mark Vero, Jasper Dekoninck, Florian E. Dorner, Robin Staab, and Martin Vechev. Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RL VR. InConference on Language Modeling, 2026 on Pith
- [8] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866, 2023 on Pith
- [9] Sunghwan Kim, Dongjin Kang, Taeyoon Kwon, Hyungjoo Chae, Dongha Lee, and Jinyoung Y eo. Rethinking reward model evaluation through the lens of reward overoptimiza- tion. In Annual Meeting of the Association for Computational Linguistics, 2025 on Pith
- [10] Tony Lee, Andrew Wagenmaker, Karl Pertsch, Percy Liang, Sergey Levine, and Chelsea Finn. RoboReward: General- Purpose Vision-Language Reward Models for Robotics. arXiv preprint arXiv:2601.00675, 2026 on PitharXiv
- [11] Zeyi Liu, Arpit Bahety, and Shuran Song. REFLECT: Sum- marizing robot experiences for failure explanation and correc- tion. In Conference on Robot Learning , pages 3468–3484, 2023 on Pith
- [12] Fiona Luo. Vision-language models for robot success detec- tion. Proceedings of the AAAI Conference on Artificial Intel- ligence, 38(21):23750–23752, 2024 on Pith
- [13] Jiaxuan Luo, Xingguo Xu, Shanshan Wang, Yuhan Zhou, and Zhen Zhang. CriticHack: Evaluating Visual Re- wards Under Robot Policy Optimization. arXiv preprint arXiv:2610.02527, 2026 on PithPith's reviewarXiv
- [14] David Manheim and Scott Garrabrant. Categorizing variants of Goodhart’s law. arXiv preprint arXiv:1803.04585, 2018 on PithPith's reviewarXiv
- [15] Zaruhi Navasardyan, Tatul Danielyan, and Hrant Davtyan. FailBench: How Reliable are VLMs at Judging Robot Task Success? arXiv preprint arXiv:2609.03611, 2026 on PithPith's reviewarXiv
- [16] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mah- moud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Y ao Huang, Shang-Wen Li, Ishan Misra, Michael R... on Pith
- [17] Juan Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, and Moritz Hardt. Performative prediction. In International Con- ference on Machine Learning, pages 7599–7609, 2020 on Pith
- [18] Qwen Team. Qwen3.8-27B. Official model card, 2026. Archived experiment revision: 1d4bf0f on Pith
- [19] Dushyant Rajput, Nirdesh Chauhan, and Siddharth Kosaraju. Cheap Verifiers, Large Blind Spots: Measuring the Re- liability Cost of Cost-Saving Cascades. arXiv preprint arXiv:2609.01345, 2026 on PithPith's reviewarXiv
- [20] Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision-language models are zero- shot reward models for reinforcement learning. In Interna- tional Conference on Learning Representations , 2024 on Pith
show 5 more
- [21] Sumedh A. Sontakke, Jesse Zhang, Sébastien M. R. Arnold, Karl Pertsch, Erdem Bıyık, Dorsa Sadigh, Chelsea Finn, and Laurent Itti. RoboCLIP: One demonstration is enough to learn robot policies. In Advances in Neural Information Pro- cessing Systems, 2023 on Pith
- [22] Benedikt Stroebl, Sayash Kapoor, and Arvind Narayanan. The Limits of Inference Scaling Through Resampling. arXiv preprint arXiv:2411.17501, 2024 on PitharXiv
- [23] Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse-kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnav Gurha, Viswesh Nagaswamy Rajesh, Y ong Woo Choi, Y en-Ru Chen, Zhiao Huang, Roberto Calandra, Rui ... on Pith
- [24] Xueru Wen, Jie Lou, Y aojie Lu, Hongyu Lin, Xing Yu, Xinyu Lu, Ben He, Xianpei Han, Debing Zhang, and Le Sun. Re- thinking Reward Model Evaluation: Are We Barking up the Wrong Tree? In International Conference on Learning Rep- resentations, 2025 on Pith
- [25] Zhiyuan Zhou, Pranav Atreya, Abraham Lee, Homer Walke, Oier Mees, and Sergey Levine. Autonomous Improvement of Instruction Following Skills via Foundation Models. In Conference on Robot Learning, 2024. 10 Supplementary Material This supplement documents the completed evidence ... on Pith
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment