{"id":"e586cee3-23f3-4407-8b2d-eb7e4ed979c1","arxiv_id":"2506.12374","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"AntiGrounding lifts candidate robot trajectories into the VLM's visual space via multi-view rendering and structured VQA, and reports 57.5% average success across eight manipulation tasks, beating three intermediate-representation baselines.","lead":"This paper introduces AntiGrounding, a robot control framework that renders candidate trajectories as colored overlays on camera images and asks a vision-language model to score them from multiple viewpoints, selecting the best in a closed loop. It reports higher task success than three prior methods across eight tabletop manipulation tasks, in simulation and on a real robot.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core selection mechanism—VLM scoring of rendered trajectory overlays (Eqs. 3-4)—has no direct validation, and the paper's own error breakdown (Sec. 4.4) and Appendix A.8 concede that the required fine-grained geometric precision is a major weak point.","rationale":"The framework is coherent and the ablation study suggests each component contributes, which is a point in its favor. The multi-view rendering and structured VQA are a reasonable way to elicit spatial judgments from a VLM. However, the entire decision-making loop rests on the assumption that numeric scores extracted from VLM text responses reliably order trajectories at the centimeter/degree scale needed for tape stacking, pouring, and shape matching. That assumption is directly challenged by the authors' own error analysis and limitations discussion. A direct rank-correlation test would settle whether the premise holds, independent of the other pipeline components. If the test fails, the central claim of 'optimal closed-loop trajectory synthesis' falls; if it passes, the conditional acceptance can be upgraded. Secondary issues—missing code, no error bars in Table 1, and the Water Pouring/Table 3 inconsistencies—further reduce confidence in the reported numbers, but they are not the deepest vulnerability. I therefore keep the reader's CONDITIONAL verdict and recommend the scoring-validation experiment as the decisive check.","tokens_in":28161,"tokens_out":4329,"duration_ms":51089,"concrete_test":"Run a controlled trajectory-ranking study on one task (e.g., Circular Tape Stacking): sample the same candidate trajectory set used by the MPC, render the multi-view overlays exactly as in Sec. 3.3, and have the AntiGrounding VLM ensemble score them without any execution feedback. Independently obtain ground-truth success labels by executing each candidate in the calibrated simulator (or on the real robot with multiple trials). Compute Kendall's tau between VLM scores and ground-truth success, and compare the success rate of the VLM argmax choice against an oracle argmax and random selection. If tau is not significantly positive or VLM argmax is near random, the central selection mechanism is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that pre-trained VLMs can rank candidate trajectories in their native representation space well enough to choose optimal closed-loop actions. This requires the VLM scores S_{j,t} in Eq. (3) to be positively correlated with real task success, and the argmax in Eq. (4) to select near-best trajectories. The paper never measures this correlation. It reports only end-of-task success, which couples VLM scoring with trajectory sampling, constraint filtering, sim-to-real transfer, and execution. Section 4.4 states that VLM evaluation errors dominate the failure breakdown, and Appendix A.8 concedes that 'fine-grained, metric geometric constraints (e.g., sub-centimeter concentricity...) purely from 2D renderings remains a significant challenge for current VLMs.' This is precisely the regime the method targets. The absence of a direct scoring-accuracy measurement is therefore load-bearing: if VLM scores are weakly correlated with physical feasibility, the argmax selection is essentially arbitrary. The empirical tables also give no error bars for the main comparison (Table 1) and are internally inconsistent (Water Pouring 5/10 vs Rekep 6/10 despite overall superiority; Table 3 'No Prior' numbers, e.g., 55.24% for Circular Tape Stacking, do not match Table 1's 60%). These inconsistencies reinforce, but do not replace, the core concern: the VLM-as-evaluator premise is asserted, not demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AntiGrounding, a framework that inverts conventional instruction-grounding pipelines by rendering candidate robot trajectories from multiple viewpoints and asking a pre-trained VLM ensemble to score them via structured visual question answering. The highest-scoring trajectory is executed in a closed-loop MPC, with an optional offline meta-VLM module that refines the evaluation template from prior execution logs. The authors report success rates on eight real-robot manipulation tasks, comparing against Code as Policies, Voxposer, and Rekep, and claim zero-shot synthesis of closed-loop trajectories for novel tasks. The paper also includes ablations, an offline policy-optimization study, and a system error breakdown.","tokens_in":28500,"tokens_out":2322,"duration_ms":29024,"significance":"If the central premise holds, the paper contributes an interesting alternative to intermediate-representation approaches: instead of compressing VLM knowledge into symbolic skills or value maps, it uses rendered trajectory overlays as a native interface to the VLM's spatial and semantic priors. The use of an ensemble of five pre-trained VLMs in a closed-loop MPC, without task-specific fine-tuning, is a distinctive and potentially useful design point. The paper also honestly documents its own limitations in Section 4.4 and Appendix A.8, including that VLM evaluation errors dominate and that fine-grained metric geometric constraints remain a challenge. However, the empirical support for the central premise is currently incomplete: the manuscript never directly measures whether the VLM scores in Eq. (3) are positively correlated with physical task success, and the reported numerical comparisons suffer from inconsistencies and lack of statistical rigor. The concept is promising, but the evidence as presented is not yet commensurate with the strength of the central claim.","major_comments":[{"comment":"The load-bearing assumption is that the VLM scores S_{j,t} in Eq. (3) and the argmax in Eq. (4) reliably rank physically feasible trajectories. The paper never validates this correlation directly, instead reporting end-of-task success that couples VLM scoring with trajectory sampling, constraint filtering, and execution. Section 4.4 states that VLM evaluation errors dominate the failure breakdown, and Appendix A.8 concedes that \"fine-grained, metric geometric constraints (e.g., sub-centimeter concentricity...) purely from 2D renderings remains a significant challenge for current VLMs.\" This is precisely the regime the method targets. I ask the authors to provide a direct scoring-accuracy measurement, for example by comparing VLM rankings against ground-truth or human-annotated trajectory quality on a held-out set of candidate trajectories, with per-task correlation coefficients or top-k selection accuracy. Without this, the central claim that anti-grounding enables reliable decision making remains asserted rather than demonstrated.","section":"Section 3.3, Eqs. (2)-(4); Section 4.4; Appendix A.8"},{"comment":"The success numbers are internally inconsistent. Table 1 reports AntiGrounding success of 6/10 for Circular Tape Stacking, while Table 3 reports 55.24% for the \"No Prior\" condition on the same task; Waste Sorting is 7/10 in Table 1 but 68.49% in Table 3. Additionally, in Table 1, Rekep beats AntiGrounding on Water Pouring (6/10 vs. 5/10) despite AntiGrounding's higher aggregate score. The authors need to clarify whether these tables use different trial counts, different task configurations, or different evaluation criteria, and report the exact number of trials and the definition of success for each table. As written, the inconsistencies undermine the quantitative comparison.","section":"Table 1 vs. Table 3"},{"comment":"The main comparison in Table 1 is based on 10 trials per task with no confidence intervals, error bars, or significance tests, and Table 2 reports ablation variants from a single run while only AntiGrounding is averaged over three runs. Given the high variance suggested by the AntiGrounding ablation standard deviations in Table 2 (e.g., 46.67±9.87 for Water Pouring), the differences between AntiGrounding and Rekep on individual tasks (e.g., 5/10 vs. 6/10 on Water Pouring) could easily be noise. The authors should provide per-task confidence intervals and a significance test (or at least additional repetitions) for the main comparison, and report multiple runs for the ablations.","section":"Section 4.1, Table 1; Section 4.2, Table 2"},{"comment":"The offline policy refinement is evaluated on the same tasks whose execution outcomes are used to tune the sub-question weights and templates, which risks overfitting and makes the \"improvement\" numbers in Table 3 partly circular. The paper should clarify whether the offline refinement was evaluated on held-out task instances or only on the same task set, and ideally report performance on tasks not used for refinement. In addition, Table 3 does not state the number of trials per entry or whether the same trial set was used for both conditions; these details are necessary to assess the reported improvements.","section":"Section 4.3 and Appendix A.6"}],"minor_comments":[{"comment":"The sentence \"Extensive experiments in both simulation and real-world environments demonstrate that AntiGrounding significantly outperforms conventional methods... particularly in complex manipulation tasks in complex manipulation tasks encompassing precise spatial reasoning\" contains a duplicated phrase that should be removed.","section":"Section 1"},{"comment":"The caption contains a typo: \"Sim Buider\" should be \"Sim Builder.\"","section":"Figure 3 caption"},{"comment":"The sentence \"Each model processes the same pipline in parallel\" contains a typo: \"pipline\" should be \"pipeline.\"","section":"Appendix A.5.1"},{"comment":"The Water Pouring task is described as pouring from a \"green cup\" into a \"stainless steel bowl,\" whereas Section 1 and Figure 7 refer to a teapot and cup; please reconcile the object descriptions across the paper.","section":"Appendix A.2.3"},{"comment":"The ablation paragraph says AG-MVECW \"led to unstable trajectory assessments\" and AG-VGTG \"severely affected trajectory quality,\" but the corresponding rows in Table 2 do not include error bars or significance tests; the text should be moderated to match the evidence presented.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising core idea and a reasonably complete system description, but the empirical validation is currently too thin for the strength of the central claim. The missing direct validation of VLM scoring accuracy is the key technical gap, and the numerical inconsistencies across tables need to be resolved before the paper can be considered for publication. I also note that several baselines are evaluated with very few trials, which may bias the comparison; the authors should be asked to provide additional trials or statistical evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a system-building paper, not a proof that VLM-as-evaluator works. The integration is new-ish: multi-view adaptive evaluation, five-model ensemble, real2sim2real, offline template refinement. Prior work (PIVOT, VLMPC) already rendered candidate actions and asked the VLM to choose; this paper adds engineering on top and evaluates on eight tabletop tasks. That evaluation is the main contribution, and it is not strong. No error bars for baselines, no significance tests, single-run ablations, and Table 3 numbers disagree with Table 1 for the same method (Circular Tape Stacking 55.24% vs 60%). Water Pouring is worse than Rekep (5/10 vs 6/10) despite the overall claim. The offline refinement tunes sub-question weights on the same tasks' outcomes and then reports improvement on those tasks—that is leakage, not learning.\n\nThe deeper problem is the load-bearing premise: Eq. (3) aggregates VLM scores and Eq. (4) takes the argmax, but the paper never checks whether those scores actually correlate with task success. The paper's own Section 4.4 says VLM evaluation errors dominate failures, and Appendix A.8 concedes that fine-grained metric geometry from 2D renderings remains a challenge. That is precisely the regime the method targets. End-to-end success rates give indirect evidence that the VLM is doing something useful, but they conflate scoring with sampling, constraint filtering, and sim-to-real transfer. A direct scoring-accuracy measurement would be cheap and decisive.\n\nTo its credit, the paper is honest. The limitations section and appendix are unusually candid about failure modes, and the system description is detailed enough to reproduce approximately. The real2sim2real pipeline and the rigid-coupling rotation module are thoughtful engineering. The idea of keeping the VLM in its native representation space is worth taking seriously even if this particular evaluation does not nail it.\n\nWho is this for? Robotics researchers interested in VLM-based trajectory evaluation and real2sim2real systems. It deserves a serious referee: the framework is coherent, the claim is testable, and the weaknesses are fixable. I would recommend a conditional accept with major revision: release code and data, add variance and significance tests, fix the table inconsistencies, directly validate the scoring mechanism, and report offline refinement in a way that separates tuning on the test tasks from genuine adaptation.","headline":"A plausible system paper with an honest limitations section, but the core claim—VLM scores rank trajectories well enough to drive MPC—is never directly tested, and the reported numbers have inconsistencies.","tokens_in":29038,"tokens_out":2847,"would_cite":false,"duration_ms":29435,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AntiGrounding lets a pretrained vision-language model pick robot trajectories directly from rendered images.","keywords":["vision-language models","robotic manipulation","trajectory evaluation","visual question answering","model predictive control","zero-shot learning","real2sim2real","closed-loop control"],"falsifier":"Take a fixed scene, generate candidate trajectories whose true success or failure is known from execution, render them with the paper's multi-view protocol, and ask the VLM ensemble to rank them. If the top-ranked trajectory fails more often than a randomly chosen feasible trajectory, or if ranking accuracy on fine-grained geometric perturbations is at chance, the central claim is falsified.","tokens_in":27958,"feed_emoji":"🤖","tokens_out":5250,"duration_ms":60477,"temperature":0.7,"pith_summary":"This paper tries to show that a pre-trained vision-language model (VLM) can act as the decision-maker for fine-grained robot manipulation without task-specific training or intermediate symbolic representations. The idea is to 'anti-ground': instead of compressing an instruction into skills or constraints, the system renders each candidate robot trajectory as colored overlays on images from several camera views and lets the VLM answer structured questions about safety, task alignment, efficiency, and physical feasibility. The scoring is wrapped in a closed-loop model predictive controller, and an offline meta-review refines the scoring template from logged experience. Across eight tabletop tasks, the method reports a 57.5% overall success rate, versus 41.25% for the strongest baseline family tested. If the claim holds, general-purpose VLMs can serve as trajectory evaluators for novel tasks with zero fine-tuning.","feed_headline":"VLM-as-judge lifts robot task success to 57.5 percent","feed_subtitle":"Rendering candidate paths as colored overlays lets a pretrained vision-language model evaluate them in closed loop.","key_machinery":"The load-bearing object is the trajectory-based structured VQA evaluation inside the MPC loop. Candidate end-effector trajectories are sampled under an annealed exploration bias, filtered by collision and target-interaction constraints, and then color-coded and rendered onto multi-view scene images. A VLM ensemble answers four weighted sub-questions—safety, task alignment, efficiency, and physical feasibility—plus a viewpoint-clarity question; viewpoint confidence weights suppress unreliable views, and the aggregated score ranks trajectories, with the top scorer selected by the argmax rule. The 'anti-grounding' move is that the robot action itself becomes the visual input to the VLM rather than being converted to a compressed intermediate representation.","core_discovery":"The central discovery is that reversing the instruction-grounding pipeline removes the information bottleneck. The paper argues that existing methods project a language instruction into compressed intermediate forms—symbolic skill sequences, voxel value maps, or keypoint constraints—and that this discards fine-grained spatial and physical detail the VLM has already encoded. AntiGrounding instead lifts candidate action trajectories into the VLM's native representation space by rendering them from robot-centric viewpoints and evaluating the augmented images with a structured visual-question-answering template. Scores from an ensemble of VLMs are aggregated across sub-questions, views weighted by viewpoint confidence, and agents, and the argmax trajectory is executed in a closed-loop MPC. The paper reports that this zero-shot procedure outperforms all three baseline families on the tested manipulation suite, and that the optional offline policy refinement further raises success rates by adjusting the evaluation template from past execution outcomes.","pith_inferences":["The mechanism is not tied to a specific robot arm: any planner that can propose candidate trajectories and render them could adopt the same VLM-as-judge loop, potentially extending to mobile manipulation, navigation, or bimanual tasks.","A direct stress test would measure how the VLM ensemble's trajectory ranking degrades as candidate differences shrink below a few centimeters or degrees; the paper's own error analysis suggests this metric-level discrimination is the fragile point.","The offline refinement could become an online self-supervised loop that uses execution success flags as reward labels to tune template weights, a step the paper leaves for future work.","Because evaluation is purely image-and-text based, the framework could combine with other search strategies beyond the spherical sampler to explore higher-dimensional or contact-rich trajectory spaces."],"forward_implications":["A single pretrained VLM can serve as the cost function for closed-loop trajectory optimization, so a new manipulation task requires only a language instruction rather than reward engineering or fine-tuning.","Multi-view rendering with viewpoint confidence weighting lets the VLM infer 3D scene structure without a dedicated 3D encoder.","The offline meta-VLM refinement loop lets the evaluation policy self-correct after failures, so long-horizon performance improves with accumulated experience.","On the reported task suite, the method beats approaches built on symbolic skill sequences, voxel value maps, and keypoint constraints, with the largest gaps on tasks requiring precise alignment and multi-stage sequencing."],"supporting_citations":[{"why":"Supplies the code-as-policies baseline that AntiGrounding outperforms.","marker":"[13]"},{"why":"Supplies the voxel value-map baseline representing compressed spatial representations.","marker":"[14]"},{"why":"Supplies the relational-keypoint baseline motivating the anti-grounding approach.","marker":"[16]"},{"why":"Supplies single-image 3D mesh reconstruction used in the real2sim2real pipeline.","marker":"[94]"},{"why":"Supplies 6D pose estimation used for simulated scene alignment.","marker":"[79]"},{"why":"Supplies physical-property identification used to bridge sim-to-real dynamics.","marker":"[90]"},{"why":"Inspires the trajectory-based visual prompting strategy for VQA evaluation.","marker":"[15]"},{"why":"Supplies the Mixture-of-Agents ensemble strategy for aggregating VLM evaluations.","marker":"[130]"},{"why":"Supplies the simulated-annealing schedule used for the exploration bias in trajectory generation.","marker":"[128]"}],"fun_headline_variants":["AntiGrounding lifts robot success to 57.5% via VLM as judge","VLM judges rendered robot paths for zero-shot manipulation","Reverse grounding: lift actions into VLM space to pick winners","Robot actions rendered into VLM space boost task success rate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole system rests on the VLM's rendered-image scores being good enough to rank trajectories that differ by only centimeter-level placement and degree-level orientation; if those scores do not track real physical feasibility, the argmax selection picks the wrong motion.","fun_headline_variants_meta":{"raw":{"variants":["AntiGrounding lifts robot success to 57.5% via VLM as judge","VLM judges rendered robot paths for zero-shot manipulation","Reverse grounding: lift actions into VLM space to pick winners","Robot actions rendered into VLM space boost task success rate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001028,"raw_usage":{"total_tokens":4285,"prompt_tokens":851,"completion_tokens":3434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":3361}},"tokens_in":467,"tokens_out":3434,"duration_ms":29110,"temperature":1.0,"reasoning_tokens":3361,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:52:06.586516+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed scene, generate candidate trajectories whose true success or failure is known from execution, render them with the paper's multi-view protocol, and ask the VLM ensemble to rank them. If the top-ranked trajectory fails more often than a randomly chosen feasible trajectory, or if ranking accuracy on fine-grained geometric perturbations is at chance, the central claim is falsified.","supporting_citations":[],"review_version":1}