{"id":"146a317a-9135-46d5-91ae-97c185c944e0","arxiv_id":"2508.02629","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"HyCodePolicy closes the loop between generated robot code, visual checkpoint monitoring, and iterative repair, raising average success rates on 10 simulated manipulation tasks by up to 16.5 points over one-shot code-as-policy baselines.","lead":"This paper describes HyCodePolicy, a system that asks a language model to write robot control code, then uses a vision-language model to watch execution snapshots and repair the code when steps fail. It reports higher average success rates on a simulation benchmark, but the gains are inconsistent, lack statistical backing, and come from a test set the system was tuned on.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main reported gains are measured on the same 10-task set used to tune the framework (Sec. 4.5), while the adaptive trial-selection score ψ (Eq. 3.4.1) is undefined and VLM diagnosis is never validated; the central claim therefore lacks a held-out and mechanism-level check.","rationale":"The reader's CONDITIONAL verdict is appropriate, and my stress-test does not move it. The reader's weakest assumption focuses on VLM reliability and trial representativeness; I agree that this is a real gap, but I would place the tuning-set issue earlier in the causal chain. Section 4.5 explicitly admits that the framework architecture, feedback logic, and prompting structure were jointly tuned on the shared subset reported in Table 3, which is also the source of the headline results. Even a perfectly accurate VLM and a well-defined ψ would not make an in-sample comparison establish robustness or sample efficiency on unseen tasks. The paper deserves credit for its detailed appendices, open-source pointer, per-task results, and the absolute zero-shot execution of all 50 Bi2Code tasks in Table 4; those are useful. However, Table 4 has no baseline, so it does not support the comparative claim. The per-task table also reveals non-monotonic effects of adding VLM feedback, which the aggregate numbers hide. These observations strengthen the existing CONDITIONAL verdict rather than changing it: the authors should supply held-out results, statistical variability, and a concrete definition or ablation of ψ before the central claim is accepted.","tokens_in":17847,"tokens_out":5059,"duration_ms":56136,"concrete_test":"Re-run the three-way comparison (Code as Policies, CodeAct, HyCodePolicy) on the 40 held-out Bi2Code tasks in Table 4, using the exact prompts and no further tuning, and report per-condition ASR and CR-Iter. If HyCodePolicy does not outperform CodeAct on these held-out tasks, the generalization and robustness claim fails. As a secondary check, log VLM diagnoses against ground-truth failure labels on a random subset of trials and ablate ψ (e.g., random trial vs. max-severity trial); if repair success does not track diagnostic accuracy or if ψ is inert, the adaptive-repair mechanism is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing issue is the evaluation-design chain supporting the headline numbers. Section 4.5 states that the framework architecture, feedback logic, and prompting structure were jointly tuned only on the shared subset reported in Table 3; those 10 tasks are exactly the tasks in Tables 1 and 3. Hence the reported ASR gains (47.4→63.9, 62.1→71.3) and CR-Iter reduction (2.42→1.76) are in-sample. The full 50-task Table 4 provides no CodeAct or Code-as-Policies baseline, and several zero-rate tasks are impossible through the available API, so it cannot substitute for a held-out comparison. Separately, the method's adaptive mechanism is underspecified: Eq. 3.4.1 defines trial selection through ψ(FailureSeverity_i, TraceDivergence_i), but ψ is never defined or implemented, and the VLM's binary success judgments and causal hypotheses are never checked against ground truth. Table 3 shows cases where adding VLM feedback reduces success (e.g., Handover Block on Bi2Code: 50%→27%; Pick Diverse Bottles on RoboTwin: 65%→64%), which is consistent with misdiagnosis driving bad repairs. The central claim therefore depends on two unverified links: that the tuned tasks are representative, and that the VLM/ψ loop gives correct selective feedback.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces HyCodePolicy, a closed-loop framework for language-conditioned robot manipulation that decomposes an instruction into subgoals, synthesizes Python code grounded in geometric primitives, executes the program in simulation, monitors execution through checkpoint images with a VLM, and fuses symbolic execution logs with VLM perceptual feedback to diagnose and repair failures. The authors also present Bi2Code, a re-engineered interface built on RoboTwin 2.0 with dual-arm API support, structured prompts, symbolic logging, and observation hooks. The main empirical claims are that HyCodePolicy raises average success rates from 47.4% to 63.9% on RoboTwin 1.0 and from 62.1% to 71.3% on Bi2Code, and that it reduces the mean number of repair iterations from 2.42 to 1.76. The paper additionally reports a zero-shot extension to 50 Bi2Code tasks and a set of per-task results in Table 3 and Table 4.","tokens_in":18146,"tokens_out":5552,"duration_ms":57723,"significance":"If the reported results hold, the hybrid symbolic-perceptual feedback loop is a useful step toward self-correcting code-based robot policies, and the Bi2Code interface could be a practical contribution to the community. The paper is transparent about its limitations, including tasks that fail because of missing API skills, and it ships prompt templates and open-source code that aid reproducibility. However, the significance is currently limited by evaluation-design issues: the headline gains come from the same 10-task subset on which the framework was tuned, the adaptive trial-selection function is not defined, the VLM's diagnostic accuracy is never validated, and no statistical uncertainty is reported. These issues affect the central claim and need to be addressed before the results can be taken as strong evidence.","major_comments":[{"comment":"The headline results are in-sample. The manuscript states that \"the framework architecture, feedback logic, and prompting structure were jointly tuned only on the shared subset reported in Tab. 3\", and those 10 shared tasks are exactly the tasks aggregated in Table 1. The ASR gains (47.4→63.9 and 62.1→71.3) and the CR-Iter reduction are therefore measured on the same tasks used for development. The 40-task extension in Table 4 provides no CodeAct or Code-as-Policies baseline, and the paper itself notes that several tasks are impossible through the available API; it cannot serve as a held-out comparison. A held-out task split, an external benchmark, or an explicit development-versus-test separation is needed to support the generalization claim.","section":"Section 4.5, Tables 1 and 3"},{"comment":"The claimed CR-Iter reduction from 2.42 to 1.76 is not a controlled comparison of the repair mechanism: 2.42 is HyCodePolicy on RoboTwin 1.0 and 1.76 is HyCodePolicy on Bi2Code, and Section 4.4 explicitly warns that the two platforms use different motion-planning backends and are not directly comparable. Furthermore, the Code-as-Policies baseline has CR-Iter 1.00 by construction because it performs no repair. The sample-efficiency claim is therefore supported only by the smaller CodeAct-versus-HyCodePolicy differences (2.46 versus 2.42 on RoboTwin 1.0; 1.89 versus 1.76 on Bi2Code), for which no variance or significance testing is reported.","section":"Abstract and Table 1"},{"comment":"The adaptive trial-selection function ψ(FailureSeverity_i, TraceDivergence_i) is never defined or implemented. This is load-bearing because ψ selects the single trial that the VLM inspects; without its concrete form, thresholds, or learned parameters, the method is not reproducible and the claimed efficiency of \"selecting a single representative trial\" cannot be evaluated. The paper should provide the exact scoring function and an ablation showing its effect on overall performance.","section":"Section 3.4.1, Eq. (3.4.1)"},{"comment":"The VLM's binary success judgments ŷ_i and causal hypotheses c_i are never validated against ground truth. The entire repair loop depends on these judgments being correct, yet no diagnostic-accuracy study is reported. Table 3 provides internal evidence that VLM feedback can be harmful: Handover Block on Bi2Code drops from 50% under CodeAct to 27% under HyCodePolicy, and Pick Diverse Bottles on RoboTwin drops from 65% to 64%. Without an evaluation of the VLM's diagnostic accuracy, the central mechanism is unverified and these regressions remain unexplained.","section":"Sections 3.3.1–3.3.2 and Table 3"},{"comment":"No error bars, confidence intervals, or significance tests are reported for ASR or CR-Iter. Each program is executed only 10 times and 10 candidate programs are generated per task, so the differences between CodeAct and HyCodePolicy (60.4→63.9 and 66.7→71.3) may fall within stochastic variation. Please report per-seed or per-candidate statistics and a significance test, or at least show the variance across runs, to support the claim of significant improvement.","section":"Section 4.1 and Table 1"}],"minor_comments":[{"comment":"The header contains \"A vg Success Rate\" with an errant space, and the \"T ask\" column header in Table 4 appears truncated; these look like table-generation artifacts that should be fixed.","section":"Table 4"},{"comment":"The BASIC_INFO string contains typos: \"representated\" should be \"represented\", and \"7 dimention\" should be \"7-dimensional\".","section":"Appendix A.2.2"},{"comment":"Please use consistent naming for \"Code as Policies\" versus \"Code-as-Policies\", and for \"Unixcoder\" versus \"UniXCoder\" and \"Curobo\" versus \"CuRobo\".","section":"Throughout"},{"comment":"The legend for Figure 3 does not clearly explain how dot size encodes AST similarity; please add an explicit caption or a scale.","section":"Figure 3"},{"comment":"The CR-Iter metric is not fully defined for the case where success never exceeds 50%; please state the convention used (for example, a capped maximum number of iterations).","section":"Section 4.1 and Table 1"}],"recommendation":"major_revision","confidential_remarks":"The authors are unusually honest in Section 4.5 about tuning on the evaluation subset, which is to their credit, but that disclosure also makes the headline evaluation circular. I would give them the opportunity to add held-out validation and to define the missing adaptive-selection component; if such experiments cannot be provided, the central claims should be substantially weakened. The appendix prompt templates and open-source release are valuable for reproducibility and should be recognized in any revision decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a serious look, but the headline numbers should be read with a big caveat: the 10 tasks used for the main comparison are the same 10 tasks on which the framework, prompt structure, and feedback logic were tuned (Sec. 4.5). So the 47.4→63.9 and 62.1→71.3 ASR gains are partly in-sample. That doesn't make the work worthless, but it makes the central claim unproven rather than wrong.\n\nWhat's actually new: Bi2Code, the re-engineered interface with shorter prompts, modular APIs, and embedded observation hooks, is a concrete artifact. The hybrid dual-feedback loop—symbolic logs plus VLM perceptual diagnosis—is a sensible extension of CodeAct and self-debugging, and the idea of selecting a single diagnostic trial via a scoring function ψ is interesting even though ψ is never actually defined. Credit is also due for reporting per-task numbers, which show the VLM feedback can hurt as well as help (Handover Block on Bi2Code: 50%→27%; Pick Diverse Bottles on RoboTwin: 65%→64%).\n\nThe soft spots are mostly around evaluation. No error bars, no significance tests. The VLM's success/failure judgments and causal hypotheses are never validated against ground truth, so we don't know whether the repair signal is reliable. The CR-Iter reduction in the abstract (2.42→1.76) conflates two changes: on the same interface, CodeAct vs HyCodePolicy is 2.46→2.42 on RoboTwin and 1.89→1.76 on Bi2Code, so most of the apparent speed-up comes from the interface, not the feedback loop. The 40-task generalization table has no CodeAct or Code-as-Policies baseline, and several tasks are impossible through the current API, which the authors acknowledge honestly.\n\nThe stress-test note is on target. I'd add one nuance: the paper is unusually transparent about the tuning (Sec. 4.5) and about API limitations. That honesty is a point in its favor. Citation pattern is normal; the relevant prior work (CodeAct, Self-Debugging, Code-as-Policies) is cited.\n\nWho is this for? People working on code-as-policy and LLM-based robot repair will want to know about Bi2Code and the hybrid feedback idea. The contribution is incremental but useful. The right fix is a held-out evaluation—tune on one task set, report on another—plus a defined ψ and an isolated validation of the VLM component.\n\nMy recommendation: send it to peer review, expecting major revision. The architecture is coherent, the artifact is real, and the flaws are fixable. With a clean held-out protocol, this could be a solid paper.","headline":"A sensible hybrid feedback architecture and a useful new interface, but the headline results are partly in-sample and the VLM/ψ loop is unvalidated.","tokens_in":18712,"tokens_out":4168,"would_cite":false,"duration_ms":39111,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A closed loop that treats generated robot code as a falsifiable, evolving hypothesis—executing it, monitoring it with a vision-language model, and repairing it—raises manipulation success and reduces repair iterations.","keywords":["HyCodePolicy","code-as-policies","vision-language model monitoring","robot manipulation","geometric grounding","adaptive program repair","multimodal feedback","closed-loop control"],"falsifier":"Audit the vision-language model's per-subgoal judgments on a held-out set of execution checkpoints with human labels; if diagnostic agreement is at chance on the visually ambiguous tasks where the method shows the largest gains (stacking, placement, handover), then the reported improvements cannot be credited to the perceptual feedback loop.","tokens_in":17650,"feed_emoji":"🤖","tokens_out":7463,"duration_ms":78319,"temperature":0.7,"pith_summary":"The paper tries to show that a generated robot program should be treated as a live hypothesis, tested in simulation and revised through a closed loop rather than shipped after one synthesis pass. It argues that fusing two feedback channels—structured execution logs and vision-language model judgments on checkpoint images—lets the system locate why a manipulation failed and repair the code accordingly. Sympathetically read, this is a demonstration that multimodal monitoring can turn one-shot code-generation policies into self-correcting ones with less human intervention. The reported gains are large: average success improves in both evaluated settings and the number of repair iterations to reach a working policy drops.","feed_headline":"Vision-guided code repair lifts robot task success to 71 percent","feed_subtitle":"A closed loop of execution, visual monitoring, and targeted repair beats one-shot policies and needs fewer repair rounds.","key_machinery":"The load-bearing object is the monitor-diagnose-repair cycle, or what the paper calls code-as-monitor: the generated program carries its own observation hooks. Each subgoal is mapped to visible state changes; the execution inserts camera-capture calls only where the scene changes, plus at the beginning and end of the run. After ten stochastic trials, the system picks the single trial whose symbolic trace diverges most, asks a vision-language model whether each subgoal succeeded, and fuses that verdict with execution logs to infer a cause. Repairs are constrained by a symbolic grammar and a subgoal template so the program stays executable. Geometric point and axis operation primitives—stable grasp points, placement points, approach axes—are embedded in the synthesis prompt to make the initial code physically feasible.","core_discovery":"On the paper's own terms, the central discovery is that adding a perception layer to code-as-policy synthesis converts a one-shot generator into an adaptive controller. After decomposing an instruction into subgoals and synthesizing Python code from geometric primitives, the system executes ten trials, logs symbolic outcomes, and has a vision-language model inspect only checkpoints where the scene visibly changes. A fusion of the two diagnostics attributes the failure to a specific operation, and a repair agent rewrites that portion of the program. In the authors' evaluation this raises average success rates from 47.4% to 63.9% in one benchmark configuration and from 62.1% to 71.3% in their re-engineered modular interface, while the mean number of code-revision iterations needed to exceed 50% success falls from 2.42 to 1.76. The authors also report that the same loop transfers to a larger 50-task suite without per-task tuning, but fails on skills that are absent from the available action API.","pith_inferences":["Because the vision-language model's diagnosis is never checked against ground truth, the method's gains could partly reflect symbolic-trace repair; an ablation that corrupts visual feedback would separate the two channels.","The selection of a single most diagnostic trial is a resource-saving heuristic; analyzing all ten trials could reveal failure modes that the severity score misses, at a higher compute cost.","The same dual-feedback loop is in principle portable to other executable-code-plus-vision settings—GUI automation, warehouse sorting, or inspection—where checkpoint images can be captured.","Real-robot transfer would need to handle noisy images and physical violations that simulation checkpoints may not expose; that gap is left untested."],"forward_implications":["If the central claim holds, one-shot code generation for robots is a bottleneck: a closed loop with perceptual feedback can recover a large share of initial failures automatically.","Vision-language feedback matters most on tasks requiring spatial alignment and perceptual disambiguation; on deterministic tasks, symbolic feedback alone already saturates.","Interface design itself contributes: modular APIs with symbolic logging and observation hooks make failures localizable, which shortens the repair loop.","The same repair loop can generalize to new tasks without per-task prompt tuning, up to the limits of the action API.","Autonomous repair with vision-language monitoring reduces the need for human debugging, since each iteration produces an interpretable code update rather than a black-box retraining step."],"supporting_citations":[{"why":"Defines the one-shot code-generation approach whose single-attempt assumption the closed loop is designed to overcome.","marker":"[17]"},{"why":"Supplies the interactive repair baseline that provides symbolic, trace-driven feedback as the non-visual comparison point.","marker":"[32]"},{"why":"Provides the dual-arm simulation benchmark where the method's success rates are measured.","marker":"[26]"},{"why":"Provides the platform release of the same dual-arm benchmark, supplying the simulation tasks used for evaluation.","marker":"[27]"},{"why":"Underlies the paper's re-engineered modular interface by supplying the simulator generation foundation it is built on.","marker":"[3]"},{"why":"Gives the self-repair paradigm the method extends by adding perceptual, vision-language feedback.","marker":"[5]"}],"fun_headline_variants":["Vision-guided code repair lifts robot task success to 71%","Closed-loop code+vision repairs lift robot success to 71%","Hybrid VLM feedback reduces robot code repair attempts","Perceptual monitoring guides code self-correction for robot policies","Adaptive code loop with visual checks cuts repair rounds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole repair loop rests on the vision-language model being able to tell, from checkpoint images, whether a subgoal actually succeeded and what went wrong—and on the single selected trial representing how the program fails; the paper does not validate either part of that assumption.","fun_headline_variants_meta":{"raw":{"variants":["Vision-guided code repair lifts robot task success to 71%","Closed-loop code+vision repairs lift robot success to 71%","Hybrid VLM feedback reduces robot code repair attempts","Perceptual monitoring guides code self-correction for robot policies","Adaptive code loop with visual checks cuts repair rounds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00133,"raw_usage":{"total_tokens":5419,"prompt_tokens":960,"completion_tokens":4459,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":4377}},"tokens_in":576,"tokens_out":4459,"duration_ms":32659,"temperature":1.0,"reasoning_tokens":4377,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:37:01.682923+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit the vision-language model's per-subgoal judgments on a held-out set of execution checkpoints with human labels; if diagnostic agreement is at chance on the visually ambiguous tasks where the method shows the largest gains (stacking, placement, handover), then the reported improvements cannot be credited to the perceptual feedback loop.","supporting_citations":[{"cited_title":"Executable code actions elicit better llm agents","cited_arxiv_id":null,"evidence_quote":"Supplies the interactive repair baseline that provides symbolic, trace-driven feedback as the non-visual comparison point."},{"cited_title":"Robotwin: Dual-arm robot benchmark with generative digital twins","cited_arxiv_id":null,"evidence_quote":"Provides the platform release of the same dual-arm benchmark, supplying the simulation tasks used for evaluation."}],"review_version":2}