{"id":"46cd8b37-e977-4e94-9cd0-6b3767edd206","arxiv_id":"2505.10749","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An iterative code-generation framework (IPP) improves LLM performance on GRASP and MiniGrid grid-planning tasks by refining generated policy programs based on execution feedback.","lead":"This paper tests whether large language models can write reusable computer programs that solve grid-based planning puzzles, instead of listing action steps directly. The authors find that an iterative feedback loop, where the model rewrites its program after seeing failures, boosts performance on two benchmark suites across several popular LLMs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"IR refinement and reported evaluation appear to use the same instances, so the headline IR-vs-DG gains may reflect fitting rather than generalization.","rationale":"The reader identified the same load-bearing concern: no held-out split is described, so IR may be refining on the same instances used for scoring. This is not an internal inconsistency but an under-specified evaluation protocol, and it threatens the central generalization claim rather than a peripheral one. The pseudo-code explicitly uses a \"training set\" and selects worst cases from it; the experimental section describes generating 1,000 MiniGrid instances but never says which instances are used for refinement and which for the reported averages. Because IR is a program-level search driven by execution feedback on concrete grids, scoring on those same grids is exactly the failure mode that would inflate the reported 10%–10x improvements and the SOTA claim. The appendix provides prompts and code examples, which is useful, but it does not resolve the split question. The appropriate disposition remains CONDITIONAL: the central claim is plausible and potentially correct, but it should not be accepted without a held-out evaluation and, ideally, a direct comparison against prior GRASP methods. Therefore the reader's verdict does not need to change.","tokens_in":52140,"tokens_out":3487,"duration_ms":40769,"concrete_test":"Split instance generation into refinement and evaluation pools (e.g., 1,000 and 1,000), run Algorithm 2.1 using only the refinement pool, then evaluate the final IR program and the DG program on the evaluation pool, repeating with 3 independently drawn splits; also report the per-model IR-vs-DG delta on that held-out pool. If the 10%–10x improvements do not survive on held-out instances, the generalization claim fails; if they survive, the protocol concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that IR turns LLM-generated code into reusable policies that beat DG by 10%–10x and set a GRASP SOTA. That claim requires the reported IR scores to measure generalization to instances not used to drive refinement. Algorithm 2.1 computes D as \"evaluation results of f(t) on training set\" and iteratively refines on Dfail, the worst k=3 training instances. Section 3 says \"For each task, we generate 1,000 instances and measure two metrics\" but never states that the reported numbers come from a disjoint held-out set; Table 3 even reports \"100 instances per task,\" making the evaluation population ambiguous. If Tables 1–2 score the same instances that supplied Dfail, then each final program is adapted to exactly those grids, and the DG-vs-IR comparison is effectively a train-vs-DG comparison. It would not show that the synthesized code generalizes, which is the basis for the \"reusable policy\" and amortized-cost arguments. The missing SOTA baseline for GRASP is a separate weakness, but the absent split is the most load-bearing because it directly controls the magnitude of every headline gain.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Iterative Programmatic Planning (IPP), a framework in which an LLM is prompted to synthesize a Python policy for a grid-world task and then iteratively refine it using execution feedback. The main components are direct generation, pseudocode-conditioned extension, step-by-step curriculum prompting, and an iterative refinement loop (Algorithm 2.1) that selects the k worst-performing training instances and asks the LLM to repair the code. The paper evaluates six LLMs on GRASP and three MiniGrid tasks, reporting that IR improves over DG by 10% to 10x for five of six models, that IPP matches or beats CoT and 2-step CoT at lower API cost, and that it establishes a new state of the art on GRASP. The abstract and introduction also emphasize interpretability, reusability, and the low amortized cost of synthesized policies.","tokens_in":52385,"tokens_out":6860,"duration_ms":68242,"significance":"If the claims hold, the contribution is useful: a simple generate-execute-refine loop can turn weak LLM policies into reusable, interpretable programs and outperform per-instance prompting at lower amortized cost. The appendix is genuinely helpful, with full prompt templates and worked examples of successful refinements, and the breadth across six models and two benchmarks is a strength. However, the headline result rests on the synthesized policy generalizing to instances that did not drive refinement, and the manuscript never establishes that. The unsubstantiated GRASP state-of-the-art claim and unresolved ambiguities in the evaluation population mean that the current evidence supports a promising mechanism rather than the stated conclusions.","major_comments":[{"comment":"The evaluation never specifies a held-out split, so the central DG-vs-IR comparison may be in-sample. Algorithm 2.1 evaluates f(t) on the training set and selects Dfail from that same set (lines 4 and 10), while Section 3 says only that 'we generate 1,000 instances' per MiniGrid task and does not state that the reported numbers come from a disjoint test set. Table 6 makes the problem concrete: for GPT-o3-mini on GRASP, Iter 2 (1.41) is lower than Iter 1 (2.90), which cannot happen under Algorithm 2.1's stopping condition (Jcurr ≤ Jprev) if Jcurr is computed on the same instances used for refinement. This suggests either that the trajectory in Table 6 is evaluated on a different instance population than the one used for refinement or that the algorithm was not followed as written; either way, the population used for evaluation must be defined and reported. As written, the reported 10%–10x gains could be fitting artifacts, and the 'reusable policy' and amortized-cost arguments are unsupported. Please re-run with an explicitly disjoint held-out set and report train and test performance separately.","section":"§2.1/Algorithm 2.1 and §3"},{"comment":"The claim of establishing a new state of the art on GRASP is not supported by any comparison. Section 4 compares IR only to DG, CoT, 2-step CoT, Random, and Greedy; no prior published result or existing GRASP baseline is cited or tabulated. The phrase 'state-of-the-art' requires a baseline; either add a table with previously reported GRASP numbers under matching settings, including results from the original GRASP benchmark paper, or remove the claim.","section":"Abstract and §4"},{"comment":"The evaluation population and cost denominator are inconsistent. Section 3 states that 1,000 instances are generated per MiniGrid task, while Table 3 reports results 'averaged over 100 instances per task'; Tables 1–2 do not state the number of instances at all. For the cost comparison, Table 3 reports an IR per-instance API cost (e.g., 8.0e-4), but if the final program is reused, the marginal per-instance cost should be 0 and the reported number must be an amortization of some synthesis cost. Please state the instance count for each table, report the synthesis/refinement cost separately, and define the amortization formula.","section":"§3 and Table 3"},{"comment":"The GPT-o3-mini rows contain values identical to GPT-o1 rows, which suggests a labeling or data-processing error. In Table 1, GPT-o3-mini's entries under 'With' (e.g., 2.89/2.92, 2.85/2.92) equal GPT-o1's IR entries in the same columns, and in Table 5, GPT-o3-mini's cost-0 DG values (5.75, 5.73, 5.78, ...) equal GPT-o1's cost-0 DG values. Please verify the model assignments and regenerate these tables; if the duplication is not an error, explain why GPT-o3-mini exactly replicates GPT-o1's numbers.","section":"Tables 1 and 5"},{"comment":"No statistical tests or confidence intervals are reported for the comparisons that back the headline claims, and several reported differences appear to be within noise. For example, GPT-o3-mini Unlock-Pickup IR is 0.78 ± 0.34 versus 2-step CoT 0.84 ± 0.30, a difference of 0.06 that is far smaller than the standard deviations. The abstract's 'equal or better performance' claim needs paired significance tests or bootstrap confidence intervals over instances and seeds, at least for the IR-vs-DG and IR-vs-2-step-CoT comparisons.","section":"§4, Tables 2–3"}],"minor_comments":[{"comment":"The text refers to Algorithm 2.1, but the displayed pseudocode is titled 'Algorithm 2.2'; fix the numbering and all cross-references.","section":"§2.1"},{"comment":"The abstract's '10% to as much as 10x' phrasing is imprecise when baselines are negative: for example, Table 1 reports Gemini-2.5-Pro moving from -0.48 to 3.27 as +781%, a percentage based on a negative denominator; report absolute changes or define the improvement metric explicitly.","section":"Abstract"},{"comment":"The main paper's Table 1 omits standard deviations even though Table 4 provides them; move the standard deviations into the main table or state explicitly that Table 4 is the version with error bars.","section":"Table 1"},{"comment":"The cost claims in the abstract ($0.08 per task versus $0.002 per instance, 400x amortization) are not derived from any table in the paper; add a concrete calculation and define what constitutes a 'task' versus an 'instance'.","section":"Abstract and §4"},{"comment":"Figure 1's caption reports an IR reward of 0.96 but does not identify the task or the reward scale; align it with Table 2's MiniGrid tasks and state which configuration is shown.","section":"Figure 1"},{"comment":"The sentence 'Code for reproducing all experiments is available here' contains no visible URL or repository identifier in the manuscript; include the link or an anonymized repository.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":"The main blocker is the missing held-out split: the central claim of generalization and the amortized-cost argument depend on evaluation instances being disjoint from refinement instances. This is potentially fixable by re-running the evaluation on a held-out set, which the released code should permit. I would also ask the editor to insist that the GRASP state-of-the-art claim be either supported by a real baseline comparison or removed, and to verify the apparent GPT-o1/GPT-o3-mini duplication in Tables 1 and 5 before any further review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the one thing to know: the headline claim—IR beats direct generation by 10%–10x and sets a GRASP SOTA—depends on a train/test split that the paper never describes. Algorithm 2.1 refines on a 'training set'; Section 3 reports results on 1,000 generated instances; nothing says those are disjoint. If they aren't, the gains are partly fitted to the test set and the 'reusable policy' claim collapses.\n\nThat said, there's real value here. The IPP framework combines known pieces—code as policies, SELF-REFINE—with a wide sweep: six LLMs, two benchmarks, four prompting strategies. The appendix has all prompts and before/after code, and the examples show concrete bugs refinement fixes (Gemini 2.5 Pro conflating movement cost with action budget; Claude 3.7 not clearing picked-up cells). The finding that DeepSeek-R1 degrades under IR is honest and interesting. The cost-reuse argument is sensible in principle.\n\nSoft spots, in order. First, the missing held-out split is load-bearing, not a nit. They need to either state the split explicitly or run a clean test set; otherwise the main empirical tables are uninterpretable as generalization. Second, the 'new state-of-the-art for GRASP' is asserted without a number from any prior method on GRASP—where's the baseline? Third, the per-instance cost for IR in Table 3 assumes some amortization horizon; that should be stated.\n\nThe writing is clear, the authors engage with the right prior work, and the limitations section is candid about model variance. This is a modest but useful contribution once the evaluation is cleaned up. It deserves a serious referee—not a desk reject. I'd send it out, with instructions to demand the split and a fair SOTA comparison. I wouldn't cite it in its current form.","headline":"A useful code-as-policy refinement study whose headline gains hinge on a train/test split the paper never describes.","tokens_in":52925,"tokens_out":4845,"would_cite":false,"duration_ms":45013,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that iterative refinement of LLM-generated code turns weak grid-world policies into strong, reusable programs, with 10%-to-10x gains over direct code generation on GRASP.","keywords":["iterative programmatic planning","LLM code generation","policy synthesis","grid-world planning","iterative refinement","GRASP benchmark","MiniGrid"],"falsifier":"Re-run the full IPP pipeline with an explicit split: refine on one set of GRASP and MiniGrid instances, freeze the final program, and score it only on a disjoint held-out set; if the improvement over direct generation collapses or reverses on the held-out set, the central claim of learned generalizable policies is refuted.","tokens_in":1555,"feed_emoji":"🧩","tokens_out":1478,"duration_ms":61760,"temperature":0.7,"pith_summary":"The paper tries to establish that planning in grid worlds can be solved not by asking a language model to output an action plan, but by asking it to write a program that acts as the policy, then repeatedly executing that program, scoring it on sample tasks, and feeding the worst failures back to the model for revision. It claims this iterative programmatic planning (IPP) loop improves over one-shot direct code generation by roughly 10% to 10x on the GRASP benchmark across five of six tested LLMs, and reports a new state-of-the-art result there. A sympathetic reader would care because the resulting policy is a readable, reusable Python program: once synthesized it can be applied to any number of new instances without further model calls, which the paper estimates makes GPT-o3-mini's amortized cost about 400x cheaper than per-instance prompting.","feed_headline":"Iterative code refinement lifts LLM grid-planning by up to 10x","feed_subtitle":"A generate-execute-refine loop turns weak LLM policies into reusable programs on GRASP and MiniGrid.","key_machinery":"The load-bearing mechanism is the Iterative Refinement loop, stated as Algorithm 2.1. Starting from a directly generated program $f^{(0)}$, the loop evaluates it on a training set of task instances, computes the average task metric $J$, and if the average improved, feeds the $k=3$ worst-scoring instances to the LLM as revision feedback to produce $f^{(t+1)}$; it stops when the average stops improving. The paper describes this as a discrete, non-gradient search over program space in which worst-case feedback acts as a heuristic gradient. Two auxiliary prompting strategies, pseudocode extension and step-by-step curriculum prompting, seed or stage the synthesis.","core_discovery":"The central claim is that an LLM that cannot directly produce a good plan can still write a policy program that, when executed and scored, reveals concrete failing instances; feeding those instances back lets the same model correct logical errors, add missing constraints, and restructure control flow. On GRASP, iterative refinement turns Gemini-2.5-Pro's negative direct-generation scores (e.g., -0.48) into positive scores (3.27), lifts Claude-3.7 energy from 0.89 to 0.98 via pseudocode extension and GPT-o3-mini from 0.44 to 1.73 via curriculum prompting, and on MiniGrid takes GPT-o3-mini's Unlock-Pickup from zero reward and zero completion to 0.78 reward and 84% completion. The paper also claims that this beats chain-of-thought and two-step chain-of-thought prompting at lower API cost, because the synthesized program generalizes without per-instance inference.","pith_inferences":["The same generate-score-refine loop should transfer to any setting where an LLM can emit an executable policy and a cheap reward signal exists, such as PDDL planning, robot control programs, or decision-rule synthesis; the paper tests only grid worlds.","Because the 16,000 GRASP instances vary by movement type, carry limit, step cost, energy layout, obstacles, and start position, a natural extension is to measure how refinement on one distribution transfers to a different distribution of the same benchmark.","The $k=3$ worst-instance selection is an arbitrary choice; testing whether selecting diverse or adversarial failures rather than the three worst changes convergence rate would clarify what the feedback signal is doing.","The most decisive follow-up is an explicit train/test split, which the current write-up does not provide; without it, the reported gains could partly reflect refinement on the evaluation distribution."],"forward_implications":["The synthesized policies are reusable: one program produced by IPP can be executed on any number of task instances without further LLM calls, which the paper estimates makes GPT-o3-mini's amortized GRASP cost about 400x lower than per-instance reasoning prompting.","Weak initial policies are not fatal: GPT-o3-mini moves from 0% to 84% completion on Unlock-Pickup, so refinement can rescue models whose one-shot code fails completely.","Most of the benefit arrives in the first one or two refinement iterations, after which scores plateau or decline slightly, so the loop is cheap to run.","Structured scaffolding helps: pseudocode extension raises Claude-3.7's GRASP energy from 0.89 to 0.98, and step-by-step curriculum prompting raises GPT-o3-mini from 0.44 to 1.73.","If these results hold, code-driven planning offers an interpretable, low-cost alternative to reinforcement learning in discrete grid tasks, with human-readable and editable policies."],"supporting_citations":[{"why":"Supplies the GRASP energy-collection benchmark, the 16,000-instance task family on which the new state-of-the-art result is claimed.","marker":"[48]"},{"why":"Supplies the MiniGrid Unlock, Door-Key, and Unlock-Pickup tasks used to measure reward and completion rate.","marker":"[11]"},{"why":"Provides the code-as-policy idea that motivates generating executable programs rather than direct action sequences.","marker":"[28]"},{"why":"Establishes the iterative self-refinement paradigm that the paper adapts from code quality to task-level planning feedback.","marker":"[31]"},{"why":"Defines the chain-of-thought prompting baseline that IPP is compared against on both performance and API cost.","marker":"[59]"}],"fun_headline_variants":["Refine, execute, reuse: LLM code planning gets 10x better","Code-driven loop beats one-shot LLM planning by up to 10x","LLM-written policies improve via iterative feedback up to 10x","From code to better code: iterative planning lifts LLMs 10x","Programmatic refinement: reuseable LLM policies with 10x gains"],"cache_read_input_tokens":55040,"weakest_assumption_plain":"The load-bearing premise is that the instances used to give refinement feedback differ from the instances used to compute final scores; the paper refines on a 'training set' but never describes a held-out split, so if the sets coincide the reported gains would measure fitting rather than generalization.","fun_headline_variants_meta":{"raw":{"variants":["Refine, execute, reuse: LLM code planning gets 10x better","Code-driven loop beats one-shot LLM planning by up to 10x","LLM-written policies improve via iterative feedback up to 10x","From code to better code: iterative planning lifts LLMs 10x","Programmatic refinement: reuseable LLM policies with 10x gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000509,"raw_usage":{"total_tokens":2510,"prompt_tokens":1010,"completion_tokens":1500,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":1402}},"tokens_in":626,"tokens_out":1500,"duration_ms":13248,"temperature":1.0,"reasoning_tokens":1402,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:03:41.145090+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the full IPP pipeline with an explicit split: refine on one set of GRASP and MiniGrid instances, freeze the final program, and score it only on a disjoint held-out set; if the improvement over direct generation collapses or reverses on the held-out set, the central claim of learned generalizable policies is refuted.","supporting_citations":[],"review_version":1}