{"id":"5e66a0bb-6307-48ad-b742-512fd8fdf4ae","arxiv_id":"2411.13826","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"REPL-Plan uses an interactive code REPL with recursive sub-REPLs to improve LLM performance on long-horizon text-based planning tasks.","lead":"This paper introduces REPL-Plan, a method that lets a large language model plan by writing Python code into an interactive shell, with undefined function calls spawning sub-shells for sub-tasks. The method reports top success rates on ALFWorld and long-context web shopping tasks, but the results rely on hand-written demonstrations and small evaluation samples.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Global REPL pool persists across evaluation tasks, making REPL-Plan an external-memory method despite Table 1's 'No' label; this may confound the ALFWorld comparison with THREAD.","rationale":"The reader's weakest assumption (rewind-and-rerun semantic transparency) is plausible but speculative: the paper describes a caching mechanism that could preserve semantics, and without the code we cannot confirm a bug. The global REPL pool issue is more concrete and directly grounded in the paper's own text. Section 2.3 states the pool contains spawned REPLs from 'previous task executions', which is the same cross-task memory the paper's Table 1 uses to separate external-memory baselines. Categorizing REPL-Plan as 'No' external memory creates an internal inconsistency that undermines the headline ALFWorld comparison. If the pool is persistent, REPL-Plan is not a no-memory method, and the 1.5-point advantage over THREAD could be attributable to accumulated code rather than the REPL interaction mechanism itself. This does not invalidate the method's novelty, but it means the empirical support for the central claim needs re-evaluation with a clean no-memory control. The proposed test is simple and decisive: resetting the pool per task would either confirm the confound or exonerate the comparison. I therefore keep the reader's conditional verdict but for a different, more textually supported reason.","tokens_in":25584,"tokens_out":11671,"duration_ms":110996,"concrete_test":"Run REPL-Plan on the full ALFWorld evaluation twice: (1) with the global REPL pool reset after each task (containing only the provided demonstrations) and (2) with the pool persisting across tasks as described in Section 2.3. Compare the success rates. If the persistent-pool run is materially higher than the reset run, the 'No external memory' classification in Table 1 is wrong and the 97.0% vs. 95.5% comparison with THREAD is confounded. Alternatively, inspect the attached implementation to confirm whether the evaluation loop clears the pool between tasks.","verdict_should_be":"UNCHANGED","load_bearing_attack":"REPL-Plan's k-shot variant maintains a global REPL pool that includes 'spawned REPLs from previous task executions and demonstrations' (Section 2.3). This is precisely the 'external memory' defined in Table 1's footnote: saving information from previous test tasks to new test tasks. Yet Table 1 classifies REPL-Plan in the 'No' external memory group, directly below THREAD. The headline ALFWorld result (97.0% vs. THREAD's 95.5%) is therefore not an apples-to-apples comparison against a no-memory baseline: if REPL-Plan reuses code written on earlier test tasks, later tasks benefit from accumulated sub-task implementations (e.g., a specialized find_and_take_obj REPL) that THREAD cannot access. The same confound applies to the WebShop k=10 and Real-World Web results, where the REPL pool could accumulate task-specific strategies across trials. The paper never states that the pool is reset between evaluation tasks, and the phrase 'previous task executions' implies persistence. If the reported advantage stems from this cross-task reuse, the central claim that REPL-Plan's interactive, code-expressive mechanism alone drives strong results is weakened, and the comparison to no-memory baselines is invalid.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes REPL-Plan, a code-augmented LLM planning approach in which an LLM interacts with a recursively spawnable REPL (LLM-REPL) to write and execute code line-by-line, with child REPLs handling fuzzy subproblems. The method is evaluated on ALFWorld, WebShop, and a new Real-World Web benchmark built from live e-commerce sites. The authors report state-of-the-art success on ALFWorld (97.0% vs. THREAD's 95.5%), strong WebShop performance in a k=10 setting, and 39.6% of expert score on complex Real-World Web tasks vs. 17.6% for ReACT and 0.0% for THREAD. Ablations show that recursive spawning and zero-shot subtask inference are important to performance.","tokens_in":25701,"tokens_out":4937,"duration_ms":44340,"significance":"If the reported results are robust, REPL-Plan is a meaningful advance in code-augmented LLM planning: it is novel in proposing interactive, recursively spawnable REPLs as a planning mechanism, and the qualitative trajectories show the approach can handle long web observations and recover from hallucinated actions. The paper also contributes a new real-world web navigation benchmark and a thoughtful ablation of error correction. However, the strength of the empirical claims is substantially limited by unresolved experimental confounds and lack of statistical rigor, as detailed below.","major_comments":[{"comment":"The k-shot REPL-Plan keeps a 'global REPL pool, a set of spawned REPLs from previous task executions and demonstrations' (Section 2.3). This means REPL-Plan can reuse code from earlier test tasks during later test tasks, which is exactly the 'external memory' defined in Table 1's footnote. Yet Table 1 places REPL-Plan in the 'No' external-memory group, directly below THREAD. The ALFWorld comparison (97.0% vs. 95.5%) is therefore not apples-to-apples against a no-memory baseline unless the pool is explicitly reset between tasks. The paper must either state that the pool is reset per task (and then explain what 'previous task executions' refers to) or reclassify REPL-Plan as an external-memory method and compare against memory-equipped baselines. The same issue affects the WebShop and Real-World Web results.","section":"Section 2.3, Table 1"},{"comment":"The headline empirical claims are reported without error bars, confidence intervals, or statistical tests. On ALFWorld, the gap between REPL-Plan (97.0%) and THREAD (95.5%) is only 1.5 percentage points, which with the usual ALFWorld test set of roughly 134 tasks corresponds to about 2 tasks; this difference may easily be within run-to-run noise. The Real-World Web results use only 3 trials per task (15 trials total per condition for the complex tasks) and an expert score provided by one author, with no per-task breakdown or variance. The WebShop ablation uses n=25. The paper should provide uncertainty estimates and, where possible, significance tests or at least a discussion of the minimal detectable effect at the given sample sizes.","section":"Section 3.2, Tables 1-3; Section 3.3, Table 4"},{"comment":"The rewind-and-rerun implementation (saving execution state, intercepting NameError, re-running code with a call-counter cache) is the core mechanism that claims to preserve the exact sequence of environment actions and observations. However, the paper does not establish that this mechanism is semantically transparent. Re-running code from a saved state can re-execute statements with side effects on local variables, and the environment may have changed since the saved state because a prior act() already executed. The call-counter cache prevents duplicate calls to context-sensitive functions (act, child REPLs), but it does not obviously handle stateful local-variable updates that depend on observations. A correctness argument or a validation that the execution trace equals the trace of a single continuous interpreter (or equivalently, that no action is duplicated or dropped) is needed to support the claim that the reported trajectories reflect the LLM-written code as designed.","section":"Appendix A.1"},{"comment":"The Real-World Web benchmark is very small (5 simple and 5 complex tasks, 3 trials each) and is based on live websites that can change over time, making the results difficult to reproduce or generalize. The expert score is defined as one author's manual performance, but the scoring procedure is not described in enough detail to assess inter-rater reliability or potential bias. Given that the central claim of 'scalability' to real-world web navigation rests almost entirely on this benchmark, the paper should either substantially expand the task set, report per-task and per-trial results, provide a detailed scoring rubric with inter-rater agreement, or moderate the strength of the claims to be consistent with the evidence.","section":"Appendix A.4 and Results in Section 3.2"}],"minor_comments":[{"comment":"Typo: 'explictly' should be 'explicitly' in the second paragraph.","section":"Section 1"},{"comment":"The footnote for 'THREAD*' in Table 2 says modifications are described in Appendix A.2; it would be clearer to state in the main text that THREAD was patched to avoid infinite loops on GPT-4o-mini, since this is relevant to the interpretation of the comparison.","section":"Section 3.2, Table 2"},{"comment":"The naming is inconsistent: 'ReACT' in Table 3 and the text, 'ReAct' in the references and in Section 4; please unify.","section":"Section 3.2 and Appendix A.4"},{"comment":"The No-Subtask-REPLs ablation is described as 'conceptually similar to a code-augmented version of prior approaches, such as ReACT', but ReACT is not code-augmented; please rephrase to avoid a misleading comparison.","section":"Section 3.3"},{"comment":"The citation 'Sipser, 1996' for the undecidability of determining undefined variables should be a specific theorem or section, not just a textbook citation; alternatively, rephrase to 'the undecidability of the halting problem' with a standard reference.","section":"References"},{"comment":"The red highlighting mentioned in the caption may not be visible in grayscale print; consider using a different visual marker (e.g., bold or underlining) to indicate the hallucination errors.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The core idea of REPL-Plan is clever and well-motivated, and the qualitative results are compelling. However, the external-memory confound is serious: if the global REPL pool persists across test tasks, the ALFWorld and WebShop comparisons against no-memory baselines are not fair, and the paper's taxonomical claim in Table 1 is internally inconsistent. This is fixable (e.g., by resetting the pool or relabeling the method and adding appropriate memory baselines), but it must be addressed. The statistical concerns, especially the 1.5-point ALFWorld gap and the 15-trial Real-World Web evaluation, further weaken the central claim. I think the paper is promising and worth a major revision rather than rejection, assuming the authors can address the memory issue and provide uncertainty estimates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read REPL-Plan. The core idea is real: an LLM writes code into a REPL, and undefined function calls spawn child LLM-REPLs, letting the model decompose tasks top-down while staying interactive. That's a clean step beyond THREAD (recursive but not code-expressive) and CodeAct/Mint (interactive but not recursive). The paper is also honestly written—the ablations show how much the method depends on hand-written demonstrations (zero-shot subtask inference halves performance), and the appendix explains the implementation including the rewind-and-rerun trick to handle NameError. Credit where due: the method is well-motivated, the writing is clear, and the qualitative trajectories are informative.\n\nNow the soft spots. The biggest one is the external-memory classification. Section 2.3 says the k-shot variant keeps a global REPL pool with \"spawned REPLs from previous task executions and demonstrations,\" and that LLM-REPLs reuse code/output history. That's exactly what Table 1's footnote defines as external memory: saving information from previous test tasks to new test tasks. Yet REPL-Plan is listed in the \"No\" column. So the ALFWorld 97.0% vs THREAD's 95.5% is not a fair comparison against a no-memory baseline—later test tasks can reuse sub-task REPLs built on earlier ones. This is a real confound, and it needs to be addressed, either by resetting the pool per task and reporting both numbers, or by reclassifying and comparing against a memory-equivalent baseline. Without that, the headline claim is weaker than stated.\n\nThe Real-World Web evaluation is also thin: 3 trials per task, one expert score from an author, no error bars. The THREAD baseline needed a custom anti-loop patch to work at all on GPT-4o-mini, which inflates the relative advantage on k=10. These are disclosed, but they limit what can be concluded from the numbers.\n\nThat said, the central idea survives. The mechanism is novel, the ablation is informative, and the paper is not overclaiming zero-shot generalization—it explicitly flags the limitation. The evaluation needs strengthening before \"strong results\" is fully established, and the external-memory mislabeling is a genuine blemish.\n\nWho's it for: anyone working on LLM agents, code-augmented planning, or recursive task decomposition. It's a solid method paper worth a serious referee. I'd push for revision before acceptance—mainly re-examining the memory claim and tightening the real-world eval.\n\nRecommendation: send to peer review. The novelty and the clean exposition justify referee time, and the flaws are fixable.","headline":"REPL-Plan is a genuinely new code-augmented planning mechanism, but the ALFWorld headline is confounded by the global REPL pool that Table 1 mislabels as external memory.","tokens_in":26406,"tokens_out":3092,"would_cite":true,"duration_ms":26531,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"REPL-Plan makes an LLM plan by writing Python in a recursive REPL, reaching 97.0% on ALFWorld.","keywords":["REPL","LLM planning","code-augmented agents","recursive decomposition","ALFWorld","WebShop","web navigation","in-context learning"],"falsifier":"Attach a strict audit log to the environment that rejects any repeated act() call with identical arguments before the task is complete; if the rewind-and-rerun replay ever re-executes an act() after a NameError, the audit will flag a duplicate action, and the reported 97.0% success rate on ALFWorld should fall, showing that the trajectories do not exactly match the LLM-written code.","tokens_in":25209,"feed_emoji":"⌨️","tokens_out":5555,"duration_ms":97845,"temperature":0.7,"pith_summary":"REPL-Plan claims that large language models plan better when they solve tasks by writing Python line by line inside a recursive read-eval-print loop rather than emitting one action at a time or writing a full program in a single pass. The paper argues that this combination gives the full expressiveness of code, including loops, variables, and reusable functions, while keeping the loop dynamic enough to correct errors and hand ambiguous subtasks to fresh LLM calls. The central construction is the LLM-REPL: when the generated code calls an undefined function, the system treats that as a request to spawn a child REPL where the LLM writes code for that subtask, and it can cache and reuse those child REPLs across the task. If the reported numbers are right, the practical payoff is 97.0% success on ALFWorld, a 52% success rate on WebShop with ten results per page, and 39.6% of expert score on complex real-world web tasks where message-based baselines score 17.6% and 0.0%.","feed_headline":"Recursive code REPL lifts LLM planning to 97% on ALFWorld","feed_subtitle":"Writing plans as interactive Python lets models fix errors and delegate fuzzy subtasks, beating one-pass code and text baselines.","key_machinery":"The central object is the LLM-REPL, a read-eval-print loop whose program state is augmented with four primitives: act sends an action to the environment, get_obs reads the latest observation, get_args receives arguments from the parent REPL, and answer returns a value and gives control back. The load-bearing mechanism is the treatment of NameError: when the LLM's code calls a function that is not defined, the system spawns a child LLM-REPL for that function, waits for its answer, and then rewinds and re-runs the parent code with the newly defined function, using cached call counts so context-sensitive functions such as act are not executed twice. This is what makes planning top-down, because large tasks call helper REPLs instead of being written bottom-up; dynamic, because each line's output is visible before the next line is written; and code-expressive, because the LLM can use loops, variables, and helper functions across REPL boundaries.","core_discovery":"On its own terms, the paper's discovery is that planning can be organized as an interactive session in which the LLM is both the programmer and the interpreter of its own code: it types a line, sees the resulting observation or error, and writes the next line, with undefined function names triggering recursive child sessions that can be reused. The key result is that this arrangement is not merely expressive but also robust: the loop lets the model absorb execution feedback including syntax errors and runtime NameErrors, the recursive spawning lets it decompose tasks top-down, and the reuse of previously spawned child REPLs lets it apply the same subtask logic repeatedly. The paper reports that REPL-Plan reaches 97.0% success on ALFWorld with GPT-3.5-instruct, above a 95.5% baseline that also decomposes tasks recursively but lacks full code expressiveness, and on complex real-world web tasks with GPT-4o-mini it reaches 39.6% of an expert score, more than double the strongest message-based baseline and well above a recursive-thread baseline that scores zero.","pith_inferences":["Beyond the reported benchmarks, the same NameError-as-subtask mechanism could turn any tool-use API into a genuinely recursive language, because an LLM that calls a nonexistent function effectively declares a new tool on the fly.","The paper does not measure sample cost, and the rewind-and-rerun implementation may replay code frequently; a testable extension would be to compare success per dollar against message-based baselines on the same tasks.","A direct stress test the authors leave open is removing the entire demonstration pool and requiring the model to write all child REPLs zero-shot; their own ablation suggests performance would fall substantially, making fine-tuning on REPL-style interactions a natural next step."],"forward_implications":["On ALFWorld, the 97.0% success rate implies that near-exhaustive household search tasks can be solved by a code-generating LLM that reuses a small pool of demonstrated subtask REPLs, with only occasional interventions from the main loop.","On WebShop with ten results per page, the 52% success rate implies that storing candidates in variables and looping over pages generalizes beyond the three-item page configuration that earlier methods targeted.","On real-world web pages of 4,000 to 20,000 tokens, the complex-task score of 39.6% implies that delegating page interpretation to child REPLs is what lets planning survive long, noisy observations that overwhelm text-action baselines.","The ablation that drops recursive spawning cuts success rates roughly in half, so the paper's claim is that top-down recursive decomposition, not interactivity alone, is the critical component for these benchmarks."],"supporting_citations":[{"why":"Provides the ALFWorld benchmark that defines the primary success-rate comparison.","marker":"Shridhar et al., 2021"},{"why":"Provides the WebShop environment and its attribute-matching score, which the paper extends to a top-10 setting.","marker":"Yao et al., 2022a"},{"why":"Supplies the ReAct text-action baseline whose act-and-observe style REPL-Plan adapts into code primitives.","marker":"Yao et al., 2022b"},{"why":"The strongest recursive-decomposition baseline, which REPL-Plan is designed to beat by adding full code expressiveness.","marker":"Schroeder et al., 2024"},{"why":"Establishes the code-execution paradigm for language models that REPL-Plan extends into interactive decision-making.","marker":"Gao et al., 2023"},{"why":"A code-writing planning baseline with external memory that gives a comparison point on ALFWorld.","marker":"Sun et al., 2023"},{"why":"An external-memory baseline on ALFWorld that shows the gap REPL-Plan covers without external memory.","marker":"Shinn et al., 2023"},{"why":"An as-needed decomposition baseline on ALFWorld that contrasts with REPL-Plan's code-expressive recursive REPLs.","marker":"Prasad et al., 2024"}],"fun_headline_variants":["LLM plans better by typing and running its own code","REPL-Plan: let LLMs execute code to plan tasks","Interactive code loop boosts LLM planning to 97%","Planning with code REPL: LLMs fix errors on the fly","LLM planning via REPL beats text-only baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the rewind-and-rerun implementation, which saves execution state, intercepts NameErrors, and replays code with a cached call counter, is semantically transparent so that no environment action is duplicated, dropped, or reordered when a child REPL interrupts the parent.","fun_headline_variants_meta":{"raw":{"variants":["LLM plans better by typing and running its own code","REPL-Plan: let LLMs execute code to plan tasks","Interactive code loop boosts LLM planning to 97%","Planning with code REPL: LLMs fix errors on the fly","LLM planning via REPL beats text-only baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000486,"raw_usage":{"total_tokens":2409,"prompt_tokens":970,"completion_tokens":1439,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":1354}},"tokens_in":586,"tokens_out":1439,"duration_ms":10453,"temperature":1.0,"reasoning_tokens":1354,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:49:36.969419+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Attach a strict audit log to the environment that rejects any repeated act() call with identical arguments before the task is complete; if the rewind-and-rerun replay ever re-executes an act() after a NameError, the audit will flag a duplicate action, and the reported 97.0% success rate on ALFWorld should fall, showing that the trajectories do not exactly match the LLM-written code.","supporting_citations":[],"review_version":1}