Pith. sign in

REVIEW 4 major objections 6 minor 15 references

Interactive and Expressive Code-Augmented Planning with Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read REPL-Plan makes an LLM plan by writing Python in a recursive REPL, reaching 97.0% on ALFWorld.

desk verdict REPL-Plan is a genuinely new code-augmented planning mechanism, but the ALFWorld headline is confounded by the global REPL pool that Table 1 mislabels as external memory. read the letter →

arxiv 2411.13826 v1 pith:KBKOUFKO submitted 2024-11-21 cs.CL cs.LG

Anthony Z. Liu , Xinhe Wang , Jacob Sansom , Yao Fu , Jongwook Choi , Sungryull Sohn , Jaekyeom Kim , Honglak Lee This is my paper · ORCID
classification cs.CLcs.LG
keywords REPLLLMplanningcode-augmentedagentsrecursivedecompositionALFWorldWebShopnavigationin-contextlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

REPL-Plan claims that large language models plan better when they solve tasks by writing Python line by line inside a recursive read-eval-print loop rather than emitting one action at a time or writing a full program in a single pass. The paper argues that this combination gives the full expressiveness of code, including loops, variables, and reusable functions, while keeping the loop dynamic enough to correct errors and hand ambiguous subtasks to fresh LLM calls. The central construction is the LLM-REPL: when the generated code calls an undefined function, the system treats that as a request to spawn a child REPL where the LLM writes code for that subtask, and it can cache and reuse those child REPLs across the task. If the reported numbers are right, the practical payoff is 97.0% success on ALFWorld, a 52% success rate on WebShop with ten results per page, and 39.6% of expert score on complex real-world web tasks where message-based baselines score 17.6% and 0.0%.

What carries the argument

The central object is the LLM-REPL, a read-eval-print loop whose program state is augmented with four primitives: act sends an action to the environment, get_obs reads the latest observation, get_args receives arguments from the parent REPL, and answer returns a value and gives control back. The load-bearing mechanism is the treatment of NameError: when the LLM's code calls a function that is not defined, the system spawns a child LLM-REPL for that function, waits for its answer, and then rewinds and re-runs the parent code with the newly defined function, using cached call counts so context-sensitive functions such as act are not executed twice. This is what makes planning top-down, because large tasks call helper REPLs instead of being written bottom-up; dynamic, because each line's output is visible before the next line is written; and code-expressive, because the LLM can use loops, variables, and helper functions across REPL boundaries.

What would settle it

Attach a strict audit log to the environment that rejects any repeated act() call with identical arguments before the task is complete; if the rewind-and-rerun replay ever re-executes an act() after a NameError, the audit will flag a duplicate action, and the reported 97.0% success rate on ALFWorld should fall, showing that the trajectories do not exactly match the LLM-written code.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that planning can be organized as an interactive session in which the LLM is both the programmer and the interpreter of its own code: it types a line, sees the resulting observation or error, and writes the next line, with undefined function names triggering recursive child sessions that can be reused. The key result is that this arrangement is not merely expressive but also robust: the loop lets the model absorb execution feedback including syntax errors and runtime NameErrors, the recursive spawning lets it decompose tasks top-down, and the reuse of previously spawned child REPLs lets it apply the same subtask logic repeatedly. The paper reports that REPL-Plan reaches 97.0% success on ALFWorld with GPT-3.5-instruct, above a 95.5% baseline that also decomposes tasks recursively but lacks full code expressiveness, and on complex real-world web tasks with GPT-4o-mini it reaches 39.6% of an expert score, more than double the strongest message-based baseline and well above a recursive-thread baseline that scores zero.

Load-bearing premise

The load-bearing premise is that the rewind-and-rerun implementation, which saves execution state, intercepts NameErrors, and replays code with a cached call counter, is semantically transparent so that no environment action is duplicated, dropped, or reordered when a child REPL interrupts the parent.

Editorial extensions

If this is right

  • On ALFWorld, the 97.0% success rate implies that near-exhaustive household search tasks can be solved by a code-generating LLM that reuses a small pool of demonstrated subtask REPLs, with only occasional interventions from the main loop.
  • On WebShop with ten results per page, the 52% success rate implies that storing candidates in variables and looping over pages generalizes beyond the three-item page configuration that earlier methods targeted.
  • On real-world web pages of 4,000 to 20,000 tokens, the complex-task score of 39.6% implies that delegating page interpretation to child REPLs is what lets planning survive long, noisy observations that overwhelm text-action baselines.
  • The ablation that drops recursive spawning cuts success rates roughly in half, so the paper's claim is that top-down recursive decomposition, not interactivity alone, is the critical component for these benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the reported benchmarks, the same NameError-as-subtask mechanism could turn any tool-use API into a genuinely recursive language, because an LLM that calls a nonexistent function effectively declares a new tool on the fly.
  • The paper does not measure sample cost, and the rewind-and-rerun implementation may replay code frequently; a testable extension would be to compare success per dollar against message-based baselines on the same tasks.
  • A direct stress test the authors leave open is removing the entire demonstration pool and requiring the model to write all child REPLs zero-shot; their own ablation suggests performance would fall substantially, making fine-tuning on REPL-style interactions a natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes REPL-Plan, a code-augmented LLM planning approach in which an LLM interacts with a recursively spawnable REPL (LLM-REPL) to write and execute code line-by-line, with child REPLs handling fuzzy subproblems. The method is evaluated on ALFWorld, WebShop, and a new Real-World Web benchmark built from live e-commerce sites. The authors report state-of-the-art success on ALFWorld (97.0% vs. THREAD's 95.5%), strong WebShop performance in a k=10 setting, and 39.6% of expert score on complex Real-World Web tasks vs. 17.6% for ReACT and 0.0% for THREAD. Ablations show that recursive spawning and zero-shot subtask inference are important to performance.

Significance. If the reported results are robust, REPL-Plan is a meaningful advance in code-augmented LLM planning: it is novel in proposing interactive, recursively spawnable REPLs as a planning mechanism, and the qualitative trajectories show the approach can handle long web observations and recover from hallucinated actions. The paper also contributes a new real-world web navigation benchmark and a thoughtful ablation of error correction. However, the strength of the empirical claims is substantially limited by unresolved experimental confounds and lack of statistical rigor, as detailed below.

major comments (4)
  1. [Section 2.3, Table 1] The k-shot REPL-Plan keeps a 'global REPL pool, a set of spawned REPLs from previous task executions and demonstrations' (Section 2.3). This means REPL-Plan can reuse code from earlier test tasks during later test tasks, which is exactly the 'external memory' defined in Table 1's footnote. Yet Table 1 places REPL-Plan in the 'No' external-memory group, directly below THREAD. The ALFWorld comparison (97.0% vs. 95.5%) is therefore not apples-to-apples against a no-memory baseline unless the pool is explicitly reset between tasks. The paper must either state that the pool is reset per task (and then explain what 'previous task executions' refers to) or reclassify REPL-Plan as an external-memory method and compare against memory-equipped baselines. The same issue affects the WebShop and Real-World Web results.
  2. [Section 3.2, Tables 1-3; Section 3.3, Table 4] The headline empirical claims are reported without error bars, confidence intervals, or statistical tests. On ALFWorld, the gap between REPL-Plan (97.0%) and THREAD (95.5%) is only 1.5 percentage points, which with the usual ALFWorld test set of roughly 134 tasks corresponds to about 2 tasks; this difference may easily be within run-to-run noise. The Real-World Web results use only 3 trials per task (15 trials total per condition for the complex tasks) and an expert score provided by one author, with no per-task breakdown or variance. The WebShop ablation uses n=25. The paper should provide uncertainty estimates and, where possible, significance tests or at least a discussion of the minimal detectable effect at the given sample sizes.
  3. [Appendix A.1] The rewind-and-rerun implementation (saving execution state, intercepting NameError, re-running code with a call-counter cache) is the core mechanism that claims to preserve the exact sequence of environment actions and observations. However, the paper does not establish that this mechanism is semantically transparent. Re-running code from a saved state can re-execute statements with side effects on local variables, and the environment may have changed since the saved state because a prior act() already executed. The call-counter cache prevents duplicate calls to context-sensitive functions (act, child REPLs), but it does not obviously handle stateful local-variable updates that depend on observations. A correctness argument or a validation that the execution trace equals the trace of a single continuous interpreter (or equivalently, that no action is duplicated or dropped) is needed to support the claim that the reported trajectories reflect the LLM-written code as designed.
  4. [Appendix A.4 and Results in Section 3.2] The Real-World Web benchmark is very small (5 simple and 5 complex tasks, 3 trials each) and is based on live websites that can change over time, making the results difficult to reproduce or generalize. The expert score is defined as one author's manual performance, but the scoring procedure is not described in enough detail to assess inter-rater reliability or potential bias. Given that the central claim of 'scalability' to real-world web navigation rests almost entirely on this benchmark, the paper should either substantially expand the task set, report per-task and per-trial results, provide a detailed scoring rubric with inter-rater agreement, or moderate the strength of the claims to be consistent with the evidence.
minor comments (6)
  1. [Section 1] Typo: 'explictly' should be 'explicitly' in the second paragraph.
  2. [Section 3.2, Table 2] The footnote for 'THREAD*' in Table 2 says modifications are described in Appendix A.2; it would be clearer to state in the main text that THREAD was patched to avoid infinite loops on GPT-4o-mini, since this is relevant to the interpretation of the comparison.
  3. [Section 3.2 and Appendix A.4] The naming is inconsistent: 'ReACT' in Table 3 and the text, 'ReAct' in the references and in Section 4; please unify.
  4. [Section 3.3] The No-Subtask-REPLs ablation is described as 'conceptually similar to a code-augmented version of prior approaches, such as ReACT', but ReACT is not code-augmented; please rephrase to avoid a misleading comparison.
  5. [References] The citation 'Sipser, 1996' for the undecidability of determining undefined variables should be a specific theorem or section, not just a textbook citation; alternatively, rephrase to 'the undecidability of the halting problem' with a standard reference.
  6. [Figure 3] The red highlighting mentioned in the caption may not be visible in grayscale print; consider using a different visual marker (e.g., bold or underlining) to indicate the hallucination errors.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: REPL-Plan's central claims rest on empirical rollouts; the global-REPL-pool/external-memory labeling is a consistency concern, not a circular derivation.

full rationale

REPL-Plan is an empirical systems paper rather than a derivation. There is no fitted parameter that is later reported as a prediction, no first-principles equation whose output equals an input, and no load-bearing uniqueness theorem or ansatz imported from the authors' prior work. The claimed results (ALFWorld 97.0%, WebShop k=10 52%, Real-World Web complex 39.6%) are measured from environment interaction traces, not constructed from the method's definitions. The strongest consistency concern is the k-shot global REPL pool: Section 2.3 says it contains 'spawned REPLs from previous task executions and demonstrations,' while Table 1's footnote defines external memory as saving 'information from previous test tasks to new test tasks' and Table 1 lists REPL-Plan under 'No.' This is a real benchmark-comparison confound and a labeling inconsistency, but it does not make the success rates circular, because the rates are not defined as equal to the pool or to any fitted quantity. The ablations show dependence on hand-written demonstrations (Zero-shot Subtask-REPL drops from 52% to 28%), which is standard k-shot in-context behavior rather than a disguised fit. Self-citations such as AutoGuide and SkillAct appear as baselines or related work and are not load-bearing. No circular step meeting the quoted-evidence standard was found.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central results depend on hand-authored demonstrations and evaluation choices made by the authors. The REPL-Plan method is not a derivation; it is an empirical method whose performance relies on the LLM's code-writing ability, the rewind-and-rerun execution semantics, and the availability of expert k-shot demonstrations. The Real-World Web benchmark is newly constructed by the authors, and the expert score is defined by one author's run. These are stated assumptions rather than independent external benchmarks.

free parameters (5)
  • k-shot demonstration pool = hand-written REPL code per domain (see Appendix A.6)
    The method's performance depends on expert demonstrations; removing one subtask REPL (get_requirements) drops success from 52% to 28% on WebShop Top-3 (Table 4).
  • WebShop page setting k = k=3 and k=10; Top-20 strategy for k=10
    The paper introduces the k=10 setting to demonstrate scaling; REPL-Plan's best result (52% SR) comes from running a Top-20 strategy that THREAD cannot execute (Appendix A.3).
  • Number of trials on Real-World Web tasks = 3 trials per task, 5 tasks per category
    Scores are averaged over 15 episodes per method per category; no error bars are reported, and live websites change during the evaluation (Appendix A.4).
  • Observation truncation = unnecessary web page nodes stripped before LLM query
    To fit 4k-20k token pages into context, the authors truncate observations; the truncation rule is described qualitatively and used for all methods (Section 3.1, Appendix A.4).
  • THREAD anti-loop prompt = L = 'Do NOT output #START#...' (Appendix A.2)
    The THREAD baseline is patched with a custom prompt to run on GPT4o-mini; without it THREAD loops infinitely and scores 0%, increasing the apparent gap to REPL-Plan.
assumptions (4)
  • domain assumption LLMs can write executable Python code in a REPL and can trigger subtask spawning by calling undefined functions.
    The entire method depends on the LLM's code-writing and error-recovery behavior; the ablations show performance drops when subtask spawning is removed or zero-shot (Table 4).
  • ad hoc to paper The rewind-and-rerun execution in Appendix A.1 preserves the exact sequence of environment actions and observations.
    The implementation saves state, intercepts NameError, and re-runs code with a call counter cache; this is an unverified engineering assumption that, if wrong, would change what the method actually does.
  • ad hoc to paper The Real-World Web tasks and the expert score are valid and representative.
    Tasks are constructed by the authors on live e-commerce websites, and the expert score is one of the authors completing the task (Appendix A.4). There is no external validation or public release.
  • domain assumption Results obtained on OpenAI API models (GPT-3.5-instruct, GPT-4o-mini) generalize to the method's broader claims.
    All experiments use specific API models; the paper shows THREAD behaves very differently across models, so model choice is load-bearing (Section 3.2, Appendix A.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Interactive and Expressive Code-Augmented Planning with Large Language Models." pith.science (2026). https://pith.science/paper/KBKOUFKO

@misc{pith2026241113826,
  author       = {Pith},
  title        = {Pith review of: Interactive and Expressive Code-Augmented Planning with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KBKOUFKO}},
  note         = {Machine review of arXiv:2411.13826}
}
read the original abstract

Large Language Models (LLMs) demonstrate strong abilities in common-sense reasoning and interactive decision-making, but often struggle with complex, long-horizon planning tasks. Recent techniques have sought to structure LLM outputs using control flow and other code-adjacent techniques to improve planning performance. These techniques include using variables (to track important information) and functions (to divide complex tasks into smaller re-usable sub-tasks). However, purely code-based approaches can be error-prone and insufficient for handling ambiguous or unstructured data. To address these challenges, we propose REPL-Plan, an LLM planning approach that is fully code-expressive (it can utilize all the benefits of code) while also being dynamic (it can flexibly adapt from errors and use the LLM for fuzzy situations). In REPL-Plan, an LLM solves tasks by interacting with a Read-Eval-Print Loop (REPL), which iteratively executes and evaluates code, similar to language shells or interactive code notebooks, allowing the model to flexibly correct errors and handle tasks dynamically. We demonstrate that REPL-Plan achieves strong results across various planning domains compared to previous methods.

Figures

Figures reproduced from arXiv: 2411.13826 by the authors.

Figure 1
Figure 1. REPL-Plan is an approach for augmenting LLM-planning by using LLMs to interact with LLM-REPLs, which are an extension of REPLs (Read-Eval-Print-Loops, e.g. language shells, code notebooks). [A] Observations are printed out in the main LLM-REPL and all child LLM-REPLs. [B] [C] The LLM, which outputs code in the LLM-REPL line by line, interacts with the environment by calling the act function. Each LLM-REPL (including… view at source ↗
Figure 2
Figure 2. A toy example of context passing that is possible in REPL-Plan. In the toy example, the task is parse all [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. For a qualitative analysis, we include truncated versions of trajectories from REPL-Plan and the baseline [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Another toy example of context passing that is possible in REPL-Plan, where context is “interleaved” [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 12 canonical work pages

  1. [1]

    [id for id, price in id_to_price.items() for price < max_price]

    work page

  2. [2]

    Preprint, arXiv:2302.06706

    On the planning abilities of large language models (a critical investigation with a proposed benchmark). Preprint, arXiv:2302.06706. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Man- dlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023a. V oyager: An open- ended embodied agent with large language models. Preprint, arXiv:2305.16291. Lei Wang...

    arXiv 2023

  3. [3]

    Add any monitor to the cart

    On staples.com, search for monitors. Add any monitor to the cart

    work page

  4. [4]

    Add any backpack to the cart

    On zappos.com, search for backpacks. Add any backpack to the cart

    work page

  5. [5]

    interleaved

    [id for id, price in id_to_price.items() for price < get_max_price()] In the case that max_price and get_max_price are undefined, both lines would raise a NameError. In both cases, in our framework, an LLM-REPL will be created and named max_price and get_max_price respectively. However, in the implementation of an LLM-REPL, we overrideany non-function met...

    work page 2024

  6. [6]

    Add any laptop to the cart

    On bestbuy.com, search for laptops. Add any laptop to the cart

    work page

  7. [7]

    Add any desktop computer to the cart

    On dell.com, search for desktop computers. Add any desktop computer to the cart

    work page

  8. [10]

    Add any bath towel to the cart

    On nordstrom.com, search for bath towels. Add any bath towel to the cart. Complex tasks:

    work page

Show all 15 references
  1. [11]

    Loop through each search result and add every printer that is capable of printing at least 12 pages per minute to the cart

    On bestbuy.com, search for printers. Loop through each search result and add every printer that is capable of printing at least 12 pages per minute to the cart

  2. [12]

    Loop through each search result and add every keyboard with cherry key switches to the cart

    On dell.com, search for keyboards. Loop through each search result and add every keyboard with cherry key switches to the cart

  3. [13]

    Loop through each search result and add every shredder with at least 6 gallons of capacity to the cart

    On staples.com, search for shredders. Loop through each search result and add every shredder with at least 6 gallons of capacity to the cart

  4. [14]

    Loop through each search result and add every pair of sunglasses that is polarized to the cart

    On zappos.com, search for sunglasses. Loop through each search result and add every pair of sunglasses that is polarized to the cart

  5. [15]

    "" noise cancelling cosycost usb microphone , and price lower than 70.00 dollars

    On nordstrom.com, search for watches. Loop through each search result and add every watch that is swiss made to the cart. Approaches. We tested REPL-Plan and the baselines ReACT and THREAD on the Real-World Web environment. For each approach, to help the LLM deal with long pag...

  6. [2023]

    Preprint, arXiv:2304.11477

    Llm+p: Empowering large language mod- els with optimal planning proficiency. Preprint, arXiv:2304.11477. OpenAI. 2023a. Gpt-3.5. https://platform.openai. com/docs/models/gpt-3-5. Accessed via OpenAI API. OpenAI. 2023b. Gpt-4. https://openai.com/ research/gpt-4. Accessed via Op...

    2024 arXiv

  7. [2024]

    Preprint, arXiv:2401.00812

    If llm is the wizard, then code is the wand: A survey on how code empowers large language models to serve as intelligent agents. Preprint, arXiv:2401.00812. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022a. Webshop: Towards scalable real-world web interaction ...

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.