{"id":"83106275-0402-453f-82ec-f60c28b9e481","arxiv_id":"2506.21030","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"STEP builds a coarse-to-fine subgoal tree with LLM-based decomposition and termination checks, reporting higher task success than existing LLM planners on WAH-NL and a real robot.","lead":"A robotics planning system that uses a language model to break a long task into a tree of smaller subgoals, then checks each branch to decide when it is small enough to execute directly. The authors report better success on a household-task benchmark and on a real robot than several existing language-model planners.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract and tables report incompatible success rates (34%/25% vs 40%/60%), leaving the central quantitative claim ambiguous.","rationale":"The reader's weakest_assumption focuses on the LLM-based leaf node termination model, which is a legitimate technical risk: if mappability/consistency verdicts are unreliable, the tree may terminate at wrong nodes and invalid actions could be executed. However, the empirical success rates are the evidence that the system works despite this risk. A more fundamental, directly load-bearing concern is that the paper's own reported headline numbers contradict each other across the abstract/contributions and the results tables. This inconsistency attacks the factual basis of the central claim before any technical mechanism is considered. The reader did flag the abstract-table mismatch in the rationale, so agreement is partial: we share the concern but elevate it to the primary one. The appropriate verdict remains CONDITIONAL, pending clarification of the results and access to raw data. No new verdict adjustment is needed beyond the reader's conditional assessment.","tokens_in":10170,"tokens_out":6794,"duration_ms":77922,"concrete_test":"Obtain from the authors the raw evaluation logs for all 100 WAH-NL tasks and all 40 real-robot trials (10 per category). Recompute the overall success rate and per-category success rates using the paper's own task-success criteria. Then check whether the abstract/contributions figures (34%/25%) or the table figures (40%/60%) match the recomputed values. If the recomputed WAH-NL SR is 34% rather than 40%, Table I is erroneous; if it is 40%, the abstract and contributions are erroneous. Also request the explicit definition of the '25% real robot' metric to resolve which aggregation of Table III is meant.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline results are internally inconsistent. The Abstract and Section I/III claim 'up to 34% (WAH-NL) and 25% (real robot)', while Table I reports STEP at 40% SR on WAH-NL and Table III reports 6/10 on long-complex real-robot tasks (60%). The real-robot 25% figure does not match any straightforward aggregation of the Table III category rates (100%, 100%, 90%, 60% for short-simple, short-complex, long-simple, long-complex). Since the central claim is precisely these quantitative improvements over SOTA, the reader cannot determine which set of numbers is authoritative. This is not a harmless typo: the contributions section also repeats 34%/25%, so either the tables or the text are wrong, and the 40% / 6-of-10 figures cited in the strongest_claim may be unsupported. Without the raw per-task records, the discrepancy cannot be resolved, and the headline result is ambiguous.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes STEP, an LLM-based embodied long-horizon planner that recursively decomposes a natural-language goal into a subgoal tree. A subgoal decomposition model expands nodes, and a leaf-node termination model uses mappability and consistency criteria to decide whether a node is executable as a primitive action, should be decomposed further, or should trigger replanning. The system is evaluated on VirtualHome WAH-NL (100 annotated tasks) against SayCan, ProgPrompt, and LoTa-Bench, and on a Franka Panda with RoboScript against the same baselines plus RoboScript. The paper reports that STEP reaches 40% success rate on WAH-NL versus 6% for LoTa-Bench, and 6/10 versus 1/10 on long-complex real-robot tasks, with ablations showing that the tree structure contributes most of the gain.","tokens_in":10320,"tokens_out":7586,"duration_ms":79465,"significance":"The central idea is timely: recursively decomposing instructions into a coarse-to-fine subgoal tree is a natural way to reduce both context length and the logical distance between instruction and action. The method has no trained parameters and is evaluated against an external benchmark with objective subgoal-satisfaction criteria, which lowers circularity concerns. If the reported improvements survive a corrected comparison and more careful real-robot statistics, the result would be a useful and simple planning framework. However, the current paper's quantitative claims are obscured by inconsistent presentation, missing comparisons, and a small real-robot sample.","major_comments":[{"comment":"The abstract and contributions report 'success rates up to 34% (WAH-NL) and 25% (real robot)', while Table I reports STEP at 40% SR and Table III reports 6/10 in the long-complex real-robot category. Section IV-A clarifies that 34 percentage points and 25.23 percentage points are improvements over LoTa-Bench (40-6=34; 62.02-36.79=25.23), and the real-robot aggregate is also a 25 percentage point improvement (87.5% minus 62.5% across the four categories). Because the abstract does not say 'improvement', the headline numbers are internally inconsistent as written. The abstract and contributions should be rewritten to state the base rates and the improvement explicitly.","section":"Abstract and Section I, Contribution (III)"},{"comment":"The real-robot evaluation uses two tasks per category with five trials each, giving n=10 per condition and no confidence intervals or significance tests. For the headline long-complex comparison, STEP's 6/10 versus LoTa-Bench's 1/10 yields a Fisher exact p-value of approximately 0.03 one-sided and about 0.06 two-sided; with only two distinct tasks per category this is not a robust statistical basis for the claim of superiority. Please report the raw per-task records, add confidence intervals, and perform a suitable significance test or explicitly weaken the conclusion.","section":"Section IV-B, Real Robot"},{"comment":"The baseline set omits Tree-Planner and RAP, the tree-structured planners discussed in Section II-B as the closest related work. Without evaluating against these methods or explaining why they are excluded, the statement that STEP 'outperforms SOTA methods' (Abstract and Section I) is not supported. The authors should add these baselines to the VirtualHome comparison or restrict the claim to the compared set.","section":"Section IV-A, Baselines"},{"comment":"Algorithm 1 does not implement the three-way leaf-node decision described in Section III-C. The text states that a non-mappable but consistent subgoal should be further decomposed, while an inconsistent subgoal should trigger replanning; Algorithm 1 instead routes every non-mappable result to line 13 (subgoal = subgoal.parent) and never adds the newly decomposed subgoal to the tree. This makes the algorithm impossible to execute as written and prevents reproduction. Replace Algorithm 1 with a correct procedure and make the tree construction explicit.","section":"Section III-C, Algorithm 1"},{"comment":"The formal criteria in Eqs. (4) through (7) are not well-formed. Variables such as ai, T, Tt, and ΦT are used without definitions, and the formulas state injectivity or containment conditions rather than the mappability and consistency predicates they claim to define. Since the termination model is the central control loop, the authors should either specify the actual LLM evaluation protocol (for example, prompt template and output schema) or clearly mark these equations as illustrative rather than formal definitions.","section":"Section III-C, Eqs. (4)-(7)"}],"minor_comments":[{"comment":"There are typos such as 'permitive action' (should be 'primitive action') and 'enabliing' in Section I; the manuscript would benefit from a careful proofread.","section":"Section III-A"},{"comment":"The notation Φn({bi}|n i=0) is used with braces and subscripts that are not defined consistently; please rewrite the decomposition operator notation so that the recursion is unambiguous.","section":"Section III-B, Eq. (2)-(3)"},{"comment":"The error-type percentages in Table II do not sum to 100 (for example, the STEP row sums to 37%), and the 15% Grammar Error rate discussed in the text appears nowhere in the table. Please state the denominator and include grammar errors as a column.","section":"Table II"},{"comment":"The description 'annotated dataset of 100 entries in the WAH-NL benchmark, which consists of 350 labeled robot datasets' is ambiguous; clarify whether 100 tasks or 350 episodes were evaluated and how the two numbers relate.","section":"Section IV-A, Dataset"},{"comment":"No code, prompts, LLM version for VirtualHome, or sampling parameters are provided; please include these in the paper or supplement, as the method is otherwise difficult to reproduce.","section":"Reproducibility"},{"comment":"Reference [8] is a duplicate of reference [7] and does not appear to cite Tree-Planner as intended; please correct the citation list in Section II-B.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is not beyond repair: the 34%/25% numbers in the abstract are consistent with percentage-point improvements once Section IV-A is read carefully, but the wording is dangerously ambiguous. The more substantive issues are the absence of the closest tree-based baselines and the very small real-robot sample. I would encourage the editor to ask for a revision that fixes these before further consideration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the tables, not the abstract. The 34% and 25% cited in the abstract and contributions are the percentage-point gains over LoTa-Bench (6%→40% on WAH-NL; 62.5%→87.5% aggregate on the real robot), not absolute success rates. So the apparent discrepancy flagged in the stress-test note is a writing error, not a numeric contradiction. The central quantitative claim is internally consistent: 40% SR versus 6% for LoTa-Bench on WAH-NL, and 6/10 versus 1/10 on the real-robot long-complex category.\n\nWhat is genuinely useful: the closed loop between a decomposition model and a leaf-node termination model. The termination model checks mappability and consistency before deciding to decompose further, execute a primitive, or replan. Tree-Planner and RAP sample full plans or lean on world models; STEP instead grows a tree and stops each branch when the current subgoal can map to a single action. That is a modest but real extension. The ablations support it: removing the tree structure drops WAH-NL SR from 40% to 8%, and removing the subgoal tree drops it to 9%. The error analysis is also honest—the main residual failure mode is Additional/Missing Steps, which they attribute to LLM misjudgments in the termination model.\n\nSoft spots, in order of severity. First, no comparison against Tree-Planner or RAP, the very tree-based methods they cite as motivation. A reviewer will ask for this immediately. Second, no code or prompts released; the termination model is the heart of the system, and it is specified only in prose and equations. Third, the real-robot sample is small: two tasks per category, five trials each, so the headline long-complex result rests on ten trials. It is consistent with the simulator, but not robust on its own. Fourth, the formal notation in Section III does no work; it restates that an LLM decomposes a goal, and it could be cut or simplified. None of these sink the paper, but they determine whether it is a solid workshop-level contribution or a citable archival one.\n\nWho this is for: researchers building LLM-based planners for long-horizon embodied tasks, especially those integrating skill libraries. The termination criteria and the failure-mode table are worth reading. I would send this to peer review rather than desk-reject. It needs a serious referee, but it is not there yet—missing baselines and missing artifacts are fixable, but they must be fixed.","headline":"Read the tables, not the abstract: the 34%/25% figures are percentage-point improvements over LoTa-Bench, not absolute success rates, and the core claim of a real gain from subgoal-tree decomposition with an LLM termination check is plausible but needs a few missing baselines and artifacts.","tokens_in":10904,"tokens_out":3229,"would_cite":false,"duration_ms":38513,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"STEP constructs a hierarchical subgoal tree through two closed-loop LLM models, claiming 40% success on the WAH-NL long-horizon benchmark and 6/10 on long-complex real-robot tasks, up from 6% and 1/10 for the strongest baselines.","keywords":["long-horizon task planning","subgoal tree","large language models","embodied agents","robot task planning","hierarchical decomposition","leaf node termination","VirtualHome"],"falsifier":"Run the WAH-NL benchmark with a version of STEP where leaf-node termination is decided by a deterministic ground-truth check (a subgoal is terminal only if it exactly matches a primitive action known to be executable in the current state) instead of the LLM's mappability/consistency verdict; if this deterministic version does not reproduce or exceed the reported 40% success rate, then the framework's claimed success rides on the LLM termination judgments rather than on the tree structure itself.","tokens_in":9965,"feed_emoji":"🤖","tokens_out":4138,"duration_ms":44015,"temperature":0.7,"pith_summary":"The paper tries to establish that long-horizon embodied planning becomes reliable when a large language model never has to reason about the whole task at once. Instead, STEP recursively decomposes a natural-language goal into a subgoal tree, and only stops decomposing a branch when a separate termination model judges the current subgoal to be directly executable as a primitive action. The authors argue this shrinks two failure sources: the contextual gap, where long instruction and action histories add irrelevant information that degrades LLM reasoning, and the logical gap, where abstract instructions are too far from concrete actions for one inference step. If the claim is right, then the route to better long-horizon robot planners is not a stronger planner but a structure that feeds each reasoning step only the information it needs. The paper reports head-to-head gains on the VirtualHome WAH-NL benchmark and on a real Franka Panda robot.","feed_headline":"Subgoal-tree planner lifts long-horizon robot success to 40%","feed_subtitle":"Decomposing tasks into a coarse-to-fine subgoal tree beats direct LLM planning on WAH-NL and on a real Franka Panda.","key_machinery":"The central object is the subgoal tree: a hierarchy whose root is the natural-language goal, each child node is a subgoal that contributes to its parent's effect, and leaf nodes are primitive actions. Spinning up this tree is done by two closed-loop models: the subgoal decomposition model, which generates the next subgoal from the parent node, the previously generated subgoal at the same level, and the current observation; and the leaf node termination model, which evaluates each subgoal against mappability criteria (can it map to a distinct primitive action?) and consistency criteria (does it match the embodiment's affordances, agree with task congruence, and fit the current environmental constraints?). The termination model's verdict decides whether to execute the action, decompose further, or replan the parent. This architecture confines each LLM call to a single-level decomposition, excluding history from other branches and thereby reducing both the contextual and logical gaps identified in the paper.","core_discovery":"The central discovery is that a subgoal tree built by recursive decomposition, with a closed-loop leaf-node termination model deciding when to stop decomposing, substantially outperforms direct LLM planning and existing tree-search planning baselines on long-horizon embodied tasks. On the WAH-NL benchmark, STEP reaches 40% overall task success and 62.02% subgoal success, versus 6% and 36.79% for LoTa-Bench, and on real-robot tasks it completes 6/10 long-complex trials versus 1/10 for all baselines. The paper's error analysis attributes its remaining failures mainly to Additional/Missing Steps, which stem from misjudgments by the LLM-based leaf-node termination model. Ablations show that removing the tree structure drops success to 8%, and removing the subgoal condensation drops it to 9%, supporting the claim that both contextual-gap reduction and logical-gap reduction contribute to the gains.","pith_inferences":["Beyond the paper's claims, the subgoal-tree idea could be combined with sampling or search at each decomposition level, e.g., generating several candidate subgoal sets and selecting among them, which the paper does not explore.","A testable extension the paper leaves open is replacing the LLM termination judgment with a learned feasibility classifier or a simulator-based check; if such a replacement lifts success further, it would confirm the termination model as the bottleneck.","The contextual-gap argument suggests a measurable prediction: success rate should correlate inversely with the length of the context actually fed to each decomposition call, independent of total task length, which could be tested by varying context condensation.","The decomposition-into-subgoal-tree pattern might transfer beyond text-based planning to vision-language-action policies, where each subgoal could condition a different low-level policy or skill instead of a primitive action."],"forward_implications":["If STEP's claim holds, LLM-based planners can handle significantly longer tasks by replacing full-horizon reasoning with per-level decomposition, without requiring a stronger backbone model.","The reported error distribution points to the leaf-node termination model as the main bottleneck, so improving mappability and consistency judgments should directly raise overall success rates.","The real-robot results suggest the advantage grows with task complexity, since the largest gap over baselines appears in long-complex tasks involving hidden objects and 5-8 step sequences.","Ablation results imply that both the tree structure's context condensing and the subgoal reasoning condensation are individually necessary, not just helpful, for the reported performance.","Because the grammar-error rate stays similar to baselines using the same LLM, the gains are attributable to the planning structure rather than to superior language generation."],"supporting_citations":[{"why":"Provides the VirtualHome simulator used as the main experimental environment for household task planning.","marker":"[28]"},{"why":"Supplies the WAH-NL benchmark and the LoTa-Bench baseline, the strongest comparison method whose 6% success rate STEP seeks to beat.","marker":"[29]"},{"why":"ProgPrompt is a code-generation planning baseline compared on the same benchmark.","marker":"[30]"},{"why":"SayCan is an affordance-grounded planning baseline used for comparison in both simulation and real-robot experiments.","marker":"[15]"},{"why":"RoboScript's API provides the low-level motion planning and control on the real Franka Panda robot.","marker":"[32]"},{"why":"The EAI taxonomy supplies the error classification used in Table II to analyze failure modes across methods.","marker":"[31]"}],"fun_headline_variants":["Subgoal tree planner hits 40% on long-horizon benchmarks","Closed-loop subgoal tree beats direct LLM planning","Coarse-to-fine subgoal tree lifts robot success to 40%","STEP planner's subgoal tree outperforms SOTA on tasks","Tree-based subgoal decomposition improves embodied planning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method depends on the LLM correctly deciding when a subgoal is already a primitive action; if it says 'mappable' too early, an invalid action is executed, and if it says 'not mappable' too often, the planner over-decomposes or replans unnecessarily, with no verification against ground truth.","fun_headline_variants_meta":{"raw":{"variants":["Subgoal tree planner hits 40% on long-horizon benchmarks","Closed-loop subgoal tree beats direct LLM planning","Coarse-to-fine subgoal tree lifts robot success to 40%","STEP planner's subgoal tree outperforms SOTA on tasks","Tree-based subgoal decomposition improves embodied planning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000991,"raw_usage":{"total_tokens":4204,"prompt_tokens":950,"completion_tokens":3254,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":3169}},"tokens_in":566,"tokens_out":3254,"duration_ms":23904,"temperature":1.0,"reasoning_tokens":3169,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:34:55.073400+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the WAH-NL benchmark with a version of STEP where leaf-node termination is decided by a deterministic ground-truth check (a subgoal is terminal only if it exactly matches a primitive action known to be executable in the current state) instead of the LLM's mappability/consistency verdict; if this deterministic version does not reproduce or exceed the reported 40% success rate, then the framework's claimed success rides on the LLM termination judgments rather than on the tree structure itself.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the VirtualHome simulator used as the main experimental environment for household task planning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ProgPrompt is a code-generation planning baseline compared on the same benchmark."}],"review_version":1}