{"id":"0b3a7e23-ca4d-4074-8223-4d44ad89d5dd","arxiv_id":"2502.02067","paper_version":2,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"An LLM+knowledge-graph+human-in-the-loop framework improves simulated task-completion success over LLM-only and LLM+KG baselines, though the human oracle inflates the reported gains.","lead":"This paper describes a robot planning framework that combines LLM-suggested action sequences, a knowledge graph of the robot's world, and human corrections to complete cooking and cleaning tasks in simulation. The authors report large success-rate gains over using the LLM alone, but the comparison rests on allowing the system to query a human oracle during the task.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains are not attributable to the LLM+KG mechanism alone: human-supplied actions are executed and then scored by the same human population, so Tables I–II may measure the oracle rather than the framework.","rationale":"The paper describes a plausible and transparent integration of LLM, KG, and human feedback, and the qualitative observations—KG growth and reduced token usage—are consistent with the mechanism. The linked-trials design is a good feature, and the KG updates are explicit. However, the central experimental claim is undercut by a measurement confound: the human oracle provides missing knowledge during execution and the same human population then scores the resulting traces. Because the paper does not report the proportion of human-supplied steps, error bars, significance tests, or a blinding protocol, the reported success-rate improvements do not establish that the LLM+KG machinery, rather than the human's own knowledge, drives the gains. This is not an internal inconsistency or a disagreement with consensus; it is a correctness risk in the evidence supporting the headline. The framework may well be sound, but the paper as written does not robustly support 'substantial performance gains' as an autonomous-adaptation result. Reader's CONDITIONAL verdict is therefore appropriate; no change is needed, though the required revision is substantial: either report the masked/oracle-free analysis or explicitly re-scope the claim to human-assisted task completion.","tokens_in":11876,"tokens_out":3831,"duration_ms":41575,"concrete_test":"Re-run the evaluation with per-task logs of every human correction, then have a disjoint set of blinded annotators score two versions of each execution trace: the original output and the same trace with all human-supplied steps masked or removed. If the masked-trace success rate is statistically indistinguishable from the LLM+KG baseline, or if the human-supplied fraction is large, the reported gains are attributable to the oracle rather than to the LLM+KG mechanism, and the claim should be re-scoped accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the LLM+KG+Human interplay yields substantial gains—rests on Tables I and II, but the evaluation protocol conflates the framework's adaptation with the human oracle. In Section 3.4 and Algorithm 1 (lines 14–18), unresolved mismatches trigger ask_human, which can supply missing action names, object attributes, and existence judgments; Section 4.1.2 states this human input is 'assumed to be accurate.' The executed output is then scored by 18 human participants on a 0–20 scale (Section 4.1.4). No quantity is reported for how many final action steps were human-supplied, and no blinding or separation is described between the people who provide corrections and those who evaluate the outputs. Under this protocol, a system that simply asked the human for the whole plan whenever it was stuck would receive high scores, so the observed jumps (e.g., GPT-4o cooking success 45.2% → 91.14%; cleaning 42.33% → 75.66%) cannot be read as evidence that the LLM+KG component is doing the adaptation. The token-use reduction is suggestive but not decisive, since human input is also replacing failed LLM re-prompts. The claim must be re-scoped to 'performance with an accurate human oracle in the loop,' or the autonomous contribution must be measured separately.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AdaptBot, a framework that combines LLM-generated generic action sequences with a Knowledge Graph encoding domain-specific prior knowledge and with human-in-the-loop feedback. The LLM first produces a high-level action sequence for a task; the KG is used to refine the sequence by replacing unknown or inapplicable actions, objects, and attributes; unresolved mismatches or execution errors trigger re-prompting of the LLM and, beyond a threshold, solicitation of human input that expands or corrects the KG. The system is evaluated on 30 cooking tasks sampled from Recipe1M+ and 12 cleaning/clearing tasks in simulation, comparing three configurations: LLM only, LLM+KG, and LLM+KG+Human. Tables I and II report higher success rates, lower token usage, and higher ingredient overlap for the full configuration, and the paper claims that the interplay between LLM, KG, and human input leads to substantial performance gains and enables adaptation to new task classes through incremental knowledge refinement.","tokens_in":12074,"tokens_out":4391,"duration_ms":44404,"significance":"If the reported results were properly supported, the paper would make a useful contribution to robot task planning: the idea of using an LLM for generic decomposition and a KG for transparent, incremental refinement is attractive, and the authors include an explicit algorithm (Algorithm 1), transparent knowledge-update equations, linked trials across baselines, and two task domains with two LLMs. The main limitation is that the experimental protocol as described cannot separate the contribution of the framework from the contribution of the human oracle: the human can supply missing actions and object attributes during execution, and human evaluators then score the executed outputs. The observed gains in Tables I and II may therefore be due to the oracle rather than to the LLM+KG mechanism. The core architectural idea is defensible, but the evidence presented in support of the central claim needs re-scoping or a substantially stronger evaluation protocol.","major_comments":[{"comment":"The evaluation conflates the framework's adaptation with the human oracle. In the LLM+KG+Human arm, unresolved mismatches are resolved by ask_human, which can supply missing action names, object attributes, and existence judgments, and Section 4.1.2 states that this human input is 'assumed to be accurate.' The executed plan therefore contains human-authored steps, but Tables I and II report only final aggregated success rates; no quantity is reported for how many final action steps were human-supplied, nor for which tasks human intervention occurred. Under this protocol, a system that simply asked the human to complete the plan whenever it was stuck would receive high scores, so the reported jumps (e.g., GPT-4o cooking from 45.2% to 91.14%) cannot be read as evidence that the LLM+KG component is doing the adaptation. Please report autonomous-only performance (e.g., performance before any human correction, or performance on later trials after KG expansion), per-task human-intervention counts, and the fraction of final action steps authored by the human.","section":"Section 4.1.2 and Algorithm 1, lines 14-18"},{"comment":"The evaluation protocol does not establish that the scoring is fair across conditions. The manuscript says that 18 evaluators scored the execution outputs, that at least three raters scored each task, and that evaluators were 'not involved in the design,' but it does not state whether the evaluators were blinded to which condition produced each output, whether progress lines were shown for all conditions, or whether the people who supplied corrections during execution were the same population as the evaluators. Without blinding and separation, the success-rate scores may measure how closely the final output matches the evaluators' own expectations, which is especially problematic when the final output contains corrections supplied by a human. Please describe the evaluation interface, the blinding procedure, inter-rater agreement, and the relationship between the correcting participants and the scoring participants.","section":"Section 4.1.4"},{"comment":"No measures of uncertainty are reported for any of the headline numbers. Each table cell is a single average over 30 or 12 tasks with at least three ratings per task, and the claims of 'substantial improvement' in Section 4.2 are based on these single numbers without confidence intervals, standard deviations, or significance tests. Given the small number of tasks and the large reported differences, per-task score distributions, bootstrap confidence intervals, or pairwise significance tests are needed to determine whether the differences are robust; the paper should also report the value of Fmax used and any sensitivity analysis for this threshold, since it controls how often the human is queried.","section":"Section 4.1.4 and Tables I and II"},{"comment":"The token-use reduction is not clear evidence of efficiency gains from the KG. Because unresolved mismatches in the human arm are resolved by asking the human rather than by additional LLM feedback prompts, the lower average token count is a direct consequence of the protocol: the human's answer replaces the LLM calls that the baselines continue making. The paper should report token counts separately for the LLM-generated portion and for the human-query overhead (including the interactive attribute acquisition described in Section 3.4), and should avoid presenting the reduction as evidence of the KG's contribution without an ablation that holds the number of feedback rounds fixed across conditions.","section":"Section 4.2, H2 and Table I"}],"minor_comments":[{"comment":"The sentence 'KGs have have been used to model prior knowledge' contains a duplicated word ('have have').","section":"Section 2"},{"comment":"The sentence 'We describe out framework's components below' should read 'our framework's components'.","section":"Section 3.1"},{"comment":"The phrase 'until the tasks is completed' should be corrected to 'until the task is completed.'","section":"Text following Algorithm 1"},{"comment":"In the definition of Scurrent, the second tuple is written as (sn, sn); it should presumably be (sn, vn) to match the first tuple and the surrounding notation.","section":"Section 3.4"},{"comment":"The reported differences of 45.94 and 34.19 percentage points are labeled 'increase in success rate'; these are percentage-point differences relative to the baselines, not percentage increases, and should be stated as percentage-point gains to avoid ambiguity.","section":"Section 4.2, H2"},{"comment":"The experimental section does not name the simulator or environment used for the cooking and cleaning tasks, although the introduction cites three candidate simulators; please specify the simulator, version, and task definitions so that the experiments are reproducible.","section":"Section 4.1"},{"comment":"The success-rate measure is described as 'computed based on the scores assigned by the human participants,' but the exact evaluation question, the rubric for the 0-20 scale, and the mapping from scores to binary success are not provided; please include the precise instructions given to evaluators.","section":"Section 4.1.5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of the journal and the proposed idea has merit, but the load-bearing experimental claim is not currently supported because the HITL evaluation protocol conflates the framework's contribution with an assumed-accurate human oracle. The authors could address this by re-scoping the claim to 'performance with an accurate human oracle in the loop' and by adding an ablation that measures the autonomous contribution, for example by evaluating performance before human correction or on subsequent trials after KG expansion. Please also consider requiring the authors to make the simulator, prompts, and evaluation materials available for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a reasonable integration of LLM decomposition, KG-based checking, and human-in-the-loop refinement, with a transparent knowledge graph and a sensible algorithm. But the headline numbers do not measure what they claim. In the LLM+KG+Human condition, unresolved mismatches are resolved by asking a human for the correct action or attribute, the corrected sequence is executed, and then a human panel scores the output. The paper assumes human input is accurate (Section 4.1.2) but never reports how many steps were human-supplied. Under that protocol, the success-rate jumps (e.g., GPT-4o cooking 45.2% to 91.14%) largely measure the oracle, not the framework's adaptation. The stress-test note lands on this cleanly, and I agree with it.\n\nWhat the paper does well: the framework is clearly described (Algorithm 1 is readable), the KG expansion is tangible and transparent, and the linked-trials setup—same LLM output fed to all configurations—is a fair way to compare components. The token-use reduction is suggestive, though it is partly an artifact of replacing failed LLM re-prompts with human answers rather than a pure efficiency gain. The related work is adequate and covers ADaPT, TaskBench, and FOON+LLM.\n\nSoft spots beyond the oracle problem: no error bars, confidence intervals, or significance tests; only 30 cooking and 12 cleaning tasks; the simulator is not named in the evaluation section (the intro cites CoppeliaSim, VirtualHome, and AI2-THOR, but the experiments do not say which one was used); no code or data are provided, so the results are not independently checkable. The similarity matching (exact-word plus hypernyms) is simple but that is a minor issue given the paper's transparency goal. The claim of 'substantial performance gains' should be re-scoped to 'gains when an accurate human oracle is available in the loop,' or the autonomous contribution should be measured separately.\n\nThe central idea is worth pursuing, and the paper deserves a serious referee because the integration is timely and the framework is plausible. But the evaluation must be redone—separate the people who provide corrections from the people who score, report the number of human-supplied steps, add an autonomous-only variant, and run more trials with proper statistics. As written, I would not cite the numbers as evidence of autonomy.\n\nRecommendation: send to peer review, but expect a major revision focused on the evaluation protocol.","headline":"A sensible LLM+KG+human-in-the-loop integration whose reported gains are inflated by an evaluation that lets the human oracle both fix the plan and score the result.","tokens_in":12664,"tokens_out":1547,"would_cite":false,"duration_ms":18402,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An embodied agent can adapt to new tasks by combining a large language model's generic plans with knowledge-graph repair and occasional human corrections, and this loop substantially outperforms the language model alone in simulation.","keywords":["Large Language Models","Knowledge Graphs","Human-in-the-Loop Learning","Task Decomposition","Embodied Agents","Cooking Tasks","Cleaning Tasks","Knowledge Refinement"],"falsifier":"Run the full framework on the same 30 cooking tasks but replace the human oracle with a random-answer script and score the executions with evaluators who do not know which configuration produced each trace; if success rates stay near 91–92%, human accuracy is not load-bearing, whereas a collapse toward the LLM-only baseline would confirm that the reported gains rely on correct human input.","tokens_in":11640,"feed_emoji":"🤖","tokens_out":10216,"duration_ms":89945,"temperature":0.7,"pith_summary":"This paper tries to show that a robot can handle a new task without retraining by letting three components compensate for each other: a large language model produces a rough step-by-step plan, a knowledge graph of the specific environment filters and repairs that plan, and a human supplies the few missing details the graph does not know. The central claim is that the loop formed by these three pieces outperforms using the language model alone, and also outperforms the language model plus knowledge graph without human help. The authors support this with simulated cooking and cleaning experiments in which the full loop raises success rates from roughly 45% to 91% with one language model and from 25% to 92% with another, while reducing token usage. The payoff, if true, is a practical route to adaptable assistive robots that accumulate knowledge incrementally and transparently rather than needing comprehensive pretraining.","feed_headline":"Robots learn new chores via LLMs, knowledge graphs, human hints","feed_subtitle":"In cooking simulations, the three-way loop lifts success from 45% to 91% and cuts token use.","key_machinery":"The load-bearing mechanism is a two-layer knowledge graph used as a filter-and-repair layer between the LLM and the physical actions: a state graph stores instances with their current attributes and locations, and an attribute graph stores class-level properties such as 'sliceable' or 'fryable', both encoded in the RDF triple format. The paper checks every LLM-proposed action and object against these graphs with SPARQL queries; mismatches are repaired by direct word matching over hypernyms and hyponyms, or flagged as unknown. The second half of the machinery is a feedback controller: unresolved unknowns or execution errors send a feedback prompt back to the LLM up to a fixed number of times, and only then draw on a human, whose corrections update the graphs. This explicit triple-based update is what makes the knowledge growth transparent and reusable across subsequent tasks.","core_discovery":"The paper's discovery is that the bottleneck in LLM-based robot task execution is not the plan but the fit between the plan and the robot's actual world, and that this fit can be achieved by explicit knowledge repair. An LLM is prompted with an in-context example to produce a step-by-step action sequence for a new task. That sequence is checked against a knowledge graph stored as RDF (a standard triple-based graph format) representing states and attributes; known objects and actions pass, unknown ones are replaced with hypernym or hyponym matches from the graph, and anything that cannot be resolved triggers re-prompting of the LLM with feedback. If mismatches or execution errors persist beyond a threshold, the robot asks a human, uses the answer to correct the action, and writes the new object, attribute, or state into the knowledge graph as explicit triples. The reported result is that this combined loop succeeds far more often than the LLM alone, and that the improvement is tied to the knowledge expansion: for example, adding a single missing object such as a mopping cloth to the graph produced a 31–56% rise in success on cleaning tasks.","pith_inferences":["The paper assumes human answers are accurate; a natural stress test would be to inject occasional incorrect answers and measure how quickly errors propagate through the knowledge graph.","Because all knowledge updates are explicit RDF triples, the learned knowledge from one task class could in principle be audited or transferred to another domain, a possibility the paper leaves open.","The current repair step uses direct word matching, which suggests a probable failure mode when the LLM and the graph use different phrasing; an embedding-based matcher might extend the approach to paraphrase-heavy inputs.","The success metric comes from human evaluators who were not necessarily blinded to which configuration produced each execution; a follow-up with blinded scoring would clarify how much of the reported gain is due to the loop itself."],"forward_implications":["If the reported results hold, an assistive robot can take on a new task class after only a handful of human-supplied knowledge insertions, with no retraining or large labeled datasets.","Because the knowledge graph catches and repairs errors before re-prompting, the framework uses fewer tokens than the LLM-only baseline while succeeding more often.","The framework transfers from cooking to cleaning and clearing tasks, so the same architecture supports qualitatively different task families.","As the graph expands, the need for human intervention decreases, so the system becomes progressively more autonomous on the tasks it has seen."],"supporting_citations":[{"why":"Supplies the prior task-tree method and the progress-line evaluation that this framework is compared against.","marker":"[8]"},{"why":"Provides the functional object-oriented knowledge network that motivates the KG-based representation of cooking knowledge.","marker":"[25]"},{"why":"Provides the recipe dataset from which the 30 cooking tasks are sampled.","marker":"[29]"},{"why":"Grounds the use of LLM task decomposition into sub-tasks that the framework begins from.","marker":"[4]"},{"why":"Represents prior work that retrieves task trees from a knowledge network for cooking, a direct baseline approach.","marker":"[7]"},{"why":"Supplies the reflexive human-feedback loop that informs the human-in-the-loop component.","marker":"[32]"}],"fun_headline_variants":["LLM + knowledge graph + human input refines robot task plans","Robots adapt to new tasks using LLM, knowledge graphs, and human guidance","Knowledge repair boosts robot task success via LLM, KG, and human input","AdaptBot: LLM meets knowledge graph and human feedback to teach robots"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system's reported success rates depend on humans giving correct answers when asked and on the evaluators of the executions being impartial, so the framework alone cannot reproduce those numbers if the human oracle is wrong or the scoring is biased.","fun_headline_variants_meta":{"raw":{"variants":["LLM + knowledge graph + human input refines robot task plans","Robots adapt to new tasks using LLM, knowledge graphs, and human guidance","Knowledge repair boosts robot task success via LLM, KG, and human input","AdaptBot: LLM meets knowledge graph and human feedback to teach robots"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000806,"raw_usage":{"total_tokens":3548,"prompt_tokens":961,"completion_tokens":2587,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":2506}},"tokens_in":577,"tokens_out":2587,"duration_ms":21405,"temperature":1.0,"reasoning_tokens":2506,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T13:27:25.483176+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full framework on the same 30 cooking tasks but replace the human oracle with a random-answer script and score the executions with evaluators who do not know which configuration produced each trace; if success rates stay near 91–92%, human accuracy is not load-bearing, whereas a collapse toward the LLM-only baseline would confirm that the reported gains rely on correct human input.","supporting_citations":[{"cited_title":"From cooking recipes to robot task trees– improving planning correctness and task efficiency by leveraging llms with a knowledge network,","cited_arxiv_id":null,"evidence_quote":"Supplies the prior task-tree method and the progress-line evaluation that this framework is compared against."},{"cited_title":"Functional object-oriented network for manipulation learning,","cited_arxiv_id":null,"evidence_quote":"Provides the functional object-oriented knowledge network that motivates the KG-based representation of cooking knowledge."},{"cited_title":"Recipe1m+: A dataset for learning cross-modal embeddings for cooking recipes and food images,","cited_arxiv_id":null,"evidence_quote":"Provides the recipe dataset from which the 30 cooking tasks are sampled."},{"cited_title":"Approximate task tree retrieval in a knowledge network for robotic cooking,","cited_arxiv_id":null,"evidence_quote":"Represents prior work that retrieves task trees from a knowledge network for cooking, a direct baseline approach."},{"cited_title":"Reflexion: Language agents with verbal reinforce- ment learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the reflexive human-feedback loop that informs the human-in-the-loop component."}],"review_version":1}