{"id":"01839980-d852-48cb-aa02-743b96a3047c","arxiv_id":"2506.20394","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SPARK updates a 3D scene graph online from verbal, text, and gesture cues, and replans robot fetch tasks based on the updated graph.","lead":"SPARK is a robot system that updates a 3D semantic scene graph online from spoken, written, and gestured clues, then replans fetch tasks such as finding an apple on a cleaning table. The report shows three successful demonstrator runs on a Toyota HSR robot, but includes no quantitative metrics or comparison baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The graph-specific benefit is confounded: every experiment updates the scene graph and the plan together, with no control where the same extracted hint is given directly to the planner, so the central enhancement claim is not yet evidenced.","rationale":"The reader's weakest assumption is exactly the load-bearing concern: graph update versus direct text injection. I agree. The paper is a plausible system paper, but its central claim is an empirical comparative statement. Since all three experiments vary the graph and the hint together, the success cannot be attributed to the graph. The only evidence of graph-specific value would be a task where the graph structure itself beyond a single relation is used, such as inference through multiple relations, query answering, or relation consistency, or a control showing degraded performance without the graph. The proposed ablation is straightforward and would settle the issue. In the meantime, the appropriate verdict is conditional acceptance, not rejection, because the system is novel, runs on a real robot, and the components are coherent; the paper should be revised to include the control and metrics. This does not change the reader's verdict, so I recommend UNCHANGED.","tokens_in":3256,"tokens_out":4810,"duration_ms":53576,"concrete_test":"Run a controlled ablation in the same simulated household environment. Condition A is full SPARK: the extracted relation (e.g., apple-on-cleaning-table) is written into the scene graph and the graph-based plan is regenerated. Condition B is the direct-hint baseline: the same extracted relation is appended verbatim to the planner prompt, and no scene graph update is performed; all detection, speech, and gesture modules remain identical. Run at least 20 trials per condition per input modality, randomizing target object and location, and record task success, planning time, and number of replans. If Condition B achieves statistically indistinguishable success, the central 'graph enhancement' claim is unsupported. If Condition A is meaningfully better, especially in scenes where the relation must be reused for multiple decisions, the claim gains support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"SPARK's central claim is that graph representations of spatial relationships enhance task performance (Abstract, Sections II-B and IV). The supporting evidence consists of three single-run demonstrations (Figures 2–4) in which the robot hears speech, reads text, or sees a pointing gesture; SPARK extracts a semantic relation such as 'apple is on the cleaning table' and updates the scene graph; then the planner makes a new plan. In each case, the graph update and the informational content of the cue are varied at the same time. Nothing in the paper isolates the graph update as the cause of success. The same extracted triple could be appended to the LLM planner's prompt as an additional instruction ('the apple is on the cleaning table'), yielding the same replanning behavior without any graph representation. The description in Section II-B says the LLM is queried to decide whether to update the map and/or plan; it does not explain why updating the graph, rather than passing the extracted relation to the planner, is necessary for the demonstrated behavior. The experiments also report no repeated runs, no success-rate metric, no timing data, and no comparison to a non-graph system, so the claim of enhancement is at present a demonstration of an integrated pipeline, not evidence that the graph is the operative component. The gesture experiment is especially weak: the only stated difficulty is that the teddy bear location is abstract, and the baseline alternative of a direct location hint is again not tested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SPARK, a framework that lets a service robot acquire semantic information from speech, visual text, and gestures during task execution, update an online scene graph, and replan accordingly. The system builds on ConceptGraphs and Self-Recovery Prompting, using Whisper, GPT-4, Detic, and CLIP. The experiments are three single-run demonstrations on a Toyota HSR in a household-like environment, where the robot changes its target location after receiving a cue. The abstract and conclusion claim that graph representations of spatial relationships enhance task performance in dynamic environments.","tokens_in":3682,"tokens_out":3873,"duration_ms":44374,"significance":"If the central claim were established, the paper would address a real gap: most scene graph systems are offline, and online semantic updates from non-geometric cues are indeed underexplored. The authors demonstrate an end-to-end pipeline on real hardware with open-vocabulary perception, which is a useful feasibility result. However, the experiments do not currently support the enhancement claim: there are no baselines, no repeated trials, no success metrics, and the graph update is never isolated from the simultaneous prompt change. The contribution at this stage is a system demonstration rather than a validated improvement.","major_comments":[{"comment":"The central claim that graph representations enhance task ability is confounded. In all three experiments, the robot receives a cue, updates the scene graph, and replans in a single pipeline, so the graph update and the informational content of the cue are varied together. No control condition passes the extracted semantic relation (e.g., \"apple is on the cleaning table\") directly to the planner without updating the graph. Without this control, the observed behavior could be due entirely to adding the new location to the planner's prompt, not to the graph representation. This is a load-bearing missing experiment for the paper's main claim.","section":"Section III and Abstract"},{"comment":"There is no quantitative evaluation. The experiments are anecdotal single runs with no number of trials, success rate, error bar, timing data, or failure cases. The sentence in Section III-A that the robot \"successfully completing the command\" is not backed by any recorded metric. To support a claim of enhanced ability in dynamic environments, the authors need repeated runs and a predefined success criterion, ideally across multiple task instances.","section":"Section III-A, III-B, III-C"},{"comment":"The evaluation is self-referential. Section II-B says the LLM is queried to decide whether to update the map and/or plan, and the same LLM performs task planning; success is assessed qualitatively by the authors. There is no stated protocol for judging success independently of the planner's own output, and no check for whether the updated graph was actually used by the planner. This makes it hard to distinguish genuine integration from the LLM coincidentally producing the desired plan.","section":"Section II-B and Section III"},{"comment":"The graph update mechanism is underspecified. The paper does not describe how nodes and edges are added or removed, how conflicting semantic information is resolved, whether old edges are retained or overwritten, or how the updated graph is represented in the prompt to the planner. Since the paper's contribution is the online scene graph update, these details are central to reproducibility and to evaluating whether the graph is truly load-bearing.","section":"Section II-B"}],"minor_comments":[{"comment":"The heading \"V erbal Input\" contains a stray space; it should be \"Verbal Input\".","section":"Section III-A heading"},{"comment":"Figure 1 is referenced but not explained in the text; the pipeline for gesture input in particular is hard to follow from the figure alone. Please add a caption or text description that walks through each modality.","section":"Figure 1"},{"comment":"The phrase \"other settings were nearly identical\" is vague; please state exactly which settings differed between the verbal and written text experiments.","section":"Section III-B"},{"comment":"The paper should explicitly state what SPARK adds over the authors' prior Self-Recovery Prompting work and over ConceptGraphs; currently the novelty is only implicit.","section":"Introduction and Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper is short and currently reads as a workshop-style demonstration. The central claim is unsupported by the evidence as presented, but the missing control and metrics are obtainable within the manuscript's scope, so major revision is appropriate. I would also ask the authors to be explicit about the incremental contribution over their prior works, particularly Self-Recovery Prompting and ConceptGraphs."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a plausible integration paper with a genuinely useful framing—online semantic scene-graph updates from speech, text, and gestures—but the central comparative claim is not backed by the experiments. Worth a referee, but it needs a control condition and real metrics before the enhancement claim can stand.\n\nWhat's new: prior work like ConceptGraphs builds scene graphs offline; SPARK updates the graph mid-task from non-geometric cues. That is a real gap, and the modular prompt-based design (Whisper, GPT-4, Detic, CLIP) is sensible and reproducible in principle. The three demos show the pipeline working end-to-end on an HSR, which is non-trivial.\n\nWhere it's soft: the paper says graph representations 'enhance' task ability, but every experiment changes both the graph and the planner's information at once. There is no control where the extracted triple ('apple on cleaning table') is handed directly to the LLM planner as text, without a graph update. That is the exact control needed to isolate the graph's contribution, and without it the enhancement claim collapses to a demonstration of an integrated system. Also missing: repeated runs, success rates, timing, failure cases, and any baseline against the original system or a non-graph planner. The gesture example is illustrative but the 'abstract indication' difficulty is never quantified, so it's hard to see what problem the graph specifically solves.\n\nCitation pattern looks fine; the authors cite prior work including their own ConceptGraphs and Self-Recovery Prompting, which is appropriate given they build directly on those. No sign of fitting or circular derivation.\n\nBottom line: the idea is worth engaging. As it stands, the paper is a promising system description, not a validated claim. If I were editing, I'd send it to review, but the reviewers should insist on a baseline comparison and quantitative results. I would not cite it yet as evidence for graph-based enhancement.\n\nThe stress-test note is fair: the confound is real and the absence of a control is the load-bearing weakness. The note's suggested control is exactly what's missing.","headline":"Plausible integration system, but the graph-specific enhancement claim is untested for lack of a control condition.","tokens_in":4048,"tokens_out":1623,"would_cite":false,"duration_ms":16947,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SPARK lets a service robot take verbal, written, or gestured hints mid-task and turn them into scene graph updates that drive replanning.","keywords":["online scene graph","semantic mapping","robot task planning","service robot","foundation models","gesture recognition","spatial semantics","SPARK"],"falsifier":"Run the three scenarios (verbal, written, gesture) with a control condition in which the same LLM receives the same semantic hint as a plain text instruction and the scene graph is left unchanged; if the control succeeds equally often, the graph-update mechanism is not the source of the improvement. A second check: update the graph but suppress plan regeneration; success would show the graph alone, not replanning, matters.","tokens_in":3103,"feed_emoji":"🤖","tokens_out":4476,"duration_ms":45452,"temperature":0.7,"pith_summary":"SPARK is a framework for updating a robot's semantic scene graph while the robot is executing a task. It takes verbal statements, written signs, or pointing gestures, uses an LLM to decide whether the information is relevant, writes the new spatial relation into the graph, and regenerates the task plan if needed. The paper shows this on a physical service robot in a household-like setting: in each scenario the robot initially plans for the wrong location, then receives a cue and successfully fetches the object. The paper's central claim is that keeping this semantic information in a graph structure, rather than leaving it as a one-off instruction, is what lets the robot adapt to dynamic and unconventional cues.","feed_headline":"Scene graph updates let robots replan from new cues mid-task","feed_subtitle":"SPARK turns a spoken, written, or gestured hint into a scene graph change and a new fetch plan.","key_machinery":"The central object is the online scene graph: a graph whose nodes are objects and rooms and whose edges are spatial relations (\"apple on cleaning table\"), updated during task execution rather than built once offline. SPARK's mechanism is a promptable loop: Whisper transcribes speech, Detic and CLIP detect and recognize objects in visual input, and GPT-4 decides whether the new semantic information is relevant, updates the graph edges, and—when the update changes the goal location—regenerates the plan. The graph is what carries the new relation into subsequent planning steps, so the planner sees the world the way the cue described it rather than as the original static map did.","core_discovery":"On the paper's own terms, SPARK's discovery is that environment-embedded semantic cues — speech, written text, gestures — can be converted into online scene graph updates during execution and that those updates, drawn from the graph at plan time, let a general-purpose service robot complete fetch tasks it would otherwise fail or slow down on. The paper implements this by adapting ConceptGraphs to an online setting and layering a promptable LLM-based decision module on top: the LLM transcribes and interprets the cue, decides whether the scene graph and plan should change, and the robot re-plans with the revised graph. Three real-robot experiments, one per cue type, support the claim that graph representations of spatial relationships improve task performance in dynamic environments.","pith_inferences":["Inference: the paper does not run the control where the same location hint is given to the LLM as plain text without a graph update, so the distinctive contribution—the graph structure—remains unisolated.","Inference: a natural extension is to assign confidence or reliability scores to cues (speech vs. gestures are not equally trustworthy), letting the graph keep multiple candidate locations and the planner weigh them.","Inference: because the update mechanism is promptable and graph-based, it should transfer to other semantic relations beyond locations, such as \"the cup is fragile\" or \"the door is locked,\" as long as those relations are expressible as graph edges.","Inference: the gesture experiment points toward non-verbal human-robot communication; a stronger test would place the target where the gesture alone can identify it, without any accompanying text."],"forward_implications":["A service robot can finish a fetch task when the target object has been moved to an atypical place, because a spoken hint during execution updates the plan.","Written signs and pointing gestures work through the same graph-update path, so semantic integration does not depend on speech recognition.","The robot changes its goal mid-execution rather than restarting the whole mapping or planning pipeline, since only the relevant graph edges are revised.","Because every component is promptable, new objects, rooms, and cue types can be added without retraining the perception or planning models."],"supporting_citations":[{"why":"Supplies the open-vocabulary 3D scene graph representation that SPARK adapts for online, in-execution updates.","marker":"[2]"},{"why":"The prior self-recovery system that plans from LLM prompts but does not simultaneously update geometric and semantic information; SPARK extends it.","marker":"[3]"},{"why":"Establishes the 3D-LLM approach to spatial task planning that ignores non-geometric cues, the gap SPARK fills.","marker":"[1]"},{"why":"The LLM used for task planning and for deciding whether a cue should update the graph and the plan.","marker":"[5]"},{"why":"Transcribes verbal cues into text for the graph-update decision.","marker":"[4]"},{"why":"Detects and segments objects for visual cue grounding in the scene graph.","marker":"[6]"},{"why":"Classifies objects for visual cue grounding and graph node labeling.","marker":"[7]"}],"fun_headline_variants":["SPARK turns hints into scene graph updates for smarter replanning","Robot re-plans from speech, text, and gestures via live scene graphs","Online scene graphs let robots adapt to new cues mid-task","SPARK: live scene graph updates from semantic cues for task planning","Semantic cues become graph edits that drive robot replanning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim assumes that updating the robot's scene graph is what improves performance, rather than simply giving the same location hint to the planner as a direct text prompt.","fun_headline_variants_meta":{"raw":{"variants":["SPARK turns hints into scene graph updates for smarter replanning","Robot re-plans from speech, text, and gestures via live scene graphs","Online scene graphs let robots adapt to new cues mid-task","SPARK: live scene graph updates from semantic cues for task planning","Semantic cues become graph edits that drive robot replanning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000627,"raw_usage":{"total_tokens":2840,"prompt_tokens":827,"completion_tokens":2013,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":443,"completion_tokens_details":{"reasoning_tokens":1923}},"tokens_in":443,"tokens_out":2013,"duration_ms":14513,"temperature":1.0,"reasoning_tokens":1923,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:48:07.537255+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the three scenarios (verbal, written, gesture) with a control condition in which the same LLM receives the same semantic hint as a plain text instruction and the scene graph is left unchanged; if the control succeeds equally often, the graph-update mechanism is not the source of the improvement. A second check: update the graph but suppress plan regeneration; success would show the graph alone, not replanning, matters.","supporting_citations":[{"cited_title":"Self-Recovery Prompting: Promptable General Purpose Service Robot System with Foundation Models and Self-Recovery","cited_arxiv_id":"2309.14425","evidence_quote":"The prior self-recovery system that plans from LLM prompts but does not simultaneously update geometric and semantic information; SPARK extends it."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the 3D-LLM approach to spatial task planning that ignores non-geometric cues, the gap SPARK fills."},{"cited_title":"Radford, J","cited_arxiv_id":null,"evidence_quote":"Transcribes verbal cues into text for the graph-update decision."},{"cited_title":"a henb \\","cited_arxiv_id":null,"evidence_quote":"Detects and segments objects for visual cue grounding in the scene graph."}],"review_version":1}