{"id":"f1839ff1-c389-4ca6-b92d-28307a8c921d","arxiv_id":"2506.07454","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A multi-robot team fuses open-set object maps and 3D scene graphs, translates natural-language commands into PDDL goals with an LLM, and executes them in a large outdoor environment.","lead":"A team of robots built a shared 3D map, understood plain-English commands, and worked together to carry out outdoor inspection tasks. The paper shows a practical way to connect object-based mapping, scene graphs, and large language models for language-directed multi-robot operations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Language grounding is the load-bearing weakness: offline spatial/region disambiguation scores are 20-50%, yet end-to-end Hard commands report 100% grounding without reconciliation.","rationale":"The load-bearing requirement for the central claim is that the LLM reliably maps operator commands to PDDL goals matching user intent. Section 4 and Fig. 2 describe the prompt, but Section 5.2's Table 4 shows this is the weakest link: no model exceeds 4/10 on spatial-relation disambiguation and 5/10 on region-level disambiguation, and Section 5.1 documents a grounding failure on 'checkout two nearby boxes'. These are exactly the linguistic constructions used in the Hard end-to-end commands ('check out the two boxes in the shelter and the closest trash'). The 100% grounding success reported for multi-robot Hard trials in Table 1 is therefore in tension with the offline results. If one goal predicate is wrong, the planner optimizes the wrong objective and execution success is irrelevant to user intent. The paper gives no reconciliation: no trial counts per cell, no confidence intervals, no indication whether the end-to-end instructions were selected to avoid the hard categories, and no released artifact to inspect. Other concerns, such as 64-65% relocalization success, are real but secondary: relocalization failures can be retried or may still allow coarse navigation, and execution rates already reflect such failures, whereas a wrong goal is unrecoverable. I therefore identify LLM grounding as the single most load-bearing concern. The system is an impressive integration and the paper honestly reports its failures, but the evidence does not support the 'reliably ground' claim as stated. This matches the reader's weakest assumption, so no verdict change is needed relative to the CONDITIONAL verdict.","tokens_in":9934,"tokens_out":5906,"duration_ms":66530,"concrete_test":"Reproduce Table 4 using the exact prompt template, scene graph serialization, and LLM configuration (model, temperature) used in the Sec. 5.1 end-to-end experiments, on the 70-instruction dataset; additionally, run 50 new Hard-style instructions containing spatial and region disambiguation. If SR-D and RL-D grounding accuracy remains below 50%, the end-to-end 100% grounding result is not representative, and the reliability claim should be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the system 'reliably ground[s] and execute[s] complex instructions' (Sec. 5.1) depends on the LLM translating natural language into correct PDDL goals (Sec. 4, Fig. 2). The offline grounding study in Table 4 shows this step fails on the very categories present in the claimed Hard commands: spatial-relation disambiguation (best 4/10, GPT-4.1) and region-level disambiguation (5/10, GPT-4.1). Section 5.1 also reports an end-to-end failure where 'checkout two nearby boxes' was grounded as movement rather than inspection. Yet Table 1 reports 100% grounding success for multi-robot Hard trials. The paper does not explain how the end-to-end commands avoid the Table 4 failure modes, how many Hard trials were run, or whether the same prompt and LLM configuration were used. Because a wrong goal predicate invalidates the downstream plan and execution, the reliability claim is not supported by the reported evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an integrated multi-robot system that combines 3D scene graph mapping (Hydra/Hydra-Multi), open-set object-based relocalization (ROMAN), LLM-based translation of natural-language instructions into PDDL goals, and hierarchical task and motion planning. The system is evaluated in large-scale outdoor experiments at Camp Buckner with a Clearpath Husky and a Boston Dynamics Spot, covering scene graph fusion, view-invariant relocalization, and end-to-end execution of instructions at three difficulty levels. Ablations on the Kimera-Multi dataset characterize fusion accuracy, and offline language-grounding experiments test seven linguistic categories across several LLMs. The central claim is that the system can 'reliably ground and execute complex instructions' expressed in natural language.","tokens_in":10062,"tokens_out":5131,"duration_ms":56557,"significance":"If the central claims hold, this is a noteworthy systems integration demonstration: it brings together state-of-the-art components (Hydra-Multi, ROMAN, scene-graph-based TAMP, LLM grounding) and evaluates them on real robots in a large-scale outdoor environment. The paper's honest reporting of failures (e.g., the grounding error for 'checkout two nearby boxes', the 64-65% relocalization success) is a strength, as is the inclusion of offline ablations that probe component behavior beyond the small end-to-end trial count. However, the evidence is thin for the stated reliability claim: the end-to-end trials are few (8 single-robot, 12 multi-robot total, with no per-condition trial counts), the offline grounding ablations show weak performance on exactly the categories appearing in the Hard commands, and the relocalization success rate is only about two-thirds. The integration value is clear, but the quantitative support for the central claims needs substantial strengthening.","major_comments":[{"comment":"The paper does not reconcile the near-perfect grounding success in the end-to-end experiments with the low offline grounding scores on the same linguistic phenomena. Table 4 reports that the best model, GPT-4.1, achieves only 4/10 on spatial-relation disambiguation and 5/10 on region-level disambiguation, and GPT-4o scores 2/10 and 4/10 on these categories. Yet Table 1 reports 100% grounding success for Multi-Robot trials at all difficulty levels, including Hard commands such as 'Spot, check out the two boxes in the shelter and the closest trash, then head over to the pole', which require spatial-relation and region-level disambiguation. The paper also reports an end-to-end failure in Section 5.1 where 'checkout two nearby boxes' was grounded as a movement predicate rather than an inspection predicate. The authors should explain how the end-to-end Hard commands avoid the failure modes identified in Table 4, specify which LLM and prompt configuration were used in the end-to-end trials, and report the number of Hard trials per condition. Without this reconciliation, the claim that complex instructions are 'reliably ground[ed]' (abstract, Section 5.1) is not supported by the evidence.","section":"Section 5.1, Table 1 vs. Section 5.2, Table 4"},{"comment":"The trial counts are too small to support the reliability claim. Table 1 reports aggregate success rates over 8 single-robot and 12 multi-robot trials, but does not state how many trials were run per difficulty level and per stage. With three difficulty levels and three stages, each percentage likely corresponds to only 1-4 trials, so a single failure changes the rate by 25-50%. No confidence intervals or error bars are provided. The claim that the system 'reliably ground[s] and execute[s] complex instructions' requires either a larger trial count, per-condition trial numbers, or a more modest interpretation of the success rates, especially given that the single-robot Hard execution success is 50%.","section":"Section 5.1, Table 1"},{"comment":"The text states that 'These metrics validate the improved accuracy of ROMAN when integrated with Hydra-Multi as compared to traditional visual place recognition' (Section 5.2). This is not directly supported by the numbers in Table 2: ROMAN has a higher ATE RMSE than SALAD on both the Hybrid setting (10.36 m vs. 9.40 m) and the Outdoor setting (6.61 m vs. 5.67 m). The improvements in object IoU, precision, and recall are modest (e.g., 0.40 vs. 0.36 IoU on Hybrid) and are not statistically characterized. The paper should either report significance or reframe the result as a trade-off: ROMAN yields slightly worse trajectory accuracy but somewhat better scene-graph object consistency, while using drastically less memory. The current wording overstates the conclusion.","section":"Section 5.2, Table 2"},{"comment":"The relocalization success rates of 64% (Husky) and 65% (Spot) are reported as evidence for 'real-time, view-invariant relocalization'. While the view-invariance is interesting, a one-third failure rate means that the system frequently cannot localize, and the paper does not state how these failures are handled during task execution (e.g., retries, re-planning, or abort). The claim that this representation 'supports real-time, view-invariant relocalization' is only partially supported. The authors should report the number of relocalization trials, the failure modes (e.g., object sparsity, depth range), and the operational impact of a 64-65% success rate on the end-to-end execution results.","section":"Section 5.1, Position Estimation and Relocalization"}],"minor_comments":[{"comment":"There is a typo: 'occured' should be 'occurred'. Also, 'theacl jackal2' appears to be a formatting error for 'the acl_jackal2' (or similar robot name).","section":"Section 5.2, first paragraph"},{"comment":"The language-to-PDDL approach is described as an adaptation of AutoTAMP [13], but the novelty relative to [13] is not made precise. The paper should clarify what is new in the prompt structure, the scene-graph context, or the multi-robot goal assignment, so that the contribution is not conflated with the prior work.","section":"Section 4, first paragraph"},{"comment":"The definition of a 'correct plan' is not specified. It would help to state whether the plan is compared against a hand-authored ground-truth plan, whether any plan satisfying the PDDL goal counts, and how execution correctness is judged (e.g., reaching a goal region, visual verification). A precise rubric would make the success rates more interpretable.","section":"Table 1"},{"comment":"Several references are incomplete, e.g., reference [36] is missing volume, issue, and page numbers, and reference [29] may be a preprint rather than a published venue. Please verify all entries.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a solid systems integration paper from a strong group, and the hardware deployment is a genuine contribution. My main concern is that the central reliability claim is not supported by the reported quantitative evidence, particularly the discrepancy between the offline grounding ablations (Table 4) and the end-to-end grounding success (Table 1), and the small trial counts. These are fixable with more careful reporting and more cautious claims, so I recommend major revision rather than rejection. The paper may be a better fit for a systems-oriented venue than a purely algorithmic one, but it should still meet the quantitative bar for the claims it makes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a real systems integration: Hydra scene graphs, ROMAN open-set object maps, an existing scene-graph TAMP planner, and LLM-to-PDDL grounding, all run end-to-end on a Husky and a Spot in an outdoor environment. That integration is the contribution, and it goes beyond prior work from this group, which covered the pieces separately. The fusion ablation on Kimera-Multi is a fair comparison against a vision-based baseline, and the view-invariant relocalization test (side vs. forward cameras) is genuinely informative. They also report failures without hiding them, including the \"checkout two nearby boxes\" grounding error.\n\nThe soft spots are concentrated in the reliability claim. Trial counts are tiny: 8 and 12, no error bars. Relocalization succeeds only 64-65% of the time, which is below what \"reliably\" usually means. The offline grounding study in Table 4 shows the LLM fails on exactly the categories that appear in the claimed Hard commands: spatial-relation disambiguation scores 2-4/10, region-level 4-5/10. Yet Table 1 reports 100% grounding on multi-robot Hard trials. The paper never reconciles this. The one reported end-to-end grounding failure also looks like a spatial-relation/region problem (\"two nearby boxes\" grounded as movement). So the stress-test concern is fair: either the Hard commands avoid the Table 4 failure modes and the paper should say so, or the 100% figure is a small-sample artifact. Either way, the central claim overstretches the evidence.\n\nNone of that makes the paper worthless. As a demonstration that the components can be made to work outdoors with real robots, it succeeds. As a claim about reliable language-grounded multi-robot execution, it is not yet supported. Worth citing for the integration and as a baseline, but not for the numbers.\n\nMy recommendation is to send it to peer review. It is a legitimate systems contribution, and referees can push for the reconciliation, more trials, and an artifact release. Expect major revision on the evaluation section.","headline":"Genuine multi-robot outdoor integration, but the 'reliably ground' claim is not backed by the paper's own grounding ablations.","tokens_in":10654,"tokens_out":3463,"would_cite":true,"duration_ms":38451,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A team of robots can translate plain-English commands into formal plans and execute them outdoors using a single shared 3D scene graph.","keywords":["multi-robot systems","3D scene graphs","task and motion planning","natural language grounding","large language models","open-set object map","view-invariant relocalization","PDDL"],"falsifier":"Give the system the instruction 'Spot, check out the two boxes near the pole' in a fresh outdoor scene and observe the generated PDDL goal; if the goal is a movement to the boxes rather than an inspection, the grounding step has reproduced the documented failure mode, falsifying the claim of reliable language grounding for that class of commands.","tokens_in":9704,"feed_emoji":"🤖","tokens_out":6578,"duration_ms":66246,"temperature":0.7,"pith_summary":"This paper claims that a team of robots can build one shared 3D scene graph, relocalize within it from very different viewpoints, and use a large language model to turn plain-English operator commands into formal plans that the robots then carry out in large outdoor spaces. The key move is a single representation that serves both perception and planning: an open-set object-based map tied to a hierarchical scene graph with objects, places, and regions. If the claim holds, a human could task heterogeneous robots with inspection and navigation jobs by typing or speaking a sentence, with no programming. The authors test the claim on two real robots at Camp Buckner and report success rates for grounding, planning, and execution, along with ablations that show where the weakest link sits.","feed_headline":"Robots turn plain-English orders into outdoor missions","feed_subtitle":"A shared 3D scene graph lets two robots ground language, plan, and execute real inspection tasks.","key_machinery":"The central object is the fused 3D scene graph, a hierarchical structure whose nodes are objects (each with a CLIP embedding and a shape descriptor), places (navigable 2D regions computed by a Voronoi-like partition of the mesh), and regions (semantically consistent areas such as road or sidewalk). The object layer is produced by ROMAN, an open-set object map that enables loop-closure detection and view-invariant relocalization; the places layer supports motion planning; and the full graph, serialized to text, defines the PDDL domain. The planning pipeline is PDDLStream, and the language-to-goal step is an LLM prompted in-context with the scene-graph text, robot capabilities, and examples.","core_discovery":"The central discovery is that a fused 3D scene graph whose object layer is an open-set object map can simultaneously support view-invariant relocalization and language-grounded task and motion planning. The system builds per-robot scene graphs and object maps, fuses them through object-based loop closures into one shared scene graph, and then treats that graph as a PDDL planning domain. A large language model, prompted with a text serialization of the scene graph, robot capability descriptions, and in-context examples, translates natural-language instructions into per-robot PDDL goals. PDDLStream searches for a feasible task skeleton with continuous motion parameters, and each robot relocalizes against the fused object map before executing its part of the plan. In experiments, two heterogeneous robots executed easy, medium, and hard instructions outdoors, with the authors reporting a failure mode where an instruction to 'check out two nearby boxes' was grounded as movement rather than inspection.","pith_inferences":["If the LLM grounding step is the bottleneck, replacing it with a hybrid resolver that calls a spatial-relation program or a scene-graph query engine for the ambiguous parts could raise end-to-end reliability without changing the rest of the pipeline.","Since relocalization degrades where objects are sparse, we expect the approach to struggle in open fields and long corridor-like stretches; merging object cues with conventional geometric place recognition would be a natural test.","The open-set object map is used only to localize and to supply object names; we infer that the same CLIP embeddings could let operators refer to objects never seen during mapping, enabling genuinely open-ended commands.","A stale scene graph would silently break tasks that reference changed or moved objects; online scene-graph updating is an implied requirement before the system can operate in non-static environments."],"forward_implications":["A single 3D scene graph can serve as both the metric-semantic map for localization and the symbolic domain for planning, so the system avoids maintaining separate representations for perception and task planning.","Object-based loop closures let multiple robots fuse maps with a far smaller database than visual place recognition, while improving object IoU, precision, and recall in the fused scene graph on the tested datasets.","The language-to-PDDL translation is accurate for direct references and unambiguous commands, but spatial-relation and region-level disambiguation remain unreliable, which bounds the kinds of commands the system can safely execute.","The system achieves end-to-end execution on a wheeled robot and a quadruped with different sensors and odometry, so the approach transfers across heterogeneous platforms."],"supporting_citations":[{"why":"Builds the per-robot 3D scene graphs whose objects, places, and regions define the planning domain.","marker":"[1]"},{"why":"Provides the open-set object map used for loop-closure detection, fusion, and view-invariant relocalization.","marker":"[4]"},{"why":"Supplies the Hydra-Multi method that fuses single-robot scene graphs into the shared graph.","marker":"[2]"},{"why":"Provides the hierarchical scene-graph planner that solves the PDDL problem and outputs executable actions.","marker":"[5]"},{"why":"PDDLStream enables the joint search over task skeletons and continuous motion parameters.","marker":"[12]"},{"why":"Supplies the neuro-symbolic inference approach adapted for translating natural language into PDDL goals.","marker":"[13]"},{"why":"CLIP embeddings give objects open-set semantic descriptors used in object matching and language grounding.","marker":"[11]"},{"why":"Offers the visual-place-recognition baseline (SALAD) against which ROMAN is compared in the fusion ablation.","marker":"[15]"}],"fun_headline_variants":["Robots plan outdoor missions from plain English via 3D scene graphs","LLM turns natural language into robot plans on 3D scene graphs","Multi-robot team executes language commands using shared 3D maps","Robot pair grounds natural language in 3D scene graph planning","Natural language to multi-robot plans via shared 3D scene graphs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline rests on the assumption that a large language model will translate an operator's natural-language instruction into the correct PDDL goal from a text serialization of the scene graph, an assumption the paper's own grounding results show failing for spatial-relation and region-level commands.","fun_headline_variants_meta":{"raw":{"variants":["Robots plan outdoor missions from plain English via 3D scene graphs","LLM turns natural language into robot plans on 3D scene graphs","Multi-robot team executes language commands using shared 3D maps","Robot pair grounds natural language in 3D scene graph planning","Natural language to multi-robot plans via shared 3D scene graphs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000705,"raw_usage":{"total_tokens":3162,"prompt_tokens":909,"completion_tokens":2253,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":2160}},"tokens_in":525,"tokens_out":2253,"duration_ms":15423,"temperature":1.0,"reasoning_tokens":2160,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:32:58.821249+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the system the instruction 'Spot, check out the two boxes near the pole' in a fresh outdoor scene and observe the generated PDDL goal; if the goal is a movement to the boxes rather than an inspection, the grounding step has reproduced the documented failure mode, falsifying the claim of reliable language grounding for that class of commands.","supporting_citations":[{"cited_title":"Foundations of spatial perception for robotics: Hierarchical representations and real-time systems,","cited_arxiv_id":null,"evidence_quote":"Builds the per-robot 3D scene graphs whose objects, places, and regions define the planning domain."},{"cited_title":"Roman: Open-set object map alignment for robust view-invariant global localization,","cited_arxiv_id":null,"evidence_quote":"Provides the open-set object map used for loop-closure detection, fusion, and view-invariant relocalization."},{"cited_title":"Hydra-Multi: Collaborative online construction of 3D scene graphs with multi-robot teams,","cited_arxiv_id":null,"evidence_quote":"Supplies the Hydra-Multi method that fuses single-robot scene graphs into the shared graph."},{"cited_title":"Task and motion planning in hierarchical 3D scene graphs,","cited_arxiv_id":null,"evidence_quote":"Provides the hierarchical scene-graph planner that solves the PDDL problem and outputs executable actions."},{"cited_title":"PDDLStream: Integrat- ing symbolic planners and blackbox samplers via optimistic adaptive planning,","cited_arxiv_id":null,"evidence_quote":"PDDLStream enables the joint search over task skeletons and continuous motion parameters."},{"cited_title":"AutoTAMP: Au- toregressive task and motion planning with LLMs as translators and checkers,","cited_arxiv_id":null,"evidence_quote":"Supplies the neuro-symbolic inference approach adapted for translating natural language into PDDL goals."},{"cited_title":"Learning transferable visual models from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"CLIP embeddings give objects open-set semantic descriptors used in object matching and language grounding."},{"cited_title":"Optimal transport aggregation for visual place recog- nition,","cited_arxiv_id":null,"evidence_quote":"Offers the visual-place-recognition baseline (SALAD) against which ROMAN is compared in the fusion ablation."}],"review_version":1}