{"id":"4aa4bf89-22c8-4d69-9718-30b00a9b0f21","arxiv_id":"2411.09022","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A dependency-aware task planner that uses LLMs and a DAG to decompose and execute natural-language construction tasks on multiple mobile robots.","lead":"DART-LLM uses a large language model to break a single spoken instruction into smaller chores for a team of construction robots, and it records which chores depend on which others before the robots start moving. The system is tested on 102 construction instructions in simulation and on real robots, where it reports higher success rates than a previous LLM-based planner.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ablation behind the central claim is confounded: if the 'Without Dependency' arm removes dependency lists, Algorithm 1's default parallel scheduling alone can explain the success-rate drop, independent of LLM reasoning.","rationale":"The reader's weakest_assumption concerns the closed skill menu and expressivity beyond the benchmark—a valid generalization caveat. My concern is more direct: the internal ablation intended to establish the central mechanism is confounded by the executor's scheduling policy. The paper's own Algorithm 1 makes parallel execution the default for dependency-free subtasks, so removing the DAG from the prompt while keeping that executor would degrade success for purely mechanical reasons. Without a description of the control arm's scheduling rule, the headline mechanism claim is underdetermined. I credit the paper for a real hardware deployment, a reasonable system architecture, and a clean JSON interface, but those do not resolve the confound. The current CONDITIONAL verdict remains appropriate: acceptance should require the authors to disclose the control-arm implementation and rerun the ablation with a fair ordering-preserving baseline. I did not select REJECT because the system may still work as described and the SMART-LLM comparison, while weak, is not the only evidence; the central flaw is the unverified attribution of the ablation gap to dependency reasoning.","tokens_in":11620,"tokens_out":5835,"duration_ms":67556,"concrete_test":"Obtain the ablation prompts and executor code from the project website and rerun the 'Without Dependency' condition under two executor policies: (a) the Algorithm-1 default, where empty dependency lists cause parallel execution, and (b) a policy that executes the LLM's returned subtask list in its emitted order while ignoring dependency edges. If SR under policy (b) is close to the 'With Dependency' SR, the reported DAG benefit is a scheduling artifact; if SR under policy (b) drops to the reported no-DAG level, the benefit is attributable to explicit dependency reasoning. Report per-task counts, repeated runs, and variance for L3, and publish the no-DAG prompt template alongside the code.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's core claim—'explicit dependency modeling via DAG can effectively compensate for the limited reasoning capabilities of smaller models'—rests on the ablation in Fig. 3, run only on L3 tasks. The paper never specifies how the 'Without Dependency' condition schedules subtasks. In Algorithm 1 (lines 11-18), any subtask with no dependencies in D is executed in parallel; only subtasks with non-empty dependency lists wait. If the 'Without Dependency' arm simply omits dependency lists from the JSON prompt, then every subtask has an empty dependency list and the executor launches all subtasks concurrently. Since L3 tasks are defined by strict execution order, this control would guarantee failure on sequential portions regardless of the LLM's decomposition quality. The observed gap (e.g., Llama3.1 0.84→0.45) would then be an artifact of the changed executor policy, not evidence that DAG prompting improves model reasoning. Conversely, if the no-DAG arm preserves ordering through list order or some other mechanism, that mechanism must be described. The preprint provides neither the control-arm prompt template nor the scheduling rule. This is not a cosmetic omission: the mechanism claim is precisely that representing dependencies explicitly is what helps small models reason, and the control condition as described does not isolate that factor. The lack of code/data in the manuscript makes the ambiguity unresolvable from the preprint. Secondary issues (single baseline, undefined RTR formula, no confidence intervals) reinforce the need for a fair, fully specified comparison, but the ablation confound is the most load-bearing.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DART-LLM, an end-to-end framework for multi-robot task execution in which an LLM parses natural-language instructions into JSON subtasks with explicit dependency lists, constructs a DAG, assigns robots, and executes atomic navigation and robot-specific skills, while a VLM-based detector maintains an object map. It introduces a 102-instruction construction benchmark at three complexity levels, evaluates five LLMs inside DART-LLM against SMART-LLM, and reports success rate, instruction parsing accuracy, dependency satisfaction rate, semantic grounding success rate, and response time reliability. The central claims are that explicit DAG-based dependency modeling improves task performance and especially compensates for the limited reasoning capabilities of small models, and that DART-LLM significantly outperforms the baseline across all evaluation metrics.","tokens_in":11915,"tokens_out":4228,"duration_ms":47055,"significance":"If the claims hold, the contribution is practically valuable: a structured dependency representation in the prompt, a fixed atomic skill library, real-time ROS2-based execution, and a reusable benchmark for multi-robot construction tasks. The idea that explicit dependency lists improve small-model planning is directionally sensible, and the paper provides a concrete system plus videos and a claimed code release. However, the evidence as presented is not yet sufficient: the ablation control is underspecified and potentially confounded, the state-of-the-art claim rests on a single baseline with point estimates on small samples, and the pseudocode contains an ordering inconsistency. The contribution is therefore promising but needs a revised experimental and presentation pass.","major_comments":[{"comment":"The 'Without Dependency' ablation condition is not specified, and this ambiguity is load-bearing for the paper's central claim that explicit DAG modeling compensates for small-model reasoning limits. If the control simply removes dependency lists from the JSON output, then every subtask has an empty dependency list and Algorithm 1 launches all subtasks in parallel; since L3 tasks are defined by strict execution order, the observed drop (e.g., Llama3.1: 0.84 to 0.45 in Fig. 3) could be caused by the executor policy change alone, not by degraded LLM reasoning. Please provide the exact control prompt template and the scheduling rule used in the 'Without Dependency' arm, or rerun the control while preserving execution order through an alternative mechanism (e.g., list order) so that the only difference is the explicit DAG representation.","section":"IV-C-2 and Algorithm 1 (lines 11-18)"},{"comment":"The claim that DART-LLM 'significantly outperforms' the baseline and achieves state-of-the-art performance is not statistically supported. The comparison uses only one baseline method (SMART-LLM), and all metrics are point estimates without confidence intervals, error bars, or significance tests. This matters particularly for L3, which contains only 22 tasks, so a difference of 0.94 vs. 0.93 corresponds to about one task; small absolute differences in Table V cannot support the stated significance. Please report multiple trials with intervals and appropriate statistical comparisons, and ideally add additional baseline methods.","section":"Table V and Section IV-C-1"},{"comment":"Even if the ablation scheduling confound is resolved, the mechanism claim—that dependency modeling improves the model's reasoning—requires evidence that the effect is not solely due to output format differences. The current paper only reports end-to-end success rates; it does not compare decomposition quality, dependency satisfaction, or parsing accuracy between the with/without conditions. Adding these metrics for the ablation arms would strengthen the claim that the DAG representation changes the model's planning behavior rather than merely changing how the executor interprets the output.","section":"Section IV-C-2 and Fig. 3"}],"minor_comments":[{"comment":"The pseudocode is internally inconsistent: lines 11-18 execute subtasks in topological order before lines 19-25 assign robots to subtasks, but robot assignment must logically precede execution. Please reorder the algorithm so that assignment happens before execution, or clarify the intended control flow.","section":"Algorithm 1"},{"comment":"The prompt template P = (I, E, R, S, F) and the few-shot example set F are central to reproducibility, but the paper does not include the actual prompt or the few-shot examples. Since F is a free parameter, please release the full prompt templates, the JSON schema, and the few-shot examples along with the code.","section":"Section III-A and III-C"},{"comment":"The Response Time Reliability (RTR) metric is described only verbally as using average response time and standard deviation; the paper does not give the exact formula that maps these quantities to a value in [0,1]. Please define RTR formally.","section":"Section IV-B"},{"comment":"The skill list for L3-T2-001 is given as 'FE1, FE2, FE2, FD2', which appears to contain a duplicated FE2; please check whether this is a typo and ensure the skill lists match the task descriptions.","section":"Table IV"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new piece here is the packaging: an LLM that emits JSON subtask lists with explicit dependency edges, plus a topological scheduler that actually respects those edges on real mobile robots. That combination is absent from SMART-LLM and RoCo, and the benchmark of 102 instructions across three complexity levels is a usable resource. The observation that Llama-3.1-8B nearly matches much larger models on success rate, while beating them on response-time reliability, is interesting and worth taking seriously.\n\nWhat the paper does well is its systems integration. End-to-end real-time execution with a VLM-based object detector is nontrivial, and Table V's numbers are internally consistent. The ablation in Fig. 3 shows a clear drop without dependencies, and the related-work table is honest about what is new.\n\nThe soft spots are real, though. The stress-test concern about the ablation is the most load-bearing. The paper never specifies what changes under the \"Without Dependency\" condition. If that arm simply omits the dependency lists from the JSON prompt, then Algorithm 1's default scheduling launches every subtask with an empty dependency list in parallel. L3 tasks are explicitly defined by strict execution order, so that control would fail almost regardless of the LLM's decomposition quality. The observed gap (e.g., Llama3.1 0.84 to 0.45) could then be an artifact of the changed executor policy rather than evidence that DAG prompting improves model reasoning. The paper's central claim is precisely that explicit dependency modeling helps small models reason, so this ambiguity is not cosmetic. The authors need to describe the control-arm prompt and scheduling rule, and ideally release code or data, before the claim is supported.\n\nSecondary issues are proportionate: \"state-of-the-art\" rests on a single baseline (SMART-LLM); RTR is under-defined (average and std only, no formula); L3 has just 22 tasks with no confidence intervals. These are fixable in revision.\n\nOverall, this is a solid engineering contribution that deserves a serious referee. The authors should be asked to clarify the ablation protocol and provide code/data. For a reader building LLM-based multi-robot systems, especially with small models, this is worth reading despite the current gaps.","headline":"Useful engineering integration with a clean execution loop, but the ablation supporting the main claim is confounded by the executor policy and needs clarification.","tokens_in":674,"tokens_out":1094,"would_cite":false,"duration_ms":25809,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By adding a dependency graph to structured JSON output, DART-LLM lets smaller language models plan multi-robot construction tasks at state-of-the-art success rates.","keywords":["multi-robot task planning","task decomposition","large language models","directed acyclic graph","dependency modeling","construction robotics","natural language instruction","vision-language object detection"],"falsifier":"Run the level-3 benchmark with the same DART-LLM prompts but execute the generated subtasks in random order rather than topological order of the dependency graph; if Llama-3.1's success rate stays near 0.84, the DAG is not what produces the reported gain.","tokens_in":11448,"feed_emoji":"🏗️","tokens_out":7799,"duration_ms":68520,"temperature":0.7,"pith_summary":"This paper claims that a major bottleneck in LLM-driven multi-robot planning is not just model capability but the absence of explicit task-order information. DART-LLM makes a large language model decompose a natural-language instruction into subtasks, each named by a fixed atomic skill and each carrying a list of prerequisite subtasks, and then runs those subtasks in topological order of the resulting dependency graph. On 102 construction-robot instructions, the framework reports perfect instruction parsing at every complexity level and success rates of 0.94 on the hardest level with DeepSeek-r1 and 0.84 with the 8B-parameter Llama-3.1, against a SMART-LLM baseline of 0.65 and 0.24. An ablation on level-3 tasks shows Llama-3.1's success rate falling from 0.84 to 0.45 when dependencies are not modeled, which the paper reads as evidence that the dependency graph compensates for small models' weaker reasoning.","feed_headline":"Dependency graphs lift small-model multi-robot planning","feed_subtitle":"DART-LLM's JSON subtask graph takes an 8B model from 45% to 84% success on hardest construction tests.","key_machinery":"The load-bearing mechanism is the dependency-annotated directed acyclic graph (DAG) that the QA LLM produces as structured JSON. Each subtask $T^k_{t_j}$ names a breakdown function from the closed skill library $S$, object keywords, and a dependency list $d_k$; the system turns those lists into directed edges $D \\subseteq T \\times T$ and executes subtasks in topological order. The prompt $P=(I,E,R,S,F)$ ties the model to the environment, the robot skills, and the few-shot examples, so the planner's output is executable by construction rather than code that needs parsing or debugging.","core_discovery":"The central discovery is that a directed acyclic graph of subtask dependencies, expressed in a constrained JSON output and enforced by a runtime scheduler, is enough to make end-to-end multi-robot task execution reliable across a range of foundation models. The QA LLM does not generate code; it selects atomic skills from the skill library $S$, lists object keywords, and annotates each subtask with its dependencies $d_k$. The DAG built from these lists lets independent subtasks execute in parallel and dependent subtasks wait, while a VLM-based detector keeps the object map current. The paper reports that DART-LLM with DeepSeek-r1 reaches success rates of 1.00/0.97/0.94 across the L1/L2/L3 task levels, DART-LLM with Llama-3.1-8B reaches 1.00/0.85/0.84, and both beat the corresponding SMART-LLM baselines, which score 1.00/0.78/0.65 and 1.00/0.36/0.24 respectively. The ablation makes the mechanism explicit: removing the dependency structure drops Llama-3.1's L3 success rate from 0.84 to 0.45.","pith_inferences":["The paper leaves untested how much of the gain comes from the dependency semantics versus the strict JSON format; a version that outputs the same JSON but ignores dependency lists at execution time would separate the two.","Because the skill library is closed, transfer to new construction or rescue domains depends on whether their operations can be enumerated in advance; open-ended tasks would need a hierarchical skill expansion mechanism.","A natural prediction, testable on the released benchmark, is that error rates track the number of dependency edges rather than the number of subtasks, since the DAG offloads ordering reasoning from the model to the scheduler."],"forward_implications":["Small models become deployable on constrained robots: on the hardest task level, Llama-3.1-8B with the DAG scores 0.84 success rate versus 0.24 for the SMART-LLM baseline using the same model.","The benefit is not specific to one foundation model; all five tested LLMs improve when dependencies are explicit, so the JSON-plus-DAG format is a model-agnostic prompt intervention.","Perfect instruction parsing across all 102 instructions means failure, when it happens, lies in dependency satisfaction or execution rather than in understanding the command.","Because the skills are atomic and executed through the ROS navigation stack, the planner runs end-to-end without manual code execution, which is the practical advantage over code-generating baselines.","The DAG's parallelism rule directly supports multi-robot coordination: independent subtasks launch concurrently and dependent subtasks wait, which is what the L3 tasks require."],"supporting_citations":[{"why":"Supplies the SMART-LLM baseline that DART-LLM must beat and whose Python-code output the paper identifies as a failure source.","marker":"[12]"},{"why":"The Llama-3.1-8B foundation model used to show that small models reach competitive success rates with the DAG.","marker":"[17]"},{"why":"The DeepSeek-r1 foundation model that achieves the highest success rates in the DART-LLM evaluation.","marker":"[21]"},{"why":"The CLIP-based model used by the VLM object detector to identify objects and update the object map for grounding.","marker":"[16]"},{"why":"The ROS Navigation stack that implements the navigation atomic skills for all robots.","marker":"[14]"},{"why":"ROS2 topics, the communication layer that connects all DART-LLM modules for real-time execution.","marker":"[13]"}],"fun_headline_variants":["DAGs lift small-model multi-robot planning","Dependency graph boosts 8B model from 45% to 84%","DART-LLM: explicit DAG key to multi-robot success","Small LLMs excel with dependency-aware task graphs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every instruction a user gives can be decomposed into the fixed atomic skills and object keywords listed in the prompt; instructions requiring new actions, object types, or spatial reasoning outside that closed menu cannot be represented.","fun_headline_variants_meta":{"raw":{"variants":["DAGs lift small-model multi-robot planning","Dependency graph boosts 8B model from 45% to 84%","DART-LLM: explicit DAG key to multi-robot success","Small LLMs excel with dependency-aware task graphs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000347,"raw_usage":{"total_tokens":1954,"prompt_tokens":1052,"completion_tokens":902,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":668,"completion_tokens_details":{"reasoning_tokens":829}},"tokens_in":668,"tokens_out":902,"duration_ms":8643,"temperature":1.0,"reasoning_tokens":829,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:08:46.080237+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the level-3 benchmark with the same DART-LLM prompts but execute the generated subtasks in random order rather than topological order of the dependency graph; if Llama-3.1's success rate stays near 0.84, the DAG is not what produces the reported gain.","supporting_citations":[{"cited_title":"The marathon 2: A navigation system,","cited_arxiv_id":null,"evidence_quote":"The ROS Navigation stack that implements the navigation atomic skills for all robots."}],"review_version":1}