{"id":"3c89f320-3a82-4a71-b91e-08aafee4264f","arxiv_id":"2504.18945","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A workload characterization of 14 embodied LLM agent systems showing that planning and communication dominate latency, communication is often redundant, and multi-agent systems scale poorly.","lead":"This paper benchmarks 14 embodied AI agent systems, breaking down latency by module (planning, communication, memory, reflection, execution) and measuring how success rate and runtime scale with memory size and agent count. It identifies LLM-based planning and communication as the main latency bottleneck and redundant dialogue as a key inefficiency, then proposes optimization strategies.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 70.2% LLM-latency claim and the module-sensitivity numbers rest on unreported trial counts and no variance, so the central quantitative result is not yet statistically grounded.","rationale":"The reader's verdict is CONDITIONAL, and the weakest assumption identified--statistical stability of the measurements--is exactly the most load-bearing concern. The paper's central contribution is a quantitative system-level characterization: the 70.2% LLM-latency fraction and the module-ablation effects. These numbers motivate all subsequent recommendations (batching, quantization, communication filtering, memory compression). If those numbers are not reproducible across runs, the paper's quantitative conclusions are unsupported, though the qualitative directions (planning/communication are heavy, memory matters, multi-agent scaling is hard) are likely still correct. I considered whether a more fundamental issue exists, such as the representativeness of the 14 workloads or the 'first system-level analysis' novelty claim, but those affect framing rather than the validity of the central measurement. The statistical-confound concern is concrete, testable, and directly tied to the paper's main evidence. The reader's CONDITIONAL verdict remains appropriate: the paper should not be rejected outright because the qualitative insights and the workload taxonomy are useful, but it should not be fully accepted without addressing the missing trial counts and confidence intervals. My proposed check would settle whether the concern actually lands: if the numbers are stable across repeated runs, the concern is resolved; if not, the quantitative claims should be softened or re-derived. I agree with the reader that this is the weak spot, and no additional objection rises to the same level of importance.","tokens_in":147,"tokens_out":2217,"duration_ms":63684,"concrete_test":"Run 10 repeated executions of CoELA and JARVIS-1 on the same tasks with fixed seeds and LLM sampling parameters, measuring per-module latency. Compute the mean and 95% CI for the LLM-module percentage of total latency for each system and the pooled 70.2% aggregate using both unweighted and latency-weighted averaging. If any CI width exceeds ±5 percentage points or the weighted vs unweighted aggregates differ by more than 5 percentage points, the central quantitative claim in Sec. IV-A is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (Sec. IV-A, Fig. 2a) is that LLM-based planning and communication account for 70.2% of total latency on average across 14 workloads. This number, and the related sensitivity results in Sec. IV-B (e.g., disabling memory increases steps by 1.61x and reduces success rate by 27.7%; disabling reflection causes 1.88x steps and 33.3% success drop), are presented as single point estimates. Section III-E defines the metrics but gives no experimental protocol: no number of runs per workload, no seeds, no confidence intervals, no variance across runs. Since GPT-4 API calls are stochastic and task success is high-variance, the reported percentages could shift substantially under repetition. Moreover, the 70.2% figure is an unweighted average across heterogeneous workloads: single-agent systems have no communication module, while multi-agent systems have both planning and communication, so the aggregate is sensitive to the composition of the workload suite and to how module percentages are averaged (per step vs. per task, equal-weight vs. latency-weight). If the true LLM-module fraction were, say, 55-85% depending on run and aggregation, the headline 'planning and communication dominate' might still hold qualitatively, but the specific 70.2% number and the derived optimization priorities would not be quantitatively reliable. The paper itself does not report any limitation or caveat about run-to-run variability, which is a gap in the evidence supporting the paper's main contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a system-level workload characterization of embodied AI agents that use large language models. It proposes a taxonomy of four paradigms (single-agent modularized, single-agent end-to-end, multi-agent centralized, multi-agent decentralized), curates a suite of 14 embodied-agent systems spanning these paradigms, and benchmarks each for task success rate, runtime latency, per-module latency, and number of steps. The central empirical claim is that LLM-based planning and communication modules account for an average of 70.2% of total per-step latency across the 14 workloads (Section IV-A, Fig. 2a), and that memory and reflection modules are critical for task efficiency, with ablations showing 1.61× and 1.88× increases in steps and 27.7% and 33.3% drops in success rate, respectively (Section IV-B, Fig. 3). The paper also studies the effects of planning-model choice, memory capacity, prompt token growth, and agent count, and it concludes with a set of system optimization recommendations.","tokens_in":22595,"tokens_out":4634,"duration_ms":47513,"significance":"If the quantitative results are reliable, this would be a useful first system-level analysis for embodied-agent workloads, identifying LLM planning and communication as the dominant latency contributors and providing concrete targets for optimization. The curated workload suite, the taxonomy, and the qualitative observations (e.g., redundant dialogue, prompt-length growth, centralized versus decentralized scaling trade-offs) are potentially valuable to the architecture and systems community. However, the central numbers are presented as point estimates without any statistical grounding, and the aggregation methodology for the headline 70.2% figure is underspecified. These issues affect the credibility of the paper's main quantitative conclusions, so the current version cannot be accepted without substantial revision.","major_comments":[{"comment":"The experimental protocol is missing the number of trials, random seeds, or confidence intervals for any of the reported latency and success-rate numbers. The headline claim that LLM-based modules account for 70.2% of average latency (Fig. 2a) is a single point estimate, despite stochastic LLM API inference and high-variance task success. The same applies to the sensitivity results in Section IV-B (e.g., 1.61× steps and 27.7% success drop when memory is disabled). Without error bars or trial counts, these numbers cannot be distinguished from noise. I request that the authors specify how many independent runs per workload and per condition were performed, and that they report standard errors or confidence intervals on all point estimates, with significance tests for the ablation differences.","section":"Section III-E and Section IV-A"},{"comment":"The aggregation rule behind the 70.2% average latency fraction is not defined. It is unclear whether the per-module percentages are averaged per time step and then equally across workloads, or weighted by task latency, and how systems lacking a communication module (e.g., single-agent systems) are treated. Because the workload suite mixes single-agent and multi-agent systems, the reported average is sensitive to the suite composition and to the averaging choice. The authors should provide the exact formula, report per-workload module breakdowns (e.g., as a table or supplementary material), and discuss how the average would change if computed differently.","section":"Section IV-A, Fig. 2a"},{"comment":"The module sensitivity analysis is performed on only six systems (CoELA, COMBO, COHERENT, RoCo, HMAS, JARVIS-1), not the full suite, and the text does not justify this selection or report per-system results. The aggregate statements such as 'disabling memory increases steps by an average of 1.61× and reduces success rate by 27.7%' are presented as averages with no error bars, and the claim that disabling communication 'does not have significant impact' appears to be a hypothesis (indeed, the text says 'We hypothesize') rather than a statistically supported finding. The authors should show per-system values, report variance, and test whether the observed differences are statistically meaningful.","section":"Section IV-B, Fig. 3"},{"comment":"All results in the configuration and scalability analyses (local-model comparison, memory capacity, token-length evolution, and multi-agent scaling) are reported as point estimates with no variance. For example, Fig. 7 presents success-rate and latency curves versus number of agents without any error bars, although multi-agent task success is notoriously high-variance. I ask the authors to add replication counts and confidence intervals to these figures, or at a minimum to clearly label them as single-run illustrative results with a caveat that they are not statistically validated. The absence of any limitation statement about run-to-run variability is a gap in the paper's evidence.","section":"Sections V and VI, Figs. 4-7"}],"minor_comments":[{"comment":"The introduction says both 'the first comprehensive analysis' and 'one of the first works' when describing the contribution; these statements should be reconciled to avoid overclaiming.","section":"Section I"},{"comment":"The success rate metric is not formally defined; the authors should state how task success is determined for each workload and whether partial success is allowed.","section":"Section III-E"},{"comment":"The figures would benefit from clearer axis labels and legends: Fig. 2a's horizontal axis is labeled in seconds, but the caption refers to percentage contributions; Fig. 3's marker legend for the modules is not described in the caption.","section":"Fig. 2a and Fig. 3"},{"comment":"The action types 'V', 'T', and 'E' in Table I should be expanded in the caption (virtual, tool usage, physical) for readability.","section":"Table I"},{"comment":"References [4] and [82] appear to describe the same paper ('Aligning cyber space with physical world...'); please consolidate or distinguish them.","section":"References"},{"comment":"The text refers to 'Llama-3.1-8B' while Fig. 4 labels the model 'Llama-3-8B'; please align the notation.","section":"Section V-A"},{"comment":"The recommendations (e.g., batching, quantization, planning-then-communication) are not validated by the experiments in this paper; I suggest labeling them clearly as informed hypotheses for future work rather than findings of this study.","section":"Sections IV-VI"}],"recommendation":"major_revision","confidential_remarks":"One of the authors (Yilun Du) is also an author of two of the benchmarked systems, CoELA (Ref. [6]) and COMBO (Ref. [7]). This is not itself grounds for rejection, but the editors may want to ensure there is no undisclosed conflict of interest in benchmarking one's own systems, and perhaps to ask the authors to comment on how they mitigated bias in the setup and evaluation. Additionally, the paper does not mention any limitations regarding statistical power, which is a concern given the strong quantitative claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the first broad system-level latency map of embodied LLM agents I've seen, and the qualitative takeaways are probably right. But the headline numbers (70.2% LLM latency, 27.7% success-rate drop without memory, 33.3% without reflection) are single point estimates with no trial counts, no seeds, no variance, so I'd treat them as directional rather than quantitative.\n\nWhat earns credit: the six-building-block / four-paradigm taxonomy is a genuinely useful way to organize the space. The paper actually built and ran 14 different embodied agent systems, broke down per-step latency by module, ran ablations, tested memory capacity, and measured multi-agent scaling. That's real work, and the results match what I'd expect: planning and communication dominate latency, much dialogue is redundant (the 20% usage stat is telling), memory and reflection buy reliability, and decentralized multi-agent systems blow up in communication cost. The scalability plots are consistent with intuition.\n\nSoft spots, in proportion: the lack of statistical grounding is the main one. GPT-4 calls are stochastic, task success is high-variance, and we get no number of trials or confidence intervals anywhere. An unweighted average across 14 heterogeneous workloads (some single-agent with no communication module, some multi-agent) makes the 70.2% figure sensitive to suite composition and averaging method. That doesn't kill the qualitative story—planning and communication are clearly the bottleneck—but it means specific percentages and the derived optimization priorities shouldn't be quoted as precise. Also, no artifacts are linked: the paper says they built docker images but gives no repo or hash, which for a benchmarking paper is a real gap for reproducibility. The 'first system-level analysis' framing should be positioned more carefully against the authors' own prior work (refs [2] and [5]); that's not a fatal flaw, but the novelty claim is overstated.\n\nThe citation pattern is mostly fine. The paper cites the systems it benchmarks and related workload-characterization work; the self-citations are to directly relevant prior architecture papers, not gratuitous.\n\nWho it's for: systems and architecture people who want a map of where time goes in embodied LLM agents and what to optimize. It's a useful starting point, not the final word. I'd send it to peer review, but with the expectation that the authors add repeated-run results with variance, release the workloads, and soften the quantitative claims. If I were building a hardware/software co-design project for embodied agents, I'd cite this after those fixes.","headline":"A genuinely useful first system-level latency map of embodied LLM agents, but the headline numbers are point estimates without variance, so treat them as directional.","tokens_in":23184,"tokens_out":2275,"would_cite":true,"duration_ms":23869,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Across 14 embodied AI agent systems, LLM-based planning and communication account for 70.2% of runtime latency on average.","keywords":["embodied AI","large language models","system-level benchmarking","latency analysis","multi-agent systems","scalability","workload characterization","module sensitivity"],"falsifier":"Run the same 14 workloads with, say, ten or more random seeds per configuration, recording step counts and success rates for the intact system and for each disabled-module variant. If the claimed gaps (e.g., 27.7% success drop without memory, 33.3% without reflection) fall inside the seed-to-seed spread or reverse sign on some workloads, the central sensitivity claims are not stable. A second check would swap the planning LLM for a different API or local model and re-measure the 70.2% latency share; if the share moves by tens of percentage points, the bottleneck claim is model-specific rather than a system-level law.","tokens_in":22125,"feed_emoji":"🤖","tokens_out":5862,"duration_ms":52789,"temperature":0.7,"pith_summary":"This paper tries to establish where time and reliability go in embodied AI agent systems—robots and simulated agents that perceive an environment, plan with a large language model, and execute physical or virtual actions. By benchmarking 14 published systems, it claims that LLM-based planning and communication dominate runtime, averaging 70.2% of per-step latency and producing end-to-end task times of 10 to 40 minutes. It also claims that memory and reflection modules are load-bearing for task success: removing memory raises required steps by 1.61x and lowers success by 27.7%, while removing reflection raises steps by 1.88x and lowers success by 33.3%. These results matter because they direct system architects to the modules that deserve optimization effort, rather than treating the entire pipeline as one undifferentiated cost.","feed_headline":"Embodied AI agents spend 70% of runtime on LLM calls","feed_subtitle":"A 14-workload benchmark traces 10–40 minute task delays to planning and communication, not sensing or control.","key_machinery":"The central object is the six-module decomposition of an embodied agent—sensing, planning, communication, memory, reflection, and execution—paired with a 14-workload benchmark suite that instantiates these modules across four paradigms: single-agent modularized, end-to-end, centralized multi-agent, and decentralized multi-agent. The machinery does the essential work of turning \"how well does embodied AI work\" into per-module latency percentages and ablation-style sensitivity numbers: each system is run with a module disabled, and the change in steps, success rate, and runtime is attributed to that module. The named quantity that carries the argument is the average latency share of LLM-based modules, 70.2%, used as the headline target for optimization.","core_discovery":"The paper's central empirical claim is a module-level attribution of latency and reliability across embodied agent workloads. Using a suite of 14 published embodied AI systems, it reports that LLM-based planning and communication—whether served through GPT-4 API calls or local Llama/LLaVA models—account for an average of 70.2% of runtime latency per step, with low-level execution also contributing large shares in systems such as RoCo (49.4%), DaDu-E (38.1%), and EmbodiedGPT (24.1%). It further claims that disabling the memory module increases required steps by 1.61x and cuts success rate by 27.7% across six tested systems, while disabling reflection increases steps by 1.88x and cuts success rate by 33.3%; disabling execution causes task failure. A secondary claim is that centralized multi-agent systems lose task success as agent count grows, while decentralized systems suffer communication overhead that scales quadratically, with end-to-end latency already at 10–40 minutes for a single task.","pith_inferences":["If the 70.2% figure is roughly model-independent, future embodied-agent accelerators and schedulers should specialize in transformer inference for planning and dialogue rather than in perception or control; this is an extension the paper points toward but does not measure.","The memory ablation result suggests a concrete experiment: replacing raw observation and action histories with a compressed or learned memory should preserve most of the success-rate gain while reducing retrieval latency—something the paper recommends qualitatively but does not test.","The quadratic token growth in decentralized systems implies that API cost, not just wall-clock latency, will dominate at scale; reporting token consumption alongside runtime would let planners compare paradigms on an economic basis.","Because success rates are reported without trial counts or confidence intervals, a multi-seed re-run on a fixed task set would determine which sensitivity gaps are stable; the paper's numbers should be treated as hypotheses until then."],"forward_implications":["System optimization should target planning and communication first, using the paper's recommended levers of batching LLM queries, quantizing models, and deploying lightweight local models.","Memory and reflection are not optional extras; preserving their benefits while reducing their cost, for example through context summarization and dual short-term/long-term storage, follows directly from the ablation results.","Communication should become conditional: generate messages only when planning determines they are needed, because a large fraction of dialogue rounds are redundant.","Multi-agent scalability requires a hierarchical or clustered cooperation structure, since centralized planners lose task success with more agents and decentralized communication grows quadratically.","Low-level execution should be offloaded to specialized controllers rather than left to LLM-driven inference, because execution is both a major latency contributor and essential to task completion."],"supporting_citations":[{"why":"Supplies CoELA, a decentralized multi-agent workload whose per-step LLM inference pattern drives the communication-latency finding.","marker":"[6]"},{"why":"Supplies COMBO, a decentralized world-model workload used for the scalability and communication analysis.","marker":"[7]"},{"why":"Supplies MindAgent, the centralized multi-agent workload whose agent-count results support the scalability claims.","marker":"[8]"},{"why":"Supplies DEPS, a single-agent modular workload that anchors the per-step latency and module-sensitivity measurements.","marker":"[18]"},{"why":"Provides the CMAS, DMAS, and HMAS multi-agent systems used to compare centralized and decentralized scaling.","marker":"[23]"},{"why":"Supplies JARVIS-1, a memory-augmented single-agent workload central to the memory-capacity and reflection experiments.","marker":"[27]"},{"why":"Supplies RoCo, a decentralized multi-robot workload whose 49.4% execution-latency share supports the low-level-control bottleneck claim.","marker":"[30]"},{"why":"Supplies COHERENT, a centralized heterogeneous-robot workload whose message-generation pattern illustrates communication overhead.","marker":"[31]"},{"why":"Supplies EmbodiedGPT, a single-agent workload that grounds the end-to-end latency and planning-latency attribution.","marker":"[42]"},{"why":"Supplies DaDu-E, a robotics workload whose 38.1% execution share supports the claim that low-level execution is a major latency contributor.","marker":"[43]"}],"fun_headline_variants":["Embodied AI: 70% of runtime is LLM calls","Memory loss makes embodied AI fail 28% more often","Scaling embodied agents: latency grows quadratically","Embodied AI tasks take 10-40 min mostly LLM calls"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the latency and success-rate numbers, each reported as a single value, are stable enough that differences of several percentage points or a factor of 1.6 in steps are real and not run-to-run noise, even though LLM outputs are stochastic and the paper reports no trial counts or confidence intervals.","fun_headline_variants_meta":{"raw":{"variants":["Embodied AI: 70% of runtime is LLM calls","Memory loss makes embodied AI fail 28% more often","Scaling embodied agents: latency grows quadratically","Embodied AI tasks take 10-40 min mostly LLM calls"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000958,"raw_usage":{"total_tokens":4096,"prompt_tokens":974,"completion_tokens":3122,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":3051}},"tokens_in":590,"tokens_out":3122,"duration_ms":23113,"temperature":1.0,"reasoning_tokens":3051,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:05:04.347871+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 14 workloads with, say, ten or more random seeds per configuration, recording step counts and success rates for the intact system and for each disabled-module variant. If the claimed gaps (e.g., 27.7% success drop without memory, 33.3% without reflection) fall inside the seed-to-seed spread or reverse sign on some workloads, the central sensitivity claims are not stable. A second check would swap the planning LLM for a different API or local model and re-measure the 70.2% latency share; if the share moves by tens of percentage points, the bottleneck claim is model-specific rather than a system-level law.","supporting_citations":[{"cited_title":"Embodiedgpt: Vision-language pre-training via embodied chain of thought,","cited_arxiv_id":null,"evidence_quote":"Supplies EmbodiedGPT, a single-agent workload that grounds the end-to-end latency and planning-latency attribution."}],"review_version":1}