{"id":"7900707e-64c7-4983-a154-1bd25f61409e","arxiv_id":"2506.07232","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LIET combines a finetuned local utility function with an iteratively updated shared knowledge list to improve multi-agent LLM planning, and it outperforms CoELA, ProAgent, and RoCo on C-WAH and TDW-MAT benchmarks.","lead":"The paper proposes LIET, a framework that adapts LLM agents to multi-agent embodied tasks by adding a learned cost-prediction module and a shared, evolving communication knowledge list. In tests on two household simulators, LIET completes tasks faster and transports more objects than several existing LLM planning baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section A.1 collects exploratory data from exactly 10 C-WAH and 24 TDW-MAT episodes, matching the evaluation set sizes, with no stated disjointness; if utility finetuning saw evaluation episodes, LIET's main gains are partly memorization.","rationale":"I read the central claim as saying that LIET's two adaptation mechanisms, individual utility learning and evolving team communication, are responsible for its superior planning and cooperation. The load-bearing condition for the individual-learning half is that the utility function was not fitted on the same episodes used to evaluate it. The paper's own appendix creates a concrete reason to doubt this: exploratory data sizes exactly equal the evaluation set sizes, and no disjointness is stated. This is an internal evidence gap rather than a disagreement with consensus. If overlap exists, then the 12.4-step improvement over LIET w/o individual learning in Table 3 is inflated by memorization, and the comparisons against CoELA, ProAgent, and RoCo are unfair in LIET's favor because the utility has seen the evaluation instances. I would not reject the paper outright: the framework is coherent, the team-evolving component operates at test time and is less exposed to this particular leak, and the concern is empirically checkable. The verdict should remain conditional, with release of episode splits, variance or significance reporting, and code as conditions. I agree with the reader's weakest_assumption, so no verdict adjustment is needed.","tokens_in":15001,"tokens_out":6836,"duration_ms":74543,"concrete_test":"Ask the authors for the episode IDs, environment seeds, or layout identifiers for both the exploratory dataset and the evaluation set, and verify that the sets are disjoint. If they are not available, rerun the C-WAH and TDW-MAT experiments with the utility finetuned only on exploratory data from episodes that are provably excluded from evaluation, e.g., exploration collected from held-out floorplans or regenerated object placements. The decisive comparison is the C-WAH Table 3 ablation: recompute LIET versus LIET w/o individual learning (40.3 vs 52.7) with a disjoint-trained utility. If the gap persists, the concern is settled; if the gap shrinks materially or disappears, the reported individual-learning improvement is substantially an artifact of memorized evaluation episodes, and the central claim must be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states that the C-WAH evaluation set consists of 10 episodes and the TDW-MAT evaluation set includes 24 episodes. Section A.1 then states: 'For the exploratory dataset, we collected 10 episodes for C-WAH benchmark and 24 episodes for TDW-MAT benchmark. We used 80% of this data as the training set.' The numbers match exactly and no passage states that the exploratory episodes are drawn from different task instances, object layouts, seeds, or floorplans than the evaluation episodes. The individual utility module is the 'learn as individuals' component, and Table 3 shows that removing it degrades C-WAH from 40.3 to 52.7 steps. Because the utility is trained with MSE on cost labels for concrete actions in concrete episodes, overlap would let it memorize episode-specific object locations and action costs rather than learn transferable environment knowledge; at test time it would be recalling the evaluation instance. The 80/20 split in A.1 does not resolve this: with 10 C-WAH episodes, if the same 10 episodes supplied the exploratory data, 8 of them are still in the utility training set. This directly affects the headline claim of outperforming baselines and the attribution of the gains to individual learning, so the train/eval disjointness must be demonstrated before the result can be accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LIET (Learn as Individuals, Evolve as a Team), a semi-centralized framework for adapting LLM-based planners to multi-agent embodied environments. At the individual level, agents use a utility function, implemented by fine-tuning a small LLM with LoRA and a value head, to estimate action costs from exploratory data. At the team level, agents maintain a shared cooperation knowledge list that is iteratively updated through receiver-side reflection, and this list guides message generation during test time. The method is evaluated on Communicative Watch-And-Help (C-WAH) and ThreeDWorld Multi-Agent Transport (TDW-MAT) with LLaMA-3.1-70B and GPT-4o backbones, compared against CoELA, ProAgent, and RoCo. The paper reports improved task-completion steps and transport rates, ablation studies showing the utility function and evolving communication both contribute, and additional experiments with 3 and 4 agents.","tokens_in":15203,"tokens_out":2606,"duration_ms":31103,"significance":"If the reported results are sound, the paper makes a useful contribution to multi-agent LLM planning by combining two relatively lightweight adaptation mechanisms: a learned cost estimator for individual decision-making and an evolving communication prompt for team coordination. The framework is simple, modular, and potentially applicable to other embodied multi-agent settings, and the visualization and ablation studies help explain why each component matters. However, the central empirical claims currently lack statistical support, and there is a serious unresolved question about whether the utility function's training data overlaps with the evaluation episodes. Because both issues directly affect the validity of the headline comparisons, the significance of the contribution cannot be assessed from the manuscript as written.","major_comments":[{"comment":"The paper must clarify whether the exploratory dataset used to fine-tune the utility function is disjoint from the evaluation episodes. Section 4.1 states that the C-WAH evaluation set consists of 10 episodes and the TDW-MAT evaluation set includes 24 episodes. Section A.1 then states that the exploratory dataset was collected from 10 C-WAH episodes and 24 TDW-MAT episodes, with 80% used as the training set. These numbers match exactly, and no sentence in the paper says that the exploratory episodes come from different task instances, object layouts, seeds, or floorplans. Since the utility function is trained on (observation, action, cost) pairs and then queried during evaluation, any overlap would let it memorize episode-specific object locations and action costs rather than learn transferable environment knowledge. Table 3 shows that removing the individual utility module degrades C-WAH performance from 40.3 to 52.7 steps, so this potential leakage directly inflates the reported contribution of the 'learn as individuals' component. The authors need to either demonstrate disjointness explicitly or re-run the evaluation with held-out episodes before the headline claim can be accepted.","section":"Section 4.1 vs. Section A.1, Table 3"},{"comment":"The reported results lack any measure of statistical uncertainty. The implementation details state that LLM inference uses temperature = 0.7, so the planning process is stochastic, yet Tables 1 and 2 report only single average values for each method without standard deviations, confidence intervals, or significance tests. The abstract's claim that LIET 'significantly outperforms' baselines is therefore not supported by the evidence presented. The same issue applies to the ablation results in Table 3 and the multi-agent results in Table 4. At minimum, the paper should report means and standard deviations over multiple independent runs (with distinct random seeds) and perform a standard significance test for the main comparisons against CoELA, ProAgent, and RoCo.","section":"Tables 1, 2, 3, and Section 4.1"}],"minor_comments":[{"comment":"There is a typo in the message generator prompting template: 'placehoder' should be 'placeholder'.","section":"Section 3.3"},{"comment":"The heading 'Learn as Individuals, Envolve as a Team' contains a typo; 'Envolve' should be 'Evolve'.","section":"Section 3"},{"comment":"The baseline description says 'A-start-based low-level planner'; this should be 'A*' or 'A-star'.","section":"Section 4.1"},{"comment":"The paper states that code will be released after review, but it does not report the exact split of the exploratory dataset into training and validation, nor does it give the number of description-cost pairs per episode. Providing these details would improve reproducibility and help address the data-composition concern.","section":"Section A.1"}],"recommendation":"major_revision","confidential_remarks":"The matching episode counts between the exploratory dataset and the evaluation set is a red flag that should be resolved with the authors directly. I would not recommend rejection outright, because the issue might be fixable with a clear statement of disjointness or a re-run on held-out episodes. However, the lack of any statistical analysis, combined with the potential data leakage, makes the current empirical support insufficient for acceptance. If the authors can demonstrate that the utility training data is truly disjoint from evaluation and add proper error bars and significance tests, this paper could be a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a clearly written paper about a plausible method. The central claim—that adding a learned per-agent cost model plus an evolving shared communication prompt helps multi-agent LLM planners—is not supported as written, because the paper never shows the utility training data are disjoint from the evaluation episodes, and the numbers in A.1 make overlap look likely. Second: no result in the paper has a variance estimate, so \"significantly outperforms\" is doing a lot of work.\n\nThe integration itself is genuinely new: SayCan-style value grounding and Reflexion-style reflection have been applied in single-agent settings, and putting them together in a semi-centralized multi-agent planning loop with a shared knowledge list is a reasonable contribution. The writing is honest about limitations, and the ablations in Table 3 isolate the components sensibly.\n\nThe load-bearing issue is the exploratory dataset. Section 4.1 says C-WAH is evaluated on 10 episodes and TDW-MAT on 24. Section A.1 says they collected 10 C-WAH and 24 TDW-MAT episodes for the exploratory data, using 80% for training. No sentence says these are different episodes, different floorplans, or different object layouts. With 10 C-WAH episodes, an 80/20 split leaves 8 of the same episodes in the utility training set if they are the same collection. That means the \"learn as individuals\" component—whose removal costs 12.4 steps in Table 3—could be recalling instance-specific costs rather than learning transferable environment knowledge. The paper needs to demonstrate disjointness, not just assert an 80/20 split.\n\nSecondary but real: all tables report point estimates only. LLM planners with temperature 0.7 are stochastic; Table 1 shows differences around 8–14 steps between methods. Without error bars, significance tests, or multiple seeds, those differences are not interpretable. Ablations are single numbers too.\n\nThe framework is worth a serious look, and the problem is fixable in revision. I'd send it to peer review, because a conditional accept after the authors clarify the split, add variance estimates, and ideally release code/data would settle whether LIET actually works. As it stands, I wouldn't cite the headline result.","headline":"A sensible multi-agent LLM adaptation framework whose headline gains rest on an unverified train/eval split that the paper's own numbers make look suspicious.","tokens_in":15768,"tokens_out":2532,"would_cite":false,"duration_ms":28086,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multi-agent LLM planners adapt best when each agent learns a cost-prediction utility function and the team evolves a shared communication knowledge list, outperforming established baselines on two embodied benchmarks.","keywords":["multi-agent planning","LLM agents","embodied environments","utility function","test-time adaptation","cooperative communication","prompt evolution","Dec-POMDP"],"falsifier":"Re-run the C-WAH and TDW-MAT evaluations with a training/evaluation split guaranteed disjoint—for example, hold out entire room layouts or task types from utility-function training—and compare full LIET against its 'without individual learning' ablation; if the gap shrinks to near zero, the utility gain is memorization of evaluation episodes rather than generalization.","tokens_in":14770,"feed_emoji":"🤝","tokens_out":9982,"duration_ms":92437,"temperature":0.7,"pith_summary":"The paper proposes LIET—'Learn as Individuals, Evolve as a Team'—a semi-centralized recipe for making large-language-model agents plan together in embodied household environments. Its central claim is that LLM agents adapt better when each agent first learns a cost-prediction utility function from exploratory data and the team then evolves a shared cooperation-knowledge list through receiver-side reflection during test time. On the C-WAH and TDW-MAT benchmarks, with both LLaMA-3.1-70B and GPT-4o as backbones, LIET outperforms established decentralized and centralized baselines. If the claim is right, the bottleneck for multi-agent LLM planning in embodied settings is adaptation through experience rather than zero-shot prompting alone.","feed_headline":"LLM agents that learn costs plus team chat win embodied benchmarks","feed_subtitle":"On C-WAH and TDW-MAT, the semi-centralized LIET recipe beats established multi-agent LLM planners.","key_machinery":"The two mechanisms that carry LIET are the utility function and the shared cooperation knowledge list. The utility function is a parameter-efficiently fine-tuned LLM with an MLP value head that predicts per-action step costs from text descriptions; it supplies the individual learning that lets a planner prefer cheap, feasible actions. The cooperation knowledge list is a shared, evolving prompt of message-construction tips, maintained by a receiver-side reflector that evaluates incoming messages and merges insights back into the list; it supplies the team-level adaptation that makes communication increasingly specific and useful. The framework wraps both around a decentralized planner that broadcasts messages only at critical timesteps, which is what makes the design semi-centralized.","core_discovery":"LIET's central claim is that two layers of adaptation close the gap between off-the-shelf LLM planners and competent embodied teammates. In the individual layer, each agent queries a utility function $f(\\ell_{o_i}, \\ell_a)$ that returns the estimated number of environment steps needed to execute an action from the current observation; this function is a 1B-parameter LLM with an MLP value head, fine-tuned by mean-squared-error regression on exploratory episodes collected by agents themselves, so no human cost labels are needed. In the team layer, agents maintain a shared cooperation knowledge list—a growing set of communication tips—which is re-prompted into every message-generation call and is updated after each exchange by the receiver, who reflects on whether and how the incoming message helped planning. The paper reports that on C-WAH and TDW-MAT, with LLaMA-3.1-70B and GPT-4o as backbones, LIET achieves lower average task-completion steps and higher transport rates than the compared baselines, and its ablations attribute the gain to both layers rather than to one component.","pith_inferences":["A direct test the paper leaves implicit: publish the exploratory episodes and re-run with a guaranteed disjoint train/eval split; this would show whether the individual-learning gain reflects generalization or memorization of the fixed evaluation episodes.","The goal-agnostic cost predictor could be reused across tasks within the same simulator; reporting per-task cost-prediction error would indicate how much of the C-WAH gain is environment knowledge versus task-specific tuning.","The same two-layer pattern—a learned scalar cost/utility head on a small LLM, plus a shared prompt list updated by receiver-side reflection—could be dropped into other decentralized multi-agent settings such as warehouse or traffic coordination, since neither component depends on household chore structure."],"forward_implications":["LIET's individual utility function can be trained from self-collected exploration data, so adapting a planner to a new embodied environment does not require human expert labels for action costs.","The team-level knowledge list gives communication a test-time memory: messages become more specific as episodes progress, down to object-level task division, as the paper's visualizations show.","The utility function trained only on two-agent episodes transfers to three- and four-agent settings without retraining, which the paper presents as evidence of generalization to larger teams.","The C-WAH ablations show that both layers contribute: removing the learned utility function, replacing cost estimates with prompted guesses, or freezing the communication prompt all hurt performance."],"supporting_citations":[{"why":"Supplies both benchmarks and the modular decentralized baseline whose evaluation protocol LIET inherits.","marker":"[38]"},{"why":"Supplies a decentralized LLM baseline that models teammate intentions; a main comparison point.","marker":"[37]"},{"why":"Supplies a centralized multi-turn discussion baseline; LIET positions itself between this and decentralized planners.","marker":"[23]"},{"why":"Introduces grounding LLM actions with a learned value/affordance function, which motivates the utility module.","marker":"[2]"},{"why":"Provides LoRA, the parameter-efficient finetuning method used to adapt the utility model.","marker":"[12]"},{"why":"Provides the reflection loop pattern used by the receiver-side reflector that updates the cooperation knowledge list.","marker":"[28]"},{"why":"Defines the ThreeDWorld Transport Challenge that TDW-MAT extends.","marker":"[6]"},{"why":"Defines the Watch-And-Help Challenge that C-WAH extends.","marker":"[27]"}],"fun_headline_variants":["LLM agents adapt via individual learning and team evolution","Semi-centralized adaptation improves multi-agent LLM planning","Agents learn costs individually, evolve communication as a team","New method helps LLM agents cooperate in embodied environments"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the exploratory episodes teach a transferable cost function: the paper reports collecting 10 C-WAH and 24 TDW-MAT episodes—the same counts as the evaluation sets—and does not show the training and evaluation episodes are disjoint, so the individual-learning improvement could in part reflect memorized costs.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents adapt via individual learning and team evolution","Semi-centralized adaptation improves multi-agent LLM planning","Agents learn costs individually, evolve communication as a team","New method helps LLM agents cooperate in embodied environments"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1415,"prompt_tokens":1039,"completion_tokens":376,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":655,"completion_tokens_details":{"reasoning_tokens":311}},"tokens_in":655,"tokens_out":376,"duration_ms":4752,"temperature":1.0,"reasoning_tokens":311,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:39:12.733499+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the C-WAH and TDW-MAT evaluations with a training/evaluation split guaranteed disjoint—for example, hold out entire room layouts or task types from utility-function training—and compare full LIET against its 'without individual learning' ablation; if the gap shrinks to near zero, the utility gain is memorization of evaluation episodes rather than generalization.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies both benchmarks and the modular decentralized baseline whose evaluation protocol LIET inherits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies a decentralized LLM baseline that models teammate intentions; a main comparison point."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies a centralized multi-turn discussion baseline; LIET positions itself between this and decentralized planners."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces grounding LLM actions with a learned value/affordance function, which motivates the utility module."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides LoRA, the parameter-efficient finetuning method used to adapt the utility model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the reflection loop pattern used by the receiver-side reflector that updates the cooperation knowledge list."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the ThreeDWorld Transport Challenge that TDW-MAT extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Watch-And-Help Challenge that C-WAH extends."}],"review_version":1}