{"id":"4714702c-f739-4c38-81d5-745ebd63e3d4","arxiv_id":"2507.16382","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLM-generated, iteratively refined reward functions guide multi-agent PPO to 100% success in formation control with collision avoidance in a three-agent benchmark.","lead":"The paper uses a large language model to write and repeatedly improve the reward function for training robot swarms to keep formation while dodging obstacles. The authors claim this needs fewer training cycles and beats human-designed rewards in simulations and on real robots.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed superiority over human-designed rewards (Table II) is confounded: the LLM method receives iterative reward rewrites driven by the reported success metrics (Algorithm 2, Fig. 4), while the human baseline uses a static reward.","rationale":"I agree with the reader that the adaptive tuning budget is the weakest link. The paper's central claim is the framework's superiority over human design, and the comparison is the evidence. The LLM's reward is rewritten multiple times with access to the very metrics being optimized, while the baseline is static. This is a textbook case of an unfair comparison, and the paper provides no control. I also note the stop-at-100% issue and the small sample, which compound the problem. However, this is fixable: the paper could introduce a tuned human baseline or an automatic weight-search baseline, and report variance. The method itself (LLM generating reward code, then refining based on metrics) is plausible and the paper provides a useful demonstration, so I would not reject outright. The reader's CONDITIONAL verdict stands.","tokens_in":9716,"tokens_out":5465,"duration_ms":58795,"concrete_test":"Re-run the experiment with a tuned human baseline: take the human-designed reward of [10] and pass it through the same iterative refinement loop of Algorithm 2 (N=3) with the same evaluation metrics and PPO protocol, allowing a human or the same LLM to adjust its parameters/terms. If the tuned baseline reaches statistically indistinguishable success rate, time, and formation error (e.g., via McNemar or bootstrap test across the 3x300 episodes), then the Table II advantage is due to the tuning budget, not LLM reward design. Report per-seed results. Additionally, verify Table I's 100% success by running the final policy for at least 100 episodes and reporting the 95% confidence interval.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V.A and Algorithm 2 implement an adaptive loop: after each training run, task-level metrics (success rate, hazard incidents, formation error, total time, average acceleration) are fed back to the LLM, which rewrites the reward function. This is repeated until the success rate reaches 100% (Table I: iterations 0-3). The comparison in Table II then pits this multi-round-tuned reward against a fixed human-designed reward from [10] (success 93%, time 14.5s, formation error 37.4) and ORCA-F. The human baseline is never given an equivalent refinement budget. Fig. 4 shows the LLM making targeted weight/penalty adjustments (e.g., 'increased the penalty for collisions', 'increased the penalty for formation errors') based on those metrics; a human designer or a simple automatic weight-search could apply the same fixes. Thus the observed advantage (95% vs 93% success, 11.5 vs 14.5s, 26.7 vs 37.4 formation error) may reflect the adaptive protocol, not the LLM's reward-generation ability. Moreover, iteration 3's 100% success is a stopping condition, not a steady-state estimate, and is based on 20 evaluation episodes; Table II's numbers are averages over 3 seeds x 300 episodes, but no variance or significance test is reported. The central comparative claim therefore rests on an apples-to-oranges comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for multi-agent formation control with collision avoidance (FCCA) in which a large language model (LLM, specifically Qwen2.5-72B) generates the initial reward function for a multi-agent PPO policy and then iteratively rewrites that reward based on high-level task metrics (success rate, hazard incidents, formation error, total time, average acceleration) rather than on raw reward magnitudes. The authors claim that after three refinement iterations the method reaches 100% success in simulation, outperforms a human-designed reward baseline and ORCA-F in success rate, average time, and formation error (Table II), and transfers to Gazebo simulation and real-world robots. The paper also includes a qualitative real-world deployment with Mecanum-wheel robots using OptiTrack localization and Jetson AGX Orin compute.","tokens_in":10056,"tokens_out":5878,"duration_ms":58526,"significance":"If the stated results held, this would be a useful incremental contribution to automated reward design for multi-agent RL: the idea of closing the reward-tuning loop with task-level metrics is sensible and has practical appeal, and the paper demonstrates a plausible pipeline from LLM-generated reward to trained deployment. The main strengths are the concrete algorithm descriptions and the inclusion of both simulation and physical-robot experiments. However, the empirical support for the central comparative claim is currently weak: the LLM method is given an iterative refinement budget while the human-designed baseline is static, no variance or significance measures are reported, the ORCA-F baseline is mis-cited, and the real-world validation is qualitative. These issues are fixable within the scope of the manuscript, which is why I recommend major revision rather than rejection.","major_comments":[{"comment":"The central comparative claim that the LLM-guided reward outperforms human-designed rewards is confounded by an unequal adaptation budget. The LLM method runs Algorithm 2, in which the reward function is rewritten multiple times based on the very evaluation metrics (success rate, formation error, total time) that are then reported in Table II, while the human-designed baseline [10] is trained with a static reward. The observed differences (95% vs 93% success, 11.5 s vs 14.5 s, 26.7 vs 37.4 formation error) may therefore reflect the iterative tuning protocol rather than the LLM's reward-generation ability. Please add a matched baseline in which the human-designed reward (or an automated weight-search baseline) receives the same iterative refinement budget on the same metrics, or report the best single-shot LLM reward without online tuning.","section":"Section V.A, Table II, Algorithm 2"},{"comment":"No measure of variance is reported for any of the headline results. Table I reports averages over only 20 evaluation episodes per iteration, and Table II reports averages over 3 seeds x 300 episodes without standard deviations or significance tests. With 900 episodes, the 2-point success-rate difference between LLM (95%) and human-designed (93%) is not statistically significant (a two-proportion z-test gives p ≈ 0.07), and the 100% figure in Table I is a stopping condition, not a steady-state estimate. Please report per-seed results, standard errors, and significance tests, and evaluate the final policy on a held-out set of obstacle configurations.","section":"Section V.A, Tables I and II"},{"comment":"The baseline labeled ORCA-F is cited to [24], which is Chen et al., 'Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning' (ICRA 2017), not an ORCA-family method. This citation mismatch makes it impossible to verify the baseline implementation. Please provide the correct reference for ORCA-F (e.g., the original ORCA paper or an ORCA-based formation controller), and describe the baseline's parameters and how it was adapted to the formation-control task.","section":"Section V.A, Table II, reference [24]"},{"comment":"The real-world validation is entirely qualitative. Figure 7 shows snapshots, but the paper reports no quantitative measures for the physical experiments, such as number of trials, success rate, minimum distance to obstacles, formation error over time, or completion time. The claim that the approach is 'validated in real-world settings' is therefore unsupported. Please report repeated real-world runs with the same metrics used in simulation.","section":"Section V.B"},{"comment":"The evaluation metrics used for reporting are fed back into the reward-refinement loop (Algorithm 2 lines 3-5), so the reported success rate and formation error are fitted quantities, not independent estimates. The LLM is explicitly instructed to adjust the reward to improve these metrics (Fig. 4), which makes the later-iteration numbers a form of test-set optimization. At minimum, the final evaluation should be performed on a separate held-out set of environments and obstacle trajectories that were never used in the feedback loop.","section":"Section IV.C, Algorithm 2"}],"minor_comments":[{"comment":"The abstract contains a grammatical fragment ('However, when confronted with the complex objective of Formation Control with Collision Avoidance (FCCA): designing an effective reward function ...') and should be rewritten for clarity.","section":"Abstract, Section I"},{"comment":"The TD-error definition has an index error: δ_t = r_t + γV(s_{t+1}) − V(s_t) is standard; the printed V(s_{t−1}) appears to be a typo. Also, the symbol r_t is used both for the reward at time t and the probability ratio in Eq. (8); please disambiguate.","section":"Section III.B, Eq. (5)"},{"comment":"The input 'policy summary' used to update the prompt (line 4) is never defined; please specify what information it contains and how it is generated.","section":"Algorithm 2"},{"comment":"The reward-curve comparison is hard to interpret because the LLM reward function changes at each iteration, and rewards from different reward functions are not on a common scale; please plot a task-level metric or normalize the curves.","section":"Section V.A, Fig. 5"},{"comment":"The sentence order implies secondary objectives are only considered after primary ones are achieved, but Algorithm 1 uses a convergence threshold η that is never given a numerical value; please specify η and the termination criterion for each iteration.","section":"Section IV.B, Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reports an incremental but potentially useful extension of LLM-based reward tuning to multi-agent formation control. Its main weakness is the unmatched comparison and lack of statistical rigor, both of which are fixable within the paper's scope. I would not reject outright, but the authors need to either weaken the comparative claims or provide a matched-budget baseline. Also, the citation mismatch for ORCA-F should be corrected before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper applies an established idea — LLM-generated reward functions (Eureka, Text2Reward) — to a new setting: multi-agent formation control with collision avoidance. The genuinely new bit is that they refine the reward online using task-level evaluation metrics (success rate, formation error, hazard incidents) rather than raw reward magnitude, which is a sensible fix for a known failure mode in Eureka-style feedback. The method is clearly described, and the authors are upfront about limitations in the conclusion.\n\nThe soft spot is the comparison. In Algorithm 2, the LLM rewrites the reward for up to three iterations based on exactly the metrics that Table II reports. The human-designed baseline is trained once with a static reward. So Table II's advantage (95% vs 93% success, 11.5 vs 14.5s, 26.7 vs 37.4 formation error) could just as easily come from the extra tuning budget as from the LLM's reward-generation skill. The stress-test note has this right; the paper does not address it. The 100% success in Table I is a stopping condition, not a steady-state estimate, and it is averaged over only 20 episodes. Table II reports no variance or significance tests. The baseline reward is never described, and the ORCA-F citation (ref [24]) points to a deep-RL collision avoidance paper, not ORCA-F. Real-world validation is qualitative, with only snapshots.\n\nNone of this makes the method useless. If the authors release the prompts and code, describe the baseline reward, and run a fair comparison where the human-designed reward gets the same iterative tuning budget (or a simple automatic weight search), the result would be a useful engineering contribution. The authors are honest about the limitations of LLM-based reward design losing track of earlier objectives, which is more than many papers in this space do. The novelty is modest but real.\n\nWho this is for: robotics practitioners working on MAS reward design and people building on Eureka-style workflows. It does not change theory, but it is a reasonable data point. I would cite it if the comparison is fixed. I would send it to peer review — it deserves referee time, but the central claim needs substantial revision. The paper shows clear thinking and honest engagement with prior work; the issues are experimental rigor, not incoherence.","headline":"Sensible incremental extension of LLM-generated rewards to multi-agent formation control, but the headline comparison is confounded by an adaptive refinement loop that the human baseline does not get.","tokens_in":10534,"tokens_out":1854,"would_cite":false,"duration_ms":19229,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM that writes and rewrites reward functions drives a multi-agent formation-control policy to 100% success.","keywords":["LLM-guided reward design","multi-agent reinforcement learning","formation control","collision avoidance","reward shaping","PPO","sim-to-real transfer","task-level evaluation metrics"],"falsifier":"Train the human-designed reward function from [10] under the same protocol as Algorithm 2—same metrics, same number of rewrites, same training budget—and compare final success rates. If the human-tuned reward also reaches 100% success with comparable formation error, the LLM's specific role is not established. Alternatively, freeze the LLM's prompt after the first rewrite and check whether performance stops improving.","tokens_in":9509,"feed_emoji":"🤖","tokens_out":5850,"duration_ms":53921,"temperature":0.7,"pith_summary":"This paper argues that the hardest part of multi-agent reinforcement learning for formation control with collision avoidance—designing a reward function that balances several conflicting objectives—can be handed to a large language model. The LLM writes an initial reward from a plain-language task description and the agents' observation structure, and then, after each training round, it receives five task-level evaluation metrics (success rate, hazard incidents, formation error, total time, average acceleration) and rewrites the reward accordingly. In three refinement iterations the success rate rises from 60% to 100% in a dense obstacle environment, and the final policy outperforms both a classical reciprocal-velocity-obstacle baseline and a human-designed reward in success rate, average time, and formation error. The authors also show the learned policy transfers to a Gazebo simulation and to physical robots.","feed_headline":"LLM-written rewards push robot formation control to 100% success","feed_subtitle":"A language model rewrites the reward each round from success metrics, beating hand-tuned rewards in dynamic obstacle fields.","key_machinery":"The carrying mechanism is a closed-loop reward-design loop. The LLM is given a prompt containing the agent observation structure (own state, obstacle states, and communication-derived formation information encoded through the symmetric normalized Laplacian of the agent graph), a task list with priorities, and tips; it outputs an executable reward function. After PPO training with GAE, the policy is evaluated on five objective-level metrics, and those metrics—not the reward—are appended to the prompt so the LLM rewrites the reward. This design is intended to avoid the local-optima trap of feeding reward magnitudes back, which the authors attribute to prior one-shot LLM reward generators.","core_discovery":"The paper's central claim is that LLM-guided online reward refinement, driven by high-level task metrics rather than raw reward magnitudes, lets a multi-agent PPO policy solve formation control with collision avoidance in dynamic environments more efficiently than reward functions written once by a human. The authors demonstrate this with three iterations of the loop in which the LLM produces a reward function, the policy is trained until its loss converges, five task-level metrics are evaluated, and those metrics are fed back for the next rewrite. The final iteration reaches a 100% success rate across evaluation episodes, with formation error dropping from 74.6 to 27.2, while the comparison methods ORCA-F and the human-designed baseline reach 79% and 93% success respectively. The method's practical value is supported by sim-to-sim and sim-to-real deployment on Mecanum-wheel robots.","pith_inferences":["An implication the authors leave implicit: the same metric-driven LLM loop should apply to other MARL tasks with conflicting objectives (e.g., coverage, tracking, or sorting), where only the observation class and the metric list would need to change.","A fairer test of the LLM's contribution would give the human-designed reward the same adaptive retraining budget; until then, the advantage over the human baseline is partly a comparison between adaptive and static tuning.","If the success plateau at iteration 3 holds across random seeds, a natural next experiment is to test whether the LLM's refinements generalize to unseen obstacle configurations, which the current evaluation does not report."],"forward_implications":["If the central claim holds, multi-objective MAS tasks like formation control no longer require laborious manual reward shaping; an LLM can start from a simple objective and refine the reward on its own.","Because the LLM is used only during training, the deployed policy carries no extra inference cost, so the approach fits real-time robot control.","The reported 100% success rate, after only three rewrites, suggests the metric-based feedback loop converges faster than training with a fixed reward.","The successful sim-to-real transfer indicates the reward loop produces policies that behave sensibly on physical hardware, not just in simulation."],"supporting_citations":[{"why":"Supplies the PPO algorithm that trains the policy with the LLM-generated reward.","marker":"[3]"},{"why":"The prior LLM-reward-design method whose reward-magnitude feedback the authors identify as prone to local optima.","marker":"[15]"},{"why":"Another prior LLM-reward-shaping approach that the method extends by adding online refinement.","marker":"[14]"},{"why":"Provides the human-designed-reward multi-agent RL baseline that the LLM method is compared against.","marker":"[10]"},{"why":"Supplies the symmetric normalized Laplacian representation used to compute formation error.","marker":"[20]"},{"why":"The specific LLM (Qwen2.5-72B) used to generate and rewrite reward functions.","marker":"[23]"},{"why":"Cited as the ORCA-F comparison baseline in Table II.","marker":"[24]"}],"fun_headline_variants":["LLM rewrite loop boosts formation robot success to 100%","LLM-tuned rewards hit 100% success in robot formation control","Dynamic reward crafting by LLM beats hand-tuned in formation tests","LLM-driven reward updates perfect robot formation and avoidance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that LLM-guided rewards outperform human-designed ones rests on comparing an adaptively rewritten reward against a static baseline; if the human reward were given the same iterative tuning on the same metrics, the reported advantage could disappear.","fun_headline_variants_meta":{"raw":{"variants":["LLM rewrite loop boosts formation robot success to 100%","LLM-tuned rewards hit 100% success in robot formation control","Dynamic reward crafting by LLM beats hand-tuned in formation tests","LLM-driven reward updates perfect robot formation and avoidance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1312,"prompt_tokens":895,"completion_tokens":417,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":345}},"tokens_in":511,"tokens_out":417,"duration_ms":3918,"temperature":1.0,"reasoning_tokens":345,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:10:47.058896+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the human-designed reward function from [10] under the same protocol as Algorithm 2—same metrics, same number of rewrites, same training budget—and compare final success rates. If the human-tuned reward also reaches 100% success with comparable formation error, the LLM's specific role is not established. Alternatively, freeze the LLM's prompt after the first rewrite and check whether performance stops improving.","supporting_citations":[{"cited_title":"Text2Reward: Reward Shaping with Language Models for Reinforcement Learning,","cited_arxiv_id":null,"evidence_quote":"Another prior LLM-reward-shaping approach that the method extends by adding online refinement."},{"cited_title":"Relative distributed formation and obstacle avoidance with multi- agent reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Provides the human-designed-reward multi-agent RL baseline that the LLM method is compared against."},{"cited_title":"Distributed swarm trajectory optimization for formation flight in dense environments,","cited_arxiv_id":null,"evidence_quote":"Supplies the symmetric normalized Laplacian representation used to compute formation error."},{"cited_title":"Qwen2.5 Technical Report,","cited_arxiv_id":null,"evidence_quote":"The specific LLM (Qwen2.5-72B) used to generate and rewrite reward functions."},{"cited_title":"Decentralized non- communicating multiagent collision avoidance with deep reinforce- ment learning,","cited_arxiv_id":null,"evidence_quote":"Cited as the ORCA-F comparison baseline in Table II."}],"review_version":1}