{"id":"c1f0d217-ca65-4ef5-9b02-49e87a90b7b8","arxiv_id":"2606.07845","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"GRPO on dining-philosophers rollouts yields no significant coordination gain (p=0.66 at five philosophers) and training reward declines after step nine.","lead":"The paper tests frontier and open LLMs on the dining philosophers coordination task and then applies GRPO training on task rollouts. It reports no statistically significant improvement from GRPO and identifies reward degeneracy and early training peak as issues.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Reward function admits degenerate zero-action maximum, so non-significant GRPO result may reflect optimization to inaction rather than coordination limits.","rationale":"The reader's weakest_assumption directly isolates the same reward degeneracy that undercuts the interpretation of the GRPO result. No stronger internal inconsistency (e.g., in the t-test arithmetic or model counts) is visible from the supplied text; the reward issue is the load-bearing assumption for the headline claim.","tokens_in":1789,"tokens_out":386,"duration_ms":8113,"concrete_test":"Replace the four-term reward with an augmented version that adds a positive term proportional to total meals served (or subtracts a fixed penalty for zero-action episodes) while keeping all other experimental settings identical; recompute the Welch t-test and Hedges' g on the new per-episode rewards at five philosophers. If the p-value drops below 0.05 or |g| exceeds 0.3, the original non-significance is an artifact of the degenerate maximum.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that per-episode reward faithfully tracks coordination quality. The four-term reward (explicitly noted in the abstract) reaches its global maximum when all agents output zero actions, yielding reward 1.0 for DeepSeek-R1-Distill-Qwen-7B and 0.83 for Mistral-Small at five philosophers with zero meals served. GRPO training on this reward peaked at step 9 then declined, and the reported t-test (p=0.66, g=-0.11) compares policies that may both be near this degenerate fixed point. If the reward surface contains an attractive basin at the no-op policy, absence of improvement under GRPO does not demonstrate that coordination is unlearnable; it demonstrates only that the chosen scalarization does not penalize inaction.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper evaluates LLMs on multi-agent coordination via the dining philosophers problem across 630 episodes with seven models and three philosopher counts. Frontier closed-source models achieve mean rewards 0.45–0.87 while Mistral-Small reaches 0.83–0.99 and Qwen3-14B reaches 0.13–0.35. It then applies GRPO on task rollouts and reports no statistically significant improvement (Welch t-test p=0.66, Hedges’ g=-0.11 at five philosophers; similar at ten and fifteen). The paper notes that training rewards peak at step nine then decline and that the four-term reward admits a degenerate maximum at zero actions (reached by some models at zero meals served). It concludes that the bottleneck for open-weight models is reward shaping and checkpoint discipline rather than compute.","tokens_in":1971,"tokens_out":712,"duration_ms":19389,"significance":"If the central empirical result holds after addressing the reward issue, the work supplies a clean, reproducible test-bed for multi-agent coordination, per-episode statistics, effect sizes, and explicit acknowledgment of a degenerate reward maximum. These elements provide a useful negative result on GRPO and point toward concrete methodological improvements (reward shaping, curriculum, checkpoint selection) that future work can build on.","major_comments":[{"comment":"The four-term reward (explicitly described in the abstract) reaches its global maximum when all agents output zero actions, yielding reward 1.0 for DeepSeek-R1-Distill-Qwen-7B and 0.83 for Mistral-Small at five philosophers with zero meals served. Because both the GRPO-trained and baseline policies may lie near this attractive fixed point, the reported non-significant difference (p=0.66, g=-0.11) does not yet demonstrate that coordination is unlearnable under GRPO; it only shows that the chosen scalarization does not penalize inaction.","section":"Abstract and reward-function description"},{"comment":"The training curves for the 8B and 14B GRPO runs peak at step nine and then decline, so the default checkpoint saved at step 15 is strictly inferior to several earlier checkpoints. This choice of final checkpoint directly affects the comparison used to support the claim that GRPO cannot close the coordination gap.","section":"GRPO training results and checkpoint selection"},{"comment":"The central claim that GRPO does not close the gap rests on per-episode reward as the primary measure of coordination quality. Because the reward admits a degenerate maximum at the no-op policy, the manuscript should report auxiliary metrics (meals served, deadlock rate, or action entropy) for both baseline and GRPO runs to confirm that high-reward trajectories actually reflect coordination rather than inaction.","section":"Evaluation protocol and results tables"}],"minor_comments":[{"comment":"The abstract refers to “four frontier closed-source systems” without naming them; listing the exact models would improve reproducibility and context.","section":"Abstract"},{"comment":"Ensure every reported statistical test is accompanied by the exact sample size (number of episodes) and the precise definition of the reward terms so readers can replicate the degenerate-maximum observation.","section":"Methods and results"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the careful and constructive review. We agree that the reward degeneracy and checkpoint selection warrant further clarification and auxiliary metrics, which we will incorporate. Our responses below address each major comment directly while preserving the manuscript's reported findings.","responses":[{"response":"We thank the referee for this point. The manuscript already explicitly states that the four-term reward admits a degenerate maximum at zero actions, with the cited models achieving the reported rewards at zero meals served. The central empirical finding is that GRPO produces no statistically significant improvement over baseline under this reward; the conclusion frames this as evidence that reward shaping (rather than compute) is the bottleneck. We will revise the abstract and discussion to more explicitly caveat that the result is tied to the current scalarization and does not claim coordination is unlearnable under GRPO with improved rewards.","revision_made":"partial","referee_comment":"[Abstract and reward-function description] The four-term reward (explicitly described in the abstract) reaches its global maximum when all agents output zero actions, yielding reward 1.0 for DeepSeek-R1-Distill-Qwen-7B and 0.83 for Mistral-Small at five philosophers with zero meals served. Because both the GRPO-trained and baseline policies may lie near this attractive fixed point, the reported non-significant difference (p=0.66, g=-0.11) does not yet demonstrate that coordination is unlearnable under GRPO; it only shows that the chosen scalarization does not penalize inaction."},{"response":"The manuscript already reports that training reward peaked at step nine and declined, explicitly noting that the default checkpoint at step 15 is inferior to earlier ones. The default was used to reflect standard practice without post-hoc selection. We will add results from the best checkpoint (step nine) alongside the default in the revised results section and discuss implications for checkpoint discipline.","revision_made":"yes","referee_comment":"[GRPO training results and checkpoint selection] The training curves for the 8B and 14B GRPO runs peak at step nine and then decline, so the default checkpoint saved at step 15 is strictly inferior to several earlier checkpoints. This choice of final checkpoint directly affects the comparison used to support the claim that GRPO cannot close the coordination gap."},{"response":"We agree that auxiliary metrics are required to interpret results given the reward structure. The manuscript already notes zero meals served for the high-reward no-op cases. We will expand the evaluation tables to report meals served, deadlock rate, and action entropy for both baseline and GRPO runs across all philosopher counts.","revision_made":"yes","referee_comment":"[Evaluation protocol and results tables] The central claim that GRPO does not close the gap rests on per-episode reward as the primary measure of coordination quality. Because the reward admits a degenerate maximum at the no-op policy, the manuscript should report auxiliary metrics (meals served, deadlock rate, or action entropy) for both baseline and GRPO runs to confirm that high-reward trajectories actually reflect coordination rather than inaction."}],"tokens_in":1667,"tokens_out":670,"duration_ms":31753,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to know is that GRPO does not improve performance on the dining philosophers coordination task according to their tests, but the reward function's degeneracy at zero actions makes it hard to draw strong conclusions about coordination itself.\n\nThe paper supplies concrete data across seven models and three problem sizes, with 630 episodes total. It includes statistical tests like Welch's t-test with p=0.66 and Hedges' g of -0.11 at five philosophers, plus observations that training reward peaks early and declines. They also highlight how some models achieve high reward scores with zero meals served, which matches the degenerate case they describe. This level of detail on both the negative result and the caveats is useful.\n\nThe weakness is that the four-term reward admits a maximum when agents take no actions, so the policies being compared may both be near that fixed point. The non-significant GRPO result then reflects the reward surface rather than an inherent limit on learning coordination. Since the paper itself flags this, it's not hidden, but it does mean the evidence for the training methodology bottleneck is weaker than the title suggests.\n\nThis is relevant for researchers working on multi-agent LLM training and post-training methods. It serves as a cautionary example on reward shaping and checkpointing. The work shows clear thinking in reporting its own limitations.\n\nI would recommend sending it for peer review after they consider adding a non-degenerate metric, because the setup is reproducible and the observations are worth discussing.","headline":"GRPO shows no gain on this task but the degenerate reward undercuts what that means for coordination.","tokens_in":2435,"tokens_out":364,"would_cite":false,"duration_ms":20711,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"GRPO applied to task rollouts does not close the multi-agent coordination gap in LLMs","keywords":["multi-agent coordination","dining philosophers","GRPO","large language models","reward shaping","reinforcement learning","coordination gap"],"falsifier":"A replication that applies GRPO with a modified reward preventing zero-action maxima and records a statistically significant reward increase on the same philosopher counts.","tokens_in":2701,"feed_emoji":"📉","tokens_out":658,"duration_ms":19686,"temperature":0.7,"pith_summary":"The paper tests whether group relative policy optimization improves how large language models coordinate when multiple instances share one resource. It uses the dining philosophers problem to run 630 episodes across seven models and three agent counts. GRPO produces no statistically significant reward gains, with a Welch's t-test at five philosophers returning p = 0.66 and a small negative effect size. Rewards in the 8B and 14B runs peak at step nine then fall, and the reward function reaches high values when agents take zero actions. The authors conclude that open-weight models need changes in reward design, checkpoint choice, and scaling order rather than additional training compute.","feed_headline":"GRPO leaves LLM multi-agent coordination unchanged","feed_subtitle":"Tests on the dining philosophers problem find no significant reward gains after training, with rewards peaking early and permitting zero-act","key_machinery":"The dining philosophers problem as a test bed for multi-agent coordination, evaluated with a four-term reward function before and after GRPO training on task rollouts.","core_discovery":"GRPO on rollouts from the dining philosophers task yields no statistically significant increase in per-episode reward for 8B and 14B models at five, ten, or fifteen philosophers. A Welch's t-test at five philosophers gives p = 0.66 and Hedges' g of -0.11. Training reward for both sizes peaks at step nine and declines by the default step-15 checkpoint. The four-term reward admits a degenerate maximum at zero actions, which some models reach with mean reward near 1.0 while serving zero meals.","pith_inferences":["A reward that explicitly subtracts for inaction could allow GRPO to produce coordination gains.","Training on a curriculum that increases the number of philosophers gradually might reduce the scaling problem.","The same reward and checkpoint issues may limit GRPO on other multi-agent benchmarks."],"forward_implications":["GRPO produces no statistically significant change in per-episode reward at five, ten, or fifteen philosophers.","Training reward for both 8B and 14B models peaks at step nine and declines by step fifteen.","High reward values can be reached with zero meals served, indicating a degenerate solution.","The bottleneck for open-weight 14B models is training methodology rather than compute."],"fun_headline_variants":["GRPO yields no gains in LLM multi-agent coordination","GRPO training leaves coordination gap unchanged","No significant reward boost from GRPO in agent tests","GRPO ineffective for closing multi-agent LLM gaps","Coordination unchanged after GRPO on philosopher task"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The four-term reward function faithfully measures coordination quality rather than allowing a high score from the degenerate choice of taking no actions.","fun_headline_variants_meta":{"raw":{"variants":["GRPO yields no gains in LLM multi-agent coordination","GRPO training leaves coordination gap unchanged","No significant reward boost from GRPO in agent tests","GRPO ineffective for closing multi-agent LLM gaps","Coordination unchanged after GRPO on philosopher task"]},"model":"grok-4.3","cost_usd":0.002414,"raw_usage":{"total_tokens":1461,"prompt_tokens":776,"num_sources_used":0,"completion_tokens":69,"cost_in_usd_ticks":24137000,"prompt_tokens_details":{"text_tokens":776,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":616,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":776,"tokens_out":69,"duration_ms":5794,"temperature":1.0,"reasoning_tokens":616,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T19:59:09.643764+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A replication that applies GRPO with a modified reward preventing zero-action maxima and records a statistically significant reward increase on the same philosopher counts.","supporting_citations":[],"review_version":1}