{"id":"f14aca6e-0f7c-4939-9592-cb11d3043893","arxiv_id":"1908.06758","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"An iterative-update and shared-network variant of MADDPG that stabilizes multi-agent training and cuts wall-clock time, with mixed performance gains depending on the task.","lead":"This paper proposes a multi-agent reinforcement learning method that trains agents one at a time while trying to keep the others fixed, and that stores all agents in a single shared neural network. The claimed payoff is steadier learning in environments where agents affect each other, plus faster training and less memory as the number of agents grows.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1's actor update sums over all agents, which contradicts the 'only current agent trains' rule; the value-fixing claim is therefore untested by the reported experiments.","rationale":"The reader's CONDITIONAL verdict is well grounded. My top concern is narrower and more implementation-level than the reader's weakest_assumption: the value-fixing approximation is admitted in §3 and §5, but Algorithm 1 as printed does not even encode the restriction that only the current agent is trained. The actor-update line contains a sum over all agents and no selection of l. If the released code does the same, then the experiments measure a shared-network MADDPG variant, not IUUR, and the performance advantage cannot be attributed to iterative update. This is a load-bearing internal consistency issue rather than merely an approximation issue. The paper's own results support caution: IUUR is worse than MADDPG in the Predator_6-Prey_2 predator-replacement plot, and no error bars are shown for the five-seed runs. However, the authors provide source code, so this concern is testable and potentially fixable. A single diagnostic—masking the actor update and measuring waiting-agent output drift—would settle whether the value-fixing mechanism is real. Because the concern is addressable in revision and the wall-clock speedups of batch computation are plausible independently of the value-fixing mechanism, I do not move the verdict away from CONDITIONAL. I partially agree with the reader: we both identify the value-fixing step as weak, but I locate the failure earlier, in the actor-loss implementation.","tokens_in":9305,"tokens_out":7200,"duration_ms":69843,"concrete_test":"Inspect the released GitHub implementation of Algorithm 1 and determine whether the actor-loss tensor is masked to the current learning agent (l) or summed over all agents. Then, with a fixed batch from the replay buffer, record μ_θ(o_i) for every waiting agent before and after one optimizer step, and compute the L2 norm of ∇_θ J(μ|o_-i) using the paper's o_-i formulation. Repeat across several update steps and learning-agent switches. If waiting-agent outputs move by more than a small tolerance (e.g., more than 1% of the action scale) or the waiting gradient norm is not consistently smaller than the MADDPG baseline's corresponding norm, the value-fixing mechanism is not operational and the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central algorithmic claim is that IUUR fixes waiting agents' policies while training only the current agent, so the shared parameter update reduces nonstationarity. That requires Algorithm 1 to update only the current learning agent's objective, or at least to explicitly suppress the waiting-agent policy gradient. Section 3 states 'only the current particular agent is trained,' and the value-fixing target y_-i = Q'(s,o_-i,a') omits the reward term. But the actor update in Algorithm 1 reads ∇θJ(μ|o_i)≈ (1/NB) Σ_i Σ_j (∇θμ(oi|θ) ∇a_i Q(...)), with an outer sum over all agents i=1..N and no mask selecting the current learning agent l. If the implementation follows this pseudocode, all agents' policies are updated every step and the method degenerates to a shared-network MADDPG; the reported experiments would not test iterative update at all. The paper's own Section 5 admits the value-fixing equation 'cannot strictly hold,' but even the weaker guarantee of a smaller L2 norm for waiting-agent gradients is neither proven nor enforced by the pseudocode, and Section 4 notes IUUR underperforms MADDPG in the Predator_6-Prey_2 predator replacement. Without a clarified or masked actor update, or a measurement of waiting-agent policy drift, the headline performance claim is not attributable to the proposed mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two interlocking ideas for multi-agent reinforcement learning: iterative update (IU), in which only one agent is trained at a time while the policies of the other agents are held fixed, and unified representation (IUUR), in which all agents share a single policy network conditioned on first-person observations. The authors claim that IUUR mitigates nonstationarity, improves performance over MADDPG, reduces memory usage, and decreases wall-clock time, especially as the number of agents grows. The method is evaluated on cooperative (Spread) and mixed cooperative-competitive (Predator-Prey) continuous-control tasks, and the source code is made available online.","tokens_in":9556,"tokens_out":5262,"duration_ms":54551,"significance":"If the central mechanism worked as stated, the paper would make a useful practical contribution: parameter sharing via first-person perspective is a sensible way to reduce memory and computation in MARL, and fixing waiting agents' policies is a principled way to reduce nonstationarity. The availability of source code and the use of an external baseline (MADDPG) rather than fitted constants are strengths. However, the paper's own text and Algorithm 1 are inconsistent about which agents are updated, the value-fixing guarantee is asserted rather than proved or measured, and the headline performance claim is contradicted by one of the reported environments. As it stands, the experiments do not isolate the mechanism the paper claims to test.","major_comments":[{"comment":"The actor update in Algorithm 1 sums over all agents: ∇θJ(μ|oi) ≈ (1/NB) Σ_i Σ_j (∇θμ(oi)∇_ai Q(...)), with no mask selecting the current learning agent l. This directly contradicts Section 3's statement that 'only the current particular agent is trained.' As written, every shared parameter update changes all agents' policies, so the training procedure degenerates to a shared-network version of MADDPG and the reported experiments would not test the iterative-update mechanism. Please correct the pseudocode to update only the current learning agent (or otherwise explicitly suppress waiting-agent gradients), and state clearly which version of the algorithm produced the reported results. If the implementation followed the printed pseudocode, the nonstationarity-related performance claims need to be rerun with the intended masking.","section":"Algorithm 1"},{"comment":"The claimed guarantee in the paragraph beginning 'Luckily we can usually guarantee a smaller L2 norm of its gradient' is not established. Because all agents share the same parameters θ, any gradient step changes μ(o_-i) for waiting agents; a smaller L2 norm of the policy gradient does not imply that the waiting agents' policies are fixed, and no bound on the resulting policy change is given. The inequality is asserted without proof, and Section 5 later concedes that the equation 'cannot strictly hold.' Please provide a derivation under explicit assumptions, or empirically measure waiting-agent policy drift (for example, mean action distance or KL divergence between target policies before and after updates).","section":"Section 3"},{"comment":"The value-fixing target for waiting agents, y_-i = Q'(s, o_-i, a'_1, ..., a'_N), omits the immediate reward r_-i. Since Q is meant to represent a discounted return, regressing Q toward this target does not correspond to a Bellman fixed point for agent -i unless an additional argument is supplied. The paper should define the exact loss for waiting agents and explain why this target is valid; otherwise the value-fixing mechanism is underspecified and the reported training objective is ambiguous.","section":"Section 3"},{"comment":"In the Predator_6-Prey_2 predator-replacement condition, the paper reports that IUUR performs worse than MADDPG (Figure 6(a) and the accompanying text), attributing this to 'update error introduced by unified representation.' This is a direct counterexample to the abstract's unqualified statement that the algorithm 'achieves state-of-the-art performance.' Please qualify the performance claim to the environments and settings where the improvement is observed, and discuss whether the negative result is consistent with the value-fixing approximation or indicates a more fundamental limitation.","section":"Section 4.2.1"}],"minor_comments":[{"comment":"The text 'For each agent's policy ri : S×A1×...×AN → R' reuses the reward notation for a policy; this should be πi.","section":"Section 2"},{"comment":"The target notation y_-i = Q'(s, o_-i, ...) is confusing because Q is defined as Q(s, o_i, a_1, ..., a_N); please clarify which observation argument is used in the critic and targets.","section":"Section 3 and Algorithm 1"},{"comment":"The caption of Figure 6 states 'In (a), IU outperforms MADDPG a lot and IU's performance is worse than that of MADDPG,' which is internally inconsistent; the text indicates that IUUR, not IU, is the worse-performing method in that condition.","section":"Section 4.2.1"},{"comment":"The wall-clock time savings are reported as single numbers without variance, number of runs, or details about batching/GPU utilization; adding this information would make the computational-efficiency claim more robust.","section":"Section 4.2.2"},{"comment":"Only one value of the new hyperparameter K is used (K=5000), with no sensitivity analysis; a short ablation or at least a discussion of how K affects the performance/nonstationarity trade-off would strengthen the paper.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The discrepancy between Algorithm 1's unmasked actor update and the paper's 'only current agent trains' description is the key issue. Because the code is publicly available, the authors should be asked to confirm which implementation generated the curves and to correct the pseudocode and experiments accordingly. The paper's broad performance claim should also be aligned with its own negative result in the Predator_6-Prey_2 predator condition."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take on 1908.06758. The paper's two ideas are (a) train agents one at a time, freezing the others, to reduce nonstationarity, and (b) represent all agents with a single shared network to save memory and compute. The second idea is sound and the wall-clock numbers support it: interaction time drops substantially when agents share weights. The first idea is where it falls apart.\n\nWhat's actually new is the specific 'value fixing' rule: for waiting agents, the critic target is set to the target Q directly, without the reward term, in the hope that their policy gradient becomes small. The paper admits this can't hold strictly, and claims only a smaller L2 norm of the gradient. That's a heuristic, not a guarantee, and the paper provides no measurement of waiting-agent policy drift to back the claim.\n\nThe bigger problem is that Algorithm 1's actor update sums over all agents. Section 3 explicitly says only the current agent is trained, but the pseudocode updates the shared actor using gradients for every agent. If the implementation matches the pseudocode, then all policies change every step, and the method degenerates to shared-network MADDPG with a modified critic target. The reported results would then have nothing to do with iterative update. If the implementation actually masks the gradient, the pseudocode is simply wrong and needs fixing. Either way, the paper as written does not demonstrate the mechanism it claims.\n\nOther soft spots: no error bars or variance shading in the reward curves despite saying five seeds were run; the K hyperparameter is hand-tuned and the paper admits it; in Predator_6-Prey_2 IUUR underperforms MADDPG on the predator side. The comparison set is also thin: no other parameter-sharing baselines, no ablation that turns off the value-fixing target while keeping the shared network.\n\nCredit where due: the idea of using a first-person perspective to let one network serve all agents is a reasonable engineering choice, and the speedup numbers are concrete. The code is available. It's just that the central claim, that iterative update fixes policies under a shared network, is not established by the text.\n\nIf this came to me as an editor, I'd desk reject it in its current form, but I'd invite a resubmission with a corrected algorithm description and a direct measurement of whether waiting-agent policies actually stay fixed. The idea has enough texture to be worth a look, but the current manuscript doesn't support its own headline.","headline":"A useful parameter-sharing speedup, but the paper's iterative-update mechanism is contradicted by its own algorithm and the headline claim is not supported.","tokens_in":10064,"tokens_out":4948,"would_cite":false,"duration_ms":44617,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One shared network trains all agents and beats MADDPG","keywords":["multi-agent reinforcement learning","nonstationarity","iterative update","unified representation","value fixing","MADDPG","batch computation","first-person perspective"],"falsifier":"Run IUUR on Spread_10 and record the action distributions of waiting agents immediately before and after the current agent's update. If the KL divergence between those distributions is large while the stated L2-norm bound on their policy gradient is small, then the value-fixing step does not actually keep waiting agents fixed, and the method's stated mechanism is not what drives the reported gains.","tokens_in":9103,"feed_emoji":"🧠","tokens_out":3336,"duration_ms":34080,"temperature":0.7,"pith_summary":"The paper proposes that multi-agent reinforcement learning can be made both stable and cheap by training one agent at a time while keeping the others fixed, and by representing every agent with a single network. It argues that freezing the waiting agents' policies turns the learning problem into a single-agent one, alleviating the nonstationarity that grows as the number of agents increases. A shared network with a first-person perspective per agent changes policy computation from sequential to batch, so interaction time and memory no longer grow linearly with agent count. On cooperative and mixed cooperative-competitive tasks, the method reports better performance than MADDPG in most comparisons and large wall-clock savings, especially as the number of agents grows. The paper is candid that the value-fixing step only makes the waiting agents' policy gradients small rather than zero.","feed_headline":"One shared network trains all agents and beats MADDPG","feed_subtitle":"Iterative updates freeze waiting agents via a Bellman value-fixing target, easing nonstationarity and cutting wall-clock time.","key_machinery":"The key machinery is the value-fixing target for iterative update inside a unified network. For the current learning agent $i$, the critic target is $y_i = r_i + \\gamma Q'(s, o_i, a'_1, \\dots, a'_N)$ with $a'_j = \\mu'(o_j)$; for waiting agents $-i$, the target is simply $y_{-i} = Q'(s, o_{-i}, a'_1, \\dots, a'_N)$, so no Bellman update is applied to their part of the Q function. This is meant to keep the gradient of the waiting agents' policy objective near zero, fixing their strategies while the current agent improves. The unified representation adds each agent's own observation $o_i$ as a first-person perspective, so one network can produce distinct policies for all agents and enable batch computation.","core_discovery":"The paper claims that a single policy network can represent all agents in a multi-agent system, and that this representation can be trained stably by updating one agent at a time. The central discovery is a value-fixing rule based on the Bellman equation: for the agent currently being trained, the critic target uses the reward plus the discounted target-Q value, while for all other agents the target is taken directly from the target critic network without a Bellman expansion. The authors argue that this makes the policy gradient for the waiting agents small in L2 norm, so their policies stay approximately fixed even though they share the same network weights. This iterative update, combined with the unified one-network representation, is claimed to mitigate nonstationarity and to achieve state-of-the-art performance compared with MADDPG while saving significant wall-clock time, with the advantage growing as the number of agents increases.","pith_inferences":["A natural test the authors leave implicit is to log the L2 norm of the waiting agents' policy gradients during IUUR training; if the norm is not actually smaller than under a Bellman target, the value-fixing premise fails in practice even if results are good.","The learning-frequency hyperparameter $K$ controls the balance between stability and the learning speed of waiting agents; the paper tunes it by experience, suggesting that annealing $K$ or adapting it per environment could widen the performance gains.","The single shared network may act as an implicit regularizer, forcing agents to share features; this could explain the fast convergence in Spread_3 and could be tested by ablating the first-person perspective encoding.","If the value-fixing target is sound, it should port to value-based MARL methods such as VDN and QMIX, where the critic is central and Bellman targets are already well defined."],"forward_implications":["In fully cooperative tasks with many agents, iterative update reduces nonstationarity and outperforms the MADDPG baseline, with the gap growing as the number of agents increases.","In mixed cooperative-competitive tasks, the unified representation matches or beats MADDPG in most comparisons, with the paper noting a few cases where shared-network Q-value error hurts performance.","Wall-clock training time and environment interaction time drop substantially under IUUR, and interaction time no longer grows linearly with the number of agents.","Because policies are still computed from each agent's own observation, the method preserves decentralized execution and distributed deployment.","The paper states that the approach is compatible with other multi-agent algorithms such as IQL, VDN, and QMIX."],"supporting_citations":[{"why":"MADDPG is the baseline algorithm and the actor-critic structure that IUUR extends.","marker":"[13]"},{"why":"Generative adversarial networks inspire the iterative update idea of alternating which agent learns.","marker":"[11]"},{"why":"Continual lifelong learning provides the framing for avoiding catastrophic forgetting in the shared network.","marker":"[12]"},{"why":"DDPG supplies the single-agent deterministic policy gradient method underlying each agent's update.","marker":"[18]"},{"why":"The Bellman equation is the basis of the value-fixing targets for the current learning agent.","marker":"[25]"},{"why":"COMA is contrasted because it fixes other agents' actions for a counterfactual baseline, whereas the paper fixes their strategies.","marker":"[21]"}],"fun_headline_variants":["One shared policy net, iterative updates beat MADDPG","Single network for all agents, faster than MADDPG","Iterative value-fixing target stabilizes shared-net MARL","One agent at a time: shared net beats MADDPG in speed","GAN-style updates with one net outpace MADDPG"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that making the waiting agents' policy-gradient norm small is enough to keep their policies fixed, even though all agents share one network and any parameter update necessarily changes the network's outputs for every agent's observations.","fun_headline_variants_meta":{"raw":{"variants":["One shared policy net, iterative updates beat MADDPG","Single network for all agents, faster than MADDPG","Iterative value-fixing target stabilizes shared-net MARL","One agent at a time: shared net beats MADDPG in speed","GAN-style updates with one net outpace MADDPG"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1245,"prompt_tokens":924,"completion_tokens":321,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":236}},"tokens_in":540,"tokens_out":321,"duration_ms":3814,"temperature":1.0,"reasoning_tokens":236,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:01:43.328778+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run IUUR on Spread_10 and record the action distributions of waiting agents immediately before and after the current agent's update. If the KL divergence between those distributions is large while the stated L2-norm bound on their policy gradient is small, then the value-fixing step does not actually keep waiting agents fixed, and the method's stated mechanism is not what drives the reported gains.","supporting_citations":[{"cited_title":"Multi-agent actor-critic for mixed cooperative-competitive environments","cited_arxiv_id":null,"evidence_quote":"MADDPG is the baseline algorithm and the actor-critic structure that IUUR extends."},{"cited_title":"Continual lifelong learning with neural networks: A review","cited_arxiv_id":null,"evidence_quote":"Continual lifelong learning provides the framing for avoiding catastrophic forgetting in the shared network."},{"cited_title":"A generalized dynamic programming princi- ple and hamilton-jacobi-bellman equation","cited_arxiv_id":null,"evidence_quote":"The Bellman equation is the basis of the value-fixing targets for the current learning agent."},{"cited_title":"Counterfactual multi-agent policy gradients","cited_arxiv_id":null,"evidence_quote":"COMA is contrasted because it fixes other agents' actions for a counterfactual baseline, whereas the paper fixes their strategies."}],"review_version":1}