{"id":"4a86968a-34a3-4cb0-81e7-d5ac01dc96b7","arxiv_id":"2607.18597","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Sampling a 'default action' from an agent's replay buffer as a counterfactual baseline gives unbiased policy gradients and strong empirical results in continuous cooperative control.","lead":"SAFE is a multi-agent RL method that builds a counterfactual baseline by replacing an agent's action with a random action sampled from its own replay buffer, making credit assignment work in continuous action spaces. The authors report large collision-rate reductions in cooperative highway driving compared with discrete-action baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unbiasedness proof for Eq. (9) does not cover the deterministic policy gradient with shared actor parameters: ∇_θ b_i ≠ 0, so the baseline can bias the update and the stated convergence guarantee is unsupported.","rationale":"The reader's weakest assumption points to the hand-waved σ→0 limit and the inapplicable Konda & Tsitsiklis theorem. I agree those are problems, but the more decisive issue is the shared-actor parameterization. Because all agents share θ, a^{-i} = π_θ(τ_{-i}) makes b_i a function of θ. The deterministic policy gradient of the baseline is then nonzero unless explicitly detached. Lemma 0.1 never addresses this; its stochastic-PG cancellation cannot be ported to DPG. This is a direct internal inconsistency, not just a missing proof step. Nevertheless, the empirical contribution may stand, so the paper should be conditionally accepted with required clarifications/repairs. This matches the reader's CONDITIONAL verdict, so no verdict change.","tokens_in":10148,"tokens_out":13207,"duration_ms":145396,"concrete_test":"Independently derive ∇_θ b_i for b_i = Q(s, a^{-i}, \\bar{a}^i) with shared parameter θ and deterministic actions a_i = π_θ(τ_i). If ∇_θ b_i ≠ 0, run SAFE on a two-agent LQR task with and without stop-gradient on the baseline; if the learning curves or final rewards differ beyond seed variance, the baseline is not inert. Report the norm of ∇_θ b_i over training.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the counterfactual baseline b_i = Q(s, a^{-i}, \\bar{a}^i(t)) in Eq. (9) is unbiased for the deterministic policy gradient and ensures convergence. Lemma 0.1 proves the baseline term vanishes only for a stochastic policy gradient, using the identity E[∇_θ log μ_i] = 0 for each agent. The transfer to the deterministic policy gradient via the σ→0 limit in Eq. (10) is not valid: the deterministic policy gradient is not obtained by taking the limit of the stochastic policy gradient in a way that preserves this cancellation. More seriously, the algorithm shares actor parameters θ. The baseline depends on the joint action a, including the other agents' actions a^{-i}, and since a^{-i} = π_θ(τ^{-i}), b_i is a function of θ. In the deterministic actor update, ∇_θ b_i = Σ_{j≠i} ∇_θ π_θ(τ_j) ∇_{a_j} Q(s, a^{-i}, \\bar{a}^i) ≠ 0 in general. Unless the implementation explicitly detaches the baseline (stop-gradient) or uses independent actor parameters, the baseline contributes a nonzero term to the gradient and therefore biases the update. The paper never specifies such detachment. Even if the baseline were unbiased, the cited convergence result (Konda & Tsitsiklis 1999) is for on-policy actor-critic with linear function approximation, not for the off-policy deep actor-critic with replay and shared parameters used here. Hence the central theoretical guarantee is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAFE, a counterfactual baseline for continuous-action cooperative MARL. Rather than computing a COMA-style expectation over discrete actions, SAFE replaces agent i's action by a 'default action' sampled uniformly from that agent's replay buffer: b_i = Q(s, a^{-i}, \\bar{a}^i) (Eq. 9). The authors claim that this baseline introduces no bias into the deterministic policy gradient and thereby ensures convergence to local optima (Lemma 0.1), and they support the method with experiments on Highway-Env cooperative driving tasks, where SAFE outperforms VDN, QMIX, IQL, COMA, and MAPPO. The paper also includes a parameter study on the number of sampled default actions and ablations comparing against a zero default action, a batch-mean default, continuous COMA, and a plain centralized critic.","tokens_in":10509,"tokens_out":6614,"duration_ms":95081,"significance":"The core idea is attractive: a replay-buffer-sampled default action is task-agnostic, requires no extra simulations or reward models, and naturally extends to continuous action spaces. If the unbiasedness and convergence claims were correct, this would be a meaningful step beyond COMA for continuous-action cooperative tasks. The empirical results on 2V1O through 7V2O are also suggestive of practical utility. However, the theoretical justification in Lemma 0.1 does not cover the actual deep off-policy deterministic actor-critic with shared parameters that the paper implements. The main novelty therefore rests on a proof that is not valid as stated, and the paper does not specify the exact update rule used in the experiments, leaving a gap between theory and implementation. The empirical study is well structured, but the central theoretical claim needs substantial reworking.","major_comments":[{"comment":"Lemma 0.1 proves unbiasedness for a stochastic policy gradient using the identity E[∇θ log μ_i] = 0, and then transfers to deterministic policies through the informal limit σ→0 in Eq. (10). This transfer is not valid. In the implemented algorithm the baseline depends on θ through the other agents' actions a^{-i} = π_θ(τ^{-i}) and, in the shared-parameter setting, through the actor parameters themselves. Consequently ∇θ b_i ≠ 0 in general, and subtracting b_i from Q in a deterministic actor update biases the gradient unless the baseline is explicitly detached (no stop-gradient is mentioned). The proof also assumes b_i is not a function of a_i, but for the deterministic policy a_i = π_θ(τ_i), so the chain rule introduces additional terms. The paper must state the actual update rule and prove unbiasedness for that rule, or substantially weaken the claim.","section":"Algorithm, Eq. (9)-(13), Lemma 0.1"},{"comment":"The convergence to local optima is attributed to Konda & Tsitsiklis (1999). That theorem covers on-policy actor-critic with linear function approximation, not the deep off-policy actor-critic with replay buffer and shared actor parameters used in this paper. Moreover, Eq. (14) only asserts that the gradient norm tends to zero; this is not by itself a statement about convergence to a local optimum of the objective, especially under the non-stationary, off-policy update used in deep MARL. The convergence claim in the abstract and Lemma 0.1 is therefore unsupported by the cited theorem. A valid convergence argument for the actual algorithm is needed, or the claim should be replaced by an empirical statement.","section":"Algorithm, Eq. (14)"},{"comment":"The paper never specifies the deterministic actor update used in the experiments. Eq. (4) is the single-agent deterministic policy gradient ∇θ Q(s, π_θ(s)), while Eq. (11) defines a stochastic policy gradient with an advantage A_i. The experiments presumably use some multi-agent variant of DDPG with the counterfactual baseline, but the connection between the theory and the implementation is missing. In particular, if the actor minimizes a loss such as -Σ_i Q(s, a) + λ Σ_i b_i, the effect of the baseline depends on the precise loss and on whether gradients flow through the baseline. Without this specification, the empirical results cannot be interpreted as validating the proposed unbiasedness mechanism. This gap is load-bearing because the central claim concerns the deterministic policy gradient.","section":"Algorithm and Experimental Setup"},{"comment":"All empirical comparisons are based on three independent runs and no error bars or confidence intervals are reported. Given that the main quantitative claims are comparative ('consistently outperforms', '97% lower collision rate'), the absence of variance information makes it difficult to assess statistical reliability, especially in the 7V2O scenario where the paper itself attributes benchmark behavior to randomness. The authors should report standard deviations, show error bars in learning curves, and ideally perform multiple seeds with a significance test. This does not invalidate the empirical work but is necessary for the strength of the stated conclusions.","section":"Experimental Results, Figs. 3 and 5, Table 1"}],"minor_comments":[{"comment":"Typos and language issues: 'reply buffer' in Eq. (3) should be 'replay buffer'; 'gain considerable success' should be 'gained'; 'as while as' in Fig. 1 caption should be 'as well as'; 'SAFE_K is deviced by' should be 'SAFE_K is defined by'; 'The reminder of Eq. (11)' should be 'The remainder of Eq. (11)'.","section":"Global"},{"comment":"The description of MADDPG says it 'requires the joint observation and action of all agents as input' and does not scale; this is correct but it would be helpful to cite the continuous-action value-factorization extensions (e.g., DOP, FACMAC) to position SAFE more precisely. Currently the related work treats discrete-action COMA and value factorization as the only prior art.","section":"Related Work"},{"comment":"The parameter study shows that K=1 outperforms K=30-100, but the paper does not discuss variance or why averaging sampled Q-values should increase bias. The comparison SAFE_batch_mean replaces the action with the mean of 32 sampled actions, which is different from the expectation of Q used in SAFE_K; this difference should be spelled out so the ablation is interpretable.","section":"Parameter Study and Ablations"},{"comment":"The Bellman equation as written uses Q_tot but the paper later uses Q for the centralized critic; the notation should be made consistent, and the distinction between the global state s and the observation history τ should be maintained throughout.","section":"Background, Eq. (1)"},{"comment":"The reward function is described only as 'collision penalty, and components encouraging formation and efficiency.' Since the paper compares methods on this reward, the exact formulation should be given or a precise reference to Highway-Env should be provided. Also, the number of hidden layers for the critic and the learning rates are not stated, making reproducibility harder.","section":"Experimental Setup"}],"recommendation":"major_revision","confidential_remarks":"The paper's central theoretical claim—unbiased deterministic policy gradient and guaranteed convergence—is not supported by the provided proof. The empirical method may still be useful, but the authors need to either prove unbiasedness for the actual implemented update (e.g., by detaching the baseline and stating the exact loss) or reframe the paper as an empirical study. If the proof cannot be repaired, the 'ensuring convergence' wording in the abstract and title would be misleading. I would not reject outright because the baseline idea is simple and plausibly effective, and the experimental design is reasonable, but the manuscript in its current form overclaims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the replay-buffer default action is a simple and genuinely new idea for continuous-action counterfactual baselines, and the ablation in Fig. 5b suggests the gains are really from credit assignment, not from the continuous action representation. The problem is the stated theory: Lemma 0.1 is a correct stochastic policy-gradient unbiasedness result, but the paper's claim that it 'generalizes to deterministic policy gradients' via the σ→0 limit is not a proof. The DPG is not obtained by taking the limit of the stochastic PG in a way that preserves the baseline cancellation, and the actual algorithm (off-policy, replay buffer, deep networks, shared actor parameters) is not the setting of Konda-Tsitsiklis. Moreover, if the actor update is deterministic and the baseline Q(s,a^{-i},\\bar{a}^i) is differentiated through the shared parameters, then ∇_θ b_i ≠ 0 and the baseline biases the update. The paper never specifies the exact actor loss or a stop-gradient, so the convergence guarantee in the abstract is unsupported.\n\nThe empirical study is suggestive but thin: one environment, no reported variance for the three runs, no code, and no strong continuous-action MARL baselines like MADDPG, FACMAC, or DOP. Comparing against discrete-action COMA and QMIX is a sanity check, not a SOTA claim. The parameter study is interesting—K=1 beating K=30/50/80/100 is surprising—but the 'average performance' explanation conflicts with the mode of the sampled actions shown in Fig. 4b.\n\nNone of this kills the core idea. The baseline is cheap, plausible, and the ablations back it up. I'd want the authors to re-scope the theoretical claims, add error bars and code, and include proper continuous baselines before accepting it. As is, it's a useful practical contribution wrapped in an overclaimed guarantee. I'd bring it up in reading group and I'd cite the baseline trick, but I'd not take the convergence proof at face value.","headline":"The replay-buffer default action is a neat, practical idea with decent ablations, but the convergence proof does not cover the actual algorithm—the σ→0 transfer and the off-policy deep setting are both hand-waved.","tokens_in":11009,"tokens_out":8508,"would_cite":true,"duration_ms":97208,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SAFE argues that a default action sampled from each agent's own replay buffer gives a counterfactual baseline for continuous cooperative control that adds no bias to the deterministic policy gradient.","keywords":["multi-agent reinforcement learning","counterfactual credit assignment","deterministic policy gradient","continuous action space","cooperative control","experience replay buffer","autonomous driving","policy gradient bias"],"falsifier":"Run the actual training procedure on a small two-agent cooperative MDP with a known optimal policy and record the empirical mean of $\\sum_i \\text{grad}_\\theta \\log \\pi_i(a_i|\\tau_i) b_i$ over many updates as the exploration noise anneals toward zero. If this quantity is systematically nonzero, the no-bias claim is refuted; if it shrinks to numerical noise, the claim is supported.","tokens_in":9972,"feed_emoji":"🚗","tokens_out":7903,"duration_ms":87723,"temperature":0.7,"texified_at":"2026-08-05T21:31:35.684627+00:00","pith_summary":"This paper is trying to extend counterfactual credit assignment, a technique that works well when agents choose from a small discrete set of actions, to tasks where agents choose continuous values such as steering angles. Its proposed mechanism replaces one agent's action in the centralized critic with a default action drawn uniformly from that agent's experience replay buffer, making the baseline a data-driven estimate of the agent's average contribution. The paper claims this baseline introduces no bias into the deterministic policy gradient and therefore preserves convergence to local optima, while avoiding the extra simulations, reward models, or hand-picked default actions that prior continuous extensions require. If right, this would give continuous-action cooperative systems, such as coordinated autonomous driving, a principled and practical credit-assignment method that scales beyond what discrete-action baselines can do.","texify_model":"deepseek-v4-flash","texify_usage":{"total_tokens":4084,"prompt_tokens":719,"completion_tokens":3365,"prompt_tokens_details":{"cached_tokens":0},"prompt_cache_hit_tokens":0,"prompt_cache_miss_tokens":719,"completion_tokens_details":{"reasoning_tokens":2734}},"feed_headline":"Replay-buffer default action keeps multi-agent gradients unbiased","feed_subtitle":"Replacing an agent's action with one sampled from its own memory makes credit assignment work in continuous control.","key_machinery":"The self-evolving default action: at each training step, a baseline replaces agent i's action in the centralized critic with $\\bar a^i(t)$, drawn uniformly from the agent's own replay buffer. This single object does double duty: it makes the baseline action-independent, which is what the unbiasedness argument needs, and it gives the baseline a data-driven meaning, since actions that dominate the buffer become the default and come to represent the agent's average behavior. It replaces the discrete-action sum over all possible actions with one continuous, experience-grounded sample, removing the need for simulator rollouts or task-specific defaults.","core_discovery":"The central claim, as the author states it, is that the counterfactual baseline $b_i = Q(s, a^{-i}, \\bar a^i(t))$, with $\\bar a^i(t)$ sampled uniformly from agent i's experience buffer at training step t, solves the continuous-action credit-assignment problem. Because the baseline does not depend on the agent's own action $a_i$, Lemma 0.1 asserts that its contribution to the expected policy gradient vanishes; because the sampled action is one the agent has actually executed, the critic's value for it is well-trained and the baseline reflects the agent's average performance. The paper concludes that the resulting deterministic policy gradient is unbiased and converges to a local optimum, and that a","pith_inferences":["Not stated in the paper, but the core recipe is generic: any action drawn from an agent's own behavioral distribution and held independent of the current action could act as a data-driven counterfactual baseline, so the idea may transfer to other actor-critic and policy-gradient algorithms.","A straightforward way to test the unbiasedness claim directly is to measure the baseline's empirical contribution to the gradient on a small cooperative control problem with a known Q-function; a systematic nonzero contribution as exploration noise decays would require qualifying the no-bias conclusion.","Because the baseline depends on the replay buffer's action distribution, its adaptivity may lag in non-stationary environments where the optimal policy keeps changing; sampling from a more recent or prioritized buffer is a natural extension to test."],"forward_implications":["Continuous cooperative MARL can use counterfactual credit assignment without extra simulations or learned reward models, lowering the cost of training.","The baseline improves as training proceeds: the sampled default action tracks the agent's most frequent behavior, so credit assignment becomes more accurate exactly when policies stabilize.","A single default action outperforms averaging many sampled actions in the paper's experiments, so the mechanism stays computationally light.","The paper's experiments show large collision-rate reductions on cooperative highway driving relative to discrete-action and on-policy baselines across scenarios with 2 to 7 controlled vehicles."],"fun_headline_variants":["Replay buffer default action removes bias in continuous MARL","Self-evolving baseline from experience buffer fixes credit assignment","Unbiased gradients via default actions sampled from replay","Experience-buffer default action enables unbiased continuous credit assignment","Default action from replay buffer gives unbiased MARL gradients"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the unbiasedness argument for stochastic policy gradients transfers unchanged to the deterministic, off-policy deep actor-critic algorithm actually trained, through the limit that a deterministic policy is a zero-noise stochastic policy; if that transfer fails, the convergence guarantee stated in Lemma 0.1 does not follow.","fun_headline_variants_meta":{"raw":{"variants":["Replay buffer default action removes bias in continuous MARL","Self-evolving baseline from experience buffer fixes credit assignment","Unbiased gradients via default actions sampled from replay","Experience-buffer default action enables unbiased continuous credit assignment","Default action from replay buffer gives unbiased MARL gradients"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00036,"raw_usage":{"total_tokens":1751,"prompt_tokens":681,"completion_tokens":1070,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":425,"completion_tokens_details":{"reasoning_tokens":995}},"tokens_in":425,"tokens_out":1070,"duration_ms":9268,"temperature":1.0,"reasoning_tokens":995,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T14:56:32.076248+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the actual training procedure on a small two-agent cooperative MDP with a known optimal policy and record the empirical mean of $\\sum_i \\text{grad}_\\theta \\log \\pi_i(a_i|\\tau_i) b_i$ over many updates as the exploration noise anneals toward zero. If this quantity is systematically nonzero, the no-bias claim is refuted; if it shrinks to numerical noise, the claim is supported.","supporting_citations":[],"review_version":1}