Pith. sign in

REVIEW 4 major objections 5 minor 38 references

SAFE argues that a default action sampled from each agent's own replay buffer gives a counterfactual baseline for continuous cooperative control that adds no bias to the deterministic policy gradient.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:56 UTC pith:VKBP3EXK

load-bearing objection The replay-buffer default action is a neat, practical idea with decent ablations, but the convergence proof does not cover the actual algorithm—the σ→0 transfer and the off-policy deep setting are both hand-waved. the 4 major comments →

arxiv 2607.18597 v3 pith:VKBP3EXK submitted 2026-07-21 cs.LG cs.MA

A Self-Evolving Default Action for Cooperative Tasks with Continuous Action Space

Shuangyao Huang This is my paper
classification cs.LG cs.MA
keywords multi-agent reinforcement learningcounterfactual credit assignmentdeterministic policy gradientcontinuous action spacecooperative controlexperience replay bufferautonomous drivingpolicy gradient bias
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper is trying to extend counterfactual credit assignment, a technique that works well when agents choose from a small discrete set of actions, to tasks where agents choose continuous values such as steering angles. Its proposed mechanism replaces one agent's action in the centralized critic with a default action drawn uniformly from that agent's experience replay buffer, making the baseline a data-driven estimate of the agent's average contribution. The paper claims this baseline introduces no bias into the deterministic policy gradient and therefore preserves convergence to local optima, while avoiding the extra simulations, reward models, or hand-picked default actions that prior continuous extensions require. If right, this would give continuous-action cooperative systems, such as coordinated autonomous driving, a principled and practical credit-assignment method that scales beyond what discrete-action baselines can do.

Core claim

The central claim, as the author states it, is that the counterfactual baseline b_i = Q(s, a^{-i}, \bar a^i(t)), with \bar a^i(t) sampled uniformly from agent i's experience buffer at training step t, solves the continuous-action credit-assignment problem. Because the baseline does not depend on the agent's own action a_i, Lemma 0.1 asserts that its contribution to the expected policy gradient vanishes; because the sampled action is one the agent has actually executed, the critic's value for it is well-trained and the baseline reflects the agent's average performance. The paper concludes that the resulting deterministic policy gradient is unbiased and converges to a local optimum, and that a

What carries the argument

The self-evolving default action: at each training step, a baseline replaces agent i's action in the centralized critic with \bar a^i(t), drawn uniformly from the agent's own replay buffer. This single object does double duty: it makes the baseline action-independent, which is what the unbiasedness argument needs, and it gives the baseline a data-driven meaning, since actions that dominate the buffer become the default and come to represent the agent's average behavior. It replaces the discrete-action sum over all possible actions with one continuous, experience-grounded sample, removing the need for simulator rollouts or task-specific defaults.

Load-bearing premise

The load-bearing premise is that the unbiasedness argument for stochastic policy gradients transfers unchanged to the deterministic, off-policy deep actor-critic algorithm actually trained, through the limit that a deterministic policy is a zero-noise stochastic policy; if that transfer fails, the convergence guarantee stated in Lemma 0.1 does not follow.

What would settle it

Run the actual training procedure on a small two-agent cooperative MDP with a known optimal policy and record the empirical mean of sum_i grad_theta log pi_i(a_i|tau_i) b_i over many updates as the exploration noise anneals toward zero. If this quantity is systematically nonzero, the no-bias claim is refuted; if it shrinks to numerical noise, the claim is supported.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Continuous cooperative MARL can use counterfactual credit assignment without extra simulations or learned reward models, lowering the cost of training.
  • The baseline improves as training proceeds: the sampled default action tracks the agent's most frequent behavior, so credit assignment becomes more accurate exactly when policies stabilize.
  • A single default action outperforms averaging many sampled actions in the paper's experiments, so the mechanism stays computationally light.
  • The paper's experiments show large collision-rate reductions on cooperative highway driving relative to discrete-action and on-policy baselines across scenarios with 2 to 7 controlled vehicles.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Not stated in the paper, but the core recipe is generic: any action drawn from an agent's own behavioral distribution and held independent of the current action could act as a data-driven counterfactual baseline, so the idea may transfer to other actor-critic and policy-gradient algorithms.
  • A straightforward way to test the unbiasedness claim directly is to measure the baseline's empirical contribution to the gradient on a small cooperative control problem with a known Q-function; a systematic nonzero contribution as exploration noise decays would require qualifying the no-bias conclusion.
  • Because the baseline depends on the replay buffer's action distribution, its adaptivity may lag in non-stationary environments where the optimal policy keeps changing; sampling from a more recent or prioritized buffer is a natural extension to test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SAFE, a counterfactual baseline for continuous-action cooperative MARL. Rather than computing a COMA-style expectation over discrete actions, SAFE replaces agent i's action by a 'default action' sampled uniformly from that agent's replay buffer: b_i = Q(s, a^{-i}, \bar{a}^i) (Eq. 9). The authors claim that this baseline introduces no bias into the deterministic policy gradient and thereby ensures convergence to local optima (Lemma 0.1), and they support the method with experiments on Highway-Env cooperative driving tasks, where SAFE outperforms VDN, QMIX, IQL, COMA, and MAPPO. The paper also includes a parameter study on the number of sampled default actions and ablations comparing against a zero default action, a batch-mean default, continuous COMA, and a plain centralized critic.

Significance. The core idea is attractive: a replay-buffer-sampled default action is task-agnostic, requires no extra simulations or reward models, and naturally extends to continuous action spaces. If the unbiasedness and convergence claims were correct, this would be a meaningful step beyond COMA for continuous-action cooperative tasks. The empirical results on 2V1O through 7V2O are also suggestive of practical utility. However, the theoretical justification in Lemma 0.1 does not cover the actual deep off-policy deterministic actor-critic with shared parameters that the paper implements. The main novelty therefore rests on a proof that is not valid as stated, and the paper does not specify the exact update rule used in the experiments, leaving a gap between theory and implementation. The empirical study is well structured, but the central theoretical claim needs substantial reworking.

major comments (4)
  1. [Algorithm, Eq. (9)-(13), Lemma 0.1] Lemma 0.1 proves unbiasedness for a stochastic policy gradient using the identity E[∇θ log μ_i] = 0, and then transfers to deterministic policies through the informal limit σ→0 in Eq. (10). This transfer is not valid. In the implemented algorithm the baseline depends on θ through the other agents' actions a^{-i} = π_θ(τ^{-i}) and, in the shared-parameter setting, through the actor parameters themselves. Consequently ∇θ b_i ≠ 0 in general, and subtracting b_i from Q in a deterministic actor update biases the gradient unless the baseline is explicitly detached (no stop-gradient is mentioned). The proof also assumes b_i is not a function of a_i, but for the deterministic policy a_i = π_θ(τ_i), so the chain rule introduces additional terms. The paper must state the actual update rule and prove unbiasedness for that rule, or substantially weaken the claim.
  2. [Algorithm, Eq. (14)] The convergence to local optima is attributed to Konda & Tsitsiklis (1999). That theorem covers on-policy actor-critic with linear function approximation, not the deep off-policy actor-critic with replay buffer and shared actor parameters used in this paper. Moreover, Eq. (14) only asserts that the gradient norm tends to zero; this is not by itself a statement about convergence to a local optimum of the objective, especially under the non-stationary, off-policy update used in deep MARL. The convergence claim in the abstract and Lemma 0.1 is therefore unsupported by the cited theorem. A valid convergence argument for the actual algorithm is needed, or the claim should be replaced by an empirical statement.
  3. [Algorithm and Experimental Setup] The paper never specifies the deterministic actor update used in the experiments. Eq. (4) is the single-agent deterministic policy gradient ∇θ Q(s, π_θ(s)), while Eq. (11) defines a stochastic policy gradient with an advantage A_i. The experiments presumably use some multi-agent variant of DDPG with the counterfactual baseline, but the connection between the theory and the implementation is missing. In particular, if the actor minimizes a loss such as -Σ_i Q(s, a) + λ Σ_i b_i, the effect of the baseline depends on the precise loss and on whether gradients flow through the baseline. Without this specification, the empirical results cannot be interpreted as validating the proposed unbiasedness mechanism. This gap is load-bearing because the central claim concerns the deterministic policy gradient.
  4. [Experimental Results, Figs. 3 and 5, Table 1] All empirical comparisons are based on three independent runs and no error bars or confidence intervals are reported. Given that the main quantitative claims are comparative ('consistently outperforms', '97% lower collision rate'), the absence of variance information makes it difficult to assess statistical reliability, especially in the 7V2O scenario where the paper itself attributes benchmark behavior to randomness. The authors should report standard deviations, show error bars in learning curves, and ideally perform multiple seeds with a significance test. This does not invalidate the empirical work but is necessary for the strength of the stated conclusions.
minor comments (5)
  1. [Global] Typos and language issues: 'reply buffer' in Eq. (3) should be 'replay buffer'; 'gain considerable success' should be 'gained'; 'as while as' in Fig. 1 caption should be 'as well as'; 'SAFE_K is deviced by' should be 'SAFE_K is defined by'; 'The reminder of Eq. (11)' should be 'The remainder of Eq. (11)'.
  2. [Related Work] The description of MADDPG says it 'requires the joint observation and action of all agents as input' and does not scale; this is correct but it would be helpful to cite the continuous-action value-factorization extensions (e.g., DOP, FACMAC) to position SAFE more precisely. Currently the related work treats discrete-action COMA and value factorization as the only prior art.
  3. [Parameter Study and Ablations] The parameter study shows that K=1 outperforms K=30-100, but the paper does not discuss variance or why averaging sampled Q-values should increase bias. The comparison SAFE_batch_mean replaces the action with the mean of 32 sampled actions, which is different from the expectation of Q used in SAFE_K; this difference should be spelled out so the ablation is interpretable.
  4. [Background, Eq. (1)] The Bellman equation as written uses Q_tot but the paper later uses Q for the centralized critic; the notation should be made consistent, and the distinction between the global state s and the observation history τ should be maintained throughout.
  5. [Experimental Setup] The reward function is described only as 'collision penalty, and components encouraging formation and efficiency.' Since the paper compares methods on this reward, the exact formulation should be given or a precise reference to Highway-Env should be provided. Also, the number of hidden layers for the critic and the learning rates are not stated, making reproducibility harder.

Circularity Check

0 steps flagged

No significant circularity: the baseline is defined independently, the unbiasedness argument relies on external results, and the main issue (the sigma-to-zero limit) is a correctness gap rather than a circular reduction.

full rationale

The paper's central claim is that the counterfactual baseline b_i = Q(s, a^{-i}, \bar{a}^i(t)), with \bar{a}^i(t) sampled uniformly from the agent's experience buffer, introduces no bias into the deterministic policy gradient and ensures convergence to local optima. This is not a circular claim: the baseline is defined independently of the policy-gradient estimator, and the unbiasedness argument in Lemma 0.1 proceeds from the standard score-function identity E[\nabla_\theta \log \mu_i] = 0, rather than by assuming the desired conclusion. The default action is drawn from the replay buffer, which is part of the algorithm's data distribution; it is not a fitted parameter that is later renamed as a prediction. The convergence statement invokes Konda and Tsitsiklis (1999) as an external theorem, and the deterministic-policy-gradient background invokes Silver et al. (2014); these are external citations, not self-citations, and the derivation does not reduce to a chain of the authors' own prior results. The empirical evaluation is benchmarked against external methods and includes ablations (SAFE_a=0, SAFE_batch_mean, COMA_cont, Centralized_critic) that vary the credit-assignment mechanism, so the reported gains are not forced by the definition of the baseline. The most serious issue is the \sigma \to 0 limit argument in Eq. (10), which is mathematically unsupported for deep off-policy actor-critics with shared parameters; however, an unsound proof step is a correctness risk, not an instance of circularity, because it does not make the conclusion equivalent to the input by construction. No specific circular reduction can be exhibited from the paper's equations, so the appropriate finding is low circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The method's theoretical guarantee inherits classic baseline-unbiasedness and a non-applicable convergence theorem; empirical performance depends on several un-swept hyperparameters. No new physical entities are introduced.

free parameters (4)
  • K (number of sampled default actions) = 1
    Selected after parameter study on 3V2O; SAFE1 outperforms SAFE30/50/80/100. This is a post-hoc choice on the test scenario.
  • Exploration noise annealing (epsilon) = 1.0 -> 0.05 over 50,000 episodes
    Used for actor exploration; no sensitivity analysis; affects behavior but not the theoretical unbiasedness claim.
  • GRU hidden size = 128
    A single architecture choice; not swept.
  • Replay buffer size / minibatch size = unspecified
    Not reported in paper; needed to reproduce.
axioms (5)
  • standard math Score function identity: ∇θ log μ = ∇θ μ / μ, and Σ μ = 1, implying Σ μ ∇θ log μ = 0.
    Used in Lemma 0.1 to show the baseline term has zero expectation.
  • standard math A baseline that does not depend on the agent's own action does not bias the policy gradient (Sutton et al. 2000; COMA).
    This is the entire content of Lemma 0.1.
  • ad hoc to paper Stochastic policy with variance σ→0 converges to the deterministic policy, so unbiasedness transfers (Silver et al. 2014 argument).
    This limit is used to bridge the proof to deterministic policy gradient; it is not rigorously established and is the weakest step.
  • domain assumption Konda-Tsitsiklis actor-critic convergence theorem applies to the deep off-policy setting.
    Cited to conclude convergence to local optima; the theorem assumes on-policy sampling and linear function approximation, which are violated by the deep replay-buffer algorithm.
  • domain assumption Actions sampled from the replay buffer are 'sufficiently trained' in the critic.
    Needed for the claim that the baseline accurately quantifies contribution; never validated in the paper.

pith-pipeline@v1.3.0-alltime-deepseek · 9730 in / 18357 out tokens · 285841 ms · 2026-08-01T14:56:32.076248+00:00 · methodology

0 comments
read the original abstract

Counterfactual credit assignment has proven effective in multi-agent reinforcement learning (MARL) for discrete action spaces, yet its extension to continuous-action cooperative tasks remains challenging. Existing methods that approximate the counterfactual baseline via Monte Carlo sampling often introduce bias into policy gradients and fail to guarantee convergence to local optima, as the sampled actions may not have been sufficiently trained. To address these limitations, we propose SAFE, a novel MARL framework that employs a counterfactual baseline conditioned on a self-evolving default action sampled from each agent's experience buffer. This design naturally extends to continuous action spaces without relying on additional simulations, reward models, or environment-specific prior knowledge. The baseline accurately quantifies each agent's contribution, and introduces no bias into the deterministic policy gradient, ensuring convergence to local optima. Extensive experiments on cooperative vehicular tasks demonstrate that SAFE consistently outperforms state-of-the-art models.

Figures

Figures reproduced from arXiv: 2607.18597 by Shuangyao Huang.

Figure 1
Figure 1. Figure 1: (a). Architectures of the centralized actors and the centralized critic, as while as the information flow between them for [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Highway cooperative autonomous driving tasks. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of SAFE against benchmark models on highway task, under four scenarios: 2-Vehicle-1-Obstacle (2V1O), [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: (a). Trajectories of vehicles after 150,000 training episodes. (b). Probability density distributions of each agent’s default [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: (a). Parameter study on the number of sampled actions in constructing our baseline. (b). Ablation study on the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 1 canonical work pages

  1. [1]

    Langley , title =

    P. Langley , title =. Proceedings of the 17th International Conference on Machine Learning (ICML 2000) , address =. 2000 , pages =

    2000

  2. [2]

    T. M. Mitchell. The Need for Biases in Learning Generalizations. 1980

    1980

  3. [3]

    M. J. Kearns , title =

  4. [4]

    Machine Learning: An Artificial Intelligence Approach, Vol. I. 1983

    1983

  5. [5]

    R. O. Duda and P. E. Hart and D. G. Stork. Pattern Classification. 2000

    2000

  6. [6]

    Suppressed for Anonymity , author=

  7. [7]

    Newell and P

    A. Newell and P. S. Rosenbloom. Mechanisms of Skill Acquisition and the Law of Practice. Cognitive Skills and Their Acquisition. 1981

    1981

  8. [8]

    A. L. Samuel. Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development. 1959

    1959

  9. [9]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Counterfactual multi-agent policy gradients , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  10. [10]

    arXiv preprint arXiv:1706.05296 , year=

    Value-decomposition networks for cooperative multi-agent learning , author=. arXiv preprint arXiv:1706.05296 , year=

    Pith/arXiv arXiv

  11. [11]

    Journal of Machine Learning Research , volume=

    Monotonic value function factorisation for deep multi-agent reinforcement learning , author=. Journal of Machine Learning Research , volume=

  12. [12]

    Advances in neural information processing systems , volume=

    Multi-agent actor-critic for mixed cooperative-competitive environments , author=. Advances in neural information processing systems , volume=

  13. [13]

    Advances in neural information processing systems , volume=

    The surprising effectiveness of ppo in cooperative multi-agent games , author=. Advances in neural information processing systems , volume=

  14. [14]

    2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages=

    CoDe: A Cooperative and Decentralized Collision Avoidance Algorithm for Small-Scale UAV Swarms Considering Energy Efficiency , author=. 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages=. 2024 , organization=

    2024

  15. [15]

    Shapley Q-Value: A Local Reward Approach to Solve Global Reward Games , volume=

    Wang, Jianhong and Zhang, Yuan and Kim, Tae-Kyun and Gu, Yunjie , year=. Shapley Q-Value: A Local Reward Approach to Solve Global Reward Games , volume=. Proceedings of the AAAI Conference on Artificial Intelligence , publisher=. doi:10.1609/aaai.v34i05.6220 , number=

    work page doi:10.1609/aaai.v34i05.6220

  16. [16]

    Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning , DOI=

    Li, Jiahui and Kuang, Kun and Wang, Baoxiang and Liu, Furui and Chen, Long and Wu, Fei and Xiao, Jun , year=. Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning , DOI=. Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining , publisher=

  17. [17]

    Advances in neural information processing systems , volume=

    Actor-critic algorithms , author=. Advances in neural information processing systems , volume=

  18. [18]

    IEEE Transactions on Intelligent Transportation Systems , volume=

    Conditional DQN-based motion planning with fuzzy logic for autonomous driving , author=. IEEE Transactions on Intelligent Transportation Systems , volume=. 2020 , publisher=

    2020

  19. [19]

    2019 IEEE intelligent vehicles symposium (IV) , pages=

    Controlling an autonomous vehicle with deep reinforcement learning , author=. 2019 IEEE intelligent vehicles symposium (IV) , pages=. 2019 , organization=

    2019

  20. [20]

    Proceedings of the International Conference on Automated Planning and Scheduling , volume=

    E^2Coop : Energy efficient and cooperative obstacle detection and avoidance for uav swarms , author=. Proceedings of the International Conference on Automated Planning and Scheduling , volume=

  21. [21]

    IEEE transactions on intelligent transportation systems , volume=

    E^2CoPre : Energy Efficient and Cooperative Collision Avoidance for UAV Swarms With Trajectory Prediction , author=. IEEE transactions on intelligent transportation systems , volume=. 2024 , publisher=

    2024

  22. [22]

    IEEE Transactions on Intelligent Transportation Systems , volume=

    Experience-driven power allocation using multi-agent deep reinforcement learning for millimeter-wave high-speed railway systems , author=. IEEE Transactions on Intelligent Transportation Systems , volume=. 2021 , publisher=

    2021

  23. [23]

    Multiagent Deep Deterministic Policy Gradient-Based Computation Offloading and Resource Allocation for ISAC-Aided 6G V2X Networks , year=

    Hu, Bintao and Zhang, Wenzhang and Gao, Yuan and Du, Jianbo and Chu, Xiaoli , journal=. Multiagent Deep Deterministic Policy Gradient-Based Computation Offloading and Resource Allocation for ISAC-Aided 6G V2X Networks , year=

  24. [24]

    Computers and Electrical Engineering , volume=

    A deep reinforcement learning-based multi-agent area coverage control for smart agriculture , author=. Computers and Electrical Engineering , volume=. 2022 , publisher=

    2022

  25. [25]

    arXiv preprint arXiv:1708.04782 , year=

    Starcraft II: A new challenge for reinforcement learning , author=. arXiv preprint arXiv:1708.04782 , year=

    Pith/arXiv arXiv

  26. [26]

    , author=

    POMDPs for robotic tasks with mixed observability. , author=. Robotics: Science and systems , volume=

  27. [27]

    Advanced Robotics , volume=

    A reinforcement learning approach to fail-safe design for multiple space robots—cooperation mechanism without communication and negotiation schemes , author=. Advanced Robotics , volume=. 2003 , publisher=

    2003

  28. [28]

    Journal of Artificial Intelligence Research , volume=

    Optimal and approximate Q-value functions for decentralized POMDPs , author=. Journal of Artificial Intelligence Research , volume=

  29. [29]

    Advances in Neural Information Processing Systems , volume=

    Towards understanding cooperative multi-agent q-learning with value factorization , author=. Advances in Neural Information Processing Systems , volume=

  30. [30]

    International conference on machine learning , pages=

    Deterministic policy gradient algorithms , author=. International conference on machine learning , pages=. 2014 , organization=

    2014

  31. [31]

    arXiv preprint arXiv:1509.02971 , year=

    Continuous control with deep reinforcement learning , author=. arXiv preprint arXiv:1509.02971 , year=

    Pith/arXiv arXiv

  32. [32]

    nature , volume=

    Human-level control through deep reinforcement learning , author=. nature , volume=

  33. [33]

    Advances in Complex Systems , volume=

    Optimal payoff functions for members of collectives , author=. Advances in Complex Systems , volume=. 2001 , publisher=

    2001

  34. [34]

    cooperative agents , author=

    Multi-agent reinforcement learning: Independent vs. cooperative agents , author=. Proceedings of the tenth international conference on machine learning , pages=

  35. [35]

    arXiv preprint arXiv:2003.06709 , volume=

    Deep multi-agent reinforcement learning for decentralized continuous cooperative control , author=. arXiv preprint arXiv:2003.06709 , volume=

    Pith/arXiv arXiv 2003

  36. [36]

    Advances in neural information processing systems , pages=

    Policy gradient methods for reinforcement learning with function approximation , author=. Advances in neural information processing systems , pages=

  37. [37]

    GitHub repository , howpublished =

    Leurent, Edouard , title =. GitHub repository , howpublished =. 2018 , publisher =

    2018

  38. [38]

    arXiv preprint arXiv:1409.1259 , year=

    On the properties of neural machine translation: Encoder-decoder approaches , author=. arXiv preprint arXiv:1409.1259 , year=

    Pith/arXiv arXiv