Pith. sign in

REVIEW 4 major objections 6 minor 18 references

Comparative Study of Multi-Agent Actor-Critic Algorithms in Parameterized Action Reinforcement Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Shared replay experience across independent actor-critic agents consistently lifts Greedy Actor-Critic on parameterized-action benchmarks, while leaving Soft Actor-Critic and Truncated Quantile Critics mostly unchanged.

desk verdict A modest empirical study of shared-replay multi-agent GAC/SAC/TQC whose key claim is undermined by an undefined action-fusion rule and missing statistical comparison to single-agent baselines. read the letter →

arxiv 2607.19117 v1 pith:AUFY22DE submitted 2026-07-21 cs.AI

Ubayd Ali Bapoo , Clement N Nyirenda This is my paper
classification cs.AI
keywords parameterizedactionreinforcementlearningmulti-agentactor-criticmethodssofthybridspacesshared-experienceGreedyTruncatedQuantileCritics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a lightweight multi-agent scheme — several independent actor-critic agents sharing one replay buffer and fusing their actions into a single executable action — can improve reinforcement learning in parameterized action spaces, where each action combines a discrete choice with continuous parameters. On the Goal-v0 and Platform-v0 benchmarks, the multi-agent version of Greedy Actor-Critic (MAGAC) outperforms its single-agent baseline, while the multi-agent versions of SAC and TQC show only small differences. The paper also claims that moving from three to five agents yields little extra return, and ten agents adds almost nothing while sharply increasing training time. A sympathetic reader would care because the study isolates a simple design choice — shared experience without a centralized critic — and measures where it helps and where it does not.

What carries the argument

The load-bearing mechanism is the shared replay buffer combined with action fusion. Each of the n agents keeps its own actor and critic networks; all agents write their transitions into one buffer and sample from it during updates, so every policy sees the group's collective experience. At execution, the agents' individual parameterized actions are merged by an action-fusion strategy into the single action the environment accepts. The paper's comparisons hinge on this design: it is what distinguishes the multi-agent variants from their single-agent baselines and what the agent-count scaling analysis varies.

What would settle it

Rerun the Goal-v0 experiment with the multi-agent framework's action-fusion rule replaced by a random selection of one agent's action per timestep. If MAGAC's advantage over GAC disappears, the shared replay buffer is not the cause; the fusion rule is.

Watch

Extended reading notes

Core claim

The central claim is that shared-experience multi-agent learning changes actor-critic algorithms for parameterized action spaces unevenly. Using three, five, and ten independent actor-critic agents that store transitions in a common replay buffer and select actions through an action-fusion rule, the paper reports that Multi-Agent Greedy Actor-Critic (MAGAC) improves the Greedy Actor-Critic baseline in both environments: average return rises from -8.551 to -7.885 on Goal-v0 and from 0.170 to 0.185 on Platform-v0, with most of the gain already present at three agents. Multi-Agent Soft Actor-Critic (MASAC) and Multi-Agent Truncated Quantile Critics (MATQC) stay close to their single-agent basel

Load-bearing premise

The results depend on the action-fusion rule that combines the agents' separate actions into one executable action, and the paper never specifies this rule; if it is trivial averaging or voting, the multi-agent setup may behave like a single noisy ensemble, making the reported gains an artifact of that choice.

Editorial extensions

If this is right

  • MAGAC's improvement over GAC is statistically significant in both benchmarks, so shared-replay experience can matter for greedy update rules even without a centralized critic.
  • MASAC and MATQC gains are small and often not significant, implying the shared-experience benefit is algorithm-specific, not a general property of adding agents.
  • Agent counts beyond five produce negligible return gains, so practitioners can cap agent numbers near three to five without sacrificing performance.
  • Training time grows sharply with agent count for MAGAC (roughly six hours for ten agents on Platform-v0 versus 41 minutes for the single-agent baseline), making small configurations the cost-effective choice.
  • Because the framework is not full CTDE, the reported effects isolate shared replay rather than centralized value information; stronger coordination is left to future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the action-fusion rule is simple, such as averaging or majority voting, MAGAC's gain may be an ensemble effect rather than true multi-agent coordination; a single agent that samples several candidate actions per step could reproduce the same benefit.
  • The algorithm-specific result makes sense mechanistically: GAC's cross-entropy greedy improvement is sensitive to diverse off-policy data, while SAC's entropy term already supplies exploration, so extra agents add less — a testable hypothesis by ablating SAC's temperature.
  • Varying the fusion rule (random agent, best agent by critic value, weighted average) is a direct next experiment; the current paper treats fusion as fixed, so the fusion rule itself may be responsible for the observed ordering.
  • A parameter-sharing variant of MAGAC that keeps the shared buffer but collapses the separate networks would separate the benefit of experience diversity from the benefit of independent policy representations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper empirically evaluates shared-experience multi-agent extensions of three parameterized-action actor–critic algorithms (GAC, SAC, TQC). In the proposed framework, several independent actor–critic agents share a single replay buffer but have separate networks; their proposed actions are combined by an unspecified 'action fusion strategy' and executed in the environment. Experiments on Platform-v0 and Goal-v0 compare the multi-agent variants (3, 5, and 10 agents) with the single-agent baselines in terms of average evaluation return and training time over 10 runs. The authors report that MAGAC improves over GAC while MASAC and MATQC show modest gains, and that increasing agents beyond five gives diminishing returns. One-way ANOVA and Tukey HSD tests are used among the three multi-agent algorithms only.

Significance. If the claims were fully supported, the paper would provide a useful empirical datapoint: shared replay can help some parameterized-action actor-critic algorithms more than others, and moderate agent counts are enough. The authors are appropriately modest about the contribution, explicitly stating that this is not a new MARL algorithm and that it is not CTDE. The use of multiple seeds and formal significance tests is a positive feature, as is the explicit discussion of the performance/computation trade-off. However, the contribution is undermined by (i) an unspecified core mechanism (action fusion), (ii) statistical comparisons that exclude the single-agent baselines, and (iii) unreported hyperparameters and code. These are fixable in revision.

major comments (4)
  1. [III] The proposed framework's behavior is not fully specified. Section III states: 'The individual actions are then combined using an action fusion strategy to produce a single executable action for the environment,' but the strategy is never described. The reader cannot tell whether the fusion is averaging continuous parameters, majority voting on discrete actions, or a more complex selection rule. Every experimental result depends on this mechanism, and the paper's central attribution of MAGAC's gains to 'shared experience' is therefore confounded by the fusion rule. The authors themselves list 'alternative action fusion strategies' as future work (Section VI), which confirms that this choice is unexplored. Please specify the fusion rule precisely and provide an ablation that separates the effect of fusion from the effect of sharing a replay buffer.
  2. [V-A] Tables I and II report only mean returns and training times, with no standard deviations, confidence intervals, or per-run values. Moreover, the ANOVA analyses in Sections V-D and V-G are run on the three multi-agent algorithms only (MAGAC vs MASAC vs MATQC); they do not include the single-agent baselines. Consequently, the abstract's claim that 'the multi-agent framework consistently improves Greedy Actor-Critic performance' is supported only by point estimates (e.g., Table I: -8.551 vs -7.914; Table II: 0.170 vs 0.183), without any inferential test or effect size for those comparisons. Please report variability and perform statistical comparisons that include the baselines, or restrict the conclusions to descriptive observations.
  3. [IV-A] Reproducibility and fair-comparison claims cannot be assessed. Section IV-A states that hyperparameters were tuned with Microsoft NNI and that 'the same ... hyperparameter settings' were used, but the actual values, network sizes, optimizers, learning rates, entropy temperatures, and quantile parameters are not listed. No code is provided, and no seed list is given. At a minimum, include a hyperparameter table and a description of the computing environment; ideally, release the code and seeds so that the experiments can be reproduced and the 'same codebase with modifications only to actor–critic updates' claim verified.
  4. [V-B] Training-time claims are difficult to interpret without more context. Times in Tables I and II are single means without variance or hardware/software specifications. Some entries are surprising, e.g., MASAC with 3 agents in Goal is 17:47 versus SAC baseline 26:37; the paper gives no explanation for such nonmonotonic effects of agent count on wall-clock time. If computational efficiency is a central trade-off, report clock-time distributions, hardware details, and the cause of the timing variation (e.g., per-agent update frequency or environment interaction overhead).
minor comments (6)
  1. [I] The opening paragraphs of Section I and Section II-A are about search-engine match plans and inverted indexes. The experiments, however, are on Platform-v0 and Goal-v0, and the match-plan connection is never revisited. This framing should either be cut or integrated into the motivation.
  2. [Abstract] Use consistent spelling of 'parameterized/parameterised' throughout (the abstract and several sections mix the two).
  3. [V-D] Table III and similar tables contain malformed typesetting, e.g., '32.975.659×10 −8' under F Statistic; this should read 'F = 32.97, p = 5.659×10^-8'.
  4. [IV] The definition of average evaluation return is vague ('the final evaluation score between multiple runs'). State how evaluation was conducted (e.g., number of evaluation episodes, exploration disabled, whether the final policy after 5,000 episodes or the best policy was used).
  5. [III] Given the framework, 'multi-agent' may be a misnomer because the agents do not interact or coordinate; consider using 'shared-experience ensemble' or explicitly justify the term. The paper does acknowledge this, but the title/abstract still says 'multi-agent'.
  6. [Figures] Figures 1 and 2 are referenced but not shown in the supplied text; if present in the final PDF, ensure Figure 1 includes the action fusion box, because the current textual description omits it.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: new benchmark data; action-fusion gap is lack of specification, not a circular reduction.

full rationale

The paper is an empirical comparative study, not a derivation chain. Its central claims are average evaluation returns measured across ten independent runs and analyzed with ANOVA/Tukey HSD tests. No equation or fitted parameter is defined in terms of the target result, so there is no reduction-by-construction. The only self-citation is the authors' prior work [11], used to supply the single-agent baseline implementations and environment framing; the multi-agent variants are new implementations with separate actor/critic networks and a shared replay buffer, and the numbers in Tables I and II are measured outputs, not fitted constants renamed as predictions. No uniqueness theorem or ansatz is imported from a self-citation to force the conclusions. The most serious weakness is the unspecified action fusion strategy: Section III states 'The individual actions are then combined using an action fusion strategy to produce a single executable action for the environment' but never defines the rule, and future work lists 'alternative action fusion strategies.' This is a genuine reproducibility and attribution gap—MAGAC's improvement over GAC could be partly caused by the fusion rule rather than by shared experience—but an unspecified mechanism is not circularity, since the paper does not define the multi-agent gain in terms of the fusion rule. The explicit disclaimer that the framework 'should be interpreted as a shared-experience multi-agent extension rather than a full CTDE implementation' is an honest limitation, not evidence of circularity. Score 2 reflects the minor, non-load-bearing self-citation to [11] for baseline implementations.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper's central comparison depends on unreported hyperparameters, an unspecified action-fusion procedure, and the assumption that shared-replay training is a valid multi-agent mechanism. There are no new physical or conceptual entities.

free parameters (2)
  • NNI-tuned hyperparameters for each algorithm (learning rates, batch size, network sizes, entropy temperature, quantile s = not reported
    Section IV: 'Hyperparameter optimisation was performed using Microsoft Neural Network Intelligence (NNI).' The tuned values are not given; the same hyperparameters are used for both single- and multi-agent variants, so any mismatch in optimality could confound the comparison.
  • Action fusion weights/strategy = unspecified
    Section III describes an 'action fusion strategy' with no details; if the fusion involves tunable weights (e.g., preference for one agent), they are not listed.
assumptions (4)
  • domain assumption Platform-v0 and Goal-v0 environments faithfully represent parameterized action RL tasks and reward learning progress comparably across methods.
    The paper relies on these two benchmarks from Bester et al. [10] to generalize about parameterized action RL; no other environments are used.
  • ad hoc to paper Training multiple independent actor-critic agents on a shared replay buffer in a single-agent environment constitutes a meaningful multi-agent learning framework.
    The paper defines its own 'multi-agent' as multiple learners that share experience and fuse actions; this is not standard MARL but an ensemble. The claim that this framework 'extends' prior single-agent work assumes this definition is valid.
  • domain assumption The fused action yields a valid Markovian transition in the environment.
    Section III: actions are fused into 'a single executable action for the environment.' The paper does not prove that the fused action respects the environment's dynamics.
  • domain assumption The actor-critic algorithms GAC, SAC, and TQC are correctly implemented as described in the cited references, and the retained baselines from [11] are trustworthy.
    The paper does not provide implementation details or code, relying on previous work [11] and mismatched references for algorithm definitions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Comparative Study of Multi-Agent Actor-Critic Algorithms in Parameterized Action Reinforcement Learning." pith.science (2026). https://pith.science/paper/AUFY22DE

@misc{pith2026260719117,
  author       = {Pith},
  title        = {Pith review of: Comparative Study of Multi-Agent Actor-Critic Algorithms in Parameterized Action Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AUFY22DE}},
  note         = {Machine review of arXiv:2607.19117}
}
read the original abstract

Parameterized action reinforcement learning has shown strong performance in environments requiring both discrete action selection and continuous parameterization. Prior work established the effectiveness of single-agent actor-critic algorithms - Greedy Actor-Critic (GAC), Soft Actor-Critic (SAC), and Truncated Quantile Critics (TQC) - on benchmark parameterized action tasks, but their extension to multi-agent settings remains largely unexplored. This paper presents a comparative study of shared-experience multi-agent extensions of these algorithms: Multi-Agent Greedy Actor-Critic (MAGAC), Multi-Agent Soft Actor-Critic (MASAC), and Multi-Agent Truncated Quantile Critics (MATQC). Rather than following the centralized training, decentralized execution (CTDE) paradigm, the proposed framework uses multiple independent actor-critic agents that share a replay buffer while maintaining separate policy and value networks. We evaluate the algorithms on the Platform-v0 and Goal-v0 benchmarks against their single-agent counterparts, using three-, five-, and ten-agent configurations to assess scalability. Performance is measured by average evaluation return and training time across ten independent runs, with one-way ANOVA and Tukey HSD post-hoc tests used to assess statistical significance. Results show that the multi-agent framework consistently improves Greedy Actor-Critic performance, while MASAC and MATQC show comparatively modest gains over their single-agent versions. Increasing the number of agents beyond five yields limited additional performance while substantially raising computational cost, particularly for MAGAC. These results highlight a trade-off between learning performance and computational efficiency, offering insight into the scalability of shared-experience multi-agent actor-critic methods for parameterized action reinforcement learning.

Figures

Figures reproduced from arXiv: 2607.19117 by the authors.

Figure 1
Figure 1. Overview of the proposed shared-experience multi-agent actor–critic [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Average training time for each evaluated algorithm. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 1 canonical work pages

  1. [1]

    Inverted files for text search engines,

    J. Zobel and A. Moffat, “Inverted files for text search engines,”ACM Computing Surveys, vol. 38, no. 2, pp. 6–es, Jul. 2006

    2006

  2. [2]

    Optimizing query evaluations using reinforcement learning for web search,

    C. Rosset, D. Jose, G. Ghosh, B. Mitra, and S. Tiwary, “Optimizing query evaluations using reinforcement learning for web search,” inPro- ceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval. Ann Arbor, MI, USA: Association for Computing Machinery, Jul. 2018, pp. 1193–1196

    2018

  3. [3]

    Match plan generation in web search with parameterized action reinforcement learning,

    Z. Luo, L. Zhao, W. Cheng, S. Chen, Q. Chen, H. Xue, H. Wang, C. Liu, M. Yang, and L. Zhang, “Match plan generation in web search with parameterized action reinforcement learning,” inProceedings of the Web Conference 2021. Ljubljana, Slovenia: Association for Computing Machinery, Apr. 2021, pp. 1040–1052

    2021

  4. [4]

    Exploration in deep rein- forcement learning: A survey,

    P. Ladosz, L. Weng, M. Kim, and H. Oh, “Exploration in deep rein- forcement learning: A survey,”Information Fusion, vol. 85, pp. 1–22, Sep. 2022

    2022

  5. [5]

    Actor-critic reinforcement learning in the songbird,

    R. Chen and J. H. Goldberg, “Actor-critic reinforcement learning in the songbird,”Current Opinion in Neurobiology, vol. 65, pp. 1–9, Dec. 2020

    2020

  6. [6]

    Asynchronous methods for deep reinforcement learning,

    V . Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” inProceedings of The 33rd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. F. Balcan and K. Q. Weinberger, Eds., vol. 48. New York, New York, USA: PMLR, 2...

    2016

  7. [7]

    Averaged soft actor-critic for deep reinforcement learning,

    F. Ding, G. Ma, Z. Chen, J. Gao, and P. Li, “Averaged soft actor-critic for deep reinforcement learning,”Complexity, vol. 2021, pp. 1–16, 2021

    2021

  8. [8]

    Greedy actor-critic: A new conditional cross-entropy method for policy improvement,

    S. Neumann, S. Lim, A. Joseph, Y . Pan, A. White, and M. White, “Greedy actor-critic: A new conditional cross-entropy method for policy improvement,” 2023. [Online]. Available: https://arxiv.org/abs/1810.09103

    arXiv 2023

Show all 18 references
  1. [9]

    Drl trading with cpt actor and truncated quantile critics,

    M. J. R. Foo, N. S. Lesmana, and C. S. Pun, “Drl trading with cpt actor and truncated quantile critics,” inProceedings of the Fourth ACM International Conference on AI in Finance. New York, NY , USA: Association for Computing Machinery, 2023, p. 574–582. [Online]. Available: h...

    2023

  2. [10]

    Multi-pass q-networks for deep reinforcement learning with parameterised action spaces,

    C. J. Bester, S. D. James, and G. D. Konidaris, “Multi-pass q-networks for deep reinforcement learning with parameterised action spaces,”

  3. [11]

    Comparative analysis of parameterized action actor-critic reinforcement learning algorithms for web search match plan generation,

    U. Bapoo and C. N. Nyirenda, “Comparative analysis of parameterized action actor-critic reinforcement learning algorithms for web search match plan generation,” inProceedings of Tenth International Congress on Information and Communication Technology, X.-S. Yang, S. Sherratt, ...

    2025

  4. [12]

    Research on load frequency control of multi-microgrids in an isolated system based on the multi-agent soft actor-critic algorithm,

    L. L. Xie, Y . Li, P. Fan, L. Wan, K. Zhang, and J. Yang, “Research on load frequency control of multi-microgrids in an isolated system based on the multi-agent soft actor-critic algorithm,”IET Renewable Power Generation, vol. 18, no. 7, pp. 1230–1246, 2024. [Online]. Availabl...

    2024 doi

  5. [13]

    Multi- agent actor-critic for mixed cooperative-competitive environments,

    R. Lowe, Y . Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multi- agent actor-critic for mixed cooperative-competitive environments,” in Advances in Neural Information Processing Systems, I. Guyon, U. von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. V . N. Vishwanathan,...

    2017

  6. [14]

    Monotonic value function factorisation for deep multi-agent reinforcement learning,

    T. Rashid, M. Samvelyan, C. S. de Witt, G. Farquhar, J. Foerster, and S. Whiteson, “Monotonic value function factorisation for deep multi-agent reinforcement learning,”Journal of Machine Learning Research, vol. 21, no. 178, pp. 1–51, 2020. [Online]. Available: http://jmlr.org/...

    2020

  7. [15]

    The surprising effectiveness of PPO in cooperative multi-agent games,

    C. Yu, A. Velu, E. Vinitsky, J. Gao, Y . Wang, A. Bayen, and Y . Wu, “The surprising effectiveness of PPO in cooperative multi-agent games,” inAdvances in Neural Information Processing Systems, vol. 35. Curran Associates, Inc., 2022, pp. 24 611–24 624

    2022

  8. [16]

    Parametrized deep q-networks learning: Reinforcement learning with discrete-continuous hybrid action space,

    J. Xiong, Q. Wang, Z. Yang, P. Sun, L. Han, Y . Zheng, H. Fu, T. Zhang, J. Liu, and H. Liu, “Parametrized deep q-networks learning: Reinforcement learning with discrete-continuous hybrid action space,”

  9. [2018]

    Available: https://arxiv.org/abs/1810.06394

    [Online]. Available: https://arxiv.org/abs/1810.06394

    arXiv

  10. [2019]

    Available: https://arxiv.org/abs/1905.04388

    [Online]. Available: https://arxiv.org/abs/1905.04388

    1905 arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.