PIMAEX: Multi-Agent Exploration through Peer Incentivization

Claudia Linnhoff-Popien; Gerhard Stenzel; Johannes Tochtermann; Julian Sch\"onberger; Michael K\"olle; Philipp Altmann

REVIEW 4 major objections 5 minor 24 references

PIMAEX: Multi-Agent Exploration through Peer Incentivization

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Rewarding influence on peers improves multi-agent exploration

desk verdict A clean, incremental MARL contribution whose headline beta comparison is confounded by hyperparameter choices; the generalized reward and environment are citable, but the specific beta claim needs a controlled ablation. read the letter →

arxiv 2501.01266 v1 pith:Q2P7S2Z4 submitted 2025-01-02 cs.MA cs.AI

Michael Kölle , Johannes Tochtermann , Julian Schönberger , Gerhard Stenzel , Philipp Altmann , Claudia Linnhoff-Popien This is my paper · ORCID

classification cs.MAcs.AI

keywords multi-agentreinforcementlearningexplorationintrinsiccuriositysocialinfluencepeerincentivizationrandomnetworkdistillationsequentialdilemmacommunication

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a reward-shaping scheme that turns one agent's discovery of novel states into a reward for the agents that influenced it. The idea is a generalized social-influence reward with three weighted terms: policy influence, value influence, and a newly introduced term that multiplies an agent's influence on a peer by that peer's reward. The authors instantiate the scheme as PIMAEX, pair it with a communication-based actor-critic algorithm called PIMAEX-Communication, and evaluate it in a custom partially observable environment with deceptive rewards. They report that PIMAEX agents outperform both vanilla PPO and PPO augmented with random network distillation, with the $\beta$ variant achieving the highest team return and the lowest variance across seeds.

What carries the argument

The load-bearing object is the generalized multi-agent social-influence reward function of Eq. 4, $r_j = \sum_{k \neq j} [\alpha \cdot PI^\alpha_{j \to k} + \beta \cdot PI^\beta_{j \to k} \cdot r^w_k + \gamma \cdot VI^w_{j \to k}]$. PIMAEX is the special case where $r^w_k$ and $VI^w_{j \to k}$ are weighted sums of extrinsic and intrinsic (RND-based) rewards and values. The training side is PIMAEX-Communication, which adds a discrete communication head to each agent, samples counterfactual messages to build marginal policies and values, and computes influence on the actor side while intrinsic and PIMAEX rewards are computed on the learner side. The mechanism works by turning the communication channel into a channel of influence that the reward explicitly prices.

What would settle it

Train PIMAEX-Communication ($\beta$) and PPO+RND on a second, independently designed partially observable coordination task with sparse rewards, using the same hyperparameter search budget and seeds; if $\beta$ does not exceed RND in mean team return or variance reduction, the central empirical claim fails to generalize beyond its single environment.

Watch

Extended reading notes

Core claim

The paper's central claim is that multi-agent exploration can be improved by giving each agent a reward that depends on its causal influence on the exploration of others. Concretely, agent $j$ receives $r_j = \sum_{k \neq j} [\alpha \cdot PI^\alpha_{j \to k} + \beta \cdot PI^\beta_{j \to k} \cdot r^w_k + \gamma \cdot VI^w_{j \to k}]$, where the $\alpha$ term measures policy influence (the divergence or pointwise mutual information between a peer's informed and counterfactually marginalized policy), the $\gamma$ term measures value influence (the difference between a peer's informed and marginalized value), and the $\beta$ term—the paper's new contribution—multiplies influence by the peer's weighted reward, letting $j$ share in the outcome of actions it caused. The authors claim this $\beta$ term is what lifts overall return in their Consume/Explore task, and they further observe that the $\alpha$-only agent explores most while $\beta$ yields the most stable policies. The empirical support is an internal comparison between best-tuned models of each reward variant.

Load-bearing premise

The single Consume/Explore environment, designed by the authors, is a valid and representative testbed for multi-agent exploration, and the separate hyperparameter search run for each method does not inadvertently favor PIMAEX.

Editorial extensions

If this is right

Any actor-critic MARL algorithm can be augmented with PIMAEX by adding a communication policy, counterfactual message sampling, and the three-term reward, without changing the underlying policy updates.
The $\beta$ term's low variance suggests peer-incentivized rewards can act as a variance-reduction mechanism, which the paper ties to social influence's known stabilizing effect on policy gradients.
The Consume/Explore environment provides a tunable sequential social dilemma (credit-assignment difficulty, coordination threshold, resource abundance) for future MARL exploration studies.
The $\alpha$-only variant's strong exploration indicates that policy influence combined with individual intrinsic curiosity may be sufficient to drive exploration, even without sharing intrinsic returns.
PIMAEX $\beta$ agents specialize into explorer and consumer roles in pairs, suggesting the reward induces division of labor within the team.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct testable extension is to re-run the same comparison on a second partially observable coordination task with a larger state space; if the $\beta$ advantage disappears, the claim is limited to the Consume/Explore setting.
The low-variance property of $\beta$ could be investigated mechanically by logging per-step PIMAEX rewards and advantage estimates during training to see whether the $\beta$ term compresses the spread of returns across agents.
Since the generalized reward separates influence measurement from the reward stream, one could substitute learned or learned-models-based influence measures and preserve the framework; the paper does not test this.
The pairwise specialization pattern suggests a testable prediction: in tasks with complementary roles, peer incentives should amplify role differentiation, measurable with a role-divergence metric.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is an incremental but competently made contribution. The generalized social-influence reward in Eq. (4) is a clean unification of Jaques et al. and Wang et al., and the beta term forcing influence times peer reward is new, as far as I can tell. The Consume/Explore environment is small but thoughtfully built to combine deceptive reward, credit assignment, and coordination. The authors also do something rare: the conclusion states the limitations without spin and even reports that alpha, not beta, produced the most exploratory behavior. That honesty buys real goodwill.

The experimental section is the weak part, and the stress-test note is correct. Table 5 is not an ablation. PIMAEX beta differs from PIMAEX alpha in both the reward term and two hyperparameters: communication loss weight (1.0 vs 0.0758) and influence measure (PMI vs KLD). So the claim that the beta term achieves the highest return cannot be attributed to beta alone; it could be the larger communication loss weight or the choice of PMI. The missing cells (alpha with PMI and weight 1.0, beta with KLD and weight 0.0758) are exactly the cells that would settle it. The comparison against PPO+RND also conflates the reward with the communication channel, since that baseline lacks communication. That is less serious because all PIMAEX variants share the channel, but it still qualifies the headline.

Other soft spots: three seeds, no significance tests, no released code or data, and one self-designed environment with a small state/action space and short training. Those are addressable, and the authors already list most of them. If the central claim were just that the PIMAEX formulation can work in this toy setting, the evidence would be enough. It is the beta-specific claim that exceeds the data.

Who should read it: anyone working on influence-based rewards or multi-agent exploration might want the unified formulation and the environment. I would not cite it as evidence that beta improves return, but I would cite Eq. (4) and Consume/Explore as related work. A serious editor should send this to review rather than desk-reject: the idea is clean, the writeup is careful, the related work is fair, and the confounds are fixable with a modest set of additional experiments. I would ask for the controlled ablation, a code release, and ideally a second environment before accepting.

Referee Report

4 major / 5 minor

Summary. The paper proposes PIMAEX, a peer-incentivized multi-agent exploration reward, and PIMAEX-Communication, a training algorithm with a discrete communication channel and counterfactual influence estimation. The reward is a weighted sum of three influence terms: alpha (policy influence), beta (policy influence multiplied by the peer's reward, claimed as the novel contribution), and gamma (value influence). The method is evaluated on a purpose-built partially observable environment, Consume/Explore, with PPO and PPO+RND baselines. Single-term variants PIMAEX alpha, beta, and gamma are compared. The reported results show that the best PIMAEX beta configuration achieves the highest joint episode return and lower variance than baselines in this environment. The conclusion is appropriately narrowed to overall return in Consume/Explore compared with baselines without social influence, but the attribution of this advantage to the beta term is not established by the reported experiments.

Significance. If the beta-term advantage were cleanly established, this would be a modest but useful contribution to multi-agent exploration, since the beta term is simple and the generalized reward formulation is a convenient unification of prior influence-based rewards. The paper is clearly written, the environment is described transparently with parameters and hyperparameters, and the authors explicitly acknowledge the main scope limitations. However, the paper provides no code or other reproducibility artifact, the evaluation is on a single self-designed environment, and the experimental design does not isolate the beta term. The main value of the paper at this stage is as a proposal of a reward form and an environment, not as a validated mechanism.

major comments (4)

[Section 6, Table 5] The comparison between PIMAEX alpha and PIMAEX beta is not a controlled ablation, and this is load-bearing for the paper's main claim that the novel beta term improves return. In Table 5, PIMAEX beta uses communication loss weight 1.0 and PMI as the influence measure, while PIMAEX alpha uses communication loss weight 0.0758 and KLD. Thus two hyperparameters change together with the reward term, and the reported advantage of beta over alpha could be due to the 13.2x larger communication loss weight, the choice of PMI, or their interaction. The missing cells—alpha with PMI and loss weight 1.0, and beta with KLD and loss weight 0.0758—are needed to attribute the improvement to the beta term. As it stands, Sections 6 and 7 overstate the evidence for the beta contribution.
[Section 5.2 and Sections 6-7] The paper reports only the best-performing hyperparameter configuration for each method, selected on the same evaluation environment, and provides no statistical significance tests. With three seeds per model and no per-seed confidence intervals or hypothesis tests, statements such as "PIMAEX beta exhibits significantly less standard deviation than other methods" (Section 6) are not supported. Moreover, Figures 1 and 3 are described as the "best training run of each agent category" while the methodology says results were averaged over three seeds; this ambiguity affects which quantity is being plotted. The authors should report all searched configurations or a clear selection procedure, use a validation/test split if hyperparameters are tuned, and provide per-seed results or significance tests for the central return comparison.
[Section 4 and Section 6] The comparison against PPO+RND conflates the PIMAEX reward with the presence of the communication channel. PPO+RND has no communication channel, so the headline comparison "PIMAEX reward with PIMAEX-Communication outperforms those that do not" does not isolate the reward function's contribution. A more appropriate control would be a PPO+RND agent with the same communication channel and counterfactual machinery but without the social influence reward, or a PIMAEX-Communication variant with the reward terms set to zero. This issue is less sharp than the alpha-versus-beta confound because all PIMAEX variants share the communication channel, but it affects the abstract's claim that agents using the PIMAEX reward outperform those that do not.
[Section 7] The paper's own limitation statement acknowledges that the evaluation was limited to a single task with small state and action spaces and short training. This is not by itself an error, but the central claim of the paper is empirical, so the lack of any second environment or scalability evidence means the general conclusion is necessarily tentative. The abstract should be aligned with the conclusion's narrow phrasing: the demonstrated results are for Consume/Explore, not for multi-agent exploration generally. At minimum, the title and abstract should not imply broader generality than the single environment supports.

minor comments (5)

[Equation (4)] The notation PI^alpha and PI^beta is confusing because alpha and beta are also the weights in the same equation. The superscripts appear to denote policy influence variants, but the text does not define a distinct PI^alpha versus PI^beta; please clarify by renaming the influence measures or the weights.
[Equations (5) and (6)] The symbols beta_env and beta_int in Equation (5) reuse the Greek letter beta, which already denotes the second reward term in Equation (4). This is a readability issue; consider using different coefficient names for the intrinsic/extrinsic mixing weights.
[Section 5.2] The hyperparameter search for PPO+RND and PIMAEX-Communication lists the searched values in Tables 3 and 4 but does not specify the search procedure, the number of trials, or the criterion for selecting the "best" configuration. Reporting this would help assess the risk of overfitting the reported result to the evaluation environment.
[Abstract] The abstract states that "agents using the PIMAEX reward with PIMAEX-Communication outperform those that do not" without noting that this holds for a single environment and for best-tuned configurations. The conclusion is more careful; please align the abstract with the qualified claim.
[Figure 4(a)] Figure 4(a) reports per-episode exploration state space coverage, but the text notes that agents cover only a small portion of the agent state space within one episode and therefore reports local state space coverage in Figure 4(b). It would help to state explicitly why exploration state space coverage remains meaningful within an episode despite the small agent-state coverage.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the PIMAEX reward is an explicit heuristic and the reported comparisons are empirical model-selection results, not fitted predictions.

full rationale

The paper's derivation chain consists of defining the generalized social influence reward in Eq. 4 as a weighted sum of policy influence, peer-reward-weighted influence, and value influence, then instantiating it as the PIMAEX reward. This is a stated ansatz, not a result derived from the evaluation data; nothing in the definition presupposes the empirical outcome. The empirical claim in Section 6 is that, after hyperparameter search, the best PIMAEX-beta configuration reaches the highest overall return on the authors' Consume/Explore task. That is a fitted comparison standard in RL papers, and the paper does not relabel it as a prediction: the abstract says 'empirically demonstrate', and the conclusion explicitly acknowledges the single-task, small-state-space, short-training scope. The hyperparameter differences between alpha/beta/gamma in Table 5 (communication loss weight and influence measure) are a real experimental confound, but they do not make any quantity equal to another by construction, so they are a validity concern rather than circularity. The only self-citation (Schmid et al. 2021) appears in a related-work list of peer-incentivization methods and is not load-bearing for the central claim. No self-definitional, fitted-input-as-prediction, or self-citation-chain reduction is present.

Assumptions & free parameters 6 free parameters · 3 assumptions · 3 invented entities

The central result depends on several hyperparameters tuned on the single test environment, on the authors' own environment as a benchmark, and on the assumption that counterfactual sampling approximates true influence. The PIMAEX reward itself is a newly proposed heuristic, not a derived law. There are no new physical entities, only the new reward function, algorithm, and environment, none of which currently have independent evidence outside the paper's own experiments.

free parameters (6)

alpha term weight = 1.0 (PIMAEX alpha)
Selected via hyperparameter search (Table 5) to maximize performance in the Consume/Explore environment.
beta term weight = 1.0 (PIMAEX beta)
Selected via hyperparameter search; the main novel term is weighted at 1.0 in the best beta configuration.
gamma term weight = 0.01 (PIMAEX gamma)
Tuned to a small value in the best gamma configuration.
communication loss weight = 1.0 (beta), 0.0758 (alpha, gamma)
Selected via hyperparameter search (Tables 4 and 5).
extrinsic/intrinsic reward coefficients = (1.0, 0.5) for PIMAEX agents; (2.0, 1.0) or (1.0, 0.5) for RND baselines
Chosen via hyperparameter search (Table 3 and 4).
policy influence measure = KLD for alpha, PMI for beta, none for gamma
Selected per configuration during the search (Table 5).

assumptions (3)

domain assumption The Consume/Explore environment is a valid proxy for multi-agent exploration problems with deceptive rewards.
The environment was designed by the authors and is the sole testbed; its representativeness is asserted but not externally validated. Section 5.1.
domain assumption Sampling counterfactual messages uniformly approximates the marginal policy and value functions needed for influence estimation.
Section 4.1 and 4.2 use M = |A_comm| - 1 counterfactual samples per agent and average, assuming this yields a sufficiently accurate marginal for reward computation.
standard math Standard RL convergence assumptions for PPO and RND hold in the asynchronous actor-learner setup.
The methods rely on the established PPO and RND algorithms as provided in acme, with modifications described in Section 4.3.

invented entities (3)

PIMAEX reward function
purpose: To incentivize agents to influence peers toward novel states by combining policy influence, reward sharing, and value influence.
The reward is a new heuristic formula (Eq. 4-6); its effectiveness is only demonstrated on the authors' own environment with tuned hyperparameters.
Consume/Explore environment
purpose: To evaluate exploration versus exploitation and credit assignment in a partially observable multi-agent setting with deceptive rewards.
This is a new synthetic environment introduced in Section 5.1; no external benchmark or third-party validation is provided.
PIMAEX-Communication algorithm
purpose: To enable agents to influence each other via a discrete message channel and compute PIMAEX rewards through counterfactual reasoning.
The algorithm is a modification of existing actor-critic frameworks; it is described in Section 4 but no independent implementation is released.

how reviews work

0 comments

Cite this review

Pith. "Pith review of PIMAEX: Multi-Agent Exploration through Peer Incentivization." pith.science (2026). https://pith.science/paper/Q2P7S2Z4

@misc{pith2026250101266,
  author       = {Pith},
  title        = {Pith review of: PIMAEX: Multi-Agent Exploration through Peer Incentivization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q2P7S2Z4}},
  note         = {Machine review of arXiv:2501.01266}
}

read the original abstract

While exploration in single-agent reinforcement learning has been studied extensively in recent years, considerably less work has focused on its counterpart in multi-agent reinforcement learning. To address this issue, this work proposes a peer-incentivized reward function inspired by previous research on intrinsic curiosity and influence-based rewards. The \textit{PIMAEX} reward, short for Peer-Incentivized Multi-Agent Exploration, aims to improve exploration in the multi-agent setting by encouraging agents to exert influence over each other to increase the likelihood of encountering novel states. We evaluate the \textit{PIMAEX} reward in conjunction with \textit{PIMAEX-Communication}, a multi-agent training algorithm that employs a communication channel for agents to influence one another. The evaluation is conducted in the \textit{Consume/Explore} environment, a partially observable environment with deceptive rewards, specifically designed to challenge the exploration vs.\ exploitation dilemma and the credit-assignment problem. The results empirically demonstrate that agents using the \textit{PIMAEX} reward with \textit{PIMAEX-Communication} outperform those that do not.

Figures

Figures reproduced from arXiv: 2501.01266 by the authors.

**Figure 3.** Final state space coverage of the best training run of each agent category, PPO, PPO+RND, PPO+RND+PIMAEX α/β/γ, for per-agent state space and (global) exploration state space. Results are averaged over three seeds, plots show mean and standard deviation. (a) Per-episode exploration state space coverage, from evaluation runs (b) Per-episode local agent state space coverage, from evaluation runs [PITH_FULL_IMAGE:figu… view at source ↗

**Figure 1.** Overall return per episode, i.e., sum of all individ [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗

Figures from the paper (2 more)

**Figure 6.** Figure 6: Number of simultaneous consume and explore [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: Final yield level and number of environment steps [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 19 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
[2]

G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R

Bellemare, M. G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. (2016). Unifying count-based exploration and intrinsic motivation

work page 2016
[3]

J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q

Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q. (2018). JAX : composable transformations of P ython+ N um P y programs

2018
[4]

Burda, Y., Edwards, H., Storkey, A., and Klimov, O. (2018). Exploration by random network distillation

work page 2018
[5]

Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., Legg, S., and Kavukcuoglu, K. (2018). Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures

work page 2018
[6]

Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. (2017). Counterfactual multi-agent policy gradients

work page 2017
[7]

Fu, W., Yu, C., Xu, Z., Yang, J., and Wu, Y. (2022). Revisiting some common practices in cooperative multi-agent reinforcement learning

work page 2022
[8]

Hoffman, M. W., Shahriari, B., Aslanides, J., Barth-Maron, G., Momchev, N., Sinopalnikov, D., Sta\'nczyk, P., Ramos, S., Raichuk, A., Vincent, D., Hussenot, L., Dadashi, R., Dulac-Arnold, G., Orsini, M., Jacq, A., Ferret, J., Vieillard, N., Ghasemipour, S. K. S., Girgin, S., Pietquin, O., Behbahani, F., Norman, T., Abdolmaleki, A., Cassirer, A., Yang, F.,...

arXiv 2020

Show all 24 references

[9]

A., Strouse, D., Leibo, J

Jaques, N., Lazaridou, A., Hughes, E., Gulcehre, C., Ortega, P. A., Strouse, D., Leibo, J. Z., and de Freitas, N. (2018). Social influence as intrinsic motivation for multi-agent deep reinforcement learning

2018
[10]

Z., Zambaldi, V., Lanctot, M., Marecki, J., and Graepel, T

Leibo, J. Z., Zambaldi, V., Lanctot, M., Marecki, J., and Graepel, T. (2017). Multi-agent reinforcement learning in sequential social dilemmas

2017
[11]

Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I. (2017). Multi-agent actor-critic for mixed cooperative-competitive environments

2017
[12]

G., Oord, A

Ostrovski, G., Bellemare, M. G., Oord, A. v. d., and Munos, R. (2017). Count-based exploration with neural density models

2017
[13]

Oudeyer, P.-Y., Kaplan, F., and Hafner, V. V. (2007). Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation , 11(2):265--286

2007
[14]

A., and Darrell, T

Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. (2017). Curiosity-driven exploration by self-supervised prediction

2017
[15]

Peng, P., Wen, Y., Yang, Y., Yuan, Q., Tang, Z., Long, H., and Wang, J. (2017). Multiagent bidirectionally-coordinated nets: Emergence of human-level coordination in learning to play starcraft combat games

2017
[16]

S., Farquhar, G., Foerster, J., and Whiteson, S

Rashid, T., Samvelyan, M., de Witt, C. S., Farquhar, G., Foerster, J., and Whiteson, S. (2018). Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning

2018
[17]

Schmid, K., Belzner, L., and Linnhoff-Popien, C. (2021). Learning to penalize other learning agents. In Proceedings of the Artificial Life Conference 2021 , volume 2021. MIT Press

2021
[18]

Schmidhuber, J. (1991). A possibility for implementing curiosity and boredom in model-building neural controllers. In Proceedings of the First International Conference on Simulation of Adaptive Behavior on From Animals to Animats , page 222–227, Cambridge, MA, USA. MIT Press

1991
[19]

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms

2017
[20]

C., Levine, S., and Abbeel, P

Stadie, B. C., Levine, S., and Abbeel, P. (2015). Incentivizing exploration in reinforcement learning with deep predictive models

2015
[21]

Sukhbaatar, S., Szlam, A., and Fergus, R. (2016). Learning multiagent communication with backpropagation

2016
[22]

Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, X., Duan, Y., Schulman, J., De Turck, F., and Abbeel, P. (2016). \#exploration: A study of count-based exploration for deep reinforcement learning

2016
[23]

Wang, T., Wang, J., Wu, Y., and Zhang, C. (2019). Influence-based multi-agent exploration

2019
[24]

Yang, J., Li, A., Farajtabar, M., Sunehag, P., Hughes, E., and Zha, H. (2020). Learning to incentivize other learning agents

2020

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

[2] [2]

G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R

Bellemare, M. G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. (2016). Unifying count-based exploration and intrinsic motivation

work page 2016

[3] [3]

J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q

Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q. (2018). JAX : composable transformations of P ython+ N um P y programs

2018

[4] [4]

Burda, Y., Edwards, H., Storkey, A., and Klimov, O. (2018). Exploration by random network distillation

work page 2018

[5] [5]

Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., Legg, S., and Kavukcuoglu, K. (2018). Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures

work page 2018

[6] [6]

Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. (2017). Counterfactual multi-agent policy gradients

work page 2017

[7] [7]

Fu, W., Yu, C., Xu, Z., Yang, J., and Wu, Y. (2022). Revisiting some common practices in cooperative multi-agent reinforcement learning

work page 2022

[8] [8]

Hoffman, M. W., Shahriari, B., Aslanides, J., Barth-Maron, G., Momchev, N., Sinopalnikov, D., Sta\'nczyk, P., Ramos, S., Raichuk, A., Vincent, D., Hussenot, L., Dadashi, R., Dulac-Arnold, G., Orsini, M., Jacq, A., Ferret, J., Vieillard, N., Ghasemipour, S. K. S., Girgin, S., Pietquin, O., Behbahani, F., Norman, T., Abdolmaleki, A., Cassirer, A., Yang, F.,...

arXiv 2020