Pith. sign in

REVIEW 3 major objections 5 minor 71 references

Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Story

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that in parallel reinforcement learning, replacing $N$ identical agents with $N$ specialized policies that jointly maximize the entropy of their combined state distribution yields faster concentration of state entropy…

desk verdict The parallel entropy objective is a natural framing, but the central concentration theorem is unproven—the proof relies on a false Chernoff bound, so the paper's headline claim doesn't hold. read the letter →

arxiv 2505.01336 v2 pith:GJLXTN46 submitted 2025-05-02 cs.LG

Vincenzo De Paola , Riccardo Zamboni , Mirco Mutti , Marcello Restelli This is my paper · ORCID
classification cs.LG
keywords parallelreinforcementlearningstateentropymaximizationexplorationpolicygradientagentdiversityofflineconcentrationinequalitiesMarkovdecisionprocesses
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether the usual parallel reinforcement learning recipe of running $N$ identical copies of the same policy is leaving sample efficiency on the table. It proposes a parallel state entropy maximization objective in which the $N$ agents are treated as one mixture policy, so the total entropy is the sum of individual entropies plus an explicit inter-agent diversity term. Its central claim is that specialized policies with low individual entropy but distinct state coverage concentrate faster than one high-entropy policy collecting the same number of trajectories, and it supports that claim with an exponential concentration bound whose rate depends on $H^2(d_\pi)/\mathrm{Var}(d_\pi)$. On this basis it introduces PGPSE, a centralized policy gradient that rewards each agent by the entropy of the joint empirical state distribution, and reports gridworld experiments where parallel agents beat a single agent matched for total trajectories, both in state entropy and in offline Q-learning success. If the claim holds, parallel simulation should be used not only to multiply throughput but also to create data diversity.

What carries the argument

The load-bearing object is the parallel mixture distribution $d_{\pi_p}(s)=(1/m)\sum_i d_{\pi_i}(s)$ and its finite-trials counterpart $J_p(\pi_p)=\mathbb{E}_{d_{n,p}\sim p_{\pi_p}}[H(d_{n,p})]$. The entropy decomposition $H(d_{\pi_p}) = \frac{1}{m}\sum_i H(d_{\pi_i}) + \frac{1}{m}\sum_i D_{\mathrm{KL}}(d_{\pi_i}\|d_{\pi_p})$ turns diversity into an explicit term in the objective. Theorem 4.1's concentration bound is the mechanism: it ties the number of samples needed for the empirical entropy to approach the true entropy to $H^2(d_\pi)/\mathrm{Var}(d_\pi)$, so low-entropy specialized agents concentrate their own estimates quickly while the KL term supplies the mixture's total entropy. PGPSE then carries the machinery into optimization: the gradient for agent $i$ is $\mathbb{E}\left[\left(\sum_t \nabla_\theta\log\pi_{\theta_i}(a_t^i|s_t^i)\right) H(d_p)\right]$, an unbiased REINFORCE-style estimator in which every agent is pushed by the entropy of the shared empirical state distribution.

What would settle it

Evaluate Eq. (4) directly. For a Bernoulli random variable with $p=0.2$, take $\epsilon=0.1$ and $n=1000$; the true probability $P(|\hat p - p|>\epsilon)$ computed from the binomial CDF is about $6\times 10^{-13}$, while the claimed upper bound $2\exp(-n\epsilon^2/(2p(1-p)))$ is about $5.4\times 10^{-14}$. The inequality fails, so the proof of Theorem 4.1 as written does not stand.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the entropy of the empirical state distribution in a parallel system concentrates faster when the $m$ agents follow specialized, low-entropy policies than when a single policy is asked to cover the whole state space with the same total number of samples. Writing the mixture state distribution as $d_{\pi_p} = (1/m)\sum_i d_{\pi_i}$, the entropy decomposes as $H(d_{\pi_p}) = (1/m)\sum_i H(d_{\pi_i}) + (1/m)\sum_i D_{\mathrm{KL}}(d_{\pi_i}\|d_{\pi_p})$, so a high-entropy mixture can be built from individually low-entropy, mutually distinct agents. Theorem 4.1 bounds $\Pr(H(d_\pi)-H(d_n)>\epsilon)$ by $2S\exp\left(-n\epsilon^2\mathrm{Var}(d_\pi)/(2S^3H^2(d_\pi))\right)$, and since the sample bound scales with $H^2(d_\pi)/\mathrm{Var}(d_\pi)$, low-entropy distributions need fewer samples to concentrate; the paper reads this as the quantitative mechanism by which specialized parallel exploration stabilizes faster than homogeneous exploration. The accompanying PGPSE algorithm gives each agent the policy-gradient score of its own trajectory weighted by the entropy of the joint empirical state distribution, so agents learn to cover different states without an explicit diversity penalty.

Load-bearing premise

The theoretical claim that parallel specialization concentrates faster rests on a tail bound for Bernoulli estimates that is not valid for all frequencies; if that bound is false, the theorem's rates and the sample-complexity comparison do not follow.

Editorial extensions

If this is right

  • Running $m$ specialized agents with the same total interaction budget as one agent should produce higher state entropy than $m$ identical copies, because mixture entropy contains an explicit diversity term.
  • For strategies with disjoint state supports, the mixture entropy gains an additive $\log m$, so sample complexity for covering a state space should drop by roughly a factor of $m$ rather than staying flat.
  • Datasets collected by parallel specialized agents should make offline RL more robust to goal position, since broader and lower-variance coverage translates into success at more goals from the same number of trajectories.
  • Because the concentration theorem is stated for any categorical distribution, the same fast-concentration argument applies to trajectory distributions, pointing toward trajectory-entropy parallel exploration as well as state entropy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A corrected concentration proof would need to replace the quadratic tail bound in Eq. (4) with the exact KL exponent, and whether the 'faster for low-entropy distributions' conclusion survives with a valid bound is an open question.
  • The mixture-entropy decomposition suggests a direct scaling law: if agents keep their supports disjoint, the sample size needed to reach entropy $\epsilon$-close to the optimum should scale like $|S|/(mT)$, which can be tested in gridworlds with much larger state spaces.
  • PGPSE can be seen as implicit skill discovery in which each agent is a skill; unlike mutual-information skill methods it needs no discriminator, but it also has no explicit mechanism to stop two agents from collapsing to the same support, so measuring support overlap across seeds is a natural test.
  • The empirical observation that parallel agents become quasi-deterministic while a single agent stays stochastic suggests a practical diagnostic: specialization should be measurable as the entropy of each agent's own action distribution.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies maximum state entropy exploration with parallel agents operating in independent environment copies. It defines a parallel objective that averages the agents' state distributions, proposes a centralized policy-gradient algorithm (PGPSE) that optimizes the entropy of the empirical mixture distribution, and claims an original concentration analysis showing that specialized parallel sampling distributions concentrate faster than single-policy exploration. The empirical section reports gridworld experiments in which parallel agents achieve higher normalized state entropy and support size than a single agent with the same total number of trajectories, and in which datasets collected by parallel agents improve offline Q-learning success rates.

Significance. If the concentration claims were valid, the paper would make a useful contribution by providing a principled argument for specializing parallel agents in state-entropy exploration, a question of practical interest in RL. The proposed algorithm is simple and its empirical evaluation, while limited to small gridworlds, does suggest that parallel specialization can improve data diversity. However, the theoretical contribution is not established: the proof of Theorem 4.1 relies on a false Bernoulli tail bound, and the advertised 'faster rates for specialized parallel sampling distributions' is never formally stated or proven. Without the concentration analysis, the paper is essentially an empirical study of a REINFORCE-style algorithm on toy domains, which is substantially weaker than the claims made in the abstract and Section 1.

major comments (3)
  1. [Appendix A, Eq. (4)] The proof of Theorem 4.1 (Appendix A, Eq. (4)) uses the inequality P(|p_i - \hat p_i| > \epsilon) <= 2 exp(-n \epsilon^2 / (2 p_i(1-p_i))), justified by the claim that DKL(p_i+\epsilon || p_i) >= \epsilon^2/(2 p_i(1-p_i)). This is not true globally. For example, with p_i = 0.1 and \epsilon = 0.09, DKL(0.19||0.1) = 0.0366 while \epsilon^2/(2 p_i(1-p_i)) = 0.045; at n = 100, the sum of the two Chernoff exponents is about 0.0266, which exceeds the claimed upper bound 2 exp(-4.5) \approx 0.0222. Therefore Eq. (4) is false as stated, and consequently Eq. (5), the sample-complexity statement, and Theorem 4.1 are not established. Since Section 4's 'Primary Insight' and the conclusion that parallel exploration 'enables faster concentration' rest on this theorem, the paper's central theoretical claim is unproven.
  2. [Section 4, 'Primary Insight'] The abstract, Section 1, and Section 3 advertise 'faster rates for specialized parallel sampling distributions,' but Section 4 contains no theorem or formal statement about the concentration of the empirical mixture distribution d_{n,p}. The only concentration result, Theorem 4.1, applies to a single categorical distribution. The paragraph after the entropy decomposition argues heuristically that low-entropy components concentrate faster, and that disjoint supports increase mixture entropy by log(m), but no bound is derived for H(d_{\pi_p}) - H(d_{n,p}). The 'Primary Insight' is a post-hoc rationalization rather than a derivation. Thus the advertised claim about parallel specialization is not supported by the theoretical results presented.
  3. [Appendix A, proof of Theorem A.1] Independently of the flawed tail bound, the step that converts Eq. (5) into the final bound is also unjustified. The proof asserts min_i p_i(1-p_i)/h^2(p_i) >= min_i p_i(1-p_i)/H^2(p_i) = max_i p_i(1-p_i)/max_i H^2(p_i) >= Var(p)/(K H^2(p)), but the equality 'min of a ratio = ratio of maxima' is not generally valid, and the claim that the ratio is minimized at the maximum-entropy component is not established. Even with a valid concentration inequality for each component, the final sample-complexity bound of Theorem 4.1 would not follow from the argument as written.
minor comments (5)
  1. [Algorithm 1, line 7] The empirical state distribution d_p(s) in Algorithm 1 is not normalized correctly: each trajectory contributes T states, so the denominator should be k*m*T (or the average over state visits), not k*m. As written, d_p does not generally sum to 1.
  2. [Section 5, paragraph 2] The text refers to 'Eq. 3.3' when it means Definition 3.3 (the finite-trials parallel objective); the cross-reference is incorrect.
  3. [Section 4, entropy decomposition] The sums in the entropy decomposition are written as summing from i=0 to m, but the agents are indexed from 1 to m elsewhere in the paper; this is a minor indexing inconsistency.
  4. [Throughout] There are several typos and formatting issues, including 'Intitute' in the author affiliation, 'Toghether' in the Related Work section, 'LenghT' and 'paralell' in Appendices B and C, and 'PT-1 t=1' in Algorithm 1 where the sum should start at t=0.
  5. [Appendix C] The text says the code is 'made available at the following link,' but no URL is provided; the link should be included for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's algorithm, objective, and experimental comparisons are not built from the quantities they claim to predict; the disputed concentration bound is a mathematical claim, not a fit or a self-citation.

full rationale

The paper's principal claims are the definition of a parallel state-entropy objective, a policy-gradient derivation for that objective, an analytic concentration bound, and empirical comparisons. None of these reduces to its inputs by construction. The parallel objective J_p is defined from the per-agent mixture distribution and optimized directly via an unbiased score-function estimator (Section 5), with no parameter fitted to a target entropy value. Theorem 4.1 is stated as a general high-probability bound and is not calibrated to the experiments; the 'faster concentration' passage in Section 4 is an interpretation of the bound for low-entropy component distributions, not a fitted prediction. The entropy decomposition H(d_{π_p}) = (1/m)ΣH(d_{π_i}) + (1/m)ΣD_KL(d_{π_i}||d_{π_p}) is a standard identity and is used only to motivate the design, not to define away the claimed benefit. Citations to the authors' prior work (Mutti and Restelli 2020; Mutti et al. 2021, 2022, 2023; Zamboni et al. 2024, 2025) are background, related-work, or a definition of the finite-trials objective; no load-bearing theorem is imported from those citations to forbid alternatives. The reader and skeptic are right that the proof of Theorem 4.1 contains an invalid Chernoff-type inequality at Eq. (4), so the theoretical support for the abstract's 'faster rates' is not established; however, that is a mathematical correctness problem, not circularity. The paper's derivation chain is self-contained, so no circular step is present.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central theoretical claim rests on a concentration bound whose proof uses an invalid probability inequality. The algorithm itself relies on standard REINFORCE and is well-defined. No new physical or model entities are introduced. The two hyperparameters listed are standard but unexamined.

free parameters (2)
  • Learning rate schedule = alpha0=0.1, exponential decay lambda=0.999
    Chosen by hand as standard in tabular policy gradient; no sensitivity analysis is reported.
  • Batch size B = 40
    Chosen by hand; the paper reports no tuning procedure, so the empirical results may depend on this value.
assumptions (3)
  • domain assumption The finite-horizon Parallel MDP model with independent environment copies and discrete finite state and action spaces is a valid abstraction for parallel exploration.
    Introduced in Section 2 and 3 as the interaction model; the algorithm and experiments assume this structure.
  • domain assumption The entropy functional H is beta-smooth, B-bounded, and satisfies the quadratic descent lemma used in the Frank-Wolfe analysis in Appendix A.2.
    Invoked in the proof of Theorem A.2; these smoothness assumptions are standard in convex optimization but are not verified for the specific state entropy objective.
  • ad hoc to paper The Bernoulli tail bound P(|p_i - p_hat_i| > epsilon) <= 2 exp(-n epsilon^2 / (2 p_i(1-p_i))) is a valid upper bound.
    This is Eq. (4) in Appendix A, used to prove Theorem 4.1. The inequality is not true in general for small p_i with positive deviations, so the proof of the theorem is invalid.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Story." pith.science (2026). https://pith.science/paper/GJLXTN46

@misc{pith2026250501336,
  author       = {Pith},
  title        = {Pith review of: Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Story},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GJLXTN46}},
  note         = {Machine review of arXiv:2505.01336}
}
abstract

Parallel data collection has redefined Reinforcement Learning (RL), unlocking unprecedented efficiency and powering breakthroughs in large-scale real-world applications. In this paradigm, $N$ identical agents operate in $N$ replicas of an environment simulator, accelerating data collection by a factor of $N$. A critical question arises: \textit{Does specializing the policies of the parallel agents hold the key to surpass the $N$ factor acceleration?} In this paper, we introduce a novel learning framework that maximizes the entropy of collected data in a parallel setting. Our approach carefully balances the entropy of individual agents with inter-agent diversity, effectively minimizing redundancies. The latter idea is implemented with a centralized policy gradient method, which shows promise when evaluated empirically against systems of identical agents, as well as synergy with batch RL techniques that can exploit data diversity. Finally, we provide an original concentration analysis that shows faster rates for specialized parallel sampling distributions, which supports our methodology and may be of independent interest.

Figures

Figures reproduced from arXiv: 2505.01336 by the authors.

Figure 1
Figure 1. The top row (a–d) shows the progression of normalized state entropy across different environments, while the bottom row (e–h) depicts the corresponding size of the support of the entropy. Each plot shows the performance of parallel agents (2 or 6) against single agents taking 2 and 6 trajectories, respectively. We report average results and standard deviation over 5 independent runs. Figures 5, 6). One family (refer… view at source ↗
Figure 2
Figure 2. Entropy evaluation of the dataset generated by sampling from parallel policies, compared against both random and single￾baselines across different environments. With the notation K′ = mK we denoted the set in which the single agent had access to a multiple m of trajectories available to the parallel agents. We report average and standard deviation over 5 independent runs. hibits minimal variance across runs, indicat… view at source ↗
Figure 3
Figure 3. Offline Q-learning success rate for various goal positions: Each subplot depicts the agent’s ability to reach a goal state posi￾tioned at a different location (one for each state). The results refer to datasets obtained over 5 independent runs. offline RL success rate on each dataset relabeled, one for every possible goal state, ordering the results according to their respective success rate. Notably, a dataset coll… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The plot shows that H(pi) and Var(pi) are concave symmetric function with their maximum located at pi = 0.5, while Var(pi) H(pi) 2 is a convex symmetric function with its minimum located at pi = 0.5. We now complete the proof by combining the results in Eqs (2), (3), a…
Figure 5
Figure 5. Figure 5: Two rooms gridworld, with starting position in the corridor and goal state in the left bottom corner [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 7
Figure 7. Figure 7: Performance of four parallel agents. The top row (a–d) shows the progression of normalized state entropy across different environments, while the bottom row (e–h) depicts the corresponding support size dynamics. Each plot illustrates the performance of parallel agents …
Figure 8
Figure 8. Figure 8: Dataset extrapolation based on the Maze environments. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Offline Result of Gained Experience 20 [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Analysis of stochasticity and determinism of parallel vs. single agents. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

71 extracted references · 50 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    V., Christianos, F., and Sch\"afer, L

    Albrecht, S. V., Christianos, F., and Sch\"afer, L. M ulti- A gent R einforcement L earning: Foundations and modern approaches . MIT Press, 2024

    work page 2024

  3. [3]

    and Arjun, C

    Alfredo, C. and Arjun, C. Efficient parallel methods for deep R einforcement L earning. In The Multi-disciplinary Conference on R einforcement L earning and Decision Making (RLDM) , pp.\ 1--6, 2017

    work page 2017

  4. [4]

    R einforcement L earning and optimal control , volume 1

    Bertsekas, D. R einforcement L earning and optimal control , volume 1. Athena Scientific, 2019

    work page 2019

  5. [5]

    Explore, discover and learn: Unsupervised discovery of state-covering skills

    Campos, V., Trott, A., Xiong, C., Socher, R., Gir \'o -i Nieto, X., and Torres, J. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, 2020

    work page 2020

  6. [6]

    Society of agents: Regret bounds of concurrent thompson sampling

    Chen, Y., Dong, P., Bai, Q., Dimakopoulou, M., Xu, W., and Zhou, Z. Society of agents: Regret bounds of concurrent thompson sampling. Advances in Neural Information Processing Systems, 35: 0 7587--7598, 2022

    work page 2022

  7. [7]

    and Van Roy, B

    Dimakopoulou, M. and Van Roy, B. Coordinated exploration in concurrent R einforcement L earning. In International Conference on Machine Learning, pp.\ 1271--1279. PMLR, 2018

    work page 2018

  8. [8]

    Scalable coordinated exploration in concurrent R einforcement L earning

    Dimakopoulou, M., Osband, I., and Van Roy, B. Scalable coordinated exploration in concurrent R einforcement L earning. Advances in Neural Information Processing Systems, 31, 2018

    work page 2018

Show all 71 references
  1. [9]

    Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures

    Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., Legg, S., and Kavukcuoglu, K. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. CoRR, abs/1802.01561, 2018. URL http:/...

    2018 arXiv

  2. [10]

    Diversity is all you need: Learning skills without a reward function

    Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019

    2019

  3. [11]

    A universal and generative physics engine for robotics and beyond, December 2024

    Genesis. A universal and generative physics engine for robotics and beyond, December 2024. URL https://github.com/Genesis-Embodied-AI/Genesis

    2024

  4. [12]

    J., and Wierstra, D

    Gregor, K., Rezende, D. J., and Wierstra, D. Variational intrinsic control. In International Conference on Learning Representations, 2017

    2017

  5. [13]

    and Brunskill, E

    Guo, Z. and Brunskill, E. Concurrent PAC RL . In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015

    2015

  6. [14]

    D., Azar, M

    Guo, Z. D., Azar, M. G., Saade, A., Thakoor, S., Piot, B., Pires, B. A., Valko, M., Mesnard, T., Lattimore, T., and Munos, R. Geometric entropic exploration. arXiv preprint arXiv:2101.02055, 2021

    2021 arXiv

  7. [15]

    Fast task inference with variational intrinsic successor features

    Hansen, S., Dabney, W., Barreto, A., Warde-Farley, D., Van de Wiele, T., and Mnih, V. Fast task inference with variational intrinsic successor features. In International Conference on Learning Representations, 2019

    2019

  8. [16]

    Provably efficient M aximum E ntropy E xploration

    Hazan, E., Kakade, S., Singh, K., and Van Soest, A. Provably efficient M aximum E ntropy E xploration. In International Conference on Machine Learning, pp.\ 2681--2691. PMLR, 2019

    2019

  9. [17]

    Wasserstein unsupervised R einforcement L earning

    He, S., Jiang, Y., Zhang, H., Shao, J., and Ji, X. Wasserstein unsupervised R einforcement L earning. In AAAI Conference on Artificial Intelligence, 2022

    2022

  10. [18]

    M., and Goodwin, M

    Holen, M., Andersen, P.-A., Knausg rd, K. M., and Goodwin, M. Loss and reward weighing for increased learning in distributed R einforcement L earning. arXiv preprint arXiv:2304.12778, 2023

    2023 arXiv

  11. [19]

    K., Lehnert, L., Rish, I., and Berseth, G

    Jain, A. K., Lehnert, L., Rish, I., and Berseth, G. Maximum state entropy exploration using predecessor and successor representations. In Advances in Neural Information Processing Systems, 2023

    2023

  12. [20]

    Population-guided parallel policy search for R einforcement L earning

    Jung, W., Park, G., and Sung, Y. Population-guided parallel policy search for R einforcement L earning. arXiv preprint arXiv:2001.02907, 2020

    2001 arXiv

  13. [21]

    Kakade, S. M. A natural policy gradient. In Advances in Neural Information Processing Systems, 2001

    2001

  14. [22]

    Accelerating R einforcement L earning with value-conditional state entropy exploration

    Kim, D., Shin, J., Abbeel, P., and Seo, Y. Accelerating R einforcement L earning with value-conditional state entropy exploration. In Advances in Neural Information Processing Systems, 2023

    2023

  15. [23]

    Efficient exploration via state marginal matching

    Lee, L., Eysenbach, B., Parisotto, E., Xing, E., Levine, S., and Salakhutdinov, R. Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274, 2019

    1906 arXiv

  16. [24]

    Offline R einforcement L earning: Tutorial, review, and perspectives on open problems

    Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline R einforcement L earning: Tutorial, review, and perspectives on open problems. CoRR, abs/2005.01643, 2020. URL https://arxiv.org/abs/2005.01643

    2005 arXiv

  17. [25]

    Celebrating diversity in shared multi-agent R einforcement L earning

    Li, C., Wu, C., Wang, T., Yang, J., Zhao, Q., and Zhang, C. Celebrating diversity in shared multi-agent R einforcement L earning. CoRR, abs/2106.02195, 2021. URL https://arxiv.org/abs/2106.02195

    2021 arXiv

  18. [26]

    and Abbeel, P

    Liu, H. and Abbeel, P. APS : Active pretraining with successor features. In International Conference on Machine Learning, 2021 a

    2021

  19. [27]

    and Abbeel, P

    Liu, H. and Abbeel, P. Behavior from the void: Unsupervised active pre-training. In Advances in Neural Information Processing Systems, 2021 b

    2021

  20. [28]

    Trajectory diversity for zero-shot coordination

    Lupu, A., Cui, B., Hu, H., and Foerster, J. Trajectory diversity for zero-shot coordination. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.\ 7204--7213. PMLR, ...

    2021

  21. [29]

    M., Papini, M., Faccio, F., and Restelli, M

    Metelli, A. M., Papini, M., Faccio, F., and Restelli, M. Policy optimization via importance sampling. In Advances in Neural Information Processing Systems, 2018

    2018

  22. [30]

    Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. A. Playing ATARI with deep R einforcement L earning. CoRR, abs/1312.5602, 2013. URL http://arxiv.org/abs/1312.5602

    2013 arXiv

  23. [31]

    Unsupervised R einforcement L earning via state entropy maximization

    Mutti, M. Unsupervised R einforcement L earning via state entropy maximization . PhD Thesis, Universit \`a di Bologna, 2023

    2023

  24. [32]

    and Restelli, M

    Mutti, M. and Restelli, M. An intrinsically-motivated approach for learning highly exploring and fast mixing policies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.\ 5232--5239, 2020

    2020

  25. [33]

    Task-agnostic exploration via policy gradient of a non-parametric state entropy estimate

    Mutti, M., Pratissoli, L., and Restelli, M. Task-agnostic exploration via policy gradient of a non-parametric state entropy estimate. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 9028--9036, 2021

    2021

  26. [34]

    The importance of non- M arkovianity in maximum state entropy exploration

    Mutti, M., De Santi, R., and Restelli, M. The importance of non- M arkovianity in maximum state entropy exploration. In International Conference on Machine Learning, 2022 a

    2022

  27. [35]

    Unsupervised R einforcement L earning in multiple environments

    Mutti, M., Mancassola, M., and Restelli, M. Unsupervised R einforcement L earning in multiple environments. In AAAI Conference on Artificial Intelligence, 2022 b

    2022

  28. [36]

    D., Bartolomeis, P

    Mutti, M., Santi, R. D., Bartolomeis, P. D., and Restelli, M. Challenging common assumptions in convex R einforcement L earning, 2023. URL https://arxiv.org/abs/2202.01511

    2023 arXiv

  29. [37]

    and Cook, M

    Nedergaard, A. and Cook, M. k- M eans M aximum E ntropy E xploration. arXiv preprint arXiv:2205.15623, 2022

    2022 arXiv

  30. [38]

    NVIDIA Isaac Sim. NVIDIA. 2021.https://developer.nvidia.com/isaac-sim

    2021

  31. [39]

    B., Chinnaobireddy, V., Wu, Y., and Salakhutdinov, R

    Parisotto, E., Ghosh, S., Yalamanchi, S. B., Chinnaobireddy, V., Wu, Y., and Salakhutdinov, R. Concurrent meta R einforcement L earning. arXiv preprint arXiv:1903.02710, 2019

    1903 arXiv

  32. [40]

    HIQL : Offline goal-conditioned RL with latent states as actions

    Park, S., Ghosh, D., Eysenbach, B., and Levine, S. HIQL : Offline goal-conditioned RL with latent states as actions. Advances in Neural Information Processing Systems, 36: 0 34866--34891, 2023

    2023

  33. [41]

    and Schaal, S

    Peters, J. and Schaal, S. R einforcement L earning of motor skills with policy gradients. Neural Networks, 2008

    2008

  34. [42]

    An analysis of ensemble sampling

    Qin, C., Wen, Z., Lu, X., and Van Roy, B. An analysis of ensemble sampling. Advances in Neural Information Processing Systems, 35: 0 21602--21614, 2022

    2022

  35. [43]

    Learning to walk in minutes using massively parallel deep reinforcement learning

    Rudin, N., Hoeller, D., Reist, P., and Hutter, M. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pp.\ 91--100. PMLR, 2022

    2022

  36. [44]

    Trust region policy optimization

    Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International conference on machine learning, 2015

    2015

  37. [45]

    State entropy maximization with random encoders for efficient exploration

    Seo, Y., Chen, L., Shin, J., Lee, H., Abbeel, P., and Lee, K. State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning, 2021

    2021

  38. [46]

    Dynamics-aware unsupervised discovery of skills

    Sharma, A., Gu, S., Levine, S., Kumar, V., and Hausman, K. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, 2020

    2020

  39. [47]

    Sucar, L. E. Parallel markov decision processes. Advances in Probabilistic Graphical Models, pp.\ 295--309, 2007

    2007

  40. [48]

    Sutton, R. S. and Barto, A. G. R einforcement L earning: An introduction . MIT press, 2018

    2018

  41. [49]

    S., McAllester, D., Singh, S., and Mansour, Y

    Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. Policy gradient methods for R einforcement L earning with function approximation. In Advances in Neural Information Processing Systems, 1999

    1999

  42. [50]

    and Lazaric, A

    Tarbouriech, J. and Lazaric, A. Active exploration in M arkov decision processes. In International Conference on Artificial Intelligence and Statistics, 2019

    2019

  43. [51]

    Active model estimation in M arkov decision processes

    Tarbouriech, J., Shekhar, S., Pirotta, M., Ghavamzadeh, M., and Lazaric, A. Active model estimation in M arkov decision processes. In Conference on Uncertainty in Artificial Intelligence, 2020

    2020

  44. [52]

    Fast rates for maximum entropy exploration

    Tiapkin, D., Belomestny, D., Calandriello, D., Moulines, E., Munos, R., Naumov, A., Perrault, P., Tang, Y., Valko, M., and Menard, P. Fast rates for maximum entropy exploration. In International Conference on Machine Learning, pp.\ 34161--34221. PMLR, 2023

    2023

  45. [53]

    Mujoco: A physics engine for model-based control

    Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.\ 5026--5033. IEEE, 2012. doi:10.1109/IROS.2012.6386109

    2012

  46. [54]

    U., Cola, G

    Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., Cola, G. D., Deleu, T., Goulão, M., Kallinteris, A., Krimmel, M., KG, A., Perez-Vicente, R., Pierré, A., Schulhoff, S., Tai, J. J., Tan, H., and Younis, O. G. Gymnasium: A standard interface for R einforcement L earning env...

    2024 arXiv

  47. [55]

    Influence-based multi-agent exploration

    Wang, T., Wang, J., Wu, Y., and Zhang, C. Influence-based multi-agent exploration. arXiv preprint arXiv:1910.05512, 2019

    1910 arXiv

  48. [56]

    Watkins, C. J. and Dayan, P. Q-learning. Machine learning, 8: 0 279--292, 1992

    1992

  49. [57]

    Williams, R. J. Simple statistical gradient-following algorithms for connectionist R einforcement L earning. Machine Learning, 8: 0 229--256, 1992

    1992

  50. [58]

    Population-based diverse exploration for sparse-reward multi-agent tasks

    Xu, P., Zhang, J., and Huang, K. Population-based diverse exploration for sparse-reward multi-agent tasks. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pp.\ 283--291, 2024

    2024

  51. [59]

    and Spaan, M

    Yang, Q. and Spaan, M. T. CEM : Constrained entropy maximization for task-agnostic safe exploration. In AAAI Conference on Artificial Intelligence, 2023

    2023

  52. [60]

    R einforcement L earning with prototypical representations

    Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. R einforcement L earning with prototypical representations. In International Conference on Machine Learning, 2021

    2021

  53. [61]

    Don't change the algorithm, change the data: Exploratory data for offline R einforcement L earning

    Yarats, D., Brandfonbrener, D., Liu, H., Laskin, M., Abbeel, P., Lazaric, A., and Pinto, L. Don't change the algorithm, change the data: Exploratory data for offline R einforcement L earning. arXiv preprint arXiv:2201.13425, 2022

    2022 arXiv

  54. [62]

    Discovering policies with DOM i NO : Diversity optimization maintaining near optimality

    Zahavy, T., Schroecker, Y., Behbahani, F., Baumli, K., Flennerhag, S., Hou, S., and Singh, S. Discovering policies with DOM i NO : Diversity optimization maintaining near optimality. arXiv preprint arXiv:2205.13521, 2022

    2022 arXiv

  55. [63]

    How to explore with belief: state entropy maximization in POMDP s

    Zamboni, R., Cirino, D., Restelli, M., and Mutti, M. How to explore with belief: state entropy maximization in POMDP s. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024 a

    2024

  56. [64]

    The limits of pure exploration in POMDP s: When the observation entropy is enough

    Zamboni, R., Cirino, D., Restelli, M., and Mutti, M. The limits of pure exploration in POMDP s: When the observation entropy is enough. RLJ , 2: 0 676--692, 2024 b . URL https://rlj.cs.umass.edu/2024/papers/Paper95.html

    2024

  57. [65]

    Towards principled multi-agent task agnostic exploration

    Zamboni, R., Mutti, M., and Restelli, M. Towards principled multi-agent task agnostic exploration. arXiv preprint arXiv:2502.08365, 2025

    2025

  58. [66]

    Exploration by maximizing R \'e nyi entropy for reward-free RL framework

    Zhang, C., Cai, Y., Huang, L., and Li, J. Exploration by maximizing R \'e nyi entropy for reward-free RL framework. In AAAI Conference on Artificial Intelligence, 2021 a

    2021

  59. [67]

    Self-motivated multi-agent exploration

    Zhang, S., Cao, J., Yuan, L., Yu, Y., and Zhan, D.-C. Self-motivated multi-agent exploration. arXiv preprint arXiv:2301.02083, 2023

    2023 arXiv

  60. [68]

    E., and Russell, S

    Zhang, T., Rashidinejad, P., Jiao, J., Tian, Y., Gonzalez, J. E., and Russell, S. MADE : Exploration via maximizing deviation from explored regions. Advances in Neural Information Processing Systems, 34: 0 9663--9680, 2021 b

    2021

  61. [69]

    Maximum entropy population based training for zero-shot human-ai coordination

    Zhao, R., Song, J., Hu, H., Gao, Y., Wu, Y., Sun, Z., and Wei, Y. Maximum entropy population based training for zero-shot human-ai coordination. CoRR, abs/2112.11701, 2021. URL https://arxiv.org/abs/2112.11701

    2021 arXiv

  62. [70]

    No prior mask: Eliminate redundant action for deep reinforcement learning

    Zhong, D., Yang, Y., and Zhao, Q. No prior mask: Eliminate redundant action for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 17078--17086, 2024

    2024

  63. [71]

    Explore to generalize in zero-shot RL

    Zisselman, E., Lavie, I., Soudry, D., and Tamar, A. Explore to generalize in zero-shot RL . In Advances in Neural Information Processing Systems, 2023

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.