Pith. sign in

REVIEW 4 major objections 5 minor 152 references

Decentralized Multi-Player Q-Learning in Episodic Markov Decision Processes with Information Asymmetry

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Decentralized players who cannot communicate during learning can still match the centralized Q-learning regret rate in episodic MDPs, provided they pre-agree on deterministic tie-breaking.

desk verdict Problem A is a clean reduction, but the paper's centerpiece explore-then-commit bound for full asymmetry rests on a false uniform-coverage premise, and the proof is missing; the paper needs major revision. read the letter →

arxiv 2608.12753 v1 pith:GOJOUDPS submitted 2026-08-13 cs.LG

Larissa Xu , King Bi , William Chang This is my paper · ORCID
classification cs.LG
keywords multi-playerreinforcementlearningepisodicMDPQ-learningregretboundsinformationasymmetrydecentralizedexplore-then-commitimplicitcoordination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether cooperative multi-player reinforcement learning in episodic tabular MDPs (finite-horizon Markov decision processes with finitely many states and actions) loses efficiency when players cannot observe each other's actions or rewards while learning. It answers no for two of the three information-asymmetry models: with either common rewards or observed actions, a pre-agreed deterministic protocol achieves $\tilde{O}(\sqrt{H^4 S A_{\text{joint}} T})$ regret, which matches the rate a single centralized agent would obtain on the joint-action MDP, up to logarithmic factors. For the fully asymmetric case, where players see neither the others' actions nor their rewards, the paper gives explore-then-commit algorithms with regret $\tilde{O}(H (S A_{\text{joint}})^{1/3} T^{2/3})$. The bounds are most meaningful when the joint action space $A_{\text{joint}} = \prod_{i=1}^{M} |\mathcal{A}_i|$ is small, because it grows exponentially in the number of players; the contribution is that information asymmetry adds no extra factor beyond the centralized tabular rate.

What carries the argument

Lexicographic ordering of joint actions (Definition 2): compare two $M$-tuples as multi-digit numbers and pick the smaller one. This is the load-bearing tie-break because it converts a set of tied Q-values into a unique joint action, so all players, running the same deterministic algorithm on the same shared statistics, arrive at the same choice without communication. In Problems A and B it directly selects the greedy joint action; in Problem C it defines both the least-visited exploration rule and the commit-phase greedy selection. The other workhorse is the weighted-learning-rate recursion (Lemma 11) with the identity $\sum_{t \ge i} \alpha_t^i = 1 + 1/H$ (Lemma 10(c)), which lets the regret analysis charge a future-step value error back to all earlier episodes that visited the same state-action pair.

What would settle it

Run mEXC on a two-state MDP in which state 2 is reached with probability $10^{-3}$; after the $K'$ exploration episodes, count visits to each state-2 action pair. If any pair's visit count falls well below $K'/(S A_{\text{joint}})$, the proof's coverage premise is violated and the stated regret bound need not hold; the same test can check whether near-tied joint actions are chosen consistently during the commit phase.

Watch

Extended reading notes

Core claim

The central claim is that pre-agreed deterministic rules can replace communication: the players need not observe each other's choices or rewards, because a shared lexicographic ordering plus identical update rules makes their private Q-tables select the same joint action at every step. In Problem A, identical rewards and transitions make all Q-tables exactly equal, so each player's play is identical to what a single centralized Q-learning agent would do on the joint MDP. In Problem B, observable actions allow a player's unilateral deviation to be read by everyone else as a one-bit signal, and upper/lower confidence intervals keep the players' estimates close enough to eliminate the same suboptimal actions. In Problem C, both signals are absent, so the paper uses a two-phase explore-then-commit schedule where shared visit counts coordinate exploration and lexicographic tie-breaking on learned Q-values coordinates the commit phase, giving the slower $T^{2/3}$ rate. The paper's overall thesis is that asymmetry is not free in absolute terms, because the joint action space is exponential in the number of players, but it adds no multiplicative regret penalty relative to a centralized learner on the same joint-action benchmark.

Load-bearing premise

For the fully asymmetric Problem C, the proof assumes that the exploration phase visits every state-action pair about $K'/(S A_{\text{joint}})$ times in expectation, but in a stochastic MDP the transition kernel governs how often states are reached, so rarely reachable states get far fewer visits and the uniform coverage floor does not follow.

Editorial extensions

If this is right

  • For Problems A and B, decentralized learning with zero communication matches the regret of a centralized learner on the joint-action MDP, up to log factors.
  • Either shared rewards or observable actions supplies enough of a coordination channel to keep the $\sqrt{T}$ regret rate; only when both are missing does the rate drop to $T^{2/3}$.
  • The bounds show that the multi-player curse is the exponential joint action space $A_{\text{joint}}$, not the information asymmetry: if $A_{\text{joint}}$ is held fixed, splitting one player's actions among more players does not change the regret.
  • For the fully asymmetric case the paper leaves open whether $\sqrt{T}$ regret is achievable; its $T^{2/3}$ rate is the standard penalty of explore-then-commit when the suboptimality gap is unknown.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The uniform-coverage premise of the Problem C bound is questionable: in a stochastic MDP the transition kernel controls how often states are visited, so a least-visited joint-action exploration schedule does not guarantee roughly $K'/(S A_{\text{joint}})$ visits to every state-action pair. A direct check is to count visits in a two-state chain with a rarely reached state and compare against the cl
  • The same deterministic tie-breaking idea should carry over to settings with limited or one-way communication, where a short pre-agreement plays the role of the missing coordination channel; the paper implicitly trades a priori coordination against online information.
  • The regret depends on $A_{\text{joint}}$ only, so extending the framework to factored MDPs or linear function approximation is a natural next test; the tie-breaking mechanism itself does not rely on the joint-action structure except through the size of the action space.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies decentralized multi-player reinforcement learning in episodic tabular MDPs under three information asymmetry models. For Problem A (unobserved actions, common rewards) it proposes mQ-learning and claims O~(sqrt(H^4 S A_joint T)) regret by reducing the protocol to centralized joint-action Q-learning with deterministic lexicographic tie-breaking. For Problem B (observed actions, independent rewards) it proposes mQ-learning-intervals, which maintains upper and lower confidence tables for action elimination, and claims the same regret rate. For Problem C (unobserved actions, independent rewards) it proposes mEXC and mEXC-Bellman, two-phase explore-then-commit algorithms, and claims O~(H (S A_joint)^{1/3} T^{2/3}) regret. The appendix contains lemmas on learning-rate weights, an optimism lemma, an interval-width lemma, and regret recursions that closely follow Jin et al. [1].

Significance. If the theorems were valid, the paper would establish that information asymmetry can be overcome at no cost beyond the joint-action space for Problems A and B, and would give the first sublinear regret bound for the fully asymmetric Problem C. The reduction for Problem A is a clean and sound observation, and the learning-rate lemmas are stated carefully. However, the central new contributions rest on two unsupported or false premises: the validity of the lower confidence table in Problem B, and the uniform coverage assumption in Problem C's exploration phase. Because these premises are load-bearing for the main theorems, the manuscript in its current form does not support its claims.

major comments (4)
  1. [Section IV.A, Theorem 7 proof sketch] The claim that after K' episodes of least-visited exploration every (x,a) has been visited at least floor(K'/(S A_joint)) times in expectation is false. Algorithm 3 line 5 selects the least-visited action only within the current state; the sequence of states is governed by the unknown transition kernel. A state that is reachable with probability p << 1/S will be reached about p K' times, so a state-action pair involving it receives O(p K'/|A_i|) visits rather than Omega(K'/(S A_joint)). Consequently the stated post-exploration error bound O(sqrt(H^4 S A_joint iota/K')) and the balancing K' = (S A_joint)^{1/3} K^{2/3} do not follow. The same false premise invalidates the mEXC-Bellman variant, whose proof invokes a model-based concentration bound that also requires uniform coverage; the argument fails already for M=1.
  2. [Section III.B, Lemma 12 and Algorithm 2] No proof is given that the lower confidence table satisfies Q^low <= Q*. Lemma 12 proves optimism only for updates with the +b_t bonus. The lower-table update in Algorithm 2 line 12 uses -b_t but plugs in V_{h+1}, which is computed from the optimistic upper Q-values, so the future-value term can exceed V*_{h+1}; the backward induction that would establish Q^low <= Q* is therefore unavailable. The statement in Section III.B that 'Optimism ... guarantees each interval covers Q*' is unsupported. Since the action-elimination rule at lines 6-8 compares upper and lower tables, a false lower bound can remove the optimal action, which breaks the proof of Theorem 5.
  3. [Section IV.A, Remark 9] The commit phase requires that, on the high-probability event, all players' Q-tables agree on the same lexicographic-smallest greedy joint action. This does not follow from uniform closeness to Q*. If two joint actions have values within the estimation error, independent reward samples can produce opposite orderings in different players' tables, so one player may choose a and another may choose b; because actions are unobserved in Problem C, the executed joint action is then not the intended one, and the per-episode suboptimality bound in the proof of Theorem 7 no longer applies.
  4. [Appendix C, mEXC-Bellman proof] The proof of Theorem 7 for mEXC-Bellman says the argument is 'identical, using the standard model-based plug-in error bound ||hat P - P||_1 <= O(sqrt(S iota/N))'. This is not a proof: it does not specify how the empirical transition estimate is formed from the least-visited exploration data, nor how a per-state-action-pair concentration bound can be obtained without uniform coverage. A complete argument is needed, and in light of the first major comment no such argument can go through with the current exploration rule.
minor comments (5)
  1. [Algorithm 2, line 13] The definition of V_h(x_h) uses 'Q^k_h(i,x_h,a')' without specifying whether Q^k_h is the upper or lower table; the two choices yield different updates, and the proof does not disambiguate them.
  2. [Lemma 12] Lemma 12 is stated with iota = log(SAT/p), but Theorem 5 requires a union bound over M players; the M factor should appear inside the logarithm in the lemma statement as well, not only in the theorem.
  3. [Section II.C] The sentence about an Omega(log T) lower bound 'for the multi-player problem as well' is not used anywhere in the regret analysis and distracts from the episodic-MDP lower bound quoted from [1]; it could be removed or clarified.
  4. [Abstract and Section II.A] The notation A_joint = product_i |A_i| is used together with the shorthand A = A_joint; please make the notation consistent throughout.
  5. [Section VII.C] The numerical example states that per-episode suboptimality falls below 0.1 once K is roughly 10^5, but the constants in the theorems are not specified; this should be labeled as an illustrative order-of-magnitude estimate rather than a derived guarantee.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the main bounds are honest reductions to the external single-agent Q-learning result, with no fitted parameters or self-referential definitions.

full rationale

The paper's derivation chain does not define its conclusions into existence or fit parameters and then relabel them as predictions. For Problems A and B, Algorithm 1 is explicitly identified as operationally equivalent to centralized joint-action Q-learning, and the claimed regret bound is inherited from the external result of Jin et al. [1], whose authors do not overlap with the present paper. The deterministic lexicographic tie-breaking, citing the authors' earlier bandit work [7], is a coordination mechanism rather than a source of the regret bound; the proof itself is a direct reduction to [1]. Problem B similarly leverages an interval-width argument, and while the interval idea is attributed to the authors' own prior work [8], the theorem's proof is carried out in the paper via Lemma 13 and concentration inequalities, not by appeal to the cited result for its validity. For Problem C, Theorem 7 is the paper's main load-bearing contribution; it relies on an explore-then-commit schedule and a claimed uniform-coverage property of least-visited-action exploration. That uniform-coverage assertion is unsupported and in fact false in general, because state reachability is governed by the unknown transition kernel. This is a serious correctness gap, but it is not circularity: the theorem does not have its target regret bound encoded in its assumptions, no fitted parameter is renamed as a prediction, and no self-citation chain is used to force the result. No step in the paper reduces, by construction, to the quantity it is supposed to derive. The self-citations are present but not load-bearing in the sense required for a circularity finding, so the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The regret bounds borrow the Q-learning analysis of [1], the tie-breaking idea of [7], and the interval-elimination idea of [8]. The genuinely new premises are the two coordination assumptions in Problem C and the lower-confidence premise in Problem B, none of which is proved.

assumptions (6)
  • standard math Azuma-Hoeffding inequality and union bounds for concentration of martingale noise and Q-value estimates.
    Used in Lemmas 10, 12, and the proofs of Theorems 3 and 5 to bound deviation terms; standard but unproved in this paper.
  • domain assumption Episodic tabular MDP with layered states, unknown transition kernels, and 1-subgaussian rewards in [0,1].
    Section II defines the model and states that states are layered without loss of generality; all results depend on this tabular episodic structure.
  • domain assumption Players agree on a protocol a priori and cannot communicate during learning.
    Section II.D and all algorithms rely on this constraint; it is the defining setting of the paper.
  • ad hoc to paper After K' episodes of least-visited exploration, every state-action pair has at least order K'/(S A_joint) visits in expectation.
    Section IV.A and Theorem 7 proof sketch; this premise is load-bearing for the explore-then-commit bound and is false for stochastic transition kernels because state reach counts are not controlled by the exploration rule.
  • ad hoc to paper Q-tables that are uniformly close to Q* select the same lexicographic-smallest greedy joint action.
    Remark 9 and Theorem 7 proof; uniform closeness does not imply identical argmax when actions are near-tied, so decentralized greedy commitment can disagree.
  • ad hoc to paper The lower confidence table Q^low satisfies Q^low <= Q* for all players.
    Section III.B elimination rule requires lower bounds; the paper asserts interval coverage but only proves optimism for the upper table.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decentralized Multi-Player Q-Learning in Episodic Markov Decision Processes with Information Asymmetry." pith.science (2026). https://pith.science/paper/GOJOUDPS

@misc{pith2026260812753,
  author       = {Pith},
  title        = {Pith review of: Decentralized Multi-Player Q-Learning in Episodic Markov Decision Processes with Information Asymmetry},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GOJOUDPS}},
  note         = {Machine review of arXiv:2608.12753}
}
abstract

We study decentralized multi-player reinforcement learning in episodic tabular Markov decision processes (MDPs) under three forms of information asymmetry: (A) unobserved actions with common rewards, (B) observed actions with independent rewards, and (C) unobserved actions with independent rewards. Players cannot communicate during learning but may agree on a protocol a priori. For Problems A and B we propose \texttt{mQ-learning} and \texttt{mQ-learning-intervals}, achieving $\tilde{O}(\sqrt{H^4 S A_{\text{joint}}\, T})$ regret, where $H$ is the horizon, $S$ the state count, $T = KH$ the total steps, and $A_{\text{joint}} = \prod_{i=1}^M |\mathcal{A}_i|$ the joint action space across $M$ players. For Problem C we give \texttt{mEXC} and \texttt{mEXC-Bellman}, two-phase explore-then-commit algorithms with regret $\tilde{O}(H (S A_{\text{joint}})^{1/3} T^{2/3})$. Against the centralized joint-action benchmark, decentralized learning under information asymmetry matches the single-agent Q-learning rate of \cite{jin2018q} up to logarithmic factors. Because $A_{\text{joint}}$ grows exponentially in $M$, the bounds are most meaningful for small $M$ or small per-player action sets.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

152 extracted references · 50 canonical work pages

  1. [7]

    Advances in applied mathematics , volume=

    Asymptotically efficient adaptive allocation rules , author=. Advances in applied mathematics , volume=. 1985 , publisher=

    1985

  2. [1]

    2020 , publisher=

    Bandit algorithms , author=. 2020 , publisher=

    2020

  3. [2]

    , author=

    Coordinated Versus Decentralized Exploration In Multi-Agent Multi-Armed Bandits. , author=. IJCAI , pages=

  4. [3]

    2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    Multi-armed bandits in multi-agent networks , author=. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2017 , organization=

    2017

  5. [4]

    Advances in Neural Information Processing Systems , volume=

    Multi-bandit best arm identification , author=. Advances in Neural Information Processing Systems , volume=

  6. [5]

    IEEE Transactions on Automatic Control , volume=

    Asymptotically efficient allocation rules for the multiarmed bandit problem with multiple plays-part i: Iid rewards , author=. IEEE Transactions on Automatic Control , volume=. 2003 , publisher=

    2003

  7. [6]

    Annals of Applied Probability , pages=

    Chernoff-type bound for finite Markov chains , author=. Annals of Applied Probability , pages=. 1998 , publisher=

    1998

  8. [8]

    IEEE Transactions on Information Theory , volume=

    Decentralized learning for multiplayer multiarmed bandits , author=. IEEE Transactions on Information Theory , volume=. 2014 , publisher=

    2014

Show all 152 references
  1. [9]

    1988 , institution=

    The dynamic structure of everyday life , author=. 1988 , institution=

    1988

  2. [10]

    2018 , publisher=

    Reinforcement learning: An introduction , author=. 2018 , publisher=

    2018

  3. [11]

    International Conference on Machine Learning , pages=

    Multi-player bandits--a musical chairs approach , author=. International Conference on Machine Learning , pages=. 2016 , organization=

    2016

  4. [12]

    International conference on machine learning , pages=

    Distributed clustering of linear bandits in peer to peer networks , author=. International conference on machine learning , pages=. 2016 , organization=

    2016

  5. [13]

    IEEE Journal of Selected Topics in Signal Processing , volume=

    Deterministic sequencing of exploration and exploitation for multi-armed bandit problems , author=. IEEE Journal of Selected Topics in Signal Processing , volume=. 2013 , publisher=

    2013

  6. [14]

    Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages=

    Concurrent bandits and cognitive radio networks , author=. Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages=. 2014 , organization=

    2014

  7. [15]

    IEEE Transactions on Wireless Communications , volume=

    Channel selection for network-assisted D2D communication via no-regret bandit learning with calibrated forecasting , author=. IEEE Transactions on Wireless Communications , volume=. 2014 , publisher=

    2014

  8. [16]

    Algorithmic Learning Theory , pages=

    Multi-player bandits revisited , author=. Algorithmic Learning Theory , pages=. 2018 , organization=

    2018

  9. [17]

    2011 , publisher=

    Multi-armed bandit allocation indices , author=. 2011 , publisher=

    2011

  10. [18]

    International Conference on Machine Learning , pages=

    Decentralized exploration in multi-armed bandits , author=. International Conference on Machine Learning , pages=. 2019 , organization=

    2019

  11. [19]

    International Conference on Artificial Intelligence and Statistics , pages=

    Optimal algorithms for multiplayer multi-armed bandits , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2020 , organization=

    2020

  12. [20]

    International Conference on Artificial Intelligence and Statistics , pages=

    Decentralized Multi-player Multi-armed Bandits with No Collision Information , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2020 , organization=

    2020

  13. [21]

    IEEE Transactions on Signal Processing , volume=

    Distributed learning in multi-armed bandit with multiple players , author=. IEEE Transactions on Signal Processing , volume=. 2010 , publisher=

    2010

  14. [22]

    IEEE Journal on Selected Areas in Communications , volume=

    Distributed algorithms for learning and cognitive medium access with logarithmic regret , author=. IEEE Journal on Selected Areas in Communications , volume=. 2011 , publisher=

    2011

  15. [23]

    IEEE Transactions on Control of Network Systems , volume=

    On regret-optimal learning in decentralized multiplayer multiarmed bandits , author=. IEEE Transactions on Control of Network Systems , volume=. 2018 , publisher=

    2018

  16. [24]

    IEEE/ACM Transactions on Networking , volume=

    Combinatorial network optimization with unknown variables: Multi-armed bandits with linear rewards and individual observations , author=. IEEE/ACM Transactions on Networking , volume=. 2012 , publisher=

    2012

  17. [25]

    Machine learning , volume=

    Finite-time analysis of the multiarmed bandit problem , author=. Machine learning , volume=. 2002 , publisher=

    2002

  18. [26]

    American Economic Review , volume=

    Uncoupled dynamics do not lead to Nash equilibrium , author=. American Economic Review , volume=

  19. [27]

    Journal of Economic Theory , volume=

    A general class of adaptive strategies , author=. Journal of Economic Theory , volume=. 2001 , publisher=

    2001

  20. [28]

    Econometrica , volume=

    A simple adaptive procedure leading to correlated equilibrium , author=. Econometrica , volume=. 2000 , publisher=

    2000

  21. [29]

    Economics essays , pages=

    A reinforcement procedure leading to correlated equilibrium , author=. Economics essays , pages=. 2001 , publisher=

    2001

  22. [30]

    2013 , publisher=

    Simple adaptive strategies: from regret-matching to uncoupled dynamics , author=. 2013 , publisher=

    2013

  23. [31]

    1998 , publisher=

    The theory of learning in games , author=. 1998 , publisher=

    1998

  24. [32]

    2006 , publisher=

    Prediction, learning, and games , author=. 2006 , publisher=

    2006

  25. [33]

    2004 , publisher=

    Strategic learning and its limits , author=. 2004 , publisher=

    2004

  26. [34]

    Management science , volume=

    Elements for a theory of teams , author=. Management science , volume=. 1955 , publisher=

    1955

  27. [35]

    International Conference on Machine Learning , pages=

    Gossip-based distributed stochastic bandit algorithms , author=. International Conference on Machine Learning , pages=. 2013 , organization=

    2013

  28. [36]

    IEEE Transactions on Signal Processing , volume=

    Distributed multi-agent online learning based on global feedback , author=. IEEE Transactions on Signal Processing , volume=. 2015 , publisher=

    2015

  29. [37]

    arXiv preprint arXiv:1810.04468 , year=

    Decentralized Cooperative Stochastic Bandits , author=. arXiv preprint arXiv:1810.04468 , year=

    arXiv

  30. [38]

    2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS) , pages=

    Collaborative learning with limited interaction: Tight bounds for distributed exploration in multi-armed bandits , author=. 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS) , pages=. 2019 , organization=

    2019

  31. [39]

    2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS) , pages=

    Collaborative top distribution identifications with limited interaction , author=. 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS) , pages=. 2020 , organization=

    2020

  32. [40]

    IEEE Journal on Selected Areas in Information Theory , volume=

    One for All and All for One: Distributed Learning of Fair Allocations With Multi-Player Bandits , author=. IEEE Journal on Selected Areas in Information Theory , volume=. 2021 , publisher=

    2021

  33. [41]

    arXiv preprint arXiv:2109.03818 , year=

    Online learning for cooperative multi-player multi-armed bandits , author=. arXiv preprint arXiv:2109.03818 , year=

    arXiv

  34. [42]

    Advances in neural information processing systems , volume=

    Is Q-learning provably efficient? , author=. Advances in neural information processing systems , volume=

  35. [43]

    Conference on Learning Theory , pages=

    Beyond no regret: Instance-dependent pac reinforcement learning , author=. Conference on Learning Theory , pages=. 2022 , organization=

    2022

  36. [44]

    arXiv preprint arXiv:2311.06210 , year=

    Optimal Cooperative Multiplayer Learning Bandits with Noisy Rewards and No Communication , author=. arXiv preprint arXiv:2311.06210 , year=

    arXiv

  37. [45]

    Advances in neural information processing systems , volume=

    Near-optimal regret bounds for reinforcement learning , author=. Advances in neural information processing systems , volume=

  38. [46]

    , author=

    A Reinforcement Learning Method for Maximizing Undiscounted Rewards. , author=. ICML , volume=

  39. [47]

    International conference on machine learning , pages=

    Reinforcement learning for non-stationary markov decision processes: The blessing of (more) optimism , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  40. [48]

    International conference on machine learning , pages=

    Model-free reinforcement learning in infinite-horizon average-reward markov decision processes , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  41. [49]

    Algorithmic Learning Theory , pages=

    Variance-aware regret bounds for undiscounted reinforcement learning in mdps , author=. Algorithmic Learning Theory , pages=. 2018 , organization=

    2018

  42. [50]

    IEEE Transactions on Automatic Control , volume=

    Optimal control of Markov decision processes with linear temporal logic constraints , author=. IEEE Transactions on Automatic Control , volume=. 2014 , publisher=

    2014

  43. [51]

    Advances in Neural Information Processing Systems , volume=

    Transition constrained Bayesian optimization via Markov decision processes , author=. Advances in Neural Information Processing Systems , volume=

  44. [52]

    2017 IEEE International Conference on Robotics and Automation (ICRA) , pages=

    Sequential Bayesian optimization as a POMDP for environment monitoring with UAVs , author=. 2017 IEEE International Conference on Robotics and Automation (ICRA) , pages=. 2017 , organization=

    2017

  45. [53]

    International Conference on Machine Learning , pages=

    Learning fair policies in multi-objective (deep) reinforcement learning with average and discounted rewards , author=. International Conference on Machine Learning , pages=. 2020 , organization=

    2020

  46. [54]

    arXiv preprint arXiv:2502.16387 , year=

    Simultaneous Swap Regret Minimization via KL-Calibration , author=. arXiv preprint arXiv:2502.16387 , year=

  47. [55]

    arXiv preprint arXiv:2502.17625 , year=

    Instance-Dependent Regret Bounds for Learning Two-Player Zero-Sum Games with Bandit Feedback , author=. arXiv preprint arXiv:2502.17625 , year=

    arXiv

  48. [56]

    arXiv preprint arXiv:2503.02825 , year=

    On Separation Between Best-Iterate, Random-Iterate, and Last-Iterate Convergence of Learning in Games , author=. arXiv preprint arXiv:2503.02825 , year=

    arXiv

  49. [57]

    arXiv preprint arXiv:2502.12529 , year=

    Alternating Regret for Online Convex Optimization , author=. arXiv preprint arXiv:2502.12529 , year=

    arXiv

  50. [58]

    Advances in Neural Information Processing Systems , volume=

    Alternation makes the adversary weaker in two-player games , author=. Advances in Neural Information Processing Systems , volume=

  51. [59]

    arXiv preprint arXiv:2502.12528 , year=

    Contextual Linear Bandits with Delay as Payoff , author=. arXiv preprint arXiv:2502.12528 , year=

    arXiv

  52. [60]

    arXiv preprint arXiv:2412.07120 , year=

    Corrupted Learning Dynamics in Games , author=. arXiv preprint arXiv:2412.07120 , year=

    arXiv

  53. [61]

    The Thirty Sixth Annual Conference on Learning Theory , pages=

    Contexts can be cheap: Solving stochastic contextual bandits with linear bandit algorithms , author=. The Thirty Sixth Annual Conference on Learning Theory , pages=. 2023 , organization=

    2023

  54. [62]

    arXiv preprint arXiv:2405.07637 , year=

    Near-optimal regret in linear mdps with aggregate bandit feedback , author=. arXiv preprint arXiv:2405.07637 , year=

    arXiv

  55. [63]

    Algorithmic Learning Theory , pages=

    Adaptive reward-free exploration , author=. Algorithmic Learning Theory , pages=. 2021 , organization=

    2021

  56. [64]

    arXiv preprint arXiv:1610.03295 , year=

    Safe, multi-agent, reinforcement learning for autonomous driving , author=. arXiv preprint arXiv:1610.03295 , year=

    arXiv

  57. [65]

    , author=

    Libratus: The Superhuman AI for No-Limit Poker. , author=. IJCAI , pages=

  58. [66]

    nature , volume=

    Mastering the game of Go with deep neural networks and tree search , author=. nature , volume=. 2016 , publisher=

    2016

  59. [67]

    nature , volume=

    Mastering the game of go without human knowledge , author=. nature , volume=. 2017 , publisher=

    2017

  60. [68]

    The International Journal of Robotics Research , volume=

    Reinforcement learning in robotics: A survey , author=. The International Journal of Robotics Research , volume=. 2013 , publisher=

    2013

  61. [69]

    arXiv preprint arXiv:1509.02971 , year=

    Continuous control with deep reinforcement learning , author=. arXiv preprint arXiv:1509.02971 , year=

    arXiv

  62. [70]

    Science , volume=

    Superhuman AI for multiplayer poker , author=. Science , volume=. 2019 , publisher=

    2019

  63. [71]

    nature , volume=

    Human-level control through deep reinforcement learning , author=. nature , volume=. 2015 , publisher=

    2015

  64. [72]

    IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) , volume=

    A comprehensive survey of multiagent reinforcement learning , author=. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) , volume=. 2008 , publisher=

    2008

  65. [73]

    Transportation Research Part C: Emerging Technologies , volume=

    A cooperative multi-agent transportation management and route guidance system , author=. Transportation Research Part C: Emerging Technologies , volume=. 2002 , publisher=

    2002

  66. [74]

    Computer networks , volume=

    Towards smart factory for industry 4.0: a self-organized multi-agent system with big data based feedback and coordination , author=. Computer networks , volume=. 2016 , publisher=

    2016

  67. [75]

    , author=

    Stock Trading System Using Reinforcement Learning with Cooperative Agents. , author=. MACHINE LEARNING-INTERNATIONAL WORKSHOP THEN CONFERENCE- , pages=

  68. [76]

    IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans , volume=

    A multiagent approach to q-learning for daily stock trading , author=. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans , volume=. 2007 , publisher=

    2007

  69. [77]

    IEEE Transactions on robotics and Automation , volume=

    Coverage control for mobile sensing networks , author=. IEEE Transactions on robotics and Automation , volume=. 2004 , publisher=

    2004

  70. [78]

    Automatica , volume=

    Distributed learning and cooperative control for multi-agent systems , author=. Automatica , volume=. 2009 , publisher=

    2009

  71. [79]

    Cognitive Systems Research , volume=

    The theory of social functions: challenges for computational social science and multi-agent learning , author=. Cognitive Systems Research , volume=. 2001 , publisher=

    2001

  72. [80]

    arXiv preprint arXiv:1702.03037 , year=

    Multi-agent reinforcement learning in sequential social dilemmas , author=. arXiv preprint arXiv:1702.03037 , year=

    arXiv

  73. [81]

    TARK , volume=

    Planning, learning and coordination in multiagent decision processes , author=. TARK , volume=. 1996 , organization=

    1996

  74. [82]

    IEEE Transactions on Automatic Control , volume=

    Decomposition of dynamic team decision problems , author=. IEEE Transactions on Automatic Control , volume=. 1978 , publisher=

    1978

  75. [83]

    Proceedings of the IEEE , volume=

    Team decision theory and information structures , author=. Proceedings of the IEEE , volume=. 1980 , publisher=

    1980

  76. [84]

    Advances in neural information processing systems , volume=

    Reinforcement learning to play an optimal Nash equilibrium in team Markov games , author=. Advances in neural information processing systems , volume=

  77. [85]

    2008 , publisher=

    Sequential decomposition of sequential dynamic teams: applications to real-time communication and networked control systems , author=. 2008 , publisher=

    2008

  78. [86]

    2013 , publisher=

    Discrete--time stochastic control and dynamic potential games: the Euler--Equation approach , author=. 2013 , publisher=

    2013

  79. [87]

    arXiv preprint arXiv:1802.00899 , year=

    Learning parametric closed-loop policies for markov potential games , author=. arXiv preprint arXiv:1802.00899 , year=

    arXiv

  80. [88]

    IEEE Transactions on Signal Processing , volume=

    Dynamic potential games with constraints: Fundamentals and applications in communications , author=. IEEE Transactions on Signal Processing , volume=. 2016 , publisher=

    2016

  81. [89]

    International conference on machine learning , pages=

    Fully decentralized multi-agent reinforcement learning with networked agents , author=. International conference on machine learning , pages=. 2018 , organization=

    2018

  82. [90]

    IEEE Transactions on Signal Processing , volume=

    Q-Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus Innovations , author=. IEEE Transactions on Signal Processing , volume=. 2013 , publisher=

    2013

  83. [91]

    International Conference on Machine Learning , pages=

    Finite-time analysis of distributed TD (0) with linear function approximation on multi-agent reinforcement learning , author=. International Conference on Machine Learning , pages=. 2019 , organization=

    2019

  84. [92]

    Advances in neural information processing systems , volume=

    Multi-agent reinforcement learning via double averaging primal-dual optimization , author=. Advances in neural information processing systems , volume=

  85. [93]

    Machine learning proceedings 1994 , pages=

    Markov games as a framework for multi-agent reinforcement learning , author=. Machine learning proceedings 1994 , pages=. 1994 , publisher=

    1994

  86. [94]

    IEEE Transactions on Automatic control , volume=

    Optimal stochastic linear systems with exponential performance criteria and their relation to deterministic differential games , author=. IEEE Transactions on Automatic control , volume=. 2003 , publisher=

    2003

  87. [95]

    2008 , publisher=

    H-infinity optimal control and related minimax design problems: a dynamic game approach , author=. 2008 , publisher=

    2008

  88. [96]

    Learning for Dynamics and Control , pages=

    Policy Optimization for H_2 Linear Control with H_ Robustness Guarantee: Implicit Regularization and Global Convergence , author=. Learning for Dynamics and Control , pages=. 2020 , organization=

    2020

  89. [97]

    Journal of machine learning research , volume=

    Nash Q-learning for general-sum stochastic games , author=. Journal of machine learning research , volume=

  90. [98]

    ICML , volume=

    Friend-or-foe Q-learning in general-sum games , author=. ICML , volume=

  91. [99]

    Advances in Neural Information Processing Systems , volume=

    Learning in zero-sum team markov games using factored value functions , author=. Advances in Neural Information Processing Systems , volume=

  92. [100]

    IEEE Transactions on Automatic Control , volume=

    Finite-sample analysis for decentralized batch multiagent reinforcement learning with networked agents , author=. IEEE Transactions on Automatic Control , volume=. 2021 , publisher=

    2021

  93. [101]

    23 (SIAM, Philadelphia) , author=

    Dynamic Noncooperative Game Theory, vol. 23 (SIAM, Philadelphia) , author=

  94. [102]

    2022 56th Asilomar Conference on Signals, Systems, and Computers , pages=

    Constrained reinforcement learning via dissipative saddle flow dynamics , author=. 2022 56th Asilomar Conference on Signals, Systems, and Computers , pages=. 2022 , organization=

    2022

  95. [103]

    International Conference on Machine Learning , pages=

    Reload: Reinforcement learning with optimistic ascent-descent for last-iterate convergence in constrained mdps , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  96. [104]

    Advances in Neural Information Processing Systems , volume=

    Natural policy gradient primal-dual method for constrained markov decision processes , author=. Advances in Neural Information Processing Systems , volume=

  97. [105]

    arXiv preprint arXiv:2111.00552 , year=

    Policy optimization for constrained mdps with provable fast global convergence , author=. arXiv preprint arXiv:2111.00552 , year=

    arXiv

  98. [106]

    Uncertainty in Artificial Intelligence , pages=

    Towards painless policy optimization for constrained mdps , author=. Uncertainty in Artificial Intelligence , pages=. 2022 , organization=

    2022

  99. [107]

    arXiv preprint arXiv:2101.10895 , year=

    A primal-dual approach to constrained markov decision processes , author=. arXiv preprint arXiv:2101.10895 , year=

    arXiv

  100. [108]

    Operations Research Letters , volume=

    Faster algorithm and sharper analysis for constrained markov decision process , author=. Operations Research Letters , volume=. 2024 , publisher=

    2024

  101. [109]

    Advances in Neural Information Processing Systems , volume=

    Near-optimal sample complexity bounds for constrained MDPs , author=. Advances in Neural Information Processing Systems , volume=

  102. [110]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Achieving zero constraint violation for constrained reinforcement learning via primal-dual approach , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  103. [111]

    Conference on learning theory , pages=

    Last-iterate convergence of decentralized optimistic gradient descent/ascent in infinite-horizon competitive Markov games , author=. Conference on learning theory , pages=. 2021 , organization=

    2021

  104. [112]

    arXiv preprint arXiv:2303.03095 , year=

    Can We Find Nash Equilibria at a Linear Rate in Markov Games? , author=. arXiv preprint arXiv:2303.03095 , year=

    arXiv

  105. [113]

    2022 IEEE 61st Conference on Decision and Control (CDC) , pages=

    Finite-time complexity of online primal-dual natural actor-critic algorithm for constrained Markov decision processes , author=. 2022 IEEE 61st Conference on Decision and Control (CDC) , pages=. 2022 , organization=

    2022

  106. [114]

    International Conference on Artificial Intelligence and Statistics , pages=

    A dual approach to constrained markov decision processes with entropy regularization , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2022 , organization=

    2022

  107. [115]

    International Conference on Artificial Intelligence and Statistics , pages=

    Algorithm for constrained markov decision process with linear convergence , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2023 , organization=

    2023

  108. [116]

    arXiv preprint arXiv:2006.09517 , year=

    Linear last-iterate convergence in constrained saddle-point optimization , author=. arXiv preprint arXiv:2006.09517 , year=

    2006 arXiv

  109. [117]

    arXiv preprint arXiv:2204.09228 , year=

    Tight last-iterate convergence of the extragradient and the optimistic gradient descent-ascent algorithm for constrained monotone variational inequalities , author=. arXiv preprint arXiv:2204.09228 , year=

    arXiv

  110. [118]

    Advances in Neural Information Processing Systems , volume=

    Last-iterate convergent policy gradient primal-dual methods for constrained mdps , author=. Advances in Neural Information Processing Systems , volume=

  111. [119]

    International conference on machine learning , pages=

    Combinatorial multi-armed bandit: General framework and applications , author=. International conference on machine learning , pages=. 2013 , organization=

    2013

  112. [120]

    International Conference on Machine Learning , pages=

    Thompson sampling for (combinatorial) pure exploration , author=. International Conference on Machine Learning , pages=. 2022 , organization=

    2022

  113. [121]

    2024 IEEE 63rd Conference on Decision and Control (CDC) , pages=

    Optimal cooperative multiplayer learning bandits with noisy rewards and no communication , author=. 2024 IEEE 63rd Conference on Decision and Control (CDC) , pages=. 2024 , organization=

    2024

  114. [122]

    2010 IEEE Symposium on New Frontiers in Dynamic Spectrum (DySPAN) , pages=

    Learning multiuser channel allocations in cognitive radio networks: A combinatorial multi-armed bandit formulation , author=. 2010 IEEE Symposium on New Frontiers in Dynamic Spectrum (DySPAN) , pages=. 2010 , organization=

    2010

  115. [123]

    2022 IEEE 61st Conference on Decision and Control (CDC) , pages=

    Online learning for cooperative multi-player multi-armed bandits , author=. 2022 IEEE 61st Conference on Decision and Control (CDC) , pages=. 2022 , organization=

    2022

  116. [124]

    arXiv preprint arXiv:2503.08004 , year=

    Multiplayer Information Asymmetric Bandits in Metric Spaces , author=. arXiv preprint arXiv:2503.08004 , year=

    arXiv

  117. [125]

    Advances in Neural Information Processing Systems , volume=

    Combinatorial cascading bandits , author=. Advances in Neural Information Processing Systems , volume=

  118. [126]

    Advances in Neural Information Processing Systems , volume=

    Combinatorial bandits with linear constraints: Beyond knapsacks and fairness , author=. Advances in Neural Information Processing Systems , volume=

  119. [127]

    Decentralized cooperative multi-agent reinforcement learning with exploration , author=

  120. [128]

    International Conference on Algorithmic Learning Theory , pages=

    Decentralized cooperative reinforcement learning with hierarchical information structure , author=. International Conference on Algorithmic Learning Theory , pages=. 2022 , organization=

    2022

  121. [129]

    Journal of Computer and System Sciences , volume=

    Combinatorial bandits , author=. Journal of Computer and System Sciences , volume=. 2012 , publisher=

    2012

  122. [130]

    Advances in neural information processing systems , volume=

    Combinatorial bandits revisited , author=. Advances in neural information processing systems , volume=

  123. [131]

    Advances in neural information processing systems , volume=

    Combinatorial pure exploration of multi-armed bandits , author=. Advances in neural information processing systems , volume=

  124. [132]

    arXiv preprint arXiv:2406.01386 , year=

    Combinatorial multivariant multi-armed bandits with applications to episodic reinforcement learning and beyond , author=. arXiv preprint arXiv:2406.01386 , year=

    arXiv

  125. [133]

    2024 , school=

    Multi-agent reinforcement learning for nonzero-sum Markov games , author=. 2024 , school=

    2024

  126. [134]

    Advances in Neural Information Processing Systems , volume=

    Distributed multi-player bandits-a game of thrones approach , author=. Advances in Neural Information Processing Systems , volume=

  127. [135]

    2010 , school=

    Bandits games and clustering foundations , author=. 2010 , school=

    2010

  128. [136]

    Journal of Artificial Intelligence Research , volume=

    Combinatorial multi-armed bandits for real-time strategy games , author=. Journal of Artificial Intelligence Research , volume=

  129. [137]

    Advances in Neural Information Processing Systems , volume=

    Learning with bandit feedback in potential games , author=. Advances in Neural Information Processing Systems , volume=

  130. [138]

    Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment , volume=

    The combinatorial multi-armed bandit problem and its application to real-time strategy games , author=. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment , volume=

  131. [139]

    Advances in Neural Information Processing Systems , volume=

    Bandit learning in concave N-person games , author=. Advances in Neural Information Processing Systems , volume=

  132. [140]

    arXiv preprint arXiv:2403.03811 , year=

    Incentivized learning in principal-agent bandit games , author=. arXiv preprint arXiv:2403.03811 , year=

    arXiv

  133. [141]

    Information , volume=

    Addictive games: Case study on multi-armed bandit game , author=. Information , volume=. 2021 , publisher=

    2021

  134. [142]

    Conference on Learning Theory , pages=

    Multiplayer bandit learning, from competition to cooperation , author=. Conference on Learning Theory , pages=. 2021 , organization=

    2021

  135. [143]

    Advances in Neural Information Processing Systems , volume=

    Unifying PAC and regret: Uniform PAC bounds for episodic reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  136. [144]

    International conference on autonomous agents and multiagent systems , pages=

    Cooperative multi-agent control using deep reinforcement learning , author=. International conference on autonomous agents and multiagent systems , pages=. 2017 , organization=

    2017

  137. [145]

    Advances in neural information processing systems , volume=

    Multi-agent actor-critic for mixed cooperative-competitive environments , author=. Advances in neural information processing systems , volume=

  138. [146]

    Conference on Robot Learning , pages=

    The emergence of adversarial communication in multi-agent reinforcement learning , author=. Conference on Robot Learning , pages=. 2021 , organization=

    2021

  139. [147]

    The International Journal of Robotics Research , volume=

    Distributed multi-robot collision avoidance via deep reinforcement learning for navigation in complex scenarios , author=. The International Journal of Robotics Research , volume=. 2020 , publisher=

    2020

  140. [148]

    Machine Learning and Knowledge Extraction , volume=

    Robust reinforcement learning: A review of foundations and recent advances , author=. Machine Learning and Knowledge Extraction , volume=. 2022 , publisher=

    2022

  141. [149]

    arXiv preprint arXiv:2409.03052 , year=

    An introduction to centralized training for decentralized execution in cooperative multi-agent reinforcement learning , author=. arXiv preprint arXiv:2409.03052 , year=

    arXiv

  142. [150]

    Advances in neural information processing systems , volume=

    Learning to communicate with deep multi-agent reinforcement learning , author=. Advances in neural information processing systems , volume=

  143. [151]

    Distributed reinforcement learning in multi-agent networked systems , author=

  144. [152]

    Machine Learning: Proceedings of the Seventeenth International Conference (ICML'2000) , pages=

    Multi-agent reinforcement learning for traffic light control , author=. Machine Learning: Proceedings of the Seventeenth International Conference (ICML'2000) , pages=

    2000

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.