Opponent Aware Reinforcement Learning

David Gomez-Ullate Oteiza; David Rios Insua; Roi Naveiro; Victor Gallego

REVIEW 3 major objections 5 minor 45 references

Opponent Aware Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single RL agent can learn robust policies against reward-manipulating opponents by averaging its Q-learning update over their likely actions.

desk verdict Useful, honest paper on level-k Q-learning for adversarial settings, but the convergence theorem only covers a fixed opponent policy and the core update is essentially hyper-Q learning, which is not cited or compared against. read the letter →

arxiv 1908.08773 v2 pith:IE2FBE7L submitted 2019-08-22 cs.LG stat.ML

Victor Gallego , Roi Naveiro , David Rios Insua , David Gomez-Ullate Oteiza This is my paper · ORCID

classification cs.LGstat.ML

keywords threatenedMarkovdecisionprocessesadversarialreinforcementlearninglevel-kthinkingopponentmodelingQ-learningsecuritygamesriskanalysismulti-agent

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Threatened Markov Decision Processes (TMDPs), an extension of MDPs in which an adversary's actions modify rewards and transitions, and proposes a Q-learning variant that averages the standard update over the agent's beliefs about the adversary's action. The authors prove that the resulting update operator is a contraction, so standard reinforcement-learning convergence arguments apply. In experiments, the supported agent reaches Nash equilibria in repeated matrix games, avoids exploitation in a security gridworld, and handles multiple attackers in a resource-allocation game. The intended payoff is a practical way to make single-agent reinforcement learning robust to opponents without assuming common knowledge.

What carries the argument

The load-bearing object is the Threatened Markov Decision Process, a tuple $(S,A,B,T,R,p_A)$ that augments an MDP with a threat-action space $B$ for the opponent and a belief distribution $p_A(b|s)$ for the supported agent. Its mechanism is the averaged Q-learning update: first learn an action-pair value $Q(s,a,b)$, then collapse it to $Q(s,a)=\mathbb{E}_{p_A(b|s)}[Q(s,a,b)]$ before selecting an $\epsilon$-greedy action. That averaging is what converts opponent uncertainty into a robust single-agent policy, and Lemma 1 shows the resulting operator is a $\gamma$-contraction under the supremum norm.

What would settle it

Run a level-3 TMDP learner against a level-1 Q-learner in the stateless security game: the paper's own Fig. 9(c) shows the level-1 opponent exploits the level-3 agent, so a decisive test is whether the Bayesian model-averaging version reliably recovers positive reward when the prior is centered on the wrong level. A second test is to compare the TMDP averaged update against minimax-Q and WoLF-PHC in a zero-sum stochastic game over many seeds; if the TMDP learner's average reward is not at least as high, the robustness claim is not supported.

Watch

Extended reading notes

Core claim

The central claim is that a decision maker can defend against an opponent who interferes with the reward process by replacing the standard Q-learning update with an expectation over the adversary's likely action: $Q(s,a)=\mathbb{E}_{p_A(b|s)}[Q(s,a,b)]$, where $Q(s,a,b)$ itself updates with $r(s,a,b)+\gamma\max_{a'}\mathbb{E}_{p_A(b'|s')}[Q(s',a',b')]$. The belief $p_A(b|s)$ is learned either from empirical frequencies in the style of fictitious play, from a level-$k$ hierarchy in which each player models the other as a level-$(k-1)$ thinker, or from a Bayesian mixture over opponent models. The paper shows that the operator defining this update is a contraction in the supremum norm, so the learned Q-function converges. It then demonstrates empirically that an opponent-aware level-2 learner outperforms both naive Q-learning and the WoLF-PHC baseline in security settings.

Load-bearing premise

The whole scheme works only if the agent has a good picture of how the opponent decides: when the opponent acts outside the modeled classes, the learned policy can be exploited rather than protected.

Editorial extensions

If this is right

A level-1 fictitious-play learner reaches the Nash equilibrium in repeated Prisoner's Dilemma, Stag Hunt, and Chicken games, while an opponent-blind Q-learner does not.
Adding memory of the previous joint action lets the same framework learn to cooperate with a Tit-for-Tat opponent in the iterated Prisoner's Dilemma.
In the AI Safety Gridworlds friend-or-foe task, a level-2 learner obtains positive rewards where naive Q-learning is exploited, even when the true adversary is not exactly a level-1 Q-learner.
Maintaining a Bayesian mixture over opponent models lets the decision maker identify whether she faces a level-1 or level-2 opponent and avoids the failure of a mismatched level-3 model.
The update rule generalizes to deep Q-networks by replacing tabular Q-functions with neural function approximators.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the contraction proof takes the opponent policy $p_A(b|s)$ as known, the scheme's robustness in practice hinges on online estimation of that policy; a misspecified level can be worse than no model, so an adaptive level-selection rule with a misspecification guarantee is a natural next step.
The update rule is agnostic to how the Q-function is represented, so the same averaging idea should transfer to policy-gradient and actor-critic learners, which would cover continuous action spaces not treated in the paper.
The Bayesian mixture over opponent models points toward deployment-time adaptation: keep a distribution over opponent types, update it from observed actions, and train a single policy that is good against the mixture rather than against a fixed level-$k$ type.
The repeated-game results suggest the framework's advantage is largest when the opponent reacts to the supported agent's choices; in settings where both players learn at the same speed, level selection itself becomes the strategic decision and could be treated as an upper-level learning problem.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the short version. This paper does something genuinely useful: it gives a practical, one-sided way to make Q-learning robust to an adversary by averaging over opponent beliefs, and it wraps that in a level-k hierarchy with Bayesian model averaging over opponent types. The empirical section is broader than typical, includes honest failure cases (a level-3 DM exploited by a level-1 opponent, Fig 9c), and the code is public. If you work on adversarial RL or security games, this is worth knowing about.

What is actually new: the TMDP formalism is a clean way to extend MDPs with an opponent action, the level-k thinking scheme that estimates the opponent's Q-function is a useful concrete algorithm, and the Dirichlet model-averaging over different opponent models is a nice practical answer to the 'which level is my opponent?' problem. Credit where due: the contraction lemma in Appendix A is correct, as far as it goes, for a fixed opponent policy. The experiments span matrix games, gridworlds, and a Blotto-style allocation game, which is more than most papers in this space.

The soft spots, in order of real importance. First, the convergence claim is narrower than stated. Lemma 1 fixes p(b|s); Algorithm 1 estimates that policy online from a learned opponent Q-function, so the operator is time-varying and the standard Q-learning convergence argument does not transfer. The paper does not quite misstate this—Appendix A says 'if she knows his policy, or has a good estimate of it'—but the conclusion in Section 5 that 'we may use RL convergence results' is an overreach. The practical consequence shows in Fig 9(c), where a misspecified level assumption gets the DM exploited.

Second, the central update rule is close to hyper-Q learning (Tesauro 2003), which is not cited. The level-k hierarchy and the Bayesian averaging are the actual novelty, and the paper would be stronger and more honest if it acknowledged the lineage and compared against a hyper-Q baseline.

Third, and minor: the main evidence is qualitative reward curves; the hyperparameter robustness table is nice, but the paper does not report variance across seeds for the headline gridworld results, so I would want a few summary numbers.

Who is this for? Researchers in multi-agent RL and security games who want a practical opponent-modeling recipe. It deserves a serious referee. I would accept it with minor-to-major revision: fix the convergence statement, cite and compare against hyper-Q, and add summary statistics.

Referee Report

3 major / 5 minor

Summary. The paper introduces Threatened Markov Decision Processes (TMDPs), an extension of MDPs in which a supported decision maker (DM) faces an adversary whose actions b enter both the transition and reward processes. The DM maintains beliefs p_A(b|s) about the adversary's actions, and the paper proposes modifying Q-learning by averaging over these beliefs, together with a level-k thinking hierarchy in which the DM also estimates the opponent's Q-function. The framework includes fictitious-play-style and Bayesian opponent models, a model-averaging scheme for uncertain opponent types, and an extension to multiple adversaries. The theoretical contribution is a contraction argument in Appendix A intended to justify convergence of the update rule. Empirical results are reported for repeated matrix games, AI Safety Gridworlds, and a security resource allocation (Blotto) game, with code released online.

Significance. If the claims hold, the TMDP framework is a useful one-sided prescriptive tool for security-oriented reinforcement learning, and the level-k scheme is a natural way to inject opponent modeling into Q-learning without common-knowledge assumptions. The paper's strengths include a broad and honest empirical study, including a documented failure case (Fig. 9(c)), robustness checks in Appendix D, and publicly released code. The contraction lemma in Appendix A is correct for a fixed opponent policy, and the computational complexity analysis is sensible. The significance is tempered, however, by the gap between the proven contraction result and the online opponent-model estimator used in the actual algorithm, as well as by the reliance on the opponent being close to the modeled class.

major comments (3)

[Appendix A; Eqs. (3)-(4); Algorithm 1] Lemma 1 and the surrounding argument prove that the operator H is a contraction only for a fixed opponent policy p(b'|s') and a fixed transition kernel. In Algorithm 1, however, the expectation in the Q2 update is taken with respect to p_A(b|s1), which is recomputed at every step from the evolving estimate Q^1 of the opponent's Q-function. The resulting stochastic update is a Q-learning iteration with a time-varying target operator, so the cited standard convergence result (Ref. [45]) does not apply, and the paper's statement that 'we may use RL convergence results' overreaches what is proved. I recommend either proving convergence of the coupled stochastic approximation under explicit conditions (e.g., consistency or slow variation of the opponent-model estimates), or restating the theoretical contribution as convergence of the TMDP optimality operator for a fixed opponent model and presenting the online level-k scheme as a heuristic supported by the experiments.
[Section 4.2.2 (Fig. 9(c)-(e))] The failure of the level-3 DM against a level-1 opponent in Fig. 9(c) is a direct illustration of the load-bearing assumption that the opponent is, or is close to, the modeled type. The paper acknowledges this and proposes the Bayesian model-averaging scheme of Section 3.3, but that scheme is demonstrated on only one configuration (Figs. 9(d)-(e)), and no characterization is given of when the posterior will concentrate on the correct model. Without such a characterization, the claim that the framework 'generalize[s] between different kinds of opponents' remains an empirical observation rather than a supported property. The paper should either add a formal or systematic empirical study of model misspecification (e.g., varying the true opponent level and the candidate model set) or explicitly limit the generalization claim.
[Section 4.1, Figs. 2-4] The text repeatedly states that the FPQ-learner 'converges to the Nash equilibrium' in IPD, ISH, and IC, but the evidence shown is only reward trajectories. Since equilibrium selection is a policy-level claim, please report the empirical frequency of the DM's actions (as is done for the stateless security game in Section 4.2.2) or a distance-to-equilibrium metric for the matrix games; otherwise the reader cannot distinguish convergence to an equilibrium from convergence of average rewards.

minor comments (5)

[Abstract and Appendix A title] The abstract contains the typo 'suporting' and the Appendix A title uses 'TDMPs' instead of 'TMDPs'; please correct these.
[Fig. 9 caption] The caption for Fig. 9 says 'Rewards obtained against the exponential smoother adversary,' but panels (a)-(e) are against level-k Q-learners, as Section 4.2.2 itself states; the caption should be updated to reflect the actual opponents.
[Section 4.1] The sentence 'we focus on the stateless version of a TMDP and analyze the policies learnt by the DM, and analyze the policies learnt by the DM' repeats the same phrase; please remove the duplication.
[Algorithm 4 (Appendix B)] The deep-Q generalization omits standard DQN machinery such as target networks and replay buffers; even if these are intended to be unchanged from standard DQN, a sentence clarifying that the extension inherits the usual deep-RL components would help reproducibility.
[Section 3, Eq. (4)] Equation (4) defines Q(s,a) as an expectation over the opponent's actions, but the surrounding text sometimes refers to Q(s,a) without distinguishing the decision Q-function from the action-opponent Q(s,a,b) in Eq. (3); a brief notational clarification would avoid ambiguity.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: contraction proof is standard, experiments are external, and author self-citations are not load-bearing; the fixed-policy convergence gap is a scope limitation, not a circular reduction.

full rationale

The paper's central derivation is self-contained. Lemma 1 in Appendix A defines an operator H on Q-functions with a fixed opponent policy p(b'|s') and proves ||Hq1-Hq2||∞ ≤ γ||q1-q2||∞; this is the standard Bellman contraction for the augmented-state MDP, so invoking the standard Q-learning convergence result is ordinary theory, not a circular reduction. The level-k hierarchy in Section 3.2 is a modeling ansatz about how to represent opponents, not a theorem derived from the target result, and the experiments test it against independent agents and benchmarks (WoLF-PHC, exponential smoother, Tit-For-Tat, AI Safety Gridworlds), including a genuine failure case in Fig 9(c) and hyperparameter robustness in Appendix D.2, so the results are not fitted to force the conclusion. The only author self-citations ([9] and [34]) are background references for adversarial risk analysis and are not load-bearing: no equation or convergence claim is imported from them. A real limitation is that the convergence proof assumes a fixed, known opponent policy p(b'|s'), whereas Algorithm 1 updates the opponent-model estimate online; this makes the theoretical guarantee narrower than the stated claim, but it is an overreach in the scope of proof rather than circularity, because the theorem does not assume the conclusion that the online level-k rule converges. Overall circularity burden is low.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework rests on several load-bearing assumptions not justified by the paper's own derivations: observability of opponent rewards/actions, a Markovian opponent model in the convergence proof, and the assumption that the adversary is a level-(k-1) Q-learner. The hyperparameters listed are standard, but the forget factor and the zero-sum reward assumption for the adversary are hand-chosen. No new physical entities are introduced.

free parameters (4)

Forget factor lambda = 0.8
Controls recency weighting in the Dirichlet-Categorical update for the level-1 opponent model (Algorithm 3, Section 4.2.1, Appendix C). Chosen by hand; robustness not fully explored.
Adversary reward model r_B = r_B = -r_A (zero-sum)
In the security experiments the DM assumes the level-1 opponent maximizes -r_A because the adversary's true payoffs are unknown (Section 4.2.1). This is a modeling input that changes the opponent's estimated Q-function.
Thinking level k = 2 or 3, with a rule of thumb
The DM must choose the hierarchy depth; level-3 fails against level-1 opponents unless model averaging is used (Fig 9(c)-9(d)). This is a model selection parameter.
Dirichlet/Beta prior parameters = Beta(1,1) or Dirichlet(alpha)
Initial pseudocounts for opponent action beliefs (Section 3.1, Appendix C). Standard non-informative choice.

assumptions (5)

domain assumption Both agents observe each other's actions and rewards after they are chosen
Stated in Section 3 before Section 3.1; Algorithm 1 updates the opponent Q-function using r_B and b, so unobserved opponent rewards or actions would break the scheme.
domain assumption The opponent's next action depends only on the next state, p(s',b'|s,a,b) = p(b'|s') p(s'|s,a,b)
Used in the contraction proof in Appendix A. The memory-1 experiments, with state s_t=(a_{t-1}, b_{t-1}), satisfy this with an enlarged state, but the non-Markov cases are not covered.
ad hoc to paper The opponent is (or is well approximated by) a level-(k-1) Q-learner of the same tabular type
The level-k recursion in Section 3.2 estimates the opponent's Q-function with the same update. Fig 9(c) shows a level-3 DM is exploited by a level-1 opponent, so the method's success depends on this model class being correct.
standard math Standard Q-learning convergence conditions (infinite exploration, Robbins-Monro step sizes) hold for the tabular updates
The proof invokes the standard Q-learning convergence result [45] for the update in Eq. (3), but does not verify these conditions for the time-varying estimated p.
ad hoc to paper The adversary's reward in the security experiments can be set to zero-sum r_B = -r_A
Section 4.2.1: 'we do not know the actual rewards of the adversary ... model it as in a zero-sum scenario.' Robustness to other scalings is tested in Appendix D.3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Opponent Aware Reinforcement Learning." pith.science (2026). https://pith.science/paper/IE2FBE7L

@misc{pith2026190808773,
  author       = {Pith},
  title        = {Pith review of: Opponent Aware Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IE2FBE7L}},
  note         = {Machine review of arXiv:1908.08773}
}

read the original abstract

We introduce Threatened Markov Decision Processes (TMDPs) as an extension of the classical Markov Decision Process framework for Reinforcement Learning (RL). TMDPs allow suporting a decision maker against potential opponents in a RL context. We also propose a level-k thinking scheme resulting in a novel learning approach to deal with TMDPs. After introducing our framework and deriving theoretical results, relevant empirical evidence is given via extensive experiments, showing the benefits of accounting for adversaries in RL while the agent learns

Figures

Figures reproduced from arXiv: 1908.08773 by the authors.

**Figure 1.** Level-k thinking scheme, with k “ 2 Note that in the previous hierarchy of policies the decisions are obtained in a greedy manner, by maximizing the lower level Qˆ estimate. We may gain insight in a Bayesian fashion by adding uncertainty to the policy at each level. For instance, at a certain level in the hierarchy, we could consider ´greedy policies that with probability 1 ´ choose an action according to the pre… view at source ↗

**Figure 2.** Rewards obtained in IPD. We plot the trajectories of 10 simulations with shaded [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗

**Figure 3.** Rewards in ISH game (a) Q-learner vs Q-learner (b) FPQ-learner (blue) vs Q-learner (red) [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

Figures from the paper (8 more)

**Figure 4.** Figure 4: Rewards in IC game 14 [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

**Figure 5.** Figure 5: Rewards obtained in the IC game against a WoLF-PHC adversary. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

**Figure 6.** Figure 6: Rewards obtained by the DM for players: FPQ memoryless player vs TFT player [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

**Figure 7.** Figure 7: The friend or foe environment from the AI Safety Gridworlds benchmark. Figure taken from [38]. where 0 ă β ă 1 is a learning rate, unknown from the DM’s point of view, and a P tp1, 0q,p0, 1qu is a one-hot encoded vector indicating whether the DM chose targets 1 or 2. W…

**Figure 8.** Figure 8: Rewards for the DM against the adversarial opponent [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]

**Figure 9.** Figure 9: Rewards obtained against the exponential smoother adversary. [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]

**Figure 10.** Figure 10: Rewards against the adversarial (exponential smoother) opponent in the spatial [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]

**Figure 11.** Figure 11: Rewards for the DM against the adversarial opponent [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 40 canonical work pages

[45]

F. S. Melo, Convergence of q-learning: A simple proof, Tech. Rep. (2001). 36

work page 2001
[1]

I. J. Goodfellow, J. Shlens, C. Szegedy, Explaining and harnessing adversarial examples, arXiv preprint arXiv:1412.6572 (2014)

arXiv 2014
[2]

J. G. Carbonell, Introduction:paradigms for machine learning, Artiﬁcial Intelligence 40 (1989) 1 – 9

work page 1989
[3]

S. V. Albrecht, P. Stone, Autonomous agents modelling other agents: A comprehensive survey and open problems, Artif. Intell. 258 (2018) 66–95

work page 2018
[4]

Dalvi, P

N. Dalvi, P. Domingos, S. Sanghai, D. Verma, et al., Adversarial classiﬁcation, in: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, 2004, pp. 99–108

work page 2004
[5]

Menache, A

I. Menache, A. Ozdaglar, Network games: Theory, models, and dynamics, Synthesis Lectures on Communication Networks 4 (2011) 1–159

work page 2011
[6]

Biggio, F

B. Biggio, F. Roli, Wild patterns: Ten years after the rise of adversarial machine learning, Pattern Recognition 84 (2018) 317 – 331

work page 2018
[7]

Y. Zhou, M. Kantarcioglu, B. Xi, A survey of game theoretic approach for adversarial machine learning, Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery (2018) e1259

work page 2018

Show all 45 references

[8]

Hargreaves-Heap, Y

S. Hargreaves-Heap, Y. Varoufakis, Game Theory: A Critical Introduc- tion, Taylor & Francis, 2004

2004
[9]

Naveiro, A

R. Naveiro, A. Redondo, D. R. Insua, F. Ruggeri, Adversarial classiﬁ- cation: An adversarial risk analysis approach, International Journal of Approximate Reasoning (2019)

2019
[10]

D. R. Insua, J. Rios, D. Banks, Adversarial risk analysis, Journal of the American Statistical Association 104 (2009) 841–854

2009
[11]

J. B. Kadane, P. D. Larkey, Subjective probability and the theory of games, Management Science 28 (1982) 113–120. 32

1982
[12]

Raiﬀa, The Art and Science of Negotiation, Belknap Press of Harvard University Press, 1982

H. Raiﬀa, The Art and Science of Negotiation, Belknap Press of Harvard University Press, 1982

1982
[13]

V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., Human-level control through deep reinforcement learning, Nature 518 (2015) 529

2015
[14]

Silver, J

D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al., Mastering the game of go without human knowledge, Nature 550 (2017) 354

2017
[15]

Huang, N

S. Huang, N. Papernot, I. Goodfellow, Y. Duan, P. Abbeel, Adversarial attacks on neural network policies, arXiv preprint arXiv:1702.02284 (2017)

2017 arXiv
[16]

Lin, Z.-W

Y.-C. Lin, Z.-W. Hong, Y.-H. Liao, M.-L. Shih, M.-Y. Liu, M. Sun, Tactics of adversarial attack on deep reinforcement learning agents, arXiv preprint arXiv:1703.06748 (2017)

2017 arXiv
[17]

Bu¸ soniu, R

L. Bu¸ soniu, R. Babuˇ ska, B. De Schutter, Multi-agent reinforcement learning: An overview, in: Innovations in multi-agent systems and applications-1, Springer, 2010, pp. 183–221

2010
[18]

R. A. Howard, Dynamic Programming and Markov Processes, MIT Press, Cambridge, MA, 1960

1960
[19]

R. S. Sutton, A. G. Barto, Reinforcement learning: An introduction, MIT press, 2018

2018
[20]

G. W. Brown, Iterative solution of games by ﬁctitious play, Activity Analysis of Production and Allocation (1951) 374–376

1951
[21]

J. Rios, D. R. Insua, Adversarial risk analysis for counterterrorism modeling, Risk Analysis: An International Journal 32 (2012) 894–915

2012
[22]

D. O. Stahl, P. W. Wilson, Experimental evidence on players’ models of other players, Journal of economic behavior & organization 25 (1994) 309–327. 33

1994
[23]

M. L. Littman, Markov games as a framework for multi-agent reinforce- ment learning, in: Machine Learning Proceedings 1994, Elsevier, 1994, pp. 157–163

1994
[24]

J. Hu, M. P. Wellman, Nash Q-learning for general-sum stochastic games, Journal of machine learning research 4 (2003) 1039–1069

2003
[25]

M. L. Littman, Friend-or-Foe Q-learning in General-Sum Games, in: Proceedings of the Eighteenth International Conference on Machine Learning, Morgan Kaufmann Publishers Inc., 2001, pp. 322–328

2001
[26]

P. Auer, N. Cesa-Bianchi, Y. Freund, R. E. Schapire, Gambling in a rigged casino: The adversarial multi-armed bandit problem, in: Foundations of Computer Science, 1995. Proceedings., 36th Annual Symposium on, IEEE, 1995, pp. 322–331

1995
[27]

Lanctot, V

M. Lanctot, V. Zambaldi, A. Gruslys, A. Lazaridou, K. Tuyls, J. P´ erolat, D. Silver, T. Graepel, A uniﬁed game-theoretic approach to multiagent reinforcement learning, in: Advances in Neural Information Processing Systems, 2017, pp. 4190–4203

2017
[28]

P. J. Gmytrasiewicz, P. Doshi, A framework for sequential planning in multi-agent settings, Journal of Artiﬁcial Intelligence Research 24 (2005) 49–79

2005
[29]

H. He, J. Boyd-Graber, K. Kwok, H. Daum´ e III, Opponent modeling in deep reinforcement learning, in: International Conference on Machine Learning, 2016, pp. 1804–1813

2016
[30]

Foerster, R

J. Foerster, R. Y. Chen, M. Al-Shedivat, S. Whiteson, P. Abbeel, I. Mor- datch, Learning with opponent-learning awareness, in: Proceedings of the 17th International Conference on Autonomous Agents and Multi- Agent Systems, International Foundation for Autonomous Agents and Mul...

2018
[31]

D. O. Stahl, P. W. Wilson, On players’ models of other players: Theory and experimental evidence, Games and Economic Behavior 10 (1995) 218–254

1995
[32]

Altman, Constrained Markov Decision Processes, volume 7, CRC Press, 1999

E. Altman, Constrained Markov Decision Processes, volume 7, CRC Press, 1999. 34

1999
[33]

A. M. Metelli, M. Mutti, M. Restelli, Conﬁgurable Markov Deci- sion Processes, International Conference on Machine Learning (2018). arXiv:1806.05415

2018 arXiv
[34]

D. R. Insua, D. Banks, J. Rios, Modeling opponents in adversarial risk analysis, Risk Analysis 36 (2016) 742–755

2016
[35]

H. Tang, R. Houthooft, D. Foote, A. Stooke, O. X. Chen, Y. Duan, J. Schulman, F. DeTurck, P. Abbeel, # exploration: A study of count- based exploration for deep reinforcement learning, in: Advances in Neural Information Processing Systems, 2017, pp. 2750–2759

2017
[36]

A. E. Raftery, A model for high-order markov chains, Journal of the Royal Statistical Society. Series B (Methodological) (1985) 528–539

1985
[37]

C. F. Camerer, T.-H. Ho, J.-K. Chong, A cognitive hierarchy model of games, The Quarterly Journal of Economics 119 (2004) 861–898

2004
[38]

Leike, M

J. Leike, M. Martic, V. Krakovna, P. A. Ortega, T. Everitt, A. Lefrancq, L. Orseau, S. Legg, AI safety gridworlds, arXiv preprint arXiv:1711.09883 (2017)

2017 arXiv
[39]

Axelrod, The Evolution of Cooperation, Basic, New York, 1984

R. Axelrod, The Evolution of Cooperation, Basic, New York, 1984

1984
[40]

Bowling, M

M. Bowling, M. Veloso, Rational and convergent learning in stochastic games, in: Proceedings of the 17th international joint conference on Artiﬁcial intelligence-Volume 2, Morgan Kaufmann Publishers Inc., 2001, pp. 1021–1026

2001
[41]

W. H. Press, F. J. Dyson, Iterated prisoners dilemma contains strategies that dominate any evolutionary opponent, Proceedings of the National Academy of Sciences 109 (2012) 10409–10413

2012
[42]

Klima, K

R. Klima, K. Tuyls, F. Oliehoek, Markov security games: Learning in spatial security problems, NIPS Workshop on Learning, Inference and Control of Multi-Agent Systems (2016)

2016
[43]

Hart, Discrete Colonel Blotto and General Blotto Games, International Journal of Game Theory 36 (2008) 441–460

S. Hart, Discrete Colonel Blotto and General Blotto Games, International Journal of Game Theory 36 (2008) 441–460. 35

2008
[44]

Baxter, P

J. Baxter, P. L. Bartlett, Direct gradient-based reinforcement learn- ing, in: 2000 IEEE International Symposium on Circuits and Systems. Emerging Technologies for the 21st Century. Proceedings (IEEE Cat No. 00CH36353), volume 3, IEEE, 2000, pp. 271–274

2000

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [45]

F. S. Melo, Convergence of q-learning: A simple proof, Tech. Rep. (2001). 36

work page 2001

[2] [1]

I. J. Goodfellow, J. Shlens, C. Szegedy, Explaining and harnessing adversarial examples, arXiv preprint arXiv:1412.6572 (2014)

arXiv 2014

[3] [2]

J. G. Carbonell, Introduction:paradigms for machine learning, Artiﬁcial Intelligence 40 (1989) 1 – 9

work page 1989

[4] [3]

S. V. Albrecht, P. Stone, Autonomous agents modelling other agents: A comprehensive survey and open problems, Artif. Intell. 258 (2018) 66–95

work page 2018

[5] [4]

Dalvi, P

N. Dalvi, P. Domingos, S. Sanghai, D. Verma, et al., Adversarial classiﬁcation, in: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, 2004, pp. 99–108

work page 2004

[6] [5]

Menache, A

I. Menache, A. Ozdaglar, Network games: Theory, models, and dynamics, Synthesis Lectures on Communication Networks 4 (2011) 1–159

work page 2011

[7] [6]

Biggio, F

B. Biggio, F. Roli, Wild patterns: Ten years after the rise of adversarial machine learning, Pattern Recognition 84 (2018) 317 – 331

work page 2018

[8] [7]

Y. Zhou, M. Kantarcioglu, B. Xi, A survey of game theoretic approach for adversarial machine learning, Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery (2018) e1259

work page 2018