Pith. sign in

REVIEW 3 major objections 5 minor 34 references

Successor Features for Transfer in Alternating Markov Games

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Successor features, a single-agent transfer tool, extend to zero-sum alternating Markov games, where a new Game Generalized Policy Improvement selection is proven to be no worse than the worst source policy up to an approximation penalty.

desk verdict The central GGPI theorem is false as stated; the paper's main theoretical contribution collapses, though the SF-in-games idea and small experiments have some merit. read the letter →

arxiv 2507.22278 v1 pith:VGY225TQ submitted 2025-07-29 cs.MA cs.GT

Sunny Amatya , Yi Ren , Zhe Xu , Wenlong Zhang This is my paper · ORCID
classification cs.MAcs.GT MSC 91A1568T05
keywords successorfeatureszero-sumMarkovgamesalternatinggeneralizedpolicyimprovementtransferlearningmulti-agentreinforcementpursuer-evadergameminmaxQ-learning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that successor features, a single-agent technique for reusing learned value functions across tasks with related rewards, can be extended to two-player zero-sum games where players act in turns. The new ingredient is Game Generalized Policy Improvement (GGPI): when several source policies are available, the ego player selects the action that maximizes the payoff under the worst opponent response and the worst of the source policies. The authors prove that this selection is guaranteed to score at least as well as the worst source policy on the new task, up to an approximation penalty that vanishes as the learned Q-functions become exact. Because the transfer step is just a dot product between a stored successor-feature table and the new task's reward weights, adapting to a new goal is nearly instantaneous. In a grid pursuer-evader game, the resulting one-shot policy transfer beats retraining from scratch and gives shorter, more successful paths.

What carries the argument

The central object is the successor-feature table $\psi^{\Pi_i}(s,a,b)$, a per-policy, per-state-action-opponent-action vector of discounted feature sums, together with the linear-reward assumption $r_w(s,a,b)=\phi(s,a,b)^\top w$ that turns cross-task evaluation into a dot product. On top of this, Game Generalized Policy Improvement (GGPI) selects $\arg\max_a\min_b\min_i \tilde Q^{\Pi_i}(s,a,b)$, i.e. the action that maximizes the minimum over the opponent's responses and over all stored source policies. The proof mechanism is the Bellman operator's monotonicity and contraction: applying the operator to $\min_i \tilde Q^{\Pi_i}$ propagates the $\epsilon$ approximation error at rate $\mu$ per step, which after geometric summation yields the $2\epsilon/(1-\mu)$ penalty in Theorem 1. The companion task-similarity bound $\eta_{ij}/(1-\mu)$ is what lets the paper phrase transfer error as a function of reward similarity.

What would settle it

Search small tabular alternating Markov games with exactly known Q-functions (so $\epsilon=0$) for a counterexample to Theorem 1: if the GGPI policy's exact value is ever below $\min_i Q^{\Pi_i}(s,a,b)$ at some state-action pair, the bound as stated fails; equivalently, run the paper's own pursuer-evader transfer with a nonlinear target reward over the same features and check whether the initial transferred policy falls below the worst source policy's value by more than the bound allows.

Watch

Extended reading notes

Core claim

The paper's central claim is that in a two-player, zero-sum, turn-based (alternating) Markov game, knowledge acquired on a set of source tasks can be transferred to a new task with the same state and action spaces and transition structure by storing, for each source policy $\Pi_i$, the successor features $\psi^{\Pi_i}(s,a,b)$. Because each task's reward is assumed linear, $r_w(s,a,b)=\phi(s,a,b)^\top w$, evaluating any stored policy under a new reward is a dot product $\psi^{\Pi_i}(s,a,b)^\top w_{n+1}$. The paper defines the GGPI policy $\pi(s)\in\arg\max_a\min_b\min_i \tilde Q^{\Pi_i}(s,a,b)$ and proves (Theorem 1) that if each stored Q-function is within $\epsilon$ of its true value, the resulting policy satisfies $Q^{\pi}(s,a,b)\ge \min_i Q^{\Pi_i}(s,a,b) - 2\epsilon/(1-\mu)$, where $\mu$ is the discount factor. A companion lemma bounds the value gap between two tasks by $\eta_{ij}/(1-\mu)$, where $\eta_{ij}=\max_{s,a,b}|r_i(s,a,b)-r_j(s,a,b)|$, so the transferred policy's suboptimality is at most $2\eta_{ij}/(1-\mu)+2\epsilon/(1-\mu)$. The paper takes its 5x5 grid pursuer-evader experiments as evidence that this one-shot transfer works in practice.

Load-bearing premise

The load-bearing premise is that every source and target task share the same state space, action spaces, transition probabilities, and discount factor, and that every reward is exactly linear in one common feature vector whose weights for the new task are known; the paper itself notes it additionally assumes a discrete action space, a conservative best-responding opponent, and comparable Q-value ranges across tasks.

Editorial extensions

If this is right

  • A library of source policies and their successor-feature tables is enough to act immediately in a new task: the ego computes $\psi^{\Pi_i}(s,a,b)^\top w_{n+1}$ for each stored $i$ and plays the GGPI action, with no gradient updates.
  • The guarantee is worst-case over both the opponent and the source set, so the transferred policy is conservative by construction in adversarial settings.
  • The task-similarity bound $\eta_{ij}/(1-\mu)$ gives a quantitative criterion for deciding which source tasks are worth storing: tasks whose rewards are far apart contribute little.
  • In the tabular grid experiments, the authors find that value transfer and one-shot policy transfer occur when the goal changes, and that SFminmax attains higher success rates and path efficiency than MinMax Q-learning, the epsilon-reset ablation, and Probabilistic Policy Reuse.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the linear-reward assumption fails, one could still run GGPI with learned feature embeddings, but the constant $\epsilon$ would need to absorb representation error; the paper does not analyze this case.
  • Beyond the paper: because GGPI selects against the worst source policy, it may be overly conservative when one source task is much harder than the rest; weighting source policies by their relevance to the target could improve average performance, an option not tested here.
  • Beyond the paper: the supplementary's failed preliminary run without a terminal-reward feature suggests the method's success depends on the feature set being expressive enough to encode the goal; a diagnostic experiment would vary feature completeness and measure how tightly the GGPI bound holds.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an extension of successor features (SFs) to alternating, turn-based, zero-sum Markov games. The authors define a Game Generalized Policy Improvement (GGPI) operator that, from n source policies, selects actions by argmax over the ego action a of min over opponent action b and min over source policies i of the approximate Q-function. The central theoretical claim is Theorem 1 (Eq. 12): the GGPI policy's Q-function is pointwise at least min_i Q^{Π_i}(s,a,b) - 2ε/(1-γ). The paper also states Lemma 1, bounding reward-induced differences of value functions, and Proposition 1, combining these bounds. Experiments on a grid pursuer-evader game compare SFminmax with MinMax Q-learning, PRQL, and an exploration-reset variant, reporting higher cumulative return, one-shot policy transfer, and better success-weighted path length in some settings. The supplementary material contains proofs of the theorem and lemma plus implementation details.

Significance. If Theorem 1 were correct, the paper would provide a principled transfer guarantee for adversarial multi-agent settings, extending Barreto et al.'s generalized policy improvement to alternating Markov games. That would be a useful contribution, and the paper also provides a concrete algorithmic instantiation and a nontrivial experimental setup. However, the central theoretical result is false as stated, even for exact value functions. The counterexample in the report shows that the minimum over source policies is not closed under the game Bellman operator, so the GGPI policy can be substantially worse than every source policy. Since the one-shot transfer claim rests on Eq. (12), the main contribution is not supported. The experiments may still show useful heuristics, but they do not compensate for an incorrect load-bearing theorem.

major comments (3)
  1. [Section IV-A, Eq. (12)] Theorem 1 is false as stated, even in the exact-value case ε=0. Let S={s0,s1}, with any action pair at s0 transitioning deterministically to s1 with reward 0, and no transition out of s1. At s1 the terminal rewards are r(a1,b1)=10, r(a1,b2)=-100, r(a2,b1)=-100, r(a2,b2)=10. Let source policy Π1 choose (a1,b1) at s1 and source policy Π2 choose (a2,b2) at s1. With γ=0.9, for every (a,b) at s0, Q^{Π_i}(s0,a,b)=0.9*r(a_i,b_i)=9, so min_i Q^{Π_i}(s0,a,b)=9. At s1, min_i Q^{Π_i}(s1,a,b)=r(a,b); therefore the GGPI action at s1 is any a with min_b r(a,b)=-100, and the opponent then chooses the corresponding b. The GGPI policy therefore has value 0.9*(-100)=-90 at s0, which is strictly less than min_i Q^{Π_i}(s0,a,b)=9. This contradicts Eq. (12) with ε=0. The structural reason is that min_i does not commute with the max-min Bellman operator: the source attaining the minimum can switch from state to state, so min_i Q^{Π_i} is not the value function of any stationary policy.
  2. [Supplementary Material, Section I-A] The proof of Theorem 1 contains an invalid inequality. It claims |min_i Q^{Π_i}(s,a,b) - min_i \tilde Q^{Π_i}(s,a,b)| = min_i |Q^{Π_i}(s,a,b) - \tilde Q^{Π_i}(s,a,b)|, but the left-hand side is not equal to the minimum; the correct bound is ≤ max_i |Q^{Π_i} - \tilde Q^{Π_i}|. More fundamentally, the proof applies the policy Bellman operator T^Π to \tilde Q_min and then compares the result to Q^{Π_i} as if \tilde Q_min were the value function of the GGPI policy. Since min_i Q^{Π_i} is not closed under the game Bellman operator, the subsequent 'property of Bellman operator' step reverses the inequality direction and does not establish the claimed lower bound.
  3. [Supplementary Material, Lemma 1] Lemma 1 as stated in the main text claims |Q^Π_i(s,a,b)-Q^Π_j(s,a,b)| ≤ ¶_{ij}/(1-γ) for an arbitrary policy Π, but the proof in the supplementary material derives a bound of 2¶_{ij}/(1-γ) and does so for Q^i_i versus Q^j_i, i.e., for the value of the j-th optimal policy evaluated in task i, not for an arbitrary policy. The notation Q^j_i is introduced as Q^{Π*_j}_i, which is not the quantity appearing in Lemma 1. This mismatch means Lemma 1, as used in Proposition 1, is not actually proved.
minor comments (5)
  1. [Section III-B, Eq. (6)] Equation (6) writes Q^*(s,a',b') inside the max-min operator, but the Bellman recursion should use the next state s', i.e., Q^*(s',a',b').
  2. [Section III-A, Eqs. (2)-(3)] The successor feature definition in Eq. (2) uses features of the next state-action pair, ϕ(s_{t+1},a_{t+1}), and a discount μ^t, while Eq. (3) equates ψ^π(s,a)^T w to the expected discounted reward from time 0. The indexing should be aligned with the reward function in Eq. (1), otherwise the equality in Eq. (3) is off by one time step and by the terminal reward convention.
  3. [Algorithm 1, lines 7 and 18] The notation argmax_b argmin_a min_i Ψ^{Π_i}(s,a,b)^T w_i is confusing: it is not the same as the GGPI rule in Theorem 1, and it suggests a different order of the max and min operations. The algorithm should use the same operator as Eq. (12), and Ψ should be explicitly distinguished from Q.
  4. [Section V, Tables II and Fig. 1] The experimental tables report point estimates without standard errors or repeated seeds; several entries are marked 'NA' or contain inconsistent decimal notation, which makes the value-discrepancy and SPL comparisons difficult to interpret.
  5. [Section VI] The limitation paragraph states that the method 'relies on the Q-values of the training and test tasks being within comparable bounds' and 'becomes less effective when the nature or weighting of the tasks changes significantly.' This is directly related to the failure mode of the main theorem and deserves a concrete test; the manuscript does not measure robustness to that assumption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the GGPI transfer bound is derived from the linear-reward successor-feature identity and not from a fitted input or self-citation.

full rationale

The paper's central transfer computation Q^{Π_i}_{n+1} = ψ^{Π_i}(s,a,b)^T w_{n+1} is a direct consequence of the definitions in Eqs. (1)-(3): since r_w = φ^T w and ψ is the expected discounted feature sum, multiplying ψ by the new task's weight vector is exactly policy evaluation under the linear reward, with no fitted parameter inserted into the bound. Theorem 1's guarantee (Eq. 12) is not a renamed input; the GGPI action is defined by argmax_a min_b min_i Q~^{Π_i}, while the claimed lower bound concerns the actual Q of the induced policy, so the conclusion is not identical to the definition of the policy. The supplementary proof's treatment of Qmin = min_i Q^{Π_i} as if it satisfied the game Bellman equation is a substantive mathematical error—min_i does not commute with the max-min Bellman operator—but an incorrect inference is not circularity. The paper's self-citations ([5], [16]) concern intent estimation and empathetic parameter estimation and are not load-bearing for the transfer theorem; the cited SF/GPI results [10], [18] are external and the game extension is argued directly. The limitation section concedes that the approach assumes discrete actions, worst-case opponent modeling, and comparable Q-value ranges, and several experimental constants (terminal reward 0.7, learning rates, feature weights) are tuned; these are empirical and assumption risks, not cases where a predicted quantity is forced by construction. Thus no circular step meets the required evidence standard.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central derivation rests on the linear reward decomposition and shared-dynamics assumption inherited from successor features, plus the unproven error propagation property in the GGPI proof. No new physical or conceptual entities are introduced. Several experimental constants are tuned rather than predicted.

free parameters (4)
  • terminal_reward_token = 0.7
    Chosen based on parameter tuning (Sec. V-A.1); all tasks reward reaching the goal with token 0.7, so the experimental results depend on this scaling.
  • task_feature_weights = Table I weights, e.g., [0.7,-1.3,0.7,0,0,0,0]
    Weights define the tasks; the authors selected weights via ablation 'revealed the weights listed yielded the best results' (Supplementary II-A1), so the transfer results are conditioned on this choice.
  • learning_rates = alpha_sf=0.1, alpha_minmax=0.3
    Selected by ablation per algorithm (Sec. V-D); the comparison between algorithms uses different tuned learning rates, which can favor the proposed method.
  • discount_factor_gamma = 0.9
    Standard choice and best in tests (Supplementary II-A); all transfer bounds and learning updates assume this value.
assumptions (5)
  • domain assumption r_w(s,a,b) = phi(s,a,b)^T w for every task, with known weights w (Eq. 1, Eq. 10)
    The whole SF decomposition and GGPI bound depend on rewards being exactly linear in a shared feature vector; if this fails the transfer guarantee has no meaning.
  • domain assumption All source and target tasks share the same state space, action spaces, transition kernel P, and discount factor, differing only in reward weights (Sec. IV-B)
    Successor features can only be reused without re-estimating dynamics if dynamics are unchanged across tasks; the paper does not test or relax this.
  • ad hoc to paper The approximation error bound epsilon survives taking min and max over actions and source policies (used in proof of Theorem 1)
    The supplementary proof asserts 'This property should remain at a minimum as well as a maximum' without proof; this is where the sign error occurs and is load-bearing for the claimed bound.
  • standard math There exist optimal stationary deterministic policies for both agents in alternating zero-sum Markov games (Sec. III-B, citing [22])
    Standard minimax result for finite turn-based games; acceptable background.
  • standard math Bellman operators are monotone and contractive with modulus gamma (Supplementary, 'Monotonicity' and 'Contraction' paragraphs)
    Standard background used in the convergence arguments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Successor Features for Transfer in Alternating Markov Games." pith.science (2026). https://pith.science/paper/VGY225TQ

@misc{pith2026250722278,
  author       = {Pith},
  title        = {Pith review of: Successor Features for Transfer in Alternating Markov Games},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VGY225TQ}},
  note         = {Machine review of arXiv:2507.22278}
}
read the original abstract

This paper explores successor features for knowledge transfer in zero-sum, complete-information, and turn-based games. Prior research in single-agent systems has shown that successor features can provide a ``jump start" for agents when facing new tasks with varying reward structures. However, knowledge transfer in games typically relies on value and equilibrium transfers, which heavily depends on the similarity between tasks. This reliance can lead to failures when the tasks differ significantly. To address this issue, this paper presents an application of successor features to games and presents a novel algorithm called Game Generalized Policy Improvement (GGPI), designed to address Markov games in multi-agent reinforcement learning. The proposed algorithm enables the transfer of learning values and policies across games. An upper bound of the errors for transfer is derived as a function the similarity of the task. Through experiments with a turn-based pursuer-evader game, we demonstrate that the GGPI algorithm can generate high-reward interactions and one-shot policy transfer. When further tested in a wider set of initial conditions, the GGPI algorithm achieves higher success rates with improved path efficiency compared to those of the baseline algorithms.

Figures

Figures reproduced from arXiv: 2507.22278 by the authors.

Figure 1
Figure 1. Cumulative return per task in the Pursuer-Evader game. SFminmax [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Trajectories taken by the evader (robber) and pursuer (police) for three tasks determined by the goal position (exits). The trajectories for case [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. (a) Possible initial position and goals for testing transfer with significant reward change (b) Path taken by SFminmax agents where pursuer agent [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 2
Figure 2. Figure 2: Asynchronous q table update with one step look out horizon for both [PITH_FULL_IMAGE:figures/full_fig_p011_2.png]
Figure 3
Figure 3. Figure 3: Schematic representation of how the ψ is taken from the table and GGPI is implemented. Given a state s ∈ S, the first step is to compute,for each a∗b ∈ A×B, the values of the n SFs: ψ πi (s, a, b). We have sf-table for each task for both pursuer and evader. Based on th…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 31 canonical work pages

  1. [1]

    M. L. Littman and C. Szepesv \'a ri, ``A generalized reinforcement-learning model: Convergence and applications,'' in ICML, vol. 96, 1996, pp. 310--318

    work page 1996

  2. [2]

    Barreto, W

    A. Barreto, W. Dabney, R. Munos, J. J. Hunt, T. Schaul, H. P. van Hasselt, and D. Silver, ``Successor features for transfer in reinforcement learning,'' Advances in neural information processing systems, vol. 30, 2017

    work page 2017

  3. [3]

    0y^9gy g)J H

    11em plus .33em minus .07em @technote 4000 4000 100 4000 4000 500 `\.=1000 = #1 #1 #1 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEauthorblockAstyle \@IEEEauthordefaulttextstyle \@IEEEauthorblockconfadjspace -0.25em \@IEEEauthorblockNtopspace 0.0ex \@IEEEauthorblockAtopspace 0.0ex \@IEEEauthorblockNinterlinespace 2.6ex \@IEEEauthorblockAinte...

    arXiv 2011

  4. [4]

    Canese, G

    L. Canese, G. C. Cardarilli, L. Di Nunzio, R. Fazzolari, D. Giardino, M. Re, and S. Span \`o , ``Multi-agent reinforcement learning: A review of challenges and applications,'' Applied Sciences, vol. 11, no. 11, p. 4948, 2021

    work page 2021

  5. [5]

    Baker, I

    B. Baker, I. Kanitscheider, T. Markov, Y. Wu, G. Powell, B. McGrew, and I. Mordatch, ``Emergent tool use from multi-agent autocurricula,'' arXiv preprint arXiv:1909.07528, 2019

    arXiv 1909

  6. [6]

    C. Ma, A. Li, Y. Du, H. Dong, and Y. Yang, ``Efficient and scalable reinforcement learning for large-scale network control,'' Nature Machine Intelligence, vol. 6, no. 9, pp. 1006--1020, 2024

    work page 2024

  7. [7]

    F. L. Da Silva and A. H. R. Costa, ``A survey on transfer learning for multiagent reinforcement learning systems,'' Journal of Artificial Intelligence Research, vol. 64, pp. 645--703, 2019

    work page 2019

  8. [8]

    Amatya, M

    S. Amatya, M. Ghimire, Y. Ren, Z. Xu, and W. Zhang, ``When shall i estimate your intent? costs and benefits of intent inference in multi-agent interactions,'' in 2022 American Control Conference (ACC). 1em plus 0.5em minus 0.4em IEEE, 2022, pp. 586--592

    work page 2022

Show all 34 references
  1. [9]

    Sinapov, S

    J. Sinapov, S. Narvekar, M. Leonetti, and P. Stone, ``Learning inter-task transferability in the absence of target task samples,'' in Proceedings of the 2015 international conference on autonomous agents and multiagent systems, 2015, pp. 725--733

    2015

  2. [10]

    Braylan and R

    A. Braylan and R. Miikkulainen, ``Object-model transfer in the general video game domain,'' in Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, vol. 12, no. 1, 2016, pp. 136--142

    2016

  3. [11]

    Barrett and P

    S. Barrett and P. Stone, ``Cooperating with unknown teammates in complex domains: A robot soccer case study of ad hoc teamwork,'' in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 29, no. 1, 2015

    2015

  4. [12]

    M. L. Littman et al., ``Friend-or-foe q-learning in general-sum games,'' in Proceedings of the International Conference on Machine Learning, vol. 1, 2001, pp. 322--328

    2001

  5. [13]

    W. Liu, L. Dong, D. Niu, and C. Sun, ``Efficient exploration for multi-agent reinforcement learning via transferable successor features,'' IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 9, pp. 1673--1686, 2022

    2022

  6. [14]

    Hinrichs and K

    T. Hinrichs and K. D. Forbus, ``Transfer learning through analogy in games,'' Ai Magazine, vol. 32, no. 1, pp. 70--70, 2011

    2011

  7. [15]

    Sharma, M

    M. Sharma, M. P. Holmes, J. C. Santamar \' a, A. Irani, C. L. Isbell Jr, and A. Ram, ``Transfer learning in real-time strategy games using hybrid cbr/rl.'' in IJCAI, vol. 7, 2007, pp. 1041--1046

    2007

  8. [16]

    Cheng, Z

    C. Cheng, Z. Zhu, B. Xin, and C. Chen, ``A multi-agent reinforcement learning algorithm based on stackelberg game,'' in 2017 6th Data Driven Control and Learning Systems (DDCLS). 1em plus 0.5em minus 0.4em IEEE, 2017, pp. 727--732

    2017

  9. [17]

    Y. Liu, Y. Hu, Y. Gao, Y. Chen, and C. Fan, ``Value function transfer for deep multi-agent reinforcement learning based on n-step returns.'' in IJCAI. 1em plus 0.5em minus 0.4em Macao, 2019, pp. 457--463

    2019

  10. [18]

    Y. Chen, L. Zhang, T. Merry, S. Amatya, W. Zhang, and Y. Ren, ``When shall i be empathetic? the utility of empathetic parameter estimation in multi-agent interactions,'' in 2021 IEEE International Conference on Robotics and Automation (ICRA). 1em plus 0.5em minus 0.4em IEEE, 2...

    2021

  11. [19]

    Y. Hu, Y. Gao, and B. An, ``Accelerating multiagent reinforcement learning by equilibrium transfer,'' IEEE transactions on cybernetics, vol. 45, no. 7, pp. 1289--1302, 2014

    2014

  12. [20]

    Barreto, S

    A. Barreto, S. Hou, D. Borsa, D. Silver, and D. Precup, ``Fast reinforcement learning with generalized policy updates,'' Proceedings of the National Academy of Sciences, vol. 117, no. 48, pp. 30\,079--30\,087, 2020

    2020

  13. [21]

    Gimelfarb, A

    M. Gimelfarb, A. Barreto, S. Sanner, and C.-G. Lee, ``Risk-aware transfer in reinforcement learning using successor features,'' Advances in Neural Information Processing Systems, vol. 34, pp. 17\,298--17\,310, 2021

    2021

  14. [22]

    Abdolshah, H

    M. Abdolshah, H. Le, T. K. George, S. Gupta, S. Rana, and S. Venkatesh, ``A new representation of successor features for transfer across dissimilar environments,'' in International Conference on Machine Learning. 1em plus 0.5em minus 0.4em PMLR, 2021, pp. 1--9

    2021

  15. [23]

    Szepesv \'a ri and M

    C. Szepesv \'a ri and M. L. Littman, ``Generalized markov decision processes: Dynamic-programming and reinforcement-learning algorithms,'' in Proceedings of International Conference of Machine Learning, vol. 96, 1996

    1996

  16. [24]

    M. L. Littman, ``Markov games as a framework for multi-agent reinforcement learning,'' in Machine learning proceedings 1994. 1em plus 0.5em minus 0.4em Elsevier, 1994, pp. 157--163

    1994

  17. [25]

    Bertsekas, Dynamic programming and optimal control: Volume I

    D. Bertsekas, Dynamic programming and optimal control: Volume I. 1em plus 0.5em minus 0.4em Athena scientific, 2012, vol. 4

    2012

  18. [26]

    Zhu and D

    Y. Zhu and D. Zhao, ``Online minimax q network learning for two-player zero-sum markov games,'' IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 3, pp. 1228--1241, 2020

    2020

  19. [27]

    R. A. Howard, ``Dynamic programming and markov processes.'' 1960

    1960

  20. [28]

    Bertsekas and J

    D. Bertsekas and J. N. Tsitsiklis, Neuro-dynamic programming. 1em plus 0.5em minus 0.4em Athena Scientific, 1996

    1996

  21. [29]

    Lee, ``Finite-time analysis of minimax q-learning for two-player zero-sum markov games: Switching system approach,'' arXiv preprint arXiv:2306.05700, 2023

    D. Lee, ``Finite-time analysis of minimax q-learning for two-player zero-sum markov games: Switching system approach,'' arXiv preprint arXiv:2306.05700, 2023

    2023 arXiv

  22. [30]

    Fern \'a ndez, J

    F. Fern \'a ndez, J. Garc \' a, and M. Veloso, ``Probabilistic policy reuse for inter-task transfer learning,'' Robotics and Autonomous Systems, vol. 58, no. 7, pp. 866--871, 2010

    2010

  23. [31]

    Anderson, A

    P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva et al., ``On evaluation of embodied navigation agents,'' arXiv preprint arXiv:1807.06757, 2018

    2018 arXiv

  24. [32]

    Lehnert, S

    L. Lehnert, S. Tellex, and M. L. Littman, ``Advantages and limitations of using successor features for transfer in reinforcement learning,'' arXiv preprint arXiv:1708.00102, 2017

    2017 arXiv

  25. [33]

    R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch, ``Multi-agent actor-critic for mixed cooperative-competitive environments,'' Advances in neural information processing systems, vol. 30, 2017

    2017

  26. [34]

    T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, ``Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,'' in Conference on robot learning. 1em plus 0.5em minus 0.4em PMLR, 2020, pp. 1094--1100

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.