{"id":"56d84e50-dadb-43fc-b50f-72348786e85c","arxiv_id":"2507.22278","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A proposed transfer algorithm, GGPI, applies successor features to alternating zero-sum Markov games, but its central theorem is not proven as written.","lead":"This paper applies successor features and generalized policy improvement to two-player, zero-sum, turn-based Markov games, introducing the GGPI transfer algorithm with a claimed performance bound. The main theorem is not proven as written, so the central theoretical claim is unsupported despite plausible grid-world experiments.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1 is false as stated: min_i Q^{Π_i} is not closed under the game Bellman operator, so GGPI can be far worse than every source policy even with exact Q values.","rationale":"In good faith, the paper's intended contribution is a GGPI transfer guarantee for alternating Markov games, and the successor-feature pipeline depends on that guarantee. The reader's REJECT verdict is supported, but the strongest reason is not merely a sign error in the supplementary proof; the claimed inequality is genuinely false. The counterexample above uses exact value functions and a deterministic transition, so no approximation error, function approximation, or reward-feature assumption is involved. The empirical observations in the 5x5 grid may be valid, but they cannot establish one-shot transfer without a correct theoretical bound. The supplementary's false identity |min_i Q_i - min_i tilde_Q_i| = min_i |Q_i - tilde_Q_i|, and the later replacement of Qmin by Q^{Π_i}, are symptoms of the same structural problem: min_i of several policy values is not a value function of the game. I therefore see no basis to change the reader's REJECT verdict.","tokens_in":16338,"tokens_out":13888,"duration_ms":174677,"concrete_test":"Run the analytic counterexample above: with epsilon = 0 and exact Q functions, evaluate Eq. 12 at the preceding state s0 for all (a,b). Every evaluation gives LHS = -90 and RHS = 9, refuting Theorem 1. As a robustness check, also run a random search over small tabular games with |S| <= 3, |A| = |B| = 2, two to three source policies, and gamma in {0.5, 0.9}, checking whether any setting satisfies Eq. 12; the constructed counterexample should already appear as a violation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is Theorem 1 (Eq. 12). It fails even with exact value functions (epsilon = 0). The supplementary proof defines Qmin = min_i Q^{Π_i} and applies the max-min Bellman operator to it as if it were the value of a policy, but Qmin is not closed under that operator: min_i does not commute with the expectation and best-response operations in Eq. 6. Concretely, take A = {a1, a2}, B = {b1, b2}, terminal rewards r(a1,b1) = 10, r(a1,b2) = -100, r(a2,b1) = -100, r(a2,b2) = 10. Let source policy 1 choose (a1, b1) and source policy 2 choose (a2, b2) at the terminal state. In a preceding state s0 with reward 0, deterministic transition to the terminal state, and gamma = 0.9, both source policies have value 9 at s0, so min_i Q^{Π_i}(s0,a,b) = 9. The GGPI action at the terminal state is any action a with min_b r(a,b) = -100; the opponent then chooses the minimizing b, so Q^Π(s0,a,b) = 0.9 * (-100) = -90. Thus -90 >= 9 is false, contradicting Eq. 12. The flaw is structural: argmax_a min_b min_i chooses actions by worst case across all sources at each state, while a source policy's value is evaluated using its own action, and the minimum over sources can switch across states.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an extension of successor features (SFs) to alternating, turn-based, zero-sum Markov games. The authors define a Game Generalized Policy Improvement (GGPI) operator that, from n source policies, selects actions by argmax over the ego action a of min over opponent action b and min over source policies i of the approximate Q-function. The central theoretical claim is Theorem 1 (Eq. 12): the GGPI policy's Q-function is pointwise at least min_i Q^{Π_i}(s,a,b) - 2ε/(1-γ). The paper also states Lemma 1, bounding reward-induced differences of value functions, and Proposition 1, combining these bounds. Experiments on a grid pursuer-evader game compare SFminmax with MinMax Q-learning, PRQL, and an exploration-reset variant, reporting higher cumulative return, one-shot policy transfer, and better success-weighted path length in some settings. The supplementary material contains proofs of the theorem and lemma plus implementation details.","tokens_in":16715,"tokens_out":5950,"duration_ms":68168,"significance":"If Theorem 1 were correct, the paper would provide a principled transfer guarantee for adversarial multi-agent settings, extending Barreto et al.'s generalized policy improvement to alternating Markov games. That would be a useful contribution, and the paper also provides a concrete algorithmic instantiation and a nontrivial experimental setup. However, the central theoretical result is false as stated, even for exact value functions. The counterexample in the report shows that the minimum over source policies is not closed under the game Bellman operator, so the GGPI policy can be substantially worse than every source policy. Since the one-shot transfer claim rests on Eq. (12), the main contribution is not supported. The experiments may still show useful heuristics, but they do not compensate for an incorrect load-bearing theorem.","major_comments":[{"comment":"Theorem 1 is false as stated, even in the exact-value case ε=0. Let S={s0,s1}, with any action pair at s0 transitioning deterministically to s1 with reward 0, and no transition out of s1. At s1 the terminal rewards are r(a1,b1)=10, r(a1,b2)=-100, r(a2,b1)=-100, r(a2,b2)=10. Let source policy Π1 choose (a1,b1) at s1 and source policy Π2 choose (a2,b2) at s1. With γ=0.9, for every (a,b) at s0, Q^{Π_i}(s0,a,b)=0.9*r(a_i,b_i)=9, so min_i Q^{Π_i}(s0,a,b)=9. At s1, min_i Q^{Π_i}(s1,a,b)=r(a,b); therefore the GGPI action at s1 is any a with min_b r(a,b)=-100, and the opponent then chooses the corresponding b. The GGPI policy therefore has value 0.9*(-100)=-90 at s0, which is strictly less than min_i Q^{Π_i}(s0,a,b)=9. This contradicts Eq. (12) with ε=0. The structural reason is that min_i does not commute with the max-min Bellman operator: the source attaining the minimum can switch from state to state, so min_i Q^{Π_i} is not the value function of any stationary policy.","section":"Section IV-A, Eq. (12)"},{"comment":"The proof of Theorem 1 contains an invalid inequality. It claims |min_i Q^{Π_i}(s,a,b) - min_i \\tilde Q^{Π_i}(s,a,b)| = min_i |Q^{Π_i}(s,a,b) - \\tilde Q^{Π_i}(s,a,b)|, but the left-hand side is not equal to the minimum; the correct bound is ≤ max_i |Q^{Π_i} - \\tilde Q^{Π_i}|. More fundamentally, the proof applies the policy Bellman operator T^Π to \\tilde Q_min and then compares the result to Q^{Π_i} as if \\tilde Q_min were the value function of the GGPI policy. Since min_i Q^{Π_i} is not closed under the game Bellman operator, the subsequent 'property of Bellman operator' step reverses the inequality direction and does not establish the claimed lower bound.","section":"Supplementary Material, Section I-A"},{"comment":"Lemma 1 as stated in the main text claims |Q^Π_i(s,a,b)-Q^Π_j(s,a,b)| ≤ ¶_{ij}/(1-γ) for an arbitrary policy Π, but the proof in the supplementary material derives a bound of 2¶_{ij}/(1-γ) and does so for Q^i_i versus Q^j_i, i.e., for the value of the j-th optimal policy evaluated in task i, not for an arbitrary policy. The notation Q^j_i is introduced as Q^{Π*_j}_i, which is not the quantity appearing in Lemma 1. This mismatch means Lemma 1, as used in Proposition 1, is not actually proved.","section":"Supplementary Material, Lemma 1"}],"minor_comments":[{"comment":"Equation (6) writes Q^*(s,a',b') inside the max-min operator, but the Bellman recursion should use the next state s', i.e., Q^*(s',a',b').","section":"Section III-B, Eq. (6)"},{"comment":"The successor feature definition in Eq. (2) uses features of the next state-action pair, ϕ(s_{t+1},a_{t+1}), and a discount μ^t, while Eq. (3) equates ψ^π(s,a)^T w to the expected discounted reward from time 0. The indexing should be aligned with the reward function in Eq. (1), otherwise the equality in Eq. (3) is off by one time step and by the terminal reward convention.","section":"Section III-A, Eqs. (2)-(3)"},{"comment":"The notation argmax_b argmin_a min_i Ψ^{Π_i}(s,a,b)^T w_i is confusing: it is not the same as the GGPI rule in Theorem 1, and it suggests a different order of the max and min operations. The algorithm should use the same operator as Eq. (12), and Ψ should be explicitly distinguished from Q.","section":"Algorithm 1, lines 7 and 18"},{"comment":"The experimental tables report point estimates without standard errors or repeated seeds; several entries are marked 'NA' or contain inconsistent decimal notation, which makes the value-discrepancy and SPL comparisons difficult to interpret.","section":"Section V, Tables II and Fig. 1"},{"comment":"The limitation paragraph states that the method 'relies on the Q-values of the training and test tasks being within comparable bounds' and 'becomes less effective when the nature or weighting of the tasks changes significantly.' This is directly related to the failure mode of the main theorem and deserves a concrete test; the manuscript does not measure robustness to that assumption.","section":"Section VI"}],"recommendation":"reject","confidential_remarks":"The central theoretical guarantee is demonstrably false, so the paper's main claim cannot be accepted. The experiments may describe a useful heuristic, but the incorrect theorem and the inconsistent proof of Lemma 1 are load-bearing; correcting them would require reworking the core contribution rather than a local revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know first: the stress-test note is right in substance, even though its example has a miscomputed value. The corrected counterexample is cleaner and kills Theorem 1.\n\nWhat is actually new: applying successor features and generalized policy improvement to alternating zero-sum games is a natural extension, and the formulation of GGPI (maximin over the pointwise min of source Q-functions) is not in prior SF literature. The experiments compare against MinMax Q-learning, PRQL, and an epsilon-reset ablation in a small grid world, and the authors honestly list limitations (discrete actions, comparable reward scales). None of that is embarrassing.\n\nThe soft spot is load-bearing: Theorem 1 (Eq. 12) is false even with exact Q-values. Take a terminal state with rewards r(a1,b1)=10, r(a1,b2)=-100, r(a2,b1)=-100, r(a2,b2)=10. Let source policies 1 and 2 pick (a1,b1) and (a2,b2) there, and a preceding state s0 with zero reward, deterministic transition, gamma=0.9. The GGPI policy at any state chooses an action that maximizes the minimum over b of the pointwise min; at the terminal state both actions give min=-100, so the continuation value is -100. Thus Q^Π(s0,a1,b1) = -90. But min_i Q^{Π_i}(s0,a1,b1) = 0.9 * 10 = 9. The inequality Q^Π >= 9 fails. The proof in the supplementary breaks at the step where pointwise min is treated as a policy value; min over i does not commute with the max-min Bellman operator. The proof also has a sign error and Lemma 1 yields a factor of 2 that does not match the theorem's bound—minor compared to the theorem being wrong.\n\nDoes the empirical work rescue it? Not really. The experiments are one 5x5 grid world, no seeds/error bars, and the algorithm's motivation is the false theory. The transfer effects might be real for other reasons, but the central claim as written is not established.\n\nFor a referee: yes, this deserves peer-review time, because a false theorem is an important finding and the idea may be salvageable if the theory is corrected or removed. My recommendation would be to send it out, but expect a major revision or rejection on the theory.\n\nWho gets value: someone working on SF transfer in adversarial multi-agent settings, who can see the counterexample and decide whether a variant of GGPI can be fixed.","headline":"The central GGPI theorem is false as stated; the paper's main theoretical contribution collapses, though the SF-in-games idea and small experiments have some merit.","tokens_in":17213,"tokens_out":7323,"would_cite":false,"duration_ms":77720,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["91A15","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"Successor features, a single-agent transfer tool, extend to zero-sum alternating Markov games, where a new Game Generalized Policy Improvement selection is proven to be no worse than the worst source policy up to an approximation penalty.","keywords":["successor features","zero-sum Markov games","alternating games","generalized policy improvement","transfer learning","multi-agent reinforcement learning","pursuer-evader game","minmax Q-learning"],"falsifier":"Search small tabular alternating Markov games with exactly known Q-functions (so $\\epsilon=0$) for a counterexample to Theorem 1: if the GGPI policy's exact value is ever below $\\min_i Q^{\\Pi_i}(s,a,b)$ at some state-action pair, the bound as stated fails; equivalently, run the paper's own pursuer-evader transfer with a nonlinear target reward over the same features and check whether the initial transferred policy falls below the worst source policy's value by more than the bound allows.","tokens_in":16135,"feed_emoji":"🎮","tokens_out":10931,"duration_ms":116096,"temperature":0.7,"pith_summary":"This paper claims that successor features, a single-agent technique for reusing learned value functions across tasks with related rewards, can be extended to two-player zero-sum games where players act in turns. The new ingredient is Game Generalized Policy Improvement (GGPI): when several source policies are available, the ego player selects the action that maximizes the payoff under the worst opponent response and the worst of the source policies. The authors prove that this selection is guaranteed to score at least as well as the worst source policy on the new task, up to an approximation penalty that vanishes as the learned Q-functions become exact. Because the transfer step is just a dot product between a stored successor-feature table and the new task's reward weights, adapting to a new goal is nearly instantaneous. In a grid pursuer-evader game, the resulting one-shot policy transfer beats retraining from scratch and gives shorter, more successful paths.","feed_headline":"Successor features give one-shot transfer in adversarial games","feed_subtitle":"The transferred policy is never worse than its worst source task, up to a small approximation penalty.","key_machinery":"The central object is the successor-feature table $\\psi^{\\Pi_i}(s,a,b)$, a per-policy, per-state-action-opponent-action vector of discounted feature sums, together with the linear-reward assumption $r_w(s,a,b)=\\phi(s,a,b)^\\top w$ that turns cross-task evaluation into a dot product. On top of this, Game Generalized Policy Improvement (GGPI) selects $\\arg\\max_a\\min_b\\min_i \\tilde Q^{\\Pi_i}(s,a,b)$, i.e. the action that maximizes the minimum over the opponent's responses and over all stored source policies. The proof mechanism is the Bellman operator's monotonicity and contraction: applying the operator to $\\min_i \\tilde Q^{\\Pi_i}$ propagates the $\\epsilon$ approximation error at rate $\\mu$ per step, which after geometric summation yields the $2\\epsilon/(1-\\mu)$ penalty in Theorem 1. The companion task-similarity bound $\\eta_{ij}/(1-\\mu)$ is what lets the paper phrase transfer error as a function of reward similarity.","core_discovery":"The paper's central claim is that in a two-player, zero-sum, turn-based (alternating) Markov game, knowledge acquired on a set of source tasks can be transferred to a new task with the same state and action spaces and transition structure by storing, for each source policy $\\Pi_i$, the successor features $\\psi^{\\Pi_i}(s,a,b)$. Because each task's reward is assumed linear, $r_w(s,a,b)=\\phi(s,a,b)^\\top w$, evaluating any stored policy under a new reward is a dot product $\\psi^{\\Pi_i}(s,a,b)^\\top w_{n+1}$. The paper defines the GGPI policy $\\pi(s)\\in\\arg\\max_a\\min_b\\min_i \\tilde Q^{\\Pi_i}(s,a,b)$ and proves (Theorem 1) that if each stored Q-function is within $\\epsilon$ of its true value, the resulting policy satisfies $Q^{\\pi}(s,a,b)\\ge \\min_i Q^{\\Pi_i}(s,a,b) - 2\\epsilon/(1-\\mu)$, where $\\mu$ is the discount factor. A companion lemma bounds the value gap between two tasks by $\\eta_{ij}/(1-\\mu)$, where $\\eta_{ij}=\\max_{s,a,b}|r_i(s,a,b)-r_j(s,a,b)|$, so the transferred policy's suboptimality is at most $2\\eta_{ij}/(1-\\mu)+2\\epsilon/(1-\\mu)$. The paper takes its 5x5 grid pursuer-evader experiments as evidence that this one-shot transfer works in practice.","pith_inferences":["Beyond the paper: if the linear-reward assumption fails, one could still run GGPI with learned feature embeddings, but the constant $\\epsilon$ would need to absorb representation error; the paper does not analyze this case.","Beyond the paper: because GGPI selects against the worst source policy, it may be overly conservative when one source task is much harder than the rest; weighting source policies by their relevance to the target could improve average performance, an option not tested here.","Beyond the paper: the supplementary's failed preliminary run without a terminal-reward feature suggests the method's success depends on the feature set being expressive enough to encode the goal; a diagnostic experiment would vary feature completeness and measure how tightly the GGPI bound holds."],"forward_implications":["A library of source policies and their successor-feature tables is enough to act immediately in a new task: the ego computes $\\psi^{\\Pi_i}(s,a,b)^\\top w_{n+1}$ for each stored $i$ and plays the GGPI action, with no gradient updates.","The guarantee is worst-case over both the opponent and the source set, so the transferred policy is conservative by construction in adversarial settings.","The task-similarity bound $\\eta_{ij}/(1-\\mu)$ gives a quantitative criterion for deciding which source tasks are worth storing: tasks whose rewards are far apart contribute little.","In the tabular grid experiments, the authors find that value transfer and one-shot policy transfer occur when the goal changes, and that SFminmax attains higher success rates and path efficiency than MinMax Q-learning, the epsilon-reset ablation, and Probabilistic Policy Reuse."],"supporting_citations":[{"why":"Introduces successor features, generalized policy evaluation/improvement, and the single-agent transfer framework this paper extends to games.","marker":"[10]"},{"why":"Supplies the alternating Markov game model with ego action a then opponent action b used throughout.","marker":"[21]"},{"why":"Establishes that optimal stationary deterministic policies exist in Markov games, the foundation for the minmax policies in (7)-(8).","marker":"[22]"},{"why":"Provides the minimax Q-value iteration convergence result the algorithm relies on for stable learning.","marker":"[24]"},{"why":"Sources the approximation-error penalty form (epsilon/(1-gamma)) that Theorem 1 mirrors in the game setting.","marker":"[26]"},{"why":"The MinMax Q-learning baseline the paper compares against in transfer experiments.","marker":"[27]"},{"why":"The Probabilistic Policy Reuse baseline adapted for multi-agent transfer comparisons.","marker":"[28]"},{"why":"Earlier successor-feature study of initial/goal-condition transfer that motivates the epsilon-decay schedule in the paper's experiments.","marker":"[30]"}],"fun_headline_variants":["One-shot transfer in games via successor features","Game transfer guarantee: not worse than worst source","Successor features enable fast policy transfer in games","Pursuer-evader game: transfer with bounded suboptimality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every source and target task share the same state space, action spaces, transition probabilities, and discount factor, and that every reward is exactly linear in one common feature vector whose weights for the new task are known; the paper itself notes it additionally assumes a discrete action space, a conservative best-responding opponent, and comparable Q-value ranges across tasks.","fun_headline_variants_meta":{"raw":{"variants":["One-shot transfer in games via successor features","Game transfer guarantee: not worse than worst source","Successor features enable fast policy transfer in games","Pursuer-evader game: transfer with bounded suboptimality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1584,"prompt_tokens":1053,"completion_tokens":531,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":469}},"tokens_in":669,"tokens_out":531,"duration_ms":6613,"temperature":1.0,"reasoning_tokens":469,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:53:17.623198+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Search small tabular alternating Markov games with exactly known Q-functions (so $\\epsilon=0$) for a counterexample to Theorem 1: if the GGPI policy's exact value is ever below $\\min_i Q^{\\Pi_i}(s,a,b)$ at some state-action pair, the bound as stated fails; equivalently, run the paper's own pursuer-evader transfer with a nonlinear target reward over the same features and check whether the initial transferred policy falls below the worst source policy's value by more than the bound allows.","supporting_citations":[{"cited_title":"Braylan and R","cited_arxiv_id":null,"evidence_quote":"Introduces successor features, generalized policy evaluation/improvement, and the single-agent transfer framework this paper extends to games."},{"cited_title":"Gimelfarb, A","cited_arxiv_id":null,"evidence_quote":"Supplies the alternating Markov game model with ego action a then opponent action b used throughout."},{"cited_title":"Abdolshah, H","cited_arxiv_id":null,"evidence_quote":"Establishes that optimal stationary deterministic policies exist in Markov games, the foundation for the minmax policies in (7)-(8)."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the minimax Q-value iteration convergence result the algorithm relies on for stable learning."},{"cited_title":"Zhu and D","cited_arxiv_id":null,"evidence_quote":"Sources the approximation-error penalty form (epsilon/(1-gamma)) that Theorem 1 mirrors in the game setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The MinMax Q-learning baseline the paper compares against in transfer experiments."},{"cited_title":"Bertsekas and J","cited_arxiv_id":null,"evidence_quote":"The Probabilistic Policy Reuse baseline adapted for multi-agent transfer comparisons."},{"cited_title":"Fern \\'a ndez, J","cited_arxiv_id":null,"evidence_quote":"Earlier successor-feature study of initial/goal-condition transfer that motivates the epsilon-decay schedule in the paper's experiments."}],"review_version":1}