{"id":"95f4596a-7c6d-4f0c-8389-f17636da595a","arxiv_id":"2501.12199","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"ERID uses experience-replay reward averages with BNN, Smith, and Smith-replicator revision protocols, claiming convergence of its policy trajectories to those dynamics.","lead":"A new multi-agent reinforcement learning algorithm, ERID, uses a replay buffer of recent rewards to make learning trajectories follow BNN, Smith, and Smith-replicator evolutionary dynamics. The paper claims this extends formal convergence guarantees beyond replicator dynamics, but two of the three main theorems are stated without proof.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1 fails at the simplex boundary: Eq (2) forces \\bar r_i=0 for unsampled actions, so ERID cannot reproduce BNN dynamics from zero-probability initial policies.","rationale":"The reader's weakest assumption identified the invalid expectation interchange in Eq (7) and the omitted proofs of Theorems 2 and 3. I agree that these are serious gaps. However, the more decisive issue is that Theorem 1 is false as stated because of the boundary rule in Eq (2). The proof only covers \\pi_i>0, but the theorem does not. The counterexample with payoff matrix [[2,2],[0,0]] shows that ERID cannot replicate BNN dynamics from a boundary initial condition. This is not a stylistic objection: the algorithm's definition of \\bar r_i=0 for unsampled actions is scale-dependent and breaks the equivalence exactly where BNN's positive-part term matters. Hence the central claim of convergence to innovative dynamics is not supported. I set verdict_should_be to REJECT, consistent with the reader's verdict. The concrete test would settle whether the theorem needs an interior-initialization assumption; if a modified theorem with that assumption is intended, the paper should state it, and Theorems 2 and 3 still need proofs.","tokens_in":15026,"tokens_out":11131,"duration_ms":113161,"concrete_test":"Simulate ERID and BNN in the two-player game with payoff matrix A=B=[[2,2],[0,0]] for both players, starting from \\pi=(0,1) for both players, with e.g. \\alpha=1e-4 and K=100 (so \\alpha K=0.01). Record \\pi_1(t) over time. BNN will increase \\pi_1 toward 1, while ERID will keep \\pi_1=0 because action 1 is never sampled and \\bar r_1=0. If the trajectories diverge, Theorem 1 needs an explicit interior-initialization assumption or a modified rule for unsampled actions.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that ERID's policy trajectory converges to BNN, Smith, and Smith-replicator dynamics. Theorem 1, the only proof provided, fails for boundary states. In Equation (2), \\bar r_i is set to 0 whenever action i has no samples in the buffer. If at some time \\pi_i(t)=0, action i is never selected, so I_i is empty and \\bar r_i=0 forever. The update (6) then gives \\pi_i(t+1)=\\pi_i(t)+\\alpha([0-\\bar r]_+ - 0), which is identically zero for nonnegative rewards. BNN dynamics (4), by contrast, has \\dot x_i = [\\text{payoff}_i - \\text{average payoff}]_+, which can be strictly positive at x_i=0, so x_i leaves the boundary. Concretely, in the two-player game with payoff matrix [[2,2],[0,0]] for both players and initial \\pi=(0,1), BNN converges to (1,0), while ERID stays at (0,1) for any \\alpha,K satisfying \\alpha K \\to 0. The theorem statement does not restrict initial policies to the interior, and the proof explicitly assumes \\pi_i^{[1]}>0 before Equation (7). Additionally, Equation (7) uses the invalid interchange E((1/|I_i|)\\sum b_j) = \\sum E(b_j)/E(|I_i|), a further gap even in the interior. Theorems 2 and 3 have no proofs. The central claim is therefore not established, and Theorem 1 is false as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Experience-replay Innovative Dynamics (ERID), a stateless multi-agent reinforcement learning algorithm that uses a replay buffer and a tunable revision-protocol factor to approximate three evolutionary dynamics: Brown-von Neumann-Nash (BNN), Smith, and Smith-replicator-based pairwise dynamics. The central theoretical claim is that, under the joint limit αK→0 and K→∞, the policy trajectories of ERID converge to the trajectories of these dynamics (Theorems 1–3). The paper also reports experiments in matching pennies, biased Rock-Paper-Scissors, and a nonstationary Rock-Paper-Scissors game, comparing ERID with cross learning and claiming that ERID adapts better to environmental changes.","tokens_in":1653,"tokens_out":1604,"duration_ms":66808,"significance":"If the theoretical claims were established, ERID would provide a general mechanism for transferring the convergence guarantees of innovative dynamics to MARL, going beyond the replicator dynamics that dominate the current EGT-MARL literature. The algorithmic idea of using experience replay to implement non-linear revision protocols is original, and the empirical comparisons, especially the nonstationary RPS experiment, are suggestive and align with the qualitative story. The paper also makes a useful conceptual point about time-averaged replicator dynamics being slow to adapt. However, the central theorems are not proven, and Theorem 1 is false as stated; these issues currently prevent the paper from delivering on its main promise.","major_comments":[{"comment":"Theorem 1 is false as stated because it does not restrict initial policies to the interior of the simplex. If an action i has probability zero at some time, then by Eq. (2) the set I_i is empty and r̄_i is set to 0; since action i is never selected, I_i remains empty forever and r̄_i stays 0. In the two-player normal-form game with payoff matrix [[2,2],[0,0]] for both players and initial policy (0,1), the BNN dynamics (4) has a positive derivative for the first action at the boundary and converges to (1,0), whereas ERID update (6) keeps the first action at probability 0 for all time because all positive-part terms are zero. The proof explicitly assumes the initial probability of action i is positive immediately before Eq. (7), so the proof does not cover the statement as written.","section":"Theorem 1, Section 3.3"},{"comment":"The first equality in Eq. (7) is mathematically invalid. The expected average reward of action i is the expectation of the ratio of the sum of rewards in I_i to the size of I_i, and this is not equal to the ratio of the expectation of the sum to the expectation of the size, because |I_i| is a random variable depending on the sequence of sampled actions. The proof therefore does not justify the subsequent replacement of buffer averages by current expected payoffs, even when all probabilities are positive. A correct argument would need a proper stochastic-approximation or two-time-scale analysis, which is not supplied.","section":"Equation (7), Section 3.3"},{"comment":"The proof asserts that under αK→0 and K→∞ the policies remain effectively constant over the buffer window and then invokes the law of large numbers, but this stochastic-averaging step is not made rigorous. The bound on the policy difference over j steps only controls the drift for a fixed j; summing up to K requires a uniform argument with condition αK→0, and the interaction between the two limits, K growing for averaging and αK shrinking for policy constancy, is never formalized. In addition, the passage from the discrete update to the ODE (9) via the parameter θ is only asserted; no error bounds or compactness argument is given.","section":"Limit αK→0 and K→∞, Section 3.3"},{"comment":"Theorems 2 and 3 are stated without proofs, with the remark that they are similar to Theorem 1. Since the proof of Theorem 1 is invalid, these results are unsupported. Moreover, the boundary failure identified in Theorem 1 applies equally to the Smith update (11) and the Smith-replicator pairwise update (13), because r̄_i is zero for any never-sampled action, preventing the policy from leaving the boundary. The manuscript needs either complete proofs or a clear restriction to interior initial conditions and non-degenerate exploration for all three theorems.","section":"Theorems 2 and 3, Sections 3.4 and 3.5"}],"minor_comments":[{"comment":"The heading contains a typo: 'main contibution' should be 'main contribution'.","section":"Section 3 heading"},{"comment":"Algorithm 1 lists the input learning rate as theta, while the text and update rules consistently use alpha; please unify the notation.","section":"Algorithm 1"},{"comment":"The text refers to 'Hennis et al.', but the cited work is by Hennes et al.; please correct the name.","section":"Section 1.1 and reference [19]"},{"comment":"The Smith-replicator-based pairwise dynamics is attributed to reference [9], which is a paper on urban drainage systems; this reference does not appear to define the dynamics in Eq. (12). Please cite the correct source for these dynamics.","section":"Section 3.5, reference [9]"},{"comment":"The superscript notation in Eq. (7) is confusing: r̄ with superscript [1] is used for the player-1 empirical average, but the expectation E is not formally defined with respect to the buffer randomness conditional on the current policy; please clarify the probability space and conditioning.","section":"Equation (7) notation"},{"comment":"The figures compare simulated dynamics and ERID trajectories only visually; adding a quantitative distance metric, such as average Euclidean distance over time, would make the claimed agreement more precise.","section":"Figures 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The core algorithm and experiments are interesting, but the theoretical contribution is not in a publishable state. The false boundary behavior of Theorem 1 is a serious correctness issue; the invalid expectation interchange in Eq. (7) is the central gap. I believe these can be repaired by restricting to interior initial policies, adding an exploration mechanism, and rewriting the proof with a rigorous stochastic approximation argument, so I am not recommending rejection. However, the authors should be told that the current proof is not merely missing details; the theorem statement itself is false as written. Also, the citation for the Smith-replicator dynamics should be checked, and Theorems 2 and 3 need actual proofs or explicit dependence on a corrected Theorem 1."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is worth a look: use experience replay to estimate the payoffs that feed the BNN and Smith revision protocols, giving a MARL update that tracks nonstationary Nash equilibria better than time-averaged replicator methods. That combination is genuinely new as far as I know, and the paper is clearly written. The trajectory plots in Figures 2-4 support the intuition, though there is no code and no error bars.\n\nThe trouble is that Theorem 1, the only proof given, is false as stated. The stress-test note identifies a real boundary failure: Eq (2) defines \\bar r_i = 0 whenever action i has no buffer samples. If \\pi_i(0)=0, action i is never sampled, \\bar r_i stays 0, and the BNN update (6) leaves \\pi_i unchanged. Meanwhile the BNN ODE can move \\pi_i away from 0. Concrete counterexample: both players have payoff matrix [[2,2],[0,0]], initial \\pi=(0,1). BNN goes to (1,0); ERID stays at (0,1) for any \\alpha,K. That kills the theorem's unrestricted statement.\n\nEven in the interior, the proof's Eq (7) uses E(X/Y)=E(X)/E(Y), which is not valid. The \\alpha K \\to 0 and K \\to \\infty double limit is never made rigorous; a proper stochastic approximation argument would need to fix the order of the limits or show uniform convergence. Theorems 2 and 3 are stated without proof, which would be acceptable if Theorem 1 were solid, but here they're resting on the same broken reasoning.\n\nSo: novel framing, good motivation, but the central claims are not established. I would absolutely send this to a serious referee—the idea deserves one—but I wouldn't accept it in current form. The authors need to either add an exploration term that keeps all actions supported and prove a proper mean-field limit, or restrict the theorem and discuss the boundary behavior explicitly. As it stands, I'd regard it as a promising working paper rather than a finished result.","headline":"Novel and readable, but Theorem 1 is false as stated—the boundary counterexample holds—and the proof's key expectation step is invalid.","tokens_in":15889,"tokens_out":4064,"would_cite":true,"duration_ms":41583,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ERID is a stateless multi-agent learning algorithm whose experience-replay updates, with a tunable revision protocol, prove convergence to the trajectories of BNN, Smith, and Smith-replicator pairwise dynamics in the limit of vanishing…","keywords":["experience replay","multi-agent reinforcement learning","evolutionary game theory","innovative dynamics","BNN dynamics","Smith dynamics","revision protocols","Nash equilibrium"],"falsifier":"Run ERID with the BNN protocol on a zero-sum game over a grid of $K$ and $\\alpha$ values with $\\alpha K$ small and compare the empirical policy trajectory to the BNN ODE trajectory; if the distance does not shrink to zero, or if a direct calculation exhibits a case with random $|I_i|$ where $E(\\sum_{j\\in I_i} b_j / |I_i|) \\neq E(\\sum_{j\\in I_i} b_j)/E(|I_i|)$, the central convergence claim is refuted.","tokens_in":14812,"feed_emoji":"♻️","tokens_out":6793,"duration_ms":64389,"temperature":0.7,"pith_summary":"The paper introduces ERID, a stateless multi-agent reinforcement learning algorithm that uses an experience-replay buffer to average rewards and a tunable revision protocol to update policies. Its central claim is that when the buffer size grows and the learning rate shrinks together, the policy trajectories of ERID converge to the trajectories of BNN, Smith, and Smith-replicator-based pairwise dynamics, three 'innovative' evolutionary dynamics. This matters because replicator-dynamics-based MARL is unstable or slow in null-stable games like zero-sum games, whereas these innovative dynamics converge there; ERID inherits those guarantees. The paper verifies the match in matching pennies, biased rock-paper-scissors, and a nonstationary rock-paper-scissors setting.","feed_headline":"Experience replay extends MARL guarantees beyond replicator dynamics","feed_subtitle":"ERID tunes revision protocols so its learning paths match BNN, Smith, and pairwise dynamics—not just replicator.","key_machinery":"The load-bearing mechanism is experience replay interpreted as a reward-smoothing average. At each step the algorithm keeps a buffer of the last $K$ action–reward pairs, computes per-action averages $\\bar{r}_i$ and the global average $\\bar{r}$, then updates the policy with a discrete revision-protocol equation, $\\pi_i(t+1) \\leftarrow \\pi_i(t) + \\alpha(\\sum_j \\pi_j(t)\\eta_{ji} - \\pi_i(t)\\sum_j \\eta_{ij})$. Each target dynamics is selected by a protocol factor $\\eta_{ij}$ built from the reward averages: $[\\bar{r}_j - \\bar{r}]_+$ for BNN, $[\\bar{r}_j - \\bar{r}_i]_+$ for Smith, and the constrained version (12) for Smith-replicator pairwise dynamics. The proof works by showing that as $\\alpha K\\to 0$ and $K\\to\\infty$, the buffer averages converge to current expected payoffs, so the discrete update becomes the corresponding ODE.","core_discovery":"The central claim is that a single experience-replay learning rule can reproduce the trajectories of three 'innovative' evolutionary dynamics. For the BNN protocol factor $\\eta_{ij} = [\\bar{r}_j - \\bar{r}]_+$, the update in equation (6) becomes, in the limit $\\alpha K \\to 0$ with $K \\to \\infty$, the BNN differential equation (4)–(5); Theorems 2 and 3 state the same trajectory convergence for Smith dynamics and for Smith-replicator-based pairwise dynamics. The result is a bridge: MARL algorithms built on ERID inherit the convergence properties of these dynamics in stable and null-stable games, a guarantee previously available mainly for replicator dynamics. The empirical section shows the stochastic ERID trajectories tracking the deterministic dynamics in matching pennies and biased RPS, and tracking the shifting Nash equilibrium in a nonstationary RPS game.","pith_inferences":["The same replay-averaging construction could be applied to other revision protocols, such as projection or logit dynamics, yielding a whole family of MARL algorithms indexed by protocol factor; the paper does not explore these.","A finite-sample analysis of the buffer averages would reveal how $K$ and $\\alpha$ must be coupled in practice; the theorem only states the joint limit $\\alpha K\\to 0$, leaving the rate of convergence unspecified.","Because ERID is stateless and uses only aggregate rewards, it could be combined with function approximation by applying the update to a parameterized policy in expectation, though the paper does not address deep RL."],"forward_implications":["In zero-sum games, ERID with the BNN or Smith protocol factor can converge to the Nash equilibrium, because the underlying innovative dynamics do; this is exactly where replicator-dynamics learners only orbit or need time averaging.","In nonstationary environments, ERID's NashConv keeps dropping after payoff changes, while time-averaged replicator-based learners lag because their accumulated averages retain old equilibrium bias.","Swapping the revision protocol in the update rule selects a different evolutionary dynamics, so the same algorithm covers BNN, Smith, and constrained pairwise dynamics without changing the replay mechanism.","The matching-pennies and biased-RPS experiments show the stochastic ERID trajectories track the deterministic dynamics trajectories closely enough to inherit their qualitative convergence behavior."],"supporting_citations":[{"why":"Supplies the original cross-learning-to-replicator-dynamics convergence and the continuous-time limiting argument that ERID adapts.","marker":"[11]"},{"why":"Introduces Brown-von Neumann-Nash dynamics, one of the three target dynamics.","marker":"[12]"},{"why":"Defines strictly stable and null-stable games and establishes global convergence of BNN dynamics in null-stable games, the property ERID inherits.","marker":"[22]"},{"why":"Introduces Smith dynamics and its stability to equilibrium sets, the second target dynamics.","marker":"[50]"},{"why":"Defines the Smith-replicator-based pairwise dynamics used as the third target.","marker":"[9]"},{"why":"Supplies the nonstationary rock-paper-scissors benchmark and the NeuRD comparison that motivates the replay-based approach.","marker":"[19]"},{"why":"Original cross-learning model whose update rule is the starting point for the ERID update.","marker":"[15]"}],"fun_headline_variants":["Experience replay unlocks innovative dynamics for MARL","MARL algorithm reproduces BNN and Smith dynamics via replay","ERID: MARL with experience replay matches innovative dynamics","Replay-based MARL extends guarantees beyond replicator dynamics","Innovative dynamics meet experience replay in MARL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof assumes that the average reward observed for an action in the replay buffer equals the ratio of expected sums and that policies stay essentially constant over the whole buffer window, so buffer averages behave like current expected payoffs; if those assumptions fail, the claimed trajectory match is not established.","fun_headline_variants_meta":{"raw":{"variants":["Experience replay unlocks innovative dynamics for MARL","MARL algorithm reproduces BNN and Smith dynamics via replay","ERID: MARL with experience replay matches innovative dynamics","Replay-based MARL extends guarantees beyond replicator dynamics","Innovative dynamics meet experience replay in MARL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1387,"prompt_tokens":924,"completion_tokens":463,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":396}},"tokens_in":540,"tokens_out":463,"duration_ms":4515,"temperature":1.0,"reasoning_tokens":396,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:24:25.164744+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ERID with the BNN protocol on a zero-sum game over a grid of $K$ and $\\alpha$ values with $\\alpha K$ small and compare the empirical policy trajectory to the BNN ODE trajectory; if the distance does not shrink to zero, or if a direct calculation exhibits a case with random $|I_i|$ where $E(\\sum_{j\\in I_i} b_j / |I_i|) \\neq E(\\sum_{j\\in I_i} b_j)/E(|I_i|)$, the central convergence claim is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the original cross-learning-to-replicator-dynamics convergence and the continuous-time limiting argument that ERID adapts."},{"cited_title":"1950.Solutions of games by differential equations","cited_arxiv_id":null,"evidence_quote":"Introduces Brown-von Neumann-Nash dynamics, one of the three target dynamics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines strictly stable and null-stable games and establishes global convergence of BNN dynamics in null-stable games, the property ERID inherits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Smith dynamics and its stability to equilibrium sets, the second target dynamics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Smith-replicator-based pairwise dynamics used as the third target."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the nonstationary rock-paper-scissors benchmark and the NeuRD comparison that motivates the replay-based approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Original cross-learning model whose update rule is the starting point for the ERID update."}],"review_version":1}