{"id":"1e4e547e-639f-4108-b345-65816404505f","arxiv_id":"2501.00160","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A frequency-aware mean-field model of incremental Boltzmann Q-learning in the Prisoner's Dilemma predicts that apparent stable cooperation is a long metastable transient and that high discount factors induce oscillations via a Neimark-Sacker bifurcation.","lead":"Independent Q-learning agents in the Prisoner's Dilemma can look like they have learned to cooperate for millions of steps, but this cooperation is a long-lived illusion: the learning dynamics eventually break into oscillations. A new mathematical model explains this behavior and shows that standard simplified models of multi-agent learning miss it.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The exactness of the one-step conditional expectation (14) does not justify treating its iterates as the stochastic process; without an α→0 or noise-rescaling argument, the bifurcation and multi-billion-step metastability claims about actual Q-learning rest on five visual runs.","rationale":"The paper's contribution is a deterministic approximation, and the weakest point is exactly the step from exact conditional expectation (14) to the claim that the iterated map describes the stochastic algorithm. I agree with the reader's weakest_assumption. The internal mathematics—fixed point derivation (15), Jacobian in Appendix D, and the Neimark-Sacker threshold—appears coherent; my concern is not about internal inconsistency but about external validity of the proxy. The self-admitted limitation in Section 3.2 and the Discussion that noise would be needed for exit times and oscillation periods is direct manuscript evidence. A concrete ensemble test at the paper's headline parameters would settle whether the long metastable plateau and the γ≈0.75 threshold survive stochasticity. If they do, the central claim is strengthened; if not, the title claim should be downgraded to a statement about the deterministic model. The verdict stays CONDITIONAL because the required check is well-posed and the current evidence is suggestive rather than conclusive.","tokens_in":19829,"tokens_out":4996,"duration_ms":53807,"concrete_test":"Run an ensemble of independent stochastic Q-learning simulations (e.g., 10^3 seeds) for the paper's own parameters: T=1, α=0.01, γ=0.8, Qbase=0, initial joint policy (0.9,0.7). Record the first time t where either agent's cooperation probability leaves a neighborhood such as (0.8,1.0], and the fraction of runs still in that cooperative region at t=10^6, 10^7, 10^8. Compare median exit time against the deterministic map's predicted persistence beyond 10^9 steps. If stochastic runs exit orders of magnitude earlier in most seeds, or if a substantial fraction never show the long cooperative plateau, then the model's metastable-transient timescales are not representative of incremental Q-learning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the deterministic map (14), obtained by replacing δ_{A_i,a} in (2) with π_i^a, faithfully reproduces the stochastic algorithm's asymptotic and transient behavior. Exactness of the one-step conditional expectation does not supply this: because the policy π is a nonlinear function of Q, iterating conditional expectations is not the same as the conditional expectation of the iterated process, and the true process has O(α) per-step fluctuations (only the played action is updated) whose nonlinear feedback can alter stability, exit times, and even the effective bifurcation threshold. No α→0 stochastic-approximation theorem, batch/law-of-large-numbers limit, or systematic error bound connects (14) to (2) for the fixed α=0.01 used here; the paper itself states that the model 'cannot precisely capture the exact timing of specific runs' and that adding noise would improve 'exit times from metastable phases and average periods of oscillations' (Section 3.2 and Discussion). The evidence is a visual match to five stochastic runs (Figures 1–2), which is especially weak for the headline 10^9-step metastable transient: a single run or a few runs cannot establish that the stochastic process typically stays near cooperation that long, and rare early exits due to fluctuations would make the 'exploitable metastable cooperation' claim much weaker in practice.","agreement_with_reader":"agree"},"referee_report":null,"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nYou should know about arXiv:2501.00160. It convincingly shows that the standard FAQL/BQL deterministic models approximate modified variants of Boltzmann Q-learning, not the original incremental algorithm. The genuinely new piece is a 4D mean-field map (eq. 14) that keeps the discount factor and weights updates by current policy probabilities. The fixed-point calculation is exact, the stability analysis is standard, and the Neimark-Sacker bifurcation with a stable limit cycle for gamma above about 0.75 is a clean, internally consistent result. No fitted parameters, and the discrepancy with prior models is traced to update frequencies, which is the right intuition.\n\nThe soft spots are real but addressable. The one-step conditional expectation (14) is exact, but that does not make its iterates a faithful proxy for the stochastic process. No alpha-to-zero stochastic approximation theorem or noise-rescaling argument is supplied; the evidence is visual comparison with five runs. For the headline four-billion-step metastable transient, no stochastic run approaches that length, so that prediction comes entirely from the deterministic map. The paper itself concedes in Section 3.2 and the Discussion that the model cannot capture exact timing and that adding noise would improve exit-time predictions, which partly answers the criticism but also means the abstract overstates the connection to the actual algorithm. The 'supercritical' classification of the bifurcation appears to be inferred from numerical eigenvalues rather than proven, a minor issue here. And there is no code or data release, which is my biggest practical complaint.\n\nThis is not a takedown. The paper performs a service by clearly separating FAQL/BQL from incremental Q-learning, and the 4D model is a useful tool for studying update-frequency effects. I would send it to peer review, asking for two things: a formal limit argument or a systematic stochastic benchmark over many seeds and longer runs, and a release of code. With that, it could be a solid contribution.\n\nFor your reading group: worth discussing, especially the modeling choice. I would cite the paper if the authors tighten the approximation claim.","headline":"A serious, mostly sound paper that correctly separates FAQL/BQL from incremental Q-learning and offers a plausible 4D mean-field model; its main weakness is that the link to the actual stochastic algorithm is empirical, not theorem-backed.","tokens_in":20625,"tokens_out":2039,"would_cite":true,"duration_ms":38619,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A first-principles model of incremental Boltzmann Q-learning shows apparent cooperation in the Prisoner's Dilemma is a metastable transient, and discounting drives oscillations through a Neimark-Sacker bifurcation.","keywords":["independent Q-learning","Boltzmann exploration","deterministic approximation model","Prisoner's Dilemma","Neimark-Sacker bifurcation","metastability","multi-agent reinforcement learning","moving target problem"],"falsifier":"Run many long stochastic simulations of the original algorithm in this Prisoner's Dilemma with $T=1$, $\\alpha=0.01$, $\\gamma=0.8$, and an asymmetric initial policy such as $\\pi_C=(0.5,0.48)$: if over tens of millions of steps the policy distribution does not settle onto the limit cycle predicted by the deterministic map, or instead converges to the logit QRE for most seeds, the central claim fails. For the metastability claim, start from $\\pi_C=0.9$ with $Q_{base}=0$ and $\\gamma=0.8$ and check whether the escape from mutual cooperation occurs on the roughly billion-step timescale predicted by the map, or whether the predicted dependence of escape time on learning rate and discount factor is contradicted by rescaled shorter runs.","tokens_in":19555,"feed_emoji":"🎲","tokens_out":15235,"duration_ms":135762,"temperature":0.7,"pith_summary":"The paper argues that standard deterministic approximations of independent Boltzmann Q-learning in games—the frequency-adjusted and batch Q-learning models—describe modified algorithms rather than the original incremental update rule, in which only the chosen action's Q-value changes. It introduces a discrete-time map built from the one-step conditional expectation of the stochastic update, replacing the random update indicator by the action's current Boltzmann probability, and shows that this map reproduces the qualitative behaviour of stochastic runs in the two-agent Prisoner's Dilemma. The central results are that the logit Quantal Response equilibrium is stable only for small discount factors, loses stability at $\\gamma \\approx 0.75$ through a supercritical Neimark-Sacker bifurcation, and that apparent long-term cooperation is a metastable phase lasting up to billions of steps before collapsing. If the model is right, prior policy-space analyses of Q-learning have been studying different algorithms, and quasi-equilibrium cooperation observed in multi-agent learning should be treated as potentially exploitable transient behaviour.","feed_headline":"Q-learning cooperation is a transient, not an equilibrium","feed_subtitle":"Previous models describe modified algorithms; update frequencies yield oscillations or billion-step transients.","key_machinery":"The load-bearing object is the one-step conditional-expectation map that replaces the indicator $\\delta_{A_i(t),a_i}$ in the stochastic update with the Boltzmann probability $\\pi_i^{a_i}(t)$ of choosing that action. This keeps the learning dynamics in four-dimensional Q-space instead of collapsing them into two-dimensional policy space, because the update probabilities multiply the reward and target terms in a way that cannot be rewritten using only the Q-value difference $\\Delta Q_i$. The map's fixed point $Q^*$ is characterised by Eq. (15), and the stability analysis is carried out on the eigenvalues of the Jacobian at that fixed point; for $T=1$ the critical eigenvalue pair crosses the unit circle at $\\gamma_{cr}\\approx0.75$, giving the supercritical Neimark-Sacker bifurcation and the stable limit cycle. The metastable phases are driven by the slow updating of a rarely chosen action's Q-value: when $\\pi_i^C$ is near zero, $Q_i^C$ is updated so infrequently that it takes millions to billions of steps to approach its target, so the policy looks frozen even though the system is far from equilibrium.","core_discovery":"On the paper's own terms, the discovery is that the Kronecker-delta choice of which Q-value gets updated is not a technical detail: it changes the asymptotic dynamics. The conditional-expectation model $E[Q_i^a(t+1)|Q(t)] = Q_i^a(t) + \\alpha \\pi_i^a(t)\\left(E_{A^{-i}\\sim\\pi^{-i}} R_i^{a,A^{-i}} + \\gamma \\max_b Q_i^b(t) - Q_i^a(t)\\right)$ shares its policy-space fixed point with the FAQL/BQL models—the logit Quantal Response equilibrium (the soft best-response equilibrium), with $\\pi_C^*\\approx0.227$ for $T=1$—but that fixed point's stability depends on $\\gamma$. At $T=1,\\alpha=0.01$ it is a stable focus for $\\gamma\\lesssim0.75$; at $\\gamma_{cr}\\approx0.75$ a supercritical Neimark-Sacker bifurcation makes it an unstable focus encircled by a stable limit cycle, so asymmetric initial conditions lead to permanent oscillations rather than convergence. For $\\gamma\\gtrsim0.95$ the fixed point becomes a saddle. The apparent convergence to mutual cooperation, e.g. from initial $\\pi_C=0.9$ at $\\gamma=0.8$, is shown to be a metastable phase lasting up to roughly four billion steps, after which the trajectory escapes; the paper's stability analysis of all four Q-values, not just the two policies, explains both the slow escape and the oscillations.","pith_inferences":["An untested extension of the same mechanism: in multi-state environments or games with rarely visited states, the same update-frequency lag should create analogous metastable and oscillatory phases; a generalised conditional-expectation map could be checked against simulations before attributing those phenomena to other causes.","Because the deterministic map omits update variance, adding a small noise term should predict not just qualitative regimes but quantitative exit times from metastable phases and the average period of the limit cycle, which would be a sharper test of the map's faithfulness.","For symmetric two-action games more generally, the critical discount factor could be derived as a function of payoffs, temperature, and learning rate; if the predicted bifurcation threshold matched stochastic simulations across a family of payoff matrices, the mechanism would be confirmed as general rather than specific to the Prisoner's Dilemma."],"forward_implications":["With discount factors above the bifurcation threshold, two Boltzmann Q-learners in the Prisoner's Dilemma do not converge to their logit Quantal Response equilibrium; they settle onto a stable limit cycle and keep oscillating indefinitely.","The discount factor, though it does not move the fixed-point policy, becomes a control parameter for convergence: small values give convergence, intermediate values give oscillations, and large values turn the fixed point into a saddle.","Apparent cooperation in social dilemmas observed over finite simulation horizons should not be read as equilibrium behaviour, because the metastable phase eventually collapses and is exploitable in the meantime.","Results derived from frequency-adjusted or batch Q-learning models cannot be transferred to the original incremental algorithm without re-deriving them under update frequencies.","The four-dimensional deterministic map provides a cheap way to map parameter regimes into convergence, oscillation, or metastable behaviour before running long stochastic simulations."],"supporting_citations":[{"why":"Defines the original incremental Q-learning update rule whose Kronecker delta is the feature the new model explicitly accounts for.","marker":"[2]"},{"why":"Gives the first replicator-equation approximation of Q-learning that the paper shows implicitly updates all Q-values, i.e. a modified variant.","marker":"[18]"},{"why":"Identifies the update-frequency discrepancy in prior models and introduces the frequency-adjusted variant that the new model contrasts with.","marker":"[42]"},{"why":"Provides the batch Q-learning deterministic model in policy space whose stable fixed point and gamma-independence are contradicted by the new analysis.","marker":"[35]"},{"why":"Defines the logit Quantal Response equilibrium, the common policy-space fixed point whose stability the paper re-examines.","marker":"[56]"},{"why":"Supplies a recent continuity-equation model of Q-learning in population games, limited to gamma=0, which motivates extending the approach to discounted settings.","marker":"[38]"},{"why":"Documents that the batch model matches actual learning only for very large batch sizes, supporting the claim that prior models are not the incremental algorithm.","marker":"[37]"},{"why":"Provides the canonical statement of the moving target problem used to explain the oscillations.","marker":"[1]"}],"fun_headline_variants":["Cooperation in Q-learning is a transient, not a stable state","Q-learning's cooperation is metastable and exploitable","Update order changes Q-learning dynamics: oscillations emerge","Billion-step transient cooperation in Q-learning","Neimark-Sacker bifurcation causes Q-learning oscillations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire analysis assumes that iterating the one-step average of the random update gives an accurate picture of what the stochastic algorithm does over long times, even though no formal limit theorem ties the deterministic map's attractors or transient durations to the random process; the support shown is a visual comparison with five stochastic runs.","fun_headline_variants_meta":{"raw":{"variants":["Cooperation in Q-learning is a transient, not a stable state","Q-learning's cooperation is metastable and exploitable","Update order changes Q-learning dynamics: oscillations emerge","Billion-step transient cooperation in Q-learning","Neimark-Sacker bifurcation causes Q-learning oscillations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1554,"prompt_tokens":1159,"completion_tokens":395,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":775,"completion_tokens_details":{"reasoning_tokens":318}},"tokens_in":775,"tokens_out":395,"duration_ms":4293,"temperature":1.0,"reasoning_tokens":318,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:59:28.704475+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run many long stochastic simulations of the original algorithm in this Prisoner's Dilemma with $T=1$, $\\alpha=0.01$, $\\gamma=0.8$, and an asymmetric initial policy such as $\\pi_C=(0.5,0.48)$: if over tens of millions of steps the policy distribution does not settle onto the limit cycle predicted by the deterministic map, or instead converges to the logit QRE for most seeds, the central claim fails. For the metastability claim, start from $\\pi_C=0.9$ with $Q_{base}=0$ and $\\gamma=0.8$ and check whether the escape from mutual cooperation occurs on the roughly billion-step timescale predicted by the map, or whether the predicted dependence of escape time on learning rate and discount factor is contradicted by rescaled shorter runs.","supporting_citations":[{"cited_title":"A selection-mutation model for q-learning in multi-agent systems","cited_arxiv_id":null,"evidence_quote":"Gives the first replicator-equation approximation of Q-learning that the paper shows implicitly updates all Q-values, i.e. a modified variant."},{"cited_title":"Frequency adjusted multi-agent q-learning","cited_arxiv_id":null,"evidence_quote":"Identifies the update-frequency discrepancy in prior models and introduces the frequency-adjusted variant that the new model contrasts with."},{"cited_title":"Donges, and J¨ urgen Kurths","cited_arxiv_id":null,"evidence_quote":"Provides the batch Q-learning deterministic model in policy space whose stable fixed point and gamma-independence are contradicted by the new analysis."},{"cited_title":"Quantal response equilibria for normal form games","cited_arxiv_id":null,"evidence_quote":"Defines the logit Quantal Response equilibrium, the common policy-space fixed point whose stability the paper re-examines."},{"cited_title":"Dynamical systems as a level of cognitive analysis of multi-agent learning","cited_arxiv_id":null,"evidence_quote":"Documents that the batch model matches actual learning only for very large batch sizes, supporting the claim that prior models are not the incremental algorithm."},{"cited_title":"Sutton and Andrew G","cited_arxiv_id":null,"evidence_quote":"Provides the canonical statement of the moving target problem used to explain the oscillations."}],"review_version":1}