{"id":"737d0e0b-ae06-4dc3-b6c7-602308683c09","arxiv_id":"2507.18333","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Cooperative MARL agents solve many benchmarks with brittle conventions that bypass observations and memory, so the field needs tasks that genuinely require grounded, history-based reasoning.","lead":"This paper argues that cooperative multi-agent reinforcement learning successes often come from agents learning brittle conventions that ignore observations and memory, not from genuine reasoning. The authors test this with a custom game and benchmark audits, then call for new environments that force grounded, memory-based behavior.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central benchmark conclusion rests on correlational MI/parity evidence; no causal ablation shows trained policies actually bypass observations or memory in Hanabi/SMAX/MaBrax.","rationale":"The reader's conditional verdict is well founded. The strongest claim is about why model-free MARL methods succeed and what benchmarks test. For the toy, blind agents and partner-swap failures are direct causal evidence. For the benchmarks, the paper relies on MI estimates and FF-vs-RNN parity. Low MI is not a causal demonstration of non-use: it can reflect sparse but indispensable information. FF≈RNN parity is also ambiguous without knowing whether both are converged or whether the observation function leaks the belief state; footnote 2 explicitly acknowledges one such leak. A frozen-policy ablation with observations removed or recurrent state reset would resolve this unambiguously. The absence of such a control is the single largest gap between the toy demonstration and the general benchmark conclusion. This does not require rejecting the paper; it makes the CONDITIONAL verdict appropriate and gives authors a concrete path to strengthen from position-claim to demonstrated phenomenon.","tokens_in":12621,"tokens_out":13670,"duration_ms":165520,"concrete_test":"Take the final checkpoints used for Figures 5a–5c and evaluate them with frozen weights under three ablations: (1) observations replaced by zero/noise at every timestep; (2) RNN hidden state reset to zero at every timestep; (3) both. Compare returns to the unablated checkpoints. If returns stay at parity, the policies indeed bypass observations/memory and the benchmark conclusions hold; if returns drop, the MI/FF-vs-RNN interpretation is confounded and the paper's Section 5 claims need revision. Run the same protocol on the Prediction Game concurrent agents as a positive control, where returns should be invariant to all three ablations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is causal: model-free MARL methods succeed on benchmarks without recovering a Markov signal. The Prediction Game provides a causal control (blind agents, heuristic-partner failure), but the Section 5 benchmark conclusions do not. Hanabi's 'memory not needed' is inferred from FF≈RNN and from I(H;A) estimates; SMAX and MaBrax conclusions are similarly built on MI values and FF/RNN comparisons. These are correlational proxies. Low I(H;A) can coexist with rare but essential memory use, and FF≈RNN can be an artifact of under-training or of an observation function that already exposes the belief state (the authors' footnote 2 concedes the discard-pile observation, and the JaxMARL observation also exposes per-card accumulated knowledge). To make the benchmark claims load-bearing, the authors need to intervene on the trained policies: replace observations with noise and/or reset the recurrent state at evaluation. No such ablation appears in Section 5. Without it, the strongest reading—that modern benchmarks measure brittle conventions rather than Markov-signal recovery—is not directly evidenced outside the toy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that the empirical success of model-free cooperative MARL methods on standard benchmarks is explained not by recovery of a Markov signal (i.e., state estimation and agent modelling) but by learning simple, brittle conventions that bypass observations and memory. The authors first build a custom Prediction Game in which four IPPO agents with feed-forward or recurrent policies, and even blind variants receiving no observations, all reach near-optimal returns while exhibiting low mutual information between observations/actions and hidden state/actions; when two agents are replaced by fixed heuristic partners, performance collapses, showing the learned behaviour is a non-generalising convention. In a heterogeneous variant where partners follow random-phase periodic policies, RNN policies succeed with markedly higher MI, FF policies do worse, and blind agents fail, indicating that grounded, memory-based policies are learnable when the task requires them. The paper then applies the same MI and feed-forward-versus-recurrent diagnostics to Hanabi, MaBrax, and SMAX, concluding that Hanabi does not require memory, MaBrax can be solved without observations, and SMAX, while improved, still does not strongly require history-based reasoning. The authors conclude that benchmarks should be redesigned to enforce observation-grounded and memory-based behaviour.","tokens_in":12725,"tokens_out":3824,"duration_ms":41208,"significance":"If the central claim holds, the paper makes a valuable diagnostic contribution: benchmark rankings in cooperative MARL may partly reflect co-adapted conventions rather than the temporal reasoning and belief maintenance that Dec-POMDP theory treats as essential. The Prediction Game is a well-designed controlled study with a genuinely informative negative result (blind agents succeed under concurrent training) and a positive control (random-phase heuristic partners force grounded memory use). The paper also usefully connects its findings to prior work on SMAC-v2 open-loop policies and extends the critique to newer benchmarks such as MaBrax and SMAX. The main weakness is that the benchmark conclusions in Section 5 rest on correlational evidence—MI estimates and architecture comparisons—rather than direct interventions on trained policies, so the strong interpretation of those results is not yet fully supported.","major_comments":[{"comment":"The Section 5 benchmark claims that Hanabi, MaBrax, and SMAX policies do not rely on observations or memory are inferred only from I(O;A), I(H;A), and feed-forward-versus-recurrent performance parity; these are correlational proxies rather than direct evidence about the trained policies. Low MI can coexist with rare but decision-critical observation use, and FF≈RNN can arise from under-training or from an observation function that already exposes the belief state. The authors should add evaluation-time interventions, for example replacing observations with noise, resetting or zeroing the recurrent state, or freezing the partner policy, and then report the resulting returns. Without such ablations, the strong reading that modern benchmarks measure brittle conventions rather than Markov-signal recovery is established only in the Prediction Game, not in Section 5.","section":"Section 5, Fig. 5"},{"comment":"The k-NN mutual information estimator is applied to high-dimensional observations and recurrent hidden states, but the paper does not report the neighbor count k, does not validate the estimator on variables with known MI, and does not discuss bias for continuous or near-deterministic features. The I(O;A) and I(H;A) values in Figures 3–5 are therefore not yet interpretable as reliable measures of policy dependence. The authors should provide calibration experiments on synthetic data with known MI and report estimator settings for each environment.","section":"Section 3, Mutual Information"},{"comment":"The Hanabi experiments use an observation that includes the entire discard pile, as acknowledged in footnote 2. This makes the feed-forward policy's observation nearly sufficient for the information that a memory-based policy would otherwise need to accumulate, so the FF≈RNN result may be an artifact of the chosen observation function rather than evidence that Hanabi does not test memory. The claim that the task 'can be solved with no notion of history' should be tested with a top-card-only observation, or the conclusion should be explicitly restricted to this specific observation encoding.","section":"Section 5, Hanabi paragraph and footnote 2"},{"comment":"For MaBrax, the claim that blind agents achieve 'non-trivial' performance is not quantified against a random-policy baseline or a fully-observable upper bound, and the return scales differ widely across the five instances shown in Fig. 7. Please report normalized scores or explicit baselines so the reader can assess how much of each task is solvable without observations, and clarify what threshold would count as 'non-trivial'.","section":"Section 5, MaBrax and Fig. 8"}],"minor_comments":[{"comment":"The text contains a typo: 'Esch agent then selects' should read 'Each agent then selects'.","section":"Section 4, first paragraph"},{"comment":"The phrase 'We address push this idea further' is grammatically incomplete and should be revised.","section":"Section 4, introductory paragraph"},{"comment":"The caption labels panel (c) as 'histories' but the quantity plotted is the mutual information between the GRU hidden state and actions; please use consistent terminology.","section":"Figure 3 caption"},{"comment":"The captions state approximate MI maxima but do not specify the action-space cardinality for Hanabi; adding H(A) explicitly would help readers interpret the percentages quoted in the SMAX paragraph.","section":"Figure 5, panels d–g"},{"comment":"The heuristic policy formula uses floor notation without defining it; adding a short definition would improve clarity.","section":"Appendix A.1"},{"comment":"The limitation to Independent PPO without parameter sharing and without centralised critics is appropriately acknowledged only in the conclusions; it would be clearer to state this restriction in the experimental setup of Section 4 as well.","section":"Section 6, Next-steps"}],"recommendation":"major_revision","confidential_remarks":"This is a thoughtful position paper with a well-controlled toy study that demonstrates the core phenomenon convincingly. My recommendation of major_revision is driven by the gap between the strength of the Prediction Game evidence and the broader benchmark claims in Section 5, which currently rely on correlational MI and FF/RNN comparisons without evaluation-time interventions. I do not see a circularity problem: the Prediction Game was designed to exhibit conventions, but the random-phase heuristic-partner condition provides an independent control that supports the paper's interpretation. The paper's fit is more diagnostic than methodological, which may matter for the journal's scope, but the message is timely for the MARL benchmarking community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe part of this paper worth your time is the Prediction Game. They design a cooperative task where success requires predicting the next action of neighboring agents, train IPPO agents together, and show that blind agents (no observations at all) reach near-optimal reward by locking into a brittle convention. When fixed heuristic partners are introduced, performance collapses for all architectures. Then, when the heuristic partners follow random-phase cycles, the same RNNs can learn to track them, and the mutual information between observations/actions and hidden state/actions rises sharply. That is a clean, internally controlled demonstration that co-adaptive training can bypass any Markov signal recovery, and that the same models can ground their policies when the task requires it. Reframing conventions as a benchmark diagnostic, not just an algorithmic nuisance, is genuinely useful.\n\nSection 5 is where I get skeptical. The Hanabi and SMAX conclusions are built on MI estimates and on FF≈RNN parity, which are correlational proxies. The k-NN MI estimator is not validated on these high-dimensional or recurrent policies, and low I(O;A) can also mean the observation carries little variability. FF≈RNN can be under-training or an observation function that already exposes the belief state — the authors' own footnote 2 concedes exactly that for Hanabi's discard pile. To make these claims load-bearing, they should intervene on the trained policies: feed noise observations or reset the recurrent state at evaluation, and measure the performance drop. Without that, the claim that Hanabi and SMAX measure brittle conventions rather than Markov-signal recovery remains unproven outside the toy case. The MaBrax blind-agent results are more direct and do extend the SMAC-v2 open-loop critique, though they only cover 5 configurations from one environment family.\n\nThe paper is honest about its limits, which counts in its favor. No code or data has been released yet, which hurts reproducibility, though the hyperparameters and standard environments make reimplementation feasible.\n\nNet: the Prediction Game deserves citing, Section 5 should be read as hypothesis generation, and the paper deserves a serious referee — with revisions, mostly to Section 5, adding causal ablations or at least MI positive controls.","headline":"Strong Prediction Game case study, but the Section 5 benchmark claims rest on correlational MI and FF/RNN parity without causal ablations — treat them as a hypothesis, not settled evidence.","tokens_in":13345,"tokens_out":2715,"would_cite":true,"duration_ms":27013,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that the empirical success of model-free cooperative MARL is driven by brittle conventions that ignore observations and memory, and that cooperative benchmarks must be redesigned to force grounded, memory-based behavior.","keywords":["cooperative multi-agent reinforcement learning","Dec-POMDP","conventions","mutual information","Markov property","partially observable environments","benchmark design","memory-based reasoning"],"falsifier":"Mask the observation input and hidden state of trained agents in Hanabi, SMAX, and MaBrax and measure the return drop; if policies with low measured I(O;A) and I(H;A) still lose substantial performance when observations are masked, then low mutual information does not indicate that the policy bypasses sensory input. Conversely, on a task where the optimal policy provably requires history, the estimator should return high values; a failure there would undercut the diagnostic.","tokens_in":12351,"feed_emoji":"🧠","tokens_out":7218,"duration_ms":64301,"temperature":0.7,"pith_summary":"This paper argues that the strong empirical performance of model-free cooperative multi-agent reinforcement learning does not come from agents recovering a Markovian signal about the environment and other agents. Instead, co-trained agents often settle into simple, synchronized conventions that ignore both sensory input and recurrent memory, and these conventions are brittle when partners change. The evidence includes a prediction game where blind agents with no observations match sighted agents, and mutual-information measurements showing actions carry little information about observations or hidden states. The same architectures switch to grounded, memory-based policies when the task blocks conventions, which points to benchmark design rather than model capacity as the problem. The authors conclude that many Dec-POMDP benchmarks do not test the temporal reasoning and partner modeling they claim to measure, and they propose environments requiring observation-grounded and memory-based behavior.","feed_headline":"MARL benchmark wins may be brittle conventions, not reasoning","feed_subtitle":"Blind agents rival sighted ones in several cooperative tasks, so benchmarks must force memory and observation use.","key_machinery":"The central measurement device is paired mutual information statistics: I(O;A), between an agent's observation and its chosen action, and I(H;A), between its recurrent hidden state and its action, estimated with a k-NN estimator and averaged over agents. Low values are interpreted as evidence that the policy bypasses sensory input and memory. The complementary mechanism is the blind environment variant, which removes observations entirely, and the Prediction Game, which explicitly rewards predicting neighbors' next actions. This combination lets the paper attribute high returns to conventions rather than Markov-state recovery, and to show that when conventions are impossible, recurrent policies learn grounded, history-dependent behavior.","core_discovery":"The central claim is that model-free MARL policies succeed on cooperative benchmarks not by recovering a Markovian signal—a belief over the joint state and other agents' policies—but by learning simple conventions that bypass both observations and memory. In the Prediction Game, concurrently trained IPPO agents, including fully blind agents with no observations, all reach near-optimal returns, and mutual information I(O;A) and I(H;A) is low. Performance collapses when two agents are replaced by fixed heuristic partners, demonstrating that the learned policies are brittle conventions. When partners follow phase-randomized periodic cycles, learning agents must infer the hidden phase from observations; recurrent policies succeed and mutual information rises, while blind agents fail, showing the models can learn grounded policies when the task requires it. A mutual-information analysis of Hanabi, MaBrax, and SMAX then shows Hanabi demands observation grounding but not memory, MaBrax lets blind agents achieve non-trivial returns, and SMAX requires only moderate history reliance. The paper concludes that modern MARL environments do not adequately force the reasoning capacities that Dec-POMDPs assume, and that benchmarks should be redesigned around observation-grounded behavior and memory-based reasoning about other agents.","pith_inferences":["The same mutual-information diagnostic could be applied to zero-shot coordination and ad-hoc teamwork benchmarks as a pre-registered sanity check before trusting a ranking, an extension the paper does not perform.","If the diagnostic logic is sound, a testable prediction follows: in redesigned benchmarks, the gap between recurrent and memoryless policies, and between sighted and blind agents, should widen substantially; current small gaps are the failure signature.","The critique may carry over to single-agent POMDP benchmarks, where recurrent agents could also exploit task-specific conventions; the paper's blind-agent test could be adapted there."],"forward_implications":["If co-trained successes are conventions, then existing benchmark rankings may partly measure how easily agents coordinate shortcuts, not how well they track hidden state or model teammates.","Hanabi, despite requiring observation-grounded play, can be solved with no history, so it tests groundedness but not memory-based reasoning; feed-forward matching recurrent performance is the evidence.","MaBrax locomotion instances do not require partner modeling or even observations, since blind agents achieve non-trivial returns; using them as cooperative MARL benchmarks can overstate progress.","SMAC-v2/SMAX is an improvement over the original SMAC, yet its mutual-information values remain moderate, so it still leaves room for environments that force stronger history-based reasoning.","Following the two design principles—observation-grounded behavior and memory-based reasoning about others—should make success require genuine Dec-POMDP skill rather than fragile co-adaptation."],"supporting_citations":[{"why":"Defines the Dec-POMDP model and its NEXP-complete decentralized control problem, establishing the theoretical setting the paper claims benchmarks fail to test.","marker":"Bernstein et al., 2002"},{"why":"Specifies that agents should form multi-agent beliefs to recover a Markovian signal, the assumption the paper argues is bypassed.","marker":"Oliehoek et al., 2016"},{"why":"Provides the k-NN estimator used to compute I(O;A) and I(H;A), the paper's main diagnostic.","marker":"Kraskov et al., 2004"},{"why":"Extends mutual-information estimation to mixed discrete/continuous data, backing the estimator's application to policy activations and actions.","marker":"Ross, 2014"},{"why":"Introduces Independent PPO (IPPO), the algorithm used in all Prediction Game, Hanabi, MaBrax, and SMAX experiments.","marker":"De Witt et al., 2020"},{"why":"Shows original SMAC maps can be solved open-loop and introduces SMAC-v2 with meaningful partial observability, the benchmark SMAX instantiates and the paper re-analyzes.","marker":"Ellis et al., 2023"},{"why":"Formalizes the Hanabi challenge, the environment where the paper shows grounding without memory demand.","marker":"Bard et al., 2020"},{"why":"Provides JAXMARL implementations and hyperparameters for Hanabi, MaBrax, and SMAX used in the benchmark study.","marker":"Rutherford et al., 2023"},{"why":"Documents the surprising effectiveness of PPO in cooperative MARL, the kind of empirical success the paper argues is convention-driven.","marker":"Yu et al., 2022"},{"why":"Prior work treating conventions as an algorithm-level problem; the paper reframes conventions as a benchmark-design diagnostic.","marker":"Hu et al., 2020"}],"fun_headline_variants":["MARL wins may rely on conventions, not Markov reasoning","Blind agents match sighted in MARL: benchmarks too easy","Cooperative RL successes are brittle, not observation-grounded","MARL benchmarks fail to demand observation and memory"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the k-NN mutual-information estimates, averaged over agents, faithfully measure how much a policy relies on its observations and memory; if that estimator is biased for recurrent or high-dimensional policies, the benchmark conclusions weaken.","fun_headline_variants_meta":{"raw":{"variants":["MARL wins may rely on conventions, not Markov reasoning","Blind agents match sighted in MARL: benchmarks too easy","Cooperative RL successes are brittle, not observation-grounded","MARL benchmarks fail to demand observation and memory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000922,"raw_usage":{"total_tokens":3977,"prompt_tokens":994,"completion_tokens":2983,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":2916}},"tokens_in":610,"tokens_out":2983,"duration_ms":22696,"temperature":1.0,"reasoning_tokens":2916,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:14:40.886008+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Mask the observation input and hidden state of trained agents in Hanabi, SMAX, and MaBrax and measure the return drop; if policies with low measured I(O;A) and I(H;A) still lose substantial performance when observations are masked, then low mutual information does not indicate that the policy bypasses sensory input. Conversely, on a task where the optimal policy provably requires history, the estimator should return high values; a failure there would undercut the diagnostic.","supporting_citations":[{"cited_title":"The complexity of decentralized control of markov decision processes","cited_arxiv_id":null,"evidence_quote":"Defines the Dec-POMDP model and its NEXP-complete decentralized control problem, establishing the theoretical setting the paper claims benchmarks fail to test."}],"review_version":2}