{"id":"53b0ae76-b0ee-47e7-b095-48d0796db0f5","arxiv_id":"2506.16718","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MRDG outperforms RPM, CSP, and ODITS on the new ACCA generalization benchmark in three multi-agent environments, with significance and true out-of-distribution novelty remaining unproven.","lead":"The paper defines a new multi-agent generalization benchmark, ACCA, in which one learner must collaborate with unseen teammates and compete against unseen opponents across new tasks, and introduces MRDG, a retrieval plus hypernetwork method that adapts the learner's policy from the other agents' trajectories.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-time partners are drawn from the training DPP/predefined policy set, so reported gains may reflect in-distribution retrieval rather than adaptation to genuinely unseen agents.","rationale":"The paper is a plausible engineering contribution: the ACCA framework is clearly defined, MRDG is modular, and the ablations in Tables 5 and 6 show that each component contributes. However, the paper's own description of the evaluation undercuts its headline. In Section 4, test-time G−1 strategies are drawn 'from a predefined set, potentially including policies from the DPP held out from the training set'; Section 5.1 further says Overcooked tests on a held-out layout and Melting Pot uses pre-trained policies from the same substrates. Nowhere do the authors construct test agents that are behaviorally out-of-distribution with respect to the retrieval memory. Because episodic memory is populated from DPP-sampled interactions, a held-out DPP policy is still in the support of the retrieval index. The reported win rates and rewards are thus evidence for the effectiveness of the retrieval/hypernetwork machinery at selecting among known policy types, not for zero-shot adaptation to genuinely novel agents. This matches the reader's weakest_assumption and is the main risk to the central claim. Secondary issues—overlapping error bars in several Melting Pot rows, no significance testing, and unreported hyperparameters—weaken individual comparisons but do not affect the central logic as directly. The correct remedy is a direct out-of-distribution evaluation, and the public code release makes that check feasible. The CONDITIONAL verdict remains appropriate; my read does not change it.","tokens_in":27917,"tokens_out":4250,"duration_ms":46137,"concrete_test":"Train a set of evaluation-only teammate and opponent policies from scratch on each test substrate using an independent algorithm not used to populate the DPP (e.g., QMIX or PPO with different seeds and reward shaping), and ensure their trajectories are excluded from D_r. Rerun the SMAC 5m_vs_6m, Overcooked Coordination Ring, and Melting Pot evaluations under this protocol, reporting per-seed results and a permutation test. If MRDG's advantage over CSP, RPM, and ODITS persists, the out-of-distribution generalization claim is supported; if the gap narrows to overlap, the reported gains largely reflect memory interpolation over training-distribution policies.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that MRDG generalizes to 'unseen' teammates and opponents—requires that evaluation agents lie outside the training-time behavioral distribution. The paper's own setup contradicts that. Section 4 states that test-time strategies are 'drawn from a predefined set, potentially including policies from the DPP held out from the training set.' Since episodic memory D_r is populated with behavioral trajectories of DPP-sampled G−1 during training, held-out DPP policies remain inside the retrieval index's behavioral manifold. MRDG's retrieval then finds near-neighbor trajectories and the hypernetwork interpolates parameters; this is in-distribution memory lookup, not adaptation to out-of-distribution agents. The SMAC, Overcooked-AI, and Melting Pot evaluations contain no partner policies trained independently (different algorithm, human data, or different substrate-specific training) that are guaranteed absent from D_r. Section 3.3 also assumes G−1 is non-adaptive, further narrowing the regime tested. The measured improvements over RPM, CSP, and ODITS are therefore consistent with a weaker claim: better memory indexing over a known policy zoo.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces a new problem setting, Agent Collaborative-Competitive Adaptation (ACCA), in which a single learner must generalize across tasks and environments while cooperating with unseen teammates and competing against unseen opponents. To address it, the authors propose MRDG, a method combining a Diversity Policy Pool (DPP), episodic memory with multi-retrieval, a viewpoint-alignment encoder, positional encoding, and a hypernetwork that dynamically generates policy parameters. The method is evaluated on SMAC, Overcooked-AI, and Melting Pot against RPM, CSP, and ODITS, with MRDG reported as best in all evaluated scenarios; ablation studies on two Melting Pot substrates are also provided. The paper includes a public code repository.","tokens_in":28175,"tokens_out":5311,"duration_ms":51108,"significance":"If the central claim were fully supported, MRDG would be a useful contribution to agent modeling and multi-agent generalization, and the ACCA framing is a reasonable extension of zero-shot MARL and ad hoc teamwork. The paper has notable strengths: it tackles a broad and practically motivated problem, evaluates on three established benchmarks, includes multiple baselines and ablations, and releases code. However, the current evidence for generalization to genuinely unseen agents is weakened by the overlap between test policies and the training-time policy pool, and the reported performance advantages are not backed by statistical significance tests. The contribution is therefore promising but needs stronger evaluation to support the claims.","major_comments":[{"comment":"The evaluation protocol does not support the central claim that MRDG adapts to 'unseen' teammates and opponents. Section 4 states that test strategies are 'drawn from a predefined set, potentially including policies from the DPP held out from the training set.' Since the episodic memory D_r is populated during training with trajectories of DPP-sampled G-1, any held-out DPP policy is still represented in the retrieval index; the reported gains can therefore be explained by in-distribution memory interpolation rather than by adaptation to out-of-distribution agents. I ask for an evaluation with test policies that are guaranteed absent from both DPP and D_r (e.g., independently trained policies from different algorithms, human data, or a strict policy-family split), and for a clear statement of the distributional relationship between training and test policies. Section 3.3's assumption that G-1 is non-adaptive should also be explicitly tested or softened, since real deployments may involve adaptive partners.","section":"Section 4, Methodology (test-phase protocol)"},{"comment":"The word 'significantly' is used repeatedly (e.g., in the abstract and in Section 5.2), but no statistical test is reported. In Table 3, MRDG's 205±18 overlaps with CSP's 162±61 and RPM's 158±56 within one standard deviation; in Table 4, Stag Hunt MRDG 14.6±1.3 overlaps with RPM 13.7±2.2, and Chicken MRDG 34±7 overlaps with RPM 27±5 at one standard deviation. With only four seeds, these differences are not established. Please report per-scenario significance tests or, if four seeds are insufficient, report individual seed values, confidence intervals, and effect sizes; also align the caption conventions, since Tables 2-3 say 'standard error' while Table 4 says 'standard deviation.'","section":"Section 5.2, Tables 2-4"},{"comment":"The ablation claims are not supported by the reported tables. For example, the text says that removing DPP yields 'an improvement of over 40% in generalization reward retention,' but Table 5 shows W/O(DPP) 7.9±2.2 vs MRDG 14.6±1.3, a performance drop of about 46%, while Table 6 shows 10.2±1.67 vs 12.3±0.56, a drop of about 17%. Other quantitative statements in this section (20%, 26%, 35%, 60%, 55%) are not backed by any table or figure. Please report complete ablation results for all modules and all tested substrates, define each claimed metric (e.g., 'reward retention,' 'convergence speed'), and remove or substantiate unsupported quantitative assertions.","section":"Section 5.3, Tables 5-6"},{"comment":"The formal objective maximizes the joint team reward under a 'shared reward structure,' but the paper's mixed-motive experiments (Chicken, Prisoner's Dilemma) involve individual payoffs that conflict with team returns; Eq. (2) uses a single r_t without specifying whose reward is used or how the mixed-motive payoff matrix enters the objective. This makes it unclear whether MRDG optimizes team return, individual return, or a combination. Please clarify the reward used in Eq. (2), in Algorithm 1, and in the Melting Pot evaluation, and discuss why optimizing a team objective is appropriate for mixed-motive substrates.","section":"Section 3.3, Eqs. (2)-(3)"}],"minor_comments":[{"comment":"The symbol ∏_f is introduced in Section 3.2 but never defined; please define it formally.","section":"Section 3.2"},{"comment":"The indicator 1(b=i) in Eq. (10) is unclear because b has not been defined; either define b or remove the indicator, and clarify the indexing of the summation.","section":"Eq. (10)"},{"comment":"The method is called both 'Multi-Retriever' and 'Multi-Retrieval' in different places; please standardize the name throughout.","section":"Abstract and Highlights"},{"comment":"There are typographical issues such as 'trival' for 'trivial,' and repeated phrasing such as 'In the scenario with pure common interest' in Section 5.1; a proofread would improve clarity.","section":"Section 5.2"},{"comment":"The main text refers to Figure 7 for ablation curves while Figure 6 is labelled as the ablation study figure; please check the figure references and captions for consistency.","section":"Figures 6-7 and Tables 5-6"},{"comment":"Appendix F reports architecture sizes but not key hyperparameters (retrieval count m, memory length l, re-initialization factors λ and γ, learning rates, update schedules); adding a hyperparameter table would improve reproducibility, even with the code link.","section":"Appendix F"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a competent, well-scoped engineering paper. It proposes a new benchmark protocol, ACCA, that combines MARL zero-shot transfer with ad hoc teamwork and adds competition, plus a modular method, MRDG, built from retrieval, positional encoding, a hypernetwork, and viewpoint alignment. On all three benchmarks—SMAC, Overcooked-AI, Melting Pot—MRDG beats RPM, CSP, and ODITS, and the results are consistent across settings. That is a real contribution; someone working on MARL generalization should read this.\n\nThe execution has real strengths. The benchmark design is sensible: train on a few substrates, test on a held-out one, with both teammates and opponents changing. The method is modular and the ablation study is at least structured. The code is released, which is more than many papers in this area do. The paper also honestly lists its own limitations—memory overhead, potential latency—without being asked.\n\nThe soft spots are not fatal but they materially weaken the central claim. The biggest one is the 'unseen' evaluation. Section 4 states that test strategies are drawn from a predefined set, \"potentially including policies from the DPP held out from the training set.\" Since episodic memory is populated with trajectories from DPP policies during training, held-out DPP policies live on the same behavioral manifold as the memory contents. The retrieval module is then doing near-neighbor lookup over a known policy zoo, not adapting to genuinely out-of-distribution agents. The paper would need partners trained independently—different algorithms, human data, or at least policies guaranteed absent from D_r—to support the generalization claim. With the current protocol, the numbers support a weaker but still useful claim: better memory indexing over a known set of policies.\n\nSecond, the statistics are thinner than the text suggests. Several headline comparisons have overlapping error bars: Overcooked 205±18 vs 158±56, Stag Hunt 14.6±1.3 vs 13.7±2.2. The paper calls these 'significant' without any significance test. That is a fixable weakness, but it should be fixed before publication.\n\nThird, the ablations report percentages like '40% improvement' and '60% improvement' without raw numbers or confidence intervals. The tables give some values, but the prose claims more than the reported data can verify. This is a minor-to-moderate transparency issue.\n\nBottom line: this deserves a serious referee. The benchmark protocol and the method are worth engaging with, and the flaws are addressable. The authors should either add a genuinely out-of-distribution evaluation or reframe the claim as in-distribution policy interpolation. For my own work, I would cite the benchmark but treat the 'generalization to unseen agents' phrasing with caution.","headline":"Solid engineering with a useful new benchmark, but the 'unseen' claim is undercut by test partners being drawn from the same policy pool used in training.","tokens_in":28702,"tokens_out":1518,"would_cite":true,"duration_ms":17580,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a single agent can adapt to unfamiliar teammates and opponents by retrieving their behavior from memory and generating its own policy parameters on the fly, and it validates this claim across three multi-agent…","keywords":["multi-agent system","agent modeling","zero-shot coordination","ad hoc teamwork","retrieval-augmented generation","hypernetwork","generalization","reinforcement learning"],"falsifier":"Train MRDG on one predefined policy pool, then test it against partners produced independently, for example agents trained by a different MARL algorithm or human demonstrations, with no policy overlap with the pool. If the learner's performance drops to the level of RPM, CSP, or ODITS, then the reported generalization is memory interpolation rather than adaptation to genuinely new behavior.","tokens_in":27716,"feed_emoji":"🤖","tokens_out":3855,"duration_ms":42523,"temperature":0.7,"pith_summary":"The paper is trying to establish that one learner agent can generalize across new tasks, new environments, and interactions with both unknown teammates and unknown opponents, without retraining. It proposes a setting called Agent Collaborative-Competitive Adaptation (ACCA) that combines the difficulties of zero-shot coordination and ad hoc teamwork. To solve this, it introduces Multi-Retrieval and Dynamic Generation (MRDG), which stores snippets of other agents' behavior, retrieves relevant snippets at decision time, and uses a hypernetwork to rewrite the learner's own policy parameters. The paper reports that MRDG outperforms three established baselines on SMAC, Overcooked-AI, and Melting Pot, with the largest headline win being 76% on the unseen SMAC map 5m_vs_6m. The sympathetic reading is that adaptation to novel partners can be achieved by dynamically composing policy behavior from retrieved experience rather than relying on a fixed network.","feed_headline":"A single agent rewires its policy for unseen teammates and rivals","feed_subtitle":"MRDG retrieves partners' behavior and generates fresh policy weights on the fly, beating all baselines in three benchmarks.","key_machinery":"The central mechanism is the sample-interact-retrieve-generate-store loop of Multi-Retrieval and Dynamic Generation (MRDG). A Diversity Policy Pool (DPP) randomly samples substrates and teammate/opponent policies so the learner experiences a wide range of coordination schemes. The learner's episodic memory stores short trajectories of other agents' observations and actions, encoded by per-agent Viewpoint Alignment (VA) encoders that map third-person perspectives into the learner's observational frame. At decision time, a retrieval network uses the learner's observation as a query and returns the most frequent action among the closest memory entries for each other agent. A positional encoder marks each agent's index and structural role, and a hypernetwork takes the summed retrieved actions and positional encodings to generate the parameters of the first layers of the learner's policy network, so the policy is rebuilt for the current situation rather than selected from a fixed library. Base MARL parameters are updated separately and interpolated with re-initialization weights, giving a stable base strategy plus a dynamically adapted component.","core_discovery":"The central claim is that a single agent, called the learner, can simultaneously handle unfamiliar teammates and opponents by explicitly modeling their behavior through retrieval and by dynamically generating its own policy parameters. The paper defines ACCA as a framework where the learner must adapt to changes in substrate environment, task objective, and the strategies and numbers of other agents. MRDG builds a Diversity Policy Pool of training-time teammate and opponent policies, records their observation-action trajectories in episodic memory, retrieves the most similar past actions given the learner's current observation, and feeds those retrieved actions together with positional encodings into a hypernetwork that generates parameters for the learner's policy network. The paper reports that this method outperforms RPM, CSP, and ODITS in the tested scenarios: 76% win rate on SMAC's unseen 5m_vs_6m map, a score of 205 on Overcooked-AI's Coordination Ring, and leading rewards across Chicken Game, Clean Up, Pure Coordination, Prisoner's Dilemma, Rational Coordination, and Stag Hunt in Melting Pot. The paper also reports ablations showing that each component, the policy pool, positional encoding, hypernetwork, and viewpoint alignment, contributes to the final performance.","pith_inferences":["This is an inference: because the test-time 'unseen' policies are drawn from the same predefined policy set that includes held-out Diversity Policy Pool entries, the reported gains may reflect interpolation within the training behavioral distribution rather than adaptation to genuinely out-of-distribution partners; a test with partners trained independently by a different algorithm would settle th","This is an inference: the paper assumes in its formalization that all other agents are non-adaptive, so in deployment settings where teammates or opponents also update their strategies in response to the learner, the retrieval memory could become stale and the method would need an extension that tracks or anticipates the other agents' learning.","This is an inference: the same retrieve-and-generate architecture, hypernetwork output from episodic memory, could be applied to single-agent continual or meta-RL settings where the task itself changes, with episodic memory storing rewarding behaviors instead of agent strategies.","This is an inference: the viewpoint alignment module is a general technique for merging partial observations from agents with different sensors or fields of view; a direct testable extension would be measuring whether the aligned representations transfer to new observability configurations not seen during training."],"forward_implications":["If MRDG's central claim holds, a learner trained on a few map sizes and task layouts can transfer to unseen team sizes and enemy counts without new gradient updates, as demonstrated by training on SMAC 3m and 8m and testing on 5m_vs_6m.","Explicit opponent modeling alongside teammate modeling should give an advantage in mixed-motive settings, where choosing cooperate or defect depends on the partner's current behavior, which matches the reported Melting Pot results.","Retrieval plus hypernetwork parameter generation is presented as faster and more flexible than static policy libraries such as RPM or fixed latent-conditioned policies such as ODITS, because the policy can be rebuilt every timestep from the current behavioral evidence.","The ablation results imply that each module has a distinct role: the diversity pool supplies the behavioral coverage, positional encoding handles varying team structure, the hypernetwork provides dynamic policy generation, and viewpoint alignment enables cross-agent experience reuse.","Dynamic parameter generation could make a single agent adaptable to new tasks and partners without maintaining a separate policy or sub-policy per scenario."],"supporting_citations":[{"why":"Supplies the SMAC benchmark used for the pure common-interest evaluation and the 5m_vs_6m generalization test.","marker":"[13]"},{"why":"Supplies the Overcooked-AI benchmark and its Coordination Ring evaluation layout.","marker":"[14]"},{"why":"Supplies the Melting Pot mixed-motive evaluation scenarios.","marker":"[15]"},{"why":"Provides the RPM baseline and inspires the Diversity Policy Pool design.","marker":"[45]"},{"why":"Provides the CSP baseline that MRDG is compared against on unseen teammate coordination.","marker":"[37]"},{"why":"Provides the ODITS baseline for online ad hoc teamwork under partial observability.","marker":"[44]"},{"why":"Motivates the use of large-scale retrieval as a way to inject new information into a reinforcement learning agent.","marker":"[48]"},{"why":"Supplies the hypernetwork mechanism used to generate the learner's policy parameters.","marker":"[50]"},{"why":"Supplies the positional encoding method used to represent agent indices and roles.","marker":"[49]"}],"fun_headline_variants":["MRDG: one agent adapts to new allies and foes via retrieval and dynamic nets","One-shot adaptation to unseen teammates and opponents via retrieval-generated policies","Agent adapts to new allies and rivals with retrieved memories and fresh weights","MRDG lets one agent model any teammate and opponent on the fly"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on the assumption that test-time teammates and opponents are drawn from the same predefined policy set that supplied the training-time Diversity Policy Pool, so the episodic memory already covers the behavioral distribution the learner must adapt to.","fun_headline_variants_meta":{"raw":{"variants":["MRDG: one agent adapts to new allies and foes via retrieval and dynamic nets","One-shot adaptation to unseen teammates and opponents via retrieval-generated policies","Agent adapts to new allies and rivals with retrieved memories and fresh weights","MRDG lets one agent model any teammate and opponent on the fly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000881,"raw_usage":{"total_tokens":3849,"prompt_tokens":1031,"completion_tokens":2818,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":2739}},"tokens_in":647,"tokens_out":2818,"duration_ms":19531,"temperature":1.0,"reasoning_tokens":2739,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:19:58.046332+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train MRDG on one predefined policy pool, then test it against partners produced independently, for example agents trained by a different MARL algorithm or human demonstrations, with no policy overlap with the pool. If the learner's performance drops to the level of RPM, CSP, or ODITS, then the reported generalization is memory interpolation rather than adaptation to genuinely new behavior.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Melting Pot mixed-motive evaluation scenarios."},{"cited_title":"RPM: Generalizable Behaviors for Multi-Agent Reinforcement Learning","cited_arxiv_id":"2210.09646","evidence_quote":"Provides the RPM baseline and inspires the Diversity Policy Pool design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CSP baseline that MRDG is compared against on unseen teammate coordination."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ODITS baseline for online ad hoc teamwork under partial observability."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the use of large-scale retrieval as a way to inject new information into a reinforcement learning agent."},{"cited_title":"Vaswani, N","cited_arxiv_id":null,"evidence_quote":"Supplies the positional encoding method used to represent agent indices and roles."}],"review_version":2}