{"id":"3c89268d-fc4e-43e3-9387-ba396d75e2ed","arxiv_id":"2505.11100","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"BiDist trains MARL agents to handle unseen co-players by using forward and reverse distillation to generate diverse partner behaviors without storing past policies.","lead":"This paper introduces BiDist, a multi-agent reinforcement learning training scheme that alternates between distilling background agents toward the current team behavior and pushing them away from it. The goal is to make trained agents perform well when paired with never-seen co-players, a problem self-play methods handle poorly.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.4's generalization bound is not established for BiDist: it conflates loss at sampled actions with loss at a*, assumes a well-defined optimal action in MARL, and never proves reverse distillation reduces δ.","rationale":"The reader's weakest assumption identifies Theorem 4.4 as the key theoretical weak point. I agree and sharpen it: beyond the ill-defined optimal action, the proof conflates policy-sampled actions with ground-truth actions, so the stated bound is not even derived for the quantity it claims to bound. This is load-bearing because the paper's stated contribution includes theoretical analysis, and the introduction explicitly claims reverse distillation reduces δ and hence the generalization error bound. Without this theorem, the paper is left with an empirical method and ablations but no formal account of why mixed-play helps. I do not escalate to REJECT: the empirical pattern (BiDist best in all 25 normalized scenarios, ablations showing both forward and reverse distillation matter, Proposition 4.1's preference-shift intuition, and the t-SNE diversity evidence) is coherent and could be true. But the theory needs repair or explicit downgrading to a heuristic motivation. Missing error bars and hyperparameter selection on evaluation tasks (Appendix C.2, Figure C.2) are additional reporting concerns but secondary to the invalid theoretical link. Therefore the reader's CONDITIONAL verdict remains appropriate.","tokens_in":26529,"tokens_out":9840,"duration_ms":104842,"concrete_test":"Independently re-derive Theorem 4.4 from Definition 4.2, replacing every occurrence of a_i in Appendix A.3 with a*_i, and attempt to satisfy the resulting assumptions on a minimal two-agent Dec-POMDP where the background policy is drawn from two different pre-trained policies. If the bound cannot be derived, or if no single a*_i and δ-cover exist for the training distribution BiDist actually generates, then the theorem does not justify the δ-reduction claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4's Theorem 4.4 is the paper's only formal justification that reducing δ improves zero-shot generalization, and the authors state BiDist 'efficiently reduces δ' (Section 1). Two gaps make this unsupported. First, the proof does not track the quantity the theorem claims to bound. Definition 4.2 defines l(o_i,a*_i; ΦP) with a*_i the ground-truth optimal action, and Theorem 4.4 bounds E_{o_i,a*_i∼P_Z'}[l(o_i,a*_i; ΦP)] under the assumption l=0 on the training distribution P. But the proof of Theorem 4.4 (Appendix A.3, Eq. A.20) instead bounds E_{a_i∼π_i(·|o_i)}[l(o_i,a_i; ΦP)] — the loss evaluated at actions sampled from the learned policy. No step connects l(o_i,a_i) to l(o_i,a*_i); if a_i is a one-hot action, these are different functions. Second, the theorem's premise that a single a*_i exists for each observation is not satisfied in the zero-shot co-player setting: the optimal action for agent i depends on the joint policy of the unseen background population, which varies across scenarios. The paper never defines a distribution over co-player policies for which P is a δ-cover, nor proves that reverse distillation decreases δ. Thus the theoretical pillar advertised in the abstract and Section 4.4 does not actually support the mechanism claimed; the empirical results would need to stand alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Bidirectional Distillation (BiDist), a training framework for population-population generalization in multi-agent reinforcement learning. During training, a random subset of agents is designated as a 'fictitious' population that uses distilled policies. Forward distillation periodically distills the learning policies into the fictitious population to emulate self-play without storing past policies; reverse distillation pushes the distilled policies away via KL-divergence maximization to induce preference shifts and explore behavior outside the historical policy space. The learning agents are trained with MAPPO against this fictitious population. The paper claims that this mixed-play procedure improves zero-shot generalization to unseen co-players, supports the claim with a theoretical analysis based on a δ-cover covering radius (Theorem 4.4), and reports experiments on five Melting Pot substrates (Pure Coordination, Coop Mining, Chicken, Coins, Prisoners Dilemma) with baselines MAPPO, RanNet, OPRE, PP, and RPM. The central empirical claim is that BiDist attains the highest min-max normalized focal per-capita return in nearly all reported scenarios.","tokens_in":26839,"tokens_out":6635,"duration_ms":59800,"significance":"If substantiated, BiDist is a valuable practical contribution: it replaces explicit policy pools with a single distilled network, is algorithm-agnostic (instantiated with MAPPO), and is evaluated across cooperative, competitive, and social-dilemma settings. The paper includes ablation studies (Section 5.5, Figure 8), a perturbation comparison validating the choice of KL maximization (Section 5.4), and resource-efficiency experiments (Appendix C), which are genuine strengths. The training procedure itself is not circular: the distilled policies act as legitimate co-players generated from the learning policies. However, the theoretical argument advertised in the abstract and Section 1 is not established: the proof of Theorem 4.4 does not track the quantity it claims to bound, and the paper does not prove that reverse distillation reduces δ. The empirical evidence also lacks error bars and significance tests. The core algorithmic idea is plausible and worth pursuing, but the manuscript currently overstates the support it provides.","major_comments":[{"comment":"Theorem 4.4 states a bound on E_{o_i,a*_i∼P_Z'}[l(o_i,a*_i; Φ_P)], but the proof bounds E_{a_i∼π_i(·|o_i)}[l(o_i,a_i; Φ_P)]. The step labeled Hoeffding (Eq. A.24) compares these two different expectations under different measures, which is not a valid application of Hoeffding's inequality. Moreover, the proof uses the zero-loss assumption l(o_j,a_i; Φ_P)=0 for all actions a_i for o_j∈P, while Definition 4.2 defines l only at the single ground-truth optimal action a*_i. The stronger assumption is never stated, so the proof is incomplete even for the surrogate quantity.","section":"Section 4.4; Appendix A.3, Eqs. (A.20)-(A.25)"},{"comment":"The theorem assumes a well-defined ground-truth optimal action a*_i for each observation. In the zero-shot co-player setting formalized in Section 3, the optimal action for agent i depends on the joint policy of the unseen background population g, which varies across testing scenarios Z'. The manuscript does not define a distribution over co-player policies for which P is a δ-cover of P_Z', nor does it justify that such an a*_i exists. Consequently, the application of Theorem 4.4 to BiDist is not established.","section":"Definition 4.2; Section 3"},{"comment":"The paper's central claim that BiDist 'efficiently reduces δ' is not proven. The discussion after Theorem 4.4 and the schematic in Figure 2 are qualitative; no formal statement connects the forward and reverse distillation updates to a decrease in the covering radius δ. Without a proof or at least a quantitative characterization of how the fictitious population generated by BiDist relates to P_Z', the theoretical mechanism advertised in the abstract remains unsupported.","section":"Section 1; Section 4.4"},{"comment":"The primary empirical result reports min-max normalized focal per-capita returns without any standard deviations, confidence intervals, or significance tests. Min-max normalization within each scenario can exaggerate small raw differences, and the near-universal score of 1.00 for BiDist across scenarios may in part be an artifact of this normalization. The paper should report unnormalized returns with error bars across the three seeds mentioned in Appendix B.3, and provide statistical comparisons to the strongest baseline.","section":"Table 1; Section 5.2"}],"minor_comments":[{"comment":"The legend text appears to contain missing symbols (the markers are described as 'denotes' without visible glyphs). Please ensure the figure renders correctly in the final PDF.","section":"Figure 2"},{"comment":"The definition of the sampling distribution 𝔅(v_<i; p,N) is confusing, particularly the role of δ_i; please define the joint distribution over v explicitly.","section":"Equations (3)-(5)"},{"comment":"The notation θ←arg max_θ J(θ) is inconsistent with the clipped PPO objective in Eq. (9); please describe the actual parameter update step.","section":"Algorithm 1, Line 8"},{"comment":"'Distillation internal' should be 'Distillation interval'.","section":"Table B.2"},{"comment":"The text contains broken Unicode glyphs in the figure captions (e.g., '0/uni...' sequences); please check the source files.","section":"Appendix C"},{"comment":"The 70%/30% sampling probabilities for the PP baseline are stated without justification; citing the source or adding a sensitivity note would help.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The theoretical portion of the manuscript is the main obstacle. If the authors cannot repair Theorem 4.4 within a revision, I would recommend allowing them to remove the theoretical claims and reframe the contribution as an empirical method; as written, the gap between claims and proofs makes the paper unsuitable for publication without significant changes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zhen,\n\nThis one is worth reading if you care about zero-shot co-player generalization in MARL. The core idea is simple: keep a separate 'distilled' policy network, and alternate between distilling it from the current learning policies (which acts like implicit self-play) and pushing it away by KL maximization. That gives you a co-player population without storing a history of policies. As far as I can tell, that combination is new. Prior methods like RPM, PP, or OPRE either keep explicit pools or rely on entropy-based diversity. The empirical section also does the right kind of coverage: cooperation, competition, and social dilemma tasks in Melting Pot. The ablations show that both forward and reverse directions matter, and the single-network resource story is a genuine advantage over policy-pool methods.\n\nBut there are two soft spots that keep me from taking the headline numbers at face value.\n\nFirst, the experimental reporting. Table 1 reports min-max normalized returns with no error bars, no standard deviations, and no raw scores. BiDist scores 1.0 or close to it in every scenario. That could be a real and consistent win, or it could be an artifact of normalization when the absolute differences are small. The text says results are averaged over 3 seeds, but without variance we can't distinguish a robust method from a lucky run. This is fixable and necessary.\n\nSecond, the theory. Theorem 4.4 is advertised as showing that reducing the covering radius delta lowers the generalization error bound. But the proof in Appendix A.3 bounds expectation over actions sampled from the learned policy, while the theorem statement concerns the ground-truth optimal action a*_i. Nothing connects those two quantities. On top of that, in the zero-shot co-player setting the 'optimal action' is not a well-defined function of the observation alone—it depends on the background population's policy. So the theorem doesn't actually apply to this problem. The claim that BiDist reduces delta is asserted from a schematic, not proved. My take: the theory is not load-bearing. The method might still work for reasons this theorem doesn't capture, and the authors should either fix the proof or honestly label the theoretical section as intuition.\n\nThe central method itself is coherent and not circular. The hyperparameters include some hand-tuning (distillation interval chosen on one task), but that is minor.\n\nWho's this for? Anyone working on population-based training or zero-shot coordination. It deserves a serious referee—send it to review—but I would ask for major revision: add error bars and raw returns, and either repair or demote the theoretical analysis. As it stands, the empirical claim is plausible but not yet demonstrated to the level the paper asserts.","headline":"A plausible new training trick with an overclaimed theory and thin experimental reporting—worth a serious look, but not as-is.","tokens_in":27349,"tokens_out":3442,"would_cite":true,"duration_ms":36811,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BiDist claims that training against a fictitious population updated by forward and reverse distillation lets multi-agent policies generalize to unseen co-players without storing past policies.","keywords":["multi-agent reinforcement learning","population-population generalization","unseen co-players","mixed-play framework","knowledge distillation","self-play","zero-shot coordination","Melting Pot"],"falsifier":"Train BiDist on a substrate, then test against a background population whose preferred action is the one farthest from the action the reverse-distilled fictitious population shifted toward. If the focal population's per-capita return in that scenario collapses to the MAPPO level, the reverse step has not actually covered outside-space preferences and the central generalization claim is falsified.","tokens_in":26302,"feed_emoji":"🎭","tokens_out":10716,"duration_ms":106783,"temperature":0.7,"pith_summary":"BiDist proposes a way to train a team of reinforcement-learning agents that keeps performing well when matched with co-players it has never seen. The method randomly detaches a subset of agents into a fictitious population whose policies are distilled copies of the learning policies, then updates those copies in two alternating directions: forward distillation pulls them toward the current learned policies, and reverse distillation pushes them away. The authors argue that this mix reproduces the benefit of self-play without storing a policy library, while also exploring interactions outside the historical policy space. On five Melting Pot tasks spanning cooperation, competition, and social dilemmas, BiDist attains the top normalized focal per-capita return in 22 of 25 scenarios and near-top performance in the rest, supported by a δ-cover generalization bound that ties smaller training-distribution covering radius to smaller test loss.","feed_headline":"Two-way distillation beats self-play at unseen co-players","feed_subtitle":"BiDist trains agents against a shifted fictitious population and outperforms policy-pool self-play on five Melting Pot tasks.","key_machinery":"Random fictitious population with alternating forward and reverse distillation. A fictitious population is a subset of agents randomly detached from the training population at each iteration; those agents execute distilled policies $\\pi_\\phi$ instead of the learning policies $\\pi_\\theta$ while collecting trajectories in the substrate. Every $k_d$ iterations, the distilled parameters $\\phi$ are updated by one step of forward distillation ($\\phi \\leftarrow \\phi - \\eta_f \\nabla_\\phi L_{\\mathrm{KL}}$, minimizing KL to the learning policies) or one step of reverse distillation ($\\phi \\leftarrow \\phi + \\eta_r \\nabla_\\phi L_{\\mathrm{KL}}$, maximizing KL). The alternating schedule makes the distilled policies lag behind the learning policies, which supplies implicit self-play coverage, and then pushes them away, which supplies outside-space exploration. The theoretical carrier is Theorem 4.4: if the training distribution $P$ is a $\\delta$-cover of the testing distribution $P_{\\mathcal{Z}'}$ and the loss is $\\lambda$-Lipschitz and zero on $P$, then $\\mathbb{E}_{o_i,a^*_i\\sim P_{\\mathcal{Z}'}}[l(o_i,a^*_i;\\Phi_P)] \\le \\delta\\lambda(1+L|\\mathcal{A}_i|) + \\sqrt{\\frac{L^2\\log(1/\\gamma)}{2n}}$ with probability at least $1-\\gamma$; this ties smaller $\\delta$ to smaller worst-case test loss, and BiDist's reverse phase is what reduces $\\delta$ beyond self-play.","core_discovery":"The paper's central claim is that a multi-agent population can learn to interact well with unseen co-players by training against a 'fictitious population' whose policies are distilled copies of the learning agents, updated in two alternating directions. Forward distillation (minimizing KL divergence from the learning policies) recreates the coverage that self-play obtains from storing historical policies, without the storage cost; reverse distillation (maximizing KL divergence) shifts the distilled policies' preferred actions so that the learning agents encounter partners whose behavior lies outside the historical policy space. The authors show that this preference shift is the key difference from entropy maximization, randomization, and noise injection, and they give a δ-cover generalization bound in which reducing the covering radius δ of the training distribution tightens the guarantee on the testing distribution. Empirically, BiDist attains the maximum normalized focal per-capita return in 22 of the 25 tested Melting Pot scenarios and stays near the top in the remaining three, with t-SNE visualizations showing its joint action distribution spreading well beyond the clusters produced by self-play baselines.","pith_inferences":["A testable extension would apply the same forward/reverse distillation loop to human-AI coordination, where the reverse step could be steered toward plausible human biases rather than arbitrary preference shifts; the paper only evaluates pre-trained background populations.","One measurable prediction of the δ-cover argument is that the empirical covering radius of the joint action distribution should shrink as BiDist training proceeds and zero-shot returns rise, giving a cheap diagnostic that does not require running the test scenarios.","The ablation result that forward distillation alone slightly surpasses RPM hints that distillation error itself may add useful diversity; isolating a perfect-copy baseline would separate the effect of lagged policies from the effect of distillation noise.","Because reverse distillation uses a small fixed learning rate, a natural next step is a schedule for $\\eta_r$ that grows the preference shift over training, potentially covering farther outside-space behaviors without generating infeasible policies."],"forward_implications":["Zero-shot evaluation against unseen background populations is improved without storing past policies; one distilled network per agent is enough.","Any MARL algorithm that samples policies from a mixed population can carry BiDist, so the benefit is not tied to the MAPPO instantiation.","The δ-cover analysis says that whatever shrinks the covering radius of the training distribution tightens the generalization guarantee, placing BiDist in a broader family of coverage-reducing training schemes.","Reverse distillation's KL maximization is qualitatively different from entropy maximization, randomization, or noise injection because it targets the identity of the most-preferred action.","Both distillation directions are necessary: removing either forward or reverse distillation degrades performance, so implicit self-play retention and outside-space exploration each contribute."],"supporting_citations":[{"why":"Supplies the Melting Pot 2.0 substrate and test-scenario definitions used for the zero-shot co-player evaluations.","marker":"[1]"},{"why":"Defines the focal/background population-population generalization task and the evaluation protocol that frames the paper's training and testing setup.","marker":"[30]"},{"why":"Contributes the knowledge-distillation KL objective that forward and reverse distillation are built on.","marker":"[19]"},{"why":"Provides the ranked-policy-memory self-play baseline and the shared network architecture and hyperparameters used across algorithms.","marker":"[39]"},{"why":"Establishes self-play with historical policies, the inside-space approach that BiDist contrasts with and extends.","marker":"[48]"},{"why":"Defines δ-cover, the coverage concept at the center of the generalization error bound.","marker":"[43]"},{"why":"Motivates using the δ-cover radius as a control on the generalization bound via the core-set perspective.","marker":"[47]"},{"why":"Supplies the OPRE hierarchical baseline that BiDist is compared against on unseen-opponent generalization.","marker":"[54]"},{"why":"Supplies the RanNet randomization baseline used to test whether input perturbation alone improves co-player generalization.","marker":"[29]"},{"why":"Supplies MAPPO, the multi-agent PPO algorithm on which BiDist is instantiated for the policy updates.","marker":"[61]"}],"fun_headline_variants":["Two-way distillation beats self-play at unseen co-players","BiDist: distill both ways, beat unseen co-players","No policy storage? Two-way distillation still wins","Mixed-play distillation generalizes beyond self-play limits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof assumes each observation has one ground-truth optimal action and that the loss is exactly zero on the training distribution; in zero-shot multi-agent tasks the best action depends on the unseen co-players' joint behavior, so that ground-truth action is not well defined for the generalization scenario.","fun_headline_variants_meta":{"raw":{"variants":["Two-way distillation beats self-play at unseen co-players","BiDist: distill both ways, beat unseen co-players","No policy storage? Two-way distillation still wins","Mixed-play distillation generalizes beyond self-play limits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1356,"prompt_tokens":960,"completion_tokens":396,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":332}},"tokens_in":576,"tokens_out":396,"duration_ms":4118,"temperature":1.0,"reasoning_tokens":332,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:57:09.950810+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train BiDist on a substrate, then test against a background population whose preferred action is the one farthest from the action the reverse-distilled fictitious population shifted toward. If the focal population's per-capita return in that scenario collapses to the MAPPO level, the reverse step has not actually covered outside-space preferences and the central generalization claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the focal/background population-population generalization task and the evaluation protocol that frames the paper's training and testing setup."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ranked-policy-memory self-play baseline and the shared network architecture and hyperparameters used across algorithms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines δ-cover, the coverage concept at the center of the generalization error bound."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the OPRE hierarchical baseline that BiDist is compared against on unseen-opponent generalization."}],"review_version":1}