{"id":"760cc69b-e93a-4792-bd00-db3cd19c624a","arxiv_id":"2608.03644","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"For Other-Play in Yokai, agents trained with different implementation details coordinate across implementations about as well as across seeds, supporting inter-seed cross-play as a proxy for cross-implementation evaluation.","lead":"This paper tests whether the usual way of evaluating zero-shot coordination algorithms, by training one implementation with different random seeds, captures the variation between independently written implementations. For the Other-Play algorithm in the Yokai game, the answer is yes: cross-play between different implementations scored about the same as cross-play between seeds.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Arbitrary competence filter and one-at-a-time PPO-detail variation leave the no-gap result conditional: the proxy claim is not yet tested on the full implementation population or on independently written code.","rationale":"The paper is a careful empirical study with a useful new evaluation scheme, and the within-scope result (Other-Play/IPPO/Yokai, competent implementations) is credible: the XIXP matrix and aggregate numbers are consistent with no large average gap. However, the central claim is broader than the evidence. Two conditions make the no-gap result fragile: (1) the SP<5 filter excludes half the generated implementations without any reported sensitivity analysis, so the comparison set is selected post hoc; (2) the simulated implementations vary single PPO details from one codebase, not the combinatorial/independent code differences that occur in practice. The paper itself acknowledges the single-environment/algorithm limitation. These do not invalidate the paper but do mean the conclusion should remain conditional. The proposed check—threshold sensitivity plus a second, disjoint set of implementation details—would resolve whether the concern lands. I therefore keep the reader's CONDITIONAL verdict.","tokens_in":7920,"tokens_out":10820,"duration_ms":120619,"concrete_test":"Using the released code (or the Appendix Figure 4 data), recompute WIXP and XIXP for all 22 implementations and for thresholds SP≥3, 4, 5, 6, reporting a paired bootstrap 95% CI on WIXP−XIXP in each case; also report the maximum pairwise implementation gap. If the gap remains small under all thresholds and the worst pair is also small, the filter-based concern is settled. To test representativeness, run a second XIXP batch that varies a disjoint set of details (optimizer, observation normalization, hidden size, training steps) and check whether the gap stays negligible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The no-gap result is established only after discarding 11 of 22 implementations with SP<5 (§5) and only for implementations produced by varying one PPO detail at a time from a single reference codebase. Because the discarded half includes all lower-entropy variants and several architectural/hyperparameter changes, the filter could be removing the very implementations that would reveal a WIXP−XIXP gap; no threshold sensitivity analysis is reported. Moreover, the conclusion in §6 generalizes from this single environment (Yokai), single base algorithm (IPPO), and single ZSC algorithm (Other-Play) to \"the standard evaluation practice in ZSC research,\" while the paper itself lists these as open questions. The central claim would fail if real independent implementations differ in combinations of unvaried details (optimizer, observation preprocessing, training budget) or if the competence threshold is moved; the current data do not rule this out.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether the standard zero-shot coordination (ZSC) evaluation practice — training a single implementation across random seeds and using inter-seed cross-play as a proxy for cross-implementation coordination — is justified. It introduces a new evaluation scheme, cross-implementation cross-play (XIXP), in which implementation variants of the same ZSC algorithm are trained and evaluated against each other. Using Other-Play with IPPO in the Yokai environment, the authors vary nine PPO implementation details (λGAE, learning-rate schedule, gradient/value clipping, initialization, hidden layers, minibatches, discount factor, entropy coefficient, architecture). They train 176 policies across 22 implementations with 8 seeds each, discard 11 implementations whose self-play score is below 5, and report overlapping 95% confidence intervals for WIXP (4.8892, CI 4.6109–5.1675) and XIXP (4.8487, CI 4.7548–4.9427). They conclude that there is no meaningful WIXP−XIXP gap and that inter-seed cross-play is a reliable proxy for cross-implementation evaluation. The paper explicitly limits its scope to one environment, one base algorithm, and one ZSC algorithm, leaving generalization as an open question.","tokens_in":8161,"tokens_out":7166,"duration_ms":78161,"significance":"If the result holds, this is a valuable contribution: it is the first systematic empirical evaluation of a widely used but unexamined evaluation shortcut in ZSC research, and it introduces a reusable XIXP framework. The study is methodologically careful in several respects: it uses 8 seeds per implementation, adopts a CI estimator that avoids the non-independence of all-pairs cross-play, and is attentive to multimodality in cross-play scores. These are real strengths and should be credited. However, the central claim is currently supported only by an informal reading of overlapping confidence intervals and by a filtering step that removes half of the implementations without a sensitivity analysis. The paper is therefore suggestive rather than conclusive, and the breadth of the conclusion in the abstract and Section 6 exceeds what the evidence can bear.","major_comments":[{"comment":"The central inference — 'no meaningful WIXP−XIXP gap' — is based solely on overlapping 95% confidence intervals. Overlap of two CIs is not evidence of equivalence; it is compatible with a range of true differences, including ones that would undermine the 'reliable proxy' claim. The authors should report a confidence interval for the pairwise difference WIXP−XIXP, or perform a two one-sided tests (TOST) equivalence analysis against a pre-specified and justified bound. Additionally, the CI estimator is only cited to Forkel et al. (2025) and not described; without the estimator formula, the reported intervals are not reproducible from the text.","section":"Section 5, Table 1"},{"comment":"Eleven of the 22 trained implementations are discarded because their self-play score is below 5. This threshold is arbitrary and no sensitivity analysis is provided. As Figure 4 shows, the filter removes all feedforward implementations and several entropy/architecture/PPO variants; these are precisely the implementations that could exhibit a WIXP−XIXP gap if their poor self-play does not translate equally to cross-play. The conclusion is therefore conditional on an unexamined selection step. The authors should justify the threshold from the score distribution, report results for alternative thresholds (e.g., 3, 4, 5, 6), or include all 22 implementations in a supplementary analysis.","section":"Section 5, SP<5 filter (Figures 4 and 5)"},{"comment":"The implementations used in the study are generated by varying one PPO detail at a time from a single reference codebase. Independent implementations typically differ in combinations of details and in choices not varied here (optimizer details, observation preprocessing, training budgets, replay buffer, etc.). Yet the abstract and Section 6 generalize to 'the standard evaluation practice in ZSC research.' This is a large inferential leap from one environment (Yokai), one base algorithm (IPPO), and one ZSC algorithm (Other-Play). The Section 6 limitations paragraph acknowledges this, but the main claim should be proportionately restricted or supported by additional experiments with combined variations or independently written code.","section":"Sections 4.2 and 6"}],"minor_comments":[{"comment":"There is an inconsistency: Eq. (4) defines XIXP(L_k,L_k) = XP(L_k), i.e., within-implementation inter-seed cross-play, and WIXP is the average of these diagonal entries. However, the main text says the diagonal tiles in Figure 2 'represent self-play scores for each policy.' Please clarify whether the diagonal in the XIXP matrix shows self-play or within-implementation cross-play; if it is self-play, then WIXP is not computed from the displayed matrix.","section":"Equations (4)-(5) and Figure 2/5 captions"},{"comment":"The paper does not provide a full hyperparameter table, code release, or environment details needed to reproduce the 22 implementations. Given that the paper's entire argument is about implementation details, a complete list of all varied and fixed hyperparameters (optimizer, learning rate, normalizations, etc.) is essential. Also specify the score range in Yokai and the units of the SP<5 threshold.","section":"Section 4.2 and Appendix"},{"comment":"The 'estimators proposed by Forkel et al. [2025]' are central to the statistical conclusion but are not defined. Please include the estimator equations or an appendix derivation so the reader can verify the claimed independence properties.","section":"Section 4.2, CI estimator"},{"comment":"Minor language and labeling issues: Figure 3/4/5 axis labels are incomplete (e.g., x-axis labeled 'Metric'), and the duplicated implementation names in Figure 5 make it hard to count the 11 retained implementations. Please clean up the figure presentation.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central claim is plausible but currently rests on an equivalence argument that is not formally established and on a filtering step that removes half of the data without sensitivity analysis. I recommend major revision. The authorship overlaps with the cited CI estimator (Forkel et al. 2025) and the Yokai environment are not problematic per se, but the editor may wish to confirm that these resources are publicly available and that the statistical methodology is described in a self-contained way in the paper or its supplement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:2608.03644. The paper does something useful: it defines cross-implementation cross-play (XIXP) as an explicit evaluation scheme and applies it to Other-Play with IPPO in Yokai, varying a list of PPO implementation details from Huang et al. This is a step beyond earlier single-detail studies like Forkel et al. The empirical work is careful in a few respects: they use a CI estimator that respects seed independence, they deliberately train 8 seeds per implementation because they observed multimodal cross-play distributions, and they are transparent about discarding implementations with self-play below 5. Those are real contributions.\n\nThe central null result—no WIXP–XIXP gap—is plausible but not as strong as the abstract suggests. The biggest problem is the self-play threshold: 11 of 22 implementations were dropped, and we are not told how sensitive the conclusion is to moving that threshold. It is at least conceivable that the dropped implementations (which include all lower-entropy variants and several architecture/hyperparameter changes) are exactly the ones where cross-implementation differences would appear. Second, the implementations are all generated by varying one detail at a time from a single reference codebase. Independent implementers vary many things simultaneously—optimizer, preprocessing, training budget, network sizes—and interaction effects could break the null result. Third, overlapping confidence intervals on WIXP and XIXP do not establish equivalence; a formal equivalence test or a bound on the gap would be needed. Fourth, no code or data release is mentioned, which makes the whole thing hard to check.\n\nTo the paper's credit, the conclusion clearly lists these open questions, so the authors are not hiding the limitations. The abstract, though, says inter-seed cross-play is 'a reliable proxy'—that is a broad claim from one environment and one algorithm. I would want at least one more environment or algorithm, plus a sensitivity analysis for the threshold, before trusting the general claim.\n\nWho should read this: anyone working on ZSC evaluation or reproducibility of MARL training. The XIXP scheme itself is worth taking seriously. I would cite it for the evaluation framework, not for the empirical conclusion. I would send it to peer review, provided the artifacts are made available and the authors address threshold sensitivity; without that, it is a preliminary study.\n\nRecommendation: engage with it, but treat the no-gap result as conditional.","headline":"Useful XIXP evaluation framework and a carefully run null result, but the 'reliable proxy' claim outruns the evidence: unexamined threshold discards half the implementations, and one-at-a-time variation from a single codebase is not independent implementation.","tokens_in":8631,"tokens_out":2868,"would_cite":true,"duration_ms":32268,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For Other-Play, the standard zero-shot-coordination evaluation—inter-seed cross-play from a single implementation—survives variation in implementation details, with no meaningful gap to full cross-implementation cross-play.","keywords":["Zero-shot coordination","Other-Play","Independent PPO","Cross-play evaluation","Implementation details","PPO implementation details","Yokai","Multi-agent reinforcement learning"],"falsifier":"Train two or more genuinely independent implementations of Other-Play from the same specification (ideally written by separate teams, then exchanged), apply the same XIXP protocol on Yokai, and see whether the WIXP−XIXP gap stays within the overlapping-confidence-interval range; a large gap would refute the proxy. A cheaper check: add an unvaried detail such as optimizer choice (Adam vs SGD), observation preprocessing, or network width; if the resulting gap is statistically meaningful, inter-seed cross-play is not enough even within the simulated-variation setting.","tokens_in":7860,"feed_emoji":"🤝","tokens_out":8087,"duration_ms":70053,"temperature":0.7,"pith_summary":"Zero-shot coordination (ZSC) algorithms are meant to let agents trained by separate parties coordinate without any shared training. Because evaluating that ideal requires multiple independent implementations—too expensive for regular use—the field has settled for a cheaper proxy: train one implementation with many random seeds and cross-play the resulting policies. This paper asks whether that proxy is trustworthy. It builds a new evaluation scheme, cross-implementation cross-play (XIXP), that varies code-level implementation details of the underlying PPO algorithm, and runs it on Other-Play, a leading ZSC method, in the Yokai environment. The result is a null gap between within-implementation and cross-implementation cross-play, supporting the standard practice for this algorithm and benchmark.","feed_headline":"Inter-seed cross-play is a reliable proxy for Other-Play evaluation","feed_subtitle":"Varying PPO implementation details barely moves cross-play scores, so the standard ZSC check still holds.","key_machinery":"Cross-implementation cross-play (XIXP): a scheme that turns the standard single-implementation, multi-seed evaluation into a full matrix of implementations. For each pair of distinct implementations it averages cross-play over all seed pairs, and the within-implementation average (WIXP) is the traditional inter-seed cross-play score. The decisive object is the gap WIXP−XIXP; if it stays near zero, implementation details are not creating new coordination failures beyond those already seen across seeds. The paper also uses a paired standard-error estimator, in which each seed contributes exactly one inter-seed pairing per implementation, to keep confidence intervals valid.","core_discovery":"The paper's claim is that inter-seed cross-play is a reliable proxy for cross-implementation evaluation in zero-shot coordination. To support this, it defines XIXP as the average cross-play score over all policy pairs drawn from two different implementations, and WIXP as the average within-implementation cross-play score; the gap WIXP−XIXP is the measure of implementation-induced miscoordination. The authors train 22 implementations of Other-Play with IPPO in the Yokai environment, varying λGAE, learning-rate scheduling, gradient and value-function clipping, weight initialization, hidden-layer counts, minibatch counts, discount factor, entropy coefficient, and network architecture. After dis","pith_inferences":["Because the paper varies only a curated list of PPO details, the representative-details assumption is a testable extension: compare XIXP against implementations written truly independently from the same specification, and if a gap shows up, the null result is limited to simulated variation.","If the null result generalizes to other ZSC algorithms, it would suggest that the symmetry-avoiding design principle behind Other-Play also absorbs implementation noise, making algorithmic robustness an emergent property rather than a separate engineering concern.","The WIXP−XIXP gap could become a standard regression metric in ZSC research—reported alongside seed counts—so that a claimed ZSC algorithm is judged on how much its coordination survives code-level variation, not just seed variation.","A natural extension is to run the same protocol on Off-Belief Learning and Q-learning variants, since the paper explicitly leaves those open; finding a large gap there would map the boundary of the proxy's validity."],"forward_implications":["ZSC papers can keep using inter-seed cross-play as the primary evaluation without commissioning multiple independent implementations, at least for Other-Play-style algorithms on benchmarks where the competence filter is applied.","The XIXP protocol—generate implementations, filter by self-play competence, compare WIXP to XIXP—gives future work a concrete template for testing whether a new ZSC algorithm is more or less sensitive to implementation details.","Nearly half of the generated implementations (11 of 22) failed the self-play competence threshold, so the proxy only holds once implementations are competent; the null result does not license skipping quality control.","The seed-pairing standard-error estimator shows how a small seed count can distort conclusions; adopting it would make reported confidence intervals in ZSC papers more honest.","The finding gives an empirical precedent for treating Other-Play as robust to specification ambiguity in the Yokai environment, shifting the burden of proof onto claims that implementation details do break coordination."],"supporting_citations":[{"why":"Supplies Other-Play, the zero-shot coordination algorithm whose robustness is being evaluated.","marker":"Hu et al. [2020]"},{"why":"Formalizes zero-shot coordination and defines the cross-play evaluation that the paper tests as a proxy.","marker":"Treutlein et al. [2021]"},{"why":"Provides the list of PPO implementation details the paper varies to simulate independent implementations.","marker":"Huang et al. [2022]"},{"why":"Showed that entropy and λGAE affect IPPO cross-play, motivating the study, and supplies the standard-error estimator used for confidence intervals.","marker":"Forkel et al. [2025]"},{"why":"Introduces the Yokai learning environment, the benchmark used for all training and evaluation.","marker":"Ruhdorfer et al. [2026]"},{"why":"Defines PPO, the base algorithm whose code-level details are varied across implementations.","marker":"Schulman et al. [2017]"},{"why":"Highlights design decisions needed to adapt PPO to cooperative multi-agent settings, informing which details matter.","marker":"Yu et al. [2022]"},{"why":"Introduces IPPO, the independent PPO variant used as the base MARL algorithm.","marker":"Witt et al. [2020]"}],"fun_headline_variants":["Inter-seed cross-play holds up for Other-Play evaluation","Standard ZSC check remains valid for Other-Play","Inter-seed cross-play scores match cross-implementation","Other-Play survives implementation variation in ZSC","Implementation tweaks don't break Other-Play's ZSC scores"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the curated list of varied implementation details (λGAE, clipping, initialization, architecture, etc.) faithfully represents the spread of genuinely independent implementations—if independent codebases differ in ways not covered here, the null result may not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Inter-seed cross-play holds up for Other-Play evaluation","Standard ZSC check remains valid for Other-Play","Inter-seed cross-play scores match cross-implementation","Other-Play survives implementation variation in ZSC","Implementation tweaks don't break Other-Play's ZSC scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000953,"raw_usage":{"total_tokens":3906,"prompt_tokens":754,"completion_tokens":3152,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":3086}},"tokens_in":498,"tokens_out":3152,"duration_ms":25446,"temperature":1.0,"reasoning_tokens":3086,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:19:27.597809+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two or more genuinely independent implementations of Other-Play from the same specification (ideally written by separate teams, then exchanged), apply the same XIXP protocol on Yokai, and see whether the WIXP−XIXP gap stays within the overlapping-confidence-interval range; a large gap would refute the proxy. A cheaper check: add an unvaried detail such as optimizer choice (Adam vs SGD), observation preprocessing, or network width; if the resulting gap is statistically meaningful, inter-seed cross-play is not enough even within the simulated-variation setting.","supporting_citations":[],"review_version":1}