{"id":"7277e698-d432-4a6f-9482-c90f5ee0fd37","arxiv_id":"2603.00374","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"COffeE-PSRO combines conservative uncertainty penalties with robust replicator dynamics to extract lower-regret equilibrium profiles from offline multi-agent datasets.","lead":"This paper introduces COffeE-PSRO, an offline multi-agent reinforcement learning method that uses fixed datasets and conservative uncertainty penalties to find low-regret equilibrium strategies without a live simulator. It matters because many real-world systems cannot afford online data collection, and reliable offline game-solving would let practitioners evaluate agent strategies from existing logs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline advantage rests on a tuned condition: excluding (1000,πU), COffeE+R2D is better on only 3 of 5 conditions and significant on just one; the 'tends to outperform' claim is not yet established.","rationale":"The reader's weakest_assumption focused on the calibration of the uncertainty signal ρ and the admitted non-equivalence in Eq. 4. Those are legitimate methodological concerns, but the empirical claim is comparative, and the most load-bearing condition for that claim is the fairness of the evaluation itself. The paper honestly reports limitations and does evaluate against true-game regret, which is a strength. However, tuning on a dataset that appears in the headline table, combined with uncorrected multiple testing and a single benchmark, means the central 'tends to outperform' claim is not currently supported independently of the tuning condition. This is not a rejection because the weakness is addressable with a proper holdout protocol and more benchmark games; it reinforces the reader's CONDITIONAL verdict rather than changing it.","tokens_in":15602,"tokens_out":7597,"duration_ms":81895,"concrete_test":"Run a two-part holdout protocol: (1) select (λ,α) and α_bc only on D=(500,πU) or on a newly generated dataset not listed in Table 1, and freeze parameters; (2) evaluate on the remaining five distributions and recompute the win/loss table with Benjamini-Hochberg correction across all pairwise comparisons. If COffeE+R2D no longer has at least two significant wins and no significant losses on held-out distributions, the paper should downgrade the 'tends to outperform' conclusion and present the method as conditionally useful.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical. In §5.2 the authors state that COffeE-PSRO and OEF-BC are tuned using D=(1000,πU), and the same dataset appears as a column in Table 1. The (1000,πU) column is where COffeE+R2D shows its largest, most significant win (2.12 vs 2.70–3.25; p≤.01). That column is therefore a model-selection artifact, not independent evidence. Removing it and using the Meval=R2D block of Table 1 leaves: (500,σeq) 1.96 vs best baseline 2.11 (p=.22); (500,πU) 3.98 vs 3.79 (worse); (1000,σeq) 1.22 vs 1.01 (worse); (2000,σeq) .64 vs .73 (p=.28); (2000,πU) 1.40 vs 1.93 (p=.03). So COffeE+R2D is numerically better on 3/5 conditions, significantly better on only 1, and loses on the two conditions the paper itself flags as exceptions. With 36 uncorrected Welch tests in the table, isolated p<.05 outcomes are expected by chance. The claim of 'higher consistency' therefore currently rests on a single tuned dataset plus one significant untuned condition. This is addressable by a clean holdout evaluation, but as reported the central claim is under-supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses offline equilibrium discovery in general-sum stochastic games, where an agent must identify low-regret strategy profiles from a fixed dataset of trajectories. The authors extend PSRO to the offline setting by: (1) training an ensemble of dynamics models and defining a reward-prediction spread rho (Eq. 1) to quantify uncertainty; (2) modifying the PSRO response objective (Eq. 4) to penalize uncertainty and to encourage coverage of strategic deviations; and (3) proposing a new meta-strategy solver R2D, a replicator-dynamics variant that uses pessimistic/optimistic ensemble utility bounds to estimate regret. The method, COffeE-PSRO, is evaluated on a sequential bargaining game against OEF and OEF-BC baselines under dataset sizes 500/1000/2000 and behavior policies uniform-random and equilibrium mixture. The main claimed result is that COffeE-PSRO with R2D tends to produce lower true-game regret with higher consistency than state-of-the-art baselines, with ablations examining the role of hyperparameters and model fidelity.","tokens_in":16032,"tokens_out":12973,"duration_ms":125568,"significance":"The problem is timely and the conceptual framing—treating offline game-solving as equilibrium selection under uncertainty—is useful. Extending PSRO with conservatism and a pessimistic replicator-dynamics solver is a reasonable, nontrivial contribution. The paper is transparent about limitations, including the admitted non-equivalence of the deviation-coverage surrogate in Eq. (4), and it provides a fairly detailed experimental setup. If validated on a clean holdout with appropriate multiple-comparison correction, the method would be a meaningful step for offline MARL. However, the current statistical support for the headline claim is weak: the tuning set is included in the evaluation table, the number of tests is large, and the 'true-game regret' evaluation is an approximation. The empirical contribution is therefore conditional on re-analysis.","major_comments":[{"comment":"The headline claim is not supported by the reported statistical design. The authors tune COffeE-PSRO and OEF-BC on D=(1000,πU) and then include that exact column in Table 1; its largest win (2.12±0.27 vs. 2.70–3.25, p≤.01) is thus a selection artifact. Excluding that column and using the Meval=R2D block, COffeE-PSRO+R2D is numerically better in 3 of 5 conditions, significantly better in only one (2000,πU, p=.03), and worse on (500,πU) and (1000,σeq). The table contains dozens of uncorrected pairwise Welch tests (10 rows × 6 columns = 60 tests), so isolated p<.05 results are expected by chance. The §7 conclusion that the method 'tends to produce lower regret solutions with higher consistency than SOTA' needs a clean holdout evaluation and multiple-comparison correction or explicit CI reporting.","section":"§5.2 / Table 1 / §7"},{"comment":"True-game regret is approximated by training online best responses only to the last S_eval=T/2=20 PSRO profiles and evaluating the final profile against this restricted set. This yields a lower bound on true regret, and the bias may differ across algorithms if some final profiles are harder to exploit. The main text labels these as 'true-game regret' without this qualification. Headline Table 1 should use S_eval=T (as the ablations do) or explicitly report approximate regret with a sensitivity analysis; otherwise the central comparisons may be biased.","section":"Supp. E / §5.2"},{"comment":"The R2D definition is notationally inconsistent. The text says lower and upper utility bounds are the min and max ensemble estimates, and 'respectively' maps lower→min, upper→max, then writes u_i(π)=max_j u^j_i(π); immediately after, 'pessimistic utilities u(π) are minimized over K models.' The displayed update uses a single u_i for both the candidate's own payoff and the deviator's payoff, so it is unclear whether regret uses upper or lower bounds. Since R2D is a claimed contribution, please define u_i^L and u_i^U and specify exactly which enters UBDP and UBDR; otherwise the method is not reproducible.","section":"§4.3"},{"comment":"The deviation-coverage term in Eq. (4) relies on the claim that responding to a uniform-random action policy is 'roughly equivalent' to sampling a different other-player policy each episode, but Footnote 2 concedes this does not generally hold. This equivalence is load-bearing for the α term, one of the two novel components. The paper should either state a precise condition under which the equivalence holds, or explicitly relabel α as a heuristic exploratory term without the deviation-coverage interpretation. As written, the conceptual justification for Eq. (4) is unsupported.","section":"§4.2, Eq. (4), Footnote 2"},{"comment":"The method's uncertainty signal ρ (Eq. 1) is assumed to be a calibrated proxy for model error and true-game regret. Fig. 2a supports this qualitatively but only on tuning trials from (1000,πU); no quantitative correlation or calibration measure is reported, and no such analysis is given for the other five dataset conditions. Since both the response penalty and R2D's worst-case bounds are driven by ensemble spread, the paper should report a calibration measure (e.g., correlation between ρ and δ(Π̂)) per dataset, or justify why (1000,πU) is representative. Without this, the 'conservatism' mechanism is not empirically established.","section":"§4.1 / §5.4, Fig. 2"}],"minor_comments":[{"comment":"The caption should state that Welch's t-tests are two-sided and that no multiple-comparison correction has been applied. It would help to report confidence intervals for the key comparisons.","section":"Table 1 caption"},{"comment":"The symbols M_expl and M_eval are used in Table 1 and the text but are not defined at first use. Please define them explicitly as the meta-solver used during training and the meta-solver used for evaluation.","section":"§5.2"},{"comment":"The color coding is described only vaguely ('proportional to the magnitude of regret and parameters'). Please add clear colorbars and legends; currently the reader cannot map colors to values.","section":"Fig. 2"},{"comment":"Figure 4 is actually a table of regret values. It should be renumbered as a table or reformatted as a proper figure.","section":"Fig. 4"},{"comment":"The letter T is used in Supp. E to denote the number of PSRO iterations, but T is already the game horizon in Section 3. Use S or a different symbol to avoid confusion.","section":"Supp. E"},{"comment":"No code is provided. Given the complexity of R2D and the ablations, a public implementation would substantially improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is currently under-supported because of the tuning/test overlap and the lack of multiple-comparison correction, but these issues are fixable with additional experiments or by softening the claim. The methodological novelty is real, so I would not reject on novelty grounds. Please also make the evaluation protocol (S_eval) fully transparent in the main text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe paper lands in a useful middle ground: it's a competent extension of PSRO to offline game-solving, but the headline result is not as robust as the abstract implies. The main issue is that the hyperparameters for COffeE-PSRO were tuned on the (1000,πU) dataset, and that same dataset is where the method shows its largest and most significant win. Excluding that column, the method is numerically better on only three of five remaining conditions, and most of those differences are not statistically significant. With 36 uncorrected Welch tests in Table 1, a few p<.05 outcomes are expected by chance. This is a real soft spot in the central empirical claim.\n\nWhat is genuinely new here is the combination of a conservative response objective that penalizes ensemble uncertainty (Eq. 4) with a pessimistically biased replicator dynamics meta-solver (R2D). That combination is not in the prior work, and the authors are appropriately careful to separate the effects of strategy exploration from the meta-solver. The evaluation against true-game regret via online best responses is a strength—it avoids circularity—and the model fidelity analysis in Figure 2 gives some insight into why moderate conservatism works.\n\nThe paper also acknowledges its own limitations: the deviation-coverage term in Eq. 4 is not exactly equivalent to minimizing uncertainty against all deviations (footnote 2), and the experiments are confined to a single bargaining game. Those are honest caveats, but they reinforce the picture that the empirical support is real but thin.\n\nI think this paper deserves a serious referee, but the authors need to address the tuning issue before the central claim can be taken at face value. A clean holdout evaluation—for example, tuning on one dataset and reporting all results on the others without showing the tuning column in the headline table—would go a long way. Adding a second game would also help. If those changes are made, the contribution is a reasonable subfield advance.\n\nFor your reading group, I'd say maybe—the ideas are relevant, but the main result is conditional. I wouldn't cite it yet in my own work. But I'd send it to review.","headline":"Solid incremental contribution, but the tuning artifact undercuts the headline claim; worth refereeing if the authors fix evaluation.","tokens_in":16459,"tokens_out":5583,"would_cite":false,"duration_ms":52624,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Offline multiagent equilibrium discovery is improved by steering strategy search toward dataset-supported regions and using a pessimistic regret-minimizing meta-solver.","keywords":["offline reinforcement learning","multiagent reinforcement learning","game theory","equilibrium selection","policy space response oracles","conservatism","replicator dynamics","uncertainty quantification"],"falsifier":"Run COffeE-PSRO on a game where the ensemble's rho is artificially inflated in well-covered states (e.g., by increasing ensemble seed variance while keeping the dataset fixed) and deflated in genuinely out-of-distribution states; if the method then produces higher regret than OEF, the uncertainty proxy is not calibrated and the central claim fails.","tokens_in":15514,"feed_emoji":"⚖️","tokens_out":5328,"duration_ms":51742,"temperature":0.7,"pith_summary":"This paper claims that offline multiagent equilibrium discovery can be improved by deliberately steering strategy exploration toward regions the data supports, and by using a pessimistic meta-solver that chooses profiles with low worst-case regret. The authors extend policy-space response oracles (PSRO) with a dynamics-model ensemble whose prediction spread (rho) penalizes uncertain rewards, and add a second penalty for uncertain responses to potential deviations. They also introduce R2D, a replicator-dynamics meta-solver that uses lower-bound utility estimates to minimize an upper-bound regret estimate. On a sequential bargaining game, the full method (COffeE-PSRO with R2D) tends to produce final mixed strategies with lower true-game regret than the state-of-the-art offline baselines OEF and OEF-BC across dataset sizes and behavior policies. The paper also shows that moderate conservatism beats both no conservatism and excessive conservatism, and that simple behavior-cloning conservatism wins only in the lowest-data regimes.","feed_headline":"Uncertainty penalties lower regret in offline game-solving","feed_subtitle":"PSRO plus a pessimistic meta-solver picks equilibrium candidates the data supports","key_machinery":"The central object is the ensemble dynamics model and its disagreement signal rho (Eq 1), which serves as a proxy for model uncertainty on state-action pairs. The paper uses rho in two places: (i) in the conservative response objective (Eq 4), where it penalizes reward predictions in uncertain regions and under a uniform-random opponent; and (ii) in R2D, where ensemble utility estimates are replaced by their minima to compute pessimistic regret bounds for the replicator-dynamics update. The load-bearing identity is the assumed equivalence between ensemble prediction spread and true-game uncertainty—if rho is not calibrated, both conservative components lose their grounding.","core_discovery":"The central discovery is that conservatism, previously a single-agent offline RL idea, transfers to offline game solving when applied at two distinct points: strategy generation and meta-strategy solving. In strategy generation, COffeE-PSRO replaces the online simulator with an ensemble dynamics model and augments the best-response objective with a penalty lambda*rho(state,action), where rho is the maximum reward prediction difference across ensemble members; a second term with coefficient alpha penalizes rho under a uniform-random opponent policy, which the authors argue approximates reducing uncertainty on unilateral deviations. In meta-solving, R2D computes a replicator-dynamics fixed poi","pith_inferences":["The paper's rho-based penalty could be interpreted as a form of epistemic-risk aversion; a natural extension is to adapt lambda and alpha per PSRO iteration based on the current model fidelity, rather than fixed hyperparameters.","The authors frame offline game-solving as equilibrium selection by regret probability; a testable extension is to apply the same conservative PSRO pipeline to zero-sum games, where the regret metric coincides with exploitability, to see whether the observed benefits transfer.","The uniform-random policy approximation in Eq 4 (footnote 2) suggests a concrete failure mode; replacing it with sampling from the dataset's behavior distribution or from the current strategy population could reduce the approximation gap and improve the deviation-coverage term.","The paper's model-fidelity analysis implies that COffeE-PSRO's advantage should widen as the dataset becomes more informative about equilibrium-relevant strategy spaces; one could test this by constructing datasets that mix equilibrium trajectories with random noise and measuring regret as a function of mixing proportion."],"forward_implications":["If correct, offline game-solving can be made safer without any online interaction by biasing exploration toward well-covered regions, making equilibrium extraction feasible for applications where data collection is expensive.","The R2D meta-solver can be plugged into any PSRO variant (online or offline) to extract lower-regret profiles, as its pessimistic evaluation only requires ensemble payoff estimates.","The observed tradeoff between model fidelity and regret implies that offline equilibrium discovery should tune lambda and alpha as a balance between conservatism and exploration, not simply maximize confidence.","In low-data regimes with weak behavior policies, the paper's results suggest that simple behavior-cloning conservatism is a strong baseline that strategic conservatism must beat, and that general-sum games make behavior-cloning mixtures unreliable when data contains equilibrium strategies."],"fun_headline_variants":["Offline game solving gets a pessimistic twist","Two-sided conservatism lowers regret in offline games","Pessimistic PSRO for low-regret offline equilibria","Penalty plus cautious meta-solver lower offline regret"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method's conservatism is only as good as its uncertainty signal: if rho does not reliably track true model error on unseen transitions, then both the reward penalty and the pessimistic regret bounds will steer exploration away from, rather than toward, low-regret equilibria.","fun_headline_variants_meta":{"raw":{"variants":["Offline game solving gets a pessimistic twist","Two-sided conservatism lowers regret in offline games","Pessimistic PSRO for low-regret offline equilibria","Penalty plus cautious meta-solver lower offline regret"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000583,"raw_usage":{"total_tokens":2573,"prompt_tokens":734,"completion_tokens":1839,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":1776}},"tokens_in":478,"tokens_out":1839,"duration_ms":14713,"temperature":1.0,"reasoning_tokens":1776,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T19:54:51.949326+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run COffeE-PSRO on a game where the ensemble's rho is artificially inflated in well-covered states (e.g., by increasing ensemble seed variance while keeping the dataset fixed) and deflated in genuinely out-of-distribution states; if the method then produces higher regret than OEF, the uncertainty proxy is not calibrated and the central claim fails.","supporting_citations":[],"review_version":1}