Pith. sign in

REVIEW 5 major objections 6 minor 35 references

Offline multiagent equilibrium discovery is improved by steering strategy search toward dataset-supported regions and using a pessimistic regret-minimizing meta-solver.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

COffeE-PSRO combines conservative uncertainty penalties with robust replicator dynamics to extract lower-regret equilibrium profiles from offline multi-agent datasets.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection Solid incremental contribution, but the tuning artifact undercuts the headline claim; worth refereeing if the authors fix evaluation. the 5 major comments →

arxiv 2603.00374 v2 pith:XT445SZJ submitted 2026-02-27 cs.AI cs.MA

Conservative Equilibrium Discovery in Offline Game-Theoretic Multiagent Reinforcement Learning

Austin A. Nguyen , Michael P. Wellman This is my paper
classification cs.AI cs.MA
keywords offline reinforcement learningmultiagent reinforcement learninggame theoryequilibrium selectionpolicy space response oraclesconservatismreplicator dynamicsuncertainty quantification
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that offline multiagent equilibrium discovery can be improved by deliberately steering strategy exploration toward regions the data supports, and by using a pessimistic meta-solver that chooses profiles with low worst-case regret. The authors extend policy-space response oracles (PSRO) with a dynamics-model ensemble whose prediction spread (rho) penalizes uncertain rewards, and add a second penalty for uncertain responses to potential deviations. They also introduce R2D, a replicator-dynamics meta-solver that uses lower-bound utility estimates to minimize an upper-bound regret estimate. On a sequential bargaining game, the full method (COffeE-PSRO with R2D) tends to produce final mixed strategies with lower true-game regret than the state-of-the-art offline baselines OEF and OEF-BC across dataset sizes and behavior policies. The paper also shows that moderate conservatism beats both no conservatism and excessive conservatism, and that simple behavior-cloning conservatism wins only in the lowest-data regimes.

Core claim

The central discovery is that conservatism, previously a single-agent offline RL idea, transfers to offline game solving when applied at two distinct points: strategy generation and meta-strategy solving. In strategy generation, COffeE-PSRO replaces the online simulator with an ensemble dynamics model and augments the best-response objective with a penalty lambda*rho(state,action), where rho is the maximum reward prediction difference across ensemble members; a second term with coefficient alpha penalizes rho under a uniform-random opponent policy, which the authors argue approximates reducing uncertainty on unilateral deviations. In meta-solving, R2D computes a replicator-dynamics fixed poi

What carries the argument

The central object is the ensemble dynamics model and its disagreement signal rho (Eq 1), which serves as a proxy for model uncertainty on state-action pairs. The paper uses rho in two places: (i) in the conservative response objective (Eq 4), where it penalizes reward predictions in uncertain regions and under a uniform-random opponent; and (ii) in R2D, where ensemble utility estimates are replaced by their minima to compute pessimistic regret bounds for the replicator-dynamics update. The load-bearing identity is the assumed equivalence between ensemble prediction spread and true-game uncertainty—if rho is not calibrated, both conservative components lose their grounding.

Load-bearing premise

The method's conservatism is only as good as its uncertainty signal: if rho does not reliably track true model error on unseen transitions, then both the reward penalty and the pessimistic regret bounds will steer exploration away from, rather than toward, low-regret equilibria.

What would settle it

Run COffeE-PSRO on a game where the ensemble's rho is artificially inflated in well-covered states (e.g., by increasing ensemble seed variance while keeping the dataset fixed) and deflated in genuinely out-of-distribution states; if the method then produces higher regret than OEF, the uncertainty proxy is not calibrated and the central claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If correct, offline game-solving can be made safer without any online interaction by biasing exploration toward well-covered regions, making equilibrium extraction feasible for applications where data collection is expensive.
  • The R2D meta-solver can be plugged into any PSRO variant (online or offline) to extract lower-regret profiles, as its pessimistic evaluation only requires ensemble payoff estimates.
  • The observed tradeoff between model fidelity and regret implies that offline equilibrium discovery should tune lambda and alpha as a balance between conservatism and exploration, not simply maximize confidence.
  • In low-data regimes with weak behavior policies, the paper's results suggest that simple behavior-cloning conservatism is a strong baseline that strategic conservatism must beat, and that general-sum games make behavior-cloning mixtures unreliable when data contains equilibrium strategies.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's rho-based penalty could be interpreted as a form of epistemic-risk aversion; a natural extension is to adapt lambda and alpha per PSRO iteration based on the current model fidelity, rather than fixed hyperparameters.
  • The authors frame offline game-solving as equilibrium selection by regret probability; a testable extension is to apply the same conservative PSRO pipeline to zero-sum games, where the regret metric coincides with exploitability, to see whether the observed benefits transfer.
  • The uniform-random policy approximation in Eq 4 (footnote 2) suggests a concrete failure mode; replacing it with sampling from the dataset's behavior distribution or from the current strategy population could reduce the approximation gap and improve the deviation-coverage term.
  • The paper's model-fidelity analysis implies that COffeE-PSRO's advantage should widen as the dataset becomes more informative about equilibrium-relevant strategy spaces; one could test this by constructing datasets that mix equilibrium trajectories with random noise and measuring regret as a function of mixing proportion.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper addresses offline equilibrium discovery in general-sum stochastic games, where an agent must identify low-regret strategy profiles from a fixed dataset of trajectories. The authors extend PSRO to the offline setting by: (1) training an ensemble of dynamics models and defining a reward-prediction spread rho (Eq. 1) to quantify uncertainty; (2) modifying the PSRO response objective (Eq. 4) to penalize uncertainty and to encourage coverage of strategic deviations; and (3) proposing a new meta-strategy solver R2D, a replicator-dynamics variant that uses pessimistic/optimistic ensemble utility bounds to estimate regret. The method, COffeE-PSRO, is evaluated on a sequential bargaining game against OEF and OEF-BC baselines under dataset sizes 500/1000/2000 and behavior policies uniform-random and equilibrium mixture. The main claimed result is that COffeE-PSRO with R2D tends to produce lower true-game regret with higher consistency than state-of-the-art baselines, with ablations examining the role of hyperparameters and model fidelity.

Significance. The problem is timely and the conceptual framing—treating offline game-solving as equilibrium selection under uncertainty—is useful. Extending PSRO with conservatism and a pessimistic replicator-dynamics solver is a reasonable, nontrivial contribution. The paper is transparent about limitations, including the admitted non-equivalence of the deviation-coverage surrogate in Eq. (4), and it provides a fairly detailed experimental setup. If validated on a clean holdout with appropriate multiple-comparison correction, the method would be a meaningful step for offline MARL. However, the current statistical support for the headline claim is weak: the tuning set is included in the evaluation table, the number of tests is large, and the 'true-game regret' evaluation is an approximation. The empirical contribution is therefore conditional on re-analysis.

major comments (5)
  1. [§5.2 / Table 1 / §7] The headline claim is not supported by the reported statistical design. The authors tune COffeE-PSRO and OEF-BC on D=(1000,πU) and then include that exact column in Table 1; its largest win (2.12±0.27 vs. 2.70–3.25, p≤.01) is thus a selection artifact. Excluding that column and using the Meval=R2D block, COffeE-PSRO+R2D is numerically better in 3 of 5 conditions, significantly better in only one (2000,πU, p=.03), and worse on (500,πU) and (1000,σeq). The table contains dozens of uncorrected pairwise Welch tests (10 rows × 6 columns = 60 tests), so isolated p<.05 results are expected by chance. The §7 conclusion that the method 'tends to produce lower regret solutions with higher consistency than SOTA' needs a clean holdout evaluation and multiple-comparison correction or explicit CI reporting.
  2. [Supp. E / §5.2] True-game regret is approximated by training online best responses only to the last S_eval=T/2=20 PSRO profiles and evaluating the final profile against this restricted set. This yields a lower bound on true regret, and the bias may differ across algorithms if some final profiles are harder to exploit. The main text labels these as 'true-game regret' without this qualification. Headline Table 1 should use S_eval=T (as the ablations do) or explicitly report approximate regret with a sensitivity analysis; otherwise the central comparisons may be biased.
  3. [§4.3] The R2D definition is notationally inconsistent. The text says lower and upper utility bounds are the min and max ensemble estimates, and 'respectively' maps lower→min, upper→max, then writes u_i(π)=max_j u^j_i(π); immediately after, 'pessimistic utilities u(π) are minimized over K models.' The displayed update uses a single u_i for both the candidate's own payoff and the deviator's payoff, so it is unclear whether regret uses upper or lower bounds. Since R2D is a claimed contribution, please define u_i^L and u_i^U and specify exactly which enters UBDP and UBDR; otherwise the method is not reproducible.
  4. [§4.2, Eq. (4), Footnote 2] The deviation-coverage term in Eq. (4) relies on the claim that responding to a uniform-random action policy is 'roughly equivalent' to sampling a different other-player policy each episode, but Footnote 2 concedes this does not generally hold. This equivalence is load-bearing for the α term, one of the two novel components. The paper should either state a precise condition under which the equivalence holds, or explicitly relabel α as a heuristic exploratory term without the deviation-coverage interpretation. As written, the conceptual justification for Eq. (4) is unsupported.
  5. [§4.1 / §5.4, Fig. 2] The method's uncertainty signal ρ (Eq. 1) is assumed to be a calibrated proxy for model error and true-game regret. Fig. 2a supports this qualitatively but only on tuning trials from (1000,πU); no quantitative correlation or calibration measure is reported, and no such analysis is given for the other five dataset conditions. Since both the response penalty and R2D's worst-case bounds are driven by ensemble spread, the paper should report a calibration measure (e.g., correlation between ρ and δ(Π̂)) per dataset, or justify why (1000,πU) is representative. Without this, the 'conservatism' mechanism is not empirically established.
minor comments (6)
  1. [Table 1 caption] The caption should state that Welch's t-tests are two-sided and that no multiple-comparison correction has been applied. It would help to report confidence intervals for the key comparisons.
  2. [§5.2] The symbols M_expl and M_eval are used in Table 1 and the text but are not defined at first use. Please define them explicitly as the meta-solver used during training and the meta-solver used for evaluation.
  3. [Fig. 2] The color coding is described only vaguely ('proportional to the magnitude of regret and parameters'). Please add clear colorbars and legends; currently the reader cannot map colors to values.
  4. [Fig. 4] Figure 4 is actually a table of regret values. It should be renumbered as a table or reformatted as a proper figure.
  5. [Supp. E] The letter T is used in Supp. E to denote the number of PSRO iterations, but T is already the game horizon in Section 3. Use S or a different symbol to avoid confusion.
  6. [General] No code is provided. Given the complexity of R2D and the ablations, a public implementation would substantially improve reproducibility.

Circularity Check

0 steps flagged

No significant circularity: the headline regret claim is evaluated by independent true-game best responses, and neither the conservative objective nor R2D reduces to its inputs by construction.

full rationale

The central claim is that COffeE-PSRO+R2D 'tends to produce lower regret solutions with higher consistency than SOTA approaches' (Section 7). This is an empirical claim, and the paper's evaluation metric is independently defined: true-game regret is computed by training online best responses to the output profiles and evaluating them in the true game (Supp. E), not by the ensemble model or by the fitted penalties. Thus the headline result is not equivalent to the method's own inputs. The conservative response objective (Eqs. 3-4) and the R2D meta-solver are presented as heuristics built on external methods (PSRO, MOPO, BCEL, replicator dynamics), with explicit disclaimers that R2D has 'no theoretical convergence guarantees' and that the Eq. 4 reformulation is only 'roughly equivalent' and 'does not generally hold' (footnote 2). These are acknowledged approximations, not circular definitions. Self-citations to prior empirical-game work (e.g., [6,22,26,27,28]) are contextual and not load-bearing; no uniqueness theorem or ansatz is imported solely from the authors' own prior work to force the design. The paper also states its own limitation that COffeE-PSRO provides no benefit when data lack sufficient information, which is a falsifiable boundary condition rather than a circular rescue. The tuning of hyperparameters on D=(1000,πU) and then including that dataset in Table 1 is a statistical validation weakness, but it is not circularity in the derivation sense: the reported regrets are computed independently, and the paper does not claim to predict that column from the fitted values. Concerns about multiple comparisons and isolated significance are correctness/robustness issues, not evidence that the derivation reduces to its inputs. Overall, no circular step meeting the required evidentiary bar is present.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central method rests on two tuned hyperparameters (λ, α) and on the untested validity of ensemble disagreement as uncertainty, plus an admitted non-equivalent proxy for deviation coverage. No new physical entities are introduced. Since the final claim is evaluated against true-game regret, the ledger is moderate rather than empty.

free parameters (2)
  • λ (uncertainty penalty weight) = 4 (selected after tuning on D=(1000,πU); values 2,4,6 tested)
    Weights the ensemble-disagreement penalty in the response objective (Eq 3/4); selected by lowest true-game regret during tuning (Section 5.2, Table 5).
  • α (deviation coverage weight) = 0.2 for R2D, 0.3 for RD
    Controls how often training optimizes deviation-coverage uncertainty against a uniform random policy (Eq 4); annealed to zero over 10 iterations; tuned on D=(1000,πU).
axioms (4)
  • domain assumption Ensemble reward-prediction range ρ is a calibrated proxy for model uncertainty on unseen state-action pairs.
    Invoked in Section 4.1, Eq 1 and used throughout to penalize exploration and construct R2D bounds; if models are similarly wrong on uncovered data, conservatism is miscalibrated.
  • ad hoc to paper The uniform-random action policy is roughly equivalent to sampling a different other-player policy each episode for minimizing deviation uncertainty.
    Section 4.2, Eq 4 and footnote 2; the authors explicitly state this equivalence 'does not generally hold'.
  • domain assumption Replicator dynamics and R2D reach approximate equilibria in empirical games despite no convergence guarantee.
    Section 4.3; R2D is heuristic, justified only by empirical demonstration in one game.
  • standard math Normal-form game equilibrium and regret definitions from game theory.
    Section 3 notation; standard background.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Conservative Equilibrium Discovery in Offline Game-Theoretic Multiagent Reinforcement Learning." pith.science (2026). https://pith.science/paper/XT445SZJ

@misc{pith2026260300374,
  author       = {Pith},
  title        = {Pith review of: Conservative Equilibrium Discovery in Offline Game-Theoretic Multiagent Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XT445SZJ}},
  note         = {Machine review of arXiv:2603.00374}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Offline learning of strategies takes data efficiency to its extreme by restricting algorithms to a fixed dataset of state-action trajectories. We consider the problem in a mixed-motive multiagent setting, where the goal is to solve a game under the offline learning constraint. We first frame this problem in terms of selecting among candidate equilibria. Since datasets may inform only a small fraction of game dynamics, it is generally infeasible in offline game-solving to even verify a proposed solution is a true equilibrium. Therefore, we consider the relative probability of low regret (i.e., closeness to equilibrium) across candidates based on the information available. Specifically, we extend Policy Space Response Oracles (PSRO), an online game-solving approach, by quantifying game dynamics uncertainty and modifying the RL objective to skew towards solutions more likely to have low regret in the true game. We further propose a novel meta-strategy solver, tailored for the offline setting, to guide strategy exploration in PSRO. Our incorporation of Conservatism principles from Offline reinforcement learning approaches for strategy Exploration gives our approach its name: COffeE-PSRO. Experiments demonstrate COffeE-PSRO's ability to extract lower-regret solutions than state-of-the-art offline approaches and reveal relationships between algorithmic components empirical game fidelity, and overall performance.

Figures

Figures reproduced from arXiv: 2603.00374 by Austin A. Nguyen, Michael P. Wellman.

Figure 1
Figure 1. Figure 1: COffeE-PSRO follows the basic structure of PSRO, using a [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 1
Figure 1. Figure 1: An iteration of COffeE-PSRO. During strategy generation, a rollout optimizes [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Relationships among model fidelity, performance, and conservatism. Each datapoint is a single trial ( [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Final regret under various parameter settings. Re [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Regret over iterations plots under various param [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ablation over COffeE-PSRO parameters. The top two rows ablate over [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 7 linked inside Pith

  1. [1]

    David Balduzzi, Marta Garnelo, Yoram Bachrach, Wojciech Czarnecki, Julien Per- olat, Max Jaderberg, and Thore Graepel. 2019. Open-ended learning in symmetric zero-sum games. In 36th International Conference on Machine Learning . 434–443

    2019

  2. [2]

    Oliehoek

    Ariyan Bighashdel, Yongzhao Wang, Stephen McAleer, Rahul Savani, and Frans A. Oliehoek. 2024. Policy space response oracles: A survey. In 33rd International Joint Conference on Artificial Intelligence (Jeju, Korea). 7951–7961

    2024

  3. [3]

    Jingxiao Chen, Weiji Xie, Weinan Zhang, Yong Yu, and Ying Wen. 2026. Offline fictitious self-play for competitive games. In 40th AAAI Conference on Artificial Intelligence (Singapore)

    2026

  4. [4]

    Qiwen Cui and Simon S. Du. 2022. When are offline two-player zero-sum Markov games solvable?. In 36th Annual Conference on Neural Information Processing Systems. 25779–25791

    2022

  5. [5]

    Jiechuan Jiang and Zongqing Lu. 2023. Offline decentralized multi-agent reinforce- ment learning. In 26th European Conference on Artificial Intelligence . 1148–1155

    2023

  6. [6]

    Jordan, L

    Patrick R. Jordan, L. Julian Schvartzman, and Michael P. Wellman. 2010. Strategy Exploration in Empirical Games. In 9th International Conference on Autonomous Agents and Multiagent Systems (Toronto). 1131–1138

    2010

  7. [7]

    Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims

  8. [8]

    Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. 2019. Stabilizing off- policy Q-learning via bootstrapping error reduction. In 33rd Annual Conference on Neural Information Processing Systems

    2019

  9. [9]

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conserva- tive Q-learning for offline reinforcement learning. In 34th Annual Conference on Neural Information Processing Systems . 1179–1191

    2020

  10. [10]

    Marc Lanctot, Edward Lockhart, Jean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, et al. 2019. OpenSpiel: A framework for reinforcement learning in games. arXiv preprint arXiv:1908.09453 (2019)

    Pith/arXiv arXiv 2019

  11. [11]

    Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. 2017. A unified game- theoretic approach to multiagent reinforcement learning. In 31st Annual Confer- ence on Neural Information Processing Systems . 4190–4203

    2017

  12. [12]

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. 2020. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 (2020)

    Pith/arXiv arXiv 2020

  13. [13]

    Dauphin, Devi Parikh, and Dhruv Batra

    Mike Lewis, Denis Yarats, Yann N. Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? End-to-end learning for negotiation dialogues. In Conference on Empirical Methods in Natural Language Processing . 2443–2453

    2017

  14. [14]

    Shuxin Li, Xinrun Wang, Youzhi Zhang, Jakub Cerny, Pengdeng Li, Hau Chan, and Bo An. 2022. Offline equilibrium finding. arXiv preprint arXiv:2207.05285 (2022)

    Pith/arXiv arXiv 2022

  15. [15]

    Luke Marris, Paul Muller, Marc Lanctot, Karl Tuyls, and Thore Graepel. 2021. Multi-agent training beyond zero-sum with correlated equilibrium meta-solvers. In 38th International Conference on Machine Learning . 7480–7491

    2021

  16. [16]

    Paul Muller, Shayegun Omidshafiei, Mark Rowland, Karl Tuyls, Julien Pérolat, Siqi Liu, Daniel Hennes, Luke Marris, Marc Lanctot, Edward Hughes, et al. 2020. A generalized training approach for multiagent learning. In 8th International Conference on Learning Representations

    2020

  17. [17]

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. 2020. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 (2020)

    Pith/arXiv arXiv 2020

  18. [18]

    Nguyen, Anri Gu, and Michael P

    Austin A. Nguyen, Anri Gu, and Michael P. Wellman. 2025. Explicit exploration for high-welfare equilibria in game-theoretic multiagent reinforcement learning. In 42nd International Conference on Machine Learning (Vancouver). 45988–46007

    2025

  19. [19]

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. 2019. Advantage- weighted regression: Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177 (2019)

    Pith/arXiv arXiv 2019

  20. [20]

    Nicolas Perez-Nieves, Yaodong Yang, Oliver Slumbers, David Henry Mguni, Ying Wen, and Jun Wang. 2021. Modelling behavioural diversity for learning in open- ended games. In 38th International Conference on Machine Learning . 8514–8524

    2021

  21. [21]

    Zhengdao Shao, Liansheng Zhuang, Houqiang Li, and Shafei Wang. 2025. COP- SRO: An Offline Empirical Game Theoretic Method With Conservative Critic. IEEE Transactions on Neural Networks and Learning Systems (2025)

    2025

  22. [22]

    Max Olan Smith, Thomas Anthony, and Michael P. Wellman. 2021. Iterative empirical game solving via single policy best response. In 9th International Con- ference on Learning Representations (Virtual)

    2021

  23. [23]

    Taylor and Leo B

    Peter D. Taylor and Leo B. Jonker. 1978. Evolutionary stable strategies and game dynamics. Mathematical Biosciences 40, 1-2 (1978), 145–156

    1978

  24. [24]

    Hado Van Hasselt, Arthur Guez, and David Silver. 2016. Deep reinforcement learning with double Q-learning. In30th AAAI Conference on Artificial Intelligence. 2094–2100

    2016

  25. [25]

    Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan. 2024. Offline multi-agent reinforcement learning with implicit global-to-local value regular- ization. Advances in Neural Information Processing Systems 36 (2024)

    2024

  26. [26]

    Yongzhao Wang, Qiurui Ma, and Michael P. Wellman. 2022. Evaluating strategy exploration in empirical game-theoretic analysis. In 21st International Conference on Autonomous Agents and Multiagent Systems . 1346–1354

    2022

  27. [27]

    Yongzhao Wang and Michael P. Wellman. 2023. Regularization for Strategy Ex- ploration in Empirical Game-Theoretic Analysis. In22nd International Conference on Autonomous Agents and Multiagent Systems . 2484–2486

    2023

  28. [28]

    Yongzhao Wang and Michael P. Wellman. 2024. Generalized response objectives for strategy exploration in empirical game-theoretic analysis. In23rd International Conference on Autonomous Agents and Multiagent Systems . 1892–1900

    2024

  29. [29]

    Wellman, Karl Tuyls, and Amy Greenwald

    Michael P. Wellman, Karl Tuyls, and Amy Greenwald. 2025. Empirical Game- Theoretic Analysis: A Survey. Journal of Artificial Intelligence Research 82 (2025), 1017–1076

    2025

  30. [30]

    Yifan Wu, George Tucker, and Ofir Nachum. 2019. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 (2019)

    Pith/arXiv arXiv 2019

  31. [31]

    Yaodong Yang and Jun Wang. 2021. An overview of multi-agent reinforcement learning from game theoretical perspective. arXiv preprint arXiv:2011.00583v3 (2021)

    Pith/arXiv arXiv 2021

  32. [32]

    Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. 2020. MOPO: Model-based offline policy opti- mization. In 34th Annual Conference on Neural Information Processing Systems . 14129–14142

    2020

  33. [33]

    Yuheng Zhang, Yu Bai, and Nan Jiang. 2023. Offline learning in Markov games with general function approximation. In40th International Conference on Machine Learning. PMLR, 40804–40829

    2023

  34. [34]

    Han Zhong, Wei Xiong, Jiyuan Tan, Liwei Wang, Tong Zhang, Zhaoran Wang, and Zhuoran Yang. 2022. Pessimistic minimax value iteration: Provably efficient equilibrium learning from offline datasets. In 39th International Conference on Machine Learning. PMLR, 27117–27142. A HYPERPARAMETERS Table 2: All hyperparameters Parameter Description Chosen Parameter DD...

    2022

  35. [2020]

    In 34th Annual Confer- ence on Neural Information Processing Systems

    Morel: Model-based offline reinforcement learning. In 34th Annual Confer- ence on Neural Information Processing Systems . 21810–21823

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.