Pith. sign in

REVIEW 3 major objections 5 minor 146 references

The Problem of Social Cost in Multi-Agent General Reinforcement Learning: Survey and Synthesis

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A VCG-style market protocol can quantify and price the social harm that learning agents impose on each other in general reinforcement learning settings.

desk verdict Valuable synthesis of VCG and general RL, but the headline BIC theorem overreaches: the proof breaks after an agent's horizon, which is exactly where the paper claims novelty. read the letter →

arxiv 2412.02091 v2 pith:KGH6ISJH submitted 2024-12-03 cs.AI cs.GTcs.LGcs.MA

Kee Siong Ng , Samuel Yang-Zhao , Timothy Cadogan-Cowper This is my paper · ORCID
classification cs.AIcs.GTcs.LGcs.MA
keywords socialcostmulti-agentreinforcementlearningmechanismdesignVCGgeneralAIsafetycap-and-tradeBayes-Nashincentivecompatibility
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper takes on an open problem in AI safety: how to measure and control the social harms that utility-maximising agents cause one another in a shared environment. It proposes wrapping a multi-agent general reinforcement learning environment in a VCG mechanism, so that at every step the joint action maximizes the sum of the agents' submitted valuations and each agent pays the Clark pivot cost its action imposes on the others. The central construction is a rational valuation function $v_{t,i}=q_{t,i}-c_{t,i}$, where $q_{t,i}$ is the expected future reward under the socially chosen actions and $c_{t,i}$ is the expected future payment. The paper proves that with these valuations the protocol is Bayes-Nash incentive compatible and individually rational, giving a concrete partial answer to a problem the AI-safety literature had left open. If the construction holds, social harm in multi-agent AI systems becomes quantifiable and priceable, opening the way to market-based controls such as cap-and-trade.

What carries the argument

The load-bearing object is the rational valuation function of Definition 14, $v_{t,i}(h_{t-1},a_t)=q_{t,i}(h_{t-1},a_t)-c_{t,i}(h_{t-1},a_t)$, defined through the fixed-point equations (16)-(21). Here $q_{t,i}$ is the value of taking action $a_t$ under the policy that the mechanism will choose the socially optimal joint action at all future steps, and $c_{t,i}$ is the stream of expected future VCG payments. The Clark pivot payment function (14), $p_i(v_t)=\max_a\sum_{j\neq i}v_{t,j}(a)-\sum_{j\neq i}v_{t,j}(a^*_t)$, converts the externality into an internal cost: the agent pays what its presence costs the others. The protocol first aggregates the submitted valuations into the joint action $a^*_t=\arg\max_a\sum_i v_{t,i}(a)$, then charges the pivot payments; the truthfulness result is an adaptation of the classical VCG incentive-compatibility theorem to this dynamic, history-based setting.

What would settle it

In a finite-horizon multi-agent environment with differently sized horizons, compute by exhaustive search the expected realisable cumulative utility of truthful versus false valuation reports at a time step beyond one agent's horizon; Corollary 7 predicts truthful reports always win, so any counterexample where a false report does strictly better would settle the question. For the infinite-horizon claim, a rigorous derivation of the One-Shot Deviation Principle for the fixed-point equations (16)-(21) would confirm the extension the paper assumes.

Watch

Extended reading notes

Core claim

The central claim is that for any history-based multi-agent environment $\varphi$ and any VCG mechanism $M=(f,p_1,\dots,p_k)$, the interaction protocol $M\triangleright\varphi$ makes truthful reporting the best response. Each agent $i$ declares the valuation $v_{t,i}(h_{t-1},a_t)=q_{t,i}(h_{t-1},a_t)-c_{t,i}(h_{t-1},a_t)$, with $q_{t,i}$ the expected cumulative reward when the mechanism picks the social-welfare-maximising action at every future step and $c_{t,i}$ the expected cumulative Clark pivot payment. Corollary 7 asserts the protocol is Bayes-Nash incentive compatible with respect to the agent's realisable cumulative utility, and Corollary 8 asserts individual rationality when the valuations are non-negative. The environment is the history-based general reinforcement learning setting, so the framework covers single-agent RL, static and dynamic mechanism design, and ordinary multi-agent RL as special cases.

Load-bearing premise

The rational valuation functions are defined by backward induction under the assumption that every agent has full knowledge of the environment $\varphi$ and can exactly compute the fixed point of equations (16)-(21); if that assumption fails, the Bayes-Nash guarantee is not established.

Editorial extensions

If this is right

  • If agents report their rational valuations, the joint action at every step maximises total declared social welfare, and the payments make each agent internalise the externality it causes.
  • The framework reduces to the standard single-agent general reinforcement learning problem when $k=1$, to the Vickrey second-price auction in the static one-shot case, and to ordinary multi-agent reinforcement learning when actions are not mutually exclusive.
  • Bayesian reinforcement-learning agents such as DynamicHedge AIXI can estimate $q_{t,i}$ and $c_{t,i}$ online, so the truthfulness guarantee can be approached in environments the agents do not initially know.
  • In the cap-and-trade application, a VCG auction of pollution permits allocates emissions to the refineries that can turn them into the most value; the same mechanism gives other agents leverage to block runaway paperclip-style production.
  • When agents are rewarded for joint profit, Q-learning can learn a collusive bidding policy that drives permit prices to zero without explicit communication, showing that the mechanism's incentive properties depend on the agents' reward functions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical corollary the paper does not dwell on: if real agents learn only approximate valuations, the exact Bayes-Nash guarantee degrades, and the relevant open question is how approximation error in $q$ and $c$ translates into welfare loss.
  • Because the Clark pivot in (14) charges for removing an agent only from the current time step rather than from all future steps, the mechanism is not a fully dynamic VCG in the multi-period sense; this makes it natural for long-lived agents but means longer-horizon externalities are priced only indirectly.
  • The collusion phenomenon in the cap-and-trade simulation suggests that market-based control of AI agents may need anti-collusion safeguards or the budget-balanced, collusion-proof Guaranteed Utility Mechanism variant developed in Appendix A.2.
  • Since Corollary 7's infinite-horizon extension invokes the One-Shot Deviation Principle without proof, a rigorous infinite-horizon theorem is needed before the result can be relied on for AGI-scale agents; strictly speaking the paper establishes the finite-horizon case.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a general framework, denoted M▷ϕ, for multi-agent general reinforcement learning in which a VCG-style mechanism coordinates agents by charging payments that internalize social costs. The central contribution is Definition 14, which defines each agent's rational valuation function qt,i(ht−1, at) − ct,i(ht−1, at) by backward induction under full knowledge of the environment, and Corollary 7, which claims that truthful submission of these valuation functions is Bayes-Nash incentive compatible with respect to the agents' realisable cumulative utility at every time step. The paper also surveys relevant background in GRL and mechanism design, describes learning approximations including Dynamic Hedge AIXI, and presents applications to the paperclip maximiser problem and a cap-and-trade pollution control simulation.

Significance. If the main incentive-compatibility claim were correct, the paper would provide a valuable synthesis that extends dynamic mechanism design to history-based general reinforcement learning environments and to agents with heterogeneous planning horizons, thereby offering a partial answer to the open problem of measuring social harms in multi-agent AGI settings. The paper is genuinely expository in parts, shipping self-contained proofs of VCG and exponential-VCG properties, concrete worked examples, and reproducible-style simulation results for cap-and-trade. These are real strengths. However, the central theorem is not correct as stated: the proof of Corollary 7 fails exactly in the case of agents whose finite horizon has expired while the protocol continues, which is one of the paper's claimed improvements over prior work. The flaw is load-bearing and requires a substantive revision of the claim, not merely a presentational fix.

major comments (3)
  1. [§4.1, Corollary 7 proof] The proof's identity for the case t ≥ mi is incorrect. The text states: 'for the t ≥ mi case, the qt+1,i and ct+1,i terms are both zero and vt,i(ht−1, at) = E[rt,i].' Under Definition 14, qt,i(ht−1, at) = 0 for t > mi and ct,i(ht−1, at) = 0 for t ≥ mi, so vt,i(ht−1, at) = 0 for t > mi. The equality vt,i = E[rt,i] holds at t = mi (because qt,i(ht−1, at) = E[rt,i + qt+1,i] and qt+1,i = 0), but not beyond the agent's horizon. Consequently the derivation of equation (23), which converts expected realisable utility into sum_j vt,j(ht−1, at) − max_b sum_{j≠i} vt,j(ht−1, b), is invalid at t > mi.
  2. [§4.1, Corollary 7 statement] The claimed 'at every time step' Bayes-Nash incentive compatibility is false when agents have different finite horizons. A concrete counterexample: let k = 2, m1 = 1, m2 = 2, and at t = 2 suppose deterministic rewards r_{2,1}(A) = 10, r_{2,1}(B) = 0, while agent 2 values only B at 5. Submitting the true valuation v_{2,1} = 0 leads the mechanism to pick B, giving agent 1 utility 0. If agent 1 instead reports v~_{2,1}(A) = 100, the mechanism picks A; agent 1 pays the pivot price 5, receives reward 10, and obtains utility 5. This profitable misreport contradicts Corollary 7 exactly when one agent's horizon has expired. The backward-induction argument in the proof also cannot handle t > mi because agent i's valuation function is zero there while its realisable utility still includes the current reward rt,i.
  3. [§4.1, Corollary 7 proof, infinite horizon] The sentence 'For mi → ∞, the same argument can be made using the One-Shot Deviation Principle in place of backward induction' is an unproved invocation. The One-Shot Deviation Principle requires conditions (such as continuity at infinity, bounded rewards, or a discount factor) that are not stated in the paper. Since the finite-horizon proof is already invalid for t > mi, this extension to infinite horizons is not currently supported and needs either a proof or a clearly stated set of assumptions.
minor comments (5)
  1. [Example 4, Tables 1 and 2] The cell entries in Tables 1 and 2 combine three numbers (vt,i, rt,i, pt,i) without a caption or legend describing their order; adding a table note would make the example significantly easier to follow.
  2. [Definition 15 and Corollary 7 proof] The notation for submitted valuations alternates between \tilde{v}_t and \tilde{~v}_{t,i}; unify the notation for clarity.
  3. [§6.2, caption of Figure 2] The axis label in Figure 2 says 'Cubic tons' instead of 'cubic tons'; fix the capitalization for consistency.
  4. [§3.2, Example 2] In the second solution for bilateral trade, the displayed equation for u_S is missing a closing parenthesis after the expression for v_S(d); correct the typo so the formula is unambiguous.
  5. [References] The paper cites [69] as the source for AIXI in the Introduction, while [67] is the classic AIXI reference used elsewhere; consider citing [67] at the first occurrence to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Corollary 7 is a disclosed adaptation of VCG; the t>m_i proof gap is a correctness issue, not circularity.

full rationale

The central derivation is self-contained. Definition 14 defines rational q-, social-cost, and valuation functions by backward induction from the environment and the VCG payment rule; Corollary 7 is explicitly presented as "an adaptation of Theorem 2" and its key identity — expected realisable utility equals sum_j v_t,j(a_t) minus max_b sum_{j≠i} v_t,j(b) — is exactly the standard VCG utility expression. No fitted parameter is renamed as a prediction, and no quantity is defined in terms of the conclusion it is used to prove. Self-citations such as [139] (Dynamic Hedge AIXI), [131], and [140] occur in the survey/learning sections but are not load-bearing for the protocol's incentive-compatibility claim; the convergence-to-Nash conditions are attributed to external work [73, 86]. The paper even discloses the known result: the proof states "The next result is an adaptation of Theorem 2". A genuine correctness concern exists but is not circularity: in the proof of Corollary 7, the parenthetical "for the t ≥ mi case, the qt+1,i and ct+1,i terms are both zero and vt,i(ht−1, at) = Eortrt,i" contradicts Definition 14, which sets vt,i=0 for t>mi and c=0 for t≥mi, so the "at every time step" BIC claim is not established beyond an agent's horizon; the infinite-horizon appeal to the One-Shot Deviation Principle is likewise unproved. Section 7's acknowledged limitation (no enforcement for agents "in the wild") is an honest scope statement, not a circular move. Overall, the central derivation reduces to a correct application of VCG, not to its own assumptions, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central protocol depends on standard mechanism design theorems (VCG, Clark pivot) and on the assumption that agents can compute their rational valuation functions exactly. The illustrative cap-and-trade example introduces hand-chosen parameters. No new physical or computational entities are postulated.

free parameters (3)
  • Emission function coefficients = 1/5, -12, 200, 888; m_i = i
    Chosen by hand for the cap-and-trade illustration to produce a non-monotonic emissions curve; not fitted to real data and does not affect the central mechanism claim.
  • Inverse emission function tie-breaking rule = max root of the cubic
    The definition of s_i^{-1}(g) selects the largest real root, a modeling choice that affects the auction bids in Section 6.2.
  • Q-learning hyperparameters = unspecified
    The RL experiments in Section 6.2 do not state learning rate, exploration schedule, or number of episodes, so the reported profits are not reproducible.
assumptions (6)
  • standard math VCG mechanism is incentive compatible (Theorem 2)
    Used in Section 4.1 to establish Corollary 7 for the protocol M▷φ.
  • standard math Clark pivot rule yields individual rationality for non-negative valuations (Theorem 3)
    Used in Corollary 8 to show agents are never worse off participating.
  • standard math Bayesian mixture converges rapidly to the true environment (Theorem 1)
    Used in Section 5.2 to justify learning the environment from data.
  • domain assumption Grain-of-truth condition suffices for convergence to Nash equilibrium
    Used in Section 5.2.5; noted not to hold for incomputable AIXI agents.
  • ad hoc to paper One-Shot Deviation Principle holds for infinite-horizon undiscounted games
    Invoked without proof to extend Corollary 7 to mi → ∞; standard conditions for the principle are not verified in this GRL-with-mechanism setting.
  • ad hoc to paper Existence and uniqueness of the rational valuation functions (16)-(21) for infinite horizons
    Definition 14 is inductive for finite mi; the infinite-horizon case is asserted in the text, not proved.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Problem of Social Cost in Multi-Agent General Reinforcement Learning: Survey and Synthesis." pith.science (2026). https://pith.science/paper/KGH6ISJH

@misc{pith2026241202091,
  author       = {Pith},
  title        = {Pith review of: The Problem of Social Cost in Multi-Agent General Reinforcement Learning: Survey and Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KGH6ISJH}},
  note         = {Machine review of arXiv:2412.02091}
}
read the original abstract

The AI safety literature is full of examples of powerful AI agents that, in blindly pursuing a specific and usually narrow objective, ends up with unacceptable and even catastrophic collateral damage to others. In this paper, we consider the problem of social harms that can result from actions taken by learning and utility-maximising agents in a multi-agent environment. The problem of measuring social harms or impacts in such multi-agent settings, especially when the agents are artificial generally intelligent (AGI) agents, was listed as an open problem in Everitt et al, 2018. We attempt a partial answer to that open problem in the form of market-based mechanisms to quantify and control the cost of such social harms. The proposed setup captures many well-studied special cases and is more general than existing formulations of multi-agent reinforcement learning with mechanism design in two ways: (i) the underlying environment is a history-based general reinforcement learning environment like in AIXI; (ii) the reinforcement-learning agents participating in the environment can have different learning strategies and planning horizons. To demonstrate the practicality of the proposed setup, we survey some key classes of learning algorithms and present a few applications, including a discussion of the Paperclips problem and pollution control with a cap-and-trade system.

Figures

Figures reproduced from arXiv: 2412.02091 by the authors.

Figure 1
Figure 1. A horizon-2 game tree with small action and percept spaces [PITH_FULL_IMAGE:figures/full_fig_p017_1.png] view at source ↗
Figure 2
Figure 2. Plots of s and s −1 for refineries R1 and R2 [PITH_FULL_IMAGE:figures/full_fig_p039_2.png] view at source ↗
Figure 3
Figure 3. Cubic tons of greenhouse gas emitted for different production of fuel [PITH_FULL_IMAGE:figures/full_fig_p039_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: RL with reward function r (1) (incentivise individual profit) [PITH_FULL_IMAGE:figures/full_fig_p042_4.png]
Figure 5
Figure 5. Figure 5: RL with reward function r (2) (incentivise joint profit) solely by observing the permit holdings of each participant in the auction – there was never any explicit communication between them. This phenomenon is analysed more carefully in [PITH_FULL_IMAGE:figures/full_f…
Figure 6
Figure 6. Figure 6: RL with reward function r (3) (zero-sum reward) moderation of spread of misinformation [2] in social media. These more detailed applications will be written up elsewhere. 7 Discussion and Conclusion In the spirit of [110], we considered in this paper the problem of soc…
Figure 5
Figure 5. Figure 5: In the Estimated Optimal Strategies matrix in Fig 7, rows denote permits [PITH_FULL_IMAGE:figures/full_fig_p065_5.png]
Figure 7
Figure 7. Figure 7: Final policy for a sample run of r (2) . 66 [PITH_FULL_IMAGE:figures/full_fig_p066_7.png]
Figure 8
Figure 8. Figure 8: Final policy for a sample run of r (3) . 67 [PITH_FULL_IMAGE:figures/full_fig_p067_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

146 extracted references · 69 canonical work pages

  1. [1]

    David Abel, Dilip Arumugam, Lucas Lehnert, and Michael L. Littman. State abstractions for lifelong reinforcement learning. In Jennifer G. Dy and Andreas Krause, editors,ICML, pages 10–19, 2018

    2018

  2. [2]

    A model of online misinformation

    Daron Acemoglu, Asuman Ozdaglar, and James Siderius. A model of online misinformation. Review of Economic Studies, page rdad111, 2023

    2023

  3. [3]

    The multiplicative weights updatemethod: ameta-algorithmandapplications

    Sanjeev Arora, Elad Hazan, and Satyen Kale. The multiplicative weights updatemethod: ameta-algorithmandapplications. Theory of Computing, 8(1):121–164, 2012

    2012

  4. [4]

    Deep reinforcement learning: A brief survey.IEEE Signal Processing Magazine, 34(6):26–38, 2017

    Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil An- thony Bharath. Deep reinforcement learning: A brief survey.IEEE Signal Processing Magazine, 34(6):26–38, 2017

    2017

  5. [5]

    An efficient dynamic mechanism.Economet- rica, 81(6):2463–2485, 2013

    Susan Athey and Ilya Segal. An efficient dynamic mechanism.Economet- rica, 81(6):2463–2485, 2013

    2013

  6. [6]

    Using confidence bounds for exploitation-exploration trade- offs

    Peter Auer. Using confidence bounds for exploitation-exploration trade- offs. J. Mach. Learn. Res., 3:397–422, 2003

    2003

  7. [7]

    The nonstochastic multiarmed bandit problem.SIAM Journal on Com- puting, 32(1):48–77, 2002

    Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem.SIAM Journal on Com- puting, 32(1):48–77, 2002

    2002

  8. [8]

    Correlated equilibrium as an expression of Bayesian rationality

    Robert J Aumann. Correlated equilibrium as an expression of Bayesian rationality. Econometrica: Journal of the Econometric Society, pages 1– 18, 1987. 44

    1987

Show all 146 references
  1. [9]

    The emergence of cooperation among egoists.American Political Science Review, 75(2):306–318, 1981

    Robert Axelrod. The emergence of cooperation among egoists.American Political Science Review, 75(2):306–318, 1981

    1981

  2. [10]

    The Formula: The Universal Laws of Success

    Albert-László Barabási. The Formula: The Universal Laws of Success. Hachette UK, 2018

    2018

  3. [11]

    Dynamic incentives for congestion control

    Jorge Barrera and Alfredo Garcia. Dynamic incentives for congestion control. IEEE Transactions on Automatic Control, 60(2):299–310, 2014

    2014

  4. [12]

    A neural probabilistic language model.J

    Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic language model.J. Mach. Learn. Res., 3:1137–1155, 2003

    2003

  5. [13]

    Taming the Matthew effect in online markets with social influence

    Franco Berbeglia and Pascal Van Hentenryck. Taming the Matthew effect in online markets with social influence. InAAAI, pages 10–16, 2017

    2017

  6. [14]

    The dynamic pivot mechanism

    Dirk Bergemann and Juuso Välimäki. The dynamic pivot mechanism. Econometrica, 78(2):771–789, 2010

    2010

  7. [15]

    Dynamic mechanism design: An introduction

    Dirk Bergemann and Juuso Välimäki. Dynamic mechanism design: An introduction. Journal of Economic Literature, 57(2):235–274, 2019

    2019

  8. [16]

    From external to internal regret.Jour- nal of Machine Learning Research, 8(6), 2007

    Avrim Blum and Yishay Mansour. From external to internal regret.Jour- nal of Machine Learning Research, 8(6), 2007

    2007

  9. [17]

    An Introduction to the Theory of Mechanism Design

    Tilman Börgers. An Introduction to the Theory of Mechanism Design. Oxford University Press, 2015

    2015

  10. [18]

    Superintelligence: Paths, Dangers, Strategies

    Nick Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, 2014

    2014

  11. [19]

    Ethical issues in advanced artificial intelligence.Machine Ethics and Robot Ethics, pages 69–75, 2020

    Nick Bostrom. Ethical issues in advanced artificial intelligence.Machine Ethics and Robot Ethics, pages 69–75, 2020

    2020

  12. [20]

    Learn- ing to mitigate AI collusion on economic platforms.Advances in Neural Information Processing Systems, 35:37892–37904, 2022

    GianlucaBrero, EricMibuari, NicolasLepore, andDavidCParkes. Learn- ing to mitigate AI collusion on economic platforms.Advances in Neural Information Processing Systems, 35:37892–37904, 2022

    2022

  13. [21]

    A survey of monte carlo tree search methods

    Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lu- cas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI ...

    2012

  14. [22]

    A comprehensive survey of graph embedding: Problems, techniques, and applications

    Hongyun Cai, Vincent W Zheng, and Kevin Chen-Chuan Chang. A comprehensive survey of graph embedding: Problems, techniques, and applications. IEEE Transactions on Knowledge and Data Engineering, 30(9):1616–1637, 2018. 45

    2018

  15. [23]

    Artificial intelligence, algorithmic pricing, and collusion.American Economic Review, 110(10):3267–3297, 2020

    Emilio Calvano, Giacomo Calzolari, Vincenzo Denicolo, and Sergio Pas- torello. Artificial intelligence, algorithmic pricing, and collusion.American Economic Review, 110(10):3267–3297, 2020

    2020

  16. [24]

    Self-predictive universal AI

    Elliot Catt, Jordi Grau-Moya, Marcus Hutter, Matthew Aitchison, Tim Genewein, Gregoire Deletang, Kevin Li, and Joel Veness. Self-predictive universal AI. Advances in Neural Information Processing Systems , 36:27181–27198, 2023

    2023

  17. [25]

    Optimal coordi- nated planning amongst self-interested agents with private state

    Ruggiero Cavallo, David C Parkes, and Satinder Singh. Optimal coordi- nated planning amongst self-interested agents with private state. InUAI, pages 55–62, 2006

    2006

  18. [26]

    Cambridge University Press, 2006

    Nicolo Cesa-Bianchi and Gábor Lugosi.Prediction, Learning, and Games. Cambridge University Press, 2006

    2006

  19. [27]

    Evolu- tionary dynamics of biological auctions.Theoretical Population Biology, 81(1):69–80, 2012

    Krishnendu Chatterjee, Johannes G Reiter, and Martin A Nowak. Evolu- tionary dynamics of biological auctions.Theoretical Population Biology, 81(1):69–80, 2012

    2012

  20. [28]

    Dynamic pricing in a labor market: Surge pricing and flexible work on the Uber platform.Ec, 16:455, 2016

    M Keith Chen and Michael Sheldon. Dynamic pricing in a labor market: Surge pricing and flexible work on the Uber platform.Ec, 16:455, 2016

    2016

  21. [29]

    Hedging in games: Faster convergence of external and swap regrets

    Xi Chen and Binghui Peng. Hedging in games: Faster convergence of external and swap regrets. Advances in Neural Information Processing Systems, 33:18990–18999, 2020

    2020

  22. [30]

    Prediction with expert evaluators’ advice

    Alexey Chernov and Vladimir Vovk. Prediction with expert evaluators’ advice. In Ricard Gavaldà, Gábor Lugosi, Thomas Zeugmann, and Sandra Zilles, editors,Algorithmic Learning Theory, pages 8–22. Springer, 2009

    2009

  23. [31]

    Brian Christian.The alignment problem: How can machines learn human values? Atlantic Books, 2021

    2021

  24. [32]

    A formulation of the simple theory of types.Journal of Symbolic Logic, 5:56–68, 1940

    Alonzo Church. A formulation of the simple theory of types.Journal of Symbolic Logic, 5:56–68, 1940

    1940

  25. [33]

    The problem of social cost.Journal of Law and Economics, 3(1):1–44, 1960

    Robert Coase. The problem of social cost.Journal of Law and Economics, 3(1):1–44, 1960

    1960

  26. [34]

    The Firm, The Market, and The Law

    Robert Coase. The Firm, The Market, and The Law. UCP, 2012

    2012

  27. [35]

    Law for the platform economy.UCDL Rev., 51:133, 2017

    Julie E Cohen. Law for the platform economy.UCDL Rev., 51:133, 2017

    2017

  28. [36]

    Holliday, Bob M

    Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H. Holliday, Bob M. Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Rus- sell, Hailey Schoelkopf, Emanuel Tewolde, and William S. Zwicker. Social choice for AI alignment: Dealing with diverse human feedback.CoRR...

    2024 arXiv

  29. [37]

    Cambridge University Press, 1996

    Richard Cornes and Todd Sandler.The Theory of Externalities, Public Goods, and Club Goods. Cambridge University Press, 1996

    1996

  30. [38]

    A collusion-proof dynamic mechanism.SSRN, 2024

    Endre Csóka, Heng Liu, Alexander Rodivilov, and Alexander Teytelboym. A collusion-proof dynamic mechanism.SSRN, 2024

    2024

  31. [39]

    From external to swap regret 2.0: An efficient reduction for large action spaces

    Yuval Dagan, Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. From external to swap regret 2.0: An efficient reduction for large action spaces. InProceedings of the 56th Annual ACM Symposium on Theory of Computing, pages 1216–1222, 2024

    2024

  32. [40]

    Logical and relational learning

    LucDeRaedt. Logical and relational learning. SpringerScience&Business Media, 2008

    2008

  33. [41]

    The algorithmic foundations of differ- ential privacy.Found

    Cynthia Dwork and Aaron Roth. The algorithmic foundations of differ- ential privacy.Found. Trends Theor. Comput. Sci., 9(3-4):211–407, 2014

    2014

  34. [42]

    Relational reinforce- ment learning

    Saso Dzeroski, Luc De Raedt, and Kurt Driessens. Relational reinforce- ment learning. Mach. Learn., 43(1/2):7–52, 2001

    2001

  35. [43]

    Matchmakers: the new economics of multisided platforms

    David S Evans. Matchmakers: the new economics of multisided platforms. Harvard Business Review Press, 2016

    2016

  36. [44]

    Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective.Synthese, 198(Suppl 27):6435–6467, 2021

    Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective.Synthese, 198(Suppl 27):6435–6467, 2021

    2021

  37. [45]

    Re- inforcement learning with a corrupted reward channel

    Tom Everitt, Victoria Krakovna, Laurent Orseau, and Shane Legg. Re- inforcement learning with a corrupted reward channel. InIJCAI, pages 4705–4713, 2017

    2017

  38. [46]

    AGI safety literature review

    Tom Everitt, Gary Lea, and Marcus Hutter. AGI safety literature review. In Jérôme Lang, editor,IJCAI, pages 5441–5449. ijcai.org, 2018

    2018

  39. [47]

    Reflective or- acles: A foundation for game theory in artificial intelligence

    Benja Fallenstein, Jessica Taylor, and Paul F Christiano. Reflective or- acles: A foundation for game theory in artificial intelligence. In Inter- national Workshop on Logic, Rationality and Interaction, pages 411–415. Springer, 2015

    2015

  40. [48]

    W.M. Farmer. The seven virtues of simple type theory.Journal of Applied Logic, 6(3):267–286, 2008

    2008

  41. [49]

    Economicsofoilrefining

    Jean-PierreFavennec. Economicsofoilrefining. In The Palgrave Handbook of International Energy Economics, pages 59–74. Springer, 2022

    2022

  42. [50]

    Calibrated learning and correlated equilibrium

    Dean P Foster and Rakesh V Vohra. Calibrated learning and correlated equilibrium. Games and Economic Behavior, 21(1-2):40–55, 1997. 47

    1997

  43. [51]

    A decision-theoretic generalization of on-line learning and an application to boosting.Journal of Computer and System Sciences, 55(1):119–139, 1997

    Yoav Freund and Robert E Schapire. A decision-theoretic generalization of on-line learning and an application to boosting.Journal of Computer and System Sciences, 55(1):119–139, 1997

    1997

  44. [52]

    Schapire, Yoram Singer, and Manfred K

    Yoav Freund, Robert E. Schapire, Yoram Singer, and Manfred K. War- muth. Using and combining predictors that specialize. InProceedings of the 29th Annual ACM Symposium on Theory of Computing, page 334–343. ACM, 1997

    1997

  45. [53]

    The rationality of quali- fied lotteries

    Bruno S Frey, Margit Osterloh, and Katja Rost. The rationality of quali- fied lotteries. European Management Review, 20(4):698–710, 2023

    2023

  46. [54]

    MIT press, 1998

    Drew Fudenberg and David K Levine.The theory of learning in games, volume 2. MIT press, 1998

    1998

  47. [55]

    Artificial intelligence, values, and alignment.Minds and Machines, 30(3):411–437, 2020

    Iason Gabriel. Artificial intelligence, values, and alignment.Minds and Machines, 30(3):411–437, 2020

    2020

  48. [56]

    Bayesian reinforcement learning: A survey

    Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian reinforcement learning: A survey. Foundations and Trends in Machine Learning, 8(5-6):359–483, 2015

    2015

  49. [57]

    Graph embedding techniques, applica- tions, and performance: A survey.Knowledge-Based Systems, 151:78–94, 2018

    Palash Goyal and Emilio Ferrara. Graph embedding techniques, applica- tions, and performance: A survey.Knowledge-Based Systems, 151:78–94, 2018

    2018

  50. [58]

    Inconsistency of Bayesian in- ference for misspecified linear models, and a proposal for repairing it

    Peter Grünwald and Thijs Van Ommen. Inconsistency of Bayesian in- ference for misspecified linear models, and a proposal for repairing it. Bayesian Analysis, 12:1069–1103, 2017

    2017

  51. [59]

    The off-switch game

    Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. The off-switch game. InWorkshops at the Thirty-First AAAI Conference on Artificial Intelligence, 2017

    2017

  52. [60]

    Cooperative inverse reinforcement learning.Advances in Neural Informa- tion Processing Systems, 29, 2016

    Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning.Advances in Neural Informa- tion Processing Systems, 29, 2016

    2016

  53. [61]

    The tragedy of the commons

    Garrett Hardin. The tragedy of the commons. Science, 162:1243–1248, 1968

    1968

  54. [62]

    Completeness in the theory of types.Journal of Symbolic Logic, 15(2):81–91, 1950

    Leon Henkin. Completeness in the theory of types.Journal of Symbolic Logic, 15(2):81–91, 1950

    1950

  55. [63]

    Tracking the best expert.Ma- chine learning, 32(2):151–178, 1998

    Mark Herbster and Manfred K Warmuth. Tracking the best expert.Ma- chine learning, 32(2):151–178, 1998

    1998

  56. [64]

    The many faces of exponentialweightsinonlinelearning

    Dirk Hoeven, Tim van Erven, and Wojciech Kotłowski. The many faces of exponentialweightsinonlinelearning. In Conference On Learning Theory, pages 2067–2092. PMLR, 2018. 48

    2018

  57. [65]

    The exponential mechanism for social welfare: Private, truthful, and nearly optimal

    Zhiyi Huang and Sampath Kannan. The exponential mechanism for social welfare: Private, truthful, and nearly optimal. In53rd Annual Symposium on Foundations of Computer Science, pages 140–149. IEEE, 2012

    2012

  58. [66]

    Nash incentive-compatible onlinemechanismlearningviaweaklydifferentiallyprivateonlinelearning

    Joon Suk Huh and Kirthevasan Kandasamy. Nash incentive-compatible onlinemechanismlearningviaweaklydifferentiallyprivateonlinelearning. arXiv preprint arXiv:2407.04898, 2024

    2024 arXiv

  59. [67]

    Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability

    Marcus Hutter. Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability. Springer, Berlin, 2005

    2005

  60. [68]

    Feature reinforcement learning: Part I

    Marcus Hutter. Feature reinforcement learning: Part I. Unstructured MDPs. In J. Artif. Gen. Intell., 2009

    2009

  61. [69]

    CRC Press, 2024

    Marcus Hutter, David Quarel, and Elliot Catt.An Introduction to Uni- versal Artificial Intelligence. CRC Press, 2024

    2024

  62. [70]

    Principal-agent reinforcement learning: Orchestrating AI agents with contracts.arXiv preprint arXiv:2407.18074, 2024

    Dima Ivanov, Paul Dütting, Inbal Talgam-Cohen, Tonghan Wang, and David C Parkes. Principal-agent reinforcement learning: Orchestrating AI agents with contracts.arXiv preprint arXiv:2407.18074, 2024

    2024 arXiv

  63. [71]

    Reward-free exploration for reinforcement learning

    Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-free exploration for reinforcement learning. InInternational Con- ference on Machine Learning, pages 4870–4879. PMLR, 2020

    2020

  64. [72]

    Provably efficient reinforcement learning with linear function approximation

    Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pages 2137–2143. PMLR, 2020

    2020

  65. [73]

    Rational learning leads to Nash equilib- rium

    Ehud Kalai and Ehud Lehrer. Rational learning leads to Nash equilib- rium. Econometrica: Journal of the Econometric Society, pages 1019– 1045, 1993

    1993

  66. [74]

    Gonzalez, Michael I

    Kirthevasan Kandasamy, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. VCG mechanism design with unknown agent values under stochas- tic bandit feedback.J. Mach. Learn. Res., 24:53:1–53:45, 2023

    2023

  67. [75]

    A survey of reinforcement learning from human feedback.arXiv:2312.14925, 10, 2023

    Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback.arXiv:2312.14925, 10, 2023

    2023

  68. [76]

    Mech- anism design in large games: Incentives and privacy

    Michael Kearns, Mallesh Pai, Aaron Roth, and Jonathan Ullman. Mech- anism design in large games: Incentives and privacy. InInnovations in Theoretical Computer Science, pages 403–410, 2014

    2014

  69. [77]

    The rise of the platform economy

    Martin Kenney and John Zysman. The rise of the platform economy. Issues in Science and Technology, 32(3):61, 2016

    2016

  70. [78]

    Bellman goes relational

    Kristian Kersting, Martijn van Otterlo, and Luc De Raedt. Bellman goes relational. In ICML, volume 69. ACM, 2004. 49

    2004

  71. [79]

    Bandit based monte-carlo planning

    Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In Johannes Fürnkranz, Tobias Scheffer, and Myra Spiliopoulou, editors, ECML 2006, pages 282–293. Springer Berlin Heidelberg, 2006

    2006

  72. [80]

    Paying to do better: Games with payments between learning agents.arXiv:2405.20880, 2024

    Yoav Kolumbus, Joe Halpern, and Éva Tardos. Paying to do better: Games with payments between learning agents.arXiv:2405.20880, 2024

    2024

  73. [81]

    Universal codes from switching strategies

    Wouter M Koolen and Steven de Rooij. Universal codes from switching strategies. IEEE Transactions on Information Theory, 59(11):7168–7185, 2013

    2013

  74. [82]

    Krichevsky and V

    R. Krichevsky and V. Trofimov. The performance of universal encoding. IEEE Trans. Inf. Theor., 27(2):199–207, sep 2006

    2006

  75. [83]

    A unified game-theoretic approach to multiagent reinforcement learning.Advances in Neural Information Processing Systems, 30, 2017

    Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning.Advances in Neural Information Processing Systems, 30, 2017

    2017

  76. [84]

    Bandit Algorithms

    Tor Lattimore and Csaba Szepesvári. Bandit Algorithms. Cambridge University Press, 2020

    2020

  77. [85]

    Thomp- son sampling is asymptotically optimal in general environments

    Jan Leike, Tor Lattimore, Laurent Orseau, and Marcus Hutter. Thomp- son sampling is asymptotically optimal in general environments. InPro- ceedings of the Conference on Uncertainty in Artificial Intelligence, pages 417–426, 2016

    2016

  78. [86]

    A formal solution to the grain of truth problem

    Jan Leike, Jessica Taylor, and Benya Fallenstein. A formal solution to the grain of truth problem. InProceedings of the Conference on Uncertainty in Artificial Intelligence, pages 427–436, 2016

    2016

  79. [87]

    Walsh, and Michael L

    Lihong Li, Thomas J. Walsh, and Michael L. Littman. Towards a unified theory of state abstraction for MDPs. In International Symposium on Artificial Intelligence and Mathematics, 2006

    2006

  80. [88]

    An Introduction to Kolmogorov Complexity and Its Applications

    Ming Li and Paul Vitányi. An Introduction to Kolmogorov Complexity and Its Applications. Springer, fourth edition, 2019

    2019

  81. [89]

    Markov games as a framework for multi-agent rein- forcement learning

    Michael L Littman. Markov games as a framework for multi-agent rein- forcement learning. InMachine Learning, pages 157–163. Elsevier, 1994

    1994

  82. [90]

    John W. Lloyd. Logic for Learning: Learning Comprehensible Theories from Structured Data. Springer, 2003

    2003

  83. [91]

    Lloyd and Kee Siong Ng

    John W. Lloyd and Kee Siong Ng. Declarative programming for agent applications. Autonomous Agents Multi Agent Systems, 23(2):224–272, 2011

    2011

  84. [92]

    Pes- simism meets VCG: Learning dynamic mechanism design via offline rein- forcement learning

    Boxiang Lyu, Zhaoran Wang, Mladen Kolar, and Zhuoran Yang. Pes- simism meets VCG: Learning dynamic mechanism design via offline rein- forcement learning. InICML, pages 14601–14638, 2022. 50

    2022

  85. [93]

    Mechanism design via differential privacy

    Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science, pages 94–103. IEEE, 2007

    2007

  86. [94]

    Kenneth Arrow’s last theorem.The Journal of Mechanism and Institution Design, 9(1):7–11, 2024

    Paul Milgrom. Kenneth Arrow’s last theorem.The Journal of Mechanism and Institution Design, 9(1):7–11, 2024

    2024

  87. [95]

    Human-level control through deep re- inforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep re- inforcement learning. Nature, 518(7540):529–533, 2015

    2015

  88. [96]

    Efficient tracking of a growing number of experts

    Jaouad Mourtada and Odalric-Ambrym Maillard. Efficient tracking of a growing number of experts. InInternational Conference on Algorithmic Learning Theory, pages 517–539, 2017

    2017

  89. [97]

    Exploratory engineering in artificial intelligence

    Luke Muehlhauser and Bill Hibbard. Exploratory engineering in artificial intelligence. Communications of the ACM, 57(9):32–34, 2014

    2014

  90. [98]

    Lloyd, and William Uther

    Kee Siong Ng, John W. Lloyd, and William Uther. Probabilistic mod- elling, inferenceandlearningusinglogicaltheories. Annals of Mathematics and Artificial Intelligence, 54:159–205, 2008

    2008

  91. [99]

    Feature re- inforcement learning in practice

    Phuong Minh Nguyen, Peter Sunehag, and Marcus Hutter. Feature re- inforcement learning in practice. In Recent Advances in Reinforcement Learning, volume 7188 ofLNCS, pages 66–77. Springer, 2011

    2011

  92. [100]

    Introduction to mechanism design (for computer scientist)

    Noam Nisan. Introduction to mechanism design (for computer scientist). In Algorithmic Game Theory. Cambridge University Press, 2007

    2007

  93. [101]

    Vazirani, edi- tors

    Noam Nisan, Tim Roughgarden, Éva Tardos, and Vijay V. Vazirani, edi- tors. Algorithmic Game Theory. Cambridge University Press, 2007

    2007

  94. [102]

    A strategy of win-stay, lose-shift that outperforms tit-for-tat in the prisoner’s dilemma game

    Martin Nowak and Karl Sigmund. A strategy of win-stay, lose-shift that outperforms tit-for-tat in the prisoner’s dilemma game. Nature, 364(6432):56–58, 1993

    1993

  95. [103]

    Tit for tat in heterogeneous popula- tions

    Martin A Nowak and Karl Sigmund. Tit for tat in heterogeneous popula- tions. Nature, 355(6357):250–253, 1992

    1992

  96. [104]

    Universal knowledge-seeking agents

    Laurent Orseau. Universal knowledge-seeking agents. Theoretical Com- puter Science, 519:127–139, 2014

    2014

  97. [105]

    Soft-bayes: Prod for mixtures of experts with log-loss

    Laurent Orseau, Tor Lattimore, and Shane Legg. Soft-bayes: Prod for mixtures of experts with log-loss. InInternational Conference on Algo- rithmic Learning Theory, pages 372–399. PMLR, 2017

    2017

  98. [106]

    An Introduction to Game Theory

    Martin J Osborne. An Introduction to Game Theory. Oxford university press, 2004. 51

    2004

  99. [107]

    Governing the Commons: The Evolution of Institutions for Collective Action

    Elinor Ostrom. Governing the Commons: The Evolution of Institutions for Collective Action. Cambridge University Press, 1990

    1990

  100. [108]

    Online mechanisms

    David Parkes. Online mechanisms. InAlgorithmic Game Theory. Cam- bridge University Press, 2007

    2007

  101. [109]

    Parkes and Satinder Singh

    David C. Parkes and Satinder Singh. An MDP-based approach to online mechanism design. InNIPS, pages 791–798. MIT Press, 2003

    2003

  102. [110]

    Economicreasoningandartificial intelligence

    DavidCParkesandMichaelPWellman. Economicreasoningandartificial intelligence. Science, 349(6245):267–272, 2015

    2015

  103. [111]

    Dynamic mechanism design: A myersonian approach.Econometrica, 82(2):601–653, 2014

    Alessandro Pavan, Ilya Segal, and Juuso Toikka. Dynamic mechanism design: A myersonian approach.Econometrica, 82(2):601–653, 2014

    2014

  104. [112]

    Fast swap regret minimization and applications to approximate correlated equilibria

    Binghui Peng and Aviad Rubinstein. Fast swap regret minimization and applications to approximate correlated equilibria. In Proceedings of the 56th Annual ACM Symposium on Theory of Computing, pages 1223–1234, 2024

    2024

  105. [113]

    The Economics of Welfare

    Arthur Pigou. The Economics of Welfare. Routledge, 2002

    2002

  106. [114]

    Learning dynamic mechanisms in un- known environments: A reinforcement learning approach.arXiv preprint arXiv:2202.12797, 2022

    Shuang Qiu, Boxiang Lyu, Qinglin Meng, Zhaoran Wang, Zhuoran Yang, and Michael I Jordan. Learning dynamic mechanisms in un- known environments: A reinforcement learning approach.arXiv preprint arXiv:2202.12797, 2022

    2022 arXiv

  107. [115]

    Statistical Relational Artificial Intelligence: Logic, Probability, and Com- putation

    Luc De Raedt, Kristian Kersting, Sriraam Natarajan, and David Poole. Statistical Relational Artificial Intelligence: Logic, Probability, and Com- putation. Morgan & Claypool Publishers, 2016

    2016

  108. [116]

    Biological auctions with multiple rewards.Pro- ceedings of the Royal Society B: Biological Sciences, 282(1812):20151041, 2015

    Johannes G Reiter, Ayush Kanodia, Raghav Gupta, Martin A Nowak, and Krishnendu Chatterjee. Biological auctions with multiple rewards.Pro- ceedings of the Royal Society B: Biological Sciences, 282(1812):20151041, 2015

    2015

  109. [117]

    The Matthew Effect: How Advantage Begets Further Ad- vantage

    Daniel Rigney. The Matthew Effect: How Advantage Begets Further Ad- vantage. Columbia University Press, 2010

    2010

  110. [118]

    Pen- guin, 2019

    Stuart Russell.Human Compatible: AI and the Problem of Control. Pen- guin, 2019

    2019

  111. [119]

    Symbolic dynamic programming for first-order POMDPs

    Scott Sanner and Kristian Kersting. Symbolic dynamic programming for first-order POMDPs. InProceedings of the Twenty-Fourth Conference on Artificial Intelligence. AAAI Press, 2010

    2010

  112. [120]

    POMDPs make bet- ter hackers: Accounting for uncertainty in penetration testing

    Carlos Sarraute, Olivier Buffet, and Jörg Hoffmann. POMDPs make bet- ter hackers: Accounting for uncertainty in penetration testing. InProceed- ings of the AAAI Conference on Artificial Intelligence, pages 1816–1824, 2021. 52

    2021

  113. [121]

    Autonomous penetration test- ing using reinforcement learning.arXiv preprint arXiv:1905.05965, 2019

    Jonathon Schwartz and Hanna Kurniawati. Autonomous penetration test- ing using reinforcement learning.arXiv preprint arXiv:1905.05965, 2019

    1905 arXiv

  114. [122]

    CUP, 2008

    Yoav Shoham and Kevin Leyton-Brown.Multiagent Systems: Algorith- mic, Game-theoretic, and Logical Foundations. CUP, 2008

    2008

  115. [123]

    A formal theory of inductive inference

    Ray J Solomonoff. A formal theory of inductive inference. Part I.Infor- mation and control, 7(1):1–22, 1964

    1964

  116. [124]

    A formal theory of inductive inference

    Ray J Solomonoff. A formal theory of inductive inference. Part II.Infor- mation and control, 7(2):224–254, 1964

    1964

  117. [125]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. MIT Press, second edition, 2018

    2018

  118. [126]

    Monte carlo tree search: A review of recent modifications and applications

    Maciej Świechowski, Konrad Godlewski, Bartosz Sawicki, and Jacek Mańdziuk. Monte carlo tree search: A review of recent modifications and applications. Artificial Intelligence Review, 56(3):2497–2562, 2023

    2023

  119. [127]

    Catching up faster by switching sooner: A predictive approach to adaptive estimation with an application to the aic–bic dilemma

    Tim van Erven, Peter Grünwald, and Steven De Rooij. Catching up faster by switching sooner: A predictive approach to adaptive estimation with an application to the aic–bic dilemma. Journal of the Royal Statistical Society Series B: Statistical Methodology, 74(3):361–417, 2012

    2012

  120. [128]

    Fast rates in statistical and online learning

    Tim van Erven, Peter Grunwald, Nishant A Mehta, Mark Reid, and Robert Williamson. Fast rates in statistical and online learning. Jour- nal of Machine Learning Research, 2015

    2015

  121. [129]

    Catching up faster in bayesian model selection and model averaging

    Tim van Erven, Steven Rooij, and Peter Grünwald. Catching up faster in bayesian model selection and model averaging. Advances in Neural Information Processing Systems, 20, 2007

    2007

  122. [130]

    Vellambi and Marcus Hutter

    Badri N. Vellambi and Marcus Hutter. Convergence of binarized context- tree weighting for estimating distributions of stationary sources. InIEEE International Symposium on Information Theory, pages 731–735, 2018

    2018

  123. [131]

    Joel Veness, Kee Siong Ng, Marcus Hutter, and Michael H. Bowling. Con- text tree switching. In James A. Storer and Michael W. Marcellin, editors, 2012 Data Compression Conference, pages 327–336. IEEE, 2012

    2012

  124. [132]

    A Monte-Carlo AIXI approximation.Journal of Artificial Intelligence Research, 40:95–142, 2011

    Joel Veness, Kee Siong Ng, Marcus Hutter, William Uther, and David Sil- ver. A Monte-Carlo AIXI approximation.Journal of Artificial Intelligence Research, 40:95–142, 2011

    2011

  125. [133]

    On monte carlo tree search and reinforcement learning.Journal of Artificial Intelligence Research, 60:881–936, 2017

    Tom Vodopivec, Spyridon Samothrakis, and Branko Ster. On monte carlo tree search and reinforcement learning.Journal of Artificial Intelligence Research, 60:881–936, 2017

    2017

  126. [134]

    Switching between two universal source coding algorithms

    Paul AJ Volf and Frans MJ Willems. Switching between two universal source coding algorithms. InProceedings of the Data Compression Con- ference, pages 491–500. IEEE, 1998. 53

    1998

  127. [135]

    Princeton University Press, second edition, 1947

    John von Neumann and Oskar Morgenstern.Theory of Games and Eco- nomic Behavior. Princeton University Press, second edition, 1947

    1947

  128. [136]

    Derandomizing stochastic prediction strategies

    Vladimir Vovk. Derandomizing stochastic prediction strategies. InPro- ceedings of the Annual Conference on Computational Learning Theory, pages 32–44, 1997

    1997

  129. [137]

    Willems, Y.M

    F.M.J. Willems, Y.M. Shtarkov, and T.J. Tjalkens. The context-tree weighting method: basic properties. IEEE Transactions on Information Theory, 41(3):653–664, 1995

    1995

  130. [138]

    Utility function security in artificially intelligent agents

    Roman Yampolskiy. Utility function security in artificially intelligent agents. Journal of Experimental and Theoretical Artificial Intelligence, 26:373–389, 2014

    2014

  131. [139]

    Dynamic knowl- edge injection for AIXI agents

    Samuel Yang-Zhao, Kee Siong Ng, and Marcus Hutter. Dynamic knowl- edge injection for AIXI agents. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38(15), pages 16388–16397, 2024

    2024

  132. [140]

    A direct approxi- mation of AIXI using logical state abstractions.Advances in Neural In- formation Processing Systems, 35:36640–36653, 2022

    Samuel Yang-Zhao, Tianyu Wang, and Kee Siong Ng. A direct approxi- mation of AIXI using logical state abstractions.Advances in Neural In- formation Processing Systems, 35:36640–36653, 2022

    2022

  133. [141]

    Steering no-regret learners to a desired equilibrium.arXiv preprint arXiv:2306.05221, 2023

    Brian Hu Zhang, Gabriele Farina, Ioannis Anagnostides, Federico Caccia- mani, Stephen Marcus McAleer, Andreas Alexander Haupt, Andrea Celli, Nicola Gatti, Vincent Conitzer, and Tuomas Sandholm. Steering no-regret learners to a desired equilibrium.arXiv preprint arXiv:2306.05221, 2023

    2023

  134. [142]

    Multi-agent reinforce- ment learning: A selective overview of theories and algorithms.Handbook of Reinforcement Learning and Control, pages 321–384, 2021

    Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforce- ment learning: A selective overview of theories and algorithms.Handbook of Reinforcement Learning and Control, pages 321–384, 2021

    2021

  135. [143]

    The AI economist: Taxation policy design via two-level deep multiagent reinforcement learning

    Stephan Zheng, Alexander Trott, Sunil Srinivasa, David C Parkes, and Richard Socher. The AI economist: Taxation policy design via two-level deep multiagent reinforcement learning. Science Advances, 8(18):eabk2607, 2022. 54 A Variations of the Social Cost Formulation A.1 Notes ...

    2022

  136. [144]

    A.2 Guaranteed Utility Mechanism We show in this section an adaptation of the Guaranteed Utility Mechanism (GUM) proposed in [38] for our setting

    for a survey of key ideas. A.2 Guaranteed Utility Mechanism We show in this section an adaptation of the Guaranteed Utility Mechanism (GUM) proposed in [38] for our setting. Suppose we have a multi-agent envi- ronmentϕ withk agents and a mechanismM = (f,p 1,...,p k). We assume...

  137. [145]

    EM▷ϕ [ Ui(q∗,i 1:T, q−i 1:T) ] ≥Ci

    For each agenti, we have∀q−i 1:T. EM▷ϕ [ Ui(q∗,i 1:T, q−i 1:T) ] ≥Ci

  138. [146]

    Estimated Returns

    The best possible outcome is given bysup q1:k 1:T ∑ i EM▷ϕ [ Ui(q1:k 1:T) ] =∑ iCi. 61 Theorem 12 ([38]). An interaction protocolM ▷ϕ satisfies the Guaranteed Utility Property if and only if ∑ i sup qi 1:T inf q−i 1:T EM▷ϕ [ Ui(qi 1:T, q−i 1:T) ] = sup q1:k 1:T ∑ i EM▷ϕ [ Ui(q...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.