Pith. sign in

REVIEW 4 major objections 8 minor 2 cited by

Interpretable Risk Mitigation in LLM Agent Systems

T0 review · 4 major / 8 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Steering one sparse-autoencoder feature moves an LLM agent's defection probability from 47% to 75%.

desk verdict A credible proof-of-concept that SAE steering shifts LLM cooperation, but the headline 28pp figure is a selected uniform average over fixed histories, not a measurement of real game behavior. read the letter →

arxiv 2505.10670 v1 pith:RYIDSCHI submitted 2025-05-15 cs.AI cs.CYcs.GT

Jan Chojnacki This is my paper · ORCID
classification cs.AIcs.CYcs.GT
keywords LLMagentsIteratedPrisoner'sDilemmasparseautoencodersfeaturesteeringinterpretabilityAIalignmentdefectionprobabilitysafety
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language model agents behave unreliably in strategic settings: with an identical prompt and game history, a small non-zero temperature can make a Mixtral agent's defection probability jump from near 0.1 to near 0.9. The paper's proposed intervention is to modify the model's residual stream at inference time by adding a decoded sparse-autoencoder feature, a method it argues is independent of both the particular game and the prompt. In an Iterated Prisoner's Dilemma worded as an investment game, steering the 'sacrifice' feature of Gemma-2b moves the average fourth-round defection probability from 22% to 69%, and steering the monosemantic 'good faith/bad faith' feature of LLaMA3-IT-8B moves it by 28 percentage points, between 47% and 75%. The paper positions this as a step toward transparent, inference-time alignment of agent behavior, with game-theoretic evaluation serving as the testbed.

What carries the argument

Sparse autoencoder feature steering is the mechanism: at inference time, the model's residual-stream activation at a chosen layer is modified by adding the decoded feature vector, $x'_l = x_l + \omega W_{\mathrm{dec}}(f_{ID})$. The paper relies on pre-trained SAEs whose features are sparse and often monosemantic, and selects features using two criteria: the feature's activation-density histogram (bimodal for single-meaning features) and the preservation of $P(\mathrm{green}) + P(\mathrm{blue}) \approx 1$ in the next-token distribution. The steering strength $\omega$ is chosen empirically for each model. This machinery carries the argument because it connects a human-readable concept such as 'good faith' to a measurable change in strategic behavior without retraining or prompt redesign.

What would settle it

Run the same steering experiment with full autoregressive decoding and sample complete four-round games, then compare actual defection rates to $P(\mathrm{blue})$; if the 47%-to-75% separation does not survive full-game sampling, or if under steering $P(\mathrm{green})+P(\mathrm{blue})$ drops far below 1, the next-token proxy is falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper reports that a single sparse-autoencoder feature can act as a semantic steering dial for an LLM agent's strategy. In the LLaMA3-IT-8B model, the feature labeled 'good faith/bad faith' (index 30695) is monosemantic, and steering it in the residual stream changes the average fourth-round defection probability by 28 percentage points, from 47% to 75%, across all 64 possible three-round histories. The change is approximately monotonic in steering strength for each history. The same procedure applied to the 'sacrifice' feature of Gemma-2b moves the average defection probability from 22% to 69%, and an 'environment' feature appears to steer both the Gemma and LLaMA families. The paper concludes that game-theoretic evaluation, combined with representation-steering alignment, may generalize to real-world applications on end-user devices and embodied platforms.

Load-bearing premise

The load-bearing assumption is that the next-token probability of the 'blue' token on a fixed fourth-round prompt equals the agent's real defection probability; if decoding and multi-turn dynamics break that link, the reported 28-point steering effect may not describe actual agent strategy.

Editorial extensions

If this is right

  • A monosemantic feature can serve as a continuous strategy dial: the 'good faith/bad faith' direction in LLaMA3-IT-8B moves average fourth-round defection between 47% and 75% with a near-monotonic response.
  • Because the intervention happens in the residual stream at inference time, it can be applied to open models with publicly available sparse autoencoders, avoiding fine-tuning cost and catastrophic forgetting.
  • Abstract features such as 'sacrifice' and 'environment' steer behavior even though they are not game tokens, and at least one feature ('environment') has effects across model families.
  • Feature effectiveness is selective: the monosemantic 'trust' feature leaves defection nearly unchanged (47% to 50%), so automatic feature selection matters for any deployment.
  • The hypothesized generalization is that the same steering procedure could align agents in real-world settings beyond the toy game, such as consumer devices and embodied platforms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the 28-percentage-point result is measured on next-token probability, not on full game trajectories; the natural next experiment is to sample complete games and see whether actual defection rates track $P(\mathrm{blue})$ under the same steering strengths.
  • The near-monotonic good-faith axis suggests the model has an internal dimension that links negotiation 'faith' to cooperation; if that axis is general, the same feature should shift behavior in other trust- and negotiation-based games, which is testable.
  • Because polysemantic features like 'sacrifice' also steer strongly, the practical extension would be an automated pre-screen that ranks features by activation-density bimodality and by the stability of $P(\mathrm{green})+P(\mathrm{blue})$, replacing manual token inspection.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes an inference-time steering method for LLM agents in an Iterated Prisoner's Dilemma (IPD) environment. The authors use sparse autoencoder (SAE) features on the residual streams of Gemma-2B, Gemma2-2B, and LLaMA3-IT-8B, and measure the effect of positive and negative steering on the next-token probability of the defection token ('blue') on the fourth round, averaged uniformly over all 64 possible three-round histories. They report that steering the LLaMA3-IT-8B 'good/bad faith' feature (index 30695) shifts this average from 47% to 75% (or the reverse, depending on the passage), that a 'sacrifice' feature in Gemma-2B shifts it by 47 percentage points, and that these effects are consistent with the semantic interpretations of the features. The paper also discusses monosemanticity and argues that such steering could generalize to real-world agent alignment.

Significance. If the quantitative claims were validated in full multi-turn play, this would be a valuable empirical demonstration that an interpretable feature-level intervention can shift an LLM agent's game behavior in a semantically coherent direction. The use of public SAEs, the comparison across three model families, and the promise of reproducible code and data are strengths. However, as reported, the headline effect is a uniform average of next-token probabilities over fixed prompts, not an expected defection rate under the steered policy in an actual game, so the central quantitative claim needs revalidation before the significance of the result can be assessed.

major comments (4)
  1. [§4.2, §5] The headline 28 percentage point shift is an average of P('blue') over all 64 fixed histories, i.e., (1/64) Σ_h P_{θ,w}(blue|h), where h enumerates all possible three-round histories. In an actual IPD, the expected fourth-round defection probability is Σ_h P_{θ,w}(h) P_{θ,w}(blue|h), with P_{θ,w}(h) induced by the steered policy. Steering with positive and negative w changes the agent's own earlier actions and therefore the distribution over histories, so a uniform average over all 64 histories is not an estimate of the agent's defection rate in the game. Section 4.2 explicitly replaces full IPD simulation with this exhaustive sweep, and Section 5 then reports the result as 'average defection probability'. The authors should either re-run full game simulations under both steering directions (as in the §4.1 preliminary study) or clearly relabel the result as a fixed-prompt next-token metric and provide a reweighted estimate.
  2. [§4.2, §5] The paper equates the next-token probability P('blue') with the agent's defection probability, but the agent's actual choice is a generated response, often in a specified format such as JSON for the Mixtral prompt. Decoding with temperature, sampling, or multi-token outputs such as 'project blue' may decouple the top next-token probability from the final action. The manuscript provides no validation that P('blue') matches the probability of selecting 'project blue' in a complete response. At minimum, the authors should report agreement between the next-token probability and decoded/sampled actions, or explicitly restrict the claim to the next-token distribution.
  3. [§5 vs Table 1] There is an internal inconsistency in the reported direction of the effect for the 'good/bad faith' feature (index 30695). Section 5 states ⟨P('blue'|+good/bad faith)⟩ = 47% and ⟨P('blue'|−good/bad faith)⟩ = 75%, whereas Table 1 lists 0.75 and 0.47 for the same row. These two statements imply opposite steering directions. The abstract's claim that the 'good-faith negotiation feature lowers the average defection probability' depends on which convention is correct, and this inconsistency must be resolved before the central claim can be evaluated.
  4. [§4.2, §5] The magnitude of the reported effects is selected from a large scan and depends on the steering strength w, which is chosen empirically per model so that P('green')+P('blue')≈1. The paper scans thousands of features (2,339 for Gemma-2B), identifies features in the tails of the δ distribution, and then reports the largest effects without multiple-comparison correction or confidence intervals. Because w is a free parameter and the headline features are selected post hoc, the 28pp and 47pp numbers are not parameter-free estimates of intervention efficacy. The authors should report the distribution of δ with variance estimates, show the sensitivity of the headline features to w (Figure 7 does this for one feature), and state the total number of features scanned for each model.
minor comments (8)
  1. [§1] The sentence 'cannot be directly applied tor high-stakes tasks' contains a typo; 'tor' should be 'to'.
  2. [§4.2] The text states w ∈ (−10, 8) but does not define the scale or normalization of w; Figure 3 notes that w is 'scaled down by a factor of 10', which should be explained in the method.
  3. [Appendix A.3] The text reports ⟨P('blue'|−blue)⟩ = 0.04% for the Gemma-2b 'blue' feature, while Table 1 gives 0.04; if this is a probability, 0.04 is 4%, not 0.04%, and the formatting should be made consistent.
  4. [Appendix A.4] Figure captions 8-11 refer to 'Gamma-2b' instead of 'Gemma-2b'; please correct the model name.
  5. [§5, Table 1] Section 5 says 'we also find the 'green' and 'blue' features' for LLaMA3, but Table 1 lists only 'Blue' for that model; the text and table should agree.
  6. [§5, Appendix A.3] The phrase 'denouncement of violence' in Section 5 and 'denounce of violence' in Appendix A.3 are inconsistent; please standardize.
  7. [Equation (1)] The sign convention for δ is not stated explicitly for the headline features; for the 'sacrifice' feature, the reported 47 percentage point effect is an absolute change, but the sign of δ is left implicit. Please state δ for each headline feature.
  8. [Appendix A.2] The Gemma prompt template shown ends at 'round: 2' and does not display the exact formatting for the third round or the current-round request; please confirm that the full 64-history prompt template is specified in the released code.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the 28pp steering effect is a measured empirical outcome, not a derived or self-referential quantity.

full rationale

The paper's central claim is an empirical measurement rather than a derivation. The SAE features are public external artifacts (GemmaScope and Hugging Face residual-stream SAEs), the steering operation is explicitly defined as x'_l = x_l + ω W_dec(f_ID), and the reported 28 percentage point shift is the difference between measured next-token probabilities averaged over 64 fixed three-round histories. No equation defines the result in terms of its own inputs, no fitted parameter is relabeled as a prediction, and no load-bearing uniqueness or ansatz argument is imported from the authors' prior work. The only self-citation ([31], on ITI affecting unrelated QA benchmarks) supports a side remark and is not load-bearing; it is also an externally published, reproducible result. The post-hoc selection of tail features and the experimentally chosen steering strengths are statistical-selection concerns, not circularity, because the paper does not present these as predictions and the underlying measurements are fully reproducible from the published code and data. Accordingly, no specific circular reduction can be exhibited, so the appropriate finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central result depends on one fitted steering strength w, the interpretive labeling of SAE features, and the proxy of next-token probability for defection. No new physical or conceptual entities are introduced.

free parameters (1)
  • Steering strength w = not reported; chosen experimentally per model
    Section 4.2 footnote 3 says 'We found these values of steering strength experimentally, selecting appropriate steering strengths separately for each model.' The reported 47% and 75% probabilities depend on these choices.
assumptions (3)
  • domain assumption SAE features are monosemantic and their top-activation labels reflect causal semantics
    Section 6.1 uses activation density histograms and top-token tables to label feature 30695 as 'good faith/bad faith' and to argue that monosemantic steering is safer and generalizes beyond the IPD.
  • domain assumption Next-token probability of the single 'blue' token equals the agent's defection probability
    Section 4.2 measures only the last-token distribution on a fixed fourth-round prompt and does not sample full games or calibrate the probabilities against actual decoding.
  • ad hoc to paper P('green')+P('blue')≈1 is an appropriate criterion for game understanding and for choosing steering strengths
    Section 4.2 states that this near-sum-to-one condition 'is the desired behavior, indicating that the model understands the game context'; it is used only inside this study to justify the empirically chosen w.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Interpretable Risk Mitigation in LLM Agent Systems." pith.science (2026). https://pith.science/paper/RYIDSCHI

@misc{pith2026250510670,
  author       = {Pith},
  title        = {Pith review of: Interpretable Risk Mitigation in LLM Agent Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RYIDSCHI}},
  note         = {Machine review of arXiv:2505.10670}
}
read the original abstract

Autonomous agents powered by large language models (LLMs) enable novel use cases in domains where responsible action is increasingly important. Yet the inherent unpredictability of LLMs raises safety concerns about agent reliability. In this work, we explore agent behaviour in a toy, game-theoretic environment based on a variation of the Iterated Prisoner's Dilemma. We introduce a strategy-modification method-independent of both the game and the prompt-by steering the residual stream with interpretable features extracted from a sparse autoencoder latent space. Steering with the good-faith negotiation feature lowers the average defection probability by 28 percentage points. We also identify feasible steering ranges for several open-source LLM agents. Finally, we hypothesise that game-theoretic evaluation of LLM agents, combined with representation-steering alignment, can generalise to real-world applications on end-user devices and embodied platforms.

Figures

Figures reproduced from arXiv: 2505.10670 by the authors.

Figure 1
Figure 1. Example of generation steering of the Gemma-2-9B-it model, layer 31-gemmascope-res [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Schematic presentation of the transformer generation steering with SAE. The SAE network [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Steering result of ‘sacrifice’ (7155) direction. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: (left) Histogram of δ values. Negative values suggest that steering with a given features leads to more cooperative actions, while positive values correspond to features that steer the agent towards defection. (right) Comparison of defection probabilities reached with …
Figure 5
Figure 5. Figure 5: Defection probabilities p1defect for two simulated strategies: (left) win-stay lose-change strategy, (right) Mixtral 7x8B choices. For each strategy, the opponent is randomly defecting with probability p2defect. A.2 Prompts Prompt used in the preliminary Mixtral experi…
Figure 6
Figure 6. Figure 6: (left) Scores averaged with respect to round length. Y-axis corresponds to number of points the win-stay lose-change strategy scored per turn. X-axis is the deception probability of Player 2 (random). (right) Gaussian Mixture clusters show strategy patterns of Mixtral …
Figure 7
Figure 7. Figure 7: llama-3 steering with good/bad faith feature. [PITH_FULL_IMAGE:figures/full_fig_p026_7.png]
Figure 8
Figure 8. Figure 8: Activation density distribution - ‘Green’ feature of Gamma-2b. Top 10 activations on the [PITH_FULL_IMAGE:figures/full_fig_p027_8.png]
Figure 9
Figure 9. Figure 9: Activation density distribution - ‘blue’ feature of Gamma-2b. Top 10 activations on the [PITH_FULL_IMAGE:figures/full_fig_p027_9.png]
Figure 10
Figure 10. Figure 10: Activation density distribution - ‘trust’ feature of Gamma-2b. Top 10 activations on the [PITH_FULL_IMAGE:figures/full_fig_p028_10.png]
Figure 11
Figure 11. Figure 11: Activation density distribution - polysemantic feature ‘sacrifice’ of Gamma-2b. Top 10 [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]
Figure 12
Figure 12. Figure 12: Activation density distribution - polysemantic feature ‘abuse’ of LLaMA3-it-8b. Top 10 [PITH_FULL_IMAGE:figures/full_fig_p028_12.png]
Figure 13
Figure 13. Figure 13: Activation density distribution - feature ‘environment’ of Gemma2-2b. Top 10 activations [PITH_FULL_IMAGE:figures/full_fig_p029_13.png]
Figure 14
Figure 14. Figure 14: Activation density distribution - feature ‘good faith’ of LLaMA3-it-8b Top 10 activations [PITH_FULL_IMAGE:figures/full_fig_p029_14.png]
Figure 15
Figure 15. Figure 15: Activation density distribution - feature ‘good faith’ of LLaMA3-it-8b. Top 10 activations [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomy

    cs.AI 2025-06 conditional novelty 4.0 of 10

    A position paper arguing that LLM-based human-agent systems, not fully autonomous agents, should be the immediate goal for AI development.

  2. LLM Harms: A Taxonomy and Discussion

    cs.CY 2025-12 unverdicted novelty 3.0 of 10

    This paper proposes a taxonomy of LLM harms in five categories and suggests mitigation strategies plus a dynamic auditing system for responsible development.

Reference graph

Works this paper leans on

77 extracted references · 34 canonical work pages · cited by 2 Pith papers

  1. [1]

    Artificial intelligence and the future of work: Evidence from OECD countries

    Philipp Ager, Jan Bena, Maximiliano Coutin-Churchman, Julian Leon, and David Wiczer. Artificial intelligence and the future of work: Evidence from OECD countries. OECD Publishing, 2020

    work page 2020

  2. [2]

    Dai, Chelsea Finn, Justin Fu, Kanishka Gopalakrishnan, et al

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Yunfei Chow, Colin Chu, Alice X. Dai, Chelsea Finn, Justin Fu, Kanishka Gopalakrishnan, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning (CoRL) , 2022

    work page 2022

  3. [3]

    Mistral 7b: Open foundation models, 2023

    Mistral AI. Mistral 7b: Open foundation models, 2023. https://mistral.ai/technology/ #models

    work page 2023

  4. [4]

    Playing repeated games with large language models

    Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. ArXiv, abs/2305.16867, 2023. URL https://api.semanticscholar.org/CorpusID:258947115

    arXiv 2023

  5. [5]

    Concrete problems in AI safety

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man´ e. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 , 2016

    arXiv 2016

  6. [6]

    R. Axelrod. The Evolution of Cooperation. Basic books. Basic Books, 1984. ISBN 9780465021215. URL https://books.google.pl/books?id=NJZBCGbNs98C

    work page 1984

  7. [7]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Neal DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022

    arXiv 2022

  8. [8]

    Emergent tool use from multi-agent autocurricula

    Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In Proceedings of the International Conference on Learning Representations, 2020

    work page 2020

Show all 77 references
  1. [9]

    Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell

    Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623. ACM, 2021

    2021

  2. [10]

    Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, et al

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021

    2021 arXiv

  3. [11]

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L. Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLe...

    2023

  4. [12]

    RT-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Yevgen Chebotar, Jacky Liang, Chelsea Finn, Karol Hausman, Alex Irpan, Julian Ibarz, and Sergey Levine. RT-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 , 2022

    2022 arXiv

  5. [13]

    Playing games with gpt: What can we learn about a large language model from canonical strategic games? SSRN Electronic Journal, 2023

    Philip Brookins and Jason Debacker. Playing games with gpt: What can we learn about a large language model from canonical strategic games? SSRN Electronic Journal, 2023. URL https://api.semanticscholar.org/CorpusID:259714625

    2023

  6. [14]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 18...

    1901

  7. [15]

    What can machine learning do? workforce implications

    Erik Brynjolfsson and Tom Mitchell. What can machine learning do? workforce implications. Science, 358(6370):1530–1534, 2017

    2017

  8. [16]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021 arXiv

  9. [17]

    Instigating cooperation among llm agents using adaptive information modulation

    Qiliang Chen, Alireza Sepehr Ilami, Nunzio Lor` e, and Babak Heydari. Instigating cooperation among llm agents using adaptive information modulation. ArXiv, abs/2409.10372, 2024. URL https://api.semanticscholar.org/CorpusID:272690037

    2024

  10. [19]

    Sparse autoencoders find highly interpretable features in language models, 2023

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023. URL https://arxiv. org/abs/2309.08600

    2023 arXiv

  11. [20]

    Reinforcement learning in a prisoner’s dilemma

    Arthur Dolgopolov. Reinforcement learning in a prisoner’s dilemma. Games Econ. Behav. , 144: 84–103, 2024. URL https://api.semanticscholar.org/CorpusID:267111895

    2024

  12. [21]

    Toy models of superposition

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Baker Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of sup...

    2022 arXiv

  13. [22]

    Toy models of superposition

    Nelson Elhage, Neel Nanda, Catherine Olsson, et al. Toy models of superposition. arXiv preprint arXiv:2210.04866, 2022

    2022 arXiv

  14. [23]

    Michaud, Wes Gurnee, and Max Tegmark

    Joshua Engels, Isaac Liao, Eric J. Michaud, Wes Gurnee, and Max Tegmark. Not all language model features are linear. ArXiv, abs/2405.14860, 2024. URL https://api.semanticscholar. org/CorpusID:269983112

    2024 arXiv

  15. [24]

    Some experimental games

    Merrill M Flood. Some experimental games. Management Science, 5(1):5–26, 1958

    1958

  16. [25]

    Nicer than humans: How do large language models behave in the prisoner’s dilemma? ArXiv, abs/2406.13605, 2024

    Nicol’o Fontana, Francesco Pierri, and Luca Maria Aiello. Nicer than humans: How do large language models behave in the prisoner’s dilemma? ArXiv, abs/2406.13605, 2024. URL https://api.semanticscholar.org/CorpusID:270619642

    2024 arXiv

  17. [26]

    Artificial intelligence, values, and alignment

    Iason Gabriel. Artificial intelligence, values, and alignment. Minds and Machines , 30:411–437, 09 2020. doi: 10.1007/s11023-020-09539-2

    2020 doi

  18. [27]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravanku- mar, Artem Korenev, A...

    2024 arXiv

  19. [28]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https: //arxiv.org/abs/2009.03300

    2021 arXiv

  20. [29]

    Measuring mathematical problem solving with the math dataset,

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset,

  21. [30]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), ...

    2024

  22. [31]

    Non-linear inference time intervention: Improving llm truthfulness

    Jakub Ho´ sci lowicz, Adam Wiacek, Jan Chojnacki, Adam Cie´ slak, Leszek Micho´ n, Vitalii Ur- banevych, and Artur Janicki. Non-linear inference time intervention: Improving llm truthfulness. Interspeech 2024, 2024. URL https://api.semanticscholar.org/CorpusID:268724230

    2024

  23. [32]

    Towards reasoning in large language models: A survey, 2023

    Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey, 2023. URL https://arxiv.org/abs/2212.10403

    2023 arXiv

  24. [33]

    Large language models for uavs: Current state and pathways to the future

    Shumaila Javaid, Hamza Fahim, Bin He, and Nasir Saeed. Large language models for uavs: Current state and pathways to the future. IEEE Open Journal of Vehicular Technology , 5: 1166–1192, 2024. URL https://api.semanticscholar.org/CorpusID:269588084

    2024

  25. [34]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L´ elio Renard Lavaud, Lucile Saulnier, Marie-An...

    2024 arXiv

  26. [35]

    llama-3-8b-it-res (revision 53425c3), 2024

    Jiatong Han. llama-3-8b-it-res (revision 53425c3), 2024. URL https://huggingface.co/ Juliushanhanhan/llama-3-8b-it-res

    2024

  27. [36]

    Curt Tigges Joseph Bloom and David Chanin. Saelens. https://github.com/jbloomAus/ SAELens, 2024. 15

    2024

  28. [37]

    Martin, Hans-Theo Normann, and T

    Bernhard Kasberger, Simon P. Martin, Hans-Theo Normann, and T. Werner. Algorithmic cooperation. SSRN Electronic Journal , 2023. URL https://api.semanticscholar.org/ CorpusID:257673245

    2023

  29. [38]

    Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Scie...

    2017

  30. [39]

    Inference-time intervention: Eliciting truthful answers from a language model

    Kenneth Li, Oam Patel, Fernanda Vi’egas, Hans-R¨ udiger Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. ArXiv, abs/2306.03341, 2023. URL https://api.semanticscholar.org/CorpusID:259088877

    2023 arXiv

  31. [40]

    Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2, 2024

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, J´ anos Kram´ ar, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2, 2024. URL https://arxiv. org/abs/2408.05147

    2024 arXiv

  32. [41]

    The mythos of model interpretability

    Zachary Lipton. The mythos of model interpretability. Communications of the ACM , 61, 10

  33. [42]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 55(9):1–35, 2023

    2023

  34. [43]

    Large model strategic thinking, small model efficiency: Transferring theory of mind in large language models

    Nunzio Lor` e, Alireza Sepehr Ilami, and Babak Heydari. Large model strategic thinking, small model efficiency: Transferring theory of mind in large language models. ArXiv, abs/2408.05241,

    arXiv

  35. [44]

    Linguistic regularities in continuous space word representations

    Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Lucy Vanderwende, Hal Daum´ e III, and Katrin Kirchhoff, editors, Proceedings of the 2013 Conference of the North American Chapter of the Association for Comput...

    2013

  36. [45]

    Large language models: A survey, 2024

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey, 2024. URL https://arxiv. org/abs/2402.06196

    2024 arXiv

  37. [46]

    A strategy of win-stay, lose-shift that outperforms tit-for-tat in the prisoner’s dilemma game

    Martin Nowak and Karl Sigmund. A strategy of win-stay, lose-shift that outperforms tit-for-tat in the prisoner’s dilemma game. Nature, 364(6432):56–58, 1993

    1993

  38. [47]

    Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022

    2022 arXiv

  39. [48]

    Cooperation: A systematic review of how to enable agent to circumvent the prisoner’s dilemma

    Jiateng Pan, Atsushi Yoshikawa, and Masayuki Yamamura. Cooperation: A systematic review of how to enable agent to circumvent the prisoner’s dilemma. SHS Web of Conferences , 178, 10 2023. doi: 10.1051/shsconf/202317803005. 16

    2023

  40. [49]

    Wang, Linxi Wang, Alex Wang, Allie He, Qian Liao, David Kempe, et al

    Joon Sung Park, Michael Shum, Joseph Xu, Kenneth Zhang, Roger G. Wang, Linxi Wang, Alex Wang, Allie He, Qian Liao, David Kempe, et al. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442 , 2023

    2023 arXiv

  41. [50]

    Tinyclick: Single-turn agent for empowering gui automation, 2024

    Pawel Pawlowski, Krystian Zawistowski, Wojciech Lapacz, Marcin Skorupa, Adam Wiacek, Sebastien Postansque, and Jakub Hoscilowicz. Tinyclick: Single-turn agent for empowering gui automation, 2024. URL https://arxiv.org/abs/2410.11871

    2024 arXiv

  42. [51]

    Steve Phelps and Yvan I. Russell. The machine psychology of cooperation: Can gpt models operationalise prompts for altruism, cooperation, competitiveness and selfishness in economic games? 2023. URL https://api.semanticscholar.org/CorpusID:258685424

    2023

  43. [52]

    Effect of private deliberation: Deception of large language models in game play

    Kristijan Poje, Mario Brcic, Mihael Kovavc, and Marina Bagic Babac. Effect of private deliberation: Deception of large language models in game play. Entropy, 26, 2024. URL https://api.semanticscholar.org/CorpusID:270613663

    2024

  44. [53]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof qa benchmark, 2023. URL https://arxiv.org/abs/2311.12022

    2023 arXiv

  45. [54]

    A primer in BERTology: What we know about how BERT works

    Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics , 8:842– 866, 2020. doi: 10.1162/tacl a 00349. URL https://aclanthology.org/2020.tacl-1.54

    2020 doi

  46. [55]

    Research priorities for robust and beneficial artificial intelligence

    Stuart Russell, Daniel Dewey, and Max Tegmark. Research priorities for robust and beneficial artificial intelligence. AI Magazine, 36(4):105–114, 2015

    2015

  47. [56]

    P. Scharre. Army of None: Autonomous Weapons and the Future of War . WW Norton, 2019. ISBN 9780393356588. URL https://books.google.se/books?id=kF2NEAAAQBAJ

    2019

  48. [57]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Nathan Scales, David Dohan, Justin Gilmer, Richard Tanburn, Vedant Misra, Kyle Mills, Jos´ e Susano Pinto, Nathanael Sch¨ arli, et al. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761 , 2023

    2023 arXiv

  49. [58]

    An evolutionary model of personality traits related to cooperative behavior using a large language model

    Reiji Suzuki and Takaya Arita. An evolutionary model of personality traits related to cooperative behavior using a large language model. Scientific Reports, 14, 2023. URL https://api. semanticscholar.org/CorpusID:263830498

    2023

  50. [59]

    A comparative analysis of the definitions of autonomous weapons systems

    Mariarosaria Taddeo and Alexander Blanchard. A comparative analysis of the definitions of autonomous weapons systems. Science and Engineering Ethics , 28(5):1–22, 2022. doi: 10.1007/s11948-022-00392-3

    2022 doi

  51. [60]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi` ere, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, L´ eonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, ...

    2024 arXiv

  52. [61]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L´ eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram´ e, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le...

    2024 arXiv

  53. [62]

    Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet

    Adly Templeton. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Anthropic, 2024

    2024

  54. [63]

    Moral alignment for llm agents

    Elizaveta Tennant, Stephen Hailes, and Mirco Musolesi. Moral alignment for llm agents. 2024. URL https://api.semanticscholar.org/CorpusID:273026159

    2024

  55. [64]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation lang...

    2023 arXiv

  56. [65]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  57. [66]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019. URL https://arxiv.org/abs/1804.07461

    2019 arXiv

  58. [67]

    Mobile-agent: Autonomous multi-modal mobile device agent with visual perception,

    Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception,

  59. [68]

    Dai, and Quoc V Le

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL https://openreview.net/ forum?id=gEZrGCozdqR

    2022

  60. [69]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https://arxiv.org/abs/2201.11903

    2023 arXiv

  61. [70]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Howard Chen Yu, Shunyu Cao, Yuan Zhao, Dong Yu, Hanjun Sun, Ofir Press, Mike Lewis, Yuan Cao, Karthik Narasimhan, et al. ReAct: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 , 2022

    arXiv

  62. [71]

    Appagent: Multimodal agents as smartphone users, 2023

    Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users, 2023. URL https://arxiv. org/abs/2312.13771

    2023 arXiv

  63. [72]

    URL https://arxiv.org/abs/2401.16158

    arXiv

  64. [73]

    Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico...

    2023 arXiv

  65. [77]

    You only look at screens: Multimodal chain-of-action agents, 2024

    Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents, 2024. URL https://arxiv.org/abs/2309.11436. 19

    2024 arXiv

  66. [2016]

    doi: 10.1145/3233231

    doi

  67. [2021]

    URL https://arxiv.org/abs/2103.03874

    arXiv

  68. [2023]

    URL https://api.semanticscholar.org/CorpusID:261934663

  69. [2024]

    URL https://api.semanticscholar.org/CorpusID:271854780

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.