Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Because we have LLMs, we Can and Should Pursue Agentic Interpretability

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large language models open a new path to interpretability: letting the model itself teach humans through multi-turn conversation, building and using a mental model of the user.

desk verdict A well-framed position paper that names a real research program but leans heavily on the untested assumption that LLMs can teach us their own concepts reliably. read the letter →

arxiv 2506.12152 v1 pith:YFPJSDAX submitted 2025-06-13 cs.AI

Been Kim , John Hewitt , Neel Nanda , Noah Fiedel , Oyvind Tafjord This is my paper
classification cs.AI
keywords agenticinterpretabilitylargelanguagemodelsmentalexplainableAIhuman-AIinteractionsuperhumanconceptsevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that large language models open up a new mode of interpretability called agentic interpretability: instead of only inspecting a model's internals from outside, we should converse with the model, letting it proactively explain itself while building a mental model of what we already know and where we are confused. The central claim is that this mutual mental-model building lets humans learn machine concepts, including superhuman ones, in a way that static inspective methods cannot. The paper argues this is both possible, because LLMs converse coherently, and urgent, because human understanding risks falling behind rapid model advances. It also spells out the costs, such as loss of completeness and vulnerability to deceptive models, and proposes evaluation proxies based on whether the human can predict the model's behavior or improve the model's behavior on a target concept.

What carries the argument

The load-bearing object is the mutual mental model in an extended dialogue: the LLM maintains, implicitly or explicitly, a representation of the user's knowledge, confusions, and goals, and uses it to choose proactive explanations, while the human builds a mental model of how the LLM behaves. This teacher-student dynamic is supported by the concept function $f : (x, y) \mapsto c$, which formalizes the property to be understood, and by the two evaluation regimes of case improve and case learn that give the program testable targets. The paper leans on the Zone of Proximal Development to argue that such dialogue can scaffold learning just beyond the user's current ability.

What would settle it

A controlled study could settle this: take a machine concept, such as the model's notion of a good response, let one group of humans learn it through a multi-turn agentic dialogue, and give a control group a single static written explanation. If the dialogue group does not predict the concept on new examples more accurately, or does not improve the model's behavior more effectively, than the control group, agentic interpretability's core benefit fails to appear. A sharper version would test whether the LLM's self-explanations of its own failures, such as why appending a token enables a jailbreak, are actually correct when checked against causal interventions on the model's internals.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a definition and a research program: a method pursues agentic interpretability if it proactively assists human understanding in a multi-turn interactive process by developing and leveraging a mental model of the user, which in turn enables humans to develop better mental models of the LLM. The authors argue that current LLMs' conversational ability makes this feasible for the first time, and that it should be pursued as a complement to inspective interpretability, particularly for teaching humans the model's own concepts, some of which may be superhuman. They identify two evaluation settings: case improve, where the goal is to change the model's behavior on a human-defined concept, and case learn, where the goal is for humans to accurately predict the model's concept on new examples.

Load-bearing premise

The whole program rests on the premise that an LLM can reliably explain its own behavior to a human and can build a useful model of that human's knowledge, even though the paper concedes models have little meta-understanding of their own successes and failures.

Editorial extensions

If this is right

  • Model development cycles become collaborative dialogues: a meta-model informed by a project's full history could proactively suggest hypotheses and debugging strategies tailored to a developer's current understanding.
  • Humans could learn superhuman machine concepts, such as chess strategies discovered by AlphaZero, through Socratic tutoring that paces content to the learner's zone of proximal development.
  • Agentic interpretability could be combined with mechanistic intervention, called open-model surgery, where a model must reconcile its explanations with the effects of direct internal manipulation, potentially exposing inconsistencies in deceptive models.
  • Evaluation of interpretability shifts from static artifacts to end-task metrics: whether humans predict the model's concept on new examples (case learn) or successfully steer the model toward a human-defined concept (case improve).
  • LLM proxies for human users in the loop could make evaluation cheap and fast, but the paper insists the ultimate test remains whether real humans' mental models improve.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If agentic interpretability works, the boundary between explanation and education dissolves: the value of an explanation is measured by whether the learner can act on it, which suggests interpretability evaluation should borrow methodologies from intelligent tutoring systems, such as knowledge tracing.
  • The paper's admission that models lack meta-understanding suggests a testable research direction: agentic dialogue may function as a co-discovery process where the LLM discovers its own behavioral rules through the human's questions, rather than reading them off its internals, making agentic interpretability a form of behavioral science of machines.
  • If human mental models can be built faster than models evolve, agentic interpretability could become a mechanism for democratic oversight of AI, since the goal is explicitly to keep lay people from being excluded by opacity, and a concrete extension would be building model-literacy curricula for non-experts using these dialogues.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that LLMs' conversational capabilities enable a new mode of interpretability, 'agentic interpretability,' in which the model proactively assists human understanding through multi-turn interaction, builds a mental model of the user, and thereby helps the user build a better mental model of the model. It contrasts this with 'inspective' interpretability, which opens the black box, and discusses applications including model development, teaching superhuman concepts (e.g., AlphaZero chess concepts), and mechanistic interpretability via interactive 'open surgery.' The paper also addresses objections such as deceptive or uncooperative models, analyzes trade-offs such as completeness and computational hill-climbing, and proposes evaluation strategies based on concept prediction and end-task metrics. The central normative claim, stated in the title, is that because we have LLMs, we can and should pursue agentic interpretability. The paper is explicitly a position statement with no experiments or derivations; it does, however, candidly acknowledge limitations, including models' limited meta-understanding and the human-entangled nature of evaluation.

Significance. If the proposal is accepted, it reframes interpretability as a collaborative, pedagogical process rather than artifact generation, connecting interpretability to cognitive science, HCI, and concept discovery in ways that could shape future research agendas. The paper offers a clear definition of agentic interpretability, a useful distinction between 'case improve' and 'case learn' goals, and a thoughtful discussion of evaluation proxies. Its candid treatment of deception, incompleteness, and the difficulty of human evaluation is a strength, as is its integration of related work from outside interpretability. However, the paper's central 'can and should' claim rests on empirical assumptions about LLM introspection and cooperativeness that the paper itself flags but does not resolve. The significance is therefore programmatic rather than demonstrated: the paper would be a strong call-to-arms if the identified gaps were addressed or explicitly scoped as open problems, but as written the load-bearing claim is under-supported.

major comments (3)
  1. [Section 5.2–5.3] The central promise that agentic interpretability can teach 'potentially superhuman' machine concepts is not supported by the proposed evaluation and grounding strategy. Section 5.3 concedes that 'Models empirically have little meta-understanding of why they sometimes perform well and other times don't,' and the proposed 'co-discovery' process relies on the human posing discriminating hypotheticals. For a superhuman concept, however, the human does not know which probes are discriminating, and the model's answers are generated by the same unreliable self-explanation process. Section 5.2's proposed evaluation—testing whether the human can predict the concept on new examples—is external to the interaction and could in principle detect false mental models, but nothing in the agentic loop forces such validation before the user acts on the learned model. The paper acknowledges this only indirectly through the fallback to end-task metrics. As written, the loop can converge on a confidently false model, and no argument is given that interaction prevents this. Please either add a concrete mechanism by which the conversation is grounded against the model's actual decision rule (e.g., in-loop generation and checking of held-out predictions) or substantially weaken the 'potentially superhuman' claim to an explicitly open research hope.
  2. [Section 3.1 and Section 2.2.3] The feasibility claim 'we can pursue agentic interpretability' depends on current LLMs being cooperative and capable of forming useful user models, but the paper does not specify how to recognize when this precondition holds. It correctly notes that agentic interpretability is inappropriate for deliberately deceptive models, then argues that interactive 'open surgery' can expose deception (Section 2.2.3). This is an interesting conjecture, not a demonstrated capability; a deceptive model could produce explanations that are coherent and consistent with the manipulated internals. The paper should state the empirical or conceptual conditions under which a model's teaching is trustworthy enough to count as interpretability, and propose tests that would falsify the 'can' claim. Without such conditions, the title's normative force rests on an untested assumption about LLM cooperativeness.
  3. [Section 5.3] Using LLMs as proxies for human turns in evaluation risks circularity when the proxy shares the target model's blind spots. The paper acknowledges that models lack meta-understanding; if the proxy LLM is the same kind of system, its simulated confusions and questions may not reflect human learning trajectories, and success in proxy-based evaluation would not establish that real humans benefit. The paper should justify the proxy assumption with evidence (e.g., demonstrations that LLM-simulated users produce similar errors and learning curves to human users) or specify validation studies comparing proxy and human conversational trajectories. As written, the proxy proposal is plausible but not yet grounded.
minor comments (5)
  1. [Section 3.1] The sentence 'agentic interpretability may not the right tool' is missing a verb and should read 'may not be the right tool.'
  2. [Section 5] The items '1)case improveto make' and '2)case learnto learn' have missing spaces and inconsistent punctuation; they should be formatted as numbered list items, e.g., '1) case improve: to make...'.
  3. [Section 5.1] In the example 'may provide information my understanding of this code base is poor', the user's self-assessment appears to be missing quotation marks, which makes the sentence difficult to parse.
  4. [Section 4.1] The sentence 'for example in achieving reproducibility, conducting controlled comparisons, and isolating the impact of specific variables exceptionally difficult' is a fragment; a verb such as 'makes' or 'is' is needed.
  5. [Section 4.2] The phrase 'achieving this is arguably be harder using only input and output' contains a typo and should read 'is arguably harder.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper argues by definition, example, and external frameworks, not by fitting or self-citation chains.

full rationale

This is a position paper, not a derivation chain. The central claim is a stipulated definition: 'A method pursues agentic interpretability if it proactively assists human understanding in a multi-turn interactive process by developing and leveraging a mental model of the user, which in turn enables humans to develop better mental models of the LLM' (Section 1). The paper argues that LLM conversational capabilities make this definition instantiable, and it grounds the mental-model concept in established cognitive science and HCI literature. There are no fitted parameters, no equations whose outputs equal their inputs, and no prediction that is statistically forced by a fit. The evaluation proposals in Section 5.2 use external end-task metrics such as predicting the concept on new examples or measuring Elo improvement, which are independent checks. Section 5.3's suggestion to use LLMs as proxies for human turns is explicitly framed as a development-time approximation, and the same section insists that 'end-task metrics (e.g., did it enable faster model debugging?) remain critical.' Thus the LLM-proxy idea is not presented as validation by the same model. The paper's self-citations, including Schut et al. (2025), Hewitt et al. (2025), and Doshi-Velez and Kim (2017), are used as examples or evaluation frameworks, not as load-bearing uniqueness theorems or as substitutes for independent evidence. No step in the argument reduces to its own inputs by construction.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim depends on several domain assumptions about LLM capability and human learning; none are new physical entities or fitted parameters. The paper identifies some of these assumptions itself, especially the concern about model introspection.

assumptions (5)
  • domain assumption LLMs' capacity for coherent, contextual conversation enables building and leveraging mental models of users.
    Section 1 and Section 2.1 assume conversational ability suffices for mental modeling and proactive teaching; no empirical demonstration is provided.
  • domain assumption Current LLMs are generally cooperative enough to be useful for agentic interpretability.
    Section 3.1 says most advanced LLMs do not consistently exhibit severe deceptive behaviors; this is an empirical claim about current models, treated as a premise.
  • domain assumption Human learning from conversation can be measured by end-task metrics and prediction accuracy.
    Section 5.2 proposes simulatability and end-task metrics as proxies; it assumes these capture mental model improvement.
  • domain assumption Mental models from cognitive science transfer to human-machine collaboration.
    Section 2.1.1 argues established human collaboration principles ought to hold in human-machine collaboration; this is an analogy, not a proven transfer.
  • domain assumption LLMs can be used as proxies for humans in evaluation.
    Section 5.3 recommends LLM proxies for human turns; the validity of this proxy for evaluation is unvalidated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Because we have LLMs, we Can and Should Pursue Agentic Interpretability." pith.science (2026). https://pith.science/paper/YFPJSDAX

@misc{pith2026250612152,
  author       = {Pith},
  title        = {Pith review of: Because we have LLMs, we Can and Should Pursue Agentic Interpretability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YFPJSDAX}},
  note         = {Machine review of arXiv:2506.12152}
}
read the original abstract

The era of Large Language Models (LLMs) presents a new opportunity for interpretability--agentic interpretability: a multi-turn conversation with an LLM wherein the LLM proactively assists human understanding by developing and leveraging a mental model of the user, which in turn enables humans to develop better mental models of the LLM. Such conversation is a new capability that traditional `inspective' interpretability methods (opening the black-box) do not use. Having a language model that aims to teach and explain--beyond just knowing how to talk--is similar to a teacher whose goal is to teach well, understanding that their success will be measured by the student's comprehension. While agentic interpretability may trade off completeness for interactivity, making it less suitable for high-stakes safety situations with potentially deceptive models, it leverages a cooperative model to discover potentially superhuman concepts that can improve humans' mental model of machines. Agentic interpretability introduces challenges, particularly in evaluation, due to what we call `human-entangled-in-the-loop' nature (humans responses are integral part of the algorithm), making the design and evaluation difficult. We discuss possible solutions and proxy goals. As LLMs approach human parity in many tasks, agentic interpretability's promise is to help humans learn the potentially superhuman concepts of the LLMs, rather than see us fall increasingly far from understanding them.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Reality Monitoring in Large Language Models: Self-Knowledge That Transforms with Conversation Memory

    cs.AI 2026-07 conditional novelty 7.0 of 10

    LLMs' source-attribution ability is not fixed: it flips with conversational memory structure, and corrective feedback can invert judgments or sever confidence from accuracy.

  2. From Features to Actions: Explainability in Traditional and Agentic AI Systems

    cs.AI 2026-02 conditional novelty 5.0 of 10

    Attribution explanations that work for static classifiers do not diagnose failures in multi-step AI agents; trace-grounded rubric evaluation does, with state-tracking inconsistency 2.7x more common in failed agent runs.

Reference graph

Works this paper leans on

28 extracted references · 6 canonical work pages · cited by 2 Pith papers

  1. [1]

    Abdul, J

    A. Abdul, J. Vermeulen, D. Wang, B. Y. Lim, and M. Kankanhalli. Trends and trajectories for ex- plainable, accountable and intelligible systems: An HCI research agenda.Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems,

    work page 2018

  2. [10]

    Grosse, J

    R. Grosse, J. Bae, C. Anil, N. Elhage, A. Tamkin, A. Tajdini, B. Steiner, D. Li, E. Durmus, E. Perez, et al. Studying large language model generalization with influence functions.arXiv preprint arXiv:2308.03296,

    arXiv

  3. [12]

    URLhttps://arxiv.org/abs/2412.06771. J. Hewitt, R. Geirhos, and B. Kim. We can’t understand AI using our existing vocabulary,

    arXiv

  4. [14]

    doi: 10.1016/j.tics.2024.07.007. S. M. Lundberg and S.-I. Lee. A unified approach to interpreting model predictions.Advances in neural informationprocessing systems, 30,

    doi:10.1016/j.tics.2024.07.007 2024

  5. [18]

    doi: 10.1145/3564240

    ISSN 0360-0300. doi: 10.1145/3564240. URL https://doi.org/10.1145/3564240. L. Schut, N. Tomašev, T. McGrath, D. Hassabis, U. Paquet, and B. Kim. Bridging the human–AI knowledge gap through concept discovery and transfer in AlphaZero.Proceedings of the National Academy of Sciences, 122, 03

    doi:10.1145/3564240

  6. [19]

    doi: 10.1073/pnas.2406675122. R. Shah, A. Irpan, A. M. Turner, A. Wang, A. Conmy, D. Lindner, J. Brown-Cohen, L. Ho, N. Nanda, R. A. Popa, et al. An approach to technical agi safety and security.arXivpreprint arXiv:2504.01849,

    arXiv

  7. [20]

    Sharkey, B

    L. Sharkey, B. Chughtai, J. Batson, J. Lindsey, J. Wu, L. Bushnaq, N. Goldowsky-Dill, S. Heimer- sheim, A. Ortega, J. Bloom, et al. Open problems in mechanistic interpretability.arXiv preprint arXiv:2501.16496,

    arXiv

  8. [21]

    URL https: //arxiv.org/abs/2406.09264. X. Shi, I. Padhi, and K. Knight. Does string-based neural MT learn source syntax? In J. Su, K. Duh, and X. Carreras, editors,Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526–1534, Austin, Texas, Nov

    arXiv 2016

Show all 28 references
  1. [22]

    doi: 10.18653/v1/D16-1159

    Association for Computational Linguistics. doi: 10.18653/v1/D16-1159. URLhttps://aclanthology.org/D16-1159/. T. Shin, Y. Razeghi, R. L. L. IV, E. Wallace, and S. Singh. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts,

    doi

  2. [24]

    URLhttps://arxiv.org/abs/ 2412.02674. T. Sorensen, J. Moore, J. Fisher, M. Gordon, N. Mireshghallah, C. M. Rytting, A. Ye, L. Jiang, X. Lu, N. Dziri, T. Althoff, and Y. Choi. A roadmap to pluralistic alignment,

    arXiv

  3. [25]

    org/abs/2402.05070

    URLhttps://arxiv. org/abs/2402.05070. A. Tversky and D. Kahneman. Judgment under uncertainty: Heuristics and biases.Science, 185 (4157):1124–1131, Sept

    arXiv

  4. [28]

    URLhttps: //arxiv.org/abs/2201.11903. D. Wood, J. S. Bruner, and G. Ross. The role of tutoring in problem solving.Journalofchildpsychology and psychiatry, 17(2):89–100,

    arXiv

  5. [29]

    URLhttps://arxiv.org/abs/2211.01910. 15

    arXiv

  6. [1974]

    URLhttps://www.ncbi

    doi: 10.1126/science.185.4157.1124. URLhttps://www.ncbi. nlm.nih.gov/pubmed/17835457. J. A. G. M. Van Dijk.The Deepening Divide: Inequality in the InformationSociety. Sage Publications,

  7. [1982]

    Merriam-webster, Accessed on 2025-05-22

    Merriam-Webster. Merriam-webster, Accessed on 2025-05-22. URL https://www. merriam-webster.com/slang/agentic. In Merriam-Webster.com slang dictionary. N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt. Progress measures for grokking via mechanistic interpretability,

    2025

  8. [1993]

    Y. Chen, A. Wu, T. DePodesta, C. Yeh, K. Li, N. C. Marin, O. Patel, J. Riecke, S. Raval, O. Seow, et al. Designing a dashboard for transparency and control of conversational ai.arXiv preprint arXiv:2406.07882,

    arXiv

  9. [2012]

    URLhttps://www.science.org/doi/ abs/10.1126/science.1218633

    doi: 10.1126/science.1218633. URLhttps://www.science.org/doi/ abs/10.1126/science.1218633. R. Greenblatt, C. Denison, B. Wright, F. Roger, M. MacDiarmid, S. Marks, J. Treutlein, T. Be- lonax, J. Chen, D. Duvenaud, et al. Alignment faking in large language models.arXiv preprint...

    arXiv

  10. [2014]

    org/CorpusID:127197

    URLhttps://api.semanticscholar. org/CorpusID:127197. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran- Johnson, E. Perez, J. Kerr, J. ...

    arXiv

  11. [2016]

    doi: 10.18653/v1/W16-2524

    Association for Com- putational Linguistics. doi: 10.18653/v1/W16-2524. URL https://aclanthology.org/ W16-2524/. M. C. Frank and N. D. Goodman. Predicting pragmatic reasoning in language games.Science, 336 (6084):998–998,

    doi

  12. [2017]

    Marks, J

    S. Marks, J. Treutlein, T. Bricken, J. Lindsey, J. Marcus, S. Mishra-Sharma, D. Ziegler, E. Ameisen, J. Batson, T. Belonax, et al. Auditing language models for hidden objectives.arXiv preprint arXiv:2503.10965,

    arXiv

  13. [2018]

    semanticscholar.org/CorpusID:5063596

    URL https://api. semanticscholar.org/CorpusID:5063596. G. Alain and Y. Bengio. Understanding deep learning with linear models. arXiv preprint arXiv:1606.05382,

    arXiv

  14. [2019]

    URLhttp://arxiv.org/abs/1902. 02960. J. A. Cannon-Bowers, E. Salas, and S. A. Converse. Shared mental models in expert team performance. In N. J. Castellan Jr., editor,Individual and group decision making, pages 221–245. Lawrence Erlbaum Associates,

    1902

  15. [2020]

    URLhttps://arxiv.org/abs/ 2010.15980. Y. Song, H. Zhang, C. Eisenach, S. Kakade, D. Foster, and U. Ghai. Mind the gap: Examining the self-improvement capabilities of large language models,

    2010 arXiv

  16. [2021]

    doi: 10.1016/j.tics.2021.07.008. M. Hahn, W. Zeng, N. Kannen, R. Galt, K. Badola, B. Kim, and Z. Wang. Proactive agents for multi-turn text-to-image generation under uncertainty,

    2021 doi

  17. [2022]

    URLhttps://api.semanticscholar.org/CorpusID:254823489. G. Bansal, T. Wu, J. Zhou, R. Fok, B. Nushi, E. Kamar, M. T. Ribeiro, and D. Weld. Does the whole exceed its parts? The effect of ai explanations on complementary team performance. InProceedings of the 2021 CHI conference ...

    2021

  18. [2023]

    URLhttps://arxiv.org/abs/2301.05217. D. A. Norman. Some observations on mental models.Mental models, 7:7–14,

    arXiv

  19. [2024]

    URLhttps://arxiv.org/abs/2412.10380. C. J. Cai, E. Reif, N. Hegde, J. D. Hipp, B. Kim, D. Smilkov, M. Wattenberg, F. B. Viégas, G. S. Corrado, M. C. Stumpe, and M. Terry. Human-centered tools for coping with imperfect algorithms during medical decision-making.CoRR, abs/1902.02960,

    1902 arXiv

  20. [2025]

    URL https://arxiv.org/abs/2502.07586. R. R. Hoffman, S. T. Mueller, G. Klein, and J. Litman. Metrics for explainable AI: Challenges and prospects. ArXiv, abs/1812.04608,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.