Pith. sign in

REVIEW 1 major objections 2 minor 28 references

Feature attribution for generative language models requires an explicit Attribution Contract to resolve ambiguities in what is being explained.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 16:35 UTC pith:ZX442SPB

load-bearing objection The Attribution Contract is a clear framing for ambiguities in feature attribution on generative models, but the paper stays conceptual with no tests or new methods. the 1 major comments →

arxiv 2605.23080 v2 pith:ZX442SPB submitted 2026-05-21 cs.LG

The Attribution Contract: Feature Attribution for Generative Language Models

Giang Nguyen This is my paper
classification cs.LG
keywords feature attributiongenerative language modelsattribution contractautoregressive modelsdiffusion modelsexplainable AIinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that ambiguities in feature attribution for generative language models arise because earlier tokens serve as both outputs and later inputs in autoregressive generation, and because diffusion models generate through iterative denoising rather than fixed next-token steps. It introduces the Attribution Contract as a specification that must name the output explained, the eligible features, the assumed generative process, what is held fixed, and the model score being attributed. This contract shows why one attribution method can answer different questions depending on how it is set up. A sympathetic reader would care because the paper argues that many apparent disagreements in the field are really disagreements over these unstated contracts rather than over the algorithms themselves.

Core claim

The central claim is that the ambiguity about what counts as a feature when applying attribution methods to generative language models is a conceptual limitation, not merely an implementation detail. The Attribution Contract addresses it by explicitly specifying what output is being explained, which features are eligible to receive attribution, what generative process is assumed, what is held fixed, and what model score is being attributed. Using autoregressive and diffusion language models as case studies, the paper shows when attribution to earlier tokens, intermediate states, or denoising stages is informative versus misleading, and concludes that feature-attribution methods should be eva

What carries the argument

The Attribution Contract, a specification that names the output being explained, eligible features, assumed generative process, elements held fixed, and model score attributed.

Load-bearing premise

Many disagreements about feature attribution in generative language models are disagreements about unstated explanatory contracts rather than about attribution algorithms.

What would settle it

An analysis of published disagreements that traces them primarily to differences in how attribution algorithms are implemented or tuned, rather than to differing assumptions about outputs, features, or generative processes.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The same attribution method can produce informative results under one contract and misleading results under another.
  • Attribution to previously generated tokens is valid only when the contract explicitly allows it.
  • In diffusion models, targeting intermediate denoising states requires the contract to define those states as the output being explained.
  • Feature attribution methods must be assessed together with the contracts under which they are applied.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This approach could be extended to require explicit contract statements in open-source attribution libraries so users cannot run explanations without declaring the setup.
  • It suggests a path for standardizing how attribution results are reported in papers, making comparisons across studies more reliable.
  • Researchers might test whether models trained with explicit contract awareness produce attributions that better match human expectations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 2 minor

Summary. The paper claims that feature attribution in generative language models (autoregressive and diffusion-based) faces a conceptual ambiguity in defining what counts as a 'feature,' since earlier tokens or diffusion states serve dual roles as outputs and inputs. This is presented not as an implementation issue but as a limitation of directly applying classifier-era attribution methods. The authors introduce the 'Attribution Contract'—a specification naming the output explained, eligible features, assumed generative process, elements held fixed, and attributed model score—to clarify that the same method can yield informative or misleading results depending on the contract. They argue via case studies that many field disagreements concern unstated contracts rather than algorithms, and recommend evaluating attributions as method-contract pairs.

Significance. If the framework holds, the Attribution Contract offers a useful conceptual lens for making feature-attribution claims explicit in generative modeling, potentially improving consistency and reducing misinterpretation in interpretability research. The case-study illustrations for autoregressive token attribution and diffusion-stage explanations are a strength, as they concretely show when attributions align with different explanatory goals. This definitional approach could encourage better reporting practices without requiring new algorithms.

major comments (1)
  1. [Abstract and §1] Abstract and §1 (motivation): The load-bearing claim that the ambiguity 'is not merely an implementation detail, but a conceptual limitation' of classifier-era methods requires showing that the contract framework uniquely resolves issues that adapted classifier methods cannot; the case studies demonstrate context-dependence but do not include a direct counter-example where a standard method fails without the contract specification.
minor comments (2)
  1. [Definition section] The definition of the Attribution Contract (likely §3) would benefit from an explicit enumerated list or pseudocode template to make the five components (output, features, process, fixed elements, score) immediately usable by readers.
  2. [Case studies] Case studies (likely §4–5): Clarify whether the 'informative vs. misleading' judgments are based on human evaluation, downstream task performance, or logical consistency with the stated contract.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the positive assessment and recommendation for minor revision. The feedback on the motivation section is helpful, and we address it directly below.

read point-by-point responses

  1. Referee: [Abstract and §1] Abstract and §1 (motivation): The load-bearing claim that the ambiguity 'is not merely an implementation detail, but a conceptual limitation' of classifier-era methods requires showing that the contract framework uniquely resolves issues that adapted classifier methods cannot; the case studies demonstrate context-dependence but do not include a direct counter-example where a standard method fails without the contract specification.

    Authors: We appreciate the referee's identification of the load-bearing claim. The manuscript's argument is that generative models introduce an ambiguity because prior tokens or diffusion states simultaneously serve as model outputs and subsequent inputs, violating the fixed-input assumption of classifier-era attribution; this is not resolvable by adaptation alone because the choice of what to hold fixed or attribute remains underspecified. The case studies illustrate that the same method yields different (sometimes opposing) interpretations under different contracts. We agree, however, that an explicit counter-example of a standard method producing contradictory conclusions across two unstated contracts would make the conceptual claim sharper. We will revise §1 to include such a constructed example (e.g., attributing the same token prediction under a 'next-token only' versus 'full-sequence' contract) to demonstrate the point. revision: yes

Circularity Check

0 steps flagged

No significant circularity; definitional framework without derivations or self-referential reductions

full rationale

The paper proposes the Attribution Contract as a conceptual specification that names output, eligible features, generative process, fixed elements, and attributed score to resolve ambiguities in applying feature attribution to generative language models. No equations, parameter fits, or derivations appear in the provided text or abstract; the central claim rests on logical reframing of existing ambiguities illustrated via case studies rather than any reduction of a result to its own inputs or load-bearing self-citations. The argument is therefore self-contained as a definitional framework with independent content.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The paper introduces a new conceptual specification without new empirical data or formal derivations. It relies on domain assumptions about how feature attribution has been applied to classifiers and generative models.

axioms (1)
  • domain assumption Feature attribution methods developed for classifiers can be conceptually transferred to generative language models but require explicit specification to avoid ambiguity.
    Invoked throughout the abstract as the basis for arguing that the ambiguity is a conceptual limitation.
invented entities (1)
  • Attribution Contract no independent evidence
    purpose: A specification that names the output explained, eligible features, generative process, fixed elements, and attributed model score.
    Newly introduced construct to address the stated conceptual limitation.

pith-pipeline@v0.9.1-grok · 5761 in / 1448 out tokens · 38416 ms · 2026-06-30T16:35:16.883819+00:00 · methodology

0 comments
read the original abstract

Feature attribution methods promise to identify which input features matter for a model output. In generative language models, however, it is often unclear what should count as a feature in the first place. In autoregressive language models, earlier generated tokens are both outputs of the model and inputs to later predictions. In diffusion language models, generation proceeds through iterative denoising or unmasking rather than fixed left-to-right prediction, so local explanation may target a state of diffusion rather than a next token. We argue that this ambiguity is not merely an implementation detail, but a conceptual limitation of carrying classifier-era feature attribution directly into generative language modeling. We introduce the Attribution Contract, a specification for feature-attribution claims that names what output is being explained, which features are eligible to receive attribution, what generative process is assumed, what is held fixed, and what model score is being attributed. The contract clarifies why the same attribution method can answer different questions depending on how it is instantiated. We argue that many disagreements about feature attribution in generative language models are not disagreements about attribution algorithms, but about unstated explanatory contracts. Using autoregressive and diffusion language models as case studies, we show when attribution to earlier generated tokens, intermediate states, or denoising stages is informative, when it is misleading, and why feature-attribution methods in generative language models should be evaluated as method-contract pairs.

Figures

Figures reproduced from arXiv: 2605.23080 by Giang Nguyen.

Figure 1
Figure 1. Figure 1: The same attribution method produces different attribution maps under different settings. All three rows use the same model, prompt, generation, and attribution method (Integrated Gradients [13]). Only the setting differs. Top: a local next-token setting attributes the prediction of noir over both prompt and prefix tokens, and mass concentrates on the generated prefix Le chien est because it is predictive … view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 28 canonical work pages · 4 internal anchors

  1. [1]

    ContextCite: Attributing Model Generation to Context

    Benjamin Cohen-Wang, Harshay Shah, Kristian Georgiev, and Aleksander Mądry. ContextCite: Attributing Model Generation to Context. InAdvances in Neural Information Processing Sys- tems, volume 37, 2024.https://proceedings.neurips.cc/paper_files/paper/2024/hash/ adbea136219b64db96a9941e4249a857-Abstract-Conference.html

    work page 2024

  2. [2]

    Sequential Integrated Gradients: a Simple but Effective Method for Explaining Language Models

    Joseph Enguehard. Sequential Integrated Gradients: a Simple but Effective Method for Explaining Language Models. InFindings of the Association for Computational Linguistics: ACL 2023, pages 7555–7565, 2023.https://aclanthology.org/2023.findings-acl.477/

    work page 2023

  3. [3]

    Jacobian Scopes: token-level causal attributions in LLMs

    Toni J.B. Liu, Baran Zadeoğlu, Nicolas Boullé, Raphaël Sarfati, and Christopher J. Earls. Jacobian Scopes: Token-level Causal Attributions in LLMs. arXiv preprint arXiv:2601.16407, 2026.https://arxiv.org/abs/2601.16407

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  4. [4]

    Locating and Edit- ing Factual Associations in GPT

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Edit- ing Factual Associations in GPT. InAdvances in Neural Information Processing Sys- tems, volume 35, 2022.https://proceedings.neurips.cc/paper_files/paper/2022/hash/ 6f1d43d5a82a37e89b0665b33bf3a182-Abstract-Conference.html

    work page 2022

  5. [5]

    Markosyan, Diego Garcia-Olano, and Narine Kokhlikyan

    Vivek Miglani, Aobo Yang, Aram H. Markosyan, Diego Garcia-Olano, and Narine Kokhlikyan. Using Captum to Explain Generative Language Models. InProceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), pages 165–173, 2023. https://aclanthology.org/2023.nlposs-1.19/

    work page 2023

  6. [6]

    Large Language Diffusion Models

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large Language Diffusion Models. InAdvances in Neural Information Processing Systems, 2025.https://openreview.net/forum?id=KnqiC0znVF

    work page 2025

  7. [7]

    Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 32819–32848. PMLR, 2024.https://proceedings.mlr.press/v235/lou24a.html

    work page 2024

  8. [8]

    Continuous Diffusion Model for Language Modeling

    Jaehyeong Jo and Sung Ju Hwang. Continuous Diffusion Model for Language Modeling. arXiv preprint arXiv:2502.11564, 2025.https://arxiv.org/abs/2502.11564

    work page arXiv 2025

  9. [9]

    Bastian, and Sumit Kumar Jha

    Vishal Pramanik, Maisha Maliha, Nathaniel D. Bastian, and Sumit Kumar Jha. Hessian- Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMs. InInternational Con- ference on Learning Representations, 2026.https://openreview.net/forum?id=XsEZcigEjq 12

    work page 2026

  10. [10]

    Measuring Attribution in Natural Language Generation Models.Computational Linguistics, 49(4):777–840, 2023

    Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. Measuring Attribution in Natural Language Generation Models.Computational Linguistics, 49(4):777–840, 2023. https://aclanthology.org/2023.cl-4.2/

    work page 2023

  11. [11]

    Discretized Integrated Gradients for Explaining Language Models

    Soumya Sanyal and Xiang Ren. Discretized Integrated Gradients for Explaining Language Models. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10285–10299, 2021.https://aclanthology.org/2021.emnlp-main.805/

    work page 2021

  12. [12]

    Inseq: An Interpretability Toolkit for Sequence Generation Models

    Gabriele Sarti, Nils Feldhus, Ludwig Sickert, and Oskar van der Wal. Inseq: An Interpretability Toolkit for Sequence Generation Models. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 421–435, 2023.https://aclanthology.org/2023.acl-demo.40/

    work page 2023

  13. [13]

    Axiomatic Attribution for Deep Networks

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic Attribution for Deep Networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 ofProceedings of Machine Learning Research, pages 3319–3328. PMLR, 2017.https://proceedings.mlr. press/v70/sundararajan17a.html

    work page 2017

  14. [14]

    What the DAAM: Interpreting Stable Diffusion Using Cross Attention

    Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp, Jimmy Lin, and Ferhan Ture. What the DAAM: Interpreting Stable Diffusion Using Cross Attention. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5644–5659, 2023.https://aclanthology.org...

    work page 2023

  15. [15]

    Explaining the Reasoning of Large Language Models Using Attribution Graphs

    Chase Walker and Rickard Ewetz. Explaining the Reasoning of Large Language Models Using Attribution Graphs. arXiv preprint arXiv:2512.15663, 2025.https://arxiv.org/abs/2512. 15663

    work page arXiv 2025

  16. [16]

    Unifying Corroborative and Contributive Attributions in Large Language Models

    Theodora Worledge, Judy Hanwen Shen, Nicole Meister, Caleb Winston, and Carlos Guestrin. Unifying Corroborative and Contributive Attributions in Large Language Models. arXiv preprint arXiv:2311.12233, 2023.https://arxiv.org/abs/2311.12233

    work page arXiv 2023

  17. [17]

    Towards Unified Attri- bution in Explainable AI, Data-Centric AI, and Mechanistic Interpretability

    Shichang Zhang, Tessa Han, Usha Bhalla, and Himabindu Lakkaraju. Towards Unified Attri- bution in Explainable AI, Data-Centric AI, and Mechanistic Interpretability. arXiv preprint arXiv:2501.18887, 2025.https://arxiv.org/abs/2501.18887

    work page arXiv 2025

  18. [18]

    arXiv preprint arXiv:2402.00794 , year=

    Zhixue Zhao and Boxuan Shan. ReAGent: A Model-Agnostic Feature Attribution Method for Generative Language Models. InProceedings of the AAAI Workshop on Responsible Language Models, 2024.https://arxiv.org/abs/2402.00794

    work page arXiv 2024

  19. [19]

    Impossibility theorems for feature attribution.Proceedings of the National Academy of Sciences, 121(2):e2304406120, January 2024

    Blair Bilodeau, Natasha Jaques, Pang Wei Koh, and Been Kim. Impossibility Theorems for Feature Attribution.Proceedings of the National Academy of Sciences, 121(2):e2304406120, 2024.https://www.pnas.org/doi/10.1073/pnas.2304406120

    work page doi:10.1073/pnas.2304406120 2024

  20. [20]

    The Effectiveness of Feature Attribution Methods and Its Correlation with Automatic Evaluation Scores

    Giang Nguyen, Daeyoung Kim, and Anh Nguyen. The Effectiveness of Feature Attribution Methods and Its Correlation with Automatic Evaluation Scores. InAdvances in Neural Information Processing Systems, volume 34, pages 26422–26436, 2021.https://proceedings. neurips.cc/paper/2021/hash/de043a5e421240eb846da8effe472ff1-Abstract.html

    work page 2021

  21. [21]

    Post Hoc Explanations May Be Ineffective for Detecting Unknown Spurious Correlation

    Julius Adebayo, Michael Muelly, Harold Abelson, and Been Kim. Post Hoc Explanations May Be Ineffective for Detecting Unknown Spurious Correlation. InInternational Conference on Learning Representations, 2022.https://openreview.net/forum?id=xNOVfCCvDpM 13

    work page 2022

  22. [22]

    Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Joh...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  23. [23]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Chen, Wenhao Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions.ACM Transactions on Information Systems, 2024.https://dl.acm.org/doi/10.1145/3703155

    work page doi:10.1145/3703155 2024

  24. [24]

    Survey of hallucination in natural language generation

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12):1–38, 2023.https://dl.acm.org/doi/10.1145/3571730

    work page doi:10.1145/3571730 2023

  25. [25]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, 2022.https://aclanthology.org/2022.emnlp-main.759/

    work page 2022

  26. [26]

    Red Teaming Language Models with Language Models

    Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red Teaming Language Models with Language Models. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022.https://arxiv.org/abs/2202.03286

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  27. [27]

    arXiv preprint arXiv:2512.18792 , year=

    Maxime Méloux, Giada Dirupo, François Portet, and Maxime Peyrard. The Dead Salmons of AI Interpretability. arXiv preprint arXiv:2512.18792, 2025.https://arxiv.org/abs/2512.18792

    work page arXiv 2025

  28. [28]

    Zachary C. Lipton. The Mythos of Model Interpretability.Communications of the ACM, 61(10):36–43, 2018.https://arxiv.org/abs/1606.03490 14

    work page internal anchor Pith review Pith/arXiv arXiv 2018