Pith. sign in

REVIEW 3 major objections 5 minor 51 references

Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Symmetric spectral diagnostics of attention operators are provably blind to the direction of information flow, and only the antisymmetric residual G can recover it.

desk verdict A solid structural result in attention diagnostics—symmetric spectral methods provably cannot see flow direction—plus a genuinely new Cheeger floor for uniform causal attention; the empirical bridge to real heads is softer than the theorems, but the paper is honest and worth refereeing. read the letter →

arxiv 2605.04893 v3 pith:E5IJG7GY submitted 2026-05-06 cs.LG cs.CLstat.ML

Dominik Dahlem , Diego Maniloff , Mac Misiura This is my paper · ORCID
classification cs.LGcs.CLstat.ML MSC 05C5068T07
keywords orientationblindnessattentiontransportoperatorspectraldiagnosticsCheegerinequalityconductancefloorhallucinationdetectionlength-controlledevaluationasymmetrycoefficient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper treats each attention head as a degree-normalized bipartite transport operator and asks what spectral diagnostics can and cannot infer. It proves that every transpose-invariant diagnostic—any function of singular values or of the spectrum of the symmetrized operator—returns the same value for M and its transpose, so methods that read singular values or symmetric spectra cannot distinguish an operator from its reversed flow. It then proves a converse: any diagnostic sensitive to direction must pay a cost proportional to the antisymmetric component's norm G. On the capacity axis, it derives a closed-form conductance floor for uniform causal attention (phi >= 1/5) that window attention pierces, and shows empirically that the polarity of the conductance signal reverses across benchmarks as predicted.

What carries the argument

The degree-normalized bipartite transport operator M = D_Q^{-1/2} B D_K^{-1/2}, its orthogonal split into symmetric part M_sym (capacity) and antisymmetric part M_asym (direction), the Hermitian dilation H(M) that converts the rectangular operator into a symmetric eigenvalue problem whose spectrum is the signed singular values, and the closed-form cut/volume identities for temporal cuts in uniform causal attention that yield the 1/5 floor.

What would settle it

Compute exact graph conductance by brute force on real attention heads with n <= 20 tokens and compare with the spectral-sweep estimator bphi; if bphi deviates beyond the degree-ratio bound even on near-regular heads, the floor-transfer assumption fails. Alternatively, a benchmark with known bottleneck-type hallucinations that does not reverse polarity on low-conductance terciles would falsify the two-sided diagnostic.

Watch

Extended reading notes

Core claim

Every attention head defines a degree-normalized transport operator M = D_Q^{-1/2} B D_K^{-1/2}. The paper's central claim is a structural identifiability limit: any spectral diagnostic that depends only on singular values of M or on the spectrum of the symmetric part M_sym is invariant under transpose, hence blind to the orientation of information flow. The proof uses the Hermitian dilation H(M) and a block-swap similarity showing H(M) and H(M^T) share a spectrum. A quantitative converse (Prop. 8) bounds the transpose sensitivity of any L-Lipschitz diagnostic by 2L||M_asym||_F, so the asymmetry coefficient G = ||M_asym||_F / (||M||_F + eps) is the unique control parameter for directional se

Load-bearing premise

The measured spectral-sweep estimate bphi on the degree-normalized graph tracks the exact conductance phi of the raw attention graph closely enough that the closed-form 1/5 floor, proved for the idealized object, can serve as a fixed benchmark for real attention heads.

Editorial extensions

If this is right

  • Symmetric spectral diagnostics such as LLM-Check, EigenTrack, and LapEigvals are structurally unable to detect the direction of information flow; any improvement must explicitly use the antisymmetric residual.
  • The asymmetry coefficient G provides a guaranteed bound on directional sensitivity: if G is small, no Lipschitz diagnostic can have large transpose sensitivity.
  • The conductance floor phi >= 1/5 distinguishes failure modes by shape: uniform causal attention can never be a global bottleneck, while window attention provably pierces the floor on balanced cuts.
  • Polarity of the conductance signal is regime-dependent and reverses between bottleneck-dominated and diffuse-dominated datasets, so a universal sign is not expected.
  • Length-controlled evaluation is required; raw AUROC inflates by up to 0.28 points due to length confounding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The orientation-blindness result likely extends to any diagnostic built on symmetrized representations of the same operator, even if it reads eigenvectors, as long as the summary is block-swap symmetric; hidden-state methods are not covered by the theorem but face a similar structural limitation if they symmetrize.
  • The closed-form conductance template (row-sum/column-sum decomposition) can probably be applied to other causal masks such as exponential decay or RoPE-style decay to produce per-architecture floors; the paper sketches this but leaves the closed forms open.
  • The empirical status of the 1/5 floor suggests a testable design rule: architectures with high floor-violation fractions should exhibit bottleneck polarity on datasets with concentrated hallucinations, which could guide dataset construction.
  • The degree-sufficiency result implies future attention diagnostics should separate degree-based signal from coupling-based signal, or risk misattributing hallucination signal to routing structure when it is largely a degree artifact.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies what spectral diagnostics of the degree-normalized attention operator M = D_Q^{-1/2} B D_K^{-1/2} can and cannot infer about information-flow orientation. It proves that every functional of the singular values of M or of the spectrum of the symmetric part M_sym is transpose-invariant (Theorem 5), so such diagnostics are orientation-blind; a Lipschitz converse bounds the transpose sensitivity of any Lipschitz functional by 2L||M_asym||_F (Proposition 8). On the capacity axis, it derives closed-form conductance bounds for canonical causal attention: uniform causal attention satisfies an n-independent temporal-cut floor phi >= 1/5 (Theorems 19, 20), while window attention pierces it as O(w/n) (Lemma 16). Empirically, under length-controlled AUROC, the conductance/spectral features are reported to retain 0.62--0.84 LC-AUROC, with a polarity reversal between HaluEval (bottleneck) and MedHallu (diffuse), and the fraction of heads whose empirical estimator falls below 1/5 is claimed as an architectural signature (Table 3). The mathematical core is elementary and mostly correct, but the bridge from the exact theoretical object (conductance phi on the raw graph H(B)) to the empirical estimator (spectral sweep on the degree-normalized dilation H(M)) is not adequately validated, and the polarity 'prediction' is partly circular as presented.

Significance. If the empirical bridge were validated, the paper would provide a clean identifiability boundary for a class of attention diagnostics and a useful closed-form Cheeger benchmark for causal architectures. The formal results are sound as stated: the transpose-invariance of singular-value and symmetric-spectrum functionals is correctly proved, and the 1/5 floor for uniform causal attention is a correct closed-form computation. The paper also ships a Lean 4 formalization of the core structural lemmas, which is a strength. However, the central empirical claim that the floor-violation fraction is an architectural descriptor depends on the unexamined transfer from phi on H(B) to the estimator on H(M), and the paper's own null analysis shows the estimator bphi is largely degree-reducible. The contribution is therefore a solid theoretical clarification with empirical claims that currently outrun the evidence.

major comments (3)
  1. [Sec. 3, Table 3, Theorem 3] The empirical floor-violation fractions are computed with the spectral-sweep estimator bphi on the degree-normalized dilation N_bip = H(M), while Theorem 20's floor phi >= 1/5 is proved for the exact conductance of the raw bipartite graph A_bip = H(B). The paper acknowledges this distinction but justifies the transfer by Theorem 3, which only controls the discrepancy under near-regular column degrees (kappa ~ 1). The paper's own degree-preserving nulls (Sec. 7.1) show bphi is largely degree-reducible (z-AUROC 0.51--0.57), so the reported 36--42% / 55--70% / 79--82% violation fractions may be measuring degree-distribution shape rather than violation of the temporal-cut floor. This is load-bearing for the architectural-signature claim. Please report the kappa distribution for the evaluated heads, and/or compute exact phi on A_bip for a subset of heads, or otherwise demonstrate that the swe
  2. [Secs. 1, 7.1, 8.2] The 'falsifiable polarity prediction' is not tested as stated. HaluEval is labeled bottleneck-dominated and MedHallu diffuse-dominated after the tercile analysis (Sec. 7.1), and the LC-AUROC is reported as flipped, i.e., label-informed polarity (Sec. 6.2). The paper also notes that polarity is bin-consistent in only 7 of 12 model--dataset pairings (Sec. 8). To support the claim, the authors should specify an a priori criterion (e.g., the sign of the length-label correlation, r=+0.70 vs -0.21, or an independent failure-mode taxonomy) and evaluate unflipped directional AUROC. As written, the reversal is a descriptive pattern, not a confirmation of the prediction.
  3. [Intro, Corollary 7, Sec. 8.1] The set of methods claimed to be limited by Theorem 5 is inconsistent. Corollary 7 and the Introduction list EigenTrack, but Sec. 8.1 states that the orientation-blindness theorem 'does not bear directly on hidden-state-based detection,' and EigenTrack operates on hidden-state covariance spectra, not on the degree-normalized attention operator. For LapEigvals, the main text asserts that its raw Laplacian eigenvalues are functions of symmetrized spectra but defers the proof to the Online Supplement. Since naming the affected detectors is part of the claimed contribution, the coverage claim should be reconciled, and the explicit reduction for each named method should be shown in the main text or the names removed.
minor comments (5)
  1. [Abstract, Corollary 7] The phrase 'every transpose-invariant spectral diagnostic is orientation-blind' is definitional; the substantive statement is that singular-value and symmetric-spectrum diagnostics are transpose-invariant. Corollary 7 already notes that eigenvector summaries reading query/key blocks separately can break invariance, so the abstract should be qualified to avoid overstatement.
  2. [Sec. 4, Sec. 8.1] The manuscript refers to 'Theorem 8' in the proof structure of Theorem 5 and in Sec. 8.1, but the displayed result is Proposition 8. Similarly, Appendix B's roadmap refers to 'Theorem 9' for the asymmetry-coefficient characterization while the main text states Proposition 9. Numbering should be unified.
  3. [Sec. 3, Table 3] The text says the 1/5-floor violation fractions are '36--42% for GPT-2, 55--70% for Pythia-160M, and 79--82% for Flan-T5 decoder across HaluEval, MedHallu, and TruthfulQA.' Table 3 omits the Flan-T5 decoder/TruthfulQA row because no samples passed the n>=32 filter; the phrasing 'across ... TruthfulQA' is therefore inaccurate for Flan-T5.
  4. [Table 5 caption] The caption does not state that the AUROC values are flipped (label-informed polarity), although Sec. 6.2 says 'unless stated otherwise, reported LC-AUROC values are flipped.' Please make this explicit in the table caption.
  5. [Appendix B, Conductance transfer corollary] The notation phi(M), phi_ref, delta_ref, and epsilon is introduced informally. The displayed bound 'phi(M)-phi_ref <= sqrt(2(delta_ref+epsilon)) - delta_ref/2' can be negative (trivial), and the derivation of epsilon from the spectral-gap difference should be stated precisely.

Circularity Check

1 steps flagged · score 4.0 of 10

The spectral theorems and Cheeger floor are self-contained; the empirical 'polarity prediction' reduces to a label-calibrated sign choice.

  1. fitted input called prediction [Abstract; Sec. 6.2 (flipped LC-AUROC); Sec. 7.1 (tercile assignment); Sec. 8 (practical calibration)]
    "Flipping selects the polarity using the labels, so a flipped value answers “does this feature carry signal” rather than “does a label-free detector exist”... polarity reverses between HaluEval and MedHallu, directionally as predicted though asymmetric in strength, with decision polarity calibrated per regime."

    The polarity direction is not fixed by the theory before measurement: Sec. 8 states 'the sign of the diagnostic cannot be fixed a priori and must be calibrated per configuration', and the reported flipped LC-AUROC selects the sign from the labels. The assignment of HaluEval to the bottleneck/low-OC regime and MedHallu to the diffuse/high-OC regime is made from the same tercile statistics offered as confirmation ('On HaluEval, hallucinations cluster in the low-OC tercile... On MedHallu, hallucinations cluster in the high-OC tercile'). The 'directionally as predicted' finding is therefore a restatement of the label-informed sign calibration, not an independent test of the two-sided theory.

full rationale

The paper's mathematical spine is not circular. Theorem 5 follows from the block-swap similarity of H(M), Lemma 2 from the SVD, and the 1/5 Cheeger floor from the closed-form cut/volume identities; none of these are fitted to data or imported from the authors' own prior work. The paper also explicitly separates the exact object φ on A_bip from the estimator bφ on N_bip, describing the 1/5 floor as 'an idealized-architecture benchmark, not an empirical attractor', so the estimator-vs-object gap is a correctness/validation limitation rather than a circular step. There are no load-bearing self-citations (the Lean formalization is machine-checked; other cited results are external). The only step that approaches circularity is the empirical polarity claim: polarity is label-calibrated per configuration and the dataset-level bottleneck/diffuse labels are derived from the same conductance terciles, after which the reversal is presented as a confirmed prediction. Because the central theorems retain independent content and the paper discloses the label-dependent calibration, this is a partial empirical circularity, not a collapse of the derivation.

Assumptions & free parameters 3 free parameters · 6 assumptions · 1 invented entities

The free parameters are mostly evaluation choices and polarity calibration rather than derivation parameters. The axioms are standard spectral graph theory plus two domain assumptions about attention structure. The only invented object is G, a statistic with testable bounds. The geometry/derivation itself has no fitted constants.

free parameters (3)
  • polarity sign = per model-dataset regime; bin-consistent in 7/12 pairings
    The flipped LC-AUROC selects the sign of the φ–hallucination relationship from labels; the paper states decision polarity cannot be fixed a priori and needs 50–100 calibration examples.
  • G regularization ε = not specified (0 in theory)
    Appears in G = ||M_asym||_F/(||M||_F + ε); a free small constant to avoid division by zero, with no principled value.
  • CVaR tail thresholds and length-bin count B = CVaR75/CVaR25; B adaptive
    Aggregation hyperparameters that shape the reported LC-AUROC numbers; not fitted to maximize a target but freely chosen.
assumptions (6)
  • standard math Cheeger inequality and the spectral gap–conductance bridge
    Used as the backbone (Eq. 3) to interpret σ2 as a transport-capacity proxy; assumed from Cheeger 1970 / Chung 1997.
  • standard math Hilbert–Schmidt orthogonality of symmetric and antisymmetric subspaces
    Underlies the M = Msym + Masym decomposition and the definition of G.
  • domain assumption Attention heads are degree-normalized row-stochastic transport operators
    Def. 1 models B (mask applied, softmax over keys) as row-stochastic; all theory builds on this representation.
  • domain assumption Causal masking is lower-triangular for decoder self-attention
    Used for the G>0 guarantees and the temporal-cut conductance landscape (Sec. 5, App. B).
  • ad hoc to paper Conductance of the raw attention graph is a meaningful hallucination diagnostic
    The paper states 'The claim that diffuse mixing indicates failure is empirical and task-dependent' (Sec. 3). It is an interpretive assumption, not a theorem.
  • ad hoc to paper Temporal-cut family is the right diagnostic family
    The 1/5 floor and O(w/n) claims are proved for temporal cuts S_t only; global min-cut conductance of real attention graphs may differ.
invented entities (1)
  • Asymmetry coefficient G independent evidence
    purpose: Normalized Frobenius distance to the symmetric subspace; quantifies directional/asymmetry content unavailable to symmetric spectral diagnostics
    G is a new diagnostic statistic, not a physical entity; the paper gives falsifiable bounds (e.g., G ≤ 1/√2 for causal Toeplitz attention) checkable on any model's attention maps.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics." pith.science (2026). https://pith.science/paper/E5IJG7GY

@misc{pith2026260504893,
  author       = {Pith},
  title        = {Pith review of: Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E5IJG7GY}},
  note         = {Machine review of arXiv:2605.04893}
}
abstract

Every attention head defines a degree-normalized transport operator, and a growing family of diagnostics reads model behavior (hallucination among them) from its spectrum. We ask what such diagnostics can and cannot infer. The operator splits orthogonally into a symmetric part governing transport \emph{capacity} and an antisymmetric part encoding \emph{orientation}. We prove an identifiability limit: every transpose-invariant spectral diagnostic is \emph{orientation-blind} (unable to distinguish an operator from its transpose, hence blind to the orientation of information flow), with a transpose-stability bound limiting any Lipschitz diagnostic's transpose sensitivity by the asymmetry coefficient $G$. This bounds what spectral diagnostics of the attention operator can resolve (e.g.\ LapEigvals and the attention-spectral branch of LLM-Check). On the surviving axis, a closed-form bipartite-Cheeger landscape shows uniform causal attention obeys an $n$-independent \emph{temporal-cut} floor $\phi \ge 1/5$ while window attention pierces it as $O(w/n)$; the floor is an idealized benchmark, not an empirical attractor, and the fraction of real heads falling below it is itself an empirically stable architectural descriptor. The two-axis diagnostic ($\phi$ for capacity, $G$ for asymmetry magnitude) yields a falsifiable polarity prediction, borne out \emph{in sign} under length-controlled, forced-scoring evaluation across decoder-only, encoder-only, and encoder--decoder models (capacity-axis signal 0.62--0.84 LC-AUROC): polarity reverses between HaluEval and MedHallu, directionally as predicted though asymmetric in strength, with decision polarity calibrated per regime.

Figures

Figures reproduced from arXiv: 2605.04893 by the authors.

Figure 1
Figure 1. Conductance has a bounded healthy range; asymmetry detects temporal isolation. Each attention head defines bipartite transport between queries (Q) and keys (K). (a)–(c) Conductance spectrum: healthy attention occupies an optimal band; too low indicates bottleneck (a), too high indicates diffuse dilution (c). (a) Bottleneck: concentrated attention yields low ϕb, missing relevant context. (b) Healthy: selective routin… view at source ↗
Figure 2
Figure 2. Conductance landscape: theory and empirics. (a) Theoretical ϕ(St) vs. cut location t/n for canonical causal architectures at n = 100, derived in closed form from Theorems 12 and 13. Uniform causal (blue, solid) is U-shaped with worst case at t ∗/n ≈ 0.32 and asymptotic floor u∞/(2+u∞) ≈ 0.36, strictly above the 1/5 Cheeger floor (Theorem 16). Window attention (red and orange, dashed) follows ϕ ≤ w/(n−t) → 0, piercin… view at source ↗
Figure 2
Figure 2. Conductance has a bounded healthy range; asymmetry detects temporal isolation. Each attention head defines bipartite transport between queries (Q) and keys (K). (a)–(c) Conductance spectrum: healthy attention occupies an optimal band; too low indicates bottleneck (a), too high indicates diffuse dilution (c). (a) Bottleneck: concentrated attention yields low ϕb, missing relevant context. (b) Healthy: selective routin… view at source ↗
Figures from the paper (7 more)
Figure 3
Figure 3. Figure 3: Per-layer transport profiles reveal distinct failure signatures. Conductance ϕˆ (a) and spectral norm σ2 (b) averaged across heads within each layer (Pythia-160M). Shaded bands show population ±1 std around dotted mean lines (green: factual, red: hallucinated from Halu…
Figure 3
Figure 3. Figure 3: Conductance landscape: theory and empirics. (a) Theoretical ϕ(St) vs. cut location t/n for canonical causal architectures at n = 100, derived in closed form from Theorems 16 and 17. Uniform causal (blue, solid) is U-shaped with an n-independent floor, strictly above th…
Figure 4
Figure 4. Figure 4: Conductance–spectral norm scatter reveals regime-dependent polarity. Each point is one sample; axes show conductance ϕˆ and spectral norm σ2 averaged across all heads and layers (Pythia-160M, 500 subsampled per class for HaluEval). The near-perfect anti-correlation (ρ=…
Figure 4
Figure 4. Figure 4: Per-layer transport profiles reveal distinct failure signatures. Conductance ϕb (a) and spectral norm σ2 (b) averaged across heads within each layer (Pythia-160M). Shaded bands show population ±1 std around dotted mean lines (green: factual, red: hallucinated from Halu…
Figure 5
Figure 5. Figure 5: Temporal isolation (G) is sparse but architecture- and position-encoding￾dependent. (a) LC-AUROC for G std across all 15 model-dataset combinations with 95% bootstrap CIs. Most configurations cluster near chance; Flan-T5 decoder/HaluEval (0.78) and Pythia/HaluEval (0.8…
Figure 6
Figure 6. Figure 6: Scaling validation and aggregation crossover (HaluEval). (a) LC-AUROC across Pythia 70M–1.4B (same training data, varying parameter count) plus LLaMA 3.1 8B (cross-architecture; GQA, RoPE). Conductance features (σ2 std, ϕˆ CVaR75) retain signal at all scales; G std dro…
Figure 6
Figure 6. Figure 6: Temporal isolation (G) is sparse but architecture- and position-encoding￾dependent. (a) LC-AUROC for G std across all 15 model-dataset combinations with 95% bootstrap CIs. Most configurations cluster near chance; Flan-T5 decoder/HaluEval (0.78) and Pythia/HaluEval (0.8…

Review history (3 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 1 canonical work pages

  1. [1]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  2. [2]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  3. [3]

    0362 CVaR _ 75 and @ @

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv 2010

  4. [4]

    Hallucination detection in LLMs using spectral features of attention maps

    Jakub Binkowski, Denis Janiak, Albert Sawczyn, Bogdan Gabrys, and Tomasz Jan Kajdanowicz. Hallucination detection in LLMs using spectral features of attention maps. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 24354--24385, Suzhou, China, 2025. Association for Computational Linguistics. doi:10.18...

    doi:10.18653/v1/2025.emnlp-main.1239 2025

  5. [5]

    A Lower Bound for the Smallest Eigenvalue of the Laplacian

    Jeff Cheeger. A Lower Bound for the Smallest Eigenvalue of the Laplacian . In Robert C. Gunning, editor, Problems in Analysis : A Symposium in Honor of Salomon Bochner , pages 195--199. Princeton University Press, Princeton, NJ, 1970. Princeton Legacy Library reprint: 2015, ISBN 978-1-4008-6931-2

    1970

  6. [6]

    INSIDE : LLMs ' internal states retain the power of hallucination detection

    Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE : LLMs ' internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations (ICLR 2024), 2024

    2024

  7. [7]

    Yingyi Chen, Qinghua Tao, Francesco Tonin, and Johan A. K. Suykens. Primal-attention: Self-attention through asymmetric kernel SVD in primal representation. In Advances in Neural Information Processing Systems, volume 36, 2023

    2023

  8. [8]

    Yung-Sung Chuang, Linlu Qiu, Cheng-Yu Hsieh, Ranjay Krishna, Yoon Kim, and James R. Glass. Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1419--1436, Miami, Florida, USA, 2024. Association ...

    doi:10.18653/v1/2024.emnlp-main.84 2024

Show all 51 references
  1. [9]

    Fan R. K. Chung. Spectral Graph Theory, volume 92 of CBMS Regional Conference Series in Mathematics. American Mathematical Society, 1997. ISBN 978-0-8218-0315-8

    1997

  2. [10]

    Fan R. K. Chung. Laplacians and the Cheeger Inequality for Directed Graphs . Annals of Combinatorics, 9 0 (1): 0 1--19, April 2005. ISSN 0219-3094. doi:10.1007/s00026-005-0237-z

    2005 doi

  3. [11]

    Hermitian matrices for clustering directed graphs: Insights and applications

    Mihai Cucuringu, Huan Li, He Sun, and Luca Zanetti. Hermitian matrices for clustering directed graphs: Insights and applications. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), volume 108 of Proceedings of Machine Learn...

    2020

  4. [12]

    Attention is not all you need: Pure attention loses rank doubly exponentially with depth

    Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, p...

    2021

  5. [13]

    Yotam Erel, Olaf D \"u nkel, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, and Amit H. Bermano. Attention (as discrete-time Markov ) chains. In Advances in Neural Information Processing Systems, volume 38, 2025

    2025

  6. [14]

    EigenTrack : Spectral activation feature tracking for hallucination and out-of-distribution detection in LLMs and VLMs

    Davide Ettori, Nastaran Darabi, Sina Tayebati, Ranganath Krishnan, Mahesh Subedar, Omesh Tickoo, and Amit Ranjan Trivedi. EigenTrack : Spectral activation feature tracking for hallucination and out-of-distribution detection in LLMs and VLMs . arXiv preprint arXiv:2509.15735, 2025

    2025

  7. [15]

    Detecting hallucinations in large language models using semantic entropy

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630 0 (8017): 0 625--630, June 2024. ISSN 1476-4687. doi:10.1038/s41586-024-07421-0

    2024 doi

  8. [16]

    Eigenvalue bounds on convergence to stationarity for nonreversible M arkov chains, with an application to the exclusion process

    James Allen Fill. Eigenvalue bounds on convergence to stationarity for nonreversible M arkov chains, with an application to the exclusion process. The Annals of Applied Probability, 1 0 (1): 0 62--87, 1991. doi:10.1214/aoap/1177005981

    1991

  9. [17]

    The emergence of clusters in self-attention dynamics

    Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. The emergence of clusters in self-attention dynamics. In Advances in Neural Information Processing Systems, volume 36, pages 57026--57037, 2023

    2023

  10. [18]

    Transformer feed-forward layers are key-value memories

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5484--5495, 2021. doi:10.18653/v1/2021.emnlp-main.446

    2021 doi

  11. [19]

    Golub and Charles F

    Gene H. Golub and Charles F. van Loan. Matrix Computations. Johns Hopkins University Press, 4th edition, 2013. ISBN 978-1-4214-0794-4

    2013

  12. [20]

    Graham, Donald E

    Ronald L. Graham, Donald E. Knuth, and Oren Patashnik. Concrete Mathematics: A Foundation for Computer Science. Addison-Wesley, 2nd edition, 1994. ISBN 978-0-201-55802-9

    1994

  13. [21]

    Higher dimensional discrete Cheeger inequalities

    Anna Gundert and May Szedl\'ak. Higher dimensional discrete Cheeger inequalities. Journal of Computational Geometry, 6 0 (2): 0 54--71, 2015. doi:10.20382/jocg.v6i2a4

    2015 doi

  14. [22]

    Horn and Charles R

    Roger A. Horn and Charles R. Johnson. Matrix Analysis . Cambridge University Press, 2nd edition, 2012. ISBN 978-0-521-83940-2. doi:10.1017/CBO9781139020411

    2012 doi

  15. [23]

    The illusion of progress: Re-evaluating hallucination detection in LLMs

    Denis Janiak, Jakub Binkowski, Albert Sawczyn, Bogdan Gabrys, Ravid Shwartz-Ziv, and Tomasz Jan Kajdanowicz. The illusion of progress: Re-evaluating hallucination detection in LLMs . In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pag...

    2025 doi

  16. [24]

    Improved Cheeger's inequality: Analysis of spectral partitioning algorithms through higher order spectral gap

    Tsz Chiu Kwok, Lap Chi Lau, Yin Tat Lee, Shayan Oveis Gharan, and Luca Trevisan. Improved Cheeger's inequality: Analysis of spectral partitioning algorithms through higher order spectral gap. In Proceedings of the Forty-Fifth Annual ACM Symposium on Theory of Computing (STOC),...

    2013

  17. [25]

    Frustration index and Cheeger inequalities for discrete and continuous magnetic Laplacians

    Carsten Lange, Shiping Liu, Norbert Peyerimhoff, and Olaf Post. Frustration index and Cheeger inequalities for discrete and continuous magnetic Laplacians . Calculus of Variations and Partial Differential Equations, 54 0 (4): 0 4165--4196, 2015. doi:10.1007/s00526-015-0935-x

    2015 doi

  18. [26]

    Lee, Shayan Oveis Gharan, and Luca Trevisan

    James R. Lee, Shayan Oveis Gharan, and Luca Trevisan. Multiway spectral partitioning and higher-order Cheeger inequalities. Journal of the ACM, 61 0 (6): 0 1--30, 2014. doi:10.1145/2665063. Conference version in STOC 2012

    2014 doi

  19. [27]

    Levin, Yuval Peres, and Elizabeth L

    David A. Levin, Yuval Peres, and Elizabeth L. Wilmer. Markov chains and mixing times . American Mathematical Society, 2006

    2006

  20. [28]

    HaluEval : A large-scale hallucination evaluation benchmark for large language models

    Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. HaluEval : A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449--6464, Singapore...

    2023 doi

  21. [29]

    TruthfulQA : Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA : Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214--3252, Dublin, Ireland, 2022. Association for ...

    2022 doi

  22. [30]

    Lov\'asz

    L. Lov\'asz. Random walks on graphs: A survey. In D. Mikl\'os , V. T. S\'os , and T. Sz o nyi , editors, Combinatorics, Paul Erd o s is Eighty , volume 2, pages 353--398. J\'anos Bolyai Mathematical Society, Budapest, 1996

    1996

  23. [31]

    Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9004--9017, 2023. ...

    2023 doi

  24. [32]

    Clustering by weighted cuts in directed graphs

    Marina Meila and William Pentney. Clustering by weighted cuts in directed graphs. In Proceedings of the 2007 SIAM International Conference on Data Mining, pages 135--144, 2007. doi:10.1137/1.9781611972771.13

    2007 doi

  25. [33]

    Locating and editing factual associations in GPT

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT . In Advances in Neural Information Processing Systems, volume 35, 2022

    2022

  26. [34]

    FActScore : Fine-grained atomic evaluation of factual precision in long form text generation

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen - tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore : Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empiri...

    2023 doi

  27. [35]

    Mind the gap: A spectral analysis of rank collapse and signal propagation in attention layers

    Thiziri Nait Saada , Alireza Naderi, and Jared Tanner. Mind the gap: A spectral analysis of rank collapse and signal propagation in attention layers. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2025

    2025

  28. [36]

    MedHallu : A comprehensive benchmark for detecting medical hallucinations in large language models

    Shrey Pandit, Jiawei Xu, Junyuan Hong, Zhangyang Wang, Tianlong Chen, Kaidi Xu, and Ying Ding. MedHallu : A comprehensive benchmark for detecting medical hallucinations in large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Pro...

    2025 doi

  29. [37]

    Ori Parzanchevski, Ron Rosenthal, and Ran J. Tessler. Isoperimetric inequalities in simplicial complexes. Combinatorica, 36 0 (2): 0 195--227, 2016. doi:10.1007/s00493-014-3002-x

    2016 doi

  30. [38]

    Robins, Andrea Rotnitzky, and Lue Ping Zhao

    James M. Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American Statistical Association, 89 0 (427): 0 846--866, 1994. doi:10.1080/01621459.1994.10476818

    1994

  31. [39]

    Tyrrell Rockafellar and Stanislav Uryasev

    R. Tyrrell Rockafellar and Stanislav Uryasev. Optimization of conditional value-at-risk. Journal of Risk, 2 0 (3): 0 21--41, 2000

    2000

  32. [40]

    Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyr \'e

    Michael E. Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyr \'e . Sinkformers: Transformers with doubly stochastic attention. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 151 of Proceedings of Machine Le...

    2022

  33. [41]

    Revisiting uncertainty quantification evaluation in language models: Spurious interactions with response length bias results

    Andrea Santilli, Adam Golinski, Michael Kirchhof, Federico Danieli, Arno Blaas, Miao Xiong, Luca Zappella, and Sinead Williamson. Revisiting uncertainty quantification evaluation in language models: Spurious interactions with response length bias results. In Proceedings of the...

    2025 doi

  34. [42]

    On the NP -completeness of some graph cluster measures

    Ji r \' S \' ma and Satu Elisa Schaeffer. On the NP -completeness of some graph cluster measures. In SOFSEM 2006: Theory and Practice of Computer Science, volume 3831 of Lecture Notes in Computer Science, pages 530--537. Springer, 2006. doi:10.1007/11611257_51

    2006 doi

  35. [43]

    Approximate counting, uniform generation and rapidly mixing Markov chains

    Alistair Sinclair and Mark Jerrum. Approximate counting, uniform generation and rapidly mixing Markov chains. Information and Computation, 82 0 (1): 0 93--133, 1989. ISSN 0890-5401. doi:10.1016/0890-5401(89)90067-9

    1989 doi

  36. [44]

    LLM-Check : Investigating detection of hallucinations in large language models

    Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kattakinda, and Soheil Feizi. LLM-Check : Investigating detection of hallucinations in large language models. In Advances in Neural Information Processing Systems, volume 37, 2024

    2024

  37. [45]

    RoFormer : Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer : Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024. doi:10.1016/j.neucom.2023.127063

    2024

  38. [46]

    Gomez, ukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998--6008, 2017

    2017

  39. [47]

    Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned

    Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797-...

    2019 doi

  40. [48]

    A tutorial on spectral clustering

    Ulrike von Luxburg . A tutorial on spectral clustering. Statistics and Computing, 17 0 (4): 0 395--416, December 2007. ISSN 0960-3174. doi:10.1007/s11222-007-9033-z

    2007 doi

  41. [49]

    Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V. Le. Long-form factuality in large language models. In Advances in Neural Information Processing Systems, volume 37, 2024

    2024

  42. [50]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations (ICLR), 2024

    2024

  43. [51]

    Susskind

    Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M. Susskind. Stabilizing transformer training by preventing attention entropy collapse. In Proceedings of the 40th International Conference on Machine Learnin...

    2023

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.