Pith. sign in

REVIEW 3 major objections 5 minor 64 references

Activation dispersion at the prompt point reveals entity familiarity near ceiling from 1.5B, while factual reliability on known entities scales separately and models almost never refuse.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 18:21 UTC pith:I7LCRBJF

load-bearing objection Solid one-pass familiarity detector on Bielik with real controls; the familiarity-vs-reliability dissociation is the interesting claim and is only half-settled because the reliability curve is athletes-only and Claude-judged. the 3 major comments →

arxiv 2607.07670 v1 pith:I7LCRBJF submitted 2026-07-08 cs.CL cs.LG

Does Bielik Know What It Doesn't Know? Activation Dispersion Separates Entity Familiarity from Factual Reliability Across Model Scale

Grzegorz Brzezinka This is my paper
classification cs.CL cs.LG
keywords activation dispersionentity familiarityhallucination detectioninverse participation ratiospectral entropyLLM scalingfactual reliabilityBielik
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether a language model’s internal activations already show that it does or does not know an entity, before any answer token is produced, and whether that signal predicts correct answers. On four Polish Bielik models and four entity domains, two unsupervised dispersion statistics on post-SwiGLU MLP activations separate well-known from fabricated entities at AUROC 0.95–1.00 from a single forward pass, already at ceiling at 1.5B; the same holds for known versus real-but-obscure names and transfers across domains. Behavioral correctness about known athletes, by contrast, rises sharply with scale under a strict judge. Within known entities the same metrics do not cleanly separate correct answers from hallucinations, and an audit finds almost no refusals despite the internal signal. The practical claim is that familiarity and reliability are distinct, on different scaling curves.

Core claim

Entity familiarity and factual reliability are distinct phenomena on different scaling curves. Unsupervised inverse participation ratio and spectral entropy of post-SwiGLU MLP activations at the prompt point separate known from fabricated entities at AUROC 0.95–1.00 across athletes, cities, writers, and musicians and across Bielik scales 1.5B–11B, already at ceiling at 1.5B; the signal survives real-name controls and cross-domain transfer. Fully correct answers on the same known athletes rise from 0/42 to 19/42 under a strict judge. Dispersion does not reliably read correctness within known entities, and models almost never abstain despite a near-perfectly decodable familiarity signal.

What carries the argument

Inverse participation ratio and spectral entropy of winsorized post-SwiGLU MLP activations at the last prompt token: closed-form, unsupervised, single-forward-pass concentration statistics that measure how localized versus smeared the activation pattern is.

Load-bearing premise

The rising correctness curve and near-zero refusal claim rest on one family of LLM judges correctly scoring Polish sports-biography answers and refusals.

What would settle it

Have independent human annotators or a non-Claude judge re-label the 42 known athletes’ answers under the same strict and soft rubrics; if the 0→2→10→19 fully-correct counts flatten or reverse, or if refusals are no longer near zero, the dissociation and abstention-gap claims fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A free prefill-time dispersion gate can route high-dispersion (unfamiliar) queries to retrieval, escalation, or abstention without extra samples or labels.
  • Familiarity detectors must not be marketed as hallucination detectors for answers about entities the model already knows.
  • Representational ‘I don’t know this entity’ can be present at every scale while calibrated refusal remains almost absent—an alignment gap, not a capability gap.
  • Entity recognition can saturate early while long-tail factual reliability continues to improve with scale, matching frequency-based accounts of recall.
  • Global volumetric statistics remain useful even when the underlying circuitry is diffuse across heads rather than localized.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A cheap two-stage stack is natural: dispersion for prompt-side unfamiliarity, then multi-sample semantic entropy for generation-side consistency on familiar entities.
  • The forced one-sentence prompt may suppress abstention; a prompt that invites ‘say if unsure’ could close the refusal gap more cheaply than steering or fine-tuning.
  • Because the signal is diffuse across heads, global dispersion may stay a practical monitor across model families even when specific circuits differ.
  • English and other-language replications would test whether the early ceiling is specific to Polish tokenization and Bielik pretraining frequency.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper tests whether post-SwiGLU MLP activation dispersion (IPR and spectral entropy) at the prompt point, before any answer token, encodes entity familiarity in four Bielik models (1.5B–11B) across athletes, cities, writers, and musicians (known / obscure-real / fabricated; 504 prompts per model). Unsupervised dispersion separates known from fabricated at AUROC 0.95–1.00 (probe 0.99–1.00), clears selection-aware permutation floors (~0.70–0.74), survives held-out layer selection, persists on known vs. obscure-real names, and transfers across domains after a matched-template control for the cities stem. The familiarity signal is already near ceiling at 1.5B, while strict fully-correct answers on known athletes rise 0→2→10→19 of 42; within-known correctness is not cleanly read by dispersion; semantic entropy is weaker on familiarity but stronger on correctness; and an audit of 2,520 answers finds almost no refusals. The central claim is that entity familiarity and factual reliability are distinct phenomena on different scaling curves.

Significance. If the dissociation holds, the work cleanly separates a cheap, label-free, single-pass familiarity gate from answer-level factual reliability, with unusually careful selection controls, a fabricated-string-free real-name control, cross-domain transfer with a template counterfactual, a Gemma-4 qualitative control, and an honest mostly-negative result on within-known correctness. The public pipeline and frozen artifacts make the activation side reproducible. For Polish LLM deployment and for the broader internal-state hallucination literature, a prompt-point volumetric detector that is already strong at 1.5B is practically and scientifically useful even if it does not track correctness. The near-zero-refusal audit, if robust, is a concrete alignment gap rather than a capability gap.

major comments (3)
  1. [§3.2, §4.3, Fig. 1(b), §6] §3.2, §4.3, Fig. 1(b), and §6: the reliability half of the dissociation (strict 0→2→10→19 and soft 6→16→24→33 of 42 known athletes) rests entirely on Claude Opus 4.8, with parse failures defaulting to incorrect under the bare strict prompt. The paper itself notes that no non-Claude judge was run and that a family-level blind spot would contaminate behavioral counts and the SE correctness column together. Manual review of 615 soft verdicts (~0.3% anomalies) does not validate the absolute scale of the curve. Because the title/abstract claim is that familiarity and reliability move on different scaling curves, this is load-bearing: please add an independent judge family and/or human labels on a stratified subset of known-athlete answers, and report agreement with Opus so the scaling dissociation is not a single-judge artifact.
  2. [Abstract, §3.1–3.2, §5–6] Abstract, §1 contributions, and §5 state a general dissociation, but §3.1–3.2 and §6 restrict behavioral labels, semantic entropy, and the refusal audit to athletes only; cities/writers/musicians carry condition-based detection labels only. The familiarity detector generalizes well (Table 7, Fig. 5), yet the claim that reliability scales while familiarity does not is demonstrated on one domain. Either extend soft/strict judging (or a lighter human audit) to at least one additional domain, or narrow the abstract and discussion so the dissociation is explicitly athletes-only while cross-domain claims are limited to familiarity detection.
  3. [§3.5, Table 1] §3.5: winsorization quantile q=0.99 is acknowledged as a researcher degree of freedom and is the most favorable of {0.95, 0.99, 0.999, 1.0} on the 1.5B sweep; best layer moves with q and per-layer AUROCs degrade substantially at other settings (e.g., entropy at layer 22). The existence of a signal at every q is reassuring, but headline numbers and layer choices are tied to the most favorable setting on one model. Please either (i) fix q by a pre-registered rule independent of the known-vs-fabricated AUROC, (ii) report best-of-pair under a held-out q selection, or (iii) show that the multi-model, multi-domain ceiling claim is stable under a less favorable fixed q.
minor comments (5)
  1. [Table 5, §4.6] Table 5 uses a single-permutation floor that the text itself says understates the selection-aware null (§3.6); either recompute selection-aware floors for the behavioral grid or mark those floors more clearly as optimistic lower bounds so readers do not over-read the 11B probe cell.
  2. [Figure 2] Figure 2 note on MLP-hook vs residual-stream index offset is easy to miss; a one-line axis caption that residual and MLP depths are not identically indexed would help.
  3. [§3.1] §3.1: the 42-per-condition Adams nod is fine in a footnote, but the main text should state the sample-size rationale in statistical terms (power / CI width) given n=84 per contrast.
  4. [§5, Appendix A] Appendix A Gemma-4 refusal counts are marker-based entity-level all-five-sample refusals, unlike the Bielik LLM answer-level audit; state this contrast more prominently when comparing abstention regimes in §5.
  5. [§3.5, front matter] Typos / wording: “Inverse Participation Ration” (§3.5); “labelled spectral entropy in the figures” vs. Shannon entropy of activations (clarify naming once); arXiv date line “July 2026” looks like a placeholder.

Circularity Check

0 steps flagged

Empirical measurement paper with external condition labels and closed-form dispersion metrics; no derivation reduces to its inputs by construction.

full rationale

The paper’s load-bearing claims are empirical contrasts, not forced derivations. IPR and spectral entropy are closed-form functions of post-SwiGLU activations (Eqs. 1–2); known/unknown-real/fabricated labels are assigned from external entity lists, not from the metrics. AUROC is evaluated against those labels, against selection-aware permutation nulls that re-select layers, and against held-out layer selection—so the headline separability is not a fit renamed as a prediction. Cross-domain transfer, the matched-template counterfactual, and the known-vs-unknown-real control further test the signal against independent structure rather than restate the training contrast. Behavioral reliability is scored by an external LLM judge on sampled answers; that is a validity risk (single judge family, athletes-only), not circularity: correctness labels are not defined from dispersion, and dispersion is not fitted to correctness. There is no self-citation uniqueness theorem, no ansatz smuggled from the author’s prior work, and no renaming of a known result as a first-principles derivation. Solo author; Bielik citations are third-party model reports. Score 0 is the honest finding.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

The work is empirical NLP/interpretability. Load-bearing choices are experimental design parameters (winsorization quantile, entity set size, measurement token, judge family) and domain assumptions about what counts as known/obscure/fabricated for Polish models—not new physical entities. No free parameters enter a closed-form theory of hallucination; free parameters affect metric stability and label quality.

free parameters (4)
  • winsorization quantile q = 0.99
    Activations clipped above the q-quantile before IPR/entropy; q=0.99 is used in all headline numbers and is the most favorable of {0.95,0.99,0.999,1.0} on the 1.5B sweep (§3.5).
  • entities per condition = 42
    Fixed at 42 per known/unknown-real/fabricated cell; sets statistical power for all AUROCs and behavioral fractions.
  • best-layer / best-metric selection = argmax over layers and metric pair
    Headline dispersion is max over layers and {IPR, entropy}; controlled with selection-aware permutation floors and held-out splits, but still a free analysis choice.
  • sampling temperature and sample count for behavioral/SE = T=0.7, n=5
    Five answers at temperature 0.7, unseeded; freezes correctness, semantic entropy, and refusal statistics to shipped artifacts (§3.2, §7).
axioms (5)
  • domain assumption Famous entities in the known tier are substantially more present in pretraining than obscure-real and fabricated names for Bielik (and, with caveats, Gemma).
    Condition labels are the ground truth for the familiarity detector; §3.1 and Appendix A acknowledge frequency/popularity as the intended axis.
  • domain assumption Fabricated names do not collide with real entities the models know, so fabricated labels remain valid negatives.
    §6 notes screening but not exhaustive verification; rare collisions are assumed not to matter because models would not know them either.
  • ad hoc to paper Post-SwiGLU MLP activation vectors at the last prompt token carry a volumetric familiarity signal measurable by IPR and Shannon entropy of normalized squared activations.
    Motivated by a localization analogy the authors call ‘no more than that’ (§1); the paper treats this as an empirical hypothesis, not a theorem.
  • domain assumption Claude Opus/Sonnet/Haiku judgments are accurate enough proxies for factual correctness, meaning clusters, and refusal/hedge labels on Polish one-sentence answers.
    Entire behavioral axis, SE baseline, and refusal audit rest on this stack (§3.2, §3.7, §5); no independent human or non-Claude judge family is reported.
  • standard math Standard AUROC, logistic probes, and permutation tests correctly quantify separability under the stated selection procedures.
    Ordinary evaluation machinery; selection-aware nulls are the careful part (§3.6).
invented entities (1)
  • Prompt-point volumetric familiarity signal (IPR / spectral entropy of post-SwiGLU activations) independent evidence
    purpose: Unsupervised single-pass detector of entity familiarity before generation.
    Not a new particle or force; an operational statistic. Independent evidence is the multi-domain, multi-scale AUROCs and Gemma control, but the construct is defined by the paper’s measurement choice.

pith-pipeline@v1.1.0-grok45 · 30246 in / 3919 out tokens · 45660 ms · 2026-07-10T18:21:21.956504+00:00 · methodology

0 comments
read the original abstract

Large language models hallucinate most about entities they have never seen. We ask whether a model's activations betray entity familiarity before a single answer token is generated, and whether that signal predicts the factual reliability of the answers. On four Polish Bielik models (1.5B-11B parameters), we probe four entity domains (athletes, cities, writers, musicians), each with 42 well-known, 42 obscure-but-real, and 42 fabricated entities addressed by a one-sentence question (504 prompts per model). Two unsupervised, single-forward-pass dispersion measures over post-SwiGLU MLP activations, inverse participation ratio and spectral entropy, separate known from fabricated entities at AUROC 0.95-1.00 across all domains and scales; a supervised linear probe reaches 0.99-1.00. Both clear selection-aware permutation floors of about 0.70-0.74 (empirical p<=1e-3), survive held-out layer selection (0.93-0.99), and persist on real names (known vs. obscure-but-real: 0.96-1.00). The signal transfers across entity types (mean off-diagonal AUROC 0.92-0.99); a matched-template counterfactual shows the only large drops are template-caused, not entity-type effects, and the signal is diffuse across heads. This representational signal is already at ceiling at 1.5B, whereas behavioral factual reliability scales sharply: 0, 2, 10, and 19 of 42 known athletes are answered fully correctly by the 1.5B, 4.5B, 7B, and 11B models under a strict judge. Within known entities, separating correct from hallucinated answers is much harder (probe 0.93; dispersion no better than a first-token-entropy baseline). A five-sample semantic-entropy baseline reaches only 0.71-0.83 at 5x the inference cost. Despite this internal awareness, the models almost never abstain: an audit of 2,520 answers finds 2 refusals and 1 hedge. Entity familiarity and factual reliability are distinct phenomena on different scaling curves.

Figures

Figures reproduced from arXiv: 2607.07670 by Grzegorz Brzezinka.

Figure 1
Figure 1. Figure 1: Two scaling curves that do not move together (athletes domain). [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: AUROC (known vs. fabricated, prompt point) versus relative depth (layer index normalized by the deepest layer of its family) for all four models. The supervised probe (pink) rises sharply at 15–30% depth and stays near ceiling thereafter - the familiarity information is present from early-middle layers on. The unsupervised dispersion metrics (IPR, spectral entropy) peak in a broad mid-depth band and oscill… view at source ↗
Figure 3
Figure 3. Figure 3: Best per-metric AUROC (known vs. fabricated, prompt point) across the model sweep. MLP-side dispersion (IPR, spectral entropy) and the supervised probe are strong everywhere; attention entropy is robust across scale; logit-lens entropy strengthens with scale (0.72 at 1.5B to ≈0.98 at 7B/11B); effective rank is unstable (0.67–0.94). All internal metrics except effective rank at some scales clearly beat the … view at source ↗
Figure 4
Figure 4. Figure 4: Semantic entropy versus one-pass activation signals on [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The familiarity signal generalizes across entity types. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Per-head attention-entropy separability ( [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 64 canonical work pages · 16 internal anchors

  1. [1]

    Ferrando, Javier and Obeso, Oscar and Rajamanoharan, Senthooran and Nanda, Neel , booktitle =. Do. 2025 , eprint =

    work page 2025

  2. [3]

    2024 , eprint =

    Chen, Chao and Liu, Kai and Chen, Ze and Gu, Yi and Wu, Yue and Tao, Mingyuan and Fu, Zhihang and Ye, Jieping , booktitle =. 2024 , eprint =

    work page 2024

  3. [4]

    2024 , eprint =

    Du, Xuefeng and Xiao, Chaowei and Li, Yixuan , booktitle =. 2024 , eprint =

    work page 2024

  4. [6]

    2024 , url =

    Sriramanan, Gaurang and Bharti, Siddhant and Sadasivan, Vinu Sankar and Saha, Shoumik and Kattakinda, Priyatham and Feizi, Soheil , booktitle =. 2024 , url =

    work page 2024

  5. [7]

    Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps

    Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps , author =. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages =. 2024 , publisher =. 2407.07071 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  6. [9]

    Hallucination Detection in

    Binkowski, Jakub and Janiak, Denis and Sawczyn, Albert and Gabrys, Bogdan and Kajdanowicz, Tomasz , booktitle =. Hallucination Detection in. 2025 , eprint =

    work page 2025

  7. [11]

    Nature , volume =

    Detecting Hallucinations in Large Language Models Using Semantic Entropy , author =. Nature , volume =. 2024 , publisher =

    work page 2024

  8. [13]

    2022 , eprint =

    Language Models (Mostly) Know What They Know , author =. 2022 , eprint =

    work page 2022

  9. [14]

    2025 , howpublished =

    On the Biology of a Large Language Model , author =. 2025 , howpublished =

    work page 2025

  10. [15]

    Deja Vu: Contextual Sparsity for Efficient

    Liu, Zichang and Wang, Jue and Dao, Tri and Zhou, Tianyi and Yuan, Binhang and Song, Zhao and Shrivastava, Anshumali and Zhang, Ce and Tian, Yuandong and R. Deja Vu: Contextual Sparsity for Efficient. Proceedings of the 40th International Conference on Machine Learning (ICML) , series =. 2023 , eprint =

    work page 2023

  11. [16]

    The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers

    The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers , author =. International Conference on Learning Representations (ICLR) , year =. 2210.06313 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [18]

    Physical Review , volume =

    Absence of Diffusion in Certain Random Lattices , author =. Physical Review , volume =. 1958 , publisher =

    work page 1958

  13. [19]

    The Effective Rank:

    Roy, Olivier and Vetterli, Martin , booktitle =. The Effective Rank:. 2007 , address =

    work page 2007

  14. [20]

    2025 , eprint =

    Bielik v3 Small: Technical Report , author =. 2025 , eprint =

    work page 2025

  15. [21]

    2025 , eprint =

    Bielik 11B v2 Technical Report , author =. 2025 , eprint =

    work page 2025

  16. [23]

    Proceedings of the 40th International Conference on Machine Learning (ICML) , series =

    Large Language Models Struggle to Learn Long-Tail Knowledge , author =. Proceedings of the 40th International Conference on Machine Learning (ICML) , series =. 2023 , eprint =

    work page 2023

  17. [25]

    interpreting

    nostalgebraist , year =. interpreting

    work page

  18. [26]

    Advances in Neural Information Processing Systems (NeurIPS) , volume =

    Compact Language Models via Pruning and Knowledge Distillation , author =. Advances in Neural Information Processing Systems (NeurIPS) , volume =. 2024 , eprint =

    work page 2024

  19. [27]

    1979 , address =

    Adams, Douglas , title =. 1979 , address =

    work page 1979

  20. [30]

    Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Models

    Su, Weihang and Wang, Changyue and Ai, Qingyao and Hu, Yiran and Wu, Zhijing and Zhou, Yujia and Liu, Yiqun , booktitle =. 2024 , publisher =. 2403.06448 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  21. [31]

    2025 , eprint =

    Real-Time Detection of Hallucinated Entities in Long-Form Generation , author =. 2025 , eprint =

    work page 2025

  22. [32]

    No Answer Needed: Predicting

    Moreno Cencerrado, Iv. No Answer Needed: Predicting. 2025 , eprint =

    work page 2025

  23. [33]

    2025 , eprint =

    Orgad, Hadas and Toker, Michael and Gekhman, Zorik and Reichart, Roi and Szpektor, Idan and Kotek, Hadas and Belinkov, Yonatan , booktitle =. 2025 , eprint =

    work page 2025

  24. [35]

    Faiyaz Abdullah , booktitle =

    Alvi, Riasad and Sayeedi, Nurul Labib and Sayeedi, Md. Faiyaz Abdullah , booktitle =. 2026 , publisher =

    work page 2026

  25. [37]

    Retrieval Head Mechanistically Explains Long-Context Factuality

    Retrieval Head Mechanistically Explains Long-Context Factuality , author =. International Conference on Learning Representations (ICLR) , year =. 2404.15574 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv

  26. [38]

    2026 , eprint =

    Bielik-Minitron-7B: Compressing Large Language Models via Structured Pruning and Knowledge Distillation for the Polish Language , author =. 2026 , eprint =

    work page 2026

  27. [39]

    HalluVerse25: Fine-grained Multilingual Benchmark Dataset for LLM Hallucinations

    Samir Abdaljalil, Hasan Kurban, and Erchin Serpedin. HalluVerse25 : Fine-grained multilingual benchmark dataset for LLM hallucinations, 2025. URL https://arxiv.org/abs/2503.07833

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  28. [40]

    The Hitchhiker's Guide to the Galaxy

    Douglas Adams. The Hitchhiker's Guide to the Galaxy. Pan Books, London, 1979

    work page 1979

  29. [41]

    Faiyaz Abdullah Sayeedi

    Riasad Alvi, Nurul Labib Sayeedi, and Md. Faiyaz Abdullah Sayeedi. MultiHaluDet : Multilingual hallucination detection via LLM hidden state probing. In Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026). Association for Computational Linguistics, 2026. URL https://aclanthology.org/2026.mellm-1.6/

    work page 2026

  30. [42]

    P. W. Anderson. Absence of diffusion in certain random lattices. Physical Review, 109 0 (5): 0 1492--1505, 1958. doi:10.1103/PhysRev.109.1492

    work page doi:10.1103/physrev.109.1492 1958

  31. [43]

    The internal state of an LLM knows when it's lying

    Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it's lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967--976. Association for Computational Linguistics, 2023. doi:10.18653/v1/2023.findings-emnlp.68. URL https://aclanthology.org/2023.findings-emnlp.68/

    work page doi:10.18653/v1/2023.findings-emnlp.68 2023

  32. [44]

    Hallucination detection in LLM s using spectral features of attention maps

    Jakub Binkowski, Denis Janiak, Albert Sawczyn, Bogdan Gabrys, and Tomasz Kajdanowicz. Hallucination detection in LLM s using spectral features of attention maps. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025. URL https://arxiv.org/abs/2502.17598

    work page arXiv 2025

  33. [45]

    INSIDE : LLM 's internal states retain the power of hallucination detection

    Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE : LLM 's internal states retain the power of hallucination detection. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=Zj12nzlQbz

    work page 2024

  34. [46]

    Yung-Sung Chuang, Linlu Qiu, Cheng-Yu Hsieh, Ranjay Krishna, Yoon Kim, and James R. Glass. Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1419--1436. Association for Computational L...

    work page 2024

  35. [47]

    D 2 HScore : Reasoning-aware hallucination detection via semantic breadth and depth analysis in LLM s, 2025

    Yue Ding, Xiaofang Zhu, Tianze Xia, Junfei Wu, Xinlong Chen, Qiang Liu, and Liang Wang. D 2 HScore : Reasoning-aware hallucination detection via semantic breadth and depth analysis in LLM s, 2025. URL https://arxiv.org/abs/2509.11569. Under review

    work page arXiv 2025

  36. [48]

    HaloScope : Harnessing unlabeled LLM generations for hallucination detection

    Xuefeng Du, Chaowei Xiao, and Yixuan Li. HaloScope : Harnessing unlabeled LLM generations for hallucination detection. In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL https://openreview.net/forum?id=nfK0ZXFFSn

    work page 2024

  37. [49]

    EigenTrack : Spectral activation feature tracking for hallucination and out-of-distribution detection in LLM s and VLM s, 2025

    Davide Ettori, Nastaran Darabi, Sina Tayebati, Ranganath Krishnan, Mahesh Subedar, Omesh Tickoo, and Amit Ranjan Trivedi. EigenTrack : Spectral activation feature tracking for hallucination and out-of-distribution detection in LLM s and VLM s, 2025. URL https://arxiv.org/abs/2509.15735. Submitted to ICASSP 2026

    work page arXiv 2025

  38. [50]

    Detecting hallucinations in large language models using semantic entropy

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630 0 (8017): 0 625--630, 2024. doi:10.1038/s41586-024-07421-0

    work page doi:10.1038/s41586-024-07421-0 2024

  39. [51]

    Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models

    Javier Ferrando, Oscar Obeso, Senthooran Rajamanoharan, and Neel Nanda. Do I know this entity? K nowledge awareness and hallucinations in language models. In International Conference on Learning Representations (ICLR), 2025. URL https://arxiv.org/abs/2411.14257

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  40. [52]

    Language Models (Mostly) Know What They Know

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec,...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  41. [53]

    Large language models struggle to learn long-tail knowledge

    Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202 of PMLR, pages 15696--15707, 2023. URL https://proceedings.mlr.press/v202/kandpal23a.html

    work page 2023

  42. [54]

    Perez, Krzysztof Ociepa, ukasz Flis, Krzysztof Wr \'o bel, and Adrian Gwo \'z dziej

    Remigiusz Kinas, Pawe Kiszczak, Sergio P. Perez, Krzysztof Ociepa, ukasz Flis, Krzysztof Wr \'o bel, and Adrian Gwo \'z dziej. Bielik-minitron-7b: Compressing large language models via structured pruning and knowledge distillation for the polish language, 2026. arXiv:2603.11881

    work page arXiv 2026

  43. [55]

    Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs

    Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. Semantic entropy probes: Robust and cheap hallucination detection in LLM s, 2024. URL https://arxiv.org/abs/2406.15927. ICML 2024 Workshop on Foundation Models in the Wild

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  44. [56]

    Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In International Conference on Learning Representations (ICLR), 2023. URL https://arxiv.org/abs/2302.09664

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  45. [57]

    Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, and Sanjiv Kumar

    Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J. Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, and Sanjiv Kumar. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. In International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id=TJ2nxciYCk-

    work page 2023

  46. [58]

    Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivo...

    work page 2025

  47. [59]

    Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time

    Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher R \'e , and Beidi Chen. Deja vu: Contextual sparsity for efficient LLM s at inference time. In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202 of PMLR, pages 22137--22176, 2023. URL https...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  48. [60]

    When not to trust language models: Investigating effectiveness of parametric and non-parametric memories

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802--9822. Association for Comput...

    work page doi:10.18653/v1/2023.acl-long.546 2023

  49. [61]

    Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. SelfCheckGPT : Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2023. URL https://arxiv.org/abs/2303.08896

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  50. [62]

    No answer needed: Predicting LLM answer accuracy from question-only linear probes, 2025

    Iv \'a n Vicente Moreno Cencerrado, Arnau Padr \'e s Masdemont, Anton Gonzalvez Hawthorne, David Demitri Africa, and Lorenzo Pacchiardi. No answer needed: Predicting LLM answer accuracy from question-only linear probes, 2025. URL https://arxiv.org/abs/2509.10625. Accepted (poster), Principled Design for Trustworthy AI Workshop, ICLR 2026

    work page arXiv 2025

  51. [63]

    Compact language models via pruning and knowledge distillation

    Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024. URL https://openreview.net/forum?id=9U0nLnNMJ7

    work page 2024

  52. [64]

    interpreting GPT : the logit lens

    nostalgebraist. interpreting GPT : the logit lens. LessWrong, 2020. URL https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens. Published August 31, 2020

    work page 2020

  53. [65]

    Real-time detection of hallucinated entities in long-form generation, 2025

    Oscar Obeso, Andy Arditi, Javier Ferrando, Joshua Freeman, Cameron Holmes, and Neel Nanda. Real-time detection of hallucinated entities in long-form generation, 2025. URL https://arxiv.org/abs/2509.03531

    work page arXiv 2025

  54. [66]

    Bielik v3 Small: Technical Report

    Krzysztof Ociepa, ukasz Flis, Remigiusz Kinas, Krzysztof Wr \'o bel, and Adrian Gwo \'z dziej. Bielik v3 small: Technical report, 2025 a . URL https://arxiv.org/abs/2505.02550

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  55. [67]

    Bielik 11B v2 Technical Report

    Krzysztof Ociepa, ukasz Flis, Krzysztof Wr \'o bel, Adrian Gwo \'z dziej, and Remigiusz Kinas. Bielik 11b v2 technical report, 2025 b . URL https://arxiv.org/abs/2505.02410

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  56. [68]

    LLM s know more than they show: On the intrinsic representation of LLM hallucinations

    Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. LLM s know more than they show: On the intrinsic representation of LLM hallucinations. In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=KRnsX5Em3W

    work page 2025

  57. [69]

    Steer LLM Latents for Hallucination Detection

    Seongheon Park, Xuefeng Du, Min-Hsuan Yeh, Haobo Wang, and Yixuan Li. Steer LLM latents for hallucination detection, 2025. URL https://arxiv.org/abs/2503.01917. International Conference on Machine Learning (ICML) 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  58. [70]

    The effective rank: A measure of effective dimensionality

    Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 15th European Signal Processing Conference (EUSIPCO), pages 606--610, Pozna \'n , Poland, 2007. URL https://www.eurasip.org/Proceedings/Eusipco/Eusipco2007/Papers/a5p-h05.pdf

    work page 2007

  59. [71]

    LLM -check: Investigating detection of hallucinations in large language models

    Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kattakinda, and Soheil Feizi. LLM -check: Investigating detection of hallucinations in large language models. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 34188--34216, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash...

    work page 2024

  60. [72]

    MIND : Unsupervised real-time hallucination detection based on the internal states of large language models

    Weihang Su, Changyue Wang, Qingyao Ai, Yiran Hu, Zhijing Wu, Yujia Zhou, and Yiqun Liu. MIND : Unsupervised real-time hallucination detection based on the internal states of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14379--14391. Association for Computational Linguistics, 2024. URL https://aclanth...

    work page 2024

  61. [73]

    Massive Activations in Large Language Models

    Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. Massive activations in large language models. Conference on Language Modeling (COLM), 2024. URL https://arxiv.org/abs/2402.17762

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  62. [74]

    SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared Task on Hallucinations and Related Observable Overgeneration Mistakes

    Ra \'u l V \'a zquez, Timothee Mickus, Elaine Zosa, Teemu Vahtola, J \"o rg Tiedemann, Aman Sinha, Vincent Segonne, Fernando S \'a nchez-Vega, Alessandro Raganato, Jind r ich Libovick \'y , Jussi Karlgren, Shaoxiong Ji, Jind r ich Helcl, Liane Guillou, Ona de Gibert, Jaione Bengoetxea, Joseph Attieh, and Marianna Apidianaki. SemEval -2025 task 3: M u- SHR...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  63. [75]

    Retrieval head mechanistically explains long-context factuality

    Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically explains long-context factuality. In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=EytBpUGB1Z. Oral presentation

    work page 2025

  64. [76]

    Do large language models know what they don't know? In Findings of the Association for Computational Linguistics: ACL 2023, pages 8653--8665

    Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they don't know? In Findings of the Association for Computational Linguistics: ACL 2023, pages 8653--8665. Association for Computational Linguistics, 2023. doi:10.18653/v1/2023.findings-acl.551. URL https://aclanthology.org/2023.findings-acl.551/

    work page doi:10.18653/v1/2023.findings-acl.551 2023