Pith. sign in

REVIEW 3 major objections 6 minor 90 references

This paper claims that a sharp drop in representation isotropy between two adjacent layers marks the moment a decoder-only language model commits to a multiple-choice answer, and the steepness of that drop predicts task accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-05 00:11 UTC pith:PPDWS4OP

load-bearing objection A plausible geometric observation undermined by post hoc layer selection; worth a serious referee but far from established. the 3 major comments →

arxiv 2608.00828 v1 pith:PPDWS4OP submitted 2026-08-01 cs.AI

Isotropy Cliffs: The Geometric Signature of Decision-Making in Large Language Models

Okan S. Coskun , Florian Rottach , Carsten Eickhoff , William Rudman This is my paper
classification cs.AI
keywords isotropy cliffIsoScorerepresentation geometrylayer-wise analysismultiple-choice question answeringinterpretabilityCKAintrinsic dimensionality
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Sentence embeddings inside a language model do not gradually sharpen their view of a multiple-choice question; at one specific layer pair, the representation space abruptly becomes anisotropic, task-relevant clusters pop out, and the model commits to an answer. The paper tries to establish that this 'isotropy cliff' is a universal geometric signature of decision-making in decoder-only models, present across five architectures, four QA benchmarks, four prompt formats, and few-shot settings. If true, the steepness of the cliff is a practical, label-free predictor of task performance—better than intrinsic dimensionality—and pinpoints where interventions like activation steering or fine-tuning should be aimed. The paper reports Spearman correlations of -0.84 globally and as high as -0.92 for a single model between the log-isotropy change at the transition and task accuracy.

Core claim

The central claim is that the middle-to-late layers of decoder-only LLMs contain a geometric phase transition: isotropy rises until a critical layer L, then drops sharply at layer L+1, at the same layers where CKA shows maximal representational change and the k-NN score jumps from near-zero to high purity. The paper calls this drop an isotropy cliff and argues that it marks the moment the model 'becomes decisive'—sacrificing uniform use of the embedding space to compress information into task-relevant clusters. The steepness of the cliff, measured as the change in log-IsoScore between those two layers, is claimed to be a strong predictor of downstream accuracy, outperforming intrinsic dimens

What carries the argument

The central object is the isotropy cliff: the sharp drop in IsoScore between two adjacent transition layers (for example, layers 23 to 24 in Qwen3-8B). IsoScore measures how uniformly embedding variance is spread across dimensions, with 1 meaning perfect uniformity and 0 meaning variance collapsed into a few directions. The argument is carried by aligning three measurements at the same layer pair: the IsoScore drop, a spike in k-NN Score (local cluster purity), and a dip in CKA (representational similarity between consecutive layers), with t-SNE projections as qualitative confirmation. The transition layers are selected from the joint alignment of these three signals.

Load-bearing premise

That the transition layers for each model can be chosen after the fact from where the isotropy cliff, k-NN spike, and CKA dip line up, and that the correlations computed at those chosen layers are not inflated by that selection.

What would settle it

Recompute the delta-log-IsoScore versus accuracy Spearman correlation with transition layers fixed in advance (for example, selected on one task and applied to all others, or chosen by CKA alone without looking at isotropy). If the correlation collapses to near zero or changes sign, the cliff-accuracy link is an artifact of post hoc layer selection. A second test: find a configuration where isotropy drops sharply between adjacent layers but k-NN purity does not rise and accuracy does not improve; that lone counterexample would break the claimed necessity of the cliff for decision-making.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the cliff is universal, layer-wise accuracy prediction becomes possible without labels: measuring the log-isotropy change across candidate layer pairs could rank task difficulty or model competence.
  • The cliff layers are natural targets for activation steering, fine-tuning, or freezing, because the model commits to an answer there; intervening just before the cliff may offer the most directional freedom.
  • In-context learning appears to sharpen the same pre-existing transition rather than create a new mechanism, so few-shot gains and prompt-format robustness can be studied through this geometry.
  • The superiority of the isotropy cliff over intrinsic dimensionality gives model evaluators a cheaper geometric diagnostic for comparing models or checkpoints than running full behavioral evaluations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An implicit testable extension: if the cliff marks decision commitment, the same transition should appear in open-ended generation at the point where the next-token distribution collapses onto a small set of choices; the paper does not test this.
  • The correlation is computed at post hoc selected layers; a fairer protocol would hold out the layer-selection rule (for example, choose layers on a subset of tasks) and then measure the delta-log-isotropy versus accuracy on held-out tasks. Such a test is not in the paper, but it would determine whether the -0.84 is robust.
  • The cliff-steepness predictor could be compared against other geometric summary statistics at the same layers, such as spectral outlier counts or participation ratio; the paper only compares with intrinsic dimensionality.
  • For model families with multiple CKA dips (Qwen2.5-3B, Mistral-7B-v0.3), the single-pair analysis may understate the dynamics; a multi-transition extension could reveal whether accuracy correlates with the steepest cliff or with the sum of cliffs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes that decoder-only LLMs undergo a sharp 'isotropy cliff' at a specific transition layer pair, where IsoScore drops while k-NN cluster purity rises and CKA similarity to the previous layer falls. This cliff is claimed to mark the formation of task-relevant clusters, and the steepness of the cliff (measured as ΔLI, the change in log-IsoScore from L1 to L2) is reported to correlate strongly with downstream MCQA accuracy (Spearman ρ up to -0.92, global -0.84) across five models, four benchmarks, four prompt formats, and three shot counts. The paper uses this correlation to argue that ΔLI is a superior predictor of accuracy compared to intrinsic dimension, and that the transition is robust to prompt variation. The analysis is performed on intermediate residual-stream embeddings at the final token position, using IsoScore, k-NN score, CKA, and TwoNN.

Significance. If the central claim were fully supported, the paper would provide a practical and interpretable geometric signature for locating decision-critical layers in LLMs, with potential value for activation steering and for predicting task accuracy from internal states. The paper is transparent in releasing per-configuration numerical results (Appendix Table 2), and the multi-model, multi-task, multi-prompt setup is a useful contribution. The authors also make an explicit comparison with intrinsic dimension and CKA, two active lines of work. However, the significance is currently limited by a load-bearing methodological issue: the transition layers are selected post hoc using the very signals that are then reported as synchronized and correlated with accuracy. This makes the headline correlations in-sample and partly tautological, and the paper's own appendix shows many configurations where the selected 'cliff' is actually an increase in isotropy, contradicting the robustness claim. These concerns are fixable with further analysis, but they must be addressed before the empirical claims can be accepted.

major comments (3)
  1. [Section 4, Table 1] Post hoc selection of transition layers is load-bearing. The paper states (Section 4): 'we will focus on the two transition layers for each model, chosen based on the jointly observed cliff in isotropy, k-NN Score, and CKA overlap.' The synchronization correlations (ΔLI–KS, ΔLI–CKA) in Table 1 are then computed at these same selected layers, so the co-occurrence is at least partly guaranteed by construction. More importantly, the accuracy correlations are only computed at these pairs. No fixed rule (e.g., maximal IsoScore drop) or out-of-sample validation is provided, so the word 'predictor' in the abstract is not justified. Please validate with a pre-registered layer-selection rule, or leave-one-task/prompt/out-of-sample selection, and report correlations on the held-out data.
  2. [Appendix Table 2] The selected layers do not consistently exhibit a cliff. For Qwen2.5-3B on MMLU-Pro, the IsoScore deltas at the selected L1→L2 pair are +.13, -.02, +.01, +.05, +.15, +.06, +.02, +.15, +.02, +.09, +.06, -.01 across formats/shots; most are positive. Similar positive deltas appear in Llama-3.1-8B and Llama-3.2-3B rows on MMLU-Pro. These configurations are included in the correlations of Table 1 and Figure 2, so the global ρ=-0.84 mixes genuine cliffs with anti-cliffs. This directly contradicts the abstract's claim that the transition is 'robust to prompt variations.' The paper must either exclude such configurations under a pre-specified rule or report the correlation separately for configurations where ΔLI is actually negative.
  3. [Section 6, Limitations] The admitted existence of multiple representational shifts undermines the single-pair design. The limitations state that Qwen2.5-3B and Mistral-7B-v0.3 'undergo multiple representational shifts ... rather than a single isolated transition.' Yet Table 1 reports only one pair per model. If several shifts exist, the chosen pair may be one of several, and the reported cliff steepness may be sensitive to which pair is chosen. Please report how the ΔLI–accuracy correlation varies over a range of candidate transition pairs (e.g., all pairs with a local IsoScore decrease, or all pairs within the CKA-defined transition zone) to demonstrate the result is not an artifact of selecting a single favorable pair.
minor comments (6)
  1. [Section 1, Figure 1 caption] 'showing anisotropy cliff' should be 'showing an isotropy cliff.'
  2. [Section 5] The text uses both 'hump-shaped' and 'hunchback' for the intrinsic-dimension profile; please use one term for consistency.
  3. [Figure 2 caption] The caption says 'two consecutive respective layers' but does not specify that these are the L1→L2 pairs from Table 1 per model; please state this explicitly.
  4. [Appendix Table 2] Please clarify whether the reported IsoScore delta is the standard IsoScore difference (L2−L1) or the log-IsoScore difference (ΔLI), and include this definition in the caption for self-containedness.
  5. [Section 3.1, Equation 5] The CKA formulation uses linear kernel (X X^T), but this should be stated explicitly. Also specify whether the representations are centered and whether the kernel is linear or RBF.
  6. [References] Several references are incomplete: 'Ansuini et al.' and 'Turner et al.' appear in the text but lack years and full details in the bibliography; please complete them.

Circularity Check

2 steps flagged

Transition layers are selected on the joint cliff/kNN/CKA signal, making the reported synchronization partly tautological and the accuracy 'prediction' an in-sample fit at fitted layers.

specific steps
  1. self definitional [Section 4, paragraph after CKA analysis (before Table 1)]
    "For the following analysis, we will focus on the two transition layers for each model, chosen based on the jointly observed cliff in isotropy, k-NN Score, and CKA overlap, which together indicate a major reorganization of the representation geometry."

    The paper first defines the two transition layers as the layers where the isotropy cliff, k-NN spike, and CKA drop are jointly observed, and then reports the co-occurrence of these signals as a discovered 'synchronized geometric behavior.' The synchronization is therefore guaranteed by the selection rule rather than independently evidenced. The claim in Figure 3 that clustering and the drop in isotropy are 'precisely aligned within these zones' is a restatement of how the layers were chosen.

  2. fitted input called prediction [Section 4.2 'Statistical Validation and Predictive Power' (Table 1 and surrounding text)]
    "Across all models, a highly consistent pattern aligns with our phase transition hypothesis. A strong negative correlation between ΔLI and both KS and accuracy demonstrates that a drop in isotropy is fundamentally associated with an increase in both clustering quality and task performance."

    The correlations in Table 1 are computed at layer pairs that were selected post hoc from the same data on which the correlations are measured. The layer identity acts as a fitted input: it is chosen because the cliff, cluster spike, and CKA drop are visible there, and then ΔLI at those same layers is called a 'predictor' of accuracy. No fixed-rule or out-of-sample validation is provided, so the reported Spearman values are in-sample fits, not predictions. The appendix data aggravates this: many configurations show positive Δ, e.g., Qwen2.5-3B on MMLU-Pro (+.13, +.05, +.02, +.02, +.09, +.06) across formats, meaning the selected pair is one of several possible transitions and the 'consistent trend' is not a stable pre-specified feature.

full rationale

The paper's central derivation chain contains partial circularity. The two 'transition layers' are not discovered independently; they are selected as the layers where the isotropy cliff, k-NN spike, and CKA drop jointly appear. Reporting that these three signals coincide at those layers is therefore partly tautological. The accuracy correlation is then computed only at those selected pairs, making the 'predictor' framing an in-sample description rather than an out-of-sample prediction; the paper's own Section 6 limitation and Appendix Table 2 show that many configurations lack a cliff or show multiple representational shifts, confirming that the selected pair is a post hoc choice. This warrants a score of 6, not higher, because the underlying geometric measures (IsoScore, k-NN, CKA) are independently defined and the paper does not rely on a load-bearing self-citation chain or rename a known result. The circularity is real but confined to the layer-selection and in-sample validation of the headline correlation.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central claims rest on the per-model choice of transition layers, which is an ad hoc selection made from the same measurements used for validation. The only free parameters are the layer indices and the k-NN neighborhood size. No new physical or conceptual entities are introduced.

free parameters (2)
  • Per-model transition layer pair (L1 to L2) = Qwen2.5-3B: 26 to 27; Llama-3.2-3B: 14 to 15; Qwen3-8B: 23 to 24; Llama-3.1-8B: 16 to 17; Mistral-7B-v0.3: 18 to 19
    Chosen post hoc based on the jointly observed cliff in isotropy, k-NN score, and CKA overlap. This selection is used as the basis for all subsequent correlation analyses and is effectively fitted to the data.
  • k in k-NN score = 5
    The neighborhood size is fixed at 5 without an ablation or justification; the results may depend on this choice.
axioms (4)
  • domain assumption Final-token residual stream embeddings capture the model's decision-relevant state
    The paper uses s_l = H_l_T for the final input token as the sentence embedding for all geometric analyses (Eq. 1), assuming this position represents the decision.
  • domain assumption Ground-truth labels define the relevant clusters for the k-NN score
    Equation 3 uses ground-truth labels L(n_j), assuming that behavioral clustering on the true answer is the meaningful structure to detect.
  • standard math IsoScore as defined by Rudman et al. (2022) is a valid measure of isotropy
    The paper imports IsoScore without re-derivation, relying on a prior published metric.
  • standard math TwoNN estimator gives reliable intrinsic dimension estimates
    The baseline metric in Eq. 4 is imported from Facco et al. (2017) without verification in this setting.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Isotropy Cliffs: The Geometric Signature of Decision-Making in Large Language Models." pith.science (2026). https://pith.science/paper/PPDWS4OP

@misc{pith2026260800828,
  author       = {Pith},
  title        = {Pith review of: Isotropy Cliffs: The Geometric Signature of Decision-Making in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PPDWS4OP}},
  note         = {Machine review of arXiv:2608.00828}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We investigate the geometry of decision-making in Multiple Choice Question Answering (MCQA) through the lens of isotropy. Analyzing five open-weight models across diverse datasets, we identify decision-critical transition layers characterized by a shift in isotropy, coinciding with a major representational change and the emergence of task-relevant clusters. We demonstrate that this synchronized geometric behavior is strongly correlated with downstream accuracy ($r\approx0.84$), displaying its relevance for successful decision-making. Furthermore, we show that this transition is robust to prompt variations, suggesting that it reflects a general mechanism of model behavior.

Figures

Figures reproduced from arXiv: 2608.00828 by Carsten Eickhoff, Florian Rottach, Okan S. Coskun, William Rudman.

Figure 1
Figure 1. Figure 1: Dynamics of Representation Collapse and Cluster Formation. Analysis of the Qwen3-8B model on the Head-QA dataset. The dual-axis plot tracks the IsoScore and k-NN Score, showing a critical transition between the layers 23–24. The t-SNE projection at layer 23 shows overlapping representations with high isotropy, while t-SNE projection at layer 24 shows the emergence of distinct class clusters following the s… view at source ↗
Figure 2
Figure 2. Figure 2: Relationship between layer-wise isotropy dynamics and model performance. Each data point represents a specific evaluation configuration (task, prompt format, and shot-count) for each of the five models at two consecutive respective layers. geometric organization of representations, analyz￾ing how isotropy evolves across layers and relates to the emergence of task-aligned structure. Liu et al. (2025) identi… view at source ↗
Figure 3
Figure 3. Figure 3: Layer-wise evolution of mean IsoScore and k-NN with 95% Confidence Intervals (CI) across experiments. Shaded regions indicate the layers of maximal representational change as identified by CKA (see [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Layer-wise evolution of mean Intrinsic Dimension with 95% CI across experiments. tion tensor of shape (N, L, D). This tensor forms the empirical basis for our geometric analysis of the model’s internal representations. 3.1 Geometric and Similarity Measures We employ a suite of metrics to characterize the embedding space, distinguishing between global geometry, local separability, and representational simil… view at source ↗
Figure 5
Figure 5. Figure 5: Layer-wise CKA trajectories for all evalu￾ated models. Averaged across all experimental settings, including datasets, prompts, and in-context learning con￾figurations. Shaded regions indicate 95% confidence intervals across tasks. 4 Results We begin by examining the layer-wise evolution of representational geometry across all five model families [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Clustering analysis of model activations on OpenBookQA using prompt format 0. Low-dimensional t-SNE projections of the residual stream show representations grouped by ground-truth answer choices. This alignment demonstrates behavioral clustering (Rimsky et al., 2024), where semantic distinctions between choices are reflected in the geometry of the latent space. ric transition mechanism by increasing the se… view at source ↗
Figure 7
Figure 7. Figure 7: t-SNE projections of hidden representations across adjacent layers for the task OpenBookQA with prompt [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

90 extracted references · 31 canonical work pages · 2 internal anchors

  1. [1]

    2018 , eprint=

    Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering , author=. 2018 , eprint=

    2018

  2. [2]

    2024 , eprint=

    MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark , author=. 2024 , eprint=

    2024

  3. [3]

    2018 , eprint=

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge , author=. 2018 , eprint=

    2018

  4. [5]

    2024 , eprint=

    The Llama 3 Herd of Models , author=. 2024 , eprint=

    2024

  5. [6]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  6. [7]

    2023 , eprint=

    Mistral 7B , author=. 2023 , eprint=

    2023

  7. [8]

    2024 , eprint=

    Qwen2 Technical Report , author=. 2024 , eprint=

    2024

  8. [9]

    Estimating the intrinsic dimension of datasets by a minimal neighborhood information , volume=

    Facco, Elena and d’Errico, Maria and Rodriguez, Alex and Laio, Alessandro , year=. Estimating the intrinsic dimension of datasets by a minimal neighborhood information , volume=. Scientific Reports , publisher=. doi:10.1038/s41598-017-11873-y , number=

    doi:10.1038/s41598-017-11873-y

  9. [10]

    2024 , eprint=

    ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs , author=. 2024 , eprint=

    2024

  10. [11]

    Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions

    Pezeshkpour, Pouya and Hruschka, Estevam. Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions. Findings of the Association for Computational Linguistics: NAACL 2024. 2024. doi:10.18653/v1/2024.findings-naacl.130

    doi:10.18653/v1/2024.findings-naacl.130 2024

  11. [12]

    2024 , eprint=

    PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts , author=. 2024 , eprint=

    2024

  12. [13]

    Exploring the Sensitivity of LLM s' Decision-Making Capabilities: Insights from Prompt Variations and Hyperparameters

    Loya, Manikanta and Sinha, Divya and Futrell, Richard. Exploring the Sensitivity of LLM s' Decision-Making Capabilities: Insights from Prompt Variations and Hyperparameters. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023. doi:10.18653/v1/2023.findings-emnlp.241

    work page doi:10.18653/v1/2023.findings-emnlp.241 2023

  13. [14]

    2025 , eprint=

    Semantic Consistency for Assuring Reliability of Large Language Models , author=. 2025 , eprint=

    2025

  14. [15]

    2024 , eprint=

    Navigating Prompt Complexity for Zero-Shot Classification: A Study of Large Language Models in Computational Social Science , author=. 2024 , eprint=

    work page 2024

  15. [16]

    2021 , eprint=

    All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality , author=. 2021 , eprint=

    work page 2021

  16. [17]

    2024 , eprint=

    The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models , author=. 2024 , eprint=

    work page 2024

  17. [18]

    2025 , eprint=

    The Geometry of Tokens in Internal Representations of Large Language Models , author=. 2025 , eprint=

    work page 2025

  18. [19]

    2025 , eprint=

    Geometry of Decision Making in Language Models , author=. 2025 , eprint=

    2025

  19. [20]

    2024 , eprint=

    The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets , author=. 2024 , eprint=

    2024

  20. [21]

    Is Anisotropy Truly Harmful? A Case Study on Text Clustering

    Ait-Saada, Mira and Nadif, Mohamed. Is Anisotropy Truly Harmful? A Case Study on Text Clustering. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2023. doi:10.18653/v1/2023.acl-short.103

    doi:10.18653/v1/2023.acl-short.103 2023

  21. [22]

    Dense Passage Retrieval for Open-Domain Question Answering

    Karpukhin, Vladimir and Oguz, Barlas and Min, Sewon and Lewis, Patrick and Wu, Ledell and Edunov, Sergey and Chen, Danqi and Yih, Wen-tau. Dense Passage Retrieval for Open-Domain Question Answering. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. doi:10.18653/v1/2020.emnlp-main.550

    doi:10.18653/v1/2020.emnlp-main.550 2020

  22. [23]

    HIL : Hybrid Isotropy Learning for Zero-shot Performance in Dense retrieval

    Kim, Jaeyoung and Lee, Dohyeon and Hwang, Seung-won. HIL : Hybrid Isotropy Learning for Zero-shot Performance in Dense retrieval. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024. doi:10.18653/v1/2024.naacl-long.437

    work page doi:10.18653/v1/2024.naacl-long.437 2024

  23. [24]

    2025 , eprint=

    Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings , author=. 2025 , eprint=

    work page 2025

  24. [25]

    International Conference on Learning Representations , year=

    Isotropy in the Contextual Embedding Space: Clusters and Manifolds , author=. International Conference on Learning Representations , year=

    work page

  25. [26]

    How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States

    Zhou, Zhenhong and Yu, Haiyang and Zhang, Xinghua and Xu, Rongwu and Huang, Fei and Li, Yongbin. How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024. doi:10.18653/v1/2024.findings-emnlp.139

    doi:10.18653/v1/2024.findings-emnlp.139 2024

  26. [27]

    2024 , eprint=

    Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models , author=. 2024 , eprint=

    2024

  27. [28]

    2025 , eprint=

    Effect of Selection Format on LLM Performance , author=. 2025 , eprint=

    work page 2025

  28. [29]

    2024 , eprint=

    Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting , author=. 2024 , eprint=

    2024

  29. [30]

    doi:10.48550/arXiv.1905.12784 , url =

    Intrinsic Dimension of Data Representations in Deep Neural Networks , author =. doi:10.48550/arXiv.1905.12784 , url =. 1905.12784 , eprinttype =

    Pith/arXiv arXiv doi:10.48550/arxiv.1905.12784 1905

  30. [31]

    Xiao, Chenghao and Long, Yang and Al Moubayed, Noura , editor =. On. Findings of the

    work page

  31. [32]

    Ethayarajh, Kawin , editor =. How. Proceedings of the 2019

    work page 2019

  32. [33]

    Bihani, Geetanjali and Rayz, Julia , year = 2021, pages =. Low. Proceedings of

    work page 2021

  33. [34]

    Persistent

    Gardinazzi, Yuri and Viswanathan, Karthik and Panerai, Giada and Ansuini, Alessio and Cazzaniga, Alberto and Biagetti, Matteo , year = 2025, month = jun, number =. Persistent. doi:10.48550/arXiv.2410.11042 , archiveprefix =. 2410.11042 , primaryclass =

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.11042 2025

  34. [35]

    Emergence of a

    Cheng, Emily and Doimo, Diego and Kervadec, Corentin and Macocco, Iuri and Yu, Jade and Laio, Alessandro and Baroni, Marco , year = 2025, month = apr, number =. Emergence of a. doi:10.48550/arXiv.2405.15471 , archiveprefix =. 2405.15471 , primaryclass =

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.15471 2025

  35. [36]

    Bihani, Geetanjali and Rayz, Julia , date =. Low. Proceedings of. doi:10.18653/v1/2021.deelio-1.9 , url =

    work page doi:10.18653/v1/2021.deelio-1.9 2021

  36. [37]

    Inference-

    Li, Kenneth and Patel, Oam and Pfister, Fernanda Viégas Hanspeter and Wattenberg, Martin , year=. Inference-

    work page

  37. [38]

    In-Context

    Liu, Sheng and Ye, Haotian and Xing, Lei and Zou, James , date =. In-Context. doi:10.48550/arXiv.2311.06668 , url =. 2311.06668 , eprinttype =

    Pith/arXiv arXiv doi:10.48550/arxiv.2311.06668

  38. [39]

    2025 , eprint=

    Spectral Insights into Data-Oblivious Critical Layers in Large Language Models , author=. 2025 , eprint=

    work page 2025

  39. [40]

    Steering Llama 2 via Contrastive Activation Addition

    Rimsky, Nina and Gabrieli, Nick and Schulz, Julian and Tong, Meg and Hubinger, Evan and Turner, Alexander. Steering Llama 2 via Contrastive Activation Addition. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024. doi:10.18653/v1/2024.acl-long.828

    doi:10.18653/v1/2024.acl-long.828 2024

  40. [41]

    IsoScore: Measuring the Uniformity of Embedding Space Utilization , url=

    Rudman, William and Gillman, Nate and Rayne, Taylor and Eickhoff, Carsten , year=. IsoScore: Measuring the Uniformity of Embedding Space Utilization , url=. doi:10.18653/v1/2022.findings-acl.262 , booktitle=

    doi:10.18653/v1/2022.findings-acl.262 2022

  41. [42]

    and Kaiser, Lukasz and Polosukhin, Illia , year=

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser, Lukasz and Polosukhin, Illia , year=. Attention. doi:10.48550/arXiv.1706.03762 , url =. 1706.03762 , eprinttype =

    Pith/arXiv arXiv doi:10.48550/arxiv.1706.03762

  42. [43]

    Dong, Qingxiu and Li, Lei and Dai, Damai and Zheng, Ce and Ma, Jingyuan and Li, Rui and Xia, Heming and Xu, Jingjing and Wu, Zhiyong and Liu, Tianyu and Chang, Baobao and Sun, Xu and Li, Lei and Sui, Zhifang , date =. A. doi:10.48550/arXiv.2301.00234 , url =. 2301.00234 , eprinttype =

    Pith/arXiv arXiv doi:10.48550/arxiv.2301.00234

  43. [44]

    Language

    Radford, Alec and Wu, Jeffrey and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya , year =. Language

  44. [45]

    Improving

    Radford, Alec and Narasimhan, Karthik and Salimans, Tim and Sutskever, Ilya , year=. Improving

    work page

  45. [46]

    Language

    Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel and Wu, Jeffrey and Winte...

    work page

  46. [47]

    Chain-of-

    Wei, Jason and Wang, Xuezhi and Schuurmans, Dale and Bosma, Maarten and Ichter, Brian and Xia, Fei and Chi, Ed H and Le, Quoc V and Zhou, Denny , year=. Chain-of-

    work page

  47. [48]

    Liu, Jiachang and Shen, Dinghan and Zhang, Yizhe and Dolan, Bill and Carin, Lawrence and Chen, Weizhu , date =. What. doi:10.48550/arXiv.2101.06804 , url =. 2101.06804 , eprinttype =

    Pith/arXiv arXiv doi:10.48550/arxiv.2101.06804

  48. [49]

    Taxonomy,

    Wehner, Jan and Abdelnabi, Sahar and Tan, Daniel and Krueger, David and Fritz, Mario , date =. Taxonomy,. doi:10.48550/arXiv.2502.19649 , url =. 2502.19649 , eprinttype =

    work page doi:10.48550/arxiv.2502.19649

  49. [50]

    and Mini, Ulisse and MacDiarmid, Monte , date =

    Turner, Alexander Matt and Thiergart, Lisa and Leech, Gavin and Udell, David and Vazquez, Juan J. and Mini, Ulisse and MacDiarmid, Monte , date =. Steering. doi:10.48550/arXiv.2308.10248 , url =. 2308.10248 , eprinttype =

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.10248

  50. [51]

    2024 , url=

    From Scaling Law to Sub-Scaling Law: Understanding the Diminishing Returns of Larger Models , author=. 2024 , url=

    work page 2024

  51. [52]

    arXiv:1803.05457v1 , year =

    Peter Clark and Isaac Cowhey and Oren Etzioni and Tushar Khot and Ashish Sabharwal and Carissa Schoenick and Oyvind Tafjord , title =. arXiv:1803.05457v1 , year =

    Pith/arXiv arXiv

  52. [53]

    Proceedings of the 57th

    Vilares, David and Gómez-Rodríguez, Carlos , editor =. Proceedings of the 57th. doi:10.18653/v1/P19-1092 , url =

    doi:10.18653/v1/p19-1092

  53. [54]

    doi:10.48550/arXiv.2406.01574 , url =

    Wang, Yubo and Ma, Xueguang and Zhang, Ge and Ni, Yuansheng and Chandra, Abhranil and Guo, Shiguang and Ren, Weiming and Arulraj, Aaran and He, Xuan and Jiang, Ziyan and Li, Tianle and Ku, Max and Wang, Kai and Zhuang, Alex and Fan, Rongqi and Yue, Xiang and Chen, Wenhu , date =. doi:10.48550/arXiv.2406.01574 , url =. 2406.01574 , eprinttype =

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.01574

  54. [55]

    EMNLP , year=

    Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering , author=. EMNLP , year=

  55. [56]

    2023 , eprint=

    Leveraging Large Language Models for Multiple Choice Question Answering , author=. 2023 , eprint=

    2023

  56. [57]

    Lyu, Chenyang and Wu, Minghao and Aji, Alham , editor =. Beyond. Proceedings of the 1st. doi:10.18653/v1/2024.knowllm-1.10 , url =

    work page doi:10.18653/v1/2024.knowllm-1.10 2024

  57. [58]

    NAACL , year =

    BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions , author =. NAACL , year =

  58. [59]

    Evaluating Consistency and Reasoning Capabilities of Large Language Models

    Saxena, Yash and Chopra, Sarthak and Tripathi, Arunendra Mani , date =. Evaluating. doi:10.48550/arXiv.2404.16478 , url =. 2404.16478 , eprinttype =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2404.16478

  59. [60]

    doi:10.48550/arXiv.2109.07958 , url =

    Lin, Stephanie and Hilton, Jacob and Evans, Owain , date =. doi:10.48550/arXiv.2109.07958 , url =. 2109.07958 , eprinttype =

    Pith/arXiv arXiv doi:10.48550/arxiv.2109.07958

  60. [61]

    Proceedings of the 36th International Conference on Machine Learning , pages =

    Similarity of Neural Network Representations Revisited , author =. Proceedings of the 36th International Conference on Machine Learning , pages =. 2019 , editor =

    2019

  61. [62]

    The effective rank: A measure of effective dimensionality , year=

    Roy, Olivier and Vetterli, Martin , booktitle=. The effective rank: A measure of effective dimensionality , year=

  62. [63]

    int8 (): 8-bit matrix multiplication for transformers at scale , author=

    Llm. int8 (): 8-bit matrix multiplication for transformers at scale , author=. arXiv preprint arXiv:2208.07339 , year=

    Pith/arXiv arXiv

  63. [64]

    and Wang, Zifan and Mallen, Alex and Basart, Steven and Koyejo, Sanmi and Song, Dawn and Fredrikson, Matt and Kolter, J

    Zou, Andy and Phan, Long and Chen, Sarah and Campbell, James and Guo, Phillip and Ren, Richard and Pan, Alexander and Yin, Xuwang and Mazeika, Mantas and Dombrowski, Ann-Kathrin and Goel, Shashwat and Li, Nathaniel and Byun, Michael J. and Wang, Zifan and Mallen, Alex and Basart, Steven and Koyejo, Sanmi and Song, Dawn and Fredrikson, Matt and Kolter, J. ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.01405

  64. [65]

    2023 , eprint=

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena , author=. 2023 , eprint=

    2023

  65. [66]

    Allenai/Truthfulqa-Truth-Judge-Llama2-

    work page

  66. [67]

    Allenai/Truthfulqa-Info-Judge-Llama2-

    work page

  67. [68]

    Isotropy, Clusters, and Classifiers

    Mickus, Timothee and Gr. Isotropy, Clusters, and Classifiers. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2024. doi:10.18653/v1/2024.acl-short.7

    work page doi:10.18653/v1/2024.acl-short.7 2024

  68. [69]

    and Chess, Benjamin and Child, Rewon and Gray, Scott and Radford, Alec and Wu, Jeffrey and Amodei, Dario , date =

    Kaplan, Jared and McCandlish, Sam and Henighan, Tom and Brown, Tom B. and Chess, Benjamin and Child, Rewon and Gray, Scott and Radford, Alec and Wu, Jeffrey and Amodei, Dario , date =. Scaling. doi:10.48550/arXiv.2001.08361 , url =. 2001.08361 , eprinttype =

    Pith/arXiv arXiv doi:10.48550/arxiv.2001.08361 2001

  69. [70]

    Ben-Artzy, Amit and Schwartz, Roy , date =. Attend. doi:10.48550/arXiv.2409.03621 , url =. 2409.03621 , eprinttype =

    Pith/arXiv arXiv doi:10.48550/arxiv.2409.03621

  70. [71]

    , date =

    O'Neill, Charles and Ye, Christine and Iyer, Kartheik and Wu, John F. , date =. Disentangling. doi:10.48550/arXiv.2408.00657 , url =. 2408.00657 , eprinttype =

    Pith/arXiv arXiv doi:10.48550/arxiv.2408.00657

  71. [72]

    2019 , eprint=

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , author=. 2019 , eprint=

    2019

  72. [73]

    , date =

    Wang, Shang and Zhu, Tianqing and Liu, Bo and Ding, Ming and Guo, Xu and Ye, Dayong and Zhou, Wanlei and Yu, Philip S. , date =. Unique. doi:10.48550/arXiv.2406.07973 , url =. 2406.07973 , eprinttype =

    work page doi:10.48550/arxiv.2406.07973

  73. [74]

    A Survey on Large Language Model (

    Yao, Yifan and Duan, Jinhao and Xu, Kaidi and Cai, Yuanfang and Sun, Zhibo and Zhang, Yue , date =. A Survey on Large Language Model (. doi:10.1016/j.hcc.2024.100211 , url =

    arXiv 2024

  74. [75]

    2024 , eprint=

    DeepSeek-V3 Technical Report , author=. 2024 , eprint=

    2024

  75. [76]

    2021 , journal=

    A Mathematical Framework for Transformer Circuits , author=. 2021 , journal=

    2021

  76. [77]

    and Hashimoto, Tatsunori and Vinyals, Oriol and Liang, Percy and Dean, Jeff and Fedus, William , date =

    Wei, Jason and Tay, Yi and Bommasani, Rishi and Raffel, Colin and Zoph, Barret and Borgeaud, Sebastian and Yogatama, Dani and Bosma, Maarten and Zhou, Denny and Metzler, Donald and Chi, Ed H. and Hashimoto, Tatsunori and Vinyals, Oriol and Liang, Percy and Dean, Jeff and Fedus, William , date =. Emergent. doi:10.48550/arXiv.2206.07682 , url =. 2206.07682 ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2206.07682

  77. [78]

    Arora, Sanjeev and Li, Yuanzhi and Liang, Yingyu and Ma, Tengyu and Risteski, Andrej , date =. Linear. doi:10.48550/arXiv.1601.03764 , url =. 1601.03764 , eprinttype =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1601.03764

  78. [79]

    arXiv preprint arXiv:2308.12966 , year=

    Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities , author=. arXiv preprint arXiv:2308.12966 , year=

    Pith/arXiv arXiv

  79. [80]

    Lin, Bill Yuchen and Deng, Yuntian and Chandu, Khyathi and Brahman, Faeze and Ravichander, Abhilasha and Pyatkin, Valentina and Dziri, Nouha and Bras, Ronan Le and Choi, Yejin , date =

    work page

  80. [81]

    Fighting Hate Speech, Silencing Drag Queens? Artificial Intelligence in Content Moderation and Risks to LGBTQ Voices Online , journal =

    Oliva, Thiago , year =. Fighting Hate Speech, Silencing Drag Queens? Artificial Intelligence in Content Moderation and Risks to LGBTQ Voices Online , journal =

    work page

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.