Transformer Geometry Observatory TGO-I: Spectral Geometry Observatory

Kaustubh Kapil; Kishor P. Upla

arxiv: 2606.19249 · v1 · pith:BA5YNYBFnew · submitted 2026-06-17 · 💻 cs.CV · cs.LG

Transformer Geometry Observatory TGO-I: Spectral Geometry Observatory

Kaustubh Kapil , Kishor P. Upla This is my paper

Pith reviewed 2026-06-26 21:38 UTC · model grok-4.3

classification 💻 cs.CV cs.LG

keywords Vision Transformersrepresentational geometryspectral analysiseffective dimensionalityanisotropyCLS tokeneigenspectra

0 comments

The pith

Vision Transformer training redistributes variance across representational dimensions rather than concentrating it.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents the Transformer Geometry Observatory framework to examine the spectral properties of Vision Transformer representations. It trains a ViT-Small/16 on ImageNet-100 and computes multiple spectral metrics including effective rank and spectral entropy at different stages. The key finding is that training increases dimensional utilization with flatter eigenspectra and reduced anisotropy, contradicting the expectation of information concentration in few directions. This effect is strongest in the CLS token output.

Core claim

Analysis of ViT representations shows a consistent increase in dimensional utilization, decreasing anisotropy, increasing spectral entropy, and progressively flatter eigenspectra during training, with the final CLS token representation exhibiting the highest effective dimensionality and lowest anisotropy.

What carries the argument

Spectral geometry analysis using metrics such as Effective Rank, Stable Rank, Participation Ratio, Spectral Entropy, Spectral Flatness, Spectral Anisotropy, and examination of covariance structure, eigenspectra, and singular value spectra.

If this is right

Effective dimensionality of representations increases over the course of training.
Anisotropy in the representations decreases as training progresses.
The CLS token becomes the representation with the highest effective dimensionality within the network.
Spectral entropy and participation ratio increase, indicating more uniform use of dimensions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

This suggests that ViTs may learn more robust features by spreading information across dimensions.
Similar redistribution might be observable in other transformer-based models beyond vision.
Training dynamics could be monitored using these spectral metrics to detect convergence or issues.

Load-bearing premise

The spectral properties observed in this specific ViT-Small/16 model trained on ImageNet-100 reflect general behavior of Vision Transformers rather than being unique to this setup.

What would settle it

If a different Vision Transformer model or training on a larger dataset shows decreasing effective dimensionality and increasing anisotropy in the CLS token, the observed phenomenon would not hold generally.

Figures

Figures reproduced from arXiv: 2606.19249 by Kaustubh Kapil, Kishor P. Upla.

**Figure 1.** Figure 1: Effective Rank evolution across 100 epochs for all layers. It can be observed that over the course of training ViT, the [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

**Figure 2.** Figure 2: Evolution of spectral anisotropy and spectral entropy throughout training. Spectral anisotropy decreases across most [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Figure 3: CLS Covariance for Epoch 100 shows how the off [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 4.** Figure 4: Evolution of the final CLS representation across epochs 20, 50, and 100. Rows correspond to covariance matrices, [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

read the original abstract

Despite the widespread adoption of Vision Transformers (ViTs) and their success across numerous computer vision applications, the fundamental understanding of their dimensional and representational geometry remains relatively underexplored. To address this gap, we introduce Transformer Geometry Observatory (TGO), a systematic framework of experiments and analysis pipelines designed to investigate the representational geometry and dynamics of Vision Transformers. TGO-I, the first installment of the framework, focuses on the spectral geometry of ViT representations. Using a ViT-Small/16 model trained on ImageNet-100, we analyze Effective Rank, Stable Rank, Participation Ratio, Spectral Entropy, Spectral Flatness, Spectral Anisotropy, covariance structure, eigenspectra, and singular value spectra throughout training. Our results reveal a consistent increase in dimensional utilization, accompanied by decreasing anisotropy, increasing spectral entropy, increasing participation ratio, and progressively flatter eigenspectra. Contrary to the common intuition that training should concentrate information into a small number of dominant directions, we observe a progressive redistribution of variance across representational dimensions. This phenomenon is particularly pronounced in the final CLS token representation, which exhibits the highest effective dimensionality and lowest anisotropy within the network.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This is a clean but narrow measurement study: one ViT-Small training run on ImageNet-100 shows spectral flattening and rising effective dimensionality, especially in the CLS token.

read the letter

The main observation is that variance redistributes across more dimensions as training progresses instead of concentrating in a few dominant ones. They track this with standard tools—effective rank, participation ratio, spectral entropy, flatness, and anisotropy—on activations from a single ViT-Small/16 checkpoint sequence.

The paper does the measurement work carefully. Computing the full set of related spectral quantities on the same data and showing they move together gives a coherent picture. The CLS token ending up with the highest effective dimensionality is a concrete detail worth noting.

The limitation is the scope. All trends come from one model, one dataset subset, and one training trajectory. No multiple seeds, no other ViT widths or depths, no different datasets, and no error bars or variability estimates. Spectral metrics are sensitive to the precise covariance structure, so an idiosyncratic run could produce the same flattening without it being a general ViT property. The abstract frames the result as contrary to common intuition about training, but that framing rests on limited evidence.

This is useful for researchers who study representation geometry in transformers and want empirical baselines on how covariance evolves. It is not a new method or a broad claim that has been stress-tested.

I would bring the paper to a reading group to discuss the plots and what controls would be needed next. I would not cite it in my own work until the finding is checked on more runs and architectures. It deserves peer review because the measurements are reproducible in principle and the observation is clear enough to be worth testing further.

Referee Report

2 major / 1 minor

Summary. The manuscript introduces the Transformer Geometry Observatory (TGO-I) framework for investigating the spectral geometry of Vision Transformer representations. On a single ViT-Small/16 model trained on ImageNet-100, it computes Effective Rank, Stable Rank, Participation Ratio, Spectral Entropy, Spectral Flatness, Spectral Anisotropy, covariance structure, eigenspectra, and singular value spectra across training, reporting a progressive increase in effective dimensionality, participation ratio, spectral entropy and flatness together with decreasing anisotropy—particularly pronounced in the final CLS token—contrary to the intuition that training concentrates variance into few dominant directions.

Significance. If the trends prove robust, the work supplies a useful empirical baseline and systematic pipeline for spectral analysis of ViT activations, directly computing standard quantities without fitted parameters. This could inform representational studies in computer vision, though the single-run design restricts claims of generality.

major comments (2)

[Abstract] Abstract: the central claim that the observed redistribution of variance constitutes a general property of ViT representational geometry during training is load-bearing for the paper's contribution, yet all trends derive from one ViT-Small/16 checkpoint sequence on ImageNet-100 with no reported error bars, statistical tests, ablation on metrics, or controls for training stochasticity.
[Abstract] Abstract: the assumption that the chosen spectral metrics on this specific architecture, dataset subset, and training run capture general properties (rather than run- or data-specific covariance structure) is not tested; no additional seeds, ViT depths/widths, or datasets are shown to support reproducibility of the flattening trend.

minor comments (1)

The abstract could more explicitly separate descriptive observations on this model from broader implications for Vision Transformers.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed review and for identifying the need to carefully bound the scope of our empirical observations. TGO-I is presented as an initial case study within a larger framework, and we will revise the manuscript to make this explicit while preserving the reported measurements on the single ViT-Small/16 run.

read point-by-point responses

Referee: [Abstract] Abstract: the central claim that the observed redistribution of variance constitutes a general property of ViT representational geometry during training is load-bearing for the paper's contribution, yet all trends derive from one ViT-Small/16 checkpoint sequence on ImageNet-100 with no reported error bars, statistical tests, ablation on metrics, or controls for training stochasticity.

Authors: The manuscript reports concrete observations obtained from a single, fully specified training trajectory rather than asserting that the redistribution is a general property of all ViT models. The abstract employs the phrasing “we observe” and “our results reveal” to reflect this empirical scope. Because the spectral quantities are deterministic functions of the saved activations, error bars and statistical tests across stochastic runs are not applicable to the current data release. We will add an explicit limitations paragraph stating that the trends are specific to this architecture–dataset pair and that multi-seed controls are planned for subsequent TGO installments. revision: partial
Referee: [Abstract] Abstract: the assumption that the chosen spectral metrics on this specific architecture, dataset subset, and training run capture general properties (rather than run- or data-specific covariance structure) is not tested; no additional seeds, ViT depths/widths, or datasets are shown to support reproducibility of the flattening trend.

Authors: TGO-I deliberately restricts itself to a canonical ViT-Small/16 model on ImageNet-100 to establish a reproducible baseline pipeline. The paper does not claim that the flattening trend generalizes beyond the reported configuration; the contribution lies in the systematic computation of the listed spectral descriptors on this setup. We will revise the abstract and introduction to foreground the case-study nature of the work and will add a dedicated “Future Work” subsection outlining extensions to additional seeds, depths, and datasets. revision: yes

Circularity Check

0 steps flagged

No circularity: direct empirical computation of standard spectral metrics

full rationale

The paper reports direct computation of Effective Rank, Participation Ratio, Spectral Entropy, and related quantities on activations from one ViT-Small/16 training run on ImageNet-100. No equations derive a result from fitted parameters, no predictions are made that reduce to the input data by construction, and no self-citations are invoked as load-bearing uniqueness theorems. The central observations are therefore independent of any internal fitting or definitional loop.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Empirical observational study; relies on standard definitions of effective rank, participation ratio, and spectral entropy from prior literature with no new free parameters, axioms, or postulated entities introduced.

pith-pipeline@v0.9.1-grok · 5731 in / 1014 out tokens · 20825 ms · 2026-06-26T21:38:45.737397+00:00 · methodology

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Transformer Geometry Observatory TGO-II: Representational Similarity Observatory
cs.CV 2026-07 unverdicted novelty 4.0

TGO-II analysis of ViT-Small/16 training finds decreasing CKA and SVCCA, rising then stable intrinsic dimensionality, and persistent token covariance, indicating simultaneous specialization and manifold expansion with...

Reference graph

Works this paper leans on

3 extracted references · 1 canonical work pages · cited by 1 Pith paper · 1 internal anchor

[1]

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” 2021. [Online]. Available: https://arxiv.org/abs/2010.11929

work page internal anchor Pith review Pith/arXiv arXiv 2021
[2]

Imagenet: A large-scale hierarchical image database,

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” inIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009, pp. 248–255

2009
[3]

Xcit: Cross-covariance image transformers,

A. Ali, H. Touvron, M. Caron, P. Bojanowski, M. Douze, A. Joulin, I. Laptev, N. Neverova, G. Synnaeve, J. Verbeek, and H. Jegou, “Xcit: Cross-covariance image transformers,” inAdvances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y . Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 20 014–20 ...

2021

[1] [1]

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” 2021. [Online]. Available: https://arxiv.org/abs/2010.11929

work page internal anchor Pith review Pith/arXiv arXiv 2021

[2] [2]

Imagenet: A large-scale hierarchical image database,

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” inIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009, pp. 248–255

2009

[3] [3]

Xcit: Cross-covariance image transformers,

A. Ali, H. Touvron, M. Caron, P. Bojanowski, M. Douze, A. Joulin, I. Laptev, N. Neverova, G. Synnaeve, J. Verbeek, and H. Jegou, “Xcit: Cross-covariance image transformers,” inAdvances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y . Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 20 014–20 ...

2021