Pith. sign in

REVIEW 4 major objections 5 minor 16 references

IRIS: A Visual Cortex-Inspired Framework for Analyzing Orientation Selectivity in Vision Transformers

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single brain-inspired score, read off a frozen vision transformer, predicts where to start fine-tuning.

desk verdict A promising, carefully probed heuristic for picking LoRA unfreeze depth from a frozen backbone, whose 'best predictor' claim outruns the evidence. read the letter →

arxiv 2608.05122 v1 pith:UYDPTH7M submitted 2026-08-05 cs.CV

Vaishnavi B Mohan , Vijayakrishna Naganoor , Yashas Annadani , Shashank Hegde This is my paper
classification cs.CV
keywords orientationselectivityvisiontransformerrepresentationalsimilarityanalysisfine-tuningdepthLoRAmechanisticinterpretabilityV1-likefeaturesgratingstimuli
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision transformers are usually studied through their outputs, not their internals. The paper asks whether a globally-mixing architecture nevertheless builds a V1-like orientation code — units that prefer specific orientations of local structure — and shows that it does, in a depth-dependent way. Its central practical claim is that a population-level score, called RSS, computed on a completely frozen vision transformer before any fine-tuning, predicts the depth at which unfreezing with LoRA will give the best downstream performance; across six backbones spanning different pretraining objectives, the RSS peak lands on or almost exactly on the empirical fine-tuning optimum. The paper also claims that the training objective, not model scale or dataset, is the strongest determinant of where this orientation code lives, with models trained under the same paradigm peaking at comparable relative depths. If right, this turns a neuroscience-inspired probe into a cheap mechanistic heuristic for model selection and fine-tuning decisions.

What carries the argument

The central object is the representational similarity score (RSS): for a battery of 36 oriented gratings spanning phases and spatial frequencies, each stimulus's pooled patch-token activation is $\ell^2$-normalized, and RSS is the Pearson correlation between pairwise cosine dissimilarities of these activations and the circular distance between the orientations that evoked them. RSS is basis-invariant, so it can be compared across architectures, and it is computed block-by-block at matched relative depths $d \in \{0, 0.1, 0.25, 0.4, 0.5, 0.6, 0.75, 0.9, 1.0\}$, mapped to a block via $\mathrm{block}(d,L) = \operatorname{round}(d(L-1))$. It is complemented by three unit-level metrics — OSI (vector circular variance), ORS (fraction of units passing a shuffle-based significance test with a frequency-selection null), and von Mises half-width at half-maximum — applied to raw-coordinate, MLP-neuron, and SAE-feature readouts. The load-bearing role is played by RSS's peak: it marks the depth where activation geometry still preserves the orientation manifold, and the paper uses that peak both as a predictor of frozen-encoder generalization and as the a-priori choice of LoRA start layer.

What would settle it

Take a frozen backbone and compute its RSS-vs-depth curve; then run the paper's nine-depth LoRA unfreeze sweep on a held-out task or a backbone family not used in the paper, such as a contrastive model on a purely semantic dense task. If the pooled fine-tuning performance peaks at a start depth more than one relative-depth step away from the RSS peak, or if a model with a flat RSS curve nonetheless shows a clear single best unfreeze depth, the central claim is falsified. The paper itself reports this alignment for six backbones, so the concrete test is to find one counterexample on a new task or backbone.

Watch

Extended reading notes

Core claim

On the paper's own terms: orientation selectivity is a real, measurable property of ViT internal representations, organized by relative depth. Using a battery of Gaussian-windowed sinusoidal gratings and three readout bases (raw coordinates, MLP neurons, and sparse-autoencoder features), the authors define RSS as the Pearson correlation between pairwise cosine dissimilarities of pooled activations and the circular orientation distance of the stimuli that evoked them. They report an inverted-U RSS profile in nearly every backbone — weak early, peaking mid-depth, decaying toward semantic layers — whose peak position is determined primarily by the pretraining objective, with the DINO family sustaining it deepest, AIMv2 collapsing earliest, and MAE keeping it rising to the final layer. The same RSS curve, read off the frozen backbone, predicts both frozen-encoder generalization on nine tasks and the optimal LoRA unfreeze depth on six backbones, beating 18 other a-priori candidates in a head-to-head ranking. The explanation offered is that RSS marks where a general-purpose, V1-like low-level code hands off to task-specific semantic encoding, so unfreezing from that point preserves already-learned structure while adding adaptivity.

Load-bearing premise

The entire chain rests on the assumption that the synthetic-grating battery and mean-pooled activations faithfully capture the orientation code that these models actually use on natural images and during fine-tuning; if RSS on gratings is a proxy that misses how orientation is used in real adaptation, the RSS-to-fine-tuning-depth link would be an artifact of the probe rather than a property of the model.

Editorial extensions

If this is right

  • Fine-tuning budgets shrink: the RSS peak, computed once on a frozen backbone, selects the LoRA start layer without running an expensive sweep of unfreeze depths.
  • Model choice becomes mechanistically grounded: RSS curve shape predicts frozen-encoder generalization, with DINO-family peaks deep and broad, MAE never handing off, and AIMv2's early collapse explaining its poor transfer.
  • Scale laws for representation: within every family except AIMv2 (which scales width while fixing depth), RSS-vs-depth curves overlap across tiers, so relative depth, not block count, governs where the orientation code lives.
  • Training dynamics: early-to-middle layers recruit more orientation-selective units over training while deep layers broaden toward semantic encoding, a developmental pattern that mirrors biological critical periods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same RSS handoff point could generalize beyond LoRA to adapters, partial fine-tuning, or QLoRA, since the mechanism — preserving already-built low-level code while adapting deeper layers — is not specific to low-rank updates.
  • Because RSS is basis-invariant and defined on any activation geometry, the heuristic should transfer to non-transformer encoders and to other continuous low-level features such as spatial frequency, color contrast, or motion direction, which the paper names as future work.
  • A cheaper variant may exist: if RSS on synthetic gratings predicts natural-image transfer, a natural-image-only orientation statistics probe could replace the synthetic battery, making the diagnostic usable during pretraining itself.
  • The AIMv2 result suggests a training-time use for the metric: an RSS curve that collapses early could flag that a pretraining objective is sacrificing general-purpose low-level structure before any downstream benchmark is run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces IRIS, a suite of neuroscience-inspired metrics (RSS, ORS, OSI, HWHM) to quantify orientation selectivity in vision transformers. Using a battery of oriented gratings and natural-image-driven SAE features, the authors probe frozen backbones at matched relative depths and report three findings: training paradigm is the strongest determinant of the orientation code; scaling depth preserves the code at matched relative depth while scaling width changes it; and the peak of RSS measured on a frozen backbone predicts the optimal LoRA unfreeze depth for downstream fine-tuning, outperforming 18 other a-priori predictors (on one backbone) and common defaults. The paper also tracks the emergence of selectivity during training on a single OpenCLIP ViT-H/14.

Significance. The probing methodology is careful: the permutation null in Eq. (5) correctly accounts for frequency selection in the ORS statistic, the use of three readout bases (raw coordinates, MLP neurons, SAE features) is thoughtful, and the authors are candid about limitations (e.g., App. A.5's statement that no significance test is provided, Sec. 6's restriction to synthetic gratings, Sec. 4.3's unvalidated freeze-schedule conjecture). If the RSS-to-fine-tuning-depth prediction held with proper uncertainty quantification, the result would be practically valuable and would constitute a useful interpretability contribution. As it stands, the headline claims exceed the statistical evidence: the key head-to-head comparison is a single-backbone result, the six-backbone peak-coincidence evidence is qualitative, and the 'strongest determinant' claim is inferred without attribution analysis.

major comments (4)
  1. [Section 5 and App. A.5] The claim that RSS is the best a-priori predictor of the optimal LoRA unfreeze depth is supported head-to-head only on DINOv3 ViT-S/16, where 18 predictors are ranked; the appendix explicitly states that 'No single statistic here carries a formal significance test.' For the other five backbones in Fig. 5, no alternative predictors are evaluated, and the evidence is visual coincidence of two smooth curves with no null model for how often two such curves share an argmax. Because the paper's contribution (3) is that RSS is 'the best a-priori predictor,' this needs multi-backbone comparisons against at least a small set of baselines (e.g., ORS, decoding accuracy, mean activation norm, layer index heuristics) with bootstrap confidence intervals for the argmax gap.
  2. [Eq. (11) and Fig. 5] The empirical performance peak is reported as a single value (e.g., k=4 for DINOv3 ViT-S/16) and compared to the RSS peak without uncertainty quantification. With nine depth grid points and five seeds, adjacent depths may be statistically indistinguishable; a bootstrap over tasks and seeds, or a paired significance test, is needed to establish that the peak coincides beyond chance. The shaded band in Fig. 5 shows seed spread of the pooled score but does not give a confidence interval for the peak location, and RSS has no error bar at all since it is computed from a single frozen backbone.
  3. [Fig. 3 and Sec. 4.2] The claim that RSS curves 'overlap almost exactly' across scale within a family, and that relative depth rather than block count governs the orientation code, is made without error bars or a quantitative similarity criterion. The single exception (AIMv2) is then used to attribute the difference to width-vs-depth scaling, but AIMv2 also differs in training paradigm and data from the other families, and with one model per (family, tier) the comparison cannot separate scale from paradigm or architecture. A variance decomposition or repeated model instances would be needed to support the 'relative depth governs' claim.
  4. [Sec. 4.1 and Abstract] The conclusion that 'training paradigm is the strongest determinant of orientation selectivity' rests on qualitative inspection of RSS/ORS curves for one representative model per paradigm and a within-contrastive control in which data, loss, and patch size are not independently varied (SigLIP2 changes all three together). No quantitative attribution is provided (e.g., variance explained across paradigms vs data vs architecture), so the strongest-determinant statement is not established. The authors should either soften the claim or add a quantitative analysis.
minor comments (5)
  1. [Abstract] The phrase 'thede facto' should be 'the de facto'.
  2. [Section 3.2] The MLP-neuron basis is used only for the training-trajectory analysis; this choice should be motivated earlier, as it limits comparability of the training-dynamics result with the paradigm and scale results.
  3. [Fig. 2] The legend for hollow circles should also state how many models or blocks are included and provide a citation for the V1-like bandwidth range (15°-40°), since the shaded band is used as a reference throughout.
  4. [Section 4.3] The sentence 'We have not validated that this preserves downstream performance' is an important caveat and should appear in the main conclusions as well, since the abstract lists training-dynamics insights as a contribution.
  5. [Section 5 vs App. A.5] The main text says RSS is 'scored against 9 other a priori candidates' while A.5 describes 18 predictors; reconcile the counts, and clarify whether the fine-tuning sweep in Fig. 5 uses scale-matched L-tier models or a family/scale-spanning set, since DINOv3-B, DINOv3-S, MAE-B, and DeiT3-S are not L-tier.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; RSS-to-fine-tuning-depth link is an empirical correlation between independently measured curves.

full rationale

The derivation is self-contained. RSS, ORS, and HWHM are defined directly from responses to synthetic gratings (Eqs. 1-6), and the LoRA unfreeze-depth claim compares the RSS curve measured on a frozen backbone against pooled fine-tuning performance measured independently in a 9-depth x 9-task x 5-seed sweep (Sec. 5, App. A.4). No parameter of RSS is fitted to downstream performance; the permutation null in Eq. 5 and the cross-validated decoding in Eq. 10 use only grating data. The paper's own caveat that 'no single statistic here carries a formal significance test' (App. A.5) is a statistical robustness limitation, not a circularity: the 18-predictor ranking and the six-backbone peak coincidences are empirical claims that could have failed. The synthetic-grating proxy and the single-backbone head-to-head comparison are validity and generalization concerns, not circularity, and there are no load-bearing self-citations. Hence score 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The metrics themselves are definitions, not derivations, so the ledger is dominated by experimental choices: stimulus parameters, SAE configuration, and pooling strategy. The main unstated assumptions are that gratings reveal task-relevant orientation code and that mean-pooled activations preserve that code.

free parameters (3)
  • Gaussian envelope width sigma = not reported
    Chosen by hand in Eq. (1); affects the spatial extent of gratings, and its value is not reported.
  • SAE dictionary size and sparsity lambda = not reported
    Held constant across models to make ORS comparable, but actual values omitted in A.1.
  • Number of shuffles B in ORS null = not reported
    Eq. (5) defines the null using B shuffles but the text never gives B, making the p-values non-reproducible.
assumptions (4)
  • standard math Representational similarity analysis (RSA) with cosine dissimilarity and circular orientation distance captures the degree of orientation encoding.
    Eq. (3), from Kriegeskorte et al. 2008; the specific circular-distance mapping is an assumption about how orientation geometry manifests.
  • domain assumption Synthetic sinusoidal gratings with 36 orientations, 4 phases, and 5 spatial frequencies are a sufficient probe for orientation selectivity in ViTs.
    Section 3.1; the paper itself lists this as a limitation in Section 6.
  • domain assumption Sparse autoencoder features trained on 256 Imagenette images are monosemantic and comparable across models.
    Appendix A.1; no reconstruction fidelity or feature interpretability checks are reported, yet ORS comparisons assume unit comparability.
  • domain assumption Pooling over patch tokens does not destroy orientation information.
    Section 3.2; population responses are means over patch tokens, which discards spatial layout.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IRIS: A Visual Cortex-Inspired Framework for Analyzing Orientation Selectivity in Vision Transformers." pith.science (2026). https://pith.science/paper/UYDPTH7M

@misc{pith2026260805122,
  author       = {Pith},
  title        = {Pith review of: IRIS: A Visual Cortex-Inspired Framework for Analyzing Orientation Selectivity in Vision Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UYDPTH7M}},
  note         = {Machine review of arXiv:2608.05122}
}
read the original abstract

Vision transformers (ViTs) have become the de facto standard for image encoding across many perception tasks. Despite their empirical success, it remains mechanistically unclear how they encode low-level features, given their lack of inductive biases: ViTs process information globally rather than relying on local structure. Biological visual systems, in contrast, build low-level features, such as orientation selectivity in the primary visual cortex, by combining information from small, localized regions of the visual field. These features are general-purpose representations, shared and required across multiple specialized neural pathways, unlike higher-level, task-specific semantic features. This raises the question if such biologically-grounded features arise in ViTs. In this work, we systematically study how orientation selectivity emerges in ViTs by introducing a suite of neuroscience-inspired metrics: representational similarity score (RSS), orientation recruitment score (ORS), and orientation tuning bandwidth to quantify how orientation is encoded in representational geometry and as a function of model depth. Through extensive analysis, we find that: (1) the training paradigm is the strongest determinant of orientation selectivity, with models sharing an objective, peaking at comparable relative depths regardless of scale (2) many units are orientation-selective early in training, with early-to-middle layers recruiting more such units over time, while deeper layers lose selectivity and broaden their tuning toward semantic encoding and (3) our metrics offer a mechanistic heuristic for how many layers to unfreeze for best downstream generalization. Our framework presents a way to track biologically-grounded features during ViT training, probes how desired properties are encoded in transformer representations, and builds a systematic understanding of how ViTs generalize across tasks.

Figures

Figures reproduced from arXiv: 2608.05122 by the authors.

Figure 1
Figure 1. Probing pipeline. Oriented gratings and natural images pass through each frozen backbone, with [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Orientation selectivity across training paradigms. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. We plot RSS vs. relative depth, for each family, across tiers. Tiers overlay almost exactly within [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Orientation selectivity over OpenCLIP ViT-H/14’s training trajectory. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Pooled downstream performance (solid) against a-priori RSS on the frozen backbone (dashed), [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: We plot mean min–max normalized performance [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Protocol schematic. Top: the frozen backbone is probed once with the RSA battery at every block, giving an a-priori RSS-vs-block curve, no fine-tuning required. Bottom: for each block k, the same backbone is adapted with LoRA on blocks [k :] and fine-tuned across the 9…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 5 canonical work pages

  1. [6]

    Chen, Zhenyu Li, Guang Shi, Jiashi Feng, and Bingyi Kang

    Haotong Lin, Sili Chen, Jun Hao Liew, Donny Y. Chen, Zhenyu Li, Guang Shi, Jiashi Feng, and Bingyi Kang. Depth anything 3: Recovering the visual space from any views.arXiv preprint arXiv:2511.10647,

    arXiv

  2. [7]

    8), with the decoder rows constrained to unit norm so the sparsity penalty cannot be evaded by rescaling

    and is trained to minimize reconstruction error with anℓ 1 penalty on the code (Eq. 8), with the decoder rows constrained to unit norm so the sparsity penalty cannot be evaded by rescaling. The SAE is fit on the model’s activations for a battery of 256 natural images from Imagenette, a 10-class subset of ImageNet-1k (Deng et al., 2009), taken at the probe...

    work page arXiv 2009

  3. [11]

    Griffiths

    Shikhar Tuli, Ishita Dasgupta, Erin Grant, and Thomas L. Griffiths. Are convolutional neural networks or transformers more like human vision?arXiv preprint arXiv:2105.07197,

    arXiv

  4. [13]

    arXiv:2502.20578. Amir R. Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskon- omy: Disentangling task transfer learning. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),

    arXiv

  5. [14]

    A large- scale study of representation learning with the visual task adaptation benchmark.arXiv preprint arXiv:1910.04867,

    Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large- scale study of representation learning with the visual task a...

    arXiv 1910

  6. [16]

    DINOv3 generalizes best (0.958), followed by DINOv2 (0.670) and SigLIP2 (0.521); MAE (0.405) and DeiT3 (0.393) are similarly mid-low, and AIMv2 generalizes worst (0.138)

    for the six L-tier frozen backbones, pooled across benchmark tasks (0 = worst, 1 = best backbone per task). DINOv3 generalizes best (0.958), followed by DINOv2 (0.670) and SigLIP2 (0.521); MAE (0.405) and DeiT3 (0.393) are similarly mid-low, and AIMv2 generalizes worst (0.138). APRIORI PREDICTION OF BEST FINE-TUNING DEPTH k USING RSS ENCODER STATE PROBING...

    work page 2022

  7. [1962]

    Prisma: An open source toolkit for mechanistic interpretability in vision and video.arXiv preprint arXiv:2504.19475,

    Sonia Joseph, Praneet Suresh, Lorenz Hufe, Edward Stevinson, Robert Graham, Yash Vadi, Danilo Bzdok, Sebastian Lapuschkin, Lee Sharkey, and Blake Aaron Richards. Prisma: An open source toolkit for mechanistic interpretability in vision and video.arXiv preprint arXiv:2504.19475,

    arXiv

  8. [1993]

    13-12-05251.1993

    doi: 10.1523/JNEUROSCI. 13-12-05251.1993. Taco Cohen and Max Welling. Group equivariant convolutional networks. InInternational Conference on Machine Learning, pages 2990–2999,

    arXiv 1993

Show all 16 references
  1. [2001]

    Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson

    doi: 10.1038/35082568. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? InNeurIPS,

    doi

  2. [2002]

    Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J

    doi: 10.1523/ JNEUROSCI.22-13-05639.2002. Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, Franziska Geiger, Daniel L. K. Yamins, and James J. DiCarlo. Brain-score: W...

    2002

  3. [2009]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer

    doi: 10.1109/CVPR.2009.5206848. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. InNeurIPS,

    2009

  4. [2018]

    Oriane Sim´ eoni, Huy V

    doi: 10.1101/407007. Oriane Sim´ eoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Micha¨ el Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timoth´ ee Darcet, Th´ eo M...

    doi

  5. [2021]

    Steering clip’s vision transformer with sparse autoencoders.arXiv preprint arXiv:2504.08729,

    Sonia Rao et al. Steering clip’s vision transformer with sparse autoencoders.arXiv preprint arXiv:2504.08729,

    arXiv

  6. [2023]

    Horace B

    doi: 10.1016/j.neunet.2023.04.032. Horace B. Barlow. Possible principles underlying the transformation of sensory messages. In Walter A. Rosenblith, editor,Sensory Communication, pages 217–234. MIT Press, Cambridge, MA,

    2023 doi

  7. [2024]

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey

    doi: 10.1038/s41467-024-53147-y. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models.arXiv preprint arXiv:2309.08600,

    arXiv

  8. [2025]

    Vo, Camille Couprie, Patrick Labatut, Piotr Bojanowski, Valentin Wyart, and Jean-R´ emi King

    Jos´ ephine Raugel, Marc Szafraniec, Huy V. Vo, Camille Couprie, Patrick Labatut, Piotr Bojanowski, Valentin Wyart, and Jean-R´ emi King. Disentangling the factors of convergence between brains and computer vision models.arXiv preprint arXiv:2508.18226,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.