Representation biases: will we achieve complete understanding by analyzing representations?

Andrew Kyle Lampinen; Katherine Hermann; Stephanie C. Y. Chan; Yuxuan Li

Representation biases: will we achieve complete understanding by analyzing representations?

T0 review · 0 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Learned representation bias can make neural analyses rank a system closer to one that does not compute a feature than to one that does.

desk verdict A clear, honest perspective on how representation biases distort common neural analyses, but the neuroscience conclusion rests on an untested transfer from toy MLPs to biological brains. read the letter →

arxiv 2507.22216 v2 pith:IILQ46VW submitted 2025-07-29 q-bio.NC cs.LG

Andrew Kyle Lampinen , Stephanie C. Y. Chan , Yuxuan Li , Katherine Hermann This is my paper

classification q-bio.NCcs.LG

keywords representationanalysisbiasrepresentationalsimilarityprincipalcomponentneuralencodingcomputationalneurosciencedeeplearninghomomorphicencryption

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a perspective on a hard limit to representational analysis: the internal representations learned by neural networks are systematically lopsided, with simple, prevalent, or early-learned features accounting for far more variance than equally computed but harder features. The paper shows that because PCA, regression, and RSA all lean on variance, they inherit this lopsidedness and can produce strongly biased inferences about what a system computes—for example, a model that computes both an easy and a hard feature can look nearly identical to a model that computes only the easy one. The authors argue this bias is a general property of feature learning and that no simple change of metric fully fixes it, because most comparison metrics still depend on variance or correlation. The practical stake is that complete understanding of a brain or model through representational analysis alone may be out of reach without prior knowledge of which features matter; the paper therefore recommends pairing representation studies with full behavioral characterization and causal intervention.

What carries the argument

The mechanism is the representation-variance bias: the fraction of a layer's activity variance that can be attributed to a feature, together with the linking assumption that variance equals computational significance. Simpler features are learned first and have essentially one natural linear representation, so their variance concentrates; nonlinear features admit several non-linearly-equivalent representations whose inconsistent dimensions dilute one another. The paper's demonstrations rely on controlled multi-layer perceptron experiments with easy linear versus hard nonlinear (XOR-like) features, and on homomorphic encryption as a limiting case where arbitrary computation runs over intentionally opaque representations.

What would settle it

Run the paper's easy-versus-hard training setup with a sweep of seeds and training orders and measure variance explained in the penultimate layer for each feature while both are perfectly classified; the paper's claim predicts the easy feature dominates by roughly an order of magnitude. A trained model in which the hard feature explains comparable variance—or in which RSA ranks hard-only models as the closest matches to each other—would show the bias is not the systematic pattern described.

Watch

Extended reading notes

Core claim

The central claim is that patterns of representation are not a clean mirror of computation: variance in a neural layer is heavily concentrated on certain features, and the standard tools used to read representations therefore read those features and miss others. In the paper's illustrative experiments, an MLP (a simple feedforward neural network) is trained to classify an easy linear feature and a hard nonlinear feature; it generalizes perfectly on both, yet the easy feature explains more than 50 percent of the variance in the penultimate layer while the hard feature explains roughly 5 percent. The top principal components cluster by the easy feature, RSA makes a multi-task model look very similar to an easy-only model and less similar to models that compute the same hard task, and the units most driven by the hard feature still carry stronger easy-feature signals. The authors stress that these are biases in the inferences drawn, not necessarily flaws in the representations, and they use homomorphic encryption to show how far computation and superficial representation can diverge in the worst case.

Load-bearing premise

The weight-bearing premise is that real brains show the same kind of representation bias as trained artificial networks; all concrete demonstrations come from small synthetic models, and the paper itself leaves the transfer to biology as an open question.

Editorial extensions

If this is right

Principal component analysis of a layer's activity will foreground easy or prevalent features and can miss hard features entirely, so top components are not a safe inventory of a system's computations.
Representational similarity analysis can rank a multi-task model as closer to a model that only computes the easy feature than to a model that computes the same hard feature, so similarity scores do not reliably track functional overlap.
Regression-based encoding analyses will attribute most explained variance to simple or prevalent features even when harder features are equally relevant to behavior.
Between-subject noise ceilings for model–brain comparison can overstate the achievable prediction because two systems can compute the same feature in non-linearly-equivalent representation patterns.
Attaining complete understanding of a system will require pairing representational analysis with thorough behavioral characterization and causal tests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

By extension, interpretability methods outside neuroscience that select representational directions by variance or linear decodability—circuit discovery, linear probes, activation patching—may inherit the same blind spots for features that are computed nonlinearly.
A testable prediction follows from the paper's learning-order account: training curricula that interleave or pretraining that schedules hard features first should shrink the variance gap; existing training logs could be re-checked for this effect.
The homomorphic-encryption case implies that the absence of a linearly decodable compositional representation is not evidence against compositional computation, and the presence of one is not proof that it is the computation actually performed.
If cross-model and brain–model convergence is driven largely by shared easy features, then representational-alignment benchmarks may overstate agreement in computations; adding behavioral task-overlap measures alongside similarity scores would test this.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, this is a perspective piece, not a new empirical study. The core demonstrations of representation bias come from the authors' own TMLR paper; what's added here is the neuroscience-facing interpretation: how these biases distort PCA, regression, RSA, and noise-ceiling comparisons. Second, the argument is honest about its main limitation: whether biological brains show the same variance-complexity bias is left as an open question, and the paper says so explicitly.

What it does well, the RSA and noise-ceiling implications are the strongest new material. The point that between-subject noise ceilings can overstate model quality when two systems represent the same feature in non-equivalent ways is genuinely useful, and the Fig. 3 demonstration makes it concrete. The homomorphic encryption example is a good conceptual worst case: it proves that computation can proceed on representations deliberately engineered to thwart decodability, which sharpens the claim that representational alignment is not necessary for computational alignment. The writing is clear, and the hedging is appropriate rather than evasive.

Where it's soft, the central claim about neuroscience rests on that untested transfer premise. All demonstrations are small MLPs on binary features. If real neural populations do not exhibit analogous variance biases, the streetlight-effect warning is a logical possibility rather than an established risk. The paper flags this, which is to its credit, but it means the title's 'we' is doing a lot of work. The PCA point is close to definitional, since the bias is defined in terms of variance; the RSA and noise-ceiling consequences are more informative. The paper also leans heavily on the authors' own prior work, which is reasonable given that the underlying results are theirs, but a reader new to the area will need to check the TMLR paper for the actual evidence.

Overall, this is a fair, useful perspective that reframes how representational analyses should be interpreted. It doesn't resolve the long-open question of when representations track computations, but it gives the field a sharper caveat. A reader working on model-brain alignment will get real value from the noise-ceiling and RSA discussion. It deserves a serious referee: a good review would push the authors to be more explicit about the conditions under which biological systems would show the same bias, and perhaps to include evidence from more naturalistic models. My recommendation is to send it to peer review.

Referee Report

0 major / 8 minor

Summary. This perspective argues that learned feature representations in deep networks are systematically biased toward simple or prevalent features, and that this biases the conclusions drawn from standard representational analyses (PCA, linear regression, RSA). The authors review controlled experiments from Lampinen et al. (2024) on small MLPs that classify both a linear 'easy' feature and a nonlinear 'hard' feature; although the model generalizes perfectly on both, variance explained in the penultimate representation is an order of magnitude higher for the easy feature. They show how this variance bias leads PCA to cluster by the easy feature, makes RSA similarity matrices rank models in a way that disagrees with functional similarity, and can mislead between-subject 'noise ceiling' interpretations. A conceptual homomorphic-encryption example is used to argue that systematic computation does not require decodable representations, and the implications for brain-model comparisons, single-unit analyses, and causal interventions in neuroscience are discussed. The paper concludes that representational analysis remains useful but that complete understanding of a system from representations alone is likely more difficult than commonly assumed.

Significance. The paper is a timely and clearly written perspective that connects the machine-learning phenomenon of simplicity bias to methodological practice in systems neuroscience and interpretability. Its main value is in making the 'linking assumption' explicit and in showing, with concrete synthetic examples, that a feature's variance share is not a reliable index of its computational role. The homomorphic-encryption case supplies an independent, non-neural demonstration that representation and computation can in principle diverge. The discussion of noise ceilings is a particularly useful conceptual contribution: it distinguishes genuine measurement noise from systematic but non-shared representational formats across individuals. The authors are careful to hedge their claims, explicitly acknowledging that whether biological brains exhibit such biases is an open empirical question. The paper does not present new neural data or statistical tests, and its quantitative demonstrations are drawn from prior simulations; for a perspective, this is appropriate.

minor comments (8)

[Section 1, page 3] The word 'commmon' in the sentence 'how they impact commmon representational analyses' should be corrected to 'common'.
[Section 4] The word 'contatenatively' in 'without requiring representations that are straightforwardly (e.g. contatenatively) compositional' should be corrected to 'concatenatively'.
[Section 5] The phrase 'It natural to ask' should be 'It is natural to ask' in the first paragraph of 'What are the potential solutions?'.
[Abstract] The citation 'Lampinen, 2024' in the abstract should be 'Lampinen et al. (2024)' for consistency with the reference list.
[Section 2, Fig. 2 caption] The caption reports 'over 50%' and 'around 5%' of variance explained but does not specify how these values were computed (e.g., linear regression R²) or how many model instances/seeds were used; please add this detail or point to the exact method in Lampinen et al. (2024).
[Section 2, Fig. 3 caption] The caption should define 'multiple easy' and 'multiple hard' tasks explicitly and state whether the displayed similarity matrices are averaged over the five trained models or show a representative run; clarifying this will help readers assess the reproducibility of the qualitative result.
[Section 2.1] The claim that accounting for multiple representation patterns 'explains part of the gap' between linear and nonlinear features is only described verbally; a pointer to the specific analysis in Lampinen et al. (2024) would strengthen the reader's ability to evaluate the mechanism.
[Section 3, noise-ceiling paragraph] The Tuckute et al. (2025) example is cited to suggest that unexplained variance in between-subject predictions may reflect incompatible representational formats, but the logical link is not made explicit; one clarifying sentence about why shared dimensions imply the residual is computational rather than noise would help.

Circularity Check

1 steps flagged · score 2.0 of 10

The PCA demonstration is definitionally entailed by the variance-based definition of bias, but the broader perspective is not circular and retains independent content.

self definitional [Section 2, paragraph following Fig. 2]
"Because of this variance bias, when analyzing this model's representations, many analyses tend to be biased towards the easy feature. For example, Principal Components Analysis (PCA) identifies precisely the dimensions in the representations that carry the most variance. Thus, the first several PCs show clear clusters according to the easy feature, and little organization according to the more difficult feature."

The 'variance bias' was defined just before in the same section as a difference in the variance explained in the representations (Fig. 2 caption: 'representations can be substantially biased towards certain features, in the sense that those features carry much more variance in the representations'). PCA is then described as identifying 'precisely the dimensions in the representations that carry the most variance.' The PCA outcome is therefore entailed by the bias definition plus the PCA definition: it is an illustration of the definition rather than an independent empirical discovery. This definitional step is real but local; the RSA dissociations, noise-ceiling discussion, and homomorphic-encryption case study do not reduce to the variance definition.

full rationale

The paper is a perspective that synthesizes prior empirical results, chiefly Lampinen et al. (2024), and draws implications for representational analyses and neuroscience. It fits no parameters and makes no numerical prediction that is then validated against held-out data. The one definitionally forced step is the PCA consequence, because the bias is defined by variance and PCA, by definition, selects maximal-variance directions. That step is explicitly presented as an illustration of the bias, not as a new derivation, so it is a minor circularity rather than a load-bearing failure. The broader claim that representation biases can distort RSA, regression, noise-ceiling interpretations, and model-to-brain comparisons is supported by additional cited work (e.g., Friedman et al., 2023; Braun et al., 2025) and by the independent homomorphic-encryption case study, which does not depend on the authors' simulations. The neuroscience extension is also explicitly hedged as an open question ('Do natural systems similarly produce representations that are biased by these (or other) factors?'), so the central argument is conditional rather than circularly forced. Overall, the paper is largely self-contained as a perspective, with only a minor definitional step, yielding a score of 2.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No free parameters are fit in this paper; it contains no new derivation and reuses controlled experiments from prior work. No new entities, forces, or mediators are introduced. 'Representation bias' names an observed phenomenon rather than a new theoretical object.

assumptions (5)

domain assumption The empirical findings of Lampinen et al. (2024) on feature representation biases are valid and representative.
Section 2 is built on this self-authored prior study; the paper presents no independent replication. This is an upstream result the reader must accept.
domain assumption Representations can be pragmatically identified by reliable decodability and causal efficacy.
Section 1.1 adopts this framing from prior philosophy of neuroscience (Cao, 2022b; Poldrack, 2021; Baker et al., 2022). The paper's examples are interpreted under this definition.
domain assumption Biological neural systems are analogous enough to trained deep networks for the demonstrated biases to transfer.
The neuroscience implications in Section 3 depend on this transfer, which the authors explicitly leave as an open question rather than a demonstrated fact.
domain assumption Common analytic methods (PCA, regression, RSA, neural predictivity) treat variance explained as a proxy for feature importance.
The critique of biased inferences targets this linking assumption, illustrated in Figure 1. The paper must be right that this assumption underlies common practice for its argument to bite.
standard math Fully homomorphic encryption schemes can evaluate arbitrary algebraic circuits over encrypted data.
Section 4 uses this cryptographic theorem, cited to Gentry (2009) and Van Dijk et al. (2010), as the basis of the dissociation case study.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Representation biases: will we achieve complete understanding by analyzing representations?." pith.science (2026). https://pith.science/paper/IILQ46VW

@misc{pith2026250722216,
  author       = {Pith},
  title        = {Pith review of: Representation biases: will we achieve complete understanding by analyzing representations?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IILQ46VW}},
  note         = {Machine review of arXiv:2507.22216}
}

read the original abstract

A common approach in neuroscience is to study neural representations as a means to understand a system -- increasingly, by relating the neural representations to the internal representations learned by computational models. However, a recent work in machine learning (Lampinen, 2024) shows that learned feature representations may be biased to over-represent certain features, and represent others more weakly and less-consistently. For example, simple (linear) features may be more strongly and more consistently represented than complex (highly nonlinear) features. These biases could pose challenges for achieving full understanding of a system through representational analysis. In this perspective, we illustrate these challenges -- showing how feature representation biases can lead to strongly biased inferences from common analyses like PCA, regression, and RSA. We also present homomorphic encryption as a simple case study of the potential for strong dissociation between patterns of representation and computation. We discuss the implications of these results for representational comparisons between systems, and for neuroscience more generally.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safety
cs.AI 2026-01 unverdicted novelty 6.0 of 10

The paper formalizes homogenization in LLMs as a loss of deviance and core entropy, and proposes xeno-reproduction—a structure-aware diversity-pursuit objective—with a proof that diversity and fairness trade off.

Reference graph

Works this paper leans on

12 extracted references · 6 canonical work pages · cited by 1 Pith paper

[4]

Feather, M

J. Feather, M. Khosla, N. Murty, and A. Nayebi. Brain-model evaluations need the neuroai turing test. arXiv preprint arXiv:2502.16238,

arXiv
[6]

12 Representation biases: will we achieve complete understanding by analyzing representations? Z. He, J. Achterberg, K. Collins, K. Nejad, D. Akarca, Y. Yang, W. Gurnee, I. Sucholutsky, Y. Tang, R. Ianov, et al. Multilevel interpretability of artificial neural networks: leveraging framework and methods from neuroscience.arXiv preprint arXiv:2408.12664,

arXiv
[8]

M. Huh, H. Mobahi, R. Zhang, B. Cheung, P. Agrawal, and P. Isola. The low-rank simplicity bias in deep networks.arXiv preprint arXiv:2103.10427,

arXiv
[11]

Tuckute, E

G. Tuckute, E. J. Lee, Y. Ou, E. Fedorenko, and K. Kay. A two-dimensional space of linguistic representations shared across individuals.bioRxiv, pages 2025–05,

work page 2025
[12]

Van Dijk, C

M. Van Dijk, C. Gentry, S. Halevi, and V. Vaikuntanathan. Fully homomorphic encryption over the integers. InAdvances in Cryptology–EUROCRYPT 2010: 29th Annual International Conference on the Theory and Applications of Cryptographic Techniques, French Riviera, May 30–June 3,

work page 2010
[1991]

A. Mueller. Missed causes and ambiguous effects: Counterfactuals pose challenges for interpreting neural networks. InICML 2024 Workshop on Mechanistic Interpretability,

work page 2024
[1998]

Conwell, J

C. Conwell, J. S. Prince, K. N. Kay, G. A. Alvarez, and T. Konkle. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines?BioRxiv, pages 2022–03,

work page 2022
[2012]

Sucholutsky, L

14 Representation biases: will we achieve complete understanding by analyzing representations? I. Sucholutsky, L. Muttenthaler, A. Weller, A. Peng, A. Bobu, B. Kim, B. C. Love, E. Grant, J. Achter- berg, J. B. Tenenbaum, et al. Getting aligned on representational alignment.arXiv preprint arXiv:2310.13018,

arXiv

Show all 12 references

[2020]

K. L. Hermann, H. Mobahi, T. Fel, and M. C. Mozer. On the foundations of shortcut learning.arXiv preprint arXiv:2310.16228,

arXiv
[2023]

Dujmovic, J

M. Dujmovic, J. Bowers, F. Adolfi, and G. Malhotra. Inferring DNN-brain alignment using representa- tional similarity analyses can be problematic. InICLR 2024 Workshop on Representational Alignment,

2024
[2024]

Carvalho and A

W. Carvalho and A. Lampinen. Naturalistic computational cognitive science: Towards generalizable models and theories that capture the full range of natural behavior.arXiv preprint arXiv:2502.20349,

arXiv
[2025]

T. Fel, L. Bethune, A. K. Lampinen, T. Serre, and K. Hermann. Understanding visual feature reliance through the lens of complexity.arXiv preprint arXiv:2407.06076,

arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [4]

Feather, M

J. Feather, M. Khosla, N. Murty, and A. Nayebi. Brain-model evaluations need the neuroai turing test. arXiv preprint arXiv:2502.16238,

arXiv

[2] [6]

12 Representation biases: will we achieve complete understanding by analyzing representations? Z. He, J. Achterberg, K. Collins, K. Nejad, D. Akarca, Y. Yang, W. Gurnee, I. Sucholutsky, Y. Tang, R. Ianov, et al. Multilevel interpretability of artificial neural networks: leveraging framework and methods from neuroscience.arXiv preprint arXiv:2408.12664,

arXiv

[3] [8]

M. Huh, H. Mobahi, R. Zhang, B. Cheung, P. Agrawal, and P. Isola. The low-rank simplicity bias in deep networks.arXiv preprint arXiv:2103.10427,

arXiv

[4] [11]

Tuckute, E

G. Tuckute, E. J. Lee, Y. Ou, E. Fedorenko, and K. Kay. A two-dimensional space of linguistic representations shared across individuals.bioRxiv, pages 2025–05,

work page 2025

[5] [12]

Van Dijk, C

M. Van Dijk, C. Gentry, S. Halevi, and V. Vaikuntanathan. Fully homomorphic encryption over the integers. InAdvances in Cryptology–EUROCRYPT 2010: 29th Annual International Conference on the Theory and Applications of Cryptographic Techniques, French Riviera, May 30–June 3,

work page 2010

[6] [1991]

A. Mueller. Missed causes and ambiguous effects: Counterfactuals pose challenges for interpreting neural networks. InICML 2024 Workshop on Mechanistic Interpretability,

work page 2024

[7] [1998]

Conwell, J

C. Conwell, J. S. Prince, K. N. Kay, G. A. Alvarez, and T. Konkle. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines?BioRxiv, pages 2022–03,

work page 2022

[8] [2012]

Sucholutsky, L

14 Representation biases: will we achieve complete understanding by analyzing representations? I. Sucholutsky, L. Muttenthaler, A. Weller, A. Peng, A. Bobu, B. Kim, B. C. Love, E. Grant, J. Achter- berg, J. B. Tenenbaum, et al. Getting aligned on representational alignment.arXiv preprint arXiv:2310.13018,

arXiv