{"id":"d029ccf1-9153-4903-a060-c6f68b07cae2","arxiv_id":"2507.22216","paper_version":2,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Feature representation biases in trained models can distort PCA, regression, RSA, and model-brain comparisons, so representational analyses may not reveal all of a system's computations.","lead":"This perspective argues that learned feature representations in AI models are systematically biased toward simple features, and that common neuroscience analyses such as PCA, regression, and RSA inherit these biases. The authors use simulations and a homomorphic-encryption example to ask whether representational analysis alone can ever yield complete understanding of a system.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The neuroscience relevance of the paper rests on an untested transfer premise: real neural representations may not show the same variance-complexity bias as the toy MLPs. The paper itself flags this as an open question, so the perspective is sound but conditional.","rationale":"The reader's weakest assumption is exactly the transfer from trained MLPs to biological neural systems, and Section 3 explicitly acknowledges this as an open question. I agree that this is the most load-bearing premise for the neuroscience-facing claims. However, the paper is a perspective, not an empirical study, and it consistently hedges the neuroscience implications with 'could', 'raises questions', and the explicit open question. The central demonstration that variance-based analyses are biased in the controlled MLP setting is internally coherent and supported by the cited prior work; the homomorphic encryption example is a logical counterexample to the sufficiency of representational analysis, not an empirical claim about brains. Because the argument is appropriately scoped and the limitation is stated in the manuscript, the reader's ACCEPT verdict remains appropriate. The concern does not require a change in verdict; it would become decisive only if the paper claimed to have established that biological representations are biased, which it does not. The proposed reanalysis of neural data would directly test the transfer premise and would either strengthen or bound the neuroscience implications.","tokens_in":11816,"tokens_out":7035,"duration_ms":97345,"concrete_test":"Reanalyze a publicly available neural dataset with a behaviorally matched feature pair, one linearly decodable and one nonlinearly decodable (e.g., in primate IT or V4: object category vs. a conjunction of two stimulus features), by computing: (1) R^2 of each feature regressed onto the population response; (2) fraction of variance in the top principal components explained by each feature; and (3) between-animal RSA similarity for each feature. If the nonlinear feature explains comparable variance and appears comparably in RSA despite matched behavioral relevance, the transfer premise fails; if the nonlinear feature explains substantially less variance and is underrepresented in RSA, the paper's concern lands in real neural data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is the claim that representation biases in trained models can systematically distort common representational analyses (PCA, regression, RSA), and the abstract explicitly extends this to neuroscience. The direct demonstrations all come from small synthetic MLPs trained on binary features (Lampinen et al., 2024). For the neuroscience conclusion to bite, biological neural populations must exhibit analogous biases: computationally matched features should differ by orders of magnitude in variance explained, with simpler features dominating. Section 3 states this premise as an open question: 'Do natural systems similarly produce representations that are biased by these (or other) factors? If so...' This is not an internal inconsistency, but it is load-bearing: if real brains do not show this variance-complexity asymmetry, then the demonstrated distortions may not transfer, and the 'streetlight effect' warning for neuroscience would rest on logical possibility rather than empirical evidence. The homomorphic encryption case establishes a strong dissociation in principle, but it is a deliberately engineered worst case, not evidence about natural systems. Thus the paper's strongest claim about neuroscience is a well-hedged hypothesis, not a demonstrated result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This perspective argues that learned feature representations in deep networks are systematically biased toward simple or prevalent features, and that this biases the conclusions drawn from standard representational analyses (PCA, linear regression, RSA). The authors review controlled experiments from Lampinen et al. (2024) on small MLPs that classify both a linear 'easy' feature and a nonlinear 'hard' feature; although the model generalizes perfectly on both, variance explained in the penultimate representation is an order of magnitude higher for the easy feature. They show how this variance bias leads PCA to cluster by the easy feature, makes RSA similarity matrices rank models in a way that disagrees with functional similarity, and can mislead between-subject 'noise ceiling' interpretations. A conceptual homomorphic-encryption example is used to argue that systematic computation does not require decodable representations, and the implications for brain-model comparisons, single-unit analyses, and causal interventions in neuroscience are discussed. The paper concludes that representational analysis remains useful but that complete understanding of a system from representations alone is likely more difficult than commonly assumed.","tokens_in":12018,"tokens_out":9529,"duration_ms":107576,"significance":"The paper is a timely and clearly written perspective that connects the machine-learning phenomenon of simplicity bias to methodological practice in systems neuroscience and interpretability. Its main value is in making the 'linking assumption' explicit and in showing, with concrete synthetic examples, that a feature's variance share is not a reliable index of its computational role. The homomorphic-encryption case supplies an independent, non-neural demonstration that representation and computation can in principle diverge. The discussion of noise ceilings is a particularly useful conceptual contribution: it distinguishes genuine measurement noise from systematic but non-shared representational formats across individuals. The authors are careful to hedge their claims, explicitly acknowledging that whether biological brains exhibit such biases is an open empirical question. The paper does not present new neural data or statistical tests, and its quantitative demonstrations are drawn from prior simulations; for a perspective, this is appropriate.","major_comments":[],"minor_comments":[{"comment":"The word 'commmon' in the sentence 'how they impact commmon representational analyses' should be corrected to 'common'.","section":"Section 1, page 3"},{"comment":"The word 'contatenatively' in 'without requiring representations that are straightforwardly (e.g. contatenatively) compositional' should be corrected to 'concatenatively'.","section":"Section 4"},{"comment":"The phrase 'It natural to ask' should be 'It is natural to ask' in the first paragraph of 'What are the potential solutions?'.","section":"Section 5"},{"comment":"The citation 'Lampinen, 2024' in the abstract should be 'Lampinen et al. (2024)' for consistency with the reference list.","section":"Abstract"},{"comment":"The caption reports 'over 50%' and 'around 5%' of variance explained but does not specify how these values were computed (e.g., linear regression R²) or how many model instances/seeds were used; please add this detail or point to the exact method in Lampinen et al. (2024).","section":"Section 2, Fig. 2 caption"},{"comment":"The caption should define 'multiple easy' and 'multiple hard' tasks explicitly and state whether the displayed similarity matrices are averaged over the five trained models or show a representative run; clarifying this will help readers assess the reproducibility of the qualitative result.","section":"Section 2, Fig. 3 caption"},{"comment":"The claim that accounting for multiple representation patterns 'explains part of the gap' between linear and nonlinear features is only described verbally; a pointer to the specific analysis in Lampinen et al. (2024) would strengthen the reader's ability to evaluate the mechanism.","section":"Section 2.1"},{"comment":"The Tuckute et al. (2025) example is cited to suggest that unexplained variance in between-subject predictions may reflect incompatible representational formats, but the logical link is not made explicit; one clarifying sentence about why shared dimensions imply the residual is computational rather than noise would help.","section":"Section 3, noise-ceiling paragraph"}],"recommendation":"minor_revision","confidential_remarks":"The manuscript is a perspective built largely on the authors' own previous TMLR paper; it is transparent about this. The homomorphic-encryption and Tuckute et al. references provide external anchoring. I do not see a scope problem, though the fit with q-bio.NC is more about methods than new neuroscience results; this seems appropriate for a perspective."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a perspective piece, not a new empirical study. The core demonstrations of representation bias come from the authors' own TMLR paper; what's added here is the neuroscience-facing interpretation: how these biases distort PCA, regression, RSA, and noise-ceiling comparisons. Second, the argument is honest about its main limitation: whether biological brains show the same variance-complexity bias is left as an open question, and the paper says so explicitly.\n\nWhat it does well, the RSA and noise-ceiling implications are the strongest new material. The point that between-subject noise ceilings can overstate model quality when two systems represent the same feature in non-equivalent ways is genuinely useful, and the Fig. 3 demonstration makes it concrete. The homomorphic encryption example is a good conceptual worst case: it proves that computation can proceed on representations deliberately engineered to thwart decodability, which sharpens the claim that representational alignment is not necessary for computational alignment. The writing is clear, and the hedging is appropriate rather than evasive.\n\nWhere it's soft, the central claim about neuroscience rests on that untested transfer premise. All demonstrations are small MLPs on binary features. If real neural populations do not exhibit analogous variance biases, the streetlight-effect warning is a logical possibility rather than an established risk. The paper flags this, which is to its credit, but it means the title's 'we' is doing a lot of work. The PCA point is close to definitional, since the bias is defined in terms of variance; the RSA and noise-ceiling consequences are more informative. The paper also leans heavily on the authors' own prior work, which is reasonable given that the underlying results are theirs, but a reader new to the area will need to check the TMLR paper for the actual evidence.\n\nOverall, this is a fair, useful perspective that reframes how representational analyses should be interpreted. It doesn't resolve the long-open question of when representations track computations, but it gives the field a sharper caveat. A reader working on model-brain alignment will get real value from the noise-ceiling and RSA discussion. It deserves a serious referee: a good review would push the authors to be more explicit about the conditions under which biological systems would show the same bias, and perhaps to include evidence from more naturalistic models. My recommendation is to send it to peer review.","headline":"A clear, honest perspective on how representation biases distort common neural analyses, but the neuroscience conclusion rests on an untested transfer from toy MLPs to biological brains.","tokens_in":12529,"tokens_out":1735,"would_cite":true,"duration_ms":19750,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Learned representation bias can make neural analyses rank a system closer to one that does not compute a feature than to one that does.","keywords":["representation analysis","representation bias","representational similarity analysis","principal component analysis","neural encoding","computational neuroscience","deep learning","homomorphic encryption"],"falsifier":"Run the paper's easy-versus-hard training setup with a sweep of seeds and training orders and measure variance explained in the penultimate layer for each feature while both are perfectly classified; the paper's claim predicts the easy feature dominates by roughly an order of magnitude. A trained model in which the hard feature explains comparable variance—or in which RSA ranks hard-only models as the closest matches to each other—would show the bias is not the systematic pattern described.","tokens_in":11633,"feed_emoji":"🧠","tokens_out":9363,"duration_ms":110822,"temperature":0.7,"pith_summary":"This paper is a perspective on a hard limit to representational analysis: the internal representations learned by neural networks are systematically lopsided, with simple, prevalent, or early-learned features accounting for far more variance than equally computed but harder features. The paper shows that because PCA, regression, and RSA all lean on variance, they inherit this lopsidedness and can produce strongly biased inferences about what a system computes—for example, a model that computes both an easy and a hard feature can look nearly identical to a model that computes only the easy one. The authors argue this bias is a general property of feature learning and that no simple change of metric fully fixes it, because most comparison metrics still depend on variance or correlation. The practical stake is that complete understanding of a brain or model through representational analysis alone may be out of reach without prior knowledge of which features matter; the paper therefore recommends pairing representation studies with full behavioral characterization and causal intervention.","feed_headline":"Easy features can hide real computations from brain analyses","feed_subtitle":"A model computes two features equally but represents one ten times more strongly; standard tools read the wrong story.","key_machinery":"The mechanism is the representation-variance bias: the fraction of a layer's activity variance that can be attributed to a feature, together with the linking assumption that variance equals computational significance. Simpler features are learned first and have essentially one natural linear representation, so their variance concentrates; nonlinear features admit several non-linearly-equivalent representations whose inconsistent dimensions dilute one another. The paper's demonstrations rely on controlled multi-layer perceptron experiments with easy linear versus hard nonlinear (XOR-like) features, and on homomorphic encryption as a limiting case where arbitrary computation runs over intentionally opaque representations.","core_discovery":"The central claim is that patterns of representation are not a clean mirror of computation: variance in a neural layer is heavily concentrated on certain features, and the standard tools used to read representations therefore read those features and miss others. In the paper's illustrative experiments, an MLP (a simple feedforward neural network) is trained to classify an easy linear feature and a hard nonlinear feature; it generalizes perfectly on both, yet the easy feature explains more than 50 percent of the variance in the penultimate layer while the hard feature explains roughly 5 percent. The top principal components cluster by the easy feature, RSA makes a multi-task model look very similar to an easy-only model and less similar to models that compute the same hard task, and the units most driven by the hard feature still carry stronger easy-feature signals. The authors stress that these are biases in the inferences drawn, not necessarily flaws in the representations, and they use homomorphic encryption to show how far computation and superficial representation can diverge in the worst case.","pith_inferences":["By extension, interpretability methods outside neuroscience that select representational directions by variance or linear decodability—circuit discovery, linear probes, activation patching—may inherit the same blind spots for features that are computed nonlinearly.","A testable prediction follows from the paper's learning-order account: training curricula that interleave or pretraining that schedules hard features first should shrink the variance gap; existing training logs could be re-checked for this effect.","The homomorphic-encryption case implies that the absence of a linearly decodable compositional representation is not evidence against compositional computation, and the presence of one is not proof that it is the computation actually performed.","If cross-model and brain–model convergence is driven largely by shared easy features, then representational-alignment benchmarks may overstate agreement in computations; adding behavioral task-overlap measures alongside similarity scores would test this."],"forward_implications":["Principal component analysis of a layer's activity will foreground easy or prevalent features and can miss hard features entirely, so top components are not a safe inventory of a system's computations.","Representational similarity analysis can rank a multi-task model as closer to a model that only computes the easy feature than to a model that computes the same hard feature, so similarity scores do not reliably track functional overlap.","Regression-based encoding analyses will attribute most explained variance to simple or prevalent features even when harder features are equally relevant to behavior.","Between-subject noise ceilings for model–brain comparison can overstate the achievable prediction because two systems can compute the same feature in non-linearly-equivalent representation patterns.","Attaining complete understanding of a system will require pairing representational analysis with thorough behavioral characterization and causal tests."],"supporting_citations":[{"why":"Supplies the central experiments: trained MLPs compute easy and hard features equally well yet the easy feature explains an order of magnitude more representational variance.","marker":"Lampinen et al., 2024"},{"why":"Defines representational similarity analysis (RSA), the method whose similarity judgments the paper shows are distorted by variance biases.","marker":"Kriegeskorte et al., 2008"},{"why":"Earlier demonstration that feature representations can dissociate from task function; the paper's RSA multi-task versus easy-only comparison extends this.","marker":"Hermann and Lampinen, 2020"},{"why":"Shows representational and functional similarity dissociate in small transformer language models, supporting the generality of the bias beyond MLPs.","marker":"Friedman et al., 2023"},{"why":"Provides an analytical dissociation of functional and representational similarity in deep linear networks, supporting the theoretical framing.","marker":"Braun et al., 2025"},{"why":"Introduces fully homomorphic encryption over ideal lattices, the case study showing computation can proceed over representations deliberately made opaque.","marker":"Gentry, 2009"},{"why":"Presents a simpler homomorphic encryption scheme over integers, strengthening the case-study illustration with a second construction.","marker":"Van Dijk et al., 2010"},{"why":"Documents the simplicity and low-rank bias in deep networks, used in the paper to explain why easy features are learned first and dominate representations.","marker":"Huh et al., 2021"}],"fun_headline_variants":["Easy features dominate brain maps, hiding complex computations","Representation analysis can fool you about what a brain computes","Overrepresented easy features skew inferences from neural data","Variance bias: why neural representations mislead brain science","When representation and computation diverge: a cautionary tale"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The weight-bearing premise is that real brains show the same kind of representation bias as trained artificial networks; all concrete demonstrations come from small synthetic models, and the paper itself leaves the transfer to biology as an open question.","fun_headline_variants_meta":{"raw":{"variants":["Easy features dominate brain maps, hiding complex computations","Representation analysis can fool you about what a brain computes","Overrepresented easy features skew inferences from neural data","Variance bias: why neural representations mislead brain science","When representation and computation diverge: a cautionary tale"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1360,"prompt_tokens":905,"completion_tokens":455,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":378}},"tokens_in":521,"tokens_out":455,"duration_ms":5628,"temperature":1.0,"reasoning_tokens":378,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:57:15.221782+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's easy-versus-hard training setup with a sweep of seeds and training orders and measure variance explained in the penultimate layer for each feature while both are perfectly classified; the paper's claim predicts the easy feature dominates by roughly an order of magnitude. A trained model in which the hard feature explains comparable variance—or in which RSA ranks hard-only models as the closest matches to each other—would show the bias is not the systematic pattern described.","supporting_citations":[],"review_version":1}