REVIEW 3 major objections 8 minor 4 references
Unambiguous Representations in Neural Networks: An Information-Theoretic Approach to Intentionality
T0 review · 3 major / 8 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read This paper establishes that representational content can be fully fixed by a network's relational geometry, and demonstrates 100% decoding in dropout-trained digit classifiers.
desk verdict The empirical decoding results are real and interesting, but the inference to intrinsic unambiguous content is undercut by the paper's own H(I|R,C) concession. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Gram matrix of row-wise (or column-wise) cosine similarities of a weight layer, which encodes the relational geometry among neurons invariant to scale and permutation. The paper's two decoding tools are: geometric matching, which searches all permutations of a test network's output neurons to minimize Frobenius distance to a reference Gram matrix averaged from five labeled networks; and a permutation-invariant transformer decoder with no positional encodings, trained on thousands of permuted networks. The theoretical link is an Ambiguity Reduction Score (ARS) derived from Fano's inequality, mapping decoding accuracy and R² to a lower bound on how much H(I|R,C) falls
What would settle it
Scan a large set of dropout-trained MNIST networks seeded differently from the reference networks and check whether the true class permutation always yields the minimal Frobenius distance to the reference Gram matrix — one counterexample would break the zero-variance perfect result. A stronger test: train networks with class labels cyclically permuted, then apply the original decoder; if accuracy drops to chance, the relational geometry encodes a learned label convention rather than intrinsic class meaning.
Extended reading notes
Core claim
On its own terms, the paper claims that the pattern of pairwise cosine similarities between a neuron's weights and those of other neurons—the relational geometry—carries enough information to specify what that neuron represents, without knowing its position, training method, or even which architecture produced it. The central evidence: for dropout-trained feedforward MNIST classifiers, matching a test network's Gram matrix against a reference built from five labeled networks identifies the correct class permutation for all output neurons in every tested network (100%, zero standard deviation); standard backpropagation yields 38% and untrained networks 10%. A transformer decoder without posit
Load-bearing premise
The result stands only if the consistency of Gram matrices across trained networks, as read by a decoder that was itself trained on labels, captures what the representation intrinsically means rather than a convention shared by the training set and the decoder.
Editorial extensions
If this is right
- If correct, representational ambiguity is a measurable, optimizable dimension of neural systems, independent of task performance: dropout changed output decoding from near chance to perfect while classification accuracy stayed nearly constant.
- Richer relational structures—more output neurons in the Gram matrix—improve decoding relative to chance monotonically (from 1.0x at two neurons to 7.36x at ten), implying relational complexity itself reduces ambiguity.
- Because geometric matching transfers across hidden-layer architectures, the content encoded in relational geometry is not tied to a specific network shape.
- Dataset identity can be read off connectivity (99.8% for dropout, 84.3% for standard training), so the context a decoder needs for disambiguation may itself be encoded relationally.
Reading between the lines
- I would not take perfect decoding as proof of intrinsic, decoder-independent content: the decoder and reference Gram matrix are built with labels and from the same distribution, so the result may reflect a shared training-set convention rather than an intrinsic property of the representation (the paper concedes it measures H(I|R,C), not H(I|R)).
- A testable extension: train networks with class labels permuted relative to a fixed reference; if geometric matching still aligns neurons perfectly, the geometry tracks label statistics rather than true class semantics, whereas if it fails, the geometry is conventional.
- Applied to biological recordings, the framework suggests a concrete prediction: neural populations supporting conscious perception should have lower representational ambiguity (higher decodability of stimulus identity from population geometry) than populations processing the same stimuli unconsciously.
- The layer-specific reversal (standard training better for input position, dropout better for output class) suggests ambiguity is not a single global property but decomposes by layer and representational domain; any theory linking consciousness to a single ambiguity value would need to specify which layer's geometry is relevant.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an information-theoretic measure of representational ambiguity, H(I|R), and tests whether the relational structure of a neural network's weights (cosine-similarity Gram matrices) can be decoded to identify which digit class an output neuron represents and which spatial position an input neuron represents. On MNIST-trained fully-connected networks, geometric matching achieves 100% accuracy for dropout-trained networks (on 10 validation networks), 38% for standard backpropagation, and 10% for untrained networks, despite similar task accuracies. For input neurons, a transformer decoder achieves R^2 = 0.844 (standard) and 0.695 (dropout) in predicting distance from image center. The authors interpret these results as showing that relational structure alone can unambiguously determine representational content, and connect this to narrow representationalism and IIT. The paper includes a theoretical framework, two experiments, ablations, and a discussion of limitations.
Significance. If established, the empirical findings would be of interest to computational neuroscience and representation analysis: they suggest that training paradigm can affect how consistently class identity is encoded in weight geometry, independently of task accuracy. The paper provides a clean operationalization of 'representational ambiguity' and a reproducible experimental pipeline, with code made available. The core observation—that dropout-trained networks exhibit substantially more consistent Gram matrices across seeds than standard backpropagation—is new and worth reporting. However, the central theoretical claim that relational structure alone determines content, independent of any convention, is not supported by the data as presented. The paper's own limitation section concedes that the measured quantity is H(I|R,C), not H(I|R), and the finite-sample evidence for H=0 is weak. The significance of the contribution therefore depends on whether the claims can be tempered to context-relative decodability.
major comments (3)
- [§3.1.2, Table 1, §2.3] The central claim that 'relational structure alone ... completely determines what each neuron represents' (§4.1) rests on the 100% geometric matching accuracy in Table 1. This accuracy is computed on only 10 validation networks. A perfect score on 10 samples cannot establish zero conditional entropy: with 10 trials, the one-sided 95% confidence lower bound for accuracy is about 0.69 (Clopper-Pearson). Consequently, the ARS = 1.0 in Table 2, which the text equates with H(I|R,C) = 0, is not justified by the data. Please report confidence intervals, increase the validation set, or explicitly treat the 100% figure as a finite-sample estimate and weaken the corresponding claim.
- [§2.1.4, §4.1, §4.4] Both decoding methods are conditioned on an external label convention. Geometric matching uses a reference Gram matrix built from 5 networks with known class ordering (Eq. 2); the learned decoder is supervised on class labels. If a global permutation were applied to the class labels of the reference, the optimal matching for the same test network would change accordingly, so the recovered 'content' is relative to the reference labeling. The paper acknowledges in §4.4 that it measures H(I|R,C), not H(I|R), but the rejoinder—dataset identity is decodable and cross-architecture transfer works—does not address label-convention dependence. Therefore the conclusion that content is 'structurally determined, not conventionally assigned' (§4.1) is a non-sequitur. To support the strong claim, the authors would need to demonstrate label-invariance, e.g., by showing that a decoder trained without la
- [§2.3, Eq. (6), Table 2] The ARS formula for regression is not scale-invariant as written. Eq. (6) sets Var(Y) = 1 implicitly through the denominator log(2πe). The target in Experiment 2, f(i,j) = Euclidean distance from the image center, has a variance determined by the pixel grid and is not 1. Unless the target is explicitly standardized to unit variance before computing R², the reported ARS values (0.419 and 0.654) are not valid entropy reductions and are likely overestimates. Please state whether Y was standardized; if not, derive the correct expression using the actual variance of Y. The same issue affects cross-task comparisons of ARS in Table 2.
minor comments (8)
- [Abstract / §1] The abstract says 'red square cannot alternatively encode the experience of a green triangle' while the introduction says 'green square.' Please harmonize.
- [§2.1.4] The description of the transformer decoder is ambiguous: 'the only positional information is which row corresponds to the target neuron.' Clarify how the target row is indicated to the model (e.g., a special token, ordering, or masking).
- [§3.1.5] The cross-architecture transfer results are reported only as 'Figure 6 shows strong decoding performance.' Provide the numerical accuracies for the 3x3 matrix so the claim can be assessed.
- [§3.2] The spatial-position regression experiment does not explicitly state the train/validation split. The classification experiment uses 800/200 networks; specify whether the same split was used for the regression decoder.
- [§3.1.4] The claim that 'performance relative to random chance increases systematically with neuron count' conflates changes in absolute accuracy with changes in the chance baseline. Report absolute accuracies for all k values; the 10-neuron absolute accuracy (73.6%) is lower than the 5-neuron value (79.1%), so the 'systematic improvement' is driven mostly by the decrease in chance level.
- [§4.4] The 'universal decoder' paragraph is speculative. It is presented as evidence that the context dependence is practical rather than fundamental, but no proof or simulation is provided. Label it as a research direction.
- [§2.3, Eq. (5)] Fano's inequality bounds the conditional entropy in terms of the true error probability. Using the empirical accuracy as a plug-in estimate is fine, but the finite-sample uncertainty should be acknowledged where ARS values are interpreted.
- [§2.1.1] The choice of 2 training epochs is arbitrary and may underfit MNIST. A sentence justifying this choice or a sensitivity check would strengthen the paper.
Circularity Check
No self-citation or fitted-input circularity: the decoding experiments are held-out and self-contained, but the central claim that relational structure alone determines content is undercut by the paper's own admission that it measures H(I|R,C), not H(I|R).
-
other
[§2.3 Eq. (4), §3.1.2, §4.1, §4.4]
"Second, we measure H(I|R, C) rather than H(I|R), where C represents context... [yet] the relational structure alone—without any information about neuron ordering or training procedure—completely determines what each neuron represents. ... the content is structurally determined, not conventionally assigned."
The paper defines its theoretical quantity as H(I|R) in Eq. (1), but the only quantity actually computed is H(I|R,C) in Eq. (4), as §4.4 concedes. The strong conclusion that 'relational structure alone' determines content is therefore not an independent derivation; it is the context-conditioned decodability result restated as a context-free property. The reference Gram matrix is built from 5 labeled networks with known class ordering (§2.1.4), so the decoded 'content' is anchored to the experimenter's label convention; a global relabeling of the reference would reassign the decoded content of the same test network. Thus the inference from finite decoding accuracy to 'structurally determined, not conventionally assigned' reduces to the operationalization rather than following from it.
full rationale
The core experimental chain is not circular: the learned decoder is trained on 800 networks and validated on 200 held-out networks, geometric matching is evaluated on 10 held-out networks against a reference built from 5 separate networks, and the cross-architecture and dataset-discrimination tests are also held-out. No parameter is fitted to the target claim, and there are no self-citations. The only notable issue is interpretive: the paper measures H(I|R,C), not H(I|R), and then uses the result to support the claim that relational structure alone fixes content. This is a conceded limitation (§4.4) and is an overreach rather than a hidden circularity in the measurement itself. Finite-sample perfect accuracy on 10 validation networks also does not strictly prove H(I|R,C)=0, though Fano's inequality with A=1 gives a lower bound of 1.0 for ARS. Under the review rules, this warrants a low score (2) but not a finding of substantive circularity.
Assumptions & free parameters
free parameters (5)
- Dropout probability =
0.20
- Training epochs =
2
- Reference networks for geometric matching =
5
- Validation networks for geometric matching =
10
- Neuron subset size k =
2..10
assumptions (6)
- domain assumption Consciousness is intrinsic: a neural state's representational content is fixed by the brain state itself, independent of external interpretation.
- domain assumption Narrow representationalism / IIT intrinsicality: intentional contents determine experience, and these contents supervene on brain states.
- ad hoc to paper The cosine-similarity Gram matrix of layer weights is the relevant relational structure encoding representational content.
- ad hoc to paper Class labels (digit identities) and pixel-distance-from-center are the interpretations I for output and input neurons respectively.
- domain assumption Fano's inequality and Gaussian-residual assumptions yield valid lower bounds on ambiguity reduction (ARS).
- domain assumption Decoding accuracy on held-out networks translates to conditional entropy H(I|R,C) of the representation.
Cite this review
Pith. "Pith review of Unambiguous Representations in Neural Networks: An Information-Theoretic Approach to Intentionality." pith.science (2026). https://pith.science/paper/56VZ4GKX
@misc{pith2026251211000,
author = {Pith},
title = {Pith review of: Unambiguous Representations in Neural Networks: An Information-Theoretic Approach to Intentionality},
year = {2026},
howpublished = {\url{https://pith.science/paper/56VZ4GKX}},
note = {Machine review of arXiv:2512.11000}
}
read the original abstract
Representations pervade our daily experience, from letters representing sounds to bit strings encoding digital files. While such representations require externally defined decoders to convey meaning, conscious experience is fundamentally different: a neural state corresponding to perceiving a red square cannot alternatively encode the experience of a green triangle. This intrinsic property of consciousness suggests that conscious representations must be unambiguous in a way that conventional representations are not. We formalize this intuition using information theory, defining representational ambiguity as the conditional entropy H(I|R) over possible interpretations I given a representation R. Through experiments on neural networks trained to classify MNIST digits, we demonstrate that relational structures in network connectivity can unambiguously encode representational content. From relational structure alone, we achieve perfect (100%) accuracy for dropout-trained networks and 38% for standard backpropagation (chance: 10%) in identifying output neuron class identity, despite identical task performance, demonstrating that representational ambiguity can arise orthogonally to behavioral accuracy. We further show that spatial position of input neurons, relevant to phenomenal properties like visual field location, can be decoded from network connectivity with R^2 up to 0.844. These results provide a quantitative method for measuring representational ambiguity in neural systems and demonstrate that neural networks can exhibit the low-ambiguity representations posited as necessary (though not sufficient) by theoretical accounts such as narrow representationalism and IIT.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1998]
Representational theories of consciousness
William Lycan. Representational theories of consciousness. In Edward N. Zalta, editor,The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, fall 2019 edition,
2019
-
[2004]
The platonic representation hypothesis.arXiv preprint arXiv:2405.07987,
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis.arXiv preprint arXiv:2405.07987,
-
[2016]
This metric, originally applied to large cross-modal models by Huh et al., measures the alignment of representations across different network instances
A Representational Alignment and Kernel Similarity A.1 Mutual k-NN Kernel Similarity To further validate the connection between our work and the Platonic Representation Hypoth- esis [Huh et al., 2024], we computed the mutual k-NN kernel similarity metric on our MNIST networks. This metric, originally applied to large cross-modal models by Huh et al., meas...
2024
-
[2019]
Neurophenomenal structuralism
Holger Lyre. Neurophenomenal structuralism. a philosophical agenda for a structuralist neuro- science of consciousness.Neuroscience of Consciousness, 2022(1):niac012,
2022
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.