Token-level intrinsic dimension of internal representations correlates with next-token cross-entropy loss across layers in three LLMs; higher-loss prompts live in higher-dimensional token manifolds.
Generic controllability of equivariant systems and applications to particle systems and neural networks
1 Pith paper cite this work. Polarity classification is still indexing.
abstract
There exist many examples of systems which have some symmetries, and which one may monitor with symmetry preserving controls. Since symmetries are preserved along the evolution, full controllability is not possible, and controllability has to be considered inside sets of states with same symmetries. We prove that generic systems with symmetries are controllable in this sense. This result has several applications, for instance: (i) generic controllability of particle systems when the kernel of interaction between particles plays the role of a mean-field control; (ii) generic controllability for families of vector fields on manifolds with boundary; (iii) universal interpolation for neural networks architectures with "generic" self attention-type layers - a type of layers ubiquitous in recent neural networks architectures, e.g., in the Transformers architecture. The tools we develop could help address various other questions of control of equivariant systems.
citation-role summary
citation-polarity summary
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
The Geometry of Tokens in Internal Representations of Large Language Models
Token-level intrinsic dimension of internal representations correlates with next-token cross-entropy loss across layers in three LLMs; higher-loss prompts live in higher-dimensional token manifolds.