Unembedding collapse in transformers prevents distinguishing unseen tokens in symbolic reasoning, but targeted interventions restore generalization.
Positional description matters for transformers arithmetic
3 Pith papers cite this work. Polarity classification is still indexing.
3
Pith papers citing it
representative citing papers
Grokking delay on Collatz prediction is a decoder access bottleneck after early encoder structure learning, with numeral base as a strong inductive bias on learnability.
FoNE encodes numbers as single tokens via Fourier features and outperforms subword and digit-wise embeddings on addition, subtraction, and multiplication with far less data.
citing papers explorer
-
To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoning
Unembedding collapse in transformers prevents distinguishing unseen tokens in symbolic reasoning, but targeted interventions restore generalization.
-
The Long Delay to Arithmetic Generalization: When Learned Representations Outrun Behavior
Grokking delay on Collatz prediction is a decoder access bottleneck after early encoder structure learning, with numeral base as a strong inductive bias on learnability.
-
FoNE: Precise Single-Token Number Embeddings via Fourier Features
FoNE encodes numbers as single tokens via Fourier features and outperforms subword and digit-wise embeddings on addition, subtraction, and multiplication with far less data.