A deterministic Fourier expansion of token IDs plus a lightweight shared MLP matches learned embeddings on STS-B at a fraction of the parameter count in small transformers.
Quantifying unique information
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Parameter-Efficient Transformer Embeddings
A deterministic Fourier expansion of token IDs plus a lightweight shared MLP matches learned embeddings on STS-B at a fraction of the parameter count in small transformers.