A deterministic Fourier expansion of token IDs plus a lightweight shared MLP matches learned embeddings on STS-B at a fraction of the parameter count in small transformers.
The Gram-Schmidt Walk: A Cure for the Banaszczyk Blues
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
An important result in discrepancy due to Banaszczyk states that for any set of $n$ vectors in $\mathbb{R}^m$ of $\ell_2$ norm at most $1$ and any convex body $K$ in $\mathbb{R}^m$ of Gaussian measure at least half, there exists a $\pm 1$ combination of these vectors which lies in $5K$. This result implies the best known bounds for several problems in discrepancy. Banaszczyk's proof of this result is non-constructive and a major open problem has been to give an efficient algorithm to find such a $\pm 1$ combination of the vectors. In this paper, we resolve this question and give an efficient randomized algorithm to find a $\pm 1$ combination of the vectors which lies in $cK$ for $c>0$ an absolute constant. This leads to new efficient algorithms for several problems in discrepancy theory.
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Parameter-Efficient Transformer Embeddings
A deterministic Fourier expansion of token IDs plus a lightweight shared MLP matches learned embeddings on STS-B at a fraction of the parameter count in small transformers.