REVIEW 2 cited by
Is In-Context Universality Enough? MLPs are Also Universal In-Context
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
The success of transformers is often linked to their ability to perform in-context learning. Recent work shows that transformers are universal in context, capable of approximating any real-valued continuous function of a context (a probability measure over $\mathcal{X}\subseteq \mathbb{R}^d$) and a query $x\in \mathcal{X}$. This raises the question: Does in-context universality explain their advantage over classical models? We answer this in the negative by proving that MLPs with trainable activation functions are also universal in-context. This suggests the transformer's success is likely due to other factors like inductive bias or training stability.
Forward citations
Cited by 2 Pith papers
-
How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Off
Heavy-tailed pretraining distributions improve in-context task selection under distribution shift but worsen ICL generalization, especially in low-data regimes.
-
Beyond Universal Approximation Theorems: Algorithmic Uniform Approximation by Neural Networks Trained with Noisy Data
An explicit randomized training pipeline is claimed to yield uniform approximators from noisy data with minimax-optimal trainable parameters, but key sample-complexity claims are algebraically reversed and the proof s...
Discussion (0). Sign in to comment.