Fixed transformers achieve universality for a class of models by encoding target descriptions in input embeddings, with random initializations being universal almost surely.
Attention retrieves, mlp memorizes: Disentangling trainable components in the transformer
6 Pith papers cite this work. Polarity classification is still indexing.
6
Pith papers citing it
representative citing papers
Conflict and hallucination in transformers are basin competition versus basin absence in hidden-state space; geometric margin detects them with zero false refusals while entropy cannot, and confident hallucinations scale as exp(-c/Δ̄).
MLP activations measured as massive activations or first four moments correlate weakly (max |Spearman| = 0.33) with in-context example quality across Llama-3.2-3B, Qwen2.5-3B, and multiple classification/generative tasks, so activation-based active learning should not be used for ICL.
Geometry-calibrated conformal abstention lets language models abstain from uncertain queries with finite-sample guarantees on both participation rate and conditional correctness of answers.
In a stylized one-layer transformer, pre-training encodes factual knowledge via relation-specific feature directions and attention patterns; fine-tuning extracts it through a relation-covering mechanism that succeeds when enough latent templates are triggered, with a failure regime explaining inauds
SPON adds a small set of trainable input-independent activation vectors as representational anchors, trained by distribution matching, to stabilize sparse activation in LLMs and recover performance lost to hidden-state distribution shifts.
citing papers explorer
-
Fixed Universal Transformers
Fixed transformers achieve universality for a class of models by encoding target descriptions in input embeddings, with random initializations being universal almost surely.
-
Attractor Geometry of Transformer Memory: From Conflict Arbitration to Confident Hallucination
Conflict and hallucination in transformers are basin competition versus basin absence in hidden-state space; geometric margin detects them with zero false refusals while entropy cannot, and confident hallucinations scale as exp(-c/Δ̄).
-
Activation-Based Active Learning for In-Context Learning: Challenges and Insights
MLP activations measured as massive activations or first four moments correlate weakly (max |Spearman| = 0.33) with in-context example quality across Llama-3.2-3B, Qwen2.5-3B, and multiple classification/generative tasks, so activation-based active learning should not be used for ICL.
-
Geometry-Calibrated Conformal Abstention for Language Models
Geometry-calibrated conformal abstention lets language models abstain from uncertain queries with finite-sample guarantees on both participation rate and conditional correctness of answers.
-
Provable Knowledge Acquisition and Extraction in One-Layer Transformers
In a stylized one-layer transformer, pre-training encodes factual knowledge via relation-specific feature directions and attention patterns; fine-tuning extracts it through a relation-covering mechanism that succeeds when enough latent templates are triggered, with a failure regime explaining inauds
-
Resting Neurons, Active Insights: Robustifying Activation Sparsity in LLMs via Spontaneity
SPON adds a small set of trainable input-independent activation vectors as representational anchors, trained by distribution matching, to stabilize sparse activation in LLMs and recover performance lost to hidden-state distribution shifts.