Transformer weights at early training stages are closed-form compositions of bigram, token-interchangeability, and context mappings that directly reflect text-corpus statistics and explain the emergence of semantic associations.
An analysis for reasoning bias of language models with small initialization
3 Pith papers cite this work. Polarity classification is still indexing.
3
Pith papers citing it
representative citing papers
LoRA modules are a complementary, finite-capacity parametric memory for LLMs: capacity grows with rank, small ranks are most parameter-efficient, synthetic QA data helps most, and practical multi-LoRA systems are bottlenecked by routing and merging degradation.
Neural networks exhibit condensation of neurons into clusters with similar outputs whose number increases monotonically during training, facilitated by small initializations or dropout, providing insights into generalization and reasoning.
citing papers explorer
-
How Do Transformers Learn to Associate Tokens: Gradient Leading Terms Bring Mechanistic Interpretability
Transformer weights at early training stages are closed-form compositions of bigram, token-interchangeability, and context mappings that directly reflect text-corpus statistics and explain the emergence of semantic associations.
-
Understanding LoRA as Knowledge Memory: An Empirical Analysis
LoRA modules are a complementary, finite-capacity parametric memory for LLMs: capacity grows with rank, small ranks are most parameter-efficient, synthetic QA data helps most, and practical multi-LoRA systems are bottlenecked by routing and merging degradation.
-
An overview of condensation phenomenon in deep learning
Neural networks exhibit condensation of neurons into clusters with similar outputs whose number increases monotonically during training, facilitated by small initializations or dropout, providing insights into generalization and reasoning.