REVIEW 13 cited by
A Theory for Emergence of Complex Skills in Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
A major driver of AI products today is the fact that new skills emerge in language models when their parameter set and training corpora are scaled up. This phenomenon is poorly understood, and a mechanistic explanation via mathematical analysis of gradient-based training seems difficult. The current paper takes a different approach, analysing emergence using the famous (and empirical) Scaling Laws of LLMs and a simple statistical framework. Contributions include: (a) A statistical framework that relates cross-entropy loss of LLMs to competence on the basic skills that underlie language tasks. (b) Mathematical analysis showing that the Scaling Laws imply a strong form of inductive bias that allows the pre-trained model to learn very efficiently. We informally call this {\em slingshot generalization} since naively viewed it appears to give competence levels at skills that violate usual generalization theory. (c) A key example of slingshot generalization, that competence at executing tasks involving $k$-tuples of skills emerges essentially at the same scaling and same rate as competence on the elementary skills themselves.
Forward citations
Cited by 13 Pith papers
-
Training with (Swap) Regret Loss in a Single-Layer Self-Attention Model: A Case Study on the Probability Simplex
Training single-layer attention with squared regret loss has stationary points that implement smoothed fictitious play (external regret) and, via a new swap-regret loss, the Blum–Mansour no-swap-regret algorithm.
-
Explaining Data Mixing Scaling Laws
Under a shared-head/disjoint-tail assumption, multi-domain loss decomposes into a capacity-competition term c_i x_i^*(h)^{-b_i} plus a per-domain noise term A_i(Dh_i)^{-a_i}, and the fitted law extrapolates optimal mi...
-
Olmo Hybrid: From Theory to Practice and Back
Hybrid attention+GDN models express tasks neither pure transformers nor pure linear RNNs can, and a matched 7B Olmo Hybrid scales more data-efficiently than Olmo 3.
-
Universal One-third Time Scaling in Learning Peaked Distributions
Softmax + cross-entropy on peaked targets yields loss ∼ t^{−1/3}, giving an architecture-driven explanation for power-law LLM training time without power-law data.
-
The Power of Power Law: Asymmetry Enables Compositional Reasoning
Power-law data sampling creates beneficial asymmetry in the loss landscape that lets models acquire high-frequency skill compositions first, enabling more efficient learning of rare long-tail skills than uniform distr...
-
Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learning
From Open LLM Leaderboard data grouped by base model, the authors recover a three-factor ordering of LLM capabilities and claim instruction-following causally supports math reasoning.
-
A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search
A skill-graph random-walk model gives closed-form accuracy-versus-compute formulas for four reasoning strategies and connects them to training scaling.
-
Circuit Stability Characterizes Language Model Generalization
Circuit stability, measured as rank correlation between soft circuits across subtasks, is proposed as a predictor of language model generalization.
-
A Mathematical Framework for AI-Human Integration in Work
A formal job-accuracy model shows that job success can jump sharply near a critical ability threshold, and combining workers with complementary decision and action subskills can create superadditive gains, formalizing...
-
Concealment of Intent: A Game-Theoretic Analysis
Intent-hiding adversarial prompting that mixes malicious intents with innocuous skills bypasses prompt and response filters, and a game-theoretic analysis quantifies the attacker's scaling advantage.
-
The Role of Diversity in In-Context Learning for Large Language Models
Diversity-aware selection of in-context examples improves performance on complex and out-of-distribution tasks, though effect sizes are often modest.
-
Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law
A fitted token-level loss weighting, optimized against known model accuracies, predicts held-out downstream task performance more accurately than mean validation loss on five of six benchmarks.
-
Decomposing Complex Visual Comprehension into Atomic Visual Skills for Vision Language Models
VLMs score far below adult humans on a new 13,188-question benchmark of 36 atomic 2D geometry perception skills.
Discussion (0). Sign in to comment.