In a stochastic k-ary tree, a two-head transformer learns randomized DFS via policy gradient under depth-wise curriculum, generalizes to deeper trees, and adapts to imbalanced goals via discounting.
A theory of emergent in-context learning as implicit structure induction
7 Pith papers cite this work. Polarity classification is still indexing.
7
Pith papers citing it
representative citing papers
Manifold steering along activation geometry induces behavioral trajectories matching the natural manifold of outputs, while linear steering produces off-manifold unnatural behaviors.
Hawk raises NPU kernel generation accuracy from 49.4% to 80% and yields up to 2.2× speedups by retrieving and distilling structured hardware-aware knowledge without any model training.
Humans and LLMs exhibit similar error patterns in common-sense reasoning, consistent with shared pattern-matching mechanisms rather than abstract world models.
LLMs dynamically construct and causally rely on structured conceptual subspaces in middle-to-late layers for in-context inference.
Curriculum post-training on reasoning trees yields polynomial sample complexity for accurate Chain-of-Thought generation in transformers, unlike exponential requirements without curriculum.
The paper surveys definitions, techniques, applications, and challenges in in-context learning for large language models.
citing papers explorer
-
Agentic Transformers Provably Learn to Search via Reinforcement Learning
In a stochastic k-ary tree, a two-head transformer learns randomized DFS via policy gradient under depth-wise curriculum, generalizes to deeper trees, and adapts to imbalanced goals via discounting.
-
Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior
Manifold steering along activation geometry induces behavioral trajectories matching the natural manifold of outputs, while linear steering produces off-manifold unnatural behaviors.
-
Hawk: Harnessing Hardware-Aware Knowledge for High-Performance NPU Kernel Generation
Hawk raises NPU kernel generation accuracy from 49.4% to 80% and yields up to 2.2× speedups by retrieving and distilling structured hardware-aware knowledge without any model training.
-
Reasoning as Pattern Matching: Shared Mechanisms in Human and LLM Everyday Reasoning
Humans and LLMs exhibit similar error patterns in common-sense reasoning, consistent with shared pattern-matching mechanisms rather than abstract world models.
-
Emergent Structured Representations Support Flexible In-Context Inference in Large Language Models
LLMs dynamically construct and causally rely on structured conceptual subspaces in middle-to-late layers for in-context inference.
-
Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Training
Curriculum post-training on reasoning trees yields polynomial sample complexity for accurate Chain-of-Thought generation in transformers, unlike exponential requirements without curriculum.
-
A Survey on In-context Learning
The paper surveys definitions, techniques, applications, and challenges in in-context learning for large language models.