Pythia releases 16 identically trained LLMs with full checkpoints and data tools to study training dynamics, scaling, memorization, and bias in language models.
arXiv preprint arXiv:2210.14891 , year=
8 Pith papers cite this work, alongside 15 external citations. Polarity classification is still indexing.
8
Pith papers citing it
15
external citations · external index
representative citing papers
Self-evolving rubric with anti-gaming fitness reveals that objective capability scaling fails to transfer to subjective LLM behaviors, with advice-restraint as the universal lowest dimension that can regress.
Language models show a scale-dependent switch from anticorrelated to correlated reasoning-truthfulness coupling at a family-specific critical parameter count, with architecture and data choices shifting the transition point.
Pre-training loss predicts LLM math reasoning better than parameter count; rejection sampling fine-tuning with diverse paths raises LLaMA-7B accuracy on GSM8K from 35.9% with SFT to 49.3%.
The paper argues flat minima are an illusion and that a reparameterization-invariant 'weakness' score predicts generalization where raw sharpness fails.
Presents a single functional form for neural scaling that unifies multiple scaling dimensions and claims higher extrapolation accuracy than prior forms across diverse tasks and architectures.
Frontier models show positive capability coupling (r=0.72) across SWE-bench and GPQA, with lab-specific emphasis shifts measured by an h-field residual that distinguishes permanent pretraining changes from reversible post-training ones.
Introduces ANAI framework with Autonomy Index (AIx), Infrastructure Coupling Coefficient (ICC), and Technological Transition Potential (TTP) to model AI-driven infrastructural transition via nonlinear coevolution and recursive feedback loops.
citing papers explorer
-
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
Pythia releases 16 identically trained LLMs with full checkpoints and data tools to study training dynamics, scaling, memorization, and bias in language models.
-
Does Capability Transfer to Subjective Behavior -- and Would Our Instruments Tell Us? A Self-Evolving, Trust-by-Construction Evaluation Paradigm
Self-evolving rubric with anti-gaming fitness reveals that objective capability scaling fails to transfer to subjective LLM behaviors, with advice-restraint as the universal lowest dimension that can regress.
-
Lying Is Just a Phase: The Hidden Alignment Transition in Language Model Scaling
Language models show a scale-dependent switch from anticorrelated to correlated reasoning-truthfulness coupling at a family-specific critical parameter count, with architecture and data choices shifting the transition point.
-
Scaling Relationship on Learning Mathematical Reasoning with Large Language Models
Pre-training loss predicts LLM math reasoning better than parameter count; rejection sampling fine-tuning with diverse paths raises LLaMA-7B accuracy on GSM8K from 35.9% with SFT to 49.3%.
-
Are Flat Minima an Illusion?
The paper argues flat minima are an illusion and that a reparameterization-invariant 'weakness' score predicts generalization where raw sharpness fails.
-
Unified Neural Scaling Laws
Presents a single functional form for neural scaling that unifies multiple scaling dimensions and claims higher extrapolation accuracy than prior forms across diverse tasks and architectures.
-
The Growing Pains of Frontier Models: When Leaderboards Stop Separating and What to Measure Next
Frontier models show positive capability coupling (r=0.72) across SWE-bench and GPQA, with lab-specific emphasis shifts measured by an h-field residual that distinguishes permanent pretraining changes from reversible post-training ones.
-
AI-Native Autonomous Infrastructure (ANAI): A Formal Framework for the Next General-Purpose Technology
Introduces ANAI framework with Autonomy Index (AIx), Infrastructure Coupling Coefficient (ICC), and Technological Transition Potential (TTP) to model AI-driven infrastructural transition via nonlinear coevolution and recursive feedback loops.