Pythia releases 16 identically trained LLMs with full checkpoints and data tools to study training dynamics, scaling, memorization, and bias in language models.
Markosyan, Luke Zettlemoyer, and Armen Aghajanyan
7 Pith papers cite this work. Polarity classification is still indexing.
7
Pith papers citing it
representative citing papers
During pretraining, language models exhibit natural ungrokking where learned rules are forgotten based on their support frequency in the corpus, with asymmetric editability of rule survival.
Derives three-force decomposition of squared weight norm under AdamW and validates it on Pythia-70M models, plus spline recovery of alignment force from checkpoints.
NumLeak detects high-fidelity recall of public numeric time series in frontier LLMs, with correlations of 0.97-0.99 on Fama-French data and similar patterns on economic indicators.
Interaction-layer antidistillation watermarks use system-prompt-induced behavioral markers like explicit follow-up questions that transfer to distilled student models at 45-89% relative fidelity and can be audited via black-box LLM-as-judge queries.
DUET is a global-to-local method that optimizes LLM training data mixtures via Bayesian optimization guided by influence-based selection and feedback from unseen evaluation tasks, with a regret bound showing convergence to the optimal mixture.
Self-RAG trains LLMs to adaptively retrieve passages on demand and self-critique using reflection tokens, outperforming ChatGPT and retrieval-augmented Llama2 on QA, reasoning, and fact verification.
citing papers explorer
-
Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining
During pretraining, language models exhibit natural ungrokking where learned rules are forgotten based on their support frequency in the corpus, with asymmetric editability of rule survival.
-
NumLeak: Public Numeric Benchmarks as Latent Labels in Foundation Models
NumLeak detects high-fidelity recall of public numeric time series in frontier LLMs, with correlations of 0.97-0.99 on Fama-French data and similar patterns on economic indicators.
-
Asking Back: Interaction-Layer Antidistillation Watermarks
Interaction-layer antidistillation watermarks use system-prompt-induced behavioral markers like explicit follow-up questions that transfer to distilled student models at 45-89% relative fidelity and can be audited via black-box LLM-as-judge queries.
-
DUET: Optimizing Training Data Mixtures via Feedback from Unseen Evaluation Tasks
DUET is a global-to-local method that optimizes LLM training data mixtures via Bayesian optimization guided by influence-based selection and feedback from unseen evaluation tasks, with a regret bound showing convergence to the optimal mixture.
-
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
Self-RAG trains LLMs to adaptively retrieve passages on demand and self-critique using reflection tokens, outperforming ChatGPT and retrieval-augmented Llama2 on QA, reasoning, and fact verification.