REVIEW 14 cited by
Datamodels: Predicting Predictions from Training Data
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
We present a conceptual framework, datamodeling, for analyzing the behavior of a model class in terms of the training data. For any fixed "target" example $x$, training set $S$, and learning algorithm, a datamodel is a parameterized function $2^S \to \mathbb{R}$ that for any subset of $S' \subset S$ -- using only information about which examples of $S$ are contained in $S'$ -- predicts the outcome of training a model on $S'$ and evaluating on $x$. Despite the potential complexity of the underlying process being approximated (e.g., end-to-end training and evaluation of deep neural networks), we show that even simple linear datamodels can successfully predict model outputs. We then demonstrate that datamodels give rise to a variety of applications, such as: accurately predicting the effect of dataset counterfactuals; identifying brittle predictions; finding semantically similar examples; quantifying train-test leakage; and embedding data into a well-behaved and feature-rich representation space. Data for this paper (including pre-computed datamodels as well as raw predictions from four million trained deep neural networks) is available at https://github.com/MadryLab/datamodels-data .
Forward citations
Cited by 14 Pith papers
-
Domain-Aware Scaling Laws Uncover Data Synergy
Domain-aware scaling laws with fitted γ and σ synergy terms recover stable code-math interactions from observational LLM mixtures and correctly predict mixture rankings in controlled small-scale trainings.
-
An Asymptotic Analysis of the Shapley Value for Dataset Valuation
Under smooth RKHS embedding utilities, a fixed owner's Shapley value is O(1/I)-close in L1 to an explicit leading term of scale (log I)/I driven by a first-order population signal.
-
Small edits, large models: How Wikipedia advocacy shapes LLM values
Wikipedia edits by animal welfare advocates measurably influence LLM outputs on animal welfare topics, shown via retrieval and gradient attribution plus fine-tuning experiments.
-
Understanding Data Influence with Differential Approximation
This paper introduces Diff-In, an influence estimator that accumulates second-order approximations of influence differences across training steps and shows strong accuracy in data cleaning, deletion, and coreset selec...
-
Better Training Data Attribution via Better Inverse Hessian-Vector Products
ASTRA, an EKFAC-preconditioned Neumann series iteration, computes more accurate inverse Hessian-vector products and improves training data attribution scores over EKFAC baselines.
-
AdaDeDup: Adaptive Hybrid Data Pruning for Efficient Large-Scale Object Detection Training
ADADEDUP adaptively prunes object detection datasets by combining semantic clustering with proxy-model loss feedback, matching full-data mAP at 20% pruning.
-
ClusterUCB: Efficient Gradient-Based Data Selection for Targeted Fine-Tuning of LLMs
ClusterUCB uses gradient clustering plus a modified UCB bandit to match full-budget gradient influence data selection at a 20% computing budget.
-
Daunce: Data Attribution through Uncertainty Estimation
DAUNCE computes training-data attribution as the covariance of per-example losses across an ensemble of perturbed fine-tuned models, reporting state-of-the-art LDS scores and the first attribution runs on proprietary LLMs.
-
Expert Survey: AI Reliability & Security Research Priorities
Expert ratings place capability forecasting and dangerous-capability evaluations at the top of a 105-area AI reliability and security research priority list.
-
Merge to Mix: Mixing Datasets via Model Merging
Merge to Mix shows that the performance of a parameter-averaged model predicts the performance of a model fine-tuned on any dataset mixture, enabling fast and accurate dataset mixture selection.
-
MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning
Multi-domain RLVR data mixing, guided by a quadratic surrogate fitted to 11 pilot runs, improves a Qwen2-VL-2B model's out-of-distribution accuracy by about 5 points over uniform mixing.
-
A Survey of LLM $\times$ DATA
A comprehensive survey of the bidirectional links between LLMs and data management, organized as DATA4LLM and LLM4DATA with a new 'IaaS' data-quality framework.
-
What Is The Performance Ceiling of My Classifier? Utilizing Category-Wise Influence Functions for Pareto Frontier Analysis
Category-wise influence vectors plus linear programming and a genetic algorithm reweight training data to improve all classes at once, with an unproven criterion for when a classifier has reached its Pareto ceiling.
-
Scaling laws for activation steering with Llama 2 models and refusal mechanisms
Activation steering with contrastive vectors becomes less effective as Llama 2 models scale from 7B to 70B parameters, with peak effect at roughly 40% of the model's layers.
Discussion (0). Sign in to comment.