REVIEW 21 cited by
What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large language models (LLMs) are trained on a vast amount of human-written data, but data providers often remain uncredited. In response to this issue, data valuation (or data attribution), which quantifies the contribution or value of each data to the model output, has been discussed as a potential solution. Nevertheless, applying existing data valuation methods to recent LLMs and their vast training datasets has been largely limited by prohibitive compute and memory costs. In this work, we focus on influence functions, a popular gradient-based data valuation method, and significantly improve its scalability with an efficient gradient projection strategy called LoGra that leverages the gradient structure in backpropagation. We then provide a theoretical motivation of gradient projection approaches to influence functions to promote trust in the data valuation process. Lastly, we lower the barrier to implementing data valuation systems by introducing LogIX, a software package that can transform existing training code into data valuation code with minimal effort. In our data valuation experiments, LoGra achieves competitive accuracy against more expensive baselines while showing up to 6,500x improvement in throughput and 5x reduction in GPU memory usage when applied to Llama3-8B-Instruct and the 1B-token dataset.
Forward citations
Cited by 21 Pith papers
-
Identifying Prompted Artist Names from Generated Images
A new 1.95M-image benchmark measures how well vision models identify artist names explicitly prompted into text-to-image systems, across artists, prompts, generators, and artist counts.
-
A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
VIF approximates leave-one-out retraining influence for non-decomposable losses (contrastive, ranking, Cox) using a finite-difference of the loss evaluated via auto-differentiation.
-
Sketching the Readout of Large Language Models for Scalable Data Attribution and Valuation
RISE applies CountSketch to dual lexical and semantic channels derived from output-layer gradient outer products, cutting data attribution storage by up to 112x and enabling retrospective and prospective influence ana...
-
LLM generation novelty through the lens of semantic similarity
An output is called novel when no semantically similar pretraining context can be retrieved; using this embedding-based measure, SmolLM models show longer-range reuse and instruction tuning increases novelty.
-
Understanding Data Influence with Differential Approximation
This paper introduces Diff-In, an influence estimator that accumulates second-order approximations of influence differences across training steps and shows strong accuracy in data cleaning, deletion, and coreset selec...
-
Better Training Data Attribution via Better Inverse Hessian-Vector Products
ASTRA, an EKFAC-preconditioned Neumann series iteration, computes more accurate inverse Hessian-vector products and improves training data attribution scores over EKFAC baselines.
-
The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text
A new 8TB openly-licensed text corpus trains 7B LLMs that are competitive with Llama 1/2, showing that performant models need not depend on unlicensed web data.
-
MedPAIR: Measuring Physicians and AI Relevance Alignment in Medical Question Answering
LLM relevance judgments align with physician trainees on only about 45 to 66 percent of sentences, and pruning contexts to physician-labeled relevant sentences improves LLM and trainee accuracy.
-
Daunce: Data Attribution through Uncertainty Estimation
DAUNCE computes training-data attribution as the covariance of per-example losses across an ensemble of perturbed fine-tuned models, reporting state-of-the-art LDS scores and the first attribution runs on proprietary LLMs.
-
Position: The Most Expensive Part of an LLM should be its Training Data
Even at conservative wages, recreating LLM training data from scratch would cost 10 to 1000 times more than the compute and energy used to train the models.
-
Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilities
BIDS, a balanced influence-based data selection algorithm using per-task normalization and iterative greedy selection, improves balanced multi-capability instruction tuning and can outperform full-dataset training on ...
-
TAROT: Targeted Data Selection via Optimal Transport
Targeted data selection via whitened feature distance and optimal transport minimization consistently outperforms influence-based greedy heuristics across vision and language tasks.
-
AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution
AttriBoT combines caching, hierarchical pruning, and smaller proxy models to approximate leave-one-out context attribution with a >300x speedup and little loss in faithfulness.
-
Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models
Influence traces show LLM reasoning on simple math is driven by documents that demonstrate procedures, not by documents containing the answers, suggesting a generalization strategy unlike retrieval.
-
KAIROS: Scalable Model-Agnostic Data Valuation
KAIROS derives a closed-form Maximum Mean Discrepancy influence score that approximates leave-one-out data rankings and detects noise, mislabels, and backdoors without retraining.
-
Counterfactual Influence as a Distributional Quantity
Near-duplicate training samples lower a model's self-influence on a record while raising its extractability, so self-influence alone underestimates memorization risk.
-
Where Did Your Model Learn That? Label-free Influence for Self-supervised Learning
Influence-SSL defines influence as the sensitivity of a sample's representation to augmentation, and shows the resulting scores can identify duplicates, outliers, and fairness-relevant examples in SSL models.
-
Capturing the Temporal Dependence of Training Data Influence
A new influence-estimation method approximates order-aware data influence with a dot product and reveals that early and late training data matter most.
-
Class-Proportional Coreset Selection for Difficulty-Separable Data
Class-proportional variants of difficulty-based coreset selection outperform class-agnostic methods on class-imbalanced security and medical datasets, particularly at 90-99.9% pruning rates.
-
QLESS: A Quantized Approach for Data Valuation and Selection in Large Language Model Fine-Tuning
QLESS quantizes the gradient features used in LESS data selection down to 1 bit without losing much fine-tuning quality, cutting gradient storage up to 16x across several LLM families.
-
Methods to Assess the UK Government's Current Role as a Data Provider for AI
Using unlearning-based ablation and information-leakage tests, the paper finds UK government websites matter for LLM performance on welfare queries while data.gov.uk datasets are not recalled.
Discussion (0). Continue with ORCID to comment.