Pith. sign in

REVIEW 10 cited by

What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.13954 v1 pith:TDJXDWZL submitted 2024-05-22 cs.LG cs.AIcs.CL

Sang Keun Choe , Hwijeen Ahn , Juhan Bae , Kewen Zhao , Minsoo Kang , Youngseog Chung , Adithya Pratapa , Willie Neiswanger
show 6 more authors
Emma Strubell Teruko Mitamura Jeff Schneider Eduard Hovy Roger Grosse Eric Xing
This is my paper
classification cs.LGcs.AIcs.CL
keywords datavaluationfunctionsgradientinfluencebeencodeexisting
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) are trained on a vast amount of human-written data, but data providers often remain uncredited. In response to this issue, data valuation (or data attribution), which quantifies the contribution or value of each data to the model output, has been discussed as a potential solution. Nevertheless, applying existing data valuation methods to recent LLMs and their vast training datasets has been largely limited by prohibitive compute and memory costs. In this work, we focus on influence functions, a popular gradient-based data valuation method, and significantly improve its scalability with an efficient gradient projection strategy called LoGra that leverages the gradient structure in backpropagation. We then provide a theoretical motivation of gradient projection approaches to influence functions to promote trust in the data valuation process. Lastly, we lower the barrier to implementing data valuation systems by introducing LogIX, a software package that can transform existing training code into data valuation code with minimal effort. In our data valuation experiments, LoGra achieves competitive accuracy against more expensive baselines while showing up to 6,500x improvement in throughput and 5x reduction in GPU memory usage when applied to Llama3-8B-Instruct and the 1B-token dataset.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sketching the Readout of Large Language Models for Scalable Data Attribution and Valuation

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    RISE applies CountSketch to dual lexical and semantic channels derived from output-layer gradient outer products, cutting data attribution storage by up to 112x and enabling retrospective and prospective influence ana...

  2. LLM generation novelty through the lens of semantic similarity

    cs.LG 2025-10 conditional novelty 6.0 of 10

    An output is called novel when no semantically similar pretraining context can be retrieved; using this embedding-based measure, SmolLM models show longer-range reuse and instruction tuning increases novelty.

  3. Understanding Data Influence with Differential Approximation

    cs.LG 2025-08 conditional novelty 6.0 of 10

    This paper introduces Diff-In, an influence estimator that accumulates second-order approximations of influence differences across training steps and shows strong accuracy in data cleaning, deletion, and coreset selec...

  4. Better Training Data Attribution via Better Inverse Hessian-Vector Products

    cs.LG 2025-07 conditional novelty 6.0 of 10

    ASTRA, an EKFAC-preconditioned Neumann series iteration, computes more accurate inverse Hessian-vector products and improves training data attribution scores over EKFAC baselines.

  5. The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A new 8TB openly-licensed text corpus trains 7B LLMs that are competitive with Llama 1/2, showing that performant models need not depend on unlicensed web data.

  6. MedPAIR: Measuring Physicians and AI Relevance Alignment in Medical Question Answering

    cs.CL 2025-05 conditional novelty 6.0 of 10

    LLM relevance judgments align with physician trainees on only about 45 to 66 percent of sentences, and pruning contexts to physician-labeled relevant sentences improves LLM and trainee accuracy.

  7. Daunce: Data Attribution through Uncertainty Estimation

    cs.LG 2025-05 conditional novelty 6.0 of 10

    DAUNCE computes training-data attribution as the covariance of per-example losses across an ensemble of perturbed fine-tuned models, reporting state-of-the-art LDS scores and the first attribution runs on proprietary LLMs.

  8. KAIROS: Scalable Model-Agnostic Data Valuation

    cs.LG 2025-06 conditional novelty 5.0 of 10

    KAIROS derives a closed-form Maximum Mean Discrepancy influence score that approximates leave-one-out data rankings and detects noise, mislabels, and backdoors without retraining.

  9. Counterfactual Influence as a Distributional Quantity

    cs.LG 2025-06 conditional novelty 5.0 of 10

    Near-duplicate training samples lower a model's self-influence on a record while raising its extractability, so self-influence alone underestimates memorization risk.

  10. Class-Proportional Coreset Selection for Difficulty-Separable Data

    cs.LG 2025-07 conditional novelty 4.0 of 10

    Class-proportional variants of difficulty-based coreset selection outperform class-agnostic methods on class-imbalanced security and medical datasets, particularly at 90-99.9% pruning rates.

Pith tools