Pith. sign in

REVIEW 17 cited by

Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.20541 v1 pith:QLELZKGR submitted 2024-05-30 cs.LG cs.CL

Zachary Ankner , Cody Blakeney , Kartik Sreenivasan , Max Marion , Matthew L. Leavitt , Mansheej Paul This is my paper · ORCID
classification cs.LGcs.CL
keywords pruningdataperformancemodelsperplexity-baseddownstreammodeldemonstrate
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

In this work, we investigate whether small language models can determine high-quality subsets of large-scale text datasets that improve the performance of larger language models. While existing work has shown that pruning based on the perplexity of a larger model can yield high-quality data, we investigate whether smaller models can be used for perplexity-based pruning and how pruning is affected by the domain composition of the data being pruned. We demonstrate that for multiple dataset compositions, perplexity-based pruning of pretraining data can \emph{significantly} improve downstream task performance: pruning based on perplexities computed with a 125 million parameter model improves the average performance on downstream tasks of a 3 billion parameter model by up to 2.04 and achieves up to a $1.45\times$ reduction in pretraining steps to reach commensurate baseline performance. Furthermore, we demonstrate that such perplexity-based data pruning also yields downstream performance gains in the over-trained and data-constrained regimes.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bridging Compute- and Data-Optimal Pretraining

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Pretraining loss obeys a single law in which repeated or paraphrased tokens count as η(N, data-per-parameter, expansion-ratio) fresh tokens, with total effective data saturating as derived tokens grow.

  2. Towards Efficient and Effective Alignment of Large Language Models

    cs.CL 2025-06 conditional novelty 7.0 of 10

    A thesis presenting Lion, WebR, LTE, BMC, and FollowBench, five empirical methods that together address LLM alignment data, training, and evaluation.

  3. A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Only gradient-based (LESS) representations make subset-to-query distance a reliable predictor of instruction-tuning performance; greedy round-robin helps most at small budgets, and random selection is surprisingly com...

  4. Language Models Improve When Pretraining Data Matches Target Tasks

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Ranking pretraining documents by similarity to benchmark training examples (BETR) yields consistent benchmark gains and a 2.1x compute multiplier over DCLM-Baseline.

  5. Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Training

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Domain2Vec finds better LLM pretraining data mixtures by aligning, in a training-free way, the meta-domain distribution of the training set with the validation set's distribution.

  6. AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection

    cs.CL 2025-05 conditional novelty 6.0 of 10

    AttentionInfluence uses retrieval-head masking loss differences in a 1.3B model to select 73B tokens that improve a 7B model's reasoning benchmarks by 1.4 to 3.5pp.

  7. Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Using a pretrained reference model inside a distributionally robust risk objective can improve generalization bounds and yields a CLIP variant that matches baseline performance with half the data.

  8. Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples

    cs.CL 2025-02 conditional novelty 6.0 of 10

    Preference examples vary in difficulty; overly difficult examples degrade DPO alignment, and filtering them out improves AlpacaEval 2 win rates by 9-16 percentage points.

  9. How to Synthesize Text Data without Model Collapse?

    cs.CL 2024-12 reject novelty 6.0 of 10

    Token-level editing of human text with a high-confidence threshold yields modest performance gains over the original data and avoids the sharp degradation seen with purely synthetic data.

  10. Weak-to-Strong Generalization Through the Data-Centric Lens

    cs.LG 2024-12 conditional novelty 6.0 of 10

    Weak-to-strong generalization is driven by data points that contain both easy and hard patterns, and such points can be detected and actively sourced.

  11. ADMIRE-BayesOpt: Accelerated Data MIxture RE-weighting for Language Models with Bayesian Optimization

    stat.ML 2025-08 conditional novelty 5.0 of 10

    Using Bayesian optimization over Gaussian-process surrogates, data mixtures for LLM training can be found much faster than with linear or exponential regression baselines, including across model sizes.

  12. MagicVL-2B: Empowering Vision-Language Models on Mobile Devices with Lightweight Visual Encoders via Curriculum Learning

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    MagicVL-2B is a 2B vision-language model for mobile phones that claims state-of-the-art-matching accuracy at 41.1% lower on-device power, via a lightweight encoder, dynamic resolution, and curriculum learning.

  13. Efficient Data Selection at Scale via Influence Distillation

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Influence Distillation selects LLM fine-tuning data by approximating each sample's gradient influence on a target task via landmarks and JVP embeddings, matching or beating RDS+ accuracy at roughly one third the selec...

  14. A Survey of LLM $\times$ DATA

    cs.DB 2025-05 conditional novelty 5.0 of 10

    A comprehensive survey of the bidirectional links between LLMs and data management, organized as DATA4LLM and LLM4DATA with a new 'IaaS' data-quality framework.

  15. Improving Chemical Understanding of LLMs via SMILES Parsing

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Pretraining LLMs on deterministic SMILES parsing tasks improves molecular structural understanding and downstream chemistry performance.

  16. Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativeness

    cs.CV 2024-12 conditional novelty 5.0 of 10

    Selecting 15% of visual instruction data with DataTailor's three-principle scores matches or slightly beats full-data fine-tuning on multimodal benchmarks.

  17. Efficient Code LLM Training via Distribution-Consistent and Diversity-Aware Data Selection

    cs.CL 2025-07 conditional novelty 4.0 of 10

    Selecting 10K code instructions with a parametric feature-space model lifts DeepSeekCoder-Base-6.7B from 67.1% to 69.5% on HumanEval and from 74.9% to 77.2% on MBPP versus full 92K training, in single-run evaluations.

Pith tools