Pith. sign in

REVIEW 21 cited by

What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.15685 v2 pith:KIEUJ2E4 submitted 2023-12-25 cs.CL cs.AIcs.LG

Wei Liu , Weihao Zeng , Keqing He , Yong Jiang , Junxian He This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords dataalignmentmodelsinstructiontuningsamplesautomaticselected
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Instruction tuning is a standard technique employed to align large language models to end tasks and user preferences after the initial pretraining phase. Recent research indicates the critical role of data engineering in instruction tuning -- when appropriately selected, only limited data is necessary to achieve superior performance. However, we still lack a principled understanding of what makes good instruction tuning data for alignment, and how we should select data automatically and effectively. In this work, we delve deeply into automatic data selection strategies for alignment. We start with controlled studies to measure data across three dimensions: complexity, quality, and diversity, along which we examine existing methods and introduce novel techniques for enhanced data measurement. Subsequently, we propose a simple strategy to select data samples based on the measurement. We present deita (short for Data-Efficient Instruction Tuning for Alignment), a series of models fine-tuned from LLaMA and Mistral models using data samples automatically selected with our proposed approach. Empirically, deita performs better or on par with the state-of-the-art open-source alignment models with only 6K SFT training data samples -- over 10x less than the data used in the baselines. When further trained with direct preference optimization (DPO), deita-Mistral-7B + DPO trained with 6K SFT and 10K DPO samples achieve 7.55 MT-Bench and 90.06% AlpacaEval scores. We anticipate this work to provide tools on automatic data selection, facilitating data-efficient alignment. We release our models as well as the selected datasets for future researches to effectively align models more efficiently.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 21 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Training Large Language Models for Self-Explanation Faithfulness

    cs.LG 2026-07 conditional novelty 6.0 of 10

    RL fine-tuning with a counterfactual mention/influence reward raises LLM self-explanation faithfulness (Phi-CCT) from near zero to ~0.66 in-distribution for two 8B models, with partial transfer to held-out tasks.

  2. SFGA: A Statistics-First Gating Architecture with Adjudicative Escalation for Trustworthy SFT Data Procurement

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A cost-aware gate that routes simple SFT procurement cases to cheap statistics and hard cases to an adjudicated LLM debate reports 0.90 accuracy on a synthetic closed-loop benchmark.

  3. What makes Reasoning Models Different? Follow the Reasoning Leader for Efficient Decoding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    FoReaL-Decoding lets a strong reasoning model generate the first few tokens of each sentence and a weaker model complete the sentence, cutting theoretical FLOPs by 30-55% while retaining 86-100% of accuracy on four ma...

  4. SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesis

    cs.LG 2025-06 conditional novelty 6.0 of 10

    SynthRL synthesizes harder, answer-preserving visual math questions from easy seed questions and reports small but mixed out-of-domain RLVR gains for Qwen2.5-VL-7B.

  5. Tag-Evol: Achieving Efficient Instruction Evolving via Tag Injection

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Tag-Evol generates harder, more diverse instruction data by injecting sampled knowledge tags into seed instructions, improving downstream SFT accuracy across math, code, and general benchmarks.

  6. R.I.P.: Better Models by Survival of the Fittest Prompts

    cs.CL 2025-01 conditional novelty 6.0 of 10

    RIP filters preference-optimization training data by keeping prompts whose rejected responses are high quality and whose chosen/rejected reward gap is small, yielding consistent benchmark gains over unfiltered data.

  7. Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilities

    cs.CL 2025-01 conditional novelty 6.0 of 10

    BIDS, a balanced influence-based data selection algorithm using per-task normalization and iterative greedy selection, improves balanced multi-capability instruction tuning and can outperform full-dataset training on ...

  8. Understanding the Logic of Direct Preference Alignment through Logic

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Direct preference alignment losses can be expressed as logical programs over model predictions, yielding an organized landscape of billions of definable losses and a route to new variants.

  9. ResoFilter: Fine-grained Synthetic Data Filtering for Large Language Models through Data-Parameter Resonance Analysis

    cs.CL 2024-12 conditional novelty 6.0 of 10

    ResoFilter keeps fine-tuning examples that produce small parameter updates in the last layers, matching full-data fine-tuning on GSM8k with 50% of the math data.

  10. Preference-Oriented Supervised Fine-Tuning: Favoring Target Model Over Aligned Large Language Models

    cs.CL 2024-12 conditional novelty 6.0 of 10

    PoFT is a new supervised fine-tuning loss that makes the target model outrank aligned reference LLMs on the same data, improving robustness and slightly raising benchmark scores.

  11. Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning

    cs.CL 2024-11 conditional novelty 6.0 of 10

    A multi-agent LLM framework that generates, selects, and evolves instruction-tuning data improves downstream instruction-following by about 12% over Evol-Instruct baselines.

  12. Less Data, Faster Convergence: Goal-Driven Data Optimization for Multimodal Instruction Tuning

    cs.CV 2026-03 conditional novelty 5.5 of 10

    Goal-driven selection of 1× multimodal instruction subsets reaches a 512k Uni-10x baseline after ~27–35k samples and improves accuracy by up to +3.08 pp under a fixed Qwen3-VL recipe.

  13. Active Domain Knowledge Acquisition with 100-Dollar Budget: Enhancing LLMs via Cost-Efficient, Expert-Involved Interaction in Sensitive Domains

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    A budget-aware framework (PU-ADKA) selects which domain expert an LLM should query under a fixed $100 budget, improving specialized-domain answers at low cost.

  14. MINT: Multimodal Instruction Tuning with Multimodal Interaction Grouping

    cs.LG 2025-06 conditional novelty 5.0 of 10

    Grouping instruction-tuning datasets by redundancy, uniqueness, or synergy of text-image interaction improves vision-language model accuracy over single-task and unselective multi-task tuning.

  15. A Survey of LLM $\times$ DATA

    cs.DB 2025-05 conditional novelty 5.0 of 10

    A comprehensive survey of the bidirectional links between LLMs and data management, organized as DATA4LLM and LLM4DATA with a new 'IaaS' data-quality framework.

  16. Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs

    cs.LG 2024-12 conditional novelty 5.0 of 10

    For 3B-7B LLMs, larger batch sizes with lower learning rates improve instruction-tuning benchmarks, early gradient and loss signals predict final quality, and stacked training matches phased training with fewer samples.

  17. BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment

    cs.CL 2024-11 conditional novelty 5.0 of 10

    BPO balances knowledge breadth and depth in preference data by compressing prompts and dynamically augmenting the number of response pairs per prompt using gradient-based clustering, achieving stronger alignment with ...

  18. Efficient Alignment of Large Language Models via Data Sampling

    cs.LG 2024-11 reject novelty 5.0 of 10

    Alignment performance plateaus quickly with data, and a GMM-plus-entropy sampling method (ISA) can allegedly match full-dataset alignment with under 10% of the data, though the supporting evidence is weak.

  19. Data Diversification Methods In Alignment Enhance Math Performance In LLMs

    cs.AI 2025-07 reject novelty 4.0 of 10

    DTS, which generates diverse solution strategies before writing solutions, improves GSM8K by 7.1 points and MATH by 4.2 points over an untuned base model at 1.03x baseline compute.

  20. Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models

    cs.LG 2024-12 conditional novelty 4.0 of 10

    This survey organizes LLM synthetic data research around quality, diversity, and complexity, claiming quality mainly helps in-distribution generalization, diversity mainly helps out-of-distribution generalization, and...

  21. Optimising Language Models for Downstream Tasks: A Post-Training Perspective

    cs.CL 2025-06 conditional novelty 3.0 of 10

    A dissertation that repackages the author's previously published papers on continued pre-training, prompt tuning, and instruction modelling into a single narrative.

Pith tools