Pith. sign in

REVIEW 25 cited by

Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.13064 v1 pith:Z5UPBIAK submitted 2024-02-20 cs.CL

Haoran Li , Qingxiu Dong , Zhengyang Tang , Chaojun Wang , Xingxing Zhang , Haoyang Huang , Shaohan Huang , Xiaolong Huang
show 12 more authors
Zeqiang Huang Dongdong Zhang Yuxian Gu Xin Cheng Xun Wang Si-Qing Chen Li Dong Wei Lu Zhifang Sui Benyou Wang Wai Lam Furu Wei
This is my paper · ORCID
classification cs.CL
keywords instructiondataglanhumantuningknowledgelanguagellms
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We introduce Generalized Instruction Tuning (called GLAN), a general and scalable method for instruction tuning of Large Language Models (LLMs). Unlike prior work that relies on seed examples or existing datasets to construct instruction tuning data, GLAN exclusively utilizes a pre-curated taxonomy of human knowledge and capabilities as input and generates large-scale synthetic instruction data across all disciplines. Specifically, inspired by the systematic structure in human education system, we build the taxonomy by decomposing human knowledge and capabilities to various fields, sub-fields and ultimately, distinct disciplines semi-automatically, facilitated by LLMs. Subsequently, we generate a comprehensive list of subjects for every discipline and proceed to design a syllabus tailored to each subject, again utilizing LLMs. With the fine-grained key concepts detailed in every class session of the syllabus, we are able to generate diverse instructions with a broad coverage across the entire spectrum of human knowledge and skills. Extensive experiments on large language models (e.g., Mistral) demonstrate that GLAN excels in multiple dimensions from mathematical reasoning, coding, academic exams, logical reasoning to general instruction following without using task-specific training data of these tasks. In addition, GLAN allows for easy customization and new fields or skills can be added by simply incorporating a new node into our taxonomy.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 25 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Efficient and Effective Alignment of Large Language Models

    cs.CL 2025-06 conditional novelty 7.0 of 10

    A thesis presenting Lion, WebR, LTE, BMC, and FollowBench, five empirical methods that together address LLM alignment data, training, and evaluation.

  2. HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs

    cs.CL 2026-07 conditional novelty 6.0 of 10

    HSS-Synth generates 230k instruction-tuning samples for 14 humanities/social-science fields and reports state-of-the-art fine-tuning results on 16 benchmarks.

  3. Search Hardness-Aware LLM-Based Problem Formulation for Expensive Simulation-Driven Design

    cs.NE 2026-07 conditional novelty 6.0 of 10

    SHA-PF uses initial simulation data to select a 'hard but promising' anchor satisfaction state and evolves LLM-generated formulations that prioritize it, reaching target designs with fewer expensive simulations on ant...

  4. NexForge: Scaling Agent Capabilities through Requirement-Driven Task Synthesis for LLMs

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Requirement-driven synthesis generates executable agent training tasks and trajectories from a capability description, improving a 35B model from 22.5% to 52.0% on Terminal-Bench 2.0 and from 813 to 1338 GDPval Elo.

  5. GLAN-QnA-KR: A Seedless Taxonomy-Driven Korean Instruction Corpus

    cs.CL 2026-05 conditional novelty 6.0 of 10

    A 303,581-row Korean instruction corpus generated seedlessly from a 1,084-discipline taxonomy, with near-zero duplicates and low measured overlap with KMMLU, KoBEST, and HAE-RAE-Bench.

  6. Icon$^{2}$: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulation

    cs.CL 2025-09 conditional novelty 6.0 of 10

    ICON2 uses representation-space steering to generate preference data from the model itself, improving alignment benchmarks and cutting cost.

  7. Not All Documents Are What You Need for Extracting Instruction Tuning Data

    cs.CL 2025-05 conditional novelty 6.0 of 10

    EQUAL iteratively selects document clusters via multi-armed bandit and optimal transport scores to extract only the instruction-tuning QA pairs that matter, saving 5-10x cost with small accuracy gains.

  8. Instruction-Tuning Data Synthesis from Scratch via Web Reconstruction

    cs.CL 2025-04 conditional novelty 6.0 of 10

    WebR generates instruction-tuning data by reconstructing raw web documents from two perspectives, as instructions and as responses, and training with it outperforms previous fully automated and human-labeled datasets ...

  9. BitNet b1.58 2B4T Technical Report

    cs.CL 2025-04 conditional novelty 6.0 of 10

    A natively 1.58-bit, 2B-parameter model trained on 4T tokens roughly matches 1-2B full-precision open LLMs on average across 16 benchmarks while using far less memory and energy.

  10. Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A large synthetic instruction corpus with guidelines, preference rules, and format variants improves LLM performance on five NLU benchmarks by an average of 3.1%.

  11. BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation

    cs.CL 2025-02 conditional novelty 6.0 of 10

    BARE generates diverse, high-quality synthetic training data from only three seed examples by having a base model draft and an instruction-tuned model refine, improving downstream fine-tuning accuracy in few-shot settings.

  12. CDS: Knowledge Component-Driven Data Synthesis Guided by Cognitive Diagnosis Theory

    cs.AI 2025-01 conditional novelty 6.0 of 10

    A knowledge-component diagnostic pipeline, inspired by cognitive diagnosis theory, generates weakness-targeted synthetic data that improves small LLMs on math, code, and exam benchmarks by up to 13.1 percentage points.

  13. Dynamic Skill Adaptation for Large Language Models

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A training pipeline that orders generated textbook and exercise data by a skill dependency graph and dynamically updates the data during fine-tuning improves LLM performance on calculus and social studies evaluations.

  14. Error-driven Data-efficient Large Multimodal Model Tuning

    cs.CL 2024-12 conditional novelty 6.0 of 10

    An error-driven teacher-student pipeline extracts a student LMM's missing skills from validation mistakes and retrieves targeted samples from a task-agnostic dataset to fine-tune it.

  15. Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A controlled, multi-family study shows the relative generation-verification gap grows with pretraining flops for stable verification methods, and iterative self-improvement saturates quickly.

  16. On Domain-Adaptive Post-Training for Multimodal Large Language Models

    cs.CL 2024-11 conditional novelty 6.0 of 10

    A generate-then-filter, open-source-only synthesis pipeline plus single-stage post-training consistently improves MLLM performance across biomedicine, food, and remote sensing.

  17. ChemActor: Enhancing Automated Extraction of Chemical Synthesis Actions with LLM-Generated Data

    cs.AI 2025-06 conditional novelty 5.0 of 10

    A fine-tuned LLaMA-2-7B model trained with selected LLM-generated data improves extraction of chemical synthesis actions from experimental text.

  18. A Survey of LLM $\times$ DATA

    cs.DB 2025-05 conditional novelty 5.0 of 10

    A comprehensive survey of the bidirectional links between LLMs and data management, organized as DATA4LLM and LLM4DATA with a new 'IaaS' data-quality framework.

  19. From Tens of Hours to Tens of Thousands: Scaling Back-Translation for Speech Recognition

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Fine-tuning TTS models on tens of hours of real audio enables generation of 500,000 hours of synthetic speech that reduces ASR error rates by over 30% on Whisper-large-v3.

  20. LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions

    cs.CL 2025-05 conditional novelty 5.0 of 10

    By prompting an aligned LLM with a document and the special token that precedes a user query, LongMagpie synthesizes long-context instruction data that outperforms prior datasets when used to fine-tune Llama-3-8B.

  21. Two Experts Are All You Need for Steering Thinking: Reinforcing Cognitive Effort in MoE Reasoning Models Without Additional Training

    cs.AI 2025-05 conditional novelty 5.0 of 10

    Reinforcing the two experts most correlated with thinking tokens improves reasoning accuracy and efficiency in MoE large reasoning models, with gains of up to 10 points on AIME benchmarks.

  22. OpenCharacter: Training Customizable Role-Playing LLMs with Large-Scale Synthetic Personas

    cs.CL 2025-01 conditional novelty 5.0 of 10

    Using 20,000 LLM-generated personas and 306k synthetic dialogues, supervised fine-tuning gives an 8B model role-playing performance comparable to GPT-4o.

  23. Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs

    cs.LG 2024-12 conditional novelty 5.0 of 10

    For 3B-7B LLMs, larger batch sizes with lower learning rates improve instruction-tuning benchmarks, early gradient and loss signals predict final quality, and stacked training matches phased training with fewer samples.

  24. A Penalty Goes a Long Way: Measuring Lexical Diversity in Synthetic Texts Under Prompt-Influenced Length Variations

    cs.CL 2025-07 conditional novelty 4.0 of 10

    PATTR adds a target-length penalty to the Type-Token Ratio, producing a lexical diversity score with tunable, reduced short-text bias for LLM synthetic data.

  25. SyNeg: LLM-Driven Synthetic Hard-Negatives for Dense Retrieval

    cs.IR 2024-12 conditional novelty 4.0 of 10

    LLM-generated synthetic hard negatives, combined with retrieved negatives in a hybrid mix, improve dense retrieval accuracy on BEIR benchmarks.

Pith tools