REVIEW 38 cited by
Best Practices and Lessons Learned on Synthetic Data
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The success of AI models relies on the availability of large, diverse, and high-quality datasets, which can be challenging to obtain due to data scarcity, privacy concerns, and high costs. Synthetic data has emerged as a promising solution by generating artificial data that mimics real-world patterns. This paper provides an overview of synthetic data research, discussing its applications, challenges, and future directions. We present empirical evidence from prior art to demonstrate its effectiveness and highlight the importance of ensuring its factuality, fidelity, and unbiasedness. We emphasize the need for responsible use of synthetic data to build more powerful, inclusive, and trustworthy language models.
Forward citations
Cited by 38 Pith papers
-
SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models
SPaR trains LLMs by having them refine their own failed responses through tree search, producing preference pairs with minimal irrelevant variation, and lifts LLaMA3-8B above GPT-4-Turbo on IFEval.
-
DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data
A learned orchestrator builds per-example drop/untouch/clean pipelines over noise pruning and instruction-conditioned rewriting, improving from-scratch and math continued pretraining over fixed curation methods.
-
Search Hardness-Aware LLM-Based Problem Formulation for Expensive Simulation-Driven Design
SHA-PF uses initial simulation data to select a 'hard but promising' anchor satisfaction state and evolves LLM-generated formulations that prioritize it, reaching target designs with fewer expensive simulations on ant...
-
Synthetic Interaction Data for Scalable Personalization in Large Language Models
PersonaGym simulates noisy multi-turn user–assistant interactions to build PersonaAtlas, and PPOpt learns to rewrite user prompts from interaction history, improving judged personalization on synthetic benchmarks.
-
CultureSynth: A Hierarchical Taxonomy-Guided and Retrieval-Augmented Framework for Cultural Question-Answer Synthesis
A taxonomy-guided retrieval-augmented framework generates CultureSynth-7, a multilingual cultural QA benchmark, and its evaluation of 14 LLMs suggests cultural competence emerges around 3B parameters.
-
Synthetic CVs To Build and Test Fairness-Aware Hiring Tools
A new synthetic CV dataset, generated from donated real CVs, is proposed as a benchmark for fairness-aware algorithmic hiring research.
-
MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidance
MDBench is a synthetically generated, knowledge-guided benchmark for multi-document QA on which frontier LLMs achieve only about 60% exact match.
-
StoryBench: A Dynamic Benchmark for Evaluating Long-Term Memory with Multi Turns
StoryBench introduces a branching interactive-fiction benchmark with immediate-feedback and self-recovery modes, and shows that current LLMs fail at long-term memory tasks, especially self-correction.
-
Seed-Coder: Let the Code Model Curate Data for Itself
Seed-Coder shows that an LLM-trained quality scorer can filter 6T tokens of code data and yield 8B models that outperform similar-size open code models.
-
EasyDistill: A Comprehensive Toolkit for Effective Knowledge Distillation of Large Language Models
EasyDistill packages established LLM knowledge-distillation techniques into a single modular toolkit with released distilled models, datasets, and Alibaba Cloud integration.
-
GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation
A knowledge-graph-guided method that scores an LLM's knowledge gaps and generates atomic, aggregated, and multi-hop QA pairs, improving closed-book QA after fine-tuning.
-
OrgAccess: A Benchmark for Role Based Access Control in Organization Scale LLMs
OrgAccess, a 70k-query synthetic RBAC benchmark, shows current LLMs including GPT-4.1 (F1 0.27 on the hardest split) struggle badly with permission adherence.
-
BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation
BARE generates diverse, high-quality synthetic training data from only three seed examples by having a base model draft and an instruction-tuned model refine, improving downstream fine-tuning accuracy in few-shot settings.
-
Examining the Expanding Role of Synthetic Data Throughout the AI Development Pipeline
Twenty-nine interviews show AI practitioners rely on synthetic data across nearly every pipeline stage while validation remains mostly manual spot-checking.
-
Aligning Instruction Tuning with Pre-training
AITP selects pre-training texts that are underrepresented relative to instruction-tuning data, rewrites them into instruction-response pairs, and improves average SFT performance on three open LLMs.
-
How to Synthesize Text Data without Model Collapse?
Token-level editing of human text with a high-confidence threshold yields modest performance gains over the original data and avoids the sharp degradation seen with purely synthetic data.
-
Bridging the Data Provenance Gap Across Text, Speech and Video
A manual audit of nearly 4,000 text, speech, and video datasets finds AI training data increasingly comes from web and social media sources, carries hidden non-commercial restrictions, and remains Western-centric with...
-
On Domain-Adaptive Post-Training for Multimodal Large Language Models
A generate-then-filter, open-source-only synthesis pipeline plus single-stage post-training consistently improves MLLM performance across biomedicine, food, and remote sensing.
-
A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection
A synthetic-data-driven method trains lightweight classifiers to flag user prompts as off-topic relative to a system prompt, outperforming heuristics and partially transferring to jailbreak and harmful content.
-
Enterprise Large Language Model Evaluation Benchmark
A 14-task enterprise LLM benchmark built mostly from GPT-4o-generated labels and scored by GPT-4o-as-judge shows open-source models closing the reasoning gap, but the dataset is not public and the evaluation is partly...
-
Using Sign Language Production as Data Augmentation to enhance Sign Language Translation
Adding synthetic sign-language data produced by stitching, a GAN, or Gaussian splatting to the training set improves sign-language translation, with the largest gains for skeleton-pose models.
-
Two Experts Are All You Need for Steering Thinking: Reinforcing Cognitive Effort in MoE Reasoning Models Without Additional Training
Reinforcing the two experts most correlated with thinking tokens improves reasoning accuracy and efficiency in MoE large reasoning models, with gains of up to 10 points on AIME benchmarks.
-
Generative AI at the Crossroads: Light Bulb, Dynamo, or Microscope?
GenAI is likely both a general-purpose technology and an invention of a method of invention, supporting a future rise in productivity levels with uncertain growth effects.
-
Even Small Reasoners Should Quote Their Sources: Introducing the Pleias-RAG Model Family
Two mid-trained small language models (350M and 1B) are claimed to achieve state-of-the-art RAG accuracy in their size class while generating native literal-quote citations.
-
Salamandra Technical Report
Salamandra is an open, from-scratch multilingual LLM family with 2B, 7B, and 40B checkpoints, instruction-tuned variants, a vision proof-of-concept, and detailed evaluations across Iberian and European languages.
-
Engineering AI Judge Systems
A constitution-based, search-driven development framework for AI judge systems improves judged accuracy by up to 6.2% on commit message generation, with about 58% of general principles reused across five languages.
-
A Penalty Goes a Long Way: Measuring Lexical Diversity in Synthetic Texts Under Prompt-Influenced Length Variations
PATTR adds a target-length penalty to the Type-Token Ratio, producing a lexical diversity score with tunable, reduced short-text bias for LLM synthetic data.
-
A Comparative Study of Open-Source Libraries for Synthetic Tabular Data Generation: SDV vs. SynthCity
On one energy-consumption dataset, Synthcity's Bayesian Network had the highest statistical fidelity and SDV's TVAE the best predictive utility at 1:10 scale, with no clear library winner.
-
Does Prompt Design Impact Quality of Data Imputation by LLMs?
Group-wise CSV prompts with correlation-based column pruning reduce LLM imputation prompt size while roughly maintaining or slightly improving classifier-based imputation quality on two imbalanced datasets.
-
The Aloe Family Recipe for Open and Specialized Healthcare LLMs
Aloe Beta, a family of open-weights health LLMs built from Llama 3.1 and Qwen 2.5, matches or exceeds closed medical models on MCQA benchmarks while improving safety via DPO.
-
Guided Persona-based AI Surveys: Can we replicate personal mobility preferences at scale using LLMs?
A persona-based LLM survey generator given MiD 2017 response statistics reproduces those statistics almost exactly, but the evaluation is circular and the main claim is unsupported.
-
Powering LLM Regulation through Data: Bridging the Gap from Compute Thresholds to Customer Experiences
LLM products should be certified on curated, domain-specific datasets rather than regulated through compute thresholds or general benchmarks.
-
Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models
This survey organizes LLM synthetic data research around quality, diversity, and complexity, claiming quality mainly helps in-distribution generalization, diversity mainly helps out-of-distribution generalization, and...
-
The Hitchhikers Guide to Production-ready Trustworthy Foundation Model powered Software (FMware)
The paper catalogs the lifecycle stages and production-readiness challenges of software built around foundation models (FMware) and proposes an action plan of engineering practices and research directions.
-
MARCO: Multi-Agent Code Optimization with Real-Time Knowledge Integration for High-Performance Computing
MARCO reports that a multi-agent LLM loop with web search improves LeetCode Python runtime by 14.6% over Claude 3.5 Sonnet, but the evidence omits code, error bars, and HPC workloads.
-
SafeSynthDP: Leveraging Large Language Models for Privacy-Preserving Synthetic Data Generation Using Differential Privacy
A study of an LLM-based synthetic data pipeline with noise injection that claims differential privacy, evaluated on news classification, but with no valid privacy analysis.
-
Unlocking the Potential of Large Language Models in the Nuclear Industry with Synthetic Data
A pipeline converts CANDU textbook chapters into synthetic QA pairs using LLMs, embedding clustering, and similarity metrics, with no downstream validation.
-
Sycophancy in Large Language Models: Causes and Mitigations
A survey of sycophancy in LLMs that organizes measurement, causes, and mitigations, but its citation errors and lack of original evidence make it unreliable.
Discussion (0). Continue with ORCID to comment.