Pith. sign in

REVIEW 45 cited by

On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.15126 v1 pith:W5IJQHMB submitted 2024-06-14 cs.CL

Lin Long , Rui Wang , Ruixuan Xiao , Junbo Zhao , Xiao Ding , Gang Chen , Haobo Wang This is my paper · ORCID
classification cs.CL
keywords datagenerationsyntheticllms-drivenacademicadventaimsalleviate
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Within the evolving landscape of deep learning, the dilemma of data quantity and quality has been a long-standing problem. The recent advent of Large Language Models (LLMs) offers a data-centric solution to alleviate the limitations of real-world data with synthetic data generation. However, current investigations into this field lack a unified framework and mostly stay on the surface. Therefore, this paper provides an organization of relevant studies based on a generic workflow of synthetic data generation. By doing so, we highlight the gaps within existing research and outline prospective avenues for future study. This work aims to shepherd the academic and industrial communities towards deeper, more methodical inquiries into the capabilities and applications of LLMs-driven synthetic data generation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 45 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Keywords: Evaluating Large Language Model Classification of Nuanced Ableism

    cs.CL 2025-05 conditional novelty 7.0 of 10

    LLMs identify autism-related words but frequently misclassify nuanced ableism, over-flagging intra-community language and under-flagging harmful stereotypes.

  2. Mind the Quote: Enabling Quotation-Aware Dialogue in LLMs via Plug-and-Play Modules

    cs.AI 2025-05 conditional novelty 6.5 of 10

    QuAda, a trainable attention adapter using under 2.8% extra parameters, gives instruction-tuned LLMs strong performance on five quotation-aware dialogue tasks.

  3. Evaluating Zero-Shot and One-Shot Adaptation of Small Language Models in Leader-Follower Interaction

    cs.HC 2026-02 conditional novelty 6.0 of 10

    Fine-tuned Qwen2.5-0.5B classifies leader-follower roles with 86.66% accuracy in single-turn interactions, but accuracy falls to chance in one-shot multi-turn interactions.

  4. Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoders

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Coverage of sparse-autoencoder-identified task features predicts post-training performance and can guide synthesis of small, high-impact datasets (2,000 vs. 300,000 samples).

  5. Ensembling LLM-Induced Decision Trees for Explainable and Robust Error Detection

    cs.CL 2025-12 conditional novelty 6.0 of 10

    LLM-induced hybrid decision trees (rules + trained graph checks) ensembled via EM detect erroneous table cells with an average 16.1-point F1 gain over the best baseline.

  6. Building Large-Scale English-Romanian Literary Translation Resources with Open Models

    cs.CL 2025-09 conditional novelty 6.0 of 10

    LoRA-fine-tuning a 12B open model on 15,000 GPT-o3 fable translations yields rubric scores close to GPT-o3 (4.83 vs 4.92) for English-to-Romanian literary translation at roughly one percent of the API cost.

  7. StaAgent: An Agentic Framework for Testing Static Analyzers

    cs.SE 2025-07 conditional novelty 6.0 of 10

    An LLM-powered four-agent framework performs metamorphic testing on static analyzers and reports 64 faulty rule implementations across SpotBugs, SonarQube, ErrorProne, Infer, and PMD.

  8. Multimodal Mathematical Reasoning with Diverse Solving Perspective

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Training a multimodal language model on multiple diverse solution paths per problem, plus rewards for distinguishing correct from incorrect solutions, improves math benchmark accuracy and output diversity.

  9. MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidance

    cs.CL 2025-06 conditional novelty 6.0 of 10

    MDBench is a synthetically generated, knowledge-guided benchmark for multi-document QA on which frontier LLMs achieve only about 60% exact match.

  10. RefEdit: A Benchmark and Method for Improving Instruction-based Image Editing Model on Referring Expressions

    cs.CV 2025-06 conditional novelty 6.0 of 10

    RefEdit-Bench measures referring-expression image editing; the RefEdit model, trained on 20K synthetic triplets, reports state-of-the-art results over million-scale baselines.

  11. Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMs

    cs.CL 2025-05 conditional novelty 6.0 of 10

    UCerF scores LLM fairness by both correctness and confidence, and SynthBias provides 31,756 gender-occupation coreference samples for benchmark testing.

  12. Self-Error-Instruct: Generalizing from Errors for LLMs Mathematical Reasoning

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Self-Error-Instruct clusters a model's math errors into types, synthesizes targeted practice data per type, and selects the best samples for fine-tuning, improving held-out math test accuracy.

  13. GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A knowledge-graph-guided method that scores an LLM's knowledge gaps and generates atomic, aggregated, and multi-hop QA pairs, improving closed-book QA after fine-tuning.

  14. JARVIS: A Multi-Agent Code Assistant for High-Quality EDA Script Generation

    cs.SE 2025-05 conditional novelty 6.0 of 10

    A multi-agent LLM framework with rule enforcement, compiler feedback, and retrieval achieves 92/93/81% pass@1 on three self-built EDA benchmarks, up from 67/62/43% for the best single model.

  15. A Design Space for the Critical Validation of LLM-Generated Tabular Data

    cs.HC 2025-05 conditional novelty 6.0 of 10

    A design space spanning analysis granularity and data source organizes existing approaches for critically validating LLM-generated tabular data and reveals unexplored combinations.

  16. A New HOPE: Domain-agnostic Automatic Evaluation of Text Chunking

    cs.CL 2025-05 conditional novelty 6.0 of 10

    HOPE is a domain-agnostic, automatic evaluation metric for text chunking that correlates with RAG performance and highlights semantic independence as the most important chunking property.

  17. An LLM-Empowered Low-Resolution Vision System for On-Device Human Behavior Understanding

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Llambda adapts LVLMs to low-resolution human-behavior videos by generating pseudo-label-guided captions from unlabeled data and fine-tuning with LoRA, reporting Bert-Score gains over LVLM baselines.

  18. Measuring Diversity in Synthetic Datasets

    cs.CL 2025-02 conditional novelty 6.0 of 10

    DCScore measures dataset diversity as the sum of self-classification probabilities under a softmax similarity matrix, and the paper shows it tracks generation temperature, human judgment, and LLM rankings.

  19. LLMs can be easily Confused by Instructional Distractions

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A new benchmark, DIM-Bench, shows that LLMs frequently follow instructions hidden inside the target input rather than the user's actual instruction, even when explicitly told to ignore them.

  20. Understanding the Effectiveness of LLMs in Automated Self-Admitted Technical Debt Repayment

    cs.SE 2025-01 conditional novelty 6.0 of 10

    Large new Python/Java benchmarks, diff-based metrics (BLEU-diff, CrystalBLEU-diff, LEMOD), and an LLM evaluation showing ~10% exact-match SATD repayment and larger models winning on fine-grained metrics.

  21. Mind the Gap: Towards Generalizable Autonomous Penetration Testing via Domain Randomization and Meta-Reinforcement Learning

    cs.LG 2024-12 conditional novelty 6.0 of 10

    GAP combines domain randomization via LLM-generated environments with meta-RL to improve generalization of autonomous pentesting agents across unseen host configurations and vulnerabilities.

  22. A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection

    cs.CL 2024-11 conditional novelty 6.0 of 10

    A synthetic-data-driven method trains lightweight classifiers to flag user prompts as off-topic relative to a system prompt, outperforming heuristics and partially transferring to jailbreak and harmful content.

  23. Can Structured Templates Facilitate LLMs in Tackling Harder Tasks? : An Exploration of Scaling Laws by Difficulty

    cs.AI 2025-08 reject novelty 5.0 of 10

    Training on easy synthetic math data lowers accuracy on hard benchmarks, and the proposed SST framework, which teaches explicit procedural chains, aims to reverse that drop.

  24. Separation Logic of Generic Resources via Sheafeology

    cs.LO 2025-08 unverdicted novelty 5.0 of 10

    Sheafeology uses sheaf categories to make first-order logic resource-aware, yielding separation logics for generic resources such as memory and random variables.

  25. Enterprise Large Language Model Evaluation Benchmark

    cs.AI 2025-06 reject novelty 5.0 of 10

    A 14-task enterprise LLM benchmark built mostly from GPT-4o-generated labels and scored by GPT-4o-as-judge shows open-source models closing the reasoning gap, but the dataset is not public and the evaluation is partly...

  26. A Survey of LLM $\times$ DATA

    cs.DB 2025-05 conditional novelty 5.0 of 10

    A comprehensive survey of the bidirectional links between LLMs and data management, organized as DATA4LLM and LLM4DATA with a new 'IaaS' data-quality framework.

  27. Large language model as user daily behavior data generator: balancing population diversity and individual personality

    cs.LG 2025-05 conditional novelty 5.0 of 10

    LLM-generated synthetic behavior data improves mobility and smartphone-use prediction models by up to 18.9% and captures roughly 60 to 88% of the gains from real-data fine-tuning.

  28. QBD-RankedDataGen: Generating Custom Ranked Datasets for Improving Query-By-Document Search Using LLM-Reranking with Reduced Human Effort

    cs.IR 2025-05 conditional novelty 5.0 of 10

    LLM-generated rankings for query-by-document search do not improve BM25 tuning over default parameters unless validated by human ground truth.

  29. Synthline: A Product Line Approach for Synthetic Requirements Engineering Data Generation using Large Language Models

    cs.SE 2025-05 conditional novelty 5.0 of 10

    A feature-model-driven LLM pipeline that generates synthetic requirements data improves defect classification when combined with real data, but the headline gains rest on a 40-sample test set with high variance.

  30. Less is More: Adaptive Coverage for Synthetic Training Data

    cs.LG 2025-04 conditional novelty 5.0 of 10

    A max-coverage graph algorithm with an adaptive similarity threshold selects 10-30% of synthetic data that trains classifiers as well as or better than the full dataset on sentiment, relation extraction, and NER tasks.

  31. Salamandra Technical Report

    cs.CL 2025-02 conditional novelty 5.0 of 10

    Salamandra is an open, from-scratch multilingual LLM family with 2B, 7B, and 40B checkpoints, instruction-tuned variants, a vision proof-of-concept, and detailed evaluations across Iberian and European languages.

  32. Few-shot LLM Synthetic Data with Distribution Matching

    cs.CL 2025-02 conditional novelty 5.0 of 10

    SynAlign generates LLM synthetic text from diversity-guided demonstrations, then reweights it by MMD-based distribution matching, improving downstream classification accuracy.

  33. OceanChat: The Effect of Virtual Conversational AI Agents on Sustainable Attitude and Behavior Change

    cs.HC 2025-02 conditional novelty 5.0 of 10

    Chatting with AI marine characters increased self-reported pro-environmental intentions and sustainable product choices compared to static information, but did not shift deeper climate beliefs.

  34. LLM Distillation for Efficient Few-Shot Multiple Choice Question Answering

    cs.CL 2024-12 conditional novelty 5.0 of 10

    Using Llama-3.1-8B to generate and score synthetic MCQA data, then distilling those soft labels into DeBERTa-v3-base, improves few-shot MMLU accuracy from 28.9% to 39.3%.

  35. Towards Data Governance of Frontier AI Models

    cs.AI 2024-12 conditional novelty 5.0 of 10

    Training data can serve as a governance lever for frontier AI through five proposed mechanisms: canary tokens, mandatory filtering, dataset reporting, data security, and know-your-customer rules.

  36. LLM for Barcodes: Generating Diverse Synthetic Data for Identity Documents

    cs.CL 2024-11 conditional novelty 5.0 of 10

    LLM-generated synthetic barcode data yields a 4.2-point mAP@0.5 improvement over Faker-based data for barcode detection in identity documents.

  37. A Penalty Goes a Long Way: Measuring Lexical Diversity in Synthetic Texts Under Prompt-Influenced Length Variations

    cs.CL 2025-07 conditional novelty 4.0 of 10

    PATTR adds a target-length penalty to the Type-Token Ratio, producing a lexical diversity score with tunable, reduced short-text bias for LLM synthetic data.

  38. Software Engineering for Large Language Models: Research Status, Challenges and the Road Ahead

    cs.SE 2025-06 conditional novelty 4.0 of 10

    A literature review organizes LLM development into a six-phase software engineering lifecycle and identifies challenges and research directions for each phase.

  39. Foundation Model Empowered Synesthesia of Machines (SoM): AI-native Intelligent Multi-Modal Sensing-Communication Integration

    eess.SP 2025-06 conditional novelty 4.0 of 10

    The paper proposes a systematic classification and two roadmaps for using foundation models (LLMs and wireless foundation models) to design Synesthesia of Machines systems for 6G, with preliminary case-study evidence ...

  40. Does Prompt Design Impact Quality of Data Imputation by LLMs?

    cs.LG 2025-06 conditional novelty 4.0 of 10

    Group-wise CSV prompts with correlation-based column pruning reduce LLM imputation prompt size while roughly maintaining or slightly improving classifier-based imputation quality on two imbalanced datasets.

  41. Tiny QA Benchmark++: Ultra-Lightweight, Synthetic Multilingual Dataset Generation & Smoke-Tests for Continuous LLM Evaluation

    cs.AI 2025-05 conditional novelty 4.0 of 10

    A lightweight multilingual QA smoke-test suite with a 52-item English core and an LLM-based generator is shown to reflect model size and language performance differences in seconds.

  42. Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models

    cs.LG 2024-12 conditional novelty 4.0 of 10

    This survey organizes LLM synthetic data research around quality, diversity, and complexity, claiming quality mainly helps in-distribution generalization, diversity mainly helps out-of-distribution generalization, and...

  43. Adaptable and Precise: Enterprise-Scenario LLM Function-Calling Capability Training Pipeline

    cs.AI 2024-12 reject novelty 3.0 of 10

    A LoRA fine-tuned 7B model trained on AI-synthesized enterprise HR API data beat GPT-4 and GPT-4o on the authors' private benchmark.

  44. A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarios

    cs.CL 2024-12 conditional novelty 3.0 of 10

    LLM-based game-playing agents are surveyed across choice-focused and communication-focused games, with a comparative performance table and future directions.

  45. Unlocking the Potential of Large Language Models in the Nuclear Industry with Synthetic Data

    cs.CL 2025-06 conditional novelty 2.0 of 10

    A pipeline converts CANDU textbook chapters into synthetic QA pairs using LLMs, embedding clustering, and similarity metrics, with no downstream validation.

Pith tools