Pith. sign in

REVIEW 8 cited by

Datasets for Large Language Models: A Comprehensive Survey

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.18041 v1 pith:NVJCIWAL submitted 2024-02-28 cs.CL cs.AI

Yang Liu , Jiahuan Cao , Chongyu Liu , Kai Ding , Lianwen Jin This is my paper
classification cs.CLcs.AI
keywords datasetscomprehensivelanguagefuturesurveyavailablecorporacurrent
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

This paper embarks on an exploration into the Large Language Model (LLM) datasets, which play a crucial role in the remarkable advancements of LLMs. The datasets serve as the foundational infrastructure analogous to a root system that sustains and nurtures the development of LLMs. Consequently, examination of these datasets emerges as a critical topic in research. In order to address the current lack of a comprehensive overview and thorough analysis of LLM datasets, and to gain insights into their current status and future trends, this survey consolidates and categorizes the fundamental aspects of LLM datasets from five perspectives: (1) Pre-training Corpora; (2) Instruction Fine-tuning Datasets; (3) Preference Datasets; (4) Evaluation Datasets; (5) Traditional Natural Language Processing (NLP) Datasets. The survey sheds light on the prevailing challenges and points out potential avenues for future investigation. Additionally, a comprehensive review of the existing available dataset resources is also provided, including statistics from 444 datasets, covering 8 language categories and spanning 32 domains. Information from 20 dimensions is incorporated into the dataset statistics. The total data size surveyed surpasses 774.5 TB for pre-training corpora and 700M instances for other datasets. We aim to present the entire landscape of LLM text datasets, serving as a comprehensive reference for researchers in this field and contributing to future studies. Related resources are available at: https://github.com/lmmlzn/Awesome-LLMs-Datasets.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Test-Time Scaling via Error Localization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    TTEL uses feedback-induced token probability drops to localize the first error in a failed reasoning trace and branch a new generation from that prefix, improving pass@k per token on coding and math benchmarks.

  2. Pop Quiz Attack: Black-box Membership Inference Attacks Against Large Language Models

    cs.CR 2026-05 unverdicted novelty 6.0 of 10

    PopQuiz Attack infers LLM training data membership by turning examples into quiz questions and measuring answer accuracy, reaching 0.873 average ROC-AUC across six models and outperforming prior methods by 20.6%.

  3. Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code

    cs.SE 2026-05 accept novelty 6.0 of 10

    A review of 114 studies creates taxonomies for code and data quality issues, formalizes 18 propagation mechanisms from training data defects to LLM-generated code defects, and synthesizes detection and mitigation techniques.

  4. Malicious ML Model Detection by Learning Dynamic Behaviors

    cs.CR 2026-04 unverdicted novelty 6.0 of 10

    DynaHug trains an OCSVM on dynamic runtime behaviors of benign PTMs and achieves up to 44% higher F1-score than static, dynamic, and LLM-based baselines on over 25,000 models.

  5. Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomy

    cs.SE 2026-03 conditional novelty 6.0 of 10

    MCP server faults form five empirical categories—server setting, server/tool configuration, server/host configuration, documentation, and general programming—confirmed by a 41-practitioner survey.

  6. Distributional Statistics Restore Training Data Auditability in One-step Distilled Diffusion Models

    cs.LG 2025-02 unverdicted novelty 6.0 of 10

    Kernel-based distributional discrepancy enables auditing of upstream training data in distilled one-step diffusion models by detecting preserved distributional alignment rather than per-instance memorization.

  7. Gaokerena: A Small Persian Medical Language Model Family

    cs.CL 2026-08 conditional novelty 5.0 of 10

    Fine-tuned Persian medical language models reach 49-53% on translated medical MMLU, with datasets released, but the reasoning variant's gain depends on extra test-time compute and a verifier.

  8. STELLAR-E: a Synthetic, Tailored, End-to-end LLM Application Rigorous Evaluator

    cs.AI 2026-04 unverdicted novelty 5.0 of 10

    STELLAR-E modifies the TGRT Self-Instruct framework to produce tailored synthetic LLM evaluation datasets that score an average 5.7% higher on LLM-as-a-judge metrics than existing language-specific benchmarks.

Pith tools