Pith. sign in

REVIEW 4 major objections 6 minor 46 references

This paper introduces DataPrep-Bench, a benchmark that evaluates data construction and data quality evaluation under one shared protocol, and proposes a distribution-based score, DAS, that predicts which candidate datasets will improve down

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 13:39 UTC pith:V5T6XRKF

load-bearing objection A useful, honestly-reported benchmark skeleton; treat the construction rankings and DAS correlations as provisional until the contamination audit and error bars arrive. the 4 major comments →

arxiv 2607.20465 v1 pith:V5T6XRKF submitted 2026-05-19 cs.LG cs.CL

DataPrep-Bench: Benchmarking LLMs as Training Data Preparators

Hao Liang , Qifeng Cai , Yibo Lin , Jianzhuo Du , Qifeng Xia , Sizhe Qiu , Linzhuang Sun , Meiyi Qiang
show 6 more authors
Zhaoyang Han Xiaochen Ma Bohan Zeng Ruichuan An Conghui He Wentao Zhang
This is my paper
classification cs.LG cs.CL
keywords LLM data preparationdata constructiondata quality evaluationdownstream-grounded benchmarkmaximum mean discrepancydistributional alignment scoreagentic data constructionsynthetic training data
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the two core abilities in LLM-driven data preparation—turning raw sources into supervised training data, and predicting which candidate datasets will actually help before training—should be measured by downstream task performance, not by surface-level text quality. To make that possible, it builds DataPrep-Bench, a unified benchmark spanning six domains and multiple base models, where every method consumes the same raw sources, uses the same fine-tuning protocol, and is scored on the same held-out benchmarks. The central empirical findings are that adding domain-specific synthetic data on top of a general instruction corpus often hurts downstream performance, that no single construction method wins across all domains, and that a distribution-based evaluator called DAS, which compares a candidate dataset to a domain proxy using maximum mean discrepancy, is the most reliable cheap predictor of downstream utility in most domains. If these claims hold, researchers can meaningfully compare data preparation methods without the confounding variability that currently makes cross-method comparison anecdotal.

Core claim

The paper's central discovery is that a simple distributional alignment measure—negative MMD between a candidate dataset's embedding distribution and a domain proxy—is a strong, training-free predictor of whether fine-tuning on that candidate will improve downstream performance. Across three base models and six domains, DAS attains the strongest average correlation with ground-truth downstream scores in four of six domains and is the only metric that clears r > 0.70 simultaneously in Math, Science, and Medical, while existing quality-, diversity-, and heuristic-based metrics are either narrow specialists or sign-inconsistent. The paper also introduces Data-Construction-Skill, an agent that u

What carries the argument

The benchmark protocol itself is the central object: a shared, downstream-grounded evaluation where data construction methods are scored by fine-tuning a base model on their outputs jointly with a fixed instruction-following corpus, and quality metrics are scored by Pearson correlation between their scalar outputs and the ground-truth downstream performance of models fine-tuned on the same candidate pools. The proposed metric DAS instantiates this protocol by computing the maximum mean discrepancy (a kernel-based distance between two distributions) between a candidate dataset and a domain proxy, using a fixed text encoder and Gaussian RBF kernel; the proxy is chosen to approximate the target

Load-bearing premise

The benchmark assumes that the candidate datasets and raw source corpora have no substantial overlap with the downstream test sets, so that measured performance reflects genuine training utility rather than memorization of test-like content.

What would settle it

Compute n-gram or embedding overlap between candidate datasets such as the math-focused public corpora and the GSM8K, MATH, and AIME test items; if substantial overlap exists, remove overlapping items from the candidates, recompute the downstream scores, and check whether DAS's r > 0.70 correlations in Math, Science, and Medical persist.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • DataPrep-Bench provides a common testbed that allows construction methods and quality metrics to be compared under identical raw sources, base models, training protocols, and downstream benchmarks, replacing anecdotal cross-method comparison.
  • The finding that synthetic domain data often degrades the Dolly-only baseline shows that 'more synthetic data' is not a safe default, and that downstream-grounded evaluation is necessary to catch regressions that surface-level quality proxies would miss.
  • DAS can be used as a training-free filter to rank candidate datasets before paying the cost of fine-tuning, with the strongest reliability in reasoning- and knowledge-intensive domains.
  • Finance and Law remain open challenges for every quality metric tested, indicating that distributional alignment alone is insufficient in these specialized professional domains.
  • The candidate pools are designed to be extensible: new in-domain datasets can be added without invalidating prior metric scores, so the benchmark can grow as more high-quality SFT corpora become available.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A key unverified assumption is that the candidate pools and source corpora are disjoint from the downstream test sets; if leakage exists, the ground-truth utility scores would reflect memorization rather than training quality, potentially inflating DAS's correlation and undermining the construction rankings.
  • DAS's reliance on a chosen proxy and a single fixed encoder suggests its predictive power may vary with proxy quality and embedding space; a natural extension would be to test DAS with multiple proxies or encoders to quantify this sensitivity.
  • The benchmark's protocol could be extended to other data-preparation tasks, such as deduplication or curriculum ordering, as long as they can be cast as dataset-level decisions scored by downstream performance.
  • The finding that synthetic data often hurts could be stress-tested by varying the amount of synthetic data mixed with the instruction corpus, to see whether there is a safe mixing ratio or a threshold beyond which degradation kicks in.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. DataPrep-Bench proposes a two-track benchmark for LLM-driven data preparation. Track 1 (Data Construction) evaluates construction methods by fine-tuning base models on method-generated QA data jointly with Dolly-15k and measuring downstream performance across six domains. Track 2 (Data Quality Evaluation) evaluates 18 quality metrics by Pearson correlation of metric scores with actual downstream fine-tuning results on per-domain candidate pools, and introduces Distributional Alignment Score (DAS), an MMD-based score comparing each candidate to a domain proxy. The paper reports that adding synthetic domain data often hurts, that no construction family dominates, and that DAS is the strongest quality predictor in 4 of 6 domains.

Significance. If the results hold, the benchmark would fill a real gap: a shared, downstream-grounded protocol for comparing data construction methods and quality metrics under identical raw sources, base models, training recipes, and evaluation sets. The strengths are concrete: ground-truth utility scores come from actual fine-tuning, the protocol is uniform across methods, and the authors release the pools, proxies, constructed datasets, and per-candidate scores. DAS is also refreshingly simple, with no parameters fitted to downstream outcomes. The main risks are contamination of the ground-truth scores, uncontrolled dataset size in the construction track, and missing uncertainty estimates; all are addressable and should be resolved before the benchmark is relied upon.

major comments (4)
  1. [Section 3.4, Table 1, Eq. (4)] The validity of both tracks rests on the candidate pools and source corpora being disjoint from the downstream test sets, but no contamination audit is reported. Section 3.4 criterion (iii) is an assertion, not a verification. The Math candidate pool includes OpenR1-Math-220k, Synthetic-1b, and ScaleQuest, and the Math source corpus includes competition materials; these are synthetic or derived corpora that plausibly overlap GSM8K/MATH/AIME/AMC. If overlap exists, s_{k,i} in Eq. (4) measures memorization rather than training utility, which would corrupt both the construction rankings in Tables 2–5 and the DAS correlations in Tables 6–7. Please add an explicit exact/near-duplicate overlap analysis (e.g., n-gram or embedding-based with thresholds and manual verification) for every candidate dataset, every proxy, and every construction-track source against every item in the downstream bench
  2. [Appendix B.2, Tables 2–5] Appendix B.2 explicitly states that the downstream scores reflect 'the combined effect of construction quality and construction yield.' Since no cap is imposed and per-method counts differ by orders of magnitude (Table 8), the construction-track comparisons are not made under identical conditions despite the Section 1 claim of a shared protocol. This confounds both T1 ('adding synthetic data often hurts') and T2 (the method ranking). For example, on Llama-3.1-8B Finance, Skill emits 106k samples while the Dolly-only baseline receives only Dolly-15k, so the claimed +19.1 point gain mixes extraction yield with data quality. Please either (i) add a yield-controlled comparison (e.g., subsampling every method's output to a common size) or (ii) recast the construction track as a joint quality-plus-yield measure and temper the 'quality' interpretation accordingly.
  3. [Tables 2–7] No multiple seeds, error bars, or confidence intervals are reported for any fine-tuning result. Many of the differences that drive rankings are 1–3 accuracy points (e.g., Table 3 Math average ranges from 7.4 to 11.0 across agents; Table 6 DAS correlations differ by 0.1–0.2 across models within a domain). For a 7–8B parameter SFT setup, such differences are within typical seed variance. The paper should report at least 3 seeds with standard deviations for the construction tables, and confidence intervals (or bootstrap intervals) for the Pearson correlations in Tables 6–7, especially given candidate-pool sizes of only 8–14. Without this, the headline claims that DAS is the 'most reliable' metric and that Skill 'lifts' a baseline by nearly 20 points are under-supported.
  4. [Section 4.2.3, Tables 6–7] DAS's superiority claim depends on several fixed implementation choices: Qwen3-Embedding-8B, Gaussian RBF with σ = 1.0, and 5,000 sampled instances per dataset. No sensitivity analysis is reported with respect to the encoder, kernel bandwidth, or sample size. Because the candidate pools for Finance and Law contain only one in-domain candidate each, the domain-level correlations in those cases are extremely sensitive to a single point, and Table 7 shows DAS at 0.18 and 0.36 there. The paper partly acknowledges this, but the cross-domain summary 'strongest in four of six domains' would be more convincing if accompanied by sensitivity checks and a discussion of the effective sample size for each correlation.
minor comments (6)
  1. [Section 3.5] General Text is evaluated on MMLU-Redux only, so the 'domain average' for General is a single benchmark. This should be stated explicitly when comparing domain-level averages across domains.
  2. [Section 4.1 / Tables 2–5] The name 'Data-Construction-Skill' is very close to 'DataFlow-Skill', and Table 2 uses 'Skill (Claude Opus 4.6)' while DataFlow-Skill appears nearby. A naming convention or footnote distinguishing the two would reduce confusion.
  3. [Appendix C.3] The full content of the construction skill is omitted from the paper and only available via an external repository. For reproducibility, the skill text should be included in the supplementary material or an appendix.
  4. [Figure 1] The line connecting 'Candidate Dataset' to 'MMD' to 'Target Dataset' is unclear; it is better to show explicitly that DAS compares the candidate with the proxy, not with the target, and that the target is used only for ground-truth correlation.
  5. [Section 5.1.2] The sentence explaining why LESS is excluded is reasonable, but it also implies that the benchmark is limited to training-free or task-agnostic metrics. Please state this scope limitation in Section 3.3 as well, since influence-based methods are a major family of data-quality estimators.
  6. [Table 8] The table is informative, but the 'Total' column would benefit from being sorted or grouped by method family to make the yield patterns easier to read.

Circularity Check

0 steps flagged

No significant circularity; DAS parameters are fixed and evaluated against independently measured downstream scores.

full rationale

DataPrep-Bench does not derive its headline findings from its own inputs. The Data Construction Track scores methods by actually fine-tuning base models on the emitted datasets plus Dolly-15k and evaluating on held-out benchmarks (Eq. 3); the released baseline Data-Construction-Skill is just another method in that protocol and has no parameter fitted to the leaderboard. The Data Quality Evaluation Track defines ground truth s_{k,i} by fine-tuning on candidate pools (Eq. 4) and evaluates every metric by Pearson correlation with that ground truth (Eq. 5). DAS is specified with a fixed encoder (Qwen3-Embedding-8B), fixed sample size (5,000), and fixed bandwidth (sigma=1.0), and its proxy datasets are public SFT corpora selected by stated criteria; no DAS hyperparameter is fitted to s_{k,i}. The theoretical support (Redko et al. domain-adaptation bound; MMD triangle inequality) is standard mathematics, and the paper explicitly concedes that Eq. (17) does not guarantee ranking preservation. The only author-overlapping references are DataFlow (a baseline method) and ODA-Math-460k (the Math proxy); both are public artifacts and neither is used to define the evaluation target. The unresolved risk of candidate-pool/test-set overlap in datasets such as OpenR1-Math-220k and ScaleQuest is a contamination/validity concern, not a demonstrated circular reduction, since the paper asserts disjointness as a design criterion but reports no audit. Overall, the central claims are self-contained and empirically grounded.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 1 invented entities

The central benchmark claim rests mainly on the validity of downstream-grounded evaluation. The most fragile inputs are the hand-chosen proxies, the fixed encoder/kernel settings, and the asserted-but-unchecked disjointness between candidate data and test sets. The construction track additionally carries the free variable of dataset size, which confounds quality with yield.

free parameters (3)
  • MMD RBF bandwidth sigma = 1.0
    Hand-set fixed value used for all DAS computations; not fitted to downstream scores, but the choice affects MMD rankings.
  • DAS sample size per dataset = 5,000 instances
    Chosen for statistical stability and compute efficiency; not tuned on downstream performance.
  • Fine-tuning hyperparameters = lr=5e-6, 3 epochs, global batch 32
    Fixed protocol across all construction methods; not tuned per method, which affects fairness of construction comparisons.
axioms (5)
  • domain assumption The domain adaptation bound (Redko et al. Theorem 36) transfers to LLM post-training distributions.
    Invoked in Section 4.2.1 to motivate DAS; the bound is stated for classification-style risk, and its applicability to generative SFT is asserted, not proved.
  • ad hoc to paper The hand-picked domain proxy approximates the latent target distribution P* closely enough that triangle-inequality slack is small.
    Section 4.2.3 states this as the validity condition for DAS; no quantitative closeness check between proxy and target is given.
  • domain assumption Candidate datasets and downstream test sets are disjoint and uncontaminated.
    Section 3.4 asserts non-contamination, but no overlap analysis is provided for synthetic candidates such as OpenR1-Math-220k.
  • domain assumption Linear (Pearson) predictability is the correct criterion for a data-quality metric.
    Section 3.3 chooses Pearson because downstream scores are on a numerical scale; rank-based or threshold-based utility is not evaluated.
  • domain assumption Qwen3-Embedding-8B embeddings capture distributional properties relevant to downstream training utility.
    Section 4.2.2 and 5.1.2 fix this encoder; its relevance to SFT utility is an empirical premise, not a proven fact.
invented entities (1)
  • Data-Construction-Skill 'skill layer' no independent evidence
    purpose: A reusable control layer specifying output schemas, filtering rules, coverage constraints, and validation utilities for agentic data construction.
    Software artifact rather than a scientific entity; it has in-paper empirical support but no out-of-paper falsifiable prediction beyond the benchmark results.

pith-pipeline@v1.3.0-alltime-deepseek · 29216 in / 12704 out tokens · 138872 ms · 2026-08-02T13:39:01.726591+00:00 · methodology

0 comments
read the original abstract

The quality of training data fundamentally determines the capabilities of large language models (LLMs), yet no unified benchmark exists to measure how well LLMs, agents, and data-centric workflows actually prepare training data end to end. We view LLM-driven data preparation as comprising two complementary capabilities: data construction, which transforms raw sources into supervised training data, and data quality evaluation, which predicts the training value of candidate datasets before downstream training; throughout, "quality" refers to downstream training utility rather than surface-level textual properties. We introduce DataPrep-Bench, the first unified benchmark that jointly evaluates both capabilities under a shared downstream-grounded protocol over six domains and multiple base models. For data construction, methods consume identical raw sources and are scored by fine-tuning a base model on their outputs jointly with Dolly-15k; alongside this track we release Data-Construction-Skill, a skill-guided agent that lifts the Dolly-only baseline by nearly 20 points absolute on Llama-3.1-8B Finance and is competitive with the strongest agent- and DataFlow-based methods in knowledge-extraction-dense domains. For data quality evaluation, scoring functions are scored by Pearson correlation with downstream performance on a shared candidate pool; we release the Distributional Alignment Score (DAS), a distribution-based evaluator that uses MMD between a candidate dataset and a domain proxy. DAS attains the strongest cross-model correlation in four of six domains and is the only metric clearing r > 0.70 simultaneously in Math, Science, and Medical, outperforming existing quality-, diversity-, and heuristic-based evaluators. DataPrep-Bench provides a unified, downstream-grounded framework for measuring progress on both capabilities as co-equal targets of LLM-driven data preparation.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 18 linked inside Pith

  1. [1]

    Fowlkes, Stefano Soatto, and Pietro Perona

    Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6430–6439, 2019

    2019

  2. [2]

    A survey of multimodal large language model from a data-centric perspective.arXiv preprint arXiv:2405.16640, 2024

    Tianyi Bai, Hao Liang, Binwang Wan, Ling Yang, Bozhou Li, Yifan Wang, Bin Cui, Conghui He, Binhang Yuan, and Wentao Zhang. A survey of multimodal large language model from a data-centric perspective.arXiv preprint arXiv:2405.16640, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Text2sql-flow: A robust sql-aware data augmentation framework for text-to-sql.arXiv preprint arXiv:2511.10192, 2025

    Qifeng Cai, Hao Liang, Chang Xu, Tao Xie, Wentao Zhang, and Bin Cui. Text2sql-flow: A robust sql-aware data augmentation framework for text-to-sql.arXiv preprint arXiv:2511.10192, 2025

    arXiv 2025

  4. [4]

    Data-juicer: A one-stop data processing system for large language models

    Daoyuan Chen, Yilun Huang, Zhijian Ma, Hesen Chen, Xuchen Pan, Ce Ge, Dawei Gao, Yuexiang Xie, Zhaoyang Liu, Jinyang Gao, Yaliang Li, Bolin Ding, and Jingren Zhou. Data-juicer: A one-stop data processing system for large language models. InInternational Conference on Management of Data, 2024

    2024

  5. [5]

    Dc-bench: Dataset condensation benchmark.Advances in Neural Information Processing Systems, 35:810–822, 2022

    Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Dc-bench: Dataset condensation benchmark.Advances in Neural Information Processing Systems, 35:810–822, 2022

    2022

  6. [6]

    Enhancing chat language models by scaling high-quality instructional conversations, 2023

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations, 2023

    2023

  7. [7]

    Unleashing reasoning capability of llms via scalable question synthesis from scratch.arXiv preprint arXiv:2410.18693, 2024

    Yuyang Ding, Xinyu Shi, Xiaobo Liang, Juntao Li, Qiaoming Zhu, and Min Zhang. Unleashing reasoning capability of llms via scalable question synthesis from scratch.arXiv preprint arXiv:2410.18693, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Megascience: Pushing the frontiers of post-training datasets for science reasoning

    Run-Ze Fan, Zengzhi Wang, and Pengfei Liu. Megascience: Pushing the frontiers of post-training datasets for science reasoning. arXiv preprint arXiv:2507.16812, 2025. URLhttps://arxiv.org/abs/2507.16812

    Pith/arXiv arXiv 2025

  9. [9]

    The vendi score: A diversity evaluation metric for machine learning.arXiv preprint arXiv:2210.02410, 2022

    Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning.arXiv preprint arXiv:2210.02410, 2022

    Pith/arXiv arXiv 2022

  10. [10]

    Datacomp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36:27092–27112, 2023

    2023

  11. [11]

    Closing the data loop: Using opendataarena to engineer superior training datasets.arXiv preprint arXiv:2601.09733, 2025

    Xin Gao, Xiaoyang Wang, Yun Zhu, Mengzhang Cai, Conghui He, and Lijun Wu. Closing the data loop: Using opendataarena to engineer superior training datasets.arXiv preprint arXiv:2601.09733, 2025

    arXiv 2025

  12. [12]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Textbooks are all you need.arXiv preprint arXiv:2306.11644, 2023

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need.arXiv preprint arXiv:2306.11644, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Lawyer llama technical report, 2023

    Quzhe Huang, Mingxu Tao, Chen Zhang, Zhenwei An, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. Lawyer llama technical report, 2023

    2023

  15. [15]

    Mistral 7b.arXiv preprint arXiv:2310.06825, 2023

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b.arXiv preprint arXiv:2310.06825, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Scaling laws for neural language models.arXiv preprintarXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprintarXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  17. [17]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi...

    2024

  18. [18]

    Datacomp-lm: In search of the next generation of training sets for language models

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems, 37:14200–14282, 2024. DataPrep-Bench 22

    2024

  19. [19]

    Infinity instruct: Scaling instruction selection and synthesis to enhance language models, 2025

    Jijie Li, Li Du, Hanyu Zhao, Bo wen Zhang, Liangdong Wang, Boyan Gao, Guang Liu, and Yonghua Lin. Infinity instruct: Scaling instruction selection and synthesis to enhance language models, 2025. URLhttps: //arxiv.org/abs/2506.11116

    Pith/arXiv arXiv 2025

  20. [20]

    Superfiltering: Weak-to-strong data filtering for fast instruction-tuning

    Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. Superfiltering: Weak-to-strong data filtering for fast instruction-tuning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14255...

    2024

  21. [21]

    Dataflow: An llm-driven framework for unified data preparation and workflow automation in the era of data-centric ai.arXiv preprint arXiv:2512.16676, 2025

    Hao Liang, Xiaochen Ma, Zhou Liu, Zhen Hao Wong, Zhengyang Zhao, Zimo Meng, Runming He, Chengyu Shen, Qifeng Cai, Zhaoyang Han, et al. Dataflow: An llm-driven framework for unified data preparation and workflow automation in the era of data-centric ai.arXiv preprint arXiv:2512.16676, 2025

    arXiv 2025

  22. [22]

    Data preparation for large language models.Journal of Computer Science and Technology, 2026

    Hao Liang, Zhen Hao Wong, Ruitong Liu, Yuhan Wang, Meiyi Qiang, Zhengyang Zhao, Chengyu Shen, Conghui He, Wentao Zhang, and Bin Cui. Data preparation for large language models.Journal of Computer Science and Technology, 2026. ISSN 1000-9000(Print) /1860-4749(Online). doi: 10.1007/s11390-026-5948-8. URL https://jcst.ict.ac.cn/en/article/doi/10.1007/s11390-...

    doi:10.1007/s11390-026-5948-8 2026

  23. [23]

    What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning

    Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. InThe TwelfthInternational Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=BTKAeLqLMw

    2024

  24. [24]

    Dataperf: Benchmarks for data-centric ai development

    Mark Mazumder, Colby Banbury, Xiaozhe Yao, Bojan Karlaš, William Gaviria Rojas, Sudnya Diamos, Greg Diamos, Lynn He, Alicia Parrish, Hannah Rose Kirk, et al. Dataperf: Benchmarks for data-centric ai development. Advances in Neural Information Processing Systems, 36:5320–5347, 2023

    2023

  25. [25]

    GPT-4 technical report, 2023

    OpenAI. GPT-4 technical report, 2023

    2023

  26. [26]

    Towards tailored recovery of lexical diversity in literary machine translation

    Esther Ploeger, Huiyuan Lai, Rik Van Noord, and Antonio Toral. Towards tailored recovery of lexical diversity in literary machine translation. InProceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1), pages 286–299, 2024

    2024

  27. [27]

    Fino1: On the transferability of reasoning-enhanced llms and reinforcement learning to finance

    Lingfei Qian, Weipeng Zhou, Yan Wang, Xueqing Peng, Han Yi, Yilun Zhao, Jimin Huang, Qianqian Xie, and Jian-yun Nie. Fino1: On the transferability of reasoning-enhanced llms and reinforcement learning to finance. arXiv preprint arXiv:2502.08127, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    A survey on domain adaptation theory: learning bounds and theoretical guarantees.arXiv preprint arXiv:2004.11829, 2020

    Ievgen Redko, Emilie Morvant, Amaury Habrard, Marc Sebban, and Younès Bennani. A survey on domain adaptation theory: learning bounds and theoretical guarantees.arXiv preprint arXiv:2004.11829, 2020

    Pith/arXiv arXiv 2004

  29. [29]

    Let’s verify math questions step by step.arXiv preprint arXiv:2505.13903, 2025

    Chengyu Shen, Zhen Hao Wong, Runming He, Hao Liang, Meiyi Qiang, Zimo Meng, Zhengyang Zhao, Bohan Zeng, Zhengzhou Zhu, Bin Cui, et al. Let’s verify math questions step by step.arXiv preprint arXiv:2505.13903, 2025

    arXiv 2025

  30. [30]

    Reasonmed: A 370k multi-agent generated dataset for advancing medical reasoning,

    Yu Sun, Xingyu Qian, Weiwen Xu, Hao Zhang, Chenghao Xiao, Long Li, Deli Zhao, Wenbing Huang, Tingyang Xu, Qifeng Bai, and Yu Rong. Reasonmed: A 370k multi-agent generated dataset for advancing medical reasoning,

  31. [31]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    Self-instruct: Aligning language models with self-generated instructions

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. InProceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), pages 13484–13508, 2023

    2023

  33. [33]

    Qurating: Selecting high-quality data for training language models.arXiv preprint arXiv:2402.09739, 2024

    Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models.arXiv preprint arXiv:2402.09739, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    Less: Selecting influential data for targeted instruction tuning.arXiv preprint arXiv:2402.04333, 2024

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning.arXiv preprint arXiv:2402.04333, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Wizardlm: Empowering large pre-trained language models to follow complex instructions, 2023

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions, 2023. DataPrep-Bench 23

    2023

  36. [36]

    Logics-stem: Empowering llm reasoning via failure-driven post-training and document knowledge enhancement, 2026

    Mingyu Xu, Cheng Fang, Keyue Jiang, Yuqian Zheng, Yanghua Xiao, Baojian Zhou, Qifang Zhao, Suhang Zheng, Xiuwen Zhu, Jiyang Tang, Yongchi Zhao, Yijia Luo, Zhiqi Bai, Yuchi Xu, Wenbo Su, Wei Wang, Bing Zhao, Lin Qu, and Xiaoxiao Xu. Logics-stem: Empowering llm reasoning via failure-driven post-training and document knowledge enhancement, 2026. URLhttps://a...

    arXiv 2026

  37. [37]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu X...

    Pith/arXiv arXiv 2024

  39. [39]

    Opencsg chinese corpus: A series of high-quality chinese datasets for llm training, 2025

    Yijiong Yu, Ziyun Dai, Zekun Wang, Wei Wang, Ran Chen, and Ji Pei. Opencsg chinese corpus: A series of high-quality chinese datasets for llm training, 2025. URLhttps://arxiv.org/abs/2501.08197

    Pith/arXiv arXiv 2025

  40. [40]

    Disc-lawllm: Fine-tuning large language models for intelligent legal services, 2023

    Shengbin Yue, Wei Chen, Siyuan Wang, Bingxuan Li, Chenchen Shen, Shujun Liu, Yuxuan Zhou, Yao Xiao, Song Yun, Xuanjing Huang, and Zhongyu Wei. Disc-lawllm: Fine-tuning large language models for intelligent legal services, 2023

    2023

  41. [41]

    Ultramedical: Building specialized generalists in biomedicine, 2024

    Kaiyan Zhang, Sihang Zeng, Ermo Hua, Ning Ding, Zhang-Ren Chen, Zhiyuan Ma, Haoxin Li, Ganqu Cui, Biqing Qi, Xuekai Zhu, Xingtai Lv, Hu Jinfang, Zhiyuan Liu, and Bowen Zhou. Ultramedical: Building specialized generalists in biomedicine, 2024

    2024

  42. [42]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistic...

    Pith/arXiv arXiv 2024

  43. [43]

    Google-proof

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment.Advances in Neural Information Processing Systems, 36: 55006–55021, 2023. DataPrep-Bench 24 Appendix A Detailed Introduction to the Benchmarks The following provides a detailed description of the ben...

    2023

  44. [45]

    Strict Data Adherence: All QA content must be entirely based on the provided Markdown files

  45. [46]

    Autonomous Operation: Run automatically without manual confirmation Figure 3Data construction prompt used to guide the agent. B.3 Data Quality Evaluation: Implementation Details To evaluate whether data quality metrics generalize across model architectures, we conduct experiments with three representative base models: Qwen2.5-7B [38], Llama-3.1-8B [12], a...

  46. [2025]

    URLhttps://arxiv.org/abs/2506.09513

    arXiv