Pith. sign in

REVIEW 4 major objections 6 minor 49 references

LP Data Pipeline: Lightweight, Purpose-driven Data Pipeline for Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A fully CPU-based pipeline can curate large-scale, domain-specific LLM training data with quality comparable to GPU-based filtering, at a fraction of the cost.

desk verdict A useful CPU-only data-curation system, but the headline quality-parity claim is untested, so treat it as an engineering report, not a validated method. read the letter →

arxiv 2411.11289 v1 pith:KPZ6XFZV submitted 2024-11-18 cs.CL cs.AI

Yungi Kim , Hyunsoo Ha , Seonghoon Yang , Sukyung Lee , Jihoo Kim , Chanjun Park This is my paper · ORCID
classification cs.CLcs.AI
keywords datacurationlargelanguagemodelsCPU-onlypipelinequalityfilteringdomain-specificdatasetsCommonCrawlFastTextKenLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that high-quality, large-scale training data for large language models does not have to be curated with expensive GPU-accelerated quality filters. It presents the Lightweight, Purpose-driven (LP) Data Pipeline, which runs entirely on CPUs and combines heuristic cleaning, deduplication, and lightweight FastText-based quality and domain classifiers. On a 4TB CommonCrawl dump, the pipeline completed all stages in 4 hours 22 minutes at an estimated cost of about $353, producing separate finance, law, and medical corpora in English and Korean. The central claim is that this CPU-only pipeline maintains data quality comparable to GPU-based reward-model filtering, making LLM development accessible to organizations without heavy GPU infrastructure. The paper notes that direct LLM training experiments on the curated data were not run, so quality equivalence is asserted rather than demonstrated end-to-end.

What carries the argument

The load-bearing piece is the quality-filtering module: a FastText classifier trained on labels produced by a Good KenLM (an n-gram language model trained on high-quality sources) and a Bad KenLM (trained on toxic and low-quality text), replacing GPU-based reward models for document scoring. FastText is a lightweight linear text classifier, and because it runs on CPUs it lets every stage of the pipeline avoid GPU inference. Supporting machinery includes line-level deduplication grouped by URL domain rather than random buckets, which removes boilerplate while lowering compute, plus MinHash locality-sensitive hashing for global near-duplicate removal and a second FastText model for domain classification. Together these components make the claimed CPU-only, low-cost curation possible.

What would settle it

Train two identically sized models, one on an LP-filtered corpus and one on a corpus filtered by a GPU-based reward model, and compare held-out downstream performance; if the LP-filtered model is materially worse, the claim of comparable quality fails. A quicker check: measure agreement between the pipeline's quality classifier and human judgments on a random sample of web documents.

Watch

Extended reading notes

Core claim

The LP Data Pipeline is a sequential, fully CPU-based curation framework built on four principles: CPU-only processing, an optimized stage order that defers expensive operations until after cheap filtering, scheduled continuous updates from fresh web dumps, and purpose-driven construction of domain- and language-specific datasets. Its key novelty is replacing GPU-based reward-model quality filtering with a FastText classifier trained on documents labeled by two KenLM models, one trained on high-quality sources and one on low-quality content; the classifier is then applied at scale on CPUs. A second FastText model, trained using LLM-annotated samples, classifies documents into finance, law, healthcare, or other. The paper reports that processing the 4TB CC-MAIN-2024-10 dump across 128 8-core machines took 4 hours 22 minutes and an estimated $352.83, and that scaling to ten dumps produced 17.95 billion legal, 34.02 billion finance, and 34.84 billion medical tokens for English, with smaller but analogous Korean corpora. The paper's stated conclusion is that the pipeline significantly reduces computational costs while maintaining data quality comparable to that of GPU-based methods, with future LLM training experiments planned to validate that claim.

Load-bearing premise

That a lightweight text classifier, trained on documents labeled good or bad by two language-model scorers, can separate high-quality from low-quality text as reliably as expensive GPU-based reward models, even though the paper never checks this against independent quality judgments.

Editorial extensions

If this is right

  • Organizations without GPU clusters can process CommonCrawl-scale dumps for a few hundred dollars in under a day, removing the main compute barrier to custom LLM pretraining.
  • Purpose-driven corpora for finance, law, and healthcare, and for languages like Korean, can be produced on demand, making specialized LLMs feasible where they previously were not.
  • Because the pipeline is scheduled to pick up new dumps automatically, LLM training data can stay current without manual curation effort.
  • If the quality-filtering FastText truly matches GPU reward models, the cost of data curation becomes a minor line item in LLM development budgets rather than a dominant one.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step, which the paper leaves open, is to train comparable LLMs on LP-filtered and GPU-filtered corpora of equal size; if metrics match, the CPU-only approach becomes a standard baseline for low-resource data curation.
  • The Good/Bad KenLM labeling scheme could be reused to bootstrap quality filters for new languages cheaply, since it requires no human quality annotations, only high- and low-quality seed corpora.
  • The domain-grouping deduplication trick is independent of the quality model and could be dropped into existing GPU-based pipelines as a cheap prefilter, reducing the amount of data sent to expensive filters.
  • The strongest version of the paper's claim, that CPU quality is comparable to GPU quality, will remain untested until someone runs the downstream training experiment; the paper itself flags this as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces the LP Data Pipeline, a fully CPU-based framework for curating large-scale text corpora for LLM training, with stages for raw text extraction, URL filtering, language identification, domain-grouped line-level deduplication, heuristic filtering, global MinHash deduplication, model-based quality filtering with a FastText classifier trained on Good/Bad KenLM labels, and FastText domain classification for finance, law, and healthcare. The empirical sections report processing time and estimated cost for one CommonCrawl dump, document/token counts for English and Korean domain corpora, and scaled-up counts across ten dumps. The central claim, stated in the abstract and conclusion, is that the pipeline significantly reduces computational cost while maintaining data quality comparable to GPU-based methods such as FineWeb-edu.

Significance. If the central claim were established, the pipeline would be a genuinely useful contribution: CPU-only curation would lower the infrastructure barrier for organizations that cannot afford GPU-based reward-model filtering, and the domain-specific English/Korean corpora could support specialized LLM development. The paper's reported time and cost figures for processing a 4TB CommonCrawl dump are plausible and potentially valuable, and the use of lightweight models is an appropriate direction. However, the paper provides no evidence that the output corpora are actually high quality: there are no LLM training experiments, no downstream task evaluations, no comparison with FineWeb-edu or other curated corpora, no human quality judgments, and no measurement of the FastText filters' fidelity to their training labels. The significance of the claimed quality parity is therefore currently unsubstantiated.

major comments (4)
  1. [Abstract, Section 5, Limitations] The central claim that the pipeline 'maintains data quality comparable to that of GPU-based methods' is not supported by any quality measurement in the paper. Section 4 reports only processing time, estimated cost, and document/token counts (Tables 2 and 3, Figure 3). The Limitations section explicitly states that 'extensive LLM training experiments using the curated datasets were not conducted,' and no comparison to FineWeb-edu, a reward-model baseline, human judgments, or any external quality benchmark is provided. As stated, the quality-parity claim is an assertion rather than a demonstrated result.
  2. [Section 3.2.6] The model-based quality filter is load-bearing for every downstream quality claim, yet its validity is never tested. A FastText classifier is trained on labels produced by the authors' own Good and Bad KenLM models (Kim et al., 2024b), but the paper does not report the FastText classifier's accuracy, its agreement with the KenLM labels on held-out data, or any analysis of whether the filtered corpus is superior to unfiltered data. Three links are required for the central claim: (i) KenLM scores approximate document quality, (ii) the FastText classifier reproduces those scores on unseen CommonCrawl documents, and (iii) filtering by this classifier yields data comparable in utility to GPU reward-model filtering. None of these links is tested here.
  3. [Section 3.2.4, Table 1] The heuristic filtering thresholds in Table 1 are described as 'determined through qualitative assessments,' with no validation of their effect on corpus quality and no sensitivity analysis. Since these thresholds discard a large fraction of documents, the paper should at least report how the thresholds were chosen, show that they remove documents independently judged to be low quality, and demonstrate that the surviving documents retain useful content. Without such evidence, the heuristic filtering stage cannot support the claim that the pipeline preserves high data quality.
  4. [Section 3.2.7] The domain classification model is trained on labels produced by an LLM, but the paper gives no details of the annotation prompt, no estimate of label quality, no classifier accuracy, and no human evaluation of the resulting domain-specific corpora. Given that the paper's purpose-driven dataset contribution depends entirely on the reliability of this classifier, the absence of any validation of the three-way finance/law/healthcare distinction is a major gap.
minor comments (6)
  1. [Section 4.1] The text states that the data was processed on '128 machines with 8-core CPUs,' but the footnote says the estimated cost was calculated using 120 AWS m7a.2xlarge instances; this discrepancy should be resolved.
  2. [Section 4.3] The claim that total time and cost for ten dumps can be approximated by scaling the single-dump result by a factor of ten is presented as a finding, but it is an assumption rather than an empirical result; no scaling experiment with varying cluster sizes is reported.
  3. [Section 3.1] There is a typo in the subsection title: 'Purpose-Driven Datset Construction' should be 'Purpose-Driven Dataset Construction.'
  4. [Section 3.2.2] The language identification tool is described only as 'based on FastText'; the specific model or checkpoint should be identified so that the CPU-only claim can be reproduced.
  5. [Section 3.2.3] The proposed domain-grouped line-level deduplication is motivated as more efficient than random bucketing, but no comparison against a random-bucket baseline is provided; the claim of reduced computational cost is therefore not directly demonstrated.
  6. [General] The paper does not state whether the curated datasets or the pipeline code will be released; for a data-curation contribution, public release of at least a sample or a reproducibility package would strengthen the paper.

Circularity Check

1 steps flagged · score 7.0 of 10

The central claim of GPU-comparable data quality is not independently validated; it reduces to the authors' self-cited Good/Bad KenLM labels feeding the FastText filter.

  1. self citation load bearing [Section 3.2.6 (Model-based Quality Filtering); Section 5 (Conclusion); Limitations]
    "To facilitate the curation of high-quality data using only CPU computation, we enhance the performance of the FastText model by utilizing datasets generated through both Good KenLM and Bad KenLM (Kim et al., 2024b). ... A FastText classifier is trained on CommonCrawl samples labeled as positive by Good KenLM and negative by Bad KenLM, and is then utilized. ... Based on our four core principles, the LP Data Pipeline significantly reduces computational costs while maintaining data quality comparable to that of GPU-based methods. ..."

    The pipeline's operational definition of 'high-quality data' is the output of the FastText classifier, and that classifier is fitted to labels produced by Good and Bad KenLM from Kim et al. (2024b), the authors' own companion paper. The quality-parity claim is therefore a restatement of the classifier's agreement with those self-cited labels: documents that survive filtering are, by construction, documents that the authors' KenLM-derived model scores highly. No external quality benchmark, human evaluation, reward-model comparison, or downstream LLM training is reported, and the Limitations explicitly state that no extensive LLM training experiments were conducted.

full rationale

The measured processing time and cost results (Section 4, Tables 2 and 3) are self-contained and are not circular. The circularity is confined to the quality dimension of the central claim. Section 3.2.6 constructs the quality filter from Good/Bad KenLM labels (Kim et al., 2024b, same authorship group), and the Abstract and Section 5 assert that the pipeline 'maintains high data quality' / 'quality comparable to that of GPU-based methods' without any external validation. That assertion reduces to the pipeline's own filter scoring the data highly. The domain classification step also relies on LLM-generated labels, but that is a labeling choice rather than a definitional reduction of the central claim. Because the quality parity claim is the key contribution beyond CPU efficiency and is supported only by a self-citation chain, the circularity score is 7 rather than lower; the independent cost/efficiency results keep it from being a complete 8-10 self-definition.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The pipeline introduces no new physical or mathematical entities. It relies on a set of hand-chosen heuristic thresholds, unspecified labeling thresholds from companion work, and unvalidated assumptions about the quality and domain classifiers. The most important assumption, that the FastText model trained on the authors' own KenLM labels is a valid quality measure, is load-bearing and untested.

free parameters (4)
  • Heuristic filtering thresholds (Table 1) = Not fully listed; examples include minimum 5 lines, document length >= 200, terminal punctuation fraction > 0.12…
    The thresholds were 'determined through qualitative assessments' (Section 3.2.4), which means they were tuned on the target data and are not derived from first principles.
  • Good/Bad KenLM labeling thresholds = Not specified in this paper
    The quality classifier training data is created by thresholding Good and Bad KenLM scores, but the exact thresholds are not reported here and are taken from the companion paper.
  • FastText classifier hyperparameters = Not reported
    Training details for the quality and domain classifiers are omitted, so the exact configuration is a free choice affecting results.
  • LLM annotation prompt for domain labels = Not reported
    The domain classification training data is annotated by an LLM, but the prompt, model, and annotation protocol are not described; these choices determine the domain labels.
assumptions (6)
  • domain assumption FastText classifier trained on Good/Bad KenLM labels is a valid proxy for data quality.
    This is the foundation of Section 3.2.6 and the central quality claim, but no external validation is provided.
  • domain assumption Bad KenLM, trained on toxic and informal content, correctly identifies low-quality documents.
    The negative training set for the quality filter is generated by Bad KenLM; its validity is assumed from the companion paper.
  • domain assumption LLM-based annotations for finance, law, and healthcare domains are accurate.
    The domain classifier is trained on LLM-annotated data, but no annotation quality check is reported.
  • ad hoc to paper Domain-grouped line-level deduplication removes boilerplate without discarding unique content.
    This is a new variant proposed in Section 3.2.3; the paper only shows that common boilerplate lines are removed, not that unique lines are retained.
  • ad hoc to paper Heuristic filtering thresholds are appropriately stringent for high-quality data.
    The thresholds in Table 1 are set by qualitative assessment, so they are an unverified modeling choice.
  • standard math MinHash LSH deduplication behaves as described in the literature.
    MinHash LSH is a standard technique, so this is a reasonable background assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LP Data Pipeline: Lightweight, Purpose-driven Data Pipeline for Large Language Models." pith.science (2026). https://pith.science/paper/KPZ6XFZV

@misc{pith2026241111289,
  author       = {Pith},
  title        = {Pith review of: LP Data Pipeline: Lightweight, Purpose-driven Data Pipeline for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KPZ6XFZV}},
  note         = {Machine review of arXiv:2411.11289}
}
read the original abstract

Creating high-quality, large-scale datasets for large language models (LLMs) often relies on resource-intensive, GPU-accelerated models for quality filtering, making the process time-consuming and costly. This dependence on GPUs limits accessibility for organizations lacking significant computational infrastructure. To address this issue, we introduce the Lightweight, Purpose-driven (LP) Data Pipeline, a framework that operates entirely on CPUs to streamline the processes of dataset extraction, filtering, and curation. Based on our four core principles, the LP Data Pipeline significantly reduces preparation time and cost while maintaining high data quality. Importantly, our pipeline enables the creation of purpose-driven datasets tailored to specific domains and languages, enhancing the applicability of LLMs in specialized contexts. We anticipate that our pipeline will lower the barriers to LLM development, enabling a wide range of organizations to access LLMs more easily.

Figures

Figures reproduced from arXiv: 2411.11289 by the authors.

Figure 1
Figure 1. Overview of the Lightweight, Purpose-driven (LP) Data Pipeline: Data extraction and cleansing flow from Common Crawl WARC dumps, illustrating the filtering processes and the sizes of data being filtered at each stage. Purpose-Driven Datset Construction. The pipeline supports the creation of datasets tailored to specific domains, such as finance, law, and medicine. With advanced module for domain classi￾fication, it … view at source ↗
Figure 2
Figure 2. Overview of the training process for the quality filtering model and the domain classification model. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The token distribution of domain-specific [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: System Architecture of the Lightweight, Purpose-driven (LP) Data Pipeline A System Architecture [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 11 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Amazon Web Services . 2023. https://aws.amazon.com/ec2/instance-types/m7a/ Amazon ec2 m7a instances . Accessed: November 14, 2024

    work page 2023

  4. [4]

    Apache. 2024. https://spark.apache.org/ Spark

    work page 2024

  5. [5]

    Janek Bevendorff, Benno Stein, Matthias Hagen, and Martin Potthast. 2018. Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl . In Advances in Information Retrieval. 40th European Conference on IR Research (ECIR 2018), Lecture Notes in Computer Science, Berlin Heidelberg New York. Springer

    work page 2018

  6. [6]

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the association for computational linguistics, 5:135--146

    2017

  7. [7]

    Isaac Caswell, Julia Kreutzer, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. 2021. Quality at a glance: An audit of web-crawled multilingual datasets. arXiv e-prints, pages arXiv--2103

    work page 2021

  8. [8]

    Daixuan Cheng, Shaohan Huang, and Furu Wei. 2024. Adapting large language models to domains via reading comprehension. arXiv preprint arXiv:2309.09530

    arXiv 2024

Show all 49 references
  1. [9]

    Orestis Christou, Nikolaos Pitropakis, Pavlos Papadopoulos, Sean McKeown, and William J Buchanan. 2020. Phishing url detection through top-level domain analysis: A descriptive approach. arXiv preprint arXiv:2005.06599

    2020 arXiv

  2. [10]

    Colin B Clement, Matthew Bierbaum, Kevin P O'Keeffe, and Alexander A Alemi. 2019. On the use of arxiv as a dataset. arXiv preprint arXiv:1905.00075

    2019 arXiv

  3. [11]

    CommonCrawl. 2024. https://commoncrawl.org/ Commoncrawl corpus

    2024

  4. [12]

    Together Computer. 2023. https://github.com/togethercomputer/RedPajama-Data Redpajama: An open source recipe to reproduce llama training dataset

    2023

  5. [13]

    Jesse Dodge, Maarten Sap, Ana Marasovi \'c , William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint arXiv:2104.08758

    2021 arXiv

  6. [14]

    Yaxin Du, Rui Ye, Fengting Yuchi, Wanru Zhao, Jingjing Qu, Yanfeng Wang, and Siheng Chen. 2024. Data quality control in federated instruction-tuning of large language models. arXiv preprint arXiv:2410.11540

    2024 arXiv

  7. [15]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  8. [16]

    Wikimedia Foundation. 2024. https://dumps.wikimedia.org Wikimedia downloads

    2024

  9. [17]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027

    2020 arXiv

  10. [18]

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C \'e sar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. 2023. Textbooks are all you need. arXiv preprint arXiv:2306.11644

    2023 arXiv

  11. [19]

    Kai He, Rui Mao, Qika Lin, Yucheng Ruan, Xiang Lan, Mengling Feng, and Erik Cambria. 2023. A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics. arXiv preprint arXiv:2310.05694

    2023 arXiv

  12. [20]

    Kenneth Heafield. 2011. Kenlm: Faster and smaller language model queries. In Proceedings of the sixth workshop on statistical machine translation, pages 187--197

    2011

  13. [21]

    Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Juncai He, et al. 2023. Acegpt, localizing large language models in arabic. arXiv preprint arXiv:2309.12053

    2023 arXiv

  14. [22]

    Piotr Indyk and Rajeev Motwani. 1998. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of the thirtieth annual ACM symposium on Theory of computing, pages 604--613

    1998

  15. [23]

    Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759

    2016 arXiv

  16. [24]

    Arham Khan, Robert Underwood, Carlo Siebenschuh, Yadu Babuji, Aswathy Ajith, Kyle Hippe, Ozan Gokdemir, Alexander Brace, Kyle Chard, and Ian Foster. 2024. Lshbloom: Memory-efficient, extreme-scale document deduplication. arXiv preprint arXiv:2411.04257

    2024

  17. [25]

    Dahyun Kim, Sukyung Lee, Yungi Kim, Attapol Rutherford, and Chanjun Park. 2024 a . Representing the under-represented: Cultural and core capability benchmarks for developing thai large language models. arXiv preprint arXiv:2410.04795

    2024 arXiv

  18. [26]

    Yungi Kim, Hyunsoo Ha, Sukyung Lee, Jihoo Kim, Seonghoon Yang, and Chanjun Park. 2024 b . Rethinking kenlm: Good and bad model ensembles for efficient text quality filtering in large web corpora. arXiv preprint arXiv:2409.09613

    2024 arXiv

  19. [27]

    Jinqi Lai, Wensheng Gan, Jiayang Wu, Zhenlian Qi, and S Yu Philip. 2024. Large language models in law: A survey. AI Open

    2024

  20. [28]

    Jean Lee, Nicholas Stevens, Soyeon Caren Han, and Minseok Song. 2024. A survey of large language models in finance (finllms). arXiv preprint arXiv:2402.02315

    2024 arXiv

  21. [29]

    Ruihang Li, Yixuan Wei, Miaosen Zhang, Nenghai Yu, Han Hu, and Houwen Peng. 2024. Scalingfilter: Assessing data quality through inverse utilization of scaling laws. arXiv preprint arXiv:2408.08310

    2024 arXiv

  22. [30]

    Yuanzhi Li, S \'e bastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463

    2023 arXiv

  23. [31]

    Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, et al. 2023. Domain specialization as the key to make large language models disruptive: A comprehensive survey. arXiv preprint arXiv:2305.18703

    2023 arXiv

  24. [32]

    Yang Liu, Jiahuan Cao, Chongyu Liu, Kai Ding, and Lianwen Jin. 2024. Datasets for large language models: A comprehensive survey. arXiv preprint arXiv:2402.18041

    2024 arXiv

  25. [33]

    Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. 2023. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. arXiv preprint...

    2023 arXiv

  26. [34]

    Diksha Meghwal, Katharina Kann, Iacer Calixto, and Stanislaw Jastrz e bski. 2020. Can wikipedia categories improve masked language model pretraining? In Proceedings of the Fourth Widening Natural Language Processing Workshop

    2020

  27. [35]

    Guilherme Penedo, Hynek Kydl \' c ek, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557

    2024 arXiv

  28. [36]

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data only. Advances in Neur...

    2023

  29. [37]

    Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S Yu. 2024. Multilingual large language model: A survey of resources, taxonomy and frontiers. arXiv preprint arXiv:2404.04925

    2024 arXiv

  30. [38]

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446

    2021 arXiv

  31. [39]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. http://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . Journal of Machine Lea...

    2020

  32. [40]

    Rick Rejeleene, Xiaowei Xu, and John Talburt. 2024. Towards trustable language models: Investigating information quality of large language models. arXiv preprint arXiv:2401.13086

    2024 arXiv

  33. [41]

    Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. 2024. How to train data-efficient llms. arXiv preprint arXiv:2402.09668

    2024 arXiv

  34. [42]

    Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, et al. 2023. Slimpajama-dc: Understanding data combinations for llm training. arXiv preprint arXiv:2309.10818

    2023 arXiv

  35. [43]

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. https://huggingface.co/datasets/cerebras/SlimPajama-627B SlimPajama: A 627B token cleaned and deduplicated version of RedPajama

    2023

  36. [44]

    Neil C Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F Manso. 2020. The computational limits of deep learning. arXiv preprint arXiv:2007.05558, 10

    2020 arXiv

  37. [45]

    Chi Tran and Huong Le Thanh. 2024. Lavy: Vietnamese multimodal large language model. arXiv preprint arXiv:2404.07922

    2024 arXiv

  38. [46]

    Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzm \'a n, Armand Joulin, and Edouard Grave. 2019. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359

    2019 arXiv

  39. [47]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223

    2023 arXiv

  40. [48]

    Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024. How do large language models handle multilingualism? arXiv preprint arXiv:2402.18815

    2024 arXiv

  41. [49]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P Xing, et al. 2023. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. arXiv preprint arXiv:2309.11998

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.