{"id":"51db6fab-126a-4912-9722-5fc226f954cb","arxiv_id":"2411.11289","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"This paper presents a CPU-only data curation pipeline for LLMs, but the central claim of high-quality output is not supported by any training or quality evaluation.","lead":"A team from Upstage AI built a data pipeline that cleans and filters web text for training large language models using only CPU machines, avoiding GPU costs. The paper reports time and cost numbers for processing a CommonCrawl dump, but does not show whether the resulting data actually improves model quality.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim of maintaining quality comparable to GPU-based filters is unsupported: no LLM training or external quality comparison is reported, and the Section 3.2.6 FastText filter's fidelity to KenLM labels is never tested.","rationale":"The reader's weakest_assumption identifies Section 3.2.6's FastText classifier and the absence of external validation as the load-bearing gap. My review reaches the same conclusion: the central claim is about data quality, but the paper operationalizes quality only through throughput and corpus size. The pipeline may well be cost-effective, but cost-effectiveness plus large corpus quantities does not establish quality parity with GPU-based methods. The lack of LLM training experiments is acknowledged in the Limitations section, and no substitute evidence (e.g., human evaluation, agreement with an established quality filter, or downstream task performance) is supplied. This is not a disagreement with consensus; it is an internally unsupported claim in the paper itself. I considered whether the companion KenLM paper could supply the missing validation, but this paper does not report or reproduce any such validation for the LP Pipeline's output, so cross-paper reliance does not fill the gap. A system paper could legitimately limit itself to engineering claims about cost and scalability; but because the Abstract and Conclusion explicitly claim maintained quality comparable to GPU-based methods, the burden of evidence falls on the authors. The reader's REJECT verdict is appropriate, and my stress-test does not move it.","tokens_in":12928,"tokens_out":4956,"duration_ms":50175,"concrete_test":"Train a small model (e.g., 1B parameters) on a fixed token budget (e.g., 10B tokens) on three corpora sampled from the same CommonCrawl dump: (a) the LP-filtered corpus produced by Section 3.2, (b) a GPU-filtered corpus such as FineWeb-edu or a reward-model-filtered subset, and (c) a heuristically filtered baseline without Section 3.2.6. Evaluate all models on identical held-out benchmarks (e.g., MMLU, HellaSwag, ARC, plus domain-specific legal/finance/medical tasks in English and Korean). If the LP-filtered model does not perform within a pre-specified margin (e.g., within 1%) of the GPU-filtered model, the central quality-parity claim is falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, stated in the Abstract and Section 5, is that the LP Data Pipeline 'maintains data quality comparable to that of GPU-based methods.' The only empirical evidence in Section 4 is wall-clock time, estimated cost, and document/token counts (Tables 2 and 3). No experiment measures the quality of the output corpus. Section 3.2.6 replaces GPU reward models with a FastText classifier trained on labels produced by the authors' companion Good/Bad KenLM work (Kim et al., 2024b). For the central claim to hold, three links must be sound: (i) KenLM labels approximate true document quality; (ii) FastText trained on those labels reproduces the KenLM ranking on unseen CommonCrawl data; (iii) this ranking is comparable in utility to GPU reward-model filtering for downstream LLM training. None of these links is tested in this paper. The Limitations section explicitly states that 'extensive LLM training experiments using the curated datasets were not conducted,' and no comparison to FineWeb-edu, a reward-model baseline, human judgments, or any external quality benchmark is provided. The claim of quality parity is therefore an assertion, not a demonstrated result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the LP Data Pipeline, a fully CPU-based framework for curating large-scale text corpora for LLM training, with stages for raw text extraction, URL filtering, language identification, domain-grouped line-level deduplication, heuristic filtering, global MinHash deduplication, model-based quality filtering with a FastText classifier trained on Good/Bad KenLM labels, and FastText domain classification for finance, law, and healthcare. The empirical sections report processing time and estimated cost for one CommonCrawl dump, document/token counts for English and Korean domain corpora, and scaled-up counts across ten dumps. The central claim, stated in the abstract and conclusion, is that the pipeline significantly reduces computational cost while maintaining data quality comparable to GPU-based methods such as FineWeb-edu.","tokens_in":13172,"tokens_out":3676,"duration_ms":38528,"significance":"If the central claim were established, the pipeline would be a genuinely useful contribution: CPU-only curation would lower the infrastructure barrier for organizations that cannot afford GPU-based reward-model filtering, and the domain-specific English/Korean corpora could support specialized LLM development. The paper's reported time and cost figures for processing a 4TB CommonCrawl dump are plausible and potentially valuable, and the use of lightweight models is an appropriate direction. However, the paper provides no evidence that the output corpora are actually high quality: there are no LLM training experiments, no downstream task evaluations, no comparison with FineWeb-edu or other curated corpora, no human quality judgments, and no measurement of the FastText filters' fidelity to their training labels. The significance of the claimed quality parity is therefore currently unsubstantiated.","major_comments":[{"comment":"The central claim that the pipeline 'maintains data quality comparable to that of GPU-based methods' is not supported by any quality measurement in the paper. Section 4 reports only processing time, estimated cost, and document/token counts (Tables 2 and 3, Figure 3). The Limitations section explicitly states that 'extensive LLM training experiments using the curated datasets were not conducted,' and no comparison to FineWeb-edu, a reward-model baseline, human judgments, or any external quality benchmark is provided. As stated, the quality-parity claim is an assertion rather than a demonstrated result.","section":"Abstract, Section 5, Limitations"},{"comment":"The model-based quality filter is load-bearing for every downstream quality claim, yet its validity is never tested. A FastText classifier is trained on labels produced by the authors' own Good and Bad KenLM models (Kim et al., 2024b), but the paper does not report the FastText classifier's accuracy, its agreement with the KenLM labels on held-out data, or any analysis of whether the filtered corpus is superior to unfiltered data. Three links are required for the central claim: (i) KenLM scores approximate document quality, (ii) the FastText classifier reproduces those scores on unseen CommonCrawl documents, and (iii) filtering by this classifier yields data comparable in utility to GPU reward-model filtering. None of these links is tested here.","section":"Section 3.2.6"},{"comment":"The heuristic filtering thresholds in Table 1 are described as 'determined through qualitative assessments,' with no validation of their effect on corpus quality and no sensitivity analysis. Since these thresholds discard a large fraction of documents, the paper should at least report how the thresholds were chosen, show that they remove documents independently judged to be low quality, and demonstrate that the surviving documents retain useful content. Without such evidence, the heuristic filtering stage cannot support the claim that the pipeline preserves high data quality.","section":"Section 3.2.4, Table 1"},{"comment":"The domain classification model is trained on labels produced by an LLM, but the paper gives no details of the annotation prompt, no estimate of label quality, no classifier accuracy, and no human evaluation of the resulting domain-specific corpora. Given that the paper's purpose-driven dataset contribution depends entirely on the reliability of this classifier, the absence of any validation of the three-way finance/law/healthcare distinction is a major gap.","section":"Section 3.2.7"}],"minor_comments":[{"comment":"The text states that the data was processed on '128 machines with 8-core CPUs,' but the footnote says the estimated cost was calculated using 120 AWS m7a.2xlarge instances; this discrepancy should be resolved.","section":"Section 4.1"},{"comment":"The claim that total time and cost for ten dumps can be approximated by scaling the single-dump result by a factor of ten is presented as a finding, but it is an assumption rather than an empirical result; no scaling experiment with varying cluster sizes is reported.","section":"Section 4.3"},{"comment":"There is a typo in the subsection title: 'Purpose-Driven Datset Construction' should be 'Purpose-Driven Dataset Construction.'","section":"Section 3.1"},{"comment":"The language identification tool is described only as 'based on FastText'; the specific model or checkpoint should be identified so that the CPU-only claim can be reproduced.","section":"Section 3.2.2"},{"comment":"The proposed domain-grouped line-level deduplication is motivated as more efficient than random bucketing, but no comparison against a random-bucket baseline is provided; the claim of reduced computational cost is therefore not directly demonstrated.","section":"Section 3.2.3"},{"comment":"The paper does not state whether the curated datasets or the pipeline code will be released; for a data-curation contribution, public release of at least a sample or a reproducibility package would strengthen the paper.","section":"General"}],"recommendation":"reject","confidential_remarks":"The paper is essentially an engineering report on a CPU-only data pipeline. The efficiency numbers are interesting, but the central claim of quality parity with GPU-based methods is unverified, and the authors explicitly acknowledge the absence of LLM training experiments. In my view this is not a minor fix: supporting the central claim would require substantial new experiments, and without them the manuscript does not meet the standard for publication. If the authors were to reframe the paper strictly as a cost-efficiency study and remove the quality-parity claim, a resubmission might be considered, but as written the gap between claim and evidence is too large."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read.\n\nThe one thing you should know: this is a legitimate engineering report about a CPU-only data curation pipeline, but the headline claim—quality comparable to GPU-based methods—is not tested anywhere in the paper. The time and cost numbers are concrete; the quality claim is an assertion.\n\nWhat's actually new: the pipeline integrates known pieces (FastText, KenLM, MinHash, heuristic filters) in a sensible order, with two small variations. The domain-grouped line-level deduplication is a nice practical trick that targets boilerplate per site rather than random buckets. And the model-based filter is a FastText classifier trained on labels from the authors' own Good/Bad KenLM work, which is a reasonable way to make a cheap classifier, but it is not independently validated. The cost breakdown for a 4TB CommonCrawl dump (4h22m, ~$353) is useful and the scaling estimate is straightforward.\n\nWhere it's soft: the central quality-parity claim is unsupported. Three links would need to hold—KenLM labels approximate true quality, FastText reproduces KenLM on unseen data, and KenLM filtering is comparable in utility to a GPU reward model for downstream training. None is tested. There is no LLM training eval, no comparison to FineWeb-edu or any external quality benchmark, no human eval, and no error bars on the measurements. The paper's own Limitations section admits that extensive LLM training experiments were not conducted, so this is not a hidden flaw—but the abstract and conclusion state the claim as fact. That overstatement is the main problem. The heuristic thresholds in Table 1 were set by qualitative assessment, which is fine for a recipe but not a validated filter. No code or data is released, so the cost numbers can't be checked independently.\n\nWhat's good: the paper is honest about its main limitation, the related work is appropriate, and the system design is coherent. This is not a sloppy paper; it is a paper that promises more than it demonstrates.\n\nWho it's for: practitioners who want a concrete recipe for building domain-specific corpora on CPUs. For them, the pipeline description and cost model have real value. As a research claim about quality parity, it doesn't hold up yet.\n\nMy recommendation: I would send this to peer review, but with the clear expectation that the authors must add some form of validation—at minimum a small-scale LM training comparison against a baseline like FineWeb-edu, or an intrinsic evaluation showing the FastText filter agrees with Good/Bad KenLM and that KenLM-based filtering gives downstream gains. Without that, it's a technical report, not a validated method. Current form: reject or major revision; the idea deserves a serious look.","headline":"A useful CPU-only data-curation system, but the headline quality-parity claim is untested, so treat it as an engineering report, not a validated method.","tokens_in":13715,"tokens_out":3278,"would_cite":false,"duration_ms":30957,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fully CPU-based pipeline can curate large-scale, domain-specific LLM training data with quality comparable to GPU-based filtering, at a fraction of the cost.","keywords":["data curation","large language models","CPU-only pipeline","quality filtering","domain-specific datasets","CommonCrawl","FastText","KenLM"],"falsifier":"Train two identically sized models, one on an LP-filtered corpus and one on a corpus filtered by a GPU-based reward model, and compare held-out downstream performance; if the LP-filtered model is materially worse, the claim of comparable quality fails. A quicker check: measure agreement between the pipeline's quality classifier and human judgments on a random sample of web documents.","tokens_in":12710,"feed_emoji":"⚙️","tokens_out":9123,"duration_ms":79562,"temperature":0.7,"pith_summary":"The paper sets out to show that high-quality, large-scale training data for large language models does not have to be curated with expensive GPU-accelerated quality filters. It presents the Lightweight, Purpose-driven (LP) Data Pipeline, which runs entirely on CPUs and combines heuristic cleaning, deduplication, and lightweight FastText-based quality and domain classifiers. On a 4TB CommonCrawl dump, the pipeline completed all stages in 4 hours 22 minutes at an estimated cost of about $353, producing separate finance, law, and medical corpora in English and Korean. The central claim is that this CPU-only pipeline maintains data quality comparable to GPU-based reward-model filtering, making LLM development accessible to organizations without heavy GPU infrastructure. The paper notes that direct LLM training experiments on the curated data were not run, so quality equivalence is asserted rather than demonstrated end-to-end.","feed_headline":"CPU-only pipeline claims LLM data quality comparable to GPU for $353","feed_subtitle":"A lightweight classifier replaces GPU reward models, so a 4TB web dump is cleaned in 4 hours for about $353.","key_machinery":"The load-bearing piece is the quality-filtering module: a FastText classifier trained on labels produced by a Good KenLM (an n-gram language model trained on high-quality sources) and a Bad KenLM (trained on toxic and low-quality text), replacing GPU-based reward models for document scoring. FastText is a lightweight linear text classifier, and because it runs on CPUs it lets every stage of the pipeline avoid GPU inference. Supporting machinery includes line-level deduplication grouped by URL domain rather than random buckets, which removes boilerplate while lowering compute, plus MinHash locality-sensitive hashing for global near-duplicate removal and a second FastText model for domain classification. Together these components make the claimed CPU-only, low-cost curation possible.","core_discovery":"The LP Data Pipeline is a sequential, fully CPU-based curation framework built on four principles: CPU-only processing, an optimized stage order that defers expensive operations until after cheap filtering, scheduled continuous updates from fresh web dumps, and purpose-driven construction of domain- and language-specific datasets. Its key novelty is replacing GPU-based reward-model quality filtering with a FastText classifier trained on documents labeled by two KenLM models, one trained on high-quality sources and one on low-quality content; the classifier is then applied at scale on CPUs. A second FastText model, trained using LLM-annotated samples, classifies documents into finance, law, healthcare, or other. The paper reports that processing the 4TB CC-MAIN-2024-10 dump across 128 8-core machines took 4 hours 22 minutes and an estimated $352.83, and that scaling to ten dumps produced 17.95 billion legal, 34.02 billion finance, and 34.84 billion medical tokens for English, with smaller but analogous Korean corpora. The paper's stated conclusion is that the pipeline significantly reduces computational costs while maintaining data quality comparable to that of GPU-based methods, with future LLM training experiments planned to validate that claim.","pith_inferences":["A natural next step, which the paper leaves open, is to train comparable LLMs on LP-filtered and GPU-filtered corpora of equal size; if metrics match, the CPU-only approach becomes a standard baseline for low-resource data curation.","The Good/Bad KenLM labeling scheme could be reused to bootstrap quality filters for new languages cheaply, since it requires no human quality annotations, only high- and low-quality seed corpora.","The domain-grouping deduplication trick is independent of the quality model and could be dropped into existing GPU-based pipelines as a cheap prefilter, reducing the amount of data sent to expensive filters.","The strongest version of the paper's claim, that CPU quality is comparable to GPU quality, will remain untested until someone runs the downstream training experiment; the paper itself flags this as future work."],"forward_implications":["Organizations without GPU clusters can process CommonCrawl-scale dumps for a few hundred dollars in under a day, removing the main compute barrier to custom LLM pretraining.","Purpose-driven corpora for finance, law, and healthcare, and for languages like Korean, can be produced on demand, making specialized LLMs feasible where they previously were not.","Because the pipeline is scheduled to pick up new dumps automatically, LLM training data can stay current without manual curation effort.","If the quality-filtering FastText truly matches GPU reward models, the cost of data curation becomes a minor line item in LLM development budgets rather than a dominant one."],"supporting_citations":[{"why":"Supplies the Good/Bad KenLM labels used to train the quality-filtering FastText classifier.","marker":"Kim et al., 2024b"},{"why":"Defines the GPU-based quality filtering approach whose data quality the paper claims to match.","marker":"Penedo et al., 2024"},{"why":"Provides the FastText classifier used for both quality filtering and domain classification.","marker":"Joulin et al., 2016"},{"why":"Supplies the KenLM language-model implementation that Good and Bad KenLM extend.","marker":"Heafield, 2011"},{"why":"Foundational for the MinHash LSH method used in global deduplication.","marker":"Indyk and Motwani, 1998"},{"why":"Source of heuristic metadata thresholds for the C4-style filtering stage.","marker":"Raffel et al., 2020"},{"why":"Provides additional metadata features adopted for heuristic filtering.","marker":"Longpre et al., 2023"},{"why":"Guides the rule-based filtering stages with RefinedWeb-style practices.","marker":"Penedo et al., 2023"}],"fun_headline_variants":["CPU-only data pipeline: 4TB cleaned in 4 hours for $353","Purpose-driven LLM datasets built with CPUs alone","LP pipeline: GPU-free, $353 for 4TB of curated data","CPU-only pipeline curates 4TB for $353 in 4 hours"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"That a lightweight text classifier, trained on documents labeled good or bad by two language-model scorers, can separate high-quality from low-quality text as reliably as expensive GPU-based reward models, even though the paper never checks this against independent quality judgments.","fun_headline_variants_meta":{"raw":{"variants":["CPU-only data pipeline: 4TB cleaned in 4 hours for $353","Purpose-driven LLM datasets built with CPUs alone","LP pipeline: GPU-free, $353 for 4TB of curated data","CPU-only pipeline curates 4TB for $353 in 4 hours"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001306,"raw_usage":{"total_tokens":5323,"prompt_tokens":940,"completion_tokens":4383,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":4305}},"tokens_in":556,"tokens_out":4383,"duration_ms":31025,"temperature":1.0,"reasoning_tokens":4305,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:41:04.620331+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two identically sized models, one on an LP-filtered corpus and one on a corpus filtered by a GPU-based reward model, and compare held-out downstream performance; if the LP-filtered model is materially worse, the claim of comparable quality fails. A quicker check: measure agreement between the pipeline's quality classifier and human judgments on a random sample of web documents.","supporting_citations":[],"review_version":1}