{"id":"a466751d-7d52-42de-9ceb-d8d303387661","arxiv_id":"2506.14111","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A 24T-token web corpus with automatic document-level taxonomy labels enables competitive domain-specific datasets via simple filters.","lead":"Essential-Web v1.0 is a 24-trillion-token web dataset in which every document is labeled with a 12-category taxonomy covering topic, format, complexity, and quality. The authors show that simple SQL filters on these labels can produce competitive pre-training datasets for math, code, STEM, and medical domains.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's headline numbers conflate pure-taxonomy filters with hybrid pipelines: math -8% uses FineMath, STEM/medical gains use DCLM, so 'nothing more than SQL-style filters' is not established.","rationale":"The reader's CONDITIONAL verdict already notes that STEM and medical gains come from combined taxonomy+DCLM filters; my concern sharpens this into the central correctness issue by showing that the math headline also depends on a domain-specific FineMath classifier, making the abstract's 'nothing more than SQL-style filters' claim inconsistent with the experiments as reported. I did not select the reader's weakest_assumption (student label quality at scale) as the primary concern because the filters that drive the headline results use FDC, Document Type, Reasoning Depth, and Technical Correctness, while the categories with the largest student-teacher kappa drops (Extraction Artifacts, Missing Content) are not used in the downstream filters; the held-out set size remains a secondary worry. The paper's internal tables are transparent, so this is a scoping/overclaiming issue rather than a sign of methodological fraud. Because the dataset and the hybrid-pipeline results retain substantial value, the appropriate verdict remains CONDITIONAL, with the condition that the abstract and headline claims be re-scoped to distinguish pure-taxonomy filters from taxonomy+classifier pipelines. Hence UNCHANGED.","tokens_in":42008,"tokens_out":8863,"duration_ms":87419,"concrete_test":"Re-run the downstream annealing evaluation for the pure-taxonomy filters only (Algorithms 2, 4, 6, 8), without any added FineMath or DCLM thresholds, and recompute the four headline relative gains in the abstract against the same SOTA baselines. If the math number reverts to roughly -15% on GSM8K and the STEM/medical gains shrink to +5% or to within standard error, the abstract must be re-scoped to say 'taxonomy metadata combined with existing quality classifiers' rather than 'nothing more than SQL-style filters.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the abstract is that SQL-style filters over the taxonomy alone yield competitive math (-8.0%), code (+14.3%), STEM (+24.5%), and medical (+8.6%) datasets. The paper's own tables show this is not what the headline numbers measure. In math, EAI-TAXONOMY Top Math (Algorithm 2, pure taxonomy) trails FineMath 3+ by 19.3% relative on GSM8K (21.3 vs 26.4) and by 6.0% on MATH; the abstract-level -8.0% figure comes from EAI-TAXONOMY Math w/ FM (Algorithm 3), which runs the FineMath classifier, a domain-specific math classifier, over 116M recalled documents. That is an additional bespoke curation step, not 'nothing more than SQL-style filters.' For STEM and medical, the +24.5% and +8.6% gains are from the w/ DCLM variants (Algorithms 9 and 7), which add the DCLM instruction-density classifier threshold; the pure taxonomy STEM gain is +5.1% relative (29.1 vs 27.7) and the pure taxonomy medical results are mixed (e.g., PubMedQA 67.0 vs DCLM 70.6, MMLU-Med 29.5 vs 31.0). The paper is transparent in the per-dataset tables, but the abstract's central claim overstates what pure taxonomy filters deliver. The load-bearing question is whether the promised 'no domain-specific training' result is actually demonstrated by the headline numbers; currently it is demonstrated only for the hybrid taxonomy+classifier pipelines.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Essential-Web v1.0, a 24-trillion-token, deduplicated Common Crawl dataset in which every document carries labels from a 12-category taxonomy (subject, document type, content complexity, quality, extraction issues). The labels are produced by EAI-Distill-0.5b, a distilled 0.5B model trained on synthetic labels from Qwen2.5-32B-Instruct. The authors argue that simple SQL-style filters over the taxonomy yield competitive domain-specific pretraining datasets for math, code, STEM, and medicine, without the bespoke pipelines used by prior work. They also propose a metric suite (NMI, annotator kappa, domain recall) for taxonomy evaluation and report downstream annealing experiments on 2.3B-parameter models with decontamination and controlled base models.","tokens_in":42319,"tokens_out":6389,"duration_ms":64339,"significance":"If the central claim holds, this is a substantial contribution to reproducible data-centric LM research: a 24T-token public corpus with interpretable document-level metadata could let practitioners assemble domain-specific datasets in hours rather than months, and the release of the dataset and annotator model would be a community resource. The downstream evaluation is careful in design: identical base models, the same annealing protocol, and a 13-gram decontamination filter. The domain-recall metric and the use of external gold annotators (GPT-4o, Claude) are also strengths. However, the headline empirical claims overstate what the pure taxonomy filters deliver, and the full-scale label-quality evidence is thinner than the scale of the release requires. The paper needs a major revision to align the abstract and contributions with the actual pipelines and to provide stronger validation of label quality at corpus scale.","major_comments":[{"comment":"The abstract's claim that 'with nothing more than SQL-style filters' yields math (-8.0%), web code (+14.3%), STEM (+24.5%), and medical (+8.6%) is not supported by the tables. The math figure is achieved by Algorithm 3, which applies the FineMath classifier to 116M recalled documents; the pure-taxonomy Top Math dataset trails FineMath 3+ by 19.3% relative on GSM8K (21.3 vs 26.4, Table 3), even though it is close on MATH and MMLU-Math. The STEM +24.5% and medical +8.6% figures correspond to the w/ DCLM variants (Algorithms 9 and 7), while pure taxonomy STEM gains are +5.1% relative (29.1 vs 27.7, Table 9) and pure medical results are mixed (e.g., PubMedQA 67.0 vs 70.6 for DCLM-baseline, Table 7). The paper is transparent in the per-dataset tables, but the abstract, the introduction, and contribution 2 in §2.1 overstate what the pure taxonomy filters deliver. Please revise the abstract and contributions to attribute each headline number to the exact pipeline, or report pure-taxonomy headline numbers.","section":"Abstract; §4.2–4.5, Tables 3/5/7/9, Algorithms 3/5/7/9"},{"comment":"The quality of the student model's labels at full 23.6B-document scale is a load-bearing premise, but the evidence is thin. The student is evaluated on only 2,017 random and 871 STEM documents, and on categories that drive the filters it shows large agreement drops relative to the teacher: Extraction Artifacts random kappa falls from 0.74 to 0.27, and Missing Content from 0.66 to 0.48 (Table 13). At the same time, the student anomalously exceeds the teacher on Reasoning Depth (0.87 vs 0.67 random) and Technical Correctness (0.72 vs 0.52 random). Since Algorithms 2–9 rely on exactly these categories, and no held-out evaluation is performed at corpus scale, the label distribution in Essential-Web v1.0 could differ substantially from these small sets. Please add a stratified validation sample (e.g., 5–10k documents) comparing EAI-Distill-0.5b labels against the teacher or gold annotators, at least for the categories used in the downstream filters.","section":"§6.3, Table 13; §5.1.4"},{"comment":"The paper does not report how the taxonomy filter rules and thresholds were selected. If any of the label sets or thresholds (e.g., DCLM_baseline_thresh = 0.01811, the reasoning-depth and technical-correctness criteria, or the FDC+document-type combinations) were chosen after inspecting results on the same benchmarks, the 'simple filters' comparison is partially circular. Please state the filter development procedure, hold out the downstream benchmarks when tuning, or provide ablation results showing that reasonable perturbations of the filters do not change the qualitative conclusions.","section":"§4.1, Algorithms 2–9"}],"minor_comments":[{"comment":"Algorithm 6 references d.dds.primary and d.dds.secondary, but the dataset schema described elsewhere uses d.fdc; please fix this typo so the medical filter is reproducible.","section":"Appendix A.8, Algorithm 6"},{"comment":"The statement that EAI-Distill-0.5b remains 'within 3%, 14%, 1% of the teacher' on annotator agreement, NMI, and domain-recall is not accurate as written: Table 14 shows the average inter-category NMI increases by 16.5% on the random set (0.079 to 0.092), and Table 13 shows the random-set kappa drops by 4.1% (0.74 to 0.71). Please qualify these numbers by evaluation set or report the exact range.","section":"§2.1, contribution 4; §6.3.2, Table 14"},{"comment":"The abstract says EAI-Distill-0.5b achieves 'annotator agreement within 3%' of Qwen2.5-32B-Instruct, but Table 13 shows a 4.1% relative drop on the random set; please specify that the 3% figure refers to the STEM set or report the average over both sets accurately.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The dataset release and evaluation framework are valuable, and the paper contains enough transparency in the per-dataset tables to fix the overclaim. I see no grounds for reject, but the abstract and contribution list must be revised to state which pipelines produce each headline number, and the label-quality validation needs to be strengthened before the central claim can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the Essential-Web v1.0 paper. Bottom line: this is a serious, large-scale data release, but the abstract's central claim is overstated. The \"nothing more than SQL-style filters\" line is true only for some of the reported numbers; the math, STEM, and medical headline gains come from adding domain-specific classifiers (FineMath or DCLM) on top of the taxonomy. The stress-test note is right.\n\nWhat's genuinely new: a 24T-token corpus with 12-category document-level labels, plus a distilled 0.5B annotator that runs 50x faster than the teacher with only a small average kappa drop. The evaluation protocol is careful: same base model, annealing, decontamination, and they compare against the right open baselines. They also ship the datasets and the annotator on HuggingFace, which makes this reproducible. The domain-recall tests on vetted URL sets are a good addition.\n\nSoft spots, in order of importance:\n\n1) Abstract overclaims. In the per-dataset tables, pure taxonomy math (EAI-TAXONOMY Top Math) trails FineMath 3+ by ~19% relative on GSM8K; the abstract's -8% comes from the FineMath-classifier variant. Pure taxonomy STEM gives +5% relative over DCLM, not +24.5%; the big gain is from adding DCLM's instruction-density classifier. Pure taxonomy medical results are mixed, sometimes below DCLM baseline. This doesn't invalidate the corpus, but the headline numbers need to be labeled as hybrid.\n\n2) Label quality at full scale is a load-bearing assumption. The student's kappa on Extraction Artifacts collapses from 0.74 to 0.27 (and Missing Content to 0.48), and the held-out sets are only 2,017 random and 871 STEM docs. They didn't validate the 0.5B labels on a large independent sample. This matters because filtering on technical_correctness or reasoning_depth depends on labels being consistent across billions of docs.\n\n3) Filter thresholds and design are underdocumented. The DCLM threshold is reused, but the taxonomy filter choices are manual and not ablated beyond the main tables. Some free parameters, so the \"no tuning\" claim is weaker than advertised.\n\nAll that said, the paper is transparent enough that a reader can reverse-engineer the actual results. It's a valuable resource for the community. I'd send it to peer review, but the authors need to rewrite the abstract to distinguish pure taxonomy filters from hybrid pipelines and add at least one large-scale label-validation experiment. It's not desk-reject material.\n\nRecommendation: engage with it; cite the dataset if you're working on web-scale data curation, but trust the tables more than the abstract.","headline":"Substantial dataset release, but the abstract's 'SQL-style filters' headline conflates pure taxonomy with taxonomy+classifier hybrids; the per-dataset tables tell a more modest and honest story.","tokens_in":42986,"tokens_out":3862,"would_cite":true,"duration_ms":40275,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Twenty-four trillion web tokens, each document tagged with a twelve-category taxonomy, reduce domain data curation to SQL queries that match or beat specialist pipelines on math, code, STEM, and medical benchmarks.","keywords":["Essential-Web","web-scale pretraining corpus","data curation","document taxonomy","Common Crawl","taxonomy distillation","SQL-style filtering","domain-specific datasets"],"falsifier":"Take a fresh sample of roughly 20,000 documents from the released corpus, have the two reference annotators (GPT-4o and Claude Sonnet-3.5) relabel them, and compare per-category agreement against EAI-Distill-0.5b's stored labels. If categories such as Extraction Artifacts (kappa 0.27 on the current 2,017-document set) and Missing Content (kappa 0.48) show similarly low agreement at this larger scale, and if re-running the paper's four published filter algorithms on the corrected labels no longer reproduces the reported margins (math within 8.0%, web code +14.3%, STEM +24.5%, medical +8.6%), the central claim that taxonomy quality carries to the full corpus fails.","tokens_in":41792,"feed_emoji":"🗂️","tokens_out":12781,"duration_ms":111982,"temperature":0.7,"pith_summary":"This paper claims that the costliest part of open model-building — bespoke, months-long pipelines that extract domain corpora (math, code, medicine) from the web — can be replaced by one organized corpus plus a query. The authors release Essential-Web v1.0, a 24-trillion-token Common Crawl corpus in which each of 23.6 billion documents carries labels for twelve taxonomy categories spanning subject matter, page type, content complexity, and quality. Using nothing more than SQL-style filters over those labels, they curate datasets that score within 8.0% of the best open web-curated baseline on math and beat it by 14.3% on web code, 24.5% on STEM, and 8.6% on medical benchmarks. If the claim holds, pre-training data curation becomes a search problem any researcher can solve in minutes, and the one-off cost of labeling the corpus is amortized across everyone who uses it.","feed_headline":"SQL filters on 24T labeled web tokens rival specialist pipelines","feed_subtitle":"Math lands within 8% of state of the art; code, STEM, and medical beat it by 9-25% — straight from metadata queries.","key_machinery":"The load-bearing object is EAI-Taxonomy, a fixed twelve-category labeling scheme whose categories act as orthogonal axes: each document receives one primary and one optional secondary label per category, so a curated dataset is exactly a conjunction of predicates over those columns. Subject matter uses Free Decimal Correspondence, a public-domain analogue of the Dewey Decimal System, whose three hierarchical levels let a filter like 'FDC = 51' isolate mathematics at 98% recall while keeping 0.5% of the crawl. The second machinery piece is EAI-Distill-0.5b, a 0.5B-parameter annotator fine-tuned with context distillation and condensed outputs, which raises throughput to 70 requests per second per GPU and makes labeling 23.6 billion documents feasible at roughly 90k GPU-hours. A third piece is the paper's metric suite — normalized mutual information for category independence, a set-overlap variant of Cohen's kappa for label clarity, and URL-based domain recall for retrieval power — which is what justifies the taxonomy's shape before any downstream training.","core_discovery":"On the paper's own terms, the discovery is that a document-level taxonomy turns corpus curation into a query interface. Every document in Essential-Web v1.0 is annotated with EAI-Taxonomy, a fixed twelve-category scheme covering subject (three levels of the public-domain Free Decimal Correspondence), cognitive demand (Bloom's taxonomy), page type (two overlapping label sets), content quality (reasoning depth, educational level, technical correctness), and extraction health (artifacts, missing content), with a primary and optional secondary label per category. The labels come from EAI-Distill-0.5b, a 0.5B-parameter model fine-tuned on 82B tokens of teacher labels, that matches Qwen2.5-32B-Instruct within 3% on annotator agreement while running roughly 50 times faster. With filters alone — for example FDC code 51 (Mathematics) plus thresholds on reasoning depth and technical correctness — annealed 2.3B-parameter models land within standard error of specialist corpora: from 8.0% behind state of the art on math to 24.5% ahead on STEM, with the strongest results coming from combining taxonomy filters with an instruction-density classifier.","pith_inferences":["My read: if label quality survives at scale, the quality axes of the taxonomy (reasoning depth, technical correctness) could replace monolithic quality classifiers in general-purpose pipelines, because they are per-document, interpretable, and queryable rather than a single opaque score.","The paper's own numbers suggest a testable recipe for new domains: a subject-code filter plus reasoning/correctness thresholds may be a sufficient universal prior, making even the DCLM classifier combination unnecessary for domains with clean subject codes.","The largest internal risk is the student model's per-category agreement drops — Extraction Artifacts kappa falls from 0.74 to 0.27 and Missing Content from 0.66 to 0.48 — so the two extraction-health categories are the ones most likely to fail at full-corpus scale; a larger held-out sample would settle this.","Because the taxonomy is built on a public-domain subject hierarchy, the same filter definitions could become standardized, reusable data-selection recipes — effectively turning every domain benchmark into a benchmark of data curation methods."],"forward_implications":["A practitioner who wants, say, a chemistry corpus can write a metadata query in under 15 minutes instead of training a high-recall classifier, running it over hundreds of millions of pages, and manually inspecting the output.","Combining taxonomy filters with an existing instruction-density classifier is the paper's strongest recipe: it lifts STEM 24.5% and medical 8.6% over baselines and adds 15.0 percentage points over DCLM-baseline on MMLU computer science.","The taxonomy is dense enough to audit other datasets: annotating FineMath 3+ with it shows only 61.9% of that 'math' corpus is labeled Mathematics, with physics, economics, and applied physics making up much of the rest.","Domain-recall measurements show simple filters retrieve 95.8-97.6% of human-vetted math and code pages while keeping under 5% of the crawl, a far higher density than fastText classifiers at comparable volume.","The one-off labeling cost (about 90k MI300x GPU-hours) is a fixed investment that the community amortizes, so subsequent domain corpus construction is nearly free."],"supporting_citations":[{"why":"Technical report for the Qwen2.5 family; the 32B-Instruct variant is the teacher whose labels train EAI-Distill-0.5b.","marker":"Qwen [2025]"},{"why":"Supplies the normalized mutual information metric and the Document Type V2 label set that EAI-Taxonomy reuses.","marker":"Wettig et al. [2025]"},{"why":"Defines the Free Decimal Correspondence hierarchy that carries the subject-matter filters.","marker":"Ockerbloom [2010]"},{"why":"DataComp-LM provides the DCLM Pool corpus, the DCLM fastText classifier used in several filters, and the DCLM-baseline comparison set.","marker":"[Li et al., 2025]"},{"why":"FineMath 3+ is the math state-of-the-art baseline and supplies the classifier behind the 'Math w/ FM' filter.","marker":"Allal et al. [2025]"},{"why":"DeepSeekMath is a math baseline and the iterative-recall paradigm behind the domain-recall metric.","marker":"[Shao et al., 2024]"},{"why":"OpenCoder's FineWeb Code Corpus is the only open large-scale web code baseline.","marker":"[Huang et al., 2025]"},{"why":"MegaMath is a further web-math baseline for the math comparison table.","marker":"[Zhou et al., 2025]"},{"why":"TheBlueScrubs-v1 is the medical baseline the taxonomy-based medical datasets are measured against.","marker":"[Felipe et al., 2025]"},{"why":"The Stack v2 Python subset forms half of the Code-Base model used in the annealing protocol for code evaluation.","marker":"[Lozhkov et al., 2024]"}],"fun_headline_variants":["24T tokens get 12 labels; filters beat specialist sets","Small labeler, big win: 0.5B tags 24T tokens, rivals 32B","Query your corpus: SQL filters on 24T labeled web docs","Taxonomy turns 24T web docs into a queryable dataset","12-label scheme on 24T tokens enables SQL-like curation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the 0.5B annotator's labels are as accurate across all 23.6 billion documents as they are on the small held-out sets it was checked on (2,017 random and 871 STEM documents), even though agreement drops sharply for some categories in the student model, with Extraction Artifacts falling from kappa 0.74 to 0.27.","fun_headline_variants_meta":{"raw":{"variants":["24T tokens get 12 labels; filters beat specialist sets","Small labeler, big win: 0.5B tags 24T tokens, rivals 32B","Query your corpus: SQL filters on 24T labeled web docs","Taxonomy turns 24T web docs into a queryable dataset","12-label scheme on 24T tokens enables SQL-like curation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1287,"prompt_tokens":985,"completion_tokens":302,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":204}},"tokens_in":601,"tokens_out":302,"duration_ms":3568,"temperature":1.0,"reasoning_tokens":204,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:17:51.357866+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh sample of roughly 20,000 documents from the released corpus, have the two reference annotators (GPT-4o and Claude Sonnet-3.5) relabel them, and compare per-category agreement against EAI-Distill-0.5b's stored labels. If categories such as Extraction Artifacts (kappa 0.27 on the current 2,017-document set) and Missing Content (kappa 0.48) show similarly low agreement at this larger scale, and if re-running the paper's four published filter algorithms on the corrected labels no longer reproduces the reported margins (math within 8.0%, web code +14.3%, STEM +24.5%, medical +8.6%), the central claim that taxonomy quality carries to the full corpus fails.","supporting_citations":[{"cited_title":"Free decimal correspondence","cited_arxiv_id":null,"evidence_quote":"Defines the Free Decimal Correspondence hierarchy that carries the subject-matter filters."},{"cited_title":"TheBlueScrubs-v1, a comprehensive curated medical dataset derived from the internet","cited_arxiv_id":"2504.02874","evidence_quote":"TheBlueScrubs-v1 is the medical baseline the taxonomy-based medical datasets are measured against."}],"review_version":1}