{"id":"1f469db6-8acb-44e6-98d3-279236569eeb","arxiv_id":"2411.19668","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ChineseWebText2.0 is a 3.8TB Chinese web text dataset with per-text quality scores, domain labels, toxicity labels and toxicity scores, built and released by the paper's MDFG-tool pipeline.","lead":"This paper releases ChineseWebText2.0, a 3.8 terabyte Chinese web text corpus in which every text carries a quality score, domain labels, and a toxicity label and score. The resource matters because it lets Chinese LLM researchers filter pretraining data by domain, quality, and toxicity instead of training on raw, unlabeled web text.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Toxicity evaluation in §3.5.2 (Table 4) is internally inconsistent, so the reported precision is unreliable and the 0.99-threshold toxic subset lacks validated precision.","rationale":"The paper's central offer is a 3.8TB corpus with per-text quality, domain, and toxicity annotations. The weakest point in the evidence chain is the toxicity annotation, because it is the only annotation for which the paper claims a safety-critical use and a 'largest toxicity dataset' milestone. The reader flagged the lack of validation at the 0.99 threshold; this review goes further: the validation that exists (Table 4) cannot be correct as written. With 300 toxic and 300 benign test samples, precision = 83.67% and specificity = 97.67% cannot both hold given TP = 251 and TN = 293. This suggests a label swap (precision is actually recall) or an incorrect denominator; either way, the actual false-positive rate at the operating threshold used to produce the toxic subset is unknown. The threshold of 0.99 is selected after looking at the score distribution, so there is no independent estimate of precision at that point. This does not invalidate the dataset as a resource, but it does mean the 'high-quality' and 'fine-grained toxicity' claims are not supported by the reported evidence. The requested changes (correct the evaluation, validate the 0.99 threshold on a held-out human-annotated sample, and release the validation sets) should be conditions for acceptance. The reader's CONDITIONAL verdict is appropriate; I do not think the issues justify rejection because the dataset and pipeline may still be usable once the evaluation is fixed.","tokens_in":15743,"tokens_out":8194,"duration_ms":63856,"concrete_test":"First, derive the full 2x2 confusion matrix from the reported TP, FP, TN, FN and compare with the stated 300/300 balance; if it does not reconcile, the table must be corrected. Second, human-annotate a random sample of ~300 texts from the 1,632,620-text subset with toxicity scores above 0.99 (Section 4.4) and compute precision at that operating point; report a confidence interval. This directly tests whether the released toxicity labels and the 'largest toxicity dataset' claim are accurate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The toxicity classifier's only reported validation (Section 3.5.2, Table 4) is mathematically inconsistent with the stated balanced test set. The text says the test set contains 300 toxic and 300 benign samples. Table 4 reports precision = 83.67% with TP = 251 and TP+FP = 300, implying FP = 49, and specificity = 97.67% with TN = 293 and TN+FN = 300, implying FN = 7. If these counts are correct, actual toxic = TP+FN = 258 and actual benign = FP+TN = 342, not 300/300. If the test set is actually balanced, then either TP+FP = 300 is really TP+FN = 300 (making 'precision' a recall), or one of the denominators is wrong. The paper's formula (4) also defines 'Specificity' as TN/(TN+FN) instead of the standard TN/(TN+FP). This error is not cosmetic: precision at the default operating threshold is not established, and the paper then uses an empirically chosen threshold of 0.99 to extract a 1.63M-text 'toxic' subset without any human-validated precision at that threshold. Because the released toxicity labels are a central advertised feature, the evaluation must be repaired before the annotation can be relied on.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MDFG-tool, a pipeline that combines rule-based cleaning, a BERT-based quality model, a FastText domain classifier, and a FastText toxicity model to annotate Chinese web text with quality scores, domain labels, toxicity labels, and toxicity scores. The authors collect roughly 6.6 TB from public Chinese datasets, reduce it to 3.8 TB through preparation and preprocessing, and release the resulting corpus as ChineseWebText2.0, including a 3.16 GB subset of texts classified as toxic. The paper evaluates each annotation component, reports corpus statistics and distributions, and states that data, code, and the tool chain are publicly available.","tokens_in":15955,"tokens_out":7533,"duration_ms":63402,"significance":"If the annotations are reliable, this is a significant resource paper: ChineseWebText2.0 would be the largest public Chinese pre-training corpus with per-text quality, domain, and toxicity metadata, and the toxic subset would be the largest Chinese toxicity dataset with continuous toxicity scores. The release of code and the tool chain is a concrete reproducibility strength, and the corpus would directly support domain-specific and safety-oriented work on Chinese LLMs. The contribution is, however, contingent on the validity of the annotation results, and the toxicity evaluation currently contains internal inconsistencies that undermine confidence in the released labels. The paper is valuable in scope and ambition, but the validation of the central advertised annotations needs repair before the resource can be relied on.","major_comments":[{"comment":"The reported toxicity evaluation is internally inconsistent. With a balanced test set of 300 toxic and 300 benign texts, precision = TP/(TP+FP) and the printed values TP=251, TP+FP=300 imply FP=49; specificity as printed, TN/(TN+FN)=293/300, implies FN=7. These counts cannot both hold, and they would imply actual class totals of Toxic=258 and Benign=342, contradicting the stated 300/300 balance. If the counts are instead taken as TP=251, FN=49, TN=293, FP=7, then the reported value 83.67% is recall, not precision, and the standard precision would be 251/(251+7)=97.29%. In addition, Eq. (4) defines specificity as TN/(TN+FN), whereas the standard definition is TN/(TN+FP). Because the released toxicity labels are a central advertised feature, this evaluation must be corrected and re-reported before the labels can be relied on.","section":"§3.5.2, Table 4 and Eq. (4)"},{"comment":"The threshold of 0.99 used to extract the 1,632,620-text toxic subset is introduced through 'manual analysis of the toxicity scores', but no precision or recall is reported at this operating point. The only model evaluation in §3.5.2 is at the default decision threshold, and that evaluation is itself misreported. Since the paper releases a 3.16 GB subset as the 'largest Chinese toxicity dataset', the authors should sample texts with scores in intervals around 0.99, have them human-annotated, and report precision, recall, and confidence intervals at the actual extraction threshold. Without this, the size claim does not establish the reliability of the toxic subset.","section":"§4.4, Figure 10"},{"comment":"The LLM-in-the-loop step uses Qwen2.5-32B-Instruct to label candidate texts that are added to the training set, but the paper reports no human verification or agreement analysis for these LLM-generated labels. This matters because the teacher labels shape the training distribution of the final toxicity classifier, and the only downstream validation is the problematic Table 4. I recommend reporting an agreement statistic (e.g., Cohen's kappa) between the LLM labels and human annotations on a held-out sample, and describing how disagreements were resolved.","section":"§3.5.1"},{"comment":"The domain classifier is validated on only 300 manually annotated texts, with some classes represented by 2–20 test examples (math=2, law=13). The iterative optimization in §3.4.1 uses model predictions to update the rule-based keyword sets, so the training signal is partly self-generated; a 300-text test set is too small to establish per-class reliability, especially for the multi-label micro-averaged metrics, and the single-label recall of 64.15% already indicates substantial under-coverage. Please report per-class precision and recall with confidence intervals, and clarify how the test texts were kept disjoint from all iterations of the self-training process.","section":"§3.4.2 and §3.4.3"},{"comment":"The human evaluation for quality scores is described inconsistently: §3.3 says 50 samples per score interval, §4.2 and Figure 7 say 100 samples per interval, and Appendix 5.1 says 1,000 examples total. The appendix also reports no inter-annotator agreement statistic despite using five annotators. Since the acceptance rate is already 80% in the lowest interval [0.1,0.2), the current evaluation has limited power to show that the quality scores are meaningfully discriminative. Please reconcile the sample sizes, report agreement, and provide acceptance rates per interval with confidence intervals.","section":"§3.3, §4.2, and Appendix 5.1"}],"minor_comments":[{"comment":"The manuscript contains several typos that should be corrected, including 'satefy' in the Introduction, 'Pratly Public' in Table 5, 'Classfier' in Figure 1, 'Initail' in §3.5.1, and inconsistent capitalization of 'BertEval'/'B ERTEval'.","section":"Throughout"},{"comment":"References [9] and [2] are the same T5 paper, reference [12] is formatted as a title rather than an authorial citation, and references [28] and [29] are incomplete URL-style entries; these should be cleaned up.","section":"References"},{"comment":"The table would be more informative if it reported recall and F1 for the toxic class in addition to precision, since the paper currently claims only precision and specificity but uses a balanced test set that also permits recall and F1.","section":"§3.5.2, Table 4"},{"comment":"The two panels of Figure 7 use different y-axis scales and the caption does not state this; the visual comparison would be clearer with a shared scale or an explicit note.","section":"§4.2, Figure 7"},{"comment":"The comparison in Table 7 mixes labeled datasets with unlabeled or weakly labeled collections; the caption should state which entries have human-verified labels and which are model-generated.","section":"§4.4, Table 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is a resource contribution with a substantial engineering component, and the data/code release is a genuine strength. The main risk is that the toxicity label validation is not currently trustworthy: the reported precision is internally inconsistent, and the 0.99 extraction threshold has no human-validated precision estimate. These are fixable with targeted experiments, so I would support publication after the authors provide corrected metrics and threshold-specific validation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Brief take: the resource is real and worth engaging, but the toxicity evaluation has a load-bearing arithmetic inconsistency and an unvalidated threshold, so the paper needs revision before its safety claims can be trusted.\n\nWhat's new: a 3.8TB Chinese pretraining corpus with per-text quality scores, domain labels, and toxicity labels/scores, plus an open tool-chain. That combination on this scale is new. The pipeline is described in enough detail to reproduce, and the quality-scoring human-acceptance curve shows a sensible monotonic relationship. Domain classification at 88% single-label precision is okay given only 300 test texts. The toxicity subset, if valid, would be the largest Chinese toxicity dataset; that's a genuinely useful asset for safety evaluation.\n\nWhere it's soft: the toxicity evaluation in Section 3.5.2 doesn't add up. Table 4 reports precision 83.67% with TP=251 and TP+FP=300 (so FP=49) and specificity 97.67% with TN=293 and TN+FN=300 (so FN=7). That would imply 258 actual toxic and 342 actual benign, not the stated balanced 300/300. Also, Formula (4) defines specificity as TN/(TN+FN), which is recall of the negative class, not specificity. These aren't cosmetic: the reported precision is the only quantitative support for the model, and it's unreliable. Then Figure 10 sets the toxic threshold at 0.99 without any human-validated precision at that operating point. The released toxicity labels inherit that uncertainty. There are also smaller inconsistencies: the quality-evaluation human sample is described as 50 per interval in Section 3.3 but 100 per interval in Section 4.2; and the test sets for domain (300) and toxicity (600) are small, so the error bars are wide. The circular elements—quality model seeded from ChineseWebText1.0, domain classifier self-training, LLM teacher without human verification—are worth acknowledging, but they don't by themselves sink the dataset; they just temper the \"high-quality\" claim.\n\nBottom line: this is a potentially valuable infrastructure contribution for Chinese LLM work. It deserves serious peer review, but the toxicity validation has to be fixed: re-run the evaluation with consistent counts, report precision at the actual threshold, and ideally validate on a sample drawn from the release distribution. I'd send it to review with the expectation of major revision, and I'd cite the corpus only with those caveats in mind.","headline":"A large Chinese pretraining corpus with useful annotations, but the toxicity evaluation is internally inconsistent and the 0.99 threshold lacks validation.","tokens_in":16549,"tokens_out":2248,"would_cite":true,"duration_ms":18465,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 3.8TB Chinese web-text corpus labels every document with quality, domain, and toxicity annotations via a public tool-chain, MDFG-tool.","keywords":["ChineseWebText2.0","Chinese pre-training corpus","fine-grained text annotations","quality scoring","domain classification","toxicity detection","data filtering"],"falsifier":"Take a random sample of texts from the released 3.16GB toxic subset and have human annotators judge toxicity; if the observed precision falls far below 83.67%, the claimed reliability of the toxicity labels and the usefulness of the toxic subset for safety work would be undermined.","tokens_in":15495,"feed_emoji":"📚","tokens_out":9660,"duration_ms":68973,"temperature":0.7,"pith_summary":"This paper describes the construction and public release of ChineseWebText2.0, a 3.8TB Chinese web-text corpus in which every text carries four fine-grained annotations: a quality score, domain labels, a toxicity label, and a toxicity score. The proposed tool-chain, MDFG-tool, removes explicit noise with handcrafted rules and then runs a BERT-based quality evaluator, a FastText domain classifier, and a FastText toxicity evaluator refined by an LLM-in-the-loop procedure to attach the annotations. The release gives LLM researchers the ability to select Chinese pre-training data by quality, domain, and toxicity at a scale and granularity not previously available, including a 3.16GB toxic subset that the paper identifies as the largest publicly available Chinese toxicity dataset. The data, code, and tool-chain are released with the paper.","feed_headline":"Chinese corpus hits 3.8TB with quality, domain, and toxicity tags","feed_subtitle":"Every document carries a quality score, domain labels, a toxicity label and score, enabling selective LLM pre-training.","key_machinery":"The load-bearing object is MDFG-tool, a pipeline that converts raw Chinese web text into annotated records in two stages. The first stage applies handcrafted rules: average line length at least 10 characters, document length at least 200 characters, at least 30% Chinese characters, a sensitive-word line filter, and 13-gram deduplication. The second stage runs three annotation models: a BERT-based quality evaluator that combines text-level and max-pooled token-level representations with MSE, margin-ranking, and cosine-similarity losses plus self-training; a FastText domain classifier seeded by keyword rules and iteratively retrained; and a FastText toxicity classifier trained on combined public Chinese toxicity datasets and refined through two LLM-in-the-loop iterations. The pipeline's output is one record per text carrying quality score, domain labels, toxicity score, and toxicity label.","core_discovery":"ChineseWebText2.0 is presented as the largest publicly available Chinese pre-training corpus, totaling 3.8TB after rule-based cleaning, and every text in it receives four annotations: a quality score from a BERT-based evaluator, domain labels from a FastText classifier over 11 domains, and a toxicity score plus toxicity label from a FastText evaluator refined by an LLM-in-the-loop procedure. On manual evaluation, the paper reports that quality scores correlate positively with human acceptance, domain classification reaches 88.33% precision for single-label prediction and 74.48% micro-precision for multi-label prediction, and toxicity classification reaches 83.67% precision and 97.67% specificity on a balanced test set. The released package also includes a toxic subset of 1,632,620 texts (3.16GB), which the paper identifies as the largest publicly available Chinese toxicity dataset.","pith_inferences":["Because the reported toxicity precision comes from a balanced 600-text test set, users should validate precision on a sample drawn from the actual 0.99-threshold toxic subset before relying on those labels for training or evaluation.","The quality-score human acceptance curve stays above 80% even at the lowest score interval, so quality filtering is likely most useful for removing a noisy tail rather than for fine-grained ranking of typical web text.","The Math domain accounts for only 0.55% of the corpus, so users training mathematical LLMs should expect to supplement this dataset with specialized sources despite the fine-grained domain labels."],"forward_implications":["Researchers can build domain-specific pre-training subsets, such as law or medicine, by filtering on the domain labels and a quality threshold.","The 3.16GB toxic subset provides a far larger Chinese toxicity resource than existing public Chinese toxicity datasets, enabling more robust safety evaluation.","The public tool-chain allows the same four-dimensional annotation to be regenerated or extended when new Chinese web text becomes available.","Joint filtering on quality, domain, and toxicity lets a user reduce harmful content while preserving high-quality, domain-relevant training material."],"supporting_citations":[{"why":"ChineseWebText1.0 supplies the quality-scoring approach, the BERTEval base model, and several preprocessing rules that ChineseWebText2.0 extends.","marker":"[1]"},{"why":"WanJuan is one of the public Chinese corpora collected as raw input for the new dataset.","marker":"[4]"},{"why":"MAP-CC is another public Chinese corpus used as a raw source.","marker":"[5]"},{"why":"WuDaoCorpora is a raw Chinese text source whose texts go through the MDFG-tool pipeline.","marker":"[14]"},{"why":"COLD contributes toxic and benign Chinese examples to the initial training set for the toxicity classifier.","marker":"[27]"},{"why":"The Tran-BERT-MS-ML-R architecture defines the multi-scale essay-scoring model that the quality evaluator is based on.","marker":"[30]"},{"why":"ToxiCN contributes Chinese toxic-language examples to the initial training set for toxicity evaluation.","marker":"[36]"},{"why":"CDial-Bias contributes Chinese dialogue bias samples to the initial toxicity training set.","marker":"[38]"},{"why":"The instruction-tuned LLM is used in the LLM-in-the-loop iterations to label candidate toxic texts for retraining.","marker":"[39]"}],"fun_headline_variants":["ChineseWebText2.0: 3.8TB with quality, domain, toxicity tags","Largest Chinese text corpus now ships with per-doc quality and toxicity scores","3.8TB Chinese pre-training data, each text tagged for quality, domain, toxicity","New 3.8TB Chinese dataset adds quality, domain, and toxicity metadata"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The release depends on the automatic toxicity model's labels being trustworthy at the 0.99 score threshold across the full corpus, even though the reported precision was measured on a small balanced test set rather than at that operating point on the released distribution.","fun_headline_variants_meta":{"raw":{"variants":["ChineseWebText2.0: 3.8TB with quality, domain, toxicity tags","Largest Chinese text corpus now ships with per-doc quality and toxicity scores","3.8TB Chinese pre-training data, each text tagged for quality, domain, toxicity","New 3.8TB Chinese dataset adds quality, domain, and toxicity metadata"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001057,"raw_usage":{"total_tokens":4480,"prompt_tokens":1036,"completion_tokens":3444,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":652,"completion_tokens_details":{"reasoning_tokens":3353}},"tokens_in":652,"tokens_out":3444,"duration_ms":20134,"temperature":1.0,"reasoning_tokens":3353,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:58:25.461488+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of texts from the released 3.16GB toxic subset and have human annotators judge toxicity; if the observed precision falls far below 83.67%, the claimed reliability of the toxicity labels and the usefulness of the toxic subset for safety work would be undermined.","supporting_citations":[{"cited_title":"Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models, 2023","cited_arxiv_id":null,"evidence_quote":"WanJuan is one of the public Chinese corpora collected as raw input for the new dataset."},{"cited_title":"Chinese tiny llm: Pretraining a chinese-centric large language model, 2024","cited_arxiv_id":null,"evidence_quote":"MAP-CC is another public Chinese corpus used as a raw source."},{"cited_title":"Wudaocorpora: A super large-scale chinese corpora for pre-training language models","cited_arxiv_id":null,"evidence_quote":"WuDaoCorpora is a raw Chinese text source whose texts go through the MDFG-tool pipeline."},{"cited_title":"Towards identifying social bias in dialog systems: Framework, dataset, and benchmark","cited_arxiv_id":null,"evidence_quote":"CDial-Bias contributes Chinese dialogue bias samples to the initial toxicity training set."}],"review_version":1}