Pith. sign in

REVIEW 5 major objections 5 minor 43 references

ChineseWebText 2.0: Large-Scale High-quality Chinese Web Text with Multi-dimensional and fine-grained information

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 3.8TB Chinese web-text corpus labels every document with quality, domain, and toxicity annotations via a public tool-chain, MDFG-tool.

desk verdict A large Chinese pretraining corpus with useful annotations, but the toxicity evaluation is internally inconsistent and the 0.99 threshold lacks validation. read the letter →

arxiv 2411.19668 v1 pith:NJKS3MNX submitted 2024-11-29 cs.CL cs.AI

Wanyue Zhang , Ziyong Li , Wen Yang , Chunlin Leng , Yinan Bai , Qianlong Du , Chengqing Zong , Jiajun Zhang This is my paper · ORCID
classification cs.CLcs.AI
keywords ChineseWebText2.0pre-trainingcorpusfine-grainedtextannotationsqualityscoringdomainclassificationtoxicitydetectiondatafiltering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper describes the construction and public release of ChineseWebText2.0, a 3.8TB Chinese web-text corpus in which every text carries four fine-grained annotations: a quality score, domain labels, a toxicity label, and a toxicity score. The proposed tool-chain, MDFG-tool, removes explicit noise with handcrafted rules and then runs a BERT-based quality evaluator, a FastText domain classifier, and a FastText toxicity evaluator refined by an LLM-in-the-loop procedure to attach the annotations. The release gives LLM researchers the ability to select Chinese pre-training data by quality, domain, and toxicity at a scale and granularity not previously available, including a 3.16GB toxic subset that the paper identifies as the largest publicly available Chinese toxicity dataset. The data, code, and tool-chain are released with the paper.

What carries the argument

The load-bearing object is MDFG-tool, a pipeline that converts raw Chinese web text into annotated records in two stages. The first stage applies handcrafted rules: average line length at least 10 characters, document length at least 200 characters, at least 30% Chinese characters, a sensitive-word line filter, and 13-gram deduplication. The second stage runs three annotation models: a BERT-based quality evaluator that combines text-level and max-pooled token-level representations with MSE, margin-ranking, and cosine-similarity losses plus self-training; a FastText domain classifier seeded by keyword rules and iteratively retrained; and a FastText toxicity classifier trained on combined public Chinese toxicity datasets and refined through two LLM-in-the-loop iterations. The pipeline's output is one record per text carrying quality score, domain labels, toxicity score, and toxicity label.

What would settle it

Take a random sample of texts from the released 3.16GB toxic subset and have human annotators judge toxicity; if the observed precision falls far below 83.67%, the claimed reliability of the toxicity labels and the usefulness of the toxic subset for safety work would be undermined.

Watch

Extended reading notes

Core claim

ChineseWebText2.0 is presented as the largest publicly available Chinese pre-training corpus, totaling 3.8TB after rule-based cleaning, and every text in it receives four annotations: a quality score from a BERT-based evaluator, domain labels from a FastText classifier over 11 domains, and a toxicity score plus toxicity label from a FastText evaluator refined by an LLM-in-the-loop procedure. On manual evaluation, the paper reports that quality scores correlate positively with human acceptance, domain classification reaches 88.33% precision for single-label prediction and 74.48% micro-precision for multi-label prediction, and toxicity classification reaches 83.67% precision and 97.67% specificity on a balanced test set. The released package also includes a toxic subset of 1,632,620 texts (3.16GB), which the paper identifies as the largest publicly available Chinese toxicity dataset.

Load-bearing premise

The release depends on the automatic toxicity model's labels being trustworthy at the 0.99 score threshold across the full corpus, even though the reported precision was measured on a small balanced test set rather than at that operating point on the released distribution.

Editorial extensions

If this is right

  • Researchers can build domain-specific pre-training subsets, such as law or medicine, by filtering on the domain labels and a quality threshold.
  • The 3.16GB toxic subset provides a far larger Chinese toxicity resource than existing public Chinese toxicity datasets, enabling more robust safety evaluation.
  • The public tool-chain allows the same four-dimensional annotation to be regenerated or extended when new Chinese web text becomes available.
  • Joint filtering on quality, domain, and toxicity lets a user reduce harmful content while preserving high-quality, domain-relevant training material.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the reported toxicity precision comes from a balanced 600-text test set, users should validate precision on a sample drawn from the actual 0.99-threshold toxic subset before relying on those labels for training or evaluation.
  • The quality-score human acceptance curve stays above 80% even at the lowest score interval, so quality filtering is likely most useful for removing a noisy tail rather than for fine-grained ranking of typical web text.
  • The Math domain accounts for only 0.55% of the corpus, so users training mathematical LLMs should expect to supplement this dataset with specialized sources despite the fine-grained domain labels.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents MDFG-tool, a pipeline that combines rule-based cleaning, a BERT-based quality model, a FastText domain classifier, and a FastText toxicity model to annotate Chinese web text with quality scores, domain labels, toxicity labels, and toxicity scores. The authors collect roughly 6.6 TB from public Chinese datasets, reduce it to 3.8 TB through preparation and preprocessing, and release the resulting corpus as ChineseWebText2.0, including a 3.16 GB subset of texts classified as toxic. The paper evaluates each annotation component, reports corpus statistics and distributions, and states that data, code, and the tool chain are publicly available.

Significance. If the annotations are reliable, this is a significant resource paper: ChineseWebText2.0 would be the largest public Chinese pre-training corpus with per-text quality, domain, and toxicity metadata, and the toxic subset would be the largest Chinese toxicity dataset with continuous toxicity scores. The release of code and the tool chain is a concrete reproducibility strength, and the corpus would directly support domain-specific and safety-oriented work on Chinese LLMs. The contribution is, however, contingent on the validity of the annotation results, and the toxicity evaluation currently contains internal inconsistencies that undermine confidence in the released labels. The paper is valuable in scope and ambition, but the validation of the central advertised annotations needs repair before the resource can be relied on.

major comments (5)
  1. [§3.5.2, Table 4 and Eq. (4)] The reported toxicity evaluation is internally inconsistent. With a balanced test set of 300 toxic and 300 benign texts, precision = TP/(TP+FP) and the printed values TP=251, TP+FP=300 imply FP=49; specificity as printed, TN/(TN+FN)=293/300, implies FN=7. These counts cannot both hold, and they would imply actual class totals of Toxic=258 and Benign=342, contradicting the stated 300/300 balance. If the counts are instead taken as TP=251, FN=49, TN=293, FP=7, then the reported value 83.67% is recall, not precision, and the standard precision would be 251/(251+7)=97.29%. In addition, Eq. (4) defines specificity as TN/(TN+FN), whereas the standard definition is TN/(TN+FP). Because the released toxicity labels are a central advertised feature, this evaluation must be corrected and re-reported before the labels can be relied on.
  2. [§4.4, Figure 10] The threshold of 0.99 used to extract the 1,632,620-text toxic subset is introduced through 'manual analysis of the toxicity scores', but no precision or recall is reported at this operating point. The only model evaluation in §3.5.2 is at the default decision threshold, and that evaluation is itself misreported. Since the paper releases a 3.16 GB subset as the 'largest Chinese toxicity dataset', the authors should sample texts with scores in intervals around 0.99, have them human-annotated, and report precision, recall, and confidence intervals at the actual extraction threshold. Without this, the size claim does not establish the reliability of the toxic subset.
  3. [§3.5.1] The LLM-in-the-loop step uses Qwen2.5-32B-Instruct to label candidate texts that are added to the training set, but the paper reports no human verification or agreement analysis for these LLM-generated labels. This matters because the teacher labels shape the training distribution of the final toxicity classifier, and the only downstream validation is the problematic Table 4. I recommend reporting an agreement statistic (e.g., Cohen's kappa) between the LLM labels and human annotations on a held-out sample, and describing how disagreements were resolved.
  4. [§3.4.2 and §3.4.3] The domain classifier is validated on only 300 manually annotated texts, with some classes represented by 2–20 test examples (math=2, law=13). The iterative optimization in §3.4.1 uses model predictions to update the rule-based keyword sets, so the training signal is partly self-generated; a 300-text test set is too small to establish per-class reliability, especially for the multi-label micro-averaged metrics, and the single-label recall of 64.15% already indicates substantial under-coverage. Please report per-class precision and recall with confidence intervals, and clarify how the test texts were kept disjoint from all iterations of the self-training process.
  5. [§3.3, §4.2, and Appendix 5.1] The human evaluation for quality scores is described inconsistently: §3.3 says 50 samples per score interval, §4.2 and Figure 7 say 100 samples per interval, and Appendix 5.1 says 1,000 examples total. The appendix also reports no inter-annotator agreement statistic despite using five annotators. Since the acceptance rate is already 80% in the lowest interval [0.1,0.2), the current evaluation has limited power to show that the quality scores are meaningfully discriminative. Please reconcile the sample sizes, report agreement, and provide acceptance rates per interval with confidence intervals.
minor comments (5)
  1. [Throughout] The manuscript contains several typos that should be corrected, including 'satefy' in the Introduction, 'Pratly Public' in Table 5, 'Classfier' in Figure 1, 'Initail' in §3.5.1, and inconsistent capitalization of 'BertEval'/'B ERTEval'.
  2. [References] References [9] and [2] are the same T5 paper, reference [12] is formatted as a title rather than an authorial citation, and references [28] and [29] are incomplete URL-style entries; these should be cleaned up.
  3. [§3.5.2, Table 4] The table would be more informative if it reported recall and F1 for the toxic class in addition to precision, since the paper currently claims only precision and specificity but uses a balanced test set that also permits recall and F1.
  4. [§4.2, Figure 7] The two panels of Figure 7 use different y-axis scales and the caption does not state this; the visual comparison would be clearer with a shared scale or an explicit note.
  5. [§4.4, Table 7] The comparison in Table 7 mixes labeled datasets with unlabeled or weakly labeled collections; the caption should state which entries have human-verified labels and which are model-generated.

Circularity Check

2 steps flagged · score 4.0 of 10

No clean by-construction circularity in the pipeline; two flagged items: the quality model is inherited from the authors' own ChineseWebText1.0, and the toxicity subset rests on a self-referential 0.99 threshold whose only precision evidence (Table 4) is internally inconsistent.

  1. self citation load bearing [Section 3.3 (Quality Evaluation), building on Section 3.2 (Preprocessing)]
    "Building on the approach used in ChineseWebText1.0 [1], we develop a BERT-based classification model to assign a quality score to each text. The BERT model in the ChineseWebText1.0 achieves excellent classification performance, so we utilize it as our base model. Building upon the work of ChineseWebText-1.0, this paper employs four types of handcrafted rules in the preprocessing module."

    ChineseWebText1.0 [1] (Chen et al., 2023) is the same group's prior work: its authors include Qianlong Du, Chengqing Zong, and Jiajun Zhang, all authors of the present paper. The quality-score annotation, one of the four advertised fine-grained outputs, inherits its base model, preprocessing rules, and training-data philosophy from that first-party paper, so the same-author citation supplies the core of the quality dimension rather than an independent derivation. This is partial rather than fatal: the prior artifact is public and externally checkable, and the present paper adds its own human-acceptance evaluation (Figures 3 and 7b), so the quality claim retains independent content.

  2. fitted input called prediction [Section 4.4 (Data Toxicity Analysis, Figure 10) and Section 3.5.2 (Evaluation, Table 4 and Eq. 4)]
    "Additionally, through manual analysis of the toxicity scores, we identify that data with scores above 0.99 are classified as toxic. By applying this empirical threshold, we filter our dataset to obtain a 3.16GB subset of toxic texts, comprising 1,632,620 samples. The final test set consists of 300 toxic samples and 300 benign samples. From this table, we can observe that the precision of our toxicity evaluation model for toxic texts reaches 83.67%, while the specificity for benign texts is 97.67%."

    The released toxicity label is, by construction, the model's own score thresholded at 0.99, and that threshold is chosen from the very same scores via manual analysis; the 1,632,620-text toxic subset and the 'largest publicly available Chinese toxicity dataset' claim are therefore outputs of the same classifier rather than independently measured quantities. The validation that would break this self-reference is unavailable at the operating point: Table 4 states no threshold, and its counts contradict the stated balanced test set, since with 300 toxic and 300 benign samples, TP=251 (TP+FP=300) and TN=293 (TN+FN=300) force FP=49 and FN=7, implying 258 actual toxic and 342 actual benign texts, not 300/300. Eq.

full rationale

The pipeline is not circular by construction: the domain classifier is judged on a manually annotated 300-text test set (Table 2), the quality model is judged by human acceptance (Figures 3 and 7b), and the toxicity model is nominally judged on a manually labeled test set (Table 4), all external anchors. The concrete circularity-adjacent findings are two. First, the quality-evaluation module is explicitly inherited from the authors' own ChineseWebText1.0 (Chen et al. 2023, with overlapping authors Du, Zong, and Zhang): the base BERT model, the handcrafted preprocessing rules, and the training approach all follow that same-group paper, so the quality-score annotation is second-generation rather than first-principles; this is mitigated because the present paper adds its own human-acceptance evaluation and the prior artifact is public and externally checkable. Second, the toxicity label is defined as the model's own score exceeding threshold 0.99, a threshold chosen by manual analysis of those same scores, so the released 1,632,620-sample toxic subset and the 'largest Chinese toxicity dataset' claim reduce to the classifier's own outputs; the validation that would break this self-reference is not available at the operating point, since Table 4 does not state the threshold used and its counts (TP=251 with TP+FP=300; TN=293 with TN+FN=300) are mutually inconsistent with the stated 300/300 balanced test set, implying 258 actual toxic and 342 actual benign texts, while Eq. (4) defines Specificity with the predicted-negative denominator TN/(TN+FN) instead of TN/(TN+FP). A further mild self-reference is that the toxicity model's 80,000 benign training samples are drawn from the same high-quality corpus it is then deployed to label, which mildly self-confirms the '97.41% non-toxic' corpus statistic; this is standard domain-matched negative sampling, however, and is not by itself circular. The iterative self-training loops in the quality and domain modules are anchored by external manual evaluations and are therefore not circular in the rubric's sense.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests mainly on hand-chosen thresholds (toxicity 0.99, candidate 0.5, domain 0.3, keyword frequencies) and on the accuracy of the three learned models. The models are standard architectures trained on data that is partly self-generated (LLM labels, self-training), which adds an indirect circularity to the annotation quality. No new physical or conceptual entities are postulated.

free parameters (5)
  • Toxicity threshold (0.99) = 0.99
    Chosen empirically by manual analysis of toxicity scores in Section 4.4; determines the 3.16GB/1,632,620-sample toxic subset and the 'largest toxicity dataset' claim.
  • Candidate toxicity threshold = 0.5
    Used in Section 3.5.1 to screen candidate texts for LLM-in-the-loop labeling, affecting the toxicity model's training distribution.
  • Domain multi-label probability threshold = 0.3
    FastText multi-label predictions return all labels with probability above 0.3 (Section 3.4.1), shaping the domain labels assigned to the full corpus.
  • Rule-based keyword frequency threshold = 3-5 non-repeating instances
    Sets the number of keyword hits needed for initial rule-based domain labels (Section 3.4.1), which in turn trains the FastText classifier.
  • Source exclusion threshold = 30% irrelevant text
    A raw source is entirely discarded if more than 30% of sampled texts appear irrelevant (Section 3.1), determining which corpora enter the pipeline.
assumptions (5)
  • domain assumption The 11 selected domains are adequate and non-overlapping categories for Chinese web text.
    Section 3.4 defines these as the most common application areas of LLMs without empirical justification for the taxonomy; all domain labels are limited to these, so classification quality depends on this taxonomy.
  • domain assumption Qwen2.5-32B-Instruct provides sufficiently accurate toxicity labels for candidate texts.
    Section 3.5.1 uses LLM labels as supervision to retrain the FastText toxicity model; no human verification of the LLM's labels is reported, so the model's accuracy relies on the LLM's reliability.
  • domain assumption Existing public datasets (MAP-CC, SkyPile, WanJuan, WuDao, ChineseWebText1.0) are suitable raw sources with license compatibility for redistribution.
    Section 3.1 treats these datasets as raw material for a newly released corpus, but the paper does not discuss licenses or redistribution terms, which could affect the dataset's legal availability.
  • standard math Standard techniques (BERT fine-tuning, FastText classification, self-training) work as described in their cited sources.
    The paper relies on these established methods without re-deriving them; any failure of these methods would directly affect the annotation models.
  • domain assumption The Common Crawl sample used as negative data for the quality model is representative of low-quality Chinese web text.
    Section 3.3 samples negatives from Common Crawl assuming it has high noise; if the sample is not representative, the quality scores could be biased, affecting the entire score distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ChineseWebText 2.0: Large-Scale High-quality Chinese Web Text with Multi-dimensional and fine-grained information." pith.science (2026). https://pith.science/paper/NJKS3MNX

@misc{pith2026241119668,
  author       = {Pith},
  title        = {Pith review of: ChineseWebText 2.0: Large-Scale High-quality Chinese Web Text with Multi-dimensional and fine-grained information},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NJKS3MNX}},
  note         = {Machine review of arXiv:2411.19668}
}
read the original abstract

During the development of large language models (LLMs), pre-training data play a critical role in shaping LLMs' capabilities. In recent years several large-scale and high-quality pre-training datasets have been released to accelerate the research of LLMs, including ChineseWebText1.0, C4, Pile, WanJuan, MAPCC and others. However, as LLMs continue to evolve, focus has increasingly shifted to domain-specific capabilities and safety concerns, making those previous coarse-grained texts insufficient for meeting training requirements. Furthermore, fine-grained information, such as quality, domain and toxicity, is becoming increasingly important in building powerful and reliable LLMs for various scenarios. To address these challenges, in this paper we propose a new tool-chain called MDFG-tool for constructing large-scale and high-quality Chinese datasets with multi-dimensional and fine-grained information. First, we employ manually crafted rules to discard explicit noisy texts from raw contents. Second, the quality evaluation model, domain classifier, and toxicity evaluation model are well-designed to assess the remaining cleaned data respectively. Finally, we integrate these three types of fine-grained information for each text. With this approach, we release the largest, high-quality and fine-grained Chinese text ChineseWebText2.0, which consists of 3.8TB and each text is associated with a quality score, domain labels, a toxicity label and a toxicity score, facilitating the LLM researchers to select data based on various types of fine-grained information. The data, codes and the tool-chain are available on this website https://github.com/CASIA-LM/ChineseWebText-2.0

Figures

Figures reproduced from arXiv: 2411.19668 by the authors.

Figure 1
Figure 1. The pipeline of MDFG-tool. 3.1 Data Collection and Preparation In recent years, many pre-training datasets have been publicly released, containing a variety of texts from different sources. As a result, we consider these datasets as the sources of our raw content. In this paper, we collect some publicly available datasets from the internet, including MAP-CC [5], SkyPile [16], WanJuan [4], WuDao [14], and ChineseWebT… view at source ↗
Figure 2
Figure 2. The Overview of BERTEval Training Data Composition and Model Architecture. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The Quality Evaluation. Evaluation To evaluate the performance of BertEval in quality assessment, we conduct a human evaluation based on BertEval scores. First, we sample a validation set consisting of 50 samples from each BertEval score interval. Since the scores in the interval [0,0.1) are zero, we exclude this range from the evaluation. Then, we perform human evaluation of the quality acceptability of samples in … view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The Overview of BERTEval Training Data Composition and Model Architecture. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The Architecture of Toxicity Evaluator and lacks diversity, which substantially restricts the effectiveness of these models. At the same time, testing publicly available large language models (LLMs) has shown that some of these models possess strong toxicity evaluation…
Figure 6
Figure 6. Figure 6: The proportion of data removed from the originally collected data in each processing step. The gray bars [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: The Data Analysis on Quality Evaluation. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Data Distribution Across Domains Book Dialogue Education Encyclopedia Finance Law Math Medicine News Technology General 0 5 10 15 20 25 30 35 Proportion (%) [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 10
Figure 10. Figure 10: The Distribution of Toxicity: A threshold of 0.99 was established, and samples with scores exceeding 0.99 [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 18 canonical work pages

  1. [1]

    Chinesewebtext: Large-scale high-quality chinese web text extracted with effective evaluation model, 2023

    Jianghao Chen, Pu Jian, Tengxiao Xi, Dongyi Yi, Qianlong Du, Chenglin Ding, Guibo Zhu, Chengqing Zong, Jinqiao Wang, and Jiajun Zhang. Chinesewebtext: Large-scale high-quality chinese web text extracted with effective evaluation model, 2023

    2023

  2. [2]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020

    2020

  3. [3]

    The Pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    arXiv 2020

  4. [4]

    Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models, 2023

    Conghui He, Zhenjiang Jin, Chao Xu, Jiantao Qiu, Bin Wang, Wei Li, Hang Yan, Jiaqi Wang, and Dahua Lin. Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models, 2023. 14 ChineseWebText 2.0: Large-Scale High-quality Chinese Web Text with Multi-dimensional and Fine-grained Information

    work page 2023

  5. [5]

    Chinese tiny llm: Pretraining a chinese-centric large language model, 2024

    Xinrun Du, Zhouliang Yu, Songyang Gao, Ding Pan, Yuyang Cheng, Ziyang Ma, Ruibin Yuan, Xingwei Qu, Jiaheng Liu, Tianyu Zheng, Xinchen Luo, Guorui Zhou, Wenhu Chen, and Ge Zhang. Chinese tiny llm: Pretraining a chinese-centric large language model, 2024

    work page 2024

  6. [6]

    Bloom: A 176b-parameter open-access multilingual language model

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili ´c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022

    arXiv 2022

  7. [7]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023

    2023

  8. [8]

    The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023

    2023

Show all 43 references
  1. [9]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023

    2023

  2. [10]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022

    2022 arXiv

  3. [11]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  4. [12]

    Corr, abs/2303.08774, 2023

    GPT-4 Technical Report. Corr, abs/2303.08774, 2023

    2023 arXiv

  5. [13]

    Learning to reason with llms

    OpenAI. Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/ , 2024

    2024

  6. [14]

    Wudaocorpora: A super large-scale chinese corpora for pre-training language models

    Sha Yuan, Hanyu Zhao, Zhengxiao Du, Ming Ding, and Jie Tang. Wudaocorpora: A super large-scale chinese corpora for pre-training language models. AI Open, 2021

    2021

  7. [15]

    Yuan 1.0: Large-scale pre-trained language model in zero-shot and few-shot learning, 2021

    Shaohua Wu, Xudong Zhao, Tong Yu, Rongguo Zhang, Chong Shen, Hongli Liu, Feng Li, Hong Zhu, Jiangang Luo, Liang Xu, and Xuanwei Zhang. Yuan 1.0: Large-scale pre-trained language model in zero-shot and few-shot learning, 2021

    2021

  8. [16]

    Skywork: A more open bilingual foundation model, 2023

    Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, Chenxia Li, Liu Yang, Xilin Luo, Xuejie Wu, Lunan Liu, Wenjun Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuanhai Dong, Yanq...

    2023

  9. [17]

    Alexandra Sasha Luccioni and Joseph D. Viviano. What’s in the box? a preliminary analysis of undesirable content in the common crawl corpus, 2021

    2021

  10. [18]

    Deduplicating training data makes language models better, 2022

    Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better, 2022

    2022

  11. [19]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    2005 arXiv

  12. [20]

    M. V . Koroteev. Bert: A review of applications in natural language processing and understanding, 2021

    2021

  13. [21]

    Bag of tricks for efficient text classification, 2016

    Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification, 2016

    2016

  14. [22]

    Efficient estimation of word representations in vector space, 2013

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space, 2013

    2013

  15. [23]

    A review on offensive language detection

    Rahul Pradhan, Ankur Chaturvedi, Aprna Tripathi, and Dilip Kumar Sharma. A review on offensive language detection. Advances in Data and Information Sciences: Proceedings of ICDIS 2019, pages 433–439, 2020

    2019

  16. [24]

    Chinese offensive language detection: Current status and future directions

    Yunze Xiao, Houda Bouamor, and Wajdi Zaghouani. Chinese offensive language detection: Current status and future directions. arXiv preprint arXiv:2403.18314, 2024

    2024 arXiv

  17. [25]

    Hatebert: Retraining bert for abusive language detection in english

    Tommaso Caselli, Valerio Basile, Jelena Mitrovi´c, and Michael Granitzer. Hatebert: Retraining bert for abusive language detection in english. arXiv preprint arXiv:2010.12472, 2020

    2010 arXiv

  18. [26]

    Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection

    Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. arXiv preprint arXiv:2203.09509, 2022. 15 ChineseWebText 2.0: Large-Scale High-quali...

    2022 arXiv

  19. [27]

    Cold: A benchmark for chinese offensive language detection

    Jiawen Deng, Jingyan Zhou, Hao Sun, Chujie Zheng, Fei Mi, Helen Meng, and Minlie Huang. Cold: A benchmark for chinese offensive language detection. arXiv preprint arXiv:2201.06025, 2022

    2022 arXiv

  20. [28]

    Baai. 2024. industrycorpus. https://huggingface.co/datasets/BAAI/IndustryCorpus/tree/main

    2024

  21. [29]

    Together computer. 2023c. redpajama-incite-base-3bv1

  22. [30]

    On the use of bert for automated essay scoring: Joint learning of multi-scale essay representation

    Yongjie Wang, Chuan Wang, Ruobing Li, and Hui Lin. On the use of bert for automated essay scoring: Joint learning of multi-scale essay representation. arXiv preprint arXiv:2205.03835, 2022

    2022 arXiv

  23. [31]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  24. [32]

    A neural local coherence model for text quality assessment

    Mohsen Mesgar and Michael Strube. A neural local coherence model for text quality assessment. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 4328–4339, 2018

    2018

  25. [33]

    Temp: taxonomy expansion with dynamic margin loss through taxonomy-paths

    Zichen Liu, Hongyuan Xu, Yanlong Wen, Ning Jiang, Haiying Wu, and Xiaojie Yuan. Temp: taxonomy expansion with dynamic margin loss through taxonomy-paths. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3854–3863, 2021

    2021

  26. [34]

    Probability of error of some adaptive pattern-recognition machines

    Henry Scudder. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory, 11(3):363–371, 1965

    1965

  27. [35]

    Uncertainty-aware self-training for few-shot text classification

    Subhabrata Mukherjee and Ahmed Awadallah. Uncertainty-aware self-training for few-shot text classification. Advances in Neural Information Processing Systems, 33:21199–21212, 2020

    2020

  28. [36]

    Facilitating fine-grained detection of chinese toxic language: Hierarchical taxonomy, resources, and benchmarks

    Junyu Lu, Bo Xu, Xiaokun Zhang, Changrong Min, Liang Yang, and Hongfei Lin. Facilitating fine-grained detection of chinese toxic language: Hierarchical taxonomy, resources, and benchmarks. arXiv preprint arXiv:2305.04446, 2023

    2023 arXiv

  29. [37]

    Swsr: A chinese dataset and lexicon for online sexism detection

    Aiqi Jiang, Xiaohan Yang, Yang Liu, and Arkaitz Zubiaga. Swsr: A chinese dataset and lexicon for online sexism detection. Online Social Networks and Media, 27:100182, 2022

    2022

  30. [38]

    Towards identifying social bias in dialog systems: Framework, dataset, and benchmark

    Jingyan Zhou, Jiawen Deng, Fei Mi, Yitong Li, Yasheng Wang, Minlie Huang, Xin Jiang, Qun Liu, and Helen Meng. Towards identifying social bias in dialog systems: Framework, dataset, and benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3...

    2022

  31. [39]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024

    2024

  32. [40]

    The bigscience roots corpus: A 1.6 tb composite multilingual dataset

    Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro V on Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neura...

    2022

  33. [41]

    Semeval- 2019 task 6: Identifying and categorizing offensive language in social media (offenseval)

    Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. Semeval- 2019 task 6: Identifying and categorizing offensive language in social media (offenseval). arXiv preprint arXiv:1903.08983, 2019

    2019 arXiv

  34. [42]

    I feel offended, don’t be abusive! implicit/explicit messages in offensive and abusive language

    Tommaso Caselli, Valerio Basile, Jelena Mitrovi´c, Inga Kartoziya, and Michael Granitzer. I feel offended, don’t be abusive! implicit/explicit messages in offensive and abusive language. In Proceedings of the twelfth language resources and evaluation conference, pages 6193–6202, 2020

    2020

  35. [43]

    True" or

    Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. In Proceedings of the 13th ...

    2019

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.