Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

FineWeb-zhtw: Scalable Curation of Traditional Chinese Text Data from the Web

T0 review · 4 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A multi-stage filtering pipeline turns raw Common Crawl web pages into a Traditional Chinese pretraining corpus, and the paper argues the resulting samples score significantly higher on LLM-rated naturalness and educational value than…

desk verdict A useful Traditional Chinese corpus with a clear pipeline, but its quality claims rest on an unvalidated LLM rubric and one self-contradictory sentence. read the letter →

arxiv 2411.16387 v1 pith:4ECPA3GR submitted 2024-11-25 cs.CL cs.DB

Cheng-Wei Lin , Wan-Hsuan Hsieh , Kai-Xin Guan , Chan-Jan Hsu , Chia-Chen Kuo , Chuan-Lin Lai , Chung-Wei Chung , Ming-Jen Wang
show 1 more author
Da-Shan Shiu
This is my paper · ORCID
classification cs.CLcs.DB
keywords TraditionalChinesepretrainingdatasetwebtextfilteringCommonCrawlLLM-basedevaluationlanguageidentificationFineWeb-zhtwdatacuration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to build FineWeb-zhtw, a publicly available pretraining corpus for Traditional Chinese, by adapting the FineWeb curation recipe to a language that lacks spacing and has two character variants. Its central claim is that a multi-stage pipeline of basic filtering, custom Traditional Chinese language identification, Gopher/C4/FineWeb quality filters, and minhash deduplication yields a corpus whose samples score consistently higher than earlier-stage samples on GPT-3.5-rated naturalness, educational value, and total quality. If the claim holds, researchers get a ready-to-use dataset and a transferable recipe for a comparatively underserved language, and the reported 40x data-size gap between English and Traditional Chinese would urge new collection efforts beyond Common Crawl.

What carries the argument

The central object is the filter cascade: a fuzzy unicode-range pre-filter, a fastText-based language identifier augmented by phrase-level Traditional/Simplified Chinese discrimination, Gopher quality heuristics (document length, symbol ratio, ellipsis ratio, stop words), C4 line-level filters (JavaScript, policy boilerplate, bracket ratio), FineWeb document-level filters (line punctuation, short line, character duplication, new line ratios), and minhash deduplication. The evaluation machinery is GPT-3.5 used as a scoring agent with a specified 0-5 rubric for naturalness, educational value, and sensitive content, with t-tests comparing pipeline stages.

What would settle it

Take about 200 documents sampled from the basic-filtered, language-identified, and final stages; have human annotators apply the same 0-5 rubric and compute agreement with the GPT-3.5 scores, or train a small Traditional Chinese language model on the final dataset and on the basic-filtered baseline and compare perplexity on a held-out set. If human ratings diverge from the LLM ratings, or the model trained on the final dataset shows no downstream gain, the paper's quality-improvement claim would be undercut.

Watch

Extended reading notes

Core claim

The paper reports that after applying the full pipeline to Common Crawl dump CC-MAIN-2024-26, the final dataset retains about 0.5% of documents (214.04 GB of text) and that 1,000 randomly sampled documents from the final dataset score significantly higher on 0-5 GPT-3.5 ratings than samples from basic-filtered and language-identified stages: naturalness rises from 1.72 to 2.42, educational value from 1.54 to 2.04, and total score from 7.53 to 9.17, with t-test p-values below 0.05. Sensitive-content scores stay high and show no significant gain, indicating that Common Crawl under this pipeline is not a major source of harmful content. The paper also quantifies a roughly 40x gap in document volume between English and Traditional Chinese after language identification and argues on Chinchilla scaling grounds that Common Crawl alone is insufficient for a 70B-parameter Traditional Chinese model.

Load-bearing premise

The central claims rest on using GPT-3.5 ratings of naturalness and educational value as proxies for pretraining-data quality, with no human validation of those ratings and no downstream training run to confirm the proxy.

Editorial extensions

If this is right

  • If the reported quality gains are real, FineWeb-zhtw (214.04 GB for Common Crawl dump CC-MAIN-2024-26) is a ready-to-use pretraining corpus for Traditional Chinese.
  • The statistically significant t-test improvements across naturalness, educational value, and total score indicate that the full cascade removes more low-quality content than basic filtering or language identification alone.
  • The roughly 40x gap in document count between English and Traditional Chinese after language identification, combined with the Chinchilla scaling estimate, supports the paper's conclusion that Common Crawl alone is insufficient for a 70B-parameter Traditional Chinese model.
  • The public release of code and dataset makes the pipeline applicable to future Common Crawl dumps and adaptable to other Chinese language variants.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct downstream check should be run: pretrain a small Traditional Chinese model on FineWeb-zhtw versus a basic-filtered control and compare on a Traditional Chinese benchmark; the paper does not include this test.
  • Because the LLM scoring agent is GPT-3.5, the method's validity for other low-resource languages remains contingent on the scoring model's competence in those languages.
  • The grid-searched thresholds were tuned by manual inspection on one Common Crawl dump; re-validating them on a later dump would tell whether the recipe is stable over time.
  • The reported 40x English-to-Traditional-Chinese gap implies that simply re-running web-scale pipelines will not close the scale deficit; complementary curated sources or synthetic data would be needed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper presents FineWeb-zhtw, a pipeline for curating Traditional Chinese text from Common Crawl. It applies a cascade of filters—basic HTML/text extraction, a custom Traditional/Simplified language identifier, Gopher, C4, and FineWeb quality filters, and minhash deduplication—to one Common Crawl dump (CC-MAIN-2024-26), yielding 14.04 GB of text. The authors evaluate the final dataset by scoring 1,000 random samples with GPT-3.5 on three criteria (Traditional Chinese naturalness, educational value, sensitive content) and compare the scores against the intermediate outputs of the pipeline. They report statistically significant improvements in naturalness and educational value, and argue the filtering is effective. Code and data are released.

Significance. If the evaluation were valid, the dataset would fill a genuine gap: public Traditional Chinese pretraining corpora are scarce, and the authors' pipeline adapts established English filters to the linguistic properties of Traditional Chinese. The use of datatrove and the release of code/data are concrete community contributions. However, the paper's only evidence is an unvalidated LLM rubric on 1,000 samples, with no human agreement, no downstream training, no baseline comparison, and no cross-dump validation. The announced 'consistent improvements across all categories' is contradicted by the paper's own reporting for the sensitive-content category. The resource may be useful, but the central quality claim is not yet established.

major comments (4)
  1. [Section 3.2] The paper states that 'FineWeb-zhtw dataset has consistent improvements across all categories,' yet in the same section it reports that for Sensitive Content 'no statistically significant gains are achieved from the FineWeb filtering stage.' This is a direct contradiction in the paper's central claim. The authors should either revise the claim to 'all categories except sensitive content' or provide a different interpretation that is consistent with the reported p-values.
  2. [Section 3.1] The evaluation relies entirely on GPT-3.5 scores without human validation, inter-annotator agreement, or a reliability analysis. The rubric explicitly rewards logical structure, coherence, and educationally relevant content, while the filters in Sections 2.3–2.5 explicitly remove short lines, high symbol ratios, repetitive text, and poorly punctuated lines. Thus the observed score gains may be mechanical consequences of the filtering criteria rather than evidence that the data is better for pretraining. The paper should add a human-annotated subsample (e.g., 100–200 documents scored by native speakers) to validate the LLM scores, and ideally a downstream check such as language-model perplexity on a held-out Traditional Chinese benchmark or fine-tuning on a downstream task.
  3. [Section 2] All filter thresholds are chosen by grid search with manual inspection of filtered-in/filtered-out data on a single Common Crawl dump (CC-MAIN-2024-26), as stated in Section 2. No held-out dump or sensitivity analysis is provided, so the paper's title claim of 'scalable curation' is not supported by evidence. The authors should at least apply the pipeline to a second dump and report the resulting retention rates and quality scores, or provide a threshold sensitivity analysis.
  4. [Section 3.1] The evaluation compares FineWeb-zhtw only against its own intermediate stages (basic filtering, language identification). The paper does not compare against existing public Traditional Chinese corpora such as mC4-zh, CC-100-zh, or the Chinese split of FineWeb. Without such a comparison, the authors cannot justify the claim that FineWeb-zhtw is a notable advance over the current publicly available options for Traditional Chinese pretraining.
minor comments (7)
  1. [Throughout] The dataset is referred to as both 'FineWeb-zhtw' (title, abstract) and 'FineWeb-TC' (Sections 2 and 5). Pick one name and use it consistently.
  2. [Throughout] There are several typos and formatting issues: 'sufficient' (Section 2.3), 'efficiency' (Section 2.1 or discussion), 'Labatories' (author affiliation), 'efficacy' (Section 5), and the Unicode range in Section 2.2 appears corrupted.
  3. [Section 2.6] The description of line-level deduplication is vague about how 'leading or trailing' lines are identified and whether the 100-occurrence threshold is per dump or cumulative. Please specify the algorithm.
  4. [Figure 1] Figure 1 mixes document counts and byte counts within a single figure; annotate each stage with its unit or separate the two types of rates into different panels.
  5. [Appendix] The evaluation prompt is provided only in Traditional Chinese. An English translation would help readers who do not read Chinese to understand the rubric.
  6. [Section 3.1] The phrase 'GPT3.5 on the DaVinci 3 platform' is unclear; specify the exact model identifier (e.g., text-davinci-003) and API settings used for scoring.
  7. [Section 2] The sentence 'For the parameters mentioned in the paragraph, we determined the parameters using grid search by manually inspecting the quality of filtered-in and filtered-out data' is redundant and should be rewritten.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the filtering pipeline and the GPT-3.5 evaluation are not connected by a fitted parameter or definitional equivalence.

full rationale

The paper's derivation chain is: heuristic filters (Sections 2.1-2.6) produce FineWeb-zhtw, then 1,000 samples from pipeline stages are scored by GPT-3.5 (Section 3.1), and t-tests compare the score distributions (Section 3.2). No quantity in the evaluation is computed from the filter thresholds by an equation: the thresholds are set by grid search with manual inspection (Section 2), and the LLM scores are a separate measurement that could, in principle, disagree with the filters (and does disagree for the Sensitive Content category, where the paper reports no significant gain). The cited works on LLM-as-judge (Chiang and Lee, 2023; Chiang et al., 2024) are not by the present authors and are not load-bearing for the dataset construction. The real weaknesses are validity risks rather than circularity: the scoring rubric rewards coherence, structure, and punctuation, which the filters are designed to enforce, and no human validation of GPT-3.5 scores is reported, so the evaluation may partly reflect the filters' own selection criteria. That is a construct-validity/confounding concern, not a definitional reduction or a fitted-input-called-prediction, so it does not raise the circularity score under the specified criteria.

Assumptions & free parameters 12 free parameters · 4 assumptions · 0 invented entities

The quality claims rest on a chain of manually tuned thresholds and evaluation assumptions. The filter thresholds are not derived from theory or prior work; they are grid-searched by inspection on the same data. The evaluation assumes LLM scoring is a valid proxy for corpus quality, and the scoring criteria overlap with the filter heuristics. No downstream language model training or external corpus comparison is provided.

free parameters (12)
  • Unicode prefilter length = 5 consecutive characters
    Section 2.1: documents without 5 consecutive Traditional Chinese-range characters are removed; threshold chosen by inspection.
  • Traditional/Simplified phrase list = not enumerated in paper
    Section 2.2: custom word-phrase filter separates Traditional from Simplified Chinese; the list is not disclosed.
  • URL blocklist = not enumerated in paper
    Section 2.1: flagged low-value websites are blocked; the blocklist content is not given.
  • Gopher document length bounds = 50 to 100,000 words
    Section 2.3: documents outside this range are excluded.
  • Gopher symbol-to-word ratio = 0.1
    Section 2.3: documents with a higher ratio are excluded.
  • Gopher ellipsis lines ratio = 0.3
    Section 2.3: documents with more than 30% of lines ending in ellipses are excluded.
  • Stopword list = predefined, not enumerated
    Section 2.3: documents without any predefined stopwords are excluded.
  • C4 bracket ratio = 0.01
    Section 2.4: documents with a bracket ratio above 0.01 are excluded.
  • FineWeb line punctuation ratio = 0.04
    Section 2.5: documents with a ratio below 0.04 are excluded.
  • FineWeb short line ratio = 0.8
    Section 2.5: documents with more than 80% short lines are excluded.
  • FineWeb character duplication ratio = 0.3
    Section 2.5: documents with a duplication ratio above 0.3 are excluded.
  • FineWeb new line ratio = 0.3
    Section 2.5: documents with a new line ratio greater than 0.3 are excluded.
assumptions (4)
  • domain assumption Common Crawl snapshot CC-MAIN-2024-26 is a representative source for Traditional Chinese web text.
    The entire dataset is built from one dump, and the paper does not analyze coverage bias or temporal stability.
  • domain assumption fasttext plus phrase matching can accurately separate Traditional Chinese from Simplified Chinese.
    Section 2.2 relies on this custom filter, but no precision or recall numbers are reported for the language identification stage.
  • domain assumption The Gopher, C4, and FineWeb quality heuristics transfer to Traditional Chinese after threshold tuning.
    Sections 2.3 through 2.5 reuse filters designed primarily for English, with thresholds adjusted by manual inspection rather than by theory or external benchmarks.
  • domain assumption LLM-as-scorer ratings correlate with human judgment and with pretraining data quality.
    Section 3.1 cites prior work on LLM evaluation, but the paper provides no human validation or downstream training to confirm the correlation for this task.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FineWeb-zhtw: Scalable Curation of Traditional Chinese Text Data from the Web." pith.science (2026). https://pith.science/paper/4ECPA3GR

@misc{pith2026241116387,
  author       = {Pith},
  title        = {Pith review of: FineWeb-zhtw: Scalable Curation of Traditional Chinese Text Data from the Web},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4ECPA3GR}},
  note         = {Machine review of arXiv:2411.16387}
}
read the original abstract

The quality and size of a pretraining dataset significantly influence the performance of large language models (LLMs). While there have been numerous efforts in the curation of such a dataset for English users, there is a relative lack of similar initiatives for Traditional Chinese. Building upon this foundation of FineWeb, we introduce FineWeb-zhtw, a dataset tailored specifically for Traditional Chinese users. We came up with multiple stages of meticulously designed filters to cater to the linguistic difference between English and Traditional Chinese, to ensure comprehensiveness and quality. We determined effectiveness from querying dataset samples with three main objectives. Our code and datasets are publicly available.

Figures

Figures reproduced from arXiv: 2411.16387 by the authors.

Figure 1
Figure 1. Relative removal rate and global kept rate of Common Crawl during each filtering stage. We report the removal rate with respect to each previous stage, and the overall kept rate (Penedo et al., 2023a). Rates are calculated by the number of documents until the language identification stage. For later line-by-line filtering, rates are measure in tokens (bytes). 3 Evaluation 3.1 Evaluation Methodology To ensure the qua… view at source ↗
Figure 2
Figure 2. presents the evaluation results for the different datasets, highlighting the distri￾bution of scores and mean values for each crite￾rion. We conducted a t-test on the null hypoth￾esis (H0) that the scores are identical. Based on the resulting t-statistic and p-values, we re￾jected the null hypothesis for p-values less than 0.05. Results show that the FineWeb-zhtw dataset has consistent improvements across all catego… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Breeze 2 Herd of Models: Traditional Chinese LLMs Based on Llama with Vision-Aware and Function-Calling Capabilities

    cs.CL 2025-01 conditional novelty 3.0 of 10

    Breeze2 continues pretraining Llama on a 900GB Traditional Chinese corpus, then adds LLaVA-style vision and function-calling fine-tuning, reporting top size-class scores on a mix of public and self-built benchmarks.

Reference graph

Works this paper leans on

6 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    繁體中文與語言自然性：內容是否以 繁體中文寫成，並使用正確的繁體中 文字符；文本是否自然流暢，符合繁 體中文的語法和用詞習慣，並且易於 閱讀和理解、內容完整。滿分 5 分， 評分時請考慮以下因素： • 若語法正確，句子結構自然，得 1 分。 • 若使用正確的繁體中文字符，得 1 分。 • 若用詞符合繁體中文的習慣，得 1 分。 • 若句子簡潔明瞭，易於理解，得 1 分。 • 若內容有邏輯性，有頭有尾，得 1 分。

    work page

  2. [2]

    教育價值：內容是否具有值得學習的 正面價值。滿分 5 分，評分時請考慮 以下因素： • 若內容提供與教育話題相關的 基本資訊，即使其中包含一些無 關或非學術的內容（如廣告和促 銷） ，得1 分。 • 若內容涉及某些教育相關元素， 但 不 完 全 符 合 教 育 標 準， 得1 分。它可能會混合教育內容和非 教育教材，提供潛在有用主題的 概述，或者以無條理和不連貫的 寫作風格呈現訊息。 • 若內容適用於教育用途並介紹與 學校課程相關的關鍵概念，得 1 分。它是連貫的，儘管可能不全 面或包含一些無關的內容。它可 能類似於教科書的介紹部分或適 合學習但有顯著局限性的基礎教 程，比如將概念處理得對於初中 學生來說過於複雜。 • 若內容對於小學或初中等級的 教育目的是高度相關且有益的， 得 1 分。它展現出清晰和一...

    work page

  3. [3]

    敏感內容：是否包含敏感或不適當的 內容。滿分 5 分，評分時請考慮以下 因素： • 若不包含暴力言論、行為、與內 容，不宣揚言語、肢體等任一形 式暴力，不宣揚槍枝與血腥等內 容，得 1 分。 • 若不包含色情言論、行為、與內 容，得 1 分。 • 若不包含歧視言論、行為、與內 容，不貶低、侮辱或仇恨任一種 族、國家、族群、與個人，得 1 分。 • 若不涉及政治和宗教等敏感話 題，不以言論、行為等任一方式 支持或反對任一政黨或教派，得 1 分。 • 若不包含侵犯隱私或個人權利的 言論、行為、與內容，得 1 分。 評分格式 請按照以下格式提供評分，分數應以整 數型態表示，並將每個標準所得分數加 總，計算總分：

    work page

  4. [6]

    敏感內容：< 分數 > 總分：< 總分 > 評分範例 請對以下文本進行評分： < 待評估的文本 >

    work page

  5. [2020]

    The pile: An 800gb dataset of diverse text for language modeling . Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen S...

    work page

  6. [2022]

    Albert Q

    Training compute-optimal large language models. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep ...

    work page 2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.