Pith. sign in

REVIEW 3 major objections 6 minor 13 references

WETBench: A Benchmark for Detecting Task-Specific Machine-Generated Text on Wikipedia

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read WETBench, a multilingual benchmark built around the editing tasks Wikipedia editors actually use LLMs for, shows that detectors underperform in realistic scenarios: training-based detectors average 78% accuracy and zero-shot detectors 58%.

desk verdict A useful new benchmark for MGT detection on Wikipedia; the headline numbers are real, but the 'task-specific is harder' claim is under-supported by the experiment design. read the letter →

arxiv 2507.03373 v2 pith:H2LO5NIA submitted 2025-07-04 cs.CL

Gerrit Quaremba , Elizabeth Black , Denny Vrandečić , Elena Simperl This is my paper
classification cs.CL
keywords machine-generatedtextdetectionWikipediabenchmarkmultilingualstyletransfersummarisationlargelanguagemodelszero-shot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that machine-generated text (MGT) detectors are usually tested the wrong way: benchmarks prompt a model to “write a Wikipedia article about X” from scratch, while real Wikipedia editing is task-specific and incremental. It builds WETBench, a benchmark of 101,940 texts that recasts detection around three editor-grounded tasks — Paragraph Writing, Summarisation, and Text Style Transfer — across English, Portuguese, and Vietnamese, four generators, and eight detectors. The central finding is that performance drops considerably: training-based detectors average 78% accuracy and zero-shot detectors 58%, with sentence-level style transfer hardest, where even trained detectors land near 61–65%. If the claim holds, published detector accuracy on generic generation overstates how ready these tools are to protect Wikipedia’s knowledge integrity.

What carries the argument

The load-bearing object is the task-specific generation setup itself, built on three components. WikiPS and mWNC are two new corpora assembled from Wikipedia revisions dated before ChatGPT’s public release on 30 November 2022, so the human-written side is uncontaminated by MGT; mWNC extends the existing Wikipedia Neutrality Corpus to Portuguese and Vietnamese and adds English paragraph-level biased–neutralised pairs. The generation pipelines use empirically selected prompts: Naive RAG (retrieval-augmented generation that grounds each paragraph in web context) for Paragraph Writing, one-shot prompting for Summarisation, and five-shot prompting for Text Style Transfer. These choices make the machine text resemble human reference text more closely than open-ended generation does, which is precisely what stresses the detectors.

What would settle it

Collect a set of real Wikipedia revisions confirmed by editors (through self-report or edit-audit) to be LLM-assisted, run the same eight detectors on them, and compare accuracy against the benchmark's 58–78% range; substantially higher accuracy would show the benchmark overstates detection difficulty, and substantially lower accuracy would show it understates it.

Watch

Extended reading notes

Core claim

On the paper’s own terms, the discovery is that task-specific, content-conditioned MGT is textually closer to human writing than open-ended generation is, and detectors trained and evaluated on generic setups do not transfer well to it. Across the benchmark, training-based detectors (fine-tuned XLM-RoBERTa and mDeBERTa) average 78% accuracy and zero-shot detectors average 58%, with strong variation by task, language, and generator. Summarisation is the easiest task for detection (supervised average 89%), because Wikipedia lead sections follow a distinctive style, while Text Style Transfer is the hardest: minimal edits to human text at the sentence level leave supervised detectors at 61–65% and zero-shot detectors at 52–56%, close to chance. The paper reads these results as evidence that detectors struggle with MGT in realistic generation scenarios and that diverse, task-specific data are necessary to judge their reliability in editor-driven contexts.

Load-bearing premise

The load-bearing premise is that the three benchmark tasks — paragraph writing, summarisation, and style transfer — capture how Wikipedia editors actually use LLMs; the paper concedes in its limitations that there is no empirical evidence editors use them this way, so if real workflows differ, the measured detection accuracies would not reflect real-world difficulty.

Editorial extensions

If this is right

  • Published accuracy figures from open-ended generation benchmarks overstate how well detectors would perform on the editing tasks Wikipedia editors actually carry out.
  • Only training-based detectors approach usable accuracy, and only on the generation-heavy tasks; zero-shot detectors are near chance on sentence-level style transfer.
  • Detection difficulty tracks the degree of LLM intervention: full summarisation is easiest to detect and minimal text modification hardest, so moderation tools should be scoped to the editing task they will face.
  • Detector behaviour differs by language resource level and generator size, so single-language or single-generator evaluations are not reliable guides to deployment.
  • The released WikiPS and mWNC corpora support downstream work beyond detection, including multilingual bias detection and single-document summarisation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper explicitly concedes there is no empirical evidence that editors use LLMs the way its tasks assume; if real workflows lean more on translation or grammar correction, real-world detection difficulty could shift in either direction from the measured 58–78% range.
  • A natural extension is LLM-assisted translation detection, which the paper flags as underexplored; since translations preserve the human author’s structure, detection difficulty could plausibly sit between summarisation and style transfer.
  • The paper reports that its style-transfer prompt evaluation relies on binary style classifiers that are near chance for Vietnamese and English paragraph-level data; if those classifiers mis-ranked prompts, the TST generation settings — and hence the TST detection numbers — would shift.
  • The large sentence-versus-paragraph gap for style transfer suggests deployed screening should work on revision-level or paragraph-level signals rather than sentence-by-sentence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces WETBench, a multilingual benchmark for detecting machine-generated text (MGT) in task-specific Wikipedia editing scenarios. The benchmark covers three tasks—Paragraph Writing, Summarisation, and Text Style Transfer—implemented with two new corpora, WikiPS and mWNC, across English, Portuguese, and Vietnamese. The authors generate MGT with four LLMs using prompts selected by automatic metrics, then evaluate eight detectors (fine-tuned classifiers and zero-shot white-/black-box methods). The main empirical finding is that training-based detectors average 78% accuracy and zero-shot detectors average 58% across settings, which the paper interprets as evidence that detectors 'struggle with MGT in realistic generation scenarios' and generalise poorly beyond open-ended generation tasks.

Significance. If the findings hold, WETBench addresses a real gap: existing MGT benchmarks largely evaluate open-ended, topic-to-text generation, whereas Wikipedia editors plausibly use LLMs for more constrained editorial tasks. The two released corpora (WikiPS and mWNC) are potentially valuable resources beyond detection, including for summarisation, bias detection, and low-resource multilingual research. The paper is largely reproducible in principle: prompt templates, generation protocols, detector implementations, and hyperparameter settings are documented in the appendices, and the authors state that data and code are released. The main caveat is that the central comparative claim—that task-specific MGT is harder to detect than generic open-ended MGT—is not supported by a matched within-benchmark control, so the headline conclusion needs either additional experimentation or more careful reframing.

major comments (3)
  1. [§5–§6; Figure 2] The paper's central claim that detectors 'struggle with MGT in realistic generation scenarios' relative to generic open-ended generation is not directly supported by the experiments. WETBench contains only the three task-specific conditions described in Section 5; there is no within-benchmark open-ended condition using the same generators, detectors, languages, and evaluation protocol. The comparisons with MULTITuDE, M4, and other prior benchmarks in Section 6 and the Abstract are cross-paper and differ in detector instances, generator families, thresholds, and splits, so the task-specificity inference is confounded. Figure 2 is a text-similarity analysis, not a detection experiment. Please add a matched open-ended condition (e.g., the Minimal paragraph prompt or a full-article prompt) evaluated with the same eight detectors, or explicitly restrict the conclusion to the claim that these detectors score 78%/58% on WETBench.
  2. [§7 Limitations; Real-World Relevance] The motivation and title emphasise 'realistic' Wikipedia editing scenarios, but the paper itself states in the Limitations that 'we lack empirical evidence that editors systematically use LLMs in the ways we design them.' This is a load-bearing assumption: if real editor workflows differ (for example, involving more translation or grammar correction), the benchmark measures plausible scenarios rather than actual Wikipedia MGT. The conclusions should be reframed accordingly, or the paper should provide evidence—from editor surveys, edit histories, or LLM-assisted revision traces—that the three tasks and prompts are representative of actual usage.
  3. [§4.3, Appendix B.3, Table 9] The prompt evaluation for Text Style Transfer relies on style classifiers whose accuracy is near chance for Vietnamese sentences (58%) and English paragraphs (58%). Because the ST column in Table 3 is used to conclude that five-shot prompting is best and to select the prompt for MGT generation, the construction of the TST benchmark is affected. The semantic metrics (BLEU/ROUGE/BERTScore) are more stable, but the style-transfer dimension is uncertain for these subsets. The authors acknowledge the issue, but it should be addressed rather than only listed as a limitation—for example, by validating prompt choice on a human-annotated sample, basing selection on semantic preservation alone, or labelling the affected TST subsets as provisional.
minor comments (6)
  1. [§5] The text says 'We evaluate six detectors from three different families,' but the list that follows and Table 4 contain eight detectors (two supervised, three white-box, three black-box).
  2. [§6, Summarisation paragraph] The sentence 'Binoculars achieves the highest average accuracy (65%) among black-box detectors ... while GECScore performs best among white-box methods (68%)' appears to reverse the families: in Section 5 and Table 4, Binoculars is a white-box detector and GECScore is a black-box detector.
  3. [Figure 2] The figure does not explain what is plotted on each axis, how the N=600 sample was selected, or which 'dimensions' are being compared; please add a legend or a more detailed caption.
  4. [Appendix B.3, Table 9 vs Table 6] Table 9 reports 300,000 English sentence pairs for the style classifier, while Table 6 reports 286,626 processed sentence pairs for the English TST subset; if the classifier was trained on a smaller filtered subset, the table caption should state this explicitly.
  5. [§6, Table 4] No confidence intervals or significance tests are reported for the accuracy differences in Table 4; given finite sample sizes (2,700 human texts per condition), the authors should add uncertainty estimates or note which differences are within noise.
  6. [§4.1, Appendix B.1.3] QAFactEval scores for Portuguese and Vietnamese are computed via GPT-4 translations; this could bias the cross-language prompt comparison, and the manuscript does not validate this translation step.

Circularity Check

1 steps flagged · score 1.0 of 10

Minimal circularity: only a supporting 'closer proximity' observation is built in by prompt selection; central detector accuracies are empirically measured.

  1. self definitional [Section 1 (Figure 2) and Section 4.1 (Paragraph Writing prompt selection)]
    "We evaluate these prompts using standard automatic metrics: BLEU ... and ROUGE ... for n-gram overlap, BERTScore ... for semantic similarity, and QAFactEval ... for factual consistency between HWT and MGT. ... Based on these findings, we adopt Naive RAG as the prompting strategy for the Paragraph Writing task in our MGT detection experiments. ... Figure 2 shows that the textual characteristics of task-specific MGT—unlike open-ended, topic-to-text MGT—more closely resemble their human-written text (HWT) references."

    The same closeness metrics that define 'best-performing prompt' are later reported as the finding that WETBench MGT resembles HWT. Section 4.1 selects prompts by BLEU/ROUGE/BERTScore/QAFactEval between HWT and MGT, and analogous overlap/similarity-based selection picks one-shot summarisation and five-shot TST; generators are then run with those prompts. Figure 2's 'closer proximity' is therefore partly an artefact of the selection criterion, not an independent property of task-specific editing. This is a self-definitional supporting observation. It does not make the main accuracy results circular, because Table 4 reports measured detector performance rather than a quantity derived from the prompt-quality metrics, and no fitted detector parameter is renamed as a prediction.

full rationale

The central claims of WETBench are benchmark measurements, not derivations. The benchmark is constructed from pre-ChatGPT Wikipedia text and independently generated MGT; detectors are then evaluated on their own held-out splits, so the reported supervised 78% / zero-shot 58% accuracies do not reduce to any fitted parameter of the paper. The footnote self-citation to the extended ICLR version (Quaremba et al. 2026) is not load-bearing. The acknowledged limitations—no empirical evidence that editors systematically use LLMs in exactly these ways, weak NPOV classifiers for Vietnamese and English paragraphs, and the absence of a within-benchmark open-ended control when comparing to prior work—are validity or generalisability concerns, not circularity. The only mildly circular element is the Figure 2 'closer proximity' observation, which is a byproduct of selecting prompts with the same textual-similarity metrics; this supports, but is not required for, the independently measured detector results. Overall circularity is therefore minimal.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The benchmark's findings rest on empirical and design assumptions rather than mathematical axioms. The most consequential are the contamination-free pre-ChatGPT Wikipedia corpus, the validity of perceived editor use cases as a proxy for real workflows, the reliability of automatic metrics for prompt selection, and the precision of low-accuracy style classifiers for NPOV pair extraction. No new theoretical entities are introduced.

free parameters (3)
  • Zero-shot detector thresholds (Youden's J) = Tuned per detector, language, and task
    Detectors like Binoculars, LLR, FastDetectGPT, GECScore, and Revise require a threshold to convert scores to predictions; the paper tunes this on each evaluation subset to maximize Youden's J, so detection accuracy includes a fitted threshold.
  • Prompt configuration per task = Naive RAG (Paragraph Writing), one-shot (Summarisation), five-shot (TST)
    Chosen from a 10% sample using automatic quality metrics. The benchmark's generation difficulty (and hence detector accuracy) depends on this data-driven selection.
  • Style classifier decision rule for TST = Precision-maximising configuration, exact threshold not reported
    For Vietnamese and English paragraph TST, classifiers achieve only about 58% accuracy; the paper selects a conservative, precision-maximizing configuration, but the threshold is not disclosed, making the prompt selection partially unquantified.
assumptions (4)
  • domain assumption Wikipedia mainspace articles as of November 2022 are free of machine-generated text.
    Section 3.1 and Appendix A use only pre-ChatGPT revisions to construct WikiPS, stating 'that our data is not contaminated by MGT'. If contamination exists, the human-written reference texts are biased.
  • domain assumption Ford et al. (2023) perceived use cases are a valid basis for defining realistic editing tasks.
    Section 4 and Section 7: tasks are grounded in editors' perceived opportunities, but the paper acknowledges the lack of empirical evidence that editors use LLMs in these ways.
  • domain assumption Automatic metrics (BLEU, ROUGE, BERTScore, QAFactEval) reliably rank generation quality for prompt selection.
    Sections 4.1-4.3 select prompts based on these metrics, including QAFactEval scores computed via GPT-4 translations for Portuguese and Vietnamese, which introduces translation-model error.
  • domain assumption Low-accuracy TST style classifiers still yield high-precision NPOV pairs.
    Appendix B.3.2: for Vietnamese sentences (58%) and English paragraphs (58%), the classifiers are near chance; the paper asserts high precision in the extracted pairs, but this is not independently verified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of WETBench: A Benchmark for Detecting Task-Specific Machine-Generated Text on Wikipedia." pith.science (2026). https://pith.science/paper/H2LO5NIA

@misc{pith2026250703373,
  author       = {Pith},
  title        = {Pith review of: WETBench: A Benchmark for Detecting Task-Specific Machine-Generated Text on Wikipedia},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H2LO5NIA}},
  note         = {Machine review of arXiv:2507.03373}
}
read the original abstract

Given Wikipedia's role as a trusted source of high-quality, reliable content, concerns are growing about the proliferation of low-quality machine-generated text (MGT) produced by large language models (LLMs) on its platform. Reliable detection of MGT is therefore essential. However, existing work primarily evaluates MGT detectors on generic generation tasks rather than on tasks more commonly performed by Wikipedia editors. This misalignment can lead to poor generalisability when applied in real-world Wikipedia contexts. We introduce WETBench, a multilingual, multi-generator, and task-specific benchmark for MGT detection. We define three editing tasks, empirically grounded in Wikipedia editors' perceived use cases for LLM-assisted editing: Paragraph Writing, Summarisation, and Text Style Transfer, which we implement using two new datasets across three languages. For each writing task, we evaluate three prompts, generate MGT across multiple generators using the best-performing prompt, and benchmark diverse detectors. We find that, across settings, training-based detectors achieve an average accuracy of 78%, while zero-shot detectors average 58%. These results show that detectors struggle with MGT in realistic generation scenarios and underscore the importance of evaluating such models on diverse, task-specific data to assess their reliability in editor-driven contexts.

Figures

Figures reproduced from arXiv: 2507.03373 by the authors.

Figure 1
Figure 1. We define task-specific editing scenarios on Wikipedia, test various prompting techniques, generate LLM-written text using the best-performing prompts, and benchmark SOTA detectors on these data. This contrasts with prior work, which primarily focuses on a single, open-ended generation task that only partially captures the real-world editorial use of LLMs. increasing prevalence of machine-generated text (MGT) produc… view at source ↗
Figure 2
Figure 2. Comparison of MGT and HWT (N=600) for English Wikipedia Open-Ended Generation (WOE) vs. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 5 canonical work pages

  1. [3]

    Shen Gao, Xiuying Chen, Chang Liu, Dongyan Zhao, and Rui Y an

    Bias and fairness in large language models: A survey.Preprint, arXiv:2309.00770. Shen Gao, Xiuying Chen, Chang Liu, Dongyan Zhao, and Rui Y an. 2021. Biogen: Generating biogra- phy summary under table guidance on wikipedia. In Findings of the Association for Computational Lin- guistics: ACL-IJCNLP 2021, pages 4752–4757. Yunfan Gao, Yun Xiong, Xinyu Gao, K...

    arXiv 2021

  2. [5]

    Hiroaki Hayashi, Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan, and Graham Neubig

    Spotting llms with binoculars: Zero-shot detection of machine-generated text.arXiv preprint arXiv:2401.12070. Hiroaki Hayashi, Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan, and Graham Neubig

    arXiv

  3. [7]

    InProceedings of the ACM Web Conference 2023, WWW ’23, page 1446–1456, New Y ork, NY , USA

    Descartes: Generating short descriptions of wikipedia articles. InProceedings of the ACM Web Conference 2023, WWW ’23, page 1446–1456, New Y ork, NY , USA. Association for Computing Machin- ery. K. Salas-Jimenez, Francisco Fernando Lopez-Ponce, Sergio-Luis Ojeda-Trueba, and Gemma Bel-Enguix

    work page 2023

  4. [8]

    InProceedings of the First Work- shop on Advancing Natural Language Processing for Wikipedia, pages 46–52, Miami, Florida, USA

    WikiBias as an extrapolation corpus for bias detection. InProceedings of the First Work- shop on Advancing Natural Language Processing for Wikipedia, pages 46–52, Miami, Florida, USA. Association for Computational Linguistics. Christina Sauper and Regina Barzilay. 2009. Auto- matically generating Wikipedia articles: A structure- aware approach. InProceedi...

    arXiv 2009

  5. [10]

    See also

    WIKIBIAS: Detecting multi-span subjective biases in language. InFindings of the Association for Computational Linguistics: EMNLP 2021, pages 1799–1814, Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics. Biru Zhu, Lifan Yuan, Ganqu Cui, Y angyi Chen, Chong Fu, Bingxiang He, Y angdong Deng, Zhiyuan Liu, Maosong Sun, and Ming Gu. 20...

    arXiv 2021

  6. [11]

    for Wikimedia content, CNN/DM (Nalla- pati et al., 2016) for news, and arXiv (Cohan et al.,

    work page 2016

  7. [2018]

    { section_title}

    for academic writing. On average, our summaries are considerably longer than those in WikiLingua and CNN/DM, but shorter than arXiv abstracts. The average body length in our datasets is comparable to CN- N/DM but significantly shorter than arXiv. De- spite this, our datasets exhibit higher ROUGE-1 and ROUGE-2 scores, indicating improved con- tent overlap....

    work page 2020

  8. [2019]

    mDeBERTaV3mDeBERTaV342 is the multilin- gual version of DeBERTa (He et al., 2023), which enhances BERT and RoBERTa using disentangled attention and an improved masked decoder

    through longer and more extensive training, as well as dynamic masking. mDeBERTaV3mDeBERTaV342 is the multilin- gual version of DeBERTa (He et al., 2023), which enhances BERT and RoBERTa using disentangled attention and an improved masked decoder. Both models are fine-tuned per task and lan- guage on an 80/10/10 split with the hyperparameter choices displ...

    work page 2023

Show all 13 references
  1. [2020]

    Preprint, arXiv:2005.10070

    A large-scale multi-document summariza- tion dataset from the wikipedia current events portal. Preprint, arXiv:2005.10070. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3.arXiv preprint arXiv:2209.12356. Max Grusky, Mo...

    2005 arXiv

  2. [2021]

    Pengcheng He, Jianfeng Gao, and Weizhu Chen

    Wikiasp: A dataset for multi-domain aspect- based summarization.Transactions of the Associa- tion for Computational Linguistics, 9:211–225. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangl...

    2023 arXiv

  3. [2023]

    Accessed: 2025-04-06

    Implications of chatgpt for knowledge integrity on wikipedia. Accessed: 2025-04-06. Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernon- court, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed

    2025

  4. [2024]

    Preprint, arXiv:2410.08044

    The rise of ai-generated content in wikipedia. Preprint, arXiv:2410.08044. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen ...

    arXiv

  5. [2025]

    Jianhao Y an, Pingchuan Y an, Yulong Chen, Judy Li, Xianchao Zhu, and Yue Zhang

    Who wrote this? the key to zero-shot llm-generated text detection is gecscore.Preprint, arXiv:2405.04286. Jianhao Y an, Pingchuan Y an, Yulong Chen, Judy Li, Xianchao Zhu, and Yue Zhang. 2024. Gpt-4 vs. human translators: A comprehensive evaluation of translation quality acros...

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.