Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Beyond Easy Wins: A Text Hardness-Aware Benchmark for LLM-generated Text Detection

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Conventional AUROC overestimates AI-text detector quality, and a new reliability-stability benchmark exposes it.

desk verdict A serious benchmark paper with a credible central result, but a ground-truth contamination in the pink slime subset needs fixing before the numbers can be trusted. read the letter →

arxiv 2507.15286 v1 pith:CLEXO4LS submitted 2025-07-21 cs.CL

Navid Ayoobi , Sadat Shahriar , Arjun Mukherjee This is my paper
classification cs.CL
keywords LLM-generatedtextdetectionbenchmarkevaluationzero-shothumanificationAUROCfalsepositiveratedetectorstabilityhardness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the standard AUROC statistic paints a misleadingly rosy picture of AI-text detectors, because it averages over false-positive regions that are unacceptable in practice and ignores how much a detector's operating point wobbles across domains. To fix this, it introduces SHIELD, a benchmark whose evaluation metric weights performance at low false-positive rates and multiplies it by a stability score, so a detector must be both precise where it matters and steady across writing styles and generator models. The paper also builds a "humanification" pipeline that rewrites machine text with word-level substitutions of increasing hardness. On this benchmark, six zero-shot detectors lose on average about 80 percent of their reliability-stability score after word-level edits that resemble ordinary user editing, while a paraphrase-trained supervised detector does not degrade. The upshot is a call to evaluate detectors where they must actually operate, not on easy unmodified text.

What carries the argument

The machinery is a mutual-information word ranker combined with a masked language model replacer. Each word gets an MI score measuring how much observing it shifts belief about authorship, and the vocabulary is split into AI-associated and human-associated sets based on frequency. Three humanification strategies then edit AI text: random meaning-preserving mutation masks a random p fraction of non-stop words and fills them with top-1 MLM predictions; AI-flagged word swap masks the p fraction of words with the highest AI-MI and replaces each with the highest-MI human candidate; and the recursive loop repeats the swap for R rounds at a fixed 10% share. A hardness knob, p or R, controls how much of the text is edited. On the evaluation side, the load-bearing identities are W-AUROC as an FPR-decayed expectation of TPR, SFD as exp(-lambda times the standard deviation of FPR over Youden thresholds), and URSS as their product, which enforces that high discrimination cannot compensate for instability or vice versa.

What would settle it

Manually audit a random sample of the pink-slime subset for signs of automated generation, such as boilerplate repetition and template structure; if a notable fraction are machine-written, recompute the detector scores and URSS rankings to see whether the reported degradations and equivalences survive.

Watch

Extended reading notes

Core claim

The central claim is that AUROC overestimates detector efficacy in real deployments, and that a fair comparison needs to score detectors in the low-FPR region and penalize instability. The paper's SHIELD metrics do this: W-AUROC is expected true positive rate under an exponential weighting that halves at 5% FPR, SFD is exp(-lambda times the standard deviation of FPR at each scenario's Youden-optimal threshold), and URSS multiplies the average W-AUROC by SFD. Applying these metrics, detectors with similar AUROC separate sharply: in the Reddit style, Radar and Rank have equal URSS despite very different AUROC, while Binoculars, despite the highest AUROC, ranks last on URSS because its optimal-threshold FPR varies so much. In the humanification experiments, all six zero-shot detectors collapse under the word-replacement strategies, with average URSS falling roughly 80 percent, because their token-level statistical signals are fragile, whereas Radar, trained adversarially against paraphrasing, improves under the same replacements. The paper's conclusion is that conventional metrics hide both practical equivalences and practical failures, and that detection methods should be judged by reliability at low false positives and stability across scenarios.

Load-bearing premise

The benchmark's ground truth assumes every text labeled "human" was genuinely written by a person, but the pink-slime style samples 2021 articles from local-news content farms that may themselves be automated.

Editorial extensions

If this is right

  • Deployment decisions should compare detectors by URSS-style scores rather than AUROC, since AUROC can rank a practically unstable detector above a stable equivalent.
  • Zero-shot detectors that rely on next-token probability signals need hardening against word-level edits, because even untargeted random substitutions resembling normal editing collapse their scores.
  • Supervised detectors trained against paraphrasing, like Radar, can be more robust to word-level humanification, suggesting adversarial training is a viable defense path.
  • Benchmark results will change as generator models change, so maintaining SHIELD-style datasets requires periodic regeneration of LLM samples.
  • The equal-URSS cases provide a fair comparison criterion: when two detectors trade off low-FPR performance and stability, URSS treats them as practically equivalent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence outside the paper: feeding humanified text back into detector training, or training detectors on MI-based substitutions, should make zero-shot methods harder to fool; the paper does not run that experiment.
  • The same hardness ladder could be applied to watermarking schemes, because word-level humanification that erases statistical signals likely also weakens watermark traces, a question the paper leaves open.
  • The paper's stated limitations, English-only data, open-source generators only, and evolving model signatures, imply the 80% figure should not be read as a universal constant; re-running SHIELD on multilingual text and commercial models could shift the numbers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. SHIELD is a benchmark for evaluating LLM-written-text detectors under realistic operating conditions. It introduces three new metrics (W-AUROC, SFD, URSS), a humanification framework with three strategies (RMM, AWS, RHL), and a large dataset spanning seven writing styles and seven LLMs. The central empirical claim is that conventional AUROC overestimates detector efficacy and that zero-shot detectors suffer severe URSS degradation under word-level humanification, with an average degradation of about 80% reported in the conclusion.

Significance. If validated, the paper makes a useful contribution: it provides a large benchmark with structured hardness levels, publicly releases data and code, and proposes metrics that penalize high-FPR operation and threshold instability. The qualitative finding that zero-shot detectors collapse under simple word substitutions is practically important and broadly consistent with existing robustness results. However, the quantitative claims are conditional on fixing a ground-truth contradiction and on adding uncertainty quantification; the reported rankings and the 80% headline number currently rest on point estimates over possibly contaminated labels.

major comments (4)
  1. [3.1, A.1] Section 3.1 claims that to guarantee human authorship the dataset comprises exclusively pre-2021 data, but Appendix A.1 states that the pink slime style is sampled from NELA-PS articles 'published in 2021' and describes these outlets as producing content that 'mimics legitimate local journalism.' This is a direct internal contradiction. If a non-negligible fraction of the 12.5k pink slime documents is machine-generated or templated, the positive class for one of the seven styles is contaminated, and the headline ~80% average URSS degradation (Section 4.2, Conclusion) as well as all per-style rankings inherit label error. The authors should either replace this style with verifiable human-authored local news, filter the subset for genuine authorship, or rerun all experiments without pink slime and report both versions.
  2. [Tables 2-4, Figure 3] Tables 2 and 4 report every metric as a single percentage with no error bars, confidence intervals, or significance tests, and Figure 3 plots point trajectories without variance information. The paper's load-bearing comparative claims, including the 'practical equivalence' of Radar and Rank on Reddit (Section 4.1), the degradation percentages in Section 4.2, and the conclusion's 80% average, are therefore not statistically supported. The authors should provide bootstrap intervals over documents, styles, and LLMs, or at minimum report the sample sizes and pairwise significance tests for the URSS rankings.
  3. [3.3.2, Eq. (11)-(13)] Section 3.3.2 defines the Youden threshold t*_i as the threshold maximizing TPR_i(t)-FPR_i(t) on each scenario's ROC curve, then evaluates FPR_i(t*_i) on the same data. This is an in-sample optimal threshold, not the predetermined deployment threshold that Section 1 motivates, so SFD measures the variability of an optimistically chosen FPR rather than threshold stability under fixed operational configuration. Please re-estimate SFD using a fixed operational FPR (e.g., 1% or 5%) or thresholds calibrated on a held-out portion of each scenario, and report sensitivity to that choice.
  4. [3.2] The humanification strategies estimate the A and H vocabularies from corpus-level frequency and MI scores, but the paper does not state whether the human corpus used to construct these vocabularies is disjoint from the human documents in the evaluation set. If the same documents are used, the attack is white-box with respect to the test distribution and the measured degradation is an upper bound rather than a realistic estimate. Please specify the split; if it is not disjoint, rerun with a held-out human corpus for ranker construction.
minor comments (6)
  1. [5, Conclusion] The acronym is spelled 'USRR' once in Section 5; it should be URSS.
  2. [Table 3] The RHL rows are labeled '@R=5%' and similar, but R is the number of rounds, not a percentage, and the listed values (5, 15, 20, 25, 30, 35, 40) omit R=10; fix the labels and the sequence.
  3. [3.3.1] The statement that exp(-k*FPR) decays to 50% at FPR=0.05 calibrates the unnormalized kernel, while W-AUROC integrates the normalized density p(t); please state this distinction explicitly.
  4. [A.1] The term 'pink slime' carries a pejorative connotation; the description should be neutral and should explicitly note that these outlets are suspected of automated content generation, which is precisely the ground-truth concern raised in the major comments.
  5. [Table 1] Several entries contain stray punctuation such as 'MAGE ;' and 'BUST;'; please clean up the table formatting.
  6. [3.1, Table 3] Section 3.1 says the dataset contains 87.5k human documents, but Table 3 reports only 3000 human documents per style for the evaluation; clarify whether Table 3 is a subsample of the full dataset.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the SHIELD metrics and humanification pipeline are self-contained definitions plus external detector evaluations, with only a minor non-load-bearing self-citation and a non-circular pink-slime label inconsistency.

full rationale

The paper's claimed derivation chain is not circular. W-AUROC (Eq. 10), SFD (Eqs. 11-12), and URSS (Eq. 13) are explicit definitions over detector score distributions; the decay constants k=20 ln2 and lambda=10 ln2 are hand-set calibration choices anchored to external conventions (TPR at FPR below 5%; sigma_FPR=0.1 as a mid-range instability), not fitted to detector outputs, so the metrics measure detectors independently of their construction. The humanification attack is an external perturbation generator: the A/H vocabularies in Section 3.2 are estimated from the corpus, but detector scores on the resulting texts are measured, not derived from the attack parameters; the fact that Radar improves under humanification confirms the outcome is not forced by construction. The only self-citation (ESPERANTO, Ayoobi et al. 2024) appears in the related-work and benchmark comparison (Table 1) and is not load-bearing. Two non-circular weaknesses are noted for the record: first, the A/H vocabulary is estimated on the same corpus from which test documents are drawn, a self-referential attack-design choice that may overstate attack success but does not make the derivation circular; second, Section 3.1's pre-2021 human-authorship guarantee is contradicted by Appendix A.1, which samples 12.5k NELA-PS pink-slime articles 'published in 2021' from outlets that 'generate content that mimics legitimate local journalism' — a ground-truth consistency risk that does not by itself reduce any result to its inputs. The Limitations section acknowledges scope restrictions (monolingual English, open-source models only) that are not circularity.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central results depend on two hand-set decay constants (k, lambda), an ad hoc frequency threshold defining the AI-associated and human-associated vocabularies, the unvalidated assumption that masked-LM replacements preserve meaning, and the "pre-2021 means human" label guarantee that the pink slime subset contradicts. The hardness knobs p and R are experimental controls, not fitted constants. No new physical or conceptual entities are introduced.

free parameters (6)
  • k (W-AUROC decay) = 20 ln2 ≈ 13.86
    Hand-set so the exponential weight equals 0.5 at FPR=0.05, motivated by TPR-at-low-FPR conventions; every W-AUROC and URSS value depends on it.
  • lambda (SFD decay) = 10 ln2 ≈ 6.93
    Hand-set so SFD=0.5 when sigma_FPR=0.1; every stability score depends on it.
  • p (hardness knob for RMM and AWS) = 10% to 100%
    Fraction of non-stop words replaced; controls hardness levels in Figure 3.
  • R (rounds for RHL) = 5 to 40
    Number of recursive humanification rounds; controls hardness levels.
  • p_o (per-round replacement in RHL) = 10%
    Fixed fraction masked in each RHL round, chosen without stated justification.
  • Per-scenario Youden threshold t* = argmax_t (TPR(t)-FPR(t))
    SFD evaluates false-positive rates at each scenario's own optimal threshold instead of a single predetermined deployment threshold.
assumptions (6)
  • domain assumption Pre-2021 publication implies human authorship (Section 3.1); the pink slime subset using 2021 articles contradicts this.
    Invoked in Section 3.1; the dataset's human labels rest on it, but Appendix A.1 samples pink slime articles published in 2021 from automated content farms.
  • domain assumption Empirical mutual information over the benchmark corpus ranks per-word authorship signal.
    Underpins the A/H vocabulary split used by AWS and RHL in Sections 3.2.2 and 3.2.3; no validation that the MI ranking matches human judgments.
  • ad hoc to paper The f_H,i > f_A,i threshold separates human- from AI-associated vocabularies.
    Used in the Ranker/Splitter (Figure 1a); the threshold is arbitrary and unvalidated.
  • domain assumption Longformer masked-LM replacements preserve meaning and fluency.
    Assumed throughout Section 3.2; no human or automatic semantic-preservation evaluation is reported, and Appendix B shows only illustrative samples.
  • ad hoc to paper Exponential weighting exp(-k*FPR) models operationally relevant low-FPR regions.
    Definition of W-AUROC (Eq. 10); the functional form and the decay constant k are chosen by the authors.
  • domain assumption Youden's J is the appropriate per-scenario operating threshold for stability measurement.
    Used in Eq. 11; conflicts with the paper's stated real-world premise of a single predetermined deployment threshold.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Easy Wins: A Text Hardness-Aware Benchmark for LLM-generated Text Detection." pith.science (2026). https://pith.science/paper/CLEXO4LS

@misc{pith2026250715286,
  author       = {Pith},
  title        = {Pith review of: Beyond Easy Wins: A Text Hardness-Aware Benchmark for LLM-generated Text Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CLEXO4LS}},
  note         = {Machine review of arXiv:2507.15286}
}
read the original abstract

We present a novel evaluation paradigm for AI text detectors that prioritizes real-world and equitable assessment. Current approaches predominantly report conventional metrics like AUROC, overlooking that even modest false positive rates constitute a critical impediment to practical deployment of detection systems. Furthermore, real-world deployment necessitates predetermined threshold configuration, making detector stability (i.e. the maintenance of consistent performance across diverse domains and adversarial scenarios), a critical factor. These aspects have been largely ignored in previous research and benchmarks. Our benchmark, SHIELD, addresses these limitations by integrating both reliability and stability factors into a unified evaluation metric designed for practical assessment. Furthermore, we develop a post-hoc, model-agnostic humanification framework that modifies AI text to more closely resemble human authorship, incorporating a controllable hardness parameter. This hardness-aware approach effectively challenges current SOTA zero-shot detection methods in maintaining both reliability and stability. (Data and code: https://github.com/navid-aub/SHIELD-Benchmark)

Figures

Figures reproduced from arXiv: 2507.15286 by the authors.

Figure 1
Figure 1. Humanification strategies based on the ranked [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 4
Figure 4. Word count distribution of human-written and LLM-generated texts aggregated across all LLMs. “All” [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ARB: A Matched Authorship-Rewriting Benchmark Dataset for AI-Text Detector Evaluation

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A matched four-regime benchmark shows AI-text detectors catch direct LLM output but lose most of their recall on human text rewritten by an LLM.

  2. AEyeDE: An Attention-Based Attribution Framework for AI-Generated Text Detection

    cs.CL 2026-04 conditional novelty 6.0 of 10

    Attention attribution maps from a white-box proxy Transformer, classified by a lightweight CNN, provide a competitive and interpretable signal for AI-generated text detection.

Reference graph

Works this paper leans on

57 extracted references · 18 canonical work pages · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Mervat Abassy, Kareem Elozeiri, Alexander Aziz, Minh Ngoc Ta, Raj Vardhan Tomar, Bimarsha Adhikari, Saad El Dine Ahmed, Yuxia Wang, Osama Mohammed Afzal, Zhuohan Xie, Jonibek Mansurov, Ekaterina Artemova, Vladislav Mikhailov, Rui Xing, Jiahui Geng, Hasan Iqbal, Zain Muhammad Mujahid, Tarek Mahmoud, Akim Tsvigun, and 5 others. 2024. https://doi.org/10.1865...

    doi:10.18653/v1/2024.emnlp-demo.35 2024

  4. [4]

    Navid Ayoobi, Lily Knab, Wen Cheng, David Pantoja, Hamidreza Alikhani, Sylvain Flamant, Jin Kim, and Arjun Mukherjee. 2024. Esperanto: Evaluating synthesized phrases to enhance robustness in ai detection for text origination. arXiv preprint arXiv:2409.14285

    work page Pith review arXiv 2024

  5. [5]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  6. [6]

    Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. 2023. Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature. arXiv preprint arXiv:2310.05130

    arXiv 2023

  7. [7]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150

    arXiv 2020

  8. [8]

    Hung-Yun Chiang, Yi-Syuan Chen, Yun-Zhu Song, Hong-Han Shuai, and Jason S. Chang. 2023. https://doi.org/10.1145/3580305.3599502 Shilling black-box review-based recommender systems through fake review generation . In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, page 286–297, New York, NY, USA. Association f...

    arXiv 2023

Show all 57 references
  1. [9]

    Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. 2021. https://doi.org/10.18653/v1/2021.acl-long.565 All that`s human' is not gold: Evaluating human evaluation of generated text . In Proceedings of the 59th Annual Meeting of the...

    2021 doi

  2. [10]

    Colin B Clement, Matthew Bierbaum, Kevin P O'Keeffe, and Alexander A Alemi. 2019. On the use of arxiv as a dataset. arXiv preprint arXiv:1905.00075

    2019 arXiv

  3. [11]

    Joseph Cornelius, Oscar Lithgow-Serrano, Sandra Mitrovic, Ljiljana Dolamic, and Fabio Rinaldi. 2024. https://doi.org/10.18653/v1/2024.naacl-long.444 BUST : Benchmark for the evaluation of detectors of LLM -generated text . In Proceedings of the 2024 Conference of the North Ame...

    2024 doi

  4. [12]

    Debby R. E. Cotton, Peter A. Cotton, and J. Reuben Shipway. 2024. https://doi.org/10.1080/14703297.2023.2190148 Chatting and cheating: Ensuring academic integrity in the era of chatgpt . Innovations in Education and Teaching International, 61(2):228--239

    2024

  5. [13]

    Smith, and Yejin Choi

    Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A. Smith, and Yejin Choi. 2022. https://doi.org/10.18653/v1/2022.acl-long.501 Is GPT -3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text . In Proceedings of the 60th Annual Meeting...

    2022 doi

  6. [14]

    Liam Dugan, Alyssa Hwang, Filip Trhl \'i k, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, and Chris Callison-Burch. 2024. https://doi.org/10.18653/v1/2024.acl-long.674 RAID : A shared benchmark for robust evaluation of machine-generated text detectors . In Proceed...

    2024 doi

  7. [15]

    Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. https://doi.org/10.18653/v1/P19-1346 ELI 5: Long form question answering . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558--356...

    2019 doi

  8. [16]

    Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019. https://doi.org/10.18653/v1/P19-3019 GLTR : Statistical detection and visualization of generated text . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrat...

    2019 doi

  9. [17]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  10. [18]

    Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597

    2023 arXiv

  11. [19]

    Hanxi Guo, Siyuan Cheng, Xiaolong Jin, Zhuo Zhang, Kaiyuan Zhang, Guanhong Tao, Guangyu Shen, and Xiangyu Zhang. 2024 a . https://proceedings.neurips.cc/paper_files/paper/2024/file/bc808cf2d2444b0abcceca366b771389-Paper-Conference.pdf Biscope: Ai-generated text detection by ch...

    2024

  12. [20]

    Xun Guo, Shan Zhang, Yongxin He, Ting Zhang, Wanquan Feng, Haibin Huang, and Chongyang Ma. 2024 b . https://proceedings.neurips.cc/paper_files/paper/2024/file/a117a3cd54b7affad04618c77c2fb18b-Paper-Conference.pdf Detective: Detecting ai-generated text via multi-level contrasti...

    2024

  13. [21]

    Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2024. Spotting LLM s with binoculars: zero-shot detection of machine-generated text. In Proceedings of the 41st International Conference on ...

    2024

  14. [22]

    Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. 2024. https://doi.org/10.1145/3658644.3670344 Mgtbench: Benchmarking machine-generated text detection . In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS '24, pa...

    2024

  15. [23]

    Horne and Maurício Gruppi

    Benjamin D. Horne and Maurício Gruppi. 2024. https://doi.org/10.1609/icwsm.v18i1.31439 Nela-ps: A dataset of pink slime news articles for the study of local news ecosystems . Proceedings of the International AAAI Conference on Web and Social Media, 18(1):1958--1966

    2024 doi

  16. [24]

    Guiyang Hou, Yongliang Shen, and Weiming Lu. 2024. https://doi.org/10.18653/v1/2024.findings-acl.855 Progressive tuning: Towards generic sentiment abilities for large language models . In Findings of the Association for Computational Linguistics: ACL 2024, pages 14392--14402, ...

    2024 doi

  17. [25]

    Xiaomeng Hu, Pin-Yu Chen, and Tsung-Yi Ho. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/30e15e5941ae0cdab7ef58cc8d59a4ca-Paper-Conference.pdf Radar: Robust ai-text detection via adversarial learning . In Advances in Neural Information Processing Systems, vo...

    2023

  18. [26]

    Guanhua Huang, Yuchen Zhang, Zhe Li, Yongjian You, Mingze Wang, and Zhouwang Yang. 2024. https://doi.org/10.18653/v1/2024.acl-long.327 Are AI -generated text detectors robust to adversarial perturbations? In Proceedings of the 62nd Annual Meeting of the Association for Computa...

    2024 doi

  19. [27]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  20. [28]

    Ehsan Kamalloo, Nouha Dziri, Charles Clarke, and Davood Rafiei. 2023. https://doi.org/10.18653/v1/2023.acl-long.307 Evaluating open-domain question answering in the era of large language models . In Proceedings of the 61st Annual Meeting of the Association for Computational Li...

    2023 doi

  21. [29]

    John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. https://proceedings.mlr.press/v202/kirchenbauer23a.html A watermark for large language models . In Proceedings of the 40th International Conference on Machine Learning, volume 202 o...

    2023

  22. [30]

    Kristian Kuznetsov, Eduard Tulchinskii, Laida Kushnareva, German Magai, Serguei Barannikov, Sergey Nikolenko, and Irina Piontkovskaya. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.992 Robust AI -generated text detection by restricted embeddings . In Findings of the As...

    2024 doi

  23. [31]

    Zhixin Lai, Xuesheng Zhang, and Suiyao Chen. 2024. https://doi.org/10.1109/IJCNN60899.2024.10651296 Adaptive ensembles of fine-tuned transformers for LLM -generated text detection . In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1--7

    2024

  24. [32]

    Thomas Lavergne, Tanguy Urvoy, and Fran c ois Yvon. 2008. Detecting fake content with relative entropy scoring. In Proceedings of the 2008 International Conference on Uncovering Plagiarism, Authorship and Social Software Misuse - Volume 377, PAN'08, page 27–31, Aachen, DEU. CE...

    2008

  25. [33]

    Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Zhilin Wang, Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. 2024. https://doi.org/10.18653/v1/2024.acl-long.3 MAGE : Machine-generated text detection in the wild . In Proceedings of the 62nd Annual Meeting of the Association for ...

    2024 doi

  26. [34]

    Yepeng Liu and Yuheng Bu. 2024. Adaptive text watermark for large language models. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org

    2024

  27. [35]

    Zhuang Liu, Wayne Lin, Ya Shi, and Jun Zhao. 2021. A robustly optimized bert pre-training approach with post-training. In Chinese Computational Linguistics, pages 471--484, Cham. Springer International Publishing

    2021

  28. [36]

    Shixuan Ma and Quan Wang. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.971 Zero-shot detection of LLM -generated text using token cohesiveness . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17538--17553, Miami, Florida, ...

    2024 doi

  29. [37]

    Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, and 1 others. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

    2024 arXiv

  30. [38]

    Manning, and Chelsea Finn

    Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. Detectgpt: zero-shot machine-generated text detection using probability curvature. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org

    2023

  31. [39]

    Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. https://doi.org/10.18653/v1/D19-1018 Justifying recommendations using distantly-labeled reviews and fine-grained aspects . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In...

    2019 doi

  32. [40]

    Michael-Andrei Panaitescu-Liess, Zora Che, Bang An, Yuancheng Xu, Pankayaraj Pathmanathan, Souradip Chakraborty, Sicheng Zhu, Tom Goldstein, and Furong Huang. 2025. https://doi.org/10.1609/aaai.v39i23.34684 Can watermarking large language models prevent copyrighted text genera...

    2025 doi

  33. [41]

    Qi Pang, Shengyuan Hu, Wenting Zheng, and Virginia Smith. 2024. No free lunch in LLM watermarking: Trade-offs in watermarking design choices. arXiv preprint arXiv:2402.16187

    2024 arXiv

  34. [42]

    Shushanta Pudasaini, Luis Miralles, David Lillis, and Marisa Llorens Salvador. 2025. https://aclanthology.org/2025.genaidetect-1.4/ Benchmarking AI text detection: Assessing detectors against new datasets, evasion tactics, and enhanced LLM s . In Proceedings of the 1stWorkshop...

    2025

  35. [43]

    Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, S.M Towhidul Islam Tonmoy, Aman Chadha, Amit Sheth, and Amitava Das. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.155 The troubling emergence of hallucination in large language models - an extensive definit...

    2023 doi

  36. [44]

    Jinyan Su, Terry Zhuo, Di Wang, and Preslav Nakov. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.827 D etect LLM : Leveraging log rank information for zero-shot detection of machine-generated text . In Findings of the Association for Computational Linguistics: EMNLP 20...

    2023 doi

  37. [45]

    Adaku Uchendu, Zeyu Ma, Thai Le, Rui Zhang, and Dongwon Lee. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.172 TURINGBENCH : A benchmark environment for T uring test in the age of neural text generation . In Findings of the Association for Computational Linguistics: EM...

    2021 doi

  38. [46]

    Ivan Vykopal, Mat \'u s Pikuliak, Ivan Srba, Robert Moro, Dominik Macko, and Maria Bielikova. 2024. https://doi.org/10.18653/v1/2024.acl-long.793 Disinformation capabilities of large language models . In Proceedings of the 62nd Annual Meeting of the Association for Computation...

    2024 doi

  39. [47]

    Jan Philip Wahle, Terry Ruas, Frederic Kirstein, and Bela Gipp. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.62 How large language models are transforming machine-paraphrase plagiarism . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Proces...

    2022 doi

  40. [48]

    Yiming Wang, Zhuosheng Zhang, and Rui Wang. 2023. https://doi.org/10.18653/v1/2023.acl-long.482 Element-aware summarization with large language models: Expert-aligned evaluation and chain-of-thought method . In Proceedings of the 61st Annual Meeting of the Association for Comp...

    2023 doi

  41. [49]

    Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Puccetti, Thomas Arnold, Alham Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024. https://doi.org/10.18653/v1/2024.acl-long.218 M 4 GT -...

    2024 doi

  42. [50]

    Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Lidia Sam Chao, and Derek Fai Wong. 2025. https://doi.org/10.1162/coli_a_00549 A survey on LLM -generated text detection: Necessity, methods, and future directions . Computational Linguistics, 51(1):275--338

    2025 doi

  43. [51]

    Wong, Shu Yang, Xinyi Yang, Yulin Yuan, and Lidia S

    Junchao Wu, Runzhe Zhan, Derek F. Wong, Shu Yang, Xinyi Yang, Yulin Yuan, and Lidia S. Chao. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/b61bdf7e9f64c04ec75a26e781e2ad51-Paper-Datasets_and_Benchmarks_Track.pdf Detectrl: Benchmarking LLM -generated text det...

    2024

  44. [52]

    Xianjun Yang, Wei Cheng, Yue Wu, Linda Petzold, William Yang Wang, and Haifeng Chen. 2023. Dna-gpt: Divergent n-gram analysis for training-free detection of gpt-generated text. arXiv preprint arXiv:2305.17359

    2023 arXiv

  45. [53]

    WJ Youden. 1950. Index for rating diagnostic tests. Cancer, 3(1):32--35

    1950

  46. [54]

    Sungduk Yu, Man Luo, Avinash Madusu, Vasudev Lal, and Phillip Howard. 2025. Is your paper being reviewed by an LLM ? a new benchmark dataset and approach for detecting ai text in peer review. arXiv preprint arXiv:2502.19614

    2025

  47. [55]

    Xiao Yu, Kejiang Chen, Qi Yang, Weiming Zhang, and Nenghai Yu. 2024 a . https://doi.org/10.18653/v1/2024.emnlp-main.885 Text fluoroscopy: Detecting LLM -generated text through intrinsic features . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language P...

    2024 doi

  48. [56]

    Xiao Yu, Yuang Qi, Kejiang Chen, Guoqiang Chen, Xi Yang, Pengyuan Zhu, Xiuwei Shang, Weiming Zhang, and Nenghai Yu. 2024 b . https://proceedings.neurips.cc/paper_files/paper/2024/file/1d35af80e775e342f4cd3792e4405837-Paper-Conference.pdf Dpic: Decoupling prompt and intrinsic c...

    2024

  49. [57]

    Ying Zhou, Ben He, and Le Sun. 2024. https://aclanthology.org/2024.lrec-main.739/ Humanizing machine-generated content: Evading AI -text detection through adversarial attack . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Reso...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.