Pith. sign in

REVIEW 4 major objections 5 minor 15 references

A zero-shot language model can distinguish AI-generated from human-written English poems with 90% accuracy, and its stated reasons expose the poetic attributes that drive correct and mistaken calls.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Zero-shot Gemma 4 classification separates AI from human poems at 90% weighted F1, while human evaluators reach only 45% accuracy, and the study catalogs the linguistic attributes behind correct and incorrect detections.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection Useful new dataset and a plausible 90% result, but the zero-shot claim is under-supported by a weak contamination check and the attribute analysis is self-referential. the 4 major comments →

arxiv 2607.26221 v1 pith:5E6L3CM4 submitted 2026-07-28 cs.CL

Characterizing Human-Likeness in AI Generated Poetry: A Zero-shot Classification Study

A. N. Biswas , T. Tabassum , A. A. Shohid , R. M. Mou , A. A. Esha , F. Sadeque , A. Ahmed This is my paper
classification cs.CL
keywords Generative AIAI Poem DetectionAI detectionHuman-like AttributesMachine-like AttributesPoetic IntegrityZero-shot classificationPoetry generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that AI-generated English poems are machine-detectable even when the detector is given no training on the poems. Using a zero-shot prompt, a single 31-billion-parameter language model labeled 5,970 poems as AI or human with 90.0% overall accuracy and 0.900 weighted F1, beating the best traditional detector by 3.5 points and far exceeding human evaluators, who reached only 44.7% overall accuracy. The paper further claims that the model's reasoning can be mined into stable human-like and machine-like attributes—rigid rhyme and clichéd metaphor mark AI, while grounded imagery, non-standard grammar, and colloquial text-speak mark human writing—and that these attributes explain both correct detection and evasion. If true, this gives an open, lightweight path to AI-poetry detection that does not depend on closed commercial detectors and points to the specific stylistic traits that future generators might use to hide.

Core claim

The paper's central claim is that AI-generated poems are in fact distinguishable from human-authored poems by a zero-shot language model, contrary to the common finding that human readers rate them as indistinguishable. Gemma 4-31B, prompted with five evaluation criteria plus a classification label, confidence score, and reasoning factor, achieved 90.0% overall accuracy and 0.900 weighted F1 on a dataset of 5,970 poems (1,513 human, 4,539 AI), outperforming Log-Likelihood, Log-Rank Ratio, Binoculars, and Fast-DetectGPT. The model's reasoning factors, aggregated by modal frequency, show that literary devices and originality are the most decisive criteria for correct AI identification, while p

What carries the argument

The load-bearing mechanism is the zero-shot classification prompt applied to the Gemma 4-31B language model. For every poem, the model receives the text plus instructions to rate five criteria on a Likert scale—clarity, literary devices, punctuation and spacing, grammar and spelling, and originality—and to output a human/AI label, a 0–100 confidence score, and a reason stating which criterion most influenced the label. This design lets the same model both classify and generate attribute evidence; the paper then aggregates modal reasoning factors across true positives, true negatives, false positives, and false negatives to define human-like and machine-like attributes. A secondary mechanism

Load-bearing premise

The load-bearing premise is that Gemma 4-31B has no prior knowledge of the human or AI poems it classifies—the paper's only evidence is a next-line completion test on 100 random lines from human authors, which does not rule out memorization of style or leaked training data.

What would settle it

Take a set of human and AI poems composed after Gemma 4-31B's training cutoff, run the same five-criterion zero-shot prompt, and compare accuracy. If accuracy drops toward the 45% human baseline, the 90% figure is leakage, not detection. A second check: apply the next-line completion test to poems known to be in the model's training data; if many pass despite being memorized, the contamination filter is invalid.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A zero-shot LLM can serve as a lightweight, open-access alternative to closed commercial AI detectors, reaching 90% accuracy on poetry with no fine-tuning.
  • Fine-tuning the detector only on hard negatives—poems that carry misleading attributes—could improve detection without requiring a large training corpus.
  • The attribute list (rigid rhyme, cliché metaphor, formulaic structure as AI markers; grounded imagery, non-standard grammar, colloquial spelling as human markers) can be used as an explainable checklist for human reviewers.
  • Because Qwen's concrete, distinctive phrasing caused the most evasion, models that imitate human lexical diversity will be harder to detect; detectors must track phrase-level fingerprints, not just vocabulary.
  • Human evaluators' near-chance performance with a strong correlation between clarity and originality suggests that human-like 'polish' is a misleading cue, and criteria-based automated scoring can correct this specific bias.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Outside the paper: if the 90% figure holds on poems written after the model's training cutoff, zero-shot attribution could become a screening layer in creative-writing platforms and academic-integrity tools, producing an explanation for every flag rather than a bare probability.
  • The attribute analysis implies an adversarial cycle: as detectors learn to cue on rigid rhyme and cliché metaphor, generators will shift toward non-standard grammar and concrete imagery (the Qwen profile). A concrete extension is to run the same five-criterion prompt against newer flagship models and measure whether evasion rises.
  • The reported 0.67% trigram overlap between human and AI corpora suggests a cheap, explainable feature—phrase-level n-gram overlap—that any future detector could combine with the LLM's judgment; the paper leaves this as a descriptive statistic rather than a deployed detector.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper constructs a poetry corpus of 1,513 human-authored poems and 4,539 AI-generated poems from three LLMs (GPT-OSS-120B, Llama-3.3-70B, Qwen-3-32B) generated under a style-mimicking few-shot protocol. It then evaluates a zero-shot Gemma 4-31B classifier against four traditional detectors and a human survey. The main claimed result is that Gemma 4-31B achieves 90.0% overall accuracy / 0.900 weighted F1 (Table IX), outperforming the best traditional detector (Log-Likelihood) by 3.5 weighted F1, while humans achieve only 44.69% accuracy (Table VIII). The paper additionally derives qualitative 'human-like' and 'AI-like' attributes from Gemma's self-reported reasoning factors (Table XIII) and uses these to explain classification successes and failures.

Significance. If the central zero-shot claim and attribute characterization are valid, the paper offers a valuable contribution: a large, style-controlled poetry dataset, a reproducible zero-shot detection pipeline, a multi-detector comparison, and human-perception evidence. The dataset size (5,970 poems) is a genuine strength, and the explicit attempt to probe contamination in §III-D1 is a useful step. However, the current evidence for the zero-shot guarantee is weak, and the attribute explanation relies on unvalidated self-reports. Both issues bear directly on the headline results, so the manuscript needs substantial revision before the findings can be accepted.

major comments (4)
  1. [§III-A2a, §III-B1, Table IX] The contamination check is insufficient to establish the zero-shot setting for the AI corpus. The test asks Gemma to complete the next line for only 100 human-authored lines; it never probes the 4,539 AI-generated poems, even though those outputs (or web mirrors of them) could plausibly appear in Gemma's training data. Exact next-line recall is also a much stricter condition than memorization that would inflate classification: stylistic and lexical leakage can persist without exact continuation. No results of the test are reported (how many of 100 lines were completed, how many of 11 authors were discarded). Because Table IX's 90.0% weighted F1 and the 3.5-point advantage over Log-Likelihood depend on the zero-shot claim, this is load-bearing. Please report the completion-test results in full, extend the probe to AI-generated poems (e.g., membership-inference or canary-style tests), or o
  2. [§IV-D, Table XIII] The generation protocol introduces a potential surface-level confound between classes. The Universal Style Guide requires all AI poems to have a title and explicitly prohibits em dashes (subsection III-A2a); preprocessing then removes em dashes from AI outputs (subsection III-B1). Human poems were collected from social media and literary websites and are not described as undergoing the same em-dash removal or title standardization. If titles or em dashes differ systematically between the two classes, Gemma's 90.0% accuracy may partly reflect these formatting cues rather than poetic content. Table XIII does not list title presence or em dashes among the reasoning factors, but self-reported factors need not match actual decision cues. Please report the distribution of titles and em dashes in both classes, and run an ablation (e.g., remove title lines and normalize em dashes in both classes
  3. [§IV-D, Tables IX and XII] The paper's attribute characterization is based on Gemma's self-reported 'reasoning factor'—a single multiple-choice/optional-text answer about which of five criteria most influenced each prediction. This is presented in Table XIII as the explanation of why poems are classified correctly or incorrectly. LLM self-reports are not reliable causal explanations of model decisions; the model may rationalize a choice after the fact. The central claim about 'human-like' and 'AI-like' attributes therefore needs independent support, for example by ablating individual attributes (e.g., removing clichéd metaphors, normalizing grammar) and testing whether Gemma's predictions change in the expected direction, or by human annotation of the same attributes. Without such validation, the attribute findings are suggestive but not established.
  4. [§IV-E, Table XII] The headline comparison—Gemma 4 outperforming Log-Likelihood by 3.5 weighted F1—is reported without confidence intervals, significance tests, or effect-size variability. Given the dataset has 5,970 items, a 3.5-point difference may or may not be statistically reliable. Please provide confidence intervals for the weighted F1 (e.g., bootstrap), and a paired significance test (e.g., McNemar on the binary predictions) for the comparison with the best traditional detector. This is particularly important because the central recommendation of the paper is that Gemma is a viable alternative to existing detectors.
minor comments (5)
  1. [References] Numerous typos and grammatical slips: 'signifiacntly' (abstract), 'outpermoning' (IV-D), 'acheieved' and 'albeit' (V-A), 'reserach' (V-A), 'grammartical' (V-A), 'occured' (V-A), 'dificult' (V-A). A careful proofread is needed.
  2. [§IV-C] Reference numbering is inconsistent between sections: §III-E cites Log-Likelihood as [11] and LRR as [12], while §IV-E cites Log-Likelihood as [12] and LRR as [11]. Please unify and verify all citations against the reference list.
  3. [Table XIII] The survey description is hard to follow: 320 evaluations, 16 evaluators, 180 unique poems, 9 slots, 2 independent slots, 7 pairwise slots. The relationship between these numbers and the support counts in Table VIII (240 AI, 80 human) should be spelled out. Also clarify whether 'human misclassification rate of 83.33% for human-authored poems' is consistent with Table VIII's 42.50% human recall (which implies 57.50% misclassification), not 83.33%.
  4. [Dataset] The percentages in Table XIII do not sum to 100% for each scenario (e.g., Qwen misclassified as human sums to 83.1%; GPT sums to 72.5%). State explicitly that the table reports only the top reasons and the remaining percentage is distributed among other factors.
  5. [Dataset] The dataset is not released. Given the contamination and confound concerns above, public release (with appropriate permissions) would substantially strengthen verifiability and is strongly recommended.

Circularity Check

1 steps flagged

Gemma's 90% accuracy is an external measurement, but the paper's attribute explanation is self-referential: 'defining attributes' are Gemma's own self-reported reasoning factors presented as discovered causes.

specific steps
  1. self definitional [Section III-D (Gemma 4 Zero-Shot Classification), Section IV-D2, Section IV-F / Table XIII]
    "These machine-like and human-like attributes produced hard positive results, which we call defining attributes, and hard negative results, which are attributes that force misclassification. ... The reasoning factor asked which of the five criteria most influenced the label, or requested another factor if none of the five applied."

    The 'defining attributes' and 'misleading attributes' are defined by Gemma's own classification outcomes and by its self-reported 'reasoning factor' (recorded during each prediction and tabulated in Table XIII). The paper then presents these self-reports as discovered attributes of the poems that 'contribute classification and misclassification.' No independent measurement links the attributes to the labels; the explanation is the classifier's self-report by construction, so the attribute characterization is self-referential rather than an external derivation. The 90% accuracy itself is measured against held-out labels and is not circular, but the attribute explanation reduces to the model's own stated reasons.

full rationale

The headline result, 90.0% weighted F1 in Table IX, is an external measurement against the poem labels and does not reduce to the paper's inputs. The comparisons with traditional detectors (Table XII) and the human survey (Table VIII) are also independent measurements. The zero-shot contamination check in §III-D1 is weak—it probes only 100 human lines and never probes AI-generated poems—so training-data leakage is a real validity threat, but an inadequate check is a validity risk, not a circular derivation. The one genuinely self-referential step is the attribute analysis: the 'defining attributes' and 'misleading attributes' are extracted from Gemma's own self-reported reasoning factor and then presented as discovered properties that drive classification. That attribution is guaranteed by construction because the attributes are the model's stated reasons; no external validation supports them. Since the central accuracy claim does not depend on the attribute attribution, the circularity is partial rather than total.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical or formal entities. Its design relies on several unverified premises: the adequacy of the contamination check, the assumption that no formatting artifact separates the classes, and the validity of the model's self-reported reasoning as evidence for attribute importance. The free parameters listed are fixed design choices that affect the difficulty of the task but are not fitted to data.

free parameters (3)
  • few-shot example count (n) = 2
    Each generation job uses exactly 2 style examples per author (Eq. 2-3). No ablation is run for n=1 or n=3, so the chosen value may affect generation diversity and detection difficulty.
  • zero-shot contamination check size = 100 human samples
    Only 100 next-line completions from 11 authors are used to verify that Gemma has not memorized the human poems (Section III-D1). This threshold is arbitrary and too small to be a reliable membership test.
  • theme pool size (20 themes) = 20
    A fixed array of 20 themes is sampled uniformly (Eq. 1, Table II). Theme choice and distribution influence vocabulary and may interact with the classifier.
axioms (4)
  • domain assumption Gemma 4-31B has no prior exposure to the human or AI poems, as verified by a 100-sample next-line completion test.
    The zero-shot classification claim (Section III-D1) depends on this. The test is a weak proxy for training-data membership because a model can memorize texts without being able to generate the exact next line.
  • domain assumption The few-shot style-mimicry pipeline produces AI poems that differ from human poems only in origin, not in prompt-induced artifacts.
    The style guide, titles, few-shot examples, and cleaning are assumed to leave no class-specific surface cue. The paper itself notes models are biased to few-shot examples and that em dashes had to be cleaned (Sections IV-A5, V-A), which suggests cues may remain.
  • ad hoc to paper Gemma's self-reported 'reasoning factor' is a valid explanation of its classification decisions.
    The attribute analysis (Section IV-F, Table XIII) counts which criterion Gemma said influenced each decision. No controlled ablation or human validation is used, so the causal role of each attribute is assumed from the model's introspections.
  • domain assumption Preprocessing removed formatting signals (titles, em dashes, headers) equally from both classes.
    The style guide required titles and prohibited em dashes for AI poems, while the cleaning pipeline described in Section III-B1 targets AI raw outputs. If titles or dash usage remain in one class, the classifier can exploit them.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Characterizing Human-Likeness in AI Generated Poetry: A Zero-shot Classification Study." pith.science (2026). https://pith.science/paper/5E6L3CM4

@misc{pith2026260726221,
  author       = {Pith},
  title        = {Pith review of: Characterizing Human-Likeness in AI Generated Poetry: A Zero-shot Classification Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5E6L3CM4}},
  note         = {Machine review of arXiv:2607.26221}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

With the advancement of AI technologies, Generative AI (GenAI) and human written text have become nearly indistinguishable. Additionally, the global standardization of AI chatbots made academic malpractice more frequent. Furthermore, existing research indicates GenAI poems are the most difficult to distinguish even without any modification thus, GenAI poems are naturally deemed human-like by modern detectors. However, the objectivity of such dissertations needs to be verified against modern detection tools but the subjectivity of poetry and the black-box nature of the modern LLMs (Large Language Models) architectures made verification of such work quite complicated. Hence, the main objective of the research is to deduce the attributes of English poetry that contribute classification and misclassification of both human and AI poems and provide corroborating or contradicting evidence to the poetry distinguishability claim. For such characterizations, we propose a Zero-shot detection pipeline with a dataset consisting of both human and AI poems to verify the distinguishability of human and AI creation and extract the aforementioned crucial attributes for accurate classification. Extraction of such attributes provides benefits in two ways: firstly, it reduces the margin of training needed as only the poems based on misclassifying attributes need to be trained and fine tuned and finally provides a critical insight to the GenAI detection dilemma to strengthen the modern detection pipelines.

Figures

Figures reproduced from arXiv: 2607.26221 by A. A. Esha, A. Ahmed, A. A. Shohid, A. N. Biswas, F. Sadeque, R. M. Mou, T. Tabassum.

Figure 2
Figure 2. Figure 2: Data generation pipeline after permission was requested, and open-access literary web￾sites where authors publish their poems. We approached the authors individually for permission to work with their poems, and 1,513 human poems were collected through this process, including 11 authors and diverse writing styles. 2) Initialization and LLM Generation Loop: For poem generation, three LLMs were selected based… view at source ↗
Figure 1
Figure 1. Figure 1: Overview of the proposed methodology for detecting AI-generated [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Data cleaning visualization and Qwen 3-32B. All poems were generated under the few￾shot prompt setting. Each model generated 1,513 poems, and each human poem served as a style reference during few-shot generation. From a pool of 20 universal themes, each theme was assigned randomly. 1) Data Preprocessing: A cleaning pipeline was applied to preprocess raw outputs from AI-generated poems. First, chain-of-tho… view at source ↗
Figure 4
Figure 4. Figure 4: Theme percentage distribution across all LLM invocation calls [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Word frequency distribution visualization [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Semantic consistency analysis visualization [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 6
Figure 6. Figure 6: Theme-wise word cloud for same and different theme pairs [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 9
Figure 9. Figure 9: Inter-rater exact agreement rates by survey evaluation metric. [PITH_FULL_IMAGE:figures/full_fig_p007_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Linguistic metric correlation heatmap. Human: 0.2061) is primarily a result of imbalanced class support in the experimental corpus. The important metric for assessing human capability is recall, which is balanced yet suppressed across both AI recall at 45.42% and human recall at 42.50%. The severely low F1-score for human detection (0.2776) establishes that generative language models have attained stylist… view at source ↗
Figure 11
Figure 11. Figure 11: Number of False Negative classification produced by LLMs. [PITH_FULL_IMAGE:figures/full_fig_p008_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 5 linked inside Pith

  1. [1]

    De- tectGPT: Zero-shot machine-generated text detection using probability curvature,

    E. Mitchell, Y . Lee, A. Khazatsky, C. D. Manning, and C. Finn, “De- tectGPT: Zero-shot machine-generated text detection using probability curvature,” inProc. ICML, 2023

    2023

  2. [2]

    GenAI Content Detection Task 3: Cross-Domain Machine- Generated Text Detection Challenge,

    L. Dugan, A. Zhu, F. Alam, P. Nakov, M. Apidianaki, and C. Callison- Burch, “GenAI Content Detection Task 3: Cross-Domain Machine- Generated Text Detection Challenge,”arXiv preprint arXiv:2501.08913, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    AI-generated poetry is indistinguishable from human-written poetry and is rated more favorably,

    B. Porter and E. Machery, “AI-generated poetry is indistinguishable from human-written poetry and is rated more favorably,”Scientific Reports, vol. 14, no. 1, p. 26133, 2024

    2024

  4. [4]

    Evaluating diversity in automatic poetry generation,

    Y . Chen, H. Gr ¨oner, S. Zarrieß, and S. Eger, “Evaluating diversity in automatic poetry generation,”arXiv preprint arXiv:2406.15267, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    MERMAID: Metaphor generation with symbolism and discriminative decoding,

    T. Chakrabarty, X. Zhang, S. Muresan, and N. Peng, “MERMAID: Metaphor generation with symbolism and discriminative decoding,” arXiv preprint arXiv:2103.06779, 2021

    Pith/arXiv arXiv 2021

  6. [6]

    GPT-OSS-120B model card,

    OpenAI, “GPT-OSS-120B model card,” 2025

    2025

  7. [7]

    Llama 3.3 70B model card,

    Meta AI, “Llama 3.3 70B model card,” 2024

    2024

  8. [8]

    Qwen3 technical report and model card,

    Qwen Team, “Qwen3 technical report and model card,” 2025

    2025

  9. [9]

    Fast-DetectGPT: Efficient zero-shot detection of machine-generated text via conditional probability curvature,

    G. Bao, Y . Zhao, H. Teng, L. Yang, and Y . Zhang, “Fast-DetectGPT: Efficient zero-shot detection of machine-generated text via conditional probability curvature,” inProc. ICLR, 2024

    2024

  10. [10]

    Spotting LLMs with Binoculars: Zero-shot detection of machine-generated text,

    A. Hans et al., “Spotting LLMs with Binoculars: Zero-shot detection of machine-generated text,” 2024

    2024

  11. [11]

    J. Su, T. Zhuo, D. Wang, and P. Nakov. 2023. ”Detectllm: Leveraging log rank information for zero-shot detection of machine-generated text.” InFindings of the Association for Computational Linguistics: EMNLP 2023, 12395–12412

    2023

  12. [12]

    Release strategies and the social impacts of language models,

    I. Solaiman et al., “Release strategies and the social impacts of language models,” 2019

    2019

  13. [13]

    Google DeepMind. 2026. ”Gemma 4 model card — Google AI for Developers.” https://ai.google.dev/gemma/docs/core/model card 4

    2026

  14. [14]

    B. Li, H. Wang, and H. Wilkinson. Poemetric: The last stanza of humanity.arXiv preprint arXiv:2604.03695, 2026

    Pith/arXiv arXiv 2026

  15. [15]

    Seeing the poem: Image-semantic detection of AI-generated modern Chinese poetry with MLLMs,

    S. Wang, F. Ye, H. Lyu, C. Gou, J. Wu, J. Yao, C. Xu, J. Luo, and D. F. Wong, “Seeing the poem: Image-semantic detection of AI-generated modern Chinese poetry with MLLMs,”arXiv preprint arXiv:2605.22654, 2026

    Pith/arXiv arXiv 2026

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.