{"id":"e9c95b93-1515-4cf1-9841-74a1ece278c1","arxiv_id":"2607.26221","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Zero-shot Gemma 4 classification separates AI from human poems at 90% weighted F1, while human evaluators reach only 45% accuracy, and the study catalogs the linguistic attributes behind correct and incorrect detections.","lead":"AI poems generated by three LLMs in a few-shot style-mimicry setup are classified with a zero-shot LLM (Gemma 4-31B) at 90% weighted F1, well above traditional detectors and human judges. The paper also identifies poem attributes—rigid rhyme and clichéd metaphor for AI, irregular grammar and idiosyncratic imagery for human poems—that drive correct and incorrect detection.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot classifier result is vulnerable to training-data leakage: the next-line completion check in §III-D1 only probes 100 human lines, never AI-generated poems, and cannot rule out memorization that persists without exact continuation recall.","rationale":"The paper assembles a substantial dataset and a reasonable pipeline; the human survey and lexical analyses are useful. The zero-shot classification result, however, is only as strong as the guarantee that Gemma has not seen the test poems. The completion test in §III-D1 is the sole evidence, and it is both underpowered and asymmetric. I agree with the reader that this is the weakest assumption; I would add that the AI side is never tested and that no completion-test results are reported. The proposed post-cutoff evaluation would settle the matter. If the fresh evaluation reproduces ~90% weighted F1, the paper's central claim is supported; if not, the conditional verdict should become reject. Given current evidence, the reader's CONDITIONAL verdict remains appropriate.","tokens_in":12038,"tokens_out":5516,"duration_ms":60980,"concrete_test":"Build a leakage-controlled test set that postdates Gemma 4's training cutoff: (i) collect, after the cutoff date, new human poems from live author-consented platforms with publication timestamps; (ii) generate new AI poems after the cutoff using the same three generators and the same few-shot prompt format, but with no overlap with the original few-shot examples; (iii) run the exact §III-D2 classification pipeline on this fresh set and compare weighted F1 to Table IX. If weighted F1 drops by more than 5 points or human recall falls below the Table IX value, the zero-shot claim is compromised. Also report the outcome of the original next-line check (number of lines completed, authors discarded) as a secondary diagnostic.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The 90.0% weighted F1 in Table IX depends on the claim that Gemma 4-31B never saw the test poems. The only evidence is §III-D1: Gemma was given 100 random lines from human poems and asked for the next line; authors with any correct continuation were discarded. This test is inadequate in two ways. First, exact next-line generation is far stricter than what is needed for leakage to inflate classification: an LLM can have memorized a poem's surface statistics, lexical choices, or stylistic patterns without being able to regenerate an exact next line. Second, the test samples only human-authored poems. The AI corpus (4,539 poems generated by GPT-OSS-120B, Llama-3.3-70B, Qwen-3-32B under few-shot prompting) is never probed, even though Gemma's training data could plausibly contain those models' outputs or their web mirrors. No results of the completion test are reported (how many of the 100 lines were completed correctly? how many of the 11 authors were discarded?), so the reader cannot verify that any data was actually removed. Without a credible contamination check, the 3.5-point F1 advantage over Log-Likelihood in §IV-D/E may reflect memorization rather than generalizable zero-shot detection. Since the dataset is not released, this cannot be independently checked.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper constructs a poetry corpus of 1,513 human-authored poems and 4,539 AI-generated poems from three LLMs (GPT-OSS-120B, Llama-3.3-70B, Qwen-3-32B) generated under a style-mimicking few-shot protocol. It then evaluates a zero-shot Gemma 4-31B classifier against four traditional detectors and a human survey. The main claimed result is that Gemma 4-31B achieves 90.0% overall accuracy / 0.900 weighted F1 (Table IX), outperforming the best traditional detector (Log-Likelihood) by 3.5 weighted F1, while humans achieve only 44.69% accuracy (Table VIII). The paper additionally derives qualitative 'human-like' and 'AI-like' attributes from Gemma's self-reported reasoning factors (Table XIII) and uses these to explain classification successes and failures.","tokens_in":12374,"tokens_out":3762,"duration_ms":40049,"significance":"If the central zero-shot claim and attribute characterization are valid, the paper offers a valuable contribution: a large, style-controlled poetry dataset, a reproducible zero-shot detection pipeline, a multi-detector comparison, and human-perception evidence. The dataset size (5,970 poems) is a genuine strength, and the explicit attempt to probe contamination in §III-D1 is a useful step. However, the current evidence for the zero-shot guarantee is weak, and the attribute explanation relies on unvalidated self-reports. Both issues bear directly on the headline results, so the manuscript needs substantial revision before the findings can be accepted.","major_comments":[{"comment":"The contamination check is insufficient to establish the zero-shot setting for the AI corpus. The test asks Gemma to complete the next line for only 100 human-authored lines; it never probes the 4,539 AI-generated poems, even though those outputs (or web mirrors of them) could plausibly appear in Gemma's training data. Exact next-line recall is also a much stricter condition than memorization that would inflate classification: stylistic and lexical leakage can persist without exact continuation. No results of the test are reported (how many of 100 lines were completed, how many of 11 authors were discarded). Because Table IX's 90.0% weighted F1 and the 3.5-point advantage over Log-Likelihood depend on the zero-shot claim, this is load-bearing. Please report the completion-test results in full, extend the probe to AI-generated poems (e.g., membership-inference or canary-style tests), or o","section":"§III-A2a, §III-B1, Table IX"},{"comment":"The generation protocol introduces a potential surface-level confound between classes. The Universal Style Guide requires all AI poems to have a title and explicitly prohibits em dashes (subsection III-A2a); preprocessing then removes em dashes from AI outputs (subsection III-B1). Human poems were collected from social media and literary websites and are not described as undergoing the same em-dash removal or title standardization. If titles or em dashes differ systematically between the two classes, Gemma's 90.0% accuracy may partly reflect these formatting cues rather than poetic content. Table XIII does not list title presence or em dashes among the reasoning factors, but self-reported factors need not match actual decision cues. Please report the distribution of titles and em dashes in both classes, and run an ablation (e.g., remove title lines and normalize em dashes in both classes","section":"§IV-D, Table XIII"},{"comment":"The paper's attribute characterization is based on Gemma's self-reported 'reasoning factor'—a single multiple-choice/optional-text answer about which of five criteria most influenced each prediction. This is presented in Table XIII as the explanation of why poems are classified correctly or incorrectly. LLM self-reports are not reliable causal explanations of model decisions; the model may rationalize a choice after the fact. The central claim about 'human-like' and 'AI-like' attributes therefore needs independent support, for example by ablating individual attributes (e.g., removing clichéd metaphors, normalizing grammar) and testing whether Gemma's predictions change in the expected direction, or by human annotation of the same attributes. Without such validation, the attribute findings are suggestive but not established.","section":"§IV-D, Tables IX and XII"},{"comment":"The headline comparison—Gemma 4 outperforming Log-Likelihood by 3.5 weighted F1—is reported without confidence intervals, significance tests, or effect-size variability. Given the dataset has 5,970 items, a 3.5-point difference may or may not be statistically reliable. Please provide confidence intervals for the weighted F1 (e.g., bootstrap), and a paired significance test (e.g., McNemar on the binary predictions) for the comparison with the best traditional detector. This is particularly important because the central recommendation of the paper is that Gemma is a viable alternative to existing detectors.","section":"§IV-E, Table XII"}],"minor_comments":[{"comment":"Numerous typos and grammatical slips: 'signifiacntly' (abstract), 'outpermoning' (IV-D), 'acheieved' and 'albeit' (V-A), 'reserach' (V-A), 'grammartical' (V-A), 'occured' (V-A), 'dificult' (V-A). A careful proofread is needed.","section":"References"},{"comment":"Reference numbering is inconsistent between sections: §III-E cites Log-Likelihood as [11] and LRR as [12], while §IV-E cites Log-Likelihood as [12] and LRR as [11]. Please unify and verify all citations against the reference list.","section":"§IV-C"},{"comment":"The survey description is hard to follow: 320 evaluations, 16 evaluators, 180 unique poems, 9 slots, 2 independent slots, 7 pairwise slots. The relationship between these numbers and the support counts in Table VIII (240 AI, 80 human) should be spelled out. Also clarify whether 'human misclassification rate of 83.33% for human-authored poems' is consistent with Table VIII's 42.50% human recall (which implies 57.50% misclassification), not 83.33%.","section":"Table XIII"},{"comment":"The percentages in Table XIII do not sum to 100% for each scenario (e.g., Qwen misclassified as human sums to 83.1%; GPT sums to 72.5%). State explicitly that the table reports only the top reasons and the remaining percentage is distributed among other factors.","section":"Dataset"},{"comment":"The dataset is not released. Given the contamination and confound concerns above, public release (with appropriate permissions) would substantially strengthen verifiability and is strongly recommended.","section":"Dataset"}],"recommendation":"major_revision","confidential_remarks":"The skeptical reader's contamination concern is well-founded and lands on the manuscript. The exact-next-line test in §III-D1 is not a credible zero-shot guarantee for the AI-generated portion of the corpus, and the title/em-dash confound could explain part of the performance gap. These are fixable with additional experiments and reporting, so major revision rather than rejection seems appropriate. The paper's strengths—a large stylized corpus, a multi-detector comparison, and a human survey—make it worth saving if the leakage and confound issues are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper has a genuinely useful new dataset (5,970 poems with few-shot style mimicry from three LLMs) and a striking result: Gemma 4-31B gets 90% weighted F1 on AI vs human poetry, with Qwen-3-32B's concrete imagery emerging as the main evasion tactic. The human survey (44.69% accuracy) corroborates earlier indistinguishability findings. That is the core value.\n\nBut the central \"zero-shot\" claim doesn't hold up as defended. The only contamination check is a next-line completion test on 100 lines from human poems. Even if we grant that exact next-line recall is a reasonable probe for memorization, the test never touches the AI-generated poems. Gemma's training data could plausibly contain web mirrors of those models' outputs. And no results are reported—how many authors were discarded? So the 90% could reflect leakage rather than generalizable detection. This is the load-bearing flaw.\n\nThere are smaller issues. The style guide forces titles and bans em dashes, then the cleaning pipeline strips em dashes and other symbols—so both classes are artificially formatted. The attribute analysis (Table XIII) is built from Gemma's own self-reported reasoning factors and then presented as the explanation of its behavior; that's circular in a mild way, though it still yields a useful list of candidate markers. No confidence intervals or significance tests on the main F1 differences, and the dataset/code aren't released, so no independent replication.\n\nWhat the paper does well: the lexical diversity analysis (TTR, n-gram overlap) is consistent with the classification outcome, the traditional detector comparison is informative (Fast-DetectGPT's poor showing is worth noting), and the qualitative examples in the appendix actually illustrate the claimed attributes. The Qwen finding is the kind of concrete result that could strengthen detection pipelines.\n\nIn short: a solid empirical contribution in construction, but the headline claim is oversold. I'd send this to peer review—the dataset and the evasion finding deserve scrutiny—but the letter should demand a real contamination analysis on both human and AI corpora, plus error bars. Right now I wouldn't cite the 90% as a trustworthy number, but I'd cite the dataset and the attribute list once they're cleaned up.","headline":"Useful new dataset and a plausible 90% result, but the zero-shot claim is under-supported by a weak contamination check and the attribute analysis is self-referential.","tokens_in":12883,"tokens_out":2245,"would_cite":false,"duration_ms":22224,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A zero-shot language model can distinguish AI-generated from human-written English poems with 90% accuracy, and its stated reasons expose the poetic attributes that drive correct and mistaken calls.","keywords":["Generative AI","AI Poem Detection","AI detection","Human-like Attributes","Machine-like Attributes","Poetic Integrity","Zero-shot classification","Poetry generation"],"falsifier":"Take a set of human and AI poems composed after Gemma 4-31B's training cutoff, run the same five-criterion zero-shot prompt, and compare accuracy. If accuracy drops toward the 45% human baseline, the 90% figure is leakage, not detection. A second check: apply the next-line completion test to poems known to be in the model's training data; if many pass despite being memorized, the contamination filter is invalid.","tokens_in":11958,"feed_emoji":"🤖","tokens_out":6661,"duration_ms":62384,"temperature":0.7,"pith_summary":"The paper tries to establish that AI-generated English poems are machine-detectable even when the detector is given no training on the poems. Using a zero-shot prompt, a single 31-billion-parameter language model labeled 5,970 poems as AI or human with 90.0% overall accuracy and 0.900 weighted F1, beating the best traditional detector by 3.5 points and far exceeding human evaluators, who reached only 44.7% overall accuracy. The paper further claims that the model's reasoning can be mined into stable human-like and machine-like attributes—rigid rhyme and clichéd metaphor mark AI, while grounded imagery, non-standard grammar, and colloquial text-speak mark human writing—and that these attributes explain both correct detection and evasion. If true, this gives an open, lightweight path to AI-poetry detection that does not depend on closed commercial detectors and points to the specific stylistic traits that future generators might use to hide.","feed_headline":"Zero-shot LLM spots AI-written poems with 90% accuracy","feed_subtitle":"No training needed: one language model beats statistical detectors and names the poetic traits that expose AI authorship.","key_machinery":"The load-bearing mechanism is the zero-shot classification prompt applied to the Gemma 4-31B language model. For every poem, the model receives the text plus instructions to rate five criteria on a Likert scale—clarity, literary devices, punctuation and spacing, grammar and spelling, and originality—and to output a human/AI label, a 0–100 confidence score, and a reason stating which criterion most influenced the label. This design lets the same model both classify and generate attribute evidence; the paper then aggregates modal reasoning factors across true positives, true negatives, false positives, and false negatives to define human-like and machine-like attributes. A secondary mechanism","core_discovery":"The paper's central claim is that AI-generated poems are in fact distinguishable from human-authored poems by a zero-shot language model, contrary to the common finding that human readers rate them as indistinguishable. Gemma 4-31B, prompted with five evaluation criteria plus a classification label, confidence score, and reasoning factor, achieved 90.0% overall accuracy and 0.900 weighted F1 on a dataset of 5,970 poems (1,513 human, 4,539 AI), outperforming Log-Likelihood, Log-Rank Ratio, Binoculars, and Fast-DetectGPT. The model's reasoning factors, aggregated by modal frequency, show that literary devices and originality are the most decisive criteria for correct AI identification, while p","pith_inferences":["Outside the paper: if the 90% figure holds on poems written after the model's training cutoff, zero-shot attribution could become a screening layer in creative-writing platforms and academic-integrity tools, producing an explanation for every flag rather than a bare probability.","The attribute analysis implies an adversarial cycle: as detectors learn to cue on rigid rhyme and cliché metaphor, generators will shift toward non-standard grammar and concrete imagery (the Qwen profile). A concrete extension is to run the same five-criterion prompt against newer flagship models and measure whether evasion rises.","The reported 0.67% trigram overlap between human and AI corpora suggests a cheap, explainable feature—phrase-level n-gram overlap—that any future detector could combine with the LLM's judgment; the paper leaves this as a descriptive statistic rather than a deployed detector."],"forward_implications":["A zero-shot LLM can serve as a lightweight, open-access alternative to closed commercial AI detectors, reaching 90% accuracy on poetry with no fine-tuning.","Fine-tuning the detector only on hard negatives—poems that carry misleading attributes—could improve detection without requiring a large training corpus.","The attribute list (rigid rhyme, cliché metaphor, formulaic structure as AI markers; grounded imagery, non-standard grammar, colloquial spelling as human markers) can be used as an explainable checklist for human reviewers.","Because Qwen's concrete, distinctive phrasing caused the most evasion, models that imitate human lexical diversity will be harder to detect; detectors must track phrase-level fingerprints, not just vocabulary.","Human evaluators' near-chance performance with a strong correlation between clarity and originality suggests that human-like 'polish' is a misleading cue, and criteria-based automated scoring can correct this specific bias."],"fun_headline_variants":["AI poems exposed: zero-shot LLM hits 90% accuracy","Why AI poetry is detectable: zero-shot model finds the flaws","Gemma zero-shot beats four detectors at telling AI poems apart","90% accuracy: zero-shot LLM names traits that reveal AI poetry","Literary devices and originality betray AI poems, zero-shot study shows"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that Gemma 4-31B has no prior knowledge of the human or AI poems it classifies—the paper's only evidence is a next-line completion test on 100 random lines from human authors, which does not rule out memorization of style or leaked training data.","fun_headline_variants_meta":{"raw":{"variants":["AI poems exposed: zero-shot LLM hits 90% accuracy","Why AI poetry is detectable: zero-shot model finds the flaws","Gemma zero-shot beats four detectors at telling AI poems apart","90% accuracy: zero-shot LLM names traits that reveal AI poetry","Literary devices and originality betray AI poems, zero-shot study shows"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000165,"raw_usage":{"total_tokens":1099,"prompt_tokens":765,"completion_tokens":334,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":244}},"tokens_in":509,"tokens_out":334,"duration_ms":3475,"temperature":1.0,"reasoning_tokens":244,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T00:26:31.700447+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of human and AI poems composed after Gemma 4-31B's training cutoff, run the same five-criterion zero-shot prompt, and compare accuracy. If accuracy drops toward the 45% human baseline, the 90% figure is leakage, not detection. A second check: apply the next-line completion test to poems known to be in the model's training data; if many pass despite being memorized, the contamination filter is invalid.","supporting_citations":[],"review_version":1}